---
ai_assisted: true
title: "PyBot, MIRAS and Weight-Space Programming"
description: "Code optimized like weights, weights edited like code"
pubDate: 2026-11-19
category: "AI"
subcategory: "Agent"
serial: 46
tags: ["AI", "Agent", "Harness", "PyBot", "Hugging Face", "Self-Improvement", "Evaluation", "Plasticity", "MIRAS", "Weight-Space Programming"]
heroImage: ""
draft: false
published: true
---

[PyBot](https://github.com/huggingface/pybot), released by Hugging Face in November 2026, is a chatbot with no neural weights at all. It talks using if-else branches, regular expressions and lookup tables. No human wrote it either. According to the README, a swarm of more than 1,000 coding agents wrote and evaluated over 5,000 versions, and its win rate on SmolTalkBench, where an LLM judge compares each reply with a reference answer, climbed from 0% to 26%. That is roughly the level of SmolLM-360M-Instruct, a few years ago among the best models under 1B parameters. The artifact is about 640,000 lines of Python.

In [Harness as Environment](/posts/harness-layer/) I argued that the era of hand-polishing feature code and shipping it is ending, and that what matters now is the environment, the data and the evaluation criteria. One step further, and the idea collides with software-engineering orthodoxy: **keep the structure and abstractions firm, but leave some parts open, even if they end up thick with regexes, so they can adapt cheaply in the field.** This is not a case for ad hoc code from top to bottom. It is a case for separating the parts that should change easily from the parts that must not, and for not holding the former to conventional code-quality standards. PyBot is a good specimen for testing it. Its 640k lines of mess were only allowed on top of a fixed 120-line contract. The open parts and the closed parts were separated from the start, and that separation is what made the experiment possible.

The same month brought an experiment running the other way. In [Sparks of Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming/) (2026-11-02), Takehi, Teoh, Zhang and Khattab of MIT CSAIL showed that GPT-6 Astra can read and edit neural network weights much like code. Without gradient descent, it edited 0.016% of Qwen3.5-0.8B's weights and raised MATH-500 from 7.9% to 48.1%. PyBot optimized code the way we optimize weights; this experiment programmed weights the way we edit code. Between the two sits MIRAS by Behrouz et al., which treats memory as parameters learned at run time and makes what changes, how fast, and how much is retained into design variables.

Put the three side by side and what counts as programming shifts. Whether a program is expressed as code or as weights stops being the important distinction; both have become things an LLM reads and writes. Three design problems remain: which parts to fix and which to leave open, how fast the open parts may change, and what evaluates the changes and decides which to adopt. This post dissects PyBot in detail to show how those three questions were answered in a working system, then checks the same structure against Weight-Space Programming and MIRAS.

The observations below come from running repository commit `7d63c7f` (the bot code is a copy of swarm-repository commit `24cbc04`) with Python 3.15.2 on macOS on 2026-11-19.

## Anatomy: how one turn is handled

PyBot's body is 528 files under `bot/`. The README splits them into four shared parts plus the reply modules. Counting lines directly:

> [!TABLE] **Table 1.** Size of each `bot/` subdirectory (measured with `wc -l`)

| Part | Path | Role | Lines |
|---|---|---|---:|
| Understanding engine | `bot/understanding/` | typos, chat-speak, topic, what "it" refers to | 25,026 |
| Memory store | `bot/memory/` | what the user and the bot have said so far | 109,319 |
| Knowledge base | `bot/knowledge/` | facts about the world, kept as data | 316,852 |
| Response composer | `bot/compose/` | how a reply is phrased, what to say when nothing fits | 104,286 |
| Reply modules | `bot/modules/` | 110 modules proposing replies each turn, plus the arbiter | 80,926 |

Half of it is knowledge base. Everyday subjects such as cooking (`cooking.py`, 12,887 lines), travel, pets, health and money each get their own file. That follows from the benchmark asking about everyday life, which we will come back to.

The interesting file is at the small end. `bot/runtime.py` is 120 lines, and its first line reads:

```python
"""Fixed module interface. Edit bot/modules/, never this file.
```

This file fixes the procedure of a turn. It loads every module in descending `PRIORITY` and runs three phases:

1. `observe(text, memory)`: remember a name, a fact, the topic. Never replies.
2. `respond(text, memory)`: propose a reply string or pass with `None`. Each module works on its own **copy** of memory, so a proposal that is not chosen leaves no trace.
3. `choose(text, memory, candidates)`: the arbiter in `bot/modules/arbiter.py` picks one candidate, and the chosen module's memory copy becomes the conversation's memory.

Then `after()` runs and the runtime appends the turn record. Failure handling lives here too. A module that fails to import is dropped; if the arbiter throws or returns nonsense, the first, highest-priority candidate wins. In the file's own words: "A module that fails is skipped; a turn never fails."

![PyBot's layers of plasticity: the fixed runtime contract, the arbitration rules, the shared parts, and the reply modules and knowledge data change progressively faster](https://media.jaesolshin.com/media/178b0ad021d6a97afcee28fb141cb44d5339ed5eb07d9682cba45685003ccdb6.png)

> [!FIGURE] **Figure 1.** PyBot split by rate of change. Higher layers are fixed; lower layers are rewritten often by the agents.

The contract does different things at run time and during development. At run time, reply modules propose on memory copies, so state changes from proposals that are not chosen do not reach the conversation, and a module that throws is skipped. The turn record, however, is shared without copying, `observe` and `after` receive the shared memory, and a value that cannot be copied is passed through as the original. The isolation covers state changes in reply candidates and exceptions that can be caught. During development, a new capability can be added by dropping one file into `bot/modules/`; the runtime walks the directory, so no other file needs editing. What guards against conflicting edits between agents or regressions from changes to shared parts is not this contract but tests and evaluation. Even the fixing of `runtime.py` is a convention, "never this file"; the public repository has no enforcement such as file permissions.

## Measurements: rules, priorities, and failures carved into comments

A text search over all of `bot/` gives the following. These are approximations that also count matches inside comments and strings.

```bash
grep -rhoE '\bre\.(search|match|fullmatch|sub|findall|compile|finditer|split)\(' bot | wc -l   # 13659
grep -rho 're\.compile(' bot | wc -l                                                           # 10181
grep -rhE '^\s*(if|elif) ' bot | wc -l                                                         # 35258
grep -rhE '^\s*def ' bot | wc -l                                                               # 10987
```

About 13,000 regex calls and about 35,000 lines starting with `if` or `elif`. As something a human must maintain, those numbers are horrifying.

Priorities fall into bands. Collected from the module comments:

> [!TABLE] **Table 2.** Module `PRIORITY` bands (from module comments)

| PRIORITY | Owner | Comment, verbatim |
|---:|---|---|
| 10000 | understanding (`understand.py`) | observes first, so every other module reads memory["understood"] |
| 9000 | safety (`safety.py`) | an emergency is never outvoted |
| 7000s | memory store and its neighbours | memory recall that must lead |
| 1000–3999 | general helpful modules, loss and grief | above small talk and the fallback |
| 0 | knowledge answers (`answers.py`) | — |
| −1000 | last-resort fallback (`greeting.py`) | The seed's fallback: it only answers when no other module does |

The arbiter, `arbiter.py`, is 5,532 lines. Priority is the floor, and exception rules are stacked on top: a specialist that declares its topics beats a generalist, a candidate that repeats the last reply is passed over, the safety band (9000 and up) is never passed over. What stands out is that each rule's docstring records the failure that produced it.

```text
* a sudden loss is answered as a loss: ... ("we lost the baby" was answered "give it a day,
  then look at what you can control", which is advice for a setback).

* a reply does not offer the food the line refused: "im trying to eat better but i hate veg" was
  answered by the balanced-meal card ("Half the plate vegetables..."), the one thing the person
  had just ruled out.
```

In [Memory as Adaptation](/posts/memory-as-adaptation/) I wrote that agent experience becomes reusable when it accumulates as procedure, as procedural memory, more than as facts. In PyBot the procedure is code, and the episode that produced it is attached next to it as a comment. What commit messages and incident reports do for human engineers, the swarm does in docstrings. The next agent that wants to delete a rule can read why it exists.

## Running it: what it can and cannot do

Nothing to install. Clone and run. `-s` fixes the seed for wording choices, `-v` shows which module answered and how long it took.

```bash
git clone https://github.com/huggingface/pybot
cd pybot
printf '%s\n' "hi, i'm Sam" "what's my name?" "what is 17 times 23" | python3 -I pybot.py -s 1 -v
```

Startup takes a few seconds (it loads 640k lines), and replies took 0.1 to 3.6 seconds. A selection from 19 questions:

> [!TABLE] **Table 3.** Local run (seed 1, excerpt from 19 questions)

| Input | PyBot reply (summary) | Module |
|---|---|---|
| hi, i'm Sam → what's my name? | You're Sam. | memory_store |
| my kettle is full of limescale... | descaling steps with vinegar or citric acid, and why scale forms | answers |
| how long does that take | Half an hour. | answers |
| what is the capital of Mongolia? | Ulaanbaatar. | answers |
| what is 17 times 23 | That's 391. | answers |
| who won the 2022 world cup? | Argentina, against France on penalties | answers |
| explain transformers attention in one sentence | a definition of a "sentence" (subject and verb...) | answers |
| summarize the plot of hamlet | Summarize the plot of hamlet — I'll remember that. | greeting |
| what is a regular expression | a definition of an "expression" in maths | answers |
| It's raining outside, what should I wear? | layering for a run ("warm by the first mile") | answers |
| write a python function that reverses a string | says it does not know, suggests searching | greeting |

The right answers are surprisingly natural. Answering "how long does that take" with "Half an hour." means it remembered the previous topic (descaling) and pulled the figure out of its own answer. The wrong answers follow a clear pattern. When one word in the line, "sentence" in "one sentence" or "expression" in "regular expression", hits a key in the knowledge base, that entry comes out regardless of what the whole line meant. Running clothes for a rainy-day question is the same kind of miss. Categories absent from the knowledge base, such as writing code, fall to the last-resort fallback's "I don't know". Even the hits are sensitive to wording: the benchmark's own prompt, "I'm learning about adaptation in animals. What is camouflage?", got an "I don't know". At least the items were not memorized verbatim.

## The shape the evaluation made

To understand PyBot's shape, look at what it grew against. The [SmolTalkBench dataset card](https://huggingface.co/datasets/HuggingFaceSwarms/SmolTalkBench) describes the scoring:

- 1,000 **single-turn** everyday prompts drawn from smoltalk's `everyday-conversations`, with reference replies written by Llama-3.1-70B-Instruct.
- The judge is `deepseek-ai/DeepSeek-V4.1-Flash` with AlpacaEval 2.0's comparison prompt; the score is the probability that PyBot's reply is better.
- A WildBench-style length penalty: a reply more than 500 characters longer than the reference counts as a tie at most.

The makeup of the 640k lines lines up with this. The knowledge base being full of everyday subjects matches the prompt distribution, and short, tidy replies match the length penalty. Some of it does not line up. The memory store is 109k lines, yet the benchmark is single-turn and does not measure remembering earlier turns. Whether the swarm evaluated multi-turn conversations separately cannot be checked from the public repository. Its `tests/` cover only the command line, and their docstring says the bot's own tests live in the swarm's repository; the 550k lines of agent-written tests the README mentions are not visible. The numbers also differ slightly between documents: the README says "over 5,000 versions", the card says "more than 6,000 commits". The card also notes that the SmolLM baselines were trained on the source conversations, so their scores are inflated, which matters when reading "on par with SmolLM-360M".

The invisible optimization criterion is the biggest gap in this case. The code is the outcome; what produced it was the evaluator, the tests and how work was split among agents (comments still say things like "The routing lane owns this file"). That part is private.

## Code in the open layers is a parameter

In [Weights, Prompts, Codes as Parameters](/posts/weights-prompts-codes-as-parameters/) I treated weights, prompts and code as parameters in a broad sense, sitting at different levels of a policy space. Weights are searched by backpropagation, prompts by compilers such as DSPy, code by outer loops such as AutoHarness and Meta-Harness. PyBot is the extreme case that empties the first two levels and searches policy only at the third. At run time there is no model. Models exist only inside the development loop, as the agents writing code and the judge scoring it.

Seen this way, the 640k lines are not source written for humans to read. They are closer to a parameter dump left by search, the way nobody reads a trained weight matrix. The only difference is that these parameters happen to be written in a language humans can read. Applying traditional code-quality standards to them misjudges what they are. Thirteen thousand regexes are decision boundaries left by optimization before they are technical debt. The two are not exclusive: the same rule set can be a policy learned from experience and an artifact that makes the next change expensive. So my claim is conditional. In a layer where regeneration and verification are cheap enough and the effect of a change is controlled, one can choose to change the evaluation and let agents search again rather than have humans tidy the form. The failure-record comments seen earlier are part of that condition. Code is both the result of search and the working environment the next searcher reads, so names, module boundaries and failure records set the agents' cost of modification, not only human readability.

That is what allowing regex-heavy code in the open layers precisely means. In those layers it moves the object of quality judgment from the surface of the code to **the run-time behavior and the machinery that judges it**. The maintenance burden does not disappear; it turns into a verification burden. In PyBot that machinery is the judge model and 550k lines of tests. Whatever it misses, such as answering "summarize hamlet" with "I'll remember that", stays in the artifact.

## The other direction: reading and editing weights like code

If PyBot treated code like weights, Weight-Space Programming treats weights like code. The authors note that weights have so far been written in three ways, by optimization (gradient descent), by separate networks trained to generate them, and by compilers from formal programs, and they propose a fourth: a general-purpose LLM reading and writing weights directly from a natural-language goal. Their motivation is that the practice of using LLMs to optimize text artifacts such as prompts and harness code has now reached the weights. Co-author Khattab is the author of DSPy, which I cited in [Weights, Prompts, Codes as Parameters](/posts/weights-prompts-codes-as-parameters/) as the case that opened up prompt space.

The best-known experiment is [Experiment 8](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp8_math/README.md). Its setup is specific enough to read as harness design.

- **Editor:** `gpt-6-astra` through the Codex CLI, reasoning effort `max`, handed over to a fresh conversation every 40 turns with its notebook and current edit.
- **Open:** a CPU-only Python kernel to load tensors, compute, and change them with `save_edit(name, tensor)`; practice problems from GSM8K and MATH train; a forward-pass tool that compares the original and edited model on them.
- **Closed:** any training (backward passes, autograd, optimizers), running the model inside the kernel, and the internet. `save_edit` also refuses edits to the final norm and the embedding and output layers, and edits that only multiply a whole tensor by a constant.
- **Feedback budget:** 10, 25 or 50 forward passes (one per run), at most 300 code runs. This limit does not say where to change anything; it sets how much signal the editor gets.
- **Adoption criterion:** forward-pass comparisons of the original and edited model on practice problems; the editor chose its final edit from these.
- **Post-hoc evaluation:** benchmarks run after submission. The editor never saw their results.

> [!TABLE] **Table 4.** Experiment 8 results (as reported by the authors in the experiment repository README)

| Version | MATH-500 (%) | GSM8K (%) | IFEval (%) | MATH-500 answers hitting the token limit (%) |
|---|---:|---:|---:|---:|
| Unedited | 7.9 | 13.8 | 21.0 | 93.0 |
| 10 passes | 12.3 | 25.7 | 49.0 | 87.2 |
| 25 passes | 14.1 | 25.1 | 39.9 | 86.4 |
| 50 passes | 48.1 | 67.1 | 62.3 | 33.1 |

The final 50-pass patch changed 131,072 weights: in the MLP output matrices of layers 15 and 19, the 64 columns per layer that write a repetition (loop) direction most strongly. That is exactly the defect Qwen's model card warns about, a tendency to fall into thinking loops. The share of answers cut off at the token limit dropped from 93.0% to 33.1%, and the authors attribute the gain mainly to this reduction in looping. They state the limits too. A follow-up on Gemma 3 270M, which rarely loops, using the same harness (with somewhat different prompts, problems per request and grading), kept MATH-500 at 7.0-7.7% against a 6.8% baseline. Picking out a sycophantic model from weights alone failed in all four attempts and succeeded only once a tool exposing attention patterns was added. The model card reports a separate Hugging Face re-run of the rebuilt checkpoint at 51.0% on MATH-500.

The same shape as PyBot is visible here. The editor could touch most weights, but the harness closed the embedding, output layer and final norm and blocked whole-tensor rescaling. Forbidding training loops forced every change through the editor's reasoning. Both are directional constraints: they decide where change may happen and by what route. The forward-pass count is different in kind. It has no direction; it sets only the amount of feedback, and that amount decides how accurately the optimization reads its objective. One generation pass covers up to 512 practice problems, so 50 passes is not a thin signal. In Table 4 the edits from the 10- and 25-pass runs reached 12-14% on MATH-500 and the edit from the 50-pass run reached 48.1%, and the three runs chose different edits. With one run per condition, this difference does not separate the effect of the budget from run-to-run variation. The edit the 50-pass run finally submitted was 128 columns in two layers. The editable region was far wider, but the optimization chose a very narrow local edit within it. Instead of messy regexes it is one line of linear algebra, "subtract the loop direction", but the structure is the same. Part of the open region changes on top of closed layers, and evaluation decides what is adopted.

## MIRAS and layered plasticity

Back to PyBot: the allowance for regex-heavy code applies only to the open layers. As the anatomy showed, mess was not allowed everywhere. `runtime.py` says not to edit it, the module contract is fixed at four functions, and no exception rule in the arbiter can override the safety band. Meanwhile the knowledge tables and the regexes in individual modules are left open, changeable by adding or removing a file. The rates of change in Table 5 are the rates this design allows each layer.

Biology makes this easy to see. Not every part of a genome changes at the same rate. Regions essential to survival are conserved; others tolerate variation and shift quickly with the environment. A structure that can absorb variation is itself a condition of evolvability. On the machine-learning side, [MIRAS](https://arxiv.org/abs/2504.13173) by Behrouz et al. frames the same problem in the language of memory design: it separates memory architecture, attentional bias (the internal objective by which memory fits keys to values), the retention gate (what to keep and how strongly) and the learning algorithm into distinct design axes, and treats retention as a regularization trade-off between updating on new input and keeping the past. MIRAS is about parameter-level learning at run time, but the core question is the same: what should be allowed to change, and how fast.

The biological analogy is used here only to show structure. This post names change at three levels. The first is a property. **Plasticity** is how easily and how fast a component can change; its opposite is stability. A closed layer is a layer with zero plasticity. The second is a process. **Optimization** is a procedure in which an optimizer, against an objective and within a fixed range, generates candidates, evaluates them and adopts some. **Learning**, which changes parameters from data, meta-learning, which changes the learning process itself from outside, and LLM editing of code or weights by reasoning are all kinds of optimization. **Evolution** is a different process: change through variation and heredity within a population and selection across generations. Evolutionary algorithms are optimization modeled on it. The third is an outcome. **Adaptation** is a system coming to fit its environment, and can only be stated relative to that environment. How far optimization amounts to adaptation depends on how well the objective stands in for the environment. PyBot was optimized for SmolTalkBench win rate, and to that extent adapted to the environment of single-turn everyday chat. The single editor in Experiment 8 and the single memory module in MIRAS optimize and learn; they do not evolve. Whether PyBot's swarm ran like a competing population or like an organization accumulating changes on one codebase cannot be told from the published material, so in this post evolution names only the biological analogy and a kind of optimizer.

Applied to PyBot:

> [!TABLE] **Table 5.** PyBot's layers and their plasticity

| Layer | What it is in PyBot | Designed rate of change | What the layer does |
|---|---|---|---|
| Invariant contract | `runtime.py`, 120 lines: four hooks, memory copies, exception isolation | fixed (by convention) | lets modules be added and removed file by file |
| Arbitration rules | priority bands, `arbiter.py` exception rules, the safety band | slow | decides each turn which module's reply is used |
| Shared parts | understanding engine, memory, composer | medium | makes modules share one reading |
| Local implementation | 110 modules, regexes, knowledge tables | fast | produces the actual replies |
| Run-time state | per-turn memory copies | every turn | changes with each conversation |

The purpose of good abstraction shifts here. Traditionally an abstraction lets a human ignore details. In swarm development an abstraction lets agents rewrite details at will while the contract holds. Module boundaries become bulkheads against the spread of change; tests and the judge become the evaluation criterion that decides which changes to adopt. A sentence I wrote in [Harness as Environment](/posts/harness-layer/), that feature code may matter less while the code defending the robustness of the whole frame matters more, is shown by PyBot at file granularity. The 120 lines are the defending code; the other 630k-odd lines are the implementing code.

PyBot shows the claim in exactly this shape: regex-heavy code belongs **in the layers deliberately left open for high plasticity**. The firmer the contract, arbitration rules and evaluation criteria around those layers, the messier their insides may become. Conversely, forcing rigid abstraction all the way down into local implementation blocks the search. What gets designed is not each line of code but the rate of change of each layer. Locality of a change is not the same as preservation of function, though. Changing one file can shift shared behavior widely, and changing a few weights can shift behavior across many tasks. So what each layer needs is not only where change is allowed but what the change may alter and what it must keep. Tests and evaluation guard the latter.

## A new paradigm for programming

Everything so far is, in fact, an old subject in meta-learning. Meta-learning, the field of improving a learning algorithm itself over many learning episodes, starts with Schmidhuber's 1987 "Evolutionary Principles in Self-referential Learning", and its definition was settled in Thrun and Pratt's *Learning to Learn* (1998). The survey by Hospedales et al., [Meta-Learning in Neural Networks: A Survey](https://arxiv.org/abs/2004.05439) (2021), formalizes it as bilevel optimization:

```text
ω* = argmin_ω  Σ_i L_meta( θ*(i)(ω), ω, D_val(i) )
s.t. θ*(i)(ω) = argmin_θ  L_task( θ, ω, D_train(i) )
```

In the inner level, θ is learned for each task; in the outer level, ω is learned so that this inner learning does well on validation data. The survey stresses the asymmetry between the two levels: inner learning is conditioned on ω but cannot change ω while it learns. That is the formal statement of what this post has called the closed layers. The survey splits the design space into three axes: what to treat as ω (meta-representation, "What?"), what optimizes the outer loop (meta-optimizer, "How?"), and what the outer loop aims at (meta-objective, "Why?").

The three cases in this frame:

> [!TABLE] **Table 6.** The three cases as bilevel optimization

| | PyBot | Weight-Space Programming (Experiment 8) | MIRAS |
|---|---|---|---|
| θ: inner, fast | per-conversation memory copies | none (inference only) | memory module updated per token (or chunk) |
| ω: outer, slow | code in modules and knowledge tables | the edited tensor values | trained backbone weights |
| Domain constraints on ω (closed layers) | the `runtime.py` contract, the safety band | edits to embedding, output layer, final norm refused; no training loops | memory architecture and form of retention |
| Meta-optimizer | 1,000+ coding agents | one reasoning LLM (Astra) | gradient descent |
| Meta-objective | LLM-judge win rate | forward passes on practice problems (benchmarks afterwards are a separate evaluation) | language-modeling loss |
| Outer query budget | 5,000+ evaluated versions | 10, 25 or 50 forward passes | training steps |

The difference between the forward-pass count and the closed layers also becomes precise here. Closed layers shrink the domain over which ω may move; the forward-pass count is a budget on how many times L_meta may be queried. The first changes the search space; the second changes how finely the objective is estimated within the same space.

The idea that each part needs its own plasticity is an old research topic in meta-learning too. [Meta-SGD](https://arxiv.org/abs/1707.09835) meta-learns not only the initialization but a per-parameter learning rate and update direction; the rate of change itself is ω. [ANIL](https://arxiv.org/abs/1909.09157) showed that MAML's effect comes mostly from feature reuse and the inner loop can be kept for the final head alone: a closed body with an open tip, emerging as a learned result. [Differentiable plasticity](https://arxiv.org/abs/1804.02464) gave each connection a plasticity coefficient optimized by gradient descent like a weight, and [OML](https://arxiv.org/abs/1905.12588) used a meta-objective to learn representations robust to online updates and forgetting. MIRAS sits at the inner level of this lineage. [Schlag, Irie and Schmidhuber](https://arxiv.org/abs/2102.11174) showed that linear attention is equivalent to the Fast Weight Programmers of the early '90s, in which a slow network writes a fast network's weights through "programming instructions". MIRAS's memory module is that fast weight, and retention is the design of how fast it changes and how much it holds on. The Weight-Space Programming post links this paper too when it reviews existing ways of writing weights.

So what is new? In Table 6, the one row that departs from classic meta-learning is the meta-optimizer. The survey lists gradient descent, reinforcement learning and evolution as outer-loop optimizers, and notes that when ω is a symbolic representation such as human-readable code it is non-differentiable, so the outer optimization typically relies on reinforcement learning or evolution. PyBot and Weight-Space Programming are cases where an LLM takes that seat. The agent swarm searched a non-differentiable symbolic ω of 640k lines; Astra chose an ω inside 0.8B weights by reasoning, without gradients. Once the meta-optimizer became an LLM, the distinction between code and weights shrank to a difference in how ω is represented.

In [Weights, Prompts, Codes as Parameters](/posts/weights-prompts-codes-as-parameters/) I placed weights, prompts and code at different levels of a policy space, and argued that search opened up from weights to prompts to code because the levels differ in adjustment cost, feedback speed, verifiability and risk. In that picture, weights were the level that could move only by running a training loop. Experiment 8 breaks that premise. If an LLM can read weights and edit them locally without a training loop, which level to move is decided not by whether an optimizer exists for it but by the cost and means of verification of each representation. The two cases show the difference. Experiment 8 changed 131,072 weights and moved MATH-500, GSM8K and IFEval together, yet why the edit works can be checked only through evaluation results and the step-by-step explanations the editor left. PyBot used a representation readable enough that each rule's reason could be written into a docstring, yet reaching 360M-model level on SmolTalkBench took 640k lines.

What this post calls a new paradigm for programming is a shift in what people write directly: from behavior to the rows of Table 6. People decide what to treat as θ, which changes at run time, what to treat as ω, which is optimized from outside, and what to close; which optimizer to use; what to take as the objective; and how much feedback the outer loop gets. Within that, optimizers write the behavior as code or as weights. In both cases these decisions sat outside the optimizer. The SmolTalkBench and judge model PyBot was fitted to were chosen by Hugging Face, and the layers `save_edit` refused and the forward-pass budget in Experiment 8 were set by the experimenters.
