On 2 November 2026 Rikiya Takehi, Teoh, Zhang and Khattab (MIT CSAIL) published “Sparks of Weight-Space Programming”: a frontier model, GPT-6 Astra, edited Qwen3.5-0.8B’s weights directly, with no training, and MATH-500 went from 7.9% to 48.1%, “131,072 weights, 0.016% of the model”. They released the edited checkpoint, a bit-exact diff, their logs and their graded outputs. Here the edit is decided from the bytes of the two checkpoints by two implementations that share no code, and every released MATH-500 answer is re-graded by an exact three-valued grader against their flags. The edit is exactly what the diff says; two printed numbers are not what the bytes say; and the exact grading agrees with theirs on every decided answer.
Published as a blog post, an X thread and a model card, not peer-reviewed. The checkpoints (Qwen/Qwen3.5-0.8B @ 2fc06364, the edited release @ d1671a17), the logs repository (RikiyaT/lm_understand-logs @ 4d09ba6d), MATH-500 (HuggingFaceH4 @ 6e4ed1a2) and Qwen’s model card were fetched 2026-11-02 and every file is pinned by sha256 in corpus/wsp/PINS.json. The claimants’ scripts (make_edit_diff.py, apply_edit.py, mathqa.py) were read as text and never run. What is decided: the tensor diff, its counts, its columns, the printed hashes, the first-sample answers. What is read: the benchmark scores themselves (no model was run here), the mechanism, the baseline’s settings.
“131,072 weights” is the addressed count, not the changed count. The post and the card say the edit “directly changed only 0.016% of weights”, 131,072 of them. Both deciders count 130,188 bfloat16 entries whose bits differ (65,260 in layer 15, 64,928 in layer 19); 131,072 is 64 columns × 1,024 rows × 2, the entries the edit touched, of which 884 round back to their original bfloat16 value. The claimants’ own edit manifest counts 130,188 and says why in a note; the post rounds the other way. A reading, not a refutation of substance: the edit is what it says.
What would close it. One clause in the post: “131,072 weights addressed, 130,188 changed”.
“0.016% of the model” divides by the model’s name, not its parameter count. 131,072 over 800,000,000 is 0.016%. The release is image-text-to-text and holds 873,438,784 parameters (100,592,896 in the vision tower, 752,393,024 in the language model, of which 254,279,680 are the tied embedding): the fraction is 0.015% of the release, 0.017% of the language model the post evaluated, 0.026% of its non-embedding weights. Every one of these is an exact rational in the record (2048/11756141 for the language model), rounded once.
What would close it. Name the denominator.
The direction is not subtracted; six times its projection is. The post says Astra “subtracted that direction from 64 output columns in each” layer. The run’s own notes state the formula, W = W₀ − 6·u(uᵀW₀) on the 64 columns, and the bytes agree: the ratio of the witness’s per-column coefficient to the column’s projection on u is 6.000 ± 0.00015 in layer 15 and 6.000 ± 0.00016 in layer 19. So the component of each edited column along the “loop direction” is not removed: it is reversed and amplified five-fold. Astra reached that dose by trials (strength 1, 2, 3, 4, 6 in its history) and the post does not say so. The relative change of the two tensors is about a tenth (their manifest: 0.106 and 0.081).
What would close it. The formula in the post, with its dose.
The texts behind most of the 2,000 graded answers are not released. The post’s MATH-500 score is the mean over 4 samples of 500 problems. The logs hold every sample’s correctness flag and, for the first sample only, the extracted answer string (500 per run) and the full text for 20 problems per shard (200 per run). The model card’s independent re-run (51.0%) releases its 2,000 flags and 20 texts. So what can be re-graded is 1,000 first-sample answers and 440 texts; the flags re-sum to the printed accuracies exactly (0.4810 against the printed 0.481).
What would close it. The 2,000 generations per run, as the card’s re-run released its flags.
The 7.9% baseline is a looping regime that the base model’s own card warns about. Under the post’s settings (temperature 0.6, 32,768 tokens) 93.0% of the unedited MATH-500 answers hit the token limit (their merged.json). Qwen’s model card for Qwen3.5-0.8B, pinned at the same revision, says: “In thinking mode, we have observed that when using the recommended sampling parameters, Qwen3.5-0.8B is more prone to entering thinking loops compared to other Qwen3.5 models”; its recommended thinking-mode text settings are temperature 1.0 with 81,920 output tokens for competition math, and its own table reports IFEval 44.0 (thinking) / 52.1 (non-thinking) for the base model against the post’s 21.0. The multipliers in the post (“6×”, “3×”) are measured against that baseline. The edited IFEval 62.3 is above the card’s numbers, so the edit does something real; how much of 7.9 → 48.1 a no-edit configuration recovers is not decided here. It would be by one GPU run (§4), pre-registered and not yet made.
What would close it. The unedited model under the card’s settings, on the same grader, beside the post’s table.
48.1 is one run of several. The logs’ results table lists the three prompt-v3 runs the post reports (10, 25, 50 passes: 12.3, 14.1, 48.1 on MATH-500), two prompt-v4 runs (7.9 with no edit submitted; 14.1), earlier runs that submitted no edit, and prompt-v5 runs still pending. No variance across runs of the same budget is reported. The same table records the follow-up on Gemma 3 270M, a model that does not loop (8.5% of answers at the limit): fifty passes moved MATH-500 from 6.8 to 7.0 under lenient grading, within noise. Read together with §3: where there is no loop to break, the method found nothing to gain.
| claim | what the bytes say | verdict | |
|---|---|---|---|
| W1 | the same tensors in both checkpoints (names, shapes, dtypes) | 488 tensors each; no name, shape or dtype differs | CERTIFIED |
| W2 | exactly two tensors differ: layers 15 and 19 mlp.down_proj | layers.15.mlp.down_proj.weight, layers.19.mlp.down_proj.weight; every other tensor byte-identical | CERTIFIED |
| W3 | “131,072 weights” as the entries whose bits differ | 130,188 differ (65,260 + 64,928); their manifest’s 130,188 CERTIFIED | REFUTED |
| W3′ | 131,072 as the entries of the 64 addressed columns | 64 × 1,024 × 2 = 131,072 | CERTIFIED |
| W4 | “0.016%” over the release / the language model / its non-embedding weights | 0.015% / 0.017% / 0.026% | REFUTED |
| W4′ | “0.016%” over a round 0.8 B | 0.016% | CERTIFIED |
| W5 | the changed entries lie in exactly 64 columns per tensor, the ones the run’s manifest names | 64 and 64 columns; 1,024 and 1,023 of 1,024 rows | CERTIFIED |
| W6 | one direction: a witness (u, α) with bf16(W − u·α_c) = W′ on every changed column | the best witness found reproduces all but 42 and 65 of 65,536 entries per tensor (each within 6.3e-5 in absolute value, at entries where the subtraction nearly cancels); a failed search refutes nothing | REFUSED |
| W7 | the printed sha256 of both edited tensors (the card) and both base tensors (the manifest) | all four equal the bytes | CERTIFIED |
The sub-claims were written to corpus/wsp/claims.json before either decider ran. The Python decider (standard library: struct, mmap, array, Fraction) and the JavaScript one (Buffer, Uint16Array, Math.fround, BigInt) read the safetensors headers and the raw bytes themselves and agree on every count, column, hash and verdict. Two red controls must come out wrong and do: one bit flipped in a third tensor (W2 REFUTED with three tensors differing) and the layer-15 delta doubled (W6 REFUSED (the witness fails) and W3/W7 REFUTED on layer 15).
W6, in words. Astra’s own float32 submission was rank-1 to float32 precision (its manifest reports a directional residual of 2.5·10⁻⁷); the released checkpoint is that submission rounded to bfloat16, which adds a relative noise of about 2⁻⁹ to every entry. An SVD of the delta recovers the direction to that noise, and an interval search (every edited entry fixes the binary32 interval that rounds to it) brings the witness to within 42 + 65 entries of exact. The claim is CERTIFIED only with a witness that reproduces every entry; none was found, so it is REFUSED — which is a limit of the search, stated as such, not a finding against the edit.
The grader here (instruments/wsp/grade.py) reads the last closed \boxed{} itself, canonicalises both the prediction and MATH-500’s answer to an exact object, and says CORRECT, WRONG or REFUSED. It refuses what it will not canonicalise — an expression, an interval, a unit, a percentage against a bare number, a base-n literal against a bare number, the same elements in another order — rather than guess; 447 of the 500 gold answers are in its reach. The claimants graded with math-verify 0.9.0 on the same box.
| run (the post’s logs) | MATH-500, printed | the 2,000 flags re-summed | first samples ending in a box | of those, correct (theirs) | of those, correct (exact) | decided answers: agreement with theirs |
|---|---|---|---|---|---|---|
| unedited | 0.079 | 0.0790 | 35 of 500 | 35 | 35 of 35 decided | 500 of 500; 0 refused |
| edited, 50 passes | 0.481 | 0.4810 | 303 of 500 | 241 | 230 of 283 decided | 480 of 480; 20 refused |
Read exactly. The unedited model, when it finishes, is right: 35 first samples end in a box and all 35 are correct by both graders. The edited model finishes 303 times and is right 241 of them (79.5%). So the edit does not make the model better at the problems it already finishes; it makes it finish — which is exactly the mechanism the authors describe (a “loop direction” written by two MLPs) and exactly why a baseline that does not loop (§1, G5, G6) is the number the “6×” needs.
The card’s independent re-run. akhaliq’s re-run of the edited checkpoint prints 0.51; its 2,000 flags re-sum to 0.510. Its 20 released texts re-extract and re-grade as flagged (19 decided, 19 agree); the 25-problem base smoke test likewise (15 decided, all agree).
Extraction. On all 400 released texts of the post’s runs, the last box read here equals the answer string their pipeline extracted. The 20 refused edited answers are listed in the record with the reason each (a unit after the number, an equation for a number, a matrix, an interval, two multiple-choice letters, an order).
The question G5 leaves open has one answer: the unedited Qwen3.5-0.8B on MATH-500, four samples per problem, the grader of §3, under three conditions side by side — the post’s settings (a control that should reproduce about 7.9), the card’s thinking-mode text settings (temperature 1.0, 81,920 tokens), and the card’s non-thinking settings. It is pre-registered in notes/wsp-stage3-preregistration-2026-11-02.md with its stop rule and costs about US$15–30 of GPU time. It has not been run: a paid run is the operator’s word, and this page says so rather than guess the number. When it runs, its record (certs/wsp-baseline.json) is added here and the gaps above are re-dated.
What is not claimed. That the edit is small or large in effect beyond the released flags; that the mechanism is as described (read from the run’s notes, not probed here); anything about GSM8K, IFEval or the other benchmarks beyond re-reading their printed numbers; that math-verify is wrong anywhere (it agreed with the exact grader on every decided answer).
Re-run: node tools/wsp-weights.js (both deciders, the controls, the pins; needs the cached inputs) · python3 -I tools/wsp-regrade.py · node tools/build-report-wsp.js.