Attention weights are nonnegative and sum to one, so a row is a point in a simplex — and a bar chart throws that away. Here is one real row from a tiny GPT, drawn where it actually lives, with the one thing about it that is decided rather than drawn: sharpening the temperature must move the point toward a vertex, and that is proved in exact arithmetic.
The 31 vertices of the simplex, placed on a regular 31-gon. The point is Σ pᵢvᵢ — a linear map, so the uniform row lands dead centre and a row that attends to one position lands on that position. The solid trail is the part of the temperature sweep the decision covers; the dashed part is drawn only. Two different rows can project to the same dot and no contour drawn here would be true, so nothing is drawn here but the point, its trail, and the vertices it is heading for.
Restrict the row to positions 20, 2, 13 and renormalise. That is a real 2-face of the same simplex, in barycentric coordinates, where PR = 1/Σpᵢ² is 1/Σpᵢ² — so its level sets are circles and can be drawn honestly. The rings are labelled in effective positions: 3.00 at the centre, 1.00 at a corner.
PR against β on the decided grid. The row descends strictly; the flat mutant sits at 31 forever and the (β−3)²·s mutant turns around, because its factor passes through zero. An instrument that cannot go red on those two is theatre.
The object is one causal attention row — last query, layer 0, head 0 — from a tiny GPT at seed 0, frozen once, 31 positions. It is not trained on anything you care about and it is not a claim about how language models work. It is a real row, and the point of the page is what a real row looks like when you stop drawing it as bars.
The weights are nonnegative and sum to one. That is the definition of the standard simplex Δ30, and it means every question about the row is a question about position: “this head is focused” means far from the barycentre, “it spreads” means near it, and “it sharpened” means it moved toward a vertex. Participation ratio, PR = 1/Σpᵢ², is the number of positions the row effectively attends to — 31 at the centre, 1 at a corner — and its level sets are spheres cut by the simplex, so concentration is a landscape rather than a statistic.
For the rational kernel w ∝ (1 + βs)2, PR strictly decreases across the grid, decided in exact arithmetic on BigInt fractions. That matters here and is not pedantry: consecutive values on this row differ in the fourteenth decimal place, and a float comparison of them decides nothing at all.
| β | PR (effective positions) | Σpᵢ² (the dual) |
|---|---|---|
| 0.25 | 30.998727743 | 0.032259388 |
| 0.5 | 30.994916043 | 0.032263356 |
| 1 | 30.979711562 | 0.032279190 |
| 1.5 | 30.954477885 | 0.032305504 |
| 2 | 30.919332701 | 0.032342225 |
| 3 | 30.819904622 | 0.032446564 |
| 4 | 30.682856946 | 0.032591489 |
| 6 | 30.303311868 | 0.032999693 |
| 8 | 29.798199871 | 0.033559074 |
Every IEEE double is exactly a dyadic rational, so turning the frozen scores into fractions loses nothing — it is a change of representation, not a rounding.
Two mutants are planted and must fail. Flat scores give PR = 31 at every β, exactly — no temperature sharpens a row that has nothing to sharpen. The scale (β−3)²·s passes through zero at β = 3, so its PR climbs back to 31 and descends again. Both are checked and both go red, which is the only reason to believe the green one.
Softmax is transcendental. Its curve here is ordinary floating point, calibrated against the source pack's own stored rows to worst Δ 1.42e-14 — close enough to trust for a picture and not close enough to call a proof. The sound interval enclosure of the softmax story exists in the lab this fixture came from and is not restated here as though we had re-derived it.
And the honest proportion: the decided grid runs to β = 8, where the row still has 29.80 effective positions out of 31. The dashed trail runs to β = 400, where it reaches 1.08 — a point essentially at a vertex. Almost the whole journey you can see is the part that is only drawn. The page says so with a dashed line rather than a footnote.
The fixture is fixtures/frozen_attn_scores.json from the attention-geometry pack in this operator's own research tree, carried here with its sha256 — 4f15743ea33e7972fb91ed4df412c6cd… — checked at every run, and re-derived rather than quoted: the softmax rows on this page are recomputed and then compared against the stored ones, which is the only version of “agrees with the source” worth anything. The concentration phenomenon itself is classical (majorization; Mattei & Loureiro, Dabah & Tirer). Nothing here claims otherwise. What is contributed is the room the row is drawn in, and the exact decision underneath it.
node playground/simplex/decide.js # 9 checks, exact node playground/build.js # the pages