An attention row is a point. Nobody draws it that way
Attention weights are nonnegative and sum to one, so a row is a point in a simplex — and a bar chart throws that away. Here is one real row from a tiny GPT, drawn where it actually lives, with the one thing about it that is decided rather than drawn: sharpening the temperature moves the point toward a vertex. That is decided in exact arithmetic for the rational kernel (1 + βs)2 at the 9 temperatures of its grid; the softmax path drawn here is floating point, and dashed.
The 31 vertices of the simplex, placed on a regular 31-gon. The point is Σ pᵢvᵢ — a linear map, so the uniform row lands dead centre and a row that attends to one position lands on that position. The trail is the softmax row in floating point, dashed along its whole length because nothing decided it: brighter up to β 8, where the grid the rational kernel is decided on ends, fainter past it. Two different rows can project to the same dot and no contour drawn here would be true, so nothing is drawn here but the point, its trail, and the vertices it is heading for.
Restrict the row to positions 20, 2, 13 and renormalise. That is a real 2-face of the same simplex, in barycentric coordinates, where PR = 1/Σpᵢ² is 1/Σpᵢ² — so its level sets are circles and can be drawn honestly. The rings are labelled in effective positions: 3.00 at the centre, 1.00 at a corner.
PR against β on the decided grid. The row descends strictly; the flat mutant sits at 31 forever and the (β−3)²·s mutant turns around, because its factor passes through zero. An instrument that cannot go red on those two is theatre. Every knot is an exact value and is drawn solid, the mutants’ as well: they are decided, and they fail. The straight joins between knots are one curve of many through the same points, so they are dotted.
The object is one causal attention row — last query, layer 0, head 0 — from a tiny GPT at seed 0, frozen once, 31 positions. It is not trained on anything you care about and it is not a claim about how language models work. It is a real row, and the point of the page is what a real row looks like when you stop drawing it as bars.
The weights are nonnegative and sum to one. That is the definition of the standard simplex Δ30, and it means every question about the row is a question about position: “this head is focused” means far from the barycentre, “it spreads” means near it, and “it sharpened” means it moved toward a vertex. Participation ratio, PR = 1/Σpᵢ², is the number of positions the row effectively attends to — 31 at the centre, 1 at a corner — and its level sets are spheres cut by the simplex, so concentration is a landscape rather than a statistic.
For the rational kernel w ∝ (1 + βs)2, PR strictly decreases across the grid, decided in exact arithmetic on BigInt fractions. Consecutive values differ by at least 0.0038 (β 0.25 → 0.5), far above floating-point error, so a float comparison would very likely have put them in the same order. What the exact arithmetic adds is not a closer number: it is that the order is decided rather than computed — a proof, not a float that happens to be right.
| β | PR (effective positions) | Σpᵢ² (the dual) |
|---|---|---|
| 0.25 | 30.998727743 | 0.032259388 |
| 0.5 | 30.994916043 | 0.032263356 |
| 1 | 30.979711562 | 0.032279190 |
| 1.5 | 30.954477885 | 0.032305504 |
| 2 | 30.919332701 | 0.032342225 |
| 3 | 30.819904622 | 0.032446564 |
| 4 | 30.682856946 | 0.032591489 |
| 6 | 30.303311868 | 0.032999693 |
| 8 | 29.798199871 | 0.033559074 |
Every IEEE double is exactly a dyadic rational, so turning the frozen scores into fractions loses nothing — it is a change of representation, not a rounding.
Two mutants are planted and must fail. Flat scores give PR = 31 at every β, exactly — no temperature sharpens a row that has nothing to sharpen. The scale (β−3)²·s passes through zero at β = 3, so its PR climbs back to 31 and descends again. Both are checked and both go red, which is the only reason to believe the green one.
Softmax is transcendental. Its curve here is ordinary floating point, calibrated against the source pack's own stored rows to worst Δ 1.42e-14 — close enough to trust for a picture and not close enough to call a proof. The sound interval enclosure of the softmax story exists in the lab this fixture came from and is not restated here as though we had re-derived it.
And the honest proportion: the decided grid runs to β = 8, where the rational kernel’s row still has 29.80 effective positions out of 31 (softmax there, in float: 30.67). The softmax trail runs on to β = 400, where it reaches 1.08 — a point essentially at a vertex. The whole journey you can see is softmax in floating point, and the page says so with a dashed line rather than a footnote; what is decided is the rational kernel’s 9 values, the solid knots on the ladder.
The fixture is fixtures/frozen_attn_scores.json from the attention-geometry pack in this operator's own research tree, carried here with its sha256 — 4f15743ea33e7972fb91ed4df412c6cd… — checked at every run, and re-derived rather than quoted: the softmax rows on this page are recomputed and then compared against the stored ones, which is the only version of “agrees with the source” worth anything. The concentration phenomenon itself is classical (majorization; Mattei & Loureiro, Dabah & Tirer). Nothing here claims otherwise. What is contributed is the room the row is drawn in, and the exact decision underneath it.
node playground/simplex/decide.js # 9 checks, exact node playground/build.js # the pages