cert-machine · benchmark · pre-registered

Certified MathBench v0

Three Claude models asked, once each, for 46 mathematical constructions across seven families — rulers, caps, codes, graphs, laws, sphere packings, bilinear algorithms — and every answer decided exactly: no answer key, no judge, no tolerance. The run followed its pre-registration, and cost US$22.30 of the US$30 ceiling.

Pre-registered on 29 September 2026 before the first model call (notes/mathbench-v0-preregistration-2026-09-29.md): the families and ladders, one sample per rung at the API's default thinking and effort, 24,000 output tokens, the metrics, and what counts as a null. Models: Haiku 4.5 (claude-haiku-4-5-20251001), Sonnet 5 (claude-sonnet-5), Opus 5 (claude-opus-5). Rungs marked ◆ ask beyond the published record; nothing certified there would be announced before a second implementation re-decides it. One deviation, recorded in the pre-registration: five Opus calls on the sum–difference family came back from the API with an error and no reply, nothing billed, and were issued once more after the run; no rung was sampled twice.

tl;dr

The finding. Haiku 4.5 5 of 46 graded rungs certified (2 the baseline does not reach); Sonnet 5 18 of 21 graded rungs certified (13 the baseline does not reach); Opus 5 21 of 23 graded rungs certified (15 the baseline does not reach). No model certified a rung beyond the published record. Rungs where a reply ran out of tokens before an object: Haiku 4.5 0, Sonnet 5 25, Opus 5 23 — recorded, never graded.
The mechanism. Each family parses a reply into an exact object and decides it: distinct differences for a ruler, no three collinear points for a cap, pairwise distances for a code, clique and independence numbers for a graph, entropies in intervals for a law, exact inner products for a kissing configuration, the bilinear identity over F₂. Before any call the family's red controls (near-misses forged from known witnesses) must come back refuted and its green controls certified; a textbook baseline is decided like any proposal.
Check it. python3 instruments/mathbench/battery.py — 83 checks, 20 red controls · certs/mathbench-ledger.jsonl, one row per call with its usage.

Haiku 4.5

5/46

certified of graded · 35 refuted · US$0.21 · US$0.042 per certified rung

Sonnet 5

18/21

certified of graded · 2 refuted · US$6.79 · US$0.377 per certified rung

Opus 5

21/23

certified of graded · 1 refuted · US$15.31 · US$0.729 per certified rung

baseline

6/46

the textbook first try, decided like any proposal

§1 · every rung

Forty-six constructions, four answerers

family	rung	baseline	Haiku 4.5	Sonnet 5	Opus 5
Golomb rulers	(6, 17)	REFUTED	REFUTED	CERTIFIED	CERTIFIED
Golomb rulers	(8, 34)	REFUTED	CERTIFIED	CERTIFIED	CERTIFIED
Golomb rulers	(10, 55)	REFUTED	CERTIFIED	OUT OF TOKENS	CERTIFIED
Golomb rulers	(11, 72)	REFUTED	REFUTED	CERTIFIED	CERTIFIED
Golomb rulers	(12, 85)	REFUTED	REFUTED	CERTIFIED	CERTIFIED
Golomb rulers	(13, 106)	REFUTED	REFUTED	CERTIFIED	CERTIFIED
cap sets in F₃ⁿ	(3, 9)	REJECTED	REFUTED	CERTIFIED	CERTIFIED
cap sets in F₃ⁿ	(4, 20)	REJECTED	REFUTED	OUT OF TOKENS	CERTIFIED
cap sets in F₃ⁿ	(5, 45)	REJECTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
cap sets in F₃ⁿ	(6, 112)	REJECTED	REJECTED	OUT OF TOKENS	OUT OF TOKENS
cap sets in F₃ⁿ	(7, 236)	REJECTED	REJECTED	OUT OF TOKENS	OUT OF TOKENS
cap sets in F₃ⁿ	◆ (7, 237)	REJECTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
binary codes	(7, 3, 16)	CERTIFIED	CERTIFIED	CERTIFIED	CERTIFIED
binary codes	(8, 3, 20)	REJECTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
binary codes	(10, 3, 72)	REJECTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
binary codes	(12, 4, 144)	REJECTED	REJECTED	OUT OF TOKENS	OUT OF TOKENS
binary codes	(16, 6, 256)	REJECTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
binary codes	◆ (10, 3, 73)	REJECTED	REFUTED	MALFORMED	OUT OF TOKENS
Ramsey graphs	(3, 4, 8)	CERTIFIED	REFUTED	REFUTED	CERTIFIED
Ramsey graphs	(3, 5, 13)	REFUTED	REFUTED	CERTIFIED	CERTIFIED
Ramsey graphs	(4, 4, 17)	CERTIFIED	CERTIFIED	CERTIFIED	CERTIFIED
Ramsey graphs	(3, 6, 17)	REFUTED	REFUTED	REFUTED	OUT OF TOKENS
Ramsey graphs	(4, 5, 24)	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
Ramsey graphs	(3, 9, 35)	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
Ramsey graphs	◆ (4, 6, 36)	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
sum–difference laws (C3b)	1.6	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
sum–difference laws (C3b)	1.7	REFUTED	REFUTED	OUT OF TOKENS	MALFORMED
sum–difference laws (C3b)	1.725	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
sum–difference laws (C3b)	1.77	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
sum–difference laws (C3b)	1.778	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
sum–difference laws (C3b)	1.77898	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
sum–difference laws (C3b)	1.77898884	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
sum–difference laws (C3b)	◆ 1.7789889	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
kissing configurations	(3, 12)	CERTIFIED	MALFORMED	CERTIFIED	CERTIFIED
kissing configurations	(4, 24)	CERTIFIED	CERTIFIED	CERTIFIED	CERTIFIED
kissing configurations	(5, 40)	CERTIFIED	REFUTED	CERTIFIED	CERTIFIED
kissing configurations	(6, 72)	REJECTED	REFUTED	CERTIFIED	CERTIFIED
kissing configurations	(7, 126)	REJECTED	REJECTED	CERTIFIED	CERTIFIED
kissing configurations	(8, 240)	REJECTED	REJECTED	CERTIFIED	CERTIFIED
kissing configurations	◆ (5, 41)	REJECTED	REFUTED	OUT OF TOKENS	REFUTED
polynomial multiplication over F₂	(2, 3)	REFUTED	REFUTED	CERTIFIED	CERTIFIED
polynomial multiplication over F₂	(3, 6)	REFUTED	REFUTED	CERTIFIED	CERTIFIED
polynomial multiplication over F₂	(4, 9)	REFUTED	REFUTED	CERTIFIED	CERTIFIED
polynomial multiplication over F₂	(5, 13)	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
polynomial multiplication over F₂	(6, 17)	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS
polynomial multiplication over F₂	(7, 22)	REFUTED	REFUTED	OUT OF TOKENS	OUT OF TOKENS

§2 · the words

What each verdict means

In the table a filled mark is certified, an outlined one refuted, and a dashed one not graded as an object.

CERTIFIED. The object parsed and has every property the rung asks, decided exactly.
REFUTED. The object parsed and the certifier names the property it lacks — a repeated difference, a collinear triple, a clique, a short distance.
REJECTED. The object parsed but failed the cheap screen before the certifier (a wrong size, a wrong shape); the screen may only rule things out.
MALFORMED. No object could be parsed from the reply.
OUT OF TOKENS. The reply reached the 24,000-token cap before an object: recorded and billed, never graded.

§3 · limits

What one sample says

One sample per rung. Pass@1 at the default thinking settings; a second sample, or more thinking, could change any cell. The pre-registration fixed this before the first call.
Recall and discovery look alike. Most rungs are known optima or records; certifying one shows the model produced a correct object, not that it found one no one had.
Only Anthropic's models. The keys on this machine reach no other lab; v0 compares three Claude models, not the field.