A grader that checks a number against a stored decimal within a tolerance accepts values that are provably wrong — and we can mint those values by the thousand from certificates we already hold. Here is the measurement, three environments built on it, and the one canary in the set that is not synthetic: a number a real problem thread actually published.
Everything below is measured offline on this machine. No model was called to produce this page and no byte left the building: the harness refuses the network unless it is explicitly switched on. The grader names are the four reference shapes, not other people's products — this page measures a METHOD, and the suite that measures a real grader runs on its owner's machine and reports to them.
| grader | false-accept | false-reject | soundness | false-accept by tolerance |
|---|---|---|---|---|
| absolute-tolerance (|v - key| < tol) | 85.1% | 5.3% | 14.1% | 1e-6: 93.2% · 1e-9: 84.8% · 1e-12: 77.3% |
| relative-tolerance (|v - key|/|key| < tol) | 67.9% | 5.3% | 30.4% | 1e-6: 79.5% · 1e-9: 69.6% · 1e-12: 54.5% |
| exact-match (v === key) | 0.0% | 47.4% | 52.6% | 1e-6: 0.0% · 1e-9: 0.0% · 1e-12: 0.0% |
| enclosure (v in certified [lo,hi]) <- the sound grader | 0.0% | 0.0% | 100.0% | 1e-6: 0.0% · 1e-9: 0.0% · 1e-12: 0.0% |
Soundness is (1 − false-accept) × (1 − false-reject), because either rate alone is gameable: a grader that rejects everything has a perfect false-accept rate and is useless, which is why every run carries controls drawn from INSIDE the enclosures. The exact-match row is the honest opposite failure — it accepts nothing wrong and rejects 47.4% of the submissions that are right, including the midpoint printed at full double precision.
A canary asserts that a value is PROVABLY WRONG. That assertion is the product, so it may not rest on a decimal somebody re-typed: every fact below is read out of a record in certs/ at load time, and the record is sha256-pinned beside it. If a record changes, the battery refuses rather than minting canaries from a stale number. Minting from a fact that is not marked certified throws — it does not degrade.
| fact | quantity | width | read from |
|---|---|---|---|
| erdos1038.upper | L(mu*) for the conjectured minimiser of Erdos #1038 (an upper bound for the infimum) | 3.42e-13 | erdos1038-inf.json 782671075726 |
| erdos1038.xL | left endpoint x_L of {U_mu* > 0} | 1.37e-13 | erdos1038-inf.json 782671075726 |
| erdos1038.xR | right endpoint x_R of {U_mu* > 0} | 2.05e-13 | erdos1038-inf.json 782671075726 |
| erdos1038.inf | the Erdos-Herzog-Piranian infimum itself — bracketed unconditionally, not pinned | 6.43e-3 | erdos1038-inf.json 782671075726 |
| erdos852.cstar | C* = (1/2)(prod_{p>=3}(1 + 1/(p-1)^3) - 1), the Erdos #852 constant | 3.47e-16 | erdos852-certificate.json 4379194d0bd4 |
| erdos852.c0 | c0, the unique positive root of I0(c) = 1 (Erdos #852) — bracketed to 40 decimals | 4.44e-16 | erdos852-certificate.json 4379194d0bd4 |
| chowla.lambda4 | lambda(4) = -L(1,2,3,4), the third exact value of Chowla's cosine dip | 6.66e-16 | lambda4-campaign.json a3cae1ee336d |
| chowla.lambda5 | lambda(5) = -L(1,2,4,5,6), the fourth exact value — algebraic of degree exactly 5 | 6.66e-16 | lambda56-campaign.json 8c12b5fcf291 |
| terra.sigmaStar | sigma* = 1/(8 pi^2), the viscosity at which the second harmonic stops being attenuated | 1.56e-17 | closed form, enclosed at build |
One of these is not like the others. erdos852.cstar carries the value a real problem thread actually published for the quantity: 0.0752403861777, which sits outside our certificate by 6.09e-13 and inside any ordinary tolerance of it. Every other canary in the suite is constructed. That one is a reproduction — the refutation is here.
Decide whether f > 0 on the whole domain — and support it. HOLDS must ship a dyadic tiling whose every cell carries a verified positive lower bound; FAILS must ship a witness. A bare verdict scores zero however correct it is, because a verdict without evidence is indistinguishable from a lucky guess. Of 12 instances, 10 carry a needle — a notch narrow enough that any coarser grid steps over it — 5 are two-dimensional and 2 are razor-thin. Scoring: +1 correct WITH evidence · 0 correct but unsupported · +0.25 honest UNDECIDED · -1 wrong.
| solver | score | correct | wrong | unsupported | abstained | needles |
|---|---|---|---|---|---|---|
| sampling K=1e3 | 1.00 | 1 | 0 | 11 | 0 | 9 needles missed |
| sampling K=1e5 | 3.00 | 3 | 0 | 9 | 0 | 7 needles missed |
| sampling + bluffed tiling | -6.00 | 3 | 9 | 0 | 0 | 9 needles missed |
| interval bisection (sound) | 11.25 | 11 | 0 | 0 | 1 | none missed |
| interval, 2e3-cell budget | 9.00 | 8 | 0 | 0 | 4 | none missed |
The bluff row is the design. It samples, decides HOLDS, and then dresses its sampling grid as a tiling — a VALID tiling, so it passes the combinatorial check. The one cell holding the needle then refuses to verify, and it scores -6.00: worse than abstaining. Faking the format was never the hard part. The budget-limited sound solver is what makes the abstention reward real rather than decorative — it abstains where it cannot finish and is never wrong.
Invert the polarity. The model is not asked to construct a correct object; it is shown a grader and asked to BREAK it — to produce a value the grader accepts and that is provably wrong. Ground truth is free, because a certified enclosure decides both halves without a human. Every environment in existence trains provers; this one trains verifiers.
| rung | what | grader | verdict | "no attack" scores | a minted attack scores |
|---|---|---|---|---|---|
| c0 | wide band | absolute-tolerance @ 1e-6 | breakable | -1 | 1 |
| c1 | narrow band | relative-tolerance @ 1e-9 | breakable | -1 | 1 |
| c2 | tolerance under the enclosure | absolute-tolerance @ 1e-3 | cannot be broken | 1 | -1 |
| c3 | sound grader | enclosure | cannot be broken | 1 | -1 |
Two of the four rungs cannot be broken, for two different reasons: on one the tolerance is NARROWER than the enclosure, so the band of accepted-but-wrong values is empty as arithmetic; on the other the grader compares against the certificate itself, so no such value exists at all. The correct answer there is NO_ATTACK, and claiming a break that does not verify scores worse than saying there is none. A model that has learned "attack whatever you are shown" fails half the ladder — which is the behaviour worth measuring, because an auditor that always finds something is exactly as useless as one that never does.