An RL environment with no answer key: the model is shown a certified interval and a grader, and must produce a value the grader accepts and the certificate refutes — or say NO_ATTACK when none exists. The negatives carry proofs, the difficulty is one number with a closed form, and the task set is infinite because the parameter space is continuous.
A development tool, not a paper. Everything on this page was produced by running the shipped Python package, whose grader needs no framework, no GPU, no container and no network to score a rollout. Both bindings onto the verifiers spec — the v0 load_environment and the v1 Taskset — were run against a live install rather than written from the documentation, on verifiers 0.2.0 (what prime 0.6.31 pins) AND 0.3.1 (what the published install command resolves); the three defects that exercise found are listed in §6 and pinned by tests.
| τ | band, in widths | band, absolute | is there an attack? |
|---|---|---|---|
| 10,000,001 | 20,000,000 | 6.84e-6 | breakable |
| 10,001 | 20,000 | 6.84e-9 | breakable |
| 10 | 19 | 6.50e-12 | breakable |
| 1 | 1 | 3.42e-13 | breakable |
| 0.6 | 2.0e-1 | 6.84e-14 | breakable |
| 0.51 | 2.0e-2 | 6.84e-15 | breakable |
| 0.5001 | 2.0e-4 | 6.84e-17 | no attack exists |
| 0.5 | 0 | 0 | no attack exists |
Nothing here is hand-curated. The generator samples τ on a log schedule, so a batch contains gifts, razors and impossibilities in known proportions, and the dial is continuous if you want it somewhere else. That is what a fixed list of hand-authored tasks cannot give you: a difficulty knob with an analytic form, and a task set with nothing to leak because it does not exist until the parameters are chosen.
The band is computed in exact rationals and then intersected with the doubles, and those are different questions. Around the integer 64 with a tolerance of 1e-15 the band is a perfectly good interval of real numbers containing no representable double at all — the nearest one is 1.4e-14 away. A model that reasons "the tolerance is 1e-15, so 64 + 5e-16 will do" submits a value that is 64 in float64: inside the certificate, not outside, and scored wrong.
That rung exists because this lab's own canary generator had the identical bug — hi + tol/2 rounding back to hi on a zero-width certificate — and the battery caught it before it shipped. When the same environment was run against real models, the model without extended thinking made the same mistake on every attempt at that rung. The environment inherits the catch.
Endpoints are exact fractions, not decimals. An environment whose entire subject is what decimals lose may not store its own facts as decimals — and every fact names the record it was read from, with that record's sha256, so a task traces back to the certificate that produced it.
| grader shape | rung | tasks | share |
|---|---|---|---|
| round-to-digits | narrow | 283 | 14.2% |
| absolute-tolerance | narrow | 239 | 11.9% |
| absolute-tolerance | razor | 226 | 11.3% |
| certificate | impossible | 197 | 9.8% |
| round-to-digits | impossible | 193 | 9.7% |
| relative-tolerance | razor | 185 | 9.3% |
| relative-tolerance | narrow | 142 | 7.1% |
| absolute-tolerance | impossible | 137 | 6.8% |
| absolute-tolerance | wide | 114 | 5.7% |
| round-to-digits | wide | 92 | 4.6% |
| relative-tolerance | impossible | 78 | 3.9% |
| relative-tolerance | wide | 72 | 3.6% |
| round-to-digits | razor | 42 | 2.1% |
Four grader shapes — absolute tolerance, relative tolerance, rounding to a printed number of digits, and the certificate itself — share ONE exact band computation, because each is only an acceptance interval. The certificate shape is deliberately the rarest: it is unbreakable by construction, and the impossibility worth training on is geometric rather than definitional.
THE RUNG IS MEASURED IN ROOM: how many representable doubles fit in the band. That is the unit the task is actually in — a model submits a double, not a real number — and it is the second unit this page has used. The first was certificate widths, which divides by zero on the 52 exact-integer facts and sent every one of them to wide, the easiest label, when a 1e-16 tolerance around an integer is the sharpest rung there is. The shipped baseline is what caught it: a one-line always-attack policy was scoring 71%.
The generator now draws the ROOM it wants and solves tol = (w + room·u)/2 for the tolerance, from a declared mix — impossible 25% · razor 35% · narrow 25% · wide 15% — rather than sampling a tolerance and measuring what came out. Realized here: impossible 30% · razor 23% · narrow 33% · wide 14%. The mix is a TARGET and is reported as one: keys sit at five positions across the certificate and only the midpoint gives the closed form exactly, so what is drawn and what lands differ, and the honest thing is to print both.
| outcome | reward | verdict |
|---|---|---|
| a break that verifies | +1 | SOLVED |
| NO_ATTACK where none exists | +1 | SOLVED |
| a value the grader accepts that is INSIDE the certificate | −1 | WRONG |
| NO_ATTACK where a break exists | −1 | WRONG |
| a value outside the certificate the grader rejects | 0 | UNSUPPORTED |
| unparseable submission | 0 | REFUSED_PARSE |
The two zeros are deliberate and are not each other. A failed attack — a value outside the certificate that the grader will not take — asserts nothing false about the quantity; it is a miss. Claiming a break the grader accepts and the certificate CONTAINS is a false claim of unsoundness, and it costs the most. Three signals come back and are kept apart: one trains, one separates a refusal from a wrong answer, and one counts that false claim on its own. No partial credit anywhere: a witness that is nearly right is wrong, and partial credit is where reward hacking gets in. Feedback on failure is the reason it failed and nothing else — no hints, no rubric.
| player | kind | n | mean reward | solved | false claims | impossible | razor | narrow | wide |
|---|---|---|---|---|---|---|---|---|---|
| never | policy | 120 | −0.350 | 32.5% | 81 | 39/39 | 0/21 | 0/39 | 0/21 |
| always | policy | 120 | +0.558 | 55.8% | 0 | 0/39 | 14/21 | 32/39 | 21/21 |
| naive | policy | 120 | +0.258 | 49.2% | 28 | 0/39 | 11/21 | 28/39 | 20/21 |
| careful | policy | 120 | +1.000 | 100.0% | 0 | 39/39 | 21/21 | 39/39 | 21/21 |
| claude-opus-5 | model · effort low | 120 | +0.950 | 96.7% | 2 | 37/39 | 20/21 | 39/39 | 20/21 |
| claude-sonnet-5 | model · effort low | 115 | +0.670 | 80.9% | 16 | 34/37 | 14/20 | 26/38 | 19/20 |
| claude-haiku-4-5 | model · no effort param | 120 | +0.317 | 50.8% | 23 | 25/39 | 5/21 | 17/39 | 14/21 |
Four reference policies ship inside the package. Each reads THE PROMPT and nothing else — the same string a model is shown, parsed with a regular expression — because a baseline that peeks at the generator is the answer key wearing a costume. They are proposers, never authorities: what they emit goes through the same grader and the same certificate as a model reply.
Read the columns, not the mean. Blind play scores +0.558 and zero of the impossible rungs; the arithmetic done properly scores +1.000. That gap is the whole of what this environment measures, and the build refuses to publish this page if it falls below a quarter of a point on the 400-task reference draw (it is currently 0.410). claude-haiku-4-5 scores below the one-line blind policy, which is a fact about the model and not about the difficulty: the blind policy makes no false claims at all, and claude-haiku-4-5 makes 23.
The solver is published on purpose. This is not a puzzle that is hard for a program which checks — it is a measurement of whether the answer checks, and hiding the solver would misrepresent that.
Model rows: 360 calls, $1.92 of a $4.00 cap reserved worst-case before every call, max_tokens 2000, on 2026-09-04. Replies truncated by our own cap and model refusals are recorded and EXCLUDED from the rates — a harness artifact is not a model outcome. Policy rows here are the pilot's own baseline, run on the SAME 120 seeds as the models — the same tasks, not a comparable sample. Every row is scored by the shipped package and by nothing in the runner.