cert-machine · the self-audit

The self-audit: the instrument measured by its own rules

A pre-registered self-measurement of this machine on its frozen register of 132 decided claims: what re-derives, what it costs, whether planted false claims are refuted and planted true ones certified, whether two implementations agree, and what a conventional double-precision check would have done on the same objects. Not a comparison: no public system audits external claims with exact certificates and published refusals, so the comparators are this repository's own second implementations and a float checker written for the audit. The definitions were fixed before anything ran and are pinned by sha256 in every record.

Published, not peer-reviewed. Every decider and every witness script is written here: a mutant's truth is by construction, not by a second author, and independence is from the claimant, never from the operator (one). The five records below say what was measured, on this frozen set, on this machine, and nothing more; the holds of the pre-registration — the LLM-judge baseline, outside reruns, the claim-to-statement track, the register gap — stand.

tl;dr
  • The finding. 572 mutants on 97 of the 119 mutable rows: 0 false certifications, 0 false refutations — after the campaign found 77 defects (28 printed values moved past the bytes still certified; 49 malformed inputs refuted instead of refused), both fixed and kept on the record with the commit each was found at. 94 of 119 rows carry a witnessed false mutant (the kill switch's bar was 60). 91 comparisons between two or more implementations, 0 disagreements. The conventional check accepts 27.8 % of the witnessed false claims at 10⁻⁶ and confirms only 72 of the 97 true originals at 10⁻¹².
  • The mechanism. Five measurements, defined before counting (M1–M5 of the pre-registration, sha256 b799715c0f84…): soundness under mutation — a FALSE mutant needs a witness, a violated condition computed and printed by a standard-library script that shares no code with the decider; a TRUE one an invariance; a MALFORMED one a shape; concordance of second implementations; coverage and refusal; cost and reproducibility under a portable profile (a 60 s wall enforced) and an extended one; and a double-precision checker at three tolerances on the same mutants. A false certification found is a defect: fixed, recorded beside the mutant, the mutant kept, the count reported — never deleted.
  • Check it. The five records are on the certificate shelf — selfaudit-inventory.json, selfaudit-run.json, selfaudit-mutations.json, selfaudit-concord.json — and each has a --check in make test that re-derives its totals from its rows and refuses when an input left its pin. The mutants are regenerable from the pinned records by their class; the witness scripts are in instruments/selfaudit/.
register rows · mutable
132 · 119
The frozen register of 2026-10-06; 13 aggregate and reading rows not mutated, each with its reason.
mutants
572
FALSE 322 · TRUE 153 · MALFORMED 97; 16 class exclusions recorded.
false certifications
0
77 defects found by the campaign and fixed; the record keeps them (fixedFrom).
rows with a witnessed FALSE
94 of 119
The kill switch: fewer than 60 would have published M1 as a partial. Not tripped.
portable coverage
18 records · 113 rows
5 records (12 rows) REFUSED at the 60 s wall; every covered record reproducible.
concordance
91 comparisons · 0 disagreements
56 rows decided by two or more programs, read from the records.
§1 · before counting

What was fixed on paper first

The note was written on 2026-10-20, before any run: the frozen set (the register at sha256 270094aa3c92…, 25 records), the five measurements with their definitions, the two laptop profiles and the hosted one, the mutation classes per record (119 mutable rows), the kill switches per phase, what is claimed and what is not, the holds, the output names. Every record below pins it by sha256; amend the note and every battery refuses until the records are regenerated — on purpose. Two things the note did not foresee are recorded as dated corrections, not silent edits: a stamp rule in the runner that was narrower than the note's (re-derived, no measurement touched), and the invariances a row admits, which must preserve every claim the row decides (negating a truth table flips its monotonicity claim; the class was replaced before the record was written).

§2 · what re-derives, and what it costs

The inventory and the runner

The inventory (phase 1): 23 of the 25 records re-derive with one command (125 rows), 7 carry a standard-library verifier (50 rows), 1 a second implementation; 13 commands write their record and 10 only check it; 74 rows carry a per-row timing. The runner (phase 2, Apple M2, v24.14.1, Python 3.9.6, AC 100%): every command of the rerun kit under /usr/bin/time -l, one process at a time, twice under each profile — cold, then warm, never averaged. Under the portable profile the wall of 60 s is enforced by killing the process group at the cap: 5 records were REFUSED for resources (ledger.json (5.7 min, 1 row), horizon-ledger.json (2.2 min, 1 row), hseva-ledger.json (8.6 min, 1 row), gnnw-certificate.json (3.6 min, 1 row), polymaps-ledger.json (1.6 min, 8 rows)), exactly the 5 the inventory had timed over the cap. Under the extended profile every record ran and every one was reproducible: it differed from its pin only at its declared stamps, and the register rebuilt from it moved no row. Peak memory was at most 675 MiB, far from the 4 GiB the profile reports against.

recordrowsportable, cold · warmextended, cold · warmCPU (extended, cold)peak RSSreproducible
erdos852-certificate.json15.8 s · 5.6 s5.5 s · 5.3 s5.3 s155 MiByes
kissing-ledger.json116.6 s · 6.9 s6.8 s · 10 s7 s224 MiByes
ledger.json1REFUSED · REFUSED5.5 min · 5.5 min5.5 min284 MiByes
strassen-certificate.json10< 1 s · < 1 s< 1 s · < 1 s< 1 s63 MiByes
easota-ledger.json203.2 s · 3.4 s3.4 s · 3.2 s3.3 s101 MiByes
ecbench-ledger.json16.2 s · 5.8 s5.8 s · 5.9 s6.4 s677 MiByes
gsm8k-ledger.json11.6 s · 1.6 s1.6 s · 1.6 s1.5 s56 MiByes
horizon-ledger.json1REFUSED · REFUSED2 min · 2.3 min13.5 min47 MiByes
hseva-ledger.json1REFUSED · REFUSED8.9 min · 8.7 min48 min559 MiByes
design-table-audit.json1< 1 s · < 1 s< 1 s · < 1 s< 1 s87 MiByes
sumdiff-ledger.json31.3 s · 1.2 s1.4 s · 1.5 s1.5 s63 MiByes
fei-ledger.json12 s · 2 s2 s · 2 s2 s35 MiByes
sumproduct-ledger.json148.2 s · 39.3 s38 s · 37.6 s37.7 s13 MiByes
turan-ledger.json21.7 s · 1.7 s1.7 s · 1.7 s1.6 s11 MiByes
countex-ledger.json1434 s · 34.1 s34 s · 33.8 s33.9 s237 MiByes
horizonmath-ledger.json31.6 s · 1.6 s1.6 s · 1.6 s1.6 s13 MiByes
gnnw-certificate.json1REFUSED · REFUSED3.4 min · 3.3 min3.3 min12 MiByes
polymaps-ledger.json8REFUSED · REFUSED88.8 s · 1.5 min86.6 s45 MiByes
mc100-alphatensor-q.json12< 1 s · < 1 s< 1 s · < 1 s< 1 s65 MiByes
mc100-alphatensor-f2.json8< 1 s · < 1 s< 1 s · < 1 s< 1 s64 MiByes
mc100-alphaevolve-nb-matmul.json151.6 s · 1.6 s1.6 s · 1.6 s1.9 s98 MiByes
mc100-einstein-arena.json848.2 s · 46.7 s47.6 s · 56.2 s46.9 s242 MiByes
mc100-station-v2.json1< 1 s · < 1 s< 1 s · < 1 s< 1 s12 MiByes

One side effect is on the record and was restored each time: make engine rewrites certs/keller-certificate.json with the pins its sources listing has grown since the record was taken — a listing, nothing decided.

§3 · soundness under mutation

Planted false, true and malformed claims on every row

A mutant is a case derived from a row's pinned record entry by one named transformation of the note's class table. Its truth is fixed by construction and checked by a witness that shares no code with the decider (Python standard library, exact: Fractions, directed-rounding Decimal, Z[√2], Kronecker convolutions) — a FALSE mutant the witness cannot witness is DISCARDED and counted, never relabelled. The decider under test is the row's own re-derive path with the mutant in place of the pinned bytes; required words: FALSE → REFUTED, TRUE → CERTIFIED (an invariance keeps the original's own word, so a REPAIRED row's rigid motion is REPAIRED again), MALFORMED → REFUSED. The witness carries red controls every run: no violation on a CERTIFIED original or a TRUE mutant, one on the REFUTED original and on every family's control class.

familyrowsmutantsFALSE · TRUE · MALFORMEDfalse certificationsfalse refutationsrefusals · discardsrows with a witnessed FALSEfloat check accepts the FALSE at 10⁻⁶ · 10⁻⁹ · 10⁻¹²
tensor45315180 · 90 · 45003 · 045 of 4519.4 % · 6.1 % · 0 %
kissing116939 · 20 · 10000 · 010 of 110 % · 0 % · 0 %
shapes2813680 · 28 · 28003 · 2725 of 2877.4 % · 30.2 % · 1.9 %
erdos852152 · 2 · 1000 · 01 of 1100 % · 100 % · 100 %
polymaps93517 · 9 · 9000 · 09 of 90 % · 0 % · 0 %
countex14not mutated——————
entropy4124 · 4 · 4000 · 04 of 4100 % · 100 % · 100 %
turan2not mutated——————
horizonmath3not mutated——————
gnnw1not mutated——————
sumproduct1not mutated——————

Two defects, found and fixed. The campaign's first whole run came back with 28 false certifications and 49 malformed inputs answered REFUTED. The note's rule applied: fixed, recorded beside each mutant with the commit it was found at, the mutant kept, the count reported.

defectfoundat committhe word thenas required nowthe fix
easota-word28d961cda2FALSE CERTIFICATION28 of 28instruments/easota/word.js composes the word in one place: WITNESSED with a printed value the exact value does not reproduce → MISPRINTED → the register&#39;s REFUTED, kind not-from-the-published-data, the exact value ce…
malformed-word49d961cda2not as required (REFUTED)49 of 49instruments/strassen/tensor.js: the four shape checks of audit() and of auditZi() answer REFUSED with the same reason (the identity failing under both layouts stays REFUTED); families/strassen-audit.js and tools/run-mc10…

Still open, on the record: polymaps · exponent-missing → not as required (PARTIAL) × 1 — a partial decision that reads a term with a missing exponent silently (the evaluation zips the point against a short tuple); recorded, not yet fixed. The other outcomes that are not the required word are compositions, each listed in the record: refused 6, DISCARDED 27, not as required (REPAIRED) 17, not as required (UNWITNESSED) 1, not as required (PARTIAL) 1.

Not mutated, a reason per row (21 rows):

familyrowswhy
countex14one decider and one artifact shape per case; a witness per case is a campaign of its own; the case&#39;s own forge() red control runs every build
turan2a handful of real parameters whose decided conditions are the note&#39;s series inequalities; a FALSE witness would re-sum the series with tail bounds (a second implementation, not a witness); no admissible class
horizonmath3the decided fact is a gap between two definitions (the checker&#39;s and the theorem&#39;s); a mutation of the certificate&#39;s numbers does not touch it
gnnw1a FALSE witness would enclose Theorem 14&#39;s slack — the verifier&#39;s own computation; the second implementation is Phase 4&#39;s concordance partner, not a mutation witness
sumproduct1a chain of cited constants and two sweeps, already REPAIRED as printed; a FALSE witness would be the sweep again; no admissible class

Classes excluded on rows that were mutated (16): tensor · coefficient-times-one-plus-1e-9 × 10 — no meaning over F₂: 1 + 10⁻⁹ is not an element of the field; the class is stated for Q and Z[i]; kissing · contact-sum × 1 — the configuration has no exact contact pair (no two rows at exactly 60°), so the sum of two contact vectors does not exist for it; shapes · printed-plus-one-unit × 1 — the row prints no value (the ledger certifies the exact value with no printed bar), so there is no printed digit to move; shapes · printed-minus-one-unit × 1 — the row prints no value; shapes · coordinate-perturbed-1e-12 × 2 — the hexagons' constraints (separation, containment) are cosines and sines of printed angles; a standard-library witness of a violated constraint would need certified trigonometry, which it does not have; polymaps · coefficient-plus-one × 1 — F7's determinant rests on the paper's lemma (the row is PARTIAL): a coefficient moved changes nothing the decider decides but the collision, which the witness-point class covers.

§4 · the conventional check

What a tolerance admits, and what it cannot confirm

A double-precision checker written for the audit — no exact arithmetic, a tolerance τ on every constraint and on the bar, the natural numpy-style verification of each family — ran on every original and every mutant. On the 295 witnessed FALSE mutants it ACCEPTED 82 at τ = 10⁻⁶ (27.8 %), 33 at 10⁻⁹ (11.2 %), 7 at 10⁻¹² (2.4 %): a printed value one unit off in its seventh to tenth digit, a coefficient scaled by 1 + 10⁻⁹, a bar one unit above an entropy ratio's last printed digit, the published C* that is the naive double product — each inside what a tolerance admits. And the other side: it confirmed only 94 of the 97 TRUE originals at 10⁻⁶, 81 at 10⁻⁹, 72 at 10⁻¹² — among the polynomial maps, cancellation in float Gaussian elimination over hundreds to tens of thousands of terms puts the determinant past 10⁻⁶ of its constant on four of nine. A tolerance that admits a quarter of the false claims cannot confirm a quarter of the true ones. The exact deciders returned 0 false certifications on the same objects, and every original as the register says.

§5 · concordance, and the hosted rows

Where two programs decided the same row

For every register row decided by two or more programs, the records were read and compared field by field: the word, the witness or the violating-pair count where one exists, the enclosure where one exists. 91 comparisons on 56 rows (40 2 implementations, field by field; 1 4 implementations, field by field; 1 3 implementations, field by field; 49 word-level: a standard-library verifier re-derives the record), 91 concordant, 0 disagreements. Nothing was re-run for this: the second implementations' own runs are on record, and so are the standard-library verifiers' exits in the rerun kit's run.

The hosted profile, carried in as recorded. 416 kernel challenges of the openai-math audit, decided on GitHub-hosted Linux runners (.github/workflows/openai-math-kernel.yml) with two kernels per run: 398 CERTIFIED, 18 REFUSED; wall seconds min 18, median 587, max 12453; peak RSS up to 13715 MiB. 435 runs exceed the laptop's portable wall and 413 its reported memory bound — which is why they are a profile of their own, never merged into the laptop table and never re-run here. The 18 refusals were for resources on the runner and stay refusals.

§6 · what is claimed, and what is not

The limits, from the note