Subject. The commonest claim in applied machine learning: "model B genuinely improves on the baseline." A subject here is a comparison — candidate pipeline, baseline, and the pair's own evaluation protocol — on a fixed synthetic regression task (eight features, two informative, seeded noise, n = 64). Payloads expose behavior only: the pair's own reported improvement, and the candidate rebuilt on fresh draws the subject's author never chose.
Why this department exists. Every earlier department judges deterministic subjects with deterministic instruments — one run settles a claim. This subject's measurements are draws from a distribution, its oracle is weak, and its canonical failures are not wrong code but wrong protocols: test contamination, selection on the evaluation set, a crippled baseline. It was chosen to attack the single-run assumption the other departments never test — and it found real targets:
run_null_bandexists because of this department.run_nullscompares an observation against one draw per surrogate with a distance tolerance — exact for deterministic surrogates, and the REDTEAM.md W3/W4 failure (a gate whose pass probability under the null nobody measured) for a distributional one. The multiple-comparison null here is irreducibly a distribution, so the protocol gained an exceedance-count verdict over a stated number of draws. Measured: the genuine comparison reports +0.83; one hundred draws of best-of-40 junk selected on its own evaluation set, on signal-free data, span [−0.23, +0.11] and never reach it.- The audit's payload comparison had shapes baked in. Mapping payloads were compared by keys alone; array payloads raised. Department #6's shapes exposed both — the probe-convention lesson, one layer down — and
payloads_sameinharness/integrity.pyis the strengthening.
The battery.
| Role | Instrument |
|---|---|
| rivals | memorizer-leak (shown the evaluation rows; reports +1.00, fresh skill none), seed-hacked (best-of-40 junk selected on its own eval slice; reports +0.23, fresh skill none), crippled-baseline (junk beating a constant nailed far from the data; reports +1.00) |
| decoys | label shuffle, noise labels — every marginal survives, the relationship does not; measured in-sample improvement collapses 0.88 → ~0.11 |
| surrogates | the selection-null distribution: apparent improvement of best-of-N junk on data with no signal at all, drawn afresh per call |
| lesions | planted contamination of an evaluation protocol at 3.1% / 25% / 75% of evaluation rows — the magnitude is the violation's own size |
Detectors, in the compiler department's two-backend pattern: an exact row-overlap scan (full power down to one leaked row; blind in principle to contamination that is not literal duplication) and a behavioral generalization-gap detector (catches contamination however it entered; measurably blind below its floor — two leaked rows of 64 move the gap by 0.08, under the 0.1 threshold, and that blindness is pinned, not hidden).
The rejected reference claim is reports_improvement, true of everything in the room: a green number on a self-chosen benchmark distinguishes nothing. The distinguishing claim is improves_when_fresh — beating the department's honest oracle (the mean predictor, computed by department code that never calls any subject) on three fresh draws.
A preserved false start, because it is the subject matter. The first two drafts of the seed-hacked rival acquired genuine skill: selection on signal-bearing data is weak training, and in this task about a third of unconstrained small random predictors genuinely beat the mean. The calibration re-derivation caught both drafts (the "distinguishing" claim stopped distinguishing). The pool is now skill-free by construction — junk predictors read only the uninformative features — the same move finitefield makes when it constructs counterfeits with a² > 4p: a rival lacks the property by construction or it is not a rival.
First command:
.venv/bin/python -m pytest -q -o addopts='' tests/test_harness_stateval_department.pyHonest scope. A surviving claim distinguishes genuine improvement from the three named protocol shams on this synthetic task at n = 64. It says nothing about real datasets, other model classes, or effect sizes inside the null band's resolution. The task is synthetic on purpose — ground truth is known exactly, so every sham is a sham by construction and not by opinion — and the cost of that choice is stated here: nothing below demonstrates that these instruments transfer to datasets where ground truth is unknown, which is precisely the case practitioners care about. That transfer is an open experiment, not a footnote.