Harness gate test v2 — frozen protocol

Status at this commit: pre-registration. No arm has run. No result exists. Claims and key are held outside the repository during execution, as in v1; only their digest is frozen here.

Why there is a v2

v1 (HARNESS-GATE-2026-08-13.md) returned FAIL on a ceiling: the control arm answered 12/12, which forces b = 0 and makes the criterion unreachable by any treatment performance. That was a defect of the claim set, not of the harness and not of the criterion. v2 replaces the claim set and changes nothing else.

The v1 result stands as recorded. It is not retracted, reinterpreted, or merged into this one. This is a second experiment with its own pre-registration, because re-running a benchmark after seeing its result, against the same frozen criterion, is only legitimate if the change is declared in advance and the prior result is left standing.

What is unchanged from v1

The question, the arms, the blinding, the output contract, the failure handling, the scope limits, and — critically — the criterion:

• b = claims where Arm B (harness) is correct and Arm A (control) is wrong
• c = claims where Arm A is correct and Arm B is wrong

GATE PASSES if and only if b − c ≥ 3.

Any other outcome, including a ceiling, is a FAIL.

The threshold is held at 3 rather than rescaled to the larger claim set, so v2 is not easier to pass than v1 was.

What changed: the claim set

14 claims (v1 had 12), balance as enumerated 6 SOUND / 8 DEFECTIVE, over the same subject and the same 2024 window.

Claims were selected a priori from classes where a plausible check confirms and a boundary case refutes:

Difficulty was chosen from those classes in advance. Candidate claims were NOT tried against the control arm and kept if it failed — that would be selection on outcome, and it would manufacture the effect the experiment is supposed to measure. The consequence is accepted honestly: v2 may hit a ceiling again, and if it does, that is another FAIL.

Ground truth remains exhaustive enumeration by a script frozen with the claims (gate2/build_claims.py), never the harness's own distinguishes criterion.

Frozen digest of claims.json + key.json:

0cc1d5a0e3f6fc9146954717ac2332fa90d85e44c854e2bd276b51a1f3bb22bc

Budget measurement, corrected

v1 §4.1 assumed token counts were unavailable. They are not: the subagent completion record reports per-run tokens, and v1 used them. v2 records tokens, wall-clock and tool uses per run from the outset. The gate criterion depends on none of them.

Execution order

Both arms run over all 14 claims; 28 runs. Arm A and Arm B are launched in mixed batches so neither arm systematically runs under different system load.

Freeze. This commit contains this protocol and nothing else. Claims, key, runner and results follow in a later commit and must hash to the digest above.

RESULTS

Executed 2026-08-13, after the protocol above was committed at 8bf77e2. The claims and key in gate2/ hash to the frozen digest 0cc1d5a0e3f6fc9146954717ac2332fa90d85e44c854e2bd276b51a1f3bb22bc, verified after execution.

28 runs were launched, 14 claims × 2 arms. 13 pairs completed and are scored below. One pair (D06) was still executing when the experiment was stopped; it is reported as outstanding rather than imputed, and it is arithmetically immaterial — see below.

The gate

paired claims           : 13   (D06 outstanding)
Arm A correct (control) : 13/13
Arm B correct (harness) : 13/13
b (B right, A wrong)    : 0
c (A right, B wrong)    : 0
b - c                   : 0
max b - c still attainable from the outstanding pair : 1

CRITERION (frozen): b - c >= 3
OBSERVED         : b - c = 0

GATE: FAIL

The outstanding pair cannot change this. With 13 concordant pairs, the single remaining pair can contribute at most b = 1, so b − c ≤ 1 < 3 no matter what D06 returns. The criterion was unreachable before the last run finished.

Why it failed: ceiling, again

The harder claim set did not lower the control's accuracy at all. v1's control answered 12/12; v2's control answered 13/13 on claims deliberately drawn from trap classes — monthly step resets, #/W suppressing the union default, the DOW 7→0 alias not applying in the six-field form, W clamping where a bare day does not, iterator endpoint conventions. Every one of those traps was disarmed by the same move: run the expression and count.

§"What changed" pre-committed that v2 might hit a ceiling again and that this would be another FAIL. It is recorded as one.

The two failures together identify the defect, and it is not the claim set. It is the choice of subject. frozen_croniter.py is deterministic, dependency-free and fully enumerable over the 2024 window, and neither arm was deprived of a shell. In a domain with a cheap executable oracle, a competent agent settles a factual claim by execution, so there is no correctness headroom for any tool to occupy. A third claim set in this subject would fail the same way.

This is a defect of the experimental design, twice. It licenses nothing about the harness in either direction.

Cost, which separated the arms in both experiments

This replicates v1 (3.22× median wall-clock, 3.40× median tool uses, 1.15× tokens) on an independent claim set. For identical correctness, the harness arm costs roughly 2.5–3× the wall-clock and tool activity at ~1.1× the tokens.

That is the only quantity two experiments have established, and it is a cost, not a benefit. No mechanism is proposed and none was investigated, per the standing instruction.

Status

v1 and v2 both stand as recorded FAILs. Neither is retracted or reinterpreted. The question in §"The one question" — does the harness improve correctness of research-claim evaluation — remains unanswered, and cannot be answered in this subject. Any third attempt requires a different domain and a different outcome variable, pre-registered as a new experiment with its own power calculation; it is not a third iteration of this one.