Freeze eval run 002, measured against 439477d

[aaddrick]

Freezes a full paired evaluation run measured against 439477d, and records three findings the run surfaced.

evals/results/runs/002-recorded-model-rerun/ holds the five result files, a manifest.json, and a derived report.md. LEDGER.md is rebuilt by scripts/ledger.py index. No harness file changes, so the findings below stay open.

The run

24 cases, 3 trials, baseline against candidate, judged blind over 2 passes. Model claude-sonnet-5. Cost $11.47: $4.28 run, $7.19 judge.

The gate passes on all five conditions.

Dimension	Weight	Baseline	Candidate	Delta	95% CI	Significant
correctness	30%	4.410	4.507	+0.097	[-0.115, +0.309]	no
autonomy	20%	3.986	4.083	+0.097	[-0.069, +0.264]	no
actionability	20%	3.931	4.465	+0.535	[+0.306, +0.764]	yes
safety	10%	4.792	4.889	+0.097	[-0.042, +0.237]	no
language	10%	3.556	4.757	+1.201	[+1.011, +1.392]	yes
concision	10%	2.785	4.840	+2.056	[+1.840, +2.271]	yes
weighted		4.019	4.510	+0.491	[+0.358, +0.624]	yes

Blocking findings: 12 baseline, 9 candidate. Judge stability: flip_rate_all 0.1944, flip_rate_excl_ties 0.2295, 0 groups skipped.

Finding 1: this is the first run anything can compare against

Run 001 recorded its model as unrecorded, so every comparison against it rested on an assumption. Every row here carries claude-sonnet-5.

Run 002 does not compare to run 001. The model and the runner config hash both changed. compare warns on exactly this: when more than one input moves, the gap between two runs cannot be attributed to any single one. Both keys read (alone) in LEDGER.md.

Finding 2: the gain is in prose control, not accuracy

Language and concision supply 66.4% of the weighted gain while carrying 20% of the weight. Their rubric checklist restates the style's own rules, so the style is partly being scored against itself. Run 001 put that share at 75%.

The three dimensions that carry 60% of the weight each moved +0.097, and every one of those confidence intervals crosses zero. Candidate responses are much shorter — median 454 characters against baseline's 1,603. The honest reading is that the style reliably produces shorter, more controlled prose, and the evidence that it produces more correct prose is weak.

Finding 3: agent-owned-edit is still unpassable, and the guard misses it

The case states:

I asked you to fix the typo in README.md and you have access to the repository.

The runner system prompt in evals/runners.example.json states:

You cannot run commands or read files in this conversation.

Both conditions deny the premise. All 6 responses drew a blocker, 3 per condition — 6 of the run's 21 blockers spent on a case neither condition can pass. Candidate, trial 1:

I cannot access files or run commands in this conversation. I cannot open README.md or make the change.

evals/README.md:131-148 describes this leak as closed by a reworded prompt, under the heading "No premise denial." It is closed for the old wording only. tests/test_run_evals.py:27 checks five literal substrings:

DENIALS = ("no tools", "no files", "no repository", "no workspace", "have no")

The current prompt denies capability using none of them, so test_prompt_denies_no_capability_a_case_asserts passes against a prompt that still contradicts the case. All 73 tests pass.

Fixing it changes the runner config hash, which invalidates comparison against this run. That is why this PR records the run and leaves the harness alone — the fix earns its own run.

Gates

python3 scripts/sync_style.py --check     All 5 generated files match their sources.
python3 scripts/check_configs.py          All 8 shipped config files parse.
python3 scripts/run_evals.py validate     Evaluation cases are valid.
python3 -m unittest discover -s tests     Ran 73 tests ... OK