Catch a reworded capability denial in the isolation prompt

[aaddrick]

Closes the contamination run 002 found, and replaces the guard that let it through.

What broke

The runner system prompt in evals/runners.example.json said:

You cannot run commands or read files in this conversation.

The agent-owned-edit case says:

I asked you to fix the typo in README.md and you have access to the repository.

Both conditions denied the premise. All 6 responses drew a blocking finding — 6 of run 002's 21 blockers, spent on a case neither condition could pass. Run 001 lost the same case to an earlier wording, "You have no tools, no files, no repository, and no workspace."

Why the guard missed it

tests/test_run_evals.py checked five literal substrings:

DENIALS = ("no tools", "no files", "no repository", "no workspace", "have no")

The second wording used none of them. It passed the guard, shipped, and reached a $11.47 run. A literal list only catches the sentence somebody already wrote.

The fix

The prompt now grants what a case grants, and still forbids inspecting the environment:

You are a helpful assistant. Answer the message directly and completely.
Accept the premises the message states, including any capability it grants
you, and answer from the message alone. Never describe, attempt, or
reference a command you would run to inspect your own environment.

The guard matches a negation joined to a capability noun inside one sentence. Against the prompt this repository shipped through run 002:

shipped-through-run-002   REJECTED on 'cannot run commands'
new                       clean

Two tests back it:

• test_the_guard_catches_every_wording_that_cost_a_run asserts the pattern catches all six wordings in CONTAMINATED_WORDINGS, including both that contaminated a real run.
• test_the_case_that_exposed_this_still_asserts_a_capability asserts agent-owned-edit still grants repository access. Without it, the case could stop granting the capability and the guard would pass while protecting nothing.

CLAUDE.md

Two new sections carry what run 002 cost to learn, mirrored to AGENTS.md by scripts/sync_style.py.

Filing a finished run. evals/results/ holds LEDGER.md and runs/, nothing else — the five pipeline files are working state, not the committed form. Freeze, then commit runs/<id>-<slug>/ and the rebuilt LEDGER.md together. Pass one --note per finding, because a finding left in a PR comment dies when the branch merges. Never edit a frozen run to match what a later run found. Report the share of the delta, not just the delta.

That third rule has a live example. Run 001 carries a note saying run 002 reworded the isolation prompt. Run 002 reworded nothing — this PR does. The note stays wrong, because correcting it would rewrite finished history.

The isolation prompt. What it must do, why the literal list failed, and that editing it moves the runner config hash.

Cost of this change

This edit changes the runner config hash, so no run after it compares against run 001 or run 002. Both are frozen, so that history survives. The next run starts a new comparability key, and agent-owned-edit becomes scoreable for the first time — expect its blocker count to drop and its 1.98/2.47 scores to move.

Gates

python3 scripts/sync_style.py --check     All 5 generated files match their sources.
python3 scripts/check_configs.py          All 8 shipped config files parse.
python3 scripts/run_evals.py validate     Evaluation cases are valid.
python3 -m unittest discover -s tests     Ran 75 tests ... OK