Correct twelve documented facts that drifted from the files - #10
Merged
Conversation
Every gate passed while these were wrong. No check reads prose. The isolation prompt in evals/README.md was the expensive one. It still quoted "You cannot run commands or read files in this conversation," which 375b7b4 removed from evals/runners.example.json and which tests/test_run_evals.py lists in CONTAMINATED_WORDINGS. A reader who copied the documented prompt into evals/runners.json would repeat the run 002 contamination: 6 of 21 blockers on agent-owned-edit alone. The case catalog reached 24 in 04b16f7. Seven places still said 20: the README and all five translations, the two attribution counts in evals/README.md, and the position-balance claim. Both frozen key files measure 36 of 72, not 30 of 60. NOTICE.md carried three attribution errors. The shape layer holds 11 rules, not 10; rule 2 "Do the work you own" is new here, added in 04b16f7, so crediting it upstream overstates what the source contributed. The style no longer ships "not a skill" — six harnesses get it as one. Ten cases are new here, not six: evals/README.md already named the verbatim / language / decision / uncertainty set. The freeze example in evals/README.md used run 002's id with run 001's judge cost and a note the ledger contradicts: run 002 reworded nothing. It now shows an unclaimed 003. CI runs a fifth gate. CLAUDE.md and CONTRIBUTING.md both undercounted it, so a maintainer who ran the documented set could still fail on the ledger diff. CONTRIBUTING.md also named four generated files; sync_style.py writes five, and AGENTS.md comes from CLAUDE.md, not from the style.
A centered line under the badge row, above the language selector: "Connect on LinkedIn!" in English, translated in the five others so no reader hits an untranslated string.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every gate passed while all twelve were wrong. No check reads prose.
The expensive one
evals/README.mdstill quoted the isolation prompt as "You cannot run commands or read files in this conversation." 375b7b4 removed that sentence fromevals/runners.example.json, andtests/test_run_evals.pylists it inCONTAMINATED_WORDINGS. A reader who copied the documented prompt intoevals/runners.jsonwould repeat the run 002 contamination: 6 of 21 blockers onagent-owned-editalone, at about $11 and two hours per run. The section now quotes the shipped text and tells readers to copy from the config, not from the doc.Case count: 20 -> 24
04b16f7 added four
uncertaintycases. Seven places still said 20:README.mdand all five translationsevals/README.mdBoth frozen
blind-key.jsonlfiles measure 36 of 72, not 30 of 60.NOTICE.md
Three attribution errors:
language, twoverbatim, onedecision, fiveuncertainty.evals/README.mdalready named that set.14 of the caseswas correct and stayed correct.The rest
120 responsesisolation numbers predate the ledger and the 24-case catalog. They now say so, and the blocker counts name their runs.CLAUDE.mdandCONTRIBUTING.mdboth undercounted it, so a maintainer running the documented set could still fail on the ledger diff.CONTRIBUTING.mdnamed four generated files.sync_style.pywrites five, andAGENTS.mdcomes fromCLAUDE.md, not from the style.Also here
A "Connect on LinkedIn!" line in all six README headers, translated in the five non-English files.
Gates
No frozen run changed.
AGENTS.mdis regenerated, not hand-edited.