Skip to content

Correct twelve documented facts that drifted from the files - #10

Merged
aaddrick merged 2 commits into
mainfrom
fix-doc-drift
Aug 3, 2026
Merged

Correct twelve documented facts that drifted from the files#10
aaddrick merged 2 commits into
mainfrom
fix-doc-drift

Conversation

@aaddrick

@aaddrick aaddrick commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Every gate passed while all twelve were wrong. No check reads prose.

The expensive one

evals/README.md still quoted the isolation prompt as "You cannot run commands or read files in this conversation." 375b7b4 removed that sentence from evals/runners.example.json, and tests/test_run_evals.py lists it in CONTAMINATED_WORDINGS. A reader who copied the documented prompt into evals/runners.json would repeat the run 002 contamination: 6 of 21 blockers on agent-owned-edit alone, at about $11 and two hours per run. The section now quotes the shipped text and tells readers to copy from the config, not from the doc.

Case count: 20 -> 24

04b16f7 added four uncertainty cases. Seven places still said 20:

  • README.md and all five translations
  • both attribution counts in evals/README.md
  • the position-balance claim, twice

Both frozen blind-key.jsonl files measure 36 of 72, not 30 of 60.

NOTICE.md

Three attribution errors:

  1. The shape layer holds 11 rules, not 10. Rule 2 "Do the work you own" was added in 04b16f7, so crediting it upstream overstates what the source contributed. It is now listed as new here.
  2. "ships as a Claude Code output style, not a skill" is false. Six harnesses get it as a skill.
  3. Six new cases is now ten: two language, two verbatim, one decision, five uncertainty. evals/README.md already named that set.

14 of the cases was correct and stayed correct.

The rest

  • The freeze example used run 002's id with run 001's judge cost and a note the ledger contradicts. It now shows an unclaimed 003.
  • The two 120 responses isolation numbers predate the ledger and the 24-case catalog. They now say so, and the blocker counts name their runs.
  • CI runs a fifth gate. CLAUDE.md and CONTRIBUTING.md both undercounted it, so a maintainer running the documented set could still fail on the ledger diff.
  • CONTRIBUTING.md named four generated files. sync_style.py writes five, and AGENTS.md comes from CLAUDE.md, not from the style.

Also here

A "Connect on LinkedIn!" line in all six README headers, translated in the five non-English files.

Gates

sync_style.py --check     All 5 generated files match their sources.
check_configs.py          All 8 shipped config files parse.
run_evals.py validate     Evaluation cases are valid.
unittest discover         Ran 75 tests ... OK
ledger.py index           no diff

No frozen run changed. AGENTS.md is regenerated, not hand-edited.

Every gate passed while these were wrong. No check reads prose.

The isolation prompt in evals/README.md was the expensive one. It still
quoted "You cannot run commands or read files in this conversation," which
375b7b4 removed from evals/runners.example.json and which
tests/test_run_evals.py lists in CONTAMINATED_WORDINGS. A reader who copied
the documented prompt into evals/runners.json would repeat the run 002
contamination: 6 of 21 blockers on agent-owned-edit alone.

The case catalog reached 24 in 04b16f7. Seven places still said 20: the
README and all five translations, the two attribution counts in
evals/README.md, and the position-balance claim. Both frozen key files
measure 36 of 72, not 30 of 60.

NOTICE.md carried three attribution errors. The shape layer holds 11 rules,
not 10; rule 2 "Do the work you own" is new here, added in 04b16f7, so
crediting it upstream overstates what the source contributed. The style no
longer ships "not a skill" — six harnesses get it as one. Ten cases are new
here, not six: evals/README.md already named the verbatim / language /
decision / uncertainty set.

The freeze example in evals/README.md used run 002's id with run 001's judge
cost and a note the ledger contradicts: run 002 reworded nothing. It now
shows an unclaimed 003.

CI runs a fifth gate. CLAUDE.md and CONTRIBUTING.md both undercounted it,
so a maintainer who ran the documented set could still fail on the ledger
diff. CONTRIBUTING.md also named four generated files; sync_style.py writes
five, and AGENTS.md comes from CLAUDE.md, not from the style.
A centered line under the badge row, above the language selector:
"Connect on LinkedIn!" in English, translated in the five others so no
reader hits an untranslated string.
@aaddrick
aaddrick merged commit e52a2d1 into main Aug 3, 2026
3 checks passed
@aaddrick
aaddrick deleted the fix-doc-drift branch August 3, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant