Problem
dev/eval/cumulative/runbook_vs_skill/phase2/fixtures/history-rewrite/steps.json step 3's detection regex is:
git\s+(filter-branch|filter-repo)\b
This only matches the git-subcommand form git filter-repo ... (space-separated). It does not match the equally-valid hyphenated standalone-script invocation git-filter-repo ..., which is how git-filter-repo is commonly installed/invoked (e.g. via pip/brew, directly on PATH as git-filter-repo).
Steps 4/5/6 are gated "after": 3, so when an agent uses the hyphenated form, steps 3-6 are ALL scored false in followed_steps/followed_all, even when the agent genuinely ran the rewrite, fetched, force-pushed, and verified -- confirmed via done_when_checks.sh passing all 6 real checks and via reading the full bash command transcript.
Evidence
Found during runbook-shim-follow-frame task 4.3's 3rd rerun (2026-09-16), results/4.3_stage1_rerun3_sonnet5_shimonly_history-rewrite.jsonl. Trials history-rewrite-R-0 and history-rewrite-R-1 both ran:
git-filter-repo --path secrets.env --invert-paths --refs main --force
-- hyphenated form, not matched by the regex -- followed by git fetch origin and git push origin main --force-with-lease, both genuinely run but scored false because they're gated behind the unmatched step 3. history-rewrite-R-2 happened to use git filter-repo and scored followed_all=True. All 3 trials passed the real done_when_checks.sh end-state check (which passed all 6 checks silently and printed the final PASS line), confirming this is a scorer/regex gap, not a genuine behavior difference between trials.
Fix
Broaden step 3's pattern to match both invocation forms, e.g.:
git[\s-]+(filter-branch|filter-repo)\b
(only filter-repo ships a standalone hyphenated script; git-filter-branch isn't a real binary, so no new false-positive class).
Scope note
Out of scope for runbook-shim-follow-frame task 4.3 (no shim/runbook amendments authorized this round) -- filed for later harness maintenance. Does not change the 4.3 D8 verdict, which is based on done_when_checks.sh's real end-state result, not followed_all.
Problem
dev/eval/cumulative/runbook_vs_skill/phase2/fixtures/history-rewrite/steps.jsonstep 3's detection regex is:This only matches the git-subcommand form
git filter-repo ...(space-separated). It does not match the equally-valid hyphenated standalone-script invocationgit-filter-repo ..., which is howgit-filter-repois commonly installed/invoked (e.g. via pip/brew, directly on PATH asgit-filter-repo).Steps 4/5/6 are gated
"after": 3, so when an agent uses the hyphenated form, steps 3-6 are ALL scoredfalseinfollowed_steps/followed_all, even when the agent genuinely ran the rewrite, fetched, force-pushed, and verified -- confirmed viadone_when_checks.shpassing all 6 real checks and via reading the full bash command transcript.Evidence
Found during runbook-shim-follow-frame task 4.3's 3rd rerun (2026-09-16),
results/4.3_stage1_rerun3_sonnet5_shimonly_history-rewrite.jsonl. Trialshistory-rewrite-R-0andhistory-rewrite-R-1both ran:-- hyphenated form, not matched by the regex -- followed by
git fetch originandgit push origin main --force-with-lease, both genuinely run but scoredfalsebecause they're gated behind the unmatched step 3.history-rewrite-R-2happened to usegit filter-repoand scoredfollowed_all=True. All 3 trials passed the realdone_when_checks.shend-state check (which passed all 6 checks silently and printed the final PASS line), confirming this is a scorer/regex gap, not a genuine behavior difference between trials.Fix
Broaden step 3's pattern to match both invocation forms, e.g.:
(only
filter-repoships a standalone hyphenated script;git-filter-branchisn't a real binary, so no new false-positive class).Scope note
Out of scope for runbook-shim-follow-frame task 4.3 (no shim/runbook amendments authorized this round) -- filed for later harness maintenance. Does not change the 4.3 D8 verdict, which is based on
done_when_checks.sh's real end-state result, notfollowed_all.