Skip to content

steps.json history-rewrite step-3 regex misses hyphenated git-filter-repo invocation #754

Description

@toejough

Problem

dev/eval/cumulative/runbook_vs_skill/phase2/fixtures/history-rewrite/steps.json step 3's detection regex is:

git\s+(filter-branch|filter-repo)\b

This only matches the git-subcommand form git filter-repo ... (space-separated). It does not match the equally-valid hyphenated standalone-script invocation git-filter-repo ..., which is how git-filter-repo is commonly installed/invoked (e.g. via pip/brew, directly on PATH as git-filter-repo).

Steps 4/5/6 are gated "after": 3, so when an agent uses the hyphenated form, steps 3-6 are ALL scored false in followed_steps/followed_all, even when the agent genuinely ran the rewrite, fetched, force-pushed, and verified -- confirmed via done_when_checks.sh passing all 6 real checks and via reading the full bash command transcript.

Evidence

Found during runbook-shim-follow-frame task 4.3's 3rd rerun (2026-09-16), results/4.3_stage1_rerun3_sonnet5_shimonly_history-rewrite.jsonl. Trials history-rewrite-R-0 and history-rewrite-R-1 both ran:

git-filter-repo --path secrets.env --invert-paths --refs main --force

-- hyphenated form, not matched by the regex -- followed by git fetch origin and git push origin main --force-with-lease, both genuinely run but scored false because they're gated behind the unmatched step 3. history-rewrite-R-2 happened to use git filter-repo and scored followed_all=True. All 3 trials passed the real done_when_checks.sh end-state check (which passed all 6 checks silently and printed the final PASS line), confirming this is a scorer/regex gap, not a genuine behavior difference between trials.

Fix

Broaden step 3's pattern to match both invocation forms, e.g.:

git[\s-]+(filter-branch|filter-repo)\b

(only filter-repo ships a standalone hyphenated script; git-filter-branch isn't a real binary, so no new false-positive class).

Scope note

Out of scope for runbook-shim-follow-frame task 4.3 (no shim/runbook amendments authorized this round) -- filed for later harness maintenance. Does not change the 4.3 D8 verdict, which is based on done_when_checks.sh's real end-state result, not followed_all.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions