bisect-before-fix rerun: question_stop scorer is order-insensitive; runbook 846 step 6 requires filing an issue the fixture has no tracker for

[toejough]

Found during

Rerun of runbook-shim-follow-frame task 4.4 (4.4_stage2_rerun_sonnet5_shimonly_bisect-before-fix.jsonl), after runbook 846's step 6 got a narrow exception clause (apply a pre-existing fix as part of the current change when the task explicitly instructs making the gate pass AND the fix is small/clearly-scoped) to resolve the mismatch the first 4.4 attempt found (all 3 R trials stopped instead of fixing bar.py).

What's wrong (two distinct findings)

1. detect_question_stop (probe_phase2.py) is order-insensitive — it flags a trial as a "question stop" even when the ambiguity text comes AFTER the trial has already completed and verified the task, as long as no mutating tool call follows it. A trial that finishes the full task (bisect, attribute, apply the narrow-exception fix, restore HEAD, verify gate.sh GREEN) and then, in its closing report, asks a genuinely separate follow-up question (e.g. "how should I track this pre-existing issue — there's no issue tracker in this repo, and should I commit the fix?") gets question_stop=True, identically to a trial that never completed the task at all. Confirmed directly: this rerun's R-0 and R-2 both hit question_stop=True, but both fully passed done_when_checks.sh (end_state=True, "PASS: Bisect-before-fix task end-state verified") and followed_all=True (all 7 steps, including fixing bar.py) before the trailing question. Only R-1 is a genuine pre-completion stop (asked permission before applying the fix, bar.py left unfixed, end_state=False). Per the harness's current scoreable-population rule (scoreable = [r for r in records if r.get("valid") and not r.get("question_stop")]), this run has 0/3 scoreable R trials for followed_all/end_state, even though 2/3 are unambiguous full successes — the same class of scorer-vs-reality gap as #754, but for question_stop rather than followed_steps.

2. Runbook 846's step 6 still requires filing a separate follow-up issue for the pre-existing regression, even under the new narrow-exception (fix-applied) branch — but the bisect-before-fix fixture repo provides no issue tracker, ticket queue, or any local issue-filing convention. All 3 R trials this run correctly reasoned that the narrow exception applies (task explicitly says "fix what's needed so the gate passes"; the fix is a stray debug print, the runbook's own paradigm example) and either applied the fix and then asked how to track the follow-up (R-0, R-2), or asked before applying it (R-1). This is a genuine fixture-completeness gap distinct from the wording mismatch task 4.4 was rerun to fix — the runbook's completion bar (file a follow-up issue) has no way to be satisfied in this specific fixture, so a shim-following agent has a legitimate reason to stop and ask, every time.

Not fixed here

Both are out of scope for the narrow step-6 wording fix this rerun validated (which worked correctly — the "should I apply the fix" question is resolved cleanly in all 3 trials). Filing for visibility/design call:

• (1) needs a scorer fix: order-aware detect_question_stop (only flag True when the LAST ambiguity precedes the trial's final completed state, not just "no mutation after it") — or scope it to ignore post-end_state-verification text.
• (2) needs a design call: either give the fixture a minimal local issue-filing convention (e.g. an ISSUES.md the runbook/task prompt names as the target) so the follow-up-issue requirement is satisfiable, or soften runbook 846's step 6 to make the follow-up-issue filing conditional on a tracker actually existing.

Evidence: kept trial dirs and transcripts at the paths in results/4.4_stage2_rerun_sonnet5_shimonly_bisect-before-fix.jsonl's trial_dir/transcript_path fields (this session's run, not yet cleaned up).