fix(hw-gate): accept a decide verdict written as a markdown headline - #711
Conversation
Run 33905366422 (#691): Fable's decide phase returned a complete, correct investigation — as a markdown report headed `## hw-gate decide — PR #691: **merge-staging**` instead of a JSON object. review.py reported "no JSON object in assistant text", decision=None, decision_final=hold, and the run went red on a complete green verdict. _markdown_decision() recognizes the decide headline / bold decision word (`**merge-staging**`, `**decision:** hold`, `decision: block`) and synthesizes the decision dict; wired into both decide paths (omp_investigate's no-JSON branch and omp_review's retry loop). fable_raw is now also set on the non-investigate failure path. test_decide_markdown_headline_is_a_decision: headline verdict → merge-staging decision + staging merge; prose without a verdict word → hold with fable_error + raw tail in the artifact. 104/104.
hw-gate sol prelimsummary: Adds a decide-phase fallback that recognizes a Fable verdict expressed in the first 2,000 characters of a markdown report, synthesizes the decision object, preserves raw assistant text, and allows the existing floor and staging-merge logic to consume it; also preserves the assistant tail when the non-investigate path remains unparseable. run_hardware: true routes:
unavailable_routes: (none) claim_assessment: The author claims markdown-headline verdicts now produce decision_final=merge-staging and invoke the staging merge, while verdict-free prose remains hold with error and raw-tail evidence, with 104 tests passing. This is proved by running the complete scripts/hw-gate test suite and inspecting focused positive, negative, and adversarial parser cases; model-generation routes do not exercise the claim. questions_for_author:
|
hw-gate evidence — 2 lane(s) — verdict passlane hiptrx (gfx1201)hw-gate evidence
fixturesqwen3.6:27bsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 33.2 status pass
qwen3.6:27b battery turn 0qwen3.6:27b battery turn 1qwen3.6:27b battery turn 2qwen3.6:27b battery turn 3qwen3.6:27b battery turn 4chain — exit 0 seconds 21.5 status pass
qwen3.6:27b chain turn 0qwen3.6:27b chain turn 1qwen3.6:27b chain turn 2qwen3.6:27b chain turn 3qwen3.6:27b chain turn 4ornith-1.5:35b-a3b-mq4rsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 31.8 status pass
ornith-1.5:35b-a3b-mq4r battery turn 0ornith-1.5:35b-a3b-mq4r battery turn 1ornith-1.5:35b-a3b-mq4r battery turn 2ornith-1.5:35b-a3b-mq4r battery turn 3ornith-1.5:35b-a3b-mq4r battery turn 4chain — exit 0 seconds 21.4 status pass
ornith-1.5:35b-a3b-mq4r chain turn 0ornith-1.5:35b-a3b-mq4r chain turn 1ornith-1.5:35b-a3b-mq4r chain turn 2ornith-1.5:35b-a3b-mq4r chain turn 3ornith-1.5:35b-a3b-mq4r chain turn 4lfm2.5:1.2bsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 8.1 status pass
lfm2.5:1.2b battery turn 0lfm2.5:1.2b battery turn 1lfm2.5:1.2b battery turn 2lfm2.5:1.2b battery turn 3lfm2.5:1.2b battery turn 4chain — exit 0 seconds 14.6 status pass
lfm2.5:1.2b chain turn 0lfm2.5:1.2b chain turn 1lfm2.5:1.2b chain turn 2lfm2.5:1.2b chain turn 3lfm2.5:1.2b chain turn 4qwen3.8:27b-mq4-xtsource: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason: battery — exit 0 seconds 11.5 status pass
qwen3.8:27b-mq4-xt battery turn 0qwen3.8:27b-mq4-xt battery turn 1qwen3.8:27b-mq4-xt battery turn 2qwen3.8:27b-mq4-xt battery turn 3qwen3.8:27b-mq4-xt battery turn 4chain — exit 0 seconds 18.4 status pass
qwen3.8:27b-mq4-xt chain turn 0qwen3.8:27b-mq4-xt chain turn 1qwen3.8:27b-mq4-xt chain turn 2qwen3.8:27b-mq4-xt chain turn 3qwen3.8:27b-mq4-xt chain turn 4kernelstatus: pass report pass: True lane hipx (gfx1100)hw-gate evidence
fixturesno fixtures kernelnot run |
hw-gate sol verdict{
"claim_verdict": "not-exercised",
"confidence": 0.98,
"coverage": {
"gaps": [
"The hardware report exercises kernel, load, and serve behavior, not the changed markdown decision parser or staging-merge authorization path.",
"The reported hw-run result is failure.",
"No adversarial tests cover quoted, negated, or conflicting verdict phrases."
],
"surfaces_evidenced": [],
"surfaces_touched": [
"policy"
]
},
"decision": "block",
"eyeball": [
"Review the unanchored fallback patterns at scripts/hw-gate/review.py:447-450; quoted or explanatory prior verdicts can become authoritative.",
"The supplied model outputs are generally coherent, but they are unrelated to the policy-parser claim."
],
"phase": "verdict",
"rationale": "Block because the diff introduces a demonstrated merge-authorization ambiguity: an unanchored quoted or negated `decision: merge-staging` phrase is accepted as the current verdict. The supplied hardware evidence passes unrelated runtime routes but does not exercise this parser, and the overall hw-run result is failure. Narrow the fallback to an unambiguous verdict heading or standalone line, reject conflicting matches, and add positive plus quoted/negated/conflicting end-to-end tests.",
"regressions": [
{
"beta_behavior": "The fallback accepts a decision phrase anywhere within the first 2,000 characters. Text such as \u201cPrevious **decision:** merge-staging was wrong\u201d matches the second pattern and can authorize a staging merge despite explicitly rejecting that verdict.",
"evidence": "Diff at scripts/hw-gate/review.py:449 uses an unanchored search for **decision:** followed by merge-staging/hold/block; no negative, quotation, conflict, or unique-verdict checks are present.",
"file": "scripts/hw-gate/review.py",
"line": 449,
"master_behavior": "Non-JSON decide output is rejected and held for human review.",
"severity": "high"
}
]
}Floor: hard=['hw_run_result=failure', 'policy_paths: scripts/hw-gate/review.py,scripts/hw-gate/tests/test_review.py'] soft=["coverage_gaps: ['The hardware report exercises kernel, load, and serve behavior, not the changed markdown decision parser or staging-merge authorization path.', 'The reported hw-run result is failure.', 'No adversarial tests cover quoted, negated, or conflicting verdict phrases.']"] model_decision=block final=block |
There was a problem hiding this comment.
hw-gate sol verdict block: Block because the diff introduces a demonstrated merge-authorization ambiguity: an unanchored quoted or negated decision: merge-staging phrase is accepted as the current verdict. The supplied hardware evidence passes unrelated runtime routes but does not exercise this parser, and the overall hw-run result is failure. Narrow the fallback to an unambiguous verdict heading or standalone line, reject conflicting matches, and add positive plus quoted/negated/conflicting end-to-end tests.
|
announcement: Blocking, and the block is on the code rather than the floor: the headline pattern is sound and does merge the #691 shape end-to-end, but the two unanchored investigation: | question | route | result | evidence | |---|---|---|---| | Does the 104/104 claim hold, and does the new test pass on the PR head (base for comparison)? | cd pr && python3 -m pytest scripts/hw-gate/tests -q -p no:cacheprovider ; same in base ; pr: -k test_decide_markdown_headline_is_a_decision -vv | PR: 104 passed in 4.56s. Base: 103 passed. New test: 1 passed. The claim is true, and the suite has no negative case for a verdict word that is not the report's verdict. | pytest-pr-markdown-test.txt | | What does _markdown_decision (review.py:438-464) accept when the text is a realistic Fable report rather than the one fixture in the test? | python3 $HW_GATE_EVIDENCE/markdown_decision_probe.py — 17 named cases run through the PR's _markdown_decision, each re-matched against the three patterns to record which fired | Headline cases (pattern 0, anchored | Does a markdown verdict merge end-to-end through _run_decide, including the #691 shape (Sol needs-human), and what does base do with the same text? | python3 $HW_GATE_EVIDENCE/e2e_markdown_probe.py — the suite's own _run_decide harness (fake gh/omp) on the PR checkout for 7 scenarios and on base for 4 | PR: headline + Sol greenlight -> merge-staging, merged. Headline + Sol needs-human (the #691 case) -> merge-staging, merged, override={of: needs-human, why: } — the fix does cover the run it was written for. Headline + policy_paths floor -> hold, no merge (floor precedence intact). Prose with no verdict word -> hold, fable_error set, fable_raw.assistant_text_tail present. Quoted | Can a report whose verdict is block be merged to staging by the fallback? (Sol's regression, review.py:449-453) | python3 $HW_GATE_EVIDENCE/e2e_wrong_direction_merge.py — same harness, Sol greenlight, fable text = "The previous run's decision: merge-staging was wrong; the regression is real. ... turn 3 emits a single-token attractor ... My decision: block." | PR: decision_final=merge-staging, markdown_fallback=true, one POST to /merges, merged.merge_sha set, an --approve review posted. Base: decision_final=hold, decision=null, fable_error carries the text. A report that says block and describes an attractor merges to staging on this diff. | wrong_direction_block_report_merges.decision.json | | Why did the floor fire hw_run_result=failure when the merged evidence says pass on both lanes, and is the hardware evidence even this PR's? | stat/jq over the reused hiptrx workspace (hw-gate.json, lanes/, evidence/, prelim/*), git log for the recorded SHAs; then base/scripts/hw-gate/run.py --repo pr --buckets "" [--routes evidence/routes.json] --skip-build --device 3 with this PR's select (empty buckets, routes []) | The hiptrx lane in evidence/hw-gate.json is head 31dd5e3 / base e98b461 — a different PR's run (mtime 20:07:33, lanes/ copy 20:10:29); only the hipx lane (656 bytes, buckets [], no fixtures, head b106a4a) belongs to this PR (downloaded 20:12:05). merge_evidence.py takes base/head verbatim from the first lane (merge_evidence.py:89-95) and hard_floor never compares evidence.head to --head (review.py:94-151); sol-verdict and fable-decide download into lanes/ with continue-on-error and no rm -rf (hw-gate.yml:286-297, 374-385). So every battery, chain, and Redline number in the evidence comment on #711 is another commit's. run.py with this PR's actual selection exits 0 and writes verdict=pass with head b106a4a (binaries md5 17ba7dfa/cac7ede8, matching the hipx lane), so the hiptrx job failure was not the runner script on this input; the leg simply produced no artifact for this head. | runpy-empty-buckets-{routes,noroutes}.{json,md,log} | unproven:
rationale: The hard floor fired twice and neither reason is mine to lift: policy_paths (scripts/hw-gate/review.py, tests/test_review.py) is a hold by rule, and hw_run_result=failure is a block by rule. On the merits I also block, for a regression I demonstrated rather than inferred. hard floor: ['hw_run_result=failure', 'policy_paths: scripts/hw-gate/review.py,scripts/hw-gate/tests/test_review.py'] soft floor: ["coverage_gaps: ['The hardware report exercises kernel, load, and serve behavior, not the changed markdown decision parser or staging-merge authorization path.', 'The reported hw-run result is failure.', 'No adversarial tests cover quoted, negated, or conflicting verdict phrases.']"] |
There was a problem hiding this comment.
Blocking, and the block is on the code rather than the floor: the headline pattern is sound and does merge the #691 shape end-to-end, but the two unanchored decision: patterns take the first verdict word in the report, and a report that says block while quoting a prior merge-staging merges to staging on this diff — I ran that exact text through the suite's own _run_decide harness ($HW_GATE_EVIDENCE/e2e_wrong_direction_merge.json: PR merges with an approving review, base holds). A report with no verdict that mentions the daemon's admission decision posts block, and a report quoting Sol's block before its own merge-staging posts block. Keep the headline pattern, drop or line-anchor the other two with a single-distinct-verdict rule, prefer a toolless re-ask for the JSON over regex (the investigate path has no retry today), stop duplicating text[:6000] into both comment fields, and add the quoted/negated/conflicting/no-verdict cases as tests asserting hold. One thing for you rather than the PR: the hardware evidence comment on this PR is another commit's (hiptrx lane head 31dd5e3) — lanes/ in the reused runner workspace is not cleared and merge_evidence.py never checks a lane's head against the run's; details in $HW_GATE_EVIDENCE/stale-lane-evidence.txt. The hard floor fired twice and neither reason is mine to lift: policy_paths (scripts/hw-gate/review.py, tests/test_review.py) is a hold by rule, and hw_run_result=failure is a block by rule. On the merits I also block, for a regression I demonstrated rather than inferred. _markdown_decision (review.py:438-464) tries three patterns; the first, anchored to the ## hw-gate decide headline, is correct in every case I threw at it, including a headline after a quoted Sol decision, and it merges the #691 shape end-to-end over a Sol needs-human with the report as override.why ($HW_GATE_EVIDENCE/e2e_markdown_probe.json S2). The second and third (review.py:452-453) search the first 2000 characters for any decision: <word> and return the first hit, and the first hit in a real report is routinely a quotation: $HW_GATE_EVIDENCE/e2e_wrong_direction_merge.json runs a report that says block and describes an attractor through the PR's own harness and gets decision_final=merge-staging, one /merges call and an approving review; base holds the same text with fable_error. The other direction is also live: a report quoting Sol's block before its own merge-staging posts block (e2e S3), and a sentence about the daemon's admission decision in a report with no verdict posts block (probe case prose_daemon_decision). Test evidence is 104/104 on the PR and 103/103 on base ($HW_GATE_EVIDENCE/pytest-*.txt), and none of those tests contains a verdict word that is not the verdict. What changes this to a merge: keep pattern 0; delete patterns 1-2 or replace them with a line-anchored standalone verdict (^\s*\**decision:?\**\s*[:=]?\s*?(merge-staging|hold|block)?\s*$, MULTILINE) that returns None when more than one distinct verdict word matches; in omp_review try the existing 'Return only the JSON object' retry before falling back (the investigate path has no retry at all — a second, toolless omp call that hands back the report and asks for the JSON is safer than any regex); make announcement the headline plus first paragraph rather than a second copy of text[:6000]; and add the probe's quoted/negated/conflicting/no-verdict cases as tests asserting hold. Separately and not this PR's fault: the hardware evidence posted on #711 is not this PR's. evidence/hw-gate.json's hiptrx lane is head 31dd5e3 / base e98b461 (a different PR, written 20:07:33 on this reused hiptrx workspace); only the hipx lane (buckets [], no fixtures, head b106a4a) is real. merge_evidence.py copies base/head from the first lane verbatim (merge_evidence.py:89-95), hard_floor never checks evidence.head against --head (review.py:94-151), and both sol-verdict and fable-decide download into lanes/ without clearing it (hw-gate.yml:286-297, 374-385) — the same class #705 fixed for fable-evidence. This time hw_run_result=failure caught it; a stale lane under a successful or cancelled leg would pass the floor. Forensics in $HW_GATE_EVIDENCE/stale-lane-evidence.txt. run.py with this PR's empty selection exits 0 and writes head b106a4a ($HW_GATE_EVIDENCE/runpy-empty-buckets.json), so re-running hw-run clears the failure reason; the policy hold and the parser fix remain.
…e-gate a merged PR Two defects observed on 2026-09-04 while driving the ladder, both of which cost live rungs their hardware lane: 1. The decide phase takes `flock --exclusive` on all five GPUs around the whole `review.py --phase decide` process, for its full 45-minute budget. warpfront#711's seat had no provider credits and could never produce a verdict, yet it held that lock for ~20 minutes while warpfront#687 and warpfront#688 sat queued with 'hardware (gfx1201)' unable to start. A seat that cannot answer must not own the hardware. Fixed with a preflight: one 90 s toolless probe of the decide model before the lock is taken. If it replies, the locked phase runs exactly as before. If it does not, the locked step is skipped entirely and a new unlocked step records the hold via `--decider-unavailable REASON`, which short-circuits the model call in review.py and lets the floors, comment, and labels run as usual. The GPUs are never claimed. 2. `pull_request_target` fires on `labeled` even for a merged PR, so label churn re-ran the entire gate on warpfront#711 five minutes after it merged (run 33915818350) — taking the runner and the exclusive lock from live rungs. `select` now refuses any event whose PR is already merged; `workflow_dispatch` is unaffected, so a manual re-gate still works. Test: `test_decide_unavailable_seat_holds_without_calling_the_model` asserts hold, the reason recorded, no merge, and no model invocation at all (the fake never opens its log). 105/105 hw-gate tests pass.
Summary
Run 33905366422 (#691): Fable's decide phase returned a complete, correct investigation — as a markdown report headed
## hw-gate decide — PR #691: **merge-staging**instead of a JSON object.review.pyreported "no JSON object in assistant text",decision=None,decision_final=hold— red run on a green verdict, and the staging merge never happened.(This class of failure is why the ladder loses a run: the seat did the work and the gate lost it at the serializer.)
Change
_markdown_decision()recognizes the decide headline / bold decision word (**merge-staging**,**decision:** hold,decision: block) within the report head and synthesizes the decision dict; the full text is kept as announcement/rationale and infable_raw.markdown_fallback.omp_investigate's no-JSON branch andomp_review's retry loop (decide phase only).fable_rawis now also set on the non-investigate failure path (the omp_review error message carries the assistant tail).Evidence
test_decide_markdown_headline_is_a_decision: a headline verdict yieldsdecision_final=merge-stagingand the staging merge fires; prose without a verdict word yieldsholdwithfable_error+ raw tail in the artifact. Full suite: 104/104.Which surface(s) does this touch?
scripts/hw-gate/review.py(hard floor: a human merges this)