fix(hw-gate): isolate each Fable session's evidence dir; keep the seat's raw output on a no-decision - #705
Conversation
…t's raw output on a no-decision Run 33866758629 (#702) ended the decide phase in 10 s with "omp decide: no JSON object in assistant text", and the uploaded fable-evidence/ contained #700's fable-summary.md and #686's route outputs: the workflow does `mkdir -p fable-evidence fable-home` in a reused runner workspace, so every session inherits the previous PR's files and can cite them as its own. `rm -rf` both before the mkdir. On the no-JSON path review.py discarded the assistant text it had already extracted, so the artifact carried nothing to diagnose the failure with. decision.json now records `fable_error` and `fable_raw` {assistant_text_tail, stderr_tail}; the step log gets the tail too. 103/103 in scripts/hw-gate/tests; workflow YAML parses.
hw-gate sol prelimsummary: The workflow now deletes and recreates the reused runner's relative run_hardware: true routes:
unavailable_routes: (none) claim_assessment: The author claims isolation between reused-runner Fable sessions and retention of raw diagnostics specifically when assistant text contains no JSON decision. The diff directly establishes the fixed-directory cleanup and the no-JSON diagnostic fields; proving end-to-end behavior requires a workflow or focused test that seeds both directories before startup and simulates non-JSON assistant output, then verifies stale files are absent and |
hw-gate evidence — 2 lane(s) — verdict passlane hiptrx (gfx1201)hw-gate evidence
fixturesno fixtures kernelnot run lane hipx (gfx1100)hw-gate evidence
fixturesno fixtures kernelnot run |
hw-gate sol verdict{
"claim_verdict": "not-exercised",
"confidence": 0.96,
"coverage": {
"gaps": [
"The hardware run executed no fixture or focused route proving that seeded fable-evidence/fable-home contents are removed before a session.",
"No supplied behavioral evidence simulates successful OMP output without a JSON object and verifies the bounded fable_error/fable_raw fields in decision.json.",
"No supplied evidence verifies the uploaded hw-gate-decision artifact contains the new diagnostics after the decide step exits nonzero."
],
"surfaces_evidenced": [
"build compatibility",
"static workflow cleanup logic",
"static no-JSON diagnostic serialization"
],
"surfaces_touched": [
"policy",
"ci",
"filesystem",
"diagnostics"
]
},
"decision": "needs-human",
"eyeball": [
".github/workflows/hw-gate.yml:405-410 \u2014 confirm the fixed relative-directory rm occurs at the intended lifecycle point and cannot erase evidence downloaded into the separate evidence/ directory.",
"scripts/hw-gate/review.py:1350-1355 and 1631-1635 \u2014 confirm retaining up to 4,000 characters of assistant output and stderr in an uploaded artifact is acceptable for CI diagnostics and does not expose material that should remain log-only."
],
"phase": "verdict",
"rationale": "Both hardware lanes built successfully and there are no fixture failures, but the empty bucket run does not exercise either behavioral claim. The diff directly shows cleanup of the two fixed session directories and serialization of bounded diagnostics on the returncode-zero/no-JSON branch, with no evident regression. Because .github/workflows/hw-gate.yml and scripts/hw-gate/review.py are gate-policy files, the mandatory human-review floor applies regardless of the clean build.",
"regressions": []
}Floor: hard=['policy_paths: .github/workflows/hw-gate.yml,scripts/hw-gate/review.py'] soft=["coverage_gaps: ['The hardware run executed no fixture or focused route proving that seeded fable-evidence/fable-home contents are removed before a session.', 'No supplied behavioral evidence simulates successful OMP output without a JSON object and verifies the bounded fable_error/fable_raw fields in decision.json.', 'No supplied evidence verifies the uploaded hw-gate-decision artifact contains the new diagnostics after the decide step exits nonzero.']", 'model needs-human'] model_decision=needs-human final=needs-human |
![hipfire-sol[bot]](https://avatars.githubusercontent.com/u/308720658?s=60&v=4){kind=small}
There was a problem hiding this comment.
hw-gate sol verdict needs-human: Both hardware lanes built successfully and there are no fixture failures, but the empty bucket run does not exercise either behavioral claim. The diff directly shows cleanup of the two fixed session directories and serialization of bounded diagnostics on the returncode-zero/no-JSON branch, with no evident regression. Because .github/workflows/hw-gate.yml and scripts/hw-gate/review.py are gate-policy files, the mandatory human-review floor applies regardless of the clean build.
|
announcement: Holding only because this touches gate-policy files, which a human merges by rule; on the evidence the change does what it says. Defect 1 is not hypothetical: this very session started with 238 inherited evidence entries and 80 inherited home entries from #686/#689/#700, and your two lines, run verbatim against a seeded workspace, leave both dirs empty and everything else alone. Defect 2 checks out too: driven with the tree's fake omp on the investigate path, head writes fable_error and a 4,000-char-bounded fable_raw before exiting 1 while base writes neither. Two things for the merge: the 103-test suite never reaches the changed branch (no test uses --investigate with garbage output), so please add one asserting on fable_raw; and the same reuse leak still exists for the workspace-root decision.json, which if: always() will upload from the previous PR if review.py dies before writing — adding it to the rm line is a one-word fix. investigation: | question | route | result | evidence | |---|---|---|---| | Is defect 1 real on this runner — does a Fable session actually inherit prior PRs' evidence and home? | ls/find/grep over $HW_GATE_EVIDENCE and $HIPFIRE_HOME at session start (this run executes the base workflow, which has no rm); recorded before writing anything else | $HW_GATE_EVIDENCE held 238 inherited top-level entries (282 files, mtimes 08:33–15:59 today) with provenance markers #686 x7, #689 x5, #700 x1 (the unprefixed fable-summary.md opens '# PR #700 — Fable decide-phase evidence summary'); $HIPFIRE_HOME held 80 inherited entries incl. 8 prior sessions' config.toml and a 68 KB registry.cache.json from 13:00. This run's hw-gate-decision artifact will upload all of it as #705's evidence; only pr705-* files are this session's. | pr705-stale-inheritance-inventory.txt | | Do the two added workflow lines (409–410) empty both dirs at the right lifecycle point without collateral, and do they target the dirs review.py is given? | sed -n '409,410p' pr/.github/workflows/hw-gate.yml > snippet.sh; seed fable-evidence/ (3 files) + fable-home/ (2 files, incl. config.toml) in a scratch workspace under $HIPFIRE_HOME; bash -euo pipefail snippet.sh; rerun on empty dirs; rerun with evidence/ decision.json lanes/ base/ pr/ present; python3 yaml.safe_load of the step | After the snippet both dirs exist with 0 files (exit 0); idempotent on empty dirs; evidence/verdict.json, decision.json, lanes/, base/, pr/ untouched. YAML parses; step has no working-directory; offsets show rm < mkdir < flock < review.py; --home "$PWD/fable-home" / --evidence-dir "$PWD/fable-evidence" share the rm's $PWD base. Base workflow has 0 occurrences of the rm, PR has 1. | pr705-rm-snippet-sim.txt | | On the investigate path, when omp exits 0 with no JSON object, does the PR's review.py record fable_error and a bounded fable_raw in decision.json and print the tail to stderr — and does the base not? | python3 $HW_GATE_EVIDENCE/pr705-nojson-sim.py — runs review.py --phase decide --seat fable --investigate with HW_GATE_OMP_BIN=tests/fake_omp.py (FAKE_OMP_GARBAGE=1 / FAKE_OMP_RESPONSES) and HW_GATE_GH_BIN=tests/fake_gh.py, four cases: pr-investigate-garbage, base-investigate-garbage, pr-investigate-longtail (9,990 chars + END-MARKER-705), pr-noninvestigate-garbage | 25 assertions, 0 failures, verified by me from the artifacts. PR/investigate: exit 1, decision.json written, decision_final=hold, decision=null, fable_error='omp decide: no JSON object in assistant text', fable_raw={assistant_text_tail:'this is not json at all — garbage !@#', stderr_tail:''}, stderr has 'fable assistant text (tail): ...'. Base/investigate (identical argv): exit 1, decision.json has neither key, stderr is the single 'fable omp failed' line. Longtail: assistant_text_tail is exactly 4,000 chars ending in END-MARKER-705; stderr tail line 2,003 chars. Non-investigate: fable_raw=null (capture is investigate-only; the workflow only uses --investigate, line 413). Fake gh log shows repo o/r PR 1 — nothing reached GitHub. | pr705-nojson-summary.txt and pr705-nojson-.{decision.json,stderr.txt,exit.txt,argv.txt} | | Is decision.json (with the new fields) on disk before the nonzero exit that the artifact upload survives? | exit codes + decision.json presence from the three exit-1 cases above; review.py:1637–1645 read | Write at review.py:1637–1641 precedes 'return 1 if fable_unavailable' at :1645; all three exit-1 cases have a complete decision.json; the upload step is if: always(). Sol gap #3 closed. | pr705-nojson-pr-investigate-garbage.exit.txt, .decision.json | | Does the author's '103 passed' exercise the change? | cd pr && TMPDIR=$HIPFIRE_HOME/pr705-tmp python3 -m pytest scripts/hw-gate/tests -q -p no:cacheprovider; git diff --stat $HW_GATE_BASE_SHA..HEAD -- scripts/hw-gate/tests; grep -c fable_raw tests/test_review.py | 103 passed in 4.18 s. The tests dir is byte-identical to base and mentions fable_raw 0 times; the only garbage-omp decide test (test_fable_unavailable_hold_exit1, line 720) runs without --investigate and never reaches review.py:1349–1357. The count is pre-existing coverage, not evidence for this diff. | pr705-pytest-pr-head.log | unproven:
rationale: Hard floor fired on policy_paths (.github/workflows/hw-gate.yml, scripts/hw-gate/review.py), so the decision is hold or block; nothing here is an evidence failure, so hold. Both claims are now evidenced on this host rather than asserted. Defect 1 is real and live: this session started with 238 inherited entries in $HW_GATE_EVIDENCE (#686/#689/#700 provenance) and 80 in $HIPFIRE_HOME including prior config.toml files (pr705-stale-inheritance-inventory.txt); the added lines, run verbatim, empty both dirs, are idempotent, touch no sibling dir, precede flock/review.py, and target the same $PWD-relative paths review.py receives at hw-gate.yml:425–426 (pr705-rm-snippet-sim.txt). Defect 2: with the tree's own fake omp on the investigate path, the PR's review.py writes fable_error and fable_raw {assistant_text_tail<=4000, stderr_tail<=4000} into decision.json before exiting 1 and prints a 2,000-char tail to stderr; base writes neither and leaves a one-line error (pr705-nojson-*, 25/25). All three of Sol's coverage gaps are closed by pr705-nojson-sim.py and pr705-rm-snippet-sim.txt. The author's 103-test run is real but the suite is unchanged from base and never reaches review.py:1349–1357, so it should not be read as coverage; a 10-line test with --investigate and FAKE_OMP_GARBAGE=1 asserting on fable_raw would be the durable form of what I ran. For the human: (1) same-class gap left open — the workspace root decision.json is stale (#686's at session start, mtime 16:01) and hw-gate-decision uploads it if: always(), so a review.py crash before review.py:1637 ships the previous PR's decision, which #706 would then read as this PR's decision_final; 'rm -rf fable-evidence fable-home decision.json' closes it. (2) Sol's eyeball on 4,000-char seat text + omp stderr in a downloadable artifact is a policy call; the child env is credential-stripped (review.py:368–371) so environment tokens cannot appear, omp's own config output is outside what I can inspect. (3) fable_raw is not captured on the returncode!=0 or timeout branches; scope choice. Index: $HW_GATE_EVIDENCE/pr705-fable-summary.md. hard floor: ['policy_paths: .github/workflows/hw-gate.yml,scripts/hw-gate/review.py'] soft floor: ["coverage_gaps: ['The hardware run executed no fixture or focused route proving that seeded fable-evidence/fable-home contents are removed before a session.', 'No supplied behavioral evidence simulates successful OMP output without a JSON object and verifies the bounded fable_error/fable_raw fields in decision.json.', 'No supplied evidence verifies the uploaded hw-gate-decision artifact contains the new diagnostics after the decide step exits nonzero.']", 'model needs-human'] |
There was a problem hiding this comment.
Holding only because this touches gate-policy files, which a human merges by rule; on the evidence the change does what it says. Defect 1 is not hypothetical: this very session started with 238 inherited evidence entries and 80 inherited home entries from #686/#689/#700, and your two lines, run verbatim against a seeded workspace, leave both dirs empty and everything else alone. Defect 2 checks out too: driven with the tree's fake omp on the investigate path, head writes fable_error and a 4,000-char-bounded fable_raw before exiting 1 while base writes neither. Two things for the merge: the 103-test suite never reaches the changed branch (no test uses --investigate with garbage output), so please add one asserting on fable_raw; and the same reuse leak still exists for the workspace-root decision.json, which if: always() will upload from the previous PR if review.py dies before writing — adding it to the rm line is a one-word fix. Hard floor fired on policy_paths (.github/workflows/hw-gate.yml, scripts/hw-gate/review.py), so the decision is hold or block; nothing here is an evidence failure, so hold. Both claims are now evidenced on this host rather than asserted. Defect 1 is real and live: this session started with 238 inherited entries in $HW_GATE_EVIDENCE (#686/#689/#700 provenance) and 80 in $HIPFIRE_HOME including prior config.toml files (pr705-stale-inheritance-inventory.txt); the added lines, run verbatim, empty both dirs, are idempotent, touch no sibling dir, precede flock/review.py, and target the same $PWD-relative paths review.py receives at hw-gate.yml:425–426 (pr705-rm-snippet-sim.txt). Defect 2: with the tree's own fake omp on the investigate path, the PR's review.py writes fable_error and fable_raw {assistant_text_tail<=4000, stderr_tail<=4000} into decision.json before exiting 1 and prints a 2,000-char tail to stderr; base writes neither and leaves a one-line error (pr705-nojson-*, 25/25). All three of Sol's coverage gaps are closed by pr705-nojson-sim.py and pr705-rm-snippet-sim.txt. The author's 103-test run is real but the suite is unchanged from base and never reaches review.py:1349–1357, so it should not be read as coverage; a 10-line test with --investigate and FAKE_OMP_GARBAGE=1 asserting on fable_raw would be the durable form of what I ran. For the human: (1) same-class gap left open — the workspace root decision.json is stale (#686's at session start, mtime 16:01) and hw-gate-decision uploads it if: always(), so a review.py crash before review.py:1637 ships the previous PR's decision, which #706 would then read as this PR's decision_final; 'rm -rf fable-evidence fable-home decision.json' closes it. (2) Sol's eyeball on 4,000-char seat text + omp stderr in a downloadable artifact is a policy call; the child env is credential-stripped (review.py:368–371) so environment tokens cannot appear, omp's own config output is outside what I can inspect. (3) fable_raw is not captured on the returncode!=0 or timeout branches; scope choice. Index: $HW_GATE_EVIDENCE/pr705-fable-summary.md.
hipfire-fable
Bot
added
the
needs-human
…judged #702 was blocked tonight by #682's verdict. Its own lanes were 8/8 pass (run 33929015500, head 87233ea, evidence `verdict: pass`), yet the run published a decision carrying `hw_run_result=failure`, `evidence verdict='fail'`, and an announcement about "six refused loads ... master says 'no model loaded' for four of them" -- which is #682's source-aware admission work, not a DFlash kernel PR. Cause: the runner workspace is reused and `upload-artifact` runs `if: always()`. #702's decide phase failed before writing its own decision.json, so the file left behind by the previous run on that runner -- #682's re-gate -- was uploaded as `hw-gate-decision` for #702, and the status job read it and blocked the PR. #705 fixed the same hazard for `fable-evidence/` and `fable-home/`; decision.json was missed, and it is worse, because that file is the gate's verdict rather than an input to it. Two changes, because cleaning is necessary but not sufficient: 1. The decide step removes a stale `decision.json` alongside the evidence dirs, so the common case cannot arise. 2. review.py records `base` and `head` in decision.json, and the status job refuses a decision whose `head` is not this run's head: "decision artifact is for <sha> but this run is <sha> -- stale decision.json from a reused workspace; re-run the gate". An artifact from another commit is a gate malfunction, not a verdict, so it fails as one instead of being obeyed. Artifacts predating this field warn rather than fail, so an in-flight run does not break on merge. The lane evidence already carried base/head for exactly this reason (hw-gate.json records both); the decision did not. Test: `test_decision_records_the_commit_it_judged` asserts both fields match the commit under review. 107/107 hw-gate tests pass.
1 participant
Summary
Two defects from the campaign PR's first gate pass (run 33866758629 on #702):
mkdir -p fable-evidence fable-homein a reused runner workspace. Thehw-gate-decisionartifact for perf(dflash, gfx1100): dedicated MQ4-V2 verify kernels, split-K residual, and launch fusion — merge-sort xt 209 → 285 tok/s #702 contained chore: retire tools/change_gate and the agentic-review route superseded by hw-gate #700'sfable-summary.mdand feat(dflash): registry-declared draft sidecars — pull fetches, auto uses, on requires #686's route outputs (explicit-draft-wins-qwen3.8-27b-mq4-xt-pr.*,attempt1-hipfire-home-leak/). The seat can read and cite another PR's evidence as its own. Fix:rm -rf fable-evidence fable-homebefore themkdir.omp decide: no JSON object in assistant text, andreview.pydiscarded the assistant text it had already extracted.decision.jsonnow recordsfable_errorandfable_raw {assistant_text_tail, stderr_tail}; the step log gets the tail too.Which surface(s) does this touch?
hw-gate.yml,scripts/hw-gate/review.py(hard floor: a human merges this)Evidence
python3 -m pytest scripts/hw-gate/tests -q→ 103 passed.No change to seat prompts, routes, or the hard floor.