ci: run the backlog status invariant ungated, and correct #320 to the measurement - #128
Open
wshallwshall wants to merge 1 commit into
Open
ci: run the backlog status invariant ungated, and correct #320 to the measurement#128wshallwshall wants to merge 1 commit into
wshallwshall wants to merge 1 commit into
Conversation
… measurement Two fixes from the same investigation, both correcting my own work. 1. THE GUARD GAP THAT RED main. Two guards police docs/BACKLOG.md. The number-space gate is DELIBERATELY ungated, with a comment saying gating it "would skip it on exactly the pull requests it exists to police". The STATUS invariant was reachable only through pytest, which IS gated on `code == 'true'` -- so on a BACKLOG-only PR, the exact shape it exists to check, it did not run at all. On 2026-08-01 that let #117 merge with a #320 banner using an emoji the invariant rejects. Docs-only, so the leg went green in seconds without compiling the suite; main was then red for every session until another session's PR tripped over it. A 0.6s markdown parse would have caught it. One hardened and one unhardened guard over the same file is worse than neither: the hardened one makes it reasonable to assume BACKLOG changes are covered. Adds an ungated step running scripts/docs/backlog_status_check.py. The step above advertises "git + stdlib python (no install), costs seconds" and that property is preserved exactly -- the script imports only argparse/re/sys/pathlib and has its own CLI. tests/test_backlog_status_check.py imports the SAME module, so there is one implementation, the unit tests keep covering the edge cases, and this only adds the always-on invocation. 2. #320's DIAGNOSIS WAS WRONG. It claimed windows-2025 "cannot service 60/s" and is "~10x slower". A 36-run sweep across all three hosted SKUs (run 30705885914, 3 repeats per cell) refutes both: stranded ubuntu win-2022 win-2025 60/s 0,0,0 0,0,0 0,0,0 150/s 0,0,0 0,0,0 0,0,0 300/s 0,0,0 0,0,0 0,0,0 600/s 0,0,0 4.6/8.9/19.2% 25.4/30.7/31.1% wall ubuntu win-2022 win-2025 2025 vs ubuntu 60/s 2.8s 4.7s 4.9s 1.8x 150/s 4.3s 10.1s 11.9s 2.8x 300/s 8.4s 22.0s 29.8s 3.5x What holds: a consistent ubuntu > win-2022 > win-2025 ordering, and ~3x more stranding than its sibling at saturation. It is genuinely the slowest leg. What is refuted: the gap is a LATENCY gap of 1.8x-3.5x, not a capacity cliff. windows-2025 ingests everything up to 300/s -- five times the rate the failing test offers -- and strands 0% at 60/s in three of three runs. So the 60/s failures are still unexplained: real, byte-identical twice, but not a property of the SKU at that rate or the sweep would show them. The untested variable is CONTENTION -- the probe runs the engine alone on an idle runner, while the CI failure happens with the full suite alongside it in a ~20-minute job. That also matches the developer-box outlier already retracted in this item. Consequence: this is a TOLERANCE problem, not a throughput one, which is what #115's widened stranding budget already absorbs -- that fix is better supported now than when it was made. And the product concern this item originally raised is not supported: a 1.8x-3.5x gap on a hosted VM image says nothing about Windows Server 2025 as a deployment target. Verified: backlog_status_check exits 0 (244 items); the new step parses and is ungated (`runner.os == 'Linux'` only, no `code` condition); 122 backlog / required-context / workflow guard tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wshallwshall
enabled auto-merge (squash)
August 1, 2026 16:13
wshallwshall
added a commit
that referenced
this pull request
Aug 2, 2026
…s that can never merge (#131) * ci: report pull requests that are green, armed, and can never merge Measured on this repo 2026-08-01: nine open pull requests with zero failing checks and zero pending checks, not one of which could merge. Six had auto-merge ARMED, which will never fire. #74 had been in that state since 2026-07-30 and was found only because somebody went hunting for "stuck CI" by hand. The mechanism is that `strict = true` plus no merge queue plus a ~20-minute suite makes merging a race: a PR is mergeable only between going green and the next thing landing on main. Losing that race is silent. Armed auto-merge does NOT update a BEHIND branch -- it waits on checks that already passed -- so the PR sits with no failing check, no notification, and no run in flight. No existing signal can see it, because every existing signal is a check OUTCOME and nothing has failed. nightly-notice.yml watches CI runs and there is no failing run to watch; the author's last signal was a full pass. A green dashboard and a wedged repository are indistinguishable unless something asks "can this still merge at all?". This does not fix the race -- only a merge queue does, filed separately as BACKLOG #340. It converts a SILENT failure into a LOUD one, which is the part that let #74 sit for days. Scheduled rather than per-PR: the stall arrives when a DIFFERENT pull request merges, so the affected PR has no run to hang a check on. Advisory by placement and must never become required -- it reports on OTHER pull requests, so a stall on #71 would block #128, wedging the repo with the tool meant to unwedge it. Verified against the live repo: 14 scanned, 8 stalled, 6 armed, exit 1. The count differs from the hand survey's 9 because #120 was re-synced in between, which the check correctly excluded. Tests carry a positive control (the exact stall shape MUST be detected) alongside negative controls for failing, pending, BLOCKED, DIRTY and closed PRs, and assert that an unclassifiable rollup node counts as unsettled rather than green. * ci: raise the Windows step cap, which had 1.06x margin while claiming 2x PR #119 was killed at 26:07 against ci.yml's 26:00 `step_timeout` with ZERO tests failing. What moved was the suite, not the code under test: #74 landed tests/test_worktree_prune_merged.py (1,506 lines) and windows-2025 went 19:35 -> 26:07 on the same branch. The comment beside the cap said the Windows legs were "unchanged because 26 min against the same suite is still ~2x headroom". Measured over the 11 PASSING windows-2025 runs on 2026-08-01: leg max passing step old cap old margin ubuntu-latest 12:27 19:00 1.53x windows-2022 18:39 26:00 1.39x windows-2025 24:35 26:00 1.06x windows-2025 had already PASSED at 24:35 -- 85 seconds of margin -- before #119 died. The "2x" figure matched no leg when it was written. The same file records this exact failure happening on the ubuntu leg on 2026-07-31 (775s green against a 780s cap) and concludes a watchdog that cannot separate "deadlocked" from "slow today" becomes a coin flip; ubuntu's budget was raised then and Windows was left alone on the false claim. Raised to step_timeout 36 (1.46x over the 24:35 max) and job_timeout 40, preserving the nesting invariant that the step must expire strictly before the job. Both Windows legs take the same number: windows-2022 is faster, so sizing on windows-2025 only leaves it more room. The replacement comment states the measured value and its date rather than a multiple -- a bare multiple is what let this rot undetected. Timing note recorded in the comment because it cost two sessions an error during triage: step_timeout gates the STEP, not the job. c53f752's JOB ran 28:41 and PASSED, against job cap 30 / step cap 26. Also files BACKLOG #340 (enable a merge queue -- 9 green PRs could not merge, 6 armed and never firing) and #344 (fixed wall-clock bounds as a class, with this cap as instance 1 and test_stage_dispatcher.py's hardcoded 8.0s poll budget against an injected ManualClock as instance 2). COORDINATION: ci.yml and docs/BACKLOG.md were each held by another live session, and the collision gate (scripts/hooks/collision_gate.ps1) refuses an Edit while a live session's BRANCH carries a diff to the file -- it cannot represent "coordinated, verified disjoint". Both counterparties gave explicit written consent before these edits were applied outside the Edit tool: zizmor-1280-adoption ("You land it. I'm standing down on ci.yml timeouts", its only hunk being a one-line pin comment ~160 lines away) and ha-construct-pickle-sandbox ("Go ahead with #340 now -- append after #338 exactly as you planned. I'll absorb the conflict"). No hooks were skipped; this commit ran the full pre-commit suite. * ci: record that #119's leg passed on a re-run at the same cap The ADR-0154 session re-ran #119's windows-2025 leg on the SAME commit against the SAME 26:00 cap. Attempt 1 was killed at the cap; attempt 2 concluded success. Same code, same config, same ceiling, two outcomes. This closes the one gap in the case for raising the cap: it rules out "that PR's tests are just slow". The leg was not failing, it was coin-flipping against the ceiling -- which is precisely the state the ubuntu note above already names, now demonstrated rather than argued. It also disposes of "re-run it and see" as a diagnosis. A green re-run at 26:00 does not show the suite fits; it shows that runner was fast enough that time. Recorded in the comment so the next person reaching for a retry knows what a green retry does and does not prove. Evidence contributed by the session holding #119, which is deliberately holding its branch update until this lands so it re-rolls under the raised cap rather than spending another coin flip at 26. Merged main (8f01cef, #120) in the same push: #131 had gone BEHIND, which is the stall this PR exists to report -- the fix for the cap has to survive the cap, and the fix for silent stalls can itself stall silently.
wshallwshall
added a commit
that referenced
this pull request
Aug 2, 2026
…ide (#144) Follow-on to #143, which merged before this measurement existed. Adds the one argument #340 was missing, and it is a better one than the cycle-count case. Measured 2026-08-02T13:41Z, re-derived here rather than relayed: open=14 armed=9 armed_and_inert=6 armed_and_CLEAN=0 #142 BEHIND #139 BEHIND #128 BEHIND #101 BEHIND #96 BEHIND #71 DIRTY Two-thirds of the armed PRs in this repo cannot land, and NOT ONE armed PR was CLEAN. #71 is armed and DIRTY, so it can never land at all. Why this belongs in the item: everything else in #340 is an efficiency argument, and an efficiency argument has a "then be patient" answer. This one does not. Every session here reads autoMergeRequest != null as "this will land" -- I said exactly that about my own PR an hour before measuring this -- when for six of nine it means "this waits until a human runs gh pr update-branch", with nothing reporting the difference. That is the ADR 0158 defect class (a green signal that means nothing) caught live rather than in retrospect. ADR 0158 is referenced by number, not linked: it is not on main yet. The measurement came out of the sandbox-codec session's queue claim, checked by the announce-hook session, and the connection to 0158's class is sandbox-codec's. Both routed it to me rather than writing it, since #340 is claimed here. Re-derived independently before writing; their figures and mine agree exactly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes from the same investigation, both correcting my own work.
1. The guard gap that red
mainTwo guards police
docs/BACKLOG.md. The number-space gate is deliberately ungated, with a comment saying gating it "would skip it on exactly the pull requests it exists to police". The status invariant was reachable only through pytest, which is gated oncode == 'true'— so on a BACKLOG-only PR, the exact shape it exists to check, it did not run at all.On 2026-08-01 that let #117 merge with a #320 banner using an emoji the invariant rejects. Docs-only, so the leg went green in seconds without compiling the suite.
mainwas then red for every session in the repo until another session's PR tripped over it. A 0.6s markdown parse would have caught it.One hardened guard and one unhardened guard over the same file is worse than neither — the hardened one makes it reasonable to assume BACKLOG changes are covered.
The fix turned out to be clean. The invariant already lives in a standalone stdlib script,
scripts/docs/backlog_status_check.py— onlyargparse/re/sys/pathlib, with its own CLI. So the neighbouring step's advertised property ("git + stdlib python (no install), costs seconds") is preserved exactly. Andtests/test_backlog_status_check.pyimports that same module, so there is one implementation, the unit tests keep covering its edge cases, and this only adds the always-on invocation.2. #320's diagnosis was wrong
It claimed windows-2025 "cannot service 60/s" and is "~10x slower". A 36-run sweep across all three hosted SKUs (run 30705885914, 3 repeats per cell) refutes both.
Stranded fraction:
Wall time — the cleaner signal, since all three still ingest everything up to 300/s:
What holds: a consistent
ubuntu > win-2022 > win-2025ordering, and ~3× more stranding than its sibling at saturation. It is genuinely the slowest leg.What is refuted: the gap is a latency gap of 1.8×–3.5×, not a capacity cliff. windows-2025 ingests everything up to 300/s — five times the rate the failing test offers — and strands 0% at 60/s in three of three runs.
So the 60/s failures remain unexplained. Real (byte-identical twice) but not a property of the SKU at that rate, or the sweep would show them. The untested variable is contention: the probe runs the engine alone on an idle runner, while the CI failure happens with the full suite alongside it in a ~20-minute job. That also matches the developer-box outlier already retracted in this item.
Consequence. This is a tolerance problem, not a throughput one — exactly what #115's widened stranding budget absorbs, so that fix is better supported now than when it was made. And the product concern this item originally raised is not supported: a 1.8×–3.5× gap on a hosted VM image says nothing about Windows Server 2025 as a deployment target.
Verification
scripts/docs/backlog_status_check.pyexits 0 — 244 items, each declaring exactly one status.runner.os == 'Linux'only, nocodecondition).main(2a6649fb) rather than committed against a stale base — another session had editedci.ymlmeanwhile; auto-merged cleanly and re-verified after.