ci(ubuntu): give the pytest watchdog room to tell a hang from a slow run - #104
Merged
Conversation
wshallwshall
enabled auto-merge (squash)
July 31, 2026 15:26
wshallwshall
force-pushed
the
ci-ubuntu-timeout-headroom
branch
from
July 31, 2026 15:56
183bc81 to
1737b4d
Compare
The ubuntu leg's step timeout was 13 minutes. On 2026-07-31 the suite finished GREEN in 775s -- 9598 passed, 855 skipped -- against a 780s cap, and the step was killed anyway. The run is reported as a failure whose last line of output is a clean pass summary. A watchdog five seconds from firing on a healthy run is not distinguishing a deadlock from a slow day; it is a coin flip that reddens PRs at random and trains people to re-run without reading. That is worse than no watchdog, because a real hang now looks like the usual flake. Raises the ubuntu budget to job 22 / step 19, preserving the property the surrounding comment is built on: the STEP must expire before the JOB, so a missed hang still fails with step attribution rather than as a silent job-level kill. Verified on all three legs -- ubuntu 19<22, windows-2022 and windows-2025 unchanged at 26<30. Windows is deliberately untouched. It runs the same suite against a 26-min step cap, which is still roughly double the observed runtime; it has the headroom ubuntu lost. Not a fix for anything slow. The suite grew into a cap that was set when it was shorter, and the comment now says to re-check the margin as it grows rather than leaving the next person to rediscover this from a green-but-failed run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
wshallwshall
force-pushed
the
ci-ubuntu-timeout-headroom
branch
from
July 31, 2026 20:35
1737b4d to
fcd2c5f
Compare
wshallwshall
added a commit
that referenced
this pull request
Aug 1, 2026
…ution's centre (#115) test_load_runner::test_run_load_end_to_end_no_loss red the required windows-2025 leg on `main` at 9b03057. Nothing was lost: sent=90 acked=44 timeouts=46, engine_read=52, engine_written=104, sink_received=104, backlog=0, drain 4.7s of a 30s bound, leg done in 20:05 inside every cap. It failed the reconcile. THE BUDGET WAS THE BUG. `_reconcile` excuses unconfirmed sends up to `max(unconfirmed_budget, sent // 2)`. That fraction was sized when the worst teardown stranding on record was 14/90 (~16%) -- report.py documented it as "half is ~3x the worst seen". windows-2025 has since produced 46/90 (~51%) on a lossless run, so the headroom was 0.98x and the run failed by ONE message over 45. A threshold on the centre of the healthy distribution is a coin flip, not a detector -- same defect as the ubuntu step cap in #104 (775s against a 780s bound) and test_coord_lock's arrival-spread assumption. THE TWO DETECTORS DISAGREED: tests/test_load_runner.py already recorded the 46/90 observation and tuned ITS check to `acked >= sent // 4` (tolerating 75% stranding) while this budget failed at half + 1, so the tuning never reached the path that fires. Raises the fraction to three quarters in all three synchronised copies (report, connscale, estate) plus the multishard doc-comment: ~1.5x the worst healthy value on record, while a dead ACK path strands ~100% and still blows it, and the two detectors now encode the same tolerance. NOT changed: `excused = 0 if over_budget else unconfirmed`. Clamping gives a better message (the current one claims "lost 38 on intake" for a lossless run) but test_harness_reconcile pins all-or-nothing deliberately, and the cliff is only reachable once the run has already failed. Noted in-code instead of overridden as a side-effect. The anti-vacuity floor is untouched and unconditional (`read >= sent // 2`); on the failing run it PASSED (52 >= 45) -- the run met the guarantee the budget protects and failed anyway. Tests: cap pin renamed and moved to the 67/68 boundary, plus a regression pin carrying the exact 9b03057 counters (90/46/52/104) that must reconcile clean. 1,371 harness/load tests green; ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The symptom
PR #101's
test (ubuntu-latest, py3.14)failed. The last lines of that job:Every test passed. The suite finished in 775s against a 780s cap — five seconds of margin — and the step was killed anyway.
mainis green, so this isn't inherited; it's the cap.Why this matters more than one red PR
The 13-minute step timeout is a deliberate deadlock watchdog — the comment beside it explains it's "the outer backstop, well under the job's 15-min cap." That reasoning is sound and preserved here.
But a watchdog five seconds from firing on a healthy run isn't distinguishing a deadlock from a slow day. It reddens PRs at random, and it trains people to re-run without reading — which is exactly how a real hang gets waved through as "the usual flake." That's worse than no watchdog.
The change
Ubuntu only:
job_timeout15 → 22,step_timeout13 → 19.The invariant the surrounding comment depends on — step expires before job, so a missed hang fails with step attribution instead of a silent job-level kill — is preserved and verified on every leg:
Windows is deliberately untouched. It runs the same suite against a 26-minute step cap — still roughly double the observed runtime. Ubuntu is the only leg that lost its headroom.
This is not a fix for anything slow. The suite grew into a cap set when it was shorter. The comment now says to re-check the margin as the suite grows, so the next person meets it as a documented constraint rather than rediscovering it from a green-but-failed run.
Verification
Checked before writing: #71 is the only other open PR touching
ci.yml, and it changes zero timeout lines — no collision.One file, +12/−3. No test, code, or behaviour change.
🤖 Generated with Claude Code