ci: convert Test Core mid-suite stalls into labeled fast failures (#4250) - #4309
Merged
Conversation
) Test Core has hung mid-suite with frozen log output three times in one day (#4250): the job sits in_progress with zero new bytes for 20+ minutes while a healthy run finishes in ~9-12. The job-level timeout from #4156 only bounds the hang; it cannot distinguish stopped from slow, and each occurrence still cost a manual log-timestamp diagnosis, a cancel, and a rerun. Make the stall declare itself instead: - scripts/run-with-stall-guard.mjs wraps the turbo test command. It watches output progress -- the mechanical signature #4250 identified (frozen flush timestamp, log length no longer advancing) -- and when nothing is emitted for --stall-minutes (10 in CI, ~5x the longest healthy quiet gap) it prints a verdict naming the last output line, kills the test process group, and exits 75 (EX_TEMPFAIL: every occurrence so far passed on plain rerun of the same commit). - The guard also owns the log tee for check-test-completeness.mjs and propagates the suite's real exit status, retiring the `… 2>&1 | tee` + `set -o pipefail` idiom and its masked-red footgun. - Test Core's timeout-minutes drops 45 -> 30 as a pure backstop: 2.5-3x a normal run, with cold-cache margin. Root cause (suspected unreleased engine/driver handles at worker teardown) is deliberately untouched -- #4250 tracks that; this change makes the next occurrence a clear red at ~10 minutes of silence instead of a half-hour of ambiguous in_progress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014NNaWXVEyG6Pv4EFRfYGg6
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014NNaWXVEyG6Pv4EFRfYGg6
github-actions
Bot
added
documentation
os-zhuang
marked this pull request as ready for review
July 31, 2026 05:31
os-zhuang
added a commit
that referenced
this pull request
Jul 31, 2026
#4309 gave Test Core the output stall guard; the same #4250 failure mode elsewhere still presented as hang-until-timeout. Extend the guard to every remaining suite-running job: - Dogfood shards (ci.yml): the guard replaces the `| tee dogfood.log` + pipefail idiom (log path unchanged, so the completeness guard needs no edit) and declares a stall at 10 min of silence. Job timeout drops 45 -> 30 as a pure backstop, mirroring Test Core. Dogfood boots real engines in-process -- exactly the population #4250's stalls came from. - rerun-safety-nightly: both full passes run under the guard at 15 min (a nightly nobody watches previously hung until the 120-min timeout). Pass 2 now tells the two reds apart: guard exit 75 gets a stall verdict pointing at #4250, everything else keeps the #4065 rerun-unsafe text -- a stall says nothing about working-tree pollution, and the old copy would have sent triage hunting for state leaks that aren't there. - coverage-nightly: wrapped at 10 min; trivial cost. No `... | tee` + pipefail combination remains in suite-running workflows (the `find`-based dist check in ci.yml keeps its pipefail; it is not this pattern). Closes #4314 Claude-Session: https://claude.ai/code/session_014NNaWXVEyG6Pv4EFRfYGg6 Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
落实 #4250 的「让停摆自己说话」:停摆的机械判据是输出不再推进(两次相隔 9 分钟的日志逐字节相同),而 job 级 timeout 只能兜底、无法区分「慢」和「停」。本 PR 把这个判据直接自动化。
改动
新增
scripts/run-with-stall-guard.mjs— 包住 turbo test 命令的输出看门狗:$RUNNER_TEMP/test-core.log(供check-test-completeness.mjs继续读);--stall-minutes(CI 设 10 分钟,约为健康运行最长静默间隔的 5 倍、正常总时长的一半以下)即判定停摆:打印醒目判词(冻结时刻、最后一行输出、"这是 CI:Test Corestalls mid-suite with frozen log output — three occurrences in one day, each costing a manual diagnosis + rerun #4250 型停摆不是慢、rerun 即可"的分诊指引),杀掉整个测试进程组,以 exit 75(EX_TEMPFAIL,语义即"重试")收场;… 2>&1 | tee+set -o pipefail的老 idiom 及其"tee 掩红"隐患(workflow 注释里已标明不要再引回| tee)。ci.yml:Test Core两个测试 step(PR affected / push 全量)改为经看门狗执行;timeout-minutes45 → 30,降级为纯兜底(main 全量约 9.5 分钟、PR 约 12 分钟,30 ≈ 正常的 2.5-3 倍,含冷缓存余量)。效果
下一次停摆的呈现从「in_progress 挂 30+ 分钟 → 人工判读日志时间戳 → 取消 → rerun」变成:静默 10 分钟后一条明确的红,判词直接给出最后输出位置和处置建议,全程无需人工取证。
本地验证
ci.ymlYAML 解析通过;脚本node --check通过明确不做的
疑似根因(每用例新建 ObjectQL engine 后未释放的 driver 连接/定时器句柄拖住 vitest worker 收尾)本 PR 不碰 —— 与 #4250 中「不要求立刻根治」的定位一致,排查方向仍由 #4250 跟踪。
Closes 无(不关闭 #4250,根因仍开放);ref #4250、#4156。
🤖 Generated with Claude Code
https://claude.ai/code/session_014NNaWXVEyG6Pv4EFRfYGg6
Generated by Claude Code