[P0] Stabilize the complete Vitest suite and protected CI merge gate

[mohanagy]

Parent: #649
Blocks merge of: every roadmap implementation issue

Outcome

Make the complete test suite a reliable, reproducible merge gate on supported environments.

No roadmap PR may be described as merge-ready while the complete suite can time out, fail to start workers, or fail to terminate for unexplained reasons.

Verified baseline evidence

At 06b373a447acfce895412ac10eb4e5228c5df0b7:

• build passed;
• typecheck passed;
• 206 focused tests passed;
• a broader run under Node 22.22.3 ended with one timeout and worker-start failures;
• 530 tests passed before the run ended;
• the timed-out benchmark test passed alone with --maxWorkers=1 in roughly three seconds.

Observed affected tests included:

• tests/unit/compare-native-agent.test.ts near the previously identified timeout;
• tests/unit/extract-frameworks.test.ts worker startup;
• tests/unit/retrieve.test.ts worker startup;
• tests/unit/stdio-server.test.ts worker startup.

This points to suite-level resource pressure, worker scheduling, ordering/global state, filesystem/process contention, or a lingering handle. It is not yet proof of any one cause.

Reference environment

Use and record:

• Node 22.22.3 or a repository-supported equivalent recent Node 20/22 patch line;
• npm 12.0.1 or the lock-compatible package manager version;
• exact package-lock.json;
• audited lock SHA-256: 0144eb0ddf92f78c69f10089d0e0414485594966ff5be4f36f655c7aa5cff53e.

Node 22.9.0 is not an acceptable qualification environment because locked tooling requires a newer patch line.

Scope

1. Reproduce the failure on a clean checkout with exact logs, environment, process/memory observations, and test ordering.
2. Determine whether the cause is:
   • four-worker resource pressure;
   • worker startup starvation;
   • global state or ordering interaction;
   • filesystem/process contention;
   • lingering handles or child processes;
   • benchmark harness behavior;
   • platform-specific behavior;
   • another evidenced cause.
3. Add the smallest safe correction.
4. Preserve or improve parallelism rather than hiding the defect globally.
5. Add diagnostic output sufficient to identify future worker-start or nontermination failures.
6. Verify protected CI on the supported Node/OS matrix.

Explicit non-goals

• Do not increase global timeouts merely to hide contention.
• Do not permanently force the entire suite to --maxWorkers=1 without evidence that this is the correct supported policy.
• Do not skip, quarantine, or weaken assertions for the affected tests merely to produce green output.
• Do not retry until a run happens to pass and call the issue resolved.
• Do not mix graph, retrieval, Pack, extraction, or product behavior changes into this issue.
• Do not expand the supported Node claim beyond what the locked toolchain actually supports.

Investigation requirements

Capture for every failed attempt:

• exact commit;
• OS/architecture;
• Node/npm versions;
• lock checksum;
• command and Vitest options;
• worker count;
• peak memory/process observations where practical;
• full stderr/stdout;
• failed or unstarted test files;
• whether affected tests pass independently;
• whether order changes the result;
• whether the process exits cleanly.

Use binary-search or grouped test runs where useful. Any suspected open handle must be demonstrated rather than inferred from nontermination alone.

Acceptance criteria

• Root cause is documented with reproducible evidence.
• The correction is narrowly scoped and does not weaken test semantics.
• Three consecutive complete local runs succeed under the pinned reference environment, or a documented local-resource distinction is established and the complete protected CI matrix succeeds repeatedly.
• No worker-start errors occur.
• No unexplained timeout or nontermination remains.
• Focused affected tests still pass independently and as part of the full suite.
• Exact failing logs from the original and corrected runs are retained in the issue/PR artifacts.
• Supported Node documentation and package/tooling constraints agree.
• The repository has one authoritative complete-suite command used by protected CI.

Validation

At minimum run:

npm ci
npm run typecheck
npm run build
npm run test:run
npm run test:coverage
npm run verify:pack-parity
npm pack --dry-run
npm run registry:validate
npm run release:verify

Also run the affected files alone and in the grouping that originally reproduced the failure.

Agent execution contract

• Begin from a clean supported environment.
• Do not modify product semantics.
• Stop and report when the evidence contradicts the working hypothesis.
• Return exact commands, logs, files changed, before/after behavior, and remaining uncertainty.

Closure rule

Close only when the complete suite is a trustworthy protected merge gate, not merely when one local run passes.

[mohanagy]

Status: partial fix landed, root cause of the broader symptom not yet established

Draft PR: #682 (does not close this issue — see verdicts in the PR description).

Environment

• Worktree base: 06b373a447acfce895412ac10eb4e5228c5df0b7 (audited commit)
• Node v22.22.3 (matches issue reference)
• npm 12.0.2 — issue names 12.0.1; flagging this discrepancy, not normalizing it
• package-lock.json SHA-256 0144eb0ddf92f78c69f10089d0e0414485594966ff5be4f36f655c7aa5cff53e — matches the audited value exactly

What was found and fixed

tests/unit/stdio-server.test.ts > stdio runtime > refreshes an active stdio session after an agent changes its workspace intermittently timed out at its 10s limit in the flagged 4-file group (compare-native-agent.test.ts, extract-frameworks.test.ts, retrieve.test.ts, stdio-server.test.ts) at --maxWorkers=4. Debug instrumentation showed it stuck in event_mode: 'polling', waiting on the production default auto-refresh poll interval — too slow to land inside the test's 10s budget under concurrent load. Injecting a fast poll interval through the pre-existing autoRefreshStarter/startGraphAutoRefresh seams (already present in src/runtime/stdio-server.ts and src/infrastructure/watch.ts — no new production seam added) made the failure disappear deterministically: 303/303 passing across --maxWorkers=1/2/4, repeated, vs. 1 failure before the fix.

This is a real, narrowly-scoped fix. It is not the root cause of the issue's central symptom.

What remains unresolved: full-suite worker-start failures

Both before and after the fix above, full npm run test:run / npm run test:coverage runs fail with:

Error: [vitest-pool]: Failed to start forks worker for test files <file>.
Caused by: Error: [vitest-pool-runner]: Timeout waiting for worker to respond

This is an IPC handshake timeout, not EMFILE/ENOMEM (ulimit -n is 1048576 on this machine).

Across 6 full-suite runs (3 before the fix, 3 after, including one run I executed myself fresh after the fix landed), the affected files were different and effectively random every time — never the same set twice:

• Run 1 (before): 29 files failed to start (cli.test.ts, benchmark-suite.test.ts, install.test.ts, implementation-pack.test.ts, context-pack.test.ts, benchmark.test.ts, watch.test.ts, stdio-slice-surface.test.ts, retrieve-production-correctness.test.ts, why-madar-doc.test.ts, pr-impact.test.ts, and others) — 609 failed / 1748 passed tests, 340.62s
• Run 2 (before): 8 files (install.test.ts, implementation-pack.test.ts, benchmark.test.ts, watch.test.ts, stdio-slice-surface.test.ts, retrieve-production-correctness.test.ts, why-madar-doc.test.ts, pr-impact.test.ts) — 4 failed / 2515 passed, 373.16s
• Run 3 (before, coverage): 11 files (spi-cache, spi-framework-nextjs, benchmark-artifact, extract-go, spi-symbols, spi-calls, spi-build, spi-nest-di-runtime-calls-realistic, spi-framework-redux, pack-quality-fixtures, release-hygiene) — 12 failed / 2624 passed, 560.49s
• Run 4 (after the fix, coverage): 5 files (spi-projector, spi-cache, spi-framework-nextjs, benchmark-artifact, extract-go, spi-symbols, generate-spi-flag, mcp-registry-metadata, spi-calls, spi-build, spi-nest-di-runtime-calls-realistic) — 12 failed / 2550 passed, 597.13s
• Run 5 (after the fix, test:run, run fresh by me independently): 12 files (benchmark-artifact, extract-go, generate-spi-flag, mcp-registry-metadata, pack-quality-fixtures, pipeline, spi-cache, spi-calls, spi-framework-nextjs, spi-framework-route-metadata, spi-projector, spi-symbols) — 2 failed (both 60s timeouts in compare-native-agent.test.ts, matching this issue's originally recorded symptom) / 2605 passed, 511.37s, exit 1

No two runs failed on the same file set. This is the signature of resource starvation at the point Vitest starts a worker, not a defect in any specific test file.

Corroborating evidence (independent, cross-branch)

A teammate working on an unrelated branch (roadmap/656-plural-graph-apis, different code changes entirely) reproduced the identical signature: running 21 test files concurrently produced 8× Failed to start forks worker plus 2× 60s timeouts in compare-native-agent.test.ts; small batches (~40 files split up) had zero failures across 1000+ test cases; every file that failed to start passed cleanly when run alone. This reproduces cross-branch and cross-codebase, which argues against a specific broken test or a defect introduced by any particular change, and for a suite-level/environmental cause tied to concurrency/batch width.

Leading hypothesis: local machine resource contention (not yet confirmed against protected CI)

During one failing run, an lsof snapshot taken at the moment of failure showed multiple concurrent node processes with cwd pointing at a different worktree (this machine's primary checkout, not the one under test) plus other unrelated Node processes running simultaneously — i.e., this development machine had other concurrent agent/test workloads competing for CPU during the run. Worker-handshake timeouts under system-wide contention is consistent with all the evidence above (random affected files, files passing cleanly alone or in small batches, cross-branch reproduction).

This has not been confirmed against the actual protected CI matrix (dedicated, single-purpose GitHub Actions runners with no concurrent unrelated load). That is the next evidence needed: repeated complete protected CI runs across the {ubuntu, macos, windows} x {node 20, node 22} matrix, watching specifically for Failed to start forks worker / worker-handshake timeouts.

Acceptance criteria status

• Root cause documented with reproducible evidence — partially: one contributing test-level cause is fully root-caused and fixed; the dominant full-suite symptom is not.
• The correction (stdio-server.test.ts fix) is narrowly scoped and does not weaken test semantics.
• Three consecutive complete local runs succeed under the pinned reference environment, or a documented local-resource distinction plus repeated protected CI success — neither achieved. Local-resource distinction is hypothesized with supporting evidence (lsof snapshot, cross-branch corroboration) but not yet confirmed against protected CI.
• No worker-start errors occur — not met, reproduced fresh as of this comment.
• No unexplained timeout or nontermination remains — not met for the broader symptom; the one instance we could fully explain is fixed.
• Focused affected tests still pass independently and as part of the full suite — true for the 4-file flagged group specifically.
• Exact failing logs from the original and corrected runs are retained (in PR test: fix flaky stdio auto-refresh polling timeout (partial evidence for #654) #682's investigation, not committed to the repo — .issue-654-artifacts/ is gitignored).
• Supported Node documentation and package/tooling constraints agree — npm version discrepancy (12.0.2 local vs. 12.0.1 issue-named) unresolved, flagged only.
• One authoritative complete-suite command used by protected CI — unchanged, not evaluated as part of this partial fix.

Recommendation

Do not close #654. Merge PR #682 as a legitimate partial improvement. Continue #654 with protected-CI-matrix evidence (dedicated runners, no concurrent load) to test the local-resource-contention hypothesis, since that is the only remaining path to satisfying the issue's acceptance criteria without further repository changes — or, if CI reproduces the same symptom on dedicated runners, the hypothesis is falsified and a different, still-undetermined cause must be investigated.

[mohanagy]

Protected CI evidence on exact head af16e41b14a4e63f2dc6b92025c520effc79ecfc (PR #682), plus a newly discovered, undiagnosed flake

Ran 3 complete six-job protected-CI attempts on the exact PR #682 head, sequential, each run to completion (gh run rerun + gh run watch --exit-status), workflow run 31568735268.

Attempt	Trigger	Result	Per-job
1	PR push	✅ 6/6 green	macos20 2m39s, macos22 2m29s, ubuntu20 2m22s, ubuntu22 7m23s (coverage), windows20 7m37s, windows22 6m56s
2	gh run rerun	❌ 5/6 green, 1 failed	validate (ubuntu-latest, Node 20) FAILED — see below. Others green: macos22 3m43s, ubuntu22 7m28s, windows20 6m8s, windows22 8m24s, macos20 3m39s
3	gh run rerun	✅ 6/6 green	macos22 3m17s, ubuntu20 3m32s, macos20 4m18s, windows20 7m24s, ubuntu22 7m32s, windows22 7m51s

The forks-worker symptom: 0/18 job-runs on CI vs 6/6 local runs

None of the 18 job executions across these 3 attempts produced Failed to start forks worker / Timeout waiting for worker to respond — the symptom this issue exists to resolve. That contrasts with 6/6 local full-suite runs in this investigation reproducing it (see the earlier evidence-only comment). This 0-of-18-vs-6-of-6 split is real, meaningful evidence for the working hypothesis that the forks-worker symptom is a local/shared-machine resource-contention artifact rather than a defect reproducible on dedicated, single-purpose CI runners.

But attempt 2's failure is a new, separate, undiagnosed flake — not noise, not this symptom

FAIL tests/unit/watch.test.ts > watch > backs off authoritative reconciliation while idle and reports resource measurements
AssertionError: expected false to be true // Object.is equality
❯ tests/unit/watch.test.ts:478:80
Test Files  1 failed | 203 passed (204)
Tests  1 failed | 2803 passed | 2 skipped (2806)

Job: validate (ubuntu-latest, Node 20), run 31568735268, attempt 2, job id 94059711350. All 204 test files completed normally in this job — zero worker-start errors, nothing missing. This is unrelated to PR #682's diff (tests/unit/stdio-server.test.ts, .gitignore) and unrelated to worker-start starvation; it's a distinct pre-existing test at origin/main.

The test (tests/unit/watch.test.ts:459-479) starts a watcher with pollIntervalMs: 20, maxPollIntervalMs: 80, waits a fixed await delay(190) of wall-clock time, then asserts the backoff sequence reached both the 40ms and 80ms rungs within that window:

await delay(190)
controller.abort()
await watcher

expect(reconciliations[0]).toMatchObject({ trigger: 'initial', fileCount: 1, nextIntervalMs: 20 })
expect(reconciliations.some((metrics) => metrics.nextIntervalMs === 40)).toBe(true)
expect(reconciliations.some((metrics) => metrics.nextIntervalMs === 80)).toBe(true)   // line 478 — the assertion that failed
expect(reconciliations.length).toBeLessThanOrEqual(5)
expect(reconciliations.every((metrics) => metrics.durationMs >= 0 && metrics.directoryCount >= 1)).toBe(true)

This is squeezed from both sides and can't be stabilized by simply widening the wall-clock window: it must reach the 80ms backoff rung inside a fixed 190ms budget (a slower/momentarily-loaded runner misses it — exactly what happened here), and it must keep reconciliations.length <= 5 (a faster runner risks exceeding that upper bound if the window is widened). It is a genuine timing race baked into the test's construction, not simply "flaky CI infra."

Why this matters for #654's outcome, not just as a side note

The stdio-server.test.ts fix in PR #682 and this watch.test.ts failure are the same family of defect: a test asserting real-time-sensitive polling/backoff behavior against a fixed wall-clock budget, which passes on a fast/unloaded machine and fails under load or scheduling jitter. That's now two independent instances of this pattern found in one investigation. It suggests part of this suite's unreliability may be a timing-assumption problem in specific tests, distinct from (and possibly in addition to) worker-scheduling/resource-pressure causes — worth factoring into how #654's root cause is ultimately framed, alongside the forks-worker hypothesis.

Net effect on #654's acceptance criteria

• Criterion 3 ("three consecutive clean local runs, OR local-resource distinction + repeatedly passing protected CI") is not met on a plain reading of "repeatedly" — 2 of 3 CI attempts on the exact same head were clean, one was not.
• Criterion 5 ("no unexplained timeout or nontermination remains") is now unmet for two independent reasons: the local forks-worker symptom, and this newly surfaced watch.test.ts timing flake.

Recommend: this belongs to #654's stated outcome ("the complete suite is a reliable, reproducible merge gate"), but I have not investigated or touched watch.test.ts — flagging it here as newly discovered and undiagnosed for a maintainer to decide whether to track it as part of #654 or split it into its own issue.

No product source was changed as part of gathering this evidence. PR #682 remains unaffected — its diff is not implicated in the attempt-2 failure.

[mohanagy]

PR #682 merged as a standalone test-flakiness fix — #654 remains open

PR #682 has been squash-merged as a standalone test-flakiness and diagnostic correction only. It does not resolve this issue and must not be treated as satisfying it.

Item	Value
Merged PR	#682
Exact PR head SHA	af16e41b14a4e63f2dc6b92025c520effc79ecfc
Squash commit on main	3371ada8425efa7f8cabdac781fa227feaea7a6a
Merge method	squash, with --match-head-commit exact-head protection
Production source changes	none — .gitignore and tests/unit/stdio-server.test.ts only
Exact-head protected CI	6/6 green

What it fixes

The stdio auto-refresh test depended on the production default polling interval, which is too slow under load to land inside the test's 10s budget. It now injects a fast poll interval through the pre-existing autoRefreshStarter / startGraphAutoRefresh seams — no new seam was introduced — adds timeout-detail diagnostics to the waitFor helper, and ends the input stream before destroying it to release a lingering handle.

What it does not fix

The broader complete-suite reliability problem. Current evidence:

• The forks-worker startup failure reproduced in 6 of 6 full runs on the shared local environment.
• It reproduced in 0 of 18 dedicated GitHub Actions job executions.
• That strongly supports a local / shared-machine contention distinction for that failure class.
• However, one repeated protected matrix failed on a separate assertion — the tests/unit/watch.test.ts idle-backoff timing assertion.
• The repeated protected-CI requirement therefore stands at 2 successful matrices out of 3.
• Uncontrolled wall-clock assertions remain an active repository-level reliability concern.
• This issue's acceptance criteria remain unmet, and this issue remains open.

Incident — automatic closure, corrected

Merging this PR automatically closed this issue, and it was reopened immediately.

Cause, recorded precisely: PR #682's body correctly used only Refs #654 with no closing keyword. The auto-closure was triggered by the squash commit message, which contained the sentence "This does NOT resolve #654". GitHub's closing-keyword parser matches the substring resolve #654 and does not interpret negation, so a sentence written to deny resolution was read as declaring it.

The lesson is worth keeping: when a commit or PR body must state that it does not close an issue, the issue reference and the words close / fix / resolve must never appear adjacent in any grammatical form, negated or otherwise. Phrase it as "#654 remains open" rather than "does not resolve #654".

The squash commit message on main retains that wording permanently, but the closure event fires only once at merge, so no further automatic closure will occur.

[mohanagy]

Evidence refinement: a green local test:run does not prove the worker-start symptom is absent

Surfaced while hardening the prerelease pipeline on PR #688. No code here changes anything tracked by this issue; this is an evidence-quality correction that affects how earlier results should be read.

1. npm run test:run can report green while the symptom occurred

vitest.config.ts sets maxWorkers: 4 and defines no test.retry. In a local run, 8 files logged Failed to start forks worker / Timeout waiting for worker to respond, yet the same single invocation's final tally counted all 8 as passed: 182 files, 2131 passed, 2 skipped, 0 assertion failures.

That recovery is vitest's forks-pool respawning workers inside one run. It is not a configured retry, and it was not a manual rerun. The green summary line does not disclose the failed starts — only the full log does.

Consequence: any historical "the suite passed locally" claim is weaker than it appears, because a pass is compatible with the symptom having occurred and been absorbed. Future evidence for this issue should grep the full log for the worker-start strings rather than trusting the summary.

2. The dedicated-runner result is not an absorption artifact — it holds

The obvious worry is that CI's green runs absorbed the same failures silently, which would have made the earlier "0 of 18 job-runs" claim meaningless. It does not.

I downloaded the raw logs for all three attempts of run 31568735268 (the exact-head matrix on af16e41b) and searched them directly:

Failed to start forks worker      → 0 occurrences
Timeout waiting for worker        → 0 occurrences

Method validated with controls on the same extracted logs so a zero could not be a silent read failure: vitest appears in 6 job logs and Test Files in 6 job logs.

So the dedicated-runner evidence is stronger than first reported: the symptom did not merely fail to break a job, it never appeared in the logs at all. Local 6/6 versus CI 0/18 stands, and now rests on log content rather than job conclusions.

3. Direct observation of the contending process

During a second local run, ps aux captured an unrelated project's npm exec vitest run … executing roughly 15 forked workers on the same machine at the moment of failure. That is a concrete, observed contention source rather than an inference from randomness.

Net effect on this issue

The local-resource-contention hypothesis is better supported: a specific competing multi-worker process was observed, and dedicated runners show no trace of the symptom in raw logs. What remains unresolved is unchanged — the mechanism is still not proven, criterion 5 is still unmet, and the separate tests/unit/watch.test.ts wall-clock backoff flake is still undiagnosed.

One practical suggestion for whoever picks this up: consider making a fork-worker start failure fail the run rather than being absorbed, so the gate stops reporting green over a symptom this issue exists to eliminate. That is a change to test infrastructure and belongs to this issue's scope, not to PR #688.

#654 remains open.