Wall-clock-sensitive tests fail intermittently on loaded CI runners

[mohanagy]

Current disposition — 10 September 2026

OPEN — unresolved asynchronous session failures remain here. The former collection has been separated so clock assertions and worker startup are no longer treated as one cause.

Reported behavior	Current owner
Recovery-budget assertion depends on a 1 ms real-time window and 3 ms busy wait	#755 — focused deterministic-clock story
Requested test module fails to start or execute	#756 — deferred, bounded lifecycle diagnosis
MCP discovery times out while a graph request waits for startup	This issue — unresolved
An active stdio session times out while refreshing after a workspace change	This issue — unresolved

For the two remaining session cases, the desired outcome is that the public session remains responsive and observes a workspace change. Their causal explanation is still uncertain. Preserve the original failures and existing guards. A same-code retry passing does not prove an infrastructure-only cause or exclude a product race; the historical heading claiming these are not product regressions is not an established conclusion.

Do not close this issue when #755 is fixed. Close it only when the remaining session failures have specific resolved evidence, or when each has a focused successor retaining the evidence and unresolved status. No diagnosis, tests, timeout changes or implementation are launched by this triage.

This issue is independent of #754 except when a concrete failure prevents trustworthy execution. It is no longer an umbrella directing automatic recovery work.

Historical issue body — preserved; current disposition above takes precedence

Current disposition — 5 September 2026

OPEN — independent timing and worker-lifecycle maintenance. Current roadmap: #740. Keep deterministic test-timing failures distinct from worker-start/lifecycle failures; preserve the guarded requested-versus-executed test ledger and do not treat isolated passes as a full-suite pass. This work is independent of #741's research and evaluation-contract preparation, and blocks future evaluation only when trustworthy execution is prevented. This reconciliation does not authorize coding or automatically resume implementation.

The original scope and evidence below, and all existing comments, are preserved as history.

---

Three consecutive protected matrices during PR #709 each lost exactly one lane,
a different lane each time, always to a wall-clock-sensitive test. Two of the
tests are untouched by that PR. This issue collects the family so it is tracked
rather than absorbed as noise.

Affected tests

Test	File	Touched by #709
keeps MCP discovery responsive while one graph request waits for slow startup	tests/unit/background-auto-refresh.test.ts	no — last changed 2026-07-17
refreshes an active stdio session after an agent changes its workspace	tests/unit/stdio-server.test.ts	yes, read migration only (218dd7c7)
does not relabel a completed final attempt as a time-budget abort	tests/unit/context-pack-recovery.test.ts	no

Evidence

Run	Lane	Test	Observed
32142031007	windows-latest Node 20	background-auto-refresh	timed out at 30,008 ms
32145436260	windows-latest Node 20	same test	4,009 ms — pass
32148986352	windows-latest Node 22	stdio-server watcher	timed out at 11,344 ms
32145436260	windows-latest Node 22	same test	4,442 ms — pass
32139276516	windows-latest Node 22	same test	4,719 ms — pass
32150482216 (1st attempt)	macos-latest Node 22	context-pack-recovery	expected 1 call, got 0
32150482216 (rerun)	macos-latest Node 22	same test	pass

The background-auto-refresh case was retested on byte-identical production
code — 71cce3f3 is documentation-only on top of a8a94a8c — and ran in
3,615 ms. Same code, both a timeout and a 3.6 s pass.

Why these are not product regressions

The outcomes are bimodal: roughly 4 s or the full timeout, never a gradual
slope. Two of the three tests have no commits in PR #709 at all. Every failure
was retested on identical code and passed. Nothing was changed to make any of
them pass — no timeout raised, no test skipped, no assertion loosened.

context-pack-recovery is the clearest of the three: it asserts one call under
maxElapsedMs: 1 while the callback it measures busy-waits 3 ms. On a loaded
runner the budget can expire before the first call is made, giving 0 calls.
A 1 ms wall-clock budget is not reliably observable on a shared CI runner.

Why raising Vitest timeouts is not the fix

Measured on vitest 4.1.10 in this repository: a synchronous test body that
busy-waits 3,000 ms under a 1,000 ms timeout runs to completion and is only
then reported as timed out. A per-test timeout is therefore a post-hoc
elapsed-time verdict with no phase attribution and no deadlock protection
for the many synchronous tests here. Raising a budget past a real one hides
contention rather than preventing a hang; real hang protection has to come from
timeout on each execFileSync/spawnSync call, or from moving the work into
a child with a parent-side deadline.

Suggested direction

• Replace absolute wall-clock budgets with injected clocks or fake timers where
  the test is about ordering rather than duration.
• Where a real deadline is the contract, assert it in a child process with a
  parent-side timeout so a regression fails fast and attributably.
• Keep the durations in the record: a test that normally runs in 4 s against a
  30 s budget is not short of budget, and raising it would not have helped.

Refs #705.

[mohanagy]

Another instance, and the cleanest controlled evidence yet — both lanes in the
same matrix on the same head.

Lane	Head	Test	Result
windows-latest Node 22	9502d3d2	benchmark-suite > writes summary.json and summary.md with per-repo rows and no aggregate headline	timed out at 30,000 ms
windows-latest Node 20	9502d3d2	same test	1,394 ms — passed, 272/272 files

Prior runs of the same test on the same Windows Node 22 lane: 1,149 ms
(32150482216) and 1,322 ms (32145436260).

tests/unit/benchmark-suite.test.ts has no commits in PR #709. The
production change on that head reworks how artifact files are opened, and the
test does exercise artifact paths — which is why it was checked rather than
assumed. The sibling lane running the identical code in 1.4 s rules that out:
a 25× excursion on one lane and not the other, from the same binary, is
scheduling, not semantics.

Run 32172774837, job 95827605400.

Adds tests/unit/benchmark-suite.test.ts to the affected set.

[mohanagy]

Local full-suite timeouts observed during #658 Stage 2 validation

Recording an observation against this issue rather than fixing it in #658.

Running npm test on the #658 branch at 24ba54ef (macOS, Node v22.22.3):

Test Files  4 failed | 264 passed (268)
Tests       7 failed | 3699 passed | 1 expected fail | 2 skipped (3709)

Every failure is a timeout — Test timed out in 60000ms and
Test timed out in 15000ms — in these files:

tests/unit/artifact-special-files.test.ts
tests/unit/compare-native-agent.test.ts
tests/unit/release-pipeline.test.ts
tests/unit/run-guarded-vitest.test.ts

All 193 tests in those four files pass when the files are run alone, so this
is contention under parallel execution rather than a functional failure.

The victim set also shifts between runs. Earlier full-suite runs on this branch
and at the pre-remediation baseline dccf00cf produced overlapping but
different sets, including benchmark-artifact and qualification-contract,
which is the signature of a timing-sensitive condition rather than a specific
broken test.

This is not a #658 functional regression: no #658 suite appears in any failing
set, and none of these files references the #658 modules. It is recorded here
so the local suite is not described as green anywhere in #658, and is not being
remediated on that branch.

[mohanagy]

Repository-wide local suite at 63cf17ad — 10 timeout failures, none functional

Full npm test run on the #658 branch at 63cf17ad, macOS, Node v22.22.3.

Test Files  6 failed | 263 passed (269)
Tests      10 failed | 3801 passed | 1 expected fail | 4 skipped (3816)
Duration   1050.53s

Exact failed-test inventory

artifact-special-files.test.ts
  > special files are never treated as graph artifacts
    > opens a FIFO with the production flags without waiting

benchmark-tooling-split.test.ts
  > pack metrics never turn a missing field into a zero
    > reads the serialized token count and matched nodes

compare-native-agent.test.ts
  > executeNativeAgentCompare
    > classifies implement outcomes from isolated touched-file overlap and passing validation
    > records implement-task outcome scoring with isolated per-arm workspaces and an external graph artifact

release-hygiene.test.ts
  > release hygiene
    > requires prerelease README changelog links to target next
    > requires the README changelog link to match the current release heading exactly

release-pipeline.test.ts
  > next release state guards
    > preserves latest while moving next to the published version
  > release tag classifier
    > classifies stable SemVer tags

run-guarded-vitest.test.ts
  > guarded vitest CLI
    > deletes successful logs and retains failed logs under explicit paths
    > detects and retains a failure at a log path containing spaces and Unicode

Failure mode

Every failure is a timeout. There are no assertion failures.

7 x  Error: Test timed out in 15000ms.
2 x  Error: Test timed out in 60000ms.
1 x  Error: spawnSync ... node ETIMEDOUT

The victim set shifts between runs

Membership changes run to run while the family stays the same, which is the
signature of a timing-sensitive condition rather than specific broken tests:

head	failing files
dccf00cf	artifact-special-files, benchmark-artifact, compare-native-agent, qualification-contract
ba3fb9f1 era	artifact-special-files, compare-native-agent, release-hygiene, release-pipeline, run-guarded-vitest
later run	compare-native-agent, qualification-contract, release-pipeline, run-guarded-vitest
24ba54ef	artifact-special-files, compare-native-agent, release-pipeline, run-guarded-vitest
63cf17ad	the six listed above

benchmark-tooling-split and release-hygiene appear in some runs and not
others; qualification-contract and benchmark-artifact appear in earlier runs
and not this one. No file is consistently red.

An orphaned process on this machine

Worth recording as an environment observation. A vitest process from a
different worktree has been hung for nearly two days:

PID 11251, started Tue 18 Aug 19:52, parent init
node .../madar-705/node_modules/.bin/vitest run \
  tests/unit/artifact-special-files.test.ts --maxWorkers=1

It is blocked on that test's FIFO probe -- openSync(path, 'r') on a FIFO with
no writer waits indefinitely -- and holds two Node processes.

It is not the cause of these timeouts. The obvious hypothesis was that it
collides with the same-named test here, and that hypothesis is wrong: the test
creates its FIFO under mkdtempSync(join(tmpdir(), 'artifact-special-')), so
every run gets a unique path and no cross-worktree collision is possible. The
stray adds background load and should be cleaned up, but it does not explain the
failures and is not being recorded as if it did.

Isolation

All 220 tests across those six files pass when the files are run alone.

[mohanagy]

Worker-start failures presenting as a smaller green summary — lifecycle evidence

Instrumented diagnosis of silent module omission during #658 qualification.

branch head        06423547 (+ uncommitted qualification-harness work)
manifest           tests/manifests/658-focused.json
manifest checksum  98437e747f8f9c3e8154e63f03c051a4f32820760b1e0905d48124299f4b32a5
requested modules  21

The failure

The underlying failure is worker startup.
The misleading smaller-green summary is the observable consequence.

No claim is made that Vitest failed to discover or filter the files.

Vitest resolved all 21 specifications correctly. Every omitted module reached
specificationResolved and then never reached queued. For each one, Vitest
emitted an unhandled pool error:

[vitest-pool]: Failed to start forks worker for test files \
  .../tests/unit/graph-integrity-receipt.test.ts
[vitest-pool]: Failed to start forks worker for test files \
  .../tests/unit/integrity-record-share-safety.test.ts
[vitest-pool]: Failed to start forks worker for test files \
  .../tests/unit/integrity-validation-policy.test.ts
[vitest-pool]: Failed to start forks worker for test files \
  .../tests/unit/integrity-record-identity.test.ts

Failed to start forks worker is the first entry in this repository's own
WORKER_FAILURE_SIGNATURES, so this is the existing timing/environment family
rather than a new defect.

The result summary counts only modules that produced results. Four modules whose
workers never started therefore rendered as a smaller passing total instead of an
error — a green summary over a silently reduced file set.

Invocation types tested

invocation	requested	executed	omitted
npx vitest run with 21 literal paths	21	12	9
./node_modules/.bin/vitest run with 21 literal paths	21	17	4
programmatic runTestSpecifications	21	15	6
instrumented programmatic, attempt 1	21	21	0
instrumented programmatic, attempts 2–6	21	17	4

Attempt 1 was clean; attempts 2 through 6 each lost the same four modules. The
condition is load-sensitive rather than file-specific — earlier invocations lost
different subsets.

First missing lifecycle event, attempts 2–6

Identical on every attempt:

tests/unit/graph-integrity-receipt.test.ts
  reached: specificationResolved
  first missing: queued
  pool error: Failed to start forks worker

tests/unit/integrity-record-identity.test.ts      (same)
tests/unit/integrity-record-share-safety.test.ts  (same)
tests/unit/integrity-validation-policy.test.ts    (same)

Why it went unnoticed

The repository's canonical guarded runner (scripts/run-guarded-vitest.mjs)
already scans for exactly this signature. The safeguard existed and worked; the
multi-file commands used during qualification bypassed it by invoking Vitest
directly.

Two changes follow on the #658 branch, neither touching production source:

• the [P0][Graph PR C] Account for every normalized graph candidate and retain unresolved facts #658 focused qualification now runs one module per process through the
  guarded path, with a requested-versus-executed ledger and an independent
  per-entry signature scan, so a worker-start failure can never again present as
  a smaller passing count;
• raw multi-file Vitest invocation is diagnostic only and is not accepted as
  qualification evidence.

A prior aggregate count published from a bypassing invocation has been marked an
unverified subset rather than deleted. No #658 production change was made in
response to this issue.

[mohanagy]

Retained as architecture-independent test reliability work under #735. Non-blocking for #736 unless it prevents trustworthy prototype evaluation.