test-stability: orchestration timing assertions + tests-watchdog exhaustion (two distinct modes)

[GionaGranchelli]

Symptom

CI's tests job fails on :tramai-orchestration:test with timing-sensitive lifecycle assertions. Observed twice on unrelated exact heads in the same review cycle:

Exact head	Failing tests
48e1dafe280b94b2688271d71d563b78f4b4448a (#407)	ShutdownHookTest > recording observer captures lifecycle events during workflow execution and shutdown(), WorkerShutdownCoordinatorTest > shutdown drains a running execution to completion before finishing()
741a76fd96d9fd7fc74a9dc5d2d66400999795ad (#409)	CheckpointPollerTest > poll failure is observable and cancellation is preserved() (java.lang.AssertionError at CheckpointPollerTest.kt:255), WorkerShutdownCoordinatorTest > shutdown drains a running execution to completion before finishing()

Two distinct failure modes tracked here

1. Orchestration timing assertions — test defect. CheckpointPollerTest,
WorkerShutdownCoordinatorTest, ShutdownHookTest fail on event-ordering / drain-completion
assertions with AssertionError. Evidence and attribution in the sections below. The fix
belongs in the tests themselves (deterministic waits instead of fixed windows).

2. Whole-suite watchdog exhaustion — CI calibration defect. The tests job background-runs
./gradlew test and polls it for 18 x sleep 40 = 12 minutes inside a timeout-minutes: 15
step, killing the build when it is still running. Ordinary successful runs of this suite reach
~12.8 minutes, so the containment threshold now overlaps normal runtime and the job is aborted
with no failing test and no tests completed tally. Observed twice on
a4eab8b628f8f037db37ec32f6373eea91baca21 (PR #412, a one-line YAML change), both times
Run tests exceeded 12 minutes — dumping threads and aborting with every downstream step
skipped. Being fixed separately as a calibration change (watchdog 18 -> 21 iterations = 14 min,
hard step timeout unchanged at 15).

They may share runner-load pressure as a contributing factor, but they are not the same
defect: one needs test-level determinism, the other needs CI budget calibration. Do not read
evidence for one as evidence for the other.

Evidence that this is a test/environment flake, not a regression

• Both candidate diffs are unrelated to orchestration: 0.7.1d: governed run attribution & identity continuity #407's failing run was in the same module but under a diff touching JDBC/ops persistence; fix(build-logic): record post-freeze published module in the description oracle #409 changed a single build-logic test fixture (LegacyPublicationDescriptions.kt).
• Local focused reruns pass: ./gradlew :tramai-orchestration:test --tests "*CheckpointPollerTest*" --tests "*WorkerShutdownCoordinatorTest*" → BUILD SUCCESSFUL on the same head.
• An identical-SHA rerun of the failed CI job passes (no new commit).
• Failure text is AssertionError on event ordering / drain completion, not compilation or classpath errors.

Hypothesis to investigate

Fixed timing windows (await/withTimeout budgets), scheduler contention under CI parallelism, and Gradle daemon load sensitivity on the runner. The drain assertions wait for a worker to finish, so a slower runner can observe a shorter event sequence than the fixture expects.

Constraint

Do not "stabilize" these opportunistically inside unrelated PRs — every such PR would otherwise carry an unaudited timing change. Handle it as its own change with a stated window/await policy.

[GionaGranchelli]

Third CI failure mode on this same theme, seen on an unrelated exact head (a4eab8b628f8f037db37ec32f6373eea91baca21, PR #412 — a YAML-only deviation entry):

The tests job was killed by its own watchdog, not by an assertion:

failure | Run tests exceeded 12 minutes — dumping threads and aborting
failure | Process completed with exit code 1.

Job timeline: started 11:48:38Z, aborted 12:01:06Z (12m28s), no tests completed tally was ever printed, and every downstream step (cancellation safety, critical coverage, mutation ratchet, all against PR base and previous master) was skipped after the abort. The log ends with the thread-dump upload and orphan-process cleanup.

For comparison: tests took ~10m27s on #409 and #410 — i.e. this runner is operating within ~1.5 minutes of the 12-minute ceiling, so a slower or busier runner trips the watchdog before any test can fail.

Nothing in the candidate diff can affect suite duration (one line in config/quality/maintainability-deviations.yml). Identical-SHA re-run is the retry path, but the margin is thin enough that this will recur: worth either raising the watchdog budget or addressing suite wall-clock, in addition to the assertion-level flake in the issue body.

[GionaGranchelli]

New failure mode, same lane — proven by thread dump rather than correlation. Extending this issue instead of opening another, as it is the third distinct mode in the tests lane.

Mode 3: MCP test-server teardown hang (fixed in #432)

Observed on 6361362e (PR #431, which touches no orchestration code). The tests job reported no failed test; the watchdog aborted the run:

Run tests made no progress for 271s (limit 270s) — dumping threads and aborting

Thread dump, trimmed:

"Test worker" TIMED_WAITING
  kotlinx.coroutines.runBlocking(...)
  WorkflowMcpStepTest.createEchoServer$lambda$1 → WorkflowMcpStepTest.kt:831
  WorkflowMcpStepTest.tearDown → WorkflowMcpStepTest.kt:48

with MCP stdio reader coroutines still parked in StdioClientTransport.readSource → PipedInputStream.read.

Root cause: at 0f36ab87 the fixture's teardown was

servers.forEach { runCatching { it.close() } }   // tearDown
runCatching { pipeCloseable.close() }            // close()
scope.cancel()
runBlocking { server.close() }                   // no bound

Blocking Java stream reads are not interruptible through coroutine cancellation, so runBlocking { server.close() } had no upper bound. runCatching cannot help here: it catches exceptions, not hangs.

Why it belongs in this issue: unlike mode 2 (calibration — the suite was killed for total runtime), this is an inactivity stall with a concrete thread dump. The watchdog behaved correctly. WATCHDOG_INACTIVITY_SECONDS must not be raised for this; that would only delay diagnosis of the next real deadlock.

Evidence and the bounded, fail-loud replacement are in #432. One correction to the record there: the hang did not reproduce locally at BASE in 100 repeated create/connect/call/close cycles (BASE's close returned every time, ~4.8s for the loop), so the defect is intermittent/order-dependent. The claim that holds is the structural one — teardown was unbounded, therefore it can stall the worker indefinitely whenever a close blocks, which is what CI observed.