CI flake: telemetry_kill_switch_dispatch loses its dry-run receipt under Ubuntu load (250 ms CLI_PERSIST_TIMEOUT)

[Hmbown]

Symptom

codewhale-cli::telemetry_kill_switch_dispatch::missing_preference_defaults_on_without_inventing_acceptance fails on Ubuntu CI at crates/cli/tests/telemetry_kill_switch_dispatch.rs:180:

thread '...' panicked at crates/cli/tests/telemetry_kill_switch_dispatch.rs:180:18:
default-on writes dry run

Two occurrences, both Ubuntu-only, both on PRs that touch neither telemetry nor the CLI:

PR	Commit	Date
#6104	c810bc0458	2026-09-12
#6267	75e032b61d	2026-09-16

It is not an env race

CI runs cargo-nextest, which gives each test its own process, and this test never mutates process-global state — every case spawns the real binary with env_clear() and a fresh TempDir HOME/CODEWHALE_HOME (telemetry_kill_switch_dispatch.rs:231-276). An intra-process env race is structurally impossible here.

The panic is a missing file: $CODEWHALE_HOME/telemetry/dryrun.jsonl was never written.

Mechanism

features list resolves to Surface::Cli, whose exit path waits CLI_PERSIST_TIMEOUT — 250 ms (crates/telemetry/src/lib.rs:75) — for a detached writer thread that must:

1. re-run decision::re_decide against disk (crates/telemetry/src/actor.rs:162-166), then
2. fsync an append (buffer.rs, file.sync_data())

before the process exits. The path deliberately fails open, so a missed deadline silently produces no receipt and the assertion sees an absent file.

The runner was demonstrably saturated at the failure: neighbouring subprocess-spawning tests took 3.5–3.6 s for work that normally finishes well under a second, while this one failed in 0.747 s.

Partial fix already landed

PR #6270 bounds the binary into the telemetry-contract test group (max-threads = 1). All three pre-existing overrides filtered binary(integration), and this is a different binary, so none matched it — the five cases had been running at full parallelism beside 15,774 tests.

That removes the load which makes the deadline reachable. It does not fix the deadline.

What remains — a behavioural decision

A slow enough host can still miss 250 ms. Options, in rough order of preference:

1. Hoist the disk read out of the deadline window. decision::re_decide re-reads config and setup state from disk inside the 250 ms budget. Resolving it before arming the timer would leave the budget covering only the fsync.
2. Join the writer for the local (non-network) case. A dry-run or pending-buffer append is bounded local work; racing it buys nothing. The network path is the only one that needs a deadline.
3. Raise CLI_PERSIST_TIMEOUT. Cheapest, least principled — it moves the cliff rather than removing it.
4. Assert the outcome, not the artifact. Have the test check the logged telemetry local persistence outcome= line instead of the file, which matches the fail-open contract the code actually implements.

Option 4 alone would make the test honest but would stop pinning the behaviour users care about (a short CLI run does leave its receipt), so it is probably a complement to 1 or 2 rather than a substitute.

Not SHA-6380

The handoff note attributed this to the process-global env-var flake class tracked as SHA-6380. That is a closed tracking issue (#5929) covering six named codewhale-tui lib tests; this binary is not among them, and the nextest process model rules the mechanism out. Recording that here so the misattribution does not get re-derived.

Note on "main is green"

main being green says nothing about this test. On push events the Ubuntu Run tests step is skipped by its if: guard (.github/workflows/ci.yml:632), so this binary only runs on Ubuntu in PR runs. Every Ubuntu-only regression is discovered exclusively in PR CI. Whether that is intended is worth a separate decision.