Skip to content

test(cli): make the #4873 exit-code pin deterministic — literal duration, no wall clock (#6266) - #6481

Merged
os-project-manager merged 1 commit into
mainfrom
claude/issue-6266-exit-code-flaky
Aug 8, 2026
Merged

test(cli): make the #4873 exit-code pin deterministic — literal duration, no wall clock (#6266)#6481
os-project-manager merged 1 commit into
mainfrom
claude/issue-6266-exit-code-flaky

Conversation

@os-project-manager

Copy link
Copy Markdown
Collaborator

Fixes #6266

What was wrong

packages/cli/src/utils/format.exit-code.test.ts read its duration off a live wall clock and then asserted a hard-coded truncation of it:

const timer = createTimer();
const durationMs = timer.elapsed() + 531;
...
expect(durationMs & 0xff).toBe(19);

531 & 0xff is 19, 532 & 0xff is 20, 533 & 0xff is 21. The assertion therefore held only while timer.elapsed() returned exactly 0 — and an await emitJson(...) plus module work sits between the two statements. One millisecond ticking on a busy runner turned the case red, which is why it kicked three unrelated PRs in 24h (#6248, #6375, #6429 — the last from a merge-queue generation) through CI's merge ref.

Mechanism chosen, and why

The issue thread carried two candidate shapes. I chose (a) the literal, and folded in the useful half of (b):

  • (a) literal durationconst durationMs = 531;. Zero clock reads, so the truncation assertion is deterministic by construction rather than by luck.
  • (b) keep the live duration, assert only the property — also deterministic in itself (process.exitCode === durationMs compares against the same variable whatever the clock says), but it leaves a Date.now() read in a case where nothing under test is a function of elapsed time. Neither the type rejection nor Node's 8-bit truncation depends on how long anything took, so the clock supplied a failure mode and no coverage. Keeping it invites the next edit to assert something about that variable's value and re-open exactly this bug.

So: (a)'s determinism, plus (b)'s framing — the property assertion and the truncation illustration are both kept, I only removed the wall clock that fed them.

The #4873 guard still bears load

Nothing was weakened to silence the flake. All three load-bearing parts are intact, and I verified each by breaking it:

Load-bearing part How I broke it Result
The two @ts-expect-error directives (the lasting pin) widened CliExitCode to number in format.ts error TS2578: Unused '@ts-expect-error' directive. at both lines 114 and 119
The runtime half — the duration reaches the exit-code slot verbatim clamped emitText to process.exitCode = 1 AssertionError: expected 1 to be 531
The flake mechanism itself restored the old shape, forced 1ms between the two statements AssertionError: expected 21 to be 19 at the & 0xff line — the same signature CI reported as expected 20 to be 19

That third row is worth reading closely: with the millisecond forced, only the & 0xff line went red; expect(process.exitCode).toBe(durationMs) stayed green. The clock-dependent assertion was the one line, and it is the one line this PR changes.

I also added one assertion naming why 19 is a defect rather than a curiosity — it is not one of the two codes CliExitCode defines, so a scripted caller read a successful run as a failure it could not name. In fairness both & 0xff lines are now arithmetic on a literal: they state the truncation rather than measure it. That is deliberate and is what the previous version was too, minus the coin flip.

Determinism evidence

  • Old shape, 200 samples with a 0-2ms gap forced between the two statements: the assertion failed 200/200. Same 200 samples with the literal: 0/200.
  • Fixed file, 20 consecutive runs: 20/20 green.
  • Full @objectstack/cli suite: 91 files, 928 tests, 928 passed. (The CI run that reported this flake was 927/928 with this case as the sole failure.)

Scope

Test-only, one file. No changeset, per the ruling on the issue — an empty changeset stalls the release pipeline (#4898), so the skip-changeset label carries this instead.


Generated by Claude Code

…ion, no wall clock (#6266)

The case read the duration off a live `createTimer()`:

    const durationMs = timer.elapsed() + 531;
    expect(durationMs & 0xff).toBe(19);

which holds only while `elapsed()` returns exactly 0 — and an
`await emitJson(...)` sits between the two statements. One millisecond
turns 531 into 532 and `& 0xff` from 19 into 20, the exact number CI
reported. The clock supplied a failure mode and no coverage: neither the
type rejection nor Node's 8-bit truncation is a function of how long
anything took.

The duration is now the literal 531. The guard keeps bearing load — both
`@ts-expect-error` directives are untouched, and the runtime half still
asserts that the duration reaches the exit-code slot verbatim; the
truncation line is joined by an explicit statement of why 19 is a defect
(it is not one of the two codes `CliExitCode` defines).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017uFVNMmTxLpmfQYiuKM1Yx
@vercel

vercel Bot commented Aug 8, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
objectstack Ignored Ignored Aug 8, 2026 1:38am

Request Review

@github-actions github-actions Bot added the size/s label Aug 8, 2026
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

No hand-written docs reference the 0 changed package(s). ✅

@os-project-manager os-project-manager added skip-changeset PR has no user-facing published change; bypasses the changeset gate and removed size/s labels Aug 8, 2026 — with Claude
@github-actions github-actions Bot added the tests label Aug 8, 2026
@os-project-manager
os-project-manager marked this pull request as ready for review August 8, 2026 01:49
@os-project-manager
os-project-manager added this pull request to the merge queue Aug 8, 2026
Merged via the queue into main with commit b532f8a Aug 8, 2026
31 checks passed
@os-project-manager
os-project-manager deleted the claude/issue-6266-exit-code-flaky branch August 8, 2026 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changeset PR has no user-facing published change; bypasses the changeset gate tests

Projects

None yet

2 participants