Skip to content

test(cli): run the telemetry mutation 7x faster without weakening it - #1043

Merged
blafourcade merged 16 commits into
nextfrom
test/faster-mutation
Oct 16, 2026
Merged

blafourcade merged 16 commits into
nextfrom
test/faster-mutation

Conversation

@blafourcade

@blafourcade blafourcade commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

🎯 What & why

The telemetry mutation run took 45 to 62 min locally. Each mutant ran the slow integration test files before the unit test that would have killed it in milliseconds. After this change the run takes 6.7 min locally (CI shards: 3.6, 4.9 and 6.9 min). No test was removed or weakened.

🛠️ How it works

  • Unit tests run first, smallest file first, for mutation only. A sequencer (mutation-sequencer.ts) is loaded only by vitest.mutation.config.ts. Stryker stops a mutant at its first failing test, so the cheapest test that kills it now runs first. The sequencer changes the order of tests, never which tests run. pnpm test keeps its order. Measured: 45.5 to 8.9 min.
  • Fewer static mutants (330 to 158). Values that tests built at module load are now built inside the tests that use them. The assertions and it.each titles are unchanged. Test runs per full telemetry run drop from 94k to 39k.
  • timeoutFactor 3 (stryker.conf.json:9). 165 of the old timeouts were slow kills, not hangs, and are now counted as killed. timeoutMS stays at 20000, the value test(cli): count a mutant as killed only when a test failed #844 set.
  • pnpm test. The 33 s telemetry report e2e file is split into 3 files, with 22 tests before and after. The golden tests capture each cell twice instead of three times; the determinism check (capture 2 against capture 1) is kept. The shared code lives in two-captures.ts. The overall wall-time gain is within noise; the e2e subset went from 17.1 s to 12.1 s.

🧪 How to verify

  • cd cli && pnpm test: 6641 passed, 1 skipped.
  • pnpm test:mutation:telemetry, compared mutant by mutant with the baseline:
    • no mutant killed in the baseline survives;
    • 165 went from Timeout to Killed;
    • 7 went from Timeout to Survived, and they already survived in a fresh baseline run on unchanged code;
    • 2 went from Killed to Timeout, and these are real hangs in ledger-lock.ts.
  • Every mutation scope holds its floor. framework is at 93.0 against 93.1 in CI before this change, with no regression per mutant.
  • Sequencer guard: mutation-sequencer.unit.test.ts goes red when sort returns the files unsorted, or when a file size reads 0.
  • Timings, the per-mutant comparison and the load on each run are in evidence.md. Independent review, 2 rounds: review.md.

⚠️ Heads-up

  • Telemetry score: 96.35, against 96.52 before (floor 96). The baseline counted 7 survivors as timeouts because the machine was loaded. Counted the same way, both runs score 96.33 on the 4169 mutants they share. The maintainer accepted this score rule; it is recorded in the contract.
  • This PR touches the shared mutation harness, so CI runs every mutation scope.
  • git-repository-locator-adapter.ts:78 is a flaky mutant: Timeout, Survived or Killed depending on the run. It is reported, not fixed.
  • The merge commit 7639a681 has git's default message. The squash replaces it.

🔗 Linked issue

Closes #1036

✅ I certify

  • I DO CERTIFY I READ EACH LINE OF THE PULL REQUEST BECAUSE I AM A SOFTWARE ENGINEER, NOT A AI PUPPY.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp

blafourcade and others added 16 commits October 16, 2026 10:07
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Stryker stops a mutant at its first failing test, but Vitest ran the
integration project before the unit one and the longest file first, so
most mutants paid for slow integration files before the unit test that
kills them in milliseconds. The mutation config now orders files unit
before integration, then by size, then by path. Size is the same on every
machine and sandbox; the ordinary suite keeps Vitest's order.

Refs #1036

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Six telemetry test files called product code while Vitest collected them:
parsers at file level, repositoryIdOf, daysOf in an it.each table. Stryker
counts code reached outside a test as static and reruns every related test
for each of its mutants, so 172 mutants in task-declaration, stored-usage,
repository-identity and text-order ran up to 881 tests each. Each value is
now computed inside the tests that read it: static mutants 330 to 158,
test runs 54490 to 39327, no mutant status changed.

Refs #1036

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
tests/e2e/telemetry-report.e2e.test.ts ran 22 tests in one file for 33 s,
the critical path of the whole suite, since Vitest runs files in
parallel, not describes. Its three describes are now three files, report,
report narrowed and identity, starting each test from the same sandbox in
telemetry-report-sandbox.ts. Same tests, same assertions.

Refs #1036

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Both golden suites captured every cell twice to prove determinism, then a
third time to compare with the stored snapshot. The baseline test now
compares the first of the two captures, which are still taken one after
the other in two sandboxes, so a difference that depends on the time
still shows. A third of the capture work goes; no comparison is dropped.

Refs #1036

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…t static

Refs #1036

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Under load, a survivor that runs all its covering tests (up to 37 s net)
crossed 1.5 x net + 20 s and was scored as a timeout, which counts as
detected: 12 survivors read Timeout in one sharded telemetry run. Since
unit tests run first, the only timeouts left are 26 real hangs, so a
factor of 3 costs about 360 worker-seconds and no survivor reads Timeout
in the run that measured it (6.7 min).

Refs #1036

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Replacing sort with return files left every sequencer test green, and the
default statSync size was never exercised. The new test sorts two temp
files of known sizes through CheapestKillerFirst and fails on that change.
sort now takes any spec that has a project name and a module id, so the
test passes plain values instead of widening a type.

Refs #1036

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Both golden suites carried the same block that takes two captures one
after the other and hands the first to the baseline test. It lives once
in tests/golden/two-captures.ts, given the capture to run.

Refs #1036

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
The baseline read 96.5 because load scored some survivors as timeouts;
counted alike on both sides, both score 96.33, and no baseline kill
survives. pnpm test gains only in the e2e subset and CPU, not above noise
in wall time. The evidence names the seventh load-hidden survivor, keeps
the flaky locator mutant apart, and gives framework's CI score before.

Refs #1036

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
@blafourcade
blafourcade marked this pull request as ready for review October 16, 2026 14:35
@blafourcade
blafourcade requested a review from a team as a code owner October 16, 2026 14:35
@blafourcade
blafourcade merged commit 1c5ae72 into next Oct 16, 2026
61 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant