Repository navigation
test(cli): run the telemetry mutation 7x faster without weakening it - #1043
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Stryker stops a mutant at its first failing test, but Vitest ran the integration project before the unit one and the longest file first, so most mutants paid for slow integration files before the unit test that kills them in milliseconds. The mutation config now orders files unit before integration, then by size, then by path. Size is the same on every machine and sandbox; the ordinary suite keeps Vitest's order. Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Six telemetry test files called product code while Vitest collected them: parsers at file level, repositoryIdOf, daysOf in an it.each table. Stryker counts code reached outside a test as static and reruns every related test for each of its mutants, so 172 mutants in task-declaration, stored-usage, repository-identity and text-order ran up to 881 tests each. Each value is now computed inside the tests that read it: static mutants 330 to 158, test runs 54490 to 39327, no mutant status changed. Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
tests/e2e/telemetry-report.e2e.test.ts ran 22 tests in one file for 33 s, the critical path of the whole suite, since Vitest runs files in parallel, not describes. Its three describes are now three files, report, report narrowed and identity, starting each test from the same sandbox in telemetry-report-sandbox.ts. Same tests, same assertions. Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Both golden suites captured every cell twice to prove determinism, then a third time to compare with the stored snapshot. The baseline test now compares the first of the two captures, which are still taken one after the other in two sandboxes, so a difference that depends on the time still shows. A third of the capture work goes; no comparison is dropped. Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
…t static Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Under load, a survivor that runs all its covering tests (up to 37 s net) crossed 1.5 x net + 20 s and was scored as a timeout, which counts as detected: 12 survivors read Timeout in one sharded telemetry run. Since unit tests run first, the only timeouts left are 26 real hangs, so a factor of 3 costs about 360 worker-seconds and no survivor reads Timeout in the run that measured it (6.7 min). Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Replacing sort with return files left every sequencer test green, and the default statSync size was never exercised. The new test sorts two temp files of known sizes through CheapestKillerFirst and fails on that change. sort now takes any spec that has a project name and a module id, so the test passes plain values instead of widening a type. Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Both golden suites carried the same block that takes two captures one after the other and hands the first to the baseline test. It lives once in tests/golden/two-captures.ts, given the capture to run. Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
The baseline read 96.5 because load scored some survivors as timeouts; counted alike on both sides, both score 96.33, and no baseline kill survives. pnpm test gains only in the e2e subset and CPU, not above noise in wall time. The evidence names the seventh load-hidden survivor, keeps the flaky locator mutant apart, and gives framework's CI score before. Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Refs #1036 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What & why
The telemetry mutation run took 45 to 62 min locally. Each mutant ran the slow integration test files before the unit test that would have killed it in milliseconds. After this change the run takes 6.7 min locally (CI shards: 3.6, 4.9 and 6.9 min). No test was removed or weakened.
🛠️ How it works
vitest.mutation.config.ts. Stryker stops a mutant at its first failing test, so the cheapest test that kills it now runs first. The sequencer changes the order of tests, never which tests run.pnpm testkeeps its order. Measured: 45.5 to 8.9 min.it.eachtitles are unchanged. Test runs per full telemetry run drop from 94k to 39k.timeoutFactor3 (stryker.conf.json:9). 165 of the old timeouts were slow kills, not hangs, and are now counted as killed.timeoutMSstays at 20000, the value test(cli): count a mutant as killed only when a test failed #844 set.pnpm test. The 33 s telemetry report e2e file is split into 3 files, with 22 tests before and after. The golden tests capture each cell twice instead of three times; the determinism check (capture 2 against capture 1) is kept. The shared code lives in two-captures.ts. The overall wall-time gain is within noise; the e2e subset went from 17.1 s to 12.1 s.🧪 How to verify
cd cli && pnpm test: 6641 passed, 1 skipped.pnpm test:mutation:telemetry, compared mutant by mutant with the baseline:ledger-lock.ts.frameworkis at 93.0 against 93.1 in CI before this change, with no regression per mutant.sortreturns the files unsorted, or when a file size reads 0.git-repository-locator-adapter.ts:78is a flaky mutant: Timeout, Survived or Killed depending on the run. It is reported, not fixed.7639a681has git's default message. The squash replaces it.🔗 Linked issue
Closes #1036
✅ I certify
🤖 Generated with Claude Code
https://claude.ai/code/session_01VWNxk63AGKkqE8HRqHLjGp