Conversation
…umber I published "~118s to ~46s" in the changelog, the PR, and on pm-plugin.dev. It is reproducible, but only on an otherwise-idle 16-core workstation, and presenting it without conditions overstates what most readers will see. Measured across three environments: idle 16-core workstation 118s -> 46s 2.6x same machine, load ~12 146s -> 122s 1.2x CI (whole workflow) ~55s -> ~42s ~1.3x Parallel execution is precisely what degrades when cores are already contended, so the benefit largely evaporates on a busy laptop or a few-core runner -- which is what CI is. A contributor reading "2.6x" and seeing 1.2x would reasonably conclude the claim was inflated. This also violated the project's own performance-reporting rule: report wall-clock WITH its conditions, never a bare headline number. Corrected to give the range and say plainly that it depends on free cores. Method note on how I got this wrong: I measured the original 46s three times, but while other work was running I later saw the same suite take 685s and then 122s. A benchmark taken while the machine is busy is not a benchmark, and I never re-measured after the test count grew 255 -> 282. Claude-Session: https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Correction to a performance claim I published in 0.24.0's changelog, PR #72, and on pm-plugin.dev.
"~118s to ~46s" is reproducible, but only on an otherwise-idle 16-core workstation. Presenting it without conditions overstates what most readers will see.
Measured across three environments on this project:
Parallel execution is exactly what degrades when cores are already contended, so the benefit largely evaporates on a busy laptop or a few-core runner — which is what CI is. A contributor reading "2.6×" and measuring 1.2× would reasonably conclude the claim was inflated.
This also broke the project's own performance-reporting rule: report wall-clock with its conditions, never a bare headline number.
How I got it wrong: I measured the original 46s three times and it was real — but later saw the same suite take 685s and then 122s while other work was running on the machine. A benchmark taken on a busy machine is not a benchmark, and I never re-measured after the test count grew 255 → 282 (several additions perform extra commits per test, and this suite is subprocess-bound).
The structural claims are unchanged and still verifiable: 11 files, 676 engine spawns at ~73ms each,
node --testparallelizes across files only, split verified by test-name-set equality.The matching pm-plugin.dev changelog entry is updated in the same cycle.
https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC