From b19fa76ee11f53512d1813cbbd53998e19b64a3e Mon Sep 17 00:00:00 2001 From: Rob Sherman Date: Thu, 30 Jul 2026 13:39:33 -0700 Subject: [PATCH] =?UTF-8?q?docs(pm):=20qualify=20the=20test-suite=20speedu?= =?UTF-8?q?p=20=E2=80=94=202.6x=20was=20an=20idle-machine=20number?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I published "~118s to ~46s" in the changelog, the PR, and on pm-plugin.dev. It is reproducible, but only on an otherwise-idle 16-core workstation, and presenting it without conditions overstates what most readers will see. Measured across three environments: idle 16-core workstation 118s -> 46s 2.6x same machine, load ~12 146s -> 122s 1.2x CI (whole workflow) ~55s -> ~42s ~1.3x Parallel execution is precisely what degrades when cores are already contended, so the benefit largely evaporates on a busy laptop or a few-core runner -- which is what CI is. A contributor reading "2.6x" and seeing 1.2x would reasonably conclude the claim was inflated. This also violated the project's own performance-reporting rule: report wall-clock WITH its conditions, never a bare headline number. Corrected to give the range and say plainly that it depends on free cores. Method note on how I got this wrong: I measured the original 46s three times, but while other work was running I later saw the same suite take 685s and then 122s. A benchmark taken while the machine is busy is not a benchmark, and I never re-measured after the test count grew 255 -> 282. Claude-Session: https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC --- CHANGELOG.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a1378cc..60ded66 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,12 +22,19 @@ This project adheres to [Semantic Versioning](https://semver.org/). ### Changed - **The test suite is now `scripts/test/*.test.mjs` (11 files) instead of one - `scripts/conductor.test.mjs`, cutting a full run from ~118s to ~46s.** Contributor-facing: the + `scripts/conductor.test.mjs`, so it runs in parallel.** Contributor-facing: the command is `node --test scripts/test/*.test.mjs`. Measured rather than guessed — one engine spawn costs ~73ms and the suite makes 676 of them, so ~49s was pure `node` startup, serialized in a single file on a 16-core machine; `node --test` parallelizes across *files* only. The split is verbatim, verified by test-name-set equality (255 → 255). + **How much faster depends entirely on free cores, so take a single headline number with + salt.** Measured on this project: an otherwise-idle 16-core workstation went ~118s → ~46s + (2.6×), but the *same* machine under load (~12) went ~146s → ~122s — only 1.2×, because + parallel execution is exactly what suffers when cores are already contended. On CI, whole- + workflow time went from ~52–58s to ~38–48s (~1.3×). If you are on a busy laptop or a + few-core runner, expect the low end. + Note `node --test scripts/test` (a **directory**) does not work — Node treats the argument as a module to execute and dies with `MODULE_NOT_FOUND` while reporting "1 test". It exits non-zero so it cannot pass silently, but use the glob.