From fb45ade48994279cddb045fd55870e67aa9f661a Mon Sep 17 00:00:00 2001 From: elrrrrrrr Date: Thu, 7 May 2026 14:29:22 +0800 Subject: [PATCH] chore(bench): add hyperfine --warmup 1 to phase-bench MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bench iterates PMs in fixed order (`utoo, utoo-next, utoo-npm, bun` per `PM_LIST`) within each phase. The first PM pays the cold-network tax — DNS resolver miss, TLS session ticket cold, npm CDN edge POP unpopulated — and `--runs 3` averages that cold iteration into the wall mean. PMs that run later inherit the warm state for free. A standalone smoke test confirmed the magnitude: the same utoo binary, run twice back-to-back through `pm-bench-pcap.sh`, showed ~3s wall and 270 MB pcap-size delta on `p3_install` between the first and second run, attributed to CDN/DNS/TLS warm-up. Folded through `--runs 3` averaging that's roughly +1s on the first PM's mean — the same magnitude as the per-PR p0 deltas observed on #2903 / #2904 / #2905, which makes those mean-deltas indistinguishable from ordering bias. `--warmup 1` makes hyperfine run one untimed iteration through `--prepare` before the timed runs start, so each PM enters its measurement window with the network path warmed independently. The cross-PM ordering bias in the LATER measurements collapses because every PM starts from a primed-network state (the first PM's warmup primes the runner; subsequent PMs' warmups cost nothing extra but hold the same shape). This costs one extra iteration per PM × phase (≈ 33% wall increase on the bench job, well within tolerance), and is independent of the script's PM-ordering question — that one we'll address separately with round-robin if `--warmup` alone doesn't close the gap. Co-Authored-By: Claude Opus 4.7 (1M context) --- bench/pm-bench-phases.sh | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/bench/pm-bench-phases.sh b/bench/pm-bench-phases.sh index 7d3d37c496..226ffb751d 100755 --- a/bench/pm-bench-phases.sh +++ b/bench/pm-bench-phases.sh @@ -347,8 +347,17 @@ run_phase() { echo -e " ${CYAN}$pm${NC} · $phase" export PROJECT_DIR UTOO_CACHE BUN_CACHE + # `--warmup 1`: untimed iteration before the timed runs so each PM enters + # its measurement window with DNS resolver cached, TLS session ticket + # primed, and the CDN edge POP populated. Without this, the first PM in + # the per-phase loop pays the cold-network tax (~+1s on p0) which the + # 3-run `--runs` mean cannot fully amortize, biasing the first PM + # systematically slow vs PMs that follow it. `--prepare` resets the + # filesystem state (lockfile / cache / node_modules) on every iteration + # — warmup included — so warmup leaves no on-disk artifact behind. if ! hyperfine \ --runs "$RUNS" \ + --warmup 1 \ --prepare "bash $prep_script" \ --export-json "$json" \ --show-output \