ci(pm): pm-e2e-bench unified workflow + phase-isolated bench infrastructure by elrrrrrrr · Pull Request #2824 · utooland/utoo

elrrrrrrr · 2026-04-25T14:11:33Z

Summary

First of 4 split PRs from #2818. Pure infrastructure — no business code changes — establishes the unified PM CI workflow so subsequent perf PRs can cite empirical phase deltas.

Three layers:

pm-e2e-bench.yml — single workflow that consolidates the previous pm-e2e.yml + pm-bench.yml + pm-bench-phases.yml. Each target builds once (build-{linux,mac-arm64,mac-x64,windows}) and downstream jobs (e2e-*, bench-phases-*, pm-bench-all-*, pm-bench-pcap-linux) reuse the artifact via needs: — DAG visible in the run's Summary view.
bench/pm-bench-phases.sh — splits a full install into p0_full_cold / p1_resolve / p3_cold_install / p4_warm_link, captures wall + RSS + ctx + IO + network per phase under hyperfine. Runs three PMs side-by-side: utoo (this branch) · utoo-npm (latest published) · bun (latest).
Sticky PR comment — bench-phases-{linux,mac} jobs render the per-phase summary as markdown tables (one per phase × per registry × per metric group) and upsert via marocchino/sticky-pull-request-comment@v2. Two comments per PR (one per platform), idempotent across commits.

Triggers

Job group	Trigger	Notes
`build-*`	union of all downstream gates	runs whenever any consumer needs it
`e2e-*`	push/PR with `(pm)` strict in commit msg / title	always-on smoke test
`bench-phases-{linux,mac}`	PR labeled `benchmark` OR `workflow_dispatch target=pm-bench-phases`	utoo / utoo-npm / bun, ~25 min/platform
`pm-bench-all-{linux,mac}`	`workflow_dispatch target=pm-bench-all` only	full shootout: utoo + pnpm + yarn + bun + aube — too slow for label trigger
`pm-bench-pcap-linux`	`workflow_dispatch target=pm-bench-pcap` only	tcpdump capture for resolve phase

Architecture

Build job gates are the union of every downstream consumer's gate
Build artifacts (utoo-{linux-x64,mac-arm64,mac-x64,windows-x64}) cached for 1 day
utoopm-e2e-success aggregator always runs (if: always()) — preserves the old branch-protection check name
utoo-npm baseline installed locally (npm install utoo into temp dir; uses node_modules/utoo/bin/utoo directly) so it doesn't shadow this branch's utoo on PATH
Bench scripts live under bench/ (not e2e/) since they're perf benchmarks, not tests

Test plan

All 4 split PRs branched off this one (perf(pm): chunked parallel writes within tarball extraction + Tar Slip guard #2825 install-path, perf(pm): manifest cache & resolver alloc cleanup #2826 manifest-cache, perf(pm): preload worker-pool replaces FuturesUnordered #2827 worker-pool)
CI green on consolidated pm-e2e-bench run
Markdown sticky comments render correctly with utoo + utoo-npm + bun across all 4 phases × 2 registries (npmjs + npmmirror)
Comment idempotent — same pm-bench-phases-{linux,mac} headers update in place across commits
Bench-phases requires benchmark label; pm-bench-all is dispatch-only — no auto-trigger on every (pm-scoped PR

Context

Full exploration journey + failed-experiments catalog in #2818.

🤖 Generated with Claude Code

… + PR auto-comment Adds a comprehensive benchmark infrastructure to support data-driven review of subsequent perf PRs. Infrastructure layers: 1. Phase-isolated bench (`pm-bench-phases.yml` + `e2e/pm-bench-phases.sh`) - Splits a full install into `p0_full_cold`, `p1_resolve`, `p3_cold_install`, `p4_warm_link` so each phase's wall + RSS + ctx-switches + IO + network metrics is captured separately - Runs both `utoo` and `bun` under hyperfine for direct comparison - Per-iteration prepare via generated bash scripts (lockfile + cache + node_modules wiped or selectively kept per phase) - Triggers on `bench-phases` PR label so it stays opt-in and runs before the default-branch merge gate - Captures `/usr/bin/time -v` per phase for memory+ctx visibility 2. Standalone manifest-bench (`crates/manifest-bench`) - New non-published crate: a single binary that fires parallel HTTP GETs for every package name in a `package-lock.json`, with no parsing / no resolver state / no caching. Same `reqwest + rustls + aws-lc-rs + tokio` stack as ruborist. - Exposes the same diag fields (wall, busy, sum, avg_conc, p50, p95, p99, max) so its output is directly comparable to ruborist's `Preload HTTP diag` log line - CLI flags sweep concurrency / H1 vs H2 / single-version endpoint / User-Agent — useful for empirically establishing the HTTP stack ceiling under the same Cloudflare conditions as the real bench - Wired into the workflow as a sweep step after the phase bench 3. PCAP capture (`e2e/pm-bench-pcap.sh`) - Single-iteration tcpdump of the resolve phase per PM, scoped to the registry SNI, uploaded as artifact for offline analysis 4. PR auto-comment - Workflow extracts the `## p<N>_<name>` summary tables from the bench output and upserts them into a single sticky PR comment keyed by an HTML marker (re-runs replace, not append) - Uses `actions/github-script@v7`; only fires on `bench-phases` -labeled PRs so workflow_dispatch + push runs do not spam - Job declares `permissions: pull-requests: write` to allow comment writes on a normally read-only fork-PR token Why this lands first: subsequent perf PRs (preload worker-pool, manifest cache cleanup, install path tweaks) need this in place so the PR description can cite real before/after numbers and the PR comment can show the per-commit delta. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

gemini-code-assist

Code Review

This pull request introduces a standalone HTTP-only manifest fetch benchmark tool called manifest-bench, along with new E2E scripts for phase-isolated benchmarking and network trace capture. These additions are designed to isolate network behavior from the resolver pipeline for more accurate performance measurement. The review feedback highlights several improvement opportunities, including the need for recursive traversal when parsing npm v2 lockfiles, optimizing string splitting operations for efficiency, adding a request timeout to the HTTP client for robustness, and refining filename parsing logic in the summary script to avoid potential collisions with project names.

CI runs tombi 0.7.32 which aligns column widths across the file (local 0.8.0 normalises per-table and disagreed). Apply the formatting CI expects. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The headline architectural change of #2818. ruborist's preload phase shifts from a single-task `FuturesUnordered` cooperative poller to N long-lived `tokio::spawn` workers (or `wasm_bindgen_futures::spawn_local` on wasm32 where Send isn't satisfied). Stacks on top of #2826. ## Why Old design: main task owned `FuturesUnordered`, polled all preload futures cooperatively, and ran every per-future continuation (post-await body, completion handler, dispatch refill) on the same single task. The deeper await chain inside `resolve_package` (cache check + `OnceMap::get_or_init` + `RetryIf` + `request.send` + `bytes` + parse spawn_blocking) made each future yield 5+ times, and every yield round-tripped through main — saturating it. CI ant-design preload sustained avg_conc=55-61 even after Mutex / allocator hot-path eliminations, while the standalone manifest-bench (same reqwest stack, no resolver — see #2824) hit 92 at the same cap. ## How N long-lived `tokio::spawn` workers pulling from a shared lock-free `SegQueue<Dep>` with `DashSet` dedup. Each worker owns an `Arc<R>` clone and runs `resolve_package` on tokio's global executor — futures progress fully independently, no cooperative poll bottleneck. Main task only drains an `mpsc::unbounded_channel` of completions to fire receiver events + on_manifest callback. Termination: workers track `dispatched` / `completed: AtomicUsize` and park on a shared `Notify` when the queue is empty. When the last completion makes `completed == dispatched` and the queue is empty, the finishing worker raises a `shutdown` flag and wakes others; all workers drop their result_tx clones, the channel closes, and the main `recv().await` loop exits. ## Trait surface change - `MockRegistryClient` + `MockPackage` now `derive(Clone)` so tests can wrap the mock in `Arc` for the new signature - `preload_manifests` takes `registry: Arc<R>` (was `&R`); call site in `run_preload_phase` clones the borrowed registry into a fresh `Arc`. Bound at every public surface up the chain bumped to `R: RegistryClient + Clone + MaybeSend + MaybeSync + 'static`, `R::Error: MaybeSend`. The `MaybeSend` / `MaybeSync` shims (added in #2826) keep the trait surface wasm-compatible. ## Companion changes folded in - **Inline simd_json parse** — drop `tokio::task::spawn_blocking` in `service/manifest.rs`. Worker-pool surfaced parse blocking- pool queue saturation: `queue p95=200ms sum=70-89s` over 2730 manifests on cap=4 CI runners. Inline parse on the worker thread eliminates dispatch + queue overhead; 1-5ms CPU per manifest is acceptable on async worker. - **Workspace package.json parallel reads** — `find_workspaces_from_pkg` switched from sequential `for path in matched_paths { read }` loop to `FuturesUnordered` fan-out. ant-design has ~200 workspace packages; saved ~150ms. - **Setup phase + lockfile-write timing logs** — round out the per-phase wall account for the bench-comment infrastructure. - **Manifests concurrency cap 64 → 128** — worker-pool delivered the parallelism that justifies the cap raise. CI ant-design avg_conc 84 at cap=128 (up from 55 under the old architecture); preload wall 3.10s → 2.15s. ## Tests `#[tokio::test(flavor = "multi_thread", worker_threads = 2)]` since worker-pool needs a spawn-able runtime; ruborist's dev-dependencies on `tokio` add the `rt-multi-thread` feature. 164 ruborist + 10 doctests + 248/249 utoo-pm pass (1 pre-existing flake on `test_update_package_binary_fsevents`, runs green alone). ## Wasm CI cfg-gates `tokio::spawn` to `wasm_bindgen_futures::spawn_local` on wasm32 since wasm-bindgen's `JsFuture` is `!Send`. Workers still run independently — single-threaded under wasm but the queue + Notify + mpsc termination story is unchanged. `cargo check -p utoo-wasm --target wasm32-unknown-unknown` clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Mirror `pack-perf.yml` pattern: - Trigger on `pull_request` to main/next AND `push` to main/next when PR title / commit message contains `(pm` (matches `perf(pm)`, `refactor(pm)`, `ci(pm-bench-phases)`, etc — same scope convention as pack uses for `(pack`) - Replace the ad-hoc `actions/github-script` upsert with battle-tested `marocchino/sticky-pull-request-comment@v2` (same action pack-perf uses) - Build the comment body in a separate shell step that strips ANSI escapes and wraps each `## p<N>_<name>` summary section as a fenced code block - Auto-detect PR number by branch name → SHA fallback (handles push events where the PR isn't on the event payload) - Drop the prior `pull_request: types: [labeled]` trigger — labels are no longer needed; scope-in-title is the convention This makes the bench results auto-post on every PR push for any pm- scoped change. workflow_dispatch still works for manual runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The headline architectural change of #2818 — preload phase shifts from a single-task `FuturesUnordered` cooperative poller to N long-lived `tokio::spawn` workers (or `wasm_bindgen_futures::spawn_local` on wasm32 where Send isn't satisfied). Stacks on top of #2826. ## Why Old design: main task owned `FuturesUnordered`, polled all preload futures cooperatively, and ran every per-future continuation (post-await body, completion handler, dispatch refill) on the same single task. The deeper await chain inside `resolve_package` (cache check + `OnceMap::get_or_init` + `RetryIf` + `request.send` + `bytes` + parse spawn_blocking) made each future yield 5+ times, and every yield round-tripped through main — saturating it. CI ant-design preload sustained avg_conc=55-61 even after Mutex / allocator hot-path eliminations, while the standalone manifest-bench (#2824) hit 92 on the same reqwest stack. ## How N long-lived `tokio::spawn` workers pulling from a shared lock-free `SegQueue<Dep>` with `DashSet` dedup. Each worker owns an `Arc<R>` clone and runs `resolve_package` on tokio's global executor — futures progress fully independently, no cooperative poll bottleneck. Main task only drains an `mpsc::unbounded_channel` of completions to fire receiver events + on_manifest callback. Termination: workers track `dispatched` / `completed: AtomicUsize` and park on a shared `Notify` when the queue is empty. When the last completion makes `completed == dispatched` and the queue is empty, the finishing worker raises a `shutdown` flag and wakes others; all workers drop their result_tx clones, the channel closes, and the main `recv().await` loop exits. ## Trait surface change - `MockRegistryClient` + `MockPackage` `derive(Clone)` so tests can wrap the mock in `Arc` for the new signature - `preload_manifests` takes `registry: Arc<R>` (was `&R`); call site in `run_preload_phase` clones the borrowed registry into a fresh `Arc`. Bound at every public surface up the chain bumped to `R: RegistryClient + Clone + MaybeSend + MaybeSync + 'static`, `R::Error: MaybeSend`. The `MaybeSend` / `MaybeSync` shims (added in #2826) keep the trait surface wasm-compatible. ## Companion changes folded in - **Inline simd_json parse** — drop `tokio::task::spawn_blocking` in `service/manifest.rs`. Worker-pool surfaced parse blocking- pool queue saturation: `queue p95=200ms sum=70-89s` over 2730 manifests on cap=4 CI runners. Inline parse on the worker thread eliminates dispatch + queue overhead. - **Workspace package.json parallel reads** — switch the per-pattern `for path in matched_paths` serial loop to `FuturesUnordered` fan-out. ant-design has ~200 workspace packages; saved ~150ms. - **Setup phase + lockfile-write timing logs** — round out the per-phase wall account for the bench-comment infrastructure. - **Manifests concurrency cap 64 → 128** — worker-pool delivers the parallelism that justifies the cap raise. CI ant-design avg_conc 84 at cap=128 (up from 55 under the old architecture); preload wall 3.10s → 2.15s. ## Wasm CI cfg-gates `tokio::spawn` to `wasm_bindgen_futures::spawn_local` on wasm32 since wasm-bindgen's `JsFuture` is `!Send`. Workers still run independently — single-threaded under wasm but the queue + Notify + mpsc termination story is unchanged. `cargo check -p utoo-wasm --target wasm32-unknown-unknown` clean. Tests: 164 ruborist + 10 doctests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…egacy bench script Three changes per review feedback: 1. **Revert `e2e/pm-bench.sh` changes** — those (`--show-output`, JSON parse fault tolerance) were part of the legacy non-phase bench. Bench-phases lives in its own script (`pm-bench-phases.sh`); the legacy script doesn't need our touch and was bloating the diff. 2. **Add npmmirror run after npmjs** with full cache wipe between. Both `pm-bench-phases.sh` and `manifest-bench` now run twice per bench job. The wipe step removes: /tmp/pm-bench /tmp/pm-bench-results /tmp/utoo-bench-cache /tmp/bun-bench-cache ~/.bun/install/cache ~/.cache/nm so the second registry starts truly cold (lockfile, project clone, manifest cache, tarball cache — all wiped). The PR comment renders both registries' phase tables under labeled sections. 3. **Cross-platform matrix** — `build` and `bench` jobs both run on `[ubuntu-latest, macos-latest]`. Each platform posts its own sticky comment (header `pm-bench-phases-{linux,mac}`) so they don't race. `gnu-time` and `hyperfine` installed via brew on mac. PCAP step is linux-only (raw sockets + tcpdump availability). Bonus: extracted manifest-bench sweep loop to `.github/scripts/run-manifest-bench.sh` so the workflow yaml stays readable and the sweep can be reused by ad-hoc scripts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

pm-e2e and pm-bench-phases both build `utoo` from the same source on every (pm)-scoped PR — wasteful (~3 min per build). Bench now downloads pm-e2e's `utoo-linux-x64` artifact from the matching commit's pm-e2e run. Linux flow: bench job → `gh run list` finds pm-e2e run for this commit's SHA → download-artifact pulls `utoo-linux-x64` from there → falls back to local cargo build only if pm-e2e didn't run for this commit (e.g. workflow_dispatch path) manifest-bench: still built locally (pm-e2e doesn't build it). Mac flow unchanged: pm-e2e builds mac in its e2e matrix but doesn't upload an artifact, so bench still builds utoo + manifest-bench locally on macos-latest. Permissions: - `actions: read` added to bench job for `gh run list` + cross-run artifact download. Net CI savings on (pm)-scoped PRs: linux utoo cargo build (~3 min) runs once across both workflows instead of twice. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The bench job's `gh run list` would find a pm-e2e run for the commit even when its build job had been skipped (pm-e2e gates on `(pm)` with closing paren — `ci(pm-bench-phases): ...` doesn't match), causing `download-artifact` to fail. Check that the run actually uploaded `utoo-linux-x64` before deciding to reuse it, otherwise fall back to the local build path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…e workflow Single workflow that builds utoo once per target and fans out to every consumer (e2e, bench-phases, pm-bench, pm-bench-pcap), so each target's `cargo build --release` runs at most once per commit instead of being duplicated across the three previous workflows. Build jobs (one per target, gated on the union of consumer gates): - build-linux x86_64-unknown-linux-gnu - build-mac-arm64 aarch64-apple-darwin - build-mac-x64 x86_64-apple-darwin - build-windows x86_64-pc-windows-msvc Downstream jobs (pure download-and-run, no cargo build): - e2e-{linux,mac-arm64,mac-x64,windows} auto on `(pm)` scope - bench-phases-{linux,mac} auto on `(pm` scope - pm-bench-{linux,mac} manual workflow_dispatch - pm-bench-pcap-linux manual workflow_dispatch Job names like `utoopm-e2e-success` and the per-target e2e check names are preserved so existing branch-protection rules continue to work. Also delete `crates/manifest-bench` (debug-only HTTP fetch tool) and its `.github/scripts/run-manifest-bench.sh` helper — no longer used. The pcap step previously ran inline on every bench-phases run; it now lives in its own workflow_dispatch target since pcap is only useful during manual perf analysis. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…own tables Three changes to make the PR comment more useful: 1. Add `utoo-npm` (latest published utoo from npm) as a third PM column alongside `utoo` (this branch) and `bun`. Lets reviewers see whether a perf change in this PR is a regression vs the published baseline, not just a delta vs bun. Implementation: pm-bench-phases.sh now recognises `utoo-npm` as a distinct PM with its own bench-scoped cache (UTOO_NPM_CACHE) and binary path (UTOO_NPM_BIN). The bench job installs latest utoo from npm into a temp dir BEFORE running the bench so it doesn't shadow $GITHUB_WORKSPACE/dist/utoo on PATH; the script invokes it via the absolute UTOO_NPM_BIN path. If UTOO_NPM_BIN is unset (local runs), `utoo-npm` is silently filtered out of PM_LIST. 2. Render the sticky comment as markdown tables instead of fenced code blocks. The fixed-width tables emitted by the bench script's summary are parsed by awk and converted to one markdown table per phase, with PMs as rows and metrics as columns. Renamed `net RX`/`net TX` columns to `netRX`/`netTX` so the header tokenises cleanly. 3. (No code change) Idempotency: the comment is already idempotent — marocchino/sticky-pull-request-comment uses `header: pm-bench-phases-{linux,mac}` so each commit edits the same two comments per platform rather than appending new ones. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

npm install for the utoo wrapper package didn't populate node_modules/.bin/utoo on the runner (npm's local-install bin-link behavior depends on version/flags). The wrapper's package.json declares `bin: { utoo: 'bin/utoo' }`, so point at the actual file under the package directory instead of relying on the symlink. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Earlier replace_all only matched the linux block (the comments diverged between linux and mac). Mac was still pointing at the non-existent node_modules/.bin/utoo symlink. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The pm-bench, pm-bench-phases and pm-bench-pcap scripts are perf benchmarks, not e2e tests. Group them under bench/ so the directory intent is obvious. Update pm-build.yml's path references. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Restructure: pm-build.yml is now build-only — builds utoo for the four targets and uploads each as an artifact. Each downstream concern lives in its own workflow file: - pm-e2e.yml workflow_run after pm-build, runs e2e per target - pm-bench-phases.yml workflow_run after pm-build, posts sticky PR comment - pm-bench.yml workflow_dispatch + label trigger, builds inline - pm-bench-pcap.yml workflow_dispatch only, builds inline pm-e2e and pm-bench-phases reuse pm-build's artifact via cross-workflow download (`actions/download-artifact@v4` with `run-id` from the workflow_run event). Each per-target/per-platform job first checks `gh api .../actions/runs/.../artifacts` for the matching artifact — if pm-build skipped that target (e.g. mac-x64 + windows are gated on `(pm)` strict only), the downstream job no-ops cleanly. The `utoopm-e2e-success` aggregator + per-target check names are preserved verbatim, so any branch protection rules continue to match. Note: workflow_run-triggered workflows use the file on the default branch. After this branch merges to `next`, pm-e2e/pm-bench-phases fire on every pm-build completion. Until then, the OLD pm-e2e.yml on `next` remains the canonical e2e check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

This reverts commit 70f8df5.

The workflow does e2e + bench (in addition to building utoo) — the old `pm-build` name only described the first stage. `pm-e2e-bench` explicitly lists what the consolidated workflow covers and reads better alongside `pm-ci` (unit tests) and `pm-release` (publishing). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Phase-isolated bench takes ~25 min per platform — too expensive to fire on every (pm-scoped PR. Switch the gate to require an explicit `bench` label (the PR author / reviewer adds it when they actually want bench numbers in the comment). Build jobs' gate union updated to include the `bench` and `benchmark` labels so the build artifact still gets produced when either trigger fires. Push to next/e2e-pm with `(pm)` in commit msg keeps building + running e2e (no bench), so trunk doesn't lose its smoke test. Also tightened the `(pm` loose match to `(pm)` strict everywhere — `(pm-foo)` scopes shouldn't auto-trigger e2e. They still trigger manual paths (label / dispatch) on demand. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ispatch-only - bench-phases now fires on the `benchmark` label (was `bench`) - pm-bench (full PM shootout — utoo + bun + pnpm + yarn + aube) is workflow_dispatch only. Comparing every PM is slow enough that even a label trigger isn't worth the runner time; user must explicitly dispatch when they want it. - build gates trimmed to drop the unused `bench` label entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Job names `pm-bench-{linux,mac}` → `pm-bench-all-{linux,mac}` and the workflow_dispatch target value `pm-bench` → `pm-bench-all`. Makes it explicit that this is the all-PMs comparison (utoo + pnpm + yarn + bun + aube), not the lighter phase-isolated bench. The shell script filename (bench/pm-bench.sh) is unchanged — it's a stable on-disk asset. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Pull repeated step sequences out of pm-e2e-bench.yml into composite actions under .github/actions/. The workflow shrinks from ~947 lines to ~792 (~16%) and each shared concern now lives in one place instead of being copy-pasted across linux + mac variants. New actions: - setup-utoo-bin download artifact + chmod + (optional ut symlink) + PATH - install-utoo-npm-baseline npm install utoo + set UTOO_NPM_BIN - install-bench-tools per-platform hyperfine + GNU time install - post-bench-phases-comment render markdown + find PR + post sticky comment Behaviour unchanged — only structural refactor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

github-actions · 2026-04-26T16:17:00Z

📊 pm-bench-phases · `16f8203` · linux (`ubuntu-latest`)

Workflow run — ant-design

PMs: utoo (this branch) · utoo-npm (latest published) · bun (latest)

npmjs.org

p0_full_cold

PM	wall	±σ	user	sys	RSS	pgMinor
bun	8.92s	0.13s	10.07s	9.92s	700M	320.3K
utoo-npm	10.06s	0.38s	11.30s	12.99s	1.39G	157.7K
utoo	10.86s	0.73s	11.67s	13.23s	1.21G	148.1K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	15.0K	17.2K	1.17G	6M	1.85G	1.74G	1M
utoo-npm	160.8K	147.8K	1.15G	4M	1.70G	1.69G	2M
utoo	172.3K	118.6K	1.15G	4M	1.70G	1.69G	2M

p1_resolve

PM	wall	±σ	user	sys	RSS	pgMinor
bun	4.42s	2.07s	3.94s	1.08s	498M	187.5K
utoo-npm	5.57s	0.60s	5.93s	1.11s	432M	76.3K
utoo	5.21s	0.09s	5.94s	1.14s	431M	74.0K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	9.2K	4.1K	201M	3M	104M	-	1M
utoo-npm	65.2K	2.9K	202M	2M	9M	5M	2M
utoo	65.1K	2.8K	201M	2M	9M	5M	2M

p3_cold_install

PM	wall	±σ	user	sys	RSS	pgMinor
bun	7.41s	0.84s	6.21s	9.87s	575M	197.2K
utoo-npm	8.09s	1.56s	5.56s	11.43s	893M	115.6K
utoo	6.36s	0.17s	5.45s	10.94s	898M	111.6K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	5.2K	6.6K	1004M	4M	1.75G	1.75G	1M
utoo-npm	118.9K	80.6K	975M	3M	1.69G	1.69G	2M
utoo	96.0K	66.3K	974M	2M	1.69G	1.69G	2M

p4_warm_link

PM	wall	±σ	user	sys	RSS	pgMinor
bun	3.29s	0.05s	0.19s	2.38s	134M	31.0K
utoo-npm	2.31s	0.11s	0.61s	3.91s	81M	19.4K
utoo	2.23s	0.06s	0.62s	3.88s	82M	19.0K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	204	21	2K	6K	1.85G	1.75G	1M
utoo-npm	49.2K	22.0K	46K	14K	1.69G	1.69G	2M
utoo	48.8K	21.9K	46K	41K	1.70G	1.69G	2M

npmmirror.com

p0_full_cold

PM	wall	±σ	user	sys	RSS	pgMinor
bun	49.13s	33.75s	9.44s	9.98s	540M	399.6K
utoo-npm	21.92s	1.96s	9.15s	14.59s	744M	109.5K
utoo	26.32s	6.86s	8.88s	14.38s	783M	114.2K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	62.0K	4.4K	1.13G	10M	1.85G	1.74G	2M
utoo-npm	237.2K	119.9K	1008M	8M	1.69G	1.69G	2M
utoo	235.8K	105.8K	990M	8M	1.69G	1.69G	2M

p1_resolve

PM	wall	±σ	user	sys	RSS	pgMinor
bun	2.09s	0.32s	4.11s	1.14s	550M	174.7K
utoo-npm	4.81s	0.09s	2.17s	0.60s	75M	16.1K
utoo	12.79s	14.04s	2.31s	0.67s	76M	15.8K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	8.5K	5.3K	151M	3M	106M	-	2M
utoo-npm	46.7K	854	13M	2M	-	4M	2M
utoo	47.4K	590	13M	2M	-	4M	2M

p3_cold_install

PM	wall	±σ	user	sys	RSS	pgMinor
bun	17.71s	0.82s	5.81s	9.00s	249M	104.1K
utoo-npm	25.21s	2.47s	6.72s	13.00s	712M	90.6K
utoo	23.96s	1.34s	6.71s	13.21s	633M	88.2K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	34.6K	3.1K	1008M	7M	1.75G	1.75G	2M
utoo-npm	191.2K	99.5K	975M	6M	1.69G	1.69G	2M
utoo	190.0K	100.8K	994M	6M	1.69G	1.69G	2M

p4_warm_link

PM	wall	±σ	user	sys	RSS	pgMinor
bun	3.22s	0.09s	0.21s	2.30s	135M	32.1K
utoo-npm	2.32s	0.06s	0.59s	3.91s	84M	19.5K
utoo	2.39s	0.14s	0.59s	3.88s	85M	20.0K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	470	25	7M	62K	1.90G	1.73G	2M
utoo-npm	49.1K	22.0K	441K	17K	1.69G	1.69G	2M
utoo	50.3K	22.4K	439K	26K	1.69G	1.69G	2M

github-actions · 2026-04-26T16:36:11Z

📊 pm-bench-phases · `16f8203` · mac (`macos-latest`)

Workflow run — ant-design

PMs: utoo (this branch) · utoo-npm (latest published) · bun (latest)

npmjs.org

p0_full_cold

PM	wall	±σ	user	sys	RSS	pgMinor
bun	19.10s	5.78s	6.67s	20.03s	780M	50.3K
utoo-npm	21.35s	0.89s	10.42s	23.65s	1.06G	102.3K
utoo	24.98s	0.21s	11.68s	27.75s	1021M	103.1K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	15.4K	140.3K	-	-	1.79G	1.91G	1M
utoo-npm	13.1K	389.1K	-	-	1.64G	1.87G	2M
utoo	13.1K	370.6K	-	-	1.64G	1.84G	2M

p1_resolve

PM	wall	±σ	user	sys	RSS	pgMinor
bun	2.44s	0.22s	2.32s	0.96s	506M	33.0K
utoo-npm	6.23s	0.62s	4.09s	2.22s	541M	36.8K
utoo	8.26s	2.36s	4.50s	2.60s	551M	37.0K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	29	24.5K	-	-	110M	-	1M
utoo-npm	56	80.9K	-	-	28M	5M	2M
utoo	64	85.4K	-	-	28M	5M	2M

p3_cold_install

PM	wall	±σ	user	sys	RSS	pgMinor
bun	19.39s	4.38s	3.97s	21.19s	509M	33.1K
utoo-npm	15.24s	1.59s	4.08s	17.70s	760M	81.3K
utoo	11.48s	0.98s	3.71s	15.67s	814M	81.3K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	4.8K	139.1K	-	-	1.70G	1.94G	1M
utoo-npm	1.5K	240.6K	-	-	1.61G	1.87G	2M
utoo	1.3K	230.6K	-	-	1.61G	1.87G	2M

p4_warm_link

PM	wall	±σ	user	sys	RSS	pgMinor
bun	7.23s	0.62s	0.14s	3.03s	52M	3.9K
utoo-npm	6.29s	0.53s	0.87s	4.85s	90M	6.7K
utoo	5.10s	0.55s	0.75s	3.89s	98M	7.3K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	15.5K	902	-	-	1.87G	1.92G	1M
utoo-npm	12.8K	81.7K	-	-	1.61G	1.87G	2M
utoo	12.7K	76.3K	-	-	1.64G	1.87G	2M

npmmirror.com

p0_full_cold

PM	wall	±σ	user	sys	RSS	pgMinor
bun	29.43s	6.24s	6.74s	19.09s	570M	36.9K
utoo-npm	34.12s	3.74s	8.43s	27.15s	785M	74.6K
utoo	37.69s	10.82s	8.83s	28.31s	735M	78.6K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	14.2K	158.0K	-	-	1.77G	1.92G	2M
utoo-npm	4.1K	410.1K	-	-	1.61G	1.84G	2M
utoo	814	382.6K	-	-	1.61G	1.84G	2M

p1_resolve

PM	wall	±σ	user	sys	RSS	pgMinor
bun	1.90s	0.09s	2.67s	1.30s	580M	37.7K
utoo-npm	7.15s	2.26s	2.24s	1.40s	77M	5.6K
utoo	5.70s	0.96s	2.27s	1.41s	79M	5.8K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	10	24.0K	-	-	111M	-	2M
utoo-npm	4	42.5K	-	-	-	4M	2M
utoo	29	41.0K	-	-	-	4M	2M

p3_cold_install

PM	wall	±σ	user	sys	RSS	pgMinor
bun	32.65s	21.29s	4.09s	17.01s	265M	17.5K
utoo-npm	45.02s	21.68s	6.30s	22.58s	914M	78.4K
utoo	32.30s	2.58s	5.58s	19.71s	766M	78.9K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	1.7K	155.6K	-	-	1.65G	1.92G	2M
utoo-npm	1.6K	371.9K	-	-	1.61G	1.84G	2M
utoo	1.4K	349.4K	-	-	1.61G	1.84G	2M

p4_warm_link

PM	wall	±σ	user	sys	RSS	pgMinor
bun	3.73s	0.44s	0.09s	1.75s	44M	3.4K
utoo-npm	3.11s	0.38s	0.46s	2.42s	95M	7.1K
utoo	3.10s	0.33s	0.46s	2.36s	93M	7.0K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	12.2K	922	-	-	1.78G	1.91G	2M
utoo-npm	12.0K	70.1K	-	-	1.61G	1.83G	2M
utoo	12.4K	70.3K	-	-	1.61G	1.83G	2M

The headline architectural change of #2818 — preload phase shifts from a single-task `FuturesUnordered` cooperative poller to N long-lived `tokio::spawn` workers (or `wasm_bindgen_futures::spawn_local` on wasm32 where Send isn't satisfied). Stacks on top of #2826. ## Why Old design: main task owned `FuturesUnordered`, polled all preload futures cooperatively, and ran every per-future continuation (post-await body, completion handler, dispatch refill) on the same single task. The deeper await chain inside `resolve_package` (cache check + `OnceMap::get_or_init` + `RetryIf` + `request.send` + `bytes` + parse spawn_blocking) made each future yield 5+ times, and every yield round-tripped through main — saturating it. CI ant-design preload sustained avg_conc=55-61 even after Mutex / allocator hot-path eliminations, while the standalone manifest-bench (#2824) hit 92 on the same reqwest stack. ## How N long-lived `tokio::spawn` workers pulling from a shared lock-free `SegQueue<Dep>` with `DashSet` dedup. Each worker owns an `Arc<R>` clone and runs `resolve_package` on tokio's global executor — futures progress fully independently, no cooperative poll bottleneck. Main task only drains an `mpsc::unbounded_channel` of completions to fire receiver events + on_manifest callback. Termination: workers track `dispatched` / `completed: AtomicUsize` and park on a shared `Notify` when the queue is empty. When the last completion makes `completed == dispatched` and the queue is empty, the finishing worker raises a `shutdown` flag and wakes others; all workers drop their result_tx clones, the channel closes, and the main `recv().await` loop exits. ## Trait surface change - `MockRegistryClient` + `MockPackage` `derive(Clone)` so tests can wrap the mock in `Arc` for the new signature - `preload_manifests` takes `registry: Arc<R>` (was `&R`); call site in `run_preload_phase` clones the borrowed registry into a fresh `Arc`. Bound at every public surface up the chain bumped to `R: RegistryClient + Clone + MaybeSend + MaybeSync + 'static`, `R::Error: MaybeSend`. The `MaybeSend` / `MaybeSync` shims (added in #2826) keep the trait surface wasm-compatible. ## Companion changes folded in - **Inline simd_json parse** — drop `tokio::task::spawn_blocking` in `service/manifest.rs`. Worker-pool surfaced parse blocking- pool queue saturation: `queue p95=200ms sum=70-89s` over 2730 manifests on cap=4 CI runners. Inline parse on the worker thread eliminates dispatch + queue overhead. - **Workspace package.json parallel reads** — switch the per-pattern `for path in matched_paths` serial loop to `FuturesUnordered` fan-out. ant-design has ~200 workspace packages; saved ~150ms. - **Setup phase + lockfile-write timing logs** — round out the per-phase wall account for the bench-comment infrastructure. - **Manifests concurrency cap 64 → 128** — worker-pool delivers the parallelism that justifies the cap raise. CI ant-design avg_conc 84 at cap=128 (up from 55 under the old architecture); preload wall 3.10s → 2.15s. ## Wasm CI cfg-gates `tokio::spawn` to `wasm_bindgen_futures::spawn_local` on wasm32 since wasm-bindgen's `JsFuture` is `!Send`. Workers still run independently — single-threaded under wasm but the queue + Notify + mpsc termination story is unchanged. `cargo check -p utoo-wasm --target wasm32-unknown-unknown` clean. Tests: 164 ruborist + 10 doctests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Resolves overlap with merged upstream PRs: - #2856 rustls + aws-lc-rs (gated to macOS only, prevents Linux regression) - #2862 single-flight manifest fetches via OnceMap (full Result-aware variant) - #2824 pm-e2e-bench unified workflow + phase-isolated bench infrastructure - #2858 mdx option, #2857 externalType: promise, #2829 sync next.js Resolution choices: - Cargo.toml: keep next's macOS-only aws-lc-rs gating (validated by #2856 to avoid Linux/x86_64 CI regression). Add crossbeam-queue for the worker-pool SegQueue. Keep rt-multi-thread test feature for worker-pool tests. - util/oncemap.rs + util/mod.rs: take next's superset (adds get_or_try_init, Debug impl, refined failure semantics). - service/* + model/manifest.rs + traits/registry.rs: keep PR's worker-pool API surface — its preload.rs depends on these shapes for the FuturesUnordered -> worker-pool migration. - Drop redundant `use utoo_ruborist::util::oncemap::OnceMap;` lines in pm/util/{cloner,downloader}.rs after import-path consolidation. cargo clippy --all-targets -- -D warnings clean for utoo-pm + utoo-ruborist cargo test passes: 245 pm + 167 ruborist + 10 doctests

next merged #2868 which added `utoo-next` (latest published utoo) as a baseline alongside `utoo-npm` / `bun` in `bench-phases-linux`. Also resolves a PR-side staleness left from the earlier #2818 ↔ #2824 race: PR #2818 had seeded bench scripts under `e2e/`, but #2824 reorganized them into `bench/` on next while we were merging. Auto-merge couldn't reconcile the rename (both sides "added" files at different paths), so the duplicates lingered. - Drop `e2e/pm-bench-phases.sh` (412 lines, superseded by `bench/pm-bench-phases.sh`) - Drop `e2e/pm-bench-pcap.sh` (83 lines, superseded by `bench/pm-bench-pcap.sh`) - Drop `.github/workflows/pm-bench-phases.yml` (182 lines, superseded by `pm-e2e-bench.yml`) cargo check -p utoo-ruborist -p utoo-pm clean.

elrrrrrrr and others added 2 commits April 25, 2026 22:10

chore(manifest-bench): tombi format + fix typo

e190e41

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

gemini-code-assist Bot reviewed Apr 25, 2026

View reviewed changes

Comment thread crates/manifest-bench/src/main.rs Resolved

Comment thread crates/manifest-bench/src/main.rs Resolved

Comment thread crates/manifest-bench/src/main.rs Resolved

Comment thread bench/pm-bench-phases.sh Resolved

chore(manifest-bench): tombi 0.7.32 format

b136dc0

CI runs tombi 0.7.32 which aligns column widths across the file (local 0.8.0 normalises per-table and disagreed). Apply the formatting CI expects. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

This was referenced Apr 25, 2026

perf(pm): chunked parallel writes within tarball extraction + Tar Slip guard #2825

Closed

perf(pm): manifest cache & resolver alloc cleanup #2826

Closed

perf(pm): preload worker-pool replaces FuturesUnordered #2827

Closed

elrrrrrrr marked this pull request as draft April 25, 2026 15:22

elrrrrrrr and others added 7 commits April 26, 2026 00:03

utooland deleted a comment from github-actions Bot Apr 26, 2026

elrrrrrrr requested review from killagu and xusd320 April 26, 2026 13:54

elrrrrrrr and others added 4 commits April 26, 2026 22:04

Revert "ci(pm): split pm-build into build-only + per-consumer workflows"

469fb5d

This reverts commit 70f8df5.

utooland deleted a comment from github-actions Bot Apr 26, 2026

elrrrrrrr changed the title ~~ci(pm-bench-phases): phase-isolated bench + manifest-bench tool + PR auto-comment~~ ci(pm): pm-e2e-bench unified workflow + phase-isolated bench infrastructure Apr 26, 2026

elrrrrrrr added the A-Pkg Manager Area: Package Manager label Apr 26, 2026

elrrrrrrr added the benchmark Run pm-bench on PR label Apr 26, 2026

elrrrrrrr requested review from fireairforce and yuzheng14 April 26, 2026 15:21

elrrrrrrr marked this pull request as ready for review April 26, 2026 15:21

elrrrrrrr and others added 3 commits April 26, 2026 23:41

xusd320 approved these changes Apr 27, 2026

View reviewed changes

xusd320 merged commit 81d01c7 into next Apr 27, 2026
32 checks passed

xusd320 deleted the perf/bench-infra branch April 27, 2026 02:14

elrrrrrrr mentioned this pull request Apr 30, 2026

perf(pm): full PR #2818 perf bundle on top of worker-pool #2870

Closed

4 of 6 tasks

Navigation Menu

Repository navigation

ci(pm): pm-e2e-bench unified workflow + phase-isolated bench infrastructure - #2824

ci(pm): pm-e2e-bench unified workflow + phase-isolated bench infrastructure#2824
xusd320 merged 19 commits into
nextfrom
perf/bench-infra

elrrrrrrr commented Apr 25, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 26, 2026

Uh oh!

github-actions Bot commented Apr 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

elrrrrrrr commented Apr 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Triggers

Architecture

Test plan

Context

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 26, 2026

📊 pm-bench-phases · 16f8203 · linux (ubuntu-latest)

npmjs.org

p0_full_cold

p1_resolve

p3_cold_install

p4_warm_link

npmmirror.com

p0_full_cold

p1_resolve

p3_cold_install

p4_warm_link

Uh oh!

github-actions Bot commented Apr 26, 2026

📊 pm-bench-phases · 16f8203 · mac (macos-latest)

npmjs.org

p0_full_cold

p1_resolve

p3_cold_install

p4_warm_link

npmmirror.com

p0_full_cold

p1_resolve

p3_cold_install

p4_warm_link

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

elrrrrrrr commented Apr 25, 2026 •

edited

Loading

📊 pm-bench-phases · `16f8203` · linux (`ubuntu-latest`)

📊 pm-bench-phases · `16f8203` · mac (`macos-latest`)