experiment(pm): install hot-path streaming triplet (TTY-gate + FuturesUnordered + zero-copy) by elrrrrrrr · Pull Request #2902 · utooland/utoo

elrrrrrrr · 2026-05-06T15:10:34Z

What this PR is for

Observation experiment, not a merge candidate. Companion to #2887.

#2887 ships the 5 small structurally-clean micro-opts. This PR carries the bigger triplet that got A/B-reverted out of #2887 for widening utoo's p0_full_cold σ to ~14× baseline. We want to keep tracking it because the mean is fine and the p3 behavior is meaningfully better with it active — the issue is purely σ-shape, and that needs a fix (probably a layered semaphore for long-tail tarballs) before it's mergeable.

What's in the triplet (single commit `b92fc85a`)

TTY-gate progress_inc / progress_set_length helpers — non-TTY callers (CI, piped output) skip indicatif's Mutex<ProgressState> write-lock. ~9k Mutex acquisitions removed per CI install.
install_packages per-layer FuturesUnordered streaming — drains tasks in completion order so a slow tarball doesn't pin its siblings; surfaces the first error early.
Zero-copy tar entries — ExtractedEntry { content: Vec<u8> } → (data_offset, data_len) slicing into the shared decompressed buffer. Eliminates per-file Vec<u8> alloc + copy.

Tradeoff observed in earlier CI A/B (#2887 σ investigation)

State	utoo `p0_full_cold` σ ratio (vs utoo-next)	utoo `p3_cold_install` σ
Triplet ACTIVE	11–31× baseline (wide)	0.07–0.5s (very tight)
Triplet REVERTED (#2887 ships state)	0.3–1.1× (parity)	0.5–14s (occasional 39s long-tail outlier)

The streaming refactor IS doing its job — preventing one long-tail tarball from pinning the whole layer. But by widening the per-task latency distribution, p0's wall variance climbs.

What we want to observe

p0 σ behavior across multiple bench-phases-linux runs
p3 σ tightness — should be visibly better than perf(pm): install hot-path micro-opts (OnceMap API + cache-hit fast-paths, file_type, binary mirror gate) #2887's
Whether p3 occasional outliers (39s) really go away with streaming active
iCtx / vCtx delta — streaming changes the spawn_blocking dispatch pattern

Future work (not in this PR)

A layered approach to long-tail handling:

Detect long-tail tarballs (size estimate from gzip footer, or measured download time)
Route them through a smaller secondary semaphore so they can't starve install layer's slot turnover
Or: split the install loop's drain into "fast" (cache hit / small) and "slow" (long-tail) lanes, drain fast in completion order but slow in spawn order

This PR is just the data-gathering vehicle. No merge intent until the design is sound.

Companion

perf(pm): install hot-path micro-opts (OnceMap API + cache-hit fast-paths, file_type, binary mirror gate) #2887 — the 5 small micro-opts that did ship

🤖 Generated with Claude Code

…ero-copy tar entries Three independent install-path changes that compound on cold installs. 1. **TTY-gate `PROGRESS_BAR.inc`/`set_length`** — `install.rs` had 8 raw `PROGRESS_BAR.inc(1)` plus a `set_length` that bypassed the `IS_TTY` short-circuit already present in `ProgressReceiver`. indicatif always takes the internal `Mutex<ProgressState>` write-lock even with a hidden draw target, so non-TTY runs (CI, piped output) were paying ~9k Mutex acquisitions per install. Wrapped them in `progress_inc` / `progress_set_length` helpers that match the receiver's gating pattern. 2. **`install_packages` per-layer streaming** — replaced `for task in clone_tasks { task.await?? }` with `FuturesUnordered`. The previous serial join walked tasks in spawn order, so a single slow tarball blocked the layer's `await` loop while later finishers piled up unobserved — pure idle latency between layers on deep trees. Now the layer drains in completion order and propagates the first error early. 3. **Zero-copy tar entries in `extract_tarball_sync`** — `ExtractedEntry` previously held an owned `Vec<u8>` copied via `read_to_end` for every file. Each tarball with N files paid N allocs + N memcpys before the parallel-write phase even started. Switched to `(data_offset, data_len)` indices into the decompressed buffer; the rayon write phase slices into the shared `&[u8]` directly. Tar Slip + path-traversal guards untouched, plus a buffer-overrun check on the recorded offsets. Built on top of #2874 to compound with the rayon offload + h1+pool+dns network layer. CI bench-phases-linux will validate the deltas.

gemini-code-assist

Code Review

This pull request optimizes package installation and tarball extraction. Key improvements include the use of FuturesUnordered to process tasks in completion order, a zero-copy approach for file extraction using a shared buffer with offsets, and TTY-gated progress bar updates to reduce mutex contention. Feedback was provided regarding the use of is_none_or, which may cause compatibility issues with older Rust versions.

I am having trouble creating individual review comments. Click here to see my feedback.

crates/pm/src/util/extractor.rs (194)

The is_none_or method on Option was stabilized in Rust 1.82.0. If this project targets an older MSRV (Minimum Supported Rust Version), this will cause a compilation error. For better compatibility, consider using map_or(true, ...) instead.

.map_or(true, |end| end > buf.len())

github-actions · 2026-05-07T01:12:30Z

📊 pm-bench-phases · `95d9a49` · linux (`ubuntu-latest`)

Workflow run — ant-design

PMs: utoo (this branch) · utoo-npm (latest published) · bun (latest)

npmjs.org

p0_full_cold

PM	wall	±σ	user	sys	RSS	pgMinor
bun	14.25s	1.73s	7.69s	7.42s	790M	330.3K
utoo-next	12.75s	3.39s	8.24s	9.42s	1.43G	193.5K
utoo-npm	10.74s	1.16s	8.31s	9.40s	1.35G	187.8K
utoo	18.80s	10.63s	7.94s	9.37s	959M	124.6K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	17.7K	15.5K	1.16G	6M	1.84G	1.72G	1M
utoo-next	135.2K	82.7K	1.14G	5M	1.68G	1.68G	2M
utoo-npm	138.9K	95.4K	1.14G	5M	1.68G	1.68G	2M
utoo	168.2K	75.5K	1.14G	5M	1.68G	1.68G	2M

p1_resolve

PM	wall	±σ	user	sys	RSS	pgMinor
bun	2.23s	0.40s	3.17s	0.87s	492M	180.3K
utoo-next	2.70s	0.14s	3.95s	1.64s	617M	90.3K
utoo-npm	2.58s	0.10s	4.02s	1.67s	588M	89.3K
utoo	2.98s	0.42s	4.05s	1.58s	609M	84.4K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	9.9K	3.9K	203M	3M	105M	-	1M
utoo-next	72.2K	107.5K	198M	2M	7M	3M	2M
utoo-npm	75.9K	115.3K	198M	2M	7M	3M	2M
utoo	72.6K	108.7K	198M	2M	7M	3M	2M

p3_cold_install

PM	wall	±σ	user	sys	RSS	pgMinor
bun	13.81s	10.46s	4.55s	7.08s	671M	215.6K
utoo-next	12.86s	8.44s	4.23s	8.38s	956M	116.9K
utoo-npm	11.21s	4.69s	4.23s	8.37s	911M	116.6K
utoo	8.00s	1.20s	3.91s	8.10s	490M	61.3K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	9.2K	4.1K	994M	4M	1.74G	1.74G	1M
utoo-next	125.8K	55.2K	965M	3M	1.68G	1.68G	2M
utoo-npm	126.0K	58.3K	965M	3M	1.68G	1.68G	2M
utoo	114.5K	66.8K	965M	3M	1.68G	1.68G	2M

p4_warm_link

PM	wall	±σ	user	sys	RSS	pgMinor
bun	4.67s	0.16s	0.17s	1.88s	137M	32.2K
utoo-next	3.29s	0.45s	0.39s	2.86s	78M	18.4K
utoo-npm	2.97s	0.27s	0.38s	2.93s	82M	18.6K
utoo	4.26s	0.67s	0.37s	2.91s	78M	18.3K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	250	33	7M	37K	1.88G	1.70G	1M
utoo-next	41.3K	16.6K	3K	7K	1.68G	1.68G	2M
utoo-npm	44.0K	17.3K	21K	21K	1.68G	1.68G	2M
utoo	43.3K	18.9K	4K	29K	1.68G	1.68G	2M

npmmirror.com: no output captured.

The pcap-only bench was previously a one-off that captured `p1_resolve` across `utoo` and `bun`, and assumed the project tree was already cloned by `pm-bench-phases.sh` running in the same job. That gave us metadata fan-out, but install-phase regressions (#2902 / #2903 / #2904 / #2905 σ widening on `p0_full_cold`) live in the tarball download path, not in resolve. This commit makes the pcap bench self-contained and covers both phases for three PMs: - Self-clone the project if `$PROJECT_DIR` is missing (mirrors `pm-bench-phases.sh`), so this script runs as a standalone CI job. - Add a `<pm>-install` capture per PM: lock pre-existing, `cache + node_modules` wiped, then `<pm> install`. This is the cold-tarball-download phase where the σ-widening lives. - Add `utoo-next` as a third PM: built upstream by `build-linux`'s bench-baseline step (now also gated on `pm-bench-pcap`), downloaded via the same artifact path as `bench-phases-linux`. Skipped in local runs where `$UTOO_NEXT_BIN` is unset. Workflow change: - `pm-bench-pcap-linux` now downloads the `utoo-next-linux-x64` artifact and exports `UTOO_NEXT_BIN` exactly like `bench-phases-linux` does. - `Build next branch utoo` and `Upload utoo-next binary` steps in `build-linux` now also fire for `inputs.target == 'pm-bench-pcap'`, not only `pm-bench-phases`. Outputs in `/tmp/pm-bench-pcap`: dns.txt utoo-{resolve,install}.{pcap,log} utoo-next-{resolve,install}.{pcap,log} (when UTOO_NEXT_BIN set) bun-{resolve,install}.{pcap,log} Drives the analysis of whether the install hot-path's increased concurrency (FuturesUnordered streaming, zero-copy tar, TTY-gate) saturates outbound TCP and starves the download path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(pm): zero-copy tar entries in extract_tarball_sync `ExtractedEntry` previously held an owned `Vec<u8>` copied via `read_to_end` for every file. Each tarball with N files paid N allocs + N memcpys before the parallel-write phase even started. Switched to `(data_offset, data_len)` indices into the decompressed buffer; the rayon write phase slices into the shared `&[u8]` directly. Tar Slip + path-traversal guards untouched, plus a buffer-overrun check on the recorded offsets. This is 3/3 of the triplet from #2902, split out for independent A/B benchmarking against the recently-merged baseline #2887. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(bench): extend pcap to install phase + utoo-next baseline The pcap-only bench was previously a one-off that captured `p1_resolve` across `utoo` and `bun`, and assumed the project tree was already cloned by `pm-bench-phases.sh` running in the same job. That gave us metadata fan-out, but install-phase regressions (#2902 / #2903 / #2904 / #2905 σ widening on `p0_full_cold`) live in the tarball download path, not in resolve. This commit makes the pcap bench self-contained and covers both phases for three PMs: - Self-clone the project if `$PROJECT_DIR` is missing (mirrors `pm-bench-phases.sh`), so this script runs as a standalone CI job. - Add a `<pm>-install` capture per PM: lock pre-existing, `cache + node_modules` wiped, then `<pm> install`. This is the cold-tarball-download phase where the σ-widening lives. - Add `utoo-next` as a third PM: built upstream by `build-linux`'s bench-baseline step (now also gated on `pm-bench-pcap`), downloaded via the same artifact path as `bench-phases-linux`. Skipped in local runs where `$UTOO_NEXT_BIN` is unset. Workflow change: - `pm-bench-pcap-linux` now downloads the `utoo-next-linux-x64` artifact and exports `UTOO_NEXT_BIN` exactly like `bench-phases-linux` does. - `Build next branch utoo` and `Upload utoo-next binary` steps in `build-linux` now also fire for `inputs.target == 'pm-bench-pcap'`, not only `pm-bench-phases`. Outputs in `/tmp/pm-bench-pcap`: dns.txt utoo-{resolve,install}.{pcap,log} utoo-next-{resolve,install}.{pcap,log} (when UTOO_NEXT_BIN set) bun-{resolve,install}.{pcap,log} Drives the analysis of whether the install hot-path's increased concurrency (FuturesUnordered streaming, zero-copy tar, TTY-gate) saturates outbound TCP and starves the download path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(bench): add tshark TCP-stress metrics to pcap bench Building on the install-phase pcap capture from the previous commit, post-process each .pcap with tshark to extract pre-TLS metrics that directly probe the "install greediness starves download" hypothesis without needing TLS session-key dumping: zero_windows — receive buffer full → server paused. Direct evidence that the app's tokio runtime is not draining the socket fast enough between extracts. retransmits — server resent because ACK was late. Indirect evidence of receive-side stall. duplicate_acks — receiver re-sent ACK because it perceived a gap. stream_gap_* — inter-packet gap distribution per TCP stream (p50 / p99 / max in microseconds). p99 / max measure the longest pause an active connection experienced — if utoo shows multi-hundred-ms gaps where utoo-next shows tens of ms, install is freezing the runtime mid-download. Per-capture summaries land at $PCAP_DIR/<name>.summary.json. They are aggregated into a top-level summary.json via jq -s, so artifact consumers can compare metrics across PMs without re-parsing the 100s of MB of raw pcaps. Single-pass tshark over the pcap with -T fields keeps cost bounded to ~1 minute per 1 GB of capture; the full analysis pass runs after all captures so it does not bleed into wall-clock measurement. Workflow change: Install pcap tools step now also installs tshark + jq, with wireshark-common pre-seeded so tshark installs non-interactively (we only read existing pcaps, no setuid dumpcap needed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(bench): make tshark analysis best-effort, awk-only wall-time parse The analysis pass aborted the whole job on the first .pcap because the wall-time grep returned no match, and `set -eo pipefail` propagated that exit-1 through `local x=$(grep | awk)` (the multi-line `local x; x=$(...)` form does NOT mask the exit code, unlike `local x=$(...)` on one line — bash gotcha). Two-part fix: 1. Drop into `set +e` / `set +o pipefail` for the analysis function body. The metrics are diagnostic — one tshark hiccup or an empty log line should not nuke a 25-minute capture run. Strict mode is restored at the end of the function so the rest of the script keeps its safety net. 2. Replace `grep -oE | awk` with awk-only. awk returns 0 even when no record matches, so empty-result log files no longer trip pipefail. Same parse, fewer pipes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(bench): add iostat sampling to pcap bench for disk-saturation diagnosis The TCP-level analysis (zero-copy retx=123 vs baseline 4-18) gave strong evidence that utoo's receiver runtime is under back-pressure during install, but it doesn't tell us *why*. The leading hypothesis is disk IO saturation: rayon's parallel `fs::create + write_all` over 80k+ files in the ant-design tarball burst can outrun GitHub Actions runners' Azure-disk IOPS budget, blocking write threads → tokio threads back up → socket buffers fill. This commit adds an iostat-x sampler to each capture: capture_one() now spawns `iostat -x -y 1` in parallel with tcpdump, writing per-second device samples to $PCAP_DIR/<name>.iostat.txt. Both samplers are torn down with the workload command. analyze_pcap() parses the iostat log via column-position lookup (sysstat header row → column index map) and extracts: io_util_max_pct — peak disk-busy percentage io_util_avg_pct — average disk-busy percentage io_w_iops_max — peak write IOPS io_w_kbs_max — peak write throughput (kB/s) io_w_await_max_ms — peak write queue wait (ms) io_samples — sample count for sanity check These six fields land in summary.json alongside the TCP metrics, so artifact consumers can directly cross-correlate disk pressure with TCP back-pressure within the same capture window. The decision rule: * If io_util_max_pct stays high (>80%) on the experiment branch while baseline same-PM utoo-next stays low → install path is saturating disk and that's the mechanism. * If both branches show similar low %util, disk is not the bottleneck and we keep looking (e.g. CPU contention). Workflow: apt install adds `sysstat` (iostat lives there). It is preinstalled on ubuntu-latest images today, but pinning the dependency makes future image rebuilds resilient. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(pm): chunked write in extract_tarball_sync to bound disk burst Direct A/B follow-up: pcap+iostat instrumentation on this branch (par_iter zero-copy) showed install-time disk burst saturating GHA's disk queue — util_max=92%, w_await peaks of 490ms, paired with TCP-level retx=123 vs utoo-next's retx=4 in the same run. The mechanism: rayon spreads every individual file write across all workers, so each tarball's extract submits N concurrent IO syscalls and the in-kernel queue depth blows up. Write threads block on the saturated queue, the tokio runtime can't drain socket buffers between extracts, ACKs delay, server retransmits. Switch from `par_iter()` to `par_chunks(64)` with a sequential body inside each chunk. Cross-thread parallelism is preserved: each rayon worker still picks up a chunk and runs in parallel with sibling chunks. What changes is the in-flight write count *per worker* — it goes from "as many as rayon will divide into me" down to one. The disk queue stays drained. Same pcap+iostat A/B with this commit: retx 123 → 10 (Δ utoo-vs-utoo-next +119 → +4) zwin 9 → 1 util_max 92% → 81% (no longer saturating) util_avg 19% → 19% (work done, just smoother) w_await 490ms → 160ms (disk queue not backed up) wall 18s → 20s (run-to-run noise, no real loss) Companion to the iter() / rayon-cap experiments on experiment/zero-copy-{iter-write,rayon-cap}: par_chunks is the only option that brought retx back to baseline *without* increasing zwin or wall, so it's the recommended ship form. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pm): use try_from for tar offset/size to avoid 32-bit truncation Address Gemini review feedback on #2905 (lines 204-205): the previous \`as usize\` cast on \`entry.raw_file_position()\` and \`entry.size()\` silently truncates on 32-bit targets when a tar header claims a >4GB value. utoo only ships 64-bit binaries, so this is defensive — but the \`try_from\` path also turns a malformed tar header into a clean error instead of a buffer-overrun bug. The buffer-overrun guard a few lines below catches the same class of issue, so this is belt-and-suspenders. Gemini's second comment on \`is_none_or\` does not apply: the project pins to \`nightly-2026-04-02\` (rust-toolchain.toml), which is far past \`is_none_or\`'s 1.82 stabilization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(pm): use bytes::Bytes for zero-copy tar entry slicing The previous form stored \`(data_offset, data_len)\` and reconstructed the slice at every write call site: let data = &buf[entry.data_offset..entry.data_offset + entry.data_len]; file.write_all(data)?; That worked but leaked offset/len arithmetic into the hot loop and forced the writer to know the parent buffer existed. \`bytes::Bytes\` is already a workspace dep — converting the decompressed buffer into \`Bytes\` once after \`output.truncate\` lets each entry hold a reference-counted slice that shares the underlying Arc allocation (no data copy, one atomic increment per slice). Diff at the call site: -let data = &buf[entry.data_offset..entry.data_offset + entry.data_len]; -file.write_all(data)?; +file.write_all(&entry.data)?; The Tar Slip / path-traversal guards and buffer-overrun check are preserved — \`Bytes::slice\` panics on out-of-bounds, so the explicit length validation is still load-bearing. The atomic ref-count cost (~ns per slice) is negligible against the write-syscall path; pcap+iostat metrics from the prior par_chunks(64) work are unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

elrrrrrrr · 2026-05-11T08:41:46Z

Superseded by #2924 / #2929 stack — experiment archived.

elrrrrrrr added the benchmark Run pm-bench on PR label May 6, 2026

gemini-code-assist Bot reviewed May 6, 2026

View reviewed changes

elrrrrrrr added benchmark Run pm-bench on PR and removed benchmark Run pm-bench on PR labels May 6, 2026

Merge branch 'next' into experiment/install-streaming-triplet

d1099e8

utooland deleted a comment from github-actions Bot May 6, 2026

elrrrrrrr added the A-Pkg Manager Area: Package Manager label May 7, 2026

utooland deleted a comment from github-actions Bot May 7, 2026

elrrrrrrr closed this May 11, 2026

Navigation Menu

Repository navigation

experiment(pm): install hot-path streaming triplet (TTY-gate + FuturesUnordered + zero-copy) - #2902

experiment(pm): install hot-path streaming triplet (TTY-gate + FuturesUnordered + zero-copy)#2902
elrrrrrrr wants to merge 2 commits into
nextfrom
experiment/install-streaming-triplet

elrrrrrrr commented May 6, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

github-actions Bot commented May 7, 2026

Uh oh!

elrrrrrrr commented May 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

elrrrrrrr commented May 6, 2026

What this PR is for

What's in the triplet (single commit b92fc85a)

Tradeoff observed in earlier CI A/B (#2887 σ investigation)

What we want to observe

Future work (not in this PR)

Companion

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

crates/pm/src/util/extractor.rs (194)

Uh oh!

github-actions Bot commented May 7, 2026

📊 pm-bench-phases · 95d9a49 · linux (ubuntu-latest)

npmjs.org

p0_full_cold

p1_resolve

p3_cold_install

p4_warm_link

Uh oh!

elrrrrrrr commented May 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

What's in the triplet (single commit `b92fc85a`)

📊 pm-bench-phases · `95d9a49` · linux (`ubuntu-latest`)