perf(pm): parallelize workspace package.json reads by elrrrrrrr · Pull Request #2831 · utooland/utoo

elrrrrrrr · 2026-04-27T09:15:55Z

Summary

Workspace discovery: replace serial `for path in matched_paths { read_package_json(...).await }` with `FuturesUnordered` fan-out
Single-variable probe — closed PR perf(pm): preload worker pool replaces FuturesUnordered #2830 confirmed worker-pool architecture (alone, with cap=128, or both) is not the perf driver behind PR perf(pm): sequential writes within tarball extraction #2818/perf(pm): preload worker-pool replaces FuturesUnordered #2827's claims
This was one of the companion changes folded into perf(pm): preload worker-pool replaces FuturesUnordered #2827; isolating it should reveal whether parallel workspace reads is the actual driver

Hypothesis to test

PR #2827's commit msg claimed ~150ms savings on ant-design (~200 workspace packages). If `p1_resolve` drops measurably here, this is one of the real drivers; if no change, the perf gain came from elsewhere (e.g. multi-thread parse parallelism only available with `tokio::spawn`).

Test plan

`cargo fmt -p utoo-ruborist`
`cargo clippy -p utoo-ruborist --all-targets -- -D warnings --no-deps`
`cargo test -p utoo-ruborist` — all unit tests + doctests pass
CI `pm-e2e-*` (4 platforms)
CI `pm-bench-phases-*` with `benchmark` label — compare `utoo` to `utoo-next`

🤖 Generated with Claude Code

Replace the per-pattern serial `for path in matched_paths { read_package_json(...).await }` loop with a `FuturesUnordered` fan-out across all matched workspace directories. Each `package.json` read is small (typical < 4 KB) so completion order doesn't matter; results land into the `workspaces` vector as they complete. ant-design has ~200 workspace packages, and the serial loop paid 200× single-file FS round-trips on the resolver hot path between glob completion and BFS start. PR #2818 (closed) bundle reported ~150ms savings here on ant-design on macOS, but never got isolated bench numbers — this PR extracts the change as a single-variable probe to verify. Replaces the `workspace package.json parallel reads` companion change folded into PR #2827, which never landed in next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

gemini-code-assist

Code Review

This pull request optimizes workspace discovery by transitioning from serial to concurrent package.json reads using FuturesUnordered. Feedback suggests using a HashSet to collect workspace paths to avoid duplicates from overlapping glob patterns and utilizing buffer_unordered to limit concurrent I/O operations, preventing potential file descriptor exhaustion in large monorepos.

gemini-code-assist · 2026-04-27T09:17:28Z

+        // Collect every matched workspace directory across all patterns
+        // first, then read each `package.json` concurrently. The previous
+        // serial `for path in matched_paths { read_package_json(...).await }`
+        // paid one FS round-trip per workspace on the resolver hot path —
+        // ant-design has ~200 workspaces and the read takes the largest
+        // unmeasured chunk of `p1_resolve` between glob completion and
+        // BFS start.
+        let mut workspace_paths: Vec<PathBuf> = Vec::new();
        for pattern_str in patterns {
-            // Prepare glob pattern for package.json files
            let package_json_pattern = root_path.join(pattern_str).join("package.json");
-
-            // Match workspace directories using Glob::glob
            let matched_paths = self
                .glob
                .glob(&package_json_pattern)
                .await
                .map_err(|e| anyhow::anyhow!("Failed to glob pattern: {}", e))?;
-
            for path in matched_paths {
-                let workspace_path = match path.parent() {
-                    Some(p) => p.to_path_buf(),
-                    None => continue,
-                };
-
-                // Read workspace package.json
-                let workspace_pkg: PackageJson = match read_package_json(&workspace_path).await {
-                    Ok(pkg) => pkg,
-                    Err(e) => {
-                        tracing::debug!("Failed to read workspace package.json: {}", e);
-                        continue;
-                    }
-                };
-
-                tracing::debug!(
-                    "Found workspace: {} at {:?}",
-                    workspace_pkg.name,
-                    workspace_path
-                );
-                workspaces.push(WorkspacePackage {
-                    name: workspace_pkg.name.clone(),
-                    path: workspace_path,
-                    package_json: workspace_pkg,
-                });
+                if let Some(p) = path.parent() {
+                    workspace_paths.push(p.to_path_buf());
+                }
+            }
+        }


Overlapping glob patterns in the workspaces configuration (e.g., ["packages/*", "packages/my-pkg"]) can lead to the same directory being matched multiple times. This results in redundant file reads and duplicate WorkspacePackage entries in the final result, which may cause issues in downstream dependency resolution. Using a HashSet to collect unique paths avoids this.

// Collect every matched workspace directory across all patterns // first, then read each package.json concurrently. The previous // serial for path in matched_paths { read_package_json(...).await } // paid one FS round-trip per workspace on the resolver hot path — // ant-design has ~200 workspaces and the read takes the largest // unmeasured chunk of p1_resolve between glob completion and // BFS start. let mut workspace_paths = std::collections::HashSet::new(); for pattern_str in patterns { let package_json_pattern = root_path.join(pattern_str).join("package.json"); let matched_paths = self .glob .glob(&package_json_pattern) .await .map_err(|e| anyhow::anyhow!("Failed to glob pattern: {}", e))?; for path in matched_paths { if let Some(p) = path.parent() { workspace_paths.insert(p.to_path_buf()); } } }

gemini-code-assist · 2026-04-27T09:17:28Z

+        // Fan out the per-workspace package.json reads. Each is small
+        // (typical workspace package.json < 4 KB) so completion order
+        // doesn't matter — push results into `workspaces` as they land.
+        use futures::stream::{FuturesUnordered, StreamExt};
+        let mut futs: FuturesUnordered<_> = workspace_paths
+            .into_iter()
+            .map(|workspace_path| async move {
+                let pkg = read_package_json(&workspace_path).await;
+                (workspace_path, pkg)
+            })
+            .collect();


The current implementation initiates all package.json reads concurrently without a limit. In very large monorepos (e.g., thousands of packages), this could lead to file descriptor exhaustion or high memory pressure. Consider using futures::stream::iter with buffer_unordered to cap the maximum number of concurrent I/O operations to a reasonable limit.

Suggested change

// Fan out the per-workspace package.json reads. Each is small

// (typical workspace package.json < 4 KB) so completion order

// doesn't matter — push results into `workspaces` as they land.

use futures::stream::{FuturesUnordered, StreamExt};

let mut futs: FuturesUnordered<_> = workspace_paths

.into_iter()

.map(|workspace_path| async move {

let pkg = read_package_json(&workspace_path).await;

(workspace_path, pkg)

})

.collect();

// Fan out the per-workspace package.json reads. Each is small

// (typical workspace package.json < 4 KB) so completion order

// doesn't matter — push results into workspaces as they land.

use futures::stream::StreamExt;

let mut futs = futures::stream::iter(workspace_paths)

.map(|workspace_path| async move {

let pkg = read_package_json(&workspace_path).await;

(workspace_path, pkg)

})

.buffer_unordered(64);

elrrrrrrr · 2026-04-27T09:20:47Z

Closing — workspace count too small (~200 on ant-design, ~150ms theoretical savings) for the impact to surface above bench noise (p1_resolve σ ~0.1-0.7s). Not the driver candidate.

github-actions · 2026-04-27T09:38:20Z

📊 pm-bench-phases · `bb8f422` · linux (`ubuntu-latest`)

Workflow run — ant-design

PMs: utoo (this branch) · utoo-npm (latest published) · bun (latest)

npmjs.org

p0_full_cold

PM	wall	±σ	user	sys	RSS	pgMinor
bun	9.23s	0.15s	9.94s	10.14s	568M	290.1K
utoo-npm	10.02s	0.27s	11.21s	12.81s	1.33G	166.4K
utoo	10.45s	0.52s	11.28s	12.93s	1.30G	171.1K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	19.6K	17.2K	1.16G	7M	1.83G	1.72G	1M
utoo-npm	158.8K	131.7K	1.14G	4M	1.68G	1.68G	2M
utoo	160.8K	115.0K	1.14G	4M	1.68G	1.68G	2M

p1_resolve

PM	wall	±σ	user	sys	RSS	pgMinor
bun	2.57s	0.06s	3.83s	1.14s	495M	196.3K
utoo-npm	5.40s	0.02s	5.97s	1.08s	433M	65.3K
utoo	9.60s	7.17s	5.83s	1.10s	430M	69.7K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	13.3K	3.3K	200M	3M	104M	-	1M
utoo-npm	66.3K	3.0K	200M	2M	9M	5M	2M
utoo	66.0K	3.0K	203M	2M	9M	5M	2M

p3_cold_install

PM	wall	±σ	user	sys	RSS	pgMinor
bun	7.39s	0.76s	6.08s	10.09s	549M	198.7K
utoo-npm	7.79s	1.15s	5.48s	11.40s	814M	105.2K
utoo	6.61s	0.17s	5.46s	10.97s	857M	116.3K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	7.7K	7.7K	993M	5M	1.73G	1.73G	1M
utoo-npm	112.3K	72.4K	963M	3M	1.67G	1.67G	2M
utoo	97.5K	63.9K	963M	2M	1.67G	1.67G	2M

p4_warm_link

PM	wall	±σ	user	sys	RSS	pgMinor
bun	3.16s	0.08s	0.20s	2.33s	138M	32.3K
utoo-npm	2.46s	0.19s	0.58s	3.95s	82M	18.9K
utoo	2.21s	0.03s	0.58s	3.89s	82M	18.9K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	449	29	7M	42K	1.88G	1.72G	1M
utoo-npm	48.7K	21.9K	29K	11K	1.67G	1.67G	2M
utoo	49.3K	21.0K	29K	11K	1.68G	1.67G	2M

npmmirror.com

p0_full_cold

PM	wall	±σ	user	sys	RSS	pgMinor
bun	26.09s	3.22s	9.32s	10.37s	538M	387.4K
utoo-npm	22.46s	0.86s	8.35s	14.55s	730M	133.3K
utoo	48.17s	33.14s	8.40s	14.98s	693M	130.8K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	100.0K	5.0K	1.12G	13M	1.83G	1.72G	2M
utoo-npm	245.2K	113.6K	998M	9M	1.67G	1.68G	2M
utoo	264.1K	88.5K	1013M	9M	1.67G	1.68G	2M

p1_resolve

PM	wall	±σ	user	sys	RSS	pgMinor
bun	2.63s	0.04s	3.90s	1.30s	540M	202.2K
utoo-npm	4.80s	0.37s	2.18s	0.58s	74M	15.9K
utoo	4.79s	0.11s	2.10s	0.60s	76M	16.0K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	15.4K	4.0K	151M	3M	106M	-	2M
utoo-npm	46.5K	822	14M	2M	-	4M	2M
utoo	46.7K	751	14M	2M	-	4M	2M

p3_cold_install

PM	wall	±σ	user	sys	RSS	pgMinor
bun	17.64s	0.41s	6.13s	9.94s	237M	91.4K
utoo-npm	24.95s	0.28s	6.38s	13.08s	606M	98.7K
utoo	26.18s	1.65s	6.51s	13.11s	601M	93.0K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	59.1K	4.6K	999M	9M	1.73G	1.73G	2M
utoo-npm	178.7K	100.4K	976M	7M	1.67G	1.67G	2M
utoo	181.0K	100.0K	970M	7M	1.67G	1.67G	2M

p4_warm_link

PM	wall	±σ	user	sys	RSS	pgMinor
bun	3.20s	0.06s	0.17s	2.36s	134M	31.0K
utoo-npm	2.08s	0.07s	0.59s	3.92s	82M	18.9K
utoo	2.31s	0.18s	0.64s	3.89s	82M	19.2K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	240	21	3K	13K	1.84G	1.73G	2M
utoo-npm	47.9K	21.5K	69K	17K	1.67G	1.67G	2M
utoo	47.9K	23.0K	70K	17K	1.67G	1.67G	2M

github-actions · 2026-04-27T10:05:32Z

📊 pm-bench-phases · `bb8f422` · mac (`macos-latest`)

Workflow run — ant-design

PMs: utoo (this branch) · utoo-npm (latest published) · bun (latest)

npmjs.org

p0_full_cold

PM	wall	±σ	user	sys	RSS	pgMinor
bun	22.98s	3.39s	7.21s	23.20s	794M	51.3K
utoo-npm	23.15s	2.12s	10.99s	26.54s	1003M	104.0K
utoo	25.66s	2.40s	11.72s	27.98s	1.10G	103.9K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	16.9K	139.9K	-	-	1.79G	1.90G	1M
utoo-npm	13.8K	374.1K	-	-	1.63G	1.83G	2M
utoo	13.4K	374.4K	-	-	1.63G	1.84G	2M

p1_resolve

PM	wall	±σ	user	sys	RSS	pgMinor
bun	3.18s	1.08s	2.96s	1.50s	495M	32.1K
utoo-npm	6.51s	0.59s	5.00s	3.40s	546M	37.0K
utoo	6.70s	0.56s	5.32s	3.72s	551M	36.9K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	29	21.3K	-	-	110M	-	1M
utoo-npm	71	74.3K	-	-	28M	5M	2M
utoo	40	73.1K	-	-	28M	5M	2M

p3_cold_install

PM	wall	±σ	user	sys	RSS	pgMinor
bun	21.78s	2.56s	4.12s	22.77s	550M	35.8K
utoo-npm	18.06s	3.58s	4.62s	21.77s	697M	75.2K
utoo	15.94s	0.86s	4.54s	21.66s	776M	80.0K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	5.5K	136.7K	-	-	1.70G	1.94G	1M
utoo-npm	1.6K	244.0K	-	-	1.61G	1.83G	2M
utoo	1.4K	233.7K	-	-	1.61G	1.83G	2M

p4_warm_link

PM	wall	±σ	user	sys	RSS	pgMinor
bun	7.03s	0.54s	0.14s	2.85s	52M	3.9K
utoo-npm	5.60s	0.57s	0.73s	4.04s	100M	7.4K
utoo	5.46s	0.26s	0.73s	4.05s	92M	6.9K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	15.8K	1.4K	-	-	1.86G	1.91G	1M
utoo-npm	12.9K	84.2K	-	-	1.61G	1.84G	2M
utoo	13.2K	78.2K	-	-	1.63G	1.84G	2M

npmmirror.com

p0_full_cold

PM	wall	±σ	user	sys	RSS	pgMinor
bun	34.92s	5.72s	7.94s	24.86s	571M	36.9K
utoo-npm	31.76s	1.35s	8.33s	25.62s	702M	74.3K
utoo	36.38s	2.10s	8.77s	26.14s	710M	73.1K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	14.8K	166.3K	-	-	1.79G	1.90G	2M
utoo-npm	966	390.9K	-	-	1.61G	1.87G	2M
utoo	4.1K	400.2K	-	-	1.61G	1.88G	2M

p1_resolve

PM	wall	±σ	user	sys	RSS	pgMinor
bun	4.19s	0.11s	3.33s	2.32s	572M	37.2K
utoo-npm	7.46s	0.63s	1.97s	1.16s	77M	5.6K
utoo	13.53s	10.82s	2.13s	1.17s	77M	5.6K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	25	25.8K	-	-	111M	-	2M
utoo-npm	5	38.5K	-	-	-	4M	2M
utoo	22	40.4K	-	-	-	4M	2M

p3_cold_install

PM	wall	±σ	user	sys	RSS	pgMinor
bun	25.69s	4.32s	5.06s	27.32s	313M	20.6K
utoo-npm	48.84s	4.10s	7.36s	26.48s	656M	74.6K
utoo	37.09s	2.46s	6.54s	25.11s	723M	76.7K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	2.2K	165.2K	-	-	1.68G	1.95G	2M
utoo-npm	1.7K	369.3K	-	-	1.61G	1.84G	2M
utoo	1.6K	312.8K	-	-	1.61G	1.84G	2M

p4_warm_link

PM	wall	±σ	user	sys	RSS	pgMinor
bun	8.46s	1.29s	0.20s	3.65s	48M	3.6K
utoo-npm	6.69s	0.90s	0.88s	4.78s	96M	6.9K
utoo	7.23s	0.79s	0.91s	5.03s	96M	7.1K

PM	vCtx	iCtx	netRX	netTX	cache	node_mod	lock
bun	14.8K	2.6K	-	-	1.87G	1.91G	2M
utoo-npm	12.3K	79.2K	-	-	1.61G	1.84G	2M
utoo	12.3K	79.5K	-	-	1.61G	1.84G	2M

elrrrrrrr added the benchmark Run pm-bench on PR label Apr 27, 2026

gemini-code-assist Bot reviewed Apr 27, 2026

View reviewed changes

elrrrrrrr closed this Apr 27, 2026

Navigation Menu

Repository navigation

perf(pm): parallelize workspace package.json reads - #2831

perf(pm): parallelize workspace package.json reads#2831
elrrrrrrr wants to merge 1 commit into
nextfrom
perf/workspace-parallel-reads

elrrrrrrr commented Apr 27, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Apr 27, 2026

Uh oh!

gemini-code-assist Bot Apr 27, 2026

Uh oh!

elrrrrrrr commented Apr 27, 2026

Uh oh!

github-actions Bot commented Apr 27, 2026

Uh oh!

github-actions Bot commented Apr 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

elrrrrrrr commented Apr 27, 2026

Summary

Hypothesis to test

Test plan

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

elrrrrrrr commented Apr 27, 2026

Uh oh!

github-actions Bot commented Apr 27, 2026

📊 pm-bench-phases · bb8f422 · linux (ubuntu-latest)

npmjs.org

p0_full_cold

p1_resolve

p3_cold_install

p4_warm_link

npmmirror.com

p0_full_cold

p1_resolve

p3_cold_install

p4_warm_link

Uh oh!

github-actions Bot commented Apr 27, 2026

📊 pm-bench-phases · bb8f422 · mac (macos-latest)

npmjs.org

p0_full_cold

p1_resolve

p3_cold_install

p4_warm_link

npmmirror.com

p0_full_cold

p1_resolve

p3_cold_install

p4_warm_link

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

📊 pm-bench-phases · `bb8f422` · linux (`ubuntu-latest`)

📊 pm-bench-phases · `bb8f422` · mac (`macos-latest`)