Skip to content

perf(pm): parallelize workspace package.json reads #2831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
elrrrrrrr wants to merge 1 commit into from
Closed

perf(pm): parallelize workspace package.json reads #2831

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 43 additions & 28 deletions crates/ruborist/src/resolver/workspace.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,42 +68,57 @@ impl<G: Glob> WorkspaceDiscovery<G> {
None => return Ok(workspaces),
};

// Collect every matched workspace directory across all patterns
// first, then read each `package.json` concurrently. The previous
// serial `for path in matched_paths { read_package_json(...).await }`
// paid one FS round-trip per workspace on the resolver hot path —
// ant-design has ~200 workspaces and the read takes the largest
// unmeasured chunk of `p1_resolve` between glob completion and
// BFS start.
let mut workspace_paths: Vec<PathBuf> = Vec::new();
for pattern_str in patterns {
// Prepare glob pattern for package.json files
let package_json_pattern = root_path.join(pattern_str).join("package.json");

// Match workspace directories using Glob::glob
let matched_paths = self
.glob
.glob(&package_json_pattern)
.await
.map_err(|e| anyhow::anyhow!("Failed to glob pattern: {}", e))?;

for path in matched_paths {
let workspace_path = match path.parent() {
Some(p) => p.to_path_buf(),
None => continue,
};

// Read workspace package.json
let workspace_pkg: PackageJson = match read_package_json(&workspace_path).await {
Ok(pkg) => pkg,
Err(e) => {
tracing::debug!("Failed to read workspace package.json: {}", e);
continue;
}
};

tracing::debug!(
"Found workspace: {} at {:?}",
workspace_pkg.name,
workspace_path
);
workspaces.push(WorkspacePackage {
name: workspace_pkg.name.clone(),
path: workspace_path,
package_json: workspace_pkg,
});
if let Some(p) = path.parent() {
workspace_paths.push(p.to_path_buf());
}
}
}
Comment on lines +71 to +91

@gemini-code-assist gemini-code-assist Bot Apr 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Overlapping glob patterns in the workspaces configuration (e.g., ["packages/*", "packages/my-pkg"]) can lead to the same directory being matched multiple times. This results in redundant file reads and duplicate WorkspacePackage entries in the final result, which may cause issues in downstream dependency resolution. Using a HashSet to collect unique paths avoids this.

        // Collect every matched workspace directory across all patterns
        // first, then read each package.json concurrently. The previous
        // serial for path in matched_paths { read_package_json(...).await }
        // paid one FS round-trip per workspace on the resolver hot path —
        // ant-design has ~200 workspaces and the read takes the largest
        // unmeasured chunk of p1_resolve between glob completion and
        // BFS start.
        let mut workspace_paths = std::collections::HashSet::new();
        for pattern_str in patterns {
            let package_json_pattern = root_path.join(pattern_str).join("package.json");
            let matched_paths = self
                .glob
                .glob(&package_json_pattern)
                .await
                .map_err(|e| anyhow::anyhow!("Failed to glob pattern: {}", e))?;
            for path in matched_paths {
                if let Some(p) = path.parent() {
                    workspace_paths.insert(p.to_path_buf());
                }
            }
        }


// Fan out the per-workspace package.json reads. Each is small
// (typical workspace package.json < 4 KB) so completion order
// doesn't matter — push results into `workspaces` as they land.
use futures::stream::{FuturesUnordered, StreamExt};
let mut futs: FuturesUnordered<_> = workspace_paths
.into_iter()
.map(|workspace_path| async move {
let pkg = read_package_json(&workspace_path).await;
(workspace_path, pkg)
})
.collect();
Comment on lines +93 to +103

@gemini-code-assist gemini-code-assist Bot Apr 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation initiates all package.json reads concurrently without a limit. In very large monorepos (e.g., thousands of packages), this could lead to file descriptor exhaustion or high memory pressure. Consider using futures::stream::iter with buffer_unordered to cap the maximum number of concurrent I/O operations to a reasonable limit.

Suggested change
// Fan out the per-workspace package.json reads. Each is small
// (typical workspace package.json < 4 KB) so completion order
// doesn't matter — push results into `workspaces` as they land.
use futures::stream::{FuturesUnordered, StreamExt};
let mut futs: FuturesUnordered<_> = workspace_paths
.into_iter()
.map(|workspace_path| async move {
let pkg = read_package_json(&workspace_path).await;
(workspace_path, pkg)
})
.collect();
// Fan out the per-workspace package.json reads. Each is small
// (typical workspace package.json < 4 KB) so completion order
// doesn't matter — push results into workspaces as they land.
use futures::stream::StreamExt;
let mut futs = futures::stream::iter(workspace_paths)
.map(|workspace_path| async move {
let pkg = read_package_json(&workspace_path).await;
(workspace_path, pkg)
})
.buffer_unordered(64);


while let Some((workspace_path, pkg)) = futs.next().await {
match pkg {
Ok(workspace_pkg) => {
tracing::debug!(
"Found workspace: {} at {:?}",
workspace_pkg.name,
workspace_path
);
workspaces.push(WorkspacePackage {
name: workspace_pkg.name.clone(),
path: workspace_path,
package_json: workspace_pkg,
});
}
Err(e) => {
tracing::debug!("Failed to read workspace package.json: {}", e);
}
}
}

Expand Down
Loading