[elrrrrrrr]

• Implement pipeline architecture using oncemap for manifest parsing, TGZ download, extraction, and file cloning.
• use libdeflater for decompression and rayon for parallel file writing.
• Remove tokio-runtime spawn_blocking limitations.

[gemini-code-assist]

Summary of Changes

Hello @elrrrrrrr, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors the dependency resolution and installation process within the utoo-pm crate. The core change is the implementation of a pipeline architecture that enables highly concurrent operations for fetching package manifests, downloading tarballs, and cloning packages into the node_modules directory. This design aims to drastically improve the speed and efficiency of utoo install commands by parallelizing tasks that were previously more sequential. The refactor also includes updates to dependency versions and the adoption of more performant compression libraries.

Highlights

• Pipeline Architecture: Implemented a new pipeline for dependency resolution and installation, allowing concurrent manifest fetching, tarball downloading, and package cloning. This design is inspired by Bun's approach to speed up utoo install operations.
• Concurrency Control & Deduplication: Introduced global OnceMap instances and semaphores to manage and deduplicate concurrent download and clone operations, ensuring efficiency and preventing redundant work across different stages of the pipeline.
• Performance Optimizations: Switched from stream-based tarball unpacking to the libdeflater library for Gzip decompression, which is expected to offer significant performance improvements during package extraction.
• Dependency Management Refinements: Updated Cargo.lock and Cargo.toml to reflect new dependencies (libdeflater, bytes) and removed/updated others (async-compression, tokio-tar, redox_syscall), aligning with the new pipeline's requirements.
• Simplified Tokio Runtime: The main function now leverages the #[tokio::main] macro for simpler and more idiomatic Tokio runtime initialization, reducing boilerplate code.

Changelog

• Cargo.lock
  • Updated async-compression from 0.3.15 to 0.4.32.
  • Updated bytes from 1.10.1 to 1.11.1.
  • Added new dependencies libdeflate-sys and libdeflater (version 1.25.0).
  • Removed tokio-tar dependency.
  • Adjusted redox_syscall dependency references.
• crates/pm/Cargo.toml
  • Removed explicit version and features for async-compression.
  • Added bytes = "1.11.0".
  • Added libdeflater = "1.25".
  • Removed tokio-tar = "0.3.1".
  • Removed features = ["io"] from tokio-util.
• crates/pm/src/cmd/deps.rs
  • Integrated PipelineInstaller to start and await pipeline workers during dependency building.
• crates/pm/src/cmd/rebuild.rs
  • Modified rebuild to await pipeline workers if they were initiated during ensure_package_lock.
• crates/pm/src/helper/fs.rs
  • Updated BuildDepsOptions to use PipelineReceiver for progress events and to return PipelineChannels for worker management.
• crates/pm/src/helper/lock.rs
  • Introduced BuildDepsResult to encapsulate PackageLock and optional PipelineHandles.
  • Replaced build_deps_with_download with build_deps_with_pipeline to leverage the new pipeline.
• crates/pm/src/main.rs
  • Switched to #[tokio::main] macro for simplified Tokio runtime initialization.
• crates/pm/src/service/install.rs
  • Removed manual semaphore-based concurrency control, now handled by the pipeline.
  • Updated install_packages to use clone_package_once from the pipeline service.
  • Ensures pipeline workers complete after package installation.
• crates/pm/src/service/mod.rs
  • Added pipeline module.
• crates/pm/src/service/pipeline.rs
  • New file introducing the core pipeline logic, including PipelineStats, OnceMap for caching, semaphores for concurrency, and workers for concurrent download and clone operations.
• crates/pm/src/util/cloner.rs
  • Refactored Linux cloning logic to a phased approach (collect, create dirs, clone files) for better control and performance.
• crates/pm/src/util/config.rs
  • Added get_manifests_concurrency_limit_sync for synchronous access to concurrency settings.
• crates/pm/src/util/downloader.rs
  • Replaced DashMap based locking with OnceMap for download deduplication.
  • Introduced download_bytes and extract_and_write functions.
  • Switched to libdeflater for Gzip decompression, improving performance.
• crates/pm/src/util/oncemap.rs
  • Added wait_if_pending method to OnceMap to allow tasks to wait for in-progress computations.
• crates/ruborist/src/lib.rs
  • Exported PackageTarballInfo for use in progress events.
• crates/ruborist/src/resolver/builder.rs
  • Modified BFS phase to emit PackagePlaced events for pipeline cloning, including parent path and depth information.
  • Updated BuildEvent parameters to use &str for efficiency.
• crates/ruborist/src/resolver/preload.rs
  • Increased DEFAULT_CONCURRENCY for manifest fetching to 64.
  • Emits PackageResolved events during preloading to trigger early tarball downloads.
  • Updated BuildEvent parameters to use &str.
• crates/ruborist/src/traits/progress.rs
  • Introduced PackageTarballInfo and new BuildEvent::PackageResolved and BuildEvent::PackagePlaced events.
  • Refactored BuildEvent and EventReceiver to use references (&'a str, &'a Path) for improved performance and reduced allocations.

Activity

• The pull request was opened by elrrrrrrr with the title "refactor(pm): pipeline deps resolver", indicating a significant refactoring effort.
• The changes introduce a new pipeline service to parallelize dependency resolution, downloading, and cloning, aiming to improve installation speed.
• Updates to Cargo.toml and Cargo.lock reflect the introduction of new dependencies like libdeflater and changes in existing ones to support the new architecture.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-executed refactoring to implement a pipelined dependency installation process, which should greatly improve performance by parallelizing downloads and installations. The introduction of the pipeline service and the use of OnceMap for deduplicating work are excellent changes. However, I have identified a few areas for improvement. The switch to #[tokio::main] removes some potentially important runtime performance tuning. Additionally, the new download and extraction logic in downloader.rs has moved from a streaming to an in-memory model, which could cause high memory usage and potential OOM errors for large packages. I've also suggested a minor structural improvement in the same file to combine blocking operations.

[Copilot]

@xusd320 I've opened a new pull request, #2581, to work on those changes. Once the pull request is ready, I'll request review from you.