Closed
Conversation
This change implements Phase 2 of the TESTING_PLAN.md by adding a NixOS integration test that verifies the end-to-end functionality of Loft. The test runs in a VM with Garage (S3 compatible storage) and Loft. It verifies: 1. Basic upload of a store path. 2. End-to-end cache fetching (substitution) by deleting the local path and rebuilding with `--max-jobs 0` from the Garage bucket. Changes: - Created `nixos/tests/integration.nix`. - Updated `flake.nix` to include the `integration` check, using a dedicated `pkgsForTest` with the Loft overlay. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
Contributor
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
- Add 'rpc_secret' to Garage config in tests - Implement robust node ID extraction for Garage layout - Fix bucket creation and permissions in test script - Add safety timeouts to VM test wait operations - Enable 'force_path_style' in S3 uploader for local S3 compatibility - Optimize builds by cleaning cargo source in flake.nix
- Added `list_all_hashes` to `RemoteCacheStorage` trait. - Added `is_scan_complete` and `set_scan_complete` to `LocalCacheStorage` trait. - Implemented bulk warmup logic in `CacheChecker::check_paths` to fetch and store all remote hashes on startup if the local cache is empty. - Implemented `list_all_hashes` for `S3Uploader` using the existing `list_all_narinfo_keys` method. - Linked `LocalCache`'s `redb` state methods (`is_scan_complete`, `set_scan_complete`) to the `LocalCacheStorage` trait. - Increased default `upload_threads` to 32 in `config.rs` to optimize concurrency for remote and local operations. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Fixes the SQLite connection exhaustion in `s3_uploader.rs` by extracting the Nix store hash from the file path string locally, completely bypassing the need to connect to the Nix daemon and open a new SQLite DB connection for every checked path in highly concurrent map iterations. - Fixes the dangling file descriptor/locking crash bug in `main.rs` by safely using `local_cache.clear()` to clear the cache via a safe `redb` transaction, instead of calling `std::fs::remove_file` on the actively open memory-mapped `redb` database. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Add 'reset_state' helper to sync S3 and local DB state - Add 'Mass Watcher' test using nix-build to trigger inotify correctly - Add 'Cache Recovery' test for 'populateCacheOnStartup' feature - Add 'Handling Inconsistency' test for S3 wipe vs local DB state - Use '--ignore-liveness' for robust store path deletion in tests - Configure dummy nixpkgs in VM for standalone nix-builds
…-11670314188469532262 Refactor Cache Optimization to Bulk-Sync Model
…on-15958177944138370681 fix: resolve SQLite connection exhaustion and open-file deletion bug
…bility
- Pass `LocalCache` to the pruner and explicitly remove hashes from it
when deleting .narinfo files from S3 to prevent cache desync.
- Wrap multipart uploads inside `s3_uploader.rs` in `async {}.await`
blocks, capturing any errors during the `upload_part` or `read` loops
and executing `abort_multipart_upload` on the AWS S3 client to prevent
S3 bucket storage leaks from incomplete uploads.
- Remove hardcoded `/nix/store/` strings in `local_cache.rs` by querying
the active Nix store path directory `NixStore::connect()?.store_dir()`
once upon application startup and caching it in the `LocalCache` struct.
Use this dynamic path as the strip prefix.
- Implement a graceful shutdown mechanism using
`tokio_util::sync::CancellationToken` and `tokio::signal::ctrl_c()`.
Use a `tokio::task::JoinSet` inside the watcher loop to track active
spawns, allowing the process to wait for all currently active S3 uploads
to safely finish and `redb` to flush when terminating the application.
Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Add 'Concurrency' subtest with 30 rapid builds - Add 'skipSignedByKeys' subtest with real Nix path signing - Add 'Pruning' subtest to verify CLI pruner execution - Increase VM disk size to 4GB for larger datasets - Use manual config for CLI tests to ensure robustness
…4582372062740923' into nixos-integration-test-10565240411368607104 # Conflicts: # src/local_cache.rs # src/pruner.rs
- Invert cache check and signature fetching logic to minimize expensive subprocess calls. - Enforce backpressure by acquiring Tokio semaphore permits before spawning tasks. - Avoid manual S3 upload retries by offloading logic to AWS SDK RetryConfig. - Reduce heap memory fragmentation by preferring string slices (`&[String]`) over cloned vectors. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Correct 'watcher_handle' move in 'main.rs' tokio::select! - Fix 'extract_hash_from_path' to be static in 'local_cache.rs' and update all call sites - Remove unused 'NixStore' import in 's3_uploader.rs' - Fix 'local_cache.rs' tests after API changes
…y-13760093137456892221' into nixos-integration-test-10565240411368607104 # Conflicts: # src/cache_checker.rs # src/nix_store_watcher.rs # src/pruner.rs # src/s3_uploader.rs
This introduces a GitHub Actions workflow to run `nix flake check` on push and pull requests to the `main` branch. To ensure unit tests and clippy checks are run, `flake.nix` and `crane.nix` were refactored to use a split-phase build strategy yielding `cargoArtifacts`. These artifacts are passed into `cargoClippy` and `cargoTest` checks. Several preexisting clippy warnings that were preventing the clippy check from passing with the `--deny warnings` flag were resolved. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Implement (1)$ memory streaming uploads using async-compression and S3 multipart upload. - Eliminate nested Tokio runtimes and replace CLI sub-processes with native attic::nix_store::NixStore FFI. - Parallelize metadata queries and closure uploads for significant performance gains. - Batch local cache disk I/O to reduce write transactions and wear. - Optimize S3 pruning to use a single-pass ListObjectsV2 iteration. - Integrate CI workflow with clippy and unit-test flake checks. - Optimize NixOS integration tests by parallelizing stress-test builds and disabling external substituters. - Resolve merge conflicts between performance refactors and CI linting fixes.
The CI pipeline runs `cachix/install-nix-action@v22`, which installs Nix 2.16.1. Recently, `nixos-unstable` introduced a requirement for `builtins.nixVersion` to be at least 2.18, causing the evaluation of `nix flake check -L` to fail in CI. This commit downgrades the `nixpkgs` input from `nixos-unstable` to `nixos-24.05` to restore compatibility with Nix 2.16.1 and fix the CI failure. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
Owner
Author
|
don't downgrade, the previous commit just bumped the nix install to v30. please revert |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implemented Phase 2 of the testing plan by adding a NixOS integration test.
This test verifies the entire system in a realistic, fully automated environment using NixOS's testing framework.
It simulates a real deployment with a real S3 backend (Garage).
Key components:
nixos/tests/integration.nix: Defines the test scenario.flake.nix: Addedchecks.integrationto run the test.The test verifies:
PR created automatically by Jules for task 10565240411368607104 started by @ProjectInitiative