Merged
Conversation
This change implements Phase 2 of the TESTING_PLAN.md by adding a NixOS integration test that verifies the end-to-end functionality of Loft. The test runs in a VM with Garage (S3 compatible storage) and Loft. It verifies: 1. Basic upload of a store path. 2. End-to-end cache fetching (substitution) by deleting the local path and rebuilding with `--max-jobs 0` from the Garage bucket. Changes: - Created `nixos/tests/integration.nix`. - Updated `flake.nix` to include the `integration` check, using a dedicated `pkgsForTest` with the Loft overlay. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Add 'rpc_secret' to Garage config in tests - Implement robust node ID extraction for Garage layout - Fix bucket creation and permissions in test script - Add safety timeouts to VM test wait operations - Enable 'force_path_style' in S3 uploader for local S3 compatibility - Optimize builds by cleaning cargo source in flake.nix
- Added `list_all_hashes` to `RemoteCacheStorage` trait. - Added `is_scan_complete` and `set_scan_complete` to `LocalCacheStorage` trait. - Implemented bulk warmup logic in `CacheChecker::check_paths` to fetch and store all remote hashes on startup if the local cache is empty. - Implemented `list_all_hashes` for `S3Uploader` using the existing `list_all_narinfo_keys` method. - Linked `LocalCache`'s `redb` state methods (`is_scan_complete`, `set_scan_complete`) to the `LocalCacheStorage` trait. - Increased default `upload_threads` to 32 in `config.rs` to optimize concurrency for remote and local operations. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Fixes the SQLite connection exhaustion in `s3_uploader.rs` by extracting the Nix store hash from the file path string locally, completely bypassing the need to connect to the Nix daemon and open a new SQLite DB connection for every checked path in highly concurrent map iterations. - Fixes the dangling file descriptor/locking crash bug in `main.rs` by safely using `local_cache.clear()` to clear the cache via a safe `redb` transaction, instead of calling `std::fs::remove_file` on the actively open memory-mapped `redb` database. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Add 'reset_state' helper to sync S3 and local DB state - Add 'Mass Watcher' test using nix-build to trigger inotify correctly - Add 'Cache Recovery' test for 'populateCacheOnStartup' feature - Add 'Handling Inconsistency' test for S3 wipe vs local DB state - Use '--ignore-liveness' for robust store path deletion in tests - Configure dummy nixpkgs in VM for standalone nix-builds
…-11670314188469532262 Refactor Cache Optimization to Bulk-Sync Model
…on-15958177944138370681 fix: resolve SQLite connection exhaustion and open-file deletion bug
…bility
- Pass `LocalCache` to the pruner and explicitly remove hashes from it
when deleting .narinfo files from S3 to prevent cache desync.
- Wrap multipart uploads inside `s3_uploader.rs` in `async {}.await`
blocks, capturing any errors during the `upload_part` or `read` loops
and executing `abort_multipart_upload` on the AWS S3 client to prevent
S3 bucket storage leaks from incomplete uploads.
- Remove hardcoded `/nix/store/` strings in `local_cache.rs` by querying
the active Nix store path directory `NixStore::connect()?.store_dir()`
once upon application startup and caching it in the `LocalCache` struct.
Use this dynamic path as the strip prefix.
- Implement a graceful shutdown mechanism using
`tokio_util::sync::CancellationToken` and `tokio::signal::ctrl_c()`.
Use a `tokio::task::JoinSet` inside the watcher loop to track active
spawns, allowing the process to wait for all currently active S3 uploads
to safely finish and `redb` to flush when terminating the application.
Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Add 'Concurrency' subtest with 30 rapid builds - Add 'skipSignedByKeys' subtest with real Nix path signing - Add 'Pruning' subtest to verify CLI pruner execution - Increase VM disk size to 4GB for larger datasets - Use manual config for CLI tests to ensure robustness
…4582372062740923' into nixos-integration-test-10565240411368607104 # Conflicts: # src/local_cache.rs # src/pruner.rs
- Invert cache check and signature fetching logic to minimize expensive subprocess calls. - Enforce backpressure by acquiring Tokio semaphore permits before spawning tasks. - Avoid manual S3 upload retries by offloading logic to AWS SDK RetryConfig. - Reduce heap memory fragmentation by preferring string slices (`&[String]`) over cloned vectors. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Correct 'watcher_handle' move in 'main.rs' tokio::select! - Fix 'extract_hash_from_path' to be static in 'local_cache.rs' and update all call sites - Remove unused 'NixStore' import in 's3_uploader.rs' - Fix 'local_cache.rs' tests after API changes
…y-13760093137456892221' into nixos-integration-test-10565240411368607104 # Conflicts: # src/cache_checker.rs # src/nix_store_watcher.rs # src/pruner.rs # src/s3_uploader.rs
This introduces a GitHub Actions workflow to run `nix flake check` on push and pull requests to the `main` branch. To ensure unit tests and clippy checks are run, `flake.nix` and `crane.nix` were refactored to use a split-phase build strategy yielding `cargoArtifacts`. These artifacts are passed into `cargoClippy` and `cargoTest` checks. Several preexisting clippy warnings that were preventing the clippy check from passing with the `--deny warnings` flag were resolved. Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
- Implement (1)$ memory streaming uploads using async-compression and S3 multipart upload. - Eliminate nested Tokio runtimes and replace CLI sub-processes with native attic::nix_store::NixStore FFI. - Parallelize metadata queries and closure uploads for significant performance gains. - Batch local cache disk I/O to reduce write transactions and wear. - Optimize S3 pruning to use a single-pass ListObjectsV2 iteration. - Integrate CI workflow with clippy and unit-test flake checks. - Optimize NixOS integration tests by parallelizing stress-test builds and disabling external substituters. - Resolve merge conflicts between performance refactors and CI linting fixes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.