Repository navigation
Conversation
The upstream `inflight_version<(name, original_spec)>` gate keys on the *original* spec while `store_version_manifest` keys on the *resolved* version. Two sibling specs (`^4.17.0`, `~4.17.20`) resolving to the same `4.17.21` each pass through different gates and would each `tokio::spawn(write_json)` for the same path — duplicate serialize, duplicate disk write, and a brief truncate-race window where a concurrent reader sees partial JSON. Add an `OnceMap<(name, version), ()>` to `DiskManifestStore` and use `register` as a sync atomic "first wins" check. Subsequent callers for the same `(name, resolved_version)` skip the spawn entirely. For ant-design preload (~600 specs → ~250 unique versions), this drops ~350 redundant spawn+serialize+write cycles (~1.75 MB redundant disk write). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a deduplication mechanism for disk writes in DiskManifestStore using a OnceMap to prevent redundant concurrent tasks for the same version manifest. Feedback was provided regarding the error handling in the asynchronous write task; specifically, the operation is marked as complete even if the write fails, which would prevent subsequent retry attempts for that manifest. It is recommended to only mark the operation as complete upon a successful write.
| write_json(&path, &*manifest).await; | ||
| inflight.complete(key, Some(()), notify); |
There was a problem hiding this comment.
The write is currently marked as complete with Some(()) regardless of whether write_json succeeds or fails. Since OnceMap::register returns None for any key already in the Done state, a single failed write (e.g., due to a transient I/O error or disk full condition) will prevent any subsequent attempts to persist that specific version manifest for the lifetime of the process. This results in persistent cache misses for that version.
Consider updating write_json to return a success boolean and using inflight.complete(key, success.then_some(()), notify) here. This ensures that if a write fails, the key is removed from the map (via None), allowing future calls to retry the operation.
|
Superseded by #2874 commit |
Problem
DiskManifestStore::store_version_manifestis called fromservice/registry.rs:388inside theinflight_version<(name, original_spec)>gate. The gate keys on the original spec, but the disk write keys on the resolved version. Two sibling specs (^4.17.0,~4.17.20) resolving to the same4.17.21each pass through different gates and both spawnwrite_jsonfor the same path:Each duplicate does:
serde_json::to_vec(~5 KB × N)store_versions(versions.json) is safe — it sits insideinflight_full<name>whose key matches the write key.Fix
Add an
OnceMap<(name, resolved_version), ()>toDiskManifestStore. Reuse the existingOnceMap::register(sync, atomic "first wins") that ruborist already uses forinflight_full/inflight_version. Subsequent callers for the same(name, version)getNonefromregisterand skip the spawn entirely.Impact
ant-design preload: ~600 specs → ~250 unique versions → ~350 redundant
spawn + serialize + writecycles eliminated, ~1.75 MB redundant disk write avoided. Wall-clock impact within σ (writes are fire-and-forget off the resolve hot path), but reduces CPU/IO contention with preload workers and closes the truncate-race window.Test plan
cargo build --profile release-local -p utoo-pmcleancargo clippy -p utoo-pm -p utoo-ruborist --all-targets -- -D warnings --no-depscleancargo test -p utoo-pm— 245 passed, 0 failed🤖 Generated with Claude Code