[8.8.0] Use FileArtifactValue#setContentsProxy for remote repo contents cache#29975
Merged
Conversation
github-actions
Bot
added
team-Performance
Fixes bazelbuild#27517 by checking Skyframe deps in batches that stop right before any dep that may cause a cycle if checked while previous deps are out-of-date. This is accompanied by a restructuring of `RepoRecordedInput` that consolidates all Skyframe logic associated with the computation of the corresponding value exclusively within that class. This will also be helpful in adding support for dynamic inputs to the remote repo contents cache in future work. The upstream commit additionally moved the entirety of `RepositoryFetchFunction` to Skyframe workers so that checking the up-to-dateness of local repo contents cache entries isn't quadratic. That worker refactor is omitted in this 8.x backport, which keeps the existing split `RepositoryDelegatorFunction`/`StarlarkRepositoryFunction` architecture and restart-based evaluation; the cycle fix and the `RepoRecordedInput` restructuring are applied on that model. The batched up-to-dateness check is restart-safe (`isAnyValueOutdated` short-circuits while its batch's values are still missing). Closes bazelbuild#28206. Co-authored-by: Xudong Yang <wyverald@gmail.com> PiperOrigin-RevId: 855252657 Change-Id: Ica18760ae79da5155fc0f3d8cd4f24c52a034c86 (cherry picked from commit 72a25a9)
* The cache was always written to, even if not enabled. (On 8.x this guard was already folded into the remote repo contents cache port, bazelbuild#29075, so only the second fix below is applied here.) * Google RBE doesn't accept `Command`s without the (deprecated) `Platform` field set. We set it both on `Command` and `Action`, just to be safe. Fixes bazelbuild#28294 (comment) Closes bazelbuild#28295. PiperOrigin-RevId: 856169835 Change-Id: I2479119a173e325a7d39643a36536569f5f831fc (cherry picked from commit a9946096847e22de98e0e11b1f5dfbb6ec6ecdbb)
Important outputs and runfiles from external repos that are remote repo contents cache hits got stuck at various levels of the materialization pipeline for being source artifacts. This is fixed by consolidating the skip logic in a `RemoteOutputChecker.mayBeRemote` static helper and letting external-repo source artifacts flow through the toplevel-output download path. 8.x adaptation: v9 routes toplevel outputs through `RemoteImportantOutputHandler`, which does not exist on 8.x (`ImportantOutputHandler` has no production implementor). The equivalent toplevel-output materialization on 8.x lives in `CompletionFunction.ensureToplevelArtifacts`/`downloadArtifact`, which previously bailed out on non-`DerivedArtifact`s and only ran under skymeld. This change lets external-repo source artifacts (which have no generating action and are thus never downloaded by `finalizeAction`) flow through that path in both skymeld and non-skymeld builds, so they honor `--remote_download_outputs` just like build outputs do. `AbstractActionInputPrefetcher.prefetchFiles` already only skips main-repo source artifacts, so it is unchanged beyond annotating the `action` parameter `@Nullable` (source artifacts have no generating action). Closes bazelbuild#28308. PiperOrigin-RevId: 881618604 Change-Id: Ifaae8e39b0bcab3803653ca82bcf00d26c487316 (cherry picked from commit 16613f1)
Even though expiration times don't matter in Bazel (they aren't honored), supporting the contents proxy optimization avoids Skyframe invalidation when external repo files are materialized later. Along the way ensure that the persistent action cache always recreates remote metadata via `FileArtifactValue.createForRemoteFileWithMaterializationData`. Before this change, such metadata would roundtrip into a `RemoteFileArtifactValue` (without the content proxy optimization) if `expirationTime` is set to `null`. 8.x adaptation: the remote metadata factories still live on the nested `FileArtifactValue.RemoteFileArtifactValue` class (`create`/`createWithMaterializationData`) rather than the v9 top-level `createForRemoteFile*` overloads. The overlay filesystem therefore injects external repo files via `RemoteFileArtifactValue.createWithMaterializationData` (passing the expiration as epoch millis and a null materialization path), and `CompactPersistentActionCache.decodeRemoteMetadata` drops its early return to the non-materialization `RemoteFileArtifactValue.create` so decoded remote metadata always supports the `FileContentsProxy` optimization. Closes bazelbuild#28654. PiperOrigin-RevId: 884796123 Change-Id: Ib399aff3864f5fbbef230b64a7a5ef619bf854d0 (cherry picked from commit 257a224)
coeuvre
approved these changes
Jun 25, 2026
Merged
via the queue into
bazelbuild:release-8.8.0
with commit Jun 25, 2026
023a3f6
46 checks passed
github-actions
Bot
removed
the
awaiting-review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Even though expiration times don't matter in Bazel (they aren't honored), supporting the contents proxy optimization avoids Skyframe invalidation when external repo files are materialized later.
Along the way ensure that the persistent action cache always recreates remote metadata via the materialization-data variant. Before this change, such metadata would roundtrip into a plain
RemoteFileArtifactValue(without the content proxy optimization) when no expiration time is set.8.x adaptation: the remote metadata factories still live on the nested
FileArtifactValue.RemoteFileArtifactValueclass (create/createWithMaterializationData) rather than the v9 top-levelcreateForRemoteFile*overloads. The overlay filesystem therefore injects external repo files viaRemoteFileArtifactValue.createWithMaterializationData(passing the expiration as epoch millis and a null materialization path), andCompactPersistentActionCache.decodeRemoteMetadatadrops its early return to the non-materializationRemoteFileArtifactValue.createso decoded remote metadata always supports theFileContentsProxyoptimization.Closes #28654.
PiperOrigin-RevId: 884796123
Change-Id: Ib399aff3864f5fbbef230b64a7a5ef619bf854d0
(cherry picked from commit 257a224)