[8.8.0] Materialize important outputs from remote external repos#29089
Closed
fmeum wants to merge 3 commits into
Closed
[8.8.0] Materialize important outputs from remote external repos#29089fmeum wants to merge 3 commits into
fmeum wants to merge 3 commits into
Conversation
iancha1992
added
the
soft-release-blocker
iancha1992
changed the title
|
Thank you for contributing to the Bazel repository! This pull request has been marked as stale since it has not had any activity in the last 30 days. It will be closed in the next 30 days unless any other activity occurs. If you think this PR is still relevant and should stay open, please post any comment here and the PR will no longer be marked as stale. |
github-actions
Bot
added
the
stale
fmeum
changed the title
github-actions
Bot
removed
the
stale
fmeum
force-pushed
the
rrcc-8.7.0-23
branch
3 times, most recently
from
8b22c28 to
468cbb3
Compare
fmeum
force-pushed
the
rrcc-8.7.0-23
branch
2 times, most recently
from
119e3c8 to
ddbaca0
Compare
github-actions
Bot
added
team-Performance
Fixes bazelbuild#27517 by checking Skyframe deps in batches that stop right before any dep that may cause a cycle if checked while previous deps are out-of-date. This is accompanied by a restructuring of `RepoRecordedInput` that consolidates all Skyframe logic associated with the computation of the corresponding value exclusively within that class. This will also be helpful in adding support for dynamic inputs to the remote repo contents cache in future work. The upstream commit additionally moved the entirety of `RepositoryFetchFunction` to Skyframe workers so that checking the up-to-dateness of local repo contents cache entries isn't quadratic. That worker refactor is omitted in this 8.x backport, which keeps the existing split `RepositoryDelegatorFunction`/`StarlarkRepositoryFunction` architecture and restart-based evaluation; the cycle fix and the `RepoRecordedInput` restructuring are applied on that model. The batched up-to-dateness check is restart-safe (`isAnyValueOutdated` short-circuits while its batch's values are still missing). Closes bazelbuild#28206. Co-authored-by: Xudong Yang <wyverald@gmail.com> PiperOrigin-RevId: 855252657 Change-Id: Ica18760ae79da5155fc0f3d8cd4f24c52a034c86 (cherry picked from commit 72a25a9)
* The cache was always written to, even if not enabled. (On 8.x this guard was already folded into the remote repo contents cache port, bazelbuild#29075, so only the second fix below is applied here.) * Google RBE doesn't accept `Command`s without the (deprecated) `Platform` field set. We set it both on `Command` and `Action`, just to be safe. Fixes bazelbuild#28294 (comment) Closes bazelbuild#28295. PiperOrigin-RevId: 856169835 Change-Id: I2479119a173e325a7d39643a36536569f5f831fc (cherry picked from commit a9946096847e22de98e0e11b1f5dfbb6ec6ecdbb)
Important outputs and runfiles from external repos that are remote repo contents cache hits got stuck at various levels of the materialization pipeline for being source artifacts. This is fixed by consolidating the skip logic in a `RemoteOutputChecker.mayBeRemote` static helper and letting external-repo source artifacts flow through the toplevel-output download path. 8.x adaptation: v9 routes toplevel outputs through `RemoteImportantOutputHandler`, which does not exist on 8.x (`ImportantOutputHandler` has no production implementor). The equivalent toplevel-output materialization on 8.x lives in `CompletionFunction.ensureToplevelArtifacts`/`downloadArtifact`, which previously bailed out on non-`DerivedArtifact`s and only ran under skymeld. This change lets external-repo source artifacts (which have no generating action and are thus never downloaded by `finalizeAction`) flow through that path in both skymeld and non-skymeld builds, so they honor `--remote_download_outputs` just like build outputs do. `AbstractActionInputPrefetcher.prefetchFiles` already only skips main-repo source artifacts, so it is unchanged beyond annotating the `action` parameter `@Nullable` (source artifacts have no generating action). Closes bazelbuild#28308. PiperOrigin-RevId: 881618604 Change-Id: Ifaae8e39b0bcab3803653ca82bcf00d26c487316 (cherry picked from commit 16613f1)
auto-merge was automatically disabled
June 25, 2026 07:47
Head branch was pushed to by a user without write access
Collaborator
Author
|
Merged in 023a3f6 |
auto-merge was automatically disabled
June 26, 2026 11:09
Pull request was closed
github-actions
Bot
removed
the
awaiting-review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important outputs and runfiles from external repos that are remote repo contents cache hits got stuck at various levels of the materialization pipeline for being source artifacts. This is fixed by consolidating the skip logic in a
RemoteOutputChecker.mayBeRemotestatic helper and letting external-repo source artifacts flow through the toplevel-output download path, so they honor--remote_download_outputsjust like build outputs.8.x adaptation: v9 routes toplevel outputs through
RemoteImportantOutputHandler, which does not exist on 8.x (ImportantOutputHandlerhas no production implementor there). The equivalent toplevel-output materialization on 8.x lives inCompletionFunction.ensureToplevelArtifacts/downloadArtifact, which previously bailed out on non-DerivedArtifacts and only ran under skymeld. This backport lets external-repo source artifacts — which have no generating action and are thus never downloaded byfinalizeAction— flow through that path in both skymeld and non-skymeld builds.AbstractActionInputPrefetcher.prefetchFileson 8.x already only skips main-repo source artifacts, so it is unchanged beyond annotating theactionparameter@Nullable.Closes #28308.
PiperOrigin-RevId: 881618604
Change-Id: Ifaae8e39b0bcab3803653ca82bcf00d26c487316
(cherry picked from commit 16613f1)