[8.8.0] Include rctx.os.{name,arch} in the pre declared inputs hash#30027
Open
fmeum wants to merge 3 commits into
Open
[8.8.0] Include rctx.os.{name,arch} in the pre declared inputs hash#30027fmeum wants to merge 3 commits into
rctx.os.{name,arch} in the pre declared inputs hash#30027fmeum wants to merge 3 commits into
Conversation
With this change, all reproducible repository rules can now be cached in a disk or remote cache, including those with dependencies recorded dynamically during evaluation. This is made possible by introducing a new intermediate type of synthetic AC entries. When looking up the predeclared inputs hash for a repo rule with dynamic dependencies, the action result for such an intermediate entry lists one or more sets of inputs (e.g. a particular file in another repo or an environment variable name). These inputs are then requested from Skyframe and their current values are hashed to obtain the key of the next AC entry, which is again either an intermediate entry or a final entry containing the contents of the repository. 8.x: adapted to the split RepositoryDelegatorFunction/StarlarkRepositoryFunction restart model (there is no merged RepositoryFetchFunction worker on 8.x). The cache lookup is wired through RepositoryDelegatorFunction, which returns null to trigger a Skyframe restart when lookupCache reports missing values via env.valuesMissing(). Since DigestWriter is a nested class of RepositoryDelegatorFunction on 8.x rather than a standalone target, the marker file is parsed inline in RemoteRepoContentsCacheImpl via RepoRecordedInput.WithValue.parse to avoid a dependency cycle between the remote and repository rule libraries. RELNOTES: The remote repo contents cache now supports all reproducible repo rules. Closes bazelbuild#27634. PiperOrigin-RevId: 889750228 Change-Id: I9c7e4fed9d86432a85a96b3318f6eccc9c0558eb (cherry picked from commit ffebc5b)
On 8.x the remote repo contents cache lookup runs in RepositoryDelegatorFunction's restart-based compute (there is no merged repository fetch worker). Looking up a repository with dynamically recorded inputs walks a DAG of action cache entries, requesting the current values of recorded inputs from Skyframe along the way. Each such request can trigger a Skyframe restart, after which the lookup re-walks the DAG from the root and re-downloads every action cache entry it already fetched, turning an O(depth) lookup into O(depth^2) remote round-trips. Memoize the action cache entries fetched during a lookup, keyed by the input hash under which they were looked up, in a SkyKeyComputeState that survives restarts. Action cache entries don't change within a command, so a memoized entry can be reused on the next restart instead of being re-downloaded. The repository's compute-state slot is already occupied by StarlarkRepositoryFunction's worker state (created by wasJustFetched before the lookup runs), and Skyframe stores a single state instance per key. The lookup therefore cannot allocate its own SkyKeyComputeState via env.getState. Instead, the memo is stored on the existing state and handed to lookupCache through a new RepositoryFunction.getRemoteRepoContentsCacheLookupState hook; the default returns a fresh, non-memoizing instance for handlers without a compute state. Follow-up to the dynamic inputs support (cherry-pick of bazelbuild#27634).
This info is accessible via `rctx.os.{name,arch}` and can also influence the
result of a repo rule in subtle ways (e.g. behavior of host tools, line
breaks, etc), so it is now mixed into the predeclared inputs hash used as the
key for the local and remote repo contents cache.
8.x adaptation: `DigestWriter` is a nested class of `RepositoryDelegatorFunction`
rather than a standalone target, and the predeclared inputs hash is computed in
`computePredeclaredInputHash` from the serialized `Rule` instead of a
`RepoDefinition`. The host OS name and CPU architecture are therefore mixed into
that fingerprint right before the environment variable inputs.
Closes bazelbuild#29148.
PiperOrigin-RevId: 893352625
Change-Id: I02c0ded7ffe2b5aa9bc2ef489dc5240b5716ebdf
(cherry picked from commit 2ec24b4)
This was referenced Jun 26, 2026
github-actions
Bot
added
team-ExternalDeps
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This info is accessible via
rctx.os.{name,arch}and can also influence the result of a repo rule in subtle ways (e.g. behavior of host tools, line breaks, etc), so it is now mixed into the predeclared inputs hash used as the key for the local and remote repo contents cache.8.x adaptation:
DigestWriteris a nested class ofRepositoryDelegatorFunctionrather than a standalone target, and the predeclared inputs hash is computed incomputePredeclaredInputHashfrom the serializedRuleinstead of aRepoDefinition. The host OS name and CPU architecture are therefore mixed into that fingerprint right before the environment variable inputs.Closes #29148.
PiperOrigin-RevId: 893352625
Change-Id: I02c0ded7ffe2b5aa9bc2ef489dc5240b5716ebdf
(cherry picked from commit 2ec24b4)