Skip to content

[8.8.0] Add a remote repo contents cache#29075

Merged
Wyverald merged 5 commits into
bazelbuild:release-8.8.0from
fmeum:rrcc-8.7.0-09
Jun 18, 2026
Merged

[8.8.0] Add a remote repo contents cache#29075
Wyverald merged 5 commits into
bazelbuild:release-8.8.0from
fmeum:rrcc-8.7.0-09

Conversation

@fmeum

@fmeum fmeum commented Mar 24, 2026

Copy link
Copy Markdown
Collaborator

Contains the following adaptions required for Bazel 8:

  • DetailedIOException propagation had to be added to ensure that rewinding works correctly
  • Since the path of the execroot depends on the name set in WORKSPACE (gone in Bazel 9), AbstractActionInputPrefetcher needed a combination of a lazy Supplier for the path as well as moving the temp directory to the output base, whose path is known at startup.

Original commit info follows:

Repositories are cached in a regular remote cache as AC entries for a synthetic command with the predeclared input hash as the salt. The contents are represented as an output file for the marker file and an output directory for the contents.

Upon a cache hit, the metadata of the files comprising the repository is downloaded and injected into an in-memory file system that is overlaid on the external directory on the native file system. Downloads of file contents only occur when Bazel needs to read a file (e.g., a BUILD or .bzl file) or if a file is an input to an action executed locally. This can save time taken to execute repo rules and compute file digests and disk space required to store the contents of external repositories.

The output of du -h $(bazel info output_base) after bazel build //src:bazel-dev and a fully up-to-date remote cache

  • without the flag:
$ bazel info peak-heap-size used-heap-size-after-gc
peak-heap-size: 271MB
used-heap-size-after-gc: 123MB
$ du -h $(bazel info output_base) | tail -1
1.3G	/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722
  • with the flag:
$ bazel info peak-heap-size used-heap-size-after-gc
peak-heap-size: 266MB
used-heap-size-after-gc: 120MB
$ du -h $(bazel info output_base) | tail -1
380M	/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722

Some repos are still materialized eagerly, which may not be necessary. Patched http_archive also can't be cached yet, so these numbers are likely to improve with further work on this feature.

Fixes #6359

RELNOTES[NEW]: The results of reproducible repository rules without dependencies added at runtime (e.g., via repository_ctx.watch or .getenv) can now be cached in a regular HTTP or gRPC remote cache if the new --experimental_remote_repo_contents_cache startup option is provided.

Closes #26860.

PiperOrigin-RevId: 827569755
Change-Id: I3a4e20c8c717c8503b5eb43545a8db162daef585
(cherry-picked from b8589c3, also includes 4f46e82)

@iancha1992 iancha1992 added this to the 8.7.0 release blockers milestone Mar 27, 2026
@iancha1992 iancha1992 added the soft-release-blocker Soft release blockers that are nice to have, but shouldn't block the release if it's the last one. label Apr 24, 2026
@iancha1992 iancha1992 changed the title [8.7.0] 09/23: Add a remote repo contents cache [8.8.0] 09/23: Add a remote repo contents cache May 7, 2026
@iancha1992 iancha1992 changed the base branch from release-8.7.0 to release-8.8.0 May 7, 2026 18:02
@fmeum fmeum changed the title [8.8.0] 09/23: Add a remote repo contents cache [8.8.0] Add a remote repo contents cache Jun 2, 2026
@fmeum fmeum force-pushed the rrcc-8.7.0-09 branch 6 times, most recently from b055d93 to 967de9a Compare June 6, 2026 22:34
@fmeum fmeum force-pushed the rrcc-8.7.0-09 branch 3 times, most recently from c845c2e to 635532b Compare June 11, 2026 07:33
@fmeum fmeum marked this pull request as ready for review June 11, 2026 07:36
@fmeum fmeum requested a review from a team as a code owner June 11, 2026 07:36
@fmeum fmeum requested a review from Wyverald June 11, 2026 07:36
@github-actions github-actions Bot added team-Performance Issues for Performance teams team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. team-Rules-CPP Issues for C++ rules team-Rules-Server Issues for serverside rules included with Bazel team-Core Skyframe, bazel query, BEP, options parsing, bazelrc team-Remote-Exec Issues and PRs for the Execution (Remote) team awaiting-review PR is awaiting review from an assigned reviewer labels Jun 11, 2026
@iancha1992 iancha1992 enabled auto-merge June 11, 2026 18:04
Makes it possible to build `FileSystem`s composed out of existing `FileSystem`s outside the `vfs` package. Some methods already had their visibility upgraded for this purpose.

Closes bazelbuild#26796.

PiperOrigin-RevId: 798226663
Change-Id: Ic9e4cdefadc7ad8d50a5b69df91837c99c5f6c64
(cherry picked from commit 4f46e82)
@fmeum

fmeum commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator Author

I realized that this lacks one more cherry-pick that simplifies it, will send it inline and mark as RFR when done.

@fmeum fmeum marked this pull request as draft June 15, 2026 13:19
auto-merge was automatically disabled June 15, 2026 13:19

Pull request was converted to draft

fmeum added 3 commits June 15, 2026 15:28
(cherry picked from commit b8589c3)
(cherry picked from commit f741d36)
@fmeum fmeum force-pushed the rrcc-8.7.0-09 branch 3 times, most recently from 5b14e05 to 9aad6ba Compare June 15, 2026 14:15
@fmeum fmeum marked this pull request as ready for review June 15, 2026 14:15
@fmeum

fmeum commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator Author

Ready again, I included 4f46e82

@github-actions github-actions Bot added area-Windows Windows-specific issues and feature requests team-Configurability platforms, toolchains, cquery, select(), config transitions labels Jun 15, 2026
@fmeum

fmeum commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator Author

@iancha1992

@iancha1992 iancha1992 enabled auto-merge June 15, 2026 18:39
auto-merge was automatically disabled June 17, 2026 08:52

Head branch was pushed to by a user without write access

@Wyverald Wyverald added this pull request to the merge queue Jun 18, 2026
Merged via the queue into bazelbuild:release-8.8.0 with commit 3e991ca Jun 18, 2026
46 checks passed
@github-actions github-actions Bot removed the awaiting-review PR is awaiting review from an assigned reviewer label Jun 18, 2026
@fmeum fmeum deleted the rrcc-8.7.0-09 branch June 19, 2026 07:11
fmeum added a commit to fmeum/bazel that referenced this pull request Jun 23, 2026
* The cache was always written to, even if not enabled. (On 8.x this guard was already folded into the remote repo contents cache port, bazelbuild#29075, so only the second fix below is applied here.)
* Google RBE doesn't accept `Command`s without the (deprecated) `Platform` field set. We set it both on `Command` and `Action`, just to be safe.

Fixes bazelbuild#28294 (comment)

Closes bazelbuild#28295.

PiperOrigin-RevId: 856169835
Change-Id: I2479119a173e325a7d39643a36536569f5f831fc
(cherry picked from commit a9946096847e22de98e0e11b1f5dfbb6ec6ecdbb)
fmeum added a commit to fmeum/bazel that referenced this pull request Jun 24, 2026
* The cache was always written to, even if not enabled. (On 8.x this guard was already folded into the remote repo contents cache port, bazelbuild#29075, so only the second fix below is applied here.)
* Google RBE doesn't accept `Command`s without the (deprecated) `Platform` field set. We set it both on `Command` and `Action`, just to be safe.

Fixes bazelbuild#28294 (comment)

Closes bazelbuild#28295.

PiperOrigin-RevId: 856169835
Change-Id: I2479119a173e325a7d39643a36536569f5f831fc
(cherry picked from commit a9946096847e22de98e0e11b1f5dfbb6ec6ecdbb)
fmeum added a commit to fmeum/bazel that referenced this pull request Jun 25, 2026
* The cache was always written to, even if not enabled. (On 8.x this guard was already folded into the remote repo contents cache port, bazelbuild#29075, so only the second fix below is applied here.)
* Google RBE doesn't accept `Command`s without the (deprecated) `Platform` field set. We set it both on `Command` and `Action`, just to be safe.

Fixes bazelbuild#28294 (comment)

Closes bazelbuild#28295.

PiperOrigin-RevId: 856169835
Change-Id: I2479119a173e325a7d39643a36536569f5f831fc
(cherry picked from commit a9946096847e22de98e0e11b1f5dfbb6ec6ecdbb)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-Windows Windows-specific issues and feature requests soft-release-blocker Soft release blockers that are nice to have, but shouldn't block the release if it's the last one. team-Configurability platforms, toolchains, cquery, select(), config transitions team-Core Skyframe, bazel query, BEP, options parsing, bazelrc team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. team-Performance Issues for Performance teams team-Remote-Exec Issues and PRs for the Execution (Remote) team team-Rules-CPP Issues for C++ rules team-Rules-Server Issues for serverside rules included with Bazel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants