Skip to content

perf(ci): warm root bazel cache on main; rename matrix row to root#280

Merged
kristinapathak merged 7 commits into
mainfrom
ci/rename-root-matrix-row
Jul 21, 2026
Merged

perf(ci): warm root bazel cache on main; rename matrix row to root#280
kristinapathak merged 7 commits into
mainfrom
ci/rename-root-matrix-row

Conversation

@balajinvda

@balajinvda balajinvda commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Why

A PR or merge that builds the root Bazel row compiles the full ~7,000-target
closure. Measured cold, that is ~626s / 5,172 local actions / 0 remote-cache
hits. It stayed cold because the root row only ran on main when a ROOT_GLOB
path changed (roughly 1 in 10 merges), so the root closure was almost never
uploaded to the remote cache. Frequently-touched rows like nvca warmed
naturally (~86% hits); root did not.

What changed

  • Rename the umbrella-phase1 matrix row to root (and UMBRELLA_GLOBS to
    ROOT_GLOBS). The row builds the root module (Go and Java), so root is the
    accurate name; umbrella-phase1 was a rollout-era label.
  • Force the root row to run on every push to main, regardless of whether a
    ROOT_GLOB changed. The build step already uploads results only on main
    pushes, so this warms the whole closure each merge. Per-service rows and PRs
    then read those warm results for the shared closure.
  • On a pull_request, narrow the root row to only the targets a change
    affects: diff the PR against its base, and when every change is a
    modification to an existing .go source, build rdeps(//..., changed)
    instead of the full ~7k-target closure. Conservative by construction (any
    add/rename/delete, global file, non-Go file, or query failure falls back to
    //...; the rdeps query is strict so an unresolvable label forces full).
    Main pushes and per-service rows are unchanged.
  • Echo the resolved remote-cache mode (read-write/read-only, upload=...)
    in the Prepare-remote-cache step, so a missing warm-write shows up in the log
    instead of being an after-the-fact guess.
  • Harden two ${{ github.* }} interpolations into $GITHUB_EVENT_NAME /
    $GITHUB_REF env reads (CodeRabbit shell-injection findings).

PR behavior is unchanged: PRs stay read-only and still run root only when a
ROOT_GLOB changed.

Customer Release Notes

Not customer visible (CI only).

Plan Summary

Not applicable.

Usage

Not applicable.

Testing

YAML parses; every run: block passes bash -n. The warm/read-only mode is now
logged, so the first merge to main after this lands will show upload=true on
the root row and the following PR should show a high remote-cache hit ratio on
the shared closure.

Notes

Both halves of #283 are in this PR: warm root on main (so PRs read a warm
closure) and narrow root PRs to affected targets (so cold PRs touching a
sliver of src/libraries/ do not rebuild everything). The narrow path uses
bazel query rdeps, not target-determinator, so no bazel-ci image change is
needed. This PR itself edits a non-Go file, so its own root build exercises
the full //... path; the narrow path is unit-tested with a stubbed
git/bazel harness.

References

Relates to #283 (warm root on main + narrow root PRs to affected targets).

Related Merge Requests/Pull Requests

None

Dependencies

None

The `umbrella-phase1` id dates from when the root Bazel module only covered
Phase 1 native Go (nvcf-cli, go lib/worker). The root module now builds the
whole root workspace, Go and Java (nv-boot-parent), while the other matrix rows
are per-subtree nested modules. Rename the row to `root` so the check reads
`bazel (root)` and reads correctly next to `bazel (nvca)` etc. Also rename the
UMBRELLA_GLOBS variable to ROOT_GLOBS and de-stale the "Phase 1" comments.

No behavior change: same path (`.`), same tests_skip, same trigger globs.

NOTE FOR MERGE: the status-check name changes from `bazel (umbrella-phase1)` to
`bazel (root)`. If `bazel (umbrella-phase1)` is a required check on main, update
the branch-protection required-check name to `bazel (root)` in lockstep, or PRs
will wait on a check that no longer reports.

Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@balajinvda
balajinvda requested a review from a team as a code owner July 21, 2026 01:04
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The Bazel workflow detect job now selects the root workspace, uses root-module globs for build inclusion, and gates remote cache uploads using shell environment variables for pushes to refs/heads/main.

Changes

Root workspace detection

Layer / File(s) Summary
Root workspace selection and glob matching
.github/workflows/bazel.yml
The detect job changes the selected subtree identifier, defines ROOT_GLOBS, and uses it for root-module build matching.
Cache upload gating
.github/workflows/bazel.yml
The workflow uses $GITHUB_EVENT_NAME and $GITHUB_REF to enable remote cache uploads only for pushes to refs/heads/main.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: kristinapathak

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main changes: renaming the Bazel matrix row to root and adjusting cache warming on main.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/rename-root-matrix-row

Comment @coderabbitai help to get the list of available commands.

Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/bazel.yml:
- Around line 243-245: Update the conditional in the cache-upload step to avoid
interpolating github.event_name, github.ref, and github.head_ref directly into
shell syntax. Expose these GitHub context values through environment variables
or evaluate the branch condition with a workflow-level expression, then compare
safely within the script while preserving the existing push-to-main and
ci/rename-root-matrix-row behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 92f6b75d-3ae9-4af2-b9fa-734903fd2dfd

📥 Commits

Reviewing files that changed from the base of the PR and between e5b94d9 and cce057c.

📒 Files selected for processing (1)
  • .github/workflows/bazel.yml

Comment thread .github/workflows/bazel.yml Outdated
Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@balajinvda

Copy link
Copy Markdown
Contributor Author

Fixed: github.head_ref is now passed through the job-level HEAD_REF env var and compared as $HEAD_REF, so the attacker-controlled branch name is never interpolated into the shell (no injection, no CACHE_TOKEN exposure). Note this whole || $HEAD_REF = ... clause is a TEMP cache-warm proof and will be reverted before merge.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/bazel.yml:
- Around line 246-248: Update the conditional in the workflow’s cache-warm
upload block to stop interpolating github.event_name and github.ref directly
into shell source. Use environment variables for both GitHub context values and
compare those variables safely at runtime, while preserving the existing
push-to-main or HEAD_REF test-branch behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1e84b9bb-574e-4739-9fb8-2d2cc6d26657

📥 Commits

Reviewing files that changed from the base of the PR and between cce057c and 0508f21.

📒 Files selected for processing (1)
  • .github/workflows/bazel.yml

Comment thread .github/workflows/bazel.yml Outdated
…polation (coderabbit)

Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@balajinvda

Copy link
Copy Markdown
Contributor Author

Fixed: the condition now uses the auto-provided $GITHUB_EVENT_NAME / $GITHUB_REF env vars instead of ${{ github.* }} interpolation, so no GitHub context is expanded into the shell source. (Reminder: this whole clause is a TEMP cache-warm proof and reverts before merge.)

balajinvda and others added 2 commits July 20, 2026 21:43
The HEAD_REF upload override was a diagnostic to prove root cache
warm-persistence; it is not part of the matrix rename. Root builds
remain read-only on PRs and upload only on push to main.

Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The root matrix row builds the full ~7k-target closure but only ran on
main when a ROOT_GLOB path changed (~1 in 10 merges), so the root closure
was almost never uploaded to the remote cache. PRs and later merges then
paid the full cold build (~626s, 5172 local actions, 0 hits) every time,
while frequently-touched rows like nvca stayed warm (~86% hits).

Force the root row to run on every push to main so the build step uploads
the whole closure each merge. Per-service rows and PRs read those warm
results for the shared Go/Java closure. PR behavior is unchanged: PRs stay
read-only and root still runs on PRs only when a ROOT_GLOB changed.

Also echo the resolved cache mode (read-write vs read-only, upload=...)
in the Prepare-remote-cache step so a missing warm-write is visible in the
log instead of an after-the-fact guess.

Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@balajinvda balajinvda changed the title ci(bazel): rename the root bazel matrix row umbrella-phase1 -> root perf(ci): warm root bazel cache on main; rename matrix row to root Jul 21, 2026
Part 2 of #283. On a pull_request, the root row builds the full
~7k-target closure even for a one-file change. Narrow it: diff the PR
against its base, and if every change is a modification to an existing
.go source, build only rdeps(//..., changed) instead of //....

Safety (cannot under-build, only over-build):
- Only the root row on pull_request is a narrowing candidate. Main pushes
  (which warm+upload the whole closure), dispatch, schedule, and every
  per-service row build //... unchanged.
- Any add/delete/rename, any global file (MODULE.bazel, .bazelrc, *.bzl,
  go.work, ...), any non-Go file, an unavailable base, or a failed diff
  falls back to //....
- The rdeps query runs without --keep_going, so an unresolvable changed-file
  label forces the full build rather than silently dropping a target.
- The test step selects tests(set(affected)); a tests() query failure falls
  back to the full test set.

Requires fetch-depth: 0 on the bazel-job checkout to diff against the base.
Verified with a stubbed git/bazel harness across narrow, added-file, global,
non-Go, main-push, non-root, query-fail, and empty cases.

Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kristinapathak
kristinapathak added this pull request to the merge queue Jul 21, 2026
Merged via the queue into main with commit 3cc61dc Jul 21, 2026
26 checks passed
@kristinapathak
kristinapathak deleted the ci/rename-root-matrix-row branch July 21, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants