fix(arc): raise sccache cache size to 50G on ak-ci-runners#163
Open
brandonrc wants to merge 2 commits into
Open
fix(arc): raise sccache cache size to 50G on ak-ci-runners#163brandonrc wants to merge 2 commits into
brandonrc wants to merge 2 commits into
Conversation
The shared sccache hostPath was capped at sccache's 10 GiB default (SCCACHE_CACHE_SIZE unset) and verified at exactly 10G on 2026-06-12 — continuously evicting and degrading hit rates. Baseline measurement: ~30 cold rust builds/week at ~12m penalty each, ~22 runner-hours/week lost to cache churn across GHA quota + local sccache eviction. Also adds SCCACHE_ERROR_LOG so compile-cache failures are observable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Missing linked issueThis PR does not reference a tracking issue in its body. Every PR must link to an issue in this repository so we can trace work back to a planned change. How to fix
Accepted keywords (case-insensitive, any tense): Policy reference: see the PR template. Maintainer bypass: apply the |
/cache (hostPath root) is root:755; only the subdirs get chmod 777 from init-cache. sccache could not create /cache/sccache-error.log, the server died at startup, and every compile job on ak-ci-runners failed with 'Timed out waiting for server startup' (~04:00Z 2026-06-12). Coverage stayed green only because it sets RUSTC_WRAPPER=''.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
SCCACHE_CACHE_SIZEwas never set on theak-ci-runnersscale set, so sccache used its 10 GiB default. The shared/home/runner-cache/sccachehostPath was verified at exactly 10G on 2026-06-12 — at cap and continuously evicting.Baseline (measured 2026-06-12, last 33h of CI / 595 runs over 7 days)
Change
SCCACHE_CACHE_SIZE: 50G(/home has 1.2T free)SCCACHE_ERROR_LOG: /cache/sccache-error.logfor observabilityApply
Per the file header:
helm upgrade --install ak-ci-runners --namespace arc-runners --version 0.13.1 -f argocd/arc-ci-runners-values.yaml oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-setNote: this upgrade also deploys the iac#83
init-dockerhub-credsinitContainer already in git but not yet on the cluster (rev 24 predates it).dockerhub-pullsecret anddind-registry-mirrorconfigmap verified present inarc-runners.Companion PR (buildkit type=gha → GHCR registry cache) follows in artifact-keeper/artifact-keeper.
🤖 Generated with Claude Code