Skip to content

The depot cache has a 0% hit rate: N shards save the SAME key, each uploading the whole depot, and the repo quota evicts everything #72

Description

@sotashimozono

Adjacent to #62 but a different defect, and this one hurts hosted consumers too — where #62's
"the depot already persists" argument does not apply. Measured on ParaLinearAlgebra.jl, 2026-08-05.

The cache never hits

Every restore across two runs and every shard inspected logs the same line:

Cache key: testshards;os=Linux;run_id=30961107784;run_attempt=1
No cache found

Not "restored from a prefix key" — nothing. The restore side of this action has been returning
zero value, for every run, while the save side costs 4–5 minutes per shard.

Why: N shards, one key, N full uploads, one 10 GB quota

The key is deliberately shared (cache-name: testshards, include-matrix: 'false') so the shards
do not pay precompilation N times — right in intent. But each shard still uploads the whole
depot
under that one key. The repo's cache store right now:

active caches: 6      total: 11.2 GB      (GitHub's per-repo limit: 10 GB)

1.90 GB  refs/heads/main        testshards;os=Linux;run_id=30962234474;run_attempt=1
1.89 GB  refs/pull/426/merge    testshards;os=Linux;run_id=30961107784;run_attempt=1
1.88 GB  refs/pull/426/merge    testshards;os=Linux;run_id=30961107784;run_attempt=1
1.87 GB  refs/pull/426/merge    testshards;os=Linux;run_id=30961107784;run_attempt=1
1.84 GB  refs/pull/426/merge    testshards;os=Linux;run_id=30961107784;run_attempt=1

Four entries, one run, one key, slightly different sizes — concurrent saves racing, each having
tarred a slightly different depot state. A single 8-shard run can therefore push ~15 GB into a 10 GB
quota by itself, which evicts everything including the entry the next run would have restored. That
is the 0 % hit rate, and it is self-inflicted rather than transient.

And the prune that would have contained it is failing

delete-old-caches: true is set, and its cleanup step errors on every run:

┌ Error: An error occurred while managing existing caches
│    failed process: Process(`gh api -X GET /repos/…/actions/caches …`), ProcessExited(1)
└ @ Main …/julia-actions/cache/v3/handle_caches.jl:88

So nothing is pruned, and the quota stays blown.

What is worth changing (author's call)

  1. Save once, restore everywhere. There is already a prebuild job that runs once per run. If
    the save happened only there — or only from one shard — the upload cost divides by N and the
    racing duplicates disappear. julia-actions/cache has no restore-only switch, but
    actions/cache/restore does, and an if: on the shard-level step is the cheap version.
  2. Stop caching compiled in the shards. This workflow already ships the compiled cache as an
    artifact (prebuild → compiled.tgz → "Adopt the precompiled cache"), so cache-compiled: true
    duplicates a mechanism that exists — and compiled/ is the part that changes every run, i.e. the
    part that never dedupes and drives the 1.9 GB.
  3. Make the prune work (token:), or the quota will keep evicting whatever is saved.
  4. julia-actions/cache@v3.2.0 also takes a gcp-bucket, which removes the 10 GB quota and the
    GitHub uplink entirely — infrastructure this fleet does not have today, but it is the option that
    makes a large depot cacheable at all.

For the self-hosted pool none of this changes #62's conclusion — a perfect hit would restore files
already on the box — which is why #71 turns it off there. This issue is about the case where the
cache is supposed to help.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions