Cache intermediates when build fails

[dprotaso]

I'm trying to build a fork of https://github.com/envoyproxy/envoy/ in my own repository. I'm hoping to use this action so that caching can improve build times.

I'm hitting timeouts on GitHub's hosted runners - retriggering the job I would assume the units that were compiled in earlier runs would have been cached but I notice they are not.

I'm using the following config

    - uses: bazel-contrib/setup-bazel@0.14.0
      with:
        bazelisk-cache: true
        repository-cache: true
        external-cache: true
        disk-cache: ${{ github.workflow }}

Does this action save the cache on failed workflows?

[dprotaso]

I'm also seeing the follow messages

Failed to save: Failed to CreateCacheEntry: Received non-retryable error: Failed request: (409) Conflict: cache entry with the same key, version, and scope already exists

[p0deje]

Does this action save the cache on failed workflows?

Yes, as far as I recall, GitHub Actions would call post.js even if it fails, and the action should save whatever caches are available.

I'm also seeing the follow messages
Failed to save: Failed to CreateCacheEntry: Received non-retryable error: Failed request: (409) Conflict: cache entry with the same key, version, and scope already exists

Can you share links to workflow runs you are seeing issues with? There is currently an issue when multiple jobs attempt to upload the same cache, addressed in #71

[dprotaso]

https://github.com/pintail-dev/istio-proxy/actions/runs/13913186022

[p0deje]

I see, it's being addressed in #71 as I mentioned.

[dprotaso]

Any work arounds? I could also test Caleb's fork

[p0deje]

Apart from 409 warnings, the caches are saved in your builds and should be usable - https://github.com/pintail-dev/istio-proxy/actions/caches

[dprotaso]

it's odd then that I see things being compiled again - let me retry the last action run

[dprotaso]

Ok - i see the cache being restored. Are there some bazel commands to help with introspection

 Cache hit for: setup-bazel-1-linux-repository-
  Received 25165824 of 902240816 (2.8%), 24.0 MBs/sec
  Received 130023424 of 902240816 (14.4%), 61.9 MBs/sec
  Received 218103808 of 902240816 (24.2%), 69.3 MBs/sec
  Received 268435456 of 902240816 (29.8%), 64.0 MBs/sec
  Received 398458880 of 902240816 (44.2%), 75.9 MBs/sec
  Received 461373440 of 902240816 (51.1%), 73.3 MBs/sec
  Received 536870912 of 902240816 (59.5%), 73.1 MBs/sec
  Received 666894336 of 902240816 (73.9%), 79.4 MBs/sec
  Received 746586112 of 902240816 (82.7%), 79.0 MBs/sec
  Received 805306368 of 902240816 (89.3%), 76.7 MBs/sec
  Received 902240816 of 902240816 (100.0%), 80.9 MBs/sec
  Cache Size: ~860 MB (902240816 B)

[dprotaso]

For context - I'm building arm and x86

I think I'm not qualifying the arm build in my disk cache key - I'm assuming that would help - is there other stuff I should do?

https://github.com/pintail-dev/istio-proxy/blob/315b13e5cf1621147c8bb7eb84d12b6c6e130916/.github/workflows/release.yaml#L41

[p0deje]

Oh, I don't think our cache key currently uses the platform, so maybe the cache is populated incorrectly. Can you try using and see if it helps?

 disk-cache: ${{ runner.arch }}

[dprotaso]

So I setup some self hosted runners (just standard ubuntu 24LTS)

I have my workflow disk-cache set to

disk-cache: ${{ github.workflow }}-arm

and

disk-cache: ${{ github.workflow }}

but now I'm getting warnings

Error: Path Validation Error: Path(s) specified in the action for caching do(es) not exist, hence no cache is being saved.
    at /home/runner/actions-runner/_work/_actions/bazel-contrib/setup-bazel/0.14.0/dist/node_modules/@actions/cache/lib/cache.js:384:1
    at Generator.next (<anonymous>)
    at fulfilled (/home/runner/actions-runner/_work/_actions/bazel-contrib/setup-bazel/0.14.0/dist/node_modules/@actions/cache/lib/cache.js:28:1)

[karl3wm]

I'm also seeing the follow messages

Failed to save: Failed to CreateCacheEntry: Received non-retryable error: Failed request: (409) Conflict: cache entry with the same key, version, and scope already exists

This indicates reuse of a cache key; cache keys are write-only once.
I am not sure how this action works, but the workaround I found was to load a prefix key since keys are prefix matched, and write to a key suffixed with the run id and run attempt. I also remove all other cache keys (because it's random which one it loads).

My code for deciding cache keys presently looks somewhat like:

      - name: Plan caching
        id: caching
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          sudo apt-get install --yes --quiet jq coreutils || brew install jq
          KEY_PREFIX="bazel-cache-os${{ runner.os }}-tfv${{ matrix.tensorflow }}-py${{ steps.setup_python.outputs.python-version }}-${{ matrix.acceleration }}"
          CACHES_API="/repos/${{ github.repository }}/actions/caches"
          LATEST_KEY="$(gh api "${CACHES_API}"?key="$KEY_PREFIX"\&sort=created_at | jq -r '.actions_caches[0].key')"
          gh api "${CACHES_API}"?key="$KEY_PREFIX"\&sort=created_at | jq -r '.actions_caches[1:30][].key' | while read old_key
          do
                echo "Deleting key $old_key"
                gh api --method DELETE "${CACHES_API}"?key="$old_key"
          done
          NEXT_KEY="${KEY_PREFIX}-run${{ github.run_id }}.${{ github.run_attempt }}"
          echo "Load key: $LATEST_KEY"
          echo "Save key: $NEXT_KEY"
          echo "lastkey=$LATEST_KEY" >> $GITHUB_OUTPUT
          echo "nextkey=$NEXT_KEY" >> $GITHUB_OUTPUT

Then when restoring from the cache, it uses lastkey, and when saving the cache, it uses nextkey.

I am using this code to terminate bazel before cache save, which otherwise can fail.

      - name: Always Shutdown Bazel Before Caching
        if: always()
        working-directory: tensorflow
        run: |
          killall bazel
          bazel --block_for_lock shutdown
          sleep 5

For me it looks like the problem with the missing directories was using the wrong directories for my platform and/or specifying the --disk_cache option differently and/or bazel failing to launch.