Use setup bazel gha - #166
Conversation
|
+a:@jwnimmer-tri (trying again after permissions were updated) |
jwnimmer-tri
left a comment
There was a problem hiding this comment.
Fyi In the GHA Actions page, I clicked "re-run all jobs". The re-launched bazel_test job is here: https://github.com/RobotLocomotion/drake-blender/actions/runs/20720901652/job/59585025755. Notice the final output Executed 1 out of 18 tests: 18 tests pass.. That's pretty good (the expensive server tests were cached). It's a bit concerning that there was 1 cache miss (//test:pycodestyle_py_lint_test) but I don't think that's worth investigating.
@jwnimmer-tri reviewed 3 files and all commit messages, and made 3 comments.
Reviewable status: 2 unresolved discussions, LGTM missing from assignee jwnimmer-tri (waiting on @tom-osika).
.github/workflows/main.yml line 48 at r1 (raw file):
- uses: actions/cache/restore@v5 with: path: ~/.cache/pip
In the "install_test" job below, we use setup-python and set "cache: pip" and that gives us pip caching.
It is possible that we could use that same technique here, instead of doing a manual pip cache? That would be a lot simpler.
.github/workflows/main.yml line 74 at r1 (raw file):
# This dumps configuration details to the log. bazelrc: common --announce_rc=yes - name: Report cache sizes
nit Ditto -- we can drop this now.
tom-osika
left a comment
There was a problem hiding this comment.
Thanks for catching that. Its up to you, I can either look into it more before we merge or I can create a ticket for later.
@tom-osika made 2 comments.
Reviewable status: 2 unresolved discussions, LGTM missing from assignee jwnimmer-tri (waiting on @jwnimmer-tri).
.github/workflows/main.yml line 48 at r1 (raw file):
Previously, jwnimmer-tri (Jeremy Nimmer) wrote…
In the "install_test" job below, we use setup-python and set "cache: pip" and that gives us pip caching.
It is possible that we could use that same technique here, instead of doing a manual pip cache? That would be a lot simpler.
Yes, I think so. According to this, the setup-python action will compute the hash of the dependency file (either pyproject.toml or requirements.txt) and use that as part of the cache key. For pip, the action will cache the global cache directory
Would you like me to switch it?
561ab3d to
0c4d54b
Compare
jwnimmer-tri
left a comment
There was a problem hiding this comment.
@jwnimmer-tri reviewed all commit messages, made 1 comment, and resolved 1 discussion.
Reviewable status: 1 unresolved discussion, LGTM missing from assignee jwnimmer-tri (waiting on @tom-osika).
.github/workflows/main.yml line 48 at r1 (raw file):
Previously, tom-osika (Tom Osika) wrote…
Yes, I think so. According to this, the
setup-pythonaction will compute the hash of the dependency file (eitherpyproject.tomlorrequirements.txt) and use that as part of the cache key. Forpip, the action will cache the global cache directoryWould you like me to switch it?
Yes, that sounds like a good idea.
e72bb29 to
7bca0a6
Compare
tom-osika
left a comment
There was a problem hiding this comment.
@tom-osika made 1 comment.
Reviewable status: 1 unresolved discussion, LGTM missing from assignee jwnimmer-tri (waiting on @jwnimmer-tri).
.github/workflows/main.yml line 48 at r1 (raw file):
Previously, jwnimmer-tri (Jeremy Nimmer) wrote…
Yes, that sounds like a good idea.
Hmm, for some reason using setup-python caching is causing all of the 18 tests to run instead of the 1/18 we saw before.... Investigating now.
f0cadde to
51a6ef3
Compare
tom-osika
left a comment
There was a problem hiding this comment.
@tom-osika made 1 comment.
Reviewable status: 1 unresolved discussion, LGTM missing from assignee jwnimmer-tri (waiting on @jwnimmer-tri).
.github/workflows/main.yml line 48 at r1 (raw file):
Previously, tom-osika (Tom Osika) wrote…
Hmm, for some reason using setup-python caching is causing all of the 18 tests to run instead of the 1/18 we saw before.... Investigating now.
If I remove the setup-python caching altogether, the tests still don't get cached. i.e., if setup-python is configured like this:
- uses: actions/setup-python@v6
with:
python-version: "3.11"All of the tests run. This makes me think that the python version that setup-python is using is interfering with bazel / somehow causing the tests to no longer be hermetic. Looking more into it...
jwnimmer-tri
left a comment
There was a problem hiding this comment.
@jwnimmer-tri reviewed all commit messages.
Reviewable status: 1 unresolved discussion, LGTM missing from assignee jwnimmer-tri (waiting on @tom-osika).
jwnimmer-tri
left a comment
There was a problem hiding this comment.
@jwnimmer-tri reviewed all commit messages.
Reviewable status: 1 unresolved discussion, LGTM missing from assignee jwnimmer-tri (waiting on @tom-osika).
b419ca4 to
0780f12
Compare
0780f12 to
2ed8768
Compare
tom-osika
left a comment
There was a problem hiding this comment.
@tom-osika made 1 comment.
Reviewable status: 1 unresolved discussion, LGTM missing from assignee jwnimmer-tri (waiting on @jwnimmer-tri).
.github/workflows/main.yml line 48 at r1 (raw file):
Previously, tom-osika (Tom Osika) wrote…
If I remove the setup-python caching altogether, the tests still don't get cached. i.e., if setup-python is configured like this:
- uses: actions/setup-python@v6 with: python-version: "3.11"All of the tests run. This makes me think that the python version that
setup-pythonis using is interfering with bazel / somehow causing the tests to no longer be hermetic. Looking more into it...
Update on this: I don't think that the pip caching we originally had was necessary for the bazel_test job. As far as I understand, bazel manages all of the dependencies itself in a totally isolated, hermetic way, so the packages we were saving / restoring from the pip cache ourselves were irrelevant.
If I check out the $(bazel info output_base)/external/ directory, you can find subdirectories for the python dependencies. AFAIK this is totally separate from the normal pip cache.
I removed the pip caching from the bazel_test job and the runtime stayed the same. See here: https://github.com/RobotLocomotion/drake-blender/actions/runs/21525900666/job/62029310737
I re-ran the job and saw 0 tests run, instead of the 1/18 we were occasionally seeing before: https://github.com/RobotLocomotion/drake-blender/actions/runs/21525900666/job/62033212206
This is ready for re-review, @jwnimmer-tri
jwnimmer-tri
left a comment
There was a problem hiding this comment.
@jwnimmer-tri reviewed 2 files and all commit messages, made 1 comment, and resolved 1 discussion.
Reviewable status:complete! all discussions resolved, LGTM from assignee jwnimmer-tri (waiting on @tom-osika).
Towards #111
The first run shows the expected cache misses during the
Run bazel-contrib/setup-bazel@0.15.0GHA step here. The second run shows cache hits hereJust like RobotLocomotion/models#111, I added a temporary commit modifying the MODULE.bazel to ensure that the repository caching still works even when modifying this file (which, be default, is used as part of the restore key).
This change is