feat(release): incremental combos — skip unchanged via a content-hash tag#239
Closed
elronbandel wants to merge 1 commit into
Closed
feat(release): incremental combos — skip unchanged via a content-hash tag#239elronbandel wants to merge 1 commit into
elronbandel wants to merge 1 commit into
Conversation
… tag Every release rebuilt all ~12,800 combos+standalones (the bulk of the ~6h wall clock) even when nothing changed. Add a content-hash skip on the existing buildx bake — no account, no SaaS, no Dockerfile rewrite (git + buildx + bash): - containers/scripts/combo-src-hash.sh: a combo's hash = its build source + the manifest digest of every base it's FROM (benchmark, agent, gosu, otel, process-compose, model). Identical hash -> skip; a changed base -> new digest -> new hash -> rebuild. That last property is the cascade, for free. - combos job: compute the hash, skip (under skip_published) if the evals/<b>--<a>:src-<hash> tag already exists, and stamp that tag on a successful build so the next run skips this exact source. - tests/build/hash-cascade.sweep.sh + a Rust wrapper (no docker, every cargo test) prove the hash is deterministic, cascades over all 6 parents, and is source-sensitive. imagetools can't read config labels (verified both on the manifest list and the per-arch sub-manifest), so the hash lives in a TAG, not a label. Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
f28570a to
b4774e5
Compare
Contributor
Author
|
Superseded by #241 — the git-diff affected-set is the standard approach (Nx/Turborepo's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stop rebuilding the ~12,800 combos+standalones every release. Add a content-hash skip on the existing
buildx bake— no account, no SaaS, no Dockerfile rewrite (git + buildx + bash).How
containers/scripts/combo-src-hash.sh— a combo's hash = its build source (the two Dockerfiles +runner/+entrypoint/+ the bake graph) folded with the manifest digest of every base it's FROM (benchmark, agent, gosu, otel, process-compose, model).skip_published, skip ifevals/<b>--<a>:src-<hash>already exists; stamp that tag on a successful build so the next run skips this exact source.Why a tag, not a label
docker buildx imagetools inspect --format '{{.Config.Labels…}}'fails on multi-arch images — verified on both the manifest list and the per-arch sub-manifest (can't evaluate field Config). Tag-existence checks work cleanly, so the hash lives in a:src-<hash>tag (the ko / content-addressed-tag pattern).Verified
tests/build/hash-cascade.sweep.sh+ a Rust wrapper (combo_src_hash_cascade, no docker → runs everycargo test) assert the hash is deterministic, cascades over all 6 parents, and is source-sensitive. Both pass locally.dry_rundispatch on this branch.Follow-ups (not in this PR)
:src-*tag GC — each source version leaves a hash-tag; a periodic prune keeps GHCR tidy.This is the no-account, simplest-standard-tools version of incremental builds, validated end-to-end before wiring (the apt/QEMU misfires taught me to measure first).