Skip to content

feat(release): incremental combos — skip unchanged via a content-hash tag#239

Closed
elronbandel wants to merge 1 commit into
mainfrom
elron/combo-hash-skip
Closed

feat(release): incremental combos — skip unchanged via a content-hash tag#239
elronbandel wants to merge 1 commit into
mainfrom
elron/combo-hash-skip

Conversation

@elronbandel

Copy link
Copy Markdown
Contributor

What

Stop rebuilding the ~12,800 combos+standalones every release. Add a content-hash skip on the existing buildx bake — no account, no SaaS, no Dockerfile rewrite (git + buildx + bash).

How

  • containers/scripts/combo-src-hash.sh — a combo's hash = its build source (the two Dockerfiles + runner/ + entrypoint/ + the bake graph) folded with the manifest digest of every base it's FROM (benchmark, agent, gosu, otel, process-compose, model).
    • Identical hash → skip.
    • A changed base → new parent digest → new hash → rebuild. That's the cascade, for free — a base rebuild ripples into every combo on top of it.
  • combos job — compute the hash; under skip_published, skip if evals/<b>--<a>:src-<hash> already exists; stamp that tag on a successful build so the next run skips this exact source.

Why a tag, not a label

docker buildx imagetools inspect --format '{{.Config.Labels…}}' fails on multi-arch images — verified on both the manifest list and the per-arch sub-manifest (can't evaluate field Config). Tag-existence checks work cleanly, so the hash lives in a :src-<hash> tag (the ko / content-addressed-tag pattern).

Verified

  • Logic, two ways: tests/build/hash-cascade.sweep.sh + a Rust wrapper (combo_src_hash_cascade, no docker → runs every cargo test) assert the hash is deterministic, cascades over all 6 parents, and is source-sensitive. Both pass locally.
  • Lint: shellcheck (scripts + the workflow bash), actionlint, cargo fmt — all green.
  • The live combos-job wiring (digest reads + skip-check) is exercised by a dry_run dispatch on this branch.

Follow-ups (not in this PR)

This is the no-account, simplest-standard-tools version of incremental builds, validated end-to-end before wiring (the apt/QEMU misfires taught me to measure first).

… tag

Every release rebuilt all ~12,800 combos+standalones (the bulk of the ~6h wall
clock) even when nothing changed. Add a content-hash skip on the existing
buildx bake — no account, no SaaS, no Dockerfile rewrite (git + buildx + bash):

- containers/scripts/combo-src-hash.sh: a combo's hash = its build source + the
  manifest digest of every base it's FROM (benchmark, agent, gosu, otel,
  process-compose, model). Identical hash -> skip; a changed base -> new digest
  -> new hash -> rebuild. That last property is the cascade, for free.
- combos job: compute the hash, skip (under skip_published) if the
  evals/<b>--<a>:src-<hash> tag already exists, and stamp that tag on a
  successful build so the next run skips this exact source.
- tests/build/hash-cascade.sweep.sh + a Rust wrapper (no docker, every cargo
  test) prove the hash is deterministic, cascades over all 6 parents, and is
  source-sensitive.

imagetools can't read config labels (verified both on the manifest list and the
per-arch sub-manifest), so the hash lives in a TAG, not a label.

Signed-off-by: Elron Bandel <elron.bandel@ibm.com>
@elronbandel

Copy link
Copy Markdown
Contributor Author

Superseded by #241 — the git-diff affected-set is the standard approach (Nx/Turborepo's affected) and far simpler than content-hash tags. No script, no per-job skip.

@elronbandel elronbandel deleted the elron/combo-hash-skip branch June 29, 2026 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant