CI: Separate builds with native runners #277
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This avoids emitting lines such as
[None] booting buildkit
[None] load build definition from Dockerfile
Preparing to add a matrix to the build job. Variables should only be set once and can be done outside of the matrix.
Member
Author
Instead of using one long-running job to build 6 images (3 stages x 2 platforms) and 3 manifest lists, use 2 platform-specific jobs to build 3 images (one per stage) and a third job to build the manifest lists. This change comes with two functional improvements: 1. Most significantly, per-runner disk space usage and build times go down. This is due to both parallelizing the build per platform and removing emulation for the linux/arm64 build. The new time bottleneck for the workflow is now the linux/arm64 test job, which allows for an easy follow-up improvement (see added comment). 2. As a byproduct of the platform-specific images being tagged, they can now be retrieved from GitHub REST API results, removing the need to fetch from GHCR's undocumented Docker Registry API.
victorlin
force-pushed
the
victorlin/split-build
branch
from
9b4087e to
52477eb
Compare
joverlee521
approved these changes
Jan 7, 2026
Contributor
joverlee521
left a comment
joverlee521
left a comment
There was a problem hiding this comment.
Nice! Can't say I fully understand the details, but does seem like the added bit of complexity is worth it.
Comment on lines
+116
to
+135
| - name: Create base-builder-build-platform image | ||
| run: | | ||
| docker buildx imagetools create \ | ||
| -t "ghcr.io/nextstrain/base-builder-build-platform:$TAG" \ | ||
| "ghcr.io/nextstrain/base-builder-build-platform:$TAG-amd64" \ | ||
| "ghcr.io/nextstrain/base-builder-build-platform:$TAG-arm64" | ||
| - name: Create base-builder-target-platform image | ||
| run: | | ||
| docker buildx imagetools create \ | ||
| -t "ghcr.io/nextstrain/base-builder-target-platform:$TAG" \ | ||
| "ghcr.io/nextstrain/base-builder-target-platform:$TAG-amd64" \ | ||
| "ghcr.io/nextstrain/base-builder-target-platform:$TAG-arm64" | ||
| - name: Create base image | ||
| run: | | ||
| docker buildx imagetools create \ | ||
| -t "ghcr.io/nextstrain/base:$TAG" \ | ||
| "ghcr.io/nextstrain/base:$TAG-amd64" \ | ||
| "ghcr.io/nextstrain/base:$TAG-arm64" |
Contributor
There was a problem hiding this comment.
non-blocking
The general pattern in this repo seems to run docker buildx commands in /devel/ scripts, should these steps be wrapped in a separate script?
Member
Author
There was a problem hiding this comment.
Good point, I'll keep it as is for a couple reasons:
- The README instructions for building locally using the existing
develscripts still work fine. The split build +merge-buildspattern is an optimization specific to the GitHub Actions workflow. - Since these commands are simple, keeping them inline and split across 3 steps feels more readable, both in the code and on the GitHub run page. If it ever gets more complex, we can move to a
develwrapper script.
Member
Author
|
Merging, but happy to walk through details in a dev chat session! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of proposed changes
This PR contains two prep commits and one main commit. Message from main commit:
Instead of using one long-running job to build 6 images (3 stages x 2 platforms) and 3 manifest lists, use 2 platform-specific jobs to build 3 images (one per stage) and a third job to build the manifest lists.
This change comes with two functional improvements:
Most significantly, per-runner disk space usage and build times go down. This is due to both parallelizing the build per platform and removing emulation for the linux/arm64 build. The new time bottleneck for the workflow is now the linux/arm64 test job, which allows for an easy follow-up improvement (see added comment).
As a byproduct of the platform-specific images being tagged, they can now be retrieved from GitHub REST API results, removing the need to fetch from GHCR's undocumented Docker Registry API.
Related issue(s)
Checklist
Checks pass
linux/arm64image works on M1 macTested with
zika-tutorial:- [ ] Update changelog