build: split the image into runtime and uat targets, publish multi-arch - #6
Merged
Merged
Conversation
Closes #5. The published image was the UAT drill image. Its own header said so, verbatim: "for the UAT drill environment only" and "NOT hardened for production". That is what v0.1.0 pushed to GHCR and what studios pull, so the artifact carrying an explicit not-for-production warning was the one running in production. Two targets off the same Go build stage now: runtime alpine + ca-certificates + the binary default, published by v* tags uat ruby:3.3-alpine, unchanged drill environment only runtime is last, so a bare `docker build` with no --target yields the safe artifact, and release.yml names `target: runtime` explicitly so reordering the file can never quietly publish the drill image again. The Ruby stays where it is load-bearing. fake_p4 is a Ruby script the UAT compose file bind-mounts over /usr/local/bin/p4, and the connector execs whatever binary connector.yml names. The seven-drill harness is unaffected: `make drills` runs `ruby test/drills.rb` on the host (RUBY ?= ruby), never inside an image. alpine rather than scratch or distroless, deliberately. The connector execs perforce.binary (default "p4", internal/tools/perforce.go), so a studio has to be able to supply a real p4 client by bind-mount or in a derived image. scratch forecloses both. Nothing is installed beyond CA certificates and there is no interpreter. Also fixes the arch asymmetry: v0.1.0 shipped an amd64-only image while the release archives carried linux/arm64. setup-qemu-action plus platforms: linux/amd64,linux/arm64. Turned on provenance and SBOM attestations while in the file, which covers one of the #1575 survival conditions. Entrypoint, config path, and uid/gid 10001 are identical between the two targets, so an existing deployment moves by changing only the digest. Verified by building both: runtime 9.6 MiB compressed (21.7 MB on disk) against the published v0.1.0's 40.6 MiB, ruby absent from runtime, `-config` flag intact, and id reporting uid=10001(connector) gid=10001(connector).
Corrects the runtime base from alpine to distroless/base-debian12. The split
itself is unchanged; the base is not a preference, it is forced.
The connector execs the binary named by connector.yml's perforce.binary
(default "p4", internal/tools/perforce.go). A studio supplies that binary
itself, and on bsg-cp-01 it does so by bind-mounting the p4 client out of the
Perforce container, so the client version tracks the server automatically.
Perforce's packaged p4 is dynamically linked against glibc (ldd: libc.so.6,
librt, libdl, libm, libpthread, /lib64/ld-linux-x86-64.so.2), so a musl
runtime rejects it outright. Measured against the real binary from the
helix-p4d package rather than assumed:
alpine:3.22 p4 -V -> "Dynamic loader not found:
/lib64/ld-linux-x86-64.so.2"
distroless/base-debian12 p4 -V -> runs
this runtime image + mount p4 -V -> runs
distroless/base rather than debian-slim because it supplies glibc and CA
certificates and nothing else: no shell, no package manager, no interpreter.
That matters more than usual in an image whose job includes executing an
operator-supplied binary. Verified there is no /bin/sh in the result.
Pinned by digest, which covers survival condition 4 (#1575). The digest is a
manifest index, so linux/amd64 and linux/arm64 both still resolve.
uid/gid stay 10001, and the entrypoint and config path are unchanged, so a
deployment still moves between targets by changing only the digest. distroless
has no shell to run adduser in, so the account and directory skeleton are built
in a small alpine stage and copied in; COPY --from preserves numeric ownership.
USER is numeric so it does not depend on an /etc/passwd lookup a derived image
could overwrite.
Size is still most of the win: 15.7 MiB compressed against the published
v0.1.0's 40.6 MiB.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5.
What was wrong
The published image was the UAT drill image. Its own header said so, verbatim: "for the UAT drill environment only" and "NOT hardened for production". That is what
v0.1.0pushed to GHCR and what a studio pulls, so the artifact carrying an explicit not-for-production warning was the one running in production.ButterStack/terraformbranchdevops/bsg-azureruns it onbsg-cp-01today.Size was the symptom that led here: 40.6 MiB compressed, of which 33.9 MiB is a Ruby interpreter, to ship a static Go binary.
What this does
Two publishable targets off the same Go build stage:
runtimegcr.io/distroless/base-debian12, digest-pinnedv*tags publishuatruby:3.3-alpine, otherwise unchangedruntimeis last in the file, so a baredocker buildwith no--targetproduces the safe artifact, andrelease.ymlnamestarget: runtimeexplicitly so reordering the Dockerfile can never quietly publish the drill image to studios again.The Ruby stays where it is load-bearing.
fake_p4is a Ruby script the UAT compose file bind-mounts over/usr/local/bin/p4. The seven-drill harness is unaffected:make drillsrunsruby test/drills.rbon the host (RUBY ?= ruby), never inside an image.The base is glibc because it has to be, not because I preferred it
The first commit here used alpine. That was wrong, and the second commit fixes it.
The connector execs the binary named by
connector.yml'sperforce.binary(default"p4",internal/tools/perforce.go). A studio supplies that binary itself; onbsg-cp-01it will bind-mount the p4 client straight out of the Perforce container, so the client version tracks the server automatically. Perforce's packagedp4is dynamically linked against glibc (ldd:libc.so.6,librt,libdl,libm,libpthread,/lib64/ld-linux-x86-64.so.2), so a musl runtime rejects it outright.Measured against the real binary from the
helix-p4dpackage, not assumed:distroless/baserather thandebian-slimbecause it supplies glibc and CA certificates and nothing else: no shell, no package manager, no interpreter. That matters more than usual in an image whose job includes executing an operator-supplied binary. Confirmed there is no/bin/shin the result.Multi-arch and supply chain
v0.1.0shipped an amd64-only image while the release archives carriedlinux/arm64. Addeddocker/setup-qemu-action(without it buildx silently emits amd64 only from an amd64 runner) andplatforms: linux/amd64,linux/arm64.Also switched on
provenanceandsbomattestations, and digest-pinned the runtime base. That covers survival conditions 1 and 4 from #1575, which the Dockerfile header lists as unmet. The pinned digest is a manifest index, so both architectures still resolve.Compatibility
Entrypoint, config path (
/etc/butterstack/connector.yml), and uid/gid10001are identical between the two targets, so an existing deployment moves by changing only the digest. distroless has no shell to runadduserin, so the account and directory skeleton are built in a small alpine stage and copied in (COPY --frompreserves numeric ownership).USERis numeric so it does not depend on an/etc/passwdlookup a derived image could overwrite.Verified, both targets built
[/usr/local/bin/butterstack-connector -config /etc/butterstack/connector.yml], user10001:10001-configflag intact/bin/shabsent from runtime, present in uatCorrection: the second commit's message says 15.7 MiB compressed. The measured figure is 11.4 MiB. The commit message is wrong; this number is right.
Multi-arch itself is not verified locally, since it only exercises on the release runner.
Sequencing
target: uattodocker-compose.uat-connector.yml. Merge that before or with this, or the next UAT run builds the runtime image and losesfake_p4anduat-entrypoint.sh.ButterStack/terraformbsg-azure/files/compose/docker-compose.ymlneeds its pinned digest updated and the p4 bind-mount added.