Skip to content

build: split the image into runtime and uat targets, publish multi-arch - #6

Merged
ryanlitalien merged 2 commits into
mainfrom
build/split-runtime-uat-images
Sep 4, 2026
Merged

build: split the image into runtime and uat targets, publish multi-arch#6
ryanlitalien merged 2 commits into
mainfrom
build/split-runtime-uat-images

Conversation

@ryanlitalien

@ryanlitalien ryanlitalien commented Sep 4, 2026

Copy link
Copy Markdown
Member

Closes #5.

What was wrong

The published image was the UAT drill image. Its own header said so, verbatim: "for the UAT drill environment only" and "NOT hardened for production". That is what v0.1.0 pushed to GHCR and what a studio pulls, so the artifact carrying an explicit not-for-production warning was the one running in production. ButterStack/terraform branch devops/bsg-azure runs it on bsg-cp-01 today.

Size was the symptom that led here: 40.6 MiB compressed, of which 33.9 MiB is a Ruby interpreter, to ship a static Go binary.

What this does

Two publishable targets off the same Go build stage:

target base who publishes it
runtime gcr.io/distroless/base-debian12, digest-pinned default, and what v* tags publish
uat ruby:3.3-alpine, otherwise unchanged drill environment only

runtime is last in the file, so a bare docker build with no --target produces the safe artifact, and release.yml names target: runtime explicitly so reordering the Dockerfile can never quietly publish the drill image to studios again.

The Ruby stays where it is load-bearing. fake_p4 is a Ruby script the UAT compose file bind-mounts over /usr/local/bin/p4. The seven-drill harness is unaffected: make drills runs ruby test/drills.rb on the host (RUBY ?= ruby), never inside an image.

The base is glibc because it has to be, not because I preferred it

The first commit here used alpine. That was wrong, and the second commit fixes it.

The connector execs the binary named by connector.yml's perforce.binary (default "p4", internal/tools/perforce.go). A studio supplies that binary itself; on bsg-cp-01 it will bind-mount the p4 client straight out of the Perforce container, so the client version tracks the server automatically. Perforce's packaged p4 is dynamically linked against glibc (ldd: libc.so.6, librt, libdl, libm, libpthread, /lib64/ld-linux-x86-64.so.2), so a musl runtime rejects it outright.

Measured against the real binary from the helix-p4d package, not assumed:

alpine:3.22                 p4 -V -> Dynamic loader not found: /lib64/ld-linux-x86-64.so.2
distroless/base-debian12    p4 -V -> Perforce - The Fast Software Configuration Management System
this runtime image + mount  p4 -V -> Perforce - The Fast Software Configuration Management System

distroless/base rather than debian-slim because it supplies glibc and CA certificates and nothing else: no shell, no package manager, no interpreter. That matters more than usual in an image whose job includes executing an operator-supplied binary. Confirmed there is no /bin/sh in the result.

Multi-arch and supply chain

v0.1.0 shipped an amd64-only image while the release archives carried linux/arm64. Added docker/setup-qemu-action (without it buildx silently emits amd64 only from an amd64 runner) and platforms: linux/amd64,linux/arm64.

Also switched on provenance and sbom attestations, and digest-pinned the runtime base. That covers survival conditions 1 and 4 from #1575, which the Dockerfile header lists as unmet. The pinned digest is a manifest index, so both architectures still resolve.

Compatibility

Entrypoint, config path (/etc/butterstack/connector.yml), and uid/gid 10001 are identical between the two targets, so an existing deployment moves by changing only the digest. distroless has no shell to run adduser in, so the account and directory skeleton are built in a small alpine stage and copied in (COPY --from preserves numeric ownership). USER is numeric so it does not depend on an /etc/passwd lookup a derived image could overwrite.

Verified, both targets built

runtime  27.6 MB on disk   11.4 MiB compressed
uat      83.6 MB on disk   45.2 MiB compressed
published v0.1.0:          40.6 MiB compressed
  • entrypoint [/usr/local/bin/butterstack-connector -config /etc/butterstack/connector.yml], user 10001:10001
  • mounted p4 from the Perforce image executes inside the runtime image
  • connector binary runs, -config flag intact
  • /bin/sh absent from runtime, present in uat

Correction: the second commit's message says 15.7 MiB compressed. The measured figure is 11.4 MiB. The commit message is wrong; this number is right.

Multi-arch itself is not verified locally, since it only exercises on the release runner.

Sequencing

  • ButterStack/butter_stack#1668 adds target: uat to docker-compose.uat-connector.yml. Merge that before or with this, or the next UAT run builds the runtime image and loses fake_p4 and uat-entrypoint.sh.
  • After this merges and a release is cut, ButterStack/terraform bsg-azure/files/compose/docker-compose.yml needs its pinned digest updated and the p4 bind-mount added.

Closes #5.

The published image was the UAT drill image. Its own header said so, verbatim:
"for the UAT drill environment only" and "NOT hardened for production". That is
what v0.1.0 pushed to GHCR and what studios pull, so the artifact carrying an
explicit not-for-production warning was the one running in production.

Two targets off the same Go build stage now:

  runtime  alpine + ca-certificates + the binary   default, published by v* tags
  uat      ruby:3.3-alpine, unchanged              drill environment only

runtime is last, so a bare `docker build` with no --target yields the safe
artifact, and release.yml names `target: runtime` explicitly so reordering the
file can never quietly publish the drill image again.

The Ruby stays where it is load-bearing. fake_p4 is a Ruby script the UAT
compose file bind-mounts over /usr/local/bin/p4, and the connector execs
whatever binary connector.yml names. The seven-drill harness is unaffected:
`make drills` runs `ruby test/drills.rb` on the host (RUBY ?= ruby), never
inside an image.

alpine rather than scratch or distroless, deliberately. The connector execs
perforce.binary (default "p4", internal/tools/perforce.go), so a studio has to
be able to supply a real p4 client by bind-mount or in a derived image. scratch
forecloses both. Nothing is installed beyond CA certificates and there is no
interpreter.

Also fixes the arch asymmetry: v0.1.0 shipped an amd64-only image while the
release archives carried linux/arm64. setup-qemu-action plus
platforms: linux/amd64,linux/arm64. Turned on provenance and SBOM attestations
while in the file, which covers one of the #1575 survival conditions.

Entrypoint, config path, and uid/gid 10001 are identical between the two
targets, so an existing deployment moves by changing only the digest. Verified
by building both: runtime 9.6 MiB compressed (21.7 MB on disk) against the
published v0.1.0's 40.6 MiB, ruby absent from runtime, `-config` flag intact,
and id reporting uid=10001(connector) gid=10001(connector).
Corrects the runtime base from alpine to distroless/base-debian12. The split
itself is unchanged; the base is not a preference, it is forced.

The connector execs the binary named by connector.yml's perforce.binary
(default "p4", internal/tools/perforce.go). A studio supplies that binary
itself, and on bsg-cp-01 it does so by bind-mounting the p4 client out of the
Perforce container, so the client version tracks the server automatically.
Perforce's packaged p4 is dynamically linked against glibc (ldd: libc.so.6,
librt, libdl, libm, libpthread, /lib64/ld-linux-x86-64.so.2), so a musl
runtime rejects it outright. Measured against the real binary from the
helix-p4d package rather than assumed:

  alpine:3.22                  p4 -V -> "Dynamic loader not found:
                                         /lib64/ld-linux-x86-64.so.2"
  distroless/base-debian12     p4 -V -> runs
  this runtime image + mount   p4 -V -> runs

distroless/base rather than debian-slim because it supplies glibc and CA
certificates and nothing else: no shell, no package manager, no interpreter.
That matters more than usual in an image whose job includes executing an
operator-supplied binary. Verified there is no /bin/sh in the result.

Pinned by digest, which covers survival condition 4 (#1575). The digest is a
manifest index, so linux/amd64 and linux/arm64 both still resolve.

uid/gid stay 10001, and the entrypoint and config path are unchanged, so a
deployment still moves between targets by changing only the digest. distroless
has no shell to run adduser in, so the account and directory skeleton are built
in a small alpine stage and copied in; COPY --from preserves numeric ownership.
USER is numeric so it does not depend on an /etc/passwd lookup a derived image
could overwrite.

Size is still most of the win: 15.7 MiB compressed against the published
v0.1.0's 40.6 MiB.
@ryanlitalien
ryanlitalien merged commit e2f45d9 into main Sep 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ship a thin runtime image: move the Ruby UAT layer to a separate build target

1 participant