Repository navigation
docker: build and publish a multi-arch image (#548) - #552
Conversation
The published image was amd64-only, so every Apple Silicon user -- most of the audience the MCP work targets -- got a platform-mismatch warning on the very first command in the README. #550 established the code builds and passes its full test suite on aarch64, which is what makes this possible. Native runner per architecture, not buildx+QEMU. This image compiles the whole source tree: 1753 build jobs, ~10 minutes natively. Emulated that is hours and would blow the 90-minute timeout. `ubuntu-24.04-arm` builds it natively in about the same wall time as x86, in parallel. That choice pays a second dividend: every gate already in this workflow -- the MCP smoke against the containerised server, the in-container orlyc compile, the REPL entrypoint mode -- now runs ON the arm image. The published arm64 artifact is therefore proven to work, not merely to compile, which emulated cross-builds would not have given us. Publishing is two stages: each leg pushes an arch-suffixed tag, then a manifest job stitches them with `imagetools create`. Arch tags are left pullable on purpose -- they are the manifest's inputs and the thing you want when only one architecture misbehaves. The manifest job runs only if BOTH legs passed their gates; a manifest pointing at an image that failed its smoke would be worse than no manifest. It then verifies the result rather than assuming it: the published manifest must advertise both amd64 and arm64, which is the actual claim being made. Note this does not change the published image by itself -- that needs a dispatch or a v* tag. README still says the published image is amd64, which stays true until the first multi-arch publish lands; updating it before then would be describing something that has not happened.
The arm64 leg was killed at the 90-minute cap, still inside `jhm -c release`. So it is >90 minutes at 4 workers, and how much over is unknown. The same runner class does `make debug` in 14 minutes (ci.yml, merged in #550), so aarch64 is not simply slow. The difference in the Dockerfile is `-c release` -- an LTO build. LTO is memory-hungry and largely serial at link time, which makes the leading hypothesis memory pressure rather than CPU: if the arm runner has less RAM per core, four concurrent LTO jobs thrash and FEWER workers is faster. Hence per-arch matrix values: 4 workers / 90 min on x86 (unchanged, it builds in ~10), 2 workers / 180 min on arm. The generous timeout is so a slow-but-working build is not killed while we are still learning what it actually costs. That is a hypothesis, not a diagnosis, and the reason the first failure taught us so little is that the job reported only `uname -m`. It now reports nproc and free -g as well. "How many cores and how much RAM did that runner have" is the first question on any slow leg and the one that cannot be answered after the fact. If the capacity report shows ample free memory, the hypothesis is wrong and the next thing to look at is aarch64 LTO codegen time directly.
df1e132 to
eb197b1
Compare
Correcting a claim in the description aboveThe PR body says the arm runner "builds it natively in about the same time as x86, in parallel, at roughly x86 wall time." That is wrong, and I am correcting it here rather than quietly editing it, because the reasoning it supported — native runners over QEMU — deserves to be judged against the real numbers. What I extrapolated from: ci.yml's aarch64 job does What the Dockerfile actually runs is
So the arm leg is at least 9x the x86 leg for this build, not "roughly the same". Does that invalidate the approach?No, and this is worth being precise about. The alternative was buildx+QEMU, which emulates the arm half. If a native arm LTO build exceeds 90 minutes, an emulated one is categorically worse — this result makes the case for native runners stronger, not weaker. What it invalidates is my estimate of the cost, not the choice. Where the investigation standsThe same runner class does Latest push therefore does two things: reports That is a hypothesis. If the capacity report shows ample free memory, it is wrong and the next thing to examine is aarch64 LTO codegen time directly. Not ready to mergeDo not merge this until the arm leg goes green and its cost is understood. A multi-arch manifest whose arm half was never successfully built and smoked would be worse than the amd64-only image it replaces — the current image at least works under emulation, with a warning. |
The memory-pressure hypothesis is deadThe capacity report settles it, and not in my favour: The two runners are identical — same cores, same RAM, both with ~13GB free and zero swap used. There was never any memory pressure, so LTO thrash cannot be the explanation. And halving the workers made it worse, not better: >90 min at 4, >180 min at 2. That is the opposite of what the thrash theory predicts, which is the cleanest possible refutation. What is actually going onThe remaining explanation is the one I listed as the fallback: aarch64 LTO codegen for this tree is enormously more expensive than x86. At least 18x the x86 leg, and still no upper bound — it has never finished. Consistent with everything else we know: the same runner class does What I am not going to doKeep raising the timeout until it passes. Three plausible directions, none of them guesses I should make unilaterally at this point:
My inclination is (2) before choosing between (1) and (3), because right now nobody knows whether this is one bad translation unit or a systemic aarch64 LTO problem — and those have very different answers. StatusNot ready to merge. The amd64 leg and every other check are green; the arm leg has never completed. Merging would publish a manifest whose arm half was never built or smoked, which is worse than today's amd64-only image. |
…me (#548) The multi-arch image is blocked on an arm leg that never finishes `jhm -c release`: >90 min at 4 workers, >180 min at 2, against 10m13s for the entire amd64 job. What is already ruled out: resource starvation. Capacity is identical on both runners (4 cores, 15GB, zero swap used), and halving the workers made it WORSE rather than better, which is the opposite of what memory thrash predicts. The same runner class does `make debug` -- no LTO -- in 14m7s, slightly FASTER than x86, so aarch64 compiles this tree fine. That leaves one question worth answering before choosing a fix: is the cost spread across ~1753 compile targets, or concentrated in the LTO link? Dropping LTO for the image and chasing one pathological translation unit are very different pieces of work, and picking between them by intuition is how the last two rounds were wasted. Two instruments, both cheap: - jhm's progress output timestamped. It prints [done/total], so the timestamps show directly whether the counter crawls (compile-bound) or races to the end and then stalls (link-bound). - a 30s process sampler. If lto1 holds a core for an hour, that settles it. Runs on BOTH arches so the comparison is like-for-like rather than against a number remembered from another workflow, uploads both logs as artifacts, and prints the progress curve even when killed -- a partial run still shows the shape. workflow_dispatch only: this is expensive and diagnostic, not a gate.
workflow_dispatch is only dispatchable once the workflow exists on the DEFAULT branch, and this diagnostic has to run before anything merges. Branch-scoped push trigger, to be removed with the rest of this file once the arm question is answered.
…as emulated (#548) The profiling run reframes this entirely. `jhm -c release` on the BARE arm runner completes in 8m06s -- 884 targets, steady progress, no stall, and faster than the x86 leg. The same command inside `docker build` on the same runner class never finished in 180 minutes. So neither of my two previous explanations survives. It is not memory pressure (identical 4-core/15GB runners, zero swap, and halving the workers made it worse) and it is not aarch64 LTO codegen (8 minutes, natively, at 4 workers). The build is healthy on arm. What is pathological is the build INSIDE docker on arm, and a >20x gap on identical hardware is the signature of qemu rather than of slow codegen. Hence: pass --platform explicitly rather than trusting the daemon default, log docker's client/server arch, the registered binfmt handlers, and what ubuntu:24.04 actually resolves to on that runner, and assert after the build that the image really is the architecture we asked for. If the default was resolving to amd64, every RUN line in the Dockerfile was going through emulation, which would explain everything. If it was already correct, pinning costs nothing and the diagnostics say where to look next. Workers on arm go back to 4, since the reason they were lowered has been refuted. The timeout stays generous until a native arm image build has actually been observed to finish.
The profiling round answered #548's question and produced a better one. `jhm -c release` is fine on arm: 482s inside docker, 8m06s on the bare runner, both faster than amd64's 11m05s. What hangs is the NEXT docker step -- compiling the example packages with the just-built orlyc, which printed its header and then nothing for 37 minutes before the job was cancelled. Per the Dockerfile's own comment, each orlyc run stands up and tears down an embedded mem-sim server, so that is the suspect. This reproduces it in ~9 minutes rather than a 45-minute docker build: build orlyc alone, run it on one package with a 300s deadline, and if it is still alive, interrogate the live process rather than merely noting that it never returned -- process state, kernel wait channel per thread, and all-thread gdb backtraces. amd64 runs as the control, since orlyc demonstrably works there. Replaces the LTO profiler, which has served its purpose.
The first run proved nothing. $GITHUB_WORKSPACE already ends in /orly (it is the repo root, the Dockerfile's WORKDIR /src/orly), and I appended another, so orlyc got a doubled path, exited 1 in two seconds on BOTH arches, and the job reported green. That is worse than a failed repro: a diagnostic that exits fast and clean while testing nothing is indistinguishable from good news. So the inputs are now asserted to exist before anything is timed -- if the path is wrong again the job says so instead of quietly passing.
…554) The last run piped gdb through `tail -80`, which cut exactly the frame that matters -- the worker running the main job. Its stack is what distinguishes "RunPackageTests never returned" from "server->Shutdown() wedged", and those have different fixes. Now: full stacks to a file and uploaded as an artifact, the teardown-relevant frames grepped inline, and a SECOND snapshot 60s later. Identical stacks across the two means genuinely wedged; moving frames would mean livelock or very slow progress, which would be a different bug and worth knowing before assuming deadlock.
1 participant
Closes the last open box on #548 — the one the whole thread started from.
Why
The published image is
linux/amd64only, so an Apple Silicon user gets a platform-mismatch warning on the first command in the README. That is the audience the MCP/agent work is aimed at. #550 established the code builds and passes its full test suite on aarch64 (1753 build jobs, 203 test binaries, zero failures), which is what makes this possible at all.Native runners, not buildx + QEMU
The reflex here is
docker/setup-qemu-action+buildx --platform linux/amd64,linux/arm64. That would be wrong for this image.This image compiles the whole source tree — 1753 build jobs, ~10 minutes natively. Under emulation that is plausibly hours and would exceed the 90-minute timeout. So:
ubuntu-24.04andubuntu-24.04-arm, each building its own architecture natively, in parallel, at roughly x86 wall time.That choice pays a second dividend I'd argue is the bigger one. Every gate already in this workflow now runs on the arm image:
orlyccompile to a loadable.soSo the published arm64 artifact is proven to work, not merely to compile. An emulated cross-build would have given us a binary nobody had ever run.
Publishing
Two stages. Each leg pushes an arch-suffixed tag; a
manifestjob stitches them withimagetools create.amd64andarm64— that is the actual claim being made, so it should be checked, not hoped for.fail-fast: falseon the matrix, because "arm broke, x86 is fine" is the useful signal and fail-fast hides it.The Dockerfile needed no changes
Worth stating since it is the obvious place for trouble: both stages are
ubuntu:24.04(multi-arch official), every dependency is an apt package by name, nothing downloads a prebuilt binary, and node comes from Ubuntu's repo. Noamd64, nox86_64, nouname -manywhere in it.This does not change the published image by itself
Merging only changes how it is built. The registry still serves amd64-only until someone runs the workflow via dispatch or pushes a
v*tag.So the README is deliberately untouched here. It currently says the published image is
linux/amd64and runs under emulation on Apple Silicon — which stays true until that first multi-arch publish. Updating it now would be documenting something that has not happened, and this repo has history with that particular failure mode.Follow-up once a multi-arch image is actually live: drop the emulation caveat from the Quick start section and the "the one the published docker image targets" qualifier under Supported platforms.
Testing
This PR touches
.github/workflows/docker.yml, which is in the workflow's ownpathsfilter — so the docker workflow runs on this PR and builds both architectures with all gates, no push. That is the test.