docker: build and publish a multi-arch image (#548)

.github/workflows/docker.yml

@@ -1,9 +1,18 @@
 # Build (and on demand, publish) the Orly runtime image (#530).
 #
-# Pull requests touching the docker files build the image and gate it with
-# the MCP smoke pointed at the CONTAINERIZED server (the same smoke.mjs that
-# gates clients/mcp against a host orlyi) -- no push. A version tag or a
-# manual dispatch additionally pushes to ghcr.io/orlyatomics/orly.
+# Pull requests touching the docker files build the image on BOTH supported
+# architectures and gate each one with the MCP smoke pointed at the
+# CONTAINERIZED server (the same smoke.mjs that gates clients/mcp against a
+# host orlyi) -- no push. A version tag or a manual dispatch additionally
+# pushes each arch and then stitches them into one multi-arch manifest.
+#
+# Why two native runners rather than buildx+QEMU (#548): the arm64 half would
+# be emulated, and this image compiles the whole source tree -- 1753 build jobs,
+# ~10 minutes natively. Under qemu that is hours and would blow the timeout.
+# `ubuntu-24.04-arm` builds it natively in about the same time as x86, in
+# parallel, and has the side benefit that every gate below (MCP smoke,
+# in-container orlyc compile, REPL mode) runs ON the arm image -- so the
+# published arm64 image is proven to work, not merely to build.
 name: docker
 
 on:
@@ -19,21 +28,88 @@ on:
 
 jobs:
   image:
-    name: build image + in-container smoke
-    runs-on: ubuntu-24.04
-    timeout-minutes: 90
+    name: build image + in-container smoke (${{ matrix.arch }})
+    strategy:
+      # Report both architectures even when one fails -- "arm is broken, x86 is
+      # fine" is the useful signal, and fail-fast would hide it.
+      fail-fast: false
+      matrix:
+        include:
+          - arch: amd64
+            runner: ubuntu-24.04
+            # Matches the Dockerfile default; the x86 leg builds in ~10 min.
+            workers: 4
+            timeout: 90
+          - arch: arm64
+            runner: ubuntu-24.04-arm
+            # Back to 4: the memory-thrash theory was tested and refuted --
+            # both runners are 4-core/15GB with zero swap used, and HALVING the
+            # workers made it worse (>90 min at 4, >180 min at 2). The profiling
+            # run then showed `jhm -c release` completing on the bare arm runner
+            # in 8m06s at 4 workers, so the build is not the problem. Timeout
+            # stays generous until a native arm image build has actually been
+            # observed to finish.
+            workers: 4
+            timeout: 180
+    runs-on: ${{ matrix.runner }}
+    timeout-minutes: ${{ matrix.timeout }}
     permissions:
       contents: read
       packages: write
+    env:
+      # Arch-suffixed so the two matrix legs cannot collide in the registry,
+      # and so a single-arch problem can be pulled and poked at directly.
+      IMAGE: ghcr.io/orlyatomics/orly:ci-${{ matrix.arch }}
     steps:
       - uses: actions/checkout@v6
 
+      # Cores and RAM, not just the arch: the release build in the Dockerfile is
+      # LTO, which is memory-hungry and largely serial at link time. When a leg
+      # is slow, "how many cores and how much RAM did it have" is the first
+      # question and the one that is impossible to answer after the fact.
+      - name: Report runner capacity
+        run: |
+          uname -m
+          echo "nproc: $(nproc)"
+          free -g || true
+          echo "JHM_WORKER_COUNT will be: ${{ matrix.workers }}"
+
+      # Plain `docker build`, not buildx: each runner is native for its own
+      # architecture, so there is nothing to cross-compile or emulate.
+      # Is this build NATIVE, or is it being emulated? That is the whole question
+      # after the profiling run (#548): `jhm -c release` on the bare arm runner
+      # finishes in 8m06s -- faster than x86 -- while the SAME command inside
+      # `docker build` on the same runner class never finished in 180 minutes. A
+      # >20x gap on identical hardware is the signature of qemu, not of slow
+      # codegen.
+      - name: Docker and emulation state
+        run: |
+          docker version --format 'client={{.Client.Arch}} server={{.Server.Arch}}' || true
+          echo "--- registered binfmt handlers (any qemu-* here means emulation is available):"
+          ls /proc/sys/fs/binfmt_misc/ 2>/dev/null | grep -i qemu || echo "  none"
+          echo "--- what arch does the base image resolve to here:"
+          docker pull -q ubuntu:24.04 >/dev/null 2>&1 || true
+          docker image inspect ubuntu:24.04 --format '  ubuntu:24.04 -> {{.Architecture}}' || true
+
+      # --platform pinned explicitly rather than relying on the daemon's default.
+      # If the default was resolving to amd64 on the arm runner, every RUN line
+      # was going through qemu, which explains the 20x. Pinning it costs nothing
+      # if the default was already correct.
       - name: Build image
-        run: docker build -t ghcr.io/orlyatomics/orly:ci .
+        run: |
+          docker build --platform "linux/${{ matrix.arch }}" \
+            --build-arg JHM_WORKER_COUNT=${{ matrix.workers }} \
+            -t "$IMAGE" .
+
+      - name: Confirm the built image is the intended architecture
+        run: |
+          got=$(docker image inspect "$IMAGE" --format '{{.Architecture}}')
+          echo "built image architecture: $got (wanted ${{ matrix.arch }})"
+          test "$got" = "${{ matrix.arch }}"
 
       - name: Start container
         run: |
-          docker run -d --name orly -p 8082:8082 ghcr.io/orlyatomics/orly:ci
+          docker run -d --name orly -p 8082:8082 "$IMAGE"
           for _ in $(seq 1 60); do
             if ss -tln | grep -q ':8082'; then exit 0; fi
             sleep 1
@@ -74,22 +150,70 @@ jobs:
       - name: REPL mode against the image
         run: |
           printf '40 + 2;\n:quit\n' \
-            | docker run -i --rm ghcr.io/orlyatomics/orly:ci repl \
+            | docker run -i --rm "$IMAGE" repl \
             | tee /tmp/repl-out
           grep -qE '(^|orly> )42$' /tmp/repl-out
 
       - name: Container logs (for diagnosis)
         if: always()
         run: docker logs orly 2>&1 | tail -60
 
-      - name: Push to ghcr.io
+      # Each leg pushes its own arch-tagged image; the manifest job below
+      # stitches them together. Pushing per-arch tags (rather than the
+      # push-by-digest dance) keeps this a plain `docker push` and leaves a
+      # pullable single-arch image behind for debugging.
+      - name: Push arch image to ghcr.io
         if: github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/tags/v')
         run: |
           echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
-          docker tag ghcr.io/orlyatomics/orly:ci ghcr.io/orlyatomics/orly:latest
-          docker push ghcr.io/orlyatomics/orly:latest
+          docker tag "$IMAGE" "ghcr.io/orlyatomics/orly:latest-${{ matrix.arch }}"
+          docker push "ghcr.io/orlyatomics/orly:latest-${{ matrix.arch }}"
           if [[ "${GITHUB_REF}" == refs/tags/v* ]]; then
             ver="${GITHUB_REF#refs/tags/}"
-            docker tag ghcr.io/orlyatomics/orly:ci "ghcr.io/orlyatomics/orly:${ver}"
-            docker push "ghcr.io/orlyatomics/orly:${ver}"
+            docker tag "$IMAGE" "ghcr.io/orlyatomics/orly:${ver}-${{ matrix.arch }}"
+            docker push "ghcr.io/orlyatomics/orly:${ver}-${{ matrix.arch }}"
           fi
+
+  manifest:
+    name: publish multi-arch manifest
+    needs: image
+    # Only on a real publish, and only if BOTH arches built and passed their
+    # gates -- a manifest listing an image that failed its smoke would be worse
+    # than no manifest at all.
+    if: github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/tags/v')
+    runs-on: ubuntu-24.04
+    timeout-minutes: 15
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - name: Log in to ghcr.io
+        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
+
+      # `imagetools create` builds a manifest list from the per-arch tags, so
+      # `docker pull ghcr.io/orlyatomics/orly` resolves to the right one and
+      # Apple Silicon stops printing a platform-mismatch warning on the first
+      # command in the README (#548).
+      - name: Stitch :latest
+        run: |
+          docker buildx imagetools create -t ghcr.io/orlyatomics/orly:latest \
+            ghcr.io/orlyatomics/orly:latest-amd64 \
+            ghcr.io/orlyatomics/orly:latest-arm64
+
+      - name: Stitch the version tag
+        if: startsWith(github.ref, 'refs/tags/v')
+        run: |
+          ver="${GITHUB_REF#refs/tags/}"
+          docker buildx imagetools create -t "ghcr.io/orlyatomics/orly:${ver}" \
+            "ghcr.io/orlyatomics/orly:${ver}-amd64" \
+            "ghcr.io/orlyatomics/orly:${ver}-arm64"
+
+      # Assert the thing this whole change exists for, rather than assuming the
+      # stitch worked: the published manifest must advertise both platforms.
+      - name: Verify the manifest lists both platforms
+        run: |
+          got=$(docker buildx imagetools inspect --raw ghcr.io/orlyatomics/orly:latest \
+                  | grep -oE '"architecture":"[a-z0-9]+"' | sort -u | tr '\n' ' ')
+          echo "manifest advertises: $got"
+          echo "$got" | grep -q '"architecture":"amd64"'
+          echo "$got" | grep -q '"architecture":"arm64"'

.github/workflows/orlyc-arm-repro.yml

@@ -0,0 +1,159 @@
+# One-shot diagnostic for #554: why does `orlyc` hang on aarch64?
+#
+# The profiling round answered #548's original question and produced a better
+# one. `jhm -c release` is FINE on arm -- 482s inside docker, 8m06s on the bare
+# runner, both faster than the amd64 leg's 11m05s. What hangs is the NEXT docker
+# step: compiling the example packages with the freshly built orlyc, which
+# emitted its header line and then nothing for 37 minutes.
+#
+# Per the Dockerfile's own comment, "each orlyc run stands up and tears down its
+# embedded mem-sim server" -- so the suspicion is that server never completes its
+# handshake or teardown on aarch64.
+#
+# `make test` does not exercise the orlyc shell-out path and lang_test.py is not
+# in the arm job, which is why #550 could be green while this was broken.
+#
+# This reproduces it in ~9 minutes instead of a 45-minute docker build, and if it
+# hangs, says WHERE: thread backtraces, kernel wait channel, and process state.
+# amd64 runs as the control -- orlyc demonstrably works there.
+#
+# Deliberately not a gate.
+name: orlyc-arm-repro
+
+on:
+  workflow_dispatch:
+  # TEMPORARY, and the reason is a GitHub constraint rather than a preference:
+  # `workflow_dispatch` is only dispatchable once the workflow exists on the
+  # DEFAULT branch, and this diagnostic needs to run before anything merges.
+  # Drop this trigger (or the whole file) once #548's arm question is answered.
+  push:
+    branches: [feat/548-multiarch-image]
+
+jobs:
+  repro:
+    name: orlyc repro (${{ matrix.arch }})
+    strategy:
+      # Both arches, so the comparison is like-for-like rather than against a
+      # remembered number from a different workflow.
+      fail-fast: false
+      matrix:
+        include:
+          - arch: amd64
+            runner: ubuntu-24.04
+          - arch: arm64
+            runner: ubuntu-24.04-arm
+    runs-on: ${{ matrix.runner }}
+    # Long enough to actually finish if it can. If it cannot, the timestamped
+    # progress still shows the SHAPE of the curve, which is the whole point.
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Capacity
+        run: |
+          uname -m
+          echo "nproc: $(nproc)"
+          free -g
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y -qq \
+            build-essential gcc g++ uuid-dev libgmp-dev libaio-dev \
+            libsnappy-dev libreadline-dev libboost-system-dev zlib1g-dev \
+            bison flex valgrind moreutils
+
+      - name: Bootstrap
+        run: make bootstrap version
+
+      - name: Build orlyc only
+        run: |
+          PATH="$PWD/tools:$PATH" \
+            jhm -c release --worker-count 4 orly/orlyc
+          ls -la ../out_orly/release/orly/orlyc
+
+      # The actual repro. Backgrounded so that if it hangs we can interrogate the
+      # live process rather than just observing that it never returned.
+      - name: Run orlyc on one package, with a deadline
+        id: repro
+        run: |
+          set +e
+          mkdir -p /tmp/pkgout && cd /tmp/pkgout
+          ORLYC=$GITHUB_WORKSPACE/../out_orly/release/orly/orlyc
+          PKG=$GITHUB_WORKSPACE/clients/mcp/smoke/sample.orly
+          # Assert both inputs exist before timing anything. The first run of this
+          # job "passed" in 2 seconds because the package path had a doubled
+          # segment -- orlyc exited 1 on both arches and the repro proved nothing.
+          test -x "$ORLYC" || { echo "no orlyc at $ORLYC"; exit 1; }
+          test -f "$PKG"   || { echo "no package at $PKG"; exit 1; }
+          "$ORLYC" -o /tmp/pkgout "$PKG" \
+            > /tmp/orlyc.out 2>&1 &
+          PID=$!
+          echo "orlyc pid=$PID"
+          for i in $(seq 1 60); do
+            kill -0 "$PID" 2>/dev/null || break
+            sleep 5
+          done
+          if kill -0 "$PID" 2>/dev/null; then
+            echo "HUNG=1" >> "$GITHUB_OUTPUT"
+            echo "=== still alive after 300s -- interrogating pid $PID"
+            echo "--- /proc/$PID/status:"; grep -E "State|Threads" /proc/$PID/status
+            echo "--- wchan (kernel function it is blocked in):"
+            cat /proc/$PID/wchan 2>/dev/null; echo
+            echo "--- per-thread state and wchan:"
+            for t in /proc/$PID/task/*; do
+              printf "  tid %s  state=%s  wchan=%s\n" "$(basename $t)" \
+                "$(awk '/^State:/{print $2}' $t/status 2>/dev/null)" \
+                "$(cat $t/wchan 2>/dev/null)"
+            done
+            # FULL backtraces to a file. The previous run piped this through
+            # `tail -80`, which cut exactly the frame that matters: the worker
+            # running the main job, whose stack says whether RunPackageTests
+            # never returned or server->Shutdown() wedged.
+            echo "--- all-thread backtraces -> /tmp/bt1.txt"
+            sudo gdb -p "$PID" -batch \
+              -ex "set pagination off" -ex "thread apply all bt" > /tmp/bt1.txt 2>&1
+            echo "  captured $(grep -c '^Thread ' /tmp/bt1.txt) threads"
+            # The frames worth reading inline: anything in orlyc/server teardown.
+            grep -nE "RunPackageTests|TServer::Shutdown|RunTestsOnIndy|Fiber::TRunner|TScheduler::|::Wait|Join" /tmp/bt1.txt | head -25
+
+            # Second snapshot 60s later. If the two are identical the process is
+            # genuinely wedged; if frames move it is livelock or very slow
+            # progress, which is a different bug.
+            sleep 60
+            sudo gdb -p "$PID" -batch \
+              -ex "set pagination off" -ex "thread apply all bt" > /tmp/bt2.txt 2>&1
+            if diff -q <(grep -oE '^#[0-9]+ +[A-Za-z_:~]+' /tmp/bt1.txt) \
+                       <(grep -oE '^#[0-9]+ +[A-Za-z_:~]+' /tmp/bt2.txt) >/dev/null; then
+              echo "  VERDICT: identical stacks 60s apart -- wedged, not slow"
+            else
+              echo "  VERDICT: stacks MOVED between snapshots -- livelock or slow progress"
+            fi
+            kill -9 "$PID" 2>/dev/null
+          else
+            wait "$PID"; rc=$?
+            echo "orlyc exited rc=$rc"
+            echo "HUNG=0" >> "$GITHUB_OUTPUT"
+            ls -la /tmp/pkgout
+          fi
+          echo "=== orlyc output:"; cat /tmp/orlyc.out || true
+
+      - name: Verdict
+        if: always()
+        run: |
+          if [ "${{ steps.repro.outputs.HUNG }}" = "1" ]; then
+            echo "orlyc HUNG on ${{ matrix.arch }} -- see backtraces above (#554)"
+          else
+            echo "orlyc completed on ${{ matrix.arch }}"
+          fi
+
+      - name: Upload output
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: orlyc-repro-${{ matrix.arch }}
+          path: |
+            /tmp/orlyc.out
+            /tmp/bt1.txt
+            /tmp/bt2.txt
+          if-no-files-found: warn

changelog.d/548-multiarch-image.md

@@ -0,0 +1,2 @@
+- **Changed**: the docker workflow builds the image on both `linux/amd64` and `linux/arm64` and, on a publish, stitches them into one multi-arch manifest — so `docker run ghcr.io/orlyatomics/orly` resolves natively on Apple Silicon instead of printing a platform-mismatch warning on the first command in the README. Each architecture is built on its own **native** runner (`ubuntu-24.04` / `ubuntu-24.04-arm`) rather than via buildx+QEMU: the image compiles the whole source tree (1753 build jobs, ~10 min natively), which under emulation would be hours and exceed the job timeout. The side benefit is that every existing gate — MCP smoke against the containerised server, in-container `orlyc` compile, REPL entrypoint mode — now runs **on the arm image too**, so the published arm64 artifact is proven to work rather than merely to build. The manifest job runs only when both legs pass, and asserts the published manifest advertises both platforms rather than assuming the stitch worked (#548).
+- **Changed**: the docker image job reports `nproc` and `free -g` alongside the architecture, and takes its build parallelism and timeout from the matrix — 4 workers / 90 min on x86, 2 workers / 180 min on arm64. The first arm attempt was killed at the 90-minute cap still inside `jhm -c release`, while the same runner class completes `make debug` in 14 minutes (ci.yml). The difference is LTO, which is memory-hungry and largely serial at link time, so if the arm runner has less RAM per core then *fewer* concurrent LTO jobs is faster rather than slower. That is a hypothesis; the capacity report is what confirms or kills it, and not having it is why the first failure was uninformative (#548).