Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: CI

on:
push:
branches: [main]
pull_request:

permissions:
contents: read

jobs:
image:
name: Build and smoke-test rust-dev
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- name: Shellcheck
run: shellcheck scripts/* tests/smoke-image
- name: Build flyingrobots/rust-dev:1.96.0
run: scripts/build-image 1.96.0
- name: Smoke test
run: tests/smoke-image 1.96.0
- name: Unchanged rebuild reuses the cache (same image ID)
run: |
first="$(docker image inspect -f '{{.Id}}' flyingrobots/rust-dev:1.96.0)"
scripts/build-image 1.96.0
second="$(docker image inspect -f '{{.Id}}' flyingrobots/rust-dev:1.96.0)"
echo "first=$first second=$second"
test "$first" = "$second"
20 changes: 18 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,24 @@ containers, never on the host.
- One base image per Rust toolchain, pinned to the version in each repository's
`rust-toolchain.toml`.
- One guarded runner that every repository calls. The runner claims `git-locks`
keys, applies a timeout and resource limits, mounts only named volumes and
binds, and audits Docker for litter before and after each run.
keys, applies a timeout and resource limits, mounts only named volumes (never
a host path), and audits Docker for litter before and after each run.

The rules are in [`docs/hygiene.md`](docs/hygiene.md). Work is tracked in GitHub
Issues with `issue-dag/2` frontmatter.

## Build and test the base image

```sh
scripts/build-image 1.96.0 # builds flyingrobots/rust-dev:1.96.0 under the git-locks keys
tests/smoke-image 1.96.0 # checks it in offline, read-only, unprivileged containers
```

Pinned base digests live in `images/rust-dev/base-digests`, one line per Rust
version. The image runs as UID 10001. Mount points for the named volumes:
`/work` (the copy), `/scratch` (inputs, logs, exports), `/target` (build output),
`/usr/local/cargo/registry`, and `/usr/local/cargo/git` (cargo caches).

The guarded runner `bin/devrun` is not here yet (flyingrobots/devcontainers#3
and #5). Until it lands, these two scripts are the only supported way to use
Docker from this repository.
65 changes: 56 additions & 9 deletions docs/hygiene.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ is hostile. The host, its git repositories, its credentials, and other projects'
Docker resources must stay unreachable and unchanged. The runner (`bin/devrun`)
enforces every rule here; a rule that only lives in this file is not enforced.

**Status:** `bin/devrun` is not merged yet (flyingrobots/devcontainers#3). Until
it is, the rules about runs (copy-in, network phases, run-phase mounts, prep,
outputs, audit) are its specification, and nothing enforces them.
`scripts/build-image` and `tests/smoke-image` enforce the rules that apply to an
image build and a smoke test.

## 1. Source isolation

- **Copy in, never mount.** No host path is bind-mounted into a container: not
Expand Down Expand Up @@ -40,9 +46,27 @@ enforces every rule here; a rule that only lives in this file is not enforced.

- Non-root user with a fixed UID (`10001`), `--user 10001:10001`.
- `--cap-drop ALL`, `--security-opt no-new-privileges`.
- `--read-only` root filesystem. Writable places: `/work` (the copy, in a tmpfs
or the scratch volume), `/tmp` (tmpfs with a size limit), and the named
volumes in section 5.
- `--read-only` root filesystem. Writable places: `/work` (the copy, at the
root of the work volume), `/scratch`, `/tmp` (tmpfs with a size limit), the
other named volumes in section 5, and Docker's own `/dev/shm` and
`/dev/mqueue`.
- **No writable directory above the copy.** Cargo reads `.cargo/config.toml` in
the working directory and in every parent directory. The copy is the root of
its own volume, so its only parent is the read-only root. Before each copy-in,
the runner empties the work and scratch volumes completely, hidden entries
included, so a `.cargo/config.toml` that one run leaves behind never reaches
the next.
- `CARGO_HOME` is the image path `/usr/local/cargo`. It is read-only only
because the run uses `--read-only`: the base image leaves it and the toolchain
world-writable. Under `--read-only`, a run cannot plant a cargo `config.toml`
or a `credentials.toml` there.
- No `RUSTUP_TOOLCHAIN` in the image or the run: a repository's
`rust-toolchain.toml` channel must match the image. Otherwise rustup tries to
install the other toolchain, and that fails on the read-only root. This
catches an honest version mismatch only. A hostile repository can still run
its own compiler, for example through a `path` toolchain or `build.rustc` in
its own `.cargo/config.toml`, so a run's output never proves which compiler
built it. The isolation rules contain such a compiler like any other code.
- Limits on every run: `--pids-limit`, `--memory` with `--memory-swap` equal to
it, `--cpus`, `--ulimit nofile`.
- Never `--privileged`, `--network host`, `--pid host`, `--ipc host`, or added
Expand All @@ -52,8 +76,19 @@ enforces every rule here; a rule that only lives in this file is not enforced.

## 3. Network

- **Phase 1, fetch:** `cargo fetch --locked` with network access, writing only
to the project's registry volume.
- **Phase 1, fetch:** `cargo fetch --locked` with network access and the cargo
caches writable. It must run no code that the repository chooses, because a
program that runs now can rewrite cached sources for every later run.
`cargo fetch` asks the compiler for target information, so a repository's
`build.rustc-wrapper`, `build.rustc`, or `path` toolchain would run here.
Therefore:
- The fetch copy contains no `.cargo/config` or `.cargo/config.toml` file
and no `rust-toolchain` or `rust-toolchain.toml` file.
- The fetch command runs with `RUSTUP_TOOLCHAIN` set to the image's version,
`RUSTC=/usr/local/cargo/bin/rustc`, and `RUSTC_WRAPPER` and
`RUSTC_WORKSPACE_WRAPPER` set empty. These override the same settings in
any configuration file.
- `cargo fetch` runs no build scripts and no procedural macros.
- **Phase 2, build and test:** `--network none` and `CARGO_NET_OFFLINE=true`. A
build script cannot download or send anything.
- A command that truly needs the network in phase 2 must say so with
Expand All @@ -64,6 +99,10 @@ enforces every rule here; a rule that only lives in this file is not enforced.

- Base images are pinned by digest (`rust@sha256:…`), not by tag. The digest is
in the receipt.
- Known gap: `clippy`, `rustfmt`, and the `wasm32-unknown-unknown` standard
library come from the rustup server at image build time. rustup checks each
file against the channel manifest for that exact version, but the manifest
itself is not pinned here. `cargo-deny` is pinned by SHA-256.
- Every cargo command uses `--locked`. A changed `Cargo.lock` leaves the
container only through an explicit patch export.
- Images build with the default builder. No new `buildx` builder. A new image is
Expand All @@ -77,13 +116,19 @@ and `org.flyingrobots.role=<role>`. Cleanup selects by these labels only.
| Resource | Name | Role label |
| --- | --- | --- |
| Base image | `flyingrobots/rust-dev:<rust-version>` | `base-image` |
| Upstream base, the build input | `rust@sha256:<digest>` from `images/rust-dev/base-digests`, pulled by digest. It carries only upstream labels. Keep it while its digest line exists; after removing the line, remove it with `docker image rm`. | none (upstream image) |
| Run container | `<project>-dev-run` (one at a time, under a lock) | `dev-run` |
| Cargo registry cache | `<project>-cargo-registry` (per project: no cross-project cache poisoning) | `cargo-registry` |
| Cargo registry cache, at `/usr/local/cargo/registry` | `<project>-cargo-registry` (per project: no cross-project cache poisoning) | `cargo-registry` |
| Cargo git cache, at `/usr/local/cargo/git` | `<project>-cargo-git` | `cargo-git` |
| Build output, iteration | `<project>-target` | `target` |
| Build output, evidence | `<project>-target-evidence`, emptied before each merge-gate run | `target-evidence` |
| Logs, test data, exports | `<project>-scratch` | `scratch` |
| The copy, at `/work` | `<project>-work`, emptied before each copy-in | `work` |
| Inputs, logs, test data, exports, at `/scratch` | `<project>-scratch`, emptied before each copy-in | `scratch` |

- No anonymous volumes. Images must not declare `VOLUME`.
- The cargo caches are writable only in the fetch phase, which runs no code
that the repository chooses (section 3). The build phase mounts them
read-only, so a hostile build script cannot poison the cache for a later run.
- **Evidence runs use a fresh target.** A merge-gate run never reuses build
output, so a stale cache cannot fabricate a passing result.

Expand Down Expand Up @@ -123,8 +168,10 @@ and `org.flyingrobots.role=<role>`. Cleanup selects by these labels only.

## 10. Locks

- Each run claims `host/heavy-work` and `host/docker/<project>-dev/` in the
workstation `git-locks` store, with a lease longer than the timeout.
- Each run claims `host/heavy-work` and `host/docker/<project>-dev-run/` in the
workstation `git-locks` store, with a lease longer than the timeout. The
second key follows the workstation registry rule
`host/docker/<exact-container-name>/`.

## 11. Not ours

Expand Down
71 changes: 71 additions & 0 deletions images/rust-dev/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Shared Rust development image for the flyingrobots repositories.
# Rules: docs/hygiene.md. Build with scripts/build-image, which passes the digest from base-digests.

# Fallback only. scripts/build-image always passes BASE_IMAGE from images/rust-dev/base-digests;
# keep this line equal to the 1.96.0 entry there.
ARG BASE_IMAGE=rust@sha256:58fe97504a0e4cbba5d85599619a589923d3e779472a6fb0840d58d1c4ba99d7
FROM ${BASE_IMAGE}

ARG RUST_VERSION=1.96.0
# Published checksums: https://github.com/EmbarkStudios/cargo-deny/releases/download/<version>/
# cargo-deny-<version>-<triple>.tar.gz.sha256
ARG CARGO_DENY_VERSION=0.20.2
ARG CARGO_DENY_SHA256_AARCH64=995c82be0defc7a025cae49a2aa2644ce8245c9a3318fc4103907c6a285e8c7d
ARG CARGO_DENY_SHA256_X86_64=9f12ed4c49936e09b48bf862b595cde2fe64fcbd9d74dfacac6131ca824c8d5f
# Set by BuildKit (the default builder). An empty value fails the build below.
ARG TARGETARCH

LABEL org.flyingrobots.owner="devcontainers" \
org.flyingrobots.role="base-image" \
org.flyingrobots.rust-version="${RUST_VERSION}" \
org.opencontainers.image.source="https://github.com/flyingrobots/devcontainers"

# The pinned base must carry the requested toolchain.
RUN test "$(rustc --version | cut -d' ' -f2)" = "${RUST_VERSION}"

# Known gap (docs/hygiene.md section 4): these components come from the rustup server at build
# time. rustup checks each one against the channel manifest for this exact version; the manifest
# itself is not pinned in this repository.
RUN rustup component add clippy rustfmt \
&& rustup target add wasm32-unknown-unknown

RUN set -eu; \
case "${TARGETARCH}" in \
arm64) triple=aarch64-unknown-linux-musl; sum="${CARGO_DENY_SHA256_AARCH64}" ;; \
amd64) triple=x86_64-unknown-linux-musl; sum="${CARGO_DENY_SHA256_X86_64}" ;; \
*) echo "unsupported or empty TARGETARCH '${TARGETARCH}' (build with BuildKit)" >&2; exit 1 ;; \
esac; \
name="cargo-deny-${CARGO_DENY_VERSION}-${triple}"; \
curl -fsSL -o /tmp/deny.tgz "https://github.com/EmbarkStudios/cargo-deny/releases/download/${CARGO_DENY_VERSION}/${name}.tar.gz"; \
echo "${sum} /tmp/deny.tgz" | sha256sum -c -; \
tar -xzf /tmp/deny.tgz -C /tmp; \
install -m 0755 "/tmp/${name}/cargo-deny" /usr/local/bin/cargo-deny; \
rm -rf /tmp/deny.tgz "/tmp/${name}"

# Unprivileged user (fixed UID per docs/hygiene.md section 2) and the mount points for the named
# volumes (section 5). New named volumes copy the ownership of these directories, so UID 10001 can
# write to them. The copy lives at /work, the root of its own volume: its only parent directory is
# the root filesystem, so no writable directory above the copy can hold a .cargo/config.toml that
# cargo would read. The work volume itself persists, so the runner must empty it before each
# copy-in (docs/hygiene.md section 2). CARGO_HOME stays at the image path /usr/local/cargo, with only its registry and
# git caches as volumes. CARGO_HOME and the toolchain are read-only only because every run uses
# --read-only: the base image leaves those directories world-writable.
RUN groupadd --gid 10001 dev \
&& useradd --uid 10001 --gid 10001 --no-create-home --home-dir /tmp/home --shell /usr/sbin/nologin dev \
&& mkdir -p /work /scratch /target /usr/local/cargo/registry /usr/local/cargo/git /tmp/home \
&& chown 10001:10001 /work /scratch /target /usr/local/cargo/registry /usr/local/cargo/git /tmp/home

# HOME is /tmp/home. The image creates it; a run that mounts a fresh tmpfs on /tmp must create it
# again (devrun and tests/smoke-image do).
# No RUSTUP_TOOLCHAIN: a repository's rust-toolchain.toml channel must match this image. Otherwise
# rustup tries to install the other toolchain, which fails on the read-only root. This catches an
# honest mismatch only: a repository can still run its own compiler (a path toolchain, build.rustc).
# docs/hygiene.md section 2. CARGO_INCREMENTAL=0 keeps audit
# builds free of incremental state. GIT_CONFIG_NOSYSTEM=1 ignores any system git config.
ENV CARGO_TARGET_DIR=/target \
CARGO_INCREMENTAL=0 \
HOME=/tmp/home \
GIT_CONFIG_NOSYSTEM=1

USER 10001:10001
WORKDIR /work
3 changes: 3 additions & 0 deletions images/rust-dev/base-digests
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Rust version, then the pinned rust image index digest (multi-arch).
# Update a line only on purpose, with the new digest from `docker buildx imagetools inspect rust:<version>`.
1.96.0 rust@sha256:58fe97504a0e4cbba5d85599619a589923d3e779472a6fb0840d58d1c4ba99d7
100 changes: 100 additions & 0 deletions scripts/build-image
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
#!/usr/bin/env bash
# Build flyingrobots/rust-dev:<rust-version> from the digest pinned in images/rust-dev/base-digests.
# On the workstation it runs under the git-locks keys host/heavy-work and host/docker/devcontainers-dev-run/
# (docs/hygiene.md section 10), with a 30-minute timeout inside a 40-minute lease (section 7), and it
# refuses to start below 50 GiB free on the host or in the Docker VM (section 8). Time budget inside
# the lease: base pull 300 s + free-space check 60 s + build 1800 s = 2160 s, below the 2400 s lease.
set -euo pipefail
version="${1:?usage: scripts/build-image <rust-version>}"
root="$(cd "$(dirname "$0")/.." && pwd)"
base="$(awk -v v="$version" '$1 == v { print $2 }' "$root/images/rust-dev/base-digests")"
if [[ -z "$base" ]]; then
echo "build-image: no pinned digest for Rust $version in images/rust-dev/base-digests" >&2
exit 2
fi
tag="flyingrobots/rust-dev:$version"
build=(docker build
--file "$root/images/rust-dev/Dockerfile"
--build-arg "BASE_IMAGE=$base"
--build-arg "RUST_VERSION=$version"
--label "org.flyingrobots.base-digest=$base"
# No provenance or SBOM attestations: they carry a per-build timestamp, so an unchanged rebuild
# would get a new image ID (issue #2 acceptance criterion: same ID on an unchanged rebuild).
--provenance=false --sbom=false
--tag "$tag"
"$root/images/rust-dev")
timeout_s=1800
limits=(--pids-limit 32 --memory 128m --memory-swap 128m --cpus 0.5 --ulimit nofile=256:256)

# Only a docker-driver builder loads the result into the local image store under its tag (section 4).
driver="$(docker buildx inspect | awk '$1 == "Driver:" { print $2; exit }')"
if [[ "$driver" != "docker" ]]; then
echo "build-image: the selected buildx builder uses driver '${driver:-unknown}'; select a docker-driver builder first" >&2
exit 4
fi

# GitHub Actions runners are single-use machines with no shared git-locks store. Keep the timeout.
if [[ "${GITHUB_ACTIONS:-}" == "true" ]]; then
exec timeout "$timeout_s" "${build[@]}"
fi

if [[ -z "${DEVCONTAINERS_LOCK_HELD:-}" ]]; then
# Default store: the shared workstation store on this macOS machine.
exec env GIT_LOCKS_STORE="${GIT_LOCKS_STORE:-/Users/Shared/git-locks/workstation.git}" DEVCONTAINERS_LOCK_HELD=1 \
git locks with --job "devcontainers-build-$(uuidgen)" --holder "${USER:-agent}-build-image" \
--ttl 2400 --wait 600 --note "build $tag" host/heavy-work host/docker/devcontainers-dev-run/ -- "$0" "$@"
fi

# perl is the timeout tool, because macOS has no coreutils timeout by default.
# deadline SECONDS COMMAND...: run COMMAND; after SECONDS, kill it with SIGKILL and return 124. A
# plain alarm is not enough: docker run and docker start -a forward SIGALRM to the container instead
# of exiting, so the client would keep waiting. Killing the client does not stop a container.
deadline() {
perl -e '$t = shift; $p = fork; die "fork: $!" unless defined $p;
if (!$p) { exec @ARGV or exit 127 }
$SIG{ALRM} = sub { kill "KILL", $p; waitpid $p, 0; exit 124 };
alarm $t; waitpid $p, 0; exit($? & 127 ? 128 + ($? & 127) : $? >> 8)' "$@"
}

if ! docker image inspect "$base" >/dev/null 2>&1; then
echo "build-image: pulling $base" >&2
deadline 300 docker pull --quiet "$base" >/dev/null
fi
# The free-space container's / is on the image store's filesystem only for overlay storage. Refuse
# other storage, such as a stargz snapshotter, rather than measure the wrong filesystem.
storage="$(docker info -f '{{.Driver}}')"
case "$storage" in
overlay2|overlayfs) ;;
*) echo "build-image: cannot check free space for Docker storage driver '$storage'; use overlay2 or overlayfs" >&2; exit 3 ;;
esac
min_kib=$((50 * 1024 * 1024))
host_free="$(df -Pk "$root" | awk 'NR == 2 { print $4 }')"
Comment thread
flyingrobots marked this conversation as resolved.
Resolved
run_name=devcontainers-dev-run
run_id="build-$(uuidgen)"
# A leftover from an interrupted run stays for inspection; this script never removes it. A failed
# listing stops the script (set -e), so an error never reads as "no leftover".
leftover="$(docker ps -a -q --filter "name=^${run_name}\$")"
if [[ -n "$leftover" ]]; then
echo "build-image: leftover container $run_name exists (interrupted run?); inspect it, then: docker rm -f $run_name" >&2
exit 3
fi
# The deadline stops only the docker client, so on failure remove the container this run started. It
# carries this run's own label, so a container that someone else created under the name stays.
vm_out="$(deadline 60 docker run --rm --pull never --name "$run_name" --network none \
--label org.flyingrobots.owner=devcontainers --label org.flyingrobots.role=dev-run \
--label "org.flyingrobots.run=$run_id" \
--read-only --cap-drop ALL --security-opt no-new-privileges --user 10001:10001 "${limits[@]}" \
--entrypoint df "$base" -Pk /)" || {
mine="$(docker ps -a -q --filter "label=org.flyingrobots.run=$run_id" || true)"
[[ -z "$mine" ]] || docker rm -f "$mine" >/dev/null 2>&1 || true
echo "build-image: could not measure Docker VM free space (failed or timed out)" >&2
exit 3
}
vm_free="$(awk 'NR == 2 { print $4 }' <<<"$vm_out")"
[[ "$vm_free" =~ ^[0-9]+$ ]] || { echo "build-image: unexpected df output: $vm_out" >&2; exit 3; }
if (( host_free < min_kib || vm_free < min_kib )); then
echo "build-image: refusing to start: less than 50 GiB free (host ${host_free} KiB, Docker VM ${vm_free} KiB)" >&2
exit 3
fi

deadline "$timeout_s" "${build[@]}"
Loading
Loading