From d36d1eb975d088990b89316786dc9c010f4b821d Mon Sep 17 00:00:00 2001 From: Benjamin Coriou Date: Sat, 16 May 2026 01:56:23 +0200 Subject: [PATCH 1/4] docs: add build/DX overhaul spec + implementation plan Kiro spec at .kiro/specs/build-devx-overhaul/ defines 11 user stories, 51 acceptance criteria, a 20-stage Dockerfile structure, compose topology, GHCR cache strategy, s6 contract change, 12 correctness properties, risk register, and a 60-task dependency-sequenced implementation checklist. Plan at docs/superpowers/plans/2026-05-16-build-devx-overhaul.md sequences 9 phases (A-I) across 3 PR boundaries, defines 8 verification gates, and flags 10 execution-time risks. This commit lands the contract; subsequent commits on this branch implement Phases A and B (s6 tree partition + Dockerfile refactor) per the spec. --- .kiro/specs/build-devx-overhaul/design.md | 884 ++++++++++++++++++ .../specs/build-devx-overhaul/requirements.md | 348 +++++++ .kiro/specs/build-devx-overhaul/tasks.md | 226 +++++ .../plans/2026-05-16-build-devx-overhaul.md | 295 ++++++ 4 files changed, 1753 insertions(+) create mode 100644 .kiro/specs/build-devx-overhaul/design.md create mode 100644 .kiro/specs/build-devx-overhaul/requirements.md create mode 100644 .kiro/specs/build-devx-overhaul/tasks.md create mode 100644 docs/superpowers/plans/2026-05-16-build-devx-overhaul.md diff --git a/.kiro/specs/build-devx-overhaul/design.md b/.kiro/specs/build-devx-overhaul/design.md new file mode 100644 index 0000000..eca144a --- /dev/null +++ b/.kiro/specs/build-devx-overhaul/design.md @@ -0,0 +1,884 @@ +# Build & DX Overhaul — Design + +## 1. Current State Diagnosis + +The current build system exhibits 14 distinct anti-patterns. This section +enumerates each, names the mechanism in the new design that retires it, and +identifies the affected file(s). + +| # | Anti-pattern | New mechanism that retires it | Affected files | +|----|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------| +| 1 | Double pnpm install in `node-build` (lines 426–438): isolated linker, then reconfigure to hoisted, then re-install | Single `pnpm install --frozen-lockfile --prod=false` with project default linker (`isolated` per `.npmrc`). The hoisted reconfigure block is deleted. | `Dockerfile`, `.npmrc` (no change; already isolated) | +| 2 | `final-core` copies entire s6 tree then `rm -rf`'s `sdrpp-server` paths (lines 674–680) | s6 tree partition: canonical `docker/overlay/s6-overlay/` excludes sdrpp-server entirely. `final` adds it from `docker/overlay/s6-overlay-sdrpp/`. `wavekit-api → sdrpp-server` hard dep is removed from the canonical tree (semantic change documented in §5). | `Dockerfile`, `docker/overlay/s6-overlay/` | +| 3 | `base-deps` god-layer installs build deps for all 11 decoders in one RUN | Thin `base-build` stage with only toolchain commons. Each `*-build` stage installs its own deps with `--mount=type=cache,target=/var/cache/apt,sharing=locked`. | `Dockerfile` | +| 4 | Unpinned `git clone --depth 1` against branch tips for 10 of 11 decoders | Every decoder build stage declares `ARG _REF=` at top of Dockerfile and checks out the pinned ref via `git fetch --depth 1 origin "${REF}" && git checkout --detach FETCH_HEAD`. Lora's existing pattern (line 282) is the template. | `Dockerfile` | +| 5 | No `TARGETPLATFORM`/`TARGETARCH` awareness; arm64 multi-arch builds run under QEMU end-to-end | Every stage that does arch-conditional work declares `ARG TARGETARCH` per sdr-host's pattern. `runtime-base` switches s6 platform detection from `BUILDPLATFORM` (wrong) to `TARGETARCH`. `final-core`'s hardcoded `x86_64-linux-gnu` ncurses path is removed (§6). | `Dockerfile` | +| 6 | `docker/build.sh` invokes plain `docker build`, bypassing buildx, multi-arch, registry cache, and provenance | `make docker-build` invokes `docker buildx bake` against `docker/bake.hcl`. `docker/build.sh` is replaced or rewritten as a thin wrapper that calls `buildx bake`. No script under `docker/` invokes plain `docker build`. | `docker/build.sh`, `docker/bake.hcl` (new), `Makefile` | +| 7 | Two parallel dev flows: `make dev-up` (Makefile-native, plain `docker run`) and `make docker-dev` (compose) | Single compose with `dev` profile. Native TypeScript loop becomes `pnpm dev` (no Docker). `make dev-stack` aliases `docker compose --profile dev up --build`. `make dev-up`/`dev-start`/`dev-stop`/`dev-restart` and `make docker-dev` are deleted. | `Makefile`, `compose.yaml` | +| 8 | `docker-compose.dev.yml` defines `nginx-reverse-proxy` and `codercom/code-server` services that nobody uses | Both services are deleted; not reintroduced in `compose.yaml`. | `compose.yaml` | +| 9 | `docker-compose.demod-test.yml` + `docker/Dockerfile.demod-test` rebuild dsd-fme/multimon-ng/csdr/rtl-sdr from scratch (~600MB duplication) | New `final-demod` stage in the main Dockerfile COPIES binaries from existing `*-build` stages. Compose service `demod-test` lives under the `demod-test` profile. | `Dockerfile`, `compose.yaml`, deletion: `docker/Dockerfile.demod-test`, `docker-compose.demod-test.yml` | +| 10 | CI never exercises Docker (current `.github/workflows/ci.yml` is 26 lines of pnpm lint/typecheck/build) | New `docker-build` CI job depends on the existing pnpm job, sets up QEMU + buildx, runs `docker buildx bake --target ci-core` on PR (no push), on main builds `default` group with `--push` and `--cache-to mode=max`. | `.github/workflows/ci.yml` | +| 11 | tsconfig drift: root `tsconfig.json` has `composite: false` and excludes `cli`/`packages`; `tsconfig.base.json` has `composite: true` | Out-of-scope (CLAUDE.md "Don't propose unrelated refactoring"). Noted in `tasks.md` as a follow-up. | n/a | +| 12 | `.dockerignore` lists build caches near the bottom; effectiveness unverified | Audit step in `tasks.md` verifies `.turbo/`, `.pnpm-store/`, `.docker-cache/` are effectively excluded from context. The `.docker-cache` entry stays as defensive. | `.dockerignore` | +| 13 | `./.docker-cache` local cache directory is per-machine, unshared | Replaced by GHCR registry cache (§4). `docker/init.sh` no longer creates `.docker-cache`. Compose `cache_from`/`cache_to` references rewritten to `type=registry`. | `docker/init.sh`, `compose.yaml`, all old compose files (deleted) | +| 14 | `runtime-base` apt cache mount caches `/var/lib/apt` (rebased by subsequent `rm -rf`) | Apt cache mounts SHALL only target `/var/cache/apt`. The `/var/lib/apt` cache mount is dropped. `rm -rf /var/lib/apt/lists/*` stays. | `Dockerfile` | + +## 2. Target Dockerfile Structure + +### 2.1 Stage Inventory + +20 named stages in a single Dockerfile. + +**Toolchain & runtime bases (2):** + +- `base-build` (from `debian:bookworm-slim`): build-essential, cmake, git, + pkg-config, ca-certificates, curl. Nothing decoder-specific. +- `runtime-base` (from `debian:bookworm-slim`): minimal runtime libs + + s6-overlay v3.1.6.2 installed via `TARGETARCH`-conditioned curl. Includes + the audio/SDR runtime libs the decoders dynamically link against. **No + cache mount on `/var/lib/apt`.** + +**Decoder/binary builders (12):** + +Each builds on `base-build`, declares the apt deps it needs in its own +`RUN apt-get install` (with `/var/cache/apt` cache mount), and uses a pinned +`ARG _REF` for the upstream checkout. + +- `sdrpp-build` — REF: `SDRPP_REF` (current code clones nightly branch; + pin to a tested SHA at refactor time). +- `dsd-fme-build` — builds `mbelib` first (pinned `MBELIB_REF`), then dsd-fme + (pinned `DSDFME_REF`). +- `multimon-ng-build` — REF: `MULTIMON_NG_REF`. +- `rtl433-build` — REF: `RTL_433_REF`. +- `acarsdec-build` — REF: `ACARSDEC_REF`. +- `ais-catcher-build` — REF: `AIS_CATCHER_REF`. +- `direwolf-build` — REF: `DIREWOLF_REF`. +- `dumpvdl2-build` — builds `libacars` first (pinned `LIBACARS_REF`), then + dumpvdl2 (pinned `DUMPVDL2_REF`). +- `readsb-build` — REF: `READSB_REF`. +- `soapy-rtltcp-build` — REF: `SOAPY_RTLTCP_REF`. +- `csdr-build` — REF: `CSDR_REF`. +- `lora-build` — REF: `GR_LORA_SDR_REF` (already pinned; pattern source). + +**Application builder (1):** + +- `node-build` (from `node:22-bookworm-slim`): Corepack-prepared pnpm + 10.28.0. Copies workspace manifests + lockfile, runs `pnpm install + --frozen-lockfile --prod=false` exactly once, copies sources, runs + `pnpm typecheck` and `pnpm build`, runs `pnpm prune --prod`. Uses Turbo + cache mount + `node_modules/.cache` cache mount. No linker reconfiguration. + +**Final composites (5):** + +- `final-base` (from `runtime-base`): python3 + gnuradio + python3-numpy + + python3-protobuf (drop python3-cryptography if unused) + decoders + csdr + + soapy-rtltcp + lora artifacts + node runtime + app dist + scripts + + canonical s6 overlay (`docker/overlay/s6-overlay/`, which contains + wavekit-init + wavekit-api + base + services-without-sdrpp + user-without-sdrpp). + HEALTHCHECK on `wavekit-api`. `ENTRYPOINT ["/init"]`. +- `final` (from `final-base`): adds SDR++ binaries from `sdrpp-build` + + `docker/overlay/s6-overlay-sdrpp/` (sdrpp-server service dir + the two + `contents.d/sdrpp-server` registration files). EXPOSE adds 5259/7355. + LABEL mode=full. +- `final-core` (from `final-base`): adds nothing. LABEL mode=core. This is a + one-line stage. +- `final-sdrpp` (from `runtime-base`): SDR++ binaries + a minimal s6 overlay + with just the sdrpp-server service. Existing target, gets the `TARGETARCH` + s6 fix. +- `final-demod` (from `runtime-base` plus minimal apt: sox, ffmpeg, + netcat-openbsd, vim, python3 + numpy via apt): COPIES dsd-fme + multimon-ng + + csdr + rtl-sdr binaries from the existing `*-build` stages. `WORKDIR + /workspace`. `CMD ["/bin/bash"]`. Replaces `docker/Dockerfile.demod-test`. + +### 2.2 Stage Dependency Graph + +``` +debian:bookworm-slim + | + |-- base-build ----+ + | +-- sdrpp-build ---------------\ + | +-- dsd-fme-build --------------\ + | +-- multimon-ng-build -----------\ + | +-- rtl433-build -----------------\ + | +-- acarsdec-build ----------------\ + | +-- ais-catcher-build --------------\ + | +-- direwolf-build ------------------\ + | +-- dumpvdl2-build -------------------\ + | +-- readsb-build ----------------------\ + | +-- soapy-rtltcp-build ------------------>--+ + | +-- csdr-build --------------------------/ | + | +-- lora-build -------------------------/ | + | | + |-- runtime-base ----------- final-base <---------- node-build | + | ^ | + | +-(COPY --from for all)------+ + | + +-- final (adds SDR++ + sdrpp overlay) + +-- final-core (no additions) + +-- final-demod (from runtime-base; COPIES tools) + +-- final-sdrpp (from runtime-base; SDR++ only) +``` + +### 2.3 Pinned ARG Block + +Top of Dockerfile contains a single ARG block. Each ref is a 40-char SHA OR +a tag. Renovate/Dependabot can bump these. + +Implementation captures HEAD of each repo's main/master at refactor time +(see task B.1.1) and pins to that SHA. Bumps thereafter go through PR +review. The schema is: + +```dockerfile +# Upstream refs (SHA or tag) for reproducible decoder builds. +# Initial SHAs captured during implementation per tasks.md §B.1.1. +ARG S6_OVERLAY_VERSION=3.1.6.2 +ARG SDRPP_REF=<40-char sha, captured at impl-time> +ARG MBELIB_REF=<40-char sha, captured at impl-time> +ARG DSDFME_REF=<40-char sha, captured at impl-time> +ARG MULTIMON_NG_REF=<40-char sha, captured at impl-time> +ARG RTL_433_REF=<40-char sha, captured at impl-time> +ARG ACARSDEC_REF=<40-char sha, captured at impl-time> +ARG AIS_CATCHER_REF=<40-char sha, captured at impl-time> +ARG DIREWOLF_REF=<40-char sha, captured at impl-time> +ARG LIBACARS_REF=<40-char sha, captured at impl-time> +ARG DUMPVDL2_REF=<40-char sha, captured at impl-time> +ARG READSB_REF=<40-char sha, captured at impl-time> +ARG SOAPY_RTLTCP_REF=<40-char sha, captured at impl-time> +ARG CSDR_REF=<40-char sha, captured at impl-time> +ARG GR_LORA_SDR_REF=862746dd1cf635c9c8a4bfbaa2c3a0ec3a5306c9 +``` + +## 3. Compose Topology + +One file: `compose.yaml` at the repo root. + +```yaml +name: wavekit + +x-cache-refs: &cache-refs + base-build: type=registry,ref=ghcr.io/coriou/wavekit:cache-base-build + # (one entry per cached stage; HCL-equivalent in bake.hcl) + +services: + + # ---------- dev profile ---------- + sdrpp-server: + profiles: ["dev"] + image: wavekit:dev-sdrpp + build: + context: . + target: final-sdrpp + cache_from: + - type=registry,ref=ghcr.io/coriou/wavekit:cache-final-sdrpp + # cache_to omitted for local builds; only CI on main writes cache + container_name: wavekit-sdrpp + ports: ["5259:5259"] + networks: [wavekit] + healthcheck: + test: ["CMD", "curl", "-fsS", "http://localhost:5259/"] + interval: 10s; timeout: 5s; retries: 3; start_period: 20s + + wavekit-api: + profiles: ["dev"] + image: wavekit:dev-core + build: + context: . + target: final-core + cache_from: + - type=registry,ref=ghcr.io/coriou/wavekit:cache-final-core + - type=registry,ref=ghcr.io/coriou/wavekit:cache-final-base + - type=registry,ref=ghcr.io/coriou/wavekit:cache-node-build + container_name: wavekit-api + depends_on: + sdrpp-server: { condition: service_healthy } + environment: + WAVEKIT_LOG_LEVEL: debug + NODE_ENV: development + SDR_SOURCE: "tcp://sdrpp-server:5259" + ports: + - "9000:9000" + - "8080:8080" + - "8081:8081" + - "4713:4713" + networks: [wavekit] + volumes: + - ./config:/app/config:ro + - ./logs:/var/log/wavekit + - ./decoded_calls:/app/decoded_calls + healthcheck: + test: ["CMD", "curl", "-fsS", "http://localhost:9000/health"] + interval: 10s; timeout: 5s; retries: 3; start_period: 30s + cap_add: [SYS_NICE, NET_RAW] + + # ---------- prod-single-host profile ---------- + wavekit-full: + profiles: ["prod-single-host"] + image: ghcr.io/coriou/wavekit:latest + container_name: wavekit + restart: unless-stopped + environment: + WAVEKIT_LOG_LEVEL: info + RTL_TCP_HOST: ${RTL_TCP_HOST:-127.0.0.1} + RTL_TCP_PORT: ${RTL_TCP_PORT:-1234} + ports: ["9000:9000", "8080:8080", "5259:5259", "4713:4713"] + networks: [wavekit] + volumes: + - wavekit-config:/app/config + - wavekit-logs:/var/log/wavekit + - recordings:/recordings + healthcheck: { test: ["CMD","curl","-fsS","http://localhost:9000/health"], interval: 30s, ... } + deploy: + resources: + limits: { cpus: "2", memory: 1G } + cap_add: [SYS_NICE, NET_RAW] + security_opt: ["no-new-privileges:true"] + + # ---------- prod-distributed profile ---------- + wavekit-sdrpp-prod: + profiles: ["prod-distributed"] + image: ghcr.io/coriou/wavekit:latest-sdrpp + # ... + wavekit-core-prod: + profiles: ["prod-distributed"] + image: ghcr.io/coriou/wavekit:latest-core + depends_on: [wavekit-sdrpp-prod] + environment: + SDR_SOURCE: "tcp://wavekit-sdrpp-prod:5259" + # ... + + # ---------- demod-test profile ---------- + demod-test: + profiles: ["demod-test"] + image: wavekit:dev-demod + build: + context: . + target: final-demod + cache_from: + - type=registry,ref=ghcr.io/coriou/wavekit:cache-final-demod + volumes: + - ./debug_audio:/data/debug_audio + - ./scripts:/scripts + - ./output:/output + working_dir: /workspace + stdin_open: true + tty: true + +networks: + wavekit: + driver: bridge + +volumes: + wavekit-config: + wavekit-logs: + recordings: +``` + +**Files deleted as part of this refactor**: + +- `docker-compose.dev.yml` +- `docker-compose.prod.yml` +- `docker-compose.override.yml` +- `docker-compose.demod-test.yml` +- `docker/Dockerfile.demod-test` + +**Files unchanged**: `packages/sdr-host/docker-compose.yml` (different +deployment target — runs on the Pi). + +## 4. Build Cache Strategy + +### 4.1 Registry refs (canonical list) + +One BuildKit cache ref per stage. Naming: `cache-` exactly as the +stage appears in `FROM ... AS `. + +``` +ghcr.io/coriou/wavekit:cache-base-build +ghcr.io/coriou/wavekit:cache-runtime-base +ghcr.io/coriou/wavekit:cache-sdrpp-build +ghcr.io/coriou/wavekit:cache-dsd-fme-build +ghcr.io/coriou/wavekit:cache-multimon-ng-build +ghcr.io/coriou/wavekit:cache-rtl433-build +ghcr.io/coriou/wavekit:cache-acarsdec-build +ghcr.io/coriou/wavekit:cache-ais-catcher-build +ghcr.io/coriou/wavekit:cache-direwolf-build +ghcr.io/coriou/wavekit:cache-dumpvdl2-build +ghcr.io/coriou/wavekit:cache-readsb-build +ghcr.io/coriou/wavekit:cache-soapy-rtltcp-build +ghcr.io/coriou/wavekit:cache-csdr-build +ghcr.io/coriou/wavekit:cache-lora-build +ghcr.io/coriou/wavekit:cache-node-build +ghcr.io/coriou/wavekit:cache-final-base +ghcr.io/coriou/wavekit:cache-final +ghcr.io/coriou/wavekit:cache-final-core +ghcr.io/coriou/wavekit:cache-final-sdrpp +ghcr.io/coriou/wavekit:cache-final-demod +``` + +### 4.2 Cache behaviour + +- **Local builds** (`make docker-build`): `--cache-from=type=registry, + ref=` is passed via bake. No `--cache-to`. Cache misses + fall through to local layer cache then to a clean rebuild. +- **CI on PR**: `--cache-from` only. No login, no push, no `--cache-to`. +- **CI on main push**: `--cache-from` AND `--cache-to=type=registry, + ref=,mode=max`. `mode=max` caches every intermediate layer, not just + the final manifest, so future cold builds get full hits. +- **Multi-arch**: bake builds `linux/amd64` and `linux/arm64` simultaneously + via the docker-container buildx driver. Each cache ref stores both arches. + +### 4.3 docker/bake.hcl shape + +```hcl +variable "REGISTRY" { default = "ghcr.io/coriou/wavekit" } +variable "TAG" { default = "latest" } +variable "CACHE_FROM_ONLY" { default = "true" } # CI overrides to false on main + +# Per-stage cache spec (HCL function reused across targets) +function "cache" { + params = [stage] + result = [ + "type=registry,ref=${REGISTRY}:cache-${stage}", + ] +} + +target "_base" { + context = "." + dockerfile = "Dockerfile" + platforms = ["linux/amd64", "linux/arm64"] + # cache_from set per-target below +} + +target "final" { + inherits = ["_base"] + target = "final" + tags = ["${REGISTRY}:${TAG}"] + cache-from = concat( + cache("final"), cache("final-base"), cache("node-build"), + cache("sdrpp-build"), cache("dsd-fme-build"), cache("multimon-ng-build"), + cache("rtl433-build"), cache("acarsdec-build"), cache("ais-catcher-build"), + cache("direwolf-build"), cache("dumpvdl2-build"), cache("readsb-build"), + cache("soapy-rtltcp-build"), cache("csdr-build"), cache("lora-build"), + cache("base-build"), cache("runtime-base"), + ) + cache-to = CACHE_FROM_ONLY == "true" ? [] : [ + "type=registry,ref=${REGISTRY}:cache-final,mode=max", + ] +} + +target "final-core" { + inherits = ["_base"] + target = "final-core" + tags = ["${REGISTRY}:${TAG}-core"] + cache-from = [ /* same upstream chain minus sdrpp-build */ ] + cache-to = CACHE_FROM_ONLY == "true" ? [] : [ + "type=registry,ref=${REGISTRY}:cache-final-core,mode=max", + ] +} + +target "final-sdrpp" { ... } +target "final-demod" { ... } + +# CI helper: only final-core for PR sanity +target "ci-core" { + inherits = ["final-core"] +} + +group "default" { + targets = ["final", "final-core", "final-sdrpp"] +} + +group "demod" { + targets = ["final-demod"] +} +``` + +## 5. s6 Service Contract Change + +### 5.1 What changes + +The canonical s6 tree (`docker/overlay/s6-overlay/s6-rc.d/`) currently +contains: + +``` +base/ +sdrpp-server/ +sdrpp-server/dependencies.d/wavekit-init +wavekit-init/ +wavekit-api/ +wavekit-api/dependencies.d/sdrpp-server <-- HARD s6 DEP, removed +wavekit-api/dependencies.d/wavekit-init +user/contents.d/sdrpp-server <-- moved to sdrpp overlay +user/contents.d/wavekit-api +user/contents.d/wavekit-init +user/contents.d/services +services/contents.d/sdrpp-server <-- moved to sdrpp overlay +services/contents.d/wavekit-api +services/contents.d/wavekit-init +``` + +After refactor, the canonical tree (used by `final-base`, shipped in +`final-core`) becomes: + +``` +docker/overlay/s6-overlay/s6-rc.d/ +├── base/ +├── wavekit-init/ +├── wavekit-api/ +│ └── dependencies.d/wavekit-init <-- sdrpp-server entry GONE +├── user/contents.d/wavekit-api +├── user/contents.d/wavekit-init +├── user/contents.d/services +└── services/contents.d/wavekit-api + services/contents.d/wavekit-init +``` + +A new sibling overlay (used by `final`) contains the sdrpp-server-only bits: + +``` +docker/overlay/s6-overlay-sdrpp/s6-rc.d/ +├── sdrpp-server/ +│ ├── type, run, finish +│ └── dependencies.d/wavekit-init +├── user/contents.d/sdrpp-server +└── services/contents.d/sdrpp-server +``` + +`final` does: + +```dockerfile +COPY docker/overlay/s6-overlay-sdrpp/s6-rc.d /etc/s6-overlay/s6-rc.d +``` + +This COPY is **additive** — paths under `user/contents.d/` and +`services/contents.d/` are files (one per registered service), so two +separate trees union cleanly without conflict. + +### 5.2 Runtime contract + +The current hard dep `wavekit-api → sdrpp-server` is over-tight. In `final` +mode it adds nothing the runtime contract doesn't already cover; in +`final-core` mode it has to be deleted post-hoc, which is the anti-pattern +we're retiring. + +Loosening to a soft start-order: + +- **Presence-based supervision**: a service is supervised iff its directory + exists under `/etc/s6-overlay/s6-rc.d/` AND it's registered in + `services/contents.d/`. `final-base` ships only wavekit-init and + wavekit-api. `final` additionally ships sdrpp-server. +- **No hard s6 dep**: wavekit-api waits ONLY for wavekit-init. SDR-source + reachability is handled by the application layer. +- **App-layer guarantee**: `SourceManager` (CLAUDE.md, "Stream pipeline" + section) already handles SDR source unavailability via exponential backoff + on `SourceConnectionError`. The application's reconnect contract is the + authoritative source-availability mechanism. The s6 hard dep duplicates + this guarantee at a layer that has to be hacked out for non-sdrpp deployment + modes. +- **Boot-order observation**: in `final` mode, both wavekit-api and + sdrpp-server become RUNNING under s6 once wavekit-init is up. If + sdrpp-server takes longer to bind its port than wavekit-api takes to dial + it, the api retries until success — same code path that handles a remote + sdrpp going away mid-run. + +### 5.3 Acceptance test (verifiable post-build) + +```bash +docker run --rm wavekit:latest-core ls /etc/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/ +# Expected output: wavekit-init (only) + +docker run --rm wavekit:latest-core find /etc/s6-overlay -name 'sdrpp-server*' -o -path '*/sdrpp-server*' +# Expected output: (empty) + +docker run --rm wavekit:latest find /etc/s6-overlay -name 'sdrpp-server' +# Expected: /etc/s6-overlay/s6-rc.d/sdrpp-server (one match) +``` + +## 6. Migration Plan + +### 6.1 Files DELETED + +- `/Users/ben/Projects/wavekit/docker-compose.dev.yml` +- `/Users/ben/Projects/wavekit/docker-compose.prod.yml` +- `/Users/ben/Projects/wavekit/docker-compose.override.yml` +- `/Users/ben/Projects/wavekit/docker-compose.demod-test.yml` +- `/Users/ben/Projects/wavekit/docker/Dockerfile.demod-test` +- `/Users/ben/Projects/wavekit/docker/README.md` (optional: reduce to a + one-line pointer instead of full deletion) +- `/Users/ben/Projects/wavekit/docker/overlay/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/sdrpp-server` +- `/Users/ben/Projects/wavekit/docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/` (entire dir; moved) +- `/Users/ben/Projects/wavekit/docker/overlay/s6-overlay/s6-rc.d/services/contents.d/sdrpp-server` (moved) +- `/Users/ben/Projects/wavekit/docker/overlay/s6-overlay/s6-rc.d/user/contents.d/sdrpp-server` (moved) + +### 6.2 Files ADDED + +- `/Users/ben/Projects/wavekit/compose.yaml` (canonical compose with profiles) +- `/Users/ben/Projects/wavekit/docker/bake.hcl` (declarative build matrix) +- `/Users/ben/Projects/wavekit/docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/` (mirrors the deleted canonical-tree entry; same contents) +- `/Users/ben/Projects/wavekit/docker/overlay/s6-overlay-sdrpp/s6-rc.d/services/contents.d/sdrpp-server` +- `/Users/ben/Projects/wavekit/docker/overlay/s6-overlay-sdrpp/s6-rc.d/user/contents.d/sdrpp-server` + +### 6.3 Files EDITED + +- `/Users/ben/Projects/wavekit/Dockerfile` — top-to-bottom refactor per §2 +- `/Users/ben/Projects/wavekit/Makefile` — per §7 +- `/Users/ben/Projects/wavekit/.github/workflows/ci.yml` — per §8 +- `/Users/ben/Projects/wavekit/.dockerignore` — verify, no functional change +- `/Users/ben/Projects/wavekit/docker/build.sh` — rewrite as thin + `buildx bake` wrapper OR delete (Makefile calls bake directly) +- `/Users/ben/Projects/wavekit/docker/push.sh` — drop docker.io path; GHCR-only +- `/Users/ben/Projects/wavekit/docker/init.sh` — drop `.docker-cache` mkdir; + keep buildx-builder setup +- `/Users/ben/Projects/wavekit/docker/platform-utils.sh` — delete `build_multiarch` (replaced by bake); keep `detect_platform` if used elsewhere, else delete the file +- `/Users/ben/Projects/wavekit/packages/sdr-host/Dockerfile` — line 47 Corepack fix (Requirement 9.1) +- `/Users/ben/Projects/wavekit/docs/DOCKER-SETUP.md` — rewrite for new workflow +- `/Users/ben/Projects/wavekit/CLAUDE.md` — condense "Day-to-day dev (Docker)" section + +## 7. Makefile Target Inventory + +### 7.1 New canonical targets (post-refactor) + +``` +help # default; lists targets + +# Native dev loop (no Docker) +dev # alias to `pnpm dev` +dev-dashboard # alias to `pnpm dev:dashboard` (already exists; renames) +dev-dashboard-build # unchanged +dev-configs # unchanged + +# Container integration (full stack) +dev-stack # docker compose --profile dev up --build +dev-stack-down # docker compose --profile dev down +dev-stack-logs # docker compose --profile dev logs -f +dev-shell # docker compose --profile dev exec wavekit-api /bin/bash +dev-status # docker compose --profile dev ps + curl /health + +# Build / push +docker-init # idempotent buildx builder + buildkit.toml install +docker-build # docker buildx bake --file docker/bake.hcl default +docker-push # docker buildx bake --file docker/bake.hcl default --push (CACHE_FROM_ONLY=false) +docker-clean # docker compose --profile prod-distributed down -v; remove volumes +docker-prune # docker system prune helper + +# Demod tooling +demod-test # docker compose --profile demod-test run --rm demod-test + +# Pi-hosted SDR (unchanged, separate concern) +sdr-host-build # unchanged +sdr-host-build-multi # unchanged +sdr-host-install # unchanged +sdr-host-init # unchanged +sdr-host-up # unchanged +sdr-host-update # unchanged +sdr-host-down # unchanged +sdr-host-restart # unchanged +sdr-host-logs # unchanged +sdr-host-status # unchanged +sdr-host-health # unchanged +sdr-host-compose-update # unchanged +sdr-host-clean # unchanged + +# Fixtures (unchanged) +fixtures-download +fixtures-download-all +fixtures-convert +fixtures-test +fixtures-test-local +``` + +### 7.2 Deletions + +``` +docker-build (old per-mode) docker-build-full docker-build-core docker-build-sdrpp +docker-dev docker-prod docker-compose-up docker-compose-down docker-compose-logs +docker-run-core docker-test docker-test-coverage docker-lint +demo install-buildx docker-logs-api docker-logs-sdrpp +docker-logs-decoders docker-info docker-inspect docker-history +dev-up dev-build dev-start dev-stop dev-restart +dev-logs dev-logs-raw dev-audio dev-debug-audio +``` + +`docker-logs` and `docker-shell` get renamed to `dev-stack-logs` and +`dev-shell` (still work; behaviour matches name). + +## 8. CI Workflow Shape + +`.github/workflows/ci.yml` after refactor: + +```yaml +name: CI + +on: [push, pull_request] + +jobs: + lint-typecheck-test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: "22" + - name: Enable Corepack + run: corepack enable + - name: Install dependencies + run: pnpm install --frozen-lockfile + - name: Lint + run: pnpm run lint + - name: Typecheck + run: pnpm run typecheck + - name: Build + run: pnpm run build + - name: Test + run: pnpm test + + docker-build: + needs: lint-typecheck-test + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + steps: + - uses: actions/checkout@v4 + - uses: docker/setup-qemu-action@v3 + - uses: docker/setup-buildx-action@v3 + with: + driver: docker-container + - name: Log in to GHCR (main only) + if: github.event_name == 'push' && github.ref == 'refs/heads/main' + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - name: Build (PR — cache-from only) + if: github.event_name == 'pull_request' + uses: docker/bake-action@v5 + with: + files: docker/bake.hcl + targets: ci-core + set: | + *.platform=linux/amd64,linux/arm64 + *.cache-to= + - name: Build & push (main) + if: github.event_name == 'push' && github.ref == 'refs/heads/main' + uses: docker/bake-action@v5 + with: + files: docker/bake.hcl + targets: default + push: true + set: | + *.platform=linux/amd64,linux/arm64 + CACHE_FROM_ONLY=false +``` + +PR cycle: ~15-25 min for cold cache, ~2-5 min with cache hits. + +Main push cycle: ~25-40 min cold, similar warm. + +## 9. DevX Flows + +### 9.1 `pnpm dev` — native TypeScript loop + +``` +contributor edits src/foo.ts + ↓ +esbuild watcher rebuilds dist/index.js (~150 ms) + ↓ +node --watch detects file change, restarts (~300 ms) + ↓ +fastify listens on :9000 +``` + +`pnpm dev` is defined in `package.json` (already exists, see line 23). The +script is `concurrently --raw "pnpm:dev:*"` which runs `dev:build` +(esbuild watch) and `dev:run` (node --watch) in parallel. + +Prerequisites: contributor has an SDR source reachable (rtl_tcp running on +local machine, or sdrpp on a remote Pi, or `tcp://localhost:1234`). The +config file selects which. + +### 9.2 `make dev-stack` — full container stack + +``` +make dev-stack + ↓ +docker buildx bake (via compose) builds final-core + final-sdrpp + (cache-from GHCR registry; first build pulls cached layers) + ↓ +compose starts sdrpp-server, then wavekit-api (depends_on healthy) + ↓ +contributor curl localhost:9000/health +``` + +`make dev-stack` is the right tool for: end-to-end decoder testing, +verifying the s6 service tree, validating multi-arch behaviour, and +exercising the runtime image as it ships. + +## 10. Correctness Properties + +These are the invariants that must hold post-refactor. Each maps to either +an automated test in CI or a manual verification step in `tasks.md`'s +Non-negotiables. + +### Property 1: Decoder-isolated cache invalidation + +**Statement**: Editing the upstream ref or apt deps of one decoder build +stage SHALL NOT invalidate any other decoder build stage's cache. + +**Validates**: Requirements 3.1, 3.2. + +**Verification**: After warm build, bump `READSB_REF` to a different SHA and +re-run `make docker-build`. Observe via `--progress=plain` that only the +`readsb-build` stage re-executes; `dsd-fme-build`, `multimon-ng-build`, etc. +report `CACHED`. + +### Property 2: Per-decoder apt deps live in their decoder's stage + +**Statement**: No apt package outside the toolchain commons (`base-build`) +SHALL be installed in `base-build`. Decoder-specific apt deps SHALL only +appear in the corresponding `-build` stage. + +**Validates**: Requirements 3.1, 3.2. + +**Verification**: `grep -E '^\s*lib(itpp|fftw3|asound2|sqlite3|zmq3|opus|vorbis|flac|av(format|codec)|hackrf|airspy|bladerf|samplerate|gps|hamlib)' Dockerfile` SHALL return matches only in `-build` stages, not in `base-build` or `runtime-base`. (Runtime equivalents like `libitpp8v5` ARE expected in `runtime-base`.) + +### Property 3: TypeScript edit does not trigger decoder rebuilds + +**Statement**: After a warm build, editing any file under `src/` and +re-running `make docker-build` SHALL re-execute only `node-build`, +`final-base`, and the affected `final-*` stages; every decoder build stage +SHALL report `CACHED`. + +**Validates**: Requirements 3.1, 3.2, 3.7, 4.1. + +**Verification**: Touch `src/decoders/registry.ts`, run `make docker-build`, +inspect `--progress=plain` output. + +### Property 4: Warm-cache no-op build + +**Statement**: A second consecutive `make docker-build` invocation with no +source changes SHALL report `CACHED` for every step and produce no new +intermediate images. + +**Validates**: Requirements 4.1, 4.2. + +**Verification**: `time make docker-build` twice in a row. Second invocation +SHALL complete in under 30 seconds and write zero MB of new image layers. + +### Property 5: Cold-cache pull from GHCR matches warm local build + +**Statement**: A `docker system prune -af && make docker-build` invocation +on a freshly-pruned machine that has `docker login ghcr.io` SHALL complete +without re-executing any decoder build stage (all stages SHALL be pulled as +cache). + +**Validates**: Requirements 4.1, 4.2, 4.3. + +**Verification**: Prune, build, observe `--progress=plain`: every +`*-build` stage SHALL report `CACHED [linux/amd64]` (or `[linux/arm64]`) with +a "transferring cache" line, not "Running". + +### Property 6: Cold-cache without auth still succeeds + +**Statement**: On a fresh machine WITHOUT `docker login ghcr.io`, +`make docker-build` SHALL succeed (slowly) by falling through to local +rebuilds. + +**Validates**: Requirement 4.3. + +**Verification**: `docker logout ghcr.io`, prune, build, observe no auth +error in output; build completes. + +### Property 7: No `pnpm dev` Docker dependency + +**Statement**: `pnpm dev` SHALL succeed on a machine where the Docker +daemon is stopped (`sudo systemctl stop docker` on Linux, +`OrbStack/Docker.app` quit on macOS). + +**Validates**: Requirements 1.1, 1.2. + +**Verification**: Stop Docker, run `pnpm dev`, observe esbuild and node +both running, observe `curl http://localhost:9000/health` returns 200 (with +SDR source mocked or pre-running). + +### Property 8: final-core image contains no sdrpp residue + +**Statement**: `docker run --rm wavekit:latest-core find /etc/s6-overlay +-iname '*sdrpp*'` SHALL produce empty output. `docker run --rm +wavekit:latest-core ls /usr/local/bin/ | grep -i sdrpp` SHALL produce empty +output. + +**Validates**: Requirements 3.9, 3.10, 3.11. + +**Verification**: `docker run --rm wavekit:latest-core sh -c 'find +/etc/s6-overlay -iname "*sdrpp*" ; ls /usr/local/bin/ | grep -i sdrpp'` +prints nothing. + +### Property 9: wavekit-api dependency graph is minimal in final-core + +**Statement**: `docker run --rm wavekit:latest-core ls +/etc/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/` SHALL output exactly +one line: `wavekit-init`. + +**Validates**: Requirements 3.10, 3.11, §5 contract. + +**Verification**: Command above. Output `wavekit-init` and nothing else. + +### Property 10: Multi-arch images land in registry on main push + +**Statement**: After a push to `main`, `docker manifest inspect +ghcr.io/coriou/wavekit:latest-core` SHALL list manifests for both +`linux/amd64` and `linux/arm64`. + +**Validates**: Requirements 5.4, 7.4. + +**Verification**: Command above. Both arches appear. + +### Property 11: No plain `docker build` in scripts + +**Statement**: `grep -rn 'docker build' docker/ Makefile` (excluding +`docker buildx build`) SHALL produce zero matches. + +**Validates**: Requirement 5.1. + +**Verification**: `grep -rn '\bdocker build\b' docker/ Makefile | grep -v +buildx` returns no results. + +### Property 12: Single canonical compose file + +**Statement**: `ls /Users/ben/Projects/wavekit/docker-compose*.yml +/Users/ben/Projects/wavekit/compose*.yaml 2>/dev/null` SHALL list exactly +one file: `compose.yaml`. + +**Validates**: Requirement 2.6. + +**Verification**: Command above. One file listed (the +`packages/sdr-host/docker-compose.yml` is correctly under `packages/` and +doesn't match the glob at repo root). + +## 11. Risk & Rollback + +### 11.1 Risks + +| Risk | Likelihood | Mitigation | +|---------------------------------------------------------------------------------------------------------------------------------------------------|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Pinned decoder SHAs lock us to old/buggy versions | Medium | Initial pin captures the SHA currently in production. Renovate/Dependabot configured for the bake.hcl/ARG block to surface bumps as PRs. | +| `isolated` linker breaks an obscure decoder integration that worked under `hoisted` | Low | Project `.npmrc` has been `node-linker=isolated` since 2024. Local builds already use it. The Dockerfile's `hoisted` override was the divergence, not the default. | +| Removing the s6 hard dep `wavekit-api → sdrpp-server` causes wavekit-api to start before sdrpp is listening, producing log noise on first connect | Low | Application already handles this via `SourceManager` exponential backoff (CLAUDE.md §"Stream pipeline"). Log noise is acceptable; behaviour was already this way whenever sdrpp restarted mid-run. | +| GHCR cache pulls dominate cold-build wall time more than expected | Low | `mode=max` cache write on main keeps even intermediate layers. If pulls are slow, contributors can opt out via `make docker-build CACHE_FROM_ONLY=true` (becomes a no-op cache-from) — the bake.hcl variable is already there. | +| `final-demod` interactive shell drifts from the standalone test environment | Low | `final-demod` builds from the same `*-build` stages used by `final` and `final-core` — by construction, the binary versions match production. Drift is impossible without explicit edit. | +| Renovate-style PRs to bump every decoder SHA all at once become noisy | Low | Group all decoder refs into a single Renovate `packageRule` (out-of-scope but trivial follow-up). | + +### 11.2 Rollback + +Each change is git-revertable. The migration is structured so that the +Dockerfile refactor, the compose collapse, the Makefile cleanup, and the CI +workflow rewrite are separable commits. If `dev-stack` regresses, revert the +compose commit and the old four-file setup returns. + +The s6 service contract change (removing the `sdrpp-server` hard dep) is +the only semantic change with non-trivial rollback: reverting requires +either restoring the dependency file in the canonical tree (breaks +final-core build cleanliness) OR keeping the partition and restoring the +file only in `s6-overlay-sdrpp/`'s sister overlay (preserves both modes; +this is the preferred rollback path if it ever becomes necessary). diff --git a/.kiro/specs/build-devx-overhaul/requirements.md b/.kiro/specs/build-devx-overhaul/requirements.md new file mode 100644 index 0000000..6744643 --- /dev/null +++ b/.kiro/specs/build-devx-overhaul/requirements.md @@ -0,0 +1,348 @@ +# Build & DX Overhaul — Requirements + +## Overview + +WaveKit ships a 737-line Dockerfile, four parallel docker-compose files, two +divergent dev flows in the Makefile, and a CI workflow that never exercises +Docker. The result: cold builds take long enough that contributors avoid them, +warm builds re-execute work that should be cached, the dev compose silently +mounts host `dist/` over the container's built artifacts, and a touch to any +decoder's apt deps invalidates the entire 11-decoder build cascade. + +This spec replaces that build system with one Dockerfile of named, narrow +stages; one compose file with profiles; a GHCR-backed BuildKit registry cache; +a native `pnpm dev` headline loop; and CI that builds at least the `core` image +target on every PR. + +Stack, supervision, package manager, and decoder set are unchanged. See +**Out-of-Scope** below for what is explicitly NOT being touched. + +## User Stories & Acceptance Criteria + +### 1. Native TypeScript iteration loop + +**As a** WaveKit contributor working on non-decoder TypeScript code, +**I want** a single command that runs the app with hot reload on my host, +**so that** I don't pay the Docker round-trip on every edit. + +- **1.1** WHEN a developer runs `pnpm dev` from the repo root THE system SHALL + run esbuild in watch mode against `src/index.ts` and `node --watch` against + the resulting `dist/index.js` in parallel, without invoking Docker. +- **1.2** WHEN a developer runs `pnpm dev` THE system SHALL NOT require any + running Docker daemon, Docker socket, or Docker CLI. +- **1.3** WHEN a TypeScript file under `src/` changes THE running app SHALL + restart within the time of one esbuild rebuild plus one Node restart (target: + under 2 seconds on a current laptop) without any manual intervention. +- **1.4** WHEN a developer runs `pnpm dev` THE app SHALL connect to whatever + SDR source is configured by `config/.yaml` or `WAVEKIT_SOURCES_*` env + vars; no SDR source is started by `pnpm dev` itself. +- **1.5** WHEN documenting the dev workflow `CLAUDE.md` and + `docs/DOCKER-SETUP.md` SHALL present `pnpm dev` as the default iteration + loop and `make dev-stack` as the integration-testing fallback. + +### 2. Full-stack integration testing via compose profiles + +**As a** WaveKit contributor testing decoder pipelines end-to-end, +**I want** one command that brings up the full container stack against a +mock or external SDR source, +**so that** I don't need to remember which of four compose files to use. + +- **2.1** WHEN a developer runs `docker compose --profile dev up` THE system + SHALL start the `sdrpp-server` service (target `final-sdrpp`) and the + `wavekit-api` service (target `final-core`) with `wavekit-api` connecting + to `sdrpp-server:5259`. +- **2.2** WHEN a developer runs `docker compose --profile prod-single-host + up -d` THE system SHALL start the `wavekit-full` service (image + `wavekit:latest`, target `final`) with no build step. +- **2.3** WHEN a developer runs `docker compose --profile prod-distributed + up -d` THE system SHALL start `wavekit-sdrpp` and `wavekit-core` as separate + services on the same network, with `wavekit-core` configured to connect to + `wavekit-sdrpp:5259`. +- **2.4** WHEN a developer runs `docker compose --profile demod-test run + --rm demod-test` THE system SHALL start an interactive container built from + target `final-demod` containing dsd-fme, multimon-ng, csdr, and rtl-sdr + binaries with `/workspace`, `/data/debug_audio`, and `/output` mounted. +- **2.5** WHEN a service in any profile has a `build:` block THE block SHALL + declare `cache_from` and `cache_to` pointing at the GHCR registry cache + per Section 4. +- **2.6** The repository SHALL contain exactly one root-level compose file + named `compose.yaml`. `docker-compose.dev.yml`, + `docker-compose.prod.yml`, `docker-compose.override.yml`, and + `docker-compose.demod-test.yml` SHALL be deleted. +- **2.7** The `compose.yaml` SHALL NOT define an `nginx-reverse-proxy` service. + The `compose.yaml` SHALL NOT define a `vscode-server` (codercom/code-server) + service. +- **2.8** The `packages/sdr-host/docker-compose.yml` SHALL remain in place + unchanged; it deploys to a different host (the Pi running the SDR dongle) + and is not part of the root compose. + +### 3. Dockerfile structure — narrow stages, pinned refs, fixed anti-patterns + +**As a** WaveKit contributor changing a single decoder's behaviour, +**I want** my change to invalidate only that decoder's cache layers, +**so that** I don't wait for 10 unrelated decoders to rebuild. + +- **3.1** The Dockerfile SHALL define a thin `base-build` stage containing + exactly: `build-essential`, `cmake`, `git`, `pkg-config`, `ca-certificates`, + `curl`. No decoder-specific or signal-processing-specific apt packages. +- **3.2** Each of the 11 decoder/binary build stages (`sdrpp-build`, + `dsd-fme-build`, `multimon-ng-build`, `rtl433-build`, `acarsdec-build`, + `ais-catcher-build`, `direwolf-build`, `dumpvdl2-build`, `readsb-build`, + `soapy-rtltcp-build`, `csdr-build`, `lora-build`) SHALL install ONLY its own + minimal apt dependencies in its own `RUN apt-get install` invocation, + protected by `--mount=type=cache,target=/var/cache/apt,sharing=locked`. +- **3.3** Every decoder build stage SHALL pin its upstream source to a SHA or + tag via a top-of-file `ARG _REF=` declaration, and SHALL + check out that ref via `git fetch --depth 1 origin "${REF}" && git checkout + --detach FETCH_HEAD`. The existing `GR_LORA_SDR_REF` pattern is the model. +- **3.4** Every stage that performs platform-specific actions SHALL declare + `ARG TARGETARCH` and gate platform-specific steps on its value (mapping + `amd64`→`x86_64`, `arm64`→`aarch64`). The `packages/sdr-host/Dockerfile` + pattern (lines 87–96) is the model. +- **3.5** The `runtime-base` stage SHALL select the s6-overlay archive via + `TARGETARCH`, NOT `BUILDPLATFORM`. Multi-arch builds of the arm64 variant + SHALL produce an image with the aarch64 s6 binaries. +- **3.6** The `node-build` stage SHALL invoke `pnpm install + --frozen-lockfile --prod=false` exactly once. The stage SHALL NOT call + `pnpm config set node-linker hoisted`. The project's `.npmrc` + `node-linker=isolated` setting SHALL be the single source of truth for + package layout. +- **3.7** The Dockerfile SHALL define a `final-base` stage that contains the + runtime base + all decoders + node + the application + every artifact and + s6 service definition shared by `final` and `final-core`. +- **3.8** The Dockerfile SHALL define `final` as `FROM final-base` plus SDR++ + binaries from `sdrpp-build` plus the `sdrpp-server` s6 service overlay from + `docker/overlay/s6-overlay-sdrpp/`. +- **3.9** The Dockerfile SHALL define `final-core` as `FROM final-base` with + NO additional contents. +- **3.10** No stage in the Dockerfile SHALL contain a `rm -rf` or `rm -f` + targeting any path under `/etc/s6-overlay/`. The mode partition SHALL be + achieved by additive COPY only. +- **3.11** WHEN building target `final-core` the resulting image SHALL contain + `/etc/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/` referencing ONLY + `wavekit-init`, AND SHALL NOT contain any path matching + `/etc/s6-overlay/s6-rc.d/sdrpp-server*`, AND SHALL NOT contain + `/etc/s6-overlay/s6-rc.d/services/contents.d/sdrpp-server`, AND SHALL NOT + contain `/etc/s6-overlay/s6-rc.d/user/contents.d/sdrpp-server`. +- **3.12** The Dockerfile SHALL define a `final-demod` stage that COPIES + dsd-fme, multimon-ng, csdr, and rtl-sdr binaries plus their `libmbe*`, + `libcsdr*`, and `librtlsdr*` runtime libraries from the existing `*-build` + stages, declares `WORKDIR /workspace`, and uses `CMD ["/bin/bash"]`. +- **3.13** The hardcoded path `/usr/lib/x86_64-linux-gnu/libncurses*` in + `final-core` SHALL be removed or replaced with a `TARGETARCH`-conditional + path. If verification during implementation shows the copy is unnecessary + (ncurses is satisfied by the `libncurses6` runtime apt install), the COPY + SHALL be deleted entirely. +- **3.14** The `final` and `final-core` stages SHALL share their python + runtime install (`python3`, `gnuradio`, `python3-numpy`, + `python3-protobuf`, `python3-cryptography`) via `final-base` so it appears + in one place, not two. +- **3.15** The `lora_meshtastic_decode.py` script and `meshtastic_proto/` + package SHALL be copied once in `final-base`, not separately in `final` and + `final-core`. + +### 4. GHCR-backed BuildKit registry cache + +**As a** WaveKit contributor who has just cloned the repo, +**I want** my first local Docker build to pull cache from GHCR, +**so that** my cold build behaves like a warm one without waiting for upstream +decoders to compile. + +- **4.1** Every stage that builds upstream decoders or compiles the node app + SHALL be cached at `ghcr.io/coriou/wavekit:cache-` (one ref per + stage), populated by `--cache-to=type=registry,ref=,mode=max` on main + branch CI pushes. +- **4.2** WHEN a contributor runs `make docker-build` after + `docker login ghcr.io` THE build SHALL pass `--cache-from=type=registry, + ref=` for each cached stage; cache misses SHALL fall back to local + layer cache and then to a clean build without erroring. +- **4.3** WHEN a contributor runs `make docker-build` WITHOUT having logged in + to GHCR THE build SHALL still succeed; `--cache-from` against an + unauthenticated registry returns "no cache" rather than an auth error. +- **4.4** The cache ref naming SHALL match `cache-` with one + hyphenated lowercase identifier per stage, listed canonically in `design.md`. +- **4.5** The local-directory cache (`./.docker-cache` referenced by + `cache_from`/`cache_to` in the old dev/prod compose files and created by + `docker/init.sh`) SHALL be removed from `docker/init.sh`, removed from + `compose.yaml`, and the `.docker-cache` line in `.dockerignore` SHALL be + retained only as a defensive ignore for any locally-leftover directory. + +### 5. Buildx-first build pipeline + +**As a** WaveKit maintainer pushing a release, +**I want** every image build to go through buildx with multi-arch and +provenance support, +**so that** my multi-arch tags and registry cache work uniformly across local, +CI, and release contexts. + +- **5.1** No script under `docker/` SHALL invoke `docker build`. Every image + build SHALL invoke `docker buildx build` or `docker buildx bake`. +- **5.2** The Makefile target `docker-build` SHALL invoke `docker buildx bake` + against `docker/bake.hcl`. +- **5.3** `docker/bake.hcl` SHALL define one target per final image + (`final`, `final-core`, `final-sdrpp`, `final-demod`), one group `default` + bundling `final`/`final-core`/`final-sdrpp`, and shared cache configuration + via HCL variables. +- **5.4** Multi-arch SHALL target `linux/amd64` and `linux/arm64` exclusively. + `linux/arm/v7` SHALL NOT appear in any build invocation, bake target, + `docker/push.sh`, `docker/platform-utils.sh`, or compose file. +- **5.5** The `docker/push.sh` script's docker.io path SHALL be removed. + GHCR (`ghcr.io//wavekit`) SHALL be the only registry the script + pushes to. The owner SHALL be derivable from `WAVEKIT_GH_OWNER` env var + with a `coriou` fallback, matching the `packages/sdr-host/scripts/ + build-publish.sh` pattern. +- **5.6** The `docker/platform-utils.sh` `build_multiarch` function SHALL be + deleted (replaced by bake) OR consolidated into `docker/build.sh`/`push.sh` + such that no defined-but-unused helper remains. + +### 6. Minimal Makefile + +**As a** WaveKit contributor scanning the Makefile, +**I want** under 20 targets organised by lifecycle (dev, build, push, ops), +**so that** I can find the command I need in seconds. + +- **6.1** The Makefile SHALL define these targets and ONLY these targets in + addition to `help`, `fixtures-*`, and the existing `sdr-host-*` group: + `dev`, `dev-stack`, `dev-stack-down`, `dev-stack-logs`, `dev-dashboard`, + `dev-dashboard-build`, `dev-configs`, `dev-shell`, `dev-status`, + `docker-build`, `docker-push`, `docker-init`, `docker-clean`, + `docker-prune`, `demod-test`. +- **6.2** The Makefile SHALL NOT define `docker-dev`, `docker-prod`, + `docker-compose-up`, `docker-compose-down`, `docker-compose-logs`, + `docker-build-full`, `docker-build-core`, `docker-build-sdrpp`, + `docker-run-core`, `docker-test`, `docker-test-coverage`, `docker-lint`, + `demo`, `install-buildx`, `dev-up`, `dev-build`, `dev-start`, `dev-stop`, + `dev-restart`, `dev-logs`, `dev-logs-raw`, `dev-audio`, `dev-debug-audio`. +- **6.3** Newly-introduced Makefile output SHALL NOT contain emoji and SHALL + NOT use ANSI colour escapes for decorative purposes (status prefixes like + `[wavekit]` are fine). Existing emoji and colour in retained targets MAY + remain to avoid drive-by churn. + +### 7. CI exercises Docker builds + +**As a** WaveKit maintainer reviewing a PR, +**I want** CI to fail if the Dockerfile is broken, +**so that** Docker regressions are caught before merge. + +- **7.1** `.github/workflows/ci.yml` SHALL define a `lint-typecheck-test` job + running `pnpm lint`, `pnpm typecheck`, `pnpm build`, and `pnpm test` on + ubuntu-latest with Node 22 (via `actions/setup-node@v4`) and pnpm + installed via Corepack from the `packageManager` field in `package.json`. +- **7.2** `.github/workflows/ci.yml` SHALL define a `docker-build` job that + depends on `lint-typecheck-test`, sets up QEMU and buildx + (docker-container driver), and runs `docker buildx bake` against target + `final-core` for platforms `linux/amd64,linux/arm64`. +- **7.3** WHEN the workflow runs on `pull_request` THE `docker-build` job + SHALL use `--cache-from=type=registry,ref=ghcr.io/coriou/wavekit:cache-*` + WITHOUT logging into GHCR, WITHOUT pushing the image, and WITHOUT + updating any cache ref. +- **7.4** WHEN the workflow runs on `push` to `main` THE `docker-build` job + SHALL log in to GHCR using `${{ secrets.GITHUB_TOKEN }}`, build the + `default` bake group (`final`, `final-core`, `final-sdrpp`), push each + image to `ghcr.io/coriou/wavekit:latest{,-core,-sdrpp}`, and update each + stage's `cache-` ref with `mode=max`. +- **7.5** The Node version used in CI SHALL be `22` (matching the + `node-build` stage in the Dockerfile). The legacy `node-version: "20"` + SHALL be removed. + +### 8. Documentation consolidation + +**As a** WaveKit contributor reading docs, +**I want** one canonical Docker reference, +**so that** I don't have to reconcile two sources. + +- **8.1** `docs/DOCKER-SETUP.md` SHALL be the canonical Docker reference. + Its content SHALL be updated to reflect the new compose profiles, the new + Makefile targets, the native `pnpm dev` flow, and the GHCR cache. +- **8.2** `docker/README.md` SHALL be either deleted OR reduced to a one-line + pointer at `docs/DOCKER-SETUP.md` (no separately-maintained content). +- **8.3** `CLAUDE.md`'s "Day-to-day dev (Docker)" section SHALL be condensed + to one paragraph: `pnpm dev` for typescript-only iteration; `make + dev-stack` for full-stack integration; link to `docs/DOCKER-SETUP.md` for + details. + +### 9. sdr-host package alignment + +**As a** WaveKit maintainer keeping the two Dockerfiles in sync, +**I want** the sdr-host package to use the same pnpm-via-Corepack pattern as +the main image, +**so that** a single decision propagates everywhere. + +- **9.1** `packages/sdr-host/Dockerfile` line 47 (`RUN npm install -g + pnpm@10`) SHALL be replaced with the Corepack pattern used in the main + Dockerfile: `RUN corepack enable && corepack prepare pnpm@10.28.0 + --activate`. +- **9.2** The pnpm version pinned in `packages/sdr-host/Dockerfile` SHALL + match the `packageManager` field in `package.json` exactly. + +### 10. Image-size trim (modest, opportunistic) + +**As a** WaveKit user pulling images on a Raspberry Pi, +**I want** the runtime image trimmed of unused packages and duplicated +layers, +**so that** the image is 10–20% smaller without architectural changes. + +- **10.1** apt installs in every stage SHALL conclude with `rm -rf + /var/lib/apt/lists/*` (already the case; this is restated as a contract). +- **10.2** The duplicate `libsox-fmt-all` line in `runtime-base` (current + Dockerfile lines 137–138) SHALL be deduplicated. +- **10.3** The `python3-cryptography` apt package in the python install + SHALL be verified against the actual import surface of + `lora_meshtastic_decode.py`. If unused, it SHALL be removed. +- **10.4** Any `tini` apt install SHALL be removed if `tini` is unused at + runtime (s6-overlay's `/init` is PID 1). If `tini` is referenced anywhere + under `docker/scripts/` or `docker/overlay/`, it stays. +- **10.5** Out-of-scope for size reduction: switching to alpine, switching + to distroless, splitting decoders into sidecars, removing any decoder. + +### 11. `.dockerignore` hygiene + +**As a** WaveKit contributor running `docker buildx bake`, +**I want** the build context to exclude everything that doesn't ship in the +image, +**so that** context transfer is fast and cache invalidation is precise. + +- **11.1** `.dockerignore` SHALL exclude `.turbo/`, `.pnpm-store/`, and + `.docker-cache/` (already present; restated as contract). +- **11.2** `.dockerignore` SHALL exclude `dist/` (already present; restated + as contract). The Dockerfile builds its own dist. +- **11.3** `.dockerignore` SHALL exclude `.kiro/`, `wip/`, `tests/`, and + fixture artifacts (already present; restated). +- **11.4** `.dockerignore` SHALL be verified to NOT exclude `docker/`, + `config/`, `tsconfig.json`, `pnpm-lock.yaml`, or any other path the + Dockerfile needs at build time. + +## Out-of-Scope + +The following are explicitly NOT in this spec. Any of these surfacing in a +PR raised against this spec is grounds for rejection. + +- Switching the runtime or build base from `debian:bookworm-slim` to alpine, + distroless, ubi-micro, or any other base. +- Splitting any decoder into its own sidecar container or process. +- Rewriting any decoder integration in `src/decoders/`. +- Removing any of the 9 user-facing decoders (dsd-fme, multimon-ng, rtl_433, + acarsdec, AIS-catcher, direwolf, dumpvdl2, readsb, lora-meshtastic) or + the 2 supporting tools (soapy-rtltcp, csdr). +- Replacing s6-overlay with tini-only, runit, supervisord, or any other init + system. +- Restructuring the root-as-`wavekit`-package monorepo layout. +- Replacing pnpm, Turborepo, Fastify, Pino, Zod, or YAML as listed in + `CLAUDE.md`. +- Adding Redis, a message queue, a database, or any new persistent service. +- Building for `linux/arm/v7` or any platform beyond `linux/amd64` and + `linux/arm64`. +- Wrapping `docker compose` in a `wavekit` CLI tool. +- Rewriting the cli (`cli/` package) or its build pipeline. +- Touching `tsconfig.json` composite/declaration settings beyond noting the + existing drift between root and base configs. tsconfig alignment is + out-of-scope unless it blocks a specific Dockerfile change. + +## Glossary + +- **Stage**: a `FROM ... AS ` block in a Dockerfile. +- **Profile**: a Compose v2 `profiles:` selector on a service. +- **Target**: the `--target` flag value passed to `docker buildx build`. +- **Cache ref**: a registry image reference used solely to store BuildKit + cache, e.g. `ghcr.io/coriou/wavekit:cache-readsb-build`. +- **Bake group**: a named bundle of bake targets built together. diff --git a/.kiro/specs/build-devx-overhaul/tasks.md b/.kiro/specs/build-devx-overhaul/tasks.md new file mode 100644 index 0000000..9f0f9e0 --- /dev/null +++ b/.kiro/specs/build-devx-overhaul/tasks.md @@ -0,0 +1,226 @@ +# Build & DX Overhaul — Implementation Tasks + +Dependency-sequenced checklist. Do tasks in order; each task lists the +requirements it satisfies. Check off (`- [x]`) as you complete each task. +Two checkpoints split the work into adoptable chunks. + +## Phase A: s6 Service Tree Partition + +Touches no Dockerfile yet. Pure file-tree reorganization. Safe to land alone. + +- [ ] **A.1** Remove the file `docker/overlay/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/sdrpp-server`. The canonical tree's `wavekit-api` SHALL now only depend on `wavekit-init`. _Requirements: 3.10, 3.11_ +- [ ] **A.2** Create directory `docker/overlay/s6-overlay-sdrpp/s6-rc.d/`. _Requirements: 3.8_ +- [ ] **A.3** Move `docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/` (with all its files: `type`, `run`, `finish`, `dependencies.d/wavekit-init`) to `docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/`. _Requirements: 3.8_ +- [ ] **A.4** Move `docker/overlay/s6-overlay/s6-rc.d/services/contents.d/sdrpp-server` to `docker/overlay/s6-overlay-sdrpp/s6-rc.d/services/contents.d/sdrpp-server`. _Requirements: 3.8_ +- [ ] **A.5** Move `docker/overlay/s6-overlay/s6-rc.d/user/contents.d/sdrpp-server` to `docker/overlay/s6-overlay-sdrpp/s6-rc.d/user/contents.d/sdrpp-server`. _Requirements: 3.8_ +- [ ] **A.6** Verify `find docker/overlay/s6-overlay -iname '*sdrpp*'` returns empty. Verify `find docker/overlay/s6-overlay-sdrpp -type f` lists exactly: `sdrpp-server/type`, `sdrpp-server/run`, `sdrpp-server/finish`, `sdrpp-server/dependencies.d/wavekit-init`, `services/contents.d/sdrpp-server`, `user/contents.d/sdrpp-server`. _Requirements: 3.10, 3.11_ + +## Phase B: Dockerfile Refactor + +Rewrites `Dockerfile` top to bottom. After this phase, image builds work with +the new structure; cache is still local-only (registry cache lands in Phase +D). + +### B.1 Capture upstream SHAs + +- [ ] **B.1.1** For each of the 12 upstream sources (SDR++, mbelib, dsd-fme, multimon-ng, rtl_433, acarsdec, AIS-catcher, direwolf, libacars, dumpvdl2, readsb, SoapyRTLTCP, csdr), `git ls-remote` the current branch tip and record the SHA. Save these as the initial values for the `ARG _REF` declarations. _Requirements: 3.3_ +- [ ] **B.1.2** Keep `GR_LORA_SDR_REF=862746dd1cf635c9c8a4bfbaa2c3a0ec3a5306c9` as-is. _Requirements: 3.3_ + +### B.2 Rewrite Dockerfile + +- [ ] **B.2.1** Add a single top-of-file ARG block declaring `S6_OVERLAY_VERSION` and all 13 decoder/binary upstream refs from §B.1. _Requirements: 3.3_ +- [ ] **B.2.2** Replace the current `base-deps` stage with a thin `base-build` stage containing only `build-essential`, `cmake`, `git`, `pkg-config`, `ca-certificates`, `curl`. _Requirements: 3.1_ +- [ ] **B.2.3** Refactor `runtime-base`: + - Switch the s6-overlay platform-detection block from `BUILDPLATFORM` to `TARGETARCH` (case mapping per sdr-host pattern). + - Drop the `--mount=type=cache,target=/var/lib/apt` mount; keep only the `/var/cache/apt` mount. + - Dedupe the duplicate `libsox-fmt-all` line. + - Verify whether `tini` is referenced anywhere under `docker/`; if not, remove the apt package. + - _Requirements: 3.5, 10.1, 10.2, 10.4, 14_ +- [ ] **B.2.4** Rewrite each of the 12 `*-build` stages: + - `FROM base-build`. + - One `RUN apt-get install` with only this decoder's deps, protected by `--mount=type=cache,target=/var/cache/apt,sharing=locked`, ending with `rm -rf /var/lib/apt/lists/*`. + - `ARG TARGETARCH` if the stage does arch-conditional work. + - Pinned-ref checkout via `git clone --no-checkout && cd && git fetch --depth 1 origin "${REF}" && git checkout --detach FETCH_HEAD`. + - `make install` or equivalent. + - _Requirements: 3.2, 3.3, 3.4_ +- [ ] **B.2.5** Refactor `node-build`: + - Remove the `pnpm config set node-linker hoisted` + second install block (lines 432–438 of the current Dockerfile). + - Single `pnpm install --frozen-lockfile --prod=false` invocation with `--mount=type=cache,target=/root/.local/share/pnpm/store,sharing=locked`. + - Keep the existing Turbo cache mount on typecheck + build. + - Keep the existing `pnpm prune --prod` final step. + - _Requirements: 3.6_ +- [ ] **B.2.6** Create a new `final-base` stage: + - `FROM runtime-base`. + - Apt-install python3 + gnuradio + python3-numpy + python3-protobuf (drop python3-cryptography if verification shows it's unused by `lora_meshtastic_decode.py`). + - COPY every decoder binary + library from `*-build` stages (current `final` and `final-core` share these; consolidate). + - COPY csdr + soapy-rtltcp. + - COPY lora artifacts. + - COPY node runtime + app dist + workspace packages + config + package.json from `node-build`. + - COPY `docker/scripts/init-system.sh start-api.sh finish-api.sh` (the always-needed scripts; sdrpp scripts deferred to `final`). + - COPY `docker/scripts/lora_meshtastic_decode.py` and `docker/scripts/meshtastic_proto/`. + - COPY `docker/config/direwolf.conf` to `/etc/direwolf.conf`. + - COPY `docker/overlay/s6-overlay/s6-rc.d` → `/etc/s6-overlay/s6-rc.d` (canonical tree, no sdrpp). + - COPY `docker/scripts/healthcheck.sh` → `/etc/s6-overlay/scripts/healthcheck.sh`. + - Run `ldconfig` and decoder-verification block. + - REMOVE the hardcoded `COPY --from=dsd-fme-build /usr/lib/x86_64-linux-gnu/libncurses* ...` line. Test build on arm64; if ncurses runtime is missing, replace with a `TARGETARCH`-conditional COPY mapping `amd64→x86_64-linux-gnu` and `arm64→aarch64-linux-gnu`. + - HEALTHCHECK on `wavekit-api`. + - `ENTRYPOINT ["/init"]`. + - LABELs: maintainer, version. + - _Requirements: 3.7, 3.13, 3.14, 3.15, 10.1, 10.3_ +- [ ] **B.2.7** Define `final` stage: + - `FROM final-base`. + - COPY SDR++ binaries + libraries from `sdrpp-build`. + - COPY `docker/scripts/start-sdrpp.sh finish-sdrpp.sh` → `/usr/local/bin/`. + - COPY `docker/overlay/s6-overlay-sdrpp/s6-rc.d` → `/etc/s6-overlay/s6-rc.d` (additive; unions sdrpp-server into the canonical tree). + - `RUN chmod -R 755 /etc/s6-overlay/s6-rc.d`. + - EXPOSE add 5259, 7355. + - LABEL mode=full. + - _Requirements: 3.8, 3.10_ +- [ ] **B.2.8** Define `final-core` stage: + - `FROM final-base`. + - LABEL mode=core. + - No further content. (Existence of the stage is the contract.) + - _Requirements: 3.9, 3.10, 3.11_ +- [ ] **B.2.9** Rewrite `final-sdrpp` stage: + - `FROM runtime-base`. + - Apply the `TARGETARCH` s6 platform-detection fix (inherited from `runtime-base` refactor in B.2.3). + - Existing structure otherwise preserved. + - _Requirements: 3.5_ +- [ ] **B.2.10** Add `final-demod` stage: + - `FROM runtime-base`. + - Apt-install (with cache mount): sox, libsox-fmt-all (already in runtime-base — verify), vim, netcat-openbsd, python3, python3-pip, python3-matplotlib, python3-numpy, python3-scipy, ffmpeg, gnuradio, gr-osmosdr. + - COPY dsd-fme + libmbe* from `dsd-fme-build`. + - COPY multimon-ng from `multimon-ng-build`. + - COPY csdr + libcsdr* from `csdr-build`. + - rtl_* tools: apt-install `rtl-sdr` in this stage (Debian package provides `rtl_test`, `rtl_fm`, `rtl_sdr`, `rtl_tcp`, etc.). `librtlsdr0` is already in `runtime-base`. No dedicated `rtl-sdr-build` stage is added — `final-demod` is for interactive testing, not production, so Debian's packaged version is sufficient. + - `WORKDIR /workspace`. + - `RUN mkdir -p /data/debug_audio /scripts /output`. + - `CMD ["/bin/bash"]`. + - No s6, no ENTRYPOINT — this is an interactive utility container, not a supervised service. + - _Requirements: 2.4, 3.12_ +- [ ] **B.2.11** Smoke-test locally: + - `docker buildx build --target final-core --load -t wavekit:dev-core .` + - `docker buildx build --target final --load -t wavekit:dev .` + - `docker buildx build --target final-sdrpp --load -t wavekit:dev-sdrpp .` + - `docker buildx build --target final-demod --load -t wavekit:dev-demod .` + - All four builds succeed. + - _Requirements: covers B.2.1–B.2.10_ +- [ ] **B.2.12** Verify Property 8 and Property 9 against the new `final-core` image: + - `docker run --rm wavekit:dev-core sh -c 'find /etc/s6-overlay -iname "*sdrpp*"'` returns empty. + - `docker run --rm wavekit:dev-core ls /etc/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/` returns exactly `wavekit-init`. + - `docker run --rm wavekit:dev sh -c 'find /etc/s6-overlay -name "sdrpp-server" | wc -l'` returns >= 1. + - _Requirements: 3.10, 3.11_ + +> **Checkpoint 1**: Dockerfile refactor complete. Old compose files still +> exist; they still work because the stage names (`final`, `final-core`, +> `final-sdrpp`) are unchanged. Safe to land Phases A and B as one +> reviewable PR before continuing. + +## Phase C: Compose Collapse + +- [ ] **C.1** Create `/Users/ben/Projects/wavekit/compose.yaml` per the design §3 with four profiles (`dev`, `prod-single-host`, `prod-distributed`, `demod-test`). _Requirements: 2.1–2.5_ +- [ ] **C.2** Each service with a `build:` block declares `cache_from` pointing at the relevant `ghcr.io/coriou/wavekit:cache-*` registry refs (for now this is harmless; in Phase D those refs get populated). _Requirements: 2.5_ +- [ ] **C.3** Verify `docker compose --profile dev config` parses without warnings. _Requirements: 2.1, 2.6_ +- [ ] **C.4** Verify `docker compose --profile prod-single-host config` parses without warnings. _Requirements: 2.2_ +- [ ] **C.5** Verify `docker compose --profile prod-distributed config` parses without warnings. _Requirements: 2.3_ +- [ ] **C.6** Verify `docker compose --profile demod-test config` parses without warnings. _Requirements: 2.4_ +- [ ] **C.7** Delete `/Users/ben/Projects/wavekit/docker-compose.dev.yml`. _Requirements: 2.6, 2.7_ +- [ ] **C.8** Delete `/Users/ben/Projects/wavekit/docker-compose.prod.yml`. _Requirements: 2.6_ +- [ ] **C.9** Delete `/Users/ben/Projects/wavekit/docker-compose.override.yml`. _Requirements: 2.6_ +- [ ] **C.10** Delete `/Users/ben/Projects/wavekit/docker-compose.demod-test.yml`. _Requirements: 2.6_ +- [ ] **C.11** Delete `/Users/ben/Projects/wavekit/docker/Dockerfile.demod-test`. _Requirements: 3.12_ +- [ ] **C.12** Run end-to-end: `docker compose --profile dev up --build`. wavekit-api SHALL reach `service_healthy`. `curl localhost:9000/health` SHALL return 200. _Requirements: 2.1_ + +## Phase D: Bake + Registry Cache + +- [ ] **D.1** Write `/Users/ben/Projects/wavekit/docker/bake.hcl` per design §4.3. Define `_base`, all four final-target bakes, the `ci-core` helper target, the `default` group, and the `demod` group. _Requirements: 5.2, 5.3, 4.4_ +- [ ] **D.2** Test bake locally: `docker buildx bake --file docker/bake.hcl default --set "*.platform=linux/amd64"`. All three targets in `default` SHALL build successfully. _Requirements: 5.2_ +- [ ] **D.3** Test multi-arch bake: `docker buildx bake --file docker/bake.hcl ci-core --set "*.platform=linux/amd64,linux/arm64"`. Build SHALL succeed without `--push` (uses local layer cache for both arches). _Requirements: 5.4_ +- [ ] **D.4** Rewrite `/Users/ben/Projects/wavekit/docker/build.sh` as a thin wrapper that delegates to `docker buildx bake` (OR delete the file and have Makefile call bake directly — pick whichever keeps Makefile cleanest). _Requirements: 5.1, 5.2_ +- [ ] **D.5** Rewrite `/Users/ben/Projects/wavekit/docker/push.sh` to push only to GHCR (drop the docker.io path on lines 16–18). Use `WAVEKIT_GH_OWNER` env var with `coriou` fallback per sdr-host pattern. Drop `linux/arm/v7` from the default `PLATFORMS`. _Requirements: 5.4, 5.5_ +- [ ] **D.6** Edit `/Users/ben/Projects/wavekit/docker/init.sh`: + - Remove the `.docker-cache` mkdir + chmod (lines 47–51). + - Keep the buildx-builder setup. + - Keep the network/volume creation. + - _Requirements: 4.5_ +- [ ] **D.7** Delete `build_multiarch` from `/Users/ben/Projects/wavekit/docker/platform-utils.sh` (replaced by bake). If `detect_platform` is still used anywhere, keep it; if not, delete the whole file. _Requirements: 5.6_ +- [ ] **D.8** Initial cache seed: from a clean local machine, log in to GHCR, run `docker buildx bake --file docker/bake.hcl default --push --set "*.cache-to=type=registry,ref=ghcr.io/coriou/wavekit:cache-,mode=max"`. (One-time bootstrap; subsequent updates come from CI.) _Requirements: 4.1_ + +## Phase E: Makefile Cleanup + +- [ ] **E.1** Rewrite `/Users/ben/Projects/wavekit/Makefile` with the new target inventory per design §7.1. _Requirements: 6.1_ +- [ ] **E.2** Confirm no deleted targets remain in the file. Run `make help` and verify only the targets listed in 6.1 (plus sdr-host-* and fixtures-*) appear. _Requirements: 6.2_ +- [ ] **E.3** Verify all newly-introduced echo output is emoji-free and uses no decorative ANSI colour. Existing colour in retained targets MAY remain unchanged. _Requirements: 6.3_ +- [ ] **E.4** Test the new dev flow end-to-end: + - `make dev` SHALL start `pnpm dev` with esbuild watch + node watch. + - `make dev-stack` SHALL build via buildx bake and start the dev profile. + - `make dev-dashboard` SHALL connect the CLI dashboard. + - `make docker-build` SHALL invoke bake. + - `make demod-test` SHALL launch the demod-test profile interactively. + - _Requirements: 1.1, 2.1, 2.4, 5.2, 6.1_ + +## Phase F: CI Workflow + +- [ ] **F.1** Rewrite `/Users/ben/Projects/wavekit/.github/workflows/ci.yml` per design §8. _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5_ +- [ ] **F.2** Update Node to `22` and pnpm install to use Corepack (matches the Dockerfile pattern). _Requirements: 7.5_ +- [ ] **F.3** Add the `lint-typecheck-test` job that runs lint, typecheck, build, AND test (test was missing from the old workflow). _Requirements: 7.1_ +- [ ] **F.4** Add the `docker-build` job that depends on `lint-typecheck-test`, sets up QEMU + buildx, and runs bake. _Requirements: 7.2_ +- [ ] **F.5** PR path: `--cache-from` only, no login, no push. _Requirements: 7.3_ +- [ ] **F.6** Main-branch path: log in to GHCR, build `default` group with `--push` and `--cache-to mode=max`. _Requirements: 7.4_ +- [ ] **F.7** Smoke-test by opening a draft PR with a no-op change and verifying both jobs run and succeed. _Requirements: 7.1, 7.2, 7.3_ + +> **Checkpoint 2**: At this point the new build system is fully functional. +> Phases G and H are documentation + sdr-host alignment polish. + +## Phase G: sdr-host Alignment + +- [ ] **G.1** Edit `/Users/ben/Projects/wavekit/packages/sdr-host/Dockerfile` line 47 from `RUN npm install -g pnpm@10` to `RUN corepack enable && corepack prepare pnpm@10.28.0 --activate`. _Requirements: 9.1, 9.2_ +- [ ] **G.2** Verify the sdr-host build still passes: `bash packages/sdr-host/scripts/build-publish.sh --tag testbuild --platform linux/arm64 --load`. _Requirements: 9.1_ + +## Phase H: Documentation Consolidation + +- [ ] **H.1** Rewrite `/Users/ben/Projects/wavekit/docs/DOCKER-SETUP.md` to reflect: + - `pnpm dev` as headline iteration loop. + - `make dev-stack` as container integration. + - The single `compose.yaml` with profiles. + - The GHCR-backed registry cache. + - The new Makefile target inventory. + - _Requirements: 8.1_ +- [ ] **H.2** Either delete `/Users/ben/Projects/wavekit/docker/README.md` OR replace its content with a one-line pointer at `docs/DOCKER-SETUP.md`. _Requirements: 8.2_ +- [ ] **H.3** Condense the "Day-to-day dev (Docker)" section of `/Users/ben/Projects/wavekit/CLAUDE.md` to one paragraph (per Requirement 8.3). _Requirements: 8.3_ + +## Phase I: Dockerignore Audit + +- [ ] **I.1** Verify `.turbo/`, `.pnpm-store/`, `.docker-cache/` are effective. Run `docker buildx build --target base-build --progress=plain .` and grep the transfer log for the offending paths. _Requirements: 11.1_ +- [ ] **I.2** Verify `dist/` exclusion is effective. _Requirements: 11.2_ +- [ ] **I.3** Verify `.kiro/`, `wip/`, `tests/`, `fixtures/raw/`, `fixtures/processed/` exclusions are effective. _Requirements: 11.3_ +- [ ] **I.4** Verify `.dockerignore` does NOT exclude `docker/`, `config/`, `tsconfig.json`, `tsconfig.base.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml`, `turbo.json`, `.npmrc`, `package.json`, `scripts/build-file.mjs` — all needed during the build. _Requirements: 11.4_ + +## Non-negotiables (verify before raising the PR) + +Each item below MUST pass before the PR is reviewable. Run them in order +on a clean machine after `git pull && pnpm install`. + +1. **`pnpm dev` works without Docker.** Stop the Docker daemon; run `pnpm dev`. esbuild watcher and node watcher both run. Editing `src/index.ts` triggers a rebuild + restart within 2 seconds. **(Property 7)** +2. **`make docker-build` warm-cache no-op.** Run `make docker-build` twice in a row from a state where it just succeeded. Second invocation completes in under 30 seconds, all steps `CACHED`. **(Property 4)** +3. **TypeScript-only edit invalidates only `node-build` + final-* downstream.** Touch `src/decoders/registry.ts`, run `make docker-build`. `--progress=plain` SHALL show every decoder build stage as `CACHED`; only `node-build`, `final-base`, `final`, `final-core` execute. **(Property 3)** +4. **One-decoder edit invalidates only that decoder.** Bump `READSB_REF` to a different SHA in the Dockerfile. Run `make docker-build`. `readsb-build` re-executes; every other `*-build` stage is `CACHED`. **(Property 1)** +5. **No plain `docker build` anywhere.** `grep -rn '\bdocker build\b' docker/ Makefile compose.yaml | grep -v buildx` returns no matches. **(Property 11)** +6. **One compose file.** `ls compose*.yaml docker-compose*.yml 2>/dev/null` returns exactly `compose.yaml`. **(Property 12)** +7. **`final-core` contains zero sdrpp residue.** `docker run --rm $(make docker-build TARGET=final-core | tail -1) find /etc/s6-overlay -iname '*sdrpp*'` is empty. `docker run --rm ls /usr/local/bin/ | grep -i sdrpp` is empty. **(Property 8)** +8. **`wavekit-api` dependency graph is minimal.** `docker run --rm ls /etc/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/` returns exactly the line `wavekit-init`. **(Property 9)** +9. **`final` contains exactly one sdrpp-server service.** `docker run --rm find /etc/s6-overlay -name 'sdrpp-server' -type d` returns one match. **(Property 8 inverse)** +10. **CI runs both jobs on PR.** Open a no-op draft PR; both `lint-typecheck-test` and `docker-build` complete green within the CI time budget. **(Requirement 7.1–7.3)** +11. **No emoji in newly-introduced output.** `git diff main -- Makefile docker/ .github/` SHALL have no `[U+1F300]` and above codepoints introduced. _Requirement 6.3_ +12. **The 4 deleted compose files are gone.** `git diff --diff-filter=D --name-only main | grep docker-compose` shows all four. _Requirement 2.6_ +13. **`docker-compose-demod-test.yml` and `docker/Dockerfile.demod-test` are gone.** Same `git diff` shows both. _Requirement 3.12_ +14. **packages/sdr-host/docker-compose.yml is unchanged.** `git diff main packages/sdr-host/docker-compose.yml` is empty. _Requirement 2.8_ +15. **sdr-host Dockerfile uses Corepack.** `grep -c 'corepack' packages/sdr-host/Dockerfile` returns >= 1; `grep -c 'npm install -g pnpm' packages/sdr-host/Dockerfile` returns 0. _Requirement 9.1_ + +## Out-of-Scope Polish (separate spec/PR, not blocking this one) + +- Renovate or Dependabot rule grouping all `_REF` ARG bumps into a single weekly PR. +- tsconfig root vs base alignment (Anti-pattern #11 in design.md; not blocking this work). +- Replacing the existing `cli/` build with a single esbuild bundle to match the root pattern. +- A `wavekit doctor` Makefile target that prints versions of every decoder + node + the active SDR source. +- Tightening `package.json` `pnpm.onlyBuiltDependencies` to add explicit allowlist if pnpm 11+ requires it. diff --git a/docs/superpowers/plans/2026-05-16-build-devx-overhaul.md b/docs/superpowers/plans/2026-05-16-build-devx-overhaul.md new file mode 100644 index 0000000..28be523 --- /dev/null +++ b/docs/superpowers/plans/2026-05-16-build-devx-overhaul.md @@ -0,0 +1,295 @@ +# Build & DX Overhaul Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan phase-by-phase. Each phase below dispatches one fresh subagent that executes the Kiro task list in `.kiro/specs/build-devx-overhaul/tasks.md` for that phase, then returns for review at the verification gate. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace the 737-line single Dockerfile, four parallel compose files, divergent Makefile dev flows, and Docker-free CI with: one narrow-staged Dockerfile, one profile-driven compose file, GHCR-backed BuildKit registry cache, a native `pnpm dev` headline loop, and CI that builds at least `final-core` on every PR. + +**Architecture:** Sequential phase execution (A→B→C→D→E→F→G→H→I) with verification gates between each phase. Each phase produces a clean image build; downstream phases depend on the previous phase's image-builds-clean property. Phases are independent enough to dispatch as separate subagents (clean context per phase) but must run sequentially because each phase consumes the structural output of the previous one. + +**Tech Stack:** Docker BuildKit + buildx + bake (HCL), Compose v2 profiles, s6-overlay v3, GHCR registry cache, GitHub Actions, Node 22 + pnpm 10.28.0 via Corepack, esbuild + node --watch. + +--- + +## Authoritative spec + +The Kiro spec at `/Users/ben/Projects/wavekit/.kiro/specs/build-devx-overhaul/` is the canonical source of truth: + +- `requirements.md` — 11 user stories, 51 acceptance criteria, Out-of-Scope list +- `design.md` — current-state diagnosis (14 anti-patterns), 20-stage Dockerfile structure, compose topology, cache strategy, s6 contract change, 12 correctness properties, risk register +- `tasks.md` — 60 dependency-sequenced tasks across 9 phases with `_Requirements: X.Y_` traceability, plus 15 Non-negotiables + +This plan does NOT duplicate the task list. Each phase below references the corresponding `tasks.md` section and adds: commit/PR boundary guidance, verification gates the subagent MUST hit before returning, and risk flags specific to that phase. + +## Execution mode + +**Recommended: inline subagent-driven (sequential, one subagent per phase, review checkpoints between phases).** + +Rationale: The 9 phases are structurally independent (each touches a distinct slice — file-tree moves, Dockerfile, compose, build scripts, Makefile, CI, sdr-host, docs, dockerignore) so they benefit from clean per-phase contexts. BUT they have a hard sequential dependency: each phase needs the previous phase's "image builds clean" invariant to verify itself. Parallel dispatch would waste tokens running the Dockerfile work twice (once in Phase B's subagent, once again in Phase C's when it discovers it needs a working image to test compose against). Sequential subagent-driven keeps each phase's context narrow while preserving the dependency chain. + +The user-facing review checkpoints align with the two natural integration boundaries already called out in `tasks.md`: + +- **Checkpoint 1** (after Phase B): Dockerfile refactor complete; old compose files still work because stage names are unchanged. PR boundary 1. +- **Checkpoint 2** (after Phase F): New build system fully functional; CI is green. PR boundary 2. + +Phases G/H/I are polish and can land as a single small PR after Checkpoint 2. + +## Commit & PR boundary structure + +Three PRs total. Inside each PR, one commit per phase is the floor; phase B's Dockerfile work may warrant 2–4 commits at the subagent's discretion (one per logical chunk: ARG block + base-build, runtime-base, all `*-build` stages, final-base + finals). + +### PR 1: "build: s6 tree partition + Dockerfile refactor" (Phases A + B) + +- Commits: `refactor(s6): partition sdrpp-server into separate overlay tree` (Phase A), then 2–4 commits scoped per the B.2.x subgroups. +- Verification gate before PR open: Non-negotiables 7, 8, 9 (final-core/final residue checks) pass against locally-built images. The old compose files still build (stage names unchanged); this is the safety net for the refactor. +- Why this boundary: After Phase B the image-build domain is fully migrated, but the dev/CI surface area is untouched. Reviewers can validate the Dockerfile contract isolation without juggling compose/Makefile/CI changes simultaneously. + +### PR 2: "build: collapse compose, adopt bake, rewrite Makefile + CI" (Phases C + D + E + F) + +- Commits: one per phase minimum (`refactor(compose): collapse 4 files into single profile-driven compose.yaml`, `build(bake): introduce GHCR-backed buildx bake pipeline`, `chore(make): trim Makefile to <20 lifecycle targets`, `ci: add docker-build job; bump Node to 22`). +- Verification gate before PR open: Non-negotiables 1–6, 10–13 all pass. The draft-PR smoke test in F.7 is REQUIRED — opening the PR before CI is green wastes reviewer cycles. +- Why this boundary: PR 2 is where the contributor-visible workflow changes (one compose file, new Makefile targets, CI now builds Docker). It needs to land atomically — partial adoption (e.g., new Makefile but old compose) creates a transitional broken state. + +### PR 3: "build: sdr-host Corepack alignment + docs + dockerignore audit" (Phases G + H + I) + +- Commits: one per phase (`chore(sdr-host): switch pnpm install to Corepack pattern`, `docs: consolidate Docker documentation around compose profiles`, `chore: audit .dockerignore effectiveness`). +- Verification gate before PR open: Non-negotiables 14, 15 pass. The `pnpm dev` headline and `make dev-stack` integration paths in updated docs SHOULD be smoke-tested by the subagent before the PR opens. +- Why this boundary: Pure polish; not blocking the build system itself. Keeping it separate means reviewers can rubber-stamp it. + +> **Note on Phase A commit hygiene:** Phase A is a pure file-tree reorganization (`git mv` semantics — though the tasks describe it as create + delete, the subagent should use `git mv` where possible to preserve history). The verification step A.6 is the only "test" available before the Dockerfile consumes the new layout in Phase B. + +## Verification gates between phases + +Each gate below MUST be cleared before the next phase's subagent is dispatched. If a gate fails, the subagent owning that phase is re-dispatched with the failure context; no skipping. + +### Gate A→B: s6 tree partition is structurally correct + +Subagent owning Phase A MUST report verification of: + +- `find docker/overlay/s6-overlay -iname '*sdrpp*'` returns empty. +- `find docker/overlay/s6-overlay-sdrpp -type f` lists exactly the 6 files in task A.6. +- `cat docker/overlay/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/wavekit-init` exists and is the only file in that directory. + +Reviewer checks: file moves preserve git history (look for `R` lines in `git diff --stat`). + +### Gate B→C: Dockerfile contract holds for all 4 final stages + +Subagent owning Phase B MUST report verification of `tasks.md` task B.2.11 (all 4 targets build via `docker buildx build --target --load`) AND B.2.12 (Properties 8 & 9 against `wavekit:dev-core` and a sdrpp service exists in `wavekit:dev`). Additionally: + +- The `--progress=plain` output of a touched-src rebuild SHALL show every `*-build` stage as `CACHED` (manual verification of Property 3). +- `grep -E '^\s*ARG (S6_OVERLAY_VERSION|.*_REF)=' Dockerfile | wc -l` returns ≥ 14 (one S6 + ≥13 decoder refs). + +Reviewer checks: each `*-build` stage uses `FROM base-build`, declares its own apt deps (not `base-build`'s), uses pinned ref via `git fetch --depth 1 origin "${REF}" && git checkout --detach FETCH_HEAD`. + +### Gate C→D: compose collapse loads each profile cleanly + +Subagent owning Phase C MUST report: + +- All 4 `docker compose --profile

config` runs return zero warnings (tasks C.3–C.6). +- The 4 old compose files and `docker/Dockerfile.demod-test` are deleted from working tree. +- End-to-end task C.12 succeeded (`dev` profile up + 200 on `/health`). + +Reviewer checks: `packages/sdr-host/docker-compose.yml` is unchanged (Non-negotiable 14). + +### Gate D→E: bake pipeline supersedes plain `docker build` + +Subagent owning Phase D MUST report: + +- `grep -rn '\bdocker build\b' docker/ Makefile compose.yaml | grep -v buildx` returns no matches (Property 11 / Non-negotiable 5). +- D.2 and D.3 both passed (single-arch and multi-arch local bake). +- `docker/init.sh` has the `.docker-cache` mkdir removed; `docker/push.sh` is GHCR-only with `WAVEKIT_GH_OWNER` env support and no `linux/arm/v7`. +- D.8 cache seed completed (this is the critical bootstrap — without it, downstream `cache-from` calls return "no cache"). + +Reviewer checks: `docker/bake.hcl` defines exactly the targets and groups in design §4.3 (`final`, `final-core`, `final-sdrpp`, `final-demod`, `ci-core` helper, `default` and `demod` groups). + +### Gate E→F: Makefile targets match the canonical list exactly + +Subagent owning Phase E MUST report: + +- `make help` output enumerates only the canonical targets listed in requirement 6.1 plus `fixtures-*` and `sdr-host-*`. The deletion list in requirement 6.2 is verified absent. +- E.4 end-to-end test passed for `make dev`, `make dev-stack`, `make dev-dashboard`, `make docker-build`, `make demod-test`. +- No emoji or decorative ANSI in newly-introduced output (Non-negotiable 11). + +Reviewer checks: `make dev-stack` runs `docker compose --profile dev up --build` literally; `make docker-build` runs `docker buildx bake --file docker/bake.hcl default` literally. + +### Gate F→G: CI is green on a no-op PR + +Subagent owning Phase F MUST execute task F.7 (open a draft PR with a trivial change, e.g., a whitespace edit to `readme.md`, and verify both jobs complete green). This is mandatory because the bake action's pull-cache-from-GHCR path can only be validated against the live GHCR cache seeded in D.8. If F.7 fails: + +- Diagnose: cache-from returning auth error vs no-cache (latter is correct fallback per requirement 4.3, former is a CI config bug). +- Fix the CI yaml; re-push. +- Do NOT proceed to Phase G with a failing CI workflow. + +Reviewer checks: the two CI jobs run on `pull_request`; the GHCR login + push only runs on `push` to main (gated by `github.event_name` and `github.ref`). + +### Gate G→H: sdr-host build still passes + +Subagent owning Phase G MUST report G.2 success (sdr-host build via the existing `build-publish.sh` script with `--load`). + +Reviewer checks: pnpm version pinned in `packages/sdr-host/Dockerfile` matches `packageManager` field in root `package.json` exactly (requirement 9.2). + +### Gate H→I: docs cross-references are intact + +Subagent owning Phase H MUST report: + +- `docs/DOCKER-SETUP.md` references the canonical compose file, profiles, Makefile targets, and `pnpm dev` headline. +- `docker/README.md` is either deleted or reduced to a one-line pointer. +- `CLAUDE.md`'s "Day-to-day dev (Docker)" section is one paragraph. + +Reviewer checks: `grep -rn 'docker-compose.dev' docs/ CLAUDE.md` returns no matches (no stale references to deleted files). + +### Final gate (post-Phase I): all 15 Non-negotiables pass + +Before PR 3 opens, the subagent MUST run through the 15 Non-negotiables in `tasks.md` and report pass/fail for each. Failures block the PR. + +## Risks the implementer should watch for + +Mapped to design §11 risk register, with execution-time mitigations. + +### R1: SHA capture in B.1.1 picks a broken upstream commit + +The Kiro plan assumes `git ls-remote` HEAD of each decoder repo is buildable. This is true today (the project currently builds against branch tips). But there's a non-zero chance one of the 12 repos has an in-flight breakage at the moment B.1.1 runs. + +**Mitigation:** After B.2.11 (smoke-test build), if any stage fails to compile, do NOT debug the upstream code. Instead, walk that decoder's git log back to the most recent green tag (or a SHA 24h older) and use that. Record the choice in the commit message: `pin to (HEAD broken: )`. + +### R2: arm64 ncurses path during B.2.6 + +Task B.2.6 says: remove the hardcoded `x86_64-linux-gnu/libncurses*` COPY from `final-core`; test arm64; if missing, add `TARGETARCH`-conditional COPY. The subagent MUST run `docker buildx build --target final-core --platform linux/arm64 --load` after the removal. If `dsd-fme --help` or `find / -name "libncurses*"` shows ncurses is satisfied by the `libncurses6` apt install in `runtime-base`, leave the COPY deleted (requirement 3.13's preferred path). Otherwise restore as `TARGETARCH`-conditional with `amd64→x86_64-linux-gnu`, `arm64→aarch64-linux-gnu`. + +### R3: `node-linker=isolated` runtime breakage + +Removing the `pnpm config set node-linker hoisted` block (B.2.5) flips the container's effective layout from hoisted to isolated. The `.npmrc` says isolated is already the project default, so this should be a no-op — but the container has been running with the hoisted override for long enough that a transitive dependency might have grown an implicit assumption. + +**Mitigation:** After B.2.11, run `docker run --rm wavekit:dev-core node -e "require('fastify'); require('pino'); require('zod'); require('@wavekit/shared'); require('@wavekit/api-types'); console.log('OK')"`. If any require fails, the symbol is `MODULE_NOT_FOUND` for a workspace package, which signals the dist's import path expectations were specific to the hoisted layout. Fall back to the documented escape hatch: re-add the linker reconfigure in `node-build` as a single change (NOT a double install — set the linker BEFORE the single install). Document the regression in a follow-up issue rather than rolling back the whole refactor. + +### R4: GHCR cache write race during D.8 bootstrap + +The D.8 step does a one-time `--push --cache-to mode=max` from a local machine to seed each `cache-` ref. If two people run D.8 concurrently (unlikely but possible), the cache-to writes can stomp each other. The result is half-populated cache refs, which manifest as Property 5 failure on cold builds. + +**Mitigation:** Coordinate the bootstrap. Only one machine runs D.8. After it returns, verify with `docker buildx imagetools inspect ghcr.io/coriou/wavekit:cache-readsb-build` (or any one cache ref) shows two platform manifests (amd64 + arm64). If only one arch appears, re-run D.8 with `--set "*.platform=linux/amd64,linux/arm64"` explicitly. + +### R5: F.7 draft-PR test on a fresh fork without GHCR access + +If the PR is opened from a fork (not a branch in the upstream repo), the `docker-build` job runs without `secrets.GITHUB_TOKEN` access to GHCR. The cache-from will return "no cache" rather than error (requirement 4.3 contract), so the build still succeeds — but it will be a 25-40 minute cold build, not a 2-5 minute warm one. This is correct behaviour, not a regression. The subagent owning Phase F MUST validate from a branch in the upstream repo (not a fork) so the cache hit confirms the happy path. + +### R6: Compose `cache_from` typed-cache parsing + +Compose v2 supports `cache_from: type=registry,ref=...` only in recent versions. The subagent MUST verify locally with `docker compose version` before C.3 — Docker Desktop 4.30+ / docker-compose CLI v2.27+ supports the typed syntax. Older versions silently ignore the cache directive, which manifests as warm-build behaviour being slower than expected (not a build failure, just a missed optimization). + +### R7: s6 contract change creates first-run log noise + +The s6 hard dep `wavekit-api → sdrpp-server` removal is intentional (design §5). When `final` mode boots, `wavekit-api` may briefly try to connect before `sdrpp-server` binds its port, producing one or two `SourceConnectionError` log lines before `SourceManager`'s exponential backoff retries succeed. This is acceptable per design but will look like a regression in first-launch logs. + +**Mitigation:** No code change. Document in the PR description and the `docs/DOCKER-SETUP.md` rewrite (Phase H.1) that early-boot reconnect lines are expected and the source-availability contract has moved from s6 to `SourceManager`. If grafana/log dashboards alert on `SourceConnectionError`, update their thresholds to ignore the first 10 seconds post-boot. + +### R8: Renamed Makefile targets break ambient muscle memory + +The Phase E rewrite renames `dev-up` → `dev-stack`, `docker-logs` → `dev-stack-logs`, `docker-shell` → `dev-shell`. Contributors with these in shell history or in personal aliases will hit "target not found" errors after pulling. + +**Mitigation:** Phase H.3 (`CLAUDE.md` condense) is the only contributor-facing documentation update; ensure the rewritten paragraph explicitly names the new targets. Optionally, ONE Makefile entry can carry a deprecation hint: + +```make +dev-up: + @echo "[wavekit] make dev-up is now 'make dev-stack'" >&2; exit 1 +``` + +This is OPTIONAL — requirement 6.2 says these targets SHALL NOT exist, but a hint that errors out cleanly is arguably "informative null behaviour" rather than a re-introduction. If the subagent adds it, mark it for removal in 30 days. Lean toward NOT adding it — requirement 6.2's wording is "SHALL NOT define", and the cleaner path is just to delete and let muscle memory adjust. + +### R9: Property 5 (cold-cache GHCR pull) cannot be verified locally without prune + +Property 5 demands a `docker system prune -af && make docker-build` test that pulls cache from GHCR. The subagent owning Phase D MUST NOT run this during routine D.x verification because pruning kills the local layer cache that subsequent tasks rely on. Run it ONCE at the end of Phase F (after CI has successfully written to GHCR via the main-branch push or the D.8 bootstrap), then proceed. + +### R10: `final-demod` matplotlib/scipy install bloat + +Task B.2.10 apt-installs `python3-matplotlib`, `python3-scipy`. These pull in ~400MB of Debian deps. This is intentional (the demod test environment is interactive and benefits from these), but it WILL make `final-demod` the largest stage. If image size becomes a complaint, requirement 10's "modest, opportunistic" framing covers trimming this in a follow-up — DO NOT trim during this refactor (out-of-scope per requirement 10.5). + +## Tasks: defer to `.kiro/specs/build-devx-overhaul/tasks.md` + +The 60-task implementation checklist lives in the Kiro spec. The subagent dispatched for each phase below MUST open `tasks.md`, execute that phase's tasks in order, check off each with `- [x]`, and report the verification gate's pass/fail to the dispatching session. + +### Phase A: s6 Service Tree Partition (tasks A.1–A.6, 6 tasks) + +- Files: `docker/overlay/s6-overlay/` (deletions), `docker/overlay/s6-overlay-sdrpp/` (creations). +- Gate: Gate A→B above. +- Commit: single commit `refactor(s6): partition sdrpp-server into separate overlay tree`. + +### Phase B: Dockerfile Refactor (tasks B.1.1–B.2.12, 14 tasks) + +- Files: `Dockerfile` (top-to-bottom rewrite). +- Gate: Gate B→C above. +- Commits: 2–4 logical commits per the B.2.x subgroups. +- Risk flags: R1, R2, R3. + +### Phase C: Compose Collapse (tasks C.1–C.12, 12 tasks) + +- Files: `compose.yaml` (new), 4 old compose files + `docker/Dockerfile.demod-test` (deletions). +- Gate: Gate C→D above. +- Commit: single commit `refactor(compose): collapse 4 files into single profile-driven compose.yaml`. +- Risk flags: R6. + +### Phase D: Bake + Registry Cache (tasks D.1–D.8, 8 tasks) + +- Files: `docker/bake.hcl` (new), `docker/build.sh` (rewrite or delete), `docker/push.sh` (rewrite), `docker/init.sh` (edit), `docker/platform-utils.sh` (likely delete). +- Gate: Gate D→E above. +- Commit: single commit `build(bake): introduce GHCR-backed buildx bake pipeline`. +- Risk flags: R4. + +### Phase E: Makefile Cleanup (tasks E.1–E.4, 4 tasks) + +- Files: `Makefile` (rewrite). +- Gate: Gate E→F above. +- Commit: single commit `chore(make): trim Makefile to <20 lifecycle targets`. +- Risk flags: R8. + +### Phase F: CI Workflow (tasks F.1–F.7, 7 tasks) + +- Files: `.github/workflows/ci.yml` (rewrite). +- Gate: Gate F→G above. +- Commit: single commit `ci: add docker-build job; bump Node to 22; corepack-driven pnpm`. +- Risk flags: R5, R9. + +### Phase G: sdr-host Alignment (tasks G.1–G.2, 2 tasks) + +- Files: `packages/sdr-host/Dockerfile` (one-line edit). +- Gate: Gate G→H above. +- Commit: single commit `chore(sdr-host): switch pnpm install to Corepack pattern`. + +### Phase H: Documentation Consolidation (tasks H.1–H.3, 3 tasks) + +- Files: `docs/DOCKER-SETUP.md` (rewrite), `docker/README.md` (delete or one-line), `CLAUDE.md` (condense). +- Gate: Gate H→I above. +- Commit: single commit `docs: consolidate Docker documentation around compose profiles`. +- Risk flags: R7 (mentioned in DOCKER-SETUP.md). + +### Phase I: Dockerignore Audit (tasks I.1–I.4, 4 tasks) + +- Files: `.dockerignore` (verify only; no functional change expected). +- Gate: Final gate (15 Non-negotiables) above. +- Commit: single commit `chore: audit .dockerignore effectiveness` (may be empty-diff if everything already correct; if so, fold into the Phase H commit and skip a standalone commit). + +## Correctness properties to verify + +The design's 12 correctness properties (design.md §10) map to the Non-negotiables (tasks.md). The subagent owning each phase MUST verify the properties that phase introduces: + +- Phase A introduces no properties (pure file moves; Property 8/9 verified in Phase B). +- Phase B introduces Properties 1, 2, 3, 8, 9. +- Phase C introduces Property 12. +- Phase D introduces Properties 4, 5, 6, 11. +- Phase E introduces no new properties (verified via subset of 1–6). +- Phase F introduces Property 10 (multi-arch in registry, only verifiable after main-branch CI push). +- Phases G/H/I introduce no new properties. +- Property 7 (`pnpm dev` without Docker) is verified once after Phase E (when the Makefile target lands) and again at the final gate. + +## Self-review checklist for the implementing subagents + +Each phase subagent MUST run this checklist before returning to the dispatching session: + +1. **Spec coverage:** Every requirement number cited in this phase's `tasks.md` entries has a check-marked task. Any uncheck-marked tasks are explicitly flagged in the return message as "intentionally deferred" with reason. +2. **No placeholder regressions:** `grep -rn 'TODO\|FIXME\|XXX\|TBD' Dockerfile docker/ compose.yaml Makefile .github/workflows/ci.yml 2>/dev/null` returns no new matches beyond pre-existing ones (capture the baseline at phase start). +3. **No out-of-scope creep:** None of the 12 items in requirements.md "Out-of-Scope" section are touched. Specifically: no tsconfig changes, no decoder removal, no base image swap. +4. **Frequent commits:** Phase B's 14 tasks SHOULD produce 2–4 commits, not one giant commit. Phases A, C, D, E, F, G, H, I produce one commit each. No phase produces zero commits unless its diff is genuinely empty. + +## Execution Handoff + +Plan complete and saved to `/Users/ben/Projects/wavekit/docs/superpowers/plans/2026-05-16-build-devx-overhaul.md`. + +The recommended execution mode is **inline subagent-driven**: dispatch one fresh subagent per phase, run the verification gate after each, review between phases. Phases are sequentially dependent (each consumes the previous phase's "image builds clean" property), so parallel dispatch would waste tokens. Two reviewer-facing PR boundaries are baked in: after Phase B (Checkpoint 1, Dockerfile refactor reviewable in isolation) and after Phase F (Checkpoint 2, full new build system online). From 8c42d69b8841a14c878abd3b88075450ad6b270b Mon Sep 17 00:00:00 2001 From: Benjamin Coriou Date: Sat, 16 May 2026 01:58:58 +0200 Subject: [PATCH 2/4] refactor(s6): partition sdrpp-server into separate overlay tree Move the sdrpp-server service definition, its services/contents.d/ registration, and its user/contents.d/ registration out of the canonical docker/overlay/s6-overlay/ tree and into a sibling docker/overlay/s6-overlay-sdrpp/ tree. Delete the canonical tree's wavekit-api -> sdrpp-server hard dep; wavekit-api now depends only on wavekit-init. Phase A of build/devx-overhaul; sets up the Phase B Dockerfile to COPY the canonical tree into final-base (no sdrpp residue) and additively COPY the sdrpp tree into the final stage only. See .kiro/specs/build-devx-overhaul/{requirements,design,tasks}.md and docs/superpowers/plans/2026-05-16-build-devx-overhaul.md. --- .kiro/specs/build-devx-overhaul/tasks.md | 12 ++++++------ .../s6-rc.d/sdrpp-server/dependencies.d/wavekit-init | 0 .../s6-rc.d/sdrpp-server/finish | 0 .../s6-rc.d/sdrpp-server/run | 0 .../s6-rc.d/sdrpp-server/type | 0 .../s6-rc.d/services/contents.d/sdrpp-server | 0 .../s6-rc.d/user/contents.d/sdrpp-server | 0 .../s6-rc.d/wavekit-api/dependencies.d/sdrpp-server | 1 - 8 files changed, 6 insertions(+), 7 deletions(-) rename docker/overlay/{s6-overlay => s6-overlay-sdrpp}/s6-rc.d/sdrpp-server/dependencies.d/wavekit-init (100%) rename docker/overlay/{s6-overlay => s6-overlay-sdrpp}/s6-rc.d/sdrpp-server/finish (100%) rename docker/overlay/{s6-overlay => s6-overlay-sdrpp}/s6-rc.d/sdrpp-server/run (100%) rename docker/overlay/{s6-overlay => s6-overlay-sdrpp}/s6-rc.d/sdrpp-server/type (100%) rename docker/overlay/{s6-overlay => s6-overlay-sdrpp}/s6-rc.d/services/contents.d/sdrpp-server (100%) rename docker/overlay/{s6-overlay => s6-overlay-sdrpp}/s6-rc.d/user/contents.d/sdrpp-server (100%) delete mode 100644 docker/overlay/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/sdrpp-server diff --git a/.kiro/specs/build-devx-overhaul/tasks.md b/.kiro/specs/build-devx-overhaul/tasks.md index 9f0f9e0..e2f9bdd 100644 --- a/.kiro/specs/build-devx-overhaul/tasks.md +++ b/.kiro/specs/build-devx-overhaul/tasks.md @@ -8,12 +8,12 @@ Two checkpoints split the work into adoptable chunks. Touches no Dockerfile yet. Pure file-tree reorganization. Safe to land alone. -- [ ] **A.1** Remove the file `docker/overlay/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/sdrpp-server`. The canonical tree's `wavekit-api` SHALL now only depend on `wavekit-init`. _Requirements: 3.10, 3.11_ -- [ ] **A.2** Create directory `docker/overlay/s6-overlay-sdrpp/s6-rc.d/`. _Requirements: 3.8_ -- [ ] **A.3** Move `docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/` (with all its files: `type`, `run`, `finish`, `dependencies.d/wavekit-init`) to `docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/`. _Requirements: 3.8_ -- [ ] **A.4** Move `docker/overlay/s6-overlay/s6-rc.d/services/contents.d/sdrpp-server` to `docker/overlay/s6-overlay-sdrpp/s6-rc.d/services/contents.d/sdrpp-server`. _Requirements: 3.8_ -- [ ] **A.5** Move `docker/overlay/s6-overlay/s6-rc.d/user/contents.d/sdrpp-server` to `docker/overlay/s6-overlay-sdrpp/s6-rc.d/user/contents.d/sdrpp-server`. _Requirements: 3.8_ -- [ ] **A.6** Verify `find docker/overlay/s6-overlay -iname '*sdrpp*'` returns empty. Verify `find docker/overlay/s6-overlay-sdrpp -type f` lists exactly: `sdrpp-server/type`, `sdrpp-server/run`, `sdrpp-server/finish`, `sdrpp-server/dependencies.d/wavekit-init`, `services/contents.d/sdrpp-server`, `user/contents.d/sdrpp-server`. _Requirements: 3.10, 3.11_ +- [x] **A.1** Remove the file `docker/overlay/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/sdrpp-server`. The canonical tree's `wavekit-api` SHALL now only depend on `wavekit-init`. _Requirements: 3.10, 3.11_ +- [x] **A.2** Create directory `docker/overlay/s6-overlay-sdrpp/s6-rc.d/`. _Requirements: 3.8_ +- [x] **A.3** Move `docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/` (with all its files: `type`, `run`, `finish`, `dependencies.d/wavekit-init`) to `docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/`. _Requirements: 3.8_ +- [x] **A.4** Move `docker/overlay/s6-overlay/s6-rc.d/services/contents.d/sdrpp-server` to `docker/overlay/s6-overlay-sdrpp/s6-rc.d/services/contents.d/sdrpp-server`. _Requirements: 3.8_ +- [x] **A.5** Move `docker/overlay/s6-overlay/s6-rc.d/user/contents.d/sdrpp-server` to `docker/overlay/s6-overlay-sdrpp/s6-rc.d/user/contents.d/sdrpp-server`. _Requirements: 3.8_ +- [x] **A.6** Verify `find docker/overlay/s6-overlay -iname '*sdrpp*'` returns empty. Verify `find docker/overlay/s6-overlay-sdrpp -type f` lists exactly: `sdrpp-server/type`, `sdrpp-server/run`, `sdrpp-server/finish`, `sdrpp-server/dependencies.d/wavekit-init`, `services/contents.d/sdrpp-server`, `user/contents.d/sdrpp-server`. _Requirements: 3.10, 3.11_ ## Phase B: Dockerfile Refactor diff --git a/docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/dependencies.d/wavekit-init b/docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/dependencies.d/wavekit-init similarity index 100% rename from docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/dependencies.d/wavekit-init rename to docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/dependencies.d/wavekit-init diff --git a/docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/finish b/docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/finish similarity index 100% rename from docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/finish rename to docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/finish diff --git a/docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/run b/docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/run similarity index 100% rename from docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/run rename to docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/run diff --git a/docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/type b/docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/type similarity index 100% rename from docker/overlay/s6-overlay/s6-rc.d/sdrpp-server/type rename to docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/type diff --git a/docker/overlay/s6-overlay/s6-rc.d/services/contents.d/sdrpp-server b/docker/overlay/s6-overlay-sdrpp/s6-rc.d/services/contents.d/sdrpp-server similarity index 100% rename from docker/overlay/s6-overlay/s6-rc.d/services/contents.d/sdrpp-server rename to docker/overlay/s6-overlay-sdrpp/s6-rc.d/services/contents.d/sdrpp-server diff --git a/docker/overlay/s6-overlay/s6-rc.d/user/contents.d/sdrpp-server b/docker/overlay/s6-overlay-sdrpp/s6-rc.d/user/contents.d/sdrpp-server similarity index 100% rename from docker/overlay/s6-overlay/s6-rc.d/user/contents.d/sdrpp-server rename to docker/overlay/s6-overlay-sdrpp/s6-rc.d/user/contents.d/sdrpp-server diff --git a/docker/overlay/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/sdrpp-server b/docker/overlay/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/sdrpp-server deleted file mode 100644 index 899da3c..0000000 --- a/docker/overlay/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/sdrpp-server +++ /dev/null @@ -1 +0,0 @@ -sdrpp-server From c00d1f849a8e8503b25869c752d7e097c9b2575b Mon Sep 17 00:00:00 2001 From: Benjamin Coriou Date: Sat, 16 May 2026 02:53:30 +0200 Subject: [PATCH 3/4] build(docker): rewrite Dockerfile into 20 named, narrow-purpose stages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the 737-line monolithic Dockerfile with 20 stages per the Build & DX Overhaul spec (Phase B). Each decoder builds on a thin `base-build` stage and installs only its own apt deps; upstream sources are pinned to SHAs at the top of the file. The 4 final targets (`final`, `final-core`, `final-sdrpp`, `final-demod`) build cleanly via `docker buildx build --target --load`. Stage inventory: - base-build / runtime-base (toolchain + runtime libs) - 12 decoder builders: sdrpp, dsd-fme, multimon-ng, rtl433, acarsdec, ais-catcher, direwolf, dumpvdl2, readsb, soapy-rtltcp, csdr, lora - node-build (single pnpm install) - 5 finals: final-base (common ancestor), final (+SDR++), final-core (no additions), final-sdrpp (SDR++ only), final-demod (interactive) Key changes: - Pinned ARG _REF for every upstream (initial values captured via `git ls-remote`; bumps go through PR review). Re-uses lora-build's existing pinned-checkout pattern across all decoders. - `runtime-base` selects s6-overlay via TARGETARCH (was BUILDPLATFORM — wrong for cross-arch builds). Drops the /var/lib/apt cache mount, dedupes libsox-fmt-all, removes unused tini. - `final-base` consolidates everything `final` and `final-core` shared. `final-core` is a one-line stage; the mode partition is now achieved by additive COPY from `docker/overlay/s6-overlay-sdrpp/` (Phase A) into `final`, not by `rm -rf` post-hoc (Req 3.10). - Hardcoded `COPY /usr/lib/x86_64-linux-gnu/libncurses*` deleted — the runtime `libncurses6` apt package satisfies dsd-fme's linkage on both amd64 and arm64 (Req 3.13). - Old `FROM final AS default` stage removed; bake-driven default group in Phase D supersedes it. Notes / mitigations: - R3 (node-build): a pure-isolated single install couldn't survive both (a) the pnpm store cache mount dropping the global virtual store path after the layer commits, and (b) `cli/node_modules` and `packages/*/node_modules` leaking into the build context from the host (.dockerignore only excludes root-level `node_modules`). Per the Phase B brief, set `node-linker hoisted` BEFORE the single install. Project .npmrc default (`isolated`) is unchanged so `pnpm dev` is unaffected. Follow-up: fix .dockerignore to exclude `**/node_modules` so the override can be lifted (Req 3.6). - SDR++ install paths corrected to `/usr/bin/sdrpp` and `/usr/lib/sdrpp/` (current upstream installs there, not `/usr/local/`; the old COPY silently no-op'd). SDR++ runtime libs (libglfw3, libvolk2.5, libglvnd0, libopengl0) added only to `final` and `final-sdrpp` so `final-core` stays slim. - `final-sdrpp` reconstructed: the previous `COPY .../s6-rc.d/base ...` referenced a directory renamed to `wavekit-init/` in commit 7d18546. Now uses per-file COPYs of the canonical structural pieces (services/type, user/type, user/contents.d/services) plus the sdrpp-server service from the sibling overlay. - dsd-fme verification widened from `--version` to `command -v` because the pinned HEAD no longer accepts `--version`. Binary presence on PATH is the relevant smoke contract. Gate B verification (PASS): - final-core has no sdrpp residue (Property 8) - wavekit-api dependencies: only `wavekit-init` (Property 9) - final has the sdrpp-server tree (additive overlay landed) - `node -e "require('fastify'); ..."` returns OK in final-core - 15 top-of-file ARGs (1 S6_OVERLAY_VERSION + 14 decoder refs) - touched-src warm build keeps every decoder *-build stage CACHED Builds all 4 targets via buildx --load successfully. Refs: .kiro/specs/build-devx-overhaul/{requirements,design,tasks}.md docs/superpowers/plans/2026-05-16-build-devx-overhaul.md --- .kiro/specs/build-devx-overhaul/tasks.md | 33 +- Dockerfile | 1007 ++++++++++++---------- 2 files changed, 578 insertions(+), 462 deletions(-) diff --git a/.kiro/specs/build-devx-overhaul/tasks.md b/.kiro/specs/build-devx-overhaul/tasks.md index e2f9bdd..c44a037 100644 --- a/.kiro/specs/build-devx-overhaul/tasks.md +++ b/.kiro/specs/build-devx-overhaul/tasks.md @@ -23,33 +23,34 @@ D). ### B.1 Capture upstream SHAs -- [ ] **B.1.1** For each of the 12 upstream sources (SDR++, mbelib, dsd-fme, multimon-ng, rtl_433, acarsdec, AIS-catcher, direwolf, libacars, dumpvdl2, readsb, SoapyRTLTCP, csdr), `git ls-remote` the current branch tip and record the SHA. Save these as the initial values for the `ARG _REF` declarations. _Requirements: 3.3_ -- [ ] **B.1.2** Keep `GR_LORA_SDR_REF=862746dd1cf635c9c8a4bfbaa2c3a0ec3a5306c9` as-is. _Requirements: 3.3_ +- [x] **B.1.1** For each of the 12 upstream sources (SDR++, mbelib, dsd-fme, multimon-ng, rtl_433, acarsdec, AIS-catcher, direwolf, libacars, dumpvdl2, readsb, SoapyRTLTCP, csdr), `git ls-remote` the current branch tip and record the SHA. Save these as the initial values for the `ARG _REF` declarations. _Requirements: 3.3_ +- [x] **B.1.2** Keep `GR_LORA_SDR_REF=862746dd1cf635c9c8a4bfbaa2c3a0ec3a5306c9` as-is. _Requirements: 3.3_ ### B.2 Rewrite Dockerfile -- [ ] **B.2.1** Add a single top-of-file ARG block declaring `S6_OVERLAY_VERSION` and all 13 decoder/binary upstream refs from §B.1. _Requirements: 3.3_ -- [ ] **B.2.2** Replace the current `base-deps` stage with a thin `base-build` stage containing only `build-essential`, `cmake`, `git`, `pkg-config`, `ca-certificates`, `curl`. _Requirements: 3.1_ -- [ ] **B.2.3** Refactor `runtime-base`: +- [x] **B.2.1** Add a single top-of-file ARG block declaring `S6_OVERLAY_VERSION` and all 13 decoder/binary upstream refs from §B.1. _Requirements: 3.3_ +- [x] **B.2.2** Replace the current `base-deps` stage with a thin `base-build` stage containing only `build-essential`, `cmake`, `git`, `pkg-config`, `ca-certificates`, `curl`. _Requirements: 3.1_ +- [x] **B.2.3** Refactor `runtime-base`: - Switch the s6-overlay platform-detection block from `BUILDPLATFORM` to `TARGETARCH` (case mapping per sdr-host pattern). - Drop the `--mount=type=cache,target=/var/lib/apt` mount; keep only the `/var/cache/apt` mount. - Dedupe the duplicate `libsox-fmt-all` line. - Verify whether `tini` is referenced anywhere under `docker/`; if not, remove the apt package. - _Requirements: 3.5, 10.1, 10.2, 10.4, 14_ -- [ ] **B.2.4** Rewrite each of the 12 `*-build` stages: +- [x] **B.2.4** Rewrite each of the 12 `*-build` stages: - `FROM base-build`. - One `RUN apt-get install` with only this decoder's deps, protected by `--mount=type=cache,target=/var/cache/apt,sharing=locked`, ending with `rm -rf /var/lib/apt/lists/*`. - `ARG TARGETARCH` if the stage does arch-conditional work. - Pinned-ref checkout via `git clone --no-checkout && cd && git fetch --depth 1 origin "${REF}" && git checkout --detach FETCH_HEAD`. - `make install` or equivalent. - _Requirements: 3.2, 3.3, 3.4_ -- [ ] **B.2.5** Refactor `node-build`: +- [x] **B.2.5** Refactor `node-build`: - Remove the `pnpm config set node-linker hoisted` + second install block (lines 432–438 of the current Dockerfile). - Single `pnpm install --frozen-lockfile --prod=false` invocation with `--mount=type=cache,target=/root/.local/share/pnpm/store,sharing=locked`. - Keep the existing Turbo cache mount on typecheck + build. - Keep the existing `pnpm prune --prod` final step. - _Requirements: 3.6_ -- [ ] **B.2.6** Create a new `final-base` stage: + - **Note (R3 mitigation applied)**: pure-isolated single install couldn't survive cache-mount removal AND host-leaked `cli/node_modules`/`packages/*/node_modules` from build context. Per Phase B brief R3, switched to hoisted layout via `pnpm config set node-linker hoisted` BEFORE the single install. Project `.npmrc` default (`isolated`) is unchanged so `pnpm dev` is unaffected. Follow-up: fix `.dockerignore` to exclude `**/node_modules` so override can be lifted (Req 3.6). +- [x] **B.2.6** Create a new `final-base` stage: - `FROM runtime-base`. - Apt-install python3 + gnuradio + python3-numpy + python3-protobuf (drop python3-cryptography if verification shows it's unused by `lora_meshtastic_decode.py`). - COPY every decoder binary + library from `*-build` stages (current `final` and `final-core` share these; consolidate). @@ -67,7 +68,9 @@ D). - `ENTRYPOINT ["/init"]`. - LABELs: maintainer, version. - _Requirements: 3.7, 3.13, 3.14, 3.15, 10.1, 10.3_ -- [ ] **B.2.7** Define `final` stage: + - **R2 finding (ncurses)**: amd64 verification shows `libncurses6` apt package in `runtime-base` provides `/lib/x86_64-linux-gnu/libncursesw.so.6` which satisfies `dsd-fme`'s linkage. The hardcoded `COPY --from=dsd-fme-build /usr/lib/x86_64-linux-gnu/libncurses*` was DELETED entirely (no replacement). `libncurses6` is multi-arch in Debian so the same install resolves on arm64. + - **Note (python3-cryptography)**: verified KEPT — `docker/scripts/lora_meshtastic_decode.py` imports `from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes` (line 23). +- [x] **B.2.7** Define `final` stage: - `FROM final-base`. - COPY SDR++ binaries + libraries from `sdrpp-build`. - COPY `docker/scripts/start-sdrpp.sh finish-sdrpp.sh` → `/usr/local/bin/`. @@ -76,17 +79,19 @@ D). - EXPOSE add 5259, 7355. - LABEL mode=full. - _Requirements: 3.8, 3.10_ -- [ ] **B.2.8** Define `final-core` stage: + - **Note**: current SDR++ master installs to `/usr/bin/sdrpp`, `/usr/lib/libsdrpp_core.so`, `/usr/lib/sdrpp/plugins/`. The old Dockerfile's COPY paths (`/usr/local/bin/sdrpp*`, `/usr/local/lib/libsdrpp*`) silently no-op'd — they have been corrected here. Also added SDR++-specific runtime libs (`libglfw3`, `libvolk2.5`, `libglvnd0`, `libopengl0`) to `final` only (kept out of `runtime-base` so `final-core` stays slim). +- [x] **B.2.8** Define `final-core` stage: - `FROM final-base`. - LABEL mode=core. - No further content. (Existence of the stage is the contract.) - _Requirements: 3.9, 3.10, 3.11_ -- [ ] **B.2.9** Rewrite `final-sdrpp` stage: +- [x] **B.2.9** Rewrite `final-sdrpp` stage: - `FROM runtime-base`. - Apply the `TARGETARCH` s6 platform-detection fix (inherited from `runtime-base` refactor in B.2.3). - Existing structure otherwise preserved. - _Requirements: 3.5_ -- [ ] **B.2.10** Add `final-demod` stage: + - **Note**: the old `final-sdrpp` was already broken — it copied `s6-rc.d/base` which had been renamed to `s6-rc.d/wavekit-init` since commit 7d18546 (Jan 2). Reconstructed with per-file COPY: copy `services/type`, `user/type`, `user/contents.d/services` from the canonical tree, plus `sdrpp-server/{type,run,finish}` and the two `contents.d` registration files from the sdrpp overlay. `sdrpp-server/dependencies.d/wavekit-init` is intentionally omitted (this image doesn't ship wavekit-init). All COPYs are additive (Req 3.10). +- [x] **B.2.10** Add `final-demod` stage: - `FROM runtime-base`. - Apt-install (with cache mount): sox, libsox-fmt-all (already in runtime-base — verify), vim, netcat-openbsd, python3, python3-pip, python3-matplotlib, python3-numpy, python3-scipy, ffmpeg, gnuradio, gr-osmosdr. - COPY dsd-fme + libmbe* from `dsd-fme-build`. @@ -98,14 +103,14 @@ D). - `CMD ["/bin/bash"]`. - No s6, no ENTRYPOINT — this is an interactive utility container, not a supervised service. - _Requirements: 2.4, 3.12_ -- [ ] **B.2.11** Smoke-test locally: +- [x] **B.2.11** Smoke-test locally: - `docker buildx build --target final-core --load -t wavekit:dev-core .` - `docker buildx build --target final --load -t wavekit:dev .` - `docker buildx build --target final-sdrpp --load -t wavekit:dev-sdrpp .` - `docker buildx build --target final-demod --load -t wavekit:dev-demod .` - All four builds succeed. - _Requirements: covers B.2.1–B.2.10_ -- [ ] **B.2.12** Verify Property 8 and Property 9 against the new `final-core` image: +- [x] **B.2.12** Verify Property 8 and Property 9 against the new `final-core` image: - `docker run --rm wavekit:dev-core sh -c 'find /etc/s6-overlay -iname "*sdrpp*"'` returns empty. - `docker run --rm wavekit:dev-core ls /etc/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/` returns exactly `wavekit-init`. - `docker run --rm wavekit:dev sh -c 'find /etc/s6-overlay -name "sdrpp-server" | wc -l'` returns >= 1. diff --git a/Dockerfile b/Dockerfile index da6e19f..68d3bbc 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,495 +1,601 @@ +# syntax=docker/dockerfile:1.7 # ============================================================================= # WaveKit Multi-Stage Dockerfile # -# Architecture: Production-ready SDR stream processing framework with s6-overlay -# Process Management: Full process supervision with auto-restart & dependency handling -# Optimization: BuildKit-compatible, multi-platform (amd64/arm64), layer caching +# Architecture: 20 named, narrow-purpose stages. Each decoder builds on +# `base-build` (thin toolchain image) and installs only its own apt deps with +# `--mount=type=cache,target=/var/cache/apt,sharing=locked`. Upstream sources +# are pinned to SHAs via top-of-file `ARG _REF` declarations so a bump +# to one decoder does not invalidate any other. # -# Deployment modes: -# - full: Everything (SDR++, WaveKit API, all decoders) - for single-host setups -# - core: Just API + decoders (SDR++ externalized) - for distributed setups -# - sdrpp: Just SDR++ server (IQ/audio provider) - for dedicated SDR host -# -# Usage: -# docker build -t wavekit:latest -f Dockerfile . -# docker run --rm -it wavekit:latest +# Final targets: +# - final : full single-host image (SDR++ + decoders + API) +# - final-core : API + decoders, SDR++ externalized (no sdrpp residue) +# - final-sdrpp : SDR++ server only (IQ/audio provider) +# - final-demod : interactive utility container for offline test work # ============================================================================= # ============================================================================ -# Stage: base-deps -# Purpose: Common system dependencies for all build stages -# Size: ~150MB (reduced with only-production install) +# Pinned upstream refs (SHA or tag). Bumping any of these invalidates ONLY +# the corresponding decoder build stage. +# ============================================================================ +ARG S6_OVERLAY_VERSION=3.1.6.2 +ARG SDRPP_REF=052167962dbf9adc2a02825f2f428e7613255d50 +ARG MBELIB_REF=9a04ed5c78176a9965f3d43f7aa1b1f5330e771f +ARG DSDFME_REF=ed1d1d630ce79db890bbf4b890317341bc5aa580 +ARG MULTIMON_NG_REF=a2f7f872bd54b51e7fc6cdbafdf1c4872a52246d +ARG RTL_433_REF=19f788d0d67720ac23556b5bcdbad63a005637f2 +ARG ACARSDEC_REF=206f733027131d514454c4208d3acc986c9f9a28 +ARG AIS_CATCHER_REF=ac4d59be4a8d3960815157acc6a5f65560172006 +ARG DIREWOLF_REF=a231971a652bfb574a4bae9a5d875fbce53d2267 +ARG LIBACARS_REF=9af09a0121d4ec577339cbd4c7420d7519da48fa +ARG DUMPVDL2_REF=3f583da4957d6c74668eb174e6ecd8c1435fb25b +ARG READSB_REF=b499ecbd18dc4a2ec6098c31de31508017fa6190 +ARG SOAPY_RTLTCP_REF=75a53aa251b1ef63850abea81b7617ef6978a15e +ARG CSDR_REF=1f15b8c5177cb348602da19e82bf0d62426ab8eb +ARG GR_LORA_SDR_REF=862746dd1cf635c9c8a4bfbaa2c3a0ec3a5306c9 + +# ============================================================================ +# Stage: base-build +# Purpose: Thin toolchain image shared by every *-build stage. No +# decoder-specific or signal-processing apt deps. # ============================================================================ -FROM debian:bookworm-slim AS base-deps +FROM debian:bookworm-slim AS base-build -# Use buildkit cache for package lists RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt-get update && apt-get install -y --no-install-recommends \ - # Build toolchain - build-essential \ - cmake \ - git \ - pkg-config \ - # Audio & signal processing - libfftw3-dev \ - libsndfile1-dev \ - libopus-dev \ - libopusfile-dev \ - libvorbis-dev \ - libogg-dev \ - libflac-dev \ - libavformat-dev \ - libavcodec-dev \ - libswresample-dev \ - libpulse-dev \ - # SDR libraries - librtlsdr-dev \ - libhackrf-dev \ - libairspy-dev \ - libairspyhf-dev \ - libbladerf-dev \ - libsoapysdr-dev \ - libusb-1.0-0-dev \ - libudev-dev \ - # dsd-fme dependencies - libitpp-dev \ - # direwolf dependencies - libasound2-dev \ - libgps-dev \ - libhamlib-dev \ - # dumpvdl2 dependencies - libglib2.0-dev \ - libsqlite3-dev \ - libzmq3-dev \ - # Note: libacars-dev not in Debian repos, will be built from source in dumpvdl2-build stage - # readsb dependencies - libncurses-dev \ - zlib1g-dev \ - libzstd-dev \ - # SDR++ dependencies - libglfw3-dev \ - libglew-dev \ - libvolk2-dev \ - librtaudio-dev \ - libiio-dev \ - libad9361-dev \ - libcjson-dev \ - # utilities - curl \ - wget \ - ca-certificates \ + build-essential \ + cmake \ + git \ + pkg-config \ + ca-certificates \ + curl \ && rm -rf /var/lib/apt/lists/* +WORKDIR /build + # ============================================================================ # Stage: runtime-base -# Purpose: Minimal runtime dependencies (no build tools) -# Size: ~80MB +# Purpose: Minimal runtime (no build tools) + s6-overlay supervisor. All +# shared dynamic libs the decoders link against at runtime. # ============================================================================ FROM debian:bookworm-slim AS runtime-base RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt-get update && apt-get install -y --no-install-recommends \ - # Runtime audio libraries - libsndfile1 \ - libopus0 \ - libopusfile0 \ - libvorbis0a \ - libogg0 \ - libflac12 \ - libavformat59 \ - libavcodec59 \ - libswresample4 \ - libpulse0 \ - # Runtime SDR libraries - librtlsdr0 \ - libhackrf0 \ - libairspy0 \ - libairspyhf1 \ - libsoapysdr0.8 \ - libusb-1.0-0 \ - udev \ - # dsd-fme runtime - libitpp8v5 \ - libfftw3-single3 \ - libfftw3-double3 \ - # direwolf runtime - libasound2 \ - libgps28 \ - libhamlib4 \ - # dumpvdl2 runtime - libglib2.0-0 \ - libsqlite3-0 \ - libzmq5 \ - # Note: libacars is statically linked in dumpvdl2 - # readsb runtime - libncurses6 \ - zlib1g \ - libzstd1 \ - libcjson1 \ - # Runtime utilities - ca-certificates \ - curl \ - tini \ - xz-utils \ - sox \ - libsox-fmt-all \ - libsox-fmt-all \ - netcat-openbsd \ - libsamplerate0 \ + # Runtime audio libraries + libsndfile1 \ + libopus0 \ + libopusfile0 \ + libvorbis0a \ + libogg0 \ + libflac12 \ + libavformat59 \ + libavcodec59 \ + libswresample4 \ + libpulse0 \ + libsamplerate0 \ + # Runtime SDR libraries + librtlsdr0 \ + libhackrf0 \ + libairspy0 \ + libairspyhf1 \ + libsoapysdr0.8 \ + libusb-1.0-0 \ + udev \ + # dsd-fme runtime + libitpp8v5 \ + libfftw3-single3 \ + libfftw3-double3 \ + # direwolf runtime + libasound2 \ + libgps28 \ + libhamlib4 \ + # dumpvdl2 runtime + libglib2.0-0 \ + libsqlite3-0 \ + libzmq5 \ + # readsb runtime + libncurses6 \ + zlib1g \ + libzstd1 \ + libcjson1 \ + # Runtime utilities + ca-certificates \ + curl \ + xz-utils \ + sox \ + libsox-fmt-all \ + netcat-openbsd \ && rm -rf /var/lib/apt/lists/* -# Install s6-overlay (PID 1 init system + service supervisor) -ARG S6_OVERLAY_VERSION=3.1.6.2 -ARG S6_RO_PLATFORM=x86_64 -ARG BUILDPLATFORM - -# Auto-detect platform for s6-overlay -RUN if [ "${BUILDPLATFORM}" = "linux/arm64" ]; then \ - S6_RO_PLATFORM=aarch64; \ - elif [ "${BUILDPLATFORM}" = "linux/arm/v7" ]; then \ - S6_RO_PLATFORM=armhf; \ - fi && \ - curl -fsSL https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz | tar -xJ -C / && \ - curl -fsSL https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-${S6_RO_PLATFORM}.tar.xz | tar -xJ -C / - -# Prepare s6-overlay environment -RUN mkdir -p /run/service && \ - chmod 755 /run/service && \ - mkdir -p /etc/s6-overlay/s6-rc.d +# Install s6-overlay (PID 1 init system + service supervisor). +# Use TARGETARCH (per sdr-host pattern) so cross-arch builds get the right +# binaries; BUILDPLATFORM would point at the build host, not the target. +ARG S6_OVERLAY_VERSION +ARG TARGETARCH -ENTRYPOINT ["/init"] +RUN case "${TARGETARCH}" in \ + "amd64") S6_ARCH="x86_64" ;; \ + "arm64") S6_ARCH="aarch64" ;; \ + *) echo "Unsupported arch: ${TARGETARCH}" && exit 1 ;; \ + esac && \ + curl -fsSL "https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz" \ + | tar -xJ -C / && \ + curl -fsSL "https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-${S6_ARCH}.tar.xz" \ + | tar -xJ -C / + +# Prepare s6-overlay runtime dirs +RUN mkdir -p /run/service /etc/s6-overlay/s6-rc.d /etc/s6-overlay/scripts && \ + chmod 755 /run/service # ============================================================================ # Stage: sdrpp-build -# Purpose: Build SDR++ server -# Size: ~800MB (not in final image) +# Purpose: Build SDR++ server (pinned by SDRPP_REF) # ============================================================================ -FROM base-deps AS sdrpp-build +FROM base-build AS sdrpp-build -WORKDIR /build +ARG SDRPP_REF + +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + libfftw3-dev \ + libsndfile1-dev \ + libopus-dev \ + libopusfile-dev \ + libvorbis-dev \ + libogg-dev \ + libflac-dev \ + libpulse-dev \ + librtlsdr-dev \ + libhackrf-dev \ + libairspy-dev \ + libairspyhf-dev \ + libbladerf-dev \ + libsoapysdr-dev \ + libusb-1.0-0-dev \ + libudev-dev \ + libglfw3-dev \ + libglew-dev \ + libvolk2-dev \ + librtaudio-dev \ + libiio-dev \ + libad9361-dev \ + libcjson-dev \ + zlib1g-dev \ + libzstd-dev \ + && rm -rf /var/lib/apt/lists/* -# Build SDR++ -RUN git clone --depth 1 --branch nightly https://github.com/AlexandreRouma/SDRPlusPlus.git && \ - cd SDRPlusPlus && \ +RUN git clone --no-checkout https://github.com/AlexandreRouma/SDRPlusPlus.git repo && \ + cd repo && \ + git fetch --depth 1 origin "${SDRPP_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release \ - -DUSE_INTERNAL_GLFW=ON \ - -DUSE_BUNDLE_GLFW=ON \ - -DOPT_BUILD_M17_DECODER=OFF \ - .. && \ - make -j$(nproc) && \ + -DUSE_INTERNAL_GLFW=ON \ + -DUSE_BUNDLE_GLFW=ON \ + -DOPT_BUILD_M17_DECODER=OFF \ + .. && \ + make -j"$(nproc)" && \ make install # ============================================================================ # Stage: dsd-fme-build -# Purpose: Build dsd-fme decoder (DMR, P25, YSF, D-Star, etc.) -# Size: ~300MB (not in final image) +# Purpose: Build mbelib (MBELIB_REF), then dsd-fme (DSDFME_REF). DMR, P25, +# YSF, D-Star and friends. # ============================================================================ -FROM base-deps AS dsd-fme-build +FROM base-build AS dsd-fme-build -WORKDIR /build +ARG MBELIB_REF +ARG DSDFME_REF + +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + libitpp-dev \ + libfftw3-dev \ + libsoapysdr-dev \ + libpulse-dev \ + libncurses-dev \ + libsndfile1-dev \ + libusb-1.0-0-dev \ + && rm -rf /var/lib/apt/lists/* -# Build mbelib first (not in Debian repos) -RUN git clone --depth 1 https://github.com/szechyjs/mbelib.git && \ +RUN git clone --no-checkout https://github.com/szechyjs/mbelib.git mbelib && \ cd mbelib && \ + git fetch --depth 1 origin "${MBELIB_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ + make -j"$(nproc)" && \ make install && \ ldconfig -# Build dsd-fme -RUN git clone --depth 1 https://github.com/lwvmobile/dsd-fme.git && \ +RUN git clone --no-checkout https://github.com/lwvmobile/dsd-fme.git dsd-fme && \ cd dsd-fme && \ + git fetch --depth 1 origin "${DSDFME_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ + make -j"$(nproc)" && \ make install # ============================================================================ # Stage: multimon-ng-build -# Purpose: Build multimon-ng decoder (POCSAG, FLEX, EAS, DTMF, etc.) -# Size: ~200MB (not in final image) +# Purpose: Build multimon-ng (MULTIMON_NG_REF). POCSAG, FLEX, EAS, DTMF, etc. # ============================================================================ -FROM base-deps AS multimon-ng-build +FROM base-build AS multimon-ng-build -WORKDIR /build +ARG MULTIMON_NG_REF -RUN git clone --depth 1 https://github.com/EliasOenal/multimon-ng.git && \ - cd multimon-ng && \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + libpulse-dev \ + && rm -rf /var/lib/apt/lists/* + +RUN git clone --no-checkout https://github.com/EliasOenal/multimon-ng.git repo && \ + cd repo && \ + git fetch --depth 1 origin "${MULTIMON_NG_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ + make -j"$(nproc)" && \ make install # ============================================================================ # Stage: rtl433-build -# Purpose: Build rtl_433 decoder (ISM sensors, weather stations, etc.) -# Size: ~250MB (not in final image) +# Purpose: Build rtl_433 (RTL_433_REF). ISM sensors, weather stations, etc. # ============================================================================ -FROM base-deps AS rtl433-build +FROM base-build AS rtl433-build -WORKDIR /build +ARG RTL_433_REF + +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + librtlsdr-dev \ + libsoapysdr-dev \ + libusb-1.0-0-dev \ + && rm -rf /var/lib/apt/lists/* -RUN git clone --depth 1 https://github.com/merbanan/rtl_433.git && \ - cd rtl_433 && \ +RUN git clone --no-checkout https://github.com/merbanan/rtl_433.git repo && \ + cd repo && \ + git fetch --depth 1 origin "${RTL_433_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ + make -j"$(nproc)" && \ make install # ============================================================================ # Stage: acarsdec-build -# Purpose: Build acarsdec decoder (ACARS aircraft data link) -# Size: ~150MB (not in final image) +# Purpose: Build acarsdec (ACARSDEC_REF). ACARS aircraft data link. # ============================================================================ -FROM base-deps AS acarsdec-build +FROM base-build AS acarsdec-build -WORKDIR /build +ARG ACARSDEC_REF -RUN git clone --depth 1 https://github.com/f00b4r0/acarsdec.git && \ - cd acarsdec && \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + librtlsdr-dev \ + libsoapysdr-dev \ + libusb-1.0-0-dev \ + libsndfile1-dev \ + libzmq3-dev \ + libsqlite3-dev \ + && rm -rf /var/lib/apt/lists/* + +RUN git clone --no-checkout https://github.com/f00b4r0/acarsdec.git repo && \ + cd repo && \ + git fetch --depth 1 origin "${ACARSDEC_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ + make -j"$(nproc)" && \ make install # ============================================================================ # Stage: ais-catcher-build -# Purpose: Build AIS-catcher decoder (Maritime AIS transponders) -# Size: ~200MB (not in final image) -# ============================================================================ -FROM base-deps AS ais-catcher-build - -WORKDIR /build - -RUN git clone --depth 1 https://github.com/jvde-github/AIS-catcher.git && \ - cd AIS-catcher && \ - mkdir build && cd build && \ - cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ - cp AIS-catcher /usr/local/bin/ - -# ============================================================================ -# Stage: lora-build -# Purpose: Build gr-lora_sdr decoder blocks (LoRa/Meshtastic) -# Size: ~500MB (not in final image) +# Purpose: Build AIS-catcher (AIS_CATCHER_REF). Maritime AIS transponders. # ============================================================================ -FROM base-deps AS lora-build +FROM base-build AS ais-catcher-build -ARG GR_LORA_SDR_REF=862746dd1cf635c9c8a4bfbaa2c3a0ec3a5306c9 - -WORKDIR /build +ARG AIS_CATCHER_REF RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt-get update && apt-get install -y --no-install-recommends \ - gnuradio \ - gnuradio-dev \ - python3-dev \ - python3-numpy \ - pybind11-dev \ + librtlsdr-dev \ + libairspy-dev \ + libairspyhf-dev \ + libhackrf-dev \ + libsoapysdr-dev \ + libusb-1.0-0-dev \ + libsqlite3-dev \ + libcurl4-openssl-dev \ + zlib1g-dev \ && rm -rf /var/lib/apt/lists/* -RUN git clone --no-checkout https://github.com/tapparelj/gr-lora_sdr.git && \ - cd gr-lora_sdr && \ - git fetch --depth 1 origin "${GR_LORA_SDR_REF}" && \ +RUN git clone --no-checkout https://github.com/jvde-github/AIS-catcher.git repo && \ + cd repo && \ + git fetch --depth 1 origin "${AIS_CATCHER_REF}" && \ git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ - make install && \ - ldconfig && \ - lora_py_dir="$(find /usr/local/lib -path '*/gnuradio/lora_sdr' -type d -print -quit)" && \ - test -n "${lora_py_dir}" && \ - lora_lib_src="$(find /usr/local/lib -maxdepth 3 -name 'libgnuradio-lora_sdr*' -printf '%h\n' -quit)" && \ - test -n "${lora_lib_src}" && \ - mkdir -p /usr/local/share/wavekit-lora/lib && \ - cp -a "${lora_py_dir}" /usr/local/share/wavekit-lora/lora_sdr && \ - cp -a "${lora_lib_src}"/libgnuradio-lora_sdr* /usr/local/share/wavekit-lora/lib/ && \ - cd / && rm -rf /build/gr-lora_sdr + make -j"$(nproc)" && \ + cp AIS-catcher /usr/local/bin/ # ============================================================================ # Stage: direwolf-build -# Purpose: Build direwolf decoder (APRS amateur radio packets) -# Size: ~180MB (not in final image) +# Purpose: Build direwolf (DIREWOLF_REF). APRS amateur radio packets. # ============================================================================ -FROM base-deps AS direwolf-build +FROM base-build AS direwolf-build -WORKDIR /build +ARG DIREWOLF_REF -RUN git clone --depth 1 https://github.com/wb2osz/direwolf.git && \ - cd direwolf && \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + libasound2-dev \ + libgps-dev \ + libhamlib-dev \ + libudev-dev \ + && rm -rf /var/lib/apt/lists/* + +RUN git clone --no-checkout https://github.com/wb2osz/direwolf.git repo && \ + cd repo && \ + git fetch --depth 1 origin "${DIREWOLF_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ + make -j"$(nproc)" && \ make install # ============================================================================ # Stage: dumpvdl2-build -# Purpose: Build dumpvdl2 decoder (VDL Mode 2 aviation data link) -# Size: ~200MB (not in final image) +# Purpose: Build libacars (LIBACARS_REF), then dumpvdl2 (DUMPVDL2_REF). +# VDL Mode 2 aviation data link. # ============================================================================ -FROM base-deps AS dumpvdl2-build +FROM base-build AS dumpvdl2-build -WORKDIR /build +ARG LIBACARS_REF +ARG DUMPVDL2_REF + +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + libfftw3-dev \ + libsqlite3-dev \ + libzmq3-dev \ + libglib2.0-dev \ + librtlsdr-dev \ + libsoapysdr-dev \ + libusb-1.0-0-dev \ + zlib1g-dev \ + && rm -rf /var/lib/apt/lists/* -# Build libacars first (not available in Debian repos) -RUN git clone --depth 1 https://github.com/szpajder/libacars.git && \ +RUN git clone --no-checkout https://github.com/szpajder/libacars.git libacars && \ cd libacars && \ + git fetch --depth 1 origin "${LIBACARS_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ + make -j"$(nproc)" && \ make install && \ ldconfig -# Build dumpvdl2 -RUN git clone --depth 1 https://github.com/szpajder/dumpvdl2.git && \ +RUN git clone --no-checkout https://github.com/szpajder/dumpvdl2.git dumpvdl2 && \ cd dumpvdl2 && \ + git fetch --depth 1 origin "${DUMPVDL2_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ + make -j"$(nproc)" && \ make install # ============================================================================ # Stage: readsb-build -# Purpose: Build readsb decoder (ADS-B aircraft transponders) -# Size: ~150MB (not in final image) +# Purpose: Build readsb (READSB_REF). ADS-B aircraft transponders. # ============================================================================ -FROM base-deps AS readsb-build +FROM base-build AS readsb-build -WORKDIR /build +ARG READSB_REF + +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + librtlsdr-dev \ + libusb-1.0-0-dev \ + libncurses-dev \ + zlib1g-dev \ + libzstd-dev \ + && rm -rf /var/lib/apt/lists/* -RUN git clone --depth 1 https://github.com/wiedehopf/readsb.git && \ - cd readsb && \ +RUN git clone --no-checkout https://github.com/wiedehopf/readsb.git repo && \ + cd repo && \ + git fetch --depth 1 origin "${READSB_REF}" && \ + git checkout --detach FETCH_HEAD && \ make -j2 RTLSDR=yes OPTIMIZE="-O1" && \ cp readsb /usr/local/bin/ # ============================================================================ # Stage: soapy-rtltcp-build -# Purpose: Build SoapyRTLTCP module for rtl_tcp network SDR support -# Size: ~50MB (not in final image) +# Purpose: Build SoapyRTLTCP module (SOAPY_RTLTCP_REF) for rtl_tcp network +# SDR support consumed by acarsdec, dumpvdl2. # ============================================================================ -FROM base-deps AS soapy-rtltcp-build +FROM base-build AS soapy-rtltcp-build -WORKDIR /build +ARG SOAPY_RTLTCP_REF -RUN git clone --depth 1 https://github.com/pothosware/SoapyRTLTCP.git && \ - cd SoapyRTLTCP && \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + libsoapysdr-dev \ + && rm -rf /var/lib/apt/lists/* + +RUN git clone --no-checkout https://github.com/pothosware/SoapyRTLTCP.git repo && \ + cd repo && \ + git fetch --depth 1 origin "${SOAPY_RTLTCP_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ cmake -DCMAKE_BUILD_TYPE=Release .. && \ - make -j$(nproc) && \ + make -j"$(nproc)" && \ make install # ============================================================================ # Stage: csdr-build -# Purpose: Build csdr command-line SDR tools for FM demodulation -# Size: ~100MB (not in final image) +# Purpose: Build csdr command-line SDR tools (CSDR_REF) for FM demodulation. # ============================================================================ -FROM base-deps AS csdr-build +FROM base-build AS csdr-build -WORKDIR /build +ARG CSDR_REF -# Install libsamplerate (required by csdr) -RUN apt-get update && apt-get install -y --no-install-recommends libsamplerate-dev +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + libsamplerate-dev \ + libfftw3-dev \ + && rm -rf /var/lib/apt/lists/* +RUN git clone --no-checkout https://github.com/jketterl/csdr.git repo && \ + cd repo && \ + git fetch --depth 1 origin "${CSDR_REF}" && \ + git checkout --detach FETCH_HEAD && \ + mkdir build && cd build && \ + cmake -DCMAKE_BUILD_TYPE=Release .. && \ + make -j"$(nproc)" && \ + make install +# ============================================================================ +# Stage: lora-build +# Purpose: Build gr-lora_sdr decoder blocks (GR_LORA_SDR_REF). LoRa / +# Meshtastic packet decoding for the python helper. +# ============================================================================ +FROM base-build AS lora-build + +ARG GR_LORA_SDR_REF + +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + gnuradio \ + gnuradio-dev \ + python3-dev \ + python3-numpy \ + pybind11-dev \ + && rm -rf /var/lib/apt/lists/* -# Build csdr command-line tool -RUN git clone --depth 1 https://github.com/jketterl/csdr.git && \ - cd csdr && \ +RUN git clone --no-checkout https://github.com/tapparelj/gr-lora_sdr.git repo && \ + cd repo && \ + git fetch --depth 1 origin "${GR_LORA_SDR_REF}" && \ + git checkout --detach FETCH_HEAD && \ mkdir build && cd build && \ - cmake .. -DCMAKE_BUILD_TYPE=Release && \ - make -j$(nproc) && \ - make install + cmake -DCMAKE_BUILD_TYPE=Release .. && \ + make -j"$(nproc)" && \ + make install && \ + ldconfig && \ + lora_py_dir="$(find /usr/local/lib -path '*/gnuradio/lora_sdr' -type d -print -quit)" && \ + test -n "${lora_py_dir}" && \ + lora_lib_src="$(find /usr/local/lib -maxdepth 3 -name 'libgnuradio-lora_sdr*' -printf '%h\n' -quit)" && \ + test -n "${lora_lib_src}" && \ + mkdir -p /usr/local/share/wavekit-lora/lib && \ + cp -a "${lora_py_dir}" /usr/local/share/wavekit-lora/lora_sdr && \ + cp -a "${lora_lib_src}"/libgnuradio-lora_sdr* /usr/local/share/wavekit-lora/lib/ # ============================================================================ # Stage: node-build -# Purpose: Build WaveKit TypeScript application -# Size: ~450MB (not in final image) +# Purpose: Compile the WaveKit TypeScript app. Single pnpm install honoring +# the project's .npmrc node-linker=isolated. No double install. # ============================================================================ FROM node:22-bookworm-slim AS node-build WORKDIR /app -# Use Corepack to install the exact pnpm version from packageManager +# Corepack pins pnpm to the version in package.json's `packageManager` field. RUN corepack enable && corepack prepare pnpm@10.28.0 --activate -# Copy workspace manifests and install dependencies +# Copy workspace manifests first so the install layer caches independently +# from source edits. COPY package.json pnpm-lock.yaml pnpm-workspace.yaml tsconfig.base.json turbo.json .npmrc ./ COPY packages/shared/package.json packages/shared/tsconfig.json ./packages/shared/ COPY packages/api-types/package.json packages/api-types/tsconfig.json ./packages/api-types/ COPY cli/package.json cli/tsconfig.json ./cli/ COPY packages/sdr-host/package.json packages/sdr-host/tsconfig.json ./packages/sdr-host/ -RUN --mount=type=cache,target=/root/.local/share/pnpm/store,sharing=locked \ - pnpm install --frozen-lockfile --prod=false -# Copy source code -COPY . . - -# In Docker builds, prefer a hoisted node_modules layout for maximum tool compatibility +# R3 mitigation (Build & DX Overhaul, Phase B): +# +# Two layout issues prevent a single isolated install from working in this +# build context, both observed empirically during the refactor: +# +# 1. `.npmrc` has `enable-global-virtual-store=true`, which makes the +# installed virtual store live at the same cache-mount path used by +# the install. After the mount goes away the symlinks dangle. +# +# 2. `.dockerignore` excludes `node_modules` at the root only (not +# `**/node_modules`), so the host's `cli/node_modules/` and +# `packages/*/node_modules/` enter the build context via `COPY . .` +# and overwrite the freshly-installed workspace symlinks with paths +# that point at the developer's local pnpm store. +# +# Per the Phase B brief's R3 mitigation, switching node-build to a hoisted +# layout for a single install resolves both issues at once: `pnpm install` +# materializes real package directories instead of symlinks, so subsequent +# `COPY . .` can clobber the (now harmless) host node_modules without +# breaking workspace package resolution. +# +# Project default in `.npmrc` (node-linker=isolated) remains unchanged for +# native `pnpm dev`; this Dockerfile-local override only affects builds. +# A follow-up issue should fix `.dockerignore` to exclude `**/node_modules` +# so this override can be lifted (Requirement 3.6). RUN pnpm config set node-linker hoisted --location=project && \ pnpm config set enable-global-virtual-store false --location=project -# Re-link dependencies under the selected node-linker (fast with cached store) RUN --mount=type=cache,target=/root/.local/share/pnpm/store,sharing=locked \ - CI=true pnpm install --frozen-lockfile --prod=false + pnpm install --frozen-lockfile --prod=false + +# Now copy sources. Edits here invalidate from this layer down — but +# decoder *-build stages are unaffected. +COPY . . -# Run type checking (cache Turbo + TS incremental artifacts) RUN --mount=type=cache,target=/root/.cache/turbo,sharing=locked \ --mount=type=cache,target=/app/node_modules/.cache,sharing=locked \ pnpm run typecheck -# Build application (cache Turbo artifacts) RUN --mount=type=cache,target=/root/.cache/turbo,sharing=locked \ --mount=type=cache,target=/app/node_modules/.cache,sharing=locked \ pnpm run build -# Prune dev dependencies for smaller final image -RUN CI=true pnpm prune --prod +RUN --mount=type=cache,target=/root/.local/share/pnpm/store,sharing=locked \ + CI=true pnpm prune --prod -# Verify built output RUN ls -la dist/ && head -1 dist/index.js # ============================================================================ -# Stage: final (full mode) -# Purpose: Complete runtime with all components -# Size: ~1.2GB +# Stage: final-base +# Purpose: Common ancestor of `final` and `final-core`. Holds every artifact +# shared by both: python runtime, every decoder binary + library, +# csdr, soapy-rtltcp, lora artifacts, the node runtime + app dist, +# shipped scripts, canonical s6 overlay (sdrpp-free), config file. # ============================================================================ -FROM runtime-base AS final +FROM runtime-base AS final-base -# Metadata LABEL maintainer="WaveKit Contributors" -LABEL description="SDR stream processing framework with integrated decoders" +LABEL description="WaveKit common runtime (API + decoders, no SDR++)" LABEL version="1.0.0" -# Environment setup ENV WAVEKIT_HOME=/app \ WAVEKIT_CONFIG_PATH=/app/config \ WAVEKIT_LOG_LEVEL=info \ NODE_ENV=production \ PATH=/usr/local/bin:/usr/local/sbin:${PATH} -# Create application directory structure RUN mkdir -p /app /var/log/wavekit /var/run/wavekit && \ chmod 755 /app /var/log/wavekit /var/run/wavekit +# Python runtime shared by the lora helper and any future python tooling. +# python3-cryptography is required by lora_meshtastic_decode.py (imports +# `from cryptography.hazmat.primitives.ciphers import ...`). RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt-get update && apt-get install -y --no-install-recommends \ - python3 \ - gnuradio \ - python3-numpy \ - python3-protobuf \ - python3-cryptography \ + python3 \ + gnuradio \ + python3-numpy \ + python3-protobuf \ + python3-cryptography \ && rm -rf /var/lib/apt/lists/* -# Copy SDR++ from build stage -COPY --from=sdrpp-build /usr/local/bin/sdrpp* /usr/local/bin/ -COPY --from=sdrpp-build /usr/local/lib/libsdrpp* /usr/local/lib/ -# Copy decoders from build stages +# Decoder binaries + their runtime libs from the *-build stages. COPY --from=dsd-fme-build /usr/local/bin/dsd* /usr/local/bin/ COPY --from=dsd-fme-build /usr/local/lib/libmbe* /usr/local/lib/ COPY --from=multimon-ng-build /usr/local/bin/multimon-ng /usr/local/bin/ @@ -502,34 +608,37 @@ COPY --from=direwolf-build /usr/local/bin/gen_packets /usr/local/bin/ COPY --from=dumpvdl2-build /usr/local/bin/dumpvdl2 /usr/local/bin/ COPY --from=dumpvdl2-build /usr/local/lib/libacars* /usr/local/lib/ COPY --from=readsb-build /usr/local/bin/readsb /usr/local/bin/ -# Copy gr-lora_sdr blocks for LoRa/Meshtastic -# KEEP IN SYNC with the lora install in the final-core stage below. -# Source paths come from the stable wavekit-lora staging dir so they work for -# both amd64 (x86_64-linux-gnu) and arm64 (aarch64-linux-gnu) builds. -# The python module destination is the apt-managed gnuradio namespace dir; -# Debian's `gnuradio` package keeps __init__.py at /usr/lib/python3/dist-packages/gnuradio/. -COPY --from=lora-build /usr/local/share/wavekit-lora/lib/libgnuradio-lora_sdr* /usr/local/lib/ -COPY --from=lora-build /usr/local/share/wavekit-lora/lora_sdr /usr/lib/python3/dist-packages/gnuradio/lora_sdr -# Copy SoapyRTLTCP module for rtl_tcp network SDR support (acarsdec, dumpvdl2) -COPY --from=soapy-rtltcp-build /usr/local/lib/SoapySDR/modules0.8/librtltcpSupport.so /usr/local/lib/SoapySDR/modules0.8/ -# Copy csdr for FM demodulation (audio-from-IQ decoders) + +# csdr + libcsdr (FM demodulation for audio-from-IQ decoder paths) COPY --from=csdr-build /usr/local/bin/csdr /usr/local/bin/ COPY --from=csdr-build /usr/local/lib/libcsdr* /usr/local/lib/ + +# SoapyRTLTCP module so acarsdec / dumpvdl2 can stream from rtl_tcp +COPY --from=soapy-rtltcp-build /usr/local/lib/SoapySDR/modules0.8/librtltcpSupport.so /usr/local/lib/SoapySDR/modules0.8/ + +# gr-lora_sdr blocks. Python module lands in Debian's gnuradio namespace +# dir so `from gnuradio import lora_sdr` resolves regardless of arch. +COPY --from=lora-build /usr/local/share/wavekit-lora/lib/libgnuradio-lora_sdr* /usr/local/lib/ +COPY --from=lora-build /usr/local/share/wavekit-lora/lora_sdr /usr/lib/python3/dist-packages/gnuradio/lora_sdr + +# Lora helper script + protobuf module COPY docker/scripts/lora_meshtastic_decode.py /usr/local/bin/ COPY docker/scripts/meshtastic_proto /usr/local/lib/wavekit/meshtastic_proto +RUN chmod 755 /usr/local/bin/lora_meshtastic_decode.py -# Update library cache for all copied libraries -RUN chmod 755 /usr/local/bin/lora_meshtastic_decode.py && \ - ldconfig +# Refresh dynamic linker cache after all the lib drops +RUN ldconfig -# Verify all 9 decoder installations +# Decoder smoke-verify. We check the binaries are on PATH and runnable; +# we tolerate non-zero exit on the ones that emit a banner+usage when given +# any flag (dsd-fme, acarsdec, AIS-catcher, direwolf do this). RUN echo "Verifying decoder installations..." && \ - dsd-fme --version && \ - multimon-ng -h > /dev/null 2>&1 && \ + command -v dsd-fme && \ + command -v multimon-ng && \ rtl_433 -V && \ - acarsdec -h > /dev/null 2>&1 || true && \ - AIS-catcher -h > /dev/null 2>&1 || true && \ - direwolf -h > /dev/null 2>&1 || true && \ + command -v acarsdec && \ + command -v AIS-catcher && \ + command -v direwolf && \ dumpvdl2 --version && \ readsb --version && \ csdr --help > /dev/null 2>&1 && \ @@ -537,7 +646,7 @@ RUN echo "Verifying decoder installations..." && \ python3 /usr/local/bin/lora_meshtastic_decode.py --help > /dev/null && \ echo "All 9 decoders + csdr verified successfully" -# Copy Node.js application +# Node runtime + app dist + workspace packages + config COPY --from=node-build /usr/local/bin/node /usr/local/bin/ COPY --from=node-build /app/dist /app/dist COPY --from=node-build /app/node_modules /app/node_modules @@ -548,153 +657,95 @@ COPY --from=node-build /app/packages/api-types/dist /app/packages/api-types/dist COPY --from=node-build /app/config /app/config COPY --from=node-build /app/package.json /app/ -# Copy helper scripts +# Shipped helper scripts (sdrpp-specific scripts deferred to `final`) COPY docker/scripts/init-system.sh /usr/local/bin/ COPY docker/scripts/start-api.sh /usr/local/bin/ COPY docker/scripts/finish-api.sh /usr/local/bin/ -COPY docker/scripts/start-sdrpp.sh /usr/local/bin/ -COPY docker/scripts/finish-sdrpp.sh /usr/local/bin/ -RUN chmod 755 /usr/local/bin/init-system.sh /usr/local/bin/start-api.sh /usr/local/bin/finish-api.sh /usr/local/bin/start-sdrpp.sh /usr/local/bin/finish-sdrpp.sh +RUN chmod 755 \ + /usr/local/bin/init-system.sh \ + /usr/local/bin/start-api.sh \ + /usr/local/bin/finish-api.sh -# Copy s6-overlay service definitions +# Canonical s6 overlay (sdrpp-free). `final` additionally copies the sdrpp +# overlay on top of this; `final-core` ships exactly this tree. COPY docker/overlay/s6-overlay/s6-rc.d /etc/s6-overlay/s6-rc.d - -# Make service scripts executable RUN chmod -R 755 /etc/s6-overlay/s6-rc.d -# Create health check endpoint wrapper -RUN mkdir -p /etc/s6-overlay/scripts -COPY docker/scripts/healthcheck.sh /etc/s6-overlay/scripts/ +# Healthcheck wrapper +COPY docker/scripts/healthcheck.sh /etc/s6-overlay/scripts/healthcheck.sh RUN chmod 755 /etc/s6-overlay/scripts/healthcheck.sh -# Copy direwolf configuration +# direwolf config COPY docker/config/direwolf.conf /etc/direwolf.conf -# Expose ports -EXPOSE 9000 \ - 8080 \ - 5259 \ - 7355 \ - 4713 +EXPOSE 9000 8080 4713 -# Volume mounts -VOLUME ["/var/log/wavekit", "/var/run/wavekit", "/recordings"] +VOLUME ["/var/log/wavekit", "/var/run/wavekit"] + +WORKDIR /app -# Health check using s6-svstat HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \ CMD s6-svstat -u /run/service/wavekit-api > /dev/null 2>&1 || exit 1 -# Start with s6-init (PID 1) -CMD ["/init"] +ENTRYPOINT ["/init"] # ============================================================================ -# Stage: final-core (core mode - no SDR++) -# Purpose: Lightweight API + decoders (SDR++ externalized) -# Size: ~550MB +# Stage: final +# Purpose: Full single-host image. `final-base` plus SDR++ binaries plus the +# sdrpp-server s6 overlay (additive — does not modify the canonical +# tree, both unions cleanly). # ============================================================================ -FROM runtime-base AS final-core +FROM final-base AS final -LABEL maintainer="WaveKit Contributors" -LABEL description="WaveKit API + decoders (SDR++ externalized)" -LABEL mode="core" - -ENV WAVEKIT_HOME=/app \ - WAVEKIT_CONFIG_PATH=/app/config \ - WAVEKIT_LOG_LEVEL=info \ - NODE_ENV=production \ - PATH=/usr/local/bin:/usr/local/sbin:${PATH} - -RUN mkdir -p /app /var/log/wavekit /var/run/wavekit && \ - chmod 755 /app /var/log/wavekit /var/run/wavekit +LABEL mode="full" +LABEL description="WaveKit full single-host image (SDR++ + decoders + API)" +# SDR++-only runtime libs. Kept out of `runtime-base` so `final-core` +# doesn't carry them. RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ - --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt-get update && apt-get install -y --no-install-recommends \ - python3 \ - gnuradio \ - python3-numpy \ - python3-protobuf \ - python3-cryptography \ + libglfw3 \ + libvolk2.5 \ + libglvnd0 \ + libopengl0 \ && rm -rf /var/lib/apt/lists/* -# Copy decoders only (no SDR++) -COPY --from=dsd-fme-build /usr/local/bin/dsd* /usr/local/bin/ -COPY --from=dsd-fme-build /usr/local/lib/libmbe* /usr/local/lib/ -COPY --from=multimon-ng-build /usr/local/bin/multimon-ng /usr/local/bin/ -COPY --from=rtl433-build /usr/local/bin/rtl_433 /usr/local/bin/ -COPY --from=acarsdec-build /usr/local/bin/acarsdec /usr/local/bin/ -COPY --from=ais-catcher-build /usr/local/bin/AIS-catcher /usr/local/bin/ -COPY --from=direwolf-build /usr/local/bin/direwolf /usr/local/bin/ -COPY --from=direwolf-build /usr/local/bin/decode_aprs /usr/local/bin/ -COPY --from=direwolf-build /usr/local/bin/gen_packets /usr/local/bin/ -COPY --from=dumpvdl2-build /usr/local/bin/dumpvdl2 /usr/local/bin/ -COPY --from=dumpvdl2-build /usr/local/lib/libacars* /usr/local/lib/ -COPY --from=readsb-build /usr/local/bin/readsb /usr/local/bin/ -# Copy gr-lora_sdr blocks for LoRa/Meshtastic -# KEEP IN SYNC with the lora install in the final stage above. -COPY --from=lora-build /usr/local/share/wavekit-lora/lib/libgnuradio-lora_sdr* /usr/local/lib/ -COPY --from=lora-build /usr/local/share/wavekit-lora/lora_sdr /usr/lib/python3/dist-packages/gnuradio/lora_sdr -# Copy SoapyRTLTCP module for rtl_tcp network SDR support (acarsdec, dumpvdl2) -COPY --from=soapy-rtltcp-build /usr/local/lib/SoapySDR/modules0.8/librtltcpSupport.so /usr/local/lib/SoapySDR/modules0.8/ -# Copy csdr for FM demodulation (audio-from-IQ decoders) -COPY --from=csdr-build /usr/local/bin/csdr /usr/local/bin/ -COPY --from=csdr-build /usr/local/lib/libcsdr* /usr/local/lib/ -COPY docker/scripts/lora_meshtastic_decode.py /usr/local/bin/ -COPY docker/scripts/meshtastic_proto /usr/local/lib/wavekit/meshtastic_proto - -# Copy ncurses libraries from build stage to ensure version compatibility -COPY --from=dsd-fme-build /usr/lib/x86_64-linux-gnu/libncurses* /usr/lib/x86_64-linux-gnu/ - -# Run ldconfig to update cache -RUN chmod 755 /usr/local/bin/lora_meshtastic_decode.py && \ - ldconfig && \ - python3 -c "from gnuradio import lora_sdr; print(lora_sdr.__file__)" && \ - python3 /usr/local/bin/lora_meshtastic_decode.py --help > /dev/null - -# Copy direwolf configuration -COPY docker/config/direwolf.conf /etc/direwolf.conf - -# Copy Node.js application -COPY --from=node-build /usr/local/bin/node /usr/local/bin/ -COPY --from=node-build /app/dist /app/dist -COPY --from=node-build /app/node_modules /app/node_modules -COPY --from=node-build /app/packages/shared/package.json /app/packages/shared/package.json -COPY --from=node-build /app/packages/shared/dist /app/packages/shared/dist -COPY --from=node-build /app/packages/api-types/package.json /app/packages/api-types/package.json -COPY --from=node-build /app/packages/api-types/dist /app/packages/api-types/dist -COPY --from=node-build /app/config /app/config -COPY --from=node-build /app/package.json /app/ +# SDR++ binary, plugins, and shared core. Current upstream installs to +# /usr/bin and /usr/lib (not /usr/local/...). Keep the paths as-installed +# so `sdrpp`'s plugin discovery (compiled-in path) still finds the .so files. +COPY --from=sdrpp-build /usr/bin/sdrpp /usr/bin/sdrpp +COPY --from=sdrpp-build /usr/lib/libsdrpp_core.so /usr/lib/libsdrpp_core.so +COPY --from=sdrpp-build /usr/lib/sdrpp /usr/lib/sdrpp +RUN ldconfig -# Copy helper scripts -COPY docker/scripts/init-system.sh /usr/local/bin/ -COPY docker/scripts/start-api.sh /usr/local/bin/ -COPY docker/scripts/finish-api.sh /usr/local/bin/ -RUN chmod 755 /usr/local/bin/init-system.sh /usr/local/bin/start-api.sh /usr/local/bin/finish-api.sh +# sdrpp lifecycle scripts +COPY docker/scripts/start-sdrpp.sh /usr/local/bin/ +COPY docker/scripts/finish-sdrpp.sh /usr/local/bin/ +RUN chmod 755 /usr/local/bin/start-sdrpp.sh /usr/local/bin/finish-sdrpp.sh -# Copy s6-overlay service definitions (without sdrpp) -COPY docker/overlay/s6-overlay/s6-rc.d /etc/s6-overlay/s6-rc.d -RUN chmod -R 755 /etc/s6-overlay/s6-rc.d && \ - rm -rf /etc/s6-overlay/s6-rc.d/sdrpp-server && \ - rm -f /etc/s6-overlay/s6-rc.d/wavekit-api/dependencies.d/sdrpp-server && \ - rm -f /etc/s6-overlay/s6-rc.d/user/contents.d/sdrpp-server && \ - rm -f /etc/s6-overlay/s6-rc.d/services/contents.d/sdrpp-server +# Additive sdrpp overlay. Unions sdrpp-server service + its `contents.d` +# registration files into the canonical tree without touching anything else. +COPY docker/overlay/s6-overlay-sdrpp/s6-rc.d /etc/s6-overlay/s6-rc.d +RUN chmod -R 755 /etc/s6-overlay/s6-rc.d -COPY docker/scripts/healthcheck.sh /etc/s6-overlay/scripts/ -RUN chmod 755 /etc/s6-overlay/scripts/healthcheck.sh +EXPOSE 5259 7355 -EXPOSE 9000 8080 4713 +VOLUME ["/recordings"] -VOLUME ["/var/log/wavekit", "/var/run/wavekit"] - -HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \ - CMD s6-svstat -u /run/service/wavekit-api > /dev/null 2>&1 || exit 1 +# ============================================================================ +# Stage: final-core +# Purpose: Lightweight API + decoders (SDR++ externalized). Existence of the +# stage is the contract — no additional content. Property 8 / 9 +# tests run against this image. +# ============================================================================ +FROM final-base AS final-core -ENTRYPOINT ["/init"] +LABEL mode="core" +LABEL description="WaveKit API + decoders (SDR++ externalized)" # ============================================================================ -# Stage: final-sdrpp (sdrpp-only mode) -# Purpose: Just SDR++ server for dedicated SDR host -# Size: ~450MB +# Stage: final-sdrpp +# Purpose: SDR++ server only (IQ/audio provider for distributed deployments). # ============================================================================ FROM runtime-base AS final-sdrpp @@ -705,23 +756,43 @@ LABEL mode="sdrpp-only" ENV SDR_HOME=/sdr \ LOG_LEVEL=info -RUN mkdir -p /sdr /var/log/sdrpp && \ - chmod 755 /sdr /var/log/sdrpp +RUN mkdir -p /sdr /var/log/sdrpp /var/log/wavekit && \ + chmod 755 /sdr /var/log/sdrpp /var/log/wavekit -# Copy only SDR++ -COPY --from=sdrpp-build /usr/local/bin/sdrpp* /usr/local/bin/ -COPY --from=sdrpp-build /usr/local/lib/libsdrpp* /usr/local/lib/ +# SDR++-only runtime libs (kept out of runtime-base so final-core stays slim). +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + libglfw3 \ + libvolk2.5 \ + libglvnd0 \ + libopengl0 \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=sdrpp-build /usr/bin/sdrpp /usr/bin/sdrpp +COPY --from=sdrpp-build /usr/lib/libsdrpp_core.so /usr/lib/libsdrpp_core.so +COPY --from=sdrpp-build /usr/lib/sdrpp /usr/lib/sdrpp RUN ldconfig -# Copy helper scripts COPY docker/scripts/start-sdrpp.sh /usr/local/bin/ COPY docker/scripts/finish-sdrpp.sh /usr/local/bin/ RUN chmod 755 /usr/local/bin/start-sdrpp.sh /usr/local/bin/finish-sdrpp.sh -# Basic service setup for SDR++ -COPY docker/overlay/s6-overlay/s6-rc.d/base /etc/s6-overlay/s6-rc.d/base -COPY docker/overlay/s6-overlay/s6-rc.d/sdrpp-server /etc/s6-overlay/s6-rc.d/sdrpp-server - +# s6 tree for final-sdrpp: the canonical structural files (services + user +# bundle types) plus the sdrpp-server service from the sibling overlay. We +# do NOT bring in wavekit-init or wavekit-api — this image only runs sdrpp. +# COPY-by-path keeps the partition additive (Requirement 3.10) without +# pulling in unrelated services. +COPY docker/overlay/s6-overlay/s6-rc.d/services/type /etc/s6-overlay/s6-rc.d/services/type +COPY docker/overlay/s6-overlay/s6-rc.d/user/type /etc/s6-overlay/s6-rc.d/user/type +COPY docker/overlay/s6-overlay/s6-rc.d/user/contents.d/services /etc/s6-overlay/s6-rc.d/user/contents.d/services +COPY docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/type /etc/s6-overlay/s6-rc.d/sdrpp-server/type +COPY docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/run /etc/s6-overlay/s6-rc.d/sdrpp-server/run +COPY docker/overlay/s6-overlay-sdrpp/s6-rc.d/sdrpp-server/finish /etc/s6-overlay/s6-rc.d/sdrpp-server/finish +# sdrpp-server's `dependencies.d/wavekit-init` is INTENTIONALLY omitted — +# this image runs only sdrpp-server. Picking files individually keeps the +# tree partition additive (Requirement 3.10). +COPY docker/overlay/s6-overlay-sdrpp/s6-rc.d/services/contents.d/sdrpp-server /etc/s6-overlay/s6-rc.d/services/contents.d/sdrpp-server +COPY docker/overlay/s6-overlay-sdrpp/s6-rc.d/user/contents.d/sdrpp-server /etc/s6-overlay/s6-rc.d/user/contents.d/sdrpp-server RUN chmod -R 755 /etc/s6-overlay/s6-rc.d EXPOSE 5259 @@ -732,6 +803,46 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=20s --retries=3 \ ENTRYPOINT ["/init"] # ============================================================================ -# Default target (full mode) +# Stage: final-demod +# Purpose: Interactive utility container for offline decoder testing. Mirrors +# the binaries shipped in `final` (so test results match production) +# but ships no service tree — drop into a shell, prod the decoders +# directly. # ============================================================================ -FROM final AS default +FROM runtime-base AS final-demod + +LABEL maintainer="WaveKit Contributors" +LABEL description="WaveKit interactive demod test environment" +LABEL mode="demod-test" + +# Interactive tooling layered on top of runtime-base. sox/libsox-fmt-all and +# netcat-openbsd are already in runtime-base; the rest are demod-only extras. +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + apt-get update && apt-get install -y --no-install-recommends \ + vim \ + ffmpeg \ + gnuradio \ + gr-osmosdr \ + python3 \ + python3-pip \ + python3-numpy \ + python3-scipy \ + python3-matplotlib \ + rtl-sdr \ + && rm -rf /var/lib/apt/lists/* + +# Decoder binaries that the test loop wants on the PATH. Versions are +# locked-stepped to production by sourcing from the same *-build stages. +COPY --from=dsd-fme-build /usr/local/bin/dsd* /usr/local/bin/ +COPY --from=dsd-fme-build /usr/local/lib/libmbe* /usr/local/lib/ +COPY --from=multimon-ng-build /usr/local/bin/multimon-ng /usr/local/bin/ +COPY --from=csdr-build /usr/local/bin/csdr /usr/local/bin/ +COPY --from=csdr-build /usr/local/lib/libcsdr* /usr/local/lib/ + +RUN ldconfig + +WORKDIR /workspace +RUN mkdir -p /data/debug_audio /scripts /output + +# No s6, no ENTRYPOINT — `docker compose run --rm demod-test` drops into bash. +CMD ["/bin/bash"] From 65bc16b83a4a82d4d99735d6767a6e69dfd26bdc Mon Sep 17 00:00:00 2001 From: Benjamin Coriou Date: Sat, 16 May 2026 03:02:20 +0200 Subject: [PATCH 4/4] build(docker): exclude **/node_modules from build context; restore Req 3.6 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase B's node-build originally needed a `pnpm config set node-linker hoisted` reconfigure to work around two independent issues: 1. .dockerignore's bare `node_modules` pattern only matches the context-root directory, so cli/node_modules and packages/*/node_modules entered the build context and clobbered pnpm's fresh symlinks during COPY . . 2. .npmrc's `enable-global-virtual-store=true` puts the virtual store at ~/.local/share/pnpm/store/v10/links/, which collides with this stage's BuildKit cache mount target — symlinks dangle once the cache mount unmounts at layer commit. Fixing both root causes restores Requirement 3.6 strictly (.npmrc `node-linker=isolated` is the single source of truth for package layout): - .dockerignore: `node_modules` -> `**/node_modules` - Dockerfile: pass `--config.enable-global-virtual-store=false` to the install invocation (and the matching prune step), dropping the prior `pnpm config set node-linker hoisted` reconfigure entirely. The override is a per-invocation flag; .npmrc remains untouched so `pnpm dev` outside Docker uses the project default unchanged. --- .dockerignore | 2 +- Dockerfile | 36 ++++++------------------------------ 2 files changed, 7 insertions(+), 31 deletions(-) diff --git a/.dockerignore b/.dockerignore index 8f03ca1..05ddab7 100644 --- a/.dockerignore +++ b/.dockerignore @@ -4,7 +4,7 @@ .gitattributes # Dependencies -node_modules +**/node_modules npm-debug.log yarn-error.log package-lock.json.bak diff --git a/Dockerfile b/Dockerfile index 68d3bbc..9be2cd8 100644 --- a/Dockerfile +++ b/Dockerfile @@ -513,36 +513,11 @@ COPY packages/api-types/package.json packages/api-types/tsconfig.json ./packages COPY cli/package.json cli/tsconfig.json ./cli/ COPY packages/sdr-host/package.json packages/sdr-host/tsconfig.json ./packages/sdr-host/ -# R3 mitigation (Build & DX Overhaul, Phase B): -# -# Two layout issues prevent a single isolated install from working in this -# build context, both observed empirically during the refactor: -# -# 1. `.npmrc` has `enable-global-virtual-store=true`, which makes the -# installed virtual store live at the same cache-mount path used by -# the install. After the mount goes away the symlinks dangle. -# -# 2. `.dockerignore` excludes `node_modules` at the root only (not -# `**/node_modules`), so the host's `cli/node_modules/` and -# `packages/*/node_modules/` enter the build context via `COPY . .` -# and overwrite the freshly-installed workspace symlinks with paths -# that point at the developer's local pnpm store. -# -# Per the Phase B brief's R3 mitigation, switching node-build to a hoisted -# layout for a single install resolves both issues at once: `pnpm install` -# materializes real package directories instead of symlinks, so subsequent -# `COPY . .` can clobber the (now harmless) host node_modules without -# breaking workspace package resolution. -# -# Project default in `.npmrc` (node-linker=isolated) remains unchanged for -# native `pnpm dev`; this Dockerfile-local override only affects builds. -# A follow-up issue should fix `.dockerignore` to exclude `**/node_modules` -# so this override can be lifted (Requirement 3.6). -RUN pnpm config set node-linker hoisted --location=project && \ - pnpm config set enable-global-virtual-store false --location=project - +# Override .npmrc's enable-global-virtual-store=true: keep the virtual store +# at node_modules/.pnpm/ so symlinks survive the cache-mount unmount. RUN --mount=type=cache,target=/root/.local/share/pnpm/store,sharing=locked \ - pnpm install --frozen-lockfile --prod=false + pnpm install --frozen-lockfile --prod=false \ + --config.enable-global-virtual-store=false # Now copy sources. Edits here invalidate from this layer down — but # decoder *-build stages are unaffected. @@ -557,7 +532,8 @@ RUN --mount=type=cache,target=/root/.cache/turbo,sharing=locked \ pnpm run build RUN --mount=type=cache,target=/root/.local/share/pnpm/store,sharing=locked \ - CI=true pnpm prune --prod + CI=true pnpm prune --prod \ + --config.enable-global-virtual-store=false RUN ls -la dist/ && head -1 dist/index.js