diff --git a/README.md b/README.md index cdfdcda..02b5341 100644 --- a/README.md +++ b/README.md @@ -40,50 +40,42 @@ thing yourself. ## Run your own -You need a machine that can run Firecracker — a Linux box with `/dev/kvm`, **or -just your Mac**: - -**On a Linux box** — Ubuntu 24.04, **x86_64 or arm64**, with `/dev/kvm` -(bare-metal, or a VM with nested virtualization) that you can root-SSH into. One -command turns it into a running nehemiahd: +Nehemiah runs as a managed cloud: hosts are provisioned from **signed release +artifacts** onto machines the control plane enrolls, not from a local source +build. The supported path stands up one approved +[Latitude.sh](https://latitude.sh) bare-metal host from a published `v` +release: ```sh git clone https://github.com/boringcomputers/nehemiah cd nehemiah && npm install -# set it up on your box (installs Firecracker, builds the images, runs nehemiahd) -NEHEMIAH_ANTHROPIC_KEY=sk-ant-... ./infra/setup.sh root@YOUR_BOX_IP +# provision one managed host from a signed release (needs operator inputs — see the runbook) +infra/latitude/provision.sh --config "${XDG_CONFIG_HOME:-$HOME/.config}/nehemiah/latitude-host.env" ``` -Don't have a box? If you use [Latitude.sh](https://latitude.sh), -[`infra/latitude/provision.sh`](infra/latitude/provision.sh) creates one for you -first. Any other provider works too — just point `setup.sh` at it. - -**On an Apple Silicon Mac** (M3 or later) — no server needed. One command builds -the whole arm64 stack in a nested-virt [Lima](https://lima-vm.io) VM; real -microVMs boot on your laptop (a shell restores from snapshot in ~5 ms): - -```sh -brew install lima -NEHEMIAH_ANTHROPIC_KEY=sk-ant-... ./infra/local/setup-local.sh -# nehemiahd is now at http://localhost:8088 — details in infra/local/README.md -``` +Follow [`infra/latitude/README.md`](infra/latitude/README.md) for the full +managed-host runbook: the signed-release trust boundary, the one-use enrollment +grant, the WireGuard overlay, and the canary checks to run before admitting +workloads. Tear a host back down (and stop billing) with +[`infra/latitude/teardown.sh`](infra/latitude/teardown.sh). -(Windows 11 via WSL2 is designed but not yet wired up — see -[`infra/local/README.md`](infra/local/README.md).) +> **The earlier self-serve one-command installers are no longer supported.** +> `infra/setup.sh` (a Linux box over SSH) and `infra/local/setup-local.sh` (an +> Apple Silicon Mac in Lima) built the host from source, but host bootstrap now +> installs Firecracker, the jailer, and the kernel only from signed +> managed-release artifacts, which those scripts cannot supply. They now exit with +> a pointer to the managed runbook above. -Then run the site against it: +Then run the site against your host: ```sh # apps/web/.env -PUBLIC_NEHEMIAH_URL=http://YOUR_BOX_IP:8080 # or a tunnel — see apps/web/.env.example +PUBLIC_NEHEMIAH_URL=http://YOUR_HOST_IP:8080 # or a tunnel — see apps/web/.env.example npm run dev -w web ``` -`setup.sh` options (env): `NEHEMIAH_TOKEN` (require auth), `NEHEMIAH_S3_*` -(persistent volumes), `BIND_LOCALHOST=1` (reach it only via SSH tunnel — most -private), `SKIP_DESKTOP=1` (skip the ~8-min desktop image). Full REST + WebSocket -API in the [docs](https://boringcomputers.com/docs). +Full REST + WebSocket API in the [docs](https://boringcomputers.com/docs). **From any AI** — an MCP server ([`nehemiah-mcp`](packages/mcp)) lets Claude Desktop, Cursor, and other @@ -110,8 +102,9 @@ Real hardware-virtualized isolation — a kernel per machine, not a shared container. Each VM is jailed and resource-capped, restored from a memory snapshot in ~3 ms, and self-destructs on a TTL (or runs until you stop it, when the server enables `NEHEMIAH_ALLOW_PERSISTENT`). Guests are network-isolated -behind an egress firewall. The host daemon is [`nehemiahd/`](nehemiahd) (Go); host -setup is one command ([`infra/setup.sh`](infra/setup.sh)). +behind an egress firewall. The host daemon is [`nehemiahd/`](nehemiahd) (Go); +hosts are provisioned from a signed release via the managed runbook +([`infra/latitude/`](infra/latitude)). ## Repo @@ -122,8 +115,7 @@ apps/web/ the site — SvelteKit nehemiahd/ the host daemon — Go, runs the microVMs packages/sdk/ nehemiah-sdk — Effect-native TypeScript client packages/mcp/ nehemiah-mcp — MCP server -infra/setup.sh one-command host setup (any Ubuntu + KVM box) -infra/latitude/ rootfs/kernel/image builds, networking, Caddy, Latitude helpers +infra/latitude/ managed-host provisioning (provision/teardown), image builds, networking ``` ```sh diff --git a/apps/nehemiah/src/billing/metering.ts b/apps/nehemiah/src/billing/metering.ts index 599c122..e245ee1 100644 --- a/apps/nehemiah/src/billing/metering.ts +++ b/apps/nehemiah/src/billing/metering.ts @@ -195,7 +195,10 @@ const observationIntegrityReason = ( if (observation.kind === 'final') { const terminal = ['stopping', 'stopped', 'failed', 'lost'].includes(identity.state); if (observation.quality === 'exact') { - if (!terminal && receivedAt.getTime() < identity.expires_at.getTime() - maximumExactFinalLeadMs) { + if ( + !terminal && + receivedAt.getTime() < identity.expires_at.getTime() - maximumExactFinalLeadMs + ) { return 'premature_final'; } } else if (!terminal) { diff --git a/infra/latitude/README.md b/infra/latitude/README.md index 8af61ff..89a0dcb 100644 --- a/infra/latitude/README.md +++ b/infra/latitude/README.md @@ -1,204 +1,259 @@ -# Nehemiah — Latitude.sh runbook +# Manual Latitude managed-host runbook -This is the operator runbook for the one-box prototype: a single Latitude.sh -bare-metal server running **nehemiahd**, which launches **Firecracker microVMs** you -can create over REST and drive with a live shell over WebSocket. +This is the executable provisioning path for an approved beta host. It creates one Latitude bare-metal server, supplies a self-contained cloud-config, enrolls the daemon into the managed control plane, and removes the reusable Latitude user-data record after the server reaches `status=on`. -- **Server:** `c3.small.x86` @ **MIA2** (Miami), Ubuntu 24.04 x86_64, 6 cores / 32 GB, `/dev/kvm` present. -- **Cost:** ~**$0.52 / hr** (~$12.5/day, ~$375/mo) — see [COST](#cost). -- **IP / SSH key:** in `~/.config/latitude/server.env` (never committed). +It is deliberately manual. It does not implement provider autoscaling, capacity reconciliation, or control-plane-driven provisioning. Do not admit tenant workloads until the canary checks below pass. -> The server is **already provisioned**. This runbook covers everything after that: -> bootstrap → deploy → tunnel → demo → teardown. +## Trust and secret boundaries ---- +Use only a published `v` release that completed the protected release workflow. Managed cloud-init requires all of the following before it installs anything from the release: -## 0. Prerequisites (operator laptop) +- a valid Minisign signature over `SHA256SUMS`; +- the exact schema 5 / managed-host contract 4 release manifest and artifact matrix; +- the versioned daemon, guest-agent, and host-bootstrap archives; and +- signed, retained per-architecture Firecracker, jailer, and guest-kernel artifacts with exact SHA-256 pins; and +- signed `python` and `desktop` ext4 images plus their final-filesystem vulnerability-scan evidence; and +- a signed per-architecture Ubuntu 24.04 package closure built from the reviewed immutable snapshot, including its exact package/index identities. -Create `~/.config/latitude/server.env`: +There is no mutable `latest` lookup. The release image builder uses exact +per-architecture Node 24.19 OCI digests, SHA-256-pinned Alpine 3.23 repository +indexes, and SHA-256-pinned npm runtime inputs from the signed manifest. +Cloud-init never resolves apk, network apt, npm, or OCI inputs and has no +minimal-image fallback. It installs host packages only from the selected signed +release's flat repository with `--no-download`, then continuously checks exact +package versions, architecture and non-documentation package files for drift. +The trusted release build is the only step that contacts the reviewed upstream +Firecracker/kernel locations. Provisioning consumes those bytes only from the +immutable signed release, so a fresh host or rollback does not depend on +upstream object retention. -```sh -# ~/.config/latitude/server.env (chmod 600 — do NOT commit) -SERVER_IP=203.0.113.10 # public IPv4 of the box -SSH_KEY=/Users/you/.ssh/latitude # private key that can root@ the box -NEHEMIAH_TOKEN= # optional bearer token for /v1/* (leave empty for tunnel-only) - -# for teardown.sh (billing): -LATITUDE_API_KEY=... # or put it in ~/.config/latitude/api_key -LATITUDE_SERVER_ID=sv_... # or put it in ~/.config/latitude/server_id -``` - -All scripts here `source` this file. The API key is never printed. - ---- +The operator config contains a short-lived, one-use, host-bound enrollment grant and a WireGuard private key. The grant is issued by an allowlisted fleet operator only after the provider identity, region, overlay address, architecture, and exact capacity are known; it is not a reusable control-plane environment secret. The rendered cloud-config contains recoverable base64 encodings of those values. Both files must remain mode 0600. The renderer and provisioner never print credentials or place them in command arguments, but Latitude necessarily receives the cloud-config to install the host. Deleting the API user-data object cannot prove deletion from provider audit logs or backups. -## 1. Bootstrap the box (once) - -`bootstrap.sh` (owned separately, run **on the box**) installs the Go toolchain, -downloads the Firecracker binary, an uncompressed guest kernel, and builds the -Alpine rootfs with `python3` + the `NEHEMIAH_READY` serial marker. It lays down the -fixed host paths nehemiahd expects: - -``` -/opt/boring/bin/firecracker firecracker (also jailer) -/opt/boring/kernel/vmlinux uncompressed firecracker-compatible kernel -/opt/boring/rootfs/rootfs.ext4 base Alpine rootfs (busybox init, /bin/sh on ttyS0, python3) -/opt/boring/templates// optional snapshot template (snapshot_file, mem_file, rootfs.ext4) -/opt/boring/run/ per-machine sockets + rootfs overlays -``` +## 1. Prepare private operator inputs -Run it once after provisioning: +Copy the example outside the checkout and lock it before editing: ```sh -scp -i "$SSH_KEY" infra/bootstrap.sh root@$SERVER_IP:/root/ -ssh -i "$SSH_KEY" root@$SERVER_IP 'bash /root/bootstrap.sh' +install -d -m 0700 "${XDG_CONFIG_HOME:-$HOME/.config}/nehemiah" +install -m 0600 infra/latitude/managed-host.env.example \ + "${XDG_CONFIG_HOME:-$HOME/.config}/nehemiah/latitude-host.env" ``` -Sanity check on the box: `ls -la /dev/kvm` and `/opt/boring/bin/firecracker --version`. - ---- - -## 2. Deploy nehemiahd - -From the repo root on your laptop: +Fill every placeholder in that file: + +- `NEHEMIAH_RELEASE_BASE` is the release download collection, normally `https://github.com/boringcomputers/nehemiah/releases/download`. +- `NEHEMIAH_RELEASE_VERSION` is an exact tag version without the leading `v`. +- `NEHEMIAH_RELEASE_MINISIGN_KEY` is the reviewed `RW...` public key matching the protected release environment. +- `NEHEMIAH_HOST_ID` and `NEHEMIAH_PROVIDER_ID` are stable, unique identifiers for this approved physical host. Reusing a provider identity after enrollment is rejected. +- `LATITUDE_OS_ID` is an operator-reviewed opaque Latitude operating-system ID; + the adjacent slug, version and architecture must be copied from the same live + API record. Before any user-data or billable server request, the provisioner + re-fetches the bounded OS inventory and requires the exact ID-to-fields match + plus `LATITUDE_PLAN` membership in `provisionable_on`. It records the selected + evidence beside the server ID. There is deliberately no default production OS. +- `NEHEMIAH_FLEET_BOOTSTRAP_TOKEN` is the per-host `nhe_...` enrollment grant returned once by `POST /v1/operator/host-enrollments`. It expires in at most 30 minutes and is bound to this file's provider identity, region, overlay address, architecture, and capacity. Never reuse a token or configure it as a control-plane-wide secret. +- `NEHEMIAH_TEMPLATE_OBJECT_ORIGIN` is optional unless durable template transfers + are enabled. Set it to the exact HTTPS origin produced by presigned template + URLs (bucket hostname for virtual-hosted S3, endpoint origin for path-style S3). + The host rejects redirects and every other object origin. +- `NEHEMIAH_ADVERTISE_ADDRESS` is this host's literal managed-overlay address, not its public Latitude address. +- `NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS` and + `NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS` are distinct, reviewed overlay host + addresses for those two roles. They must use the host overlay's address + family and must not overlap the managed guest subnet. +- `NEHEMIAH_WIREGUARD_CONFIG_B64` is `base64 -w0` of the complete host `wg0.conf`. + The renderer parses and canonically re-renders exactly one Interface and one + Peer. The interface must contain only its private key and the advertised host + address; the peer must contain only its key, bounded endpoint, the exact two + separately configured CP/gateway host routes, and `PersistentKeepalive = 25`. + Missing, extra, arbitrary, self, or guest-subnet routes fail closed. Hooks, DNS, + Table, SaveConfig, comments, duplicate/unknown fields, and broad/default routes + fail at render, cloud-init and every managed service preflight. +- All nine telemetry values are mandatory. Use an origin-only HTTPS collector + URL with a DNS hostname, exact `NEHEMIAH_OTEL_ENABLED=true`, bounded export + settings, the immutable release/service and stable host identities, and a + per-host authorization credential that is distinct from the fleet, + daemon-internal, and gateway credentials. +- `NEHEMIAH_ROOT_SSH_AUTHORIZED_KEY_B64` is `base64 -w0` of one recovery **public** SSH key line. Never put an SSH private key here. + +The file format is literal `KEY=value`, not shell. Unknown keys, duplicates, shell syntax, unsafe URLs, malformed base64, and loose file permissions fail closed. + +Store the Latitude API key in a separate private file: ```sh -infra/latitude/deploy.sh +install -d -m 0700 "${XDG_CONFIG_HOME:-$HOME/.config}/latitude" +install -m 0600 /dev/stdin "${XDG_CONFIG_HOME:-$HOME/.config}/latitude/api_key" ``` -This rsyncs `nehemiahd/` to `/opt/boring/src/`, builds a static binary to -`/usr/local/bin/nehemiahd`, installs `nehemiahd.service`, writes -`/etc/boring/nehemiahd.env` (with `NEHEMIAH_TOKEN` if you set one), enables the -service, and curls `localhost:8080/healthz` to confirm. Re-run any time to ship -a new build. +Paste only the API key, then end input with Ctrl-D. The provisioner also accepts `LATITUDE_API_KEY` for ephemeral automation, but the file avoids putting it in a long-lived shell environment. -Logs: `ssh -i "$SSH_KEY" root@$SERVER_IP journalctl -u nehemiahd -f` +## 2. Rehearse rendering offline ---- - -## 3. Open the tunnel - -nehemiahd binds `0.0.0.0:8080` on the box but is **firewalled to localhost only** -(see [SECURITY](#security)). Reach it from your laptop over SSH: +Rendering makes no network or provider request and refuses to overwrite a path: ```sh -infra/latitude/tunnel.sh # localhost:8080 -> box:8080 +infra/latitude/render-user-data.sh \ + --config "${XDG_CONFIG_HOME:-$HOME/.config}/nehemiah/latitude-host.env" \ + --output "${XDG_CONFIG_HOME:-$HOME/.config}/nehemiah/latitude-user-data.yaml" ``` -Leave it running. Verify: `curl http://localhost:8080/healthz` → `{"ok":true,...}`. - ---- - -## 4. Run the demo - -In another terminal, from the repo root: +The output is mode 0600. Delete it securely after review; it is not a harmless public artifact. The equivalent provisioner rehearsal is: ```sh -NEHEMIAH_URL=http://localhost:8080 node packages/sdk/demo.mjs -# if you set a token: -NEHEMIAH_URL=http://localhost:8080 NEHEMIAH_TOKEN=... node packages/sdk/demo.mjs +infra/latitude/provision.sh \ + --config "${XDG_CONFIG_HOME:-$HOME/.config}/nehemiah/latitude-host.env" \ + --output "${XDG_CONFIG_HOME:-$HOME/.config}/nehemiah/latitude-user-data.yaml" \ + --render-only ``` -The demo exercises the full contract: +Use a new output pathname for each run because no existing file is overwritten. -1. `POST /v1/machines {"template":"python","ttl_seconds":120}` → a microVM (`mode` is `coldboot` or `snapshot`, with a measured `boot_ms`). -2. Attaches to `GET /v1/machines/{id}/tty` (WebSocket, binary frames) — a **live serial shell**. -3. Runs `python3` inside the guest and prints the output. -4. `POST /v1/machines/{id}/branch` — **forks** the VM from its snapshot (best-effort; `501` if snapshots aren't available). -5. `DELETE /v1/machines/{id}` to clean up. +## 3. Provision one approved host -You can also drive it by hand: +Latitude project, SSH-key, plan, site, and hostname values are identifiers rather than credentials. The approved OS identity is read only from the private config above: ```sh -curl -s localhost:8080/v1/machines -XPOST -H 'content-type: application/json' \ - -d '{"template":"python","ttl_seconds":120}' -curl -s localhost:8080/v1/machines +export LATITUDE_PROJECT=proj_... +export LATITUDE_SSH_KEY=ssh_... +export LATITUDE_PLAN=c3-small-x86 +export LATITUDE_SITE=MIA2 +export LATITUDE_HOSTNAME=nehemiah-metal-01 # base name, max 23 chars + +infra/latitude/provision.sh \ + --config "${XDG_CONFIG_HOME:-$HOME/.config}/nehemiah/latitude-host.env" ``` -### HTTP/WS contract (reference) +The hostname doubles as the durable recovery correlation value for the +billable create, so every run appends a unique random suffix to the base +(`nehemiah-metal-01-3fa9c2d1`) and refuses to provision while any live server +already carries the resulting name. The full suffixed hostname is what +provisioning records for `teardown.sh` recovery. + +The provisioner: + +1. renders into a private temporary directory; +2. resolves and records the exact approved Latitude OS ID/slug/version/architecture/plan record; +3. creates a one-time Latitude user-data object with base64 content; +4. creates an hourly-billed server referencing that object and the uploaded SSH key; +5. waits for Latitude to report the server online; +6. deletes the reusable user-data API object; and +7. stores the non-secret server ID and mode-0600 provider-image evidence under `${XDG_CONFIG_HOME:-$HOME/.config}/latitude` without overwriting prior records. + +It never prints an API response because the create-user-data response echoes decoded credential material. On failure it still attempts to delete the one-time object. A server ID in an error means the hourly-billed server exists; inspect or tear it down promptly. + +Cloud-init then verifies the published release, atomically installs the exact +`python` developer and VNC `desktop` ext4 images, validates their signed size, +compression, ext4 type, and filesystem health, installs the pinned host inputs, +verifies their runtime cohort ID (contract, arch and five installed hashes), +configures canonical WireGuard and the isolated guest bridge, starts the daemon, and waits +for durable control-plane enrollment. Only after enrollment does it remove the +fleet token, private bootstrap environment, and cached cloud-init user data. +Success creates `/var/lib/nehemiahd/bootstrap.complete`. + +The signed systemd units are managed-only. They require the root-owned +`/etc/boring/nehemiahd.env` and `/etc/boring/managed-host` marker created after +the verified runtime bootstrap, and their command lines hard-pin +`NEHEMIAH_MODE=1` even if the environment file says otherwise. `nehemiahd` is +bound to `boring-net`: a failed or stopped network unit stops the daemon, and +the network unit disables and detaches every guest tap on failed setup or stop. +Local setup scripts install separate `*-local.service` units and never create +the managed marker. + +### Recover a lost enrollment response + +The grant is consumed in the same database statement that creates or recovers +the host row. If that `201` response is lost before the daemon durably stores its +heartbeat credential, automatic retries correctly fail because the grant cannot +be replayed. This is a manual static-fleet recovery case: + +1. Confirm `/var/lib/nehemiahd/enrollment.json` is absent and record the exact + provider ID, region, overlay address, architecture, and capacity from the + original operator config. Do not change or recycle the provider identity. +2. From an allowlisted fleet-operator session, issue another + `POST /v1/operator/host-enrollments` with those exact bindings. The control + plane reserves the existing pre-heartbeat host ID and revokes any other + pending grant. +3. Over the recovery SSH channel, stop `nehemiahd`, edit + `/etc/boring/nehemiahd.env` with a credential-safe editor (do not put the + returned token in command arguments or shell history), replace only the + `NEHEMIAH_FLEET_BOOTSTRAP_TOKEN=` value, and retain root ownership and mode + `0600`. +4. Restart `nehemiahd`. Require a mode-`0600` enrollment file, an accepted fresh + heartbeat, and automatic removal of the token line from the environment file + before resuming the canary. Revoke the unused grant through + `POST /v1/operator/host-enrollments/:id/revoke` if recovery is abandoned. + +After the first accepted heartbeat, enrollment is permanently closed; use the +separate operator credential-rotation lifecycle rather than issuing another +enrollment grant. + +## 4. Canary acceptance + +The provider's `status=on` is not proof that cloud-init or enrollment succeeded. Before workloads are admitted, use the recovery SSH key and confirm: -``` -GET /healthz -> {"ok":true,"machines":,"kvm":} (no auth) -POST /v1/machines -> 201 {"id","status","mode","boot_ms","created_at","expires_at"} -GET /v1/machines -> 200 {"machines":[...]} -GET /v1/machines/{id} -> 200 | 404 -DELETE /v1/machines/{id} -> 204 | 404 -POST /v1/machines/{id}/branch -> 201 | 501 -GET /v1/machines/{id}/tty -> WebSocket, BINARY frames both ways (serial stdin/stdout) +```sh +test -r /var/lib/nehemiahd/bootstrap.complete +test -r /var/lib/nehemiahd/enrollment.json +test "$(stat -c '%a:%u' /etc/boring/managed-host)" = 400:0 +test "$(cat /etc/boring/managed-host)" = nehemiah-managed-host-v1 +systemctl is-active --quiet wg-quick@wg0 boring-net nehemiahd +test "$(systemctl show nehemiahd.service -p KillMode --value)" = control-group +test -f /opt/boring/rootfs/rootfs.ext4 +test -f /opt/boring/rootfs/desktop.ext4 +/opt/boring/bin/managed-host-packages verify-installed \ + --arch "$(dpkg --print-architecture)" --release-version 0.2.0-beta.0 +/opt/nehemiah/infra/latitude/verify-isolation.sh +journalctl -u nehemiahd --since=-10m --no-pager ``` -Auth: if `NEHEMIAH_TOKEN` is set, send `Authorization: Bearer ` on `/v1/*` -(the WebSocket also accepts `?token=`). `/healthz` is always open. +Also cold-boot one `python` machine and one `desktop` machine. Require Node 24.19, +npm 11.19, git, curl with a valid CA store, shell and guest-agent exec in both; require +the existing VNC/vsock desktop contract in `desktop`. Confirm the control plane +reports this exact provider identity healthy, the advertised address is the +expected managed-overlay IP, `/dev/kvm` is usable, the one-time user-data record +no longer exists in Latitude, and dnsmasq is DHCP-only while the daemon owns +managed DNS interception. Treat any mismatch as a failed canary. ---- +For the restart canary, create one test machine through the control plane and +record its host-local ID, lease, VMM PID, and +`nehemiah-vmm-.scope`. Restart `nehemiahd.service`, then require +the same scope and PID to remain active, the daemon to report the same current +lease, and fresh exec/TTY/VNC connections to succeed. Also require a terminal +opened before restart to disconnect without leaking its guest shell and the SDK +reconnect path to open a fresh PTY. Delete the canary through the control plane +and require the exact scope, cgroup, tap, socket, and overlay to disappear. -## 5. Teardown (STOP BILLING) +## 5. Stop billing -When you're done, **delete the server** so the meter stops: +Deleting the server is destructive and is the only way to stop its hourly meter: ```sh -infra/latitude/teardown.sh # type "yes" to confirm +infra/latitude/teardown.sh ``` -This calls `DELETE https://api.latitude.sh/servers/`. It never prints the API -key. Deleting the server does not delete the (free) Latitude **project** — if it's -now empty you can remove it from the dashboard too. - ---- - -## SECURITY - -> **This platform runs untrusted, arbitrary code inside the microVMs.** Treat the -> whole box as hostile-tenant territory. - -Current posture (prototype): - -- **Bound to localhost / SSH tunnel only.** Do **not** expose `:8080` publicly. - Keep the box firewalled (e.g. `ufw` default-deny inbound except `22`). -- **Set a `NEHEMIAH_TOKEN`** even behind the tunnel as defense-in-depth. -- Firecracker already gives you a KVM hardware boundary + minimal device model — - much stronger isolation than containers. - -**Hardening TODO before any public exposure (not done yet):** - -- **jailer** — run each firecracker under `jailer` (chroot, `cgroups`, `pid`/`net` - namespaces, drop to an unprivileged uid). Today nehemiahd runs firecracker as root. -- **seccomp** — enforce firecracker's seccomp filters (advanced/custom profile), - and confine nehemiahd itself. -- **egress limits** — the demo skips guest networking entirely. If/when you add a - tap per VM, put the guest behind a default-deny NAT with strict egress - allow-lists and per-VM rate limits; block link-local/metadata ranges. -- **resource caps** — enforce vCPU/mem/disk quotas (already 1 vCPU / 256 MiB / - overlay per VM), plus `NEHEMIAH_MAX` (default 20) and TTLs (15–900 s) to bound blast radius. -- **rootfs is copy-on-write per VM** (`cp --reflink=auto`) so tenants can't corrupt - the base image, and VMs are destroyed on TTL/DELETE. - -Until all of the above lands, this stays a single-operator prototype reachable -only through the SSH tunnel. - ---- +The teardown script reads the private API key and saved server ID from the operator's `.config/latitude` directory and asks for explicit confirmation. If provisioning preserved an older `server_id`, pass the newly printed ID as `LATITUDE_SERVER_ID`. -## COST +## Evidence still required outside the repository -- **Server:** `c3.small.x86` @ MIA2 ≈ **$0.52 / hr** - - ≈ **$12.48 / day** - - ≈ **$375 / month** if left running 24×7. -- Billing is **hourly while the server exists** — it accrues whether or not nehemiahd - is running or any VMs are up. The only way to stop it is `teardown.sh` (delete - the server). -- microVMs themselves are free (they're just processes on the box); the cost is the - bare-metal host. -- **Habit:** provision → bootstrap → deploy → demo → **teardown** in one sitting, or - you'll pay for idle hours. Latitude projects are free; only servers bill. +Offline tests validate rendering, strict WireGuard grammar, provider-image mismatch denial before billing, API request structure, one-time user-data deletion, exact release filenames, signature-before-manifest ordering, mutable-URL rejection, and exact template-object-origin propagation. They cannot prove current Latitude inventory, provider OS image bytes/attestation, KVM availability, provider retention behavior, live native-arm package installation, WireGuard routing, or a real control-plane heartbeat. Each release/provider-image combination therefore needs an approved no-workload Latitude canary and recorded acceptance evidence before beta use. ---- +## Files -## Files in this directory +| File | Purpose | +| --------------------------- | -------------------------------------------------------------------------- | +| `managed-host.env.example` | Strict operator-input template with no real credentials. | +| `render-user-data.sh` | Offline mode-0600 cloud-config renderer. | +| `provision.sh` | Manual Latitude user-data/server API flow. | +| `cloud-init.sh` | Fail-closed signed-release installer and enrollment gate. | +| `bootstrap.sh` | Installs retained release Firecracker/kernel; requires both signed images. | +| `build-rootfs.sh` | Mutable Alpine builder for local development only; rejects managed use. | +| `build-desktop-rootfs.sh` | Mutable desktop builder for local development only; rejects managed use. | +| `managed-host-preflight.sh` | Exact marker/env ownership gate shared by both managed systemd units. | +| `net-setup.sh` | Isolated bridge, DHCP, and egress firewall. | +| `boring-net.service` | Fail-closed managed networking unit; requires verified marker and env. | +| `nehemiahd.service` | Managed daemon unit bound to networking; hard-pins managed mode. | +| `*-local.service` | Permissive units used only by local/prototype setup scripts. | +| `verify-isolation.sh` | On-host network/isolation acceptance checks. | +| `teardown.sh` | Confirmed server deletion to stop billing. | -| File | Purpose | -| ------------------- | ------------------------------------------------------------------------------- | -| `deploy.sh` | Build & deploy nehemiahd to the box; install/enable systemd unit; health-check. | -| `tunnel.sh` | SSH tunnel `localhost:8080 → box:8080` for running the demo. | -| `teardown.sh` | Delete the Latitude server via API to stop billing (confirmation required). | -| `nehemiahd.service` | systemd unit installed on the box by `deploy.sh`. | -| `README.md` | This runbook. | +The older `deploy.sh` and `tunnel.sh` remain local/prototype conveniences. They are not substitutes for the managed signed-release path. diff --git a/infra/latitude/bootstrap.sh b/infra/latitude/bootstrap.sh index 6083636..359a02d 100755 --- a/infra/latitude/bootstrap.sh +++ b/infra/latitude/bootstrap.sh @@ -6,8 +6,15 @@ # Run as root on the target box: # sudo bash infra/latitude/bootstrap.sh # -# Idempotent and safe to re-run. Installs firecracker + jailer, fetches a -# firecracker-compatible uncompressed kernel, and builds the base Alpine rootfs. +# Idempotent and safe to re-run. Installs Firecracker, jailer, and the kernel +# from local artifacts already verified against the signed release. Production +# guest images must also have been installed by cloud-init. +# +# MANAGED HOSTS ONLY. This installs exclusively from signed managed-release +# artifacts and requires the managed-release inputs below (NEHEMIAH_RELEASE_VERSION, +# the signed archive/kernel, and the managed-host package cohort). It is driven by +# infra/latitude/provision.sh + cloud-init; the self-serve infra/setup.sh and +# infra/local/setup-local.sh flows do not satisfy this contract and refuse to run it. # set -euo pipefail @@ -22,31 +29,20 @@ ROOTFS_DIR="${NEHEMIAH_ROOT}/rootfs" RUN_DIR="${NEHEMIAH_ROOT}/run" TEMPLATE_DIR="${NEHEMIAH_ROOT}/templates" -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" - -GITHUB_API="https://api.github.com/repos/firecracker-microvm/firecracker/releases/latest" +: "${NEHEMIAH_FIRECRACKER_ARCHIVE:?signed release Firecracker archive is required}" +: "${NEHEMIAH_FIRECRACKER_SHA256:?signed manifest Firecracker SHA-256 is required}" +: "${NEHEMIAH_KERNEL_IMAGE:?signed release kernel image is required}" +: "${NEHEMIAH_KERNEL_SHA256:?signed manifest kernel SHA-256 is required}" +: "${NEHEMIAH_FIRECRACKER_INSTALLED_SHA256:?signed installed Firecracker SHA-256 is required}" +: "${NEHEMIAH_JAILER_INSTALLED_SHA256:?signed installed jailer SHA-256 is required}" +: "${NEHEMIAH_RELEASE_VERSION:?signed release version is required}" +: "${NEHEMIAH_RUNTIME_PYTHON_SHA256:?signed python rootfs SHA-256 is required}" +: "${NEHEMIAH_RUNTIME_DESKTOP_SHA256:?signed desktop rootfs SHA-256 is required}" # Arch — firecracker + kernel artifacts differ between x86_64 and aarch64. uname's # names (x86_64 / aarch64) match firecracker's release naming, so ARCH drives both. ARCH="$(uname -m)" -# Kernel candidate URLs (arch-specific), tried in order. First one that downloads -# AND passes the "file" ELF/Linux-kernel check wins. -case "${ARCH}" in - x86_64) - KERNEL_URLS=( - "https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/v1.11/x86_64/vmlinux-6.1.128" - "https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/v1.10/x86_64/vmlinux-6.1.102" - "https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/x86_64/kernels/vmlinux.bin" - ) ;; - aarch64) - KERNEL_URLS=( - "https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/v1.12/aarch64/vmlinux-6.1.128" - "https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/v1.10/aarch64/vmlinux-6.1.102" - "https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/aarch64/kernels/vmlinux.bin" - ) ;; -esac - # -------------------------------------------------------------------------- # Logging helpers # -------------------------------------------------------------------------- @@ -54,6 +50,73 @@ log() { printf '\033[1;34m[bootstrap]\033[0m %s\n' "$*"; } warn() { printf '\033[1;33m[bootstrap:warn]\033[0m %s\n' "$*" >&2; } die() { printf '\033[1;31m[bootstrap:error]\033[0m %s\n' "$*" >&2; exit 1; } +# The first identity in nehemiahd's bounded per-VM UID/GID pool has a named +# bootstrap account. Every other slot remains numeric-only. Never hide account +# creation errors: a stale name or an unrelated owner of either numeric id is a +# host-identity collision and must abort provisioning. +ensure_boringjail_account() { + local account_name=boringjail expected_uid=30000 expected_gid=30000 + local expected_home=/nonexistent expected_shell=/usr/sbin/nologin + local group_by_name group_by_id user_by_name user_by_id password_status + local name uid gid gecos home shell members extra status + + lookup_record() { + local database="$1" key="$2" destination="$3" output lookup_status + set +e + output="$(getent "$database" "$key" 2>/dev/null)" + lookup_status=$? + set -e + case "$lookup_status" in + 0) [[ -n "$output" && "$output" != *$'\n'* ]] || die "ambiguous ${database} identity for ${key}" ;; + 2) output="" ;; + *) die "cannot query ${database} identity for ${key}" ;; + esac + printf -v "$destination" '%s' "$output" + } + + lookup_record group "$account_name" group_by_name + lookup_record group "$expected_gid" group_by_id + if [[ -z "$group_by_name" && -z "$group_by_id" ]]; then + groupadd --gid "$expected_gid" "$account_name" \ + || die "cannot create the managed jailer group" + elif [[ -z "$group_by_name" || -z "$group_by_id" || "$group_by_name" != "$group_by_id" ]]; then + die "managed jailer group name or gid is already owned by another identity" + fi + lookup_record group "$account_name" group_by_name + lookup_record group "$expected_gid" group_by_id + IFS=: read -r name _ gid members extra <<<"$group_by_name" + [[ "$group_by_name" == "$group_by_id" && "$name" == "$account_name" && \ + "$gid" == "$expected_gid" && -z "$members" && -z "$extra" ]] \ + || die "managed jailer group does not match the signed host contract" + + lookup_record passwd "$account_name" user_by_name + lookup_record passwd "$expected_uid" user_by_id + if [[ -z "$user_by_name" && -z "$user_by_id" ]]; then + useradd --uid "$expected_uid" --gid "$expected_gid" --no-create-home \ + --home-dir "$expected_home" --shell "$expected_shell" --comment "" "$account_name" \ + || die "cannot create the managed jailer account" + elif [[ -z "$user_by_name" || -z "$user_by_id" || "$user_by_name" != "$user_by_id" ]]; then + die "managed jailer account name or uid is already owned by another identity" + fi + lookup_record passwd "$account_name" user_by_name + lookup_record passwd "$expected_uid" user_by_id + IFS=: read -r name _ uid gid gecos home shell extra <<<"$user_by_name" + [[ "$user_by_name" == "$user_by_id" && "$name" == "$account_name" && \ + "$uid" == "$expected_uid" && "$gid" == "$expected_gid" && -z "$gecos" && \ + "$home" == "$expected_home" && "$shell" == "$expected_shell" && -z "$extra" ]] \ + || die "managed jailer account does not match the signed host contract" + + password_status="$(passwd --status "$account_name" 2>/dev/null)" \ + || die "cannot verify that the managed jailer account is locked" + read -r name status extra <<<"$password_status" + [[ "$name" == "$account_name" && "$status" == "L" ]] \ + || die "managed jailer account must remain password-locked" + + install -d -o root -g root -m 0755 /srv/jailer + [[ "$(stat -c '%a:%u:%g' /srv/jailer)" == "755:0:0" ]] \ + || die "managed jailer chroot base must be root-owned mode 0755" +} + # -------------------------------------------------------------------------- # Preconditions # -------------------------------------------------------------------------- @@ -65,16 +128,33 @@ case "${ARCH}" in esac log "Target arch: ${ARCH}" +for digest in \ + "${NEHEMIAH_FIRECRACKER_SHA256}" "${NEHEMIAH_KERNEL_SHA256}" \ + "${NEHEMIAH_FIRECRACKER_INSTALLED_SHA256}" "${NEHEMIAH_JAILER_INSTALLED_SHA256}" \ + "${NEHEMIAH_RUNTIME_PYTHON_SHA256}" "${NEHEMIAH_RUNTIME_DESKTOP_SHA256}"; do + [[ "${digest}" =~ ^[0-9a-f]{64}$ ]] || die "signed manifest contains an invalid SHA-256" +done +for asset in "${NEHEMIAH_FIRECRACKER_ARCHIVE}" "${NEHEMIAH_KERNEL_IMAGE}"; do + [[ -f "${asset}" && ! -L "${asset}" && -s "${asset}" ]] \ + || die "signed release runtime input is missing or unsafe" +done +[[ "$(stat -c %s "${NEHEMIAH_FIRECRACKER_ARCHIVE}")" -le 16777216 ]] \ + || die "signed release Firecracker archive exceeds its size policy" +[[ "$(stat -c %s "${NEHEMIAH_KERNEL_IMAGE}")" -le 67108864 ]] \ + || die "signed release kernel exceeds its size policy" + +ASSET_WORK="$(mktemp -d /var/tmp/nehemiah-assets.XXXXXX)" +trap 'rm -rf -- "${ASSET_WORK}"' EXIT + # -------------------------------------------------------------------------- -# 1. Packages +# 1. Signed offline package cohort # -------------------------------------------------------------------------- -log "Updating apt and installing dependencies..." -export DEBIAN_FRONTEND=noninteractive -apt-get update -y -apt-get install -y --no-install-recommends \ - curl ca-certificates jq e2fsprogs iproute2 iptables cpio util-linux \ - git build-essential file debootstrap -log "Dependencies installed." +package_arch=amd64 +[[ "$ARCH" == aarch64 ]] && package_arch=arm64 +python3 /opt/boring/bin/managed-host-packages verify-installed \ + --arch "$package_arch" --release-version "$NEHEMIAH_RELEASE_VERSION" \ + || die "signed managed-host package cohort is missing or has drifted" +log "Signed offline package cohort verified." # -------------------------------------------------------------------------- # 2. KVM verification + ip_forward @@ -93,7 +173,18 @@ log "Enabling net.ipv4.ip_forward..." sysctl -w net.ipv4.ip_forward=1 >/dev/null # Persist across reboots (idempotent). if [ -d /etc/sysctl.d ]; then - echo "net.ipv4.ip_forward=1" > /etc/sysctl.d/99-boring.conf + cat > /etc/sysctl.d/99-nehemiah.conf <<'EOF' +net.ipv4.ip_forward=1 +net.ipv4.conf.all.accept_redirects=0 +net.ipv4.conf.default.accept_redirects=0 +net.ipv4.conf.all.send_redirects=0 +net.ipv4.conf.default.send_redirects=0 +net.ipv4.conf.all.route_localnet=0 +net.ipv4.conf.default.route_localnet=0 +kernel.unprivileged_bpf_disabled=1 +fs.protected_fifos=2 +fs.protected_regular=2 +EOF fi # -------------------------------------------------------------------------- @@ -101,120 +192,127 @@ fi # -------------------------------------------------------------------------- log "Creating ${NEHEMIAH_ROOT} layout..." mkdir -p "${BIN_DIR}" "${KERNEL_DIR}" "${ROOTFS_DIR}" "${RUN_DIR}" "${TEMPLATE_DIR}" +install -d -m0700 /var/lib/nehemiahd -# Jailer prerequisites: the chroot base + the unprivileged uid/gid the jailer -# drops firecracker into (NEHEMIAH_JAILER=1). Without these, jailed boots fail with +# Jailer prerequisites: the chroot base + the first unprivileged uid/gid in the +# per-machine pool. Without these, jailed boots fail with # "Canonicalize(/srv/jailer)" / "fc.sock did not appear". -mkdir -p /srv/jailer -groupadd -g 991 boringjail 2>/dev/null || true -useradd -u 30000 -g 991 -M -s /usr/sbin/nologin boringjail 2>/dev/null || true +ensure_boringjail_account # -------------------------------------------------------------------------- # 4. Install firecracker + jailer # -------------------------------------------------------------------------- install_firecracker() { - if [ -x "${BIN_DIR}/firecracker" ] && "${BIN_DIR}/firecracker" --version >/dev/null 2>&1; then - log "firecracker already installed: $("${BIN_DIR}/firecracker" --version | head -n1)" - return 0 - fi - - log "Resolving latest firecracker release tag from GitHub API..." - local tag - tag="$(curl -fsSL "${GITHUB_API}" | jq -r '.tag_name')" - [ -n "${tag}" ] && [ "${tag}" != "null" ] || die "could not resolve firecracker release tag" - log "Latest firecracker tag: ${tag}" - - local tmp - tmp="$(mktemp -d)" - # Ensure temp dir is cleaned up on any exit from this function's subshell scope. - trap 'rm -rf "${tmp}"' RETURN - - local tgz="${tmp}/firecracker.tgz" - local url="https://github.com/firecracker-microvm/firecracker/releases/download/${tag}/firecracker-${tag}-${ARCH}.tgz" - log "Downloading ${url}" - curl -fSL --retry 3 -o "${tgz}" "${url}" || die "failed to download firecracker release tarball" - - log "Extracting release tarball..." - tar -xzf "${tgz}" -C "${tmp}" - - # Layout inside tarball: release--/firecracker-- and jailer-- - local rel_dir="${tmp}/release-${tag}-${ARCH}" - local fc_bin="${rel_dir}/firecracker-${tag}-${ARCH}" - local jail_bin="${rel_dir}/jailer-${tag}-${ARCH}" - - # Fall back to a glob search if the expected path differs. - if [ ! -f "${fc_bin}" ]; then - fc_bin="$(find "${tmp}" -type f -name "firecracker-*-${ARCH}" ! -name '*.debug' | head -n1)" - fi - if [ ! -f "${jail_bin}" ]; then - jail_bin="$(find "${tmp}" -type f -name "jailer-*-${ARCH}" ! -name '*.debug' | head -n1)" - fi - - [ -f "${fc_bin}" ] || die "firecracker binary not found in release tarball" - [ -f "${jail_bin}" ] || warn "jailer binary not found in release tarball (continuing without jailer)" + local extracted="${ASSET_WORK}/firecracker" + mkdir -p "${extracted}" + log "Installing the retained signed-release Firecracker archive..." + printf '%s %s\n' "${NEHEMIAH_FIRECRACKER_SHA256}" "${NEHEMIAH_FIRECRACKER_ARCHIVE}" \ + | sha256sum --check --strict --status \ + || die "signed release Firecracker checksum verification failed" + python3 - "${NEHEMIAH_FIRECRACKER_ARCHIVE}" "${extracted}" <<'PY' +import pathlib +import sys +import tarfile + +archive, destination = sys.argv[1:] +with tarfile.open(archive, "r:gz") as bundle: + members = bundle.getmembers() + if not members or len(members) > 128 or sum(member.size for member in members) > 256 * 1024 * 1024: + raise SystemExit("Firecracker archive exceeds extraction bounds") + seen = set() + for member in members: + path = pathlib.PurePosixPath(member.name) + canonical = str(path) + if ( + path.is_absolute() + or ".." in path.parts + or canonical in seen + or not (member.isdir() or member.isfile()) + ): + raise SystemExit("Firecracker archive contains an unsafe entry") + seen.add(canonical) + bundle.extractall(destination, members=members, filter="data") +PY + + local -a firecracker_bins jailer_bins + mapfile -t firecracker_bins < <(find "${extracted}" -type f -name "firecracker-*-${ARCH}" ! -name '*.debug' | sort) + mapfile -t jailer_bins < <(find "${extracted}" -type f -name "jailer-*-${ARCH}" ! -name '*.debug' | sort) + [ "${#firecracker_bins[@]}" -eq 1 ] || die "Firecracker archive has an unexpected binary set" + [ "${#jailer_bins[@]}" -eq 1 ] || die "Firecracker archive has an unexpected jailer set" + local fc_bin="${firecracker_bins[0]}" + local jail_bin="${jailer_bins[0]}" + [[ "$(sha256sum "$fc_bin" | cut -d' ' -f1)" == "$NEHEMIAH_FIRECRACKER_INSTALLED_SHA256" ]] \ + || die "installed Firecracker digest does not match the signed cohort" + [[ "$(sha256sum "$jail_bin" | cut -d' ' -f1)" == "$NEHEMIAH_JAILER_INSTALLED_SHA256" ]] \ + || die "installed jailer digest does not match the signed cohort" + local expected_machine="x86-64" + [[ "${ARCH}" == aarch64 ]] && expected_machine="ARM aarch64" + for binary in "${fc_bin}" "${jail_bin}"; do + local description + description="$(file -b "${binary}")" + [[ "${description}" == *"ELF 64-bit LSB"* && \ + "${description}" == *"${expected_machine}"* ]] \ + || die "Firecracker archive contains a wrong-architecture binary" + done install -m 0755 "${fc_bin}" "${BIN_DIR}/firecracker" - [ -f "${jail_bin}" ] && install -m 0755 "${jail_bin}" "${BIN_DIR}/jailer" + install -m 0755 "${jail_bin}" "${BIN_DIR}/jailer" log "firecracker installed: $("${BIN_DIR}/firecracker" --version | head -n1)" } install_firecracker # -------------------------------------------------------------------------- -# 5. Fetch an uncompressed firecracker-compatible kernel +# 5. Install the retained uncompressed Firecracker-compatible kernel # -------------------------------------------------------------------------- kernel_is_valid() { local path="$1" [ -s "${path}" ] || return 1 - # Accept ELF or "Linux kernel x86 boot executable" (vmlinux.bin bzImage form). local desc desc="$(file -b "${path}" 2>/dev/null || true)" - case "${desc}" in - *ELF*) return 0 ;; - *"Linux kernel x86 boot"*) return 0 ;; - *boot*executable*) return 0 ;; - *) return 1 ;; - esac + if [[ "${ARCH}" == x86_64 ]]; then + [[ "${desc}" == *"Linux kernel x86 boot executable"* || \ + ( "${desc}" == *"ELF 64-bit LSB"* && "${desc}" == *"x86-64"* ) ]] + else + [[ "${desc}" == *"Linux kernel ARM64 boot executable"* || \ + ( "${desc}" == *"ELF 64-bit LSB"* && "${desc}" == *"ARM aarch64"* ) ]] + fi } install_kernel() { - if kernel_is_valid "${KERNEL_PATH}"; then - log "Kernel already present and valid: ${KERNEL_PATH} ($(file -b "${KERNEL_PATH}"))" - return 0 - fi - - local url - for url in "${KERNEL_URLS[@]}"; do - log "Trying kernel URL: ${url}" - local tmp - tmp="$(mktemp)" - if curl -fSL --retry 2 -o "${tmp}" "${url}"; then - if kernel_is_valid "${tmp}"; then - install -m 0644 "${tmp}" "${KERNEL_PATH}" - rm -f "${tmp}" - log "Kernel installed from ${url}: $(file -b "${KERNEL_PATH}")" - return 0 - else - warn "Downloaded file did not look like a kernel: $(file -b "${tmp}")" - fi - else - warn "Download failed: ${url}" - fi - rm -f "${tmp}" - done - - die "Could not fetch a valid kernel from any candidate URL. - Please supply a firecracker-compatible uncompressed kernel at: - ${KERNEL_PATH} - (e.g. build one with the firecracker kernel config, or copy a known-good vmlinux.)" + log "Installing the retained signed-release guest kernel..." + printf '%s %s\n' "${NEHEMIAH_KERNEL_SHA256}" "${NEHEMIAH_KERNEL_IMAGE}" \ + | sha256sum --check --strict --status \ + || die "signed release kernel checksum verification failed" + kernel_is_valid "${NEHEMIAH_KERNEL_IMAGE}" \ + || die "signed release kernel has an invalid architecture or executable format" + install -m 0644 "${NEHEMIAH_KERNEL_IMAGE}" "${KERNEL_PATH}" + log "Kernel installed: $(file -b "${KERNEL_PATH}")" } install_kernel # -------------------------------------------------------------------------- -# 6. Build base rootfs +# 6. Require both signed guest images # -------------------------------------------------------------------------- -log "Building base rootfs (build-rootfs.sh)..." -bash "${SCRIPT_DIR}/build-rootfs.sh" +rootfs_is_ext4() { + local image="$1" magic + [[ -f "$image" && ! -L "$image" && -s "$image" ]] || return 1 + magic="$(dd if="$image" bs=1 skip=1080 count=2 status=none | od -An -tx1 | tr -d ' \n')" + [[ "$magic" == 53ef ]] || return 1 + file -b "$image" | grep -q 'ext4 filesystem data' +} +for image_and_digest in \ + "${ROOTFS_DIR}/rootfs.ext4:${NEHEMIAH_RUNTIME_PYTHON_SHA256}" \ + "${ROOTFS_DIR}/desktop.ext4:${NEHEMIAH_RUNTIME_DESKTOP_SHA256}"; do + image="${image_and_digest%%:*}" + expected_digest="${image_and_digest##*:}" + rootfs_is_ext4 "$image" \ + || die "required signed guest image is missing or invalid: $image" + [[ "$(sha256sum "$image" | cut -d' ' -f1)" == "$expected_digest" ]] \ + || die "required signed guest image checksum mismatch: $image" + e2fsck -fn "$image" >/dev/null \ + || die "required signed guest image failed filesystem validation: $image" +done # -------------------------------------------------------------------------- # 7. Success banner @@ -228,18 +326,9 @@ cat </dev/null | head -n1 || echo "(not installed)") kernel : ${KERNEL_PATH} ($(file -b "${KERNEL_PATH}")) rootfs : ${ROOTFS_DIR}/rootfs.ext4 ($(du -h "${ROOTFS_DIR}/rootfs.ext4" 2>/dev/null | cut -f1)) + desktop : ${ROOTFS_DIR}/desktop.ext4 ($(du -h "${ROOTFS_DIR}/desktop.ext4" 2>/dev/null | cut -f1)) run dir : ${RUN_DIR} templates : ${TEMPLATE_DIR} ----------------------------------------------------------------------------- - NEXT STEPS: - 1. (optional) Build the python snapshot template: - sudo bash ${SCRIPT_DIR}/build-template.sh python - 2. Deploy the nehemiahd binary to /usr/local/bin/nehemiahd - 3. (optional) Set a token: echo 'NEHEMIAH_TOKEN=...' > /etc/boring/nehemiahd.env - 4. Install the service: - install -m0644 ${SCRIPT_DIR}/nehemiahd.service /etc/systemd/system/nehemiahd.service - systemctl daemon-reload && systemctl enable --now nehemiahd - 5. Verify: curl -s http://localhost:8080/healthz | jq ============================================================================ BANNER diff --git a/infra/latitude/boring-net-local.service b/infra/latitude/boring-net-local.service new file mode 100644 index 0000000..d537447 --- /dev/null +++ b/infra/latitude/boring-net-local.service @@ -0,0 +1,15 @@ +[Unit] +Description=Nehemiah local/prototype guest networking +After=network-online.target dnsmasq.service +Wants=network-online.target +Before=nehemiahd.service + +[Service] +Type=oneshot +# Local development keeps the historical optional environment and mode. +EnvironmentFile=-/etc/boring/nehemiahd.env +ExecStart=/opt/boring/bin/net-setup.sh +RemainAfterExit=yes + +[Install] +WantedBy=multi-user.target diff --git a/infra/latitude/boring-net.service b/infra/latitude/boring-net.service index 9791b2c..28737ef 100644 --- a/infra/latitude/boring-net.service +++ b/infra/latitude/boring-net.service @@ -1,12 +1,20 @@ [Unit] -Description=Nehemiah guest networking (bridge, dnsmasq, NAT, egress firewall) +Description=Nehemiah guest networking (bridge, DHCP, NAT, egress firewall) After=network-online.target dnsmasq.service Wants=network-online.target Before=nehemiahd.service +AssertPathExists=/etc/boring/managed-host +AssertPathExists=/etc/boring/nehemiahd.env [Service] Type=oneshot -ExecStart=/opt/boring/bin/net-setup.sh +# The signed managed unit fails rather than silently falling back to prototype +# defaults. The local unit remains intentionally permissive for development. +EnvironmentFile=/etc/boring/nehemiahd.env +ExecStartPre=/opt/boring/bin/managed-host-preflight.sh +ExecStart=/usr/bin/env NEHEMIAH_MODE=1 /opt/boring/bin/net-setup.sh +ExecStop=/usr/bin/env NEHEMIAH_MODE=1 /opt/boring/bin/net-setup.sh --fail-closed +ExecStopPost=/usr/bin/env NEHEMIAH_MODE=1 /opt/boring/bin/net-setup.sh --fail-closed RemainAfterExit=yes [Install] diff --git a/infra/latitude/build-desktop-rootfs.sh b/infra/latitude/build-desktop-rootfs.sh index 4f52980..34ea78a 100644 --- a/infra/latitude/build-desktop-rootfs.sh +++ b/infra/latitude/build-desktop-rootfs.sh @@ -1,6 +1,8 @@ #!/usr/bin/env bash # -# build-desktop-rootfs.sh - Build the "desktop" ext4 rootfs for Nehemiah. +# build-desktop-rootfs.sh - LOCAL DEVELOPMENT ONLY: build a mutable desktop. +# Managed hosts install the signed, scanned desktop ext4 release artifact and +# never run this script or resolve these mutable apt/npm/curl inputs. # # A minimal Debian rootfs that boots straight into a headless X session # (Xvfb + openbox + xterm + xclock) served over VNC, and bridges the VNC port @@ -16,6 +18,7 @@ NEHEMIAH_ROOT="/opt/boring" ROOTFS_DIR="${NEHEMIAH_ROOT}/rootfs" IMG="${ROOTFS_DIR}/desktop.ext4" IMG_SIZE_MB="${IMG_SIZE_MB:-6144}" # room for chromium + node + coding agents +GUEST_AGENT_BIN="${GUEST_AGENT_BIN:-${NEHEMIAH_ROOT}/bin/bc-guest-agent}" SUITE="${SUITE:-bookworm}" MIRROR="${MIRROR:-http://deb.debian.org/debian}" # xcalc/xclock come from x11-apps; figlet for banners. (galculator was dropped — @@ -26,6 +29,9 @@ PKGS="xvfb,x11vnc,openbox,xterm,x11-xserver-utils,xfonts-base,x11-apps,fonts-lib log() { printf '\033[1;34m[desktop]\033[0m %s\n' "$*"; } die() { printf '\033[1;31m[desktop:error]\033[0m %s\n' "$*" >&2; exit 1; } [ "$(id -u)" -eq 0 ] || die "must run as root" +[[ "${NEHEMIAH_MANAGED_ROOTFS:-0}" == 0 && "${NEHEMIAH_MODE:-0}" != 1 ]] \ + || die "managed desktop builds are forbidden; install the signed release image" +[ -x "${GUEST_AGENT_BIN}" ] || die "guest agent not found at ${GUEST_AGENT_BIN} (build guest-agent first)" WORK="$(mktemp -d /tmp/boring-desktop.XXXXXX)" MNT="${WORK}/mnt" @@ -51,6 +57,7 @@ mount -o loop "${IMG}" "${MNT}" log "debootstrap ${SUITE} (minbase + desktop packages)..." debootstrap --variant=minbase --include="${PKGS}" "${SUITE}" "${MNT}" "${MIRROR}" \ || die "debootstrap failed" +install -D -m0755 "${GUEST_AGENT_BIN}" "${MNT}/usr/local/sbin/bc-guest-agent" log "Configuring guest (chromium + node 22 + coding agents)..." cp -f /etc/resolv.conf "${MNT}/etc/resolv.conf" # DNS for apt/npm inside chroot @@ -106,8 +113,17 @@ mkdir -p /tmp/.X11-unix && chmod 1777 /tmp/.X11-unix hostname boring 2>/dev/null # Firecracker guests boot with loopback DOWN; x11vnc + socat talk over 127.0.0.1. ip link set lo up 2>/dev/null || true -# eth0 is configured by the kernel (ip=dhcp); publish its DNS so the browser works. -cat /proc/net/pnp > /etc/resolv.conf 2>/dev/null || echo "nameserver 1.1.1.1" > /etc/resolv.conf +# eth0 is configured by the kernel (ip=dhcp). Keep DNS on the managed bridge so +# every answer passes through nehemiahd's policy binding; never fall back to a +# direct public resolver. /proc/net/pnp is primary, with the DHCP default gateway +# (or the deployment default) used only when the kernel did not publish it. +if [ -s /proc/net/pnp ]; then + cat /proc/net/pnp > /etc/resolv.conf +else + DNS_GATEWAY=$(ip -4 route show default | awk '$1 == "default" { print $3; exit }') + [ -n "$DNS_GATEWAY" ] || DNS_GATEWAY=10.200.0.1 + printf 'nameserver %s\n' "$DNS_GATEWAY" > /etc/resolv.conf +fi export HOME=/root DISPLAY=:0 Xvfb :0 -screen 0 1280x800x24 -ac -nolisten tcp >/var/log/xvfb.log 2>&1 & @@ -137,7 +153,21 @@ x11vnc -display :0 -forever -shared -nopw -rfbport 5900 -noxdamage -threads -def # Bridge guest vsock port 5900 -> local VNC. The host connects via the vsock UDS. socat VSOCK-LISTEN:5900,fork,reuseaddr TCP:127.0.0.1:5900 >/var/log/socat.log 2>&1 & -echo NEHEMIAH_READY > /dev/ttyS0 +# Keep the control agent alive independently of the desktop processes. Only +# report serial readiness once the agent has actually bound its vsock port. +( + while true; do + rm -f /run/bc-guest-agent.ready + /usr/local/sbin/bc-guest-agent >>/var/log/bc-guest-agent.log 2>&1 + sleep 1 + done +) & +i=0; while [ ! -e /run/bc-guest-agent.ready ] && [ "$i" -lt 100 ]; do i=$((i+1)); sleep 0.05; done +if [ -e /run/bc-guest-agent.ready ]; then + echo NEHEMIAH_READY > /dev/ttyS0 +else + echo "guest agent failed to become ready" > /dev/ttyS0 +fi exec /bin/sh INIT_EOF chmod +x "${MNT}/sbin/boring-init" diff --git a/infra/latitude/build-rootfs.sh b/infra/latitude/build-rootfs.sh index cc82a99..f212793 100755 --- a/infra/latitude/build-rootfs.sh +++ b/infra/latitude/build-rootfs.sh @@ -1,10 +1,12 @@ #!/usr/bin/env bash # -# build-rootfs.sh - Build the base Alpine ext4 rootfs for boring microVMs. +# build-rootfs.sh - LOCAL DEVELOPMENT ONLY: build a mutable Alpine rootfs. +# Managed hosts must install signed release ext4 artifacts through cloud-init; +# this script deliberately refuses the old managed/minimal profile. # # Produces /opt/boring/rootfs/rootfs.ext4 : # * ~512MB ext4 image -# * Alpine minirootfs (busybox init) with python3 installed +# * Caller-provided Alpine minirootfs plus mutable local apk/npm packages # * /etc/inittab that boots an interactive /bin/sh on ttyS0 and prints # the "NEHEMIAH_READY" marker (required by nehemiahd for boot_ms timing) # @@ -18,11 +20,13 @@ set -euo pipefail NEHEMIAH_ROOT="/opt/boring" ROOTFS_DIR="${NEHEMIAH_ROOT}/rootfs" IMG="${ROOTFS_DIR}/rootfs.ext4" -IMG_SIZE_MB="${IMG_SIZE_MB:-1280}" # room for the Claude Code CLI +IMG_SIZE_MB="${IMG_SIZE_MB:-1280}" # room for opt-in local development packages +GUEST_AGENT_BIN="${GUEST_AGENT_BIN:-${NEHEMIAH_ROOT}/bin/bc-guest-agent}" ALPINE_MIRROR="https://dl-cdn.alpinelinux.org/alpine" -ALPINE_BRANCH="${ALPINE_BRANCH:-v3.20}" # 3.x series -ALPINE_ARCH="$(uname -m)" # x86_64 / aarch64 — matches Alpine's arch naming +ALPINE_BRANCH="${ALPINE_BRANCH:-v3.20}" +ALPINE_MINIROOTFS_TARBALL="${ALPINE_MINIROOTFS_TARBALL:-}" +NEHEMIAH_MANAGED_ROOTFS="${NEHEMIAH_MANAGED_ROOTFS:-0}" # -------------------------------------------------------------------------- # Logging helpers @@ -32,6 +36,11 @@ warn() { printf '\033[1;33m[rootfs:warn]\033[0m %s\n' "$*" >&2; } die() { printf '\033[1;31m[rootfs:error]\033[0m %s\n' "$*" >&2; exit 1; } [ "$(id -u)" -eq 0 ] || die "must run as root" +[ -x "${GUEST_AGENT_BIN}" ] || die "guest agent not found at ${GUEST_AGENT_BIN} (build guest-agent first)" +[ -f "${ALPINE_MINIROOTFS_TARBALL}" ] \ + || die "ALPINE_MINIROOTFS_TARBALL must name a previously checksum-verified archive" +[[ "${NEHEMIAH_MANAGED_ROOTFS}" == 0 ]] \ + || die "managed rootfs builds are forbidden; install signed release images via cloud-init" # -------------------------------------------------------------------------- # Working state + cleanup trap @@ -56,25 +65,7 @@ cleanup() { trap cleanup EXIT INT TERM # -------------------------------------------------------------------------- -# 1. Resolve + download alpine minirootfs -# -------------------------------------------------------------------------- -log "Resolving latest alpine-minirootfs for ${ALPINE_BRANCH}/${ALPINE_ARCH}..." -RELEASES_URL="${ALPINE_MIRROR}/${ALPINE_BRANCH}/releases/${ALPINE_ARCH}" - -# Parse the release index for the newest alpine-minirootfs-*.tar.gz filename. -TARBALL="$(curl -fsSL "${RELEASES_URL}/" \ - | grep -oE 'alpine-minirootfs-[0-9][0-9.]*-'"${ALPINE_ARCH}"'\.tar\.gz' \ - | sort -V | tail -n1)" -[ -n "${TARBALL}" ] || die "could not find an alpine-minirootfs tarball at ${RELEASES_URL}/" -log "Selected: ${TARBALL}" - -MINIROOT_TGZ="${WORK}/${TARBALL}" -log "Downloading ${RELEASES_URL}/${TARBALL}" -curl -fSL --retry 3 -o "${MINIROOT_TGZ}" "${RELEASES_URL}/${TARBALL}" \ - || die "failed to download alpine minirootfs" - -# -------------------------------------------------------------------------- -# 2. Create + format ext4 image +# 1. Create + format ext4 image # -------------------------------------------------------------------------- mkdir -p "${ROOTFS_DIR}" log "Creating ${IMG_SIZE_MB}MB ext4 image at ${IMG}..." @@ -87,41 +78,36 @@ log "Mounting image..." mount -o loop "${IMG}" "${MNT}" # -------------------------------------------------------------------------- -# 3. Extract minirootfs +# 2. Extract the checksum-verified minirootfs # -------------------------------------------------------------------------- log "Extracting minirootfs into image..." -tar -xzf "${MINIROOT_TGZ}" -C "${MNT}" +tar -xzf "${ALPINE_MINIROOTFS_TARBALL}" -C "${MNT}" +install -D -m0755 "${GUEST_AGENT_BIN}" "${MNT}/usr/local/sbin/bc-guest-agent" # -------------------------------------------------------------------------- -# 4. Configure inside a chroot +# 3. Configure inside a chroot # -------------------------------------------------------------------------- -log "Configuring guest (resolv.conf, python3, inittab, root passwd)..." +log "Configuring guest (resolv.conf, inittab, root passwd)..." # DNS for apk inside the chroot. cp -f /etc/resolv.conf "${MNT}/etc/resolv.conf" -# Configure apk repositories explicitly (main + community) so python3 resolves. -mkdir -p "${MNT}/etc/apk" -cat > "${MNT}/etc/apk/repositories" < "${MNT}/etc/apk/repositories" </dev/null || true umount "${MNT}/dev" 2>/dev/null || true # -------------------------------------------------------------------------- -# 5. inittab - busybox init reads this. NEHEMIAH_READY marker is REQUIRED. +# 4. Guest-agent supervisor + inittab. The serial marker is emitted only after +# the guest agent has bound AF_VSOCK and written its readiness file. # -------------------------------------------------------------------------- +log "Writing guest-agent supervisor..." +cat > "${MNT}/sbin/bc-guest-agent-supervisor" <<'SUPERVISOR_EOF' +#!/bin/sh +while true; do + rm -f /run/bc-guest-agent.ready + /usr/local/sbin/bc-guest-agent >>/var/log/bc-guest-agent.log 2>&1 + sleep 1 +done +SUPERVISOR_EOF +chmod +x "${MNT}/sbin/bc-guest-agent-supervisor" + log "Writing /etc/inittab..." cat > "${MNT}/etc/inittab" <<'INITTAB_EOF' ::sysinit:/bin/mount -t proc proc /proc ::sysinit:/bin/mount -t sysfs sysfs /sys ::sysinit:/bin/mount -t devtmpfs devtmpfs /dev +::sysinit:/bin/mkdir -p /dev/pts /run /var/log +::sysinit:/bin/mount -t devpts devpts /dev/pts ::sysinit:/bin/hostname boring -::sysinit:/bin/sh -c 'echo NEHEMIAH_READY > /dev/ttyS0' +::sysinit:/bin/sh -c '/sbin/bc-guest-agent-supervisor &' +::sysinit:/bin/sh -c 'i=0; while [ ! -e /run/bc-guest-agent.ready ] && [ "$i" -lt 100 ]; do i=$((i+1)); sleep 0.05; done; [ -e /run/bc-guest-agent.ready ] && echo NEHEMIAH_READY > /dev/ttyS0 || echo "guest agent failed to become ready" > /dev/ttyS0' ttyS0::respawn:/bin/sh -l ::ctrlaltdel:/sbin/reboot ::shutdown:/bin/umount -a -r @@ -150,7 +151,7 @@ INITTAB_EOF echo "boring" > "${MNT}/etc/hostname" # -------------------------------------------------------------------------- -# 6. Unmount cleanly (trap will also handle this on failure) +# 5. Unmount cleanly (trap will also handle this on failure) # -------------------------------------------------------------------------- log "Syncing and unmounting..." sync diff --git a/infra/latitude/build-template.sh b/infra/latitude/build-template.sh index 27f73f7..85fe0e0 100755 --- a/infra/latitude/build-template.sh +++ b/infra/latitude/build-template.sh @@ -49,6 +49,7 @@ die() { printf '\033[1;31m[template:error]\033[0m %s\n' "$*" >&2; exit 1; } # -------------------------------------------------------------------------- WORK="$(mktemp -d /tmp/boring-template.XXXXXX)" SOCK="${WORK}/fc.sock" +VSOCK="${WORK}/vsock" STDOUT_LOG="${WORK}/console.log" # Boot (and snapshot) from the template's OWN stable rootfs path so the path # baked into the snapshot still exists at restore time. nehemiahd restores by @@ -132,13 +133,15 @@ wait_for_socket || die "firecracker API socket did not appear" # -------------------------------------------------------------------------- # 3. Configure the VM via the API # -------------------------------------------------------------------------- -log "Configuring boot-source, drive, machine-config..." +log "Configuring boot-source, drive, machine-config, and guest-agent vsock..." api PUT /boot-source \ "{\"kernel_image_path\":\"${KERNEL}\",\"boot_args\":\"${BOOT_ARGS}\"}" >/dev/null api PUT /drives/rootfs \ "{\"drive_id\":\"rootfs\",\"path_on_host\":\"${WORK_ROOTFS}\",\"is_root_device\":true,\"is_read_only\":false}" >/dev/null api PUT /machine-config \ "{\"vcpu_count\":${VCPU_COUNT},\"mem_size_mib\":${MEM_MIB}}" >/dev/null +api PUT /vsock \ + "{\"guest_cid\":3,\"uds_path\":\"${VSOCK}\"}" >/dev/null log "Starting instance..." api PUT /actions '{"action_type":"InstanceStart"}' >/dev/null diff --git a/infra/latitude/cloud-init.sh b/infra/latitude/cloud-init.sh new file mode 100755 index 0000000..e36c931 --- /dev/null +++ b/infra/latitude/cloud-init.sh @@ -0,0 +1,565 @@ +#!/usr/bin/env bash +# Provider-neutral cloud-init payload for an approved managed host. The manual +# renderer writes this script and a private bootstrap environment into cloud-init. +set -euo pipefail +set +x +umask 077 + +BOOTSTRAP_ENV=/etc/nehemiah/bootstrap.env +DAEMON_ENV=/etc/boring/nehemiahd.env +MANAGED_HOST_MARKER=/etc/boring/managed-host +if [[ ! -f "$BOOTSTRAP_ENV" || ! -r "$BOOTSTRAP_ENV" ]]; then + echo "missing $BOOTSTRAP_ENV" >&2 + exit 1 +fi +if [[ -L "$BOOTSTRAP_ENV" || "$(stat -c '%a:%u' "$BOOTSTRAP_ENV")" != 600:0 ]]; then + echo "$BOOTSTRAP_ENV must be a root-owned regular mode-0600 file" >&2 + exit 1 +fi +# shellcheck disable=SC1090 +source "$BOOTSTRAP_ENV" + +: "${NEHEMIAH_RELEASE_BASE:?required}" +: "${NEHEMIAH_RELEASE_VERSION:?required}" +: "${NEHEMIAH_RELEASE_MINISIGN_KEY:?required}" +: "${NEHEMIAH_HOST_ID:?required}" +: "${NEHEMIAH_REGION:?required}" +: "${LATITUDE_OS_ID:?required}" +: "${LATITUDE_OS_SLUG:?required}" +: "${LATITUDE_OS_VERSION:?required}" +: "${LATITUDE_OS_ARCH:?required}" +: "${NEHEMIAH_FLEET_BOOTSTRAP_TOKEN:?required}" +: "${NEHEMIAH_CONTROL_PLANE_URL:?required}" +: "${NEHEMIAH_ADVERTISE_ADDRESS:?required}" +: "${NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS:?required}" +: "${NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS:?required}" +: "${NEHEMIAH_WIREGUARD_CONFIG_B64:?required}" +: "${NEHEMIAH_OTEL_ENABLED:?required}" +: "${NEHEMIAH_OTEL_ENDPOINT:?required}" +: "${NEHEMIAH_OTEL_AUTHORIZATION:?required}" +: "${NEHEMIAH_SERVICE_VERSION:?required}" +: "${NEHEMIAH_INSTANCE_ID:?required}" +: "${NEHEMIAH_DEPLOYMENT_ENVIRONMENT:?required}" +: "${NEHEMIAH_OTEL_EXPORT_INTERVAL_MS:?required}" +: "${NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS:?required}" +: "${NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO:?required}" +NEHEMIAH_TEMPLATE_OBJECT_ORIGIN="${NEHEMIAH_TEMPLATE_OBJECT_ORIGIN:-}" + +[[ "$NEHEMIAH_RELEASE_VERSION" =~ ^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-[0-9A-Za-z.-]+)?(\+[0-9A-Za-z.-]+)?$ ]] \ + || { echo "invalid release version" >&2; exit 1; } +[[ "$NEHEMIAH_RELEASE_MINISIGN_KEY" =~ ^RW[A-Za-z0-9+/]{54}$ ]] \ + || { echo "invalid release minisign public key" >&2; exit 1; } +[[ "$NEHEMIAH_HOST_ID" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ ]] \ + || { echo "invalid host id" >&2; exit 1; } +[[ "$NEHEMIAH_REGION" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ ]] \ + || { echo "invalid region" >&2; exit 1; } +[[ "$LATITUDE_OS_ID" =~ ^os_[A-Za-z0-9_-]{4,128}$ ]] \ + || { echo "invalid Latitude operating-system id" >&2; exit 1; } +[[ "$LATITUDE_OS_SLUG" =~ ^ubuntu_24_04_(x64|arm64)_lts$ ]] \ + || { echo "invalid Latitude operating-system slug" >&2; exit 1; } +[[ "$LATITUDE_OS_VERSION" =~ ^24\.04([.[:space:]A-Za-z0-9_-]{0,63})$ ]] \ + || { echo "invalid Latitude operating-system version" >&2; exit 1; } +case "$LATITUDE_OS_ARCH:$LATITUDE_OS_SLUG" in + amd64:ubuntu_24_04_x64_lts | arm64:ubuntu_24_04_arm64_lts) ;; + *) echo "Latitude operating-system architecture mismatch" >&2; exit 1 ;; +esac +[[ "$NEHEMIAH_FLEET_BOOTSTRAP_TOKEN" =~ ^nhe_[A-Za-z0-9_-]{43}$ ]] \ + || { echo "invalid fleet bootstrap credential" >&2; exit 1; } +[[ "$NEHEMIAH_OTEL_ENABLED" == true ]] \ + || { echo "NEHEMIAH_OTEL_ENABLED must be exactly true" >&2; exit 1; } +[[ ${#NEHEMIAH_OTEL_AUTHORIZATION} -ge 16 && \ + ${#NEHEMIAH_OTEL_AUTHORIZATION} -le 4096 && \ + "$NEHEMIAH_OTEL_AUTHORIZATION" =~ ^[A-Za-z][A-Za-z0-9_-]{0,31}\ [-A-Za-z0-9._~+/=]+$ ]] \ + || { echo "invalid OTLP authorization value" >&2; exit 1; } +for identity in "$NEHEMIAH_SERVICE_VERSION" "$NEHEMIAH_INSTANCE_ID"; do + [[ "$identity" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ ]] \ + || { echo "invalid telemetry identity" >&2; exit 1; } +done +case "$NEHEMIAH_DEPLOYMENT_ENVIRONMENT" in + staging | production) ;; + *) echo "invalid managed telemetry environment" >&2; exit 1 ;; +esac +[[ "$NEHEMIAH_OTEL_EXPORT_INTERVAL_MS" =~ ^[1-9][0-9]*$ && \ + $NEHEMIAH_OTEL_EXPORT_INTERVAL_MS -ge 5000 && \ + $NEHEMIAH_OTEL_EXPORT_INTERVAL_MS -le 300000 ]] \ + || { echo "invalid OTLP export interval" >&2; exit 1; } +[[ "$NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS" =~ ^[1-9][0-9]*$ && \ + $NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS -ge 1000 && \ + $NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS -le 30000 && \ + $NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS -lt $NEHEMIAH_OTEL_EXPORT_INTERVAL_MS ]] \ + || { echo "invalid OTLP export timeout" >&2; exit 1; } +python3 - "$NEHEMIAH_RELEASE_BASE" "$NEHEMIAH_CONTROL_PLANE_URL" \ + "$NEHEMIAH_ADVERTISE_ADDRESS" "$NEHEMIAH_TEMPLATE_OBJECT_ORIGIN" \ + "$NEHEMIAH_OTEL_ENDPOINT" "$NEHEMIAH_REGION" "$NEHEMIAH_INSTANCE_ID" \ + "$NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO" <<'PY' +import ipaddress +import math +import re +import sys +from urllib.parse import urlsplit + +for label, raw in (("release base", sys.argv[1]), ("control-plane URL", sys.argv[2])): + if not re.fullmatch(r"https://[A-Za-z0-9.-]+(?::[0-9]{1,5})?(?:/[A-Za-z0-9._~/-]*)?", raw): + raise SystemExit(f"{label} must be a simple HTTPS URL without credentials, query, or fragment") + parsed = urlsplit(raw) + if parsed.username or parsed.password or parsed.query or parsed.fragment or not parsed.hostname: + raise SystemExit(f"invalid {label}") + try: + if parsed.port is not None and not 1 <= parsed.port <= 65535: + raise ValueError + except ValueError: + raise SystemExit(f"invalid {label} port") from None +ipaddress.ip_address(sys.argv[3]) +if sys.argv[4]: + parsed = urlsplit(sys.argv[4]) + if ( + parsed.scheme != "https" + or not parsed.hostname + or parsed.username + or parsed.password + or parsed.path not in ("", "/") + or parsed.query + or parsed.fragment + ): + raise SystemExit("template object origin must be an origin-only HTTPS URL") + try: + if parsed.port is not None and not 1 <= parsed.port <= 65535: + raise ValueError + except ValueError: + raise SystemExit("invalid template object origin port") from None + +otel_endpoint = urlsplit(sys.argv[5]) +if ( + otel_endpoint.scheme != "https" + or not otel_endpoint.hostname + or otel_endpoint.username + or otel_endpoint.password + or otel_endpoint.path not in ("", "/") + or otel_endpoint.query + or otel_endpoint.fragment +): + raise SystemExit("OTLP endpoint must be an origin-only HTTPS URL") +try: + ipaddress.ip_address(otel_endpoint.hostname) +except ValueError: + pass +else: + raise SystemExit("OTLP endpoint must use a DNS hostname") +try: + if otel_endpoint.port is not None and not 1 <= otel_endpoint.port <= 65535: + raise ValueError +except ValueError: + raise SystemExit("invalid OTLP endpoint port") from None +for label, identity in (("region", sys.argv[6]), ("instance id", sys.argv[7])): + try: + ipaddress.ip_address(identity) + except ValueError: + pass + else: + raise SystemExit(f"{label} must not be an IP address") +try: + sample_ratio = float(sys.argv[8]) +except ValueError: + raise SystemExit("invalid OTLP trace sample ratio") from None +if not math.isfinite(sample_ratio) or not 0.001 <= sample_ratio <= 1: + raise SystemExit("invalid OTLP trace sample ratio") +PY +[[ "$NEHEMIAH_WIREGUARD_CONFIG_B64" =~ ^[A-Za-z0-9+/]*={0,2}$ && \ + ${#NEHEMIAH_WIREGUARD_CONFIG_B64} -le 131072 ]] \ + || { echo "invalid WireGuard configuration encoding" >&2; exit 1; } + +NEHEMIAH_PROVIDER_ID="${NEHEMIAH_PROVIDER_ID:-${NEHEMIAH_HOST_ID}}" +[[ "$NEHEMIAH_PROVIDER_ID" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ ]] \ + || { echo "invalid provider id" >&2; exit 1; } +# Generate both inbound credentials on this host. They are sent to the control +# plane during authenticated enrollment and never shared across hosts. Preserve +# them if cloud-init is deliberately re-run on an already-enrolled host. +existing_token() { + local wanted="$1" name value found="" + if [[ -f "$DAEMON_ENV" && ! -L "$DAEMON_ENV" ]]; then + while IFS='=' read -r name value; do + if [[ "$name" == "$wanted" ]]; then + found="$value" + fi + done < "$DAEMON_ENV" + fi + if [[ ${#found} -ge 32 && ${#found} -le 4096 && "$found" =~ ^[-A-Za-z0-9._~+/=]+$ ]]; then + printf '%s' "$found" + fi + return 0 +} +NEHEMIAH_INTERNAL_TOKEN="$(existing_token NEHEMIAH_INTERNAL_TOKEN)" +NEHEMIAH_HOST_GATEWAY_TOKEN="$(existing_token NEHEMIAH_TOKEN)" +NEHEMIAH_INTERNAL_TOKEN="${NEHEMIAH_INTERNAL_TOKEN:-$(od -An -N32 -tx1 /dev/urandom | tr -d ' \n')}" +NEHEMIAH_HOST_GATEWAY_TOKEN="${NEHEMIAH_HOST_GATEWAY_TOKEN:-$(od -An -N32 -tx1 /dev/urandom | tr -d ' \n')}" +NEHEMIAH_OTEL_CREDENTIAL="${NEHEMIAH_OTEL_AUTHORIZATION#* }" +[[ "$NEHEMIAH_INTERNAL_TOKEN" != "$NEHEMIAH_HOST_GATEWAY_TOKEN" && \ + "$NEHEMIAH_INTERNAL_TOKEN" != "$NEHEMIAH_FLEET_BOOTSTRAP_TOKEN" && \ + "$NEHEMIAH_HOST_GATEWAY_TOKEN" != "$NEHEMIAH_FLEET_BOOTSTRAP_TOKEN" && \ + "$NEHEMIAH_OTEL_AUTHORIZATION" != "$NEHEMIAH_FLEET_BOOTSTRAP_TOKEN" && \ + "$NEHEMIAH_OTEL_CREDENTIAL" != "$NEHEMIAH_FLEET_BOOTSTRAP_TOKEN" && \ + "$NEHEMIAH_OTEL_AUTHORIZATION" != "$NEHEMIAH_INTERNAL_TOKEN" && \ + "$NEHEMIAH_OTEL_CREDENTIAL" != "$NEHEMIAH_INTERNAL_TOKEN" && \ + "$NEHEMIAH_OTEL_AUTHORIZATION" != "$NEHEMIAH_HOST_GATEWAY_TOKEN" && \ + "$NEHEMIAH_OTEL_CREDENTIAL" != "$NEHEMIAH_HOST_GATEWAY_TOKEN" ]] \ + || { echo "managed host credentials must be distinct" >&2; exit 1; } +if [[ "$NEHEMIAH_ADVERTISE_ADDRESS" == *:* ]]; then + NEHEMIAH_LISTEN_ADDRESS="[${NEHEMIAH_ADVERTISE_ADDRESS}]:8080" +else + NEHEMIAH_LISTEN_ADDRESS="${NEHEMIAH_ADVERTISE_ADDRESS}:8080" +fi + +for bootstrap_command in apt-get bash curl dpkg dpkg-deb openssl python3 sha256sum stat tar uname; do + command -v "$bootstrap_command" >/dev/null 2>&1 \ + || { echo "provider base image is missing bootstrap command: $bootstrap_command" >&2; exit 1; } +done +case "$(uname -m)" in + x86_64) RELEASE_ARCH=amd64 ;; + aarch64) RELEASE_ARCH=arm64 ;; + *) echo "unsupported architecture: $(uname -m)" >&2; exit 1 ;; +esac +[[ "$LATITUDE_OS_ARCH" == "$RELEASE_ARCH" ]] \ + || { echo "provider image architecture does not match the running host" >&2; exit 1; } + +RELEASE_DIR=$(mktemp -d /var/tmp/nehemiah-release.XXXXXX) +trap 'rm -rf -- "$RELEASE_DIR"' EXIT +RELEASE_URL="${NEHEMIAH_RELEASE_BASE%/}/v${NEHEMIAH_RELEASE_VERSION}" +MANIFEST_URL="${RELEASE_URL}/SHA256SUMS" +curl --fail --silent --show-error --location --proto '=https' --tlsv1.2 --max-filesize 1048576 \ + "$MANIFEST_URL" -o "$RELEASE_DIR/SHA256SUMS" +curl --fail --silent --show-error --location --proto '=https' --tlsv1.2 --max-filesize 65536 \ + "${MANIFEST_URL}.minisig" -o "$RELEASE_DIR/SHA256SUMS.minisig" +/usr/local/libexec/nehemiah-verify-minisign \ + --message "$RELEASE_DIR/SHA256SUMS" \ + --signature "$RELEASE_DIR/SHA256SUMS.minisig" \ + --public-key "$NEHEMIAH_RELEASE_MINISIGN_KEY" +python3 - "$RELEASE_DIR/SHA256SUMS" "$NEHEMIAH_RELEASE_VERSION" <<'PY' +import pathlib +import re +import sys + +checksum_path, version = sys.argv[1:] +contents = pathlib.Path(checksum_path).read_text() +if not contents.endswith("\n") or "\r" in contents: + raise SystemExit("signed checksum metadata has invalid line endings") +entries = {} +for line in contents[:-1].split("\n"): + match = re.fullmatch(r"([0-9a-f]{64}) ([A-Za-z0-9][A-Za-z0-9._+-]{0,254})", line) + if not match or match.group(2) in entries: + raise SystemExit("signed checksum metadata is malformed or duplicated") + entries[match.group(2)] = match.group(1) +expected = {"release-manifest.json", "nehemiah.rb", f"nehemiah-cli-{version}.tgz", f"nehemiah-host-bootstrap_{version}.tar.gz"} +for component in ("nehemiahd", "bc-guest-agent", "bc-gateway"): + for arch in ("amd64", "arm64"): + expected.add(f"{component}_{version}_linux_{arch}.tar.gz") +for flavor in ("python", "desktop"): + for arch in ("amd64", "arm64"): + expected.add(f"nehemiah-guest-{flavor}_{version}_linux_{arch}.ext4.gz") +for arch in ("amd64", "arm64"): + expected.add(f"nehemiah-guest-scan_{version}_linux_{arch}.json") + expected.add(f"nehemiah-host-packages_{version}_ubuntu24.04_linux_{arch}.tar.gz") + expected.add(f"nehemiah-runtime-firecracker_1.15.1_linux_{arch}.tgz") + expected.add(f"nehemiah-runtime-kernel_6.1.155_linux_{arch}.bin") +if set(entries) != expected: + raise SystemExit("signed checksum metadata does not contain the exact release artifact set") +PY + +fetch_release_artifact() { + local artifact="$1" max_bytes="${2:-536870912}" matches + [[ "$artifact" =~ ^[A-Za-z0-9][A-Za-z0-9._+-]{0,254}$ ]] \ + || { echo "unsafe release artifact name" >&2; exit 1; } + matches="$(awk -v name="$artifact" '$2 == name { print }' "$RELEASE_DIR/SHA256SUMS")" + [[ "$(printf '%s\n' "$matches" | grep -c .)" -eq 1 && \ + "$matches" =~ ^[0-9a-f]{64}[[:space:]][[:space:]] ]] || { + echo "signed checksum metadata does not contain exactly one $artifact" >&2 + exit 1 + } + [[ "$max_bytes" =~ ^[1-9][0-9]*$ && "$max_bytes" -le 2147483648 ]] \ + || { echo "unsafe release artifact size bound" >&2; exit 1; } + curl --fail --silent --show-error --location --proto '=https' --tlsv1.2 --max-filesize "$max_bytes" \ + "${RELEASE_URL}/${artifact}" \ + -o "$RELEASE_DIR/$artifact" + (cd "$RELEASE_DIR" && printf '%s\n' "$matches" | sha256sum --check --strict --status) +} + +fetch_release_artifact release-manifest.json +/usr/local/libexec/nehemiah-validate-release \ + --manifest "$RELEASE_DIR/release-manifest.json" \ + --version "$NEHEMIAH_RELEASE_VERSION" \ + --arch "$RELEASE_ARCH" \ + --output "$RELEASE_DIR/selected-release.env" +# The validator emits only fixed uppercase names and strict numeric/hash/file values. +# shellcheck disable=SC1091 +source "$RELEASE_DIR/selected-release.env" + +NEHEMIAH_FIRECRACKER_SHA256="$FIRECRACKER_ARCHIVE_SHA256" +NEHEMIAH_KERNEL_SHA256="$KERNEL_SHA256" +NEHEMIAH_FIRECRACKER_INSTALLED_SHA256="$FIRECRACKER_INSTALLED_SHA256" +NEHEMIAH_JAILER_INSTALLED_SHA256="$JAILER_INSTALLED_SHA256" +NEHEMIAH_RUNTIME_PYTHON_SHA256="$PYTHON_SHA256" +NEHEMIAH_RUNTIME_DESKTOP_SHA256="$DESKTOP_SHA256" +for artifact in "$DAEMON_ARTIFACT" "$AGENT_ARTIFACT" "$HOST_ARTIFACT"; do + fetch_release_artifact "$artifact" +done +fetch_release_artifact "$PACKAGE_ARTIFACT" "$PACKAGE_MAX_BYTES" +fetch_release_artifact "$FIRECRACKER_ARTIFACT" "$FIRECRACKER_MAX_BYTES" +fetch_release_artifact "$KERNEL_ARTIFACT" "$KERNEL_MAX_BYTES" +NEHEMIAH_FIRECRACKER_ARCHIVE="$RELEASE_DIR/$FIRECRACKER_ARTIFACT" +NEHEMIAH_KERNEL_IMAGE="$RELEASE_DIR/$KERNEL_ARTIFACT" +export NEHEMIAH_FIRECRACKER_ARCHIVE NEHEMIAH_FIRECRACKER_SHA256 +export NEHEMIAH_KERNEL_IMAGE NEHEMIAH_KERNEL_SHA256 +export NEHEMIAH_FIRECRACKER_INSTALLED_SHA256 NEHEMIAH_JAILER_INSTALLED_SHA256 +export NEHEMIAH_RUNTIME_PYTHON_SHA256 NEHEMIAH_RUNTIME_DESKTOP_SHA256 +export NEHEMIAH_RELEASE_VERSION + +safe_extract() { + local archive="$1" destination="$2" + mkdir -p "$destination" + python3 - "$archive" "$destination" <<'PY' +import pathlib +import sys +import tarfile + +archive, destination = sys.argv[1:] +with tarfile.open(archive, "r:gz") as bundle: + members = bundle.getmembers() + if len(members) > 4096 or sum(member.size for member in members) > 512 * 1024 * 1024: + raise SystemExit("release archive exceeds extraction limits") + seen = set() + for member in members: + normalized = pathlib.PurePosixPath(member.name) + canonical = str(normalized) + if normalized.is_absolute() or ".." in normalized.parts or canonical in seen: + raise SystemExit("release archive contains an unsafe or duplicate path") + if not (member.isdir() or member.isfile()): + raise SystemExit("release archive contains a non-regular entry") + seen.add(canonical) + bundle.extractall(destination, members=members, filter="data") +PY +} + +safe_extract "$RELEASE_DIR/$HOST_ARTIFACT" "$RELEASE_DIR/host" +for relative in managed-host-packages.py validate-managed-release.py verify-minisign.py wireguard-config.py; do + [[ -f "$RELEASE_DIR/host/infra/latitude/$relative" && \ + ! -L "$RELEASE_DIR/host/infra/latitude/$relative" ]] \ + || { echo "host bootstrap archive is missing $relative" >&2; exit 1; } +done +python3 "$RELEASE_DIR/host/infra/latitude/managed-host-packages.py" install \ + --archive "$RELEASE_DIR/$PACKAGE_ARTIFACT" \ + --manifest-sha256 "$PACKAGE_MANIFEST_SHA256" \ + --arch "$RELEASE_ARCH" \ + --release-version "$NEHEMIAH_RELEASE_VERSION" +# Cross-check the bootstrap verifier with the exact retained Minisign package. +minisign -Vm "$RELEASE_DIR/SHA256SUMS" -x "$RELEASE_DIR/SHA256SUMS.minisig" \ + -P "$NEHEMIAH_RELEASE_MINISIGN_KEY" + +install_guest_image() { + local flavor="$1" destination="$2" artifact="$3" max_bytes="$4" + local expected_bytes="$5" expected_sha="$6" temporary magic + fetch_release_artifact "$artifact" "$max_bytes" + gzip --test "$RELEASE_DIR/$artifact" + install -d -m 0755 /opt/boring/rootfs + [[ ! -L /opt/boring/rootfs ]] || { echo "rootfs directory must not be a symlink" >&2; exit 1; } + temporary="$(mktemp "/opt/boring/rootfs/.${flavor}.ext4.XXXXXX")" + gzip -dc "$RELEASE_DIR/$artifact" | dd of="$temporary" bs=4M conv=sparse,fsync status=none + [[ "$(stat -c %s "$temporary")" == "$expected_bytes" ]] \ + || { echo "$flavor image uncompressed size mismatch" >&2; exit 1; } + [[ "$(sha256sum "$temporary" | cut -d' ' -f1)" == "$expected_sha" ]] \ + || { echo "$flavor image uncompressed checksum mismatch" >&2; exit 1; } + magic="$(dd if="$temporary" bs=1 skip=1080 count=2 status=none | od -An -tx1 | tr -d ' \n')" + [[ "$magic" == 53ef ]] || { echo "$flavor image is not ext4" >&2; exit 1; } + file -b "$temporary" | grep -q 'ext4 filesystem data' \ + || { echo "$flavor image has an invalid filesystem type" >&2; exit 1; } + e2fsck -fn "$temporary" >/dev/null + chmod 0644 "$temporary" + chown root:root "$temporary" + mv -fT "$temporary" "$destination" + [[ -f "$destination" && ! -L "$destination" && "$(stat -c '%a:%u' "$destination")" == 644:0 ]] \ + || { echo "$flavor image was not installed safely" >&2; exit 1; } +} + +install_guest_image python /opt/boring/rootfs/rootfs.ext4 \ + "$PYTHON_ARTIFACT" "$PYTHON_MAX_BYTES" "$PYTHON_UNCOMPRESSED_BYTES" "$PYTHON_SHA256" +install_guest_image desktop /opt/boring/rootfs/desktop.ext4 \ + "$DESKTOP_ARTIFACT" "$DESKTOP_MAX_BYTES" "$DESKTOP_UNCOMPRESSED_BYTES" "$DESKTOP_SHA256" + +safe_extract "$RELEASE_DIR/$DAEMON_ARTIFACT" "$RELEASE_DIR/daemon" +safe_extract "$RELEASE_DIR/$AGENT_ARTIFACT" "$RELEASE_DIR/agent" +[[ -f "$RELEASE_DIR/daemon/nehemiahd" && ! -L "$RELEASE_DIR/daemon/nehemiahd" ]] \ + || { echo "daemon archive is missing nehemiahd" >&2; exit 1; } +[[ -f "$RELEASE_DIR/agent/bc-guest-agent" && ! -L "$RELEASE_DIR/agent/bc-guest-agent" ]] \ + || { echo "agent archive is missing bc-guest-agent" >&2; exit 1; } +install -D -m 0755 "$RELEASE_DIR/daemon/nehemiahd" /usr/local/bin/nehemiahd +install -D -m 0755 "$RELEASE_DIR/agent/bc-guest-agent" /opt/boring/bin/bc-guest-agent + +ASSET_ROOT=/opt/nehemiah +for relative in \ + bootstrap.sh cloud-init.sh managed-host-packages.py managed-host-preflight.sh \ + net-setup.sh validate-managed-release.py verify-isolation.sh verify-minisign.py \ + wireguard-config.py; do + source_path="$RELEASE_DIR/host/infra/latitude/$relative" + [[ -f "$source_path" && ! -L "$source_path" ]] \ + || { echo "host bootstrap archive is missing $relative" >&2; exit 1; } + install -D -m 0755 "$source_path" "$ASSET_ROOT/infra/latitude/$relative" +done +for relative in boring-net.service nehemiahd.service; do + source_path="$RELEASE_DIR/host/infra/latitude/$relative" + [[ -f "$source_path" && ! -L "$source_path" ]] \ + || { echo "host bootstrap archive is missing $relative" >&2; exit 1; } + install -D -m 0644 "$source_path" "$ASSET_ROOT/infra/latitude/$relative" +done + +install -d -m 0700 /etc/wireguard +wireguard_tmp="$(mktemp /etc/wireguard/.wg0.conf.XXXXXX)" +printf '%s' "$NEHEMIAH_WIREGUARD_CONFIG_B64" \ + | base64 --decode > "$wireguard_tmp" +chown root:root "$wireguard_tmp" +chmod 0600 "$wireguard_tmp" +NEHEMIAH_WIREGUARD_CONFIG_SHA256="$(sha256sum "$wireguard_tmp" | cut -d' ' -f1)" +python3 "$ASSET_ROOT/infra/latitude/wireguard-config.py" verify \ + --advertise-address "$NEHEMIAH_ADVERTISE_ADDRESS" \ + --control-plane-address "$NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS" \ + --gateway-address "$NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS" \ + --guest-subnet 10.200.0.0/24 \ + --expected-sha256 "$NEHEMIAH_WIREGUARD_CONFIG_SHA256" \ + --path "$wireguard_tmp" +mv -fT "$wireguard_tmp" /etc/wireguard/wg0.conf +wg-quick strip wg0 >/dev/null +systemctl enable --now wg-quick@wg0 + +install -d -m 0750 /etc/boring +cat > "$DAEMON_ENV" <> "$DAEMON_ENV" +fi +chmod 0600 "$DAEMON_ENV" + +install -d -m 0700 /var/lib/nehemiahd +provider_image_tmp="$(mktemp /var/lib/nehemiahd/.provider-image.XXXXXX)" +cat > "$provider_image_tmp" < "$managed_marker_tmp" +chown root:root "$managed_marker_tmp" +chmod 0400 "$managed_marker_tmp" +mv -fT "$managed_marker_tmp" "$MANAGED_HOST_MARKER" +systemctl daemon-reload +systemctl enable --now boring-net.service +systemctl enable nehemiahd.service +systemctl restart nehemiahd.service + +# Do not erase the only usable enrollment credential until the daemon has +# durably enrolled and atomically removed the one-time token from its service +# environment. A failed bootstrap remains retryable and fails visibly. +enrollment_ready=0 +for _ in $(seq 1 120); do + if [[ -f /var/lib/nehemiahd/enrollment.json && \ + ! -L /var/lib/nehemiahd/enrollment.json && \ + "$(stat -c '%a:%u' /var/lib/nehemiahd/enrollment.json)" == 600:0 ]] && \ + systemctl is-active --quiet nehemiahd.service && \ + ! grep -Eq '^(NEHEMIAH|BORING)_FLEET_BOOTSTRAP_TOKEN=' "$DAEMON_ENV"; then + enrollment_ready=1 + break + fi + sleep 1 +done +if [[ "$enrollment_ready" != 1 ]]; then + echo "managed host did not complete enrollment within 120 seconds" >&2 + exit 1 +fi + +# The source bundle and cloud-init's cached copy both contain recoverable +# base64-encoded credentials. Remove them only after durable enrollment. +shred -u "$BOOTSTRAP_ENV" 2>/dev/null || rm -f -- "$BOOTSTRAP_ENV" +for cached_user_data in \ + /var/lib/cloud/instances/*/user-data.txt \ + /var/lib/cloud/instances/*/user-data.txt.i; do + [[ -e "$cached_user_data" || -L "$cached_user_data" ]] || continue + if [[ -L "$cached_user_data" ]]; then + unlink "$cached_user_data" + elif [[ -f "$cached_user_data" ]]; then + shred -u "$cached_user_data" 2>/dev/null || rm -f -- "$cached_user_data" + fi +done +install -d -m 0700 /var/lib/nehemiahd +printf 'release=%s\n' "$NEHEMIAH_RELEASE_VERSION" \ + > /var/lib/nehemiahd/bootstrap.complete +chmod 0600 /var/lib/nehemiahd/bootstrap.complete diff --git a/infra/latitude/deploy.sh b/infra/latitude/deploy.sh index 7864061..7d52b5c 100755 --- a/infra/latitude/deploy.sh +++ b/infra/latitude/deploy.sh @@ -101,7 +101,7 @@ REMOTE_BUILD # ---- 3. install systemd unit + env ------------------------------------------ echo "==> [3/5] installing systemd unit + env file" -scp "${SSH_OPTS[@]}" "${SCRIPT_DIR}/nehemiahd.service" "${REMOTE}:/etc/systemd/system/nehemiahd.service" +scp "${SSH_OPTS[@]}" "${SCRIPT_DIR}/nehemiahd-local.service" "${REMOTE}:/etc/systemd/system/nehemiahd.service" # Write /etc/boring/nehemiahd.env. Only set NEHEMIAH_TOKEN if provided; never echo it. if [[ -n "${NEHEMIAH_TOKEN}" ]]; then diff --git a/infra/latitude/managed-host-packages.py b/infra/latitude/managed-host-packages.py new file mode 100755 index 0000000..5a4b0c6 --- /dev/null +++ b/infra/latitude/managed-host-packages.py @@ -0,0 +1,496 @@ +#!/usr/bin/env python3 +"""Install and continuously verify the signed managed-host package closure.""" + +from __future__ import annotations + +import argparse +import gzip +import hashlib +import json +import os +import pathlib +import re +import shutil +import subprocess +import sys +import tarfile +import tempfile + + +SHA256 = re.compile(r"[0-9a-f]{64}") +SAFE_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9._+~-]{0,254}") +SAFE_CACHE_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9%._+~-]{0,254}") +SAFE_PACKAGE = re.compile(r"[a-z0-9][a-z0-9+.-]{0,127}") +SAFE_VERSION = re.compile(r"[A-Za-z0-9][A-Za-z0-9.+:~_-]{0,255}") +ARCHES = ("amd64", "arm64") +INSTALLED_MANIFEST = pathlib.Path("/var/lib/nehemiahd/managed-host-packages.json") +MAX_ARCHIVE_BYTES = 268435456 +MAX_PACKAGE_BYTES = 67108864 +MAX_PACKAGE_COUNT = 256 +MAX_UNPACKED_BYTES = 1073741824 + + +def stop(message: str) -> "NoReturn": + raise SystemExit(f"managed host package guard: {message}") + + +def digest(path: pathlib.Path) -> str: + value = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + value.update(block) + return value.hexdigest() + + +def exact_keys(value: object, keys: set[str], label: str) -> dict: + if not isinstance(value, dict) or set(value) != keys: + stop(f"{label} does not contain the exact key set") + return value + + +def validate_manifest(value: object, arch: str, release_version: str | None) -> dict: + manifest = exact_keys( + value, + { + "architecture", + "contractVersion", + "operatingSystem", + "packageCount", + "packages", + "releaseVersion", + "repository", + "rootPackages", + "snapshot", + "unpackedBytes", + }, + "manifest", + ) + if manifest["contractVersion"] != 1 or manifest["architecture"] != arch: + stop("manifest contract or architecture mismatch") + if release_version is not None and manifest["releaseVersion"] != release_version: + stop("manifest release version mismatch") + if manifest["operatingSystem"] != { + "codename": "noble", + "id": "ubuntu", + "version": "24.04", + }: + stop("manifest targets an unsupported host operating system") + if not isinstance(manifest["releaseVersion"], str) or not re.fullmatch( + r"(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?", + manifest["releaseVersion"], + ): + stop("manifest release version is invalid") + roots = manifest["rootPackages"] + if ( + not isinstance(roots, list) + or roots != sorted(set(roots)) + or not roots + or not all(isinstance(name, str) and SAFE_PACKAGE.fullmatch(name) for name in roots) + ): + stop("manifest root package set is unsafe") + required = { + "apt", + "bash", + "ca-certificates", + "curl", + "dnsmasq", + "e2fsprogs", + "file", + "iproute2", + "ipset", + "iptables", + "jq", + "kmod", + "minisign", + "openssl", + "python3", + "systemd", + "wireguard-tools", + } + if not required.issubset(roots): + stop("manifest omits a required runtime package") + repository = exact_keys( + manifest["repository"], {"packagesGzipSha256", "packagesSha256"}, "repository" + ) + if not all(isinstance(value, str) and SHA256.fullmatch(value) for value in repository.values()): + stop("manifest repository digest is invalid") + snapshot = exact_keys( + manifest["snapshot"], {"baseUrl", "capturedAt", "indexes", "suites"}, "snapshot" + ) + if ( + not isinstance(snapshot["baseUrl"], str) + or not re.fullmatch( + r"https://snapshot\.ubuntu\.com/ubuntu/[0-9]{8}T[0-9]{6}Z", + snapshot["baseUrl"], + ) + or not re.fullmatch(r"[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z", snapshot["capturedAt"]) + or not isinstance(snapshot["indexes"], list) + or len(snapshot["indexes"]) != 6 + or not isinstance(snapshot["suites"], dict) + or set(snapshot["suites"]) != {"noble", "noble-security", "noble-updates"} + ): + stop("manifest snapshot evidence is invalid") + packages = manifest["packages"] + if ( + not isinstance(packages, list) + or not packages + or len(packages) != manifest["packageCount"] + or len(packages) > MAX_PACKAGE_COUNT + ): + stop("manifest package count is invalid") + expected_order = sorted(packages, key=lambda item: (item["name"], item["architecture"], item["version"])) + if packages != expected_order: + stop("manifest packages are not canonically sorted") + filenames = set() + package_names = set() + for package in packages: + exact_keys( + package, + { + "architecture", + "filename", + "installedBytes", + "name", + "sha256", + "size", + "sourcePath", + "version", + }, + "package", + ) + source_path = pathlib.PurePosixPath(package["sourcePath"]) + if ( + not isinstance(package["name"], str) + or not SAFE_PACKAGE.fullmatch(package["name"]) + or package["name"] in package_names + or not isinstance(package["version"], str) + or not SAFE_VERSION.fullmatch(package["version"]) + or package["architecture"] not in (arch, "all") + or not isinstance(package["filename"], str) + or not SAFE_NAME.fullmatch(package["filename"]) + or package["filename"] in filenames + or not isinstance(package["sha256"], str) + or not SHA256.fullmatch(package["sha256"]) + or not isinstance(package["size"], int) + or not 0 < package["size"] <= MAX_PACKAGE_BYTES + or not isinstance(package["installedBytes"], int) + or package["installedBytes"] < 0 + or source_path.is_absolute() + or ".." in source_path.parts + ): + stop("manifest contains an unsafe package entry") + package_names.add(package["name"]) + filenames.add(package["filename"]) + if ( + not isinstance(manifest["unpackedBytes"], int) + or manifest["unpackedBytes"] != sum(item["installedBytes"] for item in packages) + or manifest["unpackedBytes"] > MAX_UNPACKED_BYTES + ): + stop("manifest unpacked size is invalid") + return manifest + + +def parse_packages(contents: str) -> dict[str, dict[str, str]]: + result = {} + for raw in contents.strip().split("\n\n"): + fields = {} + current = None + for line in raw.splitlines(): + if line.startswith((" ", "\t")): + if current is None: + stop("Packages has an orphan continuation") + fields[current] += "\n" + line + continue + if ":" not in line: + stop("Packages has a malformed field") + name, value = line.split(":", 1) + if value.startswith(" "): + value = value[1:] + if name in fields: + stop("Packages has a duplicate field") + fields[name] = value + current = name + filename = fields.get("Filename") + if not filename or filename in result: + stop("Packages has a missing or duplicate filename") + result[filename] = fields + return result + + +def command(arguments: list[str], *, environment: dict | None = None) -> str: + result = subprocess.run( + arguments, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + env=environment, + check=False, + ) + if result.returncode != 0: + detail = (result.stdout + "\n" + result.stderr).strip() + stop(f"command failed ({' '.join(arguments[:3])}): {detail}") + return result.stdout + + +def deb_identity(path: pathlib.Path) -> tuple[str, str, str]: + values = [] + for field in ("Package", "Version", "Architecture"): + value = command(["dpkg-deb", "--field", str(path), field]).strip() + if not value or "\n" in value or "\r" in value: + stop(f"{path.name} has invalid package metadata") + values.append(value) + return tuple(values) + + +def safe_extract(archive: pathlib.Path, destination: pathlib.Path) -> None: + if not archive.is_file() or archive.is_symlink() or not 0 < archive.stat().st_size <= MAX_ARCHIVE_BYTES: + stop("package archive is missing, unsafe, or oversized") + total = 0 + seen = set() + with tarfile.open(archive, "r:gz") as bundle: + members = bundle.getmembers() + if not members or len(members) > MAX_PACKAGE_COUNT + 16: + stop("package archive entry count is invalid") + for member in members: + path = pathlib.PurePosixPath(member.name) + normalized = str(path) + if path.is_absolute() or ".." in path.parts or normalized in seen: + stop("package archive contains an unsafe or duplicate path") + if not (member.isdir() or member.isfile()): + stop("package archive contains a non-regular entry") + seen.add(normalized) + total += member.size + if total > MAX_ARCHIVE_BYTES + MAX_UNPACKED_BYTES: + stop("package archive exceeds the extraction size policy") + bundle.extractall(destination, members=members, filter="data") + + +def validate_repository(root: pathlib.Path, manifest: dict) -> None: + packages_path = root / "repo/Packages" + packages_gzip_path = root / "repo/Packages.gz" + if ( + digest(packages_path) != manifest["repository"]["packagesSha256"] + or digest(packages_gzip_path) != manifest["repository"]["packagesGzipSha256"] + ): + stop("repository metadata checksum mismatch") + try: + if gzip.decompress(packages_gzip_path.read_bytes()) != packages_path.read_bytes(): + stop("compressed repository metadata does not reproduce Packages") + except (OSError, gzip.BadGzipFile): + stop("compressed repository metadata is invalid") + index = parse_packages(packages_path.read_text()) + expected_files = { + "manifest.json", + "repo/Packages", + "repo/Packages.gz", + *(f"repo/packages/{package['filename']}" for package in manifest["packages"]), + } + actual_files = { + path.relative_to(root).as_posix() for path in root.rglob("*") if path.is_file() + } + if actual_files != expected_files: + stop("package archive does not contain the exact file set") + if set(index) != {f"packages/{package['filename']}" for package in manifest["packages"]}: + stop("Packages and manifest file sets differ") + for package in manifest["packages"]: + package_path = root / "repo/packages" / package["filename"] + fields = index[f"packages/{package['filename']}"] + if ( + package_path.stat().st_size != package["size"] + or digest(package_path) != package["sha256"] + or deb_identity(package_path) + != (package["name"], package["version"], package["architecture"]) + or fields.get("Package") != package["name"] + or fields.get("Version") != package["version"] + or fields.get("Architecture") != package["architecture"] + or fields.get("SHA256") != package["sha256"] + or fields.get("Size") != str(package["size"]) + ): + stop(f"retained package identity mismatch: {package['filename']}") + + +def os_identity() -> tuple[str, str]: + values = {} + for line in pathlib.Path("/etc/os-release").read_text().splitlines(): + if "=" in line: + key, value = line.split("=", 1) + values[key] = value.strip('"') + return values.get("ID", ""), values.get("VERSION_ID", "") + + +def install(args: argparse.Namespace) -> None: + if os.geteuid() != 0: + stop("installation requires root") + if args.arch not in ARCHES or not SHA256.fullmatch(args.manifest_sha256): + stop("invalid expected package identity") + if os_identity() != ("ubuntu", "24.04"): + stop("provider base image is not Ubuntu 24.04") + archive = args.archive.resolve() + with tempfile.TemporaryDirectory(prefix="nehemiah-package-install-", dir="/var/tmp") as temporary: + root = pathlib.Path(temporary) / "archive" + root.mkdir() + safe_extract(archive, root) + manifest_path = root / "manifest.json" + if digest(manifest_path) != args.manifest_sha256: + stop("package manifest does not match the signed release") + try: + manifest = validate_manifest( + json.loads(manifest_path.read_text()), args.arch, args.release_version + ) + except (OSError, json.JSONDecodeError): + stop("package manifest is invalid JSON") + validate_repository(root, manifest) + apt_root = pathlib.Path(temporary) / "apt" + lists = apt_root / "lists" + archive_cache = apt_root / "archives" + lists.joinpath("partial").mkdir(parents=True) + archive_cache.joinpath("partial").mkdir(parents=True) + empty_parts = apt_root / "apt.conf.d" + empty_parts.mkdir() + # --no-download accepts only packages already present in APT's archive + # cache. Seed that cache from the verified flat repository so neither a + # misconfiguration nor a maintainer changing APT URI handling can turn + # this install into a network fetch. + for package in manifest["packages"]: + source_package = root / "repo/packages" / package["filename"] + shutil.copyfile(source_package, archive_cache / package["filename"]) + cache_version = package["version"].replace(":", "%3a") + cache_name = ( + f"{package['name']}_{cache_version}_{package['architecture']}.deb" + ) + if not SAFE_CACHE_NAME.fullmatch(cache_name): + stop("package identity produced an unsafe APT cache name") + if cache_name != package["filename"]: + shutil.copyfile(source_package, archive_cache / cache_name) + source = apt_root / "sources.list" + source.write_text(f"deb [trusted=yes] file:{root / 'repo'} ./\n") + options = [ + "-o", "Dir::Etc::main=/dev/null", + "-o", f"Dir::Etc::parts={empty_parts}", + "-o", f"Dir::Etc::sourcelist={source}", + "-o", "Dir::Etc::sourceparts=-", + "-o", f"Dir::State::lists={lists}", + "-o", f"Dir::Cache::archives={archive_cache}", + "-o", "Acquire::Languages=none", + "-o", "Acquire::Retries=0", + "-o", "Acquire::http::Proxy=false", + "-o", "Acquire::https::Proxy=false", + ] + environment = { + **os.environ, + "APT_CONFIG": "/dev/null", + "DEBIAN_FRONTEND": "noninteractive", + "LC_ALL": "C", + } + command(["apt-get", *options, "update"], environment=environment) + exact_packages = [f"{package['name']}={package['version']}" for package in manifest["packages"]] + command( + [ + "apt-get", + "--yes", + "--no-download", + "--no-install-recommends", + "--no-remove", + "--allow-downgrades", + "--allow-change-held-packages", + *options, + "install", + *exact_packages, + ], + environment=environment, + ) + INSTALLED_MANIFEST.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + temporary_manifest = INSTALLED_MANIFEST.parent / ".managed-host-packages.json.tmp" + temporary_manifest.write_bytes(manifest_path.read_bytes()) + os.chown(temporary_manifest, 0, 0) + temporary_manifest.chmod(0o600) + temporary_manifest.replace(INSTALLED_MANIFEST) + verify_installed(args.arch, args.release_version) + + +def verify_installed(arch: str, release_version: str | None) -> None: + if arch not in ARCHES: + stop("invalid installed architecture") + try: + stats = INSTALLED_MANIFEST.lstat() + except OSError: + stop("installed package manifest is missing") + if not INSTALLED_MANIFEST.is_file() or INSTALLED_MANIFEST.is_symlink(): + stop("installed package manifest is unsafe") + if (stats.st_mode & 0o777) != 0o600 or stats.st_uid != 0 or stats.st_gid != 0: + stop("installed package manifest ownership is unsafe") + try: + manifest = validate_manifest( + json.loads(INSTALLED_MANIFEST.read_text()), arch, release_version + ) + except (OSError, json.JSONDecodeError): + stop("installed package manifest is invalid") + for package in manifest["packages"]: + result = command( + [ + "dpkg-query", + "--show", + "--showformat=${db:Status-Status}\n${Version}\n${Architecture}\n", + package["name"], + ] + ).splitlines() + if result != ["installed", package["version"], package["architecture"]]: + stop(f"installed package drift detected: {package['name']}") + verification = subprocess.run( + ["dpkg", "--verify", *(package["name"] for package in manifest["packages"])], + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + check=False, + ) + allowed_missing_prefixes = ( + "/usr/share/doc/", + "/usr/share/info/", + "/usr/share/locale/", + "/usr/share/man/", + ) + allowed_missing_exact = { + "/var/cache/apt/archives/partial", + "/var/lib/apt/lists/partial", + } + unexpected = [] + for line in verification.stdout.splitlines(): + match = re.fullmatch(r"missing +(/\S+)", line) + if match and ( + match.group(1).startswith(allowed_missing_prefixes) + or match.group(1) in allowed_missing_exact + ): + # Provider images may use dpkg path-exclude for documentation and + # translations. Those files are not executable/runtime inputs; + # every present file and every non-documentation path remains + # covered by dpkg's package checksum database. + continue + unexpected.append(line) + if unexpected or verification.stderr.strip(): + stop( + "installed package file drift detected " + f"(paths={len(unexpected)}, verifier_error={bool(verification.stderr.strip())})" + ) + + +def main() -> None: + parser = argparse.ArgumentParser() + subparsers = parser.add_subparsers(dest="command", required=True) + install_parser = subparsers.add_parser("install") + install_parser.add_argument("--archive", type=pathlib.Path, required=True) + install_parser.add_argument("--manifest-sha256", required=True) + install_parser.add_argument("--arch", required=True) + install_parser.add_argument("--release-version", required=True) + install_parser.set_defaults(action=lambda args: install(args)) + verify_parser = subparsers.add_parser("verify-installed") + verify_parser.add_argument("--arch", required=True) + verify_parser.add_argument("--release-version") + verify_parser.set_defaults(action=lambda args: verify_installed(args.arch, args.release_version)) + args = parser.parse_args() + args.action(args) + + +if __name__ == "__main__": + main() diff --git a/infra/latitude/managed-host-preflight.sh b/infra/latitude/managed-host-preflight.sh new file mode 100644 index 0000000..fd7ccb6 --- /dev/null +++ b/infra/latitude/managed-host-preflight.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# Exact, credential-safe preflight shared by the signed managed systemd units. +set -euo pipefail +set +x + +ENV_PATH=/etc/boring/nehemiahd.env +MARKER_PATH=/etc/boring/managed-host +NET_SETUP=/opt/boring/bin/net-setup.sh +PACKAGE_GUARD=/opt/boring/bin/managed-host-packages +WIREGUARD_GUARD=/opt/boring/bin/wireguard-config +WIREGUARD_PATH=/etc/wireguard/wg0.conf +PROVIDER_IMAGE_PATH=/var/lib/nehemiahd/provider-image + +fail_closed() { + local label="$1" + if [[ -x "$NET_SETUP" ]]; then + /usr/bin/env NEHEMIAH_MODE=1 "$NET_SETUP" --fail-closed >/dev/null 2>&1 || true + fi + echo "managed host preflight failed: $label" >&2 + exit 1 +} + +require_exact_file() { + local path="$1" expected_metadata="$2" label="$3" actual_metadata + [[ -f "$path" && ! -L "$path" ]] || fail_closed "$label is not a regular file" + actual_metadata="$(stat -c '%a:%u:%g' -- "$path")" \ + || fail_closed "$label metadata is unreadable" + [[ "$actual_metadata" == "$expected_metadata" ]] \ + || fail_closed "$label ownership or mode is invalid" +} + +require_exact_file "$ENV_PATH" 600:0:0 'managed environment' +require_exact_file "$MARKER_PATH" 400:0:0 'managed marker' +[[ "$(< "$MARKER_PATH")" == nehemiah-managed-host-v1 ]] \ + || fail_closed 'managed marker content is invalid' +require_exact_file "$WIREGUARD_PATH" 600:0:0 'WireGuard configuration' +require_exact_file "$PROVIDER_IMAGE_PATH" 600:0:0 'provider image evidence' +for required in \ + NEHEMIAH_RELEASE_VERSION NEHEMIAH_RUNTIME_ARCH NEHEMIAH_ADVERTISE_ADDRESS \ + NEHEMIAH_NET_SUBNET NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS \ + NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS \ + NEHEMIAH_WIREGUARD_CONFIG_SHA256 NEHEMIAH_PROVIDER_IMAGE_ID \ + NEHEMIAH_PROVIDER_IMAGE_SLUG NEHEMIAH_PROVIDER_IMAGE_VERSION \ + NEHEMIAH_PROVIDER_IMAGE_ARCH; do + [[ -n "${!required:-}" ]] || fail_closed "$required is missing" +done +[[ -x "$PACKAGE_GUARD" ]] || fail_closed 'package guard is missing' +[[ -x "$WIREGUARD_GUARD" ]] || fail_closed 'WireGuard guard is missing' +python3 "$PACKAGE_GUARD" verify-installed \ + --arch "$NEHEMIAH_RUNTIME_ARCH" --release-version "$NEHEMIAH_RELEASE_VERSION" \ + || fail_closed 'managed package cohort has drifted' +python3 "$WIREGUARD_GUARD" verify \ + --advertise-address "$NEHEMIAH_ADVERTISE_ADDRESS" \ + --control-plane-address "$NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS" \ + --gateway-address "$NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS" \ + --guest-subnet "${NEHEMIAH_NET_SUBNET}.0/24" \ + --expected-sha256 "$NEHEMIAH_WIREGUARD_CONFIG_SHA256" \ + --path "$WIREGUARD_PATH" \ + || fail_closed 'WireGuard configuration has drifted' +if ! python3 - "$PROVIDER_IMAGE_PATH" \ + "$NEHEMIAH_PROVIDER_IMAGE_ID" "$NEHEMIAH_PROVIDER_IMAGE_SLUG" \ + "$NEHEMIAH_PROVIDER_IMAGE_VERSION" "$NEHEMIAH_PROVIDER_IMAGE_ARCH" <<'PY' +import pathlib +import sys + +path, image_id, slug, version, arch = sys.argv[1:] +expected = f"provider=latitude\nid={image_id}\nslug={slug}\nversion={version}\narch={arch}\n" +if pathlib.Path(path).read_text() != expected: + raise SystemExit(1) +PY +then + fail_closed 'provider image evidence has drifted' +fi diff --git a/infra/latitude/managed-host.env.example b/infra/latitude/managed-host.env.example new file mode 100644 index 0000000..b1f20e0 --- /dev/null +++ b/infra/latitude/managed-host.env.example @@ -0,0 +1,50 @@ +# Copy to a private file, replace every placeholder, then chmod 600. +# This file is parsed as literal KEY=value data; shell syntax is not accepted. +NEHEMIAH_RELEASE_BASE=https://github.com/boringcomputers/nehemiah/releases/download +NEHEMIAH_RELEASE_VERSION=0.2.0-beta.0 +NEHEMIAH_RELEASE_MINISIGN_KEY= +NEHEMIAH_HOST_ID=latitude-manual-01 +NEHEMIAH_REGION=MIA2 +# Use a stable, unique provider identity. It cannot be changed after enrollment. +NEHEMIAH_PROVIDER_ID=latitude-manual-01 +# Operator-reviewed immutable launch input. Resolve the opaque id with Latitude's +# live operating-system API and copy its exact current slug/version/architecture. +# The provisioner refuses to create user data or a billable server if any field, +# or the selected plan's provisionable_on membership, differs at launch time. +LATITUDE_OS_ID= +LATITUDE_OS_SLUG=ubuntu_24_04_x64_lts +LATITUDE_OS_VERSION= +LATITUDE_OS_ARCH=amd64 +NEHEMIAH_FLEET_BOOTSTRAP_TOKEN= +NEHEMIAH_CONTROL_PLANE_URL=https://api.example.com +# Optional. Set this to the exact origin used by template presigned URLs. For +# virtual-hosted S3, include the bucket in the hostname; for path-style S3, use +# the endpoint origin. Managed hosts reject HTTP and every other object origin. +# NEHEMIAH_TEMPLATE_OBJECT_ORIGIN=https://nehemiah-template-artifacts.objects.storage-provider.example +# Literal address reachable only through the managed WireGuard overlay. +NEHEMIAH_ADVERTISE_ADDRESS=10.42.0.10 +# Exact, distinct overlay host addresses assigned to the control-plane and +# gateway roles. AllowedIPs in the canonical file must equal this two-address +# set and neither address may be in the managed guest subnet. +NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS=10.42.0.1 +NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS=10.42.0.2 +# Base64 of a canonical wg-quick file with exactly one Interface (PrivateKey, +# the advertised /32 or /128 Address) and one Peer (PublicKey, bounded Endpoint, +# the exact two configured CP/gateway host-route AllowedIPs, PersistentKeepalive=25). Hooks, +# DNS, Table, SaveConfig, comments, broad/default routes and duplicates fail. +NEHEMIAH_WIREGUARD_CONFIG_B64= +# Base64 of one recovery SSH public-key line, never a private key. +NEHEMIAH_ROOT_SSH_AUTHORIZED_KEY_B64= + +# Managed hosts require central telemetry. Provision a private TLS collector and +# a unique per-host collector credential that is distinct from every fleet, +# daemon-internal, and gateway credential. +NEHEMIAH_OTEL_ENABLED=true +NEHEMIAH_OTEL_ENDPOINT=https://otel-collector.internal.example +NEHEMIAH_OTEL_AUTHORIZATION=Basic REPLACE_WITH_BASE64_UNIQUE_HOST_CREDENTIAL +NEHEMIAH_SERVICE_VERSION=0.2.0-beta.0 +NEHEMIAH_INSTANCE_ID=latitude-manual-01 +NEHEMIAH_DEPLOYMENT_ENVIRONMENT=production +NEHEMIAH_OTEL_EXPORT_INTERVAL_MS=15000 +NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS=10000 +NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO=0.1 diff --git a/infra/latitude/nehemiahd-local.service b/infra/latitude/nehemiahd-local.service new file mode 100644 index 0000000..b61f188 --- /dev/null +++ b/infra/latitude/nehemiahd-local.service @@ -0,0 +1,31 @@ +[Unit] +Description=nehemiahd — local/prototype Firecracker microVM daemon +Documentation=https://github.com/boringcomputers/nehemiah +After=network.target +Wants=network.target +Conflicts=boringd.service + +[Service] +Type=simple +# Prototype installs may start without credentials and retain legacy config +# compatibility. Never install this unit on an approved managed host. +EnvironmentFile=-/etc/boring/boringd.env +EnvironmentFile=-/etc/boring/nehemiahd.env +ExecStart=/usr/local/bin/nehemiahd +WorkingDirectory=/opt/boring +Restart=always +RestartSec=2 +KillMode=control-group +User=root +Group=root +UMask=0077 +LimitNOFILE=1048576 +LimitNPROC=65536 +Delegate=yes +RuntimeDirectory=nehemiahd +RuntimeDirectoryMode=0700 +StateDirectory=nehemiahd +StateDirectoryMode=0700 + +[Install] +WantedBy=multi-user.target diff --git a/infra/latitude/net-setup.sh b/infra/latitude/net-setup.sh index bb536e0..a38bd05 100644 --- a/infra/latitude/net-setup.sh +++ b/infra/latitude/net-setup.sh @@ -3,7 +3,7 @@ # net-setup.sh - Host networking for guest internet egress (idempotent). # # Creates a bridge (boring0, 10.200.0.1/24) that nehemiahd attaches per-VM taps to, -# runs dnsmasq for DHCP + DNS on it, NATs guest traffic out the uplink, and +# runs dnsmasq for DHCP only, NATs guest traffic out the uplink, and # installs a strict EGRESS FIREWALL. This box runs untrusted public code, so the # firewall must hold: guests may reach the public internet, but NOT the cloud # metadata endpoint, private ranges, the host, other guests, or SMTP, and their @@ -13,67 +13,276 @@ # set -euo pipefail -BR="boring0" -SUBNET="10.200.0" +BR="${NEHEMIAH_NET_BRIDGE:-boring0}" +SUBNET="${NEHEMIAH_NET_SUBNET:-10.200.0}" +STATE_PATH="${NEHEMIAH_STATE_PATH:-/var/lib/nehemiahd/state.json}" +MANAGED_MODE="${NEHEMIAH_MODE:-0}" +NETWORK_SETUP_COMPLETE=0 +ACTION="${1:-apply}" + +log() { printf '\033[1;34m[net]\033[0m %s\n' "$*"; } + +fail_closed_guest_port() { + local tap="$1" status=0 + ip link set "$tap" down >/dev/null 2>&1 || status=1 + ip link set "$tap" nomaster >/dev/null 2>&1 || status=1 + return "$status" +} + +fail_closed_all_guest_ports() { + local tap_path tap_name status=0 + for tap_path in /sys/class/net/bt*; do + [[ -e "$tap_path" ]] || continue + tap_name="${tap_path##*/}" + [[ "$tap_name" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,14}$ ]] || continue + fail_closed_guest_port "$tap_name" || status=1 + done + return "$status" +} + +freeze_all_guest_ports() { + local tap_path tap_name status=0 + for tap_path in /sys/class/net/bt*; do + [[ -e "$tap_path" ]] || continue + tap_name="${tap_path##*/}" + [[ "$tap_name" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,14}$ ]] || continue + ip link set "$tap_name" down >/dev/null 2>&1 || status=1 + done + return "$status" +} + +managed_fail_closed_on_exit() { + local status=$? + trap - EXIT + if [[ "$status" -ne 0 || "$NETWORK_SETUP_COMPLETE" -ne 1 ]]; then + fail_closed_all_guest_ports || true + [[ "$status" -ne 0 ]] || status=1 + fi + exit "$status" +} + +# A managed network restart starts by disabling every guest port. The EXIT trap +# repeats that operation after every unsuccessful path, including validation, +# missing-tool, partial-firewall, dnsmasq, and isolation failures. Only the fully +# verified success path below disarms the fail-closed behavior. +if [[ "$MANAGED_MODE" == "1" ]]; then + trap managed_fail_closed_on_exit EXIT + if ! fail_closed_all_guest_ports; then + echo "failed to disable one or more guest taps before managed setup" >&2 + exit 1 + fi + : "${NEHEMIAH_RELEASE_VERSION:?managed release version is required}" + : "${NEHEMIAH_RUNTIME_ARCH:?managed runtime architecture is required}" + : "${NEHEMIAH_ADVERTISE_ADDRESS:?managed advertised address is required}" + : "${NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS:?managed control-plane overlay address is required}" + : "${NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS:?managed gateway overlay address is required}" + : "${NEHEMIAH_WIREGUARD_CONFIG_SHA256:?managed WireGuard digest is required}" + python3 /opt/boring/bin/managed-host-packages verify-installed \ + --arch "$NEHEMIAH_RUNTIME_ARCH" --release-version "$NEHEMIAH_RELEASE_VERSION" + python3 /opt/boring/bin/wireguard-config verify \ + --advertise-address "$NEHEMIAH_ADVERTISE_ADDRESS" \ + --control-plane-address "$NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS" \ + --gateway-address "$NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS" \ + --guest-subnet "${SUBNET}.0/24" \ + --expected-sha256 "$NEHEMIAH_WIREGUARD_CONFIG_SHA256" \ + --path /etc/wireguard/wg0.conf +fi + +[[ "$#" -le 1 ]] || { echo "usage: net-setup.sh [--fail-closed]" >&2; exit 2; } +case "$ACTION" in + apply) ;; + --fail-closed) + if ! fail_closed_all_guest_ports; then + echo "failed to disable one or more guest taps" >&2 + exit 1 + fi + NETWORK_SETUP_COMPLETE=1 + exit 0 + ;; + *) echo "usage: net-setup.sh [--fail-closed]" >&2; exit 2 ;; +esac + +[[ "$BR" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,14}$ ]] || { echo "invalid bridge name" >&2; exit 1; } +[[ "$SUBNET" =~ ^10\.[0-9]{1,3}\.[0-9]{1,3}$|^172\.(1[6-9]|2[0-9]|3[01])\.[0-9]{1,3}$|^192\.168\.[0-9]{1,3}$ ]] || { echo "guest subnet must be a private IPv4 /24 prefix" >&2; exit 1; } CIDR="${SUBNET}.0/24" +for required_command in awk bridge dnsmasq ip iptables ip6tables jq sha1sum; do + command -v "$required_command" >/dev/null 2>&1 || { echo "missing required command: ${required_command}" >&2; exit 1; } +done UPLINK="$(ip route show default | awk '{print $5; exit}')" [ -n "$UPLINK" ] || { echo "no default route uplink"; exit 1; } -log() { printf '\033[1;34m[net]\033[0m %s\n' "$*"; } +guest_mac_for_id() { + local id="$1" digest + digest="$(printf '%s' "$id" | sha1sum | awk '{print $1}')" + printf '06:00:%s:%s:%s:%s\n' "${digest:0:2}" "${digest:2:2}" "${digest:4:2}" "${digest:6:2}" +} + +guest_ip_is_usable() { + local address="$1" suffix + [[ "$address" == "${SUBNET}."* ]] || return 1 + suffix="${address##*.}" + [[ "$suffix" =~ ^[0-9]+$ ]] || return 1 + (( 10#$suffix >= 2 && 10#$suffix <= 254 )) +} + +fdb_selects_port() { + local mac="$1" tap="$2" output + output="$(bridge fdb get "$mac" dev "$tap" master 2>/dev/null)" || return 1 + awk -v wanted_mac="${mac,,}" -v wanted_tap="$tap" -v wanted_bridge="$BR" ' + tolower($1) == wanted_mac { + for (i = 1; i <= NF; i++) { + if ($i == "dev" && $(i + 1) == wanted_tap) dev_ok = 1 + if ($i == "master" && $(i + 1) == wanted_bridge) master_ok = 1 + if ($i == "static") static_ok = 1 + } + } + END { exit !(dev_ok && master_ok && static_ok) } + ' <<<"$output" +} + +secure_guest_port() { + local tap="$1" mac="$2" address="${3:-}" neighbor port_state + [[ "$tap" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,14}$ ]] || return 1 + [[ "$mac" =~ ^06:00:([[:xdigit:]]{2}:){3}[[:xdigit:]]{2}$ ]] || return 1 + + # The port is down before it joins the bridge, so there is no window where a + # live guest can send ordinary shared-L2 traffic before isolation is active. + ip link set "$tap" down || return 1 + if ! ip link set "$tap" master "$BR" || + ! bridge link set dev "$tap" isolated on locked on learning off flood off guard on hairpin off || + ! bridge fdb replace "$mac" dev "$tap" master static; then + fail_closed_guest_port "$tap" || true + return 1 + fi + + port_state="$(bridge -j -details link show dev "$tap")" || { + fail_closed_guest_port "$tap" || true + return 1 + } + jq -e --arg bridge "$BR" 'length == 1 and .[0].master == $bridge and .[0].isolated == true and .[0].locked == true and .[0].learning == false and .[0].flood == false and .[0].hairpin == false and .[0].guard == true' <<<"$port_state" >/dev/null || { + fail_closed_guest_port "$tap" || true + return 1 + } + fdb_selects_port "$mac" "$tap" || { + fail_closed_guest_port "$tap" || true + return 1 + } + + if [[ -n "$address" ]]; then + guest_ip_is_usable "$address" || { + fail_closed_guest_port "$tap" || true + return 1 + } + ip neigh replace "$address" lladdr "$mac" nud permanent dev "$BR" || { + fail_closed_guest_port "$tap" || true + return 1 + } + neighbor="$(ip neigh get "$address" dev "$BR")" || { + fail_closed_guest_port "$tap" || true + return 1 + } + grep -Eiq "lladdr ${mac}([[:space:]]|$).*PERMANENT" <<<"$neighbor" || { + fail_closed_guest_port "$tap" || true + return 1 + } + fi + ip link set "$tap" up || { + fail_closed_guest_port "$tap" || true + return 1 + } +} # --- bridge ----------------------------------------------------------------- ip link show "$BR" >/dev/null 2>&1 || ip link add "$BR" type bridge ip addr replace "${SUBNET}.1/24" dev "$BR" ip link set "$BR" up sysctl -qw net.ipv4.ip_forward=1 +# Make bridged guest-to-guest frames traverse the same FORWARD policy as routed +# egress. Without bridge netfilter, two taps can bypass iptables at layer 2. +modprobe br_netfilter 2>/dev/null || [ -e /proc/sys/net/bridge/bridge-nf-call-iptables ] +sysctl -qw net.bridge.bridge-nf-call-iptables=1 +sysctl -qw "net.ipv6.conf.${BR}.disable_ipv6=1" log "bridge $BR up at ${SUBNET}.1/24, uplink=$UPLINK" -# --- dnsmasq (DHCP + DNS on the bridge only) -------------------------------- -command -v dnsmasq >/dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq dnsmasq >/dev/null; } +# Freeze existing guest taps before rebuilding shared firewall/bridge state. +# Local prototype reruns retain the authenticated FDB entry used by the narrow +# state-save race fallback; managed setup already detached every port above. +if ! freeze_all_guest_ports; then + echo "failed to disable one or more guest taps before firewall rebuild" >&2 + exit 1 +fi + +# --- dnsmasq (DHCP only; nehemiahd exclusively owns managed DNS on :53) ----- mkdir -p /etc/dnsmasq.d cat > /etc/dnsmasq.d/boring.conf </dev/null 2>&1 || true systemctl restart dnsmasq -log "dnsmasq serving DHCP+DNS on $BR" +log "dnsmasq serving DHCP only on $BR; nehemiahd owns ${SUBNET}.1:53" # --- NAT -------------------------------------------------------------------- iptables -t nat -C POSTROUTING -s "$CIDR" -o "$UPLINK" -j MASQUERADE 2>/dev/null \ || iptables -t nat -A POSTROUTING -s "$CIDR" -o "$UPLINK" -j MASQUERADE -# --- INPUT: guests may only reach the host for DHCP + DNS ------------------- +# --- INPUT: guests may only reach the host for DHCP + rate-limited DNS ------ # Allow replies to host-initiated connections (e.g. the preview proxy reaching a # guest's port) — without this the blanket DROP below kills those return packets. -iptables -C INPUT -i "$BR" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT 2>/dev/null \ - || iptables -I INPUT -i "$BR" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT -iptables -C INPUT -i "$BR" -p udp -m multiport --dports 67,53 -j ACCEPT 2>/dev/null \ - || iptables -I INPUT -i "$BR" -p udp -m multiport --dports 67,53 -j ACCEPT -iptables -C INPUT -i "$BR" -p tcp --dport 53 -j ACCEPT 2>/dev/null \ - || iptables -I INPUT -i "$BR" -p tcp --dport 53 -j ACCEPT -iptables -C INPUT -i "$BR" -j DROP 2>/dev/null \ - || iptables -A INPUT -i "$BR" -j DROP +iptables -N NEHEMIAH_INPUT 2>/dev/null || true +# Remove rules emitted by older versions of this script, then ensure our chain +# is first so an existing broad host ACCEPT cannot bypass the guest policy. +while iptables -D INPUT -i "$BR" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT 2>/dev/null; do :; done +while iptables -D INPUT -i "$BR" -p udp -m multiport --dports 67,53 -j ACCEPT 2>/dev/null; do :; done +while iptables -D INPUT -i "$BR" -p tcp --dport 53 -j ACCEPT 2>/dev/null; do :; done +while iptables -D INPUT -i "$BR" -j DROP 2>/dev/null; do :; done +while iptables -D INPUT -i "$BR" -j NEHEMIAH_INPUT 2>/dev/null; do :; done +iptables -I INPUT 1 -i "$BR" -j NEHEMIAH_INPUT +iptables -F NEHEMIAH_INPUT +iptables -A NEHEMIAH_INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT +iptables -A NEHEMIAH_INPUT -p udp --dport 67 -j ACCEPT +# DNS query flood protection is per guest IP, for both UDP and TCP fallback. +for proto in udp tcp; do + limit_name="neh-dns-${proto:0:1}" + iptables -A NEHEMIAH_INPUT -p "$proto" --dport 53 \ + -m hashlimit --hashlimit-above 50/sec --hashlimit-burst 100 \ + --hashlimit-mode srcip --hashlimit-name "$limit_name" -j DROP + iptables -A NEHEMIAH_INPUT -p "$proto" --dport 53 -j ACCEPT +done +iptables -A NEHEMIAH_INPUT -j DROP # --- FORWARD: the egress firewall ------------------------------------------ iptables -N NEHEMIAH_FWD 2>/dev/null || true -iptables -C FORWARD -j NEHEMIAH_FWD 2>/dev/null || iptables -I FORWARD -j NEHEMIAH_FWD +while iptables -D FORWARD -j NEHEMIAH_FWD 2>/dev/null; do :; done +iptables -I FORWARD 1 -j NEHEMIAH_FWD iptables -F NEHEMIAH_FWD # return traffic to guests iptables -A NEHEMIAH_FWD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT # only guest-sourced traffic is filtered below; anything else falls through iptables -A NEHEMIAH_FWD ! -s "$CIDR" -j RETURN -# block the cloud metadata endpoint + private/link-local/loopback + guest↔guest -for net in 169.254.0.0/16 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 127.0.0.0/8 100.64.0.0/10; do +# Non-overridable hard floor: metadata, current/future private space, +# loopback/link-local, CGNAT, benchmarking networks, multicast and broadcast. +# The RFC1918 blocks also prevent guest↔guest and guest→bridge traffic. +for net in \ + 0.0.0.0/8 \ + 10.0.0.0/8 \ + 100.64.0.0/10 \ + 127.0.0.0/8 \ + 169.254.0.0/16 \ + 172.16.0.0/12 \ + 192.168.0.0/16 \ + 198.18.0.0/15 \ + 224.0.0.0/4 \ + 240.0.0.0/4; do iptables -A NEHEMIAH_FWD -s "$CIDR" -d "$net" -j DROP done # no spam @@ -82,7 +291,65 @@ iptables -A NEHEMIAH_FWD -s "$CIDR" -p tcp --dport 25 -j DROP iptables -A NEHEMIAH_FWD -s "$CIDR" -p tcp --syn \ -m hashlimit --hashlimit-above 80/sec --hashlimit-burst 120 \ --hashlimit-mode srcip --hashlimit-name boringrate -j DROP +# Bound connectionless abuse as well; DNS to the bridge was already handled by +# NEHEMIAH_INPUT and never reaches this chain. +iptables -A NEHEMIAH_FWD -s "$CIDR" -p udp \ + -m hashlimit --hashlimit-above 200/sec --hashlimit-burst 400 \ + --hashlimit-mode srcip --hashlimit-name nehemiah-udp -j DROP +iptables -A NEHEMIAH_FWD -s "$CIDR" -p icmp \ + -m hashlimit --hashlimit-above 20/sec --hashlimit-burst 40 \ + --hashlimit-mode srcip --hashlimit-name nehemiah-icmp -j DROP # everything else out to the public internet is allowed iptables -A NEHEMIAH_FWD -s "$CIDR" -j ACCEPT -log "egress firewall installed (metadata + private + SMTP blocked, rate-capped)" +# Managed machines attach a NIC for host-initiated preview/file traffic, but +# outbound guest forwarding is off unless a separately verified allowlist is +# installed. Rebuild per-tap deny rules when this oneshot is rerun so flushing +# the shared chain can never silently grant egress to already-running VMs. +declare -A tap_macs=() +declare -A tap_ips=() +if [[ -r "$STATE_PATH" ]]; then + while IFS=$'\t' read -r machine_id persisted_tap persisted_ip; do + [[ -n "$machine_id" && -n "$persisted_tap" ]] || continue + [[ "$persisted_tap" == "bt$(printf '%s' "$machine_id" | sha1sum | awk '{print substr($1,1,8)}')" ]] || continue + tap_macs["$persisted_tap"]="$(guest_mac_for_id "$machine_id")" + tap_ips["$persisted_tap"]="$persisted_ip" + done < <(jq -r '.machines[]? | select(.runtime.tap? != null and .runtime.tap != "") | [.id, .runtime.tap, (.runtime.ip // "")] | @tsv' "$STATE_PATH") +fi + +tap_isolation_failed=0 +for tap_path in /sys/class/net/bt*; do + [[ -e "$tap_path" ]] || continue + tap_name="${tap_path##*/}" + [[ "$tap_name" =~ ^[A-Za-z0-9][A-Za-z0-9_.-]{0,14}$ ]] || continue + if [[ "$MANAGED_MODE" == "1" ]]; then + iptables -I NEHEMIAH_FWD 1 -i "$tap_name" -j DROP + fi + expected_mac="${tap_macs[$tap_name]:-}" + expected_ip="${tap_ips[$tap_name]:-}" + if [[ -z "$expected_mac" ]]; then + # A port created by the new daemon already has a static authenticated FDB + # entry. This fallback makes a manual network-service restart idempotent + # even if it races the next atomic state save. Old unauthenticated ports + # have no such entry and remain down/detached below. + expected_mac="$(bridge fdb show br "$BR" brport "$tap_name" state static 2>/dev/null | awk 'tolower($1) ~ /^06:00:([[:xdigit:]]{2}:){3}[[:xdigit:]]{2}$/ { print tolower($1); exit }')" + fi + if [[ -z "$expected_mac" ]] || ! secure_guest_port "$tap_name" "$expected_mac" "$expected_ip"; then + fail_closed_guest_port "$tap_name" || true + log "ERROR: tap ${tap_name} has no verifiable isolated identity; left down and detached" + tap_isolation_failed=1 + else + log "tap ${tap_name} isolated and locked to ${expected_mac}" + fi +done +# Guests have no IPv6 egress path; make that a fail-closed invariant if IPv6 is +# enabled elsewhere on the host in the future. +while ip6tables -D FORWARD -i "$BR" -j DROP 2>/dev/null; do :; done +ip6tables -I FORWARD 1 -i "$BR" -j DROP +log "egress firewall installed (metadata/private/peer/multicast blocked; DNS + egress rate-capped)" +if [[ "$tap_isolation_failed" -ne 0 ]]; then + fail_closed_all_guest_ports || true + echo "one or more guest taps failed bridge isolation" >&2 + exit 1 +fi +NETWORK_SETUP_COMPLETE=1 log "done." diff --git a/infra/latitude/provision.sh b/infra/latitude/provision.sh index a2adebc..33f8cfd 100755 --- a/infra/latitude/provision.sh +++ b/infra/latitude/provision.sh @@ -1,59 +1,554 @@ #!/usr/bin/env bash -# -# provision.sh — create a Latitude.sh bare-metal box for Nehemiah, then -# tell you how to set it up. Optional convenience for Latitude users; if you have -# a box anywhere else (Ubuntu 24.04 x86_64 + /dev/kvm), skip this and run -# ../setup.sh directly. -# -# Needs a Latitude API key + a project + an uploaded SSH key. Get them from the -# Latitude dashboard, then: -# -# LATITUDE_API_KEY=... LATITUDE_PROJECT=proj_... LATITUDE_SSH_KEY=ssh_... \ -# ./provision.sh -# -# Options (env): LATITUDE_PLAN (default c3-small-x86), LATITUDE_SITE (default MIA2), -# LATITUDE_OS (default ubuntu_24_04_x64_lts), LATITUDE_HOSTNAME (default nehemiah-metal-01). -# +# Manually render and provision one approved Latitude managed host. Bootstrap +# credentials stay in private files and never appear in curl argv or stdout. set -euo pipefail +set +x +umask 077 + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +usage() { + cat <<'EOF' +Usage: + infra/latitude/provision.sh --config FILE [--output NEW_FILE] + infra/latitude/provision.sh --config FILE --output NEW_FILE --render-only + +Live provisioning reads these non-bootstrap values from the environment or +the documented private files: + LATITUDE_API_KEY or LATITUDE_API_KEY_FILE + LATITUDE_PROJECT, LATITUDE_SSH_KEY + +--output retains the private rendered cloud-config for operator inspection. +Without it, the payload is held only in a mode-0700 temporary directory. +EOF +} + +die() { + printf 'provision: %s\n' "$*" >&2 + exit 1 +} + +log() { + printf '[provision] %s\n' "$*" +} + +CONFIG_FILE="" +OUTPUT_FILE="" +RENDER_ONLY=0 +while (($#)); do + case "$1" in + --config | --output) + (($# >= 2)) || die "missing value for $1" + option="$1" + value="$2" + shift 2 + case "$option" in + --config) + [[ -z "$CONFIG_FILE" ]] || die "duplicate --config" + CONFIG_FILE="$value" + ;; + --output) + [[ -z "$OUTPUT_FILE" ]] || die "duplicate --output" + OUTPUT_FILE="$value" + ;; + esac + ;; + --render-only) + [[ "$RENDER_ONLY" == 0 ]] || die "duplicate --render-only" + RENDER_ONLY=1 + shift + ;; + -h | --help) + usage + exit 0 + ;; + *) die "unknown argument: $1" ;; + esac +done +[[ -n "$CONFIG_FILE" ]] || { usage >&2; exit 2; } +if [[ "$RENDER_ONLY" == 1 && -z "$OUTPUT_FILE" ]]; then + die "--render-only requires --output" +fi + +WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/nehemiah-provision.XXXXXX")" +chmod 0700 "$WORK_DIR" +USER_DATA_ID="" +API_READY=0 + +latitude_request() { + local method="$1" path="$2" request_file="$3" response_file="$4" expected="$5" + local code + local -a arguments=( + --config "$WORK_DIR/curl.conf" + --request "$method" + --output "$response_file" + --write-out '%{http_code}' + ) + if [[ -n "$request_file" ]]; then + arguments+=(--data-binary "@$request_file") + fi + : > "$response_file" + if ! code="$(curl "${arguments[@]}" "${LATITUDE_API_BASE}${path}")"; then + printf 'provision: Latitude request transport failed for %s %s\n' \ + "$method" "$path" >&2 + return 1 + fi + if [[ " $expected " != *" $code "* ]]; then + printf 'provision: Latitude request %s %s returned HTTP %s\n' \ + "$method" "$path" "$code" >&2 + return 1 + fi +} + +delete_user_data() { + [[ -n "$USER_DATA_ID" && "$API_READY" == 1 ]] || return 0 + if latitude_request DELETE "/user_data/$USER_DATA_ID" "" \ + "$WORK_DIR/delete-user-data-response.json" "200 202 204 404"; then + log "deleted one-time Latitude user-data record $USER_DATA_ID" + USER_DATA_ID="" + else + printf 'provision: could not delete one-time user-data record %s\n' \ + "$USER_DATA_ID" >&2 + fi +} + +cleanup() { + local status=$? + set +e + delete_user_data + rm -rf -- "$WORK_DIR" + return "$status" +} +trap cleanup EXIT + +if [[ -n "$OUTPUT_FILE" ]]; then + RENDERED_USER_DATA="$OUTPUT_FILE" +else + RENDERED_USER_DATA="$WORK_DIR/user-data.yaml" +fi +"$SCRIPT_DIR/render-user-data.sh" \ + --config "$CONFIG_FILE" \ + --output "$RENDERED_USER_DATA" +if [[ "$RENDER_ONLY" == 1 ]]; then + log "render-only complete; no Latitude API request was made" + exit 0 +fi + +STATE_DIR="${LATITUDE_STATE_DIR:-${XDG_CONFIG_HOME:-${HOME}/.config}/latitude}" +API_KEY_FILE="${LATITUDE_API_KEY_FILE:-$STATE_DIR/api_key}" +API_KEY="${LATITUDE_API_KEY:-}" +if [[ -z "$API_KEY" ]]; then + [[ -f "$API_KEY_FILE" && ! -L "$API_KEY_FILE" ]] \ + || die "set LATITUDE_API_KEY or create the private API key file $API_KEY_FILE" + [[ "$(stat -c '%a:%u' "$API_KEY_FILE")" == "600:$(id -u)" ]] \ + || die "Latitude API key file must be mode 0600 and owned by the invoking user" + API_KEY="$(< "$API_KEY_FILE")" +fi +[[ "$API_KEY" =~ ^[A-Za-z0-9][A-Za-z0-9._~+/-]{19,4095}$ ]] \ + || die "Latitude API key has an invalid format" -: "${LATITUDE_API_KEY:?set LATITUDE_API_KEY}" : "${LATITUDE_PROJECT:?set LATITUDE_PROJECT (proj_...)}" -: "${LATITUDE_SSH_KEY:?set LATITUDE_SSH_KEY (ssh_... — an uploaded SSH key id)}" -# Latitude's ssh_keys create field does not reliably inject the key; the reliable -# path is a cloud-init user_data that appends the key to /root/.ssh/authorized_keys. -# Set LATITUDE_USER_DATA to a user_data id (ud_...) that does this, or SSH will fail. -USER_DATA="${LATITUDE_USER_DATA:-}" +: "${LATITUDE_SSH_KEY:?set LATITUDE_SSH_KEY (ssh_...)}" PLAN="${LATITUDE_PLAN:-c3-small-x86}" SITE="${LATITUDE_SITE:-MIA2}" -OS="${LATITUDE_OS:-ubuntu_24_04_x64_lts}" -HOST="${LATITUDE_HOSTNAME:-nehemiah-metal-01}" - -log() { printf '\033[1;34m[provision]\033[0m %s\n' "$*"; } - -log "Creating ${PLAN} @ ${SITE} (${OS}) in ${LATITUDE_PROJECT}…" -RESP="$(curl -sS -g --max-time 90 -X POST 'https://api.latitude.sh/servers' \ - -H "Authorization: Bearer ${LATITUDE_API_KEY}" \ - -H 'Accept: application/vnd.api+json' -H 'Content-Type: application/vnd.api+json' \ - -d "{\"data\":{\"type\":\"servers\",\"attributes\":{\"project\":\"${LATITUDE_PROJECT}\",\"plan\":\"${PLAN}\",\"site\":\"${SITE}\",\"operating_system\":\"${OS}\",\"hostname\":\"${HOST}\",\"ssh_keys\":[\"${LATITUDE_SSH_KEY}\"]${USER_DATA:+,\"user_data\":\"${USER_DATA}\"},\"billing\":\"hourly\"}}}")" - -ID="$(printf '%s' "${RESP}" | python3 -c 'import sys,json;d=json.load(sys.stdin);print(d.get("data",{}).get("id","") if "errors" not in d else "ERR:"+json.dumps(d["errors"])[:300])')" -[[ "${ID}" == ERR:* || -z "${ID}" ]] && { echo "provision failed: ${ID:-$RESP}" >&2; exit 1; } - -log "Server ${ID} created. Waiting for it to come online + get an IP…" -IP="" -for _ in $(seq 1 60); do - INFO="$(curl -sS -g --max-time 30 "https://api.latitude.sh/servers/${ID}" \ - -H "Authorization: Bearer ${LATITUDE_API_KEY}" -H 'Accept: application/vnd.api+json')" - IP="$(printf '%s' "${INFO}" | python3 -c 'import sys,json;a=json.load(sys.stdin).get("data",{}).get("attributes",{});print(a.get("primary_ipv4") or "")')" - ST="$(printf '%s' "${INFO}" | python3 -c 'import sys,json;print(json.load(sys.stdin).get("data",{}).get("attributes",{}).get("status",""))')" - [[ -n "${IP}" && "${ST}" == "on" ]] && break - sleep 15 +HOSTNAME_BASE="${LATITUDE_HOSTNAME:-nehemiah-metal-01}" +LATITUDE_API_BASE="${LATITUDE_API_BASE:-https://api.latitude.sh}" +POLL_ATTEMPTS="${LATITUDE_POLL_ATTEMPTS:-60}" +POLL_INTERVAL_SECONDS="${LATITUDE_POLL_INTERVAL_SECONDS:-15}" + +mapfile -t provider_image_config < <(python3 - "$CONFIG_FILE" <<'PY' +import pathlib +import re +import sys + +wanted = { + "LATITUDE_OS_ID", + "LATITUDE_OS_SLUG", + "LATITUDE_OS_VERSION", + "LATITUDE_OS_ARCH", +} +values = {} +for line in pathlib.Path(sys.argv[1]).read_text().splitlines(): + if not line or line.startswith("#"): + continue + match = re.fullmatch(r"([A-Z][A-Z0-9_]*)=(.*)", line) + if match and match.group(1) in wanted: + if match.group(1) in values: + raise SystemExit("duplicate provider image field") + values[match.group(1)] = match.group(2) +if set(values) != wanted or any("\n" in value or "\r" in value for value in values.values()): + raise SystemExit("provider image identity is incomplete") +for key in ("LATITUDE_OS_ID", "LATITUDE_OS_SLUG", "LATITUDE_OS_VERSION", "LATITUDE_OS_ARCH"): + print(values[key]) +PY +) +[[ "${#provider_image_config[@]}" -eq 4 ]] \ + || die "could not read the exact provider image identity from the private config" +LATITUDE_OS_ID="${provider_image_config[0]}" +OPERATING_SYSTEM="${provider_image_config[1]}" +LATITUDE_OS_VERSION="${provider_image_config[2]}" +LATITUDE_OS_ARCH="${provider_image_config[3]}" +unset provider_image_config + +[[ "$LATITUDE_PROJECT" =~ ^proj_[A-Za-z0-9_-]{4,128}$ ]] \ + || die "invalid LATITUDE_PROJECT" +[[ "$LATITUDE_SSH_KEY" =~ ^ssh_[A-Za-z0-9_-]{4,128}$ ]] \ + || die "invalid LATITUDE_SSH_KEY" +[[ "$PLAN" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ ]] || die "invalid LATITUDE_PLAN" +[[ "$SITE" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,31}$ ]] || die "invalid LATITUDE_SITE" +[[ "$LATITUDE_OS_ID" =~ ^os_[A-Za-z0-9_-]{4,128}$ ]] \ + || die "invalid LATITUDE_OS_ID" +[[ "$LATITUDE_OS_VERSION" =~ ^24\.04([.[:space:]A-Za-z0-9_-]{0,63})$ ]] \ + || die "invalid LATITUDE_OS_VERSION" +case "$LATITUDE_OS_ARCH:$OPERATING_SYSTEM" in + amd64:ubuntu_24_04_x64_lts | arm64:ubuntu_24_04_arm64_lts) ;; + *) die "provider image architecture and slug do not match" ;; +esac +# The hostname is the durable recovery correlation value for this run's +# billable create, so it must never be shared by two servers. Latitude +# enforces neither hostname uniqueness nor create idempotency, so uniqueness +# is generated locally: every run appends a random suffix, which makes +# concurrent runs (even with the same LATITUDE_HOSTNAME) use distinct +# correlation values. The provider caps hostnames at 32 characters, so the +# base leaves room for the 9-character suffix. +[[ "$HOSTNAME_BASE" =~ ^[A-Za-z0-9]([A-Za-z0-9.-]{0,21}[A-Za-z0-9])?$ ]] \ + || die "invalid LATITUDE_HOSTNAME: up to 23 characters (letters, digits, dots, hyphens; alphanumeric at both ends) so the unique per-run suffix fits the provider's 32-character hostname limit" +HOSTNAME_SUFFIX="$(od -An -N4 -tx1 /dev/urandom | tr -d ' \n')" +[[ "$HOSTNAME_SUFFIX" =~ ^[0-9a-f]{8}$ ]] \ + || die "could not generate a random hostname suffix" +HOSTNAME_VALUE="${HOSTNAME_BASE}-${HOSTNAME_SUFFIX}" +[[ "$POLL_ATTEMPTS" =~ ^[1-9][0-9]{0,2}$ && "$POLL_ATTEMPTS" -le 240 ]] \ + || die "LATITUDE_POLL_ATTEMPTS must be between 1 and 240" +[[ "$POLL_INTERVAL_SECONDS" =~ ^[0-9]{1,2}$ && "$POLL_INTERVAL_SECONDS" -le 60 ]] \ + || die "LATITUDE_POLL_INTERVAL_SECONDS must be between 0 and 60" + +if [[ "${LATITUDE_ALLOW_HTTP_FOR_TESTS:-0}" == 1 ]]; then + [[ "$LATITUDE_API_BASE" =~ ^http://127\.0\.0\.1:[0-9]{1,5}$ ]] \ + || die "the HTTP test escape hatch is restricted to IPv4 loopback" + CURL_PROTOCOL=http +else + [[ "$LATITUDE_API_BASE" == "https://api.latitude.sh" ]] \ + || die "Latitude API base must be exactly https://api.latitude.sh" + [[ "$POLL_INTERVAL_SECONDS" -ge 1 ]] \ + || die "live provisioning requires a positive poll interval" + CURL_PROTOCOL=https +fi + +cat > "$WORK_DIR/curl.conf" < 100: + raise SystemExit("Latitude returned an invalid operating-system page") +matches = [entry for entry in data if isinstance(entry, dict) and entry.get("id") == image_id] +if len(matches) > 1: + raise SystemExit("Latitude returned a duplicate operating-system id") +if not matches: + print("more" if len(data) == 100 else "end") + raise SystemExit(0) + +entry = matches[0] +attributes = entry.get("attributes") +if entry.get("type") != "operating_system" or not isinstance(attributes, dict): + raise SystemExit("Latitude operating-system identity has an invalid type") +provisionable = attributes.get("provisionable_on") +expected_architecture = { + "ubuntu_24_04_x64_lts": "amd64", + "ubuntu_24_04_arm64_lts": "arm64", +}.get(attributes.get("slug")) +if ( + attributes.get("slug") != slug + or attributes.get("version") != version + or expected_architecture != architecture + or not isinstance(provisionable, list) + or not provisionable + or not all( + isinstance(value, str) + and re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}", value) + for value in provisionable + ) + or len(provisionable) != len(set(provisionable)) + or plan not in provisionable +): + raise SystemExit("Latitude operating-system record differs from the approved image identity") +evidence = { + "architecture": architecture, + "id": image_id, + "plan": plan, + "provisionableOn": sorted(provisionable), + "provider": "latitude", + "slug": slug, + "version": version, +} +path = pathlib.Path(evidence_path) +path.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n") +path.chmod(0o600) +print("found") +PY +)" + case "$provider_image_status" in + found) + provider_image_found=1 + break + ;; + end) break ;; + more) ;; + *) die "could not validate the Latitude operating-system response" ;; + esac +done +[[ "$provider_image_found" == 1 ]] \ + || die "approved Latitude operating-system id was not found within the bounded inventory" +log "validated approved Latitude operating-system id $LATITUDE_OS_ID for $PLAN" + +# Defense-in-depth for the recovery correlation value: the per-run random +# suffix already makes duplicate hostnames vanishingly unlikely, but teardown's +# hostname recovery requires a single match, so refuse to create a second +# server behind an already-live hostname (a pre-existing operator-created +# duplicate, or a suffix collision) — a duplicate would strand a billed host +# whose create response was unparseable. HOSTNAME_VALUE is built above from +# URL-safe characters only. +latitude_request GET \ + "/servers?filter%5Bhostname%5D=${HOSTNAME_VALUE}&page%5Bsize%5D=200" \ + "" "$WORK_DIR/hostname-collision.json" 200 +python3 - "$WORK_DIR/hostname-collision.json" "$HOSTNAME_VALUE" <<'PY' +import json +import pathlib +import sys + +response_path, hostname = sys.argv[1:] +try: + response = json.loads(pathlib.Path(response_path).read_text()) +except (OSError, json.JSONDecodeError): + raise SystemExit("Latitude returned an invalid server-listing response") from None +data = response.get("data") if isinstance(response, dict) else None +if not isinstance(data, list): + raise SystemExit("Latitude returned an invalid server-listing page") +matches = [ + entry + for entry in data + if isinstance(entry, dict) + and isinstance(entry.get("attributes"), dict) + and entry["attributes"].get("hostname") == hostname +] +if matches: + raise SystemExit( + f"a live server already uses hostname {hostname!r}, so hostname recovery" + " would be ambiguous. Tear that host down first or set a different" + " LATITUDE_HOSTNAME." + ) +PY +log "hostname $HOSTNAME_VALUE is not in use by any live server" + +python3 - "$RENDERED_USER_DATA" "$LATITUDE_PROJECT" "$HOSTNAME_VALUE" \ + > "$WORK_DIR/create-user-data.json" <<'PY' +import base64 +import json +import pathlib +import sys + +payload = pathlib.Path(sys.argv[1]).read_bytes() +if not payload.startswith(b"#cloud-config\n") or len(payload) > 1024 * 1024: + raise SystemExit("rendered user-data is invalid or too large") +print(json.dumps({ + "data": { + "type": "user_data", + "attributes": { + "content": base64.b64encode(payload).decode("ascii"), + "description": f"one-time-nehemiah-bootstrap-{sys.argv[3]}", + "project": sys.argv[2], + }, + }, +}, separators=(",", ":"))) +PY +chmod 0600 "$WORK_DIR/create-user-data.json" +latitude_request POST /user_data "$WORK_DIR/create-user-data.json" \ + "$WORK_DIR/create-user-data-response.json" 201 +USER_DATA_ID="$(python3 - "$WORK_DIR/create-user-data-response.json" <<'PY' +import json +import pathlib +import re +import sys + +identifier = json.loads(pathlib.Path(sys.argv[1]).read_text())["data"]["id"] +if not isinstance(identifier, str) or not re.fullmatch(r"ud_[A-Za-z0-9_-]{4,128}", identifier): + raise SystemExit("Latitude returned an invalid user-data id") +print(identifier) +PY +)" +log "created one-time Latitude user-data record $USER_DATA_ID" + +python3 - "$LATITUDE_PROJECT" "$PLAN" "$SITE" "$OPERATING_SYSTEM" \ + "$HOSTNAME_VALUE" "$LATITUDE_SSH_KEY" "$USER_DATA_ID" \ + > "$WORK_DIR/create-server.json" <<'PY' +import json +import sys + +project, plan, site, operating_system, hostname, ssh_key, user_data = sys.argv[1:] +print(json.dumps({ + "data": { + "type": "servers", + "attributes": { + "project": project, + "plan": plan, + "site": site, + "operating_system": operating_system, + "hostname": hostname, + "ssh_keys": [ssh_key], + "user_data": user_data, + "billing": "hourly", + }, + }, +}, separators=(",", ":"))) +PY +chmod 0600 "$WORK_DIR/create-server.json" +latitude_request POST /servers "$WORK_DIR/create-server.json" \ + "$WORK_DIR/create-server-response.json" 201 +# Latitude has accepted the (billable) create. Persist durable recovery inputs +# BEFORE parsing so an accepted-but-unparseable response can still be identified +# and torn down — WORK_DIR is removed on exit. The requested hostname is the +# durable correlation value teardown.sh uses to find the server via a provider +# lookup when the id cannot be validated from the response body. +mkdir -p "$STATE_DIR" +chmod 0700 "$STATE_DIR" +# Commit the new host's recovery record before touching the previously recorded +# server_id: clearing first would open a window (a kill between the rm and the +# rename) where the state directory identifies NO host at all and teardown could +# not discover the one that is now billing. The record is staged inside +# STATE_DIR so the final mv is an atomic rename (WORK_DIR may be on another +# filesystem). +hostname_record_tmp="$(mktemp "$STATE_DIR/.last-created-hostname.XXXXXX")" +printf '%s\n' "$HOSTNAME_VALUE" > "$hostname_record_tmp" +chmod 0600 "$hostname_record_tmp" +mv -- "$hostname_record_tmp" "$STATE_DIR/last-created-hostname" +raw_creation_tmp="$(mktemp "$STATE_DIR/.created-server.XXXXXX")" +cp -- "$WORK_DIR/create-server-response.json" "$raw_creation_tmp" +chmod 0600 "$raw_creation_tmp" +mv -- "$raw_creation_tmp" "$STATE_DIR/last-created-server.json" +# Only now drop the stale server_id — this host is already recoverable by +# hostname. Clearing before the id parse below keeps the parse-failure path +# safe: teardown falls through to hostname recovery for THIS host instead of +# deleting the prior server. (A kill before this rm leaves the stale id +# alongside the new hostname record, which teardown would target first; that +# state is recoverable — remove the server_id file and rerun teardown — unlike +# a window holding no record of the new host at all.) +rm -f -- "$STATE_DIR/server_id" +log "persisted recovery inputs (hostname + raw response) under $STATE_DIR for teardown" +if ! SERVER_ID="$(python3 - "$WORK_DIR/create-server-response.json" <<'PY' +import json +import pathlib +import re +import sys + +identifier = json.loads(pathlib.Path(sys.argv[1]).read_text())["data"]["id"] +if not isinstance(identifier, str) or not re.fullmatch(r"sv_[A-Za-z0-9_-]{4,128}", identifier): + raise SystemExit("Latitude returned an invalid server id") +print(identifier) +PY +)"; then + die "Latitude accepted the billable create but returned an unparseable server id. +A host may be billing now. Do NOT read the id from the malformed body +($STATE_DIR/last-created-server.json); recover it by its hostname: + LATITUDE_HOSTNAME='$HOSTNAME_VALUE' infra/latitude/teardown.sh" +fi +# Persist the validated id before polling so teardown can discover it even if the +# status poll below fails after the host is already billing. +if [[ ! -e "$STATE_DIR/server_id" && ! -L "$STATE_DIR/server_id" ]]; then + server_id_tmp="$(mktemp "$STATE_DIR/.server_id.XXXXXX")" + printf '%s\n' "$SERVER_ID" > "$server_id_tmp" + chmod 0600 "$server_id_tmp" + mv -- "$server_id_tmp" "$STATE_DIR/server_id" + log "saved the server id to $STATE_DIR/server_id" +else + log "left existing $STATE_DIR/server_id unchanged; use LATITUDE_SERVER_ID=$SERVER_ID for teardown" +fi +log "created hourly-billed server $SERVER_ID; waiting for provider status=on" + +SERVER_IP="" +SERVER_STATUS="" +for ((attempt = 1; attempt <= POLL_ATTEMPTS; attempt += 1)); do + latitude_request GET "/servers/$SERVER_ID" "" \ + "$WORK_DIR/get-server-response.json" 200 + read -r SERVER_STATUS SERVER_IP < <( + python3 - "$WORK_DIR/get-server-response.json" <<'PY' +import ipaddress +import json +import pathlib +import sys + +attributes = json.loads(pathlib.Path(sys.argv[1]).read_text())["data"]["attributes"] +status = attributes.get("status", "") +address = attributes.get("primary_ipv4") or "" +if address: + ipaddress.ip_address(address) +print(status, address) +PY + ) + [[ "$SERVER_STATUS" == on && -n "$SERVER_IP" ]] && break + sleep "$POLL_INTERVAL_SECONDS" done -[[ -n "${IP}" ]] || { echo "server created (${ID}) but no IP yet — check the Latitude dashboard" >&2; exit 1; } - -log "Ready: ${HOST} = ${IP} (server ${ID})." -echo -echo "Next — set it up (from the repo root):" -echo " NEHEMIAH_ANTHROPIC_KEY=sk-ant-... ./infra/setup.sh root@${IP}" -echo -echo "To delete it later (stops billing): ./infra/latitude/teardown.sh (server_id ${ID})" +if [[ "$SERVER_STATUS" != on || -z "$SERVER_IP" ]]; then + die "server $SERVER_ID exists but did not become online; inspect it before billing continues" +fi + +# Latitude has consumed this record for the completed deployment. Delete the +# reusable API object; the host also erases cloud-init's cached payload after +# successful control-plane enrollment. +delete_user_data + +mkdir -p "$STATE_DIR" +chmod 0700 "$STATE_DIR" +# The server id was persisted before polling (above); only provider evidence +# remains to record now that the host is confirmed online. +provider_evidence_path="$STATE_DIR/provider-image-${SERVER_ID}.json" +if [[ ! -e "$provider_evidence_path" && ! -L "$provider_evidence_path" ]]; then + provider_evidence_tmp="$(mktemp "$STATE_DIR/.provider-image.XXXXXX")" + cp -- "$provider_image_evidence" "$provider_evidence_tmp" + chmod 0600 "$provider_evidence_tmp" + mv "$provider_evidence_tmp" "$provider_evidence_path" + log "saved provider image evidence to $provider_evidence_path" +else + log "left existing provider image evidence unchanged: $provider_evidence_path" +fi + +log "server $SERVER_ID is online at $SERVER_IP" +printf '%s\n' \ + "Cloud-init may still be enrolling. Verify the host in the control plane and" \ + "check /var/lib/nehemiahd/bootstrap.complete before admitting workloads." \ + "To stop hourly billing: LATITUDE_SERVER_ID=$SERVER_ID infra/latitude/teardown.sh" diff --git a/infra/latitude/render-user-data.sh b/infra/latitude/render-user-data.sh new file mode 100755 index 0000000..a89a436 --- /dev/null +++ b/infra/latitude/render-user-data.sh @@ -0,0 +1,396 @@ +#!/usr/bin/env bash +# Render a private, self-contained cloud-config for one approved managed host. +# Secrets are read from a mode-0600 config file and are never written to stdout. +set -euo pipefail +set +x +umask 077 + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +usage() { + cat <<'EOF' +Usage: infra/latitude/render-user-data.sh --config FILE --output NEW_FILE + +FILE must be a non-symlink, mode-0600 KEY=value file. NEW_FILE must not +already exist; it is created mode 0600 because it contains recoverable +base64-encoded bootstrap credentials. +EOF +} + +die() { + printf 'render-user-data: %s\n' "$*" >&2 + exit 1 +} + +CONFIG_FILE="" +OUTPUT_FILE="" +while (($#)); do + case "$1" in + --config | --output) + (($# >= 2)) || die "missing value for $1" + option="$1" + value="$2" + shift 2 + case "$option" in + --config) + [[ -z "$CONFIG_FILE" ]] || die "duplicate --config" + CONFIG_FILE="$value" + ;; + --output) + [[ -z "$OUTPUT_FILE" ]] || die "duplicate --output" + OUTPUT_FILE="$value" + ;; + esac + ;; + -h | --help) + usage + exit 0 + ;; + *) die "unknown argument: $1" ;; + esac +done + +[[ -n "$CONFIG_FILE" && -n "$OUTPUT_FILE" ]] || { usage >&2; exit 2; } +[[ -f "$CONFIG_FILE" && ! -L "$CONFIG_FILE" ]] \ + || die "config must be a regular, non-symlink file" +[[ "$(stat -c '%a:%u' "$CONFIG_FILE")" == "600:$(id -u)" ]] \ + || die "config must be mode 0600 and owned by the invoking user" +[[ -f "$SCRIPT_DIR/cloud-init.sh" && ! -L "$SCRIPT_DIR/cloud-init.sh" ]] \ + || die "cloud-init.sh is missing or is a symlink" +for required_script in validate-managed-release.py verify-minisign.py wireguard-config.py; do + [[ -f "$SCRIPT_DIR/$required_script" && ! -L "$SCRIPT_DIR/$required_script" ]] \ + || die "$required_script is missing or is a symlink" +done +[[ ! -e "$OUTPUT_FILE" && ! -L "$OUTPUT_FILE" ]] \ + || die "output already exists; refusing to overwrite it" +OUTPUT_PARENT="$(dirname "$OUTPUT_FILE")" +[[ -d "$OUTPUT_PARENT" && ! -L "$OUTPUT_PARENT" ]] \ + || die "output parent must be an existing, non-symlink directory" + +declare -A allowed=() +for key in \ + NEHEMIAH_RELEASE_BASE \ + NEHEMIAH_RELEASE_VERSION \ + NEHEMIAH_RELEASE_MINISIGN_KEY \ + NEHEMIAH_HOST_ID \ + NEHEMIAH_REGION \ + NEHEMIAH_PROVIDER_ID \ + LATITUDE_OS_ID \ + LATITUDE_OS_SLUG \ + LATITUDE_OS_VERSION \ + LATITUDE_OS_ARCH \ + NEHEMIAH_FLEET_BOOTSTRAP_TOKEN \ + NEHEMIAH_CONTROL_PLANE_URL \ + NEHEMIAH_TEMPLATE_OBJECT_ORIGIN \ + NEHEMIAH_ADVERTISE_ADDRESS \ + NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS \ + NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS \ + NEHEMIAH_WIREGUARD_CONFIG_B64 \ + NEHEMIAH_OTEL_ENABLED \ + NEHEMIAH_OTEL_ENDPOINT \ + NEHEMIAH_OTEL_AUTHORIZATION \ + NEHEMIAH_SERVICE_VERSION \ + NEHEMIAH_INSTANCE_ID \ + NEHEMIAH_DEPLOYMENT_ENVIRONMENT \ + NEHEMIAH_OTEL_EXPORT_INTERVAL_MS \ + NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS \ + NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO \ + NEHEMIAH_ROOT_SSH_AUTHORIZED_KEY_B64; do + allowed["$key"]=1 +done + +declare -A values=() +line_number=0 +while IFS= read -r line || [[ -n "$line" ]]; do + ((line_number += 1)) + [[ -z "$line" || "$line" == \#* ]] && continue + [[ "$line" =~ ^([A-Z][A-Z0-9_]*)=(.*)$ ]] \ + || die "config line $line_number must be KEY=value without shell syntax" + key="${BASH_REMATCH[1]}" + value="${BASH_REMATCH[2]}" + [[ -n "${allowed[$key]:-}" ]] || die "unknown config key on line $line_number" + [[ ! -v "values[$key]" ]] || die "duplicate config key on line $line_number" + values["$key"]="$value" +done < "$CONFIG_FILE" + +required=( + NEHEMIAH_RELEASE_BASE + NEHEMIAH_RELEASE_VERSION + NEHEMIAH_RELEASE_MINISIGN_KEY + NEHEMIAH_HOST_ID + NEHEMIAH_REGION + LATITUDE_OS_ID + LATITUDE_OS_SLUG + LATITUDE_OS_VERSION + LATITUDE_OS_ARCH + NEHEMIAH_FLEET_BOOTSTRAP_TOKEN + NEHEMIAH_CONTROL_PLANE_URL + NEHEMIAH_ADVERTISE_ADDRESS + NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS + NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS + NEHEMIAH_WIREGUARD_CONFIG_B64 + NEHEMIAH_OTEL_ENABLED + NEHEMIAH_OTEL_ENDPOINT + NEHEMIAH_OTEL_AUTHORIZATION + NEHEMIAH_SERVICE_VERSION + NEHEMIAH_INSTANCE_ID + NEHEMIAH_DEPLOYMENT_ENVIRONMENT + NEHEMIAH_OTEL_EXPORT_INTERVAL_MS + NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS + NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO + NEHEMIAH_ROOT_SSH_AUTHORIZED_KEY_B64 +) +for key in "${required[@]}"; do + [[ -n "${values[$key]:-}" ]] || die "missing required config key $key" +done +values[NEHEMIAH_PROVIDER_ID]="${values[NEHEMIAH_PROVIDER_ID]:-${values[NEHEMIAH_HOST_ID]}}" + +[[ "${values[NEHEMIAH_RELEASE_VERSION]}" =~ ^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-[0-9A-Za-z.-]+)?(\+[0-9A-Za-z.-]+)?$ ]] \ + || die "invalid release version" +[[ "${values[NEHEMIAH_RELEASE_MINISIGN_KEY]}" =~ ^RW[A-Za-z0-9+/]{54}$ ]] \ + || die "invalid Minisign public key" +for key in NEHEMIAH_HOST_ID NEHEMIAH_REGION NEHEMIAH_PROVIDER_ID; do + [[ "${values[$key]}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ ]] \ + || die "invalid $key" +done +[[ "${values[LATITUDE_OS_ID]}" =~ ^os_[A-Za-z0-9_-]{4,128}$ ]] \ + || die "invalid LATITUDE_OS_ID" +[[ "${values[LATITUDE_OS_SLUG]}" =~ ^ubuntu_24_04_(x64|arm64)_lts$ ]] \ + || die "LATITUDE_OS_SLUG must be an approved Ubuntu 24.04 image slug" +[[ "${values[LATITUDE_OS_VERSION]}" =~ ^24\.04([.[:space:]A-Za-z0-9_-]{0,63})$ ]] \ + || die "invalid LATITUDE_OS_VERSION" +case "${values[LATITUDE_OS_ARCH]}:${values[LATITUDE_OS_SLUG]}" in + amd64:ubuntu_24_04_x64_lts | arm64:ubuntu_24_04_arm64_lts) ;; + *) die "Latitude image architecture and slug do not match" ;; +esac +[[ "${values[NEHEMIAH_FLEET_BOOTSTRAP_TOKEN]}" =~ ^nhe_[A-Za-z0-9_-]{43}$ ]] \ + || die "invalid fleet bootstrap credential" +[[ "${values[NEHEMIAH_OTEL_ENABLED]}" == true ]] \ + || die "NEHEMIAH_OTEL_ENABLED must be exactly true" +[[ ${#values[NEHEMIAH_OTEL_AUTHORIZATION]} -ge 16 && \ + ${#values[NEHEMIAH_OTEL_AUTHORIZATION]} -le 4096 && \ + "${values[NEHEMIAH_OTEL_AUTHORIZATION]}" =~ ^[A-Za-z][A-Za-z0-9_-]{0,31}\ [-A-Za-z0-9._~+/=]+$ ]] \ + || die "invalid OTLP authorization value" +otel_credential="${values[NEHEMIAH_OTEL_AUTHORIZATION]#* }" +[[ "${values[NEHEMIAH_OTEL_AUTHORIZATION]}" != "${values[NEHEMIAH_FLEET_BOOTSTRAP_TOKEN]}" && \ + "$otel_credential" != "${values[NEHEMIAH_FLEET_BOOTSTRAP_TOKEN]}" ]] \ + || die "OTLP authorization must be distinct from the fleet credential" +for key in NEHEMIAH_SERVICE_VERSION NEHEMIAH_INSTANCE_ID; do + [[ "${values[$key]}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ ]] \ + || die "invalid $key" +done +case "${values[NEHEMIAH_DEPLOYMENT_ENVIRONMENT]}" in + staging | production) ;; + *) die "managed telemetry environment must be staging or production" ;; +esac +[[ "${values[NEHEMIAH_OTEL_EXPORT_INTERVAL_MS]}" =~ ^[1-9][0-9]*$ && \ + ${values[NEHEMIAH_OTEL_EXPORT_INTERVAL_MS]} -ge 5000 && \ + ${values[NEHEMIAH_OTEL_EXPORT_INTERVAL_MS]} -le 300000 ]] \ + || die "invalid OTLP export interval" +[[ "${values[NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS]}" =~ ^[1-9][0-9]*$ && \ + ${values[NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS]} -ge 1000 && \ + ${values[NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS]} -le 30000 && \ + ${values[NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS]} -lt ${values[NEHEMIAH_OTEL_EXPORT_INTERVAL_MS]} ]] \ + || die "invalid OTLP export timeout" +[[ "${values[NEHEMIAH_WIREGUARD_CONFIG_B64]}" =~ ^[A-Za-z0-9+/]*={0,2}$ && \ + ${#values[NEHEMIAH_WIREGUARD_CONFIG_B64]} -le 131072 ]] \ + || die "invalid WireGuard configuration encoding" +[[ "${values[NEHEMIAH_ROOT_SSH_AUTHORIZED_KEY_B64]}" =~ ^[A-Za-z0-9+/]*={0,2}$ && \ + ${#values[NEHEMIAH_ROOT_SSH_AUTHORIZED_KEY_B64]} -le 32768 ]] \ + || die "invalid SSH public key encoding" + +python3 - "${values[NEHEMIAH_RELEASE_BASE]}" \ + "${values[NEHEMIAH_CONTROL_PLANE_URL]}" \ + "${values[NEHEMIAH_ADVERTISE_ADDRESS]}" \ + "${values[NEHEMIAH_TEMPLATE_OBJECT_ORIGIN]:-}" \ + "${values[NEHEMIAH_OTEL_ENDPOINT]}" \ + "${values[NEHEMIAH_REGION]}" \ + "${values[NEHEMIAH_INSTANCE_ID]}" \ + "${values[NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO]}" <<'PY' +import ipaddress +import math +import re +import sys +from urllib.parse import urlsplit + +for label, raw in (("release base", sys.argv[1]), ("control-plane URL", sys.argv[2])): + if not re.fullmatch(r"https://[A-Za-z0-9.-]+(?::[0-9]{1,5})?(?:/[A-Za-z0-9._~/-]*)?", raw): + raise SystemExit(f"invalid {label}") + parsed = urlsplit(raw) + if parsed.username or parsed.password or parsed.query or parsed.fragment or not parsed.hostname: + raise SystemExit(f"invalid {label}") + try: + if parsed.port is not None and not 1 <= parsed.port <= 65535: + raise ValueError + except ValueError: + raise SystemExit(f"invalid {label}") from None +ipaddress.ip_address(sys.argv[3]) +if sys.argv[4]: + parsed = urlsplit(sys.argv[4]) + if parsed.scheme != "https" or not parsed.hostname or parsed.username or parsed.password or parsed.path not in ("", "/") or parsed.query or parsed.fragment: + raise SystemExit("invalid template object origin") + +otel_endpoint = urlsplit(sys.argv[5]) +if otel_endpoint.scheme != "https" or not otel_endpoint.hostname or otel_endpoint.username or otel_endpoint.password or otel_endpoint.path not in ("", "/") or otel_endpoint.query or otel_endpoint.fragment: + raise SystemExit("invalid OTLP endpoint") +try: + ipaddress.ip_address(otel_endpoint.hostname) +except ValueError: + pass +else: + raise SystemExit("OTLP endpoint must use a DNS hostname") +try: + if otel_endpoint.port is not None and not 1 <= otel_endpoint.port <= 65535: + raise ValueError +except ValueError: + raise SystemExit("invalid OTLP endpoint port") from None +for label, identity in (("region", sys.argv[6]), ("instance id", sys.argv[7])): + try: + ipaddress.ip_address(identity) + except ValueError: + pass + else: + raise SystemExit(f"{label} must not be an IP address") +try: + sample_ratio = float(sys.argv[8]) +except ValueError: + raise SystemExit("invalid OTLP trace sample ratio") from None +if not math.isfinite(sample_ratio) or not 0.001 <= sample_ratio <= 1: + raise SystemExit("invalid OTLP trace sample ratio") +PY + +validate_encoded_file() { + local kind="$1" encoded="$2" + printf '%s' "$encoded" | python3 -c ' +import base64 +import binascii +import re +import sys + +kind = sys.argv[1] +encoded = sys.stdin.buffer.read() +try: + decoded = base64.b64decode(encoded, validate=True) +except binascii.Error: + raise SystemExit(f"invalid {kind} base64") from None +if not decoded or b"\x00" in decoded: + raise SystemExit(f"invalid {kind} content") +if kind != "WireGuard": + if len(decoded) > 16384 or decoded.count(b"\n") > 1 or (b"\n" in decoded and not decoded.endswith(b"\n")): + raise SystemExit("SSH authorized key must contain exactly one line") + line = decoded.rstrip(b"\n") + if not re.match(rb"^(ssh-ed25519|ssh-rsa|ecdsa-sha2-nistp(?:256|384|521)|sk-ssh-ed25519@openssh.com|sk-ecdsa-sha2-nistp256@openssh.com) [A-Za-z0-9+/]+={0,3}(?: .*)?$", line): + raise SystemExit("invalid SSH authorized key") +' "$kind" +} +validate_encoded_file WireGuard "${values[NEHEMIAH_WIREGUARD_CONFIG_B64]}" +validate_encoded_file SSH "${values[NEHEMIAH_ROOT_SSH_AUTHORIZED_KEY_B64]}" +canonical_wireguard="$({ + printf '%s' "${values[NEHEMIAH_WIREGUARD_CONFIG_B64]}" \ + | python3 "$SCRIPT_DIR/wireguard-config.py" canonicalize-base64 \ + --advertise-address "${values[NEHEMIAH_ADVERTISE_ADDRESS]}" \ + --control-plane-address "${values[NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS]}" \ + --gateway-address "${values[NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS]}" \ + --guest-subnet 10.200.0.0/24 +} 2>&1)" || die "$canonical_wireguard" +values[NEHEMIAH_WIREGUARD_CONFIG_B64]="$(printf '%s' "$canonical_wireguard" | base64 --wrap=0)" + +bootstrap_keys=( + NEHEMIAH_RELEASE_BASE + NEHEMIAH_RELEASE_VERSION + NEHEMIAH_RELEASE_MINISIGN_KEY + NEHEMIAH_HOST_ID + NEHEMIAH_REGION + NEHEMIAH_PROVIDER_ID + LATITUDE_OS_ID + LATITUDE_OS_SLUG + LATITUDE_OS_VERSION + LATITUDE_OS_ARCH + NEHEMIAH_FLEET_BOOTSTRAP_TOKEN + NEHEMIAH_CONTROL_PLANE_URL + NEHEMIAH_TEMPLATE_OBJECT_ORIGIN + NEHEMIAH_ADVERTISE_ADDRESS + NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS + NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS + NEHEMIAH_WIREGUARD_CONFIG_B64 + NEHEMIAH_OTEL_ENABLED + NEHEMIAH_OTEL_ENDPOINT + NEHEMIAH_OTEL_AUTHORIZATION + NEHEMIAH_SERVICE_VERSION + NEHEMIAH_INSTANCE_ID + NEHEMIAH_DEPLOYMENT_ENVIRONMENT + NEHEMIAH_OTEL_EXPORT_INTERVAL_MS + NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS + NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO +) +bootstrap_environment="" +for key in "${bootstrap_keys[@]}"; do + printf -v escaped_value '%q' "${values[$key]:-}" + bootstrap_environment+="${key}=${escaped_value}"$'\n' +done + +bootstrap_b64="$(printf '%s' "$bootstrap_environment" | base64 --wrap=0)" +cloud_init_b64="$(base64 --wrap=0 "$SCRIPT_DIR/cloud-init.sh")" +minisign_verifier_b64="$(base64 --wrap=0 "$SCRIPT_DIR/verify-minisign.py")" +release_validator_b64="$(base64 --wrap=0 "$SCRIPT_DIR/validate-managed-release.py")" +wireguard_parser_b64="$(base64 --wrap=0 "$SCRIPT_DIR/wireguard-config.py")" +ssh_key_b64="${values[NEHEMIAH_ROOT_SSH_AUTHORIZED_KEY_B64]}" + +output_owned=0 +render_complete=0 +cleanup() { + local status=$? + exec 9>&- 2>/dev/null || true + if [[ "$output_owned" == 1 && "$render_complete" != 1 ]]; then + rm -f -- "$OUTPUT_FILE" + fi + return "$status" +} +trap cleanup EXIT +set -o noclobber +if ! exec 9> "$OUTPUT_FILE"; then + die "could not create output without overwriting an existing path" +fi +output_owned=1 +set +o noclobber +chmod 0600 "$OUTPUT_FILE" +cat >&9 <&- +output_sha256="$(sha256sum "$OUTPUT_FILE" | awk '{print $1}')" +render_complete=1 +printf 'rendered private cloud-config: %s (sha256=%s)\n' "$OUTPUT_FILE" "$output_sha256" diff --git a/infra/latitude/teardown.sh b/infra/latitude/teardown.sh index 90d895e..7f86d16 100755 --- a/infra/latitude/teardown.sh +++ b/infra/latitude/teardown.sh @@ -5,11 +5,20 @@ # This is destructive and irreversible: the box (and everything on it) is gone. # Use it when you're done with the prototype so the ~$0.52/hr meter stops. # -# Config from ~/.config/latitude/: -# - api_key: read from LATITUDE_API_KEY (env or ~/.config/latitude/server.env), -# or from the file ~/.config/latitude/api_key +# State is read from the same directory provision.sh writes its recovery +# records to: LATITUDE_STATE_DIR, else $XDG_CONFIG_HOME/latitude, else +# ~/.config/latitude. Within that directory: +# - api_key: read from LATITUDE_API_KEY (env or server.env), or from the +# file named by LATITUDE_API_KEY_FILE (default /api_key) # - server_id: read from LATITUDE_SERVER_ID / SERVER_ID (env or server.env), -# or from the file ~/.config/latitude/server_id +# or from the file /server_id +# - hostname: fallback recovery when no server_id is known — read from +# LATITUDE_HOSTNAME or /last-created-hostname, +# then resolved to a unique server via the Latitude API. +# When the server_id came from the state file and a hostname +# record exists, both must agree (verified via the provider) +# before anything is deleted; conflicting records abort. +# An explicit LATITUDE_SERVER_ID bypasses the check. # # The API key is NEVER printed. # @@ -18,16 +27,22 @@ # set -euo pipefail -CONF_DIR="${HOME}/.config/latitude" +# Must match the state-directory precedence in provision.sh, or a server +# provisioned with an alternate state location cannot be found and the +# hourly-billed host keeps running. +CONF_DIR="${LATITUDE_STATE_DIR:-${XDG_CONFIG_HOME:-${HOME}/.config}/latitude}" ENV_FILE="${CONF_DIR}/server.env" +API_KEY_FILE="${LATITUDE_API_KEY_FILE:-${CONF_DIR}/api_key}" usage() { cat <<'EOF' Usage: infra/latitude/teardown.sh DELETES the Latitude.sh server via API to stop billing. Prompts for -confirmation (type "yes"). Reads api_key and server_id from -~/.config/latitude/ (server.env or api_key/server_id files). +confirmation (type "yes"). Reads api_key and server_id from the same state +directory provision.sh writes to — LATITUDE_STATE_DIR, else +$XDG_CONFIG_HOME/latitude, else ~/.config/latitude (server.env or +api_key/server_id files). EOF } @@ -43,21 +58,134 @@ if [[ -f "${ENV_FILE}" ]]; then fi API_KEY="${LATITUDE_API_KEY:-}" -if [[ -z "${API_KEY}" && -f "${CONF_DIR}/api_key" ]]; then - API_KEY="$(tr -d '[:space:]' < "${CONF_DIR}/api_key")" +if [[ -z "${API_KEY}" && -f "${API_KEY_FILE}" ]]; then + API_KEY="$(tr -d '[:space:]' < "${API_KEY_FILE}")" fi SERVER_ID="${LATITUDE_SERVER_ID:-${SERVER_ID:-}}" +SERVER_ID_SOURCE="env" if [[ -z "${SERVER_ID}" && -f "${CONF_DIR}/server_id" ]]; then SERVER_ID="$(tr -d '[:space:]' < "${CONF_DIR}/server_id")" + SERVER_ID_SOURCE="file" fi if [[ -z "${API_KEY}" ]]; then - echo "error: no API key found. Set LATITUDE_API_KEY in ${ENV_FILE} or create ${CONF_DIR}/api_key." >&2 + echo "error: no API key found. Set LATITUDE_API_KEY in ${ENV_FILE} or create ${API_KEY_FILE}." >&2 exit 1 fi + +# The durable hostname correlation value that provision.sh records (or an +# explicit LATITUDE_HOSTNAME). Used to recover a host when no id is known, and +# to verify a file-sourced id before the destructive delete. +HOSTNAME_LOOKUP="${LATITUDE_HOSTNAME:-}" +if [[ -z "${HOSTNAME_LOOKUP}" && -f "${CONF_DIR}/last-created-hostname" ]]; then + HOSTNAME_LOOKUP="$(tr -d '[:space:]' < "${CONF_DIR}/last-created-hostname")" +fi + +# Resolve HOSTNAME_LOOKUP to a validated server id via the provider. We never +# parse an id from a possibly-malformed local body — we query the provider and +# require a single validated match. Prints the id on success. Returns 0 on a +# unique match, 2 when no server matches, 4 when several match, 3 when the +# lookup itself failed. +lookup_server_by_hostname() { + local response code resolved status=0 + response="$(mktemp "${TMPDIR:-/tmp}/latitude_lookup.XXXXXX")" + code="$( + curl -sS -o "${response}" -w '%{http_code}' \ + -G "https://api.latitude.sh/servers" \ + --data-urlencode "filter[hostname]=${HOSTNAME_LOOKUP}" \ + --data-urlencode "page[size]=200" \ + -H "Authorization: Bearer ${API_KEY}" \ + -H "Accept: application/vnd.api+json" + )" || { rm -f "${response}"; return 3; } + if [[ "${code}" != "200" ]]; then + rm -f "${response}" + echo "error: hostname lookup failed (HTTP ${code})." >&2 + return 3 + fi + resolved="$(python3 - "${response}" "${HOSTNAME_LOOKUP}" <<'PY' +import json, pathlib, re, sys +resp = json.loads(pathlib.Path(sys.argv[1]).read_text()) +wanted = sys.argv[2] +ids = [] +for item in resp.get("data", []) or []: + if not isinstance(item, dict): + continue + attrs = item.get("attributes") or {} + if attrs.get("hostname") != wanted: + continue + sid = item.get("id") + if isinstance(sid, str) and re.fullmatch(r"sv_[A-Za-z0-9_-]{4,128}", sid): + ids.append(sid) +ids = sorted(set(ids)) +if len(ids) == 1: + print(ids[0]) + raise SystemExit(0) +raise SystemExit(2 if not ids else 4) +PY +)" || status=$? + rm -f "${response}" + [[ "${status}" -eq 0 ]] && printf '%s\n' "${resolved}" + return "${status}" +} + +# Recovery path: when no id is known (e.g. provisioning saw a malformed create +# response, or a status poll failed before the id was saved), recover the host +# by the recorded hostname. +if [[ -z "${SERVER_ID}" && -n "${HOSTNAME_LOOKUP}" ]]; then + echo "==> no server_id on file; looking up the server by hostname '${HOSTNAME_LOOKUP}'" >&2 + LOOKUP_STATUS=0 + SERVER_ID="$(lookup_server_by_hostname)" || LOOKUP_STATUS=$? + if [[ "${LOOKUP_STATUS}" -eq 0 ]]; then + echo "==> resolved server ${SERVER_ID} by hostname" >&2 + else + SERVER_ID="" + if [[ "${LOOKUP_STATUS}" -ne 3 ]]; then + echo "error: hostname lookup could not resolve a unique server id for '${HOSTNAME_LOOKUP}'." >&2 + fi + echo " Inspect the Latitude dashboard and set LATITUDE_SERVER_ID explicitly." >&2 + fi +fi + +# Conflicting-records guard: a stale server_id file alongside a newer +# last-created-hostname (an interrupted provisioning run) must not silently +# delete the PREVIOUS server while the newly billed host keeps running. When +# the id came from the state file and a hostname record exists, require the +# provider to agree before deleting; reject a conflict instead of preferring +# the id. An explicit LATITUDE_SERVER_ID bypasses this check. +if [[ -n "${SERVER_ID}" && "${SERVER_ID_SOURCE}" == "file" && -n "${HOSTNAME_LOOKUP}" ]]; then + echo "==> verifying the on-file server_id against hostname '${HOSTNAME_LOOKUP}'" >&2 + VERIFY_STATUS=0 + RESOLVED_ID="$(lookup_server_by_hostname)" || VERIFY_STATUS=$? + case "${VERIFY_STATUS}" in + 0) + if [[ "${RESOLVED_ID}" != "${SERVER_ID}" ]]; then + echo "error: conflicting recovery records — ${CONF_DIR}/server_id says '${SERVER_ID}' but hostname '${HOSTNAME_LOOKUP}' resolves to '${RESOLVED_ID}' (likely an interrupted provisioning run)." >&2 + echo " Inspect the Latitude dashboard, tear down each host explicitly with LATITUDE_SERVER_ID=, and remove the stale ${CONF_DIR}/server_id file." >&2 + exit 1 + fi + echo "==> verified: the hostname resolves to the same server" >&2 + ;; + 2) + # No live server carries the hostname, so the recorded host is already + # gone; deleting by the on-file id is a safe no-op (404) at worst. + echo "==> hostname matches no live server; proceeding with the on-file server_id" >&2 + ;; + 4) + echo "error: multiple servers match hostname '${HOSTNAME_LOOKUP}'; cannot verify the on-file server_id." >&2 + echo " Tear down explicitly with LATITUDE_SERVER_ID=." >&2 + exit 1 + ;; + *) + echo "error: could not verify the on-file server_id (hostname lookup failed); refusing a blind destructive delete." >&2 + echo " Retry, or tear down explicitly with LATITUDE_SERVER_ID=." >&2 + exit 1 + ;; + esac +fi + if [[ -z "${SERVER_ID}" ]]; then - echo "error: no server_id found. Set LATITUDE_SERVER_ID/SERVER_ID in ${ENV_FILE} or create ${CONF_DIR}/server_id." >&2 + echo "error: no server_id found. Set LATITUDE_SERVER_ID/SERVER_ID in ${ENV_FILE}, create ${CONF_DIR}/server_id, or set LATITUDE_HOSTNAME to recover the host by provider lookup." >&2 exit 1 fi diff --git a/infra/latitude/test/managed-provisioning.test.sh b/infra/latitude/test/managed-provisioning.test.sh new file mode 100755 index 0000000..4c885b2 --- /dev/null +++ b/infra/latitude/test/managed-provisioning.test.sh @@ -0,0 +1,1041 @@ +#!/usr/bin/env bash +set -euo pipefail +set +x +umask 077 + +REPOSITORY_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +# The signed-release pipeline (scripts/release/) ships in its own stack slice. +# When this checkout does not contain it, its invariants cannot be asserted +# here, so those checks are skipped loudly instead of failing on missing files. +if [[ -f "$REPOSITORY_ROOT/scripts/release/build.mjs" ]]; then + RELEASE_PIPELINE_PRESENT=1 +else + RELEASE_PIPELINE_PRESENT=0 + echo "note: scripts/release/ is not in this checkout; skipping release-pipeline assertions" >&2 +fi +TASK_TEMP="$(mktemp -d "${TMPDIR:-/tmp}/managed-provisioning-test.XXXXXX")" +FAKE_API_PID="" +cleanup() { + local status=$? + if [[ -n "$FAKE_API_PID" ]]; then + kill "$FAKE_API_PID" 2>/dev/null || true + wait "$FAKE_API_PID" 2>/dev/null || true + fi + rm -rf -- "$TASK_TEMP" + return "$status" +} +trap cleanup EXIT +trap 'printf "managed provisioning test failed at line %s\n" "$LINENO" >&2' ERR + +MINISIGN_KEY="RW$(printf 'A%.0s' {1..54})" +FLEET_TOKEN="nhe_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +OTEL_CREDENTIAL="otel-host-test-credential-abcdef0123456789" +WIREGUARD_PRIVATE="QUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUE=" +WIREGUARD_CONFIG="[Interface] +PrivateKey = ${WIREGUARD_PRIVATE} +Address = 10.42.0.10/32 + +[Peer] +PublicKey = QkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkI= +Endpoint = 192.0.2.10:51820 +AllowedIPs = 10.42.0.1/32, 10.42.0.2/32 +PersistentKeepalive = 25" +WIREGUARD_B64="$(printf '%s\n' "$WIREGUARD_CONFIG" | base64 --wrap=0)" +SSH_PUBLIC_KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE4xY2hhbmdlVGhpcyBUZXN0S2V5 managed-test" +SSH_PUBLIC_KEY_B64="$(printf '%s\n' "$SSH_PUBLIC_KEY" | base64 --wrap=0)" +CONFIG_FILE="$TASK_TEMP/managed-host.env" +cat > "$CONFIG_FILE" < "$TASK_TEMP/wireguard-${label}.stdout" \ + 2> "$TASK_TEMP/wireguard-${label}.stderr"; then + echo "WireGuard parser accepted unsafe input: $label" >&2 + exit 1 + fi +} +expect_wireguard_rejection hook "${WIREGUARD_CONFIG} +PostUp = touch /root/pwned" +expect_wireguard_rejection default-route "${WIREGUARD_CONFIG/10.42.0.1\/32, 10.42.0.2\/32/0.0.0.0\/0}" +expect_wireguard_rejection duplicate-address "${WIREGUARD_CONFIG/Address = 10.42.0.10\/32/Address = 10.42.0.10\/32 +Address = 10.42.0.10\/32}" +expect_wireguard_rejection dns "${WIREGUARD_CONFIG/Address = 10.42.0.10\/32/Address = 10.42.0.10\/32 +DNS = 1.1.1.1}" +expect_wireguard_rejection broad-route "${WIREGUARD_CONFIG/10.42.0.1\/32, 10.42.0.2\/32/10.42.0.0\/24}" +expect_wireguard_rejection wrong-host-route "${WIREGUARD_CONFIG/10.42.0.1\/32, 10.42.0.2\/32/10.42.0.1\/32, 10.42.0.3\/32}" +expect_wireguard_rejection missing-role "${WIREGUARD_CONFIG/10.42.0.1\/32, 10.42.0.2\/32/10.42.0.1\/32}" +expect_wireguard_rejection extra-role "${WIREGUARD_CONFIG/10.42.0.1\/32, 10.42.0.2\/32/10.42.0.1\/32, 10.42.0.2\/32, 10.42.0.3\/32}" +for label_and_addresses in \ + 'guest-role 10.200.0.2 10.42.0.2' \ + 'duplicate-role 10.42.0.1 10.42.0.1' \ + 'family-mismatch 10.42.0.1 fd00::2'; do + read -r label control_plane gateway <<< "$label_and_addresses" + if printf '%s' "$WIREGUARD_B64" \ + | python3 "$wireguard_validator" canonicalize-base64 \ + --advertise-address 10.42.0.10 \ + --control-plane-address "$control_plane" \ + --gateway-address "$gateway" \ + --guest-subnet 10.200.0.0/24 \ + > "$TASK_TEMP/wireguard-${label}.stdout" \ + 2> "$TASK_TEMP/wireguard-${label}.stderr"; then + echo "WireGuard parser accepted unsafe typed routes: $label" >&2 + exit 1 + fi +done +wireguard_file="$TASK_TEMP/wg0.conf" +printf '%s\n' "$canonical_wireguard" > "$wireguard_file" +chmod 0600 "$wireguard_file" +wireguard_digest="$(sha256sum "$wireguard_file" | awk '{print $1}')" +python3 "$wireguard_validator" verify \ + --advertise-address 10.42.0.10 \ + --control-plane-address 10.42.0.1 \ + --gateway-address 10.42.0.2 \ + --guest-subnet 10.200.0.0/24 \ + --expected-sha256 "$wireguard_digest" \ + --path "$wireguard_file" +chmod 0644 "$wireguard_file" +if python3 "$wireguard_validator" verify \ + --advertise-address 10.42.0.10 \ + --control-plane-address 10.42.0.1 \ + --gateway-address 10.42.0.2 \ + --guest-subnet 10.200.0.0/24 \ + --expected-sha256 "$wireguard_digest" \ + --path "$wireguard_file" >/dev/null 2>&1; then + echo "WireGuard verifier accepted a loose configuration mode" >&2 + exit 1 +fi + +assert_no_secret_output() { + local output="$1" + for secret in "$FLEET_TOKEN" "$OTEL_CREDENTIAL" "$WIREGUARD_PRIVATE" "$SSH_PUBLIC_KEY" \ + "latitude-test-api-key-1234567890"; do + if grep -Fq "$secret" "$output"; then + echo "credential appeared in command output" >&2 + exit 1 + fi + done +} + +RENDERED="$TASK_TEMP/rendered.yaml" +"$REPOSITORY_ROOT/infra/latitude/render-user-data.sh" \ + --config "$CONFIG_FILE" --output "$RENDERED" \ + > "$TASK_TEMP/render.stdout" 2> "$TASK_TEMP/render.stderr" +assert_no_secret_output "$TASK_TEMP/render.stdout" +assert_no_secret_output "$TASK_TEMP/render.stderr" +[[ "$(stat -c '%a' "$RENDERED")" == 600 ]] +grep -Fxq '#cloud-config' "$RENDERED" +! grep -Fq "$FLEET_TOKEN" "$RENDERED" +! grep -Fq "$WIREGUARD_PRIVATE" "$RENDERED" + +python3 - "$RENDERED" "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" \ + "$FLEET_TOKEN" "$SSH_PUBLIC_KEY" \ + "https://nehemiah-template-artifacts.objects.example.com" <<'PY' +import base64 +import pathlib +import re +import sys + +rendered, cloud_init, fleet_token, ssh_key, object_origin = sys.argv[1:] +document = pathlib.Path(rendered).read_text() +matches = re.findall( + r" - path: ([^\n]+)\n" + r" owner: root:root\n" + r" permissions: '(0[67]00)'\n" + r" encoding: b64\n" + r" content: ([A-Za-z0-9+/=]+)\n", + document, +) +files = {path: {"permissions": permissions, "content": content} for path, permissions, content in matches} +assert set(files) == { + "/etc/nehemiah/bootstrap.env", + "/usr/local/sbin/nehemiah-cloud-init", + "/usr/local/libexec/nehemiah-validate-release", + "/usr/local/libexec/nehemiah-verify-minisign", + "/usr/local/libexec/nehemiah-wireguard-config", + "/root/.ssh/authorized_keys", +} +assert files["/etc/nehemiah/bootstrap.env"]["permissions"] == "0600" +assert files["/usr/local/sbin/nehemiah-cloud-init"]["permissions"] == "0700" +for helper in ( + "/usr/local/libexec/nehemiah-validate-release", + "/usr/local/libexec/nehemiah-verify-minisign", + "/usr/local/libexec/nehemiah-wireguard-config", +): + assert files[helper]["permissions"] == "0700" +bootstrap = base64.b64decode(files["/etc/nehemiah/bootstrap.env"]["content"], validate=True) +assert f"NEHEMIAH_FLEET_BOOTSTRAP_TOKEN={fleet_token}\n".encode() in bootstrap +assert f"NEHEMIAH_TEMPLATE_OBJECT_ORIGIN={object_origin}\n".encode() in bootstrap +for telemetry_key in ( + "NEHEMIAH_OTEL_ENABLED", + "NEHEMIAH_OTEL_ENDPOINT", + "NEHEMIAH_OTEL_AUTHORIZATION", + "NEHEMIAH_SERVICE_VERSION", + "NEHEMIAH_INSTANCE_ID", + "NEHEMIAH_DEPLOYMENT_ENVIRONMENT", + "NEHEMIAH_OTEL_EXPORT_INTERVAL_MS", + "NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS", + "NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO", +): + assert f"{telemetry_key}=".encode() in bootstrap +for wireguard_role_key in ( + "NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS", + "NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS", +): + assert f"{wireguard_role_key}=".encode() in bootstrap +installed_cloud_init = base64.b64decode( + files["/usr/local/sbin/nehemiah-cloud-init"]["content"], validate=True +) +assert installed_cloud_init == pathlib.Path(cloud_init).read_bytes() +installed_ssh_key = base64.b64decode( + files["/root/.ssh/authorized_keys"]["content"], validate=True +).decode().rstrip("\n") +assert installed_ssh_key == ssh_key +assert document.endswith("runcmd:\n - [ /usr/local/sbin/nehemiah-cloud-init ]\n") +PY + +before_digest="$(sha256sum "$RENDERED" | awk '{print $1}')" +if "$REPOSITORY_ROOT/infra/latitude/render-user-data.sh" \ + --config "$CONFIG_FILE" --output "$RENDERED" \ + > "$TASK_TEMP/overwrite.stdout" 2> "$TASK_TEMP/overwrite.stderr"; then + echo "renderer overwrote an existing output" >&2 + exit 1 +fi +[[ "$(sha256sum "$RENDERED" | awk '{print $1}')" == "$before_digest" ]] + +expect_render_rejection() { + local config="$1" label="$2" + chmod 0600 "$config" + if "$REPOSITORY_ROOT/infra/latitude/render-user-data.sh" \ + --config "$config" --output "$TASK_TEMP/rejected-${label}.yaml" >/dev/null 2>&1; then + echo "renderer accepted invalid telemetry config: $label" >&2 + exit 1 + fi +} + +sed '/^NEHEMIAH_OTEL_ENDPOINT=/d' "$CONFIG_FILE" > "$TASK_TEMP/missing-otel.env" +expect_render_rejection "$TASK_TEMP/missing-otel.env" missing +sed 's#^NEHEMIAH_OTEL_ENDPOINT=.*#NEHEMIAH_OTEL_ENDPOINT=https://192.0.2.7#' \ + "$CONFIG_FILE" > "$TASK_TEMP/ip-otel.env" +expect_render_rejection "$TASK_TEMP/ip-otel.env" ip-endpoint +sed "s#^NEHEMIAH_OTEL_AUTHORIZATION=.*#NEHEMIAH_OTEL_AUTHORIZATION=Basic ${FLEET_TOKEN}#" \ + "$CONFIG_FILE" > "$TASK_TEMP/shared-otel.env" +expect_render_rejection "$TASK_TEMP/shared-otel.env" shared-credential +sed 's/^NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS=.*/NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS=15000/' \ + "$CONFIG_FILE" > "$TASK_TEMP/timeout-otel.env" +expect_render_rejection "$TASK_TEMP/timeout-otel.env" timeout-bound +sed 's/^NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO=.*/NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO=nan/' \ + "$CONFIG_FILE" > "$TASK_TEMP/sample-otel.env" +expect_render_rejection "$TASK_TEMP/sample-otel.env" sample-bound + +chmod 0644 "$CONFIG_FILE" +if "$REPOSITORY_ROOT/infra/latitude/render-user-data.sh" \ + --config "$CONFIG_FILE" --output "$TASK_TEMP/loose.yaml" >/dev/null 2>&1; then + echo "renderer accepted a loose config file" >&2 + exit 1 +fi +chmod 0600 "$CONFIG_FILE" +ln -s "$CONFIG_FILE" "$TASK_TEMP/config-link" +if "$REPOSITORY_ROOT/infra/latitude/render-user-data.sh" \ + --config "$TASK_TEMP/config-link" --output "$TASK_TEMP/link.yaml" >/dev/null 2>&1; then + echo "renderer accepted a symlink config file" >&2 + exit 1 +fi + +INJECTION_CONFIG="$TASK_TEMP/injection.env" +cp "$CONFIG_FILE" "$INJECTION_CONFIG" +printf 'EVIL=$(touch %s)\n' "$TASK_TEMP/injected" >> "$INJECTION_CONFIG" +chmod 0600 "$INJECTION_CONFIG" +if "$REPOSITORY_ROOT/infra/latitude/render-user-data.sh" \ + --config "$INJECTION_CONFIG" --output "$TASK_TEMP/injection.yaml" >/dev/null 2>&1; then + echo "renderer accepted an unknown shell expression" >&2 + exit 1 +fi +[[ ! -e "$TASK_TEMP/injected" ]] + +"$REPOSITORY_ROOT/infra/latitude/provision.sh" \ + --config "$CONFIG_FILE" --output "$TASK_TEMP/render-only.yaml" --render-only \ + > "$TASK_TEMP/render-only.stdout" 2> "$TASK_TEMP/render-only.stderr" +assert_no_secret_output "$TASK_TEMP/render-only.stdout" +assert_no_secret_output "$TASK_TEMP/render-only.stderr" + +cat > "$TASK_TEMP/fake-latitude.py" <<'PY' +import base64 +import http.server +import json +import pathlib +import sys +import urllib.parse + +port_file, request_log = map(pathlib.Path, sys.argv[1:]) + +class Handler(http.server.BaseHTTPRequestHandler): + def log_message(self, _format, *_args): + return + + def body(self): + length = int(self.headers.get("content-length", "0")) + return self.rfile.read(length) + + def respond(self, status, payload=None): + body = b"" if payload is None else json.dumps(payload).encode() + self.send_response(status) + self.send_header("Content-Type", "application/vnd.api+json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def record(self, event): + with request_log.open("a") as output: + output.write(json.dumps(event, separators=(",", ":")) + "\n") + + def do_POST(self): + request = json.loads(self.body()) + authorized = self.headers.get("authorization") == "Bearer latitude-test-api-key-1234567890" + if self.path == "/user_data": + attributes = request["data"]["attributes"] + cloud_config = base64.b64decode(attributes["content"], validate=True) + self.record({ + "event": "user_data_created", + "authorized": authorized, + "project": attributes["project"], + "cloud_config": cloud_config.startswith(b"#cloud-config\n"), + }) + # Deliberately echo content, matching Latitude's documented response. + self.respond(201, {"data": {"id": "ud_test1234", "attributes": attributes}}) + elif self.path == "/servers": + attributes = request["data"]["attributes"] + self.record({ + "event": "server_created", + "authorized": authorized, + "user_data": attributes["user_data"], + "billing": attributes["billing"], + }) + self.respond(201, {"data": {"id": "sv_test1234", "attributes": {"status": "off"}}}) + else: + self.respond(404, {"errors": [{"title": "not found"}]}) + + def do_GET(self): + if self.path.startswith("/plans/operating_systems?"): + self.record({ + "event": "provider_image_read", + "authorized": self.headers.get("authorization") == "Bearer latitude-test-api-key-1234567890", + }) + self.respond(200, {"data": [{ + "id": "os_test1234", + "type": "operating_system", + "attributes": { + "slug": "ubuntu_24_04_x64_lts", + "version": "24.04 LTS", + "provisionable_on": ["c3-small-x86"], + }, + }], "meta": {}}) + elif self.path.startswith("/servers?"): + query = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query) + wanted = query.get("filter[hostname]", [""])[0] + self.record({ + "event": "hostname_checked", + "authorized": self.headers.get("authorization") == "Bearer latitude-test-api-key-1234567890", + "hostname": wanted, + }) + if wanted.startswith("collision-host-"): + self.respond(200, {"data": [ + {"id": "sv_conflict9999", "attributes": {"hostname": wanted}}, + ]}) + else: + self.respond(200, {"data": []}) + elif self.path == "/servers/sv_test1234": + self.record({"event": "server_read"}) + self.respond(200, {"data": {"id": "sv_test1234", "attributes": { + "status": "on", "primary_ipv4": "192.0.2.44" + }}}) + else: + self.respond(404, {"errors": [{"title": "not found"}]}) + + def do_DELETE(self): + if self.path == "/user_data/ud_test1234": + self.record({"event": "user_data_deleted"}) + self.respond(204) + else: + self.respond(404) + +server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) +port_file.write_text(str(server.server_address[1])) +server.serve_forever() +PY +python3 "$TASK_TEMP/fake-latitude.py" \ + "$TASK_TEMP/api.port" "$TASK_TEMP/api.log" & +FAKE_API_PID=$! +for _ in {1..100}; do + [[ -s "$TASK_TEMP/api.port" ]] && break + sleep 0.05 +done +[[ -s "$TASK_TEMP/api.port" ]] +FAKE_API_PORT="$(< "$TASK_TEMP/api.port")" + +LATITUDE_API_KEY=latitude-test-api-key-1234567890 \ +LATITUDE_PROJECT=proj_test1234 \ +LATITUDE_SSH_KEY=ssh_test1234 \ +LATITUDE_STATE_DIR="$TASK_TEMP/state" \ +LATITUDE_API_BASE="http://127.0.0.1:$FAKE_API_PORT" \ +LATITUDE_ALLOW_HTTP_FOR_TESTS=1 \ +LATITUDE_POLL_INTERVAL_SECONDS=0 \ +LATITUDE_POLL_ATTEMPTS=2 \ + "$REPOSITORY_ROOT/infra/latitude/provision.sh" --config "$CONFIG_FILE" \ + > "$TASK_TEMP/provision.stdout" 2> "$TASK_TEMP/provision.stderr" +assert_no_secret_output "$TASK_TEMP/provision.stdout" +assert_no_secret_output "$TASK_TEMP/provision.stderr" +[[ "$(stat -c '%a' "$TASK_TEMP/state/server_id")" == 600 ]] +[[ "$(< "$TASK_TEMP/state/server_id")" == sv_test1234 ]] +[[ "$(stat -c '%a' "$TASK_TEMP/state/provider-image-sv_test1234.json")" == 600 ]] + +python3 - "$TASK_TEMP/api.log" <<'PY' +import json +import pathlib +import re +import sys + +events = [json.loads(line) for line in pathlib.Path(sys.argv[1]).read_text().splitlines()] +assert [event["event"] for event in events] == [ + "provider_image_read", + "hostname_checked", + "user_data_created", + "server_created", + "server_read", + "user_data_deleted", +] +assert events[0]["authorized"] is True +assert events[1]["authorized"] is True +# Every run appends a unique random suffix to the recovery hostname. +assert re.fullmatch(r"nehemiah-metal-01-[0-9a-f]{8}", events[1]["hostname"]) +assert events[2]["authorized"] is True +assert events[2]["project"] == "proj_test1234" +assert events[2]["cloud_config"] is True +assert events[3]["authorized"] is True +assert events[3]["user_data"] == "ud_test1234" +assert events[3]["billing"] == "hourly" +PY + +# Teardown must read the same state directory provisioning wrote: a server +# created under LATITUDE_STATE_DIR or XDG_CONFIG_HOME has to be discovered and +# deleted by a no-argument teardown, or the hourly-billed host keeps running. +# Teardown pins the production API base, so the provider is stubbed at the +# curl boundary; HOME points at an empty directory so a regression to the +# hard-coded ~/.config/latitude path cannot pass. +teardown_bin="$TASK_TEMP/teardown-bin" +mkdir -p "$teardown_bin" +cat > "$teardown_bin/curl" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +: "${TEARDOWN_CURL_LOG:?}" +output="" method=GET url="" hostname="" +args=("$@") +for ((i = 0; i < ${#args[@]}; i++)); do + case "${args[$i]}" in + -o) output="${args[$((i + 1))]}"; i=$((i + 1)) ;; + -w | -H) i=$((i + 1)) ;; + -X) method="${args[$((i + 1))]}"; i=$((i + 1)) ;; + --data-urlencode) + case "${args[$((i + 1))]}" in + "filter[hostname]="*) hostname="${args[$((i + 1))]#"filter[hostname]="}" ;; + esac + i=$((i + 1)) + ;; + -*) ;; + *) url="${args[$i]}" ;; + esac +done +printf '%s %s%s\n' "$method" "$url" "${hostname:+ $hostname}" >> "$TEARDOWN_CURL_LOG" +if [[ "$method" == DELETE ]]; then + : > "$output" + printf '204' +else + printf '{"data":[{"id":"sv_test1234","attributes":{"hostname":"%s"}}]}' \ + "$hostname" > "$output" + printf '200' +fi +EOF +chmod 0755 "$teardown_bin/curl" + +recorded_hostname="$(< "$TASK_TEMP/state/last-created-hostname")" +mkdir -p "$TASK_TEMP/xdg-home/latitude" +cp "$TASK_TEMP/state/server_id" "$TASK_TEMP/state/last-created-hostname" \ + "$TASK_TEMP/xdg-home/latitude/" +for alternate_state in state-dir xdg-config; do + teardown_curl_log="$TASK_TEMP/teardown-curl-$alternate_state.log" + : > "$teardown_curl_log" + teardown_env=() + case "$alternate_state" in + state-dir) teardown_env+=("LATITUDE_STATE_DIR=$TASK_TEMP/state") ;; + xdg-config) teardown_env+=("XDG_CONFIG_HOME=$TASK_TEMP/xdg-home") ;; + esac + env PATH="$teardown_bin:$PATH" \ + TEARDOWN_CURL_LOG="$teardown_curl_log" \ + LATITUDE_API_KEY=latitude-test-api-key-1234567890 \ + HOME="$TASK_TEMP/empty-home" \ + "${teardown_env[@]}" \ + "$REPOSITORY_ROOT/infra/latitude/teardown.sh" <<< "yes" \ + > "$TASK_TEMP/teardown-$alternate_state.stdout" \ + 2> "$TASK_TEMP/teardown-$alternate_state.stderr" + assert_no_secret_output "$TASK_TEMP/teardown-$alternate_state.stdout" + assert_no_secret_output "$TASK_TEMP/teardown-$alternate_state.stderr" + grep -Fq 'Billing stopped' "$TASK_TEMP/teardown-$alternate_state.stdout" + # The file-sourced id must be verified against the recorded hostname before + # the delete, and exactly one delete of the recovered server must be issued. + [[ "$(sed -n '1p' "$teardown_curl_log")" == \ + "GET https://api.latitude.sh/servers ${recorded_hostname}" ]] + [[ "$(sed -n '2p' "$teardown_curl_log")" == \ + "DELETE https://api.latitude.sh/servers/sv_test1234" ]] + [[ "$(wc -l < "$teardown_curl_log")" -eq 2 ]] +done + +# A valid but unapproved opaque image id must fail before user-data creation or +# a billable server POST. The sole additional provider request is inventory GET. +sed 's/^LATITUDE_OS_ID=.*/LATITUDE_OS_ID=os_missing1234/' \ + "$CONFIG_FILE" > "$TASK_TEMP/unapproved-image.env" +chmod 0600 "$TASK_TEMP/unapproved-image.env" +events_before="$(wc -l < "$TASK_TEMP/api.log")" +if LATITUDE_API_KEY=latitude-test-api-key-1234567890 \ + LATITUDE_PROJECT=proj_test1234 \ + LATITUDE_SSH_KEY=ssh_test1234 \ + LATITUDE_STATE_DIR="$TASK_TEMP/unapproved-state" \ + LATITUDE_API_BASE="http://127.0.0.1:$FAKE_API_PORT" \ + LATITUDE_ALLOW_HTTP_FOR_TESTS=1 \ + LATITUDE_POLL_INTERVAL_SECONDS=0 \ + LATITUDE_POLL_ATTEMPTS=2 \ + "$REPOSITORY_ROOT/infra/latitude/provision.sh" \ + --config "$TASK_TEMP/unapproved-image.env" \ + > "$TASK_TEMP/unapproved.stdout" 2> "$TASK_TEMP/unapproved.stderr"; then + echo "provisioner accepted an unapproved provider image id" >&2 + exit 1 +fi +assert_no_secret_output "$TASK_TEMP/unapproved.stdout" +assert_no_secret_output "$TASK_TEMP/unapproved.stderr" +[[ "$(wc -l < "$TASK_TEMP/api.log")" -eq $((events_before + 1)) ]] +[[ "$(tail -n 1 "$TASK_TEMP/api.log")" == *'"event":"provider_image_read"'* ]] + +# A hostname already carried by a live server must fail before user-data +# creation or a billable server POST: the hostname is the durable recovery +# correlation value, so it has to identify exactly one server. +events_before="$(wc -l < "$TASK_TEMP/api.log")" +if LATITUDE_API_KEY=latitude-test-api-key-1234567890 \ + LATITUDE_PROJECT=proj_test1234 \ + LATITUDE_SSH_KEY=ssh_test1234 \ + LATITUDE_HOSTNAME=collision-host \ + LATITUDE_STATE_DIR="$TASK_TEMP/collision-state" \ + LATITUDE_API_BASE="http://127.0.0.1:$FAKE_API_PORT" \ + LATITUDE_ALLOW_HTTP_FOR_TESTS=1 \ + LATITUDE_POLL_INTERVAL_SECONDS=0 \ + LATITUDE_POLL_ATTEMPTS=2 \ + "$REPOSITORY_ROOT/infra/latitude/provision.sh" \ + --config "$CONFIG_FILE" \ + > "$TASK_TEMP/collision.stdout" 2> "$TASK_TEMP/collision.stderr"; then + echo "provisioner created a server behind an already-live hostname" >&2 + exit 1 +fi +assert_no_secret_output "$TASK_TEMP/collision.stdout" +assert_no_secret_output "$TASK_TEMP/collision.stderr" +grep -Fq 'already uses hostname' "$TASK_TEMP/collision.stderr" +[[ "$(wc -l < "$TASK_TEMP/api.log")" -eq $((events_before + 2)) ]] +[[ "$(tail -n 1 "$TASK_TEMP/api.log")" == *'"event":"hostname_checked"'* ]] +[[ ! -e "$TASK_TEMP/collision-state/last-created-hostname" ]] + +for script in \ + bootstrap.sh build-desktop-rootfs.sh build-rootfs.sh cloud-init.sh managed-host-preflight.sh net-setup.sh \ + provision.sh render-user-data.sh teardown.sh verify-isolation.sh; do + bash -n "$REPOSITORY_ROOT/infra/latitude/$script" +done +grep -Fq 'minisign -Vm' "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +grep -Fq '"DAEMON_ARTIFACT": f"nehemiahd_{version}_linux_{arch}.tar.gz"' \ + "$REPOSITORY_ROOT/infra/latitude/validate-managed-release.py" +grep -Fq 'host["inputs"][candidate] != RUNTIME[candidate]' \ + "$REPOSITORY_ROOT/infra/latitude/validate-managed-release.py" +grep -Fq 'exact release artifact set' "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +grep -Fq 'manifest["schemaVersion"] != 5' \ + "$REPOSITORY_ROOT/infra/latitude/validate-managed-release.py" +grep -Fq 'host["contractVersion"] != 4' \ + "$REPOSITORY_ROOT/infra/latitude/validate-managed-release.py" +grep -Fq 'signed-developer-ext4-v1' \ + "$REPOSITORY_ROOT/infra/latitude/validate-managed-release.py" +grep -Fq 'install_guest_image python /opt/boring/rootfs/rootfs.ext4' \ + "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +grep -Fq 'install_guest_image desktop /opt/boring/rootfs/desktop.ext4' \ + "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +grep -Fq 'gzip --test' "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +grep -Fq 'e2fsck -fn' "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +for key in \ + NEHEMIAH_OTEL_ENABLED NEHEMIAH_OTEL_ENDPOINT NEHEMIAH_OTEL_AUTHORIZATION \ + NEHEMIAH_SERVICE_VERSION NEHEMIAH_INSTANCE_ID NEHEMIAH_DEPLOYMENT_ENVIRONMENT \ + NEHEMIAH_OTEL_EXPORT_INTERVAL_MS NEHEMIAH_OTEL_EXPORT_TIMEOUT_MS \ + NEHEMIAH_OTEL_TRACE_SAMPLE_RATIO; do + grep -Fq ": \"\${${key}:?required}\"" "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" + grep -Fq "${key}=\${${key}}" "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +done +for key in \ + NEHEMIAH_RUNTIME_COHORT_ID NEHEMIAH_RUNTIME_CONTRACT_VERSION \ + NEHEMIAH_RUNTIME_ARCH NEHEMIAH_RUNTIME_PYTHON_SHA256 \ + NEHEMIAH_RUNTIME_DESKTOP_SHA256 NEHEMIAH_RUNTIME_KERNEL_SHA256 \ + NEHEMIAH_RUNTIME_FIRECRACKER_SHA256 NEHEMIAH_RUNTIME_JAILER_SHA256; do + grep -Fq "${key}=" "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +done +grep -Fq 'managed-host-packages.py" install' \ + "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +grep -Fq '"--no-download"' \ + "$REPOSITORY_ROOT/infra/latitude/managed-host-packages.py" +for managed_script in cloud-init.sh bootstrap.sh net-setup.sh; do + if grep -Eq '\b(apt|apt-get|aptitude)[[:space:]]+(update|install|upgrade|full-upgrade|dist-upgrade)\b' \ + "$REPOSITORY_ROOT/infra/latitude/$managed_script"; then + echo "managed runtime mutates a network package repository: $managed_script" >&2 + exit 1 + fi +done +grep -Fq 'NEHEMIAH_FIRECRACKER_ARCHIVE' "$REPOSITORY_ROOT/infra/latitude/bootstrap.sh" +grep -Fq 'NEHEMIAH_KERNEL_IMAGE' "$REPOSITORY_ROOT/infra/latitude/bootstrap.sh" +grep -Fq 'fetch_release_artifact "$FIRECRACKER_ARTIFACT"' \ + "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +grep -Fq 'fetch_release_artifact "$KERNEL_ARTIFACT"' \ + "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" +if grep -Eq 'NEHEMIAH_(FIRECRACKER|KERNEL)_URL|download_verified|curl --fail' \ + "$REPOSITORY_ROOT/infra/latitude/bootstrap.sh"; then + echo "managed bootstrap still downloads runtime inputs outside the signed release" >&2 + exit 1 +fi + +# Execute the exact bootstrap account function against a fake NSS database. +# Fresh and idempotent states succeed; name, UID, GID, account-shape, and lock +# collisions fail before a user/group mutation can be hidden. +account_fixture="$TASK_TEMP/account-fixture" +account_fake_bin="$account_fixture/bin" +mkdir -p "$account_fake_bin" +account_runner="$account_fixture/ensure-account.sh" +{ + printf '%s\n' '#!/usr/bin/env bash' 'set -euo pipefail' + printf '%s\n' 'die() { printf "%s\\n" "$*" >&2; exit 1; }' + awk ' + /^ensure_boringjail_account\(\) \{/ { copying = 1 } + copying { print } + copying && /^}$/ { exit } + ' "$REPOSITORY_ROOT/infra/latitude/bootstrap.sh" + printf '%s\n' 'ensure_boringjail_account' +} > "$account_runner" +chmod 0755 "$account_runner" +cat > "$account_fake_bin/account-command" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +command_name="${0##*/}" +: "${FAKE_ACCOUNT_STATE:?}" +case "$command_name" in + getent) + database="$1" + key="$2" + file="$FAKE_ACCOUNT_STATE/$database" + [[ -f "$file" ]] || : > "$file" + awk -F: -v key="$key" ' + $1 == key || $3 == key { print; found = 1 } + END { exit(found ? 0 : 2) } + ' "$file" + ;; + groupadd) + [[ "$1" == --gid && "$#" == 3 ]] + printf '%s:x:%s:\n' "$3" "$2" >> "$FAKE_ACCOUNT_STATE/group" + printf 'groupadd %s %s\n' "$3" "$2" >> "$FAKE_ACCOUNT_STATE/actions" + ;; + useradd) + uid="" gid="" home="" shell="" comment="" name="" + while [[ "$#" -gt 0 ]]; do + case "$1" in + --uid) uid="$2"; shift 2 ;; + --gid) gid="$2"; shift 2 ;; + --no-create-home) shift ;; + --home-dir) home="$2"; shift 2 ;; + --shell) shell="$2"; shift 2 ;; + --comment) comment="$2"; shift 2 ;; + *) name="$1"; shift ;; + esac + done + printf '%s:x:%s:%s:%s:%s:%s\n' "$name" "$uid" "$gid" "$comment" "$home" "$shell" \ + >> "$FAKE_ACCOUNT_STATE/passwd" + printf 'useradd %s %s %s %s %s\n' "$name" "$uid" "$gid" "$home" "$shell" \ + >> "$FAKE_ACCOUNT_STATE/actions" + ;; + passwd) + [[ "$1" == --status && "$#" == 2 ]] + printf '%s %s 01/01/1970 0 99999 7 -1\n' "$2" "$(< "$FAKE_ACCOUNT_STATE/password-status")" + ;; + install) + [[ "${*: -1}" == /srv/jailer ]] + printf 'install-jailer\n' >> "$FAKE_ACCOUNT_STATE/actions" + ;; + stat) + [[ "${*: -1}" == /srv/jailer ]] + printf '%s\n' '755:0:0' + ;; + *) exit 127 ;; +esac +EOF +chmod 0755 "$account_fake_bin/account-command" +for command_name in getent groupadd useradd passwd install stat; do + ln -s account-command "$account_fake_bin/$command_name" +done + +new_account_state() { + local name="$1" + local state="$account_fixture/$name" + mkdir -p "$state" + : > "$state/group" + : > "$state/passwd" + : > "$state/actions" + printf '%s\n' L > "$state/password-status" + printf '%s\n' "$state" +} +run_account_fixture() { + local state="$1" + FAKE_ACCOUNT_STATE="$state" PATH="$account_fake_bin:$PATH" "$account_runner" +} +expect_account_rejection() { + local label="$1" state="$2" + if run_account_fixture "$state" > "$TASK_TEMP/account-${label}.stdout" \ + 2> "$TASK_TEMP/account-${label}.stderr"; then + echo "bootstrap accepted a colliding jailer account: $label" >&2 + exit 1 + fi +} + +fresh_account_state="$(new_account_state fresh)" +run_account_fixture "$fresh_account_state" +grep -Fxq 'boringjail:x:30000:' "$fresh_account_state/group" +grep -Fxq 'boringjail:x:30000:30000::/nonexistent:/usr/sbin/nologin' \ + "$fresh_account_state/passwd" +: > "$fresh_account_state/actions" +run_account_fixture "$fresh_account_state" +! grep -Eq '^(groupadd|useradd)' "$fresh_account_state/actions" + +group_name_collision="$(new_account_state group-name-collision)" +printf '%s\n' 'boringjail:x:29999:' > "$group_name_collision/group" +expect_account_rejection group-name "$group_name_collision" + +group_id_collision="$(new_account_state group-id-collision)" +printf '%s\n' 'unrelated:x:30000:' > "$group_id_collision/group" +expect_account_rejection group-id "$group_id_collision" + +uid_collision="$(new_account_state uid-collision)" +printf '%s\n' 'boringjail:x:30000:' > "$uid_collision/group" +printf '%s\n' 'unrelated:x:30000:30000::/nonexistent:/usr/sbin/nologin' > "$uid_collision/passwd" +expect_account_rejection uid "$uid_collision" + +account_shape_collision="$(new_account_state account-shape-collision)" +printf '%s\n' 'boringjail:x:30000:' > "$account_shape_collision/group" +printf '%s\n' 'boringjail:x:30000:30000::/home/boringjail:/bin/bash' > "$account_shape_collision/passwd" +expect_account_rejection account-shape "$account_shape_collision" + +unlocked_account="$(new_account_state unlocked-account)" +printf '%s\n' 'boringjail:x:30000:' > "$unlocked_account/group" +printf '%s\n' 'boringjail:x:30000:30000::/nonexistent:/usr/sbin/nologin' > "$unlocked_account/passwd" +printf '%s\n' P > "$unlocked_account/password-status" +expect_account_rejection unlocked "$unlocked_account" + +# The dollar sign is intentionally literal; bootstrap must not expand it. +# shellcheck disable=SC2016 +if grep -Fq '${SCRIPT_DIR}' "$REPOSITORY_ROOT/infra/latitude/bootstrap.sh"; then + echo "managed bootstrap success path references an undefined SCRIPT_DIR" >&2 + exit 1 +fi +if [[ "$RELEASE_PIPELINE_PRESENT" == 1 ]]; then + grep -Fq '"ipset"' \ + "$REPOSITORY_ROOT/scripts/release/managed-host-packages-policy.json" +fi +grep -Fxq 'port=0' "$REPOSITORY_ROOT/infra/latitude/net-setup.sh" +grep -Fxq 'KillMode=control-group' "$REPOSITORY_ROOT/infra/latitude/nehemiahd.service" + +# Signed managed units must never degrade into the permissive local contract. +for managed_unit in nehemiahd.service boring-net.service; do + managed_path="$REPOSITORY_ROOT/infra/latitude/$managed_unit" + grep -Fxq 'AssertPathExists=/etc/boring/managed-host' "$managed_path" + grep -Fxq 'AssertPathExists=/etc/boring/nehemiahd.env' "$managed_path" + grep -Fxq 'EnvironmentFile=/etc/boring/nehemiahd.env' "$managed_path" + grep -Fxq 'ExecStartPre=/opt/boring/bin/managed-host-preflight.sh' "$managed_path" + grep -Fq 'ExecStart=/usr/bin/env NEHEMIAH_MODE=1 ' "$managed_path" + if grep -Fq 'EnvironmentFile=-' "$managed_path"; then + echo "managed unit contains an optional environment file: $managed_unit" >&2 + exit 1 + fi +done + +preflight="$REPOSITORY_ROOT/infra/latitude/managed-host-preflight.sh" +grep -Fq "stat -c '%a:%u:%g' -- \"\$path\"" "$preflight" +grep -Fq "require_exact_file \"\$ENV_PATH\" 600:0:0" "$preflight" +grep -Fq "require_exact_file \"\$MARKER_PATH\" 400:0:0" "$preflight" +grep -Fq 'NEHEMIAH_MODE=1 "$NET_SETUP" --fail-closed' "$preflight" + +# Execute the exact preflight logic against private fixture paths. Valid files +# pass; loose mode, wrong content, and (on non-root CI) wrong ownership all fail +# and invoke the network fail-closed action without exposing file contents. +preflight_env="$TASK_TEMP/preflight.env" +preflight_marker="$TASK_TEMP/preflight.marker" +preflight_net="$TASK_TEMP/preflight-net.sh" +preflight_log="$TASK_TEMP/preflight-net.log" +preflight_guard="$TASK_TEMP/preflight-guard.py" +preflight_wireguard="$TASK_TEMP/preflight-wg0.conf" +preflight_provider="$TASK_TEMP/preflight-provider-image" +cat > "$preflight_net" <> "$preflight_log" +EOF +chmod 0755 "$preflight_net" +cat > "$preflight_guard" <<'PY' +#!/usr/bin/env python3 +import sys + +if len(sys.argv) < 2: + raise SystemExit(2) +PY +chmod 0755 "$preflight_guard" +: > "$preflight_env" +printf '%s\n' nehemiah-managed-host-v1 > "$preflight_marker" +printf '%s\n' fixture-wireguard > "$preflight_wireguard" +cat > "$preflight_provider" <<'EOF' +provider=latitude +id=os_test1234 +slug=ubuntu_24_04_x64_lts +version=24.04 LTS +arch=amd64 +EOF +chmod 0600 "$preflight_env" +chmod 0400 "$preflight_marker" +chmod 0600 "$preflight_wireguard" "$preflight_provider" +export NEHEMIAH_RELEASE_VERSION=0.2.0-beta.0 +export NEHEMIAH_RUNTIME_ARCH=amd64 +export NEHEMIAH_ADVERTISE_ADDRESS=10.42.0.10 +export NEHEMIAH_NET_SUBNET=10.200.0 +export NEHEMIAH_WIREGUARD_CONTROL_PLANE_ADDRESS=10.42.0.1 +export NEHEMIAH_WIREGUARD_GATEWAY_ADDRESS=10.42.0.2 +export NEHEMIAH_WIREGUARD_CONFIG_SHA256="$(sha256sum "$preflight_wireguard" | awk '{print $1}')" +export NEHEMIAH_PROVIDER_IMAGE_ID=os_test1234 +export NEHEMIAH_PROVIDER_IMAGE_SLUG=ubuntu_24_04_x64_lts +export NEHEMIAH_PROVIDER_IMAGE_VERSION='24.04 LTS' +export NEHEMIAH_PROVIDER_IMAGE_ARCH=amd64 +fixture_uid="$(id -u)" +fixture_gid="$(id -g)" +fixture_preflight="$TASK_TEMP/managed-host-preflight-fixture.sh" +sed \ + -e "s#^ENV_PATH=.*#ENV_PATH=$preflight_env#" \ + -e "s#^MARKER_PATH=.*#MARKER_PATH=$preflight_marker#" \ + -e "s#^NET_SETUP=.*#NET_SETUP=$preflight_net#" \ + -e "s#^PACKAGE_GUARD=.*#PACKAGE_GUARD=$preflight_guard#" \ + -e "s#^WIREGUARD_GUARD=.*#WIREGUARD_GUARD=$preflight_guard#" \ + -e "s#^WIREGUARD_PATH=.*#WIREGUARD_PATH=$preflight_wireguard#" \ + -e "s#^PROVIDER_IMAGE_PATH=.*#PROVIDER_IMAGE_PATH=$preflight_provider#" \ + -e "s#600:0:0#600:$fixture_uid:$fixture_gid#" \ + -e "s#400:0:0#400:$fixture_uid:$fixture_gid#" \ + "$preflight" > "$fixture_preflight" +chmod 0755 "$fixture_preflight" +"$fixture_preflight" +[[ ! -e "$preflight_log" ]] + +chmod 0640 "$preflight_env" +if "$fixture_preflight" > "$TASK_TEMP/preflight-mode.stdout" 2> "$TASK_TEMP/preflight-mode.stderr"; then + echo "managed preflight accepted a loose environment mode" >&2 + exit 1 +fi +grep -Fxq -- '--fail-closed' "$preflight_log" +! grep -Fq 'nehemiah-managed-host-v1' "$TASK_TEMP/preflight-mode.stderr" +chmod 0600 "$preflight_env" +: > "$preflight_log" + +chmod 0600 "$preflight_marker" +printf '%s\n' invalid-managed-marker > "$preflight_marker" +chmod 0400 "$preflight_marker" +if "$fixture_preflight" >/dev/null 2>&1; then + echo "managed preflight accepted invalid marker content" >&2 + exit 1 +fi +grep -Fxq -- '--fail-closed' "$preflight_log" +chmod 0600 "$preflight_marker" +printf '%s\n' nehemiah-managed-host-v1 > "$preflight_marker" +chmod 0400 "$preflight_marker" + +if [[ "$fixture_uid" != 0 ]]; then + ownership_preflight="$TASK_TEMP/managed-host-preflight-ownership.sh" + sed \ + -e "s#^ENV_PATH=.*#ENV_PATH=$preflight_env#" \ + -e "s#^MARKER_PATH=.*#MARKER_PATH=$preflight_marker#" \ + -e "s#^NET_SETUP=.*#NET_SETUP=$preflight_net#" \ + "$preflight" > "$ownership_preflight" + chmod 0755 "$ownership_preflight" + : > "$preflight_log" + if "$ownership_preflight" >/dev/null 2>&1; then + echo "managed preflight accepted non-root ownership" >&2 + exit 1 + fi + grep -Fxq -- '--fail-closed' "$preflight_log" +fi + +grep -Fxq 'Requires=boring-net.service' \ + "$REPOSITORY_ROOT/infra/latitude/nehemiahd.service" +grep -Fxq 'BindsTo=boring-net.service' \ + "$REPOSITORY_ROOT/infra/latitude/nehemiahd.service" +grep -Fxq 'ExecStop=/usr/bin/env NEHEMIAH_MODE=1 /opt/boring/bin/net-setup.sh --fail-closed' \ + "$REPOSITORY_ROOT/infra/latitude/boring-net.service" +grep -Fxq 'ExecStopPost=/usr/bin/env NEHEMIAH_MODE=1 /opt/boring/bin/net-setup.sh --fail-closed' \ + "$REPOSITORY_ROOT/infra/latitude/boring-net.service" + +# The fail-closed trap is armed before every managed validation or mutation, +# and only a fully verified end-to-end apply marks the network setup complete. +python3 - "$REPOSITORY_ROOT/infra/latitude/net-setup.sh" <<'PY' +import pathlib +import sys + +script = pathlib.Path(sys.argv[1]).read_text() +armed = script.index('trap managed_fail_closed_on_exit EXIT') +bridge_validation = script.index('[[ "$BR" =~') +bridge_mutation = script.index('ip link show "$BR"') +completed = script.rindex('NETWORK_SETUP_COMPLETE=1') +done = script.rindex('log "done."') +assert armed < bridge_validation < bridge_mutation < completed < done +assert script.count('fail_closed_all_guest_ports') >= 5 +assert script.count('freeze_all_guest_ports') >= 2 +input_remove = script.index('while iptables -D INPUT -i "$BR" -j NEHEMIAH_INPUT') +input_insert = script.index('iptables -I INPUT 1 -i "$BR" -j NEHEMIAH_INPUT') +input_flush = script.index('iptables -F NEHEMIAH_INPUT') +forward_remove = script.index('while iptables -D FORWARD -j NEHEMIAH_FWD') +forward_insert = script.index('iptables -I FORWARD 1 -j NEHEMIAH_FWD') +forward_flush = script.index('iptables -F NEHEMIAH_FWD') +ipv6_remove = script.index('while ip6tables -D FORWARD -i "$BR" -j DROP') +ipv6_insert = script.index('ip6tables -I FORWARD 1 -i "$BR" -j DROP') +assert input_remove < input_insert < input_flush +assert forward_remove < forward_insert < forward_flush +assert ipv6_remove < ipv6_insert < completed +assert 'iptables -C FORWARD -j NEHEMIAH_FWD' not in script +assert 'ip6tables -C FORWARD -i "$BR" -j DROP' not in script +PY + +# Verified cloud-init publishes the root-only marker atomically after bootstrap +# succeeds and before either managed unit can be enabled. +python3 - "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" <<'PY' +import pathlib +import sys + +script = pathlib.Path(sys.argv[1]).read_text() +bootstrap = script.index('"$ASSET_ROOT/infra/latitude/bootstrap.sh"') +marker = script.index("printf '%s\\n' nehemiah-managed-host-v1") +marker_mode = script.index('chmod 0400 "$managed_marker_tmp"') +publish = script.index('mv -fT "$managed_marker_tmp" "$MANAGED_HOST_MARKER"') +start = script.index('systemctl enable --now boring-net.service') +assert bootstrap < marker < marker_mode < publish < start +PY + +# Prototype installers must deliberately install the separate permissive units; +# those files are not part of the exact signed managed-host archive allowlist. +for local_unit in nehemiahd-local.service boring-net-local.service; do + local_path="$REPOSITORY_ROOT/infra/latitude/$local_unit" + grep -Fq 'EnvironmentFile=-/etc/boring/' "$local_path" + if grep -Eq 'managed-host|NEHEMIAH_MODE=1' "$local_path"; then + echo "local unit unexpectedly carries the managed contract: $local_unit" >&2 + exit 1 + fi +done +grep -Fq '/root/infra/boring-net-local.service' "$REPOSITORY_ROOT/infra/setup.sh" +grep -Fq '/root/infra/nehemiahd-local.service' "$REPOSITORY_ROOT/infra/setup.sh" +grep -Fq '/root/infra/boring-net-local.service' "$REPOSITORY_ROOT/infra/local/setup-local.sh" +grep -Fq '/root/infra/nehemiahd-local.service' "$REPOSITORY_ROOT/infra/local/setup-local.sh" +grep -Fq '"${SCRIPT_DIR}/nehemiahd-local.service"' "$REPOSITORY_ROOT/infra/latitude/deploy.sh" +if [[ "$RELEASE_PIPELINE_PRESENT" == 1 ]] \ + && grep -Eq '(boring-net|nehemiahd)-local\.service' "$REPOSITORY_ROOT/scripts/release/build.mjs"; then + echo "prototype systemd unit entered the signed managed-host archive" >&2 + exit 1 +fi + +grep -Fq 'mount -t devpts devpts /dev/pts' "$REPOSITORY_ROOT/infra/latitude/build-rootfs.sh" +grep -Fq 'managed rootfs builds are forbidden' "$REPOSITORY_ROOT/infra/latitude/build-rootfs.sh" +if grep -Eq 'MINIROOTFS|build-rootfs\.sh' \ + "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" \ + "$REPOSITORY_ROOT/infra/latitude/bootstrap.sh"; then + echo "managed provisioning still has a mutable/minimal rootfs fallback" >&2 + exit 1 +fi +if grep -Eq '^(server|resolv-file|dns-forward-max|cache-size)=' \ + "$REPOSITORY_ROOT/infra/latitude/net-setup.sh"; then + echo "managed dnsmasq still owns resolver configuration" >&2 + exit 1 +fi +if grep -Eq 'releases/latest|/latest/download' \ + "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" \ + "$REPOSITORY_ROOT/infra/latitude/bootstrap.sh"; then + echo "managed bootstrap contains a mutable latest URL" >&2 + exit 1 +fi +if grep -Fq 'nameserver 1.1.1.1' \ + "$REPOSITORY_ROOT/infra/latitude/build-desktop-rootfs.sh"; then + echo "desktop guest bypasses managed DNS" >&2 + exit 1 +fi + +if command -v shellcheck >/dev/null 2>&1; then + shellcheck_targets=( + "$REPOSITORY_ROOT/infra/latitude/bootstrap.sh" + "$REPOSITORY_ROOT/infra/latitude/build-rootfs.sh" + "$REPOSITORY_ROOT/infra/latitude/cloud-init.sh" + "$REPOSITORY_ROOT/infra/latitude/managed-host-preflight.sh" + "$REPOSITORY_ROOT/infra/latitude/net-setup.sh" + "$REPOSITORY_ROOT/infra/latitude/provision.sh" + "$REPOSITORY_ROOT/infra/latitude/render-user-data.sh" + "$REPOSITORY_ROOT/infra/latitude/teardown.sh" + ) + if [[ "$RELEASE_PIPELINE_PRESENT" == 1 ]]; then + shellcheck_targets+=( + "$REPOSITORY_ROOT/scripts/release/fetch-managed-runtime-assets.sh" + "$REPOSITORY_ROOT/scripts/release/inspect-managed-runtime-assets.sh" + "$REPOSITORY_ROOT/scripts/release/build-guest-images.sh" + "$REPOSITORY_ROOT/scripts/release/guest-images/assemble-rootfs.sh" + "$REPOSITORY_ROOT/scripts/release/guest-images/inspect-guest-image.sh" + "$REPOSITORY_ROOT/scripts/release/guest-images/prepare-vulnerability-scanner.sh" + "$REPOSITORY_ROOT/scripts/release/guest-images/scan-final-rootfs.sh" + ) + fi + shellcheck "${shellcheck_targets[@]}" +fi + +printf 'managed Latitude provisioning tests passed\n' diff --git a/infra/latitude/validate-managed-release.py b/infra/latitude/validate-managed-release.py new file mode 100755 index 0000000..e7c0f3d --- /dev/null +++ b/infra/latitude/validate-managed-release.py @@ -0,0 +1,359 @@ +#!/usr/bin/env python3 +"""Validate signed release schema 5 and emit one architecture's safe inputs.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import pathlib +import re + + +SHA256 = re.compile(r"[0-9a-f]{64}") +SAFE_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9._+-]{0,254}") +ARCHES = ("amd64", "arm64") +RUNTIME = { + "amd64": { + "firecracker": { + "version": "1.15.1", + "artifact": "nehemiah-runtime-firecracker_1.15.1_linux_amd64.tgz", + "format": "tgz", + "maxBytes": 16777216, + "sha256": "d4a32ab2322d887ca1bc4a4e7afa9cc35393e6362dfc2b3becb389d362e4275a", + "firecrackerSha256": "7e8b57e88c459396d4680d83dcdd8c7f72305447cb55b11f4ac98ad70a3f7825", + "jailerSha256": "4830a9b1fc6cece036d8992ff12f1fe9c5247aacad77f42c7aba683c7a08622e", + }, + "kernel": { + "version": "6.1.155", + "artifact": "nehemiah-runtime-kernel_6.1.155_linux_amd64.bin", + "format": "linux-kernel", + "maxBytes": 67108864, + "sha256": "e20e46d0c36c55c0d1014eb20576171b3f3d922260d9f792017aeff53af3d4f2", + }, + }, + "arm64": { + "firecracker": { + "version": "1.15.1", + "artifact": "nehemiah-runtime-firecracker_1.15.1_linux_arm64.tgz", + "format": "tgz", + "maxBytes": 16777216, + "sha256": "00654ac1e702a22744121ea9f10a4f792ebd7c3a744cba587dfac9fcb79b41a5", + "firecrackerSha256": "e9ce7466c3b0d879d7a9158f4bf710dd5e131bbc5e580e5269fec66d5b5a0f0a", + "jailerSha256": "7faa581395fd1994ee005efc0a9c8826b4a9f0616dd942c2486adb8a8eac13f0", + }, + "kernel": { + "version": "6.1.155", + "artifact": "nehemiah-runtime-kernel_6.1.155_linux_arm64.bin", + "format": "linux-kernel", + "maxBytes": 67108864, + "sha256": "e3544b10603acbf3db492cb52e000d22ba202cb4b63b9add027565683e11c591", + }, + }, +} + + +def stop(message: str) -> "NoReturn": + raise SystemExit(f"signed release manifest: {message}") + + +def exact(value: object, keys: set[str], label: str) -> dict: + if not isinstance(value, dict) or set(value) != keys: + stop(f"{label} does not contain the exact key set") + return value + + +def artifact_names(version: str) -> list[str]: + names = [ + "nehemiah.rb", + f"nehemiah-cli-{version}.tgz", + f"nehemiah-host-bootstrap_{version}.tar.gz", + ] + for component in ("nehemiahd", "bc-guest-agent", "bc-gateway"): + for arch in ARCHES: + names.append(f"{component}_{version}_linux_{arch}.tar.gz") + for flavor in ("python", "desktop"): + for arch in ARCHES: + names.append(f"nehemiah-guest-{flavor}_{version}_linux_{arch}.ext4.gz") + for arch in ARCHES: + names.extend( + [ + f"nehemiah-guest-scan_{version}_linux_{arch}.json", + f"nehemiah-host-packages_{version}_ubuntu24.04_linux_{arch}.tar.gz", + RUNTIME[arch]["firecracker"]["artifact"], + RUNTIME[arch]["kernel"]["artifact"], + ] + ) + return sorted(names) + + +def runtime_cohort(arch: str, images: dict) -> dict: + runtime = RUNTIME[arch] + fields = { + "contractVersion": 4, + "arch": arch, + "pythonSha256": images["python"]["uncompressedSha256"], + "desktopSha256": images["desktop"]["uncompressedSha256"], + "kernelSha256": runtime["kernel"]["sha256"], + "firecrackerSha256": runtime["firecracker"]["firecrackerSha256"], + "jailerSha256": runtime["firecracker"]["jailerSha256"], + } + canonical = ( + "contract_version=4\n" + f"arch={arch}\n" + f"python={fields['pythonSha256']}\n" + f"desktop={fields['desktopSha256']}\n" + f"kernel={fields['kernelSha256']}\n" + f"firecracker={fields['firecrackerSha256']}\n" + f"jailer={fields['jailerSha256']}\n" + ).encode() + fields["cohortId"] = hashlib.sha256(canonical).hexdigest() + return fields + + +def validate_guest_policy(policy: object) -> None: + if not isinstance(policy, dict): + stop("guest image policy is missing") + if ( + policy.get("contractVersion") != 1 + or policy.get("rootfsProfile") != "signed-developer-ext4-v1" + ): + stop("guest image policy contract is unsupported") + architectures = policy.get("architectures") + if not isinstance(architectures, dict) or set(architectures) != set(ARCHES): + stop("guest image architecture policy is invalid") + for arch in ARCHES: + base = architectures[arch].get("ociBase", {}) + if ( + not re.fullmatch(r"24\.[0-9]+\.[0-9]+", str(base.get("nodeVersion", ""))) + or not re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+", str(base.get("bundledNpmVersion", ""))) + or not re.fullmatch( + r"docker\.io/library/node@sha256:[0-9a-f]{64}", + str(base.get("reference", "")), + ) + ): + stop(f"{arch} guest OCI policy is mutable or unsupported") + snapshot = policy.get("alpineRepositorySnapshot", {}) + if ( + snapshot.get("release") != "v3.23" + or not isinstance(snapshot.get("maxIndexAgeHours"), int) + or not 0 < snapshot["maxIndexAgeHours"] <= 168 + or not re.fullmatch( + r"2026-08-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z", + str(snapshot.get("capturedAt", "")), + ) + ): + stop("guest APK snapshot policy is invalid") + for arch, apk_arch in (("amd64", "x86_64"), ("arm64", "aarch64")): + repositories = snapshot.get("architectures", {}).get(arch, {}) + if repositories.get("apkArchitecture") != apk_arch: + stop(f"{arch} APK architecture is invalid") + for component in ("main", "community"): + source = repositories.get(component, {}) + if ( + source.get("url") + != f"https://dl-cdn.alpinelinux.org/alpine/v3.23/{component}/{apk_arch}/APKINDEX.tar.gz" + or not SHA256.fullmatch(str(source.get("sha256", ""))) + ): + stop(f"{arch} {component} APK index is not digest-pinned") + npm = policy.get("npmRuntime", {}) + if ( + npm.get("version") != "11.19.0" + or npm.get("tarball", {}).get("url") + != "https://registry.npmjs.org/npm/-/npm-11.19.0.tgz" + or not SHA256.fullmatch(str(npm.get("tarball", {}).get("sha256", ""))) + ): + stop("npm runtime policy is not exact") + python = policy.get("pythonRuntime", {}) + for name, version in (("pip", "26.2.1"), ("setuptools", "84.0.0")): + wheel = python.get(name, {}) + if wheel.get("version") != version or not SHA256.fullmatch(str(wheel.get("sha256", ""))): + stop(f"Python {name} runtime is not exact") + + +def validate(path: pathlib.Path, version: str, arch: str) -> dict[str, str]: + try: + manifest = json.loads(path.read_text()) + except (OSError, json.JSONDecodeError): + stop("manifest is not valid JSON") + exact( + manifest, + { + "artifacts", + "commit", + "managedCloudInitCompatible", + "managedHost", + "repository", + "schemaVersion", + "sourceDateEpoch", + "version", + }, + "manifest", + ) + if ( + manifest["schemaVersion"] != 5 + or manifest["managedCloudInitCompatible"] is not True + or manifest["version"] != version + or not re.fullmatch(r"[0-9a-f]{40,64}", str(manifest["commit"])) + or not isinstance(manifest["sourceDateEpoch"], int) + or manifest["sourceDateEpoch"] <= 0 + ): + stop("manifest release identity is invalid") + artifacts = manifest["artifacts"] + if not isinstance(artifacts, list): + stop("artifact matrix is invalid") + names = [] + for artifact in artifacts: + if not isinstance(artifact, dict) or not SAFE_NAME.fullmatch(str(artifact.get("name", ""))): + stop("artifact matrix contains an unsafe entry") + names.append(artifact["name"]) + if names != artifact_names(version): + stop("manifest does not contain the exact release artifact set") + host = exact( + manifest["managedHost"], + { + "bootstrapArtifact", + "contractVersion", + "guestImagePolicy", + "guestImages", + "inputs", + "packageRepositories", + "rootfsProfile", + "runtimeCohorts", + }, + "managed-host contract", + ) + if ( + host["contractVersion"] != 4 + or host["bootstrapArtifact"] != f"nehemiah-host-bootstrap_{version}.tar.gz" + or host["rootfsProfile"] != "signed-developer-ext4-v1" + ): + stop("managed-host contract is unsupported") + validate_guest_policy(host["guestImagePolicy"]) + if not isinstance(host["inputs"], dict) or set(host["inputs"]) != set(ARCHES): + stop("managed runtime inputs are invalid") + for candidate in ARCHES: + if host["inputs"][candidate] != RUNTIME[candidate]: + stop(f"{candidate} runtime inputs differ from reviewed pins") + if not isinstance(host["guestImages"], dict) or set(host["guestImages"]) != set(ARCHES): + stop("managed guest image matrix is invalid") + for candidate in ARCHES: + images = host["guestImages"][candidate] + if not isinstance(images, dict) or set(images) != {"desktop", "python", "scanEvidence"}: + stop(f"{candidate} guest image set is not exact") + for flavor, uncompressed, maximum in ( + ("python", 2147483648, 805306368), + ("desktop", 6442450944, 2147483648), + ): + image = exact( + images[flavor], + { + "artifact", + "format", + "maxCompressedBytes", + "uncompressedBytes", + "uncompressedSha256", + }, + f"{candidate} {flavor} image", + ) + if ( + image["artifact"] + != f"nehemiah-guest-{flavor}_{version}_linux_{candidate}.ext4.gz" + or image["format"] != "ext4.gz" + or image["uncompressedBytes"] != uncompressed + or image["maxCompressedBytes"] != maximum + or not SHA256.fullmatch(str(image["uncompressedSha256"])) + ): + stop(f"{candidate} {flavor} image contract is invalid") + scan = images["scanEvidence"] + if scan.get("artifact") != f"nehemiah-guest-scan_{version}_linux_{candidate}.json": + stop(f"{candidate} guest scan evidence is invalid") + expected_cohort = runtime_cohort(candidate, images) + if host.get("runtimeCohorts", {}).get(candidate) != expected_cohort: + stop(f"{candidate} runtime cohort is invalid") + packages = exact( + host.get("packageRepositories", {}).get(candidate), + { + "artifact", + "contractVersion", + "format", + "manifestSha256", + "maxBytes", + "operatingSystem", + "packageCount", + "snapshot", + }, + f"{candidate} package repository", + ) + if ( + packages["contractVersion"] != 1 + or packages["artifact"] + != f"nehemiah-host-packages_{version}_ubuntu24.04_linux_{candidate}.tar.gz" + or packages["format"] != "tar.gz" + or packages["maxBytes"] != 268435456 + or not SHA256.fullmatch(str(packages["manifestSha256"])) + or not isinstance(packages["packageCount"], int) + or not 0 < packages["packageCount"] <= 256 + or packages["operatingSystem"] + != {"codename": "noble", "id": "ubuntu", "version": "24.04"} + or packages["snapshot"] + != { + "baseUrl": "https://snapshot.ubuntu.com/ubuntu/20260809T000000Z", + "capturedAt": "2026-08-09T00:00:00Z", + } + ): + stop(f"{candidate} package repository contract is invalid") + + images = host["guestImages"][arch] + packages = host["packageRepositories"][arch] + runtime = RUNTIME[arch] + cohort = host["runtimeCohorts"][arch] + return { + "DAEMON_ARTIFACT": f"nehemiahd_{version}_linux_{arch}.tar.gz", + "AGENT_ARTIFACT": f"bc-guest-agent_{version}_linux_{arch}.tar.gz", + "HOST_ARTIFACT": host["bootstrapArtifact"], + "PACKAGE_ARTIFACT": packages["artifact"], + "PACKAGE_MAX_BYTES": str(packages["maxBytes"]), + "PACKAGE_MANIFEST_SHA256": packages["manifestSha256"], + "FIRECRACKER_ARTIFACT": runtime["firecracker"]["artifact"], + "FIRECRACKER_MAX_BYTES": str(runtime["firecracker"]["maxBytes"]), + "FIRECRACKER_ARCHIVE_SHA256": runtime["firecracker"]["sha256"], + "FIRECRACKER_INSTALLED_SHA256": runtime["firecracker"]["firecrackerSha256"], + "JAILER_INSTALLED_SHA256": runtime["firecracker"]["jailerSha256"], + "KERNEL_ARTIFACT": runtime["kernel"]["artifact"], + "KERNEL_MAX_BYTES": str(runtime["kernel"]["maxBytes"]), + "KERNEL_SHA256": runtime["kernel"]["sha256"], + "PYTHON_ARTIFACT": images["python"]["artifact"], + "PYTHON_MAX_BYTES": str(images["python"]["maxCompressedBytes"]), + "PYTHON_UNCOMPRESSED_BYTES": str(images["python"]["uncompressedBytes"]), + "PYTHON_SHA256": images["python"]["uncompressedSha256"], + "DESKTOP_ARTIFACT": images["desktop"]["artifact"], + "DESKTOP_MAX_BYTES": str(images["desktop"]["maxCompressedBytes"]), + "DESKTOP_UNCOMPRESSED_BYTES": str(images["desktop"]["uncompressedBytes"]), + "DESKTOP_SHA256": images["desktop"]["uncompressedSha256"], + "RUNTIME_COHORT_ID": cohort["cohortId"], + "RUNTIME_CONTRACT_VERSION": str(cohort["contractVersion"]), + } + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--manifest", required=True, type=pathlib.Path) + parser.add_argument("--version", required=True) + parser.add_argument("--arch", required=True, choices=ARCHES) + parser.add_argument("--output", required=True, type=pathlib.Path) + args = parser.parse_args() + if not re.fullmatch( + r"(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?", + args.version, + ): + stop("release version is invalid") + if args.output.exists() or args.output.is_symlink(): + stop("output already exists") + values = validate(args.manifest, args.version, args.arch) + args.output.write_text("".join(f"{key}={value}\n" for key, value in sorted(values.items()))) + args.output.chmod(0o600) + + +if __name__ == "__main__": + main() diff --git a/infra/latitude/verify-isolation.sh b/infra/latitude/verify-isolation.sh new file mode 100755 index 0000000..37cc5d9 --- /dev/null +++ b/infra/latitude/verify-isolation.sh @@ -0,0 +1,297 @@ +#!/usr/bin/env bash +# Verify the non-overridable Nehemiah host-isolation floor. +# +# Default mode is read-only and safe on a live host. `--active` creates two +# short-lived network namespaces on boring0 and performs connection probes; use +# that mode only on a disposable Latitude staging host with no addresses in the +# reserved .252-.253 range. +set -euo pipefail + +BR="${NEHEMIAH_NET_BRIDGE:-boring0}" +SUBNET="${NEHEMIAH_NET_SUBNET:-10.200.0}" +CIDR="${SUBNET}.0/24" +STATE_PATH="${NEHEMIAH_STATE_PATH:-/var/lib/nehemiahd/state.json}" +ACTIVE=0 +[[ "${1:-}" == "--active" ]] && ACTIVE=1 + +pass_count=0 +fail_count=0 +pass() { printf '[PASS] %s\n' "$*"; pass_count=$((pass_count + 1)); } +fail() { printf '[FAIL] %s\n' "$*" >&2; fail_count=$((fail_count + 1)); } +require_cmd() { command -v "$1" >/dev/null 2>&1 || { fail "missing command: $1"; return 1; }; } + +check_cmd() { + local description="$1" + shift + if "$@" >/dev/null 2>&1; then pass "$description"; else fail "$description"; fi +} + +fdb_selects_port() { + local mac="$1" tap="$2" output + output="$(bridge fdb get "$mac" dev "$tap" master 2>/dev/null)" || return 1 + awk -v wanted_mac="${mac,,}" -v wanted_tap="$tap" -v wanted_bridge="$BR" ' + tolower($1) == wanted_mac { + for (i = 1; i <= NF; i++) { + if ($i == "dev" && $(i + 1) == wanted_tap) dev_ok = 1 + if ($i == "master" && $(i + 1) == wanted_bridge) master_ok = 1 + if ($i == "static") static_ok = 1 + } + } + END { exit !(dev_ok && master_ok && static_ok) } + ' <<<"$output" +} + +[[ "$(id -u)" -eq 0 ]] || { echo "run as root" >&2; exit 2; } +prerequisites_ok=1 +for command in awk bridge ip iptables iptables-save ip6tables jq sha1sum sysctl systemctl; do + require_cmd "$command" || prerequisites_ok=0 +done +[[ "$prerequisites_ok" -eq 1 ]] || { echo "required isolation verification tools are unavailable" >&2; exit 2; } + +check_cmd "bridge ${BR} exists" ip link show "$BR" +if ip -4 address show dev "$BR" | grep -Fq "inet ${SUBNET}.1/24"; then pass "bridge has gateway ${SUBNET}.1/24"; else fail "bridge gateway missing"; fi +if [[ "$(sysctl -n net.ipv4.ip_forward 2>/dev/null)" == "1" ]]; then pass "IPv4 forwarding enabled"; else fail "IPv4 forwarding disabled"; fi +if [[ "$(sysctl -n net.bridge.bridge-nf-call-iptables 2>/dev/null)" == "1" ]]; then pass "bridged peer traffic enters iptables"; else fail "bridge netfilter disabled"; fi + +check_cmd "guest INPUT policy is hooked" iptables -C INPUT -i "$BR" -j NEHEMIAH_INPUT +check_cmd "guest INPUT defaults to drop" iptables -C NEHEMIAH_INPUT -j DROP +if iptables-save | grep -q -- '--hashlimit-name neh-dns-u'; then pass "UDP DNS rate limit present"; else fail "UDP DNS rate limit missing"; fi +if iptables-save | grep -q -- '--hashlimit-name neh-dns-t'; then pass "TCP DNS rate limit present"; else fail "TCP DNS rate limit missing"; fi +check_cmd "egress policy is hooked" iptables -C FORWARD -j NEHEMIAH_FWD +declare -A tap_macs=() +declare -A tap_ips=() +if [[ -r "$STATE_PATH" ]]; then + while IFS=$'\t' read -r machine_id persisted_tap persisted_ip; do + [[ -n "$machine_id" && -n "$persisted_tap" ]] || continue + digest="$(printf '%s' "$machine_id" | sha1sum | awk '{print $1}')" + expected_tap="bt${digest:0:8}" + [[ "$persisted_tap" == "$expected_tap" ]] || { + fail "state tap ${persisted_tap} does not match machine ${machine_id}" + continue + } + tap_macs["$persisted_tap"]="06:00:${digest:0:2}:${digest:2:2}:${digest:4:2}:${digest:6:2}" + tap_ips["$persisted_tap"]="$persisted_ip" + done < <(jq -r '.machines[]? | select(.runtime.tap? != null and .runtime.tap != "") | [.id, .runtime.tap, (.runtime.ip // "")] | @tsv' "$STATE_PATH") +fi +for tap_path in /sys/class/net/bt*; do + [[ -e "$tap_path" ]] || continue + tap_name="${tap_path##*/}" + check_cmd "managed tap ${tap_name} defaults to no egress" \ + iptables -C NEHEMIAH_FWD -i "$tap_name" -j DROP + port_state="$(bridge -j -details link show dev "$tap_name" 2>/dev/null || true)" + if jq -e --arg bridge "$BR" 'length == 1 and .[0].master == $bridge and .[0].isolated == true and .[0].locked == true and .[0].learning == false and .[0].flood == false and .[0].hairpin == false and .[0].guard == true' <<<"$port_state" >/dev/null 2>&1; then + pass "managed tap ${tap_name} has fail-closed isolated/locked bridge flags" + else + fail "managed tap ${tap_name} lacks fail-closed isolated/locked bridge flags" + fi + expected_mac="${tap_macs[$tap_name]:-}" + if [[ -n "$expected_mac" ]]; then + check_cmd "managed tap ${tap_name} has its expected static FDB identity" \ + fdb_selects_port "$expected_mac" "$tap_name" + expected_ip="${tap_ips[$tap_name]:-}" + if [[ -n "$expected_ip" ]]; then + neighbor="$(ip neigh get "$expected_ip" dev "$BR" 2>/dev/null || true)" + if grep -Eiq "lladdr ${expected_mac}([[:space:]]|$).*PERMANENT" <<<"$neighbor"; then + pass "managed tap ${tap_name} host neighbor is permanently pinned" + else + fail "managed tap ${tap_name} host neighbor is not permanently pinned" + fi + fi + else + fail "managed tap ${tap_name} has no expected identity in daemon state" + fi +done + +for destination in \ + 0.0.0.0/8 \ + 10.0.0.0/8 \ + 100.64.0.0/10 \ + 127.0.0.0/8 \ + 169.254.0.0/16 \ + 172.16.0.0/12 \ + 192.168.0.0/16 \ + 198.18.0.0/15 \ + 224.0.0.0/4 \ + 240.0.0.0/4; do + check_cmd "blocked destination ${destination}" iptables -C NEHEMIAH_FWD -s "$CIDR" -d "$destination" -j DROP +done +check_cmd "SMTP egress blocked" iptables -C NEHEMIAH_FWD -s "$CIDR" -p tcp --dport 25 -j DROP +check_cmd "IPv6 guest forwarding blocked" ip6tables -C FORWARD -i "$BR" -j DROP + +DNSMASQ_CONF=/etc/dnsmasq.d/boring.conf +if grep -Fxq 'port=0' "$DNSMASQ_CONF" 2>/dev/null; then + pass "dnsmasq DNS is disabled" +else + fail "dnsmasq must be DHCP-only with port=0" +fi +if grep -Eq '^(server|resolv-file|dns-forward-max|cache-size)=' "$DNSMASQ_CONF" 2>/dev/null; then + fail "dnsmasq still contains DNS resolver configuration" +else + pass "dnsmasq has no resolver upstream" +fi + +if [[ -x /opt/boring/bin/jailer ]]; then pass "jailer installed"; else fail "jailer missing"; fi +if [[ -e "$STATE_PATH" ]]; then + if [[ "$(stat -c '%a' "$STATE_PATH" 2>/dev/null)" == "600" ]]; then pass "state file is owner-only"; else fail "state file permissions are not 600"; fi +else + pass "state file not created yet" +fi +if [[ "$(systemctl show nehemiahd.service -p Delegate --value 2>/dev/null)" == "yes" ]]; then pass "cgroup delegation enabled"; else fail "cgroup delegation missing"; fi +if [[ "$(systemctl show nehemiahd.service -p KillMode --value 2>/dev/null)" == "control-group" ]]; then pass "daemon cgroup is fully cleaned while sibling VMM scopes survive"; else fail "KillMode is not control-group"; fi +if command -v systemd-run >/dev/null 2>&1 && command -v systemctl >/dev/null 2>&1; then pass "systemd transient-scope tools installed"; else fail "systemd transient-scope tools missing"; fi +if pgrep -a firecracker 2>/dev/null | grep -q -- '--no-seccomp'; then + fail "a Firecracker process disabled seccomp" +else + pass "no Firecracker process disables seccomp" +fi + +if [[ "$ACTIVE" -eq 1 ]]; then + active_prerequisites_ok=1 + for command in curl python3; do + require_cmd "$command" || active_prerequisites_ok=0 + done + [[ "$active_prerequisites_ok" -eq 1 ]] || { echo "active isolation verification tools are unavailable" >&2; exit 2; } + suffix="$$" + ns_a="nehemiah-a-${suffix}" + ns_b="nehemiah-b-${suffix}" + va="nva${suffix: -6}" + vb="nvb${suffix: -6}" + pa="npa${suffix: -6}" + pb="npb${suffix: -6}" + mac_a="06:00:00:00:fc:01" + mac_b="06:00:00:00:fd:01" + peer_pid="" + cleanup() { + if [[ -n "$peer_pid" ]]; then kill "$peer_pid" >/dev/null 2>&1 || true; fi + ip neigh del "${SUBNET}.253" lladdr "$mac_b" dev "$BR" >/dev/null 2>&1 || true + ip link del "$va" >/dev/null 2>&1 || true + ip link del "$vb" >/dev/null 2>&1 || true + while iptables -D NEHEMIAH_FWD -i "$va" -j DROP >/dev/null 2>&1; do :; done + while iptables -D NEHEMIAH_FWD -i "$va" ! -s "${SUBNET}.252/32" -j DROP >/dev/null 2>&1; do :; done + ip netns del "$ns_a" >/dev/null 2>&1 || true + ip netns del "$ns_b" >/dev/null 2>&1 || true + } + trap cleanup EXIT + + ip netns add "$ns_a" + ip netns add "$ns_b" + ip link add "$va" type veth peer name "$pa" + ip link add "$vb" type veth peer name "$pb" + ip link set "$pa" netns "$ns_a" + ip link set "$pb" netns "$ns_b" + ip link set "$va" master "$BR" + ip link set "$vb" master "$BR" + ip -n "$ns_a" link set "$pa" address "$mac_a" + ip -n "$ns_b" link set "$pb" address "$mac_b" + bridge link set dev "$va" isolated on locked on learning off flood off guard on hairpin off + bridge link set dev "$vb" isolated on locked on learning off flood off guard on hairpin off + bridge fdb replace "$mac_a" dev "$va" master static + bridge fdb replace "$mac_b" dev "$vb" master static + ip neigh replace "${SUBNET}.253" lladdr "$mac_b" nud permanent dev "$BR" + ip link set "$va" up + ip link set "$vb" up + ip -n "$ns_a" link set lo up + ip -n "$ns_b" link set lo up + ip -n "$ns_a" addr add "${SUBNET}.252/24" dev "$pa" + ip -n "$ns_b" addr add "${SUBNET}.253/24" dev "$pb" + ip -n "$ns_a" link set "$pa" up + ip -n "$ns_b" link set "$pb" up + ip -n "$ns_a" route add default via "${SUBNET}.1" + ip -n "$ns_b" route add default via "${SUBNET}.1" + + ip netns exec "$ns_b" python3 -m http.server 18080 --bind "${SUBNET}.253" >/dev/null 2>&1 & + peer_pid=$! + for _ in 1 2 3 4 5; do + ip netns exec "$ns_b" curl -fsS --max-time 1 "http://${SUBNET}.253:18080/" >/dev/null 2>&1 && break + sleep 0.1 + done + if ! ip netns exec "$ns_b" curl -fsS --max-time 1 "http://${SUBNET}.253:18080/" >/dev/null 2>&1; then + fail "peer test listener failed to start" + fi + + if curl -fsS --connect-timeout 1 --max-time 2 "http://${SUBNET}.253:18080/" >/dev/null 2>&1; then + pass "host-initiated preview path reaches an isolated guest" + else + fail "host-initiated preview path cannot reach an isolated guest" + fi + + blocked_probe() { + local description="$1" target="$2" + if ip netns exec "$ns_a" curl -fsS --connect-timeout 1 --max-time 2 "$target" >/dev/null 2>&1; then + fail "$description" + else + pass "$description" + fi + } + blocked_probe "guest cannot contact peer guest" "http://${SUBNET}.253:18080/" + blocked_probe "guest cannot contact host daemon" "http://${SUBNET}.1:8080/healthz" + blocked_probe "guest cannot contact metadata" "http://169.254.169.254/" + blocked_probe "guest cannot contact RFC1918 services" "http://192.168.1.1/" + + # Send two forged gratuitous ARP replies from guest A: one uses A's admitted + # MAC while claiming B's IP, and one spoofs B's MAC outright. The permanent + # neighbor must not move, the locked FDB must still select B's port, and a + # host preview must continue reaching B. + for forged_mac in "$mac_a" "$mac_b"; do + ip netns exec "$ns_a" python3 - "$pa" "$forged_mac" "${SUBNET}.253" <<'PY' +import socket +import struct +import sys + +interface, source_text, claimed_ip = sys.argv[1:] +source = bytes.fromhex(source_text.replace(":", "")) +target_ip = socket.inet_aton(claimed_ip) +ethernet = b"\xff" * 6 + source + struct.pack("!H", 0x0806) +arp = struct.pack("!HHBBH6s4s6s4s", 1, 0x0800, 6, 4, 2, source, target_ip, b"\x00" * 6, target_ip) +sock = socket.socket(socket.AF_PACKET, socket.SOCK_RAW) +sock.bind((interface, 0)) +for _ in range(3): + sock.send(ethernet + arp) +sock.close() +PY + done + sleep 0.2 + neighbor="$(ip neigh get "${SUBNET}.253" dev "$BR" 2>/dev/null || true)" + if grep -Eiq "lladdr ${mac_b}([[:space:]]|$).*PERMANENT" <<<"$neighbor"; then + pass "forged ARP cannot move the host's permanent guest neighbor" + else + fail "forged ARP changed the host guest neighbor: ${neighbor}" + fi + check_cmd "foreign source MAC cannot move the static FDB selection" \ + fdb_selects_port "$mac_b" "$vb" + if fdb_selects_port "$mac_b" "$va"; then + fail "victim MAC was also selected on the attacker port" + else + pass "victim MAC is absent from the attacker port" + fi + if curl -fsS --connect-timeout 1 --max-time 2 "http://${SUBNET}.253:18080/" >/dev/null 2>&1; then + pass "host preview still selects the victim after forged ARP" + else + fail "forged ARP diverted the host preview path" + fi + if ip netns exec "$ns_a" curl -kfsS --connect-timeout 3 --max-time 8 https://1.1.1.1/cdn-cgi/trace >/dev/null 2>&1; then + pass "host firewall can carry explicitly allowed public egress" + else + fail "host firewall public egress path failed" + fi + # Mirror the daemon's first per-tap policy rule, then prove a root guest + # cannot rotate source addresses to multiply machine-scoped abuse buckets. + iptables -I NEHEMIAH_FWD 1 -i "$va" ! -s "${SUBNET}.252/32" -j DROP + ip -n "$ns_a" addr add "${SUBNET}.251/24" dev "$pa" + if ip netns exec "$ns_a" curl -kfsS --interface "${SUBNET}.251" --connect-timeout 2 --max-time 4 https://1.1.1.1/cdn-cgi/trace >/dev/null 2>&1; then + fail "spoofed guest source IPv4 bypassed the per-tap identity binding" + else + pass "per-tap identity binding drops spoofed guest source IPv4" + fi + ip -n "$ns_a" addr del "${SUBNET}.251/24" dev "$pa" + while iptables -D NEHEMIAH_FWD -i "$va" ! -s "${SUBNET}.252/32" -j DROP >/dev/null 2>&1; do :; done + iptables -I NEHEMIAH_FWD 1 -i "$va" -j DROP + if ip netns exec "$ns_a" curl -kfsS --connect-timeout 2 --max-time 4 https://1.1.1.1/cdn-cgi/trace >/dev/null 2>&1; then + fail "managed per-tap default deny allowed public egress" + else + pass "managed per-tap default deny blocks public egress" + fi +fi + +printf '\nIsolation verification: %d passed, %d failed\n' "$pass_count" "$fail_count" +[[ "$fail_count" -eq 0 ]] diff --git a/infra/latitude/verify-minisign.py b/infra/latitude/verify-minisign.py new file mode 100755 index 0000000..753a6f3 --- /dev/null +++ b/infra/latitude/verify-minisign.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +"""Minimal strict verifier for the prehashed Minisign release envelope. + +Managed cloud-init runs before the retained offline package repository has +installed ``minisign``. Ubuntu's base-image bootstrap contract already contains +Python and OpenSSL; this verifier supports only Minisign's current prehashed +Ed25519 form and verifies both the payload and trusted-comment signatures. The +newly installed pinned Minisign package verifies the envelope again afterward. +""" + +from __future__ import annotations + +import argparse +import base64 +import binascii +import hashlib +import pathlib +import re +import subprocess +import tempfile + + +def stop(message: str) -> "NoReturn": + raise SystemExit(f"release signature verification failed: {message}") + + +def decode_exact(value: str, size: int, label: str) -> bytes: + if not re.fullmatch(r"[A-Za-z0-9+/]+={0,2}", value): + stop(f"invalid {label} encoding") + try: + decoded = base64.b64decode(value, validate=True) + except binascii.Error: + stop(f"invalid {label} encoding") + if len(decoded) != size: + stop(f"invalid {label} length") + return decoded + + +def verify(public_key: str, message: pathlib.Path, signature: pathlib.Path) -> None: + if not message.is_file() or message.is_symlink() or message.stat().st_size > 1024 * 1024: + stop("unsafe checksum metadata") + if not signature.is_file() or signature.is_symlink() or signature.stat().st_size > 65536: + stop("unsafe signature metadata") + public_packet = decode_exact(public_key, 42, "public key") + if public_packet[:2] != b"Ed": + stop("unsupported public-key algorithm") + key_id = public_packet[2:10] + raw_public_key = public_packet[10:] + + try: + signature_bytes = signature.read_bytes() + signature_text = signature_bytes.decode("utf-8") + except (OSError, UnicodeDecodeError): + stop("signature file is not strict UTF-8") + if b"\r" in signature_bytes or b"\x00" in signature_bytes or not signature_text.endswith("\n"): + stop("signature file has invalid line endings") + lines = signature_text[:-1].split("\n") + if len(lines) != 4 or not lines[0].startswith("untrusted comment: "): + stop("signature file has an invalid envelope") + if not lines[2].startswith("trusted comment: "): + stop("signature file omits its trusted comment") + trusted_comment = lines[2][len("trusted comment: ") :] + if not trusted_comment or len(trusted_comment.encode()) > 1024: + stop("trusted comment is invalid") + if not all(character == "\t" or 0x20 <= ord(character) <= 0x7E for character in trusted_comment): + stop("trusted comment contains non-printable data") + + signature_packet = decode_exact(lines[1], 74, "payload signature") + if signature_packet[:2] != b"ED": + stop("legacy or unsupported Minisign signature") + if signature_packet[2:10] != key_id: + stop("signature key identifier does not match") + payload_signature = signature_packet[10:] + comment_signature = decode_exact(lines[3], 64, "trusted-comment signature") + payload_hash = hashlib.blake2b(message.read_bytes(), digest_size=64).digest() + + # RFC 8410 SubjectPublicKeyInfo prefix for a raw Ed25519 public key. + public_der = bytes.fromhex("302a300506032b6570032100") + raw_public_key + with tempfile.TemporaryDirectory(prefix="nehemiah-minisign-") as temporary: + directory = pathlib.Path(temporary) + key_path = directory / "public.der" + payload_path = directory / "payload.blake2b" + payload_signature_path = directory / "payload.sig" + comment_path = directory / "comment" + comment_signature_path = directory / "comment.sig" + key_path.write_bytes(public_der) + payload_path.write_bytes(payload_hash) + payload_signature_path.write_bytes(payload_signature) + comment_path.write_bytes(payload_signature + trusted_comment.encode()) + comment_signature_path.write_bytes(comment_signature) + for data, detached, label in ( + (payload_path, payload_signature_path, "payload"), + (comment_path, comment_signature_path, "trusted comment"), + ): + result = subprocess.run( + [ + "openssl", + "pkeyutl", + "-verify", + "-pubin", + "-inkey", + str(key_path), + "-keyform", + "DER", + "-rawin", + "-in", + str(data), + "-sigfile", + str(detached), + ], + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + check=False, + ) + if result.returncode != 0: + stop(f"{label} signature is invalid") + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--public-key", required=True) + parser.add_argument("--message", required=True, type=pathlib.Path) + parser.add_argument("--signature", required=True, type=pathlib.Path) + args = parser.parse_args() + verify(args.public_key, args.message, args.signature) + + +if __name__ == "__main__": + main() diff --git a/infra/latitude/wireguard-config.py b/infra/latitude/wireguard-config.py new file mode 100755 index 0000000..346ca9f --- /dev/null +++ b/infra/latitude/wireguard-config.py @@ -0,0 +1,269 @@ +#!/usr/bin/env python3 +"""Strict parser and canonical renderer for the managed WireGuard contract.""" + +from __future__ import annotations + +import argparse +import base64 +import binascii +import hashlib +import ipaddress +import pathlib +import re +import sys + + +KEY_FIELDS = { + "Interface": ("PrivateKey", "Address"), + "Peer": ("PublicKey", "Endpoint", "AllowedIPs", "PersistentKeepalive"), +} + + +def stop(message: str) -> "NoReturn": + raise SystemExit(f"managed WireGuard config: {message}") + + +def key(value: str, label: str) -> str: + if not re.fullmatch(r"[A-Za-z0-9+/]{43}=", value): + stop(f"{label} is not a canonical WireGuard key") + try: + decoded = base64.b64decode(value, validate=True) + except binascii.Error: + stop(f"{label} is invalid base64") + if len(decoded) != 32 or decoded == bytes(32): + stop(f"{label} is invalid") + return value + + +def endpoint(value: str) -> str: + if len(value) > 253 or any(character.isspace() for character in value): + stop("peer endpoint is invalid") + host = "" + port_text = "" + if value.startswith("["): + match = re.fullmatch(r"\[([^]]+)]:(\d{1,5})", value) + if not match: + stop("peer endpoint is invalid") + host, port_text = match.groups() + try: + parsed_host = ipaddress.ip_address(host) + except ValueError: + stop("bracketed peer endpoint must contain IPv6") + if parsed_host.version != 6: + stop("bracketed peer endpoint must contain IPv6") + host = parsed_host.compressed + rendered = f"[{host}]:{int(port_text)}" + else: + if value.count(":") != 1: + stop("peer endpoint is invalid") + host, port_text = value.rsplit(":", 1) + if not re.fullmatch(r"\d{1,5}", port_text): + stop("peer endpoint port is invalid") + try: + parsed_host = ipaddress.ip_address(host) + except ValueError: + if ( + not re.fullmatch( + r"(?=.{1,253}\Z)(?:[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?\.)*[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?", + host, + ) + or "." not in host + ): + stop("peer endpoint hostname is invalid") + host = host.lower() + else: + if parsed_host.version != 4: + stop("IPv6 peer endpoints must be bracketed") + host = parsed_host.compressed + rendered = f"{host}:{int(port_text)}" + port = int(port_text) + if not 1 <= port <= 65535: + stop("peer endpoint port is out of range") + return rendered + + +def canonicalize( + raw: bytes, + advertise_address: str, + control_plane_address: str, + gateway_address: str, + guest_subnet: str, +) -> bytes: + if not raw or len(raw) > 65536 or b"\x00" in raw or b"\r" in raw: + stop("configuration is empty, oversized, or has unsafe bytes") + try: + contents = raw.decode("ascii") + except UnicodeDecodeError: + stop("configuration must be ASCII") + try: + advertised = ipaddress.ip_address(advertise_address) + except ValueError: + stop("advertised address is invalid") + try: + control_plane = ipaddress.ip_address(control_plane_address) + gateway = ipaddress.ip_address(gateway_address) + guests = ipaddress.ip_network(guest_subnet, strict=True) + except ValueError: + stop("approved peer or guest network is invalid") + approved_routes = (control_plane, gateway) + if ( + control_plane == gateway + or guests.version != 4 + or guests.prefixlen != 24 + or any(address.version != advertised.version for address in approved_routes) + or any( + address == advertised + or address.is_unspecified + or address.is_loopback + or address.is_link_local + or address.is_multicast + or address in guests + for address in approved_routes + ) + ): + stop("approved CP/gateway addresses are unsafe or ambiguous") + sections: dict[str, dict[str, str]] = {} + current = None + observed_order = [] + for line_number, raw_line in enumerate(contents.splitlines(), 1): + line = raw_line.strip() + if not line: + continue + if line.startswith(("#", ";")): + stop(f"comments are not permitted (line {line_number})") + section_match = re.fullmatch(r"\[([A-Za-z]+)]", line) + if section_match: + current = section_match.group(1) + if current not in KEY_FIELDS or current in sections: + stop(f"unknown or duplicate section on line {line_number}") + sections[current] = {} + observed_order.append(current) + continue + if current is None: + stop(f"field appears before a section on line {line_number}") + field_match = re.fullmatch(r"([A-Za-z]+)\s*=\s*(\S(?:.*\S)?)", line) + if not field_match: + stop(f"malformed field on line {line_number}") + name, value = field_match.groups() + if name not in KEY_FIELDS[current] or name in sections[current]: + stop(f"unknown or duplicate {current} field on line {line_number}") + sections[current][name] = value + if observed_order != ["Interface", "Peer"]: + stop("configuration must contain exactly one Interface then one Peer") + for section, names in KEY_FIELDS.items(): + if set(sections[section]) != set(names): + stop(f"{section} does not contain the exact field set") + + private_key = key(sections["Interface"]["PrivateKey"], "private key") + public_key = key(sections["Peer"]["PublicKey"], "peer public key") + if private_key == public_key: + stop("interface and peer keys must be distinct") + try: + interface = ipaddress.ip_interface(sections["Interface"]["Address"]) + except ValueError: + stop("interface address is invalid") + if ( + interface.ip != advertised + or interface.network.prefixlen != interface.max_prefixlen + or interface.ip.is_unspecified + or interface.ip.is_multicast + ): + stop("interface address must be the advertised host route") + + routes = [] + route_values = [part.strip() for part in sections["Peer"]["AllowedIPs"].split(",")] + if not 1 <= len(route_values) <= 8 or any(not value for value in route_values): + stop("AllowedIPs must contain between one and eight host routes") + for value in route_values: + try: + route = ipaddress.ip_network(value, strict=True) + except ValueError: + stop("AllowedIPs contains an invalid route") + if ( + route.prefixlen != route.max_prefixlen + or route.network_address == advertised + or route.network_address.is_unspecified + or route.network_address.is_multicast + ): + stop("AllowedIPs may contain only distinct CP/gateway host routes") + routes.append(route) + if len(routes) != len(set(routes)): + stop("AllowedIPs contains a duplicate route") + expected_routes = { + ipaddress.ip_network(f"{address}/{address.max_prefixlen}") + for address in approved_routes + } + if set(routes) != expected_routes: + stop("AllowedIPs must exactly match the approved CP/gateway host routes") + routes.sort(key=lambda route: (route.version, int(route.network_address))) + if sections["Peer"]["PersistentKeepalive"] != "25": + stop("PersistentKeepalive must be exactly 25") + peer_endpoint = endpoint(sections["Peer"]["Endpoint"]) + canonical = ( + "[Interface]\n" + f"PrivateKey = {private_key}\n" + f"Address = {interface.ip.compressed}/{interface.max_prefixlen}\n" + "\n" + "[Peer]\n" + f"PublicKey = {public_key}\n" + f"Endpoint = {peer_endpoint}\n" + f"AllowedIPs = {', '.join(str(route) for route in routes)}\n" + "PersistentKeepalive = 25\n" + ) + return canonical.encode() + + +def main() -> None: + parser = argparse.ArgumentParser() + subparsers = parser.add_subparsers(dest="command", required=True) + canonical_parser = subparsers.add_parser("canonicalize-base64") + canonical_parser.add_argument("--advertise-address", required=True) + verify_parser = subparsers.add_parser("verify") + verify_parser.add_argument("--advertise-address", required=True) + verify_parser.add_argument("--expected-sha256", required=True) + verify_parser.add_argument("--path", required=True, type=pathlib.Path) + for subparser in (canonical_parser, verify_parser): + subparser.add_argument("--control-plane-address", required=True) + subparser.add_argument("--gateway-address", required=True) + subparser.add_argument("--guest-subnet", required=True) + args = parser.parse_args() + if args.command == "canonicalize-base64": + encoded = sys.stdin.buffer.read() + if len(encoded) > 131072 or not re.fullmatch(rb"[A-Za-z0-9+/]*={0,2}", encoded): + stop("encoded configuration is invalid") + try: + raw = base64.b64decode(encoded, validate=True) + except binascii.Error: + stop("encoded configuration is invalid") + sys.stdout.buffer.write( + canonicalize( + raw, + args.advertise_address, + args.control_plane_address, + args.gateway_address, + args.guest_subnet, + ) + ) + return + if not re.fullmatch(r"[0-9a-f]{64}", args.expected_sha256): + stop("expected configuration digest is invalid") + try: + stats = args.path.lstat() + except OSError: + stop("configuration file is missing") + if not args.path.is_file() or args.path.is_symlink() or (stats.st_mode & 0o777) != 0o600: + stop("configuration file is unsafe") + raw = args.path.read_bytes() + rendered = canonicalize( + raw, + args.advertise_address, + args.control_plane_address, + args.gateway_address, + args.guest_subnet, + ) + if rendered != raw or hashlib.sha256(raw).hexdigest() != args.expected_sha256: + stop("configuration is not the approved canonical file") + + +if __name__ == "__main__": + main() diff --git a/infra/local/README.md b/infra/local/README.md index a951157..d701ee4 100644 --- a/infra/local/README.md +++ b/infra/local/README.md @@ -1,29 +1,21 @@ -# Running Nehemiah locally (Mac & Windows) - -**The Mac path is built and proven** — one command -([`setup-local.sh`](setup-local.sh)) turns an Apple Silicon Mac into a boring -computers host (in a Lima nested-virt VM), and a real arm64 Firecracker microVM -boots on it in **~5 ms**. Windows is designed but not yet wired up (it's the -easier path — see below). +# Running Nehemiah locally (Mac & Windows) — descoped + +> **This self-serve local path is no longer supported.** `setup-local.sh` built +> the host stack from source in a Lima VM, but host bootstrap now installs +> Firecracker, the jailer, and the kernel only from **signed managed-release +> artifacts**, which a local build cannot supply. `infra/local/setup-local.sh` +> now exits with a pointer to the managed runbook. Provision a managed host from a +> signed release instead — see [`../latitude/README.md`](../latitude/README.md). +> +> The rest of this document is retained as a record of the local architecture (how +> a Mac or Windows box can host a KVM-capable Linux guest), in case a +> local-artifact bootstrap mode is built in the future. nehemiahd runs Firecracker microVMs, which need **Linux + a functional `/dev/kvm`**. Neither macOS nor Windows provides that natively, but both can host a Linux VM that _does_ — Firecracker needs only **one** level of nested virtualization, which modern Macs and Windows 11 both expose. -## Quickstart (Mac) - -```sh -brew install lima # once -NEHEMIAH_ANTHROPIC_KEY=sk-ant-... ./infra/local/setup-local.sh -# → builds the arm64 stack in a Lima VM, forwards :8080 to the Mac at :8088 -echo 'NEHEMIAH_URL=http://localhost:8088' > apps/web/.env -npm run dev -w web # → http://localhost:5173 -``` - -`SKIP_DESKTOP=1` skips the ~8-min desktop image (the python shell still works). -`limactl stop boring` frees the VM's RAM. - | Path | Status | Why | Extra work vs a Linux box | | ----------------------- | ------------------------------- | ---------------------------------------------------------------- | ----------------------------------------------------- | | **Apple Silicon Mac** | ✅ **built + booted a microVM** | nested virt on M3+/macOS 15+ exposes `/dev/kvm` in a Linux guest | arm64 rebuilds — done, automated by `setup-local.sh` | diff --git a/infra/local/setup-local.sh b/infra/local/setup-local.sh index 1ce320f..0d45504 100755 --- a/infra/local/setup-local.sh +++ b/infra/local/setup-local.sh @@ -3,6 +3,10 @@ # setup-local.sh — run a full Nehemiah host locally on an Apple Silicon # Mac, inside a Lima nested-virt Linux VM (which is where /dev/kvm lives). # +# DESCOPED — this self-serve path is no longer supported. Host bootstrap installs +# only signed managed-release artifacts, which this Lima flow cannot supply; it now +# exits with a pointer to the managed runbook (../latitude/README.md). +# # From the repo root on your Mac: # NEHEMIAH_ANTHROPIC_KEY=sk-ant-... ./infra/local/setup-local.sh # @@ -24,6 +28,12 @@ log() { printf '\033[1;34m[local]\033[0m %s\n' "$*"; } die() { printf '\033[1;31m[local:error]\033[0m %s\n' "$*" >&2; exit 1; } invm() { limactl shell "${VM}" -- sudo bash -c "$*"; } +# --- self-serve local bootstrap is descoped --------------------------------- +# bootstrap.sh installs Firecracker/jailer/kernel only from signed managed-release +# artifacts, which this Lima flow cannot stage or forward into the guest. It is +# disabled rather than fail cryptically part-way through provisioning. +die "self-serve local bootstrap is no longer supported: bootstrap.sh installs only from signed managed-release artifacts, which this Lima flow cannot supply. Provision managed hosts with infra/latitude/provision.sh + cloud-init (see infra/latitude/README.md)." + # --- 0. host preflight ------------------------------------------------------- [ "$(uname -s)" = "Darwin" ] || die "this script is for a Mac host; on Linux use infra/setup.sh directly" command -v limactl >/dev/null || die "Lima not installed — run: brew install lima" @@ -46,18 +56,21 @@ invm 'test -e /dev/kvm' || die "/dev/kvm missing in the guest — nested virtual GUEST_ARCH="$(limactl shell "${VM}" -- uname -m)" log " ok: guest is ${GUEST_ARCH} with /dev/kvm" -# --- 3. cross-build nehemiahd for the guest arch on the Mac -------------------- -log "Cross-building nehemiahd for linux/${GUEST_ARCH}…" +# --- 3. cross-build the host and guest agents for the nested VM --------------- +log "Cross-building nehemiahd + guest agent for linux/${GUEST_ARCH}…" GOARCH="arm64"; [ "${GUEST_ARCH}" = "x86_64" ] && GOARCH="amd64" ( cd "${REPO_ROOT}/nehemiahd" && GOOS=linux GOARCH="${GOARCH}" CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /tmp/nehemiahd-local . ) +( cd "${REPO_ROOT}/guest-agent" && GOOS=linux GOARCH="${GOARCH}" CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /tmp/bc-guest-agent-local . ) log " built $(file -b /tmp/nehemiahd-local | cut -d, -f1-2)" # --- 4. ship infra scripts + nehemiahd binary into the guest ------------------ -log "Shipping infra scripts + nehemiahd into the guest…" +log "Shipping infra scripts + nehemiahd/guest agent into the guest…" invm 'mkdir -p /root/infra /opt/boring/bin' tar czf - -C "${REPO_ROOT}/infra/latitude" . | limactl shell "${VM}" -- sudo tar xzf - -C /root/infra limactl shell "${VM}" -- sudo cp /dev/stdin /usr/local/bin/nehemiahd < /tmp/nehemiahd-local +limactl shell "${VM}" -- sudo cp /dev/stdin /opt/boring/bin/bc-guest-agent < /tmp/bc-guest-agent-local invm 'chmod +x /usr/local/bin/nehemiahd' +invm 'chmod +x /opt/boring/bin/bc-guest-agent' invm 'ln -sfn /usr/local/bin/nehemiahd /usr/local/bin/boringd' # pre-rename name keeps working # --- 5. build the stack in the guest (arch-adapted scripts auto-detect) ------ @@ -73,11 +86,11 @@ else invm 'bash /root/infra/build-desktop-rootfs.sh' || log " desktop image build had issues (python shell still works)" fi log "guest networking (bridge + NAT + egress firewall)…" -invm 'install -m0755 /root/infra/net-setup.sh /opt/boring/bin/net-setup.sh && bash /opt/boring/bin/net-setup.sh && cp /root/infra/boring-net.service /etc/systemd/system/ && systemctl daemon-reload && systemctl enable boring-net.service || true' +invm 'install -m0755 /root/infra/net-setup.sh /opt/boring/bin/net-setup.sh && bash /opt/boring/bin/net-setup.sh && cp /root/infra/boring-net-local.service /etc/systemd/system/boring-net.service && systemctl daemon-reload && systemctl enable boring-net.service || true' # --- 6. nehemiahd config + service (bind 0.0.0.0 so Lima can forward it) ------- log "installing nehemiahd service…" -invm "cp /root/infra/nehemiahd.service /etc/systemd/system/nehemiahd.service" +invm "cp /root/infra/nehemiahd-local.service /etc/systemd/system/nehemiahd.service" limactl shell "${VM}" -- sudo bash -c "install -d -m0755 /etc/boring && umask 077 && cat > /etc/boring/nehemiahd.env" <&2; exit 1; } +# --- self-serve local bootstrap is descoped --------------------------------- +# infra/latitude/bootstrap.sh installs Firecracker/jailer/kernel only from signed +# managed-release artifacts (it requires NEHEMIAH_RELEASE_VERSION, the signed +# archive/kernel inputs, and the managed-host package cohort). This self-serve +# script cannot stage or forward that signed contract into the remote host, so it +# is disabled rather than fail cryptically part-way through provisioning. +die "self-serve local bootstrap is no longer supported: bootstrap.sh installs only from signed managed-release artifacts, which this script cannot supply. Provision managed hosts with infra/latitude/provision.sh + cloud-init (see infra/latitude/README.md)." + # --- 0. preflight ------------------------------------------------------------ log "Preflight on ${TARGET}…" "${SSH[@]}" 'true' || die "can't SSH to ${TARGET}" @@ -44,14 +56,16 @@ esac [[ "${KVM}" == "yes" ]] || die "/dev/kvm missing — the box needs hardware/nested virtualization" log " ok: Ubuntu ${ID:-?} ${ARCH} with /dev/kvm" -# --- 1. ship infra scripts + nehemiahd source ---------------------------------- -log "Copying infra scripts + nehemiahd source…" -"${SSH[@]}" 'mkdir -p /root/infra /opt/boring/src' +# --- 1. ship infra scripts + host/guest agent source -------------------------- +log "Copying infra scripts + nehemiahd/guest-agent source…" +"${SSH[@]}" 'mkdir -p /root/infra /opt/boring/bin /opt/boring/src /opt/boring/guest-agent-src' scp -q -o StrictHostKeyChecking=accept-new \ "${REPO_ROOT}"/infra/latitude/*.sh "${REPO_ROOT}"/infra/latitude/*.service \ "${REPO_ROOT}"/infra/latitude/Caddyfile "${TARGET}:/root/infra/" rsync -az --delete -e "ssh -o StrictHostKeyChecking=accept-new" \ --exclude '*_test.go' "${REPO_ROOT}/nehemiahd/" "${TARGET}:/opt/boring/src/" +rsync -az --delete -e "ssh -o StrictHostKeyChecking=accept-new" \ + --exclude '*_test.go' "${REPO_ROOT}/guest-agent/" "${TARGET}:/opt/boring/guest-agent-src/" # --- 2. install Go (matching go.mod) ----------------------------------------- log "Ensuring Go ${GO_VERSION}…" @@ -61,10 +75,12 @@ if ! /usr/local/go/bin/go version 2>/dev/null | grep -q "go${GO_VERSION}"; then rm -rf /usr/local/go && tar -C /usr/local -xzf /tmp/go.tgz && rm -f /tmp/go.tgz fi /usr/local/go/bin/go version +cd /opt/boring/guest-agent-src +CGO_ENABLED=0 GOOS=linux GOARCH=${GOARCH} /usr/local/go/bin/go build -trimpath -ldflags="-s -w" -o /opt/boring/bin/bc-guest-agent . EOF # --- 3. bootstrap: firecracker, jailer, kernel, base rootfs ------------------ -log "Bootstrap (firecracker + jailer + kernel + base rootfs)…" +log "Bootstrap (firecracker + jailer + kernel + guest-agent rootfs)…" "${SSH[@]}" 'bash /root/infra/bootstrap.sh' # --- 4. build the guest images + snapshot ------------------------------------ @@ -84,7 +100,7 @@ log "Setting up guest networking…" "${SSH[@]}" bash -euo pipefail <<'EOF' install -m0755 /root/infra/net-setup.sh /opt/boring/bin/net-setup.sh bash /opt/boring/bin/net-setup.sh -cp /root/infra/boring-net.service /etc/systemd/system/ 2>/dev/null || true +cp /root/infra/boring-net-local.service /etc/systemd/system/boring-net.service 2>/dev/null || true systemctl daemon-reload && systemctl enable boring-net.service 2>/dev/null || true EOF @@ -94,7 +110,7 @@ log "Building + installing nehemiahd…" cd /opt/boring/src CGO_ENABLED=0 /usr/local/go/bin/go build -trimpath -ldflags="-s -w" -o /usr/local/bin/nehemiahd . ln -sfn /usr/local/bin/nehemiahd /usr/local/bin/boringd # pre-rename name keeps working -cp /root/infra/nehemiahd.service /etc/systemd/system/nehemiahd.service +cp /root/infra/nehemiahd-local.service /etc/systemd/system/nehemiahd.service EOF log "Writing config (secrets not printed)…"