From 3ab465d9aa011a393b107d92c2ecdfbeec064814 Mon Sep 17 00:00:00 2001 From: Mao Nakamoto <41178744+maonakamoto@users.noreply.github.com> Date: Fri, 4 Sep 2026 13:03:11 +0200 Subject: [PATCH] fix(sweep): a test that rotted behind a wrong excuse, and a box spec off by 2x MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three follow-ups from sweeping the earlier findings. Two of the four I had carried forward did NOT survive checking and are dropped: /api/control/dispatch is a real route, so the docs referencing it are correct; and scripts/test/push-notifications.ts exists. Verifying first is why this commit is smaller than the list that prompted it. 1. THE TEST NOBODY RAN, FOR A REASON THAT WAS NEVER TRUE scripts/test/push-notifications.ts read scripts/agent-hook-bridge.sh, which was deleted on 2026-06-11 in 956ccf64 ("delete the bash daemon and bridge files"). It threw ENOENT and the suite exited 1. Its entry in test-unit.ts SKIP said "needs push/web-push env — run manually". That was never the reason: every check in the file is a static file read and it needs no environment at all. A wrong skip reason is worse than no skip. It answers the question that would have found the bug, so the red looked accounted for and nobody looked again for three months. Retargeted rather than deleted. The behaviour moved instead of going away — a closing run reaches the operator through notify-close.ts → pushToUser → push-fanout — and scripts/test/notify-close.ts covers that module without mentioning push at all. So "an agent finished, tell the operator" was asserted by nobody. Deleting the check would have quietly ratified that hole. Removed from SKIP, so it now runs in test:unit (132/132) and therefore in CI. Mutation-proven: renaming the pushToUser call fails with "notify-close must call pushToUser" (exit 1); restored, exit 0. 2. THE BOX IS HALF THE MACHINE THE DOCS DESCRIBE Measured 2026-09-04: nproc 4, 7746 MiB, 75 G disk, 4 GB swap. That is a CX33. Three docs said CX43, 8 vCPU / 16 GB, 40 GB disk — every figure but the swap wrong, in the direction that matters: planning against 16 GB on an 8 GB box is how you meet the OOM killer. src/app/api/system/hetzner/route.ts has said "the box is a cx33" all along, so this was code and docs disagreeing with the code being right. Note the shape of the drift: CPU and RAM went DOWN while disk went UP (40→75), which is what a disk-inclusive rescale to a smaller tier looks like — plausibly to escape the disk-full problem the old caveat describes. The June runbook entry is kept as history and marked superseded rather than rewritten. The docs now point at /api/system/hetzner instead of restating numbers. A hardcoded spec is a second source of truth that rots silently, which is exactly what happened here between June and September. 3. THE COMPOSER STATUS, FINISHED #479 gave this line its own row below `sm` but kept `sm:truncate`, which left it at 57% on a 1440 viewport — "Autopilot on: queue is empty, so FleetCr…". That is the same defect as the phone's 14%, just less obvious, and there is no `title` here so the remainder is unreachable at every width. It now wraps everywhere. Costs one toolbar line when the sentence is long; truncating cost the half of the sentence that says what will happen. pnpm run verify passes (repo moved to pnpm in #478). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UvjGNAS9CMfEGNW26tUR4P --- docs/HANDOFF.md | 2 +- .../box-rescale-ivy-migration-runbook.md | 7 +++ docs/infrastructure/hetzner-migration.md | 24 ++++++++-- scripts/test-unit.ts | 8 +++- scripts/test/push-notifications.ts | 46 ++++++++++++++++--- src/components/control/prompt-input.tsx | 14 ++++-- 6 files changed, 85 insertions(+), 16 deletions(-) diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index d38642d41..001f67009 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -46,7 +46,7 @@ FleetCrown is a multi-user SaaS for builders who run **multiple AI agents across | Surface | URL / path | Status | |---|---|---| | Cloud web app | `https://fleetcrown.orangecat.ch` | Production, self-hosted on Hetzner (Caddy + systemd), Postgres 17 on the same box | -| SSE bridge | `https://bridge.orangecat.ch` | Production, same Hetzner CX43 (8 vCPU / 16 GB, 40 GB disk, €17.29/mo) | +| SSE bridge | `https://bridge.orangecat.ch` | Production, same Hetzner box — **CX33: 4 vCPU / 8 GB / 80 GB** (measured 2026-09-04: `nproc` 4, 7746 MiB, 75 G, 4 GB swap). Live figures: `/api/system/hetzner` | | DB | `postgresql://fleetcrown@postgresqlbridge.orangecat.ch:5432/fleetcrown` | Postgres 17.10, 10 MB used, 39 tables, all healthy | | Desktop app | `Fleet Runner` (Electron 33) | v0.7.5 latest, ships as .deb / .dmg / .exe / AppImage | | Releases | `https://github.com/bitbaum/fleetcrown-releases/releases` | Mirror of build artifacts | diff --git a/docs/development/box-rescale-ivy-migration-runbook.md b/docs/development/box-rescale-ivy-migration-runbook.md index c1f82a1c3..7014a2664 100644 --- a/docs/development/box-rescale-ivy-migration-runbook.md +++ b/docs/development/box-rescale-ivy-migration-runbook.md @@ -42,6 +42,13 @@ The #1 data-loss risk: nightly dumps currently live on the same disk as the DBs. CX22 → **CX43** (8 vCPU / 16 GB) completed 2026-06-16. Verified: 15 GB total, 10 GB free, **0 swap**, no failed services, site 200. No action remaining. +> **Superseded — the box is no longer on this tier.** Measured 2026-09-04: +> 4 vCPU, 7746 MiB, 75 G disk, 4 GB swap — a **CX33**. CPU/RAM went down and the +> disk went up, i.e. a later disk-inclusive rescale to a smaller tier (see +> `docs/infrastructure/hetzner-migration.md`). This entry stays as the record of +> what was done in June; it is not the current spec. For that, read +> `/api/system/hetzner`, never a number written in a doc. + > (Separate note, not gating Ivy: the box was swapping pre-rescale; it's healthy > now. If app count grows, re-evaluate — but that's its own decision.) diff --git a/docs/infrastructure/hetzner-migration.md b/docs/infrastructure/hetzner-migration.md index d214192be..fef6db677 100644 --- a/docs/infrastructure/hetzner-migration.md +++ b/docs/infrastructure/hetzner-migration.md @@ -6,9 +6,27 @@ Neon entirely. Every real app and every database now lives on the `bitbaum` box. Hosted accounts are kept frozen for 14 days as a fallback, then deleted (see Decommission below). -## Box layout (bitbaum · 167.233.22.31 · CX43 8 vCPU / 16 GB · 40 GB disk + 4GB swap) - -> **Disk caveat (2026-06-30):** the CPU/RAM were rescaled to CX43 (8 vCPU / 16 GB) +## Box layout (bitbaum · 167.233.22.31 · CX33 4 vCPU / 8 GB · 80 GB disk + 4 GB swap) + +> **Current spec, measured 2026-09-04:** `nproc` = 4, `free -m` = 7746 MiB, +> `df -h /` = 75 G, swap 4095 MiB. That is a **CX33**, and it is what +> `src/app/api/system/hetzner/route.ts` has said all along ("the box is a cx33 +> and cannot simply be resized: Falkenstein is capacity-blocked"). This heading +> claimed CX43 / 8 vCPU / 16 GB / 40 GB — every figure but the swap was wrong, +> in the direction that matters: planning against 16 GB on an 8 GB machine is +> how you meet the OOM killer. +> +> Note the shape of the drift. CPU and RAM went DOWN (8→4, 16→8) while the disk +> went UP (40→75), which is what a disk-inclusive rescale to a smaller tier +> looks like — a trade made to escape the disk-full problem the caveat below +> describes. The caveat is kept as the historical record of how that arose. +> +> **Do not re-state these numbers elsewhere.** `/api/system/hetzner` reports +> them live; a hardcoded spec is a second source of truth that rots silently, +> which is exactly what happened here between June and September. + +> **Disk caveat (HISTORICAL, 2026-06-30 — superseded by the measurement above):** +> the CPU/RAM were rescaled to CX43 (8 vCPU / 16 GB) > ~2026-06-16, but the rescale was done **"Nur CPU und RAM"** (CPU & RAM only), so > the disk is still stuck at the original **40 GB** (`df` shows 38 GB, ~85–92% full). > The full CX43 tier ships ~160 GB. To claim it requires a **disk-inclusive rescale** diff --git a/scripts/test-unit.ts b/scripts/test-unit.ts index 2ed482ed8..602c1a44d 100644 --- a/scripts/test-unit.ts +++ b/scripts/test-unit.ts @@ -30,7 +30,13 @@ const SKIP: Record = { "authenticated-smoke.ts": "needs a running server + FLEETCROWN_SESSION_TOKEN (pre-push/prod dogfood)", "rag-retrieval.ts": "needs EMBEDDINGS_BASE_URL (fastembed service)", - "push-notifications.ts": "needs push/web-push env — run manually", + // push-notifications.ts was here, excluded as "needs push/web-push env — + // run manually". It never needed env: every check is a static file read. It + // was failing because scripts/agent-hook-bridge.sh was deleted on 2026-06-11 + // (956ccf64), and the skip entry gave a reason that was never the real one — + // so the failure looked accounted for and nobody looked for three months. + // A wrong skip reason is worse than no skip: it answers the question that + // would have found the bug. "inject-prompt.ts": "needs a live DB (only passes locally via .env.local)", "verify-project-brief.ts": "needs a live DB + Groq API (network + GROQ_API_KEY)", }; diff --git a/scripts/test/push-notifications.ts b/scripts/test/push-notifications.ts index adcb3935c..e7246c604 100644 --- a/scripts/test/push-notifications.ts +++ b/scripts/test/push-notifications.ts @@ -1,7 +1,11 @@ /** * Inline self-tests for Web Push (Stage 5) — keeps the subscribe/notify - * surface, service worker, and Stop-hook bridge wired together. + * surface, the service worker, and the run-close notification wired together. * Run: npm run test:push-notifications + * + * Needs no environment: every check is a static file read. It was excluded from + * test:unit for years as "needs push/web-push env", which was never true — see + * the note on the run-close check below. */ import { readFileSync, existsSync } from "fs"; @@ -46,12 +50,40 @@ function runTests(): void { assert(/showNotification/.test(sw), "sw.js must show OS notifications"); }); - check("Stop hook calls push_notify_stop", () => { - const sh = readFileSync("scripts/agent-hook-bridge.sh", "utf8"); - assert(/push_notify_stop/.test(sh), "push_notify_stop helper must exist"); - assert(/\/api\/push\/notify/.test(sh), "push_notify_stop must POST /api/push/notify"); - const stopCalls = sh.match(/push_notify_stop\s+"\$TAB_NAME"/g) ?? []; - assert(stopCalls.length >= 1, "handle_stop must invoke push_notify_stop"); + // "An agent finished, tell the operator" — the point of this whole surface. + // + // This used to read scripts/agent-hook-bridge.sh and assert a bash helper + // called push_notify_stop. That file was deleted on 2026-06-11 (956ccf64, + // "delete the bash daemon and bridge files") when the Stop hook moved into + // TypeScript, so the check threw ENOENT and the whole suite exited 1 — while + // its entry in scripts/test-unit.ts SKIP said it "needs push/web-push env", + // which was never why it failed. A test excluded from CI for a reason that + // was not the real one is a test that rots unread: nothing has asserted this + // path for three months. + // + // Retargeted, not deleted. The behaviour did not go away, it moved — + // notify-close.ts reaches the operator through pushToUser now — and + // scripts/test/notify-close.ts covers that module without mentioning push at + // all. Deleting the check would have quietly ratified the coverage hole. + check("a closing run still pushes to the operator", () => { + const closer = readFileSync("src/lib/orchestration/notify-close.ts", "utf8"); + assert( + /from "@\/lib\/push-fanout"/.test(closer), + "notify-close must reach the operator through push-fanout", + ); + assert(/pushToUser\s*\(/.test(closer), "notify-close must call pushToUser"); + const fanout = readFileSync("src/lib/push-fanout.ts", "utf8"); + assert( + /listSubscriptionsForUser\s*\(/.test(fanout), + "push-fanout must look up the operator's subscribed devices", + ); + // A fan-out that throws would take its caller down with it, and the caller + // is a run finishing — the notification is the least important thing on + // that path. The module's own contract says "must never throw". + assert( + /catch\b/.test(fanout), + "push-fanout must be fire-and-forget, never throwing at callers", + ); }); check("NotificationsPill is wired into AppTopBar", () => { diff --git a/src/components/control/prompt-input.tsx b/src/components/control/prompt-input.tsx index 5920a1d0b..fa6669715 100644 --- a/src/components/control/prompt-input.tsx +++ b/src/components/control/prompt-input.tsx @@ -235,10 +235,16 @@ export function PromptInput({