feat(323): P3a — a local model's window is sized from the machine it runs on

[munchist]

Plan 323 P3a: each model's context window is now sized from the machine it runs on, and the operator can see and manage what a local model is holding. Follows #118.

Why

On the 24 GB M3, Helmsman on Gemma 4 26B worked but was slow. The cause was measured (plan §9 G6):

• Memory cost: the model takes its weights (16.18 GB) plus about 120 KB for every token of context.
• The cache only grows: on the MLX runner it grows with what the model has read, keeps more than one conversation, and is freed only on unload. The live session reached 22.7 GB.
• Past Metal's GPU limit (19.07 GB here) it doesn't fail; it spills. Prompt reading roughly halves in speed and swap grows.

The fixed 32K default context window was too large for this machine and too small for a big one.

What changes

• The window follows the machine. A connection with no window set gets, for each model, the largest window that fits the GPU memory, never below the 32K the platform's agents need. Helmsman's first prompt alone is about 18K tokens, and slow is better than refused.
  • A 64 GB machine gets Gemma's full 256K window.
  • The 24 GB Mac fits only 16K, so it runs at 32K and says it will spill.
  • A window you set yourself is used exactly as set.
• Fit (localModelFit) answers fits / tight / does not fit / unknown for a model, a machine and a window.
• The machine is read once. The host executor reads the GPU limit through Metal (or nvidia-smi on Linux), caches it in ~/.aflow/machine.json, and publishes it in its inventory. The AI executor, titles and judges all size the same window.
• yarn models:verify-local measures each model's memory footprint and fails on drift. Gemma's entry passes at 16.15 GB and 120,469 bytes per token.
• Readiness reports each local role's window, its fit, and what the runtime holds now. A model spilling because it holds every conversation's cache says so, under the model picker.
• Load now / Unload sit beside each local role, through operator-only routes refused outside the local edition. Loading is the operator's choice, never a side effect of picking a model.
• A runtime on another machine: the connection's new gpu_memory_gb field sizes its windows. This machine's figures are never applied to a runtime elsewhere.

Verified

• Unit tests: fit rules, the window per connection, the machine probe and its cache, the machine-resolution rule, the runtime client, the route refusals, and the UI wording.
• Full checks: the full yarn test with no database (16,142 passed; the one failure is the shared-Redis test below), tsc -b, the UI typecheck, format:check, and repo-wide lint with 0 errors.
• Live on the 24 GB M3: the Metal probe read 19.07 GB; the probe measured Gemma's footprint within tolerance; Gemma loaded at 32K and unloaded on request.

Not yet run

The dev stack on this branch. This means the host executor publishing the machine, the note and the Load/Unload buttons in the UI, and a Helmsman conversation at the automatic window. Once this branch is merged into live, open Models & reasoning and look under the role on Gemma.

Notes

• One pre-existing test touches the live stack. streamRetention.test.ts › "never lets consumer hygiene delete a consumer…" runs consumer cleanup across every stream on this machine's shared Redis, the running dev stack's included. It can fail while a stack runs, and it briefly removes that stack's idle consumers, which re-register on their next read. It's unrelated to this branch and worth its own fix.
• Biggest remaining win for small machines: a smaller Helmsman prompt for local models (plan D8). It is not built.

🤖 Generated with Claude Code