Metal: platform-aware expert-cache defaults via honest storage probe (#379)

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic

Problem

On Apple Silicon (Metal backend, unified memory, fast NVMe), colibri's stock
defaults leave a large performance win on the table. Measured campaign
(issue #379, M5 Max 128 GB):

config	tok/s
stock defaults	0.4
tuned (MTP=0 CAP_RAISE=0 --ram 90 --cap 1)	~2.0

Scope caveat, to prevent over-reading: the tuned row is a four-knob
configuration. This PR changes the defaults of exactly two of those knobs
(cap, CAP_RAISE); MTP and RAM sizing remain manual and out of scope. Stock
Metal runs with these defaults should therefore land somewhere between 0.4
and 2.0 tok/s, not at 2.0 — this PR is a partial capture of the measured
configuration, and how much of the 5x the two cache knobs alone recover is
exactly what the on-box validation run (below) will measure.

Issues #180 (M3 Ultra) and #107 (M4 Pro) corroborate the same shape on other
Apple Silicon boxes. As the maintainer noted in #379: no one maintaining this
repo has Metal hardware, and this PR was explicitly invited from the field.

Mechanism (#379)

Metal's per-buffer residency cost makes the engine's own expert LRU cache
anti-productive on fast NVMe. The OS page cache already streams cold
experts back in cheaper than Metal can re-pin them into GPU-visible buffers,
so a large expert cache just adds residency churn on top of I/O that was
already going to happen. On this class of hardware, a minimal cache (so the
engine gets out of the page cache's way) beats a large one.

Whether a volume is "fast" has to be measured, not assumed: buffered
reads lie (page-cache hits read 23-97 GB/s on the reference box; a real
F_NOCACHE read measured 14 GB/s on the same file). That's why this PR adds
an honest storage probe rather than just flipping the default unconditionally
on every Metal build.

Design

S1 -- honest storage probe (glm.c, coli_ssd_probe_raw/coli_ssd_probe_cached)
C-side, in the engine's own startup path, so it covers glm direct users and
the coli wrapper alike -- Python-only logic would miss direct users.
Mirrors compat_open_direct()/iobench.c's existing __APPLE__ branch: open
a real model shard (readdir order -- any shard is representative for a
bandwidth probe), fcntl(F_NOCACHE,1), disable readahead, random 16K-aligned
pread, single thread, ~0.35s wall-clock budget. The result is cached in
<model>/.coli_ssd (plain-text GB/s) so every startup after the first reads
a file instead of re-probing; a missing or unparsable cache file just
re-probes. The cache write is atomic (tmp+rename, the same model-dir
convention as stats_dump_q); if two engines race on a virgin dir (e.g. a
concurrent serve + run pair), both probe under mutual contention and may
cache a low reading -- the safe direction (defaults simply stay historic);
delete .coli_ssd to re-measure, and an unparsable file re-probes
automatically on the next start.

S2 -- platform-aware cache defaults (glm.c, coli, openai_server.py)
When g_metal_enabled && darwin && probe-fast:

• expert LRU cap defaults to 1/layer.
• CAP_RAISE defaults to 0 (previously 1) -- raising the cache back up
  under a generous RAM budget is the same anti-pattern the cap default exists
  to avoid.

MTP, RAM sizing, and autopin are untouched -- out of scope for this PR.

Precedence (coli_resolve_cap(), pure function, unit-tested in isolation):

tier	cap	CAP_RAISE
1 (highest)	explicit --cap N CLI	--
2	explicit CAP env	explicit CAP_RAISE env
3	platform default (Metal + darwin + fast SSD) -> 1	platform default (same condition) -> 0, independently of how cap was set -- see note
4 (lowest)	historic default: 8 via the coli wrapper ("0 = auto" sentinel), 64 for a bare ./glm with no positional argument	historic default -> 1

An explicit setting is never overridden by the probe, at any tier.

Note on the cap/CAP_RAISE coupling (deliberate): on a qualifying box, an
explicit --cap 16 does not re-enable auto-raise -- CAP_RAISE still
defaults to 0 there. Rationale: the Metal residency cost scales with cache
growth, so auto-raise is the same mechanism the minimal-cache default avoids;
a user who explicitly sizes the cache gets exactly that size. Explicit
CAP_RAISE=1 re-enables auto-raise.

S3 -- the one-line notice (glm.c, stderr, alongside the existing config banner)
Printed only when the platform cap default actually engages:

METAL: fast SSD (X.X GB/s) — page cache favored, expert cache minimal (cap 1); override with --cap

Silent when the probe ran but storage was slow -- defaults stay at their
historic values with no extra output.

S4 -- doctor/plan surfacing (doctor.py, resource_plan.py)
coli doctor and coli plan read <model>/.coli_ssd if present and display
it. Read-and-display only: neither command re-measures or guesses. A new
storage.ssd_probe doctor check reports "pass" with the cached GB/s, or
"skip" before the first Metal+darwin startup.

What's explicitly out of scope

• MTP, RAM sizing (--ram), autopin -- their defaults are untouched (and are
  the other half of the measured 5x; see the scope caveat at the top).
• Routing, CACHE_ROUTE, sampling -- nothing here can change a generated
  token. Cache sizing is a pure performance knob: it changes how often an
  expert is re-read from disk, never which expert gets computed for a given
  token.
• Non-Metal, non-darwin, or slow-storage runs: the probe never engages
  (g_metal_enabled stays 0, or the measured GB/s is under the threshold,
  default 4.0 GB/s, COLI_SSD_FAST_GBS to override), and every invocation
  resolves byte-identically to today -- including the bare ./glm fallback
  of 64.

Behavioral disclosures (small, but stated plainly)

1. --cap 0 / CAP=0 now mean "auto" on every platform, not just Metal.
   This is the sentinel that lets the wrapper say "not explicitly set."
   Previously a literal cap=0 was accepted and produced a working zero-slot,
   always-cold-load cache mode; that mode was undocumented and unused
   repo-wide (no test, tool, doc, or wrapper passes 0). Anyone who somehow
   relied on it can get the nearest supported equivalent with --cap 1.
2. coli's --cap argparse default changed from 8 to 0 (auto).
   Without this, coli always forced an explicit 8 onto the engine command
   line, which (per the precedence table) would have permanently shadowed the
   platform default for every wrapped invocation. openai_server.py's
   --cap flag and its internal Engine/serve() defaults got the same
   sentinel so programmatic callers don't silently bypass auto.
3. The benchmark harness (tools/eval_glm.py) keeps its fixed --cap 64
   deliberately -- benchmarks need a reproducible cache size, not a
   platform-dependent auto (commented in the source).

Durable vs. current-state (a review map)

Two kinds of change ship here, and they age differently:

Durable machinery -- the F_NOCACHE probe and .coli_ssd cache, the
coli_resolve_cap() precedence chain and cap sentinel, the notice
mechanism, and the doctor/plan surfacing. None of these encode an opinion
about how big the cache should be; they survive any change in residency
economics.

Current-state calibration -- the qualifying-platform values: cap 1
and CAP_RAISE 0. These encode today's measured economics (per-buffer
Metal residency cost makes a growing expert cache anti-productive, #379)
and are deliberately concentrated so that revisiting them is a two-constant
edit. "Today" is a checkable coordinate, not rhetoric: measured 2026-07 on
macOS 26.5 (M5 Max, 128 GB) against engine base caa49f7 (dev) -- the
in-code marker comments carry the same stamp. A reader far from those
coordinates (newer macOS Metal runtime, or an engine that has absorbed the
#379 residency work) should treat the constants as suspect and re-run the
#379 A/B (cap 1 vs cap 16, hit rate vs tok/s) before trusting them. Two
foreseeable developments would flip them:

1. The residency-mechanism work in Metal — growing the expert cache makes decode SLOWER #379 itself. The MTLHeap experiment
   already flips the sign of caching on the reference box (cap16 ~ +5% over
   cap1 with heap-backed slabs); a follow-up PR is planned from the same
   investigation. If it lands, the right qualifying default likely becomes a
   larger, RAM-scaled cap -- not 1.
2. Metal's move toward first-class tensor objects (MTLTensor and
   runtime-managed residency in current and upcoming macOS releases). If
   expert storage migrates there, the per-buffer registration cost -- the
   entire reason cap 1 wins today -- changes wholesale.

In both cases the edit is confined to the platform-default branch of
coli_resolve_cap() and the single CAP_RAISE default expression (both
carry a marker comment in the code); probe, precedence, notice, and
surfacing are unaffected. The COLI_SSD_FAST_GBS threshold (4.0) is
likewise a current-economics number, with the env override as escape hatch.

Testing

Author-side testing was parse-level and unit-level only -- no model was
loaded and no inference was run:

• tests/test_cap_precedence.c -- exhaustive precedence-table coverage of
  coli_resolve_cap(), including bare-invocation vs wrapper-sentinel vs
  explicit CLI/env on fast and slow platforms. Portable (no Metal/Apple
  hardware required), part of the default make test-c gate.
• tests/test_resource_plan.py / tests/test_doctor.py -- cache-file
  parsing (read_ssd_probe: missing, empty, malformed, negative, valid) and
  the doctor/plan surfacing contract.
• make glm METAL=1 and plain make glm both compile clean
  (-Wall -Wextra); full make test-c + python unittest suites pass.
• Startup-path smoke tests against a scratch directory containing only a
  dummy *.safetensors file (no config.json, so every run exits before
  model_init -- deliberately): notice printed + cap=1 on a fast volume;
  .coli_ssd written once (atomically) and reused on the next start;
  --cap 16 / CAP=4 override silently; forced-slow threshold and
  non-Metal builds stay silent at historic defaults; corrupted .coli_ssd
  re-probes; bare ./glm still resolves cap 64.

On-box validation (performed 2026-07-18): M5 Max 128 GB, macOS 26.5.2,
real GLM-5.2 int4 model, engine at this branch's HEAD (base caa49f7,
dev). Three-run block, fixed prompt,
temp 0, 32 decoded tokens each:

1. Stock defaults (no cache flags, no env): notice printed
   (METAL: fast SSD (9.7 GB/s) -- page cache favored, expert cache minimal (cap 1); override with --cap), cap resolved to 1, .coli_ssd written
   once. Decode 1.48 tok/s vs 0.4 at the old defaults -- the two cache
   knobs alone recover roughly 3.7x of the measured 5x (MTP and RAM sizing
   remain manual, as scoped above).
2. Explicit --cap 16: silent (no notice), cap honored at 16/layer,
   .coli_ssd untouched. Decode 0.86 tok/s despite the hit rate rising
   24% -> 50% -- a same-session replication of the Metal — growing the expert cache makes decode SLOWER #379 mechanism this PR
   encodes: growing the Metal-registered cache makes decode slower even as
   it hits more.
3. Restart, stock again: cached probe reused (.coli_ssd mtime unchanged,
   same 9.7 GB/s in the notice), cap=1 again, decode 1.49 tok/s
   replicating run 1; temp-0 output text identical across the two stock runs.

[JustVugg]

Heads-up: c/glm.c was just split into c/colibri.c + header modules (#391, now merged into dev). This PR touches the old glm.c, so it will need a rebase onto current dev with its changes moved into colibri.c (or the relevant extracted header: quant.h, sample.h, kv_persist.h, telemetry.h, grammar.h). The make glm target still works (alias of make colibri), so no build scripts break. Apologies for the churn — ping me if the rebase gets thorny and I can help place the hunks.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic

Rebase after #391

This PR was rebased (relocation-only, no behavior change) from engine base
caa49f7 onto dev @ 61004dc, after upstream PR #391 split c/glm.c into
c/colibri.c + four extracted header modules (quant.h, sample.h,
kv_persist.h, telemetry.h). Everything this PR touches -- the F_NOCACHE
probe, coli_resolve_cap(), the cap/CAP_RAISE wiring in main(), and
cap_for_ram()'s conditional auto-raise -- lives in code that #391 kept in
colibri.c itself; none of it landed in an extracted header. git range-diff caa49f7..f3a7fb9 61004dc..HEAD confirms the only tree-level changes are the
file rename (glm.c → colibri.c in diff hunk headers) and updating this
PR's own stray glm.c filename references to colibri.c: a Makefile
prerequisite list (which also gained the four split-out headers --
quant.h sample.h kv_persist.h telemetry.h -- mirroring the sibling
test_dsa_select target's own convention post-#391, not just a filename
swap), a test's #include, and doc-comment mentions of "where this lives".
The two commit titles had the same stray reference (glm.c: honest F_NOCACHE... and glm.c/coli/openai_server.py: platform-aware...) and have
likewise been reworded to colibri.c: ... / colibri.c/coli/openai_server.py: ... -- message-only, trees unchanged. make glm is still a working alias
for make colibri (unaffected by this PR either way). Below, every glm.c
reference in the Design section has been updated to colibri.c to match;
the on-box validation numbers in Testing predate the rebase (measured
against caa49f7) and will be re-verified against 61004dc by the
orchestrator separately -- no model was run as part of this rebase.

Problem

On Apple Silicon (Metal backend, unified memory, fast NVMe), colibri's stock
defaults leave a large performance win on the table. Measured campaign
(issue #379, M5 Max 128 GB):

config	tok/s
stock defaults	0.4
tuned (MTP=0 CAP_RAISE=0 --ram 90 --cap 1)	~2.0

Scope caveat, to prevent over-reading: the tuned row is a four-knob
configuration. This PR changes the defaults of exactly two of those knobs
(cap, CAP_RAISE); MTP and RAM sizing remain manual and out of scope. Stock
Metal runs with these defaults should therefore land somewhere between 0.4
and 2.0 tok/s, not at 2.0 — this PR is a partial capture of the measured
configuration, and how much of the 5x the two cache knobs alone recover is
exactly what the on-box validation run (below) will measure.

Issues #180 (M3 Ultra) and #107 (M4 Pro) corroborate the same shape on other
Apple Silicon boxes. As the maintainer noted in #379: no one maintaining this
repo has Metal hardware, and this PR was explicitly invited from the field.

Mechanism (#379)

Metal's per-buffer residency cost makes the engine's own expert LRU cache
anti-productive on fast NVMe. The OS page cache already streams cold
experts back in cheaper than Metal can re-pin them into GPU-visible buffers,
so a large expert cache just adds residency churn on top of I/O that was
already going to happen. On this class of hardware, a minimal cache (so the
engine gets out of the page cache's way) beats a large one.

Whether a volume is "fast" has to be measured, not assumed: buffered
reads lie (page-cache hits read 23-97 GB/s on the reference box; a real
F_NOCACHE read measured 14 GB/s on the same file). That's why this PR adds
an honest storage probe rather than just flipping the default unconditionally
on every Metal build.

Design

S1 -- honest storage probe (colibri.c, coli_ssd_probe_raw/coli_ssd_probe_cached)
C-side, in the engine's own startup path, so it covers glm direct users and
the coli wrapper alike -- Python-only logic would miss direct users.
Mirrors compat_open_direct()/iobench.c's existing __APPLE__ branch: open
a real model shard (readdir order -- any shard is representative for a
bandwidth probe), fcntl(F_NOCACHE,1), disable readahead, random 16K-aligned
pread, single thread, ~0.35s wall-clock budget. The result is cached in
<model>/.coli_ssd (plain-text GB/s) so every startup after the first reads
a file instead of re-probing; a missing or unparsable cache file just
re-probes. The cache write is atomic (tmp+rename, the same model-dir
convention as stats_dump_q); if two engines race on a virgin dir (e.g. a
concurrent serve + run pair), both probe under mutual contention and may
cache a low reading -- the safe direction (defaults simply stay historic);
delete .coli_ssd to re-measure, and an unparsable file re-probes
automatically on the next start.

S2 -- platform-aware cache defaults (colibri.c, coli, openai_server.py)
When g_metal_enabled && darwin && probe-fast:

• expert LRU cap defaults to 1/layer.
• CAP_RAISE defaults to 0 (previously 1) -- raising the cache back up
  under a generous RAM budget is the same anti-pattern the cap default exists
  to avoid.

MTP, RAM sizing, and autopin are untouched -- out of scope for this PR.

Precedence (coli_resolve_cap(), pure function, unit-tested in isolation):

tier	cap	CAP_RAISE
1 (highest)	explicit --cap N CLI	--
2	explicit CAP env	explicit CAP_RAISE env
3	platform default (Metal + darwin + fast SSD) -> 1	platform default (same condition) -> 0, independently of how cap was set -- see note
4 (lowest)	historic default: 8 via the coli wrapper ("0 = auto" sentinel), 64 for a bare ./glm with no positional argument	historic default -> 1

An explicit setting is never overridden by the probe, at any tier.

Note on the cap/CAP_RAISE coupling (deliberate): on a qualifying box, an
explicit --cap 16 does not re-enable auto-raise -- CAP_RAISE still
defaults to 0 there. Rationale: the Metal residency cost scales with cache
growth, so auto-raise is the same mechanism the minimal-cache default avoids;
a user who explicitly sizes the cache gets exactly that size. Explicit
CAP_RAISE=1 re-enables auto-raise.

S3 -- the one-line notice (colibri.c, stderr, alongside the existing config banner)
Printed only when the platform cap default actually engages:

METAL: fast SSD (X.X GB/s) — page cache favored, expert cache minimal (cap 1); override with --cap

Silent when the probe ran but storage was slow -- defaults stay at their
historic values with no extra output.

S4 -- doctor/plan surfacing (doctor.py, resource_plan.py)
coli doctor and coli plan read <model>/.coli_ssd if present and display
it. Read-and-display only: neither command re-measures or guesses. A new
storage.ssd_probe doctor check reports "pass" with the cached GB/s, or
"skip" before the first Metal+darwin startup.

What's explicitly out of scope

• MTP, RAM sizing (--ram), autopin -- their defaults are untouched (and are
  the other half of the measured 5x; see the scope caveat at the top).
• Routing, CACHE_ROUTE, sampling -- nothing here can change a generated
  token. Cache sizing is a pure performance knob: it changes how often an
  expert is re-read from disk, never which expert gets computed for a given
  token.
• Non-Metal, non-darwin, or slow-storage runs: the probe never engages
  (g_metal_enabled stays 0, or the measured GB/s is under the threshold,
  default 4.0 GB/s, COLI_SSD_FAST_GBS to override), and every invocation
  resolves byte-identically to today -- including the bare ./glm fallback
  of 64.

Behavioral disclosures (small, but stated plainly)

1. --cap 0 / CAP=0 now mean "auto" on every platform, not just Metal.
   This is the sentinel that lets the wrapper say "not explicitly set."
   Previously a literal cap=0 was accepted and produced a working zero-slot,
   always-cold-load cache mode; that mode was undocumented and unused
   repo-wide (no test, tool, doc, or wrapper passes 0). Anyone who somehow
   relied on it can get the nearest supported equivalent with --cap 1.
2. coli's --cap argparse default changed from 8 to 0 (auto).
   Without this, coli always forced an explicit 8 onto the engine command
   line, which (per the precedence table) would have permanently shadowed the
   platform default for every wrapped invocation. openai_server.py's
   --cap flag and its internal Engine/serve() defaults got the same
   sentinel so programmatic callers don't silently bypass auto.
3. The benchmark harness (tools/eval_glm.py) keeps its fixed --cap 64
   deliberately -- benchmarks need a reproducible cache size, not a
   platform-dependent auto (commented in the source).

Durable vs. current-state (a review map)

Two kinds of change ship here, and they age differently:

Durable machinery -- the F_NOCACHE probe and .coli_ssd cache, the
coli_resolve_cap() precedence chain and cap sentinel, the notice
mechanism, and the doctor/plan surfacing. None of these encode an opinion
about how big the cache should be; they survive any change in residency
economics.

Current-state calibration -- the qualifying-platform values: cap 1
and CAP_RAISE 0. These encode today's measured economics (per-buffer
Metal residency cost makes a growing expert cache anti-productive, #379)
and are deliberately concentrated so that revisiting them is a two-constant
edit. "Today" is a checkable coordinate, not rhetoric: current engine-base
coordinate: 61004dc (dev); the underlying cap-1-vs-cap-16 measurement
itself is unchanged -- taken 2026-07 on macOS 26.5, M5 Max, 128 GB, engine
base caa49f7 -- not redone as part of this rebase. The in-code marker
comments still carry that original caa49f7 stamp, untouched (per the
relocation-only rebase: only this document's coordinate label moves to the
new base). A reader far from those coordinates (newer macOS Metal runtime,
or an engine that has
absorbed the #379 residency work) should treat the constants as suspect and
re-run the #379 A/B (cap 1 vs cap 16, hit rate vs tok/s) before trusting
them. Two foreseeable developments would flip them:

1. The residency-mechanism work in Metal — growing the expert cache makes decode SLOWER #379 itself. The MTLHeap experiment
   already flips the sign of caching on the reference box (cap16 ~ +5% over
   cap1 with heap-backed slabs); a follow-up PR is planned from the same
   investigation. If it lands, the right qualifying default likely becomes a
   larger, RAM-scaled cap -- not 1.
2. Metal's move toward first-class tensor objects (MTLTensor and
   runtime-managed residency in current and upcoming macOS releases). If
   expert storage migrates there, the per-buffer registration cost -- the
   entire reason cap 1 wins today -- changes wholesale.

In both cases the edit is confined to the platform-default branch of
coli_resolve_cap() and the single CAP_RAISE default expression (both
carry a marker comment in the code); probe, precedence, notice, and
surfacing are unaffected. The COLI_SSD_FAST_GBS threshold (4.0) is
likewise a current-economics number, with the env override as escape hatch.

Testing

Author-side testing was parse-level and unit-level only -- no model was
loaded and no inference was run:

• tests/test_cap_precedence.c -- exhaustive precedence-table coverage of
  coli_resolve_cap(), including bare-invocation vs wrapper-sentinel vs
  explicit CLI/env on fast and slow platforms. Portable (no Metal/Apple
  hardware required), part of the default make test-c gate.
• tests/test_resource_plan.py / tests/test_doctor.py -- cache-file
  parsing (read_ssd_probe: missing, empty, malformed, negative, valid) and
  the doctor/plan surfacing contract.
• make glm METAL=1 and plain make glm both compile clean
  (-Wall -Wextra); full make test-c + python unittest suites pass.
• Startup-path smoke tests against a scratch directory containing only a
  dummy *.safetensors file (no config.json, so every run exits before
  model_init -- deliberately): notice printed + cap=1 on a fast volume;
  .coli_ssd written once (atomically) and reused on the next start;
  --cap 16 / CAP=4 override silently; forced-slow threshold and
  non-Metal builds stay silent at historic defaults; corrupted .coli_ssd
  re-probes; bare ./glm still resolves cap 64.

Post-rebase gates (performed against dev @ 61004dc, this branch's HEAD):
make glm METAL=1 and make glm METAL=0 both build clean, zero warnings
(-Wall -Wextra, matched against a zero-warning stock 61004dc build --
zero new warnings either way); make check (portable build + full C test
suite, 18 binaries including test_cap_precedence + full Python
unittest discover, 86 tests) passes, exit 0. test_cap_precedence was
spot-checked for vacuous-pass risk by temporarily neutering the
platform_fast branch in coli_resolve_cap(): the mutation produced 3 real
test failures (reverted immediately after). No model was run (no coli run/
coli serve) as part of this rebase, per the rebase task's constraints.

On-box validation (performed 2026-07-18, predates this rebase): M5 Max 128
GB, macOS 26.5.2, real GLM-5.2 int4 model, engine at this branch's
pre-rebase HEAD (base caa49f7, dev). Three-run block, fixed prompt,
temp 0, 32 decoded tokens each:

1. Stock defaults (no cache flags, no env): notice printed
   (METAL: fast SSD (9.7 GB/s) -- page cache favored, expert cache minimal (cap 1); override with --cap), cap resolved to 1, .coli_ssd written
   once. Decode 1.48 tok/s vs 0.4 at the old defaults -- the two cache
   knobs alone recover roughly 3.7x of the measured 5x (MTP and RAM sizing
   remain manual, as scoped above).
2. Explicit --cap 16: silent (no notice), cap honored at 16/layer,
   .coli_ssd untouched. Decode 0.86 tok/s despite the hit rate rising
   24% -> 50% -- a same-session replication of the Metal — growing the expert cache makes decode SLOWER #379 mechanism this PR
   encodes: growing the Metal-registered cache makes decode slower even as
   it hits more.
3. Restart, stock again: cached probe reused (.coli_ssd mtime unchanged,
   same 9.7 GB/s in the notice), cap=1 again, decode 1.49 tok/s
   replicating run 1; temp-0 output text identical across the two stock runs.

These numbers predate the rebase onto 61004dc and will be re-verified
on-box against the rebased branch by the orchestrator separately -- this
rebase performed no model runs.

Post-rebase on-box verification (2026-07-19, M5 Max 128 GB, macOS 26.5, base 61004dc)

Smoke on the rebased branch (0822b18), METAL=1, GLM-5.2 int4, run via this
branch's own coli wrapper:

• Stock (no cap given): F_NOCACHE probe ran and cached to <model>/.coli_ssd
  (9.98 GB/s), notice fired verbatim: METAL: fast SSD (10.0 GB/s) — page cache favored, expert cache minimal (cap 1); override with --cap; cap resolved to 1
  ([RAM_GB=90.0] cap=1 ok).
• --cap 16: silent (no notice), honored (cap=16 ok).
• Explicit-wrapper tier (incidental but real): driving this branch's engine
  through a pre-Metal: platform-aware expert-cache defaults via honest storage probe (#379) #386 coli wrapper — which injects its historic explicit --cap 8 —
  yields silent cap=8: tier-1 explicit-wins precedence verified end-to-end.
• Usage-file snapshot/restore md5-verified around every run.

[JustVugg]

Rebase needed: this went stale against dev after today's merges (#298 reworked scale plumbing; #192 touched the engine). Your Metal storage-probe logic itself shouldn't need changes — likely just contextual conflicts in colibri.c. If you'd rather I do the mechanical rebase, say so and I'll push it to your branch (maintainer edits are enabled); otherwise all yours.

[monotophic]

Re-based to dev 68ac9ff and verified clean at my end.

[JustVugg]

Rebase heads-up: dev is green and moved (Metal PRs #426/#447 among others landed). Please rebase on current dev, then — since this changes the storage-probe cache defaults — a quick before/after tok/s on Apple Silicon confirming the new defaults don't regress the small-cache case (the #379 symptom). CI green + that measurement and I merge. Thanks @monotophic.

[monotophic]

Rebased, requested verification in my queue. Should be available and posted tonight or tomorrow AM. Machine is pretty tied up and hot at the moment.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic.

Rebase: head 758ceb2 on dev @ 008ecc3. One conflict (both sides appended a
test binary to TEST_BINS — resolved as the superset), everything else
patch-identical to the previously-reviewed content (git range-diff clean; the
CURRENT-STATE CALIBRATION markers and their measurement coordinates untouched).
Gates on the new base: build zero warnings, 22/22 C test binaries, 137 Python tests
OK, metal-test green with COLI_METAL_RESSET both off and on.

The before/after measurement (M5 Max 128 GB, macOS 26.5, plugged; interleaved
A/B/A/B with 100 s gaps, temp-0 fixed prompt, ngen 64, both arms at FULL STOCK
defaults — no env overrides, each arm using its own coli wrapper):

arm	binary	defaults chosen	tok/s
A1	dev @ 008ecc3	stock	0.78
B1	this PR @ 758ceb2	platform tier → cap=1 (notice fired, projected peak 60.3 GB)	1.73
A2	dev @ 008ecc3	stock	0.40
B2	this PR @ 758ceb2	platform tier → cap=1 (projected peak 67.7 GB)	1.82

Two observations:

1. No small-cache regression — the small-cache case IS the new default, and it wins
   2.2–4.5×. The PR's platform tier selects cap=1 on this hardware; both branch
   arms sit stable within ±2.5% of each other.
2. The stock arms' own instability (0.78 → 0.40 across two interleaved runs on a
   quiet machine) is the Metal — growing the expert cache makes decode SLOWER #379 symptom class showing up live: with the historic cache
   defaults, Metal decode both underperforms and drifts as the run history warms the
   engine cache. The branch arms don't exhibit it.

Let me know if you have questions or if there is anything else I can do.

[monotophic]

Rebased onto dev dce7012 to absorb the #509 test_temp_env TEST_BINS addition; range-diff patch-identical; gates re-run green both RESSET states.

[JustVugg]

This conflicts with dev now, and part of that is my doing — several PRs landed today that touch the same files (openai_server.py gained an Anthropic /v1/messages endpoint in #525 and an engine-path fix in #526; colibri.c changed in #532). Sorry for the churn.

The substance of this one — platform-aware Metal cache defaults driven by an honest storage probe rather than a guess — is something I want, and it's the same discipline as your fmt=7 proposal: measure, then decide, and refuse to pretend when the measurement isn't available. Rebase onto current dev and I'll review it properly.

One thing worth confirming while you're in there: #457 (your grouped-int4 Metal GEMV) and this PR both touch Metal defaults. If they interact — e.g. if fmt=4 support changes what a sensible default cache size is — say which should land first and I'll sequence them that way rather than discovering it at merge time.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic
(we/our = the joint work; me/my = @monotophic alone).

Rebased: r6, head 3e6c086 on dev @ f1b0f7e.

Same discipline as r4/r5: git range-diff shows commits 1 and 3
patch-identical to what you've already seen; commit 2 differs in exactly
one line — the serve() signature conflict you predicted, where our
cap=8 → cap=0 change met #526's engine=None fix. The resolution keeps
both (cap=0, … engine=None); the cap=0 "engine resolves it" handling
below the signature is untouched. The CURRENT-STATE CALIBRATION markers
and their measurement coordinates (2026-07, macOS 26.5, M5 Max, base
commit stamp) are untouched — they date the measurement, not the rebase.

Gates on the new base (M5 Max, macOS 26.5):

• METAL=1 build zero warnings; per-commit builds clean
• make check: all C test binaries + 151 Python tests OK (10 platform
  skips; count includes this branch's own probe/cap-precedence tests)
• metal-test ok under both COLI_METAL_RESSET states
• behavior evidence (notice fires, probe caches to .coli_ssd, explicit
  --cap stays silent and honored, cached-probe reuse on restart) carries
  from the r4/r5 validation runs — the rebase is patch-identical at those
  code paths; happy to re-run the smoke on request

Sequencing: full four-PR order + reasoning is on #457 (short version:
#457 first, this second, #528 revised third, #529 stacked last). Your
interaction question is answered there in full — the one-line version:
no interaction; the cap-1 default is #379 residency economics,
format-independent, and its revisit trigger is the residency-set default
flip, not fmt=4 support.

[JustVugg]

This conflicts with dev now — several PRs landed today touching colibri.c/coli/openai_server.py. The substance (platform-aware Metal cache defaults from an honest storage probe) is something I want, same discipline as your fmt=7 work: measure, then decide. Rebase onto current dev and I'll review it. If it interacts with #457 (both touch Metal defaults), say which should land first and I'll sequence them.

[monotophic]

Rebased and ready for review. Verification on b68cdf7: git range-diff against r6 shows commits 1 and 3 patch-identical; commit 2 differs only in the Makefile TEST_BINS line (the disclosed superset — dev's test_router_nan + this PR's test_cap_precedence). All three CURRENT-STATE CALIBRATION markers are present with their measurement coordinates intact (2026-07, macOS 26.5, M5 Max). Gates on this head: make check (full C suite including this PR's own test_cap_precedence, plus the Python suite with its probe/cap tests — 13 platform skips), METAL=1 build with zero warnings, and metal-test under both COLI_METAL_RESSET states. Behavior evidence (notice fires, probe caches to .coli_ssd, explicit --cap honored, cached-probe reuse) carries from the r4–r6 validation runs — the rebase is patch-identical at those code paths, and we'll re-run the smoke on request.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic (We/our = the joint analysis; me/my = @monotophic's own hardware and containers).

First, an overdue answer to your question from the 23rd — apologies, our last comment was a rebase log when you'd asked a sequencing question:

#457 interaction: none of substance. #457 is fmt=4 Metal kernels (GEMV + entry-point/allocator fixes); this PR is cache-default policy (probe + precedence). Disjoint surfaces — the only file both touch is colibri.c, in unrelated regions. Land in either order; given the gate now in motion on #457, #457 first and this one whenever it suits you.

Second, r8 is pushed: rebased onto current dev (21e7a35) to clear the conflict from the serve-fix wave. Range-diff r7→r8: commits 1 and 3 patch-identical; commit 2 differs only in the conflicted serve() signature line, resolved as the union (our cap=0 platform default + #606's allowed_hosts). Gates re-run on the exact head: METAL build zero warnings, make check 206 OK, metal-test under both COLI_METAL_RESSET states, all three commits build individually.

Third — an observation from watching this PR eat five rebases, offered as your choice, since all three shapes are the same work for us:

1. Merge as-is — it's current and green.
2. Split it: S1 (the storage probe, 92 lines, pure addition) + S4 (doctor/resource_plan surfacing, pure addition) merge trivially and stop rotting; the default-change commit (the conflict magnet — it threads serve()/coli) becomes its own small PR.
3. Fold the default change forward: post-Metal: rebase MTLResidencySet expert residency onto split runtime #426, our measurements say the deeper default fix is flipping COLI_METAL_RESSET on for macOS 15+ (with residency fixed, the cap curve flattens to 2.28–2.39 across cap1→96 — the protective cap-1 mostly matters in the unflipped world; numbers on Metal — growing the expert cache makes decode SLOWER #379). If you'd rather change platform defaults once, we'd merge the probe+doctor now and bring one combined defaults PR (RESSET flip + probe-driven cap recalibration together, fresh measurements on the current base).

We have a mild preference for (3) as the cleanest end state, but any of the three gets our full follow-through — say the word and it happens.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic (We/our = the joint analysis; me/my = @monotophic's own hardware).

Rebased onto current dev c031307 — new head 491c0a1, rebase-only, no semantic change; the three-shape offer from the previous comment stands unchanged.

One conflict existed and its resolution is the only content difference: #615's deep-doctor rewrite of run_doctor collided with this PR's S4 ssd-probe surfacing in doctor.py. Resolved as the union — both features present, #615's wider TypeError-inclusive except kept.

Verification, one decisive check per claim:

claim	evidence
commits other than the conflicted one are byte-identical to the reviewed r8	git range-diff: 2/3 =; the third differs only in the union hunk
the merged run_doctor region works with both features	tests/test_doctor.py all 24 green on the resolved file — #615's 16 test_deep_check_* and this PR's 2 test_ssd_probe_check_* (pass-path and skip-path) in one suite
nothing else regressed	make check 211 OK · METAL=1 build zero warnings · metal-test under both COLI_METAL_RESSET states · per-commit bisect builds 3/3

The S1–S3 measurement evidence from r8 carries under patch identity (those commits replayed byte-identical); only the S4 commit changed, and its evidence above is re-run on the new head.