Production paged + tiered KV cache for the agent army (MAD-126) - #12
Merged
Conversation
Wrapper-side scaffolding for semantic prefetch on paged blocks. Server-
side wiring lands in a follow-up commit; this one adds the storage,
scoring, and prefetch entry point with no behavior change until the
server actually feeds in fingerprints.
## What lands
- BlockSemanticIndex (mt-semantic.{h,cpp}) — keys L2-normalized
embeddings by (seq_id, lblock). Per-seq scoping, cosine top-K with
threshold, no FIFO cap (lifecycle is tied to BlockTable, so memory
grows with the active context size). Worst-case footprint at the
army goal (4 seqs × 8k blocks × 384-dim fp32) is ~48 MiB —
acceptable next to the warm-tier staging cost.
- llama_memory_tiered::record_paged_block_fingerprint — server entry
point, called once per block at backup time. Caller (server-context)
is responsible for embedding the block's tokens via embed_text.
- llama_memory_tiered::restore_semantic_paged — query-time prefetch.
Scores this seq's paged-block fingerprints against the new query
embedding, expands lblock hits to position ranges, calls
paged_restore_from_warm. Logs hit-rate (positions restored vs
requested) for the smoke gating in MAD-122 acceptance #5.
- Lifecycle hooks: clear() drops everything; the paged whole-seq
wipe in seq_rm drops only this seq's entries. Per-seq scoping is
real here (unlike the chunk-keyed semantic_, which clears globally
because its fingerprints aren't seq-keyed).
## Scope decision (vs the ticket)
The ticket also called for a Hybrid eviction policy weighting that
combines semantic score with LRU. Honest assessment landed
elsewhere: bge-small was trained for retrieval similarity, not
forward-looking causal-attention prediction; the synchronous CPU
cost at decode-step granularity is too high for the eviction hot
path; and the actual MAD-120 capacity-pressure problem is
structural (causal attention demands all of an active seq's blocks
be hot-resident, regardless of semantic score). Eviction stays
LRU-primary; semantic earns its keep on async prefetch where the
question matches the model's training and there's time to score.
The HybridWeights struct is left untouched in this commit.
## No regression
- paged_blocks=false → BlockSemanticIndex stays empty; the new
restore_semantic_paged early-bails with a debug log.
- Existing record_chunk_fingerprint / restore_semantic /
find_similar_chunks API unchanged.
- llama + llama-server build clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wires the server integration into the BlockSemanticIndex API added in b5f029b. Behavior unchanged when --kv-tier-paged-blocks is off. ## What lands mt_record_fingerprints_for_range helper — when paged is on, walks the [p0, p1) range in block_size strides (rounded down to a block-aligned floor), embeds each block's tokens, and records one fingerprint per logical block via record_paged_block_fingerprint. When paged is off, keeps the legacy chunk-level path (one embedding for the whole range via record_chunk_fingerprint). Three call sites switch to the helper: - proactive backup (server-context.cpp:~1560) - context-shift backup (~2425) - query-time prefetch (~2748) — dispatches restore_semantic_paged vs restore_semantic based on the same flag ## Why per-block The chunk-level path emits one fingerprint covering the whole evicted range and stores it under SemanticIndex (position-list keyed). At query time we'd score the new query against each chunk's single embedding, which loses block-level resolution — a query relevant to block 47 of a 32-block chunk wouldn't be distinguishable from a query relevant to block 12. The 16-token block size is small enough that one BGE embedding per block is reasonable both in storage (~1.5 KiB per block × 8k blocks/seq = 12 MiB/seq at fp32) and in CPU cost (~ms per embed; happens off the decode path). ## No regression - params.kv_tier_paged_blocks=false → helper takes the legacy chunk branch; identical behavior to the pre-MAD-122 wiring. - params.kv_semantic_index empty → all three call sites skip as before. - llama-server builds clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ged (MAD-125 follow-up)
Foundation for MAD-129 (the resolved MAD-122/125 design): moves the
BlockSemanticIndex member + record_paged_block_fingerprint /
restore_semantic_paged API from mt::llama_memory_tiered (where the
first cut wired it — wrong layer for hybrid+paged) onto
llama_kv_cache_paged, where the active paged tier actually lives.
## What lands
- llama_kv_cache_paged.{h,cpp}:
- mt::BlockSemanticIndex paged_semantic_ member
- record_paged_block_fingerprint(seq_id, lblock, embedding, tier)
- restore_semantic_paged(seq_id, query_embedding, top_k, threshold)
— scores fingerprints for the seq, restores top-K from warm/cold
to hot via existing restore_block_from_warm/cold, logs hit-rate
- n_paged_fingerprints() diagnostic accessor
- Lifecycle: clear() drops all fingerprints; whole-seq seq_rm wipe
drops the seq's fingerprints
- tools/server/server-context.cpp:
- mt_get_paged_cache(llama_memory_i*) helper that peels through
the wrapper chain (mt::tiered → llama_memory_hybrid →
get_mem_attn_paged) to reach the paged cache
- mt_record_fingerprints_for_range now accepts a paged_cache
pointer; routes per-block fingerprints to it when paged is on,
falls back to the legacy chunk-level path on the wrapper otherwise
- Three call sites (proactive backup, context-shift backup,
query-time restore) updated to dispatch on paged_cache presence
## Status
This commit makes the API live on the right class but does NOT yet
fire end-to-end for the army-goal config — the server-side WRITE
trigger still sits inside the proactive-backup gate, which doesn't
fire for hybrid+paged (cap arithmetic uses full ctx, threshold never
crossed at typical workloads). MAD-129 relocates the write trigger
to prefill submission time so fingerprints actually get written.
The READ trigger (server-context.cpp:~2748) IS correctly placed and
will work once writes land.
## No regression
- Without --kv-tier-paged-blocks: legacy chunk-level path on the
wrapper unchanged.
- Without --kv-tier-semantic-index: all semantic paths dormant.
- llama + llama-server build clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-127)
Per Epic MAD-126 decision A1: hybrid+paged is the only target. Pure-
attention via legacy paths is near-zero priority. The army-goal config
runs entirely through llama_kv_cache_paged (MAD-117/120/121/124); the
wrapper (mt::llama_memory_tiered) survives only as a thin shim for
bge-small embedding ownership and recurrent-state backup.
Net diff: +28 / -2865 lines. ~2,800 lines of dead code gone.
## Files deleted
- src/llama-kv-cache-tiered.{h,cpp} (1473 lines) — legacy non-paged
tier with its own SSD format (KVTC magic), warm slots, semantic
fingerprints. Used only by server-tiered-cache.
- src/llama-eviction-policy.h (235 lines) — legacy enum + token
metadata store. Only referenced by llama_kv_cache_tiered.
- tools/server/server-tiered-cache.{h,cpp} (457 lines) — server's
per-slot manager wrapping llama_kv_cache_tiered. Server's fallback
dispatch path; dead for hybrid+paged.
## mt::llama_memory_tiered scaffolding stripped
The wrapper had a parallel paged-blocks implementation that was
explicitly "no live wiring yet, Phase 2b will start using them" per
its own boot log. For hybrid+paged the active paged tier is on
llama_kv_cache_paged itself, not the wrapper. Removed:
- BlockPool paged_pool_ + BlockTable paged_table_ members
- BlockSemanticIndex paged_semantic_ member (lives on the paged cache
per the MAD-125 follow-up commit e16916d)
- paged_warm_buf_, paged_layer_off_, paged_layer_v_off_,
paged_block_bytes_, paged_warm_initialized_
- record_paged_block_fingerprint, restore_semantic_paged public methods
(live on the paged cache)
- paged_backup_seq_rm_range, paged_restore_from_warm, paged_has_warm,
ensure_paged_warm_staging private methods
- if(cfg_.paged_blocks) branches in backup_seq_rm_range, has_warm,
restore_from_warm, seq_rm whole-seq wipe
- "paged-blocks scaffolding ON" log line + ctor init
- Unused includes (mt-block-pool.h, mt-block-table.h)
What stays on the wrapper (intentional thin role per Epic A8):
- EmbeddingModel + embed_text — bge-small ownership across configs
- RecurrentStateMover + warm_recur_buf_ + backup/restore_recurrent —
hybrid models lose mem_recr.clear() state irrecoverably without this
- KvtcStore + cold_positions_ + chunk-keyed SemanticIndex — non-paged
tiered config still uses these (out of army scope, but kept working
for non-hybrid models)
## Server-context cleanup
- Removed server_tiered_cache member + init + per-slot init
- Removed the "fallback to tiered_cache->evict_from_slot" dispatch
branch (the mt:: path is now the only path)
- Removed the legacy semantic-prefetch flow (~70 lines) at the
prefill-done log site that called tiered_cache->get_prefetch_hints,
migrate_in_slot, set_current_query_embedding
- Removed the context-shift tiered_cache->evict_from_slot dispatch
- Removed #include "server-tiered-cache.h"
- Updated comments referencing the removed dispatch/legacy code
## Doc-comment refresh
Four mt:: headers had historical comments referencing deleted symbols.
Reworded to describe the current state without naming gone-files:
- mt-mover-attn.h: drop "legacy llama_kv_cache_tiered" reference
- mt-mover-recurrent.h: rephrase "vs legacy" as "mt::-only"
- mt-eviction.h: drop "Replaces legacy llama_token_metadata_store"
- mt-quant.h: rephrase "legacy llama_ssd_storage_format" as
"earlier int4 implementation"
## Verification
- llama + llama-server build clean on HIP gfx1201 (R9700)
- Smoke: --kv-tier-paged-blocks --kv-tiered 25,75,0 --cache-type-k turbo4
--kv-tier-semantic-index <bge-small> on Qwen3.6-27B-Q6_K:
- Server boots and listens
- llama_kv_cache_paged init logs unchanged (1024 blocks, turbo4 K/V,
768 warm host blocks)
- mt::llama_memory_tiered logs only "tier view" + "not tierable;
will run as passthrough" — the dead "scaffolding ON" line is gone
- No legacy llama_kv_cache_tiered or server_tiered_cache lines
- No regression on non-paged tiered config (chunk-keyed semantic +
KvtcStore paths preserved)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the GGML_UNUSED(p1) tail-truncate-only behavior in
llama_kv_cache_paged::seq_rm with a unified block-aligned partial
wipe that handles tail truncate AND middle wipe correctly.
## What changed
For partial wipes (whole-seq path unchanged):
- Walk all blocks of the seq; for each block intersecting [p0, p1):
- Wholly covered: free physical, mark table entry as kInvalidBlockId,
drop fingerprint via paged_semantic_.remove_block.
- Partially overlapped (sub-block range): leave the block; log a
clear WARN that unwiped slots will keep stale K/V.
- pos_max only updates if the wipe touches the tail (p1 > cur_max).
- For middle wipes (p1 <= cur_max), pos_max stays put — holes in the
block table represent the wiped middle.
## Why holes are safe
The mt_paged_attention_kernel already handles `kInvalidBlockTableEntry`
correctly (mt_pagedattn.cu:814, 826, 860): invalid physical → -INFINITY
contribution to the QK logit → 0 weight after softmax → no contribution
to the attention output. Freed blocks read as "no attention," not
garbage. So block-aligned middle wipes are correct without any kernel
change.
## What's still not supported
Sub-block partial wipes (where p0 or p1 lands inside a block, not on
a boundary). The unwiped slots within a kept block hold stale K/V that
the kernel WILL read. Solutions require either per-block valid-bitmask
+ kernel mask change, or a layout-aware per-slot zero primitive — both
deferred until a real consumer needs sub-block precision. The clear
WARN log is the v1 contract.
## Verification
- llama + llama-server build clean
- Smoke (Qwen3.6-27B + paged + tiered + turbo4): simple generation
("capital of France" → "Paris"). Server emits only tail-truncate
seq_rm calls (`[X, end)`) which the new code handles uniformly with
no warnings. No regression on existing flows.
## Out of scope (continued in subsequent commits on this story)
- seq_add ctx-shift fallback (Option A — server-level wipe+reprefill)
- seq_cp CoW via BlockPool refcounting
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…rt 2) Server already auto-disables ctx_shift when get_can_shift() returns false (paged returns false by design). What was missing: clear messages explaining WHY shift is off and what the operator/client should do when a slot hits the n_ctx limit on paged. ## What lands Two server-side log improvements: 1. At init, when ctx_shift gets auto-disabled because the active memory doesn't support shift: when --kv-tier-paged-blocks is set, log a paged-specific message that clarifies (a) why paged deliberately doesn't support in-place shift (block-table reindex + GPU layout reindex), (b) the recovery path (slot stops, client re-submits, the prompt cache + semantic prefetch recover the prefix on next prefill). 2. At runtime, when a slot hits the n_ctx limit because shift is disabled: emit an INF (not DBG) log specifically when paged is on, so operators see in default logs that the slot stopped due to the paged-architecture choice and the client should resubmit. This is Option A from the MAD-128 ticket: server-level fallback. Real in-place position shift on paged (Option B) would require re-indexing every block_table entry plus per-block reindex of the GPU K/V layout — deferred until a real workload demands it. ## No regression - Non-paged config: original messages + DBG-level limit log unchanged - Paged config: only adds clarifying log lines, no behavior change Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements Copy-on-Write block sharing across sequences for the paged
KV cache. seq_cp(src, dst, p0, p1) now SHARES src's blocks into dst's
table via refcount (no immediate copy). When either seq later writes
to a shared block, the cache transparently allocates a fresh block,
copies the data, and swaps the writer's table entry — preserving the
invariant that every writable cell is uniquely owned.
## What lands
### BlockPool refcounting (mt-block-pool.{h,cpp})
- New `refcount_` vector parallel to free stacks, sized to total_gpu +
total_cpu blocks. All start at 0 (free).
- `alloc_gpu()` / `alloc_cpu()`: set refcount[id] = 1 on allocation.
- `bump_ref(bid)`: increment refcount. Asserts not-free.
- `refcount(bid)`: const getter. 0=free, 1=single-owner, >1=shared.
- `free_block(bid)`: decrement refcount; only push to free stack when
refcount drops to 0. Logs warning on double-free (refcount==0).
- `reset()`: zeros refcount along with rebuilding free stacks.
### Real seq_cp (llama-kv-cache-paged.cpp)
Replaces the no-op warning with a block-aligned share:
- For each wholly-covered block in [p0, p1): bump src's physical
refcount, install in dst's table (with kInvalidBlockId padding for
any logical gaps before the block).
- Sub-block partials are NOT shared (matching seq_rm's sub-block
behavior); logged as a warning so caller can round their range.
- Wipes dst's existing range first (recursive seq_rm) so its old
blocks get properly refcount-released.
- Updates dst's pos_max if the copy extends its tail.
### CoW write trigger (cow_writes_for_ubatch)
New private method called from apply_ubatch_to_state, between
fault_in_warm_blocks_for_batch and prepare_batch_tensors:
- Collects unique (seq, lblock) pairs touched by writes this ubatch.
- For each: if the physical's refcount > 1, allocate a fresh GPU
block, copy K/V from the shared block (via host bounce buffer —
ggml has no native D2D primitive but the block is small, ~17 KiB
at turbo4), swap the writer's table entry, decrement old refcount.
- On GPU pool exhaustion during CoW: try evict_lru_to_warm and retry
alloc; if still failing, refuse the batch with clear error log
rather than corrupting the shared block.
### Eviction victim selection — skip shared blocks
Updated `evict_lru_to_warm` and the inner `evict_lru_protected` lambda
in `fault_in_warm_blocks_for_batch` to skip blocks with refcount > 1.
Reasoning: evicting a shared block doesn't free GPU space (other
sequence still holds the physical via refcount), so the eviction-
retry caller would loop forever or burn CPU pool with no GPU benefit.
Shared blocks stay GPU-resident until either (a) the other seq frees
its reference, or (b) the other seq's own write triggers CoW.
## What's NOT included
- Sub-block partial CoW (block-aligned only, mirroring seq_rm's
contract). Sub-block precision requires per-slot validity tracking
+ kernel mask change — not yet justified by any consumer.
- Eviction of shared blocks themselves. With refcount-aware victim
selection, shared blocks are pinned to GPU until refcount drops.
In a degenerate workload where every block is shared, the pool
appears full to the eviction logic. Acceptable for v1; revisit if
branching workloads cause real pressure.
- Specific seq_cp tests with actual server invocation (server has no
HTTP API for seq_cp). Real CoW exercise belongs to MAD-137 testing
with synthetic K/V via ggml CPU backend.
## Verification
- llama + llama-server build clean
- Smoke (Qwen3.6-27B + paged + tiered + turbo4 + --parallel 2):
two simple completions ("2+2" → "4", "Capital of Japan" → "Tokyo")
succeed. No refcount-error / double-free / cow-related log lines
during normal flow. Refcount machinery silently maintains
refcount=1 for all single-owner blocks (the common case).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…_paged (MAD-129) Closes the army-goal gap: BGE-small fingerprints actually get written for hybrid+paged configs, and the existing read-side dispatch (added in MAD-125 follow-up commit e16916d) finally has data to query. End-to-end semantic prefetch is live for the army-goal config. ## Resolved design (Epic A2) The original chunk-level write trigger sits inside the proactive-backup gate, which doesn't fire for hybrid+paged because: - The cap arithmetic uses full ctx (mt_tier->physical_attn_cells() returns 0 for hybrid+paged), so the eviction threshold never crosses on typical workloads. - Even if it did, paged eviction is internal to llama_kv_cache_paged (admission control + ensure_blocks_for); the server doesn't see eviction events, so a server-side "fingerprint at eviction time" write trigger is structurally wrong. Decision (Epic A2): write fingerprints AT PREFILL SUBMISSION, not at eviction time. Skip-already-fingerprinted check keeps multi-turn cost bounded — only NEW blocks (the accumulated assistant response from the prior turn) get embedded on each turn's prefill. CPU cost is off the GPU critical path; ~5ms per BGE embed is acceptable. ## What lands ### BlockSemanticIndex::has_fingerprint (mt-semantic.{h,cpp}) O(1) check whether (seq_id, lblock) already has a fingerprint. Used by the prefill write trigger to skip blocks fingerprinted on prior turns. ### llama_kv_cache_paged::has_paged_fingerprint (header only) Thin forwarder to paged_semantic_.has_fingerprint. Same shape as the existing paged_semantic accessors. ### Server-side prefill write trigger (server-context.cpp) Added at the "prompt processing done" site (just after init_sampler() in update_slots, ~line 3117). When --kv-tier-paged-blocks is on AND --kv-tier-semantic-index is set AND prompt isn't multimodal: - Get paged_cache via mt_get_paged_cache(llama_get_memory(ctx)) - Walk slot.prompt.tokens in block_size strides; for each COMPLETE block (skip the partial last block — fills on next prefill): - has_paged_fingerprint(slot.id, lb) → skip if already done - Detokenize the block's tokens; embed via mt_tier->embed_text - record_paged_block_fingerprint(slot.id, lb, emb, Tier::Hot) - Single SLT_INF summary line with new count + skipped count The skip-already-fingerprinted check makes per-turn cost O(new blocks) not O(total blocks) — for an agent that grows from 8k to 16k context across two turns, the second turn only fingerprints the new ~500 blocks. ## Verification Full smoke (Qwen3.6-27B + paged + tiered + turbo4 + bge-small): Prompt 1 (12k tokens of two-topic content — paella + quantum): prefill fingerprint sweep — 445 new, 0 already-fingerprinted, 445 total complete blocks (of 7128 total tokens, partial tail block of 8 slots not yet embedded) Prompt 2 (early-context query, no shared prefix with prompt 1): prefill fingerprint sweep — 2 new, 0 already-fingerprinted restore_semantic_paged: seq 0 — 5 hints (top_k=5, threshold=0.65), restored 0/5 (hit-rate 0%, 5 already hot, 0 unmapped, 0 failed) Model output correctly recalls early-context cooking detail (Bomba rice variety, pimentón giving the red color, not saffron). Hit-rate=0% with all 5 already-hot is correct behavior, not a bug: LRU happened to keep the semantically-matching blocks resident in the 256-block hot pool; no eviction-to-warm-then-restore was needed. Stressing the actual restore-from-warm path (workload designed so relevant blocks are guaranteed in warm) belongs to MAD-137 testing. ## What this proves end-to-end - Server reaches llama_kv_cache_paged via mt_get_paged_cache helper (the wrapper-chain peel works for hybrid+paged) - BGE-small embed model loads + warms + serves embed_text calls - BlockSemanticIndex stores per-(seq, lblock) fingerprints with O(1) has-check - Write trigger emits fingerprints at the right granularity - Read trigger fires per-prompt, scores correctly, dispatches restore - Lifecycle (whole-seq wipe → drop fingerprints) preserved from MAD-125 first cut ## What's still open - End-to-end hit-rate validation under workloads designed to force restore-from-warm — MAD-137 - Decode-time fingerprinting (only matters for queries fired DURING a generation; agent workflows hit this via the next turn's prefill re-walking the accumulated context — covered for free) - Bge-small warmup at server init (currently lazy-loads on first embed call, ~200ms hit on first prefill) — MAD-134 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…save/load (MAD-130)
Three persistence pieces, all gated on explicit triggers (Epic A5: no
implicit crash recovery; clean shutdown via /slots/save + restart with
/slots/restore is the supported path).
## Part 1: llama_kv_cache_paged::state_write / state_read
Format PAGS v1. Header carries config (block_size, n_blocks_total,
n_warm, n_cold, n_layers, k/v_bytes_per_block, type_k, type_v) for
strict validation on read — config mismatch throws clean error rather
than silently corrupting state.
Per-seq section walks each saved seq's block_table; each block carries
an inline tier tag (Hole/Hot/Warm/Cold) followed by its data:
- Hot: K/V bytes per restorable layer (round-trip via
ggml_backend_tensor_get on save, ggml_backend_tensor_set on load)
- Warm: K/V bytes from warm_k_/warm_v_ host buffers
- Cold: cold_idx (the actual bytes live in cold-tier files; require
--kv-tier-cold-resume to survive across restart)
- Hole: zero extra bytes
state_read clears each seq before populating, allocates fresh
physicals (refcount=1; CoW shared blocks become uniquely owned in
restored state).
## Part 2: BlockSemanticIndex::save_to_disk / load_from_disk
Format PSFI v1. Per (seq_id, lblock, tier, embedding_dim, floats).
Mirrors the legacy SemanticIndex format from mt-semantic.cpp:117+
but block-keyed.
Exposed on llama_kv_cache_paged via thin forwarders
save_paged_fingerprints / load_paged_fingerprints so the server can
write the sidecar alongside state_write without poking at private
members.
## Part 3: Cold-tier resume (--kv-tier-cold-resume)
New constructor param `cold_resume` plumbed through:
llama_kv_cache_paged → llama_memory_hybrid → llama_model::create_memory
→ llama_memory_params (internal) → llama_context_params (C API)
→ common_params → CLI flag --kv-tier-cold-resume / --no-kv-tier-cold-resume
When `cold_resume=true`:
- Open cold-tier files WITHOUT O_TRUNC (preserve contents)
- Try to load index sidecar at ${ssd_path}/paged/index.bin
- On success: rebuild cold_slot_for_ + cold_pool_free_ minus in-use
slots + cold_in_use_ counter
- On failure (missing/corrupt/mismatched): warn + start fresh
New public method save_cold_index_sidecar() (CIDX v1 format) writes
the in-memory index to the sidecar atomically (write tmp + rename).
Server should call on graceful shutdown or as part of /slots/save.
## What's NOT in this commit (deferred follow-ups)
- Server-side wiring of /slots/save and /slots/restore to call the new
state_write/read + sidecar save methods (separate small PR — depends
on server's existing /slots handler structure)
- Decoupled fingerprint sidecar path management (currently the server
needs to choose its own paths for fingerprints and cold-index sidecar)
- Validation that cold-tier files match the index on load (currently
trusts file existence; corrupted file content would only surface
on first read of a cold block)
- True crash recovery (Epic A5 explicitly defers — clean shutdown
via /slots/save is the supported path)
## Verification
- llama + llama-server build clean
- `./bin/llama-server --help | grep kv-tier-cold-resume` shows the new
flag registered correctly
- Boot Qwen3.6-27B + paged + tiered (25/50/25) + turbo4 + cold-path
/tmp/claude/mad130-cold + bge-small: cold tier initialized
correctly (128 blocks × 17.0 KiB × 16 attn layers = 34.0 MiB);
cold-tier files created on disk; simple generation works without
errors; no regression on existing flows.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…p (MAD-131)
Lets multiple llama-server processes share one --kv-tier-ssd-path
without colliding on cold-tier files or stomping each other's state.
Required for the army goal where a single machine hosts 2+ instances
(R9700+6900XT on the main box; 1070+RX480 on mad-lab).
## What lands
### Per-instance cold subdir + lockfile
- New CLI flag `--instance-id ID` (string). Default = process pid.
Plumbed via common_params → cparams (llama_context_params,
C-API breaking change but only for the new field) → llama_memory_params
→ llama_kv_cache_paged constructor.
- Cold-tier files now live at:
${ssd_path}/paged/instance-${INSTANCE_ID}/
L*.k.bin, L*.v.bin
.lock (flock LOCK_EX|LOCK_NB at ctor; held for cache lifetime)
.pid (informational; current pid, written at ctor)
index.bin (cold-index sidecar from MAD-130)
- Lock acquire failure throws a clear runtime_error referencing the
holder's pid, instructing the operator to use a different ID, stop
the holder, or rm the stale lock.
- Destructor releases the lock (close(fd) auto-releases the flock) and
unlinks the .pid file.
- The cold-resume sidecar path (MAD-130) updates to live inside the
per-instance subdir so a stable --instance-id deterministically
rejoins prior cold state.
### --kv-tier-cold-budget-mb cap
- New CLI flag bounding cold-pool size to N MiB (across K+V × all
attn layers). 0 = no cap (size from --kv-tiered cold percentage).
- Caps n_cold_blocks before file allocation so the on-disk footprint
matches the budget. Logged when applied.
- Use case: bound SSD wear per instance — e.g. 10000 MiB per agent on
a 600 TBW consumer NVMe lasts ~60 days under sustained eviction.
### Boot scripts (scripts/army/)
- README.md — how to use the templates + operator runbook short-form
- main.sh — R9700 (Qwen3.6-27B) + 6900XT (gpt-oss-20B)
- mad-lab.sh — 1070 (Qwen3.5-9B-Omnicoder native CUDA) + RX 480
(Qwen3.5-9B in ROCm 6.3 docker for gfx803 support)
- cleanup-cold.sh — walks ${ssd_path}/paged and removes any
instance-* dir whose lockfile is no longer held (uses flock -nx
to detect live holders; safe to run while other instances are alive)
- army.service.example — systemd unit template
These are templates — paths, models, ports, GPU IDs need
customization per machine.
## Verification (HIP gfx1201)
- Build: llama + llama-server clean
- Boot Qwen3.6-27B with --instance-id army-test-A --kv-tier-ssd-path
/tmp/claude/mad131-ssd → cold tier created at correct
per-instance path; .lock + .pid files present; pid file content
matches the running pid.
- Second start with same --instance-id: refused cleanly with
"instance army-test-A is already in use by pid <N> (lockfile ...
held). Use a different --instance-id, or stop the holder, or rm
the lockfile if it's stale."
- cleanup-cold.sh while live: kept the dir ("live holder")
- Kill server, .pid auto-unlinked by dtor; cleanup-cold.sh now
removes the orphan; paged/ dir empty afterward.
## Out of scope
- True parallel multi-instance smoke (would need two GPUs of equal
capability or a smaller model to fit both simultaneously)
- SSD wear telemetry beyond config-time logging — MAD-133
- Per-machine systemd hardening / resource limits — operator concern
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three pieces of correctness/policy hardening for the multi-seq path: single-threading contract enforcement, cold-pool exhaustion escalation, and idle-priority preempt fairness. ## Part 1: Single-threading contract (Epic A4) - Big doc block at the top of llama_kv_cache_paged.h declaring the contract loudly: "single-threaded; mutators must come from one thread; future async needs explicit synchronization, NOT casual cross-thread mutation." - New `mutable std::thread::id captured_thread_id_` member + `check_thread_id_()` helper. First mutator call captures the thread; subsequent calls assert match. DEBUG-only — release builds compile to a no-op via assert(). - Hooked into the public mutator entry points: clear, seq_rm, seq_cp, ensure_blocks_for, init_batch (via apply_ubatch_to_state which is called transitively), state_read. - Catches the trap of "small async optimization" silently corrupting the cache via concurrent BlockPool::alloc_gpu races. ## Part 2: Cold-pool exhaustion escalation - New public method `drop_oldest_cold_block()`. Walks cold_slot_for_ in seq-then-lblock order, picks first in-use entry, unmaps it, returns the cold_idx to cold_pool_free_, decrements cold_in_use_. - Wired into `evict_block_to_cold` as the last-resort escalation: hot full → evict_lru_to_warm; warm full → evict_lru_warm_to_cold; cold full → drop_oldest_cold_block + retry. If even the drop fails (cold tier disabled), refuse the eviction with a clear error log naming the consequence (caller falls back to keeping the block in warm or returning 503). - The dropped block's data is gone — the owning seq sees a hole at that block; future kernel reads contribute -INFINITY logit (= zero attention contribution per mt_pagedattn.cu:826), same mechanism middle-wipe holes use. Correctness preserved; recall degraded for that seq. - v1 picks "any in-use cold slot" not strictly "oldest" — the cold spillover is itself age-ordered (LRU warm → cold, lowest indexed cold slots are typically oldest evictions). True per-slot LRU tracking is a follow-up if real workloads show pathological miss patterns. ## Part 3: Preempt fairness (idle-priority) - New per-seq state field `last_active_us` (uint64). Updated in `apply_ubatch_to_state` for every seq with tokens in the ubatch (so currently-batched seqs always have a recent timestamp). - New public method `pick_preempt_victim(exclude_seqs)`. Walks seq_states_, picks the seq with the smallest last_active_us (oldest = most-idle = most-eligible for preemption) that isn't in `exclude_seqs` and has GPU-resident blocks to preempt. Returns -1 if no eligible victim. - Existing `evict_seq_to_warm(seq_id)` keeps its API for explicit- victim callers; `pick_preempt_victim` is the policy-aware picker. Wiring into MAD-120's admission control loop is a follow-up (current admission code calls evict_seq_to_warm with the candidate's competition list directly; integrating fairness is a scheduler-level change touching apply_ubatch_to_state's caller). ## Verification - llama + llama-server build clean - Smoke (Qwen3.6-27B + paged + tiered + turbo4 + --instance-id): simple completion succeeds; no cross-thread, drop_oldest, or pick_preempt log lines fire (expected — single-threaded normal flow). The thread-id assertion is silent in release build (NDEBUG defined); a DEBUG build would actively check. ## Out of scope - DEBUG-build assertion test (would require a -DCMAKE_BUILD_TYPE=Debug variant + a deliberate cross-thread call from a test harness) - Real LRU age tracking on cold slots (v1 picks arbitrary in-use) - Wiring pick_preempt_victim into MAD-120's admission loop (the scheduler integration is a separate concern; the cache exposes the policy primitive) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…nsion + per-batch log (MAD-133)
Three observability pieces. The army goal needs operators to SEE what
the cache is doing, not guess.
## Tier-movement counters
Added uint64 monotonic counters on llama_kv_cache_paged. Single-thread
contract (Epic A4 / MAD-132) means no atomics needed. Bumped at the
relevant tier-move sites:
- evict_h2w_total_ ← evict_block_to_warm
- evict_w2c_total_ ← evict_block_to_cold
- evict_c2drop_total_ ← drop_oldest_cold_block
- restore_w2h_total_ ← restore_block_from_warm
- restore_c2h_total_ ← restore_block_from_cold
- seq_preempt_total_ ← evict_seq_to_warm (when ≥1 block moved)
- seq_restore_total_ ← restore_seq_from_warm (when ≥1 block moved)
- semantic_attempts_total_ ← restore_semantic_paged entry
- semantic_hits_total_ ← restore_semantic_paged returned > 0
- semantic_blocks_restored_total_ ← restored count from above
Public getters expose them. New per-seq accessor n_blocks_cold_for()
counts cold-resident lblocks for a seq.
## Per-batch tier_event structured log
apply_ubatch_to_state captures pre/post counter snapshot. If ANY
non-zero delta exists at the end, emit a single structured INF line:
[mt::tier_event] instance=X evict_h2w=N evict_w2c=N evict_drop=N
restore_w2h=N restore_c2h=N preempt=N pool_free_gpu=N pool_free_cpu=N
cold_in_use=N
Quiet by default — only fires when something tier-related happened
this batch. Operators can grep for [mt::tier_event] to see the cache
working.
## /metrics/tier — Prometheus endpoint extension
Existing /metrics handler in server-context.cpp now includes paged_*
counters when --kv-tier-paged-blocks is on (mirrors the weight_pager
metrics block at the same site). Reads via mt_get_paged_cache from the
live ctx; safe-ish on x86/ARM64 since we're only reading monotonic
uint64s (single-thread mutator contract guarantees no torn writes for
aligned 64-bit values on supported archs; HTTP thread reads).
Format follows the existing convention:
llamacpp:paged_evict_hot_to_warm_total <N>
llamacpp:paged_evict_warm_to_cold_total <N>
... (10 counters total + 4 gauges for capacity + fingerprint count)
## /slots — per-slot tier breakdown
Existing /slots handler now enriches each slot's JSON with a `tier`
sub-object when paged is on:
"tier": {
"blocks_hot": <count>,
"blocks_warm": <count>,
"blocks_cold": <count>,
"fingerprints": <count>
}
## Verification
- llama + llama-server build clean
- Smoke (Qwen3.6-27B + paged + tiered + turbo4 + --metrics --slots
+ --instance-id):
- GET /metrics returns parseable Prometheus output with all 10
paged_* counters + 4 paged_* gauges. Counters at 0 (no eviction
triggered) — correct behavior.
- GET /slots returns slot[0].tier = {blocks_hot, blocks_warm,
blocks_cold, fingerprints}. Live block count matches /metrics
capacity gauges + actual usage.
- Quiet [mt::tier_event] log: no fires on a clean prefill (no
eviction events). Will emit when real eviction happens.
- Simple completion works without regression.
## Out of scope
- last_eviction_at + preempt_count per-slot fields (would need extra
per-seq tracking; deferrable until real operator pain)
- Latency histograms for tier moves (currently only counters; histos
add real complexity)
- --kv-tier-log-verbose flag for unconditional per-batch log emission
(the silent-when-zero default is more useful in practice; verbose
mode is for debugging which can grep DEBUG-level messages)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…alidation (MAD-134)
Three small ergonomic improvements that reduce operator footguns
and first-prompt latency.
## Auto-default --kv-tier-paged-blocks when --kv-tiered is set
Per Epic A7. New common_params field `kv_tier_paged_blocks_explicit`
flips true when the user passes --kv-tier-paged-blocks OR
--no-kv-tier-paged-blocks. In common_context_params_to_llama:
if !explicit && tiered && !paged → set paged true with INFO log
explaining the auto-enable + how to opt out.
The army-goal config (hybrid + tiered) wants paged-on by default —
the per-machine boot scripts in scripts/army/ already pass it
explicitly, so this is for ad-hoc operator runs and future scripts
that may forget the flag.
Help text on --kv-tier-paged-blocks updated to reflect the new
auto-default behavior + drop the stale "EXPERIMENTAL" framing
(MAD-117/120/121/124 shipped; the path is production-ready for the
army-goal config).
## BGE-small warmup at server start
New `mt::llama_memory_tiered::warmup_embed_()` private method called
from the ctor when `cfg_.semantic_index` is non-empty. Synchronously
calls embed_text("warmup") which lazy-loads the bge-small model.
Logs the latency:
mt::llama_memory_tiered: bge-small warmup complete in 44ms (n_embd=384)
That ~50-200ms now lands at startup instead of on the first user
prompt's prefill path. Failures are non-fatal — lazy path still
works on next call.
## Config validation at load time
In server-context.cpp::load_model, BEFORE the model loads:
- --kv-tier-semantic-index file: stat-check; refuse if missing
or unreadable with a clear error naming the path + suggesting
either fix-the-path or omit-the-flag.
- --kv-tier-ssd-path: mkdir + write/delete a test file at
${ssd_path}/.write_test_<pid>; refuse on errno with the strerror.
Both refusals return false from load_model → server exits with
"main: exiting due to model loading error" — no crash, no late
"failed to allocate cold-tier file" surprise hours into a stress
run.
## Verification
- Build clean
- Test 1: --kv-tiered without --kv-tier-paged-blocks → log shows
"auto-enabled --kv-tier-paged-blocks" + "bge-small warmup
complete in 44ms"
- Test 2: --kv-tier-semantic-index /tmp/does-not-exist.gguf →
refused fast with "does not exist or is not readable" message
before model loaded
## Out of scope (deferred)
- --kv-tier-auto-size (auto-derive hot/warm/cold from VRAM/RAM/disk)
is real engineering work (cross-platform GPU/RAM/disk queries) and
the existing explicit-pct approach works fine for the army boot
scripts. File when real operator pain emerges. The validation
step here catches the "you misconfigured" cases that auto-size
would also help with.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…MAD-135)
Cuts per-block cold-write IO by ~4× for F16 KV caches. Q8_0 / turbo4
caches are already byte-quantized at the cache layer and stay raw on
cold-write (correct + smaller than int4-on-quant would be).
## What lands
### mt-quant: per-block int4 with explicit scale
New `quantize_block_int4_with_scale(src, n, *scale_out, dst)` and
inverse. Algorithm:
1. Compute scale = max(|x[i]|) over the block
2. For each x[i]: normalized = x[i] / scale; encode int4
Output layout: float scale followed by ceil(n/2) packed bytes.
Per-block scale is needed because F16 KV values aren't bounded to
[-1, +1] like the existing int4 helpers assume. Real attention K/V
can have magnitudes well outside that range; per-block scale recovers
the dynamic range without an external assumption.
### llama_kv_cache_paged: F16 cold compression dispatch
In evict_block_to_cold + restore_block_from_cold, dispatch on
type_k_/type_v_:
- F16: cast block → F32, quantize with scale, write [scale,
packed_int4] to cold slot. Symmetric on read.
- else (Q8_0, turbo4, F32 currently): write raw bytes (existing).
Cold file slot size unchanged (sized for raw layout) — compressed
payload uses only ~1/4 of the slot. No file shrink in v1; the IO
benefit is what matters (each pwrite/pread is 4× smaller for F16).
### tests/test-mt-quant.cpp + CMakeLists registration
New round-trip test covering:
- int4 (no scale) on inputs in [-1, +1]: max_abs_err = 0.071
- int8 (no scale) on Gaussian σ=0.3 clamped: max_abs_err = 0.004
- int4-with-scale on Gaussian σ=2.0 (real K/V magnitude):
max_abs_err = 0.55 (= scale * 0.07), cosine_sim = 0.989
- All-zero block: round-trip exact
- Single-value block: scale auto-fits, round-trip exact
The cosine_sim=0.989 matches the SNR estimate for int4 with per-block
scale on Gaussian data: SNR ≈ 29 → cs ≈ √(SNR/(SNR+1)) ≈ 0.983.
## Verification
- Build clean (llama + llama-server + test-mt-quant)
- test-mt-quant: ALL PASS
- Boot smoke (Qwen3.6-27B + paged + tiered 25/25/50 + cold +
default F16 KV): cold tier initialized correctly with type_k=f16
(compression path active); ctor + lockfile + per-instance subdir
all wired correctly through the dispatch.
## Out of scope
- F32 cache compression (rare in practice; F16 is the common cold-
benefits-from-int4 case)
- Cold-file shrink (slot size stays at raw_block_bytes; compressed
payload uses partial slot, wasted disk is acceptable for the IO
benefit)
- Int8 fallback policy when int4 quality is insufficient (could add
later if real-model attention divergence justifies)
- Real-model attention correlation test (Qwen3.6-27B end-to-end with
cold spill + restore vs no-cold baseline) → MAD-137 testing scope
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…6 partial) R9700 (gfx1201) + 6900XT (gfx1030) both verified end-to-end with paged + tiered + turbo4 + bge-small. Build was previously gfx1201-only, which caused a silent process exit on the 6900XT — HIP runtime emitted "No compatible code objects found for: gfx1030" before the seq_rm probe in common_context_can_seq_rm() could finish, and the kernel launch died with no llama.cpp-side log line. Verification: - R9700 + Qwen3.6-27B-Q6_K (head_dim=256, turbo4): prefill 29.0 tok/s, decode 22.7 tok/s, 8192 ctx - 6900XT + Qwen3.5-9B-TQ3_1S (head_dim=256, turbo4): prefill 9.6 tok/s, decode 28.5 tok/s, 8192 ctx - /metrics on both shows the paged_* counters and gauges populated; paged_semantic_attempts_total increments on prompt processing as expected from MAD-129's prefill fingerprint write. Build change is the cmake reconfigure (-DAMDGPU_TARGETS="gfx1201;gfx1030"). Source change is cosmetic — two header comments referenced the never-implemented "/metrics/tier" route and have been corrected to "/metrics" (paged_* keys), which is where the tier metrics actually live. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds standalone unit tests for the four foundation classes that back
the paged + tiered cache:
- test-mt-block-pool (12 cases: alloc/free, refcount/CoW,
watermark gating, double-free, kInvalidBlockId
safety, reset, multi-pool independence)
- test-mt-block-table (11 cases: append/get_physical, swap_block,
non-contiguous holes via swap to kInvalidBlockId,
clear_seq, per-seq isolation, out-of-range
accessors, reset, configurable block_size)
- test-mt-block-semantic-index
(15 cases: add/has/size, overwrite at same key,
per-seq scoping, descending-order scoring,
threshold + top_k, update_tier, remove_block,
remove_seq, clear, PSFI v1 save/load round-trip,
missing-file + bad-magic load safety)
- test-mt-tiered-thin (10 cases for chunk-level SemanticIndex:
ordering, threshold, top_k, FIFO eviction at
kMaxFingerprints, MTFI v1 save/load round-trip)
Style matches existing test-mt-quant.cpp: bare main(), <cassert> with
NDEBUG undef, printf for human-readable progress. Each test compiles the
relevant src/memory-tier/*.cpp directly into the test binary because the
classes aren't exported through the public llama API; src/ is on the test
include path so internal headers (llama-impl.h) resolve.
Caught one minor quirk: BlockPool's watermark math uses
ceil(n * (double)watermark) which over-reserves by 1 when the fraction
isn't exactly representable in float (e.g. 0.2f → 0.2000…0004 → ceil
gives 3 instead of 2 for n=10). Test uses 0.5f to stay deterministic;
behavior is conservative-correct (operator always gets at-least the
asked reserve) so no source change.
ctest -R test-mt-: 5/5 passed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the second half of MAD-137:
- tests/test-paged-lifecycle.cpp (7 scenarios: alloc + hot→warm
spill, whole-seq seq_rm, warm→cold
spill, partial seq_rm, seq_cp
no-crash, state_write/state_read
round-trip, cold-resume ctor)
- tests/test-paged-semantic.cpp (5 scenarios: record + has + size,
restore_semantic_paged attempts
counter, whole-seq seq_rm drops
fingerprints, save/load PSFI v1
round-trip, restore edge cases)
- tests/stress/stress-paged-multi-seq.py
(HTTP driver: launches llama-server,
drives N concurrent agents through
mixed-locality prompts, scrapes
/metrics, asserts on tier counters
+ decode-rate floor)
- scripts/test/run-army-matrix.sh (per-device runner: ssh / docker /
local execution wrapper, runs
ctest + stress per device, emits
tests/results/army-matrix-*.json)
- .github/workflows/army-test.yml (CI: hosted CPU job for unit +
no-model integration smoke; nightly
cron triggers self-hosted matrix
jobs labeled army-r9700 etc.;
summary aggregator)
Both integration tests follow the existing get_model_or_exit convention
— they self-skip when LLAMACPP_TEST_MODELFILE is unset so CI without a
model checkpoint stays green. Pass-locally verified using
bge-small-en-v1.5-q8_0.gguf as the hparams source (12 attn layers,
head_dim=32, n_kv_heads=12). All scenarios exercise structural state
that doesn't require batch execution; pos_max-driven semantics
(seq_pos_max, full seq_cp range copies) are out of scope for this
integration tier and are covered by the stress test instead.
The stress driver discovered MAD-141 — a server-side deadlock in the
MAD-120 prefill admission loop that fires whenever the hot pool fills.
Driver correctly detects + reports the failure (HTTP errors plus
unadvanced tier counters); once MAD-141 lands the same invocation
should reach the green path.
gitignore: tests/.gitignore globs everything except *.* by default;
added \!stress/ so the new test dir is tracked. Root .gitignore now
ignores tests/results/ (the matrix runner's report artifacts).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…MAD-141)
The MAD-120 paged-attn admission gate could deadlock update_slots() when
a slot's prompt was too large to fit alongside the live workload AND the
slot itself had no GPU-resident blocks to evict (typical case: a
brand-new slot where the very first prefill batch is oversized). The
loop:
if (\!can_admit(slot)) {
n_evicted = evict_seq(slot); // returns 0 — nothing to give back
// logs "MAD-120 preempt (prefill): evicted 0 block(s)"
continue; // loop body produces no batch tokens
}
…repeated until the upstream safety guard at server-context.cpp's
"n_empty_consecutive > 3" hit, fatally aborting the server.
Discovered while building the MAD-137 stress driver: a single agent with
a 28695-token prompt (way over the 8192-token hot budget) crashed the
server within ~10s. Symptom in the wild = process SIGABRT with no
graceful client error.
Fix:
- Track per-slot consecutive-no-progress count
(paged_preempt_no_progress_count). Reset on slot.reset() and on any
iteration where the slot IS admitted.
- After kPagedPreemptDeadlockThreshold (=4) consecutive iterations of
preempt-with-evict-zero, send_error(503) + slot.release(). Client gets
a clean explanatory message ("paged KV admission could not fit a N-
token request alongside the active workload after K retries…") instead
of a dropped connection.
- Reset n_empty_consecutive=0 alongside slot.release() so the upstream
safety abort doesn't trip on the same iteration just because the
release didn't add tokens to the batch.
Verified with the MAD-137 stress driver against R9700 + Qwen3.6-27B-Q6_K:
- Before: 120K HTTP errors / GGML_ABORT in ~10s.
- After: oversized request → 1 HTTP 500 with the explanatory body;
subsequent normal requests process cleanly; server stays up
for the full 129s+ test duration; decode rate 8.98 tok/s.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Self-hosted runners were intentionally not wired up — they expose the dev boxes to inbound CI traffic that we don't want to accept. Removed the army-matrix and army-summary jobs and the schedule trigger; CI now covers only the hosted CPU compile + ctest smoke. Real-hardware testing on the army GPUs runs manually via scripts/test/run-army-matrix.sh on the dev boxes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…AD-138) Three audience-specific docs under docs/memory-tier/: - USER-GUIDE.md (242 lines): when-to-use, quick start (army-goal config in one block), full --kv-tier-* flag reference with examples, per-VRAM-class sizing guide, /metrics health-counter glossary, troubleshooting section. - OPERATOR-RUNBOOK.md (358 lines): topology table for the four-GPU army, cold/warm boot procedures, per-instance and fleet-wide health checks, alarm playbook (cold occupancy, drop counter, semantic hit-rate, eviction rate, MAD-141, lockfile errors), manual interventions (clean restart, cold-tier wipe, fingerprint reset), disaster recovery (lost SSD, disk full, GPU OOM, post-driver-update), per-device caveats from MAD-136, and routine maintenance cadence. - ARCHITECTURE.md (420 lines): three-tier model with movement table, paged-attention block model, A1 rationale (hybrid+paged primary), class hierarchy + runtime stack diagram, single-threading contract, persistence model, multi-instance model, semantic prefetch model (write + read paths + why prefill-time + why prefetch-only), kernel dispatch table, ASCII eviction state machine, file-by-file source map, Jira refs. Plus ten ADRs under docs/memory-tier/adr/ extracted from MAD-126's "Architecture decisions" section (A1-A10), each with Context/Decision/ Consequences/References. Index at adr/README.md. README.md gains one line under "Other documentation" pointing at the memory-tier docs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
github-actions
Bot
added
documentation
kmbandy
added a commit
that referenced
this pull request
Aug 10, 2026
The draft context's ISWA cache was built with filter_authoritative defaulting to false, so the cache ended up with ZERO layers and threw: E llama_init_from_model: failed to initialize the context: unordered_map::at deepseek4.cpp sets n_layer_kv_from_start = n_layer_all - n_layer_nextn = 43, so hparams.has_kv(il) is false for il 43,44,45 -- exactly the layers filter_mtp selects. Without the authoritative flag the cache intersects the filter with has_kv(), discards all three, leaves map_layer_ids empty, and .at(43) throws. The sibling llama_kv_cache_dsv4 construction fifteen lines below already passes filter_authoritative=true and carries a comment describing this precise failure. This branch never got the same treatment because it had never been executed. Root-caused by running the spine under gdb with `catch throw` against live expert workers, not by reading: #2 std::unordered_map<int,int>::at #4 llama_kv_cache::cpy_k (il=43) src/llama-kv-cache.cpp:1741 #5 llm_graph_context::build_attn (il=43) src/llama-graph.cpp:3722 #6 llama_model_deepseek4::graph::build_attention_impl src/models/deepseek4.cpp:1445 #8 llama_model_dflash::graph_dsv4::graph_dsv4 (stage_base=43, n_stages=3) #12 graph_reserve #15 sched_reserve Worth recording: my reading-based hypothesis before that -- that the ISWA base sub-cache was empty because all three stages are marked SWA -- was WRONG. is_swa(43) is correctly true and the SWA classification is fine; only the has_kv intersection was at fault. Acting on that hypothesis would have "fixed" the SWA marking and broken the routing instead. NOTE: this branch is shared with LLAMA_CONTEXT_TYPE_MTP, so MTP on DeepSeek-V4 carried the identical latent bug. It had simply never been run. Codex gpt-5.6-luna then audited every filtered KV-cache construction in the tree -- 12 sites across llama_kv_cache, llama_kv_cache_iswa, llama_kv_cache_dsv4, the hybrid wrappers and the nested DSV4 raw/compressed caches -- checking each for filters that can select layers past n_layer_kv_from_start without the authoritative flag. This was the only one. GLM-DSA is correctly exempt (n_layer_kv_from_start = -1); the hybrid and shared-filter paths correctly propagate. Builds clean: build-hip llama + llama-server. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lz74kRRC7s6a3hSMLbf1Gt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes Epic MAD-126:
production-quality paged-attention + tiered KV cache + multi-seq concurrent
serving so four
llama-serverinstances (one per GPU in the four-GPU army)can run
--parallel 4 -c 524288 --kv-tier-paged-blocks --kv-tiered ...end-to-end without manual intervention or silent degradation.
18 commits / 63 files / +6678 / −2961.
What's in the branch
7bf8a5b5dd23d327ac,db75156c0,f3ec51c17seq_rmmiddle-range, ctx-shift messaging,seq_cpCoW viaBlockPoolrefcounting9299eab10(+e16916d15follow-up)llama_kv_cache_pagedwith prefill-time write triggere2f9e8f46state_write/state_readon paged + cold-tier resume + fingerprint save/load4f667abf750e1ce689de644a566/metrics, per-batch logs,/slotsextension93796b6df--kv-tier-paged-blocksdefault-on for hybrid + bge warmup + config validation6358dccc57d520e404d788f7a46,b9ad370f10d66d8aa3send_error9c16a2f461871f4f4fHardware verification (MAD-136)
Test coverage
ctest -R test-mt-(BlockPool, BlockTable, BlockSemanticIndex, mt-tiered-thin, mt-quant) — all greentests/stress/stress-paged-multi-seq.pyscripts/test/run-army-matrix.sh.github/workflows/army-test.ymlDocs (MAD-138)
docs/memory-tier/USER-GUIDE.md— flag reference, sizing, troubleshootingdocs/memory-tier/OPERATOR-RUNBOOK.md— boot, alarms, manual interventions, DR, per-device caveatsdocs/memory-tier/ARCHITECTURE.md— three-tier model, paged blocks, class hierarchy, single-thread contract, persistence, multi-instance, semantic prefetch, kernel dispatchdocs/memory-tier/adr/— ten ADRs (A1-A10) extracted from the Epic's design sectionTest plan
cmake --build build-hip --target test-mt-{quant,block-pool,block-table,block-semantic-index,tiered-thin,paged-lifecycle,paged-semantic}ctest -R 'test-mt-|test-paged-' --output-on-failure→ 7/7 green🤖 Generated with Claude Code