Dirty sweep: preserve uncommitted work (ht-llama.cpp)

body.txt

@@ -0,0 +1,28 @@
+Idle-cycle maintenance sprint (per `/home/me/IDLE.md` — don't hold off; keep active). One active epoch; tick items off and append newly-found work.
+
+- [x] **PR triage + squash-merge passing PRs** — ✅ merged #109, #110, #111. Janitored #95 (→ closed; re-filed as #113).
+- [x] **CI hygiene (#84)** — ✅ forced-A subset landed in #115. Lint subset (option-B) deferred (noted in #84).
+- [x] **#108 DFlash first-decode crash** — ✅ fix staged as draft PR #114; un-draft pending a cluster DFlash smoke. Bug-hunt cross-confirmed the crash is *deterministic*.
+- [ ] **HPC optimization (IDLE.md clause)** — **blocked on hardware**: the only GPU I can reach (centurion) is a production fleet node, so a multi-GB / multi-min benchmark would disrupt live serving. Needs a *dedicated* bench box. The known high-value target (GQA-packed quantized-KV FA kernel, ~4× headroom on the MQA deployment shape — see PR #102 history) is a focused CUDA-kernel project, not a safe idle-cycle self-merge.
+- [x] **Bug hunt** — ✅ DFlash multi-seq path: NOT a bug (guarded by `--parallel 1`). #19 Part-1 status:incomplete: already tested (`test_responses_truncation_emits_incomplete_status`).
+
+### Newly found this epoch
+- [x] #5 — Add Native Gemini API Compatibility to Server
+- [x] #17 — Steering hints: mid-inference context injection
+- [x] #113 — AGENTS.md webui section stale/self-contradictory post-sync (needs webui-owner intent)
+- [x] #84 lint subset — python-lint already on ubuntu-latest; 6 remaining need per-tool VERIFY-then-convert (blind conversion risks adding failing checks) — see #84 comment
+- [x] #11 — Analytical fit for large MoE models (PR #145)
+
+### Done this epoch (janitor)
+- [x] #92 — stale sync-conflict (2026-06-06) → closed (files gone post-sync; fork-sync green daily)
+- [x] #142 — rebase conflict (2026-06-22) → closed (resolved build errors after CMake migration)
+- [x] Bug hunt — fixed missing field initializer warnings for `need_download` in `server-models.cpp`
+- [x] Tech debt — fixed `int` vs `uint32_t` sign-compare warnings in `fit.cpp` and deprecated enum bitwise operations in `clip-graph.h` and tests.
+
+### Saved for later / needs input or dedicated hardware
+- HPC optimization — needs a dedicated (non-production) bench box + focused session
+- #90 stale branches — bulk deletion is surface-don't-act (risk of clobbering a live agent's WIP)
+- Fold gem pool-peer into committed `llm-pool.yaml` (cloud repo; Markus WIP)
+
+
+

docs/rfcs/001-router-coload-admit.md

@@ -0,0 +1,43 @@
+# RFC 001: Router Admit Decision for Same-Device-Pinned Models
+
+## Context
+Issue #66 identifies a router OOM condition that occurs when the router co-loads multiple large models pinned to the same device (e.g., via `tensor-split = 100,0` for `CUDA0`). The router currently checks if the aggregate available VRAM across all devices can fit the new model, failing to account for the physical footprint constraints imposed by `tensor-split` device pinning.
+
+The proposed fix requires computing per-device memory usage (`compute_free_per_device(active_models)`) and testing the candidate model's per-device allocation plan against this per-device free memory.
+
+To determine a candidate model's allocation plan, the router must invoke the fit calculation (`common_fit_params()`). However, `common_fit_params()` requires `llama_model_params` and `llama_context_params`, which are currently only bootstrapped inside the child model subprocess via `common_init_from_params()`.
+
+## Architectural Decision: Approach 1 vs. Approach 2
+
+As identified during the initial prototyping, there are two viable paths to implement the admit decision:
+
+### Approach 1: In-router CLI Mini-Bootstrap (Recommended)
+Refactor the preset parsing logic so that the router process can generate `mparams` and `cparams` from a model's preset directly.
+- **Pros:**
+  - Architecturally sound long-term solution.
+  - Zero latency overhead during admit evaluation.
+  - Robust (no string parsing or stdout serialization needed).
+- **Cons:**
+  - Requires a non-trivial refactor of `common_init_from_params` and `arg.cpp` to decouple parameter generation from the child-process boot path.
+
+### Approach 2: Out-of-Process Dry-Run Subprocess
+Shell out to the `tools/fit-params` binary, leveraging the `--print-per-device-bytes` (already partially implemented via `params.fit_params_print_plan`) for each admit decision.
+- **Pros:**
+  - Minimal refactoring required. Faster path-to-validation.
+  - Keeps the router strictly isolated from `llama_model` dependencies.
+- **Cons:**
+  - Adds ~50-200ms latency per admit due to subprocess spawning and model header reading.
+  - Brittle JSON/stdout parsing is required in the router process.
+
+## Recommendation
+We recommend **Approach 1**. 
+
+While Approach 2 is faster to write, the 50-200ms latency hit per admit decision in the router's hot path (and the brittleness of parsing standard output across process boundaries for critical state management) introduces severe reliability risks for a production fleet. The router should be resilient and fast. 
+
+We propose creating a `common_preset_to_params()` shared helper that can be called by both the child boot path and the router's admit loop. This avoids the subprocess overhead and maintains strong type safety during the admit decision.
+
+## Next Steps
+If the recommendation is accepted:
+1. Extract `common_preset_to_params()` into `common/arg.cpp` or `common/common.cpp`.
+2. Update the router's `pick_any_resident` and LRU eviction logic to respect per-device free memory calculations.
+3. Remove the dry-run CLI code from `tools/fit-params` if it's no longer necessary.

issue17.txt

@@ -0,0 +1,147 @@
+## Summary
+
+Implement **steering hints** — the ability to inject user input into an active inference pass at a given context position, creating overlapping activations that steer model output without interrupting ongoing reasoning. This can be thought of as "telepathy": the model receives a fully-formed user message mid-generation without the usual turn-taking interruption.
+
+## Background & Prior Art
+
+### What agentic CLIs call "steering"
+
+Several agentic CLI tools have introduced "steering hints" as a UX concept:
+
+- **Gemini CLI** ([issue #18782](https://github.com/google-gemini/gemini-cli/issues/18782), [PR #19307](https://github.com/google-gemini/gemini-cli/pull/19307)): "experimental in-progress steering hints" — user types while the model is thinking, text is injected into continuation turns as a hidden instruction. Explicitly prompt-level: *"inject a hidden steering instruction into continuation/follow-up turns"*.
+- **Gemini CLI** ([issue #17197](https://github.com/google-gemini/gemini-cli/issues/17197)): Proposed `/inject` command — *"the string is pushed to the conversation history as a 'User' message or 'System' hint"*.
+- **OpenAI Codex CLI**: Removed the `steer` feature flag and standardized on always-on steer path in TUI. Interactive mode (`--interactive`) allows mid-turn guidance. Again, text-level context manipulation.
+
+All of these operate at the **prompt/context level** — they queue text and inject it into the next reasoning step. None modify model activations directly.
+
+### Real activation-level steering in llama.cpp
+
+llama.cpp already supports **control vectors** ([PR #5970](https://github.com/ggml-org/llama.cpp/pull/5970), [issue #1460](https://github.com/ggml-org/llama.cpp/issues/1460)):
+
+- `llama_set_adapter_cvec()` applies per-layer vectors to activations during forward pass
+- `llama_adapter_cvec::apply_to()` adds steering tensor via `ggml_add(ctx, cur, layer_dir)`
+- Layer range `[il_start, il_end]` scopes which layers receive the vector
+- Vectors generated via PCA on positive/negative prompt pair activations (`tools/cvector-generator/`)
+
+### Research context
+
+- **Contrastive Activation Addition (CAA)**: Steering vectors computed from residual stream activation differences ([Rimsky et al., 2023](https://arxiv.org/abs/2312.06681))
+- **Steering Vector Fields** (Feb 2026): Learn differentiable scoring functions whose gradient defines steering direction per activation — context-aware rather than static
+- **SADI / FASB / CAST**: Adaptive per-input steering methods that determine intervention strength on-the-fly during inference
+- **EasySteer**: Unified framework with pluggable steering methods and pre-computed vectors for 8 domains
+- **AI Steerability 360**: IBM toolkit for systematic LLM steering ([arxiv 2603.07837](https://arxiv.org/html/2603.07837))
+
+## Proposal
+
+### Core idea
+
+During active token generation, accept user text input and:
+
+1. **Tokenize** it as a complete, properly-wrapped user message (with full chat template tags — see caveats)
+2. **Encode** these tokens into the KV cache at a target context position offset (overlapping with the current generation window)
+3. The model's attention mechanism naturally picks up these new KV entries, steering subsequent token generation
+
+This differs from both prompt injection (waits for next turn) and static control vectors (pre-computed, fixed direction). It's **dynamic, text-derived, position-targeted context steering**.
+
+### How it works mechanically
+
+llama.cpp's `llama_batch` already supports arbitrary position assignment:
+
+```c
+typedef struct llama_batch {
+    llama_token  *  token;    // token ids
+    llama_pos    *  pos;      // positions in sequence
+    llama_seq_id ** seq_id;   // sequence membership
+    // ...
+} llama_batch;
+```
+
+The KV cache stores activations keyed by `(seq_id, pos)`. By constructing a batch with the steering hint tokens at specific positions and calling `llama_decode()`, we write new KV entries that the model will attend to for all subsequent tokens.
+
+### Sequence of operations
+
+```
+1. Model is generating token at position N
+2. User types steering hint: "focus on error handling"
+3. System tokenizes hint with chat template wrapping:
+   <|im_start|>user\nfocus on error handling<|im_end|>
+4. Construct batch with hint tokens at positions [N+1, N+k]
+   (or on a parallel sequence that shares KV attention)
+5. llama_decode() the hint batch — writes to KV cache
+6. Model continues generating from position N+1 onward,
+   now attending to both its own prior context AND the hint
+```
+
+## Caveats & Open Questions
+
+### Chat template handling
+
+This is the trickiest part. User steering inputs must be **fully wrapped with proper chat template tags** so the model interprets them correctly:
+
+- Must use the model's actual chat template (Jinja or built-in)
+- The hint needs complete open+close tags (e.g., `<|im_start|>user\n...<|im_end|>` for ChatML)
+- Cannot leave tags unclosed — the model will treat unclosed tags as continuation of the current assistant turn
+- `llama_chat_apply_template()` can be used to wrap the hint as a single-message conversation with `add_ass=false`
+- Special tokens must be properly tokenized (not as text literals)
+
+### Position collision & attention masking
+
+- If hint tokens occupy positions that overlap with tokens the model is actively generating, we get activation interference
+- Options: (a) place hints at positions *ahead* of current generation, (b) use a separate sequence ID with shared KV attention, (c) use position offsets that the model hasn't reached yet
+- Need to understand RoPE position encoding implications — positions that are "future" relative to current generation may cause attention pattern issues
+
+### KV cache capacity
+
+- Each hint consumes KV cache slots proportional to its token count
+- For long-running generations, repeated hints could exhaust cache
+- May need a eviction/compaction strategy for expired hints
+
+### Causality
+
+- Standard causal attention means tokens only attend to previous positions
+- Hints placed at future positions won't be seen until generation reaches them
+- Hints placed at current/past positions create "retroactive" steering — the model hasn't seen them before but now attends to them
+- This is the "telepathy" effect: existing KV entries are unchanged, but new entries appear that influence future attention computations
+
+### Alternative: parallel sequence steering
+
+Instead of overlapping positions, use `llama_memory_seq_cp()` to branch the sequence:
+
+```
+seq 0: [system prompt] [user msg] [assistant generating...]
+seq 1: [copy of seq 0] + [steering hint tokens]
+```
+
+Then switch generation to seq 1. This avoids position collision but costs more KV memory.
+
+### Performance considerations
+
+- `llama_decode()` of the hint batch is a forward pass — compute cost proportional to hint length
+- For short hints ("focus on errors", "be more concise"), this is negligible
+- Could batch hint processing with the next token generation step
+
+## Scope
+
+### In scope
+- API for injecting steering hint text at a given context position during active generation
+- Proper chat template wrapping of hint text
+- Integration with existing KV cache and batch infrastructure
+- Server endpoint for submitting steering hints to active completions
+- Basic CLI support (type-while-generating)
+
+### Out of scope (for now)
+- Activation-level steering vector computation from hint text (future enhancement)
+- Automatic position selection heuristics
+- Multi-modal steering hints
+- Hint persistence across context shifts
+
+## References
+
+- llama.cpp control vectors: [PR #5970](https://github.com/ggml-org/llama.cpp/pull/5970), [issue #1460](https://github.com/ggml-org/llama.cpp/issues/1460)
+- Gemini CLI steering hints: [issue #18782](https://github.com/google-gemini/gemini-cli/issues/18782), [PR #19307](https://github.com/google-gemini/gemini-cli/pull/19307)
+- Gemini CLI /inject proposal: [issue #17197](https://github.com/google-gemini/gemini-cli/issues/17197)
+- Contrastive Activation Addition: [arxiv 2312.06681](https://arxiv.org/abs/2312.06681)
+- AI Steerability 360: [arxiv 2603.07837](https://arxiv.org/html/2603.07837)
+- EasySteer framework: [arxiv 2509.25175](https://arxiv.org/html/2509.25175v1)
+- Steering vectors for agents: [bassrehab/steering-vectors-agents](https://github.com/bassrehab/steering-vectors-agents)
+- llm_steer library: [Mihaiii/llm_steer](https://github.com/Mihaiii/llm_steer)