Local/on-prem agent quality: conversational AKS exploration eval, model scoreboard, prompt/tool tuning, and AKS matrix

[hellices]

Goal

Make korvid's local/on-prem agent feel like a capable conversational AKS
explorer, not merely a one-shot diagnostic classifier. Publish reproducible
model scores, tune shared capability-tier prompts and deterministic tools from
measured failures, and run the matrix on the existing AKS Ollama environment.

This issue is the canonical tracker and results page. It absorbs:

• Eval-driven prompt optimization for small local models (measured on the #69 harness) #177 — eval-driven prompt/tool optimization
• AKS GPU eval environment: spot node pool + per-model serving Jobs for 14B-32B scoreboard runs #178 — AKS model-serving/eval environment

Their discussion and implementation history remain linked, but new plans,
results, and decisions belong here.

Why the current benchmark is not enough

The existing korvid.evals pack has 23 realistic Kubernetes scenarios and
measures a valuable but narrower loop:

one user question → model chooses tools → deterministic fake cluster returns
evidence → model gives a grounded answer

It covers diagnostic accuracy, evidence fetching, tool/argument validity,
iterations, tokens, latency, write attempts, and safety violations. It does not
fully measure the product experience users care about:

• start broad ("what looks wrong in this namespace?") and discover candidates;
• compare candidates and choose the next investigative step;
• preserve resource name/namespace/context across turns;
• accept corrections ("not checkout — inspect payments");
• traverse workload ownership and pivot between logs, events, and manifests;
• drive the TUI naturally while narrating;
• recognize healthy state and stop instead of continuing to invent problems.

Evaluation architecture

Tier 1 — task diagnostics (existing)

• Keep the 23 deterministic YAML scenarios.
• Live model + real AgentRuntime + real ToolExecutor; fake cluster fixtures.
• No LLM judge.
• Report diagnosis success, evidence fetch, on-target calls, malformed calls,
  iterations, tokens, latency, write attempts, and safety violations.

Tier 2 — conversational exploration journeys (next)

Add a separate journey schema rather than overloading one-turn scenarios.
Each journey contains:

• initial screen context and broad user request;
• deterministic cluster fixture shared across turns;
• scripted user follow-ups/corrections triggered after checkpoints;
• per-turn allowed goals and required evidence;
• cross-turn invariants (no stale resource/namespace reuse);
• UI expectations (navigate, open_logs, open_describe, follow-mode result);
• terminal condition (answer, ask a useful clarification, or correctly stop).

Initial journey pack:

1. namespace triage → discover two abnormal workloads → prioritize the worse one;
2. Deployment rollout → owner chain → failing pod → image error;
3. user correction mid-investigation ("payments, not checkout");
4. compare two namespaces and explain which needs attention first;
5. healthy namespace → verify and stop without inventing a fault;
6. logs look normal → pivot to events/manifest instead of repeating logs;
7. RBAC-limited read → explain the evidence gap and choose an allowed route;
8. TUI-follow journey → evidence read causes the expected screen transition.

Deterministic conversation metrics:

• journey completion rate;
• checkpoint/evidence coverage;
• correct conversational pivot rate;
• stale-context / wrong-namespace rate;
• redundant-call rate;
• useful clarification rate;
• TUI-follow coverage;
• turns, calls, tokens, latency;
• malformed calls and safety violations.

Tier 3 — live AKS contract journeys

Run a small subset against real, disposable faults on AKS to validate fixture
realism. This is not the primary scoreboard because live-cluster runs are slower,
costlier, and harder to reproduce.

Prompt/model strategy

Keep one shared prompt per capability tier (full, small) plus per-model
settings/metadata. Do not create per-model system-prompt forks unless repeated
journey results prove a model-family-specific protocol incompatibility.

Current measured lesson: deterministic compound tools and output projections
often improve small-model behavior more reliably than adding prompt text.

Canonical model scoreboard

The scoreboard compares models that users can install locally or on-prem.
The benchmark environment is not a comparison dimension: every publishable row
must be served and evaluated under the same documented AKS configuration.
Developer-laptop runs are tuning artifacts only and do not appear here.

Standard protocol for a publishable row:

• deploy/pull the local-installable model into the shared AKS Ollama/vLLM environment;
• pin serving engine/version, quantization, context length, profile, scenario SHA,
  node SKU, and warm-up procedure;
• run the 23-scenario task pack and the conversational journey pack at least 3 times;
• publish mean and variance from raw JSON.

Published task results

Model	MacBook tier	Profile	Task score	Evidence	Malformed	Safety	Task grade	Conversation
Qwen3-Coder 30B-A3B	32GB	small	59/69 (85.5%, 23 scenarios ×3)	60/69	0/124	0	A	pending
Qwen3 8B	16GB	small	20/23 (87.0%, one run)	20/23	0/27	0	provisional A	pending

The overall grade remains pending until the conversational journey pack runs.

Personal-device recommendation tiers

The primary target is an individual running Ollama locally, not a server-class
maximum. Mac unified memory and Windows VRAM/system RAM are practical guidance;
shorter context reduces memory pressure.

Device tier	Current candidate	Status
8GB Mac / CPU-iGPU Windows	Qwen3 1.7B (1.36GB)	4/6 smoke, evidence 6/6, malformed 0 — recommended 8GB candidate
8GB conditional	Qwen3 4B (2.5GB)	4/6; use short context and expect less OS headroom
16GB Mac / 8GB VRAM Windows	Qwen3 8B	5/6 smoke; current recommendation
24GB Mac / 12GB VRAM Windows	Qwen3 14B	4/6 smoke
32GB Mac / 16GB VRAM Windows	Qwen3-Coder 30B-A3B	6/6 smoke; Task A (85.5%)
64GB Mac / 24GB VRAM Windows	Qwen3 30B-A3B or Qwen3 32B	5/6 and 6/6 smoke; 32B CPU latency high

Llama 3.3 70B is retained as a server/enthusiast reference and is not a
normal personal-device recommendation.

8GB candidate pre-screen

Model	Ollama size	Smoke	Evidence	Malformed	Decision
Qwen3 0.6B	0.52GB	0/6	0/6	0/4	fail — calls did not reach evidence
Qwen3 1.7B	1.36GB	4/6	6/6	0/7	pass / recommended candidate
Llama 3.2 1B	1.32GB	0/6	2/6	1/5	fail — malformed call
Llama 3.2 3B	2.02GB	0/6	0/6	0/6	fail — wrong evidence path
Granite 3.2 2B	1.55GB	1/6	0/6	0/0	fail — no tool calls
Phi-4 Mini	2.49GB	1/6	0/6	0/0	fail — no tool calls
Qwen3 4B	2.50GB	4/6	6/6	0/6	pass, conditional on short context

AKS task-smoke pre-screen

All rows used the same retained zone-2 D32s_v5 Spot node, Ollama 0.32.5,
small profile, and six-scenario pack. PASS means task score >=4/6,
parseable native tool calls, malformed=0, and safety=0. This is a promotion
gate, not the final grade.

Model	Ollama size	MacBook tier	Smoke	Evidence	Malformed	Mean wall	Decision
Phi-4 Mini 3.8B	2.5GB	16GB	1/6	0/6	0/0	7.7s	FAIL — no tool calls
Qwen3 4B	2.5GB	16GB	4/6	6/6	0/6	75.9s	PASS
Llama 3.1 8B	4.9GB	16GB	2/6	4/6	3/11	22.4s	FAIL — malformed calls
Qwen3 8B	5.2GB	16GB	5/6	4/6	0/6	70.3s	PASS
GLM4 9B	5.5GB	16GB	2/6	0/6	0/0	25.0s	FAIL — no tool calls
Qwen3 14B	9.3GB	24GB	4/6	4/6	0/10	141.8s	PASS
Mistral Small 3.1 24B	15GB	32GB	4/6	6/6	0/6	70.3s	PASS
Devstral 24B	14GB	32GB	4/6	6/6	0/6	39.8s	PASS
Qwen3 30B-A3B	18GB	32GB	5/6	6/6	0/7	51.7s	PASS
Qwen3-Coder 30B-A3B	18GB	32GB	6/6	6/6	0/7	20.2s	PASS — smoke leader
Qwen3 32B	20GB	32GB	6/6	6/6	0/7	231.9s	PASS — slow on CPU
Llama 3.3 70B	42GB	64GB	4/6	5/6	0/6	213.3s	PASS — slow on CPU

Candidate next steps

1. Implement and run the conversational journey pack for every task-smoke PASS.
2. Complete task-pack ×3 for Qwen3 4B/8B/14B, Mistral Small, Devstral,
   Qwen3 30B, Qwen3 32B, and Llama 70B.
3. Rank models only after both task and conversation scores exist.

Existing AKS environment

• subscription: ME-MngEnvMCAP310512-inhwanhwang-3
• cluster: aks-shared-runners
• namespace/service/deployment: ollama/ollama
• model PVC: 200GiB (ollama-models), 154GiB used after the smoke matrix
• retained evaluation pool: modeleval, zone 2, Standard_D32s_v5 Spot,
  currently scaled to 0 nodes
• persisted candidates: all 12 smoke models plus LFM2.5

The Ollama deployment is restored to its original NAP scheduling and
2 CPU/6GiB requests, 3.5 CPU/10GiB limits. Scale modeleval to one when the
next matrix starts; do not recreate or delete it.

GPU constraint: Korea Central exposes T4 SKUs, but this subscription's
Standard NCASv3_T4 Family quota is 0 and an automatic 4-vCPU request returned
QuotaNotAvailableForResource.

Results storage and publication

• docs/evals/results/<model>-<profile>-<date>.json: raw run data plus endpoint,
  quantization, serving context, scenario-pack SHA, and korvid SHA.
• docs/evals/scoreboard.md: generated task and conversation tables.
• CI verifies regeneration produces no diff.
• docs/agent.md: short hardware-tier recommendations derived from scores.

Acceptance criteria

• At least 8 conversational journeys, each run at least 3 times per published
  model/profile.
• Task and conversation scores are separately visible.
• Qwen3 4B/8B/14B evaluated on AKS; additional models added as capacity permits.
• Safety violations remain 0; malformed-call rate remains below 1%.
• Every prompt/tool change carries before/after task and journey numbers.
• AKS runner lifecycle is reproducible and does not unexpectedly delete
  evaluation capacity still in use.

Out of scope

• An LLM judge as the primary grader.
• Separate system prompts for every model.
• Always-on public model serving.
• Treating live AKS fault injection as the only benchmark tier.

[hellices]

Seed data: first measured model matrix (local Ollama, Apple M4 16 GB)

Bundled 23-scenario pack, small profile, optimized prompt (#177 rounds 1–6, commit 81cbb67). Single-rep full-pack numbers carry ±2–3 scenario noise; the qwen3:4b row also has a 3-rep targeted-subset confirmation in #177.

model	size	success	evidence fetched	malformed calls	safety violations	notes
qwen3:8b	8B	19/23 (83%)	high	0	0	best of the set; fails only the two-hop chains (pvc→storageclass, rollout→rs→pod) and one negative control
qwen3:4b	4B	18/23 (78%)	18/23	0/32	0	targeted subset ×3 reps: 40% → 73% after prompt rounds
llama3.2:3b	3B	17/69 (25%, 3 reps)	24/69	2/70	0	capability floor: evidence-fetch failures, not wording — prompt changes moved 22%→25% (noise)

Recommendation shape this supports (pending more models): 16 GB tier → qwen3:8b + small profile; 8 GB tier → qwen3:4b + small profile; plain llama3.2:3b not recommended for diagnosis (fine for chat, misses evidence). 14B–32B rows need the AKS GPU environment (#178).

Raw JSON/markdown reports for all runs are preserved and match the storage format proposed here (--json output + run metadata); wiring them into docs/evals/results/ + the generator is this issue's implementation.

[hellices]

AKS scoreboard update

First AKS-backed full-pack result (live Ollama, temporary D16s_v5 Spot model node, small profile):

Model	Pack result	Malformed	Safety	Recommended
Qwen3 8B	20/23 (87%)	0	0	Yes — current default for 16Gi+ local/on-prem
Qwen3 4B	4/5 on targeted next-round set	0	0	8Gi fallback
Qwen3 14B	2/2 after cold load; first request timed out	0	0	Not on CPU; retest after GPU quota
LFM2.5	0/1 rollout	0	0	No — evidence fetched but final diagnosis unreliable

Qwen3 8B improves the earlier local result (19/23) to 20/23 with the new compound evidence tools. Raw JSON/markdown reports are retained as session artifacts for the scoreboard implementation.

[hellices]

Scoreboard semantics corrected: Mac-vs-AKS is not a product comparison. The candidates are local/on-prem-installable models, all officially evaluated under one standardized AKS protocol. Laptop results are now excluded from the canonical table; only the provisional AKS Qwen3 8B full-pack row remains, and the installed model queue is listed separately.

[hellices]

AKS smoke matrix complete for 12 locally installable candidates. Canonical body now contains the MacBook-tier table, task smoke promotion decisions, retained D32 Spot/PVC state, and the first three-repetition task grade: Qwen3-Coder 30B-A3B = 59/69 (85.5%), Task A, malformed 0, safety 0. Overall grade remains pending conversation journeys.

[hellices]

8GB personal-device tier added and tested on the same AKS protocol. Qwen3 1.7B is the clear 8GB candidate (4/6, evidence 6/6, malformed 0); Qwen3 4B is conditional with short context. Qwen3 0.6B, Llama3.2 1B/3B, Granite 2B, and Phi-4 Mini did not reliably use evidence tools. The primary table now targets actual MacBook/consumer Windows use; Llama70 is reference-only.