feat: per-model benchmark evaluation, composite scores, and web dashboard

[yeerliin]

Summary

This PR enhances the benchmark system to evaluate each (agent, model) combination independently instead of mixing all models into one metric set per agent. It also adds a browser-based web dashboard embedded directly in the daemon — one service, one process, everything unified.

Problem

1. v1 mixed all models into one metric set per agent — when sdd-orchestrator ran with opus, sonnet, and gpt, all events were aggregated together. If one model degraded, another compensated statistically, masking the problem.
2. TUI rendering issues on Windows PowerShell — ANSI escaping, bar characters, layout glitches made the terminal dashboard unreliable.
3. No way to compare models quantitatively — users had to interpret raw metrics manually.

Architecture

metronous install → ONE Windows service (StartType: Automatic)
  ├── MCP ingest (dynamic port, OpenCode shims connect here)
  ├── Web dashboard (localhost:9100, always available)
  ├── Benchmark scheduler (daily at 2AM)
  └── On-demand benchmark (dashboard Refresh button)

1 OpenCode session  → 1 shim → 1 daemon → 1 dashboard
10 OpenCode sessions → 10 shims → 1 daemon → 1 dashboard
0 OpenCode sessions  → 0 shims → 1 daemon → 1 dashboard

What changed

Per-model evaluation pipeline:

• GroupEventsByModel() partitions events by model before aggregation
• Each (agent_id, model) pair gets independent metrics, thresholds evaluation, and verdict
• NormalizeModelName() prevents duplicates from inconsistent provider prefixes

Composite score (0-1):

• ComputeCompositeScore() = accuracy x 0.40 + (1-latency_norm) x 0.20 + tool_rate x 0.20 + roi_clamped x 0.20
• Weights configurable via score_weights in thresholds.json

Pairwise model comparison:

• CompareModels() pure function with per-metric deltas and auto-generated recommendation

Store extensions:

• ListAgentModels(), GetLatestRunByAgentModel(), GetVerdictTrendByModel()
• composite_score column + compound index

Unified daemon with embedded web dashboard:

• Dashboard served directly by the daemon on port 9100 — no separate metronous web process needed
• Single process handles MCP ingest + web dashboard + benchmark scheduler
• metronous web still works as standalone fallback

Web Dashboard features:

• Benchmark tab: agent overview grouped by type, detail panel (context, recommendation, decision reason, trend), model comparison with Chart.js bar charts
• Tracking tab: real-time session stream with expandable events, 5s auto-refresh
• On-demand benchmark: Refresh button executes a real benchmark run, not just data reload
• i18n: English/Spanish toggle (auto-detects browser, persists to localStorage)
• Backend translations: context, recommendation, trend direction translated client-side
• Responsive: horizontal scroll on narrow viewports
• Visibility API: refresh pauses when tab is hidden, resumes on focus

Windows service stability:

• SCM failure recovery configured: auto-restart after 5s/10s/30s on crash
• Clean shutdown: 0 processes, PID/port files cleaned, port freed
• Tested: daemon killed twice, recovered both times with new PID + dashboard

Commits (16)

#	Commit	Area
1	feat(install): Windows service via kardianos/service	Windows
2	feat(mcp): Windows MCP stdio shim	Windows
3	docs: Windows installation instructions	Docs
4	feat(benchmark): per-model evaluation	Core
5	feat(benchmark): composite score	Scoring
6	feat(benchmark): pairwise model comparison	Analysis
7	fix(benchmark): normalize model names	Bug fix
8	feat(store): per-model queries	Database
9	feat(runner): per-model pipeline	Pipeline
10	feat(tui): ranked comparison panel	TUI
11	feat(web): browser-based benchmark dashboard	Web
12	feat(web): tracking tab, i18n, detail panel	Web
13	docs: add web dashboard section to README	Docs
14	feat(web): on-demand benchmark run	Web
15	fix(install): SCM auto-recovery on failure	Windows
16	feat(daemon): embed web dashboard into daemon	Architecture

Test results (41/41 PASS)

Phase	Tests	Result
Installation from scratch	7	PASS
API performance (<400ms all endpoints)	7	PASS
Benchmark on-demand	2	PASS
Crash recovery (kill daemon 2x)	2	PASS
Clean shutdown (0 orphans)	4	PASS
Clean restart	4	PASS
Test suite (14 packages)	14	PASS
Git status (clean working tree)	1	PASS

Installation (Windows)

go install github.com/kiosvantra/metronous/cmd/metronous@latest
metronous install       # registers service + configures OpenCode + sets recovery
# Open http://localhost:9100 — dashboard is already running

Updating

metronous service stop
go install github.com/kiosvantra/metronous/cmd/metronous@latest
metronous install       # re-registers with new binary

[kiosvantra]

Please update your branch to the latest main and push again (rebase/sync), since this PR currently shows merge conflicts and some recent UI/decision-engine changes landed on main.

After updating, please verify:

• go test ./...
• The TUI tabs ([1] Tracking, [2] Benchmark Summary, [3] Benchmark Detailed) render correctly
• Intraweek manual runs triggered via F5 still work

Thanks.

[kiosvantra]

Alignment review — based on current state of `main`

Hi! Reviewed this PR against what was merged into `main` over the last few days. Here's a summary of what needs to be reconciled before this can merge cleanly.

---

What this PR adds (great work overall)

• Per-model benchmark evaluation — `GroupEventsByModel` and separate metric aggregation per `(agent, model)` pair. This is already done in `main` inside `runner.processAgentAllModels`.
• Composite score (`benchmark/score.go`) — weighted normalization of accuracy, latency, tool success, ROI into a 0–1 score stored as `BenchmarkRun.CompositeScore`.
• Model comparison (`benchmark/comparison.go`) — pairwise delta analysis between two runs.
• Web dashboard (`internal/web/`, `metronous web` command) — browser UI at `localhost:9100` as an alternative to the TUI.
• `normalize.go` — model name normalization with provider prefix inference.
• `config/score_weights.go` — configurable weights with validation.

---

Conflicts and drift with `main`

1. `BenchmarkRun` struct — field removals that broke `main`

This PR removes `RunKind`, `WindowStart`, and `WindowEnd` from the top of the struct and re-adds them at the bottom. In `main` these fields are actively written by the runner and read by the TUI (Benchmark Detailed tab cycles by `WindowStart`). The reordering is fine but make sure the SQLite column mapping in `benchmark_store.go` is consistent — the store uses positional scan order in some queries.

2. `NormalizeModelName` — duplicate, different implementations

`main` already has `store.NormalizeModelName()` (in `internal/store/`). This PR introduces `benchmark.NormalizeModelName()` in `internal/benchmark/normalize.go` with a different implementation (prefix inference table vs. the existing one). Before merging, decide on one canonical location and remove the other. Prefer keeping it in `store` since both runner and TUI already import it from there.

3. `score_weights` in `configs/thresholds.json` — conflicts with active config fields

`main` removed `max_p95_latency_ms` and `min_tool_success_rate` from the active Config tab (they were noisy/always-1.0). This PR re-introduces `tool_success_rate` as a score weight (0.20). That's a different concept (a weight, not a threshold), but `MaxLatencyP95Ms` being used for `latency_norm` calculation needs to be validated — in `main` this field is present in `DefaultThresholds` but is not an active SWITCH trigger. Confirm the normalization fallback when `MaxLatencyP95Ms == 0`.

4. `ROI formula` mismatch

In `main`, ROI = `accuracy / cost_per_session` (tool_success_rate was dropped because it is always 1.0). This PR's `score_weights` still includes `tool_success_rate: 0.20` as a separate dimension in the composite score. That's a valid design choice, but it diverges from the current verdict logic. Make sure the decision engine verdict (SWITCH/URGENT_SWITCH) and the composite score use a consistent definition — or document the intentional divergence.

5. Tab order — README is outdated in this PR

The README diff in this PR still shows the old 4-tab layout and old tab order (Tracking first). `main` already has the correct 5-tab order: `[1] Benchmark Summary → [2] Benchmark Detailed → [3] Tracking → [4] Charts → [5] Config`. When you rebase, the README will conflict — use the version from `main` as the base and layer the web dashboard section on top.

6. `docs/` — new files on `main` not present here

`main` now has `docs/ARCHITECTURE.md` and `docs/BENCHMARKS.md`. This PR still references the old `docs/architecture.md` and `docs/how-it-works.md`. After rebasing, update any cross-references.

---

Suggested rebase steps

git fetch origin
git rebase origin/main
# resolve conflicts in: README.md, internal/store/interface.go, internal/benchmark/fetcher.go
# consolidate NormalizeModelName → keep store.NormalizeModelName, delete benchmark/normalize.go
# verify benchmark_store.go column scan order matches updated BenchmarkRun struct

---

Minor notes

• `comparison.go` looks solid — no conflicts expected, just make sure `CompositeScore` is populated before `CompareModels` is called (the runner needs to persist it via the score pipeline first).
• `internal/cli/web.go` — clean addition, no conflicts.
• The EN/ES toggle in the web dashboard is a nice touch; just make sure it doesn't rely on any i18n file that isn't embedded.

Let me know if any of the above needs clarification. Happy to review again after the rebase.

[kiosvantra]

Hi! Thanks for the comprehensive work on this PR. I've reviewed it against the current state of main and wanted to provide some guidance on the next steps.

Current Situation

Since this PR was opened on Mar 31, main has evolved significantly — we've landed 84 commits with important changes:

• macOS support for the installer (not just Windows)
• Tab reordering: now [1] Benchmark Summary → [2] Benchmark Detailed → [3] Tracking → [4] Charts → [5] Config
• Cost accounting refinements and plugin hardening
• Architecture docs in docs/ARCHITECTURE.md and docs/BENCHMARKS.md
• Removal of obsolete config fields (max_p95_latency_ms, min_tool_success_rate)

In Git, main is always our source of truth. When main evolves, any open PR needs to adapt to the new context—this isn't a setback, it's just part of keeping everything aligned.

What Needs to Happen

Your PR is still valuable, but we need to rebase it cleanly against the current main. Here's the suggested workflow:

git fetch origin
git rebase origin/main
# Resolve any conflicts (favor main when in doubt)
# Run tests to ensure everything still works
go test ./...
git push -f origin feat/benchmark-v2-web-dashboard

Key Reconciliations to Watch

When you rebase, you'll want to carefully review:

1. NormalizeModelName duplication — main has one in internal/store/, your PR adds another in internal/benchmark/. We should consolidate to just the one in store/.

2. ROI + composite score formula — main removed tool_success_rate from active decision triggers (it was always 1.0), but your PR still includes it as a weight in the composite score. Let's align these definitions so the verdict engine and the score are consistent.

3. Tab references in docs — the README and any references to tab order need to match the new 5-tab layout.

4. macOS in installer logic — main now supports macOS alongside Windows. When you rebase, make sure the installer changes work cleanly with both platforms.

Next Steps

1. Rebase against main (as above)
2. Run the full test suite to catch any conflicts or regressions
3. Force-push your branch
4. Let me know if you run into any snags during the rebase—happy to help clarify the changes on main or discuss design decisions

Looking forward to getting this merged once everything aligns. Great work so far!

[kiosvantra]

Updated Status — Apr 5, 2026

Hi! Wanted to give you a comprehensive update since main has moved significantly since the last comment (Apr 4). As of today, main is 84+ commits ahead of when this PR was opened — so I want to make sure you have the full picture before rebasing.

---

What landed on main since Apr 4

The benchmark system on main has been substantially redesigned. Key changes that directly affect this PR:

Run Status system (run_status field)

Every benchmark run now has run_status = 'active' | 'superseded'. At benchmark time, the runner reads ~/.config/opencode/opencode.json to determine the currently configured model for each agent — that model gets run_status='active', all other models benchmarked in the same cycle get run_status='superseded'. This is the source of truth for the ● active marker in the TUI. Cross-cycle superseding (older runs of the same (agent, model)) is handled via MarkSupersededRuns().

Raw model field (raw_model)

A new raw_model TEXT column stores the full provider-prefixed name (e.g. opencode/claude-sonnet-4-6). The table columns display a normalized name (no prefix), while the Decision Rationale panel shows the full provider-qualified name. Your NormalizeModelName in benchmark/normalize.go should be consolidated with the existing store.NormalizeModelName() — keep the one in store/ and delete benchmark/normalize.go.

Active model from config

runner.go now has a AgentModelLookup function type injected at construction time. LoadDefaultAgentModelLookup() reads opencode.json and returns the configured model per agent. All call sites (NewRunner, daemon, TUI F5, CLI) use NewRunnerWithModelLookup. Your per-model pipeline in runner.processAgentAllModels is structurally aligned with this — just make sure to wire the lookup.

TUI overhaul

• Tab 1 is now "Benchmark History Summary" (not "Benchmark Summary")
• Tab 2 is "Benchmark Detailed" (unchanged)
• Tab order: [1] Benchmark History Summary → [2] Benchmark Detailed → [3] Tracking → [4] Charts → [5] Config
• Benchmark History Summary: cascade sort (active first per agent), shows only (agent, model) pairs active in the last 4 weekly cycles, verdict shown only for active model, legend: "Weighted historical averages (weekly + intraweek) — showing models active in the last 4 weekly cycles"
• Benchmark Detailed: cascade sort, superseded rows in orange, CHANGED verdict for superseded, Decision Rationale panel shows full provider prefix
• Verdict trend renamed to "Weekly Trend" and computed from run_kind='weekly' + run_status='active' runs only

BenchmarkRun struct additions

New fields added to interface.go: RawModel string, RunKind RunKindType, WindowStart time.Time, WindowEnd time.Time, AvgTurnMs float64, P95TurnMs float64, Status RunStatus. Make sure your rebase preserves these — they are actively used in store scan queries.

Documentation

docs/ARCHITECTURE.md, docs/BENCHMARKS.md, docs/tui-controls.md, docs/how-it-works.md, and README.md have all been updated. Use main's versions as the base and layer your web dashboard section on top.

---

Rebase guidance

git fetch origin
git rebase origin/main

Conflicts to expect and how to resolve them:

File	Expected conflict	Resolution
internal/store/interface.go	BenchmarkRun struct fields	Keep main's struct, add CompositeScore float64 from your PR
internal/benchmark/normalize.go	Duplicate NormalizeModelName	Delete your file, use store.NormalizeModelName() everywhere
internal/runner/runner.go	Per-model pipeline + active model lookup	Keep main's NewRunnerWithModelLookup wiring, integrate your GroupEventsByModel approach
internal/store/sqlite/benchmark_store.go	Column scan order	Keep main's scan order (positional), add composite_score column at the end
README.md	Tab order + dashboard section	Use main's README as base, add your web dashboard section after Tab 5
docs/	All doc files	Use main's versions, add web dashboard references where appropriate

What from your PR is NOT yet in main (still valuable):

• internal/benchmark/score.go — composite score (0-1 weighted normalization)
• internal/benchmark/comparison.go — pairwise model comparison
• internal/web/ — browser-based dashboard at localhost:9100
• internal/cli/web.go — metronous web command
• config/score_weights.go — configurable weights
• Windows service improvements

One design note on tool_success_rate: in main, this metric is excluded from both verdict triggers and ROI because it is always 1.0 in practice. If you include it as a composite score weight (0.20), it effectively becomes a dead dimension. Consider replacing it with a more meaningful signal or redistributing that weight. Happy to discuss.

---

After rebasing and resolving conflicts, run:

go test ./...

Let me know if anything is unclear or if you want to discuss any of the design decisions. Looking forward to getting this merged!

[kiosvantra]

Auditoría técnica (hoy): este PR no es mergeable en su estado actual.\n\nBlockers:\n1) Tiene conflictos de merge con (merge local Auto-merging README.md
CONFLICT (content): Merge conflict in README.md
Auto-merging cmd/metronous/commands/root.go
CONFLICT (content): Merge conflict in cmd/metronous/commands/root.go
Auto-merging configs/thresholds.json
Auto-merging internal/benchmark/fetcher.go
Auto-merging internal/cli/install_windows.go
CONFLICT (content): Merge conflict in internal/cli/install_windows.go
Auto-merging internal/cli/mcp_shim_windows.go
Auto-merging internal/config/thresholds.go
CONFLICT (content): Merge conflict in internal/config/thresholds.go
Auto-merging internal/daemon/service.go
CONFLICT (content): Merge conflict in internal/daemon/service.go
Auto-merging internal/decision/engine.go
Auto-merging internal/decision/engine_test.go
Auto-merging internal/mcp/server.go
Auto-merging internal/runner/runner.go
CONFLICT (content): Merge conflict in internal/runner/runner.go
Auto-merging internal/runner/runner_test.go
Auto-merging internal/store/interface.go
CONFLICT (content): Merge conflict in internal/store/interface.go
Auto-merging internal/store/interface_test.go
CONFLICT (content): Merge conflict in internal/store/interface_test.go
Auto-merging internal/store/sqlite/benchmark_store.go
CONFLICT (content): Merge conflict in internal/store/sqlite/benchmark_store.go
Auto-merging internal/store/sqlite/benchmark_store_test.go
Auto-merging internal/tui/benchmark_view.go
CONFLICT (content): Merge conflict in internal/tui/benchmark_view.go
Auto-merging internal/tui/benchmark_view_test.go
CONFLICT (add/add): Merge conflict in internal/tui/benchmark_view_test.go
Auto-merging internal/tui/export_test.go
Automatic merge failed; fix conflicts and then commit the result. falló) en archivos críticos como , , , , , , entre otros.\n2) Está desfasado respecto a (157 commits behind en la revisión local).\n3) Validación automática insuficiente: en checks del PR solo aparece ; no hay evidencia de pipeline de tests/build para este diff grande (benchmark/store/tui/web/daemon).\n\nAcción solicitada para destrabar:\n- Rebase/merge con y resolver conflictos.\n- Ejecutar y adjuntar resultado de en CI (o check equivalente obligatorio).\n\nCuando esté actualizado y con tests verdes, se puede re-auditar para merge.

[kiosvantra]

Corrección de formato del comentario anterior:

Dictamen: CHANGES_REQUESTED (no merge hoy).

Blockers concretos:

1. Conflictos de merge contra rama main (validado localmente con merge de prueba).
   Archivos con conflicto: README.md, cmd/metronous/commands/root.go, internal/cli/install_windows.go, internal/config/thresholds.go, internal/daemon/service.go, internal/runner/runner.go, internal/store/interface.go, internal/store/interface_test.go, internal/store/sqlite/benchmark_store.go, internal/tui/benchmark_view.go, internal/tui/benchmark_view_test.go.
2. PR desfasado respecto a main (157 commits behind en revisión local).
3. Checks insuficientes: en GitHub solo está verde conventional-commits; no hay check obligatorio de build/test para este alcance de cambios.

Para destrabar:

• Rebase/actualizar con main y resolver conflictos.
• Correr suite de pruebas completa en CI (incluyendo go test ./...).

Con eso listo, se puede re-auditar para merge.

[kiosvantra]

Se destrabó este trabajo mediante PR reemplazo en rama de origin: #21 (mergeado).\n\nMotivo: la rama head de este PR es cross-repo (yeerliin) y no quedó mergeable contra main.\nAcción aplicada: se resolvieron conflictos contra main priorizando comportamiento vigente en choques, se abrió PR #21 y se mergeó por squash.\n\nCommit final en main: 388e8e9

[kiosvantra]

Superseded by #21, which has been merged into main as commit 388e8e9. Closing this original cross-repo PR to keep history clean.