feat: per-model benchmark evaluation, composite scores, and web dashboard (replace #4) - #21
Merged
Merged
Conversation
…ervice
Add install_windows.go that provides a native 'metronous install' command
on Windows:
1. Initializes ~/.metronous directory structure (via existing runInit)
2. Registers Metronous as a Windows service via kardianos/service
3. Starts the service immediately
4. Patches opencode.json (checks %APPDATA%\opencode first, then
falls back to ~/.config/opencode)
Update install_other.go build tag from '!linux' to '!linux && !windows'
so macOS and other platforms still get the stub, but Windows gets the
real implementation.
Includes 3 tests for patchOpencodeJSON: basic patching, APPDATA priority
over .config fallback, and missing file error handling.
Port the Linux MCP shim to Windows:
- Replace unix.Flock with windows.LockFileEx/UnlockFileEx for
serializing concurrent shim processes
- Replace syscall.SysProcAttr{Setsid: true} with CREATE_NEW_PROCESS_GROUP
and DETACHED_PROCESS flags for daemon detachment on Windows
- All JSON-RPC protocol handling, health checks, and tool forwarding
remain identical to the Linux implementation
Update mcp_shim_other.go build tag from '!linux' to '!linux && !windows'
so the stub only applies to macOS and other unsupported platforms.
- Add Windows installation section with PowerShell commands - Document elevated terminal requirement for service registration - Add manual service control commands (start/stop/status/uninstall) - Update architecture diagram to mention Windows SCM alongside systemd
Adds a normalized 0-1 composite score that combines accuracy (40%), latency (20%), tool success rate (20%), and ROI (20%) into a single comparable metric. Weights are configurable via thresholds.json. - internal/benchmark/score.go: ComputeCompositeScore pure function - internal/config/score_weights.go: ScoreWeights type with validation - internal/decision/engine.go: ScoreWeights accessor method - configs/thresholds.json: score_weights section added
Pure function CompareModels() produces side-by-side metric deltas between two benchmark runs with auto-generated recommendation text. Includes tie detection (delta < 0.01) and per-metric better/worse. - comparison.go: CompareModels, MetricDelta, ModelComparison types - comparison_test.go: table-driven tests for all comparison paths
OpenCode sometimes emits model names without provider prefix (e.g. "claude-opus-4-6" instead of "anthropic/claude-opus-4-6"). NormalizeModelName() infers the provider from known prefixes and GroupEventsByModel() applies normalization before grouping. Supported providers: anthropic, openai, google, mistral.
Extends BenchmarkStore with compound (agent_id, model) queries: - ListAgentModels: distinct agent+model pairs - GetLatestRunByAgentModel: most recent run per model - GetVerdictTrendByModel: verdict history per model - composite_score column added to benchmark_runs table - Compound index idx_benchmark_agent_model for query performance
Changes processAgent() to group events by model via GroupEventsByModel() before aggregation. Each (agent_id, model) pair gets independent metrics, evaluation, and composite score. Resolves the v1 limitation where all models were mixed into a single metric set per agent.
Rewrites benchmark tab to show one row per (agent, model) with: - Score column with color coding (green/yellow/red) - Model column with shortened names (opus-4-6 vs full path) - Verdict colors: KEEP=green, SWITCH=red, INSUFFICIENT=yellow - Ranked comparison panel (press 'c') with visual bars - Toggle NO DATA rows with 'h' key (hidden by default) The comparison panel shows all models for an agent ranked by composite score with proportional bars, BEST/KEEP/SWITCH/INSUFFICIENT labels, cost deltas, and a recommendation sentence from pairwise comparison.
Adds 'metronous web' command serving a dashboard at localhost:9100. Built with Go's embed FS, net/http, and a single HTML file using Tailwind CSS + Chart.js (CDN, no build step). API endpoints: - GET /api/overview: all latest runs per (agent, model) - GET /api/compare?agent=X: ranked model comparison with deltas - GET /api/trend?agent=X&model=Y: verdict history Dashboard features: - Dark mode, auto-refresh every 30s - Agent overview table grouped by type - Click agent → model ranking with bar charts - Verdict trend visualization - Only shows agents with actual benchmark data
- Tracking tab: session list with expandable events, 5s auto-refresh - i18n: EN/ES language selector with localStorage persistence - Backend string translations for context, recommendation, trend - Detail panel bugfix: compound key (agent+model) for row selection - Responsive: horizontal scroll on narrow viewports - Visibility API: pause/resume refresh when tab is hidden - Tracking API: /api/sessions and /api/sessions/events endpoints - EventStore passed alongside BenchmarkStore to web server
Documents the browser-based dashboard (metronous web) as an alternative to the TUI. Includes usage, flags, and architecture diagram update showing both dashboard options.
The Refresh button now executes a real benchmark before refreshing data, so samples update immediately instead of waiting for the daily scheduled run. Protected with mutex to prevent concurrent runs. - POST /api/benchmark/run endpoint triggers runner.RunWeekly - Runner instance created in web CLI with thresholds + decision engine - Frontend shows progress: "Running benchmark..." → "Done!" - i18n: benchmark status messages translated (EN/ES)
Configures sc failure recovery after service installation so the daemon restarts automatically if it crashes or the binary is replaced during an update (go install). Three restart attempts: 5s, 10s, 30s delays. Also documents the update flow in README: always run metronous install after updating the binary to ensure clean service state.
The web dashboard is now served directly by the daemon on port 9100. No need to run 'metronous web' separately — the browser dashboard is available as soon as the service starts. One service, one process: - MCP server (dynamic port for OpenCode shims) - Web dashboard (fixed port 9100 for browser) - Benchmark runner (on-demand from dashboard button) Architecture: metronous daemon (single process) ├── MCP ingest (dynamic port, shim→daemon) ├── Web dashboard (localhost:9100) └── Benchmark scheduler + on-demand runner The 'metronous web' command still works as a standalone fallback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reemplazo de #4 para destrabar merge desde rama en origin.
Refs: #4