Eval-Driven Model Router

Pick the best model per task, score with multi-metric eval, track win-rates on a Postgres scoreboard. Model selection as engineering, not vibes.

Given a task (summarise, extract, answer, classify), a router agent selects the optimal model and technique, runs it, scores the output with four evaluation methods (BLEU, ROUGE-L, G-Eval, BERTScore), logs results to a Postgres scoreboard, and exposes win-rates, cost-per-task, and latency p95 via a Streamlit dashboard.

Architecture

flowchart LR
    subgraph Input
        T[Task + Context]
    end

    subgraph Router Agent
        R[Policy Router]
    end

    subgraph Model Backends
        G[GPT-4o]
        B[BERT Classifier]
    end

    subgraph Eval Harness
        BL[BLEU]
        RG[ROUGE-L]
        BS[BERTScore]
        GE[G-Eval<br/>gpt-4o-mini judge]
    end

    subgraph Storage
        PG[(Postgres<br/>Scoreboard)]
    end

    subgraph Dashboard
        ST[Streamlit<br/>Win-rates / Cost / p95]
    end

    T --> R
    R -->|summarize / extract / answer| G
    R -->|classify| B
    G --> BL & RG & BS & GE
    B --> BL & RG & BS & GE
    BL & RG & BS & GE --> PG
    PG --> ST

Loading

Run It

Prerequisites

• Python 3.10+
• Docker (for Postgres)
• OpenAI API key (for live model paths and G-Eval judge)

Quick start

# Clone and install
git clone git@github.com:ghoshp83/eval-driven-model-router.git
cd eval-driven-model-router
pip install -e ".[dev]"
python -c "import nltk; nltk.download('punkt_tab', quiet=True)"

# Start Postgres
make up

# Copy and configure environment
cp .env.example .env
# Edit .env with your OPENAI_API_KEY

# Run the eval pipeline
python scripts/run_eval.py

# Or run in mocked mode (no API key needed)
python scripts/run_eval.py --mock

# View the dashboard
streamlit run app.py

# Run tests and lint
make test
make lint

Commands

Command	Description
make up	Start Postgres via docker-compose
make down	Stop and remove volumes
make test	Run pytest
make lint	Run ruff linter
make eval	Run full eval pipeline (requires OPENAI_API_KEY)
make smoke	Run CI eval regression gate

Why This Stack

• OpenAI GPT-4o as the general-purpose model path — strongest performance across summarisation, extraction, and Q&A tasks. gpt-4o-mini as the G-Eval judge (cost-efficient for evaluation).
• BERT classifier for classification — purpose-built, orders of magnitude cheaper and faster than an LLM for label-prediction tasks.
• Four eval metrics (BLEU, ROUGE-L, BERTScore, G-Eval) — covers lexical overlap (BLEU/ROUGE), semantic similarity (BERTScore), and holistic quality judgment (G-Eval). No single metric tells the full story.
• Postgres for the scoreboard — relational aggregation (win-rates, percentiles) is a natural fit; no need for a specialised store.
• Streamlit for the dashboard — fastest path to an interactive UI for exploring eval results.

Operational Characteristics

• Routing latency: <5ms (deterministic policy; no LLM call in the routing step).
• GPT-4o path: ~1-3s per task, ~$0.005-0.02 per call depending on token count.
• BERT classifier path: <10ms, effectively zero cost (local inference).
• Eval overhead: BLEU/ROUGE <1ms each; BERTScore ~2-5s (first call loads model); G-Eval ~1-2s (LLM judge call).
• Golden set: 30-item sample covering all four task types (7 summarize, 6 extract, 7 answer, 10 classify).
• CI gate: Regression detection with configurable tolerance (default 5%). Fails the build if any metric drops.

Honest Disclaimer

Component	Status	Notes
Router agent	Real	Deterministic policy router; exercises the same interface as an LLM-based function-calling router
GPT-4o path	Real	Live OpenAI API calls for summarize/extract/answer/classify
BERT classifier	Illustrative	Rule-based keyword classifier standing in for a fine-tuned BERT checkpoint. The ModelBackend protocol is identical — swap the implementation when a fine-tuned model is available
BLEU, ROUGE-L	Real	Standard implementations via nltk and rouge-score
BERTScore	Real	Via the bert-score library with DeBERTa embeddings
G-Eval	Real	LLM-as-judge via gpt-4o-mini; scores coherence, relevance, fluency, consistency
Postgres scoreboard	Real	Stores all eval results; computes win-rates and percentile stats
Streamlit dashboard	Real	Reads from Postgres or eval artifacts; shows win-rates, score distributions
Golden set	Illustrative	30-item sample dataset for demonstration; a production deployment would use a larger, domain-specific set
Fine-tuned model path	Not included	Deferred to a future iteration; noted as future work, not a hidden gap

Future Work

• LLM-based function-calling router (replace deterministic policy with GPT-4o tool-use)
• Fine-tuned BERT/GPT model paths with real checkpoints
• Latency-aware routing (factor p95 into model selection)
• Cost optimiser (budget-constrained routing)
• Larger golden sets with domain-specific test cases
• Automated weekly policy-update loop based on scoreboard trends

License

MIT — see LICENSE.