Routing Evaluation Workflow

For the public synthetic v2 fixture and execution/evidence checks, use evaluate_routing_v2.py and the v2 guide. The workflow below describes the legacy route-only evaluator for the private golden set.

scripts/evaluate_golden_qa.py evaluates route selection only. It does not import a workbook, execute DuckDB/Chroma/Python routes, call /api/query, or score answer correctness and citations.

Golden file

Legacy working set: the production-derived set remains in gitignored private/ directory and is intentionally not published. The JSON below documents the canonical schema; it is not a public benchmark dataset. Use the v2 fixture for the public workflow, or provide your own file to this older route evaluator.

Keep production-derived golden sets under the gitignored private/ directory. The canonical schema keeps source_file at the top level:

{
  "source_file": "example.xlsx",
  "profile": {
    "sheets": [
      {
        "name": "Cases",
        "rows": 100,
        "columns": ["status", "problem_description"],
        "detected": {
          "semantic": ["problem_description"]
        }
      }
    ]
  },
  "questions": [
    {
      "id": "Q001",
      "question": "How many cases do we have?",
      "expected_route": "count"
    }
  ]
}

Follow-up questions can include recent history:

{
  "id": "Q002",
  "history": [
    {"role": "user", "content": "Find cases similar to motor vibration"},
    {"role": "assistant", "content": "The closest cases are MX-1001 and MX-1003."}
  ],
  "question": "same, but only open",
  "expected_route": "hybrid"
}

The evaluator contextualizes the current question exactly as the application does before routing. It does not score the text stored in the assistant-history entry.

Run route evaluation

Run deterministic evaluation:

.venv/bin/python scripts/evaluate_golden_qa.py private/golden_qa.json --output private/golden_qa_eval.json

This uses the production router's strategy instances and order, omitting only the LLM fallback. It is deterministic and does not require provider access.

Include the production LLM fallback for ambiguous questions:

.venv/bin/python scripts/evaluate_golden_qa.py private/golden_qa.json --use-llm-router --output private/golden_qa_eval.json

This mode uses the same router instance as the application and can make provider requests. Keep questions and reports private.

Use a route-accuracy threshold in CI-like checks:

.venv/bin/python scripts/evaluate_golden_qa.py private/golden_qa.json --min-accuracy 0.9

The JSON report includes "evaluation_scope": "routing_only". The script prints summary metrics by default; --verbose adds per-question expected and actual routes, reasons, candidate chains, and contextualization state.

Interpret route results

count: broad deterministic summaries.
sql: exact filters, dates, simple aggregates, and explicit distributions.
semantic: fuzzy search over selected semantic columns.
hybrid: structured SQL filtering followed by semantic ranking.
multi: several subroutes combined into one response.
python: dataframe logic, cross-file comparisons, ratios, or missing IDs.

A mismatch can mean either a router regression or an outdated expected route. For example, an explicit group-by question normally belongs to sql, even when the grouped column resembles a status field.

Answer-quality checks are separate

Fields such as expected_answer, row_refs, and verification may be useful as manual annotations, but this script deliberately ignores them. Answer-quality testing needs a fixed synthetic workbook loaded into a test application, actual /api/query execution, and route-appropriate assertions (for example an exact count, expected row IDs, or required source fields). Do not report routing accuracy as answer accuracy.

FilesExpand file tree

evaluation.md

Latest commit

History