Skip to content

Hard-gate + ratchet the retrieval-eval baseline; expand golden corpus (ADR 0003 v2) #139

Description

@sagars01

ADR 0003 ("Consequences") explicitly defers to v2: "run-over-run regression deltas and history; scheduled runs and alerting; a CLI/CI gate (thin client over the REST API)". This issue tracks implementing that deferred scope on top of the existing tests/evals/ CI suite (#33/#35/#37/#45/#47).

What changes

  • tests/evals/eval_gate.py (+ test_eval_gate.py): pure, dependency-free comparison between a run's eval-report.json and the committed corpus/retrieval_baseline.json, with a check and a ratchet CLI subcommand.
  • test_compose_retrieval_regression.py: the baseline diff moves from print-only to a hard gate (fails on any per-mode metric regressing beyond EVAL_GATE_TOLERANCE, default 0.02), on top of the existing absolute floor backstop.
  • .github/workflows/integration.yml: two new jobs —
    • eval-baseline-ratchet: on a green gate on main, ratchets retrieval_baseline.json up to max(current, baseline) per mode/metric (never down) and appends a line to a new corpus/retrieval_history.jsonl trend log.
    • eval-regression-alert: on gate failure (push-to-main or nightly), files or updates a tracking issue labeled retrieval-eval-regression so a regression isn't only visible in CI logs.
  • Golden corpus expansion: new fixtures (error-codes.txt, release-notes-v1.txt/v2.txt) and new qrels.jsonl categories (exact_id, stale_version, paraphrase, abstention), each judgment tagged with an optional category field, with per-category reporting in the eval report (visibility only, not gated).

Explicitly out of scope (by design, per ADR 0003)

  • No answer/task-layer grading (groundedness, correctness) and no LLM-judge dependency — Inherent scores retrieval only; that ADR boundary stands.
  • No SQL/NL2SQL retrieval route — out of scope for this eval-hardening work; a separate product decision if ever pursued.
  • No permission/tenancy-boundary category in this corpus — that's owned by the security marker suite (auth/tenancy isolation), not the ranking-quality corpus, per ADR 0003's "ownership follows the knobs."
  • Still does not gate PRs (full Compose stack per PR was judged too slow/expensive); stays post-merge (push-to-main + nightly), consistent with the existing integration.yml policy.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions