ADR 0003 ("Consequences") explicitly defers to v2: "run-over-run regression deltas and history; scheduled runs and alerting; a CLI/CI gate (thin client over the REST API)". This issue tracks implementing that deferred scope on top of the existing tests/evals/ CI suite (#33/#35/#37/#45/#47).
What changes
tests/evals/eval_gate.py (+ test_eval_gate.py): pure, dependency-free comparison between a run's eval-report.json and the committed corpus/retrieval_baseline.json, with a check and a ratchet CLI subcommand.
test_compose_retrieval_regression.py: the baseline diff moves from print-only to a hard gate (fails on any per-mode metric regressing beyond EVAL_GATE_TOLERANCE, default 0.02), on top of the existing absolute floor backstop.
.github/workflows/integration.yml: two new jobs —
eval-baseline-ratchet: on a green gate on main, ratchets retrieval_baseline.json up to max(current, baseline) per mode/metric (never down) and appends a line to a new corpus/retrieval_history.jsonl trend log.
eval-regression-alert: on gate failure (push-to-main or nightly), files or updates a tracking issue labeled retrieval-eval-regression so a regression isn't only visible in CI logs.
- Golden corpus expansion: new fixtures (
error-codes.txt, release-notes-v1.txt/v2.txt) and new qrels.jsonl categories (exact_id, stale_version, paraphrase, abstention), each judgment tagged with an optional category field, with per-category reporting in the eval report (visibility only, not gated).
Explicitly out of scope (by design, per ADR 0003)
- No answer/task-layer grading (groundedness, correctness) and no LLM-judge dependency — Inherent scores retrieval only; that ADR boundary stands.
- No SQL/NL2SQL retrieval route — out of scope for this eval-hardening work; a separate product decision if ever pursued.
- No permission/tenancy-boundary category in this corpus — that's owned by the
security marker suite (auth/tenancy isolation), not the ranking-quality corpus, per ADR 0003's "ownership follows the knobs."
- Still does not gate PRs (full Compose stack per PR was judged too slow/expensive); stays post-merge (push-to-main + nightly), consistent with the existing
integration.yml policy.
ADR 0003 ("Consequences") explicitly defers to v2: "run-over-run regression deltas and history; scheduled runs and alerting; a CLI/CI gate (thin client over the REST API)". This issue tracks implementing that deferred scope on top of the existing
tests/evals/CI suite (#33/#35/#37/#45/#47).What changes
tests/evals/eval_gate.py(+test_eval_gate.py): pure, dependency-free comparison between a run'seval-report.jsonand the committedcorpus/retrieval_baseline.json, with acheckand aratchetCLI subcommand.test_compose_retrieval_regression.py: the baseline diff moves from print-only to a hard gate (fails on any per-mode metric regressing beyondEVAL_GATE_TOLERANCE, default 0.02), on top of the existing absolute floor backstop..github/workflows/integration.yml: two new jobs —eval-baseline-ratchet: on a green gate onmain, ratchetsretrieval_baseline.jsonup tomax(current, baseline)per mode/metric (never down) and appends a line to a newcorpus/retrieval_history.jsonltrend log.eval-regression-alert: on gate failure (push-to-main or nightly), files or updates a tracking issue labeledretrieval-eval-regressionso a regression isn't only visible in CI logs.error-codes.txt,release-notes-v1.txt/v2.txt) and newqrels.jsonlcategories (exact_id,stale_version,paraphrase,abstention), each judgment tagged with an optionalcategoryfield, with per-category reporting in the eval report (visibility only, not gated).Explicitly out of scope (by design, per ADR 0003)
securitymarker suite (auth/tenancy isolation), not the ranking-quality corpus, per ADR 0003's "ownership follows the knobs."integration.ymlpolicy.