Skip to content

feat: add IR evaluation benchmark (Recall@K/Precision@K/MRR/NDCG) - #10

Merged
superops-team merged 3 commits into
mainfrom
feat/ir-eval-benchmark
Sep 1, 2026
Merged

feat: add IR evaluation benchmark (Recall@K/Precision@K/MRR/NDCG)#10
superops-team merged 3 commits into
mainfrom
feat/ir-eval-benchmark

Conversation

@superops-team

Copy link
Copy Markdown
Owner

Summary

Adds a reproducible information-retrieval (IR) quality benchmark to okf, quantifying search quality with canonical metrics. This closes the gap identified in the okf-vs-OpenKB analysis: okf previously proved search correctness (functional assertions) but could not quantify search quality (Recall/Precision/MRR/NDCG).

What's included

  • pkg/eval — pure-Go IR metric functions: RecallAtK, PrecisionAtK, MRR, NDCG (standard library only, no new dependencies)
  • Golden query setpkg/eval/testdata/golden_queries.json: 20 cases over all 7 document formats (7 single-hit, 5 multi-hit, 3 table-content, 2 list-content, 1 title-match, 2 negative)
  • Benchmark runnerRunBenchmark loads the golden set, runs query.Search against a real 7-document bundle (fixtures converted via pkg/convert), scores every case, and outputs aggregate + per-case report
  • tools/eval.sh — one-command reproducible benchmark entrypoint (consistent with tools/gauntlet.sh)
  • openspec — proposal/design/spec/tasks/conformance (20/20 scenarios fully aligned)
  • docs — README Evaluation section + releases.md v1.3.0 baseline scores

Baseline scores (K=5, 20 cases, actual run)

Metric All cases Positive only
Recall@5 1.0000 1.0000
Precision@5 0.9000 1.0000
MRR 0.9000 1.0000
NDCG@5 1.0000 1.0000
  • 18/18 positive queries: correct top-1, all metrics = 1.0
  • 2/2 negative queries: zero results
  • All-cases Precision/MRR = 0.9 because 2 negative cases (Precision=0, MRR=0) pull down the 20-case average; positive-only is 1.0

Test coverage

  • pkg/eval/metrics_test.go: 19 unit tests + 1 property test (500 random cases verifying all metrics ∈ [0,1])
  • pkg/eval/benchmark_test.go: end-to-end benchmark (real fixtures, golden set, baseline assertions) + golden set validation + empty-bundle edge case
  • Total: 23 tests, all green

Design decisions

  • New package pkg/eval, not extending pkg/query — separation of concerns (retrieval vs measurement), no circular dependency
  • Metrics operate on []string (doc identifiers), not []*Concept — pure functions, trivially testable, reusable for future retrieval backends (BM25/vector)
  • Binary relevance NDCG — minimal implementation; graded relevance would require extending EvalCase.ExpectedDocs to weighted pairs (documented as known limit)
  • okf Search is unordered (results in bundle insertion order) — MRR=1.0 reflects correct first result for these fixtures, not a ranking algorithm; benchmark becomes more meaningful when ranked retrieval is added

Known limits (documented in conformance.md)

  1. Binary relevance only (no graded relevance)
  2. Small fixture set (7 small docs); large-corpus stress is future work
  3. No LLM-as-judge metrics (faithfulness/relevancy) — okf has no generation layer; belongs to future RAG integration

Verification

  • go build ./...
  • go vet ./...
  • gofmt -l (tracked files) ✅ clean
  • go test ./pkg/eval/ ✅ 23/23 pass
  • go test ./pkg/convert/ ./pkg/git/ ./pkg/lint/ ./pkg/mcp/ ./pkg/okf/ ./pkg/parser/ ./pkg/query/ ./pkg/tool/ ✅ all pass
  • tools/eval.sh ✅ exits 0, prints full report
  • Determinism ✅ two consecutive runs produce identical scores

Note: cmd/okf CLI smoke tests are slow (~55s each, recompiling the binary every run) and were not re-run locally due to time; they are pre-existing tests unaffected by this change (pkg/eval does not import cmd/okf). CI will verify the full suite.

superops-bot added 3 commits September 1, 2026 17:58
…h golden query set

- pkg/eval: pure-Go IR metric functions (Recall@K, Precision@K, MRR, NDCG@K)
  with 19 unit tests + 1 property test (500 random cases, all metrics in [0,1])
- pkg/eval: benchmark runner (EvalCase/GoldenSet/EvalReport/RunBenchmark)
  with end-to-end test on 7 real document fixtures (convert → search → score)
- pkg/eval/testdata/golden_queries.json: 20 golden cases (7 single-hit,
  5 multi-hit, 3 table-content, 2 list-content, 1 title-match, 2 negative)
- tools/eval.sh: one-command reproducible benchmark entrypoint
- Baseline (K=5, 20 cases): Recall@5=1.0, Precision@5=0.9 (positive=1.0),
  MRR=0.9 (positive=1.0), NDCG@5=1.0; all 18 positive queries correct top-1
- openspec: proposal/design/spec/tasks/conformance (20/20 scenarios fully)
- docs: README Evaluation section + releases.md v1.3.0 baseline scores
@superops-team
superops-team merged commit 877d5bb into main Sep 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant