feat: add IR evaluation benchmark (Recall@K/Precision@K/MRR/NDCG) - #10
Merged
Conversation
added 3 commits
September 1, 2026 17:58
…h golden query set - pkg/eval: pure-Go IR metric functions (Recall@K, Precision@K, MRR, NDCG@K) with 19 unit tests + 1 property test (500 random cases, all metrics in [0,1]) - pkg/eval: benchmark runner (EvalCase/GoldenSet/EvalReport/RunBenchmark) with end-to-end test on 7 real document fixtures (convert → search → score) - pkg/eval/testdata/golden_queries.json: 20 golden cases (7 single-hit, 5 multi-hit, 3 table-content, 2 list-content, 1 title-match, 2 negative) - tools/eval.sh: one-command reproducible benchmark entrypoint - Baseline (K=5, 20 cases): Recall@5=1.0, Precision@5=0.9 (positive=1.0), MRR=0.9 (positive=1.0), NDCG@5=1.0; all 18 positive queries correct top-1 - openspec: proposal/design/spec/tasks/conformance (20/20 scenarios fully) - docs: README Evaluation section + releases.md v1.3.0 baseline scores
# Conflicts: # docs/knowledge/releases.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a reproducible information-retrieval (IR) quality benchmark to okf, quantifying search quality with canonical metrics. This closes the gap identified in the okf-vs-OpenKB analysis: okf previously proved search correctness (functional assertions) but could not quantify search quality (Recall/Precision/MRR/NDCG).
What's included
pkg/eval— pure-Go IR metric functions:RecallAtK,PrecisionAtK,MRR,NDCG(standard library only, no new dependencies)pkg/eval/testdata/golden_queries.json: 20 cases over all 7 document formats (7 single-hit, 5 multi-hit, 3 table-content, 2 list-content, 1 title-match, 2 negative)RunBenchmarkloads the golden set, runsquery.Searchagainst a real 7-document bundle (fixtures converted viapkg/convert), scores every case, and outputs aggregate + per-case reporttools/eval.sh— one-command reproducible benchmark entrypoint (consistent withtools/gauntlet.sh)Baseline scores (K=5, 20 cases, actual run)
Test coverage
pkg/eval/metrics_test.go: 19 unit tests + 1 property test (500 random cases verifying all metrics ∈ [0,1])pkg/eval/benchmark_test.go: end-to-end benchmark (real fixtures, golden set, baseline assertions) + golden set validation + empty-bundle edge caseDesign decisions
pkg/eval, not extendingpkg/query— separation of concerns (retrieval vs measurement), no circular dependency[]string(doc identifiers), not[]*Concept— pure functions, trivially testable, reusable for future retrieval backends (BM25/vector)EvalCase.ExpectedDocsto weighted pairs (documented as known limit)Known limits (documented in conformance.md)
Verification
go build ./...✅go vet ./...✅gofmt -l(tracked files) ✅ cleango test ./pkg/eval/✅ 23/23 passgo test ./pkg/convert/ ./pkg/git/ ./pkg/lint/ ./pkg/mcp/ ./pkg/okf/ ./pkg/parser/ ./pkg/query/ ./pkg/tool/✅ all passtools/eval.sh✅ exits 0, prints full report