feat(server): kb.explain_ranking — expose why a result ranked where it did (#432)#457
feat(server): kb.explain_ranking — expose why a result ranked where it did (#432)#457moorsecopers99 wants to merge 3 commits into
Conversation
adds a read-only method that re-runs the kb.context retrieval pipeline in an instrumented mode and reports, per candidate, the lexical (fts5) rank, the semantic (embedding) rank, the rrf contribution, the rerank delta (when the reranker extras are installed), the salience factor, and the gate outcome (kept / budget-dropped / uncited / status-filtered). it reuses _retrieve's own primitives rather than duplicating the scoring math, and is viewer-scoped identically to kb.context so it can never expose an artifact the caller couldn't already retrieve. recency and frequency factors are reported null until vouchdev#317 lands; the rerank delta is populated only when vouchdev#5's reranker extras are present. registered on all four surfaces — mcp (kb_explain_ranking), jsonl (kb.explain_ranking), capabilities.METHODS, and cli (vouch explain-ranking with --format text|json) — plus tests in tests/test_explain_ranking.py covering a fused-only query, each gate drop, viewer-scoping parity, and the four-site registration. closes vouchdev#432
|
Warning Review limit reached
Next review available in: 37 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdds read-only ChangesRanking introspection
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant CLI_JSONL_MCP
participant explain_ranking
participant KBStore
participant _classify_gates
Caller->>CLI_JSONL_MCP: request ranking explanation
CLI_JSONL_MCP->>explain_ranking: forward query, filters, and viewer scope
explain_ranking->>KBStore: retrieve scoped candidates
explain_ranking->>_classify_gates: apply status, budget, and citation gates
_classify_gates-->>explain_ranking: candidate outcomes
explain_ranking-->>CLI_JSONL_MCP: structured explanation
CLI_JSONL_MCP-->>Caller: JSON or formatted results
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
docs
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/vouch/context.py (1)
176-211: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftBackend dispatch logic is duplicated from
_retrieve(lines 92–126).The
if/elif/elsecontrol flow — backend selection, hybrid fallback to substring, per-backendfilter_hitscalls — mirrors_retrievebut is maintained separately. Adding a new backend or changing fallback behavior requires updating both functions in sync, risking silent divergence.Consider extracting a shared dispatch helper that returns the raw hits plus metadata (backend name, fusion flag, sem/lex rank maps), so both
_retrieveandexplain_rankingconsume the same primitive.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/vouch/context.py` around lines 176 - 211, Extract the duplicated backend dispatch from `_retrieve` and `explain_ranking` into a shared helper that performs backend selection, hybrid substring fallback, filtering inputs, and rank-map/fusion metadata assembly. Have the helper return raw hits plus `used_backend`, `fusion`, `sem_rank`, and `lex_rank`, then update both callers to consume it so backend behavior and fallback logic remain synchronized.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_explain_ranking.py`:
- Around line 223-237: Expand test_jsonl_handler_round_trip to assert the
response id equals "r1" and verify the complete successful envelope, including
ok and result. Add a failure-case test for kb.explain_ranking using invalid or
missing parameters, asserting the response preserves the request id, has ok set
to false, and includes an error field.
---
Nitpick comments:
In `@src/vouch/context.py`:
- Around line 176-211: Extract the duplicated backend dispatch from `_retrieve`
and `explain_ranking` into a shared helper that performs backend selection,
hybrid substring fallback, filtering inputs, and rank-map/fusion metadata
assembly. Have the helper return raw hits plus `used_backend`, `fusion`,
`sem_rank`, and `lex_rank`, then update both callers to consume it so backend
behavior and fallback logic remain synchronized.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: e7b54a35-90ac-4cae-a953-9b28df815d16
📒 Files selected for processing (7)
CHANGELOG.mdsrc/vouch/capabilities.pysrc/vouch/cli.pysrc/vouch/context.pysrc/vouch/jsonl_server.pysrc/vouch/server.pytests/test_explain_ranking.py
add the request-id assertion on the success round-trip and a
failure-case test for a missing query param, asserting the
{id, ok: false, error} envelope with code missing_param — per the
coding guideline for new kb.* methods.
add docstrings to the nested retrieval helpers (_ranks, _clipped_len, _fmt), the jsonl handler, and the test helpers/cases that lacked them, clearing the 80% docstring-coverage threshold on the diff.
2 participants
what changed
adds
kb.explain_ranking, a read-only introspection method over the retrieval pipeline. per candidate it returns the lexical (fts5) rank, the semantic (embedding) rank, the rrf contribution, the rerank delta (when #5's reranker extras are installed), the salience factor, and the gate outcome —kept,budget-dropped,uncited, orstatus-filtered. it re-runs_retrieve's own primitives (search_semantic,search,rrf_fuse,filter_hits) in an instrumented mode rather than duplicating the scoring math, and classifies each candidate's gate by mirroringbuild_context_pack's status → budget → uncited pipeline. exposed on all four surfaces:vouch explain-ranking(cli,--format text|json),kb_explain_ranking(mcp),kb.explain_ranking(jsonl), and registered incapabilities.METHODS.why
_retrievehands back each hit as(kind, id, summary, score, backend)— one opaque score and a backend label. a reviewer tuning retrieval (fusion weights, the reranker in #5, the recency/frequency signals in #317) has no way to see why an artifact surfaced or got dropped: how much came from lexical vs semantic rank, what the rrf contribution was, whether a gate cut it. this makes the ranker inspectable exactly where it was a black box. the ui panel stays in the vouch-ui console per the issue — this is the backend that feeds it. closes #432.proof
captured from the shipped cli against a fresh kb — two approved claims, one archived. the fusion path is reported active, lexical rank and rrf contribution are populated, and the archived claim surfaces as
status-filtered; tightening--max-charsflips the kept claim tobudget-dropped:the json surface returns the same breakdown per candidate and carries the usual
_meta.vouch_truststamp.what might break
kb.*method.kb.explain_rankingjoins the method surface on all three transports plus the cli. adapters that enumerate methods will see it;test_capabilities(methods ↔ jsonl handlers ↔ mcp tools parity) is updated and green.kb.approve, no lifecycle change, no new audit events. it exposes only whatkb.contextalready computes internally and is viewer-scoped identically, so it can't reveal an artifact a caller couldn't already retrieve (covered by a private-claim parity test).storage.pystays pure i/o; the instrumentation lives in the retrieval layer.vep
this adds a new
kb.*method, which per the template andproposals/README.mdis a surface change that wants an accepted vep first. flagging it honestly: the issue (#432) is assigned and in milestone 2 and already specifies the surface, and this is a read-only, additive, viewer-scoped introspection method with no object-model or on-disk change. happy to write up a short vep if you'd like this gated on one before review.tests
make checkpasses locally (lint + mypy + pytest) — pytest 1092 passed / 28 skipped, mypy clean across 89 files, ruff clean. run with the[dev,web]extras to match ci.tests/test_explain_ranking.py, 12 cases: fused-only query signals, fused-rank ordering,status-filtered/budget-dropped/uncitedgates, require-citations happy path, viewer-scoping parity withkb.context, read-only invariance (no proposals, no audit growth), empty-query, and the four-site registration plus a jsonl round-trip.CHANGELOG.mdupdated under## [Unreleased].Summary by CodeRabbit
kb.explain_ranking/vouch explain-ranking) that explains per-candidate ranking components and gate outcomes.max_charsbudgeting, with JSON or human-readable output.kb_explain_ranking).