fix(metrics): count MCP retrieval modes instead of bucketing them as "unknown"#151
Merged
CGFixIT merged 1 commit intoJun 21, 2026
Merged
Conversation
…own'
print_metrics() read the retrieval mode only from the 'retrieval_mode'
field, which the graph audit path writes. The MCP server
(mcp_hybrid_server._handle_search) records the same datum under 'mode'.
As a result every mcp_rag_query was silently counted as 'unknown' in the
'Retrieval modes' breakdown — under-reporting hybrid/semantic/keyword
usage for all MCP traffic.
Fix: fall back to e.get('mode') before 'unknown'. Score stats already
covered both event types via the shared 'top_score' field, so only the
mode tally was affected.
Adds tests/test_metrics.py (metrics.py previously had no test): covers
load_events (missing file, malformed-line skip), the no-events message,
score-stat aggregation, and the mixed graph/MCP mode-tally regression
(fails on the old 'retrieval_mode'-only code). Registered in the CI
pytest list with --cov=metrics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NXYYNSfqvBrAgghyNmzbHs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
metrics.pybuilds its "Retrieval modes" breakdown from a single field:But the two producers of RAG audit events disagree on the key name:
graph.audit_logger_noderag_queryretrieval_modemcp_hybrid_server._handle_searchmcp_rag_querymoderag_queriesincludes both event types (event in ("rag_query", "mcp_rag_query")), so every MCP query — which has noretrieval_modekey — fell through to"unknown". The result: the mode breakdown under-reportshybrid/semantic/keywordand inflatesunknownfor any deployment that uses the MCP server.The fix
Fall back to
e.get("mode")before"unknown":Score stats were already correct (both event shapes share
top_score), so only the mode tally needed fixing.Tests
metrics.pyhad no test file. Addstests/test_metrics.py:load_events: missing file →[], and malformed JSON lines skippedprint_metrics: the no-events messagerag_query+mcp_rag_querylog assertshybrid: 2,semantic: 1,keyword: 1, and"unknown" not in out— this fails on the oldretrieval_mode-only code.Registered in the CI pytest list with
--cov=metrics.Risk
Low. One-line logic fix in a read-only reporting script (no effect on the request path), plus a new test file and two additive CI lines. New tests pass locally (5 passed).
🤖 Generated with Claude Code
https://claude.ai/code/session_01NXYYNSfqvBrAgghyNmzbHs
Generated by Claude Code