Repository navigation
Feat/rag hybrid retrieval - #11
Conversation
- 分工写清楚：本人负责业务口径、契约、评测门禁、验收与合并决定，编码实现由 Claude Code 完成 - 删除“接管人：Claude Code（唯一主开发）”一类表述，HANDOFF 改为交付报告与验证记录 - 新增 docs/COLLABORATION.md：分工表、本人负责的判断、AI 完成的部分、质量门禁、可核对项、不主张的事 - AGENTS.md 头部声明它是给编码 Agent 的工作约定，随代码版本管理 - 修 README 已知限制里“实际 MCP Server 未实现”的过时矛盾句
- 评测入口统一：backend/evals/rag.py 改读 rag_queries.json 的 42 条固定标注（36 可回答 / 6 不可回答），一次运行同时对比 weighted 与 hybrid_rrf - 新增指标 MRR、nDCG@5、按 keyword/paraphrase/cross_doc/unanswerable 分层、cross_doc 的 coverage@3，以及 no_rank_regression 门禁 - 新增 backend/app/bm25.py（内存 Okapi BM25，复用同一套中文 n-gram tokenizer），检索改为三路排名 + RRF(k=60) - score 与 confidence 分离：score 是融合排名，no_answer 只由可解释的 confidence 判定（RRF 是排名量，不能当概率用） - 只有具备排序能力的 provider 才让稠密路投票：hashing 下实测相关片段余弦为 0.0，等权投票会把正确结果平均掉 - 新增 KNOWLEDGE_MAX_PER_DOCUMENT=2，避免 top-k 被单篇 SOP 占满；新增 KNOWLEDGE_RETRIEVAL_MODE/RRF_K/BM25_K1/BM25_B - BGE-M3 接入：Q8_0 GGUF 导入 Ollama，42 条查询 hit@1 100%、MRR 1.0000；新增 scripts/reindex_knowledge.py 就地重嵌入旧库 - 评测新增阈值校准输出（max 不可回答 / min 可回答 / 建议值）：0.15 在 bge-m3 与 qwen3-embedding 下失效，分别校准到 0.42 / 0.40 - 测试与本地 .env 解耦：conftest 增加 session 级环境覆写，避免本机 embedding 配置让整套测试调用真实模型
- update retrieval description, 42-query eval results and test counts - document knowledge reindex and hashing/weighted equivalence - add knowledge search, no-answer, agent trace and mobile screenshots Co-Authored-By: Claude Code <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 185fa44e0d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Interpretable per-chunk relevance used for abstention. ``score`` is the | ||
| # ranking value (RRF in hybrid mode), which is rank-based and therefore | ||
| # cannot decide whether anything relevant exists at all. | ||
| confidence: float | None = Field(default=None, ge=0, le=1) |
There was a problem hiding this comment.
The default retrieval strategy and the meaning of score change here, and search responses now include confidence, retrieval_mode, and rankers, but docs/CONTRACT.md remains unchanged. It still specifies weighted ranking, abstention based on the highest score, and the old response fields. Consumers following that contract would incorrectly interpret the normalized RRF score as relevance confidence. Update the contract alongside the implementation; AGENTS.md explicitly requires contract changes to be synchronized with docs/CONTRACT.md, backend tests, and frontend types.
AGENTS.md reference: AGENTS.md:L25-L25
Useful? React with 👍 / 👎.
| probe = search_knowledge( | ||
| session, | ||
| KnowledgeSearchInput(query=query["query"], top_k=1), |
There was a problem hiding this comment.
When a query is rejected, the calibration probe changes top_k to 1 even though production abstention uses the maximum confidence across all selected hits. RRF order is not confidence order: the committed BGE-M3 report already shows fulfillment-04 with confidences [0.4978, 0.4941, 0.5038]. At a threshold above 0.5038, this probe would record 0.4978 instead of the actual decision statistic, making the reported separation and suggested threshold unreliable. Re-run with the original top_k and only disable abstention.
Useful? React with 👍 / 👎.
| "mode": "offline_retrieval_evaluation", | ||
| "network_calls": 0, |
There was a problem hiding this comment.
With the newly supported --embedding-provider openai option, corpus seeding, query retrieval, and calibration probes call the configured HTTP embedding endpoint, but the generated report still unconditionally declares network_calls: 0 and labels the run offline. This is already present in the committed BGE-M3 and Qwen3 reports, so the evaluation artifacts misrepresent how their results were obtained. Track the actual remote embedding calls, including the connectivity check, and distinguish remote-provider runs from the offline hashing baseline.
Useful? React with 👍 / 👎.
变更摘要
关联任务
实际验证
请只勾选实际执行过的检查，并在下方粘贴关键结果。
python -m pytest backend/tests -qnpm --prefix frontend testnpm --prefix frontend run buildpython scripts/smoke.py（涉及 HTTP/前端流程时）docker compose config --quiet（涉及 Compose 时）验证结果：
安全与边界
.env、API Key、数据库、缓存、构建产物或本机运行数据契约与文档
docs/CONTRACT.md、后端测试和前端类型docs/EVALUATION.md审查提示
请重点检查金额守恒、证据 ID 可追溯性、LLM 失败/降级可见性、审批执行幂等和敏感信息。自动审查工具的评论只是辅助，不替代人工审查。