Repository navigation
feat: add offline RAG knowledge base and harden retrieval - #7
Conversation
- add knowledge documents/chunks, Markdown chunking, deterministic hashing embeddings and optional OpenAI-compatible embeddings - add hybrid retrieval with no-answer threshold, retrieve_knowledge workflow node and search_knowledge tool - validate LLM knowledge references and expose cited chunks in analyses and recommendations - add knowledge APIs, frontend knowledge workspace and citation panels - add offline RAG evaluation, knowledge tests and documentation Co-Authored-By: Claude Code <noreply@anthropic.com>
- Cap merged knowledge hits to the schema limit, re-rank, and revalidate before downstream use.\n- Exclude knowledge retrieval from the generic step timeout so slow embeddings degrade to knowledge_error.\n- Sanitize and redact knowledge tool results, and strengthen prompts against embedded instructions and credential leakage. Co-Authored-By: Claude Code <noreply@anthropic.com>
- Validate citations against chunk IDs actually provided to the model and prefer cited chunks when capping merged hits.\n- Degrade startup on remote embedding failure, expose seed_error, and keep database failures fatal.\n- Preserve short/long sentence order during chunking.\n- Reject malformed embedding values before persistence.\n\nCo-Authored-By: Claude Code <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Co-Authored-By: Claude Code <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cff917dcbf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| merged.sort(key=lambda hit: hit.score, reverse=True) | ||
| merged = [ | ||
| hit.model_copy(update={"rank": rank}) | ||
| for rank, hit in enumerate(merged[:KNOWLEDGE_MAX_HITS], start=1) |
There was a problem hiding this comment.
When pre-retrieval and tool searches return more than ten unique chunks, this truncation can discard chunks already supplied to the model. verify_llm subsequently validates citations only against the truncated result, so citing one of those previously supplied chunks incorrectly triggers invalid_knowledge_reference, discards an otherwise valid LLM response, and can fail the analysis under the fail policy. Keep the complete set of supplied citation IDs for validation, and preserve cited chunks in the persisted context rather than selecting solely by retrieval score.
AGENTS.md reference: AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
| initialize_database(engine) | ||
| if settings.demo_mode and settings.auto_seed: | ||
| seed_demo(factory) | ||
| seed_knowledge(factory, embeddings, settings) |
There was a problem hiding this comment.
On a fresh database with KNOWLEDGE_EMBEDDING_PROVIDER=openai and the default enabled demo/auto-seed settings, this startup call embeds all seed documents through the remote service. A timeout or HTTP error propagates out of the lifespan before yield, preventing the entire API from starting, including deterministic analysis and health endpoints. The retrieval node's error-degradation handling cannot help because startup never completes. Handle optional knowledge-seeding failures separately so a temporarily unavailable embedding service does not take down the application.
Useful? React with 👍 / 👎.
| if len(sentence) > max_chars: | ||
| for index in range(0, len(sentence), max_chars): | ||
| units.append(sentence[index:index + max_chars]) |
There was a problem hiding this comment.
If a paragraph contains a short sentence followed by a sentence longer than max_chars, the short sentence remains in current while the later sentence is appended directly to units; the buffered text is emitted afterward. For example, _split_long('First. ' + 'B' * 25 + '.', 10) returns the B fragments before First.. This corrupts the original ordering of indexed SOP text and can attach introductory conditions to the wrong subsequent instructions. Flush and clear current before appending fragments of the oversized sentence.
Useful? React with 👍 / 👎.
| return [self._normalize(vector) for vector in vectors] | ||
|
|
||
| @staticmethod | ||
| def _normalize(vector: list[Any]) -> list[float]: | ||
| values = [float(value) for value in vector] |
There was a problem hiding this comment.
If an embedding endpoint returns a nonnumeric or null vector element, such as {"data":[{"index":0,"embedding":[null]}]}, the response passes the list checks but raises TypeError or ValueError during normalization outside the response-validation handler. Direct knowledge-search and document-creation requests therefore return an unhandled 500 instead of the documented knowledge_embedding_failed 502. Include numeric conversion in the guarded validation and reject non-finite values before storing or scoring vectors.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab79d125b4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| stale = [ | ||
| chunk for chunk in chunks | ||
| if chunk.embedding_model != embeddings.model or not chunk.embedding |
There was a problem hiding this comment.
If an existing hashing knowledge base is restarted with a different KNOWLEDGE_EMBEDDING_DIMENSIONS value, the model name remains hashing-v1, so this check treats the old vectors as current. Both explicit reindexing and re-uploading identical content therefore become no-ops, while _cosine returns zero for every old chunk because its dimensions differ from the query vector. Retrieval silently loses its vector component and can reject previously answerable queries. Include the vector dimensions in the embedding identity or freshness check so reindexing can repair this supported configuration change.
Useful? React with 👍 / 👎.
| existing = session.scalar( | ||
| select(KnowledgeDocument).where( | ||
| KnowledgeDocument.content_sha256 == sha, | ||
| KnowledgeDocument.active.is_(True), | ||
| ) |
There was a problem hiding this comment.
In the PostgreSQL deployment, two concurrent uploads of the same new content can both observe no existing document here and then both insert successfully: KnowledgeDocument.content_sha256 has only a non-unique index. This violates the documented creation idempotency and produces identical chunks with different IDs, allowing duplicate results to consume retrieval slots. Enforce uniqueness for active content hashes, or otherwise serialize creation, and return the existing document when another request wins the race.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 044bd1e437
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if existing is not None: | ||
| count = int(session.scalar( | ||
| select(func.count()).select_from(KnowledgeChunk).where(KnowledgeChunk.document_id == existing.id) | ||
| ) or 0) | ||
| return knowledge_document_output(existing, count, created=False) |
There was a problem hiding this comment.
After changing KNOWLEDGE_EMBEDDING_MODEL, uploading an existing document again returns created=false without generating embeddings for the new model because deduplication checks only the content hash and active flag. Meanwhile, search_knowledge excludes all chunks whose embedding model differs, so the documented re-ingestion path leaves these documents permanently absent from search unless users first deactivate and recreate them. Make the duplicate-content path account for the current embedding model or provide an explicit reindex operation.
Useful? React with 👍 / 👎.
| text = str(chunk["content"]) | ||
| session.add(KnowledgeChunk( | ||
| id=new_id(), document_id=document.id, chunk_index=index, | ||
| heading=chunk["heading"], content=text, char_count=len(text), |
There was a problem hiding this comment.
In the PostgreSQL Compose deployment, an otherwise valid document containing a Markdown heading longer than 200 characters causes document creation to fail with an unhandled database error: chunk_text retains the entire heading, but KnowledgeChunk.heading is declared as String(200). The request schema limits total content length, not individual headings, so this input passes API validation and only fails when the transaction flushes. Validate heading length or truncate the stored heading before insertion so supported document uploads do not produce HTTP 500 responses.
Useful? React with 👍 / 👎.
- Reindex when a same-model provider returns a different vector size, not only when the model name changes. - Enforce one active document per content SHA-256 with a partial unique index; migrate legacy duplicates by soft-deleting all but the earliest row. - Serialize SQLite uploads with BEGIN IMMEDIATE and recover PostgreSQL insert conflicts in a savepoint, reusing the winning document. - Skip stale-dimension chunks during retrieval and add dimension, migration, and two-request concurrency tests for SQLite and PostgreSQL. Co-Authored-By: Claude Code <noreply@anthropic.com>
摘要
knowledge_documents/knowledge_chunks、Markdown 字符分块、hashing / OpenAI-compatible embedding、混合检索、no_answer、引用校验、知识库 API 与前端面板。KnowledgeSearchOutput验证失败。retrieve_knowledge不再受通用step_timeout_seconds事后检查，远程 embedding 超时降级为knowledge_error，不阻断确定性分析。044bd1e）：/health与/api/demo返回knowledge.seed_error；数据库初始化失败仍中止启动。_split_long在“短句 + 超长句 + 短句”场景的顺序问题，保留原文顺序、chunk 长度上限与 overlap。null、非数字、NaN、Infinity、空向量、重复/缺失index和维度不一致，统一映射knowledge_embedding_failed（502），非法向量不落库。ab79d12）：created=false而跳过索引：在原文档上重嵌入现有 chunk，保留文档 ID、source、version、effective_date、active 与 chunk ID/正文；新增POST /api/knowledge/documents/{id}/reindex（仅 approver），重复执行幂等，失败事务回滚，不静默降级为 hashing。knowledge_chunks.heading前截断到 200 字符，覆盖 200/201/500 字符边界；正文与段落顺序不变，避免 PostgreSQLString(200)写入失败。cff917d，第二轮 P2 为044bd1e，第三轮 P2 为ab79d12。实际执行的验证
环境：Windows 11、Python 3.12.8、Node 24.16.0。
git status044bd1e；提交并推送后远端feat/agent-evaluation-trace=ab79d12.venv/Scripts/python.exe -m pytest backend/tests -qPROFITOPS_TEST_POSTGRES_URL=... pytest backend/tests/test_knowledge.py::test_postgres_accepts_bounded_long_headingnpm --prefix frontend testnpm --prefix frontend run build.venv/Scripts/python.exe scripts/smoke.py --base-url http://127.0.0.1:18013LLM_API_KEY=""，端口 18013）npm --prefix frontend run test:e2edata/e2e.db）.venv/Scripts/python.exe backend/evals/rag.pyab79d12）；mergeStateStatus: CLEAN新增回归覆盖：
httpx.MockTransport模拟远程 embedding 超时，核心 API 启动成功且knowledge.seed_error可见；已有文档保留；数据库初始化失败仍抛出。null、非数字、NaN、Infinity、空向量、重复 index、维度不一致等异常 embedding 响应统一 502，且不写入数据库。embedding_model保持不变，旧模型仍可检索；analyst 调用 reindex 返回 403，文档数量不增加。未验证
httpx.MockTransport测试。已知限制与后续 P2
version/effective_date不参与检索；没有文档级 ACL、租户隔离、reranker 或向量数据库。Z、no_answer弃答、创建竞态、失败状态 UI、Promise.all、citation 评测自证等 P2 未在本 PR 处理。审查重点
🤖 Generated with Claude Code