Resolve Issue #64

README.md

@@ -1,5 +1,3 @@
-<img width="1000" height="233" alt="20260506-102713" src="https://github.com/user-attachments/assets/896e64d2-e50e-4158-b71c-bc69e11c7c65" />
-
 <h1 align="center">Prepare unstructured data for AI Agents</h1>
 
 <p align="center">
@@ -45,42 +43,33 @@ Knowhere is the open-source infrastructure for unstructured data processing. It
 ## How it Works
 
 > [!TIP]
-> **TL;DR**: Knowhere parses documents into structured units, maps them in a graph, and lets agents navigate that context to find and cite reliable evidence.
+> **TL;DR**: Knowhere builds navigable memory from messy documents, then lets agents retrieve and cite evidence from that memory.
 
-Knowhere turns raw documents into a structured memory store that AI agents can navigate and cite. The process follows a three-stage pipeline:
+Knowhere turns raw documents into a structured memory store that AI agents can navigate and cite. The process follows two steps:
 
-```mermaid
-flowchart LR
-    A[📄 Document Parsing] --> B[🕸️ Graph Construction]
-    B --> C[🤖 Agentic Retrieval]
-    B --> D[🔍 Vector-based RAG]
-    C --> E[✅ Cited Results]
-    D --> E
-```
+### Step 1: Parse and Build Memory
+
+<p align="center">
+  <img alt="Step 1: Parse and Build Memory" src="docs/assets/step-1-parse-build-memory.png" width="900">
+</p>
+
+Parsing, chunking, hierarchy extraction, and graph construction are unified into one outcome: a navigable memory layer for AI agents.
+
+- **Parse**: Route PDFs, Office files, images, tables, Markdown, and text to specialized parsers.
+- **Structure**: Preserve headings, section paths, multi-modal assets, and chunk relationships.
+- **Build Memory**: Store chunks, navigation trees, summaries, and graph links as agent-ready context.
+
+### Step 2: Agentic Retrieval
+
+<p align="center">
+  <img alt="Step 2: Agentic Retrieval" src="docs/assets/step-2-agentic-retrieval.png" width="900">
+</p>
+
+Agents retrieve by navigating memory instead of depending on a single flat vector lookup.
 
-### 1. Document Parsing
-Knowhere routes files to specialized parsers for PDFs, Office docs, images, and more. We don't just extract text; we preserve the document's hierarchy:
-- **Hierarchical Paths**: Every chunk knows its exact location (e.g., `Section 2.1 > Table 4`).
-- **Multi-modal Units**: Tables and images are treated as distinct assets with their own metadata.
-- **Structural Awareness**: Heading levels and section boundaries are maintained to keep context intact.
-
-### 2. Memory Graph
-Parsed content is organized into a lightweight graph. It’s designed as a practical map for agents, not a complex ontology.
-- **Nodes**: Represent documents, sections, and chunks.
-- **Edges**: Map semantic relationships (keyword overlap, summaries) and structural links.
-This graph helps agents quickly understand what a document is about and which neighboring files might be relevant.
-
-### 3a. Agentic Retrieval
-An agent navigates the memory graph to find evidence rather than relying on a single vector lookup:
-- **Hybrid Discovery**: Fuses keyword and semantic search (RRF) for broad first-pass coverage.
-- **Agent Navigation**: The agent "walks" the graph, reviewing section previews to drill down into the most relevant paths.
-- **Cited Evidence**: Results are returned as traceable evidence — source document, section, chunk, and any linked image or table assets.
-
-### 3b. Vector-based RAG
-For teams that prefer a pure retrieval pipeline without agent overhead, Knowhere's parsed chunks plug directly into standard vector stacks:
-- **Dense Search**: Chunk embeddings stored in Qdrant, pgvector, or Milvus for fast ANN lookup.
-- **Sparse Search**: BM25 term index for keyword-sensitive queries.
-- **Multi-channel Fusion**: Dense and sparse results are fused with RRF before being returned, giving you the best of both signals.
+- **Discover**: Fuse keyword, path, content, and semantic signals for broad first-pass coverage.
+- **Navigate**: Walk section trees and graph links to drill into the most relevant document regions.
+- **Cite Evidence**: Return traceable results with source document, section, chunk, and linked image or table assets.
 
 ## Ecosystem
 
@@ -140,8 +129,16 @@ cp apps/worker/.env.example apps/worker/.env
 
 - database and Redis connection settings
 - S3-compatible storage credentials
-- `DS_KEY`
-- any optional LLM, billing, or webhook providers you want to enable
+- at least one LLM provider key: `DS_KEY`, `ALI_API_KEYS`, `GPT_API_KEY`, or `GLM_API_KEY`
+- `MINERU_API_KEYS` if you need PDF parsing
+- a vision-capable model provider if you need image summaries, OCR, atlas classification, or image-aware retrieval
+- any optional billing or webhook providers you want to enable
+
+Most parser and retrieval tuning values have code defaults. Start with the
+required external services first, then override model names, provider URLs,
+budgets, or concurrency limits only when your deployment needs different
+behavior. See [docs/external-services.md](docs/external-services.md) for the
+full dependency matrix.
 
 4. Start the local infrastructure stack:
 

apps/api/.env.example

@@ -66,37 +66,26 @@ OSS_ENDPOINT=
 OSS_EVENT_CALLBACK_KEY=
 OSS_EVENT_VERIFY_SIGNATURE=true
 
-# Required for local startup: one OpenAI-compatible provider
-DS_KEY=replace-with-your-deepseek-key
-DS_URL=https://api.deepseek.com/v1
+# Required for parsing/retrieval LLM calls: configure at least one provider key.
+# Provider URLs and model names have code defaults; override them only when needed.
+DS_KEY=
 GPT_API_KEY=
 GLM_API_KEY=
-GLM_URL=https://open.bigmodel.cn/api/paas/v4
 ALI_API_KEYS=
-ALI_TOKEN_RPM_LIMIT=300
-ALI_TOKEN_DAILY_LIMIT=10000
-ALI_TOKEN_COOLDOWN_SECONDS=60
-ALI_INLINE_MAX_RETRIES=3
-ALI_SDK_MAX_RETRIES=3
-ALI_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
 ARK_API_KEY=
-ARK_URL=https://ark.cn-beijing.volces.com/api/v3/chat/completions
-NORMOL_MODEL=deepseek-chat
-HIERARCHY_LLM_MODEL=qwen3.6-flash
-IMAGE_MODEL=qwen3.5-flash
-IMAGE_MODEL_MAX=qwen3.5-flash
-RETRIEVAL_DECOMPOSITION_ENABLED=false
-RETRIEVAL_PLANNER_MODEL=
-RETRIEVAL_PLANNER_THINKING_BUDGET=4000
-RETRIEVAL_DECOMPOSITION_MAX_STEPS=5
-RETRIEVAL_WALLET_TOTAL_BUDGET=200000
-RETRIEVAL_WALLET_PER_RETRIEVE_STEP_BUDGET=40000
-RETRIEVAL_WALLET_PER_SYNTHESIZE_STEP_BUDGET=6000
-RETRIEVAL_WORKFLOW_PARALLEL_MAX=3
 
-# Agentic retrieval (LLM-driven hierarchical navigation).
-# Set to false to fall back to legacy 3-channel RRF mode.
-RETRIEVAL_AGENTIC_ENABLED=true
+# Optional overrides:
+# DS_URL=https://api.deepseek.com/v1
+# GLM_URL=https://open.bigmodel.cn/api/paas/v4
+# ALI_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
+# ARK_URL=https://ark.cn-beijing.volces.com/api/v3/chat/completions
+# NORMOL_MODEL=deepseek-chat
+# HIERARCHY_LLM_MODEL=
+# IMAGE_MODEL=qwen3.5-flash
+# IMAGE_MODEL_MAX=qwen3.5-flash
+
+# Optional retrieval overrides have code defaults. Set RETRIEVAL_AGENTIC_ENABLED=false
+# only when you need to fall back to legacy 3-channel RRF mode.
 
 # File handling defaults
 SUPPORTED_EXTENSIONS=.doc,.docx,.pdf,.txt,.xls,.xlsx,.csv,.pptx,.jpg,.jpeg,.png,.md
@@ -120,16 +109,16 @@ FRONTEND_URL=http://localhost:3000
 
 # Required for specific features: parsing providers
 MINERU_API_KEYS=
-MINERU_URL=https://mineru.net/api/v4
-MINERU_TOKEN_RPM_LIMIT=300
-MINERU_TOKEN_DAILY_LIMIT=10000
-MINERU_TOKEN_COOLDOWN_SECONDS=60
-FORCE_MINERU_UPLOAD_ENABLED=false
 ILOVEAPI_PUBLIC_KEY=
 ILOVEAPI_SECRET_KEY=
-ILOVEAPI_BASE_URL=https://api.ilovepdf.com/v1
-ILOVEAPI_TIMEOUT=120
+# MINERU_URL=https://mineru.net/api/v4
+# MINERU_TOKEN_RPM_LIMIT=300
+# MINERU_TOKEN_DAILY_LIMIT=10000
+# MINERU_TOKEN_COOLDOWN_SECONDS=60
+# FORCE_MINERU_UPLOAD_ENABLED=false
+# ILOVEAPI_BASE_URL=https://api.ilovepdf.com/v1
+# ILOVEAPI_TIMEOUT=120
 
 # Legacy parser compatibility fields.
-ALL_DF_COLS=content,path,type,length,keywords,summary,know_id,tokens,connectto,addtime,page_nums
-SPLIT_CHAR=/
+# ALL_DF_COLS=content,path,type,length,keywords,summary,know_id,tokens,connectto,addtime,page_nums
+# SPLIT_CHAR=/

apps/worker/.env.example

@@ -71,32 +71,26 @@ QSTASH_MAX_RETRIES=5
 # QSTASH_CURRENT_SIGNING_KEY=
 # QSTASH_NEXT_SIGNING_KEY=
 
-# Required for local startup: one OpenAI-compatible provider
-DS_KEY=replace-with-your-deepseek-key
-DS_URL=https://api.deepseek.com/v1
+# Required for parsing/retrieval LLM calls: configure at least one provider key.
+# Provider URLs and model names have code defaults; override them only when needed.
+DS_KEY=
 GPT_API_KEY=
 GLM_API_KEY=
-GLM_URL=https://open.bigmodel.cn/api/paas/v4
 ALI_API_KEYS=
-ALI_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
 ARK_API_KEY=
-ARK_URL=https://ark.cn-beijing.volces.com/api/v3/chat/completions
-NORMOL_MODEL=deepseek-chat
-HIERARCHY_LLM_MODEL=deepseek-chat
-IMAGE_MODEL=qwen3.5-flash
-IMAGE_MODEL_MAX=qwen3.5-flash
-RETRIEVAL_DECOMPOSITION_ENABLED=false
-RETRIEVAL_PLANNER_MODEL=
-RETRIEVAL_PLANNER_THINKING_BUDGET=4000
-RETRIEVAL_DECOMPOSITION_MAX_STEPS=5
-RETRIEVAL_WALLET_TOTAL_BUDGET=200000
-RETRIEVAL_WALLET_PER_RETRIEVE_STEP_BUDGET=40000
-RETRIEVAL_WALLET_PER_SYNTHESIZE_STEP_BUDGET=6000
-RETRIEVAL_WORKFLOW_PARALLEL_MAX=3
 
-# Agentic retrieval (LLM-driven hierarchical navigation).
-# Set to false to fall back to legacy 3-channel RRF mode.
-RETRIEVAL_AGENTIC_ENABLED=true
+# Optional overrides:
+# DS_URL=https://api.deepseek.com/v1
+# GLM_URL=https://open.bigmodel.cn/api/paas/v4
+# ALI_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
+# ARK_URL=https://ark.cn-beijing.volces.com/api/v3/chat/completions
+# NORMOL_MODEL=deepseek-chat
+# HIERARCHY_LLM_MODEL=
+# IMAGE_MODEL=qwen3.5-flash
+# IMAGE_MODEL_MAX=qwen3.5-flash
+
+# Optional retrieval overrides have code defaults. Set RETRIEVAL_AGENTIC_ENABLED=false
+# only when you need to fall back to legacy 3-channel RRF mode.
 
 # Required for specific features: billing and analytics
 BILLING_ENABLED=false
@@ -106,21 +100,21 @@ MOESIF_APPLICATION_ID=
 
 # Required for specific features: parsing providers
 MINERU_API_KEYS=
-MINERU_URL=https://mineru.net/api/v4
-MINERU_TOKEN_RPM_LIMIT=300
-MINERU_TOKEN_DAILY_LIMIT=10000
-MINERU_TOKEN_COOLDOWN_SECONDS=60
-FORCE_MINERU_UPLOAD_ENABLED=false
 ILOVEAPI_PUBLIC_KEY=
 ILOVEAPI_SECRET_KEY=
-ILOVEAPI_BASE_URL=https://api.ilovepdf.com/v1
-ILOVEAPI_TIMEOUT=120
+# MINERU_URL=https://mineru.net/api/v4
+# MINERU_TOKEN_RPM_LIMIT=300
+# MINERU_TOKEN_DAILY_LIMIT=10000
+# MINERU_TOKEN_COOLDOWN_SECONDS=60
+# FORCE_MINERU_UPLOAD_ENABLED=false
+# ILOVEAPI_BASE_URL=https://api.ilovepdf.com/v1
+# ILOVEAPI_TIMEOUT=120
 
 # File handling defaults
 SUPPORTED_EXTENSIONS=.doc,.docx,.pdf,.txt,.xls,.xlsx,.csv,.pptx,.jpg,.jpeg,.png,.md
 MAX_FILE_SIZE=104857600
 
 # Legacy parser compatibility fields.
-ALL_DF_COLS=content,path,type,length,keywords,summary,know_id,tokens,connectto,addtime,page_nums
-SPLIT_CHAR=/
+# ALL_DF_COLS=content,path,type,length,keywords,summary,know_id,tokens,connectto,addtime,page_nums
+# SPLIT_CHAR=/
 

apps/worker/app/services/document_agent/__init__.py

@@ -0,0 +1,15 @@
+"""Phase 1 document split-agent utilities."""
+
+from app.services.document_agent.manifest import (
+    GlobalSignals,
+    ShardManifest,
+    ShardSignal,
+    SpecialPage,
+)
+
+__all__ = [
+    "GlobalSignals",
+    "ShardManifest",
+    "ShardSignal",
+    "SpecialPage",
+]