Release 05/19

AGENTS.md

@@ -33,7 +33,7 @@ knowhereapi-main/
 │   ├── api/          # FastAPI REST API (port 5005)
 │   │   ├── app/
 │   │   │   ├── api/v1/routes/   # Endpoint handlers
-│   │   │   ├── services/        # Business logic (auth, knowledge, billing)
+│   │   │   ├── services/        # Business logic (auth, ingestion, billing)
 │   │   │   └── repositories/    # Data access layer
 │   │   └── main.py              # Entrypoint, runs migrations on start
 │   ├── worker/       # Celery worker for async document processing
@@ -50,7 +50,10 @@ knowhereapi-main/
 │       ├── services/retrieval/  # Core retrieval engine
 │       ├── services/chunks/     # DataFrame → ChunkPayload conversion
 │       ├── services/ai/         # LLM prompt service & AI client
-│       └── utils/               # Text, file, and chunk utilities
+│       ├── services/http/       # Public URL validation and outbound HTTP
+│       ├── services/redis/      # Redis state, key language, and retry policy
+│       ├── services/quota/      # Shared token-pool quota primitives
+│       └── utils/               # Generic text, chunk, and API helpers
 └── deploy/                      # Docker Compose & deployment scripts
 ```
 
@@ -70,14 +73,14 @@ flowchart TB
     end
 
     subgraph PARSE["② Document Parsing (Worker)"]
-        Queue --> Router["parse_service.checkerboard_inject_parse"]
-        Router --> Profiler["doc_profiler.profile_document"]
-        Profiler --> PDF["pdf_parser → MinerU"]
-        Profiler --> DOCX["doc_parser.parse_docx"]
-        Profiler --> PPTX["pptx_parser → iLoveAPI → PDF"]
-        Profiler --> XLSX["table_parser.parse_xlsx"]
-        Profiler --> MD["md_parser.parse_md"]
-        Profiler --> IMG["image_parser.parse_image"]
+        Queue --> Router["parse_service.checkerboard_parse_output"]
+        Router --> Profiler["profiling.doc_profiler.profile_document"]
+        Profiler --> PDF["formats.pdf.parser → MinerU"]
+        Profiler --> DOCX["formats.docx.parser.parse_docx"]
+        Profiler --> PPTX["formats.pptx.parser → iLoveAPI → PDF"]
+        Profiler --> XLSX["formats.excel.table_parser.parse_xlsx"]
+        Profiler --> MD["formats.markdown.parser.parse_md"]
+        Profiler --> IMG["formats.image.parser.parse_image"]
         PDF --> DF["pd.DataFrame (ALL_DF_COLS)"]
         DOCX --> DF
         PPTX --> DF
@@ -117,36 +120,36 @@ flowchart TB
 ### Entry Point
 
 `apps/worker/app/services/document_parser/parse_service.py` →
-`checkerboard_inject_parse()`
+`checkerboard_parse_output()`
 
-This is the universal entry for all file types. It:
+This is the typed `ParseOutput` entry for all file types. The parser flow:
 
-1. **Profiles** the document via `doc_profiler.profile_document()` to detect
+1. **Profiles** the document via `profiling.doc_profiler.profile_document()` to detect
    file type, page count, and special categories (e.g. `atlas`).
 2. **Routes** to the appropriate parser based on file extension.
 3. **Post-processes**: cleans up unreferenced images, compresses PNG→JPG.
-4. Returns `(output_dir, parsed_df)` — the parsed DataFrame.
+4. Returns typed parse output with task-local artifact paths.
 
 ### Parser Routing Table
 
 | Extension | Parser Module | Strategy |
 |:---|:---|:---|
-| `.pdf` | `pdf_parser.parse_pdfs` | MinerU API → `md_parser` → `layout_parser.pred_titles` |
-| `.docx` | `doc_parser.parse_docx` + `convert_doc2dics` | OXML iteration → heading detection → hierarchical tree |
-| `.doc` | `legacy_converter.doc_to_docx` → `.docx` pipeline | LibreOffice headless conversion first |
-| `.pptx` | `pptx_parser.parse_pptx` | iLoveAPI PPTX→PDF → MinerU pipeline |
-| `.xlsx` | `table_parser.parse_xlsx` | Sheet-by-sheet HTML table extraction |
-| `.xls` | `legacy_converter.xls_to_xlsx` → `.xlsx` pipeline | LibreOffice conversion first |
-| `.md` | `md_parser.parse_md` | Markdown heading parsing + LLM summaries |
-| `.txt` | `txt_parser.parse_texts` → `md_parser` | Read lines then route to MD parser |
-| `.png/.jpg` | `image_parser.parse_image` | VLM image description + OCR |
-| `.fragment` | `fragment_parser.parse_fragment` | Raw text fragment ingestion |
-
-### Heading Detection: `layout_parser.pred_titles()`
+| `.pdf` | `formats.pdf.parser.parse_pdfs` | MinerU API → Markdown parser → `structure.layout_parser.pred_titles` |
+| `.docx` | `formats.docx.parser.parse_docx` + `convert_doc2dics` | OXML iteration → heading detection → hierarchical tree |
+| `.doc` | `conversion.legacy_converter.doc_to_docx` → `.docx` pipeline | LibreOffice headless conversion first |
+| `.pptx` | `formats.pptx.parser.parse_pptx` | iLoveAPI PPTX→PDF → MinerU pipeline |
+| `.xlsx` | `formats.excel.table_parser.parse_xlsx` | Sheet-by-sheet HTML table extraction |
+| `.xls` | `conversion.legacy_converter.xls_to_xlsx` → `.xlsx` pipeline | LibreOffice conversion first |
+| `.md` | `formats.markdown.parser.parse_md` | Markdown heading parsing + LLM summaries |
+| `.txt` | `formats.text.parser.parse_texts` → Markdown parser | Read lines then route to MD parser |
+| `.png/.jpg` | `formats.image.parser.parse_image` | VLM image description + OCR |
+| `.fragment` | `formats.fragment.parser.parse_fragment` | Raw text fragment ingestion |
+
+### Heading Detection: `structure.layout_parser.pred_titles()`
 
 The core hierarchical recognition module. Determines heading levels using:
 
-1. **TOC-first**: If a DOCX TOC exists (`toc_parser.build_docx_toc_hierarchies`),
+1. **TOC-first**: If a DOCX TOC exists (`structure.toc_parser.build_docx_toc_hierarchies`),
    use it as ground truth for heading levels.
 2. **Regex patterns**: Match numbered headings like `1.2.3`, `第X章`, `（一）`.
 3. **LLM smart parse**: When `smart_title_parse=True`, send candidate headings
@@ -155,7 +158,7 @@ The core hierarchical recognition module. Determines heading levels using:
 4. **Font clustering (PDF)**: K-means on span heights from MinerU `layout.json`
    to group headings into 5 discrete tiers.
 
-### DOCX Parsing Deep Dive: `doc_parser.py`
+### DOCX Parsing Deep Dive: `formats/docx/parser.py`
 
 ```mermaid
 flowchart LR
@@ -188,11 +191,11 @@ Key logic in `parse_docx()`:
 
 ```mermaid
 flowchart LR
-    PDF[pdf_parser] --> MinerU[MinerU Cloud API]
+    PDF[formats.pdf.parser] --> MinerU[MinerU Cloud API]
     MinerU --> MDFile[Markdown + layout.json]
-    MDFile --> MDParser[md_parser.parse_md]
+    MDFile --> MDParser[formats.markdown.parser.parse_md]
     MDParser --> EvalHeadings[eval_md_headings + layout.json]
-    EvalHeadings --> PredTitles[layout_parser.pred_titles]
+    EvalHeadings --> PredTitles[structure.layout_parser.pred_titles]
     PredTitles --> Chunks[Hierarchical Chunks]
 ```
 
@@ -204,21 +207,21 @@ flowchart LR
 | Heading hierarchy recognition | `HIERARCHY_LLM_MODEL` | Falls back to `NORMOL_MODEL` |
 | Image description (VLM) | `IMAGE_MODEL` | `qwen3.5-flash` |
 | Image OCR / Q&A | `IMAGE_MODEL_MAX` | `qwen3.5-flash` |
-| Atlas classification | VLM via `atlas_classifier` | `IMAGE_MODEL` |
+| Atlas classification | VLM via `formats.atlas.classifier` | `IMAGE_MODEL` |
 
 ---
 
-## Persisted Knowledge Base Schema (On-Disk Output)
+## Persisted Document Corpus Schema (On-Disk Output)
 
-After parsing and chunk conversion, results are persisted to `~/.knowhere/{kb_name}/`.
+After parsing and chunk conversion, results are persisted to `~/.knowhere/{corpus_name}/`.
 This on-disk structure is the **authoritative persisted format** — the intermediate
 DataFrame is an internal detail. Below is the complete schema.
 
-### KB-Level Directory Layout
+### Corpus-Level Directory Layout
 
 ```text
-~/.knowhere/{kb_name}/
-├── knowledge_graph.json           # KB-wide graph: file metadata + cross-doc edges
+~/.knowhere/{corpus_name}/
+├── knowledge_graph.json           # corpus-wide graph: file metadata + cross-doc edges
 ├── chunk_stats.json               # Per-chunk retrieval hit analytics {chunk_id → stats}
 ├── {source_file_name}/            # One directory per ingested document
 │   ├── chunks.json                # All parsed chunks for this document
@@ -233,12 +236,12 @@ DataFrame is an internal detail. Below is the complete schema.
 │   └── toc_hierarchies.json       # Debug: extracted TOC structure (DOCX only)
 ```
 
-### `knowledge_graph.json` — KB-Wide Graph
+### `knowledge_graph.json` — Corpus-Wide Graph
 
 ```json
 {
   "version": "2.0",
-  "kb_id": "test_kb",
+  "corpus_id": "test-corpus",
   "stats": { "total_files": 3, "total_chunks": 364, "total_cross_file_edges": 0 },
   "files": {
     "AI_Security_Report.docx": {
@@ -422,7 +425,7 @@ Used by agentic retrieval for 2-level section browsing. Structure:
 ```
 
 The `HIERARCHY` field is a nested dict representing the full heading tree
-discovered by `layout_parser.pred_titles()`. Each key is a heading title;
+discovered by `structure.layout_parser.pred_titles()`. Each key is a heading title;
 its value is a dict of child headings (empty `{}` for leaf nodes).
 
 ### Intermediate DataFrame (`ALL_DF_COLS`)
@@ -527,6 +530,19 @@ debug CSVs (`preds_*.csv`) are saved alongside for troubleshooting.
 
 `shared/services/retrieval/app_service.py` → `run_retrieval_query()`
 
+Core retrieval internals are grouped by ownership:
+
+- `execution/`: request shaping, route selection, legacy route execution, and public response projection.
+- `search/`: lexical channels, scoring, section filters, and candidate ranking.
+- `hydration/`: row/path/reference hydration, inline assets, and result assembly.
+- `graph/`: document graph publication/query support.
+- `stats/`: retrieval hit recording.
+- `workflow/`: query planning, step execution, synthesis, and wallet state.
+- `agentic/core/`: agentic run types, token budgets, runtime config, and traces.
+- `agentic/discovery/`: bottom discovery and document selection.
+- `agentic/navigation/`: section-tree navigation, selection hydration, and asset tools.
+- `agentic/evidence/`: evidence tree rendering and budget trimming.
+
 ### Two Retrieval Modes
 
 The system supports two modes, controlled globally by `RETRIEVAL_AGENTIC_ENABLED` and locally via the per-request `use_agentic` toggle.
@@ -543,7 +559,7 @@ flowchart LR
     T --> RRF
     RRF --> Graph[Legacy Graph Routing]
     Graph --> Rank[Dual-priority ranking]
-    Rank --> Assemble[assemble_retrieval_results]
+    Rank --> Assemble[hydration.result_assembly]
 ```
 
 **Channel weights** (default): path=1.0, content=2.0, term=1.5
@@ -578,18 +594,18 @@ Unlike legacy retrieval which relied on static `hydrate_mode` tags, hydration is
 
 ### Result Assembly
 
-`assemble_retrieval_results()`:
+`hydration.result_assembly.assemble_retrieval_results()`:
 
 1. Filters by `exclude_document_ids` and `exclude_sections`
 2. Filters by `allowed_chunk_types` (data_type parameter)
 3. Hydrates `connect_to` targets (related table chunks inlined into text)
 4. Cleans asset path references from content
 5. Attaches citation: `{document_id, chunk_id, source_file_name, section_path}`
 
-### Small KB Optimization
+### Small Corpus Optimization
 
 When `total_chunks <= top_k`, skips the full pipeline and returns all chunks
-directly (router: `small_kb_all`).
+directly (router: `small_corpus_all`).
 
 ### Caching