chore: sync main into staging

AGENTS.md

@@ -124,8 +124,13 @@ flowchart TB
 
 This is the typed `ParseOutput` entry for all file types. The parser flow:
 
-1. **Profiles** the document via `profiling.doc_profiler.profile_document()` to detect
-   file type, page count, and special categories (e.g. `atlas`).
+1. **Profiles** the document via `profiling.doc_profiler.profile_document()`. PDF
+   profiling uses `document_agent` as the single PyMuPDF feature source, then runs
+   VLM coarse classification with two fields: open semantic `category` (for example,
+   `Financial Prospectus`) and routing-only `routing_category`
+   (`atlas/scanned/slides/generic`). Oversized non-atlas PDFs additionally run the
+   structural anatomy stage once at the entry point and pass the resulting shard
+   plan to PDF parsing.
 2. **Routes** to the appropriate parser based on file extension.
 3. **Post-processes**: cleans up unreferenced images, compresses PNG→JPG.
 4. Returns typed parse output with task-local artifact paths.
@@ -134,7 +139,7 @@ This is the typed `ParseOutput` entry for all file types. The parser flow:
 
 | Extension | Parser Module | Strategy |
 |:---|:---|:---|
-| `.pdf` | `formats.pdf.parser.parse_pdfs` | MinerU API → Markdown parser → `structure.layout_parser.pred_titles` |
+| `.pdf` | `formats.pdf.parser.parse_pdfs` | DOC_PROFILE category dispatch: `atlas` → atlas parser; oversized with entry anatomy → shard MinerU; otherwise MinerU API → Markdown parser → `structure.layout_parser.pred_titles` |
 | `.docx` | `formats.docx.parser.parse_docx` + `convert_doc2dics` | OXML iteration → heading detection → hierarchical tree |
 | `.doc` | `conversion.legacy_converter.doc_to_docx` → `.docx` pipeline | LibreOffice headless conversion first |
 | `.pptx` | `formats.pptx.parser.parse_pptx` | iLoveAPI PPTX→PDF → MinerU pipeline |
@@ -187,12 +192,19 @@ Key logic in `parse_docx()`:
 - **Table handling**: `table2html()` converts python-docx Table to HTML with
   accurate `rowspan`/`colspan` via direct OXML inspection.
 
-### PDF Parsing: MinerU Pipeline
+### PDF Parsing: DOC_PROFILE + MinerU Pipeline
 
 ```mermaid
 flowchart LR
-    PDF[formats.pdf.parser] --> MinerU[MinerU Cloud API]
-    MinerU --> MDFile[Markdown + layout.json]
+    PDF[profiling.doc_profiler.profile_document] --> Probe[document_agent probe_page_features]
+    Probe --> Coarse[VLM coarse category]
+    Coarse -->|atlas| Atlas[formats.atlas.parser]
+    Coarse -->|oversized generic/scanned/slides| Anatomy[document_agent structural anatomy + shard_plan]
+    Coarse -->|standard| MinerU[MinerU Cloud API]
+    Anatomy --> Shards[Shard MinerU pipeline]
+    Shards --> MDFile[Markdown + layout.json]
+    Atlas --> Chunks[Atlas page chunks]
+    MinerU --> MDFile
     MDFile --> MDParser[formats.markdown.parser.parse_md]
     MDParser --> EvalHeadings[eval_md_headings + layout.json]
     EvalHeadings --> PredTitles[structure.layout_parser.pred_titles]
@@ -205,9 +217,9 @@ flowchart LR
 |:---|:---|:---|
 | Text/table summarization | `NORMOL_MODEL` | `deepseek-chat` |
 | Heading hierarchy recognition | `HIERARCHY_LLM_MODEL` | Falls back to `NORMOL_MODEL` |
-| Image description (VLM) | `IMAGE_MODEL` | `qwen3.5-flash` |
-| Image OCR / Q&A | `IMAGE_MODEL_MAX` | `qwen3.5-flash` |
-| Atlas classification | VLM via `formats.atlas.classifier` | `IMAGE_MODEL` |
+| Image description (VLM) | `IMAGE_MODEL` | `qwen3.6-flash` |
+| Image OCR / Q&A | `IMAGE_MODEL_MAX` | `qwen3.6-flash` |
+| PDF coarse classification | `IMAGE_MODEL` | `qwen3.6-flash` |
 
 ---
 

README.md

@@ -43,6 +43,7 @@ It ingests unstructured documents and produces persistent, navigable memory: par
 
 ## 📢 News
 
+- **June 1, 2026**: 📚 **Knowhere now supports ultra-long PDFs and atlas-style documents.** The parsing pipeline can process long-form PDFs with hundreds of pages (for example, 300, 500, or more) and route technical atlases or drawing collections through a dedicated layout-aware parser.
 - **May 7, 2026**: 🚀 **Knowhere is now Open Source!** We have open-sourced our entire stack for document ingestion, parsing, and agentic RAG. You can now self-host the full platform using [knowhere-self-hosted](https://github.com/Ontos-AI/knowhere-self-hosted). Check out our [Contribution Guide](CONTRIBUTING.md) to get involved!
 
 ## How it Works
@@ -77,7 +78,7 @@ A: Knowhere uses MinerU as its default parser because it performs best in our te
 
 **Q: What LLM / VLM dependencies does Knowhere have?**
 
-A: By default, DeepSeek (`deepseek-chat`) handles text and table summarization, and Qwen-VL (`qwen3.5-flash`) handles image OCR and descriptions. Knowhere is model-agnostic. Swap in OpenAI, DashScope, Zhipu, or Volcengine via environment variables.
+A: By default, DeepSeek (`deepseek-chat`) handles text and table summarization, and Qwen-VL (`qwen3.6-flash`) handles image OCR and descriptions. Knowhere is model-agnostic. Swap in OpenAI, DashScope, Zhipu, or Volcengine via environment variables.
 
 **Q: How is Agentic Retrieval different from traditional RAG?**
 

apps/api/.env.example

@@ -81,8 +81,8 @@ ARK_API_KEY=
 # ARK_URL=https://ark.cn-beijing.volces.com/api/v3/chat/completions
 # NORMOL_MODEL=deepseek-chat
 # HIERARCHY_LLM_MODEL=
-# IMAGE_MODEL=qwen3.5-flash
-# IMAGE_MODEL_MAX=qwen3.5-flash
+# IMAGE_MODEL=qwen3.6-flash
+# IMAGE_MODEL_MAX=qwen3.6-flash
 
 # Optional retrieval overrides have code defaults. Retrieval is evidence-only:
 # evidence_text is the primary output and answer_text is always empty. Set
@@ -95,7 +95,19 @@ MAX_FILE_SIZE=314572800
 MAX_PDF_PAGE_LIMIT=200
 OVERSIZED_PDF_SHARD_ENABLED=true
 OVERSIZED_PDF_SOFT_LIMIT=1500
+PDF_PAGE_TOC_ENABLED=true
+RETRIEVAL_PAGE_MEMORY_ENABLED=false
 MINERU_SHARD_CONCURRENCY=3
+PARSE_AGENT_PLAN_BUDGET=50000
+PARSE_AGENT_VISUAL_BUDGET=80000
+PARSE_AGENT_TOC_CONFIRM_MIN_BUDGET=8000
+PARSE_AGENT_TOC_CONFIRM_CAP=24000
+PARSE_AGENT_COARSE_PLANNER_MIN_BUDGET=12000
+PARSE_AGENT_COARSE_PLANNER_CAP=36000
+PARSE_AGENT_STRUCTURAL_REACT_MIN_BUDGET=24000
+PARSE_AGENT_STRUCTURAL_REACT_CAP=64000
+PARSE_AGENT_PAGE_TAGGING_MIN_BUDGET=0
+PARSE_AGENT_PAGE_TAGGING_CAP=0
 
 # Required for specific features: webhooks and callbacks
 WEBHOOK_MASTER_KEY=

apps/api/alembic/versions/f9a0b1c2d3e4_add_doc_profile_to_document_page_plan.py

@@ -0,0 +1,29 @@
+"""add doc profile to document page plan
+
+Revision ID: f9a0b1c2d3e4
+Revises: f8a9b0c1d2e3
+Create Date: 2026-06-11 09:50:00.000000
+
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "f9a0b1c2d3e4"
+down_revision: Union[str, Sequence[str], None] = "f8a9b0c1d2e3"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "document_page_plan",
+        sa.Column("doc_profile", sa.JSON(), nullable=True),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("document_page_plan", "doc_profile")

apps/api/alembic/versions/f9b0c1d2e3f4_add_parse_track_to_documents.py

@@ -0,0 +1,35 @@
+"""add parse track to documents
+
+Revision ID: f9b0c1d2e3f4
+Revises: f9a0b1c2d3e4
+Create Date: 2026-06-11 10:05:00.000000
+
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+revision: str = "f9b0c1d2e3f4"
+down_revision: Union[str, Sequence[str], None] = "f9a0b1c2d3e4"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "documents",
+        sa.Column(
+            "parse_track",
+            sa.String(length=32),
+            nullable=False,
+            server_default="chunk",
+        ),
+    )
+    op.alter_column("documents", "parse_track", server_default=None)
+
+
+def downgrade() -> None:
+    op.drop_column("documents", "parse_track")

apps/api/alembic/versions/f9c0d1e2f3a4_add_retrieval_steps_created_index.py

@@ -0,0 +1,32 @@
+"""Add retrieval step timestamp index.
+
+Revision ID: f9c0d1e2f3a4
+Revises: f9b0c1d2e3f4
+Create Date: 2026-06-12 08:35:00.000000
+"""
+
+from __future__ import annotations
+
+from typing import Sequence, Union
+
+from alembic import op
+
+
+# revision identifiers, used by Alembic.
+revision: str = "f9c0d1e2f3a4"
+down_revision: Union[str, Sequence[str], None] = "f9b0c1d2e3f4"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_index(
+        "idx_retrieval_steps_created",
+        "retrieval_steps",
+        ["created_at"],
+        unique=False,
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("idx_retrieval_steps_created", table_name="retrieval_steps")

apps/api/app/api/v1/routes/documents.py

@@ -17,7 +17,7 @@
 router = APIRouter(tags=["Documents"])
 
 _document_service = DocumentService()
-DocumentChunkType = Literal["text", "image", "table"]
+DocumentChunkType = Literal["text", "image", "table", "page"]
 
 
 async def _archive_document_response(

apps/api/app/core/middleware/telemetry.py

@@ -0,0 +1,37 @@
+"""Anonymous aggregate telemetry middleware."""
+
+from __future__ import annotations
+
+import time
+
+from starlette.types import ASGIApp, Message, Receive, Scope, Send
+
+from shared.services.telemetry.api_metrics import ApiRequestTelemetryMetrics
+
+
+class ApiTelemetryMiddleware:
+    """Record bounded API request metrics without request payloads or raw paths."""
+
+    def __init__(self, app: ASGIApp, *, metrics: ApiRequestTelemetryMetrics) -> None:
+        self.app = app
+        self.metrics = metrics
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        if scope["type"] != "http":
+            await self.app(scope, receive, send)
+            return
+
+        status_code = 500
+        started_at = time.perf_counter()
+
+        async def send_wrapper(message: Message) -> None:
+            nonlocal status_code
+            if message["type"] == "http.response.start":
+                status_code = int(message.get("status", 500))
+            await send(message)
+
+        try:
+            await self.app(scope, receive, send_wrapper)
+        finally:
+            elapsed_ms = (time.perf_counter() - started_at) * 1000
+            self.metrics.record(status_code=status_code, latency_ms=elapsed_ms)