feat(retrieval): default to agent explore and retire map-nav

AGENTS.md

@@ -103,13 +103,12 @@ flowchart TB
     subgraph RETRIEVE["⑤ Retrieval (shared)"]
         Query["POST /v1|/v2 retrieval/query"] --> Pipeline["run_retrieval_query"]
         Pipeline --> Classic["classic_topk / small_corpus (use_agentic=False)"]
-        Pipeline --> MapNav["mapnav checklist (default / use_agentic≠False)"]
+        Pipeline --> Explore["agent_explore + cursor_sdk (default / use_agentic≠False)"]
         Classic --> Channels["map_unit_discovery: path+content BM25 -> RRF"]
         Channels --> Rank["rank_retrieval_candidates"]
-        MapNav --> NavSnap["nav_snapshot + run_nav_episode"]
-        NavSnap --> Bridge["nav_bridge referenced_chunks"]
+        Explore --> Tools["corpus.* tools + harness.run_episode"]
         Rank --> Assemble["assemble_retrieval_results"]
-        Bridge --> Assemble
+        Tools --> Assemble
         Assemble --> Results["Cited Evidence Results"]
     end
 ```
@@ -493,8 +492,6 @@ debug CSVs (`preds_*.csv`) are saved alongside for troubleshooting.
 | `content_search_text` | `Text` | Pre-tokenized for BM25 content channel |
 | `path_search_text` | `Text` | Pre-tokenized for BM25 path channel |
 | `term_search_text` | `Text` | Pre-tokenized for term/grep channel |
-| `content_search_tsv` | `TSVECTOR` (computed) | PostgreSQL GIN index for full-text |
-| `path_search_tsv` | `TSVECTOR` (computed) | PostgreSQL GIN index for path |
 | `source_chunk_path` | `Text` | Original parser path |
 | `file_path` | `Text` | Asset reference (`images/x.jpg`) |
 | `chunk_metadata` | `JSON` | Keywords, tokens, connect_to, etc. |
@@ -545,23 +542,23 @@ debug CSVs (`preds_*.csv`) are saved alongside for troubleshooting.
 
 Core retrieval internals are grouped by ownership:
 
-- `execution/`: request shaping, route selection (classic / mapnav / small_corpus), and public response projection.
-- `search/`: `map_unit_discovery` (persisted map-unit BM25 discovery, with a legacy chunk-level PG FTS fallback), scoring, section filters, candidate ranking.
+- `execution/`: request shaping, route selection (classic / agent_explore / small_corpus), and public response projection.
+- `search/`: `map_unit_discovery` (persisted map-unit BM25 discovery; incomplete or incompatible indexes raise), scoring, section filters, candidate ranking.
 - `hydration/`: row/path/reference hydration, inline assets, and result assembly.
-- `nav/` + `nav_*.py`: map-nav checklist episode (PLANNER / HARVEST / CONTROL).
+- `agent_explore/` + `agent_tools/`: default agentic route (cursor_sdk harness). Map-nav is archived under `deprecated/mapnav/`.
 - `trace/`: `DecisionTraceStep` mapping and `TraceRecorder`.
 - `graph/`: document graph publication/query support.
 - `stats/`: retrieval hit recording.
 
 ### Two Retrieval Modes
 
-Per-request `use_agentic`: `False` → classic top-K (map-unit BM25); `None`/`True` → map-nav (default).
+Per-request `use_agentic`: `False` → classic top-K (map-unit BM25); `None`/`True` → agent_explore (default harness `cursor_sdk`).
 
-#### Classic Mode (map-unit BM25 + legacy FTS fallback)
+#### Classic Mode (map-unit BM25)
 
 Primary path is `search.map_unit_discovery.map_unit_discovery`: Python BM25Okapi
 over the persisted `document_map_unit_tokens` index, path and content channels
-only, fused by RRF.
+only, fused by RRF. An incomplete or incompatible map-unit index raises.
 
 ```mermaid
 flowchart LR
@@ -577,15 +574,15 @@ flowchart LR
 `score = weight / (k + rank + 1)` per channel, summed across channels, `k=60`.
 
 There is no scored term channel in this primary path. `term_search_text` /
-`term_search_text_lower` are persisted at publish time but are only read by
-the **legacy fallback** (`_legacy_chunk_discovery`), which runs only when a
-revision's map-unit index is missing or incomplete: a single SQL query
-scoring `GREATEST(ts_rank_cd(path_search_tsv), 2 * ts_rank_cd(content_search_tsv))`
-OR `term_search_text LIKE '%query%'`, not three independently-ranked channels.
+`term_search_text_lower` are persisted at publish time and are read by
+`corpus.recall`'s term channel, not by classic discovery.
 
-#### Map-nav Mode (default)
+#### Agent-explore Mode (default)
 
-Default agentic path is checklist map-nav (`nav/`): PLANNER (`plan_query`) → HARVEST (`execute_plan` / `harvest`, recursive DISPATCH) → CONTROL (`plan_control`). Episode config lives in `nav_config.py`. Exit bridge expands kept chunks to `referenced_chunks`; `decision_trace` is mapped in `trace/mapnav.py`. Token hard-stop uses `NavConfig.token_limit`.
+Default agentic path is `agent_explore`: a `corpus.*` tool loop run by the
+resolved harness (`AGENT_EXPLORE_HARNESS`, default `cursor_sdk`). Finish refs
+are resolved to chunks, then assembled like classic. Map-nav is archived at
+`deprecated/mapnav/` and is no longer a live route.
 
 ### Result Assembly
 

README.md

@@ -1,4 +1,4 @@
-<img width="1000" height="233" alt="20260506-102713" src="https://github.com/user-attachments/assets/896e64d2-e50e-4158-b71c-bc69e11c7c65" />
+<img width="1000" height="233" alt="Knowhere 2.0" src="docs/assets/knowhere-banner-2.0.png" />
 
 <h1 align="center">Prepare unstructured data for AI Agents</h1>
 

apps/api/.env.example

@@ -100,8 +100,12 @@ ARK_API_KEY=
 
 # Optional retrieval overrides have code defaults. Retrieval is evidence-only:
 # evidence_text is the primary output and answer_text is always empty.
-# Default path is map-nav (PLANNER+HARVEST+CONTROL); set use_agentic=false for
-# classic 3-channel RRF. Classic BM25 may use Postgres FTS prefilter:
+# Default path is agent_explore (AGENT_EXPLORE_HARNESS=cursor_sdk).
+# Set use_agentic=false for classic map-unit BM25.
+# AGENT_EXPLORE_HARNESS=cursor_sdk
+# CURSOR_API_KEY=   # required when harness is cursor_sdk
+# AGENT_EXPLORE_CURSOR_MODEL=composer-2.5
+# Classic BM25 may use Postgres FTS prefilter:
 # RETRIEVAL_POSTGRES_FTS_CANDIDATE_LIMIT=2000
 
 # File handling defaults

apps/api/alembic/env.py

@@ -40,21 +40,6 @@
         "session",
     }
 )
-_AUTOGENERATE_IGNORED_COLUMNS: frozenset[tuple[str, str]] = frozenset(
-    {
-        ("document_chunks", "content_search_tsv"),
-        ("document_chunks", "path_search_tsv"),
-    }
-)
-
-
-def _resolve_table_name(object_: object, compare_to: object | None) -> str | None:
-    for candidate in (object_, compare_to):
-        table = getattr(candidate, "table", None)
-        table_name = getattr(table, "name", None)
-        if isinstance(table_name, str):
-            return table_name
-    return None
 
 
 def include_object(
@@ -64,14 +49,10 @@ def include_object(
     reflected: bool,
     compare_to: object | None,
 ) -> bool:
-    """Exclude externally managed auth tables and generated TSV columns."""
-    del reflected
+    """Exclude externally managed auth tables."""
+    del object_, compare_to, reflected
     if type_ == "table" and isinstance(name, str) and name in _EXTERNALLY_MANAGED_TABLES:
         return False
-    if type_ == "column" and isinstance(name, str):
-        table_name = _resolve_table_name(object_, compare_to)
-        if table_name is not None and (table_name, name) in _AUTOGENERATE_IGNORED_COLUMNS:
-            return False
     return True
 
 

apps/api/alembic/versions/e4f5a6b7c8d9_drop_chunk_search_tsv_columns.py

@@ -0,0 +1,55 @@
+"""Drop unused PostgreSQL FTS columns on document_chunks.
+
+Retrieval no longer reads ``content_search_tsv`` / ``path_search_tsv``.
+They were generated from ``content_search_text`` / ``path_search_text`` and
+only served the retired FTS fallback.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+from alembic import op
+
+
+revision: str = "e4f5a6b7c8d9"
+down_revision: str | None = "d3e4f5a6b7c8"
+branch_labels: Sequence[str] | None = None
+depends_on: Sequence[str] | None = None
+
+__all__ = [
+    "revision",
+    "down_revision",
+    "branch_labels",
+    "depends_on",
+    "upgrade",
+    "downgrade",
+]
+
+
+def upgrade() -> None:
+    op.execute("DROP INDEX IF EXISTS idx_chunk_path_search_tsv")
+    op.execute("DROP INDEX IF EXISTS idx_chunk_content_search_tsv")
+    op.execute("ALTER TABLE document_chunks DROP COLUMN IF EXISTS path_search_tsv")
+    op.execute("ALTER TABLE document_chunks DROP COLUMN IF EXISTS content_search_tsv")
+
+
+def downgrade() -> None:
+    op.execute(
+        "ALTER TABLE document_chunks ADD COLUMN content_search_tsv TSVECTOR "
+        "GENERATED ALWAYS AS (to_tsvector('simple', COALESCE(content_search_text, ''))) "
+        "STORED"
+    )
+    op.execute(
+        "ALTER TABLE document_chunks ADD COLUMN path_search_tsv TSVECTOR "
+        "GENERATED ALWAYS AS (to_tsvector('simple', COALESCE(path_search_text, ''))) "
+        "STORED"
+    )
+    op.execute(
+        "CREATE INDEX IF NOT EXISTS idx_chunk_content_search_tsv "
+        "ON document_chunks USING GIN (content_search_tsv)"
+    )
+    op.execute(
+        "CREATE INDEX IF NOT EXISTS idx_chunk_path_search_tsv "
+        "ON document_chunks USING GIN (path_search_tsv)"
+    )

apps/api/app/api/v1/routes/retrieval.py

@@ -80,8 +80,8 @@ class RetrievalQueryRequest(BaseModel):
     use_agentic: bool | None = Field(
         None,
         description=(
-            "Map-nav (PLANNER+HARVEST+CONTROL) is the default when unset/true. "
-            "Set false to force classic 3-channel top-K retrieval."
+            "Agent explore (cursor_sdk harness by default) when unset/true. "
+            "Set false to force classic map-unit BM25 top-K retrieval."
         ),
     )
     conversation_id: str | None = Field(

apps/api/app/mcp/dynamic_tools.py

@@ -0,0 +1,80 @@
+"""Bridge ``shared.services.retrieval.agent_tools.REGISTRY`` onto FastMCP.
+
+FastMCP's public registration API (``FastMCP.tool`` / ``ToolManager.add_tool``)
+only builds a tool's schema by introspecting a Python function's *signature*
+(``mcp.server.fastmcp.tools.base.Tool.from_function`` -> ``func_metadata``);
+there is no public entry point to register a tool from an already-built JSON
+Schema dict, which is what every ``agent_tools.ToolSpec`` carries. Since our
+schema is the one already shipped to ``agent_explore`` and meant to be
+verbatim-identical across harnesses (see ``CORPUS_SCHEMA.md``), we construct
+``Tool`` objects directly instead of round-tripping through a synthetic
+Python function signature, and insert them into the tool manager's registry
+dict — the same dict ``ToolManager.__init__`` accepts a ``tools=`` list for,
+just with no public single-tool equivalent of that constructor path.
+"""
+
+from __future__ import annotations
+
+from typing import Any, AsyncContextManager, Callable
+
+from mcp.server.fastmcp import Context, FastMCP
+from mcp.server.fastmcp.tools.base import Tool
+from mcp.server.fastmcp.utilities.func_metadata import ArgModelBase, FuncMetadata
+from pydantic import create_model
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from shared.services.retrieval.agent_tools import REGISTRY, ToolContext, ToolSpec
+from shared.services.retrieval.agent_tools import tools as _agent_tools_registered  # noqa: F401
+
+DbFactory = Callable[[], AsyncContextManager[AsyncSession]]
+
+
+def _lenient_arg_model(spec: ToolSpec) -> type[ArgModelBase]:
+    """Build a permissive pydantic arg model for FastMCP's internal validation.
+
+    The schema actually exposed to MCP clients is ``spec.json_schema``
+    (``Tool.parameters``, returned verbatim in ``tools/list`` — see
+    ``FastMCP.list_tools``); this model only has to satisfy
+    ``FuncMetadata.call_fn_with_arg_validation`` well enough to forward
+    whatever the client sent through to ``_dispatch_tool``'s ``**kwargs``.
+    Every property is optional/``Any`` — each tool already validates its own
+    required args and reports a caller-facing ``ToolResult.error`` for a
+    missing one, so duplicating "required" enforcement here would just
+    produce a less informative MCP-level error instead.
+    """
+    properties = spec.json_schema.get("properties", {})
+    fields: dict[str, Any] = {key: (Any, None) for key in properties}
+    return create_model(f"{spec.name.replace('.', '_')}_Args", __base__=ArgModelBase, **fields)
+
+
+def _make_tool(spec: ToolSpec, *, db_factory: DbFactory) -> Tool:
+    async def _dispatch_tool(
+        ctx: Context | None = None, **kwargs: Any
+    ) -> dict[str, Any]:
+        from app.mcp.retrieval_server import resolve_mcp_namespace, resolve_mcp_user_id
+
+        namespace = resolve_mcp_namespace(ctx=ctx)
+        async with db_factory() as db:
+            user_id = await resolve_mcp_user_id(ctx=ctx, db=db)
+            tool_ctx = ToolContext(db=db, user_id=user_id, namespace=namespace)
+            result = await REGISTRY.dispatch(spec.name, tool_ctx, kwargs)
+        return {"text": result.text, "payload": result.payload, "refs": result.refs, "error": result.error}
+
+    return Tool(
+        fn=_dispatch_tool,
+        name=spec.name,
+        title=None,
+        description=spec.description,
+        parameters=spec.json_schema,
+        fn_metadata=FuncMetadata(arg_model=_lenient_arg_model(spec)),
+        is_async=True,
+        context_kwarg="ctx",
+        annotations=None,
+    )
+
+
+def register_corpus_tools(server: FastMCP, *, db_factory: DbFactory) -> None:
+    """Register every ``agent_tools.REGISTRY`` tool onto ``server``."""
+    for spec in REGISTRY.all():
+        tool = _make_tool(spec, db_factory=db_factory)
+        server._tool_manager._tools[tool.name] = tool

apps/api/app/mcp/retrieval_server.py

@@ -10,8 +10,10 @@
 from pydantic import Field
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.mcp.dynamic_tools import register_corpus_tools
 from shared.core.database import get_db_context
 from shared.models.schemas.retrieval_namespace import normalize_retrieval_namespace
+from shared.services.retrieval.agent_tools import load_corpus_schema_text
 from shared.services.retrieval.app_service import run_retrieval_query
 from shared.services.retrieval.settings import DEFAULT_TOP_K
 
@@ -85,16 +87,16 @@ def create_retrieval_mcp_server(
     server = FastMCP(
         "knowhere-retrieval",
         instructions=(
-            "Use this server to search published documents. "
-            "It returns evidence_text (hierarchical evidence tree), "
-            "referenced_chunks (structured chunk citations), and "
-            "decision_trace (navigation decisions). "
-            "Downstream agents should synthesize answers from evidence_text."
+            "retrieval.query is a one-shot legacy search tool retained for "
+            "backward compatibility (see its own tool description below). "
+            "Prefer the corpus.* tools for exploration; the schema below "
+            "describes what they operate on.\n\n" + load_corpus_schema_text()
         ),
         streamable_http_path=streamable_http_path,
         stateless_http=True,
         transport_security=create_public_mcp_transport_security(),
     )
+    register_corpus_tools(server, db_factory=db_factory)
 
     @server.tool(
         name="retrieval.query",

apps/api/main.py

@@ -104,6 +104,18 @@ async def _redis_ping() -> bool:
     mcp_server = getattr(app.state, "retrieval_mcp_server", None)
     mcp_session_manager = getattr(mcp_server, "session_manager", None)
 
+    from shared.services.retrieval.agent_explore.harness.resolve import (
+        resolve_harness_name,
+    )
+
+    if resolve_harness_name() == "cursor_sdk" and not os.environ.get(
+        "CURSOR_API_KEY", ""
+    ).strip():
+        logger.error(
+            "Default retrieval harness is cursor_sdk but CURSOR_API_KEY is unset; "
+            "agentic retrieval requests will fail until the key is set"
+        )
+
     logger.info("Document API service started!")
     if mcp_session_manager is not None:
         async with mcp_session_manager.run():

apps/api/pyproject.toml

@@ -23,6 +23,7 @@ dependencies = [
     "limits>=5.8,<6",
     "logfire[celery,fastapi,httpx,sqlalchemy]>=4.25.0",
     "mcp>=1.27.0",
+    "cursor-sdk>=1.0.31",
 ]
 
 [dependency-groups]

apps/api/scripts/backfill_map_unit_statistics.py

@@ -51,7 +51,7 @@ def _bootstrap_python_path() -> None:
     DocumentMapUnit,
     DocumentMapUnitIndex,
 )
-from shared.services.retrieval.nav.knowhere_hybrid import MAP_UNIT_INDEX_FORMAT_VERSION
+from shared.services.retrieval.scoring.knowhere_hybrid import MAP_UNIT_INDEX_FORMAT_VERSION
 
 
 @dataclass(frozen=True)