Sprint 4 — Smart indexing, streaming search, production hardening

[svalench]

Sprint 4 — Smart indexing, streaming, and production hardening

This PR is stacked on top of PR #3 (Sprint 3 conversational search). Merging PR #2 → PR #3 → this one delivers the full LangGraph integration described in the original plan.

What's new

Smart indexing (SMART_INDEXING.ENABLED)

• New SmartIndexer orchestration that builds structured documents with labelled sections (Title:, Description:, Category: …) so the embedder can distinguish field roles.
• Per-model templates via SMART_INDEXING.TEMPLATES, with a deterministic heuristic fallback when none is provided.
• Drop-in replacement: get_indexer() factory returns the right implementation based on settings. Signals, index(), and the build_search_index management command all use the factory now.
• Legacy whitespace-joined text is always appended as a safety net — smart indexing never produces less searchable content than the classic indexer.

Event hub & streaming endpoint

• events.EventHub — thread-safe, error-tolerant pub/sub used by the LangGraph nodes.
• Both real LangGraph and the in-tree fallback runner emit lifecycle events: query_received, query_expanded, vector_search_completed, rerank_completed, completed.
• New StreamingSearchAPIView at /api/search/stream/:
  • NDJSON (default) or SSE transport, configurable via STREAMING.FORMAT.
  • Runs the search in a worker thread; the request loop drains events through a queue.Queue so users see progress in real time.
  • Terminal end event gives clients a reliable stream-termination marker.
  • Sets X-Accel-Buffering: no so reverse proxies don't buffer the response.
• Returns HTTP 404 when STREAMING.ENABLED=False, so the URL is safe to leave registered.

Production hardening

• All new settings are validated in _build_*_config helpers and stored in frozen dataclasses (SmartIndexingConfig, StreamingConfig).
• STREAMING.FORMAT must be "ndjson" or "sse"; invalid values raise ConfigurationError at boot.
• Searcher forwards event_hub to the graph factory only when supported (TypeError-tolerant), so custom graph factories don't have to be updated.
• No new hard dependencies — LangGraph and LLM SDKs remain optional.

Backward compatibility

• All flags default to False. Existing users see zero behavioural change.
• Public surface (Searcher.search, Searcher.find_similar, Indexer, REST endpoints) is unchanged.
• Re-indexing is not required to switch smart indexing on or off — old Document.text payloads continue to work.

Tests

• +22 tests (10 smart indexing, 12 events/streaming).
• Total: 58 tests pass (8 original + 13 PR feat(langgraph): optional LangGraph search pipeline (Sprint 1+2) #2 + 15 PR feat(conversational): session-aware conversational search (Sprint 3) #3 + 22 PR Sprint 4 — Smart indexing, streaming search, production hardening #4).

tests/test_api.py ............................. PASS
tests/test_backends.py ....................... PASS
tests/test_conversational_search.py .......... PASS (16)
tests/test_embedding_selection.py ............ PASS
tests/test_events_streaming.py ............... PASS (12) NEW
tests/test_indexer.py ......................... PASS
tests/test_langgraph_search.py ................ PASS (13)
tests/test_resolver.py ........................ PASS
tests/test_smart_indexing.py .................. PASS (10) NEW

Quick demo

GRAPH_SEARCH = {
    # ... your existing config ...
    "SMART_INDEXING": {
        "ENABLED": True,
        "TEMPLATES": {
            "shop.Product": {
                "title_field": "name",
                "sections": [
                    {"label": "Description", "field": "description", "multiline": True},
                    {"label": "Category", "field": "category__name"},
                ],
            }
        },
    },
    "STREAMING": {"ENABLED": True, "FORMAT": "ndjson"},
}

$ curl -N "http://localhost:8000/api/search/stream/?q=phone"
{"type": "query_received", "query": "phone"}
{"type": "vector_search_completed", "candidate_count": 12}
{"type": "completed", "total": 5}
{"type": "results", "results": [...], "total": 5}
{"type": "end"}

Merge order

1. #2 (Sprint 1+2) → main
2. #3 (Sprint 3) → main
3. This PR (Sprint 4) → main

Or merge into the corresponding base branches in stacked order if preferred.