feat: add contextual retrieval, reranking and vector store configuration

[franciscojavierarceo]

Summary

Synchronize this layer with its updated parent after main's AgentPipeline refactor (#274), preserving the immediate-base review boundary.

Stack layer 3 of 7: #34 foundation → #278 pgvector/CI → #280 contextual retrieval/reranking/configuration → #281 streaming Files → #282 store lifecycle/content → #283 durable batches → #284 streaming/SDK compatibility.

Base: codex/file-search-pgvector (#278). Next: #281. Review against the immediate base and land from the bottom of the stack.

Adds model-assisted retrieval to in-tree file search. Contextual chunking generates document-aware context before embedding while preserving original passages for citations. Neural/classifier rerankers score bounded candidates before final result limits and thresholds; query rewriting searches and returns the actual rewritten query.

A typed VectorStoresConfig groups provider/model choices, ingestion/retrieval defaults, contextual settings, rewrite settings and batch settings. Models resolve through configured providers with separate credentials; existing embedding settings and keyword-only setups remain supported. Direct Vector Stores search and the Responses file-search tool share the pipeline.

Reranking uses the vLLM/Cohere text protocol. Probability/logit interpretation is explicit; invalid scores/indexes and malformed provider output fail clearly. Contextual failures do not publish partial ingestion. Configurable hybrid fusion and documented OpenAI dated ranker selectors are supported, without claiming equality with OpenAI's proprietary models. OGX-specific configuration/chunking/ranker options are documented extensions.

Test Plan

Layer-specific implementation verification:

• Local model HTTP fixtures cover provider credentials, model allowlists, contextual embedding/original text, rewrite, reranker selection, malformed results, zero-score promotion, timeout, cancellation and bounded request bodies. Final model/tool covering run: 21 passed; core file-search unit run: 31 passed.
• Before the scoped review fixes, the all-feature service/model/tool run passed all 52 tests, including real PostgreSQL/pgvector and PDF cases. The fixes have red/green regressions for repeated prompt placeholders, bounded serialization, the documented hyphenated ranker selector and queued context-worker admission after cancellation.
• Configuration: 18 passed. HTTP/OpenAPI: 10 passed. All-target, all-feature clippy for both changed crates and all applicable pre-commit hooks passed, including formatting.
• Retrieval tests use local deterministic model fixtures; no hosted model quality claim. Integrated SDK contracts and final workspace verification are recorded in feat: complete file-search streaming and API compatibility #284.

After synchronization with #274: This updated layer passed locked all-feature workspace compilation, diff checks, and all-files pre-commit hooks. The combined stack in #284 passed 1,389 workspace tests, 31 PostgreSQL/pgvector tests, and 17 maintained SDK cases, plus the Rust 1.88 all-feature check and all-target workspace clippy. The pipeline integration review is complete. GitHub CI is rerunning for the updated heads; previous green checks apply to the prior revisions.