feat: add Files and Vector Stores APIs with in-tree file search

[franciscojavierarceo]

Summary

Integrate with main's AgentPipeline refactor (#274). Keep per-round citation evidence in the owned translation context, assemble native file-search items through the typed accumulator slots, and preserve already-public citations during strict JSON/SSE decoding. Regression tests cover citation preservation and lenient done-only file-search items.

Stack layer 1 of 7: #34 foundation → #278 pgvector/CI → #280 contextual retrieval/reranking/configuration → #281 streaming Files → #282 store lifecycle/content → #283 durable batches → #284 streaming/SDK compatibility.

Base: main. Next: #278. Review against the immediate base and land from the bottom of the stack.

Adds the OGX file-ingestion and retrieval flow directly to Agentic API. Clients can upload documents, attach them to vector stores, and use the Responses file_search built-in tool without running OGX or a Python service.

• Adds 14 authenticated Files and Vector Stores API operations with typed OpenAPI contracts: upload, list, retrieve, download, delete, create stores, attach/detach files, and search. SQLite/PostgreSQL retains metadata and search data through migration 0005_file_search.sql.
• Stores uploaded bytes on the local filesystem, configured with [files] storage_dir or AGENTIC_FILES_STORAGE_DIR (default $AGENTIC_API_HOME/files). Binary upload/download is independent of ingestion format support. New uploads use generated IDs and bounded, atomic file publication with cancellation/error cleanup. Earlier draft uploads stored inline in SQL remain readable.
• Implements overlapping token chunks, an OpenAI-compatible embeddings client, cosine semantic retrieval, BM25 keyword retrieval, hybrid rank fusion, typed attribute filters, and global result limits. Embeddings are configured separately from generation; keyword retrieval works without an embeddings endpoint.
• Integrates file_search_call lifecycle events, opt-in results, file citations, public tool-choice selectors, and private stored continuation with the existing Responses executor. Preserves the shell-tool implementation from current main.
• Bounds ingestion, parsing, provider responses, and retrieval. Each store supports up to 10,000 chunks and 64 MiB of serialized chunk data. Text ingestion is available by default; bounded PDF ingestion is enabled with --features file-search-pdf and requires Rust 1.88 or newer.
• Documents setup, API examples, migration requirements, and limits in docs/api/file-search.md. Container deployments need a persistent files volume; replicas sharing a database must also share the files directory. Exact retrieval in this foundation targets small corpora. Indexed retrieval, contextual chunking, query rewriting, neural reranking, and asynchronous batches are developed in the dependent stack; arbitrary OGX provider catalogs remain outside this foundation.

Test Plan

Layer-specific implementation verification:

• Passed cargo test --workspace --all-features --locked, including filesystem-backed HTTP upload/attach/search/download/delete, binary-file round trips, detachment/store-deletion preserving uploads, semantic retrieval without lexical overlap, failure rollback, pagination, filters, persistence, streaming/blocking tool use, citations, continuation, shell regressions, and OpenAPI schemas.
• Passed the default-feature file-search service tests, including filesystem persistence, rollback/cancellation cleanup, path and symlink handling, oversized-file rejection, and legacy inline-data reads. PostgreSQL tests require the isolated CI database.
• Passed cargo clippy --workspace --all-targets --all-features --locked -- -D warnings, formatting, git diff --check, and all pre-commit hooks.
• Independent review and re-review completed. Added regression coverage for long-input cancellation, backward pagination, transactional capacity limits, and built-in tool selectors.
• PostgreSQL tests require an isolated database and are wired into the PostgreSQL CI job. The original foundation revision passed its CI checks, including PostgreSQL, Rust, pre-commit and the agent launcher integrations. Local Docker could not create its temporary filesystem mount during foundation validation. The bounded-PDF extraction and decompression-limit tests passed locally and are also included in CI.

After synchronization with #274: The updated foundation passed 657 core/file-search/stream/shell tests, all-target all-feature workspace clippy, formatting, diff checks, and all-files pre-commit hooks. Scoped review and re-review passed. The combined stack in #284 passed 1,389 workspace tests, 31 PostgreSQL/pgvector tests, and 17 maintained SDK cases, plus the Rust 1.88 all-feature check and all-target workspace clippy. The pipeline integration review is complete. GitHub CI is rerunning for the updated heads; previous green checks apply to the prior revisions.

[ashwing]

@franciscojavierarceo Is this still in-progress or ready for review? Happy to take a look when you're ready — especially interested in how the OGX VectorSearch trait maps to the dispatch interface we're building.

[franciscojavierarceo]

@ashwing I'm waiting for #46 to land to incorporate it and then mark as ready for review as I need the agentic loop and to update the Responses structs.