[RFC] Pluggable Web Search Providers and Typed Result Normalization

[Zheng-Lu]

Problem statement / motivation

Summary

The gateway-owned web_search_preview tool is hardwired to You.com. This RFC proposes making the search
backend selectable through a typed WebSearchProviderKind config enum, adding Brave Search as the first
alternative provider, and — as a prerequisite — replacing the current untyped pass-through of provider JSON with
a typed normalization contract (WebSearchResult). The work is split into two PRs: a behavior-preserving refactor,
then the Brave provider with configuration, tests, and docs. SearXNG (keyless, self-hosted) and Tavily follow as
separate PRs against the same contract.

The design keeps the existing WebSearchProvider trait private, keeps the public WebSearchHandler constructors
unchanged, and adds only additive configuration. When no provider is selected, behavior is identical to today.

Motivation

web_search_preview is executed by the gateway (ARCHITECTURE.md, docs/design/tool-framework.md), which is a
real differentiator for a self-hosted, OpenAI-compatible stack. Today it can only be enabled by setting
YOU_API_KEY and YOU_API_BASE_URL
(crates/agentic-server-core/src/tool/web_search.rs:20-21, docs/deploying/kubernetes.md:419-420).

PR #85 (@franciscojavierarceo) anticipated this:

Splits the You.com HTTP integration behind an internal WebSearchProvider abstraction so additional search
providers can be added without changing the OpenAI tool adapter.

The abstraction exists (web_search.rs:253-260), but nothing else was ever plugged into it. Practical consequences:

• Barrier to entry. As of this writing, You.com's Search API is a paid product without a self-serve free
  developer tier. Brave Search offers a free plan (≈2,000 queries/month), Tavily ≈1,000/month. A personal or
  evaluation deployment of Agentic API cannot exercise web_search at all without a commercial contract.
• Air-gapped / private-cloud deployments cannot use web_search because the only supported backend is a public
  SaaS endpoint. A SearXNG instance (or an internal search API behind the same contract) is the realistic answer for
  those environments.
• Vendor coupling in the tool contract. The function schema shown to the model, the argument struct, and the
  model-facing tool output are all You.com-shaped (details below), so even an embedder who implements their own
  executor inherits You.com semantics.

Goals

1. Select the search backend by configuration; you remains the default.
2. Add Brave Search as the first alternative provider.
3. Define a typed, provider-neutral result contract so no provider builds serde_json::json! blobs.
4. Keep every existing public constructor, env var, config key, and public output item working unchanged.
5. Keep the provider trait private; extension for embedders remains GatewayExecutorRegistration::WebSearch.

Non-goals

• Pagination (not exposed by the function schema or search_context_size).
• Automatic retries on HTTP 429 (see Concurrency and rate limits).
• A public plugin ABI for providers.
• Query rewriting with site: operators to emulate domain filters (possible later optimization; Phase 1 post-filters).
• Cargo features per provider. All providers compile into the single binary.

Proposed solution

Implementation plan

Phrase 1 — refactor and typed contract (no configuration or public API change)

• Split crates/agentic-server-core/src/tool/web_search.rs (807 lines) into
  tool/web_search/{mod.rs, args.rs, you.rs}: mod.rs holds the handler, trait, typed result, and public output
  mapping; args.rs holds WebSearchArguments, Freshness, and the domain post-filter; you.rs holds the You.com
  provider and request shaping.
• Introduce WebSearchResult, WebSearchProviderMetadata, typed WebSearchProviderResponse; the You.com provider
  deserializes into them.
• Add WebSearchProviderKind with only You, WebSearchProviderConfig.provider, and
  WebSearchHandler::from_config; wire ToolExecutors::from_config.
• Add a test that serializes the handler output for the existing mock fixture
  (tests/web_search_tool_test.rs:291) and asserts it byte-for-byte; add a redacted real You.com response as a
  fixture so the mapping is checked against actual upstream shape, not only the hand-written mock.
• Existing unit and integration tests pass unmodified.

Phrase 2 — Brave Search provider

• tool/web_search/brave.rs (~300 lines): request shaping, minimal response structs, mapping, domain post-filter
  application, count clamp, freshness rendering, max_concurrent_requests() = 1.
• WebSearchProviderKind::Brave; file/env/generated-config plumbing in config_file.rs and main.rs.
• Tests (see below), docs, CHANGELOG.md.

Roadmap (separate RFC-lite issues, same contract)

1. SearXNG — keyless, self-hosted, GET /search?format=json&categories=general,news&time_range=…. This is the
   air-gapped story and the proof that the abstraction handles a provider with no credential and heterogeneous
   result quality.
2. Tavily — native include_domains / exclude_domains, LLM-oriented content; news is a separate
   topic=news request, which exercises the "one request per query" policy differently.
3. DuckDuckGo is not planned: there is no official web search API, and scraping is out of scope for this project.

Testing

All provider tests run against a local Axum mock bound to 127.0.0.1:0, following spawn_mock_you_with_response
(tests/web_search_tool_test.rs:291-322). No external network access in CI. No new replay cassettes are needed.

Phrase 2 coverage:

• 200 with mixed web + news → both sections mapped; sources derived from both.
• Empty results → empty sections, no error.
• 401 / 403 → failed web_search_call, message names the key env var, credential absent from output.
• 429 with Retry-After → failed call, no retry (single captured request), header value in message.
• Query-parameter assertions: q, count (clamped from 50 → 20), freshness=pw, country, search_lang,
  safesearch, result_filter=web,news; X-Subscription-Token present; Accept-Encoding absent.
• Domain post-filter: allowlist, blocklist, subdomain match, label-boundary negative case, uppercase host, host with
  trailing dot, unparsable URL dropped.
• Concurrency ceiling: reuse the ConcurrencyTrackingProvider pattern to assert max_active == 1 for Brave under a
  five-query batch even when the gateway limit is 5.
• Config: AGENTIC_WEB_SEARCH_PROVIDER=Brave parses case-insensitively; unknown value fails startup;
  [web_search] provider round-trips through FileConfig; generated file contains provider = "you".
• Mock hygiene: use try_send (or hold the receiver for the test's lifetime) instead of
  tx.send(..).await.unwrap() inside handlers (tests/web_search_tool_test.rs:311), to avoid the class of hang fixed
  in CHANGELOG.md:51.

Gates: cargo test, cargo clippy --all-targets -- -D warnings (pedantic is warn in Cargo.toml and is
promoted by -D warnings), cargo fmt -- --check, pre-commit run --all-files.

Additional context

Open questions for maintainers

1. page_age vs published_at. Keeping page_age makes PR 1 byte-identical for You.com; published_at is
   provider-neutral. We propose published_at and would like a decision before PR 1.
2. Default base URL for You.com. Adding provider.default_base_url() for You.com means a config that fails
   today (YOU_API_KEY set, YOU_API_BASE_URL unset) starts working. Acceptable? If so, which host is canonical —
   README.md:219 says https://api.ydc-index.io, docs/deploying/kubernetes.md:420 says https://ydc-index.io.
3. Base URL override for non-You providers: a generic AGENTIC_WEB_SEARCH_BASE_URL, or per-provider
   BRAVE_API_BASE_URL mirroring YOU_API_BASE_URL? We lean generic, with YOU_API_BASE_URL retained as-is.
4. Location of WebSearchProviderKind: config.rs (proposed, alongside SqliteTempStore) or
   tool/web_search/mod.rs with a re-export?
5. Should the provider name be surfaced to the model in metadata[] (proposed) or kept out of the model-facing
   output?

References

• PR feat: add web search gateway tool #85 — feat: add web search gateway tool (commit bf5fe8b).
• docs/design/tool-framework.md — tool ownership model; web_search is gateway-owned.
• docs/design/messages-gateway-tool-classification.md — Claude Code WebSearch aliasing into web_search.
• crates/agentic-server-core/tests/cassettes/README.md — recorder workflow (no new cassettes required here).

[franciscojavierarceo]

Very much welcome this extension. And more generally we welcome every component to be pluggable. We've even extended the inference engine to be luggable!