feat(tool): add SearXNG as a selectable web_search provider (#326)

[Zheng-Lu]

Closes #326 (Part of #291).

Summary

Adds a self-hosted, keyless SearXNG backend for the gateway-owned web_search tool so air-gapped and privacy-sensitive deployments can run web search without any direct client queries leaving the local network.

Key Changes

• Provider Implementation (crates/agentic-server-core/src/tool/web_search/searxng.rs):

  • Hits GET {base_url}/search with format=json&categories=general,news, Accept: application/json, and optional Authorization: Bearer from SEARXNG_API_KEY.
  • Base URLs must be absolute http(s) without a query or fragment; validated at startup and again by the provider, with /search built from the parsed path so sub-path mounts (e.g. http://host/searxng/) work.
  • Strictly adheres to transport rules: Accept-Encoding: gzip is never sent (core reqwest lacks gzip support).
  • Single HTTP request per query; hits are split into web and news sections based on each result item's category.

• Input & Output Adaptations (verified against SearXNG master):

  • freshness: Maps named variants to time_range=day|week|month|year (date ranges are ignored with a debug log).
  • language: Normalized to SearXNG's accepted ^[a-z]{2,3}(-[A-Z]{2})?$ format so complex BCP 47 subtags (e.g. zh-Hans) do not trigger a 400 error.
  • safesearch: Mapped to numeric strings 0|1|2.
  • Domain filtering: Enforces allowlists and blocklists client-side via the shared DomainFilter.
  • count: Applied client-side after domain filtering (falling back to search_context_size), as SearXNG only has pagination (pageno).
  • Unsupported args (country, boost_domains, You.com crawl args) are safely ignored with debug logging.

• Actionable Diagnostics (Credential-Free):

  • 403 → Points to JSON format disabled in SearXNG (search.formats: [html, json]).
  • 429 → Addresses SearXNG's bot-detection limiter (which blocks requests without gzip), guiding operators to configure botdetection.ip_lists.pass_ip with the gateway's source Pod IP / Pod CIDR, or set server.limiter: false, and echoes Retry-After.
  • 401 → Mentions SEARXNG_API_KEY without echoing secrets.
  • 400 → Parameter error with upstream body.
  • 2xx non-JSON → Hints at misconfigured HTML endpoint. Nothing is automatically retried.

• Configuration & Fail-Fast Validation:

  • Adds WebSearchProviderKind::Searxng ("searxng") and conventional SEARXNG_API_KEY.
  • Base URL is mandatory: resolve_web_search_config validates and rejects missing, non-HTTP(S), or query/fragment-bearing URLs at startup. Direct provider construction falls back to the same SEARXNG_BASE_URL_HINT.

• Architecture, File Sizes & Non-Breaking API:

  • Preserves all existing public constructors and method signatures (WebSearchProviderConfig builders and from_config are reused as-is).
  • Moves shared provider helpers (null_as_default, read_response_limited) from mod.rs to provider.rs with re-exports, keeping mod.rs well within the 500-line limit (473 lines, chore: enforce Rust production file size limits #319) without modifying baseline policies.
  • Zero new external dependencies. Adds "SearXNG" to clippy.toml doc-valid-idents.

• Documentation:

  • Fully updated README.md (quick start, provider table, SearXNG privacy boundary & limiter caveats), docs/deploying/README.md, docs/deploying/kubernetes.md (correct Pod CIDR guidance), CHANGELOG.md, and ARCHITECTURE.md.

Test Plan

• Local Verification & Lints:

  • cargo fmt -- --check — passes.
  • cargo clippy --all-targets -- -D warnings — passes with 0 warnings.
  • cargo test --workspace — all workspace unit and integration tests pass (0 failures).
  • pre-commit run --all-files (including scripts/check_rust_file_sizes.py: mod.rs 473, searxng.rs 409, config.rs 396 prod lines) — all hooks pass.

• Hermetic Integration Suite (crates/agentic-server-core/tests/web_search_searxng_test.rs):

  • 14 comprehensive Axum mock integration tests on 127.0.0.1:0:
    • Web and news result mapping with "provider": "searxng" in metadata[].
    • Exact request framing (format=json, categories, time_range, language, numeric safesearch, no Accept-Encoding, no Authorization when keyless).
    • Sub-path base URL routing (e.g. http://host/searxng/search).
    • Rejection of query/fragment base URLs without sending network requests.
    • Bearer authentication when SEARXNG_API_KEY is configured.
    • Graceful empty/null/missing results handling.
    • Client-side allowlist and blocklist domain filtering.
    • Filter-then-truncate count behavior and context size defaulting.
    • Diagnostics for 403, 401, 429 (asserts single request with Retry-After), 400, 502, and HTML responses without leaking secrets.
    • Mandatory base URL validation errors.
    • In-flight concurrency limits and override behavior.

• Unit Tests:

  • 14 inline unit tests in searxng.rs testing pure mapping helpers (language normalization, category splitting, param rendering, URL validation).
  • Updated existing config round-trip tests for the new searxng provider kind.

• Zero Regressions:

  • Existing You.com and Brave Search integration test suites (web_search_tool_test.rs, web_search_brave_test.rs) pass without regression.