Skip to content

feat: add fastCRW as a web search source - #37

Open
us wants to merge 1 commit into
magnitudedev:mainfrom
us:feat/crw-web-search
Open

us wants to merge 1 commit into
magnitudedev:mainfrom
us:feat/crw-web-search

Conversation

@us

@us us commented Aug 12, 2026

Copy link
Copy Markdown

why

web_search currently has one working source. in
packages/sdk/src/provider-client.ts:

// Cloud is disabled.
const webSearchSource: WebSearchSource = exaInstance.configured
  ? "exa"
  : "unavailable"

without EXA_API_KEY the tool fails with WebSearchNotConfigured, and
docs/get-started.mdx sends people to exa.ai to buy a key.

after the move to fully local, web search is the one part of the agent that
still needs a paid third-party account.

this adds a second source: fastCRW, an AGPL-3.0
search engine that runs as a single static binary. self-hosted it needs no key
and no account, so web_search works with nothing but a base url.

disclosure up front: i maintain fastCRW.

what changed

follows packages/providers/AGENTS.md and mirrors the exa module:

  • packages/providers/src/crw/{contract.ts,web-search.ts} implementing the
    existing WebSearchExtension from packages/ai
  • "crw" added to WebSearchProviderSchema; providerName became a lookup
  • crwApiKey / crwBaseUrl on ProviderClientConfig, wired into
    createProviderClient()
  • source selection refactored from nested ternaries to an ordered
    first-configured pick, so precedence is stated once and a fourth source is a
    one-line change
  • tests: crw/contract.test.ts + crw/web-search.test.ts (8 cases), all
    against a local Bun.serve fixture, no network and no key
  • docs, env table, changeset

two changes outside the new module, both deliberate

1. the ambient availability schema now derives from WebSearchProviderSchema.
packages/agent/src/ambient/tool-availability-ambient.ts restated
Schema.Literal('magnitude','exa'), so adding a provider broke
packages/acn typecheck. i pointed it at the existing schema instead of typing
a third literal, otherwise the next provider breaks the build the same way.

this is persisted event-sourced state, so flagging it rather than burying it:
the union only widens, so previously written events still decode. if you would
rather own that change separately i can drop it and you can take it on your own
terms, but then this PR cannot go green.

2. a new WebSearchStructuredOutputUnsupported error. see below.

the structured-output gap

webSearch(query, schema) takes an optional schema, which exa serves via
outputSchema. fastCRW's search endpoint has no equivalent.

returning results with data absent looked like the wrong call: the tool
declares data as optional output, so the model gets a clean success envelope
with no data and cannot tell "nothing matched" from "this source cannot do
that", which usually turns into a retry loop or an invented value.

so it fails loudly with a typed error naming the reason. if you would rather it
degrade quietly and be documented instead, say so and i will switch it.

behaviour

  • CRW_API_URL=http://localhost:3000 and nothing else: works, no key, and no
    Authorization header is sent at all
  • CRW_API_KEY=...: uses the hosted engine
  • both exa and crw configured: exa wins, existing installs unchanged
  • neither: unavailable, exactly as today

one thing worth knowing: CRW_API_KEY / CRW_API_URL are also fastCRW's own
CLI config vars, so a user who has run crw setup already has them exported.
that means web search can become available without them setting anything for
magnitude specifically. i think that is the behaviour you want given exa reads
EXA_API_KEY the same way, but it is a wider blast radius than exa's
single-purpose variable, so it is your call. every routing test pins all
sources explicitly so the suite does not depend on ambient env.

verification

self-host route prefix, against a locally running engine:

$ crw serve --port 3111
$ curl -s -o /dev/null -w "%{http_code}\n" -X POST http://localhost:3111/v1/search \
    -H 'Content-Type: application/json' -d '{"query":"test","limit":2}'
422        # route exists; 422 because no search backend is configured on that box
$ curl -s -o /dev/null -w "%{http_code}\n" -X POST http://localhost:3111/api/v1/search \
    -H 'Content-Type: application/json' -d '{"query":"test","limit":2}'
404        # confirming the base url in the docs carries no path suffix
  • packages/providers: 174 passed (14 new, across contract and web-search)
  • packages/sdk: 56 passed, including 5 source-selection cases
  • packages/agent web-search tool test: passed
  • both suites re-run with CRW_API_KEY / CRW_API_URL exported, still green
  • typecheck: cli, packages/sdk, packages/acn, packages/icn-protocol,
    packages/release all clean
  • end to end against a live engine: 9 sources mapped into WebSearchResult; a
    schema request returns WebSearchStructuredOutputUnsupported

note: bun run --cwd packages/icn typecheck fails on
src/lifecycle/cuda-compatibility.test.ts(38,3), but it fails identically with
this branch stashed, so it is not from this change.

scope

only web_search. i measured web_fetch too and on pages a plain GET can
already fetch, your dom-extract output beats what i get back and is roughly
ten times faster, so i am not proposing to touch it.

happy to close this if a third-party search source is not a direction you want.

Web search resolved to Exa only, so it required a paid third-party key.
Add fastCRW alongside it, which needs no key when self-hosted.

- new provider module implementing the existing WebSearchExtension
- "crw" added to WebSearchProviderSchema; providerName became a lookup
- crwApiKey / crwBaseUrl on ProviderClientConfig
- source selection refactored to an ordered first-configured pick so
  precedence is stated once
- tool availability ambient derives its source union from
  WebSearchProviderSchema instead of restating the literal
- WebSearchStructuredOutputUnsupported so a schema request fails loudly
  rather than returning a success envelope with no data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant