Make /api/search honest: explain empty results + clear config errors - #16
Merged
Conversation
/api/search returned {success:true, count:0, results:[]} with no indication
of why — you couldn't tell "no hits" from "keyless provider blocked" from
"provider not configured". The root cause of the empty results in production:
DuckDuckGo's keyless endpoint anti-bot-challenges datacenter/server IPs (HTTP
202), confirmed from two independent datacenter IPs. That's a DDG policy, not
a parser bug — it can't be fixed into working keyless from a hosted API.
So make the endpoint transparent rather than pretend:
- search.service: new `runSearch()` returns { results, provider, reason? }; the
per-provider fetchers now report an actionable `reason` (DDG 202 challenge,
request failure, …). `searchWeb()` kept as a results-only wrapper so existing
callers/tests are unchanged. New `emptyResultHint(provider)` gives the
provider-specific explanation + the real fix.
- search route: response now includes `provider` and, when empty, a `note`
saying why (e.g. the DDG datacenter block, pointing at SERPER_API_KEY /
SEARXNG_URL). Missing provider config now returns a 400 with the actionable
message instead of an opaque 500.
Reliable search still requires config — but it's free: SERPER_API_KEY (free
tier at serper.dev, auto-selected when present) or a self-hosted SEARXNG_URL.
Both already worked; now the endpoint tells you so.
Docker end-to-end verified: default provider -> 200 + note explaining the 202
block; provider=searxng w/o URL -> 400 with the fix; SEARXNG_URL configured ->
200 with real results (provider:searxng, count:2). tsc, eslint, 204 tests
(+3), openapi:check all pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The real problem
/api/searchreturned{success:true, count:0, results:[]}for every query — with no indication of why. You couldn't tell "no hits" from "provider blocked" from "provider not configured". That silent empty is what wasted debugging time.Root cause of the empty results in production: the default keyless provider (DuckDuckGo's HTML endpoint) anti-bot-challenges datacenter / server IPs — HTTP 202. I confirmed this from two independent datacenter IPs (the VPS and this sandbox), on both
html.andlite.endpoints. It's a DuckDuckGo policy, not a parser bug — the parser is fine, DDG just refuses to serve hosted IPs. So it genuinely cannot be "fixed" into working keyless from a server.What this PR does — make it transparent, not pretend
search.service: newrunSearch()returns{ results, provider, reason? }. Each provider fetcher now reports an actionablereason(DDG 202 challenge, request failure…).searchWeb()is kept as a results-only wrapper, so existing callers/tests are untouched. NewemptyResultHint(provider)gives the provider-specific explanation and the real fix.providerand, when empty, anoteexplaining why — e.g. "DuckDuckGo returned an anti-bot challenge (HTTP 202)… set SERPER_API_KEY (free at serper.dev) or SEARXNG_URL." Missing provider config now returns a 400 with the actionable message instead of an opaque 500.Reliable search still needs config, but it's free and already supported:
SERPER_API_KEY(Serper.dev free tier, auto-selected when present) or a self-hostedSEARXNG_URL.Verification (Docker, end-to-end)
200·provider:duckduckgo·count:0·noteexplaining the 202 block + fixprovider:searxng, noSEARXNG_URL400with the actionable message (was an opaque 500)SEARXNG_URLconfigured (mock)200·provider:searxng·count:2real results flow throughtsc✅ ·eslint✅ · 204 tests ✅ (+3 new) ·openapi:check✅🤖 Generated with Claude Code