Problem
When scraping DuckDuckGo (e.g. html.duckduckgo.com or duckduckgo.com) via plain HTTP requests (Tier 1) or through the MITM proxy (:8192 Tier 0), DuckDuckGo often serves an anti-bot anomaly challenge (HTTP/2 202 Accepted or HTTP 200 with <form id="challenge-form" action="//duckduckgo.com/anomaly.js..."> and <div data-testid="anomaly-modal">).
Because detect.ts does not recognize DuckDuckGo's anomaly challenge markers, isChallengeWall() returns false. Consequently:
- Tier 1 treats the response as a successful fetch (
status: "success") and returns the challenge form HTML rather than escalating to Tier 2/3.
- The MITM proxy (
directForward.ts) leaves challengeDetected: false and proxies the anomaly challenge back to the downstream client (e.g. SearXNG, Prowlarr, or scrapers).
When the request is handled by Camoufox in Tier 3, DuckDuckGo executes stealth fingerprinting and returns the real search results cleanly without triggering the anomaly gate.
Proposed solution
Add conservative DuckDuckGo anomaly challenge detection matching the established patterns in packages/tiers/src/utils/detect.ts (similar to hasDdosGuardChallenge and hasAkamaiChallenge):
- Detect
action="//duckduckgo.com/anomaly.js" or id="challenge-form" within anomaly-modal, or anomaly.js / anomaly-modal elements.
- Ensure bare domain mentions (
duckduckgo.com) or ordinary search result pages are not classified as challenges.
- In
isChallengeWall(), classify the anomaly challenge as a challenge wall.
- In
tiers/1.ts and directForward.ts, escalate detected challenges to needs-js / browser tiers.
Alternatives considered
- Relying on
MITM_PROXY_ALWAYS_SCRAPE: skips Tier 0 but Tier 1 still receives the anomaly page without escalating to Tier 3 unless the challenge is recognized.
- Solving the challenge client-side: inefficient and fragile compared to letting Camoufox handle the navigation natively in Tier 3.
Tier impact
Fits inside existing detection and Tier 1/proxy escalation (packages/tiers/src/utils/detect.ts, packages/tiers/src/tiers/1.ts, apps/api/src/proxy/directForward.ts).
Willingness to contribute
Additional context
Verified against live endpoints:
- Plain HTTP POST to
https://html.duckduckgo.com/html/ triggers HTTP/2 202 Accepted with form action //duckduckgo.com/anomaly.js and data-testid="anomaly-modal".
- Tier 3 with Camoufox returns real search results with
HTTP 200 OK.
Problem
When scraping DuckDuckGo (e.g.
html.duckduckgo.comorduckduckgo.com) via plain HTTP requests (Tier 1) or through the MITM proxy (:8192Tier 0), DuckDuckGo often serves an anti-bot anomaly challenge (HTTP/2 202 AcceptedorHTTP 200with<form id="challenge-form" action="//duckduckgo.com/anomaly.js...">and<div data-testid="anomaly-modal">).Because
detect.tsdoes not recognize DuckDuckGo's anomaly challenge markers,isChallengeWall()returnsfalse. Consequently:status: "success") and returns the challenge form HTML rather than escalating to Tier 2/3.directForward.ts) leaveschallengeDetected: falseand proxies the anomaly challenge back to the downstream client (e.g. SearXNG, Prowlarr, or scrapers).When the request is handled by Camoufox in Tier 3, DuckDuckGo executes stealth fingerprinting and returns the real search results cleanly without triggering the anomaly gate.
Proposed solution
Add conservative DuckDuckGo anomaly challenge detection matching the established patterns in
packages/tiers/src/utils/detect.ts(similar tohasDdosGuardChallengeandhasAkamaiChallenge):action="//duckduckgo.com/anomaly.js"orid="challenge-form"withinanomaly-modal, oranomaly.js/anomaly-modalelements.duckduckgo.com) or ordinary search result pages are not classified as challenges.isChallengeWall(), classify the anomaly challenge as a challenge wall.tiers/1.tsanddirectForward.ts, escalate detected challenges toneeds-js/ browser tiers.Alternatives considered
MITM_PROXY_ALWAYS_SCRAPE: skips Tier 0 but Tier 1 still receives the anomaly page without escalating to Tier 3 unless the challenge is recognized.Tier impact
Fits inside existing detection and Tier 1/proxy escalation (
packages/tiers/src/utils/detect.ts,packages/tiers/src/tiers/1.ts,apps/api/src/proxy/directForward.ts).Willingness to contribute
Additional context
Verified against live endpoints:
https://html.duckduckgo.com/html/triggersHTTP/2 202 Acceptedwith form action//duckduckgo.com/anomaly.jsanddata-testid="anomaly-modal".HTTP 200 OK.