Skip to content

[feat] Detect DuckDuckGo anomaly challenge and escalate to browser tier #119

Description

@nklowns

Problem

When scraping DuckDuckGo (e.g. html.duckduckgo.com or duckduckgo.com) via plain HTTP requests (Tier 1) or through the MITM proxy (:8192 Tier 0), DuckDuckGo often serves an anti-bot anomaly challenge (HTTP/2 202 Accepted or HTTP 200 with <form id="challenge-form" action="//duckduckgo.com/anomaly.js..."> and <div data-testid="anomaly-modal">).

Because detect.ts does not recognize DuckDuckGo's anomaly challenge markers, isChallengeWall() returns false. Consequently:

  1. Tier 1 treats the response as a successful fetch (status: "success") and returns the challenge form HTML rather than escalating to Tier 2/3.
  2. The MITM proxy (directForward.ts) leaves challengeDetected: false and proxies the anomaly challenge back to the downstream client (e.g. SearXNG, Prowlarr, or scrapers).

When the request is handled by Camoufox in Tier 3, DuckDuckGo executes stealth fingerprinting and returns the real search results cleanly without triggering the anomaly gate.

Proposed solution

Add conservative DuckDuckGo anomaly challenge detection matching the established patterns in packages/tiers/src/utils/detect.ts (similar to hasDdosGuardChallenge and hasAkamaiChallenge):

  • Detect action="//duckduckgo.com/anomaly.js" or id="challenge-form" within anomaly-modal, or anomaly.js / anomaly-modal elements.
  • Ensure bare domain mentions (duckduckgo.com) or ordinary search result pages are not classified as challenges.
  • In isChallengeWall(), classify the anomaly challenge as a challenge wall.
  • In tiers/1.ts and directForward.ts, escalate detected challenges to needs-js / browser tiers.

Alternatives considered

  • Relying on MITM_PROXY_ALWAYS_SCRAPE: skips Tier 0 but Tier 1 still receives the anomaly page without escalating to Tier 3 unless the challenge is recognized.
  • Solving the challenge client-side: inefficient and fragile compared to letting Camoufox handle the navigation natively in Tier 3.

Tier impact

  • Fit inside an existing tier?
  • Require a new tier or solver?
  • Apply cross-cutting (caching, observability, config)?

Fits inside existing detection and Tier 1/proxy escalation (packages/tiers/src/utils/detect.ts, packages/tiers/src/tiers/1.ts, apps/api/src/proxy/directForward.ts).

Willingness to contribute

  • I'd like to implement this myself and open a PR

Additional context

Verified against live endpoints:

  • Plain HTTP POST to https://html.duckduckgo.com/html/ triggers HTTP/2 202 Accepted with form action //duckduckgo.com/anomaly.js and data-testid="anomaly-modal".
  • Tier 3 with Camoufox returns real search results with HTTP 200 OK.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions