[jehrr]

Problem

A run through a metered residential proxy wasted traffic and time when the proxy refused it. Chromium reports a proxy's HTTP 407 as Page.goto: Timeout ... exceeded. The canary on 2026-10-20 showed this: /api/search/... Page.goto: Timeout 60000ms three times through the proxy, then exit 5, with nothing naming the credential.

Resulting behaviour

Pre-check, in all three engines. When a local proxy is configured, page_flow.stop_on_refused_proxy sends one request through it before any browser is launched. That request is a CONNECT to www.webmotors.com.br:443 followed by HEAD /robots.txt inside the tunnel. No third-party IP service is involved.

• 407: the run logs "the gateway refused the credential (or the traffic balance is exhausted)" and names WEBMOTORS_PROXY (or --proxy / --proxy-file). It exits 5 and writes <out>.latest_attempt.meta.json with status: failed and stop_reason: proxy_auth_failed.
• 402: the same handling, with "proxy traffic/balance exhausted" and stop_reason: proxy_balance_exhausted.
• The credential is never printed; the exit is shown masked.
• <out>.meta.json and any previous output are left untouched.
• A proxy that does not answer keeps the existing behaviour: the browser makes its own attempt.
• Nothing changes without a proxy or over --cdp-endpoint.

Media blocking, opt-in on this site (--block-media). When enabled, it aborts image, media and font requests:

• Playwright: context.route, by resource type.
• pyppeteer: CDP Fetch.enable with Image/Media/Font patterns, on its own CDP session. This leaves the engine's existing Fetch-based proxy auth alone.
• Selenium: CDP Network.setBlockedURLs, by file extension.

--load-media / WEBMOTORS_LOAD_MEDIA=1 keeps media loading. Neither flag does anything over CDP.

Why media blocking is off by default here. The rows come from the site's JSON API, which needs no media. The landing page, however, runs PerimeterX, which has refused this scraper for smaller identity inconsistencies. A free comparison run on 2026-10-20 (no proxy, one search page, media loaded vs --block-media) was refused by CloudFront both times, because this datacentre address is refused before the site sees the request. So whether PerimeterX reacts to blocked resources could not be measured, and the default keeps media.

Evidence

Offline suite

• Core-only venv: 480 passed, 0 failed.
• Playwright venv: 501 passed. Puppeteer venv: 489 passed. Selenium venv: 489 passed.
• In each engine's own venv, that engine's new checks ran.

New checks

• A stub proxy on 127.0.0.1 answering 407, 402 or 200, plus a closed port:
  • exactly one CONNECT, carrying the credential;
  • the outcome for each answer;
  • no message contains the credential.
• The attempt sidecar holds the expected values, and the old output is untouched.
• The decision table: default off, --block-media with and without a proxy, --load-media, CDP.
• The Playwright route handler, the pyppeteer Fetch patterns and failRequest, and the Selenium URL list.
• Each engine's scrape() on a 407 stub: exit 5, no browser started, sidecar written, credential absent from the log.
• Each engine's session opened on fakes, to check when blocking is installed.

Planted-fault controls. Each fault turned the named check red, and the suite was green again after the restore.

• 407 not recognised → "a 407 from the proxy is auth_failed and fatal", "gate() returns the fatal check on a 407", "playwright_scraper: a 407 exits 5 without starting a browser".
• Route handler never aborts → "Playwright: image/media/font aborted, everything else continues".
• stop_on_refused_proxy call removed from each of the three engines → ": a 407 exits 5 without starting a browser".
• Blocking install removed from each of the three engines → ": media blocking --block-media through a proxy = True (got False)".

Other

• The install functions are byte-identical to quora-scraper#16. There they were checked on real browsers (Playwright, pyppeteer with system Chromium, Selenium with chromedriver) against a local stub page: blocking stopped the image, video and font requests, while the document, script and stylesheet still loaded.
• No proxy and no paid call was used.

Not changed

• The canary workflow, versions and releases.
• Selenium still cannot send proxy credentials. This is unchanged and documented.
• The Scraper API client does not route through a local proxy, so it has no pre-check.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8