[jehrr]

Problem

Runs through the residential proxy that this site needs (datacentre addresses get a CloudFront 403) spent traffic and time they did not need to:

• A rejected credential was invisible in the browser engines. Chromium reports a proxy's HTTP 407 as Page.goto: Timeout ... exceeded. On 2026-10-20, three Playwright canary runs timed out this way for 60 s each before the requests pass reported Tunnel connection failed: 407.
• http_scraper.py did not stop on the 407 either. It saw the 407 inside a ProxyError but treated it as a transport failure and retried.
• Media went through the proxy. The browser engines downloaded every image, video and font through the proxy, although the rows come from the page's inline React Router payload.

Resulting behaviour

• One pre-check per run, before the first page request, in every entry point (http_scraper.py, Playwright, Selenium, pyppeteer). It is a CONNECT through the proxy plus HEAD /robots.txt on www.sleepnumber.com, with no third-party IP service. It is shared code in proxy_check.py, so the requests transport and the browsers use one implementation.
  • 407: the run logs "the gateway refused the credential (or the traffic balance is exhausted)", naming SLEEPNUMBER_PROXY (or --proxy / --proxy-file). It exits 5 and writes <out>.latest_attempt.meta.json with status: failed and stop_reason: proxy_auth_failed.
  • 402: the same, with "proxy traffic/balance exhausted" and proxy_balance_exhausted.
  • The credential is never printed. <out>.meta.json and the previous output are untouched.
  • A proxy that does not answer, or refuses the tunnel for another reason, keeps the existing behaviour: a warning, then the normal retries.
  • Nothing changes without a proxy, or over --cdp-endpoint.
• Media blocking, browser engines only, and only through a local proxy:
  • Playwright: context.route aborts image/media/font. This also covers --concurrency workers, which go through the same _launch_local.
  • pyppeteer: CDP Fetch.enable with Image/Media/Font patterns, on its own CDP session. page.setRequestInterception is not used because current Chromium no longer has Network.setRequestInterception.
  • Selenium: CDP Network.setBlockedURLs by file extension.
  • --load-media (or SLEEPNUMBER_LOAD_MEDIA=1) turns it off. A hidden --block-media forces it on for tests and measurements.
  • http_scraper.py fetches the HTML document only, so there is nothing to block on that path.

Evidence

• Offline suite: 112/112 checks in the core-only venv, and 119/119 in each of the playwright, puppeteer and selenium venvs. In each venv, that engine's new checks ran.
• New checks:
  • A stub proxy on 127.0.0.1 that answers 407, 402 or 200, plus a closed port.
  • http_scraper.scrape() driven on a 407: exit 5, _fetch_one_page never called, the attempt sidecar written, and the credential absent from the log.
  • Each engine's scrape() driven on a 407: exit 5 and no browser started.
  • The Playwright route handler on stub routes, the pyppeteer Fetch patterns and failRequest, and the Selenium URL list.
  • Each engine's session opened on fakes, to check when blocking is installed.
• Planted-fault controls. Each one went red on the named check, and the suite was green again after restoring:
  • 407 not recognised → "a 407 from the proxy is auth_failed and fatal", "gate() returns the fatal check on a 407", "http_scraper: a 407 exits 5 before any page request", "playwright_scraper: a 407 exits 5 without starting a browser".
  • Route handler never aborts → "Playwright: image/media/font aborted ...".
  • gate() call removed from http_scraper.py → "http_scraper: a 407 exits 5 before any page request".
  • gate() call removed from each of the three engines → ": a 407 exits 5 without starting a browser".
  • Blocking install removed from each of the three engines → ": media blocking through a proxy = True (got False)".
• Real-browser stub page, run with this repo's proxy_check.py: with blocking on, the server never saw the image, the video or the woff2 font, while the document, script and stylesheet loaded. This held on all three drivers.
• Free live run from a datacentre on 2026-10-20, with no proxy, Playwright, --category mattresses, media loaded vs --block-media:
  • Both runs got the same 923-byte CloudFront "Request blocked" 403 (exit 3). The two pages differ only in the request id.
  • So a served-page comparison was not obtainable without a residential exit, which was not used.
  • Blocking is still on by default here, because:
    • the parser reads the inline payload only;
    • http_scraper.py loads no sub-resource and produces the same rows as Playwright (measured in this repo on 2026-10-13);
    • the refusal is an address-reputation deny with no challenge script, and it was identical either way.
• No proxy and no paid call was used.

Not changed

• The canary workflow, versions and releases.
• Selenium still cannot send proxy credentials (unchanged and documented).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8