[jehrr]

Problem

A run through a paid residential proxy could start a whole browser on a proxy that refuses every request (a gateway 407 or 402), and then spend its retries and block-retries on that. Once a browser was running behind the proxy, it also downloaded every image, video and web font through the metered proxy, and no parser here reads any of them.

What changes

Proxy pre-check (all three browser engines). Before launching a local browser behind --proxy / --proxy-file, playwright_scraper.py, puppeteer_scraper.py and selenium_scraper.py send ONE HEAD https://<start host>/robots.txt through the proxy. The shared logic is in page_flow.proxy_precheck, and the helpers are in proxy_pool.py.

• 407: exit 5, with the message the gateway refused the credential (or the traffic balance is exhausted).
• 402: exit 5, with the message proxy traffic/balance exhausted.
• In both cases the message names MEDIAMARKT_PROXY and never prints its value. The repo had no attempt sidecar, so a minimal output_writer.write_failed_attempt was added. It writes only <out>.latest_attempt.meta.json (status: failed, stop_reason: proxy_auth_failed, data_updated: false), and the data and <out>.meta.json are left as they were. If even that file cannot be written, the run still exits 5 rather than crashing.
• Any other answer, or a transport error, is not a verdict on the proxy, and the run continues as it did before.
• No pre-check is sent with --cdp-endpoint or with no proxy.
• With --proxy-file, the first exit of the pool is checked. The pre-check is one request and is never retried or rotated.

Existing proxy check unified, not duplicated. .github/canary_preflight.py already asked the canary's proxy for the page before the browser run. That request is unchanged. Its 407 handling now uses the same classification and wording as the engines (proxy_pool's status regex, PROXY_PRECHECK_STATUSES and precheck_message). It also recognises a 402, and a 407/402 answered to a plain http:// request.

Media blocking. With a local browser behind a proxy, images, media and fonts are aborted:

• Playwright uses context.route by resource_type, installed in _launch_local, so every relaunch and concurrency worker gets it.
• pyppeteer uses CDP Fetch.enable with resource-type patterns. This engine opens no other Fetch interception, because it authenticates with page.authenticate.
• Selenium uses Network.setBlockedURLs by file extension, so an image served without an extension still loads. If the CDP command fails, the engine warns and loads the media.
• --load-media opts out. A hidden --force-block-media turns blocking on without a proxy, for measurement.
• Nothing is blocked over CDP or with no proxy. <out>.meta.json records media_blocked.
• No env variable was added, because env_config.ENV_KEYS maps only credentials and URLs.

Evidence

• Offline suite, python smoke_test.py:

  venv	Passed	Failed	Notes
  core only	446	0	engine groups reported skipped
  playwright	477	0
  pyppeteer	464	0
  selenium	464	0

  pytest -q, ci_checks.py --secret-check and env_config.py are green, and every engine's --help shows --load-media and hides --force-block-media.

• New test_proxy_traffic_savers checks use a local stub proxy that answers every request, CONNECT included, with a fixed status:

  • On 407 and 402: exit 5 after exactly one CONNECT www.mediamarkt.de:443. The message names the variable and does not print the credential. The attempt sidecar reads failed/proxy_auth_failed. Earlier .json/.csv/.meta.json files keep their bytes, and nothing else is written.
  • The same holds through each engine's real scrape(), with its browser launch patched to raise, which shows the browser is never started.
  • On a site 403, the browser does start. A proxy that cannot be reached is not treated as a failure.
  • No pre-check is sent with --cdp-endpoint.
  • The media-policy matrix, the Playwright route handler on fake routes, the Selenium CDP command, and the pyppeteer Fetch patterns and failRequest on a fake CDP session.
  • A source check that each engine calls should_block_media plus its blocker and records media_blocked.
  • canary_preflight.py run against the stub for 407/402 over https and http.

• The real CLIs against the stub proxy (--proxy http://user:…@127.0.0.1:<stub>): Playwright, pyppeteer and Selenium each gave rc=5 for both 407 and 402. The stub saw one CONNECT, the secret did not appear in the output, and only r.latest_attempt.meta.json was written.

• Each real engine session class (_BrowserSession / _Session) was driven against a local page containing png/jpg/woff2/mp4/js:

  Engine	Without blocking	With --force-block-media
  Playwright	fetched all assets	fetched / and /app.js only, and the JS still ran
  pyppeteer (system Chromium)	fetched all assets	fetched /, /app.js and the browser's own /favicon.ico
  Selenium (system Chromium)	fetched all assets	fetched / and /app.js only

• Live comparison on 2026-10-20, Playwright, no proxy, one category page, with and without --force-block-media: not possible. Both runs got MediaMarkt's 403 refusal page (exit 3) in about 0.25 s, the same datacentre refusal the README documents. Instead, the parser was read for anything that depends on loaded media. image_url comes from JSON-LD or from the tile's src/data-src attribute, and the asset-host block signal counts references in the markup. Nothing reads image bytes, dimensions or load events. Debug screenshots taken behind a proxy will show no images, and the README says so.

• Planted-fault controls, each restored afterwards and green again:

  • Disabling the call inside page_flow.proxy_precheck turns 15 checks red: the 407/402 checks and the per-engine exits 5 before a browser is started.
  • Replacing the engine call in selenium_scraper.scrape turns 5 checks red, including selenium_scraper: a proxy 407 exits 5 before a browser is started.
  • Making playwright_media_route continue every request turns media: playwright image/media/font -> abort red.
  • Removing context.route from _launch_local turns media: playwright_scraper applies the policy red.
  • Removing the pyppeteer blocker call turns media: puppeteer_scraper applies the policy red.

• No paid calls, no proxy and no credentials were used.

Not changed

• The canary still sends its own preflight request, then the engine's one HEAD for robots.txt (no body). The preflight's page request is what names a timeout or a dead tunnel, which the pre-check deliberately does not judge.
• .github/ci_checks.py: the stub credentials user:S3cretPw@127.0.0.1 and u:p@127.0.0.1 were added to the secret scan's allowlist.
• No version bump and no release. The canary was not dispatched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8