[jehrr]

Problem

A run through a paid proxy could start a whole browser on a proxy that refuses every request (a gateway 407 or 402). Once it was running, a local browser behind the proxy also downloaded every image, video and web font through the metered proxy, and no parser here reads any of them.

What changes

Proxy pre-check. Before a local browser starts behind --proxy / --proxy-file, playwright_scraper.py, puppeteer_scraper.py and selenium_scraper.py each send ONE HEAD https://www.farfetch.com/robots.txt through the proxy (proxy_pool.precheck_proxy).

• Playwright checks the pool's first exit, and pyppeteer checks the --proxy value.
• Selenium checks the bare scheme://host:port, because its browser's --proxy-server cannot carry credentials either. On a 407 it adds a line saying that, and points to playwright_scraper.py.
• 407: exit 5, with the message the gateway refused the credential (or the traffic balance is exhausted).
• 402: exit 5, with the message proxy traffic/balance exhausted.
• The message names FARFETCH_PROXY and never prints its value.
• A new <out>.latest_attempt.meta.json records status: failed, stop_reason: proxy_auth_failed and data_updated: false. The data files and <out>.meta.json are left untouched. This repo had no attempt sidecar, so output_writer.write_failed_attempt was added for it. proxy_auth_failed joins FETCH_FAILURE_STOP_REASONS.
• Any other answer (a 403, a timeout, an unreachable proxy or a socks5:// proxy) does not count as a verdict, and the run continues as it did before.
• Nothing is sent with --cdp-endpoint.

Media blocking, only for a local browser behind a proxy:

• Playwright uses context.route by resource type, installed in _launch_local, so it also covers rotations and --concurrency workers.
• pyppeteer uses CDP Fetch.enable with Image/Media/Font patterns.
• Selenium uses Network.setBlockedURLs by file extension.
• --load-media (in --help) opts out on all three engines. A hidden --force-block-media turns blocking on without a proxy, for measurement.
• Nothing is blocked over CDP or with no proxy.
• The README documents the trade-offs: Playwright turns its HTTP cache off while a route is installed, Selenium's blocking misses images served without an extension, and debug screenshots show no images.

Evidence

• Offline suite (python smoke_test.py):

  • core-only venv: 434 PASS, 0 FAIL;
  • playwright venv: 490/0, SKIPPED_ENGINES: pyppeteer,selenium;
  • pyppeteer venv: 435/0;
  • selenium venv: 449/0.
  • pytest -q: 492 passed. ruff check . is clean, mypy is clean on its 9 gated files, and ci_checks.py --all is ok.

• New section check_proxy_traffic_savers. It runs in the core-only job too, with each engine loaded against fake driver modules. It checks:

  • per engine, against a local stub proxy answering everything (CONNECT included) with 407 or 402: exit 5, no browser started, exactly one CONNECT www.farfetch.com:443, the message names FARFETCH_PROXY, the secret never appears in stdout, stderr or the logs, the sidecar reads failed/proxy_auth_failed, and the output directory holds only the attempt sidecar;
  • 403 and an unreachable proxy: the browser starts;
  • no proxy, and --cdp-endpoint: no pre-check;
  • the media policy matrix, and the Playwright route by resource type;
  • each engine's real launch code with fake browser objects: route installed or not, Fetch patterns and failRequest, Selenium's blocked URL list and media_blocked;
  • a source check per engine, and --help shows --load-media but not --force-block-media.

• Real CLI against the stub proxy (--proxy http://user:…@127.0.0.1:<stub>): rc=5 for both 407 and 402 on all three engines. The stub saw one CONNECT each time, and the secret did not leak.

• Each engine's real main() was run against a local stub page holding png/jpg/woff2/mp4/js:

  Engine	Without blocking	With --force-block-media
  Playwright (bundled Chromium)	all assets	/ and /app.js
  pyppeteer (system Chromium)	all assets and /favicon.ico	/, /app.js and /favicon.ico
  Selenium (system Chromium + chromedriver)	all assets and /favicon.ico	/ and /app.js

• Live comparison, 2026-10-20, no proxy: playwright_scraper.py --url …/shopping/women/dresses-1/items.aspx --pages 1, plain and with --force-block-media. Both runs got Akamai's 403 Access Denied (329 bytes, exit 3) from this datacentre address, so values could not be compared. From reading the code: listing image_url comes from JSON-LD image or the <img> src/data-src attribute, and detail images come from JSON-LD. No extraction reads image bytes or load state.

• Planted-fault controls, each restored and then green again:

  • Disabling precheck_proxy in Playwright turns 13 pre-check (playwright) checks red. Doing the same in Selenium turns 15 pre-check (selenium) checks red.
  • Making playwright_media_route continue every request turns media: playwright image/media/font -> abort red.
  • Removing selenium_block_media(driver) turns the two media: selenium … blocked by URL checks and the source check red.
  • Removing await pyppeteer_block_media(page) turns the two media: pyppeteer … Fetch pauses checks and the source check red.

• No paid calls, no proxy and no credentials were used.

Not changed

• No env variable was added for --load-media: env_config.ENV_KEYS maps only credentials and URLs.
• The pyppeteer proxy authentication (page.authenticate) is unchanged.
• Dependabot PR ci: bump docker/build-push-action from 6 to 7 #41 was not touched.
• No version bump and no release. The canary was not dispatched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.