Stop on a refused proxy before starting a browser, and skip media behind a proxy

CHANGELOG.md

@@ -8,6 +8,27 @@ a promise that every flag and exit code is contractually frozen.
 
 ## [Unreleased]
 
+> **Behaviour changes.** (1) Behind `--proxy` / `--proxy-file`, every browser
+> engine now sends one `HEAD https://www.farfetch.com/robots.txt` through the
+> proxy before starting a browser. A gateway `407` or `402` ends the run with
+> exit 5 and `stop_reason: proxy_auth_failed` in the new
+> `<out>.latest_attempt.meta.json`, and no browser is started. The data
+> files and `<out>.meta.json` are left untouched. (2) Behind a local proxy,
+> the browser no longer downloads images, video/audio or web fonts.
+> `--load-media` restores the old behaviour. Nothing changes with
+> `--cdp-endpoint` or without a proxy.
+
+### Added
+
+- Proxy pre-check in `playwright_scraper.py`, `puppeteer_scraper.py` and
+  `selenium_scraper.py`. Selenium pre-checks the proxy without credentials,
+  exactly as its browser uses it, and says so when a 407 follows. A 403, a
+  timeout or an unreachable proxy is not treated as a verdict.
+- Media blocking behind a local proxy: by resource type in Playwright
+  (`context.route`) and pyppeteer (CDP `Fetch`), and by file extension in
+  Selenium (`Network.setBlockedURLs`). `--load-media` opts out on all three.
+  Playwright turns off its HTTP cache while a route is installed.
+
 ### Changed
 
 - CI only: the whole canary, free and paid jobs alike, now runs weekly (Mondays 06:17 UTC) and on a manual dispatch; the daily schedule is removed.

README.md

@@ -248,7 +248,8 @@ The three browser engines share these:
 | `--out` | `farfetch_products` | Output prefix, also used for debug dumps |
 | `--delay` | `2.0` | Seconds between pages |
 | `--cdp-endpoint` | – | Attach to a running browser instead of launching one |
-| `--proxy` | – | Proxy for a self-launched browser; ignored with `--cdp-endpoint` |
+| `--proxy` | – | Proxy for a self-launched browser; ignored with `--cdp-endpoint`. Pre-checked before the browser starts ([traffic savers](#traffic-savers-pre-check-and-media-blocking)) |
+| `--load-media` | off | Load images, video and web fonts behind `--proxy` too (by default they are blocked there) |
 | `--retries` | `3` | Attempts per page load, pause doubling each time |
 | `--concurrency` | `1` | Playwright only — fetch pages through N parallel workers ([details](#concurrency)) |
 | `--twocaptcha-key` | – | 2Captcha API key (or set `TWOCAPTCHA_KEY`) |
@@ -283,7 +284,7 @@ A contract, not decoration — the harness and any pipeline can branch on these.
 | `2` | Bad usage |
 | `3` | **Blocked before parsing** — a challenge page, or an outright refusal |
 | `4` | Fetched the page fine, parsed **0 products**, and the page links to no products either |
-| `5` | **No content obtained** — navigation timeout, dead proxy, 4xx/5xx, a failed `--cdp-endpoint` connect, a Scraper API error, or a page that links to products and parsed to none (`stop_reason: parse_drift` — a parser problem, not a network one) |
+| `5` | **No content obtained** — navigation timeout, dead proxy, a proxy gateway that refused the pre-check with 407/402 (`stop_reason: proxy_auth_failed`), 4xx/5xx, a failed `--cdp-endpoint` connect, a Scraper API error, or a page that links to products and parsed to none (`stop_reason: parse_drift` — a parser problem, not a network one) |
 | `6` | **Partial run** — products written, but the page loop stopped early |
 | `124` | Selenium only: chromedriver could not be **started** within the watchdog |
 
@@ -802,6 +803,53 @@ all (use Selenium-Wire or an extension); and both flags are ignored with
 **Match the proxy's country to your fingerprint's.** A US fingerprint arriving on
 a German IP is a contradiction that is cheap to detect.
 
+#### Traffic savers: pre-check and media blocking
+
+Both apply only to a browser this repo launches itself behind `--proxy` /
+`--proxy-file`. Neither applies with `--cdp-endpoint` (the remote browser has
+its own exit and is configured on its own side), or with no proxy at all,
+where nothing is metered.
+
+**Pre-check.** Before any browser starts, each of the three engines sends ONE
+`HEAD https://www.farfetch.com/robots.txt` through the proxy it is about to
+use: the first exit of the pool for Playwright, the `--proxy` value for
+pyppeteer, and the bare `scheme://host:port` for Selenium. Selenium sends no
+credentials because its browser cannot send them either (see above), so the
+pre-check sees exactly what the browser will see.
+
+- A gateway `407` (the credential was refused, or the traffic balance is
+  exhausted) or `402` (traffic/balance exhausted) ends the run with **exit 5**.
+  The message names `FARFETCH_PROXY` and never prints its value.
+- No browser is started, and `<out>.latest_attempt.meta.json` records
+  `status: failed`, `stop_reason: proxy_auth_failed`. The data files and
+  `<out>.meta.json` from an earlier run are left as they were.
+- Any other answer, a 403 from the site, a timeout or a DNS error is not a
+  verdict on the proxy, and the run continues exactly as before. A `socks5://`
+  proxy is not pre-checked.
+
+**Media blocking.** Behind a local proxy, images, video/audio and web fonts
+are not downloaded. No parser here reads them: `image_url` comes from the
+markup and the JSON-LD, not from a loaded image. `--load-media` turns this
+off.
+
+| Engine | How | Limitation |
+|---|---|---|
+| Playwright | `context.route` aborts the `image`, `media` and `font` resource types | While a route is installed Playwright turns off its HTTP cache, so a run of many pages re-fetches the scripts and styles it would otherwise have cached |
+| pyppeteer | CDP `Fetch.enable` pauses only `Image`/`Media`/`Font` requests, and fails each one | – |
+| Selenium | CDP `Network.setBlockedURLs` by file extension (`*.jpg`, `*.webp`, `*.mp4`, `*.woff2`, …) | An image served from a URL without an extension still loads |
+
+Debug screenshots (`*_debug.png`) taken while blocking is on show the page
+without its images.
+
+Measured 2026-10-20 against a local stub page holding a png, a jpg, a woff2,
+an mp4 and a script. Without blocking, every engine fetched all of them. With
+blocking, Playwright and Selenium fetched only the page and the script, and
+pyppeteer fetched the page, the script and `/favicon.ico`. A live
+comparison against farfetch.com was not possible that day: from a
+datacentre address, both the plain run and the run with blocking forced
+(Playwright, no proxy, one listing page) got Akamai's 403 `Access Denied`
+(exit 3), the same refusal described in the exit-code section.
+
 ### 4. Fingerprints
 
 [Product](https://2captcha.com/buy-fingerprints) · `--fingerprint`

output_writer.py

@@ -315,7 +315,7 @@ def write_csv(products: List[Product], path: str,
 # replaced it honest: a reason cannot be both "we finished" and "we
 # never got there".
 FETCH_FAILURE_STOP_REASONS = ("page_load_timeout", "proxy_unusable",
-                              "http_error")
+                              "http_error", "proxy_auth_failed")
 
 
 def stop_reason_for(*, load_failed: bool, blocked_by: Optional[str],
@@ -375,6 +375,37 @@ def write_run_meta(out_prefix: str, meta: dict) -> str:
     return path
 
 
+# Beside `<out>.meta.json`, never instead of it: that file describes the data
+# beside it and is written only with that data. This one records an attempt
+# that ended before any data could be written.
+ATTEMPT_META_SUFFIX = ".latest_attempt.meta.json"
+
+
+def write_failed_attempt(out_prefix: str, *, stop_reason: str,
+                         start_url: Optional[str], pages_requested: int,
+                         transport_error: str, mode: str = "listing") -> str:
+    """Record a run that ended before `finish_run` could run at all.
+
+    Used when the proxy gateway refused the pre-check, so no browser was
+    started. Writes ONLY `<out>.latest_attempt.meta.json`: the data files and
+    `<out>.meta.json` from an earlier good run stay exactly as they were, so
+    the two never contradict each other.
+    """
+    meta = run_meta(status="failed", stop_reason=stop_reason,
+                    pages_requested=pages_requested, pages_completed=0,
+                    start_url=start_url or "", final_url=start_url or "",
+                    products=0, mode=mode)
+    meta.update(blocked=False, transport_error=transport_error,
+                data_updated=False)
+    path = f"{out_prefix}{ATTEMPT_META_SUFFIX}"
+    with _atomic(path) as f:
+        json.dump(meta, f, ensure_ascii=False, indent=2)
+    print(f"[+] Wrote attempt metadata -> {path} (status=failed, "
+          f"stop_reason={stop_reason}); the data and {out_prefix}.meta.json "
+          f"were left as they were.")
+    return path
+
+
 def run_meta(status: str, stop_reason: str, pages_requested: int,
              pages_completed: int, start_url: str, final_url: str,
              products: int, pages_failed: Optional[List[int]] = None,

playwright_scraper.py

@@ -64,13 +64,17 @@
 from product_detail_parser import parse_product_detail
 from output_writer import (dedupe_by_sku, finish_run, stop_reason_for,
                            COMPLETE_STOP_REASONS, new_run_id,
-                           ProductVariant, EXIT_FETCH_FAILED)
+                           ProductVariant, EXIT_FETCH_FAILED,
+                           write_failed_attempt)
 from run_state import Checkpoint
 from proxy_pool import (from_args as proxy_pool_from_args, to_playwright, mask,
                         redact_url_credentials, describe_cdp_failure,
                         cdp_failure_is_transient, CDP_CONNECT_ATTEMPTS,
                         CDP_CONNECT_PAUSE_S,
-                        ROTATE_MODES, ProxyError, ProxyPool)
+                        ROTATE_MODES, ProxyError, ProxyPool,
+                        precheck_proxy, robots_url, precheck_message,
+                        ProxyPrecheckError, PROXY_PRECHECK_STOP_REASON,
+                        should_block_media, playwright_media_route)
 import env_config
 from arg_types import positive_int, nonneg_int, nonneg_float
 
@@ -275,6 +279,13 @@ def _launch_local(pw, args, pool):
     if init_script:
         # Must be installed on the context, before any page script runs.
         context.add_init_script(init_script)
+    if should_block_media(args, proxy_in_use=bool(proxy)):
+        # Behind a metered proxy: abort images, video/audio and web fonts.
+        # No parser reads them (image_url comes from the markup, not from a
+        # loaded image). --load-media keeps them.
+        context.route("**/*", playwright_media_route)
+        logger.info("Blocking images/media/fonts "
+                    "(--load-media keeps them).")
     return browser, context, context.new_page()
 
 
@@ -983,6 +994,22 @@ def scrape(args) -> int:
                        "proxy on top would contradict it.")
         pool = None
 
+    if pool:
+        # One HEAD for robots.txt through the proxy BEFORE any browser
+        # starts: a gateway 407/402 ends the run here with exit 5 instead of
+        # launching browsers whose every request would be refused the same
+        # way. Any other answer, or a transport error, changes nothing.
+        try:
+            precheck_proxy(pool.current, robots_url(args.url), timeout=20)
+        except ProxyPrecheckError as exc:
+            print(precheck_message(exc, "FARFETCH_PROXY", "farfetch.com"),
+                  file=sys.stderr)
+            write_failed_attempt(args.out, stop_reason=PROXY_PRECHECK_STOP_REASON,
+                                 start_url=args.url, pages_requested=args.pages,
+                                 transport_error="proxy pre-check: %s" % exc,
+                                 mode=args.mode)
+            return EXIT_FETCH_FAILED
+
     concurrency = max(1, args.concurrency)
     if concurrency > 1:
         if args.cdp_endpoint:
@@ -1258,6 +1285,12 @@ def parse_args():
     p.add_argument("--format", choices=["json", "csv", "both"], default="both")
     p.add_argument("--out", default="farfetch_products", help="Output file prefix")
     p.add_argument("--proxy", default=None, help="Proxy URL, e.g. http://ACCOUNT:PASSWORD@HOST:9999 (2captcha.com/proxy)")
+    p.add_argument("--load-media", action="store_true",
+                   help="Load images, video and web fonts. By default they are "
+                        "aborted while a LOCAL browser goes through --proxy / --proxy-file "
+                        "(no parser reads them, and a proxy bills per byte); "
+                        "never blocked with --cdp-endpoint or with no proxy.")
+    p.add_argument("--force-block-media", action="store_true", help=argparse.SUPPRESS)
     p.add_argument("--proxy-file", default=None,
                    help="File with one proxy URL per line (# comments and blank "
                         "lines skipped) to rotate across. Wins over --proxy.")

proxy_pool.py

@@ -351,3 +351,159 @@ def from_args(args) -> Optional[ProxyPool]:
         return ProxyPool([entry], rotate="per-run")
 
     return None
+
+
+# ---------------------------------------------------------------------------
+# Traffic savers for runs through a paid proxy.
+#
+# 1. Pre-check. ONE small request (a HEAD for the target site's robots.txt)
+#    goes through the proxy BEFORE a browser is started. A gateway that
+#    answers 407 (credential refused, or the traffic balance is exhausted) or
+#    402 (traffic/balance exhausted) ends the run there, with exit 5, instead
+#    of launching a browser whose every request would be refused the same way.
+#    Any other outcome — a 200, a 403 from the site, a timeout, a DNS error —
+#    is not a verdict on the proxy and the run continues exactly as before.
+# 2. Media blocking. Images, video/audio and web fonts are aborted in the
+#    browser while its traffic goes through a LOCAL proxy (`--proxy` /
+#    `--proxy-file`): no parser here reads them, and they are most of a page's
+#    bytes. Not applied with `--cdp-endpoint` (the remote browser is billed
+#    and configured on its own side) or with no proxy (nothing is metered).
+#    `--load-media` keeps them.
+# ---------------------------------------------------------------------------
+
+PROXY_PRECHECK_STATUSES = {
+    407: "the gateway refused the credential (or the traffic balance is exhausted)",
+    402: "proxy traffic/balance exhausted",
+}
+PROXY_PRECHECK_STOP_REASON = "proxy_auth_failed"
+
+# Playwright `request.resource_type` values that are aborted.
+BLOCKED_RESOURCE_TYPES = frozenset({"image", "media", "font"})
+# The same three as CDP `Network.ResourceType` names (pyppeteer, via Fetch).
+BLOCKED_CDP_RESOURCE_TYPES = ("Image", "Media", "Font")
+# Selenium has no request events, only `Network.setBlockedURLs`, which
+# matches URLs, not resource types: an image served without an extension
+# still loads there.
+BLOCKED_URL_PATTERNS = tuple(
+    "*.%s%s" % (ext, tail)
+    for ext in ("jpg", "jpeg", "png", "gif", "webp", "avif", "bmp", "ico",
+                "mp4", "webm", "m4v", "mov", "mp3", "m4a", "ogg", "wav",
+                "woff", "woff2", "ttf", "otf", "eot")
+    for tail in ("", "?*"))
+
+
+class ProxyPrecheckError(RuntimeError):
+    """The proxy gateway itself refused the pre-check (407 or 402)."""
+
+    def __init__(self, status: int):
+        self.status = status
+        super().__init__("HTTP %d: %s" % (status, PROXY_PRECHECK_STATUSES[status]))
+
+
+_STATUS_IN_ERROR_RE = re.compile(r"Tunnel connection failed:\s*(40[27])\b")
+
+
+def robots_url(url: Optional[str]) -> Optional[str]:
+    """`scheme://host/robots.txt` for a page URL — the pre-check's target."""
+    try:
+        parts = urlparse(url or "")
+    except ValueError:
+        return None
+    if parts.scheme not in ("http", "https") or not parts.hostname:
+        return None
+    return "%s://%s/robots.txt" % (parts.scheme, parts.netloc.rsplit("@", 1)[-1])
+
+
+def precheck_proxy(proxy_url: Optional[str], probe_url: Optional[str],
+                   timeout: float = 20.0) -> Optional[int]:
+    """One HEAD for `probe_url` through `proxy_url`.
+
+    Raises ProxyPrecheckError on a gateway 407/402. Returns the status the
+    probe got otherwise, or None when there was nothing to check (no proxy, a
+    non-HTTP proxy scheme, no probe URL) or the request failed in transport —
+    a transport failure keeps the run's existing behaviour. The proxy URL is
+    never printed or put in the exception.
+    """
+    if not proxy_url or not probe_url:
+        return None
+    try:
+        scheme = (urlparse(proxy_url).scheme or "http").lower()
+    except ValueError:
+        return None
+    if scheme not in ("http", "https"):
+        return None
+    try:
+        import requests
+    except ImportError:
+        return None
+    try:
+        response = requests.head(probe_url, proxies={"http": proxy_url, "https": proxy_url},
+                                 timeout=timeout, allow_redirects=False)
+    except requests.exceptions.ProxyError as exc:
+        # An https target: the gateway answers the CONNECT, and urllib3
+        # reports its status line as "Tunnel connection failed: 407 ...".
+        match = _STATUS_IN_ERROR_RE.search(str(exc))
+        if match:
+            raise ProxyPrecheckError(int(match.group(1))) from None
+        return None
+    except Exception:  # noqa: BLE001 - transport errors are not a verdict
+        return None
+    if response.status_code in PROXY_PRECHECK_STATUSES:
+        raise ProxyPrecheckError(response.status_code)
+    return response.status_code
+
+
+def precheck_message(exc: ProxyPrecheckError, env_var: str, site: str,
+                     browser: bool = True) -> str:
+    """What to print. Names the variable, never its value."""
+    return ("[!] proxy pre-check failed — %s.\n"
+            "    %s%s was never reached. Check %s (or the --proxy / --proxy-file "
+            "value)." % (exc, "No browser was started and " if browser else "",
+                         site, env_var))
+
+
+def should_block_media(args, proxy_in_use: bool, over_cdp: bool = False) -> bool:
+    """Abort images/media/fonts? Only for a LOCAL browser behind a proxy.
+
+    `--load-media` opts out. The hidden `--force-block-media` turns it on
+    without a proxy, so the effect on parsing can be measured for free.
+    """
+    if getattr(args, "force_block_media", False):
+        return True
+    if getattr(args, "load_media", False) or over_cdp:
+        return False
+    return bool(proxy_in_use)
+
+
+def playwright_media_route(route) -> None:
+    """`context.route("**/*", ...)` handler: abort the heavy resource types."""
+    if route.request.resource_type in BLOCKED_RESOURCE_TYPES:
+        route.abort("blockedbyclient")
+    else:
+        route.continue_()
+
+
+async def pyppeteer_block_media(page) -> None:
+    """Abort images/media/fonts in a pyppeteer page through the CDP Fetch domain.
+
+    `page.setRequestInterception` sends Network.setRequestInterception, which
+    current Chromium no longer implements; Fetch.enable with resource-type
+    patterns pauses only the requests to be aborted.
+    """
+    import asyncio
+    client = await page.target.createCDPSession()
+
+    def on_paused(event):
+        asyncio.ensure_future(client.send("Fetch.failRequest", {
+            "requestId": event["requestId"], "errorReason": "BlockedByClient"}))
+
+    client.on("Fetch.requestPaused", on_paused)
+    await client.send("Fetch.enable", {"patterns": [
+        {"resourceType": kind, "requestStage": "Request"}
+        for kind in BLOCKED_CDP_RESOURCE_TYPES]})
+
+
+def selenium_block_media(driver) -> None:
+    """Block media URLs in a Chrome WebDriver through `Network.setBlockedURLs`."""
+    driver.execute_cdp_cmd("Network.enable", {})
+    driver.execute_cdp_cmd("Network.setBlockedURLs", {"urls": list(BLOCKED_URL_PATTERNS)})