Stop on a refused proxy before the first request; skip media through a proxy

.env.example

@@ -36,6 +36,12 @@ TWOCAPTCHA_KEY={your_2captcha_api_key}
 # ---------------------------------------------------------------------------
 SLEEPNUMBER_PROXY=http://{login}-zone-resi-region-us:{password}@proxy.2captcha.com:2334
 
+# Through a proxy, the browser engines do not download images, video/audio
+# or fonts: the proxy is billed by traffic and the catalogue is read from the
+# page's inline payload. Set to 1 to load them anyway (same as --load-media).
+# Left empty = block. http_scraper.py fetches HTML only, so it is unaffected.
+SLEEPNUMBER_LOAD_MEDIA=
+
 # ---------------------------------------------------------------------------
 # SLEEPNUMBER_CDP_ENDPOINT — the 2Captcha Scraping Browser API, over CDP.
 #

CHANGELOG.md

@@ -11,6 +11,38 @@ nobody discovers it from a bill or a broken cron job.
 
 ## [Unreleased]
 
+> **Behaviour changes.** (1) A run through a proxy now makes one pre-check
+> request before its first page request, in `http_scraper.py` and all three
+> browser engines. A proxy that answers **HTTP 407** or **402** ends the run
+> with **exit 5** and `<out>.latest_attempt.meta.json` (`status: failed`,
+> `stop_reason: proxy_auth_failed` / `proxy_balance_exhausted`) instead of
+> failing every retry. (2) Through a local proxy, the browser engines **no
+> longer download** images, video/audio or fonts; `--load-media` (or
+> `SLEEPNUMBER_LOAD_MEDIA=1`) restores them. Runs without a proxy, over
+> `--cdp-endpoint`, and the HTML-only `http_scraper.py` fetch are otherwise
+> unchanged.
+
+### Added
+
+- `proxy_check.py`: the proxy pre-check (one CONNECT plus `HEAD /robots.txt`
+  on www.sleepnumber.com, no third-party service), shared by
+  `http_scraper.py` and the browser engines, plus media blocking for the
+  three browser engines. The 407 message names `SLEEPNUMBER_PROXY` and says
+  the gateway refused the credential (or the traffic balance is exhausted);
+  a 402 says the proxy traffic/balance is exhausted. Neither prints the
+  credential. A proxy that does not answer keeps the existing retry
+  behaviour.
+- `--load-media` on the three browser engines, and `SLEEPNUMBER_LOAD_MEDIA`
+  in `.env.example`.
+- Offline checks:
+  - a stub proxy that answers 407/402/200 or nothing;
+  - `http_scraper.py` and each engine driven to exit 5 on a 407, with no
+    page request and no browser;
+  - the attempt sidecar;
+  - the Playwright route handler, the pyppeteer Fetch patterns and the
+    Selenium blocked-URL list;
+  - when each engine installs blocking.
+
 ### Changed
 
 - CI only: the canary bills the `SLEEPNUMBER_PROXY` exit, so it now runs weekly

Dockerfile

@@ -51,7 +51,7 @@ RUN pip install --no-cache-dir -r requirements.txt -r requirements-playwright.tx
 # because nothing built it.
 COPY captcha_solver.py env_config.py fingerprint_client.py output_writer.py \
      page_flow.py playwright_scraper.py product_parser.py proxy_pool.py \
-     scraper_api_client.py diff_runs.py http_scraper.py ./
+     scraper_api_client.py diff_runs.py http_scraper.py proxy_check.py ./
 
 # No test suite, no fixtures, no .github/ and no .env — see .dockerignore.
 ENTRYPOINT ["python", "playwright_scraper.py"]

README.md

@@ -458,6 +458,39 @@ address and does not change between attempts, so retrying buys a second copy
 of a certainty and is billed for it. A challenge page still retries, because
 that one can clear.
 
+### Proxy traffic savers
+
+A proxy is billed by traffic, so a run through `--proxy` / `--proxy-file` /
+`SLEEPNUMBER_PROXY` does two things to avoid paying for bytes it does not
+use:
+
+- **One pre-check before the first page request, in every entry point**
+  (`http_scraper.py` and the three browser engines). It is a CONNECT through
+  the proxy and a `HEAD /robots.txt` on www.sleepnumber.com. Chromium
+  reports a proxy's HTTP 407 as a navigation timeout, and `requests` reports
+  it inside a `ProxyError`, so a wrong or exhausted credential used to spend
+  every retry before ending as `page_load_timeout`. Now a **407** (the
+  gateway refused the credential, or the traffic balance is exhausted) or a
+  **402** (proxy traffic/balance exhausted) ends the run in seconds with
+  **exit 5** and writes `<out>.latest_attempt.meta.json` saying
+  `status: failed`, `stop_reason: proxy_auth_failed` (or
+  `proxy_balance_exhausted`). Any previous output and its `.meta.json` are
+  left as they were. The message names the variable and never prints the
+  credential. If the proxy does not answer at all, the run goes on as
+  before. No pre-check runs without a proxy or with `--cdp-endpoint`.
+- **Images, video/audio and web fonts are not downloaded** by the browser
+  engines through a local proxy. Playwright aborts them by resource type,
+  pyppeteer through CDP's Fetch domain, and Selenium by file extension with
+  CDP `Network.setBlockedURLs`. The rows come from the page's inline
+  payload, which is why `http_scraper.py` (which never loads a sub-resource)
+  produces the same rows. `http_scraper.py` fetches HTML only, so blocking
+  does not apply to it. On 2026-10-20 a run with no proxy from a datacentre
+  got the same 923-byte CloudFront "Request blocked" 403 with media loaded
+  and with media blocked, so the refusal does not depend on what the page
+  loads; a served comparison needs a residential exit and was not made.
+  `--load-media` (or `SLEEPNUMBER_LOAD_MEDIA=1`) loads media. Without a
+  proxy, or over `--cdp-endpoint`, nothing is blocked.
+
 ### Install exactly one engine
 
 `playwright` and `pyppeteer` pin mutually unsatisfiable `pyee` versions, and
@@ -484,7 +517,9 @@ python3 playwright_scraper.py --mode product --category cm-mattress
   never replaced with `[]`. `--allow-empty` is the opt-out.
 - **Exit codes:** `0` ok · `1` crash · `2` bad usage · `3` blocked ·
   `4` zero products · `5` the content was never obtained (a remote API
-  error, a navigation timeout, a dead proxy) · `6` partial.
+  error, a navigation timeout, a dead proxy, or a proxy that refused the
+  credential or the balance before the run began: `stop_reason:
+  proxy_auth_failed` / `proxy_balance_exhausted`) · `6` partial.
 - **`<out>.meta.json` beside every run that WRITES ROWS**, recording
   `status`, `stop_reason`, `transport`, which pages failed by number, and the
   site's own arithmetic about the result set — `total_results`, `per_page`,
@@ -504,7 +539,9 @@ python3 playwright_scraper.py --mode product --category cm-mattress
   "per run" and it meant "per run that writes rows". If you want a machine-
   readable record of attempts including failures, read the exit code and keep
   your own log; this repo will not put a contradicting file next to good
-  data.
+  data. The one exception is a run the proxy pre-check stopped (see "Proxy
+  traffic savers"): it writes `<out>.latest_attempt.meta.json`, a file with
+  its own name, and leaves `<out>.meta.json` alone.
 - **An empty CSV still carries its header.**
 - **Writes are atomic.** Each file is written beside its target and renamed
   over it, so a crash or a full disk mid-write leaves the previous file
@@ -544,7 +581,7 @@ Precedence, highest first: explicit flag → exported environment variable →
 ## Tests
 
 ```bash
-python3 smoke_test.py -v      # 59 checks, no network, no engine needed
+python3 smoke_test.py -v      # 112 checks, no network, no engine needed
 pytest                        # the same checks, wrapped
 ```
 

env_config.py

@@ -41,6 +41,8 @@
     "SLEEPNUMBER_CDP_ENDPOINT": "cdp_endpoint",
     "SLEEPNUMBER_PROXY": "proxy",
     "SLEEPNUMBER_URL": "url",
+    # "1" keeps images/video/fonts loading through a proxy (--load-media).
+    "SLEEPNUMBER_LOAD_MEDIA": "load_media",
 }
 # Deliberately NOT here: an output prefix. `--out` already carries a non-empty
 # default, so `apply()` would never see it as unset and the variable would be

http_scraper.py

@@ -84,13 +84,15 @@
 
 import requests
 
-from output_writer import dedupe_by_key, finish_run, SOURCE_DEFAULT
+from output_writer import (dedupe_by_key, finish_run, finish_without_fetch,
+                           SOURCE_DEFAULT)
 import page_flow
 from product_parser import (parse_products, detect_bot_challenge,
                             is_supported_url, normalise_url,
                             category_url, product_url, SITE_ASSET_MARKERS)
 from proxy_pool import from_args as proxy_pool_from_args, mask, ROTATE_MODES, ProxyError
 import env_config
+import proxy_check
 
 logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
 logger = logging.getLogger("http_scraper")
@@ -294,6 +296,16 @@ def scrape(args) -> int:
 
     pool = proxy_pool_from_args(args)
 
+    # One request through the proxy before the first page request: a 407 or 402
+    # from the gateway ends the run here, in seconds, instead of surfacing
+    # as a ProxyError on every retry.
+    refused = proxy_check.gate(pool, args.url)
+    if refused:
+        return finish_without_fetch(
+            args.out, stop_reason=refused.stop_reason,
+            pages_requested=args.pages, start_url=args.url, mode=args.mode,
+            source=SOURCE_DEFAULT)
+
     with requests.Session() as session:
         first = _fetch_one_page(session, args, pool, 1, args.url)
         outcomes.append(first)

output_writer.py

@@ -456,6 +456,35 @@ def write_run_meta(out_prefix: str, meta: dict) -> str:
     return path
 
 
+ATTEMPT_META_SUFFIX = ".latest_attempt.meta.json"
+
+
+def finish_without_fetch(out_prefix: str, *, stop_reason: str,
+                         pages_requested: int, start_url: str,
+                         mode: str = "listing", source: str = SOURCE_DEFAULT,
+                         message: str = "") -> int:
+    """End a run that was stopped before anything was fetched; exit 5.
+
+    Used by the proxy pre-check (a 407 or 402 from the proxy). The record
+    goes to `<out>.latest_attempt.meta.json`, NOT `<out>.meta.json`: the
+    latter describes the data file beside it, and a previous run's good
+    output is still there, untouched, so overwriting its metadata with
+    `status: failed` would make the pair contradict each other.
+    """
+    meta = run_meta(status="failed", stop_reason=stop_reason,
+                    pages_requested=pages_requested, pages_completed=0,
+                    start_url=start_url, final_url=start_url, products=0,
+                    mode=mode, source=source)
+    path = f"{out_prefix}{ATTEMPT_META_SUFFIX}"
+    with _atomic(path) as f:
+        json.dump(meta, f, ensure_ascii=False, indent=2)
+    print(f"[+] Wrote attempt metadata -> {path} (status=failed, "
+          f"stop_reason={stop_reason})")
+    print(f"[!] Failed run: nothing was fetched ({stop_reason}). "
+          f"{message} Any previous output and its .meta.json are untouched.")
+    return EXIT_FETCH_FAILED
+
+
 def run_meta(status: str, stop_reason: str, pages_requested: int,
              pages_completed: int, start_url: str, final_url: str,
              products: int, pages_failed: Optional[List[int]] = None,

playwright_scraper.py

@@ -79,11 +79,13 @@
                             category_url, product_url, CATEGORY_URL,
                             CANONICAL_HOST)
 from output_writer import (dedupe_by_key, finish_run, RemoteAPIError,
-                           EXIT_REMOTE_API_ERROR, SOURCE_DEFAULT)
+                           EXIT_REMOTE_API_ERROR, SOURCE_DEFAULT,
+                           finish_without_fetch)
 import page_flow
 from proxy_pool import (from_args as proxy_pool_from_args, to_playwright, mask,
                         ROTATE_MODES, ProxyError, ProxyPool)
 import env_config
+import proxy_check
 
 logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
 logger = logging.getLogger("playwright_scraper")
@@ -188,6 +190,8 @@ def _launch_local(pw, args, pool):
     context = browser.new_context(**ctx_kwargs)
     if init_script:
         context.add_init_script(init_script)
+    if proxy_check.block_media(args, pool, remote=False):
+        proxy_check.install_playwright(context)
     return browser, context, context.new_page()
 
 
@@ -847,6 +851,16 @@ def scrape(args) -> int:
                         "its whole lifetime by design, "
                         "rotated only per WORKER, not per page.")
 
+    # One request through the proxy before a browser exists: a 407 or 402
+    # from the gateway ends the run here, in seconds, instead of surfacing
+    # as a navigation timeout after every retry.
+    refused = proxy_check.gate(pool, args.url)
+    if refused:
+        return finish_without_fetch(
+            args.out, stop_reason=refused.stop_reason,
+            pages_requested=args.pages, start_url=args.url, mode=args.mode,
+            source=SOURCE_DEFAULT)
+
     with sync_playwright() as pw:
         session = _BrowserSession(pw, args, pool,
                                   remote=bool(args.cdp_endpoint)).open()
@@ -1077,6 +1091,7 @@ def parse_args():
     p.add_argument("--captcha-api", choices=["v2", "v1"], default="v2")
     p.add_argument("--solve-captcha", choices=["when-blocked", "always"], default="when-blocked")
     p.add_argument("--min-score", type=float, default=0.7)
+    proxy_check.add_media_flags(p)
     p.add_argument("--cdp-endpoint", default=None,
                    help="Connect to an already-running browser over CDP, "
                         "e.g. ws://user:pass@host:port — the Scraping "