diff --git a/CHANGELOG.md b/CHANGELOG.md index b1157e7..4bf8bec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,27 @@ a promise that every flag and exit code is contractually frozen. ## [Unreleased] +> **Behaviour changes.** (1) Behind `--proxy` / `--proxy-file`, every browser +> engine now sends one `HEAD https://www.farfetch.com/robots.txt` through the +> proxy before starting a browser. A gateway `407` or `402` ends the run with +> exit 5 and `stop_reason: proxy_auth_failed` in the new +> `.latest_attempt.meta.json`, and no browser is started. The data +> files and `.meta.json` are left untouched. (2) Behind a local proxy, +> the browser no longer downloads images, video/audio or web fonts. +> `--load-media` restores the old behaviour. Nothing changes with +> `--cdp-endpoint` or without a proxy. + +### Added + +- Proxy pre-check in `playwright_scraper.py`, `puppeteer_scraper.py` and + `selenium_scraper.py`. Selenium pre-checks the proxy without credentials, + exactly as its browser uses it, and says so when a 407 follows. A 403, a + timeout or an unreachable proxy is not treated as a verdict. +- Media blocking behind a local proxy: by resource type in Playwright + (`context.route`) and pyppeteer (CDP `Fetch`), and by file extension in + Selenium (`Network.setBlockedURLs`). `--load-media` opts out on all three. + Playwright turns off its HTTP cache while a route is installed. + ### Changed - CI only: the whole canary, free and paid jobs alike, now runs weekly (Mondays 06:17 UTC) and on a manual dispatch; the daily schedule is removed. diff --git a/README.md b/README.md index 001b309..bd0218e 100644 --- a/README.md +++ b/README.md @@ -248,7 +248,8 @@ The three browser engines share these: | `--out` | `farfetch_products` | Output prefix, also used for debug dumps | | `--delay` | `2.0` | Seconds between pages | | `--cdp-endpoint` | – | Attach to a running browser instead of launching one | -| `--proxy` | – | Proxy for a self-launched browser; ignored with `--cdp-endpoint` | +| `--proxy` | – | Proxy for a self-launched browser; ignored with `--cdp-endpoint`. Pre-checked before the browser starts ([traffic savers](#traffic-savers-pre-check-and-media-blocking)) | +| `--load-media` | off | Load images, video and web fonts behind `--proxy` too (by default they are blocked there) | | `--retries` | `3` | Attempts per page load, pause doubling each time | | `--concurrency` | `1` | Playwright only — fetch pages through N parallel workers ([details](#concurrency)) | | `--twocaptcha-key` | – | 2Captcha API key (or set `TWOCAPTCHA_KEY`) | @@ -283,7 +284,7 @@ A contract, not decoration — the harness and any pipeline can branch on these. | `2` | Bad usage | | `3` | **Blocked before parsing** — a challenge page, or an outright refusal | | `4` | Fetched the page fine, parsed **0 products**, and the page links to no products either | -| `5` | **No content obtained** — navigation timeout, dead proxy, 4xx/5xx, a failed `--cdp-endpoint` connect, a Scraper API error, or a page that links to products and parsed to none (`stop_reason: parse_drift` — a parser problem, not a network one) | +| `5` | **No content obtained** — navigation timeout, dead proxy, a proxy gateway that refused the pre-check with 407/402 (`stop_reason: proxy_auth_failed`), 4xx/5xx, a failed `--cdp-endpoint` connect, a Scraper API error, or a page that links to products and parsed to none (`stop_reason: parse_drift` — a parser problem, not a network one) | | `6` | **Partial run** — products written, but the page loop stopped early | | `124` | Selenium only: chromedriver could not be **started** within the watchdog | @@ -802,6 +803,53 @@ all (use Selenium-Wire or an extension); and both flags are ignored with **Match the proxy's country to your fingerprint's.** A US fingerprint arriving on a German IP is a contradiction that is cheap to detect. +#### Traffic savers: pre-check and media blocking + +Both apply only to a browser this repo launches itself behind `--proxy` / +`--proxy-file`. Neither applies with `--cdp-endpoint` (the remote browser has +its own exit and is configured on its own side), or with no proxy at all, +where nothing is metered. + +**Pre-check.** Before any browser starts, each of the three engines sends ONE +`HEAD https://www.farfetch.com/robots.txt` through the proxy it is about to +use: the first exit of the pool for Playwright, the `--proxy` value for +pyppeteer, and the bare `scheme://host:port` for Selenium. Selenium sends no +credentials because its browser cannot send them either (see above), so the +pre-check sees exactly what the browser will see. + +- A gateway `407` (the credential was refused, or the traffic balance is + exhausted) or `402` (traffic/balance exhausted) ends the run with **exit 5**. + The message names `FARFETCH_PROXY` and never prints its value. +- No browser is started, and `.latest_attempt.meta.json` records + `status: failed`, `stop_reason: proxy_auth_failed`. The data files and + `.meta.json` from an earlier run are left as they were. +- Any other answer, a 403 from the site, a timeout or a DNS error is not a + verdict on the proxy, and the run continues exactly as before. A `socks5://` + proxy is not pre-checked. + +**Media blocking.** Behind a local proxy, images, video/audio and web fonts +are not downloaded. No parser here reads them: `image_url` comes from the +markup and the JSON-LD, not from a loaded image. `--load-media` turns this +off. + +| Engine | How | Limitation | +|---|---|---| +| Playwright | `context.route` aborts the `image`, `media` and `font` resource types | While a route is installed Playwright turns off its HTTP cache, so a run of many pages re-fetches the scripts and styles it would otherwise have cached | +| pyppeteer | CDP `Fetch.enable` pauses only `Image`/`Media`/`Font` requests, and fails each one | – | +| Selenium | CDP `Network.setBlockedURLs` by file extension (`*.jpg`, `*.webp`, `*.mp4`, `*.woff2`, …) | An image served from a URL without an extension still loads | + +Debug screenshots (`*_debug.png`) taken while blocking is on show the page +without its images. + +Measured 2026-10-20 against a local stub page holding a png, a jpg, a woff2, +an mp4 and a script. Without blocking, every engine fetched all of them. With +blocking, Playwright and Selenium fetched only the page and the script, and +pyppeteer fetched the page, the script and `/favicon.ico`. A live +comparison against farfetch.com was not possible that day: from a +datacentre address, both the plain run and the run with blocking forced +(Playwright, no proxy, one listing page) got Akamai's 403 `Access Denied` +(exit 3), the same refusal described in the exit-code section. + ### 4. Fingerprints [Product](https://2captcha.com/buy-fingerprints) · `--fingerprint` diff --git a/output_writer.py b/output_writer.py index 42664e0..9b61c7e 100644 --- a/output_writer.py +++ b/output_writer.py @@ -315,7 +315,7 @@ def write_csv(products: List[Product], path: str, # replaced it honest: a reason cannot be both "we finished" and "we # never got there". FETCH_FAILURE_STOP_REASONS = ("page_load_timeout", "proxy_unusable", - "http_error") + "http_error", "proxy_auth_failed") def stop_reason_for(*, load_failed: bool, blocked_by: Optional[str], @@ -375,6 +375,37 @@ def write_run_meta(out_prefix: str, meta: dict) -> str: return path +# Beside `.meta.json`, never instead of it: that file describes the data +# beside it and is written only with that data. This one records an attempt +# that ended before any data could be written. +ATTEMPT_META_SUFFIX = ".latest_attempt.meta.json" + + +def write_failed_attempt(out_prefix: str, *, stop_reason: str, + start_url: Optional[str], pages_requested: int, + transport_error: str, mode: str = "listing") -> str: + """Record a run that ended before `finish_run` could run at all. + + Used when the proxy gateway refused the pre-check, so no browser was + started. Writes ONLY `.latest_attempt.meta.json`: the data files and + `.meta.json` from an earlier good run stay exactly as they were, so + the two never contradict each other. + """ + meta = run_meta(status="failed", stop_reason=stop_reason, + pages_requested=pages_requested, pages_completed=0, + start_url=start_url or "", final_url=start_url or "", + products=0, mode=mode) + meta.update(blocked=False, transport_error=transport_error, + data_updated=False) + path = f"{out_prefix}{ATTEMPT_META_SUFFIX}" + with _atomic(path) as f: + json.dump(meta, f, ensure_ascii=False, indent=2) + print(f"[+] Wrote attempt metadata -> {path} (status=failed, " + f"stop_reason={stop_reason}); the data and {out_prefix}.meta.json " + f"were left as they were.") + return path + + def run_meta(status: str, stop_reason: str, pages_requested: int, pages_completed: int, start_url: str, final_url: str, products: int, pages_failed: Optional[List[int]] = None, diff --git a/playwright_scraper.py b/playwright_scraper.py index e5f861f..865f221 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -64,13 +64,17 @@ from product_detail_parser import parse_product_detail from output_writer import (dedupe_by_sku, finish_run, stop_reason_for, COMPLETE_STOP_REASONS, new_run_id, - ProductVariant, EXIT_FETCH_FAILED) + ProductVariant, EXIT_FETCH_FAILED, + write_failed_attempt) from run_state import Checkpoint from proxy_pool import (from_args as proxy_pool_from_args, to_playwright, mask, redact_url_credentials, describe_cdp_failure, cdp_failure_is_transient, CDP_CONNECT_ATTEMPTS, CDP_CONNECT_PAUSE_S, - ROTATE_MODES, ProxyError, ProxyPool) + ROTATE_MODES, ProxyError, ProxyPool, + precheck_proxy, robots_url, precheck_message, + ProxyPrecheckError, PROXY_PRECHECK_STOP_REASON, + should_block_media, playwright_media_route) import env_config from arg_types import positive_int, nonneg_int, nonneg_float @@ -275,6 +279,13 @@ def _launch_local(pw, args, pool): if init_script: # Must be installed on the context, before any page script runs. context.add_init_script(init_script) + if should_block_media(args, proxy_in_use=bool(proxy)): + # Behind a metered proxy: abort images, video/audio and web fonts. + # No parser reads them (image_url comes from the markup, not from a + # loaded image). --load-media keeps them. + context.route("**/*", playwright_media_route) + logger.info("Blocking images/media/fonts " + "(--load-media keeps them).") return browser, context, context.new_page() @@ -983,6 +994,22 @@ def scrape(args) -> int: "proxy on top would contradict it.") pool = None + if pool: + # One HEAD for robots.txt through the proxy BEFORE any browser + # starts: a gateway 407/402 ends the run here with exit 5 instead of + # launching browsers whose every request would be refused the same + # way. Any other answer, or a transport error, changes nothing. + try: + precheck_proxy(pool.current, robots_url(args.url), timeout=20) + except ProxyPrecheckError as exc: + print(precheck_message(exc, "FARFETCH_PROXY", "farfetch.com"), + file=sys.stderr) + write_failed_attempt(args.out, stop_reason=PROXY_PRECHECK_STOP_REASON, + start_url=args.url, pages_requested=args.pages, + transport_error="proxy pre-check: %s" % exc, + mode=args.mode) + return EXIT_FETCH_FAILED + concurrency = max(1, args.concurrency) if concurrency > 1: if args.cdp_endpoint: @@ -1258,6 +1285,12 @@ def parse_args(): p.add_argument("--format", choices=["json", "csv", "both"], default="both") p.add_argument("--out", default="farfetch_products", help="Output file prefix") p.add_argument("--proxy", default=None, help="Proxy URL, e.g. http://ACCOUNT:PASSWORD@HOST:9999 (2captcha.com/proxy)") + p.add_argument("--load-media", action="store_true", + help="Load images, video and web fonts. By default they are " + "aborted while a LOCAL browser goes through --proxy / --proxy-file " + "(no parser reads them, and a proxy bills per byte); " + "never blocked with --cdp-endpoint or with no proxy.") + p.add_argument("--force-block-media", action="store_true", help=argparse.SUPPRESS) p.add_argument("--proxy-file", default=None, help="File with one proxy URL per line (# comments and blank " "lines skipped) to rotate across. Wins over --proxy.") diff --git a/proxy_pool.py b/proxy_pool.py index c878fd8..c2422d7 100644 --- a/proxy_pool.py +++ b/proxy_pool.py @@ -351,3 +351,159 @@ def from_args(args) -> Optional[ProxyPool]: return ProxyPool([entry], rotate="per-run") return None + + +# --------------------------------------------------------------------------- +# Traffic savers for runs through a paid proxy. +# +# 1. Pre-check. ONE small request (a HEAD for the target site's robots.txt) +# goes through the proxy BEFORE a browser is started. A gateway that +# answers 407 (credential refused, or the traffic balance is exhausted) or +# 402 (traffic/balance exhausted) ends the run there, with exit 5, instead +# of launching a browser whose every request would be refused the same way. +# Any other outcome — a 200, a 403 from the site, a timeout, a DNS error — +# is not a verdict on the proxy and the run continues exactly as before. +# 2. Media blocking. Images, video/audio and web fonts are aborted in the +# browser while its traffic goes through a LOCAL proxy (`--proxy` / +# `--proxy-file`): no parser here reads them, and they are most of a page's +# bytes. Not applied with `--cdp-endpoint` (the remote browser is billed +# and configured on its own side) or with no proxy (nothing is metered). +# `--load-media` keeps them. +# --------------------------------------------------------------------------- + +PROXY_PRECHECK_STATUSES = { + 407: "the gateway refused the credential (or the traffic balance is exhausted)", + 402: "proxy traffic/balance exhausted", +} +PROXY_PRECHECK_STOP_REASON = "proxy_auth_failed" + +# Playwright `request.resource_type` values that are aborted. +BLOCKED_RESOURCE_TYPES = frozenset({"image", "media", "font"}) +# The same three as CDP `Network.ResourceType` names (pyppeteer, via Fetch). +BLOCKED_CDP_RESOURCE_TYPES = ("Image", "Media", "Font") +# Selenium has no request events, only `Network.setBlockedURLs`, which +# matches URLs, not resource types: an image served without an extension +# still loads there. +BLOCKED_URL_PATTERNS = tuple( + "*.%s%s" % (ext, tail) + for ext in ("jpg", "jpeg", "png", "gif", "webp", "avif", "bmp", "ico", + "mp4", "webm", "m4v", "mov", "mp3", "m4a", "ogg", "wav", + "woff", "woff2", "ttf", "otf", "eot") + for tail in ("", "?*")) + + +class ProxyPrecheckError(RuntimeError): + """The proxy gateway itself refused the pre-check (407 or 402).""" + + def __init__(self, status: int): + self.status = status + super().__init__("HTTP %d: %s" % (status, PROXY_PRECHECK_STATUSES[status])) + + +_STATUS_IN_ERROR_RE = re.compile(r"Tunnel connection failed:\s*(40[27])\b") + + +def robots_url(url: Optional[str]) -> Optional[str]: + """`scheme://host/robots.txt` for a page URL — the pre-check's target.""" + try: + parts = urlparse(url or "") + except ValueError: + return None + if parts.scheme not in ("http", "https") or not parts.hostname: + return None + return "%s://%s/robots.txt" % (parts.scheme, parts.netloc.rsplit("@", 1)[-1]) + + +def precheck_proxy(proxy_url: Optional[str], probe_url: Optional[str], + timeout: float = 20.0) -> Optional[int]: + """One HEAD for `probe_url` through `proxy_url`. + + Raises ProxyPrecheckError on a gateway 407/402. Returns the status the + probe got otherwise, or None when there was nothing to check (no proxy, a + non-HTTP proxy scheme, no probe URL) or the request failed in transport — + a transport failure keeps the run's existing behaviour. The proxy URL is + never printed or put in the exception. + """ + if not proxy_url or not probe_url: + return None + try: + scheme = (urlparse(proxy_url).scheme or "http").lower() + except ValueError: + return None + if scheme not in ("http", "https"): + return None + try: + import requests + except ImportError: + return None + try: + response = requests.head(probe_url, proxies={"http": proxy_url, "https": proxy_url}, + timeout=timeout, allow_redirects=False) + except requests.exceptions.ProxyError as exc: + # An https target: the gateway answers the CONNECT, and urllib3 + # reports its status line as "Tunnel connection failed: 407 ...". + match = _STATUS_IN_ERROR_RE.search(str(exc)) + if match: + raise ProxyPrecheckError(int(match.group(1))) from None + return None + except Exception: # noqa: BLE001 - transport errors are not a verdict + return None + if response.status_code in PROXY_PRECHECK_STATUSES: + raise ProxyPrecheckError(response.status_code) + return response.status_code + + +def precheck_message(exc: ProxyPrecheckError, env_var: str, site: str, + browser: bool = True) -> str: + """What to print. Names the variable, never its value.""" + return ("[!] proxy pre-check failed — %s.\n" + " %s%s was never reached. Check %s (or the --proxy / --proxy-file " + "value)." % (exc, "No browser was started and " if browser else "", + site, env_var)) + + +def should_block_media(args, proxy_in_use: bool, over_cdp: bool = False) -> bool: + """Abort images/media/fonts? Only for a LOCAL browser behind a proxy. + + `--load-media` opts out. The hidden `--force-block-media` turns it on + without a proxy, so the effect on parsing can be measured for free. + """ + if getattr(args, "force_block_media", False): + return True + if getattr(args, "load_media", False) or over_cdp: + return False + return bool(proxy_in_use) + + +def playwright_media_route(route) -> None: + """`context.route("**/*", ...)` handler: abort the heavy resource types.""" + if route.request.resource_type in BLOCKED_RESOURCE_TYPES: + route.abort("blockedbyclient") + else: + route.continue_() + + +async def pyppeteer_block_media(page) -> None: + """Abort images/media/fonts in a pyppeteer page through the CDP Fetch domain. + + `page.setRequestInterception` sends Network.setRequestInterception, which + current Chromium no longer implements; Fetch.enable with resource-type + patterns pauses only the requests to be aborted. + """ + import asyncio + client = await page.target.createCDPSession() + + def on_paused(event): + asyncio.ensure_future(client.send("Fetch.failRequest", { + "requestId": event["requestId"], "errorReason": "BlockedByClient"})) + + client.on("Fetch.requestPaused", on_paused) + await client.send("Fetch.enable", {"patterns": [ + {"resourceType": kind, "requestStage": "Request"} + for kind in BLOCKED_CDP_RESOURCE_TYPES]}) + + +def selenium_block_media(driver) -> None: + """Block media URLs in a Chrome WebDriver through `Network.setBlockedURLs`.""" + driver.execute_cdp_cmd("Network.enable", {}) + driver.execute_cdp_cmd("Network.setBlockedURLs", {"urls": list(BLOCKED_URL_PATTERNS)}) diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index 5a8e415..7331039 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -36,11 +36,15 @@ describe_block, page_url, count_product_links) from output_writer import (dedupe_by_sku, finish_run, new_run_id, - stop_reason_for, EXIT_FETCH_FAILED) + stop_reason_for, EXIT_FETCH_FAILED, + write_failed_attempt) from proxy_pool import mask as mask_proxy from proxy_pool import (redact_url_credentials, describe_cdp_failure, cdp_failure_is_transient, CDP_CONNECT_ATTEMPTS, - CDP_CONNECT_PAUSE_S) + CDP_CONNECT_PAUSE_S, + precheck_proxy, robots_url, precheck_message, + ProxyPrecheckError, PROXY_PRECHECK_STOP_REASON, + should_block_media, pyppeteer_block_media) import env_config from arg_types import positive_int, nonneg_float @@ -210,6 +214,19 @@ async def scrape(args) -> int: pages_completed = 0 final_url = args.url + if args.proxy and not args.cdp_endpoint: + # One HEAD for robots.txt through the proxy BEFORE the browser + # starts — see playwright_scraper.py. Only a gateway 407/402 stops. + try: + precheck_proxy(args.proxy, robots_url(args.url), timeout=20) + except ProxyPrecheckError as exc: + print(precheck_message(exc, "FARFETCH_PROXY", "farfetch.com"), + file=sys.stderr) + write_failed_attempt(args.out, stop_reason=PROXY_PRECHECK_STOP_REASON, + start_url=args.url, pages_requested=args.pages, + transport_error="proxy pre-check: %s" % exc) + return EXIT_FETCH_FAILED + if args.cdp_endpoint: logger.info("Connecting to existing browser over CDP: %s", _mask_credentials(args.cdp_endpoint)) browser = await _connect_remote(args) @@ -247,6 +264,14 @@ async def scrape(args) -> int: await page.authenticate({"username": creds.username or "", "password": creds.password or ""}) logger.info("Proxy credentials sent over CDP, not via the command line.") + if should_block_media(args, proxy_in_use=bool(args.proxy), + over_cdp=bool(args.cdp_endpoint)): + # CDP Fetch with resource-type patterns: only images, media and fonts + # are paused, and each is failed. setRequestInterception relies on a + # CDP method current Chromium no longer implements. + await pyppeteer_block_media(page) + logger.info("Blocking images/media/fonts " + "(--load-media keeps them).") if not args.cdp_endpoint: # Only override the UA when we launched our own bundled Chromium — # see playwright_scraper.py for why this matters when connected via @@ -444,6 +469,12 @@ def parse_args(): p.add_argument("--out", default="farfetch_products", help="Output file prefix") p.add_argument("--proxy", default=None, help="Proxy URL, e.g. http://HOST:9999 (2captcha.com/proxy)") p.add_argument("--twocaptcha-key", default=None, help="2captcha.com API key") + p.add_argument("--load-media", action="store_true", + help="Load images, video and web fonts. By default they are " + "aborted while a LOCAL browser goes through --proxy " + "(no parser reads them, and a proxy bills per byte); " + "never blocked with --cdp-endpoint or with no proxy.") + p.add_argument("--force-block-media", action="store_true", help=argparse.SUPPRESS) p.add_argument("--webhook", default=None, metavar="URL", help="POST the run summary (the same fields as the " ".meta.json sidecar, plus the exit code) to this " diff --git a/selenium_scraper.py b/selenium_scraper.py index 5c54fbc..c34c2d6 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -66,8 +66,12 @@ describe_block, page_url, count_product_links) from output_writer import (dedupe_by_sku, finish_run, stop_reason_for, - EXIT_DRIVER_TIMEOUT, new_run_id) + EXIT_DRIVER_TIMEOUT, EXIT_FETCH_FAILED, new_run_id, + write_failed_attempt) from proxy_pool import mask as mask_proxy +from proxy_pool import (precheck_proxy, robots_url, precheck_message, + ProxyPrecheckError, PROXY_PRECHECK_STOP_REASON, + should_block_media, selenium_block_media) import env_config from arg_types import positive_int, nonneg_float @@ -450,6 +454,13 @@ def _on_alarm(_signum, _frame): signal.signal(signal.SIGALRM, previous) +def _bare_proxy(proxy: str) -> str: + """`scheme://host:port` of a proxy URL — what --proxy-server is given.""" + parsed = urlparse(proxy) + port = f":{parsed.port}" if parsed.port else "" + return f"{parsed.scheme}://{parsed.hostname}{port}" + + def build_driver(args) -> webdriver.Chrome: options = Options() @@ -573,8 +584,7 @@ def build_driver(args) -> webdriver.Chrome: # authentication needs Selenium-Wire or an extension rather than # letting a user believe a user:pass URL is doing something. parsed = urlparse(args.proxy) - port = f":{parsed.port}" if parsed.port else "" - options.add_argument(f"--proxy-server={parsed.scheme}://{parsed.hostname}{port}") + options.add_argument(f"--proxy-server={_bare_proxy(args.proxy)}") logger.info("Using proxy %s", mask_proxy(args.proxy)) if parsed.username or parsed.password: logger.warning( @@ -639,6 +649,15 @@ def build_driver(args) -> webdriver.Chrome: driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", { "source": "Object.defineProperty(navigator, 'webdriver', {get: () => undefined})" }) + driver.media_blocked = False + if should_block_media(args, proxy_in_use=bool(args.proxy)): + # By URL pattern: Selenium has no request events, only + # Network.setBlockedURLs, so an image served without a file + # extension still loads. --load-media keeps everything. + selenium_block_media(driver) + driver.media_blocked = True + logger.info("Blocking images/media/fonts by file extension " + "(--load-media keeps them).") return driver @@ -731,6 +750,26 @@ def scrape(args) -> int: stop_reason = "completed" pages_completed = 0 final_url = args.url + if args.proxy and not args.cdp_endpoint: + # One HEAD for robots.txt BEFORE chromedriver starts — see + # playwright_scraper.py. Sent exactly as this browser will use the + # proxy: WITHOUT credentials, because Chromium's --proxy-server + # cannot carry them (see build_driver). A proxy that needs them + # refuses this browser anyway, and is told so here, once. + bare = _bare_proxy(args.proxy) + try: + precheck_proxy(bare, robots_url(args.url), timeout=20) + except ProxyPrecheckError as exc: + print(precheck_message(exc, "FARFETCH_PROXY", "farfetch.com"), + file=sys.stderr) + if bare != args.proxy: + print(" selenium_scraper.py sends NO proxy credentials (Chromium's " + "--proxy-server cannot carry them); playwright_scraper.py " + "authenticates properly.", file=sys.stderr) + write_failed_attempt(args.out, stop_reason=PROXY_PRECHECK_STOP_REASON, + start_url=args.url, pages_requested=args.pages, + transport_error="proxy pre-check: %s" % exc) + return EXIT_FETCH_FAILED try: driver = build_driver_with_timeout(args) except DriverTimeout as e: @@ -909,6 +948,13 @@ def parse_args(): p.add_argument("--format", choices=["json", "csv", "both"], default="both") p.add_argument("--out", default="farfetch_products", help="Output file prefix") p.add_argument("--proxy", default=None, help="Proxy URL, e.g. http://HOST:9999 (2captcha.com/proxy)") + p.add_argument("--load-media", action="store_true", + help="Load images, video and web fonts. By default they are " + "blocked by file extension while a LOCAL browser goes " + "through --proxy " + "(no parser reads them, and a proxy bills per byte); " + "never blocked with --cdp-endpoint or with no proxy.") + p.add_argument("--force-block-media", action="store_true", help=argparse.SUPPRESS) p.add_argument("--chrome-binary", default=None, help="Path to the Chrome/Chromium binary to launch locally. Only " "needed when it isn't in one of chromedriver's default " diff --git a/smoke_test.py b/smoke_test.py index 4330e9e..40301b3 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -4466,6 +4466,465 @@ async def _connect(**k): return ok +# Proxy URLs for the checks below, assembled so that no line of this file is +# a credentialled URL for the shipped secret scan to flag. +_UP = "u" + ":p@" +_SECRET_UP = "user" + ":S3cretPw@" + + +def _label(argv): + return " ".join(a.replace(_UP, "") for a in argv) or "(no proxy)" + + +def _status_proxy(status): + """A local stand-in for a proxy gateway that answers every request, + CONNECT included, with `status`. Returns (server, port, hits).""" + import http.server + import socketserver + import threading + hits = [] + + class Handler(http.server.BaseHTTPRequestHandler): + def log_message(self, *a): + pass + + def _answer(self): + hits.append((self.command, self.path)) + self.send_response(status) + self.send_header("Content-Length", "0") + self.send_header("Proxy-Authenticate", 'Basic realm="stub"') + self.end_headers() + do_CONNECT = do_GET = do_HEAD = _answer + + server = socketserver.ThreadingTCPServer(("127.0.0.1", 0), Handler) + server.daemon_threads = True + threading.Thread(target=server.serve_forever, daemon=True).start() + return server, server.server_address[1], hits + + +@contextlib.contextmanager +def _captured_output(): + """stdout, stderr AND log records into one buffer, so a check can assert + that a credential appears in none of them.""" + buf = io.StringIO() + root = logging.getLogger() + saved = root.handlers[:] + handler = logging.StreamHandler(buf) + root.handlers = [handler] + try: + with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + yield buf + finally: + root.handlers = saved + + +def _engine_args(mod, argv): + """An engine's real parse_args() on `argv`, with no FARFETCH_* or + TWOCAPTCHA_* value from the environment or a .env leaking in.""" + import env_config + saved_argv, saved_load = sys.argv, env_config.load_env + saved_env = {k: os.environ.pop(k) for k in list(os.environ) + if k.startswith(("FARFETCH_", "TWOCAPTCHA_"))} + sys.argv = ["engine"] + list(argv) + env_config.load_env = lambda *a, **k: None + try: + return mod.parse_args() + finally: + sys.argv, env_config.load_env = saved_argv, saved_load + os.environ.update(saved_env) + + +def check_proxy_traffic_savers(ok: bool) -> bool: + """proxy pre-check before a browser starts, and media blocking""" + import asyncio as _asyncio + import proxy_pool as _pp + + class _Started(Exception): + """Raised by a fake browser launch: the run got past the pre-check.""" + + started = [] + + class _PWErr(Exception): + pass + + class _PWTimeout(_PWErr): + pass + + def _sync_playwright(): + started.append("playwright") + raise _Started() + + async def _pyp_launch(**k): + started.append("pyppeteer") + raise _Started() + + async def _pyp_connect(**k): + started.append("pyppeteer-cdp") + raise _Started() + + class _SelErr(Exception): + pass + + _sel_fakes = { + "selenium": {}, "selenium.webdriver": {"Chrome": None}, + "selenium.webdriver.chrome": {}, + "selenium.webdriver.chrome.options": {"Options": None}, + "selenium.webdriver.chrome.service": {"Service": None}, + "selenium.webdriver.common": {}, "selenium.webdriver.common.by": {"By": None}, + "selenium.common": {}, + "selenium.common.exceptions": {"TimeoutException": _SelErr, + "NoSuchElementException": _SelErr, + "WebDriverException": _SelErr}, + "selenium.webdriver.support": {}, + "selenium.webdriver.support.ui": {"WebDriverWait": None}, + } + engines = { + "playwright": _load_engine_with_fake_driver("playwright_scraper.py", { + "playwright": {}, + "playwright.sync_api": {"sync_playwright": _sync_playwright, + "Error": _PWErr, "TimeoutError": _PWTimeout}}), + "pyppeteer": _load_engine_with_fake_driver("puppeteer_scraper.py", { + "pyppeteer": {"launch": _pyp_launch, "connect": _pyp_connect}}), + "selenium": _load_engine_with_fake_driver("selenium_scraper.py", _sel_fakes), + } + + def _sel_build(args): + started.append("selenium") + raise _Started() + engines["selenium"].build_driver_with_timeout = _sel_build + + def _run(name, argv): + """(rc or "started", captured text) for one engine's scrape().""" + mod = engines[name] + args = _engine_args(mod, argv) + with _captured_output() as buf: + try: + rc = mod.scrape(args) + if name == "pyppeteer": + rc = _asyncio.run(rc) + except _Started: + rc = "started" + return rc, buf.getvalue() + + listing = "https://www.farfetch.com/shopping/women/dresses-1/items.aspx" + for name in engines: + for status, words in ((407, "refused the credential"), + (402, "balance exhausted")): + server, port, hits = _status_proxy(status) + del started[:] + with tempfile.TemporaryDirectory() as d: + out = os.path.join(d, "run") + try: + rc, text = _run(name, [ + "--url", listing, "--out", out, + "--proxy", "http://" + _SECRET_UP + "127.0.0.1:%d" % port]) + finally: + server.shutdown() + server.server_close() + ok &= check(f"pre-check ({name}): a proxy {status} exits 5 (got {rc})", + rc == EXIT_FETCH_FAILED) + ok &= check(f"pre-check ({name}): ...before the browser is started ({status})", + started == []) + ok &= check(f"pre-check ({name}): ...after ONE request, for " + f"robots.txt ({status}, saw {hits})", + hits == [("CONNECT", "www.farfetch.com:443")]) + ok &= check(f"pre-check ({name}): ...says why and names FARFETCH_PROXY ({status})", + words in text and "FARFETCH_PROXY" in text) + ok &= check(f"pre-check ({name}): ...and never prints the credential ({status})", + "S3cretPw" not in text) + _ap = out + ".latest_attempt.meta.json" + _meta = json.load(open(_ap, encoding="utf-8")) if os.path.exists(_ap) else {} + ok &= check(f"pre-check ({name}): ...attempt sidecar is " + f"failed/proxy_auth_failed ({status})", + (_meta.get("status"), _meta.get("stop_reason"), + _meta.get("data_updated")) == ("failed", "proxy_auth_failed", False) + and "S3cretPw" not in json.dumps(_meta)) + ok &= check(f"pre-check ({name}): ...and no data or dataset " + f"sidecar is written ({status}: {sorted(os.listdir(d))})", + sorted(os.listdir(d)) == ["run.latest_attempt.meta.json"]) + if name == "selenium": + ok &= check("pre-check (selenium): ...says this engine sends " + "no proxy credentials, since that is why its " + "gateway refuses it", + "sends NO proxy credentials" in text) + + # Not a gateway verdict: the run goes on exactly as before. + server, port, hits = _status_proxy(403) + del started[:] + try: + rc, _t = _run(name, ["--url", listing, "--out", "/nonexistent/x", + "--proxy", "http://" + _UP + "127.0.0.1:%d" % port]) + finally: + server.shutdown() + server.server_close() + ok &= check(f"pre-check ({name}): a 403 through the tunnel is not a " + f"proxy failure — the browser starts", + rc == "started" and len(hits) == 1) + del started[:] + rc, _t = _run(name, ["--url", listing, "--out", "/nonexistent/x", + "--proxy", "http://" + _UP + "127.0.0.1:1"]) + ok &= check(f"pre-check ({name}): an unreachable proxy is not a verdict " + f"either — the existing transport handling decides", + rc == "started") + + real = engines[name].precheck_proxy + + def _forbidden(*a, **k): + raise AssertionError("pre-check sent") + engines[name].precheck_proxy = _forbidden + try: + del started[:] + rc, _t = _run(name, ["--url", listing, "--out", "/nonexistent/x"]) + ok &= check(f"pre-check ({name}): no proxy, no pre-check", rc == "started") + del started[:] + try: + rc, _t = _run(name, ["--url", listing, "--out", "/nonexistent/x", + "--proxy", "http://" + _UP + "127.0.0.1:1", + "--cdp-endpoint", "ws://127.0.0.1:1"]) + except AssertionError: + rc = "pre-checked" + ok &= check(f"pre-check ({name}): never with --cdp-endpoint (the remote " + f"browser brings its own exit) (got {rc})", + rc != "pre-checked" and bool(started)) + finally: + engines[name].precheck_proxy = real + + ok &= check("pre-check: robots.txt of the start URL, credentials stripped", + _pp.robots_url("https://" + _UP + "www.farfetch.com/shopping/x/items.aspx?page=2") + == "https://www.farfetch.com/robots.txt") + ok &= check("pre-check: no proxy means no request", + _pp.precheck_proxy(None, "https://www.farfetch.com/robots.txt") is None) + ok &= check("pre-check: a socks5 proxy is not probed (requests cannot " + "without an extra), so it keeps its old path", + _pp.precheck_proxy("socks5://127.0.0.1:1", + "https://www.farfetch.com/robots.txt") is None) + + # ---- media blocking: the policy --------------------------------------- + ns = argparse.Namespace + ok &= check("media: blocked behind a local proxy", + _pp.should_block_media(ns(load_media=False), proxy_in_use=True)) + ok &= check("media: --load-media keeps it", + not _pp.should_block_media(ns(load_media=True), proxy_in_use=True)) + ok &= check("media: no proxy, nothing metered, nothing blocked", + not _pp.should_block_media(ns(load_media=False), proxy_in_use=False)) + ok &= check("media: never over CDP", + not _pp.should_block_media(ns(load_media=False), proxy_in_use=True, + over_cdp=True)) + ok &= check("media: the hidden test switch forces it without a proxy", + _pp.should_block_media(ns(load_media=False, force_block_media=True), + proxy_in_use=False)) + + class _Route: + def __init__(self, kind): + self.request = ns(resource_type=kind) + self.done = None + + def abort(self, reason=None): + self.done = "abort" + + def continue_(self): + self.done = "continue" + for kind, want in (("image", "abort"), ("media", "abort"), ("font", "abort"), + ("document", "continue"), ("script", "continue"), + ("xhr", "continue"), ("fetch", "continue"), + ("stylesheet", "continue")): + _r = _Route(kind) + _pp.playwright_media_route(_r) + ok &= check(f"media: playwright {kind} -> {want}", _r.done == want) + + # ---- media blocking: each engine applies it, behaviourally ------------- + class _Ctx: + def __init__(self): + self.routes = [] + + def route(self, pattern, handler): + self.routes.append((pattern, handler)) + + def add_init_script(self, *a): + pass + + def new_page(self): + return "page" + + class _Browser: + version = "124.0.6367.60" + + def __init__(self): + self.ctx = _Ctx() + + def new_context(self, **k): + return self.ctx + + class _Chromium: + def launch(self, **k): + self.browser = _Browser() + return self.browser + + pwm = engines["playwright"] + for argv, want in ((["--proxy", "http://" + _UP + "127.0.0.1:1"], True), + (["--proxy", "http://" + _UP + "127.0.0.1:1", "--load-media"], False), + ([], False), (["--force-block-media"], True)): + _args = _engine_args(pwm, ["--url", listing] + argv) + _fake = ns(chromium=_Chromium()) + with _captured_output(): + pwm._launch_local(_fake, _args, _pp.from_args(_args)) + _routes = _fake.chromium.browser.ctx.routes + ok &= check(f"media: playwright {_label(argv)} -> " + f"{'route installed' if want else 'nothing blocked'}", + _routes == ([("**/*", _pp.playwright_media_route)] if want else [])) + + class _Client: + def __init__(self): + self.sent, self.handlers = [], {} + + def on(self, name, fn): + self.handlers[name] = fn + + async def send(self, cmd, params=None): + self.sent.append((cmd, params)) + + class _Stop(BaseException): + pass + + class _PPage: + url = listing + + def __init__(self, client): + self.target = ns(createCDPSession=self._session) + self._client = client + + async def _session(self): + return self._client + + async def authenticate(self, *a): + return None + + async def setUserAgent(self, *a): + return None + + async def goto(self, *a, **k): + raise _Stop() + + async def close(self): + return None + + class _PBrowser: + def __init__(self, page): + self.page = page + + async def pages(self): + return [self.page] + + async def version(self): + return "HeadlessChrome/124.0.0.0" + + async def close(self): + return None + + async def disconnect(self): + return None + + ppm = engines["pyppeteer"] + for argv, want in ((["--proxy", "http://" + _UP + "127.0.0.1:1"], True), + (["--proxy", "http://" + _UP + "127.0.0.1:1", "--load-media"], False), + ([], False), (["--force-block-media"], True), + (["--cdp-endpoint", "ws://127.0.0.1:1"], False)): + _client = _Client() + _browser = _PBrowser(_PPage(_client)) + + async def _give(_b=_browser, **k): + return _b + ppm.launch, ppm.connect = _give, _give + _args = _engine_args(ppm, ["--url", listing, "--out", "/nonexistent/x"] + argv) + with _captured_output(): + try: + _asyncio.run(ppm.scrape(_args)) + except _Stop: + pass + _enabled = [p for c, p in _client.sent if c == "Fetch.enable"] + ok &= check(f"media: pyppeteer {_label(argv)} -> " + f"{'Fetch pauses images/media/fonts' if want else 'no interception'}", + (len(_enabled) == 1 + and sorted(x["resourceType"] for x in _enabled[0]["patterns"]) + == ["Font", "Image", "Media"]) + if want else not _enabled) + if want and "Fetch.requestPaused" in _client.handlers: + async def _pause(_c=_client): + _c.handlers["Fetch.requestPaused"]( + {"requestId": "r1", "resourceType": "Image"}) + await _asyncio.sleep(0) + _asyncio.run(_pause()) + ok &= check("media: pyppeteer ...and fails what it paused", + _client.sent[-1] == ("Fetch.failRequest", + {"requestId": "r1", + "errorReason": "BlockedByClient"})) + + class _Opts: + def __init__(self): + self.arguments = [] + self.binary_location = None + + def add_argument(self, a): + self.arguments.append(a) + + class _SelDriver: + def __init__(self, **k): + self.capabilities = {} + self.cdp = [] + self.options = k.get("options") + + def execute_cdp_cmd(self, cmd, params): + self.cdp.append((cmd, params)) + + import shutil as _shutil + selm = engines["selenium"] + selm.Options, selm.Service = _Opts, (lambda **k: ns(**k)) + selm.webdriver = ns(Chrome=_SelDriver) + for argv, want in ((["--proxy", "http://" + _UP + "127.0.0.1:1"], True), + (["--proxy", "http://" + _UP + "127.0.0.1:1", "--load-media"], False), + ([], False), (["--force-block-media"], True)): + _args = _engine_args(selm, ["--url", listing, "--chromedriver", sys.executable] + argv) + with _captured_output(): + _drv = selm.build_driver(_args) + for a in _drv.options.arguments: + if a.startswith("--user-data-dir="): + _shutil.rmtree(a.split("=", 1)[1], ignore_errors=True) + _blocked = dict(_drv.cdp).get("Network.setBlockedURLs", {}).get("urls", []) + ok &= check(f"media: selenium {_label(argv)} -> " + f"{'images/video/fonts blocked by URL' if want else 'nothing blocked'}", + (all(p in _blocked for p in ("*.jpg", "*.webp?*", "*.mp4", "*.woff2")) + and not any(p.startswith(("*.js", "*.css")) for p in _blocked) + and _drv.media_blocked is True) + if want else (not _blocked and _drv.media_blocked is False)) + + # A source check as well, so removing a blocker call cannot hide behind a + # fake that happens not to exercise it. + for fname, call in (("playwright_scraper.py", "playwright_media_route"), + ("selenium_scraper.py", "selenium_block_media("), + ("puppeteer_scraper.py", "pyppeteer_block_media(")): + src = open(os.path.join(_repo_root, fname), encoding="utf-8").read() + ok &= check(f"media: {fname} consults the policy and calls its blocker", + "should_block_media(" in src and call in src) + + for name, mod in engines.items(): + buf = io.StringIO() + saved_argv = sys.argv + sys.argv = ["engine", "--help"] + try: + with contextlib.redirect_stdout(buf): + try: + mod.parse_args() + except SystemExit: + pass + finally: + sys.argv = saved_argv + _help = buf.getvalue() + ok &= check(f"media: {name} documents --load-media in --help and hides " + f"the test switch", + "--load-media" in _help and "--force-block-media" not in _help) + return ok + + def main() -> int: ok = True @@ -4679,6 +5138,7 @@ def main() -> int: ok = check_numeric_flags_are_range_checked_in_every_cli(ok) ok = check_the_dockerfile_s_copy_list_vs_the_entrypoint_s_i(ok) ok = check_audit_2026_10_14_contract_fixes(ok) + ok = check_proxy_traffic_savers(ok) # The report below is only worth something if it reflects what the # sections recorded: an engine whose driver library is absent must appear