[jehrr]

Problem

A run through a residential proxy is billed per byte, and nothing measured or limited what a run sent through it.

Resulting behaviour

Only for a browser this repo launches behind --proxy / --proxy-file / FARFETCH_PROXY. Nothing changes with --cdp-endpoint or with no proxy.

• Meter (proxy_meter.py) — an estimate of the tunnelled payload (TLS/CONNECT overhead, WebSocket frames and the browser's own background requests not included):
  • Playwright: request.sizes() (request headers + body, response headers + body) on requestfinished / requestfailed, in every worker under --concurrency too.
  • pyppeteer: a CDP session on the page counts Network.loadingFinished.encodedDataLength for http(s) responses that came over the network plus the request line/headers/body from Network.requestWillBeSent. Chrome's own error page, chrome:// pages and cache hits are not counted.
  • Selenium: the same events from Chrome's performance log, which is now enabled behind a proxy.
  • The proxy pre-check's HEAD counts too.
• Budget --proxy-budget-mb on all three engines (default 10, 0 = unlimited; a negative or non-number value is exit 2). Flag only: ENV_KEYS maps only credentials and URLs. Once the total passes it, no further listing page, worker page or product page is opened. Playwright aborts every further request (a route registered after the media route, else fallback()), and pyppeteer and Selenium set Network.setBlockedURLs ["*"]. The products read so far are kept. In --mode detail, a budget spent on the listing opens no product page, and listing rows are never written as size rows.
• Exit codes: finish_run puts stop_reason: proxy_budget_exceeded above every other outcome. That gives exit 6 with rows and 5 without, never 3 or 4.
• Sidecar: proxy_bytes, proxy_budget_bytes, proxy_budget_exceeded (null/null/false without a local proxy). A proxied run that writes no rows now records its outcome and bytes in <out>.latest_attempt.meta.json, and the data and .meta.json stay untouched. The run prints one line at the end: Proxy traffic: X.X MB of Y MB budget.
• Canary (production-like job): the run passes --proxy-budget-mb 10. An always-run step, .github/proxy_traffic_summary.py, appends **production-like**: proxy traffic X.X MB of 10 MB to $GITHUB_STEP_SUMMARY. If the run stopped at the budget, the step prints ::error::stopped: proxy budget 10 MB exceeded and fails. A missing or unreadable sidecar is reported and does not fail the step. The latest_attempt sidecar is uploaded with the artifacts.

Evidence (offline; no proxy, no paid call, no canary dispatched)

• tests/test_proxy_meter.py runs a stub forward proxy that serves a page of known size (~152 KB of html, js, css and a fetched JSON) and counts the bytes crossing its sockets. Against its count:

  • requests: within 2%.
  • Real Chromium via Playwright, pyppeteer and Selenium: within 10%.
  • Playwright with a 50 KB budget: the second navigation is ERR_BLOCKED_BY_CLIENT.

• CI engine jobs now install Playwright's Chromium and run this file with PROXY_METER_REQUIRE=<engine>, so a missing browser fails rather than skips.

• The real CLIs through the stub proxy (CONNECT answered 403, so farfetch.com is never reached):

  • all three exit 5 with Proxy traffic: 0.0 MB of 10 MB budget and an attempt sidecar recording proxy_bytes;
  • Playwright counted 521 bytes and the stub saw 524;
  • Selenium counted 212 bytes against a stub count of 1,195. An earlier draft counted 2.8 MB here: Chrome's new-tab page loads chrome:// scripts that are reported as finished loads. The CDP count is now limited to http(s) responses.

• Results per environment:

  • smoke_test.py: core 490/490, playwright 546, pyppeteer 491, selenium 505, all green;
  • pytest -q: 502 passed;
  • ruff clean; mypy clean, with proxy_meter.py added to the gated files.

• New checks, per engine through fake browsers:

  • budget hit on page 2 of 4: exit 6, 2 navigations, 4 rows, sidecar (1200, 1000, True), one summary line;
  • budget hit with no rows read: exit 5, attempt sidecar written, data untouched;
  • no proxy: nothing is metered;
  • budget 0: unlimited;
  • default 10 MB budget: exit 0 with the bytes recorded;
  • --concurrency 2: workers stop taking pages;
  • --mode detail: no product page is opened;
  • finish_run precedence against blocked and completed outcomes;
  • the flag on every CLI;
  • wiring (Playwright events/routes, Selenium performance log and tee, pyppeteer CDP session), only behind a local proxy;
  • the canary step and its script.

• An existing check ("the canary never puts the proxy credential in argv") matched the substring --proxy. It now looks for --proxy / --proxy= / --proxy-file, because --proxy-budget-mb 10 carries no credential.

• Planted-fault controls. Each went red on the named checks, then green once restored:

  • A, meter disabled: 27 smoke checks and 8 test_proxy_meter tests red;
  • B1, Playwright page-loop checks off: 3 red;
  • B2, worker-loop check off: the concurrency check red;
  • B3, detail-mode checks off: the detail check red;
  • B4, pyppeteer loop checks off: 3 red;
  • B5, Selenium loop checks off: 3 red;
  • C, finish_run override off: 18 red;
  • D, canary summary step replaced: the canary check red.

• Follow-up commit:

  • A budget that trips on a thread with no event loop now still sends the pyppeteer page's Network.setBlockedURLs: it is scheduled on the page's loop with call_soon_threadsafe. A new test triggers the budget from a worker thread; it went red with the old code.
  • The canary summary now orders sidecars by a parsed finished_at (epoch or ISO-8601; this repo writes ISO). The old direct comparison raised TypeError when one sidecar carried an ISO string and the other none. A new check covers that case and went red on the old code.

Not changed

No version bump or release, and no canary was dispatched. The pyppeteer page.authenticate proxy auth is unchanged (a known gap on current Chromium). Chrome's background traffic is not suppressed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8