[jehrr]

Problem

A residential proxy is billed by the byte, and a run had no idea how many bytes it had used. A retry loop, a long walk of pages or a --concurrency run across several workers could spend a proxy balance with no record anywhere except the provider's invoice. Nothing capped it either.

What changes

Only when traffic goes through a local proxy (--proxy / --proxy-file / WEBMOTORS_PROXY). Nothing changes without a proxy or over --cdp-endpoint (the 2Captcha Scraping Browser API has its own billing).

• Meter (proxy_meter.py), installed by each engine on every browser it opens, every worker's included. All of them count into one budget per run.
  • Playwright: the sum of request.sizes() (request headers + body, response headers + body) on requestfinished / requestfailed.
  • pyppeteer: CDP Network.loadingFinished.encodedDataLength (bytes received, headers included) plus each request's line, headers and body from Network.requestWillBeSent, on a CDP session of its own. Requests the browser blocked itself (--block-media) cost nothing.
  • Selenium: the same CDP events, read from chromedriver's performance log (requested at driver creation) after every navigation and API fetch(), and before the driver quits.
  • It is an estimate of the payload through the tunnel. TLS, the CONNECT exchange and the browser's own background connections are not visible to any engine and add a few percent. The pre-check (under 1 KB) is not counted.
• Budget: --proxy-budget-mb (default 10, 0 = no cap, WEBMOTORS_PROXY_BUDGET_MB). Negative or non-numeric values are a usage error (exit 2). An unparseable env value stops the run rather than falling back. The check lives in the shared loop (page_flow), so the three engines cannot drift. Once the total is over the budget:
  • Playwright aborts every new request in a route, and pyppeteer and Selenium send Network.setBlockedURLs ["*"].
  • fetch_one_page stops before its next landing or API request.
  • Each --concurrency worker stops at its next request and returns its page to the queue, so that page is reported unattempted, not fetched.
  • An advert's optional market-price request and the currency read are skipped, leaving those columns null.
  • Rows already gathered are kept, stop_reason: proxy_budget_exceeded, exit 6 with rows and exit 5 without.
  • The stop is a BaseException, so a worker's broad except Exception cannot swallow it.
• Sidecar: proxy_bytes, proxy_budget_bytes (null with no cap) and proxy_budget_exceeded go in <out>.meta.json. Every metered run also writes <out>.latest_attempt.meta.json, so a run that wrote no rows still records what it spent. Without a proxy no such file is written, as before. The log ends with Proxy traffic: X.X MB of Y MB budget.
• Canary: the search and ad-mode runs pass --proxy-budget-mb 10. After each run, .github/proxy_traffic.py appends **search**: proxy traffic X.X MB of 10 MB (or "not metered" over WEBMOTORS_CDP_ENDPOINT, "unknown" when there is no sidecar) to $GITHUB_STEP_SUMMARY. When the cap was hit it prints ::error::stopped: proxy budget 10 MB exceeded and fails.
• Media still loads by default on this site, so a proxied run spends more per page than one with --block-media. Rows come from the JSON API, so pages are small, but the landing renders the site. If a run reaches 10 MB the canary goes red with that message, which is the intended signal.
• README (proxy savers section, exit codes), --help, .env.example (commented out, like the other optional settings), Dockerfile COPY list, CHANGELOG [Unreleased].

Evidence

• Offline suite, core-only venv: python smoke_test.py → 537 checks passed, 0 failed, and .github/ci_checks.py --all → OK. The new checks cover:

  • the meter against known Playwright sizes() and CDP events (exact byte counts), plus blocked, failed and redirect cases;
  • an HTTP client through a byte-counting stub proxy: 0.1% drift;
  • the sidecars and exit codes 6/5;
  • the shared loop on fake operations with known costs:
    • sequential: budget crossed on page 2 of 3 → exit 6, 10 rows, page 3 never requested;
    • two workers: both stop at their next request, exit 6;
    • spent on the landing → exit 5 with the attempt sidecar;
    • over budget: fetch_one_page makes no request at all, not even the landing;
  • each engine installs the meter through a proxy and not without one;
  • the canary steps, plus the summary script run against over-budget, under-budget, no-proxy and missing sidecars.

• Engine venvs (playwright 1.64, pyppeteer, selenium; no browser download), with the opt-in real-browser check (PROXY_METER_BROWSER=1, local Chromium): 565 / 553 / 553 checks passed, 0 failed.

• Real browser (the shared proxy_meter install functions the engines call), through a local stub proxy that counted every byte it carried (60 kB page + 30 kB CSS + 90 kB JS):

  Driver	Metered	Stub proxy carried	Drift
  Playwright	181,007 B	181,433 B	0.2%
  pyppeteer	180,859 B	181,936 B	0.6%
  Selenium	181,167 B	182,431 B	0.7%

  Past a 0.15 MB budget, a request the page made never reached the proxy, and check() stopped the run.

• Planted-fault controls. Each went red on the named check and was green again after restoring:

  • meter disabled: "sequential: budget crossed on page 2 of 3…" (rc 0, page 3 fetched);
  • proxy_meter.check made a no-op: the sequential and concurrent checks;
  • the loop's check before landing removed: "over budget, the shared fetch makes no request at all, not even the landing" (gotos=1);
  • the loop's check before the API fetch removed: "budget spent on the landing, before any row → exit 5…";
  • a worker no longer catching the stop: "concurrent: every worker stops at its next request…";
  • the ad-mode summary step removed from canary.yml: "each run's proxy traffic goes to the step summary…";
  • the summary script no longer failing over budget: "over budget: '::error::stopped: proxy budget 10 MB exceeded'…".

• No proxy, no paid call and no canary dispatch was used.

Not changed

• Nothing for runs without a proxy or over --cdp-endpoint.
• Media blocking stays opt-in.
• No release or version bump.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8