[jehrr]

Problem

A residential proxy is billed by the byte, and a run had no idea how many bytes it had used. A slow page, a retry loop or a multi-page listing could spend a proxy balance with no record anywhere except the provider's invoice. Nothing capped it either. This site needs a residential exit for every live run, so this is every live run.

What changes

Only when traffic goes through a local proxy (--proxy / --proxy-file / SLEEPNUMBER_PROXY). Nothing changes without a proxy or over --cdp-endpoint (the 2Captcha Scraping Browser API has its own billing).

• Meter (proxy_meter.py), in every entry point:
  • Playwright: the sum of request.sizes() (request headers + body, response headers + body) on requestfinished / requestfailed.
  • pyppeteer: CDP Network.loadingFinished.encodedDataLength (bytes received, headers included) plus each request's line, headers and body from Network.requestWillBeSent, on a CDP session of its own. Requests the browser blocked itself (media blocking) cost nothing.
  • Selenium: the same CDP events, from chromedriver's performance log. The log is now requested at driver creation and read before and after each navigation and before the driver quits.
  • http_scraper.py: each response's status line, headers and body (its Content-Length, i.e. the compressed size, when sent) plus the request line, headers and body, for every redirect hop.
  • It is an estimate of the payload through the tunnel. TLS, the CONNECT exchange and the browser's own background connections are not visible to any engine and add a few percent. The pre-check (under 1 KB) is not counted.
• Budget: --proxy-budget-mb (default 10, 0 = no cap, SLEEPNUMBER_PROXY_BUDGET_MB in ENV_KEYS and .env.example). Negative or non-numeric values are a usage error (exit 2). Once the total is over the budget:
  • Playwright aborts every new request in a route registered after the media route, so it runs first.
  • pyppeteer and Selenium send Network.setBlockedURLs ["*"].
  • Every entry point raises before its next page request or retry.
  • Rows already gathered are kept, stop_reason: proxy_budget_exceeded, exit 6 with rows and exit 5 without.
  • The stop is a BaseException, so the broad except Exception retry loops cannot swallow it and retry.
  • With --concurrency (Playwright, --mode transfers), the workers share one meter. Each stops at its next page, catches the stop itself instead of dying with a traceback, and the queued pages are reported unattempted.
• Sidecar: proxy_bytes, proxy_budget_bytes (null with no cap) and proxy_budget_exceeded go in <out>.meta.json. Every metered run also writes <out>.latest_attempt.meta.json, so a run that wrote no rows still records what it spent. Without a proxy no such file is written, as before. The log ends with Proxy traffic: X.X MB of Y MB budget.
• Canary: all four runs (three Playwright categories and the http_scraper.py run) pass --proxy-budget-mb 10. A new step runs .github/proxy_traffic.py for each, appending **live/<category>**: proxy traffic X.X MB of 10 MB (or "not metered" / "unknown") to $GITHUB_STEP_SUMMARY. When any run hit the cap it prints ::error::stopped: proxy budget 10 MB exceeded and fails.
• README (proxy savers, exit codes, sidecars), --help of every entry point, .env.example, Dockerfile COPY list, CHANGELOG [Unreleased].
• pyproject.toml py-modules gains proxy_meter and also proxy_check, which every engine imports and the list was missing.

Evidence

• Offline suite, core-only venv: python smoke_test.py → 163/163. It covers:

  • the meter against known Playwright sizes() and CDP events (exact byte counts), plus blocked, failed and redirect cases;
  • http_scraper.py through a byte-counting stub proxy: metered 30,517 B against 30,793 B carried, 0.9% drift, which includes the uncounted pre-check;
  • a 30 kB page past a 10 kB budget: exit 5, proxy_budget_exceeded, exactly one page request;
  • with rows: exit 6, no third page;
  • over budget, the HTTP fetch makes no request;
  • the canary step, plus the summary script run against over-budget, under-budget, no-proxy and missing sidecars.

• Engine venvs (playwright 1.64, pyppeteer, selenium; no browser download), with the opt-in real-browser check (PROXY_METER_BROWSER=1, local Chromium): 180/180, 179/179 and 179/179. Each engine:

  • installs the meter through a proxy and not without one;
  • refuses to navigate over budget;
  • driven past the budget on page 1 of 3, exits 6 with rows kept and no third page; with no rows, exits 5 with the attempt sidecar.

  Two --concurrency workers stop once the shared budget is spent: page 2 fetched, pages 3–7 unattempted, no traceback.

• Real browser, through a local stub proxy that counted every byte it carried (60 kB page + 30 kB CSS + 90 kB JS):

  Driver	Metered	Stub proxy carried	Drift
  Playwright	181,007 B	181,433 B	0.2%
  pyppeteer	180,859 B	181,936 B	0.6%
  Selenium	181,167 B	182,431 B	0.7%

  Past a 0.15 MB budget, a request the page made never reached the proxy.

• Planted-fault controls. Each went red on the named check and was green again after restoring:

  • meter disabled;
  • proxy_meter.check no-op;
  • the budget check removed from the fetch, in http_scraper.py, Playwright, pyppeteer and Selenium (each red on its own "over budget, the page fetch stops before …");
  • http_scraper.py not counting responses;
  • the Playwright worker not catching the stop ("two workers stop once the shared budget is spent");
  • the attempt sidecar not written;
  • the http summary line removed from canary.yml;
  • the summary script not failing over budget.

• .github/ci_checks.py --secret-check --help-check --sample-check: ok. python env_config.py: ok.

• No proxy, no paid call and no canary dispatch was used.

Not changed

• Nothing for runs without a proxy or over --cdp-endpoint.
• No release or version bump.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8