[jehrr]

Problem

A run through a residential proxy is billed per byte, and nothing measured or limited what a run sent through it. A run that kept retrying or rotating, or a site page that grew, could spend a lot before anyone looked.

Resulting behaviour

This applies only to a local browser behind --proxy / --proxy-file / MEDIAMARKT_PROXY. Nothing changes over --cdp-endpoint or with no proxy.

• Meter (proxy_meter.py). The count is an estimate of the tunnelled payload. It does not include TLS/CONNECT overhead, WebSocket frames or Chrome's own background requests.
  • Playwright: request.sizes() on requestfinished / requestfailed for every proxied context, the concurrent workers' included.
  • pyppeteer: a CDP session per page counts Network.loadingFinished.encodedDataLength of responses that came over the network, plus the request line, headers and body. Chrome's own error page and cache hits are not counted.
  • Selenium: the same events from the performance log. It is turned on only when metering and polled after each navigation, before each document read and before quit().
  • The pre-check's HEAD is counted through a requests response hook.
  • .github/canary_preflight.py runs in its own process, before the scraper, so its request is not in the count.
• Budget --proxy-budget-mb: default 10, 0 = unlimited. A negative value or a non-number is a usage error that names the flag. It is a flag only, because ENV_KEYS maps only credentials and URLs. Once the total passes the budget, in all three engines:
  • no new page, navigation retry, exit rotation or captcha solve is started;
  • with Playwright --concurrency, no worker takes another page, and pages already loading finish;
  • Playwright aborts every further request through a route that runs before the media route and passes everything else on with fallback(). pyppeteer and Selenium set Network.setBlockedURLs ["*"];
  • the rows are kept, and the run ends with stop_reason: proxy_budget_exceeded: exit 6 with rows, 5 without. finish_run puts this above a block.
• Sidecar:
  • proxy_bytes, proxy_budget_bytes and proxy_budget_exceeded. They are null / null / false without a local proxy.
  • pages_unattempted after a stop.
  • A proxied run that writes no rows now records its outcome and byte count in <out>.latest_attempt.meta.json. The data and .meta.json are left untouched.
  • One line at the end of the run: Proxy traffic: X.X MB of Y MB budget.
• Canary: the run passes --proxy-budget-mb 10. An if: always() step, .github/proxy_traffic_summary.py, then appends **live-run**: proxy traffic X.X MB of 10 MB to $GITHUB_STEP_SUMMARY. When the budget was hit, it prints ::error::stopped: proxy budget 10 MB exceeded and fails.
• CI: the engine jobs install Playwright's Chromium and run tests/test_proxy_meter.py with PROXY_METER_REQUIRE=<engine>. A missing browser fails the job instead of skipping.

Evidence (offline; no proxy, no paid call, no canary dispatched)

• tests/test_proxy_meter.py. A stub forward proxy serves a page of known size (about 152 KB) and counts the bytes on its sockets.
  • requests: within 2% of the stub's count.
  • Real Chromium through Playwright, pyppeteer and Selenium: within 10%.
• Real CLIs through the stub proxy. The stub refuses CONNECT, so mediamarkt.de was never reached. Every run used --pages 3 --retries 3.
  • Budget 10: rc 5 in all three engines and the line Proxy traffic: 0.0 MB of 10 MB budget.
  • --proxy-budget-mb 0.0001: rc 5, stop_reason: proxy_budget_exceeded, and proxy_bytes of 217–654 in the attempt sidecar.
• Suites. smoke_test.py is OK in the core, playwright, pyppeteer and selenium venvs, and each engine's checks ran in its own venv. pytest: 11 passed, 4 skipped.
• New checks:
  • Flag default and validation; the meter is active only behind a local proxy.
  • finish_run precedence for rows/blocked/failed combinations, giving 6 or 5 with proxy_budget_exceeded.
  • For each engine's real scrape(), with the browser and page fetch stubbed:
    • budget spent on page 2 of 5: exit 6, pages 1–2 only, sidecar (1200, 1000, True), pages_unattempted [3,4,5];
    • spent on a refused page 1: exit 5 rather than 3, with only the attempt sidecar;
    • no proxy: nothing metered;
    • budget 0: unlimited.
  • Concurrent workers stop once the budget is spent.
  • The real Playwright _fetch_one_page neither retries a timed-out navigation nor rotates away from a dead exit on a spent budget.
  • The canary step and its script.
• Planted-fault controls. Each went red on the named check, then green after the restore:
  • A, meter add() disabled: 10 smoke and 8 test_proxy_meter failures.
  • B1, page-loop check removed (Playwright, and separately Selenium): that engine's budget-stop checks.
  • B2, retry-loop check: "a timed-out navigation … is not retried".
  • B2b, rotation check: "nor is a dead exit rotated away from".
  • B3, worker-loop check: concurrent dispatch.
  • B4, finish_run override: 7 checks.
  • C, canary summary step replaced: canary check.

Not changed

• No version bump, release or canary dispatch.
• Chrome's own background traffic is not suppressed.
• The canary preflight (a separate process) is not metered.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8