Skip to content

Count only network traffic in the pyppeteer and Selenium proxy meter - #12

Merged
jehrr merged 1 commit into
mainfrom
fix/proxy-meter-network-only-2026-10-21
Oct 21, 2026
Merged

jehrr merged 1 commit into
mainfrom
fix/proxy-meter-network-only-2026-10-21

Conversation

@jehrr

@jehrr jehrr commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Problem

In pyppeteer and Selenium, the proxy meter added in #11 counted every CDP loadingFinished. That included loads that never cross a proxy:

  • Chrome's own chrome:// new-tab scripts;
  • data: and blob: URLs;
  • cache hits;
  • Chrome's error page, which is reported as a finished load of the failed URL with no responseReceived.

A sibling repo's Selenium CLI, run through a stub proxy, showed the meter reading 2,779,683 bytes while the proxy carried about 6 KB. That spends the default 10 MB --proxy-budget-mb on traffic that never happened.

Separately, the pyppeteer block sent once the budget is spent used asyncio.ensure_future from whatever thread crossed the cap. A thread with no event loop raised there, the error was swallowed, and the block was never sent.

Resulting behaviour

  • CdpCounter (proxy_meter.py) counts only http:// / https:// requests.
  • Response bytes are counted only after a responseReceived that came over the network. A response marked requestServedFromCache, fromDiskCache, fromPrefetchCache or fromServiceWorker counts nothing, and neither does its request. Chrome's error page costs only the request that was attempted.
  • responseReceived and requestServedFromCache are now subscribed (pyppeteer) and read from the performance log (Selenium).
  • The pyppeteer block goes to the page's own loop with call_soon_threadsafe.
  • Playwright, sidecar fields, exit codes and canary.yml are unchanged.

Evidence

  • New offline checks:
    • a chrome://new-tab-page/x.js request, response and finish (2,779,683 B), plus a data: and a blob: load, count 0;
    • a cache hit via requestServedFromCache, fromDiskCache, fromPrefetchCache and fromServiceWorker counts 0;
    • Chrome's error page counts only the attempted request;
    • a budget crossed on a worker thread still sends Network.setBlockedURLs ["*"] on the page's loop.
  • Planted-fault controls. Each went red on the named check and green again after restoring:
    • scheme filter removed: "chrome:// new-tab load … cost nothing" went red;
    • cache flags ignored: "a cache hit … costs nothing" went red;
    • the block sent from the tripping thread (the previous code): "a budget crossed on a worker thread …" went red.
  • Suite passes in a core-only venv and in the pyppeteer and Selenium venvs (real-browser check on).
  • Real browser, through a byte-counting stub proxy (opt-in check): pyppeteer still meters within 0.6% and Selenium within 0.7%.
  • No proxy, no paid call, no canary dispatch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8

Chrome's own chrome:// new-tab loads, data:/blob: URLs, cache hits and
its error page were counted as proxy traffic. Only http(s) requests and
responses that came over the network are counted now, and the pyppeteer
budget block reaches the page's loop when the budget runs out on
another thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8
@jehrr
jehrr merged commit a569902 into main Oct 21, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant