From d16c3beb03a4fa0bb55fc9d10ec3f4dde4080849 Mon Sep 17 00:00:00 2001 From: Petr Date: Fri, 25 Sep 2026 13:43:39 +0000 Subject: [PATCH] pyppeteer: answer proxy auth over CDP Fetch; re-measure the proxy path (v0.1.1) pyppeteer's page.authenticate relies on Network.setRequestInterception, which current Chromium removed, so any credentialled --proxy died before the first navigation. Answered through Fetch.authRequired instead. Measured after the fix: the residential exits reach the page on 2 of 3 tries and are refused by the listing API, so the docs now say exactly that. Co-Authored-By: Claude Opus 5.5 --- .env.example | 9 ++++---- CHANGELOG.md | 22 +++++++++++++++++++ README.md | 10 +++++---- puppeteer_scraper.py | 52 +++++++++++++++++++++++++++++++++++++++----- pyproject.toml | 2 +- smoke_test.py | 18 +++++++++++++++ 6 files changed, 99 insertions(+), 14 deletions(-) diff --git a/.env.example b/.env.example index 36fcdbd..329b55c 100644 --- a/.env.example +++ b/.env.example @@ -70,10 +70,11 @@ MAKEMYTRIP_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-in-pid-{profi # One URL here, or --proxy-file with one per line for a pool. # # NOT measured to work: three 2Captcha residential logins (eu, na, and one -# with -region-in) all ended in net::ERR_HTTP2_PROTOCOL_ERROR from local -# Chromium on this machine, 2026-09-24. Whether that is the exits or this -# machine's own Chromium being scored has not been separated, so treat the -# proxy path as untested rather than as broken. +# with -region-in) all ended in net::ERR_HTTP2_PROTOCOL_ERROR from +# Playwright's Chromium on this machine, 2026-09-24. Through pyppeteer on +# 2026-09-25 the same exits got the PAGE on 2 of 3 tries and were then +# refused by the listing API (no CORS permission on its answer), so no rows +# came back. Treat the proxy path as unproven rather than as broken. # # The credentials never reach the browser's command line: they go through # the driver's own fields, and logs mask them while keeping host and port. diff --git a/CHANGELOG.md b/CHANGELOG.md index 721787d..5983f56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,28 @@ toolkit can: a patch release means **fixes**, not that every flag and default is frozen. A default that changes behaviour for an existing user is said so at the top of its release notes. +## [0.1.1] — 2026-09-25 + +### Fixed + +- **pyppeteer with a credentialled `--proxy` died before its first + navigation**, with "'Network.setRequestInterception' wasn't found": + pyppeteer's `page.authenticate` relies on a CDP method current Chromium + removed. The engine now answers the proxy's challenge itself through the + `Fetch` domain, giving the credentials only to a proxy's challenge. + Measured after the fix: ipify answered through the proxy on 4 of 4 runs. + +### Measured + +- Through pyppeteer, the 2Captcha residential exits got makemytrip.com's + page on 2 of 3 tries, and then the listing API refused the exit (its + answer carried no CORS permission). So the proxy path still returns no + rows, and the README and `.env.example` say so in those terms. The + Scraping Browser with `country-in` remains the one path measured to work. +- The canary's first runs from a GitHub runner: refused by Akamai and + reported as exit 3, as the README says of a datacentre; and through a + Scraping Browser profile, 90 of 90 rows priced over three pages. + ## [0.1.0] — 2026-09-24 First release. makemytrip.com's hotel listings, read from the site's own diff --git a/README.md b/README.md index 8bd08b7..57ec597 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,8 @@ datacentre VPS (netcup, AS197540): | real Chromium, headless **and** headful | `net::ERR_HTTP2_PROTOCOL_ERROR`, every page | | Chrome under Selenium with its User-Agent overridden | HTTP 200 and a body of `200-OK`: a decoy, not the page | | the listing API itself (`mapi.makemytrip.com`) | HTTP 403, Akamai "Access Denied" | -| three 2Captcha residential exits (eu, na, and one with `-region-in`) | `net::ERR_HTTP2_PROTOCOL_ERROR` | +| three 2Captcha residential exits (eu, na, and one with `-region-in`), Playwright | `net::ERR_HTTP2_PROTOCOL_ERROR` | +| the same exits, pyppeteer (2026-09-25) | the page on 2 of 3 tries, then the listing API **refused** the exit: its answer carried no CORS permission, the shape Akamai's 403 has from a page | | the 2Captcha Scraper API on its own exits | HTTP 403, Akamai "Access Denied" | | **the Scraping Browser with an Indian exit (`country-in`)** | **HTTP 200, the full listing** | @@ -62,8 +63,9 @@ the service served the reconnect from a different Indian exit. The engines do that reconnect by themselves. The residential proxies above are recorded as **not measured to work**, not -as broken: whether it was those exits or this machine's own Chromium being -scored was not separated. A home connection in India has not been tried. +as broken: through pyppeteer they got past the page and were refused by the +API, and why Playwright's browser was refused earlier on the same exits was +not separated. A home connection in India has not been tried. --- @@ -169,7 +171,7 @@ counts together, so an average over the column means something. | | | |---|---| | `playwright_scraper.py` | **Primary.** Authenticates a proxy and a remote CDP endpoint. Run live, above. | -| `puppeteer_scraper.py` | pyppeteer is effectively unmaintained; here for parity. Run live through the Scraping Browser, above. `--chromium-path` points it at another browser. A credentialled `--proxy` is known NOT to authenticate through it on current Chromium (`page.authenticate` relies on a CDP method Chromium removed; found in a sibling repo, 2026-09-24) and is not fixed here, since no proxy was measured to help on this site anyway. | +| `puppeteer_scraper.py` | pyppeteer is effectively unmaintained; here for parity. Run live through the Scraping Browser, above. `--chromium-path` points it at another browser. It answers a proxy's credentials over CDP (`Fetch.authRequired`): pyppeteer's own `page.authenticate` relies on a method current Chromium removed, and until v0.1.1 this engine died on any credentialled proxy before its first navigation. | | `selenium_scraper.py` | Drives a Chrome you have. **Cannot authenticate a remote CDP endpoint** (`debuggerAddress` is a bare `host:port`), so it refuses the Scraping Browser with exit 2, and that is the one path measured to work. From a datacentre it is therefore refused on every run, and reports it as exit 3: verified live against the `200-OK` decoy this engine's Chrome is served, and against a captured copy of Chrome's own error page, which Selenium reports under the REQUESTED address. A successful Selenium run has not been measured. | | `scraper_api_client.py` | No local browser: the 2Captcha Scraper API fetches the listing PAGE, routed through a Scraping Browser session (`cdpurl`), and this client reads the page's server-rendered state. That is **the listing's first five properties, and nothing more**: everything after them is a POST endpoint, and the Scraper API fetches URLs. `--pages` above 1 is refused for that reason. | diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index 10d3333..e972f41 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -21,7 +21,8 @@ Playwright. It is here for parity, and for anyone who already has it. * Unlike the Selenium engine, it CAN authenticate a remote CDP endpoint (`browserWSEndpoint` takes a full `ws://user:pass@host:port`) and a - proxy (`page.authenticate`). + proxy (through the CDP `Fetch` domain, since pyppeteer's own + `page.authenticate` no longer works on current Chromium). Usage ----- @@ -267,8 +268,8 @@ def open(self): credentials = None if self.pool: exit_url = self.pool.current - # Credentials go through page.authenticate(), never onto the - # command line: --proxy-server= is part of the browser's argv, + # Credentials are answered over CDP (_authenticate_proxy), never + # put on the command line: --proxy-server= is part of the browser's argv, # readable by anything that can run `ps` (§8). scrubbed, credentials = split_credentials(exit_url) launch_args.append(f"--proxy-server={scrubbed}") @@ -287,10 +288,51 @@ def open(self): if self.args.fingerprint: self._apply_fingerprint() if credentials: - self.bridge.run(self.page.authenticate( - {"username": credentials[0], "password": credentials[1]})) + self._authenticate_proxy(*credentials) return self + def _authenticate_proxy(self, username: str, password: str): + """Answer the proxy's 407 through the CDP `Fetch` domain. + + pyppeteer's own `page.authenticate` is built on + `Network.setRequestInterception`, which current Chromium no longer + has. Measured here on 2026-09-25 with Playwright's Chromium under + --chromium-path: "'Network.setRequestInterception' wasn't found", + and the run died before its first navigation, so a credentialled + proxy never worked through this engine. `Fetch.enable` with + `handleAuthRequests` is what replaced it (and what Playwright + uses); the approach was worked out in a sibling repo the same day. + + Every request is paused and continued unchanged; only an auth + challenge FROM THE PROXY is answered with its credentials, which + never leave this process's memory (§8). A site's own challenge gets + the browser's default answer. + """ + async def _enable(): + session = await self.page.target.createCDPSession() + + def _paused(event): + asyncio.ensure_future(session.send( + "Fetch.continueRequest", {"requestId": event["requestId"]})) + + def _auth(event): + source = (event.get("authChallenge") or {}).get("source") + response = ({"response": "ProvideCredentials", + "username": username, "password": password} + if source == "Proxy" else {"response": "Default"}) + asyncio.ensure_future(session.send( + "Fetch.continueWithAuth", + {"requestId": event["requestId"], + "authChallengeResponse": response})) + + session.on("Fetch.requestPaused", _paused) + session.on("Fetch.authRequired", _auth) + await session.send("Fetch.enable", {"handleAuthRequests": True, + "patterns": [{"urlPattern": "*"}]}) + return session + + self._auth_session = self.bridge.run(_enable(), timeout=30) + def _enable_autosolve(self): """The Scraping Browser API's own CAPTCHA domain, as the Playwright engine enables it: if a captcha ever appears in front of the landing, diff --git a/pyproject.toml b/pyproject.toml index 8025200..4f31ffc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "makemytrip-scraper" -version = "0.1.0" +version = "0.1.1" description = "makemytrip.com hotel listing scraper: prices before and after taxes, star and guest ratings, property type, location, from the site's own listing API (Playwright, Selenium or pyppeteer)." readme = "README.md" license = "MIT" diff --git a/smoke_test.py b/smoke_test.py index a6a7d37..1ef771f 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -1257,6 +1257,24 @@ def check_selenium_reads_chromes_own_error_page(): "location.href" in src and "current_url" not in src.split('"""')[-1]) +def check_pyppeteer_answers_proxy_auth_over_cdp(): + """pyppeteer's page.authenticate relies on Network.setRequestInterception, + which current Chromium removed: measured, the engine died before its + first navigation with any credentialled proxy. The engine answers the + proxy's challenge through the Fetch domain instead, and only a PROXY's.""" + src = open(os.path.join(HERE, "puppeteer_scraper.py"), encoding="utf-8").read() + tree = ast.parse(src) + called = {n.func.attr for n in ast.walk(tree) + if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)} + check("the engine never CALLS page.authenticate", "authenticate" not in called) + check("...it enables Fetch with handleAuthRequests", + '"Fetch.enable", {"handleAuthRequests": True' in src) + check("...answers Fetch.authRequired, continues every paused request", + "Fetch.authRequired" in src and "Fetch.continueRequest" in src) + check("...and gives the credentials only to a Proxy challenge", + 'if source == "Proxy"' in src) + + def check_credentials_never_reach_a_log(): for module in ENGINES: engine = _import_engine(module)