pyppeteer: answer proxy auth over CDP Fetch; re-measure the proxy path (v0.1.1)

.env.example

@@ -70,10 +70,11 @@ MAKEMYTRIP_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-in-pid-{profi
 # One URL here, or --proxy-file with one per line for a pool.
 #
 # NOT measured to work: three 2Captcha residential logins (eu, na, and one
-# with -region-in) all ended in net::ERR_HTTP2_PROTOCOL_ERROR from local
-# Chromium on this machine, 2026-09-24. Whether that is the exits or this
-# machine's own Chromium being scored has not been separated, so treat the
-# proxy path as untested rather than as broken.
+# with -region-in) all ended in net::ERR_HTTP2_PROTOCOL_ERROR from
+# Playwright's Chromium on this machine, 2026-09-24. Through pyppeteer on
+# 2026-09-25 the same exits got the PAGE on 2 of 3 tries and were then
+# refused by the listing API (no CORS permission on its answer), so no rows
+# came back. Treat the proxy path as unproven rather than as broken.
 #
 # The credentials never reach the browser's command line: they go through
 # the driver's own fields, and logs mask them while keeping host and port.

CHANGELOG.md

@@ -7,6 +7,28 @@ toolkit can: a patch release means **fixes**, not that every flag and
 default is frozen. A default that changes behaviour for an existing user is
 said so at the top of its release notes.
 
+## [0.1.1] — 2026-09-25
+
+### Fixed
+
+- **pyppeteer with a credentialled `--proxy` died before its first
+  navigation**, with "'Network.setRequestInterception' wasn't found":
+  pyppeteer's `page.authenticate` relies on a CDP method current Chromium
+  removed. The engine now answers the proxy's challenge itself through the
+  `Fetch` domain, giving the credentials only to a proxy's challenge.
+  Measured after the fix: ipify answered through the proxy on 4 of 4 runs.
+
+### Measured
+
+- Through pyppeteer, the 2Captcha residential exits got makemytrip.com's
+  page on 2 of 3 tries, and then the listing API refused the exit (its
+  answer carried no CORS permission). So the proxy path still returns no
+  rows, and the README and `.env.example` say so in those terms. The
+  Scraping Browser with `country-in` remains the one path measured to work.
+- The canary's first runs from a GitHub runner: refused by Akamai and
+  reported as exit 3, as the README says of a datacentre; and through a
+  Scraping Browser profile, 90 of 90 rows priced over three pages.
+
 ## [0.1.0] — 2026-09-24
 
 First release. makemytrip.com's hotel listings, read from the site's own

README.md

@@ -44,7 +44,8 @@ datacentre VPS (netcup, AS197540):
 | real Chromium, headless **and** headful | `net::ERR_HTTP2_PROTOCOL_ERROR`, every page |
 | Chrome under Selenium with its User-Agent overridden | HTTP 200 and a body of `200-OK`: a decoy, not the page |
 | the listing API itself (`mapi.makemytrip.com`) | HTTP 403, Akamai "Access Denied" |
-| three 2Captcha residential exits (eu, na, and one with `-region-in`) | `net::ERR_HTTP2_PROTOCOL_ERROR` |
+| three 2Captcha residential exits (eu, na, and one with `-region-in`), Playwright | `net::ERR_HTTP2_PROTOCOL_ERROR` |
+| the same exits, pyppeteer (2026-09-25) | the page on 2 of 3 tries, then the listing API **refused** the exit: its answer carried no CORS permission, the shape Akamai's 403 has from a page |
 | the 2Captcha Scraper API on its own exits | HTTP 403, Akamai "Access Denied" |
 | **the Scraping Browser with an Indian exit (`country-in`)** | **HTTP 200, the full listing** |
 
@@ -62,8 +63,9 @@ the service served the reconnect from a different Indian exit. The engines
 do that reconnect by themselves.
 
 The residential proxies above are recorded as **not measured to work**, not
-as broken: whether it was those exits or this machine's own Chromium being
-scored was not separated. A home connection in India has not been tried.
+as broken: through pyppeteer they got past the page and were refused by the
+API, and why Playwright's browser was refused earlier on the same exits was
+not separated. A home connection in India has not been tried.
 
 ---
 
@@ -169,7 +171,7 @@ counts together, so an average over the column means something.
 | | |
 |---|---|
 | `playwright_scraper.py` | **Primary.** Authenticates a proxy and a remote CDP endpoint. Run live, above. |
-| `puppeteer_scraper.py` | pyppeteer is effectively unmaintained; here for parity. Run live through the Scraping Browser, above. `--chromium-path` points it at another browser. A credentialled `--proxy` is known NOT to authenticate through it on current Chromium (`page.authenticate` relies on a CDP method Chromium removed; found in a sibling repo, 2026-09-24) and is not fixed here, since no proxy was measured to help on this site anyway. |
+| `puppeteer_scraper.py` | pyppeteer is effectively unmaintained; here for parity. Run live through the Scraping Browser, above. `--chromium-path` points it at another browser. It answers a proxy's credentials over CDP (`Fetch.authRequired`): pyppeteer's own `page.authenticate` relies on a method current Chromium removed, and until v0.1.1 this engine died on any credentialled proxy before its first navigation. |
 | `selenium_scraper.py` | Drives a Chrome you have. **Cannot authenticate a remote CDP endpoint** (`debuggerAddress` is a bare `host:port`), so it refuses the Scraping Browser with exit 2, and that is the one path measured to work. From a datacentre it is therefore refused on every run, and reports it as exit 3: verified live against the `200-OK` decoy this engine's Chrome is served, and against a captured copy of Chrome's own error page, which Selenium reports under the REQUESTED address. A successful Selenium run has not been measured. |
 | `scraper_api_client.py` | No local browser: the 2Captcha Scraper API fetches the listing PAGE, routed through a Scraping Browser session (`cdpurl`), and this client reads the page's server-rendered state. That is **the listing's first five properties, and nothing more**: everything after them is a POST endpoint, and the Scraper API fetches URLs. `--pages` above 1 is refused for that reason. |
 

puppeteer_scraper.py

@@ -21,7 +21,8 @@
     Playwright. It is here for parity, and for anyone who already has it.
   * Unlike the Selenium engine, it CAN authenticate a remote CDP endpoint
     (`browserWSEndpoint` takes a full `ws://user:pass@host:port`) and a
-    proxy (`page.authenticate`).
+    proxy (through the CDP `Fetch` domain, since pyppeteer's own
+    `page.authenticate` no longer works on current Chromium).
 
 Usage
 -----
@@ -267,8 +268,8 @@ def open(self):
         credentials = None
         if self.pool:
             exit_url = self.pool.current
-            # Credentials go through page.authenticate(), never onto the
-            # command line: --proxy-server= is part of the browser's argv,
+            # Credentials are answered over CDP (_authenticate_proxy), never
+            # put on the command line: --proxy-server= is part of the browser's argv,
             # readable by anything that can run `ps` (§8).
             scrubbed, credentials = split_credentials(exit_url)
             launch_args.append(f"--proxy-server={scrubbed}")
@@ -287,10 +288,51 @@ def open(self):
         if self.args.fingerprint:
             self._apply_fingerprint()
         if credentials:
-            self.bridge.run(self.page.authenticate(
-                {"username": credentials[0], "password": credentials[1]}))
+            self._authenticate_proxy(*credentials)
         return self
 
+    def _authenticate_proxy(self, username: str, password: str):
+        """Answer the proxy's 407 through the CDP `Fetch` domain.
+
+        pyppeteer's own `page.authenticate` is built on
+        `Network.setRequestInterception`, which current Chromium no longer
+        has. Measured here on 2026-09-25 with Playwright's Chromium under
+        --chromium-path: "'Network.setRequestInterception' wasn't found",
+        and the run died before its first navigation, so a credentialled
+        proxy never worked through this engine. `Fetch.enable` with
+        `handleAuthRequests` is what replaced it (and what Playwright
+        uses); the approach was worked out in a sibling repo the same day.
+
+        Every request is paused and continued unchanged; only an auth
+        challenge FROM THE PROXY is answered with its credentials, which
+        never leave this process's memory (§8). A site's own challenge gets
+        the browser's default answer.
+        """
+        async def _enable():
+            session = await self.page.target.createCDPSession()
+
+            def _paused(event):
+                asyncio.ensure_future(session.send(
+                    "Fetch.continueRequest", {"requestId": event["requestId"]}))
+
+            def _auth(event):
+                source = (event.get("authChallenge") or {}).get("source")
+                response = ({"response": "ProvideCredentials",
+                             "username": username, "password": password}
+                            if source == "Proxy" else {"response": "Default"})
+                asyncio.ensure_future(session.send(
+                    "Fetch.continueWithAuth",
+                    {"requestId": event["requestId"],
+                     "authChallengeResponse": response}))
+
+            session.on("Fetch.requestPaused", _paused)
+            session.on("Fetch.authRequired", _auth)
+            await session.send("Fetch.enable", {"handleAuthRequests": True,
+                                                "patterns": [{"urlPattern": "*"}]})
+            return session
+
+        self._auth_session = self.bridge.run(_enable(), timeout=30)
+
     def _enable_autosolve(self):
         """The Scraping Browser API's own CAPTCHA domain, as the Playwright
         engine enables it: if a captcha ever appears in front of the landing,

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "makemytrip-scraper"
-version = "0.1.0"
+version = "0.1.1"
 description = "makemytrip.com hotel listing scraper: prices before and after taxes, star and guest ratings, property type, location, from the site's own listing API (Playwright, Selenium or pyppeteer)."
 readme = "README.md"
 license = "MIT"

smoke_test.py

@@ -1257,6 +1257,24 @@ def check_selenium_reads_chromes_own_error_page():
           "location.href" in src and "current_url" not in src.split('"""')[-1])
 
 
+def check_pyppeteer_answers_proxy_auth_over_cdp():
+    """pyppeteer's page.authenticate relies on Network.setRequestInterception,
+    which current Chromium removed: measured, the engine died before its
+    first navigation with any credentialled proxy. The engine answers the
+    proxy's challenge through the Fetch domain instead, and only a PROXY's."""
+    src = open(os.path.join(HERE, "puppeteer_scraper.py"), encoding="utf-8").read()
+    tree = ast.parse(src)
+    called = {n.func.attr for n in ast.walk(tree)
+              if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)}
+    check("the engine never CALLS page.authenticate", "authenticate" not in called)
+    check("...it enables Fetch with handleAuthRequests",
+          '"Fetch.enable", {"handleAuthRequests": True' in src)
+    check("...answers Fetch.authRequired, continues every paused request",
+          "Fetch.authRequired" in src and "Fetch.continueRequest" in src)
+    check("...and gives the credentials only to a Proxy challenge",
+          'if source == "Proxy"' in src)
+
+
 def check_credentials_never_reach_a_log():
     for module in ENGINES:
         engine = _import_engine(module)