From 406068d1d99419f2e0977304197616ccaaa64372 Mon Sep 17 00:00:00 2001 From: nick_sudev Date: Tue, 29 Sep 2026 14:42:26 +0300 Subject: [PATCH 1/3] Family sync 2026-09-29: fingerprint script, pyppeteer proxy auth, CI pipefail, skips, gitignore MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by a family-wide pass against the template on 2026-09-29; each one made a documented feature do less than it said while the run reported success. Details and evidence in CHANGELOG.md [Unreleased]. - --fingerprint's init script was broken JavaScript and never ran; now node --check'ed by the suite. - pyppeteer's page.authenticate() uses a CDP method current Chrome removed; proxy 407s are answered through the Fetch domain. - The engine CI step piped the suite into tee without pipefail, so a failing suite was green; now `shell: bash`, an import check, and a suite guard on every piped step. - Skipped checks were counted as passed; now counted separately. - env_config.py --help no longer reads .env. - .gitignore covers .env*, run directories and paged dumps by shape. - One Scraping Browser connect policy (template §26): retry a lock or an outage 3x3 s, never a 401, name the cause (incl. profile_locked). Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/workflows/tests.yml | 10 +++ .gitignore | 23 ++++- CHANGELOG.md | 45 ++++++++++ env_config.py | 7 ++ fingerprint_client.py | 6 +- playwright_scraper.py | 29 ++++-- proxy_pool.py | 61 +++++++++++++ puppeteer_scraper.py | 85 +++++++++++++++--- smoke_test.py | 170 +++++++++++++++++++++++++++++++++--- 9 files changed, 399 insertions(+), 37 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 7cba2f3..0d7b293 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -95,6 +95,11 @@ jobs: - name: The suite, with THIS engine importable # The other two are absent by design, so the suite will report skips # for them; what must NOT happen is this engine's group skipping. + # `shell: bash` turns on pipefail: GitHub runs an UNSPECIFIED shell as + # `bash -e` without it, so `smoke_test.py | tee` took tee's exit status + # and a failing suite went green (found by an audit of a sibling repo, + # 2026-09-29). + shell: bash run: | set -e . .venv-${{ matrix.engine }}/bin/activate @@ -103,6 +108,11 @@ jobs: puppeteer) module=puppeteer_scraper ;; selenium) module=selenium_scraper ;; esac + # The import itself, first: it fails BY NAME, because every engine + # imports its driver at module level, whatever wording the suite + # uses for a skip (family template §26 — the grep below fired in + # only 2 of 31 sibling suites when this was measured). + python -c "import $module" python smoke_test.py | tee smoke.log if grep -q "SKIPPED.*$module" smoke.log; then echo "FAILED: $module reported skipped although ${{ matrix.engine }} is installed in this job — likely a real import error." diff --git a/.gitignore b/.gitignore index 2d2b055..c502f92 100644 --- a/.gitignore +++ b/.gitignore @@ -8,7 +8,23 @@ lg_products.* *.meta.json *_debug.html *_debug.png +# --dump-html X over several pages writes X.page2, X.page3 …, which no +# suffix rule above covers (family template §26). +*.page[0-9]* + +# ANY working directory a live run writes into, not one chosen name: a +# sibling committed a 1.5 MB raw dump through `live/` because only +# `live_results/` was listed (family template §24). +live/ live_results/ +out/ +output/ +runs/ +results/ +tmp/ +scratch/ +dumps/ +captures/ __pycache__/ *.py[cod] @@ -25,6 +41,11 @@ env/ .idea/ .vscode/ *.log -.env *.key secrets* + +# Every spelling of a working .env, not just `.env`: a `.env.bak` or +# `.env.local` is exactly how a live key gets one `git add -A` from a commit +# (family template §22). The negation keeps the documented example tracked. +.env* +!.env.example diff --git a/CHANGELOG.md b/CHANGELOG.md index c137153..555bbaf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,51 @@ All notable changes to this project are documented here. The format follows [SemVer](https://semver.org/) as closely as a CLI toolkit can: a PATCH release means fixes, not that every flag is frozen. +## [Unreleased] + +### Fixed + +Found by a family-wide pass against the template on 2026-09-29; each one +made a documented feature do less than it said while the run reported +success. + +- **`CONTRIBUTING.md` promised checks the suite did not make**: that a + workflow installing anything but a hashed lock, a lock drifting from its + `.txt`, or an action referenced by tag fails the suite. It now does. +- **`--fingerprint` never patched anything.** The init script it installs + was syntactically broken JavaScript (`( => {` — the parameter list and the + IIFE call were missing), and a syntax error in an init script is silent: + the page ran unpatched. Fixed, and the suite now runs `node --check` on + that script too — 0.2.2's "every shipped script" did not include it. +- **pyppeteer could not authenticate a proxy on current Chrome.** + `page.authenticate()` uses `Network.setRequestInterception`, which + current Chrome no longer has ("wasn't found", exit 5 before the first + request — measured 2026-09-29 in rosreestr-scraper, which shares this + engine code). The engine now answers the proxy's 407 through the CDP + Fetch domain; verified against a local Basic-auth proxy. +- **A failing suite could turn the engine CI job green.** The step ran + `smoke_test.py | tee` under GitHub's default shell, which has no + pipefail, so the job took tee's exit status. It now runs with + `shell: bash`, imports the engine by name first, and the suite fails if + any piped step lacks pipefail. +- **Skipped checks were counted as passed.** A check that could not run + printed "SKIPPED" and was added to the pass count. Skips are now their own + count, listed at the end. +- **`env_config.py --help` read `.env`.** `ci_checks.py` runs every CLI's + `--help`; a help screen now opens no credentials file. +- **`.gitignore` let a renamed `.env` through** (`.env.bak`, `.env.local`), + and a run directory (`live/`) or a paged dump (`X.page3`). Now ignored by + shape, with a check. + +### Changed + +- **Connecting to the Scraping Browser API** follows one policy in every + engine (family template §26): three attempts 3 s apart for a locked + profile or an outage, none for an expired login (401), and a message that + names the status's own cause. pyppeteer's per-attempt timeout is 10 s (it + reports a refused handshake only as a timeout), and the orphaned-task + traceback that followed a correct exit is filtered. + ## [0.2.2] — 2026-09-28 ### Fixed diff --git a/env_config.py b/env_config.py index 4b21086..0eb5464 100644 --- a/env_config.py +++ b/env_config.py @@ -304,6 +304,13 @@ def apply(args, keys=None, quiet=False): if __name__ == "__main__": # `python3 env_config.py` — report what is configured, without printing # any secret. Useful as a first step when a key "isn't being picked up". + import sys + if any(arg in ("-h", "--help") for arg in sys.argv[1:]): + # Answered WITHOUT reading .env: ci_checks runs every CLI's --help, + # and a help screen has no business opening a credentials file. + print("usage: python3 env_config.py\n\nReports which variables from " + ".env / the environment were picked up, never their values.") + sys.exit(0) logging.basicConfig(level=logging.INFO, format="%(message)s") where = load_env() print(f".env file: {where or 'not found (this is fine — env vars still work)'}") diff --git a/fingerprint_client.py b/fingerprint_client.py index 3e82397..8536349 100644 --- a/fingerprint_client.py +++ b/fingerprint_client.py @@ -280,11 +280,11 @@ def playwright_init_script(fp: dict) -> str: "webglRenderer": found["webgl_renderer"]["value"], }) return """ -( => { +(() => { const fp = %s; const def = (obj, prop, value) => { if (value === null || value === undefined) return; - try { Object.defineProperty(obj, prop, {get: => value, configurable: true}); } + try { Object.defineProperty(obj, prop, {get: () => value, configurable: true}); } catch (e) { /* already non-configurable: leave it rather than throw */ } }; def(Navigator.prototype, 'platform', fp.platform); @@ -303,7 +303,7 @@ def playwright_init_script(fp: dict) -> str: return original.apply(this, arguments); }; } -}); +})(); """ % payload diff --git a/playwright_scraper.py b/playwright_scraper.py index b4c1336..350a241 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -66,7 +66,8 @@ from output_writer import dedupe_by_sku, finish_run from product_parser import (SELECTORS, category_from_url, page_url, parse_products, unsupported_locale_reason) -from proxy_pool import (ROTATE_MODES, ProxyError, ProxyPool, +from proxy_pool import (CDP_CONNECT_ATTEMPTS, CDP_CONNECT_PAUSE_S, CDP_CONNECT_TIMEOUT_S, + ROTATE_MODES, cdp_refusal_advice, cdp_retryable, ProxyError, ProxyPool, check_exit_or_raise, from_args as proxy_pool_from_args, mask, redact_secret_patterns, to_playwright) @@ -201,12 +202,26 @@ def _connect_remote(pw, args): # underneath it. Measured 2026-09-19 against a real endpoint that answered # 401: the password appeared five times in one traceback. An exception # message is a log. - try: - browser = pw.chromium.connect_over_cdp(args.cdp_endpoint, timeout=30000) - except Exception as e: # noqa: BLE001 — re-raised immediately, redacted - raise RuntimeError( - f"Could not connect to --cdp-endpoint: " - f"{redact_secret_patterns(str(e))}") from None + # + # Retried per proxy_pool's CDP policy: a profile stays locked for ~2 s + # after a clean disconnect (family template §26), so a run started right + # after another met HTTP 500. A 401 (expired credentials) is not retried. + for attempt in range(1, CDP_CONNECT_ATTEMPTS + 1): + try: + browser = pw.chromium.connect_over_cdp(args.cdp_endpoint, timeout=30000) + break + except Exception as e: # noqa: BLE001 — re-raised immediately, redacted + # Whole message, on one line: the endpoint's host:port is what + # says WHICH exit failed, and it sits past the first line. + text = " ".join(redact_secret_patterns(str(e)).split())[:600] + if attempt < CDP_CONNECT_ATTEMPTS and cdp_retryable(text): + logger.warning("Scraping Browser connect failed (%s); retrying in %.0fs " + "[%d/%d].", text, CDP_CONNECT_PAUSE_S, attempt, + CDP_CONNECT_ATTEMPTS) + time.sleep(CDP_CONNECT_PAUSE_S) + continue + raise RuntimeError(f"Could not connect to --cdp-endpoint: {text} — " + f"{cdp_refusal_advice(text)}") from None # Reuse the remote browser's existing context so its fingerprint, session # and exit stay intact. context = browser.contexts[0] if browser.contexts else browser.new_context() diff --git a/proxy_pool.py b/proxy_pool.py index 7e25c31..dec60f9 100644 --- a/proxy_pool.py +++ b/proxy_pool.py @@ -371,3 +371,64 @@ def from_args(args) -> Optional[ProxyPool]: return ProxyPool([entry], rotate="per-run") return None + + +# --------------------------------------------------------------------------- +# Connecting to the Scraping Browser API — one policy for every engine +# --------------------------------------------------------------------------- +# Measured in the family (template §26, 2026-09-24): a profile stays +# `profile_locked` for 1.6-1.9 s after a CLEAN disconnect, so a second run +# started straight after the first was refused with HTTP 500. Three attempts +# 3 s apart ride that out, while a profile genuinely held by another run +# still fails in about 9 s. And pyppeteer's connect never surfaces the 500 +# at all — only a timeout ends it — so its per-attempt timeout is short: a +# successful connect took 0.8-0.95 s. The 500's body names the cause when it +# is a lock: "profile_locked" (seen 2026-09-29 in rosreestr-scraper's canary, +# where two jobs shared one profile — no retry window covers that). +CDP_CONNECT_ATTEMPTS = 3 +CDP_CONNECT_PAUSE_S = 3.0 +CDP_CONNECT_TIMEOUT_S = 10.0 + + +def _has_status(text: str, *codes: str) -> bool: + """A status as a NUMBER on its own — never a digit run inside a port, a + duration or a request id (a bare "500" matched "15000ms").""" + return any(re.search(rf"(? str: + """What a failed Scraping Browser connect most likely means, by its status. + + The old message explained a 500 whatever the status was, and every + expired endpoint in the family answers 401. + """ + text = error_text or "" + if "profile_locked" in text: + # The service's own word for it, in the 500's body (seen 2026-09-29 + # when two canary jobs shared one profile). + return ("the profile is in use by another connection (profile_locked): " + "one live connection per pid. Wait for the other run to finish, or " + "give each concurrent run its own pid.") + if _has_status(text, "401"): + return ("HTTP 401: the profile's credentials were refused — they expire " + "(about a day). Take fresh ones from the 2Captcha dashboard.") + if _has_status(text, "403"): + return "HTTP 403: this account may not use that profile or zone." + if _has_status(text, "500", "502", "503"): + return ("HTTP 5xx on the WebSocket upgrade: usually the profile is still " + "locked by another (or just-finished) connection — one live " + "connection per pid — or a service-side outage. Wait, or use " + "another pid.") + if "imeout" in text or not text.strip(): + return ("no answer in time. pyppeteer reports a REFUSED handshake this " + "way too, so this may be the same 5xx; try the Playwright engine " + "to see the status.") + return "see the error above." + + +def cdp_retryable(error_text: str) -> bool: + """Worth another attempt: a lock or an outage, never a refused login.""" + text = error_text or "" + if _has_status(text, "401", "403"): + return False + return _has_status(text, "500", "502", "503") or "imeout" in text or not text.strip() diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index e6dc238..9417d9b 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -19,7 +19,8 @@ * **Proxy credentials never reach the command line.** Chromium's `--proxy-server` switch is argv, readable by anything that can run `ps`, so the host and port go there and the credentials go through - `page.authenticate()`. + the DevTools Fetch domain (`Fetch.authRequired`) — pyppeteer's own + `page.authenticate()` relies on a CDP method current Chrome removed. * **Its bundled Chromium is from 2018.** pyppeteer downloads and launches that one by default, and on a current macOS it dies with "Browser closed unexpectedly" before the first fetch. Set `PYPPETEER_EXECUTABLE_PATH` to a @@ -64,7 +65,8 @@ from output_writer import dedupe_by_sku, finish_run from product_parser import (SELECTORS, category_from_url, page_url, parse_products, unsupported_locale_reason) -from proxy_pool import (ROTATE_MODES, ProxyError, check_exit_or_raise, +from proxy_pool import (CDP_CONNECT_ATTEMPTS, CDP_CONNECT_PAUSE_S, CDP_CONNECT_TIMEOUT_S, + ROTATE_MODES, cdp_refusal_advice, cdp_retryable, ProxyError, check_exit_or_raise, from_args as proxy_pool_from_args, mask, redact_secret_patterns, to_pyppeteer) @@ -78,7 +80,6 @@ NEXT_PAGE_SELECTOR = None # --------------------------------------------------------------------------- -CONNECT_TIMEOUT_S = 30 GOTO_TIMEOUT_MS = 60000 @@ -124,14 +125,29 @@ async def _open_browser(args, pool): redact_secret_patterns(args.cdp_endpoint)) # pyppeteer's connect has no timeout of its own — and its failures, # like Playwright's, quote the endpoint with its password in them. - try: - browser = await asyncio.wait_for( - connect(browserWSEndpoint=args.cdp_endpoint), - timeout=CONNECT_TIMEOUT_S) - except Exception as e: # noqa: BLE001 — re-raised immediately, redacted - raise RuntimeError( - f"Could not connect to --cdp-endpoint: " - f"{redact_secret_patterns(str(e))}") from None + # pyppeteer never surfaces a refused handshake: the HTTP 500 dies in + # its receive loop, the caller sees only a timeout, and the orphaned + # task prints a traceback AFTER a correct exit. So: a short timeout + # per attempt, proxy_pool's retry policy, and that one traceback + # shape filtered (family template §26). + _quiet_refused_handshakes() + browser = None + for attempt in range(1, CDP_CONNECT_ATTEMPTS + 1): + try: + browser = await asyncio.wait_for( + connect(browserWSEndpoint=args.cdp_endpoint), + timeout=CDP_CONNECT_TIMEOUT_S) + break + except Exception as e: # noqa: BLE001 — re-raised immediately, redacted + text = redact_secret_patterns(f"{type(e).__name__}: {e}")[:300] + if attempt < CDP_CONNECT_ATTEMPTS and cdp_retryable(text): + logger.warning("Scraping Browser connect failed (%s); retrying in " + "%.0fs [%d/%d].", text, CDP_CONNECT_PAUSE_S, attempt, + CDP_CONNECT_ATTEMPTS) + await asyncio.sleep(CDP_CONNECT_PAUSE_S) + continue + raise RuntimeError(f"Could not connect to --cdp-endpoint: {text} — " + f"{cdp_refusal_advice(text)}") from None page = await browser.newPage() await _enable_auto_solve(page) return browser, page @@ -162,10 +178,55 @@ async def _open_browser(args, pool): page = await browser.newPage() if credentials: # The password goes through the DevTools protocol, not through argv. - await page.authenticate(credentials) + await _authenticate_proxy(page, credentials) return browser, page +def _quiet_refused_handshakes(): + """Drop the orphaned-task traceback a refused CDP handshake leaves behind.""" + loop = asyncio.get_running_loop() + default = loop.get_exception_handler() + + def handler(lp, context): + exc = context.get("exception") + if exc is not None and type(exc).__name__ in ("InvalidStatusCode", "InvalidStatus", + "InvalidHandshake"): + logger.debug("Ignored a refused-handshake task: %s", exc) + return + if default is not None: + default(lp, context) + else: + lp.default_exception_handler(context) + + loop.set_exception_handler(handler) + + +async def _authenticate_proxy(page, creds): + """Answer the proxy's 407 through the CDP Fetch domain. + + Not page.authenticate(): pyppeteer implements it with + Network.setRequestInterception, which current Chrome no longer has — + measured 2026-09-29 in a sibling repo against Chrome for Testing: + "Protocol error (Network.setRequestInterception): ... wasn't found", + exit 5 before the first request. + """ + cdp = await page.target.createCDPSession() + + def paused(event): + asyncio.ensure_future(cdp.send("Fetch.continueRequest", {"requestId": event["requestId"]})) + + def auth(event): + asyncio.ensure_future(cdp.send("Fetch.continueWithAuth", { + "requestId": event["requestId"], + "authChallengeResponse": {"response": "ProvideCredentials", + "username": creds["username"], + "password": creds["password"]}})) + + cdp.on("Fetch.requestPaused", paused) + cdp.on("Fetch.authRequired", auth) + await cdp.send("Fetch.enable", {"handleAuthRequests": True, "patterns": [{"urlPattern": "*"}]}) + + async def _enable_auto_solve(page) -> None: """Turn on the Scraping Browser API's own solver for this session.""" try: diff --git a/smoke_test.py b/smoke_test.py index 38dd032..d1679a4 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -83,10 +83,20 @@ def check(label, condition): return bool(condition) +SKIPPED = [] + + def skip(label, why): - """Record a check that could not run, in this suite's own convention: - a passing line that SAYS it was skipped, so it is visible in the output.""" - return check(f"{label} ({why} — SKIPPED)", True) + """Record a check that could NOT run — never as a pass. + + This used to be `check(f"{label} ({why} — SKIPPED)", True)`: a line that + said "skipped" and was COUNTED as passed, which inflates the pass count + and reads identically to a check that ran (family template §10). Skips + are now their own count, listed at the end of the run. + """ + SKIPPED.append(f"{label} ({why})") + print(f" SKIP {label} — {why}") + return False def eq(label, actual, expected): @@ -1298,8 +1308,7 @@ def check_fingerprint_kwargs_are_ones_playwright_accepts(): engine = ENGINES.get("playwright_scraper") if engine is None: SKIPPED_GROUPS.append("fingerprint kwargs (playwright not installed)") - check("fingerprint kwargs vs the real new_context signature " - "(playwright not installed — SKIPPED)", True) + skip('fingerprint kwargs vs the real new_context signature', 'playwright not installed') return from playwright.sync_api import Browser accepted = set(inspect.signature(Browser.new_context).parameters) @@ -1427,8 +1436,7 @@ def check_remote_connect_failures_are_redacted(): engine = ENGINES.get("playwright_scraper") if engine is None: - check("the engine's own connect path redacts (playwright not " - "installed — SKIPPED)", True) + skip("the engine's own connect path redacts", 'playwright not installed') return class _StubChromium: @@ -1645,7 +1653,7 @@ def _engine_flags(module): def check_engine_parity(): print("\n[engine parity]") if not ENGINES: - check("engine parity (no engine library installed here — SKIPPED)", True) + skip('engine parity', 'no engine library installed here') return flag_sets = {} for name, module in ENGINES.items(): @@ -2078,8 +2086,7 @@ def check_packaging_matches_the_tree(): try: import tomllib except ImportError: - check("pyproject.toml cross-check (tomllib needs 3.11 — SKIPPED here, " - "CI's 3.12 leg runs it)", True) + skip('pyproject.toml cross-check', "tomllib needs 3.11; here, CI's 3.12 leg runs it") return with open(path, "rb") as handle: config = tomllib.load(handle) @@ -2282,7 +2289,7 @@ def check_concurrency_machinery(): engine = ENGINES.get("playwright_scraper") if engine is None: SKIPPED_GROUPS.append("concurrency (playwright not installed)") - check("concurrency machinery (playwright not installed — SKIPPED)", True) + skip('concurrency machinery', 'playwright not installed') return # A live run cannot always reach this code: pages 1 and 2 are fetched @@ -2386,7 +2393,7 @@ def check_worker_pools_start_on_different_exits(): print("\n[worker pools]") engine = ENGINES.get("playwright_scraper") if engine is None: - check("worker pools (playwright not installed — SKIPPED)", True) + skip('worker pools', 'playwright not installed') return pool = proxy_pool.ProxyPool(["http://a:1", "http://b:2", "http://c:3"]) starts = [engine._worker_pool(pool, i).current for i in range(3)] @@ -3115,6 +3122,138 @@ def check_fingerprint_refusal_is_an_api_error(): "printed", fake_key not in text) + +def check_family_sync_2026_09_29(): + """Three defects found in this repo by a family-wide pass on 2026-09-29.""" + print("\n[family sync 2026-09-29]") + import shutil + import subprocess + import tempfile + # 1. The fingerprint init script was syntactically broken JavaScript + # ("( => {"), and a syntax error in an init script is SILENT: the + # page runs unpatched and --fingerprint did nothing for anyone. + import fingerprint_client + node = shutil.which("node") + if node: + path = os.path.join(tempfile.mkdtemp(), "fp.js") + with open(path, "w", encoding="utf-8") as f: + f.write(fingerprint_client.playwright_init_script({})) + r = subprocess.run([node, "--check", path], capture_output=True, text=True) + check("the fingerprint init script parses as JavaScript" + + ("" if r.returncode == 0 else f" — {r.stderr.strip()[:160]}"), r.returncode == 0) + else: + skip("the fingerprint init script parses as JavaScript", "node not installed; CI has it") + # 2. `env_config.py --help` read .env: ci_checks runs every CLI's --help. + r = subprocess.run([sys.executable, "env_config.py", "--help"], cwd=REPO, + capture_output=True, text=True) + check("`env_config.py --help` answers without reading .env", + r.returncode == 0 and ".env file:" not in r.stdout) + # 4. The Scraping Browser connect policy (template §26): retry a lock or + # an outage, never an expired login, and say which it was. + eq("a 500 on the upgrade is retried", proxy_pool.cdp_retryable( + "WebSocket error: ws://***:***@cb.2captcha.com:9222/ 500 Internal Server Error"), True) + eq("a 401 is not", proxy_pool.cdp_retryable(" 401 Unauthorized"), False) + eq("a number that merely CONTAINS 500 is not a 5xx", proxy_pool.cdp_retryable("port 5001 reset"), False) + check("a 401 is explained as expired credentials", + "expire" in proxy_pool.cdp_refusal_advice("401 Unauthorized")) + engine = ENGINES.get("playwright_scraper") + if engine is None: + skip("the Playwright engine retries a locked profile", "playwright not installed") + else: + calls = [] + + class _Flaky: + def connect_over_cdp(self, endpoint, timeout=None): + calls.append(1) + if len(calls) < 3: + raise RuntimeError("WebSocket error: ws://***:***@cb.2captcha.com:9222/ 500 Internal Server Error") + raise _Connected() + + class _Connected(Exception): + pass + + class _PW: + chromium = _Flaky() + + class _Args: + cdp_endpoint = "ws://***:***@cb.2captcha.com:9222" + + real_sleep = engine.time.sleep + engine.time.sleep = lambda s: None + try: + engine._connect_remote(_PW(), _Args()) + except _Connected: + pass + except RuntimeError: + pass + finally: + engine.time.sleep = real_sleep + eq("the Playwright engine tried a locked profile three times", len(calls), 3) + # 5. A pipe in a workflow step hides the left side's failure unless + # pipefail is on, and GitHub runs an unspecified shell without it. + import re as _re + wf_dir = os.path.join(REPO, ".github", "workflows") + piped = 0 + if os.path.isdir(wf_dir): + for name in sorted(os.listdir(wf_dir)): + text = open(os.path.join(wf_dir, name), encoding="utf-8").read() + for step in _re.split(r"\n\s*- (?:name|uses):", text): + code = "\n".join(ln for ln in step.splitlines() + if ln.strip() and not ln.strip().startswith("#")) + m = _re.search(r"run:\s*\|?(.*)", code, _re.S) + if m and _re.search(r"python[^\n]*\|\s*tee", m.group(1)): + piped += 1 + check(f"{name}: a piped python step runs with pipefail", + "shell: bash" in code or "pipefail" in code) + check(f"the pipefail check found the piped steps it guards ({piped})", piped >= 1) + else: + skip("workflow pipefail", "no .github/workflows in this tree") + # 3. .gitignore by SHAPE: a renamed .env, a run directory, a paged dump. + if shutil.which("git") and os.path.isdir(os.path.join(REPO, ".git")): + for name, ignored in ((".env.bak", True), (".env.local", True), ("live/dump.html", True), + ("out.json.page3", True), ("captures/x.html", True), + (".env.example", False), ("sample_output.json", False)): + r = subprocess.run(["git", "check-ignore", "-q", name], cwd=REPO) + check(f"{name} is {'ignored' if ignored else 'NOT ignored'}", + (r.returncode == 0) == ignored) + else: + skip(".gitignore shapes", "not a git checkout") + + +def check_supply_chain(): + print("\n[supply chain]") + wf_dir = os.path.join(REPO, ".github", "workflows") + if not os.path.isdir(wf_dir): + skip("workflow pins", "no .github/workflows in this tree") + return + for name in sorted(os.listdir(wf_dir)): + text = open(os.path.join(wf_dir, name), encoding="utf-8").read() + uses = re.findall(r"uses:\s*([^\s#]+)", text) + loose = [u for u in uses if not re.search(r"@[0-9a-f]{40}$", u)] + check(f"{name}: every action is pinned by commit SHA" + + (f" (not: {loose})" if loose else ""), not loose) + code = "\n".join(ln for ln in text.splitlines() if not ln.strip().startswith("#")) + installs = re.findall(r"pip install ([^\n]+)", code) + bad = [i for i in installs if "--require-hashes" not in i or ".lock" not in i] + check(f"{name}: pip installs only hash-checked locks" + (f" (not: {bad})" if bad else ""), + not bad) + for txt, lock in (("requirements.txt", "requirements.lock"), + ("requirements-playwright.txt", "requirements-playwright.lock"), + ("requirements-puppeteer.txt", "requirements-puppeteer.lock"), + ("requirements-selenium.txt", "requirements-selenium.lock")): + wanted = {re.split(r"[<>=!~\[ ]", line.strip())[0].lower() + for line in open(os.path.join(REPO, txt), encoding="utf-8") + if line.strip() and not line.startswith("#")} + if txt != "requirements.txt": + wanted |= {"requests"} + pinned = set(re.findall(r"^([a-z0-9_.-]+)==", open(os.path.join(REPO, lock), + encoding="utf-8").read(), re.M | re.I)) + pinned = {x.lower() for x in pinned} + missing = wanted - pinned + check(f"{lock} pins everything {txt} asks for" + (f" (missing {missing})" if missing else ""), + not missing) + check(f"{lock} carries hashes", "--hash=sha256:" in open(os.path.join(REPO, lock), encoding="utf-8").read()) + def main() -> int: logging.basicConfig(level=logging.ERROR) print("lg-scraper offline suite") @@ -3151,11 +3290,14 @@ def main() -> int: check_worker_pools_start_on_different_exits, check_captcha_javascript_actually_runs, check_recaptcha_version_from_markup, - check_fingerprint_refusal_is_an_api_error): + check_fingerprint_refusal_is_an_api_error, + check_family_sync_2026_09_29, check_supply_chain): group() print("\n" + "=" * 62) - print(f"{len(PASSED)} passed, {len(FAILED)} failed") + print(f"{len(PASSED)} passed, {len(FAILED)} failed, {len(SKIPPED)} skipped") + for line in SKIPPED: + print(f" skipped: {line}") if SKIPPED_GROUPS: # Printed in a shape CI greps for: "skipped, engine absent" reads # identically to a real import error, so the engine-smoke job fails From 7bc58c3ae74f70b3f6ba8c7b6cca5dc65f9fd0ee Mon Sep 17 00:00:00 2001 From: nick_sudev Date: Tue, 29 Sep 2026 15:07:38 +0300 Subject: [PATCH 2/3] Secret scan: skip a virtualenv by pyvenv.cfg, not by name MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With pipefail on, the engine CI jobs showed they had been red: the scan walked into CI's own .venv- and flagged pip's and selenium's vendored code (32-hex in packaging/_elffile.py, URLs in urllib3 and websocket tests, a token in selenium's devtools). A directory holding pyvenv.cfg is a virtualenv whatever it is called (template §23); ordinary untracked files are still read. Both planted in a suite check. Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/ci_checks.py | 25 +++++++++++++++++++++++++ CHANGELOG.md | 6 ++++++ smoke_test.py | 27 +++++++++++++++++++++++++++ 3 files changed, 58 insertions(+) diff --git a/.github/ci_checks.py b/.github/ci_checks.py index eeb5874..22fee7c 100644 --- a/.github/ci_checks.py +++ b/.github/ci_checks.py @@ -92,12 +92,37 @@ ".pytest_cache"} +_VENV_CACHE = {} + + +def _is_in_virtualenv(path): + """True if any ancestor of `path` under REPO is a virtualenv root. + + A directory holding `pyvenv.cfg` IS a virtualenv, whatever it is called + (family template §23). The name set in SKIP_DIRS missed CI's own + `.venv-`, so the engine jobs scanned pip's and selenium's vendored + code and failed on it — hidden until pipefail made that job's exit status + real (2026-09-29). + """ + for parent in path.parents: + cached = _VENV_CACHE.get(parent) + if cached is None: + cached = _VENV_CACHE[parent] = (parent / "pyvenv.cfg").is_file() + if cached: + return True + if parent == REPO: + break + return False + + def scanned_files(): for path in sorted(REPO.rglob("*")): if not path.is_file() or path.suffix not in SCANNED_SUFFIXES: continue if any(part in SKIP_DIRS for part in path.parts): continue + if _is_in_virtualenv(path): + continue yield path diff --git a/CHANGELOG.md b/CHANGELOG.md index 555bbaf..a96706d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,12 @@ success. pipefail, so the job took tee's exit status. It now runs with `shell: bash`, imports the engine by name first, and the suite fails if any piped step lacks pipefail. +- **The engine CI jobs had been failing, unseen.** Once pipefail made the + step's exit status real, all three engine jobs went red on the secret + scan: it walked into CI's own `.venv-` and read pip's and + selenium's vendored code as credentials. The scan now skips any directory + holding `pyvenv.cfg`, whatever it is called (template §23), and still + reads ordinary untracked files — both planted in a check. - **Skipped checks were counted as passed.** A check that could not run printed "SKIPPED" and was added to the pass count. Skips are now their own count, listed at the end. diff --git a/smoke_test.py b/smoke_test.py index d1679a4..7df3230 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -3208,6 +3208,33 @@ class _Args: check(f"the pipefail check found the piped steps it guards ({piped})", piped >= 1) else: skip("workflow pipefail", "no .github/workflows in this tree") + # 6. The secret scan skips a virtualenv by its marker, not its name — + # and still reads an ordinary untracked directory. Planted both ways. + import importlib.util as _ilu + spec = _ilu.spec_from_file_location("ci_checks_fs", os.path.join(REPO, ".github", "ci_checks.py")) + if spec and os.path.isdir(os.path.join(REPO, ".github")): + cc = _ilu.module_from_spec(spec) + spec.loader.exec_module(cc) + venv = os.path.join(REPO, ".venv-plantedcheck") + plain = os.path.join(REPO, "plantedcheck-plain") + key = "b" * 32 + try: + os.makedirs(os.path.join(venv, "lib"), exist_ok=True) + open(os.path.join(venv, "pyvenv.cfg"), "w").write("home = /usr\n") + open(os.path.join(venv, "lib", "vendored.py"), "w").write(f"# {key}\n") + os.makedirs(plain, exist_ok=True) + open(os.path.join(plain, "leak.py"), "w").write(f"# {key}\n") + cc._VENV_CACHE.clear() + scanned = {str(p) for p in cc.scanned_files()} + check("the scan skips a virtualenv whatever it is called (pyvenv.cfg)", + not any(".venv-plantedcheck" in p for p in scanned)) + check("...and still reads an ordinary untracked directory", + any("plantedcheck-plain" in p for p in scanned)) + finally: + shutil.rmtree(venv, ignore_errors=True) + shutil.rmtree(plain, ignore_errors=True) + else: + skip("secret scan virtualenv rule", "no .github in this tree") # 3. .gitignore by SHAPE: a renamed .env, a run directory, a paged dump. if shutil.which("git") and os.path.isdir(os.path.join(REPO, ".git")): for name, ignored in ((".env.bak", True), (".env.local", True), ("live/dump.html", True), From a1a3a8f6221ffb59d5ddb42c967f59faffc53eeb Mon Sep 17 00:00:00 2001 From: nick_sudev Date: Tue, 29 Sep 2026 15:10:24 +0300 Subject: [PATCH 3/3] Engine CI: install the browser each engine drives The fingerprint-refusal check launches playwright_scraper for real, and it starts Chromium before it fetches; the job had no browser, so it exited 1 instead of 5. Red since it was added, hidden until pipefail. Playwright installs its Chromium; pyppeteer/Selenium use the runner's Google Chrome (as in rosreestr-scraper, where all three are green). Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/workflows/tests.yml | 15 +++++++++++++++ CHANGELOG.md | 4 ++++ 2 files changed, 19 insertions(+) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 0d7b293..99c5c13 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -92,6 +92,21 @@ jobs: # Proves each lock is self-consistent on its own. pip check + - name: A browser for this engine + # The suite launches the engine for real in places (the fingerprint- + # refusal check starts Chromium before it fetches). Without a browser + # that check had been red — hidden until pipefail (2026-09-29). + # Playwright brings its own Chromium; the other two use the runner's + # Google Chrome. + run: | + . .venv-${{ matrix.engine }}/bin/activate + if [ "${{ matrix.engine }}" = playwright ]; then + python -m playwright install --with-deps chromium + else + echo "PYPPETEER_EXECUTABLE_PATH=$(command -v google-chrome)" >> "$GITHUB_ENV" + google-chrome --version + fi + - name: The suite, with THIS engine importable # The other two are absent by design, so the suite will report skips # for them; what must NOT happen is this engine's group skipping. diff --git a/CHANGELOG.md b/CHANGELOG.md index a96706d..7e368d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,10 @@ success. selenium's vendored code as credentials. The scan now skips any directory holding `pyvenv.cfg`, whatever it is called (template §23), and still reads ordinary untracked files — both planted in a check. +- **The Playwright engine job never had a browser**, so the suite's + fingerprint-refusal check — which starts Chromium before it fetches — + crashed there with exit 1 instead of the expected 5. Also hidden until + pipefail; the engine jobs now install the browser they drive. - **Skipped checks were counted as passed.** A check that could not run printed "SKIPPED" and was added to the pass count. Skips are now their own count, listed at the end.