From c160dc638e48ead078ca2d8d77c12aa620c4ea8f Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Wed, 14 Oct 2026 18:54:36 +0000 Subject: [PATCH 1/3] One paid solve per page, atomic output writes, formula-safe CSV - A page's CAPTCHA allowance now survives every landing on it. Each block-retry or exit rotation landed again with a fresh allowance, so one refused page could buy 2 AWS WAF solves with no pool and 4 with a 3-exit pool against SOLVES_PER_PAGE = 1. The pay-type pre-check no longer solves. - JSON, CSV and the run sidecar are written to a temp file in the same directory, fsynced and renamed. A failure mid-write leaves the previous file intact. New files get 0o666 & ~umask, existing files keep their mode. - CSV cells starting with = + - @ or a control character get a leading apostrophe (after list joining; numbers and JSON untouched), counted as csv_cells_escaped in the sidecar under the caller's own fields. - BINANCE_URL in the environment is a default: a typed --mode or query flag wins. .env.example ships it commented out. - HTTP 451 advice and docs no longer claim an unmeasured US refusal; a CDP 401 is "credentials refused", not "they last about a day". - Root-level *.html captures are ignored; .dockerignore covers .env*. - README heading counts six items, not five. Co-Authored-By: Claude Opus 5.5 --- .dockerignore | 1 + .env.example | 17 +- .github/ISSUE_TEMPLATE/bug_report.yml | 8 +- .gitignore | 3 + CONTRIBUTING.md | 7 +- README.md | 11 +- TROUBLESHOOTING.md | 6 +- captcha_solver.py | 6 +- http_scraper.py | 3 + output_writer.py | 124 ++++++++++++-- page_flow.py | 101 +++++++++--- playwright_scraper.py | 3 + puppeteer_scraper.py | 3 + selenium_scraper.py | 3 + smoke_test.py | 223 +++++++++++++++++++++++++- 15 files changed, 452 insertions(+), 67 deletions(-) diff --git a/.dockerignore b/.dockerignore index f8079ef..4a33d61 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,6 +1,7 @@ .git/ .github/ .env +.env* *.log *.key secrets* diff --git a/.env.example b/.env.example index 57a250f..7add00d 100644 --- a/.env.example +++ b/.env.example @@ -51,9 +51,8 @@ TWOCAPTCHA_KEY=your_2captcha_api_key_here # A remote browser you do not run or patch, with a persistent profile and a # chosen exit country. Used by --cdp-endpoint. # -# country- picks the exit country. Pick one Binance serves: its terms -# exclude the United States (binance.us is a separate exchange). -# What a US exit is answered with has not been measured here. +# country- picks the exit country. binance.us is a separate exchange +# this scraper does not read. # pid- a profile with persistent cookies. ONE live connection each, # capped per account. Reuse pids rather than minting one per # run, and expect a 500 (`profile_locked`, exit 5) while another @@ -62,8 +61,8 @@ TWOCAPTCHA_KEY=your_2captcha_api_key_here # Selenium cannot use this: chromedriver's `debuggerAddress` takes a bare # host:port with nowhere to put a password. Playwright and pyppeteer can. # -# The credentials in a profile expire in about a day, so never paste a -# working one anywhere that is committed. +# The credentials in a profile expire after a while (a 401 says so), so +# never paste a working one anywhere that is committed. BINANCE_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-de-pid-{profileId}:{password}@cb.2captcha.com:9222 # --- Proxy ---------------------------------------------------------------- @@ -71,8 +70,6 @@ BINANCE_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-de-pid-{profileI # spreads a run's volume, and --concurrency without one just sends N times # the request rate from a single address. # -# Region: anywhere Binance serves (not the United States, per its terms). -# # The credentials never reach the browser's command line: they go through # the driver's own fields, and logs mask them while keeping host and port. # Selenium's --proxy-server cannot authenticate at all, so that engine @@ -88,8 +85,10 @@ BINANCE_PROXY=http://{user}:{password}@eu.proxy.2captcha.com:2334 # https://www.binance.com/en/support/announcement/list/48 announcements # # The page itself is never fetched, only the query in its address. The -# flags (--asset, --fiat, --mode ...) do the same without a URL. -BINANCE_URL=https://p2p.binance.com/en/trade/all-payments/USDT?fiat=EUR +# flags (--asset, --fiat, --mode ...) do the same without a URL, and a flag +# you TYPE wins over this variable: it is only a default for a run given +# neither. Left commented out so a copied example changes nothing. +# BINANCE_URL=https://p2p.binance.com/en/trade/all-payments/USDT?fiat=EUR # Deliberately NOT here: an output prefix. `--out` already carries a # non-empty default, so the loader, which only fills UNSET values, would diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 6bf5f81..3229b82 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -12,11 +12,9 @@ body: A few things answer most reports faster than we can: - - **Exit 3 from a US address?** Binance's terms exclude some - jurisdictions, the United States among them (binance.us is a - separate exchange). Try an exit elsewhere, and say in the report - what the sidecar's `stop_reason` was: this repo has not measured - what such an address is answered with. + - **Exit 3 with `blocked_geo-451`?** This has not been observed on + binance.com in testing. Say in the report which country the exit + was in. - **Exit 3 with `blocked_aws-waf`?** The site's AWS WAF challenged the session. The data endpoints were not gated when this repo was written, so this is news worth reporting, together with the exit diff --git a/.gitignore b/.gitignore index 28b0046..7a38169 100644 --- a/.gitignore +++ b/.gitignore @@ -48,6 +48,9 @@ dumps/ *.html.page* *.json.page* *.page[0-9]* +# A capture saved by hand at the repo root (`--dump-html dump.html`). The +# repo ships no HTML of its own. +/*.html # --- captures kept for parser work --------------------------------------- # The smoke suite's fixtures are committed (trimmed, in fixtures_generated.json); diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4a05428..c69cbce 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -164,10 +164,9 @@ Two things about running this live that are specific to Binance: * **No mode needs an exit, a key or an account.** The endpoints answered a datacentre VPS normally, so "it worked from my laptop" is reproducible here in a way it is not on most sibling repos. -* **Binance's terms exclude some jurisdictions, the United States among - them** (binance.us is a separate exchange). What an address there is - answered with has not been measured by this repo. If a run from one is - refused, that is the terms, not a bug. +* **HTTP 451 has not been observed.** The daily canary is served from a + GitHub-hosted runner with no proxy. If a run is refused with 451, record + the exit's country in the issue. **Run more than the primary engine.** "Mirror them exactly" is a design rule, not a verification. The fetch loop is shared (`page_flow.run_pages`), diff --git a/README.md b/README.md index 4bf9d7e..a5fc5b2 100644 --- a/README.md +++ b/README.md @@ -105,7 +105,7 @@ copy-trading rows 26 columns and announcement rows 12 columns. --- -## Five things about Binance that will look like bugs +## Six things about Binance that will look like bugs ### 1. A buy query returns adverts marked "sell" @@ -225,17 +225,16 @@ One key, four separately-billed products ([2captcha.com](https://2captcha.com)): Selenium's Chrome was not shown the CAPTCHA at all on those pages (3 of 3), so its copy of the solve path has not been exercised live. * **Proxies** (`--proxy`, `--proxy-file`): volume from more than one - address, and an exit in a country Binance serves. Binance's terms exclude - some jurisdictions, the United States among them. + address, and an exit in a chosen country. * **The Scraping Browser API** (`--cdp-endpoint`): a remote browser you do not run, with a chosen exit country. One live connection per `pid`, so `--concurrency` is refused with it. Run live on 2026-09-24 through a `country-de` profile: Playwright and pyppeteer, all three modes, 8 of 8 runs complete, back to back. Two details measured on the way: a profile stays locked for 1.6-1.9 s after a clean disconnect, so the engines retry - a `profile_locked` connection rather than failing on it; and an expired - profile answers HTTP 401, which the engines report as exit 5 naming the - expiry. Selenium refuses a credentialled endpoint with exit 2. + a `profile_locked` connection rather than failing on it; and a profile + whose credentials are refused (expired or mistyped) answers HTTP 401, + which the engines report as exit 5 with that reason. Selenium refuses a credentialled endpoint with exit 2. * **Fingerprints** (`--fingerprint`): a consistent device identity for a local browser. Ignored with `--cdp-endpoint`, which brings its own. diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md index 519fb31..5bd711e 100644 --- a/TROUBLESHOOTING.md +++ b/TROUBLESHOOTING.md @@ -25,8 +25,10 @@ The log names what refused the run: * **`http-403`** — the site refused the address. Binance's API documentation uses 403 for a WAF rule, including a rate-limit violation. Slow down (`--delay`), or use a different exit. -* **`geo-451`** — the exit's country is refused. Binance's terms exclude some - jurisdictions, the United States among them. +* **`geo-451`** — HTTP 451, a refusal for legal reasons. It has not been + observed on binance.com in testing (the daily canary is served from a + GitHub-hosted runner with no proxy), so please open an issue with the + sidecar's `stop_reason` and the exit's country. A `_page_debug.html` beside the output holds what came back. diff --git a/captcha_solver.py b/captcha_solver.py index e60d28e..4787050 100644 --- a/captcha_solver.py +++ b/captcha_solver.py @@ -268,8 +268,8 @@ def detect_recaptcha_v3(html: str, page_url: str) -> Optional[CaptchaChallenge]: attributes (data-captcha-type="recaptcha" data-version="v3" data-sitekey="..." data-action="..."), with the actual execute() call happening inside a bundled JS file, never appearing as - readable inline script text at all — confirmed live on the sibling - farfetch-scraper repo's target, + readable inline script text at all — confirmed live on a retail + site's sign-up modal via DevTools inspection. A previous version of this function, which only checked for format 1, reported "no captcha" on this exact page despite one being genuinely present — caught by @@ -319,7 +319,7 @@ def detect_recaptcha_v3(html: str, page_url: str) -> Optional[CaptchaChallenge]: # --------------------------------------------------------------------------- # Runtime detection (added 2026-08-24 — the static-HTML detector below no -# longer matched the sibling farfetch-scraper repo's target) +# longer matched a retail site's sign-up modal) # --------------------------------------------------------------------------- # # What changed there: the sign-up modal used to render diff --git a/http_scraper.py b/http_scraper.py index 7de7f91..56bed11 100644 --- a/http_scraper.py +++ b/http_scraper.py @@ -321,6 +321,9 @@ def parse_args(argv=None): "success as well as failure. It is JSON; the flag " "keeps the family's name.") args = p.parse_args(argv) + # Before .env fills it: a URL from the environment is a default, + # and must not refuse a --mode or a query flag the user typed. + args.typed_url = args.url env_config.apply(args) # Read by the shared loop, which serves the browser engines too. There is # no remote browser here. diff --git a/output_writer.py b/output_writer.py index dc1753c..d44adef 100644 --- a/output_writer.py +++ b/output_writer.py @@ -29,8 +29,11 @@ an empty CSV still gets the right header for the mode that produced it. """ +import contextlib import csv import json +import os +import tempfile from dataclasses import dataclass, asdict, field, fields from datetime import datetime, timezone from typing import Optional, List, Set, Sequence, Any, Type @@ -262,12 +265,92 @@ def _csv_value(v: Any) -> Any: return v +@contextlib.contextmanager +def _atomic(path: str, newline: Optional[str] = None): + """Write to a temporary file beside `path`, then rename over it. + + `open(path, "w")` truncates before a single byte is written, so a crash, + a kill, a full disk or a Ctrl-C halfway through `json.dump` leaves a + SHORTER file where a complete one was. Reproduced here before this + existed: a good 786-byte `out.json` came back as 13 bytes of invalid + JSON after a failure inside `json.dump`. That destroys the previous good + run by the ATTEMPT to replace it, which is exactly what `save` refusing + to write an empty result exists to prevent. + + * the temp file goes in the target's OWN directory, because `os.replace` + is atomic only within one filesystem; + * `fsync` before the rename makes the bytes durable, not merely visible; + * `newline` is passed through because the csv module needs + `newline=""`. + + Each file is replaced atomically on its own. JSON, CSV and the sidecar + are three replacements, not one transaction. + """ + directory = os.path.dirname(os.path.abspath(path)) or "." + handle = tempfile.NamedTemporaryFile( + mode="w", encoding="utf-8", newline=newline, dir=directory, + prefix=os.path.basename(path) + ".", suffix=".tmp", delete=False) + try: + with handle: + yield handle + handle.flush() + os.fsync(handle.fileno()) + # `NamedTemporaryFile` creates at 0600 and `os.replace` keeps that + # mode. An existing target keeps its OWN mode (someone may have + # tightened it on purpose); a new one gets what `open()` would have + # given it, `0o666 & ~umask`. Never a flat 0o644, which narrows + # 0664 to 0644 under umask 002. + try: + mode = os.stat(path).st_mode & 0o777 + except OSError: + umask = os.umask(0) + os.umask(umask) + mode = 0o666 & ~umask + os.chmod(handle.name, mode) + os.replace(handle.name, path) + except BaseException: + # Including KeyboardInterrupt: an interrupted run must not leave + # `out.json.ab12cd.tmp` behind beside the good file. + try: + os.unlink(handle.name) + except OSError: + pass + raise + + +# A spreadsheet treats a cell beginning with one of these as a FORMULA +# rather than as text, and every string in these rows was written by a third +# party: an advertiser's nickname, a lead trader's portfolio name, an +# announcement title. The tab and the two newlines are here because some +# readers strip a leading one and then execute whatever followed it. +CSV_FORMULA_LEADS = ("=", "+", "-", "@", "\t", "\r", "\n") + +# Excel, Google Sheets and LibreOffice read a leading apostrophe as "the rest +# of this cell is text". A consumer parsing the file with the `csv` module +# DOES see it, which is why the count goes into the sidecar. +CSV_FORMULA_ESCAPE = "'" + + +def _csv_escape(v: Any): + """Neutralise a formula-shaped cell. Returns (value, was_escaped). + + Only STRINGS are touched: escaping a number would turn `-5` into text + and break every sum over the column (copy-trading ROI and PnL are + negative often). CSV only: the JSON output keeps the site's bytes, and + `csv_cells_escaped` in the sidecar declares the divergence. + """ + if isinstance(v, str) and v.startswith(CSV_FORMULA_LEADS): + return CSV_FORMULA_ESCAPE + v, True + return v, False + + def write_json(rows: Sequence[Any], path: str) -> None: - with open(path, "w", encoding="utf-8") as f: + with _atomic(path) as f: json.dump([asdict(r) for r in rows], f, ensure_ascii=False, indent=2) -def write_csv(rows: Sequence[Any], path: str, row_cls: Type = P2PAd) -> None: +def write_csv(rows: Sequence[Any], path: str, row_cls: Type = P2PAd) -> int: + """Write the CSV and return how many cells were formula-escaped.""" # An empty result still gets the header row. A zero-byte file makes a # consumer fail on read (no columns to parse) instead of reading a valid # table with zero rows — and "an empty result is still a well-formed @@ -276,11 +359,20 @@ def write_csv(rows: Sequence[Any], path: str, row_cls: Type = P2PAd) -> None: # The header comes from `row_cls`, not from the first row, so an empty # run still writes the columns of the mode that produced it. fieldnames = [f.name for f in fields(row_cls)] - with open(path, "w", encoding="utf-8", newline="") as f: + escaped = 0 + with _atomic(path, newline="") as f: writer = csv.DictWriter(f, fieldnames=fieldnames) writer.writeheader() for r in rows: - writer.writerow({k: _csv_value(v) for k, v in asdict(r).items()}) + row = {} + for k, v in asdict(r).items(): + # AFTER the list serialiser: a joined list is text too, and + # its first element can be formula-shaped. + value, was_escaped = _csv_escape(_csv_value(v)) + row[k] = value + escaped += was_escaped + writer.writerow(row) + return escaped # Exit code used when a run completes but produced nothing. Distinct from 1 @@ -355,7 +447,7 @@ def write_run_meta(out_prefix: str, meta: dict) -> str: both complete, and between runs of different `mode`. """ path = f"{out_prefix}.meta.json" - with open(path, "w", encoding="utf-8") as f: + with _atomic(path) as f: json.dump(meta, f, ensure_ascii=False, indent=2) print(f"[+] Wrote run metadata -> {path} (status={meta.get('status')})") return path @@ -413,14 +505,15 @@ def run_meta(status: str, stop_reason: str, pages_requested: int, } if extra: # Merged rather than nested under a key, so a consumer reads - # `shop_rating` at the top level beside `products`. Run fields win a + # `total_results` at the top level beside `products`. Run fields win a # name collision: a caller cannot accidentally overwrite `status`. meta.update({k: v for k, v in extra.items() if k not in meta}) return meta def save(rows: Sequence[Any], out_prefix: str, fmt: str, - allow_empty: bool = False, row_cls: Type = P2PAd) -> int: + allow_empty: bool = False, row_cls: Type = P2PAd, + stats: Optional[dict] = None) -> int: """Write JSON/CSV and return a process exit code. Returns 0 when rows were written, EXIT_NO_PRODUCTS when there were none. @@ -447,8 +540,16 @@ def save(rows: Sequence[Any], out_prefix: str, fmt: str, write_json(rows, f"{out_prefix}.json") print(f"[+] Saved {len(rows)} rows -> {out_prefix}.json") if fmt in ("csv", "both"): - write_csv(rows, f"{out_prefix}.csv", row_cls=row_cls) + escaped = write_csv(rows, f"{out_prefix}.csv", row_cls=row_cls) print(f"[+] Saved {len(rows)} rows -> {out_prefix}.csv") + # Through `stats` because this function returns an EXIT CODE. + if stats is not None: + stats["csv_cells_escaped"] = escaped + if escaped: + print(f"[!] {escaped} CSV cell(s) began with a formula character " + f"and were prefixed with {CSV_FORMULA_ESCAPE!r} so a " + f"spreadsheet reads them as text. The JSON output is " + f"unchanged; see csv_cells_escaped in the sidecar.") return 0 if rows else EXIT_NO_PRODUCTS @@ -505,8 +606,13 @@ def finish_run(rows: Sequence[Any], out_prefix: str, fmt: str, # names has a hole for every name nobody added to it. complete = stop_reason in COMPLETE_STOP_REASONS and not pages_failed row_cls = ROW_CLASS_BY_MODE.get(mode, P2PAd) - rc = save(rows, out_prefix, fmt, allow_empty=allow_empty, row_cls=row_cls) + stats: dict = {} + rc = save(rows, out_prefix, fmt, allow_empty=allow_empty, row_cls=row_cls, + stats=stats) wrote_output = bool(rows) or allow_empty + # Housekeeping goes UNDER the caller's `extra`: a collision must never + # let a count this module made up drop a fact about the site. + extra = {**stats, **(extra or {})} if wrote_output: status = "complete" if (rows and complete) else ( diff --git a/page_flow.py b/page_flow.py index 1a69632..351c646 100644 --- a/page_flow.py +++ b/page_flow.py @@ -195,10 +195,15 @@ def refusal_advice(state: str) -> str: """One sentence on what changes the answer, per refusal. Kept here so the three engines cannot give three different pieces of advice.""" if state == "restricted": - return ("HTTP 451 means the site refuses this exit's COUNTRY. " - "Binance's terms exclude some jurisdictions, the United " - "States among them. Use an exit elsewhere: --proxy, or a " - "Scraping Browser country- segment.") + # Kept as a state because it is cheap, but nobody has seen + # binance.com answer 451: the daily canary is served from a + # GitHub-hosted runner with no proxy. So the advice states the + # status and does not send anyone to buy an exit over a guess. + return ("HTTP 451 (unavailable for legal reasons) has not been " + "observed on binance.com in testing. Please open an issue " + "with the sidecar's stop_reason and the exit's country; an " + "exit in another COUNTRY (--proxy, or a Scraping Browser " + "country- segment) is what would change this answer.") if state == "challenge": return ("AWS WAF challenged this session. A residential exit " "(--proxy) may not be challenged at all; otherwise set " @@ -241,6 +246,17 @@ def build_query(args, error: Callable[[str], None]) -> Query: (the family's --country rule, §10). `error` is argparse's `p.error`, so a refusal is exit 2 with the usage line, as in every engine. """ + typed_url = getattr(args, "typed_url", args.url) + if args.url and not typed_url: + # The URL came from BINANCE_URL, not from the command line. It is a + # default for a run given nothing else, so a --mode or a query flag + # the user typed wins instead of being refused as a conflict. + typed_flags = [flag for dest, flag in QUERY_FLAGS + if getattr(args, dest, None) is not None] + if args.mode or typed_flags: + log.info("Ignoring BINANCE_URL from the environment: the query " + "was given on the command line.") + args.url = None if args.url: query, why = query_from_url(args.url) if query is None: @@ -441,15 +457,49 @@ def ok(self) -> bool: } -def land(ops, args) -> Tuple[str, Optional[str]]: +class SolveBudget: + """The paid-solve allowance for ONE page, shared by every landing on it. + + Created once per `fetch_one_page` call and handed to every `land()` that + page makes. It is checked BEFORE each purchase and is never reset by a + transport retry, a block-retry or an exit rotation: a fresh browser or a + fresh exit is a reason to fetch again, not a fresh allowance. + + Before this existed the allowance was a loop inside `land()`, so it + restarted on every landing. With the solver stubbed, one refused page + bought 2 solves with no proxy pool and 4 with a 3-exit pool, against + SOLVES_PER_PAGE = 1. + """ + + def __init__(self, limit: int = SOLVES_PER_PAGE): + self.limit = limit + self.spent = 0 + + def spend(self, solve: Callable[[], bool]) -> bool: + """Call `solve` if the allowance has room; return what it returned.""" + if self.spent >= self.limit: + log.warning("The CAPTCHA came back, and this page's solve " + "allowance (%d) is already spent — not paying for " + "another.", self.limit) + return False + self.spent += 1 + return bool(solve()) + + +def land(ops, args, budget: Optional[SolveBudget] = None + ) -> Tuple[str, Optional[str]]: """Put the page on a www.binance.com document that fetch() can use. Returns (state, error): state is a policy state for the landing, and - error is a transport failure's text or None. Solves at most - SOLVES_PER_PAGE CAPTCHAs, because a CAPTCHA that survives a solved token - is not one this run can pass, and a second solve is a second charge for - the same answer. + error is a transport failure's text or None. Every solve goes through + `budget`, the allowance of the page this landing serves, so landing + again after a retry or a rotation cannot buy a second token for the + same page. A CAPTCHA that survives a solved token is not one this run + can pass, and a second solve is a second charge for the same answer. + Without a budget the landing solves nothing. """ + if budget is None: + budget = SolveBudget(limit=0) try: status, waf = ops.goto(ORIGIN_URL) except TransportError as e: @@ -465,13 +515,10 @@ def land(ops, args) -> Tuple[str, Optional[str]]: ops.wait_ms(CHALLENGE_POLL_MS) waited += CHALLENGE_POLL_MS state = classify(ops.document_text(), None, ORIGIN_URL, None) - if state == "challenge" and should_solve(state): - for _ in range(SOLVES_PER_PAGE): - if not ops.solve_captcha(): - break - state = classify(ops.document_text(), None, ORIGIN_URL, None) - if state != "challenge": - break + while state == "challenge" and should_solve(state): + if not budget.spend(ops.solve_captcha): + break + state = classify(ops.document_text(), None, ORIGIN_URL, None) ops.landed = state in ("content", "empty") return state, None @@ -554,6 +601,8 @@ def fetch_one_page(ops, args, pool, query: Query, page_num: int, args.proxy_block_retries if has_pool else BLOCK_RETRIES_WITHOUT_POOL) throttles = 0 state, text, last_error, exit_failed = "unknown", "", None, None + # One allowance for the whole page, across every landing below. + budget = SolveBudget() for block_attempt in range(block_retries + 1): log.info("Fetching page %d/%d: %s", page_num, args.pages, req.label) @@ -563,7 +612,7 @@ def fetch_one_page(ops, args, pool, query: Query, page_num: int, attempt += 1 text, last_error = "", None if not ops.landed: - state, last_error = land(ops, args) + state, last_error = land(ops, args, budget) if last_error: exit_failed = ops.proxy_failure(last_error) or None if exit_failed: @@ -683,6 +732,8 @@ def check_pay_types(ops, args, query: Query) -> Optional[str]: if not query.pay_types: return None if not ops.landed: + # No solve here: this is a courtesy check, not a page. If the landing + # is a CAPTCHA, page 1 lands again and pays from its own allowance. land(ops, args) offered = None if ops.landed: @@ -837,18 +888,18 @@ def worker_pool(pool, worker_index: int): def cdp_connect_hint(error_text: str) -> str: """What a failed --cdp-endpoint connection means, from its status. - Two answers that want opposite fixes. Measured 2026-09-24 against four - Scraping Browser endpoints left in sibling repos' .env files: all four - answered 401 Unauthorized, because a profile's credentials last about a - day. The message used to explain a 500 (a pid another run still holds) + Two answers that want opposite fixes. A 401 is the service refusing the + credentials in the endpoint: expired, mistyped or revoked. How long a + profile's credentials live is not a constant (one endpoint was measured + still working after ten days), so the message does not guess which. + The message used to explain a 500 (a pid another run still holds) whatever the status was, which sent the reader to wait for a run that did not exist. """ if "401" in (error_text or ""): - return ("HTTP 401: the endpoint's credentials were refused. A Scraping " - "Browser profile's credentials last about a day, so an " - "endpoint copied from an older .env has usually expired. " - "Get a fresh one from your 2Captcha dashboard.") + return ("HTTP 401: the endpoint's credentials were refused, for " + "example an expired, revoked or mistyped profile login. " + "Copy the endpoint again from your 2Captcha dashboard.") return ("A Scraping Browser profile allows ONE live connection at a time, " "so an HTTP 500 here usually means another run still holds this " "`pid`. Wait for it to finish, or use a different pid.") diff --git a/playwright_scraper.py b/playwright_scraper.py index 8d11bff..3acda7b 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -608,6 +608,9 @@ def parse_args(argv=None): p.add_argument("--headless", action="store_true", default=True) p.add_argument("--headful", dest="headless", action="store_false") args = p.parse_args(argv) + # Before .env fills it: a URL from the environment is a default, + # and must not refuse a --mode or a query flag the user typed. + args.typed_url = args.url env_config.apply(args) args.query = page_flow.build_query(args, p.error) args.mode = args.query.mode diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index d50969b..37579c8 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -657,6 +657,9 @@ def parse_args(argv=None): p.add_argument("--headless", action="store_true", default=True) p.add_argument("--headful", dest="headless", action="store_false") args = p.parse_args(argv) + # Before .env fills it: a URL from the environment is a default, + # and must not refuse a --mode or a query flag the user typed. + args.typed_url = args.url env_config.apply(args) args.query = page_flow.build_query(args, p.error) args.mode = args.query.mode diff --git a/selenium_scraper.py b/selenium_scraper.py index da027ed..0531dde 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -538,6 +538,9 @@ def parse_args(argv=None): p.add_argument("--headless", action="store_true", default=True) p.add_argument("--headful", dest="headless", action="store_false") args = p.parse_args(argv) + # Before .env fills it: a URL from the environment is a default, + # and must not refuse a --mode or a query flag the user typed. + args.typed_url = args.url env_config.apply(args) args.query = page_flow.build_query(args, p.error) args.mode = args.query.mode diff --git a/smoke_test.py b/smoke_test.py index 946c347..a553c2d 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -371,6 +371,42 @@ def err(msg): equal("flag defaults apply without --url", (q.time_range, q.sort_by, q.order), ("30D", "roi", "desc")) + # BINANCE_URL from .env is a DEFAULT. A --mode or a query flag the user + # typed wins over it instead of being refused as a conflict with a URL + # they never typed. A copied example used to break every README example + # that builds its query from flags. + def built(**kw): + for k, v in kw.items(): + setattr(args, k, v) + errors.clear() + try: + q = page_flow.build_query(args, err) + except SystemExit: + return ("refused", errors[0] if errors else "") + return (q.mode, q.fiat) + + env_url = "https://p2p.binance.com/en/trade/all-payments/USDT?fiat=EUR" + got = built(url=env_url, typed_url=None, mode="announcements") + equal("an ENVIRONMENT url gives way to a typed --mode", got[0], "announcements") + equal("...and to a typed query flag", + built(url=env_url, typed_url=None, mode=None, fiat="TRY"), ("p2p", "TRY")) + equal("...and is used when nothing was typed", + built(url=env_url, typed_url=None, mode=None, fiat=None), ("p2p", "EUR")) + args.url = args.typed_url = env_url + args.fiat = "TRY" + errors.clear() + try: + page_flow.build_query(args, err) + except SystemExit: + pass + check("a TYPED --url with --fiat is still refused", + errors and "--fiat" in errors[0], repr(errors)) + for engine in list(ENGINES) + ["http_scraper"]: + src = open(os.path.join(HERE, engine + ".py"), encoding="utf-8").read() + i, j = src.find("args.typed_url = args.url"), src.find("env_config.apply(args)") + check("%s records the typed --url BEFORE .env fills it" % engine, + 0 <= i < j, "typed_url at %d, apply at %d" % (i, j)) + # --------------------------------------------------------------------------- # Page state: what the site answered with @@ -455,9 +491,23 @@ def check_state_policy(): "COUNTRY" in F.refusal_advice("restricted")) check("...and claims nothing about what binance.com has been SEEN doing", "answers" not in F.refusal_advice("restricted")) - check("a CDP 401 is explained as expired credentials, not a held pid", - "expired" in F.cdp_connect_hint("WebSocket error: 401 Unauthorized") - and "pid" not in F.cdp_connect_hint("401 Unauthorized")) + # Nobody has seen binance.com answer 451, and the daily canary is served + # from a GitHub-hosted runner with no proxy. Advice that names a country + # to avoid would send a reader to buy an exit over a guess. + check("451's advice says it has not been observed, and names no country to avoid", + "not been observed" in F.refusal_advice("restricted") + and "United States" not in F.refusal_advice("restricted")) + for doc in ("README.md", "TROUBLESHOOTING.md", "CONTRIBUTING.md", ".env.example", + os.path.join(".github", "ISSUE_TEMPLATE", "bug_report.yml")): + path = os.path.join(HERE, doc) + if os.path.exists(path): + check("%s does not repeat the unmeasured US refusal" % doc, + "United States" not in open(path, encoding="utf-8").read()) + hint = F.cdp_connect_hint("WebSocket error: 401 Unauthorized") + check("a CDP 401 is explained as refused credentials, not a held pid", + "refused" in hint and "pid" not in F.cdp_connect_hint("401 Unauthorized")) + check("...without claiming credentials last a fixed time", + "about a day" not in hint) check("...and a 500 as a held pid", "pid" in F.cdp_connect_hint("HTTP 500")) waf = fx("waf_captcha_chromium") check("the captured CAPTCHA page names its cookie domains (not vacuous)", @@ -610,6 +660,59 @@ def check_the_shared_loop_end_to_end(): equal("...no search page was fetched", [p for p in ops.fetches if p != 0], []) +class _PaidCaptchaOps(_FakeOps): + """A landing that is a real AWS WAF CAPTCHA and stays one after a + "successful" solve, with a solver that reports success every time. The + count of purchases is the only thing being measured.""" + + def __init__(self, pool=None): + _FakeOps.__init__(self, {}, landing=fx("waf_captcha_chromium")) + self.pool = pool + + def goto(self, url): + self.gotos += 1 + return 405, "captcha" + + def solve_captcha(self): + self.solves += 1 + return True + + +def check_the_solve_allowance_is_per_page_not_per_landing(): + """A refused page lands again after every block-retry and every rotation. + The allowance must survive those: one page, at most SOLVES_PER_PAGE + purchases, however many times the loop lands. With the allowance kept + inside each landing this bought 2 solves with no pool and 4 with a + 3-exit pool.""" + import page_flow + from product_parser import Query + import proxy_pool + check("the CAPTCHA landing really classifies as a challenge (not vacuous)", + page_flow.classify(fx("waf_captcha_chromium"), 405, + page_flow.ORIGIN_URL, "captcha") == "challenge") + for label, pool, block_retries in ( + ("no pool", None, 3), + ("3-exit pool", proxy_pool.ProxyPool( + ["http://10.0.0.1:1", "http://10.0.0.2:2", "http://10.0.0.3:3"]), 3)): + args = types.SimpleNamespace( + pages=1, retries=3, retry_delay=0, delay=0, + proxy_block_retries=block_retries, out="unused", format="json", + allow_empty=False, dump_html=None, url=None, cdp_endpoint=None, + concurrency=1) + ops = _PaidCaptchaOps(pool) + with tempfile.TemporaryDirectory() as tmp: + args.out = os.path.join(tmp, "out") + outcome = page_flow.fetch_one_page(ops, args, pool, + Query("p2p", fiat="EUR"), 1) + check("solve allowance, %s: the page was refused and the loop landed " + "more than once (not vacuous)" % label, + outcome.blocked_by == "aws-waf" and ops.gotos > 1, + "blocked_by=%r gotos=%d" % (outcome.blocked_by, ops.gotos)) + equal("solve allowance, %s: one page buys at most SOLVES_PER_PAGE " + "solves across every landing" % label, + ops.solves, page_flow.SOLVES_PER_PAGE) + + def _copy_shrunk_to(total): """The captured past-the-end copy-trading page, with the total a listing that SHRANK to `total` states. Measured 2026-09-30 on all three @@ -895,6 +998,116 @@ def check_csv_and_json_writers(): check("an id past 2**53 stays a string in JSON", isinstance(loaded[0]["sku"], str)) +def check_writes_are_atomic_and_keep_the_umask_mode(): + """A failure part-way through a write must leave the previous file intact + and no temp file behind. Before the atomic writer, a failure inside + json.dump turned a good 786-byte out.json into 13 bytes of invalid JSON. + And the temp file's 0600 must not become the output's mode.""" + import output_writer as O + import product_parser as P + rows = P.parse_page(fx("p2p_usdt_eur_buy"), _q("p2p", fiat="EUR"), 1) + with tempfile.TemporaryDirectory() as tmp: + prefix = os.path.join(tmp, "out") + O.finish_run(rows, prefix, "both", False, blocked=False, + stop_reason="completed", pages_requested=1, + pages_completed=1, start_url="u", final_url="u", mode="p2p") + before = {ext: open(prefix + ext, encoding="utf-8").read() + for ext in (".json", ".csv", ".meta.json")} + + class Boom(Exception): + pass + + def exploding_dump(obj, f, **kw): + f.write("[{\"partial\": ") + raise Boom() + real_dump = O.json.dump + O.json.dump = exploding_dump + try: + for name, call in ( + ("write_json", lambda: O.write_json(rows, prefix + ".json")), + ("write_run_meta", lambda: O.write_run_meta(prefix, {"status": "x"}))): + try: + call() + except Boom: + pass + finally: + O.json.dump = real_dump + real_writerow = csv.DictWriter.writerow + + def exploding_writerow(self, row): + raise Boom() + csv.DictWriter.writerow = exploding_writerow + try: + try: + O.write_csv(rows, prefix + ".csv", row_cls=O.P2PAd) + except Boom: + pass + finally: + csv.DictWriter.writerow = real_writerow + for ext, good in before.items(): + equal("a write that fails part-way leaves the previous %s intact" % ext, + open(prefix + ext, encoding="utf-8").read(), good) + equal("...and leaves no temporary file behind", + sorted(os.listdir(tmp)), ["out.csv", "out.json", "out.meta.json"]) + + old_umask = os.umask(0o022) + try: + for umask, want in ((0o022, 0o644), (0o002, 0o664), (0o077, 0o600)): + os.umask(umask) + path = os.path.join(tmp, "new_%o.json" % umask) + O.write_json(rows, path) + equal("a NEW output under umask %03o is %o, as open() would make it" + % (umask, want), os.stat(path).st_mode & 0o777, want) + os.umask(0o022) + os.chmod(prefix + ".json", 0o640) + O.write_json(rows, prefix + ".json") + equal("an EXISTING output keeps its own mode", + os.stat(prefix + ".json").st_mode & 0o777, 0o640) + finally: + os.umask(old_umask) + + +def check_csv_cells_are_formula_escaped(): + """Advertiser nicknames, portfolio names and announcement titles are + third-party text. A cell starting with = + - @ or a control character is + run by a spreadsheet; CSV gets an apostrophe, JSON keeps the bytes.""" + import output_writer as O + import product_parser as P + rows = P.parse_page(fx("p2p_usdt_eur_buy"), _q("p2p", fiat="EUR"), 1)[:2] + rows[0].title = '=HYPERLINK("x")' + rows[0].pay_methods = ["-cmd", "ok"] + rows[1].title = "@SUM(1)" + rows[1].price = -5.0 + with tempfile.TemporaryDirectory() as tmp: + prefix = os.path.join(tmp, "out") + O.finish_run(rows, prefix, "both", False, blocked=False, + stop_reason="completed", pages_requested=1, + pages_completed=1, start_url="u", final_url="u", mode="p2p", + extra={"total_results": 2}) + reader = list(csv.DictReader(open(prefix + ".csv", encoding="utf-8"))) + equal("a formula-shaped title is escaped in CSV", reader[0]["title"], + '\'=HYPERLINK("x")') + equal("...and a joined list cell too (escaped AFTER serialising)", + reader[0]["pay_methods"], "'-cmd | ok") + equal("...and @", reader[1]["title"], "'@SUM(1)") + equal("a negative NUMBER is left numeric", reader[1]["price"], "-5.0") + loaded = json.load(open(prefix + ".json", encoding="utf-8")) + equal("JSON keeps the site's bytes", (loaded[0]["title"], loaded[0]["pay_methods"]), + ('=HYPERLINK("x")', ["-cmd", "ok"])) + meta = json.load(open(prefix + ".meta.json", encoding="utf-8")) + equal("the sidecar declares csv_cells_escaped", meta.get("csv_cells_escaped"), 3) + equal("...beside the caller's own extra", meta.get("total_results"), 2) + with tempfile.TemporaryDirectory() as tmp: + prefix = os.path.join(tmp, "out") + O.finish_run(rows, prefix, "both", False, blocked=False, + stop_reason="completed", pages_requested=1, + pages_completed=1, start_url="u", final_url="u", mode="p2p", + extra={"csv_cells_escaped": "caller"}) + meta = json.load(open(prefix + ".meta.json", encoding="utf-8")) + equal("housekeeping merges UNDER the caller's extra, never over it", + meta.get("csv_cells_escaped"), "caller") + + def check_exit_codes(): import output_writer as O equal("3 blocked / 4 empty / 5 never obtained / 6 partial", @@ -1207,7 +1420,9 @@ def check_dockerfile_copies_everything_the_entrypoint_imports(): def check_env_example_documents_exactly_what_the_loader_reads(): import env_config text = open(os.path.join(HERE, ".env.example"), encoding="utf-8").read() - documented = set(re.findall(r"^([A-Z][A-Z0-9_]+)=", text, re.M)) + # A commented-out example (`# BINANCE_URL=...`) documents the variable + # without setting it. + documented = set(re.findall(r"^(?:# )?([A-Z][A-Z0-9_]+)=", text, re.M)) read = set(env_config.ENV_KEYS) equal("the example and the loader name the same variables", sorted(documented), sorted(read)) From 65b6d7dadfaba2cef4fb7880f1aebd7e809bb880 Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Wed, 14 Oct 2026 18:55:51 +0000 Subject: [PATCH 2/3] Comments state their reasons instead of pointing at private notes Every comment and docstring that cited a private note or a bare section number now says the rule in its own words, or drops the pointer. A suite check fails if such a citation reappears in any shipped file. Co-Authored-By: Claude Opus 5.5 --- .github/ci_checks.py | 2 +- .github/workflows/canary.yml | 4 +- .gitignore | 6 +-- CONTRIBUTING.md | 4 +- captcha_solver.py | 6 +-- env_config.py | 4 +- fingerprint_client.py | 8 ++-- http_scraper.py | 8 ++-- output_writer.py | 6 +-- page_flow.py | 34 +++++++-------- playwright_scraper.py | 24 +++++------ product_parser.py | 10 ++--- puppeteer_scraper.py | 19 ++++----- selenium_scraper.py | 16 +++---- smoke_test.py | 82 ++++++++++++++++++++++++++---------- 15 files changed, 134 insertions(+), 99 deletions(-) diff --git a/.github/ci_checks.py b/.github/ci_checks.py index 4a74574..163571e 100644 --- a/.github/ci_checks.py +++ b/.github/ci_checks.py @@ -295,7 +295,7 @@ def tracked_files(): # into pip's vendored code and flags a 32-hex string in `_elffile.py` as # key-shaped. Correct about the string, wrong about the file — and it is the # FIRST thing a new user sees from `python3 smoke_test.py`. A guard people -# have to argue with is one they learn to suppress (CLAUDE.md §22). +# have to argue with is one they learn to suppress. # # Deliberately NOT a narrowing of what gets scanned: an untracked file is # still read, because a key pasted into a scratch file beside the scripts is diff --git a/.github/workflows/canary.yml b/.github/workflows/canary.yml index a9e8cfa..9819857 100644 --- a/.github/workflows/canary.yml +++ b/.github/workflows/canary.yml @@ -3,7 +3,7 @@ # # No secrets are needed, and that is the point of running it: the README's # central claim is "no key, no proxy, no account", and an ungated scheduled -# canary is that sentence under test every morning (CLAUDE.md §21, §24). All +# canary is that sentence under test every morning. All # three data endpoints answered a datacentre address normally when this repo # was written (2026-09-24); a GitHub runner is a datacentre address too. # @@ -78,7 +78,7 @@ jobs: playwright install --with-deps chromium fi - # >= 3 pages, because with one page pagination is never exercised (§11). + # >= 3 pages, because with one page pagination is never exercised. - name: Run id: run run: | diff --git a/.gitignore b/.gitignore index 7a38169..52c3b59 100644 --- a/.gitignore +++ b/.gitignore @@ -23,7 +23,7 @@ binance_rows_*.* live_results/ live_results.zip # ANY working directory a live run writes into, by KIND rather than by the one -# name an older repo used (CLAUDE.md §24). Measured here: `live/` was not +# name an older repo used. Measured here: `live/` was not # listed, and a `--dump-html` of a 3-page run wrote `canary_dump.json.page3`, # which `*.json` does not cover. It sat untracked and NOT ignored, one # `git add -A` from a commit, until the credential scan flagged the 32-hex @@ -100,7 +100,7 @@ secrets* # recoverable only because that repo was private at the time. On 2026-09-17 # a sibling had an untracked `.env.bak` holding a 32-hex key in a PUBLIC # repository, one `git add -A` from publication. Craigslist was fixed then -# and the fix never reached anyone else, which is §16's rule unapplied: when -# you fix one, check the siblings. +# and the fix never reached anyone else. When you fix one, check the +# siblings. .env* !.env.example diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c69cbce..908a7e9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -92,8 +92,8 @@ Then the rest of the presentation, in the order that matters: three modes from a GitHub-hosted runner with no proxy. If a later run is refused for the runner's address, the canary's `BINANCE_PROXY` secret (an exit elsewhere) is the fix, with no workflow edit. -2. The repo description, homepage and topics set (see the family notes on - what those should say). +2. The repo description, homepage and topics set: the engines, the + 2Captcha Scraping Browser API by that name, and what the site publishes. 3. Only then the row in the org profile README — and check it with an ANONYMOUS request rather than your own logged-in browser. A row pointing at a private repo is a 404 for every visitor, which costs more trust than diff --git a/captcha_solver.py b/captcha_solver.py index 4787050..3a1468e 100644 --- a/captcha_solver.py +++ b/captcha_solver.py @@ -99,7 +99,7 @@ class CaptchaChallenge: page_url: str = "" # AWS WAF only. Its task needs three values that all live together in # `window.gokuProps` on the challenge page, plus the two script URLs the - # page loads. Unlike Cloudflare Turnstile (§19 of the family notes), + # page loads. Unlike Cloudflare Turnstile, # nothing has to be intercepted at runtime to get them: AWS WAF prints # all of it into the served HTML, so a static read is enough. iv: Optional[str] = None @@ -151,7 +151,7 @@ def aws_waf_action(self) -> str: def has_captcha_widget(self) -> bool: """Is there a widget on this page for a solver to work on? - Section 19's rule: "unsolvable" is a property of a PAGE, not of a + "Unsolvable" is a property of a PAGE, not of a vendor, and it means the page carries no widget. A challenge-action page carries none, so sending it to a solver buys a token for a puzzle that was never rendered -- measured at $0.00145 a time, since @@ -949,4 +949,4 @@ def solve_recaptcha(challenge: CaptchaChallenge, twocaptcha_api_key: Optional[st # what the address has been doing — a narrow list is how a challenge gets # reported as an empty page months later. A solver for a challenge this site # has never been observed to serve is dead code; a DETECTOR for one is cheap -# insurance. See the family note in CLAUDE.md. +# insurance. diff --git a/env_config.py b/env_config.py index 8316fe7..fda1f4b 100644 --- a/env_config.py +++ b/env_config.py @@ -66,8 +66,8 @@ # A literal-only check reported both of those as CONFIGURED, so `cp # .env.example .env` followed by a run connected to cb.2captcha.com with the # string `{login}-zone-…` as its username and got a 401 — a confusing failure -# a long way from its cause, which is precisely what §3's rule exists to -# prevent. Any `{…}` left in a value means the line was never filled in. +# a long way from its cause, which is precisely what reading .env strictly +# exists to prevent. Any `{…}` left in a value means the line was never filled in. _BRACED_PLACEHOLDER_RE = re.compile(r"\{[A-Za-z_][A-Za-z0-9_]*\}") _loaded_from = None diff --git a/fingerprint_client.py b/fingerprint_client.py index e84ce1e..7ed887f 100644 --- a/fingerprint_client.py +++ b/fingerprint_client.py @@ -249,9 +249,9 @@ def playwright_context_kwargs(fp: dict) -> dict: # stating `deviceScaleFactor: 1.25` produced a browser reporting # `window.devicePixelRatio === 1`, so the identity contradicted itself # on an axis any fingerprinter reads for free -- and a contradiction is - # exactly what this flag exists to avoid. Same shape as the three - # defects the family notes already list for this file: a key the API - # returns that nothing applies. Found by comparing a live browser + # exactly what this flag exists to avoid. Same shape as three + # earlier defects in this file: a key the API returns that nothing + # applies. Found by comparing a live browser # against the fingerprint rather than by reading the code. scale = (fp.get("screen") or {}).get("deviceScaleFactor") if isinstance(scale, (int, float)) and not isinstance(scale, bool) and scale > 0: @@ -331,7 +331,7 @@ def main() -> int: # loader. It used to read `os.environ` alone, which meant a key put in # `.env` exactly as the README and .env.example instruct worked for every # engine and failed HERE with "No API key" — a documented mechanism not - # applied on one path, which is the shape of half the defects §16 lists. + # applied on one path. # # `.env` has to be LOADED before it can be read: `env_value` looks at # os.environ, and `load_env` is what fills that from the file. Calling it diff --git a/http_scraper.py b/http_scraper.py index 56bed11..1d8e5bd 100644 --- a/http_scraper.py +++ b/http_scraper.py @@ -28,10 +28,10 @@ 15-second wait for a script to run, and names the engine that can. * **No --cdp-endpoint, --fingerprint, --locale or --headless**: they describe a browser. `--proxy`, `--proxy-file` and rotation work: a - rotation is a fresh session, so no cookie crosses exits (§8). + rotation is a fresh session, so no cookie crosses exits. * **Its own identity.** It sends requests' own User-Agent rather than claiming to be Chrome over a TLS handshake that is not Chrome's, which is - the contradiction a sibling site refused (§24). Measured here: P2P + the contradiction a sibling site refused. Measured here: P2P answered both identities 6 of 6. Usage @@ -68,7 +68,7 @@ # The words requests and urllib3 use for a proxy that could not be used. # Distinguished from a timeout because the two want opposite responses: a # timeout deserves another try at the same exit, a dead proxy a different -# one (§8). +# one. _PROXY_ERROR_MARKERS = ( "ProxyError", "Unable to connect to proxy", @@ -164,7 +164,7 @@ def proxy_failure(self, text: str) -> str: def relaunch(self): """A fresh session on the pool's current exit, so no cookie issued - against one exit is replayed from another (§8).""" + against one exit is replayed from another.""" self.close() self.open() diff --git a/output_writer.py b/output_writer.py index d44adef..169a2ea 100644 --- a/output_writer.py +++ b/output_writer.py @@ -11,7 +11,7 @@ These are three different things with nothing in common beyond an id and a name, so they are three dataclasses rather than one wide row that is two- -thirds null on every line (CLAUDE.md §9: a column that is null on every row +thirds null on every line (a column that is null on every row of a mode should not exist in that mode's file). What they DO share, byte-identical and in order, is the family prefix: @@ -120,7 +120,7 @@ class P2PAd: page: Optional[int] = None position: Optional[int] = None mode: Optional[str] = None - # "bapi": the site's own JSON endpoint. Provenance in a column (§8). + # "bapi": the site's own JSON endpoint. Provenance in a column. data_source: Optional[str] = None @@ -179,7 +179,7 @@ class LeadTrader: # column and not only a sidecar field because it decides WHICH # portfolios are in a capped run at all: the first 300 by ROI and the # first 300 by AUM are different samples. `diff_runs.py` refuses to - # compare two runs that differ here (§21). + # compare two runs that differ here. sort: Optional[str] = None data_source: Optional[str] = None diff --git a/page_flow.py b/page_flow.py index 351c646..6f80a17 100644 --- a/page_flow.py +++ b/page_flow.py @@ -2,7 +2,7 @@ page_flow.py ------------ The retry / solve / blocked decision, as DATA rather than as three copies of -an if-chain (CLAUDE.md §1). +an if-chain. binance.com answers one of this repo's requests in eight ways, and they want six different responses: @@ -24,8 +24,8 @@ would be silent: one engine reporting exit 3 where its twin reports exit 0 on the same response. -Nothing here imports a browser, and **no JavaScript crosses this boundary** -(§1). Each engine spells its fetch() in its own driver's dialect. +Nothing here imports a browser, and **no JavaScript crosses this boundary**. +Each engine spells its fetch() in its own driver's dialect. """ import logging @@ -52,7 +52,7 @@ # How long one fetch() may take before the engine gives up on it. The # largest response measured was 76 KB of P2P JSON, which arrived in well # under a second. The bound exists because a browser fetch() has no timeout -# of its own, and CLAUDE.md §8 requires every remote call to have one. +# of its own, and every remote call here has to be bounded. FETCH_TIMEOUT_MS = 30_000 # How long to wait at the SAME exit after a 429/418 before trying again. @@ -81,7 +81,7 @@ def classify(html: Optional[str], status: Optional[int] = None, The argument ORDER is the contract: every engine calls `classify(html, status, url, waf_action)`. A sibling repo shipped `classify(html, url=...)` in two of three engines against a callee that - took `status` second, and both crashed on their first fetch (§17). + took `status` second, and both crashed on their first fetch. `smoke_test.py` binds every engine's call against this signature for that reason. """ @@ -107,7 +107,7 @@ def classify(html: Optional[str], status: Optional[int] = None, # Rate limited. The retry happens at the same exit after a wait # (THROTTLE_*). It is NOT counted as blocked: calling a throttle a block # reports exit 3 for a page that was about to come back, and sends a - # reader to buy a proxy they do not need (§24). + # reader to buy a proxy they do not need. "throttled": {"retry": True, "solve": False, "blocked": False, "parse": False}, # HTTP 451, "Unavailable For Legal Reasons". NOT OBSERVED on binance.com # by this repo: Binance's own API documentation names 403 (WAF), 429 and @@ -138,7 +138,7 @@ def should_parse(state: str) -> bool: # Whether a blocked page is worth re-fetching at all. CONSULTED by every -# engine, so setting it False really does stop the retry loop (§17). +# engine, so setting it False really does stop the retry loop. RETRY_ON_BLOCKED = True # How many times to re-fetch a blocked page when there is no proxy pool to @@ -176,7 +176,7 @@ def concurrency_limit(cdp_endpoint: Optional[str]) -> Optional[int]: The Scraping Browser API allows ONE live connection per profile, so N workers sharing a `pid` collide with `profile_locked`. Several `pid`s, - one run each, is the way to parallelise that path (§7). + one run each, is the way to parallelise that path. """ return 1 if cdp_endpoint else None @@ -243,7 +243,7 @@ def build_query(args, error: Callable[[str], None]) -> Query: --url and the query flags are two ways to say the same thing. A flag the user typed that disagrees with the URL would silently scrape something neither of them named, so the combination is refused rather than merged - (the family's --country rule, §10). `error` is argparse's `p.error`, so + (the family's --country rule). `error` is argparse's `p.error`, so a refusal is exit 2 with the usage line, as in every engine. """ typed_url = getattr(args, "typed_url", args.url) @@ -309,7 +309,7 @@ def finish(args, query: Query, outcomes: List, stop_reason: str, """Merge the pages in PAGE order, write the output, return the exit code. One implementation for the three engines, so the merge order, the - dedupe and the sidecar cannot differ between them (§6). + dedupe and the sidecar cannot differ between them. """ rows, seen = [], set() for oc in sorted(outcomes, key=lambda o: o.page_num): @@ -390,7 +390,7 @@ def cookie_domain(host: Optional[str], html: Optional[str] = None) -> str: # transport failure, waiting out a throttle, rotating on a refusal, and # parsing what came back. The three engines differ only in HOW they ask # their driver, so each passes in an object with these operations and no -# JavaScript crosses this boundary (§1): +# JavaScript crosses this boundary: # # ops.goto(url) -> (status, waf_header). Raises TransportError. # ops.document_text() -> the landing document, JSON text if it is JSON @@ -402,7 +402,7 @@ def cookie_domain(host: Optional[str], html: Optional[str] = None) -> str: # ops.proxy_failure(text) -> the driver's proxy-error name in text, or "" # # Before this, each engine carried its own copy of the loop, and a family -# rule (§6) says the three must agree on exit codes and on whether a run +# rule says the three must agree on exit codes and on whether a run # spends money. One copy is how that holds by construction. @@ -416,7 +416,7 @@ class PageOutcome: Collected per page and merged afterwards, in page order, rather than folded into shared state as the loop goes, so the output cannot depend - on which page happened to finish first (§8). + on which page happened to finish first. """ page_num: int url: str @@ -567,7 +567,7 @@ def _unread(rows: List, total: Optional[int], pages: Optional[int], def _dump(args, page_num: int, text: str) -> None: - """Write the exact response the parser was given, on success too (§9).""" + """Write the exact response the parser was given, on success too.""" if not args.dump_html: return path = args.dump_html if args.pages == 1 else f"{args.dump_html}.page{page_num}" @@ -630,7 +630,7 @@ def fetch_one_page(ops, args, pool, query: Query, page_num: int, ops.landed = False if state == "throttled" and throttles < THROTTLE_RETRIES: throttles += 1 - attempt -= 1 # a throttle wait spends its own budget (§24) + attempt -= 1 # a throttle wait spends its own budget pause = THROTTLE_WAIT_S * throttles log.warning("Rate-limited on page %d (HTTP 429/418) — waiting " "%.0fs at the same exit (%d/%d).", page_num, pause, @@ -768,7 +768,7 @@ def run_pages(open_ops, close_ops, run_concurrently, args, pool, return 2 # Page 1 is always fetched alone: its total decides how many pages - # there are to address (§7). + # there are to address. first = fetch_one_page(ops, args, pool, query, 1, mask) outcomes.append(first) if not first.ok: @@ -875,7 +875,7 @@ def worker_pool(pool, worker_index: int): """A private ProxyPool for one worker, starting at a different exit. Workers start on distinct exits and share no mutable state, so rotation - needs no lock (§7). + needs no lock. """ if not pool: return None diff --git a/playwright_scraper.py b/playwright_scraper.py index 3acda7b..163257a 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -87,7 +87,7 @@ # Chromium's own names for a proxy that could not be used. Distinguished -# from a timeout because the two want opposite responses (CLAUDE.md §8). +# from a timeout because the two want opposite responses. _PROXY_ERROR_MARKERS = ( "ERR_PROXY_CONNECTION_FAILED", "ERR_TUNNEL_CONNECTION_FAILED", @@ -101,13 +101,13 @@ # The fetch() every page goes through. It is handed to `page.evaluate` as a # FUNCTION, which Playwright sends through `Runtime.callFunctionOn` rather -# than evaluating a string, so it works under any Content-Security-Policy -# (§18). It returns the status, the body and the WAF's own header, because +# than evaluating a string, so it works under any Content-Security-Policy. +# It returns the status, the body and the WAF's own header, because # a 202 challenge has an empty body and the header is then the only signal. # # `credentials: "include"` so a solved `aws-waf-token` cookie rides along. # The AbortController is the timeout: a browser fetch() has none of its own, -# and §8 requires every remote call to be bounded. +# and every remote call here has to be bounded. FETCH_JS = """ async ([url, method, body, timeoutMs]) => { const ctl = new AbortController(); @@ -137,7 +137,7 @@ def _chrome_ua(chromium_version: str) -> str: - """A desktop-Chrome UA naming the browser's OWN real version (§8).""" + """A desktop-Chrome UA naming the browser's OWN real version.""" return (f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " f"(KHTML, like Gecko) Chrome/{chromium_version} Safari/537.36") @@ -156,7 +156,7 @@ def _launch_local(pw, args, pool): A proxy rotation tears the whole browser down and calls this again. Cookies a bot manager issued against one exit, replayed from another, - are a stronger signal than either address alone (§8), and on this site + are a stronger signal than either address alone, and on this site that includes a solved `aws-waf-token`. """ launch_kwargs = {"headless": args.headless} @@ -194,7 +194,7 @@ class _Ops: A rotation means a genuinely FRESH browser: cookies a bot manager issued against one exit, replayed from another, are a stronger signal than - either address alone (§8), and here that includes a solved + either address alone, and here that includes a solved `aws-waf-token`. """ @@ -311,7 +311,7 @@ def _connect_remote(pw, args): break if browser is None: # The endpoint carries a password, and Playwright repeats it five - # times in its error text (§8). Rewritten with it masked, keeping + # times in its error text. Rewritten with it masked, keeping # host and port, which are the useful half. raise PWError( f"could not connect to --cdp-endpoint " @@ -340,7 +340,7 @@ def _connect_remote(pw, args): # Every `scheme://user:pass@` in a string, however many times it occurs. # Matching GLOBALLY is the point: a Playwright connection error repeats the -# endpoint five times (§8). +# endpoint five times. _CREDENTIALS_IN_URL_RE = re.compile(r"([a-z][a-z0-9+.\-]*://)[^\s/@]+:[^\s/@]+@", re.IGNORECASE) @@ -369,7 +369,7 @@ def handle_captcha_if_present(page, args, proxy: Optional[str] = None) -> bool: Nothing is paid for a page that holds no puzzle. The WAF's CHALLENGE action carries only `challenge.js`, and there is no answer to buy for that. A browser that runs it passes by itself, which is what the settle - wait in `_land` is for (§19: "unsolvable" is a property of a page). + wait in `_land` is for ("unsolvable" is a property of a page). """ try: html = page.content() @@ -394,7 +394,7 @@ def handle_captcha_if_present(page, args, proxy: Optional[str] = None) -> bool: token = solve_recaptcha(challenge, args.twocaptcha_key, api_version=args.captcha_api, min_score=args.min_score, proxy=proxy) - except Exception as e: # noqa: BLE001 — a solver error is a warning (§8) + except Exception as e: # noqa: BLE001 — a solver error is a warning logger.error("Solving the AWS WAF CAPTCHA failed (%s) — continuing.", _mask_credentials(str(e))) return False @@ -418,7 +418,7 @@ def _fetch_pages_concurrently(args, pool, query: Query, page_nums, concurrency: Each worker owns its own Playwright instance, browser and exit: with the sync API a browser belongs to the thread that made it, so sharing one is - not an option even in principle (§7). The page loop itself is + not an option even in principle. The page loop itself is page_flow.worker_loop, shared by all three engines. """ work = queue.Queue() diff --git a/product_parser.py b/product_parser.py index 6d67ed9..aceaa10 100644 --- a/product_parser.py +++ b/product_parser.py @@ -1,7 +1,7 @@ """ product_parser.py ----------------- -Everything this repo knows about binance.com lives here (CLAUDE.md §1). +Everything this repo knows about binance.com lives here. Three modes, three of the site's own JSON endpoints --------------------------------------------------- @@ -25,7 +25,7 @@ each of the three endpoints above, plain curl, no cookies -> HTTP 200, the full JSON the site's own front end renders -So the gate is per ROUTE, not per site (CLAUDE.md §21): the pages are +So the gate is per ROUTE, not per site: the pages are behind AWS WAF and the front end's own data calls are not. There is no HTML parser in this file because no HTML page is needed for any mode. The engines drive a real browser anyway, and the browser earns its keep in two @@ -439,7 +439,7 @@ def query_from_url(url: str) -> Tuple[Optional[Query], Optional[str]]: # `gokuProps` and the token host are the WAF's own integration vocabulary, # not text a page would carry. The bare word `captcha` is NOT a marker: the # Scraping Browser's auto-solve extension injects hunter scripts that say it -# on every page (CLAUDE.md §24). +# on every page. AWS_WAF_MARKERS = ("gokuProps", "token.awswaf.com", "awswaf.com/") # The HTTP statuses AWS WAF uses for an interstitial. 202 comes with an empty @@ -495,7 +495,7 @@ def detect_page_state(text: Optional[str], status: Optional[int] = None, blocked 403, or any other refusal with no widget unknown anything else: not JSON, not an interstitial - Signals are ordered by what they PROVE, not by what they cost (§17): + Signals are ordered by what they PROVE, not by what they cost: parseable JSON with the site's own envelope is checked first, because no interstitial carries it. """ @@ -762,7 +762,7 @@ def parse_page(payload: Any, query: Query, page: int = 1) -> List[Any]: """Every row on one page of one mode's response, in the site's order. `position` counts the rows EMITTED, not the slots in the payload, so a - record the parser drops cannot shift every later position (§24). + record the parser drops cannot shift every later position. """ p = _as_dict(payload) if str(p.get("code")) != OK_CODE: diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index 37579c8..3ecbd12 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -47,8 +47,7 @@ # At module level, deliberately, and not inside the launch path. The offline # suite guards `import puppeteer_scraper` behind try/except ImportError and # REPORTS the skip, and CI's engine-smoke job fails on any reported skip. -# That only works if importing this module actually requires the driver -# (CLAUDE.md §10). +# That only works if importing this module actually requires the driver. from pyppeteer import launch, connect from captcha_solver import detect_aws_waf, solve_recaptcha, AWS_WAF_COOKIE @@ -65,11 +64,11 @@ # Every await in this file goes through the bridge below with a timeout, so a # hung remote call ends the operation instead of the run. pyppeteer provides -# no connect timeout of its own (§8). +# no connect timeout of its own. DEFAULT_OP_TIMEOUT = 120 CONNECT_TIMEOUT = 30 -# Chromium's own names for a proxy that could not be used (§8). +# Chromium's own names for a proxy that could not be used. _PROXY_ERROR_MARKERS = ( "ERR_PROXY_CONNECTION_FAILED", "ERR_TUNNEL_CONNECTION_FAILED", @@ -83,7 +82,7 @@ # pyppeteer's dialect of the fetch() in playwright_scraper.FETCH_JS: the # arguments arrive POSITIONALLY rather than as one array. Same body, same -# return shape, same AbortController timeout (§8), same `credentials: +# return shape, same AbortController timeout, same `credentials: # "include"` so a solved aws-waf-token rides along. FETCH_JS = """ async (url, method, body, timeoutMs) => { @@ -177,7 +176,7 @@ def _cancel_and_stop(): self._thread.join(timeout=5) -# Every `scheme://user:pass@` in a string, however many times it occurs (§8). +# Every `scheme://user:pass@` in a string, however many times it occurs. _CREDENTIALS_IN_URL_RE = re.compile(r"([a-z][a-z0-9+.\-]*://)[^\s/@]+:[^\s/@]+@", re.IGNORECASE) @@ -188,7 +187,7 @@ def _mask_credentials(text: str) -> str: def _chrome_ua(version: str) -> str: - """A desktop-Chrome UA naming the browser's OWN real version (§8). + """A desktop-Chrome UA naming the browser's OWN real version. pyppeteer's `browser.version()` returns "HeadlessChrome/115.0.0.0".""" number = version.split("/")[-1] if "/" in version else version return (f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " @@ -214,7 +213,7 @@ class _Ops: """One pyppeteer browser + page, exposed as page_flow's named operations. Same contract as playwright_scraper._Ops, including the rule that a - rotation means a genuinely FRESH browser (§8). + rotation means a genuinely FRESH browser. """ def __init__(self, bridge: _AsyncBridge, args, pool): @@ -275,7 +274,7 @@ def open(self): exit_url = self.pool.current # Credentials go through page.authenticate(), never onto the # command line: --proxy-server= is part of the browser's argv, - # readable by anything that can run `ps` (§8). + # readable by anything that can run `ps`. scrubbed, credentials = split_credentials(exit_url) launch_args.append(f"--proxy-server={scrubbed}") logger.info("Using proxy exit %s", mask(exit_url)) @@ -439,7 +438,7 @@ def handle_captcha_if_present(ops, args, proxy: Optional[str] = None) -> bool: token = solve_recaptcha(challenge, args.twocaptcha_key, api_version=args.captcha_api, min_score=args.min_score, proxy=proxy) - except Exception as e: # noqa: BLE001 — a solver error is a warning (§8) + except Exception as e: # noqa: BLE001 — a solver error is a warning logger.error("Solving the AWS WAF CAPTCHA failed (%s) — continuing.", _mask_credentials(str(e))) return False diff --git a/selenium_scraper.py b/selenium_scraper.py index 0531dde..54b0889 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -72,7 +72,7 @@ # script timeout cuts the call off with an exception. SCRIPT_TIMEOUT = page_flow.FETCH_TIMEOUT_MS // 1000 + 15 -# Chromium's own names for "the proxy is the problem, not the site" (§8). +# Chromium's own names for "the proxy is the problem, not the site". _PROXY_ERROR_MARKERS = ( "ERR_PROXY_CONNECTION_FAILED", "ERR_TUNNEL_CONNECTION_FAILED", "ERR_PROXY_AUTH_UNSUPPORTED", "ERR_PROXY_AUTH_REQUESTED", @@ -83,7 +83,7 @@ # Selenium's dialect of the fetch() in playwright_scraper.FETCH_JS: a # function BODY run by execute_async_script, with the arguments in # `arguments` and the result handed to the callback Selenium appends last. -# Same request, same return shape, same AbortController timeout (§8). +# Same request, same return shape, same AbortController timeout. FETCH_JS = """ var done = arguments[arguments.length - 1]; var url = arguments[0], method = arguments[1], body = arguments[2]; @@ -109,7 +109,7 @@ BODY_TEXT_JS = "return document.body ? document.body.innerText : '';" -# Every `scheme://user:pass@` in a string, however many times it occurs (§8). +# Every `scheme://user:pass@` in a string, however many times it occurs. _CREDENTIALS_IN_URL_RE = re.compile(r"([a-z][a-z0-9+.\-]*://)[^\s/@]+:[^\s/@]+@", re.IGNORECASE) @@ -120,7 +120,7 @@ def _mask_credentials(text: str) -> str: def _chrome_ua(version: str) -> str: - """A desktop-Chrome UA naming the installed Chrome's own version (§8).""" + """A desktop-Chrome UA naming the installed Chrome's own version.""" return (f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " f"(KHTML, like Gecko) Chrome/{version} Safari/537.36") @@ -162,7 +162,7 @@ class _Ops: """One Chrome driver, exposed as page_flow's named operations. Same contract as playwright_scraper._Ops, including the rule that a - rotation means a genuinely FRESH browser (§8). + rotation means a genuinely FRESH browser. """ def __init__(self, args, pool): @@ -179,7 +179,7 @@ def open(self): logger.info("Attaching to an existing browser at %s.", options.debugger_address) # No UA, no proxy, no fingerprint on this path: the remote browser - # brings its own (§8). + # brings its own. self.driver = webdriver.Chrome(options=options) self._apply_timeouts() return self @@ -219,7 +219,7 @@ def open(self): def _apply_timeouts(self): # Explicit, because a driver that stops answering otherwise hangs - # the run (§8). + # the run. self.driver.set_page_load_timeout(PAGE_LOAD_TIMEOUT) self.driver.set_script_timeout(SCRIPT_TIMEOUT) @@ -331,7 +331,7 @@ def handle_captcha_if_present(ops, args, proxy: Optional[str] = None) -> bool: token = solve_recaptcha(challenge, args.twocaptcha_key, api_version=args.captcha_api, min_score=args.min_score, proxy=proxy) - except Exception as e: # noqa: BLE001 — a solver error is a warning (§8) + except Exception as e: # noqa: BLE001 — a solver error is a warning logger.error("Solving the AWS WAF CAPTCHA failed (%s) — continuing.", _mask_credentials(str(e))) return False diff --git a/smoke_test.py b/smoke_test.py index a553c2d..8e16960 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -121,7 +121,7 @@ def check_fixture_corpus_is_real_and_scrubbed(): # --------------------------------------------------------------------------- -# The parser, asserted on VALUES rather than on coverage (§10) +# The parser, asserted on VALUES rather than on coverage # --------------------------------------------------------------------------- def _q(mode, **kw): @@ -254,7 +254,7 @@ def check_page_sizes_are_the_measured_ones(): def check_position_counts_emitted_rows_not_payload_slots(): - """§24: a record the parser drops must not shift every later position.""" + """A record the parser drops must not shift every later position.""" import product_parser as P payload = copy.deepcopy(FIXTURES["p2p_usdt_eur_buy"]) payload["data"].insert(1, {"adv": None, "advertiser": {}}) @@ -448,7 +448,7 @@ def check_the_waf_page_is_solvable_and_detected_fully(): def check_markers_do_not_match_a_page_the_scraping_browser_served(): - """§24: the Scraping Browser's auto-solve extension injects captcha + """The Scraping Browser's auto-solve extension injects captcha hunters into EVERY page, amazon_waf among them. The marker set must score zero against that injection WITHOUT any strip, or the strip is load-bearing and the next marker inherits the hole.""" @@ -479,7 +479,7 @@ def check_state_policy(): check("challenge: retried, solved, blocked", F.should_retry("challenge") and F.should_solve("challenge") and F.counts_as_blocked("challenge")) - check("throttled: retried, NOT blocked (§24)", + check("throttled: retried, NOT blocked", F.should_retry("throttled") and not F.counts_as_blocked("throttled")) check("restricted and blocked: never solved, there is no widget", not F.should_solve("restricted") and not F.should_solve("blocked")) @@ -523,7 +523,7 @@ def check_state_policy(): def check_policy_constants_have_a_consumer(): - """§17: a policy constant nothing reads is the same defect as dead code.""" + """A policy constant nothing reads is the same defect as dead code.""" src = open(os.path.join(HERE, "page_flow.py"), encoding="utf-8").read() for constant in ("RETRY_ON_BLOCKED", "BLOCK_RETRIES_WITHOUT_POOL", "SOLVES_PER_PAGE", "THROTTLE_RETRIES", "THROTTLE_WAIT_S", @@ -648,7 +648,7 @@ def check_the_shared_loop_end_to_end(): equal("...fetched twice", ops.fetches, [1, 1]) rc, meta, rows, ops = _run({1: [(429, "", None), (200, _p2p_page(1), None)]}, pages=1, retries=1) - equal("a throttle wait spends its OWN budget, not --retries (§24): " + equal("a throttle wait spends its OWN budget, not --retries: " "with --retries 1 the page still arrives", rc, 0) rc, meta, rows, ops = _run({1: [(202, "", "challenge"), (200, _p2p_page(1), None)]}, pages=1) @@ -959,7 +959,7 @@ def check_row_schema(): ROW_CLASS_BY_MODE, UNIQUE_BY_SKU_MODES) for cls in (P2PAd, LeadTrader, Announcement): names = [f.name for f in fields(cls)] - equal("%s: the family prefix is byte-identical and in order (§9)" % cls.__name__, + equal("%s: the family prefix is byte-identical and in order" % cls.__name__, names[:5], ["source", "scraped_at", "url", "sku", "title"]) check("%s: the run-describing tail is present" % cls.__name__, {"page", "position", "mode", "data_source"} <= set(names)) @@ -1113,7 +1113,7 @@ def check_exit_codes(): equal("3 blocked / 4 empty / 5 never obtained / 6 partial", (O.EXIT_BLOCKED, O.EXIT_NO_PRODUCTS, O.EXIT_FETCH_FAILED, O.EXIT_PARTIAL), (3, 4, 5, 6)) - check("end_of_listing is a COMPLETE stop reason (§24)", + check("end_of_listing is a COMPLETE stop reason", "end_of_listing" in O.COMPLETE_STOP_REASONS) check("api_rejected is NOT complete", "api_rejected" not in O.COMPLETE_STOP_REASONS) @@ -1178,7 +1178,7 @@ def check_sidecar_shape(): # --------------------------------------------------------------------------- -# The engines — the checks CLAUDE.md §17 says to steal +# The engines — checks that catch what import and --help cannot # --------------------------------------------------------------------------- def check_engines_import_their_driver_at_module_level(): @@ -1195,9 +1195,9 @@ def check_engines_import_their_driver_at_module_level(): def check_shared_calls_bind_against_the_real_signature(): - """§17's check #1. Every call from an engine (and the Scraper API client, + """Every call from an engine (and the Scraper API client, diff_runs and page_flow itself) into a shared module is bound against the - callee's real signature. A name that does not exist FAILS (§22). A name + callee's real signature. A name that does not exist FAILS. A name bound in the calling file shadows a same-named module.""" import captcha_solver import output_writer @@ -1272,8 +1272,7 @@ def _argparse_flags(module_name): return flags -# The family's flag contract (CLAUDE.md §9, including the five it omitted -# for months), plus this repo's own query flags. +# The family's flag contract, plus this repo's own query flags. CONTRACT_FLAGS = { "--url", "--pages", "--category", "--format", "--out", "--delay", "--retries", "--retry-delay", "--concurrency", "--proxy", "--proxy-file", @@ -1287,7 +1286,7 @@ def _argparse_flags(module_name): def check_engine_flag_sets(): - """§17's check #2: against the contract AND against each other, both + """Against the contract AND against each other, both ways. The exception list IS the documentation.""" sets = {m: _argparse_flags(m) for m in ENGINES} for module, flags in sets.items(): @@ -1307,7 +1306,7 @@ def check_engine_flag_sets(): # The flags that describe a browser, and so are absent from the HTTP engine. -# The list IS the documentation (§17): the check fails if the HTTP engine +# The list IS the documentation: the check fails if the HTTP engine # gains one, loses a shared one, or if one of these leaves the browser # engines while still listed here. BROWSER_ONLY_FLAGS = {"--headless", "--headful", "--cdp-endpoint", "--locale", @@ -1340,7 +1339,7 @@ def check_banned_and_removed_flags(): def check_undefined_names_in_every_module(): - """§10: compileall proves a file PARSES, not that its names RESOLVE. + """compileall proves a file PARSES, not that its names RESOLVE. Kept COARSE (pooled bindings) so it under-reports rather than invents.""" import builtins for filename in sorted(f for f in os.listdir(HERE) if f.endswith(".py")): @@ -1456,7 +1455,7 @@ def check_credential_scan_is_one_implementation_invoked_from_both(): script = os.path.join(HERE, ".github", "ci_checks.py") if not os.path.isdir(os.path.join(HERE, ".github")): # Inside the Docker image, which copies no .github at all. Triggered - # by the WHOLE directory being absent, never by one file in it (§22). + # by the WHOLE directory being absent, never by one file in it. skip("ci_checks", "no .github directory (the image)") return check("the credential scan exists as a script", os.path.exists(script)) @@ -1505,7 +1504,7 @@ def check_the_hex_exemption_is_one_context_only(): # Assembled from pieces, so this file can be scanned like every other rather -# than exempted (§22: the file most likely to acquire a stray phrase is the +# than exempted (the file most likely to acquire a stray phrase is the # one a wholesale exemption never reads). BANNED_WORDING = ( "cloud" + " browser", "anti" + "detect browser", "2scraper " + "Anti" + "detect Browser", @@ -1514,7 +1513,7 @@ def check_the_hex_exemption_is_one_context_only(): def check_banned_wording(): - """§12, enforced by this test rather than by review.""" + """Banned product wording, enforced by this test rather than by review.""" scanned = 0 for root, dirs, files in os.walk(HERE): dirs[:] = [d for d in dirs if d not in (".git", "__pycache__", ".pytest_cache", @@ -1533,8 +1532,45 @@ def check_banned_wording(): check("the banned-wording scan read the repo (%d files)" % scanned, scanned > 20) +# References to private notes this public repo cannot link to: a reader +# can open none of them, so a rule is stated in this repo's own words instead. +# Assembled from pieces so this file is scanned like every other. +PRIVATE_CITATIONS = ( + "CLAUDE" + ".md", "claude_" + "2scraper", "2scraper_" + "repo_rules", + "play" + "book", +) +SECTION_SIGN_REF = re.compile("\u00a7" + r"\s*\d") + + +def check_no_private_citations_in_shipped_files(): + scanned = 0 + for root, dirs, files in os.walk(HERE): + dirs[:] = [d for d in dirs + if d not in (".git", "__pycache__", ".pytest_cache", "live", + "captures", ".claude") + and not os.path.exists(os.path.join(root, d, "pyvenv.cfg"))] + for filename in files: + if not filename.endswith((".py", ".md", ".yml", ".yaml", ".txt", + ".toml", ".cfg", ".example", ".json", + ".gitignore", ".dockerignore")) \ + and filename != "Dockerfile": + continue + path = os.path.join(root, filename) + text = open(path, encoding="utf-8", errors="replace").read() + scanned += 1 + rel = os.path.relpath(path, HERE) + for token in PRIVATE_CITATIONS: + if token.lower() in text.lower(): + check("%s cites no private note (%r)" % (rel, token), False) + m = SECTION_SIGN_REF.search(text) + if m: + line = text.count("\n", 0, m.start()) + 1 + check("%s:%d carries no bare section-number reference" % (rel, line), False) + check("the private-citation scan read the repo (%d files)" % scanned, scanned > 20) + + def check_concurrency_with_the_browser_stubbed(): - """§10: a live run cannot always reach this machinery. Driven through the + """A live run cannot always reach this machinery. Driven through the SHARED worker loop with the fetch replaced.""" import page_flow import queue as queue_mod @@ -1641,7 +1677,7 @@ def check_fingerprint_kwargs_are_ones_the_driver_accepts(): def check_engines_do_not_evaluate_a_string_in_the_browser(): - """§18: wait_for_function evaluates a string, which a CSP without + """`wait_for_function` evaluates a string, which a CSP without unsafe-eval kills. page.evaluate with a real function is fine.""" for module in ENGINES: tree = ast.parse(open(os.path.join(HERE, module + ".py"), encoding="utf-8").read()) @@ -1785,7 +1821,7 @@ def check_x_debug_header_is_redacted(): def check_captcha_capability_claims_match_the_code(): - """§19: the most expensive bug this family can ship is a SENTENCE.""" + """The most expensive bug this family can ship is a SENTENCE.""" readme = open(os.path.join(HERE, "README.md"), encoding="utf-8").read() solver = open(os.path.join(HERE, "captcha_solver.py"), encoding="utf-8").read() low = readme.lower() @@ -1798,7 +1834,7 @@ def check_captcha_capability_claims_match_the_code(): def check_readme_numbers_are_not_stale(): - """§17's check #4: a column count claimed in the README is a class's.""" + """A column count claimed in the README is a class's.""" readme = open(os.path.join(HERE, "README.md"), encoding="utf-8").read() from output_writer import ROW_CLASS_BY_MODE sizes = {len(fields(c)) for c in ROW_CLASS_BY_MODE.values()} From e78ddeb5f1490f60afebcffe6e82e96232b7016b Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Wed, 14 Oct 2026 18:57:06 +0000 Subject: [PATCH 3/3] Changelog: solve allowance, atomic writes, CSV escaping, env URL default Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5fa9a75..1f49a0c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,9 +15,63 @@ said so at the top of its release notes. > exit 3 or 4 with a 39-byte "capture", in tens of milliseconds. If a > script of yours passes 0, pass **1**: the flag counts ATTEMPTS, so 1 means > "try once and do not retry". +> +> **Behaviour change:** a page now buys at most one CAPTCHA solve in total, +> across every block-retry and exit rotation. It used to be one per +> LANDING, so a refused page could buy 2 AWS WAF solves with no proxy pool +> and 4 with a 3-exit pool. A page whose CAPTCHA survives that one solve is +> reported blocked (exit 3) sooner and for less. +> +> **Behaviour change:** CSV cells that start with `=`, `+`, `-`, `@`, a +> tab or a line break now carry a leading apostrophe, so a spreadsheet +> reads them as text. JSON is unchanged; the sidecar's `csv_cells_escaped` +> says how many cells differ. +> +> **Behaviour change:** `BINANCE_URL` from `.env` or the environment is a +> default only. A `--mode` or a query flag typed on the command line now +> wins over it instead of being refused as a conflict. `.env.example` +> ships it commented out. ### Fixed +- **One refused page could buy several CAPTCHA solves.** The allowance + lived inside each landing, and every block-retry or rotation landed + again. With the solver stubbed: 2 solves with no pool, 4 with a 3-exit + pool, against `SOLVES_PER_PAGE = 1`. The allowance is now one + `page_flow.SolveBudget` per page, checked before each purchase and never + reset by a retry or a rotation. The `--pay-type` pre-check no longer + solves at all. A suite check drives the shared loop with a solver that + always "succeeds" and counts purchases. +- **A failed write destroyed the previous good output.** `out.json`, + `out.csv` and `out.meta.json` were opened with truncation, so a failure + inside `json.dump` left 13 bytes of invalid JSON where a good file had + been. Each is now written to a temporary file in the same directory, + fsynced and renamed over the target. A new file gets the mode `open()` + would have given it (`0o666 & ~umask`), and an existing file keeps its + own. Each file is replaced atomically on its own; the three are not one + transaction. +- **CSV cells could run as spreadsheet formulas.** Advertiser nicknames, + portfolio names and announcement titles are third-party text. See the + behaviour change above. +- **A copied `.env.example` broke every flag-built query.** It set + `BINANCE_URL` to a P2P page, so `--mode announcements`, + `--mode copytrading` and `--asset/--fiat` all exited 2 with a conflict + against a URL nobody typed. The engines now record the typed `--url` + before `.env` is read. +- **Advice for HTTP 451 named a country nobody had measured.** The exit-3 + message, README, TROUBLESHOOTING, CONTRIBUTING and the bug template said + a US exit is refused. 451 has not been observed on binance.com, and the + daily canary is served from a GitHub-hosted runner with no proxy. The + message now says so and asks for the exit's country in an issue. +- **A Scraping Browser 401 was explained as credentials that "last about + a day".** That figure was not measured (an endpoint has been seen working + after ten days). A 401 is now reported as refused credentials, expired or + mistyped. +- A capture saved at the repo root (`--dump-html dump.html`) is ignored by + git, and `.dockerignore` excludes every `.env*` file. +- Comments that pointed at private notes now state their reasons; a suite + check keeps it that way. + - **`--retries 0` fetched nothing and called it a result.** The load loop is `range(1, retries + 1)`, so 0 never runs it and the engine judged `about:blank`. Three third-party audits of this family ran exactly that and