From e4c1d4975a3db87464500e6360389f63d739dd08 Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Fri, 9 Oct 2026 10:42:28 +0000 Subject: [PATCH 1/2] Refuse --retries 0, unify the not-served message, fix donor --help text --retries 0 made the load loop run zero times, so goto was never called and the engine judged about:blank (39 bytes) as a block: exit 3 for a site that was never asked. Now a usage error in all three engines. The not-served log blamed headless and recommended a residential exit in the same sentence, and said so for headful runs too. One shared page_flow.not_served_message states the observation first and the cause only when it applies. --help and usage errors described another marketplace scraper (product mode, Apollo, IDR, /p/). Now from one set of strings in page_flow, with a suite check over every string in the engines. Co-Authored-By: Claude Sonnet 5.5 --- CHANGELOG.md | 27 ++++++++++ SECURITY.md | 5 +- page_flow.py | 95 ++++++++++++++++++++++++++++++++++ playwright_scraper.py | 117 +++++++++--------------------------------- puppeteer_scraper.py | 63 ++++++----------------- selenium_scraper.py | 65 ++++++----------------- smoke_test.py | 72 ++++++++++++++++++++++++++ 7 files changed, 257 insertions(+), 187 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a41f51..6aef7cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,33 @@ or an empty column. ### Fixed +- **`--retries 0` skipped navigation and reported a block.** The load loop + is `range(1, retries + 1)`, so 0 ran it never, `goto` was not called, and + the engine judged `about:blank`: a 39-byte empty DOM, exit 3, for a site that + was never asked. Reproduced 2026-10-09 (Playwright, `--retries 0`: exit 3 in + ~50 ms in both modes; the same command with `--retries 1` fetched 72 lots, + `complete`, exit 0). `--retries` now takes an integer of at least 1 in all + three engines and refuses 0 as a usage error (exit 2). +- **The not-served message contradicted the README.** It said a headless + browser was the cause, then recommended a residential exit, and said it for a + headful run too. One shared `page_flow.not_served_message` now states what + was observed (bytes, whether the site's own assets appear) and only then the + likely cause: a document under 100 bytes with no site assets is called an + empty document and a navigation that did not complete; headless is blamed + only when the run was headless. +- **`--help` and usage errors described another marketplace scraper** (a + `product` mode, `/search?st=product`, `/p/` URLs, an Apollo cache, IDR + prices, an infinitely scrolling grid). They now describe the real URL kinds + and the `listing|lot|auctions` modes, from one set of strings in `page_flow`. + The `--mode lot` usage errors no longer point at `--mode product`, and the + discovery-hub warning, which could never fire, is removed. A suite check walks + every string in the three engines for the old wording. + +### Changed + +- `SECURITY.md` no longer says a raw HTML dump contains session cookies; it + holds the page markup, not the cookie jar. + - **The Scraper API path (`scraper_api_client.py`) failed whenever a wait flag was given, and never saw the target's status.** Measured 2026-09-23 against the live `/tasks/sync` endpoint: `waitFor` sent as a JSON-encoded string diff --git a/SECURITY.md b/SECURITY.md index 914fd48..9258531 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -100,7 +100,10 @@ Three places leak credentials that people do not expect, because unlike our own log lines they are **not** masked: - **raw HTML dumps** (`--dump-html`, and the automatic dump on a zero-product - run) — these can contain session cookies + run) — the page's markup as the browser serialised it, not the cookie jar, + but a site can write session identifiers or CSRF tokens into inline scripts, + so treat a dump as sensitive. A blocked page's dump is a few hundred bytes + and carries none - **the Scraper API's `x-debug` response header** - **your shell history**, if you passed a key on the command line diff --git a/page_flow.py b/page_flow.py index 88fe4df..ea3f603 100644 --- a/page_flow.py +++ b/page_flow.py @@ -41,6 +41,7 @@ from __future__ import annotations +import argparse import logging from typing import Callable, Dict, Iterable, List, Optional, Union from urllib.parse import urljoin, urlsplit @@ -54,6 +55,11 @@ logger = logging.getLogger("page_flow") +# Below this a document with no reference to the site's assets is an empty +# DOM (a bare is 39 bytes), not the +# refusal page, which is 300+ bytes. +EMPTY_DOM_BYTES = 100 + # --------------------------------------------------------------------------- # Readiness @@ -292,6 +298,95 @@ def block_advice(html: Optional[str], headless: bool, f"the path measured to work.") +def not_served_message(html: Optional[str], headless: bool, has_pool: bool, + tries: int, debug_html: str) -> str: + """The log line for a page the site did not serve (exit 3). + + One text for all three engines, built from the same facts as + `block_advice`. It states what was OBSERVED -- the byte count and whether + the document references the site's own assets -- and only then what that + usually means here. A document of a few dozen bytes is an EMPTY DOM, not + Akamai's refusal (a 394-byte "Access Denied"), and no HTTP status + reaches this line, so it must not claim the site refused anything. + """ + n = len(html or "") + served = served_by_catawiki(html or "") + seen = (f"The site did not serve this request: {n} bytes, " + f"{'which references' if served else 'with no reference to'} " + f"the site's own asset host, saved to {debug_html}.") + if n < EMPTY_DOM_BYTES and not served: + what = (f" That is an empty document, not a refusal page (Akamai's " + f"is a 394-byte \"Access Denied\"), so the navigation most " + f"likely never completed: check the network and that " + f"--retries is at least 1.") + if headless: + what += (" This run was HEADLESS, which this site refuses " + "whatever the address; --headful is the default.") + elif headless: + what = (" This run was HEADLESS, and Akamai refuses a headless " + "browser here regardless of the exit address. Run " + "--headful, or use --cdp-endpoint.") + else: + what = (" This run was headful, so the cause is not the headless " + "flag; the exit address may be scored. " + + ("Rotation is unlikely to help here; " if has_pool else "") + + "--cdp-endpoint is the path measured to work.") + return (f"{seen}{what} Tried {tries} time(s). This is exit 3, distinct " + f"from a genuinely empty result (exit 4).") + + +def attempts(value: str) -> int: + """argparse type for --retries: attempts per page load, so at least 1. + + The loop is `range(1, retries + 1)`. Zero ran it NEVER, `goto` was not + called, and the engine judged about:blank -- an empty DOM reported as + "blocked" (exit 3) for a site that was never asked. + """ + try: + n = int(value) + except ValueError: + raise argparse.ArgumentTypeError(f"{value!r} is not an integer") + if n < 1: + raise argparse.ArgumentTypeError( + "must be at least 1: it counts attempts per page load, and 0 " + "would attempt none and judge a page that was never fetched") + return n + + +def mode_url_mismatch(mode: str, url: str, kind: str) -> Optional[str]: + """The usage error for a --mode that cannot read this URL, or None.""" + if mode == "lot" and kind != "lot": + return (f"--mode lot expects a /{{loc}}/l/{{id}}-{{slug}} lot URL; " + f"{url!r} is a {kind or 'unrecognised'} page.") + if mode == "listing" and kind == "lot": + return (f"{url!r} is a single lot page. Use --mode lot for it, or " + f"pass a category (/{{loc}}/c/{{id}}-{{slug}}), a search " + f"(/{{loc}}/s?q=...) or an auction (/{{loc}}/a/{{id}}-{{slug}}) " + f"URL.") + return None + + +URL_HELP = ("A Catawiki URL: a category /{loc}/c/{id}-{slug}, a search " + "/{loc}/s?q=..., an auction /{loc}/a/{id}-{slug}, the auctions " + "index /{loc}/a (--mode auctions), or one lot " + "/{loc}/l/{id}-{slug} (--mode lot). Required, unless " + "CATAWIKI_URL is set in the environment or in .env.") +MODE_HELP = ("listing (default): a category, a search or a whole auction, " + "24 lots per page on the first two. lot: one lot page, adding " + "the estimate, the seller and their feedback score, the " + "specifications and the bid history. auctions: the auctions " + "index as a work list, one row per auction, to feed back in " + "as --url.") +PAGES_HELP = ("Listing pages to crawl (--mode listing on a category or a " + "search; ?page=N, capped by the site at 100). Ignored in " + "--mode lot and --mode auctions, and for a whole auction, " + "which is one page.") +RETRIES_HELP = ("Attempts per page load before giving up (default 3, " + "minimum 1). The pause between attempts doubles each time. " + "A page that comes back EMPTY is not retried: an empty " + "listing is a correct answer, not a fault.") + + # --------------------------------------------------------------------------- # Pagination # --------------------------------------------------------------------------- diff --git a/playwright_scraper.py b/playwright_scraper.py index 0de95dd..7eb09e6 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -921,24 +921,10 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome with open(debug_html, "w", encoding="utf-8") as f: f.write(html or "") served = served_by_catawiki(html or "") - logger.error( - - "The site did not serve this request — %d bytes, %s the site's " - "own asset host, saved to %s. On this site a refusal is a 394-byte " - "\"Access Denied\" with no vendor name and nothing to solve, and " - "it is triggered by the browser being headless rather than by the " - "address: no interstitial, no " - "vendor marker, so there is no challenge to solve and no key " - "would help. What clears it, measured 2026-09-10: a residential " - "exit. The country does not matter — an Indonesian and a US " - "residential exit returned identical pages — but a datacentre " - "address gets nothing. Use --cdp-endpoint, or a residential " - "--proxy. This is exit 3, distinct from a genuinely empty result " - "(exit 4).%s", - len(html or ""), "which references" if served else "with no " - "reference to", debug_html, - (f" Tried {block_retries + 1} exit(s)." if has_pool - else f" Re-fetched {block_retries + 1} time(s).")) + logger.error("%s", page_flow.not_served_message( + html, headless=bool(getattr(args, "headless", False)), + has_pool=has_pool, tries=block_retries + 1, + debug_html=debug_html)) outcome.blocked_by = "no-response" if not html else "not-served" outcome.final_url = session.page.url return outcome @@ -974,14 +960,13 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome # page's JSON-LD and not out of the buy box. if args.mode == "lot": logger.info("The product name did not paint within %.0fs. " - "That is not fatal: a detail row is read from the " - "page's own Apollo cache and its price meta, both " - "of which are in the first response, so the parse " - "below decides.", content_timeout / 1000) + "That is not fatal: a lot row is read from the " + "page's own __NEXT_DATA__ payload, which is in the first " + "response, so the parse below decides.", content_timeout / 1000) else: logger.info("No listing tiles appeared within %.0fs. If this " - "URL is a /p/ discovery hub or one page " - "past the end of a listing, that is the expected " + "URL is one page past the end of a listing or a search " + "with no matches, that is the expected " "answer and the run will report 0 rows (exit 4).", content_timeout / 1000) @@ -1253,7 +1238,7 @@ def scrape(args) -> int: concurrency = max(1, args.concurrency) if concurrency > 1: if args.mode == "lot": - logger.info("--concurrency is ignored in --mode product: there is " + logger.info("--concurrency is ignored in --mode lot: there is " "one page to fetch.") concurrency = 1 elif args.cdp_endpoint: @@ -1507,46 +1492,15 @@ def scrape(args) -> int: def parse_args(): p = argparse.ArgumentParser( description="Catawiki auction scraper (Playwright edition)") - p.add_argument("--url", default=None, - help="A Catawiki URL: a category listing " - "(/search?st=product&q=...), a category listing " - "(/p//[/]) or one product " - "(/{shop}/{slug}) with --mode product. One " - "storefront, one language, one currency — there is " - "no locale prefix and no per-country host. " - "Required, unless CATAWIKI_URL is set in the environment " - "or in .env.") + p.add_argument("--url", default=None, help=page_flow.URL_HELP) p.add_argument("--mode", choices=["listing", "lot", "auctions"], - default="listing", - help="listing (default): a search grid " - "(/search?st=product&q=...) or a category listing " - "(/p///) — 60 products per batch. " - "product: one /{shop}/{slug} page, read from the " - "page's own Apollo cache, which adds the site's real " - "product id, the EXACT sold count, the review count, " - "the condition, the shipping weight and the " - "category's own listing URL. --pages applies to " - "listing only; there is one page to read in product " - "mode. There is deliberately no shop mode: a " - "shop front is a different application " - "shell whose markup has not been measured, and a mode " - "that ships untested is worse than one that is " - "absent.") + default="listing", help=page_flow.MODE_HELP) p.add_argument("--category", default=None, help="Label to tag output rows with. Defaults to the " - "category leaf from a /c/ URL, the search query from a " - "/search URL, and the breadcrumb in --mode product, so " - "the column is rarely empty just because the flag was " - "omitted. A shop front spans categories and gets none, " - "so pass one if you want that column filled.") - p.add_argument("--pages", type=int, default=1, - help="Number of listing pages to crawl. Applies to " - "--mode listing; ignored in --mode product, which " - "has one page. Note that only a CATEGORY URL has " - "per-page addresses: a search is one infinitely " - "scrolling page, so --pages above 1 there is honoured " - "by scrolling further rather than by fetching more " - "URLs.") + "category leaf from a /c/ URL and the search query " + "from a /s URL, so the column is rarely empty just " + "because the flag was omitted.") + p.add_argument("--pages", type=int, default=1, help=page_flow.PAGES_HELP) p.add_argument("--delay", type=float, default=2.0, help="Delay between pages, seconds") p.add_argument("--concurrency", type=int, default=1, metavar="N", help="Fetch pages through N parallel workers (default 1 — " @@ -1554,23 +1508,18 @@ def parse_args(): "own browser and holds its own proxy exit, so N>1 " "without --proxy-file just sends N times the traffic " "from one address. Ignored with --cdp-endpoint.") - p.add_argument("--retries", type=int, default=3, - help="Attempts per page load before giving up (default 3). " - "The pause between attempts doubles each time. A page " - "that comes back EMPTY is not retried — see " - "page_flow.STATE_POLICY — because an empty hub " - "category is a correct answer, not a fault.") + p.add_argument("--retries", type=page_flow.attempts, default=3, + help=page_flow.RETRIES_HELP) p.add_argument("--retry-delay", type=float, default=2.0, help="Seconds before the first page-load retry, doubling " "thereafter (default 2.0)") p.add_argument("--format", choices=["json", "csv", "both"], default="both") p.add_argument("--out", default="catawiki_products", help="Output file prefix") p.add_argument("--locale", default="id-ID", - help="Browser locale (default id-ID, the site's only " - "one). It does NOT decide the language or the " - "currency: this site serves euro on all 18 locales and " - "IDR prices to an Indonesian and a US exit alike, so " - "this only affects what the browser claims about " + help="Browser locale (default id-ID). It does NOT decide the " + "language or the currency: the language is the URL's " + "/{loc}/ segment and prices are euro on every locale, " + "so this only affects what the browser claims about " "itself.") p.add_argument("--proxy", default=None, help="Proxy URL, e.g. http://ACCOUNT:PASSWORD@HOST:9999 " @@ -1688,25 +1637,9 @@ def parse_args(): f"storefront — one language, one currency, no per-country " f"hostname and no locale prefix.") kind = listing_kind(args.url) - if args.mode == "lot" and kind != "lot": - p.error(f"--mode product expects a /{{shop}}/{{slug}} product URL; " - f"{args.url!r} is a {kind} page.") - if args.mode == "listing" and kind == "lot": - p.error(f"{args.url!r} is a single product page. Use --mode product " - f"for it, or pass a /search?st=product&q=... or " - f"/p/// URL.") - if args.mode == "listing" and kind == "hub": - # Said as a warning rather than an error: it IS a Catawiki URL and - # the run will honestly report zero rows (exit 4). But a reader who - # tries the obvious category URL first would otherwise conclude the - # tool is broken, so name what happened. - logger.warning( - "%s is a /p/ DISCOVERY HUB, not a listing. It has no " - "product grid on it — only banners and recommendation carousels " - "— so this run will return 0 rows and exit 4. The real listings " - "are one or two levels down: /p//[/], e.g. " - "/p/makanan-minuman/minuman/kopi-bubuk. A product page's own " - "breadcrumb names its category's listing URL.", args.url) + mismatch = page_flow.mode_url_mismatch(args.mode, args.url, kind) + if mismatch: + p.error(mismatch) if args.mode in ("lot", "auctions") and args.pages != 1: # Said out loud rather than silently ignored: a user who passed # --pages 5 expects five pages of something. diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index 5710c10..cae3094 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -637,16 +637,10 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome debug_html = f"{args.out}_page{page_num}_debug.html" with open(debug_html, "w", encoding="utf-8") as f: f.write(html or "") - logger.error( - "The site did not serve this request — %d bytes, %s the site's " - "own asset host, saved to %s. There is no challenge to solve. " - "What clears it, measured 2026-09-10: a HEADFUL browser, and the " - "exit address does not matter (four residential exits and one " - "returned identical pages) — a datacentre address gets nothing. " - "This is exit 3, distinct from a genuinely empty result (exit 4).", - len(html or ""), - "which references" if served_by_catawiki(html or "") - else "with no reference to", debug_html) + logger.error("%s", page_flow.not_served_message( + html, headless=bool(getattr(args, "headless", False)), + has_pool=has_pool, tries=block_retries + 1, + debug_html=debug_html)) outcome.blocked_by = "no-response" if not html else "not-served" outcome.final_url = d["current_url"]() return outcome @@ -679,13 +673,13 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome # page's JSON-LD and not out of the buy box. if args.mode == "lot": logger.info("The product name did not paint in time. That is " - "not fatal: a detail row is read from the page's " - "own Apollo cache and its price meta, both in the " - "first response, so the parse below decides.") + "not fatal: a lot row is read from the page's own " + "__NEXT_DATA__ payload, which is in the first " + "response, so the parse below decides.") else: logger.info("No listing tiles appeared in time. If this URL " - "is a /p/ discovery hub or one page past " - "the end of a listing, that is the expected " + "is one page past the end of a listing or a search " + "with no matches, that is the expected " "answer and the run will report 0 rows (exit 4).") html = d["content"]() or html @@ -1025,30 +1019,17 @@ def parse_args(): description="Catawiki auction scraper (pyppeteer edition). pyppeteer is " "effectively unmaintained — playwright_scraper.py is the " "primary engine.") - p.add_argument("--url", default=None, - help="A Catawiki URL: /{loc}/c/{id}-{slug}, /{loc}/s?q=..., " - "/p///, or /{shop}/{slug} with " - "--mode product. Required, unless CATAWIKI_URL is " - "set in the environment or in .env.") + p.add_argument("--url", default=None, help=page_flow.URL_HELP) p.add_argument("--mode", choices=["listing", "lot", "auctions"], - default="listing", - help="listing (default) or product. product reads one " - "/{shop}/{slug} page out of its own Apollo cache and " - "adds the site's real product id, the EXACT sold " - "count, the review count, the condition and the " - "shipping weight — the columns a listing row cannot " - "carry. No --pages in product mode. There is " - "deliberately no shop mode; see playwright_scraper.") + default="listing", help=page_flow.MODE_HELP) p.add_argument("--category", default=None, help="Label to tag output rows with.") - p.add_argument("--pages", type=int, default=1, help="Listing pages to crawl") + p.add_argument("--pages", type=int, default=1, help=page_flow.PAGES_HELP) p.add_argument("--delay", type=float, default=2.0, help="Delay between pages, seconds") p.add_argument("--concurrency", type=int, default=1, metavar="N", help="Accepted for flag parity and IGNORED here: parallel " "page fetching lives in playwright_scraper.py.") - p.add_argument("--retries", type=int, default=3, - help="Attempts per page load before giving up (default 3). " - "A page that comes back EMPTY is not retried: an empty " - "hub category is a correct answer, not a fault.") + p.add_argument("--retries", type=page_flow.attempts, default=3, + help=page_flow.RETRIES_HELP) p.add_argument("--retry-delay", type=float, default=2.0, help="Seconds before the first retry, doubling thereafter") p.add_argument("--format", choices=["json", "csv", "both"], default="both") @@ -1131,19 +1112,9 @@ def parse_args(): f"storefront — one language, one currency, no per-country " f"hostname and no locale prefix.") kind = listing_kind(args.url) - if args.mode == "lot" and kind != "lot": - p.error(f"--mode product expects a /{{shop}}/{{slug}} product URL; " - f"{args.url!r} is a {kind} page.") - if args.mode == "listing" and kind == "lot": - p.error(f"{args.url!r} is a single product page. Use --mode product " - f"for it, or pass a /search?st=product&q=... or " - f"/p/// URL.") - if args.mode == "listing" and kind == "hub": - logger.warning( - "%s is a /p/ DISCOVERY HUB, not a listing — no product " - "grid, only banners and recommendation carousels — so this run " - "will return 0 rows and exit 4. The real listings are one or two " - "levels down: /p//[/].", args.url) + mismatch = page_flow.mode_url_mismatch(args.mode, args.url, kind) + if mismatch: + p.error(mismatch) return args diff --git a/selenium_scraper.py b/selenium_scraper.py index 56e49a8..778f8a7 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -605,18 +605,10 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome debug_html = f"{args.out}_page{page_num}_debug.html" with open(debug_html, "w", encoding="utf-8") as f: f.write(html or "") - logger.error( - "The site did not serve this request — %d bytes, %s the site's " - "own asset host, saved to %s. There is no challenge to solve. " - "What clears it, measured 2026-09-10: a HEADFUL browser, and the " - "exit address does not matter (four residential exits and one " - "returned identical pages) — a datacentre address gets nothing. " - "Note this engine cannot use an authenticated remote CDP endpoint " - "or an authenticated proxy; see the README's engine limits. This " - "is exit 3, distinct from a genuinely empty result (exit 4).", - len(html or ""), - "which references" if served_by_catawiki(html or "") - else "with no reference to", debug_html) + logger.error("%s", page_flow.not_served_message( + html, headless=bool(getattr(args, "headless", False)), + has_pool=has_pool, tries=block_retries + 1, + debug_html=debug_html)) outcome.blocked_by = "no-response" if not html else "not-served" outcome.final_url = d["current_url"]() return outcome @@ -650,14 +642,14 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome # page's JSON-LD and not out of the buy box. if args.mode == "lot": logger.info("The product name did not paint within %.0fs. " - "That is not fatal: a detail row is read from the " - "page's own Apollo cache and its price meta, both " - "in the first response, so the parse below " + "That is not fatal: a lot row is read from the " + "page's own __NEXT_DATA__ payload, which is in the first " + "response, so the parse below " "decides.", timeout_s) else: logger.info("No listing tiles appeared within %.0fs. If this " - "URL is a /p/ discovery hub or one page " - "past the end of a listing, that is the expected " + "URL is one page past the end of a listing or a search " + "with no matches, that is the expected " "answer and the run will report 0 rows (exit 4).", timeout_s) @@ -985,30 +977,17 @@ def parse_args(): description="Catawiki auction scraper (Selenium edition). Cannot authenticate a " "proxy or a remote CDP endpoint — see the module " "docstring; playwright_scraper.py is the primary engine.") - p.add_argument("--url", default=None, - help="A Catawiki URL: /{loc}/c/{id}-{slug}, /{loc}/s?q=..., " - "/p///, or /{shop}/{slug} with " - "--mode product. Required, unless CATAWIKI_URL is " - "set in the environment or in .env.") + p.add_argument("--url", default=None, help=page_flow.URL_HELP) p.add_argument("--mode", choices=["listing", "lot", "auctions"], - default="listing", - help="listing (default) or product. product reads one " - "/{shop}/{slug} page out of its own Apollo cache and " - "adds the site's real product id, the EXACT sold " - "count, the review count, the condition and the " - "shipping weight — the columns a listing row cannot " - "carry. No --pages in product mode. There is " - "deliberately no shop mode; see playwright_scraper.") + default="listing", help=page_flow.MODE_HELP) p.add_argument("--category", default=None, help="Label to tag output rows with.") - p.add_argument("--pages", type=int, default=1, help="Listing pages to crawl") + p.add_argument("--pages", type=int, default=1, help=page_flow.PAGES_HELP) p.add_argument("--delay", type=float, default=2.0, help="Delay between pages, seconds") p.add_argument("--concurrency", type=int, default=1, metavar="N", help="Accepted for flag parity and IGNORED here: parallel " "page fetching lives in playwright_scraper.py.") - p.add_argument("--retries", type=int, default=3, - help="Attempts per page load before giving up (default 3). " - "A page that comes back EMPTY is not retried: an empty " - "hub category is a correct answer, not a fault.") + p.add_argument("--retries", type=page_flow.attempts, default=3, + help=page_flow.RETRIES_HELP) p.add_argument("--retry-delay", type=float, default=2.0, help="Seconds before the first retry, doubling thereafter") p.add_argument("--format", choices=["json", "csv", "both"], default="both") @@ -1101,19 +1080,9 @@ def parse_args(): f"storefront — one language, one currency, no per-country " f"hostname and no locale prefix.") kind = listing_kind(args.url) - if args.mode == "lot" and kind != "lot": - p.error(f"--mode product expects a /{{shop}}/{{slug}} product URL; " - f"{args.url!r} is a {kind} page.") - if args.mode == "listing" and kind == "lot": - p.error(f"{args.url!r} is a single product page. Use --mode product " - f"for it, or pass a /search?st=product&q=... or " - f"/p/// URL.") - if args.mode == "listing" and kind == "hub": - logger.warning( - "%s is a /p/ DISCOVERY HUB, not a listing — no product " - "grid, only banners and recommendation carousels — so this run " - "will return 0 rows and exit 4. The real listings are one or two " - "levels down: /p//[/].", args.url) + mismatch = page_flow.mode_url_mismatch(args.mode, args.url, kind) + if mismatch: + p.error(mismatch) return args diff --git a/smoke_test.py b/smoke_test.py index 39bb2a6..9f68dbd 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -731,6 +731,78 @@ def test_page_flow(): ok &= check("and the block-retry budgets are 0", page_flow.BLOCK_RETRIES_WITHOUT_POOL == 0 and page_flow.BLOCK_RETRIES_WITH_POOL == 0) + # --retries 0 used to skip navigation entirely (range(1, 0 + 1) is empty), + # so the engine judged about:blank: a 39-byte empty DOM reported as + # "blocked", exit 3, for a site that was never asked. Found by an audit + # whose own live run used --retries 0 and read that as a block. + import argparse as _ap + ok &= check("--retries accepts 1 and 3", + page_flow.attempts("1") == 1 and page_flow.attempts("3") == 3) + for bad in ("0", "-1", "x"): + try: + page_flow.attempts(bad) + raised = False + except _ap.ArgumentTypeError: + raised = True + ok &= check("--retries %s is refused as a usage error" % bad, raised) + for eng in ("playwright", "selenium", "puppeteer"): + src = open(os.path.join(os.path.dirname(os.path.abspath(__file__)), + eng + "_scraper.py"), encoding="utf-8").read() + ok &= check("%s: --retries goes through page_flow.attempts" % eng, + '"--retries", type=page_flow.attempts' in src) + ok &= check("%s: the not-served log is the shared message" % eng, + "page_flow.not_served_message(" in src) + + # An empty DOM is not Akamai's refusal, and the message must not say so. + empty = "" + m = page_flow.not_served_message(empty, headless=False, has_pool=False, + tries=1, debug_html="x.html") + ok &= check("a headful empty DOM is called a navigation that did not " + "complete, not a refusal, and does not blame headless", + "empty document" in m and "--retries" in m + and "HEADLESS" not in m and "proxy" not in m.lower()) + m = page_flow.not_served_message(empty, headless=True, has_pool=False, + tries=1, debug_html="x.html") + ok &= check("a headless empty DOM also names headless", "HEADLESS" in m) + m = page_flow.not_served_message(BLOCKED_403, headless=False, + has_pool=False, tries=1, + debug_html="x.html") + ok &= check("a headful refusal page is not blamed on headless", + "headful" in m and "HEADLESS" not in m + and "empty document" not in m) + + ok &= check("--mode lot on a category URL is a usage error naming lot", + "--mode lot" in (page_flow.mode_url_mismatch( + "lot", "https://www.catawiki.com/en/c/333-watches", + "category") or "")) + ok &= check("a lot URL in listing mode points at --mode lot, not product", + "Use --mode lot" in (page_flow.mode_url_mismatch( + "listing", "https://www.catawiki.com/en/l/1-x", "lot") + or "")) + ok &= check("a matching mode and URL is accepted", + page_flow.mode_url_mismatch("listing", "u", "category") is None) + + # The user-facing text of all three engines must describe THIS site. It + # was inherited from a different marketplace scraper (a product mode, an + # Apollo cache, IDR, /p/ URLs), and a reader following --help sent + # the wrong URL kind and the wrong mode. + import ast as _ast + stale = ("st=product", "--mode product", "Apollo", "/p/<", "IDR ", + "discovery hub", "/{shop}") + for eng in ("playwright", "selenium", "puppeteer"): + path = os.path.join(os.path.dirname(os.path.abspath(__file__)), + eng + "_scraper.py") + tree = _ast.parse(open(path, encoding="utf-8").read()) + hits = sorted({t for n in _ast.walk(tree) + if isinstance(n, _ast.Constant) and isinstance(n.value, str) + for t in stale if t in n.value}) + ok &= check("%s: no stale marketplace wording in its strings %s" + % (eng, hits or ""), not hits) + for name in ("URL_HELP", "MODE_HELP", "PAGES_HELP", "RETRIES_HELP"): + txt = getattr(page_flow, name) + ok &= check("page_flow.%s has no stale wording" % name, + not any(t in txt for t in stale)) + advice = page_flow.block_advice(BLOCKED_403, headless=True, has_pool=False) ok &= check("the block advice names the real cause (headless), not the proxy", "HEADLESS" in advice and "--headful" in advice) From 9c8888c53aa9ad0a25eeeddb47aee7f511ceb1b4 Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Tue, 13 Oct 2026 08:38:54 +0000 Subject: [PATCH 2/2] Remove another site's markup, measurements and modes from this repo; let diff_runs read lot runs Comments, docstrings, CONTRIBUTING and a few help strings still described a different marketplace (its selectors, ids, page sizes, modes and currency), and some of that was code, not prose: - diff_runs refused every --mode lot run: the diffable-mode list named a mode this repo does not have. A test pinned that list, so it defended the bug. - PageOutcome carried scroll and shop_facts fields no engine ever set, and the sidecar logic around them could never fire. Removed (the always-empty scroll and pages_still_growing sidecar keys go with them). - output_writer's Product comments now describe this site's real columns. - page_url, the completeness check and the blocked-retry comments no longer state another site's measurements, one of which contradicted the parser (it said a search cannot be paginated by URL). No column or flag changed. Suite: 499 checks; planted faults for the mode list and the removed fields each turned it red on the intended check; a live --mode lot run pair now diffs. Co-Authored-By: Claude Sonnet 5.5 --- .github/ci_checks.py | 3 +- .github/workflows/canary.yml | 24 ++--- CHANGELOG.md | 19 +++- CONTRIBUTING.md | 60 ++++------- README.md | 3 +- captcha_solver.py | 6 +- diff_runs.py | 19 ++-- env_config.py | 2 +- output_writer.py | 203 ++++++++++++----------------------- page_flow.py | 20 ++-- playwright_scraper.py | 157 +++++++++------------------ product_parser.py | 8 +- puppeteer_scraper.py | 104 ++++++------------ scraper_api_client.py | 8 +- selenium_scraper.py | 89 +++++---------- smoke_test.py | 57 ++++++---- 16 files changed, 285 insertions(+), 497 deletions(-) diff --git a/.github/ci_checks.py b/.github/ci_checks.py index 5858a7b..2779f3c 100644 --- a/.github/ci_checks.py +++ b/.github/ci_checks.py @@ -81,8 +81,7 @@ HEX32 = re.compile(r"\b[0-9a-f]{32}\b") # Contexts in which a 32-hex string is plainly not a key. # -# Deliberately WITHOUT this site's asset hosts, unlike a sibling repo that -# needed them: Catawiki serves its images under dashed UUIDs +# Deliberately WITHOUT this site's asset hosts: Catawiki serves its images under dashed UUIDs # (`95be01d1-5c40-4126-94fa-...`), which never match a bare 32-hex run, so # the committed fixtures carry none and an allowlist entry for them would be # dead configuration that reads like enforcement. Verified by running this diff --git a/.github/workflows/canary.yml b/.github/workflows/canary.yml index f8c4d95..fdba4a9 100644 --- a/.github/workflows/canary.yml +++ b/.github/workflows/canary.yml @@ -21,10 +21,9 @@ name: canary # Scraping Browser API, country-nl 200, 637 KB, 24 lots # # Read it by column: the exit address changes nothing and the headless flag -# changes everything. So a proxy secret would NOT be enough here -- which is -# the opposite of the sibling repos, where the address is exactly what gets -# scored -- and there are only two ways for a runner to get in: the Scraping -# Browser API, or a real window under xvfb. +# changes everything. So a proxy secret would NOT be enough here, and there +# are only two ways for a runner to get in: the Scraping Browser API, or a +# real window under xvfb. # # THE XVFB ROUTE IS THE CANARY. It was left unverified when this file was # first written, and the canary was made dispatch-only around a credential @@ -45,8 +44,8 @@ name: canary # * The Scraping Browser route is kept, but only on request: dispatch with # `route: cdp` and a CATAWIKI_CDP_ENDPOINT secret set. It is NOT the # scheduled route, because an endpoint on this account does not survive a -# day (every credential inherited from the sibling repos was already -# returning 401 `deny_no_user`), and a daily run against it would be +# day (every credential that was available was already returning 401 +# `deny_no_user`), and a daily run against it would be # either red every day or green having tested nothing. # # gh workflow run canary.yml -R 2scraper/catawiki-scraper -f route=cdp @@ -98,12 +97,9 @@ jobs: # A CATEGORY url, not a search — and on this site that is forced # rather than preferred. # - # A search listing has NO per-page addresses. It is one infinitely - # scrolling page, and `?page=2` on a search URL does not advance it, - # it EMPTIES the result set ("Oops, produk nggak ditemukan"). So a - # search canary could never exercise pagination, which is the thing - # this workflow most needs to watch. A category listing paginates - # with ?page=N, verified against a real page 2. + # Both listing kinds paginate with ?page=N, but a category listing is the + # stable one to watch: its lot count does not depend on a query, and + # its page 2 was verified against the real site. # # --pages 3, not 1: with one page pagination is never exercised, so a # broken ?page= convention would stay invisible here — precisely the @@ -159,8 +155,8 @@ jobs: # things that are nobody's bug. On the cdp route: # # * the SECRET EXPIRED. A Scraping Browser credential is not - # forever — every one inherited from the sibling repos was - # already dead when this repo was written. When it stops + # forever — every one available when this repo was written + # was already dead. When it stops # authenticating the run exits 5. # * ACCESS IS INTERMITTENT PER PROFILE, and a profile allows one # live connection. A canary that collides with a person's run diff --git a/CHANGELOG.md b/CHANGELOG.md index 6aef7cf..8adaef0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,16 +27,27 @@ or an empty column. likely cause: a document under 100 bytes with no site assets is called an empty document and a navigation that did not complete; headless is blamed only when the run was headless. -- **`--help` and usage errors described another marketplace scraper** (a - `product` mode, `/search?st=product`, `/p/` URLs, an Apollo cache, IDR - prices, an infinitely scrolling grid). They now describe the real URL kinds +- **`--help` and usage errors described a different site** (its modes, URL + kinds, currency and page structure). They now describe the real URL kinds and the `listing|lot|auctions` modes, from one set of strings in `page_flow`. - The `--mode lot` usage errors no longer point at `--mode product`, and the + The `--mode lot` usage errors no longer point at a mode that does not exist, and the discovery-hub warning, which could never fire, is removed. A suite check walks every string in the three engines for the old wording. +- **`diff_runs.py` refused every `--mode lot` run.** The list of diffable modes + named a mode this repo does not have (`product`) instead of `lot`, so a lot + run's sidecar was rejected. `listing` and `lot` are diffable; `auctions` rows + are a different class and still are not. + ### Changed +- The listing sidecar's `extra` no longer carries `scroll` and + `pages_still_growing`. No engine ever filled them (this site has no scroll + loop), so they were always empty; `result_header` is unchanged. The + `--mode lot` branch that read never-set seller facts is removed too. +- Comments, docstrings and `CONTRIBUTING.md` no longer describe another + site's markup, ids, measurements or modes; they describe this site's + payload and card, and the real URL kinds. No column or flag changed. - `SECURITY.md` no longer says a raw HTML dump contains session cookies; it holds the page markup, not the cookie jar. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5e7c374..d941319 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,40 +40,26 @@ not the check caught it. ## Reporting a site change -Catawiki changing its markup is the normal way this stops working, and it -has its own issue template. The detail that saves the most time is WHICH -anchor broke, because on this site there is no structured data on a listing -page to fall back on — measured zero `application/ld+json`, zero -`__NEXT_DATA__` and zero Apollo state across six captures — so the DOM is -not the primary path by preference, it is the only one. - -1. **The grid container.** `[data-testid="divSRPContentProducts"]` on a - search page, `[data-ssr="productsCategoryL2/L3SSR"]` on a category - listing. If one of these moves the run reports 0 rows and exit 4, which - is loud. -2. **The tile marker.** `[data-testid="imgLeg-c"]` on a search page (one per - tile), `[data-testid="divProductWrapper"]` inside - `a[data-testid="lnkProductContainer"]` on a category listing. -3. **The reading ORDER inside the tile** — badge, title, price, was-price, - rating, sold, shop, location. The field reads rest on it, deliberately, - because the classes around each field are build hashes: - `` is the title today. If Catawiki - reorders a tile, `title` and the prices are what break. -4. **`span.flip`**, the shop name and the shop's city in that order, exactly - two per search tile. - -The one place structured data does exist is a DETAIL page's -`window.__cache` Apollo blob, which is where `--mode product` reads the real -product id, the exact sold count and the review count. - -A third thing can break without any path failing: the **join** between the -tiles and the structured data. When it breaks, the row count and the prices -stay healthy while `in_stock` and part of `brand` quietly empty out — so -every run logs its structured-price confirmation share per page and warns -below a floor set PER PAGE KIND (search 8%, category 70%, shop 80%; the -achievable share differs by a factor of eight between them). If you are -reporting a change, that percentage and the page kind are the numbers to -include. +Catawiki changing something is the normal way this stops working, and it has +its own issue template. The detail that saves the most time is WHICH source +broke, because the parser reads two different ones for two halves of a row: + +1. **The rows** come from the site's own SSR payload, + `