Refuse --retries 0, unify the not-served message, fix donor --help text

.github/ci_checks.py

@@ -81,8 +81,7 @@
 HEX32 = re.compile(r"\b[0-9a-f]{32}\b")
 # Contexts in which a 32-hex string is plainly not a key.
 #
-# Deliberately WITHOUT this site's asset hosts, unlike a sibling repo that
-# needed them: Catawiki serves its images under dashed UUIDs
+# Deliberately WITHOUT this site's asset hosts: Catawiki serves its images under dashed UUIDs
 # (`95be01d1-5c40-4126-94fa-...`), which never match a bare 32-hex run, so
 # the committed fixtures carry none and an allowlist entry for them would be
 # dead configuration that reads like enforcement. Verified by running this

.github/workflows/canary.yml

@@ -21,10 +21,9 @@ name: canary
 #   Scraping Browser API, country-nl              200, 637 KB, 24 lots
 #
 # Read it by column: the exit address changes nothing and the headless flag
-# changes everything. So a proxy secret would NOT be enough here -- which is
-# the opposite of the sibling repos, where the address is exactly what gets
-# scored -- and there are only two ways for a runner to get in: the Scraping
-# Browser API, or a real window under xvfb.
+# changes everything. So a proxy secret would NOT be enough here, and there
+# are only two ways for a runner to get in: the Scraping Browser API, or a
+# real window under xvfb.
 #
 # THE XVFB ROUTE IS THE CANARY. It was left unverified when this file was
 # first written, and the canary was made dispatch-only around a credential
@@ -45,8 +44,8 @@ name: canary
 #   * The Scraping Browser route is kept, but only on request: dispatch with
 #     `route: cdp` and a CATAWIKI_CDP_ENDPOINT secret set. It is NOT the
 #     scheduled route, because an endpoint on this account does not survive a
-#     day (every credential inherited from the sibling repos was already
-#     returning 401 `deny_no_user`), and a daily run against it would be
+#     day (every credential that was available was already returning 401
+#     `deny_no_user`), and a daily run against it would be
 #     either red every day or green having tested nothing.
 #
 #     gh workflow run canary.yml -R 2scraper/catawiki-scraper -f route=cdp
@@ -98,12 +97,9 @@ jobs:
         # A CATEGORY url, not a search — and on this site that is forced
         # rather than preferred.
         #
-        # A search listing has NO per-page addresses. It is one infinitely
-        # scrolling page, and `?page=2` on a search URL does not advance it,
-        # it EMPTIES the result set ("Oops, produk nggak ditemukan"). So a
-        # search canary could never exercise pagination, which is the thing
-        # this workflow most needs to watch. A category listing paginates
-        # with ?page=N, verified against a real page 2.
+        # Both listing kinds paginate with ?page=N, but a category listing is the
+        # stable one to watch: its lot count does not depend on a query, and
+        # its page 2 was verified against the real site.
         #
         # --pages 3, not 1: with one page pagination is never exercised, so a
         # broken ?page= convention would stay invisible here — precisely the
@@ -159,8 +155,8 @@ jobs:
         # things that are nobody's bug. On the cdp route:
         #
         #   * the SECRET EXPIRED. A Scraping Browser credential is not
-        #     forever — every one inherited from the sibling repos was
-        #     already dead when this repo was written. When it stops
+        #     forever — every one available when this repo was written
+        #     was already dead. When it stops
         #     authenticating the run exits 5.
         #   * ACCESS IS INTERMITTENT PER PROFILE, and a profile allows one
         #     live connection. A canary that collides with a person's run

CHANGELOG.md

@@ -13,6 +13,44 @@ or an empty column.
 
 ### Fixed
 
+- **`--retries 0` skipped navigation and reported a block.** The load loop
+  is `range(1, retries + 1)`, so 0 ran it never, `goto` was not called, and
+  the engine judged `about:blank`: a 39-byte empty DOM, exit 3, for a site that
+  was never asked. Reproduced 2026-10-09 (Playwright, `--retries 0`: exit 3 in
+  ~50 ms in both modes; the same command with `--retries 1` fetched 72 lots,
+  `complete`, exit 0). `--retries` now takes an integer of at least 1 in all
+  three engines and refuses 0 as a usage error (exit 2).
+- **The not-served message contradicted the README.** It said a headless
+  browser was the cause, then recommended a residential exit, and said it for a
+  headful run too. One shared `page_flow.not_served_message` now states what
+  was observed (bytes, whether the site's own assets appear) and only then the
+  likely cause: a document under 100 bytes with no site assets is called an
+  empty document and a navigation that did not complete; headless is blamed
+  only when the run was headless.
+- **`--help` and usage errors described a different site** (its modes, URL
+  kinds, currency and page structure). They now describe the real URL kinds
+  and the `listing|lot|auctions` modes, from one set of strings in `page_flow`.
+  The `--mode lot` usage errors no longer point at a mode that does not exist, and the
+  discovery-hub warning, which could never fire, is removed. A suite check walks
+  every string in the three engines for the old wording.
+
+- **`diff_runs.py` refused every `--mode lot` run.** The list of diffable modes
+  named a mode this repo does not have (`product`) instead of `lot`, so a lot
+  run's sidecar was rejected. `listing` and `lot` are diffable; `auctions` rows
+  are a different class and still are not.
+
+### Changed
+
+- The listing sidecar's `extra` no longer carries `scroll` and
+  `pages_still_growing`. No engine ever filled them (this site has no scroll
+  loop), so they were always empty; `result_header` is unchanged. The
+  `--mode lot` branch that read never-set seller facts is removed too.
+- Comments, docstrings and `CONTRIBUTING.md` no longer describe another
+  site's markup, ids, measurements or modes; they describe this site's
+  payload and card, and the real URL kinds. No column or flag changed.
+- `SECURITY.md` no longer says a raw HTML dump contains session cookies; it
+  holds the page markup, not the cookie jar.
+
 - **The Scraper API path (`scraper_api_client.py`) failed whenever a wait flag
   was given, and never saw the target's status.** Measured 2026-09-23 against
   the live `/tasks/sync` endpoint: `waitFor` sent as a JSON-encoded string

CONTRIBUTING.md

@@ -40,40 +40,26 @@ not the check caught it.
 
 ## Reporting a site change
 
-Catawiki changing its markup is the normal way this stops working, and it
-has its own issue template. The detail that saves the most time is WHICH
-anchor broke, because on this site there is no structured data on a listing
-page to fall back on — measured zero `application/ld+json`, zero
-`__NEXT_DATA__` and zero Apollo state across six captures — so the DOM is
-not the primary path by preference, it is the only one.
-
-1. **The grid container.** `[data-testid="divSRPContentProducts"]` on a
-   search page, `[data-ssr="productsCategoryL2/L3SSR"]` on a category
-   listing. If one of these moves the run reports 0 rows and exit 4, which
-   is loud.
-2. **The tile marker.** `[data-testid="imgLeg-c"]` on a search page (one per
-   tile), `[data-testid="divProductWrapper"]` inside
-   `a[data-testid="lnkProductContainer"]` on a category listing.
-3. **The reading ORDER inside the tile** — badge, title, price, was-price,
-   rating, sold, shop, location. The field reads rest on it, deliberately,
-   because the classes around each field are build hashes:
-   `<span class="+tnoqZhn89+NHUA43BpiJg==">` is the title today. If Catawiki
-   reorders a tile, `title` and the prices are what break.
-4. **`span.flip`**, the shop name and the shop's city in that order, exactly
-   two per search tile.
-
-The one place structured data does exist is a DETAIL page's
-`window.__cache` Apollo blob, which is where `--mode product` reads the real
-product id, the exact sold count and the review count.
-
-A third thing can break without any path failing: the **join** between the
-tiles and the structured data. When it breaks, the row count and the prices
-stay healthy while `in_stock` and part of `brand` quietly empty out — so
-every run logs its structured-price confirmation share per page and warns
-below a floor set PER PAGE KIND (search 8%, category 70%, shop 80%; the
-achievable share differs by a factor of eight between them). If you are
-reporting a change, that percentage and the page kind are the numbers to
-include.
+Catawiki changing something is the normal way this stops working, and it has
+its own issue template. The detail that saves the most time is WHICH source
+broke, because the parser reads two different ones for two halves of a row:
+
+1. **The rows** come from the site's own SSR payload,
+   `<script id="__NEXT_DATA__">`, which is in the first response and needs no
+   JavaScript: titles, ids, urls, images, auction ids and the reserve flag. If
+   that moves, the run reports 0 rows and exit 4, which is loud. A lot page
+   (`--mode lot`) is read from the same payload and never from its DOM.
+2. **The money** comes from the rendered card (`.c-lot-card__price` and its
+   siblings), because the payload does not carry it: bids arrive at
+   hydration. If that moves, rows come out with no prices.
+
+`price_source` on a row says which happened: `next_data+dom` is healthy and
+`next_data` means nothing hydrated. A third thing can break without any path
+failing: a status label the page's own translation store no longer maps, which
+leaves `bid_kind` null on rows that have a price. Every run logs its price
+coverage per page and warns below a floor set per page kind (`PRICE_FLOOR` in
+the engines); if you are reporting a change, that percentage and the page kind
+are the numbers to include.
 
 `--dump-html PATH` writes the exact bytes the parser was given, on success as
 well as failure, and a run that finds nothing writes a dump and a screenshot
@@ -191,10 +177,8 @@ silently regress:
   stripping, so the guard would be code that looks load-bearing and never
   runs. Broaden the set and add the guard together.
 - **A sku already written by an earlier page of the same run is dropped, not
-  duplicated.** Unlike its sibling repos this DOES fire on healthy runs
-  here: page 1 and page 2 of one category listing shared exactly 3 products,
-  all three from the "cheaper products" carousel that appears on every page.
-  So a small non-zero drop count is expected and a large one is not. See
+  duplicated.** A live auction listing reorders as lots close, so a small
+  non-zero drop count is possible and a large one is not. See
   `dedupe_by_key` in `output_writer.py`.
 
 There is also a naming check: certain phrases are banned repo-wide and the suite

README.md

@@ -37,8 +37,7 @@ Anything not measured is not claimed.
 Read it by column, not by row: the exit address changes nothing and the
 headless flag changes everything. So:
 
-* **`--headful` is the default here**, unlike every sibling repo in this
-  family. A `--headless` default would be a scraper whose default cannot
+* **`--headful` is the default here.** A `--headless` default would be a scraper whose default cannot
   fetch the site.
 * `--headless` is still accepted, because `--cdp-endpoint` ignores it and
   because a change on the site would make it work again. It will report

SECURITY.md

@@ -100,7 +100,10 @@ Three places leak credentials that people do not expect, because unlike our own
 log lines they are **not** masked:
 
 - **raw HTML dumps** (`--dump-html`, and the automatic dump on a zero-product
-  run) — these can contain session cookies
+  run) — the page's markup as the browser serialised it, not the cookie jar,
+  but a site can write session identifiers or CSRF tokens into inline scripts,
+  so treat a dump as sensitive. A blocked page's dump is a few hundred bytes
+  and carries none
 - **the Scraper API's `x-debug` response header**
 - **your shell history**, if you passed a key on the command line
 

captcha_solver.py

@@ -703,10 +703,8 @@ class CaptchaUnsolvable(RuntimeError):
 # ===========================================================================
 # What is deliberately NOT here
 # ===========================================================================
-# The sibling repo in this family carries a whole second solver for its
-# site's OWN first-party image captcha ("Enter the characters you see below",
-# a JPEG of distorted text and a GET form). Roughly 190 lines of it, and none
-# of it is ported here, because this site has no such page.
+# A solver for a site's OWN first-party image captcha (a JPEG of distorted
+# text and a form) is not here, because this site has no such page.
 #
 # What this site does instead is refuse a HEADLESS browser. Measured
 # 2026-09-10 from five different addresses, four of them residential: Akamai

diff_runs.py

@@ -93,12 +93,9 @@ def _within_tolerance(before: dict, after: dict, changes: dict,
                       tolerance_pct: float) -> bool:
     """True if every differing price field moved by less than `tolerance_pct`.
 
-    Inherited from this family rather than earned here, and said plainly
-    because the alternative is a comment inventing a reason. A sibling repo
-    needs it: that site converts prices for a cross-border visitor and the
-    exchange rate ticks between two runs of the same command. NO EQUIVALENT
-    CATAWIKI BEHAVIOUR WAS MEASURED — this site quotes euro to every
-    visitor, on all 18 locales including Japanese and Chinese and from every
+    Part of the family's diff contract, and said plainly because the
+    alternative is a comment inventing a reason. NO CATAWIKI BEHAVIOUR NEEDS
+    IT — this site quotes euro to every visitor, on all 18 locales including Japanese and Chinese and from every
     exit tried, so a run holds no conversion and every cent of a difference
     is a real move.
 
@@ -318,9 +315,9 @@ def _check_comparable(args) -> bool:
     # A CURRENCY MISMATCH, which on this site should be impossible — and is
     # checked anyway.
     #
-    # The sibling repos guard cross-storefront diffs with `source`: eleven
-    # country hostnames, so a run of one against another is refused on the
-    # hostname alone. Catawiki is ONE host with ONE currency — 18 locales as
+    # Cross-storefront diffs are normally guarded with `source`, so a run of one
+    # hostname against another is refused on the hostname alone. Catawiki is
+    # ONE host with ONE currency — 18 locales as
     # a path prefix, euro on every one of them, measured 2026-09-10 — so
     # `source` is "catawiki.com" on both sides and there is no storefront
     # split for it to catch.
@@ -382,9 +379,9 @@ def parse_args():
                         "rate tick rather than a price change: reported "
                         "separately and ignored by --fail-on-change. Default 0 "
                         "(report every cent), which is what a Catawiki "
-                        "run wants: the site quotes IDR to every visitor, so "
+                        "run wants: the site quotes euro to every visitor, so "
                         "there is no conversion drift to absorb. The flag is "
-                        "inherited from this scraper family; set it non-zero "
+                        "part of the family's diff contract; set it non-zero "
                         "only with a reason you can state.")
     p.add_argument("--fail-on-change", action="store_true",
                    help="Exit 1 if anything was added, removed or changed — "

env_config.py

@@ -200,7 +200,7 @@ def _dests_given_on_argv(dests, argv=None):
     remote browser the caller had just asked it not to use. There was no
     spelling of "ignore what the file says" at all.
 
-    That matters more here than in the sibling repos: this site's local
+    That matters here because this site's local
     headful path and its Scraping Browser path are both first-class, and
     switching between them is routine rather than exceptional.