diff --git a/.github/ci_checks.py b/.github/ci_checks.py index 5858a7b..2779f3c 100644 --- a/.github/ci_checks.py +++ b/.github/ci_checks.py @@ -81,8 +81,7 @@ HEX32 = re.compile(r"\b[0-9a-f]{32}\b") # Contexts in which a 32-hex string is plainly not a key. # -# Deliberately WITHOUT this site's asset hosts, unlike a sibling repo that -# needed them: Catawiki serves its images under dashed UUIDs +# Deliberately WITHOUT this site's asset hosts: Catawiki serves its images under dashed UUIDs # (`95be01d1-5c40-4126-94fa-...`), which never match a bare 32-hex run, so # the committed fixtures carry none and an allowlist entry for them would be # dead configuration that reads like enforcement. Verified by running this diff --git a/.github/workflows/canary.yml b/.github/workflows/canary.yml index f8c4d95..fdba4a9 100644 --- a/.github/workflows/canary.yml +++ b/.github/workflows/canary.yml @@ -21,10 +21,9 @@ name: canary # Scraping Browser API, country-nl 200, 637 KB, 24 lots # # Read it by column: the exit address changes nothing and the headless flag -# changes everything. So a proxy secret would NOT be enough here -- which is -# the opposite of the sibling repos, where the address is exactly what gets -# scored -- and there are only two ways for a runner to get in: the Scraping -# Browser API, or a real window under xvfb. +# changes everything. So a proxy secret would NOT be enough here, and there +# are only two ways for a runner to get in: the Scraping Browser API, or a +# real window under xvfb. # # THE XVFB ROUTE IS THE CANARY. It was left unverified when this file was # first written, and the canary was made dispatch-only around a credential @@ -45,8 +44,8 @@ name: canary # * The Scraping Browser route is kept, but only on request: dispatch with # `route: cdp` and a CATAWIKI_CDP_ENDPOINT secret set. It is NOT the # scheduled route, because an endpoint on this account does not survive a -# day (every credential inherited from the sibling repos was already -# returning 401 `deny_no_user`), and a daily run against it would be +# day (every credential that was available was already returning 401 +# `deny_no_user`), and a daily run against it would be # either red every day or green having tested nothing. # # gh workflow run canary.yml -R 2scraper/catawiki-scraper -f route=cdp @@ -98,12 +97,9 @@ jobs: # A CATEGORY url, not a search — and on this site that is forced # rather than preferred. # - # A search listing has NO per-page addresses. It is one infinitely - # scrolling page, and `?page=2` on a search URL does not advance it, - # it EMPTIES the result set ("Oops, produk nggak ditemukan"). So a - # search canary could never exercise pagination, which is the thing - # this workflow most needs to watch. A category listing paginates - # with ?page=N, verified against a real page 2. + # Both listing kinds paginate with ?page=N, but a category listing is the + # stable one to watch: its lot count does not depend on a query, and + # its page 2 was verified against the real site. # # --pages 3, not 1: with one page pagination is never exercised, so a # broken ?page= convention would stay invisible here — precisely the @@ -159,8 +155,8 @@ jobs: # things that are nobody's bug. On the cdp route: # # * the SECRET EXPIRED. A Scraping Browser credential is not - # forever — every one inherited from the sibling repos was - # already dead when this repo was written. When it stops + # forever — every one available when this repo was written + # was already dead. When it stops # authenticating the run exits 5. # * ACCESS IS INTERMITTENT PER PROFILE, and a profile allows one # live connection. A canary that collides with a person's run diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a41f51..8adaef0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,44 @@ or an empty column. ### Fixed +- **`--retries 0` skipped navigation and reported a block.** The load loop + is `range(1, retries + 1)`, so 0 ran it never, `goto` was not called, and + the engine judged `about:blank`: a 39-byte empty DOM, exit 3, for a site that + was never asked. Reproduced 2026-10-09 (Playwright, `--retries 0`: exit 3 in + ~50 ms in both modes; the same command with `--retries 1` fetched 72 lots, + `complete`, exit 0). `--retries` now takes an integer of at least 1 in all + three engines and refuses 0 as a usage error (exit 2). +- **The not-served message contradicted the README.** It said a headless + browser was the cause, then recommended a residential exit, and said it for a + headful run too. One shared `page_flow.not_served_message` now states what + was observed (bytes, whether the site's own assets appear) and only then the + likely cause: a document under 100 bytes with no site assets is called an + empty document and a navigation that did not complete; headless is blamed + only when the run was headless. +- **`--help` and usage errors described a different site** (its modes, URL + kinds, currency and page structure). They now describe the real URL kinds + and the `listing|lot|auctions` modes, from one set of strings in `page_flow`. + The `--mode lot` usage errors no longer point at a mode that does not exist, and the + discovery-hub warning, which could never fire, is removed. A suite check walks + every string in the three engines for the old wording. + +- **`diff_runs.py` refused every `--mode lot` run.** The list of diffable modes + named a mode this repo does not have (`product`) instead of `lot`, so a lot + run's sidecar was rejected. `listing` and `lot` are diffable; `auctions` rows + are a different class and still are not. + +### Changed + +- The listing sidecar's `extra` no longer carries `scroll` and + `pages_still_growing`. No engine ever filled them (this site has no scroll + loop), so they were always empty; `result_header` is unchanged. The + `--mode lot` branch that read never-set seller facts is removed too. +- Comments, docstrings and `CONTRIBUTING.md` no longer describe another + site's markup, ids, measurements or modes; they describe this site's + payload and card, and the real URL kinds. No column or flag changed. +- `SECURITY.md` no longer says a raw HTML dump contains session cookies; it + holds the page markup, not the cookie jar. + - **The Scraper API path (`scraper_api_client.py`) failed whenever a wait flag was given, and never saw the target's status.** Measured 2026-09-23 against the live `/tasks/sync` endpoint: `waitFor` sent as a JSON-encoded string diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5e7c374..d941319 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,40 +40,26 @@ not the check caught it. ## Reporting a site change -Catawiki changing its markup is the normal way this stops working, and it -has its own issue template. The detail that saves the most time is WHICH -anchor broke, because on this site there is no structured data on a listing -page to fall back on — measured zero `application/ld+json`, zero -`__NEXT_DATA__` and zero Apollo state across six captures — so the DOM is -not the primary path by preference, it is the only one. - -1. **The grid container.** `[data-testid="divSRPContentProducts"]` on a - search page, `[data-ssr="productsCategoryL2/L3SSR"]` on a category - listing. If one of these moves the run reports 0 rows and exit 4, which - is loud. -2. **The tile marker.** `[data-testid="imgLeg-c"]` on a search page (one per - tile), `[data-testid="divProductWrapper"]` inside - `a[data-testid="lnkProductContainer"]` on a category listing. -3. **The reading ORDER inside the tile** — badge, title, price, was-price, - rating, sold, shop, location. The field reads rest on it, deliberately, - because the classes around each field are build hashes: - `` is the title today. If Catawiki - reorders a tile, `title` and the prices are what break. -4. **`span.flip`**, the shop name and the shop's city in that order, exactly - two per search tile. - -The one place structured data does exist is a DETAIL page's -`window.__cache` Apollo blob, which is where `--mode product` reads the real -product id, the exact sold count and the review count. - -A third thing can break without any path failing: the **join** between the -tiles and the structured data. When it breaks, the row count and the prices -stay healthy while `in_stock` and part of `brand` quietly empty out — so -every run logs its structured-price confirmation share per page and warns -below a floor set PER PAGE KIND (search 8%, category 70%, shop 80%; the -achievable share differs by a factor of eight between them). If you are -reporting a change, that percentage and the page kind are the numbers to -include. +Catawiki changing something is the normal way this stops working, and it has +its own issue template. The detail that saves the most time is WHICH source +broke, because the parser reads two different ones for two halves of a row: + +1. **The rows** come from the site's own SSR payload, + `