diff --git a/.dockerignore b/.dockerignore index c2f7bcc..6afcfe6 100644 --- a/.dockerignore +++ b/.dockerignore @@ -11,7 +11,8 @@ __pycache__/ .venv/ venv/ env/ -bbb_businesses.* +kose_products.* +kose_products_*.* *_debug.html *_debug.png live_results/ diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index d39db59..5f5b8e4 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -10,26 +10,23 @@ body: mask credentials in their own logs, but raw HTML dumps and the Scraper API's `x-debug` header do not. Replace them with `***`. - Two things answer most reports faster than we can: + A few things answer most reports faster than we can: - - **Everything exit 3?** Check the exit address first. Etsy's - DataDome answers every datacentre and VPN IP with HTTP 403, so a run - from a VPS or a CI runner is blocked whatever else is set. If the - log says `t=bv`, a captcha key does not help — that challenge is - not solvable. The Scraping Browser API (`--cdp-endpoint`) is the - path that works; if a profile keeps refusing after three retries, - try a different `pid`. - - **Zero products?** You may be on a page that has none — a taxonomy - hub, or one page past the end of a listing. It answers 200 with - nothing on it, which is exit 4 and a correct answer. See - [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md). - - **`rating` empty on every row?** That is expected on a listing run: - Etsy publishes no per-listing rating there. The stars you see on a - tile are the shop's and are in `shop_rating`. See the README. - - **Wrong currency or language?** Etsy decides both from your exit - IP, not from the URL — a request for `/fr/…` from a German exit is - redirected to `/de/…`. The row's `url` says which storefront - actually answered. + - **Everything exit 3?** Maison KOSÉ has served every page tested, + datacentre addresses and a bare `python-requests` User-Agent + included, with no captcha of any vendor (see the README). A + block here is new — the log and a `--dump-html` capture are the + most useful things you can attach. + - **Zero products?** A bogus category id or goods code answers HTTP + 200 with the site's chrome and no tiles, which is exit 4 and a + correct answer. A wrong PATH is a real 404, and the run stops on + it rather than retrying. See "Traps that look like bugs" in the + README. + - **`rating` or `original_price` missing?** Expected: this site + publishes neither, and the README says what was measured. + - **Every page repeats?** A category walked past its end serves its + LAST page again; the run plans against the site's own page + counter and stops on "this page added no new sku". - type: dropdown id: engine @@ -77,8 +74,8 @@ body: render: shell placeholder: | python3 playwright_scraper.py \ - --url "https://www.etsy.com/search?q=handmade+mug" \ - --pages 3 --out grills + --url "https://maison.kose.co.jp/site/cosmedecorte/c/c15/" \ + --pages 3 validations: required: true @@ -98,7 +95,7 @@ body: attributes: label: What you expected instead placeholder: >- - 96 products, as the README says a category page yields. + 72 products, as the README says a 3-page run of /c/c15/ yields. validations: required: true diff --git a/.github/ISSUE_TEMPLATE/question.yml b/.github/ISSUE_TEMPLATE/question.yml index db40153..1544a13 100644 --- a/.github/ISSUE_TEMPLATE/question.yml +++ b/.github/ISSUE_TEMPLATE/question.yml @@ -7,8 +7,7 @@ body: value: | Worth a look first, in this order: - - [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md) — ordered by how - often each thing actually happens + - The README's "Traps that look like bugs" section - The README section on the 2Captcha product you are using - `python3 env_config.py` — reports what credentials are configured, without printing them diff --git a/.github/ISSUE_TEMPLATE/site_changed.yml b/.github/ISSUE_TEMPLATE/site_changed.yml index 69fa3b3..9d39ecc 100644 --- a/.github/ISSUE_TEMPLATE/site_changed.yml +++ b/.github/ISSUE_TEMPLATE/site_changed.yml @@ -1,5 +1,5 @@ name: Site changed / selectors broke -description: Etsy changed something and the parser no longer matches. +description: Maison KOSÉ changed something and the parser no longer matches. labels: ["site-change"] body: - type: markdown @@ -7,10 +7,12 @@ body: value: | This is the most common way a scraper stops working, and it is worth its own report because the fix is different from a code bug: something on - etsy.com moved. + maison.kose.co.jp moved. - Useful to know before filing: the parser tries **JSON-LD first**, then a - CSS + URL-pattern fallback. Which one broke narrows the fix a lot. + Useful to know before filing: a LISTING page carries no JSON-LD, so + listings are read from the tile markup (`li.c-product__item` and a + `/g/g{SKU}/` link); a PRODUCT page is read from its one JSON-LD + `Product` block. Which route broke narrows the fix a lot. - type: dropdown id: what_broke @@ -19,8 +21,7 @@ body: options: - Products (0 rows on a page that visibly has products) - A specific field is null or wrong (price, sku, image, URL) - - original_price or lowest_price_30d empty out (the tile overlay) - - DOM price confirmation dropped below 90% in the run log + - tax_rate or in_stock reads wrong - Pagination stops early or repeats a page - The captcha detector finds nothing on a page that has one - Something else @@ -31,8 +32,8 @@ body: id: url attributes: label: URL - description: The exact listing URL, and the country your exit IP was in. - placeholder: "https://www.etsy.com/search?q=handmade+mug (locale: /de/, exit IP: DE residential, Scraping Browser pid: etsy1)" + description: The exact listing or product URL, and where your exit IP was. + placeholder: "https://maison.kose.co.jp/site/cosmedecorte/c/c15/ (exit IP: DE datacentre)" validations: required: true @@ -43,9 +44,9 @@ body: description: | Whichever of these you can get. A capture beats a description. - - JSON-LD as the page actually serves it: + - For a product page, its JSON-LD as the page actually serves it: `python3 -c "import json,sys,re;h=open('dump.html').read();print([m[:400] for m in re.findall(r'',h,re.S)][:2])"` - - Or open devtools and paste one `Product` node from the `ItemList`. + - For a listing: one `li.c-product__item` tile's outer HTML. - For a field problem: the value you got, and the value on the page. render: text validations: diff --git a/CHANGELOG.md b/CHANGELOG.md index 52f2c58..97097db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,46 @@ existing user would notice, the release notes lead with it. this client. `http_code` (an int) is read now, with `status` kept as a fallback only when it is an int. +Leftovers from the sibling repos this one was bootstrapped from +(montblanc-scraper, bbb-scraper, etsy-scraper), which described those sites +as if they were this one. + +- **User-visible strings.** `--pages` help named a `--mode category` and + `--mode search` this repo does not have and said a category past its end is + "a served, empty grid" (here it re-serves its last page). The Playwright + blocked-page error said this site's edge "refuses `curl`, `python-requests` + and friends outright", the opposite of what the README measured; it now + matches the other two engines. The parse-failure error pointed at a JSON-LD + `ItemList` a listing here does not carry; the image-coverage log line + described that same `ItemList`; `--mode product` logs and `--help` spoke of + variants and a `ProductGroup`; `--retries` help spoke of hub pages; + `--dump-html` help pointed at a `TROUBLESHOOTING.md` that does not exist. +- **Engines agree on the refusal reason.** pyppeteer and Selenium recorded a + refusal as `blocked_cloudflare (hard block)`, a vendor never observed here, + while Playwright recorded `blocked_edge refusal`. All three now say + `edge refusal`. +- **`diff_runs.py` tracked Montblanc's columns** (`collection`, + `sub_collection`, `color`, `size`, `special_edition`, `base_sku`), none of + which exist in this row model. It now tracks this site's own `tax_rate` and + `subcategory`, and treats `subcategory` / `variant_of` as product-mode-only. +- **Issue templates** were Etsy's (DataDome, `shop_rating`, an etsy.com + example URL). Rewritten from this repo's README. +- **Donor prose in core modules and comments:** the BBB Turnstile paragraphs + in `captcha_solver.py`, Montblanc URLs and counts in the engines, BBB's + `find_country` in two locale comments, and an unnamed tokopedia hub + anecdote now credited to tokopedia-scraper. +- **Test data:** `smoke_test.py` built its sidecar fixtures from a + montblanc.com run; they now use this site's measured `/c/c15/` run (13 + pages, 304 products, `mode="listing"`) and assert the sidecar fields this + engine actually writes. +- **`.dockerignore`** listed `bbb_businesses.*`; it now lists this repo's + `kose_products.*`. The Dockerfile example wrote to `writing-instruments`. + +- `SECURITY.md` said this project has no releases or version tags; it has + both. "Supported versions" now names the latest release and `main`. +- `captcha_solver.py`'s docstring pointed at a "No DataDome solver" section + that does not exist in this repo (it came with the copied core). Removed. + ## [0.1.1] — 2026-09-18 A pass back over CLAUDE.md before calling the repo finished. Everything here diff --git a/Dockerfile b/Dockerfile index 602e091..2af0ec8 100644 --- a/Dockerfile +++ b/Dockerfile @@ -5,7 +5,7 @@ # docker build -t kose-scraper . # docker run --rm -v "$PWD/out:/out" kose-scraper \ # --url "https://maison.kose.co.jp/site/cosmedecorte/c/c15/" \ -# --pages 3 --out /out/writing-instruments +# --pages 3 --out /out/kose_products # # Pass --proxy/--twocaptcha-key the same way as running locally, or mount a # .env at /app/.env — nothing here bakes in a credential. diff --git a/SECURITY.md b/SECURITY.md index 4fb4aae..b3f92e0 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -79,9 +79,9 @@ Not because these do not matter, but because they belong somewhere else: ## Supported versions -`main` only. This project has no releases or version tags; fixes land on `main` -and you update by pulling. If you are running an old clone, update before -reporting. +The latest release and `main`. Fixes land on `main` first and ship in the +next tagged release (see the Releases page). If you are running an old clone, +update before reporting. ## If you have leaked a key diff --git a/captcha_solver.py b/captcha_solver.py index be35134..e57ecb3 100644 --- a/captcha_solver.py +++ b/captcha_solver.py @@ -6,13 +6,13 @@ Detection runs after EVERY page navigation in the main loop of all three scrapers, regardless of what URL was requested (category hub, product page, sign-in, checkout, anything) — this is deliberate, not scoped to any one -page. If BBB renders a reCAPTCHA challenge anywhere — account, sign-in and -review-submission flows are the usual places — this fires. +page. If Maison KOSÉ ever renders a reCAPTCHA challenge anywhere — account, +sign-in and checkout flows are the usual places — this fires. -**BBB's own gate is Cloudflare Turnstile, not reCAPTCHA.** Measured -2026-09-16: a Managed Challenge carries `cf_chl_opt` and a Turnstile widget, -and the pages this scraper reads carry no reCAPTCHA at all rather than -an interstitial. So this module is a contingency, not part of the happy path +**No challenge has been observed on this site at all.** Measured +2026-09-18 (see the README): zero captcha markers of any vendor across 13 +captures — listings, product pages, the site's own 404 and an exhausted facet +alike. So this module is a contingency, not part of the happy path — a bot manager can be switched on between deploys, and a scraper that cannot name what stopped it is much harder to fix. @@ -23,8 +23,6 @@ purchase, and `page_flow.STATE_POLICY` — not this file — decides which state is worth money at all. -There is deliberately no DataDome path here. See "No DataDome solver" below. - Flow: 1. Both detectors run and are reconciled (see reconcile_detections) to decide the variant: v3, v2-invisible or v2-checkbox. The parameters differ per @@ -707,26 +705,9 @@ class CaptchaUnsolvable(RuntimeError): # =========================================================================== # The sibling repo in this family carries a whole second solver for its # site's OWN first-party image captcha ("Enter the characters you see below", -# a JPEG of distorted text and a GET form). Roughly 190 lines of it, and none -# of it is ported here, because BBB has no such page. -# -# What BBB does instead is refuse in two shapes, and only ONE of them is -# something a solver can answer. Measured 2026-09-16: -# -# "Just a moment..." a Cloudflare Managed Challenge, ~15 KB, with -# `cf_chl_opt`, `__cf_chl` and a Turnstile -# widget on it. This IS a test, and it is what -# the machinery below is for. -# "You have been blocked | a refusal, ~12.5 KB, with no widget, no -# Better Business Bureau(R)" sitekey and no challenge of any kind. There -# is nothing for a solver to answer. -# -# Both answer HTTP 403 and both wear BBB's own branding in the title, so they -# are told apart structurally rather than by the title — -# product_parser.detect_page_state reports the first as "challenge" and the -# second as "blocked", and page_flow.STATE_POLICY spends on the first and -# never on the second. From a datacenter address it is the second that -# arrives, and the answer to it is a different exit rather than a purchase. +# a JPEG of distorted text and a GET form). None of it is ported here, +# because Maison KOSÉ has no such page: 13 captures measured 2026-09-18 +# carried no challenge of any kind (see the README). # # The reCAPTCHA / hCaptcha / Turnstile machinery above IS kept, and that is a # deliberate asymmetry rather than an inconsistency. Detection stays broad diff --git a/diff_runs.py b/diff_runs.py index 27f03b4..2b6a33a 100644 --- a/diff_runs.py +++ b/diff_runs.py @@ -15,7 +15,7 @@ product's own taxonomy source_changed the two rows were read a different way — a listing row and a - `--mode product` variant row populate different columns — so + `--mode product` row populate different columns — so the difference is about our two snapshots rather than about the product, and `--fail-on-change` ignores it (§8) @@ -66,41 +66,39 @@ # compared alongside `price_source` so a number that moved because it was # read from a different NODE is not reported as a price change (see below). # -# `in_stock` earns its place because it genuinely varies here — measured over -# 384 tiles, `available` on 377 and `soldout` on 7 — unlike the JSON-LD's own -# availability, which was InStock on 192 of 192 and would have made a -# constant column look like a monitored one. +# `in_stock` earns its place because it genuinely varies here — 19 of 171 +# measured tiles say 現在購入頂けません or 予約受付終了 (README). # -# The taxonomy fields are tracked because a product moving collection, or -# gaining a special-edition flag, is a real editorial change worth seeing. +# `tax_rate` is tracked because a product moving between Japan's 10% and 8% +# rates changes what its tax-inclusive price means. The taxonomy fields are +# tracked because a product moving category is a real editorial change +# worth seeing. # # Deliberately NOT tracked: `position` and `page` (a property of the slice, -# not of the product), `image_url` (the URL carries a content hash that -# changes when the site re-encodes an unchanged photo), and `title`, which is -# printed on every line anyway and whose whitespace the site is inconsistent -# about. +# not of the product), `image_url`, `badges` (the site's merchandising +# flags, which describe a campaign rather than the product), and `title`, which is printed on every +# line anyway. TRACKED_FIELDS = ( # the point of the exercise - "price", "currency", "in_stock", + "price", "currency", "in_stock", "tax_rate", # what the product IS - "category", "collection", "sub_collection", "color", "size", - "special_edition", "brand", - # identity, in case a variant is re-parented - "base_sku", "variant_of", + "category", "subcategory", "brand", + # identity, in case a size or shade is re-parented + "variant_of", ) # The subset that ONLY one kind of run populates. # -# A `--mode product` variant row has no tile behind it, so it leaves the -# tile-sourced columns null; a listing row has no `variant_of`. Diffing a -# listing run against a product run would report each of these as a change on -# every row, and none of it would be about the product. When the two rows -# disagree on `mode`, these are reported as `source_changed` rather than as -# changes — this family's rule (§8: a difference that comes with a -# provenance difference says something about our own two snapshots, not about -# the site). +# A `--mode product` row has no tile behind it and a listing row has no +# product page behind it: `subcategory` and `variant_of` are filled only from +# a product page. Diffing a listing run against a product run would report +# each of these as a change on every row, and none of it would be about the +# product. When the two rows disagree on `mode`, these are reported as +# `source_changed` rather than as changes — this family's rule (§8: a +# difference that comes with a provenance difference says something about our +# own two snapshots, not about the site). PROFILE_ONLY_FIELDS = ( - "collection", "sub_collection", "special_edition", "variant_of", + "subcategory", "variant_of", ) # Kept under a second name because that is what this repo's row model calls @@ -152,7 +150,7 @@ def diff_products(old: List[dict], new: List[dict]) -> dict: # A row whose `mode` differs between runs is not comparable on the # mode-specific columns: a listing row fills the tile-sourced ones - # and leaves `variant_of` null, a `--mode product` variant row does + # and leaves `variant_of` null, a `--mode product` row does # the opposite. Every one of them would read as a change and none of # it would be about the product. Reporting it as a change would be a # false alarm about the site; the other columns still compare fine. @@ -205,7 +203,7 @@ def _print_summary(result: dict) -> None: for f, v in c["changes"].items()) print(f" ? {c['sku']} {c['title']} {deltas} " f"[mode {src['old']!r} -> {src['new']!r}: a listing row and a " - f"variant row fill different columns, so this is not a change " + f"product row fill different columns, so this is not a change " f"in the product]") unmatchable = result["unmatchable_old"] + result["unmatchable_new"] if unmatchable: diff --git a/output_writer.py b/output_writer.py index c7511a1..0d3135f 100644 --- a/output_writer.py +++ b/output_writer.py @@ -11,11 +11,11 @@ Both yield the SAME class. Maison KOSÉ publishes one leaf — a PRODUCT — and a listing is a way of SELECTING products while a detail page is one product -described more fully. Unlike Montblanc, "more fully" here is NOT more rows: a -Kosé product page publishes a single `Product` block, never a `ProductGroup` -with `hasVariant`, so `--mode product` emits exactly one row per page. What it -adds is columns a tile does not carry — `subcategory`, `volume`, -`colour_count`, `release_date` — plus a structured price and availability +described more fully. Unlike montblanc-scraper's site, "more fully" here is +NOT more rows: a Kosé product page publishes a single `Product` block, never a +`ProductGroup` with `hasVariant`, so `--mode product` emits exactly one row +per page. What it adds is columns a tile does not carry — `subcategory`, +`volume`, `colour_count`, `release_date` — plus a structured price and availability straight from the site's own JSON-LD. The two listing ROUTES are one mode and not two, because they return the @@ -533,8 +533,8 @@ def save(rows: Sequence[Any], out_prefix: str, fmt: str, # Planning against it is not merely an optimisation on this site, which is # the part worth remembering. Walking off the end of a category returns the # LAST PAGE AGAIN — byte-identical, HTTP 200, forever — rather than the empty -# grid Montblanc returns or the HTTP 500 that the same overshoot produces on -# BBB. A run that trusted "keep going until a page is empty" would never +# grid montblanc-scraper's site returns or the HTTP 500 that the same +# overshoot produces for bbb-scraper. A run that trusted "keep going until a page is empty" would never # terminate. A TAG listing does end with an empty page, so the two routes # want the same data-based stop for opposite reasons. # diff --git a/page_flow.py b/page_flow.py index 86f5a28..f36e1ae 100644 --- a/page_flow.py +++ b/page_flow.py @@ -85,7 +85,7 @@ # measured page — so a class-only anchor would report a listing as ready on a # product page. Measured across 14 captures: `li.c-product__item` counts # 11-24 on every listing and **0 on every product page**, which is the clean -# separation Montblanc's `.product-tile` did not have. +# separation montblanc-scraper's `.product-tile` did not have. READY_SELECTOR_LISTING = "li.c-product__item" # A product page's readiness anchor: 1 on every product capture, 0 on every diff --git a/playwright_scraper.py b/playwright_scraper.py index 955ab45..a68e785 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -204,12 +204,12 @@ def ok(self) -> bool: # The share of rows that must carry a usable price before the run is worth # trusting as a PRICE run rather than merely as a catalogue listing. Both -# `jsonld` and `dom_range` count — a range is a real, labelled number — so -# this fires on a parsing break, not on the site's own variety. +# `tile` and `jsonld` count, so this fires on a parsing break, not on the +# site's own variety. PRICE_COVERAGE_FLOOR = 95 # A page holding less than this share of the page size is reported as thin. -# The size is the site's own `sz`, echoed back as `data-page-size`, and every +# The size is the site's fixed 24 (`product_parser.PAGE_SIZE`), and every # captured full page held exactly it — so the only legitimately short page is # the last one of a listing, which is why this can sit high without false # alarms. @@ -233,8 +233,8 @@ def _driver(page): # There is no scroll primitive here, and its absence is measured rather # than forgotten (§8: missing content is one of three things, so check # which before "fixing" it). Maison KOSÉ SERVER-RENDERS its grid: a plain - # HTTP fetch with no JavaScript at all returns the full JSON-LD ItemList - # of 24 products and 49 `data-pid` attributes. Everything the parser + # HTTP fetch with no JavaScript at all returns every `li.c-product__item` + # tile of the page (24 on a full one). Everything the parser # wants is in the document before any scrolling could happen, on every # capture, so a scroll would be ceremony that looks load-bearing. return { @@ -818,10 +818,10 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome # This site server-renders its grid, so a listing is parseable in # the FIRST response and there is nothing to wait for on a healthy - # page. Measured 2026-09-17 with no JavaScript executed at all — a - # plain HTTP fetch — /en-fi/bags/backpacks returned the full JSON-LD - # ItemList of 24 products and 49 `data-pid` attributes, and a product - # page returned its whole 16-entry ProductGroup. That is why this + # page. Measured 2026-09-18 with no JavaScript executed at all — a + # plain HTTP fetch of /site/cosmedecorte/c/c15_p3/ returned the whole + # 90,310-byte page with its tiles, and a product page carries its one + # JSON-LD `Product` block in the same way. That is why this # engine has no scroll step and no readiness pause on the happy path: # either would be ceremony that looks load-bearing. # @@ -872,7 +872,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome if not page_flow.should_retry(state): # "content" and "empty" are both final answers. An empty page is - # a CORRECT one — a hub category has no grid, and one page past + # a CORRECT one — a bogus category id is served with no tiles, and one page past # the end of a listing has no products — so retrying it would # spend the user's budget re-confirming the same right answer, # and rotating the exit would blame an address for the URL it was @@ -922,13 +922,15 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome # two have different remedies and only one of them is about the # address: # - # a page that is not this site's — the exit is scored, or - # something is intercepting. - # A different exit. - # nothing at all (the stream was reset) — the edge refused the - # REQUEST, and on this site - # that is a User-Agent - # denylist, not the address. + # a page that is not this site's — something between here + # and the site is + # intercepting, or the exit + # is scored. A different + # exit. + # nothing at all (the stream was reset) — the network path. On + # this site no refusal of + # any kind has been + # observed (README). # # Saying which one arrived is more use than a captcha hint that would # cost money for a page carrying no widget. @@ -939,13 +941,13 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome logger.error( "Maison KOSE did not serve this request — %d bytes, its own asset " "host referenced %d time(s), saved to %s. There is no widget on " - "it, so no key would help. Note what this is NOT: an ordinary " - "datacenter address is served normally by this site (measured " - "2026-09-17 from a Hetzner IP — listings, search and product " - "pages all HTTP 200, no proxy, no key), so a refusal here is " - "unusual rather than expected. Check the User-Agent first — the " - "edge refuses `curl`, `python-requests` and friends outright — " - "then try a different exit with --proxy or --cdp-endpoint. This " + "it, so no key would help. Nothing on this site has ever " + "produced this state: measured 2026-09-18 from a datacentre " + "address, a request with NO User-Agent at all is served the same " + "90,310 bytes as a browser, and 12 pages back to back were " + "12 x HTTP 200. So the first thing to check is the network path " + "— a proxy, a captive portal, or DNS — then try a different exit " + "with --proxy or --cdp-endpoint. This " "is exit 3, distinct from a genuinely empty result (exit 4).%s", len(html or ""), assets, debug_html, (f" Tried {block_retries + 1} exit(s)." if has_pool @@ -980,9 +982,9 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome # is the "detected is not blocking" rule the captcha default follows, # applied to the blocking decision instead of the spending one. But # `state != "content"` is still too wide: an EMPTY page is a correct - # answer, and a live run of a /p/ hub reported exit 3 on a 191 KB - # page the site had plainly served, because the hub's own performance - # script names `akamaihd.net` and "akamai" was in the marker list. Both + # answer, and a live run of tokopedia-scraper's /p/ hub reported + # exit 3 on a 191 KB page that site had plainly served, because its own + # performance script names `akamaihd.net` and "akamai" was in the marker list. Both # halves were wrong; the marker is gone (see # product_parser.BOT_CHALLENGE_MARKERS) and this now only refines the # REASON for a page the policy had already given up on. @@ -1045,8 +1047,8 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome "Page %d links to %d product(s) and parsed to ZERO rows. " "The site served this page — this is a failure in THIS parser, " "not an empty category and not a block. Saved to %s; the first " - "thing to check is the JSON-LD (an ItemList that was renamed or " - "dropped), then the tile markup. Reported as stop_reason " + "thing to check is the tile markup (li.c-product__item and its " + "/g/g{SKU}/ link) — a listing carries no JSON-LD. Reported as stop_reason " "'parser_found_nothing' so it cannot be read as a complete run.", page_num, links, dump) outcome.state = "parse_failed" @@ -1056,10 +1058,10 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome # gets the number rather than a threshold someone guessed. images = sum(1 for row in products if row.image_url) if images < len(products): - logger.info("Page %d: %d/%d rows carry an image. This site's own " - "ItemList sometimes names a product it renders no " - "tile for, and those entries have no image in the " - "JSON-LD either — so this is reported, not floored.", + logger.info("Page %d: %d/%d rows carry an image. This site's " + "tiles carry a /img/goods/ thumbnail on every " + "measured page (171 of 171 in the captures), so a " + "gap here points at the tile markup.", page_num, images, len(products)) for field_name in CORE_FIELDS: @@ -1087,7 +1089,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome if args.mode == "product": group = {row.variant_of for row in products if row.variant_of} - logger.info("Product: %d variant(s) of %s, %d priced.", + logger.info("Product: %d row(s) for %s, %d priced.", len(products), next(iter(group), products[0].sku), sum(1 for r in products if r.price is not None)) else: @@ -1202,8 +1204,7 @@ def scrape(args) -> int: outcomes: List[PageOutcome] = [] seen_keys = set() blocked = False - # All three modes are one row per business-at-a-location, so `sku` is the - # key for all of them. + # Both modes are one row per product, so `sku` is the key for both. dedupe_key = "sku" # Why the loop ended. "completed" means every requested page was fetched; # "no_new_products" means the listing itself ran out (also a complete @@ -1213,15 +1214,14 @@ def scrape(args) -> int: # "page_cap_reached" means the run asked for more pages than the # category holds and was clamped to the site's own count. On this site # that is a COMPLETE result in the strong sense: this site imposes no cap - # of its own, so the clamp IS the whole category — 280 results across 12 - # pages of 24, and start=288 is a served, empty grid. + # of its own, so the clamp IS the whole category — /c/c15/ states 13 + # pages, and 12 x 24 + 16 = 304 is all of Cosme Decorte. # - # Only --mode product is single-page. Both listing modes paginate - # identically — the same `start`/`sz` on the same URL — so neither may be + # Only --mode product is single-page. Both listing routes paginate — + # `_pN` on a category, `?p=N` on a tag listing — so neither may be # treated as single-page, which is the silent-success failure this family - # exists to avoid. Note that "single page" is a statement about FETCHING: - # --mode product still emits one row per variant, so a one-page run of a - # sixteen-nib pen is sixteen rows. + # exists to avoid. On this site one product page is also one row: a + # single `Product` block, never a `ProductGroup`. stop_reason = "single_page_mode" if args.mode == "product" else "completed" pool = proxy_pool_from_args(args) @@ -1235,7 +1235,7 @@ def scrape(args) -> int: if concurrency > 1: if args.mode == "product": logger.info("--concurrency is ignored in --mode product: there is " - "one page to fetch (however many variants it holds).") + "one page to fetch, and it is one row.") concurrency = 1 elif page_flow.concurrency_limit(args.cdp_endpoint) == 1: # The limit is page_flow's to state, not this engine's, so all @@ -1424,9 +1424,8 @@ def scrape(args) -> int: # where a short page hides. # # NOT "pages x rows-per-page" as a hard expectation, even though the - # page size is the run's own `sz`: the LAST page of a listing is - # legitimately short — /en-fi/writing-instruments ends 240 + 24 + 16 = - # 280 — and a threshold that fires on every healthy run teaches the + # page size is the site's fixed 24: the LAST page of a listing is + # legitimately short — /c/c15/ ends 12 x 24 + 16 = 304 — and a threshold that fires on every healthy run teaches the # reader to ignore it. What is worth warning about is a page that came # back materially THIN against its siblings, which is what a truncated # response looks like. @@ -1541,14 +1540,14 @@ def parse_args(): "asked for.") p.add_argument("--pages", type=int, default=1, help="Number of listing pages to fetch. Applies to --mode " - "category and --mode search; ignored in --mode " - "product. Page 1 prints the catalogue's own result " - "count, so a run PLANS against the site's arithmetic " - "rather than walking off the end. There is no page " - "cap on this site — asking past the last page is a " - "served, empty grid rather than an error — so a " - "request is limited only by what the category holds, " - "and the sidecar records both numbers.") + "listing; ignored in --mode product. A category " + "page prints the site's own page counter, so a run " + "PLANS against the site's arithmetic rather than " + "walking off the end — past its last page a category " + "serves that last page again. There is no page cap " + "on this site, so a request is limited only by what " + "the category holds, and the sidecar records both " + "numbers.") p.add_argument("--delay", type=float, default=2.0, help="Delay between pages, seconds") p.add_argument("--concurrency", type=int, default=1, metavar="N", help="Fetch pages through N parallel workers (default 1 — " @@ -1560,7 +1559,7 @@ def parse_args(): help="Attempts per page load before giving up (default 3). " "The pause between attempts doubles each time. A page " "that comes back EMPTY is not retried — see " - "page_flow.STATE_POLICY — because a hub page with no " + "page_flow.STATE_POLICY — because a category with no " "products on it is a correct answer, not a fault.") p.add_argument("--retry-delay", type=float, default=2.0, help="Seconds before the first page-load retry, doubling " @@ -1646,7 +1645,7 @@ def parse_args(): p.add_argument("--dump-html", default=None, metavar="PATH", help="Save the exact HTML the parser is given, on success as " "well as failure. Useful when the row count is right but " - "a column comes back empty — see TROUBLESHOOTING.md.") + "a column comes back empty — see the README's 'Traps that look like bugs'.") p.add_argument("--headless", action="store_true", default=True) p.add_argument("--headful", dest="headless", action="store_false") args = p.parse_args() diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index 733dbc1..c36ef58 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -98,13 +98,13 @@ # — 100% coverage of the wrong value, which is exactly the failure §10 means # by "assert VALUES on real fixtures, not coverage". -# The share of rows that must carry a usable price. Both `jsonld` and -# `dom_range` count, so this fires on a parsing break rather than on the +# The share of rows that must carry a usable price. Both `tile` and +# `jsonld` count, so this fires on a parsing break rather than on the # site's own variety. PRICE_COVERAGE_FLOOR = 95 # A page holding less than this share of the page size is reported as thin. -# The size is the site's own `sz`, so the only legitimately short page is the +# The size is the site's fixed 24, so the only legitimately short page is the # last one of a listing. THIN_PAGE_SHARE = 0.6 @@ -326,8 +326,8 @@ def open(self): return self # --lang really applies the locale rather than accepting the flag - # and ignoring it. It does NOT decide which market is read — that - # searched; that is find_country in the URL. + # and ignoring it. It does NOT decide which market is read: this + # site is one Japanese market whatever the browser claims. launch_args = ["--no-sandbox", "--disable-dev-shm-usage", f"--lang={BROWSER_LOCALE}"] launch_kwargs = {} @@ -773,7 +773,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome if not page_flow.should_retry(state): # "content" and "empty" are both final answers. An empty page is - # a CORRECT one — a hub category has no grid — so retrying it + # a CORRECT one — a bogus category id is served with no tiles — so retrying it # would re-confirm the same right answer, and rotating the exit # would blame an address for the URL it was given. break @@ -816,7 +816,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome "limits. This is exit 3, distinct from a genuinely empty result " "(exit 4).", len(html or ""), references_own_assets(html or ""), debug_html) - outcome.blocked_by = "cloudflare (hard block)" if html else "no-response" + outcome.blocked_by = "edge refusal" if html else "no-response" outcome.final_url = page.url return outcome @@ -832,9 +832,9 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome dump_path, len(html)) # Only for a state page_flow already counts as BLOCKED. An EMPTY - # page is a correct answer, and a live run of a /p/ hub - # reported exit 3 on a page the site had plainly served because the - # hub's own performance script names `akamaihd.net`. Mirrors + # page is a correct answer, and a live run of tokopedia-scraper's + # /p/ hub reported exit 3 on a page that site had plainly served + # because its own performance script names `akamaihd.net`. Mirrors # playwright_scraper exactly. vendor = (detect_bot_challenge(html, url=page.url) if page_flow.counts_as_blocked(state) else None) @@ -890,8 +890,8 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome "Page %d links to %d product(s) and parsed to ZERO rows. " "The site served this page — this is a failure in THIS parser, " "not an empty category and not a block. Saved to %s; the first " - "thing to check is the JSON-LD (an ItemList that was renamed or " - "dropped), then the tile markup. Reported as stop_reason " + "thing to check is the tile markup (li.c-product__item and its " + "/g/g{SKU}/ link) — a listing carries no JSON-LD. Reported as stop_reason " "'parser_found_nothing' so it cannot be read as a complete run.", page_num, links, dump) outcome.state = "parse_failed" @@ -899,10 +899,10 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome if products: images = sum(1 for row in products if row.image_url) if images < len(products): - logger.info("Page %d: %d/%d rows carry an image. This site's own " - "ItemList sometimes names a product it renders no " - "tile for, and those entries have no image in the " - "JSON-LD either — so this is reported, not floored.", + logger.info("Page %d: %d/%d rows carry an image. This site's " + "tiles carry a /img/goods/ thumbnail on every " + "measured page (171 of 171 in the captures), so a " + "gap here points at the tile markup.", page_num, images, len(products)) for field_name in CORE_FIELDS: @@ -929,7 +929,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome if args.mode == "product": group = {row.variant_of for row in products if row.variant_of} - logger.info("Product: %d variant(s) of %s, %d priced.", + logger.info("Product: %d row(s) for %s, %d priced.", len(products), next(iter(group), products[0].sku), priced) else: @@ -1130,7 +1130,7 @@ def scrape(args) -> int: "route": route_of(final_url or args.url), "tags": tags_from_url(final_url or args.url)} # No `capped_by_site` / `reachable_max`: this site imposes no page - # cap (measured — 280 results, 240 + 24 + 16 = 280), so + # cap (measured — /c/c15/ states 13 pages, 12 x 24 + 16 = 304), so # `pages_available` already says everything they would, and # pages x page_size would OVERSTATE a short last page. @@ -1184,14 +1184,14 @@ def parse_args(): "asked for.") p.add_argument("--pages", type=int, default=1, help="Number of listing pages to fetch. Applies to --mode " - "category and --mode search; ignored in --mode " - "product. Page 1 prints the catalogue's own result " - "count, so a run PLANS against the site's arithmetic " - "rather than walking off the end. There is no page " - "cap on this site — asking past the last page is a " - "served, empty grid rather than an error — so a " - "request is limited only by what the category holds, " - "and the sidecar records both numbers.") + "listing; ignored in --mode product. A category " + "page prints the site's own page counter, so a run " + "PLANS against the site's arithmetic rather than " + "walking off the end — past its last page a category " + "serves that last page again. There is no page cap " + "on this site, so a request is limited only by what " + "the category holds, and the sidecar records both " + "numbers.") p.add_argument("--delay", type=float, default=2.0, help="Delay between pages, seconds") p.add_argument("--concurrency", type=int, default=1, metavar="N", help="Fetch pages through N parallel workers (default 1 — " @@ -1203,7 +1203,7 @@ def parse_args(): help="Attempts per page load before giving up (default 3). " "The pause between attempts doubles each time. A page " "that comes back EMPTY is not retried — see " - "page_flow.STATE_POLICY — because a hub page with no " + "page_flow.STATE_POLICY — because a category with no " "products on it is a correct answer, not a fault.") p.add_argument("--retry-delay", type=float, default=2.0, help="Seconds before the first page-load retry, doubling " @@ -1289,7 +1289,7 @@ def parse_args(): p.add_argument("--dump-html", default=None, metavar="PATH", help="Save the exact HTML the parser is given, on success as " "well as failure. Useful when the row count is right but " - "a column comes back empty — see TROUBLESHOOTING.md.") + "a column comes back empty — see the README's 'Traps that look like bugs'.") p.add_argument("--chromium-path", default=None, metavar="PATH", help="Use this Chromium/Chrome binary instead of the one " "pyppeteer downloads on first run. Useful on a " diff --git a/scraper_api_client.py b/scraper_api_client.py index 3d12a19..5f21612 100644 --- a/scraper_api_client.py +++ b/scraper_api_client.py @@ -327,8 +327,8 @@ def parse_args(): p.add_argument("--mode", choices=["listing", "product"], default="listing", help="Same two modes as the browser engines, and the " - "same default. product emits one row per variant out " - "of the page's ProductGroup.") + "same default. product emits ONE row: a Kose product " + "page publishes a single Product block.") p.add_argument("--category", default=None, help="Label to tag output rows with. Defaults to the category segment of the URL, so the column is never empty just because the flag was omitted.") p.add_argument("--format", choices=["json", "csv", "both"], default="both") p.add_argument("--out", default="kose_products_scraperapi", help="Output file prefix") diff --git a/selenium_scraper.py b/selenium_scraper.py index e285059..e8b4823 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -96,13 +96,13 @@ # — 100% coverage of the wrong value, which is exactly the failure §10 means # by "assert VALUES on real fixtures, not coverage". -# The share of rows that must carry a usable price. Both `jsonld` and -# `dom_range` count, so this fires on a parsing break rather than on the +# The share of rows that must carry a usable price. Both `tile` and +# `jsonld` count, so this fires on a parsing break rather than on the # site's own variety. PRICE_COVERAGE_FLOOR = 95 # A page holding less than this share of the page size is reported as thin. -# The size is the site's own `sz`, so the only legitimately short page is the +# The size is the site's fixed 24, so the only legitimately short page is the # last one of a listing. THIN_PAGE_SHARE = 0.6 @@ -239,9 +239,9 @@ def open(self): options.add_argument("--disable-blink-features=AutomationControlled") # Flag parity with the Playwright engine, and really applied rather # than accepted and ignored: Chrome takes the locale as --lang. It - # does NOT decide which market is read — that is the locale in - # find_country in the URL — so this only affects what the browser - # claims about itself. + # does NOT decide which market is read — this site is one Japanese + # market — so this only affects what the browser claims about + # itself. options.add_argument(f"--lang={BROWSER_LOCALE}") if self.pool: @@ -673,7 +673,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome if not page_flow.should_retry(state): # "content" and "empty" are both final answers. An empty page is - # a CORRECT one — a hub category has no grid — so retrying it + # a CORRECT one — a bogus category id is served with no tiles — so retrying it # would re-confirm the same right answer, and rotating the exit # would blame an address for the URL it was given. break @@ -716,7 +716,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome "limits. This is exit 3, distinct from a genuinely empty result " "(exit 4).", len(html or ""), references_own_assets(html or ""), debug_html) - outcome.blocked_by = "cloudflare (hard block)" if html else "no-response" + outcome.blocked_by = "edge refusal" if html else "no-response" outcome.final_url = d["current_url"]() return outcome @@ -737,9 +737,10 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome # --cdp-endpoint the Scraping Browser's own auto-solve extension injects # such markers into every page it loads. # Only for a state page_flow already counts as BLOCKED. An EMPTY page is - # a correct answer, and a live run of a /p/ hub reported exit 3 on - # a page the site had plainly served because the hub's own performance - # script names `akamaihd.net`. Mirrors playwright_scraper exactly. + # a correct answer, and a live run of tokopedia-scraper's /p/ hub + # reported exit 3 on a page that site had plainly served because its own + # performance script names `akamaihd.net`. Mirrors playwright_scraper + # exactly. vendor = (detect_bot_challenge(html, url=d["current_url"]()) if page_flow.counts_as_blocked(state) else None) if vendor: @@ -794,8 +795,8 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome "Page %d links to %d product(s) and parsed to ZERO rows. " "The site served this page — this is a failure in THIS parser, " "not an empty category and not a block. Saved to %s; the first " - "thing to check is the JSON-LD (an ItemList that was renamed or " - "dropped), then the tile markup. Reported as stop_reason " + "thing to check is the tile markup (li.c-product__item and its " + "/g/g{SKU}/ link) — a listing carries no JSON-LD. Reported as stop_reason " "'parser_found_nothing' so it cannot be read as a complete run.", page_num, links, dump) outcome.state = "parse_failed" @@ -803,10 +804,10 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome if products: images = sum(1 for row in products if row.image_url) if images < len(products): - logger.info("Page %d: %d/%d rows carry an image. This site's own " - "ItemList sometimes names a product it renders no " - "tile for, and those entries have no image in the " - "JSON-LD either — so this is reported, not floored.", + logger.info("Page %d: %d/%d rows carry an image. This site's " + "tiles carry a /img/goods/ thumbnail on every " + "measured page (171 of 171 in the captures), so a " + "gap here points at the tile markup.", page_num, images, len(products)) for field_name in CORE_FIELDS: @@ -833,7 +834,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome if args.mode == "product": group = {row.variant_of for row in products if row.variant_of} - logger.info("Product: %d variant(s) of %s, %d priced.", + logger.info("Product: %d row(s) for %s, %d priced.", len(products), next(iter(group), products[0].sku), priced) else: @@ -1026,7 +1027,7 @@ def scrape(args) -> int: "route": route_of(final_url or args.url), "tags": tags_from_url(final_url or args.url)} # No `capped_by_site` / `reachable_max`: this site imposes no page - # cap (measured — 280 results, 240 + 24 + 16 = 280), so + # cap (measured — /c/c15/ states 13 pages, 12 x 24 + 16 = 304), so # `pages_available` already says everything they would, and # pages x page_size would OVERSTATE a short last page. @@ -1080,14 +1081,14 @@ def parse_args(): "asked for.") p.add_argument("--pages", type=int, default=1, help="Number of listing pages to fetch. Applies to --mode " - "category and --mode search; ignored in --mode " - "product. Page 1 prints the catalogue's own result " - "count, so a run PLANS against the site's arithmetic " - "rather than walking off the end. There is no page " - "cap on this site — asking past the last page is a " - "served, empty grid rather than an error — so a " - "request is limited only by what the category holds, " - "and the sidecar records both numbers.") + "listing; ignored in --mode product. A category " + "page prints the site's own page counter, so a run " + "PLANS against the site's arithmetic rather than " + "walking off the end — past its last page a category " + "serves that last page again. There is no page cap " + "on this site, so a request is limited only by what " + "the category holds, and the sidecar records both " + "numbers.") p.add_argument("--delay", type=float, default=2.0, help="Delay between pages, seconds") p.add_argument("--concurrency", type=int, default=1, metavar="N", help="Fetch pages through N parallel workers (default 1 — " @@ -1099,7 +1100,7 @@ def parse_args(): help="Attempts per page load before giving up (default 3). " "The pause between attempts doubles each time. A page " "that comes back EMPTY is not retried — see " - "page_flow.STATE_POLICY — because a hub page with no " + "page_flow.STATE_POLICY — because a category with no " "products on it is a correct answer, not a fault.") p.add_argument("--retry-delay", type=float, default=2.0, help="Seconds before the first page-load retry, doubling " @@ -1185,7 +1186,7 @@ def parse_args(): p.add_argument("--dump-html", default=None, metavar="PATH", help="Save the exact HTML the parser is given, on success as " "well as failure. Useful when the row count is right but " - "a column comes back empty — see TROUBLESHOOTING.md.") + "a column comes back empty — see the README's 'Traps that look like bugs'.") p.add_argument("--headless", action="store_true", default=True) p.add_argument("--headful", dest="headless", action="store_false") args = p.parse_args() diff --git a/smoke_test.py b/smoke_test.py index 8693c36..5b22eab 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -1357,7 +1357,7 @@ def check_fingerprint_kwargs_are_ones_the_driver_accepts(): def check_engines_do_not_evaluate_a_string_in_the_browser(): """§18: a site whose CSP omits `unsafe-eval` kills wait_for_function with - an EvalError and takes the run down with exit 1. BBB has not been + an EvalError and takes the run down with exit 1. Maison KOSÉ has not been measured for that, and the cheap habit costs nothing where it would have been allowed.""" for module in ENGINES: @@ -1460,7 +1460,7 @@ def check_a_run_that_finds_nothing_writes_nothing(): equal("--allow-empty WRITES the empty file...", json.load(open(prefix + ".json", encoding="utf-8")), []) # ...and still reports exit 4. Pinned deliberately (§10: pin a known - # behaviour rather than half-guarding it): "zero businesses" is true + # behaviour rather than half-guarding it): "zero products" is true # whether or not the file was written, and a caller that wanted the # file still wants to know the result was empty. equal("...and still reports exit 4, because it IS empty", code, 4) @@ -1837,16 +1837,14 @@ def check_engine_flag_sets(): def check_banned_and_removed_flags(): """Scoped to the ENGINES. - `--country` is absent here — the flag CLAUDE.md §10 bans outright — and - `--locale` takes its place, because on Montblanc the market really is a - property of the URL rather than of the browser. + `--country` is absent here — the flag CLAUDE.md §10 bans outright, + because it could disagree with the URL. `--locale` is absent too: + maison.kose.co.jp is one market (see EXTRA_FLAGS above). - That makes the ban's REASON bite harder than usual: the locale is a path - segment, so a `--locale` that disagreed with a `--url` would silently - read a different market, and on this site a different market is a - different PRICE (EUR 2000 on en-fi against EUR 1900 on de-de for one - backpack). So the flag is refused alongside --url rather than merged, - and this check pins that refusal exists in every engine. + The same reasoning applies to the flags this repo DOES have: `--tags` + and `--category` each build an address, so a `--tags` given alongside + `--url` is refused rather than merged, and `--category` takes only a + cNN id. This check pins both refusals in every engine. """ for module in ENGINES: path = os.path.join(HERE, module + ".py") @@ -1983,9 +1981,9 @@ def check_captcha_capability_claims_match_the_code(): # The pairing that matters ON THIS SITE. # # The sibling repo pins "the README must say TurnstileTaskProxyless is - # not built here", because BBB renders Cloudflare Managed Challenges and - # a reader could reasonably expect a key to clear one. Montblanc renders - # NO challenge at all, so the equivalent hazard is the opposite one: this + # not built here", because that site renders Cloudflare Managed + # Challenges and a reader could reasonably expect a key to clear one. + # Maison KOSÉ renders NO challenge at all, so the equivalent hazard is the opposite one: this # README's central claim is that no key is needed, and a claim like that # rots the moment the site switches a bot manager on. # @@ -2152,7 +2150,7 @@ def check_parser_found_nothing_is_not_a_complete_run(): list(COMPLETE_STOP_REASONS)) meta = run_meta(status="failed", stop_reason="parser_found_nothing", pages_requested=1, pages_completed=0, pages_failed=[1], - products=0, mode="category", source="montblanc.com", + products=0, mode="listing", source="maison.kose.co.jp", start_url="u", final_url="u") equal("parse-failure: the sidecar carries the reason by name", meta["stop_reason"], "parser_found_nothing") @@ -2173,23 +2171,27 @@ def check_parser_found_nothing_is_not_a_complete_run(): def check_sidecar_shape(): from output_writer import run_meta + # /c/c15/ states 13 pages and 12 x 24 + 16 = 304 products is all of + # Cosme Decorte (measured 2026-09-18, see output_writer). meta = run_meta(status="complete", stop_reason="page_cap_reached", - pages_requested=50, pages_completed=12, pages_failed=[], - products=280, mode="category", source="montblanc.com", - start_url="https://www.montblanc.com/en-fi/writing-instruments", - final_url="https://www.montblanc.com/en-fi/writing-instruments?start=264&sz=24", - extra={"total_results": 280, "pages_available": 12, - "page_size": 24, "locale": "en-fi", - "sort_requested": "price-asc", - "site_default_sort": "recommended"}) + pages_requested=50, pages_completed=13, pages_failed=[], + products=304, mode="listing", source="maison.kose.co.jp", + start_url="https://maison.kose.co.jp/site/cosmedecorte/c/c15/", + final_url="https://maison.kose.co.jp/site/cosmedecorte/c/c15_p13/", + extra={"pages_available": 13, "page_size": 24, + "canonical_url": "https://maison.kose.co.jp/site/cosmedecorte/c/c15/", + "route": "category"}) for key in ("status", "stop_reason", "pages_requested", "pages_completed", "pages_failed", "mode", "source"): check("the sidecar records %r" % key, key in meta) - equal("the sidecar carries the site's own total", meta["total_results"], 280) - equal("...and the market the prices belong to", meta["locale"], "en-fi") - equal("...and which ordering was asked for", meta["sort_requested"], "price-asc") - equal("...and what a visitor would have got instead", - meta["site_default_sort"], "recommended") + equal("the sidecar carries the site's own page count", + meta["pages_available"], 13) + equal("...and the page size it applied", meta["page_size"], 24) + equal("...and the page the SITE says was read", meta["canonical_url"], + "https://maison.kose.co.jp/site/cosmedecorte/c/c15/") + equal("...and which listing route it was", meta["route"], "category") + # No `locale` and no `sort_requested`: one market, no ordering control. + check("no locale on a one-market site", "locale" not in meta, sorted(meta)) equal("pages_failed is a LIST of numbers, not a count", isinstance(meta["pages_failed"], list), True) @@ -2205,7 +2207,7 @@ def check_row_schema(): equal("the family prefix is byte-identical and in order (§9)", names[:5], ["source", "scraped_at", "url", "sku", "title"]) - # The commerce columns this site DOES have. Montblanc is a shop, so the + # The commerce columns this site DOES have. Maison KOSÉ is a shop, so the # family's price fields are present rather than dropped. for present in ("brand", "price", "currency", "in_stock", "image_url", "category", "price_source"): @@ -2242,7 +2244,7 @@ def check_fixtures_carry_no_session_material(): of that granted anything — the session material was anonymous and expired — and it still did not belong in a public repo. - Montblanc is the easy case: it publishes products, not people, and no + Maison KOSÉ is the easy case: it publishes products, not people, and no capture here carries a name, a review or a login. So this guards the SHAPE rather than any literal, which is the half that keeps working when the next capture is taken (§10: guard with PATTERNS, not the old