Remove donor-repo leftovers (Montblanc, BBB, Etsy)

.dockerignore

@@ -11,7 +11,8 @@ __pycache__/
 .venv/
 venv/
 env/
-bbb_businesses.*
+kose_products.*
+kose_products_*.*
 *_debug.html
 *_debug.png
 live_results/

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -10,26 +10,23 @@ body:
         mask credentials in their own logs, but raw HTML dumps and the Scraper
         API's `x-debug` header do not. Replace them with `***`.
 
-        Two things answer most reports faster than we can:
+        A few things answer most reports faster than we can:
 
-        - **Everything exit 3?** Check the exit address first. Etsy's
-          DataDome answers every datacentre and VPN IP with HTTP 403, so a run
-          from a VPS or a CI runner is blocked whatever else is set. If the
-          log says `t=bv`, a captcha key does not help — that challenge is
-          not solvable. The Scraping Browser API (`--cdp-endpoint`) is the
-          path that works; if a profile keeps refusing after three retries,
-          try a different `pid`.
-        - **Zero products?** You may be on a page that has none — a taxonomy
-          hub, or one page past the end of a listing. It answers 200 with
-          nothing on it, which is exit 4 and a correct answer. See
-          [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md).
-        - **`rating` empty on every row?** That is expected on a listing run:
-          Etsy publishes no per-listing rating there. The stars you see on a
-          tile are the shop's and are in `shop_rating`. See the README.
-        - **Wrong currency or language?** Etsy decides both from your exit
-          IP, not from the URL — a request for `/fr/…` from a German exit is
-          redirected to `/de/…`. The row's `url` says which storefront
-          actually answered.
+        - **Everything exit 3?** Maison KOSÉ has served every page tested,
+          datacentre addresses and a bare `python-requests` User-Agent
+          included, with no captcha of any vendor (see the README). A
+          block here is new — the log and a `--dump-html` capture are the
+          most useful things you can attach.
+        - **Zero products?** A bogus category id or goods code answers HTTP
+          200 with the site's chrome and no tiles, which is exit 4 and a
+          correct answer. A wrong PATH is a real 404, and the run stops on
+          it rather than retrying. See "Traps that look like bugs" in the
+          README.
+        - **`rating` or `original_price` missing?** Expected: this site
+          publishes neither, and the README says what was measured.
+        - **Every page repeats?** A category walked past its end serves its
+          LAST page again; the run plans against the site's own page
+          counter and stops on "this page added no new sku".
 
   - type: dropdown
     id: engine
@@ -77,8 +74,8 @@ body:
       render: shell
       placeholder: |
         python3 playwright_scraper.py \
-          --url "https://www.etsy.com/search?q=handmade+mug" \
-          --pages 3 --out grills
+          --url "https://maison.kose.co.jp/site/cosmedecorte/c/c15/" \
+          --pages 3
     validations:
       required: true
 
@@ -98,7 +95,7 @@ body:
     attributes:
       label: What you expected instead
       placeholder: >-
-        96 products, as the README says a category page yields.
+        72 products, as the README says a 3-page run of /c/c15/ yields.
     validations:
       required: true
 

.github/ISSUE_TEMPLATE/question.yml

@@ -7,8 +7,7 @@ body:
       value: |
         Worth a look first, in this order:
 
-        - [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md) — ordered by how
-          often each thing actually happens
+        - The README's "Traps that look like bugs" section
         - The README section on the 2Captcha product you are using
         - `python3 env_config.py` — reports what credentials are configured,
           without printing them

.github/ISSUE_TEMPLATE/site_changed.yml

@@ -1,16 +1,18 @@
 name: Site changed / selectors broke
-description: Etsy changed something and the parser no longer matches.
+description: Maison KOSÉ changed something and the parser no longer matches.
 labels: ["site-change"]
 body:
   - type: markdown
     attributes:
       value: |
         This is the most common way a scraper stops working, and it is worth its
         own report because the fix is different from a code bug: something on
-        etsy.com moved.
+        maison.kose.co.jp moved.
 
-        Useful to know before filing: the parser tries **JSON-LD first**, then a
-        CSS + URL-pattern fallback. Which one broke narrows the fix a lot.
+        Useful to know before filing: a LISTING page carries no JSON-LD, so
+        listings are read from the tile markup (`li.c-product__item` and a
+        `/g/g{SKU}/` link); a PRODUCT page is read from its one JSON-LD
+        `Product` block. Which route broke narrows the fix a lot.
 
   - type: dropdown
     id: what_broke
@@ -19,8 +21,7 @@ body:
       options:
         - Products (0 rows on a page that visibly has products)
         - A specific field is null or wrong (price, sku, image, URL)
-        - original_price or lowest_price_30d empty out (the tile overlay)
-        - DOM price confirmation dropped below 90% in the run log
+        - tax_rate or in_stock reads wrong
         - Pagination stops early or repeats a page
         - The captcha detector finds nothing on a page that has one
         - Something else
@@ -31,8 +32,8 @@ body:
     id: url
     attributes:
       label: URL
-      description: The exact listing URL, and the country your exit IP was in.
-      placeholder: "https://www.etsy.com/search?q=handmade+mug (locale: /de/, exit IP: DE residential, Scraping Browser pid: etsy1)"
+      description: The exact listing or product URL, and where your exit IP was.
+      placeholder: "https://maison.kose.co.jp/site/cosmedecorte/c/c15/ (exit IP: DE datacentre)"
     validations:
       required: true
 
@@ -43,9 +44,9 @@ body:
       description: |
         Whichever of these you can get. A capture beats a description.
 
-        - JSON-LD as the page actually serves it:
+        - For a product page, its JSON-LD as the page actually serves it:
           `python3 -c "import json,sys,re;h=open('dump.html').read();print([m[:400] for m in re.findall(r'<script type=\"application/ld\\+json\">(.*?)</script>',h,re.S)][:2])"`
-        - Or open devtools and paste one `Product` node from the `ItemList`.
+        - For a listing: one `li.c-product__item` tile's outer HTML.
         - For a field problem: the value you got, and the value on the page.
       render: text
     validations:

CHANGELOG.md

@@ -24,6 +24,46 @@ existing user would notice, the release notes lead with it.
   this client. `http_code` (an int) is read now, with `status` kept as a
   fallback only when it is an int.
 
+Leftovers from the sibling repos this one was bootstrapped from
+(montblanc-scraper, bbb-scraper, etsy-scraper), which described those sites
+as if they were this one.
+
+- **User-visible strings.** `--pages` help named a `--mode category` and
+  `--mode search` this repo does not have and said a category past its end is
+  "a served, empty grid" (here it re-serves its last page). The Playwright
+  blocked-page error said this site's edge "refuses `curl`, `python-requests`
+  and friends outright", the opposite of what the README measured; it now
+  matches the other two engines. The parse-failure error pointed at a JSON-LD
+  `ItemList` a listing here does not carry; the image-coverage log line
+  described that same `ItemList`; `--mode product` logs and `--help` spoke of
+  variants and a `ProductGroup`; `--retries` help spoke of hub pages;
+  `--dump-html` help pointed at a `TROUBLESHOOTING.md` that does not exist.
+- **Engines agree on the refusal reason.** pyppeteer and Selenium recorded a
+  refusal as `blocked_cloudflare (hard block)`, a vendor never observed here,
+  while Playwright recorded `blocked_edge refusal`. All three now say
+  `edge refusal`.
+- **`diff_runs.py` tracked Montblanc's columns** (`collection`,
+  `sub_collection`, `color`, `size`, `special_edition`, `base_sku`), none of
+  which exist in this row model. It now tracks this site's own `tax_rate` and
+  `subcategory`, and treats `subcategory` / `variant_of` as product-mode-only.
+- **Issue templates** were Etsy's (DataDome, `shop_rating`, an etsy.com
+  example URL). Rewritten from this repo's README.
+- **Donor prose in core modules and comments:** the BBB Turnstile paragraphs
+  in `captcha_solver.py`, Montblanc URLs and counts in the engines, BBB's
+  `find_country` in two locale comments, and an unnamed tokopedia hub
+  anecdote now credited to tokopedia-scraper.
+- **Test data:** `smoke_test.py` built its sidecar fixtures from a
+  montblanc.com run; they now use this site's measured `/c/c15/` run (13
+  pages, 304 products, `mode="listing"`) and assert the sidecar fields this
+  engine actually writes.
+- **`.dockerignore`** listed `bbb_businesses.*`; it now lists this repo's
+  `kose_products.*`. The Dockerfile example wrote to `writing-instruments`.
+
+- `SECURITY.md` said this project has no releases or version tags; it has
+  both. "Supported versions" now names the latest release and `main`.
+- `captcha_solver.py`'s docstring pointed at a "No DataDome solver" section
+  that does not exist in this repo (it came with the copied core). Removed.
+
 ## [0.1.1] — 2026-09-18
 
 A pass back over CLAUDE.md before calling the repo finished. Everything here

Dockerfile

@@ -5,7 +5,7 @@
 #   docker build -t kose-scraper .
 #   docker run --rm -v "$PWD/out:/out" kose-scraper \
 #     --url "https://maison.kose.co.jp/site/cosmedecorte/c/c15/" \
-#     --pages 3 --out /out/writing-instruments
+#     --pages 3 --out /out/kose_products
 #
 # Pass --proxy/--twocaptcha-key the same way as running locally, or mount a
 # .env at /app/.env — nothing here bakes in a credential.

SECURITY.md

@@ -79,9 +79,9 @@ Not because these do not matter, but because they belong somewhere else:
 
 ## Supported versions
 
-`main` only. This project has no releases or version tags; fixes land on `main`
-and you update by pulling. If you are running an old clone, update before
-reporting.
+The latest release and `main`. Fixes land on `main` first and ship in the
+next tagged release (see the Releases page). If you are running an old clone,
+update before reporting.
 
 ## If you have leaked a key
 

captcha_solver.py

@@ -6,13 +6,13 @@
 Detection runs after EVERY page navigation in the main loop of all three
 scrapers, regardless of what URL was requested (category hub, product page,
 sign-in, checkout, anything) — this is deliberate, not scoped to any one
-page. If BBB renders a reCAPTCHA challenge anywhere — account, sign-in and
-review-submission flows are the usual places — this fires.
+page. If Maison KOSÉ ever renders a reCAPTCHA challenge anywhere — account,
+sign-in and checkout flows are the usual places — this fires.
 
-**BBB's own gate is Cloudflare Turnstile, not reCAPTCHA.** Measured
-2026-09-16: a Managed Challenge carries `cf_chl_opt` and a Turnstile widget,
-and the pages this scraper reads carry no reCAPTCHA at all rather than
-an interstitial. So this module is a contingency, not part of the happy path
+**No challenge has been observed on this site at all.** Measured
+2026-09-18 (see the README): zero captcha markers of any vendor across 13
+captures — listings, product pages, the site's own 404 and an exhausted facet
+alike. So this module is a contingency, not part of the happy path
 — a bot manager can be switched on between deploys, and a scraper that
 cannot name what stopped it is much harder to fix.
 
@@ -23,8 +23,6 @@
 purchase, and `page_flow.STATE_POLICY` — not this file — decides which state
 is worth money at all.
 
-There is deliberately no DataDome path here. See "No DataDome solver" below.
-
 Flow:
   1. Both detectors run and are reconciled (see reconcile_detections) to decide
      the variant: v3, v2-invisible or v2-checkbox. The parameters differ per
@@ -707,26 +705,9 @@ class CaptchaUnsolvable(RuntimeError):
 # ===========================================================================
 # The sibling repo in this family carries a whole second solver for its
 # site's OWN first-party image captcha ("Enter the characters you see below",
-# a JPEG of distorted text and a GET form). Roughly 190 lines of it, and none
-# of it is ported here, because BBB has no such page.
-#
-# What BBB does instead is refuse in two shapes, and only ONE of them is
-# something a solver can answer. Measured 2026-09-16:
-#
-#   "Just a moment..."          a Cloudflare Managed Challenge, ~15 KB, with
-#                               `cf_chl_opt`, `__cf_chl` and a Turnstile
-#                               widget on it. This IS a test, and it is what
-#                               the machinery below is for.
-#   "You have been blocked |    a refusal, ~12.5 KB, with no widget, no
-#    Better Business Bureau(R)"  sitekey and no challenge of any kind. There
-#                               is nothing for a solver to answer.
-#
-# Both answer HTTP 403 and both wear BBB's own branding in the title, so they
-# are told apart structurally rather than by the title —
-# product_parser.detect_page_state reports the first as "challenge" and the
-# second as "blocked", and page_flow.STATE_POLICY spends on the first and
-# never on the second. From a datacenter address it is the second that
-# arrives, and the answer to it is a different exit rather than a purchase.
+# a JPEG of distorted text and a GET form). None of it is ported here,
+# because Maison KOSÉ has no such page: 13 captures measured 2026-09-18
+# carried no challenge of any kind (see the README).
 #
 # The reCAPTCHA / hCaptcha / Turnstile machinery above IS kept, and that is a
 # deliberate asymmetry rather than an inconsistency. Detection stays broad

diff_runs.py

@@ -15,7 +15,7 @@
                product's own taxonomy
     source_changed
                the two rows were read a different way — a listing row and a
-               `--mode product` variant row populate different columns — so
+               `--mode product` row populate different columns — so
                the difference is about our two snapshots rather than about
                the product, and `--fail-on-change` ignores it (§8)
 
@@ -66,41 +66,39 @@
 # compared alongside `price_source` so a number that moved because it was
 # read from a different NODE is not reported as a price change (see below).
 #
-# `in_stock` earns its place because it genuinely varies here — measured over
-# 384 tiles, `available` on 377 and `soldout` on 7 — unlike the JSON-LD's own
-# availability, which was InStock on 192 of 192 and would have made a
-# constant column look like a monitored one.
+# `in_stock` earns its place because it genuinely varies here — 19 of 171
+# measured tiles say 現在購入頂けません or 予約受付終了 (README).
 #
-# The taxonomy fields are tracked because a product moving collection, or
-# gaining a special-edition flag, is a real editorial change worth seeing.
+# `tax_rate` is tracked because a product moving between Japan's 10% and 8%
+# rates changes what its tax-inclusive price means. The taxonomy fields are
+# tracked because a product moving category is a real editorial change
+# worth seeing.
 #
 # Deliberately NOT tracked: `position` and `page` (a property of the slice,
-# not of the product), `image_url` (the URL carries a content hash that
-# changes when the site re-encodes an unchanged photo), and `title`, which is
-# printed on every line anyway and whose whitespace the site is inconsistent
-# about.
+# not of the product), `image_url`, `badges` (the site's merchandising
+# flags, which describe a campaign rather than the product), and `title`, which is printed on every
+# line anyway.
 TRACKED_FIELDS = (
     # the point of the exercise
-    "price", "currency", "in_stock",
+    "price", "currency", "in_stock", "tax_rate",
     # what the product IS
-    "category", "collection", "sub_collection", "color", "size",
-    "special_edition", "brand",
-    # identity, in case a variant is re-parented
-    "base_sku", "variant_of",
+    "category", "subcategory", "brand",
+    # identity, in case a size or shade is re-parented
+    "variant_of",
 )
 
 # The subset that ONLY one kind of run populates.
 #
-# A `--mode product` variant row has no tile behind it, so it leaves the
-# tile-sourced columns null; a listing row has no `variant_of`. Diffing a
-# listing run against a product run would report each of these as a change on
-# every row, and none of it would be about the product. When the two rows
-# disagree on `mode`, these are reported as `source_changed` rather than as
-# changes — this family's rule (§8: a difference that comes with a
-# provenance difference says something about our own two snapshots, not about
-# the site).
+# A `--mode product` row has no tile behind it and a listing row has no
+# product page behind it: `subcategory` and `variant_of` are filled only from
+# a product page. Diffing a listing run against a product run would report
+# each of these as a change on every row, and none of it would be about the
+# product. When the two rows disagree on `mode`, these are reported as
+# `source_changed` rather than as changes — this family's rule (§8: a
+# difference that comes with a provenance difference says something about our
+# own two snapshots, not about the site).
 PROFILE_ONLY_FIELDS = (
-    "collection", "sub_collection", "special_edition", "variant_of",
+    "subcategory", "variant_of",
 )
 
 # Kept under a second name because that is what this repo's row model calls
@@ -152,7 +150,7 @@ def diff_products(old: List[dict], new: List[dict]) -> dict:
 
         # A row whose `mode` differs between runs is not comparable on the
         # mode-specific columns: a listing row fills the tile-sourced ones
-        # and leaves `variant_of` null, a `--mode product` variant row does
+        # and leaves `variant_of` null, a `--mode product` row does
         # the opposite. Every one of them would read as a change and none of
         # it would be about the product. Reporting it as a change would be a
         # false alarm about the site; the other columns still compare fine.
@@ -205,7 +203,7 @@ def _print_summary(result: dict) -> None:
                            for f, v in c["changes"].items())
         print(f"  ? {c['sku']}  {c['title']}  {deltas}  "
               f"[mode {src['old']!r} -> {src['new']!r}: a listing row and a "
-              f"variant row fill different columns, so this is not a change "
+              f"product row fill different columns, so this is not a change "
               f"in the product]")
     unmatchable = result["unmatchable_old"] + result["unmatchable_new"]
     if unmatchable: