From 8dc7c5f289400a6cae0e09ee80a8785e746e6d4d Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Wed, 23 Sep 2026 13:54:49 +0000 Subject: [PATCH 1/3] Remove donor leftovers: twin engines' --mode help, access badge, templates - selenium/puppeteer: --mode help and docstring described MediaMarkt's "/product/ page with brand, EAN"; now mirror Playwright's listing / product (/listing/) / shop text. - README: access badge "needs a residential exit" -> "needs Scraping Browser API" (README measures residential alone as refused, t=bv). - bug_report.yml: --out grills -> mugs; Farfetch's "96 products" -> the README's 184-row run. - site_changed.yml: JSON-LD-first / lowest_price_30d / 90% options replaced with data-listing-id anchor and per-page-kind floors. - SECURITY.md: Akamai -> DataDome. output_writer.py: MediaMarkt tv-audio URL example -> a taxonomy hub. CONTRIBUTING: "the registration form" generalised. Co-Authored-By: Claude Opus 5.5 --- .github/ISSUE_TEMPLATE/bug_report.yml | 4 ++-- .github/ISSUE_TEMPLATE/site_changed.yml | 11 ++++++---- CHANGELOG.md | 19 +++++++++++++++++ CONTRIBUTING.md | 2 +- README.md | 2 +- SECURITY.md | 2 +- output_writer.py | 5 ++--- puppeteer_scraper.py | 27 ++++++++++++++++++------- selenium_scraper.py | 27 ++++++++++++++++++------- 9 files changed, 73 insertions(+), 26 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index d39db59..e16a568 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -78,7 +78,7 @@ body: placeholder: | python3 playwright_scraper.py \ --url "https://www.etsy.com/search?q=handmade+mug" \ - --pages 3 --out grills + --pages 3 --out mugs validations: required: true @@ -98,7 +98,7 @@ body: attributes: label: What you expected instead placeholder: >- - 96 products, as the README says a category page yields. + 184 rows from three search pages, as the README's healthy run shows. validations: required: true diff --git a/.github/ISSUE_TEMPLATE/site_changed.yml b/.github/ISSUE_TEMPLATE/site_changed.yml index 69fa3b3..3f7e8e5 100644 --- a/.github/ISSUE_TEMPLATE/site_changed.yml +++ b/.github/ISSUE_TEMPLATE/site_changed.yml @@ -9,8 +9,11 @@ body: own report because the fix is different from a code bug: something on etsy.com moved. - Useful to know before filing: the parser tries **JSON-LD first**, then a - CSS + URL-pattern fallback. Which one broke narrows the fix a lot. + Useful to know before filing: the primary anchor is Etsy's own + `data-listing-id` attribute, the JSON-LD `ItemList` only enriches and + cross-checks it (it is partial — 8 entries for 64 tiles on a search + page), and the `/listing//` URL pattern is the fallback. + Which one broke narrows the fix a lot. - type: dropdown id: what_broke @@ -19,8 +22,8 @@ body: options: - Products (0 rows on a page that visibly has products) - A specific field is null or wrong (price, sku, image, URL) - - original_price or lowest_price_30d empty out (the tile overlay) - - DOM price confirmation dropped below 90% in the run log + - original_price or discount_pct empty out, or come out wrong + - The structured-price confirmation share warned below its per-page-kind floor in the run log - Pagination stops early or repeats a page - The captcha detector finds nothing on a page that has one - Something else diff --git a/CHANGELOG.md b/CHANGELOG.md index 17e2fe3..da44e36 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,25 @@ with it, so nobody discovers it from a bill or from a diff. client handed `status` onward, so a target 403/503 never reached the page classifier. It now reads `http_code`, falling back to `status` only if that is an integer. After the fix, one live call (`--wait-text mug` on the canary's search URL, no `--cdp-url`) was accepted — no 422 — but ran into the API's own sync-wait limit (HTTP 408, task still pending at the default 60 s), so no target status came back from that call to show. +- **`--help` for `--mode` in `selenium_scraper.py` and `puppeteer_scraper.py` + described mediamarkt-scraper's product mode** — "one /product/ page, with + brand, EAN, description" and "listing or product", while all three engines + take `listing`, `product` and `shop` and Etsy's detail page is + `/listing/`. Both twins now carry the Playwright engine's help text and + module docstring for the three modes. +- **The access badge said "needs a residential exit".** This README measures + a residential exit as NOT sufficient on its own (Playwright Chromium via a + residential US exit: 403, `t=bv`) and the Scraping Browser API as the path + that works, so the badge now says "needs Scraping Browser API". +- **Leftovers from the donor repos in the issue templates and docs:** the + bug-report example wrote `--out grills` and expected Farfetch's "96 + products"; it now uses `--out mugs` and the README's 184-row healthy run. + The site-change template described a JSON-LD-first parser, a + `lowest_price_30d` overlay (no such column here) and a "below 90%" DOM + confirmation; it now names `data-listing-id` as the primary anchor and the + per-page-kind confirmation floors. `SECURITY.md` named Akamai rather than + DataDome, and a comment in `output_writer.py` used MediaMarkt's + `/de/category/tv-audio-202.html` as its example of a page with no listings. ## [0.2.3] — 2026-09-16 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9a53938..ab9ceaa 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -182,7 +182,7 @@ datacentre address will be refused outright, so "it returned nothing" from a VPS is not a finding. Product counts differ by category and by URL, so a bare "worked for me" is not reproducible. -Do not add anything that submits the registration form. This project +Do not add anything that submits a registration or login form. This project deliberately never does, and a captcha token proved valid by creating a real account is not a result worth having. diff --git a/README.md b/README.md index 260be24..db79b1c 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ blocking, JSON/CSV output and a run-metadata sidecar. ![Python](https://img.shields.io/badge/python-3.9%2B-blue?logo=python&logoColor=white) ![licence](https://img.shields.io/badge/licence-MIT-green) ![engines](https://img.shields.io/badge/engines-Playwright%20%7C%20Selenium%20%7C%20Puppeteer-blueviolet) -![access](https://img.shields.io/badge/needs-a%20residential%20exit-orange) +![access](https://img.shields.io/badge/needs-Scraping%20Browser%20API-orange) --- diff --git a/SECURITY.md b/SECURITY.md index 8fec5d6..2a5d68b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -55,7 +55,7 @@ Not because these do not matter, but because they belong somewhere else: - **Bypassing Etsy's bot protection.** This scraper drives an ordinary browser and passes challenges the way a browser does. Anything about how - Akamai or reCAPTCHA behave is not a vulnerability in this repository. + DataDome behaves is not a vulnerability in this repository. - **The scraper stopped working.** Etsy changing its markup is expected — file it as a normal issue, there is a template for exactly that. - **Anything about 2Captcha's services** — the solver API, the Scraping Browser diff --git a/output_writer.py b/output_writer.py index 1a8abce..8e04b5a 100644 --- a/output_writer.py +++ b/output_writer.py @@ -286,9 +286,8 @@ def write_csv(rows: Sequence[Any], path: str, row_cls: Type = Product) -> None: # content". See product_parser.detect_bot_challenge. # # On Etsy this code specifically does NOT cover a page with no listings — a URL -# like /de/category/tv-audio-202.html that answers 200 with a real page and -# no product grid, because it is a landing page of sub-categories rather than -# a listing. That is EXIT_NO_PRODUCTS: the request was served exactly as +# like a taxonomy hub that answers 200 with a real page and no listing grid, +# because it is a landing page of sub-categories rather than a listing. That is EXIT_NO_PRODUCTS: the request was served exactly as # asked and simply has no products on it. Reporting it as blocked would send # a user hunting for a proxy problem that does not exist. EXIT_BLOCKED = 3 diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index 114cddd..866529f 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -9,9 +9,15 @@ and output_writer.finish_run(), so this file is browser plumbing and nothing else. - --mode listing (default) category grids and search results - --mode product one /product/ page, with brand, EAN, - description and the full image list + --mode listing (default) search results (/search?q=...), category + listings (/c/...) and /market/ pages, + paginated + --mode product one /listing/ page, with the shop, the + material, shipping origin, description and + the full image list + --mode shop one /shop/ front: that seller's + catalogue as listing rows, plus the shop's + own rating and location in the sidecar Two things to know before choosing this engine: @@ -1021,10 +1027,17 @@ def parse_args(): "environment or in .env.") p.add_argument("--mode", choices=["listing", "product", "shop"], default="listing", - help="listing (default) or product. product reads one " - "/product/ page and adds brand, EAN, description and " - "the full image list — the columns a listing row " - "cannot carry. No --pages in product mode.") + help="listing (default): paginated search results " + "(/search?q=...), a category listing (/c/...) or a " + "/market/ page — around 64 listings per page. " + "product: one /listing/ page, which adds the " + "material, shipping origin, description and the full " + "image list, and whose rating is the LISTING's rather " + "than the shop's. shop: one /shop/ front — that " + "seller's catalogue as listing rows, with the shop's " + "own rating and location in the sidecar. --pages " + "applies to listing and shop; there is one page to " + "read in product mode.") p.add_argument("--category", default=None, help="Label to tag output rows with.") p.add_argument("--pages", type=int, default=1, help="Listing pages to crawl") p.add_argument("--delay", type=float, default=2.0, help="Delay between pages, seconds") diff --git a/selenium_scraper.py b/selenium_scraper.py index 8e5ee5b..ead343d 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -9,9 +9,15 @@ and output_writer.finish_run(), so this file is browser plumbing and nothing else. - --mode listing (default) category grids and search results - --mode product one /product/ page, with brand, EAN, - description and the full image list + --mode listing (default) search results (/search?q=...), category + listings (/c/...) and /market/ pages, + paginated + --mode product one /listing/ page, with the shop, the + material, shipping origin, description and + the full image list + --mode shop one /shop/ front: that seller's + catalogue as listing rows, plus the shop's + own rating and location in the sidecar Two limits of this engine, stated here rather than left to be discovered. Neither is a bug in this code and neither can be fixed from here: @@ -1018,10 +1024,17 @@ def parse_args(): "environment or in .env.") p.add_argument("--mode", choices=["listing", "product", "shop"], default="listing", - help="listing (default) or product. product reads one " - "/product/ page and adds brand, EAN, description and " - "the full image list — the columns a listing row " - "cannot carry. No --pages in product mode.") + help="listing (default): paginated search results " + "(/search?q=...), a category listing (/c/...) or a " + "/market/ page — around 64 listings per page. " + "product: one /listing/ page, which adds the " + "material, shipping origin, description and the full " + "image list, and whose rating is the LISTING's rather " + "than the shop's. shop: one /shop/ front — that " + "seller's catalogue as listing rows, with the shop's " + "own rating and location in the sidecar. --pages " + "applies to listing and shop; there is one page to " + "read in product mode.") p.add_argument("--category", default=None, help="Label to tag output rows with.") p.add_argument("--pages", type=int, default=1, help="Listing pages to crawl") p.add_argument("--delay", type=float, default=2.0, help="Delay between pages, seconds") From 7e7910caad7f98dfa4c2ee91a1a0ba4d12bef4fb Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Wed, 23 Sep 2026 13:58:00 +0000 Subject: [PATCH 2/3] smoke_test: name mediamarkt-scraper as the sibling in the privacy-pattern comment Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 ++ smoke_test.py | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index da44e36..b1b66de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,8 @@ with it, so nobody discovers it from a bill or from a diff. per-page-kind confirmation floors. `SECURITY.md` named Akamai rather than DataDome, and a comment in `output_writer.py` used MediaMarkt's `/de/category/tv-audio-202.html` as its example of a page with no listings. + A smoke-test comment described "MediaMarkt's pages" without saying it was + a sibling site; it now does. ## [0.2.3] — 2026-09-16 diff --git a/smoke_test.py b/smoke_test.py index b0fa923..be671a8 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -3583,8 +3583,8 @@ def test_no_capture_leaks(): "(%d fixtures, %d chars)" % (len(names), len(fixtures)), len(names) >= 8 and len(fixtures) > 50000) # Guarded with PATTERNS rather than with the literals a previous capture - # happened to contain, so the NEXT capture is checked too. MediaMarkt's - # pages embed a front-end configuration blob — a Sentry DSN, a Woosmap + # happened to contain, so the NEXT capture is checked too. On a sibling + # site (mediamarkt-scraper) the pages embed a front-end configuration blob — a Sentry DSN, a Woosmap # public key, a store-code JWT — none of which is needed to test a # parser, and none of which belongs in a public repository. patterns = { From a6e97d22beaf64082303c4f2a908911437479771 Mon Sep 17 00:00:00 2001 From: Petr Date: Wed, 23 Sep 2026 15:04:35 +0000 Subject: [PATCH 3/3] Fix stale family-core lines (SECURITY releases / DataDome pointer) - SECURITY.md said the project has no releases or version tags; it has both (gh release list). Supported versions now names the latest release and main, in the wording bbb-scraper and foodpanda-scraper already use. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 3 +++ SECURITY.md | 6 +++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b1b66de..a726d43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,9 @@ with it, so nobody discovers it from a bill or from a diff. A smoke-test comment described "MediaMarkt's pages" without saying it was a sibling site; it now does. +- `SECURITY.md` said this project has no releases or version tags; it has + both. "Supported versions" now names the latest release and `main`. + ## [0.2.3] — 2026-09-16 ### Added diff --git a/SECURITY.md b/SECURITY.md index 2a5d68b..d169c0c 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -79,9 +79,9 @@ Not because these do not matter, but because they belong somewhere else: ## Supported versions -`main` only. This project has no releases or version tags; fixes land on `main` -and you update by pulling. If you are running an old clone, update before -reporting. +The latest release and `main`. Fixes land on `main` first and ship in the +next tagged release (see the Releases page). If you are running an old clone, +update before reporting. ## If you have leaked a key