Skip to content

Remove donor-repo leftovers (Montblanc, BBB, Etsy) - #3

Merged
jehrr merged 2 commits into
mainfrom
fix/donor-leftovers
Sep 28, 2026
Merged

jehrr merged 2 commits into
mainfrom
fix/donor-leftovers

Conversation

@jehrr

@jehrr jehrr commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Leftovers from the sibling repos kose-scraper was bootstrapped from, which described those sites as if they were this one. Replacements come from this repo's own README, CHANGELOG, code and live_results; nothing new was measured.

Fixed

  • --pages help (all three engines): named --mode category / --mode search (this repo has listing / product) and said a category past its end is "a served, empty grid". The README measured the opposite: it serves the last page again.
  • Playwright blocked-page error: said the edge "refuses curl, python-requests and friends outright" and cited a Hetzner measurement. The README measured all of those served. It now says what pyppeteer and Selenium already said.
  • Parse-failure error and image-coverage log (all three engines): pointed at a JSON-LD ItemList. A listing here has no JSON-LD (0 blocks in 8 captures), so both now point at the tile markup.
  • --mode product wording: the log line said "variant(s)", --concurrency said "however many variants it holds", and scraper_api_client --mode help said "one row per variant out of the page's ProductGroup". A Kosé product page is one Product block, so one row.
  • --retries help spoke of "a hub page". --dump-html help pointed at a TROUBLESHOOTING.md this repo does not have. It now points at the README's "Traps that look like bugs".
  • blocked_by: pyppeteer and Selenium recorded cloudflare (hard block) (no Cloudflare has been observed here), while Playwright recorded edge refusal, so the engines disagreed on stop_reason. All three now use edge refusal.
  • diff_runs.py tracked Montblanc columns (collection, sub_collection, color, size, special_edition, base_sku) that this row model does not have, plus Montblanc stock counts in the comment. It now tracks tax_rate and subcategory, and subcategory / variant_of are product-mode-only.
  • Issue templates were Etsy's (DataDome, t=bv, shop_rating, an etsy.com URL, a missing TROUBLESHOOTING.md). They were rewritten from this README.
  • captcha_solver.py: the BBB Turnstile docstring and the "what BBB does instead" block were replaced with this site's measurement (no challenge in 13 captures).
  • Engine comments: Montblanc /en-fi/... URLs and 280/sz/start= arithmetic became this site's /c/c15/ 13 pages / 304. BBB's find_country was in two locale comments. An uncredited /p/<slug> hub anecdote is now credited to tokopedia-scraper. "business-at-a-location" became "product". "Montblanc" in output_writer/page_flow now names montblanc-scraper as a sibling.
  • smoke_test.py: the sidecar fixtures came from a montblanc.com run (en-fi, sort_requested, 280 results). They now use the measured /c/c15/ run and assert the extra fields this engine actually writes (pages_available, page_size, canonical_url, route), plus that there is no locale. There is one more check than before. The prose naming BBB/Montblanc as this site was fixed.
  • .dockerignore: bbb_businesses.* became kose_products.* / kose_products_*.*, the argparse default, mirroring .gitignore. The Dockerfile example --out /out/writing-instruments became kose_products.
  • The CHANGELOG has a new [Unreleased] section. There is no version bump.

Kept deliberately

  • Sibling comparisons that name the sibling repo and state a lesson: "the opposite of bbb-scraper" (sort, page cap), "Unlike montblanc-scraper" (refused, image_url floor), "mediamarkt-scraper needs" (Omnibus), "quora-scraper made the same call", the tokopedia binding-check note, and the farfetch notes in captcha_solver.
  • The .gitignore craigslist-scraper note, which is family-wide.
  • The akamai / errors.edgesuite.net marker in product_parser, which is family detection coverage, and CLAUDE.md's assets.mmsrg.com reference.
  • Released CHANGELOG sections were left untouched.
  • category still differs between modes (c15 on a listing row, スキンケア on a product row) and is still tracked by diff_runs as before. That is pre-existing behaviour, not a donor leftover.

Verification

  • smoke_test.py: 463 passed before, 464 passed after, 0 failed. The same 4 engine-absent skips appear both times.
  • ci_checks.py --help-check --sample-check passes, and --secret-check finds nothing.
  • The repo metadata (description, topics, homepage) is clean, so nothing was changed there.

🤖 Generated with Claude Code

Petr and others added 2 commits September 28, 2026 19:48
- --pages help: no --mode category/search here; a category past its end
  re-serves its last page, not an empty grid (all three engines)
- Playwright blocked-page error: drop the false "edge refuses curl and
  python-requests" claim; match the other two engines' measured text
- parse-failure error and image-coverage log: no JSON-LD ItemList on a
  listing here; point at the tile markup
- --mode product log/--help (engines + scraper_api_client): one row per
  page, not variants of a ProductGroup; --retries help: no hub pages
- --dump-html help pointed at a TROUBLESHOOTING.md that does not exist
- pyppeteer/Selenium blocked_by "cloudflare (hard block)" -> "edge
  refusal", matching Playwright
- diff_runs.py tracked Montblanc columns absent from this row model; now
  tracks tax_rate and subcategory
- issue templates rewritten from Etsy's to this site's README
- captcha_solver.py BBB Turnstile prose replaced with this site's
  measurement; Montblanc URLs/counts, BBB find_country and an uncredited
  tokopedia anecdote fixed in engine comments
- smoke_test.py sidecar fixtures: montblanc.com run -> measured /c/c15/ run
- .dockerignore bbb_businesses.* -> kose_products.*; Dockerfile example
- CHANGELOG: new [Unreleased] section

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- SECURITY.md said the project has no releases or version tags; it has both
  (gh release list). Supported versions now names the latest release and main,
  in the wording bbb-scraper and foodpanda-scraper already use.
- captcha_solver.py's docstring pointed at a "No DataDome solver" section this
  repo does not have; the line came with the copied core. Removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr
jehrr force-pushed the fix/donor-leftovers branch from 37d744c to ec632d7 Compare September 28, 2026 19:49
@jehrr
jehrr merged commit 0b50e12 into main Sep 28, 2026
7 checks passed
@jehrr
jehrr deleted the fix/donor-leftovers branch September 28, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant