Repository navigation
Remove donor-repo leftovers (Montblanc, BBB, Etsy) - #3
Merged
Merged
Conversation
- --pages help: no --mode category/search here; a category past its end re-serves its last page, not an empty grid (all three engines) - Playwright blocked-page error: drop the false "edge refuses curl and python-requests" claim; match the other two engines' measured text - parse-failure error and image-coverage log: no JSON-LD ItemList on a listing here; point at the tile markup - --mode product log/--help (engines + scraper_api_client): one row per page, not variants of a ProductGroup; --retries help: no hub pages - --dump-html help pointed at a TROUBLESHOOTING.md that does not exist - pyppeteer/Selenium blocked_by "cloudflare (hard block)" -> "edge refusal", matching Playwright - diff_runs.py tracked Montblanc columns absent from this row model; now tracks tax_rate and subcategory - issue templates rewritten from Etsy's to this site's README - captcha_solver.py BBB Turnstile prose replaced with this site's measurement; Montblanc URLs/counts, BBB find_country and an uncredited tokopedia anecdote fixed in engine comments - smoke_test.py sidecar fixtures: montblanc.com run -> measured /c/c15/ run - .dockerignore bbb_businesses.* -> kose_products.*; Dockerfile example - CHANGELOG: new [Unreleased] section Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- SECURITY.md said the project has no releases or version tags; it has both (gh release list). Supported versions now names the latest release and main, in the wording bbb-scraper and foodpanda-scraper already use. - captcha_solver.py's docstring pointed at a "No DataDome solver" section this repo does not have; the line came with the copied core. Removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jehrr
force-pushed
the
fix/donor-leftovers
branch
from
September 28, 2026 19:49
37d744c to
ec632d7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Leftovers from the sibling repos kose-scraper was bootstrapped from, which described those sites as if they were this one. Replacements come from this repo's own README, CHANGELOG, code and live_results; nothing new was measured.
Fixed
--pageshelp (all three engines): named--mode category/--mode search(this repo haslisting/product) and said a category past its end is "a served, empty grid". The README measured the opposite: it serves the last page again.curl,python-requestsand friends outright" and cited a Hetzner measurement. The README measured all of those served. It now says what pyppeteer and Selenium already said.ItemList. A listing here has no JSON-LD (0 blocks in 8 captures), so both now point at the tile markup.--mode productwording: the log line said "variant(s)",--concurrencysaid "however many variants it holds", andscraper_api_client --modehelp said "one row per variant out of the page's ProductGroup". A Kosé product page is oneProductblock, so one row.--retrieshelp spoke of "a hub page".--dump-htmlhelp pointed at aTROUBLESHOOTING.mdthis repo does not have. It now points at the README's "Traps that look like bugs".blocked_by: pyppeteer and Selenium recordedcloudflare (hard block)(no Cloudflare has been observed here), while Playwright recordededge refusal, so the engines disagreed onstop_reason. All three now useedge refusal.diff_runs.pytracked Montblanc columns (collection,sub_collection,color,size,special_edition,base_sku) that this row model does not have, plus Montblanc stock counts in the comment. It now trackstax_rateandsubcategory, andsubcategory/variant_ofare product-mode-only.t=bv,shop_rating, an etsy.com URL, a missing TROUBLESHOOTING.md). They were rewritten from this README.captcha_solver.py: the BBB Turnstile docstring and the "what BBB does instead" block were replaced with this site's measurement (no challenge in 13 captures)./en-fi/...URLs and 280/sz/start=arithmetic became this site's/c/c15/13 pages / 304. BBB'sfind_countrywas in two locale comments. An uncredited/p/<slug>hub anecdote is now credited to tokopedia-scraper. "business-at-a-location" became "product". "Montblanc" in output_writer/page_flow now names montblanc-scraper as a sibling.smoke_test.py: the sidecar fixtures came from a montblanc.com run (en-fi,sort_requested, 280 results). They now use the measured/c/c15/run and assert the extra fields this engine actually writes (pages_available,page_size,canonical_url,route), plus that there is nolocale. There is one more check than before. The prose naming BBB/Montblanc as this site was fixed..dockerignore:bbb_businesses.*becamekose_products.*/kose_products_*.*, the argparse default, mirroring.gitignore. The Dockerfile example--out /out/writing-instrumentsbecamekose_products.[Unreleased]section. There is no version bump.Kept deliberately
refused,image_urlfloor), "mediamarkt-scraper needs" (Omnibus), "quora-scraper made the same call", the tokopedia binding-check note, and the farfetch notes in captcha_solver..gitignorecraigslist-scraper note, which is family-wide.akamai/errors.edgesuite.netmarker in product_parser, which is family detection coverage, and CLAUDE.md'sassets.mmsrg.comreference.categorystill differs between modes (c15on a listing row,スキンケアon a product row) and is still tracked by diff_runs as before. That is pre-existing behaviour, not a donor leftover.Verification
smoke_test.py: 463 passed before, 464 passed after, 0 failed. The same 4 engine-absent skips appear both times.ci_checks.py --help-check --sample-checkpasses, and--secret-checkfinds nothing.🤖 Generated with Claude Code