Skip to content

Remove donor-repo leftovers (Tokopedia/Catawiki/VRBO prose) - #11

Merged
jehrr merged 1 commit into
mainfrom
fix/donor-leftovers
Sep 29, 2026
Merged

jehrr merged 1 commit into
mainfrom
fix/donor-leftovers

Conversation

@jehrr

@jehrr jehrr commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Fixed

  • User-visible scraper_api_client.py --help: the description said these pages need JavaScript and a single fetch returns "about 5 products where a browser engine returns 60"; --url said to pass --wait-element '[data-stid="lodging-card-responsive"]' because "the grid arrives over client-side GraphQL" (VRBO/Tokopedia). This file's own docstring measures 48 of 48 vendors from one request on a server-rendered grid. Both rewritten; --category help no longer claims a URL-segment default (the client never reads args.category).
  • Issue templates were an auction site's (lots, reserve prices, seller_score, page-100 cap, /en/ locales, __NEXT_DATA__, c-lot-card, --out grills, "96 products"). Rewritten from README / CONTRIBUTING / TROUBLESHOOTING.
  • pyppeteer/Selenium docstrings listed --mode product (brand, EAN) and used foodpanda.pk/p/makanan-minuman/minuman/kopi-bubuk (a Tokopedia path) as the example. Now the one listing mode and /city/lahore/area/gulberg.
  • Dead code: the twins' if args.mode == "product" sidecar branch (shop_facts, "Shop: %s") was unreachable (choices=["listing"]). Removed with the shop_facts field; the listing branch runs unchanged. The twins' stop_reason = "single_page_mode" if args.mode == "product" is now "completed", as in playwright_scraper (same value at runtime).
  • Comments telling another site's story as this one's: /p/<slug> hub + akamaihd.net (now attributed to sibling tokopedia-scraper), "hub category" / "taxonomy hub", CATEGORY vs SEARCH grid over GraphQL, keyword=kopi + UTM tracking tail, detail-page buy box (engines); Akamai 394-byte "Access Denied" and "lots" (captcha_solver.py, replaced with the README's headless 0/4 vs headful 4/4 and PerimeterX's widget-less stub); "Oops, produk nggak ditemukan", HTTP/2 stream reset, "Orlando search", shop_rating (output_writer.py). Dangling "See 'No DataDome solver' below" removed.
  • SECURITY.md said "no releases or version tags"; v0.2.0/v0.2.1 exist.
  • Access badge runs without an account -> runs without an account · residential IP, matching README's "Do I need to buy anything?" (measured from a residential address; the canary notes a datacentre runner is worse).

Repo metadata

  • Topic price-monitoring removed (a vendor tile carries no price: README measures 0 price nodes over 1,952 tiles), restaurants added.

Kept, deliberately

  • scraper_api_client.py docstring "a sibling repo where this path reaches 3 of 50 cards": names the sibling.
  • captcha_solver.py "The sibling repo ... image captcha" paragraph, farfetch-scraper notes; product_parser.py / smoke_test.py amazon-scraper review-count notes; .gitignore craigslist note: sibling provenance.
  • product_parser.py "datadome" in the vendor marker list: a generic vendor name, not donor prose.
  • CHANGELOG.md released sections (incl. mediamarkt mention): history.
  • smoke_test.py deliveroo URL: a negative-host fixture.

Not changed, found in passing

  • selenium_scraper.py and puppeteer_scraper.py accept --category but never record it as run_label, while their help says they do (playwright does). That is twin drift in code, not donor text, so it is reported here rather than fixed.
  • The twins still log "This listing holds %d product(s)" behind a total_available that is always None on this site.

Suite: 535 checks, OK before and after (engine groups skipped, engines absent). ci_checks.py --secret-check --help-check --sample-check green.

🤖 Generated with Claude Code

- scraper_api_client --help: description and --url no longer describe a
  JS-hydrated GraphQL grid and a lodging-card selector; --category help
  no longer claims a URL-segment default the client does not implement.
- Issue templates rewritten for foodpanda (were an auction site's).
- pyppeteer/Selenium docstrings: one listing mode, real --url example;
  unreachable --mode product sidecar branch and shop_facts removed.
- Engine, captcha_solver and output_writer comments: donor stories
  (hub pages, search grid, kopi tracking tail, Akamai 394-byte refusal,
  lots, Orlando search, stream reset, shop_rating) replaced with this
  repo's measurements or attributed to the sibling; dangling
  "No DataDome solver" reference removed.
- SECURITY.md: no longer claims the project has no releases.
- Access badge: no account needed, from a residential IP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr
jehrr force-pushed the fix/donor-leftovers branch from 0406aae to 7621731 Compare September 29, 2026 08:13
@jehrr
jehrr merged commit a46d08c into main Sep 29, 2026
7 checks passed
@jehrr
jehrr deleted the fix/donor-leftovers branch September 29, 2026 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant