Skip to content

Remove donor-site leftovers - #8

Merged
jehrr merged 2 commits into
mainfrom
fix/donor-leftovers
Sep 28, 2026
Merged

jehrr merged 2 commits into
mainfrom
fix/donor-leftovers

Conversation

@jehrr

@jehrr jehrr commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Removes text and data inherited from the repos this one was bootstrapped from, which described another site as if it were dubizzle.

Fixed

  • User-visible strings: scraper_api_client.py description ("auction scraper", scrolling grid, /p/<cat> URL help); diff_runs.py --price-tolerance-pct ("quotes IDR"); Playwright --pages help (infinitely scrolling search); Playwright --concurrency warning ("refuses a headless browser ... answering NOTHING at all").
  • Core/engine prose: Akamai 394-byte headless refusal, --mode product/--mode shop, unaddressable search and /p/<slug> hub comments in all three engines, captcha_solver.py and output_writer.py — replaced with the Imperva / UAE-exit facts from this repo's README and page_flow.py, or deleted. Contradictory "HEADFUL by default" comments removed (the default is headless, as the code and suite already say). Unused shop_facts dataclass field removed from the engines.
  • diff_runs.py: euro / 18 locales / auction and bid prose -> AED, two languages, Car of the Week lifecycle bucket; donor example filenames.
  • CONTRIBUTING.md: "Reporting a site change" rewritten for __NEXT_DATA__, the JSON-LD path join and the DOM fallback; the datacentre "no response at all" note corrected; the pyppeteer first-fetch crash attributed to the sibling (tokopedia-scraper).
  • Issue templates: site_changed options for reserve prices, favorite_count, suggested lots, a 100-page cap, "headless is refused", and two orphaned donor lines; bug_report example --out grills / "96 products".
  • .gitignore / .dockerignore: dubizzle_listings.* -> dubizzle_products.* (the engines' real --out default; _*.* covers dubizzle_products_scraperapi).
  • smoke_test.py: CATAWIKI_CDP_ENDPOINT / CATAWIKI_PROXY -> DUBIZZLE_* (same assertions); EUR comment -> AED.
  • README: last badge "runs without an account" -> "needs a UAE exit" (the README's own "Do you need any of the paid products?" says a UAE exit is required). Badge order was already the family order.
  • CHANGELOG [Unreleased] entry; no version bump.

Kept deliberately

  • farfetch-scraper / Kohl's provenance in captcha_solver.py, the craigslist .env* note in .gitignore: explicit sibling provenance.
  • "a dubizzle run and a mediamarkt run with the same code" in output_writer.py: family schema statement.
  • The "sibling repos' auction columns are NOT tracked" check in smoke_test.py: a guard, phrased as sibling.
  • <captcha-widgets> / _Incapsula_Resource notes in product_parser.py and CONTRIBUTING: stated as measured on this site's own 17 captures.
  • Released CHANGELOG sections untouched.
  • Repo description/topics: no donor leftovers, unchanged.

Suite: smoke_test: OK before and after (engine groups skipped, no engine installed); ci_checks.py --secret-check and --help-check --sample-check green.

🤖 Generated with Claude Code

jehrr and others added 2 commits September 28, 2026 19:49
…o/IDR, auctions)

- --help: scraper_api_client "auction scraper" description and /p/<cat>
  URL help; diff_runs --price-tolerance-pct "quotes IDR"; playwright
  --pages "infinitely scrolling search".
- playwright --concurrency warning: "refuses a headless browser ... by
  answering NOTHING at all" -> refuses exits outside the UAE.
- Engines, captcha_solver, output_writer: Akamai 394-byte headless refusal,
  --mode product/shop, unaddressable search, /p/<slug> hub prose replaced
  with this repo's measured Imperva/UAE facts or deleted; contradictory
  "HEADFUL by default" comments removed; unused shop_facts field removed.
- diff_runs: euro/18 locales/auction prose -> AED, two languages, Car of
  the Week lifecycle; farfetch-style example filenames -> cars.
- CONTRIBUTING: site-change section rewritten for __NEXT_DATA__ / JSON-LD
  join / DOM fallback; datacentre note corrected; pyppeteer crash
  attributed to the sibling it happened in.
- Issue templates: donor options (reserve price, favorite_count, suggested
  lots, 100 cap, headless refused) and orphan lines removed; bug report
  example prefix/count fixed.
- .gitignore/.dockerignore: dubizzle_listings.* -> dubizzle_products.*.
- smoke_test: CATAWIKI_* env names -> DUBIZZLE_*; EUR comment -> AED.
- README: last badge "runs without an account" -> "needs a UAE exit".
- CHANGELOG: new [Unreleased] section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- SECURITY.md said the project has no releases or version tags; it has both
  (gh release list). Supported versions now names the latest release and main,
  in the wording bbb-scraper and foodpanda-scraper already use.
- captcha_solver.py's docstring pointed at a "No DataDome solver" section this
  repo does not have; the line came with the copied core. Removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr
jehrr force-pushed the fix/donor-leftovers branch from bf8e618 to 75cd2b1 Compare September 28, 2026 19:49
@jehrr
jehrr merged commit 2a8517d into main Sep 28, 2026
7 checks passed
@jehrr
jehrr deleted the fix/donor-leftovers branch September 28, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant