Skip to content

Remove donor-site leftovers from prose, help text and templates - #7

Merged
jehrr merged 2 commits into
mainfrom
fix/donor-leftovers
Sep 28, 2026
Merged

jehrr merged 2 commits into
mainfrom
fix/donor-leftovers

Conversation

@jehrr

@jehrr jehrr commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

This repo was bootstrapped from a Q&A-site sibling and a rental-site sibling. Their names had been scrubbed; their vocabulary and measurements had not.

Fixed

  • User-visible help text in the engines (--mode, --category, --pages, --concurrency) and in scraper_api_client.py --url described questions, topics and vote counts. --pages/--concurrency also said no mode has per-page addresses, which contradicts --mode archive and the Playwright worker pool.
  • Readiness log line named another site's card kind in all three engines; now "No story cards".
  • Scraper API refusal messages claimed the API's exit gets nothing and that a rendered browser is required. The README measured the Scraper API served on all four modes, so the messages now say that and point at --retries / --cdp-url.
  • diff_runs.py: docstring, comments, --price-tolerance-pct help and the source_changed output line rewritten for stories and tag/archive/author/post views. text_chars removed from the view fields because it is not a column here. Dropped an unmeasured "1,733 views" claim.
  • output_writer.py: mode default was "topic", which is not a mode here. It is now "tag". Every engine passes its mode explicitly.
  • CONTRIBUTING.md, bug_report.yml, site_changed.yml described another site's inline payload, card hooks, columns and URLs. They are rewritten from this repo's README and product_parser.py docstring, and every number in them comes from those files.
  • requirements*.txt: another repo's name, and its browser measurements, including playwright install chrome. They now carry this repo's own findings (bundled Chromium is fine; pyppeteer's bundled Chrome/117 was refused 3/3).
  • .dockerignore: donor output prefix replaced with medium_posts.* / medium_posts_*.*, taken from the engines' --out default.
  • Stray wording in TROUBLESHOOTING.md and suite comments. A min_matches check exercised a mode called question; it now uses tag, plus a new post-mode assertion.
  • Guard: the suite's foreign-vocabulary check now also scans .txt and .dockerignore, and it knows the Q&A sibling's vocabulary. Controlled: planting the old requirements.txt header, and separately the old Selenium log line, each turned the suite red on that check.

Kept, deliberately

  • The sibling guard list in smoke_test.py, which exists on purpose.
  • Explicit sibling provenance: foodpanda-scraper in the README and in a suite comment, the craigslist note in .gitignore, and the "a sibling repo …" comments in page_flow.py and the engines.
  • Released CHANGELOG sections are untouched.
  • fixtures_generated.json, which is real page data.
  • README badges were already in family order. "runs without an account" is honest here, because the README measures every mode with no key and no proxy from a datacentre address.

Not changed, noted

  • TROUBLESHOOTING.md says there is "nothing to hand 2Captcha" for the managed challenge, while the README and CHANGELOG 0.1.1 say the Turnstile parameters can be intercepted and solved. That is not a donor leftover, so it is out of scope for this PR.

Suite: green before and after (709 → 715 PASS). The three engine groups skip locally because no engine is installed. ci_checks.py --secret-check --help-check --sample-check all pass.

🤖 Generated with Claude Code

jehrr and others added 2 commits September 28, 2026 19:49
- engines: --mode/--category/--pages/--concurrency help described another
  site's questions/topics/vote counts, and claimed no mode has per-page
  addresses although --mode archive does; readiness log named another
  site's card kind in all three engines ("No story cards" now)
- scraper_api_client.py: refusal messages and --url help claimed the API's
  exit gets nothing and a rendered browser is required; README measured it
  served on all four modes
- diff_runs.py: docstring, comments, --price-tolerance-pct help and the
  source_changed line rewritten for stories/tag feeds; dropped text_chars
  (not a column here) from the view fields
- output_writer.py: mode default "topic" -> "tag"
- CONTRIBUTING.md, bug_report.yml, site_changed.yml rewritten from this
  repo's README and parser
- requirements*.txt: another repo's name and browser measurements replaced
  with this repo's own
- .dockerignore: donor output prefix -> medium_posts.*
- TROUBLESHOOTING.md and suite comments; min_matches check used a
  non-existent mode
- smoke_test.py: foreign-vocabulary guard now scans .txt and .dockerignore
  and knows the Q&A sibling's vocabulary (controlled red on both)
- CHANGELOG: Unreleased entry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- SECURITY.md said the project has no releases or version tags; it has both
  (gh release list). Supported versions now names the latest release and main,
  in the wording bbb-scraper and foodpanda-scraper already use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr
jehrr force-pushed the fix/donor-leftovers branch from 2f15358 to 533619c Compare September 28, 2026 19:50
@jehrr
jehrr merged commit 07407f0 into main Sep 28, 2026
7 checks passed
@jehrr
jehrr deleted the fix/donor-leftovers branch September 28, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant