Remove donor-site leftovers from prose, help text and templates

.dockerignore

@@ -11,7 +11,8 @@ __pycache__/
 .venv/
 venv/
 env/
-vrbo_products.*
+medium_posts.*
+medium_posts_*.*
 *_debug.html
 *_debug.png
 live_results/

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -12,36 +12,24 @@ body:
 
         A few things answer most reports faster than we can:
 
-        - **Everything exit 3?** That is Cloudflare's managed challenge, and
-          the first thing to check is how much that address has been
-          fetching. Measured: 8 of the first 14 fetches from one address were
-          served and everything after was refused, with a 45-minute rest not
-          enough to clear it. Raise `--delay`, rest the address, or use
-          `--proxy-file`. A captcha solve buys nothing here — a managed
-          challenge publishes no sitekey.
-        - **`upvotes`, `views`, `created_at` null on every row?** Expected if
-          you scraped a TOPIC url. A topic page's inlined payload carries 0
-          answer objects, so those columns come from nowhere. Point it at a
-          question or a profile. The `data_source` column says which view
-          built each row.
-        - **A handful of rows from a question with hundreds of answers?** The
-          session was served a short feed. Four loads of one URL gave 5, 13,
-          12 and 259 answers, and in the short ones the scroll reached the
-          document's end and the site never fetched more. Re-run it.
-        - **`title` null on some rows?** A question page carries answers to
-          OTHER questions — related ones and merged duplicates — and a few
-          name no question anywhere on the page. `question_url` is still
-          correct on those rows; inventing a title from the slug would be a
-          guess. What is NOT expected is a title that reads "Related …" or
-          one that repeats the page's own question on a row answering a
-          different one. Both of those are worth this issue.
-        - **Zero rows?** You may be on a page that has none, or on a `/search`
-          URL, which this scraper refuses: Medium answers an anonymous search
-          with the language home page rather than with results.
+        - **Everything exit 3?** Check which refusal you got — `--dump-html`
+          writes exactly what the parser saw. A ~5 KB "Sorry, you have been
+          blocked" is Cloudflare's WAF, and the usual cause is a
+          `HeadlessChrome` User-Agent or the address's recent volume. A
+          ~28 KB "Just a moment…" is the managed challenge, which was
+          measured clearing on the next attempt in a fresh context — that is
+          what `--retries` does. On a long archive walk, raise `--delay` or
+          use `--proxy-file`.
+        - **`reading_time_min`, `word_count`, `language` null on every row?**
+          Expected on a TAG FEED: its payload does not carry them. A day
+          archive, an author page or a story does. The `data_source` column
+          says which view built each row.
+        - **`--pages 3` on a tag feed gave one page of stories?** Only the
+          day archive has per-page addresses; `?page=2` on a tag feed is
+          ignored by Medium. Use `/tag/{slug}/archive/{yyyy}/{mm}/{dd}`.
+        - **A publication URL refused?** On purpose: a publication home page
+          carries post ids and no post data. The error names the alternative.
           See [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md).
-        - **Wrong language?** The language is the HOSTNAME (`medium.com`,
-          `jp.medium.com`), not something the exit IP decides. Note Japanese is
-          on `jp.`, not `ja.`.
 
   - type: dropdown
     id: engine
@@ -89,8 +77,8 @@ body:
       render: shell
       placeholder: |
         python3 playwright_scraper.py \
-          --url "https://medium.com/topic/Machine-Learning" \
-          --pages 3 --out grills
+          --url "https://medium.com/tag/python/archive/2026/09/10" \
+          --pages 3 --out python_days
     validations:
       required: true
 
@@ -110,7 +98,7 @@ body:
     attributes:
       label: What you expected instead
       placeholder: >-
-        96 products, as the README says a category page yields.
+        Up to 128 stories per day, as the README says a day archive yields.
     validations:
       required: true
 

.github/ISSUE_TEMPLATE/site_changed.yml

@@ -10,22 +10,21 @@ body:
         something on medium.com moved.
 
         Useful to know before filing, because it narrows the fix a lot — this
-        parser reads **two different sources**, and they break differently:
+        parser reads **two payloads and two fallbacks**, and they break
+        differently:
 
-        - **The inline GraphQL payload**, pushed into the served HTML as
-          `window.ansFrontendGlobals.data.inlineQueryResults…push("{…}")`. It
-          carries the numeric ids, the full answer text, `numUpvotes`,
-          `numViews` and the creation time. A question page and a profile
-          have it; a topic feed does not.
-        - **The rendered cards**, anchored on Medium's own hooks:
-          `a.answer_timestamp`, `.puppeteer_test_question_title`,
-          `.spacing_log_answer_content`, `.spacing_log_answer_header`.
+        - **`window.__APOLLO_STATE__`** — the modern site: tag feeds, author
+          pages, story pages.
+        - **`window["obvInit"]({…})`** — the legacy renderer, and the only
+          one that answers a tag's day archive, with reading time, word count
+          and language on every story.
+        - **The page's JSON-LD and the rendered cards** (`article`,
+          `.streamItem`, and the author's `/@handle` link) — both fallbacks.
 
-        If the PAYLOAD moves, a run does not fail — it quietly degrades to
-        the cards and every count comes out null while the row count and the
+        If a PAYLOAD moves, a run does not necessarily fail — it degrades to
+        a fallback and the counts empty out while the row count and the
         titles stay healthy. The `data_source` column is what shows it: rows
-        that used to read `dom+inline` now read `dom`. If a CARD hook moves,
-        the run reports 0 rows and exit 4, which is loud.
+        that used to read `apollo` or `obvinit` now read `jsonld` or `dom`.
 
         So please include the `data_source` breakdown of your run — it is the
         single most useful line in the report.
@@ -35,13 +34,12 @@ body:
     attributes:
       label: What stopped matching
       options:
-        - No rows at all on a page that visibly has answers (a card hook moved)
-        - Rows come out, but every data_source is "dom" (the payload moved)
-        - title is null, or wrong, on rows that clearly name a question
-        - author empty on every row (the named profile link moved)
-        - Scrolling no longer extends the feed at all
-        - A language host stopped being recognised
-        - A Space or Medium Session permalink is no longer read as an answer
+        - No rows at all on a page that visibly has stories
+        - Rows come out, but every data_source is "jsonld" or "dom" (a payload moved)
+        - title or author empty on rows that clearly have one
+        - claps look wrong (compare with the count the story page shows)
+        - The day-archive walk stopped, or stopped redirecting, where it should not
+        - A URL kind is no longer recognised (tag, archive, author, story)
         - Something else
     validations:
       required: true
@@ -51,10 +49,9 @@ body:
     attributes:
       label: URL
       description: >-
-        The exact URL, its locale prefix, and how the browser was reached.
-        Please say whether the run was headful or over --cdp-endpoint: a
-        headless run is refused by the site and is not a parser problem.
-      placeholder: "https://medium.com/topic/Machine-Learning (mode: topic, headful bundled Chromium, exit: DE datacentre)"
+        The exact URL and how the browser was reached. Please say whether the
+        run was headless or headful, or over --cdp-endpoint.
+      placeholder: "https://medium.com/tag/python (mode: tag, headful bundled Chromium, exit: DE datacentre)"
     validations:
       required: true
 
@@ -66,12 +63,10 @@ body:
         Whichever of these you can get. A capture beats a description, and
         `--dump-html` writes one on success as well as on failure.
 
-        - The payload as the page actually serves it:
-          `python3 -c "import json,re;h=open('dump.html').read();d=json.loads(re.search(r'id=.__NEXT_DATA__.[^>]*>(.*?)</script>',h,1).group(1));print(sorted(d['props']['pageProps']))"`
-        - For a price problem: one whole `article.c-lot-card__container` from
-          devtools, and the amount the page shows for that lot.
-        - For a `bid_kind` problem: the exact label the card prints
-          ("Current bid", "Aktuelles Gebot", ...) and the locale.
+        - Which payload the page actually serves:
+          `grep -o '__APOLLO_STATE__\|obvInit' dump.html | sort | uniq -c`
+        - For a count problem: the story URL and the number the story page
+          shows.
       render: text
     validations:
       required: true

CHANGELOG.md

@@ -26,6 +26,48 @@ empty output file.
   that is an integer). Pinned by an offline check that drives the real
   client with `requests.post` stubbed; verified by control (red with the
   old client).
+- **Help text and log lines that described a different site.** `--mode`,
+  `--category`, `--pages` and `--concurrency` help in the engines, and the
+  Scraper API client's `--url` help, described a Q&A site's questions,
+  topics and vote counts. `--pages`/`--concurrency` also claimed no mode has
+  per-page addresses, while `--mode archive` does and the Playwright worker
+  pool uses them. The readiness log named the donor sites' card kinds (an
+  answer in Playwright and pyppeteer, a rental property in Selenium); all
+  three now say "No story cards".
+- **The Scraper API client's refusal messages** claimed this site answers the
+  API's exit with nothing and that a rendered browser is required. The
+  README measured the Scraper API served on all four modes; the messages now
+  say so and point at `--retries` / `--cdp-url` as the next step.
+- **`diff_runs.py`** docstring, comments, `--price-tolerance-pct` help and
+  its `source_changed` line described answers, questions and topic feeds;
+  they now describe stories, tag feeds and archive/author/post views. The
+  `source_changed` view fields no longer include `text_chars`, a column this
+  repo's row does not have.
+- `output_writer`'s `mode` default was `"topic"`, a mode this repo does not
+  have; it is now `"tag"`. Every engine passes its mode explicitly.
+- **CONTRIBUTING.md and both issue templates were about another site** — its
+  card hooks, inline payload, columns and URLs. Rewritten from this repo's
+  README and parser: the two payloads and two fallbacks, and the properties
+  the suite pins.
+- `requirements*.txt` named another repo and described another site's
+  browser measurements (including `playwright install chrome`, which the
+  README does not need); they now carry this repo's own measurements.
+- `.dockerignore` ignored another repo's output prefix; it now ignores
+  `medium_posts.*`, the engines' real `--out` default.
+- Stray donor wording in `TROUBLESHOOTING.md` and in suite comments, and a
+  `min_matches` check that exercised a mode named `question`.
+
+### Changed
+
+- The suite's foreign-vocabulary guard also scans `.txt` files and
+  `.dockerignore`, and knows the Q&A sibling's vocabulary. Both new halves
+  were controlled: planting the old `requirements.txt` header and the old
+  Selenium log line each turned the suite red on that check. (This entry
+  describes the old wording rather than quoting it, because CHANGELOG.md is
+  scanned too.)
+
+- `SECURITY.md` said this project has no releases or version tags; it has
+  both. "Supported versions" now names the latest release and `main`.
 
 ## [0.1.1] — 2026-09-16