From 0e65fc39c335189c4c47df44c09fc461938da4b5 Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Wed, 23 Sep 2026 13:57:57 +0000 Subject: [PATCH 1/2] Remove donor-site leftovers (Q&A sibling prose, rental-site names) - engines: --mode/--category/--pages/--concurrency help described another site's questions/topics/vote counts, and claimed no mode has per-page addresses although --mode archive does; readiness log named another site's card kind in all three engines ("No story cards" now) - scraper_api_client.py: refusal messages and --url help claimed the API's exit gets nothing and a rendered browser is required; README measured it served on all four modes - diff_runs.py: docstring, comments, --price-tolerance-pct help and the source_changed line rewritten for stories/tag feeds; dropped text_chars (not a column here) from the view fields - output_writer.py: mode default "topic" -> "tag" - CONTRIBUTING.md, bug_report.yml, site_changed.yml rewritten from this repo's README and parser - requirements*.txt: another repo's name and browser measurements replaced with this repo's own - .dockerignore: donor output prefix -> medium_posts.* - TROUBLESHOOTING.md and suite comments; min_matches check used a non-existent mode - smoke_test.py: foreign-vocabulary guard now scans .txt and .dockerignore and knows the Q&A sibling's vocabulary (controlled red on both) - CHANGELOG: Unreleased entry Co-Authored-By: Claude Opus 5.5 --- .dockerignore | 3 +- .github/ISSUE_TEMPLATE/bug_report.yml | 52 +++---- .github/ISSUE_TEMPLATE/site_changed.yml | 55 ++++--- CHANGELOG.md | 39 +++++ CONTRIBUTING.md | 187 ++++++++++-------------- TROUBLESHOOTING.md | 2 +- diff_runs.py | 93 ++++++------ output_writer.py | 4 +- playwright_scraper.py | 86 ++++++----- puppeteer_scraper.py | 26 ++-- requirements-playwright.txt | 21 +-- requirements-puppeteer.txt | 10 +- requirements-selenium.txt | 6 +- requirements.txt | 4 +- scraper_api_client.py | 27 ++-- selenium_scraper.py | 21 +-- smoke_test.py | 47 +++--- 17 files changed, 340 insertions(+), 343 deletions(-) diff --git a/.dockerignore b/.dockerignore index 7c31857..64bc667 100644 --- a/.dockerignore +++ b/.dockerignore @@ -11,7 +11,8 @@ __pycache__/ .venv/ venv/ env/ -vrbo_products.* +medium_posts.* +medium_posts_*.* *_debug.html *_debug.png live_results/ diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 5423605..36a4c4b 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -12,36 +12,24 @@ body: A few things answer most reports faster than we can: - - **Everything exit 3?** That is Cloudflare's managed challenge, and - the first thing to check is how much that address has been - fetching. Measured: 8 of the first 14 fetches from one address were - served and everything after was refused, with a 45-minute rest not - enough to clear it. Raise `--delay`, rest the address, or use - `--proxy-file`. A captcha solve buys nothing here — a managed - challenge publishes no sitekey. - - **`upvotes`, `views`, `created_at` null on every row?** Expected if - you scraped a TOPIC url. A topic page's inlined payload carries 0 - answer objects, so those columns come from nowhere. Point it at a - question or a profile. The `data_source` column says which view - built each row. - - **A handful of rows from a question with hundreds of answers?** The - session was served a short feed. Four loads of one URL gave 5, 13, - 12 and 259 answers, and in the short ones the scroll reached the - document's end and the site never fetched more. Re-run it. - - **`title` null on some rows?** A question page carries answers to - OTHER questions — related ones and merged duplicates — and a few - name no question anywhere on the page. `question_url` is still - correct on those rows; inventing a title from the slug would be a - guess. What is NOT expected is a title that reads "Related …" or - one that repeats the page's own question on a row answering a - different one. Both of those are worth this issue. - - **Zero rows?** You may be on a page that has none, or on a `/search` - URL, which this scraper refuses: Medium answers an anonymous search - with the language home page rather than with results. + - **Everything exit 3?** Check which refusal you got — `--dump-html` + writes exactly what the parser saw. A ~5 KB "Sorry, you have been + blocked" is Cloudflare's WAF, and the usual cause is a + `HeadlessChrome` User-Agent or the address's recent volume. A + ~28 KB "Just a moment…" is the managed challenge, which was + measured clearing on the next attempt in a fresh context — that is + what `--retries` does. On a long archive walk, raise `--delay` or + use `--proxy-file`. + - **`reading_time_min`, `word_count`, `language` null on every row?** + Expected on a TAG FEED: its payload does not carry them. A day + archive, an author page or a story does. The `data_source` column + says which view built each row. + - **`--pages 3` on a tag feed gave one page of stories?** Only the + day archive has per-page addresses; `?page=2` on a tag feed is + ignored by Medium. Use `/tag/{slug}/archive/{yyyy}/{mm}/{dd}`. + - **A publication URL refused?** On purpose: a publication home page + carries post ids and no post data. The error names the alternative. See [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md). - - **Wrong language?** The language is the HOSTNAME (`medium.com`, - `jp.medium.com`), not something the exit IP decides. Note Japanese is - on `jp.`, not `ja.`. - type: dropdown id: engine @@ -89,8 +77,8 @@ body: render: shell placeholder: | python3 playwright_scraper.py \ - --url "https://medium.com/topic/Machine-Learning" \ - --pages 3 --out grills + --url "https://medium.com/tag/python/archive/2026/09/10" \ + --pages 3 --out python_days validations: required: true @@ -110,7 +98,7 @@ body: attributes: label: What you expected instead placeholder: >- - 96 products, as the README says a category page yields. + Up to 128 stories per day, as the README says a day archive yields. validations: required: true diff --git a/.github/ISSUE_TEMPLATE/site_changed.yml b/.github/ISSUE_TEMPLATE/site_changed.yml index b044aa7..90fae17 100644 --- a/.github/ISSUE_TEMPLATE/site_changed.yml +++ b/.github/ISSUE_TEMPLATE/site_changed.yml @@ -10,22 +10,21 @@ body: something on medium.com moved. Useful to know before filing, because it narrows the fix a lot — this - parser reads **two different sources**, and they break differently: + parser reads **two payloads and two fallbacks**, and they break + differently: - - **The inline GraphQL payload**, pushed into the served HTML as - `window.ansFrontendGlobals.data.inlineQueryResults…push("{…}")`. It - carries the numeric ids, the full answer text, `numUpvotes`, - `numViews` and the creation time. A question page and a profile - have it; a topic feed does not. - - **The rendered cards**, anchored on Medium's own hooks: - `a.answer_timestamp`, `.puppeteer_test_question_title`, - `.spacing_log_answer_content`, `.spacing_log_answer_header`. + - **`window.__APOLLO_STATE__`** — the modern site: tag feeds, author + pages, story pages. + - **`window["obvInit"]({…})`** — the legacy renderer, and the only + one that answers a tag's day archive, with reading time, word count + and language on every story. + - **The page's JSON-LD and the rendered cards** (`article`, + `.streamItem`, and the author's `/@handle` link) — both fallbacks. - If the PAYLOAD moves, a run does not fail — it quietly degrades to - the cards and every count comes out null while the row count and the + If a PAYLOAD moves, a run does not necessarily fail — it degrades to + a fallback and the counts empty out while the row count and the titles stay healthy. The `data_source` column is what shows it: rows - that used to read `dom+inline` now read `dom`. If a CARD hook moves, - the run reports 0 rows and exit 4, which is loud. + that used to read `apollo` or `obvinit` now read `jsonld` or `dom`. So please include the `data_source` breakdown of your run — it is the single most useful line in the report. @@ -35,13 +34,12 @@ body: attributes: label: What stopped matching options: - - No rows at all on a page that visibly has answers (a card hook moved) - - Rows come out, but every data_source is "dom" (the payload moved) - - title is null, or wrong, on rows that clearly name a question - - author empty on every row (the named profile link moved) - - Scrolling no longer extends the feed at all - - A language host stopped being recognised - - A Space or Medium Session permalink is no longer read as an answer + - No rows at all on a page that visibly has stories + - Rows come out, but every data_source is "jsonld" or "dom" (a payload moved) + - title or author empty on rows that clearly have one + - claps look wrong (compare with the count the story page shows) + - The day-archive walk stopped, or stopped redirecting, where it should not + - A URL kind is no longer recognised (tag, archive, author, story) - Something else validations: required: true @@ -51,10 +49,9 @@ body: attributes: label: URL description: >- - The exact URL, its locale prefix, and how the browser was reached. - Please say whether the run was headful or over --cdp-endpoint: a - headless run is refused by the site and is not a parser problem. - placeholder: "https://medium.com/topic/Machine-Learning (mode: topic, headful bundled Chromium, exit: DE datacentre)" + The exact URL and how the browser was reached. Please say whether the + run was headless or headful, or over --cdp-endpoint. + placeholder: "https://medium.com/tag/python (mode: tag, headful bundled Chromium, exit: DE datacentre)" validations: required: true @@ -66,12 +63,10 @@ body: Whichever of these you can get. A capture beats a description, and `--dump-html` writes one on success as well as on failure. - - The payload as the page actually serves it: - `python3 -c "import json,re;h=open('dump.html').read();d=json.loads(re.search(r'id=.__NEXT_DATA__.[^>]*>(.*?)',h,1).group(1));print(sorted(d['props']['pageProps']))"` - - For a price problem: one whole `article.c-lot-card__container` from - devtools, and the amount the page shows for that lot. - - For a `bid_kind` problem: the exact label the card prints - ("Current bid", "Aktuelles Gebot", ...) and the locale. + - Which payload the page actually serves: + `grep -o '__APOLLO_STATE__\|obvInit' dump.html | sort | uniq -c` + - For a count problem: the story URL and the number the story page + shows. render: text validations: required: true diff --git a/CHANGELOG.md b/CHANGELOG.md index c72ca42..122e9a0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,45 @@ empty output file. that is an integer). Pinned by an offline check that drives the real client with `requests.post` stubbed; verified by control (red with the old client). +- **Help text and log lines that described a different site.** `--mode`, + `--category`, `--pages` and `--concurrency` help in the engines, and the + Scraper API client's `--url` help, described a Q&A site's questions, + topics and vote counts. `--pages`/`--concurrency` also claimed no mode has + per-page addresses, while `--mode archive` does and the Playwright worker + pool uses them. The readiness log named the donor sites' card kinds (an + answer in Playwright and pyppeteer, a rental property in Selenium); all + three now say "No story cards". +- **The Scraper API client's refusal messages** claimed this site answers the + API's exit with nothing and that a rendered browser is required. The + README measured the Scraper API served on all four modes; the messages now + say so and point at `--retries` / `--cdp-url` as the next step. +- **`diff_runs.py`** docstring, comments, `--price-tolerance-pct` help and + its `source_changed` line described answers, questions and topic feeds; + they now describe stories, tag feeds and archive/author/post views. The + `source_changed` view fields no longer include `text_chars`, a column this + repo's row does not have. +- `output_writer`'s `mode` default was `"topic"`, a mode this repo does not + have; it is now `"tag"`. Every engine passes its mode explicitly. +- **CONTRIBUTING.md and both issue templates were about another site** — its + card hooks, inline payload, columns and URLs. Rewritten from this repo's + README and parser: the two payloads and two fallbacks, and the properties + the suite pins. +- `requirements*.txt` named another repo and described another site's + browser measurements (including `playwright install chrome`, which the + README does not need); they now carry this repo's own measurements. +- `.dockerignore` ignored another repo's output prefix; it now ignores + `medium_posts.*`, the engines' real `--out` default. +- Stray donor wording in `TROUBLESHOOTING.md` and in suite comments, and a + `min_matches` check that exercised a mode named `question`. + +### Changed + +- The suite's foreign-vocabulary guard also scans `.txt` files and + `.dockerignore`, and knows the Q&A sibling's vocabulary. Both new halves + were controlled: planting the old `requirements.txt` header and the old + Selenium log line each turned the suite red on that check. (This entry + describes the old wording rather than quoting it, because CHANGELOG.md is + scanned too.) ## [0.1.1] — 2026-09-16 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a268858..e2183f5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -41,45 +41,31 @@ not the check caught it. ## Reporting a site change Medium changing its markup is the normal way this stops working, and it has -its own issue template. The detail that saves the most time is WHICH of the -two read paths broke, because this repo has two and they fail differently. +its own issue template. The detail that saves the most time is WHICH read +path broke, because this repo has four and they fail differently. -**Path 1 — the inline GraphQL payload.** Medium server-renders no content DOM -at all (measured: 0 occurrences of its own `q-box` class in the raw response -body of six captures), and what it does send is its own query results pushed -into the page: +**The two payloads.** Medium serves the same catalogue through two +renderers, and the URL decides which one answers: -``` -window.ansFrontendGlobals.data.inlineQueryResults.results[""] - .push("{\"data\":{…}}") -``` +- `window.__APOLLO_STATE__` — the modern site: tag feeds, author pages and + story pages. Normalized `Post:{id}` nodes joined by `__ref` pointers to + `User` and `Collection`. +- `window["obvInit"]({…})` — the legacy renderer, and the only one that + answers a tag's DAY archive: up to 128 stories in one 2.3 MB response, 84 + fields each, and no Apollo state on that page at all. + +If either payload moves, a run does not necessarily fail. It degrades to the +page's JSON-LD or to the rendered cards, and the row count, the titles and +the authors can stay healthy while `claps`, `reading_time_min`, `word_count` +and `language` empty out. The `data_source` column is what shows it: every +row reads `jsonld` or `dom` where it used to read `apollo` or `obvinit`. -If that push pattern moves, a run does NOT fail. It quietly degrades to the -DOM-only path and every row comes back with a null `upvotes`, `views`, -`created_at`, `answer_id` and a body truncated to what the card rendered — -while the row count, the titles and the authors all stay healthy. The -`data_source` column is what shows it: every row reads `dom` where some used -to read `dom+inline`. The canary asserts exactly that on a question URL, and -every run logs its payload coverage per batch. - -**Path 2 — the rendered cards.** Four of Medium's own hooks, all of them -semantic rather than build hashes: - -1. **`a.answer_timestamp`** — the permalink under every answer, and THE - anchor. If it moves, a run reports 0 rows and exit 4, which is loud. -2. **`.puppeteer_test_question_title`** — the question on a feed card. Absent - on a question page by design; the page's `h1` is the fallback there. -3. **`.spacing_log_answer_content` / `.puppeteer_test_answer_content`** — the - answer body, and the thing the card scope widens until it finds. -4. **`.spacing_log_answer_header`** — the block holding the author link, the - credential and the date. The credential is read by REMOVING the author and - timestamp links from a copy of it, so a change to either of those two - shows up as a mangled credential rather than as a missing one. - -A third thing can break without either path failing: the **join** between -them. When it breaks, the row count and the titles stay healthy while the -counts empty out — so if you are reporting a change, the `data_source` -breakdown of a run is the number to include. +**The two fallbacks** — the tag feed's JSON-LD, and the rendered cards +(`article` on the modern site, `.streamItem` on the legacy archive, with the +author read from a `/@handle` link that carries no story id). + +So if you are reporting a change, the `data_source` breakdown of a run is +the number to include. `--dump-html PATH` writes the exact bytes the parser was given, on success as well as failure, and a run that finds nothing writes a dump and a screenshot @@ -106,14 +92,12 @@ Then the rest of the presentation, in the order that matters: 1. `python3 smoke_test.py` green, and the canary dispatched at least once — including its WARNING branch, which is what runs when a bare GitHub runner's datacentre address is refused and no `MEDIUM_PROXY` secret is set. - This canary needs no secret to do real work: eight of fourteen fetches - were served in full with no key and no proxy, from a DATACENTRE address - at that. What it has NOT been measured doing is getting past - Cloudflare from a shared datacentre address, and since the challenge here - tracks the address's recent request rate, a runner is the worst case for - it. That is exactly why a block there is a warning rather than a failure — - until you set `MEDIUM_PROXY`, after which it is a failure, because then it - means something. + This canary needs no secret to do real work: every number in the README + was taken from a datacentre address with no key and no proxy. Since + Medium's challenge tracks the address's recent request rate, a shared + runner is the worst case for it — which is why a block there is a warning + rather than a failure until you set `MEDIUM_PROXY`, after which it is a + failure, because then it means something. 2. The repo description, homepage and topics set (see the family notes on what those should say). 3. Only then the row in the org profile README — and check it with an @@ -127,66 +111,53 @@ Then the rest of the presentation, in the order that matters: file of plain functions with inline HTML/JSON fixtures — no pytest, no conftest, no fixtures directory. Copy the nearest existing check and edit it. -Ten properties in this repo exist because they were once absent, or because -they cost a sibling repo real time. Tests pin all ten, so a PR that breaks one -will fail rather than silently regress: - -- **`sku` is the permalink, not the numeric id.** Medium publishes a numeric - `aid` and it would be the tidier key — but a topic feed, a whole mode, - carries no numeric id anywhere (measured 0 occurrences across two topic - captures in two languages), and a key null for a third of the modes cannot - be the column `diff_runs.py` joins on. `answer_id` carries the number where - the payload reached the row. -- **The counts come from the payload and are NULL on a topic row.** An - anonymous reader is shown no upvote or view count in the DOM at all — 0 - rendered nodes across five captures. `data_source` is the column that says - whether a null means zero or means nobody looked, and `diff_runs.py` - reports a count that moved together with `data_source` as - `source_changed` rather than as a change. -- **The card's author link is not the first `/profile/` link in the card.** - The first one is the avatar and has no text, so reading it gives an author - of `""` on every row while every coverage check reads 100%. -- **The card scope stops at the answer BODY, not at the second id.** §4's - "widen until you cover more than one item" never fires here: Medium wraps - each card in a chain of single-child divs, and the ancestor sixteen levels - up still holds one permalink while having absorbed the page footer - (measured: a 161,882-byte scope on one page). The stop condition is the - positive one. -- **A relative date is never parsed into a date.** The card prints "Aug 14", - "2y", "3 años" — a display string in the page's own language, and "2y" - cannot be resolved to a day. `date_text` keeps it verbatim; `created_at` - comes from the payload's epoch-MICROSECOND timestamp or is null. -- **`created_at` is microseconds.** Read as seconds it lands in the year 57 - million and as milliseconds in 57705, and both parse without error. -- **A challenge that survives its retries is BLOCKED, not empty.** This - differs from every sibling repo and was found by running the thing: with - `blocked: False` in `STATE_POLICY` the first live run handed Cloudflare's - 6 KB interstitial to the parser and reported exit 4 ("ran fine, found - nothing") on a topic holding hundreds of answers. -- **A marker that matches every good page is not a marker.** `cf-turnstile`, - `challenges.cloudflare.com` and `recaptcha` each appear on EVERY page - Medium serves — it wires Turnstile into every page and never renders it to - an anonymous reader — and `recaptcha` appears zero times on the challenge - page. None of the three is in this repo's marker set, and the suite asserts - they stay out. +These properties exist because they were once absent, or because they cost +this repo or a sibling real time. The suite pins them, so a PR that breaks +one will fail rather than silently regress: + +- **`sku` is Medium's own 12-hex post id.** It survives a story being + renamed, re-slugged or moved into a publication; none of the URLs do. +- **`url` is the story's address on Medium, not the canonical.** A + cross-posted story's canonical points at another site entirely — 4 of 20 + JSON-LD entries on one tag feed pointed at habr.com and dev.to. It is kept + in `canonical_url` instead. +- **`virtuals.recommends` is not the clap count.** It is the retired + pre-2017 recommend count — 25 beside a `totalClapCount` of 248 on the same + story — and reading it fills the column completely and wrongly. +- **A publication home page is refused.** Its `Post` nodes are + `{__typename, id}` and nothing else, so rows built from them would carry a + sku and 26 nulls while the run reported success. +- **Only the day archive has per-page addresses.** `?page=2` on a tag feed is + ignored and the feed returns its first stories again, so `--pages` and + `--concurrency` do something only in `--mode archive`, and are refused + with the reason elsewhere. +- **A day with no stories redirects up to its month**, which is the other + renderer holding a different set of stories. The walk compares the landed + URL with the requested one and stops rather than attributing one period's + stories to another. +- **A marker that matches every good page is not a marker.** + `challenge-platform` appears on every page Medium serves, and so does its + reCAPTCHA Enterprise markup (3-4 `g-recaptcha` references on pages known to + be good, 0 on the challenge). Neither is in this repo's marker set, and the + suite asserts they stay out. +- **A challenge that survives its retries is BLOCKED, not empty.** Otherwise + Cloudflare's interstitial is parsed as a feed and reported as exit 4 ("ran + fine, found nothing"). - **A run that finds nothing writes nothing.** It must not replace a good output file with `[]`. `--allow-empty` is the opt-out. - **Exit codes are a contract**, not decoration: `0` ok, `1` crash, `2` bad - usage, `3` blocked, `4` zero rows, `5` remote API error, `6` partial. A - pipeline branches on these. And a scroll that produced nothing new is - `complete` only when nothing behind it was refused — otherwise it is - `partial`, because a throttled run reporting "complete" is the failure - §7 exists to prevent. + usage, `3` blocked, `4` zero stories, `5` remote API error, `6` partial. A + pipeline branches on these. Two more that are about the fixtures rather than the code: - **A fixture is CUT from a real capture and proven to parse identically**, column for column, by `make_fixtures.py`. Never hand-written. -- **The PEOPLE in a capture are replaced before it is committed.** Author - names, profile slugs, credentials, answer bodies and the links an answer - cited all become placeholders; question titles, ids, counts, timestamps and - every piece of markup Medium generates stay verbatim. The suite asserts both - halves — that no real name survives, and that the structure did. +- **The PEOPLE in a capture are replaced before it is committed.** Every + real author name and handle becomes a pseudonym; story ids, counts, + timestamps and every piece of markup Medium generates stay verbatim. The + suite asserts both halves — that no real name survives, and that the + structure did. There is also a naming check: certain phrases are banned repo-wide and the suite fails naming them. If it trips, read the message — the phrase is wrong for a @@ -211,27 +182,21 @@ reason, not merely unfashionable. Most do not — the suite covers the parser, the writers, the captcha classifier and the CLI contract against inline fixtures. If yours genuinely needs -medium.com, say in the PR what you ran, which URL and page kind, from -which exit, and what you got — including the price and image coverage -percentages the run prints, and the scroll trace from the sidecar. Note that -a run from a datacentre address gets NO RESPONSE AT ALL, so "it returned -nothing" from a VPS is not a finding. Product counts differ by category, by -URL and by how far the scroll got, so a bare "worked for me" is not -reproducible. +medium.com, say in the PR what you ran, which URL and mode, from which exit, +and what you got — including the coverage lines the run prints and the +`data_source` breakdown. Row counts differ by tag, by day and by mode, so a +bare "worked for me" is not reproducible. **Run more than the primary engine.** "Mirror them exactly" is a design rule, -not a verification: the first live run of the pyppeteer engine crashed on its -FIRST fetch on a signature mismatch that four separate offline checks and 400 +not a verification: in a sibling repo the first live run of the pyppeteer +engine crashed on its FIRST fetch on a signature mismatch that four separate offline checks and 400 green assertions had not caught. -Do not add anything that submits the registration form. This project -deliberately never does, and a captcha token proved valid by creating a real -account is not a result worth having. - ## Scope -This repo scrapes **public pages** on Medium: category listings, search -listings and product pages, exactly as an anonymous visitor is served them. +This repo scrapes **public pages** on Medium: tag feeds, a tag's day +archives, author pages and stories, exactly as an anonymous visitor is served +them. Out of scope: anything behind a login, anything that submits a form, and anything that defeats a protection rather than passing it the way an ordinary browser does. diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md index ce9761e..67e965e 100644 --- a/TROUBLESHOOTING.md +++ b/TROUBLESHOOTING.md @@ -243,7 +243,7 @@ python3 smoke_test.py `make_fixtures.py` refuses to write a fixture that does not parse identically to its untrimmed original, so a bad trim fails loudly rather than pinning the wrong behaviour. It also replaces the people in a capture — names, profile -slugs, credentials, answer bodies, cited links — before anything is written. +handles — with pseudonyms before anything is written. --- diff --git a/diff_runs.py b/diff_runs.py index dffa623..62236e3 100644 --- a/diff_runs.py +++ b/diff_runs.py @@ -4,8 +4,8 @@ ------------- Compares two output files from this project (JSON, as written by output_writer.save) and reports what changed between them, keyed on `sku` — -the identifier the README tells people to diff on for tracking an answer's -reception over time. +Medium's own 12-hex post id, which survives a story being renamed, +re-slugged or moved into a publication. python3 diff_runs.py --old ml.2026-09-01.json \\ --new ml.2026-09-07.json @@ -17,26 +17,27 @@ python3 diff_runs.py --old "ml_$(ls -t ml_*.json | sed -n 2p)" \\ --new "ml_$(date +%F).json" --out diff.json -Four buckets, each keyed on sku — here the answer's permalink: +Four buckets, each keyed on sku — here the story's post id: added — sku present in --new, absent from --old removed — sku present in --old, absent from --new (deleted or - collapsed, or just off this particular feed run) + unpublished, or just off this particular feed run) changed — sku present in both, with a different clap count, response - count, answer count, title or body length + count, reading time, word count, title, body length, + paywall flag or publication source_changed — sku present in both with a different count, but also a different `data_source`. That is the bucket this site - needs most: claps, responses, reading time and the full - reading time and word count come from the view that - built the row and are - NULL on a row built from the rendered card alone, so a - topic run diffed against a question run would report every - count as having appeared or vanished. Reported separately + needs most: reading time and word count are null on a + tag-feed row and populated on an archive, author or post + row, the body length exists only in post mode, and a row + built from the rendered card alone carries no clap count — + so a tag run diffed against an archive run would report + those columns as having appeared or vanished. Reported separately because it says something about our own two snapshots, not about the site — and --fail-on-change deliberately ignores it. -An answer this project's parser could not recover a sku for (None) cannot be +A story this project's parser could not recover a sku for (None) cannot be matched across runs at all, so it is counted and reported separately rather than silently folded into "added"/"removed", which would be wrong on its face. """ @@ -50,13 +51,12 @@ from output_writer import UNIQUE_BY_SKU_MODES -# What is worth watching on an answer. No price, currency, discount or stock +# What is worth watching on a story. No price, currency, discount or stock # anywhere in this list, because this site has none of them — see # output_writer's docstring for why those columns do not exist on the row # either. # -# `title` IS tracked, unusually for this family: it is the QUESTION, and -# Medium lets a story be renamed, re-slugged and moved into a publication. +# `title` IS tracked, unusually for this family: Medium lets a story be renamed, re-slugged and moved into a publication. # That is a real # event and there is no other column that would show it. # @@ -112,12 +112,12 @@ def _within_tolerance(before: dict, after: dict, changes: dict, """True if every differing count field moved by less than `tolerance_pct`. Unlike in most of this family, this flag has a real use here and the - reason is worth stating. Medium's counters are LIVE: a measured story - carried 1,733 views, and a view count that ticks by a handful between two - runs of the same command is not an event anybody wants alerted on. A - monitor watching for a post going viral wants a threshold; a monitor - watching for an answer being edited wants `text_chars`, which is not a - count field and is never absorbed by this. + reason is worth stating. Medium's clap and response counts are LIVE, and + a count that ticks by a handful between two runs of the same command is + not an event anybody wants alerted on. A monitor watching for a post + going viral wants a threshold; a monitor watching for a story being + edited wants `title` or `publication`, which are not count fields and are + never absorbed by this. It still DEFAULTS TO ZERO, because the default should report what happened rather than decide for the reader what was interesting. @@ -166,22 +166,19 @@ def diff_products(old: List[dict], new: List[dict], continue # THE TWO RUNS READ DIFFERENT VIEWS, which is not a change in the - # answer — and on this site this is the bucket that matters most. + # story — and on this site this is the bucket that matters most. # - # Upvotes, views, shares, comments and the question answer count come - # from the payload the view carried, which an archive or author page - # carries and a topic page does not. So a row read off a topic feed - # has null counts and the same row read off its question page has - # real ones, and diffing the two would report every counter as having - # appeared from nowhere. `text_chars` moves for the same reason: a - # card body is truncated to three lines and the payload one is the - # whole answer. + # Reading time and word count come from the payload the view + # carried, which an archive, author or post page carries and a tag + # feed does not; a card-only row has no clap count; and the body + # length exists only in post mode. So the same story read off two + # views would report those columns as having appeared from nowhere. # # `--fail-on-change` ignores this bucket for the same reason it # ignores a tolerance move: it says which view we read, not what # changed on the site. sources = (before.get("data_source"), after.get("data_source")) - view_fields = COUNT_FIELDS + ("text_chars",) + view_fields = COUNT_FIELDS if sources[0] != sources[1] and any( f in field_changes for f in view_fields): view_part = {f: v for f, v in field_changes.items() @@ -197,12 +194,11 @@ def diff_products(old: List[dict], new: List[dict], if not field_changes: continue - # There is no lifecycle bucket on this site, and its absence is a - # measurement rather than an omission. A sibling repo needs one + # There is no lifecycle bucket on this site. A sibling repo needs one # because an auction closing moves a bid kind and the amount beside - # it in one event; an answer has no such state machine. What it does - # have -- being deleted or collapsed -- makes it vanish from the - # feed, which is the `removed` bucket. The `lifecycle` key is still + # it in one event; a story has no such state machine. Being deleted + # or unpublished makes it vanish from the feed, which is the + # `removed` bucket. The `lifecycle` key is still # emitted, always empty, so a consumer written against the family # diff shape does not have to branch. @@ -260,9 +256,9 @@ def _print_summary(result: dict) -> None: for f, v in c["changes"].items()) print(f" ? {c['sku']} {c['title']} {deltas} " f"[data_source {src['old']!r} -> {src['new']!r}: the two runs " - f"read different views of the same answer, so this is not a " - f"site-side change. A topic feed carries no counts at all; a " - f"question or profile page does]") + f"read different views of the same story, so this is not a " + f"site-side change. A tag feed carries no reading time or word " + f"count; an archive, author or post page does]") unmatchable = result["unmatchable_old"] + result["unmatchable_new"] if unmatchable: print(f"[!] {unmatchable} row(s) across both files had no sku or a " @@ -309,8 +305,8 @@ def _check_comparable(args) -> bool: # This tool's whole premise is one row per `sku`, diffed on # A mode that produces many rows per sku would give a diff # whose every line is an artefact of two rows sharing an id, so - # it is refused outright rather than answered. Both of this - # repo's current modes qualify; the check is here so that adding + # it is refused outright rather than answered. All four of this + # repo's modes qualify; the check is here so that adding # one that does not is caught rather than discovered. problems.append( f"{label} ({path}) is a {mode!r} run, which is not one row " @@ -366,7 +362,8 @@ def _check_comparable(args) -> bool: return True # A generic headline, because the reasons below are no longer only about - # completeness: a mode mismatch and a reviews run are refused too, and a + # completeness: a mode mismatch and a mode that is not one row per sku are + # refused too, and a # message naming the wrong reason sends the reader looking in the wrong # place. print("[!] Refusing to diff these two runs:") @@ -391,11 +388,11 @@ def parse_args(): "counter ticking rather than an event: reported " "separately and ignored by --fail-on-change. Default 0 " "— report every tick. Unlike in most of this family " - "the flag has a real use here: Medium's view and " - "clap counts are live, so a monitor watching for an " - "answer taking off wants a threshold, while one " - "watching for an edit wants text_chars, which this " - "never absorbs.") + "the flag has a real use here: Medium's clap and " + "response counts are live, so a monitor watching for " + "a story taking off wants a threshold, while one " + "watching for an edit wants title or publication, " + "which this never absorbs.") p.add_argument("--fail-on-change", action="store_true", help="Exit 1 if anything was added, removed or changed — " "for a cron job that should only notify on a real diff.") @@ -427,7 +424,7 @@ def main() -> int: print(f"[+] Full diff written to {args.out}") # Neither `source_changed` nor `within_tolerance` is a reason to fail. - # The first means our two runs read different views of the same answer; + # The first means our two runs read different views of the same story; # the second means a live counter ticked. Neither says anything about the # site, and alerting on either would train whoever reads the alert to # ignore it. diff --git a/output_writer.py b/output_writer.py index f1425dc..4edac36 100644 --- a/output_writer.py +++ b/output_writer.py @@ -356,7 +356,7 @@ def write_run_meta(out_prefix: str, meta: dict) -> str: def run_meta(status: str, stop_reason: str, pages_requested: int, pages_completed: int, start_url: str, final_url: str, products: int, pages_failed: Optional[List[int]] = None, - mode: str = "topic", source: str = SOURCE_DEFAULT, + mode: str = "tag", source: str = SOURCE_DEFAULT, extra: Optional[dict] = None) -> dict: """Build the metadata dict for a finished run. @@ -480,7 +480,7 @@ def finish_run(rows: Sequence[Any], out_prefix: str, fmt: str, pages_requested: int, pages_completed: int, start_url: str, final_url: str, pages_failed: Optional[List[int]] = None, - mode: str = "topic", source: str = SOURCE_DEFAULT, + mode: str = "tag", source: str = SOURCE_DEFAULT, extra: Optional[dict] = None) -> int: """Write output + the run-metadata sidecar; return the exit code. diff --git a/playwright_scraper.py b/playwright_scraper.py index acb1350..7aa67eb 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -188,11 +188,11 @@ class PageOutcome: blocked_by: Optional[str] = None load_failed: bool = False state: Optional[str] = None - # How many answers the QUESTION has in total, where the site states one. - # NOT a per-page counter and never used as one: a question page renders - # twelve answers and reports 253, so reading it as a gap would claim 241 - # missing cards on a page that rendered everything it was going to. It - # goes in the sidecar beside what the run actually read, which is what + # How many stories the TAG holds in total, where the site states one. + # NOT a per-page counter and never used as one: a tag feed renders a + # first batch of a much larger catalogue, so reading it as a gap would + # claim missing cards on a page that rendered everything it was going to. + # It goes in the sidecar beside what the run actually read, which is what # makes the difference between the two visible rather than assumed. tag_total_posts: Optional[int] = None # None, always, on this site: Medium publishes no per-page counter, and an @@ -209,16 +209,10 @@ def ok(self) -> bool: return not self.load_failed and self.blocked_by is None -# The lowest share of rows that must carry an author and an answer body -# before the read is suspect. Measured across four captures in two -# languages: 30/30, 30/30, 18/18 and 13/13 — every card that rendered -# carried both, on every one. So the floor sits high; 90% leaves room for a -# deleted author without hiding a broken read. -# -# There is deliberately NO credential floor beside it. 28/30, 27/30, 14/18 -# and 10/13 carried one, and the gap is authors who have not written a -# credential rather than a parse that missed it — a threshold there would -# fire on a correct read of a topic full of new writers. +# The lowest share of rows that must carry a title and an author before the +# read is suspect. Every measured run in the README carried both on 100% of +# rows, in every mode, so the floor sits high; 90% leaves room for a deleted +# author without hiding a broken read. FIELD_FLOOR = 90 @@ -263,8 +257,8 @@ def _scroll_to_bottom(page) -> None: """Scroll the WINDOW to the end of the document. The opposite of a sibling repo, where the body never scrolled and the - results lived in an inner container. Measured on Medium: this took a topic - feed from 20 cards to 30 in two rounds and grew the document with it. + results lived in an inner container. On Medium the window itself is what + scrolls, and the document grows with the feed. To `document.body.scrollHeight` rather than by a fixed wheel distance: a fixed wheel stopped three rounds short of the bottom on a sibling site's @@ -607,14 +601,14 @@ def handle_captcha_if_present(page, args) -> bool: render one to an anonymous reader. Which challenge a visitor meets depends on the exit country and on what the address has been doing, and a narrow detector is how a rendered challenge gets reported as an empty - topic months later. + tag months later. """ html = _content_when_settled(page) if html is None: return False # Detected is not the same as blocking. A challenge on a page whose - # answers are already rendered guards nothing, and counting the anchors is + # stories are already rendered guards nothing, and counting the anchors is # instant — which is why this check sits here rather than after the # readiness wait. The other way round would cost 25 wasted seconds on a # page the challenge genuinely gates, where solving FIRST is what makes @@ -922,7 +916,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: Optional[str]) -> P "the run is reported as partial rather than " "complete.", state_now) return outcome - # Ours, still served, and no more answers came. On an + # Ours, still served, and no more stories came. On an # infinite scroll that is the only ending there is. outcome.state = "exhausted" outcome.final_url = session.page.url @@ -1083,7 +1077,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: Optional[str]) -> P session.page.wait_for_timeout, selector, threshold, content_timeout) session.page.wait_for_timeout(500) if found < threshold: - logger.info("No answer cards appeared within %.0fs. If this feed " + logger.info("No story cards appeared within %.0fs. If this feed " "genuinely holds nothing, that is the expected " "answer and the run will report 0 rows (exit 4).", content_timeout / 1000) @@ -1448,9 +1442,9 @@ def scrape(args) -> int: extra = { "scroll": {o.page_num: o.scroll for o in outcomes if o.scroll}, "rows_new_per_batch": fresh_by_batch, - # What the QUESTION holds, where the site states it. Beside + # What the TAG holds, where the site states it. Beside # `rows` in the sidecar rather than subtracted from it: the - # difference is Medium rendering a fraction of a large question, not + # difference is Medium rendering a fraction of a large tag, not # a gap this run failed to close (§8 — an unknown gap is not a gap). "tag_total_posts": tag_total_posts, "inline_payload_rows": sum(1 for r in all_rows @@ -1487,22 +1481,23 @@ def parse_args(): "default, and passing one that disagrees with the " "URL is an error rather than an override: the mode is " "a property of the path. All four yield the same " - "row. What differs is how much of it Medium sends — a " - "question or profile page carries its answers in the " - "page, a topic page does not, so upvotes, views, " - "creation time and the full answer text are null on " - "topic rows. The data_source column says which.") + "row. What differs is how much of it Medium sends — " + "reading time, word count and language are null on a " + "tag-feed row, and the story body is read in post " + "mode only. The data_source column says which view " + "built each row.") p.add_argument("--category", default=None, help="Label to tag output rows with. Filled from the URL " - "by default — the topic, profile or question slug — " - "so it is rarely empty.") + "by default — the tag or the author — so it is rarely " + "empty.") p.add_argument("--pages", type=int, default=1, - help=f"Number of scroll BATCHES to walk (default 1, cap " - f"{PAGE_CAP}). Medium has no per-page address in any " - f"mode: `?page=2` does not fail, it is ignored and " - f"the feed returns its first items again. So a batch " - f"is one settled scroll, batches are strictly " - f"sequential, and --concurrency cannot help.") + help=f"Number of pages to walk (default 1, cap {PAGE_CAP}). " + f"In --mode archive a page is one DAY, with its own " + f"address, and page N is the day N-1 days earlier. " + f"Every other mode has no per-page address — " + f"`?page=2` is ignored and the feed returns its " + f"first items again — so a page there is one settled " + f"scroll batch.") p.add_argument("--delay", type=float, default=3.0, help="Delay between batches, seconds (default 3.0). The " "rate matters more than the address on this site: " @@ -1510,17 +1505,18 @@ def parse_args(): "took the Cloudflare challenge rate from one-in-four " "to three-in-three. This is the cheapest lever.") p.add_argument("--concurrency", type=int, default=1, metavar="N", - help="Accepted for family compatibility and REFUSED above " - "1, with the reason: a Medium feed has no per-page " - "address, so batch 5 exists only inside the browser " - "that scrolled through batches 1-4 and there is " - "nothing to hand a second worker. Run several topics " - "or profiles in parallel instead, one process each.") + help="Parallel workers, --mode archive only, where each " + "day is an independent address (default 1). REFUSED " + "above 1 everywhere else, with the reason: a tag or " + "author feed has no per-page address, so batch 5 " + "exists only inside the browser that scrolled " + "through batches 1-4. Run several tags or authors in " + "parallel instead, one process each.") p.add_argument("--retries", type=int, default=3, help="Attempts per page load before giving up (default 3). " "The pause between attempts doubles each time. A page " "that comes back EMPTY is not retried — see " - "page_flow.STATE_POLICY — because an empty search is " + "page_flow.STATE_POLICY — because an empty feed is " "a correct answer, not a fault.") p.add_argument("--retry-delay", type=float, default=2.0, help="Seconds before the first page-load retry, doubling " @@ -1674,8 +1670,8 @@ def parse_args(): if why: # Refused rather than attempted. The parser's card hooks, its path # patterns and its scroll pagination are all this site's, so pointing - # it at another Q&A site would not fail loudly — it would return zero - # rows and look like an empty topic. + # it at another site would not fail loudly — it would return zero + # rows and look like an empty tag. p.error(why) # Tracking parameters are stripped rather than kept. Medium hangs a diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index 2748dad..db99256 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -682,7 +682,7 @@ def _fetch_one_page(session, args, pool, page_num: int, content_timeout) _sleep(500) if found < threshold: - logger.info("No answer cards appeared within %.0fs. If this feed " + logger.info("No story cards appeared within %.0fs. If this feed " "genuinely holds nothing, that is the expected " "answer and the run will report 0 rows (exit 4).", content_timeout / 1000) @@ -951,11 +951,12 @@ def parse_args(): p = argparse.ArgumentParser( description="Medium story scraper (pyppeteer edition)") p.add_argument("--url", default=None, - help="A Medium URL: a topic (/topic/{Slug}), a question " - "(/{Question-Slug}) or a profile (/profile/{Slug}). " - "An answer permalink is accepted and read as its " - "question. Required, unless MEDIUM_URL is set in the " - "environment or .env.") + help="A Medium URL: a tag (/tag/{slug}), a tag's day " + "archive (/tag/{slug}/archive/{yyyy}/{mm}/{dd}), an " + "author (/@{username}) or a story (any URL ending in " + "a 12-hex post id, or /p/{id}). A publication home " + "page is refused with the reason. Required, unless " + "MEDIUM_URL is set in the environment or in .env.") p.add_argument("--mode", choices=["tag", "archive", "author", "post"], default=None, help="Which view the URL is. Inferred from the URL by " @@ -964,14 +965,17 @@ def parse_args(): help="Label to tag output rows with. Filled from the URL " "by default.") p.add_argument("--pages", type=int, default=1, - help=f"Number of scroll BATCHES to walk (default 1, cap " - f"{PAGE_CAP}). Medium has no per-page address in any " - f"mode.") + help=f"Number of pages to walk (default 1, cap {PAGE_CAP}). " + f"In --mode archive a page is one day with its own " + f"address; every other mode has no per-page address, " + f"so a page there is one settled scroll batch.") p.add_argument("--delay", type=float, default=3.0, help="Delay between batches, seconds (default 3.0).") p.add_argument("--concurrency", type=int, default=1, metavar="N", - help="Accepted for family compatibility and REFUSED above " - "1: a Medium feed has no per-batch address.") + help="Accepted for family compatibility and not " + "implemented in this engine: the worker pool lives in " + "playwright_scraper.py, and this engine walks archive " + "days one at a time with identical output.") p.add_argument("--retries", type=int, default=3, help="Attempts per page load before giving up (default 3).") p.add_argument("--retry-delay", type=float, default=2.0, diff --git a/requirements-playwright.txt b/requirements-playwright.txt index 86d25a8..2f4623d 100644 --- a/requirements-playwright.txt +++ b/requirements-playwright.txt @@ -1,17 +1,12 @@ -# The recommended engine. After installing, fetch the browser itself — and -# note WHICH browser: +# The recommended engine. After installing, fetch the browser itself: # # pip install -r requirements.txt -r requirements-playwright.txt -# playwright install chrome +# playwright install chromium # -# `chrome`, NOT `chromium`. This is the single most important line in this -# repo's setup. Measured against Vrbo on 2026-09-14, same residential -# address, seconds apart: Playwright's bundled Chromium was answered HTTP 429 -# and real Chrome HTTP 200 with the full grid. Installing chromium here gives -# you a scraper that cannot fetch its own target and looks like a blocked IP. -# -# `playwright install chrome` installs Google Chrome through the system -# package manager, so it needs the same privileges any other browser install -# would. If that is not available, `--browser-channel msedge` also drives a -# real browser; the bundled chromium is the one to avoid. +# Playwright's own bundled Chromium is enough on Medium, and that is a +# measurement rather than an assumption: headful, it was served exactly as +# well as system Chrome on the same URL. What Medium's Cloudflare refuses is +# the `HeadlessChrome` token in the User-Agent, which this engine never sends +# (see the README). `--browser-channel chrome` is available if you would +# rather drive an installed Chrome, but nothing requires it. playwright>=1.40.0 diff --git a/requirements-puppeteer.txt b/requirements-puppeteer.txt index 48d15dc..e527ba5 100644 --- a/requirements-puppeteer.txt +++ b/requirements-puppeteer.txt @@ -2,10 +2,12 @@ # # pip install -r requirements.txt -r requirements-puppeteer.txt # -# pyppeteer downloads its OWN Chromium on first run, and Vrbo refuses that -# build: measured on one address seconds apart, bundled Chromium got HTTP 429 -# and real Chrome HTTP 200 with the full grid. So this engine needs to be -# pointed at an installed browser: +# pyppeteer downloads its OWN Chromium on first run — build 117, two years +# old — and because the User-Agent is built from the browser's own version it +# truthfully announces `Chrome/117`. Medium's Cloudflare refused that three +# times out of three; pointed at an installed Chrome the same engine returned +# 53 rows first try. So this engine needs to be pointed at an installed +# browser: # # python3 puppeteer_scraper.py --chromium-path "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" ... # python3 puppeteer_scraper.py --chromium-path /usr/bin/google-chrome ... diff --git a/requirements-selenium.txt b/requirements-selenium.txt index 9ea5eaf..29f16e7 100644 --- a/requirements-selenium.txt +++ b/requirements-selenium.txt @@ -2,10 +2,8 @@ # # pip install -r requirements.txt -r requirements-selenium.txt # -# No browser download step, and on this site that is an advantage rather than -# a detail: chromedriver drives the Google Chrome you already have, which is -# exactly the browser Vrbo answers. The Playwright engine has to be told -# (`playwright install chrome`); this one gets it for free. +# No browser download step: chromedriver drives the Google Chrome you already +# have. # # Selenium 4.6+ resolves chromedriver itself through Selenium Manager, so # there is nothing else to install. diff --git a/requirements.txt b/requirements.txt index 571d1d7..cadd99c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -# vrbo-scraper — core dependencies +# medium-scraper — core dependencies # # pip install -r requirements.txt # @@ -9,7 +9,7 @@ requests>=2.31.0 # Then install exactly ONE engine — see the file per engine: # -# pip install -r requirements-playwright.txt && playwright install chrome +# pip install -r requirements-playwright.txt && playwright install chromium # pip install -r requirements-selenium.txt # pip install -r requirements-puppeteer.txt # diff --git a/scraper_api_client.py b/scraper_api_client.py index ef787fc..af97acf 100644 --- a/scraper_api_client.py +++ b/scraper_api_client.py @@ -262,12 +262,12 @@ def _run_once(args, attempt: int = 1, attempts: int = 1) -> int: f.write(html) logger.error( "The site did not serve the Scraper API's request (upstream " - "HTTP %s, %d bytes) — saved to %s. Measured 2026-09-10: the " - "Scraper API's own exit is a datacentre address, and this site " - "answers those with NOTHING, while the same task routed through " - "a Scraping Browser session returned 200 and 417 KB. Pass " - "--cdp-url. This is exit 3, distinct from an empty result " - "(exit 4).", upstream_status, len(html), dump) + "HTTP %s, %d bytes) — saved to %s. On 2026-09-16 this path was " + "served on all four modes, so a refusal is not the usual " + "answer; the dump shows which of Medium's two refusal pages it " + "was. Retry later, or route the task through a Scraping Browser " + "session with --cdp-url. " + "This is exit 3, distinct from an empty result (exit 4).", upstream_status, len(html), dump) return 3 vendor = detect_bot_challenge(html) @@ -277,8 +277,9 @@ def _run_once(args, attempt: int = 1, attempts: int = 1) -> int: vendor, len(html), ) logger.error("A challenge page is not a final answer — retry before concluding " - "anything (--retries). This site needs a rendered browser in the " - "path: pass --cdp-url, or use playwright_scraper.py / " + "anything (--retries): Medium's managed challenge was measured " + "clearing on the next attempt. If it persists, pass " + "--cdp-url, or use playwright_scraper.py / " "puppeteer_scraper.py directly.") return 3 @@ -319,11 +320,11 @@ def parse_args(): help="2captcha.com API key (sent as a Bearer token). " "Defaults to $TWOCAPTCHA_KEY, which is the safer way to pass it.") p.add_argument("--url", default=None, - help="A Medium question or profile URL. A TOPIC url " - "returns nothing here — measured, and --wait-element " - "does not help — because a topic's answers arrive " - "over a later XHR. Required, unless MEDIUM_URL is set " - "in the environment or in .env.") + help="A Medium URL: a tag feed (/tag/{slug}), a tag's day " + "archive (/tag/{slug}/archive/{yyyy}/{mm}/{dd}), an " + "author (/@{username}) or a story (/p/{id}). " + "Required, unless MEDIUM_URL is set in the " + "environment or in .env.") p.add_argument("--category", default=None, help="Label to tag output rows with. Defaults to the category segment of the URL, so the column is never empty just because the flag was omitted.") p.add_argument("--format", choices=["json", "csv", "both"], default="both") p.add_argument("--out", default="medium_posts_scraperapi", help="Output file prefix") diff --git a/selenium_scraper.py b/selenium_scraper.py index 7aa458c..19cc0de 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -705,8 +705,8 @@ def _fetch_one_page(session, args, pool, page_num: int, content_timeout) _sleep(500) if found < threshold: - logger.info("No property cards appeared within %.0fs. If this " - "search genuinely matches nothing, that is the " + logger.info("No story cards appeared within %.0fs. If this " + "feed genuinely holds nothing, that is the " "expected answer and the run will report 0 rows " "(exit 4).", content_timeout / 1000) outcome.scroll = _scroll_the_feed(session, args, html, page_num) @@ -990,18 +990,21 @@ def parse_args(): help="Label to tag output rows with. Filled from the URL " "by default.") p.add_argument("--pages", type=int, default=1, - help=f"Number of scroll BATCHES to walk (default 1, cap " - f"{PAGE_CAP}). Medium has no per-page address in any " - f"mode: `?page=2` is ignored and the feed returns its " - f"first items again.") + help=f"Number of pages to walk (default 1, cap {PAGE_CAP}). " + f"In --mode archive a page is one day with its own " + f"address; every other mode has no per-page address " + f"(`?page=2` is ignored and the feed returns its " + f"first items again), so a page there is one settled " + f"scroll batch.") p.add_argument("--delay", type=float, default=3.0, help="Delay between batches, seconds (default 3.0). The " "rate matters more than the address on this site, so " "this is the lever that matters.") p.add_argument("--concurrency", type=int, default=1, metavar="N", - help="Accepted for family compatibility and REFUSED above " - "1: a Medium feed has no per-batch address, so there " - "is nothing to hand a second worker.") + help="Accepted for family compatibility and not " + "implemented in this engine: the worker pool lives in " + "playwright_scraper.py, and this engine walks archive " + "days one at a time with identical output.") p.add_argument("--retries", type=int, default=3, help="Attempts per page load before giving up (default 3).") p.add_argument("--retry-delay", type=float, default=2.0, diff --git a/smoke_test.py b/smoke_test.py index aad8a6e..d2a9b15 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -923,12 +923,10 @@ def test_page_flow_policy(): not page_flow.should_parse("empty") and not page_flow.should_retry("empty")) ok &= check("shell is parsed after the wait, never refetched", page_flow.should_parse("shell") and not page_flow.should_retry("shell")) - # THE DIFFERENCE FROM EVERY SIBLING REPO, and it was found by running - # the thing (§15). A challenge is retried first; if the retries are spent - # and it is still a challenge, the run is BLOCKED (exit 3), not empty - # (exit 4). With this False the first live run parsed the 6 KB - # interstitial as a feed and reported "ran fine, found nothing" on a - # topic holding hundreds of answers. + # A challenge is retried first; if the retries are spent and it is still + # a challenge, the run is BLOCKED (exit 3), not empty (exit 4). With this + # False Cloudflare's interstitial would be parsed as a feed and reported + # as "ran fine, found nothing". ok &= check("a challenge that survives its retries counts as blocked", page_flow.counts_as_blocked("challenge") is True) ok &= check("but it is retried before that verdict is reached", @@ -958,13 +956,15 @@ def test_page_flow_policy(): ok &= check("MIN_CARD_MATCHES is above 1 (§5)", page_flow.MIN_CARD_MATCHES > 1) ok &= check("the anchor is the card itself, in every mode", all(page_flow.ready_selector(m) == SELECTORS["item_card"] - for m in ("topic", "question", "profile"))) - # A question with a single answer can never reach the floor, so the - # site's own answer count lowers it. - ok &= check("min_matches is clamped by what the question says it holds", - page_flow.min_matches("question", 1) == 1) + for m in ("tag", "archive", "author", "post"))) + # A caller that knows the page holds a single story lowers the floor, or + # the wait would spend its whole timeout on a page that was ready. + ok &= check("min_matches is clamped by what the caller says the page holds", + page_flow.min_matches("tag", 1) == 1) ok &= check("and is not raised above the floor", - page_flow.min_matches("question", 50) == page_flow.MIN_CARD_MATCHES) + page_flow.min_matches("tag", 50) == page_flow.MIN_CARD_MATCHES) + ok &= check("post mode waits for its one card, not two", + page_flow.min_matches("post") == 1) # Medium publishes no per-page counter. What it does publish on a day # archive is the size of the tag's WHOLE catalogue since 2003, which is # recorded beside the run and never used as an expectation: reading it as @@ -1407,7 +1407,7 @@ def row(**kw): group("lifecycle is emitted and always empty, for the family's shape") ok &= check("the key is there", "lifecycle" in diff_products([], [])) - ok &= check("and it is empty", diff_products([row()], [row(upvotes=1)]) + ok &= check("and it is empty", diff_products([row()], [row(claps=1)]) ["lifecycle"] == []) group("--fail-on-change ignores what is about US, not the site") @@ -2228,6 +2228,15 @@ def test_no_file_describes_another_site(): "farfetch": "a sibling repo", "divsrpcontentproducts": "a sibling's grid selector", "lodging-card-responsive": "a sibling's card selector", + # The Q&A sibling this repo's prose was also copied from. Its name + # never survived the copy; its vocabulary did, in help text, log + # lines, CONTRIBUTING and both issue templates. + "quora": "a sibling repo", + "upvote": "a sibling's count column", + "answer_timestamp": "a sibling's card hook", + "ansfrontendglobals": "a sibling's payload", + "no answer cards": "a sibling's log line", + "no property cards": "a sibling's log line", } # A CONTEXT allowlist, the same shape ci_checks.py uses for credentials, # because one of these words is legitimate in exactly one place. §8 says @@ -2241,9 +2250,13 @@ def test_no_file_describes_another_site(): checked = 0 for path in sorted(pathlib.Path(REPO_ROOT).rglob("*")): rel = path.relative_to(REPO_ROOT) - if not path.is_file() or path.suffix not in (".py", ".md", ".yml", - ".yaml", ".toml", - ".example"): + # `.txt` and `.dockerignore` too: the requirements files and the + # docker ignore list both carried a sibling's name past this check + # when it scanned code and prose only. + if not path.is_file() or (path.suffix not in (".py", ".md", ".yml", + ".yaml", ".toml", + ".example", ".txt") + and path.name != ".dockerignore"): continue if any(part in {"worktrees", ".venv", "venv", "build", "dist", ".git"} for part in rel.parts) or path.name.startswith("_"): @@ -2317,7 +2330,7 @@ def test_no_capture_leaks(): # And the PEOPLE, which is the other half of §10 and the half a # credential grep does not cover. make_fixtures.py replaces every author - # name, profile slug, credential and answer body before writing; these + # name and handle with a pseudonym before writing; these # are the shapes that would show a pass having been skipped. ok &= check("no author name survives except the placeholder", "Fixture Author" in text) From 533619c178bf29d07154974e26f062c5cb7f8aab Mon Sep 17 00:00:00 2001 From: Petr Date: Wed, 23 Sep 2026 15:05:12 +0000 Subject: [PATCH 2/2] Fix stale family-core lines (SECURITY releases / DataDome pointer) - SECURITY.md said the project has no releases or version tags; it has both (gh release list). Supported versions now names the latest release and main, in the wording bbb-scraper and foodpanda-scraper already use. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 3 +++ SECURITY.md | 6 +++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 122e9a0..32fc6de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,9 @@ empty output file. describes the old wording rather than quoting it, because CHANGELOG.md is scanned too.) +- `SECURITY.md` said this project has no releases or version tags; it has + both. "Supported versions" now names the latest release and `main`. + ## [0.1.1] — 2026-09-16 > **Correction to v0.1.0.** That release's README said of Medium's two diff --git a/SECURITY.md b/SECURITY.md index a30bc7a..baa7905 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -81,9 +81,9 @@ Not because these do not matter, but because they belong somewhere else: ## Supported versions -`main` only. This project has no releases or version tags; fixes land on `main` -and you update by pulling. If you are running an old clone, update before -reporting. +The latest release and `main`. Fixes land on `main` first and ship in the +next tagged release (see the Releases page). If you are running an old clone, +update before reporting. ## If you have leaked a key