From 2acf252c16fab40149232be02636c2c4414bbcef Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Wed, 23 Sep 2026 13:54:51 +0000 Subject: [PATCH] Remove donor-repo leftovers that described other sites as this one - selenium/pyppeteer: "No property cards"/"No answer cards" readiness logs now say "No event tiles ... this listing", matching Playwright - pyppeteer block advice no longer claims its bundled Chromium is refused here (README measures it served) - engines: unrecognised refusal recorded as blocked_not-served, not a sibling's challenge-page name - scraper_api_client: block message, --help description and broken --url help no longer describe a sibling site - output_writer: run_meta/finish_run mode default "markets", not "topic" - diff_runs: tolerance help, host-mismatch message and comments describe markets, not a sibling's stories/answers - captcha_solver: drop a sibling's headless-refusal paragraph - smoke_test: docstring, worker-pool test URLs and labels, two comments - .dockerignore: polymarket_markets.* instead of a sibling's prefix - bug_report template placeholder, SECURITY.md supported versions - CHANGELOG: Unreleased/Fixed entry Co-Authored-By: Claude Opus 5.5 --- .dockerignore | 3 +- .github/ISSUE_TEMPLATE/bug_report.yml | 2 +- .github/workflows/tests.yml | 2 +- CHANGELOG.md | 35 ++++++++++ SECURITY.md | 6 +- captcha_solver.py | 19 ++---- diff_runs.py | 62 ++++++++---------- output_writer.py | 4 +- playwright_scraper.py | 2 +- puppeteer_scraper.py | 18 +++--- scraper_api_client.py | 21 +++--- selenium_scraper.py | 6 +- smoke_test.py | 92 +++++++++++++-------------- 13 files changed, 142 insertions(+), 130 deletions(-) diff --git a/.dockerignore b/.dockerignore index 7c31857..96aed58 100644 --- a/.dockerignore +++ b/.dockerignore @@ -11,7 +11,8 @@ __pycache__/ .venv/ venv/ env/ -vrbo_products.* +polymarket_markets.* +polymarket_markets_*.* *_debug.html *_debug.png live_results/ diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 0042acd..9a2e6aa 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -108,7 +108,7 @@ body: attributes: label: What you expected instead placeholder: >- - 96 products, as the README says a category page yields. + Twenty events per listing page, as the README measured (70-92 markets). validations: required: true diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 378bfdd..08dde71 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -243,7 +243,7 @@ jobs: # python:3.12-slim, where the apt half is not a pip package and cannot ride # in requirements.txt. `chromium` rather than `chrome`, and unlike a # sibling repo that is not a compromise — this site was measured serving - # the bundled build the full feed. + # the bundled build the full page. docker: runs-on: ubuntu-latest steps: diff --git a/CHANGELOG.md b/CHANGELOG.md index 10fa659..70a0391 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -55,6 +55,41 @@ from a row count, it leads the section in a blockquote. client with `requests.post` stubbed; verified by control (red with the old client). +- **Donor-repo leftovers that described another site as this one.** The + family core arrived from sibling repos about a blogging site, a Q&A site + and a rental site, and some of their prose and runtime strings survived: + - user-visible: Selenium logged "No property cards appeared … this search" + and pyppeteer "No answer cards appeared … this feed"; both now say "No + event tiles … this listing", as Playwright already did. pyppeteer's block + advice claimed its bundled Chromium was "the build this site was + measured refusing" — false here, where it was measured served; it now + names the build and says so. The Scraper API client's block message + cited a sibling's 2026-09-10 datacentre refusal, its `--help` + description a sibling's archive payload and author pages, and its + `--url` help ended in a broken sentence about a topic's answers; + - the sidecar's `stop_reason` for an unrecognised refusal was + `blocked_bot-or-not`, a sibling's challenge-page name; it is now + `blocked_not-served`; + - `output_writer.run_meta` / `finish_run` defaulted `mode` to a sibling's + `"topic"`, which `diff_runs.py` would have refused as not one row per + sku. The engines always pass `--mode`, so no run was affected; the + default is now `"markets"`; + - `diff_runs.py`: its `--price-tolerance-pct` help and three comments + explained a sibling's live counters, per-story languages and missing + lifecycle, and its host-mismatch refusal said the site "serves one story + from several addresses". All now describe markets; + - `captcha_solver.py` said this site refuses a headless browser with a + 394-byte "Access Denied" page. The README measures headless and headful + identical and no challenge met; the paragraph now says that; + - `smoke_test.py`: the docstring described a sibling's column, sign-in + widget and a test this repo does not have, and claimed the fixtures + pseudonymise author handles (`make_fixtures.py` rewrites nothing). The + worker-pool test fed a sibling's archive URLs and counted "days"; it now + uses event-page URLs, asserting the same behaviour; + - `.dockerignore` excluded a sibling's output prefix instead of this + repo's `polymarket_markets.*`; the bug-report template's "expected" + placeholder quoted a sibling's 96 products; `SECURITY.md` said the + project has no releases. ## [0.1.1] — 2026-09-16 diff --git a/SECURITY.md b/SECURITY.md index 40fbcbe..a5e8c0f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -81,9 +81,9 @@ Not because these do not matter, but because they belong somewhere else: ## Supported versions -`main` only. This project has no releases or version tags; fixes land on `main` -and you update by pulling. If you are running an old clone, update before -reporting. +The latest release and `main`. Fixes land on `main` first and ship in the next +tagged release; if you are running an old clone or an older release, update +before reporting. ## If you have leaked a key diff --git a/captcha_solver.py b/captcha_solver.py index 9097343..5957310 100644 --- a/captcha_solver.py +++ b/captcha_solver.py @@ -1138,16 +1138,11 @@ def turnstile_task_for(challenge: CaptchaChallenge) -> dict: # a JPEG of distorted text and a GET form). Roughly 190 lines of it, and none # of it is ported here, because this site has no such page. # -# What this site does instead is refuse a HEADLESS browser. Measured -# 2026-09-10 from five different addresses, four of them residential: one sibling's bot manager -# answers with HTTP 403 and a 394-byte "Access Denied" page carrying a -# reference id — no form, no image, no widget, nothing for a solver to -# answer. And the trigger is the CLIENT rather than the address: the very -# same addresses were served HTTP 200 and the full catalogue by a browser -# with a real window. So the response to a block here is `--headful` or -# `--cdp-endpoint`, not a solve and not a better proxy, and -# product_parser.detect_page_state reports it as "blocked" rather than -# "challenge" precisely so no solve is attempted and nothing is charged. +# Nor does this site refuse a headless browser: headless and headful were +# measured identical on polymarket.com, and no challenge was met in any +# capture (README). What a block would look like here is Cloudflare's — the +# site sits behind it — and a managed challenge renders a Turnstile, which +# the machinery above does implement. # # The reCAPTCHA / hCaptcha / Turnstile machinery above IS kept, and that is a # deliberate asymmetry rather than an inconsistency. Detection stays broad @@ -1155,6 +1150,6 @@ def turnstile_task_for(challenge: CaptchaChallenge) -> dict: # what the address has been doing — a narrow list is how a challenge gets # reported as an empty page months later. A solver for a challenge this site # has never been observed to serve is dead code; a DETECTOR for one is cheap -# insurance: a detection that fires on a page whose lots have +# insurance: a detection that fires on a page whose events have # already rendered guards nothing, which is why the default is -# `when-blocked` and why it counts lot links before it spends. +# `when-blocked` and why it counts event links before it spends. diff --git a/diff_runs.py b/diff_runs.py index 6e5c6ab..372ff2a 100644 --- a/diff_runs.py +++ b/diff_runs.py @@ -4,8 +4,8 @@ ------------- Compares two output files from this project (JSON, as written by output_writer.save) and reports what changed between them, keyed on `sku` — -the identifier the README tells people to diff on for tracking an answer's -reception over time. +the identifier the README tells people to diff on for tracking a market's +price, book and state over time. python3 diff_runs.py --old ml.2026-09-01.json \\ --new ml.2026-09-07.json @@ -20,8 +20,8 @@ Four buckets, each keyed on sku — here the market's slug: added — sku present in --new, absent from --old - removed — sku present in --old, absent from --new (deleted or - collapsed, or just off this particular feed run) + removed — sku present in --old, absent from --new (closed or + delisted, or just off this particular listing run) changed — sku present in both, with a different price, order book, volume, liquidity, state or question source_changed — sku present in both with a different value, but also a @@ -48,11 +48,6 @@ from output_writer import UNIQUE_BY_SKU_MODES -# What is worth watching on an answer. No price, currency, discount or stock -# anywhere in this list, because this site has none of them — see -# output_writer's docstring for why those columns do not exist on the row -# either. -# # `title` IS tracked, unusually for this family: it is the QUESTION, and # Polymarket lets a market be re-worded and re-slugged. # WHAT COUNTS AS A CHANGE ON THIS SITE, and why each of these and not more. @@ -192,10 +187,9 @@ def diff_products(old: List[dict], new: List[dict], # THE TWO RUNS READ DIFFERENT VIEWS, which is not a change in the # answer — and on this site this is the bucket that matters most. # - # Upvotes, views, shares, comments and the question answer count come - # from the payload the view carried, which an archive or author page - # carries and a topic page does not. So a row read off a topic feed - # has no spread and carries its EVENT's volume, and the same market + # `spread`, the volume windows and the on-chain ids come from the + # payload the view carried, which an event page carries and a listing + # does not. So a row read off a listing has no spread and carries its EVENT's volume, and the same market # read off its own event page has a spread and the MARKET's volume. # Diffing the two would report every one of them as having appeared # from nowhere or collapsed by two orders of magnitude. @@ -232,18 +226,16 @@ def diff_products(old: List[dict], new: List[dict], if not field_changes: continue - # There is no lifecycle bucket on this site, and its absence is a - # measurement rather than an omission. A sibling repo needs one - # because an auction closing moves a bid kind and the amount beside - # it in one event; an answer has no such state machine. What it does - # have -- being deleted or collapsed -- makes it vanish from the - # feed, which is the `removed` bucket. The `lifecycle` key is still - # emitted, always empty, so a consumer written against the family - # diff shape does not have to branch. + # There is no separate lifecycle bucket here: a market's state + # transitions (`active`, `closed`, `accepting_orders`) are TRACKED + # fields, so a market closing lands in `changed` beside everything + # else about it. The `lifecycle` key is still emitted, always empty, + # so a consumer written against the family diff shape does not have + # to branch. # A counter ticking rather than a real move -- see # `_within_tolerance`. Only when the ONLY differences are count - # fields: a title or a body length changing alongside is a real + # fields: a title or an end date changing alongside is a real # change whatever the size of the move. if (all(f in COUNT_FIELDS for f in field_changes) and _within_tolerance(before, after, field_changes, @@ -374,14 +366,10 @@ def _check_comparable(args) -> bool: # consistently on a DIFFERENT single host, because then `added` and # `removed` would be describing the address rather than the catalogue. # - # There is deliberately no language check, and the reason is worth - # writing down because the sibling repo this file came from has one. - # There, `language` meant which of twenty-four language SITES a row came - # from, and those are different catalogues. Here it is the site's own - # `detectedLanguage` for the STORY — one measured English tag's day - # archive carried 123 `en` rows and 5 `id` ones — so refusing a pair - # whose languages differ would refuse two perfectly comparable runs of - # the same tag. + # There is deliberately no language check: a locale path translates the + # words and leaves the ids and the prices alone (the suite pins this on + # the /es/ fixtures), so two runs of one listing under different locales + # are still the same markets, joined on the same sku. sources = {} for label, path in (("--old", args.old), ("--new", args.new)): try: @@ -394,7 +382,7 @@ def _check_comparable(args) -> bool: if len(set(sources.values())) > 1: problems.append( f"the two runs landed on different hosts ({sources}). This site " - f"serves one story from several addresses, so this is usually a " + f"serves every locale from one host, so this is usually a " f"different URL rather than a different catalogue — but added " f"and removed would describe the address change rather than " f"anything about the markets.") @@ -428,11 +416,11 @@ def parse_args(): "counter ticking rather than an event: reported " "separately and ignored by --fail-on-change. Default 0 " "— report every tick. Unlike in most of this family " - "the flag has a real use here: this site's own " - "clap counts are live, so a monitor watching for an " - "answer taking off wants a threshold, while one " - "watching for an edit wants text_chars, which this " - "never absorbs.") + "the flag has a real use here: a market's price " + "moves continuously, so a monitor watching for a " + "real repricing wants a threshold, while one " + "watching for a market closing wants `closed`, " + "which this never absorbs.") p.add_argument("--fail-on-change", action="store_true", help="Exit 1 if anything was added, removed or changed — " "for a cron job that should only notify on a real diff.") @@ -464,7 +452,7 @@ def main() -> int: print(f"[+] Full diff written to {args.out}") # Neither `source_changed` nor `within_tolerance` is a reason to fail. - # The first means our two runs read different views of the same answer; + # The first means our two runs read different views of the same market; # the second means a live counter ticked. Neither says anything about the # site, and alerting on either would train whoever reads the alert to # ignore it. diff --git a/output_writer.py b/output_writer.py index 7f01e72..7b3fc45 100644 --- a/output_writer.py +++ b/output_writer.py @@ -442,7 +442,7 @@ def write_run_meta(out_prefix: str, meta: dict) -> str: def run_meta(status: str, stop_reason: str, pages_requested: int, pages_completed: int, start_url: str, final_url: str, products: int, pages_failed: Optional[List[int]] = None, - mode: str = "topic", source: str = SOURCE_DEFAULT, + mode: str = "markets", source: str = SOURCE_DEFAULT, extra: Optional[dict] = None) -> dict: """Build the metadata dict for a finished run. @@ -565,7 +565,7 @@ def finish_run(rows: Sequence[Any], out_prefix: str, fmt: str, pages_requested: int, pages_completed: int, start_url: str, final_url: str, pages_failed: Optional[List[int]] = None, - mode: str = "topic", source: str = SOURCE_DEFAULT, + mode: str = "markets", source: str = SOURCE_DEFAULT, extra: Optional[dict] = None) -> int: """Write output + the run-metadata sidecar; return the exit code. diff --git a/playwright_scraper.py b/playwright_scraper.py index 364ddde..bb30a1b 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -1072,7 +1072,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: Optional[str]) -> P logger.error("%s", page_flow.block_advice( html, headless=bool(getattr(args, "headless", False)), has_pool=has_pool)) - outcome.blocked_by = vendor or ("no-response" if not html else "bot-or-not") + outcome.blocked_by = vendor or ("no-response" if not html else "not-served") outcome.final_url = session.page.url return outcome diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index 79eca69..7e66b42 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -685,14 +685,15 @@ def _fetch_one_page(session, args, pool, page_num: int, debug_html) if not args.chromium_path and not args.cdp_endpoint: logger.error( - "This engine launched pyppeteer's OWN Chromium, which is the " - "build this site was measured refusing. Pass --chromium-path " - "pointing at an installed Chrome before concluding anything " - "about the address.") + "This engine launched pyppeteer's OWN Chromium (build " + "117.0.5938.0), the one thing that differs from its twins. " + "polymarket.com was measured serving it, but pass " + "--chromium-path pointing at an installed Chrome before " + "concluding anything about the address.") logger.error("%s", page_flow.block_advice( html, headless=bool(getattr(args, "headless", False)), has_pool=has_pool)) - outcome.blocked_by = vendor or ("no-response" if not html else "bot-or-not") + outcome.blocked_by = vendor or ("no-response" if not html else "not-served") outcome.final_url = _current_url(session) return outcome @@ -713,9 +714,10 @@ def _fetch_one_page(session, args, pool, page_num: int, content_timeout) _sleep(500) if found < threshold: - logger.info("No answer cards appeared within %.0fs. If this feed " - "genuinely holds nothing, that is the expected " - "answer and the run will report 0 rows (exit 4).", + logger.info("No event tiles appeared within %.0fs. If this " + "listing genuinely holds nothing, that is the " + "expected answer and the run will report 0 rows " + "(exit 4).", content_timeout / 1000) outcome.scroll = _scroll_the_feed(session, args, html, page_num) html = _content(session) or html diff --git a/scraper_api_client.py b/scraper_api_client.py index a4063b3..f20edd0 100644 --- a/scraper_api_client.py +++ b/scraper_api_client.py @@ -259,11 +259,11 @@ def _run_once(args, attempt: int = 1, attempts: int = 1) -> int: f.write(html) logger.error( "The site did not serve the Scraper API's request (upstream " - "HTTP %s, %d bytes) — saved to %s. Measured 2026-09-10: the " - "Scraper API's own exit is a datacentre address, and this site " - "answers those with NOTHING, while the same task routed through " - "a Scraping Browser session returned 200 and 417 KB. Pass " - "--cdp-url. This is exit 3, distinct from an empty result " + "HTTP %s, %d bytes) — saved to %s. All four modes were measured " + "returning HTTP 200 through this API on 2026-09-16, so check the " + "dump for Cloudflare's block page before concluding anything; " + "--cdp-url routes the task through a Scraping Browser session " + "instead. This is exit 3, distinct from an empty result " "(exit 4).", upstream_status, len(html), dump) return 3 @@ -306,10 +306,9 @@ def parse_args(): description="Polymarket market scraper — 2captcha Scraper API edition " "(no local browser). Measured 2026-09-16 at $0.0005 a " "request: all four modes returned HTTP 200 and rows " - "identical to a local browser's, including the whole " - "2.3 MB archive-day payload in one request. It cannot " - "scroll, which costs nothing except on an author page — " - "see the module docstring.") + "identical to a local browser's. It cannot walk " + "--mode events, which needs a browser engine's page " + "loop — see the module docstring.") # NOT required: prefer the TWOCAPTCHA_KEY env var. A key passed on the # command line is visible to anyone who can run `ps`, and it lands in # shell history and in any log that echoes the command line. @@ -318,9 +317,7 @@ def parse_args(): "Defaults to $TWOCAPTCHA_KEY, which is the safer way to pass it.") p.add_argument("--url", default=None, help="A Polymarket listing or event URL. " - "returns nothing here — measured, and --wait-element " - "does not help — because a topic's answers arrive " - "over a later XHR. Required, unless POLYMARKET_URL is set " + "Required, unless POLYMARKET_URL is set " "in the environment or in .env.") p.add_argument("--category", default=None, help="Label to tag output rows with. Defaults to the category segment of the URL, so the column is never empty just because the flag was omitted.") p.add_argument("--format", choices=["json", "csv", "both"], default="both") diff --git a/selenium_scraper.py b/selenium_scraper.py index c93c5bc..e0c924a 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -707,7 +707,7 @@ def _fetch_one_page(session, args, pool, page_num: int, logger.error("%s", page_flow.block_advice( html, headless=bool(getattr(args, "headless", False)), has_pool=has_pool)) - outcome.blocked_by = vendor or ("no-response" if not html else "bot-or-not") + outcome.blocked_by = vendor or ("no-response" if not html else "not-served") outcome.final_url = _current_url(session) return outcome @@ -728,8 +728,8 @@ def _fetch_one_page(session, args, pool, page_num: int, content_timeout) _sleep(500) if found < threshold: - logger.info("No property cards appeared within %.0fs. If this " - "search genuinely matches nothing, that is the " + logger.info("No event tiles appeared within %.0fs. If this " + "listing genuinely holds nothing, that is the " "expected answer and the run will report 0 rows " "(exit 4).", content_timeout / 1000) outcome.scroll = _scroll_the_feed(session, args, html, page_num) diff --git a/smoke_test.py b/smoke_test.py index cecfd54..a3820f1 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -14,8 +14,8 @@ own venv and fails if that skip list is non-empty. The fixtures are cut from real captures by `make_fixtures.py`, which proves -each one parses IDENTICALLY to its untrimmed original, column for column, -and replaces every real author handle with a pseudonym. Do not hand-edit +each one parses IDENTICALLY to its untrimmed original, column for column. +Nothing in them is rewritten — see make_fixtures.py for why. Do not hand-edit them. WHAT THIS SUITE IS FOR, beyond the obvious @@ -24,18 +24,18 @@ a sibling. The ones worth knowing about before you change anything: * `test_values_on_real_fixtures` asserts VALUES, not coverage. A column can - be 100% populated and entirely wrong. The one that matters here is - `claps`: Medium's legacy payload carries `virtuals.totalClapCount` AND - `virtuals.recommends` side by side, both populated on 128 of 128 stories, - and the second is the retired pre-2017 recommend count — 25 against 248 - on the same story. Reading it would have filled the column completely and - wrongly, and no coverage check would have said a word. - - * `test_markers_do_not_match_a_good_page` is the §18 rule as a test. Medium - ships reCAPTCHA markup for its own sign-in widget on every page it - serves, and `challenge-platform` appears twice on good and refused pages - alike. A marker that matches every page is worse than no marker, so every - marker in every set is asserted ABSENT from four pages known to be good. + be 100% populated and entirely wrong. The ones that matter here are + `price` (outcomePrices[0], index-aligned with `outcomes`, which are not + always Yes/No) and `volume`, which is the MARKET's exact figure on an + event page and the EVENT's on a listing — `volume_scope` says which, and + reading the wrong one fills the column completely and wrongly. + + * `test_markers_do_not_match_a_good_page` is the §18 rule as a test. A + marker that matches every page is worse than no marker, so every marker + in every set is asserted ABSENT from every page known to be good — and + `cf-turnstile` is kept out of the set because 2Captcha's own auto-solve + extension injects it into a page fetched over the Scraping Browser + (the `cdp_extension` fixture). * `test_engine_parity` binds every shared-module call in every engine against the callee's REAL signature. Two engines in a sibling repo called @@ -44,11 +44,6 @@ own first live run hit the same class twice — `scroll_until_settled` was called with a `target=` this module no longer takes, and Selenium passed `session.driver` to a helper that wanted `session`. - - * `test_publication_pages_are_refused` pins the decision that keeps this - repo honest: a publication home page carries post IDS and no post data, - and rows built from it would hold a sku and 26 nulls while the run - reported success. """ import ast @@ -871,12 +866,11 @@ def test_page_flow_policy(): not page_flow.should_parse("empty") and not page_flow.should_retry("empty")) ok &= check("shell is parsed after the wait, never refetched", page_flow.should_parse("shell") and not page_flow.should_retry("shell")) - # THE DIFFERENCE FROM EVERY SIBLING REPO, and it was found by running - # the thing (§15). A challenge is retried first; if the retries are spent - # and it is still a challenge, the run is BLOCKED (exit 3), not empty - # (exit 4). With this False the first live run parsed the 6 KB - # interstitial as a feed and reported "ran fine, found nothing" on a - # topic holding hundreds of answers. + # A challenge is retried first; if the retries are spent and it is still + # a challenge, the run is BLOCKED (exit 3), not empty (exit 4). With this + # False, a Cloudflare interstitial would be parsed as a listing and + # reported as "ran fine, found nothing" — the wrong verdict and the + # wrong advice. ok &= check("a challenge that survives its retries counts as blocked", page_flow.counts_as_blocked("challenge") is True) ok &= check("but it is retried before that verdict is reached", @@ -1655,16 +1649,16 @@ def test_driver_primitives_tolerate_a_navigation(): def test_concurrency_machinery(skips): """§10: drive the worker pool with the browser stubbed out. - A live run cannot reach this. Page 1 is fetched alone and its answer - decides whether the rest may be addressed, so a blocked page 1 means the - workers never start — and on this site page 1 is blocked often enough - that a live test would pass by not running. + A live run cannot always reach this. Page 1 is fetched alone and its + answer decides whether the rest may be addressed, so a blocked page 1 + means the workers never start and a live test would pass by not running. - The pool only exists in the Playwright engine (Selenium and pyppeteer - walk the days one at a time and say so), which is why this group targets - that engine alone. + The pool only exists in the Playwright engine, and only in --mode events, + where page 1 is the listing and pages 2..N are its event pages (Selenium + and pyppeteer walk them one at a time and say so), which is why this + group targets that engine alone. """ - group("the archive worker pool, with no browser in it") + group("the event-page worker pool, with no browser in it") ok = True try: import playwright_scraper as eng @@ -1715,7 +1709,7 @@ def __exit__(self, *exc): eng.sync_playwright = _NoPlaywright try: - specs = [(n, "https://medium.com/tag/python/archive/2026/09/%02d" % n) + specs = [(n, "https://polymarket.com/event/test-event-%02d" % n) for n in pages] return eng._fetch_pages_concurrently(_Args(), None, specs, concurrency), seen @@ -1735,7 +1729,7 @@ def good(page_num, url, rows=3): # hole the run would report as complete. (results, unattempted, exhausted), seen = _run( list(range(2, 10)), lambda n, u: good(n, u)) - ok &= check("every queued day is fetched", sorted(seen) == list(range(2, 10))) + ok &= check("every queued event page is fetched", sorted(seen) == list(range(2, 10))) ok &= check("...exactly once", len(seen) == len(set(seen))) ok &= check("every fetch produced an outcome", len(results) == 8) ok &= check("nothing is left unattempted when all succeed", unattempted == []) @@ -1750,18 +1744,18 @@ def good(page_num, url, rows=3): ok &= check("...and each kept its own URL", all(str(o.page_num).zfill(2) in o.url for o in ordered)) - # 3. A day with no rows ends dispatch. Without this, asking for 40 days - # of a tag that published on three fetches 37 empty ones. + # 3. A page with no rows ends dispatch. Without this, asking for 40 + # pages of a listing that holds three fetches 37 empty ones. def empty_after_4(page_num, url): return good(page_num, url, rows=0 if page_num >= 4 else 3) (results, unattempted, exhausted), seen = _run( list(range(2, 40)), empty_after_4, concurrency=2) - ok &= check("an empty day stops dispatch", exhausted) + ok &= check("an empty page stops dispatch", exhausted) ok &= check("...and most of the queue is never fetched", len(seen) < 12) - ok &= check("...with the unfetched days reported, not counted as failed", + ok &= check("...with the unfetched pages reported, not counted as failed", len(unattempted) == 38 - len(seen)) - ok &= check("unattempted days are page NUMBERS, in order", + ok &= check("unattempted pages are page NUMBERS, in order", unattempted == sorted(unattempted)) # 4. A worker that raises must not hang the run and must not take its @@ -1909,13 +1903,13 @@ def test_engine_parity(skips): ok &= check("Selenium says there is nothing to choose", "there is nothing to choose" in sel) pup = _engine_source("puppeteer_scraper") or "" - # The opposite of what a sibling repo asserts, and it is measured: this - # engine's OWN bundled Chromium is build 117.0.5938.0, the UA follows the - # browser's real version, and Medium refused it 3 times out of 3. The - # engine must say so in its docstring and in its block advice, because a - # reader who concludes "my address is burned" from that is going to buy a - # proxy they do not need. - ok &= check("pyppeteer names the Chromium build it is refused on", + # This engine's OWN bundled Chromium is build 117.0.5938.0 and the UA + # follows the browser's real version, so it announces `Chrome/117`. On a + # sibling site that build was refused; polymarket.com was measured + # serving it (README). The engine and the README both name the build and + # --chromium-path, so a reader who does meet a refusal knows the one + # thing that differs between this engine and its twins. + ok &= check("pyppeteer names the Chromium build it bundles", "117.0.5938.0" in pup) ok &= check("...and tells the reader to pass --chromium-path", "--chromium-path" in pup) @@ -2165,8 +2159,8 @@ def test_no_file_describes_another_site(): # an issue template about seller feedback scores. All four were copied in # with the family core and all four read as authoritative. # - # Nothing here can tell a paragraph about Medium from a paragraph about a - # rental site in general. What it CAN do is notice the vocabulary of the + # Nothing here can tell a paragraph about a sibling site from a paragraph + # about prediction markets in general. What it CAN do is notice the vocabulary of the # specific siblings this repo was copied from, which is where the real # leakage comes from. foreign = {