Walk every event page, and stop reading every page as its own 404

.gitignore

@@ -60,8 +60,19 @@ dist/
 build/
 
 # --- environments --------------------------------------------------------
-.venv/
-venv/
+# Virtualenvs, by SHAPE rather than by the three names somebody thought of.
+# The README tells people to make one per engine and does not say where, so
+# they land in the working tree under whatever name the reader picked — a
+# `.venv-pw` made while working on this repo was untracked and NOT ignored,
+# one `git add -A` from being committed. A name is what churns.
+#
+# A glob still cannot recognise a virtualenv the way a program can (the
+# marker is `pyvenv.cfg`), so smoke_test.py asserts that every directory
+# holding one is actually ignored — this list is the cheap half and that
+# check is the one that cannot be out-named.
+.venv*/
+venv*/
+*venv/
 env/
 
 # --- editors / OS --------------------------------------------------------

CHANGELOG.md

@@ -10,8 +10,102 @@ from a row count, it leads the section in a blockquote.
 
 ## [Unreleased]
 
+### Fixed
+
+- **`--mode events` stopped at the first event page with no markets and
+  reported the run as `complete`.** Found by a third-party audit of 8
+  October and real. Reproduced against `main` end to end: one empty page at
+  position 4 of 20 left **17 event pages never fetched**, with `status:
+  complete`, exit 0, and `pages_completed: 4 of 21` in the same sidecar —
+  a file contradicting itself, the way a run with FAILED pages used to
+  before that case was fixed.
+
+  The rule arrived from a sibling whose pages are consecutive days, where
+  stopping early genuinely saves fetches. Here pages 2..N are the event URLs
+  page 1 named, capped at `--pages - 1`, so there is no catalogue left to
+  exhaust and nothing to save: the queue is exactly as long as the listing's
+  own event list. The walk now visits every planned URL, records the empty
+  ones in the sidecar as `pages_without_rows`, and keeps `complete` for a
+  run that actually finished its plan. `no_new_products` is set nowhere now
+  and has been removed from `COMPLETE_STOP_REASONS` rather than left as
+  policy with no reader.
+
+- **Every page the site serves was classified as its own 404 — and 12 of 20
+  event pages had every row discarded because of it.** Found while measuring
+  whether an empty event page is an edge case; it is not, because the
+  classifier was manufacturing them. The two defects compound into a
+  silently truncated run reported as complete, which is worse than either.
+
+  Next.js inlines its 404 TEMPLATE — a `"notFound":[…]` branch the router
+  would render if the route had 404'd — into the payload of every page.
+  `looks_not_found` scanned the first 200,000 bytes for that template's
+  wording. Counted 2026-10-09 over the twenty event pages `/predictions`
+  named, as the site serves them: the wording is on **20 of 20**, inside the
+  scanned window on **17 of 20** (always at offset ~134,000), rendered into
+  the body markup on **0 of 5** checked — and **12 of 20 held markets and
+  were read as "this address does not exist"**, 5 to 53 rows each, thrown
+  away by `empty`'s `parse: False`.
+
+  So the markers are gone rather than narrowed: a candidate that is on every
+  good page is not a marker, it is a fact about the site. The content check
+  moved to the top, where a page whose payload names markets is content
+  whatever templates it also ships. Two measurements decided the shape of
+  the fix: a slug that does not exist answers **HTTP 200** (4 of 4), so the
+  status was never the backstop the markers were assumed to have — the
+  docstring promising "a real HTTP 404" was simply wrong — and `"digest"`,
+  which looked like a clean replacement, is on 2 of 10 good pages and was
+  not adopted.
+
+  The browser engines escaped this by byte position rather than by design: a
+  browser serialises rendered markup first and puts the template at 511,547
+  where the served bytes put it at 134,033, on the same URL. The margin on
+  the smallest page measured was 67 KB. `scraper_api_client.py` reads the
+  served bytes and had no margin at all.
+
+- **A nonexistent event cost 67 seconds instead of 8**, as a direct
+  consequence of the reclassification above: `shell` spends the readiness
+  wait and four scroll rounds, every round logging "added no events
+  (0 -> 0)". That wait exists for one case — the payload never arrived — so
+  it is now gated on the payload ARRIVING rather than on it naming markets
+  (`page_flow.payload_arrived`, consulted by `is_unpainted` and by all three
+  engines' fast path). Back to 7.8s, same exit 4.
+
+- **The tree-wide scanners walked into a virtualenv.** Not in the audit —
+  the suite failed on its own working tree. The scanner widened to the whole
+  tree in the previous release reported that pip's vendored
+  `charset_normalizer` "describes this site", because it skipped virtualenvs
+  by NAME and this one is called `.venv-pw`. `.github/ci_checks.py` had
+  already been taught to recognise a venv by `pyvenv.cfg`; the four scanners
+  in `smoke_test.py` had not. It is the first thing a new user sees from
+  `python3 smoke_test.py`, since the README tells them to make a virtualenv
+  per engine and never says where.
+
+- **A virtualenv in the working tree was not ignored.** `.gitignore` listed
+  `.venv`, `venv` and `env`; the one made while working on this change is
+  called `.venv-pw`, so it was untracked and NOT ignored — one `add -A` from
+  being committed, which is the same name-versus-kind mistake as the
+  scanners above. The patterns are broader now, and because a glob can
+  always be out-named, the suite also asserts that every directory holding a
+  `pyvenv.cfg` really is ignored.
+
 ### Added
 
+- **`captures/event_served_bytes.html` — the first capture of what the HTTP
+  client path actually sees.** All eleven existing captures are a browser's
+  `page.content()`, which is a different document shape from the same URL,
+  and that is precisely why the 404-template defect was invisible to a green
+  suite. `make_fixtures.py` now carries the `"notFound"` branch through the
+  trim on purpose: without it the new checks could not fail, which the first
+  version of this fixture demonstrated by passing against the unfixed
+  parser.
+
+- **A check that every tree-wide scanner skips virtualenvs structurally**,
+  pinned as wiring rather than by content. The scanners fail three different
+  ways — a false FAILURE on the describes-this-site scan, a false PASS on
+  the dead-name corpus (vendored code keeps a dead name alive), and no
+  effect at all on the banned-wording scan — and only the first is visible
+  from a planted fault.
+
 - **The repo is public**, and the published surfaces are now checkable. Its
   GitHub description had shipped carrying the wording §12 bans for the
   Scraping Browser API, straight out of the family template — caught by hand

README.md

@@ -394,8 +394,11 @@ picked up, without printing secrets.
 * **Exit codes**: `0` ok · `1` crash · `2` bad usage · `3` blocked · `4` zero
   rows · `5` remote API error · `6` partial.
 * **`<out>.meta.json`** per run: status, stop reason, which pages failed by
-  number, the site's own `total_events`, the row counts per page, and the
-  category and locale the URL carried.
+  number, which event pages held no markets (`pages_without_rows`), the
+  site's own `total_events`, the row counts per page, and the category and
+  locale the URL carried. The last two of those answer different questions:
+  a page that FAILED is a hole in the data, a page with no rows is a fact
+  about that event, and `--mode events` keeps walking past the second.
 * **An empty CSV still carries its header.**
 * `diff_runs.py` compares two runs by `sku` and refuses a pair whose `mode`
   differs — a shallow run against a deep one would report a dozen columns as

make_fixtures.py

@@ -111,6 +111,23 @@
     # for the wrong reason, because its only fixture was curl-fetched).
     "cdp_extension":  ("cdp_scraping_browser.html",
                        "https://polymarket.com/predictions/crypto", 5),
+    # THE FIXTURE THE CLASSIFIER ACTUALLY NEEDS, for the same reason the one
+    # above is the one the marker checks need: every other event capture here
+    # is a browser's `page.content()`, and a browser serialises the rendered
+    # markup FIRST. The bytes the server sends interleave the inlined payload
+    # much earlier, which is what `scraper_api_client.py` and any HTTP client
+    # reads — a different document shape from the same URL.
+    #
+    # Measured 2026-10-09 on this very page: Next.js inlines its 404
+    # TEMPLATE at offset 134,033 in the served bytes and at 511,547 in the
+    # browser's DOM. The not-found check scanned the first 200,000, so it
+    # matched a template on a page holding five markets, classified it
+    # `empty`, and `empty` carries `parse: False` — every row discarded, on
+    # 12 of the 20 event pages `/predictions` named that day. Eleven browser
+    # captures could not show it and did not.
+    "event_served_bytes": ("event_served_bytes.html",
+                           "https://polymarket.com/event/"
+                           "fed-decision-in-october-20260617190323537", None),
     "not_found":      ("notfound.html",
                        "https://polymarket.com/event/this-event-does-not-exist-zzz-9999",
                        None),
@@ -374,6 +391,36 @@ def _payload_window(payload, keep_skus, keep_foreign=0):
     totals = re.search(r'"totalCount"\s*:\s*\d+\s*,\s*"hasNextPage"\s*:\s*\w+', payload)
     if totals:
         body = body + "," + totals.group(0)
+
+    # CARRIED OVER ON PURPOSE, and for the same reason `keep_foreign` keeps
+    # the rails: without it the check cannot fail.
+    #
+    # Next.js inlines its 404 TEMPLATE — a `"notfound":[…]` branch the router
+    # would render if the route had 404'd — into the payload of every page
+    # the site serves. It is data about a page that was NOT rendered, and the
+    # classifier used to match its wording and call a healthy event page
+    # `empty`, discarding every row. Trim the template away and no fixture
+    # can ever exercise that again, so the check guarding it would assert
+    # something no fixture contains.
+    #
+    # Kept as the literal branch rather than reconstructed, so the fixture
+    # carries the site's own bytes.
+    # The page ships SEVERAL `"notFound"` keys and only one of them is the
+    # rendered template: the others are a component reference or the literal
+    # "$undefined". Take the first whose own text carries the wording, since
+    # that is the branch the classifier used to match — picking the first key
+    # found gave a browser capture the component reference, and the fixture
+    # then held no template at all.
+    for notfound in re.finditer(r'"notFound"\s*:\s*\[', payload):
+        start = payload.index("[", notfound.end() - 1)
+        try:
+            _branch, end = _DECODER.raw_decode(payload, start)
+        except ValueError:
+            continue
+        if "could not be found" not in payload[start:end]:
+            continue
+        body = body + "," + payload[notfound.start():end]
+        break
     return body
 
 

output_writer.py

@@ -552,12 +552,29 @@ def save(rows: Sequence[Any], out_prefix: str, fmt: str,
 # failure, guarded by the README stating the twenty and the canary asserting
 # it.
 #
-# "no_new_products" stays as the data-side termination condition for
-# `--mode events`, whose page 2..N are event pages named by page 1: a page
-# whose markets are all already in `seen` ends the walk. "pagination_
-# exhausted" is kept for the family's shape.
+# "no_new_products" USED TO BE HERE, as the data-side termination condition
+# for `--mode events`. It is gone, and the removal is the point rather than a
+# tidy-up.
+#
+# The comment above it described a fresh-sku rule — "a page whose markets are
+# all already in `seen` ends the walk" — and the engines had long since
+# stopped doing that; what they actually did was end the walk on a page that
+# produced NO rows. The two are not the same rule and the second one is
+# indefensible here, because `--mode events` walks a FINITE LIST of event
+# URLs that page 1 already named. There is no catalogue to exhaust, so there
+# is no data-side ending to detect: the walk is over when the plan is done.
+#
+# What that cost, reproduced on the code this replaces: one empty event page
+# at position 4 of 20 ended the walk, 17 event pages were never fetched, and
+# the run reported `status: complete` and exit 0 with `pages_completed: 4 of
+# 21` in the same sidecar. Found by a third-party audit; the sidecar had been
+# carrying the evidence against itself the whole time.
+#
+# An event page with no readable markets is now recorded in the sidecar as
+# `pages_without_rows` and the walk carries on. "pagination_exhausted" is
+# kept for the family's shape.
 COMPLETE_STOP_REASONS = ("completed", "pagination_exhausted",
-                         "no_new_products", "listing_has_one_page")
+                         "listing_has_one_page")
 
 
 def finish_run(rows: Sequence[Any], out_prefix: str, fmt: str,

page_flow.py

@@ -118,6 +118,26 @@ def payload_answered(html: Optional[str]) -> bool:
                               or '"markets":[' in payload)
 
 
+def payload_arrived(html: Optional[str]) -> bool:
+    """Whether the page's own payload arrived AT ALL, markets or not.
+
+    The distinction from `payload_answered` is what the readiness wait is
+    actually for. That wait exists for one case: the payload was absent and
+    the DOM is the only thing that can still arrive. Once the payload is
+    here, waiting and scrolling cannot add a market the server did not send —
+    this site inlines its markets in the FIRST response, which is the
+    measurement the fast path is already built on.
+
+    So a payload that arrived carrying no markets is an answer, not a page
+    that is still painting, and it is answered in a second rather than in
+    sixty. Before this existed, a URL whose event does not exist spent the
+    full readiness timeout plus four scroll rounds — measured 2026-10-09 at
+    67 s against 8 s — to reach the same exit 4, and every scroll round
+    logged "added no events (0 -> 0)" while it did so.
+    """
+    return bool(flight_payload(html))
+
+
 def wait_for_count(count: Callable[[str], int], sleep: Callable[[int], None],
                    selector: str, minimum: int, timeout_ms: int,
                    poll_ms: int = 500) -> int:
@@ -297,9 +317,20 @@ def counts_as_blocked(state: str) -> bool:
 
 
 def is_unpainted(state: str, html: Optional[str]) -> bool:
-    """Whether this page is served but has not painted its grid yet."""
+    """Whether this page is served but has not painted its grid yet.
+
+    A page whose payload has ALREADY ARRIVED is not unpainted — it is
+    answered. This site inlines its markets in the first response, so once
+    the payload is in hand, waiting for a grid that is not coming spends the
+    full readiness timeout to reach the same conclusion. Measured 2026-10-09
+    on an event slug that does not exist: 25s of readiness wait and four
+    scroll rounds, every round logging "added no events (0 -> 0)", before the
+    same exit 4 the fast path reaches in eight seconds.
+    """
     if state != "shell":
         return False
+    if payload_arrived(html):
+        return False
     return served_by_polymarket(html or "")