Walk every event page, and stop reading every page as its own 404

[jehrr]

Works the third-party audit of 8 October against ed65d78, which is current main. Its P1 is real, and reproducing it turned up a second defect underneath that the audit did not see — the two compound, and the second is what makes the first common rather than theoretical.

P1 — the --mode events walk stopped on the first empty page and called it complete

Confirmed, and it is this repo's own code rather than anything inherited. Reproduced end to end against main, driving the real dispatcher into the real finish_run, with page 4 of 20 holding no markets and pages 5..21 holding three each:

event pages planned     : 20
event pages fetched     : 3  [2, 3, 4]
event pages NEVER tried : 17 [5 … 21]
stop_reason             : no_new_products
sidecar status          : complete
pages_completed         : 4 of 21
pages_failed            : []
exit code               : 0

The sidecar carries pages_completed: 4 of 21 and calls itself complete in the same file. That is the self-contradiction commit 60e65ba fixed for pages that FAILED; it never looked at pages that were never attempted.

Why the rule was wrong here, rather than just unlucky. It came from a sibling whose pages are consecutive days, where "asking for 40 days of a tag that published on three would fetch 37 empty ones" is a real saving. In this mode the specs are the event URLs page 1 named, capped at --pages - 1 — the engine already logs how long the walk really is before starting it. So the queue cannot contain a page worth skipping: the saving did not exist and the cost was the rest of the listing.

Fixed in all three engines, sequential and concurrent paths. An event page with no readable markets is recorded in the sidecar as pages_without_rows and the walk carries on. no_new_products is set nowhere now, so it is out of COMPLETE_STOP_REASONS rather than left as policy nothing reads.

Same scenario after the fix: 20 of 20 fetched, pages_completed: 21 of 21, and the file holds 77 markets instead of 26.

The defect underneath it: every served page classified as its own 404

Found while measuring whether an empty event page is an edge case. It is not, because the classifier was manufacturing them.

Next.js inlines its 404 template — a "notFound":[…] branch the router would render if the route had 404'd — into the payload of every page the site serves. looks_not_found scanned the first 200,000 bytes for the wording in that template. Counted on 2026-10-09 across the twenty event pages /predictions named, as the site serves them:

carry the wording	20 of 20, always 8 times
…inside the scanned window	17 of 20, always at offset ~134,000
…rendered into the body markup	0 of 5 checked, bogus pages included
misclassified empty while holding markets	12 of 20, holding 5 to 53 each

empty carries parse: False, so those rows were discarded and the page produced zero rows — which is exactly what tripped P1 above. The two defects compose into a silently truncated run reported as complete.

The browser engines escaped it by byte position, not by design: a browser serialises rendered markup first and pushes the template past the window (511,547 against 134,033 on the same URL). The margin on the smallest page measured was 67 KB. scraper_api_client.py reads the served bytes and had no such margin.

Three further measurements, each of which changed what the fix had to be:

• a slug that does not exist answers HTTP 200, 617 KB, 4 of 4 — so detect_page_state's docstring promising "a real HTTP 404" was wrong, and the status could not be the backstop the text markers were assumed to have;
• the wording is never rendered into body markup, not even on a genuinely bogus page, so no text marker could have distinguished anything;
• "digest" looked like a clean replacement — 4 of 4 bogus pages, 0 of the first two good ones — and is not: counted properly it is on 2 of 10 good pages. Not adopted.

So the markers are gone rather than narrowed, and the content check moved to the top: a page whose payload names markets is content, whatever templates it also ships. A nonexistent event now falls through to shell, parses to no rows and exits 4, which is what a bogus listing tag already did.

Cost of that reordering, and the second fix it needed. shell spends the readiness wait, so a bogus slug went from 8 s to 67 s. The wait exists for one case — the payload never arrived — so it is now gated on the payload ARRIVING rather than on it naming markets (page_flow.payload_arrived, used by is_unpainted and by all three engines' fast path). Back to 7.8 s, same exit 4.

Why no fixture could show it

All eleven existing captures are a browser's page.content(). The repo had no capture of what the HTTP client path sees, which is a different document shape from the same URL — the same gap as the CDP fixture added in #7, and the same lesson: a guard is only as good as the fixture it runs against.

captures/event_served_bytes.html is that capture, and make_fixtures.py now carries the "notFound" branch through the trim on purpose. Without that the new checks could not fail — the first version of the fixture had the template trimmed out and passed against the unfixed parser.

Verification

Eight faults planted, suite required to go RED on a named check:

planted fault	caught by
the not-found text markers come back	no text marker decides not-found — only an explicit 404
not-found consulted before the content check	a page naming markets is content even under a 404 status
an empty event page ends the walk again	an empty event page does not stop dispatch (+2)
no_new_products complete again	`no_new_products` is NOT a complete stop reason (+1)
the scanners stop skipping virtualenvs	describes this site on pip's vendored code (+3 wiring)
…the widened wording scanner specifically	test_wording() walks the tree and skips virtualenvs
the ignore patterns go back to three names	the virtualenv .venv-pw/ is gitignored
the generator stops carrying the template	the served-bytes fixture really does carry the 404 template

Three of those controls were wrong on the first attempt, and each is the reason the pass was worth running:

• a crash detector keyed on Traceback called every control a crash, because a GREEN run of this suite prints one deliberately;
• the virtualenv control replaced the function's own def line, producing a SyntaxError — a non-zero exit that proves the edit was reachable and nothing about the guard;
• disabling the guard in the wording scanner alone left the suite green, because nothing in a virtualenv trips a banned phrase. That is a hole in the suite rather than a pass, and it is why the wiring is now pinned structurally instead of relying on a venv happening to contain the right text.

Live, after the fix: --mode events --pages 8 complete with 192 markets over 8 pages; the event that curl classified empty parses 5 rows; a bogus slug exits 4 in 7.8 s; 0 of 20 event pages misclassified, against 12 before.

Also fixed: virtualenvs, by name instead of by kind — twice

Neither is in the audit; both were found by the suite failing on my own working tree.

The tree scanners walked in. The scanner widened to the whole tree in #8 reported that pip's vendored charset_normalizer "describes this site", because it skipped virtualenvs by NAME (.venv, venv) and mine is .venv-pw. .github/ci_checks.py had already been fixed to recognise a venv by pyvenv.cfg; the four scanners in smoke_test.py had not. It is the first thing a new user sees from python3 smoke_test.py, since the README tells them to make a virtualenv per engine and does not say where.

The four fail in three different ways, which is why the wiring is pinned rather than the content: the describes-this-site scan gives a false failure, the dead-name corpus gives a false pass (vendored code keeps a dead name alive by coincidence), and the banned-wording scan is unaffected today.

And .gitignore had the same shape of hole. It listed .venv, venv and env, so .venv-pw was untracked and not ignored — one add -A from being committed. The patterns are broader now, and since a glob can always be out-named, the suite asserts that every directory holding a pyvenv.cfg really is ignored. On a clean checkout that check reports it had nothing to find, rather than passing silently.

What is NOT changed here, and why

• The audit's other three findings are real and are not in this PR — non-atomic writes, a canary on one engine, and unpinned dependencies. They are reliability and reproducibility work, not correctness, and mixing them with a classifier change would make both harder to review. The first is next.
• No run identifier for the JSON/CSV/sidecar set. The atomic rename is right and I will take it; the transaction id is solving a problem this contract already solves by construction — a failed run writes no sidecar at all, and save refuses to overwrite good output with an empty result. An id would add a mechanism to enforce something enforced by not writing.
• Not declaring Playwright the only supported engine. The audit offers that as one of two options and it is the wrong one: the three engines were run live and returned identical rows, and parity is enforced offline by flag-set comparison in both directions and a shared finish_run. What is genuinely missing is a LIVE cross-engine check, which is the audit's other option and is cheap.
• The Bright Data / Oxylabs comparison is accurate and is not a finding. It answers a question this repo does not ask; the README already states the paid path is unnecessary for this site.
• The audit's own live run — four loads of a 39-byte empty DOM, exit 3 — is not evidence of a parser defect, and the audit says so itself. That is Chromium's empty document after a failed navigation. The site served 897 KB to this machine the same day, the canary on that commit was green, and the run above fetched 192 markets.

🤖 Generated with Claude Code