From 75f1d8cdaa114445a27ed7d1fcb3fb4be006a750 Mon Sep 17 00:00:00 2001 From: jehr Date: Wed, 9 Sep 2026 00:03:13 +0200 Subject: [PATCH 1/3] Scrub session material and one real person's data out of the fixtures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found while checking what making this repo public would actually expose, rather than by reading the code. The fixtures are real page captures, and a real page capture carries more than markup: * THE SESSION THAT FETCHED IT. Three `sessionId` values, their `anti-csrftoken-a2z` tokens, `csrfToken` and `offerListingId` values were baked into the committed HTML. All from anonymous, never-logged-in browsing and long expired, so nothing here granted access to anything — but session material does not belong in a repo about to go public, and the repo's own "no credentials committed" CI check has exactly this spirit while matching none of these shapes. * ON A REVIEW, A REAL CUSTOMER. Their display name, profile permalink (amzn1.account...), review id, the ids of photos they uploaded, their review title and the words they wrote. All of it is public on amazon.com, but republishing someone's name and writing inside a public repo is a different act from Amazon showing them on its own page — and the checks here read the STRUCTURE of a review, not the person. Both are now obvious placeholders. Everything Amazon generates around them — the accessibility filler that is part of the body node's text, the date wording, the badges, the markup — is untouched, because that is what the checks actually exercise. The review fixture is now the second of two fixtures that are not verbatim captures, and it says so where a reader of the fixtures will look, next to the image-captcha one. A one-off cleanup would rot on the next capture, so the suite grew a guard: patterns (not literals) that fail if a real session id, CSRF token or account id ever arrives with a future fixture, plus an assertion that the review's author stays a placeholder. Confirmed it fires — a copy of the tree with one real sessionId put back exits 1 and names it. 244 offline checks. README and CHANGELOG updated to describe both non-verbatim fixtures rather than claiming everything is a straight capture. NOTE ON WHAT THIS DOES NOT REACH: the pre-scrub values remain in this repository's history — in the v0.1.0 tag and in PR #1's own refs, which stay attached to the PR and cannot be deleted from it. This commit keeps HEAD, all future clones and every future release artifact clean; removing the history entirely would need a fresh repository. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Cm7Ebf8wfGq2BvsxFa7rxY --- CHANGELOG.md | 10 +++++++- README.md | 23 ++++++++++++------ smoke_test.py | 67 ++++++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 80 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4a59d06..8a823ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -51,7 +51,7 @@ scrapers. **Nothing from the previous code is source- or output-compatible.** - **A `page` column** beside `position`, because Amazon's `data-index` restarts on every page and position alone is ambiguous across a paginated run. -- 239 offline checks in `smoke_test.py`, with fixtures cut from real captures; +- 244 offline checks in `smoke_test.py`, with fixtures cut from real captures; a daily canary against a real 3-page listing; `tests.yml` on Python 3.9 and 3.12 with an `engine-smoke` job that fails on any unexpected skip. @@ -64,6 +64,14 @@ scrapers. **Nothing from the previous code is source- or output-compatible.** own requirements file into its own venv, which is both what the README tells users to do and the only way to get the versions they get; `pip check` runs per venv as well. +- **The fixtures carried session material and one real person's data.** Real + page captures bring the session that fetched them (three anonymous, expired + `sessionId` values and their CSRF tokens) and, on a review, a real + customer's display name, profile permalink, review id, photo ids and words. + All of it is now replaced with obvious placeholders — the checks read the + structure of a review, not the person — and a new guard fails the suite if + any of it arrives with a future capture. Found while reviewing what going + public would actually expose. - **The pyppeteer import was inside the launch path**, so `puppeteer_scraper` imported cleanly with no pyppeteer installed at all. That made the offline suite's skip reporting — and therefore CI's "fail on any skip" guard — diff --git a/README.md b/README.md index 97b4ae5..c2542f4 100644 --- a/README.md +++ b/README.md @@ -430,7 +430,7 @@ says something about our own two snapshots, not about Amazon. Same for ## Testing ```bash -python3 smoke_test.py # 239 offline checks, no network, no browser +python3 smoke_test.py # 244 offline checks, no network, no browser pytest # the same suite, as one test python3 env_config.py # what configuration was picked up (prints no secrets) ``` @@ -440,13 +440,20 @@ The offline suite passes with no engine library installed at all, and and fails if anything reports skipped, because "skipped, engine absent" reads identically to a passing run. -Its fixtures are real captures with `