One paid solve per page, parser failures reported, atomic CSV-safe output

.env.example

@@ -60,8 +60,8 @@ TWOCAPTCHA_KEY=your_2captcha_api_key_here
 # Selenium cannot use this: chromedriver's `debuggerAddress` takes a bare
 # host:port with nowhere to put a password. Playwright and pyppeteer can.
 #
-# The credentials in a profile expire in about a day, so never paste a
-# working one anywhere that is committed.
+# The credentials in a profile expire (an HTTP 401 says so), so never paste
+# a working one anywhere that is committed.
 GOODREADS_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-de-pid-{profileId}:{password}@cb.2captcha.com:9222
 
 # --- Proxy ----------------------------------------------------------------

.github/ci_checks.py

@@ -92,8 +92,8 @@
 # 32-hex shape in its markup is an asset's build fingerprint
 # (`loading-trans-<md5>.gif`), which no row carries and `make_fixtures.py`
 # takes out of every fixture. So a bare 32-hex anywhere in the tree fails,
-# with no exception to reason about (§24: take a value out rather than
-# exempt it).
+# with no exception to reason about: a value nothing reads is taken out
+# rather than exempted.
 SITE_PUBLIC_IDS = ()
 
 def _without_site_ids(line):
@@ -280,7 +280,7 @@ def tracked_files():
 # into pip's vendored code and flags a 32-hex string in `_elffile.py` as
 # key-shaped. Correct about the string, wrong about the file — and it is the
 # FIRST thing a new user sees from `python3 smoke_test.py`. A guard people
-# have to argue with is one they learn to suppress (CLAUDE.md §22).
+# have to argue with is one they learn to suppress.
 #
 # Deliberately NOT a narrowing of what gets scanned: an untracked file is
 # still read, because a key pasted into a scratch file beside the scripts is

.github/workflows/canary.yml

@@ -3,7 +3,7 @@
 #
 # No secrets are needed, and that is the point of running it: the README's
 # central claim is "no key, no proxy, no account", and an ungated scheduled
-# canary is that sentence under test every morning (CLAUDE.md §21, §24).
+# canary is that sentence under test every morning.
 # Every mode ran from a datacentre VPS when this repo was written
 # (2026-09-27), AWS WAF's challenge included, which the browser passes by
 # itself; a GitHub runner is a datacentre address too. Its first dispatch
@@ -73,7 +73,7 @@ jobs:
           pip install -r requirements.txt -r requirements-playwright.txt
           playwright install --with-deps chromium
 
-      # >= 3 pages, because with one page pagination is never exercised (§11).
+      # >= 3 pages, because with one page pagination is never exercised.
       - name: Run
         id: run
         run: |
@@ -87,11 +87,16 @@ jobs:
         run: |
           code="${{ steps.run.outputs.exit_code }}"
           echo "exit code: $code"
+          # A served page that stated rows and parsed to none is named on its
+          # own: it is this repo's parser that broke, not the site's listing.
+          if [ -f canary.meta.json ] && grep -q '"stop_reason": "parser_found_nothing"' canary.meta.json; then
+            echo "::error::parser_found_nothing: a served page stated rows and the parser read none. The page shape has moved; see canary_page*_debug.html."; exit 1
+          fi
           case "$code" in
             0) ;;
             3) echo "::error::Blocked. If the log names aws-waf, the WAF's challenge did not clear by itself: it showed its CAPTCHA, or refused the runner. If it names http-403, the runner's address was refused. Either way, set the GOODREADS_PROXY secret (see the header)."; exit 1 ;;
             4) echo "::error::Zero rows from a listing that is never empty. The page shape has moved; see canary_dump.html."; exit 1 ;;
-            5) echo "::error::The data never arrived, or the API REFUSED the parameters (the log names its own complaint)."; exit 1 ;;
+            5) echo "::error::The data never arrived, the API REFUSED the parameters, or page 1 stated rows and parsed to none (parser_found_nothing). The log names which."; exit 1 ;;
             6) echo "::error::Partial: a page or a book failed after others succeeded. See the sidecar's pages_failed."; exit 1 ;;
             *) echo "::error::Unexpected exit $code."; exit 1 ;;
           esac
@@ -104,6 +109,8 @@ jobs:
           rows = json.load(open("canary.json"))
           mode, floor, pages = "${{ matrix.mode }}", ${{ matrix.floor }}, ${{ matrix.pages }}
           problems = []
+          if meta["stop_reason"] == "parser_found_nothing":
+              problems.append("parser_found_nothing: a served page stated rows and parsed to none")
           if meta["status"] != "complete":
               problems.append("status %r (%s)" % (meta["status"], meta["stop_reason"]))
           if meta["pages_completed"] != pages:
@@ -114,6 +121,15 @@ jobs:
               problems.append("no total_results in the sidecar")
           if {r["mode"] for r in rows} != {mode}:
               problems.append("rows name modes %r" % {r["mode"] for r in rows})
+          # Core columns the parser fills on every row of every capture: a
+          # null here means the parser broke, whatever the row count says.
+          core = {"list": ("sku", "title", "author"),
+                  "book": ("sku", "title", "author", "work_id"),
+                  "reviews": ("sku", "reviewer", "created_at")}[mode]
+          for col in core:
+              missing = sum(1 for r in rows if r.get(col) in (None, "", []))
+              if missing:
+                  problems.append("%d of %d rows have no %s" % (missing, len(rows), col))
           pairs = [(r["page"], r["position"]) for r in rows]
           if len(set(pairs)) != len(pairs):
               problems.append("page+position is not unique")

.gitignore

@@ -23,11 +23,11 @@ goodreads_rows_*.*
 live_results/
 live_results.zip
 # ANY working directory a live run writes into, by KIND rather than by the one
-# name an older repo used (CLAUDE.md §24). Measured here: `live/` was not
-# listed, and a `--dump-html` of a 3-page run wrote `canary_dump.json.page3`,
-# which `*.json` does not cover. It sat untracked and NOT ignored, one
+# name an older repo used. In a sibling repo `live/` was not listed, and a
+# `--dump-html` of a 3-page run wrote `canary_dump.json.page3`, which
+# `*.json` does not cover. It sat untracked and NOT ignored, one
 # `git add -A` from a commit, until the credential scan flagged the 32-hex
-# announcement codes inside it.
+# ids inside it.
 live/
 out/
 output/
@@ -47,7 +47,7 @@ dumps/
 *_debug.png
 *.html.page*
 # A single-page `--dump-html page.html` lands at the repo root with no
-# suffix any rule above knows (CLAUDE.md §24). Every HTML file this repo
+# suffix any rule above knows. Every HTML file this repo
 # needs lives in a subdirectory or inside fixtures_generated.json.
 /*.html
 *.json.page*
@@ -101,7 +101,7 @@ secrets*
 # recoverable only because that repo was private at the time. On 2026-09-17
 # a sibling had an untracked `.env.bak` holding a 32-hex key in a PUBLIC
 # repository, one `git add -A` from publication. Craigslist was fixed then
-# and the fix never reached anyone else, which is §16's rule unapplied: when
-# you fix one, check the siblings.
+# and the fix never reached anyone else. When you fix one, check the
+# siblings.
 .env*
 !.env.example

CHANGELOG.md

@@ -9,22 +9,64 @@ said so at the top of its release notes.
 
 ## [Unreleased]
 
-> **Behaviour change:** `--retries 0` is now a usage error (exit 2). It used
-> to be accepted and made the page-load loop unreachable, so nothing was
-> fetched and the engine reported the browser's own blank page as a result:
-> exit 3 or 4 with a 39-byte "capture", in tens of milliseconds. If a
-> script of yours passes 0, pass **1**: the flag counts ATTEMPTS, so 1 means
-> "try once and do not retry".
+> **Behaviour changes:**
+>
+> * **A page buys at most one paid CAPTCHA solve, however many times it is
+>   fetched.** A page re-fetched after a refusal used to start a new
+>   allowance: 2 solves per page with no proxy pool, and one per exit with a
+>   pool (4 on a pool of four). With `TWOCAPTCHA_KEY` set, that was money.
+> * **A served page that states rows and parses to none is now a parser
+>   failure** (`stop_reason: parser_found_nothing`): exit 5 on page 1
+>   (was 4, "the listing is empty") and exit 6 on a later page (was 0 and
+>   `status: complete`).
+> * **CSV cells that start with `=`, `+`, `-`, `@`, a tab or a line break
+>   get a leading `'`**, so a spreadsheet does not run them. JSON is
+>   unchanged; the sidecar's `csv_cells_escaped` counts the difference.
+> * **New usage errors (exit 2):** `--retries 0`; `--pages`,
+>   `--concurrency` and `--max-books` below 1; a negative `--delay`,
+>   `--retry-delay` or `--proxy-block-retries`; a negative `--retries`,
+>   `--pages` below 1 or a negative delay in `scraper_api_client.py`. If a
+>   script of yours passes `--retries 0`, pass **1**: the flag counts
+>   ATTEMPTS, so 1 means "try once and do not retry".
 
 ### Fixed
 
 - **`--retries 0` fetched nothing and called it a result.** The load loop is
   `range(1, retries + 1)`, so 0 never runs it and the engine judged
   `about:blank`. Three third-party audits of this family ran exactly that and
   reported a working scraper as broken. `--retries` now goes through
-  `page_flow.attempts` in every entry point; a check reads the source of each
+  `page_flow.attempts` in every engine; a check reads the source of each
   one and calls the validator, so a new engine declaring the flag with a
   plain int fails the suite.
+- **The paid-solve limit was per call, not per page.** `SOLVES_PER_PAGE = 1`
+  was applied inside each navigation, and every re-fetch after a refusal (a
+  fresh browser, or the next exit in a pool) navigated again. One
+  allowance now lives for the whole page, is checked before each purchase
+  and is not reset by a retry or a rotation; the language check that lands
+  on the book before page 1 of a reviews run spends from page 1's. The
+  suite's old assertion `solves <= 2` pinned the bug; it now asserts 1, with
+  no pool, with a pool of four, and on a reviews landing.
+- **A page whose own count says it holds rows, parsed to none, was reported
+  as an empty listing or as the end of one.** It is now
+  `parser_found_nothing`, the page is saved for an issue, and the canary
+  names it.
+- **Output writes are atomic.** JSON, CSV and the sidecar are written to a
+  temporary file in the same directory and renamed over the target, so a
+  crash or a full disk mid-write leaves the previous file whole (before: a
+  539-byte `out.json` came back as 10 bytes). A new file gets the mode
+  `open()` would give it under the current umask; an existing file keeps its
+  own.
+- **`--delay`, `--retry-delay`, `--pages`, `--concurrency`,
+  `--proxy-block-retries` and `--max-books` are validated** in every engine,
+  and `--pages`, `--delay`, `--retries` and `--retry-delay` in
+  `scraper_api_client.py` (where `--retries` counts EXTRA attempts, so 0
+  stays legal there).
+- Comments, docstrings and one `--concurrency` error message pointed at
+  notes that are not public; each rule is now stated where it is used. The
+  captcha module's notes described another site; they describe this one.
+- The expired-endpoint hint no longer says a Scraping Browser profile lasts
+  "about a day": one was measured working after ten days, so the HTTP 401 is
+  the signal, not an age.
 
 ## [0.1.1] — 2026-09-28
 

CONTRIBUTING.md

@@ -86,8 +86,8 @@ Then the rest of the presentation, in the order that matters:
    the claim the README makes. If a run is refused for the runner's
    address, the canary's `GOODREADS_PROXY` secret (an exit elsewhere) is
    the fix, with no workflow edit.
-2. The repo description, homepage and topics set (see the family notes on
-   what those should say).
+2. The repo description, homepage and topics set: the description names
+   the site and what it really publishes, the topics are what people search for.
 3. Only then the row in the org profile README — and check it with an
    ANONYMOUS request rather than your own logged-in browser. A row pointing
    at a private repo is a 404 for every visitor, which costs more trust than

README.md

@@ -90,6 +90,12 @@ each mode is committed: [list](sample_output.json), [book](sample_output_book.js
 [reviews](sample_output_reviews.json) (the reviews sample's reviewer names,
 ids and words are placeholders; everything else is as the run wrote it).
 
+In the CSV, a text cell that starts with `=`, `+`, `-`, `@`, a tab or a
+line break gets a leading `'`, so a spreadsheet shows it as text instead of
+running it as a formula (review text and shelf names are written by anyone).
+The JSON keeps the site's text unchanged, and the sidecar's
+`csv_cells_escaped` says how many cells differ.
+
 ---
 
 ## Six things about Goodreads that will look like bugs
@@ -221,8 +227,8 @@ Surveyed 2026-09-27:
 | 2 | bad usage, including a `--review-language` the book has no reviews in |
 | 3 | blocked: AWS WAF's challenge did not clear, or a 403; distinct from an empty listing |
 | 4 | zero rows: the listing has nothing in it, or the book id names no book |
-| 5 | the data never arrived: a timeout, a dead proxy, a refused API parameter, a remote API error |
-| 6 | partial: some pages (or, from a listing, some books) came back and some did not |
+| 5 | the data never arrived: a timeout, a dead proxy, a refused API parameter, a remote API error, or page 1 stated rows and the parser read none (`stop_reason: parser_found_nothing`) |
+| 6 | partial: some pages (or, from a listing, some books) came back and some did not, including a later page that stated rows and parsed to none |
 
 **A run that finds nothing writes nothing**, so a failure never replaces last
 night's good output with `[]`. `--allow-empty` is the opt-out.

TROUBLESHOOTING.md

@@ -50,6 +50,11 @@ writes the empty file.
 * **"The book page carries no API key or work id"** — the page shape moved;
   same issue template.
 * **"Gave up on page N"** — a timeout or a dead proxy. The log names which.
+* **`stop_reason: parser_found_nothing`** ("Page N was served and states ...
+  match(es) ... but the parser read NO rows from it") — the page said it
+  holds rows and none were read, so the page shape has moved. The page is
+  saved as `<out>_pageN_debug.html`; open a "Site changed" issue with it.
+  On page 2 or later this is exit 6 and the earlier pages are written.
 
 ## Exit 6 — partial