Repository navigation
One paid solve per page, parser failures reported, atomic CSV-safe output - #3
Merged
Merged
Conversation
…tput - A page buys at most one CAPTCHA solve across every attempt, relaunch and proxy rotation (was 2 with no pool, one per exit with a pool). The suite's `solves <= 2` pinned the old behaviour; it now asserts 1. - A served page whose own total says it holds rows, parsed to none, is `parser_found_nothing`: exit 5 on page 1 (was 4), exit 6 later (was 0, complete). The canary names it. - JSON, CSV and sidecar writes are atomic, with the mode open() would give. - CSV cells starting with = + - @ TAB CR LF get a leading apostrophe; `csv_cells_escaped` in the sidecar, merged under the caller's extra. - Counted and timed flags refuse values below their floor in every entry point (exit 2 naming the flag). - Rules are stated in place instead of pointing at private notes; a suite guard keeps it that way. Captcha module notes describe this site. - The expired-endpoint hint no longer claims profiles last about a day. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A row count alone passes when the parser fills rows with nulls. sku, title and author (list, book; plus work_id on books), and sku, reviewer and created_at (reviews), are filled on every row of every capture, so one null fails the canary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jehrr
marked this pull request as ready for review
October 15, 2026 13:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the findings of the 2026-10-14 audit (base
3c6665d).Problems and resulting behaviour
SOLVES_PER_PAGEapplied persettle()call; a re-fetch after a refusal started a new allowance: 2 solves/page with no pool, 4 with a 4-exit poolSolveBudgetper page, threaded through every navigation, landing and the reviews language check; checked before each purchase; not reset by retry or rotation. 1 solve in every scenariocomplete,end_of_listingstop_reason: parser_found_nothing, page saved as debug HTML; exit 5 (page 1) / 6 (later), with no new exit code; canary fails on it by name. An author list past its end (404) and a search with no results are still exit 4; a shelf's constant page count and a reviews filter total of 0 are not treated as stated rowsopen(path,"w"): a failed dump left 10 bytes of a 539-byte file_atomic(temp in target dir, flush+fsync,os.replace, unlink on error,newline=""for CSV) for JSON, CSV, sidecar; new file0o666 & ~umask, existing file keeps its mode=HYPERLINK(...)written verbatim= + - @ \t \r \nget'; numbers untouched; JSON untouched;csv_cells_escapedin the sidecar, merged UNDER the caller's extra--concurrencyerrorshop_rating, "announcement codes", unmeasured "451"--delay,--retry-delay,--proxy-block-retries;--pages/--concurrency/--max-books0; negative--retriesinscraper_api_client.pysilently clampedpage_flow.boundedtype in all three engines and the API client → exit 2 naming the flag.env.exampleCHANGELOG
[Unreleased]leads with a blockquote listing the behaviour changes. Released sections untouched (they held no citations). Release notes v0.1.0/v0.1.1: 0 citations, not edited.Evidence
--helpworks and--delay -1/--proxy-block-retries -2giveerror: argument --X: --X must be at least 0.ci_checks.py --sample-check --secret-checkOK (47 files).parser_found_nothing: it fails naming it.__pycache__cleared): budget passed as None per call (2/2/4 solves) · budget reset on relaunch · parse_failed not set (4 / 0-complete) · truncating JSON writer · no chmod · flat 0644 under umask 002 · no CSV escape · escape before list join · stats merged over extra · plain float--delayin selenium · plain int--retriesin scraper_api_client · a planted citation.Canary (second commit)
The canary already ran ungated every day and failed on any non-zero exit; the last 10 scheduled runs were green with real rows. The Assert step now also fails when any row has no value in a core column (list: sku/title/author; book: also work_id; reviews: sku/reviewer/created_at), and it names
parser_found_nothing. No secret is required. Dispatched on this branch: https://github.com/2scraper/goodreads-scraper/actions/runs/38694402188, success (300 list rows, 5 books, 90 reviews).Not changed
--proxy/--fingerprint/--concurrencywith--cdp-endpoint(behaviour change; owner decision).--categoryas intentionally absent (owner decision).🤖 Generated with Claude Code