Repository navigation
Reject --retries 0 instead of fetching nothing - #21
Merged
Merged
Conversation
The page-load loop is `range(1, retries + 1)`, so 0 never runs it: nothing is fetched and the engine judges the browser's own about:blank (39 bytes) in a few dozen milliseconds. Three third-party audits of this family ran exactly that and reported a working scraper as broken. --retries now goes through page_flow.attempts in every entry point that declares it, and a check reads the source of each one so a new engine cannot reintroduce the plain int. scraper_api_client.py is left alone where present: it counts extra attempts, so 0 is legal there. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — not merged, no version bump, no tag, no release.
The defect
The page-load loop is
for attempt in range(1, args.retries + 1). With--retries 0that range is empty, sogotois never called and the engine then judges whatever the browser holds:about:blank, 39 bytes. The result is a confident answer in a few dozen milliseconds to a question nobody asked.Measured on the sibling sleepnumber-scraper, 2026-10-13, from an address the site really does refuse (so the conclusion even looked right):
--retries 0--retries 1Three third-party audits of this family have now run
--retries 0, read that output and reported a working scraper as broken or a site as unreachable.What changed
page_flow.attemptsis the argparsetype=for--retriesin every entry point that declares it (and rejects 0, negatives and non-integers with the reason).smoke_test.pyreads the source of every shipped entry point, requires the validator on each--retriesflag, and calls the validator. It does not use a hand-kept file list, so a new engine declaring the flag with a plain int fails the suite.CHANGELOG.md: the behaviour-change note, under Unreleased.Verified by running it, not by reading
--retries 0→ REJECTED (playwright_scraper.py: error: argument --retries: --retries must be at least 1, not 0. It counts ATT),--retries 1→ GOTO_CALLED (the flag is still honoured).attempts()accepts 0 again. Both did.Not changed, and why
scraper_api_client.pyis untouched where present: it counts extra attempts (retries + 1), so 0 is legal there and means one try. Validating it would break users.🤖 Generated with Claude Code