Skip to content

Reject --retries 0 instead of fetching nothing - #21

Merged
jehrr merged 1 commit into
mainfrom
fix/retries-zero
Oct 13, 2026
Merged

jehrr merged 1 commit into
mainfrom
fix/retries-zero

Conversation

@jehrr

@jehrr jehrr commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

Draft — not merged, no version bump, no tag, no release.

Behaviour change: --retries 0 is now a usage error (exit 2). It used to be accepted and made the page-load loop unreachable, so nothing was fetched and the engine reported the browser's own blank page as a result. A script that passes 0 should pass 1: the flag counts attempts.

The defect

The page-load loop is for attempt in range(1, args.retries + 1). With --retries 0 that range is empty, so goto is never called and the engine then judges whatever the browser holds: about:blank, 39 bytes. The result is a confident answer in a few dozen milliseconds to a question nobody asked.

Measured on the sibling sleepnumber-scraper, 2026-10-13, from an address the site really does refuse (so the conclusion even looked right):

exit bytes time
--retries 0 3 39 36 ms — never left the process
--retries 1 3 923 116 ms — the real CloudFront 403

Three third-party audits of this family have now run --retries 0, read that output and reported a working scraper as broken or a site as unreachable.

What changed

  • page_flow.attempts is the argparse type= for --retries in every entry point that declares it (and rejects 0, negatives and non-integers with the reason).
  • One check in smoke_test.py reads the source of every shipped entry point, requires the validator on each --retries flag, and calls the validator. It does not use a hand-kept file list, so a new engine declaring the flag with a plain int fails the suite.
  • CHANGELOG.md: the behaviour-change note, under Unreleased.

Verified by running it, not by reading

  • Suite before: green (None checks where the suite reports a count) → after: green (None).
  • Live path, with Playwright replaced by a recorder and the repo's own engine run end to end: --retries 0 → REJECTED (playwright_scraper.py: error: argument --retries: --retries must be at least 1, not 0. It counts ATT), --retries 1 → GOTO_CALLED (the flag is still honoured).
  • Two planted faults, each required to turn the suite red on the named check, not merely to exit non-zero: the engine stops validating the flag; attempts() accepts 0 again. Both did.

Not changed, and why

  • scraper_api_client.py is untouched where present: it counts extra attempts (retries + 1), so 0 is legal there and means one try. Validating it would break users.
  • No release. If you want this shipped, version it separately.

🤖 Generated with Claude Code

The page-load loop is `range(1, retries + 1)`, so 0 never runs it: nothing is
fetched and the engine judges the browser's own about:blank (39 bytes) in a
few dozen milliseconds. Three third-party audits of this family ran exactly
that and reported a working scraper as broken.

--retries now goes through page_flow.attempts in every entry point that
declares it, and a check reads the source of each one so a new engine cannot
reintroduce the plain int.

scraper_api_client.py is left alone where present: it counts extra attempts,
so 0 is legal there.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@jehrr
jehrr marked this pull request as ready for review October 13, 2026 21:40
@jehrr
jehrr merged commit 7de7da0 into main Oct 13, 2026
7 checks passed
@jehrr
jehrr deleted the fix/retries-zero branch October 13, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant