Treat a 403 refusal as a block, keep output file modes, validate numeric flags

[jehrr]

Fixes from the 2026-10-14 audit of 61212f3 (no open PRs at the start, main unchanged since the audit).

What changes for a user

• A 403 refusal is now a block, not a throttle. No engine passed the HTTP status to the classifier, and Rakuten renders its 403 refusal and its 503 rate-limit page with the same body. Every 403 was therefore read as a throttle: waited out three times (about 51 s) at the same exit, then reported as a failed page. Playwright and pyppeteer now carry goto()'s status into every classify call, and reset it to None after a solve reloads the page. Selenium passes None explicitly because WebDriver exposes no status, so it keeps the documented recoverable reading.
• pyppeteer no longer crashes with no proxy. puppeteer_scraper.py called pool.advance() on a repeated load failure with pool=None and raised AttributeError. It now rotates only for an unusable exit and only when there is a pool, matching Playwright and Selenium.
• Output modes. .json, .csv and .meta.json were always 0600. A new file is now 0666 & ~umask, and a rewrite keeps the existing file's mode.
• Captcha budget. The pre-classification call site counted only successful solves, so a solver that raised was asked twice for one page. Both call sites now count each attempted purchase. The handler appends to a purchases list right before calling the solver, so a page with no captcha spends nothing.
• Static detector. A capture with only class="g-recaptcha" + data-sitekey, or only the api2/anchor frame, was reported as having no captcha. It now yields a v2 (or v2-invisible) challenge. It fires on none of the 14 committed captures.
• CLI validation. --pages/--concurrency must be ≥ 1, and --delay/--retry-delay/--proxy-block-retries must be ≥ 0. Anything else is an argparse error naming the flag, in all three engines. scraper_api_client.py validates --retries ≥ 0, --retry-delay ≥ 0 and --timeout ≥ 1.
• Help and comments described another site. The --solve-captcha help in all three engines (an "HTTP/2 stream reset"), the EXIT_BLOCKED comment (a different site's hubs and no-results text), and the "394-byte Access Denied / headless" comments in the engines and captcha_solver.py are rewritten for Rakuten's measured refusals: a 43-byte deny under 200, a branded page under 403, and the same page under 503. I also dropped the keyword=kopi docstring example and reattached the EXIT_PARTIAL comment to its constant.
• Private citations removed. Every §N reference (139 lines) and both CLAUDE.md mentions in tracked files were removed or restated in the repo's own words. CHANGELOG and release notes v0.1.0/v0.2.0/v0.2.1 had none, so no release was edited. A new suite check scans every tracked text file (site-data fixtures excepted) for section-sign citations and for private-note names. The tokens are assembled from pieces, so the suite file is scanned too.
• Canary (section 12). Before this change, exit 3 and exit 6 ended the job green with a warning and tested=false. Now every non-zero exit is red. The exit-5 message now says "content never obtained, diagnose the transport" instead of "impossible". The job stays scheduled, uncredentialled and ungated. The last 10 scheduled runs really parsed 3×45 rows (2026-10-15 log: Parsed 45 row(s) per page, then canary OK). Thresholds are unchanged and test the parser: row floor, price coverage, JPY, points, shop_code, page+position uniqueness, rating sentinel.
• Small fixes. .gitignore used *.page[0-9] / *.page[0-9][0-9], which stopped at .page99; it now uses *.page[0-9]*. requirements-selenium.txt pointed to a nonexistent --chromedriver flag. .env.example no longer says that Scraping Browser credentials "live about a day". tests.yml dropped its inline duplicate of ci_checks.py --sample-check. BANNED_PHRASES is assembled from pieces, so smoke_test.py is no longer exempt from the wording scan.

Evidence

• Offline suite: core-only venv 673 PASS / 0 FAIL (it was 640 before; the engine groups skip there as expected). Per-engine venvs (no browser downloads): playwright 702/0, pyppeteer 697/0, selenium 690/0.
• The new browser engines' fetch loop, driven offline with a fake page group loads each browser engine, using a stub driver module when the library is absent, and drives _fetch_one_page. It checks four things:
  • pool=None with failing navigations ends as load_failed instead of raising;
  • the real block_403 fixture under status 403 ends blocked;
  • the same fixture under 503 ends throttled after 1 + THROTTLE_RETRIES fetches;
  • a solver stub that raises is called at most SOLVES_PER_PAGE times.
• An AST check confirms that every classify call in every engine passes a status.
• Mode checks under umask 022/002/077 and preservation of an existing 0664 file.
• ci_checks.py --secret-check --help-check --sample-check: ok, 43 files scanned.

Controls

All controls ran with PYTHONDONTWRITEBYTECODE=1 and __pycache__ removed. In each one I planted the fault, saw the suite go red on the named check, restored it, and saw the suite go green.

Planted fault	Named check that went red
chmod removed	a new output file is 0666 minus the umask
pyppeteer guard reverted	puppeteer_scraper: repeated load failures with no proxy pool … (AttributeError)
Playwright status dropped	playwright_scraper: the branded page under HTTP 403 …
pyppeteer status dropped	puppeteer_scraper: the branded page under HTTP 403 …
success-only budget count	a failing solver is asked at most … (2 calls)
detector bail restored	the static detector finds a v2 widget …
§17 planted in page_flow.py	no tracked file cites private notes …
old .gitignore page pattern	git also ignores canary.json.page100
selenium --pages type=int	selenium_scraper validates its numeric flags …
selenium classify without status	every classify call passes a status
canary exit 3 made non-failing	exit 3 is treated as fail
a literal banned phrase in the suite	no shipped file says …

No live request, paid call, proxy or credential was used.

Not changed

• not_found state for a 404 in --mode product. The audit only suggested it as something to consider. A 404 page still classifies as before. Adding it would be a new state, a new stop reason and changed exit codes, so it is left for a separate change.
• Whether a failed, unbilled solve should count against the page allowance. The audit marks this an owner decision. I made both call sites count attempts, the stricter reading, because the second site already did. Reverting is one line per engine.
• The live 403 behaviour on the ranking route. This repo does not read that route. The status change was not run live.
• Shared fetch loop (audit T10), schema/sinks (T11), and version bump/tag/release (T13). [Unreleased] now carries these behaviour changes on top of Reject --retries 0 instead of fetching nothing #7.
• .env.example's RAKUTEN_URL stays active. This repo has no URL-builder flags (--category is a label only), so an env URL cannot hijack anything. The existing round-trip check asserts that it stays usable.
• Repo metadata. The org-profile audit lists nothing for this repo, and the description and topics were already correct.

🤖 Generated with Claude Code