[jehrr]

Fixes from the 2026-10-14 audit of b98079a (main unchanged since; no open PRs).

What changes for a user

• No proxy, blocked page → exit 3 instead of a crash. Selenium (block retry) and pyppeteer (block retry and dead-exit retry) called pool.current / pool.advance on None: traceback, exit 1, and rows from earlier pages lost. Both now re-fetch on the same path, as Playwright did. pyppeteer also rotates only for an unusable exit, matching Playwright.
• A failed write keeps the previous good output. write_json, write_csv and write_run_meta go through one _atomic helper: temp file in the target's directory, flush + fsync, os.replace, unlink on error, newline="" kept for CSV. A new file gets 0o666 & ~umask; an existing file keeps its mode.
• CSV formula guard. String cells (after list/dict serialisation) starting with = + - @ TAB CR LF get a leading ' in CSV only; numbers and JSON untouched; csv_cells_escaped is merged into the sidecar under the caller's extra.
• 404 is terminal. Playwright/pyppeteer pass the response status to classify; BBB's 404 (by status, or by its own <title> for Selenium, which has no status) is a new not_found state: no wait, no retry, not blocked, stop_reason: "not_found" (exit 5 with no rows). A marker-less 403 now classifies as blocked. The smoke check that pinned the 404 as unknown is rewritten, with the reason.
• BBB_URL in .env no longer blocks --text/--location. The typed --url is captured before env_config.apply; an environment URL is a default the query flags replace. A typed --url + --text is still exit 2. .env.example ships BBB_URL commented out.
• Flag validation. --pages, --concurrency (≥1), --delay, --retry-delay, --proxy-block-retries (≥0) in all three engines; scraper_api_client.py --retries/--retry-delay (≥0, 0 stays legal there), --timeout (≥1). argparse names the flag.
• Credential scan. Exclusions use path.relative_to(REPO).parts (a clone under a venv/ ancestor scanned 0 files and said ok); a 0-file scan fails; .env* copies and .json/.csv/.html/.toml/... are read. The pyvenv.cfg detection and allowlists are unchanged.
• .gitignore: *.page[0-9]*, /*.html (except landing.html), live/ out/ output/ runs/ results/ tmp/ scratch/ dumps/.
• Prose: README exit 5 row; --allow-empty help no longer claims "exit 4 either way"; stale "stream reset / no challenge ever observed" text in --solve-captcha help and docstrings; ci_checks --sample-check no longer counts a nonexistent original_price; a shop_rating example.
• Private citations removed from every tracked file (instruction-file names and §N), with the rule stated in place; a new suite guard (tokens assembled from pieces) fails if they return. Release notes v1.0.0/v1.0.1 contained none, so none were edited.

Evidence

• Offline suite: core-only venv 447 passed, 0 failed (12 skips = absent engines); playwright venv 497/0 (4 skips); selenium venv 466/0; pyppeteer venv 466/0. ci_checks.py --all clean.
• New behavioural checks run the real _fetch_one_page of all three engines with the driver library stubbed (so they run in the core-only CI job too): blocked page and dead exit with pool=None, served 404 fetched once, status 404 reaching the classifier; real parse_args of all three engines for the env-URL and negative-flag cases; injected write failures; modes under umask 022/002/077 and an existing 0640 file.
• Planted-fault controls (PYTHONDONTWRITEBYTECODE=1, __pycache__ removed each time), all 16 went red on the named check and green on restore: atomic write, chmod removed, CSV escape off, extra-merge order reversed, Selenium pool guard, pyppeteer load guard, not_found off, Playwright status dropped, typed-URL logic off, --pages type=int, API --retries type=int, absolute-path exclusion, zero-scan guard, .env* scan, a planted citation, *.page[0-9]* removed.
• No live requests, no paid calls, no credentials.

Not changed (and why)

• --allow-empty overwriting good output on a blocked/failed run (F10), refusing vs warning for --fingerprint/--concurrency with --cdp-endpoint and the README's "refused" wording (F11), version/tag 1.0.2 (F14), profile canary exit-5 policy (F16), shared fetch loop (T10): owner decisions.
• Provenance comments naming sibling repos stay (only private-document citations were removed). Commit history and the "(§19)"/"leftovers" subjects are not rewritten.
• No version bump, tag or release. Repo description/topics already fine; no metadata edit.

🤖 Generated with Claude Code