No-proxy crashes, one paid solve per page, scope-aware diffs, stricter credential scan

[jehrr]

Fixes from the 2026-10-14 audit of main at fe2a381.

Problems and resulting behaviour

Finding	Before	After
Selenium/pyppeteer with no --proxy	a refused or empty page (and in pyppeteer a navigation that never loaded) hit pool.current/pool.advance on None: AttributeError, exit 1	re-fetch through the same access path, as Playwright does; exit 3/5 as appropriate
Captcha budget	the pre-classification call was unbudgeted: 2 purchases with no pool, 4–5 with a pool of four, limit 1 (stubbed solver)	one page_flow.SolveBudget per page, checked immediately before purchase at both call sites, not reset by retry or rotation (budget is a required keyword, so a new call site cannot skip it)
HTTP status	goto()'s response discarded; a branded 403/429/503 would read as "unpainted" and be waited on	Playwright and pyppeteer pass the status to classify; reset to None after a solve reloads the page. Selenium has no status. No not_found state added
Numeric flags	--pages 0, --concurrency 0, negative --delay/--retry-delay/--proxy-block-retries accepted	argparse error naming the flag, exit 2, in all three engines; scraper_api_client.py rejects negative --retries/--retry-delay (0 stays legal there: it counts extra attempts)
diff_runs.py scope	newyork price band vs losangeles diffed as added/removed, exit 0	refused unless --force when sidecar start_urls differ in host, path or query (order-free, fragment ignored); a warning when a run hit the result ceiling
Credential scan	exclusions on absolute path.parts: a clone under venv/ or .venv/ scanned 0 files and passed; .env.bak never read	exclusions on the path inside the repo, fails on 0 files scanned, reads .env.* copies (not .env itself) and .json/.csv/.html; venvs are still skipped by pyvenv.cfg
Private citations	4 code files and the CHANGELOG cited a private working file or its sections	removed and the rule stated in the repo's own words; a new suite check scans every shipped text file (tokens assembled from pieces)
Docs	README said exit 5 = remote API error, --concurrency >1 "refused", a "653 checks" count, nothing about --locale; the Scraper API help and challenge message described another site	corrected; canary exit-5 message too
.gitignore	live/, results/, foo.page3, root x.html not ignored	*.page[0-9]*, /*.html, scratch dirs by kind; .github/, tests/, samples and .env.example still tracked (checked with git check-ignore)

Also fixed: pyppeteer's "grid still had not painted" log used found <= need, so it fired on a page that had painted (log only).

Evidence

• New suite check test_fetch_loop_without_a_pool_and_within_the_solve_budget runs each engine's real _fetch_one_page in a subprocess with every driver module stubbed. It runs in every venv, the bare one included. Against old main it reproduced the audit: Playwright 2/4/5 solves, and Selenium and pyppeteer AttributeError. On this branch every case raises nothing and buys ≤1 solve.
• Offline suite: core-only venv exit 0, and one venv per engine (requirements + that engine, no browser download) exit 0, each skipping only the two absent engines. ci_checks.py --all green (36 files scanned).
• Planted-fault controls (run with PYTHONDONTWRITEBYTECODE=1, __pycache__ cleared). Each one turned the suite red on the check it was meant to trip, and the restored tree is green:
  • Selenium no-pool guard
  • pyppeteer load-failure guard
  • solve budget
  • status to the classifier
  • scan on relative parts
  • scan fails on 0 files
  • scan reads .env copies
  • private citation in README
  • diff scope refusal
  • numeric flag validator
• No requests to craigslist.org, no paid calls, no keys, no proxies.

Canary (owner request: a canary must actually test the site)

• Before: ungated daily run, 3-batch walk plus a 1-batch served-list run. Exits 3, 4, 5 and 6 of the walking run were already red. But a non-zero exit of the served-list run was a ::warning:: with exit 0, so the served list and its JSON-LD alignment could go untested on a green badge. The walking run's 90% price-share floor went red on 4 of 32 scheduled runs (79–90%), and the artefacts show free and "make offer" adverts behind it.
• After:
  • a non-zero served-run exit is ::error::;
  • every row must carry sku, url and title;
  • the price-share floor is 60%, which a broken price read still falls under (checked against the 10-09 artefact: 79% passes, and the same artefact with prices nulled and one title removed fails on both);
  • the served-batch row floor is 150, not 250. The first dispatch on this branch read 243 rows, every one parsed, and failed the 250 floor: a time-of-day total, not a parser check.
• Secrets: none required. The README and every scheduled run show the site serving a bare GitHub runner, and CRAIGSLIST_PROXY stays optional.
• Dispatch on this branch:
  • https://github.com/2scraper/craigslist-scraper/actions/runs/38694580494 (red, 250 floor vs 243 rows; fixed);
  • https://github.com/2scraper/craigslist-scraper/actions/runs/38694821748 is green: 1,000 walked rows, 99% priced; served batch 234 rows, 98% structured, USD.

Not changed, and why

• Scraper API client sidecar / --allow-empty (L3), sidecar total_results/source (L4): owner decision.
• Refusing rather than warning for --proxy/--fingerprint with --cdp-endpoint and for --concurrency > 1 (L6): owner decision. The docs now describe the warning.
• Version, tag and release (L1): out of scope. Changes are under [Unreleased], with behaviour changes in leading blockquotes.
• Shared fetch loop (T10): a refactor that needs live runs of all three engines.
• CRAIGSLIST_URL in .env.example: kept. This repo has no URL-builder flags for it to hijack, and the audit's round-trip found it fine.
• Release notes v0.1.0–v0.1.2 contain no private citations, so none were edited. Repo metadata: no finding for this repo.
• Historical commit subjects: not rewritten.

🤖 Generated with Claude Code