[jehrr]

Fixes from the 2026-10-14 audit of main at 9dbf278, in one batch.

Problems and resulting behaviour

Captcha budget (high). Each engine checks for a challenge before and after classifying page 1, and re-fetches up to 3 more times. Only the second check was counted, so a page that kept coming back as a challenge bought up to 5 solves against SOLVES_PER_PAGE = 1 in all three engines. Now both call sites take from one page_flow.SolveBudget per page. It is spent immediately before solve_recaptcha and is never reset by a retry, a fresh context or a rotation. Stub-driven probe: 1 solver call per engine.

OPENSEA_URL vs builder flags (high). With the value .env.example shipped, --collection pudgypenguins silently scraped boredapeyachtclub, and --mode collections exited 2. The cause was env_config.apply filling --url before the builder ran.

• The typed URL is now captured first, and an environment URL never outranks --collection, --category or --mode collections.
• A typed --url that names a different collection from --collection is refused.
• .env.example ships the variable commented out.

Atomic writes. write_json, write_csv and write_run_meta now write through _atomic: a temp file in the target's directory, fsync, os.replace, and cleanup on error. A new file gets 0o666 & ~umask; an existing file keeps its mode. Before, an injected error inside json.dump left 12 bytes of invalid JSON.

CSV formula guard. Strings starting with = + - @ TAB CR LF (checked after list joining) get a leading ', in the CSV only. Numbers are untouched and the JSON is untouched. csv_cells_escaped is merged into the sidecar under the caller's extra.

diff_runs.py. Fields are now tracked per mode (items, collections, activity). The mode comes from the sidecar, or is inferred from the rows. Before, planted changes to collections volume or activity event_type gave 0 changed. The docstring and help text no longer describe another site.

Credential scan. Exclusions are now evaluated on path.relative_to(REPO).parts. Before, a clone under live/, captures/ or venv-work/ scanned 0 files and reported clean. A 0-file scan now fails. Virtualenvs are still detected by pyvenv.cfg.

CLI validation. Values below the floor now exit 2 with an argparse error naming the flag:

• in all three engines: --pages 0, --limit 0, --concurrency 0, and negative --delay, --retry-delay, --proxy-block-retries;
• in scraper_api_client.py: --retries -1 (0 stays legal there, because it counts extra attempts).

Private citations. All §N / CLAUDE.md references are gone from tracked files (about 240). That includes the citation-only edits in the released CHANGELOG section. The rules are restated in plain words. A new suite check (tokens assembled from pieces) fails if they come back. The v0.1.0 release notes contained none, so none were edited.

Small items:

• .gitignore now covers *.page[0-9]* dumps, out/, output/, runs/, results/, tmp/, dumps/ and root *.html. A check pins both the ignored and the must-ship sides.
• .dockerignore now has .env*.
• "credentials live about a day" is replaced with "lifetime varies; a 401 means refresh".
• The use_antidetect wording is removed from a docstring.
• The tests.yml inline sample heredoc is replaced by ci_checks.py --sample-check. The original_price count is dropped from that check.

Canary. The canary used to end green on a block. Now:

• exit 3 in any mode, exit 5, and 0 activity rows are red (before, exit 3 and activity exit 4 only warned);
• the market-level priced >= 100 is replaced by "at least one priced row" plus the existing match against OpenSea's own listedItemCount.

No secrets are needed: the canary has passed ungated from GitHub runners (2026-10-15 run: 350/148/300 rows). The three sanity blocks were re-run against that run's artefacts and all pass.

Evidence

• Core-only venv: smoke_test.py → 11754 checks, OK. ci_checks.py --all → OK, with 36 files scanned. pytest → 1 passed.
• Per-engine venvs (no browser download): playwright 11777, puppeteer 11761, selenium 11761 checks, all OK. In each, that engine's group ran.
• 16 planted-fault controls, each with PYTHONDONTWRITEBYTECODE=1 and __pycache__ cleared. Each went red on the named check, and the tree was green after restore:
  • unbudgeted solve ×3 engines
  • env URL wins again
  • --pages type=int
  • scraper_api --retries type=int
  • truncating write_json
  • flat 0644 mode
  • CSV escape disabled
  • escape before list join
  • CSV count merged over extra
  • diff_runs collections using item columns
  • absolute-path exclusions in ci_checks
  • §3 planted in README
  • CLAUDE.md planted
  • .env.example URL uncommented
• A .gitignore control (removing *.page[0-9]*) also went red.
• The new engine checks stub the driver modules in a subprocess, so they also run in the core-only venv. The solver is stubbed. No paid call, proxy or credential was used, and nothing was sent to 2captcha.

Not changed

• Version bump, tag or release (the [Unreleased] section carries the notes).
• The shared fetch-loop refactor.
• Schema/sink features.
• Gating parse_rows on should_parse (audit L2, candidate only, no real phantom-row case).
• pyppeteer page.authenticate proxy auth (L8: needs an authorised proxy run).
• Ranking floors >= 50% threshold (L7, candidate).
• Repo metadata: the org-profile audit lists nothing to fix for this repo.

Owner questions

• Should activity rows be diffable at all? This PR tracks every event column, so a re-indexed event surfaces.
• Release: should this ship as 0.1.1 or 0.2.0?

🤖 Generated with Claude Code