Skip to content

Paid solves stay within one per page, atomic outputs, CSV formula guard, canary red unless it read Medium - #12

Merged
jehrr merged 1 commit into
mainfrom
fix/audit-2026-10-14
Oct 15, 2026
Merged

jehrr merged 1 commit into
mainfrom
fix/audit-2026-10-14

Conversation

@jehrr

@jehrr jehrr commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Fixes from the 2026-10-14 audit of main @ 23bcabb (no open PRs at the time).

Problems and resulting behaviour

  • Captcha budget bypassed (F1). The post-navigation handle_captcha_if_present call had no budget, so with detection forced and the solver stubbed, one page with no pool bought 4 solves against SOLVES_PER_PAGE = 1 in every engine. Now one page_flow.SolveBudget per page is shared by both call sites, checked right before the purchase, and not reset by retries or relaunches. README/TROUBLESHOOTING no longer say nothing can ever be charged: the managed challenge is never sent to a solver; a rendered reCAPTCHA (not seen on Medium) is the one paid path, at most once per page.
  • Non-atomic writes (F2). JSON, CSV and sidecar go through _atomic (temp file in the same dir, fsync, os.replace, cleanup on error, newline="" for CSV). New file 0o666 & ~umask; existing file keeps its mode.
  • CSV formula guard (F3). String cells starting = + - @ TAB CR LF get a leading ' in CSV only, after list joining; numbers untouched; csv_cells_escaped in the sidecar, under the caller's extra.
  • should_parse now gates the parse (F4): an empty page gives 0 rows (it gave 6 with the classifier forced).
  • Status reaches the classifier (F5) in Playwright and pyppeteer (dropped after a solve-reload). Selenium has no status; unchanged.
  • Credential scan (F6): exclusions on relative_to(REPO).parts; 0 files scanned fails; .json/.html/.csv/.toml/.cfg/.ini, Dockerfile and .env.* are read (hits in git-ignored files warn).
  • Private citations (F7) removed from all tracked files, including released CHANGELOG sections; suite guard added (tokens assembled from pieces).
  • Donor residue (F8): EXIT_BLOCKED comment describes Medium's two Cloudflare refusals; shop_rating and FINDINGS.md references gone; sample check counts is_paywalled instead of original_price.
  • --category (F9): help says it is accepted and unused; a warning when passed. (Removing it is left to the owner.)
  • README exit 5 (F10), "about a day" credential lifetime (F13) removed from .env.example and the 401 advice, dump shapes ignored (F15), "2Captcha Scraping Browser API" naming (F17).
  • Numeric flags: --pages 0, --concurrency 0, negative --delay/--retry-delay/--proxy-block-retries (and negative --retries on scraper_api_client) are argparse errors naming the flag.
  • tests.yml: inline --help/sample heredocs removed (ci_checks.py already ran both); suite fails if a workflow imports a local module inline.
  • Canary (F14): exit 3 used to be a green warning (3 of the last 14 runs tested nothing at all). Now each half gets one rested retry (120 s) on exit 3; then exit 3/4/5 or a step that never ran fails the job with an ::error:: saying whether it was access or the parser. Exit 0/6 must pass the data floors. No secret required (bare runner is served most days); MEDIUM_PROXY stays optional.

Evidence

  • Core-only venv: smoke_test: OK, 0 FAIL. Per-engine venvs (playwright, selenium, pyppeteer; no browser download): OK, 0 FAIL, only the other engines skipped, pip check clean.
  • New offline groups load each engine with stubbed driver modules (never skip): budget over 4 attempts = 1 solve; empty page = 0 rows; status 403 seen by classifier.
  • Controls, each in a throwaway copy with PYTHONDONTWRITEBYTECODE=1, all RED on the named check, no crash: budget bypass (playwright), spend() removed (selenium), parse gate removed (pyppeteer), status dropped (pyppeteer), non-atomic JSON, CSV escape off, flat 0644, scan by absolute parts, planted citation, /*.html un-ignored, --pages type=int, canary exit 3 as warning, inline workflow import.
  • No live request, no paid call, no credential used.

Not changed

  • F11 --allow-empty sidecar status, F16 version/tag/release, T15 commit-subject labels: owner decisions.
  • F12 refusing --concurrency/--fingerprint with --cdp-endpoint instead of warning: behaviour change left for the owner.
  • No not_found state for 404 (no real Medium 404 capture to build it on).
  • Shared fetch loop refactor (T10).

🤖 Generated with Claude Code

… and the canary is red unless it read Medium

- One SolveBudget per page covers both solver call sites in all three
  engines, checked before each purchase and never reset by a retry.
- JSON, CSV and the sidecar are written atomically; new files follow the
  umask, existing files keep their mode.
- CSV cells starting with a formula character are neutralised (CSV only),
  counted as csv_cells_escaped in the sidecar.
- A page the state policy does not parse yields no rows.
- Playwright and pyppeteer pass the HTTP status to the classifier.
- Numeric flags reject impossible values (--pages 0, negative delays...).
- Credential scan: exclusions relative to the repo, an empty scan fails,
  data files and .env.* backups are read.
- .gitignore covers multi-page and top-level dumps and capture directories.
- --category is documented as accepted and unused; README exit 5 row,
  captcha-charge sentences and product naming corrected; an unmeasured
  credential-lifetime claim removed; other-site comments fixed.
- Private-note citations removed from public files, with a guard.
- Canary: exit 3/4/5 fail the job after one rested retry per half.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr
jehrr marked this pull request as ready for review October 15, 2026 13:04
@jehrr
jehrr merged commit 3b100b8 into main Oct 15, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant