Skip to content

One solve per page, CSV formula guard, parser failures and 404s reported, canary that must test the site - #13

Merged
jehrr merged 1 commit into
mainfrom
fix/audit-2026-10-14
Oct 15, 2026
Merged

jehrr merged 1 commit into
mainfrom
fix/audit-2026-10-14

Conversation

@jehrr

@jehrr jehrr commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

Fixes from the 2026-10-14 audit of 8c653e3, plus the owner's canary requirement.

Behaviour changes

  • A served listing whose payload names ads but parses to zero rows ends with stop_reason: parser_found_nothing → exit 5 (or 6 with rows from earlier pages). Before: exit 4 on page 1, a complete run (exit 0) on page 2+.
  • --pages 0, --concurrency 0, negative --delay/--retry-delay/--proxy-block-retries (and in scraper_api_client.py negative --retries/--retry-delay, --timeout < 1) are usage errors (exit 2) naming the flag.
  • CSV cells starting with = + - @ \t \r \n get a leading apostrophe (CSV only; JSON untouched; csv_cells_escaped in the sidecar, merged under the caller's extra).
  • Canary: fails when DUBIZZLE_CDP_ENDPOINT is missing (named in ::error::), red on exit 3/5/6, red on a logged parser_found_nothing, weekly schedule added, no longer requires a merchandised "car of the week" row.

Fixes

Finding Fix Evidence
F1 solve budget bypassed page_flow.SolveBudget, passed to both handle_captcha_if_present call sites in all three engines, allows()/spend() immediately before solve_recaptcha Stubbed solver, forced challenge, no pool: 1 purchase per engine (audit measured 3). AST check + behavioural check per engine
F2 pyppeteer pool.advance on None guarded with has_pool as Playwright does stubbed failing goto, pool=None → load_failed, no exception
F4 CSV formula cells _csv_escape after list/dict serialisation, strings only =HYPERLINK(...) → '=HYPERLINK(...), -5.0 unchanged
F5 status dropped Playwright/pyppeteer pass goto()'s status (reset to None after a solve reload); Selenium relies on product_parser.is_not_found_page (empty <title> + empty Redux action list: matches only the 404 among the 13 captures) not_found fixture → empty in all three engines
F6 parser failure read as empty PageOutcome.parse_failed, page_flow.failure_reason used by every stop-reason site stubbed parser on motors_p1 → parser_found_nothing; finish_run → 5 / 6
F8 private citations 93 §N / private-file references reworded or removed; new suite guard (tokens assembled from pieces) over every tracked file. Release v0.1.0 notes checked: none grep → 0
F9 other-site residue search-grid/GraphQL/keyword=kopi/buy-box/scroll-count/2026-09-09 comments, umlaut URL docstring, mugs Docker example rewritten for this site grep
F13 capture shapes .gitignore + CAPTURE_SHAPES by kind (live/ out/ output/ run(s)/ result(s)/ tmp/ scratch/ dump(s)/, root *.html) git check-ignore; intent-to-add live/dump.html → ci_checks.py --secret-check FAILED
F15 stale wording README exit-5 line; smoke-test skip footer
F16 wording scan exempted the suite banned phrases assembled from pieces, suite scans itself
CONTRIBUTING dropped the unmeasured "credential does not survive a day" claim and the pointer to private notes
Metadata description now says "2Captcha Scraping Browser API" (gh repo edit)

Verification

  • Offline suite green in four venvs: core-only (528 PASS), +playwright (560), +pyppeteer (555), +selenium (548), each running its own engine's new fetch-path checks. No browsers downloaded; no network calls to 2captcha (solver stubbed) or the site.
  • Planted-fault controls (PYTHONDONTWRITEBYTECODE=1, __pycache__ cleared), each RED on the named check, then restored green: CSV escape removed; budget.spend() removed (AST check in core, behavioural in selenium venv); pyppeteer guard removed; one _classify without status; is_not_found_page → False; parse_failed not set (playwright venv); --delay back to type=float; a § in README; a banned phrase in the suite; canary exit 3 not failing; canary parser grep disabled; missing-secret step not failing.
  • ci_checks.py --all green.

Not changed

  • Atomic writes (F3) and Python 3.13 in CI (F10): covered by Audit 2026-10-08: atomic output writes, Python 3.13 in CI #11, not duplicated.
  • F7 --allow-empty sidecar status word, F12 refusing CDP combinations, F11 tagging 0.1.1 / release: owner decisions.
  • F17 pyppeteer proxy-auth/UA/connect patterns: needs a live proxy/CDP run.
  • Shared fetch loop, schema/sinks: out of scope. No version bump, tag or release.

🤖 Generated with Claude Code

…ted correctly

- Every paid captcha solve in all three engines goes through one per-page
  allowance (page_flow.SolveBudget), checked before the purchase; a
  challenged page bought 3 tokens against a limit of 1 before.
- pyppeteer no longer crashes (exit 1) when a page fails to load without a
  proxy.
- The HTTP status reaches the classifier; dubizzle's 404 is recognised from
  the body where Selenium has no status.
- A served page with ads that parses to zero rows ends as
  parser_found_nothing (exit 5, or 6 with rows), not exit 4 or a complete run.
- CSV cells that start with = + - @ or a control character are prefixed with
  an apostrophe (CSV only), counted in the sidecar as csv_cells_escaped.
- Count and delay flags reject 0/negative values with an argparse error.
- Capture directories ignored and flagged by kind; wording scan covers the
  suite; private-note citations removed and guarded.
- Canary: fails without its secret, red on exit 3/5/6, weekly schedule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr
jehrr marked this pull request as ready for review October 15, 2026 21:14
@jehrr
jehrr merged commit ec80da1 into main Oct 15, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant