[jehrr]

Problem

An audit of main at fa8b613 reproduced these defects with driver-free probes:

• No-proxy crash. Without --proxy, pyppeteer (on a blocked page or a failed navigation) and Selenium (on a blocked page) called pool.current / pool.advance on None. The result was an AttributeError and exit 1 where exit 3 or 5 was due.
• Solve budget bypassed. The captcha solve that runs before classification was not counted. A page that stayed a challenge bought 3 solves against SOLVES_PER_PAGE = 1 in Playwright. pyppeteer and Selenium bought 2 and then crashed (see above).
• HTTP status dropped. No engine passed the HTTP status to the classifier, so its status != 200 branch could never run.
• Wrong "empty" exit. Two cases exited 4, "empty catalogue": a served page that never painted (final state unknown), and a page holding tiles that parsed to zero rows.
• Credential scan blind spot. From a clone under a directory named venv, the scan read 0 files and still printed "ok".
• Canary never tested the site. Without its secret the canary skipped and stayed green. Its only run ever was that skip. It also warned and stayed green on exit 3, 5 and 6.
• Smaller items:
  • negative --pages, --delay, --concurrency, --retry-delay and --proxy-block-retries values, and the Scraper API client's --retries, --retry-delay and --timeout, were accepted;
  • --allow-empty wrote a "failed" sidecar for a completed empty run;
  • .gitignore did not cover X.pageN dumps or capture directories;
  • code comments and the changelog cited private instruction files by name and section;
  • a docstring still described --mode shop;
  • the canary relied on the retracted claim that a Scraping Browser credential lasts about a day.

Resulting behaviour

• No proxy pool. pyppeteer and Selenium now re-fetch through the same access path, without a relaunch, as Playwright already did. pyppeteer rotates after a load failure only when it has a pool.
• One solve allowance per page. page_flow.SolveBudget is shared by both call sites in all three engines and is claimed immediately before the paid request. Detecting nothing, finding content already rendered or having no key claims nothing. A retry or a rotation does not reset it.
• HTTP status. Playwright and pyppeteer keep the goto() response and pass its status to the classifier, so 403, 429 and 5xx mean blocked (exit 3). Selenium has no status and passes status=NO_HTTP_STATUS at each call. 404 and 410 are deliberately not read as a block: what Tokopedia sends for a dead URL has not been measured.
• New stop reasons. A served page that never painted ends as page_never_painted, and a content page with zero parsed rows as parser_found_nothing. Both exit 5, or 6 when earlier pages gave rows. A real empty page (no-results text, hub, past the end) is still exit 4.
• Flag validation. The counting and delay flags above now fail with exit 2, and argparse names the flag in the error.
• Credential scan. Exclusions are evaluated relative to the repository root, and a scan of 0 files fails.
• --allow-empty. A completed empty run gets a "complete" sidecar with products: 0.
• Canary. With no TOKOPEDIA_CDP_ENDPOINT secret the job fails with an ::error:: that names the secret. Only exit 0 passes, and the data checks (row floor, non-null core fields) then run. It is still dispatch-only; a schedule is to be added once the secret exists.
• Private citations removed from all tracked files, released CHANGELOG sections included (only the citation text was changed there). A suite guard fails if they reappear. The citations were also removed from the v0.1.1, v0.1.3 and v0.1.4 release notes; the rest of each body is byte-identical.

Evidence

• Core-only venv. PYTHONDONTWRITEBYTECODE=1 python smoke_test.py gives smoke_test: OK, 537 PASS, 0 FAIL. The 4 engine groups are skipped by name, as before.

• Engine venvs. Each venv installs requirements.txt plus one engine file, with no browser download:

  Venv	Result
  playwright	OK (561 PASS)
  pyppeteer	OK (556 PASS)
  selenium	OK (549 PASS)

  In each venv that engine's group ran. The new fetch-loop probe imported the real driver there and a stub driver module elsewhere.

• CLI checks. In each engine venv, --pages -1, --delay -2, --concurrency 0, --proxy-block-retries -1 and --retry-delay -1 give exit 2 with argument --<flag>: …. In scraper_api_client.py, the same holds for --retries -1, --timeout 0 and --retry-delay -1.

• Driven checks. New suite checks run each engine's real _fetch_one_page against a fake session:

  • no pool + blocked, and no pool + load failure: an outcome, not an exception;
  • a page that stays a challenge, with no pool and with a 3-exit pool: exactly 1 purchase from a stubbed solver, with no network;
  • a 403 from the fake goto reaches the classifier and the page is blocked (Playwright and pyppeteer);
  • unknown gives page_never_painted, content with 0 rows gives parser_found_nothing, and empty stays ok.

• Controls. Each fault was planted, the suite went red on the named check, the file was restored, and the suite went green again:

  • guard removal: pyppeteer block and load-failure pool guards, Selenium block guard;
  • budget claim disabled: Playwright, Selenium;
  • status: Playwright status dropped, pyppeteer classify call without a status;
  • not_obtained_reason returning None;
  • credential scan: absolute-path exclusions, no 0-file guard;
  • a planted private citation;
  • flag types: Selenium --pages type=int, Scraper API --retries type=int;
  • canary: exit 3 not failing, missing secret exiting 0;
  • --allow-empty writing "failed".

No paid calls, proxies, credentials or live requests to Tokopedia were made. The site refuses datacentre addresses.

Not changed

• Canary schedule. It stays dispatch-only until the TOKOPEDIA_CDP_ENDPOINT secret exists, because a daily run would be red for a known reason. This is the owner's decision.
• Refusing --concurrency with --cdp-endpoint or on the mirror engines. Still warned and ignored; this is an owner decision.
• No not_found state. A dead product URL now ends as exit 5 rather than 4. A separate state needs a live measurement first.
• Shared fetch-loop refactor. Separate work.
• Version bump, tag and release. [Unreleased] now holds the --retries 0 change and the changes above.
• Credential scan scope. The scan does not read .json or .env* backups, and the audit did not flag that.

🤖 Generated with Claude Code