[jehrr]

Fixes from the 2026-10-14 audit of main at 8bc19c7, plus the owner's request that the canary only goes green when it actually tested etsy.com.

Problem → behaviour now

1. Mirror engines crashed with no proxy pool (high). With no pool (the normal case over --cdp-endpoint), Etsy's usual DataDome t=bv refusal made pyppeteer and Selenium call pool.current / pool.advance on None. The run ended with a traceback and exit 1 instead of exit 3. pyppeteer did the same on a repeated load failure.
Now: both re-fetch through the same access path without relaunching, as Playwright already did, and they rotate only when a pool exists. pyppeteer now rotates only for an unusable exit, matching its twins. A timeout has already had its --retries.

2. Paid solves exceeded the per-page limit. The reCAPTCHA check runs on every block-retry and was never counted. With the solver stubbed and no pool, one page bought 4 solves against SOLVES_PER_PAGE = 1.
Now: page_flow.SolveBudget is one object per page, shared by the reCAPTCHA and DataDome call sites in all three engines. It is checked immediately before each purchase and never reset by a retry or a rotation.

3. A served page that parses to nothing was reported as an empty catalogue.
Now: a listing or shop page that Etsy itself says holds listings but yields 0 rows ends with stop_reason: parser_found_nothing. "Says holds listings" means:

• page 1: numberOfItems > 0;
• any page: the page number is within initial_total_pages.

That gives exit 5, or 6 after earlier pages. Before, the same page gave exit 4, or on page 2+ a complete run with exit 0 holding page 1 only.

4. The HTTP status never reached the classifier.
Now: Playwright and pyppeteer pass the navigation status to the classifier. The status is dropped once the document is replaced (by a solve or a DataDome interstitial resolving), so a stale 403 cannot mislabel the page. A 404 or 410 is a new not_found state: final, not blocked, not parsed. On page 1 it gives exit 5; past page 1 it ends the listing. Selenium has no status and is unchanged.

5. New output files ignored the umask. They were always 0644, including under umask 077.
Now: a new file gets 0o666 & ~umask, and an existing file keeps its mode. Tested under umask 022, 002 and 077 for JSON, CSV and the sidecar.

6. The credential scan could pass over nothing. Exclusions were tested against absolute path parts, so a clone under any venv/ or .git/ directory scanned 0 files and reported clean.
Now:

• Exclusions are judged relative to the repo.
• A scan that read no files fails.
• It also reads .json, .html, .csv, .toml, Dockerfile and .env.* backups. A hit in a git-ignored file is a warning; anywhere else it fails.

7. Housekeeping

• .gitignore now covers /*.html, *.page[0-9]* and live/. A check confirms that shipped files and fixtures are not ignored.
• tests.yml runs ci_checks.py --help-check --sample-check instead of inline copies that had drifted. diff_runs.py was added to the CLI list, and a guard rejects inline local imports in workflows.
• Fixed the garbled block log line in both mirrors, and a captcha_solver.py comment that described another site's hosts as Etsy's.
• Removed citations of private maintainer notes from all tracked files, including released CHANGELOG sections. A guard rejects new ones (its tokens are assembled from pieces). The v0.2.2 release notes were edited the same way; only that one citation was removed.

8. Canary (owner request). Before: dispatch-only, skipped green with a notice when no secret was set, and treated exit 3, 5 and 6 as green warnings. The only run so far (2026-09-10) was a green skip.
Now:

• Runs weekly (Monday 06:23 UTC) and on demand.
• A missing ETSY_CDP_ENDPOINT secret fails the job, and the error names the secret.
• Exit 3, 5 and 6 fail the job, and the message names the cause.
• Data thresholds are unchanged: row floor, field floors, invariants.
• The unmeasured claim that "an endpoint does not survive a day" was removed from the canary, README and CONTRIBUTING.

This secret is required: ETSY_CDP_ENDPOINT (a Scraping Browser API endpoint with a pid reserved for CI). It is not set today, so the canary is red until it is.

Evidence

• Offline suite, core-only venv: 626 PASS, 0 FAIL. Also run in one venv per engine, without browser downloads. Each passed with only its own engine's group running, and pip check was clean.
• The new suite group loads each engine file with stub driver modules, so it runs in every venv, CI's offline job included. It drives _fetch_one_page through five cases:
  • t=bv with no pool;
  • a dead exit with no pool;
  • a stubbed solver against a page that stays a challenge;
  • a served page that parses to nothing (pages 1 and 2);
  • a 404 on page 1 and on page 3.
• Controls: 16 faults were planted one at a time in a disposable copy, with PYTHONDONTWRITEBYTECODE=1. Each one turned the suite red on the expected check, with no crash:
  • reverted pool guards (pyppeteer block branch, Selenium block branch, pyppeteer load failure);
  • the budget removed (Playwright, Selenium);
  • a flat 0644 mode;
  • parser-failure detection off;
  • 404 classification off;
  • the status dropped in pyppeteer;
  • the scan tested against absolute path parts;
  • data files left out of the scan;
  • a planted private citation;
  • /*.html removed from .gitignore;
  • canary exit 3 green;
  • canary missing-secret green;
  • an inline workflow import.

Not changed

• Version, tag and release (the orphaned 0.2.3 and [Unreleased]): owner decision.
• Refusing rather than warning on --fingerprint, --proxy or --concurrency with --cdp-endpoint: a behaviour change that needs owner agreement.
• Flag parity between engines (--fingerprint and --locale are Playwright-only), unused page_bound, counts_as_blocked and pagination_is_addressable, and recounting the cf-turnstile/Akamai markers: low severity, left for a separate pass.
• Selenium not_found: no status, and there is no captured Etsy 404 body to key on.
• Moving to a shared fetch loop: a separate refactor.
• .env.example ETSY_URL: --category here is a label, not a URL builder, so nothing is hijacked.
• No live run: Etsy is DataDome-gated from this host, and no paid path was used.

🤖 Generated with Claude Code