Selenium and pyppeteer no longer crash on a no-proxy block, read --mode job pages, and CSV cells are formula-safe

[jehrr]

Problem

An audit of main at d92df63 (2026-10-14) reproduced these problems:

• Selenium and pyppeteer crashed on the normal no-proxy outcome. A refused page with no proxy pool reached mask(pool.current) / pool.advance() with pool = None and raised AttributeError. pyppeteer did the same on a plain load timeout. A datacentre address is refused on every route, so this was the expected path for those engines.
• Selenium and pyppeteer misread every --mode job page. Their classify calls omitted mode, so a job page was judged as a listing and came out parse_error: zero rows, exit 5.
• The HTTP status never reached the classifier. not_found was unreachable in every engine, so a removed job was waited on, re-fetched and reported as exit 5. The 0.1.2 notes said a 404 exits 4.
• There was no CSV formula guard. Employer-written text such as =HYPERLINK(...) reached the CSV unchanged.
• A copied .env.example broke the builder flags. Its active WELLFOUND_URL made --role X and --mode jobs exit 2, as if --url had been typed alongside them.
• The credential scan could check nothing and still pass. It tested exclusions on the absolute path, so a clone under build/, venv/ and similar directories scanned 0 files and printed ok.
• The canary had never tested the site. All 28 scheduled runs skipped green for lack of WELLFOUND_PROXY.
• Comments and help text were misleading. They described other sites' pages (a BBB block page, /p/<slug> hubs, Akamai, a --mode profile, a SHOP FRONT), and a challenge state that does not exist. Tracked files also cited private working notes.

Resulting behaviour

Behaviour changes are listed under [Unreleased] in the CHANGELOG.

• Refusals and timeouts without a proxy. Selenium and pyppeteer handle a refused page the way Playwright does. With a pool of more than one exit they rotate. Without one they re-fetch once through the same path and report blocked (exit 3). pyppeteer rotates on a load failure only for a dead exit with a pool.
• mode is passed to every classify call. --mode job yields its row in all three engines.
• Status reaches the classifier.
  • Playwright and pyppeteer keep the goto response. Its status goes to the first classification and is reset to None after a wait or a solve, because by then it may describe a different document.
  • A 404 is now not_found and exits 4. The Scraper API client does the same.
  • Selenium has no status to pass and still decides from the page body.
• CSV formula guard. String cells that start with = + - @ TAB, CR or LF get a leading ' in the CSV only, after list joining. Numbers and the JSON output are untouched. The sidecar records csv_cells_escaped, merged under the caller's extra.
• .env.example and the builder flags. .env.example ships WELLFOUND_URL commented out. Each engine records the URL that was typed before the environment is applied. A URL from the environment yields to --role, --location and --remote, and to an explicit --mode job or --mode jobs. A typed --url given together with --role is still refused.
• Counting flags reject bad values with exit 2, naming the flag.
  • --pages and --concurrency must be at least 1.
  • --proxy-block-retries must be at least 0. A negative value made the block loop range(0), so nothing was fetched.
  • --delay and --retry-delay must be at least 0. A negative value crashed in time.sleep.
  • In the Scraper API client, --timeout must be at least 1, and --retries and --retry-delay must be at least 0.
• Credential scan. Exclusions are now judged on path.relative_to(REPO).parts, and a scan that reads 0 files fails. pyvenv.cfg detection and the scanning of untracked files are unchanged.
• Canary.
  • The listing job fails with ::error:: naming WELLFOUND_PROXY when the secret is absent.
  • A new datacentre job is documented as "datacentre refusal expected". It runs from the bare runner and requires exit 3 with the refusal page saved, or real rows. Exit 1 or 5 there is red.
  • The exit-5 explanation now describes what exit 5 currently means.
• .gitignore now covers *.page[0-9]* (--dump-html page files with any base name), live/ and root-level *.html.
• Wording.
  • Donor references are removed.
  • Comments describe the real solve path: the solve runs on blocked, and a task is built only when a sitekey was captured.
  • Every private-notes citation is removed from tracked files, with the rule stated in plain words where it mattered. No released CHANGELOG section or release note contained one; checked for v0.1.0–v0.1.2.

Evidence

• Offline suite.

  • Core-only venv: python smoke_test.py gives 524 passed, 0 failed, 18 groups skipped (404 before this change).
  • Playwright venv: 564 passed, 0 failed.
  • Selenium venv: 550 passed, 0 failed.
  • pyppeteer venv: 550 passed, 0 failed. In each engine venv, that engine's group ran.
  • The engine venvs were installed without browser downloads.

• New checks run in every venv. They load each engine with its driver library stubbed and drive the real _fetch_one_page with a fake session, so they do not skip when the driver is absent.

• Fault controls. Each fault was planted in a fresh copy with PYTHONDONTWRITEBYTECODE=1 and __pycache__ removed. Each made the named check fail, and the restored tree is green.

  Planted fault	Result
  Selenium no-pool branch reverted	selenium_scraper: a branded refusal with no proxy is reported, not raised fails with AttributeError
  pyppeteer load-failure gate reverted	puppeteer_scraper: a load timeout with no proxy is reported, not raised fails
  mode= dropped from one Selenium classify call	selenium_scraper:637 classify(...) passes mode and ...is classified content (got parse_error) fail
  Playwright status discarded	...is classified not_found (got unknown) and ...and is not re-fetched fail
  Environment-URL drop removed in Playwright	the three env URL + builder-flag checks fail with exit 2
  pyppeteer --mode jobs gate broken	env URL + --mode jobs reads the feed fails
  CSV escape disabled	the 3 prefix checks and the sidecar counts the escaped cells fail
  path.parts restored in the scan	the venv/, build/, .venv/ and __pycache__/ parent checks fail
  Zero-file guard removed	a scan that read 0 files FAILS instead of passing fails
  A private citation added to README	README.md:1 cites no private notes fails
  --proxy-block-retries back to type=int	selenium_scraper --proxy-block-retries -1 exits 2 naming the flag fails
  WELLFOUND_URL uncommented in .env.example	WELLFOUND_URL is an example, not an active setting fails

• Probes.

  • Scraper API _run_once with a stubbed fetch_html: 404 gives 4, 403 gives 3, and a served job page that parses to nothing gives 5.
  • ci_checks.py --help-check --sample-check --secret-check: ok.
  • git check-ignore: canary_dump.json.page3, out.page2, live/x.html and foo.html are ignored. The tracked sources and the samples are not.

• No live, paid, proxy or credentialled requests were made.

Not changed, and why

• Solve accounting (audit L2). The budget counts attempts rather than purchases. Changing that is marked as an owner decision.
• pyppeteer proxy auth, connect(ignoreHTTPSErrors) and client hints (audit L4). These are candidates that need a pyppeteer browser and an authenticated proxy to verify. Neither was available within this task's limits: no browser downloads, no proxies.
• A Selenium body signal for a 404. Wellfound's real 404 body has not been captured. The site refuses datacentre addresses, so no free capture was possible.
• status: "failed" on a genuinely empty --allow-empty run (audit L6). This is a sidecar vocabulary question for the owner.
• Out of scope here: the shared fetch-loop refactor, schema and sinks, version bumps, tags and releases.
• Repository metadata already complies: the description names the 2Captcha Scraping Browser API, the topics are correct, and there is no price-monitoring.

The canary stays red until a residential WELLFOUND_PROXY secret is set. That is intended: until then it has not tested the site.

🤖 Generated with Claude Code