Selenium and pyppeteer no longer crash on a no-proxy block, read --mode job pages, and CSV cells are formula-safe

.env.example

@@ -139,7 +139,7 @@ WELLFOUND_PROXY=http://{user}:{password}@na.proxy.2captcha.com:2334
 # residential exit, and this scraper has no mode for it.
 #
 # `--role` and `--location` build a landing URL for you, so this is optional.
-WELLFOUND_URL=https://wellfound.com/role/r/software-engineer
+# WELLFOUND_URL=https://wellfound.com/role/r/software-engineer
 
 # Deliberately NOT here: an output prefix. `--out` already carries a
 # non-empty default, so the loader — which only fills UNSET values — would

.github/ci_checks.py

@@ -81,7 +81,7 @@
 HEX32 = re.compile(r"\b[0-9a-f]{32}\b")
 # Contexts in which a 32-hex string is plainly not a key. Site-specific
 # entries belong here only once a committed fixture actually carries one —
-# CLAUDE.md §16: a family-wide entry copied without checking is how dead
+# A family-wide entry copied without checking is how dead
 # code spreads. Kept as a CONTEXT allowlist rather than by loosening the
 # pattern: a bare 32-hex string anywhere else still fails, which is the
 # point of the check.
@@ -107,7 +107,7 @@ def _is_virtualenv(path):
     walked into pip's vendored code and flagged a 32-hex string in
     `_elffile.py` as key-shaped. The run was correct about the string and
     wrong about the file, and a guard people have to argue with is one they
-    learn to suppress (CLAUDE.md §22).
+    learn to suppress.
 
     Structural rather than by name, for the same reason a parser anchors on a
     URL pattern instead of a CSS class: `venv`, `.venv`, `.v`, `env39` and
@@ -128,7 +128,11 @@ def scanned_files():
             continue
         if path.suffix not in SCANNED_SUFFIXES:
             continue
-        if any(part in _SKIP_NAMES for part in path.parts):
+        # Judged on the path RELATIVE to the repository. The absolute path
+        # also holds the directories the clone happens to live under, and a
+        # checkout under ~/build, /tmp/venv or any other skipped name used to
+        # scan 0 files and report clean.
+        if any(part in _SKIP_NAMES for part in path.relative_to(REPO).parts):
             continue
         if any(root in path.parents for root in skip_roots):
             continue
@@ -220,6 +224,10 @@ def secret_check():
                 failed.append(f"{rel}:{lineno} contains {match[:6]}… — a "
                               f"32-char hex string, the shape of a 2captcha key")
 
+    if not scanned:
+        # A scan that read nothing has checked nothing; it must not pass.
+        failed.append(f"0 files scanned under {REPO} — the exclusions skipped "
+                      f"the whole tree, so nothing was checked")
     if not failed:
         print(f"ok       {scanned} files scanned, nothing credential-shaped")
     return failed

.github/workflows/canary.yml

@@ -1,30 +1,37 @@
 # One real run against wellfound.com. tests.yml (deliberately) never touches
 # the site — see its own header.
 #
-# THIS CANARY IS GATED, AND THAT IS A MEASUREMENT RATHER THAN CAUTION.
+# THIS CANARY NEEDS A RESIDENTIAL EXIT, AND THAT IS A MEASUREMENT.
 #
 # Wellfound refuses a datacenter address on EVERY route, `robots.txt`
 # included. Measured 2026-09-17 from a Hetzner VPS: HTTP 403 to plain HTTP,
 # to headless Chromium and to headful Chromium alike; from a 2captcha
 # residential US exit, HTTP 200 on `/`, `/jobs`, every `/role/…` shape and
 # `/jobs/{id}-{slug}`, three times out of three.
 #
-# A GitHub runner is a datacenter address. So an ungated live run here would
-# be red every single morning, and a check that is always red teaches
-# everyone to ignore checks (CLAUDE.md §11). This workflow therefore SKIPS
-# with a ::notice:: when no proxy secret is set, and runs for real when one
-# is:
+# A GitHub runner is a datacenter address, so the job that checks real data
+# (`listing`) runs through the WELLFOUND_PROXY secret:
 #
 #     gh secret set WELLFOUND_PROXY --repo 2scraper/wellfound-scraper
 #
+# WITHOUT that secret the job FAILS with an error naming it. It used to skip
+# with a notice and end green, and 28 green runs in a row had never touched
+# the site: a skipped canary is not a tested success.
+#
 # The secret goes through the ENVIRONMENT, never a `--proxy` flag: a
-# credential on a command line is readable by anything that can run `ps`
-# (CLAUDE.md §3). `env_config.py` already reads exactly this variable, and an
-# unset secret arrives as an empty string which the loader treats as unset —
-# so the same step is correct with and without it.
+# credential on a command line is readable by anything that can run `ps`.
+# `env_config.py` already reads exactly this variable.
+#
+# The second job (`datacentre`) needs no secret. It is documented as
+# "datacentre refusal expected": it runs the same engine from the bare
+# runner and requires the refusal to be RECOGNISED as one (exit 3 with the
+# refusal page saved), or real rows if the site ever serves the runner. A
+# crash, a timeout or an unreadable page (exit 1/5) is red there. It proves
+# the engine reaches the site and classifies what it gets; it does not
+# replace `listing`, which is the job that tests real data.
 #
 # `secrets` is not available in a job-level `if`, so presence is tested
-# through job `env` and read in each step's `if`.
+# through job `env` and read in a step.
 
 name: canary
 
@@ -40,48 +47,40 @@ permissions:
 
 jobs:
   listing:
-    name: role listing (needs a residential proxy secret)
+    name: role listing (needs the WELLFOUND_PROXY secret)
     runs-on: ubuntu-latest
     timeout-minutes: 25
     env:
       HAVE_PROXY: ${{ secrets.WELLFOUND_PROXY != '' }}
     steps:
       - uses: actions/checkout@v4
 
-      - name: Explain the skip
+      - name: Require the proxy secret
         if: env.HAVE_PROXY != 'true'
         run: |
-          echo "::notice::Skipping the live run: no WELLFOUND_PROXY secret is set."
-          echo "Wellfound answers a datacenter address with HTTP 403 on every route,"
-          echo "and a GitHub runner is a datacenter address, so this cannot pass"
-          echo "without a residential exit. The offline suite in tests.yml covers the"
-          echo "parser; this job covers the live site only."
-          echo ""
-          echo "    gh secret set WELLFOUND_PROXY --repo 2scraper/wellfound-scraper"
+          echo "::error::WELLFOUND_PROXY is not set, so this canary cannot reach wellfound.com and has tested nothing. Wellfound answers a datacenter address (a GitHub runner) with HTTP 403 on every route; set a residential exit: gh secret set WELLFOUND_PROXY --repo 2scraper/wellfound-scraper"
           {
-            echo "### Canary skipped"
-            echo "No \`WELLFOUND_PROXY\` secret. See the job log for why a datacenter"
-            echo "address cannot reach this site."
+            echo "### Canary did not run"
+            echo "No \`WELLFOUND_PROXY\` secret. A datacenter address cannot reach this"
+            echo "site, so without a residential exit there is nothing to test."
           } >> "$GITHUB_STEP_SUMMARY"
+          exit 1
 
       - uses: actions/setup-python@v5
-        if: env.HAVE_PROXY == 'true'
         with:
           python-version: "3.12"
 
       - name: Install
-        if: env.HAVE_PROXY == 'true'
         run: |
           python -m pip install --upgrade pip
           pip install -r requirements.txt -r requirements-playwright.txt
           playwright install --with-deps chromium
 
       # >= 3 pages, because with one page pagination is never exercised —
       # which is exactly how a silent single-page bug survived in a sibling
-      # repo (CLAUDE.md §11).
+      # repo.
       - name: Run
         id: run
-        if: env.HAVE_PROXY == 'true'
         env:
           # Through the environment, not argv. See the header.
           WELLFOUND_PROXY: ${{ secrets.WELLFOUND_PROXY }}
@@ -94,7 +93,6 @@ jobs:
           echo "exit_code=$?" >> "$GITHUB_OUTPUT"
 
       - name: Check what came back
-        if: env.HAVE_PROXY == 'true'
         env:
           EXIT_CODE: ${{ steps.run.outputs.exit_code }}
         run: |
@@ -114,7 +112,9 @@ jobs:
                      "credential before concluding the site changed.",
                   4: "Zero jobs (exit 4). Either this role genuinely has no "
                      "listings now, or the Apollo payload shape moved.",
-                  5: "Remote API error (exit 5) — nothing was tested.",
+                  5: "Content not obtained (exit 5): a load timeout, a dead "
+                     "exit, or a page that was served but could not be read "
+                     "(parse_error). Check the saved debug page.",
                   6: "Partial (exit 6): some pages came back and some did not.",
               }.get(code, "See the log."))
               sys.exit(1)
@@ -225,3 +225,66 @@ jobs:
             canary.meta.json
             canary_page*_debug.html
           if-no-files-found: ignore
+
+  datacentre:
+    # Datacentre refusal expected — see the header. Exit 3 is accepted here
+    # ONLY because `listing` is the job that requires real rows.
+    name: bare runner (datacentre refusal expected)
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt -r requirements-playwright.txt
+          playwright install --with-deps chromium
+
+      - name: Run from the runner's own address
+        id: run
+        run: |
+          set +e
+          python playwright_scraper.py \
+            --role software-engineer --remote \
+            --pages 1 --format json --out bare
+          echo "exit_code=$?" >> "$GITHUB_OUTPUT"
+
+      - name: Check the refusal was recognised
+        env:
+          EXIT_CODE: ${{ steps.run.outputs.exit_code }}
+        run: |
+          code="$EXIT_CODE"
+          if [ "$code" = "3" ]; then
+            if ls bare_page1_debug.html >/dev/null 2>&1; then
+              echo "::notice::Datacentre refusal recognised (exit 3), as expected from a GitHub runner. Real data is checked by the 'listing' job."
+              exit 0
+            fi
+            echo "::error::Exit 3 without the refusal page saved — the block path did not run as designed."
+            exit 1
+          fi
+          if [ "$code" = "0" ]; then
+            rows=$(python -c "import json; print(len(json.load(open('bare.json'))))")
+            if [ "$rows" -ge 20 ]; then
+              echo "::notice::Wellfound served the bare runner ($rows rows). The datacentre gating may have changed; re-measure."
+              exit 0
+            fi
+            echo "::error::Exit 0 with only $rows rows."
+            exit 1
+          fi
+          echo "::error::The bare-runner run exited $code: neither a recognised refusal (3) nor real rows (0). 5 means content was not obtained (timeout or an unreadable page); 1 is a crash."
+          exit 1
+
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: canary-datacentre
+          path: |
+            bare.json
+            bare.meta.json
+            bare_page1_debug.html
+          if-no-files-found: ignore

.github/workflows/tests.yml

@@ -59,7 +59,7 @@ jobs:
         # check still imported the row dataclass by a name this repo renamed,
         # and it failed on this repo's FIRST push while the script it
         # duplicates passed — two sources of truth, one of them stale, which
-        # is exactly the defect CLAUDE.md §17 describes.
+        # is exactly how a check turns into dead code that looks load-bearing.
         #
         # So: one implementation, and `smoke_test.py` now asserts the workflow
         # does not grow another one.

.gitignore

@@ -24,11 +24,18 @@ live_results.zip
 *_debug.html
 *_debug.png
 *.html.page*
+# `--dump-html PATH --pages N` writes PATH.pageN whatever PATH ends in, so
+# `snap.json.page2` and `out.page3` are dumps too.
+*.page[0-9]*
+# Root-level HTML is a dump or a capture; the tracked landing page is listed
+# by name below and is unaffected (ignoring a tracked file changes nothing).
+/*.html
 
 # --- captures kept for parser work ---------------------------------------
 # The smoke suite's fixtures are committed (trimmed, inside smoke_test.py);
 # the multi-megabyte originals they were cut from are not.
 captures/
+live/
 
 # --- the landing page and the proxy list ---------------------------------
 # Both live outside the repo on purpose: the landing page is published
@@ -73,7 +80,7 @@ secrets*
 # recoverable only because that repo was private at the time. On 2026-09-17
 # a sibling had an untracked `.env.bak` holding a 32-hex key in a PUBLIC
 # repository, one `git add -A` from publication. Craigslist was fixed then
-# and the fix never reached anyone else, which is §16's rule unapplied: when
+# and the fix never reached anyone else, which is the rule unapplied: when
 # you fix one, check the siblings.
 .env*
 !.env.example

CHANGELOG.md

@@ -10,6 +10,63 @@ does it leads the release notes rather than being discovered from a bill.
 
 ## [Unreleased]
 
+> **Behaviour changes:**
+> - **CSV cells that start with `=`, `+`, `-`, `@`, TAB, CR or LF now get a
+>   leading apostrophe** so a spreadsheet reads them as text. Only text
+>   cells, only in the CSV; numbers and the JSON output are unchanged, and
+>   the sidecar records how many cells were escaped as `csv_cells_escaped`.
+> - **Counting flags reject values that skipped or crashed a run** with exit
+>   2 naming the flag: `--pages` and `--concurrency` below 1,
+>   `--proxy-block-retries` below 0 (a negative value fetched nothing at
+>   all), `--delay` and `--retry-delay` below 0 (a negative delay crashed in
+>   `time.sleep`). In `scraper_api_client.py`, `--timeout` below 1 and
+>   `--retries`/`--retry-delay` below 0.
+> - **A 404 now exits 4 in the Playwright and pyppeteer engines and in the
+>   Scraper API client**, as the 0.1.2 notes already said it did: the HTTP
+>   status now reaches the page classifier. Before, a removed job was
+>   waited on, re-fetched and reported as content not obtained (exit 5).
+>   Selenium exposes no HTTP status and still decides by the page body.
+> - **`.env.example` ships `WELLFOUND_URL` commented out**, and an
+>   environment URL no longer collides with `--role`/`--location`/`--remote`
+>   or an explicit `--mode job`/`--mode jobs`: only a `--url` typed on the
+>   command line does.
+
+### Fixed
+
+- **Selenium and pyppeteer crashed when a page was refused and no proxy was
+  configured** — the normal outcome from a datacentre address. The block
+  retry touched `pool.current` with no pool and raised `AttributeError`;
+  pyppeteer also did so on a plain load timeout. Both now re-fetch through
+  the same path, as the Playwright engine does, and rotate only when a pool
+  of more than one exit exists.
+- **Selenium and pyppeteer misread every `--mode job` page.** They
+  classified it as a listing, found no listing payload and reported a parse
+  error: zero rows, exit 5. Every classify call now passes the mode.
+- **The canary skipped green for as long as it existed.** Without the
+  `WELLFOUND_PROXY` secret it now fails with an error naming the secret,
+  and a second job runs from the bare runner and requires the datacentre
+  refusal to be recognised as one (exit 3), not misread or crashed. The
+  exit-5 explanation in the canary named an old meaning.
+- **The credential scan skipped everything when the clone lived under a
+  directory named `build`, `venv`, `.venv` and similar**, because the
+  exclusions were tested on the absolute path. They are now tested on the
+  path inside the repository, and a scan that reads 0 files fails.
+- **`.gitignore`** now covers `--dump-html` page files whatever the base
+  name (`snap.json.page2`), a `live/` directory and root-level `.html`
+  dumps.
+- **Comments and help text** no longer describe other sites' pages, a
+  `challenge` state this repo does not have, or a `--mode profile` it never
+  had, and no longer cite private working notes.
+
+### Tests
+
+- The offline suite now loads each engine with its driver library stubbed
+  and drives the real page loop, so the no-proxy block, the timeout, `--mode
+  job` and the 404 status are checked in every venv, not only where the
+  engine is installed. New checks also cover the environment URL with the
+  role flags, the counting-flag bounds, CSV formula escaping, the scan's
+  path handling and private citations in tracked files.
+
 ## [0.1.2] — 2026-10-13
 
 > **Behaviour change:** `--retries 0` is now a usage error (exit 2) instead

CONTRIBUTING.md

@@ -107,13 +107,13 @@ history needs a decision, not a red check on every push.
 
 Then the rest of the presentation, in the order that matters:
 
-1. `python3 smoke_test.py` green, and the canary dispatched at least once —
-   including its SKIP branch, which is what runs when the `WELLFOUND_PROXY`
-   secret is absent. Unlike some siblings in this family, EVERY route on
-   this site is refused to a datacenter address, and a GitHub runner is one,
-   so the canary is gated in full rather than in half. Confirm the skip path
-   runs and goes green, not just the happy one — a check that is always red
-   teaches everyone to ignore checks.
+1. `python3 smoke_test.py` green, and the canary dispatched at least once
+   with the `WELLFOUND_PROXY` secret set. EVERY route on this site is
+   refused to a datacenter address, and a GitHub runner is one, so the
+   `listing` job needs a residential exit; without the secret it FAILS with
+   an error naming it, because a canary that skipped has tested nothing.
+   The `datacentre` job runs without a secret and only confirms that the
+   runner's refusal is recognised as one (exit 3) rather than misread.
 2. The repo description, homepage and topics set (see the family notes on
    what those should say).
 3. Only then the row in the org profile README — and check it with an