Audit fixes: a run that fetched nothing no longer reports "0 jobs"

.github/workflows/tests.yml

@@ -17,7 +17,7 @@ jobs:
       matrix:
         # 3.9 is the floor the README claims. Claiming it without testing it is
         # how a walrus operator or a `X | None` annotation ships and breaks it.
-        python-version: ["3.9", "3.12"]
+        python-version: ["3.9", "3.12", "3.13"]
 
     steps:
       - uses: actions/checkout@v4

CHANGELOG.md

@@ -10,6 +10,60 @@ does it leads the release notes rather than being discovered from a bill.
 
 ## [Unreleased]
 
+## [0.1.2] — 2026-10-13
+
+> **Behaviour change:** `--retries 0` is now a usage error (exit 2) instead
+> of silently running no navigation at all. It counts ATTEMPTS per page
+> load, and zero never entered the loop — nothing was fetched, and the
+> engine then judged `about:blank`. A script passing `0` used to get exit 4
+> ("no jobs") for a site it had never asked.
+
+From a third-party audit dated 2026-10-13. Four findings were real; the
+audit's headline live result was an artefact of its own command, and that
+artefact is fixed too, because it is what made the misreading possible.
+
+### Fixed
+
+- **A run that fetched nothing reported "0 jobs".** State `unknown` or
+  `parse_error` left the page outcome marked ok, so the run reported
+  `completed` with zero rows and exited **4** — an ETL trusting the exit
+  code would conclude the catalogue had emptied. They now exit **5**.
+  `empty` still exits 4 (the site served a listing and it matched nothing),
+  and so does `not_found` (a 404 is the site answering definitively that the
+  address holds nothing). `page_flow.STATE_POLICY` carries `obtained` per
+  state and `stop_reason_for()` replaces a triage the three engines were
+  deriving in seven places between them.
+- **`--retries 0` ran no navigation.** See the blockquote. Measured on the
+  audited commit: `--retries 0` gave 39 bytes and exit 4; the same command
+  with `--retries 1` gave the site's real refusal — 11,922 bytes, `blocked`,
+  exit 3 — and with a residential exit, 115 rows over three pages, exit 0.
+- **`--category` did nothing in the Scraper API adapter.** It assigned
+  `row.category` to a dataclass with no such field, so `asdict()` never saw
+  it. It is a run label and goes to the sidecar; the adapter writes one now.
+- **The same adapter returned exit 4 for a served page that parsed to
+  nothing**, which is this repo's bug rather than an empty catalogue.
+- **Output writes were not atomic.** `open(path, "w")` truncates before a
+  byte is written, so an interrupted run left a shorter file where a
+  complete one was — worst for `<out>.meta.json`, the file a consumer
+  branches on. Writes go to a temp file in the same directory, fsync, then
+  `os.replace`, preserving an existing file's mode and otherwise using the
+  process umask (never a flat 0644, and never the 0600 a naive port leaves).
+- **Donor residue in the Scraper API adapter**: its description and `--mode`
+  help described a `profile` mode that does not exist in this repo.
+
+### Changed
+
+- **The `robots.txt` claim now says what is guaranteed.** "Reads only routes
+  `robots.txt` allows" read as a mechanism this code does not have. Every
+  route the parser accepts is allowed and every disallowed route has no
+  mode — now checked by the offline suite against the published file. It is
+  deliberately not a runtime gate: a preflight fetch is itself 403 from a
+  datacentre address, so it would fail closed where the scraper already
+  fails, or fail open.
+- Python **3.13** is in the CI matrix; the badge claimed 3.9–3.13 while CI
+  tested 3.9 and 3.12.
+
+
 ### Fixed
 - **The Scraper API engine failed on every `--wait-text` / `--wait-element` /
   `--wait-state` call, and was billed for it.** It sent `waitFor` as a
@@ -167,6 +221,7 @@ date is given.
   `$100k – $180k` on the listing and 100000–180000 USD in the detail page's
   `baseSalary`.
 
-[Unreleased]: https://github.com/2scraper/wellfound-scraper/compare/v0.1.1...HEAD
+[Unreleased]: https://github.com/2scraper/wellfound-scraper/compare/v0.1.2...HEAD
+[0.1.2]: https://github.com/2scraper/wellfound-scraper/compare/v0.1.1...v0.1.2
 [0.1.1]: https://github.com/2scraper/wellfound-scraper/compare/v0.1.0...v0.1.1
 [0.1.0]: https://github.com/2scraper/wellfound-scraper/releases/tag/v0.1.0

CONTRIBUTING.md

@@ -246,7 +246,11 @@ a real company.
 
 This repo scrapes **public pages** on wellfound.com: role landing pages, the
 jobs feed and individual job pages, exactly as an anonymous visitor is served
-them. It reads only routes `robots.txt` allows.
+them. Every route it accepts is allowed by Wellfound's published
+`robots.txt`, and every disallowed route has no mode —
+`check_every_route_this_scraper_reads_is_robots_allowed` pins both halves
+against the published file. It is a check, not a runtime gate; the docstring
+there says why.
 
 Out of scope: anything behind a login, anything that submits a form
 (including the apply and sign-up flows), anything that defeats a protection

README.md

@@ -378,5 +378,14 @@ Changes: [CHANGELOG.md](CHANGELOG.md)
 
 MIT. This is an independent tool; it is not affiliated with or endorsed by
 Wellfound or AngelList. Scraping is your responsibility — check the site's
-terms and `robots.txt` (this scraper reads only routes `robots.txt` allows)
-and keep your request rate civil.
+terms and `robots.txt`, and keep your request rate civil.
+
+**On `robots.txt`:** every route this scraper accepts is allowed by
+Wellfound's published rules, and every route those rules disallow (`/u/`,
+`/_jobs/`, `/search`, `/jobs/signup`, `/auth/`, …) has no mode here. That is
+a property of the code and the offline suite checks it against the published
+file, so adding a mode for a disallowed route fails the build. It is **not**
+a runtime gate: the scraper does not fetch `robots.txt` before each run and
+does not parse rules at request time. Said precisely because the previous
+wording — "reads only routes robots.txt allows" — read as a mechanism this
+code does not have.

landing.html

@@ -224,7 +224,7 @@ <h2>2Captcha products, when you want them</h2>
       <a href="https://2captcha.com">2captcha.com</a></p>
 
     <h2>What it deliberately does not do</h2>
-    <p>It reads only routes <code>robots.txt</code> allows. It never submits Wellfound&rsquo;s
+    <p>Every route it accepts is allowed by Wellfound&rsquo;s published <code>robots.txt</code>, and every disallowed route has no mode here &mdash; checked by the offline suite, though not enforced as a runtime gate. It never submits Wellfound&rsquo;s
       apply or sign-up forms &mdash; an application filed by a scraper is a false record about a
       real person and a real company. There is no mode for candidate profiles
       (<code>/u/</code>, which <code>robots.txt</code> disallows), and no column anywhere for an

landing.md

@@ -50,7 +50,7 @@ One key, four separately-billed products: [2captcha.com](https://2captcha.com)
 
 ## What it deliberately does not do
 
-It reads only routes `robots.txt` allows. It never submits Wellfound's apply or sign-up forms — an application filed by a scraper is a false record about a real person and a real company. There is no mode for candidate profiles (`/u/`, which `robots.txt` disallows), and no column anywhere for an individual's details. It defaults to a 2-second delay between pages.
+Every route it accepts is allowed by Wellfound's published `robots.txt`, and every route those rules disallow has no mode here — checked by the offline suite against the published file, though not enforced as a runtime gate. It never submits Wellfound's apply or sign-up forms — an application filed by a scraper is a false record about a real person and a real company. There is no mode for candidate profiles (`/u/`, which `robots.txt` disallows), and no column anywhere for an individual's details. It defaults to a 2-second delay between pages.
 
 ---
 

output_writer.py

@@ -59,8 +59,11 @@
 empty CSV still gets the right header for the mode that produced it.
 """
 
+import contextlib
 import csv
 import json
+import tempfile
+import os
 from dataclasses import dataclass, asdict, field, fields
 from datetime import datetime, timezone
 from typing import Optional, List, Set, Sequence, Any, Type
@@ -252,8 +255,62 @@ def _csv_value(v: Any) -> Any:
     return v
 
 
+@contextlib.contextmanager
+def _atomic(path: str, newline: Optional[str] = None):
+    """Write to a temporary file beside `path`, then rename over it.
+
+    `open(path, "w")` truncates before a single byte is written, so a crash,
+    a kill or a full disk halfway through `json.dump` leaves a SHORTER file
+    where a complete one was. That breaks the same promise `save` keeps when
+    it refuses to overwrite good output with an empty result — the previous
+    good run destroyed by the ATTEMPT to replace it rather than by its
+    outcome.
+
+    The sidecar is the part that matters most: `<out>.meta.json` is the file
+    a consumer branches on, so a truncated one beside good rows reads as a
+    broken run over data that is fine.
+
+    Three details, each of which is wrong in the obvious version:
+
+    * the temporary file goes in the SAME directory, because `os.replace` is
+      atomic only within one filesystem and a temp file under /tmp can be on
+      another, where the rename degrades to an interruptible copy;
+    * `fsync` before the rename makes the content durable rather than merely
+      visible, which is the difference on power loss;
+    * **the MODE has to be restored.** `tempfile.NamedTemporaryFile` creates
+      at 0600, so a naive port leaves every output readable only by the user
+      who ran it — a scraper whose files a downstream job cannot open. An
+      existing file's mode is preserved; a new one gets the process umask
+      applied to 0666, which is what `open()` would have produced. Never a
+      flat 0644: that would widen the file on a deliberately strict umask.
+    """
+    directory = os.path.dirname(os.path.abspath(path)) or "."
+    try:
+        mode = os.stat(path).st_mode & 0o777
+    except FileNotFoundError:
+        umask = os.umask(0)
+        os.umask(umask)
+        mode = 0o666 & ~umask
+    handle = tempfile.NamedTemporaryFile(
+        mode="w", encoding="utf-8", newline=newline, dir=directory,
+        prefix=os.path.basename(path) + ".", suffix=".tmp", delete=False)
+    try:
+        with handle:
+            yield handle
+            handle.flush()
+            os.fsync(handle.fileno())
+        os.chmod(handle.name, mode)
+        os.replace(handle.name, path)
+    except BaseException:
+        try:
+            os.unlink(handle.name)
+        except OSError:
+            pass
+        raise
+
+
 def write_json(rows: Sequence[Any], path: str) -> None:
-    with open(path, "w", encoding="utf-8") as f:
+    with _atomic(path) as f:
         json.dump([asdict(r) for r in rows], f, ensure_ascii=False, indent=2)
 
 
@@ -266,7 +323,7 @@ def write_csv(rows: Sequence[Any], path: str, row_cls: Type = JobPosting) -> Non
     # The header comes from `row_cls`, not from the first row, so an empty
     # run still writes the columns of the mode that produced it.
     fieldnames = [f.name for f in fields(row_cls)]
-    with open(path, "w", encoding="utf-8", newline="") as f:
+    with _atomic(path, newline="") as f:
         writer = csv.DictWriter(f, fieldnames=fieldnames)
         writer.writeheader()
         for r in rows:
@@ -367,7 +424,10 @@ def write_run_meta(out_prefix: str, meta: dict) -> str:
     both complete, and between runs of different `mode`.
     """
     path = f"{out_prefix}.meta.json"
-    with open(path, "w", encoding="utf-8") as f:
+    # Atomic for the same reason as the rows, and more so: this is the file
+    # a consumer branches on, so a truncated one beside good data reads as a
+    # broken run over rows that are fine.
+    with _atomic(path) as f:
         json.dump(meta, f, ensure_ascii=False, indent=2)
     print(f"[+] Wrote run metadata -> {path} (status={meta.get('status')})")
     return path

page_flow.py

@@ -24,6 +24,7 @@
 OPERATION instead, and each engine spells it in its own dialect (§1).
 """
 
+import argparse
 import logging
 from typing import Callable, Optional
 
@@ -128,13 +129,15 @@ def classify(html: Optional[str], status: Optional[int] = None,
 
 STATE_POLICY = {
     # A rendered page with job records in it.
-    "content":     {"retry": False, "solve": False, "blocked": False, "parse": True},
+    "content":     {"retry": False, "solve": False, "blocked": False, "parse": True,
+                    "obtained": True},
     # A landing page Wellfound served that matched nothing, carrying its own
     # "no jobs found" copy. The site answered exactly what was asked and it
     # has no jobs in it — a real answer, and EXIT_NO_PRODUCTS rather than
     # EXIT_BLOCKED. Reporting it as blocked sends a user hunting for a proxy
     # problem that is not there.
-    "empty":       {"retry": False, "solve": False, "blocked": False, "parse": True},
+    "empty":       {"retry": False, "solve": False, "blocked": False, "parse": True,
+                    "obtained": True},
     # Cloudflare. On this site the two refusal skins — the branded
     # "Security Check | Wellfound" 403 and the "Just a moment..."
     # interstitial — are ONE state, because they are the same decision by the
@@ -156,24 +159,73 @@ def classify(html: Optional[str], status: Optional[int] = None,
     # can build a task for it once `TURNSTILE_INTERCEPT_JS` has captured the
     # parameters Cloudflare keeps out of the markup. `retry` is True because
     # a different exit clears it outright — measured, and much cheaper.
-    "blocked":     {"retry": True,  "solve": True,  "blocked": True,  "parse": False},
+    "blocked":     {"retry": True,  "solve": True,  "blocked": True,  "parse": False,
+                    "obtained": False},
     # A page built out of Wellfound's own assets whose payload we could not
     # read. NOT "zero jobs": a served page that links to jobs and parses to
     # nothing is OUR bug, and reporting it as an empty listing sends the
     # reader to check the URL instead of the parser (CLAUDE.md §20). Worth
     # one retry in case the document was caught mid-swap, and always worth a
     # dump.
-    "parse_error": {"retry": True,  "solve": False, "blocked": False, "parse": False},
+    "parse_error": {"retry": True,  "solve": False, "blocked": False, "parse": False,
+                    "obtained": False},
     # A 404. Retrying an address that does not exist is pure waste, and it is
     # not a block — saying so stops a user rotating proxies over a typo.
-    "not_found":   {"retry": False, "solve": False, "blocked": False, "parse": False},
+    # A 404 is OBTAINED: the site answered, definitively, that this address
+    # holds nothing. A consumer can conclude something about the catalogue
+    # from it — "that job is gone" — which is what separates it from
+    # `unknown` and `parse_error`, where nothing can be concluded. So it
+    # exits 4 like `empty` rather than 5, and it is still not parsed and not
+    # retried, because re-fetching an address that does not exist is waste.
+    "not_found":   {"retry": False, "solve": False, "blocked": False, "parse": False,
+                    "obtained": True},
     # Something that is neither the site nor a recognised refusal — an
     # upstream error page, a proxy's own response, Chromium's network-error
     # page. A wait, not a spend.
-    "unknown":     {"retry": True,  "solve": False, "blocked": False, "parse": False},
+    "unknown":     {"retry": True,  "solve": False, "blocked": False, "parse": False,
+                    "obtained": False},
 }
 
 
+def content_obtained(state: str) -> bool:
+    """Did this response give us the site's own answer?
+
+    The distinction `should_parse` does NOT make. `empty` holds no rows and
+    is still an answer — the site served a listing and it matched nothing —
+    while `unknown`, `parse_error`, `not_found` and `blocked` mean the
+    content was never obtained, so nothing can be concluded about the
+    catalogue from them.
+
+    Collapsing the two is how a run that fetched nothing reports "0 jobs"
+    (exit 4) instead of "the content never arrived" (exit 5). A third-party
+    audit on 2026-10-13 reported exactly that outcome and read it as the
+    site having emptied out; the chain was `unknown` -> an outcome still
+    marked ok -> stop_reason "completed" -> zero rows -> exit 4, with
+    `finish_run`'s own exit-5 branch never reachable because nothing ever
+    told it the run had failed (CLAUDE.md §9, §25).
+    """
+    return STATE_POLICY.get(state, STATE_POLICY["unknown"])["obtained"]
+
+
+def stop_reason_for(load_failed: bool, blocked_by: Optional[str],
+                    state: Optional[str]) -> str:
+    """The stop reason for a page that did not yield rows.
+
+    One implementation because the engines derive this in seven places
+    between them, and three copies of a triage drift silently — which is the
+    argument this module exists for (CLAUDE.md §1). `content_not_obtained_*`
+    is deliberately NOT in `COMPLETE_STOP_REASONS`, so a zero-row run
+    carrying it exits 5 rather than 4.
+    """
+    if load_failed:
+        return "page_load_timeout"
+    if blocked_by:
+        return f"blocked_{blocked_by}"
+    if state and not content_obtained(state):
+        return f"content_not_obtained_{state}"
+    return "completed"
+
+
 def should_retry(state: str) -> bool:
     return STATE_POLICY.get(state, STATE_POLICY["unknown"])["retry"]
 
@@ -232,6 +284,37 @@ def should_parse(state: str) -> bool:
 # Pagination
 # ---------------------------------------------------------------------------
 
+def attempts(value: str) -> int:
+    """argparse type for `--retries`: ATTEMPTS per page load, so at least 1.
+
+    The engines' load loop is `range(1, retries + 1)`. Zero therefore ran it
+    NEVER: `goto` was not called, nothing was fetched, and the engine then
+    classified `about:blank` — a 39-byte empty DOM with no markers and no
+    assets, which comes out as `unknown` and, because the outcome was never
+    marked failed, as a COMPLETE run holding zero jobs.
+
+    That is not a hypothetical. A third-party audit of this repo on
+    2026-10-13 ran `--retries 0`, got exactly that, and concluded the site
+    had stopped serving the scraper. The same command with `--retries 1`
+    returned the site's real refusal — 11,922 bytes, classified `blocked`,
+    exit 3 — and with a residential exit, 115 rows over three pages.
+
+    So the flag is rejected rather than clamped: a run that silently became
+    a different run is how the misreading happened, and `0` is much more
+    likely to be a misunderstanding of the name than an instruction.
+    """
+    try:
+        n = int(value)
+    except ValueError:
+        raise argparse.ArgumentTypeError(f"{value!r} is not an integer")
+    if n < 1:
+        raise argparse.ArgumentTypeError(
+            "must be at least 1: it counts ATTEMPTS per page load, and 0 "
+            "would attempt none and then judge a page that was never "
+            "fetched")
+    return n
+
+
 def pagination_is_addressable(url: str) -> bool:
     """Whether page N of this listing can be fetched without walking to it.