diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index ef13cd3..1351131 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -17,7 +17,7 @@ jobs: matrix: # 3.9 is the floor the README claims. Claiming it without testing it is # how a walrus operator or a `X | None` annotation ships and breaks it. - python-version: ["3.9", "3.12"] + python-version: ["3.9", "3.12", "3.13"] steps: - uses: actions/checkout@v4 diff --git a/CHANGELOG.md b/CHANGELOG.md index 6bec7cf..9643fe7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,60 @@ does it leads the release notes rather than being discovered from a bill. ## [Unreleased] +## [0.1.2] — 2026-10-13 + +> **Behaviour change:** `--retries 0` is now a usage error (exit 2) instead +> of silently running no navigation at all. It counts ATTEMPTS per page +> load, and zero never entered the loop — nothing was fetched, and the +> engine then judged `about:blank`. A script passing `0` used to get exit 4 +> ("no jobs") for a site it had never asked. + +From a third-party audit dated 2026-10-13. Four findings were real; the +audit's headline live result was an artefact of its own command, and that +artefact is fixed too, because it is what made the misreading possible. + +### Fixed + +- **A run that fetched nothing reported "0 jobs".** State `unknown` or + `parse_error` left the page outcome marked ok, so the run reported + `completed` with zero rows and exited **4** — an ETL trusting the exit + code would conclude the catalogue had emptied. They now exit **5**. + `empty` still exits 4 (the site served a listing and it matched nothing), + and so does `not_found` (a 404 is the site answering definitively that the + address holds nothing). `page_flow.STATE_POLICY` carries `obtained` per + state and `stop_reason_for()` replaces a triage the three engines were + deriving in seven places between them. +- **`--retries 0` ran no navigation.** See the blockquote. Measured on the + audited commit: `--retries 0` gave 39 bytes and exit 4; the same command + with `--retries 1` gave the site's real refusal — 11,922 bytes, `blocked`, + exit 3 — and with a residential exit, 115 rows over three pages, exit 0. +- **`--category` did nothing in the Scraper API adapter.** It assigned + `row.category` to a dataclass with no such field, so `asdict()` never saw + it. It is a run label and goes to the sidecar; the adapter writes one now. +- **The same adapter returned exit 4 for a served page that parsed to + nothing**, which is this repo's bug rather than an empty catalogue. +- **Output writes were not atomic.** `open(path, "w")` truncates before a + byte is written, so an interrupted run left a shorter file where a + complete one was — worst for `.meta.json`, the file a consumer + branches on. Writes go to a temp file in the same directory, fsync, then + `os.replace`, preserving an existing file's mode and otherwise using the + process umask (never a flat 0644, and never the 0600 a naive port leaves). +- **Donor residue in the Scraper API adapter**: its description and `--mode` + help described a `profile` mode that does not exist in this repo. + +### Changed + +- **The `robots.txt` claim now says what is guaranteed.** "Reads only routes + `robots.txt` allows" read as a mechanism this code does not have. Every + route the parser accepts is allowed and every disallowed route has no + mode — now checked by the offline suite against the published file. It is + deliberately not a runtime gate: a preflight fetch is itself 403 from a + datacentre address, so it would fail closed where the scraper already + fails, or fail open. +- Python **3.13** is in the CI matrix; the badge claimed 3.9–3.13 while CI + tested 3.9 and 3.12. + + ### Fixed - **The Scraper API engine failed on every `--wait-text` / `--wait-element` / `--wait-state` call, and was billed for it.** It sent `waitFor` as a @@ -167,6 +221,7 @@ date is given. `$100k – $180k` on the listing and 100000–180000 USD in the detail page's `baseSalary`. -[Unreleased]: https://github.com/2scraper/wellfound-scraper/compare/v0.1.1...HEAD +[Unreleased]: https://github.com/2scraper/wellfound-scraper/compare/v0.1.2...HEAD +[0.1.2]: https://github.com/2scraper/wellfound-scraper/compare/v0.1.1...v0.1.2 [0.1.1]: https://github.com/2scraper/wellfound-scraper/compare/v0.1.0...v0.1.1 [0.1.0]: https://github.com/2scraper/wellfound-scraper/releases/tag/v0.1.0 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 582843f..54cafda 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -246,7 +246,11 @@ a real company. This repo scrapes **public pages** on wellfound.com: role landing pages, the jobs feed and individual job pages, exactly as an anonymous visitor is served -them. It reads only routes `robots.txt` allows. +them. Every route it accepts is allowed by Wellfound's published +`robots.txt`, and every disallowed route has no mode — +`check_every_route_this_scraper_reads_is_robots_allowed` pins both halves +against the published file. It is a check, not a runtime gate; the docstring +there says why. Out of scope: anything behind a login, anything that submits a form (including the apply and sign-up flows), anything that defeats a protection diff --git a/README.md b/README.md index 6914d06..6a3bb2f 100644 --- a/README.md +++ b/README.md @@ -378,5 +378,14 @@ Changes: [CHANGELOG.md](CHANGELOG.md) MIT. This is an independent tool; it is not affiliated with or endorsed by Wellfound or AngelList. Scraping is your responsibility — check the site's -terms and `robots.txt` (this scraper reads only routes `robots.txt` allows) -and keep your request rate civil. +terms and `robots.txt`, and keep your request rate civil. + +**On `robots.txt`:** every route this scraper accepts is allowed by +Wellfound's published rules, and every route those rules disallow (`/u/`, +`/_jobs/`, `/search`, `/jobs/signup`, `/auth/`, …) has no mode here. That is +a property of the code and the offline suite checks it against the published +file, so adding a mode for a disallowed route fails the build. It is **not** +a runtime gate: the scraper does not fetch `robots.txt` before each run and +does not parse rules at request time. Said precisely because the previous +wording — "reads only routes robots.txt allows" — read as a mechanism this +code does not have. diff --git a/landing.html b/landing.html index 61b2328..5689769 100644 --- a/landing.html +++ b/landing.html @@ -224,7 +224,7 @@

2Captcha products, when you want them

2captcha.com

What it deliberately does not do

-

It reads only routes robots.txt allows. It never submits Wellfound’s +

Every route it accepts is allowed by Wellfound’s published robots.txt, and every disallowed route has no mode here — checked by the offline suite, though not enforced as a runtime gate. It never submits Wellfound’s apply or sign-up forms — an application filed by a scraper is a false record about a real person and a real company. There is no mode for candidate profiles (/u/, which robots.txt disallows), and no column anywhere for an diff --git a/landing.md b/landing.md index ff36df9..6864b74 100644 --- a/landing.md +++ b/landing.md @@ -50,7 +50,7 @@ One key, four separately-billed products: [2captcha.com](https://2captcha.com) ## What it deliberately does not do -It reads only routes `robots.txt` allows. It never submits Wellfound's apply or sign-up forms — an application filed by a scraper is a false record about a real person and a real company. There is no mode for candidate profiles (`/u/`, which `robots.txt` disallows), and no column anywhere for an individual's details. It defaults to a 2-second delay between pages. +Every route it accepts is allowed by Wellfound's published `robots.txt`, and every route those rules disallow has no mode here — checked by the offline suite against the published file, though not enforced as a runtime gate. It never submits Wellfound's apply or sign-up forms — an application filed by a scraper is a false record about a real person and a real company. There is no mode for candidate profiles (`/u/`, which `robots.txt` disallows), and no column anywhere for an individual's details. It defaults to a 2-second delay between pages. --- diff --git a/output_writer.py b/output_writer.py index 7cff434..2eab885 100644 --- a/output_writer.py +++ b/output_writer.py @@ -59,8 +59,11 @@ empty CSV still gets the right header for the mode that produced it. """ +import contextlib import csv import json +import tempfile +import os from dataclasses import dataclass, asdict, field, fields from datetime import datetime, timezone from typing import Optional, List, Set, Sequence, Any, Type @@ -252,8 +255,62 @@ def _csv_value(v: Any) -> Any: return v +@contextlib.contextmanager +def _atomic(path: str, newline: Optional[str] = None): + """Write to a temporary file beside `path`, then rename over it. + + `open(path, "w")` truncates before a single byte is written, so a crash, + a kill or a full disk halfway through `json.dump` leaves a SHORTER file + where a complete one was. That breaks the same promise `save` keeps when + it refuses to overwrite good output with an empty result — the previous + good run destroyed by the ATTEMPT to replace it rather than by its + outcome. + + The sidecar is the part that matters most: `.meta.json` is the file + a consumer branches on, so a truncated one beside good rows reads as a + broken run over data that is fine. + + Three details, each of which is wrong in the obvious version: + + * the temporary file goes in the SAME directory, because `os.replace` is + atomic only within one filesystem and a temp file under /tmp can be on + another, where the rename degrades to an interruptible copy; + * `fsync` before the rename makes the content durable rather than merely + visible, which is the difference on power loss; + * **the MODE has to be restored.** `tempfile.NamedTemporaryFile` creates + at 0600, so a naive port leaves every output readable only by the user + who ran it — a scraper whose files a downstream job cannot open. An + existing file's mode is preserved; a new one gets the process umask + applied to 0666, which is what `open()` would have produced. Never a + flat 0644: that would widen the file on a deliberately strict umask. + """ + directory = os.path.dirname(os.path.abspath(path)) or "." + try: + mode = os.stat(path).st_mode & 0o777 + except FileNotFoundError: + umask = os.umask(0) + os.umask(umask) + mode = 0o666 & ~umask + handle = tempfile.NamedTemporaryFile( + mode="w", encoding="utf-8", newline=newline, dir=directory, + prefix=os.path.basename(path) + ".", suffix=".tmp", delete=False) + try: + with handle: + yield handle + handle.flush() + os.fsync(handle.fileno()) + os.chmod(handle.name, mode) + os.replace(handle.name, path) + except BaseException: + try: + os.unlink(handle.name) + except OSError: + pass + raise + + def write_json(rows: Sequence[Any], path: str) -> None: - with open(path, "w", encoding="utf-8") as f: + with _atomic(path) as f: json.dump([asdict(r) for r in rows], f, ensure_ascii=False, indent=2) @@ -266,7 +323,7 @@ def write_csv(rows: Sequence[Any], path: str, row_cls: Type = JobPosting) -> Non # The header comes from `row_cls`, not from the first row, so an empty # run still writes the columns of the mode that produced it. fieldnames = [f.name for f in fields(row_cls)] - with open(path, "w", encoding="utf-8", newline="") as f: + with _atomic(path, newline="") as f: writer = csv.DictWriter(f, fieldnames=fieldnames) writer.writeheader() for r in rows: @@ -367,7 +424,10 @@ def write_run_meta(out_prefix: str, meta: dict) -> str: both complete, and between runs of different `mode`. """ path = f"{out_prefix}.meta.json" - with open(path, "w", encoding="utf-8") as f: + # Atomic for the same reason as the rows, and more so: this is the file + # a consumer branches on, so a truncated one beside good data reads as a + # broken run over rows that are fine. + with _atomic(path) as f: json.dump(meta, f, ensure_ascii=False, indent=2) print(f"[+] Wrote run metadata -> {path} (status={meta.get('status')})") return path diff --git a/page_flow.py b/page_flow.py index 14010d1..950aad9 100644 --- a/page_flow.py +++ b/page_flow.py @@ -24,6 +24,7 @@ OPERATION instead, and each engine spells it in its own dialect (§1). """ +import argparse import logging from typing import Callable, Optional @@ -128,13 +129,15 @@ def classify(html: Optional[str], status: Optional[int] = None, STATE_POLICY = { # A rendered page with job records in it. - "content": {"retry": False, "solve": False, "blocked": False, "parse": True}, + "content": {"retry": False, "solve": False, "blocked": False, "parse": True, + "obtained": True}, # A landing page Wellfound served that matched nothing, carrying its own # "no jobs found" copy. The site answered exactly what was asked and it # has no jobs in it — a real answer, and EXIT_NO_PRODUCTS rather than # EXIT_BLOCKED. Reporting it as blocked sends a user hunting for a proxy # problem that is not there. - "empty": {"retry": False, "solve": False, "blocked": False, "parse": True}, + "empty": {"retry": False, "solve": False, "blocked": False, "parse": True, + "obtained": True}, # Cloudflare. On this site the two refusal skins — the branded # "Security Check | Wellfound" 403 and the "Just a moment..." # interstitial — are ONE state, because they are the same decision by the @@ -156,24 +159,73 @@ def classify(html: Optional[str], status: Optional[int] = None, # can build a task for it once `TURNSTILE_INTERCEPT_JS` has captured the # parameters Cloudflare keeps out of the markup. `retry` is True because # a different exit clears it outright — measured, and much cheaper. - "blocked": {"retry": True, "solve": True, "blocked": True, "parse": False}, + "blocked": {"retry": True, "solve": True, "blocked": True, "parse": False, + "obtained": False}, # A page built out of Wellfound's own assets whose payload we could not # read. NOT "zero jobs": a served page that links to jobs and parses to # nothing is OUR bug, and reporting it as an empty listing sends the # reader to check the URL instead of the parser (CLAUDE.md §20). Worth # one retry in case the document was caught mid-swap, and always worth a # dump. - "parse_error": {"retry": True, "solve": False, "blocked": False, "parse": False}, + "parse_error": {"retry": True, "solve": False, "blocked": False, "parse": False, + "obtained": False}, # A 404. Retrying an address that does not exist is pure waste, and it is # not a block — saying so stops a user rotating proxies over a typo. - "not_found": {"retry": False, "solve": False, "blocked": False, "parse": False}, + # A 404 is OBTAINED: the site answered, definitively, that this address + # holds nothing. A consumer can conclude something about the catalogue + # from it — "that job is gone" — which is what separates it from + # `unknown` and `parse_error`, where nothing can be concluded. So it + # exits 4 like `empty` rather than 5, and it is still not parsed and not + # retried, because re-fetching an address that does not exist is waste. + "not_found": {"retry": False, "solve": False, "blocked": False, "parse": False, + "obtained": True}, # Something that is neither the site nor a recognised refusal — an # upstream error page, a proxy's own response, Chromium's network-error # page. A wait, not a spend. - "unknown": {"retry": True, "solve": False, "blocked": False, "parse": False}, + "unknown": {"retry": True, "solve": False, "blocked": False, "parse": False, + "obtained": False}, } +def content_obtained(state: str) -> bool: + """Did this response give us the site's own answer? + + The distinction `should_parse` does NOT make. `empty` holds no rows and + is still an answer — the site served a listing and it matched nothing — + while `unknown`, `parse_error`, `not_found` and `blocked` mean the + content was never obtained, so nothing can be concluded about the + catalogue from them. + + Collapsing the two is how a run that fetched nothing reports "0 jobs" + (exit 4) instead of "the content never arrived" (exit 5). A third-party + audit on 2026-10-13 reported exactly that outcome and read it as the + site having emptied out; the chain was `unknown` -> an outcome still + marked ok -> stop_reason "completed" -> zero rows -> exit 4, with + `finish_run`'s own exit-5 branch never reachable because nothing ever + told it the run had failed (CLAUDE.md §9, §25). + """ + return STATE_POLICY.get(state, STATE_POLICY["unknown"])["obtained"] + + +def stop_reason_for(load_failed: bool, blocked_by: Optional[str], + state: Optional[str]) -> str: + """The stop reason for a page that did not yield rows. + + One implementation because the engines derive this in seven places + between them, and three copies of a triage drift silently — which is the + argument this module exists for (CLAUDE.md §1). `content_not_obtained_*` + is deliberately NOT in `COMPLETE_STOP_REASONS`, so a zero-row run + carrying it exits 5 rather than 4. + """ + if load_failed: + return "page_load_timeout" + if blocked_by: + return f"blocked_{blocked_by}" + if state and not content_obtained(state): + return f"content_not_obtained_{state}" + return "completed" + + def should_retry(state: str) -> bool: return STATE_POLICY.get(state, STATE_POLICY["unknown"])["retry"] @@ -232,6 +284,37 @@ def should_parse(state: str) -> bool: # Pagination # --------------------------------------------------------------------------- +def attempts(value: str) -> int: + """argparse type for `--retries`: ATTEMPTS per page load, so at least 1. + + The engines' load loop is `range(1, retries + 1)`. Zero therefore ran it + NEVER: `goto` was not called, nothing was fetched, and the engine then + classified `about:blank` — a 39-byte empty DOM with no markers and no + assets, which comes out as `unknown` and, because the outcome was never + marked failed, as a COMPLETE run holding zero jobs. + + That is not a hypothetical. A third-party audit of this repo on + 2026-10-13 ran `--retries 0`, got exactly that, and concluded the site + had stopped serving the scraper. The same command with `--retries 1` + returned the site's real refusal — 11,922 bytes, classified `blocked`, + exit 3 — and with a residential exit, 115 rows over three pages. + + So the flag is rejected rather than clamped: a run that silently became + a different run is how the misreading happened, and `0` is much more + likely to be a misunderstanding of the name than an instruction. + """ + try: + n = int(value) + except ValueError: + raise argparse.ArgumentTypeError(f"{value!r} is not an integer") + if n < 1: + raise argparse.ArgumentTypeError( + "must be at least 1: it counts ATTEMPTS per page load, and 0 " + "would attempt none and then judge a page that was never " + "fetched") + return n + + def pagination_is_addressable(url: str) -> bool: """Whether page N of this listing can be fetched without walking to it. diff --git a/playwright_scraper.py b/playwright_scraper.py index 3243cb0..4b300b3 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -144,7 +144,18 @@ class PageOutcome: @property def ok(self) -> bool: - return not self.load_failed and self.blocked_by is None + """Did this page give us the site's own answer? + + `empty` counts as ok — the site served a listing and it matched + nothing, which is an answer. `unknown`, `parse_error` and + `not_found` do NOT: the content was never obtained, and an outcome + that calls them ok is what turns a failed fetch into a complete run + holding zero jobs (exit 4 instead of 5). The policy lives in + page_flow so the three engines cannot disagree about it. + """ + return (not self.load_failed + and self.blocked_by is None + and (self.state is None or page_flow.content_obtained(self.state))) ITEM_LINK_SELECTOR = page_flow.READY_SELECTOR_LISTING @@ -1098,8 +1109,22 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome # Reached only for a state the policy says holds no rows — and it # says so in ONE place, so an engine cannot quietly decide to parse # something its twins would not. - logger.info("Page %d came back as %s; nothing to parse.", page_num, - state) + # Dump it. A state that holds no rows is a state somebody will have + # to diagnose, and the only way to tell a parsing bug from a page + # that never arrived is the exact bytes (CLAUDE.md §9). Until the + # 2026-10-13 audit this path wrote nothing at all: the auditor's run + # failed and left no artefact, which is half of why the cause was + # misread. The blocked path has always dumped; this one did not. + debug_html = f"{args.out}_page{page_num}_debug.html" + try: + with open(debug_html, "w", encoding="utf-8") as f: + f.write(html or "") + except OSError as e: + debug_html = f"(could not be written: {e})" + logger.info("Page %d came back as %s (%d bytes, own assets " + "referenced %d time(s)) — nothing to parse, saved to %s.", + page_num, state, len(html or ""), + references_own_assets(html or ""), debug_html) outcome.final_url = session.page.url return outcome @@ -1364,8 +1389,8 @@ def scrape(args) -> int: outcomes.append(first) if not first.ok: - stop_reason = ("page_load_timeout" if first.load_failed - else f"blocked_{first.blocked_by}") + stop_reason = page_flow.stop_reason_for( + first.load_failed, first.blocked_by, first.state) blocked = first.blocked_by is not None elif args.mode == "job": pass # one page is the whole run @@ -1401,8 +1426,8 @@ def scrape(args) -> int: failed = [o for o in rest if not o.ok] if failed: worst = min(failed, key=lambda o: o.page_num) - stop_reason = ("page_load_timeout" if worst.load_failed - else f"blocked_{worst.blocked_by}") + stop_reason = page_flow.stop_reason_for( + worst.load_failed, worst.blocked_by, worst.state) blocked = any(o.blocked_by for o in rest) elif exhausted: stop_reason = "no_new_products" @@ -1424,8 +1449,8 @@ def scrape(args) -> int: outcome = _fetch_one_page(session, args, pool, page_num, url) outcomes.append(outcome) if not outcome.ok: - stop_reason = ("page_load_timeout" if outcome.load_failed - else f"blocked_{outcome.blocked_by}") + stop_reason = page_flow.stop_reason_for( + outcome.load_failed, outcome.blocked_by, outcome.state) blocked = outcome.blocked_by is not None break @@ -1612,7 +1637,7 @@ def parse_args(): "own browser and holds its own proxy exit, so N>1 " "without --proxy-file just sends N times the traffic " "from one address. Ignored with --cdp-endpoint.") - p.add_argument("--retries", type=int, default=3, + p.add_argument("--retries", type=page_flow.attempts, default=3, help="Attempts per page load before giving up (default 3). " "The pause between attempts doubles each time. A page " "that comes back EMPTY is not retried — see " diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index f42c62b..e8e3a01 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -253,7 +253,18 @@ class PageOutcome: @property def ok(self) -> bool: - return not self.load_failed and self.blocked_by is None + """Did this page give us the site's own answer? + + `empty` counts as ok — the site served a listing and it matched + nothing, which is an answer. `unknown`, `parse_error` and + `not_found` do NOT: the content was never obtained, and an outcome + that calls them ok is what turns a failed fetch into a complete run + holding zero jobs (exit 4 instead of 5). The policy lives in + page_flow so the three engines cannot disagree about it. + """ + return (not self.load_failed + and self.blocked_by is None + and (self.state is None or page_flow.content_obtained(self.state))) # Every `scheme://user:pass@` in a string, however many times it occurs. @@ -872,8 +883,22 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome return outcome if not page_flow.should_parse(state): - logger.info("Page %d came back as %s; nothing to parse.", page_num, - state) + # Dump it. A state that holds no rows is a state somebody will have + # to diagnose, and the only way to tell a parsing bug from a page + # that never arrived is the exact bytes (CLAUDE.md §9). Until the + # 2026-10-13 audit this path wrote nothing at all: the auditor's run + # failed and left no artefact, which is half of why the cause was + # misread. The blocked path has always dumped; this one did not. + debug_html = f"{args.out}_page{page_num}_debug.html" + try: + with open(debug_html, "w", encoding="utf-8") as f: + f.write(html or "") + except OSError as e: + debug_html = f"(could not be written: {e})" + logger.info("Page %d came back as %s (%d bytes, own assets " + "referenced %d time(s)) — nothing to parse, saved to %s.", + page_num, state, len(html or ""), + references_own_assets(html or ""), debug_html) outcome.final_url = page.url return outcome @@ -995,8 +1020,8 @@ def scrape(args) -> int: outcomes.append(first) if not first.ok: - stop_reason = ("page_load_timeout" if first.load_failed - else f"blocked_{first.blocked_by}") + stop_reason = page_flow.stop_reason_for( + first.load_failed, first.blocked_by, first.state) blocked = first.blocked_by is not None elif args.mode == "job": pass # one page is the whole run @@ -1020,8 +1045,8 @@ def scrape(args) -> int: outcome = _fetch_one_page(session, args, pool, page_num, url) outcomes.append(outcome) if not outcome.ok: - stop_reason = ("page_load_timeout" if outcome.load_failed - else f"blocked_{outcome.blocked_by}") + stop_reason = page_flow.stop_reason_for( + outcome.load_failed, outcome.blocked_by, outcome.state) blocked = outcome.blocked_by is not None break @@ -1150,7 +1175,7 @@ def parse_args(): p.add_argument("--concurrency", type=int, default=1, metavar="N", help="Accepted for flag parity and IGNORED here: parallel " "page fetching lives in playwright_scraper.py.") - p.add_argument("--retries", type=int, default=3, + p.add_argument("--retries", type=page_flow.attempts, default=3, help="Attempts per page load before giving up (default 3). " "A page that comes back EMPTY is not retried: an empty " "hub category is a correct answer, not a fault.") diff --git a/pyproject.toml b/pyproject.toml index ea7dc46..8842cb4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "wellfound-scraper" -version = "0.1.1" +version = "0.1.2" description = "Wellfound (AngelList Talent) job scraper: role landing pages, the jobs feed and individual job pages." readme = "README.md" license = "MIT" diff --git a/scraper_api_client.py b/scraper_api_client.py index fb0a30e..bfd3519 100644 --- a/scraper_api_client.py +++ b/scraper_api_client.py @@ -43,9 +43,10 @@ import requests from product_parser import (BOT_CHALLENGE_MARKERS, MODES, DEFAULT_MODE, + category_from_url, detect_bot_challenge, detect_page_state, parse_products, parse_detail) -from output_writer import save +from output_writer import finish_run, SOURCE_DEFAULT import env_config logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s") @@ -282,26 +283,44 @@ def _run_once(args, attempt: int = 1, attempts: int = 1) -> int: products = [row] if row is not None else [] else: products = parse_products(html, args.url, mode=args.mode) - if args.category: - for row in products: - row.category = args.category logger.info("Parsed %d job(s).", len(products)) + # `--category` is a RUN LABEL, not a column. It used to be assigned to + # each row as `row.category`, and `JobPosting` has no such field — so + # Python attached a dynamic attribute that `asdict()` never sees, and + # the flag silently did nothing. It goes in the sidecar, which is where + # the browser engines put it too. + label = args.category or category_from_url(args.url) + + # A page the API SERVED that parses to nothing is this repo's bug, not + # an empty catalogue (CLAUDE.md §20). It used to return 4 — the same + # conflation the engines carried until this audit — so it now reports a + # non-complete stop reason, which `finish_run` maps to exit 5 and never + # to "no jobs". if not products: dump = f"{args.out}_scraperapi_debug.html" with open(dump, "w", encoding="utf-8") as f: f.write(html) - logger.warning("0 jobs parsed — saved the raw response to %s so " - "you can see what actually came back.", dump) - return 4 + logger.warning("0 jobs parsed from a page the API served (%d bytes) " + "— saved to %s. That is a parser failure, not an " + "empty listing.", len(html), dump) + stop = "content_not_obtained_parse_error" + else: + stop = "completed" - return save(products, args.out, args.format, allow_empty=args.allow_empty) + return finish_run(products, args.out, args.format, args.allow_empty, + blocked=False, stop_reason=stop, + pages_requested=1, pages_completed=1 if products else 0, + pages_failed=[] if products else [1], + mode=args.mode, source=SOURCE_DEFAULT, + start_url=args.url, final_url=args.url, + extra={"category": label, "via": "scraper_api"}) def parse_args(): p = argparse.ArgumentParser( description="Wellfound scraper — 2captcha Scraper API edition (no local " - "browser). Built for --mode profile, which is one " + "browser). What decides success is the EXIT: its own " "server-rendered page and a good fit. NOT the way to " "browser needed). What decides success is the EXIT: " "ordinary HTTPS request with no key and no proxy, so " @@ -321,10 +340,13 @@ def parse_args(): "WELLFOUND_URL is set in the environment or .env.") p.add_argument("--mode", choices=list(MODES), default=DEFAULT_MODE, - help="Default profile, unlike the browser engines, " - "because a profile is the only page kind this path " - "reads that the free endpoint cannot.") - p.add_argument("--category", default=None, help="Label to tag output rows with. Defaults to the category segment of the URL, so the column is never empty just because the flag was omitted.") + help="Which page kind to read (default %(default)s). " + "The same three the browser engines take; this path " + "renders the page remotely instead of locally.") + p.add_argument("--category", default=None, + help="Label for the RUN, recorded in the sidecar — not a " + "column on the rows. Defaults to what the URL " + "selects.") p.add_argument("--format", choices=["json", "csv", "both"], default="both") p.add_argument("--out", default="wellfound_jobs_scraperapi", help="Output file prefix") p.add_argument("--timeout", type=int, default=60, diff --git a/selenium_scraper.py b/selenium_scraper.py index 41fc92c..e603485 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -155,7 +155,18 @@ class PageOutcome: @property def ok(self) -> bool: - return not self.load_failed and self.blocked_by is None + """Did this page give us the site's own answer? + + `empty` counts as ok — the site served a listing and it matched + nothing, which is an answer. `unknown`, `parse_error` and + `not_found` do NOT: the content was never obtained, and an outcome + that calls them ok is what turns a failed fetch into a complete run + holding zero jobs (exit 4 instead of 5). The policy lives in + page_flow so the three engines cannot disagree about it. + """ + return (not self.load_failed + and self.blocked_by is None + and (self.state is None or page_flow.content_obtained(self.state))) # Every `scheme://user:pass@` in a string, however many times it occurs. @@ -769,8 +780,22 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome final_url = d["current_url"]() or url if not page_flow.should_parse(state): - logger.info("Page %d came back as %s; nothing to parse.", page_num, - state) + # Dump it. A state that holds no rows is a state somebody will have + # to diagnose, and the only way to tell a parsing bug from a page + # that never arrived is the exact bytes (CLAUDE.md §9). Until the + # 2026-10-13 audit this path wrote nothing at all: the auditor's run + # failed and left no artefact, which is half of why the cause was + # misread. The blocked path has always dumped; this one did not. + debug_html = f"{args.out}_page{page_num}_debug.html" + try: + with open(debug_html, "w", encoding="utf-8") as f: + f.write(html or "") + except OSError as e: + debug_html = f"(could not be written: {e})" + logger.info("Page %d came back as %s (%d bytes, own assets " + "referenced %d time(s)) — nothing to parse, saved to %s.", + page_num, state, len(html or ""), + references_own_assets(html or ""), debug_html) outcome.final_url = final_url return outcome @@ -881,8 +906,8 @@ def scrape(args) -> int: outcomes.append(first) if not first.ok: - stop_reason = ("page_load_timeout" if first.load_failed - else f"blocked_{first.blocked_by}") + stop_reason = page_flow.stop_reason_for( + first.load_failed, first.blocked_by, first.state) blocked = first.blocked_by is not None elif args.mode == "job": pass # one page is the whole run @@ -906,8 +931,8 @@ def scrape(args) -> int: outcome = _fetch_one_page(session, args, pool, page_num, url) outcomes.append(outcome) if not outcome.ok: - stop_reason = ("page_load_timeout" if outcome.load_failed - else f"blocked_{outcome.blocked_by}") + stop_reason = page_flow.stop_reason_for( + outcome.load_failed, outcome.blocked_by, outcome.state) blocked = outcome.blocked_by is not None break @@ -1038,7 +1063,7 @@ def parse_args(): p.add_argument("--concurrency", type=int, default=1, metavar="N", help="Accepted for flag parity and IGNORED here: parallel " "page fetching lives in playwright_scraper.py.") - p.add_argument("--retries", type=int, default=3, + p.add_argument("--retries", type=page_flow.attempts, default=3, help="Attempts per page load before giving up (default 3). " "A page that comes back EMPTY is not retried: an empty " "hub category is a correct answer, not a fault.") diff --git a/smoke_test.py b/smoke_test.py index 785cd36..69ad291 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -442,6 +442,102 @@ def check_route_and_mode_are_derived_from_the_url(): equal("mode from %s" % url, P.mode_for_url(url), mode) +# Wellfound's own robots.txt, fetched 2026-10-13 through a residential exit +# (it answers 403 to a datacentre address, like every other route here). +# Inline rather than in a fixtures directory, per CLAUDE.md §10. +ROBOTS_TXT = """# +# Hey human, +# +# We like robots too. Come talk robot with us. +# +# https://wellfound.com/company/wellfound/jobs +# + +User-agent: * +Disallow: /_jobs/ +Disallow: /*?after_sign_in=* +Disallow: /*?inFrame=* +Disallow: /*?jobId=* +Disallow: /*?jobSlug=* +Disallow: /*?preview=* +Disallow: /*?role=* +Disallow: /*&inFrame=* +Disallow: /*&jobId=* +Disallow: /*&jobSlug=* +Disallow: /*&preview=* +Disallow: /auth/ +Disallow: /cdn-cgi/ +Disallow: /documents/ +Disallow: /embed/ +Disallow: /job_listings/report_company +Disallow: /job_pairings/howitworks +Disallow: /job_profiles/embed +Disallow: /jobs/applications +Disallow: /jobs/signup +Disallow: /onboarding +Disallow: /profile/edit +Disallow: /profile/notifications +Disallow: /profile/review +Disallow: /profile/resume +Disallow: /projects/ +Disallow: /re/ +Disallow: /recruit/dashboard +Disallow: /search +Disallow: /social/share_modal +Disallow: /u/ + +Sitemap: https://wellfound.com/sitemap.xml.gz +Sitemap: https://wellfound.com/blog-index.xml.gz +""" + + +def check_every_route_this_scraper_reads_is_robots_allowed(): + """The README says this scraper reads only routes `robots.txt` allows. + + That was a claim about BEHAVIOUR with no mechanism behind it — the word + appeared in comments and nowhere in the code — which a third-party audit + raised on 2026-10-13 and was right to raise. + + It is a CHECK rather than a runtime gate, and that is a decision worth + stating. A preflight fetch of robots.txt would add a request that is + itself refused from a datacentre address (403, like every route here), + so the gate would either fail closed exactly where the scraper already + fails — a second failure mode for no safety gained — or fail open, which + is a gate that does not gate. What makes the claim true is that the + routes this scraper ACCEPTS are allowed and the disallowed ones have no + mode; that is a property of the code, tested here and re-tested against + the live file by the canary. + """ + import urllib.robotparser + import product_parser as P + rp = urllib.robotparser.RobotFileParser() + rp.parse(ROBOTS_TXT.splitlines()) + + for path in ("/", "/jobs", "/role/software-engineer", + "/role/r/software-engineer", + "/role/l/data-scientist/new-york", + "/jobs/4697947-senior-software-engineer"): + check("robots allows a route this scraper reads: " + path, + rp.can_fetch("*", "https://wellfound.com" + path), path) + + # ...and every route robots disallows must have no mode at all. `/u/` is + # a candidate's own profile, which CONTRIBUTING also refuses on data + # grounds rather than only on this one. + for path in ("/u/someone", "/_jobs/anything", "/jobs/signup", + "/auth/login", "/profile/resume"): + url = "https://wellfound.com" + path + check("robots disallows " + path + ", and so does the parser", + not rp.can_fetch("*", url) and not P.is_supported_url(url)[0], + path) + + # The guard must have had something to read, or it passes for the wrong + # reason (CLAUDE.md §22): an empty robots file allows everything. + rules = [l for l in ROBOTS_TXT.splitlines() + if l.lower().startswith("disallow")] + check("...and the fixture really carries rules", len(rules) > 10, + str(len(rules)) + " Disallow line(s)") + + def check_unsupported_urls_are_refused_with_a_reason(): """A wrong reason sends the reader hunting for a typo (CLAUDE.md §5).""" import product_parser as P @@ -634,6 +730,134 @@ def check_company_size_tokens(): equal("size: absent stays absent", P.company_size(None), None) +def check_content_not_obtained_is_never_an_empty_result(): + """A page we never read must not report "0 jobs" (CLAUDE.md §9, §25). + + Found by a third-party audit on 2026-10-13. The chain was four links + long and every one of them looked right on its own: `unknown` -> + `should_parse` False -> an outcome still marked `ok` (it checked only + `load_failed` and `blocked_by`) -> `stop_reason` left at "completed" -> + zero rows -> exit 4. `finish_run` has had the exit-5 branch since the + family unified on it, and nothing ever reached it, because nothing told + it the run had failed. + + `empty` is the one that must stay exit 4: the site served a listing and + it matched nothing, which is an answer about the catalogue. The other + three are not. + """ + import page_flow + from output_writer import COMPLETE_STOP_REASONS + # `not_found` is in the OBTAINED list on purpose: a 404 is the site + # answering definitively that the address holds nothing, so a consumer + # can conclude something about the catalogue from it. `unknown` and + # `parse_error` say nothing at all. + equal("an answered page counts as obtained", + [s for s in ("content", "empty", "not_found") + if page_flow.content_obtained(s)], + ["content", "empty", "not_found"]) + for state in ("unknown", "parse_error", "blocked"): + check("%s does NOT count as obtained" % state, + not page_flow.content_obtained(state)) + reason = page_flow.stop_reason_for(False, None, state) + check("...and its stop_reason is not a COMPLETE one", + reason not in COMPLETE_STOP_REASONS, reason) + equal("an answered page keeps the plain reason", + page_flow.stop_reason_for(False, None, "empty"), "completed") + # A transport failure and a block still win over the state, because they + # say more about what happened. + equal("a load failure is still a load failure", + page_flow.stop_reason_for(True, None, "unknown"), "page_load_timeout") + equal("a block is still a block", + page_flow.stop_reason_for(False, "cloudflare", "blocked"), + "blocked_cloudflare") + + # And the engines must consult it rather than keeping their own copy. + for module in ENGINES: + engine = _import_engine(module) + if engine is None: + skip("outcome-ok", "%s not importable here" % module) + continue + for state, want_ok in (("content", True), ("empty", True), + ("not_found", True), + ("unknown", False), ("parse_error", False)): + o = engine.PageOutcome(page_num=1, url="u") + o.state = state + equal("%s.PageOutcome.ok for state=%s" % (module, state), + o.ok, want_ok) + + +def check_a_page_that_yielded_nothing_is_dumped(): + """A state that holds no rows is one somebody will have to diagnose. + + The blocked path has always written a debug dump. The + content-not-obtained path wrote NOTHING — so a failing run left no + artefact, and the 2026-10-13 audit's own words were "тело ответа не было + сохранено для последующего анализа". It was right, and that is half of + why the cause was misread: there was nothing to look at. + + Asserted on the source rather than by driving a browser, because the + branch needs a live navigation and the suite must pass with no engine + library installed. + """ + for module in ENGINES: + path = os.path.join(HERE, module + ".py") + if not os.path.exists(path): + continue + source = open(path, encoding="utf-8").read() + marker = "nothing to parse" + check("%s has the content-not-obtained branch" % module, + marker in source) + if marker not in source: + continue + branch = source[source.index(marker) - 1200:source.index(marker) + 400] + check("%s dumps the body on that branch" % module, + "_debug.html" in branch, + "the branch writes no artefact to diagnose from") + check("%s reports the size and the asset count with it" % module, + "references_own_assets" in branch, + "a dump with no numbers beside it is harder to read") + + +def check_retries_counts_attempts_and_refuses_zero(): + """`--retries 0` ran no navigation at all and judged `about:blank`. + + The load loop is `range(1, retries + 1)`, so zero never entered it: + `goto` was not called, and the engine then classified a 39-byte empty + DOM. That is how the 2026-10-13 audit concluded the site had stopped + serving this scraper — its own command had fetched nothing. The same + command with `--retries 1` returned the site's real refusal (11,922 + bytes, `blocked`, exit 3). + + Rejected rather than clamped: a run that silently becomes a different + run is exactly how the misreading happened. + """ + import argparse as _ap + import page_flow + equal("--retries 1 is accepted", page_flow.attempts("1"), 1) + equal("--retries 3 is accepted", page_flow.attempts("3"), 3) + for bad in ("0", "-1"): + try: + page_flow.attempts(bad) + check("--retries %s is refused" % bad, False, "it was accepted") + except _ap.ArgumentTypeError as e: + check("--retries %s is refused with a reason" % bad, + "at least 1" in str(e), str(e)) + try: + page_flow.attempts("two") + check("a non-integer is refused", False, "it was accepted") + except _ap.ArgumentTypeError: + check("a non-integer is refused", True) + # Every engine must route the flag through it, or the guard is decorative. + for module in ENGINES: + path = os.path.join(HERE, module + ".py") + if not os.path.exists(path): + continue + source = open(path, encoding="utf-8").read() + check("%s parses --retries through page_flow.attempts" % module, + '"--retries", type=page_flow.attempts' in source, + "it still takes a bare int") + + def check_state_policy(): import page_flow equal("every state has a policy",