diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index d39db59..f0c5453 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -10,26 +10,22 @@ body: mask credentials in their own logs, but raw HTML dumps and the Scraper API's `x-debug` header do not. Replace them with `***`. - Two things answer most reports faster than we can: + A few things answer most reports faster than we can: - - **Everything exit 3?** Check the exit address first. Etsy's - DataDome answers every datacentre and VPN IP with HTTP 403, so a run - from a VPS or a CI runner is blocked whatever else is set. If the - log says `t=bv`, a captcha key does not help — that challenge is - not solvable. The Scraping Browser API (`--cdp-endpoint`) is the - path that works; if a profile keeps refusing after three retries, - try a different `pid`. - - **Zero products?** You may be on a page that has none — a taxonomy - hub, or one page past the end of a listing. It answers 200 with - nothing on it, which is exit 4 and a correct answer. See - [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md). - - **`rating` empty on every row?** That is expected on a listing run: - Etsy publishes no per-listing rating there. The stars you see on a - tile are the shop's and are in `shop_rating`. See the README. - - **Wrong currency or language?** Etsy decides both from your exit - IP, not from the URL — a request for `/fr/…` from a German exit is - redirected to `/de/…`. The row's `url` says which storefront - actually answered. + - **Everything exit 3?** Check the exit address first. Wellfound + answers every route with HTTP 403 from a datacentre address, + `robots.txt` included, and no browser gets past it. A residential + exit (`WELLFOUND_PROXY` in `.env`) is what works — see "What is + gated and what is not" in the README. + - **`/company/{slug}` refused?** It is gated even from a residential + exit. There is no `--mode company` for that reason. + - **Zero jobs?** A listing can genuinely be empty, which is exit 4 + and a correct answer. Check the URL is a `/role/...`, `/jobs` or + `/jobs/{id}-{slug}` page. + - **A column empty or "wrong"?** Read "Traps that look like bugs" in + the README first: `salary_period` is null on landing rows, + `has_equity: false` differs from `null`, and a page is twenty + companies, not twenty jobs. - type: dropdown id: engine @@ -63,8 +59,8 @@ body: label: Exit code description: >- `echo $?` straight after the run. These mean specific things — 3 is a - challenge page, 4 is "ran fine, found nothing", 124 is a self-imposed - timeout. The table is in the README. + refusal or challenge page, 4 is "ran fine, found nothing", 124 is a + self-imposed timeout. The table is in the README. placeholder: "4" validations: required: true @@ -77,8 +73,8 @@ body: render: shell placeholder: | python3 playwright_scraper.py \ - --url "https://www.etsy.com/search?q=handmade+mug" \ - --pages 3 --out grills + --url "https://wellfound.com/role/r/software-engineer" \ + --pages 3 validations: required: true @@ -98,7 +94,7 @@ body: attributes: label: What you expected instead placeholder: >- - 96 products, as the README says a category page yields. + 32-45 job rows per page, as the README says a role landing page yields. validations: required: true diff --git a/.github/ISSUE_TEMPLATE/question.yml b/.github/ISSUE_TEMPLATE/question.yml index db40153..8fb7db2 100644 --- a/.github/ISSUE_TEMPLATE/question.yml +++ b/.github/ISSUE_TEMPLATE/question.yml @@ -7,8 +7,8 @@ body: value: | Worth a look first, in this order: - - [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md) — ordered by how - often each thing actually happens + - The README's "Traps that look like bugs" and "What is gated and + what is not" sections - The README section on the 2Captcha product you are using - `python3 env_config.py` — reports what credentials are configured, without printing them diff --git a/.github/ISSUE_TEMPLATE/site_changed.yml b/.github/ISSUE_TEMPLATE/site_changed.yml index 69fa3b3..2b59b03 100644 --- a/.github/ISSUE_TEMPLATE/site_changed.yml +++ b/.github/ISSUE_TEMPLATE/site_changed.yml @@ -1,5 +1,5 @@ name: Site changed / selectors broke -description: Etsy changed something and the parser no longer matches. +description: Wellfound changed something and the parser no longer matches. labels: ["site-change"] body: - type: markdown @@ -7,21 +7,23 @@ body: value: | This is the most common way a scraper stops working, and it is worth its own report because the fix is different from a code bug: something on - etsy.com moved. + wellfound.com moved. - Useful to know before filing: the parser tries **JSON-LD first**, then a - CSS + URL-pattern fallback. Which one broke narrows the fix a lot. + Useful to know before filing: a landing page is read from the + normalised Apollo cache in `__NEXT_DATA__`, and a job page from its + one schema.org `JobPosting` block. The row's `data_source` column + (`apollo` / `jsonld`) says which one was read, and which one broke + narrows the fix a lot. - type: dropdown id: what_broke attributes: label: What stopped matching options: - - Products (0 rows on a page that visibly has products) - - A specific field is null or wrong (price, sku, image, URL) - - original_price or lowest_price_30d empty out (the tile overlay) - - DOM price confirmation dropped below 90% in the run log - - Pagination stops early or repeats a page + - Jobs (0 rows on a page that visibly has jobs) + - A specific field is null or wrong (salary, equity, sku, company, URL) + - The landing row and the job page disagree about a job + - Pagination stops early, repeats a page, or stops with page_echo_mismatch - The captcha detector finds nothing on a page that has one - Something else validations: @@ -31,8 +33,8 @@ body: id: url attributes: label: URL - description: The exact listing URL, and the country your exit IP was in. - placeholder: "https://www.etsy.com/search?q=handmade+mug (locale: /de/, exit IP: DE residential, Scraping Browser pid: etsy1)" + description: The exact URL, and what kind of exit IP it was fetched from. + placeholder: "https://wellfound.com/role/r/software-engineer (exit IP: US residential)" validations: required: true @@ -43,9 +45,9 @@ body: description: | Whichever of these you can get. A capture beats a description. - - JSON-LD as the page actually serves it: - `python3 -c "import json,sys,re;h=open('dump.html').read();print([m[:400] for m in re.findall(r'',h,re.S)][:2])"` - - Or open devtools and paste one `Product` node from the `ItemList`. + - A `--dump-html` capture of the page, or the relevant part of it: + the `__NEXT_DATA__` script on a landing page, the + `application/ld+json` `JobPosting` on a job page. - For a field problem: the value you got, and the value on the page. render: text validations: diff --git a/CHANGELOG.md b/CHANGELOG.md index fba5d04..6bec7cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,20 @@ does it leads the release notes rather than being discovered from a bill. `http_code` (falling back to `status` only if that is an int). A regression check drives the real `fetch_html` with `requests.post` stubbed. +- The bug-report and site-changed issue templates were Etsy's, copied from + the repo this one was bootstrapped from: they told a reporter about + DataDome, `t=bv`, Etsy's shop ratings and storefront redirects, and gave an + etsy.com URL as the example. Rewritten for Wellfound from this README (the + datacentre 403, the gated `/company/` route, the Apollo cache / JSON-LD + split, `page_echo_mismatch`). +- Both issue templates and `playwright_scraper.py --dump-html --help` + pointed at a `TROUBLESHOOTING.md` this repo has never had. They now point + at the README sections that exist. +- `captcha_solver.py`'s docstring referred to a "No DataDome solver" + section that is not in the file. The dangling line is gone. + +- `SECURITY.md` said this project has no releases or version tags; it has + both. "Supported versions" now names the latest release and `main`. ## [0.1.1] — 2026-09-17 diff --git a/SECURITY.md b/SECURITY.md index b652db6..cebe996 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -79,9 +79,9 @@ Not because these do not matter, but because they belong somewhere else: ## Supported versions -`main` only. This project has no releases or version tags; fixes land on `main` -and you update by pulling. If you are running an old clone, update before -reporting. +The latest release and `main`. Fixes land on `main` first and ship in the +next tagged release (see the Releases page). If you are running an old clone, +update before reporting. ## If you have leaked a key diff --git a/captcha_solver.py b/captcha_solver.py index 3367617..c0d2273 100644 --- a/captcha_solver.py +++ b/captcha_solver.py @@ -93,8 +93,6 @@ `page_flow.STATE_POLICY` — not this file — decides which state is worth money at all. -There is deliberately no DataDome path here. See "No DataDome solver" below. - Flow: 1. Both detectors run and are reconciled (see reconcile_detections) to decide the variant: v3, v2-invisible or v2-checkbox. The parameters diff --git a/playwright_scraper.py b/playwright_scraper.py index 24369a3..3243cb0 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -1706,7 +1706,8 @@ def parse_args(): p.add_argument("--dump-html", default=None, metavar="PATH", help="Save the exact HTML the parser is given, on success as " "well as failure. Useful when the row count is right but " - "a column comes back empty — see TROUBLESHOOTING.md.") + "a column comes back empty — see \"Traps that look like " + "bugs\" in the README.") p.add_argument("--headless", action="store_true", default=True) p.add_argument("--headful", dest="headless", action="store_false") args = p.parse_args()