Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 20 additions & 24 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,26 +10,22 @@ body:
mask credentials in their own logs, but raw HTML dumps and the Scraper
API's `x-debug` header do not. Replace them with `***`.

Two things answer most reports faster than we can:
A few things answer most reports faster than we can:

- **Everything exit 3?** Check the exit address first. Etsy's
DataDome answers every datacentre and VPN IP with HTTP 403, so a run
from a VPS or a CI runner is blocked whatever else is set. If the
log says `t=bv`, a captcha key does not help — that challenge is
not solvable. The Scraping Browser API (`--cdp-endpoint`) is the
path that works; if a profile keeps refusing after three retries,
try a different `pid`.
- **Zero products?** You may be on a page that has none — a taxonomy
hub, or one page past the end of a listing. It answers 200 with
nothing on it, which is exit 4 and a correct answer. See
[TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md).
- **`rating` empty on every row?** That is expected on a listing run:
Etsy publishes no per-listing rating there. The stars you see on a
tile are the shop's and are in `shop_rating`. See the README.
- **Wrong currency or language?** Etsy decides both from your exit
IP, not from the URL — a request for `/fr/…` from a German exit is
redirected to `/de/…`. The row's `url` says which storefront
actually answered.
- **Everything exit 3?** Check the exit address first. Wellfound
answers every route with HTTP 403 from a datacentre address,
`robots.txt` included, and no browser gets past it. A residential
exit (`WELLFOUND_PROXY` in `.env`) is what works — see "What is
gated and what is not" in the README.
- **`/company/{slug}` refused?** It is gated even from a residential
exit. There is no `--mode company` for that reason.
- **Zero jobs?** A listing can genuinely be empty, which is exit 4
and a correct answer. Check the URL is a `/role/...`, `/jobs` or
`/jobs/{id}-{slug}` page.
- **A column empty or "wrong"?** Read "Traps that look like bugs" in
the README first: `salary_period` is null on landing rows,
`has_equity: false` differs from `null`, and a page is twenty
companies, not twenty jobs.

- type: dropdown
id: engine
Expand Down Expand Up @@ -63,8 +59,8 @@ body:
label: Exit code
description: >-
`echo $?` straight after the run. These mean specific things — 3 is a
challenge page, 4 is "ran fine, found nothing", 124 is a self-imposed
timeout. The table is in the README.
refusal or challenge page, 4 is "ran fine, found nothing", 124 is a
self-imposed timeout. The table is in the README.
placeholder: "4"
validations:
required: true
Expand All @@ -77,8 +73,8 @@ body:
render: shell
placeholder: |
python3 playwright_scraper.py \
--url "https://www.etsy.com/search?q=handmade+mug" \
--pages 3 --out grills
--url "https://wellfound.com/role/r/software-engineer" \
--pages 3
validations:
required: true

Expand All @@ -98,7 +94,7 @@ body:
attributes:
label: What you expected instead
placeholder: >-
96 products, as the README says a category page yields.
32-45 job rows per page, as the README says a role landing page yields.
validations:
required: true

Expand Down
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/question.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ body:
value: |
Worth a look first, in this order:

- [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md) — ordered by how
often each thing actually happens
- The README's "Traps that look like bugs" and "What is gated and
what is not" sections
- The README section on the 2Captcha product you are using
- `python3 env_config.py` — reports what credentials are configured,
without printing them
Expand Down
30 changes: 16 additions & 14 deletions .github/ISSUE_TEMPLATE/site_changed.yml
Original file line number Diff line number Diff line change
@@ -1,27 +1,29 @@
name: Site changed / selectors broke
description: Etsy changed something and the parser no longer matches.
description: Wellfound changed something and the parser no longer matches.
labels: ["site-change"]
body:
- type: markdown
attributes:
value: |
This is the most common way a scraper stops working, and it is worth its
own report because the fix is different from a code bug: something on
etsy.com moved.
wellfound.com moved.

Useful to know before filing: the parser tries **JSON-LD first**, then a
CSS + URL-pattern fallback. Which one broke narrows the fix a lot.
Useful to know before filing: a landing page is read from the
normalised Apollo cache in `__NEXT_DATA__`, and a job page from its
one schema.org `JobPosting` block. The row's `data_source` column
(`apollo` / `jsonld`) says which one was read, and which one broke
narrows the fix a lot.

- type: dropdown
id: what_broke
attributes:
label: What stopped matching
options:
- Products (0 rows on a page that visibly has products)
- A specific field is null or wrong (price, sku, image, URL)
- original_price or lowest_price_30d empty out (the tile overlay)
- DOM price confirmation dropped below 90% in the run log
- Pagination stops early or repeats a page
- Jobs (0 rows on a page that visibly has jobs)
- A specific field is null or wrong (salary, equity, sku, company, URL)
- The landing row and the job page disagree about a job
- Pagination stops early, repeats a page, or stops with page_echo_mismatch
- The captcha detector finds nothing on a page that has one
- Something else
validations:
Expand All @@ -31,8 +33,8 @@ body:
id: url
attributes:
label: URL
description: The exact listing URL, and the country your exit IP was in.
placeholder: "https://www.etsy.com/search?q=handmade+mug (locale: /de/, exit IP: DE residential, Scraping Browser pid: etsy1)"
description: The exact URL, and what kind of exit IP it was fetched from.
placeholder: "https://wellfound.com/role/r/software-engineer (exit IP: US residential)"
validations:
required: true

Expand All @@ -43,9 +45,9 @@ body:
description: |
Whichever of these you can get. A capture beats a description.

- JSON-LD as the page actually serves it:
`python3 -c "import json,sys,re;h=open('dump.html').read();print([m[:400] for m in re.findall(r'<script type=\"application/ld\\+json\">(.*?)</script>',h,re.S)][:2])"`
- Or open devtools and paste one `Product` node from the `ItemList`.
- A `--dump-html` capture of the page, or the relevant part of it:
the `__NEXT_DATA__` script on a landing page, the
`application/ld+json` `JobPosting` on a job page.
- For a field problem: the value you got, and the value on the page.
render: text
validations:
Expand Down
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,20 @@ does it leads the release notes rather than being discovered from a bill.
`http_code` (falling back to `status` only if that is an int). A
regression check drives the real `fetch_html` with `requests.post`
stubbed.
- The bug-report and site-changed issue templates were Etsy's, copied from
the repo this one was bootstrapped from: they told a reporter about
DataDome, `t=bv`, Etsy's shop ratings and storefront redirects, and gave an
etsy.com URL as the example. Rewritten for Wellfound from this README (the
datacentre 403, the gated `/company/` route, the Apollo cache / JSON-LD
split, `page_echo_mismatch`).
- Both issue templates and `playwright_scraper.py --dump-html --help`
pointed at a `TROUBLESHOOTING.md` this repo has never had. They now point
at the README sections that exist.
- `captcha_solver.py`'s docstring referred to a "No DataDome solver"
section that is not in the file. The dangling line is gone.

- `SECURITY.md` said this project has no releases or version tags; it has
both. "Supported versions" now names the latest release and `main`.

## [0.1.1] — 2026-09-17

Expand Down
6 changes: 3 additions & 3 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,9 @@ Not because these do not matter, but because they belong somewhere else:

## Supported versions

`main` only. This project has no releases or version tags; fixes land on `main`
and you update by pulling. If you are running an old clone, update before
reporting.
The latest release and `main`. Fixes land on `main` first and ship in the
next tagged release (see the Releases page). If you are running an old clone,
update before reporting.

## If you have leaked a key

Expand Down
2 changes: 0 additions & 2 deletions captcha_solver.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,6 @@
`page_flow.STATE_POLICY` — not this file — decides which state is worth
money at all.

There is deliberately no DataDome path here. See "No DataDome solver" below.

Flow:
1. Both detectors run and are reconciled (see reconcile_detections) to
decide the variant: v3, v2-invisible or v2-checkbox. The parameters
Expand Down
3 changes: 2 additions & 1 deletion playwright_scraper.py
Original file line number Diff line number Diff line change
Expand Up @@ -1706,7 +1706,8 @@ def parse_args():
p.add_argument("--dump-html", default=None, metavar="PATH",
help="Save the exact HTML the parser is given, on success as "
"well as failure. Useful when the row count is right but "
"a column comes back empty — see TROUBLESHOOTING.md.")
"a column comes back empty — see \"Traps that look like "
"bugs\" in the README.")
p.add_argument("--headless", action="store_true", default=True)
p.add_argument("--headful", dest="headless", action="store_false")
args = p.parse_args()
Expand Down
Loading