Skip to content

v0.2.0 #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jehrr merged 1 commit into from
Oct 15, 2026
Merged

v0.2.0 #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 43 additions & 49 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ the release notes rather than left to be discovered from a bill or a diff.

## [Unreleased]

## [0.2.0] — 2026-10-15

> **Behaviour changes for scripts.**
> - A search the site answers with its own "nothing matched"
> (`search-count:0`) now exits **4** (`no_results`, an empty result) in all
Expand All @@ -36,6 +38,7 @@ the release notes rather than left to be discovered from a bill or a diff.
> under your umask; an existing file keeps its mode.

### Fixed

- An interrupted write (a crash, a kill, a full disk) no longer destroys the
previous good output: `out.json`, `out.csv` and `out.meta.json` are each
written atomically. Measured before the fix: an 852-byte `out.json` came
Expand Down Expand Up @@ -67,59 +70,12 @@ the release notes rather than left to be discovered from a bill or a diff.
a fresh endpoint.
- `--proxy-block-retries` is accepted for compatibility with the other
scrapers in this family; its help now says it has no effect here.

### Added
- CSV output prefixes any text cell that begins with `=`, `+`, `-`, `@`, a
tab or a line break with an apostrophe, so a spreadsheet does not run it as
a formula. Numbers are untouched, JSON keeps the site's text, and the
sidecar's `csv_cells_escaped` counts the cells that differ.

### Changed
- The daily canary no longer skips green when the runner is blocked (exit
3): the README's claim is that a local browser needs no key and no proxy
here, and bare runners have been served on every scheduled run checked, so
a block is now a failure. Its exit 4 and 5 messages match the codes above.

### Fixed
- The daily canary was red from 2026-10-07 to 2026-10-12 although the run
itself was healthy (72 rows, `complete`, 3 of 3 pages). Its sanity check
rejected `pct_funded <= 0`, and a real campaign that raised 10,000 against
a goal of 900,000,000 rounds to 0.0. The check now fails on a negative
percentage, or one computed from a goal that is not positive; zero is a
figure. The parser was right and is unchanged, with a check pinning the case.

### Changed
- `scraper_api_client.py` warns on stderr when the key is passed with `--key`
(visible to `ps`, kept in shell history). `TWOCAPTCHA_KEY` in `.env` is
unchanged and remains the documented way.

> **Correction to v0.1.0's release notes.** They said the 2Captcha balance
> was "unchanged to five decimal places". That was measured across the four
> bare-browser runs and is true of them, but it is not true of the session as
> a whole: the balance moved **$0.0024** over the day, which is Scraping
> Browser session usage from `--cdp-endpoint` runs. The claim that matters is
> narrower and now stated that way — 2Captcha's own statistics endpoint
> reports **0 captcha solves and $0.00000** for the day. A run without
> `--cdp-endpoint` still costs nothing at all.

### The site has a SECOND captcha, configured but never rendered

Asked plainly which captcha this site uses, and answered by measuring. There
are two, at different layers:

1. **Cloudflare Turnstile**, in a managed challenge at the edge. This is the
one v0.1.0 documents: met, self-cleared, never solved.
2. **reCAPTCHA Enterprise**, wired into the application and enabled on every
page the site serves — site key `6LeRruUr…`, the enterprise loader on
**recaptcha.net** (not google.com), and an empty `<captcha-widgets>` mount
point. It is never shown to an anonymous reader.

"No challenge rendered" is not "no captcha configured", so the useful
question was whether this repo would RECOGNISE it. Constructing the shapes
this site's own captcha would take found two the static detector missed.

### Fixed

- **Scraper API: `waitFor` is now sent as a JSON object.** Measured
2026-09-23 against `scraper.2captcha.com/tasks/sync`: the JSON-encoded
string this client sent (the form older docs described) is refused with
Expand Down Expand Up @@ -147,7 +103,6 @@ this site's own captcha would take found two the static detector missed.
cannot be committed. `.github/ci_checks.py` had already learned this;
the suite now matches it.
- **`.gitignore` did not cover this repo's own default output prefixes.**

- **Leftovers from the repo this one was bootstrapped from.** `.gitignore`
listed transfermarkt-scraper's `transfermarkt_products.*` output prefixes
(this repo's own `indiegogo_projects.*` was already listed) and described
Expand All @@ -161,11 +116,50 @@ this site's own captcha would take found two the static detector missed.

### Added

- CSV output prefixes any text cell that begins with `=`, `+`, `-`, `@`, a
tab or a line break with an apostrophe, so a spreadsheet does not run it as
a formula. Numbers are untouched, JSON keeps the site's text, and the
sidecar's `csv_cells_escaped` counts the cells that differ.
- **`category_code`** on `Campaign` — the site's numeric, locale-independent
catalogue id. Join on it; `category` is display text.
- Checks pinning all of the above, each verified to fail when its fix is
reverted.

### Changed

- The daily canary no longer skips green when the runner is blocked (exit
3): the README's claim is that a local browser needs no key and no proxy
here, and bare runners have been served on every scheduled run checked, so
a block is now a failure. Its exit 4 and 5 messages match the codes above.
- `scraper_api_client.py` warns on stderr when the key is passed with `--key`
(visible to `ps`, kept in shell history). `TWOCAPTCHA_KEY` in `.env` is
unchanged and remains the documented way.

> **Correction to v0.1.0's release notes.** They said the 2Captcha balance
> was "unchanged to five decimal places". That was measured across the four
> bare-browser runs and is true of them, but it is not true of the session as
> a whole: the balance moved **$0.0024** over the day, which is Scraping
> Browser session usage from `--cdp-endpoint` runs. The claim that matters is
> narrower and now stated that way — 2Captcha's own statistics endpoint
> reports **0 captcha solves and $0.00000** for the day. A run without
> `--cdp-endpoint` still costs nothing at all.

### The site has a SECOND captcha, configured but never rendered

Asked plainly which captcha this site uses, and answered by measuring. There
are two, at different layers:

1. **Cloudflare Turnstile**, in a managed challenge at the edge. This is the
one v0.1.0 documents: met, self-cleared, never solved.
2. **reCAPTCHA Enterprise**, wired into the application and enabled on every
page the site serves — site key `6LeRruUr…`, the enterprise loader on
**recaptcha.net** (not google.com), and an empty `<captcha-widgets>` mount
point. It is never shown to an anonymous reader.

"No challenge rendered" is not "no captcha configured", so the useful
question was whether this repo would RECOGNISE it. Constructing the shapes
this site's own captcha would take found two the static detector missed.

### The proxy path, now exercised live

v0.1.0 shipped with `--proxy` untested against a real gateway -- there was
Expand Down Expand Up @@ -211,7 +205,6 @@ Measured 2026-09-17 through a 2Captcha residential gateway:
- `--concurrency 3` over 6 pages returns 144 rows with unique
`(page, position)` and no duplicate skus.
- `/de/` and `/zh/` each return 48/48 fully populated rows.

- `SECURITY.md` said this project has no releases or version tags; it has
both. "Supported versions" now names the latest release and `main`.
- `captcha_solver.py`'s docstring pointed at a "No DataDome solver" section
Expand Down Expand Up @@ -283,4 +276,5 @@ From a datacentre address (Hetzner, Helsinki, AS24940):
- `fundedInSeconds` is never null; it reports `0` for "not applicable", which
is normalised away.

[0.2.0]: https://github.com/2scraper/indiegogo-scraper/releases/tag/v0.2.0
[0.1.0]: https://github.com/2scraper/indiegogo-scraper/releases/tag/v0.1.0
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "indiegogo-scraper"
version = "0.1.0"
version = "0.2.0"
description = "Scrape indiegogo.com project listings, campaigns and reward tiers with Playwright, Selenium, Puppeteer or the 2Captcha Scraping Browser API"
readme = "README.md"
license = { file = "LICENSE" }
Expand Down
Loading