diff --git a/CHANGELOG.md b/CHANGELOG.md index 1607ccc..b9a9bf7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,8 @@ the release notes rather than left to be discovered from a bill or a diff. ## [Unreleased] +## [0.2.0] — 2026-10-15 + > **Behaviour changes for scripts.** > - A search the site answers with its own "nothing matched" > (`search-count:0`) now exits **4** (`no_results`, an empty result) in all @@ -36,6 +38,7 @@ the release notes rather than left to be discovered from a bill or a diff. > under your umask; an existing file keeps its mode. ### Fixed + - An interrupted write (a crash, a kill, a full disk) no longer destroys the previous good output: `out.json`, `out.csv` and `out.meta.json` are each written atomically. Measured before the fix: an 852-byte `out.json` came @@ -67,59 +70,12 @@ the release notes rather than left to be discovered from a bill or a diff. a fresh endpoint. - `--proxy-block-retries` is accepted for compatibility with the other scrapers in this family; its help now says it has no effect here. - -### Added -- CSV output prefixes any text cell that begins with `=`, `+`, `-`, `@`, a - tab or a line break with an apostrophe, so a spreadsheet does not run it as - a formula. Numbers are untouched, JSON keeps the site's text, and the - sidecar's `csv_cells_escaped` counts the cells that differ. - -### Changed -- The daily canary no longer skips green when the runner is blocked (exit - 3): the README's claim is that a local browser needs no key and no proxy - here, and bare runners have been served on every scheduled run checked, so - a block is now a failure. Its exit 4 and 5 messages match the codes above. - -### Fixed - The daily canary was red from 2026-10-07 to 2026-10-12 although the run itself was healthy (72 rows, `complete`, 3 of 3 pages). Its sanity check rejected `pct_funded <= 0`, and a real campaign that raised 10,000 against a goal of 900,000,000 rounds to 0.0. The check now fails on a negative percentage, or one computed from a goal that is not positive; zero is a figure. The parser was right and is unchanged, with a check pinning the case. - -### Changed -- `scraper_api_client.py` warns on stderr when the key is passed with `--key` - (visible to `ps`, kept in shell history). `TWOCAPTCHA_KEY` in `.env` is - unchanged and remains the documented way. - -> **Correction to v0.1.0's release notes.** They said the 2Captcha balance -> was "unchanged to five decimal places". That was measured across the four -> bare-browser runs and is true of them, but it is not true of the session as -> a whole: the balance moved **$0.0024** over the day, which is Scraping -> Browser session usage from `--cdp-endpoint` runs. The claim that matters is -> narrower and now stated that way — 2Captcha's own statistics endpoint -> reports **0 captcha solves and $0.00000** for the day. A run without -> `--cdp-endpoint` still costs nothing at all. - -### The site has a SECOND captcha, configured but never rendered - -Asked plainly which captcha this site uses, and answered by measuring. There -are two, at different layers: - -1. **Cloudflare Turnstile**, in a managed challenge at the edge. This is the - one v0.1.0 documents: met, self-cleared, never solved. -2. **reCAPTCHA Enterprise**, wired into the application and enabled on every - page the site serves — site key `6LeRruUr…`, the enterprise loader on - **recaptcha.net** (not google.com), and an empty `` mount - point. It is never shown to an anonymous reader. - -"No challenge rendered" is not "no captcha configured", so the useful -question was whether this repo would RECOGNISE it. Constructing the shapes -this site's own captcha would take found two the static detector missed. - -### Fixed - - **Scraper API: `waitFor` is now sent as a JSON object.** Measured 2026-09-23 against `scraper.2captcha.com/tasks/sync`: the JSON-encoded string this client sent (the form older docs described) is refused with @@ -147,7 +103,6 @@ this site's own captcha would take found two the static detector missed. cannot be committed. `.github/ci_checks.py` had already learned this; the suite now matches it. - **`.gitignore` did not cover this repo's own default output prefixes.** - - **Leftovers from the repo this one was bootstrapped from.** `.gitignore` listed transfermarkt-scraper's `transfermarkt_products.*` output prefixes (this repo's own `indiegogo_projects.*` was already listed) and described @@ -161,11 +116,50 @@ this site's own captcha would take found two the static detector missed. ### Added +- CSV output prefixes any text cell that begins with `=`, `+`, `-`, `@`, a + tab or a line break with an apostrophe, so a spreadsheet does not run it as + a formula. Numbers are untouched, JSON keeps the site's text, and the + sidecar's `csv_cells_escaped` counts the cells that differ. - **`category_code`** on `Campaign` — the site's numeric, locale-independent catalogue id. Join on it; `category` is display text. - Checks pinning all of the above, each verified to fail when its fix is reverted. +### Changed + +- The daily canary no longer skips green when the runner is blocked (exit + 3): the README's claim is that a local browser needs no key and no proxy + here, and bare runners have been served on every scheduled run checked, so + a block is now a failure. Its exit 4 and 5 messages match the codes above. +- `scraper_api_client.py` warns on stderr when the key is passed with `--key` + (visible to `ps`, kept in shell history). `TWOCAPTCHA_KEY` in `.env` is + unchanged and remains the documented way. + +> **Correction to v0.1.0's release notes.** They said the 2Captcha balance +> was "unchanged to five decimal places". That was measured across the four +> bare-browser runs and is true of them, but it is not true of the session as +> a whole: the balance moved **$0.0024** over the day, which is Scraping +> Browser session usage from `--cdp-endpoint` runs. The claim that matters is +> narrower and now stated that way — 2Captcha's own statistics endpoint +> reports **0 captcha solves and $0.00000** for the day. A run without +> `--cdp-endpoint` still costs nothing at all. + +### The site has a SECOND captcha, configured but never rendered + +Asked plainly which captcha this site uses, and answered by measuring. There +are two, at different layers: + +1. **Cloudflare Turnstile**, in a managed challenge at the edge. This is the + one v0.1.0 documents: met, self-cleared, never solved. +2. **reCAPTCHA Enterprise**, wired into the application and enabled on every + page the site serves — site key `6LeRruUr…`, the enterprise loader on + **recaptcha.net** (not google.com), and an empty `` mount + point. It is never shown to an anonymous reader. + +"No challenge rendered" is not "no captcha configured", so the useful +question was whether this repo would RECOGNISE it. Constructing the shapes +this site's own captcha would take found two the static detector missed. + ### The proxy path, now exercised live v0.1.0 shipped with `--proxy` untested against a real gateway -- there was @@ -211,7 +205,6 @@ Measured 2026-09-17 through a 2Captcha residential gateway: - `--concurrency 3` over 6 pages returns 144 rows with unique `(page, position)` and no duplicate skus. - `/de/` and `/zh/` each return 48/48 fully populated rows. - - `SECURITY.md` said this project has no releases or version tags; it has both. "Supported versions" now names the latest release and `main`. - `captcha_solver.py`'s docstring pointed at a "No DataDome solver" section @@ -283,4 +276,5 @@ From a datacentre address (Hetzner, Helsinki, AS24940): - `fundedInSeconds` is never null; it reports `0` for "not applicable", which is normalised away. +[0.2.0]: https://github.com/2scraper/indiegogo-scraper/releases/tag/v0.2.0 [0.1.0]: https://github.com/2scraper/indiegogo-scraper/releases/tag/v0.1.0 diff --git a/pyproject.toml b/pyproject.toml index fdacd51..5847279 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "indiegogo-scraper" -version = "0.1.0" +version = "0.2.0" description = "Scrape indiegogo.com project listings, campaigns and reward tiers with Playwright, Selenium, Puppeteer or the 2Captcha Scraping Browser API" readme = "README.md" license = { file = "LICENSE" }