Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions .github/workflows/canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -213,11 +213,18 @@ jobs:

# A goal at or below zero, or a percentage computed from one, means
# pct_funded's guards stopped guarding.
# Zero itself is a legitimate figure: a campaign that has raised
# 10,000 against a goal of 900,000,000 rounds to 0.0 at two
# decimals, and a campaign with nothing raised is 0% funded. What
# must never happen is a NEGATIVE percentage, or a percentage
# computed from a goal that is absent or not positive.
bad_pct = [r.get("sku") for r in rows
if r.get("pct_funded") is not None and r["pct_funded"] <= 0]
if r.get("pct_funded") is not None
and (r["pct_funded"] < 0
or r.get("goal") is None or r["goal"] <= 0)]
if bad_pct:
fail.append(f"{len(bad_pct)} row(s) report pct_funded <= 0: "
f"{bad_pct[:3]}")
fail.append(f"{len(bad_pct)} row(s) report a negative pct_funded "
f"or one without a positive goal: {bad_pct[:3]}")

# A page number never threaded into the parser produces (page,
# position) collisions across pages that look, from the row count
Expand Down
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,19 @@ the release notes rather than left to be discovered from a bill or a diff.

## [Unreleased]

### Fixed
- The daily canary was red from 2026-10-07 to 2026-10-12 although the run
itself was healthy (72 rows, `complete`, 3 of 3 pages). Its sanity check
rejected `pct_funded <= 0`, and a real campaign that raised 10,000 against
a goal of 900,000,000 rounds to 0.0. The check now fails on a negative
percentage, or one computed from a goal that is not positive; zero is a
figure. The parser was right and is unchanged, with a check pinning the case.

### Changed
- `scraper_api_client.py` warns on stderr when the key is passed with `--key`
(visible to `ps`, kept in shell history). `TWOCAPTCHA_KEY` in `.env` is
unchanged and remains the documented way.

> **Correction to v0.1.0's release notes.** They said the 2Captcha balance
> was "unchanged to five decimal places". That was measured across the four
> bare-browser runs and is true of them, but it is not true of the session as
Expand Down
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,11 @@ everything the site will serve. It is also a **sample**. The sidecar records
`total_results`, `pages_available`, `capped_by_site` and `reachable_max` so
a consumer can tell, and `diff_runs.py` says so before diffing.

To read it: `capped_by_site: true` means the run holds a sample of what
the query matched, however many pages it fetched. `pages_completed` against
`pages_available` says how much of the *reachable* list the run covered, and
`reachable_max` is the most rows the site will ever serve for it.

A narrower query is not capped: `?term=solar` reported 801 matches with
`hasCappedResults: false`.

Expand Down
4 changes: 4 additions & 0 deletions scraper_api_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -479,6 +479,10 @@ def parse_args():
p.add_argument("--dump-html", default=None,
help="Save the exact HTML the parser is given (always, even on success)")
args = p.parse_args()
if args.key:
print("warning: --key puts the API key in this process's argv, where "
"anything that can run `ps` can read it, and in shell history. "
"Set TWOCAPTCHA_KEY in .env instead.", file=sys.stderr)
# This client uses --key rather than --twocaptcha-key, so the env
# mapping is spelled out instead of defaulted -- same pattern this
# family's mediamarkt-scraper uses for its own scraper_api_client.py.
Expand Down
4 changes: 4 additions & 0 deletions smoke_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -748,6 +748,10 @@ def test_pct_funded_guards():
assert pp.pct_funded(100.0, 0) is None
assert pp.pct_funded(None, 100.0) is None
assert pp.pct_funded(12576.05, 12000.0) == 104.8
# A real campaign (Aethersworn, 2026-10-12): 10,000 raised against a goal
# of 900,000,000 is 0.0 after rounding. It is a figure, not a missing
# value, and the daily canary once failed for six days by treating it as one.
assert pp.pct_funded(10000.0, 900000000.0) == 0.0


@check("days_left is computed from the absolute end, not the localised card")
Expand Down
Loading