Treat a 403 refusal as a block, keep output file modes, validate numeric flags

.env.example

@@ -37,9 +37,10 @@
 TWOCAPTCHA_KEY=your_2captcha_api_key_here
 
 # --- The Scraping Browser API (a managed browser over CDP) ---------------
-# The endpoint SHAPE, not a working endpoint: a profile's credentials live
-# about a day, so anything pasted here is stale by tomorrow and produces an
-# auth error a long way from its cause. Get fresh ones from the dashboard.
+# The endpoint SHAPE, not a working endpoint. A profile's credentials do
+# expire (how long they last varies; do not plan around a fixed figure), and
+# a stale endpoint produces an auth error (HTTP 401) a long way from its
+# cause. Get fresh ones from the dashboard.
 #
 #   country-xx  picks the exit country. `country-jp` for a Japanese exit.
 #   pid-...     is a profile with persistent cookies. ONE live connection

.github/ci_checks.py

@@ -300,7 +300,7 @@ def tracked_files():
 # into pip's vendored code and flags a 32-hex string in `_elffile.py` as
 # key-shaped. Correct about the string, wrong about the file — and it is the
 # FIRST thing a new user sees from `python3 smoke_test.py`. A guard people
-# have to argue with is one they learn to suppress (CLAUDE.md §22).
+# have to argue with is one they learn to suppress.
 #
 # Deliberately NOT a narrowing of what gets scanned: an untracked file is
 # still read, because a key pasted into a scratch file beside the scripts is

.github/workflows/canary.yml

@@ -20,10 +20,11 @@ name: canary
 # kind: a keyword search returned 45 products a page, and a three-page run
 # returned 135 rows with 100% price coverage and status `complete`.
 #
-# The family's rule is that a canary which CANNOT pass must skip rather than
-# fail, because a check that is always red teaches everyone to ignore checks.
-# The other half of that rule applies here: a canary that CAN pass without a
-# credential must never be gated on one.
+# A canary that CAN pass without a credential must never be gated on one,
+# and a run that did not actually test the site must never end green: a
+# block (exit 3), a partial run (exit 6), content never obtained (exit 5) or
+# zero rows all fail the job. A green badge here means a real scrape of
+# rakuten.co.jp parsed real rows today, and nothing less.
 #
 # What a runner cannot test, and why there is no second job for it:
 #
@@ -99,40 +100,28 @@ jobs:
 
       - name: Interpret the exit code
         id: verdict
-        # ACCESS CONDITIONS ARE NOT DEFECTS. Failing the job on any non-zero
-        # exit would make the canary red for things that are nobody's bug —
-        # so 3 (blocked) and 6 (partial) are WARNINGS, while a run that got
-        # in and found the wrong thing stays red.
-        #
-        # Note that on THIS site a blocked canary is genuinely surprising
-        # rather than routine, because the listing routes are not gated. If
-        # exit 3 starts appearing every day, that is news: either Rakuten has
-        # begun refusing GitHub's address ranges, or something in this repo
-        # has started sending a client whose claimed identity and TLS
-        # fingerprint disagree — which is the one thing Akamai was measured
-        # to refuse here.
+        # EVERY NON-ZERO EXIT FAILS THE JOB. This canary needs no secret and
+        # the listing routes were measured ungated from datacentre
+        # addresses, GitHub's runners included, so a run that did not get in
+        # is news rather than an access condition to wave through. The
+        # messages differ only in where to look first.
         run: |
           set -e
           code="${{ steps.run.outputs.exit_code }}"
-          tested=true
           case "$code" in
             0)
               echo "OK — products parsed from a live run." ;;
             3)
-              tested=false
-              echo "::warning::Blocked before parsing (exit 3) — reported as an ACCESS condition rather than a code defect, so this run is not failed. On this site that is NOT routine: the listing routes were measured ungated from a datacentre address. Check canary_run_page1_debug.html — if it is 43 bytes of 'Reference  #', Akamai refused the client, and the first thing to suspect is a user agent or fingerprint that disagrees with the TLS handshake underneath it, not the address."
-              echo "### Canary did NOT test anything (exit 3, blocked)" >> "$GITHUB_STEP_SUMMARY"
-              echo "The site did not serve the request, so the parser was not exercised. Unusual here — see TROUBLESHOOTING.md, first section." >> "$GITHUB_STEP_SUMMARY" ;;
+              echo "::error::Blocked before parsing (exit 3). The listing routes were measured ungated from datacentre addresses, so this is news: either Rakuten has begun refusing GitHub's address ranges, or this repo has started sending a client whose claimed identity and TLS fingerprint disagree. Check canary_run_page1_debug.html — 43 bytes of 'Reference  #' is Akamai refusing the client; a branded page under HTTP 403 is a route refusal."
+              exit 1 ;;
             6)
-              tested=false
-              echo "::warning::Partial run (exit 6) — some pages were gathered and the run then stopped early. canary_run.meta.json names which pages failed. If the sidecar says stop_reason=end_of_listing this is not partial at all and the code is wrong; if it repeats every day, suspect the ?p= convention."
-              echo "### Canary tested only PART of the run (exit 6)" >> "$GITHUB_STEP_SUMMARY"
-              echo "See \`pages_failed\` in the sidecar artefact." >> "$GITHUB_STEP_SUMMARY" ;;
+              echo "::error::Partial run (exit 6): some pages were gathered and the run then stopped early, so the run did not test what it was asked to. canary_run.meta.json names which pages failed. If stop_reason is end_of_listing this is not partial at all and the code is wrong; a stop_reason of blocked_* or a throttle that never cleared means the site refused part of the run."
+              exit 1 ;;
             4)
               echo "::error::Ran fine but parsed 0 rows (exit 4) from a genre listing that holds 45 products per page. Either window.__INITIAL_STATE__.state.data.ichibaSearch has moved, or the item-URL pattern the DOM fallback anchors on has. This is the regression this canary exists to catch — the HTML dump in the artefacts tells you which."
               exit 1 ;;
             5)
-              echo "::error::Remote API error (exit 5) from a run that uses no remote API. That should be impossible here; read the log."
+              echo "::error::The content was never obtained (exit 5): a navigation timeout, a transport failure or a page that parsed to nothing although the site stated rows. This run uses no remote API and no credential, so diagnose the transport and the dump; the log names the stop_reason."
               exit 1 ;;
             2)
               echo "::error::Bad usage (exit 2) — the workflow's own arguments are wrong, not the site."
@@ -141,7 +130,7 @@ jobs:
               echo "::error::The scraper crashed (exit $code)."
               exit 1 ;;
           esac
-          echo "tested=$tested" >> "$GITHUB_OUTPUT"
+          echo "tested=true" >> "$GITHUB_OUTPUT"
 
       - name: Check what the run actually produced
         if: steps.verdict.outputs.tested == 'true'
@@ -197,7 +186,7 @@ jobs:
                "this site is a mall; shop_code came from the payload on "
                "every measured row")
 
-          # THE ARITHMETIC BUG §18 NAMES: position restarts per page, so the
+          # THE ARITHMETIC BUG: position restarts per page, so the
           # pair is what has to be unique.
           pairs = [(r.get("page"), r.get("position")) for r in rows]
           dupes = [p for p, n in collections.Counter(pairs).items() if n > 1]
@@ -218,7 +207,7 @@ jobs:
                        and not r.get("review_count") for r in rows))
 
           # NO NEGATIVE DISCOUNTS. Costs one line and catches the regression
-          # forever (§4).
+          # forever.
           want("a row has an original_price at or below its price",
                not any(r.get("original_price") is not None
                        and r.get("price") is not None

.github/workflows/tests.yml

@@ -83,52 +83,6 @@ jobs:
           sys.exit(1 if failed else 0)
           EOF
 
-      - name: Committed sample output is real
-        # The README links to these two files, so a push without them ships
-        # broken links. And a sample invented by hand drifts from the schema and
-        # teaches readers the wrong field names, so these are cut from a real
-        # run rather than written by hand.
-        run: |
-          python - <<'EOF'
-          import csv, json, sys
-          from pathlib import Path
-
-          MARKERS = ("sample-product-", "example brand", "sample product",
-                     "product description text", "lorem ipsum")
-
-          fail = []
-          for name in ("sample_output.json", "sample_output.csv"):
-              if not Path(name).is_file():
-                  fail.append(f"{name} is missing — regenerate it from a real run")
-
-          if not fail:
-              rows = json.loads(Path("sample_output.json").read_text())
-              if not rows:
-                  fail.append("sample_output.json is empty")
-              blob = json.dumps(rows).lower()
-              hit = [m for m in MARKERS if m in blob]
-              if hit:
-                  fail.append(f"sample_output.json looks fabricated: {hit}")
-
-              # Column set must match what the writers actually produce.
-              from output_writer import Product
-              from dataclasses import asdict
-              expected = list(asdict(Product()).keys())
-              for i, row in enumerate(rows):
-                  if list(row.keys()) != expected:
-                      fail.append(f"row {i} columns drifted from output_writer.Product")
-                      break
-
-              with open("sample_output.csv", newline="", encoding="utf-8") as f:
-                  header = next(csv.reader(f))
-              if header != expected:
-                  fail.append("sample_output.csv header drifted from output_writer.Product")
-
-          for line in fail:
-              print("FAILED:", line)
-          sys.exit(1 if fail else 0)
-          EOF
-
       - name: Config reporter runs with no .env present
         run: python env_config.py
 
@@ -154,10 +108,12 @@ jobs:
         # endpoint, a planted 32-hex key and a planted http proxy URL.
         run: python3 .github/ci_checks.py --secret-check
 
-      - name: The repo's own --help and sample checks agree with this workflow
-        # ci_checks.py implements all three checks. The two above are also
-        # done as workflow steps for the clearer failure output; running them
-        # here as well is what keeps the two from drifting apart again.
+      - name: The repo's own --help and sample checks
+        # One implementation, in ci_checks.py. The sample check used to be
+        # duplicated here as an inline heredoc importing the row class; it is
+        # now only run from the script, so the two cannot drift apart. The
+        # committed samples are cut from a real run, and this fails if they
+        # go missing, look fabricated, or drift from output_writer.Product.
         run: python3 .github/ci_checks.py --help-check --sample-check
 
   engine-smoke:

.gitignore

@@ -40,8 +40,9 @@ run/
 runs/
 results/
 scratch/
-*.page[0-9]
-*.page[0-9][0-9]
+# --dump-html X --pages N writes X.pageN, and --pages goes to 150, so the
+# pattern covers any number of digits rather than stopping at 99.
+*.page[0-9]*
 live_results.zip
 
 # --- debug artifacts the engines drop next to themselves -----------------
@@ -99,7 +100,7 @@ secrets*
 # recoverable only because that repo was private at the time. On 2026-09-17
 # a sibling had an untracked `.env.bak` holding a 32-hex key in a PUBLIC
 # repository, one `git add -A` from publication. Craigslist was fixed then
-# and the fix never reached anyone else, which is §16's rule unapplied: when
+# and the fix never reached anyone else, which is the rule unapplied: when
 # you fix one, check the siblings.
 .env*
 !.env.example

CHANGELOG.md

@@ -12,6 +12,32 @@ their output or their bill.
 
 ## [Unreleased]
 
+> **Behaviour change: a 403 refusal is now a block, not a throttle.** No
+> engine used to pass the HTTP status to the page classifier, and Rakuten
+> renders its 403 refusal and its 503 rate-limit page with the same body, so
+> every 403 was read as a throttle: waited out three times (about 51 s) at
+> the same exit and then reported as a failed page. Playwright and pyppeteer
+> now carry the navigation's status into the classifier, so a 403 is
+> handled as a block (re-fetch or rotate, then exit 3) and a 503 is still
+> waited out. Selenium has no status to give and keeps the recoverable
+> reading.
+>
+> **Behaviour change: output files honour your umask.** `.json`, `.csv` and
+> `.meta.json` were written mode 0600 since the atomic writer arrived in
+> 0.2.1. A new file now gets `0666` minus the umask (0644 under the usual
+> 022), and rewriting an existing file keeps that file's mode.
+>
+> **Behaviour change: more numeric flags are checked at the command line.**
+> `--pages 0`, `--concurrency 0`, a negative `--delay`, `--retry-delay` or
+> `--proxy-block-retries` are now usage errors (exit 2) naming the flag, in
+> all three engines; `scraper_api_client.py` rejects a negative `--retries`
+> or `--retry-delay` and a `--timeout` below 1. `--pages 0` used to fetch
+> page 1 anyway and record `pages_requested: 0`.
+>
+> **Behaviour change: the canary is red whenever it did not test the site.**
+> A blocked (exit 3) or partial (exit 6) run used to end the job green with
+> a warning. Every non-zero exit now fails it.
+>
 > **Behaviour change:** `--retries 0` is now a usage error (exit 2). It used
 > to be accepted and made the page-load loop unreachable, so nothing was
 > fetched and the engine reported the browser's own blank page as a result:
@@ -25,9 +51,41 @@ their output or their bill.
   `range(1, retries + 1)`, so 0 never runs it and the engine judged
   `about:blank`. Three third-party audits of this family ran exactly that and
   reported a working scraper as broken. `--retries` now goes through
-  `page_flow.attempts` in every entry point; a check reads the source of each
-  one and calls the validator, so a new engine declaring the flag with a
-  plain int fails the suite.
+  `page_flow.attempts` in all three browser engines; a check reads the
+  source of each one and calls the validator, so a new engine declaring the
+  flag with a plain int fails the suite. (`scraper_api_client.py --retries`
+  counts EXTRA attempts, so 0 stays legal there.)
+- **pyppeteer crashed with `AttributeError` when a page kept failing to load
+  and no proxy was configured.** Its load-failure branch called
+  `pool.advance()` without checking for a pool. It now rotates only for an
+  unusable exit and only when there is another exit to go to, as the
+  Playwright and Selenium engines already did.
+- **A failing captcha solver could be asked twice for one page.** The first
+  of the two solve call sites counted only successful solves against the
+  per-page allowance. Both now count every attempted purchase.
+- **The static reCAPTCHA detector missed a v2 widget** in a saved page that
+  carried only the `g-recaptcha` container or the `api2/anchor` frame,
+  without the literal `grecaptcha`, so a `--dump-html` capture of one was
+  reported as having no captcha. No captcha has been observed on Rakuten;
+  this only affects diagnosis.
+- **Help text and comments described another site's refusal** (an HTTP/2
+  stream reset, a headless-only "Access Denied" page). The `--solve-captcha`
+  help in all three engines and the exit-code comments now describe
+  Rakuten's own: a 43-byte Akamai deny under HTTP 200, a branded page under
+  HTTP 403, and the same page under HTTP 503 as a rate limit.
+- **The canary's exit-5 message** said a remote API error was impossible;
+  exit 5 means the content was never obtained, which a transport failure can
+  cause on any run.
+- `.gitignore` covered `--dump-html` page dumps only up to `.page99`;
+  `--pages` goes to 150. `requirements-selenium.txt` mentioned a
+  `--chromedriver` flag that does not exist. `.env.example` no longer gives
+  a fixed lifetime for Scraping Browser credentials.
+- Comments and docstrings no longer cite the numbered sections of internal
+  notes the public cannot read; each rule is stated in this repository's own
+  words, and the suite fails if such a citation comes back.
+- The CI sample-output check runs from one implementation
+  (`ci_checks.py --sample-check`) instead of also as an inline copy, and the
+  wording scan now covers `smoke_test.py` itself.
 
 ## [0.2.1] — 2026-09-30
 

CONTRIBUTING.md

@@ -105,10 +105,10 @@ Then the rest of the presentation, in the order that matters:
    runs daily on a schedule, which is deliberate: the README's central claim
    is that you need no key, no proxy and no account to read this site's
    listings, and a scheduled, ungated, real three-page run is that claim
-   under test every morning. The family's rule that a canary which cannot
-   pass must SKIP has a second half — a canary that CAN pass without a
-   credential must never be gated on one, or the badge goes green every day
-   while testing nothing.
+   under test every morning. A canary that CAN pass without a credential
+   must never be gated on one, and any run that did not really test the
+   site (blocked, partial, nothing obtained, zero rows) fails the job, so a
+   green badge always means real rows were parsed that day.
 
    Note that `workflow_dispatch` needs the workflow to exist on the DEFAULT
    branch: from a feature branch `gh workflow run` answers

README.md

@@ -65,7 +65,8 @@ Two more things follow from that table, and both are easy to get wrong:
 The canary in this repository is that claim under test: a real three-page
 scrape, daily, from a bare GitHub runner, with **no secrets**. If Rakuten ever
 puts the listing routes behind something, the badge above goes red the next
-morning.
+morning: a blocked, partial or empty run fails the job rather than passing
+with a warning.
 
 ## Install