Latest transfers: stop counting a repeated page as served; read the edge's 502 as a failed page

.github/workflows/canary.yml

@@ -102,6 +102,26 @@ jobs:
             --out canary_run --dump-html canary_run.html
           echo "exit_code=$?" >> "$GITHUB_OUTPUT"
 
+      - name: Scrape the latest-transfers listing, unfiltered, 3 pages
+        id: transfers
+        # The other paginated mode, on its own pagination: ?page=N on a plain
+        # query string, and an end the site marks by serving the last page
+        # again. The default view is filtered by the site to players valued
+        # EUR 500,000+ and held one 20-row page on 2026-10-15, so it cannot
+        # show whether a second page is followed; the unfiltered view held
+        # 4 pages (77 transfers) the same day.
+        continue-on-error: true
+        env:
+          TRANSFERMARKT_CDP_ENDPOINT: ${{ secrets.TRANSFERMARKT_CDP_ENDPOINT }}
+          TRANSFERMARKT_PROXY: ${{ secrets.TRANSFERMARKT_PROXY }}
+        run: |
+          set +e
+          python3 playwright_scraper.py \
+            --mode transfers --pages 3 \
+            --min-value 0 \
+            --out canary_transfers --dump-html canary_transfers.html
+          echo "exit_code=$?" >> "$GITHUB_OUTPUT"
+
       - name: Interpret the exit code
         id: verdict
         env:
@@ -213,6 +233,66 @@ jobs:
           sys.exit(1 if fail else 0)
           EOF
 
+      - name: Check the latest-transfers run
+        # Runs even when the market-values checks above failed, so one
+        # mode's break does not hide the other's.
+        if: ${{ !cancelled() }}
+        env:
+          CODE: ${{ steps.transfers.outputs.exit_code }}
+        run: |
+          if [ "$CODE" != "0" ]; then
+            echo "::error::--mode transfers exited $CODE (0 expected; 3 blocked, 4 zero rows, 5 nothing obtained, 6 partial — including stop_reason page_repeated, a page linked onward that served an earlier page again). canary_transfers.meta.json and canary_transfers_page*_debug.html in the artifacts say which."
+            exit 1
+          fi
+          python3 - <<'EOF'
+          import json, sys
+          from pathlib import Path
+
+          rows = json.loads(Path("canary_transfers.json").read_text())
+          meta = json.loads(Path("canary_transfers.meta.json").read_text())
+          fail = []
+
+          if meta.get("min_market_value_eur") != 0:
+              fail.append(f"min_market_value_eur={meta.get('min_market_value_eur')!r}, "
+                          f"expected 0 from --min-value 0")
+
+          # Tests the pagination and the parser, not the transfer season:
+          # more than one page's worth of rows (25 a page) means a second
+          # page was followed, whatever the listing's depth that day.
+          if meta.get("pages_completed", 0) < 2:
+              fail.append(f"only {meta.get('pages_completed')} page(s) served "
+                          f"(stop_reason={meta.get('stop_reason')!r}); the "
+                          f"unfiltered listing had 4 on 2026-10-15")
+          if len(rows) <= 25:
+              fail.append(f"only {len(rows)} rows: no second page's worth")
+          if meta.get("status") != "complete" or meta.get("stop_reason") not in (
+                  "completed", "listing_ended"):
+              fail.append(f"status={meta.get('status')!r} "
+                          f"stop_reason={meta.get('stop_reason')!r}")
+          skus = [r.get("sku") for r in rows]
+          if len(set(skus)) != len(skus):
+              fail.append("a transfer id appears twice: a page was counted twice")
+          pages = [r.get("page") for r in rows]
+          if pages != sorted(pages):
+              fail.append("rows are not in page order")
+          # Every row is built only when both ids are found, so these are
+          # 100%; the names come from the same cells.
+          for field, floor in (("sku", 1.0), ("player_id", 1.0), ("url", 1.0),
+                               ("title", 0.95), ("from_club", 0.9),
+                               ("fee_type", 1.0)):
+              got = sum(1 for r in rows if r.get(field) not in (None, ""))
+              if rows and got / len(rows) < floor:
+                  fail.append(f"{field} present on only {got}/{len(rows)} rows "
+                              f"(floor {floor:.0%})")
+
+          for line in fail:
+              print(f"::error::{line}")
+          print(f"[canary transfers] {len(rows)} rows, "
+                f"status={meta.get('status')} stop_reason={meta.get('stop_reason')} "
+                f"pages={meta.get('pages_completed')}/3")
+          sys.exit(1 if fail else 0)
+          EOF
+
       - name: Upload run artifacts (output, HTML snapshot, debug dumps)
         # Always, not just on failure: a run that returns the WRONG numbers
         # with no error is exactly the regression that needs the actual bytes
@@ -227,5 +307,9 @@ jobs:
             canary_run.meta.json
             canary_run.html*
             canary_run_page*_debug.*
+            canary_transfers.json
+            canary_transfers.meta.json
+            canary_transfers.html*
+            canary_transfers_page*_debug.*
           if-no-files-found: ignore
           retention-days: 14

CHANGELOG.md

@@ -37,8 +37,55 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 > - **The canary fails when it is blocked.** A bare-runner exit 3 used to
 >   end green with a notice.
 
+> **Behaviour change (latest transfers):** a page whose rows were all seen on
+> earlier pages is no longer counted. When the page before it had no next
+> link the run ends `stop_reason: listing_ended` (exit 0, was
+> `no_new_products`) with `repeated_page`, `repeat_of_page` and
+> `listing_pages` in the sidecar; when the page before it linked onward the
+> run ends `stop_reason: page_repeated`, exit 6 (was exit 0, "complete").
+> `pages_completed` no longer counts the repeated page.
+>
+> **Behaviour change (server errors):** an HTTP 5xx from Transfermarkt, or
+> the edge's bare nginx 5xx page where no status is visible, is a server
+> error, not a block. The page is fetched again on the same exit up to 2
+> times (`--retry-delay` x1, x2, no proxy rotation, no paid solve), then
+> failed with `stop_reason: target_http_5xx`: exit 5 on page 1, exit 6 on a
+> later page. A visible 5xx used to be exit 3 ("blocked"), and the bare 502
+> page used to read as an empty page.
+
+### Added
+
+- **`--mode transfers --min-value EUR`** (all four entry points) reads the
+  latest-transfers listing filtered to players valued at least EUR (0 = no
+  floor) through its filtered path,
+  `/transfers/neuestetransfers/statistik?...&minMarktwert=EUR`, keeping any
+  filters a typed `--url` already carries. Without it, the site's own default
+  view is read, as before. A negative or non-integer value, or the flag with
+  another `--mode`, is exit 2. The sidecar records `min_market_value_eur`
+  and `market_value_filter` (`url`, or `site_default` when the floor is the
+  site's own and not in the URL).
+
 ### Fixed
 
+- **`--mode transfers` counted a page the site did not serve.** Measured
+  2026-10-15: the default latest-transfers view held 20 transfers (the site
+  filters it to players valued EUR 500,000+), had no pager, and answered
+  `?page=2` with page 1 again. The run fetched page 2, found nothing new,
+  and reported `pages_completed: 2`, `status: complete`. A page number past
+  the end of a longer listing is answered the same way, with the last page
+  again. A page that only repeats earlier rows is now recorded as the
+  listing's end (when the page before it had no next link) or as a page not
+  served (when it had one), in all four entry points and in Playwright's
+  `--concurrency` path, which also stops dispatching once a page repeats.
+  The README documents the filter; `--min-value 0` reads the unfiltered
+  listing (4 pages, 77 transfers that day).
+- **A 502 after the AWS WAF challenge ended a run "complete" with page 1
+  missing.** Measured 2026-10-15: the challenge cleared and the document left
+  in the browser was the edge's 155-byte nginx `502 Bad Gateway`. Its status
+  had rightly been discarded with the challenge's, so it classified as an
+  empty page, and the run carried on to pages 2-5 and reported 52 of 77
+  rows as complete. The page is now recognised from its body as a server
+  error (see above).
 - **A failed write destroyed the previous good output.** `write_json`,
   `write_csv`, the `.meta.json` sidecar and `diff_runs.py --out` all opened
   their target with `open(path, "w")`, which truncates first: a dump that

README.md

@@ -49,9 +49,14 @@ python3 playwright_scraper.py --mode market-values
 # A club's full squad — id is transfermarkt's own numeric club id
 python3 playwright_scraper.py --mode club-squad --club-id 281 --season 2025
 
-# The latest-transfers ledger, 3 pages (safe to run with --concurrency)
+# The latest-transfers ledger, up to 3 pages (safe to run with --concurrency).
+# The default view is filtered to players valued EUR 500k+ and can fit on one
+# page; see "How deep that listing goes" below.
 python3 playwright_scraper.py --mode transfers --pages 3 --concurrency 3
 
+# The same listing with no market-value floor (0 = unfiltered)
+python3 playwright_scraper.py --mode transfers --min-value 0 --pages 6
+
 # One player's full profile
 python3 playwright_scraper.py --mode player --player-id 418560
 ```
@@ -293,11 +298,39 @@ uses whichever path page 1 says to use; `--concurrency` remains
 usually unaddressable, but because running several workers concurrently
 against a mode that can rarely switch shape mid-run is untested here.
 
-**`/statistik/neuestetransfers` was and remains simpler.** Its
-`<link rel="next">` is a bare `?page=2`, and page 2 fetched statelessly
-shared **0** transfer ids with page 1 (25 rows each, measured 2026-09-10) —
-genuinely, independently addressable on every capture taken of it so far.
-That is the one mode `--concurrency` is enabled for.
+**`/statistik/neuestetransfers` is simpler.** Its `<link rel="next">` is a
+bare `?page=2`, and page 2 fetched statelessly shared **0** transfer ids with
+page 1 (25 rows each, measured 2026-09-10). That is the one mode
+`--concurrency` is enabled for.
+
+**How deep that listing goes depends on the day, and on a filter you do not
+see.** Measured 2026-10-15:
+
+- The default view applies the site's own market-value floor: its filter
+  form and tab links carry `minMarktwert=500000` (players valued at
+  EUR 500,000 or more). That day it held **20** transfers, under one 25-row
+  page, so it had no pager and no next link. That is the whole listing, not
+  a cap on what this scraper reads.
+- Without the floor, the same listing had **4 pages, 77 transfers** (25, 25,
+  25, 2), each page's next link a plain `&page=N` with the filters kept.
+  `--min-value 0` reads it (`--mode transfers --min-value 0 --pages 6`);
+  `--min-value 250000` sets another floor. The flag builds
+  `/transfers/neuestetransfers/statistik?...&minMarktwert=EUR`, keeps any
+  filters a typed `--url` carries, and the sidecar records
+  `min_market_value_eur` (`null` with `market_value_filter: site_default`
+  when the run read the site's default view, whose floor is not in its URL).
+- A page number past the end is answered **HTTP 200 with the last page
+  again**: `?page=2` of the 20-row view served page 1, and `?page=5`,
+  `?page=401` and `?page=9999` of the unfiltered view all served page 4.
+
+So a page whose rows were all seen on earlier pages is never counted as a
+fresh page. When the page before it had no next link, the listing ended
+there: `stop_reason: listing_ended`, exit 0, and the sidecar names the
+`repeated_page`, the page it repeated (`repeat_of_page`) and how many pages
+the listing had (`listing_pages`). When the page before it DID link onward,
+the site promised a page and served an old one: `stop_reason:
+page_repeated`, exit 6. Either way `pages_completed` counts only the pages
+the site actually served.
 
 ---
 
@@ -578,13 +611,13 @@ push for the same four reasons.
 
 | Code | Meaning |
 |---|---|
-| `0` | Complete |
+| `0` | Complete (including a listing that ended before `--pages`: `stop_reason: listing_ended`) |
 | `1` | Crash |
 | `2` | Bad usage (including a URL that is not a supported Transfermarkt host) |
-| `3` | Blocked — a bot-check/challenge page |
+| `3` | Blocked — a bot-check/challenge page or a refusal. Never a server error: an HTTP 5xx is retried and then reported as `target_http_5xx` (exit 5 or 6) |
 | `4` | Zero rows, and the site's answer really was nothing: an empty squad, or the club, player or URL asked for does not exist (the site's own 404 page, or a profile id answered with a redirect elsewhere; `stop_reason: not_found`) |
-| `5` | Nothing obtained. Either the content was never fetched or read — a load timeout, a dead exit, or a served page that shows rows the parser could not read (`stop_reason: parser_found_nothing`, a markup change) — or a 2Captcha product call failed on its own terms (the Fingerprint API, or a `--cdp-endpoint` connection: bad key, rate limit, a Scraping Browser profile already in use). Distinct from `3` and `4`: nothing can be concluded about the site from it. A captcha-SOLVE failure is deliberately NOT this — it's a warning and the run continues, reporting `3` only if the page really was blocked. |
-| `6` | Partial — some pages fetched, then stopped early (including a later page that was refused, answered 404, or parsed to nothing while showing rows). Not necessarily a fault: for `--mode market-values`, this is the expected result on the rare run whose pagination link needs the fallback chain (see "the gallery pagination trap") and runs out of it; on the common run it does not happen at all. |
+| `5` | Nothing obtained. Either the content was never fetched or read — a load timeout, a dead exit, a server error from Transfermarkt that outlasted 2 retries (`stop_reason: target_http_5xx`), or a served page that shows rows the parser could not read (`stop_reason: parser_found_nothing`, a markup change) — or a 2Captcha product call failed on its own terms (the Fingerprint API, or a `--cdp-endpoint` connection: bad key, rate limit, a Scraping Browser profile already in use). Distinct from `3` and `4`: nothing can be concluded about the site from it. A captcha-SOLVE failure is deliberately NOT this — it's a warning and the run continues, reporting `3` only if the page really was blocked. |
+| `6` | Partial — some pages fetched, then stopped early (including a later page that was refused, answered 404, parsed to nothing while showing rows, answered 5xx after 2 retries, or served an earlier page again although the page before it linked onward: `stop_reason: page_repeated`). Not necessarily a fault: for `--mode market-values`, this is the expected result on the rare run whose pagination link needs the fallback chain (see "the gallery pagination trap") and runs out of it; on the common run it does not happen at all. |
 
 **A run that finds nothing writes nothing.** Last known good output is not
 replaced with `[]`; `--allow-empty` is the opt-out. `<out>.meta.json`

TROUBLESHOOTING.md

@@ -40,10 +40,32 @@ squad data yet answers 200 with a page that has no player rows on it, which
 is a correct "empty" answer, not a bug.
 
 **`--mode market-values` / `--mode transfers`** past page 1 — you may be one
-page past the end of the ranking/listing. Both modes stop and report a
-`no_new_products`/exhausted result rather than erroring, which is a
-*complete* answer, not a truncated one — check `stop_reason` in the
-`.meta.json` sidecar.
+page past the end of the ranking/listing. The site answers a page number past
+the end with the last page again (HTTP 200), so a page that brings no new
+rows is not counted. If the page before it had no next link, that is the end
+of the listing (`stop_reason: listing_ended`, exit 0, with `repeated_page`
+and `listing_pages` in the `.meta.json` sidecar). An empty page ends the run
+as `no_new_products`, also complete.
+
+**`--mode transfers` stops after one page of about 20 rows.** On 2026-10-15
+the default latest-transfers view (filtered by the site to players valued
+EUR 500,000+) held 20 transfers: one page, no pager. That is the site's whole
+listing for that filter. `--min-value 0` reads the unfiltered listing (4 pages
+that day); see "How deep that listing goes" in the README.
+
+## Exit 5 or 6 with `stop_reason: target_http_5xx`
+
+Transfermarkt answered with a server error (an HTTP 5xx, or the edge's
+155-byte nginx `502 Bad Gateway` page) and still did after 2 more tries on
+the same exit, `--retry-delay` and then twice that apart. Seen intermittently
+on 2026-10-15, one page at a time. It is not a block and not an empty
+result; run again later.
+
+## Exit 6 with `stop_reason: page_repeated`
+
+A page linked to a next page, and the next page served rows already seen.
+Pages were requested and not served, so the run is partial; the rows from
+the pages that were served are kept.
 
 ## Exit 5 with `stop_reason: parser_found_nothing`
 

output_writer.py

@@ -461,7 +461,7 @@ def save(rows: Sequence[Any], out_prefix: str, fmt: str,
 # exist (its own 404 page, or a profile redirected away): a complete answer
 # with no rows, so exit 4, not a failed fetch.
 COMPLETE_STOP_REASONS = ("completed", "pagination_exhausted", "no_new_products",
-                         "single_page_mode", "not_found")
+                         "single_page_mode", "not_found", "listing_ended")
 
 
 def finish_run(rows: Sequence[Any], out_prefix: str, fmt: str,