Skip to content

Empty searches exit 4, partial concurrent runs say so, one paid solve per page #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jehrr merged 1 commit into from
Oct 15, 2026
Merged

Empty searches exit 4, partial concurrent runs say so, one paid solve per page #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ TWOCAPTCHA_KEY=your_2captcha_api_key_here
# cookies and ONE live connection at a time, so reuse pids rather than
# minting one per run.
#
# Profile credentials expire after about a day, so never paste a working one
# into a file you keep — fetch a fresh endpoint when you need it.
# Profile credentials expire (a 401 is the signal), so never paste a working
# one into a file you keep — fetch a fresh endpoint when you need it.
# NOT supported by selenium_scraper.py: chromedriver's debuggerAddress takes
# a bare host:port with nowhere to put a password.
INDIEGOGO_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-{cc}-pid-{profileId}:{password}@cb.2captcha.com:9222
Expand Down
46 changes: 21 additions & 25 deletions .github/ISSUE_TEMPLATE/bug_report.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,25 +12,21 @@ body:

A few things answer most reports faster than we can:

- **Exit 3 (blocked)?** This repo's own research found no captcha,
proxy or browser challenge anywhere on this site from a plain
datacentre fetch (see `scraper_api_client.py`'s docstring and
`.env.example`) — so exit 3 here usually means something upstream
really changed, not that a proxy or `--twocaptcha-key` is missing.
Check `product_parser.detect_bot_challenge()` / `BOT_CHALLENGE_MARKERS`
against what the page actually returned.
- **Zero rows on `--mode search --pages N > 1`?** Check
whether this is the rare "gallery pagination trap" case in the
README first — a fallback chain exists for it, and running out of
it is an expected `exit 6` (partial), not a bug.
- **Wrong club on a `--mode campaign` row?** Indiegogo itself
shows a loaned player's *parent* club on the squad page under some
conditions — check the page's own title and canonical URL before
assuming the parser mis-attributed it.
- **`--mode campaign` or `--mode campaign` gave a 404 / wrong page?**
Double check the `--club-id`/`--season` or `--player-id` against
the numeric id in a indiegogo.com URL — a slug typo silently
resolves to a different profile.
- **Exit 3 (blocked)?** Every indiegogo.com page is behind a
Cloudflare managed challenge. A real browser usually clears it by
itself; if it did not, say whether the run was headless, which
address it came from (datacentre or residential), and whether a
`--twocaptcha-key` was set. Check `product_parser.BOT_CHALLENGE_MARKERS`
against what the page actually returned (`--dump-html`).
- **Exit 4?** The site itself answered "nothing matched"
(`search-count:0`) — check the search URL's own filters first.
- **Exit 5?** The page was never obtained (navigation failure, dead
proxy, remote API error), or a served page parsed to nothing. The
log line before the exit says which.
- **`--mode campaign` / `--mode rewards` gave a 404?** Some search
results are campaigns hosted on gamefound.com (`platform` column);
their URLs answer 404 on indiegogo.com. That is the site, not the
parser.

- type: dropdown
id: engine
Expand All @@ -53,8 +49,7 @@ body:
options:
- search
- campaign
- transfers
- player
- rewards
- not applicable
validations:
required: true
Expand All @@ -77,8 +72,9 @@ body:
label: Exit code
description: >-
`echo $?` straight after the run. These mean specific things — 3 is a
challenge page, 4 is "ran fine, found nothing", 6 is a partial run.
The table is in the README.
challenge page, 4 is "the site answered and nothing matched", 5 is
"the content was never obtained", 6 is a partial run. The table is
in the README.
placeholder: "4"
validations:
required: true
Expand All @@ -91,7 +87,7 @@ body:
render: shell
placeholder: |
python3 playwright_scraper.py \
--mode search --pages 3 --out market_values
--mode search --pages 3 --out projects
validations:
required: true

Expand All @@ -111,7 +107,7 @@ body:
attributes:
label: What you expected instead
placeholder: >-
25 rows per page, as the README says a search page yields.
24 rows per page, as the README says a search page yields.
validations:
required: true

Expand Down
34 changes: 16 additions & 18 deletions .github/ISSUE_TEMPLATE/site_changed.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ body:
own report because the fix is different from a code bug: something on
indiegogo.com moved.

Useful to know before filing: unlike some other scrapers in this family,
**this site has no JSON-LD anywhere** — `product_parser.py` anchors on
URL patterns (`/profil/spieler/{id}`, `/verein/{id}`,
`/jumplist/transfers/...`), the site's own CSS class names
(`table.items`, `posrela`, `hauptlink`, `data-header__label`), and
`schema.org/Person` microdata on player-profile pages only. Which of
those broke narrows the fix a lot.
Useful to know before filing: **this site has no JSON-LD anywhere.**
`--mode search` reads the results payload inlined in the search page
and the site's own `searchProjectsForCards` POST for later pages;
`--mode campaign` reads `window.__INITIAL_STATE__`; `--mode rewards`
reads the reward boxes by their `data-qa` attributes. The rendered
cards (`data-qa` hooks) are the fallback for a missing search
payload. Which of those broke narrows the fix a lot.

- type: dropdown
id: mode
Expand All @@ -24,8 +24,7 @@ body:
options:
- search
- campaign
- transfers
- player
- rewards
- not sure / more than one
validations:
required: true
Expand All @@ -36,11 +35,11 @@ body:
label: What stopped matching
options:
- Rows (0 rows on a page that visibly has data)
- A specific field is null or wrong (price, club, position, fee, ...)
- A specific field is null or wrong (price, backers, currency, campaign_end, ...)
- Price coverage dropped below the floor logged in the run output
- Pagination stops early or repeats a page
- The bot-challenge detector finds nothing on a page that has one
- schema.org/Person microdata missing on a player profile
- The inlined search payload or __INITIAL_STATE__ is missing
- Something else
validations:
required: true
Expand All @@ -50,7 +49,7 @@ body:
attributes:
label: URL
description: The exact page URL you ran against.
placeholder: "https://www.indiegogo.com/spieler-statistik/wertvollstespieler/marktwertetop"
placeholder: "https://www.indiegogo.com/en/projects/search?projectCatalogCategories=BoardAndCardGames"
validations:
required: true

Expand All @@ -61,13 +60,12 @@ body:
description: |
Whichever of these you can get. A capture beats a description.

- The relevant table row's HTML (right-click the row on the real site
-> Copy -> Copy outerHTML), or the output of
`--dump-html capture.html` against this URL.
- The relevant card's or reward box's HTML (right-click it on the
real site -> Copy -> Copy outerHTML), or the snapshot written by
`--dump-html` against this URL.
- For a field problem: the value you got, and the value on the page.
- For a player-profile problem: whether the page has a
`itemtype="https://schema.org/Person"` block at all (view source
and search for it).
- For a campaign problem: whether the page source still contains
`window.__INITIAL_STATE__` at all (view source and search for it).
render: text
validations:
required: true
Expand Down
19 changes: 16 additions & 3 deletions .github/ci_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,10 @@
HEX32 = re.compile(r"\b[0-9a-f]{32}\b")
HEX32_ALLOWED = ("sha", "hash", "nonce", "example", "md5", "digest", "checksum")

SCANNED_SUFFIXES = (".py", ".md", ".txt", ".yml", ".yaml", ".example")
# `.json`/`.csv` too: the sample outputs and any fixture are captured page
# data, which is exactly where a front-end key would arrive.
SCANNED_SUFFIXES = (".py", ".md", ".txt", ".yml", ".yaml", ".example",
".json", ".csv", ".toml", ".cfg", ".sh")


def git_ignored(paths):
Expand Down Expand Up @@ -125,7 +128,7 @@ def git_ignored(paths):
# into pip's vendored code and flags a 32-hex string in `_elffile.py` as
# key-shaped. Correct about the string, wrong about the file — and it is the
# FIRST thing a new user sees from `python3 smoke_test.py`. A guard people
# have to argue with is one they learn to suppress (CLAUDE.md §22).
# have to argue with is one they learn to suppress.
#
# Deliberately NOT a narrowing of what gets scanned: an untracked file is
# still read, because a key pasted into a scratch file beside the scripts is
Expand Down Expand Up @@ -157,8 +160,13 @@ def scanned_files():
for path in sorted(REPO.rglob("*")):
if not path.is_file() or path.suffix not in SCANNED_SUFFIXES:
continue
# RELATIVE parts only. `path.parts` includes every ancestor of the
# clone, so a checkout that happened to live under a directory named
# `venv` (or `.venv`) matched here for every file and the scan read
# nothing at all -- and then printed "0 files scanned, nothing
# credential-shaped", which reads as a pass.
if any(part in {".git", "__pycache__", ".venv", "venv"}
for part in path.parts):
for part in path.relative_to(REPO).parts):
continue
candidates.append(path)

Expand Down Expand Up @@ -274,6 +282,11 @@ def secret_check():
failed.append(f"{rel}:{lineno} contains {match[:6]}… — a "
f"32-char hex string, the shape of a 2captcha key")

if scanned == 0:
# A scan that read nothing has proved nothing; say so as a failure
# rather than as "clean".
failed.append("0 files scanned -- the credential scan read nothing, "
"so it cannot vouch for this tree")
if not failed:
print(f"ok {scanned} files scanned, nothing credential-shaped")
return failed
Expand Down
49 changes: 19 additions & 30 deletions .github/workflows/canary.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,22 +21,20 @@ name: canary
# themselves — 0 solves attempted, and the 2Captcha balance was unchanged to
# five decimal places before and after.
#
# Whether a GITHUB runner's address behaves the same way is a separate
# question, and it is not one this repo can answer in advance. So:
# A bare GitHub runner behaves the same way: the scheduled runs of
# 2026-10-06 to 2026-10-14 all fetched 72 rows across 3/3 pages from a runner
# with no secret configured. So this canary needs NO secret, and every
# outcome that is not real rows is RED:
#
# * If INDIEGOGO_CDP_ENDPOINT or INDIEGOGO_PROXY is configured, the run
# goes through it and being blocked anyway is a REAL failure worth a red
# check.
# goes through it.
# * If neither is configured, the run uses a bare runner and a local
# Playwright browser — the configuration the README's central claim is
# about, and the one that catches a markup change. But a block from a
# bare runner is a fact about that runner's address, not about the site,
# so it SKIPS with a ::notice:: rather than failing. A check that goes
# red for a reason nobody in this repo can fix is a check everyone
# learns to ignore.
# Playwright browser — the configuration the README's central claim
# ("no key, no proxy") is about. A block there (exit 3) is a FAILURE,
# not a skip: that claim is exactly what this job exists to test, and a
# canary that goes green without reaching the site tests nothing.
#
# Every other exit code is still interpreted and still turns this red, so a
# markup change cannot hide behind the skip.
# Exit 3, 4, 5 and 6, and fewer rows than the floor, all turn this red.
#
# Secrets reach the scraper through the ENVIRONMENT, never a command line —
# `ps` reads argv. env_config.py reads exactly these variable names, and an
Expand Down Expand Up @@ -78,7 +76,7 @@ jobs:
elif [ "$HAVE_PROXY" = "true" ]; then
echo "::notice::Running through INDIEGOGO_PROXY. A block here is a real failure."
else
echo "::notice::Running from a bare runner with a local Playwright browser and no secret. This catches a markup change; if the runner's address meets the challenge, this job SKIPS rather than failing."
echo "::notice::Running from a bare runner with a local Playwright browser and no secret -- the README's no-key, no-proxy path. Anything but real rows fails this job."
fi

- name: Scrape a real, known-good search listing
Expand Down Expand Up @@ -115,35 +113,26 @@ jobs:
CODE: ${{ steps.run.outputs.exit_code }}
run: |
code="$CODE"
skipped=false
case "$code" in
0) echo "OK — rows parsed from a live run." ;;
3) if [ "$HAVE_CDP" = "true" ] || [ "$HAVE_PROXY" = "true" ]; then
echo "::error::Blocked before parsing (exit 3) while running through a configured exit (cdp=$HAVE_CDP proxy=$HAVE_PROXY). That is the documented default path failing, not the runner's address. Check the zone or proxy credentials first — a Scraping Browser zone that refuses CONNECT by host reports ERR_TUNNEL_CONNECTION_FAILED and is an account matter, not a scraper bug — then see detect_bot_challenge()/BOT_CHALLENGE_MARKERS in product_parser.py."
echo "::error::Blocked before parsing (exit 3) while running through a configured exit (cdp=$HAVE_CDP proxy=$HAVE_PROXY). Check the zone or proxy credentials first, then BOT_CHALLENGE_MARKERS in product_parser.py against the HTML dump in the artifacts."
else
# No secret configured, so this ran from a bare runner. A
# block here is a fact about that runner's address on the
# day: 8 of 10 requests from one datacentre address were
# challenged on 2026-09-16, and none at all from six other
# network positions later the same day. Nothing in this
# repo can fix it, so it must not paint the check red.
echo "::notice::SKIPPED: this runner's address was challenged before the page was parsed (exit 3), and no INDIEGOGO_CDP_ENDPOINT or INDIEGOGO_PROXY secret is configured to route around it. That is a property of the runner's address, not of the site or of this scraper, so the canary is skipping rather than failing. Set either secret to have this run go through an exit that is not challenged -- and to make a block here a real failure."
skipped=true
code=0
# Not a skip. The README says a plain local browser reads
# this site with no key and no proxy, and bare runners have
# been served every day this was checked; a block here is
# that claim failing, which is what this job is for.
echo "::error::Blocked before parsing (exit 3) from a bare runner with no secret. The README's central claim is that a local browser needs no key and no proxy here, so this is that claim failing. Check BOT_CHALLENGE_MARKERS against the HTML dump in the artifacts (a marker that now also matches GOOD pages reads every page as blocked), then re-run once: a single runner address can be challenged on a given day. If it persists, configure INDIEGOGO_PROXY or INDIEGOGO_CDP_ENDPOINT and update the README."
fi ;;
4) echo "::error::Ran fine but parsed 0 rows (exit 4) on a search the site reports 10,000 matches for. Most likely the inlined results payload (App.Components.Search.SearchProjectsResults) moved or was renamed — check extract_inline_search_result() in product_parser.py against the HTML dump in the artifacts, which is what the parser actually saw." ;;
5) echo "::error::Remote API error (exit 5) — a 2captcha product path failed. With a secret configured (cdp=$HAVE_CDP proxy=$HAVE_PROXY) that is the endpoint or the credentials; with neither, nothing should have called a remote API at all and the run log in the artifacts is where to start." ;;
4) echo "::error::Exit 4: the site answered this unfiltered search with its own 'nothing matched' (search-count:0), on a query it normally reports 10,000 matches for. Check the HTML dump in the artifacts and detect_page_state() in product_parser.py." ;;
5) echo "::error::Exit 5: the content was never obtained. Either the navigation itself failed (a load timeout, a dead proxy, or with a secret configured (cdp=$HAVE_CDP proxy=$HAVE_PROXY) the endpoint or its credentials), or the page was served and parsed to nothing (stop_reason=parser_found_nothing): most likely the inlined results payload (App.Components.Search.SearchProjectsResults) moved or was renamed -- check extract_inline_search_result() against the HTML dump in the artifacts. The run log names which." ;;
6) echo "::error::Partial run (exit 6) — some pages were gathered and then the run stopped early. canary_run.meta.json names which pages failed. On this site the usual cause is the searchProjectsForCards POST being refused mid-run (the session's Cloudflare clearance expiring under it), which shows up as stop_reason=page_failed." ;;
2) echo "::error::Bad usage (exit 2) — the workflow's own arguments are wrong, not the site." ;;
*) echo "::error::Unexpected exit code $code (see the step above for the traceback)." ;;
esac
echo "skipped=$skipped" >> "$GITHUB_OUTPUT"
test "$code" = "0"

- name: Sanity-check the scraped data
# Nothing was fetched on the skip path, so there is nothing to check
# and canary_run.json does not exist.
if: steps.verdict.outputs.skipped != 'true'
# Exit 0 only means "some rows were written". It says nothing about
# whether the page actually rendered: a snapshot taken too early still
# writes rows, just with fields empty. Every floor below is set from a
Expand Down
Loading
Loading