[jehrr]

First release of binance-scraper: three modes over binance.com's own JSON endpoints, three browser engines over one shared fetch loop, and the Scraper API for the one mode it can reach.

main holds only the scaffold commit (the family core copied from bbb-scraper), which a new repo needs as its default branch before workflows register. Everything else is in this PR.

What it reads

--mode	one row per	per page	endpoint
p2p	advert	20	POST /bapi/c2c/v2/friendly/c2c/adv/search
copytrading	lead portfolio	30	POST /bapi/futures/v1/friendly/future/copy-trade/home-page/query-list
announcements	article	50	GET /bapi/composite/v1/public/cms/article/list/query

Every HTML page on the site is behind AWS WAF (HTTP 202 challenge to curl; the CAPTCHA to real Chromium on announcement pages). The three endpoints answer a datacentre address normally. The engines land on a GET endpoint and issue each page as a same-origin fetch().

Evidence (all 2026-09-24, datacentre VPS)

• Live, three engines: Playwright, pyppeteer and Selenium each ran the same 7 scenarios (P2P buy and sell, copy-trading, announcements, --concurrency, a refused --pay-type, --url) with identical results. A full USDT/EUR book came back as 207 of 207 adverts.
• Live, Scraping Browser (country-de): Playwright and pyppeteer, all modes, 8 of 8 runs back to back. Selenium refuses a credentialled endpoint with exit 2.
• Live, AWS WAF solve: AmazonTask on a gated announcement page took 23-40 s and cost $0.00145, and the page was served. The control (no solve) cleared 0 of 5 pages. The cookie that works is existing_token on .binance.com; the voucher on the page's own host did not work.
• Live, Scraper API: announcements only, 100 rows, $0.0005 a page. The POST modes are not implemented there, and the README says so.
• Offline: 346 checks over real, scrubbed API responses (fixtures_generated.json, cut by make_fixtures.py, which proves each fixture parses like its original). They include an end-to-end run of the shared loop with a fake browser. 7 controlled faults each turned the suite red on the expected check.
• Fresh copy: README followed verbatim with the venv inside the tree. The suite passed and the first README command ran.
• Canary: its own Run and Assert steps, extracted from canary.yml, passed locally for all three modes.

Traps handled

• The API accepts wrong values silently. An unknown copy-trading dataType gives some other ordering, pageSize above 30 is capped at 30, and an unknown payTypes entry returns an empty list. Every parameter is therefore allowlisted, and --pay-type is checked against the site's own list.
• The P2P side is inverted in the data. A buy query returns adverts marked SELL. Rows keep both sides.
• HTTP 400 with an empty body is a refused parameter, not a block. It is classified rejected and exits 5, with no retry.
• Scraping Browser profiles stay profile_locked briefly. Release takes 1.6-1.9 s after a clean disconnect, so connects retry. A 401 is explained as expired credentials.

Family findings, not changed here

• tests.yml's engine-smoke step greps for "could not be imported", which 25 of 27 sibling suites never print, so that guard cannot fail there. Fixed in this repo only. The count is a text search over local checkouts.
• transfermarkt-scraper sets the AWS WAF voucher as the cookie on the page host, which did not clear a page here.
• The local bbb-scraper checkout was one commit behind its main (the Scraper API waitFor fix); that file was taken from the remote.

Not done

• The Docker image has not been built locally (no daemon access); this PR's CI builds it.
• The canary has not been dispatched from GitHub. What binance.com answers a US runner with is still unmeasured.
• The repo page (topics) and the org profile row wait until the repo goes public.

🤖 Generated with Claude Code