Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
.git/
.github/
.env
.env*
*.log
*.key
secrets*
Expand Down
17 changes: 8 additions & 9 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,8 @@ TWOCAPTCHA_KEY=your_2captcha_api_key_here
# A remote browser you do not run or patch, with a persistent profile and a
# chosen exit country. Used by --cdp-endpoint.
#
# country- picks the exit country. Pick one Binance serves: its terms
# exclude the United States (binance.us is a separate exchange).
# What a US exit is answered with has not been measured here.
# country- picks the exit country. binance.us is a separate exchange
# this scraper does not read.
# pid- a profile with persistent cookies. ONE live connection each,
# capped per account. Reuse pids rather than minting one per
# run, and expect a 500 (`profile_locked`, exit 5) while another
Expand All @@ -62,17 +61,15 @@ TWOCAPTCHA_KEY=your_2captcha_api_key_here
# Selenium cannot use this: chromedriver's `debuggerAddress` takes a bare
# host:port with nowhere to put a password. Playwright and pyppeteer can.
#
# The credentials in a profile expire in about a day, so never paste a
# working one anywhere that is committed.
# The credentials in a profile expire after a while (a 401 says so), so
# never paste a working one anywhere that is committed.
BINANCE_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-de-pid-{profileId}:{password}@cb.2captcha.com:9222

# --- Proxy ----------------------------------------------------------------
# One URL here, or --proxy-file with one per line for a pool. A pool is what
# spreads a run's volume, and --concurrency without one just sends N times
# the request rate from a single address.
#
# Region: anywhere Binance serves (not the United States, per its terms).
#
# The credentials never reach the browser's command line: they go through
# the driver's own fields, and logs mask them while keeping host and port.
# Selenium's --proxy-server cannot authenticate at all, so that engine
Expand All @@ -88,8 +85,10 @@ BINANCE_PROXY=http://{user}:{password}@eu.proxy.2captcha.com:2334
# https://www.binance.com/en/support/announcement/list/48 announcements
#
# The page itself is never fetched, only the query in its address. The
# flags (--asset, --fiat, --mode ...) do the same without a URL.
BINANCE_URL=https://p2p.binance.com/en/trade/all-payments/USDT?fiat=EUR
# flags (--asset, --fiat, --mode ...) do the same without a URL, and a flag
# you TYPE wins over this variable: it is only a default for a run given
# neither. Left commented out so a copied example changes nothing.
# BINANCE_URL=https://p2p.binance.com/en/trade/all-payments/USDT?fiat=EUR

# Deliberately NOT here: an output prefix. `--out` already carries a
# non-empty default, so the loader, which only fills UNSET values, would
Expand Down
8 changes: 3 additions & 5 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,9 @@ body:

A few things answer most reports faster than we can:

- **Exit 3 from a US address?** Binance's terms exclude some
jurisdictions, the United States among them (binance.us is a
separate exchange). Try an exit elsewhere, and say in the report
what the sidecar's `stop_reason` was: this repo has not measured
what such an address is answered with.
- **Exit 3 with `blocked_geo-451`?** This has not been observed on
binance.com in testing. Say in the report which country the exit
was in.
- **Exit 3 with `blocked_aws-waf`?** The site's AWS WAF challenged
the session. The data endpoints were not gated when this repo was
written, so this is news worth reporting, together with the exit
Expand Down
2 changes: 1 addition & 1 deletion .github/ci_checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,7 @@ def tracked_files():
# into pip's vendored code and flags a 32-hex string in `_elffile.py` as
# key-shaped. Correct about the string, wrong about the file — and it is the
# FIRST thing a new user sees from `python3 smoke_test.py`. A guard people
# have to argue with is one they learn to suppress (CLAUDE.md §22).
# have to argue with is one they learn to suppress.
#
# Deliberately NOT a narrowing of what gets scanned: an untracked file is
# still read, because a key pasted into a scratch file beside the scripts is
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
#
# No secrets are needed, and that is the point of running it: the README's
# central claim is "no key, no proxy, no account", and an ungated scheduled
# canary is that sentence under test every morning (CLAUDE.md §21, §24). All
# canary is that sentence under test every morning. All
# three data endpoints answered a datacentre address normally when this repo
# was written (2026-09-24); a GitHub runner is a datacentre address too.
#
Expand Down Expand Up @@ -78,7 +78,7 @@ jobs:
playwright install --with-deps chromium
fi

# >= 3 pages, because with one page pagination is never exercised (§11).
# >= 3 pages, because with one page pagination is never exercised.
- name: Run
id: run
run: |
Expand Down
9 changes: 6 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ binance_rows_*.*
live_results/
live_results.zip
# ANY working directory a live run writes into, by KIND rather than by the one
# name an older repo used (CLAUDE.md §24). Measured here: `live/` was not
# name an older repo used. Measured here: `live/` was not
# listed, and a `--dump-html` of a 3-page run wrote `canary_dump.json.page3`,
# which `*.json` does not cover. It sat untracked and NOT ignored, one
# `git add -A` from a commit, until the credential scan flagged the 32-hex
Expand All @@ -48,6 +48,9 @@ dumps/
*.html.page*
*.json.page*
*.page[0-9]*
# A capture saved by hand at the repo root (`--dump-html dump.html`). The
# repo ships no HTML of its own.
/*.html

# --- captures kept for parser work ---------------------------------------
# The smoke suite's fixtures are committed (trimmed, in fixtures_generated.json);
Expand Down Expand Up @@ -97,7 +100,7 @@ secrets*
# recoverable only because that repo was private at the time. On 2026-09-17
# a sibling had an untracked `.env.bak` holding a 32-hex key in a PUBLIC
# repository, one `git add -A` from publication. Craigslist was fixed then
# and the fix never reached anyone else, which is §16's rule unapplied: when
# you fix one, check the siblings.
# and the fix never reached anyone else. When you fix one, check the
# siblings.
.env*
!.env.example
54 changes: 54 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,63 @@ said so at the top of its release notes.
> exit 3 or 4 with a 39-byte "capture", in tens of milliseconds. If a
> script of yours passes 0, pass **1**: the flag counts ATTEMPTS, so 1 means
> "try once and do not retry".
>
> **Behaviour change:** a page now buys at most one CAPTCHA solve in total,
> across every block-retry and exit rotation. It used to be one per
> LANDING, so a refused page could buy 2 AWS WAF solves with no proxy pool
> and 4 with a 3-exit pool. A page whose CAPTCHA survives that one solve is
> reported blocked (exit 3) sooner and for less.
>
> **Behaviour change:** CSV cells that start with `=`, `+`, `-`, `@`, a
> tab or a line break now carry a leading apostrophe, so a spreadsheet
> reads them as text. JSON is unchanged; the sidecar's `csv_cells_escaped`
> says how many cells differ.
>
> **Behaviour change:** `BINANCE_URL` from `.env` or the environment is a
> default only. A `--mode` or a query flag typed on the command line now
> wins over it instead of being refused as a conflict. `.env.example`
> ships it commented out.

### Fixed

- **One refused page could buy several CAPTCHA solves.** The allowance
lived inside each landing, and every block-retry or rotation landed
again. With the solver stubbed: 2 solves with no pool, 4 with a 3-exit
pool, against `SOLVES_PER_PAGE = 1`. The allowance is now one
`page_flow.SolveBudget` per page, checked before each purchase and never
reset by a retry or a rotation. The `--pay-type` pre-check no longer
solves at all. A suite check drives the shared loop with a solver that
always "succeeds" and counts purchases.
- **A failed write destroyed the previous good output.** `out.json`,
`out.csv` and `out.meta.json` were opened with truncation, so a failure
inside `json.dump` left 13 bytes of invalid JSON where a good file had
been. Each is now written to a temporary file in the same directory,
fsynced and renamed over the target. A new file gets the mode `open()`
would have given it (`0o666 & ~umask`), and an existing file keeps its
own. Each file is replaced atomically on its own; the three are not one
transaction.
- **CSV cells could run as spreadsheet formulas.** Advertiser nicknames,
portfolio names and announcement titles are third-party text. See the
behaviour change above.
- **A copied `.env.example` broke every flag-built query.** It set
`BINANCE_URL` to a P2P page, so `--mode announcements`,
`--mode copytrading` and `--asset/--fiat` all exited 2 with a conflict
against a URL nobody typed. The engines now record the typed `--url`
before `.env` is read.
- **Advice for HTTP 451 named a country nobody had measured.** The exit-3
message, README, TROUBLESHOOTING, CONTRIBUTING and the bug template said
a US exit is refused. 451 has not been observed on binance.com, and the
daily canary is served from a GitHub-hosted runner with no proxy. The
message now says so and asks for the exit's country in an issue.
- **A Scraping Browser 401 was explained as credentials that "last about
a day".** That figure was not measured (an endpoint has been seen working
after ten days). A 401 is now reported as refused credentials, expired or
mistyped.
- A capture saved at the repo root (`--dump-html dump.html`) is ignored by
git, and `.dockerignore` excludes every `.env*` file.
- Comments that pointed at private notes now state their reasons; a suite
check keeps it that way.

- **`--retries 0` fetched nothing and called it a result.** The load loop is
`range(1, retries + 1)`, so 0 never runs it and the engine judged
`about:blank`. Three third-party audits of this family ran exactly that and
Expand Down
11 changes: 5 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@ Then the rest of the presentation, in the order that matters:
three modes from a GitHub-hosted runner with no proxy. If a later run
is refused for the runner's address, the canary's `BINANCE_PROXY`
secret (an exit elsewhere) is the fix, with no workflow edit.
2. The repo description, homepage and topics set (see the family notes on
what those should say).
2. The repo description, homepage and topics set: the engines, the
2Captcha Scraping Browser API by that name, and what the site publishes.
3. Only then the row in the org profile README — and check it with an
ANONYMOUS request rather than your own logged-in browser. A row pointing
at a private repo is a 404 for every visitor, which costs more trust than
Expand Down Expand Up @@ -164,10 +164,9 @@ Two things about running this live that are specific to Binance:
* **No mode needs an exit, a key or an account.** The endpoints answered a
datacentre VPS normally, so "it worked from my laptop" is reproducible
here in a way it is not on most sibling repos.
* **Binance's terms exclude some jurisdictions, the United States among
them** (binance.us is a separate exchange). What an address there is
answered with has not been measured by this repo. If a run from one is
refused, that is the terms, not a bug.
* **HTTP 451 has not been observed.** The daily canary is served from a
GitHub-hosted runner with no proxy. If a run is refused with 451, record
the exit's country in the issue.

**Run more than the primary engine.** "Mirror them exactly" is a design
rule, not a verification. The fetch loop is shared (`page_flow.run_pages`),
Expand Down
11 changes: 5 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ copy-trading rows 26 columns and announcement rows 12 columns.

---

## Five things about Binance that will look like bugs
## Six things about Binance that will look like bugs

### 1. A buy query returns adverts marked "sell"

Expand Down Expand Up @@ -225,17 +225,16 @@ One key, four separately-billed products ([2captcha.com](https://2captcha.com)):
Selenium's Chrome was not shown the CAPTCHA at all on those pages (3 of 3),
so its copy of the solve path has not been exercised live.
* **Proxies** (`--proxy`, `--proxy-file`): volume from more than one
address, and an exit in a country Binance serves. Binance's terms exclude
some jurisdictions, the United States among them.
address, and an exit in a chosen country.
* **The Scraping Browser API** (`--cdp-endpoint`): a remote browser you do
not run, with a chosen exit country. One live connection per `pid`, so
`--concurrency` is refused with it. Run live on 2026-09-24 through a
`country-de` profile: Playwright and pyppeteer, all three modes, 8 of 8
runs complete, back to back. Two details measured on the way: a profile
stays locked for 1.6-1.9 s after a clean disconnect, so the engines retry
a `profile_locked` connection rather than failing on it; and an expired
profile answers HTTP 401, which the engines report as exit 5 naming the
expiry. Selenium refuses a credentialled endpoint with exit 2.
a `profile_locked` connection rather than failing on it; and a profile
whose credentials are refused (expired or mistyped) answers HTTP 401,
which the engines report as exit 5 with that reason. Selenium refuses a credentialled endpoint with exit 2.
* **Fingerprints** (`--fingerprint`): a consistent device identity for a
local browser. Ignored with `--cdp-endpoint`, which brings its own.

Expand Down
6 changes: 4 additions & 2 deletions TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,10 @@ The log names what refused the run:
* **`http-403`** — the site refused the address. Binance's API documentation
uses 403 for a WAF rule, including a rate-limit violation. Slow down
(`--delay`), or use a different exit.
* **`geo-451`** — the exit's country is refused. Binance's terms exclude some
jurisdictions, the United States among them.
* **`geo-451`** — HTTP 451, a refusal for legal reasons. It has not been
observed on binance.com in testing (the daily canary is served from a
GitHub-hosted runner with no proxy), so please open an issue with the
sidecar's `stop_reason` and the exit's country.

A `<out>_page<N>_debug.html` beside the output holds what came back.

Expand Down
12 changes: 6 additions & 6 deletions captcha_solver.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ class CaptchaChallenge:
page_url: str = ""
# AWS WAF only. Its task needs three values that all live together in
# `window.gokuProps` on the challenge page, plus the two script URLs the
# page loads. Unlike Cloudflare Turnstile (§19 of the family notes),
# page loads. Unlike Cloudflare Turnstile,
# nothing has to be intercepted at runtime to get them: AWS WAF prints
# all of it into the served HTML, so a static read is enough.
iv: Optional[str] = None
Expand Down Expand Up @@ -151,7 +151,7 @@ def aws_waf_action(self) -> str:
def has_captcha_widget(self) -> bool:
"""Is there a widget on this page for a solver to work on?

Section 19's rule: "unsolvable" is a property of a PAGE, not of a
"Unsolvable" is a property of a PAGE, not of a
vendor, and it means the page carries no widget. A challenge-action
page carries none, so sending it to a solver buys a token for a
puzzle that was never rendered -- measured at $0.00145 a time, since
Expand Down Expand Up @@ -268,8 +268,8 @@ def detect_recaptcha_v3(html: str, page_url: str) -> Optional[CaptchaChallenge]:
attributes (data-captcha-type="recaptcha" data-version="v3"
data-sitekey="..." data-action="..."), with the actual execute()
call happening inside a bundled JS file, never appearing as
readable inline script text at all — confirmed live on the sibling
farfetch-scraper repo's target,
readable inline script text at all — confirmed live on a retail
site's
sign-up modal via DevTools inspection. A previous version of this
function, which only checked for format 1, reported "no captcha"
on this exact page despite one being genuinely present — caught by
Expand Down Expand Up @@ -319,7 +319,7 @@ def detect_recaptcha_v3(html: str, page_url: str) -> Optional[CaptchaChallenge]:

# ---------------------------------------------------------------------------
# Runtime detection (added 2026-08-24 — the static-HTML detector below no
# longer matched the sibling farfetch-scraper repo's target)
# longer matched a retail site's sign-up modal)
# ---------------------------------------------------------------------------
#
# What changed there: the sign-up modal used to render
Expand Down Expand Up @@ -949,4 +949,4 @@ def solve_recaptcha(challenge: CaptchaChallenge, twocaptcha_api_key: Optional[st
# what the address has been doing — a narrow list is how a challenge gets
# reported as an empty page months later. A solver for a challenge this site
# has never been observed to serve is dead code; a DETECTOR for one is cheap
# insurance. See the family note in CLAUDE.md.
# insurance.
4 changes: 2 additions & 2 deletions env_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@
# A literal-only check reported both of those as CONFIGURED, so `cp
# .env.example .env` followed by a run connected to cb.2captcha.com with the
# string `{login}-zone-…` as its username and got a 401 — a confusing failure
# a long way from its cause, which is precisely what §3's rule exists to
# prevent. Any `{…}` left in a value means the line was never filled in.
# a long way from its cause, which is precisely what reading .env strictly
# exists to prevent. Any `{…}` left in a value means the line was never filled in.
_BRACED_PLACEHOLDER_RE = re.compile(r"\{[A-Za-z_][A-Za-z0-9_]*\}")

_loaded_from = None
Expand Down
8 changes: 4 additions & 4 deletions fingerprint_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -249,9 +249,9 @@ def playwright_context_kwargs(fp: dict) -> dict:
# stating `deviceScaleFactor: 1.25` produced a browser reporting
# `window.devicePixelRatio === 1`, so the identity contradicted itself
# on an axis any fingerprinter reads for free -- and a contradiction is
# exactly what this flag exists to avoid. Same shape as the three
# defects the family notes already list for this file: a key the API
# returns that nothing applies. Found by comparing a live browser
# exactly what this flag exists to avoid. Same shape as three
# earlier defects in this file: a key the API returns that nothing
# applies. Found by comparing a live browser
# against the fingerprint rather than by reading the code.
scale = (fp.get("screen") or {}).get("deviceScaleFactor")
if isinstance(scale, (int, float)) and not isinstance(scale, bool) and scale > 0:
Expand Down Expand Up @@ -331,7 +331,7 @@ def main() -> int:
# loader. It used to read `os.environ` alone, which meant a key put in
# `.env` exactly as the README and .env.example instruct worked for every
# engine and failed HERE with "No API key" — a documented mechanism not
# applied on one path, which is the shape of half the defects §16 lists.
# applied on one path.
#
# `.env` has to be LOADED before it can be read: `env_value` looks at
# os.environ, and `load_env` is what fills that from the file. Calling it
Expand Down
Loading
Loading