diff --git a/CHANGELOG.md b/CHANGELOG.md index c3f8283..5a5ab14 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,10 +6,11 @@ follow SemVer as closely as a CLI toolkit can: a **patch** means fixes, and a fix may change a default when the old one was wrong — such a change leads its entry in a blockquote. -## [1.0.1] — 2026-10-07 +## [1.0.1] — 2026-10-10 -Fixes from an external audit of v1.0.0. Each finding was reproduced before -it was fixed, and each now has a regression test. +Fixes from an external audit of v1.0.0, and from live runs on 2026-10-08, +-09 and -10. Each finding was reproduced before it was fixed, and each now +has a regression test. > **Behaviour changes for existing users.** > * An **unexpected exception** now exits **1** (crash, traceback printed @@ -18,6 +19,8 @@ it was fixed, and each now has a regression test. > * `diff_runs.py` now **refuses** two outputs when either has no readable > `.meta.json`, or when they were written by different row schemas. Pass > `--force` to compare anyway. +> * `--mode seller` now exits **6** (partial) when it wrote fewer listings +> than the profile's own counter, instead of 0. ### Fixed @@ -31,6 +34,52 @@ it was fixed, and each now has a regression test. * **No attempt metadata on a failed start or a crash.** `.latest_attempt.meta.json` is now written on every ending, with `exit_code`, `error_class` and a masked `error_message`. +* **A failed captcha solve is now traceable.** Every solve in + `transport_facts.solves` records the 2Captcha `task_id` and its own error + code (`ERROR_CAPTCHA_UNSOLVABLE`, `TIMEOUT`, …), and the solver waits 240 s + instead of 180 s: on 2026-10-08 2Captcha took 217–220 s to return + `ERROR_CAPTCHA_UNSOLVABLE` for this captcha, so the old wait abandoned + every task before its verdict and logged only "no solution". +* **A third firewall variant** — «подождите немного и обновите страницу», no + captcha form, no proof-of-work, served right after an accepted solve + (measured 2026-10-08) — is now a refusal of the exit and rotates, instead + of being waited out as a challenge and reported blocked. +* **A solve is `verified` only when the page actually came back.** The first + release marked it verified as soon as the document stopped being the wall, + which included that third variant. A solve that is followed by something + else records `after_submit`. A proof-of-work after a solve is waited out. +* **The wall's other two captchas.** The page can show Avito's own picture + captcha or hCaptcha instead of GeeTest; the scraper now recognises which + one is showing and records it (`solves[].variant`). The picture captcha is + solved with `ImageToTextTask` and answered through the page's own field — + implemented from the wall's script, not yet met live (every measured + request was assigned GeeTest). hCaptcha is solved with `HCaptchaTask` + (sitekey from the wall, the browser's own exit, `enterprisePayload.rqdata` + only if the wall ever carries one) and its token submitted through + `#h-captcha-response` — also not yet met live. One live `HCaptchaTask` on + avito's sitekey (2026-10-09) was accepted by the API and returned + ERROR_CAPTCHA_UNSOLVABLE after 51 s: the request shape is right, a solved + token for this wall is not yet measured. +* **`--mode seller` now returns all of a seller's listings**, not the 15 + the profile renders: it opens the seller's feed, presses «Показать все» + and scrolls until the profile's own counter or `--max-products` + (measured: 353 of 353). The feed's scroll makes the page fetch + `/web/1/profile/items`, which robots.txt disallows — README says so; this + is a deliberate choice. `--max-products 15` keeps to the profile page. +* **A recommendation carousel was read as search results.** On 2026-10-10 + `?q=yeezy` page 1 carried a «Подобрали для вас» block INSIDE the results + grid, with 31 cards of the same shape; page 1 came back with 82 rows, 32 + of them unrelated. Cards inside `itemsCarousel` are no longer results. +* **A seller run with fewer listings than the profile states is now + `partial` (exit 6).** A shop stating 7331 active listings was written as a + `complete` run of 12, because the profile's tabbed counter («Активные | + N») was not read and the feed never started. Both are fixed; a shop + profile without a `sellerId` now takes it from `/brands//items` + (measured: 100 of 100 with `--max-products 100`). +* The seller's name falls back to the profile heading when the page title + does not carry it (one fetch of two, 2026-10-10). +* Rotation messages name Chromium's error (`ERR_CONNECTION_RESET`) instead of + a fragment of the URL. * **`schema_version`** now fingerprints the row class of the run's mode (names, order and types), not `Product`'s field names only. * Leftovers from the repositories this one was scaffolded from, in diff --git a/README.md b/README.md index fb58e33..59a5ddc 100644 --- a/README.md +++ b/README.md @@ -35,12 +35,21 @@ residential exit. avito.ru sits behind its own firewall, «Доступ ограничен: проблема с IP». Its own text says why: *too many requests from this address*. It answers in -one of two ways: +one of these ways: | What you meet | Status | What clears it | |---|---|---| | a proof-of-work page | 439 | the browser itself, in a few seconds — the page runs `/web/3/firewallPow/…` and reloads | | a captcha page, «Продолжить» → **GeeTest v4** slide puzzle | 429 | a solve submitted through the page's own form | +| «подождите немного и обновите страницу» — no captcha, no proof-of-work | 429 | a different exit. Measured: this is what follows a **rejected** solution | + +The captcha page can carry two other captchas — Avito's own picture +captcha and hCaptcha. The server picks one per request; **every request +measured (2026-10-06 to 10-09) got GeeTest**, including the page's own +"new picture" request. The picture captcha is implemented (`ImageToTextTask`) +but has never been met live. hCaptcha is solved with `HCaptchaTask` +(sitekey from the wall, through the browser's own exit) — likewise +implemented from the wall's code and 2Captcha's task format, never met live. | Client | Result | |---|---| @@ -181,9 +190,21 @@ largest size), `category_path`, `address`, `lat`, `lng`, `views_total`, `.json`: `seller_name`, `seller_rating`, `seller_reviews`, `seller_since`, `subscribers`, `active_ads` (the profile's own counter), -`listings_on_page`, `badges`. `_listings.json`: the cards the profile -page renders, as listing rows — **15 measured**, of 107–353 active; the rest -load as the page scrolls and are not fetched by this version. +`listings_on_page`, `badges`. `_listings.json`: **all** the seller's +listings, as listing rows, up to the profile's own counter or +`--max-products`. The profile page renders 15; the rest come from the +seller's feed (`/brands//all?sellerId=…`), which the scraper opens, +arms with its «Показать все» button and scrolls — 15 more per scroll. +Measured 2026-10-09: **353 of 353** in 22 scrolls, every row priced. +Compare `listings_on_page` with `active_ads` to see whether the feed was +read to the end; `transport_facts.seller_feed` says why it stopped. + +**About robots.txt and the seller feed.** The scraper never requests a path +avito.ru's robots.txt disallows. But scrolling the feed makes the PAGE'S OWN +script fetch `/web/1/profile/items`, and `/web/` is disallowed for robots. +That is a deliberate choice (owner, 2026-10-09) — the only way to a +seller's listings beyond 15. Use `--max-products 15` to stay on the +profile page alone. ### Seller names, and what is never collected @@ -356,8 +377,10 @@ details, and does not message sellers. When avito.ru's firewall presents a captcha, the scraper solves it the way a visitor would — through the page's own form — with a solve you pay 2Captcha for. -`robots.txt` is respected: the snapshot is in `robots_snapshot.py`, and URLs -it disallows are refused before any request is made. +`robots.txt` is respected for every URL the scraper requests: the snapshot +is in `robots_snapshot.py`, and URLs it disallows are refused before any +request is made. One exception is stated above: scrolling a seller's feed +makes the page itself fetch the disallowed `/web/1/profile/items`. The firewall exists because of request volume. Default `--delay` is 2 seconds between pages, `--concurrency` above 1 is refused, and running hard from one diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md index e504420..eb6bccf 100644 --- a/TROUBLESHOOTING.md +++ b/TROUBLESHOOTING.md @@ -17,6 +17,12 @@ which variant you met: |---|---|---| | a page that reloads itself, requests to `/web/3/firewallPow/…` | 439 | the browser clears it within seconds; the scraper waits 25 s | | «Продолжить» and, after it, a GeeTest slide puzzle | 429 | the scraper solves it (below) | +| «подождите немного и обновите страницу», nothing to solve | 429 | follows a REJECTED solution; the scraper moves to a fresh exit (`--proxy-block-retries`) | +| «введите символы с картинки» — Avito's picture captcha | 429 | solved with `ImageToTextTask`; never met live yet — if you see it, `solves[].variant` is `image` and a report is welcome | +| hCaptcha («поставьте галочку») | 429 | solved with `HCaptchaTask` through the browser's exit; never met live yet — `solves[].variant` is `hcaptcha` | + +`transport_facts.solves[]` names the variant of every wall met, its 2Captcha +`task_id` and, on failure, 2Captcha's own error code. **The GeeTest wall stays up after a solve.** Look at `transport_facts.solves`: diff --git a/browser_bridge.py b/browser_bridge.py index 3766aca..10df460 100644 --- a/browser_bridge.py +++ b/browser_bridge.py @@ -195,48 +195,84 @@ def solve_wall(driver, args, url: str, html: str, facts: Dict[str, Any], log=Non facts["walls_cleared_by"].append("autosolve") return fresh - captcha_id = P.geetest_captcha_id(html) + variant = P.wall_variant(html) or "geetest" + facts.setdefault("wall_variants", []).append(variant) key = getattr(args, "twocaptcha_key", None) - if not captcha_id: - print("[!] firewall captcha without a GeeTest id — this repo implements " - "GeeTest v4 only; reporting the page as blocked.", file=sys.stderr) - return None if not key: - print("[!] firewall captcha (GeeTest v4) and no TWOCAPTCHA_KEY — " - "continuing without solving.", file=sys.stderr) + print("[!] firewall captcha (%s) and no TWOCAPTCHA_KEY — continuing without " + "solving." % variant, file=sys.stderr) return None import captcha_solver + t0 = time.monotonic() try: - t0 = time.monotonic() - solution = captcha_solver.solve_geetest_v4( - key, captcha_id, url, proxy=getattr(args, "solver_proxy", None)) + if variant == "image": + solution = captcha_solver.solve_image(key, P.wall_image_src(html)) + submit = captcha_solver.submit_text_js_body(solution["text"]) + elif variant == "hcaptcha": + solution = captcha_solver.solve_hcaptcha( + key, P.wall_hcaptcha_sitekey(html), url, + proxy=getattr(args, "solver_proxy", None), rqdata=P.wall_hcaptcha_rqdata(html)) + submit = captcha_solver.submit_hcaptcha_js_body(solution["token"]) + else: + captcha_id = P.geetest_captcha_id(html) + solution = captcha_solver.solve_geetest_v4( + key, captcha_id, url, proxy=getattr(args, "solver_proxy", None)) + submit = captcha_solver.submit_js_body(solution) except captcha_solver.SolverError as exc: - print("[!] solver: %s" % mask_text(str(exc)), file=sys.stderr) - facts["solves"].append({"ok": False, "error": mask_text(str(exc))[:200]}) + print("[!] solver (%s): %s" % (variant, mask_text(str(exc))), file=sys.stderr) + facts["solves"].append({"ok": False, "variant": variant, + "error": mask_text(str(exc))[:200], + "task_id": getattr(exc, "task_id", None), + "error_code": getattr(exc, "error_code", None), + "seconds": round(time.monotonic() - t0, 1)}) return None - record = {"ok": True, "seconds": round(time.monotonic() - t0, 1), - "cost": solution.pop("_cost", None), "verified": False} + record = {"ok": True, "variant": variant, "seconds": round(time.monotonic() - t0, 1), + "cost": solution.pop("_cost", None), "task_id": solution.pop("_task_id", None), + "verified": False} facts["solves"].append(record) if log: - log(" GeeTest solved in %ss — submitting through the page's own form" - % record["seconds"]) + log(" %s solved in %ss — submitting through the page's own form" + % ("GeeTest" if variant == "geetest" else "Image captcha", record["seconds"])) try: - submitted = driver.run_js(captcha_solver.submit_js_body(solution)) + submitted = driver.run_js(submit) except Exception as exc: # noqa: BLE001 print("[!] could not submit the solution: %s" % mask_text(str(exc))[:200], file=sys.stderr) return None if not submitted: print("[!] the wall's form was gone before the solution arrived.", file=sys.stderr) + return _after_submit(driver, url, record, facts) + + +def _after_submit(driver, url: str, record: Dict[str, Any], facts: Dict[str, Any]) -> Optional[str]: + """What the site served after a submitted solution. + + `verified` means the PAGE came back — cards, an item, a profile, or an + honest empty listing. "No longer the wall" is not enough: the page passes + through an intermediate document while it reloads. And the wait-wall + («подождите немного и обновите страницу») is what the site serves after + a REJECTED solution: measured 2026-10-09, three deliberately wrong GeeTest + submissions were each answered `verified: false` and then that wall. + """ + last = "captcha" + fresh = "" for _ in range(20): driver.sleep(1500) fresh = driver.content() or "" - if fresh and not P.is_wall(fresh): + if not fresh: + continue + last = P.detect_page_state(fresh, url=url) + if last in ("content", "empty"): record["verified"] = True facts["walls_cleared_by"].append("solver") return fresh - print("[!] the site did not accept the solved token (still walled).", file=sys.stderr) - return None + if last == "blocked": + break + record["after_submit"] = last + print("[!] after the solved captcha the site served %s, not the page%s." + % (last, " — the solution was rejected; rotating" if last == "blocked" else ""), + file=sys.stderr) + return fresh if last == "blocked" else None # --------------------------------------------------------------------------- @@ -261,9 +297,10 @@ def fetch_page(driver, args, url: str, mode: str, facts: Dict[str, Any], log=Non if attempt == attempts: raise BridgeError("navigation failed after %d attempt(s): %s: %s" % (attempts, type(exc).__name__, message)) from None - print("[i] navigation failed (%s) — retry %d/%d in %.0fs" - % (type(exc).__name__, attempt, attempts - 1, args.retry_delay), - file=sys.stderr) + code = re.search(r"(?:net::)?ERR_[A-Z_]+|Timeout \d+ms exceeded", str(exc)) + print("[i] navigation failed (%s%s) — retry %d/%d in %.0fs" + % (type(exc).__name__, ": " + code.group(0) if code else "", + attempt, attempts - 1, args.retry_delay), file=sys.stderr) time.sleep(args.retry_delay) html = wait_for_content(driver, mode, url, page_flow.content_timeout_ms(mode), log=log) @@ -284,6 +321,10 @@ def fetch_page(driver, args, url: str, mode: str, facts: Dict[str, Any], log=Non if fresh is not None: html = wait_for_content(driver, mode, url, page_flow.content_timeout_ms(mode), log=log) state = P.detect_page_state(html, url=url) + if state == "challenge": + # A proof-of-work after the solve clears itself like any other. + html = wait_out_challenge(driver, url, page_flow.CHALLENGE_SETTLE_MS, log=log) + state = P.detect_page_state(html, url=url) if state == "unknown": if log: @@ -313,7 +354,8 @@ def fetch_with_rotation(driver, args, url: str, mode: str, facts, log=None) -> D # answered at once. if retries <= 0 or not _is_exit_failure(str(exc)) or not _fresh_exit_available(args): raise - reason = "the exit failed (%s)" % str(exc).split(":")[-1].strip()[:60] + code = _EXIT_FAILURE_RE.search(str(exc)) + reason = "the exit failed (%s)" % (code.group(0) if code else type(exc).__name__) else: if not page_flow.should_rotate(result["state"]) or retries <= 0 \ or not _fresh_exit_available(args): @@ -628,16 +670,147 @@ def _run_seller(driver, args, url, facts, log): facts["failed_pages"].append(1) return [], [] facts["pages_completed"] = 1 - for row in listings: - row.page = 1 + want = seller.active_ads or 0 + if args.max_products: + want = min(want, args.max_products) if want else args.max_products + if want > len(listings): + more = _seller_feed(driver, args, url, result["html"], want, facts, log) + if more is not None: + listings = more + if args.max_products: + listings = listings[:args.max_products] + if want and len(listings) < want: + # 2026-10-10: a shop stating 7331 active listings came back as a + # `complete` run holding 12. Fewer than the profile's own counter is + # a partial view, and the run says so (exit 6). + facts["seller_listings_short"] = {"got": len(listings), "want": want} + seller.listings_on_page = len(listings) + for i, row in enumerate(listings, 1): + row.page, row.position = 1, i return [seller], listings +# Scrolling the feed: a round adds 15 cards when the site answers (measured +# 2026-10-09: 15 -> 195 in 12 rounds, ~2.7 s each). Three rounds with neither +# a new card nor a taller page mean the feed is exhausted (family rule: the +# terminating condition is data, and one still round is not enough). +SELLER_SCROLL_PAUSE_MS = 2500 +SELLER_SCROLL_STILL_ROUNDS = 3 +SELLER_SCROLL_MAX_ROUNDS = 400 +SELLER_SHOW_ALL_SETTLE_MS = 3000 +SELLER_SHOW_ALL_MAX_CLICKS = 3 + + +def _seller_feed(driver, args, profile_url, profile_html, want, facts, log): + """All of a seller's listings, from the profile's full feed. + + DECISION (owner, 2026-10-09): the profile page renders 15 listings; the + rest only arrive when the feed `/brands//all?sellerId=…` is + scrolled, and the page's own script then fetches them from + `/web/1/profile/items`, a path avito.ru's robots.txt disallows. The scraper + never requests that path itself — it scrolls, and the site's page does — + but the purpose of the scroll is that data. README says so plainly. + + Returns the parsed rows, or None (keep the profile's 15) when the feed + cannot be read. + """ + seller_hash = P.seller_hash_id(profile_html) + feed = {"want": want, "rounds": 0, "cards": 0, "stop": None} + facts["seller_feed"] = feed + if not seller_hash: + # 2026-10-10, /brands/a3boots.shop: a shop's profile carries no + # sellerId at all; its `/brands//items` page (robots: allowed) + # does, and the same `/all?sellerId=` feed then works for it. + items_url = P.seller_items_url(profile_url) + if P.robots_verdict(items_url)[0]: + time.sleep(args.delay) + found = fetch_with_rotation(driver, args, items_url, "seller", facts, log=log) + if found["state"] == "content": + seller_hash = P.seller_hash_id(found["html"]) + feed["sellerId_from"] = "items page" if seller_hash else None + if not seller_hash: + feed["stop"] = "no sellerId on the profile" + print("[!] no sellerId on the profile — keeping its first listings only.", file=sys.stderr) + return None + feed_url = P.seller_all_url(profile_url, seller_hash) + feed["url"] = feed_url + allowed, rule = P.robots_verdict(feed_url) + if not allowed: + feed["stop"] = "robots: %s" % rule + return None + time.sleep(args.delay) + result = fetch_with_rotation(driver, args, feed_url, "seller", facts, log=log) + if result["state"] != "content": + feed["stop"] = "feed page: %s" % result["state"] + print("[!] the seller's feed came back %s — keeping the profile's listings only." + % result["state"], file=sys.stderr) + return None + # The page's script has to be up before the button does anything: a + # click straight after the HTML arrived (the cards are server-rendered, + # so readiness is immediate) did nothing on 2026-10-09; the same click + # 4 s later armed the feed. Wait, click, and click again if the feed + # does not move. + driver.sleep(SELLER_SHOW_ALL_SETTLE_MS) + feed["show_all_clicks"] = 0 + + def click_show_all(): + try: + clicked = bool(driver.run_js(P.SELLER_SHOW_ALL_JS_BODY)) + except Exception as exc: # noqa: BLE001 + feed["show_all_error"] = type(exc).__name__ + return False + if clicked: + feed["show_all_clicks"] += 1 + driver.sleep(SELLER_SCROLL_PAUSE_MS) + return clicked + + click_show_all() + count = driver.count(P.SELLER_CARD_SELECTOR) + still, last = 0, (count, None) + while count < want and feed["rounds"] < SELLER_SCROLL_MAX_ROUNDS: + try: + height = driver.scroll_to_bottom() + except Exception as exc: # noqa: BLE001 + feed["stop"] = "scroll failed: %s" % type(exc).__name__ + break + driver.sleep(SELLER_SCROLL_PAUSE_MS) + feed["rounds"] += 1 + count = driver.count(P.SELLER_CARD_SELECTOR) + if (count, height) == last: + still += 1 + if still == 2 and feed["show_all_clicks"] < SELLER_SHOW_ALL_MAX_CLICKS: + click_show_all() + if still >= SELLER_SCROLL_STILL_ROUNDS: + feed["stop"] = "feed stopped growing" + break + else: + still = 0 + last = (count, height) + if log and feed["rounds"] % 5 == 0: + log(" seller feed: %d of %d listings after %d scrolls" % (count, want, feed["rounds"])) + html = driver.content() or "" + if P.is_wall(html): + # The feed can meet the firewall mid-scroll; what is on screen is gone. + feed["stop"] = "the firewall interrupted the feed" + print("[!] the firewall interrupted the seller feed — keeping the profile's listings only.", + file=sys.stderr) + return None + _, rows = P.parse_seller(html, url=profile_url) + rows = output_writer.dedupe_by_sku(rows, set()) + feed["cards"] = len(rows) + feed["stop"] = feed["stop"] or ("reached %d" % want if len(rows) >= want else "feed ended") + if log: + log(" seller feed: %d listings (%s)" % (len(rows), feed["stop"])) + return rows if rows else None + + def _stop_reason(args, facts: Dict[str, Any], pages_requested: int) -> str: if facts["blocked"]: return "blocked" if args.mode == "seller": - return "single_page_mode" if facts["pages_completed"] else "no_profile" + if not facts["pages_completed"]: + return "no_profile" + return "seller_listings_short" if facts.get("seller_listings_short") else "single_page_mode" if facts["failed_pages"]: return "pages_failed" if facts.get("max_products_reached"): diff --git a/captcha_solver.py b/captcha_solver.py index d85a2de..eb0764b 100644 --- a/captcha_solver.py +++ b/captcha_solver.py @@ -14,11 +14,16 @@ GeeTest v4, slide implemented — task `GeeTestTask`, `version: 4`, `initParameters.captcha_id`. captcha_id is hard-coded in the wall: 2d9c743cf7d63dbc9db578a608196bcd - hCaptcha this repo does not implement it yet (never served - in a measured run) - Avito image captcha this repo does not implement it yet; it appeared - once, as the server's answer to a REJECTED GeeTest - token (see below) + Avito image captcha implemented — `ImageToTextTask` on the picture the + wall shows, answer through `#form-input`. Not + live-verified: the server never assigned it in a + measured run (2026-10-06..09). + hCaptcha implemented — `HCaptchaTask` (shape confirmed by + 2Captcha, 2026-10-09; the public /api-docs/hcaptcha + page answered 404 that day), sitekey from the + wall's `data-sitekey`, through the browser's own + exit, token into `#h-captcha-response`. Not + live-verified: never assigned in a measured run. These are statements about THIS repo, not about what 2Captcha can solve (2scraper family rule). @@ -68,7 +73,17 @@ def _redact(text) -> str: class SolverError(RuntimeError): """The solver could not produce a usable answer. Always a WARNING to the - caller: the run continues and reports exit 3 if the page stays walled.""" + caller: the run continues and reports exit 3 if the page stays walled. + + Carries the 2Captcha `task_id` and `error_code` when there is one, so a + failed solve can be traced in the 2Captcha dashboard and support tickets — + a bare "no solution" was all the first release could say. + """ + + def __init__(self, message, task_id=None, error_code=None): + super().__init__(message) + self.task_id = task_id + self.error_code = error_code def proxy_task_fields(proxy: Optional[str]) -> dict: @@ -114,7 +129,7 @@ def geetest_v4_task(captcha_id: str, page_url: str, proxy: Optional[str]) -> dic def solve_geetest_v4(api_key: str, captcha_id: str, page_url: str, proxy: Optional[str] = None, allow_proxyless: bool = False, - timeout_s: int = 180, poll_s: float = 5.0, + timeout_s: int = 240, poll_s: float = 5.0, session: Optional[requests.Session] = None) -> dict: """Solve and return the solution dict: `{captcha_id, lot_number, pass_token, gen_time, captcha_output}`. @@ -129,41 +144,150 @@ def solve_geetest_v4(api_key: str, captcha_id: str, page_url: str, "from a different exit than the browser's (measured), so this " "would pay for a token the site refuses. Give the run the same " "session-pinned proxy the browser uses (AVITO_PROXY).") - http = session or requests.Session() task = geetest_v4_task(captcha_id, page_url, proxy) + # 240 s, not 180: measured 2026-10-08, 2Captcha took 217 s and 220 s to + # give up on this captcha (ERROR_CAPTCHA_UNSOLVABLE). A shorter wait + # abandons the task before its verdict and reports only "no solution". + solution, task_id, cost = _run_task(api_key, task, timeout_s, poll_s, session) + missing = [k for k in ("lot_number", "pass_token", "gen_time", "captcha_output") + if not solution.get(k)] + if missing: + raise SolverError("solution without %s (task %s)" % (", ".join(missing), task_id), + task_id=task_id) + solution.setdefault("captcha_id", captcha_id) + solution["_cost"] = cost + solution["_task_id"] = task_id + return solution + + +def _run_task(api_key: str, task: dict, timeout_s: float, poll_s: float, + session: Optional[requests.Session] = None): + """createTask + poll getTaskResult. Returns (solution, task_id, cost). + Every failure is a SolverError carrying the task id and 2Captcha's code.""" + if not api_key: + raise SolverError("no TWOCAPTCHA_KEY") + http = session or requests.Session() try: - created = http.post(API + "/createTask", - json={"clientKey": api_key, "task": task}, + created = http.post(API + "/createTask", json={"clientKey": api_key, "task": task}, timeout=30).json() except Exception as exc: # noqa: BLE001 raise SolverError("createTask failed: %s: %s" % (type(exc).__name__, _redact(exc))) from None if created.get("errorId"): raise SolverError("createTask: %s %s" % (created.get("errorCode"), - _redact(created.get("errorDescription", "")))) + _redact(created.get("errorDescription", ""))), + error_code=created.get("errorCode")) task_id = created.get("taskId") deadline = time.monotonic() + timeout_s while time.monotonic() < deadline: time.sleep(poll_s) try: got = http.post(API + "/getTaskResult", - json={"clientKey": api_key, "taskId": task_id}, - timeout=30).json() + json={"clientKey": api_key, "taskId": task_id}, timeout=30).json() except Exception as exc: # noqa: BLE001 raise SolverError("getTaskResult failed: %s: %s" - % (type(exc).__name__, _redact(exc))) from None + % (type(exc).__name__, _redact(exc)), task_id=task_id) from None if got.get("errorId"): - raise SolverError("getTaskResult: %s" % got.get("errorCode")) + raise SolverError("getTaskResult: %s (task %s)" % (got.get("errorCode"), task_id), + task_id=task_id, error_code=got.get("errorCode")) if got.get("status") == "ready": - solution = dict(got.get("solution") or {}) - missing = [k for k in ("lot_number", "pass_token", "gen_time", "captcha_output") - if not solution.get(k)] - if missing: - raise SolverError("solution without %s" % ", ".join(missing)) - solution.setdefault("captcha_id", captcha_id) - solution["_cost"] = got.get("cost") - return solution - raise SolverError("no solution within %ds" % timeout_s) + return dict(got.get("solution") or {}), task_id, got.get("cost") + raise SolverError("no solution within %ds (task %s)" % (timeout_s, task_id), + task_id=task_id, error_code="TIMEOUT") + + +def hcaptcha_task(sitekey: str, page_url: str, proxy: Optional[str], + rqdata: Optional[str] = None) -> dict: + """`HCaptchaTask` (API v2). The shape was confirmed by 2Captcha on + 2026-10-09 — its public page /api-docs/hcaptcha answered 404 that day, so + it is cited from the vendor, not from that page. `enterprisePayload.rqdata` + only for hCaptcha Enterprise; avito's wall carries no rqdata.""" + if not re.fullmatch(r"[0-9a-f-]{36}", sitekey or ""): + raise SolverError("not an hCaptcha sitekey: %r" % (sitekey,)) + task = {"type": "HCaptchaTask", "websiteURL": page_url, "websiteKey": sitekey, + "isInvisible": False} + task.update(proxy_task_fields(proxy)) + if rqdata: + task["enterprisePayload"] = {"rqdata": rqdata} + return task + + +def solve_hcaptcha(api_key: str, sitekey: str, page_url: str, proxy: Optional[str] = None, + rqdata: Optional[str] = None, timeout_s: int = 240, poll_s: float = 5.0, + session: Optional[requests.Session] = None) -> dict: + """Solve the wall's hCaptcha; returns `{token, _cost, _task_id}`. + + Through the browser's own exit when there is one — the same rule GeeTest + taught here (a token from another IP was rejected). NOT live-verified: + the server never assigned hCaptcha in a measured run (2026-10-06..09). + """ + solution, task_id, cost = _run_task(api_key, hcaptcha_task(sitekey, page_url, proxy, rqdata), + timeout_s, poll_s, session) + token = solution.get("gRecaptchaResponse") or solution.get("token") + if not token: + raise SolverError("hCaptcha solution without a token (task %s)" % task_id, task_id=task_id) + return {"token": token, "_cost": cost, "_task_id": task_id} + + +SUBMIT_HCAPTCHA_JS_BODY = ( + "var ta = document.querySelector('#h-captcha-response');" + "var form = document.querySelector('.js-firewall-form');" + "if (!ta || !form) { return false; }" + "ta.value = VALUE;" + "var g = document.querySelector('[name=g-recaptcha-response]'); if (g) { g.value = VALUE; }" + "form.dispatchEvent(new Event('submit'));" + "return true;" +) + + +def submit_hcaptcha_js_body(token: str) -> str: + """The token into `#h-captcha-response` — the field the wall's submit + handler reads — then the form's own submit handler.""" + return SUBMIT_HCAPTCHA_JS_BODY.replace("VALUE", json.dumps(token)) + + +def image_task(image_src: str) -> dict: + """`ImageToTextTask` for the wall's own picture captcha + (https://2captcha.com/api-docs/normal-captcha). The wall sets the picture + as a `data:` URL; only that form is accepted here — anything else would + mean fetching an image outside the browser's session.""" + if not image_src or not image_src.startswith("data:image") or "," not in image_src: + raise SolverError("image captcha without a data: URL picture") + return {"type": "ImageToTextTask", "body": image_src.split(",", 1)[1], "case": False} + + +def solve_image(api_key: str, image_src: str, timeout_s: int = 120, poll_s: float = 4.0, + session: Optional[requests.Session] = None) -> dict: + """Solve the wall's picture captcha; returns `{text, _cost, _task_id}`. + + NOT live-verified: every firewall request measured (2026-10-06..09) + was assigned GeeTest, including the page's own "refresh picture" request. + The task type and the submit path come from 2Captcha's documentation and + the wall's own script (`#form-input`, then the form's submit handler). + No proxy fields: an image task is a picture, not a session. + """ + solution, task_id, cost = _run_task(api_key, image_task(image_src), timeout_s, poll_s, + session) + text = (solution.get("text") or "").strip() + if not text: + raise SolverError("empty answer (task %s)" % task_id, task_id=task_id) + return {"text": text, "_cost": cost, "_task_id": task_id} + + +SUBMIT_TEXT_JS_BODY = ( + "var inp = document.querySelector('#form-input');" + "var form = document.querySelector('.js-firewall-form');" + "if (!inp || !form) { return false; }" + "inp.value = VALUE;" + "form.dispatchEvent(new Event('submit'));" + "return true;" +) + + +def submit_text_js_body(text: str) -> str: + """The picture-captcha answer into the page's own `#form-input`, then the + form's own submit handler — the same path a visitor's Enter takes.""" + return SUBMIT_TEXT_JS_BODY.replace("VALUE", json.dumps(text)) def firewall_response_value(solution: dict) -> str: diff --git a/fixtures/README.md b/fixtures/README.md index 94b6897..8058d65 100644 --- a/fixtures/README.md +++ b/fixtures/README.md @@ -14,6 +14,7 @@ apart from the fields scrubbed below. | `item_private.html` | item page, private seller | anonymised seller, masked IMEI dropped | | `seller.html` | `/brands/…` profile | profile counters, 2 of its cards | | `wall_captcha.html` | the firewall (HTTP 429) after «Продолжить» | GeeTest v4 form + `captchaId` | +| `wall_wait.html` | the firewall's third variant, served right after an accepted solve (2026-10-08) | «подождите немного и обновите страницу» → `blocked`; title and text only, SVG and script removed | ## Not verbatim diff --git a/fixtures/wall_wait.html b/fixtures/wall_wait.html new file mode 100644 index 0000000..11b0eb4 --- /dev/null +++ b/fixtures/wall_wait.html @@ -0,0 +1,26 @@ + +Доступ ограничен: проблема с IP + +
+
+ +

Доступ ограничен: проблема с IP

+

Иногда такое случается — подождите немного и обновите страницу. Если проблема не уходит, вот что можно сделать:

+
    +
  • Отключить VPN.
    • +
  • Включить и выключить режим «В самолёте».
    • +
  • Подключиться к другой сети.
    • +
  • Перезагрузить роутер.
    • +
+

+ Если и это не сработает, напишите в поддержку. + В письме укажите город, провайдера и IP-адрес (его можно посмотреть на yandex.ru/internet). + Постараемся разобраться как можно скорее. +

+
+
+ + + diff --git a/output_writer.py b/output_writer.py index d0d41a7..30f0bf4 100644 --- a/output_writer.py +++ b/output_writer.py @@ -129,9 +129,11 @@ class Seller: seller_reviews: Optional[int] = None seller_since: Optional[str] = None subscribers: Optional[int] = None - # The profile's own "Объявления N" counter — how many are LIVE. The page - # renders only the first 15 of them; `listings_on_page` says how many - # rows the companion listings file actually holds. + # The profile's own "Объявления N" counter — how many are LIVE. The + # profile page renders 15; the rest come from scrolling the seller's feed + # (up to this counter or --max-products). `listings_on_page` is how many + # rows the companion `_listings` file actually holds — compare the + # two to see whether the feed was read to the end. active_ads: Optional[int] = None listings_on_page: Optional[int] = None badges: Optional[List[str]] = None diff --git a/playwright_scraper.py b/playwright_scraper.py index 938e8f3..22e7879 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -131,6 +131,12 @@ def count(self, selector): def run_js(self, body): return self._page.evaluate("() => { %s }" % body) + def scroll_to_bottom(self): + """Scroll to the end of the document; return its height. A real + function object, not an evaluated string (CSP-safe, family rule).""" + return self._page.evaluate( + "() => { window.scrollTo(0, document.body.scrollHeight); return document.body.scrollHeight; }") + def sleep(self, ms): self._page.wait_for_timeout(ms) diff --git a/product_parser.py b/product_parser.py index 16756b3..4fe011b 100644 --- a/product_parser.py +++ b/product_parser.py @@ -77,6 +77,7 @@ _GEETEST_ID_RE = re.compile(r"captchaId\s*=\s*['\"]([0-9a-f]{32})['\"]") _FIREWALL_FORM_RE = re.compile(r"]*\bjs-firewall-form\b", re.I) +WALL_WAIT_TEXT = "подождите немного и обновите страницу" PRIVATE_SELLER_PLACEHOLDER = "Пользователь" @@ -254,6 +255,59 @@ def results_cutoff(html: str) -> Optional[int]: return min(hits) if hits else None +_WALL_BLOCKS = (("geetest", "#geetest_captcha"), ("image", "#inner-captcha"), + ("hcaptcha", "#h-captcha")) + + +def wall_variant(html: str) -> Optional[str]: + """Which of the firewall's three captchas the page is SHOWING: + `geetest`, `image` (Avito's own text-from-picture captcha) or `hcaptcha`. + + The server picks one per request (`/web/5/firewallCaptcha/get`) and the + page's script un-hides that block, so the visible one is the answer — an + inline `display: none` on the other two. Every measured request so far was + `geetest`; the other two are in the wall's code and are recognised from + its markup, not from a live capture. + """ + if not is_wall(html) or not _FIREWALL_FORM_RE.search(html or ""): + return None + soup = BeautifulSoup(html, "html.parser") + for name, selector in _WALL_BLOCKS: + node = soup.select_one(selector) + if node is None: + continue + style = re.sub(r"\s+", "", (node.get("style") or "").lower()) + if "display:none" not in style: + if name == "geetest" and not geetest_captcha_id(html): + continue + return name + return None + + +def wall_hcaptcha_sitekey(html: str) -> Optional[str]: + """The wall's hCaptcha sitekey: `data-sitekey` of its `.h-captcha` + widget (measured: 070db171-ddb9-4c93-b7f6-d25d3c9d7e28).""" + m = re.search(r'class="h-captcha"[^>]*data-sitekey="([0-9a-f-]{36})"', html or "") or \ + re.search(r'data-sitekey="([0-9a-f-]{36})"', html or "") + return m.group(1) if m else None + + +def wall_hcaptcha_rqdata(html: str) -> Optional[str]: + """hCaptcha Enterprise's `rqdata`, when the wall carries one. Avito's did + not on any capture; kept so an Enterprise switch is not silently missed.""" + m = re.search(r'["\']?rqdata["\']?\s*[:=]\s*["\']([^"\']{8,})["\']', html or "") + return m.group(1) if m else None + + +def wall_image_src(html: str) -> Optional[str]: + """The image captcha's picture: `src` of `.js-form-captcha-image`, which + the wall's script sets from the server's answer (a `data:` URL).""" + soup = BeautifulSoup(html or "", "html.parser") + img = soup.select_one("#inner-captcha .js-form-captcha-image") or soup.select_one("#inner-captcha img") + src = img.get("src") if img is not None else None + return src or None + + def _has_listing_cards(html: str) -> bool: html = html or "" if "item_list_with_filters/item(" in html: @@ -297,8 +351,14 @@ def detect_page_state(html: str, status: Optional[int] = None, # The FORM element, not the substring: the wall's own script names # `.js-firewall-form` in a querySelector whether or not the form is # on the page. - if _FIREWALL_FORM_RE.search(html) and geetest_captcha_id(html): + if wall_variant(html): return "captcha" + # The third variant, measured 2026-10-08 right after an ACCEPTED + # GeeTest solve: no form, no proof-of-work, only «подождите немного и + # обновите страницу». Nothing clears it in place — a different exit + # does. Waiting it out as a "challenge" reported the page blocked. + if WALL_WAIT_TEXT in html: + return "blocked" return "challenge" if status in (403, 451): return "blocked" @@ -604,6 +664,12 @@ def parse_listing(html: str, url: str = "", category: Optional[str] = None) -> L # Substitutes after «Похожие объявления…» are not results. results = set(re.findall(r'data-item-id="(\d+)"', (html or "")[:cutoff])) cards = [c for c in cards if c.get("data-item-id") in results] + # Measured 2026-10-10 (`/all/odezhda_obuv_aksessuary?q=yeezy`): a + # «Подобрали для вас» carousel sits INSIDE the results grid with 31 cards + # of the same `data-marker="item"` shape — personal recommendations, none + # of them in the state's `items` (they are its `extraBlockItems`). Taken + # as results they made page 1 hold 82 rows, 32 of them unrelated. + cards = [c for c in cards if c.find_parent(attrs={"data-marker": "itemsCarousel"}) is None] state = listing_state(html) by_id: Dict[str, dict] = {} order: List[str] = [] @@ -723,9 +789,58 @@ def parse_item(html: str, url: str = "", category: Optional[str] = None) -> Opti # Seller # --------------------------------------------------------------------------- +_SELLER_HASH_RE = re.compile(r"sellerId(?:=|%3D)([0-9a-f]{32})") +SELLER_CARD_SELECTOR = '[data-marker^="item_list_with_filters/item("][data-item-id]' + + +def seller_hash_id(html: str) -> Optional[str]: + """The profile's `sellerId` — a 32-hex id in the profile's own links + (`/brands//items/all?…&sellerId=…`). `hashedUserId` in the page + state was empty on both captured profiles.""" + m = _SELLER_HASH_RE.search(html or "") + return m.group(1) if m else None + + +# The feed's infinite scroll is ARMED by its «Показать все» control (a div, +# not a link): measured 2026-10-09, scrolling the feed without the click left +# it at 15; after the click every scroll added 15. A function BODY with an +# explicit return — each engine wraps it in its own dialect. +SELLER_SHOW_ALL_JS_BODY = ( + "var b = document.querySelector('[data-marker=\"item_list_with_filters/show_all_button\"]');" + "if (!b) {" + " var els = Array.prototype.filter.call(document.querySelectorAll('button')," + " function (e) { return (e.textContent || '').trim() === 'Показать все'; });" + " b = els.length ? els[0] : null;" + "}" + "if (!b) { return false; }" + "b.scrollIntoView({block: 'center'});" + "b.click();" + "return true;" +) + + +def seller_items_url(seller_url: str) -> str: + """`/brands//items` — where a shop's sellerId is published when its + profile carries none (measured 2026-10-10). Without `?s=`, which + robots.txt disallows and which the profile's own button adds.""" + return "%s/brands/%s/items" % (BASE, seller_id_from_url(seller_url) or "") + + +def seller_all_url(seller_url: str, seller_hash: str) -> str: + """The profile's full feed, which «Показать все» opens: + `/brands//all?sellerId=`. robots.txt allows it; the feed's + own infinite scroll then fetches `/web/1/profile/items`, which robots.txt + disallows — see browser_bridge._seller_feed for that decision.""" + slug = seller_id_from_url(seller_url) or "" + return "%s/brands/%s/all?sellerId=%s" % (BASE, slug, seller_hash) + + _SINCE_RE = re.compile(r"На Авито с ([^|∙·]+?)\s*(?:[|∙·]|$)") _SUBSCRIBERS_RE = re.compile(r"(\d[\d\s ]*)\s*подписчик") -_ADS_RE = re.compile(r"Объявления\s*\|\s*(\d[\d\s ]*)") +# Two profile layouts, both live: «Объявления | 353» (2026-10-09) and the +# tabbed «Активные | 11 | Завершённые | 5» (2026-10-10, /brands/i1300361). +# Missing the second left active_ads None, so the feed was never tried. +_ADS_RE = re.compile(r"(?:Объявления|Активные)\s*\|\s*(\d[\d\s ]*)") def parse_seller(html: str, url: str = "") -> Tuple[Optional[Seller], List[Product]]: @@ -741,7 +856,13 @@ def parse_seller(html: str, url: str = "") -> Tuple[Optional[Seller], List[Produ # Авито"; the name is everything before that suffix. m = re.match(r"(.+?)\s+-\s+официальная страница", _title(html)) seller.seller_name = m.group(1).strip() if m else None - score = profile.select_one('[data-marker="profile/score"]') + if not seller.seller_name: + # The profile's own heading, `

`. Two + # fetches of one profile 30 s apart (2026-10-10) gave the name once + # and None once from the title alone. + h1 = profile.select_one('h1[data-marker^="name"]') or soup.select_one('h1[data-marker^="name"]') + seller.seller_name = h1.get_text(" ", strip=True) or None if h1 is not None else None + score =profile.select_one('[data-marker="profile/score"]') seller.seller_rating = parse_number(score.get_text(strip=True)) if score is not None else None summary = profile.select_one('[data-marker="profile/summary"]') seller.seller_reviews = (count_before_word(summary.get_text(" ", strip=True)) diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index cbc3517..1f6b435 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -113,6 +113,11 @@ def count(self, selector): def run_js(self, body): return self._run(self._page.evaluate("() => { %s }" % body)) + def scroll_to_bottom(self): + """Scroll to the end of the document; return its height.""" + return self._run(self._page.evaluate( + "() => { window.scrollTo(0, document.body.scrollHeight); return document.body.scrollHeight; }")) + def sleep(self, ms): self._run(asyncio.sleep(ms / 1000.0)) diff --git a/selenium_scraper.py b/selenium_scraper.py index d2abaa2..df822a0 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -96,6 +96,12 @@ def run_js(self, body): # snippet is written in that shape. return self._driver.execute_script(body) + def scroll_to_bottom(self): + """Scroll to the end of the document; return its height (Selenium's + execute_script takes a function BODY).""" + return self._driver.execute_script( + "window.scrollTo(0, document.body.scrollHeight); return document.body.scrollHeight;") + def sleep(self, ms): time.sleep(ms / 1000.0) diff --git a/smoke_test.py b/smoke_test.py index b3c35e5..42393d1 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -185,6 +185,21 @@ def test_listing_state_values(): eq("unlabelled shop is NOT called private", rows["7924194781"].seller_type, None) +def test_recommendation_carousel_is_not_results(): + """2026-10-10: «Подобрали для вас» sits inside the grid with item-shaped cards.""" + import product_parser as P + html = fixture("listing_dom.html") + carousel = ('

Подобрали для вас

' + '
') + i = html.index('data-marker="item"') + i = html.rindex("<", 0, i) + rows = P.parse_listing(html[:i] + carousel + html[i:], url=LISTING + "?p=2") + eq("carousel card dropped, results kept", [r.sku for r in rows], + ["8199454701", "8275958708", "8354606253", "8349145903"]) + + def test_the_bare_price_field_is_pre_discount(): """The trap this repo is built around: Avito's bare price is the OLD one.""" import product_parser as P @@ -325,6 +340,22 @@ def test_seller_values(): check("dates as printed", all(r.published_text for r in rows)) +def test_seller_tabbed_layout_and_heading_name(): + """2026-10-10, /brands/i1300361: «Активные | 11 | Завершённые | 5» tabs, + and a fetch whose title did not yield the name.""" + import product_parser as P + html = fixture("seller.html") + html = re.sub(r".*?", "Авито", html, flags=re.S) + html = re.sub(r"Объявления(\s*]+>)", r"Активные\1", html, count=1) + seller, _ = P.parse_seller(html, url=SELLER) + eq("tabbed counter read", seller.active_ads, 353) + eq("no title, no heading -> None, not a guess", seller.seller_name, None) + marker = 'data-marker="profile">' + html = html.replace(marker, marker + '

Продавец 1

', 1) + seller, _ = P.parse_seller(html, url=SELLER) + eq("name falls back to the profile heading", seller.seller_name, "Продавец 1") + + # --------------------------------------------------------------------------- # 4. The captcha rung # --------------------------------------------------------------------------- @@ -425,9 +456,11 @@ def _args(**kw): class FakeDriver: name = "fake" - def __init__(self, pages, after_js=None): + def __init__(self, pages, after_js=None, scrolls=None): self.pages = list(pages) self.after_js = after_js + self.scrolls = list(scrolls or []) # documents a scroll reveals, in order + self.scrolled = 0 self.current = "" self.js = [] self.autosolve_armed = False @@ -448,7 +481,14 @@ def content(self): return self.current def count(self, selector): - return self.current.count('data-marker="item"') if "item" in selector else 0 + from bs4 import BeautifulSoup + return len(BeautifulSoup(self.current, "html.parser").select(selector)) + + def scroll_to_bottom(self): + self.scrolled += 1 + if self.scrolls: + self.current = self.scrolls.pop(0) + return 1000 + len(self.current) def run_js(self, body): self.js.append(body) @@ -528,13 +568,263 @@ def fake_solve(key, cid, url, proxy=None, **kw): eq("cleared_by", facts["walls_cleared_by"], ["solver"]) +TINY_PNG = ("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR4" + "nGNgYGBgAAAABQABpfZFQAAAAABJRU5ErkJggg==") + + +def _wall_showing(variant): + """The REAL wall, with the server-chosen block un-hidden the way the wall's + own renderCaptcha() does it. Derived on purpose: every live request so far + was assigned GeeTest, so no capture of the other two exists.""" + from bs4 import BeautifulSoup + soup = BeautifulSoup(fixture("wall_captcha.html"), "html.parser") + for sel, name in (("#geetest_captcha", "geetest"), ("#inner-captcha", "image"), + ("#h-captcha", "hcaptcha")): + node = soup.select_one(sel) + node["style"] = "display: %s" % ("block" if name == variant else "none") + if variant == "image": + soup.select_one("#inner-captcha img")["src"] = TINY_PNG + return str(soup) + + +def test_the_wall_names_its_variant(): + import product_parser as P + for variant in ("geetest", "image", "hcaptcha"): + html = _wall_showing(variant) + eq("variant %s recognised" % variant, P.wall_variant(html), variant) + eq("variant %s is a captcha page" % variant, P.detect_page_state(html, url=LISTING), "captcha") + eq("the picture is read off the wall", P.wall_image_src(_wall_showing("image")), TINY_PNG) + eq("the wait-wall has no variant", P.wall_variant(fixture("wall_wait.html")), None) + + +def test_image_captcha_goes_through_the_page_form(): + import browser_bridge + import captcha_solver + calls = {} + real = captcha_solver.solve_image + + def fake(key, src, **kw): + calls["src"] = src + return {"text": 'ab"12', "_cost": "0.001", "_task_id": 7} + captcha_solver.solve_image = fake + try: + d = FakeDriver([_wall_showing("image")], after_js=fixture("listing_dom.html")) + with tempfile.TemporaryDirectory() as tmp: + rc = _quiet(browser_bridge.run, _args(out=os.path.join(tmp, "o"), twocaptcha_key="k", + solver_proxy=PLACEHOLDER_PROXY), d) + facts = json.load(open(os.path.join(tmp, "o.meta.json")))["transport_facts"] + finally: + captcha_solver.solve_image = real + eq("image wall solved -> rows", rc, 0) + eq("the wall's own picture went to the solver", calls.get("src"), TINY_PNG) + check("answer typed into the page's own #form-input", + any("#form-input" in j and '"ab\\"12"' in j for j in d.js)) + eq("recorded as an image solve, verified", (facts["solves"][0]["variant"], + facts["solves"][0]["verified"]), ("image", True)) + try: + captcha_solver.image_task("https://example.invalid/pic.png") + check("a picture that is not a data: URL is refused", False) + except captcha_solver.SolverError: + check("a picture that is not a data: URL is refused", True) + + +def test_hcaptcha_task_shape(): + """`HCaptchaTask` in the shape 2Captcha confirmed on 2026-10-09.""" + import captcha_solver as C + import product_parser as P + html = _wall_showing("hcaptcha") + key = P.wall_hcaptcha_sitekey(html) + eq("sitekey read off the wall", key, "070db171-ddb9-4c93-b7f6-d25d3c9d7e28") + eq("no rqdata on avito's wall", P.wall_hcaptcha_rqdata(html), None) + t = C.hcaptcha_task(key, LISTING, PLACEHOLDER_PROXY) + eq("task fields", {k: t[k] for k in ("type", "websiteURL", "websiteKey", "isInvisible")}, + {"type": "HCaptchaTask", "websiteURL": LISTING, "websiteKey": key, "isInvisible": False}) + eq("through the browser's proxy", (t["proxyType"], t["proxyAddress"], t["proxyPort"]), + ("http", "proxy.invalid", 2334)) + check("no enterprisePayload without rqdata", "enterprisePayload" not in t) + eq("rqdata goes in enterprisePayload", C.hcaptcha_task(key, LISTING, None, rqdata="RQ-EXAMPLE") + ["enterprisePayload"], {"rqdata": "RQ-EXAMPLE"}) + try: + C.hcaptcha_task("not-a-key", LISTING, None) + check("a bad sitekey is refused", False) + except C.SolverError: + check("a bad sitekey is refused", True) + + +def test_hcaptcha_goes_through_the_page_form(): + import browser_bridge + import captcha_solver + calls = {} + real = captcha_solver.solve_hcaptcha + + def fake(key, sitekey, url, proxy=None, rqdata=None, **kw): + calls.update(sitekey=sitekey, proxy=proxy, rqdata=rqdata) + return {"token": "P1_EXAMPLE.token", "_cost": "0.00299", "_task_id": 9} + captcha_solver.solve_hcaptcha = fake + try: + d = FakeDriver([_wall_showing("hcaptcha")], after_js=fixture("listing_dom.html")) + with tempfile.TemporaryDirectory() as tmp: + rc = _quiet(browser_bridge.run, _args(out=os.path.join(tmp, "o"), twocaptcha_key="k", + solver_proxy=PLACEHOLDER_PROXY), d) + solve = json.load(open(os.path.join(tmp, "o.meta.json")))["transport_facts"]["solves"][0] + finally: + captcha_solver.solve_hcaptcha = real + eq("hCaptcha wall solved -> rows", rc, 0) + eq("the wall's own sitekey, through the browser's exit", + (calls.get("sitekey"), calls.get("proxy")), ("070db171-ddb9-4c93-b7f6-d25d3c9d7e28", PLACEHOLDER_PROXY)) + check("token into #h-captcha-response of the page's own form", + any("#h-captcha-response" in j and "P1_EXAMPLE.token" in j and ".js-firewall-form" in j + for j in d.js)) + eq("recorded as an hCaptcha solve, verified", (solve["variant"], solve["verified"]), ("hcaptcha", True)) + + +FAKE_SELLER_HASH = "ab" * 16 # not a real id; this file is outside the secret scan + + +def _seller_page(n_cards, active_ads=353, with_hash=True): + """The REAL seller fixture with its card cloned to `n_cards` distinct ids, + and (optionally) the profile's own link carrying a sellerId — trimmed out + of the committed fixture because a 32-hex id reads as a credential.""" + from bs4 import BeautifulSoup + soup = BeautifulSoup(fixture("seller.html"), "html.parser") + cards = soup.select(P_SELLER_CARD()) + proto, proto_id = str(cards[0]), cards[0]["data-item-id"] + for c in cards: + c.decompose() + holder = soup.body + for i in range(n_cards): + holder.append(BeautifulSoup(proto.replace(proto_id, str(900000000 + i)), "html.parser")) + html = str(soup).replace("353", str(active_ads)) + if with_hash: + html = html.replace("", 'x' + % FAKE_SELLER_HASH) + return html + + +def P_SELLER_CARD(): + import product_parser + return product_parser.SELLER_CARD_SELECTOR + + +def _run_seller_with(driver, **kw): + import browser_bridge + with tempfile.TemporaryDirectory() as tmp: + out = os.path.join(tmp, "o") + rc = _quiet(browser_bridge.run, _args(mode="seller", url=[SELLER], out=out, **kw), driver) + meta = json.load(open(out + ".latest_attempt.meta.json")) + rows = json.load(open(out + "_listings.json")) if os.path.exists(out + "_listings.json") else [] + seller = json.load(open(out + ".json")) if os.path.exists(out + ".json") else [] + return rc, meta, rows, seller + + +def test_seller_feed_scrolls_to_the_profile_counter(): + """Owner decision 2026-10-09 (option A): a seller's listings beyond the + profile's 15 come from scrolling its feed.""" + d = FakeDriver([_seller_page(15, active_ads=40), _seller_page(15, active_ads=40)], + scrolls=[_seller_page(30, 40), _seller_page(40, 40)]) + rc, meta, rows, seller = _run_seller_with(d) + feed = meta["transport_facts"]["seller_feed"] + eq("seller run ok", rc, 0) + eq("all 40 listings, not the profile's 15", len(rows), 40) + eq("distinct ids", len({r["sku"] for r in rows}), 40) + eq("positions 1..40", [r["position"] for r in rows][:3] + [rows[-1]["position"]], [1, 2, 3, 40]) + eq("the feed URL is the allowed /all one", + feed["url"], "https://www.avito.ru/brands/seller1/all?sellerId=" + FAKE_SELLER_HASH) + eq("stopped on reaching the counter", (feed["rounds"], feed["stop"]), (2, "reached 40")) + check("the feed was armed with its own «Показать все» button", + feed["show_all_clicks"] >= 1 and any("show_all_button" in j for j in d.js)) + eq("seller row says how many listings were written", seller[0]["listings_on_page"], 40) + + +def test_seller_feed_respects_max_products_and_stops_when_still(): + d = FakeDriver([_seller_page(15, 353), _seller_page(15, 353)], + scrolls=[_seller_page(30, 353), _seller_page(45, 353)]) + rc, meta, rows, _ = _run_seller_with(d, max_products=20) + eq("--max-products caps the feed", len(rows), 20) + eq("and the scrolling stopped there", meta["transport_facts"]["seller_feed"]["rounds"], 1) + d2 = FakeDriver([_seller_page(15, 353), _seller_page(15, 353)], + scrolls=[_seller_page(30, 353)]) # then nothing new arrives + rc, meta, rows, _ = _run_seller_with(d2) + feed = meta["transport_facts"]["seller_feed"] + eq("a feed that stops growing ends the scroll", feed["stop"], "feed stopped growing") + eq("with what it had", len(rows), 30) + eq("after the still rounds, not forever", feed["rounds"], 4) + + +def test_seller_without_an_id_keeps_its_first_listings(): + # Profile and its /items page both without a sellerId. + d = FakeDriver([_seller_page(15, 353, with_hash=False), _seller_page(15, 353, with_hash=False)]) + rc, meta, rows, _ = _run_seller_with(d) + from output_writer import EXIT_PARTIAL + eq("no sellerId -> the profile's 15, reported PARTIAL (15 of 353)", (rc, len(rows)), + (EXIT_PARTIAL, 15)) + eq("status partial", (meta["status"], meta["stop_reason"]), ("partial", "seller_listings_short")) + eq("and the reason is recorded", meta["transport_facts"]["seller_feed"]["stop"], + "no sellerId on the profile") + + +def test_shop_sellerid_comes_from_its_items_page(): + """2026-10-10, /brands/a3boots.shop: the profile has no sellerId; its + `/brands//items` page does, and the `/all` feed then works.""" + import product_parser as P + eq("items URL has no ?s= (robots)", P.seller_items_url(SELLER), + "https://www.avito.ru/brands/seller1/items") + check("and robots allows it", P.robots_verdict(P.seller_items_url(SELLER))[0]) + d = FakeDriver([_seller_page(15, 40, with_hash=False), _seller_page(12, 40), + _seller_page(15, 40)], scrolls=[_seller_page(40, 40)]) + rc, meta, rows, _ = _run_seller_with(d) + feed = meta["transport_facts"]["seller_feed"] + eq("all 40 via the feed, exit 0", (rc, len(rows)), (0, 40)) + eq("id found on the items page", feed["sellerId_from"], "items page") + eq("feed URL", feed["url"], "https://www.avito.ru/brands/seller1/all?sellerId=" + FAKE_SELLER_HASH) + + +def test_the_wait_wall_is_a_refusal_not_a_challenge(): + """Measured 2026-10-08: a third firewall variant with neither a captcha + form nor a proof-of-work — «подождите немного и обновите страницу». It + does not clear in place; the exit has to change.""" + import product_parser as P + html = fixture("wall_wait.html") + eq("wait-wall -> blocked", P.detect_page_state(html, url=LISTING), "blocked") + eq("and with its 429", P.detect_page_state(html, status=429, url=LISTING), "blocked") + import page_flow + check("which rotates the exit", page_flow.should_rotate("blocked")) + + +def test_a_solve_followed_by_another_wall_is_not_verified(): + """The first version marked a solve `verified` as soon as the page was + 'not the wall' — and on 2026-10-08 an accepted solve was followed by the + wait-wall while the metadata said verified.""" + import browser_bridge + import captcha_solver + real = captcha_solver.solve_geetest_v4 + + def fake_solve(key, cid, url, proxy=None, **kw): + return {"captcha_id": cid, "lot_number": "l", "pass_token": "p", "gen_time": "1", + "captcha_output": "o", "_cost": "0.00299", "_task_id": 1} + captcha_solver.solve_geetest_v4 = fake_solve + try: + d = FakeDriver([fixture("wall_captcha.html")], after_js=fixture("wall_wait.html")) + with tempfile.TemporaryDirectory() as tmp: + rc = _quiet(browser_bridge.run, _args(out=os.path.join(tmp, "o"), twocaptcha_key="k", + solver_proxy=PLACEHOLDER_PROXY), d) + solve = json.load(open(os.path.join(tmp, "o.latest_attempt.meta.json")))[ + "transport_facts"]["solves"][0] + finally: + captcha_solver.solve_geetest_v4 = real + eq("solve followed by the wait-wall -> blocked (3)", rc, 3) + eq("NOT verified", solve["verified"], False) + eq("and says what came instead", solve.get("after_submit"), "blocked") + + def test_a_solver_failure_is_a_warning_not_a_crash(): import browser_bridge import captcha_solver real = captcha_solver.solve_geetest_v4 def failing(*a, **k): - raise captcha_solver.SolverError("ERROR_CAPTCHA_UNSOLVABLE") + raise captcha_solver.SolverError("getTaskResult: ERROR_CAPTCHA_UNSOLVABLE (task 84066204532)", + task_id=84066204532, error_code="ERROR_CAPTCHA_UNSOLVABLE") captcha_solver.solve_geetest_v4 = failing try: with tempfile.TemporaryDirectory() as tmp: @@ -542,9 +832,13 @@ def failing(*a, **k): _args(out=os.path.join(tmp, "o"), twocaptcha_key="k", solver_proxy=PLACEHOLDER_PROXY), FakeDriver([fixture("wall_captcha.html")])) + solve = json.load(open(os.path.join(tmp, "o.latest_attempt.meta.json")))[ + "transport_facts"]["solves"][0] finally: captcha_solver.solve_geetest_v4 = real eq("solver failure -> blocked (3), not crash (1)", rc, 3) + eq("the failed solve is traceable: task id and 2Captcha's own code", + (solve["task_id"], solve["error_code"]), (84066204532, "ERROR_CAPTCHA_UNSOLVABLE")) def test_a_dead_exit_rotates_a_local_browser(): @@ -554,10 +848,14 @@ def test_a_dead_exit_rotates_a_local_browser(): args = _args(proxy=GATEWAY_PROXY + "", proxy_block_retries=1) args.proxy = args.proxy.replace("region-RU", "region-RU-session-abcdefghi") args.solver_proxy = args.proxy + err = io.StringIO() with tempfile.TemporaryDirectory() as tmp: args.out = os.path.join(tmp, "o") - rc = _quiet(browser_bridge.run, args, d) + with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(err): + rc = browser_bridge.run(args, d) eq("a dead exit was replaced and the page fetched", rc, 0) + check("the rotation names Chromium's error, not a URL fragment", + "the exit failed (ERR_CONNECTION_CLOSED)" in err.getvalue(), "(got %r)" % err.getvalue()[-200:]) eq("a fresh browser was started", d.starts, 2) check("the solver follows the browser to the new exit", args.solver_proxy == args.proxy) check("on a NEW session", "session-abcdefghi" not in args.proxy) @@ -909,7 +1207,8 @@ def test_engine_parity(): continue drivers = [v for k, v in vars(module).items() if k.endswith("Driver") and isinstance(v, type)] check("%s defines one driver" % name, len(drivers) == 1) - for method in ("start", "navigate", "content", "count", "run_js", "sleep", "stop"): + for method in ("start", "navigate", "content", "count", "run_js", "scroll_to_bottom", + "sleep", "stop"): check("%s.%s" % (name, method), callable(getattr(drivers[0], method, None))) check("%s has main()" % name, callable(getattr(module, "main", None))) d = drivers[0](_args())