avito-scraper v1.0.1

CHANGELOG.md

@@ -6,10 +6,11 @@ follow SemVer as closely as a CLI toolkit can: a **patch** means fixes, and a
 fix may change a default when the old one was wrong — such a change leads its
 entry in a blockquote.
 
-## [1.0.1] — 2026-10-07
+## [1.0.1] — 2026-10-10
 
-Fixes from an external audit of v1.0.0. Each finding was reproduced before
-it was fixed, and each now has a regression test.
+Fixes from an external audit of v1.0.0, and from live runs on 2026-10-08,
+-09 and -10. Each finding was reproduced before it was fixed, and each now
+has a regression test.
 
 > **Behaviour changes for existing users.**
 > * An **unexpected exception** now exits **1** (crash, traceback printed
@@ -18,6 +19,8 @@ it was fixed, and each now has a regression test.
 > * `diff_runs.py` now **refuses** two outputs when either has no readable
 >   `.meta.json`, or when they were written by different row schemas. Pass
 >   `--force` to compare anyway.
+> * `--mode seller` now exits **6** (partial) when it wrote fewer listings
+>   than the profile's own counter, instead of 0.
 
 ### Fixed
 
@@ -31,6 +34,52 @@ it was fixed, and each now has a regression test.
 * **No attempt metadata on a failed start or a crash.**
   `<out>.latest_attempt.meta.json` is now written on every ending, with
   `exit_code`, `error_class` and a masked `error_message`.
+* **A failed captcha solve is now traceable.** Every solve in
+  `transport_facts.solves` records the 2Captcha `task_id` and its own error
+  code (`ERROR_CAPTCHA_UNSOLVABLE`, `TIMEOUT`, …), and the solver waits 240 s
+  instead of 180 s: on 2026-10-08 2Captcha took 217–220 s to return
+  `ERROR_CAPTCHA_UNSOLVABLE` for this captcha, so the old wait abandoned
+  every task before its verdict and logged only "no solution".
+* **A third firewall variant** — «подождите немного и обновите страницу», no
+  captcha form, no proof-of-work, served right after an accepted solve
+  (measured 2026-10-08) — is now a refusal of the exit and rotates, instead
+  of being waited out as a challenge and reported blocked.
+* **A solve is `verified` only when the page actually came back.** The first
+  release marked it verified as soon as the document stopped being the wall,
+  which included that third variant. A solve that is followed by something
+  else records `after_submit`. A proof-of-work after a solve is waited out.
+* **The wall's other two captchas.** The page can show Avito's own picture
+  captcha or hCaptcha instead of GeeTest; the scraper now recognises which
+  one is showing and records it (`solves[].variant`). The picture captcha is
+  solved with `ImageToTextTask` and answered through the page's own field —
+  implemented from the wall's script, not yet met live (every measured
+  request was assigned GeeTest). hCaptcha is solved with `HCaptchaTask`
+  (sitekey from the wall, the browser's own exit, `enterprisePayload.rqdata`
+  only if the wall ever carries one) and its token submitted through
+  `#h-captcha-response` — also not yet met live. One live `HCaptchaTask` on
+  avito's sitekey (2026-10-09) was accepted by the API and returned
+  ERROR_CAPTCHA_UNSOLVABLE after 51 s: the request shape is right, a solved
+  token for this wall is not yet measured.
+* **`--mode seller` now returns all of a seller's listings**, not the 15
+  the profile renders: it opens the seller's feed, presses «Показать все»
+  and scrolls until the profile's own counter or `--max-products`
+  (measured: 353 of 353). The feed's scroll makes the page fetch
+  `/web/1/profile/items`, which robots.txt disallows — README says so; this
+  is a deliberate choice. `--max-products 15` keeps to the profile page.
+* **A recommendation carousel was read as search results.** On 2026-10-10
+  `?q=yeezy` page 1 carried a «Подобрали для вас» block INSIDE the results
+  grid, with 31 cards of the same shape; page 1 came back with 82 rows, 32
+  of them unrelated. Cards inside `itemsCarousel` are no longer results.
+* **A seller run with fewer listings than the profile states is now
+  `partial` (exit 6).** A shop stating 7331 active listings was written as a
+  `complete` run of 12, because the profile's tabbed counter («Активные |
+  N») was not read and the feed never started. Both are fixed; a shop
+  profile without a `sellerId` now takes it from `/brands/<slug>/items`
+  (measured: 100 of 100 with `--max-products 100`).
+* The seller's name falls back to the profile heading when the page title
+  does not carry it (one fetch of two, 2026-10-10).
+* Rotation messages name Chromium's error (`ERR_CONNECTION_RESET`) instead of
+  a fragment of the URL.
 * **`schema_version`** now fingerprints the row class of the run's mode
   (names, order and types), not `Product`'s field names only.
 * Leftovers from the repositories this one was scaffolded from, in

README.md

@@ -35,12 +35,21 @@ residential exit.
 
 avito.ru sits behind its own firewall, «Доступ ограничен: проблема с IP».
 Its own text says why: *too many requests from this address*. It answers in
-one of two ways:
+one of these ways:
 
 | What you meet | Status | What clears it |
 |---|---|---|
 | a proof-of-work page | 439 | the browser itself, in a few seconds — the page runs `/web/3/firewallPow/…` and reloads |
 | a captcha page, «Продолжить» → **GeeTest v4** slide puzzle | 429 | a solve submitted through the page's own form |
+| «подождите немного и обновите страницу» — no captcha, no proof-of-work | 429 | a different exit. Measured: this is what follows a **rejected** solution |
+
+The captcha page can carry two other captchas — Avito's own picture
+captcha and hCaptcha. The server picks one per request; **every request
+measured (2026-10-06 to 10-09) got GeeTest**, including the page's own
+"new picture" request. The picture captcha is implemented (`ImageToTextTask`)
+but has never been met live. hCaptcha is solved with `HCaptchaTask`
+(sitekey from the wall, through the browser's own exit) — likewise
+implemented from the wall's code and 2Captcha's task format, never met live.
 
 | Client | Result |
 |---|---|
@@ -181,9 +190,21 @@ largest size), `category_path`, `address`, `lat`, `lng`, `views_total`,
 
 `<out>.json`: `seller_name`, `seller_rating`, `seller_reviews`,
 `seller_since`, `subscribers`, `active_ads` (the profile's own counter),
-`listings_on_page`, `badges`. `<out>_listings.json`: the cards the profile
-page renders, as listing rows — **15 measured**, of 107–353 active; the rest
-load as the page scrolls and are not fetched by this version.
+`listings_on_page`, `badges`. `<out>_listings.json`: **all** the seller's
+listings, as listing rows, up to the profile's own counter or
+`--max-products`. The profile page renders 15; the rest come from the
+seller's feed (`/brands/<slug>/all?sellerId=…`), which the scraper opens,
+arms with its «Показать все» button and scrolls — 15 more per scroll.
+Measured 2026-10-09: **353 of 353** in 22 scrolls, every row priced.
+Compare `listings_on_page` with `active_ads` to see whether the feed was
+read to the end; `transport_facts.seller_feed` says why it stopped.
+
+**About robots.txt and the seller feed.** The scraper never requests a path
+avito.ru's robots.txt disallows. But scrolling the feed makes the PAGE'S OWN
+script fetch `/web/1/profile/items`, and `/web/` is disallowed for robots.
+That is a deliberate choice (owner, 2026-10-09) — the only way to a
+seller's listings beyond 15. Use `--max-products 15` to stay on the
+profile page alone.
 
 ### Seller names, and what is never collected
 
@@ -356,8 +377,10 @@ details, and does not message sellers. When avito.ru's firewall presents a
 captcha, the scraper solves it the way a visitor would — through the page's
 own form — with a solve you pay 2Captcha for.
 
-`robots.txt` is respected: the snapshot is in `robots_snapshot.py`, and URLs
-it disallows are refused before any request is made.
+`robots.txt` is respected for every URL the scraper requests: the snapshot
+is in `robots_snapshot.py`, and URLs it disallows are refused before any
+request is made. One exception is stated above: scrolling a seller's feed
+makes the page itself fetch the disallowed `/web/1/profile/items`.
 
 The firewall exists because of request volume. Default `--delay` is 2 seconds
 between pages, `--concurrency` above 1 is refused, and running hard from one

TROUBLESHOOTING.md

@@ -17,6 +17,12 @@ which variant you met:
 |---|---|---|
 | a page that reloads itself, requests to `/web/3/firewallPow/…` | 439 | the browser clears it within seconds; the scraper waits 25 s |
 | «Продолжить» and, after it, a GeeTest slide puzzle | 429 | the scraper solves it (below) |
+| «подождите немного и обновите страницу», nothing to solve | 429 | follows a REJECTED solution; the scraper moves to a fresh exit (`--proxy-block-retries`) |
+| «введите символы с картинки» — Avito's picture captcha | 429 | solved with `ImageToTextTask`; never met live yet — if you see it, `solves[].variant` is `image` and a report is welcome |
+| hCaptcha («поставьте галочку») | 429 | solved with `HCaptchaTask` through the browser's exit; never met live yet — `solves[].variant` is `hcaptcha` |
+
+`transport_facts.solves[]` names the variant of every wall met, its 2Captcha
+`task_id` and, on failure, 2Captcha's own error code.
 
 **The GeeTest wall stays up after a solve.** Look at `transport_facts.solves`: