Remove donor-site leftovers

.dockerignore

@@ -11,7 +11,8 @@ __pycache__/
 .venv/
 venv/
 env/
-dubizzle_listings.*
+dubizzle_products.*
+dubizzle_products_*.*
 *_debug.html
 *_debug.png
 live_results/

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -92,7 +92,7 @@ body:
       placeholder: |
         python3 playwright_scraper.py \
           --url "https://uae.dubizzle.com/motors/used-cars/" \
-          --pages 3 --out grills
+          --pages 3 --out cars
     validations:
       required: true
 
@@ -112,7 +112,7 @@ body:
     attributes:
       label: What you expected instead
       placeholder: >-
-        96 products, as the README says a category page yields.
+        25 ads a page, as the README says a motors listing yields.
     validations:
       required: true
 

.github/ISSUE_TEMPLATE/site_changed.yml

@@ -21,27 +21,24 @@ body:
           instead `brand`, `seller_name` or `in_stock` are empty while
           everything else is fine, it is the page's JSON-LD ItemList or the
           path-keyed join to it, not the payload.
-          not carry it: bids arrive at hydration.
 
-        So "no rows at all" and "rows with no prices" are two different
+        So "no rows at all" and "no brand / seller_name" are two different
         breakages with two different fixes, and `price_source` on the rows
-        tells you which happened: `next_data+dom` is healthy, `next_data`
-        means nothing hydrated.
+        tells you which source each row's price came from: `next_data+jsonld`
+        where the page publishes both, `next_data` on the verticals with no
+        ItemList, `dom` on the fallback path.
 
   - type: dropdown
     id: what_broke
     attributes:
       label: What stopped matching
       options:
-        - No rows at all on a page that visibly has lots (the payload moved)
-        - Rows come out with no prices (the card, or its price class, moved)
+        - No rows at all on a page that visibly has ads (the payload moved)
+        - Rows come out with no prices on motors, property or classified
         - brand / seller_name / in_stock empty (the JSON-LD join moved)
-        - A null price with no reserve_price_set flag (the invariant broke)
-        - favorite_count empty on every row (the card's counter moved)
-        - Pagination stops early, repeats a page, or runs past the 100 cap
+        - Pagination stops early, repeats a page, or runs past the site's totalPages
         - One vertical returns nothing while the others are fine
         - The Arabic pages lost brand / seller_name / in_stock
-        - A no-results search returns 24 suggested lots as if they were real
         - Something else
     validations:
       required: true
@@ -52,8 +49,8 @@ body:
       label: URL
       description: >-
         The exact URL, its locale prefix, and how the browser was reached.
-        Please say whether the run was headful or over --cdp-endpoint: a
-        headless run is refused by the site and is not a parser problem.
+        Please say which country the exit was in: an exit outside the UAE is
+        refused by the site and is not a parser problem.
       placeholder: "https://uae.dubizzle.com/motors/used-cars/ (English, headless Chromium over --cdp-endpoint, exit: UAE residential)"
     validations:
       required: true
@@ -74,8 +71,8 @@ body:
           here (`AED` in one, the digits in
           `[data-testid="listing-price"]`), so both are needed.
         - For a missing-column problem: which vertical, and whether the
-          page has a JSON-LD ItemList at all
-          ("Current bid", "Aktuelles Gebot", ...) and the locale.
+          page has a JSON-LD ItemList at all, and the locale (`/ar/` or
+          none).
       render: text
     validations:
       required: true

.gitignore

@@ -15,8 +15,8 @@
 # real captures, verified to parse identically to them and scrubbed;
 # it has to be IN the repo or the suite cannot run on a clean clone.
 !fixtures_generated.json
-dubizzle_listings.*
-dubizzle_listings_*.*
+dubizzle_products.*
+dubizzle_products_*.*
 
 # --- the live-test harness's working directory ---------------------------
 # `--dump-html PATH` writes PATH.page1, PATH.page2, ... — FILES, not a

CHANGELOG.md

@@ -23,6 +23,49 @@ user would notice, the release notes say so first.
   classifier. It now reads `http_code`, falling back to `status` only if that
   is an integer. After the fix, one live call (`--wait-text AED` on the canary's used-cars listing, no `--cdp-url`) answered HTTP 200 (no 422) with an integer upstream status, 200; the body was a 1,156-byte Incapsula interstitial, correctly reported as blocked (exit 3), as the module docstring already says for the API's own exit.
 
+- **Prose and strings inherited from the sites this repo was copied from.**
+  Several comments, docstrings and user-visible strings still described
+  another site's behaviour as if it were dubizzle's. Rewritten to what this
+  repo's own README measures, or deleted where it has no equivalent:
+  - `--help` text: `scraper_api_client.py` called itself an "auction
+    scraper" and described a scrolling grid and a `/p/<cat>/…` URL shape;
+    `diff_runs.py --price-tolerance-pct` said the site quotes IDR; the
+    Playwright engine's `--pages` help described an infinitely scrolling
+    search page.
+  - The Playwright engine's `--concurrency` warning said dubizzle refuses a
+    headless browser by answering nothing at all. It refuses exits outside
+    the UAE.
+  - The engines, `captcha_solver.py` and `output_writer.py` described an
+    Akamai refusal of HEADLESS browsers (a 394-byte "Access Denied"), a
+    `--mode product` / `--mode shop`, a search that cannot be paginated and
+    a `/p/<slug>` discovery hub. None of those exist here: the refusal is
+    Imperva's, triggered by the exit's country, and the only mode is
+    `listing`. A contradictory "HEADFUL by default" comment is removed from
+    all three engines and the suite (the default is, and was, headless).
+  - `diff_runs.py` said the site quotes euro on 18 locales and described
+    auctions and bids. It quotes AED in two languages; what moves without a
+    price change is the Car of the Week slot.
+  - An unused `shop_facts` field is removed from all three engines.
+- `CONTRIBUTING.md`'s "Reporting a site change" described another site's
+  selectors and page kinds; rewritten for the SSR payload, the JSON-LD join
+  and the DOM fallback this repo actually reads. Its note that a datacentre
+  run gets no response at all is corrected to the measured Imperva refusal.
+- The site-change issue template carried another site's options (reserve
+  prices, favourite counts, suggested lots, a 100-page cap) and said a
+  headless run is refused. The bug-report template's example used another
+  repo's output prefix and product count.
+- `.gitignore` / `.dockerignore` ignored `dubizzle_listings.*`, which no
+  engine writes; the default `--out` prefix is `dubizzle_products`.
+- The suite's env-placeholder and proxy-line checks used another repo's
+  variable names; they now use `DUBIZZLE_CDP_ENDPOINT` and `DUBIZZLE_PROXY`.
+- README: the last badge read "runs without an account", which is not true
+  here — a plain run needs a UAE exit. It now says so.
+
+- `SECURITY.md` said this project has no releases or version tags; it has
+  both. "Supported versions" now names the latest release and `main`.
+- `captcha_solver.py`'s docstring pointed at a "No DataDome solver" section
+  that does not exist in this repo (it came with the copied core). Removed.
+
 ## [0.1.1] — 2026-09-16
 
 ### Fixed

CONTRIBUTING.md

@@ -42,38 +42,25 @@ not the check caught it.
 
 dubizzle changing its markup is the normal way this stops working, and it
 has its own issue template. The detail that saves the most time is WHICH
-anchor broke, because on this site there is no structured data on a listing
-page to fall back on — measured zero `application/ld+json`, zero
-`__NEXT_DATA__` and zero Apollo state across six captures — so the DOM is
-not the primary path by preference, it is the only one.
-
-1. **The grid container.** `[data-testid="divSRPContentProducts"]` on a
-   search page, `[data-ssr="productsCategoryL2/L3SSR"]` on a category
-   listing. If one of these moves the run reports 0 rows and exit 4, which
-   is loud.
-2. **The tile marker.** `[data-testid="imgLeg-c"]` on a search page (one per
-   tile), `[data-testid="divProductWrapper"]` inside
-   `a[data-testid="lnkProductContainer"]` on a category listing.
-3. **The reading ORDER inside the tile** — badge, title, price, was-price,
-   rating, sold, shop, location. The field reads rest on it, deliberately,
-   because the classes around each field are build hashes:
-   `<span class="+tnoqZhn89+NHUA43BpiJg==">` is the title today. If dubizzle
-   reorders a tile, `title` and the prices are what break.
-4. **`span.flip`**, the shop name and the shop's city in that order, exactly
-   two per search tile.
-
-The one place structured data does exist is a DETAIL page's
-`window.__cache` Apollo blob, which is where `--mode product` reads the real
-product id, the exact sold count and the review count.
+source broke, because a row here is read from more than one:
+
+1. **The SSR payload** — `<script id="__NEXT_DATA__">`, the Redux action
+   `listings/fetchListingDataForQuery/fulfilled`. Every ad on every vertical,
+   with its ids, url, price, location, attributes and the pagination
+   contract. If this moves the run reports 0 rows and exit 4, which is loud.
+2. **The JSON-LD ItemList**, on motors and property only — the one source
+   for `brand`, `seller_name` and `in_stock`, joined to the payload on the
+   ad's locale-stripped path.
+3. **The DOM fallback**, anchored on the ad-URL pattern (every ad path
+   carries `/{yyyy}/{m}/{d}/`), with the tile's price read from the wrapper
+   around `[data-testid="listing-price"]`. Every class on a tile but
+   `lpv-cards` is a build hash, so nothing is anchored on one.
 
 A third thing can break without any path failing: the **join** between the
-tiles and the structured data. When it breaks, the row count and the prices
-stay healthy while `in_stock` and part of `brand` quietly empty out — so
-every run logs its structured-price confirmation share per page and warns
-below a floor set PER PAGE KIND (search 8%, category 70%, shop 80%; the
-achievable share differs by a factor of eight between them). If you are
-reporting a change, that percentage and the page kind are the numbers to
-include.
+payload and the JSON-LD. When it breaks, the row count and the prices stay
+healthy while `brand`, `seller_name` and `in_stock` quietly empty out. If
+you are reporting a change, say which vertical and which language (`/ar/`
+or none) — the Arabic join is the one that broke before.
 
 `--dump-html PATH` writes the exact bytes the parser was given, on success as
 well as failure, and a run that finds nothing writes a dump and a screenshot
@@ -223,25 +210,24 @@ Most do not — the suite covers the parser, the writers, the captcha classifier
 and the CLI contract against inline fixtures. If yours genuinely needs
 dubizzle.com, say in the PR what you ran, which URL and page kind, from
 which exit, and what you got — including the price and image coverage
-percentages the run prints, and the scroll trace from the sidecar. Note that
-a run from a datacentre address gets NO RESPONSE AT ALL, so "it returned
-nothing" from a VPS is not a finding. Product counts differ by category, by
-URL and by how far the scroll got, so a bare "worked for me" is not
-reproducible.
+percentages the run prints. Note that a run from an exit outside the UAE
+is refused by Imperva — a 403, or a "Pardon Our Interruption" page under
+HTTP 200 — so "it returned nothing" from a non-UAE address is not a
+finding. Ad counts differ by vertical and by URL, so a bare "worked for me"
+is not reproducible.
 
 **Run more than the primary engine.** "Mirror them exactly" is a design rule,
-not a verification: the first live run of the pyppeteer engine crashed on its
-FIRST fetch on a signature mismatch that four separate offline checks and 400
-green assertions had not caught.
+not a verification: in a sibling repo (tokopedia-scraper) the first live run
+of the pyppeteer engine crashed on its FIRST fetch on a signature mismatch
+that four separate offline checks and 400 green assertions had not caught.
 
-Do not add anything that submits the registration form. This project
-deliberately never does, and a captcha token proved valid by creating a real
-account is not a result worth having.
+Do not add anything that submits a form on the site. This project
+deliberately never does.
 
 ## Scope
 
-This repo scrapes **public pages** on dubizzle: category listings, search
-listings and product pages, exactly as an anonymous visitor is served them.
+This repo scrapes **public pages** on dubizzle: category listings on
+`uae.dubizzle.com`, exactly as an anonymous visitor is served them.
 Out of scope: anything behind a login, anything that submits a form, and
 anything that defeats a protection rather than passing it the way an ordinary
 browser does.

README.md

@@ -6,7 +6,7 @@
 [![python](https://img.shields.io/badge/python-3.9%20%7C%203.13-blue)](pyproject.toml)
 [![licence](https://img.shields.io/badge/licence-MIT-green)](LICENSE)
 [![engines](https://img.shields.io/badge/engines-Playwright%20%7C%20Selenium%20%7C%20pyppeteer%20%7C%20CDP-informational)](#engines)
-[![runs without an account](https://img.shields.io/badge/runs%20without-an%20account-brightgreen)](#do-you-need-any-of-the-paid-products)
+[![needs a UAE exit](https://img.shields.io/badge/needs-a%20UAE%20exit-orange)](#do-you-need-any-of-the-paid-products)
 
 Scrapes [dubizzle](https://uae.dubizzle.com) UAE classifieds — cars, property,
 electronics, jobs and community services — to JSON or CSV, with four

SECURITY.md

@@ -79,9 +79,9 @@ Not because these do not matter, but because they belong somewhere else:
 
 ## Supported versions
 
-`main` only. This project has no releases or version tags; fixes land on `main`
-and you update by pulling. If you are running an old clone, update before
-reporting.
+The latest release and `main`. Fixes land on `main` first and ship in the
+next tagged release (see the Releases page). If you are running an old clone,
+update before reporting.
 
 ## If you have leaked a key
 

captcha_solver.py

@@ -23,8 +23,6 @@
 purchase, and `page_flow.STATE_POLICY` — not this file — decides which state
 is worth money at all.
 
-There is deliberately no DataDome path here. See "No DataDome solver" below.
-
 Flow:
   1. Both detectors run and are reconciled (see reconcile_detections) to decide
      the variant: v3, v2-invisible or v2-checkbox. The parameters differ per
@@ -710,23 +708,22 @@ class CaptchaUnsolvable(RuntimeError):
 # a JPEG of distorted text and a GET form). Roughly 190 lines of it, and none
 # of it is ported here, because this site has no such page.
 #
-# What this site does instead is refuse a HEADLESS browser. Measured
-# 2026-09-10 from five different addresses, four of them residential: Akamai
-# answers with HTTP 403 and a 394-byte "Access Denied" page carrying a
-# reference id — no form, no image, no widget, nothing for a solver to
-# answer. And the trigger is the CLIENT rather than the address: the very
-# same addresses were served HTTP 200 and the full catalogue by a browser
-# with a real window. So the response to a block here is `--headful` or
-# `--cdp-endpoint`, not a solve and not a better proxy, and
-# product_parser.detect_page_state reports it as "blocked" rather than
-# "challenge" precisely so no solve is attempted and nothing is charged.
+# What this site does instead is refuse an exit outside the UAE. Imperva
+# answers with an HTTP 403 or a "Pardon Our Interruption" page (see the
+# README's table, measured 2026-09-14), and the trigger is the ADDRESS's
+# country rather than the client: a UAE residential exit was served the full
+# catalogue by headless Chromium. So the response to a block here is a UAE
+# exit (`--proxy` with region-ae, or `--cdp-endpoint` with country-ae), not
+# a solve, and product_parser.detect_page_state reports it as "blocked"
+# rather than "challenge" precisely so no solve is attempted and nothing is
+# charged.
 #
 # The reCAPTCHA / hCaptcha / Turnstile machinery above IS kept, and that is a
 # deliberate asymmetry rather than an inconsistency. Detection stays broad
 # because which challenge a visitor meets depends on the exit country and on
 # what the address has been doing — a narrow list is how a challenge gets
 # reported as an empty page months later. A solver for a challenge this site
 # has never been observed to serve is dead code; a DETECTOR for one is cheap
-# insurance: a detection that fires on a page whose lots have
+# insurance: a detection that fires on a page whose ads have
 # already rendered guards nothing, which is why the default is
-# `when-blocked` and why it counts lot links before it spends.
+# `when-blocked` and why it counts ad links before it spends.