Zimmo.be real-estate listing scraper (Playwright, Selenium, Puppeteer, or the 2Captcha Scraping Browser API via CDP) — sale and rent listings, prices, living area, bedrooms, EPC labels, proxies, captcha solving.
It reads zimmo.be search-results pages — sale and rent, the Dutch
(/nl/…/te-koop/) and French (/fr/…/a-vendre/) sites — and writes one
row per listing: price, address, property type, living area, bedrooms and
energy grade, plus the listing's own Zimmo code as sku.
The site serves an Angular grid with no listing data in any structured
block (its only JSON-LD is a BreadcrumbList, and its embedded Angular
state carries headings and counts, not rows). So the rows come from the
rendered tiles, anchored on markup the site uses as a contract rather than
on build-generated classes: each listing is its own <zimmo-listing>
element, the listing code sits in .zimmo-code, and bedrooms, living area
and energy grade are stated by icon file (bedrooms.svg,
floorspace-surface.svg, energy-labels/epc_b.svg), which is identical on
the Dutch and French sites while the labels beside it are translated.
Measured 2026-09-23 on 105 tiles over 7 live pages (gent-9000/te-koop
pages 1–3, antwerpen-2000/te-huur, bruxelles-1000/a-vendre), checked field
by field against the tiles' own accessibility labels: 0 mismatches.
Coverage on those pages: price 99% (the rest are "Prijs op aanvraag",
written as a row with price and currency null), property and listing
type 100%, living area 90%, bedrooms 72%, energy grade 70% — the gaps are
listings that state no area, no bedrooms (garages, commercial property) or
no grade.
No 2Captcha key. But a residential exit and a visible browser. Every
run below is one attempt, 2026-09-23, same URL
(/nl/gent-9000/te-koop/):
| From | Browser | Result |
|---|---|---|
| datacentre (Hetzner), no proxy | curl, any User-Agent | HTTP 403 "Just a moment…" |
| datacentre (Hetzner), no proxy | Chromium headless / headful | Cloudflare challenge, exit 3 |
| GitHub Actions runner, no proxy | Chromium headful under xvfb | Cloudflare challenge, exit 3 |
| Belgian residential exit | Chromium headless | Cloudflare challenge, exit 3 |
| Belgian residential exit | Chromium headful | served — 63 rows, 3 of 3 pages, exit 0 |
| German residential exit | Chromium headful | served — 21 rows, exit 0 |
2Captcha Scraping Browser API, country-be |
its own | served — 63 rows, 3 of 3 pages, exit 0 |
| 2Captcha Scraper API | its own | served — 21 rows, exit 0 |
So on a home connection, --headful is the whole requirement — the exit
does not have to be Belgian. On a server, you need a residential proxy
(ZIMMO_PROXY in .env) and a display for the headful browser
(xvfb-run), or the Scraping Browser API, which brings both. Headless is
refused even from a residential address, which is why every example below
passes --headful; the flag's default stays headless for parity with the
rest of this family.
All three engines were run live through the Belgian exit, headful, 3 pages: Playwright, Puppeteer and Selenium each returned the same 63 listings at the same prices. (Selenium cannot authenticate a proxy, so it was given a local forwarder; see Engines.)
What the paid products buy here: the Scraping Browser API is the one path that needs neither a residential address nor a display of your own, which is what a server-side pipeline lacks; residential proxies give a local browser the address it needs; the Scraper API returns a page with no browser at all, one request per page ($0.0005 per request on 2026-09-23). The captcha solver was not exercised in these runs: no challenge appeared on a residential exit with a headful browser, and a solve against the Cloudflare interstitial the other paths meet was not tested here.
fingerprint_client.py is shipped but not wired into any engine yet.
git clone https://github.com/2scraper/zimmo-scraper.git
cd zimmo-scraper
pip install -r requirements.txt -r requirements-playwright.txt --break-system-packages
playwright install chromium
cp .env.example .env # on a server: set ZIMMO_PROXY to a residential proxy
python3 playwright_scraper.py --url "https://www.zimmo.be/nl/gent-9000/te-koop/" --headful
# on a server with no display:
xvfb-run -a python3 playwright_scraper.py --url "https://www.zimmo.be/nl/gent-9000/te-koop/" --headfulSelenium and Puppeteer (pyppeteer) work the same way, with their own
requirements-{engine}.txt — see Engines for why you
should install exactly one at a time.
python3 playwright_scraper.py \
--url "https://www.zimmo.be/nl/gent-9000/te-koop/" \
--pages 3 \
--headful \
--category gent-koop \
--format both \
--out gent_listingsWith the 2Captcha Scraping Browser API instead of a local browser — put
the endpoint in .env, never on the command line, since a password in
argv is readable by anything that can run ps:
# .env
ZIMMO_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-be-pid-{profileId}:{password}@cb.2captcha.com:9222python3 playwright_scraper.py --url "https://www.zimmo.be/nl/gent-9000/te-koop/" --pages 3A profile's credentials can stop being valid (no fixed lifetime is
promised here): an HTTP 401 deny_no_user on connect usually means the
endpoint's credentials are no longer accepted, not a broken scraper, and
the run reports it as exit 5. Check the endpoint before changing code.
Reuse the same pid across runs rather than minting a new one each time
— the Scraping Browser API caps live connections per profile, and a
persistent pid keeps its cookies (and therefore its trust with
Cloudflare) between runs.
A run holds ONE connection for all its pages, each page in a fresh tab (Playwright and pyppeteer engines). Reconnecting per page, as versions before 1.2.1 did, was refused with HTTP 500 roughly every other page — measured 2026-10-07: five connections and two refusals for a 3-page run before, one connection and none after. The log's last line says how many connections the run opened; more than one means an attempt failed and was retried on a fresh connection.
Without a browser at all, via the Scraper API (one page per call, key
from TWOCAPTCHA_KEY in .env):
python3 scraper_api_client.py --url "https://www.zimmo.be/nl/gent-9000/te-koop/"| Flag | Default | Notes |
|---|---|---|
--url |
— (or ZIMMO_URL) |
A zimmo.be search-results URL |
--category |
none | Label written into every row |
--pages |
1 | Number of result pages (1 or more) |
--delay |
0.0s | Sleep this long after each successfully-fetched page (rate limiting) |
--retries / --retry-delay |
2 / 3.0s | Extra attempts per page (0 = one attempt; negative values are refused) and the wait between them |
--concurrency |
1 | Parallel workers for pages 2+ (page 1 always fetched alone; refused together with --cdp-endpoint) |
--format |
both | json, csv, or both |
--out |
zimmo_listings |
Output file prefix |
--proxy / --proxy-file |
none (or ZIMMO_PROXY) |
A single proxy, or one-per-line file |
--proxy-rotate / --no-proxy-rotate |
rotate ON | Rotate to a different exit immediately on any proxy failure, or retry the same one |
--proxy-shuffle / --proxy-block-retries |
off / 3 | Pool behaviour |
--twocaptcha-key |
none (or TWOCAPTCHA_KEY) |
Needed only if Cloudflare/reCAPTCHA actually appears |
--solve-captcha |
when-blocked |
always solves on every detection; when-blocked only when the page has no usable content yet; never detects but never solves |
--min-score |
solver default (0.7) | Minimum reCAPTCHA v3 score to request |
--cdp-endpoint |
none (or ZIMMO_CDP_ENDPOINT) |
Connect to an existing browser instead of launching one; refused together with --proxy/--proxy-file, since that browser uses its own exit |
--allow-empty |
off | Write output even if zero products were found |
--dump-html |
off | Always save a per-page debug .html/.png, including on success |
--headless / --headful |
headless | Toggle a visible browser window |
The full set above is identical across all three engines — asserted
automatically by smoke_test.py, not just by having copied one file
into the other two.
Secrets belong in .env, never on a shared command line or in shell
history — copy .env.example to .env and fill in real values. An
explicit CLI flag always overrides .env; an already-exported shell
variable always overrides .env too. A value still containing {...}
(a vendor's own placeholder braces, copied verbatim from documentation)
is treated as unset, the same as an empty value. Run python3 env_config.py to see which source each variable actually came from,
without ever printing a secret's real value.
Playwright is primary. Selenium and Puppeteer (via pyppeteer) exist
for parity, not as equally-recommended defaults — pyppeteer itself is
effectively unmaintained.
Install exactly one engine at a time (or use a separate virtualenv
per engine): Playwright and Puppeteer declare mutually unsatisfiable
pyee pins, and Puppeteer/Selenium collide on urllib3. pip check
will tell you if your environment has both — confirmed in this repo's
own development sandbox, where installing all three at once reproduced
both conflicts exactly as described.
Two confirmed engine-specific limits:
- Selenium cannot use an authenticated remote CDP endpoint.
debuggerAddresswas designed for a local, unauthenticated debug port and does not forward auser:passembedded in a remote CDP URL. Useplaywright_scraper.pyorpuppeteer_scraper.pyinstead if you need--cdp-endpointagainst an authenticated endpoint. - Selenium's
--proxy-servercannot authenticate at all. A credential embedded in--proxyis stripped, with a warning, rather than silently doing nothing useful.
Each run writes <out>.json / <out>.csv (per --format) plus
<out>.meta.json, a sidecar recording status (complete / partial
/ empty), stop_reason, which pages failed by number, the product
count, the search it covered (selection: the URL without its page
parameter) and whether it read the listing to its end
(listing_exhausted, with the pages that made unnecessary in
exhausted_pages). A run that finds zero products writes nothing at
all by default — pass --allow-empty to force writing (an empty CSV
still carries its header row either way).
How each page is judged, measured on zimmo.be 2026-10-07: a served search
page is HTTP 200; Cloudflare's "Even geduld..." challenge is HTTP 403 and
counts as blocked unless listings appear after it; a route the site
does not serve is HTTP 404 ("Pagina niet gevonden"); 429 and 5xx are
retried up to --retries times and then count as failed. A page number
past the end of a listing is HTTP 200 carrying page 1 again — the run
recognises that by its data (no new listing) and stops there,
complete. A page whose listing grid rendered but parsed to zero rows is
a parser failure (parser_failed_pages), never an empty search. A
failed, blocked or parser-failed page lands in failed_pages; if it is
page 1, no later page is requested. One page buys at most one paid
captcha solve, across all its attempts and proxy rotations.
Exit codes, identical across all engines: 0 ok · 1 crash ·
2 bad usage · 3 blocked · 4 zero products (no --allow-empty) ·
5 the content was never obtained (a remote API error, a dead proxy, a
load timeout, an HTTP error or 404 on page 1, or a rendered page the parser
read nothing from) · 6 partial (some pages failed, were blocked or
parsed to nothing). The Scraper API path uses the same page verdicts.
In the CSV, a text cell starting with =, +, -, @, a tab, CR or LF
is prefixed with an apostrophe so a spreadsheet does not run it as a
formula (csv_cells_escaped in the sidecar counts them); the JSON keeps
every value as served. Files are replaced atomically, so a failed write
leaves the previous output intact.
Compare two runs: python3 diff_runs.py --old run1.json --new run2.json
— refuses to compare a run whose sidecar isn't status: complete, and
two runs of different searches (--allow-selection-mismatch overrides
that). When the newer run read only a window of the listing (--pages
ended before the listing did), a listing missing from it is reported as
left selection, not removed: a new listing may simply have pushed it
to the next page. Only a run that read the listing to its end can report
a removal.
python3 smoke_test.py # or: pytest tests/Zero-network, zero-browser. Validates the parser against real captured
markup (see caveats above), the output contract, banned wording,
.env.example correctness (including a round-trip through the real
loader), that every engine's driver import is at module level, a
coarse AST name-resolution walk, that all three engines' flag sets
match the contract and each other, and that every shared-module
function call across every engine binds against that function's real
signature.
CI runs this offline suite on Python 3.9 and 3.12, an engine-smoke
job that installs each engine into its own clean venv, and a
docker-build job that actually builds the image, runs its entrypoint,
confirms Chromium launches inside it, and confirms no .env/tests/
fixtures ended up in it. The image runs headless by default, and
zimmo.be refuses headless Chromium: use it with --cdp-endpoint (a
Scraping Browser API session), or supply a display yourself and pass
--headful.
A separate canary workflow runs live against a real filtered category
once a week (Mondays): 3 pages, headful under xvfb, with assertions on the share of
rows carrying a price, living area, bedrooms, energy grade and property
type. It needs a residential exit — measured 2026-09-23 and 2026-10-13,
GitHub's runner is refused like any datacentre address, and a US
residential exit was refused while Belgian and German ones were served —
so it takes one from the ZIMMO_PROXY repository secret. Without that
secret (or with an empty one) the canary fails with an error naming
it: a run that never reached the site is not reported as a pass, and
neither is a block (exit 3), zero rows, or a parser failure (exit 5).
Besides the share of populated columns it asserts the arithmetic of the
data: every row has a unique sku, prices fall in a plausible range, no
original_price sits at or below its price, and every URL is on
zimmo.be.
product.schema.json is a JSON Schema (draft 2020-12) for one row,
generated from the Product dataclass by python3 output_writer.py and
checked for drift by smoke_test.py, so a consumer outside Python can
validate a run. x-schema-version goes up whenever a column is added,
renamed or removed, and the CHANGELOG says so.
- Private per-clone notes, if you keep any, belong in
.git/info/exclude, not.gitignore. - Before making this repo public, scan every blob that ever existed
(
git rev-list --objects --all) for credentials, not just the current working tree. - Add this repo as a row in
2scraper/.github'sprofile/README.mdonce it's public, and check that row with a logged-out request.
See CONTRIBUTING.md. Security issues: see SECURITY.md, not a public issue.
MIT