foodpanda-scraper

Scrapes foodpanda vendor listings — every restaurant and shop tile the site puts on a city, area or home page, with its rating, review count, cuisines, deal labels, delivery data and whether it is open right now. JSON and CSV, one row per vendor, the same schema across all ten country sites.

Four ways to fetch: Playwright (recommended), Selenium, pyppeteer, or a remote browser over CDP — the 2Captcha Scraping Browser API, or any browser that exposes a CDP endpoint.

pip install -r requirements.txt -r requirements-playwright.txt
playwright install chromium

python3 playwright_scraper.py \
  --url "https://www.foodpanda.pk/city/lahore/area/gulberg" \
  --pages 3 --delay 45 --retry-delay 30 \
  --out gulberg --format both

[+] Saved 143 products -> gulberg.json
[+] Saved 143 products -> gulberg.csv
[+] Wrote run metadata -> gulberg.meta.json (status=complete)

That is a real run, 2026-09-15, from an ordinary residential address with no proxy, no key and no account. See sample_output.json.

---

Do I need to buy anything?

No, and that is measured. A plain headful Chromium on a residential address was served by ten of the ten country sites on 2026-09-15, and a three-page run of a real area listing returned 143 vendors with status: complete.

What you do need is patience, because this site's refusal is a matter of degree:

A sweep of fourteen foodpanda addresses from one residential exit, each navigated up to three times with a fresh browser and roughly half a minute between attempts, was served on 10 of 30 navigations. The successes were spread across attempt numbers — four on the first, one on the second, five on the third.

The refusal is a property of the session and the moment, not of your address alone: the same URL that answers 403 answers 200 two attempts later. So the instruments that work here, in order, are --retries (each attempt launches a fresh browser), --retry-delay and --delay. Only after those does buying an exit help.

--fingerprint makes this site WORSE — do not use it here

Measured on one URL within five minutes, so the hour and the address fall on every arm equally:

Run	Served
no fingerprint	48 rows, first attempt
no fingerprint, five minutes later	48 rows, first attempt
--fingerprint --fp-country pk	refused 4 of 4
--fingerprint --fp-country de — matching the exit	refused 4 of 4

The last row is what makes this conclusive. A fingerprint whose country contradicts the exit is a known mismatch, so the obvious reading of the third row is "wrong country" — and the fourth row removes it: a fingerprint that matched the exit exactly was refused just the same.

What is left is the mismatch the flag cannot avoid. The fingerprint describes a Windows machine; it is injected into whatever Chromium you are running. The overrides say one thing while everything they do not cover — the real GPU strings, the font stack, the TLS handshake, the JS engine's own behaviour — says another, and PerimeterX reads that as worse than an honest browser.

The flag stays because it is part of this family's contract and the 2Captcha fingerprint product genuinely works on sites where the local browser is the thing being scored. On foodpanda it is a way to get refused. Use --cdp-endpoint instead, which supplies a coherent identity rather than a painted-on one — and which the scraper already refuses to stack a fingerprint on top of.

Headless is refused — so --headful is the default

Unusually for this family, this scraper runs with a window unless you ask otherwise. One URL, arms alternating, a fresh browser per navigation, 45 seconds apart so the hour and the address fall on both equally:

	Served	Tiles
--headless	0 of 4	0, HTTP 403 every time
--headful	4 of 4	47, 48, 45, 48

--headless is still there and is what the Docker image passes, because a container has no display. That is the reason an image can be refused where your laptop is not.

What the paid products actually buy on this site — measured, not guessed. The Scraping Browser API turns this site from "patience required" into "just works":

	Refusals	Retries	Wall clock
Residential address, no proxy	page 3 refused 4 times	5 attempts	~7 min
--cdp-endpoint (Scraping Browser)	none	none	29 s

Both runs took the same three pages of the same listing and returned 143 and 144 vendors with status: complete. If you are doing more than a page or two, that is the difference the money buys.

The endpoint expires. How long a Scraping Browser profile lives is not a fixed figure — one endpoint set as a repository secret was still accepted ten days later — but once it is gone it answers HTTP 401 deny_no_user and you mint new credentials. If you put one in a scheduled job, expect to refresh it when that 401 appears.

What a 2Captcha key buys here: a way past both of this site's refusals, measured. PerimeterX's denial renders reCAPTCHA Enterprise (~55s, $0.00299) and Cloudflare's "Just a moment…" is a Turnstile (11s, $0.00145); --solve-captcha clears either, and the page comes back with its full grid — see solving the block and Turnstile. The one thing no key can do is PerimeterX's widget-less stub: there is nothing on that page to solve.

---

What it reads

URL	What you get	Paginates
https://www.foodpanda.pk/	The country home page — a curated set of vendors, with delivery time, delivery fee and price level	No pagination of any kind; scrolled instead
/city/{city}	A city's listing, 48 vendor tiles	?page=N
/city/{city}/area/{area}	An area's listing — the big one	?page=N, up to 24 numbered pages on the largest measured

Refused, each with the reason:

URL	Why
/restaurant/{code}/{slug}	A vendor page cannot be reached
/city, /city/{city}/area	Directories of links, not listings of vendors — warned about, and the run honestly reports 0 rows
foodpanda.com	The brand's global landing page: a country picker with no vendor tiles and no /city tree

The ten country sites

Taken from the brand's own country footprint — foodpanda publishes no cross-country hreflang set, checked on three captures — and then probed one at a time with a real browser rather than trusted.

Host	Country	Probed 2026-09-15
foodpanda.pk	Pakistan	served, 62 vendor links on the home page
foodpanda.sg	Singapore	served, 67
foodpanda.my	Malaysia	served, 83
foodpanda.ph	Philippines	served, 56
foodpanda.com.tw	Taiwan	served, 75
foodpanda.hk	Hong Kong	served, 60
foodpanda.com.bd	Bangladesh	served, 57
foodpanda.com.kh	Cambodia	served, 47
foodpanda.la	Laos	served, 50
foodpanda.com.mm	Myanmar	served, 50

foodpanda.co.th was dropped, and it is the most surprising thing this repo found. It answers HTTP 200 and redirects to robinhood.co.th — a different company — with zero vendor tiles, zero deliveryhero references and zero /restaurant/ links on it. foodpanda has left Thailand. The scraper refuses that host with that reason rather than reporting a block, because a block would send you looking for a proxy problem that does not exist.

Each row is one to three navigations with a real browser; the raw probe output is not committed (it is a working artefact, and stale within days). Bare hosts redirect to their www. form and both are accepted.

The country is the host, so there is no --country flag to disagree with the URL.

---

The output

One row per vendor tile. The family's column prefix comes first, then this site's own. sample_output.json is cut from the run at the top of this page.

Column	Notes
source	The country site: foodpanda.pk, foodpanda.sg, …
url, sku, slug	The vendor's page, its four-character code, and its URL slug. The code is read from the tile's own data-testid and checked against its href — the two agreed on 1,904 of 1,904 measured tiles
title	From the title attribute, not the text node, which CSS truncates with an ellipsis
rating	Out of five. 69–78% of tiles carry one; the rest are vendors with no reviews yet
review_count + review_count_is_floor	Read the flag. The site caps the printed figure at (100+), (500+), (1000+), so 100 with the flag set means "at least 100"
discount_pct + discount_is_upper_bound + discount_label	Read the flag. 20% off is 20; Up to 35% off is a CEILING, and 21% of measured labels are that shape
min_order + min_order_currency	The minimum spend a voucher attaches to its percentage: 20% off Rs. 300, 15% off S$ 30
currency	ISO 4217, read from the symbol the tile prints. Null where the tile carries no money at all, which is every row of a city or area listing
category, cuisines	The site's own categorisation — "Pakistani", "Fast Food", "Cakes & Bakery". 98% of tiles
is_open + opens_at	The most volatile column in the schema: a point-in-time reading, 35–38% closed in the measured captures. opens_at is verbatim ("Sat 10:20")
tags	Every deal label, verbatim. The overflow chip (+1) is excluded — it is a count of hidden labels, not a label
free_delivery, is_super_vendor	Site badges. free_delivery is matched case-insensitively: Pakistan writes "Free Delivery" and Singapore "Free delivery"
delivery_time, delivery_fee, price_level	Home page only — see below
page_kind	home, city or area. The reason the three columns above can be null
page, position	The pair is unique across a run, and the canary asserts it
image_url	Recognised POSITIVELY by foodpanda's own media hosts, so a placeholder cannot fill the column
city, area	From the LISTING's URL. A vendor tile publishes no address of its own

Five family columns are absent, and each absence is a measurement rather than an oversight: price, original_price, price_source, brand, in_stock. A foodpanda vendor tile carries no price — the thing on sale is a restaurant, not an item — and 0 price nodes were found across 1,952 tiles on two country sites. The reasoning is written out at the top of output_writer.py.

The home page publishes more than the listings do

This surprised us, and it is worth knowing before you pick a URL. A home-page tile carries five labelled info rows where a city tile carries one:

city listing   Desserts
home page      From 25 min · $ · Biryani · Rs.69 · Free for first order
               ^delivery time  ^price level  ^cuisines  ^delivery fee

So delivery_time, delivery_fee (with its currency) and price_level are populated on --url https://www.foodpanda.pk/ and null on a /city or /area run. The home page is the one listing foodpanda renders with a delivery address already in play.

The rows are read by what they are, not by where they sit: the site labels each one for screen readers, and the classification falls back to the row's shape so it does not depend on the label being in English.

---

What a vendor page does

Nothing, for this repo. /restaurant/{code}/{slug} is where a menu, a delivery fee and a minimum order live, and it is refused:

Eight cold navigations and one click-through from a listing the same browser had just been served all came back HTTP 403 with PerimeterX's denial page, on foodpanda.pk and foodpanda.sg alike, while listing pages answered 200 in those same sessions.

No page was ever captured, so no parser was written and no mode ships. A parser written against markup nobody has seen is a guess with a docstring. The scraper refuses such a URL with that reason rather than half-working.

---

The two refusals, and why they want opposite responses

PerimeterX — Access to this page has been denied

The common one. Per-session, and it clears — see Do I need to buy anything?. Two shapes, both captured: a 10 KB page with a reCAPTCHA widget on it, and a 4.7 KB page with no widget at all.

Cloudflare — Just a moment...

Cloudflare's managed challenge, served before the request ever reaches the application — so it is a different thing from PerimeterX's denial, and it wants a different answer.

Every non-browser client gets it: measured with curl against all eleven hosts this repo probed (the ten storefronts and foodpanda.com), every one returned HTTP 403 and this document, while a real browser on the same address was served normally seconds later.

A real browser can get it too, and foodpanda.com in particular does. That is not "something stripped the context" and a proxy will not fix it either — it is a Turnstile, and this repo solves it for $0.00145.

Solving the block works, and costs about a third of a cent

PerimeterX's denial page renders a reCAPTCHA Enterprise widget, and that is solvable. Measured 2026-09-16 against a live refusal on foodpanda.sg, through this repo's own solve path:

variant	recaptcha_v2 — enterprise loader, bframe, no size, no render
task type	RecaptchaV2EnterpriseTaskProxyless
solved in	~55 seconds
after the token was injected	HTTP 200, 1,172,561 bytes, 48 vendor tiles
cost	$0.00299

And the control that makes that a claim rather than a coincidence: reloading a denial page in the same session without solving cleared the block 0 of 8 times. The token is what got in.

python3 playwright_scraper.py --solve-captcha always \
  --url "https://www.foodpanda.sg/city/singapore"   # key comes from .env

The task type is the whole game. The widget's container looks like an ordinary v2 checkbox:

<div id="px-captcha">
  <div class="g-recaptcha" data-sitekey="6Lc…" data-callback="handleCaptcha">

but the loader beside it is the enterprise API — https://www.google.com/recaptcha/enterprise.js — measured on four denial documents across two country sites, with zero occurrences of recaptcha/api.js on any of them. An enterprise widget solved through the ordinary task type returns a token the site rejects. This repo's first live attempt did exactly that, from a different mistake: it read the bframe with no size as v3, bought a RecaptchaV3TaskProxyless, and got ERROR_CAPTCHA_UNSOLVABLE after 87 seconds and $0.003. v3 never renders a challenge frame, so a frame present now settles the variant as a v2 checkbox.

Cloudflare Turnstile is implemented too, Challenge page included

The other refusal this site can produce is Cloudflare's managed challenge — "Just a moment…", which foodpanda.com serves even to a browser. That is solved as well. Measured 2026-09-16:

intercepted	sitekey, action: managed, cData, chlPageData
task type	TurnstileTaskProxyless
solved in	11 seconds, an 837-character token
after injection	`Home
cost	$0.00145

The interception is the whole trick. A Challenge page publishes no sitekey in its markup: Cloudflare calls turnstile.render(container, params) once and keeps nothing, and cData, chlPageData and action live only in that call. So every engine installs captcha_solver.TURNSTILE_INTERCEPT_JS on the context before any page script runs — add_init_script in Playwright, evaluateOnNewDocument in pyppeteer, Page.addScriptToEvaluateOnNewDocument over CDP in Selenium. A static read alone cannot produce a solvable task, and this repo refuses to build one from it rather than spend money on a request 2Captcha will reject.

One thing that did not matter, against expectation: the user agent. 2Captcha minted the token against a Windows UA while the browser was macOS, and Cloudflare accepted it anyway. The mismatch is logged so it can be ruled out if a token is ever refused.

What is NOT solvable: PerimeterX's stub. A second denial variant, 4.7 KB, carries 24 px-captcha references and no widget at all — nothing there for any solver to work on, at any price. Reported as blocked, and nothing is attempted or billed.

A marker that had to be thrown away. cf-turnstile looks like the obvious way to spot a Turnstile and is measured useless here: it appears once on a served Hong Kong listing — 2Captcha's own Scraping Browser injects a chrome-extension://…/turnstile/hunter.js with data-ts-input="cf-turnstile-response" into every page it loads — and zero times on the real Cloudflare challenge. challenges.cloudflare.com is the marker that works: 0 on every served page, 5 on the challenge.

Is a solve the right move? Usually not, on cost alone: the refusal is per-session and a fresh browser clears it for nothing — which is why --solve-captcha when-blocked is the default and the retry budget runs first. Reach for always when retries are not clearing it and you want the page now.

Pagination

Unusually for this family, the site publishes the page addresses itself. An area listing renders its own numbered links between batches of tiles:

<span class="page-number" id="2" data-testid="pageNumber">
  <a href="https://www.foodpanda.pk/city/lahore/area/gulberg?page=2">2</a>
</span>

and a cold fetch of that address returned 48 tiles sharing no vendor code with page 1. So pages are fetched independently, --concurrency above 1 is genuinely available on the /city tree, and the run stops on a data condition — "this page added no sku we had not already seen" — rather than on a selector.

Three things the code will tell you rather than guess:

• The home page has no pagination at all. --pages 5 there scrolls the one page it has, and says so. --concurrency is refused with that reason.
• A refused page is not an exhausted listing. This was a real bug, found by this repo's first live run: page 2 was refused on every attempt, the parse produced 0 rows, and the run read that as the end of the catalogue — reporting status: complete and exit 0 while holding a third of the data. A refused page is now a FAILED page, and the run exits 6 with the page number in pages_failed.
• Page size is not constant. Three consecutive cold fetches of one listing returned 48, 48 and 44 tiles, so nothing in this repo concludes completeness from a count.

---

Engines

	Playwright	Selenium	pyppeteer	Scraping Browser (CDP)
Recommended	yes	works	needs --chromium-path	for volume
--concurrency > 1	yes	no	no	refused (one connection per profile)
Authenticated proxy	yes	no	yes	n/a
Authenticated CDP endpoint	yes	no	yes	yes
--fingerprint	yes	yes	no	not applicable

Every measurement in this README was taken with the Playwright engine and Playwright's own bundled Chromium — no special channel needed there.

The pyppeteer engine is refused out of the box on this site, and the reason is its browser rather than its driver. Measured on one URL in one window, arms alternating:

	Served
pyppeteer + Playwright's Chromium 148	2 of 2, full grid
pyppeteer + its own bundled Chromium 117	0 of 2, HTTP 403
Playwright	2 of 2, full grid

pyppeteer pins a Chromium that reports 117.0.5938.0 — three years stale in September 2026. Point the engine at a browser you already have and it works:

python3 puppeteer_scraper.py --chromium-path /usr/bin/google-chrome ...

The Selenium engine does not have this problem, because chromedriver drives the Chrome already installed on the machine. It was served first try in a live run.

Two Selenium limits are real and are reported loudly rather than half-working: --proxy-server takes an address with nowhere to put a password, and chromedriver's debuggerAddress is a bare host:port. Use Playwright or pyppeteer for either. pyppeteer is also effectively unmaintained and its own README points at Playwright.

Install exactly one engine: playwright and pyppeteer pin incompatible pyee versions, and pyppeteer and selenium collide on urllib3.

---

Flags

Identical across the three engines, except where the engine's own limits make a flag impossible — --chromium-path is pyppeteer's, and Selenium cannot authenticate a proxy or a CDP endpoint at all (see Engines). Every one of these is also settable from .env; see Configuration.

Flag	Default	What it does
--url URL	—	The listing to read. Required, unless FOODPANDA_URL is set. The country is the host, so there is no --country to disagree with it
--mode {listing}	listing	The only mode. There is deliberately no vendor mode
--category LABEL	—	A label for this run, recorded as run_label in the sidecar. It does not write the category column, which holds the vendor's own cuisines
--pages N	1	Listing pages to crawl. The home page has no pagination, so --pages there scrolls further instead
--delay S	1.0	Delay between pages
--concurrency N	1	Parallel workers, each with its own browser and its own proxy exit. Ignored with --cdp-endpoint
--retries N	3	Attempts per page, each with a fresh browser — the instrument that actually clears a refusal here. Backs off by doubling. An empty page is not retried
--retry-delay S	2.0	Seconds before the first retry, doubling after
--format {json,csv,both}	json	Output format
--out PREFIX	foodpanda_products	Output prefix; the sidecar is <prefix>.meta.json
--locale L	en-US	What the browser claims about itself. It does not pick the page's language — the host does
--allow-empty	off	Write files even with 0 rows. Off so a failed run cannot overwrite good data
--dump-html PATH	—	Save the exact HTML the parser was given, on success as well as failure
--headful / --headless	--headful	Headless is refused on this site; --headless is for containers

Proxies — what they buy here:

Flag	Default	What it does
--proxy URL	—	One exit, http://user:pass@host:port
--proxy-file PATH	—	One URL per line to rotate across. Wins over --proxy
--proxy-rotate {per-run,per-page}	per-run	per-page relaunches the browser with each exit, so the session does not follow the address
--proxy-shuffle	off	Shuffle the pool at startup, so concurrent runs do not all start on the first exit
--proxy-block-retries N	2	On a refusal, retry from this many other exits. Needs a pool

2Captcha — one key, four separately-billed products:

Flag	Default	What it does
--twocaptcha-key K	—	Prefer TWOCAPTCHA_KEY in .env; a key on a command line is readable by ps
--solve-captcha {when-blocked,always}	when-blocked	Both of this site's challenges go through here — reCAPTCHA Enterprise (~$0.00299) and Turnstile ($0.00145). when-blocked is the default because a fresh browser often clears the refusal for free
--captcha-api {v2,v1}	v2	The current JSON API, or the legacy in.php/res.php pair. v1 cannot do Turnstile and says so
--min-score S	0.7	reCAPTCHA v3 score to request — 0.3, 0.7 or 0.9 are the only values the API takes. Ignored for v2 widgets
--fingerprint	off	Makes this site worse — do not use it here
--fp-tags TAG	Windows	One OS-family tag. Chrome, Desktop and Mobile are each rejected by the API with HTTP 400
--fp-country CC	—	Match it to your exit's country; a US fingerprint on a German address is a contradiction
--cdp-endpoint WS	—	Drive a remote browser (the Scraping Browser API) instead of a local one. --proxy and --headless/--headful are ignored with it, and its credentials expire

---

Exit codes

0	ok
1	crash
2	bad usage
3	blocked
4	ran fine, zero vendors
5	remote API error
6	partial — some pages were never read

A run that finds nothing writes nothing, so last night's good output is never replaced by []. --allow-empty is the opt-out. Every run writes <out>.meta.json with the status, the stop reason and which pages failed, by number — a failed run writes none, because a "failed" sidecar beside good data would contradict it.

---

Comparing two runs

python3 diff_runs.py --old gulberg.2026-09-14.json --new gulberg.2026-09-15.json

Keyed on sku, with one bucket this site needs and its siblings do not: hours_changed. A vendor opening or closing follows the clock rather than the catalogue — 735 of 1,904 tiles were closed in one measured capture — so rows whose ONLY change is the opening pair are reported separately and --fail-on-change ignores them.

It refuses to diff two runs that are not both complete, and two runs from different country sites: foodpanda issues vendor codes per country, so two rows can share a sku and be unrelated vendors.

---

Configuration

Credentials go in .env next to the scripts, never on a command line.

cp .env.example .env
python3 env_config.py     # prints what was picked up, WITHOUT secrets

Precedence, highest first: an explicit flag → an exported environment variable → .env → the default. Anything still carrying {...} braces is treated as unset, so a copied example is never sent to an API as if it were a key.

TWOCAPTCHA_KEY, FOODPANDA_PROXY, FOODPANDA_CDP_ENDPOINT and FOODPANDA_URL are the four variables the code reads, and a test asserts .env.example documents exactly those, in both directions.

All four 2Captcha paths were run end to end against this site. The solver included: a live PerimeterX denial was solved and the page came back with its grid, for about a third of a cent. See solving the block.

---

Tests

python3 smoke_test.py      # the offline suite, no engine library required
pytest                     # the same checks, wrapped as one pytest test
python3 make_fixtures.py   # regenerate the fixtures from your own captures

Over 400 offline checks, with fixtures cut from real captures by make_fixtures.py, which refuses to write unless every trimmed fixture parses identically to its untrimmed original, column for column.

The checks worth knowing about are listed at the top of smoke_test.py. Most of them exist because of a specific failure.

CI runs the offline suite on the oldest and newest supported Python, installs each engine in its own virtualenv (their pins are mutually unsatisfiable), builds and runs the Docker image, and greps for committed credentials through one implementation that both the workflow and the suite invoke.

There is also a canary workflow that takes three real pages off the live site and asserts the things only a real run can show — that pagination happened, that (page, position) is unique, that an area row carries no delivery data. It is dispatch-only, with no schedule: nothing here runs live on its own, and the canary badge shows the last dispatch. Point it at a FOODPANDA_CDP_ENDPOINT or FOODPANDA_PROXY secret and run it by hand when you want that check. Without one it fails with an error naming the missing secret, because a run that never reached the site has tested nothing.

---

Traps that look like bugs

Read TROUBLESHOOTING.md before filing one. The short list:

• review_count of 100 on a plainly busier vendor — read review_count_is_floor.
• discount_pct of 35 on a vendor that is not 35% off — read discount_is_upper_bound.
• Two runs disagreeing about a third of the rows — that is is_open, and the clock.
• delivery_fee null on every row — that is a /city run; those columns are the home page's, and page_kind says which you have.
• --category not appearing in the category column — by design. That column holds the vendor's cuisines; --category labels the run, in the sidecar.
• A Docker run being refused where your laptop is not — there is no display in a container, so it runs headless.

---

Licence

MIT. See LICENSE.

Captcha solving, the Scraping Browser API, proxies and fingerprints are four separately billed 2Captcha products behind one key. All four were run end to end against this site.