diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 21372a0..6a78b17 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -12,31 +12,28 @@ body: A few things answer most reports faster than we can: - - **Everything exit 3?** Check whether the run was **headless** before - you check anything else. Akamai refuses a headless browser on this - site whatever the exit address — measured HTTP 403 from four - residential exits and one datacentre address, against HTTP 200 and - the full catalogue from those very same addresses with a real - window. `--headful` is the default here; a proxy does not fix it, - and `--cdp-endpoint` is the other path measured to work. - - **`price` null on some rows?** That is almost certainly correct: - every blank price is a lot whose reserve has not been met, and the - row carries `reserve_price_set: true` to say so. What is NOT - correct is a null price with no reserve flag — that one is worth - this issue. - - **Zero rows?** You may be on a page that has none: one page past the - end of a listing, or a search that matched nothing (that one answers - 200 with "No results" **and 24 suggested lots**, which this scraper - deliberately does not parse). Both are exit 4 and correct answers. - See [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md). - - **`rating` and `review_count` empty on every row?** Expected: this - site publishes no per-lot rating. The seller's feedback is in - `seller_score` and `seller_feedback_count` on a `--mode lot` row. - - **The run stopped at page 100?** That is the site's own cap, not a - bug. Past it the site returns page 100's own lots under HTTP 200. - - **Wrong language?** The locale is a path segment (`/en/`, `/nl/`, - `/de/`), not something the exit IP decides. Prices are in euro on - all 18 locales. + - **Everything exit 3?** That is Expedia's "Bot or Not?" handler + (HTTP 429) refusing your exit address, and the scoring is noisy: + the same address serves some requests and challenges others + minutes apart. Slow down (`--delay 10`), move to another exit + (`--proxy` / `--proxy-file`), or wait. The log names the vendor + the handler picked, e.g. `blocked (datadome-challenge)`; only a + reCAPTCHA pick is ever offered to the solver. + - **Exit 6 after page 1?** Look at `stop_reason` in the sidecar. + `next_page_throttled` means the page turn (`POST /graphql`) was + rate limited while the HTML kept answering 200; `cards_missing` + means a page delivered fewer cards than its own counter states. + - **Every price changed overnight?** Check `stay_dates`. A search + with no dates in the URL prices each property on its own cheapest + night; pin `startDate`/`endDate` for comparable prices. + - **`image_url` or `amenities` empty on most rows?** Expected: the + gallery is not mounted below the fold, and the amenity strip is an + A/B variant of the card. + - **Zero rows?** See + [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md) first. + - **Wrong currency or language?** The storefront is the hostname + (`vrbo.com`, `fewo-direkt.de`, `abritel.fr`, `bookabach.co.nz`, + `stayz.com.au`), not something the exit IP decides. - type: dropdown id: engine @@ -84,8 +81,8 @@ body: render: shell placeholder: | python3 playwright_scraper.py \ - --url "https://www.vrbo.com/en/c/333-watches" \ - --pages 3 --out grills + --url "https://www.vrbo.com/search?destination=Orlando,%20Florida,%20United%20States%20of%20America" \ + --pages 2 --out orlando validations: required: true @@ -105,7 +102,7 @@ body: attributes: label: What you expected instead placeholder: >- - 96 products, as the README says a category page yields. + 50 properties per page, as the README says a search page holds. validations: required: true diff --git a/.github/ISSUE_TEMPLATE/site_changed.yml b/.github/ISSUE_TEMPLATE/site_changed.yml index ea565eb..8e7c9ef 100644 --- a/.github/ISSUE_TEMPLATE/site_changed.yml +++ b/.github/ISSUE_TEMPLATE/site_changed.yml @@ -9,36 +9,28 @@ body: its own report because the fix is different from a code bug: something on vrbo.com moved. - Useful to know before filing, because it narrows the fix a lot — this - parser reads **two different sources for two halves of a row**: - - - **The rows** come from the site's own SSR payload, - `',h,1).group(1));print(sorted(d['props']['pageProps']))"` - - For a price problem: one whole `article.c-lot-card__container` from - devtools, and the amount the page shows for that lot. - - For a `bid_kind` problem: the exact label the card prints - ("Current bid", "Aktuelles Gebot", ...) and the locale. + - For a price problem: one whole + `[data-stid="lodging-card-responsive"]` card from devtools, and + the amount the page shows for that property. + - For a missing-rows problem: the counter text + (`1 - 50 of 300+`) and `cards_missing` from the sidecar. render: text validations: required: true @@ -78,5 +68,5 @@ body: attributes: label: Anything else description: >- - Whether it reproduces on a second engine, whether it reproduces in a - second locale, and when it last worked. + Whether it reproduces on a second engine, whether it reproduces on a + second storefront, and when it last worked. diff --git a/.gitignore b/.gitignore index 0d951b8..98ce9bb 100644 --- a/.gitignore +++ b/.gitignore @@ -16,8 +16,8 @@ # every CI job died with FileNotFoundError while the whole suite was green # locally. Exactly the class of thing only a clean checkout finds. !fixtures_generated.json -catawiki_products.* -catawiki_products_*.* +vrbo_products.* +vrbo_products_*.* # --- the live-test harness's working directory --------------------------- live_results/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 00385aa..a62a04e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,37 @@ that is a fix, and the release notes lead with it. logged "Upstream page status success". `status` is now a fallback only when it is an integer. +### Docs — leftovers from the repos this one was copied from + +Text describing another site as if it were Vrbo, removed or rewritten from +this repo's own README, TROUBLESHOOTING and parser: + +- **`captcha_solver.py`** said no challenge had ever been observed here and + that a block was a headless browser refused by Akamai with a 394-byte + "Access Denied". Both were another site's. It now says what the README + measured: Expedia's "Bot or Not?" handler (HTTP 429) picked DataDome on + every recorded refusal, and there is deliberately no DataDome path. The + "No DataDome solver" section the docstring pointed at now exists. +- **`CONTRIBUTING.md`**: the site-change anchors (a different site's + `data-testid` grid and tile markers), the list of pinned properties + (auction lots, reserve prices, bid counts, a 100-page cap) and the + `akamai` marker note (which contradicted this repo's own measurement) + are rewritten for Vrbo. Also corrected: the canary note that the browser + build is the gate, and a "datacentre gets no response at all" claim. +- **Issue templates**: the bug-report hints and the site-change template + described another site's payload, lots and locales; both now describe + Vrbo's cards, storefronts and `stop_reason`s. Example commands use a Vrbo + search URL. +- **`diff_runs.py`**: example filenames, a `sold`/`sold_is_floor` note for a + column this repo does not have, and a docstring listing + `original_price`/`discount_pct`/`in_stock` and JSON-LD provenance. The + docstring now lists the tracked fields and the `stay_changed` bucket. +- **`.gitignore`** ignored another repo's default output prefix; it now + ignores `vrbo_products.*`, the engines' real `--out` default. +- **`SECURITY.md`** named only Akamai as the protection, said the project + has no releases (v0.1.0 exists) and pointed at a README section that does + not exist. + ## [0.1.0] — 2026-09-14 First release on this repo family's architecture. The previous contents of diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d546350..ad3176d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -43,37 +43,30 @@ not the check caught it. Vrbo changing its markup is the normal way this stops working, and it has its own issue template. The detail that saves the most time is WHICH anchor broke, because on this site there is no structured data on a listing -page to fall back on — measured zero `application/ld+json`, zero -`__NEXT_DATA__` and zero Apollo state across six captures — so the DOM is -not the primary path by preference, it is the only one. - -1. **The grid container.** `[data-testid="divSRPContentProducts"]` on a - search page, `[data-ssr="productsCategoryL2/L3SSR"]` on a category - listing. If one of these moves the run reports 0 rows and exit 4, which - is loud. -2. **The tile marker.** `[data-testid="imgLeg-c"]` on a search page (one per - tile), `[data-testid="divProductWrapper"]` inside - `a[data-testid="lnkProductContainer"]` on a category listing. -3. **The reading ORDER inside the tile** — badge, title, price, was-price, - rating, sold, shop, location. The field reads rest on it, deliberately, - because the classes around each field are build hashes: - `

` is the title today. If Vrbo - reorders a tile, `title` and the prices are what break. -4. **`span.flip`**, the shop name and the shop's city in that order, exactly - two per search tile. - -The one place structured data does exist is a DETAIL page's -`window.__cache` Apollo blob, which is where `--mode product` reads the real -product id, the exact sold count and the review count. - -A third thing can break without any path failing: the **join** between the -tiles and the structured data. When it breaks, the row count and the prices -stay healthy while `in_stock` and part of `brand` quietly empty out — so -every run logs its structured-price confirmation share per page and warns -below a floor set PER PAGE KIND (search 8%, category 70%, shop 80%; the -achievable share differs by a factor of eight between them). If you are -reporting a change, that percentage and the page kind are the numbers to -include. +page to fall back on — zero `application/ld+json`, zero `__NEXT_DATA__`, and +an `__APOLLO_STATE__` holding only a banner, because the grid arrives over +client-side POSTs to `/graphql` — so the DOM is not the primary path by +preference, it is the only one. Every anchor is one of Vrbo's own +`data-stid` attributes, never a `uitk-…` design-system class. + +1. **The card.** `[data-stid="lodging-card-responsive"]`, and the property + link inside it, `a[data-stid="open-product-information"]`. If either + moves the run reports 0 rows and exit 4, which is loud. +2. **The price container, which has two spellings.** + `data-stid="product-price-summary"` on vrbo.com and + `data-test-id="price-summary"` on the local storefronts. A third + spelling shows up as a null price column on ONE storefront while every + other column looks fine — so say which host you ran. +3. **The scroll container.** `.scrollable-result-section`. The page body + never scrolls; if this moves, first paint (3 to 18 cards) is all you get + and the sidecar reports `cards_missing`. +4. **Pagination.** `[data-stid="next-button"]` and the counter in + `[data-stid="pagination-navigation"]` (`1 - 50 of 300+`). The counter is + what the completeness arithmetic reads. + +A property page (`--mode property`) carries two JSON-LD blocks, a +`BreadcrumbList` (read for `category`) and an `FAQPage`; its price, address +and reviews still come from the DOM. `--dump-html PATH` writes the exact bytes the parser was given, on success as well as failure, and a run that finds nothing writes a dump and a screenshot @@ -100,10 +93,10 @@ Then the rest of the presentation, in the order that matters: 1. `python3 smoke_test.py` green, and the canary dispatched at least once — including its WARNING branch, which is what runs when a bare GitHub runner's datacentre address is refused and no `VRBO_PROXY` secret is set. - Unlike the sibling repos in this family, this canary needs no secret to do - real work: what Vrbo refuses is the browser BUILD rather than the address, - and a runner can install real Chrome. Whether it also gets past the - ADDRESS check has not been measured, which is exactly why a block there is + Unlike most sibling repos in this family, this canary needs no secret to + try: it installs real Chrome and runs it headful. What decides access + here is how the exit ADDRESS is scored, and whether a runner's datacentre + address is served has not been measured, which is exactly why a block there is a warning rather than a failure — until you set `VRBO_PROXY`, after which it is a failure, because then it means something. 2. The repo description, homepage and topics set (see the family notes on @@ -119,58 +112,33 @@ Then the rest of the presentation, in the order that matters: file of plain functions with inline HTML/JSON fixtures — no pytest, no conftest, no fixtures directory. Copy the nearest existing check and edit it. -Five properties in this repo exist because they were once absent and cost real -time. Tests pin all five, so a PR that breaks one will fail rather than +The properties below exist because they were once absent or are easy to get +wrong. Tests pin them, so a PR that breaks one will fail rather than silently regress: -- **`price` means three different things, and `bid_kind` says which.** - `current` is a live high bid, `final` is the last bid on a closed lot — a - hammer price only when `sold` is also true — and `starting` is a floor - nobody has bid. One captured lot reached €1,300 with its reserve unmet and - sold for nothing at all. The kind is resolved through the page's OWN - translation store (`lot_status_current_bid` and friends), not through a - table of 18 languages: two keys read "Current bid" in English and they are - different strings in Chinese, so a table built from the English page would - have matched nothing there. -- **A null price is a reserve lot, not a failure.** 57 of 57 blank prices - across 13 captures carried `reserve_price_set: true`, spread through the - page rather than clustered at its end. So there is no price-coverage - threshold worth setting, and the check that matters is the INVARIANT: a - null price always carries the reserve flag. The canary asserts exactly - that. -- **The empty-price placeholder is a ZERO-WIDTH SPACE.** - `.c-lot-card__price` is present on 24 of 24 cards while 2–3 hold nothing, - so a truthiness check on the node reports 100% coverage and writes an - invisible character into every row. Anything read out of a card goes - through the zero-width strip first. -- **`favorite_count` comes from the rendered card, never from the payload.** - The payload's own `favoriteCount` reads 0 on 288 of 288 lots across 12 - captures while the card shows the real figure on all 24 of each — present, - authoritative-looking and uniformly wrong. -- **`bid_count` is a floor.** The site returns the last ten bids and states - no total; two lots with very different activity both reported exactly ten. - `bid_count_is_floor` is what says which kind of number it is. -- **A lot page's DOM is not read.** It renders 20–40 OTHER lots in a - "similar lots" carousel using the same class a listing uses for its own - price, so "the first euro amount on the page" is a neighbour's number. - Every lot-mode column comes from the payload. -- **Pagination is capped at 100 pages by the site**, and a request past the - cap returns page 100's own lots under HTTP 200 rather than failing. The cap - is enforced on the URL this repo builds AND on any link the site offers, - because without the second half a run reports COMPLETE holding 2,400 of - 11,681 lots. -- **A search that matches nothing returns 24 suggested lots** reported as - `total: 24`. That state is read off the payload's own - `extended_search_result` flag and is NOT parsed: two dozen plausible rows - for a query that matched nothing is worse than none. -- **A block here is a HEADLESS browser, not a bad address.** HTTP 403 and a - 394-byte "Access Denied" from four residential exits and one datacentre - one, against HTTP 200 and the full catalogue from the same addresses with a - real window. So `--headful` is the default, `RETRY_ON_BLOCKED` is False, - and the block message says so rather than sending someone to buy a proxy. - Block detection is INVERTED as well: a served page is recognised by the - site's own asset host, because Chromium's own network-error page carries - the site's hostname in its title and would pass any title check. +- **A dateless search prices every property on a different night.** + `stay_dates` carries the window verbatim, and `diff_runs.py` buckets a + price move that comes with a moved window as `stay_changed`, not + `changed`. With `startDate`/`endDate` in the URL it is null on every row. +- **Completeness is arithmetic.** The pagination counter states which items + a page holds, so fewer rows than that is `cards_missing` and downgrades + the run to `partial` — never a threshold. +- **A throttled page turn is `partial`, never "the listing ended".** The + HTML keeps answering 200 while the `/graphql` POST behind the next button + answers 429, and `stop_reason` says `next_page_throttled`. +- **There is no per-page address.** `&page=2` and `&startIndex=50` answer + 200 with page 1, so `--concurrency` above 1 is refused with that reason. +- **Two different ids.** `sku` is the property id from the URL path; + `expedia_property_id` is Expedia's own, a DIFFERENT number on a `/{id}` + card. Do not merge them. +- **`rating` is out of ten**, which is why `rating_scale` is its own column. +- **The refusal names its vendor as data.** Expedia's "Bot or Not?" handler + (HTTP 429) states `whichChallenge`; only a reCAPTCHA pick is offered to + the solver, and every other pick is reported as blocked with nothing + charged. Block detection is also positive: a served page is recognised by + the site's own asset hosts (`travel-assets.com`, `media.vrbo.com`), + because Chromium's network-error page carries the site's hostname in its + title. - **A run that finds nothing writes nothing.** It must not replace a good output file with `[]`. `--allow-empty` is the opt-out. - **Exit codes are a contract**, not decoration: `0` ok, `1` crash, `2` bad @@ -183,21 +151,13 @@ silently regress: disagree about it. - **A challenge marker is only consulted for a state already counted as blocked**, and a marker that matches every page of the site is not a - marker at all. This has bitten twice in this family, and the second time - is why `akamai` is NOT in this repo's marker set: the string lives in the - response header (`server: AkamaiGHost`), not in the body of a good page or - a bad one — 0 occurrences in every capture. There is deliberately no - extension-stripping guard either: the Scraping Browser's auto-solve - extension does inject a recaptcha and a turnstile hunter into every page it - loads, but none of this repo's markers matches them even without - stripping, so the guard would be code that looks load-bearing and never - runs. Broaden the set and add the guard together. + marker at all. That is why `akamai` is NOT in this repo's marker set: Vrbo + is fronted by Akamai Bot Manager and every good page loads its sensor + script, so the string matches an 899 KB page holding the full grid. What + identifies a refusal here is the handler itself (`Bot or Not?`, + `captcha-pwa`, `wildcard-challenge-handler`). - **A sku already written by an earlier page of the same run is dropped, not - duplicated.** Unlike its sibling repos this DOES fire on healthy runs - here: page 1 and page 2 of one category listing shared exactly 3 products, - all three from the "cheaper products" carousel that appears on every page. - So a small non-zero drop count is expected and a large one is not. See - `dedupe_by_key` in `output_writer.py`. + duplicated.** See `dedupe_by_key` in `output_writer.py`. There is also a naming check: certain phrases are banned repo-wide and the suite fails naming them. If it trips, read the message — the phrase is wrong for a @@ -224,25 +184,24 @@ Most do not — the suite covers the parser, the writers, the captcha classifier and the CLI contract against inline fixtures. If yours genuinely needs vrbo.com, say in the PR what you ran, which URL and page kind, from which exit, and what you got — including the price and image coverage -percentages the run prints, and the scroll trace from the sidecar. Note that -a run from a datacentre address gets NO RESPONSE AT ALL, so "it returned -nothing" from a VPS is not a finding. Product counts differ by category, by -URL and by how far the scroll got, so a bare "worked for me" is not -reproducible. +percentages the run prints, `cards_missing`, and the `stop_reason` from the +sidecar. Access here depends on how the exit address is scored and is +noisy, so a single exit 3 is not a finding on its own. Row counts differ by +search, by storefront and by how far the scroll got, so a bare "worked for +me" is not reproducible. **Run more than the primary engine.** "Mirror them exactly" is a design rule, -not a verification: the first live run of the pyppeteer engine crashed on its -FIRST fetch on a signature mismatch that four separate offline checks and 400 -green assertions had not caught. +not a verification: in a sibling repo (tokopedia-scraper) the first live run +of the pyppeteer engine crashed on its FIRST fetch on a signature mismatch +that four separate offline checks and 400 green assertions had not caught. -Do not add anything that submits the registration form. This project -deliberately never does, and a captcha token proved valid by creating a real -account is not a result worth having. +Do not add anything that submits a booking, enquiry or sign-in form. This +project deliberately never does. ## Scope -This repo scrapes **public pages** on Vrbo: category listings, search -listings and product pages, exactly as an anonymous visitor is served them. +This repo scrapes **public pages** on Vrbo and its sibling storefronts: +search listings and property pages, exactly as an anonymous visitor is served them. Out of scope: anything behind a login, anything that submits a form, and anything that defeats a protection rather than passing it the way an ordinary browser does. diff --git a/SECURITY.md b/SECURITY.md index 99724c0..39c1ddd 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -55,7 +55,8 @@ Not because these do not matter, but because they belong somewhere else: - **Bypassing Vrbo's bot protection.** This scraper drives an ordinary browser and passes challenges the way a browser does. Anything about how - Akamai or reCAPTCHA behave is not a vulnerability in this repository. + Akamai, Expedia's "Bot or Not?" handler, DataDome or reCAPTCHA behave is + not a vulnerability in this repository. - **The scraper stopped working.** Vrbo changing its markup is expected — file it as a normal issue, there is a template for exactly that. - **Anything about 2Captcha's services** — the solver API, the Scraping Browser @@ -71,7 +72,7 @@ Not because these do not matter, but because they belong somewhere else: Several of our optional dependencies carry advisories for code paths this project never calls; a report needs to show the path. - **Rate limits, terms of service, or the legality of scraping** in your - jurisdiction. See the Legal section of the README — those are your + jurisdiction. See the "Licence and scope" section of the README — those are your responsibility as the operator, not defects. - **Reports generated entirely by an automated tool** with no analysis of whether the finding applies. We read every report, and unexamined scanner @@ -79,8 +80,8 @@ Not because these do not matter, but because they belong somewhere else: ## Supported versions -`main` only. This project has no releases or version tags; fixes land on `main` -and you update by pulling. If you are running an old clone, update before +The latest release and `main`. Fixes land on `main` first; update by +pulling. If you are running an old clone, update before reporting. ## If you have leaked a key diff --git a/captcha_solver.py b/captcha_solver.py index 1ddd103..4bbf5b3 100644 --- a/captcha_solver.py +++ b/captcha_solver.py @@ -9,12 +9,14 @@ page. If Vrbo renders a reCAPTCHA challenge anywhere — account, sign-in and checkout flows are the usual places — this fires. -**NO challenge of any kind has been observed on this site.** Measured -2026-09-10 across six live captures: the pages this scraper reads carry -none, and an address the site has scored gets no response at all rather than -an interstitial. So this module is a contingency, not part of the happy path -— a bot manager can be switched on between deploys, and a scraper that -cannot name what stopped it is much harder to fix. +**The challenge this site actually serves is not one this module solves.** +Vrbo's refusal is Expedia's own "Bot or Not?" handler (HTTP 429), which +picks its vendor per request and names the pick in its page config +(`whichChallenge`). Every measured refusal in the README picked DataDome. +product_parser offers a challenge to this module only when that pick is +reCAPTCHA, so on this site the module is a contingency rather than part of +the happy path — and a scraper that cannot name what stopped it is much +harder to fix. Detection therefore stays BROAD (which challenge a visitor meets depends on the exit and on what the address has been doing) while spending stays @@ -703,23 +705,24 @@ class CaptchaUnsolvable(RuntimeError): # =========================================================================== -# What is deliberately NOT here +# What is deliberately NOT here — No DataDome solver # =========================================================================== # The sibling repo in this family carries a whole second solver for its # site's OWN first-party image captcha ("Enter the characters you see below", # a JPEG of distorted text and a GET form). Roughly 190 lines of it, and none # of it is ported here, because this site has no such page. # -# What this site does instead is refuse a HEADLESS browser. Measured -# 2026-09-10 from five different addresses, four of them residential: Akamai -# answers with HTTP 403 and a 394-byte "Access Denied" page carrying a -# reference id — no form, no image, no widget, nothing for a solver to -# answer. And the trigger is the CLIENT rather than the address: the very -# same addresses were served HTTP 200 and the full catalogue by a browser -# with a real window. So the response to a block here is `--headful` or -# `--cdp-endpoint`, not a solve and not a better proxy, and -# product_parser.detect_page_state reports it as "blocked" rather than -# "challenge" precisely so no solve is attempted and nothing is charged. +# Nor is there a DataDome path, although DataDome is what Expedia's +# "Bot or Not?" handler picked on every measured refusal here. The README +# records why: `DataDomeSliderTask` solved the slider in testing, but +# applying the returned cookie did not reliably get a request in, and a +# control run with no solve at all was served just as often. A DataDome +# solve also needs a sticky proxy session this tool does not manage, and +# the recommended path (a real local Chrome on a residential exit) never +# met a challenge. So product_parser reports any non-reCAPTCHA pick as +# "blocked" rather than "challenge", and no solve is attempted and nothing +# is charged. This is "this repo does not implement it", not "it cannot be +# solved". # # The reCAPTCHA / hCaptcha / Turnstile machinery above IS kept, and that is a # deliberate asymmetry rather than an inconsistency. Detection stays broad @@ -727,6 +730,6 @@ class CaptchaUnsolvable(RuntimeError): # what the address has been doing — a narrow list is how a challenge gets # reported as an empty page months later. A solver for a challenge this site # has never been observed to serve is dead code; a DETECTOR for one is cheap -# insurance: a detection that fires on a page whose lots have -# already rendered guards nothing, which is why the default is -# `when-blocked` and why it counts lot links before it spends. +# insurance: a detection that fires on a page whose properties +# have already rendered guards nothing, which is why the default is +# `when-blocked` and why it counts property links before it spends. diff --git a/diff_runs.py b/diff_runs.py index bc40cb5..a1880d9 100644 --- a/diff_runs.py +++ b/diff_runs.py @@ -8,26 +8,31 @@ monitoring and assortment tracking, but that nothing in this repo actually computed. - python3 diff_runs.py --old watches.2026-09-01.json \\ - --new watches.2026-09-07.json + python3 diff_runs.py --old orlando.2026-09-01.json \\ + --new orlando.2026-09-07.json -Typical use is a scheduled re-run of one of the four scraper engines, kept +Typical use is a scheduled re-run of one of the scraper engines, kept under a dated filename, diffed against the previous one: - python3 playwright_scraper.py --url "$URL" --out "girls_$(date +%F)" - python3 diff_runs.py --old "girls_$(ls -t girls_*.json | sed -n 2p)" \\ - --new "girls_$(date +%F).json" --out diff.json + python3 playwright_scraper.py --url "$URL" --out "orlando_$(date +%F)" + python3 diff_runs.py --old "$(ls -t orlando_*.json | sed -n 2p)" \\ + --new "orlando_$(date +%F).json" --out diff.json -Four buckets, each keyed on sku: +The main buckets, each keyed on sku: added — sku present in --new, absent from --old removed — sku present in --old, absent from --new (delisted, or just - off this particular page/category run) - changed — sku present in both, with a different price, - original_price, discount_pct, currency or in_stock + off this particular search run) + changed — sku present in both, with a different value in one of + TRACKED_FIELDS (price, currency, rating, review_count, + property_type, bedrooms, beds, title) + stay_changed — a price move that came with a moved `stay_dates`: the + two runs quoted different nights, so the prices are not + comparable (see the README's first surprise) source_changed — sku present in both with a different price, but also a - different price_source: one run got the DOM-corrected - figure and the other the raw JSON-LD one, so the two are + different price_source: one run read the rendered amount + (`card`) and the other the screen-reader sentence + (`card-a11y`), or one was a property page, so the two are not comparable on price. Reported separately because this says something about our own two snapshots, not about the site — and --fail-on-change deliberately ignores it. @@ -46,13 +51,6 @@ from output_writer import UNIQUE_BY_SKU_MODES -# `sold` is tracked alongside the price, and `sold_is_floor` with it, because -# without the flag a `sold` change is unreadable: a tile's figure is a floor -# the site rounded down ("100rb+ terjual" = 100_000) while a product page's -# is exact (207785 for that same product). A monitor watching `sold` alone -# would report a jump of 107,785 the moment someone diffed a listing run -# against a product run, and none of it would be a sale. -# # No `original_price` / `discount_pct` here, because this site has neither: # 0 strike nodes across 218 cards, 6 captures and 4 storefronts, so the # columns do not exist on `Product` either (see output_writer's docstring). @@ -212,8 +210,8 @@ def diff_products(old: List[dict], new: List[dict], # `lifecycle` key is still emitted, always empty, so a consumer # written against the family's diff shape does not have to branch. # An FX tick rather than a price change — see _within_tolerance. Only - # when the ONLY differences are price fields: a currency or stock - # change alongside is a real change whatever the size of the move. + # when the ONLY differences are price fields: a currency or + # rating change alongside is a real change whatever the size of the move. if (all(f in PRICE_FIELDS for f in field_changes) and _within_tolerance(before, after, field_changes, price_tolerance_pct)):