Rebuild on the 2scraper template: three engines, three modes, 230 offline checks

.dockerignore

@@ -0,0 +1,18 @@
+.git/
+.github/
+.env
+*.log
+*.key
+secrets*
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.fingerprint_cache/
+.venv/
+venv/
+env/
+amazon_products.*
+*_debug.html
+*_debug.png
+live_results/
+live_results.zip

.env.example

@@ -0,0 +1,86 @@
+# amazon-scraper — configuration
+#
+# Copy this file to `.env` next to the scripts and fill in what you need.
+# Nothing here is required: with no .env and no environment variables at all
+# the scrapers behave exactly as their flags say.
+#
+# Measured, so you know what the paid parts actually buy: an ordinary local
+# Chromium on a clean residential IP returned 3 pages / 77 products from
+# amazon.com with NO key, NO proxy and NO account — see the README. The
+# products below buy volume from many addresses, a specific exit country, and
+# a browser you do not have to run yourself.
+#
+# CREDENTIALS BELONG HERE AND NOWHERE ELSE. A secret on a command line is
+# readable by anything on the machine that can run `ps`, and it lands in your
+# shell history. Every flag these variables back can also be typed, and an
+# explicit flag always wins over this file.
+#
+# Precedence, highest first:
+#   explicit flag  ->  exported environment variable  ->  .env  ->  default
+#
+# Run `python3 env_config.py` to see what was picked up. It prints which
+# variables were found WITHOUT printing their values.
+
+# ---------------------------------------------------------------------------
+# 2Captcha — one key, four separately-billed products
+# ---------------------------------------------------------------------------
+# Captcha solving, the Scraping Browser API, proxies and fingerprints all
+# authenticate with this one key and are billed separately.
+#
+# On Amazon the solving part is for ONE specific page: the first-party image
+# captcha ("Enter the characters you see below"). It is NOT needed for the
+# AWS WAF interstitial, which the browser clears by itself in about four
+# seconds, and no solve helps a 503 throttle or a sign-in wall.
+TWOCAPTCHA_KEY=your_2captcha_api_key_here
+
+# ---------------------------------------------------------------------------
+# Scraping Browser API (a remote browser over CDP)
+# ---------------------------------------------------------------------------
+# Endpoint shape:
+#   ws://{login}-zone-scraping_browser-country-{cc}-pid-{profileId}:{password}@cb.2captcha.com:9222
+#
+# `country-` picks the exit country. On Amazon that decides the page language
+# AND the currency you are quoted — prices are converted for a visitor whose
+# delivery country differs from the marketplace, so a European exit reading
+# amazon.co.jp is shown "EUR 25.34" rather than yen. If you want a
+# marketplace's own currency, use an exit in that country.
+#
+# `pid-` is a profile with persistent cookies and allows ONE live connection
+# each, capped per account — reuse profile ids rather than minting one per
+# run. Note the cookie persistence has a cost here: Amazon assigns some page
+# variants per session, and a profile that was given the detail page without
+# its reviews keeps getting it. A different `pid` is the fix.
+#
+# --concurrency is refused with this set, for the same one-connection reason.
+AMAZON_CDP_ENDPOINT=
+
+# ---------------------------------------------------------------------------
+# Proxy
+# ---------------------------------------------------------------------------
+# One exit, as a full URL: http://ACCOUNT:PASSWORD@HOST:PORT
+# For a pool, use --proxy-file with one URL per line.
+#
+# Ignored when AMAZON_CDP_ENDPOINT is set: the remote browser has its own
+# exit, and layering a second on top contradicts it rather than helping.
+#
+# Note selenium_scraper.py cannot authenticate a proxy at all — chromedriver's
+# --proxy-server takes no credentials. It strips them and warns. Use the
+# Playwright or pyppeteer engine for an authenticated exit.
+AMAZON_PROXY=
+
+# ---------------------------------------------------------------------------
+# Target
+# ---------------------------------------------------------------------------
+# The URL to scrape, so a scheduled run does not need it on the command line.
+# The hostname decides which of the 21 marketplaces is read — there is no
+# separate marketplace flag, so a flag and a URL cannot disagree.
+#
+#   listing mode:  https://www.amazon.com/s?k=wireless+headphones
+#                  https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics/
+#   product mode:  https://www.amazon.com/dp/B07K5214NZ
+#   reviews mode:  https://www.amazon.com/dp/B07K5214NZ
+AMAZON_URL=
+
+# Deliberately NOT configurable here: the output prefix. `--out` already has a
+# non-empty default, so the loader (which only fills UNSET values) would never
+# apply a variable for it — a setting that looks configurable and is not.

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,116 @@
+name: Bug report
+description: The scraper ran but the result was wrong, empty, or it crashed.
+labels: ["bug"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        **Before anything else: never paste an API key, a proxy password or a
+        full `ws://user:pass@...` endpoint into a public issue.** The scrapers
+        mask credentials in their own logs, but raw HTML dumps and the Scraper
+        API's `x-debug` header do not. Replace them with `***`.
+
+        Two things answer most reports faster than we can:
+
+        - **Zero products?** Check you used a *filtered* category URL, not the
+          bare `/shopping/kids/items.aspx` hub — the hub carries no product
+          JSON-LD and correctly returns nothing. See
+          [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md).
+        - **Wrong currency or localised titles?** Amazon decides both from
+          your exit IP, not from the URL. See "Geo-redirect" in the README.
+
+  - type: dropdown
+    id: engine
+    attributes:
+      label: Which engine
+      options:
+        - playwright_scraper.py
+        - puppeteer_scraper.py
+        - selenium_scraper.py
+        - scraper_api_client.py
+        - fingerprint_client.py
+        - other / not sure
+    validations:
+      required: true
+
+  - type: dropdown
+    id: browser_route
+    attributes:
+      label: How was the browser reached
+      options:
+        - A browser the script launched itself
+        - A remote browser over --cdp-endpoint
+        - No browser (scraper_api_client.py over plain HTTP)
+        - Not applicable
+    validations:
+      required: true
+
+  - type: input
+    id: exit_code
+    attributes:
+      label: Exit code
+      description: >-
+        `echo $?` straight after the run. These mean specific things — 3 is a
+        challenge page, 4 is "ran fine, found nothing", 124 is a self-imposed
+        timeout. The table is in the README.
+      placeholder: "4"
+    validations:
+      required: true
+
+  - type: textarea
+    id: command
+    attributes:
+      label: The exact command
+      description: With credentials replaced by `***`.
+      render: shell
+      placeholder: |
+        python3 playwright_scraper.py \
+          --url "https://www.amazon.com/s?k=wireless+headphones" \
+          --pages 1 --out girls
+    validations:
+      required: true
+
+  - type: textarea
+    id: output
+    attributes:
+      label: Output
+      description: >-
+        The whole run, not just the last line — the useful detail is usually
+        upstream of the error. Credentials replaced by `***`.
+      render: text
+    validations:
+      required: true
+
+  - type: textarea
+    id: expected
+    attributes:
+      label: What you expected instead
+      placeholder: >-
+        96 products, as the README says a category page yields.
+    validations:
+      required: true
+
+  - type: textarea
+    id: environment
+    attributes:
+      label: Environment
+      description: |
+        Please run and paste:
+        ```
+        python3 -c "import sys, platform; print(sys.version, platform.platform())"
+        pip list 2>/dev/null | grep -Ei "playwright|pyppeteer|selenium|beautifulsoup4|requests|webdriver"
+        python3 smoke_test.py | tail -3
+        ```
+      render: text
+    validations:
+      required: true
+
+  - type: checkboxes
+    id: checks
+    attributes:
+      label: Checks
+      options:
+        - label: The offline suite passes on my machine (`python3 smoke_test.py`)
+          required: true
+        - label: I removed every API key, password and credentialled URL from this report
+          required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,14 @@
+blank_issues_enabled: true
+contact_links:
+  - name: 2Captcha support
+    url: https://2captcha.com/support
+    about: >-
+      Billing, an API key that will not authenticate, profile or quota limits
+      (ERROR_MAX_PROFILES), proxy availability. Anything about the service
+      rather than about this code — support can see your account, we cannot.
+  - name: 2Captcha API documentation
+    url: https://2captcha.com/api-docs
+    about: Task types, parameters and error codes for the solver API.
+  - name: Other scrapers in this family
+    url: https://github.com/2scraper
+    about: Same structure, other sites.

.github/ISSUE_TEMPLATE/question.yml

@@ -0,0 +1,30 @@
+name: Question
+description: How do I ... ?
+labels: ["question"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Worth a look first, in this order:
+
+        - [TROUBLESHOOTING.md](../blob/main/TROUBLESHOOTING.md) — ordered by how
+          often each thing actually happens
+        - The README section on the 2Captcha product you are using
+        - `python3 env_config.py` — reports what credentials are configured,
+          without printing them
+
+  - type: textarea
+    id: question
+    attributes:
+      label: What are you trying to do
+      description: >-
+        The goal, not only the step that is stuck — often there is a shorter
+        route to it.
+    validations:
+      required: true
+
+  - type: textarea
+    id: tried
+    attributes:
+      label: What you tried
+      render: shell

.github/ISSUE_TEMPLATE/site_changed.yml

@@ -0,0 +1,58 @@
+name: Site changed / selectors broke
+description: Amazon changed something and the parser no longer matches.
+labels: ["site-change"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        This is the most common way a scraper stops working, and it is worth its
+        own report because the fix is different from a code bug: something on
+        amazon.com moved.
+
+        Useful to know before filing: the parser tries **JSON-LD first**, then a
+        CSS + URL-pattern fallback. Which one broke narrows the fix a lot.
+
+  - type: dropdown
+    id: what_broke
+    attributes:
+      label: What stopped matching
+      options:
+        - Products (0 rows on a page that visibly has products)
+        - A specific field is null or wrong (price, brand, sku, image, URL)
+        - Pagination stops early or repeats a page
+        - The captcha detector finds nothing on a page that has one
+        - Something else
+    validations:
+      required: true
+
+  - type: input
+    id: url
+    attributes:
+      label: URL
+      description: The exact listing URL, and the country your exit IP was in.
+      placeholder: "https://www.amazon.com/s?k=wireless+headphones (marketplace: .com, exit IP: DE)"
+    validations:
+      required: true
+
+  - type: textarea
+    id: evidence
+    attributes:
+      label: Evidence from the page
+      description: |
+        Whichever of these you can get. A capture beats a description.
+
+        - JSON-LD as the page actually serves it:
+          `python3 -c "import json,sys,re;h=open('dump.html').read();print([m[:400] for m in re.findall(r'<script type=\"application/ld\\+json\">(.*?)</script>',h,re.S)][:2])"`
+        - Or open devtools and paste one `Product` node from the `ItemList`.
+        - For a field problem: the value you got, and the value on the page.
+      render: text
+    validations:
+      required: true
+
+  - type: textarea
+    id: notes
+    attributes:
+      label: Anything else
+      description: >-
+        Whether it reproduces on a second engine, whether it reproduces from a
+        different country, and when it last worked.