Keep good output on a failed write, escape CSV formulas, fail the canary without a secret

.env.example

@@ -54,8 +54,8 @@ TWOCAPTCHA_KEY=your_2captcha_api_key_here
 # Selenium cannot use this: chromedriver's `debuggerAddress` takes a bare
 # host:port with nowhere to put a password. Playwright and pyppeteer can.
 #
-# The credentials in a profile expire in about a day, so never paste a
-# working one anywhere that is committed.
+# The endpoint carries a password: never paste a working one anywhere
+# that is committed.
 WEBMOTORS_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-br-pid-{profileId}:{password}@cb.2captcha.com:9222
 
 # --- Proxy ----------------------------------------------------------------
@@ -78,10 +78,10 @@ WEBMOTORS_PROXY=http://{user}-zone-custom-region-br:{password}@na.proxy.2captcha
 #   https://www.webmotors.com.br/carros/sp/toyota?anode=2020
 #   https://www.webmotors.com.br/motos/estoque/honda
 #
-# While it is set, the search flags (--make, --model, --state ...) are
-# refused, since they would have to agree with it and nothing checks that
-# they do. Leave it unset if you use the flags.
-WEBMOTORS_URL=https://www.webmotors.com.br/carros/estoque/volkswagen/gol
+# It is only a default: a --url, search flags (--make, --model, --state
+# ...) or an --ads-file typed on the command line replace it. Shipped
+# commented out, so a copied .env does not pick a search for you.
+# WEBMOTORS_URL=https://www.webmotors.com.br/carros/estoque/volkswagen/gol
 
 # Deliberately NOT here: an output prefix. `--out` already carries a
 # non-empty default, so the loader, which only fills UNSET values, would

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -70,8 +70,8 @@ body:
       render: shell
       placeholder: |
         python3 playwright_scraper.py \
-          --asset USDT --fiat EUR --side buy \
-          --pages 3 --out usdt_eur
+          --make volkswagen --model gol \
+          --pages 3 --out gol
     validations:
       required: true
 
@@ -91,7 +91,8 @@ body:
     attributes:
       label: What you expected instead
       placeholder: >-
-        60 adverts over 3 pages, as the site shows ~200 for USDT/EUR.
+        About 141 listings over 3 pages of 47, as the site shows over a
+        thousand Gol listings.
     validations:
       required: true
 

.github/ci_checks.py

@@ -89,8 +89,8 @@
 # This site publishes no 32-hex identifier at all: an advert's id is a short
 # integer and its photo file names are `WMIMAGEM` plus digits, checked in
 # every capture rather than assumed. So a bare 32-hex string anywhere in the
-# repo fails, with no exception to reason about (CLAUDE.md §24: before
-# adding an exemption, check whether the value is needed at all).
+# repo fails, with no exception to reason about (before adding an
+# exemption, check whether the value is needed at all).
 SITE_PUBLIC_IDS = ()
 
 def _without_site_ids(line):
@@ -279,7 +279,7 @@ def tracked_files():
 # into pip's vendored code and flags a 32-hex string in `_elffile.py` as
 # key-shaped. Correct about the string, wrong about the file — and it is the
 # FIRST thing a new user sees from `python3 smoke_test.py`. A guard people
-# have to argue with is one they learn to suppress (CLAUDE.md §22).
+# have to argue with is one they learn to suppress.
 #
 # Deliberately NOT a narrowing of what gets scanned: an untracked file is
 # still read, because a key pasted into a scratch file beside the scripts is

.github/workflows/canary.yml

@@ -1,22 +1,25 @@
-# One real run of each mode against webmotors.com.br, daily. tests.yml
-# (deliberately) never touches the site.
+# A real run of each mode against webmotors.com.br, daily: a car search, a
+# motorcycle search, and the first adverts of the car search read in ad mode.
+# tests.yml (deliberately) never touches the site.
 #
-# THIS CANARY NEEDS A SECRET, and without one it SKIPS rather than fails
-# (CLAUDE.md §11). CloudFront refused every datacentre address measured on
-# 2026-09-28 — a VPS, headful Chromium included, and the Scraper API's own
-# exits — and a GitHub-hosted runner is a datacentre address. A canary that
-# cannot pass is red every day, and a check that is always red teaches
-# everyone to ignore checks. So the live steps run only when a RESIDENTIAL
-# proxy is configured:
+# THIS CANARY NEEDS A SECRET, and without one it FAILS. CloudFront refused
+# every datacentre address measured on 2026-09-28 (a VPS, headful Chromium
+# included, and the Scraper API's own exits), and a GitHub-hosted runner is a
+# datacentre address. So a run from the bare runner cannot reach the
+# listings, and a canary that skipped green in that case reported a scraper
+# as working when nothing had been tested. Set ONE of:
 #
 #     gh secret set WEBMOTORS_PROXY --repo 2scraper/webmotors-scraper
+#         a residential exit, http://user:pass@host:port
+#     gh secret set WEBMOTORS_CDP_ENDPOINT --repo 2scraper/webmotors-scraper
+#         a 2Captcha Scraping Browser API endpoint (country-br)
 #
-# It goes through the ENVIRONMENT, not a --proxy line: env_config.py reads
-# exactly that variable, an unset secret arrives as an empty string which the
-# loader treats as unset, and the credential never reaches a command line.
-#
-# Not a Scraping Browser endpoint: a profile's credentials expire in about a
-# day, so a secret holding one would rot between runs (§19).
+# They go through the ENVIRONMENT, not a command line: env_config.py reads
+# exactly those variables, an unset secret arrives as an empty string which
+# the loader treats as unset, and the credential never reaches a command
+# line. A Scraping Browser endpoint stops working when its credentials are
+# refused; how long that takes is not a fixed figure, so a 401 here means
+# "copy the endpoint again", not a broken scraper.
 
 name: canary
 
@@ -44,40 +47,39 @@ jobs:
           - name: search
             args: "--make volkswagen --model gol --pages 3"
             floor: 120
+            ads: 3
           - name: motorcycles
             args: "--category motos --make honda --pages 3"
             floor: 120
+            ads: 0
     env:
       WEBMOTORS_PROXY: ${{ secrets.WEBMOTORS_PROXY }}
+      WEBMOTORS_CDP_ENDPOINT: ${{ secrets.WEBMOTORS_CDP_ENDPOINT }}
     steps:
-      - name: Skip without a residential proxy
-        if: env.WEBMOTORS_PROXY == ''
+      # No skip: a run that never contacted the site must not end green.
+      - name: Require a residential exit
         run: |
-          echo "::notice::No WEBMOTORS_PROXY secret. CloudFront refuses datacentre addresses (a GitHub runner is one), so a live run here could only fail. Set a residential proxy secret to enable this canary (see the header)."
-          # A green run that tested nothing must SAY so where a reader looks.
-          # The badge cannot: GitHub colours it by the run's conclusion
-          # (third-party audit, 2026-10-08).
-          echo "## ${{ matrix.name }}: SKIPPED — the site was NOT contacted" >> "$GITHUB_STEP_SUMMARY"
-          echo "No residential proxy secret is set, so no live step ran. This run proves the workflow is intact, not that the scraper works today." >> "$GITHUB_STEP_SUMMARY"
+          if [ -z "$WEBMOTORS_PROXY" ] && [ -z "$WEBMOTORS_CDP_ENDPOINT" ]; then
+            echo "::error::Neither the WEBMOTORS_PROXY nor the WEBMOTORS_CDP_ENDPOINT secret is set. CloudFront refuses datacentre addresses (a GitHub runner is one), so this canary cannot reach the site without one. See the workflow header."
+            echo "## ${{ matrix.name }}: FAILED — the site was NOT contacted" >> "$GITHUB_STEP_SUMMARY"
+            echo "Set the WEBMOTORS_PROXY or WEBMOTORS_CDP_ENDPOINT secret." >> "$GITHUB_STEP_SUMMARY"
+            exit 1
+          fi
 
       - uses: actions/checkout@v4
-        if: env.WEBMOTORS_PROXY != ''
       - uses: actions/setup-python@v5
-        if: env.WEBMOTORS_PROXY != ''
         with:
           python-version: "3.12"
 
       - name: Install
-        if: env.WEBMOTORS_PROXY != ''
         run: |
           python -m pip install --upgrade pip
           pip install -r requirements.txt -r requirements-playwright.txt
           playwright install --with-deps chromium
 
-      # >= 3 pages, because with one page pagination is never exercised (§11).
+      # >= 3 pages, because with one page pagination is never exercised.
       - name: Run
         id: run
-        if: env.WEBMOTORS_PROXY != ''
         run: |
           set +e   # `run:` is `bash -e`; without this a non-zero exit aborts
                    # the step before the echo below records it.
@@ -86,7 +88,6 @@ jobs:
           echo "exit_code=$?" >> "$GITHUB_OUTPUT"
 
       - name: Interpret
-        if: env.WEBMOTORS_PROXY != ''
         run: |
           code="${{ steps.run.outputs.exit_code }}"
           echo "exit code: $code"
@@ -101,7 +102,6 @@ jobs:
           esac
 
       - name: Assert
-        if: env.WEBMOTORS_PROXY != ''
         run: |
           python - <<'EOF'
           import json, sys
@@ -141,8 +141,56 @@ jobs:
           print("canary OK: %d rows of %s, complete" % (len(rows), meta["total_results"]))
           EOF
 
+      # Ad mode: the first adverts the search returned, each read from its
+      # own detail endpoint.
+      - name: Run ad mode
+        id: ads
+        if: matrix.ads > 0
+        run: |
+          python - <<'EOF'
+          import json
+          rows = json.load(open("canary.json"))
+          with open("canary_ads.txt", "w") as f:
+              f.write("\n".join(r["url"] for r in rows[:${{ matrix.ads }}]) + "\n")
+          EOF
+          set +e
+          python playwright_scraper.py --ads-file canary_ads.txt \
+            --out canary_ad --format both --dump-html canary_ad_dump.json
+          code=$?
+          echo "exit code: $code"
+          if [ "$code" != 0 ]; then
+            echo "::error::Ad mode exited $code (3 blocked, 5 never arrived, 6 some adverts failed). See canary_ad.meta.json."
+            exit 1
+          fi
+
+      - name: Assert ad mode
+        if: matrix.ads > 0
+        run: |
+          python - <<'EOF'
+          import json, sys
+          meta = json.load(open("canary_ad.meta.json"))
+          rows = json.load(open("canary_ad.json"))
+          want = ${{ matrix.ads }}
+          problems = []
+          if meta["status"] != "complete":
+              problems.append("status %r (%s)" % (meta["status"], meta["stop_reason"]))
+          if len(rows) != want:
+              problems.append("%d adverts, asked for %d" % (len(rows), want))
+          for r in rows:
+              if not (isinstance(r["price"], float) and r["price"] > 0):
+                  problems.append("advert %s has no price" % r["sku"])
+              if not r.get("make") or not r.get("year_model"):
+                  problems.append("advert %s lacks make/year_model" % r["sku"])
+              if r["seller_type"] == "private" and (r["seller_name"] or r["description"]):
+                  problems.append("private advert %s carries a name or text" % r["sku"])
+          if problems:
+              print("ad canary FAILED: " + "; ".join(problems))
+              sys.exit(1)
+          print("ad canary OK: %d adverts" % len(rows))
+          EOF
+
       - uses: actions/upload-artifact@v4
-        if: always() && env.WEBMOTORS_PROXY != ''
+        if: always()
         with:
           name: canary-${{ matrix.name }}
           path: |
@@ -151,4 +199,7 @@ jobs:
             canary.meta.json
             canary_dump.json*
             canary_page*_debug.html
+            canary_ad.json
+            canary_ad.meta.json
+            canary_ad_dump.json*
           if-no-files-found: ignore

.gitignore

@@ -21,7 +21,7 @@ webmotors_rows_*.*
 live_results/
 live_results.zip
 # ANY working directory a live run writes into, by KIND rather than by the one
-# name an older repo used (CLAUDE.md §24). Measured in a sibling: `live/` was not
+# name an older repo used. Measured in a sibling: `live/` was not
 # listed, and a `--dump-html` of a 3-page run wrote `canary_dump.json.page3`,
 # which `*.json` does not cover. It sat untracked and NOT ignored, one
 # `git add -A` from a commit, until the credential scan flagged the 32-hex
@@ -95,7 +95,7 @@ secrets*
 # recoverable only because that repo was private at the time. On 2026-09-17
 # a sibling had an untracked `.env.bak` holding a 32-hex key in a PUBLIC
 # repository, one `git add -A` from publication. Craigslist was fixed then
-# and the fix never reached anyone else, which is §16's rule unapplied: when
-# you fix one, check the siblings.
+# and the fix never reached anyone else. When you fix one, check the
+# siblings.
 .env*
 !.env.example

CHANGELOG.md

@@ -16,8 +16,56 @@ said so at the top of its release notes.
 > script of yours passes 0, pass **1**: the flag counts ATTEMPTS, so 1 means
 > "try once and do not retry".
 
+> **Behaviour change:** negative values of `--delay`, `--retry-delay` and
+> `--proxy-block-retries`, `--concurrency` below 1 and `--pages` below 1 are
+> now usage errors (exit 2) naming the flag, in every browser engine. In
+> `scraper_api_client.py`, `--retries`, `--delay` and `--retry-delay` below 0
+> and `--pages` below 1 are refused the same way (`--retries 0` stays legal
+> there: it counts EXTRA attempts).
+
+> **Behaviour change:** CSV cells that begin with `=`, `+`, `-`, `@`, a tab
+> or a line break now get a leading `'`, so a spreadsheet reads them as text.
+> Numbers and the JSON output are unchanged; the sidecar's
+> `csv_cells_escaped` says how many cells were changed.
+
+> **Behaviour change:** the canary now FAILS, naming the missing secret, when
+> neither `WEBMOTORS_PROXY` nor `WEBMOTORS_CDP_ENDPOINT` is set. It used to
+> skip and end green without contacting the site.
+
 ### Fixed
 
+- **A failed write destroyed the previous good output.** JSON, CSV, the
+  `.meta.json` sidecar and `diff_runs.py --out` were opened with truncation,
+  so a full disk or an interrupt mid-write left a short, invalid file where a
+  good one had been. Each file is now written to a temporary file in the same
+  directory, flushed, synced and renamed over the target; the temporary file
+  is removed on failure. A new file gets the mode `open()` would have given
+  it (`0o666` minus the umask); an existing file keeps its own mode. The
+  three files are replaced one by one, not as a single transaction.
+- **CSV cells were not protected against formula injection.** Titles,
+  versions, attributes, optionals and dealer names and descriptions are
+  third-party text, and a cell starting with `=` ran as a formula when the
+  CSV was opened in a spreadsheet. Escaping runs after list columns are
+  joined.
+- **A copied `.env.example` broke the README's flag examples.** It shipped an
+  active `WEBMOTORS_URL`, and a URL from `.env` was treated as a typed
+  `--url`, so `--make volkswagen --model gol` exited 2. The example line now
+  ships commented out, and a `WEBMOTORS_URL` from the environment is only a
+  default: typed search flags, `--ads-file` or a different `--mode` replace
+  it. A typed `--url` with filter flags is still refused.
+- **The Scraping Browser 401 hint claimed credentials "last about a day".**
+  That figure is not a constant (an endpoint was seen working after ten
+  days). The hint, README, TROUBLESHOOTING and `.env.example` now say the
+  credentials were refused (expired, revoked or mistyped) without guessing a
+  lifetime.
+- **The bug-report template's example command used flags this scraper does
+  not have.** It now shows a Volkswagen Gol search.
+- **The banned-wording check read virtualenvs.** A `.venv` inside the repo,
+  as the README suggests, had its site-packages scanned. Directories holding
+  a `pyvenv.cfg` are now skipped.
+- **Comments cited private notes by section number.** They now state the
+  reason in place, and a suite check fails if such a citation returns.
+
 - **`--retries 0` fetched nothing and called it a result.** The load loop is
   `range(1, retries + 1)`, so 0 never runs it and the engine judged
   `about:blank`. Three third-party audits of this family ran exactly that and
@@ -26,6 +74,13 @@ said so at the top of its release notes.
   one and calls the validator, so a new engine declaring the flag with a
   plain int fails the suite.
 
+### Changed
+
+- **The canary covers ad mode** (the first three adverts of the car search)
+  and accepts a 2Captcha Scraping Browser API endpoint
+  (`WEBMOTORS_CDP_ENDPOINT`) as well as a residential proxy. Without either
+  secret it fails rather than skipping.
+
 ## [0.1.1] — 2026-10-08
 
 > **Exit codes change for a page the parser cannot read.** v0.1.0 treated a

CONTRIBUTING.md

@@ -90,12 +90,12 @@ history needs a decision, not a red check on every push.
 Then the rest of the presentation, in the order that matters:
 
 1. `python3 smoke_test.py` green, and the canary dispatched at least once.
-   Without a `WEBMOTORS_PROXY` secret it SKIPS with a notice, because
-   CloudFront refuses the runner's datacentre address; dispatch it once to
-   see the skip branch run, and again with the secret set to see the real
-   one.
-2. The repo description, homepage and topics set (see the family notes on
-   what those should say).
+   It needs a `WEBMOTORS_PROXY` (residential) or `WEBMOTORS_CDP_ENDPOINT`
+   secret, because CloudFront refuses the runner's datacentre address;
+   without either it FAILS and names the missing secret.
+2. The repo description, homepage and topics set (what the scraper reads,
+   which engines and paid products it supports, and nothing it does not
+   use).
 3. Only then the row in the org profile README — and check it with an
    ANONYMOUS request rather than your own logged-in browser. A row pointing
    at a private repo is a 404 for every visitor, which costs more trust than

README.md

@@ -182,8 +182,7 @@ One key, four separately-billed products:
   the top for the acceptance rates measured.
 - **The Scraping Browser API** (`--cdp-endpoint`) — a remote browser on a
   residential exit, nothing to install; served on every attempt measured.
-  One live connection per profile, so no `--concurrency` through it; a
-  profile's credentials last about a day.
+  One live connection per profile, so no `--concurrency` through it.
 - **The Scraper API** (`scraper_api_client.py`) — no local browser at all,
   but its own exits were refused, so on this site it works only with
   `--cdp-url` (a Scraping Browser session), measured at $0.0005 a page.
@@ -227,12 +226,14 @@ captured on 2026-09-28 (`tools/capture.py`), cut and scrubbed by
 `make_fixtures.py`, which proves each one parses exactly as its original.
 CI runs the suite on Python 3.9 and 3.12, once per engine in its own
 virtualenv, builds the Docker image, and scans for committed credentials.
-The canary runs a real search daily when a residential proxy secret is set,
-and skips with a notice when it is not. **This repository has no such secret
-set, so the canary badge above is green because it skipped:** it says the
-workflow is intact, not that the site was scraped today. Each skipped run
-says so at the top of its summary page. Measure the scraper yourself with
-the commands under "Run".
+The canary runs a car search, a motorcycle search and ad mode daily
+through a residential exit: the `WEBMOTORS_PROXY` or
+`WEBMOTORS_CDP_ENDPOINT` secret, because the site refuses the runner's
+datacentre address. Without either secret the canary **fails** and names
+the missing secret, rather than reporting green for a run that never
+contacted the site. A red canary badge may therefore mean that no secret is
+set; its summary page says which. Measure the scraper yourself with the
+commands under "Run".
 
 ## Legal