Atomic output writes, CSV formula neutralisation, 0600 HTML dumps

CHANGELOG.md

@@ -13,8 +13,24 @@ with it, so nobody discovers it from a bill or from a diff.
 
 ## [Unreleased]
 
+### Changed
+
+- **CSV output now neutralises spreadsheet formulas.** A text cell that
+  begins with `=`, `+`, `-`, `@`, tab, CR or LF is prefixed with `'`; numbers
+  are not touched, JSON is not touched, and `csv_cells_escaped` in the sidecar
+  counts the cells. A consumer parsing the CSV with code will see the
+  apostrophe.
+
 ### Fixed
 
+- **Output files were written in place, so a crash mid-write truncated the
+  previous good run.** `write_json`, `write_csv` and the `.meta.json` sidecar
+  now write to a temporary file in the same directory, `fsync`, and
+  `os.replace` it. A new output is mode 0644 and a rewritten one keeps its
+  existing mode.
+- **HTML dumps (`--dump-html`, the `_debug.html` files) are now mode 0600.**
+  They are what the site served this session and are what people paste into
+  issues.
 - **The Scraper API path (`scraper_api_client.py`) failed whenever a wait flag
   was given, and never saw the target's status.** Measured 2026-09-23 against
   the live `/tasks/sync` endpoint: `waitFor` sent as a JSON-encoded string

README.md

@@ -263,6 +263,14 @@ across this scraper family — the first sixteen columns are identical in every
 repo in it — with Etsy's own columns appended. See
 [`sample_output.json`](sample_output.json), which is cut from a real run.
 
+Files are written to a temporary file beside the target and renamed over it, so
+a crash or a full disk leaves the previous run's output whole instead of a
+truncated file. In the CSV only, a text cell that starts with `=`, `+`, `-`, `@`,
+tab or a line break gets a leading `'` so a spreadsheet reads it as text (titles
+and shop names are written by sellers); the JSON keeps the site's bytes, and
+`csv_cells_escaped` in `<out>.meta.json` counts the cells changed. Numbers are
+never touched. Not measured on Etsy yet: how often this fires on real data.
+
 | column | notes |
 |---|---|
 | `source` `scraped_at` `url` `sku` `title` | `sku` is Etsy's listing id |

SECURITY.md

@@ -100,7 +100,8 @@ Three places leak credentials that people do not expect, because unlike our own
 log lines they are **not** masked:
 
 - **raw HTML dumps** (`--dump-html`, and the automatic dump on a zero-product
-  run) — these can contain session cookies
+  run) — a snapshot of what the site served this session, which can carry
+  per-session tokens. They are written owner-only (0600) but are not scrubbed
 - **the Scraper API's `x-debug` response header**
 - **your shell history**, if you passed a key on the command line
 

output_writer.py

@@ -34,11 +34,14 @@
 gets the right header for the mode that produced it.
 """
 
+import contextlib
 import csv
 import json
+import os
+import tempfile
 from dataclasses import dataclass, asdict, field, fields
 from datetime import datetime, timezone
-from typing import Optional, List, Set, Sequence, Any, Type
+from typing import Optional, List, Set, Sequence, Any, Type, Tuple
 
 
 # The hostname a row came from. Etsy is ONE host — every market lives on
@@ -255,25 +258,123 @@ def _csv_value(v: Any) -> Any:
     return v
 
 
+# A cell starting with one of these is executed by Excel, Google Sheets and
+# LibreOffice. Every string in a row was written by whoever wrote the listing.
+CSV_FORMULA_LEADS = ("=", "+", "-", "@", "\t", "\r", "\n")
+
+# Spreadsheets treat a leading apostrophe as "the rest is text" and do not
+# show it; a consumer parsing the CSV with `csv` sees it, which is why the
+# count goes in the sidecar instead of staying silent.
+CSV_FORMULA_ESCAPE = "'"
+
+
+def _csv_escape(v: Any) -> Tuple[Any, bool]:
+    """Neutralise a formula-shaped cell. Returns (value, was_escaped).
+
+    Only STRINGS are touched: turning the number -5 into the text '-5 would
+    break every sum a consumer writes over the column. CSV only — the JSON
+    output keeps the site's bytes, and `csv_cells_escaped` in the sidecar
+    declares that the two files differ.
+    """
+    if isinstance(v, str) and v.startswith(CSV_FORMULA_LEADS):
+        return CSV_FORMULA_ESCAPE + v, True
+    return v, False
+
+
+@contextlib.contextmanager
+def _atomic(path: str, newline: Optional[str] = None):
+    """Write to a temporary file beside `path`, then rename over it.
+
+    Every write here replaces a file a previous run may have left, and the
+    invariant this module exists to protect is that a bad run never destroys
+    last night's good data (`save` refuses an empty result for the same
+    reason). Writing in place gives that up at the worst moment: a kill, a
+    full disk or a crash halfway through `json.dump` leaves a TRUNCATED file
+    where a complete one was, and the sidecar beside it still describes the
+    old, good run.
+
+    The temporary file lives in the SAME directory, because a rename across
+    filesystems is not atomic and degrades to an interruptible copy. `fsync`
+    before the rename is what makes it hold after a power loss and not only
+    after a crash.
+
+    NamedTemporaryFile creates the file 0600 and a rename keeps that, so the
+    output would come out unreadable to the group and to other users where
+    `open()` gave 0644. Restore the destination's own mode, or 0644.
+    """
+    directory = os.path.dirname(os.path.abspath(path)) or "."
+    handle = tempfile.NamedTemporaryFile(
+        mode="w", encoding="utf-8", newline=newline, dir=directory,
+        prefix=os.path.basename(path) + ".", suffix=".tmp", delete=False)
+    try:
+        with handle:
+            yield handle
+            handle.flush()
+            os.fsync(handle.fileno())
+        try:
+            mode = os.stat(path).st_mode & 0o777
+        except OSError:
+            mode = 0o644
+        os.chmod(handle.name, mode)
+        os.replace(handle.name, path)
+    except BaseException:
+        # Leave the destination untouched: a failed write must not be visible.
+        try:
+            os.unlink(handle.name)
+        except OSError:
+            pass
+        raise
+
+
+def write_private_text(path: str, text: str) -> None:
+    """Write a diagnostic dump readable by the owner only (0600).
+
+    A page snapshot is the one artefact here that is not ours: it is whatever
+    the site served this session, and it is what people paste into issues.
+    `open()` would give it 0644 under the usual umask. The mode is also
+    applied to a file that already exists, since os.open only uses it on
+    creation.
+    """
+    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+    try:
+        os.chmod(path, 0o600)
+    except OSError:
+        pass
+    with os.fdopen(fd, "w", encoding="utf-8") as f:
+        f.write(text)
+
+
 def write_json(rows: Sequence[Any], path: str) -> None:
-    with open(path, "w", encoding="utf-8") as f:
+    with _atomic(path) as f:
         json.dump([asdict(r) for r in rows], f, ensure_ascii=False, indent=2)
 
 
-def write_csv(rows: Sequence[Any], path: str, row_cls: Type = Product) -> None:
+def write_csv(rows: Sequence[Any], path: str, row_cls: Type = Product) -> int:
+    """Write the rows; return how many cells were formula-neutralised."""
     # An empty result still gets the header row. A zero-byte file makes a
     # consumer fail on read (no columns to parse) instead of reading a valid
-    # table with zero rows — and "an empty result is still a well-formed
-    # result" is the same principle as `save` refusing to overwrite good data.
+    # table with zero rows.
     #
     # The header comes from `row_cls`, not from the first row, so an empty
     # run still writes the columns of the mode that produced it.
     fieldnames = [f.name for f in fields(row_cls)]
-    with open(path, "w", encoding="utf-8", newline="") as f:
+    escaped = 0
+    # newline="" is the csv module's requirement and has to survive the move
+    # into the temp file, or every row gains a stray CR on Windows.
+    with _atomic(path, newline="") as f:
         writer = csv.DictWriter(f, fieldnames=fieldnames)
         writer.writeheader()
         for r in rows:
-            writer.writerow({k: _csv_value(v) for k, v in asdict(r).items()})
+            row = {}
+            for k, v in asdict(r).items():
+                # Escape AFTER `_csv_value`: a list joined into one cell is
+                # text too, and its first element can be formula-shaped
+                # while the list itself is not a string.
+                value, was_escaped = _csv_escape(_csv_value(v))
+                row[k] = value
+                escaped += was_escaped
+            writer.writerow(row)
+    return escaped
 
 
 # Exit code used when a run completes but produced nothing. Distinct from 1
@@ -350,7 +451,7 @@ def write_run_meta(out_prefix: str, meta: dict) -> str:
     both complete, and between runs of different `mode`.
     """
     path = f"{out_prefix}.meta.json"
-    with open(path, "w", encoding="utf-8") as f:
+    with _atomic(path) as f:
         json.dump(meta, f, ensure_ascii=False, indent=2)
     print(f"[+] Wrote run metadata -> {path} (status={meta.get('status')})")
     return path
@@ -412,7 +513,8 @@ def run_meta(status: str, stop_reason: str, pages_requested: int,
 
 
 def save(rows: Sequence[Any], out_prefix: str, fmt: str,
-         allow_empty: bool = False, row_cls: Type = Product) -> int:
+         allow_empty: bool = False, row_cls: Type = Product,
+         stats: Optional[dict] = None) -> int:
     """Write JSON/CSV and return a process exit code.
 
     Returns 0 when rows were written, EXIT_NO_PRODUCTS when there were none.
@@ -428,6 +530,9 @@ def save(rows: Sequence[Any], out_prefix: str, fmt: str,
 
     `allow_empty=True` is for the legitimate case: a filter that genuinely
     matches nothing, where an empty file is the answer.
+
+    Facts about the write go into `stats` rather than the return value,
+    because the return IS the exit code every engine exits with.
     """
     if not rows and not allow_empty:
         print(f"[!] 0 products — refusing to write {out_prefix}.json/.csv, so an "
@@ -439,8 +544,15 @@ def save(rows: Sequence[Any], out_prefix: str, fmt: str,
         write_json(rows, f"{out_prefix}.json")
         print(f"[+] Saved {len(rows)} products -> {out_prefix}.json")
     if fmt in ("csv", "both"):
-        write_csv(rows, f"{out_prefix}.csv", row_cls=row_cls)
+        escaped = write_csv(rows, f"{out_prefix}.csv", row_cls=row_cls)
         print(f"[+] Saved {len(rows)} products -> {out_prefix}.csv")
+        if stats is not None:
+            stats["csv_cells_escaped"] = escaped
+        if escaped:
+            print(f"[!] {escaped} CSV cell(s) began with a formula character "
+                  f"and were prefixed with {CSV_FORMULA_ESCAPE!r} so a "
+                  f"spreadsheet reads them as text. The JSON output is "
+                  f"unchanged — see csv_cells_escaped in the sidecar.")
     return 0 if rows else EXIT_NO_PRODUCTS
 
 
@@ -495,7 +607,9 @@ def finish_run(rows: Sequence[Any], out_prefix: str, fmt: str,
     # names has a hole for every name nobody added to it.
     complete = stop_reason in COMPLETE_STOP_REASONS and not pages_failed
     row_cls = ROW_CLASS_BY_MODE.get(mode, Product)
-    rc = save(rows, out_prefix, fmt, allow_empty=allow_empty, row_cls=row_cls)
+    save_stats: dict = {}
+    rc = save(rows, out_prefix, fmt, allow_empty=allow_empty, row_cls=row_cls,
+              stats=save_stats)
     wrote_output = bool(rows) or allow_empty
 
     if wrote_output:
@@ -506,7 +620,9 @@ def finish_run(rows: Sequence[Any], out_prefix: str, fmt: str,
             pages_requested=pages_requested, pages_completed=pages_completed,
             pages_failed=pages_failed, mode=mode, source=source,
             start_url=start_url, final_url=final_url, products=len(rows),
-            extra=extra))
+            # The caller's `extra` wins a collision: housekeeping must never
+            # silently drop site data.
+            extra={**save_stats, **(extra or {})}))
 
     if not rows:
         # Nothing gathered at all, and WHY decides the code. The three

playwright_scraper.py

@@ -93,7 +93,8 @@
                             listing_kind, site_host, is_supported_host,
                             total_results, LOCALE_CURRENCY, locale_of,
                             unsupported_reason)
-from output_writer import dedupe_by_key, finish_run, EXIT_API_ERROR
+from output_writer import (dedupe_by_key, finish_run, EXIT_API_ERROR,
+                           write_private_text)
 import page_flow
 from page_flow import MIN_CARD_MATCHES
 from proxy_pool import (from_args as proxy_pool_from_args, to_playwright, mask,
@@ -923,8 +924,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome
         # Naming that, and saying what DOES clear it, is more use than a
         # captcha hint that would cost money.
         debug_html = f"{args.out}_page{page_num}_debug.html"
-        with open(debug_html, "w", encoding="utf-8") as f:
-            f.write(html)
+        write_private_text(debug_html, html)
         verdict = datadome_verdict(html)
         logger.error(
             "Etsy refused this request: DataDome answered HTTP 403 with "
@@ -992,8 +992,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome
     if args.dump_html:
         dump_path = (args.dump_html if args.pages == 1
                      else f"{args.dump_html}.page{page_num}")
-        with open(dump_path, "w", encoding="utf-8") as f:
-            f.write(html)
+        write_private_text(dump_path, html)
         logger.info("Saved the snapshot the parser sees to %s (%d bytes).",
                     dump_path, len(html))
 
@@ -1011,8 +1010,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome
               if state != "content" else None)
     if vendor:
         debug_html = f"{args.out}_page{page_num}_debug.html"
-        with open(debug_html, "w", encoding="utf-8") as f:
-            f.write(html)
+        write_private_text(debug_html, html)
         try:
             session.page.screenshot(path=f"{args.out}_page{page_num}_debug.png",
                                     full_page=True)
@@ -1099,8 +1097,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome
     if not products:
         debug_html = f"{args.out}_page{page_num}_debug.html"
         debug_png = f"{args.out}_page{page_num}_debug.png"
-        with open(debug_html, "w", encoding="utf-8") as f:
-            f.write(html)
+        write_private_text(debug_html, html)
         try:
             session.page.screenshot(path=debug_png, full_page=True)
         except Exception as e:

puppeteer_scraper.py

@@ -70,7 +70,8 @@
                             page_url, listing_kind,
                             site_host, is_supported_host, total_results,
                             unsupported_reason)
-from output_writer import dedupe_by_key, finish_run, EXIT_API_ERROR
+from output_writer import (dedupe_by_key, finish_run, EXIT_API_ERROR,
+                           write_private_text)
 import page_flow
 from page_flow import MIN_CARD_MATCHES
 from proxy_pool import (from_args as proxy_pool_from_args, mask, ROTATE_MODES,
@@ -687,8 +688,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome
         # under a 403. Saying so plainly, and saying what actually clears it,
         # is more use than a captcha hint that does not apply.
         debug_html = f"{args.out}_page{page_num}_debug.html"
-        with open(debug_html, "w", encoding="utf-8") as f:
-            f.write(html)
+        write_private_text(debug_html, html)
         logger.error("Etsy refused this request (DataDome under HTTP 403 "
                      "under HTTP 403) — saved to %s. There is no challenge on "
                      "that page to solve, so a 2Captcha key does not help; a "
@@ -736,16 +736,14 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome
     if args.dump_html:
         dump_path = (args.dump_html if args.pages == 1
                      else f"{args.dump_html}.page{page_num}")
-        with open(dump_path, "w", encoding="utf-8") as f:
-            f.write(html)
+        write_private_text(dump_path, html)
         logger.info("Saved the snapshot the parser sees to %s (%d bytes).",
                     dump_path, len(html))
 
     vendor = detect_bot_challenge(html, url=page.url)
     if vendor:
         debug_html = f"{args.out}_page{page_num}_debug.html"
-        with open(debug_html, "w", encoding="utf-8") as f:
-            f.write(html)
+        write_private_text(debug_html, html)
         try:
             bridge.run(page.screenshot({"path": f"{args.out}_page{page_num}_debug.png",
                                         "fullPage": True}))
@@ -811,8 +809,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome
 
     if not products:
         debug_html = f"{args.out}_page{page_num}_debug.html"
-        with open(debug_html, "w", encoding="utf-8") as f:
-            f.write(html)
+        write_private_text(debug_html, html)
         try:
             bridge.run(page.screenshot({"path": f"{args.out}_page{page_num}_debug.png",
                                         "fullPage": True}))