Skip to content

Atomic writes, CSV formula guard, one captcha allowance per page, Google Finance's own help text - #5

Merged
jehrr merged 2 commits into
mainfrom
fix/audit-2026-10-14
Oct 15, 2026
Merged

jehrr merged 2 commits into
mainfrom
fix/audit-2026-10-14

Conversation

@jehrr

@jehrr jehrr commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes from the 2026-10-14 audit of this repo (base 42ed314). The live earnings breakage is in a separate PR.

Problems and what happens now

Finding Before Now
Writes not atomic an injected write error turned a 748-byte out.json into 13 bytes of invalid JSON temp file in the same directory, fsync, os.replace; previous file intact, no .tmp left. New files 0o666 & ~umask, existing files keep their mode
No CSV formula guard =HYPERLINK(...) written raw CSV text cells starting = + - @ TAB CR LF get '; numbers (e.g. -1.5) and JSON unchanged; csv_cells_escaped in the sidecar, under the caller's extra
Captcha allowance a failing solver was called 2 (no pool) to 4 (3-exit pool) times against SOLVES_PER_PAGE = 1 one page_flow.SolveBudget per page, spent before the solver is called, at both call sites in all three engines: 1 call in every case
pyppeteer, no proxy AttributeError (exit 1) on a navigation timeout load failure ends the page, as in Playwright
pyppeteer/Selenium classify without --mode markets/movers/earnings pages labelled empty (readiness wait skipped) content, same as Playwright
GOOGLE_FINANCE_URL from .env --symbols MSFT:NASDAQ → "pass --symbols or --url, not both" only a typed --url conflicts; .env.example ships the URL commented out
Text about other sites block message told users to buy an exit in another site's country; issue templates, CONTRIBUTING, diff_runs --help (JPY), --headful help, scraper_api_client docstring, a "matches N, will serve M" log on every pyppeteer/Selenium market run rewritten with what was measured on Google Finance, or removed
Private citations 79 references to private notes/section numbers in 13 files removed or stated in place; a suite guard fails if one reappears (release notes v0.1.0/v0.2.0 checked: none)
CHANGELOG several behaviour changes on main since v0.2.0 unrecorded listed under [Unreleased], with a behaviour-change blockquote
Canary exit 3 (blocked) and exit 6 skipped the data checks and ended green exit 3 is an error; exit 6 warns and the row checks still run (quote status may be partial, one dropped symbol is a warning). GOOGL volume is no longer required: before the US open Google publishes 0, which the parser nulls, so a 07:00 UTC dispatch failed on it — now a notice

Evidence

  • Offline suite green in a core-only venv and in one venv per engine (requirements.txt + requirements-<engine>.txt, no browser download): each run executed its own engine's group.
  • New tests/stub_engines.py drives all three engines' _fetch_one_page and parse_args with stubbed drivers, run by the suite in a subprocess. On main it reports: pyppeteer CRASH AttributeError; 6× empty states; solver calls 2/2/2/4; --symbols with env URL exit 2 in all three engines. On this branch: no crash, content, 1/1/1/1, accepted.
  • Controls (fault planted, cache cleared, suite red on the NAMED check, restored green): non-atomic write_json; flat 0o644 & ~umask; CSV escape disabled; extra merge order reversed; Playwright allowance spend removed; pyppeteer pool guard removed; Selenium classify without mode; typed_url reverted; a section citation added to page_flow; another site's exit string added to README. All 10 named, restored suite green.
  • Canary assertion block extracted and run against the 2026-10-14 scheduled run's artefacts: the only failure is no event carries an estimate (the earnings payload change, fixed in the separate PR).

Not changed

  • The earnings parser (live breakage): separate PR on fix/site-change-2026-10-14. Until it merges, the canary on this branch stays red on that one assertion.
  • Shared fetch loop refactor, schema/sinks, version bump/tag/release: out of scope.
  • Repo description/topics: no defect found (27-word description is informational only).
  • x-debug JSON-field redaction (L2): whether the API ever echoes a key that way is not established.
  • Inline sample-output step in tests.yml (L3) and the scraper_api_client --retries double meaning (L7): left as they are.

🤖 Generated with Claude Code

… Google Finance's own help text

- Output: JSON, CSV and the .meta.json sidecar are written to a temp file
  in the target's directory and renamed; a new file gets 0o666 & ~umask,
  an existing one keeps its mode. A failed write no longer destroys the
  previous good output.
- CSV only: text cells starting with = + - @ TAB CR LF get a leading
  apostrophe; numbers and JSON untouched; csv_cells_escaped in the
  sidecar, under the caller's extra.
- Captcha: one SolveBudget per page, spent before each purchase, at both
  call sites in all three engines. A failing solver was called 2-4 times
  against a limit of 1.
- pyppeteer no longer crashes on a load failure without a proxy pool.
- pyppeteer and Selenium pass --mode to the classifier (markets, movers
  and earnings pages were labelled empty).
- A GOOGLE_FINANCE_URL from .env no longer refuses --symbols; the example
  ships it commented out.
- Issue templates, CONTRIBUTING, diff_runs --help, --headful help, the
  blocked-page message and two market-run log lines described other
  sites; rewritten for Google Finance. Private section citations removed.
- Canary: a blocked run (exit 3) is now red instead of a green skip.
- Suite: stubbed-driver probes of all three engines, writer/mode/CSV
  checks, and a guard against citations and other sites' text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Google publishes 0 volume before the US session opens and the parser
nulls it, so a dispatch at 07:00 UTC failed on the equity row's volume.
A time-of-day value is not a parser check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jehrr
jehrr marked this pull request as ready for review October 15, 2026 13:04
@jehrr
jehrr merged commit e4a1855 into main Oct 15, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant