Repository navigation
Atomic writes, CSV formula guard, one captcha allowance per page, Google Finance's own help text - #5
Merged
Merged
Conversation
… Google Finance's own help text - Output: JSON, CSV and the .meta.json sidecar are written to a temp file in the target's directory and renamed; a new file gets 0o666 & ~umask, an existing one keeps its mode. A failed write no longer destroys the previous good output. - CSV only: text cells starting with = + - @ TAB CR LF get a leading apostrophe; numbers and JSON untouched; csv_cells_escaped in the sidecar, under the caller's extra. - Captcha: one SolveBudget per page, spent before each purchase, at both call sites in all three engines. A failing solver was called 2-4 times against a limit of 1. - pyppeteer no longer crashes on a load failure without a proxy pool. - pyppeteer and Selenium pass --mode to the classifier (markets, movers and earnings pages were labelled empty). - A GOOGLE_FINANCE_URL from .env no longer refuses --symbols; the example ships it commented out. - Issue templates, CONTRIBUTING, diff_runs --help, --headful help, the blocked-page message and two market-run log lines described other sites; rewritten for Google Finance. Private section citations removed. - Canary: a blocked run (exit 3) is now red instead of a green skip. - Suite: stubbed-driver probes of all three engines, writer/mode/CSV checks, and a guard against citations and other sites' text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Google publishes 0 volume before the US session opens and the parser nulls it, so a dispatch at 07:00 UTC failed on the equity row's volume. A time-of-day value is not a parser check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jehrr
marked this pull request as ready for review
October 15, 2026 13:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes from the 2026-10-14 audit of this repo (base
42ed314). The live earnings breakage is in a separate PR.Problems and what happens now
out.jsoninto 13 bytes of invalid JSONos.replace; previous file intact, no.tmpleft. New files0o666 & ~umask, existing files keep their mode=HYPERLINK(...)written raw= + - @ TAB CR LFget'; numbers (e.g.-1.5) and JSON unchanged;csv_cells_escapedin the sidecar, under the caller'sextraSOLVES_PER_PAGE = 1page_flow.SolveBudgetper page, spent before the solver is called, at both call sites in all three engines: 1 call in every caseAttributeError(exit 1) on a navigation timeout--modeempty(readiness wait skipped)content, same as PlaywrightGOOGLE_FINANCE_URLfrom.env--symbols MSFT:NASDAQ→ "pass --symbols or --url, not both"--urlconflicts;.env.exampleships the URL commented outdiff_runs --help(JPY),--headfulhelp, scraper_api_client docstring, a "matches N, will serve M" log on every pyppeteer/Selenium market run[Unreleased], with a behaviour-change blockquotepartial, one dropped symbol is a warning). GOOGLvolumeis no longer required: before the US open Google publishes 0, which the parser nulls, so a 07:00 UTC dispatch failed on it — now a noticeEvidence
requirements.txt+requirements-<engine>.txt, no browser download): each run executed its own engine's group.tests/stub_engines.pydrives all three engines'_fetch_one_pageandparse_argswith stubbed drivers, run by the suite in a subprocess. Onmainit reports: pyppeteer CRASH AttributeError; 6×emptystates; solver calls 2/2/2/4;--symbolswith env URL exit 2 in all three engines. On this branch: no crash,content, 1/1/1/1, accepted.0o644 & ~umask; CSV escape disabled;extramerge order reversed; Playwright allowance spend removed; pyppeteer pool guard removed; Selenium classify without mode;typed_urlreverted; a section citation added to page_flow; another site's exit string added to README. All 10 named, restored suite green.no event carries an estimate(the earnings payload change, fixed in the separate PR).Not changed
fix/site-change-2026-10-14. Until it merges, the canary on this branch stays red on that one assertion.x-debugJSON-field redaction (L2): whether the API ever echoes a key that way is not established.scraper_api_client --retriesdouble meaning (L7): left as they are.🤖 Generated with Claude Code