Repository navigation
Safer output files, honest exit codes and a canary that must test the site - #16
Merged
Merged
Conversation
… site - JSON, CSV and the sidecar are replaced atomically and keep umask/existing modes; CSV formula cells are neutralised and counted in the sidecar. - A served grid that parses to nothing ends as parser_found_nothing (exit 5/6), and Playwright/pyppeteer pass the HTTP status to the classifier. - Counting and delay flags reject impossible values in every engine. - Credential scan judges exclusions inside the repo, fails on 0 files, and reads untracked .json dumps and .env.* backups. - pyppeteer load-failure retries match the other engines. - The canary fails when no paid exit is configured or nothing was fetched. - Corrected the Scraping Browser lifetime claim, removed internal section citations, widened .gitignore for dump shapes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jehrr
marked this pull request as ready for review
October 15, 2026 21:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes from the 2026-10-14 audit of
mainat0895e83, plus the canary rule: a canary that did not test real rows must not be green.What changes for a user
write_json,write_csvandwrite_run_metawrite a temp file in the target's directory, then flush, fsync andos.replace. A failed write leaves the previous file intact and no temp file behind. A new file gets0o666 & ~umaskand an existing file keeps its mode. The implementation is the one already used in the sibling repos.= + - @ TAB CR LFget a leading', in CSV only. Numbers and JSON are untouched. The sidecar getscsv_cells_escaped, merged under the caller'sextra.classify, so a marker-less 401/403/429/5xx shell isblocked(exit 3) instead of an exit-4 "empty listing". The status is dropped after a solve replaces the document. Selenium has no status to pass, and passes an explicitNonewith a comment saying so. A page served WITH tiles that parses to 0 rows is nowparser_found_nothing, which gives exit 5 with no rows and exit 6 after earlier pages. All three engines get it through onepage_flow.failure_stop_reason. A complete empty--allow-emptyrun writesstatus: complete. The deadsingle_page_modereason is removed (F14).--pages,--concurrency(≥1),--delay,--retry-delayand--proxy-block-retries(≥0) are validated in all three engines, with an argparse error that names the flag. Inscraper_api_client.py,--retriesand--retry-delaymust be ≥0 (0 retries is legal there because that flag counts extra attempts).path.relative_to(REPO).parts. Zero files scanned is now a failure. Untracked.jsonand.env.*backups are scanned..envitself is not scanned, because it holds the user's own key.--retriesfor every block retry. Only an unusable exit with a pool rotates, which matches Playwright and Selenium..env.exampleand canary.yml. The Turnstile log no longer calls a UA mismatch the likeliest cause of a refused token. About 148 internal section-number citations and one reference to a private instruction file are removed from tracked files, including released CHANGELOG sections (citations only).test_no_private_citationsassembles its tokens from pieces so it does not match itself.test_wordingnow scans.github/, the Dockerfile,.env.exampleand the suite itself, again with assembled phrases (F15). The release notes v0.1.0, v0.2.0 and v0.2.1 contained no citations, so they were not edited.*.page[0-9]*,/*.html,live/,out/,output/,dumps/andscratch/. A suite check confirms that dumps are ignored and shipped files are not.FOODPANDA_PROXYnorFOODPANDA_CDP_ENDPOINTset, the job now FAILS with::error::naming both secrets; it used to skip green. Exit 5 is red. Exit 6 runs the data floors on the rows it gathered. Exit 3 and exit 4 were already red. It stays dispatch-only: a datacentre runner cannot reach the site without a paid exit, so a schedule only makes sense once a secret exists. This repo has no secrets configured, so a dispatch is expected to be red until one is added.Evidence
_fetch_one_pagestubs (a 403 shell is blocked, a 200 shell is parsed, a li→div fixture givesparser_found_nothing). Core + pyppeteer: 608 OK. Core + selenium: 608 OK. Each engine's own groups ran in its venv.--pages 0,--delay -1,--concurrency 0,--proxy-block-retries -1and--retries 0→ exit 2 witherror: argument --<flag>in pyppeteer and Selenium.venv/,.venv/andbuild/each scan 37 files. A planted 32-hex value in.env.bakand inlive_dump.jsonis caught; one planted in.envis not read.PYTHONDONTWRITEBYTECODE=1and the cache cleared each time. Each turned the suite red, and the restored tree is green. 17 went red on the predicted check. One (addingparser_found_nothingto the COMPLETE reasons) went red on "parser_found_nothing is not a COMPLETE stop reason" instead of the exit-5 check, because that test'spages_failed=[1]already makes the run incomplete. The faults: non-atomic write_json, flat 0o644 mode, removed CSV escape, reversed extra merge, old allow_empty status, the COMPLETE-reasons change above, absolute-path exclusions,.jsondropped from the scan,--pages type=intin Selenium, the validator minimum removed, Selenium classify withoutNone, pyppeteer classify without status, the old pyppeteer load-failure branch,parser_failedremoved in Selenium,parser_found_nothingremoved from the stop-reason helper, a planted section citation,*.page[0-9]*removed from .gitignore, and a banned phrase planted in smoke_test.py.Not changed
--fingerprintREADME claim. Testing it needs a key and a paid re-measurement, so it is left for the owner.--proxy/--concurrencywith--cdp-endpoint, which are still warnings. Making them usage errors is the owner's decision..env.example'sFOODPANDA_URLis kept. This repo has no URL-builder flags for it to hijack (--categoryis only a run label), and a typed--urlstill wins.🤖 Generated with Claude Code