Repository navigation
Refuse --retries 0, unify the not-served message, fix donor --help text - #14
Merged
Merged
Conversation
--retries 0 made the load loop run zero times, so goto was never called and the engine judged about:blank (39 bytes) as a block: exit 3 for a site that was never asked. Now a usage error in all three engines. The not-served log blamed headless and recommended a residential exit in the same sentence, and said so for headful runs too. One shared page_flow.not_served_message states the observation first and the cause only when it applies. --help and usage errors described another marketplace scraper (product mode, Apollo, IDR, /p/<cat>). Now from one set of strings in page_flow, with a suite check over every string in the engines. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…let diff_runs read lot runs Comments, docstrings, CONTRIBUTING and a few help strings still described a different marketplace (its selectors, ids, page sizes, modes and currency), and some of that was code, not prose: - diff_runs refused every --mode lot run: the diffable-mode list named a mode this repo does not have. A test pinned that list, so it defended the bug. - PageOutcome carried scroll and shop_facts fields no engine ever set, and the sidecar logic around them could never fire. Removed (the always-empty scroll and pages_still_growing sidecar keys go with them). - output_writer's Product comments now describe this site's real columns. - page_url, the completeness check and the blocked-retry comments no longer state another site's measurements, one of which contradicted the parser (it said a search cannot be paginated by URL). No column or flag changed. Suite: 499 checks; planted faults for the mode list and the removed fields each turned it red on the intended check; a live --mode lot run pair now diffs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Worked through a third-party audit of 443a052. Each claim was checked against the code and a live run.
What the audit got wrong, and what was real under it
Its live run used
--retries 0. The load loop isrange(1, retries + 1), so it ran zero times,gotonever happened, and the engine classifiedabout:blank(39 bytes) as blocked. Measured 2026-10-09 from a datacentre-class Linux host under xvfb:--retries 0(headful and headless)--retries 1, 3 pagesSo the README's headful finding reproduces; the defect is that 0 attempts is reported as a block.
Changes
--retriesmust be >= 1 (all three engines, usage error exit 2)page_flow.not_served_message: shared, states observation first; no longer blames headless on a headful run or recommends a proxy beside it--help/ usage errors rewritten for Catawiki from shared strings (URL_HELP,MODE_HELP,PAGES_HELP,RETRIES_HELP); dead discovery-hub warning removedSECURITY.md: a dump is page markup, not the cookie jarn < 0, empty-DOM threshold zeroed, stale word in a help string).Not changed, deliberately
--localedefault is stillid-ID(donor leftover; only the help text was corrected, behaviour untouched).scraper_api_client.pycomputes attempts asretries + 1, unlike the engines.🤖 Generated with Claude Code