[HALCONDEORO]

Why

An empty or unreadable PARES results page currently pauses every lane as a possible block. When the source's own control still returns enough readable results, the failure is specific to that query. It should remain unreadable, never become a false zero-result answer, without pausing unrelated work.

Changes

• On an unreadable PARES search, run the declared control once, live, through the normal gate spacing.
• If the control meets its minimum, return exit 7 (unreadable), with no result files, no cache entry for the unreadable answer and no gate hit.
• If the control is unreadable, blocked or short of its minimum, preserve the existing stop behavior. A failed control reports its own gate signal.
• Leave explicit block, captcha and sign-in detection unchanged. Successful form and control answers can still be cached; the probe bypasses cached control answers.
• Add an optional on_empty callback to the shared reader after its existing data parameter, preserving positional and keyword POST behavior.
• Update documentation, add PARES and POST compatibility tests, and add two mutation checks in read/mutants.d/pares-unreadable.txt.

Validation

Prepared independently against upstream main 7c1bc0c35846c4674abad00aa4749ebad1ee8994. This proposal does not depend on the locality proposal.

• Read tests: 460 passed, 20 subtests passed.
• All 20 Python package test runs: 2,392 passed, 43 subtests passed; both extra shell checks passed.
• Both added mutation checks fail as intended: 2 mutants, 2 killed. The full existing mutation suite was not completed locally.
• Zensical clean strict build passed; built-site check: 49 pages, 0 errors.
• git diff --check passed.

Tests use invented fixtures and fake transport. No live PARES queries were made for this validation, and no personal or family records are included. This does not bypass a challenge or retry a failed control in a loop.

No new dependencies. Adapted from the unreadable-query fix merged in HALCONDEORO#165, preserving the current upstream POST API and adding explicit compatibility coverage.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 33 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: GeiserX/gramps-web-toolkit
• Branch: main
• File: .coderabbit.yaml
• Commit: 8aaec10
• Layers: Repository YAML
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 37955a09-143a-4d32-9298-9068d08aea76

Commits

Reviewing files that changed from the base of the PR and between 8aaec10 and 7139f83.

Files ignored due to path filters: 1

• read/mutants.txt is excluded by none and included by none

Files selected for processing: 5

• CHANGELOG.md
• docs/read.md
• read/gwt_read.py
• read/gwt_reader.py
• read/test_gwt_pares.py

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[GeiserX]

Review of this PR. The problem is real (it is #237) and the shape is right: one optional on_empty after data in Reader.get, used only by the PARES search, and the canary read once through the gate. Exit 7 is free in gwt-read (0 to 6 and 64 are taken), so it stays. We are pushing a few changes to this branch.

What we change, and why:

1. A control that cannot be reached keeps the wall.

   gramps-web-toolkit/read/gwt_read.py

   Line 343 in 75027a9

   _st, _rh, body, _cached = R.get(url, h, refresh=True, page=gwt_pares.is_results)

   lets any Stop from the probe escape. A 5xx or a dropped connection is the http verdict, which tells the gate nothing, so an empty page followed by an unreachable canary went from exit 3 with a gate hit to exit 1 with no gate signal. The probe now returns False on http or unexpected, and the empty page takes its normal wall. Same for a gate that another lane paused between the search and the probe (Paused), as _fs_on_403 already does.
2. A canary never proves itself. At

   gramps-web-toolkit/read/gwt_read.py

   Line 361 in 75027a9

   on_empty=_pares_control_probe(R, h))

   the probe is passed even when the search is the canary query (--controls, or a lane searching it), so an empty canary page was read again and called "this query's own problem" with no gate hit. The probe is now skipped when the search URL is the canary URL.
3. Unreadable leaves a line in the gate log.

   gramps-web-toolkit/read/gwt_reader.py

   Line 328 in 75027a9

   if verdict == 'empty' and on_empty is not None and on_empty():

   raises with no gate call at all. It now calls gate.denied(lane, 'UNREADABLE …'), like Restricted: no pause and no gap change, but a run of unreadable answers during a throttle can be seen in the log.
4. The docs name a ledger result that exists.

   gramps-web-toolkit/docs/read.md

   Line 48 in 75027a9

   | an empty or unreadable results page on PARES whose canary answers (checked once, live, at the gate's pace) | the query's own problem, not a block: the unreadable query is neither saved nor cached (the successful form and canary answers may be), and the gate hears nothing; log the row as unreadable and carry on with the batch | 7 |

   says to log the row as unreadable, but gwt-ledger has no such result. The docs now say to record it as unproven (open, to retry or narrow), never none.
5. Mutant rows move to read/mutants.txt, where the reader-core and gwt_read.py rows live; mutants.d/ holds one file per source. We add rows for the three fixes above and for the minimum-count boundary (>= to > survived before), and the main unreadable test now uses a canary at exactly its minimum.
6. One test cut:

   gramps-web-toolkit/read/test_gwt_pares.py

   Line 236 in 75027a9

   def test_post_data_keeps_its_positional_slot_and_does_not_run_empty_control(self):

   only checks that appending a parameter after data did not move data. The POST test beside it covers the behaviour and stays.
7. A ### Fixed line under Unreleased in the CHANGELOG, since lanes learn about a new exit code there.

What we leave, and why:

• Probing only pages with PARES markup (the site's error page or form), and keeping a zero-byte body or a bare frame as a wall without a probe. We don't know which of those shapes PARES sends for the per-query failure in gwt-read pares: empty page on one query pauses the source and voids the batch #237, and narrowing could switch the fix off for the real case. A site-wide block fails the canary too, the canary goes through the same gate, and the gate log line in (3) makes a burst visible.
• EXIT_UNREADABLE stays in gwt_reader.py. Another tool uses 7 for something else, but exit codes are already per tool (5 differs too).
• The probe reads only the first declared canary. PARES has one.

[GeiserX]

Merged as 14b9af7 with the changes from the review above. Thanks for this one, the canary check is the right shape.