fix(read): distinguish unreadable PARES queries from site-wide blocks

CHANGELOG.md

@@ -6,6 +6,10 @@ Every release has a **Change lists** heading. It names each change that makes a
 
 ## [Unreleased]
 
+### Fixed
+
+- `gwt-read pares search` no longer pauses every lane when one query gets an empty or unreadable page. It reads the source's canary once, live, through the gate; when the canary answers with at least its minimum, the search exits 7 (`unreadable`, new), saves and caches nothing for that query, and the gate only logs it. A canary that is empty, short, walled, unreachable or the search itself keeps the wall (exit 3). Record an exit 7 as `unproven`, never `none` (#237).
+
 ## [1.3.2] - 2026-11-07
 
 ### Change lists

docs/read.md

@@ -44,7 +44,8 @@ Every request goes the same way (the INSEE source makes none). A fresh answer in
 | HTTP 2xx with the JSON it expects, or a page with the record text it asked for | writes the envelope and the candidates | 0, or 2 with no results |
 | HTTP 403 | blocks the account's gate for 24 h | 3 |
 | HTTP 429, a captcha or block page | pauses every lane (20 min) and triples the gap | 3 |
-| HTTP 200 with an empty body, a page that is not JSON from a JSON site, or a page without the record text from a page site (an empty frame, an Imperva block inside an iframe, the site's own error page) | read as a block: pauses every lane | 3 |
+| HTTP 200 with an empty body, a page that is not JSON from a JSON site, or a page without the record text from a page site (an empty frame, an Imperva block inside an iframe, the site's own error page) | read as a block: pauses every lane (on PARES, only once the source's canary has failed to answer too) | 3 |
+| an empty or unreadable results page on PARES whose canary answers (checked once, live, at the gate's pace) | the query's own problem, not a block: the unreadable query is neither saved nor cached (the successful form and canary answers may be), and the gate only logs it (no pause); record the row as `unproven` in gwt-ledger, never `none`, and carry on with the batch | 7 |
 | HTTP 401, or a sign-in page | pauses every lane; get a fresh token | 4 |
 | HTTP 403 for one FamilySearch record whose access is restricted (see [restricted records](#a-403-for-one-record-restricted-or-blocked)) | writes the envelope with `access: restricted` and no candidates; the gate only logs it as a denial (no pause, no block) | 5 |
 | JSON of an unexpected shape, or another HTTP error | nothing saved, the gate is not told | 1 |
@@ -65,6 +66,8 @@ gwt read boe search --name "Ana Zorvalo Quemada" --controls both --out runs/   #
 control	control	--given Cristóbal --surname Colón	25 read, want at least 10	ok
 ```
 
+A PARES search that gets an empty page checks it once against the canary before it counts as a wall, at the gate's normal spacing. When the canary answers, the problem is that query, not the site: the search exits 7 (`unreadable`), the unreadable query is neither saved nor cached (the successful form and canary answers may be) and the gate only writes an `UNREADABLE` line to its log, so one odd query no longer pauses every lane or voids its batch. Record the row as `unproven` in gwt-ledger (open, to retry or narrow), never `none`. When the canary is empty too, reads short of its minimum, cannot be reached or meets a paused gate, the wall stands as before, and an empty page on the canary search itself is always a wall; a canary that answered short may be cached, and the probe always reads it live.
+
 `--controls before|after|both` on `search` runs the canaries around that search, and reads the search itself without the cache, since the canaries only prove the site works now. A short canary before the search means no search. A short canary after it, or one stopped by a block or a captcha, makes the run void (exit 3): the envelope and candidates files the search wrote are removed (one `removed` line each), and so are the answers it cached, so nothing on disk reads as "none" and the next run asks the site again. To bracket a batch, run `control` before the first search and after the last one: a canary costs one search, which is two requests on PARES (the search form, then the search), so do not add it to every search.
 
 | Source | Canary | Minimum rows |
@@ -331,4 +334,4 @@ cd read && python3 -m unittest -v test_gwt_read test_gwt_pares test_gwt_genae te
 
 `test_registry` checks every source against the registry's shape, the three-state page classifier on fixture pages (a skeleton page is `unknown`, a STOP), and that a restricted FamilySearch record never reaches the gate while a refused control record still blocks it. `test_controls` checks that a canary skips the cache, that a short one voids the run, and that `--controls` brackets a search.
 
-Invented people, descriptions and ids (the PARES pages in `read/fixtures/` keep the site's markup with invented content), a fake transport and a fake gate: the tests make no request to any site. They check that a search writes an envelope and candidates `gwt-rank` ranks, that a repeated search is served from the cache with no network call, that the gate is asked before every request, that a 403, 429, captcha, sign-in page or empty frame stops the run with nothing written or cached, and that the token never appears in a written file or a log line. The INSEE tests use invented fixed-width lines (`read/fixtures/insee-deces-1985.txt`, and lines built in the test). They check that the source asks for no gate, cache or network, that each line out of the layout exits 2 with nothing written, and that a value INSEE wrote out of its own rules is kept and marked.
+Invented people, descriptions and ids (the PARES pages in `read/fixtures/` keep the site's markup with invented content), a fake transport and a fake gate: the tests make no request to any site. They check that a search writes an envelope and candidates `gwt-rank` ranks, that a repeated search is served from the cache with no network call, that the gate is asked before every request, that a 403, 429, captcha, sign-in page or empty frame stops the run with nothing written or cached, that an empty PARES page its canary answers is unreadable (exit 7, only a line in the gate log) and stays a wall when the canary cannot answer either or is the search itself, and that the token never appears in a written file or a log line. The INSEE tests use invented fixed-width lines (`read/fixtures/insee-deces-1985.txt`, and lines built in the test). They check that the source asks for no gate, cache or network, that each line out of the layout exits 2 with nothing written, and that a value INSEE wrote out of its own rules is kept and marked.

read/gwt_read.py

@@ -35,7 +35,9 @@
 (HTTP 429: every lane on the account pauses; HTTP 403: the account's gate is blocked for 24 h; do other
 work; also a canary that read less than its minimum: the run is void), 4 signed out or the token
 expired (get a fresh token; never retry in a loop), 5 restricted (a FamilySearch record refused to this access
-level, not a block; the gate is not told), 6 the account's gate is paused (nothing was requested), 64 usage (no token).
+level, not a block; the gate is not told), 6 the account's gate is paused (nothing was requested), 7 unreadable
+(pares: an empty page the source's control answered: this query's own problem, not a block; nothing saved, the
+gate only logs it), 64 usage (no token).
 """
 import argparse, contextlib, io, json, os, re, shutil, sys, tempfile, urllib.error, urllib.parse, urllib.request
 
@@ -326,6 +328,33 @@ def cmd_record(a, R):
         return gwt_reader.EXIT_RESTRICTED
 
 
+def _pares_control_probe(R, url, h):
+    """The on_empty check for a pares search: the source's declared canary, read live through the gate at
+    its normal pace. True only when the canary answers with at least its minimum: the site is fine, so the empty
+    page is that query's own problem. A canary that reads short, cannot be reached (a 5xx, a dropped connection)
+    or meets a paused gate keeps the wall; one that meets a wall itself raises that Stop, with its own gate signal.
+    None when `url` is the canary: a canary never proves itself."""
+    ctl = next(s['controls'][0] for s in LEGACY if s['id'] == 'pares')
+    args = argparse.ArgumentParser()
+    _pares_search_args(args)
+    ns = args.parse_args(list(ctl['argv']))
+    probe = gwt_pares.search_url(ns.given, ns.surname, ns.words)
+    if url == probe:
+        return None
+
+    def on_empty():
+        try:
+            _st, _rh, body, _cached = R.get(probe, h, refresh=True, page=gwt_pares.is_results)
+        except gwt_reader.Stop as e:
+            if e.verdict in ('http', 'unexpected'):
+                return False    # no proof the site is fine: the empty page keeps its wall
+            raise
+        except Paused as _paused:   # another lane paused the gate: no proof either
+            return False
+        return len(gwt_pares.search_candidates(body.decode('utf-8', 'replace'))) >= ctl['min_count']
+    return on_empty
+
+
 def cmd_pares_search(a, R):
     if not (a.given or a.surname or a.words):
         err('give --given, --surname, --words or any of them')
@@ -337,7 +366,14 @@ def cmd_pares_search(a, R):
     if a.refresh or R.cache.lookup('GET', url, h) is None:
         R.get(gwt_pares.FORM_URL, h, refresh=True, page=gwt_pares.is_form)
     key = gwt_reader.search_key(url, ['pares', a.given] + list(a.surname) + [a.words])
-    return _save_page(a, R, url, h, gwt_pares.is_results, key, gwt_pares.search_candidates, gwt_pares.total)
+    try:
+        return _save_page(a, R, url, h, gwt_pares.is_results, key, gwt_pares.search_candidates, gwt_pares.total,
+                          on_empty=_pares_control_probe(R, url, h))
+    except gwt_reader.Unreadable as e:
+        err(str(e))
+        out("results\tunreadable: the page came back empty, but the source's control answered; this query, "
+            "not the site (nothing saved, no gate hit; log it as unproven)")
+        return gwt_reader.EXIT_UNREADABLE
 
 
 def cmd_pares_record(a, R):
@@ -497,8 +533,8 @@ def cmd_insee_search(a, R=None):
     return EXIT_OK if cands else EXIT_EMPTY
 
 
-def _save_page(a, R, url, h, page_ok, key, to_cands, total_of):
-    status, rh, body, cached = R.get(url, h, refresh=a.refresh, page=page_ok)
+def _save_page(a, R, url, h, page_ok, key, to_cands, total_of, on_empty=None):
+    status, rh, body, cached = R.get(url, h, refresh=a.refresh, page=page_ok, on_empty=on_empty)
     page = body.decode('utf-8', 'replace')
     cands = to_cands(page)
     env = gwt_reader.envelope(url, (rh or {}).get(FINAL_URL_HEADER), text=gwt_pares.text(page), html=page,

read/gwt_reader.py

@@ -24,6 +24,7 @@
 import datetime, hashlib, json, os, re, struct, unicodedata
 
 EXIT_OK, EXIT_ERROR, EXIT_EMPTY, EXIT_WALL, EXIT_DEAD, EXIT_PAUSED = 0, 1, 2, 3, 4, 6
+EXIT_UNREADABLE = 7    # a query-specific unreadable answer whose live control succeeds
 EXIT_RESTRICTED = 5      # the site refuses this one record to every account of this access level: a result, not a block
 
 # ---------------------------------------------------------------- small helpers
@@ -280,6 +281,13 @@ def __init__(self, msg, status=403, body=b''):
         self.status, self.body = status, body
 
 
+class Unreadable(Stop):
+    """An unreadable query with a successful live control: a result, not a gate hit."""
+
+    def __init__(self, msg):
+        super().__init__(msg, EXIT_UNREADABLE, verdict='unreadable')
+
+
 class Reader:
     """GET through gwt-fscache and gwt-gate. fetcher(method, url, headers, data) -> (status, headers, body)
     is gwt-fscache's fetcher signature; the token belongs inside it, never in `headers`, which the cache sees."""
@@ -290,12 +298,14 @@ def __init__(self, cache, gate, lane, fetcher, log=None):
         self.cache, self.gate, self.lane, self.fetcher = cache, gate, lane, fetcher
         self.log = log or (lambda m: None)
 
-    def get(self, url, headers=None, expect=None, refresh=False, page=None, on_403=None, data=None):
+    def get(self, url, headers=None, expect=None, refresh=False, page=None, on_403=None, data=None, on_empty=None):
         """(status, headers, body, from_cache) for a 2xx answer the site really gave; raises Stop otherwise.
         `expect` checks a JSON answer, `page` an HTML one (see classify_answer). `on_403(body)` is asked about a
         403 before the gate blocks: 'restricted' only logs it as a denial (gate.denied: no pause, no block) and raises
         Restricted; anything else is a block, as without it. `data` (bytes) makes it a POST, for a site whose search
-        is a form or a JSON body; the cache keeps a POST only when it was built with methods=('GET', 'POST')."""
+        is a form or a JSON body; the cache keeps a POST only when it was built with methods=('GET', 'POST').
+        `on_empty()` is asked about an empty answer before the gate hears it: true (a live source control answered)
+        only logs it to the gate (gate.denied: no pause) and raises Unreadable; false or absent keeps the wall."""
         state = {}
 
         def gated(method, u, h, data):
@@ -314,6 +324,11 @@ def gated(method, u, h, data):
             raise Restricted('restricted: HTTP 403 at %s is about this record\'s access, not a block; no gate hit' % url,
                              state['status'], state['body'])
         if verdict != 'ok':
+            if verdict == 'empty' and on_empty is not None and on_empty():
+                self.gate.denied(self.lane, 'UNREADABLE HTTP %s %s' % (state['status'], url[:120]))
+                raise Unreadable(f"empty or unreadable page, but the source's control answered: this query's own "
+                                 f"problem, not a block; nothing saved, no gate hit (HTTP {state['status']} "
+                                 f"at {url[:120]})")
             code, why, tell = STOP_FOR[verdict]
             what = 'HTTP %s %s' % (state['status'], url[:120])
             if tell == 'block':

read/mutants.txt

@@ -71,3 +71,9 @@ gwt_read.py | return v if isinstance(v, str) and v else None | return v | place_
 gwt_reader.py | if not (isinstance(c, dict) and isinstance(c.get('argv'), list) and c['argv'] | if not (isinstance(c, dict) and isinstance(c.get('argv'), list) | a canary must name at least one search option: an empty search is not a canary
 gwt_read.py | if not isinstance(t, str) or not t: | if not t: | place_orig keeps only text, never an object a field holds
 gwt_read.py | R.cache.store = store | pass | a void forgets only the answers its search cached, never the canaries' own
+gwt_reader.py | if verdict == 'empty' and on_empty is not None and on_empty(): | if False: | an empty pares page its own control answers is unreadable (exit 7, no gate hit), never a wall
+gwt_reader.py | self.gate.denied(self.lane, 'UNREADABLE HTTP %s %s' % (state['status'], url[:120])) | pass | an unreadable pares page leaves a line in the gate log, so a run of them can be seen
+gwt_read.py | return len(gwt_pares.search_candidates(body.decode('utf-8', 'replace'))) >= ctl['min_count'] | return True | a control that reads fewer rows than its minimum cannot clear an empty page: the wall stands
+gwt_read.py | return len(gwt_pares.search_candidates(body.decode('utf-8', 'replace'))) >= ctl['min_count'] | return len(gwt_pares.search_candidates(body.decode('utf-8', 'replace'))) > ctl['min_count'] | a control that reads exactly its minimum clears an empty page
+gwt_read.py | return False    # no proof the site is fine: the empty page keeps its wall | return True | a control that cannot be reached (a 5xx, a dropped connection) keeps the empty page's wall
+gwt_read.py | if url == probe: | if False: | an empty canary page is a wall: a canary never proves itself