fix(guards): close the silent-omission gaps — crawl, extract, money, charset

CHANGELOG.md

@@ -37,13 +37,68 @@ failure. For the trust model and the evaluation harness behind those calls, see
   satisfies, so a 92% body collapse passed CI unnoticed. It now pins real body sentences, total
   guard silence, and that silence comes from the cluster floor rather than a toothless page.
 
+### Added
+- **`crawl` now runs the same guards as `scrape`.** It previously imported `extract_content`
+  alone — no `looks_blocked`, no `assess_extraction` — so a consent banner or bot wall was
+  written to disk as a numbered page file, cached, and listed in `INDEX.md` with an ordinary
+  token count. Walls are now refused: never written, never cached, their links never followed,
+  and reported as `refused: <path> (<reason>)` with a `refused: N` count in the summary. Page
+  warnings ride in the index too. This mattered most on `crawl` precisely because its output is
+  read back off disk later rather than re-fetched — a wall written once is a wall trusted for weeks.
+- **`extract` refuses bot walls** instead of caching them. It wrote to the same `pages.html` row
+  `scrape` reads, so one `extract` call could poison both verbs, and it returned the wall's own
+  structured data — usually `"no structured data found"`, which reads as *this page has none*
+  rather than *we never saw the page*.
+
+### Fixed
+- **Money matching had four blind spots**, each of which silently disarmed the dropped-price
+  guard on the page class it exists for. Figures are now canonicalised (symbol-first,
+  space-free) via `money_figures()`, because `$ 49` and `$49` are the same price:
+  - `<span>$</span><span>49</span>` — ordinary pricing-grid markup. Tag stripping inserts a
+    space and the pattern demanded a digit immediately after the symbol, so it matched **nothing**.
+  - `$49,` in prose kept the trailing comma, so the same figure compared unequal against its
+    table rendering and the guard reported a **false** drop.
+  - `19,99 €` — the German/French suffix form, on exactly the `tarif`/`preise` pages the
+    title-arming path targets — never matched at all.
+  - `$1.1 billion` matched as `$1`, silently changing the figure.
+- **`tarif` matched English "tariff"/"tariffs"** (no word boundary), handing news articles the
+  low arming floor the design explicitly reserves for pricing pages — and tariff coverage is
+  precisely the genre that scatters real dollar figures through prose.
+- **Guard thresholds are now link-invariant.** `_STUB_MAX_MARKDOWN`, `_BLOCK_MAX_CHARS` and
+  `_CONSENT_MAX_CHARS` are absolute character counts while `scrape` passes the caller's
+  `include_links` straight through; link syntax inflates markdown by ~70%, so the same failing
+  page went silent purely because the caller asked for links.
+- **`html_to_text` is charset-aware** (declared charset, then cp1252) instead of utf-8-only.
+  This was pinned as a cosmetic `raw` limitation, which understated it: this function produces
+  the `page_text` that ARMS the price, stub and gated guards, so a latin-1 pricing page had its
+  `£` figures replaced before counting and **no guard could arm**.
+- **`fresh=True` now evicts a poisoned row.** `replace_poisoned` was only reachable through the
+  cache-read branch that `fresh=True` skips, so the caller got correct content while the cache
+  kept the wall for the rest of the 7-day TTL and the next ordinary scrape replayed it.
+- The raw-body wall check now covers every content type the extractor accepts
+  (`application/xhtml+xml`, empty) rather than `text/*` alone — defense in depth; the
+  post-extraction backstop already caught the reachable cases.
+
+Calibration: replaying all 194 cached pages produced **zero guard-verdict changes**, so none of
+the above widened or narrowed what fires on the real corpus. Corpus-wide distinct figures went
+363 → 377, every delta verified legitimate (recovered split-markup figures, `$1.1` no longer
+truncated to `$1`, and trailing-comma duplicates correctly merging).
+
 ### Known issues
 - **The collapsed-column guard has no real-page coverage under 2.2.0.** Quo is just as lossy
   (still 4 of 24 figures, 1 of 3 plan names) but emits the repeated cell once instead of three
   times, so the repetition signature the guard keys on is gone — on quo and on every other page
   in the cache replay. The dropped-price guard still catches quo, so the page is not silent.
   The quo pin was narrowed deliberately and the substantive loss pinned separately rather than
-  the assertion being deleted quietly.
+  the assertion being deleted quietly. A replacement signal — a short cell value repeated inside
+  a small window of visible text but present fewer times in the markdown, i.e. detecting the
+  deduplication itself — was built and **measured and rejected**: it separates the fixtures
+  cleanly (quo 19, heyrosie 1, article_peripheral 0) and then fires just as hard on ordinary
+  federal prose (`acquisition.gov/far/52.227-20` 19, `artificialintelligenceact.eu` 16, NASA
+  SBIR 13). No threshold separates quo from a FAR page. Shipping it would trade one missed
+  collapse for routine false warnings — the trade that destroys the value of every other
+  warning. Same reasoning that rejected the bare yield ratio in v0.1.2; the numbers are recorded
+  next to `_COLLAPSE_RUN_LEN` so it is not re-proposed.
 
 ## [0.1.5] — 2026-07-24
 

README.md

@@ -11,7 +11,7 @@ no telemetry.
 
 > *tearsheet (n.): a page torn from a publication and filed as proof it ran.*
 
-**Trust status:** qualified for heavy usage 2026-07-16 — 252 tests, a falsifiable live
+**Trust status:** qualified for heavy usage 2026-07-16 — 275 tests, a falsifiable live
 eval harness (verdict GREEN), and zero fabrications across the tool's entire recorded
 history. Its documented failure mode is *omission*, and the guards exist to make every
 omission loud. See [Trust](#trust).
@@ -129,6 +129,10 @@ a research tool for reading the public web — not for evading paywalls or bot d
   reported and never cached, and previously poisoned cache rows are evicted, not replayed.
   Note `--raw` deliberately uses the plain fetch, not the browser: a rendered DOM can be *worse*
   (on smith.ai the consent overlay replaced the pricing table the raw fetch still carried).
+  Since 2026-08-11 the money matcher also handles currency split from its digits by markup
+  (`<span>$</span><span>49</span>`), the `19,99 €` suffix form, and non-UTF-8 pages — each of
+  which previously left the guard unable to arm at all — and `crawl` and `extract` run the same
+  wall guards as `scrape` rather than saving a banner to disk as a page.
 - **Listings and rosters can come back as a boilerplate stub.** A page with no prices, no
   table, and no wall gave the earlier guards nothing to arm on, so a directory page could
   return its marketing sentence and silently drop every row. Measured 2026-07-24 across the
@@ -165,7 +169,7 @@ a research tool for reading the public web — not for evading paywalls or bot d
 
 "Can it be trusted for heavy usage?" is a measurement here, not a feeling.
 
-- **Offline suite (252 tests, runs in the gate)**: guard boundary pins, cache-poisoning
+- **Offline suite (275 tests, runs in the gate)**: guard boundary pins, cache-poisoning
   regressions, truncation honesty, charset torture, structure torture, adversarial
   robustness — enforced fully offline by a loopback-only socket guard. The five REAL
   pages that defined the tool's probation (quo, smith.ai, dialpad, heyrosie, a LinkedIn
@@ -199,7 +203,7 @@ for figures you'll quote, treat a suspiciously small extraction of a rich page a
 .venv/bin/ruff check src tests && .venv/bin/mypy && .venv/bin/python -m pytest
 ```
 
-TDD throughout; the default suite (252 tests) runs entirely offline — `httpx.MockTransport`,
+TDD throughout; the default suite (275 tests) runs entirely offline — `httpx.MockTransport`,
 fixture HTML, and a conftest socket guard that fails any test reaching for a non-loopback
 address. Extras: `pytest -m playwright` (real chromium, local server), `pytest -m live`
 (real network). The live trust evaluation lives in `evals/` (see [Trust](#trust)).

src/tearsheet/content.py

@@ -8,7 +8,30 @@
 _SCRIPT_STYLE = re.compile(r"<(script|style)\b.*?</\1>", re.S | re.I)
 _TAG = re.compile(r"<[^>]+>")
 _WHITESPACE = re.compile(r"\s+")
-_MONEY = re.compile(r"[$€£][0-9][0-9,]*(?:\.[0-9]{2})?")
+# Money matching. Four blind spots were measured out of this on 2026-08-11; each one
+# defeated the dropped-price guard SILENTLY on the page class it exists for:
+#
+#   1. `<span>$</span><span>49</span>` — ordinary pricing-grid markup. Tag stripping puts
+#      a space between symbol and digits, and the old pattern demanded a digit
+#      immediately after the symbol, so it matched NOTHING on such a page. Hence `\s?`.
+#   2. `$49,` in prose — `[0-9][0-9,]*` swallowed the trailing comma, so the same figure
+#      compared unequal against its table rendering and the guard reported a FALSE drop.
+#      The amount now has to end in a digit.
+#   3. `19,99 €` — the standard German/French suffix form, on pages _PRICING_TITLE
+#      deliberately targets (`tarif`, `preise`). Never matched at all. Hence the second
+#      alternative.
+#   4. Comparison requires a canonical form: `$ 49` on the page and `$49` in the markdown
+#      are the same price. Use money_figures()/_money_hits(), never _MONEY.findall —
+#      the raw match text is not comparable across renderings.
+#
+# The amount body allows `.` and `,` internally but must start AND end with a digit, which
+# covers `49`, `1,200`, `10,500.00`, `19,99` and `1.234,56` without eating punctuation.
+_AMOUNT = r"[0-9](?:[0-9.,]*[0-9])?"
+_MONEY = re.compile(
+    rf"(?P<sym>[$€£])\s?(?P<amt>{_AMOUNT})|(?P<amt2>{_AMOUNT})\s?(?P<sym2>[€£])"
+)
+# Markdown link syntax, stripped before any absolute character threshold is applied.
+_MD_LINK = re.compile(r"\[([^\]]*)\]\([^)]*\)")
 
 # Strong phrases + a size guard, mirroring fetch.looks_blocked: a page that merely
 # DISCUSSES cookie consent is long; a page that IS a consent wall is short.
@@ -65,7 +88,12 @@
 # window — because plan cards diluted by marketing prose can spread real figures more
 # than 1,500 chars apart. Articles never get this path (the LinkedIn FP protection):
 # their titles don't say "pricing".
-_PRICING_TITLE = re.compile(r"pricing|plans? and pricing|tarif|preise", re.I)
+# `tarif` carried no word boundary, so English "tariff"/"tariffs" matched it and handed
+# NEWS ARTICLES the low floor the comment above promises they never get — and tariff
+# coverage is exactly the genre that scatters real dollar figures through prose. `tarifs?\b`
+# keeps the French singular/plural and rejects the double-f English word. (`plans? and
+# pricing` was also dropped: anything matching it already matched the leading `pricing`.)
+_PRICING_TITLE = re.compile(r"pricing|tarifs?\b|tarification|preise", re.I)
 _MONEY_MIN_TITLED = 3
 
 # Stub-extraction guard (the roster/directory class, 2026-07-24 audit). A content-bearing
@@ -114,6 +142,30 @@
 
 # A collapsed table column reads as a run of identical consecutive rows
 # (quo: `Unlimited* / Unlimited* / Unlimited*`). One run happens naturally; two is a pattern.
+#
+# ⚠ KNOWN GAP since trafilatura 2.2.0 (2026-08-11): this guard has NO real-page coverage.
+# 2.2.0 deduplicates identical consecutive blocks, so quo — the page this was built on —
+# now emits that cell ONCE. The page is just as lossy (still 4 of 24 figures, 1 of 3 plan
+# names); only the signature is gone. A full-cache replay confirms the guard fires on no
+# page it used to catch. The dropped-price guard still catches quo, so it is not silent.
+#
+# REJECTED REPLACEMENT — measured 2026-08-11, do not re-propose without new evidence:
+# "a short cell value repeated >=3x within a 300-char window of visible text but present
+# fewer times in the markdown" (i.e. detect the dedup itself). It separates the FIXTURES
+# beautifully — quo 19 hits, heyrosie 1, article_peripheral 0 — and then falls apart on
+# the real corpus, because ordinary prose pages repeat short strings in nav and headings:
+#
+#     quo.com/pricing                      19   <- the page we want
+#     vmware.com/.../sovereign-cloud       22
+#     acquisition.gov/far/52.227-20        19   <- a regulation page, not a table
+#     artificialintelligenceact.eu/art/99  16
+#     nasa.gov/sbir_sttr/phase-i           13
+#
+# No threshold separates quo from the FAR pages, and at >=10 it would fire on 20 of 194
+# pages — mostly the federal prose corpus this tool is actually used for. Shipping it
+# would trade a missed collapse for routine false warnings, which is the trade that
+# destroys the value of every OTHER warning. Same reasoning that rejected the bare yield
+# ratio in v0.1.2. (It also costs ~63ms/page, versus microseconds for the run scan.)
 _COLLAPSE_RUN_LEN = 3
 _COLLAPSE_MIN_RUNS = 2
 _COLLAPSE_MAX_LINE = 80
@@ -135,6 +187,37 @@ class ExtractionQuality:
     warnings: list[str] = field(default_factory=list)
 
 
+_META_CHARSET = re.compile(rb"""<meta[^>]+charset=["']?\s*([a-zA-Z0-9_-]+)""", re.I)
+
+
+def _decode(html: bytes) -> str:
+    """Decode page bytes, honouring a declared charset.
+
+    This used to be a bare `decode("utf-8", errors="replace")`, which quietly diverged
+    from extract_content — that hands trafilatura raw BYTES precisely so its charset
+    detection runs. The two sides disagreeing is not just a `raw` cosmetic issue: this
+    function produces the `page_text` that ARMS the price, stub and gated guards, so a
+    latin-1 pricing page had every `£` replaced before counting, `on_page` came back
+    empty, and no guard could arm on a page whose figures were all right there.
+
+    utf-8 first because almost every page is utf-8 and this runs on every scrape; the
+    detection path costs nothing until a page actually fails to decode.
+    """
+    try:
+        return html.decode("utf-8")
+    except UnicodeDecodeError:
+        pass
+    declared = _META_CHARSET.search(html[:4096])
+    if declared:
+        try:
+            return html.decode(declared.group(1).decode("ascii", "ignore"), errors="replace")
+        except LookupError:
+            pass
+    # cp1252 over latin-1: real-world legacy Western pages use the Windows superset
+    # (curly quotes, en dashes) and it decodes the latin-1 range identically.
+    return html.decode("cp1252", errors="replace")
+
+
 def html_to_text(html: bytes) -> str:
     """Visible text of a page: scripts/styles gone, tags dropped, entities decoded.
 
@@ -143,14 +226,34 @@ def html_to_text(html: bytes) -> str:
     """
     if not html:
         return ""
-    text = _SCRIPT_STYLE.sub(" ", html.decode("utf-8", errors="replace"))
+    text = _SCRIPT_STYLE.sub(" ", _decode(html))
     text = html_module.unescape(_TAG.sub(" ", text))
     return _WHITESPACE.sub(" ", text).strip()
 
 
+def _money_hits(text: str) -> list[tuple[int, str]]:
+    """(position, canonical figure) for every money amount, symbol-first and space-free.
+
+    Canonicalising here is what makes page-vs-extraction comparison meaningful: `$ 49`
+    from split markup, `$49` from prose and `49 €` from a European grid all have to
+    reduce to one comparable token.
+    """
+    hits = []
+    for match in _MONEY.finditer(text):
+        symbol = match.group("sym") or match.group("sym2")
+        amount = match.group("amt") if match.group("amt") is not None else match.group("amt2")
+        hits.append((match.start(), f"{symbol}{amount}"))
+    return hits
+
+
+def money_figures(text: str) -> set[str]:
+    """The distinct canonical money figures in a string."""
+    return {figure for _, figure in _money_hits(text)}
+
+
 def _has_price_cluster(page_text: str) -> bool:
     """True when some 1,500-char window of visible text holds >= 4 distinct figures."""
-    hits = [(m.start(), m.group()) for m in _MONEY.finditer(page_text)]
+    hits = _money_hits(page_text)
     if len(hits) < _MONEY_MIN_CLUSTERED:
         return False
     left = 0
@@ -195,13 +298,19 @@ def assess_extraction(html: bytes, extracted: ExtractedContent | None) -> Extrac
         return quality
     markdown = extracted.markdown
     page_text = html_to_text(html)
-    lowered = markdown.lower()
-
-    if len(markdown) < _CONSENT_MAX_CHARS and any(p in lowered for p in _CONSENT_PHRASES):
+    # Every absolute character threshold below is measured against link-stripped markdown.
+    # scrape passes the caller's include_links straight through, and link syntax inflates
+    # markdown by ~70% — enough for a stub to clear _STUB_MAX_MARKDOWN or a banner to clear
+    # _CONSENT_MAX_CHARS purely because the caller asked for links. The thresholds are
+    # calibrated on prose length, so they have to see prose length.
+    measured = _MD_LINK.sub(r"\1", markdown)
+    lowered = measured.lower()
+
+    if len(measured) < _CONSENT_MAX_CHARS and any(p in lowered for p in _CONSENT_PHRASES):
         quality.consent_wall = True
         return quality
 
-    if len(markdown) < _BLOCK_MAX_CHARS and any(p in lowered for p in _BLOCK_PHRASES):
+    if len(measured) < _BLOCK_MAX_CHARS and any(p in lowered for p in _BLOCK_PHRASES):
         quality.block_wall = True
         return quality
 
@@ -214,23 +323,23 @@ def assess_extraction(html: bytes, extracted: ExtractedContent | None) -> Extrac
 
     if (
         len(page_text) >= _STUB_MIN_PAGE_TEXT
-        and len(markdown) < _STUB_MAX_MARKDOWN
-        and len(markdown) / len(page_text) < _STUB_MAX_RATIO
+        and len(measured) < _STUB_MAX_MARKDOWN
+        and len(measured) / len(page_text) < _STUB_MAX_RATIO
     ):
         quality.warnings.append(
-            f"extraction is a {len(markdown)}-char stub of a page holding ~{len(page_text)} "
+            f"extraction is a {len(measured)}-char stub of a page holding ~{len(page_text)} "
             "chars of visible text — names, rows, or listings on this page are probably "
             "missing entirely. Use raw=true, or fetch independently."
         )
 
-    on_page = set(_MONEY.findall(page_text))
+    on_page = money_figures(page_text)
     title_armed = (
         extracted.title is not None
         and _PRICING_TITLE.search(extracted.title) is not None
         and len(on_page) >= _MONEY_MIN_TITLED
     )
     if title_armed or _has_price_cluster(page_text):
-        kept = on_page & set(_MONEY.findall(markdown))
+        kept = on_page & money_figures(markdown)
         if len(kept) / len(on_page) < _MONEY_MIN_RETAINED:
             quality.warnings.append(
                 f"extraction dropped {len(on_page) - len(kept)} of {len(on_page)} distinct prices "