fix(guards): close the silent-omission gaps — crawl, extract, money, charset

[Wynelson94]

Stacked on #6 — base is fix/trafilatura-2.2-close-882, not main. Merge #6 first; this retargets to main automatically. It genuinely depends on 2.2.0 behaviour.

Every defect here fails silently, which is the class that actually costs research: the tool returns something that looks clean while the guard that should have warned could not arm.

crawl had no guards at all

It imported extract_content alone — no looks_blocked, no assess_extraction. A consent banner or bot wall was written to disk as a numbered page file, cached, and listed in INDEX.md with an ordinary token count. The failing test output said it plainly: 004-just-a-moment.md ~18tok Just a moment.

Walls are now refused — never written, never cached, links never followed — and reported as refused: <path> (<reason>) with a count in the summary. Page warnings ride in the index too. This matters more here than anywhere else: crawl output is read back off disk later instead of re-fetched, so a wall written once is a wall trusted for weeks.

extract had the same hole and wrote to the same pages.html row scrape reads, so one call could poison both verbs — while returning the wall's own structured data, usually "no structured data found", which reads as this page has none rather than we never saw the page.

Money matching had four blind spots

Each disarmed the dropped-price guard on the exact page class it exists for. Figures are now canonicalised symbol-first and space-free via money_figures(), because $ 49 and $49 are one price:

input	before	now
<span>$</span><span>49</span>	no match at all	$49
$49, in prose	$49, ≠ $49 → false drop warning	$49
19,99 € (German/French suffix)	no match at all	€19,99
$1.1 billion	$1 — silently wrong figure	$1.1

The first is ordinary pricing-grid markup; the third appears on exactly the tarif/preise pages the title-arming path deliberately targets.

Three more silent disarms

• tarif matched English "tariff"/"tariffs" — no word boundary — handing news articles the low arming floor the design reserves for pricing pages. Tariff coverage is precisely the genre that scatters real dollar figures through prose.
• Thresholds are now link-invariant. They're absolute character counts, but scrape passes the caller's include_links through and link syntax inflates markdown ~70% — enough for a failing page to go silent because the caller asked for links.
• html_to_text is charset-aware. This was pinned as a cosmetic raw limitation, which understated it: this function produces the page_text that arms the price/stub/gated guards, so a latin-1 pricing page had its £ figures replaced before counting and no guard could arm.
• fresh=True now evicts a poisoned row — replace_poisoned was only reachable through the cache-read branch fresh skips, so the caller got good content while the cache kept the wall for the rest of the TTL.

Calibration

Replaying all 194 cached pages: zero guard-verdict changes. Nothing here widened or narrowed what fires on the real corpus. Corpus distinct figures went 363 → 377, every delta verified legitimate (split-markup figures recovered, $1.1 no longer truncated, trailing-comma duplicates correctly merging).

Deliberately NOT fixed

The collapsed-column guard still has no real-page coverage. I built a replacement signal (detect the deduplication itself) and measured and rejected it — it separates the fixtures cleanly (quo 19, heyrosie 1, article_peripheral 0) then fires just as hard on ordinary federal prose:

quo.com/pricing                     19   <- the page we want
vmware.com/.../sovereign-cloud      22
acquisition.gov/far/52.227-20       19   <- a regulation page, not a table
artificialintelligenceact.eu/art/99 16
nasa.gov/sbir_sttr/phase-i          13

No threshold separates quo from a FAR page; at ≥10 it fires on 20 of 194, mostly the federal corpus this tool is used for. That trades one missed collapse for routine false warnings — the trade that destroys the value of every other warning. Same reasoning that rejected the bare yield ratio in v0.1.2. The numbers live next to _COLLAPSE_RUN_LEN so it isn't re-proposed, and a test documents the gap.

Verification

ruff · mypy · 275 tests (+23) · check_consistency.py — all green. Every fix was written test-first; the 194-page replay is the calibration guard.

🤖 Generated with Claude Code