deps: require trafilatura >=2.2 — the #882 canary fired, close the loop

CHANGELOG.md

@@ -11,6 +11,40 @@ failure. For the trust model and the evaluation harness behind those calls, see
 
 ## [Unreleased]
 
+### Changed
+- **Requires trafilatura >= 2.2**, which fixes the nested-emphasis serializer bug tearsheet
+  reported as [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882)
+  (released 2026-07-31, closed 2026-07-31). The `#882` canary test — which pinned the BROKEN
+  behavior on purpose so an upgrade would fail loudly — fired exactly as designed and is now
+  flipped to pin the corrected serialization as a permanent regression guard.
+- **Adoption was measured, not assumed.** Replaying all 194 cached pages under both versions:
+  77 extract more, 42 less, 75 identical, corpus total **+2.89%**, zero catastrophic losses.
+  2.2.0 repairs three of the five silent-omission failures the stub guard was built on —
+  `sbir.gov/awards` (421 → 3,155 chars; it previously returned only the "official website of
+  the United States government" banner with no award data), `atlas.nomic.ai/pricing`
+  (259 → 1,661) and the NOAA SBIR archive. The stub guard now fires on 2 of 194 pages
+  (was 5 of 162) because the corpus got better, not because the guard got weaker.
+- **Stub-guard calibration table recalibrated** against 2.2.0. Thresholds unchanged; the two
+  remaining fires still bracket the 10% floor tightly (8.1% / 9.7%). The old "do not raise
+  past ~570 (eur-lex)" lower bound no longer binds — eur-lex now extracts 6,167 chars.
+
+### Fixed
+- **`article_peripheral` probation fixture regenerated** (`scripts/gen_article_peripheral_fixture.py`
+  is now the reproducible source). It padded its article body with 14 byte-identical `<p>` tags;
+  2.2.0 deduplicates identical blocks, collapsing the body 1,973 → 166 chars and tripping the
+  stub guard — while the real page it models moved the opposite way (+642%). Its tests were also
+  strengthened: the old body assertion checked only `"Join Forces"`, which the page *title*
+  satisfies, so a 92% body collapse passed CI unnoticed. It now pins real body sentences, total
+  guard silence, and that silence comes from the cluster floor rather than a toothless page.
+
+### Known issues
+- **The collapsed-column guard has no real-page coverage under 2.2.0.** Quo is just as lossy
+  (still 4 of 24 figures, 1 of 3 plan names) but emits the repeated cell once instead of three
+  times, so the repetition signature the guard keys on is gone — on quo and on every other page
+  in the cache replay. The dropped-price guard still catches quo, so the page is not silent.
+  The quo pin was narrowed deliberately and the substantive loss pinned separately rather than
+  the assertion being deleted quietly.
+
 ## [0.1.5] — 2026-07-24
 
 ### Added

README.md

@@ -11,7 +11,7 @@ no telemetry.
 
 > *tearsheet (n.): a page torn from a publication and filed as proof it ran.*
 
-**Trust status:** qualified for heavy usage 2026-07-16 — 247 tests, a falsifiable live
+**Trust status:** qualified for heavy usage 2026-07-16 — 252 tests, a falsifiable live
 eval harness (verdict GREEN), and zero fabrications across the tool's entire recorded
 history. Its documented failure mode is *omission*, and the guards exist to make every
 omission loud. See [Trust](#trust).
@@ -111,7 +111,10 @@ a research tool for reading the public web — not for evading paywalls or bot d
   clean while the figures never made it. Measured 2026-07-14 on real pages — `quo.com/pricing`
   kept 4 of 24 prices and flattened a 3-column matrix into repeated rows (`Unlimited* /
   Unlimited* / Unlimited*`), while `heyrosie.com/pricing` came through perfectly. So the failure
-  is site-shaped, not universal, and you cannot eyeball it from the output alone.
+  is site-shaped, not universal, and you cannot eyeball it from the output alone. Quo still
+  loses the same 20 of 24 figures under trafilatura 2.2.0, but emits that repeated cell once
+  instead of three times, so the **collapsed-column** guard no longer fires on it — the
+  dropped-price guard is what catches the page today.
   Since v0.1.2 the tool says so itself: a `warning:` line reports dropped prices and collapsed
   columns, and a cookie banner is reported as a `consent/cookie wall` instead of being served as
   the page. **Never quote figures from a warned extraction — re-run with `--raw` / `raw=true`.**
@@ -143,12 +146,13 @@ a research tool for reading the public web — not for evading paywalls or bot d
   below every guard's floor by design (a one-figure floor would warn on every blog footer).
   The rule is procedural: any figure you are going to quote gets `--raw` or independent
   verification.
-- **Emphasis mangling (upstream)**: trafilatura 2.1.0's markdown serializer displaces
-  nested-emphasis words (`<strong><em>word</em></strong>` mid-sentence) onto the next
-  paragraph and can drop characters around inline `<em>` (observed: "i.e." → "e.").
-  Facts survive; verbatim quotes should be re-verified against the source before reuse.
-  Pinned by `tests/test_content.py::TestKnownUpstreamManglingDocumented` (fails when
-  upstream fixes it). Reported: [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882).
+- **Emphasis mangling (upstream) — FIXED, now a regression pin.** trafilatura ≤ 2.1.0
+  displaced nested-emphasis words (`<strong><em>word</em></strong>` mid-sentence) onto the
+  next paragraph and dropped characters around inline `<em>` ("i.e." → "e."). Reported as
+  [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882), fixed in
+  trafilatura 2.2.0, verified downstream 2026-08-11 — tearsheet now requires `>=2.2`.
+  `tests/test_content.py::TestNestedEmphasisSerializedInPlace` pinned the broken behavior
+  as a canary, failed on cue when the fix landed, and now pins the correct serialization.
 - Bot-walled sites (eCFR, DoD, Cloudflare in strict mode) are reported as
   `blocked by bot protection …` — deliberately not evaded; use the site's official API.
 - Wikipedia extractions can include maintenance-hatnote table noise ("This article
@@ -161,7 +165,7 @@ a research tool for reading the public web — not for evading paywalls or bot d
 
 "Can it be trusted for heavy usage?" is a measurement here, not a feeling.
 
-- **Offline suite (247 tests, runs in the gate)**: guard boundary pins, cache-poisoning
+- **Offline suite (252 tests, runs in the gate)**: guard boundary pins, cache-poisoning
   regressions, truncation honesty, charset torture, structure torture, adversarial
   robustness — enforced fully offline by a loopback-only socket guard. The five REAL
   pages that defined the tool's probation (quo, smith.ai, dialpad, heyrosie, a LinkedIn
@@ -195,7 +199,7 @@ for figures you'll quote, treat a suspiciously small extraction of a rich page a
 .venv/bin/ruff check src tests && .venv/bin/mypy && .venv/bin/python -m pytest
 ```
 
-TDD throughout; the default suite (247 tests) runs entirely offline — `httpx.MockTransport`,
+TDD throughout; the default suite (252 tests) runs entirely offline — `httpx.MockTransport`,
 fixture HTML, and a conftest socket guard that fails any test reaching for a non-loopback
 address. Extras: `pytest -m playwright` (real chromium, local server), `pytest -m live`
 (real network). The live trust evaluation lives in `evals/` (see [Trust](#trust)).

evals/reports/2026-08-11/report.md

@@ -0,0 +1,73 @@
+# Tearsheet Trust Report — 2026-08-11 13:13
+
+**VERDICT: GREEN**
+
+- tearsheet 0.1.5 · trafilatura 2.2.0 · httpx 0.28.1 · lxml 6.1.1 · python 3.14.2
+- corpus 2026-07-16.2 (sha fe68ddd7bb2c) · 40 items scored · burst: 45 scrapes across 3 domains: {'ok': 40, 'honest_refusal': 5, 'FAIL': 0, 'exception': 0}
+
+## Gates
+
+| gate | status | detail |
+|---|---|---|
+| fabrication (0 tolerated) | PASS | 0 pages fabricated figures |
+| guard calibration (0 regressions) | PASS | known-bad warns, known-good silent |
+| silent omission (<=1) | PASS | 1 pages omitted without warning |
+| minor omissions within retention floor (<=3) | PASS | tailscale: ['$5'] missing (80% retained — within the document |
+| unhandled exceptions (0) | PASS | 0 |
+| cache poison (0) | PASS | none served |
+| timeouts (<=2) | PASS | 0 |
+| baseline drift (0) | PASS | stable |
+| hard failures (<=1 beyond the above) | PASS | wiki_idaho: SILENT omission: ['$74,900'] |
+
+## Items
+
+| id | category | status | detail |
+|---|---|---|---|
+| quo | pricing_grid | warn-correct | warning riding with content; 23 figures flagged |
+| slack | pricing_grid | warn-correct | warning riding with content; 7 figures flagged |
+| notion | pricing_grid | warn-correct | warning riding with content; 3 figures flagged |
+| zapier | pricing_grid | warn-correct | warning riding with content; 9 figures flagged |
+| tailscale | pricing_grid | minor-omission | ['$5'] missing (80% retained — within the documented retention floor) |
+| zoom | pricing_tabbed | warn-correct | warning riding with content; 6 figures flagged |
+| ringcentral | pricing_tabbed | pass | 4/4 figures shown |
+| heyrosie | plan_cards | pass | 5/5 figures shown |
+| calendly | plan_cards | pass | 3/5 figures shown |
+| basecamp | plan_cards | pass | 3/3 figures shown |
+| hetzner | pricing_non_usd | pass | clean |
+| scaleway | pricing_non_usd | warn-correct | warning riding with content; 5 figures flagged |
+| giffgaff | pricing_non_usd | pass | clean |
+| linkedin_fp | article_peripheral | pass | 0/2 figures shown |
+| rfc9110 | pinned_immutable | pass | clean |
+| rfc2616_txt | pinned_immutable | pass | clean |
+| gutenberg_alice | pinned_immutable | pass | clean |
+| w3c_css2 | pinned_immutable | pass | clean |
+| gh_raw_readme | pinned_immutable | pass | clean |
+| wiki_idaho | reference_table | **FAIL** | SILENT omission: ['$74,900'] |
+| fed_h15 | data_table | pass | clean |
+| pg_essay | control_prose | pass | clean |
+| py_docs | control_docs | pass | clean |
+| fastapi_docs | control_docs | pass | clean |
+| guardian | consent_heavy | pass | content served (3951 chars) |
+| lemonde | consent_heavy | pass | content served (3788 chars) |
+| zeit | consent_heavy | pass | content served (3987 chars) |
+| ecfr | botwall | walled | honest refusal |
+| dodcio | botwall | walled | honest refusal |
+| crunchbase | botwall | walled | honest refusal |
+| todomvc | spa_shell | documented-limit | tiny output (300 chars) — the known custom-mount class |
+| aozora | charset_shift_jis | pass | clean |
+| wiki_ja | charset_utf8_ja | pass | clean |
+| gh_api | json_endpoint | pass | JSON pretty-printed |
+| berkshire_pdf | pdf_figures | pass | pdf text extracted; 3 money figures visible |
+| irs_p15 | pdf_tables | pass | pdf text extracted; 7 money figures visible |
+| map_uv | map | pass | 50 urls mapped |
+| map_fastapi | map | pass | 50 urls mapped |
+| crawl_ruff | crawl | pass | crawl: docs.astral.sh  pages: 5  errors: 0  skipped(robots/dupe/type): 0 |
+| search_smoke | search | pass | results returned |
+
+## Reading this report
+
+- `pass` — figures/content verified against the independent oracle.
+- `warn-correct` — the tool flagged its own extraction; the warning was warranted.
+- `walled` — honest refusal (bot wall / consent wall / HTTP error reported as such).
+- `FAIL` — a trust property was violated; see evidence/ for the raw bytes.
+- Every item's raw evidence is under `evidence/<id>/` for re-adjudication.

pyproject.toml

@@ -14,7 +14,7 @@ authors = [{ name = "Nate Nelson" }]
 dependencies = [
     "fastmcp>=3,<4",
     "httpx>=0.28",
-    "trafilatura>=2.1",
+    "trafilatura>=2.2",
     "ddgs>=9",
     "playwright>=1.45",
     "extruct>=0.18",

scripts/gen_article_peripheral_fixture.py

@@ -0,0 +1,100 @@
+"""Regenerate the article_peripheral synthetic probation fixture for trafilatura 2.2.0.
+
+The 2026-07-16 original bulked its article body with 14 byte-identical <p> tags.
+trafilatura 2.2.0 deduplicates identical blocks, so the body collapsed to a single
+paragraph (1,973 -> 166 chars) and the page began tripping the stub guard — while the
+REAL LinkedIn page it models went the other way (1,827 -> 13,560 chars, +642%).
+The synthetic had stopped modelling reality. This rebuild uses varied prose, which is
+what a real article has, and re-verifies the original calibration contract.
+"""
+
+import gzip
+import sys
+from pathlib import Path
+
+BODY = [
+    "The announcement details a merger between two infrastructure companies and what it "
+    "means for self-hosted deployments across the industry.",
+    "Both firms built their reputations on operator tooling rather than end-user products, "
+    "which is why the combination surprised analysts who had expected a platform vendor to "
+    "make the first move in this segment.",
+    "Customers running on-premise clusters have spent the week asking the only question "
+    "that matters to them, which is whether the existing support contracts survive the "
+    "transition intact or get renegotiated at the next renewal window.",
+    "Leadership addressed that directly in a call with partners, saying the support "
+    "organisations will merge slowly and that no contract will be reopened before its "
+    "scheduled term ends.",
+    "The engineering blog post accompanying the news is unusually candid about the parts "
+    "of the integration that remain unsolved, including two overlapping control planes "
+    "that cannot simply be collapsed into one another.",
+    "Migration tooling is promised for the second half of next year, though the post stops "
+    "short of committing to a date and notes that the schedule depends on how many "
+    "customers are running the older agent.",
+    "Competitors have already begun courting the installed base, with at least two vendors "
+    "publishing migration guides within days of the announcement.",
+    "That reaction is familiar to anyone who watched the previous consolidation wave in "
+    "this space, where the acquiring company lost roughly a third of the smaller firm's "
+    "customers inside eighteen months.",
+    "What makes this case different, according to people close to the deal, is that the "
+    "two products barely overlap in the workloads they were designed for.",
+    "One was built for long-running batch jobs on hardware the operator owns; the other "
+    "grew out of a scheduler for short-lived tasks in leased capacity.",
+    "Merging them means reconciling two scheduling models that make opposite assumptions "
+    "about how long a node is expected to live, and that is a genuinely hard problem "
+    "rather than a matter of shipping a compatibility shim.",
+    "The teams have said they will keep both schedulers alive indefinitely rather than "
+    "force a migration, which is the outcome most of the operator community was hoping "
+    "to hear.",
+    "Independent maintainers who package these tools for the major distributions have "
+    "asked for clarity on the licence, since one project carries a copyleft obligation "
+    "the other does not.",
+    "No answer has been given on that point yet, and it is the thread most likely to "
+    "generate friction over the coming months.",
+]
+
+# Peripheral feed cards: real dollar figures in related-content, which correct
+# main-content extraction excludes. Four distinct figures, deliberately spaced so no
+# 1,500-char window of visible text holds four — the v0.1.3 cluster-arming contract.
+FILLER = [
+    "Another author shares a long reflection on the state of the industry and its hiring "
+    "pipeline, with anecdotes from conference hallways and customer calls that never quite "
+    "resolve into a thesis.",
+    "A second commentator responds at length about procurement cycles, arguing that the "
+    "purchasing process shapes the architecture far more than any engineering preference "
+    "ever does.",
+    "A third post walks through a capacity planning exercise in detail, complete with the "
+    "spreadsheet assumptions that the author admits were wrong by a wide margin.",
+    "Someone else recounts a postmortem from a regional outage, noting that the review "
+    "found no single cause and that the team stopped looking for one.",
+    "A researcher summarises a preprint on scheduling fairness, then spends most of the "
+    "post explaining why the benchmark does not reflect production conditions.",
+]
+
+CARDS = [
+    "Community groups halted $130 billion of projects this quarter.",
+    "Token prices crashed from $17 per million down to just $2.",
+    "Big Tech's $8 trillion bet is reshaping electricity costs.",
+]
+
+
+def build() -> bytes:
+    body = "".join(f"<p>{p}</p>" for p in BODY)
+    cards = []
+    for i, lead in enumerate(CARDS):
+        # rotate the filler so no two cards are byte-identical (2.2.0 dedupes those)
+        pad = " ".join(FILLER[(i + j) % len(FILLER)] for j in range(5))
+        cards.append(f'<div class="feed-card"><p class="feed-card__commentary">{lead} {pad}</p></div>')
+    return (
+        "<html><head><title>Two Companies Join Forces | Industry Feed</title></head>"
+        "<body><nav>Home Feed Jobs</nav>"
+        f"<article><h1>Two Companies Join Forces</h1>{body}</article>"
+        f'<section class="related">{"".join(cards)}</section>'
+        "<footer>Terms Privacy</footer></body></html>"
+    ).encode()
+
+
+if __name__ == "__main__":
+    html = build()
+    out = Path(sys.argv[1])
+    out.write_bytes(gzip.compress(html))
+    print(f"wrote {out} ({len(html)} raw bytes, {out.stat().st_size} gzipped)")