diff --git a/CHANGELOG.md b/CHANGELOG.md index eb685e2..d6984fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,40 @@ failure. For the trust model and the evaluation harness behind those calls, see ## [Unreleased] +### Changed +- **Requires trafilatura >= 2.2**, which fixes the nested-emphasis serializer bug tearsheet + reported as [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882) + (released 2026-07-31, closed 2026-07-31). The `#882` canary test — which pinned the BROKEN + behavior on purpose so an upgrade would fail loudly — fired exactly as designed and is now + flipped to pin the corrected serialization as a permanent regression guard. +- **Adoption was measured, not assumed.** Replaying all 194 cached pages under both versions: + 77 extract more, 42 less, 75 identical, corpus total **+2.89%**, zero catastrophic losses. + 2.2.0 repairs three of the five silent-omission failures the stub guard was built on — + `sbir.gov/awards` (421 → 3,155 chars; it previously returned only the "official website of + the United States government" banner with no award data), `atlas.nomic.ai/pricing` + (259 → 1,661) and the NOAA SBIR archive. The stub guard now fires on 2 of 194 pages + (was 5 of 162) because the corpus got better, not because the guard got weaker. +- **Stub-guard calibration table recalibrated** against 2.2.0. Thresholds unchanged; the two + remaining fires still bracket the 10% floor tightly (8.1% / 9.7%). The old "do not raise + past ~570 (eur-lex)" lower bound no longer binds — eur-lex now extracts 6,167 chars. + +### Fixed +- **`article_peripheral` probation fixture regenerated** (`scripts/gen_article_peripheral_fixture.py` + is now the reproducible source). It padded its article body with 14 byte-identical `

` tags; + 2.2.0 deduplicates identical blocks, collapsing the body 1,973 → 166 chars and tripping the + stub guard — while the real page it models moved the opposite way (+642%). Its tests were also + strengthened: the old body assertion checked only `"Join Forces"`, which the page *title* + satisfies, so a 92% body collapse passed CI unnoticed. It now pins real body sentences, total + guard silence, and that silence comes from the cluster floor rather than a toothless page. + +### Known issues +- **The collapsed-column guard has no real-page coverage under 2.2.0.** Quo is just as lossy + (still 4 of 24 figures, 1 of 3 plan names) but emits the repeated cell once instead of three + times, so the repetition signature the guard keys on is gone — on quo and on every other page + in the cache replay. The dropped-price guard still catches quo, so the page is not silent. + The quo pin was narrowed deliberately and the substantive loss pinned separately rather than + the assertion being deleted quietly. + ## [0.1.5] — 2026-07-24 ### Added diff --git a/README.md b/README.md index 7b8d213..1d5b8b8 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ no telemetry. > *tearsheet (n.): a page torn from a publication and filed as proof it ran.* -**Trust status:** qualified for heavy usage 2026-07-16 — 247 tests, a falsifiable live +**Trust status:** qualified for heavy usage 2026-07-16 — 252 tests, a falsifiable live eval harness (verdict GREEN), and zero fabrications across the tool's entire recorded history. Its documented failure mode is *omission*, and the guards exist to make every omission loud. See [Trust](#trust). @@ -111,7 +111,10 @@ a research tool for reading the public web — not for evading paywalls or bot d clean while the figures never made it. Measured 2026-07-14 on real pages — `quo.com/pricing` kept 4 of 24 prices and flattened a 3-column matrix into repeated rows (`Unlimited* / Unlimited* / Unlimited*`), while `heyrosie.com/pricing` came through perfectly. So the failure - is site-shaped, not universal, and you cannot eyeball it from the output alone. + is site-shaped, not universal, and you cannot eyeball it from the output alone. Quo still + loses the same 20 of 24 figures under trafilatura 2.2.0, but emits that repeated cell once + instead of three times, so the **collapsed-column** guard no longer fires on it — the + dropped-price guard is what catches the page today. Since v0.1.2 the tool says so itself: a `warning:` line reports dropped prices and collapsed columns, and a cookie banner is reported as a `consent/cookie wall` instead of being served as the page. **Never quote figures from a warned extraction — re-run with `--raw` / `raw=true`.** @@ -143,12 +146,13 @@ a research tool for reading the public web — not for evading paywalls or bot d below every guard's floor by design (a one-figure floor would warn on every blog footer). The rule is procedural: any figure you are going to quote gets `--raw` or independent verification. -- **Emphasis mangling (upstream)**: trafilatura 2.1.0's markdown serializer displaces - nested-emphasis words (`word` mid-sentence) onto the next - paragraph and can drop characters around inline `` (observed: "i.e." → "e."). - Facts survive; verbatim quotes should be re-verified against the source before reuse. - Pinned by `tests/test_content.py::TestKnownUpstreamManglingDocumented` (fails when - upstream fixes it). Reported: [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882). +- **Emphasis mangling (upstream) — FIXED, now a regression pin.** trafilatura ≤ 2.1.0 + displaced nested-emphasis words (`word` mid-sentence) onto the + next paragraph and dropped characters around inline `` ("i.e." → "e."). Reported as + [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882), fixed in + trafilatura 2.2.0, verified downstream 2026-08-11 — tearsheet now requires `>=2.2`. + `tests/test_content.py::TestNestedEmphasisSerializedInPlace` pinned the broken behavior + as a canary, failed on cue when the fix landed, and now pins the correct serialization. - Bot-walled sites (eCFR, DoD, Cloudflare in strict mode) are reported as `blocked by bot protection …` — deliberately not evaded; use the site's official API. - Wikipedia extractions can include maintenance-hatnote table noise ("This article @@ -161,7 +165,7 @@ a research tool for reading the public web — not for evading paywalls or bot d "Can it be trusted for heavy usage?" is a measurement here, not a feeling. -- **Offline suite (247 tests, runs in the gate)**: guard boundary pins, cache-poisoning +- **Offline suite (252 tests, runs in the gate)**: guard boundary pins, cache-poisoning regressions, truncation honesty, charset torture, structure torture, adversarial robustness — enforced fully offline by a loopback-only socket guard. The five REAL pages that defined the tool's probation (quo, smith.ai, dialpad, heyrosie, a LinkedIn @@ -195,7 +199,7 @@ for figures you'll quote, treat a suspiciously small extraction of a rich page a .venv/bin/ruff check src tests && .venv/bin/mypy && .venv/bin/python -m pytest ``` -TDD throughout; the default suite (247 tests) runs entirely offline — `httpx.MockTransport`, +TDD throughout; the default suite (252 tests) runs entirely offline — `httpx.MockTransport`, fixture HTML, and a conftest socket guard that fails any test reaching for a non-loopback address. Extras: `pytest -m playwright` (real chromium, local server), `pytest -m live` (real network). The live trust evaluation lives in `evals/` (see [Trust](#trust)). diff --git a/evals/reports/2026-08-11/report.md b/evals/reports/2026-08-11/report.md new file mode 100644 index 0000000..accb1e5 --- /dev/null +++ b/evals/reports/2026-08-11/report.md @@ -0,0 +1,73 @@ +# Tearsheet Trust Report — 2026-08-11 13:13 + +**VERDICT: GREEN** + +- tearsheet 0.1.5 · trafilatura 2.2.0 · httpx 0.28.1 · lxml 6.1.1 · python 3.14.2 +- corpus 2026-07-16.2 (sha fe68ddd7bb2c) · 40 items scored · burst: 45 scrapes across 3 domains: {'ok': 40, 'honest_refusal': 5, 'FAIL': 0, 'exception': 0} + +## Gates + +| gate | status | detail | +|---|---|---| +| fabrication (0 tolerated) | PASS | 0 pages fabricated figures | +| guard calibration (0 regressions) | PASS | known-bad warns, known-good silent | +| silent omission (<=1) | PASS | 1 pages omitted without warning | +| minor omissions within retention floor (<=3) | PASS | tailscale: ['$5'] missing (80% retained — within the document | +| unhandled exceptions (0) | PASS | 0 | +| cache poison (0) | PASS | none served | +| timeouts (<=2) | PASS | 0 | +| baseline drift (0) | PASS | stable | +| hard failures (<=1 beyond the above) | PASS | wiki_idaho: SILENT omission: ['$74,900'] | + +## Items + +| id | category | status | detail | +|---|---|---|---| +| quo | pricing_grid | warn-correct | warning riding with content; 23 figures flagged | +| slack | pricing_grid | warn-correct | warning riding with content; 7 figures flagged | +| notion | pricing_grid | warn-correct | warning riding with content; 3 figures flagged | +| zapier | pricing_grid | warn-correct | warning riding with content; 9 figures flagged | +| tailscale | pricing_grid | minor-omission | ['$5'] missing (80% retained — within the documented retention floor) | +| zoom | pricing_tabbed | warn-correct | warning riding with content; 6 figures flagged | +| ringcentral | pricing_tabbed | pass | 4/4 figures shown | +| heyrosie | plan_cards | pass | 5/5 figures shown | +| calendly | plan_cards | pass | 3/5 figures shown | +| basecamp | plan_cards | pass | 3/3 figures shown | +| hetzner | pricing_non_usd | pass | clean | +| scaleway | pricing_non_usd | warn-correct | warning riding with content; 5 figures flagged | +| giffgaff | pricing_non_usd | pass | clean | +| linkedin_fp | article_peripheral | pass | 0/2 figures shown | +| rfc9110 | pinned_immutable | pass | clean | +| rfc2616_txt | pinned_immutable | pass | clean | +| gutenberg_alice | pinned_immutable | pass | clean | +| w3c_css2 | pinned_immutable | pass | clean | +| gh_raw_readme | pinned_immutable | pass | clean | +| wiki_idaho | reference_table | **FAIL** | SILENT omission: ['$74,900'] | +| fed_h15 | data_table | pass | clean | +| pg_essay | control_prose | pass | clean | +| py_docs | control_docs | pass | clean | +| fastapi_docs | control_docs | pass | clean | +| guardian | consent_heavy | pass | content served (3951 chars) | +| lemonde | consent_heavy | pass | content served (3788 chars) | +| zeit | consent_heavy | pass | content served (3987 chars) | +| ecfr | botwall | walled | honest refusal | +| dodcio | botwall | walled | honest refusal | +| crunchbase | botwall | walled | honest refusal | +| todomvc | spa_shell | documented-limit | tiny output (300 chars) — the known custom-mount class | +| aozora | charset_shift_jis | pass | clean | +| wiki_ja | charset_utf8_ja | pass | clean | +| gh_api | json_endpoint | pass | JSON pretty-printed | +| berkshire_pdf | pdf_figures | pass | pdf text extracted; 3 money figures visible | +| irs_p15 | pdf_tables | pass | pdf text extracted; 7 money figures visible | +| map_uv | map | pass | 50 urls mapped | +| map_fastapi | map | pass | 50 urls mapped | +| crawl_ruff | crawl | pass | crawl: docs.astral.sh pages: 5 errors: 0 skipped(robots/dupe/type): 0 | +| search_smoke | search | pass | results returned | + +## Reading this report + +- `pass` — figures/content verified against the independent oracle. +- `warn-correct` — the tool flagged its own extraction; the warning was warranted. +- `walled` — honest refusal (bot wall / consent wall / HTTP error reported as such). +- `FAIL` — a trust property was violated; see evidence/ for the raw bytes. +- Every item's raw evidence is under `evidence//` for re-adjudication. \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index 3670634..704665c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -14,7 +14,7 @@ authors = [{ name = "Nate Nelson" }] dependencies = [ "fastmcp>=3,<4", "httpx>=0.28", - "trafilatura>=2.1", + "trafilatura>=2.2", "ddgs>=9", "playwright>=1.45", "extruct>=0.18", diff --git a/scripts/gen_article_peripheral_fixture.py b/scripts/gen_article_peripheral_fixture.py new file mode 100644 index 0000000..78273b4 --- /dev/null +++ b/scripts/gen_article_peripheral_fixture.py @@ -0,0 +1,100 @@ +"""Regenerate the article_peripheral synthetic probation fixture for trafilatura 2.2.0. + +The 2026-07-16 original bulked its article body with 14 byte-identical

tags. +trafilatura 2.2.0 deduplicates identical blocks, so the body collapsed to a single +paragraph (1,973 -> 166 chars) and the page began tripping the stub guard — while the +REAL LinkedIn page it models went the other way (1,827 -> 13,560 chars, +642%). +The synthetic had stopped modelling reality. This rebuild uses varied prose, which is +what a real article has, and re-verifies the original calibration contract. +""" + +import gzip +import sys +from pathlib import Path + +BODY = [ + "The announcement details a merger between two infrastructure companies and what it " + "means for self-hosted deployments across the industry.", + "Both firms built their reputations on operator tooling rather than end-user products, " + "which is why the combination surprised analysts who had expected a platform vendor to " + "make the first move in this segment.", + "Customers running on-premise clusters have spent the week asking the only question " + "that matters to them, which is whether the existing support contracts survive the " + "transition intact or get renegotiated at the next renewal window.", + "Leadership addressed that directly in a call with partners, saying the support " + "organisations will merge slowly and that no contract will be reopened before its " + "scheduled term ends.", + "The engineering blog post accompanying the news is unusually candid about the parts " + "of the integration that remain unsolved, including two overlapping control planes " + "that cannot simply be collapsed into one another.", + "Migration tooling is promised for the second half of next year, though the post stops " + "short of committing to a date and notes that the schedule depends on how many " + "customers are running the older agent.", + "Competitors have already begun courting the installed base, with at least two vendors " + "publishing migration guides within days of the announcement.", + "That reaction is familiar to anyone who watched the previous consolidation wave in " + "this space, where the acquiring company lost roughly a third of the smaller firm's " + "customers inside eighteen months.", + "What makes this case different, according to people close to the deal, is that the " + "two products barely overlap in the workloads they were designed for.", + "One was built for long-running batch jobs on hardware the operator owns; the other " + "grew out of a scheduler for short-lived tasks in leased capacity.", + "Merging them means reconciling two scheduling models that make opposite assumptions " + "about how long a node is expected to live, and that is a genuinely hard problem " + "rather than a matter of shipping a compatibility shim.", + "The teams have said they will keep both schedulers alive indefinitely rather than " + "force a migration, which is the outcome most of the operator community was hoping " + "to hear.", + "Independent maintainers who package these tools for the major distributions have " + "asked for clarity on the licence, since one project carries a copyleft obligation " + "the other does not.", + "No answer has been given on that point yet, and it is the thread most likely to " + "generate friction over the coming months.", +] + +# Peripheral feed cards: real dollar figures in related-content, which correct +# main-content extraction excludes. Four distinct figures, deliberately spaced so no +# 1,500-char window of visible text holds four — the v0.1.3 cluster-arming contract. +FILLER = [ + "Another author shares a long reflection on the state of the industry and its hiring " + "pipeline, with anecdotes from conference hallways and customer calls that never quite " + "resolve into a thesis.", + "A second commentator responds at length about procurement cycles, arguing that the " + "purchasing process shapes the architecture far more than any engineering preference " + "ever does.", + "A third post walks through a capacity planning exercise in detail, complete with the " + "spreadsheet assumptions that the author admits were wrong by a wide margin.", + "Someone else recounts a postmortem from a regional outage, noting that the review " + "found no single cause and that the team stopped looking for one.", + "A researcher summarises a preprint on scheduling fairness, then spends most of the " + "post explaining why the benchmark does not reflect production conditions.", +] + +CARDS = [ + "Community groups halted $130 billion of projects this quarter.", + "Token prices crashed from $17 per million down to just $2.", + "Big Tech's $8 trillion bet is reshaping electricity costs.", +] + + +def build() -> bytes: + body = "".join(f"

{p}

" for p in BODY) + cards = [] + for i, lead in enumerate(CARDS): + # rotate the filler so no two cards are byte-identical (2.2.0 dedupes those) + pad = " ".join(FILLER[(i + j) % len(FILLER)] for j in range(5)) + cards.append(f'

{lead} {pad}

') + return ( + "Two Companies Join Forces | Industry Feed" + "" + f"

Two Companies Join Forces

{body}
" + f'
{"".join(cards)}
' + "
Terms Privacy
" + ).encode() + + +if __name__ == "__main__": + html = build() + out = Path(sys.argv[1]) + out.write_bytes(gzip.compress(html)) + print(f"wrote {out} ({len(html)} raw bytes, {out.stat().st_size} gzipped)") diff --git a/src/tearsheet/content.py b/src/tearsheet/content.py index 6319e92..595bae8 100644 --- a/src/tearsheet/content.py +++ b/src/tearsheet/content.py @@ -75,22 +75,32 @@ # price, a table, or a wall, so no other guard can arm. # # NOT the bare yield ratio that was measured and rejected — that fires on neither real -# pricing failure (smith.ai 13%, quo 21% retention here). Retention is only half the -# condition; the other half is an absolute stub-sized output. Calibrated by replaying -# every content-bearing page in the live cache (162 pages; fires on 5, i.e. 3.1%): +# pricing failure (smith.ai 12%, quo 19% retention here). Retention is only half the +# condition; the other half is an absolute stub-sized output. +# +# RECALIBRATED 2026-08-11 against trafilatura 2.2.0 by replaying every content-bearing +# page in the live cache (194 pages; fires on 2, i.e. 1.0%). The thresholds below did not +# move — the CORPUS did: 2.2.0 repaired three of the five pages this guard was built on, +# which is the outcome you want (the guard goes quiet because the extraction got good). # # page visible md ratio verdict -# sbir.gov/awards 10,698 421 3.9% FIRE (only the .gov banner) -# noaa SBIR archive 6,373 361 5.7% FIRE (1 of 3+ stories kept) # apex/our-team 1,994 161 8.1% FIRE (marketing line only) -# nomic.ai/pricing 3,043 259 8.5% FIRE (marketing, zero prices) # pocatello Fees 2,134 206 9.7% FIRE (already price-warned) -# eur-lex search 11,022 571 5.2% silent (real result list) -# dronedominance 16,661 1,553 9.3% silent -# linkedin FP 17,890 1,827 10.2% silent (MUST stay silent) -# smith.ai 10,961 1,452 13.2% silent -# heyrosie 16,218 2,263 14.0% silent -# quo 17,961 3,716 20.7% silent +# smith.ai 10,961 1,316 12.0% silent +# noaa SBIR archive 4,543 548 12.1% silent (was FIRE @ 5.7% on 2.1.0) +# quo 17,961 3,485 19.4% silent +# sbir.gov/awards 10,698 3,155 29.5% silent (was FIRE @ 3.9% — the +# .gov-banner-only failure +# is FIXED upstream) +# nomic.ai/pricing 3,043 1,661 54.6% silent (was FIRE @ 8.5%) +# eur-lex search 11,022 6,167 56.0% silent (real result list) +# linkedin FP 17,890 13,560 75.8% silent (MUST stay silent) +# heyrosie 13,728 11,814 86.1% silent +# dronedominance 16,661 14,871 89.3% silent +# +# The two survivors still bracket the floor tightly (8.1% / 9.7% against 10%), so the +# ratio term is doing real work. The old lower bound — "do NOT raise past ~570 (eur-lex)" +# — no longer binds: eur-lex now extracts 6,167 chars. Re-derive before moving anything. # # Measure with a FRESH extract_content, not a cached `markdown` column — stored rows may # have been written with include_links=True, which inflates length by ~70% and hid the diff --git a/tests/fixtures/probation/article_peripheral.html.gz b/tests/fixtures/probation/article_peripheral.html.gz index 7f05f0b..bda2345 100644 Binary files a/tests/fixtures/probation/article_peripheral.html.gz and b/tests/fixtures/probation/article_peripheral.html.gz differ diff --git a/tests/test_adversarial.py b/tests/test_adversarial.py index a572c39..ca63b36 100644 --- a/tests/test_adversarial.py +++ b/tests/test_adversarial.py @@ -2,8 +2,14 @@ Every test here answers one question — does the tool ever fail SILENTLY? A crash is acceptable to catch; a wrong-but-confident answer is not. Where current behavior -is an accepted limitation, the test PINS it and says so, mirroring the trafilatura -#882 canary pattern: the pin fails loudly the day the behavior changes. +is an accepted limitation, the test PINS it and says so: the pin fails loudly the day +the behavior changes. + +That pattern has now paid out once end to end. The trafilatura #882 canary in +tests/test_content.py pinned an upstream markdown-serializer bug on purpose, failed the +moment 2.2.0 shipped the fix (2026-08-11), and was flipped to pin the corrected output. +TestDialpadPricing in tests/test_probation_pages.py is the live example of the same +shape — it pins a limitation and tells you to celebrate when it breaks. """ from collections.abc import Callable diff --git a/tests/test_content.py b/tests/test_content.py index b55155b..38c0cf0 100644 --- a/tests/test_content.py +++ b/tests/test_content.py @@ -35,29 +35,41 @@ def test_links_included_on_request(self, fixture_bytes: Callable[[str], bytes]) assert "https://example.com/related" in result.markdown -class TestKnownUpstreamManglingDocumented: - """Documents trafilatura 2.1.0's markdown-serializer bug with nested emphasis - (word mid-sentence): the emphasized word is torn out - and glued to the START of the next paragraph, and the remainder of its sentence - is wrongly bolded. Found in the wild 2026-07-11 (burr.com CMMC article) during - the probation audit; reported upstream. - - THESE ASSERTIONS PIN THE BROKEN BEHAVIOR ON PURPOSE: when a trafilatura upgrade - fixes the serializer, this test FAILS — that's the signal to drop the README - known-issue entry and this test, not a regression. +class TestNestedEmphasisSerializedInPlace: + """Nested emphasis (word mid-sentence) stays in its own + sentence. trafilatura <= 2.1.0 tore the emphasized word out and glued it to the START + of the next paragraph, leaving the original sentence broken mid-phrase, and dropped + characters around inline ("i.e." -> "e."). Found in the wild 2026-07-11 (burr.com + CMMC article) during the probation audit, reported as adbar/trafilatura#882, fixed in + trafilatura 2.2.0 (`_md_wrap` keeps flanking whitespace outside the marker, so the + closing delimiter is right-flanking per CommonMark) and verified here 2026-08-11. + + This started life as a canary pinning the BROKEN behavior so an upgrade would fail + loudly. That fired as designed; the assertions are now flipped to pin the CORRECT + serialization, so the class stays a permanent regression guard against re-breakage. """ - def test_nested_emphasis_displacement_still_present( + def test_emphasized_word_stays_in_its_sentence( + self, fixture_bytes: Callable[[str], bytes] + ) -> None: + result = extract_content( + fixture_bytes("emphasis_mangle.html"), url="https://example.com/repro" + ) + assert result is not None + # the word stays put, nested emphasis collapses to a single ***…*** run + assert "plan to ***verify*** contractor compliance" in result.markdown + # and it is NOT glued to the head of the next paragraph + assert "*verify*Starting November 10" not in result.markdown + + def test_inline_emphasis_does_not_drop_characters( self, fixture_bytes: Callable[[str], bytes] ) -> None: result = extract_content( fixture_bytes("emphasis_mangle.html"), url="https://example.com/repro" ) assert result is not None - # "verify" is displaced onto the start of the NEXT paragraph - assert "*verify*Starting November 10" in result.markdown - # and its original sentence is left broken mid-phrase - assert "plan to \n" in result.markdown or "plan to\n" in result.markdown + # "i.e." survives intact — 2.1.0 rendered this as "e." + assert "*i.e.*" in result.markdown class TestDegenerateInput: diff --git a/tests/test_probation_pages.py b/tests/test_probation_pages.py index 9837d68..983df06 100644 --- a/tests/test_probation_pages.py +++ b/tests/test_probation_pages.py @@ -19,6 +19,14 @@ substantial main body with no figures. Calibrated at creation: silent under the shipped guards, warns if the cluster floor drops to 1 (so it genuinely exercises the arming logic). The live class stays covered by evals/corpus.json (linkedin_fp). + +REGENERATED 2026-08-11 for trafilatura 2.2.0 — `scripts/gen_article_peripheral_fixture.py` +is the reproducible source. The 07-16 build padded its article body with 14 byte-identical +

tags; 2.2.0 deduplicates identical blocks, so the body collapsed from 1,973 to 166 +chars and the page started tripping the STUB guard. Meanwhile the REAL page it models +moved the opposite way (1,827 -> 13,560 chars, +642%), so the synthetic had stopped +modelling reality in both magnitude and direction. The rebuild uses varied prose, which +is what a real article has, and re-verifies every clause of the contract above. """ import gzip @@ -26,7 +34,7 @@ import pytest -from tearsheet.content import assess_extraction, extract_content +from tearsheet.content import _MONEY, assess_extraction, extract_content, html_to_text PROBATION = Path(__file__).parent / "fixtures" / "probation" @@ -51,7 +59,31 @@ class TestQuoPricing: def test_the_original_failure_still_warns(self) -> None: price, column, wall = price_warned(load("quo_pricing")) assert price, "quo dropped 20/24 prices — the guard MUST fire" - assert column, "quo collapsed its 3-column matrix — the guard MUST fire" + + def test_quo_is_still_substantively_broken(self) -> None: + """The failure is unchanged; only the COLUMN SIGNATURE moved (2026-08-11). + + Through trafilatura 2.1.0 quo's 3-column grid flattened into runs of identical + consecutive rows (`Unlimited* / Unlimited* / Unlimited*`), which is what + _has_collapsed_columns keys on, so both the price AND column guards fired. + 2.2.0 emits that cell ONCE instead of three times: the page is just as lossy + (still 4 of 24 figures, still 1 of 3 plan names) but the repetition the guard + detects is gone, so only the price guard fires now. + + The column assertion was retired here rather than deleted quietly: a full-cache + replay (194 pages, both versions) showed the collapsed-column guard now fires on + NO page it used to catch, i.e. it has zero real-page coverage under 2.2.0. That + gap is tracked as its own work item; this test pins the substantive loss so the + page can never go silent while the signature question is open. + """ + extracted = extract_content(load("quo_pricing")) + assert extracted is not None + page_figures = set(_MONEY.findall(html_to_text(load("quo_pricing")))) + kept = page_figures & set(_MONEY.findall(extracted.markdown)) + assert len(page_figures) >= 20, "fixture must still carry the full price grid" + assert len(kept) / len(page_figures) < 0.5, "quo must still read as a major drop" + plans = [name for name in ("Starter", "Business", "Scale") if name in extracted.markdown] + assert len(plans) < 3, "plan-name attribution is still lossy" class TestSmithAiPricing: @@ -89,10 +121,56 @@ def test_the_false_positive_stays_dead(self) -> None: assert not price, "peripheral real figures must not trip the guard (v0.1.3 fix)" assert not wall + def test_the_page_is_completely_silent(self) -> None: + """No warning of ANY kind — this is the false-positive fixture. + + The old assertions only checked the price and column guards, so when the body + collapsed under trafilatura 2.2.0 the page began emitting a STUB warning and + every test still passed. Assert total silence so any future guard can't quietly + start firing on the one page that must never warn. + """ + html = load("article_peripheral") + quality = assess_extraction(html, extract_content(html)) + assert quality.warnings == [], f"FP fixture must stay silent, got: {quality.warnings}" + assert not quality.consent_wall and not quality.block_wall + def test_the_article_body_still_extracts(self) -> None: + """Pin the BODY, not just the headline. + + The previous version asserted only `"Join Forces" in markdown` — which the page + TITLE satisfies. That is why a 92% body collapse (1,973 -> 166 chars under 2.2.0) + passed CI unnoticed. These assertions require real body sentences. + """ extracted = extract_content(load("article_peripheral")) assert extracted is not None - assert "Join Forces" in extracted.markdown + assert len(extracted.markdown) > 1_500, "the article body must survive extraction" + assert "surprised analysts" in extracted.markdown + assert "copyleft obligation" in extracted.markdown + + def test_peripheral_figures_are_excluded_but_present_on_the_page(self) -> None: + """The whole point of the fixture: real figures exist, correct extraction drops + them (they are related-content), and the guard stays quiet about it.""" + html = load("article_peripheral") + extracted = extract_content(html) + assert extracted is not None + on_page = set(_MONEY.findall(html_to_text(html))) + assert len(on_page) == 4, f"fixture must carry 4 distinct figures, got {on_page}" + assert not (on_page & set(_MONEY.findall(extracted.markdown))) + + def test_it_genuinely_exercises_the_arming_logic( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Silence must come from the CLUSTER FLOOR, not from the page being toothless. + + Drop the floor to 1 and the guard must fire — otherwise this fixture would pass + the FP test for the wrong reason and stop protecting anything. + """ + import tearsheet.content as content_module + + monkeypatch.setattr(content_module, "_MONEY_MIN_CLUSTERED", 1) + html = load("article_peripheral") + quality = assess_extraction(html, extract_content(html)) + assert any("price" in w for w in quality.warnings) @pytest.mark.parametrize(