From f55c1c0a4832f0ba9d63bad8085808fd669e20f9 Mon Sep 17 00:00:00 2001 From: nathan nelson Date: Tue, 11 Aug 2026 13:10:16 -0600 Subject: [PATCH 1/2] =?UTF-8?q?deps:=20require=20trafilatura=20>=3D2.2=20?= =?UTF-8?q?=E2=80=94=20the=20#882=20canary=20fired,=20close=20the=20loop?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The canary worked. tests/test_content.py pinned trafilatura's BROKEN nested-emphasis serializer on purpose so an upgrade would fail loudly. trafilatura 2.2.0 shipped the fix on 2026-07-31 (issue #882 closed the same day), the test failed exactly as designed, and it is now flipped to pin the CORRECT serialization as a permanent regression guard. Adoption was measured, not assumed. Replaying all 194 cached pages under both versions: 77 extract more, 42 less, 75 identical, corpus total +2.89%, zero catastrophic losses. 2.2.0 repairs three of the five silent-omission failures the stub guard was built on — sbir.gov/awards (421 -> 3,155 chars; it previously returned only the "official website of the United States government" banner with no award data), atlas.nomic.ai/pricing (259 -> 1,661) and the NOAA SBIR archive. Two pins moved, both deliberately and neither quietly: - article_peripheral (the false-positive fixture) was regenerated. It padded its body with 14 byte-identical

tags; 2.2.0 deduplicates identical blocks, so the body collapsed 1,973 -> 166 chars and the page began tripping the stub guard — while the real page it models went the other way (+642%). The synthetic had stopped modelling reality in both magnitude and direction. scripts/gen_article_peripheral_fixture.py is now its reproducible source, and the rebuild re-verifies every clause of the original contract. Its tests were the reason nobody noticed: the body assertion checked only "Join Forces", which the page TITLE satisfies, so a 92% collapse passed CI. It now pins real body sentences, total guard silence, and that the silence comes from the cluster floor rather than a toothless page. - The quo collapsed-column assertion was narrowed. Quo is just as lossy (still 4 of 24 figures, 1 of 3 plan names) but 2.2.0 emits the repeated cell once instead of three times, so the repetition signature the guard keys on is gone — on quo and on every page in the cache replay. The dropped-price guard still catches quo, so the page is not silent. The substantive loss is now pinned separately, and the zero-real-page-coverage gap is recorded rather than papered over. Stub-guard calibration table recalibrated against 2.2.0: thresholds unchanged, the corpus improved. Fires on 2 of 194 (was 5 of 162). Gate: ruff, mypy, 252 tests, consistency all green. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 34 ++++++ README.md | 24 +++-- pyproject.toml | 2 +- scripts/gen_article_peripheral_fixture.py | 100 ++++++++++++++++++ src/tearsheet/content.py | 34 +++--- .../probation/article_peripheral.html.gz | Bin 534 -> 1980 bytes tests/test_adversarial.py | 10 +- tests/test_content.py | 42 +++++--- tests/test_probation_pages.py | 84 ++++++++++++++- 9 files changed, 287 insertions(+), 43 deletions(-) create mode 100644 scripts/gen_article_peripheral_fixture.py diff --git a/CHANGELOG.md b/CHANGELOG.md index eb685e2..d6984fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,40 @@ failure. For the trust model and the evaluation harness behind those calls, see ## [Unreleased] +### Changed +- **Requires trafilatura >= 2.2**, which fixes the nested-emphasis serializer bug tearsheet + reported as [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882) + (released 2026-07-31, closed 2026-07-31). The `#882` canary test — which pinned the BROKEN + behavior on purpose so an upgrade would fail loudly — fired exactly as designed and is now + flipped to pin the corrected serialization as a permanent regression guard. +- **Adoption was measured, not assumed.** Replaying all 194 cached pages under both versions: + 77 extract more, 42 less, 75 identical, corpus total **+2.89%**, zero catastrophic losses. + 2.2.0 repairs three of the five silent-omission failures the stub guard was built on — + `sbir.gov/awards` (421 → 3,155 chars; it previously returned only the "official website of + the United States government" banner with no award data), `atlas.nomic.ai/pricing` + (259 → 1,661) and the NOAA SBIR archive. The stub guard now fires on 2 of 194 pages + (was 5 of 162) because the corpus got better, not because the guard got weaker. +- **Stub-guard calibration table recalibrated** against 2.2.0. Thresholds unchanged; the two + remaining fires still bracket the 10% floor tightly (8.1% / 9.7%). The old "do not raise + past ~570 (eur-lex)" lower bound no longer binds — eur-lex now extracts 6,167 chars. + +### Fixed +- **`article_peripheral` probation fixture regenerated** (`scripts/gen_article_peripheral_fixture.py` + is now the reproducible source). It padded its article body with 14 byte-identical `

` tags; + 2.2.0 deduplicates identical blocks, collapsing the body 1,973 → 166 chars and tripping the + stub guard — while the real page it models moved the opposite way (+642%). Its tests were also + strengthened: the old body assertion checked only `"Join Forces"`, which the page *title* + satisfies, so a 92% body collapse passed CI unnoticed. It now pins real body sentences, total + guard silence, and that silence comes from the cluster floor rather than a toothless page. + +### Known issues +- **The collapsed-column guard has no real-page coverage under 2.2.0.** Quo is just as lossy + (still 4 of 24 figures, 1 of 3 plan names) but emits the repeated cell once instead of three + times, so the repetition signature the guard keys on is gone — on quo and on every other page + in the cache replay. The dropped-price guard still catches quo, so the page is not silent. + The quo pin was narrowed deliberately and the substantive loss pinned separately rather than + the assertion being deleted quietly. + ## [0.1.5] — 2026-07-24 ### Added diff --git a/README.md b/README.md index 7b8d213..1d5b8b8 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ no telemetry. > *tearsheet (n.): a page torn from a publication and filed as proof it ran.* -**Trust status:** qualified for heavy usage 2026-07-16 — 247 tests, a falsifiable live +**Trust status:** qualified for heavy usage 2026-07-16 — 252 tests, a falsifiable live eval harness (verdict GREEN), and zero fabrications across the tool's entire recorded history. Its documented failure mode is *omission*, and the guards exist to make every omission loud. See [Trust](#trust). @@ -111,7 +111,10 @@ a research tool for reading the public web — not for evading paywalls or bot d clean while the figures never made it. Measured 2026-07-14 on real pages — `quo.com/pricing` kept 4 of 24 prices and flattened a 3-column matrix into repeated rows (`Unlimited* / Unlimited* / Unlimited*`), while `heyrosie.com/pricing` came through perfectly. So the failure - is site-shaped, not universal, and you cannot eyeball it from the output alone. + is site-shaped, not universal, and you cannot eyeball it from the output alone. Quo still + loses the same 20 of 24 figures under trafilatura 2.2.0, but emits that repeated cell once + instead of three times, so the **collapsed-column** guard no longer fires on it — the + dropped-price guard is what catches the page today. Since v0.1.2 the tool says so itself: a `warning:` line reports dropped prices and collapsed columns, and a cookie banner is reported as a `consent/cookie wall` instead of being served as the page. **Never quote figures from a warned extraction — re-run with `--raw` / `raw=true`.** @@ -143,12 +146,13 @@ a research tool for reading the public web — not for evading paywalls or bot d below every guard's floor by design (a one-figure floor would warn on every blog footer). The rule is procedural: any figure you are going to quote gets `--raw` or independent verification. -- **Emphasis mangling (upstream)**: trafilatura 2.1.0's markdown serializer displaces - nested-emphasis words (`word` mid-sentence) onto the next - paragraph and can drop characters around inline `` (observed: "i.e." → "e."). - Facts survive; verbatim quotes should be re-verified against the source before reuse. - Pinned by `tests/test_content.py::TestKnownUpstreamManglingDocumented` (fails when - upstream fixes it). Reported: [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882). +- **Emphasis mangling (upstream) — FIXED, now a regression pin.** trafilatura ≤ 2.1.0 + displaced nested-emphasis words (`word` mid-sentence) onto the + next paragraph and dropped characters around inline `` ("i.e." → "e."). Reported as + [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882), fixed in + trafilatura 2.2.0, verified downstream 2026-08-11 — tearsheet now requires `>=2.2`. + `tests/test_content.py::TestNestedEmphasisSerializedInPlace` pinned the broken behavior + as a canary, failed on cue when the fix landed, and now pins the correct serialization. - Bot-walled sites (eCFR, DoD, Cloudflare in strict mode) are reported as `blocked by bot protection …` — deliberately not evaded; use the site's official API. - Wikipedia extractions can include maintenance-hatnote table noise ("This article @@ -161,7 +165,7 @@ a research tool for reading the public web — not for evading paywalls or bot d "Can it be trusted for heavy usage?" is a measurement here, not a feeling. -- **Offline suite (247 tests, runs in the gate)**: guard boundary pins, cache-poisoning +- **Offline suite (252 tests, runs in the gate)**: guard boundary pins, cache-poisoning regressions, truncation honesty, charset torture, structure torture, adversarial robustness — enforced fully offline by a loopback-only socket guard. The five REAL pages that defined the tool's probation (quo, smith.ai, dialpad, heyrosie, a LinkedIn @@ -195,7 +199,7 @@ for figures you'll quote, treat a suspiciously small extraction of a rich page a .venv/bin/ruff check src tests && .venv/bin/mypy && .venv/bin/python -m pytest ``` -TDD throughout; the default suite (247 tests) runs entirely offline — `httpx.MockTransport`, +TDD throughout; the default suite (252 tests) runs entirely offline — `httpx.MockTransport`, fixture HTML, and a conftest socket guard that fails any test reaching for a non-loopback address. Extras: `pytest -m playwright` (real chromium, local server), `pytest -m live` (real network). The live trust evaluation lives in `evals/` (see [Trust](#trust)). diff --git a/pyproject.toml b/pyproject.toml index 3670634..704665c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -14,7 +14,7 @@ authors = [{ name = "Nate Nelson" }] dependencies = [ "fastmcp>=3,<4", "httpx>=0.28", - "trafilatura>=2.1", + "trafilatura>=2.2", "ddgs>=9", "playwright>=1.45", "extruct>=0.18", diff --git a/scripts/gen_article_peripheral_fixture.py b/scripts/gen_article_peripheral_fixture.py new file mode 100644 index 0000000..78273b4 --- /dev/null +++ b/scripts/gen_article_peripheral_fixture.py @@ -0,0 +1,100 @@ +"""Regenerate the article_peripheral synthetic probation fixture for trafilatura 2.2.0. + +The 2026-07-16 original bulked its article body with 14 byte-identical

tags. +trafilatura 2.2.0 deduplicates identical blocks, so the body collapsed to a single +paragraph (1,973 -> 166 chars) and the page began tripping the stub guard — while the +REAL LinkedIn page it models went the other way (1,827 -> 13,560 chars, +642%). +The synthetic had stopped modelling reality. This rebuild uses varied prose, which is +what a real article has, and re-verifies the original calibration contract. +""" + +import gzip +import sys +from pathlib import Path + +BODY = [ + "The announcement details a merger between two infrastructure companies and what it " + "means for self-hosted deployments across the industry.", + "Both firms built their reputations on operator tooling rather than end-user products, " + "which is why the combination surprised analysts who had expected a platform vendor to " + "make the first move in this segment.", + "Customers running on-premise clusters have spent the week asking the only question " + "that matters to them, which is whether the existing support contracts survive the " + "transition intact or get renegotiated at the next renewal window.", + "Leadership addressed that directly in a call with partners, saying the support " + "organisations will merge slowly and that no contract will be reopened before its " + "scheduled term ends.", + "The engineering blog post accompanying the news is unusually candid about the parts " + "of the integration that remain unsolved, including two overlapping control planes " + "that cannot simply be collapsed into one another.", + "Migration tooling is promised for the second half of next year, though the post stops " + "short of committing to a date and notes that the schedule depends on how many " + "customers are running the older agent.", + "Competitors have already begun courting the installed base, with at least two vendors " + "publishing migration guides within days of the announcement.", + "That reaction is familiar to anyone who watched the previous consolidation wave in " + "this space, where the acquiring company lost roughly a third of the smaller firm's " + "customers inside eighteen months.", + "What makes this case different, according to people close to the deal, is that the " + "two products barely overlap in the workloads they were designed for.", + "One was built for long-running batch jobs on hardware the operator owns; the other " + "grew out of a scheduler for short-lived tasks in leased capacity.", + "Merging them means reconciling two scheduling models that make opposite assumptions " + "about how long a node is expected to live, and that is a genuinely hard problem " + "rather than a matter of shipping a compatibility shim.", + "The teams have said they will keep both schedulers alive indefinitely rather than " + "force a migration, which is the outcome most of the operator community was hoping " + "to hear.", + "Independent maintainers who package these tools for the major distributions have " + "asked for clarity on the licence, since one project carries a copyleft obligation " + "the other does not.", + "No answer has been given on that point yet, and it is the thread most likely to " + "generate friction over the coming months.", +] + +# Peripheral feed cards: real dollar figures in related-content, which correct +# main-content extraction excludes. Four distinct figures, deliberately spaced so no +# 1,500-char window of visible text holds four — the v0.1.3 cluster-arming contract. +FILLER = [ + "Another author shares a long reflection on the state of the industry and its hiring " + "pipeline, with anecdotes from conference hallways and customer calls that never quite " + "resolve into a thesis.", + "A second commentator responds at length about procurement cycles, arguing that the " + "purchasing process shapes the architecture far more than any engineering preference " + "ever does.", + "A third post walks through a capacity planning exercise in detail, complete with the " + "spreadsheet assumptions that the author admits were wrong by a wide margin.", + "Someone else recounts a postmortem from a regional outage, noting that the review " + "found no single cause and that the team stopped looking for one.", + "A researcher summarises a preprint on scheduling fairness, then spends most of the " + "post explaining why the benchmark does not reflect production conditions.", +] + +CARDS = [ + "Community groups halted $130 billion of projects this quarter.", + "Token prices crashed from $17 per million down to just $2.", + "Big Tech's $8 trillion bet is reshaping electricity costs.", +] + + +def build() -> bytes: + body = "".join(f"

{p}

" for p in BODY) + cards = [] + for i, lead in enumerate(CARDS): + # rotate the filler so no two cards are byte-identical (2.2.0 dedupes those) + pad = " ".join(FILLER[(i + j) % len(FILLER)] for j in range(5)) + cards.append(f'

{lead} {pad}

') + return ( + "Two Companies Join Forces | Industry Feed" + "" + f"

Two Companies Join Forces

{body}
" + f'
{"".join(cards)}
' + "
Terms Privacy
" + ).encode() + + +if __name__ == "__main__": + html = build() + out = Path(sys.argv[1]) + out.write_bytes(gzip.compress(html)) + print(f"wrote {out} ({len(html)} raw bytes, {out.stat().st_size} gzipped)") diff --git a/src/tearsheet/content.py b/src/tearsheet/content.py index 6319e92..595bae8 100644 --- a/src/tearsheet/content.py +++ b/src/tearsheet/content.py @@ -75,22 +75,32 @@ # price, a table, or a wall, so no other guard can arm. # # NOT the bare yield ratio that was measured and rejected — that fires on neither real -# pricing failure (smith.ai 13%, quo 21% retention here). Retention is only half the -# condition; the other half is an absolute stub-sized output. Calibrated by replaying -# every content-bearing page in the live cache (162 pages; fires on 5, i.e. 3.1%): +# pricing failure (smith.ai 12%, quo 19% retention here). Retention is only half the +# condition; the other half is an absolute stub-sized output. +# +# RECALIBRATED 2026-08-11 against trafilatura 2.2.0 by replaying every content-bearing +# page in the live cache (194 pages; fires on 2, i.e. 1.0%). The thresholds below did not +# move — the CORPUS did: 2.2.0 repaired three of the five pages this guard was built on, +# which is the outcome you want (the guard goes quiet because the extraction got good). # # page visible md ratio verdict -# sbir.gov/awards 10,698 421 3.9% FIRE (only the .gov banner) -# noaa SBIR archive 6,373 361 5.7% FIRE (1 of 3+ stories kept) # apex/our-team 1,994 161 8.1% FIRE (marketing line only) -# nomic.ai/pricing 3,043 259 8.5% FIRE (marketing, zero prices) # pocatello Fees 2,134 206 9.7% FIRE (already price-warned) -# eur-lex search 11,022 571 5.2% silent (real result list) -# dronedominance 16,661 1,553 9.3% silent -# linkedin FP 17,890 1,827 10.2% silent (MUST stay silent) -# smith.ai 10,961 1,452 13.2% silent -# heyrosie 16,218 2,263 14.0% silent -# quo 17,961 3,716 20.7% silent +# smith.ai 10,961 1,316 12.0% silent +# noaa SBIR archive 4,543 548 12.1% silent (was FIRE @ 5.7% on 2.1.0) +# quo 17,961 3,485 19.4% silent +# sbir.gov/awards 10,698 3,155 29.5% silent (was FIRE @ 3.9% — the +# .gov-banner-only failure +# is FIXED upstream) +# nomic.ai/pricing 3,043 1,661 54.6% silent (was FIRE @ 8.5%) +# eur-lex search 11,022 6,167 56.0% silent (real result list) +# linkedin FP 17,890 13,560 75.8% silent (MUST stay silent) +# heyrosie 13,728 11,814 86.1% silent +# dronedominance 16,661 14,871 89.3% silent +# +# The two survivors still bracket the floor tightly (8.1% / 9.7% against 10%), so the +# ratio term is doing real work. The old lower bound — "do NOT raise past ~570 (eur-lex)" +# — no longer binds: eur-lex now extracts 6,167 chars. Re-derive before moving anything. # # Measure with a FRESH extract_content, not a cached `markdown` column — stored rows may # have been written with include_links=True, which inflates length by ~70% and hid the diff --git a/tests/fixtures/probation/article_peripheral.html.gz b/tests/fixtures/probation/article_peripheral.html.gz index 7f05f0baeb17b00ad69aa1bcae994dc19f60fd9e..bda23454cfc148c1780b75b03899a53545db1f28 100644 GIT binary patch literal 1980 zcmV;t2SfNDiwFP!00002|Ls^^i{!`={42$>;Fq2oul-h!!|zK;DLKQN>WOy z?NU{3RcU$~{No*wC29IFez|)v2m3ZH)kkJzL}Y5tb6x!9Jc}JJ=jy7L%gZgAZ{m_{ za8k@)qYLKSn0x&FZ}XQhtfl6=`Br2&Ker9%=eE6z!~Jp&_Vx1TxQH&n_O6_t`Rr_S zbv>8QAO80pIP1gZoGvf37#l)dLobVjY6hv+`C_bDBu|n}C-o*0OvOFqqqHt8o?kY$#@o*%a~P(;Q1hfP*9-@4N!5`y5L#6?eNW(XYug4s zmtwlr`N~Bun=EOqwz`OuV=ysEwiP$j7`+RV!LJ#y*4YLlVR%{#ex)1-L|RS=&GoZ! z1%KT2T14Ktpa+_==9FDQj5b(*FO|!pneAZYbCRCXTa&!4$Z0XJIA3ozi@l1ThIk9| zj<3uIPjFF@iMj4+eS;)Xt6WT8Ltw-)Jf$oP?lrx~TfUnuPAG|G;IV-HYHYc3Ie*9C z@8;7gMZ>Oyd$Cn_A{wk%ew3+fkO4%yvpkeZ_?zN?XN1-@9+=YtQ z26;|WfqsxFR%aE87F>|e?ZwS{b3+;9ZBO=ZU@3x{T{3nUvJ_&g;SMeXPhfanTDK$;;@rJX=Jv}|dBeAxu6GRBE1l->Vs)zb8 z=YE!9^$bc-P}H;Rp;HbLCKp69xOagOvavVE%^s&9eA8cOu9GMumiOc?ZhXbo)2-loLgnpX5AVUW_c_mpil#VrYU zcjUCkdSZ>Mus=}QktEo&`_3H!Y`Q`GP|2d~r(zX%!F2=3=tst`g}#f;C%ld8G`G}< z2{@eq=d4v4DXPU)B^vIxW&mHbI*4MAS})k}rza46DteALkU0RvLn#*J+SQ`;_h6N= z)1$0V5`j6!T|CxjkdJUoYd}D2uKRWuN=2DS1zY6Qh=+*0z|j=(rfeXxcHV)$*ti^s zO{*KE=q?m(u=k^|kM{Y-BP}qPsA_^8+QoTi)i;nPi$jyS*-9`Pq*-2FTnnoWO1Pn2 ze6x?Xl5Ee|QPiec+kaYJZX8j18qdmP7K+;7wmj^imW2ppHP~O32dn`O(lg?wxzg+x z5NPa^Z*cA9!EqG>PfYchpp}jit{wJh(*npP-yEUK8EROr%p$R%}4Y{<(bXH&2Hcw zZ6w}8`9pi8ra5J~8LAk#*@LgSH8s-x$-^{JHn=mvQxf5KZ&4ptk8buId}`z0vh~GG zb@k5gGT#nYvc*As*^T|m_#iV!5tXtosTp9i7wUy+BD{cY#P(=b0G;8T-ki~~pF=n+ zj152&1ft)0g!R}u?bcA3H~m}9$2MwJ*CFb9=ZodhFRR#P?_JgnTeaxFSCM2ocJu>6 z1j5+Z_hobe$swM{5JrwC3}jc{Lr>EJt+nUWvykQ+t&cj=f!4qvNr*Y7jgy$04!U5p zQrlsdr2%8yVe3&jqIy>t3=dWEMqdtFT>LLdfavWkfD8ooR0mI}wM7MX=93_`lOV3Z#Im!FWn_0-V9@ zO(kK(qg~fleOGtf>P%;Buz7bS>;MLwAaodEvTI&N5$+>WBjuYDJRhSadW5b<#-;r6 z9puZ|;N9gsxYzyRh`IC$iSlv!uZKVUz+#~YHs9aT$hX2xu&zX8J*oWu!*73Wz&o#@ zjXQpu4}Dq@5ZashQ|oiau4qw|C=^a`RUuhMzlFPDi$;KykG{N(+6^yU-TXAUB#`CN zDT4Ha_OTHSlOC<^hq@wEe`7!30lTH@aI^}#^rbyjO@y3e#I3XFAb=V0%5$~t=w#P?PzM%uVfie%=djD>sh=1p zwG}S_MFG~rB$Bkuf}!)#^Nysp0^4C>rL-m7GK<`?*WK8|7E90l%KQsupyXhwqOQB@ zr-ufNWGk#sD{1j^!YvqMfi2X3qTqf6B^xsu!iXa{%uvr0Tb#vb#+%93%?oubN+7R~ zQD=8r10o@2D5(&7z=-S+Dj?CI@PU)5Tz-$db%fEnjDfFk^n%Ds8>oHC>NVs1>tP|g4)(UI@&&zg;NJxf`Tk@Lzr-tIL>l~r?eU$+!A2o||KWES3&62W z$oN*^2x>k;Eav@hzR>>*{r}(e|HZeEpReDq-DF;*pYdVv{`Uq`P7`Y|SkuKR<6z&(G~2!O!}y O<^KSi{{KW_761VD^X_c` literal 534 zcmV+x0_pu9iwFP!00002|LvGTtJE+M$G^(3EO_y~x_Zz<45;u>PagLE$fnaxo@pjd z(k%r)ypwHx2;xP&sDU2JH1nU|B=Bb%<;obea%g(3iZ226aVB_2V=~cWhL7YU+!0%O z{|X;sKjp#;+#&X=TN6~brk|)^w2J1U-_wZKC6a#1s;e<&xOiKayTh+*h}J=?q>m24 z#7I-LIASc&V=+EtFfbyYk>P}8MvPEI%Ey6?a80&Mj9?#?P3&QIrhqS!XJUqd7&3<8 z#!)T;*kcN`)Dp7VGUW_KYWuCxTToqUKnkgjWt}u>8IfRz3|n7hL@d|5o4U9FAJQRrj)^%P`UPp7ti42L#V}Q zfW-75r7NA!@MRJp7IhTqWX0xo@Y(-0EDh4aj+{hZ)|4z0#hxo#8 zo5PVli^C=M_1DIV#W`_?E5rWq5|We}H;3qHmi`DHgaP)?+krK*M)vO-!&`rbBU<+) z!~PW%-dsRtB9Q4o`gA5$UaU=uebo8F%D~eOtdTXczrh%~?MFkafkcS)5qZq;iT%ac YMRi+Lb?cR8cfBP22H7Y58Y>k50089word mid-sentence): the emphasized word is torn out - and glued to the START of the next paragraph, and the remainder of its sentence - is wrongly bolded. Found in the wild 2026-07-11 (burr.com CMMC article) during - the probation audit; reported upstream. - - THESE ASSERTIONS PIN THE BROKEN BEHAVIOR ON PURPOSE: when a trafilatura upgrade - fixes the serializer, this test FAILS — that's the signal to drop the README - known-issue entry and this test, not a regression. +class TestNestedEmphasisSerializedInPlace: + """Nested emphasis (word mid-sentence) stays in its own + sentence. trafilatura <= 2.1.0 tore the emphasized word out and glued it to the START + of the next paragraph, leaving the original sentence broken mid-phrase, and dropped + characters around inline ("i.e." -> "e."). Found in the wild 2026-07-11 (burr.com + CMMC article) during the probation audit, reported as adbar/trafilatura#882, fixed in + trafilatura 2.2.0 (`_md_wrap` keeps flanking whitespace outside the marker, so the + closing delimiter is right-flanking per CommonMark) and verified here 2026-08-11. + + This started life as a canary pinning the BROKEN behavior so an upgrade would fail + loudly. That fired as designed; the assertions are now flipped to pin the CORRECT + serialization, so the class stays a permanent regression guard against re-breakage. """ - def test_nested_emphasis_displacement_still_present( + def test_emphasized_word_stays_in_its_sentence( + self, fixture_bytes: Callable[[str], bytes] + ) -> None: + result = extract_content( + fixture_bytes("emphasis_mangle.html"), url="https://example.com/repro" + ) + assert result is not None + # the word stays put, nested emphasis collapses to a single ***…*** run + assert "plan to ***verify*** contractor compliance" in result.markdown + # and it is NOT glued to the head of the next paragraph + assert "*verify*Starting November 10" not in result.markdown + + def test_inline_emphasis_does_not_drop_characters( self, fixture_bytes: Callable[[str], bytes] ) -> None: result = extract_content( fixture_bytes("emphasis_mangle.html"), url="https://example.com/repro" ) assert result is not None - # "verify" is displaced onto the start of the NEXT paragraph - assert "*verify*Starting November 10" in result.markdown - # and its original sentence is left broken mid-phrase - assert "plan to \n" in result.markdown or "plan to\n" in result.markdown + # "i.e." survives intact — 2.1.0 rendered this as "e." + assert "*i.e.*" in result.markdown class TestDegenerateInput: diff --git a/tests/test_probation_pages.py b/tests/test_probation_pages.py index 9837d68..983df06 100644 --- a/tests/test_probation_pages.py +++ b/tests/test_probation_pages.py @@ -19,6 +19,14 @@ substantial main body with no figures. Calibrated at creation: silent under the shipped guards, warns if the cluster floor drops to 1 (so it genuinely exercises the arming logic). The live class stays covered by evals/corpus.json (linkedin_fp). + +REGENERATED 2026-08-11 for trafilatura 2.2.0 — `scripts/gen_article_peripheral_fixture.py` +is the reproducible source. The 07-16 build padded its article body with 14 byte-identical +

tags; 2.2.0 deduplicates identical blocks, so the body collapsed from 1,973 to 166 +chars and the page started tripping the STUB guard. Meanwhile the REAL page it models +moved the opposite way (1,827 -> 13,560 chars, +642%), so the synthetic had stopped +modelling reality in both magnitude and direction. The rebuild uses varied prose, which +is what a real article has, and re-verifies every clause of the contract above. """ import gzip @@ -26,7 +34,7 @@ import pytest -from tearsheet.content import assess_extraction, extract_content +from tearsheet.content import _MONEY, assess_extraction, extract_content, html_to_text PROBATION = Path(__file__).parent / "fixtures" / "probation" @@ -51,7 +59,31 @@ class TestQuoPricing: def test_the_original_failure_still_warns(self) -> None: price, column, wall = price_warned(load("quo_pricing")) assert price, "quo dropped 20/24 prices — the guard MUST fire" - assert column, "quo collapsed its 3-column matrix — the guard MUST fire" + + def test_quo_is_still_substantively_broken(self) -> None: + """The failure is unchanged; only the COLUMN SIGNATURE moved (2026-08-11). + + Through trafilatura 2.1.0 quo's 3-column grid flattened into runs of identical + consecutive rows (`Unlimited* / Unlimited* / Unlimited*`), which is what + _has_collapsed_columns keys on, so both the price AND column guards fired. + 2.2.0 emits that cell ONCE instead of three times: the page is just as lossy + (still 4 of 24 figures, still 1 of 3 plan names) but the repetition the guard + detects is gone, so only the price guard fires now. + + The column assertion was retired here rather than deleted quietly: a full-cache + replay (194 pages, both versions) showed the collapsed-column guard now fires on + NO page it used to catch, i.e. it has zero real-page coverage under 2.2.0. That + gap is tracked as its own work item; this test pins the substantive loss so the + page can never go silent while the signature question is open. + """ + extracted = extract_content(load("quo_pricing")) + assert extracted is not None + page_figures = set(_MONEY.findall(html_to_text(load("quo_pricing")))) + kept = page_figures & set(_MONEY.findall(extracted.markdown)) + assert len(page_figures) >= 20, "fixture must still carry the full price grid" + assert len(kept) / len(page_figures) < 0.5, "quo must still read as a major drop" + plans = [name for name in ("Starter", "Business", "Scale") if name in extracted.markdown] + assert len(plans) < 3, "plan-name attribution is still lossy" class TestSmithAiPricing: @@ -89,10 +121,56 @@ def test_the_false_positive_stays_dead(self) -> None: assert not price, "peripheral real figures must not trip the guard (v0.1.3 fix)" assert not wall + def test_the_page_is_completely_silent(self) -> None: + """No warning of ANY kind — this is the false-positive fixture. + + The old assertions only checked the price and column guards, so when the body + collapsed under trafilatura 2.2.0 the page began emitting a STUB warning and + every test still passed. Assert total silence so any future guard can't quietly + start firing on the one page that must never warn. + """ + html = load("article_peripheral") + quality = assess_extraction(html, extract_content(html)) + assert quality.warnings == [], f"FP fixture must stay silent, got: {quality.warnings}" + assert not quality.consent_wall and not quality.block_wall + def test_the_article_body_still_extracts(self) -> None: + """Pin the BODY, not just the headline. + + The previous version asserted only `"Join Forces" in markdown` — which the page + TITLE satisfies. That is why a 92% body collapse (1,973 -> 166 chars under 2.2.0) + passed CI unnoticed. These assertions require real body sentences. + """ extracted = extract_content(load("article_peripheral")) assert extracted is not None - assert "Join Forces" in extracted.markdown + assert len(extracted.markdown) > 1_500, "the article body must survive extraction" + assert "surprised analysts" in extracted.markdown + assert "copyleft obligation" in extracted.markdown + + def test_peripheral_figures_are_excluded_but_present_on_the_page(self) -> None: + """The whole point of the fixture: real figures exist, correct extraction drops + them (they are related-content), and the guard stays quiet about it.""" + html = load("article_peripheral") + extracted = extract_content(html) + assert extracted is not None + on_page = set(_MONEY.findall(html_to_text(html))) + assert len(on_page) == 4, f"fixture must carry 4 distinct figures, got {on_page}" + assert not (on_page & set(_MONEY.findall(extracted.markdown))) + + def test_it_genuinely_exercises_the_arming_logic( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Silence must come from the CLUSTER FLOOR, not from the page being toothless. + + Drop the floor to 1 and the guard must fire — otherwise this fixture would pass + the FP test for the wrong reason and stop protecting anything. + """ + import tearsheet.content as content_module + + monkeypatch.setattr(content_module, "_MONEY_MIN_CLUSTERED", 1) + html = load("article_peripheral") + quality = assess_extraction(html, extract_content(html)) + assert any("price" in w for w in quality.warnings) @pytest.mark.parametrize( From c8cbc7e75eb988988081836e485dc68273304a2b Mon Sep 17 00:00:00 2001 From: nathan nelson Date: Tue, 11 Aug 2026 13:14:47 -0600 Subject: [PATCH 2/2] =?UTF-8?q?evals:=20live=20trust=20run=20on=20trafilat?= =?UTF-8?q?ura=202.2.0=20=E2=80=94=20GREEN?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Standing protocol: re-run the eval after every dependency upgrade. This clears BOTH outstanding debts — the 2.2.0 upgrade trigger, and the 07-24 stub guard that shipped without a run (evals were last run 07-16, 26 days stale). All 9 gates PASS. 40 items + a 45-scrape burst (40 ok, 5 honest refusals, 0 FAIL, 0 exceptions). Zero fabrications, zero cache poison, zero unhandled exceptions, baselines stable across the 5 pinned immutables. The single FAIL is wiki_idaho's $74,900 infobox figure — the documented single-figure floor, the same tolerated item as the 2026-07-16 run 2, so it is stable rather than a regression. No guard can arm on one figure without spamming false positives. linkedin_fp passes live (0/2 peripheral figures shown, no warning), so the false-positive protection the regenerated fixture pins offline also holds against the real page. This report is stamped `tearsheet 0.1.5` — the first one that is. The editable install had gone stale at 0.1.0, which silently mislabeled both 07-16 reports. Co-Authored-By: Claude Fable 5 --- evals/reports/2026-08-11/report.md | 73 ++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 evals/reports/2026-08-11/report.md diff --git a/evals/reports/2026-08-11/report.md b/evals/reports/2026-08-11/report.md new file mode 100644 index 0000000..accb1e5 --- /dev/null +++ b/evals/reports/2026-08-11/report.md @@ -0,0 +1,73 @@ +# Tearsheet Trust Report — 2026-08-11 13:13 + +**VERDICT: GREEN** + +- tearsheet 0.1.5 · trafilatura 2.2.0 · httpx 0.28.1 · lxml 6.1.1 · python 3.14.2 +- corpus 2026-07-16.2 (sha fe68ddd7bb2c) · 40 items scored · burst: 45 scrapes across 3 domains: {'ok': 40, 'honest_refusal': 5, 'FAIL': 0, 'exception': 0} + +## Gates + +| gate | status | detail | +|---|---|---| +| fabrication (0 tolerated) | PASS | 0 pages fabricated figures | +| guard calibration (0 regressions) | PASS | known-bad warns, known-good silent | +| silent omission (<=1) | PASS | 1 pages omitted without warning | +| minor omissions within retention floor (<=3) | PASS | tailscale: ['$5'] missing (80% retained — within the document | +| unhandled exceptions (0) | PASS | 0 | +| cache poison (0) | PASS | none served | +| timeouts (<=2) | PASS | 0 | +| baseline drift (0) | PASS | stable | +| hard failures (<=1 beyond the above) | PASS | wiki_idaho: SILENT omission: ['$74,900'] | + +## Items + +| id | category | status | detail | +|---|---|---|---| +| quo | pricing_grid | warn-correct | warning riding with content; 23 figures flagged | +| slack | pricing_grid | warn-correct | warning riding with content; 7 figures flagged | +| notion | pricing_grid | warn-correct | warning riding with content; 3 figures flagged | +| zapier | pricing_grid | warn-correct | warning riding with content; 9 figures flagged | +| tailscale | pricing_grid | minor-omission | ['$5'] missing (80% retained — within the documented retention floor) | +| zoom | pricing_tabbed | warn-correct | warning riding with content; 6 figures flagged | +| ringcentral | pricing_tabbed | pass | 4/4 figures shown | +| heyrosie | plan_cards | pass | 5/5 figures shown | +| calendly | plan_cards | pass | 3/5 figures shown | +| basecamp | plan_cards | pass | 3/3 figures shown | +| hetzner | pricing_non_usd | pass | clean | +| scaleway | pricing_non_usd | warn-correct | warning riding with content; 5 figures flagged | +| giffgaff | pricing_non_usd | pass | clean | +| linkedin_fp | article_peripheral | pass | 0/2 figures shown | +| rfc9110 | pinned_immutable | pass | clean | +| rfc2616_txt | pinned_immutable | pass | clean | +| gutenberg_alice | pinned_immutable | pass | clean | +| w3c_css2 | pinned_immutable | pass | clean | +| gh_raw_readme | pinned_immutable | pass | clean | +| wiki_idaho | reference_table | **FAIL** | SILENT omission: ['$74,900'] | +| fed_h15 | data_table | pass | clean | +| pg_essay | control_prose | pass | clean | +| py_docs | control_docs | pass | clean | +| fastapi_docs | control_docs | pass | clean | +| guardian | consent_heavy | pass | content served (3951 chars) | +| lemonde | consent_heavy | pass | content served (3788 chars) | +| zeit | consent_heavy | pass | content served (3987 chars) | +| ecfr | botwall | walled | honest refusal | +| dodcio | botwall | walled | honest refusal | +| crunchbase | botwall | walled | honest refusal | +| todomvc | spa_shell | documented-limit | tiny output (300 chars) — the known custom-mount class | +| aozora | charset_shift_jis | pass | clean | +| wiki_ja | charset_utf8_ja | pass | clean | +| gh_api | json_endpoint | pass | JSON pretty-printed | +| berkshire_pdf | pdf_figures | pass | pdf text extracted; 3 money figures visible | +| irs_p15 | pdf_tables | pass | pdf text extracted; 7 money figures visible | +| map_uv | map | pass | 50 urls mapped | +| map_fastapi | map | pass | 50 urls mapped | +| crawl_ruff | crawl | pass | crawl: docs.astral.sh pages: 5 errors: 0 skipped(robots/dupe/type): 0 | +| search_smoke | search | pass | results returned | + +## Reading this report + +- `pass` — figures/content verified against the independent oracle. +- `warn-correct` — the tool flagged its own extraction; the warning was warranted. +- `walled` — honest refusal (bot wall / consent wall / HTTP error reported as such). +- `FAIL` — a trust property was violated; see evidence/ for the raw bytes. +- Every item's raw evidence is under `evidence//` for re-adjudication. \ No newline at end of file