feat: stub-extraction guard — the roster/directory class

CHANGELOG.md

@@ -12,6 +12,17 @@ failure. For the trust model and the evaluation harness behind those calls, see
 ## [Unreleased]
 
 ### Added
+- **Stub-extraction guard** — the roster/directory class. A page carrying real content
+  (≥ 1,000 chars of visible text) whose extraction is a stub (< 500 chars) retaining under
+  10% now emits a `warning:` line. Closes a gap where a page with no prices, no table, and
+  no wall gave every existing guard nothing to arm on: `idahoapexaccelerator.com/our-team`
+  returned a marketing sentence and dropped all four staff entries, `sbir.gov/awards`
+  returned only the government-site banner, and a NOAA SBIR archive kept 1 of 3+ stories —
+  all silently. Deliberately **not** the bare yield ratio rejected in v0.1.2 (it fires on
+  neither real pricing failure and false-positives on nav-heavy pages); pairing retention
+  with an absolute output floor is what separates the two groups. Calibrated by replaying
+  all 162 content-bearing pages in a live cache: fires on 5 (3.1%), every one a genuine
+  omission, and stays silent on the LinkedIn false-positive page and every good pricing page.
 - MIT `LICENSE`, project URLs, and a GitHub Actions CI gate (`ruff` + `mypy` + `pytest`,
   matrixed over Python 3.12 and 3.13).
 - Synthetic false-positive fixture so the probation corpus republishes no real individual's page.

README.md

@@ -126,6 +126,17 @@ a research tool for reading the public web — not for evading paywalls or bot d
   reported and never cached, and previously poisoned cache rows are evicted, not replayed.
   Note `--raw` deliberately uses the plain fetch, not the browser: a rendered DOM can be *worse*
   (on smith.ai the consent overlay replaced the pricing table the raw fetch still carried).
+- **Listings and rosters can come back as a boilerplate stub.** A page with no prices, no
+  table, and no wall gave the earlier guards nothing to arm on, so a directory page could
+  return its marketing sentence and silently drop every row. Measured 2026-07-24 across the
+  live cache: `idahoapexaccelerator.com/our-team` returned a 161-char blurb from a 1,994-char
+  roster and dropped all four staff entries; `sbir.gov/awards` returned only the "official
+  website of the United States government" banner; a NOAA SBIR news archive kept 1 of 3+
+  stories. Since **v0.1.5** a `warning:` line fires when an extraction is a stub (< 500 chars)
+  of a content-bearing page (>= 1,000 chars of visible text) it retained under 10% of.
+  It is deliberately **not** a bare yield ratio — that fires on neither real pricing failure
+  and false-positives on nav-heavy pages; the absolute floor is what separates them. Replayed
+  over the whole cache it fires on 3.1% of pages, every one a genuine omission.
 - **Figures no guard can see**: prices rendered by JS into tabs/RSC payloads (dialpad),
   served as literal `"null"` placeholders (aircall), or drawn in images — invisible to any
   non-interactive fetch, tearsheet included. And a page carrying a **single** figure sits

src/tearsheet/content.py

@@ -68,6 +68,40 @@
 _PRICING_TITLE = re.compile(r"pricing|plans? and pricing|tarif|preise", re.I)
 _MONEY_MIN_TITLED = 3
 
+# Stub-extraction guard (the roster/directory class, 2026-07-24 audit). A content-bearing
+# page whose extraction is boilerplate: idahoapexaccelerator.com/our-team returned a
+# 161-char marketing sentence and dropped every consultant name; sbir.gov/awards returned
+# only the "official website of the United States government" banner. Neither carries a
+# price, a table, or a wall, so no other guard can arm.
+#
+# NOT the bare yield ratio that was measured and rejected — that fires on neither real
+# pricing failure (smith.ai 13%, quo 21% retention here). Retention is only half the
+# condition; the other half is an absolute stub-sized output. Calibrated by replaying
+# every content-bearing page in the live cache (162 pages; fires on 5, i.e. 3.1%):
+#
+#            page                 visible     md   ratio   verdict
+#            sbir.gov/awards       10,698    421    3.9%   FIRE   (only the .gov banner)
+#            noaa SBIR archive      6,373    361    5.7%   FIRE   (1 of 3+ stories kept)
+#            apex/our-team          1,994    161    8.1%   FIRE   (marketing line only)
+#            nomic.ai/pricing       3,043    259    8.5%   FIRE   (marketing, zero prices)
+#            pocatello Fees         2,134    206    9.7%   FIRE   (already price-warned)
+#            eur-lex search        11,022    571    5.2%   silent (real result list)
+#            dronedominance        16,661  1,553    9.3%   silent
+#            linkedin FP           17,890  1,827   10.2%   silent (MUST stay silent)
+#            smith.ai              10,961  1,452   13.2%   silent
+#            heyrosie              16,218  2,263   14.0%   silent
+#            quo                   17,961  3,716   20.7%   silent
+#
+# Measure with a FRESH extract_content, not a cached `markdown` column — stored rows may
+# have been written with include_links=True, which inflates length by ~70% and hid the
+# noaa failure during the first pass of this calibration.
+#
+# The markdown floor is what separates the two groups; do NOT drop it and leave a bare
+# ratio, and do NOT raise it past ~570 (eur-lex, a legitimate terse result list).
+_STUB_MIN_PAGE_TEXT = 1_000
+_STUB_MAX_MARKDOWN = 500
+_STUB_MAX_RATIO = 0.10
+
 # A collapsed table column reads as a run of identical consecutive rows
 # (quo: `Unlimited* / Unlimited* / Unlimited*`). One run happens naturally; two is a pattern.
 _COLLAPSE_RUN_LEN = 3
@@ -137,10 +171,14 @@ def _has_collapsed_columns(markdown: str) -> bool:
 def assess_extraction(html: bytes, extracted: ExtractedContent | None) -> ExtractionQuality:
     """Judge an extraction against the page it came from.
 
-    Catches the two failure classes that reached real research (2026-07-14): a consent
-    banner served as content, and a pricing table whose figures never survived. Deliberately
-    NOT a markdown/text yield ratio — measured against both real failures, a ratio fires on
-    neither (smith.ai yields 34%, quo 21%) and would only add false positives.
+    Catches the failure classes that reached real research: a consent banner served as
+    content and a pricing table whose figures never survived (2026-07-14), plus a
+    content-bearing page whose extraction is a boilerplate stub (2026-07-24).
+
+    A BARE markdown/text yield ratio is still deliberately absent — measured against the
+    pricing failures it fires on neither and only adds false positives. The stub guard
+    pairs retention with an absolute output floor, which is what separates the real
+    failures from legitimately nav-heavy pages; see the calibration table above it.
     """
     quality = ExtractionQuality()
     if extracted is None:
@@ -164,6 +202,17 @@ def assess_extraction(html: bytes, extracted: ExtractedContent | None) -> Extrac
             "independently."
         )
 
+    if (
+        len(page_text) >= _STUB_MIN_PAGE_TEXT
+        and len(markdown) < _STUB_MAX_MARKDOWN
+        and len(markdown) / len(page_text) < _STUB_MAX_RATIO
+    ):
+        quality.warnings.append(
+            f"extraction is a {len(markdown)}-char stub of a page holding ~{len(page_text)} "
+            "chars of visible text — names, rows, or listings on this page are probably "
+            "missing entirely. Use raw=true, or fetch independently."
+        )
+
     on_page = set(_MONEY.findall(page_text))
     title_armed = (
         extracted.title is not None

tests/test_quality.py

@@ -19,9 +19,10 @@
 more than 3 near each other). A page-wide count cannot tell those apart; a
 cluster can, with margin on both sides (3 vs 5).
 
-An earlier design used a markdown/visible-text yield ratio; it is kept out on
+An earlier design used a BARE markdown/visible-text yield ratio; it stays out on
 purpose because it provably fires on neither bad page (smith.ai yields 34%,
-quo 21%) while risking false positives on legitimately terse pages.
+quo 21%) while risking false positives on legitimately terse pages. The
+low-yield guard added 2026-07-24 is NOT that design — see TestLowYield.
 """
 
 from tearsheet.content import ExtractedContent, assess_extraction, html_to_text
@@ -279,3 +280,58 @@ def test_single_repeated_triple_is_tolerated(self) -> None:
         markdown = "Included\n\nYes\n\nYes\n\nYes\n\n" + "Real prose about the product. " * 30
         quality = assess_extraction(page("Included"), extracted(markdown))
         assert not any("column" in w for w in quality.warnings)
+
+
+class TestLowYield:
+    """The roster/directory class (2026-07-24 audit): a content-bearing page whose
+    extraction is a boilerplate stub, with no price, table, or wall for any other
+    guard to arm on.
+
+    Calibrated by replaying every content-bearing page in the live cache (162 pages;
+    the guard fires on 5, i.e. 3.1%). Retention alone is NOT the signal — nav-heavy
+    pages legitimately retain under 10% (dronedominance 9.3% is a HEALTHY extraction).
+    The discriminator is retention AND an absolute stub-sized output:
+
+        page                      visible     md   ratio   verdict
+        sbir.gov/awards            10,698    421    3.9%   FIRE  (only the .gov banner)
+        noaa SBIR archive           6,373    361    5.7%   FIRE  (1 of 3+ stories kept)
+        idahoapex /our-team         1,994    161    8.1%   FIRE  (only a marketing line)
+        atlas.nomic.ai/pricing      3,043    259    8.5%   FIRE  (marketing, zero prices)
+        pocatello.gov/812/Fees      2,134    206    9.7%   FIRE  (already price-warned)
+        eur-lex search             11,022    571    5.2%   silent (real result list)
+        dronedominance             16,661  1,553    9.3%   silent (healthy)
+        linkedin post (the FP)     17,890  1,827   10.2%   silent (MUST stay silent)
+        smith.ai receptionists     10,961  1,452   13.2%   silent
+        heyrosie.com               16,218  2,263   14.0%   silent
+        quo.com/pricing            17,961  3,716   20.7%   silent
+
+    Measure with a FRESH extract_content, never a cached `markdown` column: stored rows
+    may carry include_links=True, which inflated noaa to 614 chars and hid it as
+    "healthy" during the first pass of this calibration.
+    """
+
+    def test_stub_extraction_from_a_content_bearing_page_warns(self) -> None:
+        """idahoapexaccelerator.com/our-team: 1,994 chars of roster -> 161-char blurb."""
+        body = "Meet the team. " + "Name Title City. " * 110
+        stub = "Meet the passionate team members who are committed to excellence."
+        quality = assess_extraction(page(body), extracted(stub))
+        assert any("stub" in w for w in quality.warnings)
+
+    def test_low_ratio_with_substantial_output_stays_silent(self) -> None:
+        """The noaa/dronedominance class: nav-heavy page, healthy article extracted."""
+        body = "Site navigation. " * 400
+        real = "A genuine article body. " * 40  # ~960 chars, well past a stub
+        quality = assess_extraction(page(body + real), extracted(real))
+        assert not any("stub" in w for w in quality.warnings)
+
+    def test_short_page_extracted_faithfully_stays_silent(self) -> None:
+        """A genuinely brief page is not a failure — high ratio, small output."""
+        body = "A short notice. It says one thing and stops."
+        quality = assess_extraction(page(body), extracted(body))
+        assert not any("stub" in w for w in quality.warnings)
+
+    def test_tiny_page_cannot_arm_the_guard(self) -> None:
+        """Below the page-substance floor there is nothing to have missed."""
+        body = "Hello. " * 20  # ~140 chars of visible text
+        quality = assess_extraction(page(body), extracted("Hello."))
+        assert not any("stub" in w for w in quality.warnings)