chore(hygiene): make the repo's self-checks able to see what they claim

.github/workflows/ci.yml

@@ -10,7 +10,11 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.12", "3.13"]
+        # 3.14 is here because it is what the local .venv actually runs — every local
+        # gate run and every eval report to date has used an interpreter CI never
+        # exercised. 3.12 stays as the floor (`requires-python = ">=3.12"`, and both
+        # ruff's target-version and mypy's python_version pin it).
+        python-version: ["3.12", "3.13", "3.14"]
     steps:
       # v6/v5 targeted Node 20, which runners now force onto Node 24 with a
       # deprecation warning; v6 is the Node-24-native pair.

CHANGELOG.md

@@ -11,6 +11,49 @@ failure. For the trust model and the evaluation harness behind those calls, see
 
 ## [Unreleased]
 
+### Changed — repo hygiene (2026-08-11 audit, third pass)
+- **One definition for the bot-wall phrases.** `content._BLOCK_PHRASES` said "keep phrases
+  in sync with `fetch._BLOCK_MARKERS`" and they had drifted anyway: `content` carried
+  `"checking your browser"` and the looser `"attention required"`, `fetch` carried neither,
+  so a sub-30 KB *"Checking your browser before accessing…"* interstitial was invisible to
+  the raw-body check and depended entirely on the post-extraction backstop — which only
+  fires when the extraction lands under 600 chars. `fetch._BLOCK_MARKERS` is now derived
+  from the one list, with a test asserting they cannot diverge. A comment cannot enforce a
+  shared list; one definition can.
+- **The eval's guard-calibration gate tests the guard it names again.** `expect_warning`
+  marks a known-bad *pricing* page, but it was satisfied by ANY `warning:` line, so once the
+  stub guard shipped in v0.1.5 a stub warning could satisfy a gate written for the price
+  guard — one of the three RED-triggering gates getting weaker without anyone changing it.
+  It now requires a price warning. The silent-omission gate deliberately still accepts any
+  warning: a page that stub-warns while dropping figures was **not** silent, so scoring it
+  as a silent omission would be false.
+- **The eval's own oracle is no longer blind where the tool was.** `run_eval.MONEY` carried
+  the same four gaps the tool's matcher did, so it under-counted the figures actually on the
+  page and genuine omissions could not be scored. Re-derived from the same requirements but
+  still written separately — importing the tool's matcher would recreate the shared-fate
+  flaw the harness exists to avoid. A test asserts the two implementations agree.
+- **`run_eval.py` cannot silently overwrite a report.** The output directory is the run date,
+  so a second run on the same day clobbered the first one's `report.md` in place. It now
+  moves the earlier run to `<date>-runN` first, matching the convention already on disk.
+  This is not hypothetical: two runs were needed on 2026-08-11 and the first had to be
+  rescued by hand.
+- **The trust instrument now has tests.** `evals/run_eval.py` decides whether the tool can be
+  trusted and had no coverage at all — including for the directory-renaming function above.
+  `tests/test_eval_harness.py` covers the oracle matcher, report preservation, and wall
+  detection. The network run stays manual by design.
+- **`check_consistency.py` sees four things it used to miss**, each falsified in both
+  directions before commit: single-digit counts (`\d{2,4}` made README's "6 tests" invisible);
+  which SUITE a count claims (the playwright count is now verified against
+  `pytest -m playwright` rather than compared to the default suite or skipped); an undated
+  `## [X.Y.Z] — TBD` CHANGELOG section passing as released; and **installed distribution
+  metadata**, which had been stale at 0.1.0 for three releases and silently mislabelled every
+  eval report, since `run_eval.py` stamps reports with the metadata version.
+- **README's Development command is the actual gate.** It omitted `scripts` from the lint
+  target and left out `check_consistency.py` entirely, so following the README could push a
+  failure CI would then catch.
+- **CI tests Python 3.14**, the interpreter the local venv actually runs — every local gate
+  run and every eval report to date used a version CI never exercised. 3.12 stays the floor.
+
 ### Changed
 - **Requires trafilatura >= 2.2**, which fixes the nested-emphasis serializer bug tearsheet
   reported as [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882)

README.md

@@ -11,7 +11,7 @@ no telemetry.
 
 > *tearsheet (n.): a page torn from a publication and filed as proof it ran.*
 
-**Trust status:** qualified for heavy usage 2026-07-16 — 275 tests, a falsifiable live
+**Trust status:** qualified for heavy usage 2026-07-16 — 292 tests, a falsifiable live
 eval harness (verdict GREEN), and zero fabrications across the tool's entire recorded
 history. Its documented failure mode is *omission*, and the guards exist to make every
 omission loud. See [Trust](#trust).
@@ -169,7 +169,7 @@ a research tool for reading the public web — not for evading paywalls or bot d
 
 "Can it be trusted for heavy usage?" is a measurement here, not a feeling.
 
-- **Offline suite (275 tests, runs in the gate)**: guard boundary pins, cache-poisoning
+- **Offline suite (292 tests, runs in the gate)**: guard boundary pins, cache-poisoning
   regressions, truncation honesty, charset torture, structure torture, adversarial
   robustness — enforced fully offline by a loopback-only socket guard. The five REAL
   pages that defined the tool's probation (quo, smith.ai, dialpad, heyrosie, a LinkedIn
@@ -200,13 +200,19 @@ for figures you'll quote, treat a suspiciously small extraction of a rich page a
 ## Development
 
 ```bash
-.venv/bin/ruff check src tests && .venv/bin/mypy && .venv/bin/python -m pytest
+.venv/bin/ruff check src tests scripts && .venv/bin/mypy && \
+  .venv/bin/python -m pytest && .venv/bin/python scripts/check_consistency.py
 ```
 
-TDD throughout; the default suite (275 tests) runs entirely offline — `httpx.MockTransport`,
+That is the whole gate, and it is exactly what CI runs — previously this line omitted
+`scripts` from the lint target and left out the consistency check entirely, so following
+the README could push a failure CI would catch.
+
+TDD throughout; the default suite (292 tests) runs entirely offline — `httpx.MockTransport`,
 fixture HTML, and a conftest socket guard that fails any test reaching for a non-loopback
-address. Extras: `pytest -m playwright` (real chromium, local server), `pytest -m live`
-(real network). The live trust evaluation lives in `evals/` (see [Trust](#trust)).
+address. Extras: `pytest -m playwright` (real chromium, local server) and `pytest -m live`
+(real network — the marker is wired up but currently has no tests; the live checks live in
+`evals/`). The live trust evaluation lives in `evals/` (see [Trust](#trust)).
 
 A note on `tests/fixtures/probation/`: four fixtures are real captured commercial pages
 (the tool's original documented failures, pinned forever); the article-peripheral fixture

evals/run_eval.py

@@ -40,7 +40,22 @@
 REPO = EVALS_DIR.parent
 sys.path.insert(0, str(REPO / "src"))
 
-MONEY = re.compile(r"[$€£][0-9][0-9,]*(?:\.[0-9]{2})?")
+# The oracle's money matcher. DELIBERATELY a separate definition from
+# tearsheet.content — importing the tool's matcher would recreate exactly the
+# shared-fate flaw this harness exists to avoid (an oracle that is blind wherever the
+# tool is blind cannot detect an omission).
+#
+# But independence is not an excuse for being WORSE. Until 2026-08-11 this carried the
+# same four blind spots the tool's matcher did, so the oracle under-counted the figures
+# actually on the page and genuine omissions could not be scored: currency split from
+# its digits by markup, the `19,99 €` suffix form, `$1.1` truncated to `$1`, and a
+# trailing comma captured into the figure. Re-derived here from the same requirements,
+# written separately on purpose — if these two ever disagree, that disagreement is
+# itself a finding worth reading.
+_ORACLE_AMOUNT = r"[0-9](?:[0-9.,]*[0-9])?"
+MONEY = re.compile(
+    rf"(?P<sym>[$€£])\s?(?P<amt>{_ORACLE_AMOUNT})|(?P<amt2>{_ORACLE_AMOUNT})\s?(?P<sym2>[€£])"
+)
 ITEM_TIMEOUT_S = 120
 PACING_S = 1.5
 
@@ -64,7 +79,36 @@ def oracle_text(html: bytes) -> str:
 
 
 def money_set(text: str) -> set[str]:
-    return set(MONEY.findall(text))
+    """Distinct money figures, canonicalised symbol-first so `$ 49`, `$49` and `49 €`
+    are comparable across the page body and the returned string."""
+    figures = set()
+    for match in MONEY.finditer(text):
+        symbol = match.group("sym") or match.group("sym2")
+        amount = match.group("amt") if match.group("amt") is not None else match.group("amt2")
+        figures.add(f"{symbol}{amount}")
+    return figures
+
+
+def _preserve_existing_report(report_dir: Path) -> None:
+    """Never silently overwrite a report from earlier the same day.
+
+    The report directory is the run date, so a second run on the same day used to
+    clobber the first one's report.md in place — and a report is evidence, not a
+    scratch file. On 2026-08-11 two runs were needed (one for a dependency upgrade,
+    one after a guard change) and the first had to be rescued by hand before the second
+    could start.
+
+    The existing run is moved aside to `<date>-runN`, matching the convention already
+    on disk from 2026-07-16, so the plain date always holds the LATEST run and nothing
+    is ever lost.
+    """
+    if not (report_dir / "report.md").exists():
+        return
+    n = 1
+    while (archived := report_dir.with_name(f"{report_dir.name}-run{n}")).exists():
+        n += 1
+    report_dir.rename(archived)
+    print(f"preserved the earlier run as {archived.name}/")
 
 
 @dataclass
@@ -77,6 +121,7 @@ class ItemResult:
     fabricated: set[str] = field(default_factory=set)
     silently_omitted: set[str] = field(default_factory=set)
     warned: bool = False
+    price_warned: bool = False
     retried: bool = False
 
 
@@ -169,7 +214,20 @@ async def check_content(self, item: dict, r: ItemResult) -> None:
 
         body = self.producing_body(item["url"])
         shown = self.shown_body(out)
+        # `warned` = the user was told SOMETHING is off, whatever the guard. That is the
+        # right notion for the silent-omission gate below: a page that stub-warns while
+        # also dropping figures was not silent, so scoring it as a silent omission would
+        # be false. Deliberately unchanged.
         r.warned = "warning:" in out
+        # `price_warned` is narrower, and the guard-calibration gate needs the narrow one.
+        # `expect_warning` marks a known-bad PRICING page (quo), so it means "the
+        # dropped-price guard fired" — but it was satisfied by ANY warning, so after the
+        # stub guard shipped in v0.1.5 a stub warning could satisfy a gate written to test
+        # the price guard. That is one of the three RED-triggering gates getting quietly
+        # weaker without anyone changing it. (2026-08-11 audit.)
+        r.price_warned = any(
+            "price" in line for line in out.splitlines() if line.startswith("warning:")
+        )
         oracle_figs = money_set(oracle_text(body)) if body else set()
         shown_figs = money_set(shown)
 
@@ -198,12 +256,10 @@ async def check_content(self, item: dict, r: ItemResult) -> None:
         if r.fabricated:
             r.status = "FAIL"
             r.detail = f"FABRICATED figures: {sorted(r.fabricated)}"
-        elif item.get("expect_warning") and not r.warned and truly_missing:
+        elif item.get("expect_warning") and not r.price_warned and truly_missing:
             r.status = "FAIL"
-            r.detail = "known-bad page: figures missing and no warning fired"
-        elif item.get("expect_no_price_warning") and any(
-            "price" in line for line in out.splitlines() if line.startswith("warning:")
-        ):
+            r.detail = "known-bad pricing page: figures missing and no PRICE warning fired"
+        elif item.get("expect_no_price_warning") and r.price_warned:
             r.status = "FAIL"
             r.detail = "false-positive price warning on a page that must stay silent"
         elif r.silently_omitted and (retained_ratio >= 0.5 and len(r.silently_omitted) <= 2):
@@ -501,6 +557,7 @@ async def main() -> None:
     args = parser.parse_args()
 
     report_dir = EVALS_DIR / "reports" / datetime.now().strftime("%Y-%m-%d")
+    _preserve_existing_report(report_dir)
     report_dir.mkdir(parents=True, exist_ok=True)
     os.environ["TEARSHEET_HOME"] = str(report_dir / "home")  # isolated cache per run
 

scripts/check_consistency.py

@@ -33,7 +33,9 @@
 # "243 tests", "(243 tests, runs in the gate)", "suite (243 tests)" — any prose
 # claim about how many tests exist. Deliberately broad: a claim this script
 # cannot see is a claim that can rot.
-_README_TEST_CLAIM = re.compile(r"\(?(\d{2,4})\s+tests\b")
+# `\d{2,4}` missed single-digit claims — README:170's "6 tests" was invisible to this
+# script entirely (2026-08-11 audit). A claim this cannot see is a claim that can rot.
+_README_TEST_CLAIM = re.compile(r"\(?(\d{1,5})\s+tests\b")
 _COLLECTED = re.compile(r"(\d+)(?:/\d+)?\s+tests? collected")
 
 
@@ -58,18 +60,55 @@ def check_dunder_version(expected: str) -> list[str]:
 def check_changelog_released(expected: str) -> list[str]:
     path = REPO_ROOT / "CHANGELOG.md"
     text = path.read_text()
-    if re.search(rf"^## \[{re.escape(expected)}\]", text, re.M):
-        return []
+    m = re.search(rf"^## \[{re.escape(expected)}\](.*)$", text, re.M)
+    if m:
+        # A heading alone was enough, so `## [0.1.5] — TBD` or an undated section passed.
+        # A released section carries a real date.
+        if re.search(r"\d{4}-\d{2}-\d{2}", m.group(1)):
+            return []
+        return [
+            f"{path.name}: `## [{expected}]` has no release date "
+            f"(found {m.group(1).strip()!r}) — an undated section is not a released one"
+        ]
     return [
         f"{path.name}: no released `## [{expected}]` section — the current version is "
         "still unreleased, or the section was never added"
     ]
 
 
-def collected_test_count() -> tuple[int | None, list[str]]:
-    """How many tests the DEFAULT suite collects (what the gate and README mean)."""
+def check_installed_metadata(expected: str) -> list[str]:
+    """The INSTALLED distribution's version, which nothing else compares.
+
+    An editable install does not refresh its metadata when pyproject's version changes.
+    This repo's had been stale at 0.1.0 for three releases (2026-08-11 audit) — invisible
+    to every other check here, because `tearsheet.__version__` reads the source file
+    while `importlib.metadata` reads the install. It mattered: `evals/run_eval.py` stamps
+    each trust report with the metadata version, so two reports were labelled with a
+    version that had not existed for weeks. Fix with `pip install -e ".[dev]"`.
+    """
+    try:
+        from importlib.metadata import PackageNotFoundError, version
+    except ImportError:  # pragma: no cover - stdlib since 3.8
+        return []
+    try:
+        installed = version("tearsheet")
+    except PackageNotFoundError:
+        return []  # not installed (e.g. a bare checkout) — not a claim, so not drift
+    if installed != expected:
+        return [
+            f"installed distribution metadata says {installed!r} != pyproject {expected!r} "
+            '— stale editable install; run: pip install -e ".[dev]"'
+        ]
+    return []
+
+
+def collected_test_count(marker: str | None = None) -> tuple[int | None, list[str]]:
+    """How many tests a suite collects. `marker=None` is the DEFAULT (offline) suite."""
+    argv = [sys.executable, "-m", "pytest", "--collect-only", "-q"]
+    if marker:
+        argv += ["-m", marker]
     proc = subprocess.run(  # noqa: S603 - fixed argv, no shell
-        [sys.executable, "-m", "pytest", "--collect-only", "-q"],
+        argv,
         cwd=REPO_ROOT,
         capture_output=True,
         text=True,
@@ -81,19 +120,39 @@ def collected_test_count() -> tuple[int | None, list[str]]:
         if m:
             break
     if not m:
-        return None, ["could not determine collected test count from pytest output"]
+        # "no tests collected (284 deselected)" is a real answer, not a parse failure —
+        # the `live` marker is documented in the README but currently has no tests.
+        if "no tests collected" in proc.stdout:
+            return 0, []
+        label = f" for -m {marker}" if marker else ""
+        return None, [f"could not determine collected test count{label} from pytest output"]
     return int(m.group(1)), []
 
 
-def check_readme_test_counts(actual: int) -> list[str]:
+def check_readme_test_counts(default_count: int, marked_counts: dict[str, int]) -> list[str]:
+    """Every "N tests" claim must match the suite it is actually talking about.
+
+    Claims are not all about the default suite: the README also advertises the
+    real-browser suite. Comparing those against the default collection would be wrong,
+    and simply skipping them would leave them unverifiable — which is how README:178's
+    "6 tests" sat outside this script's reach entirely (2026-08-11 audit).
+    """
     path = REPO_ROOT / "README.md"
     problems = []
     for lineno, line in enumerate(path.read_text().splitlines(), start=1):
+        lowered = line.lower()
+        # Match the pytest INVOCATION (`-m playwright`), not the bare word: README:14
+        # says "a falsifiable live eval harness" in prose, which a substring test
+        # mistook for a claim about the `live` marker.
+        suite = next((m for m in marked_counts if f"-m {m}" in lowered), None)
+        expected = marked_counts[suite] if suite else default_count
+        label = f"the {suite} suite" if suite else "the default suite"
         for m in _README_TEST_CLAIM.finditer(line):
             claimed = int(m.group(1))
-            if claimed != actual:
+            if claimed != expected:
                 problems.append(
-                    f"{path.name}:{lineno}: claims {claimed} tests, pytest collects {actual}"
+                    f"{path.name}:{lineno}: claims {claimed} tests for {label}, "
+                    f"pytest collects {expected}"
                 )
     return problems
 
@@ -103,11 +162,18 @@ def main() -> int:
     problems: list[str] = []
     problems.extend(check_dunder_version(expected))
     problems.extend(check_changelog_released(expected))
+    problems.extend(check_installed_metadata(expected))
 
     actual, count_problems = collected_test_count()
     problems.extend(count_problems)
+    marked_counts: dict[str, int] = {}
+    for marker in ("playwright", "live"):
+        count, marker_problems = collected_test_count(marker)
+        problems.extend(marker_problems)
+        if count is not None:
+            marked_counts[marker] = count
     if actual is not None:
-        problems.extend(check_readme_test_counts(actual))
+        problems.extend(check_readme_test_counts(actual, marked_counts))
 
     if problems:
         print(f"Consistency check FAILED. pyproject.toml version is {expected!r}.")

src/tearsheet/content.py

@@ -50,7 +50,13 @@
 # challenge pages CAN exceed 30 KB once their JS is inlined, sail past that guard,
 # extract to a short wall message, and get cached as "content" for the whole TTL.
 # Same lesson as the consent wall: when the extraction IS the wall, catch it here,
-# regardless of body size. Keep phrases in sync with fetch._BLOCK_MARKERS.
+# regardless of body size.
+#
+# THIS IS THE SINGLE SOURCE OF TRUTH for both checks: fetch._BLOCK_MARKERS is derived
+# from it. It used to say "keep these in sync with fetch" and they had drifted anyway —
+# this list carried "checking your browser" and the looser "attention required", fetch
+# carried neither, so a sub-30 KB "Checking your browser…" interstitial was invisible to
+# the raw-body check. A comment cannot enforce a shared list; one definition can.
 _BLOCK_PHRASES = (
     "complete the captcha",
     "solve the captcha",