diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 565e7dd..1957036 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -10,7 +10,11 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: ["3.12", "3.13"] + # 3.14 is here because it is what the local .venv actually runs — every local + # gate run and every eval report to date has used an interpreter CI never + # exercised. 3.12 stays as the floor (`requires-python = ">=3.12"`, and both + # ruff's target-version and mypy's python_version pin it). + python-version: ["3.12", "3.13", "3.14"] steps: # v6/v5 targeted Node 20, which runners now force onto Node 24 with a # deprecation warning; v6 is the Node-24-native pair. diff --git a/CHANGELOG.md b/CHANGELOG.md index d71f7d3..2b595af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,49 @@ failure. For the trust model and the evaluation harness behind those calls, see ## [Unreleased] +### Changed — repo hygiene (2026-08-11 audit, third pass) +- **One definition for the bot-wall phrases.** `content._BLOCK_PHRASES` said "keep phrases + in sync with `fetch._BLOCK_MARKERS`" and they had drifted anyway: `content` carried + `"checking your browser"` and the looser `"attention required"`, `fetch` carried neither, + so a sub-30 KB *"Checking your browser before accessing…"* interstitial was invisible to + the raw-body check and depended entirely on the post-extraction backstop — which only + fires when the extraction lands under 600 chars. `fetch._BLOCK_MARKERS` is now derived + from the one list, with a test asserting they cannot diverge. A comment cannot enforce a + shared list; one definition can. +- **The eval's guard-calibration gate tests the guard it names again.** `expect_warning` + marks a known-bad *pricing* page, but it was satisfied by ANY `warning:` line, so once the + stub guard shipped in v0.1.5 a stub warning could satisfy a gate written for the price + guard — one of the three RED-triggering gates getting weaker without anyone changing it. + It now requires a price warning. The silent-omission gate deliberately still accepts any + warning: a page that stub-warns while dropping figures was **not** silent, so scoring it + as a silent omission would be false. +- **The eval's own oracle is no longer blind where the tool was.** `run_eval.MONEY` carried + the same four gaps the tool's matcher did, so it under-counted the figures actually on the + page and genuine omissions could not be scored. Re-derived from the same requirements but + still written separately — importing the tool's matcher would recreate the shared-fate + flaw the harness exists to avoid. A test asserts the two implementations agree. +- **`run_eval.py` cannot silently overwrite a report.** The output directory is the run date, + so a second run on the same day clobbered the first one's `report.md` in place. It now + moves the earlier run to `-runN` first, matching the convention already on disk. + This is not hypothetical: two runs were needed on 2026-08-11 and the first had to be + rescued by hand. +- **The trust instrument now has tests.** `evals/run_eval.py` decides whether the tool can be + trusted and had no coverage at all — including for the directory-renaming function above. + `tests/test_eval_harness.py` covers the oracle matcher, report preservation, and wall + detection. The network run stays manual by design. +- **`check_consistency.py` sees four things it used to miss**, each falsified in both + directions before commit: single-digit counts (`\d{2,4}` made README's "6 tests" invisible); + which SUITE a count claims (the playwright count is now verified against + `pytest -m playwright` rather than compared to the default suite or skipped); an undated + `## [X.Y.Z] — TBD` CHANGELOG section passing as released; and **installed distribution + metadata**, which had been stale at 0.1.0 for three releases and silently mislabelled every + eval report, since `run_eval.py` stamps reports with the metadata version. +- **README's Development command is the actual gate.** It omitted `scripts` from the lint + target and left out `check_consistency.py` entirely, so following the README could push a + failure CI would then catch. +- **CI tests Python 3.14**, the interpreter the local venv actually runs — every local gate + run and every eval report to date used a version CI never exercised. 3.12 stays the floor. + ### Changed - **Requires trafilatura >= 2.2**, which fixes the nested-emphasis serializer bug tearsheet reported as [adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882) diff --git a/README.md b/README.md index d59c31c..7a7382e 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ no telemetry. > *tearsheet (n.): a page torn from a publication and filed as proof it ran.* -**Trust status:** qualified for heavy usage 2026-07-16 — 275 tests, a falsifiable live +**Trust status:** qualified for heavy usage 2026-07-16 — 292 tests, a falsifiable live eval harness (verdict GREEN), and zero fabrications across the tool's entire recorded history. Its documented failure mode is *omission*, and the guards exist to make every omission loud. See [Trust](#trust). @@ -169,7 +169,7 @@ a research tool for reading the public web — not for evading paywalls or bot d "Can it be trusted for heavy usage?" is a measurement here, not a feeling. -- **Offline suite (275 tests, runs in the gate)**: guard boundary pins, cache-poisoning +- **Offline suite (292 tests, runs in the gate)**: guard boundary pins, cache-poisoning regressions, truncation honesty, charset torture, structure torture, adversarial robustness — enforced fully offline by a loopback-only socket guard. The five REAL pages that defined the tool's probation (quo, smith.ai, dialpad, heyrosie, a LinkedIn @@ -200,13 +200,19 @@ for figures you'll quote, treat a suspiciously small extraction of a rich page a ## Development ```bash -.venv/bin/ruff check src tests && .venv/bin/mypy && .venv/bin/python -m pytest +.venv/bin/ruff check src tests scripts && .venv/bin/mypy && \ + .venv/bin/python -m pytest && .venv/bin/python scripts/check_consistency.py ``` -TDD throughout; the default suite (275 tests) runs entirely offline — `httpx.MockTransport`, +That is the whole gate, and it is exactly what CI runs — previously this line omitted +`scripts` from the lint target and left out the consistency check entirely, so following +the README could push a failure CI would catch. + +TDD throughout; the default suite (292 tests) runs entirely offline — `httpx.MockTransport`, fixture HTML, and a conftest socket guard that fails any test reaching for a non-loopback -address. Extras: `pytest -m playwright` (real chromium, local server), `pytest -m live` -(real network). The live trust evaluation lives in `evals/` (see [Trust](#trust)). +address. Extras: `pytest -m playwright` (real chromium, local server) and `pytest -m live` +(real network — the marker is wired up but currently has no tests; the live checks live in +`evals/`). The live trust evaluation lives in `evals/` (see [Trust](#trust)). A note on `tests/fixtures/probation/`: four fixtures are real captured commercial pages (the tool's original documented failures, pinned forever); the article-peripheral fixture diff --git a/evals/run_eval.py b/evals/run_eval.py index c9c9fa9..8a84ab3 100644 --- a/evals/run_eval.py +++ b/evals/run_eval.py @@ -40,7 +40,22 @@ REPO = EVALS_DIR.parent sys.path.insert(0, str(REPO / "src")) -MONEY = re.compile(r"[$€£][0-9][0-9,]*(?:\.[0-9]{2})?") +# The oracle's money matcher. DELIBERATELY a separate definition from +# tearsheet.content — importing the tool's matcher would recreate exactly the +# shared-fate flaw this harness exists to avoid (an oracle that is blind wherever the +# tool is blind cannot detect an omission). +# +# But independence is not an excuse for being WORSE. Until 2026-08-11 this carried the +# same four blind spots the tool's matcher did, so the oracle under-counted the figures +# actually on the page and genuine omissions could not be scored: currency split from +# its digits by markup, the `19,99 €` suffix form, `$1.1` truncated to `$1`, and a +# trailing comma captured into the figure. Re-derived here from the same requirements, +# written separately on purpose — if these two ever disagree, that disagreement is +# itself a finding worth reading. +_ORACLE_AMOUNT = r"[0-9](?:[0-9.,]*[0-9])?" +MONEY = re.compile( + rf"(?P[$€£])\s?(?P{_ORACLE_AMOUNT})|(?P{_ORACLE_AMOUNT})\s?(?P[€£])" +) ITEM_TIMEOUT_S = 120 PACING_S = 1.5 @@ -64,7 +79,36 @@ def oracle_text(html: bytes) -> str: def money_set(text: str) -> set[str]: - return set(MONEY.findall(text)) + """Distinct money figures, canonicalised symbol-first so `$ 49`, `$49` and `49 €` + are comparable across the page body and the returned string.""" + figures = set() + for match in MONEY.finditer(text): + symbol = match.group("sym") or match.group("sym2") + amount = match.group("amt") if match.group("amt") is not None else match.group("amt2") + figures.add(f"{symbol}{amount}") + return figures + + +def _preserve_existing_report(report_dir: Path) -> None: + """Never silently overwrite a report from earlier the same day. + + The report directory is the run date, so a second run on the same day used to + clobber the first one's report.md in place — and a report is evidence, not a + scratch file. On 2026-08-11 two runs were needed (one for a dependency upgrade, + one after a guard change) and the first had to be rescued by hand before the second + could start. + + The existing run is moved aside to `-runN`, matching the convention already + on disk from 2026-07-16, so the plain date always holds the LATEST run and nothing + is ever lost. + """ + if not (report_dir / "report.md").exists(): + return + n = 1 + while (archived := report_dir.with_name(f"{report_dir.name}-run{n}")).exists(): + n += 1 + report_dir.rename(archived) + print(f"preserved the earlier run as {archived.name}/") @dataclass @@ -77,6 +121,7 @@ class ItemResult: fabricated: set[str] = field(default_factory=set) silently_omitted: set[str] = field(default_factory=set) warned: bool = False + price_warned: bool = False retried: bool = False @@ -169,7 +214,20 @@ async def check_content(self, item: dict, r: ItemResult) -> None: body = self.producing_body(item["url"]) shown = self.shown_body(out) + # `warned` = the user was told SOMETHING is off, whatever the guard. That is the + # right notion for the silent-omission gate below: a page that stub-warns while + # also dropping figures was not silent, so scoring it as a silent omission would + # be false. Deliberately unchanged. r.warned = "warning:" in out + # `price_warned` is narrower, and the guard-calibration gate needs the narrow one. + # `expect_warning` marks a known-bad PRICING page (quo), so it means "the + # dropped-price guard fired" — but it was satisfied by ANY warning, so after the + # stub guard shipped in v0.1.5 a stub warning could satisfy a gate written to test + # the price guard. That is one of the three RED-triggering gates getting quietly + # weaker without anyone changing it. (2026-08-11 audit.) + r.price_warned = any( + "price" in line for line in out.splitlines() if line.startswith("warning:") + ) oracle_figs = money_set(oracle_text(body)) if body else set() shown_figs = money_set(shown) @@ -198,12 +256,10 @@ async def check_content(self, item: dict, r: ItemResult) -> None: if r.fabricated: r.status = "FAIL" r.detail = f"FABRICATED figures: {sorted(r.fabricated)}" - elif item.get("expect_warning") and not r.warned and truly_missing: + elif item.get("expect_warning") and not r.price_warned and truly_missing: r.status = "FAIL" - r.detail = "known-bad page: figures missing and no warning fired" - elif item.get("expect_no_price_warning") and any( - "price" in line for line in out.splitlines() if line.startswith("warning:") - ): + r.detail = "known-bad pricing page: figures missing and no PRICE warning fired" + elif item.get("expect_no_price_warning") and r.price_warned: r.status = "FAIL" r.detail = "false-positive price warning on a page that must stay silent" elif r.silently_omitted and (retained_ratio >= 0.5 and len(r.silently_omitted) <= 2): @@ -501,6 +557,7 @@ async def main() -> None: args = parser.parse_args() report_dir = EVALS_DIR / "reports" / datetime.now().strftime("%Y-%m-%d") + _preserve_existing_report(report_dir) report_dir.mkdir(parents=True, exist_ok=True) os.environ["TEARSHEET_HOME"] = str(report_dir / "home") # isolated cache per run diff --git a/scripts/check_consistency.py b/scripts/check_consistency.py index b13ac8c..3d19866 100644 --- a/scripts/check_consistency.py +++ b/scripts/check_consistency.py @@ -33,7 +33,9 @@ # "243 tests", "(243 tests, runs in the gate)", "suite (243 tests)" — any prose # claim about how many tests exist. Deliberately broad: a claim this script # cannot see is a claim that can rot. -_README_TEST_CLAIM = re.compile(r"\(?(\d{2,4})\s+tests\b") +# `\d{2,4}` missed single-digit claims — README:170's "6 tests" was invisible to this +# script entirely (2026-08-11 audit). A claim this cannot see is a claim that can rot. +_README_TEST_CLAIM = re.compile(r"\(?(\d{1,5})\s+tests\b") _COLLECTED = re.compile(r"(\d+)(?:/\d+)?\s+tests? collected") @@ -58,18 +60,55 @@ def check_dunder_version(expected: str) -> list[str]: def check_changelog_released(expected: str) -> list[str]: path = REPO_ROOT / "CHANGELOG.md" text = path.read_text() - if re.search(rf"^## \[{re.escape(expected)}\]", text, re.M): - return [] + m = re.search(rf"^## \[{re.escape(expected)}\](.*)$", text, re.M) + if m: + # A heading alone was enough, so `## [0.1.5] — TBD` or an undated section passed. + # A released section carries a real date. + if re.search(r"\d{4}-\d{2}-\d{2}", m.group(1)): + return [] + return [ + f"{path.name}: `## [{expected}]` has no release date " + f"(found {m.group(1).strip()!r}) — an undated section is not a released one" + ] return [ f"{path.name}: no released `## [{expected}]` section — the current version is " "still unreleased, or the section was never added" ] -def collected_test_count() -> tuple[int | None, list[str]]: - """How many tests the DEFAULT suite collects (what the gate and README mean).""" +def check_installed_metadata(expected: str) -> list[str]: + """The INSTALLED distribution's version, which nothing else compares. + + An editable install does not refresh its metadata when pyproject's version changes. + This repo's had been stale at 0.1.0 for three releases (2026-08-11 audit) — invisible + to every other check here, because `tearsheet.__version__` reads the source file + while `importlib.metadata` reads the install. It mattered: `evals/run_eval.py` stamps + each trust report with the metadata version, so two reports were labelled with a + version that had not existed for weeks. Fix with `pip install -e ".[dev]"`. + """ + try: + from importlib.metadata import PackageNotFoundError, version + except ImportError: # pragma: no cover - stdlib since 3.8 + return [] + try: + installed = version("tearsheet") + except PackageNotFoundError: + return [] # not installed (e.g. a bare checkout) — not a claim, so not drift + if installed != expected: + return [ + f"installed distribution metadata says {installed!r} != pyproject {expected!r} " + '— stale editable install; run: pip install -e ".[dev]"' + ] + return [] + + +def collected_test_count(marker: str | None = None) -> tuple[int | None, list[str]]: + """How many tests a suite collects. `marker=None` is the DEFAULT (offline) suite.""" + argv = [sys.executable, "-m", "pytest", "--collect-only", "-q"] + if marker: + argv += ["-m", marker] proc = subprocess.run( # noqa: S603 - fixed argv, no shell - [sys.executable, "-m", "pytest", "--collect-only", "-q"], + argv, cwd=REPO_ROOT, capture_output=True, text=True, @@ -81,19 +120,39 @@ def collected_test_count() -> tuple[int | None, list[str]]: if m: break if not m: - return None, ["could not determine collected test count from pytest output"] + # "no tests collected (284 deselected)" is a real answer, not a parse failure — + # the `live` marker is documented in the README but currently has no tests. + if "no tests collected" in proc.stdout: + return 0, [] + label = f" for -m {marker}" if marker else "" + return None, [f"could not determine collected test count{label} from pytest output"] return int(m.group(1)), [] -def check_readme_test_counts(actual: int) -> list[str]: +def check_readme_test_counts(default_count: int, marked_counts: dict[str, int]) -> list[str]: + """Every "N tests" claim must match the suite it is actually talking about. + + Claims are not all about the default suite: the README also advertises the + real-browser suite. Comparing those against the default collection would be wrong, + and simply skipping them would leave them unverifiable — which is how README:178's + "6 tests" sat outside this script's reach entirely (2026-08-11 audit). + """ path = REPO_ROOT / "README.md" problems = [] for lineno, line in enumerate(path.read_text().splitlines(), start=1): + lowered = line.lower() + # Match the pytest INVOCATION (`-m playwright`), not the bare word: README:14 + # says "a falsifiable live eval harness" in prose, which a substring test + # mistook for a claim about the `live` marker. + suite = next((m for m in marked_counts if f"-m {m}" in lowered), None) + expected = marked_counts[suite] if suite else default_count + label = f"the {suite} suite" if suite else "the default suite" for m in _README_TEST_CLAIM.finditer(line): claimed = int(m.group(1)) - if claimed != actual: + if claimed != expected: problems.append( - f"{path.name}:{lineno}: claims {claimed} tests, pytest collects {actual}" + f"{path.name}:{lineno}: claims {claimed} tests for {label}, " + f"pytest collects {expected}" ) return problems @@ -103,11 +162,18 @@ def main() -> int: problems: list[str] = [] problems.extend(check_dunder_version(expected)) problems.extend(check_changelog_released(expected)) + problems.extend(check_installed_metadata(expected)) actual, count_problems = collected_test_count() problems.extend(count_problems) + marked_counts: dict[str, int] = {} + for marker in ("playwright", "live"): + count, marker_problems = collected_test_count(marker) + problems.extend(marker_problems) + if count is not None: + marked_counts[marker] = count if actual is not None: - problems.extend(check_readme_test_counts(actual)) + problems.extend(check_readme_test_counts(actual, marked_counts)) if problems: print(f"Consistency check FAILED. pyproject.toml version is {expected!r}.") diff --git a/src/tearsheet/content.py b/src/tearsheet/content.py index a32b574..7acf513 100644 --- a/src/tearsheet/content.py +++ b/src/tearsheet/content.py @@ -50,7 +50,13 @@ # challenge pages CAN exceed 30 KB once their JS is inlined, sail past that guard, # extract to a short wall message, and get cached as "content" for the whole TTL. # Same lesson as the consent wall: when the extraction IS the wall, catch it here, -# regardless of body size. Keep phrases in sync with fetch._BLOCK_MARKERS. +# regardless of body size. +# +# THIS IS THE SINGLE SOURCE OF TRUTH for both checks: fetch._BLOCK_MARKERS is derived +# from it. It used to say "keep these in sync with fetch" and they had drifted anyway — +# this list carried "checking your browser" and the looser "attention required", fetch +# carried neither, so a sub-30 KB "Checking your browser…" interstitial was invisible to +# the raw-body check. A comment cannot enforce a shared list; one definition can. _BLOCK_PHRASES = ( "complete the captcha", "solve the captcha", diff --git a/src/tearsheet/fetch.py b/src/tearsheet/fetch.py index 24b3179..eec4141 100644 --- a/src/tearsheet/fetch.py +++ b/src/tearsheet/fetch.py @@ -6,6 +6,7 @@ import httpx from tearsheet.config import Settings +from tearsheet.content import _BLOCK_PHRASES _SPA_MARKERS = ( b'id="root"', @@ -80,15 +81,11 @@ async def fetch_url( ) -_BLOCK_MARKERS = ( - b"complete the captcha", - b"solve the captcha", - b"flagged as potentially automated", - b"attention required!", - b"verify you are a human", - b"are you a robot", - b"enable javascript and cookies to continue", -) +# Derived from content._BLOCK_PHRASES so the raw-body check and the post-extraction +# backstop can never drift apart again — they did, and a sub-30 KB "Checking your +# browser…" interstitial slipped past this function as a result. content has no +# tearsheet imports and defers its heavy ones, so this stays cheap at import time. +_BLOCK_MARKERS = tuple(phrase.encode() for phrase in _BLOCK_PHRASES) _BLOCK_MAX_BODY = 30_000 # real bot walls are small pages; articles about captchas aren't diff --git a/tests/test_eval_harness.py b/tests/test_eval_harness.py new file mode 100644 index 0000000..a695108 --- /dev/null +++ b/tests/test_eval_harness.py @@ -0,0 +1,122 @@ +"""Coverage for the trust instrument itself. + +`evals/run_eval.py` is what decides whether tearsheet can be trusted, and it had no +tests at all — including for a function that RENAMES DIRECTORIES containing evidence. +An unverified instrument cannot certify anything. + +Only the pure, offline pieces are covered here; the harness's network run stays manual +by design (see evals/README.md). +""" + +import importlib.util +import sys +from pathlib import Path + +import pytest + +_SPEC = importlib.util.spec_from_file_location( + "run_eval", Path(__file__).resolve().parent.parent / "evals" / "run_eval.py" +) +assert _SPEC and _SPEC.loader +run_eval = importlib.util.module_from_spec(_SPEC) +sys.modules["run_eval"] = run_eval +_SPEC.loader.exec_module(run_eval) + + +class TestOracleMoneyMatcher: + """The oracle is DELIBERATELY a separate implementation from tearsheet.content — + importing the tool's matcher would make the oracle blind wherever the tool is blind. + But it must not be WORSE, which it was until 2026-08-11: it carried the same four + gaps, so it under-counted the figures on the page and could not score an omission. + """ + + def test_canonicalises_split_currency(self) -> None: + assert run_eval.money_set("$ 49 and $99") == {"$49", "$99"} + + def test_trailing_comma_is_not_part_of_the_figure(self) -> None: + assert run_eval.money_set("costs $49, $99, $149 total") == {"$49", "$99", "$149"} + + def test_suffix_currency(self) -> None: + assert run_eval.money_set("Ab 19,99 € pro Monat") == {"€19,99"} + + def test_decimal_precision_is_not_truncated(self) -> None: + assert run_eval.money_set("$1.1 billion") == {"$1.1"} + + def test_thousands_and_cents(self) -> None: + assert run_eval.money_set("$1,200 and $10,500.00") == {"$1,200", "$10,500.00"} + + def test_agrees_with_the_tool_on_the_shared_contract(self) -> None: + """Two independent implementations, same requirements. Divergence here is a + finding worth reading, not something to paper over — hence a real assertion.""" + from tearsheet.content import money_figures + + for text in ( + "$ 49 and $99", + "costs $49, $99, $149 total", + "Ab 19,99 € pro Monat", + "$1.1 billion", + "$1,200 and $10,500.00", + "no money here at all", + ): + assert run_eval.money_set(text) == money_figures(text), text + + +class TestReportPreservation: + """A report is evidence. The output directory is the run date, so a second run on + the same day used to overwrite the first one's report.md in place.""" + + def test_existing_report_is_moved_aside_not_destroyed(self, tmp_path: Path) -> None: + day = tmp_path / "2026-08-11" + (day / "evidence" / "quo").mkdir(parents=True) + (day / "report.md").write_text("FIRST RUN") + (day / "evidence" / "quo" / "returned.txt").write_text("evidence") + + run_eval._preserve_existing_report(day) + + assert not day.exists(), "the date dir must be freed for the new run" + archived = tmp_path / "2026-08-11-run1" + assert (archived / "report.md").read_text() == "FIRST RUN" + assert (archived / "evidence" / "quo" / "returned.txt").read_text() == "evidence" + + def test_a_third_run_does_not_clobber_the_second(self, tmp_path: Path) -> None: + (tmp_path / "2026-08-11-run1").mkdir() + (tmp_path / "2026-08-11-run1" / "report.md").write_text("RUN ONE") + day = tmp_path / "2026-08-11" + day.mkdir() + (day / "report.md").write_text("RUN TWO") + + run_eval._preserve_existing_report(day) + + assert (tmp_path / "2026-08-11-run1" / "report.md").read_text() == "RUN ONE" + assert (tmp_path / "2026-08-11-run2" / "report.md").read_text() == "RUN TWO" + + def test_a_first_run_is_left_alone(self, tmp_path: Path) -> None: + day = tmp_path / "2026-08-11" + day.mkdir() + run_eval._preserve_existing_report(day) + assert day.exists() + assert not (tmp_path / "2026-08-11-run1").exists() + + def test_a_dir_without_a_report_is_not_archived(self, tmp_path: Path) -> None: + """A crashed run leaves evidence/ but no report.md — reusing it is correct.""" + day = tmp_path / "2026-08-11" + (day / "evidence").mkdir(parents=True) + run_eval._preserve_existing_report(day) + assert day.exists() + assert not (tmp_path / "2026-08-11-run1").exists() + + +class TestWallReportDetection: + @pytest.mark.parametrize( + "out", + [ + "blocked by bot protection (final url: x)", + "consent/cookie wall (final url: x)", + "error fetching https://x: HTTP 403", + ], + ) + def test_refusals_are_recognised_as_walls_not_content(self, out: str) -> None: + assert run_eval.Eval.is_wall_report(out) + + def test_ordinary_content_is_not_a_wall(self) -> None: + assert not run_eval.Eval.is_wall_report("url: x\ntokens: ~50\n---\nReal content.") diff --git a/tests/test_fetch.py b/tests/test_fetch.py index da84f1e..2171407 100644 --- a/tests/test_fetch.py +++ b/tests/test_fetch.py @@ -74,3 +74,39 @@ def handler(request: httpx.Request) -> httpx.Response: assert result.status == 0 assert result.error is not None assert "boom" in result.error + + +class TestBlockMarkersStayInSyncWithContent: + """content._BLOCK_PHRASES carries the instruction "Keep phrases in sync with + fetch._BLOCK_MARKERS" — and they had drifted apart (2026-08-11 audit). content had + "checking your browser" and the looser "attention required"; fetch had neither, so a + sub-30KB "Checking your browser before accessing…" interstitial was invisible to the + raw-body check and depended entirely on the post-extraction backstop, which only + fires when the extraction lands under 600 chars. + + A comment cannot enforce a shared list. One definition can. + """ + + def test_the_two_lists_are_the_same_list(self) -> None: + from tearsheet import content, fetch + + assert {m.decode() for m in fetch._BLOCK_MARKERS} == set(content._BLOCK_PHRASES) + + def test_checking_your_browser_is_caught_on_the_raw_body(self) -> None: + from tearsheet.fetch import looks_blocked + + body = ( + b"Just a moment" + b"

Checking your browser before accessing the site

" + b"" + ) + assert looks_blocked(body) + + def test_long_article_about_captchas_still_does_not_flag(self) -> None: + """The small-body guard is what protects prose; unifying the lists must not + weaken it.""" + from tearsheet.fetch import looks_blocked + + article = b"Sites ask you to verify you are a human. " + b"History follows. " * 3000 + assert len(article) > 30_000 + assert not looks_blocked(article)