From d9c83f0924273e8ab11f9b48cbd8a14ff751a82d Mon Sep 17 00:00:00 2001 From: nathan nelson Date: Tue, 11 Aug 2026 14:05:00 -0600 Subject: [PATCH] =?UTF-8?q?evals:=20post-guard=20live=20run=20on=20main=20?= =?UTF-8?q?=E2=80=94=20GREEN,=20one=20external=20change?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second run of the day, on merged main (trafilatura 2.2 + the guard-coverage pass). The first run covered the dependency upgrade only; this one exercises the crawl and extract refusal paths the 194-page cache replay structurally could not reach, since that replay only drives assess_extraction over stored HTML. The earlier report is preserved as reports/2026-08-11-run1/ rather than being overwritten — run_eval.py hardcodes the output directory to today's date, and that report is the pre-guard baseline referenced by #6. Matches the existing 2026-07-16 / 2026-07-16-run1 convention. VERDICT: GREEN. All 9 gates PASS. 40 items + 45-scrape burst (40 ok, 5 honest refusals, 0 FAIL, 0 exceptions). Zero fabrications, zero cache poison, zero unhandled exceptions, baselines stable. Exactly ONE item changed status against the baseline: irs_p15 pass -> unreachable (HTTP 403) That is external, and it was verified rather than assumed: - fetch.py and config.py are BYTE-IDENTICAL between 06296c8 (pre-guard) and 77196fc (post-guard), so the guard PR cannot have caused it - curl gets HTTP 200 from the same URL with tearsheet's exact User-Agent AND Accept header, over both HTTP/1.1 and HTTP/2 - the other PDF item (berkshire_pdf) still extracts figures normally So the discriminator sits below the HTTP layer — TLS client fingerprinting. irs.gov has joined the Akamai class that already 403s eCFR and dodcio, and tearsheet reported it honestly as a fetch error instead of serving anything made up. The record of zero fabrications holds. crawl_ruff passes with the new index format (`refused: 0`). Co-Authored-By: Claude Fable 5 --- evals/reports/2026-08-11-run1/report.md | 73 +++++++++++++++++++++++++ evals/reports/2026-08-11/report.md | 12 ++-- 2 files changed, 79 insertions(+), 6 deletions(-) create mode 100644 evals/reports/2026-08-11-run1/report.md diff --git a/evals/reports/2026-08-11-run1/report.md b/evals/reports/2026-08-11-run1/report.md new file mode 100644 index 0000000..accb1e5 --- /dev/null +++ b/evals/reports/2026-08-11-run1/report.md @@ -0,0 +1,73 @@ +# Tearsheet Trust Report — 2026-08-11 13:13 + +**VERDICT: GREEN** + +- tearsheet 0.1.5 · trafilatura 2.2.0 · httpx 0.28.1 · lxml 6.1.1 · python 3.14.2 +- corpus 2026-07-16.2 (sha fe68ddd7bb2c) · 40 items scored · burst: 45 scrapes across 3 domains: {'ok': 40, 'honest_refusal': 5, 'FAIL': 0, 'exception': 0} + +## Gates + +| gate | status | detail | +|---|---|---| +| fabrication (0 tolerated) | PASS | 0 pages fabricated figures | +| guard calibration (0 regressions) | PASS | known-bad warns, known-good silent | +| silent omission (<=1) | PASS | 1 pages omitted without warning | +| minor omissions within retention floor (<=3) | PASS | tailscale: ['$5'] missing (80% retained — within the document | +| unhandled exceptions (0) | PASS | 0 | +| cache poison (0) | PASS | none served | +| timeouts (<=2) | PASS | 0 | +| baseline drift (0) | PASS | stable | +| hard failures (<=1 beyond the above) | PASS | wiki_idaho: SILENT omission: ['$74,900'] | + +## Items + +| id | category | status | detail | +|---|---|---|---| +| quo | pricing_grid | warn-correct | warning riding with content; 23 figures flagged | +| slack | pricing_grid | warn-correct | warning riding with content; 7 figures flagged | +| notion | pricing_grid | warn-correct | warning riding with content; 3 figures flagged | +| zapier | pricing_grid | warn-correct | warning riding with content; 9 figures flagged | +| tailscale | pricing_grid | minor-omission | ['$5'] missing (80% retained — within the documented retention floor) | +| zoom | pricing_tabbed | warn-correct | warning riding with content; 6 figures flagged | +| ringcentral | pricing_tabbed | pass | 4/4 figures shown | +| heyrosie | plan_cards | pass | 5/5 figures shown | +| calendly | plan_cards | pass | 3/5 figures shown | +| basecamp | plan_cards | pass | 3/3 figures shown | +| hetzner | pricing_non_usd | pass | clean | +| scaleway | pricing_non_usd | warn-correct | warning riding with content; 5 figures flagged | +| giffgaff | pricing_non_usd | pass | clean | +| linkedin_fp | article_peripheral | pass | 0/2 figures shown | +| rfc9110 | pinned_immutable | pass | clean | +| rfc2616_txt | pinned_immutable | pass | clean | +| gutenberg_alice | pinned_immutable | pass | clean | +| w3c_css2 | pinned_immutable | pass | clean | +| gh_raw_readme | pinned_immutable | pass | clean | +| wiki_idaho | reference_table | **FAIL** | SILENT omission: ['$74,900'] | +| fed_h15 | data_table | pass | clean | +| pg_essay | control_prose | pass | clean | +| py_docs | control_docs | pass | clean | +| fastapi_docs | control_docs | pass | clean | +| guardian | consent_heavy | pass | content served (3951 chars) | +| lemonde | consent_heavy | pass | content served (3788 chars) | +| zeit | consent_heavy | pass | content served (3987 chars) | +| ecfr | botwall | walled | honest refusal | +| dodcio | botwall | walled | honest refusal | +| crunchbase | botwall | walled | honest refusal | +| todomvc | spa_shell | documented-limit | tiny output (300 chars) — the known custom-mount class | +| aozora | charset_shift_jis | pass | clean | +| wiki_ja | charset_utf8_ja | pass | clean | +| gh_api | json_endpoint | pass | JSON pretty-printed | +| berkshire_pdf | pdf_figures | pass | pdf text extracted; 3 money figures visible | +| irs_p15 | pdf_tables | pass | pdf text extracted; 7 money figures visible | +| map_uv | map | pass | 50 urls mapped | +| map_fastapi | map | pass | 50 urls mapped | +| crawl_ruff | crawl | pass | crawl: docs.astral.sh pages: 5 errors: 0 skipped(robots/dupe/type): 0 | +| search_smoke | search | pass | results returned | + +## Reading this report + +- `pass` — figures/content verified against the independent oracle. +- `warn-correct` — the tool flagged its own extraction; the warning was warranted. +- `walled` — honest refusal (bot wall / consent wall / HTTP error reported as such). +- `FAIL` — a trust property was violated; see evidence/ for the raw bytes. +- Every item's raw evidence is under `evidence//` for re-adjudication. \ No newline at end of file diff --git a/evals/reports/2026-08-11/report.md b/evals/reports/2026-08-11/report.md index accb1e5..805d465 100644 --- a/evals/reports/2026-08-11/report.md +++ b/evals/reports/2026-08-11/report.md @@ -1,4 +1,4 @@ -# Tearsheet Trust Report — 2026-08-11 13:13 +# Tearsheet Trust Report — 2026-08-11 14:03 **VERDICT: GREEN** @@ -47,9 +47,9 @@ | pg_essay | control_prose | pass | clean | | py_docs | control_docs | pass | clean | | fastapi_docs | control_docs | pass | clean | -| guardian | consent_heavy | pass | content served (3951 chars) | -| lemonde | consent_heavy | pass | content served (3788 chars) | -| zeit | consent_heavy | pass | content served (3987 chars) | +| guardian | consent_heavy | pass | content served (4002 chars) | +| lemonde | consent_heavy | pass | content served (3826 chars) | +| zeit | consent_heavy | pass | content served (3937 chars) | | ecfr | botwall | walled | honest refusal | | dodcio | botwall | walled | honest refusal | | crunchbase | botwall | walled | honest refusal | @@ -58,10 +58,10 @@ | wiki_ja | charset_utf8_ja | pass | clean | | gh_api | json_endpoint | pass | JSON pretty-printed | | berkshire_pdf | pdf_figures | pass | pdf text extracted; 3 money figures visible | -| irs_p15 | pdf_tables | pass | pdf text extracted; 7 money figures visible | +| irs_p15 | pdf_tables | unreachable | error fetching https://www.irs.gov/pub/irs-pdf/p15.pdf: HTTP 403 | | map_uv | map | pass | 50 urls mapped | | map_fastapi | map | pass | 50 urls mapped | -| crawl_ruff | crawl | pass | crawl: docs.astral.sh pages: 5 errors: 0 skipped(robots/dupe/type): 0 | +| crawl_ruff | crawl | pass | crawl: docs.astral.sh pages: 5 errors: 0 refused: 0 skipped(robots/dupe/type): 0 | | search_smoke | search | pass | results returned | ## Reading this report