Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions evals/reports/2026-08-11-run1/report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Tearsheet Trust Report — 2026-08-11 13:13

**VERDICT: GREEN**

- tearsheet 0.1.5 · trafilatura 2.2.0 · httpx 0.28.1 · lxml 6.1.1 · python 3.14.2
- corpus 2026-07-16.2 (sha fe68ddd7bb2c) · 40 items scored · burst: 45 scrapes across 3 domains: {'ok': 40, 'honest_refusal': 5, 'FAIL': 0, 'exception': 0}

## Gates

| gate | status | detail |
|---|---|---|
| fabrication (0 tolerated) | PASS | 0 pages fabricated figures |
| guard calibration (0 regressions) | PASS | known-bad warns, known-good silent |
| silent omission (<=1) | PASS | 1 pages omitted without warning |
| minor omissions within retention floor (<=3) | PASS | tailscale: ['$5'] missing (80% retained — within the document |
| unhandled exceptions (0) | PASS | 0 |
| cache poison (0) | PASS | none served |
| timeouts (<=2) | PASS | 0 |
| baseline drift (0) | PASS | stable |
| hard failures (<=1 beyond the above) | PASS | wiki_idaho: SILENT omission: ['$74,900'] |

## Items

| id | category | status | detail |
|---|---|---|---|
| quo | pricing_grid | warn-correct | warning riding with content; 23 figures flagged |
| slack | pricing_grid | warn-correct | warning riding with content; 7 figures flagged |
| notion | pricing_grid | warn-correct | warning riding with content; 3 figures flagged |
| zapier | pricing_grid | warn-correct | warning riding with content; 9 figures flagged |
| tailscale | pricing_grid | minor-omission | ['$5'] missing (80% retained — within the documented retention floor) |
| zoom | pricing_tabbed | warn-correct | warning riding with content; 6 figures flagged |
| ringcentral | pricing_tabbed | pass | 4/4 figures shown |
| heyrosie | plan_cards | pass | 5/5 figures shown |
| calendly | plan_cards | pass | 3/5 figures shown |
| basecamp | plan_cards | pass | 3/3 figures shown |
| hetzner | pricing_non_usd | pass | clean |
| scaleway | pricing_non_usd | warn-correct | warning riding with content; 5 figures flagged |
| giffgaff | pricing_non_usd | pass | clean |
| linkedin_fp | article_peripheral | pass | 0/2 figures shown |
| rfc9110 | pinned_immutable | pass | clean |
| rfc2616_txt | pinned_immutable | pass | clean |
| gutenberg_alice | pinned_immutable | pass | clean |
| w3c_css2 | pinned_immutable | pass | clean |
| gh_raw_readme | pinned_immutable | pass | clean |
| wiki_idaho | reference_table | **FAIL** | SILENT omission: ['$74,900'] |
| fed_h15 | data_table | pass | clean |
| pg_essay | control_prose | pass | clean |
| py_docs | control_docs | pass | clean |
| fastapi_docs | control_docs | pass | clean |
| guardian | consent_heavy | pass | content served (3951 chars) |
| lemonde | consent_heavy | pass | content served (3788 chars) |
| zeit | consent_heavy | pass | content served (3987 chars) |
| ecfr | botwall | walled | honest refusal |
| dodcio | botwall | walled | honest refusal |
| crunchbase | botwall | walled | honest refusal |
| todomvc | spa_shell | documented-limit | tiny output (300 chars) — the known custom-mount class |
| aozora | charset_shift_jis | pass | clean |
| wiki_ja | charset_utf8_ja | pass | clean |
| gh_api | json_endpoint | pass | JSON pretty-printed |
| berkshire_pdf | pdf_figures | pass | pdf text extracted; 3 money figures visible |
| irs_p15 | pdf_tables | pass | pdf text extracted; 7 money figures visible |
| map_uv | map | pass | 50 urls mapped |
| map_fastapi | map | pass | 50 urls mapped |
| crawl_ruff | crawl | pass | crawl: docs.astral.sh pages: 5 errors: 0 skipped(robots/dupe/type): 0 |
| search_smoke | search | pass | results returned |

## Reading this report

- `pass` — figures/content verified against the independent oracle.
- `warn-correct` — the tool flagged its own extraction; the warning was warranted.
- `walled` — honest refusal (bot wall / consent wall / HTTP error reported as such).
- `FAIL` — a trust property was violated; see evidence/ for the raw bytes.
- Every item's raw evidence is under `evidence/<id>/` for re-adjudication.
12 changes: 6 additions & 6 deletions evals/reports/2026-08-11/report.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Tearsheet Trust Report — 2026-08-11 13:13
# Tearsheet Trust Report — 2026-08-11 14:03

**VERDICT: GREEN**

Expand Down Expand Up @@ -47,9 +47,9 @@
| pg_essay | control_prose | pass | clean |
| py_docs | control_docs | pass | clean |
| fastapi_docs | control_docs | pass | clean |
| guardian | consent_heavy | pass | content served (3951 chars) |
| lemonde | consent_heavy | pass | content served (3788 chars) |
| zeit | consent_heavy | pass | content served (3987 chars) |
| guardian | consent_heavy | pass | content served (4002 chars) |
| lemonde | consent_heavy | pass | content served (3826 chars) |
| zeit | consent_heavy | pass | content served (3937 chars) |
| ecfr | botwall | walled | honest refusal |
| dodcio | botwall | walled | honest refusal |
| crunchbase | botwall | walled | honest refusal |
Expand All @@ -58,10 +58,10 @@
| wiki_ja | charset_utf8_ja | pass | clean |
| gh_api | json_endpoint | pass | JSON pretty-printed |
| berkshire_pdf | pdf_figures | pass | pdf text extracted; 3 money figures visible |
| irs_p15 | pdf_tables | pass | pdf text extracted; 7 money figures visible |
| irs_p15 | pdf_tables | unreachable | error fetching https://www.irs.gov/pub/irs-pdf/p15.pdf: HTTP 403 |
| map_uv | map | pass | 50 urls mapped |
| map_fastapi | map | pass | 50 urls mapped |
| crawl_ruff | crawl | pass | crawl: docs.astral.sh pages: 5 errors: 0 skipped(robots/dupe/type): 0 |
| crawl_ruff | crawl | pass | crawl: docs.astral.sh pages: 5 errors: 0 refused: 0 skipped(robots/dupe/type): 0 |
| search_smoke | search | pass | results returned |

## Reading this report
Expand Down