From 1374060c2ae1c83c619bd438ed0eaf1da7ae198c Mon Sep 17 00:00:00 2001 From: nathan nelson Date: Sat, 18 Jul 2026 22:15:34 -0600 Subject: [PATCH 1/2] docs: add CHANGELOG, CONTRIBUTING, SECURITY + wiki link in README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the conventional governance/history files the repo lacked, and make the new GitHub Wiki discoverable from the README. - CHANGELOG.md: Keep a Changelog format, v0.1.0–v0.1.4 + Unreleased, reconstructed from the version-bump commit history (no tags yet; version links point at the shipping commit). - CONTRIBUTING.md: setup, the ruff+mypy+pytest gate, TDD + offline-determinism conventions; links to the wiki deep-dive. - SECURITY.md: private disclosure channel, supported-version = latest main, and the local-tool threat model. - README.md: CI/license/python badges + a Documentation → Wiki pointer. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 96 +++++++++++++++++++++++++++++++++++++++++++++++++ CONTRIBUTING.md | 62 ++++++++++++++++++++++++++++++++ README.md | 9 +++++ SECURITY.md | 41 +++++++++++++++++++++ 4 files changed, 208 insertions(+) create mode 100644 CHANGELOG.md create mode 100644 CONTRIBUTING.md create mode 100644 SECURITY.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..3deb750 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,96 @@ +# Changelog + +All notable changes to **tearsheet** are documented here. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +For the *narrative* of how the guard system evolved — which real page broke each +guard, and why — see the wiki page +[How the Guards Evolved](https://github.com/Wynelson94/tearsheet/wiki/How-the-Guards-Evolved). + +## [Unreleased] + +### Added +- MIT `LICENSE`, project URLs, and a GitHub Actions CI gate (`ruff` + `mypy` + `pytest`, + matrixed over Python 3.12 and 3.13). +- Synthetic false-positive fixture so the probation corpus republishes no real individual's page. + +### Changed +- User-Agent string corrected; live-eval evidence artifacts untracked (only `report.md` + scorecards are committed). +- Version and test-count references made consistent across the repo. + +## [0.1.4] — 2026-07-16 + +### Added +- **Title-armed dropped-price guard** — the "notion-class" gap: genuine pricing pages whose + plan cards are diluted by prose so the real figures never share a 1,500-char window. When a + page's *title* declares it a pricing page, the guard now arms at ≥ 3 distinct figures + page-wide (instead of requiring a cluster), catching omissions the cluster heuristic missed. +- **€ and £ matching** in the price guard (previously `$` only). +- **Post-extraction bot-wall backstop** — challenge pages too large for the raw-body heuristic + are now caught after extraction, reported, and never cached; previously poisoned cache rows + are evicted rather than replayed. + +### Changed +- Live-eval harness adjudicated against run-1 evidence; `map` scoring counts relative paths, + and omissions inside the documented 50%-retention contract are bucketed as minor (not hard fails). + +## [0.1.3] — 2026-07-16 + +### Changed +- **Dropped-price guard now arms on price *clusters*, not page-wide counts.** It fires only + when some 1,500-char window of visible text holds ≥ 4 distinct price figures (a pricing-grid + signature). This eliminates the 2026-07-14 false positive on a LinkedIn post page, whose four + real dollar amounts lived in related-content cards and were correctly excluded by extraction. + Calibrated against the real cached pages (quo, smith.ai, heyrosie) with a clean separation margin. + +## [0.1.2] — 2026-07-14 + +### Added +- **Silent-failure guards** in `assess_extraction()`, each riding in the output header as a + `warning:` line: dropped-price warning, collapsed-column warning, and a gated/never-rendered + warning. +- **Consent-wall detection** — a cookie-consent banner is reported as a `consent/cookie wall`, + never cached, and never passed off as the page (root cause of the smith.ai 428-byte-banner bug). +- **`raw=` / `--raw` escape hatch** — returns the page's visible text, bypassing the extractor, + to recover figures the extractor drops. Deliberately uses the plain fetch, not the browser. + +### Changed +- Warnings surface on both the live and cache-read paths; poisoned cache rows are no longer replayed. + +## [0.1.1] — 2026-07-11 + +### Added +- **Bot-wall detection** (`looks_blocked()`, strong phrases + 30 KB size guard) — bot walls are + reported explicitly and never cached. +- **Tiny-extraction render** — `scrape` auto mode attempts a browser render when extraction is + tiny, so custom SPA mounts (e.g. TodoMVC) upgrade instead of returning near-empty. +- **MCP/CLI parity test** so the "verb available in one interface but not the other" class of gap + cannot recur. +- Documentation + a pinning test for the upstream trafilatura 2.1.0 emphasis-mangling bug + ([adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882)) — fails loudly if + upstream fixes it. + +## [0.1.0] — 2026-07-11 + +Initial working toolset (milestones M1–M5). + +### Added +- **`scrape`** — httpx fetch, trafilatura main-content extraction, SQLite cache, token-shaped output. +- **FastMCP server** exposing the tools to Claude Code / any MCP client. +- **`map`** and **`crawl`** — robots-obedient async BFS crawler with sitemap discovery and + crawl-to-disk output. +- **Playwright** headless-Chromium render fallback and the deterministic **`extract`** tool + (JSON-LD / OpenGraph / microdata / tables). +- **PDF** text extraction (pypdf) and cache eviction. +- **`search`** subcommand added to the CLI (had been MCP-only). + + +[Unreleased]: https://github.com/Wynelson94/tearsheet/compare/feacd27...main +[0.1.4]: https://github.com/Wynelson94/tearsheet/commit/feacd27 +[0.1.3]: https://github.com/Wynelson94/tearsheet/commit/282f71e +[0.1.2]: https://github.com/Wynelson94/tearsheet/commit/45f0e84 +[0.1.1]: https://github.com/Wynelson94/tearsheet/commit/87613ec +[0.1.0]: https://github.com/Wynelson94/tearsheet/commit/a5d42db diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..8176776 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,62 @@ +# Contributing to tearsheet + +Issues and pull requests are welcome. tearsheet is personal research tooling shared as-is — +there's no roadmap and no support promise — but it is tested like it matters, and contributions +that keep it that way are appreciated. + +For the full contributor deep-dive (the trust model, how the guards are calibrated against real +pages, and how the live-eval flywheel works), see the wiki: +**[Development & Testing](https://github.com/Wynelson94/tearsheet/wiki/Development-and-Testing)**. + +## Setup + +```bash +git clone https://github.com/Wynelson94/tearsheet.git +cd tearsheet +python3 -m venv .venv +.venv/bin/pip install -e ".[dev]" +``` + +Requires Python >= 3.12. Optional, for the browser-render path and its tests: +`playwright install chromium`. + +## The gate + +Every change must pass the same gate CI runs. Run it before you push: + +```bash +.venv/bin/ruff check src tests && .venv/bin/mypy && .venv/bin/python -m pytest +``` + +- **`ruff check`** — lint. **`mypy`** — strict type-checking. **`pytest`** — the offline suite. +- The default suite runs **fully offline**: a `conftest.py` socket guard fails any test that + reaches a non-loopback address, so tests stay deterministic. HTTP is faked with + `httpx.MockTransport` and fixture HTML. +- Extras (not in the default gate): `pytest -m playwright` (real Chromium against a local server) + and `pytest -m live` (real network). + +CI (`.github/workflows/ci.yml`) runs `ruff` + `mypy` + `pytest` on every push and PR, matrixed +over Python 3.12 and 3.13. A red gate blocks the merge. + +## Conventions + +- **TDD.** Write the failing test first, then the code that passes it. +- **Guards answer to real pages.** tearsheet's contract is *no figure a page carries may go + missing silently*. If you touch the extraction guards, your change must still satisfy the + permanent fixtures in `tests/fixtures/probation/` — the real pages that defined the tool's + original failures. Every new guard adds a new permanent fixture. See + [Development & Testing](https://github.com/Wynelson94/tearsheet/wiki/Development-and-Testing) + and [Trust & Evaluation](https://github.com/Wynelson94/tearsheet/wiki/Trust-and-Evaluation). +- **Keep it offline-deterministic.** New tests must not reach the network unless explicitly + marked `@pytest.mark.live`. +- **Update the docs.** User-facing changes get a `CHANGELOG.md` entry (see + [Keep a Changelog](https://keepachangelog.com/)); behavior changes that touch the guards get a + note in the wiki. + +## Pull requests + +1. Branch off `main`. +2. Make the change with tests; keep the gate green. +3. Open the PR with a clear description of the behavior change and how you verified it. + +Security issues: please see [SECURITY.md](SECURITY.md) rather than opening a public issue. diff --git a/README.md b/README.md index eda1e5c..f11ee19 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # tearsheet +[![CI](https://github.com/Wynelson94/tearsheet/actions/workflows/ci.yml/badge.svg)](https://github.com/Wynelson94/tearsheet/actions/workflows/ci.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) +[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/) + Local, self-hosted web-to-markdown for LLM research. A Firecrawl-style toolset — scrape, crawl, map, search, extract — that runs entirely on your machine, built to feed Claude Code (or any MCP client) **clean content with minimal context tokens**. No API keys, no SaaS, @@ -12,6 +16,11 @@ eval harness (verdict GREEN), and zero fabrications across the tool's entire rec history. Its documented failure mode is *omission*, and the guards exist to make every omission loud. See [Trust](#trust). +📖 **Documentation:** full install, tool/CLI reference, architecture, the guard/`raw` contract, +and troubleshooting live in the **[Wiki](https://github.com/Wynelson94/tearsheet/wiki)**. +Contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) · Security? [SECURITY.md](SECURITY.md) · +History? [CHANGELOG.md](CHANGELOG.md). + ## Why A language model pays for every token of nav, ads, and footer it reads. tearsheet extracts diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..b8d6892 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,41 @@ +# Security Policy + +## Reporting a vulnerability + +Please report security issues **privately** — do not open a public issue for anything exploitable. + +- **Preferred:** GitHub's private vulnerability reporting — the **Security** tab → + **Report a vulnerability** (if enabled on this repo). +- **Or:** email the maintainer at **nmnslr@gmail.com** with details and, if possible, a + reproduction. + +Please allow a reasonable window for a fix before any public disclosure. As personal research +tooling maintained by one person, response is best-effort — but security reports are taken +seriously and are prioritized over feature work. + +## Supported versions + +Only the latest `main` is supported. There are no backports; fixes land on `main` and in the next +version. tearsheet is not published to PyPI — you run it from source, so *updating* means pulling +`main` and re-installing. + +## Scope & threat model + +tearsheet is a **local, self-hosted** tool. It has no server, no accounts, no telemetry, and +requires no API keys; nothing you scrape leaves your machine except the outbound HTTP(S) request +to the site you pointed it at. It handles no secrets or credentials. + +Security-relevant surface, for reporters: + +- **Fetches arbitrary URLs** you give it (`httpx`) and, on the render path, loads them in a + **headless Chromium** (Playwright). Treat scraped sites as untrusted input. +- **Parses untrusted content** — HTML (trafilatura / lxml / extruct), PDFs (pypdf), sitemaps. + Parser-level issues (resource exhaustion, XML entity handling, crashes on malformed input) are + in scope. +- **Writes to disk** under `~/.tearsheet` (override with `TEARSHEET_HOME`): a SQLite cache and + extracted markdown/crawl files. Path-handling issues are in scope. +- Follows and rate-limits per `robots.txt`; strips tracking parameters. It is **not** designed to + evade paywalls or bot defenses, and reports bot/consent walls rather than bypassing them. + +Out of scope: vulnerabilities in third-party sites you scrape, and upstream bugs in dependencies +(please report those to the relevant project — e.g. trafilatura — though a heads-up here is welcome). From 38af879cc25aabd0c781fa3df0655e5624858b5f Mon Sep 17 00:00:00 2001 From: nathan nelson Date: Fri, 24 Jul 2026 11:15:20 -0600 Subject: [PATCH 2/2] docs: point links at docs that exist; make the security channel real Review-item follow-ups on this PR, both found by auditing the links rather than the prose. Dead wiki deep-links: CHANGELOG linked "How the Guards Evolved" and CONTRIBUTING linked "Development & Testing" (x2) and "Trust & Evaluation". None of those pages exist -- the wiki is a 31-byte "Welcome to the tearsheet wiki!" and Phase A hasn't started. Merging as written would have shipped four 404s. Repointed at what exists today: README's Trust / Development / Limitations sections and evals/README.md. The wiki root link stays (it resolves) and is now described as in progress. README's documentation pointer claimed full install, CLI reference, architecture, the guard/raw contract and troubleshooting "live in the Wiki". All of that is in this README. Corrected to say so. SECURITY.md offered GitHub private vulnerability reporting as the preferred channel, hedged "(if enabled on this repo)" -- and it was NOT enabled, which quietly made the maintainer's personal email the only working disclosure route. Private vulnerability reporting is now enabled on the repo, so the hedge is gone and the preferred path is real. Gate green: ruff, mypy (16 files), 243 passed / 6 deselected. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 6 +++--- CONTRIBUTING.md | 14 ++++++++------ README.md | 5 +++-- SECURITY.md | 2 +- 4 files changed, 15 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3deb750..b7bbf25 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,9 +5,9 @@ All notable changes to **tearsheet** are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -For the *narrative* of how the guard system evolved — which real page broke each -guard, and why — see the wiki page -[How the Guards Evolved](https://github.com/Wynelson94/tearsheet/wiki/How-the-Guards-Evolved). +Each guard below exists because a real page broke it; the entries name the page and the +failure. For the trust model and the evaluation harness behind those calls, see +[Trust](README.md#trust) and [`evals/README.md`](evals/README.md). ## [Unreleased] diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8176776..3d05ade 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,9 +4,11 @@ Issues and pull requests are welcome. tearsheet is personal research tooling sha there's no roadmap and no support promise — but it is tested like it matters, and contributions that keep it that way are appreciated. -For the full contributor deep-dive (the trust model, how the guards are calibrated against real -pages, and how the live-eval flywheel works), see the wiki: -**[Development & Testing](https://github.com/Wynelson94/tearsheet/wiki/Development-and-Testing)**. +For the contributor deep-dive — the trust model, how the guards are calibrated against real +pages, and how the live-eval flywheel works — see **[Trust](README.md#trust)** and +**[Development](README.md#development)** in the README, plus +**[`evals/README.md`](evals/README.md)** for the evaluation harness itself. A fuller +[wiki](https://github.com/Wynelson94/tearsheet/wiki) is in progress. ## Setup @@ -45,13 +47,13 @@ over Python 3.12 and 3.13. A red gate blocks the merge. missing silently*. If you touch the extraction guards, your change must still satisfy the permanent fixtures in `tests/fixtures/probation/` — the real pages that defined the tool's original failures. Every new guard adds a new permanent fixture. See - [Development & Testing](https://github.com/Wynelson94/tearsheet/wiki/Development-and-Testing) - and [Trust & Evaluation](https://github.com/Wynelson94/tearsheet/wiki/Trust-and-Evaluation). + [Trust](README.md#trust) and [`evals/README.md`](evals/README.md). - **Keep it offline-deterministic.** New tests must not reach the network unless explicitly marked `@pytest.mark.live`. - **Update the docs.** User-facing changes get a `CHANGELOG.md` entry (see [Keep a Changelog](https://keepachangelog.com/)); behavior changes that touch the guards get a - note in the wiki. + note in the README's [Limitations & known issues](README.md#limitations--known-issues) or + [Trust](README.md#trust) section. ## Pull requests diff --git a/README.md b/README.md index f11ee19..38edccd 100644 --- a/README.md +++ b/README.md @@ -16,8 +16,9 @@ eval harness (verdict GREEN), and zero fabrications across the tool's entire rec history. Its documented failure mode is *omission*, and the guards exist to make every omission loud. See [Trust](#trust). -📖 **Documentation:** full install, tool/CLI reference, architecture, the guard/`raw` contract, -and troubleshooting live in the **[Wiki](https://github.com/Wynelson94/tearsheet/wiki)**. +📖 **Documentation:** this README is the reference — install, tool/CLI reference, how it works, +the guard/`raw` contract, and known limitations are all below. An expanded +**[Wiki](https://github.com/Wynelson94/tearsheet/wiki)** is in progress. Contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) · Security? [SECURITY.md](SECURITY.md) · History? [CHANGELOG.md](CHANGELOG.md). diff --git a/SECURITY.md b/SECURITY.md index b8d6892..81ac29b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -5,7 +5,7 @@ Please report security issues **privately** — do not open a public issue for anything exploitable. - **Preferred:** GitHub's private vulnerability reporting — the **Security** tab → - **Report a vulnerability** (if enabled on this repo). + **Report a vulnerability**. This is enabled on this repo and is the fastest route. - **Or:** email the maintainer at **nmnslr@gmail.com** with details and, if possible, a reproduction.