diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..b7bbf25 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,96 @@ +# Changelog + +All notable changes to **tearsheet** are documented here. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +Each guard below exists because a real page broke it; the entries name the page and the +failure. For the trust model and the evaluation harness behind those calls, see +[Trust](README.md#trust) and [`evals/README.md`](evals/README.md). + +## [Unreleased] + +### Added +- MIT `LICENSE`, project URLs, and a GitHub Actions CI gate (`ruff` + `mypy` + `pytest`, + matrixed over Python 3.12 and 3.13). +- Synthetic false-positive fixture so the probation corpus republishes no real individual's page. + +### Changed +- User-Agent string corrected; live-eval evidence artifacts untracked (only `report.md` + scorecards are committed). +- Version and test-count references made consistent across the repo. + +## [0.1.4] — 2026-07-16 + +### Added +- **Title-armed dropped-price guard** — the "notion-class" gap: genuine pricing pages whose + plan cards are diluted by prose so the real figures never share a 1,500-char window. When a + page's *title* declares it a pricing page, the guard now arms at ≥ 3 distinct figures + page-wide (instead of requiring a cluster), catching omissions the cluster heuristic missed. +- **€ and £ matching** in the price guard (previously `$` only). +- **Post-extraction bot-wall backstop** — challenge pages too large for the raw-body heuristic + are now caught after extraction, reported, and never cached; previously poisoned cache rows + are evicted rather than replayed. + +### Changed +- Live-eval harness adjudicated against run-1 evidence; `map` scoring counts relative paths, + and omissions inside the documented 50%-retention contract are bucketed as minor (not hard fails). + +## [0.1.3] — 2026-07-16 + +### Changed +- **Dropped-price guard now arms on price *clusters*, not page-wide counts.** It fires only + when some 1,500-char window of visible text holds ≥ 4 distinct price figures (a pricing-grid + signature). This eliminates the 2026-07-14 false positive on a LinkedIn post page, whose four + real dollar amounts lived in related-content cards and were correctly excluded by extraction. + Calibrated against the real cached pages (quo, smith.ai, heyrosie) with a clean separation margin. + +## [0.1.2] — 2026-07-14 + +### Added +- **Silent-failure guards** in `assess_extraction()`, each riding in the output header as a + `warning:` line: dropped-price warning, collapsed-column warning, and a gated/never-rendered + warning. +- **Consent-wall detection** — a cookie-consent banner is reported as a `consent/cookie wall`, + never cached, and never passed off as the page (root cause of the smith.ai 428-byte-banner bug). +- **`raw=` / `--raw` escape hatch** — returns the page's visible text, bypassing the extractor, + to recover figures the extractor drops. Deliberately uses the plain fetch, not the browser. + +### Changed +- Warnings surface on both the live and cache-read paths; poisoned cache rows are no longer replayed. + +## [0.1.1] — 2026-07-11 + +### Added +- **Bot-wall detection** (`looks_blocked()`, strong phrases + 30 KB size guard) — bot walls are + reported explicitly and never cached. +- **Tiny-extraction render** — `scrape` auto mode attempts a browser render when extraction is + tiny, so custom SPA mounts (e.g. TodoMVC) upgrade instead of returning near-empty. +- **MCP/CLI parity test** so the "verb available in one interface but not the other" class of gap + cannot recur. +- Documentation + a pinning test for the upstream trafilatura 2.1.0 emphasis-mangling bug + ([adbar/trafilatura#882](https://github.com/adbar/trafilatura/issues/882)) — fails loudly if + upstream fixes it. + +## [0.1.0] — 2026-07-11 + +Initial working toolset (milestones M1–M5). + +### Added +- **`scrape`** — httpx fetch, trafilatura main-content extraction, SQLite cache, token-shaped output. +- **FastMCP server** exposing the tools to Claude Code / any MCP client. +- **`map`** and **`crawl`** — robots-obedient async BFS crawler with sitemap discovery and + crawl-to-disk output. +- **Playwright** headless-Chromium render fallback and the deterministic **`extract`** tool + (JSON-LD / OpenGraph / microdata / tables). +- **PDF** text extraction (pypdf) and cache eviction. +- **`search`** subcommand added to the CLI (had been MCP-only). + + +[Unreleased]: https://github.com/Wynelson94/tearsheet/compare/feacd27...main +[0.1.4]: https://github.com/Wynelson94/tearsheet/commit/feacd27 +[0.1.3]: https://github.com/Wynelson94/tearsheet/commit/282f71e +[0.1.2]: https://github.com/Wynelson94/tearsheet/commit/45f0e84 +[0.1.1]: https://github.com/Wynelson94/tearsheet/commit/87613ec +[0.1.0]: https://github.com/Wynelson94/tearsheet/commit/a5d42db diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..3d05ade --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,64 @@ +# Contributing to tearsheet + +Issues and pull requests are welcome. tearsheet is personal research tooling shared as-is — +there's no roadmap and no support promise — but it is tested like it matters, and contributions +that keep it that way are appreciated. + +For the contributor deep-dive — the trust model, how the guards are calibrated against real +pages, and how the live-eval flywheel works — see **[Trust](README.md#trust)** and +**[Development](README.md#development)** in the README, plus +**[`evals/README.md`](evals/README.md)** for the evaluation harness itself. A fuller +[wiki](https://github.com/Wynelson94/tearsheet/wiki) is in progress. + +## Setup + +```bash +git clone https://github.com/Wynelson94/tearsheet.git +cd tearsheet +python3 -m venv .venv +.venv/bin/pip install -e ".[dev]" +``` + +Requires Python >= 3.12. Optional, for the browser-render path and its tests: +`playwright install chromium`. + +## The gate + +Every change must pass the same gate CI runs. Run it before you push: + +```bash +.venv/bin/ruff check src tests && .venv/bin/mypy && .venv/bin/python -m pytest +``` + +- **`ruff check`** — lint. **`mypy`** — strict type-checking. **`pytest`** — the offline suite. +- The default suite runs **fully offline**: a `conftest.py` socket guard fails any test that + reaches a non-loopback address, so tests stay deterministic. HTTP is faked with + `httpx.MockTransport` and fixture HTML. +- Extras (not in the default gate): `pytest -m playwright` (real Chromium against a local server) + and `pytest -m live` (real network). + +CI (`.github/workflows/ci.yml`) runs `ruff` + `mypy` + `pytest` on every push and PR, matrixed +over Python 3.12 and 3.13. A red gate blocks the merge. + +## Conventions + +- **TDD.** Write the failing test first, then the code that passes it. +- **Guards answer to real pages.** tearsheet's contract is *no figure a page carries may go + missing silently*. If you touch the extraction guards, your change must still satisfy the + permanent fixtures in `tests/fixtures/probation/` — the real pages that defined the tool's + original failures. Every new guard adds a new permanent fixture. See + [Trust](README.md#trust) and [`evals/README.md`](evals/README.md). +- **Keep it offline-deterministic.** New tests must not reach the network unless explicitly + marked `@pytest.mark.live`. +- **Update the docs.** User-facing changes get a `CHANGELOG.md` entry (see + [Keep a Changelog](https://keepachangelog.com/)); behavior changes that touch the guards get a + note in the README's [Limitations & known issues](README.md#limitations--known-issues) or + [Trust](README.md#trust) section. + +## Pull requests + +1. Branch off `main`. +2. Make the change with tests; keep the gate green. +3. Open the PR with a clear description of the behavior change and how you verified it. + +Security issues: please see [SECURITY.md](SECURITY.md) rather than opening a public issue. diff --git a/README.md b/README.md index eda1e5c..38edccd 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # tearsheet +[![CI](https://github.com/Wynelson94/tearsheet/actions/workflows/ci.yml/badge.svg)](https://github.com/Wynelson94/tearsheet/actions/workflows/ci.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) +[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/) + Local, self-hosted web-to-markdown for LLM research. A Firecrawl-style toolset — scrape, crawl, map, search, extract — that runs entirely on your machine, built to feed Claude Code (or any MCP client) **clean content with minimal context tokens**. No API keys, no SaaS, @@ -12,6 +16,12 @@ eval harness (verdict GREEN), and zero fabrications across the tool's entire rec history. Its documented failure mode is *omission*, and the guards exist to make every omission loud. See [Trust](#trust). +📖 **Documentation:** this README is the reference — install, tool/CLI reference, how it works, +the guard/`raw` contract, and known limitations are all below. An expanded +**[Wiki](https://github.com/Wynelson94/tearsheet/wiki)** is in progress. +Contributing? See [CONTRIBUTING.md](CONTRIBUTING.md) · Security? [SECURITY.md](SECURITY.md) · +History? [CHANGELOG.md](CHANGELOG.md). + ## Why A language model pays for every token of nav, ads, and footer it reads. tearsheet extracts diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..81ac29b --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,41 @@ +# Security Policy + +## Reporting a vulnerability + +Please report security issues **privately** — do not open a public issue for anything exploitable. + +- **Preferred:** GitHub's private vulnerability reporting — the **Security** tab → + **Report a vulnerability**. This is enabled on this repo and is the fastest route. +- **Or:** email the maintainer at **nmnslr@gmail.com** with details and, if possible, a + reproduction. + +Please allow a reasonable window for a fix before any public disclosure. As personal research +tooling maintained by one person, response is best-effort — but security reports are taken +seriously and are prioritized over feature work. + +## Supported versions + +Only the latest `main` is supported. There are no backports; fixes land on `main` and in the next +version. tearsheet is not published to PyPI — you run it from source, so *updating* means pulling +`main` and re-installing. + +## Scope & threat model + +tearsheet is a **local, self-hosted** tool. It has no server, no accounts, no telemetry, and +requires no API keys; nothing you scrape leaves your machine except the outbound HTTP(S) request +to the site you pointed it at. It handles no secrets or credentials. + +Security-relevant surface, for reporters: + +- **Fetches arbitrary URLs** you give it (`httpx`) and, on the render path, loads them in a + **headless Chromium** (Playwright). Treat scraped sites as untrusted input. +- **Parses untrusted content** — HTML (trafilatura / lxml / extruct), PDFs (pypdf), sitemaps. + Parser-level issues (resource exhaustion, XML entity handling, crashes on malformed input) are + in scope. +- **Writes to disk** under `~/.tearsheet` (override with `TEARSHEET_HOME`): a SQLite cache and + extracted markdown/crawl files. Path-handling issues are in scope. +- Follows and rate-limits per `robots.txt`; strips tracking parameters. It is **not** designed to + evade paywalls or bot defenses, and reports bot/consent walls rather than bypassing them. + +Out of scope: vulnerabilities in third-party sites you scrape, and upstream bugs in dependencies +(please report those to the relevant project — e.g. trafilatura — though a heads-up here is welcome).