openattribution-org · jalexspringer · Feb 17, 2026 · Feb 17, 2026 · Feb 17, 2026 · Feb 17, 2026
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -18,44 +18,58 @@ Web attribution and compliance scanner. Checks robots.txt, RSL licenses, and TDM
 
 ## Architecture
 
-Single binary, seven modules:
+Cargo workspace with two crates:
 
 ```
-src/
-  main.rs        — CLI entry point (clap), dispatches to analyze or serve
-  ai_crawlers.rs — Canonical list of 26 AI crawlers (GPTBot, ClaudeBot, etc.)
-  analyzer.rs    — Core logic: parse robots.txt, extract licenses, evaluate TDM, analyze AI bots
-  fetcher.rs     — HTTP fetching for robots.txt and /.well-known/tdmrep.json
-  models.rs      — Data types: AnalysisResult, TdmPolicy, BotAnalysisResult, request/response shapes
-  output.rs      — Formatters: table, JSON, CSV (with AI bot columns), compact text
-  server.rs      — Axum HTTP API (GET /health, POST /analyze)
+crates/
+  core/                    — policycheck-core (pure library, no I/O, WASM-compatible)
+    src/
+      lib.rs               — PolicyAnalyzer facade, orchestrates all checks
+      ai_crawlers.rs       — Canonical list of 26 AI crawlers (GPTBot, ClaudeBot, etc.)
+      models.rs            — Data types: AnalysisResult, TdmPolicy, BotAnalysisResult
+      checks/
+        mod.rs             — Check module index
+        robots.rs          — RFC 9309 robots.txt parsing (user agents, paths, crawl delay)
+        rsl.rs             — RSL licence extraction (global, group-scoped, precedence)
+        content_signals.rs — Cloudflare Content Signals (search, ai-input, ai-train)
+        tdm.rs             — W3C TDMRep pattern matching and rule evaluation
+        ai_bots.rs         — Per-bot access analysis for 26 AI crawlers
+  cli/                     — policycheck (binary: CLI + HTTP server)
+    src/
+      main.rs              — CLI entry point (clap), dispatches to analyze or serve
+      analyzer.rs          — Network-aware analyzer wrapping core with HTTP fetching
+      fetcher.rs           — HTTP fetching for robots.txt and /.well-known/tdmrep.json
+      output.rs            — Formatters: table, JSON, CSV (with AI bot columns), compact text
+      server.rs            — Axum HTTP API (GET /health, POST /analyze)
 ```
 
-No workspaces, no proc macros, no feature flags. Keep it simple.
+The core crate has 4 dependencies (texting_robots, serde, serde_json, url) and no network I/O. The CLI crate owns reqwest, axum, and all I/O concerns. No proc macros, no feature flags.
 
 ## Commands
 
 ```bash
 # Build
-cargo build                    # Debug
+cargo build                    # Debug (whole workspace)
 cargo build --release          # Optimised (LTO, strip)
+cargo build -p policycheck     # CLI only
 
 # Test
-cargo test                     # All tests
+cargo test --workspace         # All tests (core + CLI)
+cargo test -p policycheck-core # Core library only
 cargo test -- --nocapture      # With stdout
 
 # Run - Single URL
-cargo run -- analyze --url https://www.nytimes.com
-cargo run -- analyze --url https://github.com --format json
+cargo run -p policycheck -- analyze --url https://www.nytimes.com
+cargo run -p policycheck -- analyze --url https://github.com --format json
 
 # Run - Bulk analysis with CSV export (advertiser use case)
-cargo run -- analyze --csv publishers.csv --format csv --output results.csv
+cargo run -p policycheck -- analyze --csv publishers.csv --format csv --output results.csv
 
 # Run - HTTP server
-cargo run -- serve --port 3000
+cargo run -p policycheck -- serve --port 3000
 
 # Lint
-cargo clippy
+cargo clippy --workspace --all-targets
 cargo fmt --check
 ```
 
@@ -69,27 +83,37 @@ cargo fmt --check
 
 ## Testing
 
-Tests live alongside code in `#[cfg(test)] mod tests` blocks. Currently in `analyzer.rs`:
+Tests live alongside code in `#[cfg(test)] mod tests` blocks. 53 tests across both crates:
 
-- Unit tests for robots.txt parsing (user agents, paths, licenses)
-- RSL licence extraction (global, group-scoped, precedence, absolute URI validation)
-- TDM pattern matching (wildcards, end markers, complex patterns)
-- TDM rule evaluation (async tests with `#[tokio::test]`)
+**Core (35 tests)** — pure unit tests, no I/O:
+- `checks::robots` — user agent extraction, path parsing, allow/disallow
+- `checks::rsl` — global/group-scoped licences, precedence, absolute URI validation
+- `checks::content_signals` — signal parsing, group scoping, Cloudflare format
+- `checks::tdm` — pattern matching (wildcards, `$` end markers), rule evaluation
+- `checks::ai_bots` — wildcard blocking, selective blocking, bot count
+- `lib.rs` — integration tests via `PolicyAnalyzer::analyze()`
+
+**CLI (17 tests)** — server, fetcher, output, CSV:
+- `server` — health check, empty URLs, too-many-URLs validation
+- `fetcher` — URL construction for robots.txt and TDM endpoints
+- `output` — CSV headers, comma escaping, JSON round-trip
+- `analyzer` — CSV column detection, bare domain prefixing, empty row skipping
 
 When adding features:
 1. Write `#[test]` or `#[tokio::test]` in the relevant module
 2. Make it fail
 3. Implement until green
-4. `cargo clippy` + `cargo fmt`
+4. `cargo clippy --workspace --all-targets` + `cargo fmt`
 
 ## Standards Implemented
 
 | Standard | Status | Where |
 |----------|--------|-------|
-| RFC 9309 (Robots Exclusion Protocol) | Done | `analyzer.rs` via `texting_robots` |
-| RSL (Responsible Sourcing License) | Done | `analyzer.rs::extract_licenses()` |
-| W3C TDMRep | Done | `fetcher.rs::fetch_tdm_policy()`, `analyzer.rs::evaluate_tdm_policy()` |
-| AI Crawler Analysis | Done | `ai_crawlers.rs`, `analyzer.rs::analyze_ai_bots()` |
+| RFC 9309 (Robots Exclusion Protocol) | Done | `core::checks::robots` via `texting_robots` |
+| RSL (Responsible Sourcing License) | Done | `core::checks::rsl` |
+| Cloudflare Content Signals | Done | `core::checks::content_signals` |
+| W3C TDMRep | Done | `cli::fetcher` + `core::checks::tdm` |
+| AI Crawler Analysis | Done | `core::ai_crawlers` + `core::checks::ai_bots` |
 | RFC 9116 (security.txt) | Planned | — |
 | RFC 8615 (Well-Known URIs) | Planned | — |
 

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -1,44 +1,15 @@
-[package]
-name = "policycheck"
-version = "0.2.1"
+[workspace]
+members = ["crates/core", "crates/cli"]
+resolver = "2"
+
+[workspace.package]
+version = "0.3.0"
 edition = "2021"
 rust-version = "1.75"
 authors = ["OpenAttribution Contributors"]
-description = "Publisher policy compliance checker - verifies robots.txt, RSL licenses, Content Signals, and TDM policies"
-readme = "README.md"
 license = "MIT"
 repository = "https://github.com/openattribution-org/policycheck"
 homepage = "https://openattribution.org"
-keywords = ["robots", "rsl", "tdm", "policy", "compliance"]
-categories = ["web-programming", "command-line-utilities"]
-exclude = [
-    ".github/",
-    ".dockerignore",
-    "Dockerfile",
-    "fly.toml",
-    "CLAUDE.md",
-    "*.csv",
-    "!example.csv"
-]
-
-[dependencies]
-texting_robots = "0.2"
-reqwest = { version = "0.12", default-features = false, features = ["rustls-tls"] }
-clap = { version = "4.5", features = ["derive"] }
-csv = "1.3"
-serde = { version = "1.0", features = ["derive"] }
-serde_json = "1.0"
-anyhow = "1.0"
-tokio = { version = "1", features = ["full"] }
-axum = "0.7"
-tower = { version = "0.4", features = ["util"] }
-http-body-util = "0.1"
-tower-http = { version = "0.5", features = ["cors"] }
-comfy-table = "=7.1.1"
-url = "2.5"
-
-[dev-dependencies]
-tempfile = "3"
 
 [profile.release]
 opt-level = 3

diff --git a/Dockerfile b/Dockerfile
@@ -5,12 +5,10 @@ WORKDIR /app
 
 # Copy manifests
 COPY Cargo.toml Cargo.lock ./
+COPY crates ./crates
 
-# Copy source
-COPY src ./src
-
-# Build release binary
-RUN cargo build --release
+# Build release binary (CLI server only — skip WASM crate)
+RUN cargo build --release -p policycheck
 
 # Runtime stage
 FROM debian:bookworm-slim

diff --git a/README.md b/README.md
@@ -17,7 +17,7 @@ PolicyCheck helps you **scrape responsibly** by checking multiple compliance sig
 - ✅ **Robots.txt** - What paths you can crawl (REP/RFC 9309)
 - 📜 **RSL Licenses** - Required licensing terms (Responsible Sourcing License)
 - 🎯 **Content Signals** - AI usage preferences (Cloudflare's policy framework)
-- 🤖 **TDM Policies** - Text & Data Mining permissions (coming soon)
+- 🤖 **TDM Policies** - Text & Data Mining permissions (W3C TDMRep)
 - 🔒 **Privacy Controls** - DNT, GPC signals (coming soon)
 - 📧 **Security Contacts** - Who to contact about scraping (coming soon)
 
@@ -65,11 +65,31 @@ For development or the latest unreleased features:
 ```bash
 git clone https://github.com/openattribution-org/policycheck.git
 cd policycheck
-cargo build --release
+cargo build --release -p policycheck
 ```
 
 The binary will be at `target/release/policycheck`.
 
+#### As a Library
+
+Use the core parsing library in your own project (no network I/O, WASM-compatible):
+
+```bash
+cargo add policycheck-core
+```
+
+```rust
+use policycheck_core::PolicyAnalyzer;
+
+let analyzer = PolicyAnalyzer::new("GPTBot".to_string());
+let result = analyzer.analyze(
+    "https://www.nytimes.com",
+    "User-agent: GPTBot\nDisallow: /\n",
+    None,
+);
+assert!(!result.is_path_allowed);
+```
+
 ### Basic Usage
 
 ```bash
@@ -617,10 +637,11 @@ Multi-platform images available for `linux/amd64` and `linux/arm64`.
 #### Building from Source
 
 ```dockerfile
-FROM rust:1.92-slim as builder
+FROM rust:1.85-bookworm as builder
 WORKDIR /app
-COPY . .
-RUN cargo build --release
+COPY Cargo.toml Cargo.lock ./
+COPY crates ./crates
+RUN cargo build --release -p policycheck
 
 FROM debian:bookworm-slim
 RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
@@ -711,9 +732,10 @@ podman-compose up -d
 - [x] CSV batch processing
 - [x] HTTP API server
 - [x] Multiple output formats
+- [x] TDM (Text & Data Mining) policy detection (`/.well-known/tdmrep.json`)
+- [x] Content Signals (Cloudflare AI policy framework)
 
 ### 🚧 In Progress
-- [ ] TDM (Text & Data Mining) policy detection (`/.well-known/tdmrep.json`)
 - [ ] Security contact discovery (`/.well-known/security.txt`)
 - [ ] Privacy control detection (DNT, GPC)
 
@@ -797,7 +819,7 @@ PolicyCheck implements the following standards:
 - ✅ **RFC 9309**: Robots Exclusion Protocol (REP)
 - ✅ **RSL Standard**: Responsible Sourcing License
 - ✅ **Content Signals**: Cloudflare's AI Policy Framework (CC0 License)
-- 🚧 **W3C TDMRep**: Text and Data Mining Reservation Protocol (planned)
+- ✅ **W3C TDMRep**: Text and Data Mining Reservation Protocol
 - 🚧 **RFC 9116**: security.txt (planned)
 - 🚧 **RFC 8615**: Well-Known URIs (planned)