Add docs

.github/workflows/docs.yml

@@ -0,0 +1,57 @@
+name: docs
+
+on:
+  pull_request:
+  push:
+    branches:
+      - main
+      - master
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: docs
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Install docs dependencies
+        run: uv sync --extra docs
+
+      - name: Build docs
+        run: uv run mkdocs build --strict
+
+      - name: Upload Pages artifact
+        if: github.event_name != 'pull_request'
+        uses: actions/upload-pages-artifact@v3
+        with:
+          path: site
+
+  deploy:
+    if: github.event_name != 'pull_request'
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

README.md

@@ -2,6 +2,35 @@
 
 A small, typed framework to evaluate pipeline outputs for single-feature, single-entity, and multi-entity extraction tasks against gold data. It aligns predictions to gold, normalizes values (text/number/date/category), computes macro metrics per feature, aggregates totals, and writes timestamped logs.
 
+## Why this library exists
+
+This library is built for evaluation-driven development (EDD): the idea that extraction and classification workflows should be improved through a repeatable loop of gold-set design, implementation, evaluation, error review, and iteration.
+
+In many real projects, building the first workflow is not the hardest part. The harder part is setting up the evaluation layer around it quickly enough that the team actually keeps the discipline to measure performance, inspect mistakes, and improve behavior over time. That problem becomes even sharper when the workflow depends on domain knowledge the developer does not have directly, but a domain expert can provide in limited bursts.
+
+The typical loop looks like this:
+
+1. Define a representative gold test set with a domain expert.
+2. Build a baseline workflow or pipeline.
+3. Compare predictions against the gold data, calculate metrics, and visualize the results.
+4. Review grouped mistakes, refine the workflow, and repeat.
+
+This library is intended to standardize step 3. Instead of rebuilding comparison logic, metrics code, logging, and visualization for every project, you can focus on:
+
+1. defining the test set correctly
+2. building the workflow itself
+3. using this library to evaluate predictions against the gold set
+
+That lowers the cost of doing EDD and makes it easier to balance speed and quality in production-oriented development.
+
+Example task shapes include:
+
+- one label per text, such as sentiment or routing class: `SINGLE_FEATURE`
+- one structured object per record, such as key fields from an article: `SINGLE_ENTITY`
+- multiple entities per source, such as contracts extracted from a report: `MULTI_ENTITY`
+
+The most important design work still lives in the test set. That is where intended behavior, definitions, and useful biases are encoded for the task.
+
 ## Install
 
 Using `uv`:

docs/concepts/alignment.md

@@ -0,0 +1,98 @@
+# Alignment
+
+Alignment decides which predicted rows are compared against which gold rows. The library has two alignment strategies: indexed matching and entity matching.
+
+## `IndexAligner`
+
+`IndexAligner` is used for `SINGLE_FEATURE` and `SINGLE_ENTITY`.
+
+Current behavior:
+
+- reads `run_config.index_key_name`
+- builds a map from each gold key value to one gold row index
+- iterates through predicted rows and matches rows with the same key
+- returns matched pairs with similarity score `1.0`
+- also returns unmatched predicted indices and unmatched gold indices
+
+### Important implications
+
+- matching is exact equality on the key field
+- duplicate key values in the gold DataFrame are ambiguous and should be avoided
+- unmatched rows are not included in per-feature scoring
+
+## `EntityAligner`
+
+`EntityAligner` is used for `MULTI_ENTITY`.
+
+Current behavior:
+
+1. build candidate predicted/gold pairs
+2. compute similarity for each pair
+3. drop pairs below `minimum_similarity_threshold`
+4. optionally keep only the top `maximum_candidate_pairs`
+5. shuffle candidates using a fixed seed
+6. stable-sort by similarity descending
+7. greedily choose non-overlapping pairs
+
+The result is deterministic for a fixed seed and input order.
+
+## Matching modes
+
+### Exact matching
+
+When `matching_mode="exact"`:
+
+- the aligner checks only features where `is_mandatory_for_matching=True`
+- if any mandatory feature is unequal, pair similarity is `0.0`
+- otherwise pair similarity is `1.0`
+
+This is strict matching. There is no partial credit.
+
+### Weighted matching
+
+When `matching_mode="weighted"`:
+
+- every feature contributes a similarity score
+- the final score is a weighted average
+- weights come from `FeatureRule.weight_for_matching`
+
+Feature-level similarity rules:
+
+- `text`: token-set overlap after normalization
+- `category`: `1.0` if equal, else `0.0`
+- `number`: `1.0` if equal under tolerance rules, else `0.0`
+- `date`: `1.0` if equal under date tolerance rules, else `0.0`
+
+## Threshold behavior
+
+`minimum_similarity_threshold` applies after pair similarity is computed.
+
+- pairs below the threshold are ignored entirely
+- pairs at or above the threshold remain candidates for greedy selection
+
+Practical effect:
+
+- a high threshold increases precision of entity matching
+- a low threshold increases recall of candidate pairs but can create more ambiguous competition
+
+## Determinism and tie-breaking
+
+For weighted matching, the aligner:
+
+- seeds Python's `random` module with `random_tie_breaker_seed`
+- shuffles candidate pairs
+- then sorts by similarity descending using a stable sort
+
+This means equal-score pairs are resolved reproducibly for a given seed.
+
+## What alignment affects downstream
+
+- matched pairs feed the per-feature metrics and row accuracy calculations
+- unmatched predicted and unmatched gold entities feed the multi-entity presence summary
+- indexed tasks do not expose entity presence metrics because alignment is assumed to be explicit
+
+## Related pages
+
+- [Task Types](task-types.md)
+- [Feature Rules](feature-rules.md)
+- [Metrics](metrics.md)

docs/concepts/evaluation-driven-development.md

@@ -0,0 +1,142 @@
+# Why This Library Exists
+
+`extraction-testing` is designed to support evaluation-driven development (EDD) for extraction and classification workflows.
+
+The short version is:
+
+- building a first workflow is often feasible
+- building a disciplined evaluation layer around it is the part teams often postpone
+- postponing evaluation makes iteration less reliable and performance harder to explain
+- this library exists to standardize that evaluation layer
+
+## The practical problem
+
+Suppose you want to extract contracts from a monthly report.
+
+Implementing a first version of the workflow is usually straightforward enough:
+
+- parse the report
+- run the extraction logic
+- return structured predictions
+
+What slows teams down is the next step: turning that workflow into something they can improve systematically.
+
+To do that well, the team needs to:
+
+- compare predictions against a gold set
+- compute useful metrics
+- inspect mistakes in a structured way
+- visualize and communicate overall performance
+
+That work is important, but it is often rewritten from scratch for each project. As a result, teams either delay it or keep the evaluation layer too ad hoc to drive disciplined iteration.
+
+## The EDD loop
+
+The usual evaluation-driven loop looks like this:
+
+```mermaid
+flowchart LR
+    A["Domain expert defines gold set"] --> B["Developer builds baseline workflow"]
+    B --> C["Run workflow on evaluation set"]
+    C --> D["Compare predictions with gold data"]
+    D --> E["Compute metrics, logs, and charts"]
+    E --> F["Review grouped errors with domain expert"]
+    F --> G["Refine workflow or prompt"]
+    G --> C
+    E --> H["Share expected performance with stakeholders"]
+```
+
+This loop matters because it gives the team two different outputs:
+
+- an iteration loop for improving the workflow
+- a performance summary that can be shared with stakeholders
+
+## Where domain experts fit
+
+Many extraction problems are domain-heavy. A developer can usually build the software pipeline, but may not have enough domain knowledge to define intended behavior alone.
+
+That is why the gold set is so important.
+
+In practice, the domain expert often contributes in two places:
+
+1. **Gold-set design**
+   They help define representative examples, labels, extracted fields, and task boundaries.
+
+2. **Error review**
+   They explain what a mistake means in domain terms.
+
+One useful review pattern is:
+
+> If a junior analyst with no prior domain experience made this mistake, what would you say to explain what they misunderstood?
+
+That kind of feedback is especially valuable in LLM workflows, where iteration often happens through prompt improvements, task definitions, extra instructions, or clearer field specifications.
+
+## What this library replaces
+
+This library is intended to standardize the repeated evaluation work in the middle of the loop.
+
+It helps with:
+
+- comparing predictions to gold data
+- aligning rows or entities
+- applying feature-specific equality rules
+- calculating metrics
+- writing logs
+- generating basic visualizations
+
+It is a way to outsource the repeated engineering work in the evaluation layer so that developers do not have to rebuild it for every new workflow.
+
+## What this library does not replace
+
+This library does not replace:
+
+- domain expertise
+- gold-set design
+- workflow implementation
+- error analysis and iteration decisions
+
+The most important design work still lives in the evaluation set. That is where the task definition, intended behavior, and acceptable tradeoffs are encoded.
+
+## Why this matters in production
+
+In many production environments, there is a real tension between speed and quality:
+
+- shipping fast favors minimal implementation effort
+- shipping safely requires an evaluation loop with visible performance
+
+The goal of this library is to lower that tradeoff. If the evaluation layer is cheap to set up, teams are more likely to keep the discipline of measuring and improving behavior instead of relying on informal spot checks.
+
+## How the supported task types fit the motivation
+
+The task types in this library match common evaluation shapes:
+
+| Real task shape | Typical example | Library task type |
+|---|---|---|
+| One label per record | sentiment, routing label, final subfolder | `SINGLE_FEATURE` |
+| One structured object per record | key fields from an article or form | `SINGLE_ENTITY` |
+| Multiple entities per source | contracts, people, products, events in a report | `MULTI_ENTITY` |
+
+Examples:
+
+- classifying a text into one final category is usually `SINGLE_FEATURE`
+- extracting several key fields from one article is usually `SINGLE_ENTITY`
+- extracting multiple contracts from a report is usually `MULTI_ENTITY`
+- routing a document into a folder hierarchy can be `SINGLE_FEATURE` or `SINGLE_ENTITY`, depending on whether you score only the final label or the full path
+
+## A useful mental model
+
+Think of the library as the reusable evaluation layer around an extraction workflow.
+
+You still need to:
+
+- define the gold set
+- build the workflow
+- improve it over time
+
+But you do not need to write the comparison, metric, logging, and visualization code from scratch each time.
+
+## Related pages
+
+- [Choosing a Task Type](../getting-started/task-selection.md)
+- [Task Types](task-types.md)
+- [Metrics](metrics.md)