feat(analyzer): Add golden-dataset evaluation harness with per-PR quality report

.github/workflows/ci.yml

@@ -204,6 +204,59 @@ jobs:
           pip install --require-hashes -r ${{ github.workspace }}/.github/pipelines/requirements-build.txt
           python -m build --wheel
 
+  analyzer-evaluation:
+    name: Analyzer Evaluation Report
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    env:
+      UV_CACHE_DIR: /mnt/uv-cache
+      UV_PROJECT_ENVIRONMENT: /mnt/uv-venv
+      UV_LINK_MODE: hardlink
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+        with:
+          python-version: '3.13'
+
+      - name: Install uv
+        run: python -m pip install uv==0.11.6
+
+      - name: Prepare uv directories on /mnt
+        run: |
+          sudo mkdir -p "$UV_CACHE_DIR" "$UV_PROJECT_ENVIRONMENT"
+          sudo chown -R "$(id -u):$(id -g)" "$UV_CACHE_DIR" "$UV_PROJECT_ENVIRONMENT"
+
+      - name: Cache uv downloads
+        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.0
+        with:
+          path: ${{ env.UV_CACHE_DIR }}
+          key: uv-eval-${{ runner.os }}-${{ hashFiles('presidio-analyzer/uv.lock') }}
+          restore-keys: |
+            uv-eval-${{ runner.os }}-
+
+      - name: Install dependencies
+        working-directory: presidio-analyzer
+        run: |
+          # The evaluation runs the default (spaCy) configuration only, so no
+          # extras are needed; dev group provides pytest and pip (spaCy's
+          # model download shells out to pip).
+          uv sync --locked --group dev --python 3.13
+          uv run --no-sync python -m spacy download en_core_web_lg
+
+      - name: Run golden-dataset evaluation
+        working-directory: presidio-analyzer
+        run: |
+          # Report-only for now: the report is published to the workflow
+          # summary; regression gating against baselines is a follow-up.
+          uv run --no-sync python -m tests.evaluation.run_evaluation \
+            --output evaluation_report.md
+          cat evaluation_report.md >> "$GITHUB_STEP_SUMMARY"
+
   build-platform-images:
     name: Build ${{ matrix.image }} (${{ matrix.platform }})
     runs-on: ${{ matrix.runner }}

presidio-analyzer/tests/evaluation/README.md

@@ -0,0 +1,142 @@
+# Analyzer evaluation harness
+
+A span-level evaluation harness for measuring analyzer detection quality
+(per-entity precision / recall / F1) and latency against a curated,
+hand-annotated golden dataset. It addresses
+[#1639](https://github.com/data-privacy-stack/presidio/issues/1639) (evaluate
+precision/recall/latency during CI) and
+[#1810](https://github.com/data-privacy-stack/presidio/issues/1810) (curated
+PII/PHI benchmark dataset), and lays the groundwork for the recipes comparison
+in [#1809](https://github.com/data-privacy-stack/presidio/issues/1809).
+
+## Running it
+
+From the `presidio-analyzer` directory (requires `en_core_web_lg`):
+
+```sh
+python -m tests.evaluation.run_evaluation            # report to stdout
+python -m tests.evaluation.run_evaluation --output report.md
+python -m tests.evaluation.run_evaluation --iou 1.0  # exact-span matching
+
+# evaluate another analyzer configuration (e.g. transformers):
+python -m tests.evaluation.run_evaluation \
+    --analyzer-conf path/to/analyzer_conf.yaml \
+    --write-baseline tests/evaluation/baselines/my_config.json
+
+# enforce the baseline locally (CI does not do this yet):
+python -m tests.evaluation.run_evaluation --fail-on-regression
+```
+
+The same evaluation also runs as part of the regular test suite
+(`pytest tests/evaluation`), and a CI job publishes the report to the
+workflow step summary on every PR.
+
+## Design decisions
+
+**Enforcement is built in but switched off.** The report compares every run
+against the checked-in baseline (`baselines/spacy_en.json`) and shows
+per-entity F1 deltas; `--fail-on-regression` exits non-zero when overall or
+per-entity F1 drops more than `--f1-tolerance` (default 0.02) below the
+baseline. CI runs report-only: switching enforcement on is a one-line CI
+change, deliberately left as a maintainer decision to take once the numbers
+have been observed to be stable across real PRs. Gating on day one would
+risk blocking contributor PRs on flaky or contested thresholds — e.g. a new
+`en_core_web_lg` release can shift NER results with no code change.
+
+**Updating the baseline** is part of the PR that changes behavior:
+`python -m tests.evaluation.run_evaluation --write-baseline
+tests/evaluation/baselines/spacy_en.json`, committed alongside the code, so
+reviewers see the metric change explicitly in the diff.
+
+**Curated golden set now, synthetic generation later.** A checked-in,
+hand-annotated dataset is deterministic, reviewable in diffs, and cheap to
+run per-PR. The template + Faker synthetic generation described in #1639 is
+the right tool for scaling coverage with every new recognizer, and is planned
+as a follow-up phase — the evaluator here consumes plain annotated samples,
+so generated data plugs into the same pipeline.
+
+**Small internal evaluator instead of depending on presidio-evaluator.** The
+matching logic needed for CI (span IoU matching, per-entity counts, a
+markdown report) is a few hundred lines. Vendoring it avoids coupling CI to
+an external research repo and keeps install time minimal. If it grows, it can
+graduate to a standalone package.
+
+**Raw analyzer output is evaluated.** The analyzer intentionally returns
+overlapping candidates of different types (e.g. `PHONE_NUMBER` over an SSN)
+and leaves conflict resolution to the anonymizer. The report therefore counts
+such overlaps as false positives — that is useful signal about analyzer
+precision, but it means per-entity precision here is a lower bound on
+end-to-end precision after anonymizer conflict resolution.
+
+**Evaluated entities are a fixed allowlist.** Predictions for entity types
+not declared in the dataset (`entities` in `golden_en.json`) are ignored, so
+recognizers without golden coverage don't pollute the report with
+unreviewable false positives. Adding coverage for a new entity = adding
+annotated samples + the entity to the allowlist.
+
+**Per-PR CI evaluates the default configuration only.** That covers spaCy
+NER (`PERSON`, `LOCATION`, `DATE_TIME`) plus the predefined pattern/checksum
+recognizers. HuggingFace, GLiNER and other NER recognizers are optional
+extras with large model downloads, so they are not run per-PR; the runner
+already supports them via `--analyzer-conf <yaml>` with a separate baseline
+per configuration (`--write-baseline`). Wiring those configurations into a
+scheduled nightly job — where model download cost is paid once a day, not
+per PR — is roadmap step 4.
+
+## Dataset
+
+`datasets/golden_en.json` — 46 English samples, 93 annotated spans across 12
+entity types (`PERSON`, `LOCATION`, `DATE_TIME`, `EMAIL_ADDRESS`,
+`PHONE_NUMBER`, `CREDIT_CARD`, `US_SSN`, `IP_ADDRESS`, `URL`, `IBAN_CODE`,
+`CRYPTO`, `UK_NHS`), organized in categories per #1810:
+
+- `simple` — single-entity one-liners
+- `medium` — multi-entity texts (support tickets, clinical notes, logs)
+- `long` — full documents (discharge summary, incident report, KYC)
+- `edge` — lowercase/inverted names, hyphenated names, adjacent entities,
+  entities at text boundaries, inline IDs
+- `negative` — no PII, but tempting lookalikes (version numbers, room
+  numbers, quantities)
+
+Entity values are checksum-valid where recognizers validate (Luhn for cards,
+mod-97 for IBANs, NHS check digit, SSN excluded ranges), reusing values from
+the unit tests where possible. Span offsets are validated by
+`test_golden_dataset.py::TestDatasetIntegrity`, so every annotation is
+guaranteed to match its text slice.
+
+### Extending the dataset
+
+`generate_golden_en.py` is the source of truth — samples are defined as
+interleaved text parts and `(value, entity_type)` tuples, so offsets are
+computed rather than hand-counted. To add coverage (e.g. for a new
+recognizer):
+
+1. Add samples to `SAMPLES` in `generate_golden_en.py` (and the entity type
+   to `ENTITIES` if new).
+2. Regenerate: `python -m tests.evaluation.generate_golden_en`
+3. Commit both files; a sync test fails if the JSON drifts from the
+   generator, so the JSON can never be hand-edited.
+
+### Other languages
+
+The dataset format and evaluator are language-agnostic: one file per
+language (`golden_en.json` today, e.g. `golden_de.json` later), each
+declaring its `language` and evaluated entity list. What is still
+English-only is the runner, which builds a default `AnalyzerEngine`;
+supporting another language means adding a per-language engine
+configuration (NLP model + registry languages) to `run_evaluation.py` and
+installing that language's model in the CI job. Planned alongside the
+nightly multi-configuration matrix (roadmap step 4).
+
+## Roadmap
+
+1. **(this PR)** Evaluator + golden dataset + baselines and regression
+   detection (enforcement off) + report-only CI job.
+2. Switch `--fail-on-regression` on in CI once metrics have proven stable —
+   a one-line CI change plus a baseline-update workflow note in
+   CONTRIBUTING.
+3. Synthetic data generation from templates + Faker providers; contributing
+   a recognizer requires contributing templates (see #1639).
+4. Nightly multi-configuration matrix (spaCy / transformers / GLiNER / LLM)
+   with per-configuration baselines and non-English datasets, feeding the
+   fast/balanced/accurate recipes comparison (#1809).

presidio-analyzer/tests/evaluation/__init__.py

@@ -0,0 +1,26 @@
+"""Span-level evaluation harness for the Presidio analyzer.
+
+This package contains a curated golden dataset and a small evaluator that
+measures per-entity precision/recall/F1 and latency for an analyzer
+configuration. See tests/evaluation/README.md for the design and roadmap.
+"""
+
+from tests.evaluation.evaluator import (
+    EntityMetrics,
+    EvaluationResult,
+    EvaluationSample,
+    GoldSpan,
+    SpanEvaluator,
+    SpanMismatch,
+    load_golden_dataset,
+)
+
+__all__ = [
+    "EntityMetrics",
+    "EvaluationResult",
+    "EvaluationSample",
+    "GoldSpan",
+    "SpanEvaluator",
+    "SpanMismatch",
+    "load_golden_dataset",
+]

presidio-analyzer/tests/evaluation/baseline.py

@@ -0,0 +1,98 @@
+"""Baseline persistence and regression comparison for evaluation results.
+
+A baseline is a checked-in snapshot of per-entity metrics for one analyzer
+configuration. Comparing a fresh evaluation against it turns the report's
+absolute numbers into deltas, and — once enforcement is switched on via
+``--fail-on-regression`` — lets CI fail when an entity's F1 drops by more
+than a tolerance.
+
+Throughput is deliberately not part of the baseline: it depends on the
+machine running the evaluation and would make regressions non-reproducible.
+"""
+
+import json
+from pathlib import Path
+from typing import Dict, List
+
+from tests.evaluation.evaluator import EvaluationResult
+
+DEFAULT_F1_TOLERANCE = 0.02
+
+
+def default_baseline_path() -> Path:
+    """Path of the checked-in baseline for the default (spaCy) config."""
+    return Path(__file__).parent / "baselines" / "spacy_en.json"
+
+
+def result_to_baseline(result: EvaluationResult, config_name: str) -> Dict:
+    """Snapshot an evaluation result as a baseline dict."""
+    return {
+        "config": config_name,
+        "overall": {
+            "precision": round(result.precision, 4),
+            "recall": round(result.recall, 4),
+            "f1": round(result.f1, 4),
+        },
+        "per_entity": {
+            entity: {
+                "support": metrics.support,
+                "precision": round(metrics.precision, 4),
+                "recall": round(metrics.recall, 4),
+                "f1": round(metrics.f1, 4),
+            }
+            for entity, metrics in sorted(result.per_entity.items())
+        },
+    }
+
+
+def save_baseline(result: EvaluationResult, config_name: str, path: Path) -> None:
+    """Write a baseline snapshot to disk."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    baseline = result_to_baseline(result, config_name)
+    path.write_text(json.dumps(baseline, indent=2) + "\n", encoding="utf-8")
+
+
+def load_baseline(path: Path) -> Dict:
+    """Load a baseline snapshot from disk."""
+    with open(path, encoding="utf-8") as f:
+        return json.load(f)
+
+
+def compare_to_baseline(
+    result: EvaluationResult,
+    baseline: Dict,
+    f1_tolerance: float = DEFAULT_F1_TOLERANCE,
+) -> List[str]:
+    """Compare a result against a baseline and describe regressions.
+
+    :param result: Fresh evaluation result.
+    :param baseline: Baseline dict, as produced by :func:`result_to_baseline`.
+    :param f1_tolerance: Maximum allowed F1 drop before it counts as a
+        regression, both overall and per entity type.
+    :return: Human-readable regression descriptions; empty when the result
+        is within tolerance. Entities absent from the baseline are skipped
+        (they are new coverage, not regressions).
+    """
+    regressions = []
+
+    overall_drop = baseline["overall"]["f1"] - result.f1
+    if overall_drop > f1_tolerance:
+        regressions.append(
+            f"overall F1 dropped {overall_drop:.3f} "
+            f"(baseline {baseline['overall']['f1']:.3f}, "
+            f"current {result.f1:.3f}, tolerance {f1_tolerance})"
+        )
+
+    for entity, metrics in sorted(result.per_entity.items()):
+        baseline_entity = baseline["per_entity"].get(entity)
+        if baseline_entity is None:
+            continue
+        drop = baseline_entity["f1"] - metrics.f1
+        if drop > f1_tolerance:
+            regressions.append(
+                f"{entity} F1 dropped {drop:.3f} "
+                f"(baseline {baseline_entity['f1']:.3f}, "
+                f"current {metrics.f1:.3f}, tolerance {f1_tolerance})"
+            )
+
+    return regressions