🔬 watermark-bench

Stress-test AI text watermarks until they break.

A modular PyTorch benchmark for measuring how statistical AI watermarks survive text transformations and model-based attacks.

---

⚡ Motivation & Overview

As international regulatory frameworks (such as the EU AI Act (Art. 50)) mandate synthetic content provenance and AI text watermarking, the common assumption is that statistical watermarks are persistent and resilient.

watermark-bench measures watermark strength before and after an attack using $z$-score, $p$-value, BLEU, perplexity, and detection status. It is a research and testing tool, not an AI-content classifier: NO WATERMARK DETECTED does not mean that a text was written by a human.

---

🧪 Supported Watermark Schemes

Scheme	Reference	Mechanism
kirchenbauer	Kirchenbauer et al. (ICML 2023)	Green-list / Red-list logit biasing with hash seeding
synthid	Google DeepMind (Nature 2024)	Keyed pseudorandom $g$-values and tournament distortion
awt	Adaptive Watermarking	Dynamic logit bias scaling based on Shannon entropy $H(P)$
unigram	Aaronson et al. / Zhao et al. (2023)	Context-free static green-list unigram baseline
pos_aware	Kirchenbauer POS / Syntax-Aware	Contextual bias conditioning on lexical vs functional tokens

---

⚔️ Implemented Attack Vectors

Attack	CLI Option	Description
Homoglyph Substitution	--attack homoglyph	Replaces Latin letters with visually identical Cyrillic/Greek characters
Zero-Width Injection	--attack zero_width	Injects invisible characters (\u200b, \u200c, \u200d, \ufeff) to disrupt tokenization
Keyboard Typo Noise	--attack typo	Injects realistic keyboard proximity typos, omissions, and character transpositions
Synonym Substitution	--attack synonym	Replaces words using WordNet or an offline fallback dictionary
Cross-Model Paraphrase	--attack paraphrase	Rewrites text using a second Hugging Face causal model (--attacker-model)
API Paraphrase	--attack api_paraphrase	Asynchronous rewrite via Ollama / vLLM / OpenAI-compatible endpoint (--api-url)
Chained / Composite	--attack homoglyph+synonym	Sequential attack pipeline chaining multiple transformations
No attack / control	--attack none	Keeps the generated text unchanged as baseline
Back-Translation	--attack backtranslate	Stub: flagged as stub (is_stub = True)
Logit Noise	--attack logit_noise	Stub: flagged as stub (is_stub = True)

---

📦 Dataset Support

watermark-bench can load prompts from standard benchmark datasets or local files:

Source	CLI Flag	Description
Built-in presets	--dataset c4 / humaneval / wmt16 / qa	Curated prompt sets for reproducible benchmarks
JSONL file	--dataset path/to/prompts.jsonl	Each line: {"text": "..."} or {"prompt": "..."}
CSV file	--dataset path/to/prompts.csv	Column text or prompt
Plain text	--dataset path/to/prompts.txt	One prompt per line

Use --max-samples N to limit the number of prompts loaded.

---

📈 Metrics & Statistical Analysis

Beyond z-score, p-value, BLEU, and perplexity, the framework provides:

Metric	Module	Description
ROC / AUC	metrics.roc	Receiver Operating Characteristic curves and area under the curve
Threshold@FPR	metrics.roc	Optimal detection threshold for a target false positive rate
BERTScore F1	metrics.semantic	Contextual embedding similarity between original and attacked text
N-gram cosine	metrics.semantic	Character/word n-gram overlap similarity

---

🚀 Quickstart

There are two ways to use the project:

Goal	Command	What it does
Check an existing text	detect	Looks for a statistical watermark in the text
Run a robustness experiment	eval	Generates, attacks, detects, and saves a report
Compare multiple experiments	compare	Side-by-side table of JSON reports with optional CSV/Parquet export

detect does not determine whether a text was written by AI. It only checks for the selected watermark signature.

1. Installation

# Clone repository
git clone https://github.com/takzen/watermark-bench.git
cd watermark-bench

# Initialize virtualenv and install dependencies with uv
uv venv
uv pip install -e ".[dev]"

2. Verify the installation

On Windows PowerShell:

& ".\.venv\Scripts\watermark-bench.exe" version

Alternatively, use uv run watermark-bench if uv can access its cache.

3. Run a small robustness evaluation

& ".\.venv\Scripts\watermark-bench.exe" eval `
  --model "Qwen/Qwen2.5-0.5B-Instruct" `
  --watermark kirchenbauer `
  --attack synonym `
  --num-samples 1 `
  --max-new-tokens 128 `
  --output-report ".\reports\run_01.json"

The first run downloads the Hugging Face tokenizer and model weights. eval generates one sample of up to 128 tokens by default. Increase --num-samples and --max-new-tokens for a fuller benchmark; CPU execution may be slow, so a GPU is recommended for repeated evaluations.

4. Check an existing text

Put each text you want to check in work/input/. This directory is intended for local working files and its contents are excluded from Git, so a submitted text is not committed by accident.

For example, save the text as work/input/tekst.txt in UTF-8 encoding and run:

& ".\.venv\Scripts\watermark-bench.exe" detect `
  --input-file ".\work\input\tekst.txt" `
  --model "Qwen/Qwen2.5-0.5B-Instruct" `
  --watermark kirchenbauer

The command accepts exactly one source: --input-file for a file, or a direct argument for a short text:

& ".\.venv\Scripts\watermark-bench.exe" detect `
  "This is a text sample with several sentences to analyze." `
  --model "Qwen/Qwen2.5-0.5B-Instruct" `
  --watermark kirchenbauer

The --model value is used to load the tokenizer for detection. For meaningful results, use the same tokenizer that was used when the watermark was generated. Longer texts produce more stable statistics. Input files must use UTF-8 encoding.

5. Compare experiment reports

& ".\.venv\Scripts\watermark-bench.exe" compare `
  .\reports\run_01.json `
  .\reports\run_02.json `
  --export-csv .\reports\comparison.csv `
  --export-parquet .\reports\comparison.parquet

This renders a Rich terminal table with z-score shift bars and optionally exports results to CSV or Parquet for further analysis in Pandas or Jupyter.

---

📊 Evaluation Output Format

Evaluation results are rendered as an interactive terminal table and saved as structured JSON reports containing statistical telemetry:

{
  "version": "0.1.0",
  "config": {
    "model_id": "Qwen/Qwen2.5-0.5B-Instruct",
    "watermark_scheme": "kirchenbauer",
    "attack_type": "synonym"
  },
  "results": [
    {
      "sample_id": 1,
      "pre_attack_detection": {
        "z_score": 5.8421,
        "p_value": 0.000001,
        "is_watermarked": true,
        "green_fraction": 0.485
      },
      "post_attack_detection": {
        "z_score": 1.1215,
        "p_value": 0.1311,
        "is_watermarked": false,
        "green_fraction": 0.291
      },
      "quality": {
        "bleu": 82.4,
        "perplexity": 14.85
      }
    }
  ]
}

Interpretation:

• pre_attack_detection should normally report a detected watermark before the attack;
• post_attack_detection shows whether the watermark survived the transformation;
• STRIPPED means the post-attack score is below the detector threshold;
• a negative Δ z-score means that the attack weakened the signal;
• if the watermark is not detected before the attack, the attack result is not meaningful.

---

📂 Source Code Architecture

src/watermark_bench/
├── watermarks/
│   ├── base.py             # BaseWatermark — abstract generate() + detect()
│   ├── kirchenbauer.py     # Green-list logit bias (Kirchenbauer et al. 2023)
│   ├── synthid.py          # Tournament sampling (Google DeepMind 2024)
│   ├── awt.py              # Adaptive entropy-aware watermarking
│   ├── unigram.py          # Context-free static green-list baseline
│   └── pos_aware.py        # POS/syntax-aware variable γ/δ
├── attacks/
│   ├── base.py             # BaseAttack — abstract apply()
│   ├── synonym.py          # WordNet token substitution
│   ├── paraphrase.py       # Cross-model HF rewrite
│   ├── api_paraphrase.py   # Async Ollama / vLLM / OpenAI-compatible
│   ├── homoglyph.py        # Cyrillic/Greek visual substitution
│   ├── zero_width.py       # Invisible Unicode injection
│   ├── typo.py             # Keyboard proximity noise
│   ├── composite.py        # Chained attack pipeline
│   ├── backtranslate.py    # Stub
│   └── logit_noise.py      # Stub
├── metrics/
│   ├── detection.py        # z-score, p-value, BER
│   ├── quality.py          # SacreBLEU, Perplexity
│   ├── roc.py              # ROC curves, AUC, threshold selection
│   └── semantic.py         # BERTScore, n-gram cosine similarity
├── datasets/
│   └── loader.py           # DatasetLoader (C4, HumanEval, WMT16, QA, .jsonl/.csv/.txt)
├── models/
│   └── api_client.py       # AsyncLLMClient for Ollama / vLLM / OpenAI endpoints
├── pipeline.py             # EvalPipeline — orchestrates the full loop
├── config.py               # Pydantic v2 configuration schemas
└── cli.py                  # Typer CLI (eval, detect, compare, version)

Models and tokenizers are loaded through Hugging Face Transformers. Paraphrase attacks can also use external Ollama / vLLM / OpenAI-compatible endpoints via AsyncLLMClient.

---

🛠️ Development & Quality Assurance

These commands test the project code. They do not analyze a supplied text; use detect for that.

# Run unit tests
uv run pytest

# Type check with mypy
uv run mypy src/

# Lint & Format
uv run ruff check src/ tests/
uv run ruff format src/ tests/

On Windows, the same checks can be run directly from the virtual environment:

& ".\.venv\Scripts\python.exe" -m pytest
& ".\.venv\Scripts\python.exe" -m ruff check src tests
& ".\.venv\Scripts\python.exe" -m mypy src

The current test suite contains 57 passing unit tests.

---

🛡️ Kanari.pl — własne testy i audyty techniczne

watermark-bench is an open-source research and red-teaming engine. Kanari.pl currently offers expert-led testing rather than a SaaS platform or continuous production monitoring.

The service scope can include:

👉 Kanari.pl — Enterprise AI Governance & Compliance

• Custom watermark robustness tests for a selected model and tokenizer
• Controlled attacks, pre/post detection, and quality measurements
• Technical reports with configuration, results, limitations, and recommendations
• Sector-specific prompt and text suites prepared for individual engagements

Reports are technical assessments, not legal certification. References to EU AI Act Article 50 should be reviewed with qualified legal counsel where needed.

---

📜 License

Distributed under the MIT License. See LICENSE for details.
Copyright (c) 2026 Krzysztof Pika.