Visual Verifier

Anonymization QA for images and videos

Test blur, redaction, masking, and other visual processing frame by frame. Get a deterministic PASS/FAIL, temporal evidence, and CI-ready reports.

Anonymizers modify. Visual Verifier verifies.

🔒 Runs entirely on your machine. Your media is never uploaded — the package has no network dependency at all, and a test enforces it on every commit.

Visual Verifier brings software-testing discipline to processed image and video output. Instead of trusting that an anonymization or visual-processing job completed, it independently checks what happened frame by frame, verifies required regions when targets are supplied, and produces reproducible PASS/FAIL evidence for CI and review.

pip install visual-verifier
visual-verifier demo

FAIL
Frames missed: 4, 8, 12

---

The problem

Your anonymizer says the video passed. Frame 4 still exposes the licence plate.

A blur, redaction, or masking pipeline reports success. Three frames out of fifteen were silently skipped. A global image metric averages the failure away. Manual frame review does not scale and does not reproduce.

Visual Verifier finds the frames your privacy pipeline missed, and answers one narrow question with evidence you can attach to a build:

Did the expected visual processing occur, in every frame, and can I catch it in CI when it does not?

Anonymization is the named use case, not the limit

The engine is a general pixel-level change verifier. It measures what actually changed between a reference and a candidate — region by region, frame by frame — so anything that alters pixels can be verified the same way:

Use it for	What a FAIL means
Blur, pixelation, redaction, masking	A frame was left unprotected
Watermark and overlay application	The mark is missing or too faint
Encoding, transcoding, filter chains	A stage silently did nothing
Compositing, inpainting, style transfer	The edit did not land everywhere
Any processed-media regression test	Output drifted from the reference

Privacy work is where the failure is most expensive, so it is what the documentation leads with. The verification contract itself makes no assumption about why the pixels changed.

The missing test after visual processing

Image and video pipelines are usually tested as software: the function ran, the model returned detections, the encoder produced a file, and the job exited successfully.

None of that proves the output media is correct.

An anonymizer can complete successfully while missing one frame. A masking pipeline can modify the frame while missing the required face or licence plate. A watermark job can finish while the mark disappears during part of a video. In every case the exit code is zero and the media is wrong.

Visual Verifier tests the produced media itself. It is not an anonymizer; it is the independent test that runs after one:

any anonymizer, redactor, or image/video processor
                      ↓
               processed output
                      ↓
              VISUAL VERIFIER
                      ↓
        Did the transformation happen?
        Did it happen in every frame?
        Did it cover the required target?
        Where exactly did it fail?
                      ↓
           PASS / FAIL + evidence
                      ↓
             fix → rerun → verify

Instead of asking only whether the software executed, it asks whether the expected visual transformation actually appeared, frame by frame, and — when reviewed targets are supplied — in the required region.

Think of it as regression testing for processed visual media: deterministic PASS/FAIL, the exact failing frames and regions, reviewable evidence, and CI-ready exit codes.

Your code has tests. Why shouldn't your processed media?

That also means tools like face blurrers, plate redactors, and OCR scrubbers are not competitors. They are upstream systems this verifies.

What it looks like

Fifteen dashcam frames. A licence-plate blur that silently skipped three of them. Watch the verdict flip on frames 4, 8, and 12:

The same run on the command line:

$ visual-verifier video \
    --reference examples/media/video_raw.mp4 \
    --candidate examples/media/video_blur_partial.mp4 \
    --output outputs/partial_check

Visual Verifier result
======================
Status:                    FAIL
Policy:                    generic_change_every_frame
Reference:                 examples/media/video_raw.mp4
Candidate:                 examples/media/video_blur_partial.mp4

Measurements
------------
Frames checked:            15
Frames with processing:    12
Frames without processing: 3
Processing coverage:       80.0%
Accepted regions:          32
Rejected regions:          0

Temporal tracking
-----------------
Tracks:                    15
Track observations:        32
Track events:              53
Tracks with gaps:          1
Mean continuity ratio:     0.986667

Failures
--------
UNPROCESSED_FRAMES: No accepted processing was detected in frames [4, 8, 12].
Failed frames:             4, 8, 12

Evidence
--------
annotated_video:           outputs/partial_check/annotated_video.mp4
frame_report:              outputs/partial_check/frame_report.csv
html_report:               outputs/partial_check/index.html
region_report:             outputs/partial_check/region_report.csv
rejected_region_report:    outputs/partial_check/rejected_region_report.csv
summary_json:              outputs/partial_check/summary.json
track_event_report:        outputs/partial_check/track_event_report.csv
track_observation_report:  outputs/partial_check/track_observation_report.csv
track_report:              outputs/partial_check/track_report.csv

$ echo $?
2

Frames 4, 8, and 12 were never blurred. The exit code fails the build, the CSV reports say exactly where, and annotated_video.mp4 shows a reviewer the same thing in ten seconds.

Install

Two ways in, depending on where you want the check to run.

Python / CLI — requires Python 3.10 or newer.

pip install visual-verifier
visual-verifier doctor

Or with uv:

uv tool install visual-verifier      # as a standalone CLI
uv add visual-verifier               # as a project dependency

GitHub Actions — no install step, straight into a workflow.

uses: SAMtheROCKET/visual-verifier@v0.3.0

To work on Visual Verifier itself, clone and run uv sync. On Windows PowerShell a bootstrap script sets the same environment up:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force
Unblock-File ./scripts/bootstrap_uv.ps1
./scripts/bootstrap_uv.ps1

visual-verifier doctor reports your Python, OpenCV, and codec support, so you can confirm the environment before trusting a result.

Quick start

No clone, no fixtures to download, no media of your own required:

visual-verifier demo

That generates a short sample clip in which a licence plate is blurred on every frame except three, verifies it, writes the full evidence set to ./visual-verifier-demo/report/, and prints the exact command to reproduce it. The sample is synthesized locally, so nothing is fetched.

Then point it at your own media:

visual-verifier video \
    --reference raw.mp4 \
    --candidate anonymized.mp4 \
    --output outputs/check

Add --json for a machine-readable document, --quiet to report only through the exit code, and visual-verifier video --help for every threshold.

Use it in CI

Your pipeline passed. Did the pixels?

On GitHub, the bundled action — available on the GitHub Marketplace — installs the tool, runs the comparison, writes a job summary, and uploads the evidence:

- name: Verify anonymization coverage
  uses: SAMtheROCKET/visual-verifier@v0.3.0
  with:
    reference: fixtures/source.mp4
    candidate: build/anonymized.mp4

Also available on GitHub Marketplace as a reusable Action.

The summary names the failing frames, draws a frame timeline, and links the evidence artifact, so a reviewer sees the answer without downloading anything. docs/github_action.md lists every input and output.

Everywhere else the exit code is the contract, so no wrapper is required:

- name: Verify anonymization coverage
  run: |
    visual-verifier video \
      --reference fixtures/source.mp4 \
      --candidate build/anonymized.mp4 \
      --output artifacts/verification \
      --quiet

Exit code	Meaning
0	Verification completed and passed
1	Verification could not be completed
2	Verification completed and failed

Errors print a stable machine-greppable code, for example ERROR [MEDIA_READ_ERROR], so a failing build tells you whether the media was bad or the processing was.

Python API

from visual_verifier import TrackingConfig, verify_video

result = verify_video(
    reference="examples/media/video_raw.mp4",
    candidate="examples/media/video_blur_partial.mp4",
    output_dir="outputs/partial_check",
    tracking_config=TrackingConfig(
        association_iou_threshold=0.20,
        minimum_confirmation_hits=2,
        maximum_gap_frames=3,
        lineage_overlap_threshold=0.20,
    ),
)

print(result.status.value)  # FAIL
print(result.failed_frames)  # (4, 8, 12)
print(result.measurements["tracking"]["track_count"])

Use it as a test assertion:

def test_every_frame_is_anonymized() -> None:
    verify_video("source.mp4", "anonymized.mp4").raise_for_failure()

Disable tracking while preserving frame-level verification:

result = verify_video("reference.mp4", "candidate.mp4", enable_tracking=False)

The package is fully typed and ships py.typed. Everything you need is exported from the top level: verify_image, verify_video, DetectionConfig, TrackingConfig, VerificationResult, VerificationStatus, and the structured error hierarchy rooted at VisualVerifierError.

How it compares

Averaging the difference between two frames is the check most teams reach for first. The Anonymization Gap Benchmark measures what that costs, over 78 generated sequences carrying 13 labelled failure modes:

Method	Recall	False alarms	Localizes region	Operating point
Visual Verifier + targets	100%	0.0%	Yes	reviewed targets + severity floor 50
Visual Verifier	79%	0.0%	Yes	shipped defaults
Mean pixel difference	42%	0.0%	No	threshold tuned on calibration
PSNR threshold	42%	0.0%	No	threshold tuned on calibration
SSIM threshold	42%	0.0%	No	threshold tuned on calibration

The global metrics are handed a held-out calibration set to pick their best threshold. Visual Verifier runs at shipped defaults. They still drop to 0% recall on compressed, noisy, and offset-blur footage, because a metric that averages the whole frame cannot separate a missing blur from a re-encode. Region-by-region measurement can.

Three families defeat plain change detection entirely: weak blur, partially covered targets, and a missed plate among several. Every method above scores 0% on all three, because all three are the same gap: something was processed, but not the thing that mattered.

Telling the tool which region had to be anonymized closes all three and takes the benchmark to 100%:

visual-verifier video --reference raw.mp4 --candidate out.mp4 \
    --targets plates.csv --min-severity 50

See Target annotation.

Verification contract

Reference media
+ Candidate processed media
+ Detection configuration
+ Optional tracking configuration
= PASS/FAIL + measurements + temporal evidence

With processing required on every frame:

• PASS: every synchronized frame contains an accepted changed region.
• FAIL: one or more frames contain no accepted changed region.

The bundled regression fixture is enforced by examples/expected/demo_expectations.json, which the test suite executes on every run so the table below cannot drift:

Comparison	Status	Failed frames
Raw vs. fully blurred	PASS	None
Raw vs. partially blurred	FAIL	4, 8, 12

Verifying a specific region

Supplying reviewed targets changes the question from did anything change to did the required region change:

visual-verifier video \
    --reference raw.mp4 \
    --candidate anonymized.mp4 \
    --targets plates.csv

A frame where two of three plates were blurred passes the generic check and fails this one. Target boxes between reviewed frames are interpolated and always marked as such, so a generated box is never mistaken for one a human drew. docs/target_annotation.md documents the file format and every limit.

Evidence outputs

Video verification with tracking enabled writes:

index.html                   Self-contained report you can email to a reviewer
annotated_video.mp4          Candidate with persistent track labels (T001)
frame_report.csv             One row per synchronized frame
region_report.csv            One row per accepted changed region
rejected_region_report.csv   One row per rejected region, with reasons
track_report.csv             One row per completed track
track_observation_report.csv One row per region-to-track assignment
track_event_report.csv       Lifecycle and split/merge lineage events
summary.json                 The complete machine-readable result

Every panel below is generated from a real run by scripts/generate_readme_assets.py, with the region boxes read back from track_observation_report.csv:

The report you actually send someone

index.html opens in any browser and needs no explanation of CSV schemas. It shows the verdict, a frame timeline where every unprotected frame links to its own evidence, and a before/after wipe slider for each failure.

It references nothing external — no CDN, no web font, no linked image. Thumbnails are embedded, so it works offline, survives being attached to a ticket, and opening it cannot signal to anyone that a privacy report exists. It is also deterministic, so two runs over the same media produce identical documents and can be diffed.

The primary temporal measures are continuity ratio, fragmentation index, missing frames and longest gap, mean and minimum association IoU, center jitter, area stability, recovery count, and split/merge involvement. See docs/temporal_tracking.md and docs/output_schema.md.

Current release

Version 0.3.0 is the V5.3 milestone: target-aware verification. The package is early software classified Alpha: the API and report schemas may still change before a stable release.

• Image-to-image and synchronized video verification
• Multiple changed-region detection, filtering, and severity scoring
• Reviewed targets declaring the regions that must be anonymized, with union coverage, bounded interpolation, and explicit provenance
• Deterministic IoU-based temporal association
• Tentative, confirmed, lost, recovered, and closed track lifecycles
• Configurable short-gap tolerance and split/merge lineage evidence
• Track continuity, fragmentation, stability, and recovery metrics
• A published benchmark measuring all of it against simpler methods
• Typed Python API and a self-documenting CLI
• Linux, macOS, and Windows CI across Python 3.10–3.13

Temporal tracking remains evidence-only: it does not silently alter the active frame- or target-level verification policy.

What this is not

A tracking ID represents a persistent changed region — not a proven face, licence plate, person, or other semantic object. V5.2 assumes synchronized media and performs no motion prediction, appearance re-identification, camera-motion compensation, or automatic temporal alignment.

Visual Verifier is an engineering QA tool. It is not a regulatory certificate and not a guarantee of anonymization. Read docs/limitations.md before relying on a PASS.

Documentation

Full documentation: https://samtherocket.github.io/visual-verifier/

Document	Purpose
docs/problem_statement.md	Why the tool exists and what it deliberately excludes
docs/verification_contract.md	The formal input/output contract
docs/architecture.md	Module boundaries and dependency direction
docs/temporal_tracking.md	Association, lifecycle, lineage, and metrics
docs/github_action.md	Every action input, output, and permission
docs/benchmarks.md	Measured comparison against global-metric baselines
docs/target_annotation.md	Reviewed target format, coverage, and interpolation
docs/output_schema.md	Every report field
docs/limitations.md	Known failure modes and honest scope
docs/use_cases.md	Suitable uses and unsupportable claims
CONTRIBUTING.md	Development setup and review requirements

Development

uv sync
uv run ruff format --check .
uv run ruff check .
uv run mypy src --python-version 3.12
uv run pytest -q --cov=visual_verifier

Or run the same gate in one step:

./scripts/run_quality.sh      # Linux and macOS

./scripts/run_quality.ps1     # Windows

Before a release, run the complete release gate. It checks the frame-level and temporal regression contracts, target-aware verification, CLI health, package contents, and distribution building:

./scripts/validate_v5_2.ps1

That script runs validate_release.ps1 first and then the frozen V5.2 temporal regression contract, so one command covers both. The V5.2 gate is retained deliberately: it pins the temporal behaviour that shipped in that milestone, so later work cannot drift it unnoticed. Run ./scripts/validate_release.ps1 alone when you only need the current gate without the historical contract.

Function length, line length, docstrings, annotations, control characters in documentation, and the agreement between the Git tag, the package version, and CITATION.cff are all enforced by Ruff and the test suite rather than by review.

Repository layout

src/visual_verifier/           Active package
src/visual_verifier/tracking/  Temporal association and integrity analysis
tests/                         Unit and end-to-end regressions
examples/media/                Reproducible 15-frame fixtures
examples/expected/             Executable demo contract
examples/targets/              Reviewed target example
docs/                          Architecture and behavior documentation
overrides/                     Documentation social and agent metadata
benchmarks/                    Measured comparison against simpler methods
scripts/                       Bootstrap, quality, cleanup, and release gates
action.yml                     Composite GitHub Action wrapping the CLI
archive/legacy_cells/          Read-only historical prototypes

Roadmap

V5.3 target-aware verification is complete. V5.4 adds the selectable policy system, and V5.5 batch verification. See ROADMAP.md.

Licence and citation

Visual Verifier is licensed under Apache-2.0. Citation metadata is provided in CITATION.cff.