synthetic-data-gen-eval

A reproducible, tool-neutral benchmark for tabular synthetic-data generation (SDG) — with a pluggable evaluation harness underneath it.

▶ Live dashboard · 📄 Comprehensive report (PDF)

Synthetic data lets an organization share and augment its most restricted datasets without exposing the real records behind them. But which tool, and is the output actually faithful, useful, private, and rule-compliant enough to trust? This project answers that empirically: it runs a fair bake-off of seven open/free synthesizers across three public enterprise-shaped datasets and scores every result on four independent metric families — all on a CPU laptop, at zero licence cost, fully reproducible from pinned data and fixed seeds.

It has two layers:

• syndata — a small, extensible evaluation harness. Every synthesizer implements one adapter interface (fit / sample / availability / info); adding a tool is one file. Fidelity, utility, and privacy metrics are tool-neutral and computed by independent libraries.
• benchmark/ — a complete, reproducible campaign built on the harness: 3 datasets × 7 tools × 3 seeds = 63 runs, scored into one append-only registry from which the reports regenerate as a pure function.

---

Architecture

flowchart LR
    D[(Real tabular<br/>datasets)] --> GEN[SDG models<br/>CTGAN / TVAE / ...]
    GEN --> SYN[(Synthetic data)]
    D --> EVAL[Evaluation harness]
    SYN --> EVAL
    EVAL --> F[Fidelity]
    EVAL --> U[Utility]
    EVAL --> P[Privacy]
    F --> R[Scorecard +<br/>privacy-utility Pareto]
    U --> R
    P --> R

Loading

Headline results

Mean across the three datasets and three seeds (higher fidelity/utility is better; membership-inference AUC closer to 0.50 = safer; exact-match and fit-time lower is better):

Tool	Family	Fidelity	Utility retained	MIA AUC	Exact copies	Fit (s)
MOSTLY AI ARGN	autoregressive	0.905	0.951	0.500	0.050	1,512
SMOTE-NC (baseline)	interpolation	0.875	1.056	0.650 ⚠	0.096 ⚠	0
Independent (baseline)	marginal floor	0.828	0.409	0.499	0.002	0
synthcity TVAE	VAE	0.769	0.940	0.500	0.000	26,238
SDV GaussianCopula	statistical	0.761	0.566	0.500	0.000	182
SDV CTGAN	GAN	0.744	1.144*	0.503	0.005	5,731
MOSTLY AI ARGN + DP	autoregressive + DP	0.735	0.847	0.503	0.033	1,441

_{Fidelity = SDMetrics overall quality. Utility retained = TSTR ÷ TRTR (a model trained on synthetic data vs one trained on real, both tested on the real holdout). *Fraud utility >1.0 is an imbalance artifact — see the report.}

What the numbers say:

1. MOSTLY AI's autoregressive model (ARGN) is the all-round winner — top fidelity, near-real utility, membership-inference at chance, and it runs locally for free.
2. The privacy dial is real. The same model under differential privacy (ε=5) trades a small, bounded slice of fidelity/utility for a formal guarantee — a knob you can show an auditor, not just describe.
3. "Just use SMOTE" is refuted for data sharing. SMOTE-NC looks great on utility but posts the highest membership-inference AUC and copies ~10% of training rows verbatim (29% on one dataset). Learned generators keep both at chance.
4. Baselines keep everyone honest. Independent-marginal sampling floors fidelity; any real tool must beat it — and the deep models do.

📊 Reports

• Comprehensive report (PDF) — the one-stop-shop: the SDG landscape, tools & datasets explained, methodology, full results, and next steps. (Renders inline on GitHub.)
• Interactive dashboard (HTML) — a self-contained Chart.js dashboard with the privacy–utility Pareto frontier and TSTR-gap bars. (Download & open locally, or view raw — GitHub shows HTML as source.)

Both are pure functions of benchmark/results/registry.parquet — re-run scoring and they refresh with zero manual edits.

---

Design philosophy

• Adapter pattern. One interface (BaseSynthesizer: fit / sample / availability / info) in src/syndata/synthesizers/base.py. Adding a tool = one adapter file + one registry entry. Benchmark backend ids (sdv.ctgan, mostlyai.tabular_argn, baseline.smote_nc, …) resolve to adapters in src/syndata/backends.py.
• Lazy imports + graceful degradation. Heavy, mutually-incompatible SDKs are imported inside adapters; a tool that isn't installed reports unavailable instead of crashing the run. (synthcity, which pins conflicting numpy/torch, runs in a dedicated venv and exchanges parquet on disk.)
• Evaluation is tool-neutral and attack-aware. Two independent fidelity evaluators (no vendor grades its own homework); utility via TSTR against an untouched holdout; privacy led by attack-based evidence (membership-inference, exact-match), with distance metrics reported but explicitly not trusted as guarantees ("The DCR Delusion", arXiv:2505.01524).
• Config-driven & reproducible. One YAML per experiment; sha256-pinned raw data; a single deterministic split (seed 42); fixed model seeds [42, 43, 44]; an append-only results registry the reports are computed from.

What's evaluated — the four metric families

Family	Question	Metrics
Fidelity	Does synthetic data match the real joint distribution?	SDMetrics (column shapes KS/TVD + pair trends), mostlyai-qa accuracy, discriminator AUC
Utility	Does a model trained on synthetic data still work on real data?	TSTR ÷ TRTR retention (LightGBM on the real holdout)
Privacy	Can an attacker recover real records?	Membership-inference AUC, exact-match count, DCR/NNDR (descriptive only), DP ε
Compliance	Do synthetic rows obey declared business rules?	Per-rule + overall pass-rate (DataCo supply-chain rules)

Tools & datasets

Synthesizers span the method families: statistical (SDV GaussianCopula), GAN (SDV CTGAN), VAE (synthcity TVAE), autoregressive (MOSTLY AI TabularARGN, + a differential-privacy variant), and two mandatory baselines — SMOTE-NC (utility bar) and independent marginals (fidelity floor). Adapters for rule-based (dbldatagen) and LLM-driven (NeMo Data Designer) generation are scaffolded for a future GPU/LLM phase.

Datasets are public, openly-licensed, and chosen to stress different enterprise concerns:

Dataset	Domain	Rows	Stresses
Adult / Census Income	cross-industry	48.8k	sharing person-level data safely
Credit-Card Fraud (ULB)	finance	284.8k	extreme class imbalance (0.17%) + rebalancing
DataCo Supply Chain	retail / SCM	180.5k	business-rule compliance on wide operational data

A broader landscape survey of candidate datasets across four modalities lives in benchmark/data/DATASETS.md.

Repository layout

synthetic-data-gen-eval/
├── src/syndata/              # the harness (pip-installable)
│   ├── synthesizers/         # base.py + one adapter per tool (lazy imports)
│   ├── evaluation/           # fidelity.py · utility.py (TSTR) · privacy.py · report.py
│   ├── backends.py           # benchmark backend-id → adapter resolution
│   ├── registry.py           # name → adapter factory
│   ├── tracks/               # text/ and qa/ design notes (future modalities)
│   └── cli.py                # python -m syndata {run,report,list}
├── benchmark/                # the reproducible campaign (see benchmark/README.md)
│   ├── data/catalog.yaml     # dataset registry + sha256 (raw data gitignored)
│   ├── configs/              # experiment_matrix.yaml → 21 generated experiment YAMLs
│   ├── scripts/              # download · split · gen-configs · run · score · build-report
│   ├── results/registry.parquet   # the scored source of truth (committed)
│   └── reports/              # interactive dashboard + comprehensive PDF
├── configs/                  # standalone harness configs (single-dataset quickstart)
├── docs/evaluation_protocol.md
└── tests/                    # smoke tests (pytest)

Quickstart

# Python 3.11 or 3.12
python -m venv .venv && . .venv/Scripts/activate      # Windows; use bin/activate on *nix
pip install -e ".[core]"                               # SDV + MOSTLY AI + evaluators + baselines

# Reproduce the benchmark (needs a free Kaggle API token at ~/.kaggle/kaggle.json):
cd benchmark
python scripts/download_data.py        # Adult via HTTP; fraud + DataCo via Kaggle (+ sha256)
python scripts/make_splits.py          # deterministic stratified 80/20, seed 42
python scripts/gen_configs.py          # expand the 3×7 grid → 21 configs
python scripts/run_bakeoff.py configs/*.yaml     # fit → sample → runs/<id>/
python scripts/score_runs.py           # 4 metric families → results/registry.parquet
python scripts/build_report.py             # interactive dashboard
python scripts/build_narrative_report.py   # comprehensive PDF

Install extras per tool to avoid dependency conflicts: .[synthcity] (dedicated venv recommended), .[data] (Kaggle), .[spark], .[nemo]. See benchmark/README.md for the full run order and compute notes.

Reproducibility contract

Raw data immutable + sha256-checked · one deterministic split (seed 42) recorded per manifest · fixed model seeds [42, 43, 44] for variance bands · append-only registry.parquet · the reports are a pure function of it. Per-run wall-clock and peak memory are recorded so scale-up estimates rest on measured numbers (~88 CPU-hours for the full grid).

Beyond tabular — future work

This repository establishes the method and the winning tools for single-table tabular data on CPU. The natural next steps reuse the same harness, registry, and reporting pipeline to go further:

• Larger models, full budgets — GPU-scale training with no subsampling caps (full-epoch CTGAN/TVAE, diffusion, hosted safe-synthesizer services).
• New modalities — relational / multi-table and time-series (SDV HMA/PAR), then LLM-driven text and Q&A generation (the tracks/ scaffolds).
• Deeper privacy — a differential-privacy ε sweep to chart the full privacy–utility curve, plus stronger membership- and attribute-inference attacks.

Adding a tool is one adapter file; adding a dataset is one catalog entry — extending the benchmark is throughput on a proven pipeline, not new engineering.

License

MIT for the original code and documentation. Third-party synthesizers invoked through the adapter layer and the public benchmark datasets each retain their own licenses.