Skip to content

feat: add a verifiable dataset quality certificate - #5

Merged
mkmuniz merged 1 commit into
mainfrom
feat/quality-certificate
Oct 3, 2026
Merged

mkmuniz merged 1 commit into
mainfrom
feat/quality-certificate

Conversation

@mkmuniz

@mkmuniz mkmuniz commented Oct 3, 2026

Copy link
Copy Markdown
Owner

A synthetic data generator has to answer an uncomfortable question: why should anyone believe the data? laranjix certify answers it with something the reader can recompute rather than a badge they have to trust. Every line is recalculated from the dataset's own files: documents are re-validated against the official algorithm, files are re-scanned, the dataset is regenerated from its manifest and hashed, and each distribution is re-measured against its calibration target.

Fidelity needed an honest threshold. No finite sample reproduces the distribution it was drawn from, so comparing a distance against zero, or against a round number like 0.01, says nothing. Each check is compared against the deviation that its own sample size produces, estimated by simulation at the 99.9th percentile. The threshold shrinks with the square root of the sample, which makes the test stricter as datasets grow -- the right direction, since in a large dataset a small deviation is already evidence of a defect.

It earned its place on the first run by failing two checks. EVP keys came out at 42.7% against a 31% target for PF, 18.6 times what sampling noise explains, because an account holds at most one key of each unique type and the generator pushed every collision into EVP. Resampling among the remaining types only halves the bias. The calibration target is a realized mix, so under the uniqueness constraint it is not the distribution to draw from; the generator now solves for the draw weights whose outcome is the target, iterating w <- w * target/realized with the realized mix computed exactly by dynamic programming over which unique types an account has used. No Monte Carlo, so the solution is identical on every machine and NumPy version, which reproducibility across releases requires. Residual error 1e-13; the two checks now sit at 0.4x and 0.3x the noise. This changes the output of existing seeds, which is free to do before v0.1.

Utility is measured by TSTR, and the direction data flows is the whole point: the real holdout stays in the caller's process and only the score comes back, so utility can be measured against reality without a real record ever entering the repository. The harness is implemented and tested against a stand-in task; the certificate reports it as pending until fraud labels arrive in milestone 4, rather than inventing a number.

The certificate is also explicit about what it does not establish -- it compares the dataset against the calibration parameters, not against Brazil, and it names the parameters still marked provisional every time it is issued.

Verified: 77 tests, ruff and mypy --strict clean, and the README's numbers are the output of the run it documents.

A synthetic data generator has to answer an uncomfortable question: why should
anyone believe the data? `laranjix certify` answers it with something the reader
can recompute rather than a badge they have to trust. Every line is recalculated
from the dataset's own files: documents are re-validated against the official
algorithm, files are re-scanned, the dataset is regenerated from its manifest and
hashed, and each distribution is re-measured against its calibration target.

Fidelity needed an honest threshold. No finite sample reproduces the distribution
it was drawn from, so comparing a distance against zero, or against a round
number like 0.01, says nothing. Each check is compared against the deviation that
its own sample size produces, estimated by simulation at the 99.9th percentile.
The threshold shrinks with the square root of the sample, which makes the test
stricter as datasets grow -- the right direction, since in a large dataset a
small deviation is already evidence of a defect.

It earned its place on the first run by failing two checks. EVP keys came out at
42.7% against a 31% target for PF, 18.6 times what sampling noise explains,
because an account holds at most one key of each unique type and the generator
pushed every collision into EVP. Resampling among the remaining types only halves
the bias. The calibration target is a *realized* mix, so under the uniqueness
constraint it is not the distribution to draw from; the generator now solves for
the draw weights whose outcome is the target, iterating w <- w * target/realized
with the realized mix computed exactly by dynamic programming over which unique
types an account has used. No Monte Carlo, so the solution is identical on every
machine and NumPy version, which reproducibility across releases requires.
Residual error 1e-13; the two checks now sit at 0.4x and 0.3x the noise. This
changes the output of existing seeds, which is free to do before v0.1.

Utility is measured by TSTR, and the direction data flows is the whole point: the
real holdout stays in the caller's process and only the score comes back, so
utility can be measured against reality without a real record ever entering the
repository. The harness is implemented and tested against a stand-in task; the
certificate reports it as pending until fraud labels arrive in milestone 4,
rather than inventing a number.

The certificate is also explicit about what it does not establish -- it compares
the dataset against the calibration parameters, not against Brazil, and it names
the parameters still marked provisional every time it is issued.

Verified: 77 tests, ruff and mypy --strict clean, and the README's numbers are
the output of the run it documents.
@mkmuniz mkmuniz self-assigned this Oct 3, 2026
@mkmuniz mkmuniz added accessibility Barrier affecting people with disabilities documentation Improvements or additions to documentation enhancement New feature or request labels Oct 3, 2026
class LabelledFrame(Protocol):
"""Minimal shape a dataset must have to take part in TSTR."""

def to_numpy(self) -> Any: ...
@mkmuniz
mkmuniz merged commit 3f927ca into main Oct 3, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accessibility Barrier affecting people with disabilities documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants