Repository navigation
feat: add a verifiable dataset quality certificate - #5
Merged
Merged
Conversation
A synthetic data generator has to answer an uncomfortable question: why should anyone believe the data? `laranjix certify` answers it with something the reader can recompute rather than a badge they have to trust. Every line is recalculated from the dataset's own files: documents are re-validated against the official algorithm, files are re-scanned, the dataset is regenerated from its manifest and hashed, and each distribution is re-measured against its calibration target. Fidelity needed an honest threshold. No finite sample reproduces the distribution it was drawn from, so comparing a distance against zero, or against a round number like 0.01, says nothing. Each check is compared against the deviation that its own sample size produces, estimated by simulation at the 99.9th percentile. The threshold shrinks with the square root of the sample, which makes the test stricter as datasets grow -- the right direction, since in a large dataset a small deviation is already evidence of a defect. It earned its place on the first run by failing two checks. EVP keys came out at 42.7% against a 31% target for PF, 18.6 times what sampling noise explains, because an account holds at most one key of each unique type and the generator pushed every collision into EVP. Resampling among the remaining types only halves the bias. The calibration target is a *realized* mix, so under the uniqueness constraint it is not the distribution to draw from; the generator now solves for the draw weights whose outcome is the target, iterating w <- w * target/realized with the realized mix computed exactly by dynamic programming over which unique types an account has used. No Monte Carlo, so the solution is identical on every machine and NumPy version, which reproducibility across releases requires. Residual error 1e-13; the two checks now sit at 0.4x and 0.3x the noise. This changes the output of existing seeds, which is free to do before v0.1. Utility is measured by TSTR, and the direction data flows is the whole point: the real holdout stays in the caller's process and only the score comes back, so utility can be measured against reality without a real record ever entering the repository. The harness is implemented and tested against a stand-in task; the certificate reports it as pending until fraud labels arrive in milestone 4, rather than inventing a number. The certificate is also explicit about what it does not establish -- it compares the dataset against the calibration parameters, not against Brazil, and it names the parameters still marked provisional every time it is issued. Verified: 77 tests, ruff and mypy --strict clean, and the README's numbers are the output of the run it documents.
| class LabelledFrame(Protocol): | ||
| """Minimal shape a dataset must have to take part in TSTR.""" | ||
|
|
||
| def to_numpy(self) -> Any: ... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A synthetic data generator has to answer an uncomfortable question: why should anyone believe the data?
laranjix certifyanswers it with something the reader can recompute rather than a badge they have to trust. Every line is recalculated from the dataset's own files: documents are re-validated against the official algorithm, files are re-scanned, the dataset is regenerated from its manifest and hashed, and each distribution is re-measured against its calibration target.Fidelity needed an honest threshold. No finite sample reproduces the distribution it was drawn from, so comparing a distance against zero, or against a round number like 0.01, says nothing. Each check is compared against the deviation that its own sample size produces, estimated by simulation at the 99.9th percentile. The threshold shrinks with the square root of the sample, which makes the test stricter as datasets grow -- the right direction, since in a large dataset a small deviation is already evidence of a defect.
It earned its place on the first run by failing two checks. EVP keys came out at 42.7% against a 31% target for PF, 18.6 times what sampling noise explains, because an account holds at most one key of each unique type and the generator pushed every collision into EVP. Resampling among the remaining types only halves the bias. The calibration target is a realized mix, so under the uniqueness constraint it is not the distribution to draw from; the generator now solves for the draw weights whose outcome is the target, iterating w <- w * target/realized with the realized mix computed exactly by dynamic programming over which unique types an account has used. No Monte Carlo, so the solution is identical on every machine and NumPy version, which reproducibility across releases requires. Residual error 1e-13; the two checks now sit at 0.4x and 0.3x the noise. This changes the output of existing seeds, which is free to do before v0.1.
Utility is measured by TSTR, and the direction data flows is the whole point: the real holdout stays in the caller's process and only the score comes back, so utility can be measured against reality without a real record ever entering the repository. The harness is implemented and tested against a stand-in task; the certificate reports it as pending until fraud labels arrive in milestone 4, rather than inventing a number.
The certificate is also explicit about what it does not establish -- it compares the dataset against the calibration parameters, not against Brazil, and it names the parameters still marked provisional every time it is issued.
Verified: 77 tests, ruff and mypy --strict clean, and the README's numbers are the output of the run it documents.