Skip to content

Keep calibration and held-out samples disjoint - #143

Merged
WestWaters merged 1 commit into
WestWaters:mainfrom
SloptimistPrime:contrib/disjoint-calibration
Oct 6, 2026
Merged

WestWaters merged 1 commit into
WestWaters:mainfrom
SloptimistPrime:contrib/disjoint-calibration

Conversation

@SloptimistPrime

Copy link
Copy Markdown
Contributor

The held-out split shuffled repeated rows, allowing bundled seeds to appear in both files.

Deduplicate content across domains before splitting, while preserving the original formatting. Report the smaller unique sample count instead of treating repeated text as new data. Reject splits with too little unique data and output paths that refer to the same file. Calibration-only runs keep their previous sample counts.

21 tests pass for repeated text, Unicode and whitespace variants, cross-domain overlap, determinism, invalid arguments and file aliases.

This prevents exact normalized sample overlap, not all near-duplicates or overlap with the model's pretraining data.

I used AI assistance and reviewed and tested the changes.

@WestWaters
WestWaters merged commit 63edbbd into WestWaters:main Oct 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants