feat(catalog): classification admissibility dataset — honest A–F trust grading per estate - #36
Merged
Conversation
…t grading per estate Registers the six-layer fingerprint classifier's artifacts in the catalog and adds the dataset that answers the question the classifier has to answer first: which of this estate's six metadata layers can actually be trusted? The layers fail independently — L1 dies where everything is declared VARCHAR, L2 dies on an empty table (anti-correlated with L1), L3 and L4 die without a glossary, L5 dies under an active drift flag, L6 dies where DDL declares no constraints. The alternative to reporting that is silently averaging over unreliable layers and producing a confident number with nothing behind it. The grade is RECOMPUTED from each estate's own layer states and n_eff, never carried from the source document — a report that asserted its own grade could assert one its measurements do not support, which is precisely the failure this dataset exists to surface. n_eff is the participation ratio of the layer covariance spectrum, not a layer count and not Herfindahl: two perfectly correlated layers give H = 0.5, which looks healthy while supplying one layer's worth of information. Since L3-L4 are correlated and L5 depends on L3, n_eff < 6 on every real estate. The seed row (Berger Foods, cold-start phase 0) grades D — four layers admissible, two of them degraded, n_eff 3.6 against a floor of 4.0. That is the correct answer, not a defect: L3 and L4 are exactly what a customer lacks when they engage, so a low cold-start grade is the deliverable rather than something to smooth away. Same principle as the country coverage grading in the Data Catalogue — the map is deliberately not uniformly green. Also registers the six new sourceos-spec schemas (ClassificationStance, FingerprintLayerEvidence, ColumnFingerprint, ColumnDriftObservation, EstateAdmissibilityReport, ClassificationEnforcementPolicy) in schemas-contracts so they percolate. validate recomputes every committed grade and fails on drift, and separately exercises the grading function across all five bands plus the degradation penalty — a grading function only ever run on one estate is a constant with extra steps.
mdheller
force-pushed
the
feat/classification-admissibility-dataset
branch
from
August 4, 2026 07:52
b9bfbf3 to
94e3522
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Catalog half of the six-layer fingerprint classifier. Pairs with SourceOS-Linux/sourceos-spec#284, which contracts the schemas.
What this answers
Before the classifier can classify anything, it has to answer: which of this estate's six metadata fingerprint layers can actually be trusted?
The layers fail independently — L1 dies where everything is declared
VARCHAR, L2 dies on an empty table (anti-correlated with L1), L3/L4 die without a glossary, L5 dies under an active drift flag, L6 dies where DDL declares no constraints. The alternative to reporting that is silently averaging over unreliable layers and producing a confident number with nothing behind it.So per-layer admissibility is a primary output, and it ships first — it needs no glossary and no calibration.
The grade is recomputed, never asserted
A report that carried its own grade could carry one its measurements do not support — precisely the failure this dataset exists to surface.
validaterecomputes every committed grade from that row's own layer states and fails on drift.n_effis the participation ratio of the layer covariance spectrum — not a layer count, and not Herfindahl. Two perfectly correlated layers give H = 0.5, which looks healthy while supplying one layer's worth of information. Since L3–L4 are correlated (both track glossary maturity) and L5 depends on L3 (it is a bag of L3 labels),n_eff < 6on every real estate.The seed row grades D, and that is correct
Berger Foods at cold-start phase 0: four layers admissible, two of them degraded,
n_eff3.6 against a floor of 4.0. The layers present are not independent enough to license a positive classification, so the honest outputs on that estate today areZEROandNEGonly.L3 and L4 are exactly what a customer lacks when they engage — a customer with a maintained glossary would not need this system. A low cold-start grade is the deliverable, not an embarrassment to smooth away. Same principle as the country coverage grading in the Data Catalogue: the map is deliberately not uniformly green.
Also
Registers the six new
sourceos-specschemas (ClassificationStance,FingerprintLayerEvidence,ColumnFingerprint,ColumnDriftObservation,EstateAdmissibilityReport,ClassificationEnforcementPolicy) inschemas-contractsso they percolate.Teeth
make validategreen. Beyond the per-row checks, the validator exercises the grading function across all five bands and asserts the degradation penalty actually changes the grade — a grading function only ever run on one estate is a constant with extra steps.Found in passing, deliberately not fixed here
Two pre-existing defects, filed separately so they don't ride along in this diff:
make validatemutates a committed dataset. It rewritesdatasets/regex-operational-dataset/contributions/Noetica.jsonl(1331 → 4434 rows) by re-harvesting from the local~/dev/Noeticaworking tree. Validate must be read-only and fail on drift, not overwrite.Noetica.jsonlandnoetica.jsonlare tracked; on case-insensitive macOS filesystems the working tree can never be clean.