Skip to content

feat(catalog): classification admissibility dataset — honest A–F trust grading per estate - #36

Merged
mdheller merged 1 commit into
mainfrom
feat/classification-admissibility-dataset
Aug 4, 2026
Merged

feat(catalog): classification admissibility dataset — honest A–F trust grading per estate#36
mdheller merged 1 commit into
mainfrom
feat/classification-admissibility-dataset

Conversation

@mdheller

@mdheller mdheller commented Aug 4, 2026

Copy link
Copy Markdown
Member

Catalog half of the six-layer fingerprint classifier. Pairs with SourceOS-Linux/sourceos-spec#284, which contracts the schemas.

What this answers

Before the classifier can classify anything, it has to answer: which of this estate's six metadata fingerprint layers can actually be trusted?

The layers fail independently — L1 dies where everything is declared VARCHAR, L2 dies on an empty table (anti-correlated with L1), L3/L4 die without a glossary, L5 dies under an active drift flag, L6 dies where DDL declares no constraints. The alternative to reporting that is silently averaging over unreliable layers and producing a confident number with nothing behind it.

So per-layer admissibility is a primary output, and it ships first — it needs no glossary and no calibration.

The grade is recomputed, never asserted

A  6 admissible, none degraded, n_eff >= floor
B  6 admissible, some degraded, n_eff >= floor
C  4-5 admissible, n_eff >= floor
D  4-5 admissible, n_eff <  floor
F  <= 3 admissible

A report that carried its own grade could carry one its measurements do not support — precisely the failure this dataset exists to surface. validate recomputes every committed grade from that row's own layer states and fails on drift.

n_eff is the participation ratio of the layer covariance spectrum — not a layer count, and not Herfindahl. Two perfectly correlated layers give H = 0.5, which looks healthy while supplying one layer's worth of information. Since L3–L4 are correlated (both track glossary maturity) and L5 depends on L3 (it is a bag of L3 labels), n_eff < 6 on every real estate.

The seed row grades D, and that is correct

Berger Foods at cold-start phase 0: four layers admissible, two of them degraded, n_eff 3.6 against a floor of 4.0. The layers present are not independent enough to license a positive classification, so the honest outputs on that estate today are ZERO and NEG only.

L3 and L4 are exactly what a customer lacks when they engage — a customer with a maintained glossary would not need this system. A low cold-start grade is the deliverable, not an embarrassment to smooth away. Same principle as the country coverage grading in the Data Catalogue: the map is deliberately not uniformly green.

Also

Registers the six new sourceos-spec schemas (ClassificationStance, FingerprintLayerEvidence, ColumnFingerprint, ColumnDriftObservation, EstateAdmissibilityReport, ClassificationEnforcementPolicy) in schemas-contracts so they percolate.

Teeth

make validate green. Beyond the per-row checks, the validator exercises the grading function across all five bands and asserts the degradation penalty actually changes the grade — a grading function only ever run on one estate is a constant with extra steps.

Found in passing, deliberately not fixed here

Two pre-existing defects, filed separately so they don't ride along in this diff:

  1. make validate mutates a committed dataset. It rewrites datasets/regex-operational-dataset/contributions/Noetica.jsonl (1331 → 4434 rows) by re-harvesting from the local ~/dev/Noetica working tree. Validate must be read-only and fail on drift, not overwrite.
  2. Case-colliding tracked paths. Both Noetica.jsonl and noetica.jsonl are tracked; on case-insensitive macOS filesystems the working tree can never be clean.

…t grading per estate

Registers the six-layer fingerprint classifier's artifacts in the catalog and adds the
dataset that answers the question the classifier has to answer first: which of this
estate's six metadata layers can actually be trusted?

The layers fail independently — L1 dies where everything is declared VARCHAR, L2 dies on an
empty table (anti-correlated with L1), L3 and L4 die without a glossary, L5 dies under an
active drift flag, L6 dies where DDL declares no constraints. The alternative to reporting
that is silently averaging over unreliable layers and producing a confident number with
nothing behind it.

The grade is RECOMPUTED from each estate's own layer states and n_eff, never carried from
the source document — a report that asserted its own grade could assert one its measurements
do not support, which is precisely the failure this dataset exists to surface.

n_eff is the participation ratio of the layer covariance spectrum, not a layer count and not
Herfindahl: two perfectly correlated layers give H = 0.5, which looks healthy while
supplying one layer's worth of information. Since L3-L4 are correlated and L5 depends on L3,
n_eff < 6 on every real estate.

The seed row (Berger Foods, cold-start phase 0) grades D — four layers admissible, two of
them degraded, n_eff 3.6 against a floor of 4.0. That is the correct answer, not a defect:
L3 and L4 are exactly what a customer lacks when they engage, so a low cold-start grade is
the deliverable rather than something to smooth away. Same principle as the country coverage
grading in the Data Catalogue — the map is deliberately not uniformly green.

Also registers the six new sourceos-spec schemas (ClassificationStance,
FingerprintLayerEvidence, ColumnFingerprint, ColumnDriftObservation,
EstateAdmissibilityReport, ClassificationEnforcementPolicy) in schemas-contracts so they
percolate.

validate recomputes every committed grade and fails on drift, and separately exercises the
grading function across all five bands plus the degradation penalty — a grading function
only ever run on one estate is a constant with extra steps.
@mdheller
mdheller force-pushed the feat/classification-admissibility-dataset branch from b9bfbf3 to 94e3522 Compare August 4, 2026 07:52
@mdheller
mdheller merged commit bfab8c4 into main Aug 4, 2026
5 checks passed
@mdheller
mdheller deleted the feat/classification-admissibility-dataset branch August 4, 2026 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant