diff --git a/CHANGELOG.md b/CHANGELOG.md index 8852f7d..589a3a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). Thi ### Added - Image-trust contracts — the Assay's measured-not-asserted discipline applied to images, for both OS and agents: `ImageTrustReport` (per-dimension trust checks — reproducibility, attestation_verify, sbom_completeness, signature_verify, measured_boot, provenance_depth — each carrying a `Measurement` via `$ref`, plus a render-time `projectedTrust` ok/sad/bad verdict; unifies OS and agent images through `subjectKind`) and `AgentImage` (agent runtime-artifact identity + provenance parallel to `OSImage`, distinct from process-level `AgentPassport`). `ImagePromotionGate` now requires a `trustReportRef` when `decision: approved` (non-breaking if/then), and the new `validate-image-trust-examples` target enforces projection soundness plus the cross-document rule that an approval must reference a report projecting `ok` — closing, for images, the self-assertable-approval gap the Assay hardening closed for verifiers. Reuses `Measurement`'s gate-eligibility invariants for free via `$ref` (a declared/assumed check cannot reach `ok`). All invariants verified biting. See `schemas/image-trust.README.md`. - Assay fleet-tier contracts (cloud-mesh): `AssayRollup` (fleet/cohort aggregate of `ReasoningAssay` verdicts over a window — ok/sad/bad distribution, calibration-drift view of which `AssayStandard` versions are live across the fleet, unassayed-reason breakdown) and `AssayStandardRollout` (canary→widen/halt promotion of a new `AssayStandard` version across cohorts, gated on an observed `AssayRollup`; rides the release-bundle + lifecycle machinery). New `validate-assay-fleet-examples` target enforces aggregation/rollout soundness (distribution sums, drift-flag agreement, rolloutPct vs node share, decision/phase consistency, and no-promotion-by-hope), all verified biting. This is the boundary between prophet-mesh single-user-local (node self-view, no fleet dashboards, structural) and cloud-mesh (rollup + rollout + dashboards). See `schemas/assay-fleet.README.md`. +- **Six-layer fingerprint stack (SP-FPRINT-STACK-001)** — classification that can distinguish "I don't know" from "my evidence contradicts itself". `ClassificationStance` over FOUR (POS/NEG/**ZERO**/**INADMISSIBLE**) with the support-vs-plausibility reading tag in the TYPE (DR-5); `FingerprintLayerEvidence` with per-layer discriminated witnesses for L1 OntoDT / L2 OntoDQ / L3 glossary / L4 operational semantics / L5 table topic / L6 key graph; `ColumnFingerprint` (pool -> guard -> quantize, in that order); `ColumnDriftObservation` (L5-D1/L5-D2, blocking); `EstateAdmissibilityReport` (the phase-0 deliverable — which layers this estate can trust, shipped BEFORE any classification); `ClassificationEnforcementPolicy` (**DR-4 settled**: POS/NEG not knobs, INADMISSIBLE never fail-open, ZERO an attested per-resource-class knob unavailable at confidential/restricted). Recompute-don't-trust throughout: stance from evidence, pooling from admissible layers, `n_eff` from the covariance spectrum (participation ratio — Herfindahl demoted to a cheap precheck, since two perfectly correlated layers give H=0.5 while supplying one layer's worth of information), drift from the measured distance. Axiom **X1** (parthood and subtyping disjoint; closure traverses subtyping only; open-world counts are intervals with INADMISSIBLE reported separately) enforced as a CI invariant. `DataClass.classifier.kind` const **retired**: TensorFlow Lattice was archived by its owner in April 2026, so the pin now enumerates the structurally-monotone constructions with the dead name deprecated-but-accepted, and gains a `fingerprintStack` binding. 81 checks + 9 schema negative vectors + **25 gates proven to bite** by mutation harness + an M5 property test verified against a deliberately non-monotone aggregator (the #265 vacuous-constraint defect, guarded against structurally this time). - Trained DataClass classifiers — adversarial hardening (follow-up to #264): stronger fixture where BOTH monotone features (token_magnitude, decimal_places) genuinely drive the label so the monotone constraint actually BINDS (was vacuous — decimal_places held constant); standardiser fit on the TRAIN split only (no test leakage); ModelManifest named honestly `monotone-logistic` (a realisation of the declared tf-lattice-wide-and-deep contract, not a full lattice). Validator now checks BOTH per-class heads monotone AND asserts the constraint binds (non-vacuous), and compares reproduced accuracy at the stored 4-decimal precision (fixes a latent tolerance bug the trivial 1.0-accuracy fixture had masked). 8 teeth. - Trained DataClass classifiers v0.1 — specified→trained→registered→reproduced (closes the program's last owed thread): `tools/train_dataclass_classifiers.py` trains real per-class LOGISTIC (one-vs-rest) + per-table SOFTMAX heads, MONOTONE-constrained in `monotonicFeatures` (weight >=0 projection = TF-Lattice essence), deterministically; emits trained weights + a conformant `ModelManifest` (id=DataClass modelRef) + train/eval `RunRecord`s (ids=runRef/evalRunRef) so the previously-dangling DataClass classifier references now RESOLVE. `validate-trained-classifiers` (recompute-don't-trust): digest matches weight bytes; eval accuracy REPRODUCED from weights+fixture (inflated claim fails); model verified genuinely monotone (perturb up → score never drops); refs resolve; artifacts schema-conform. Teeth-verified (tampered weights refused). Uses numpy. - Learned spell-correction + user dictionary v0.1 — from context, not dictionary matching (task #13): `tools/learned_dictionary.py` decides each UNKNOWN token from a count-based skip-gram word-sense predictor (PPMI + truncated SVD; SGNS≈PPMI-SVD) instead of a static wordlist. LEARN a token that recurs with a coherent word-sense (a real domain term the dictionary hadn't seen); CORRECT a rare token to the known word its SENSE matches (skip-gram cosine picks the target, not edit distance alone — a near-spelled but different-sense token is not auto-corrected); leave the rest UNKNOWN (fail-closed, human-admitted). Every decision is a proposal, never a silent rewrite. `validate-learned-dictionary` teeth: epistemiclevel learned, reciept->receipt by sense, qwzptl unknown, learned term never auto-corrected. Uses numpy. diff --git a/Makefile b/Makefile index 9ffdebd..a524c37 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ -.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live validate-kgram-differential validate-learned-dictionary train-dataclass-classifiers validate-assay-fleet-examples validate-image-trust-examples validate-schema-references +.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live validate-kgram-differential validate-learned-dictionary train-dataclass-classifiers validate-assay-fleet-examples validate-image-trust-examples validate-schema-references validate-fingerprint-stack -validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis validate-kgram-differential validate-learned-dictionary validate-trained-classifiers validate-multiverseal-twin-examples validate-assay-fleet-examples validate-image-trust-examples validate-schema-references +validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis validate-kgram-differential validate-learned-dictionary validate-trained-classifiers validate-multiverseal-twin-examples validate-assay-fleet-examples validate-image-trust-examples validate-schema-references validate-fingerprint-stack @echo "OK: validate" validate-assay-fleet-examples: @@ -24,6 +24,11 @@ validate-ingestion-pipeline-examples: python3 -m pip install --user jsonschema >/dev/null python3 tools/validate_ingestion_pipeline_examples.py +validate-fingerprint-stack: + python3 -m pip install --user jsonschema >/dev/null + python3 tools/validate_fingerprint_stack.py + python3 tools/test_fingerprint_stack_teeth.py + validate-data-class-examples: python3 -m pip install --user jsonschema >/dev/null python3 tools/validate_data_class_examples.py diff --git a/examples/classification_enforcement_policy.json b/examples/classification_enforcement_policy.json new file mode 100644 index 0000000..65d9718 --- /dev/null +++ b/examples/classification_enforcement_policy.json @@ -0,0 +1,35 @@ +{ + "id": "urn:srcos:classification-enforcement:estate-default-v1", + "type": "ClassificationEnforcementPolicy", + "specVersion": "2.0.0", + "posBehavior": "allow", + "negBehavior": "deny", + "inadmissibleBehavior": "fail-closed", + "resourceClasses": [ + { + "resourceClass": "reference-lookup", + "sensitivity": "public", + "zeroBehavior": "fail-open", + "attestationRef": "urn:srcos:attestation:dr4-reference-lookup-2026-08-01", + "rationale": "Published reference tables carry no personal data and over-blocking them strands every downstream join. Serving an unclassified reference column is a known, bounded exposure that the data steward accepted on the record." + }, + { + "resourceClass": "operational-analytics", + "sensitivity": "internal", + "zeroBehavior": "fail-closed", + "attestationRef": null, + "rationale": "Default. An unclassified column in an internal analytics surface is more likely to be an un-onboarded field than a deliberately open one." + }, + { + "resourceClass": "customer-master", + "sensitivity": "restricted", + "zeroBehavior": "fail-closed", + "attestationRef": null, + "rationale": "Customer master is where the personal data is. A column we cannot classify is exactly the column not to serve." + } + ], + "abstentionBudget": { + "maxZeroFraction": 0.35, + "onExceed": "escalate-human" + } +} diff --git a/examples/column_drift.repurposed_status.json b/examples/column_drift.repurposed_status.json new file mode 100644 index 0000000..98fcfea --- /dev/null +++ b/examples/column_drift.repurposed_status.json @@ -0,0 +1,23 @@ +{ + "id": "urn:srcos:column-drift:ops-orders-status-code-2026-08", + "type": "ColumnDriftObservation", + "specVersion": "2.0.0", + "subjectColumnRef": "urn:srcos:column:ops.orders.status_code", + "baseline": { + "profileRunId": "profile-2026-02-02-ops-orders", + "schemaVersion": "ops.orders@v12", + "observedAt": "2026-02-02T03:15:00Z" + }, + "current": { + "profileRunId": "profile-2026-08-03-ops-orders", + "schemaVersion": "ops.orders@v12", + "observedAt": "2026-08-03T04:40:00Z" + }, + "metric": "jensen-shannon", + "distance": 0.61, + "threshold": 0.15, + "schemaChanged": false, + "drifted": true, + "verdict": "silent-repurpose", + "stewardQueueRef": "urn:srcos:steward-queue:data-governance-drift" +} diff --git a/examples/column_fingerprint.customer_id.json b/examples/column_fingerprint.customer_id.json new file mode 100644 index 0000000..2d55155 --- /dev/null +++ b/examples/column_fingerprint.customer_id.json @@ -0,0 +1,257 @@ +{ + "id": "urn:srcos:column-fingerprint:crm-customers-customer-id-vs-surrogate-key", + "type": "ColumnFingerprint", + "specVersion": "2.0.0", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "layerEvidence": [ + { + "id": "urn:srcos:layer-evidence:crm-customers-customer-id-l1", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L1-ontodt-type", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "evidence": { + "alpha": 0.72, + "beta": 0.0 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "confidenceSource": "declared", + "witness": { + "kind": "L1", + "ontologyVersion": "ontodt-2.0.0", + "typePath": [ + "datatype", + "primitive-datatype", + "numeric", + "integer" + ], + "closureDepth": 3, + "closureRelation": "subtypeOf", + "axiomsApplied": [ + "X1-parthood-subtyping-disjoint" + ], + "reasoner": { + "id": "hermit", + "version": "1.4.5" + } + } + }, + { + "id": "urn:srcos:layer-evidence:crm-customers-customer-id-l2", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L2-ontodq-profile", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "evidence": { + "alpha": 0.81, + "beta": 0.0 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "witness": { + "kind": "L2", + "profileRunId": "profile-2026-08-01-crm-customers", + "sampleSize": 1240000, + "sampleMethod": "full-scan", + "partitionKey": null, + "profiledAt": "2026-08-01T02:14:00Z", + "sourceSnapshotId": "crm-snap-2026-08-01", + "promotedToHardAxiom": false, + "stewardAttestationRef": null + } + }, + { + "id": "urn:srcos:layer-evidence:crm-customers-customer-id-l3", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L3-business-glossary", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "evidence": { + "alpha": 0.88, + "beta": 0.0 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "witness": { + "kind": "L3", + "glossaryVersion": "glossary-2026-07-28", + "termId": "urn:srcos:glossary:customer-id", + "matchChannel": "lexical", + "channelScores": { + "levenshtein": 1.0, + "jaroWinkler": 1.0, + "hamming": 1.0, + "lcs": 1.0, + "mostFrequentKChars": 0.94, + "smc": 0.91 + }, + "synonymPath": [], + "stewardAttested": true, + "attestedAt": "2026-07-28T11:02:00Z" + } + }, + { + "id": "urn:srcos:layer-evidence:crm-customers-customer-id-l4", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L4-operational-semantics", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "evidence": { + "alpha": 0.66, + "beta": 0.0 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "witness": { + "kind": "L4", + "opProfileId": "opprofile-surrogate-key-v1", + "declaredOps": [ + "equal", + "not-equal", + "join" + ], + "observedOps": [ + "equal", + "join", + "group-by" + ], + "queryLogWindow": "2026-07-01/2026-08-01", + "conflictSet": [] + } + }, + { + "id": "urn:srcos:layer-evidence:crm-customers-customer-id-l5", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L5-table-topic", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "evidence": { + "alpha": 0.54, + "beta": 0.0 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "witness": { + "kind": "L5", + "schemaVersion": "crm.customers@v7", + "headerBagHash": "sha256:8c1a0f3d2b6e4a9c7f5d1e8b2a4c6d0f9e3b7a5c1d8f2e6b4a0c9d7f3e1b5a8c", + "topicModelId": "topic-lsa-bow-v3", + "topicVector": [0.61, 0.12, 0.04, 0.23], + "tableRef": "urn:srcos:table:crm.customers", + "driftRef": null + } + }, + { + "id": "urn:srcos:layer-evidence:crm-customers-customer-id-l6", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L6-schema-key-graph", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "evidence": { + "alpha": 0.91, + "beta": 0.0 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "confidenceSource": "declared", + "witness": { + "kind": "L6", + "ddlVersion": "crm@2026-07-20", + "declaredConstraints": [ + "PRIMARY KEY (customer_id)" + ], + "inferredInclusionDeps": [], + "graphSnapshotId": "keygraph-2026-08-01", + "referencedTableRef": null + } + } + ], + "pooling": { + "operator": "frechet-max", + "independenceCertificateRef": null, + "pooled": { + "alpha": 0.91, + "beta": 0.0 + } + }, + "aggregator": { + "modelRef": "urn:srcos:model-manifest:fingerprint-aggregator-v1", + "monotonicityEnforcement": "structural", + "constraints": [ + "M1-alpha-nondecreasing", + "M2-beta-nondecreasing", + "M4-zero-in-zero-out", + "M5-layer-knockout-lowers" + ] + }, + "quorum": { + "nEff": 4.8, + "nEffFloor": 3.0, + "covarianceSpectrum": [2.0, 1.0, 1.0, 1.0, 0.5, 0.5], + "herfindahl": 0.28 + }, + "stanceRef": "urn:srcos:stance:crm-customers-customer-id-surrogate-key", + "stance": { + "id": "urn:srcos:stance:crm-customers-customer-id-surrogate-key", + "type": "ClassificationStance", + "specVersion": "2.0.0", + "reading": "support", + "evidence": { + "alpha": 0.91, + "beta": 0.0 + }, + "thresholds": { + "tauPos": 0.7, + "tauNeg": 0.7, + "attestationRef": null + }, + "value": "POS", + "inadmissibleOrigin": null, + "composition": { + "depth": 1, + "baseEpsilon": 0.1, + "effectiveEpsilon": 0.1 + }, + "guards": [ + { + "name": "n-eff-floor", + "fired": false, + "inputs": ["witness.quorum.nEff"], + "reason": null + }, + { + "name": "calibration-freshness", + "fired": false, + "inputs": ["witness.calibrationRef"], + "reason": null + }, + { + "name": "l5-drift-annihilator", + "fired": false, + "inputs": ["witness.driftRef"], + "reason": null + } + ], + "calibrationRef": "urn:srcos:run:fingerprint-calibration-2026-07-30" + } +} diff --git a/examples/column_fingerprint.repurposed_status.json b/examples/column_fingerprint.repurposed_status.json new file mode 100644 index 0000000..d1c2455 --- /dev/null +++ b/examples/column_fingerprint.repurposed_status.json @@ -0,0 +1,250 @@ +{ + "id": "urn:srcos:column-fingerprint:ops-orders-status-code-silent-repurpose", + "type": "ColumnFingerprint", + "specVersion": "2.0.0", + "subjectColumnRef": "urn:srcos:column:ops.orders.status_code", + "candidateClassRef": "urn:srcos:data-class:order-status-code", + "layerEvidence": [ + { + "id": "urn:srcos:layer-evidence:ops-orders-status-code-l1", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L1-ontodt-type", + "subjectColumnRef": "urn:srcos:column:ops.orders.status_code", + "candidateClassRef": "urn:srcos:data-class:order-status-code", + "evidence": { + "alpha": 0.7, + "beta": 0.0 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "confidenceSource": "declared", + "witness": { + "kind": "L1", + "ontologyVersion": "ontodt-2.0.0", + "typePath": [ + "datatype", + "primitive-datatype", + "non-numeric", + "string-identifier" + ], + "closureDepth": 3, + "closureRelation": "subtypeOf", + "axiomsApplied": [], + "reasoner": { + "id": "hermit", + "version": "1.4.5" + } + } + }, + { + "id": "urn:srcos:layer-evidence:ops-orders-status-code-l2", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L2-ontodq-profile", + "subjectColumnRef": "urn:srcos:column:ops.orders.status_code", + "candidateClassRef": "urn:srcos:data-class:order-status-code", + "evidence": { + "alpha": 0.05, + "beta": 0.86 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "witness": { + "kind": "L2", + "profileRunId": "profile-2026-08-03-ops-orders", + "sampleSize": 8830000, + "sampleMethod": "partition-stratified", + "partitionKey": "order_date", + "profiledAt": "2026-08-03T04:40:00Z", + "sourceSnapshotId": "ops-snap-2026-08-03", + "promotedToHardAxiom": false, + "stewardAttestationRef": null + } + }, + { + "id": "urn:srcos:layer-evidence:ops-orders-status-code-l3", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L3-business-glossary", + "subjectColumnRef": "urn:srcos:column:ops.orders.status_code", + "candidateClassRef": "urn:srcos:data-class:order-status-code", + "evidence": { + "alpha": 0.82, + "beta": 0.0 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "witness": { + "kind": "L3", + "glossaryVersion": "glossary-2026-07-28", + "termId": "urn:srcos:glossary:order-status", + "matchChannel": "lexical", + "channelScores": { + "levenshtein": 0.86, + "jaroWinkler": 0.93, + "lcs": 0.88 + }, + "synonymPath": [ + "status_code", + "order status" + ], + "stewardAttested": true, + "attestedAt": "2026-07-28T11:40:00Z" + } + }, + { + "id": "urn:srcos:layer-evidence:ops-orders-status-code-l4", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L4-operational-semantics", + "subjectColumnRef": "urn:srcos:column:ops.orders.status_code", + "candidateClassRef": "urn:srcos:data-class:order-status-code", + "evidence": { + "alpha": 0.4, + "beta": 0.0 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "witness": { + "kind": "L4", + "opProfileId": "opprofile-closed-set-code-v1", + "declaredOps": [ + "equal", + "not-equal", + "count", + "group-by" + ], + "observedOps": [ + "equal", + "count", + "group-by" + ], + "queryLogWindow": "2026-07-01/2026-08-03", + "conflictSet": [] + } + }, + { + "id": "urn:srcos:layer-evidence:ops-orders-status-code-l5", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L5-table-topic", + "subjectColumnRef": "urn:srcos:column:ops.orders.status_code", + "candidateClassRef": "urn:srcos:data-class:order-status-code", + "evidence": { + "alpha": 0.0, + "beta": 0.0 + }, + "admissibility": { + "admissible": false, + "reason": "active-drift-flag" + }, + "witness": { + "kind": "L5", + "schemaVersion": "ops.orders@v12", + "headerBagHash": "sha256:3f7b1c9e5d2a8f4b6c0e7d3a9b5f1c8e2d6a4f0b7c3e9d5a1f8b2c6e4d0a7f3b", + "topicModelId": "topic-lsa-bow-v3", + "topicVector": [0.08, 0.71, 0.15, 0.06], + "tableRef": "urn:srcos:table:ops.orders", + "driftRef": "urn:srcos:column-drift:ops-orders-status-code-2026-08" + } + }, + { + "id": "urn:srcos:layer-evidence:ops-orders-status-code-l6", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L6-schema-key-graph", + "subjectColumnRef": "urn:srcos:column:ops.orders.status_code", + "candidateClassRef": "urn:srcos:data-class:order-status-code", + "evidence": { + "alpha": 0.33, + "beta": 0.0 + }, + "admissibility": { + "admissible": true, + "reason": null + }, + "confidenceSource": "inferred", + "witness": { + "kind": "L6", + "ddlVersion": "ops@2026-06-11", + "declaredConstraints": [], + "inferredInclusionDeps": [ + "ops.orders.status_code SUBSET-OF ref.status_lookup.code" + ], + "graphSnapshotId": "keygraph-2026-08-03", + "referencedTableRef": "urn:srcos:table:ref.status_lookup" + } + } + ], + "pooling": { + "operator": "frechet-max", + "independenceCertificateRef": null, + "pooled": { + "alpha": 0.82, + "beta": 0.86 + } + }, + "aggregator": { + "modelRef": "urn:srcos:model-manifest:fingerprint-aggregator-v1", + "monotonicityEnforcement": "structural", + "constraints": [ + "M1-alpha-nondecreasing", + "M2-beta-nondecreasing", + "M4-zero-in-zero-out", + "M5-layer-knockout-lowers" + ] + }, + "quorum": { + "nEff": 4.8, + "nEffFloor": 3.0, + "covarianceSpectrum": [2.0, 1.0, 1.0, 1.0, 0.5, 0.5], + "herfindahl": 0.31 + }, + "stanceRef": "urn:srcos:stance:ops-orders-status-code-order-status", + "stance": { + "id": "urn:srcos:stance:ops-orders-status-code-order-status", + "type": "ClassificationStance", + "specVersion": "2.0.0", + "reading": "support", + "evidence": { + "alpha": 0.82, + "beta": 0.86 + }, + "thresholds": { + "tauPos": 0.7, + "tauNeg": 0.7, + "attestationRef": null + }, + "value": "INADMISSIBLE", + "inadmissibleOrigin": "profile-drift", + "composition": { + "depth": 2, + "baseEpsilon": 0.1, + "effectiveEpsilon": 0.2 + }, + "guards": [ + { + "name": "l5-drift-annihilator", + "fired": true, + "inputs": ["witness.driftRef"], + "reason": "L5-D2: silent repurpose detected — L5 annihilated so it cannot outvote L2" + }, + { + "name": "n-eff-floor", + "fired": false, + "inputs": ["witness.quorum.nEff"], + "reason": null + } + ], + "calibrationRef": "urn:srcos:run:fingerprint-calibration-2026-07-30" + } +} diff --git a/examples/data_class.currency.json b/examples/data_class.currency.json index 87ae442..a1041a4 100644 --- a/examples/data_class.currency.json +++ b/examples/data_class.currency.json @@ -16,7 +16,7 @@ "dq.currency.usd.scale2" ], "classifier": { - "kind": "tf-lattice-wide-and-deep", + "kind": "monotone-logistic", "modelRef": "urn:srcos:model-manifest:dataclass-assigner-lattice-v1", "version": "1.0.0", "runRef": "urn:srcos:run:dataclass-assigner-train-001", @@ -35,6 +35,18 @@ ], "head": "logistic", "evalRunRef": "urn:srcos:run:dataclass-currency-logistic-eval-001", - "threshold": 0.5 + "threshold": 0.5, + "fingerprintStack": { + "assigningLayers": [ + "L1-ontodt-type", + "L2-ontodq-profile", + "L3-business-glossary", + "L4-operational-semantics", + "L6-schema-key-graph" + ], + "aggregatorRef": "urn:srcos:model-manifest:fingerprint-aggregator-v1", + "estateAdmissibilityRef": "urn:srcos:estate-admissibility:berger-foods-phase0-2026-08-04", + "enforcementPolicyRef": "urn:srcos:classification-enforcement:estate-default-v1" + } } } diff --git a/examples/estate_admissibility_report.phase0.json b/examples/estate_admissibility_report.phase0.json new file mode 100644 index 0000000..323914e --- /dev/null +++ b/examples/estate_admissibility_report.phase0.json @@ -0,0 +1,75 @@ +{ + "id": "urn:srcos:estate-admissibility:berger-foods-phase0-2026-08-04", + "type": "EstateAdmissibilityReport", + "specVersion": "2.0.0", + "estateRef": "urn:srcos:estate:berger-foods", + "generatedAt": "2026-08-04T06:00:00Z", + "coldStartPhase": 0, + "layers": [ + { + "layer": "L1-ontodt-type", + "admissible": true, + "reason": null, + "measured": 2.91, + "floor": 1.5, + "degraded": false + }, + { + "layer": "L2-ontodq-profile", + "admissible": true, + "reason": null, + "measured": 1240000, + "floor": 1000, + "degraded": false + }, + { + "layer": "L3-business-glossary", + "admissible": false, + "reason": "glossary-absent-or-thin", + "measured": 0, + "floor": 50, + "degraded": false + }, + { + "layer": "L4-operational-semantics", + "admissible": false, + "reason": "no-declared-operation-profile", + "measured": 0, + "floor": 1, + "degraded": false + }, + { + "layer": "L5-table-topic", + "admissible": true, + "reason": null, + "measured": null, + "floor": null, + "degraded": true + }, + { + "layer": "L6-schema-key-graph", + "admissible": true, + "reason": null, + "measured": 0.08, + "floor": 0.2, + "degraded": true + } + ], + "quorum": { + "nEff": 3.6, + "nEffFloor": 4.0, + "covarianceSpectrum": [2.0, 2.0, 1.0, 1.0], + "herfindahl": 0.34, + "posAvailable": false + }, + "knownDependencies": [ + { + "between": ["L3-business-glossary", "L4-operational-semantics"], + "note": "Both depend on glossary maturity; a thin glossary disables the pair, not one of them." + }, + { + "between": ["L3-business-glossary", "L5-table-topic"], + "note": "L5 is a bag of L3 labels, so it inherits L3's state rather than corroborating it independently." + } + ] +} diff --git a/fixtures/data-class/conformance.json b/fixtures/data-class/conformance.json index 52d31f0..d999463 100644 --- a/fixtures/data-class/conformance.json +++ b/fixtures/data-class/conformance.json @@ -3,8 +3,8 @@ "cases": [ { "schema": "DataClass.json", - "reason": "classifier kind is pinned to tf-lattice-wide-and-deep.", - "failValidator": "const", + "reason": "classifier kind must be one of the structurally-monotone constructions; 'random-forest' offers no monotonicity guarantee at all.", + "failValidator": "enum", "document": { "id": "urn:srcos:data-class:currency-amount-usd", "type": "DataClass", @@ -135,7 +135,7 @@ }, { "schema": "DataClass.json", - "reason": "ontologyClassRef is required — a data class must be ontologically typed.", + "reason": "ontologyClassRef is required \u2014 a data class must be ontologically typed.", "failValidator": "required", "document": { "id": "urn:srcos:data-class:currency-amount-usd", diff --git a/fixtures/fingerprint-stack/conformance.json b/fixtures/fingerprint-stack/conformance.json new file mode 100644 index 0000000..44ecf16 --- /dev/null +++ b/fixtures/fingerprint-stack/conformance.json @@ -0,0 +1,237 @@ +{ + "description": "Negative vectors for the six-layer fingerprint stack v0.1 (SP-FPRINT-STACK-001). Each MUST fail schema validation on its named failValidator; validate_fingerprint_stack.py enforces both the failure and the reason. These cover the structural guarantees — the ones that must be impossible to express, not merely discouraged: a stance without its reading tag, a closure walking parthood, monotonicity by regularization, a guard reading its own outcome, an estate report that omits a layer.", + "cases": [ + { + "schema": "ClassificationStance.json", + "reason": "reading tag omitted — the tag lives in the TYPE, not the witness, because a lost tag is silently wrong whereas a missing required field is loudly absent (DR-5).", + "failValidator": "required", + "document": { + "id": "urn:srcos:stance:no-reading-tag", + "type": "ClassificationStance", + "specVersion": "2.0.0", + "evidence": { "alpha": 0.9, "beta": 0.0 }, + "thresholds": { "tauPos": 0.7, "tauNeg": 0.7, "attestationRef": null }, + "value": "POS", + "composition": { "depth": 1, "baseEpsilon": 0.1, "effectiveEpsilon": 0.1 } + } + }, + { + "schema": "ClassificationStance.json", + "reason": "a fifth stance value — FOUR is closed; 'MAYBE' is the collapse of ZERO and INADMISSIBLE this family exists to prevent.", + "failValidator": "enum", + "document": { + "id": "urn:srcos:stance:fifth-value", + "type": "ClassificationStance", + "specVersion": "2.0.0", + "reading": "support", + "evidence": { "alpha": 0.5, "beta": 0.5 }, + "thresholds": { "tauPos": 0.7, "tauNeg": 0.7, "attestationRef": null }, + "value": "MAYBE", + "composition": { "depth": 1, "baseEpsilon": 0.1, "effectiveEpsilon": 0.1 } + } + }, + { + "schema": "ClassificationStance.json", + "reason": "guard reads the outcome — a guard must be measurable with respect to evidence and witness only; reading the outcome breaks exchangeability and voids the coverage guarantee.", + "failValidator": "pattern", + "document": { + "id": "urn:srcos:stance:outcome-reading-guard", + "type": "ClassificationStance", + "specVersion": "2.0.0", + "reading": "support", + "evidence": { "alpha": 0.9, "beta": 0.0 }, + "thresholds": { "tauPos": 0.7, "tauNeg": 0.7, "attestationRef": null }, + "value": "POS", + "composition": { "depth": 1, "baseEpsilon": 0.1, "effectiveEpsilon": 0.1 }, + "guards": [ + { + "name": "promote-if-already-pos", + "fired": true, + "inputs": ["value.stance"], + "reason": null + } + ] + } + }, + { + "schema": "ClassificationStance.json", + "reason": "evidence outside [0,1] — the evidence lattice is the unit square.", + "failValidator": "maximum", + "document": { + "id": "urn:srcos:stance:out-of-range", + "type": "ClassificationStance", + "specVersion": "2.0.0", + "reading": "support", + "evidence": { "alpha": 1.4, "beta": 0.0 }, + "thresholds": { "tauPos": 0.7, "tauNeg": 0.7, "attestationRef": null }, + "value": "POS", + "composition": { "depth": 1, "baseEpsilon": 0.1, "effectiveEpsilon": 0.1 } + } + }, + { + "schema": "FingerprintLayerEvidence.json", + "reason": "L1 closure declared over parthood — count aggregation traverses subtyping ONLY; a closure walking part_of makes the 'how many tables contain X' answer wrong by construction (axiom X1).", + "failValidator": "oneOf", + "document": { + "id": "urn:srcos:layer-evidence:closure-walks-parthood", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L1-ontodt-type", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "evidence": { "alpha": 0.7, "beta": 0.0 }, + "admissibility": { "admissible": true, "reason": null }, + "witness": { + "kind": "L1", + "ontologyVersion": "ontodt-2.0.0", + "typePath": ["datatype", "primitive-datatype"], + "closureDepth": 1, + "closureRelation": "partOf", + "axiomsApplied": [], + "reasoner": { "id": "hermit", "version": "1.4.5" } + } + } + }, + { + "schema": "FingerprintLayerEvidence.json", + "reason": "an unnamed inadmissibility reason — free text cannot be aggregated into an estate report, so causes are a closed set.", + "failValidator": "enum", + "document": { + "id": "urn:srcos:layer-evidence:freetext-reason", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L2-ontodq-profile", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "evidence": { "alpha": 0.0, "beta": 0.0 }, + "admissibility": { "admissible": false, "reason": "table looked a bit empty" }, + "witness": { + "kind": "L2", + "profileRunId": "p1", + "sampleSize": 0, + "sampleMethod": "full-scan", + "partitionKey": null, + "profiledAt": "2026-08-01T00:00:00Z", + "sourceSnapshotId": "s1", + "promotedToHardAxiom": false, + "stewardAttestationRef": null + } + } + }, + { + "schema": "ColumnFingerprint.json", + "reason": "monotonicity by regularization — a penalty term makes M5 probabilistic, which voids the gate-soundness argument; the constraint must be structural.", + "failValidator": "const", + "document": { + "id": "urn:srcos:column-fingerprint:soft-monotone", + "type": "ColumnFingerprint", + "specVersion": "2.0.0", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "layerEvidence": [ + { + "id": "urn:srcos:layer-evidence:soft-monotone-l1", + "type": "FingerprintLayerEvidence", + "specVersion": "2.0.0", + "layer": "L1-ontodt-type", + "subjectColumnRef": "urn:srcos:column:crm.customers.customer_id", + "candidateClassRef": "urn:srcos:data-class:customer-surrogate-key", + "evidence": { "alpha": 0.9, "beta": 0.0 }, + "admissibility": { "admissible": true, "reason": null }, + "witness": { + "kind": "L1", + "ontologyVersion": "ontodt-2.0.0", + "typePath": ["datatype"], + "closureDepth": 0, + "closureRelation": "subtypeOf", + "axiomsApplied": [], + "reasoner": { "id": "hermit", "version": "1.4.5" } + } + } + ], + "pooling": { + "operator": "frechet-max", + "independenceCertificateRef": null, + "pooled": { "alpha": 0.9, "beta": 0.0 } + }, + "aggregator": { + "modelRef": "urn:srcos:model-manifest:soft-aggregator", + "monotonicityEnforcement": "regularized", + "constraints": [ + "M1-alpha-nondecreasing", + "M2-beta-nondecreasing", + "M4-zero-in-zero-out", + "M5-layer-knockout-lowers" + ] + }, + "quorum": { + "nEff": 4.8, + "nEffFloor": 3.0, + "covarianceSpectrum": [2.0, 1.0, 1.0, 1.0, 0.5, 0.5], + "herfindahl": 0.28 + }, + "stanceRef": "urn:srcos:stance:soft-monotone", + "stance": { + "id": "urn:srcos:stance:soft-monotone", + "type": "ClassificationStance", + "specVersion": "2.0.0", + "reading": "support", + "evidence": { "alpha": 0.9, "beta": 0.0 }, + "thresholds": { "tauPos": 0.7, "tauNeg": 0.7, "attestationRef": null }, + "value": "POS", + "inadmissibleOrigin": null, + "composition": { "depth": 1, "baseEpsilon": 0.1, "effectiveEpsilon": 0.1 }, + "guards": [], + "calibrationRef": null + } + } + }, + { + "schema": "ColumnDriftObservation.json", + "reason": "a distributional metric outside the matched set — drift distance must be Jensen-Shannon (categorical), Wasserstein-1 (numeric) or pattern edit distance (strings), not an arbitrary similarity.", + "failValidator": "enum", + "document": { + "id": "urn:srcos:column-drift:wrong-metric", + "type": "ColumnDriftObservation", + "specVersion": "2.0.0", + "subjectColumnRef": "urn:srcos:column:ops.orders.status_code", + "baseline": { "profileRunId": "p0", "schemaVersion": "v12", "observedAt": "2026-02-02T00:00:00Z" }, + "current": { "profileRunId": "p1", "schemaVersion": "v12", "observedAt": "2026-08-03T00:00:00Z" }, + "metric": "cosine", + "distance": 0.61, + "threshold": 0.15, + "schemaChanged": false, + "drifted": true, + "verdict": "silent-repurpose", + "stewardQueueRef": "urn:srcos:steward-queue:data-governance-drift" + } + }, + { + "schema": "EstateAdmissibilityReport.json", + "reason": "a layer omitted from the estate report — an omitted layer is indistinguishable from an admissible one, which is exactly the silent averaging this report exists to prevent.", + "failValidator": "minItems", + "document": { + "id": "urn:srcos:estate-admissibility:five-layers", + "type": "EstateAdmissibilityReport", + "specVersion": "2.0.0", + "estateRef": "urn:srcos:estate:berger-foods", + "generatedAt": "2026-08-04T06:00:00Z", + "coldStartPhase": 0, + "layers": [ + { "layer": "L1-ontodt-type", "admissible": true, "reason": null }, + { "layer": "L2-ontodq-profile", "admissible": true, "reason": null }, + { "layer": "L3-business-glossary", "admissible": false, "reason": "glossary-absent-or-thin" }, + { "layer": "L4-operational-semantics", "admissible": false, "reason": "no-declared-operation-profile" }, + { "layer": "L6-schema-key-graph", "admissible": true, "reason": null } + ], + "quorum": { + "nEff": 3.6, + "nEffFloor": 4.0, + "covarianceSpectrum": [2.0, 2.0, 1.0, 1.0], + "herfindahl": 0.34, + "posAvailable": false + } + } + } + ] +} diff --git a/fixtures/fingerprint-stack/ontodt_graph.json b/fixtures/fingerprint-stack/ontodt_graph.json new file mode 100644 index 0000000..4f66a52 --- /dev/null +++ b/fixtures/fingerprint-stack/ontodt_graph.json @@ -0,0 +1,74 @@ +{ + "description": "OntoDT type-graph fragment exercising axiom X1 as a CI invariant. Slide 31 of the 2018 source deck poses two questions against one graph: 'how many total tables contain the personal data field Email Address' (a count over the SUBTYPE closure) and 'what attributes can I expect under the term Client Address' (an enumeration of PARTS). If is_a and part_of share an arrow type, those two answers cannot both be correct — one of them is necessarily wrong. X1 is therefore not a refinement of the original design; it is the precondition for the original design's own stated questions being simultaneously answerable.", + "ontologyVersion": "ontodt-2.0.0", + "subtypeOf": [ + ["integer", "numeric"], + ["real", "numeric"], + ["numeric", "primitive-datatype"], + ["email-address", "string-identifier"], + ["string-identifier", "non-numeric"], + ["non-numeric", "primitive-datatype"], + ["primitive-datatype", "datatype"], + ["generator-datatype", "datatype"], + ["user-defined-datatype", "datatype"], + ["bag", "generator-datatype"], + ["set", "generator-datatype"], + ["sequence", "generator-datatype"], + ["record", "generator-datatype"] + ], + "partOf": [ + ["street-line-1", "postal-address"], + ["street-line-2", "postal-address"], + ["city", "postal-address"], + ["postcode", "postal-address"], + ["country-code", "postal-address"], + ["postal-address", "client-record"], + ["email-address", "client-record"] + ], + "connectedTo": [ + ["postcode", "city"], + ["street-line-1", "street-line-2"] + ], + "aggregateQueries": [ + { + "question": "How many total tables contain the personal data field 'Email Address'?", + "root": "email-address", + "relation": "subtypeOf", + "members": { + "POS": 41, + "ZERO": 7, + "INADMISSIBLE": 2 + }, + "answerInterval": [41, 48], + "inadmissibleReported": 2, + "note": "Under the open-world assumption the answer is an INTERVAL [N+, N+ + N0], not a number: 41 columns are classified POS, 7 are ZERO (ignorance — they may or may not be email addresses, and the honest answer says so). The 2 INADMISSIBLE members are reported SEPARATELY and never folded into the interval, because a contradiction is not a maybe." + }, + { + "question": "How many tables contain a postal address?", + "root": "postal-address", + "relation": "subtypeOf", + "members": { + "POS": 12, + "ZERO": 3, + "INADMISSIBLE": 0 + }, + "answerInterval": [12, 15], + "inadmissibleReported": 0 + } + ], + "partEnumerations": [ + { + "question": "What attributes can I expect under the term 'Client Address'?", + "root": "postal-address", + "relation": "partOf", + "expected": [ + "street-line-1", + "street-line-2", + "city", + "postcode", + "country-code" + ], + "note": "Parthood does NOT roll up: a table containing a postcode is not thereby a table containing a postal address. This is the same graph as the count query above and a different arrow — which is exactly what X1 buys." + } + ] +} diff --git a/registry/contract-registry.json b/registry/contract-registry.json index 2477d21..9653392 100644 --- a/registry/contract-registry.json +++ b/registry/contract-registry.json @@ -256,6 +256,26 @@ "path": "schemas/ChunkContext.json", "sha256": "sha256:da9b97885456592ae07093e5825063cadb63604c7fc4943e7878510d306fa8a3" }, + "ClassificationEnforcementPolicy": { + "$id": "https://schemas.srcos.ai/v2/ClassificationEnforcementPolicy.json", + "path": "schemas/ClassificationEnforcementPolicy.json", + "sha256": "sha256:057f615dde592bd49585ad6245829fbfbeadf2079b9a42998aae9754eae54bfd" + }, + "ClassificationStance": { + "$id": "https://schemas.srcos.ai/v2/ClassificationStance.json", + "path": "schemas/ClassificationStance.json", + "sha256": "sha256:bc2774b41943b2229943bc5e399dff6178809ab24669976964a4f52a9f992d27" + }, + "ColumnDriftObservation": { + "$id": "https://schemas.srcos.ai/v2/ColumnDriftObservation.json", + "path": "schemas/ColumnDriftObservation.json", + "sha256": "sha256:b669ecf585317854d391d3fb77f02916773bb357e3aa3034998746a29072db5f" + }, + "ColumnFingerprint": { + "$id": "https://schemas.srcos.ai/v2/ColumnFingerprint.json", + "path": "schemas/ColumnFingerprint.json", + "sha256": "sha256:34c601c570755ad15d1b63d2c5ea4cde76492fc4e076997d77346691ab686efc" + }, "CommandBus": { "$id": "https://schemas.srcos.ai/v2/CommandBus.json", "path": "schemas/CommandBus.json", @@ -334,7 +354,7 @@ "DataClass": { "$id": "https://schemas.srcos.ai/v2/DataClass.json", "path": "schemas/DataClass.json", - "sha256": "sha256:5fdedf40896b8f3f9b51ab04012648688391ff0b1c9002b64b0f157e6b822355" + "sha256": "sha256:d5b9f431fd8be45b7c3de833d76a13f15339288206d54e98af593b43ac0ad82f" }, "DataContract": { "$id": "https://schemas.srcos.ai/v2/DataContract.json", @@ -471,6 +491,11 @@ "path": "schemas/EscalationDecision.json", "sha256": "sha256:96f2516bc4e7d13be91412b9057686dab92c9d7c2c189da02520394cbe11aa87" }, + "EstateAdmissibilityReport": { + "$id": "https://schemas.srcos.ai/v2/EstateAdmissibilityReport.json", + "path": "schemas/EstateAdmissibilityReport.json", + "sha256": "sha256:8673fb193c4b9412780f8894dbf1afb29b132e7b4e37adacf2a13fe6fd9ec925" + }, "EvalItem": { "$id": "https://schemas.srcos.ai/v2/EvalItem.json", "path": "schemas/EvalItem.json", @@ -571,6 +596,11 @@ "path": "schemas/control-plane/Fingerprint.json", "sha256": "sha256:5a35fe098e79db0f021342324a2973a625313b1980e0eb02df195c8f5b42ca25" }, + "FingerprintLayerEvidence": { + "$id": "https://schemas.srcos.ai/v2/FingerprintLayerEvidence.json", + "path": "schemas/FingerprintLayerEvidence.json", + "sha256": "sha256:8d0ff067d3821ad73ecf403e3341620428481b32ebfc3dc2df852a1b92972d12" + }, "FirewallBindingProfile": { "$id": "https://schemas.srcos.ai/v2/FirewallBindingProfile.json", "path": "schemas/FirewallBindingProfile.json", @@ -1797,5 +1827,5 @@ "sha256": "sha256:eadc32085987ffda647b090e3eba3726edd102a66d4736e239457bb74867004b" } }, - "count": 359 + "count": 365 } diff --git a/schemas/ClassificationEnforcementPolicy.json b/schemas/ClassificationEnforcementPolicy.json new file mode 100644 index 0000000..e507bf3 --- /dev/null +++ b/schemas/ClassificationEnforcementPolicy.json @@ -0,0 +1,116 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://schemas.srcos.ai/v2/ClassificationEnforcementPolicy.json", + "title": "ClassificationEnforcementPolicy", + "description": "How a four-valued classification stance is enforced at a policy enforcement point. This settles DR-4, the oldest open question in the stack: what a PEP does with ZERO. It was latent in the 2018 design — which already routed data-quality inference into access enforcement — and could not be addressed there, because without a ZERO value 'we do not know' had nowhere to live and silently became either over-blocking or a leak. POS and NEG are not knobs; ZERO is, per resource class, and only with an attestation; INADMISSIBLE is never a knob.", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "type", + "specVersion", + "posBehavior", + "negBehavior", + "inadmissibleBehavior", + "resourceClasses" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^urn:srcos:classification-enforcement:[A-Za-z0-9._~:-]+$" + }, + "type": { + "const": "ClassificationEnforcementPolicy" + }, + "specVersion": { + "type": "string" + }, + "posBehavior": { + "const": "allow", + "description": "A supported, unrefuted classification grants access under the resource's own rules. Not a knob." + }, + "negBehavior": { + "const": "deny", + "description": "A refuted classification denies. Not a knob. Refutation is available before assertion is — hard axioms need no calibration — so this is the enforcement path that works on day one of a cold-start estate." + }, + "inadmissibleBehavior": { + "const": "fail-closed", + "description": "A glut is NEVER fail-open, and this is deliberately not configurable. INADMISSIBLE means two layers disagree about what this column IS; one of the readings may well be 'personal data'. Granting access while the classification is self-contradictory would let a contradiction resolve itself in the requester's favour, which is the one resolution nobody chose." + }, + "resourceClasses": { + "type": "array", + "minItems": 1, + "description": "The ZERO knob, scoped per resource class. Ignorance is not a severity — it is a different question from refutation, and the right answer genuinely differs by resource: over-blocking a public reference table is a nuisance, over-sharing an unclassified column is a breach.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "resourceClass", + "sensitivity", + "zeroBehavior" + ], + "properties": { + "resourceClass": { + "type": "string", + "minLength": 1 + }, + "sensitivity": { + "enum": [ + "public", + "internal", + "confidential", + "restricted" + ] + }, + "zeroBehavior": { + "enum": [ + "fail-closed", + "fail-open" + ], + "description": "fail-closed over-blocks (a cost in access); fail-open leaks (a cost in exposure). Neither is universally right, which is why this is an attested decision rather than an implementation default." + }, + "attestationRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$", + "description": "REQUIRED when zeroBehavior is fail-open. Choosing to serve data whose classification is unknown is a decision somebody makes on the record, with a rationale, or it is an accident." + }, + "rationale": { + "type": [ + "string", + "null" + ] + } + } + } + }, + "abstentionBudget": { + "type": [ + "object", + "null" + ], + "additionalProperties": false, + "required": [ + "maxZeroFraction", + "onExceed" + ], + "description": "Governed abstention is honest, but unbounded abstention is a system that has stopped answering. The budget makes the failure loud rather than letting ZERO quietly become the default verdict for an estate nobody ever calibrated.", + "properties": { + "maxZeroFraction": { + "type": "number", + "minimum": 0, + "maximum": 1 + }, + "onExceed": { + "enum": [ + "escalate-human", + "block-estate" + ], + "description": "Never 'continue' — an abstention rate above budget means the classifier is not working on this estate, and continuing silently is how that goes unnoticed for a quarter." + } + } + } + } +} diff --git a/schemas/ClassificationStance.json b/schemas/ClassificationStance.json new file mode 100644 index 0000000..e022ba1 --- /dev/null +++ b/schemas/ClassificationStance.json @@ -0,0 +1,188 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://schemas.srcos.ai/v2/ClassificationStance.json", + "title": "ClassificationStance", + "description": "The four-valued stance a classification carries over the bilattice FOUR: POS (supported, unrefuted), NEG (refuted, unsupported), ZERO (ignorance — gather evidence), INADMISSIBLE (conflict — resolve contradiction). Distinguishing ZERO from INADMISSIBLE is the point: a gap and a glut are different failures with different remediations, and a two- or three-valued verdict cannot tell them apart. The stance is a QUANTIZATION of an evidence pair (alpha, beta) in [0,1]^2 and is re-derivable from it — stored `value` is a cache, never authority. This is the CLASSIFICATION vocabulary; it does not mix with the ops ternary (ok/sad/bad) or the 5-axis ReasoningAssay (see specs/fingerprint-stack.md).", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "type", + "specVersion", + "reading", + "evidence", + "thresholds", + "value", + "composition" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^urn:srcos:stance:[A-Za-z0-9._~:-]+$", + "description": "Stable URN identifier. Pattern: urn:srcos:stance:" + }, + "type": { + "const": "ClassificationStance" + }, + "specVersion": { + "type": "string" + }, + "reading": { + "enum": [ + "support", + "plausibility" + ], + "description": "MANDATORY, and carried in the TYPE rather than only in the witness (DR-5, resolved in favour of type: a lost tag is silently wrong, whereas a missing required field is loudly absent). A Belnap SUPPORT set and a conformal PLAUSIBILITY set are conflates of one another (Gamma = -S); mixing the two readings without applying the conflation operator silently exchanges gap and glut. That is a live bug class in any system that pipelines a conformal predictor into a logical reasoner, which is exactly what this stack does." + }, + "evidence": { + "type": "object", + "additionalProperties": false, + "required": [ + "alpha", + "beta" + ], + "description": "The pooled evidence pair in [0,1]^2 that `value` quantizes. alpha = support mass, beta = refutation mass. The two are independent coordinates, not endpoints of one axis — that independence is what makes ZERO and INADMISSIBLE expressible.", + "properties": { + "alpha": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Support mass." + }, + "beta": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Refutation mass." + } + } + }, + "thresholds": { + "type": "object", + "additionalProperties": false, + "required": [ + "tauPos", + "tauNeg" + ], + "description": "The quantizer Q_{tauPos,tauNeg}(alpha, beta) = (1[alpha >= tauPos], 1[beta >= tauNeg]). Q commutes with negation iff tauPos == tauNeg; asymmetric thresholds break negation-equivariance, which means deny-rules no longer come free by negating allow-rules and a separate deny path must be maintained. Asymmetry is therefore an ATTESTED platform setting, never an implementation accident (DR-6).", + "properties": { + "tauPos": { + "type": "number", + "minimum": 0, + "maximum": 1 + }, + "tauNeg": { + "type": "number", + "minimum": 0, + "maximum": 1 + }, + "attestationRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$", + "description": "REQUIRED when tauPos != tauNeg: the recorded platform decision, with rationale, that licenses breaking negation-equivariance. The validator refuses asymmetric thresholds without it." + } + } + }, + "value": { + "enum": [ + "POS", + "NEG", + "ZERO", + "INADMISSIBLE" + ], + "description": "The quantized stance. A CACHE of Q(evidence; thresholds) — the validator recomputes it and fails on drift, so an asserted stance cannot outrun its evidence. Knowledge order (<=_k) on the support/refute bit pair: ZERO=(0,0) <=_k POS=(1,0), NEG=(0,1) <=_k INADMISSIBLE=(1,1); POS and NEG are incomparable." + }, + "inadmissibleOrigin": { + "type": [ + "string", + "null" + ], + "enum": [ + "layer-conflict", + "declared-vs-observed-operation", + "profile-drift", + "inconsistent-knowledge-base", + "stale-calibration", + null + ], + "description": "REQUIRED (non-null) when value is INADMISSIBLE. The lattice element is identical in every case but the remediations differ completely, so the origin is recorded rather than inferred: a drift glut goes to a steward, an inconsistent-KB glut goes to justification scoping, a stale-calibration glut goes to recalibration." + }, + "composition": { + "type": "object", + "additionalProperties": false, + "required": [ + "depth", + "baseEpsilon", + "effectiveEpsilon" + ], + "description": "Conformal coverage does NOT compose. A chain of n witnessed bindings each valid at level epsilon gives at best n*epsilon by union bound. Silently reporting an unearned guarantee across a deep composition is the exact failure this architecture exists to prevent, so the depth and the loosened level are carried on the stance itself.", + "properties": { + "depth": { + "type": "integer", + "minimum": 1, + "description": "Number of composed witnessed bindings that produced this stance. 1 = a single calibrated prediction." + }, + "baseEpsilon": { + "type": "number", + "exclusiveMinimum": 0, + "maximum": 1, + "description": "The per-binding calibration miscoverage level." + }, + "effectiveEpsilon": { + "type": "number", + "exclusiveMinimum": 0, + "maximum": 1, + "description": "The honestly-loosened level actually guaranteed after composition. Must be >= depth * baseEpsilon (union bound); the validator refuses a stance claiming the base level at depth > 1." + } + } + }, + "guards": { + "type": "array", + "description": "Guards applied as multiplicative (tensor) factors. Guards can only LOWER stance in the knowledge order — no composition of guards can raise necessity or turn ZERO into POS. Because lowering in <=_k widens the plausibility set, gated stances RETAIN the conformal coverage guarantee: policy enforcement costs efficiency (wider sets, more abstention) and never costs validity.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "fired", + "inputs" + ], + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "fired": { + "type": "boolean" + }, + "inputs": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "pattern": "^(evidence|witness)\\." + }, + "description": "Side condition enforced statically: a guard must be measurable with respect to EVIDENCE and WITNESS only. A guard that reads the outcome breaks exchangeability and voids the coverage guarantee, so input paths are restricted to `evidence.*` and `witness.*` by pattern — `value.*` or `stance.*` cannot be named." + }, + "reason": { + "type": [ + "string", + "null" + ] + } + } + } + }, + "calibrationRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$", + "description": "The calibration run backing baseEpsilon. Freshness is an admissibility precondition." + } + } +} diff --git a/schemas/ColumnDriftObservation.json b/schemas/ColumnDriftObservation.json new file mode 100644 index 0000000..9e414e9 --- /dev/null +++ b/schemas/ColumnDriftObservation.json @@ -0,0 +1,121 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://schemas.srcos.ai/v2/ColumnDriftObservation.json", + "title": "ColumnDriftObservation", + "description": "The L5 drift detector: a rolling L2 profile distance for one column against its baseline profile, used to catch SILENT REPURPOSE — a column reassigned to a new meaning without a rename. Schema structure is highly stable (tables are not redesigned except during deliberate redesign), which is what makes the table-topic layer a low-variance prior; it is also what makes it dangerous, because under silent repurpose L5's evidence is precisely the UNCHANGED schema and it will assert stale semantics with maximum confidence exactly when it is wrong. This detector is blocking for L5 admissibility, not advisory.", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "type", + "specVersion", + "subjectColumnRef", + "baseline", + "current", + "metric", + "distance", + "threshold", + "schemaChanged", + "drifted" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^urn:srcos:column-drift:[A-Za-z0-9._~:-]+$" + }, + "type": { + "const": "ColumnDriftObservation" + }, + "specVersion": { + "type": "string" + }, + "subjectColumnRef": { + "type": "string", + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$" + }, + "baseline": { + "type": "object", + "additionalProperties": false, + "required": [ + "profileRunId", + "schemaVersion", + "observedAt" + ], + "properties": { + "profileRunId": { + "type": "string" + }, + "schemaVersion": { + "type": "string" + }, + "observedAt": { + "type": "string", + "format": "date-time" + } + } + }, + "current": { + "type": "object", + "additionalProperties": false, + "required": [ + "profileRunId", + "schemaVersion", + "observedAt" + ], + "properties": { + "profileRunId": { + "type": "string" + }, + "schemaVersion": { + "type": "string" + }, + "observedAt": { + "type": "string", + "format": "date-time" + } + } + }, + "metric": { + "enum": [ + "jensen-shannon", + "wasserstein-1", + "pattern-edit-distance" + ], + "description": "Distributional metric matched to the column shape: Jensen-Shannon for categorical, Wasserstein-1 for numeric, normalized edit distance on pattern cover for strings." + }, + "distance": { + "type": "number", + "minimum": 0, + "description": "D(c,t) = distance(profile(c,t), profile(c,t0))." + }, + "threshold": { + "type": "number", + "minimum": 0, + "description": "delta. Drift is declared when distance > threshold." + }, + "schemaChanged": { + "type": "boolean", + "description": "Whether schemaVersion moved between baseline and current. Drift WITH a schema change is an ordinary redesign; drift WITHOUT one is the silent-repurpose case that rule L5-D1 exists for." + }, + "drifted": { + "type": "boolean", + "description": "Cache of (distance > threshold); the validator recomputes it and fails on drift, so the flag cannot be set or cleared independently of the measurement." + }, + "verdict": { + "enum": [ + "no-drift", + "drift-with-redesign", + "silent-repurpose" + ], + "description": "RULE L5-D1: silent-repurpose (drifted AND NOT schemaChanged) puts L5 and L2 in conflict; the pair emits (1,1) — INADMISSIBLE — and routes to a steward. RULE L5-D2: L5 must never outvote L2 under drift; the aggregator receives the drift flag as a GUARD and a fired guard annihilates L5's contribution before pooling. The 2018 architecture could not express this, having no glut value to route to." + }, + "stewardQueueRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$", + "description": "REQUIRED (non-null) on a silent-repurpose verdict — the glut has to land somewhere a human sees it, or routing-to-steward is a claim rather than a behaviour." + } + } +} diff --git a/schemas/ColumnFingerprint.json b/schemas/ColumnFingerprint.json new file mode 100644 index 0000000..3d3dadf --- /dev/null +++ b/schemas/ColumnFingerprint.json @@ -0,0 +1,183 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://schemas.srcos.ai/v2/ColumnFingerprint.json", + "title": "ColumnFingerprint", + "description": "The aggregated six-layer classification of one column against one candidate DataClass: the per-layer evidence emissions, the pooled evidence, the aggregator that pooled them, and the resulting ClassificationStance. The aggregator produces evidence in [0,1]^2; guards can only lower stance in the knowledge order; quantization to the four-valued stance happens LAST. The ordering is the safety property — the pipeline is structurally incapable of manufacturing confidence.", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "type", + "specVersion", + "subjectColumnRef", + "candidateClassRef", + "layerEvidence", + "pooling", + "quorum", + "stanceRef", + "stance" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^urn:srcos:column-fingerprint:[A-Za-z0-9._~:-]+$" + }, + "type": { + "const": "ColumnFingerprint" + }, + "specVersion": { + "type": "string" + }, + "subjectColumnRef": { + "type": "string", + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$" + }, + "candidateClassRef": { + "type": "string", + "pattern": "^urn:srcos:data-class:[A-Za-z0-9._~:-]+$" + }, + "layerEvidence": { + "type": "array", + "minItems": 1, + "maxItems": 6, + "items": { + "$ref": "FingerprintLayerEvidence.json" + }, + "description": "The emissions being pooled. Inline rather than by reference so the pooling is auditable from one document." + }, + "pooling": { + "type": "object", + "additionalProperties": false, + "required": [ + "operator", + "pooled" + ], + "description": "Default pooling across ADMISSIBLE layers is the Frechet bound (max alpha, max beta), which assumes nothing about dependence. Its known cost is idempotence: ten corroborating layers buy nothing over one. That is deliberate and conservative — corroboration pays only where the right to claim it has been EARNED.", + "properties": { + "operator": { + "enum": [ + "frechet-max", + "probabilistic-sum" + ], + "description": "probabilistic-sum is a licensed strengthening and requires a certified independence model for the specific layer pair; the certificate — how independence was established, on what data, when — goes in the witness." + }, + "independenceCertificateRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$", + "description": "REQUIRED when operator is probabilistic-sum." + }, + "pooled": { + "type": "object", + "additionalProperties": false, + "required": [ + "alpha", + "beta" + ], + "properties": { + "alpha": { + "type": "number", + "minimum": 0, + "maximum": 1 + }, + "beta": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Refutation is monotone non-decreasing across pooling: a veto cannot be outvoted WITHIN a pooling step. Retraction requires removing the source, which matches consent-bound revocation rather than majority overrule." + } + } + } + } + }, + "aggregator": { + "type": [ + "object", + "null" + ], + "additionalProperties": false, + "required": [ + "modelRef", + "monotonicityEnforcement", + "constraints" + ], + "description": "The learned aggregator, when one is used in place of bare pooling. Constrained monotone in the knowledge order: it fits the actual interaction structure of the estate while remaining INCAPABLE of the pathology hand-tuning was protecting against — a low-evidence input producing a high-confidence output. The constraint is the governance; the learning is the fit.", + "properties": { + "modelRef": { + "type": "string", + "pattern": "^urn:srcos:model-manifest:", + "description": "A cataloged ModelManifest — the aggregator is a first-class model, not an implementation detail." + }, + "monotonicityEnforcement": { + "const": "structural", + "description": "Monotonicity must be STRUCTURALLY guaranteed, never regularization-encouraged. A penalty term makes M5 probabilistic, which voids the gate-soundness argument; a library offering only soft constraints is rejected on those grounds." + }, + "constraints": { + "type": "array", + "minItems": 4, + "items": { + "enum": [ + "M1-alpha-nondecreasing", + "M2-beta-nondecreasing", + "M3-beta-suppresses-alpha", + "M4-zero-in-zero-out", + "M5-layer-knockout-lowers" + ] + }, + "description": "M1, M2, M4 and M5 are mandatory; M3 (refutation suppressing support directly) is intuitive but may over-constrain the fit and is measured with and without. M5 is the ensemble-level restatement of gate soundness and is property-tested — and the test must be shown to BIND, not merely to pass." + } + } + }, + "quorum": { + "type": "object", + "additionalProperties": false, + "required": [ + "nEff", + "nEffFloor" + ], + "description": "Effective independence, NOT concentration. The Herfindahl index over normalized layer contributions measures concentration of MAGNITUDE, not dependence between layers: two perfectly correlated layers contributing equally give H = 0.5, which looks healthy, while supplying one layer's worth of information. H catches 'one layer dominates'. It does not catch 'my layers are secretly the same layer', which is the failure that matters for quorum.", + "properties": { + "nEff": { + "type": "number", + "exclusiveMinimum": 0, + "description": "Participation ratio of the layer covariance spectrum: (sum lambda_i)^2 / sum lambda_i^2, estimated on the calibration set. Given the known L3-L4 dependence (both track glossary maturity) and L5's dependence on L3 (it is a bag of L3 labels), expect n_eff < 6 on every real estate. Report it." + }, + "nEffFloor": { + "type": "number", + "exclusiveMinimum": 0, + "description": "The floor required for POS. Below it, stance is forced to ZERO." + }, + "covarianceSpectrum": { + "type": [ + "array", + "null" + ], + "items": { + "type": "number", + "minimum": 0 + }, + "description": "The eigenvalues n_eff was computed from. Present so the claimed n_eff can be RECOMPUTED rather than trusted." + }, + "herfindahl": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1, + "description": "Retained as a CHEAP NECESSARY condition evaluated first, never as the quorum statistic. A quorum claim resting on H alone is refused." + } + } + }, + "stanceRef": { + "type": "string", + "pattern": "^urn:srcos:stance:[A-Za-z0-9._~:-]+$" + }, + "stance": { + "$ref": "ClassificationStance.json", + "description": "Inlined so the quantization can be recomputed from the pooled evidence in one document." + } + } +} diff --git a/schemas/DataClass.json b/schemas/DataClass.json index 1fed150..522abc7 100644 --- a/schemas/DataClass.json +++ b/schemas/DataClass.json @@ -2,7 +2,7 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://schemas.srcos.ai/v2/DataClass.json", "title": "DataClass", - "description": "An OntoDT/OntoDQ data class: a data type/class defined ontologically (mereotopology in the datatype ontology) with its value domain, its business glossary term, and an optional TF-Lattice wide-and-deep classifier — registered in the model catalog with a run on Ray/TritFabric — that assigns this class by inference. Bridges the business vocabulary (GlossaryTerm) to the data (EntityField). Governs data quality: a field bound to this class must conform to its domain.", + "description": "An OntoDT/OntoDQ data class: a data type/class defined ontologically (mereotopology in the datatype ontology) with its value domain, its business glossary term, and an optional structurally-monotone classifier \u2014 registered in the model catalog with a run on Ray/TritFabric \u2014 that assigns this class by inference. Bridges the business vocabulary (GlossaryTerm) to the data (EntityField). Governs data quality: a field bound to this class must conform to its domain.", "type": "object", "additionalProperties": false, "required": [ @@ -27,12 +27,12 @@ "ontologyClassRef": { "type": "string", "pattern": "^urn:srcos:ontology-class:[A-Za-z0-9._~:-]+$", - "description": "OntoDT — the datatype-ontology class this data class IS (capture; enables inference-based data-quality logic)." + "description": "OntoDT \u2014 the datatype-ontology class this data class IS (capture; enables inference-based data-quality logic)." }, "glossaryTermRef": { "type": "string", "pattern": "^urn:srcos:glossary:", - "description": "The business GlossaryTerm this data class realizes — the biz↔data bridge." + "description": "The business GlossaryTerm this data class realizes \u2014 the biz\u2194data bridge." }, "domain": { "$ref": "ValidValues.json", @@ -51,7 +51,7 @@ "null" ], "additionalProperties": false, - "description": "Optional TF-Lattice wide-and-deep classifier that ASSIGNS this data class by inference. It is a first-class cataloged model: modelRef → ModelManifest, runRef → RunRecord, compute on Ray/TritFabric, labels are GlossaryTerms.", + "description": "Optional structurally-monotone classifier that ASSIGNS this data class by inference. It is a first-class cataloged model: modelRef \u2192 ModelManifest, runRef \u2192 RunRecord, compute on Ray/TritFabric, labels are GlossaryTerms.", "required": [ "kind", "head", @@ -64,7 +64,13 @@ ], "properties": { "kind": { - "const": "tf-lattice-wide-and-deep" + "enum": [ + "monotone-logistic", + "monotone-lattice-hll", + "monotone-constrained-nn", + "tf-lattice-wide-and-deep" + ], + "description": "The assignment model's construction. What matters is that monotonicity in the declared monotonicFeatures is STRUCTURAL, not that any particular library produced it. 'tf-lattice-wide-and-deep' is DEPRECATED and retained only so existing documents remain valid: the TensorFlow Lattice repository was archived by its owner in April 2026 and is read-only, with its last release in July 2024. Its late releases moved Keras imports to tf_keras (because tf.keras defaults to Keras 3 from TF 2.16) and deprecated Estimators \u2014 it migrated ONTO the Keras 3 era via a compatibility shim and was then archived; it did not become part of Keras. 'monotone-logistic' is what this estate actually trains and ships (weights projected >= 0 each gradient step). 'monotone-lattice-hll' is the Hierarchical Lattice Layer successor, which trains with standard SGD instead of projected gradient descent under many constraints and accepts the high-dimensional input the older lattice layer could not hold in memory \u2014 which is exactly the 6-layer x 2-component input here." }, "modelRef": { "type": "string", @@ -123,7 +129,7 @@ "type": "string", "pattern": "^urn:srcos:glossary:" }, - "description": "Output class labels — each a GlossaryTerm URN (labels are assigned in the glossary)." + "description": "Output class labels \u2014 each a GlossaryTerm URN (labels are assigned in the glossary)." }, "head": { "const": "logistic", @@ -132,7 +138,7 @@ "evalRunRef": { "type": "string", "pattern": "^urn:srcos:run:", - "description": "The per-class evaluation RunRecord — the individual test for this class/label." + "description": "The per-class evaluation RunRecord \u2014 the individual test for this class/label." }, "threshold": { "type": [ @@ -142,6 +148,52 @@ "minimum": 0, "maximum": 1, "description": "Decision threshold on the logistic probability." + }, + "fingerprintStack": { + "type": [ + "object", + "null" + ], + "additionalProperties": false, + "required": [ + "assigningLayers", + "aggregatorRef", + "enforcementPolicyRef" + ], + "description": "Binds this DataClass to the six-layer fingerprint stack: which metadata layers assign it, the monotone aggregator that pools them, and how the resulting four-valued stance is enforced. The layers are listed because per-layer admissibility is a primary OUTPUT \u2014 an estate must be told which of its layers are trustworthy rather than having unreliable ones silently averaged in.", + "properties": { + "assigningLayers": { + "type": "array", + "minItems": 1, + "maxItems": 6, + "uniqueItems": true, + "items": { + "enum": [ + "L1-ontodt-type", + "L2-ontodq-profile", + "L3-business-glossary", + "L4-operational-semantics", + "L5-table-topic", + "L6-schema-key-graph" + ] + } + }, + "aggregatorRef": { + "type": "string", + "pattern": "^urn:srcos:model-manifest:" + }, + "estateAdmissibilityRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:estate-admissibility:" + }, + "enforcementPolicyRef": { + "type": "string", + "pattern": "^urn:srcos:classification-enforcement:" + } + } } } } diff --git a/schemas/EstateAdmissibilityReport.json b/schemas/EstateAdmissibilityReport.json new file mode 100644 index 0000000..21763a7 --- /dev/null +++ b/schemas/EstateAdmissibilityReport.json @@ -0,0 +1,182 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://schemas.srcos.ai/v2/EstateAdmissibilityReport.json", + "title": "EstateAdmissibilityReport", + "description": "Which fingerprint layers are trustworthy for THIS estate, with the reason each unusable one is unusable, plus the estate-level effective independence. This is a first-class deliverable and ships BEFORE any classification does: it is derivable from the catalog and the data alone, needs no glossary and no calibration, and is therefore the honest day-one output for a customer who has neither. The system must report which layers it can trust rather than silently averaging over unreliable ones.", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "type", + "specVersion", + "estateRef", + "generatedAt", + "coldStartPhase", + "layers", + "quorum" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^urn:srcos:estate-admissibility:[A-Za-z0-9._~:-]+$" + }, + "type": { + "const": "EstateAdmissibilityReport" + }, + "specVersion": { + "type": "string" + }, + "estateRef": { + "type": "string", + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$" + }, + "generatedAt": { + "type": "string", + "format": "date-time" + }, + "coldStartPhase": { + "type": "integer", + "minimum": 0, + "maximum": 4, + "description": "0 = catalog-derived bootstrap (L1/L2/L5/L6 only, no human input; every class assignment ZERO, but hard axioms from a reference ontology carry NEG from day one — refutation is available before assertion is). 1 = candidate glossary proposed from fingerprint clusters, each steward adjudication simultaneously a glossary entry AND a calibration datum. 2 = calibration floor cleared, POS becomes available per class. 3 = query logs attached, L4's observed channel live. 4 = steady state, all six layers, n_eff reported, drift detection running. A customer with a maintained glossary would not need this system, so L3 and L4 being absent at engagement is the NORMAL state, not an error.", + "default": 0 + }, + "layers": { + "type": "array", + "minItems": 6, + "maxItems": 6, + "description": "All six layers, always — an omitted layer would be indistinguishable from an admissible one.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "layer", + "admissible" + ], + "properties": { + "layer": { + "enum": [ + "L1-ontodt-type", + "L2-ontodq-profile", + "L3-business-glossary", + "L4-operational-semantics", + "L5-table-topic", + "L6-schema-key-graph" + ] + }, + "admissible": { + "type": "boolean" + }, + "reason": { + "type": [ + "string", + "null" + ], + "enum": [ + "declared-type-entropy-below-floor", + "row-count-below-floor", + "sample-coverage-below-floor", + "glossary-absent-or-thin", + "no-declared-operation-profile", + "too-few-labelled-columns", + "active-drift-flag", + "constraint-density-below-floor", + "witness-unresolvable", + "witness-stale", + "version-incompatible", + null + ], + "description": "REQUIRED (non-null) when admissible is false." + }, + "measured": { + "type": [ + "number", + "null" + ], + "description": "The estate statistic the precondition was evaluated against (declared-type entropy, row count, attested term count, labelled-column count, constraint density)." + }, + "floor": { + "type": [ + "number", + "null" + ], + "description": "The precondition floor. Present so 'inadmissible' is a MEASUREMENT with a stated bar, not a verdict." + }, + "degraded": { + "type": "boolean", + "default": false, + "description": "Admissible but operating on a fallback channel — L4 without query logs (declared half only), or L6 falling back to inferred inclusion dependencies, which caps the alpha it may contribute. Degrade, don't fail." + } + } + } + }, + "quorum": { + "type": "object", + "additionalProperties": false, + "required": [ + "nEff", + "nEffFloor", + "posAvailable" + ], + "properties": { + "nEff": { + "type": "number", + "exclusiveMinimum": 0 + }, + "nEffFloor": { + "type": "number", + "exclusiveMinimum": 0 + }, + "covarianceSpectrum": { + "type": [ + "array", + "null" + ], + "items": { + "type": "number", + "minimum": 0 + }, + "description": "Eigenvalues of the empirical layer covariance on the calibration set; n_eff is RECOMPUTED from these rather than trusted." + }, + "herfindahl": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1, + "description": "Cheap necessary precheck only. Never the quorum statistic." + }, + "posAvailable": { + "type": "boolean", + "description": "Whether POS is reachable at all on this estate. False below the n_eff floor or before the per-class calibration floor is cleared — in which case the honest outputs are ZERO and NEG only." + } + } + }, + "knownDependencies": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "between", + "note" + ], + "properties": { + "between": { + "type": "array", + "minItems": 2, + "maxItems": 2, + "items": { + "type": "string" + } + }, + "note": { + "type": "string" + } + } + }, + "description": "Declared, not discovered-and-forgotten. L3 and L4 are correlated (both depend on glossary maturity) and L5 depends on L3 (it is a bag of L3 labels). These dependencies are exactly why n_eff is required rather than a naive count of contributing layers." + } + } +} diff --git a/schemas/FingerprintLayerEvidence.json b/schemas/FingerprintLayerEvidence.json new file mode 100644 index 0000000..96c5c8c --- /dev/null +++ b/schemas/FingerprintLayerEvidence.json @@ -0,0 +1,430 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://schemas.srcos.ai/v2/FingerprintLayerEvidence.json", + "title": "FingerprintLayerEvidence", + "description": "One metadata fingerprint layer's emission for one (column, candidate DataClass) pair: an evidence pair in [0,1]^2, a layer-specific witness, and an admissibility flag. Six layers are specified because they FAIL INDEPENDENTLY — that, not feature count, is the argument. L1 dies where every column is declared VARCHAR; L2 dies on an empty table, which is ANTI-correlated with L1's failure (a newly-declared table has good types and no data; a legacy estate has bad types and abundant data). An INADMISSIBLE layer contributes ZERO — the tensor annihilator — never a guessed value.", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "type", + "specVersion", + "layer", + "subjectColumnRef", + "candidateClassRef", + "evidence", + "admissibility", + "witness" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^urn:srcos:layer-evidence:[A-Za-z0-9._~:-]+$" + }, + "type": { + "const": "FingerprintLayerEvidence" + }, + "specVersion": { + "type": "string" + }, + "layer": { + "enum": [ + "L1-ontodt-type", + "L2-ontodq-profile", + "L3-business-glossary", + "L4-operational-semantics", + "L5-table-topic", + "L6-schema-key-graph" + ], + "description": "L1 type fingerprint (OntoDT); L2 profiling fingerprint (OntoDQ — the ONLY layer that reads data values); L3 business semantic fingerprint (closed glossary, so this is bounded supervised classification, not open-domain NLP); L4 operational semantics (which operations are admissible on the value — no counterpart in commercial catalog tooling); L5 table topic (bag of L3 labels across the header); L6 schema and key graph." + }, + "subjectColumnRef": { + "type": "string", + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$", + "description": "The column being classified." + }, + "candidateClassRef": { + "type": "string", + "pattern": "^urn:srcos:data-class:[A-Za-z0-9._~:-]+$", + "description": "The candidate DataClass this evidence is emitted FOR. Evidence is always per (column, candidate) pair — never a bare column score." + }, + "evidence": { + "type": "object", + "additionalProperties": false, + "required": [ + "alpha", + "beta" + ], + "properties": { + "alpha": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Support. L1 CANNOT refute on its own except through a hard disjointness axiom; a type-path compatibility score is support-only." + }, + "beta": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Refutation. Non-zero only from a hard axiom, a CERTIFIED (never inferred) valid-value violation, or a declared-vs-observed operation conflict. Refutation is available before assertion is — hard axioms need no calibration, so an estate can carry NEG on day one." + } + } + }, + "admissibility": { + "type": "object", + "additionalProperties": false, + "required": [ + "admissible" + ], + "description": "Per-layer admissibility. Reporting WHICH LAYERS ARE TRUSTWORTHY FOR THIS ESTATE is a primary output alongside the classification, not defensive engineering: the alternative is silently averaging over unreliable layers.", + "properties": { + "admissible": { + "type": "boolean" + }, + "reason": { + "type": [ + "string", + "null" + ], + "enum": [ + "declared-type-entropy-below-floor", + "row-count-below-floor", + "sample-coverage-below-floor", + "glossary-absent-or-thin", + "no-declared-operation-profile", + "too-few-labelled-columns", + "active-drift-flag", + "constraint-density-below-floor", + "witness-unresolvable", + "witness-stale", + "version-incompatible", + null + ], + "description": "REQUIRED (non-null) when admissible is false. Named causes, not free text, so the estate admissibility report can be aggregated." + } + } + }, + "confidenceSource": { + "type": [ + "string", + "null" + ], + "enum": [ + "declared", + "inferred", + null + ], + "description": "L6 in particular degrades rather than failing: where DDL declares no constraints (extremely common in analytic warehouses and lakehouse tables) it falls back to discovered inclusion dependencies, and an `inferred` source CAPS the alpha this layer may contribute. Same degrade-don't-fail rule applies to L4 when no query log is reachable." + }, + "witness": { + "description": "The layer-specific witness. Discriminated on `layer` — each layer must produce ITS witness shape, so provenance cannot be satisfied by an empty or generic object.", + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "ontologyVersion", + "typePath", + "closureRelation", + "reasoner" + ], + "properties": { + "kind": { + "const": "L1" + }, + "ontologyVersion": { + "type": "string" + }, + "typePath": { + "type": "array", + "minItems": 1, + "items": { + "type": "string" + }, + "description": "Path through the datatype ontology, e.g. datatype > primitive > numeric > ordered > integer." + }, + "closureDepth": { + "type": [ + "integer", + "null" + ], + "minimum": 0 + }, + "closureRelation": { + "const": "subtypeOf", + "description": "AXIOM X1 CONSEQUENCE. Count aggregation traverses subtyping ONLY, never parthood: every table with an `x` is a table with a `y` when x is a KIND of y, but not when x is a PART of y. Pinning this to a const is what stops a closure silently walking the wrong relation." + }, + "axiomsApplied": { + "type": "array", + "items": { + "type": "string" + } + }, + "reasoner": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "version" + ], + "properties": { + "id": { + "type": "string" + }, + "version": { + "type": "string" + } + } + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "profileRunId", + "sampleSize", + "sampleMethod", + "profiledAt", + "sourceSnapshotId" + ], + "properties": { + "kind": { + "const": "L2" + }, + "profileRunId": { + "type": "string" + }, + "sampleSize": { + "type": "integer", + "minimum": 0 + }, + "sampleMethod": { + "type": "string", + "enum": [ + "full-scan", + "reservoir", + "systematic", + "partition-stratified" + ] + }, + "partitionKey": { + "type": [ + "string", + "null" + ] + }, + "profiledAt": { + "type": "string", + "format": "date-time" + }, + "sourceSnapshotId": { + "type": "string" + }, + "promotedToHardAxiom": { + "type": "boolean", + "default": false, + "description": "Q3 INVARIANT, load-bearing. Profiling enters as a DEFEASIBLE PRIOR. A profiling-inferred constraint is never promoted to a hard axiom on its own; promotion requires steward attestation. Violating Q3 is the mechanism by which a bad estate poisons its own ontology — the observed becomes the mandatory, and then the estate can never be told it is wrong." + }, + "stewardAttestationRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$", + "description": "REQUIRED when promotedToHardAxiom is true." + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "glossaryVersion", + "termId", + "matchChannel", + "channelScores" + ], + "properties": { + "kind": { + "const": "L3" + }, + "glossaryVersion": { + "type": "string" + }, + "termId": { + "type": "string", + "pattern": "^urn:srcos:glossary:" + }, + "matchChannel": { + "enum": [ + "lexical", + "structural", + "deductive" + ], + "description": "Three DISTINCT problem classes, not three views of one, and they fail independently. LEXICAL is signal recovery: the intended term is present but corrupted — vowel-stripped for header width (CUST_ADDR_LN1), truncated, encoding-mangled, transposed. That is not a semantic question. STRUCTURAL is ontology-as-evidence: the exact label is absent but position in the hierarchy is informative. DEDUCTIVE is explicit override: structure and lexis both mislead and a disjointness axiom settles it. A single embedding cosine collapses all three into one scalar and destroys the ability to ask which one fired." + }, + "channelScores": { + "type": "object", + "additionalProperties": { + "type": "number" + }, + "minProperties": 1, + "description": "Per-metric scores retained individually (levenshtein, jaroWinkler, hamming, lcs, mostFrequentKChars, smc). Retained, not pooled away — the whole point of keeping the channel is being able to ask which metric carried the match." + }, + "synonymPath": { + "type": "array", + "items": { + "type": "string" + } + }, + "stewardAttested": { + "type": "boolean", + "default": false + }, + "attestedAt": { + "type": [ + "string", + "null" + ], + "format": "date-time" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "opProfileId", + "declaredOps", + "observedOps", + "conflictSet" + ], + "properties": { + "kind": { + "const": "L4" + }, + "opProfileId": { + "type": "string" + }, + "declaredOps": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Operations the ontology says are admissible on this class. A surrogate key admits equality and join but NOT addition, multiplication, mean, or ordering — ordering a surrogate key is meaningless. A measured quantity admits arithmetic but not join-as-identity." + }, + "observedOps": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Operations actually applied, mined from production query logs. Empty when no query log is reachable — the declared channel still emits (degrade, don't fail)." + }, + "queryLogWindow": { + "type": [ + "string", + "null" + ] + }, + "conflictSet": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Observed operations inadmissible under the declaration — e.g. production takes AVG() of a column the ontology declares a surrogate key. A non-empty conflict set is a GLUT (both evidence components positive), not a low-confidence match: it means either the label is wrong or downstream code has a bug, and BOTH require a human." + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "schemaVersion", + "headerBagHash", + "topicModelId", + "tableRef" + ], + "properties": { + "kind": { + "const": "L5" + }, + "schemaVersion": { + "type": "string" + }, + "headerBagHash": { + "type": "string" + }, + "topicModelId": { + "type": "string" + }, + "topicVector": { + "type": "array", + "items": { + "type": "number" + } + }, + "tableRef": { + "type": "string", + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$" + }, + "driftRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:column-drift:[A-Za-z0-9._~:-]+$", + "description": "The drift observation, when one exists. THE STABILITY THAT MAKES L5 A GOOD PRIOR MAKES IT A DANGEROUS ONE: if a column is silently repurposed without a rename, the schema does not change, L5's evidence is PRECISELY the unchanged schema, and L5 asserts the stale semantics with maximum confidence exactly when it is wrong. That is the worst possible correlation between confidence and error, and it is why the drift detector is blocking rather than advisory." + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "ddlVersion", + "declaredConstraints", + "graphSnapshotId" + ], + "properties": { + "kind": { + "const": "L6" + }, + "ddlVersion": { + "type": "string" + }, + "declaredConstraints": { + "type": "array", + "items": { + "type": "string" + } + }, + "inferredInclusionDeps": { + "type": "array", + "items": { + "type": "string" + } + }, + "graphSnapshotId": { + "type": "string" + }, + "referencedTableRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$", + "description": "For a foreign key: the target table, whose L5 topic is strong evidence for THIS column's class." + } + } + } + ] + } + } +} diff --git a/specs/fingerprint-stack.md b/specs/fingerprint-stack.md new file mode 100644 index 0000000..06d1af1 --- /dev/null +++ b/specs/fingerprint-stack.md @@ -0,0 +1,239 @@ +# Six-Layer Fingerprint Stack (v0.1) — classification that can say "I don't know" + +Consumes SP-FPRINT-STACK-001. Assigns a business data class to a **column** from six +independent metadata fingerprint layers, pooled through a structurally-monotone aggregator, +gated so that the pipeline **cannot manufacture confidence**. + +This lands the spine: the stance algebra, the layer witnesses, the admissibility gate, the +drift detector, and the enforcement policy. What it deliberately does not land is listed +under *Not in this contract* below. + +## The one thing that is actually new + +The 2018 source design already had the three-channel matcher, the datatype ontology +(OntoDT), the mereotopological framing (its own term), and the routing of data-quality +inference into access enforcement. It had no way to say **"I don't know"** distinctly from +**"my evidence contradicts itself"**. + +Everything here follows from making those two different values: + +| Stance | Support set | Plausibility set | Meaning | Remediation | +| --- | --- | --- | --- | --- | +| `POS` | {t} | {t} | supported, unrefuted | — | +| `NEG` | {f} | {f} | refuted, unsupported | — | +| `ZERO` | ∅ | {t,f} | **ignorance** | gather evidence | +| `INADMISSIBLE` | {t,f} | ∅ | **conflict** | resolve the contradiction | + +A three-valued verdict collapses the last two into one amber, and they are not the same +problem: a gap wants more data, a glut wants a human. The empty conformal set **is** the +glut — one element, not two. + +## Vocabulary boundary — read this before adding a verdict + +This repo already carries two verdict vocabularies and holds them apart deliberately +(epistemic-kernel delta 6). `ClassificationStance` is a **third**, and it is declared here +rather than introduced silently: + +| Vocabulary | Subject | Values | +| --- | --- | --- | +| Ops ternary | an **operation's** health | `ok` / `sad` / `bad` | +| `ReasoningAssay` | a **claim**, on 5 axes | projected to the ops ternary at render time | +| `ClassificationStance` | a **(column, candidate class) binding** | `POS` / `NEG` / `ZERO` / `INADMISSIBLE` | + +They do not mix and none projects onto another: a column-class binding is not a claim and +not an operation. The nearest neighbour is the Assay's `verifier.judgment` +(`supported` / `refuted` / `abstained`) — which is three-valued with **no glut**, i.e. it is +FOUR minus exactly the value this stack exists to carry. That is a coincidence worth +noticing, not a mapping to build. + +## The six layers, and why six + +The argument is **independent failure**, not feature count. + +| Layer | Signal | Fails when | Correlated with | +| --- | --- | --- | --- | +| L1 OntoDT type | type-path compatibility | everything declared `VARCHAR` | — | +| L2 OntoDQ profile | observed statistics (the only layer that reads values) | table empty / newly seeded | **anti**-correlated with L1 | +| L3 Business glossary | closed-vocabulary label match | glossary stale or absent | correlated with L4 | +| L4 Operational semantics | which operations are admissible | no query log (observed half) | correlated with L3 | +| L5 Table topic | bag of L3 labels across the header | junk-drawer staging; **silent repurpose** | depends on L3 | +| L6 Schema / key graph | PKs, FKs, inclusion dependencies | constraints undeclared in DDL | — | + +L1 and L2 failing in opposite conditions is the ensemble working as designed: a +newly-declared table has good types and no data, a legacy estate has bad types and abundant +data. But L3–L4 and L3–L5 are **dependent**, which is why quorum is measured, not counted +(see `n_eff`). + +L4 has no counterpart in commercial catalog tooling. If production takes `AVG()` of a column +the ontology declares a surrogate key, that is a detected conflict — and it is a **glut**, +not a weak match, because it means either the label is wrong or downstream code has a bug, +and both need a human. + +## The five properties the validator enforces + +Order is the safety property: **pool → guard → quantize**. + +1. **Nothing manufactures confidence.** Guards are tensor factors; they can only lower + stance in the knowledge order. Post-guard evidence never exceeds pooled evidence in + either component. Because lowering widens the plausibility set, gated stances **retain** + their conformal coverage — policy costs efficiency (more abstention), never validity. +2. **Every stored verdict is recomputed.** The stance from its evidence, the pooled evidence + from the admissible layers, `n_eff` from the covariance spectrum, the drift flag from the + measured distance. An asserted classification cannot outrun what produced it. +3. **Quorum is effective independence, not concentration.** The Herfindahl index measures + concentration of *magnitude*: two perfectly correlated layers contributing equally give + `H = 0.5`, which looks healthy while supplying one layer's worth of information. `n_eff` + — the participation ratio of the layer covariance spectrum — is the sufficient statistic; + `H` is retained only as a cheap precheck. Expect `n_eff < 6` on every real estate. +4. **Coverage does not compose.** A chain of `n` bindings each valid at `ε` gives at best + `nε`. Composition depth and the honestly-loosened level ride on the stance. +5. **An inadmissible layer contributes ZERO** — the annihilator — never a guessed value. + +Two structural details worth stating because they are easy to get wrong: + +- **Guards are measurable with respect to evidence and witness only.** A guard that reads + the outcome breaks exchangeability and voids the coverage guarantee, so guard input paths + are restricted by pattern; `value.*` cannot be named. +- **The reading tag lives in the type**, not the witness (DR-5). Belnap support and conformal + plausibility are conflates of one another; mixing them without applying `−` silently + exchanges gap and glut. A lost tag is silently wrong; a missing required field is loudly + absent. + +## Axiom X1 — why parthood and subtyping are separate arrows + +`∀x,y. x ≠ y → ¬(P(x,y) ∧ x ⊑ y)`, enforced as a CI invariant over the OntoDT fixture. + +Slide 31 of the source deck asks two questions of one graph: *how many tables contain the +field "Email Address"* (a count over the **subtype** closure) and *what attributes can I +expect under "Client Address"* (an enumeration of **parts**). If `is_a` and `part_of` share +an arrow type, those answers cannot both be right. X1 is not a refinement of the original +design — it is the precondition for the original design's own questions being simultaneously +answerable. Count aggregation traverses `⊑` only; parthood does not roll up. + +Under the open-world assumption an aggregate count is an **interval** `[N⁺, N⁺ + N⁰]`, and +`N^INADM` is reported separately — a contradiction is not a maybe. + +## L5's drift detector is blocking, not advisory + +The stability that makes the table-topic layer a good prior makes it a dangerous one. If a +column is silently repurposed without a rename, the schema does not change, so L5's evidence +is *precisely the unchanged schema* — **L5 asserts the stale semantics with maximum +confidence exactly when it is wrong.** That is the worst possible correlation between +confidence and error. + +- **L5-D1** — drift above threshold with an unchanged `schemaVersion` puts L5 and L2 in + conflict: the pair emits a glut and routes to a steward (with a queue reference, so + "routes to a steward" is a behaviour rather than a claim). +- **L5-D2** — L5 may never outvote L2 under drift: the drift flag enters the aggregator as a + guard and annihilates L5's contribution *before* pooling. + +The 2018 architecture could not express this, having no glut value to route to. + +## Cold start — the honest day-one output + +L3 and L4 are exactly what a customer lacks when they engage; a customer with a maintained +glossary would not need this system. So their absence is the **normal state**, not an error, +and phase 0 ships something real without them: + +| Phase | What runs | What ships | +| --- | --- | --- | +| 0 | L1, L2, L5 (degraded to raw header tokens), L6 | **Estate admissibility report** + candidate glossary; every assignment `ZERO`, but hard axioms carry `NEG` from day one | +| 1 | + steward adjudication of fingerprint clusters | glossary entries that are *simultaneously* calibration data | +| 2 | + per-class calibration floor cleared | `POS` becomes available | +| 3 | + query logs attached | L4's observed channel; declared-vs-observed conflicts as the highest-value steward queue | +| 4 | all six | steady state, `n_eff` reported, drift running | + +**Refutation is available before assertion is.** Hard axioms need no calibration, so +"these columns are definitely not what they claim to be" is a genuine phase-0 deliverable. + +*Resolved ambiguity:* SP-FPRINT-STACK-001 §8 runs L5 at phase 0 while §4 preconditions L5 on +having L3-labelled columns, which do not exist yet. Here L5 is admissible at phase 0 but +`degraded: true`, running on raw header tokens rather than glossary-resolved labels. + +## DR-4 — what a PEP does with ZERO + +Settled by `ClassificationEnforcementPolicy`. This was latent in the 2018 design, which +already routed data-quality inference into access enforcement and had nowhere for "we do not +know" to live, so it silently became either over-blocking or a leak. + +- `POS → allow`, `NEG → deny`. Not knobs. +- `INADMISSIBLE → fail-closed`, **deliberately not configurable**. A glut means two layers + disagree about what the column *is*, and one reading may well be "personal data". Serving + it would let a contradiction resolve itself in the requester's favour — the one resolution + nobody chose. +- `ZERO` is the knob, **per resource class**, and constrained: fail-open requires an + attestation, and is unavailable at `confidential`/`restricted` sensitivity regardless of + attestation. Over-blocking a public reference table is a nuisance; over-sharing an + unclassified column is a breach. +- An **abstention budget** keeps governed abstention from quietly becoming a system that has + stopped answering. + +## On the aggregator library + +`DataClass.classifier.kind` was pinned by a `const` to `tf-lattice-wide-and-deep`. **The +TensorFlow Lattice repository was archived by its owner in April 2026** and is read-only; +its last release was July 2024. Its late releases moved Keras imports to `tf_keras` and +deprecated Estimators — it migrated *onto* the Keras 3 era via a compatibility shim and was +then archived. It did not become part of Keras. + +The pin is now an enum. `monotone-logistic` is what this estate actually trains and ships +(#264/#265); `monotone-lattice-hll` is the successor path, whose Hierarchical Lattice Layer +trains with standard SGD rather than projected gradient descent under many constraints and +accepts high-dimensional input the older lattice layer could not hold in memory — which is +exactly the 6-layer × 2-component input here. The deprecated value is retained so existing +documents stay valid. + +**Monotonicity must be structural, never regularization-encouraged.** A penalty term makes +M5 probabilistic, which voids the gate-soundness argument; a library offering only soft +constraints is rejected on those grounds, and the schema refuses `monotonicityEnforcement: +regularized`. + +## Conformance + +``` +make validate-fingerprint-stack +``` + +Two tools, and the second is the point: + +- `tools/validate_fingerprint_stack.py` — 81 checks over the examples, the OntoDT graph + fixture, and 9 schema negative vectors. +- `tools/test_fingerprint_stack_teeth.py` — mutates each conformant example one invariant at + a time and **requires the validator to reject it, for the stated reason**. 25 gates proven + to bite. A gate that stays green under its own mutation is reported as a failure of the + *checker*. + +Plus `check_m5_binds()`, which runs the monotonicity property test against a deliberately +non-monotone aggregator and fails if the test does not catch it. + +This discipline is not decorative. Its nearest precedent in this repo shipped the exact +defect it guards against: #264's monotone classifier had a fixture holding the monotone +feature constant, so the constraint bound nothing and the test passed vacuously — found and +fixed in #265. A green check is not evidence until you have watched it go red. + +## Not in this contract + +Named so the gaps are visible rather than assumed closed: + +- **WO-15 justification-scoped `INADMISSIBLE`** — required before production. Without it, + one bad axiom makes everything derivable and every query returns `INADMISSIBLE`. Note the + architectural point: this is **classical inference over justified fragments** with the + bilattice used for *bookkeeping the outcomes* — it is **not** paraconsistent inference, and + that is the most likely misreading of this design. +- **Cold-start phases 1–4** as executable pipeline (phase 0 artifacts are contracted here). +- **KKO cross-domain transfer** — a hypothesis with a test protocol, not a capability. It + must not appear in customer-facing material until the held-out-domain label-efficiency + measurement returns a number, and the ablation arm (layers concatenated, no upper-ontology + mediation) must run: if the ablation matches, KKO is not doing the work. +- **DR-2** (marginal vs. Mondrian coverage, per-class calibration floor), **DR-7** (cascade + with early exit — must be calibrated as a unit, never per-stage), **DR-8** (constraint M3), + **DR-9** (independence-certificate schema, required before any layer pair is granted + probabilistic-sum pooling). + +## A note on the source + +Slides 28–30 of the 2018 deck embed the entire OntoDT/OntoDQ argument as three TIFFs — 95 MB +of a 120 MB file. It is un-indexable, un-greppable, un-diffable, and invisible to text +extraction. The knowledge-management architecture was stored in a form that defeats +knowledge management. Every diagram in this stack is committed as source (Mermaid, Graphviz +or TTL) with rendering as a build artifact. diff --git a/tools/fingerprint_aggregate.py b/tools/fingerprint_aggregate.py new file mode 100644 index 0000000..610d8ea --- /dev/null +++ b/tools/fingerprint_aggregate.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +"""Reference semantics for the six-layer fingerprint stack (SP-FPRINT-STACK-001). + +Everything here is RECOMPUTABLE: pooling, guard application, quantization, and effective +independence. The stored fields in ColumnFingerprint / ClassificationStance are caches, and +the validator recomputes them so an asserted stance cannot outrun its evidence. + +Order matters and is the safety property. Pool -> guard -> quantize. Guards can only lower +stance in the knowledge order, and quantization to the four-valued stance happens LAST, so +no composition of guards can raise necessity or turn ZERO into POS. + +This module also carries the M5 property test AND a deliberately non-monotone aggregator +used to prove that test bites. A monotonicity test that only ever sees a monotone model is +vacuous — that is not hypothetical here, it is the exact defect found in the trained +DataClass classifiers (#265), where the fixture held the monotone feature constant and the +constraint therefore had no teeth. +""" +from __future__ import annotations + +from itertools import product + +# Knowledge order <=_k on FOUR, as (support, refute) bits. +# ZERO=(0,0) <=_k POS=(1,0), NEG=(0,1) <=_k INADMISSIBLE=(1,1); POS and NEG incomparable. +BITS: dict[str, tuple[int, int]] = { + "ZERO": (0, 0), + "POS": (1, 0), + "NEG": (0, 1), + "INADMISSIBLE": (1, 1), +} +FROM_BITS = {v: k for k, v in BITS.items()} + +TOL = 1e-9 + + +def leq_k(a: str, b: str) -> bool: + """a <=_k b — componentwise on the support/refute bits.""" + (a1, a2), (b1, b2) = BITS[a], BITS[b] + return a1 <= b1 and a2 <= b2 + + +def quantize(alpha: float, beta: float, tau_pos: float, tau_neg: float) -> str: + """Q_{tau+,tau-}(alpha, beta) = (1[alpha >= tau+], 1[beta >= tau-]). + + Q commutes with negation iff tau_pos == tau_neg. Asymmetric thresholds break + negation-equivariance: deny-rules stop coming free by negating allow-rules and a + separate deny path has to be maintained. The validator therefore requires an + attestation for any asymmetry. + """ + return FROM_BITS[(int(alpha >= tau_pos - TOL), int(beta >= tau_neg - TOL))] + + +def pool(pairs: list[tuple[float, float]], operator: str = "frechet-max") -> tuple[float, float]: + """Pool evidence across ADMISSIBLE layers. + + Default is the Frechet bound (max alpha, max beta), which assumes nothing about + dependence. Its known cost is idempotence — ten corroborating layers buy nothing over + one — and that is deliberate: corroboration pays only where the right to claim it has + been earned via a certified independence model, which licenses probabilistic sum for + the specific layer pair it certifies. + """ + if not pairs: + return (0.0, 0.0) + if operator == "frechet-max": + return (max(a for a, _ in pairs), max(b for _, b in pairs)) + if operator == "probabilistic-sum": + a = b = 0.0 + for pa, pb in pairs: + a = a + pa - a * pa + b = b + pb - b * pb + return (min(a, 1.0), min(b, 1.0)) + raise ValueError(f"unknown pooling operator: {operator!r}") + + +def admissible_pairs(layer_evidence: list[dict]) -> list[tuple[float, float]]: + """Only admissible layers contribute. An inadmissible layer contributes ZERO — the + tensor annihilator — never a guessed or imputed value.""" + out = [] + for le in layer_evidence: + if (le.get("admissibility") or {}).get("admissible"): + ev = le.get("evidence") or {} + out.append((float(ev.get("alpha", 0.0)), float(ev.get("beta", 0.0)))) + return out + + +def n_eff(spectrum: list[float]) -> float: + """Effective number of independent layers = participation ratio of the layer + covariance spectrum, (sum lambda)^2 / sum lambda^2. + + NOT the Herfindahl index. H measures concentration of MAGNITUDE, not dependence: two + perfectly correlated layers contributing equally give H = 0.5, which looks healthy, + while supplying one layer's worth of information. H catches 'one layer dominates'; it + does not catch 'my layers are secretly the same layer', which is the failure that + matters for quorum. + """ + s1 = sum(spectrum) + s2 = sum(x * x for x in spectrum) + if s2 <= 0: + return 0.0 + return (s1 * s1) / s2 + + +def monotone_aggregate(pairs: list[tuple[float, float]]) -> tuple[float, float]: + """Reference aggregator satisfying M1/M2/M4/M5 STRUCTURALLY. + + M1 alpha non-decreasing in each input alpha; M2 beta non-decreasing in each input beta; + M4 all-ZERO in => ZERO out; M5 knocking a layer out never RAISES the result. + + Structural, not regularization-encouraged: a penalty term makes M5 probabilistic, which + voids the gate-soundness argument. A library offering only soft monotonicity is rejected + on exactly those grounds. + """ + return pool(pairs, "frechet-max") + + +def NON_MONOTONE_AGGREGATE(pairs: list[tuple[float, float]]) -> tuple[float, float]: + """A deliberately BROKEN aggregator, used only to prove the M5 test has teeth. + + It subtracts a corroboration penalty, so adding a supporting layer can LOWER alpha and + removing one can RAISE it — precisely the M5 violation. If check_m5_binds() ever passes + this function, the property test is vacuous and the validator fails. + """ + if not pairs: + return (0.0, 0.0) + a = max(p[0] for p in pairs) - 0.15 * (len(pairs) - 1) + b = max(p[1] for p in pairs) + return (max(a, 0.0), min(b, 1.0)) + + +def m5_violations(aggregate, grid: int = 3) -> list[str]: + """Property-test M5 over a grid of evidence vectors: for every layer set and every + single-layer knockout, the knocked-out result must be <=_k the full result. + + Returns the violations found. An aggregator is M5-sound iff this is empty. + """ + steps = [i / (grid - 1) for i in range(grid)] + violations: list[str] = [] + for n in (2, 3): + for combo in product(product(steps, steps), repeat=n): + pairs = [tuple(c) for c in combo] + full = aggregate(pairs) + for i in range(n): + knocked = [p for j, p in enumerate(pairs) if j != i] + out = aggregate(knocked) + # Lowering in the knowledge order is componentwise <= on (alpha, beta). + if out[0] > full[0] + TOL or out[1] > full[1] + TOL: + violations.append(f"knockout of layer {i} from {pairs} RAISED {full} -> {out}") + return violations + + +def check_m5_binds() -> tuple[bool, str]: + """Prove the M5 property test BITES before trusting a pass from it. + + The reference monotone aggregator must produce no violations, AND the deliberately + broken one must produce some. A test that cannot fail is not evidence. + """ + good = m5_violations(monotone_aggregate) + bad = m5_violations(NON_MONOTONE_AGGREGATE) + if good: + return False, f"reference monotone aggregator VIOLATES M5: {good[0]}" + if not bad: + return False, "M5 property test is VACUOUS — it did not catch the known non-monotone aggregator" + return True, f"M5 test binds ({len(bad)} violations caught on the broken aggregator, 0 on the reference)" diff --git a/tools/test_fingerprint_stack_teeth.py b/tools/test_fingerprint_stack_teeth.py new file mode 100644 index 0000000..68fb299 --- /dev/null +++ b/tools/test_fingerprint_stack_teeth.py @@ -0,0 +1,168 @@ +#!/usr/bin/env python3 +"""Prove every semantic gate in validate_fingerprint_stack.py BITES. + +A validator that passes tells you nothing until you have watched it fail. This mutates the +conformant examples one invariant at a time and requires the validator to reject each +mutation with the expected message. A gate that stays green under its own mutation is a gate +with no teeth, and is reported as a failure of the CHECKER, not of the data. + +This exists because the closest precedent in this repo shipped exactly that defect: the +trained DataClass classifiers (#264) declared a monotonicity constraint whose fixture held +the monotone feature constant, so the constraint bound nothing and the test passed vacuously +(#265). Schema negative vectors live in fixtures/fingerprint-stack/conformance.json; this +file covers the SEMANTIC checks, which no schema can express. +""" +from __future__ import annotations + +import copy +import json +import subprocess +import sys +import tempfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +EXAMPLES = ROOT / "examples" +FIXTURES = ROOT / "fixtures" / "fingerprint-stack" + +FP_A = "column_fingerprint.customer_id.json" +FP_B = "column_fingerprint.repurposed_status.json" +DRIFT = "column_drift.repurposed_status.json" +REPORT = "estate_admissibility_report.phase0.json" +GRAPH = "ontodt_graph.json" +POLICY = "classification_enforcement_policy.json" + + +def layer(doc: dict, name: str) -> dict: + return next(le for le in doc["layerEvidence"] if le["layer"].startswith(name)) + + +# Each mutation: (label, target file, mutate fn, substring the rejection must contain). +MUTATIONS: list[tuple[str, str, object, str]] = [ + ("quantizer cache drifts from evidence", FP_A, + lambda d: d["stance"].update(value="NEG"), "cannot outrun its evidence"), + ("asymmetric thresholds unattested", FP_A, + lambda d: d["stance"]["thresholds"].update(tauNeg=0.5), "negation-equivariance"), + ("glut without a recorded origin", FP_B, + lambda d: d["stance"].update(inadmissibleOrigin=None), "without an origin"), + ("pooled evidence overstated", FP_A, + lambda d: d["pooling"]["pooled"].update(alpha=0.95), "!= recomputed"), + ("inadmissible layer still contributes", FP_B, + lambda d: layer(d, "L5")["evidence"].update(alpha=0.9), "must contribute ZERO"), + ("guard raises evidence instead of lowering", FP_A, + lambda d: d["stance"]["evidence"].update(alpha=0.99), "may only lower stance"), + ("probabilistic sum without a certificate", FP_A, + lambda d: d["pooling"].update(operator="probabilistic-sum"), "independence certificate"), + ("POS below the effective-independence floor", FP_A, + lambda d: d["quorum"].update(nEffFloor=6.0), "below floor"), + ("n_eff asserted, not derived", FP_A, + lambda d: d["quorum"].update(nEff=5.9), "participation ratio"), + ("unearned coverage across composition", FP_B, + lambda d: d["stance"]["composition"].update(effectiveEpsilon=0.1), "coverage does not compose"), + ("inferred signal speaking as loudly as declared", FP_B, + lambda d: layer(d, "L6")["evidence"].update(alpha=0.9), "above the cap"), + ("L4 conflict recorded as a weak match", FP_A, + lambda d: layer(d, "L4")["witness"].update(conflictSet=["mean"]), "is not a glut"), + ("profiling self-promoted to a hard axiom", FP_A, + lambda d: layer(d, "L2")["witness"].update(promotedToHardAxiom=True), "without steward attestation"), + ("drift flag set independently of the measurement", DRIFT, + lambda d: d.update(distance=0.01), "recomputes to"), + ("silent repurpose misread as a redesign", DRIFT, + lambda d: d.update(schemaChanged=True), "measurement gives"), + ("glut never reaches a human", DRIFT, + lambda d: d.update(stewardQueueRef=None), "behaviour, not a claim"), + ("L5 survives its own drift flag", FP_B, + lambda d: layer(d, "L5")["admissibility"].update(admissible=True, reason=None), + "must be inadmissible"), + ("estate claims POS below the floor", REPORT, + lambda d: d["quorum"].update(posAvailable=True), "honest outputs are ZERO and NEG only"), + ("glossary admissible before a glossary exists", REPORT, + lambda d: d["layers"][2].update(admissible=True, reason=None), "catalog-derived bootstrap"), + ("parthood and subtyping share an arrow", GRAPH, + lambda d: d["subtypeOf"].append(["email-address", "client-record"]), "X1 violated"), + ("open-world count reported as a number", GRAPH, + lambda d: d["aggregateQueries"][0].update(answerInterval=[41, 41]), "is an interval, not a number"), + ("contradictions folded into the count", GRAPH, + lambda d: d["aggregateQueries"][0].update(inadmissibleReported=0), "reported separately"), + ("attribute enumeration walks subtyping", GRAPH, + lambda d: d["partEnumerations"][0].update(relation="subtypeOf"), "enumeration of PARTS"), + ("unknown data served without anyone deciding", POLICY, + lambda d: d["resourceClasses"][0].update(attestationRef=None), "or an accident"), + ("fail-open bought back by attestation on restricted data", POLICY, + lambda d: d["resourceClasses"][2].update( + zeroBehavior="fail-open", attestationRef="urn:srcos:attestation:override"), + "no attestation buys that back"), +] + + +def run_validator(workdir: Path) -> tuple[int, str]: + proc = subprocess.run( + [sys.executable, str(workdir / "tools" / "validate_fingerprint_stack.py")], + capture_output=True, text=True, cwd=workdir, + ) + return proc.returncode, proc.stdout + proc.stderr + + +def main() -> int: + originals = { + n: json.loads((EXAMPLES / n).read_text()) for n in (FP_A, FP_B, DRIFT, REPORT, POLICY) + } | {GRAPH: json.loads((FIXTURES / GRAPH).read_text())} + + failures: list[str] = [] + results: dict[str, bool] = {} + + with tempfile.TemporaryDirectory() as tmp: + work = Path(tmp) / "repo" + # Mirror only what the validator reads; symlink the rest so mutation is isolated. + for sub in ("schemas", "tools", "examples", "fixtures"): + (work / sub).mkdir(parents=True, exist_ok=True) + for src, dst in ( + (ROOT / "schemas", work / "schemas"), + (ROOT / "tools", work / "tools"), + (ROOT / "examples", work / "examples"), + ): + for f in src.glob("*.json"): + (dst / f.name).write_bytes(f.read_bytes()) + for f in src.glob("*.py"): + (dst / f.name).write_bytes(f.read_bytes()) + (work / "fixtures" / "fingerprint-stack").mkdir(parents=True, exist_ok=True) + for f in FIXTURES.glob("*.json"): + (work / "fixtures" / "fingerprint-stack" / f.name).write_bytes(f.read_bytes()) + + # Sanity: the unmutated mirror must PASS, or every mutation "passes" for the wrong reason. + rc, out = run_validator(work) + if rc != 0: + print("FAIL: unmutated mirror does not validate — mutation results would be meaningless", + file=sys.stderr) + print(out, file=sys.stderr) + return 1 + results["baseline-mirror-passes"] = True + + for label, target, mutate, expect in MUTATIONS: + doc = copy.deepcopy(originals[target]) + mutate(doc) + path = (work / "fixtures" / "fingerprint-stack" / target) if target == GRAPH \ + else (work / "examples" / target) + path.write_text(json.dumps(doc, indent=2)) + rc, out = run_validator(work) + path.write_text(json.dumps(originals[target], indent=2)) + + if rc == 0: + failures.append(f"NO TEETH: {label!r} was accepted — this gate does not bite") + results[label] = False + elif expect not in out: + failures.append(f"WRONG REASON: {label!r} rejected, but not for {expect!r}") + results[label] = False + else: + results[label] = True + + for m in failures: + print(f"FAIL: {m}", file=sys.stderr) + ok = not failures + print(json.dumps({"ok": ok, "gatesProvenToBite": len(MUTATIONS), "results": results}, + indent=2, sort_keys=True)) + return 0 if ok else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/validate_fingerprint_stack.py b/tools/validate_fingerprint_stack.py new file mode 100644 index 0000000..9ab6c02 --- /dev/null +++ b/tools/validate_fingerprint_stack.py @@ -0,0 +1,531 @@ +#!/usr/bin/env python3 +"""Validate the six-layer fingerprint stack (SP-FPRINT-STACK-001). + +Recompute-don't-trust throughout. Every stored verdict in this family is a CACHE of +something derivable — the stance from its evidence, the pooled evidence from the admissible +layers, n_eff from the covariance spectrum, the drift flag from the measured distance — and +each is recomputed here, so an asserted classification cannot outrun what produced it. + +Checks (fail-closed): + 1. schema conformance for every example; + 2. quantizer soundness — stance.value == Q(stance.evidence; thresholds); + 3. threshold symmetry — tau+ != tau- requires a platform attestation (DR-6); + 4. INADMISSIBLE carries an origin (the lattice element is identical across causes, the + remediations are not); + 5. pooling soundness — pooled evidence is recomputed over the ADMISSIBLE layers only; + 6. annihilation — an inadmissible layer contributes ZERO, never a guessed value; + 7. gate soundness — post-guard evidence never exceeds pooled evidence in either + component, i.e. guards only lower stance in the knowledge order; + 8. licensed strengthening — probabilistic-sum pooling requires an independence certificate; + 9. quorum — n_eff recomputed from the spectrum, and POS refused below the floor; + 10. path-length budget — effectiveEpsilon >= depth * baseEpsilon (coverage does not compose); + 11. reading tag — present on every stance, and never mixed within one aggregation; + 12. drift — `drifted` and `verdict` recomputed; a silent repurpose must route to a steward; + 13. L5-D1/L5-D2 — under silent repurpose L5 is inadmissible, contributes ZERO, and the + resulting stance is INADMISSIBLE with a drift origin; + 14. L4 glut — a declared-vs-observed operation conflict emits BOTH components positive; + 15. Q3 barrier — a profiling-inferred constraint promoted to a hard axiom needs attestation; + 16. X1 — parthood and subtyping are disjoint, closure traverses subtyping only, and an + aggregate count is an interval with INADMISSIBLE reported separately; + 17. M5 — the monotonicity property test is proven to BITE before its pass is trusted; + 18. estate report coherence — all six layers present, every inadmissible one reasoned; + 19. negative vectors fail, each on its named keyword. +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import jsonschema +from referencing import Registry, Resource + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from fingerprint_aggregate import ( # noqa: E402 + TOL, + admissible_pairs, + check_m5_binds, + n_eff, + pool, + quantize, +) + +ROOT = Path(__file__).resolve().parents[1] +SCHEMAS = ROOT / "schemas" +EXAMPLES = ROOT / "examples" +FIXTURES = ROOT / "fixtures" / "fingerprint-stack" + +# An inferred signal (a discovered inclusion dependency where DDL declares no constraint, a +# declared-only L4 with no query log) is admissible but capped: it degrades rather than failing, +# and the cap is what stops "degrade, don't fail" from quietly becoming "guess, don't say". +INFERRED_ALPHA_CAP = 0.5 + +FAILURES: list[str] = [] +CHECKS: dict[str, bool] = {} + + +def load(p: Path) -> dict: + return json.loads(p.read_text(encoding="utf-8")) + + +def registry() -> Registry: + resources = [] + for f in SCHEMAS.glob("*.json"): + schema = load(f) + res = Resource.from_contents(schema) + resources.append((schema.get("$id", f.name), res)) + resources.append((f.name, res)) + return Registry().with_resources(resources) + + +REG = registry() + + +def validator_for(schema: dict) -> jsonschema.Draft202012Validator: + return jsonschema.Draft202012Validator(schema, registry=REG) + + +def check_conformance(named: dict[str, tuple[dict, dict]]) -> None: + """Validate each document against its INTENDED schema explicitly — never pick the schema + from the instance's own `type`, or a mistyped document validates against the wrong one.""" + for name, (doc, schema) in named.items(): + errs = sorted(validator_for(schema).iter_errors(doc), key=str) + if errs: + for e in errs: + FAILURES.append(f"{name}: {e.message}") + else: + CHECKS[f"schema:{name}"] = True + + +def check_stance(name: str, stance: dict) -> None: + ev, th = stance["evidence"], stance["thresholds"] + recomputed = quantize(ev["alpha"], ev["beta"], th["tauPos"], th["tauNeg"]) + if recomputed != stance["value"]: + FAILURES.append( + f"{name}: stance claims {stance['value']} but Q({ev['alpha']}, {ev['beta']}; " + f"{th['tauPos']}, {th['tauNeg']}) = {recomputed} — an asserted stance cannot outrun its evidence" + ) + else: + CHECKS[f"quantize:{name}"] = True + + if abs(th["tauPos"] - th["tauNeg"]) > TOL and not th.get("attestationRef"): + FAILURES.append( + f"{name}: asymmetric thresholds (tau+={th['tauPos']}, tau-={th['tauNeg']}) break " + f"negation-equivariance and require a platform attestation (DR-6)" + ) + else: + CHECKS[f"tau-symmetry:{name}"] = True + + if stance["value"] == "INADMISSIBLE" and not stance.get("inadmissibleOrigin"): + FAILURES.append( + f"{name}: INADMISSIBLE without an origin — drift, layer conflict, inconsistent KB and " + f"stale calibration are the same lattice element with different remediations" + ) + else: + CHECKS[f"glut-origin:{name}"] = True + + comp = stance["composition"] + floor = comp["depth"] * comp["baseEpsilon"] + if comp["effectiveEpsilon"] < floor - TOL: + FAILURES.append( + f"{name}: claims effective epsilon {comp['effectiveEpsilon']} at depth {comp['depth']} " + f"but the union bound gives at best {floor} — coverage does not compose" + ) + else: + CHECKS[f"path-budget:{name}"] = True + + for g in stance.get("guards") or []: + for path in g["inputs"]: + if not path.startswith(("evidence.", "witness.")): + FAILURES.append( + f"{name}: guard {g['name']!r} reads {path!r} — guards must be measurable with " + f"respect to evidence and witness only; reading the outcome breaks exchangeability" + ) + else: + CHECKS[f"guard-measurable:{name}:{g['name']}"] = True + + +def check_fingerprint(name: str, fp: dict) -> None: + layers = fp["layerEvidence"] + + # 6. Annihilation — an inadmissible layer contributes ZERO, not a guess. + for le in layers: + adm = le["admissibility"] + ev = le["evidence"] + if not adm["admissible"] and (ev["alpha"] > TOL or ev["beta"] > TOL): + FAILURES.append( + f"{name}/{le['layer']}: inadmissible layer contributes ({ev['alpha']}, {ev['beta']}) " + f"— an inadmissible layer must contribute ZERO, the annihilator" + ) + else: + CHECKS[f"annihilate:{name}:{le['layer']}"] = True + if not adm["admissible"] and not adm.get("reason"): + FAILURES.append(f"{name}/{le['layer']}: inadmissible without a named reason") + + # Degrade-don't-fail has a price: an inferred signal cannot speak as loudly as a declared one. + if le.get("confidenceSource") == "inferred" and ev["alpha"] > INFERRED_ALPHA_CAP + TOL: + FAILURES.append( + f"{name}/{le['layer']}: confidenceSource 'inferred' contributes alpha {ev['alpha']} " + f"above the cap {INFERRED_ALPHA_CAP} — a discovered inclusion dependency is weaker " + f"evidence than a declared constraint and must be capped as such" + ) + else: + CHECKS[f"inferred-cap:{name}:{le['layer']}"] = True + + # 5/8. Pooling soundness over admissible layers only. + pooling = fp["pooling"] + op = pooling["operator"] + if op == "probabilistic-sum" and not pooling.get("independenceCertificateRef"): + FAILURES.append( + f"{name}: probabilistic-sum pooling without an independence certificate — corroboration " + f"pays only where the right to claim it has been earned" + ) + else: + CHECKS[f"licensed-pooling:{name}"] = True + + exp_a, exp_b = pool(admissible_pairs(layers), op) + got = pooling["pooled"] + if abs(exp_a - got["alpha"]) > 1e-6 or abs(exp_b - got["beta"]) > 1e-6: + FAILURES.append( + f"{name}: pooled evidence {(got['alpha'], got['beta'])} != recomputed " + f"{(round(exp_a, 6), round(exp_b, 6))} over the admissible layers" + ) + else: + CHECKS[f"pool:{name}"] = True + + # 7. Gate soundness — guards only lower, in both components. + stance = fp["stance"] + sev = stance["evidence"] + if sev["alpha"] > got["alpha"] + TOL or sev["beta"] > got["beta"] + TOL: + FAILURES.append( + f"{name}: post-guard evidence ({sev['alpha']}, {sev['beta']}) EXCEEDS pooled " + f"({got['alpha']}, {got['beta']}) — guards may only lower stance in the knowledge order" + ) + else: + CHECKS[f"gate-sound:{name}"] = True + + # 9. Quorum — recompute n_eff, and refuse POS below the floor. + q = fp["quorum"] + spectrum = q.get("covarianceSpectrum") + if spectrum: + recomputed = n_eff(spectrum) + if abs(recomputed - q["nEff"]) > 1e-6: + FAILURES.append( + f"{name}: claims n_eff {q['nEff']} but the participation ratio of the given spectrum " + f"is {round(recomputed, 6)}" + ) + else: + CHECKS[f"neff-recompute:{name}"] = True + if stance["value"] == "POS" and q["nEff"] < q["nEffFloor"] - TOL: + FAILURES.append( + f"{name}: POS at n_eff {q['nEff']} below floor {q['nEffFloor']} — below the floor the " + f"stance is forced to ZERO; the Herfindahl index is a cheap precheck, never the quorum statistic" + ) + else: + CHECKS[f"quorum:{name}"] = True + + # 11. Reading tag never mixed within one aggregation. + CHECKS[f"reading-tagged:{name}"] = bool(stance.get("reading")) + + # 14. L4 glut — a declared-vs-observed conflict is a glut, not a weak match. + for le in layers: + w = le["witness"] + if w.get("kind") == "L4" and w.get("conflictSet"): + ev = le["evidence"] + if not (ev["alpha"] > TOL and ev["beta"] > TOL): + FAILURES.append( + f"{name}/L4: conflict set {w['conflictSet']} recorded but evidence " + f"({ev['alpha']}, {ev['beta']}) is not a glut — a declared-vs-observed conflict " + f"means either the label is wrong or downstream code has a bug, and both need a human" + ) + else: + CHECKS[f"l4-glut:{name}"] = True + + # 15. Q3 barrier — profiling is a defeasible prior, never self-promoting. + for le in layers: + w = le["witness"] + if w.get("kind") == "L2" and w.get("promotedToHardAxiom"): + if not w.get("stewardAttestationRef"): + FAILURES.append( + f"{name}/L2: profiling-inferred constraint promoted to a hard axiom without steward " + f"attestation — this is how a bad estate poisons its own ontology" + ) + else: + CHECKS[f"q3-barrier:{name}"] = True + + +def check_drift(name: str, obs: dict, fingerprints: dict[str, dict]) -> None: + recomputed = obs["distance"] > obs["threshold"] + if recomputed != obs["drifted"]: + FAILURES.append( + f"{name}: drifted={obs['drifted']} but distance {obs['distance']} vs threshold " + f"{obs['threshold']} recomputes to {recomputed}" + ) + else: + CHECKS[f"drift-recompute:{name}"] = True + + expected = ( + "silent-repurpose" if recomputed and not obs["schemaChanged"] + else "drift-with-redesign" if recomputed + else "no-drift" + ) + if obs.get("verdict") != expected: + FAILURES.append(f"{name}: verdict {obs.get('verdict')!r} but the measurement gives {expected!r}") + else: + CHECKS[f"drift-verdict:{name}"] = True + + if expected == "silent-repurpose" and not obs.get("stewardQueueRef"): + FAILURES.append( + f"{name}: silent repurpose without a steward queue — routing the glut to a human is a " + f"behaviour, not a claim" + ) + else: + CHECKS[f"drift-routes:{name}"] = True + + # L5-D1 / L5-D2 — the rules that make L5's dangerous confidence survivable. + if expected != "silent-repurpose": + return + for fp_name, fp in fingerprints.items(): + for le in fp["layerEvidence"]: + w = le["witness"] + if w.get("kind") != "L5" or w.get("driftRef") != obs["id"]: + continue + adm = le["admissibility"] + ev = le["evidence"] + if adm["admissible"] or adm.get("reason") != "active-drift-flag": + FAILURES.append( + f"{fp_name}/L5: under silent repurpose L5 must be inadmissible with reason " + f"'active-drift-flag' (L5-D2) — L5 asserts stale semantics with maximum confidence " + f"exactly when it is wrong" + ) + elif ev["alpha"] > TOL or ev["beta"] > TOL: + FAILURES.append(f"{fp_name}/L5: drift guard fired but L5 still contributes {ev}") + elif fp["stance"]["value"] != "INADMISSIBLE": + FAILURES.append( + f"{fp_name}: silent repurpose puts L5 and L2 in conflict and must emit " + f"INADMISSIBLE (L5-D1), not {fp['stance']['value']}" + ) + elif fp["stance"].get("inadmissibleOrigin") != "profile-drift": + FAILURES.append(f"{fp_name}: drift glut must record origin 'profile-drift'") + else: + CHECKS[f"l5-d1-d2:{fp_name}"] = True + + +def check_ontodt(graph: dict) -> None: + """X1 and its consequences: the precondition that makes the source design's own two + questions simultaneously answerable. + + Q1 'how many tables contain the personal data field Email Address' is a count over the + SUBTYPE closure; Q2 'what attributes can I expect under Client Address' is an enumeration + of PARTS. If is_a and part_of share an arrow type, one of those answers is necessarily + wrong. X1 is not a refinement — it is what makes the questions answerable at all. + """ + sub = {tuple(e) for e in graph["subtypeOf"]} + part = {tuple(e) for e in graph["partOf"]} + overlap = {(x, y) for (x, y) in sub & part if x != y} + if overlap: + FAILURES.append(f"X1 violated — pairs are both parthood and subtyping: {sorted(overlap)}") + else: + CHECKS["x1-disjoint"] = True + + for qi, q in enumerate(graph["aggregateQueries"]): + if q["relation"] != "subtypeOf": + FAILURES.append( + f"aggregate query {qi}: count closure traverses {q['relation']!r} — aggregation " + f"traverses subtyping only, never parthood" + ) + continue + members = q["members"] + lo = members["POS"] + hi = members["POS"] + members["ZERO"] + if q["answerInterval"] != [lo, hi]: + FAILURES.append( + f"aggregate query {qi}: interval {q['answerInterval']} != [{lo}, {hi}] — under the " + f"open-world assumption a count is an interval, not a number" + ) + elif q.get("inadmissibleReported") != members["INADMISSIBLE"]: + FAILURES.append( + f"aggregate query {qi}: INADMISSIBLE members must be reported separately and never " + f"folded into the interval" + ) + else: + CHECKS[f"cardinality-interval:{qi}"] = True + + for qi, q in enumerate(graph.get("partEnumerations", [])): + if q["relation"] != "partOf": + FAILURES.append( + f"part enumeration {qi}: traverses {q['relation']!r} — 'what attributes can I expect " + f"under this term' is an enumeration of PARTS, and parthood does not roll up" + ) + else: + CHECKS[f"part-enumeration:{qi}"] = True + + +def check_estate_report(name: str, rep: dict) -> None: + seen = [layer["layer"] for layer in rep["layers"]] + if len(set(seen)) != 6: + FAILURES.append(f"{name}: all six layers must appear — an omitted layer is indistinguishable from an admissible one") + else: + CHECKS[f"estate-six-layers:{name}"] = True + + for layer in rep["layers"]: + if not layer["admissible"] and not layer.get("reason"): + FAILURES.append(f"{name}/{layer['layer']}: inadmissible without a named reason") + + q = rep["quorum"] + if q.get("covarianceSpectrum"): + recomputed = n_eff(q["covarianceSpectrum"]) + if abs(recomputed - q["nEff"]) > 1e-6: + FAILURES.append(f"{name}: claims n_eff {q['nEff']}, spectrum gives {round(recomputed, 6)}") + else: + CHECKS[f"estate-neff:{name}"] = True + + if q["posAvailable"] and q["nEff"] < q["nEffFloor"] - TOL: + FAILURES.append( + f"{name}: posAvailable=true at n_eff {q['nEff']} below floor {q['nEffFloor']} — below the " + f"floor the honest outputs are ZERO and NEG only" + ) + else: + CHECKS[f"estate-pos-gate:{name}"] = True + + # Phase 0 is derivable from the catalog and the data alone: no glossary, no query logs. + if rep["coldStartPhase"] == 0: + by_layer = {layer["layer"]: layer for layer in rep["layers"]} + for gated in ("L3-business-glossary", "L4-operational-semantics"): + if by_layer[gated]["admissible"]: + FAILURES.append( + f"{name}: phase 0 is the catalog-derived bootstrap — {gated} cannot be admissible " + f"before a glossary exists" + ) + else: + CHECKS[f"cold-start-phase0:{name}:{gated}"] = True + + +def check_enforcement_policy(name: str, pol: dict) -> None: + """DR-4 at the enforcement point. ZERO is a knob, but a constrained one: opening it is an + attested decision, and it is not available at all where the exposure cost is unbounded.""" + for rc in pol["resourceClasses"]: + if rc["zeroBehavior"] != "fail-open": + CHECKS[f"dr4-attested:{name}:{rc['resourceClass']}"] = True + continue + if not rc.get("attestationRef"): + FAILURES.append( + f"{name}/{rc['resourceClass']}: fail-open on ZERO without an attestation — serving data " + f"whose classification is unknown is a decision somebody makes on the record, or an accident" + ) + elif rc["sensitivity"] in ("confidential", "restricted"): + FAILURES.append( + f"{name}/{rc['resourceClass']}: fail-open on ZERO at sensitivity {rc['sensitivity']!r} — " + f"an unclassifiable column in a {rc['sensitivity']} resource is precisely the one not to serve, " + f"and no attestation buys that back" + ) + else: + CHECKS[f"dr4-attested:{name}:{rc['resourceClass']}"] = True + + +def check_dataclass_binding(name: str, dc: dict, reports: dict, policies: dict) -> None: + """A DataClass that declares the six-layer stack must point at artifacts that EXIST. + + The DataClass contract already had this failure once: it referenced a ModelManifest and + RunRecords by URN while the model was specified but never trained, so the references + pointed at nothing (#264 closed that). The same discipline applies here — a declared + binding whose target is absent is a claim, not a wiring. + """ + stack = (dc.get("classifier") or {}).get("fingerprintStack") + if not stack: + CHECKS[f"dataclass-stack:{name}:none"] = True + return + known_reports = {r["id"] for r in reports.values()} + known_policies = {p["id"] for p in policies.values()} + ref = stack.get("estateAdmissibilityRef") + if ref and ref not in known_reports: + FAILURES.append(f"{name}: estateAdmissibilityRef {ref} resolves to nothing") + elif stack["enforcementPolicyRef"] not in known_policies: + FAILURES.append( + f"{name}: enforcementPolicyRef {stack['enforcementPolicyRef']} resolves to nothing — " + f"a class whose ZERO behaviour is undefined cannot be enforced" + ) + else: + CHECKS[f"dataclass-stack:{name}"] = True + + +def check_negatives(schemas: dict[str, dict]) -> None: + fx = load(FIXTURES / "conformance.json") + for i, case in enumerate(fx["cases"]): + schema = schemas[case["schema"]] + expected = case.get("failValidator") + try: + validator_for(schema).validate(case["document"]) + except jsonschema.ValidationError as exc: + if expected is not None and exc.validator != expected: + FAILURES.append(f"negative {i}: failed on {exc.validator!r}, not {expected!r}: {case['reason']}") + else: + CHECKS[f"negative:{i}:{exc.validator}"] = True + continue + FAILURES.append(f"negative {i} unexpectedly PASSED: {case['reason']}") + + +def main() -> int: + schemas = { + n: load(SCHEMAS / n) + for n in ( + "ClassificationStance.json", + "FingerprintLayerEvidence.json", + "ColumnFingerprint.json", + "ColumnDriftObservation.json", + "EstateAdmissibilityReport.json", + "ClassificationEnforcementPolicy.json", + ) + } + + fingerprints = { + "column_fingerprint.customer_id.json": load(EXAMPLES / "column_fingerprint.customer_id.json"), + "column_fingerprint.repurposed_status.json": load(EXAMPLES / "column_fingerprint.repurposed_status.json"), + } + drifts = {"column_drift.repurposed_status.json": load(EXAMPLES / "column_drift.repurposed_status.json")} + reports = { + "estate_admissibility_report.phase0.json": load(EXAMPLES / "estate_admissibility_report.phase0.json"), + } + + policies = {"classification_enforcement_policy.json": load(EXAMPLES / "classification_enforcement_policy.json")} + + named = {n: (d, schemas["ColumnFingerprint.json"]) for n, d in fingerprints.items()} + named |= {n: (d, schemas["ColumnDriftObservation.json"]) for n, d in drifts.items()} + named |= {n: (d, schemas["EstateAdmissibilityReport.json"]) for n, d in reports.items()} + named |= {n: (d, schemas["ClassificationEnforcementPolicy.json"]) for n, d in policies.items()} + check_conformance(named) + + for name, fp in fingerprints.items(): + check_stance(name, fp["stance"]) + check_fingerprint(name, fp) + for name, obs in drifts.items(): + check_drift(name, obs, fingerprints) + for name, rep in reports.items(): + check_estate_report(name, rep) + + for name, pol in policies.items(): + check_enforcement_policy(name, pol) + + data_classes = {"data_class.currency.json": load(EXAMPLES / "data_class.currency.json")} + for name, dc in data_classes.items(): + check_dataclass_binding(name, dc, reports, policies) + + check_ontodt(load(FIXTURES / "ontodt_graph.json")) + + # 17. Prove the monotonicity property test bites BEFORE trusting a pass from it. + binds, note = check_m5_binds() + CHECKS["m5-property-test-binds"] = binds + if not binds: + FAILURES.append(f"M5: {note}") + + check_negatives(schemas) + + for m in FAILURES: + print(f"FAIL: {m}", file=sys.stderr) + ok = not FAILURES and all(CHECKS.values()) + print(json.dumps({"ok": ok, "checks": CHECKS}, indent=2, sort_keys=True)) + return 0 if ok else 1 + + +if __name__ == "__main__": + raise SystemExit(main())