diff --git a/CHANGELOG.md b/CHANGELOG.md index cf8ee6f..f0d3c01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). Thi ## [Unreleased] ### Added +- DataClass v0.1 (OntoDT/OntoDQ data-side governance, task #14; AMG 2.0 deck): a `DataClass` binds a datatype-ontology class (OntoDT), the business `GlossaryTerm` (biz↔data), and a `ValidValues` domain (enum/range/regex), with an optional **TF-Lattice wide-and-deep** classifier that is a first-class CATALOGED model — `modelRef`→`ModelManifest`, `runRef`→`RunRecord`, `compute` on **Ray/TritFabric**, `monotonicFeatures` lattice constraint, and `labels` that are `GlossaryTerm` URNs (assigned in the glossary). `EntityField` gains an optional `dataClassRef`; the drift-guard enforces field↔class domain conformance + classifier integrity (labels are glossary terms, model is cataloged, compute is ray/tritfabric) + ontological typing. 4 negative vectors, teeth-verified. Backward-compatible. Two-level classifier: per-class LOGISTIC head (`DataClass.classifier`, one-vs-rest, required `evalRunRef` — individually testable per class/glossary term) + a per-table SOFTMAX `TableClassifier` aligning an LSA bag-of-words embedding AND a doc2vec sentence-encoder (the n-ary logit→class assignment), a cataloged model on Ray/TritFabric assigning N DataClasses. Drift-guard enforces both. - Glossary alignment v0.1 (vocabulary as governance substrate, task #13): `GlossaryTerm` extended with `partOfSpeech` (NP/VP, Tesnière), `status`, a 3-method `alignment` (formal ontology class + NP↔VP vector link in the sovereign 768 space + estate entity/service binding), and typed `relations` over a closed RDF/RDFS/SKOS/FOAF predicate set (is-type/has-type, is-a, has-a, is-member/has-member, skos:broader/narrower/related, foaf:member, has-datatype). Fail-closed drift-guard (`validate_glossary_alignment_examples.py`): an `approved` term must be captured+vector-aligned+implemented (NP→entity, VP→action/service) or it's a governance hole; vector links must be reciprocal; relation inverse-pairs reciprocate; `skos:related` is symmetric; `is-a` subsumption is acyclic. Teeth-verified. Backward-compatible (optional). Follow-ups: LSA↔LDA currency detection + agreement test vs the blast-radius graph + neurosymbolic domain. - Receipt **epistemicLevel** (SP-GATE-003, closes audit G3): `ReleaseReceipt` and `SyncCycleReceipt` may now carry `epistemicLevel` (`Proved`/`Speculative`/`Refuted`) + an `attestation` block — the MEET of attestation ∧ verification. Optional + backward-compatible; `tools/validate_sync_cycle_receipts.py` RECOMPUTES the meet and rejects a declared value that disagrees, so a receipt can never claim `Proved` while unsigned or after a failed check (absence of a signature returns Speculative, not Proved). Consumes the explicit `attestation.signed` from SP-GATE-002 (source-os #312) and the SyncCycleReceipt attestation from SP-GATE-001. Examples: verified+signed→Proved, verified-but-unsigned→Speculative, failed→Refuted, dry-run→Speculative. Teeth-verified. - Data-Acquisition **citable commons** (WO-COMMONS-2): a promoted platform `CatalogEntry` may carry `promotion.doi` and the approving `DataAcquisitionRequest` records the same `mintedDoi` — a citable DOI (Zenodo/DataCite-style) minted when the governance ∧ IP/legal meet approves promotion. Invariants (schema `allOf` + validator): a DOI may appear only on a `promoted` entry, `mintedDoi` only on an `approved` DAR, and where both are present they must be equal (minted once). Grounds the governed platform tier as a citable open commons. diff --git a/Makefile b/Makefile index 46fc379..4749a7e 100644 --- a/Makefile +++ b/Makefile @@ -1,12 +1,16 @@ -.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples +.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples -validate: validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract +validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract @echo "OK: validate" validate-ingestion-pipeline-examples: python3 -m pip install --user jsonschema >/dev/null python3 tools/validate_ingestion_pipeline_examples.py +validate-data-class-examples: + python3 -m pip install --user jsonschema >/dev/null + python3 tools/validate_data_class_examples.py + validate-glossary-alignment-examples: python3 -m pip install --user jsonschema >/dev/null python3 tools/validate_glossary_alignment_examples.py diff --git a/examples/data_class.currency.json b/examples/data_class.currency.json new file mode 100644 index 0000000..87ae442 --- /dev/null +++ b/examples/data_class.currency.json @@ -0,0 +1,40 @@ +{ + "id": "urn:srcos:data-class:currency-amount-usd", + "type": "DataClass", + "specVersion": "2.0.0", + "ontologyClassRef": "urn:srcos:ontology-class:datatype.CurrencyAmount", + "glossaryTermRef": "urn:srcos:glossary:revenue", + "domain": { + "kind": "range", + "values": [], + "min": 0, + "max": null, + "pattern": null + }, + "dataQualityRules": [ + "dq.nonnegative", + "dq.currency.usd.scale2" + ], + "classifier": { + "kind": "tf-lattice-wide-and-deep", + "modelRef": "urn:srcos:model-manifest:dataclass-assigner-lattice-v1", + "version": "1.0.0", + "runRef": "urn:srcos:run:dataclass-assigner-train-001", + "compute": { + "platform": "ray", + "assetRef": "urn:srcos:physical-asset:ray-cluster-01", + "serviceRef": "urn:srcos:tritfabric-consumption-api:assigner" + }, + "monotonicFeatures": [ + "token_magnitude", + "decimal_places" + ], + "labels": [ + "urn:srcos:glossary:revenue", + "urn:srcos:glossary:cost" + ], + "head": "logistic", + "evalRunRef": "urn:srcos:run:dataclass-currency-logistic-eval-001", + "threshold": 0.5 + } +} diff --git a/examples/entity_field.revenue.json b/examples/entity_field.revenue.json new file mode 100644 index 0000000..2b9ad95 --- /dev/null +++ b/examples/entity_field.revenue.json @@ -0,0 +1,16 @@ +{ + "name": "revenue_usd", + "dataType": "decimal", + "nullable": false, + "validValues": { + "kind": "range", + "values": [], + "min": 0, + "max": null, + "pattern": null + }, + "glossaryTerms": [ + "urn:srcos:glossary:revenue" + ], + "dataClassRef": "urn:srcos:data-class:currency-amount-usd" +} diff --git a/examples/table_classifier.finance.json b/examples/table_classifier.finance.json new file mode 100644 index 0000000..3cad0df --- /dev/null +++ b/examples/table_classifier.finance.json @@ -0,0 +1,27 @@ +{ + "id": "urn:srcos:table-classifier:finance-ledger-v1", + "type": "TableClassifier", + "specVersion": "2.0.0", + "schemaRef": "urn:srcos:schema-definition:finance-ledger", + "head": "softmax", + "embeddings": [ + { + "kind": "lsa-bag-of-words", + "dimension": 300 + }, + { + "kind": "doc2vec-sentence-encoder", + "dimension": 300 + } + ], + "modelRef": "urn:srcos:model-manifest:table-assigner-softmax-v1", + "version": "1.0.0", + "runRef": "urn:srcos:run:table-assigner-train-001", + "compute": { + "platform": "tritfabric", + "assetRef": "urn:srcos:physical-asset:trit-cluster-01" + }, + "assignsClasses": [ + "urn:srcos:data-class:currency-amount-usd" + ] +} diff --git a/fixtures/data-class/conformance.json b/fixtures/data-class/conformance.json new file mode 100644 index 0000000..52d31f0 --- /dev/null +++ b/fixtures/data-class/conformance.json @@ -0,0 +1,243 @@ +{ + "description": "Negative vectors for the DataClass (OntoDT/OntoDQ) family v0.1. Each MUST fail on its named failValidator; validate_data_class_examples.py enforces both, plus: a DataClass must be ontologically typed + glossary-linked + domain-bounded; a classifier's labels are GlossaryTerms and its compute is Ray/TritFabric; and a bound EntityField's validValues must conform to the DataClass domain.", + "cases": [ + { + "schema": "DataClass.json", + "reason": "classifier kind is pinned to tf-lattice-wide-and-deep.", + "failValidator": "const", + "document": { + "id": "urn:srcos:data-class:currency-amount-usd", + "type": "DataClass", + "specVersion": "2.0.0", + "ontologyClassRef": "urn:srcos:ontology-class:datatype.CurrencyAmount", + "glossaryTermRef": "urn:srcos:glossary:revenue", + "domain": { + "kind": "range", + "values": [], + "min": 0, + "max": null, + "pattern": null + }, + "dataQualityRules": [ + "dq.nonnegative", + "dq.currency.usd.scale2" + ], + "classifier": { + "kind": "random-forest", + "modelRef": "urn:srcos:model-manifest:dataclass-assigner-lattice-v1", + "version": "1.0.0", + "runRef": "urn:srcos:run:dataclass-assigner-train-001", + "compute": { + "platform": "ray", + "assetRef": "urn:srcos:physical-asset:ray-cluster-01", + "serviceRef": "urn:srcos:tritfabric-consumption-api:assigner" + }, + "monotonicFeatures": [ + "token_magnitude", + "decimal_places" + ], + "labels": [ + "urn:srcos:glossary:revenue", + "urn:srcos:glossary:cost" + ], + "head": "logistic", + "evalRunRef": "urn:srcos:run:dataclass-currency-logistic-eval-001", + "threshold": 0.5 + } + } + }, + { + "schema": "DataClass.json", + "reason": "classifier compute platform is closed (ray|tritfabric).", + "failValidator": "enum", + "document": { + "id": "urn:srcos:data-class:currency-amount-usd", + "type": "DataClass", + "specVersion": "2.0.0", + "ontologyClassRef": "urn:srcos:ontology-class:datatype.CurrencyAmount", + "glossaryTermRef": "urn:srcos:glossary:revenue", + "domain": { + "kind": "range", + "values": [], + "min": 0, + "max": null, + "pattern": null + }, + "dataQualityRules": [ + "dq.nonnegative", + "dq.currency.usd.scale2" + ], + "classifier": { + "kind": "tf-lattice-wide-and-deep", + "modelRef": "urn:srcos:model-manifest:dataclass-assigner-lattice-v1", + "version": "1.0.0", + "runRef": "urn:srcos:run:dataclass-assigner-train-001", + "compute": { + "platform": "lambda" + }, + "monotonicFeatures": [ + "token_magnitude", + "decimal_places" + ], + "labels": [ + "urn:srcos:glossary:revenue", + "urn:srcos:glossary:cost" + ], + "head": "logistic", + "evalRunRef": "urn:srcos:run:dataclass-currency-logistic-eval-001", + "threshold": 0.5 + } + } + }, + { + "schema": "DataClass.json", + "reason": "classifier labels must be GlossaryTerm URNs (assigned in the glossary).", + "failValidator": "pattern", + "document": { + "id": "urn:srcos:data-class:currency-amount-usd", + "type": "DataClass", + "specVersion": "2.0.0", + "ontologyClassRef": "urn:srcos:ontology-class:datatype.CurrencyAmount", + "glossaryTermRef": "urn:srcos:glossary:revenue", + "domain": { + "kind": "range", + "values": [], + "min": 0, + "max": null, + "pattern": null + }, + "dataQualityRules": [ + "dq.nonnegative", + "dq.currency.usd.scale2" + ], + "classifier": { + "kind": "tf-lattice-wide-and-deep", + "modelRef": "urn:srcos:model-manifest:dataclass-assigner-lattice-v1", + "version": "1.0.0", + "runRef": "urn:srcos:run:dataclass-assigner-train-001", + "compute": { + "platform": "ray", + "assetRef": "urn:srcos:physical-asset:ray-cluster-01", + "serviceRef": "urn:srcos:tritfabric-consumption-api:assigner" + }, + "monotonicFeatures": [ + "token_magnitude", + "decimal_places" + ], + "labels": [ + "not-a-glossary-urn" + ], + "head": "logistic", + "evalRunRef": "urn:srcos:run:dataclass-currency-logistic-eval-001", + "threshold": 0.5 + } + } + }, + { + "schema": "DataClass.json", + "reason": "ontologyClassRef is required — a data class must be ontologically typed.", + "failValidator": "required", + "document": { + "id": "urn:srcos:data-class:currency-amount-usd", + "type": "DataClass", + "specVersion": "2.0.0", + "glossaryTermRef": "urn:srcos:glossary:revenue", + "domain": { + "kind": "range", + "values": [], + "min": 0, + "max": null, + "pattern": null + }, + "dataQualityRules": [ + "dq.nonnegative", + "dq.currency.usd.scale2" + ], + "classifier": { + "kind": "tf-lattice-wide-and-deep", + "modelRef": "urn:srcos:model-manifest:dataclass-assigner-lattice-v1", + "version": "1.0.0", + "runRef": "urn:srcos:run:dataclass-assigner-train-001", + "compute": { + "platform": "ray", + "assetRef": "urn:srcos:physical-asset:ray-cluster-01", + "serviceRef": "urn:srcos:tritfabric-consumption-api:assigner" + }, + "monotonicFeatures": [ + "token_magnitude", + "decimal_places" + ], + "labels": [ + "urn:srcos:glossary:revenue", + "urn:srcos:glossary:cost" + ], + "head": "logistic", + "evalRunRef": "urn:srcos:run:dataclass-currency-logistic-eval-001", + "threshold": 0.5 + } + } + }, + { + "schema": "TableClassifier.json", + "reason": "table classifier head is pinned to softmax (n-ary).", + "failValidator": "const", + "document": { + "id": "urn:srcos:table-classifier:finance-ledger-v1", + "type": "TableClassifier", + "specVersion": "2.0.0", + "schemaRef": "urn:srcos:schema-definition:finance-ledger", + "head": "logistic", + "embeddings": [ + { + "kind": "lsa-bag-of-words", + "dimension": 300 + }, + { + "kind": "doc2vec-sentence-encoder", + "dimension": 300 + } + ], + "modelRef": "urn:srcos:model-manifest:table-assigner-softmax-v1", + "version": "1.0.0", + "runRef": "urn:srcos:run:table-assigner-train-001", + "compute": { + "platform": "tritfabric", + "assetRef": "urn:srcos:physical-asset:trit-cluster-01" + }, + "assignsClasses": [ + "urn:srcos:data-class:currency-amount-usd" + ] + } + }, + { + "schema": "TableClassifier.json", + "reason": "embedding kind is closed to lsa-bag-of-words | doc2vec-sentence-encoder.", + "failValidator": "enum", + "document": { + "id": "urn:srcos:table-classifier:finance-ledger-v1", + "type": "TableClassifier", + "specVersion": "2.0.0", + "schemaRef": "urn:srcos:schema-definition:finance-ledger", + "head": "softmax", + "embeddings": [ + { + "kind": "word2vec" + }, + { + "kind": "doc2vec-sentence-encoder" + } + ], + "modelRef": "urn:srcos:model-manifest:table-assigner-softmax-v1", + "version": "1.0.0", + "runRef": "urn:srcos:run:table-assigner-train-001", + "compute": { + "platform": "tritfabric", + "assetRef": "urn:srcos:physical-asset:trit-cluster-01" + }, + "assignsClasses": [ + "urn:srcos:data-class:currency-amount-usd" + ] + } + } + ] +} diff --git a/schemas/DataClass.json b/schemas/DataClass.json new file mode 100644 index 0000000..1fed150 --- /dev/null +++ b/schemas/DataClass.json @@ -0,0 +1,149 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://schemas.srcos.ai/v2/DataClass.json", + "title": "DataClass", + "description": "An OntoDT/OntoDQ data class: a data type/class defined ontologically (mereotopology in the datatype ontology) with its value domain, its business glossary term, and an optional TF-Lattice wide-and-deep classifier — registered in the model catalog with a run on Ray/TritFabric — that assigns this class by inference. Bridges the business vocabulary (GlossaryTerm) to the data (EntityField). Governs data quality: a field bound to this class must conform to its domain.", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "type", + "specVersion", + "ontologyClassRef", + "glossaryTermRef", + "domain" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^urn:srcos:data-class:[A-Za-z0-9._~:-]+$" + }, + "type": { + "const": "DataClass" + }, + "specVersion": { + "type": "string" + }, + "ontologyClassRef": { + "type": "string", + "pattern": "^urn:srcos:ontology-class:[A-Za-z0-9._~:-]+$", + "description": "OntoDT — the datatype-ontology class this data class IS (capture; enables inference-based data-quality logic)." + }, + "glossaryTermRef": { + "type": "string", + "pattern": "^urn:srcos:glossary:", + "description": "The business GlossaryTerm this data class realizes — the biz↔data bridge." + }, + "domain": { + "$ref": "ValidValues.json", + "description": "The value domain: enumeration / range / regex. Reuses ValidValues (single source with EntityField.validValues)." + }, + "dataQualityRules": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Identifiers of data-quality validations enforced for this class (OntoDQ)." + }, + "classifier": { + "type": [ + "object", + "null" + ], + "additionalProperties": false, + "description": "Optional TF-Lattice wide-and-deep classifier that ASSIGNS this data class by inference. It is a first-class cataloged model: modelRef → ModelManifest, runRef → RunRecord, compute on Ray/TritFabric, labels are GlossaryTerms.", + "required": [ + "kind", + "head", + "modelRef", + "version", + "evalRunRef", + "runRef", + "compute", + "labels" + ], + "properties": { + "kind": { + "const": "tf-lattice-wide-and-deep" + }, + "modelRef": { + "type": "string", + "pattern": "^urn:srcos:model-manifest:", + "description": "The classifier model in the model catalog (ModelManifest)." + }, + "version": { + "type": "string" + }, + "runRef": { + "type": "string", + "pattern": "^urn:srcos:run:", + "description": "The training/eval RunRecord." + }, + "compute": { + "type": "object", + "additionalProperties": false, + "required": [ + "platform" + ], + "properties": { + "platform": { + "type": "string", + "enum": [ + "ray", + "tritfabric" + ] + }, + "assetRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$" + }, + "serviceRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$" + } + } + }, + "monotonicFeatures": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Features constrained monotone in the lattice (TF-Lattice shape constraint)." + }, + "labels": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "pattern": "^urn:srcos:glossary:" + }, + "description": "Output class labels — each a GlossaryTerm URN (labels are assigned in the glossary)." + }, + "head": { + "const": "logistic", + "description": "Per-class head: a LOGISTIC (one-vs-rest, binary) classifier so THIS class / glossary term can be tested INDIVIDUALLY (per-class precision/recall). The table-level n-ary assignment lives in TableClassifier (softmax)." + }, + "evalRunRef": { + "type": "string", + "pattern": "^urn:srcos:run:", + "description": "The per-class evaluation RunRecord — the individual test for this class/label." + }, + "threshold": { + "type": [ + "number", + "null" + ], + "minimum": 0, + "maximum": 1, + "description": "Decision threshold on the logistic probability." + } + } + } + } +} diff --git a/schemas/EntityField.json b/schemas/EntityField.json index 62abddb..7c8dfa8 100644 --- a/schemas/EntityField.json +++ b/schemas/EntityField.json @@ -45,6 +45,34 @@ "pattern": "^urn:srcos:glossary:" }, "description": "URNs of GlossaryTerms that this field represents. Pattern: urn:srcos:glossary:" + }, + "dataClassRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:data-class:[A-Za-z0-9._~:-]+$", + "description": "Optional binding to the OntoDQ DataClass this field is an instance of. When set, the field's validValues must conform to the DataClass domain (validator)." + } + }, + "allOf": [ + { + "$comment": "A field bound to a DataClass must declare its validValues, so its domain conformance is checkable.", + "if": { + "required": [ + "dataClassRef" + ], + "properties": { + "dataClassRef": { + "type": "string" + } + } + }, + "then": { + "required": [ + "validValues" + ] + } } - } + ] } diff --git a/schemas/TableClassifier.json b/schemas/TableClassifier.json new file mode 100644 index 0000000..b82eb74 --- /dev/null +++ b/schemas/TableClassifier.json @@ -0,0 +1,121 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://schemas.srcos.ai/v2/TableClassifier.json", + "title": "TableClassifier", + "description": "The per-table (per-SchemaDefinition) SOFTMAX classifier that assigns each column to one of N DataClasses — the n-ary logit→class assignment. It ALIGNS two representations of each column: an LSA bag-of-words embedding (the closed fixed vocabulary) and a doc2vec sentence-encoder (contextual). A first-class cataloged model (ModelManifest) with a run on Ray/TritFabric.", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "type", + "specVersion", + "head", + "embeddings", + "modelRef", + "assignsClasses" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^urn:srcos:table-classifier:[A-Za-z0-9._~:-]+$" + }, + "type": { + "const": "TableClassifier" + }, + "specVersion": { + "type": "string" + }, + "schemaRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$", + "description": "The SchemaDefinition/table this classifier assigns columns for." + }, + "head": { + "const": "softmax", + "description": "n-ary multinomial head — one class per column." + }, + "embeddings": { + "type": "array", + "minItems": 2, + "description": "The representations aligned by the softmax. MUST include both an LSA bag-of-words embedding and a doc2vec sentence-encoder (the n-ary logit→class examples).", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind" + ], + "properties": { + "kind": { + "type": "string", + "enum": [ + "lsa-bag-of-words", + "doc2vec-sentence-encoder" + ] + }, + "dimension": { + "type": [ + "integer", + "null" + ], + "minimum": 1 + } + } + } + }, + "modelRef": { + "type": "string", + "pattern": "^urn:srcos:model-manifest:" + }, + "version": { + "type": [ + "string", + "null" + ] + }, + "runRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:run:" + }, + "compute": { + "type": [ + "object", + "null" + ], + "additionalProperties": false, + "required": [ + "platform" + ], + "properties": { + "platform": { + "type": "string", + "enum": [ + "ray", + "tritfabric" + ] + }, + "assetRef": { + "type": [ + "string", + "null" + ], + "pattern": "^urn:srcos:[A-Za-z0-9._~:-]+$" + } + } + }, + "assignsClasses": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "pattern": "^urn:srcos:data-class:[A-Za-z0-9._~:-]+$" + }, + "description": "The DataClasses this softmax can assign — the N of the n-ary." + } + } +} diff --git a/specs/data-class-contract.md b/specs/data-class-contract.md new file mode 100644 index 0000000..023a30a --- /dev/null +++ b/specs/data-class-contract.md @@ -0,0 +1,37 @@ +# DataClass Contract (v0.1) — OntoDT / OntoDQ, the data side of governance + +The business vocabulary (`GlossaryTerm`) says what a thing MEANS; the `DataClass` says what +its DATA is, ontologically — so data quality can be inferred and enforced, not asserted. + +A `DataClass` binds three things and is fail-closed if any is missing, plus an optional fourth (assignment): +- **OntoDT** — `ontologyClassRef`: the datatype-ontology class (capture; enables reusable, + inferable data-quality logic — IBM OntoDT). +- **biz↔data** — `glossaryTermRef`: the business `GlossaryTerm` this data class realizes. +- **domain** — `domain` ($ref `ValidValues`): enumeration / range / regex (valid values, + bounds, pattern), the same constraint an `EntityField` carries. +- **assignment** (optional) — a **TF-Lattice wide-and-deep** `classifier` that assigns this + class by inference (OntoDQ). It is a first-class **cataloged model**: `modelRef` → + `ModelManifest`, `runRef` → `RunRecord`, `compute.platform` ∈ {ray, tritfabric} with an + asset/service ref, `monotonicFeatures` for the lattice shape constraint, and `labels` that + are **`GlossaryTerm` URNs** (labels are assigned in the glossary). + +`EntityField` gains an optional `dataClassRef`. The validator enforces field↔class conformance: +a field bound to a DataClass must share its `domain.kind` — a field cannot claim a class it +does not fit. + +## Conformance +`make validate-data-class-examples` (in `make validate`). + +## Two-level classifier architecture + +Assignment is tested at two granularities, so each class is verifiable on its own and the +table is verifiable as a whole: + +- **Per-class — LOGISTIC** (`DataClass.classifier`, `head: logistic`): a one-vs-rest binary + classifier per class / glossary term, with a required `evalRunRef` — the **individual test** + for that class (per-class precision/recall). A class with no eval run is untestable and refused. +- **Per-table — SOFTMAX** (`TableClassifier`, `head: softmax`): the n-ary multinomial that + assigns each column to one of N `DataClass`es. It must **align two representations** of each + column — an **LSA bag-of-words** embedding (the closed fixed vocabulary) and a **doc2vec + sentence-encoder** (contextual) — because these are the n-ary examples of the logit→class + assignment. A cataloged model (`ModelManifest`) with a run on Ray/TritFabric. diff --git a/tools/validate_data_class_examples.py b/tools/validate_data_class_examples.py new file mode 100644 index 0000000..5e77ba0 --- /dev/null +++ b/tools/validate_data_class_examples.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +"""Validate the DataClass (OntoDT/OntoDQ) family + its cross-bindings (task #14). + +A DataClass is the data-side governance node: ontologically typed (OntoDT), glossary-linked +(biz↔data), domain-bounded (ValidValues), optionally assigned by a TF-Lattice classifier that +is a CATALOGED model (ModelManifest) with a run on Ray/TritFabric and glossary-term labels. + +Checks (fail-closed): + 1. schema conformance (with ValidValues $ref resolved); + 2. classifier integrity — kind pinned, modelRef→model-manifest, runRef→run, compute on + ray|tritfabric, and every label a GlossaryTerm URN (labels live in the glossary); + 3. field↔class domain conformance — an EntityField.dataClassRef binding a DataClass in the + set must share the DataClass domain 'kind' (a field can't claim a class it doesn't fit); + 4. negative vectors fail on their named keyword. +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import jsonschema +from referencing import Registry, Resource + +ROOT = Path(__file__).resolve().parents[1] +SCHEMAS = ROOT / "schemas" + +FAILURES: list[str] = [] +CHECKS: dict[str, bool] = {} + + +def load(p: Path) -> dict: + return json.loads(p.read_text(encoding="utf-8")) + + +def registry() -> Registry: + # Resolve local $refs (e.g. "ValidValues.json") by loading every schema under its $id + # AND under its bare filename, so relative refs resolve without a network fetch. + resources = [] + for f in SCHEMAS.glob("*.json"): + schema = load(f) + res = Resource.from_contents(schema) + resources.append((schema.get("$id", f.name), res)) + resources.append((f.name, res)) + return Registry().with_resources(resources) + + +REG = registry() + + +def validator_for(schema: dict) -> jsonschema.Draft202012Validator: + return jsonschema.Draft202012Validator(schema, registry=REG) + + +def check_conformance(dataclass_schema, field_schema, dcs, fields) -> None: + # Validate each collection against its INTENDED schema explicitly — never pick the + # schema from the instance's own `type`, or a mistyped doc validates against the wrong one. + for collection, schema in ((dcs, dataclass_schema), (fields, field_schema)): + for name, d in collection.items(): + errs = sorted(validator_for(schema).iter_errors(d), key=str) + if errs: + for e in errs: + FAILURES.append(f"{name}: {e.message}") + else: + CHECKS[f"schema:{name}"] = True + + +def check_classifier(dcs) -> None: + for name, dc in dcs.items(): + clf = dc.get("classifier") + if not clf: + CHECKS[f"classifier:{name}:none"] = True + continue + if not isinstance(clf, dict): + FAILURES.append(f"{name}: classifier must be an object") # type-guard: don't crash on a bad shape + continue + labels = clf.get("labels") or [] + compute = clf.get("compute") if isinstance(clf.get("compute"), dict) else {} + if not all(l.startswith("urn:srcos:glossary:") for l in labels): + FAILURES.append(f"{name}: classifier labels must all be GlossaryTerm URNs (assigned in the glossary)") + elif not str(clf.get("modelRef", "")).startswith("urn:srcos:model-manifest:"): + FAILURES.append(f"{name}: classifier.modelRef must be a ModelManifest (cataloged model)") + elif not str(clf.get("runRef", "")).startswith("urn:srcos:run:"): + FAILURES.append(f"{name}: classifier.runRef must be a RunRecord URN (the training run that produced this model)") + elif compute.get("platform") not in ("ray", "tritfabric"): + FAILURES.append(f"{name}: classifier compute must run on ray|tritfabric") + elif clf.get("head") != "logistic": + FAILURES.append(f"{name}: per-class classifier head must be 'logistic' (one-vs-rest) so the class is individually testable") + elif not str(clf.get("evalRunRef", "")).startswith("urn:srcos:run:"): + FAILURES.append(f"{name}: per-class classifier must carry an evalRunRef (RunRecord URN) — the individual test for this class/label") + else: + CHECKS[f"classifier:{name}:logistic-individually-testable"] = True + + +def check_table_classifiers(tcs) -> None: + """A TableClassifier is the n-ary SOFTMAX; it must align BOTH a LSA bag-of-words and a + doc2vec sentence-encoder, and be a cataloged model on ray/tritfabric assigning N DataClasses.""" + for name, tc in tcs.items(): + kinds = {e.get("kind") for e in (tc.get("embeddings") or []) if isinstance(e, dict)} + if tc.get("head") != "softmax": + FAILURES.append(f"{name}: table classifier head must be 'softmax' (n-ary)") + elif not {"lsa-bag-of-words", "doc2vec-sentence-encoder"} <= kinds: + FAILURES.append(f"{name}: softmax must align BOTH lsa-bag-of-words AND doc2vec-sentence-encoder " + f"(the n-ary logit→class representations); have {sorted(kinds)}") + elif not tc.get("modelRef", "").startswith("urn:srcos:model-manifest:"): + FAILURES.append(f"{name}: modelRef must be a cataloged ModelManifest") + elif not (tc.get("assignsClasses") and all(c.startswith("urn:srcos:data-class:") for c in tc["assignsClasses"])): + FAILURES.append(f"{name}: assignsClasses must be DataClass URNs (the N of the n-ary)") + else: + CHECKS[f"table-classifier:{name}:softmax-lsa+doc2vec-cataloged"] = True + + +def check_field_conformance(dcs, fields) -> None: + by_id = {dc["id"]: dc for dc in dcs.values()} + for name, fld in fields.items(): + ref = fld.get("dataClassRef") + if not ref: + continue + dc = by_id.get(ref) + if dc is None: + continue # DataClass defined elsewhere + field_kind = (fld.get("validValues") or {}).get("kind") + class_kind = (dc.get("domain") or {}).get("kind") + if field_kind and class_kind and field_kind != class_kind: + FAILURES.append(f"{name}: validValues.kind {field_kind!r} does not conform to DataClass {ref} " + f"domain kind {class_kind!r}") + else: + CHECKS[f"field-conforms:{name}"] = True + + +def check_negatives(dataclass_schema) -> None: + fx = load(ROOT / "fixtures" / "data-class" / "conformance.json") + for i, case in enumerate(fx["cases"]): + if case["schema"] != "DataClass.json": + continue # TableClassifier negatives handled in main() against their own schema + expected = case.get("failValidator") + try: + validator_for(dataclass_schema).validate(case["document"]) + except jsonschema.ValidationError as exc: + if expected is not None and exc.validator != expected: + FAILURES.append(f"negative {i}: failed on {exc.validator!r}, not {expected!r}: {case['reason']}") + else: + CHECKS[f"negative:{i}:{exc.validator}"] = True + continue + FAILURES.append(f"negative {i} unexpectedly PASSED: {case['reason']}") + + +def main() -> int: + dataclass_schema = load(SCHEMAS / "DataClass.json") + field_schema = load(SCHEMAS / "EntityField.json") + table_schema = load(SCHEMAS / "TableClassifier.json") + dcs = {"data_class.currency.json": load(ROOT / "examples" / "data_class.currency.json")} + fields = {"entity_field.revenue.json": load(ROOT / "examples" / "entity_field.revenue.json")} + tcs = {"table_classifier.finance.json": load(ROOT / "examples" / "table_classifier.finance.json")} + + check_conformance(dataclass_schema, field_schema, dcs, fields) + for name, tc in tcs.items(): + errs = sorted(validator_for(table_schema).iter_errors(tc), key=str) + if errs: + for e in errs: + FAILURES.append(f"{name}: {e.message}") + else: + CHECKS[f"schema:{name}"] = True + check_classifier(dcs) + check_table_classifiers(tcs) + check_field_conformance(dcs, fields) + check_negatives(dataclass_schema) + # TableClassifier negatives use their own schema + fx = load(ROOT / "fixtures" / "data-class" / "conformance.json") + for i, case in enumerate(fx["cases"]): + if case["schema"] != "TableClassifier.json": + continue + try: + validator_for(table_schema).validate(case["document"]) + FAILURES.append(f"negative {i} (TableClassifier) unexpectedly PASSED: {case['reason']}") + except jsonschema.ValidationError as exc: + exp = case.get("failValidator") + if exp and exc.validator != exp: + FAILURES.append(f"negative {i} (TableClassifier): failed on {exc.validator!r}, not {exp!r}") + else: + CHECKS[f"negative-tc:{i}:{exc.validator}"] = True + + for m in FAILURES: + print(f"FAIL: {m}", file=sys.stderr) + ok = not FAILURES and all(CHECKS.values()) + print(json.dumps({"ok": ok, "checks": CHECKS}, indent=2, sort_keys=True)) + return 0 if ok else 1 + + +if __name__ == "__main__": + raise SystemExit(main())