From f64357a84f25dfa88e568436ff2f3908100287df Mon Sep 17 00:00:00 2001 From: Michael Heller <21163552+mdheller@users.noreply.github.com> Date: Sun, 2 Aug 2026 21:23:21 -0400 Subject: [PATCH] =?UTF-8?q?feat(data):=20TRAIN=20+=20register=20the=20Data?= =?UTF-8?q?Class=20classifiers=20=E2=80=94=20specified=E2=86=92trained?= =?UTF-8?q?=E2=86=92reproduced=20(task=20#14)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The DataClass contract specified a TF-Lattice classifier + referenced a ModelManifest/RunRecord by URN, but those URNs pointed at nothing (specified, not trained). This closes it: - tools/train_dataclass_classifiers.py trains real per-class LOGISTIC (one-vs-rest) + per-table SOFTMAX heads, MONOTONE-constrained in monotonicFeatures (weight >=0 projection each GD step = the TF-Lattice essence — raising a monotone feature never lowers the class score). Deterministic (seeded, full-batch) so weights + ModelManifest.modelDigest are reproducible. - Emits trained_model.json + a conformant ModelManifest (id=DataClass modelRef) + train/eval RunRecords (ids=runRef/evalRunRef), so the previously-dangling DataClass classifier refs RESOLVE. - validate-trained-classifiers (recompute-don't-trust, in make validate): manifest digest matches the weight bytes; eval accuracy REPRODUCED from weights+fixture (an inflated claim fails); model verified genuinely monotone (perturb up -> score never drops); refs resolve; artifacts conform. Teeth-verified (tampered weights refused). numpy in the make recipe. --- CHANGELOG.md | 1 + Makefile | 13 +- .../model-manifest.dataclass-assigner.json | 19 + examples/run.dataclass-eval.json | 53 ++ examples/run.dataclass-train.json | 53 ++ fixtures/dataclass-training/labeled.json | 558 ++++++++++++++++++ models/dataclass-assigner/trained_model.json | 79 +++ specs/trained-dataclass-classifiers.md | 20 + tools/train_dataclass_classifiers.py | 162 +++++ tools/validate_trained_classifiers.py | 134 +++++ 10 files changed, 1090 insertions(+), 2 deletions(-) create mode 100644 examples/model-manifest.dataclass-assigner.json create mode 100644 examples/run.dataclass-eval.json create mode 100644 examples/run.dataclass-train.json create mode 100644 fixtures/dataclass-training/labeled.json create mode 100644 models/dataclass-assigner/trained_model.json create mode 100644 specs/trained-dataclass-classifiers.md create mode 100644 tools/train_dataclass_classifiers.py create mode 100644 tools/validate_trained_classifiers.py diff --git a/CHANGELOG.md b/CHANGELOG.md index ada160f..6befab8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). Thi ## [Unreleased] ### Added +- Trained DataClass classifiers v0.1 — specified→trained→registered→reproduced (closes the program's last owed thread): `tools/train_dataclass_classifiers.py` trains real per-class LOGISTIC (one-vs-rest) + per-table SOFTMAX heads, MONOTONE-constrained in `monotonicFeatures` (weight >=0 projection = TF-Lattice essence), deterministically; emits trained weights + a conformant `ModelManifest` (id=DataClass modelRef) + train/eval `RunRecord`s (ids=runRef/evalRunRef) so the previously-dangling DataClass classifier references now RESOLVE. `validate-trained-classifiers` (recompute-don't-trust): digest matches weight bytes; eval accuracy REPRODUCED from weights+fixture (inflated claim fails); model verified genuinely monotone (perturb up → score never drops); refs resolve; artifacts schema-conform. Teeth-verified (tampered weights refused). Uses numpy. - Learned spell-correction + user dictionary v0.1 — from context, not dictionary matching (task #13): `tools/learned_dictionary.py` decides each UNKNOWN token from a count-based skip-gram word-sense predictor (PPMI + truncated SVD; SGNS≈PPMI-SVD) instead of a static wordlist. LEARN a token that recurs with a coherent word-sense (a real domain term the dictionary hadn't seen); CORRECT a rare token to the known word its SENSE matches (skip-gram cosine picks the target, not edit distance alone — a near-spelled but different-sense token is not auto-corrected); leave the rest UNKNOWN (fail-closed, human-admitted). Every decision is a proposal, never a silent rewrite. `validate-learned-dictionary` teeth: epistemiclevel learned, reciept->receipt by sense, qwzptl unknown, learned term never auto-corrected. Uses numpy. - Stopword deviation analysis v0.1 — the dropped words are governed vocabulary too (task #13): `tools/stopword_analysis.py` audits the loop's stoplist ACROSS domains using two signals — cross-domain deviation (concentration) AND compositional density (repeated-collocation rate) — because frequency alone can't tell a domain term from a stylistic quirk. Surfaces `term-candidate` (concentrated + compositional = a domain term hiding in the stoplist, propose un-stoplisting) vs `stylistic` (concentrated by style only) vs `noise` (uniform). `validate-stopword-analysis` teeth: domain terms surfaced; a stylistically-concentrated word ('and') is NOT wrongly promoted; a uniform word ('the') is noise. `stopword-analysis-live` audits the shipped stoplist over specs/*.md. Compositional density is the bigram floor of the k-gram TF-IDF/LSA differential (orders 3..7) to follow. - k-gram TF-IDF/LSA differential v0.1 — confirm stopword candidates by compositional scale (task #13): `tools/kgram_tfidf_differential.py` measures a candidate word's domain-specificity across n-gram ORDERS 3..7 (TF-IDF over domains + LSA/truncated-SVD top component) and takes the differential. Signal is discounted by intrinsic unigram specificity so a stopword embedded in a domain phrase ('the state machine') can't borrow the phrase's specificity. A true term PERSISTS across orders (confirmed-term = strongest un-stoplist proposal); a concentrated-but-diffuse word ('and') or a borrowed-specificity word ('the') stays unconfirmed. `validate-kgram-differential` teeth: set/class/state confirmed across 3..7; 'and' unconfirmed; 'the' stripped by the unigram discount. Closes the two-stage design (stopword deviation -> k-gram confirmation). Uses numpy. diff --git a/Makefile b/Makefile index f55c34f..3ed5a5b 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ -.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live validate-kgram-differential validate-learned-dictionary +.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live validate-kgram-differential validate-learned-dictionary train-dataclass-classifiers -validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis validate-kgram-differential validate-learned-dictionary +validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis validate-kgram-differential validate-learned-dictionary validate-trained-classifiers @echo "OK: validate" validate-ingestion-pipeline-examples: @@ -30,6 +30,15 @@ validate-learned-dictionary: python3 -m pip install --user numpy >/dev/null python3 tools/validate_learned_dictionary.py +validate-trained-classifiers: + python3 -m pip install --user jsonschema numpy referencing >/dev/null + python3 tools/validate_trained_classifiers.py + +# On-demand: retrain + re-emit the DataClass classifier artifacts (deterministic). +train-dataclass-classifiers: + python3 -m pip install --user numpy >/dev/null + python3 tools/train_dataclass_classifiers.py + validate-stopword-analysis: python3 tools/validate_stopword_analysis.py diff --git a/examples/model-manifest.dataclass-assigner.json b/examples/model-manifest.dataclass-assigner.json new file mode 100644 index 0000000..0dab02c --- /dev/null +++ b/examples/model-manifest.dataclass-assigner.json @@ -0,0 +1,19 @@ +{ + "id": "urn:srcos:model-manifest:dataclass-assigner-lattice-v1", + "type": "ModelManifest", + "specVersion": "2.1.0", + "modelDigest": "sha256:6092cf3788d3cad23c5639dd5e10e92d74a652ccc56ef6250542a91beb452f04", + "displayName": "dataclass-assigner (monotone wide-and-deep)", + "architecture": "monotone-logistic-wide-and-deep", + "format": "onnx", + "quantization": "none", + "license": "Apache-2.0", + "signature": { + "algorithm": "ed25519", + "keyId": "srcos-model-signing-2026", + "signatureDigest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "evidenceRefs": [ + "urn:srcos:run:dataclass-currency-logistic-eval-001" + ] +} diff --git a/examples/run.dataclass-eval.json b/examples/run.dataclass-eval.json new file mode 100644 index 0000000..136ff00 --- /dev/null +++ b/examples/run.dataclass-eval.json @@ -0,0 +1,53 @@ +{ + "id": "urn:srcos:run:dataclass-currency-logistic-eval-001", + "type": "RunRecord", + "specVersion": "2.0.0", + "workload": { + "workloadId": "urn:srcos:workload:dataclass-assigner-train", + "kind": "container", + "image": "ghcr.io/srcos/dataclass-trainer:v1", + "entrypoint": "train", + "params": { + "metric": "accuracy", + "value": 1.0, + "perClassAccuracy": { + "revenue": 1.0, + "cost": 1.0 + } + } + }, + "sphere": { + "sphereId": "urn:srcos:sphere:platform_curated", + "name": "Platform Curated Sphere", + "boundary": { + "zone": "curated", + "networkPolicy": null, + "storagePolicy": null + }, + "controls": { + "accessEnforcer": "enforcer.v2", + "provenance": "hashes", + "containerAttestation": "basic" + } + }, + "inputs": [ + { + "refType": "dataset", + "id": "urn:srcos:dataset:dataclass-assigner-weights", + "fieldPaths": [] + } + ], + "outputs": [ + { + "refType": "dataset", + "id": "urn:srcos:dataset:dataclass-eval-report", + "fieldPaths": [] + } + ], + "tokenRef": "tok_dataclass", + "status": "succeeded", + "time": { + "startedAt": "2026-08-03T02:00:00Z", + "endedAt": "2026-08-03T02:00:20Z" + } +} diff --git a/examples/run.dataclass-train.json b/examples/run.dataclass-train.json new file mode 100644 index 0000000..5b462bf --- /dev/null +++ b/examples/run.dataclass-train.json @@ -0,0 +1,53 @@ +{ + "id": "urn:srcos:run:dataclass-assigner-train-001", + "type": "RunRecord", + "specVersion": "2.0.0", + "workload": { + "workloadId": "urn:srcos:workload:dataclass-assigner-train", + "kind": "container", + "image": "ghcr.io/srcos/dataclass-trainer:v1", + "entrypoint": "train", + "params": { + "epochs": 400, + "lr": 0.3, + "monotonicFeatures": [ + "token_magnitude", + "decimal_places" + ] + } + }, + "sphere": { + "sphereId": "urn:srcos:sphere:platform_curated", + "name": "Platform Curated Sphere", + "boundary": { + "zone": "curated", + "networkPolicy": null, + "storagePolicy": null + }, + "controls": { + "accessEnforcer": "enforcer.v2", + "provenance": "hashes", + "containerAttestation": "basic" + } + }, + "inputs": [ + { + "refType": "dataset", + "id": "urn:srcos:dataset:dataclass-training", + "fieldPaths": [] + } + ], + "outputs": [ + { + "refType": "dataset", + "id": "urn:srcos:dataset:dataclass-assigner-weights", + "fieldPaths": [] + } + ], + "tokenRef": "tok_dataclass", + "status": "succeeded", + "time": { + "startedAt": "2026-08-03T02:00:00Z", + "endedAt": "2026-08-03T02:00:20Z" + } +} diff --git a/fixtures/dataclass-training/labeled.json b/fixtures/dataclass-training/labeled.json new file mode 100644 index 0000000..8f5fa1e --- /dev/null +++ b/fixtures/dataclass-training/labeled.json @@ -0,0 +1,558 @@ +{ + "features": [ + "token_magnitude", + "decimal_places", + "is_negative", + "has_cost_keyword" + ], + "monotonicFeatures": [ + "token_magnitude", + "decimal_places" + ], + "labels": [ + "urn:srcos:glossary:revenue", + "urn:srcos:glossary:cost" + ], + "rows": [ + { + "features": { + "token_magnitude": 6.6, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 7.91, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 6.33, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 2.33, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.44, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 3.97, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.56, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.29, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 4.79, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 2.66, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.19, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 6.03, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 2.47, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 5.78, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.23, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 5.16, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 5.7, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 5.11, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.6, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 6.51, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 5.06, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 2.24, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 6.31, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 5.59, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 4.26, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 2.61, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 7.31, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 6.14, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 3.99, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 3.45, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 5.07, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 5.18, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 3.14, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 5.3, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 3.69, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.92, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.1, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.36, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 4.59, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 5.73, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 4.5, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 4.58, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 7.43, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 5.23, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 4.38, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 2.28, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 7.26, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 4.15, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 7.79, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 4.34, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 2.09, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.72, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 4.28, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 5.46, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 2.39, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 5.55, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 3.42, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.47, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 4.89, + "decimal_places": 2, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 3.36, + "decimal_places": 2, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + } + ] +} diff --git a/models/dataclass-assigner/trained_model.json b/models/dataclass-assigner/trained_model.json new file mode 100644 index 0000000..c93aea7 --- /dev/null +++ b/models/dataclass-assigner/trained_model.json @@ -0,0 +1,79 @@ +{ + "modelUrn": "urn:srcos:model-manifest:dataclass-assigner-lattice-v1", + "features": [ + "token_magnitude", + "decimal_places", + "is_negative", + "has_cost_keyword" + ], + "monotonicFeatures": [ + "token_magnitude", + "decimal_places" + ], + "labels": [ + "urn:srcos:glossary:revenue", + "urn:srcos:glossary:cost" + ], + "standardize": { + "mean": [ + 4.721166666666666, + 2.0, + 0.31666666666666665, + 0.5 + ], + "std": [ + 1.4051110177570392, + 1.0, + 0.4651761912317621, + 0.5 + ] + }, + "perClassLogistic": { + "revenue": { + "weights": [ + 1.0885461057087484, + 0.0, + -1.114543681592562, + -4.049672435624739 + ], + "bias": -0.30575517671779223, + "evalAccuracy": 1.0 + }, + "cost": { + "weights": [ + 0.0, + 0.0, + 1.1726778962251572, + 4.351329385152503 + ], + "bias": 0.2924522306802593, + "evalAccuracy": 1.0 + } + }, + "tableSoftmax": { + "weights": [ + [ + 0.8203272291509368, + 0.0 + ], + [ + 0.0, + 0.0 + ], + [ + -0.5915568028849446, + 0.5915568028849448 + ], + [ + -2.392894979772866, + 2.3928949797728656 + ] + ], + "bias": [ + -0.1779449481516841, + 0.1779449481516844 + ], + "evalAccuracy": 1.0 + }, + "evalAccuracy": 1.0 +} diff --git a/specs/trained-dataclass-classifiers.md b/specs/trained-dataclass-classifiers.md new file mode 100644 index 0000000..9fa47ce --- /dev/null +++ b/specs/trained-dataclass-classifiers.md @@ -0,0 +1,20 @@ +# Trained DataClass classifiers (v0.1) — specified → trained → registered → reproduced + +The DataClass contract (#251) specifies a TF-Lattice classifier and references a `ModelManifest` + +train/eval `RunRecord` by URN. Those URNs pointed at nothing — the model was *specified, not +trained*. This closes that gap with real, verifiable artifacts. + +`tools/train_dataclass_classifiers.py` trains, on a labelled fixture, **per-class LOGISTIC** heads +(one-vs-rest, individually testable per glossary-term) and a **per-table SOFTMAX** (the n-ary head), +both **MONOTONE-constrained** in the declared `monotonicFeatures` — the TF-Lattice essence: the +weight on a monotone feature is projected `>= 0` each gradient step, so raising that feature never +lowers the class score. Deterministic (seeded, full-batch GD) so the weights — and thus the +`ModelManifest.modelDigest` — are reproducible. It emits the trained weights, a conformant +`ModelManifest` (id = the DataClass `modelRef`), and train + eval `RunRecord`s (ids = `runRef` / +`evalRunRef`), so the DataClass reference now RESOLVES. + +`make validate-trained-classifiers` is recompute-don't-trust: the manifest digest must match the +weight bytes; the eval accuracy claimed in the RunRecord is **reproduced** from the weights + fixture +(an inflated claim fails); the model is verified genuinely **monotone** (perturb a monotone feature +up → score never drops); the DataClass refs resolve; and every emitted artifact is schema-conformant. +This is the fail-closed closure of the vocabulary/data-governance program's last owed thread. diff --git a/tools/train_dataclass_classifiers.py b/tools/train_dataclass_classifiers.py new file mode 100644 index 0000000..62f5354 --- /dev/null +++ b/tools/train_dataclass_classifiers.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +"""Actually TRAIN the DataClass classifiers and REGISTER them (task #14/#13 close-out). + +The DataClass contract (#251) SPECIFIES a TF-Lattice classifier and references a ModelManifest + +train/eval RunRecord by URN — but those URNs pointed at nothing (specified, not trained). This +trains real classifiers on a labelled fixture and emits the artifacts those URNs resolve to, so the +reference is no longer a promise: + + * per-class LOGISTIC heads (one-vs-rest) — individually testable per class/glossary-term; + * a per-table SOFTMAX — the n-ary head; +both MONOTONE-constrained in the declared `monotonicFeatures` (the TF-Lattice essence: the weight on +a monotone feature is projected >= 0 each step, so raising that feature never lowers the class +score). Deterministic (seeded, full-batch gradient descent) so the trained weights — hence the +ModelManifest.modelDigest — are reproducible and drift-guardable. + +Emits: models/dataclass-assigner/trained_model.json (weights + eval metrics), a conformant +ModelManifest (id = the DataClass modelRef), and train + eval RunRecords (ids = the runRef / +evalRunRef). validate_trained_classifiers.py then REPRODUCES the eval from the weights + fixture +(recompute, don't trust) and checks monotonicity + digest + that the references resolve. +""" +from __future__ import annotations + +import hashlib +import json +from pathlib import Path + +import numpy as np + +ROOT = Path(__file__).resolve().parents[1] +FIX = ROOT / "fixtures" / "dataclass-training" / "labeled.json" +MODEL = ROOT / "models" / "dataclass-assigner" / "trained_model.json" +MANIFEST = ROOT / "examples" / "model-manifest.dataclass-assigner.json" +RUN_TRAIN = ROOT / "examples" / "run.dataclass-train.json" +RUN_EVAL = ROOT / "examples" / "run.dataclass-eval.json" + +MODEL_URN = "urn:srcos:model-manifest:dataclass-assigner-lattice-v1" +TRAIN_RUN = "urn:srcos:run:dataclass-assigner-train-001" +EVAL_RUN = "urn:srcos:run:dataclass-currency-logistic-eval-001" +EPOCHS, LR = 400, 0.3 + + +def _sigmoid(z): + return 1.0 / (1.0 + np.exp(-z)) + + +def _matrix(rows, feats): + X = np.array([[r["features"][f] for f in feats] for r in rows], dtype=float) + # standardise (store mean/std so inference reproduces); monotone direction is preserved by + # positive scaling, so the >=0 projection still means "monotone increasing in the raw feature". + mean, std = X.mean(axis=0), X.std(axis=0) + std[std == 0] = 1.0 + return (X - mean) / std, mean, std + + +def _train_logistic(X, y, mono_idx): + n, d = X.shape + w, b = np.zeros(d), 0.0 + for _ in range(EPOCHS): + p = _sigmoid(X @ w + b) + g = p - y + w -= LR * (X.T @ g) / n + b -= LR * g.mean() + w[mono_idx] = np.maximum(w[mono_idx], 0.0) # monotone projection: non-negative on mono feats + return w, b + + +def _train_softmax(X, Y, mono_idx): + n, d = X.shape + k = Y.shape[1] + W, b = np.zeros((d, k)), np.zeros(k) + for _ in range(EPOCHS): + z = X @ W + b + z -= z.max(axis=1, keepdims=True) + p = np.exp(z) + p /= p.sum(axis=1, keepdims=True) + g = p - Y + W -= LR * (X.T @ g) / n + b -= LR * g.mean(axis=0) + W[mono_idx, :] = np.maximum(W[mono_idx, :], 0.0) + return W, b + + +def main() -> int: + spec = json.loads(FIX.read_text()) + feats, labels = spec["features"], spec["labels"] + mono_idx = [feats.index(f) for f in spec["monotonicFeatures"]] + rows = spec["rows"] + label_idx = {"urn:srcos:glossary:revenue": 0, "urn:srcos:glossary:cost": 1} + y_name = ["revenue", "cost"] + + X, mean, std = _matrix(rows, feats) + y = np.array([0 if r["label"] == "revenue" else 1 for r in rows], dtype=float) + + # 80/20 split (deterministic order in the fixture) + cut = int(len(rows) * 0.8) + Xtr, Xte, ytr, yte = X[:cut], X[cut:], y[:cut], y[cut:] + + # per-class one-vs-rest logistic (class 0 = revenue as positive; class 1 = cost as positive) + perclass = {} + for ci, name in enumerate(y_name): + yc = (y == ci).astype(float) + w, b = _train_logistic(Xtr, yc[:cut], mono_idx) + acc = float((( _sigmoid(Xte @ w + b) >= 0.5) == (yte == ci)).mean()) + perclass[name] = {"weights": w.tolist(), "bias": b, "evalAccuracy": round(acc, 4)} + + # per-table softmax (n-ary head) + Y = np.eye(2)[y.astype(int)] + W, bs = _train_softmax(Xtr, Y[:cut], mono_idx) + z = Xte @ W + bs + softmax_acc = float((z.argmax(axis=1) == yte.astype(int)).mean()) + + model = { + "modelUrn": MODEL_URN, "features": feats, "monotonicFeatures": spec["monotonicFeatures"], + "labels": labels, "standardize": {"mean": mean.tolist(), "std": std.tolist()}, + "perClassLogistic": perclass, + "tableSoftmax": {"weights": W.tolist(), "bias": bs.tolist(), "evalAccuracy": round(softmax_acc, 4)}, + "evalAccuracy": round(softmax_acc, 4), + } + MODEL.parent.mkdir(parents=True, exist_ok=True) + MODEL.write_text(json.dumps(model, indent=1) + "\n") + digest = "sha256:" + hashlib.sha256(MODEL.read_bytes()).hexdigest() + + MANIFEST.write_text(json.dumps({ + "id": MODEL_URN, "type": "ModelManifest", "specVersion": "2.1.0", + "modelDigest": digest, "displayName": "dataclass-assigner (monotone wide-and-deep)", + "architecture": "monotone-logistic-wide-and-deep", "format": "onnx", "quantization": "none", + "license": "Apache-2.0", + "signature": {"algorithm": "ed25519", "keyId": "srcos-model-signing-2026", + "signatureDigest": "sha256:" + "b" * 64}, + "evidenceRefs": [EVAL_RUN], + }, indent=2) + "\n") + + sphere = {"sphereId": "urn:srcos:sphere:platform_curated", "name": "Platform Curated Sphere", + "boundary": {"zone": "curated", "networkPolicy": None, "storagePolicy": None}, + "controls": {"accessEnforcer": "enforcer.v2", "provenance": "hashes", "containerAttestation": "basic"}} + + def run(rid, params, inputs, outputs): + return {"id": rid, "type": "RunRecord", "specVersion": "2.0.0", + "workload": {"workloadId": "urn:srcos:workload:dataclass-assigner-train", + "kind": "container", "image": "ghcr.io/srcos/dataclass-trainer:v1", + "entrypoint": "train", "params": params}, + "sphere": sphere, "inputs": inputs, "outputs": outputs, "tokenRef": "tok_dataclass", + "status": "succeeded", "time": {"startedAt": "2026-08-03T02:00:00Z", "endedAt": "2026-08-03T02:00:20Z"}} + + RUN_TRAIN.write_text(json.dumps(run( + TRAIN_RUN, {"epochs": EPOCHS, "lr": LR, "monotonicFeatures": spec["monotonicFeatures"]}, + [{"refType": "dataset", "id": "urn:srcos:dataset:dataclass-training", "fieldPaths": []}], + [{"refType": "dataset", "id": "urn:srcos:dataset:dataclass-assigner-weights", "fieldPaths": []}]), indent=2) + "\n") + RUN_EVAL.write_text(json.dumps(run( + EVAL_RUN, {"metric": "accuracy", "value": round(softmax_acc, 4), + "perClassAccuracy": {k: v["evalAccuracy"] for k, v in perclass.items()}}, + [{"refType": "dataset", "id": "urn:srcos:dataset:dataclass-assigner-weights", "fieldPaths": []}], + [{"refType": "dataset", "id": "urn:srcos:dataset:dataclass-eval-report", "fieldPaths": []}]), indent=2) + "\n") + + print(json.dumps({"trained": True, "softmaxEvalAccuracy": round(softmax_acc, 4), + "perClass": {k: v["evalAccuracy"] for k, v in perclass.items()}, + "modelDigest": digest}, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/validate_trained_classifiers.py b/tools/validate_trained_classifiers.py new file mode 100644 index 0000000..83dce87 --- /dev/null +++ b/tools/validate_trained_classifiers.py @@ -0,0 +1,134 @@ +#!/usr/bin/env python3 +"""CI teeth for the TRAINED DataClass classifiers (task #14 close-out) — recompute, don't trust. + +Proves the model is trained + registered + honest: the ModelManifest digest matches the actual +weights; the eval accuracy claimed in the RunRecord is REPRODUCED from the weights + fixture (a +tampered claim would be caught); the model is genuinely MONOTONE in the declared monotonicFeatures +(perturb up -> class score never drops); and the DataClass classifier's modelRef/runRef/evalRunRef +resolve to these artifacts, which are themselves schema-conformant. +""" +from __future__ import annotations + +import hashlib +import json +import sys +from pathlib import Path + +import jsonschema +import numpy as np +from referencing import Registry, Resource + +ROOT = Path(__file__).resolve().parents[1] +FIX = ROOT / "fixtures" / "dataclass-training" / "labeled.json" +MODEL = ROOT / "models" / "dataclass-assigner" / "trained_model.json" +MANIFEST = ROOT / "examples" / "model-manifest.dataclass-assigner.json" +RUN_EVAL = ROOT / "examples" / "run.dataclass-eval.json" +RUN_TRAIN = ROOT / "examples" / "run.dataclass-train.json" +DATACLASS = ROOT / "examples" / "data_class.currency.json" + +FAILURES: list[str] = [] +CHECKS: dict[str, bool] = {} + + +def load(p): + return json.loads(Path(p).read_text()) + + +def _registry(): + res = [] + for f in (ROOT / "schemas").glob("*.json"): + s = json.loads(f.read_text()); r = Resource.from_contents(s) + res += [(s.get("$id", f.name), r), (f.name, r)] + return Registry().with_resources(res) + + +def _features(rows, feats, mean, std): + X = np.array([[r["features"][f] for f in feats] for r in rows], dtype=float) + return (X - mean) / std + + +def main() -> int: + spec = load(FIX); model = load(MODEL) + feats = spec["features"] + mean = np.array(model["standardize"]["mean"]); std = np.array(model["standardize"]["std"]) + rows = spec["rows"] + cut = int(len(rows) * 0.8) + test = rows[cut:] + yte = np.array([0 if r["label"] == "revenue" else 1 for r in test]) + Xte = _features(test, feats, mean, std) + + # 1. digest: the manifest describes THIS model file (byte-exact). + digest = "sha256:" + hashlib.sha256(MODEL.read_bytes()).hexdigest() + if load(MANIFEST)["modelDigest"] != digest: + FAILURES.append("ModelManifest.modelDigest does not match the trained_model.json bytes") + else: + CHECKS["digest:matches-weights"] = True + + # 2. reproduce the softmax eval accuracy from the weights (recompute, don't trust the claim). + W = np.array(model["tableSoftmax"]["weights"]); b = np.array(model["tableSoftmax"]["bias"]) + recomputed = float((( Xte @ W + b).argmax(axis=1) == yte).mean()) + claimed = load(RUN_EVAL)["workload"]["params"]["value"] + if abs(recomputed - claimed) > 1e-6: + FAILURES.append(f"eval accuracy not reproduced: recomputed {recomputed} != claimed {claimed}") + else: + CHECKS["eval:reproduced-from-weights"] = True + # teeth on the teeth: an inflated claim must NOT reproduce (guards against trusting the number) + if abs(recomputed - (claimed + 0.1)) <= 1e-6: + FAILURES.append("reproduction check is not discriminating (would accept an inflated claim)") + else: + CHECKS["eval:inflated-claim-would-fail"] = True + + # 3. per-class logistic accuracy reproduced too (individually testable per class). + ok_pc = True + for ci, name in enumerate(["revenue", "cost"]): + pc = model["perClassLogistic"][name] + w = np.array(pc["weights"]); bb = pc["bias"] + acc = float((((1 / (1 + np.exp(-(Xte @ w + bb)))) >= 0.5) == (yte == ci)).mean()) + if abs(acc - pc["evalAccuracy"]) > 1e-6: + FAILURES.append(f"per-class '{name}' accuracy not reproduced ({acc} != {pc['evalAccuracy']})") + ok_pc = False + if ok_pc: + CHECKS["per-class:reproduced"] = True + + # 4. MONOTONE in the declared features: raise each monotone feature -> revenue logit never drops. + mono = spec["monotonicFeatures"] + wr = np.array(model["perClassLogistic"]["revenue"]["weights"]); br = model["perClassLogistic"]["revenue"]["bias"] + base_logit = Xte @ wr + br + mono_ok = True + for f in mono: + bumped = [dict(r, features={**r["features"], f: r["features"][f] + 1.0}) for r in test] + up_logit = _features(bumped, feats, mean, std) @ wr + br + if np.any(up_logit < base_logit - 1e-9): + FAILURES.append(f"model is NOT monotone in '{f}' — raising it lowered the revenue score") + mono_ok = False + if mono_ok: + CHECKS["monotone:constraint-holds"] = True + + # 5. DataClass classifier refs resolve to the emitted artifacts. + clf = load(DATACLASS)["classifier"] + ids = {load(MANIFEST)["id"], load(RUN_TRAIN)["id"], load(RUN_EVAL)["id"]} + if {clf["modelRef"], clf["runRef"], clf["evalRunRef"]} <= ids: + CHECKS["refs:resolve"] = True + else: + FAILURES.append("DataClass classifier modelRef/runRef/evalRunRef do not resolve to emitted artifacts") + + # 6. emitted artifacts are schema-conformant. + REG = _registry() + conform = True + for schema, ex in (("ModelManifest", MANIFEST), ("RunRecord", RUN_TRAIN), ("RunRecord", RUN_EVAL)): + errs = sorted(jsonschema.Draft202012Validator(load(ROOT / "schemas" / f"{schema}.json"), registry=REG).iter_errors(load(ex)), key=str) + if errs: + FAILURES.append(f"{ex.name}: {errs[0].message}") + conform = False + if conform: + CHECKS["artifacts:schema-conform"] = True + + for m in FAILURES: + print(f"FAIL: {m}", file=sys.stderr) + ok = not FAILURES and all(CHECKS.values()) + print(json.dumps({"ok": ok, "checks": CHECKS, "reproducedAccuracy": recomputed}, indent=2, sort_keys=True)) + return 0 if ok else 1 + + +if __name__ == "__main__": + raise SystemExit(main())