feat(vocab): learned spell-correction + user dictionary — context, not dictionary matching (task #13)

CHANGELOG.md

@@ -9,6 +9,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). Thi
 ## [Unreleased]
 
 ### Added
+- Learned spell-correction + user dictionary v0.1 — from context, not dictionary matching (task #13): `tools/learned_dictionary.py` decides each UNKNOWN token from a count-based skip-gram word-sense predictor (PPMI + truncated SVD; SGNS≈PPMI-SVD) instead of a static wordlist. LEARN a token that recurs with a coherent word-sense (a real domain term the dictionary hadn't seen); CORRECT a rare token to the known word its SENSE matches (skip-gram cosine picks the target, not edit distance alone — a near-spelled but different-sense token is not auto-corrected); leave the rest UNKNOWN (fail-closed, human-admitted). Every decision is a proposal, never a silent rewrite. `validate-learned-dictionary` teeth: epistemiclevel learned, reciept->receipt by sense, qwzptl unknown, learned term never auto-corrected. Uses numpy.
 - Stopword deviation analysis v0.1 — the dropped words are governed vocabulary too (task #13): `tools/stopword_analysis.py` audits the loop's stoplist ACROSS domains using two signals — cross-domain deviation (concentration) AND compositional density (repeated-collocation rate) — because frequency alone can't tell a domain term from a stylistic quirk. Surfaces `term-candidate` (concentrated + compositional = a domain term hiding in the stoplist, propose un-stoplisting) vs `stylistic` (concentrated by style only) vs `noise` (uniform). `validate-stopword-analysis` teeth: domain terms surfaced; a stylistically-concentrated word ('and') is NOT wrongly promoted; a uniform word ('the') is noise. `stopword-analysis-live` audits the shipped stoplist over specs/*.md. Compositional density is the bigram floor of the k-gram TF-IDF/LSA differential (orders 3..7) to follow.
 - k-gram TF-IDF/LSA differential v0.1 — confirm stopword candidates by compositional scale (task #13): `tools/kgram_tfidf_differential.py` measures a candidate word's domain-specificity across n-gram ORDERS 3..7 (TF-IDF over domains + LSA/truncated-SVD top component) and takes the differential. Signal is discounted by intrinsic unigram specificity so a stopword embedded in a domain phrase ('the state machine') can't borrow the phrase's specificity. A true term PERSISTS across orders (confirmed-term = strongest un-stoplist proposal); a concentrated-but-diffuse word ('and') or a borrowed-specificity word ('the') stays unconfirmed. `validate-kgram-differential` teeth: set/class/state confirmed across 3..7; 'and' unconfirmed; 'the' stripped by the unigram discount. Closes the two-stage design (stopword deviation -> k-gram confirmation). Uses numpy.
 - Agreement test v0.1 — glossary relations vs the blast-radius graph (task #13, neurosymbolic): `tools/agreement_test.py` projects the glossary's composition relations (`has-a`/`has-member`) onto the estate via `alignment.estateBinding` and compares them to a consumed blast-radius/dependency graph (GBRG owns that graph). Fail-closed on OVERCLAIM (a declared dependency with no observed edge = governance hole); reports DRIFT (an observed edge no relation names) as a remediation candidate (a proposed `has-a` relation), like the vocab-currency loop's candidate terms. `validate-agreement` teeth: aligned agrees; overclaim refused; drift surfaced as candidate.

Makefile

@@ -1,6 +1,6 @@
-.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live validate-kgram-differential
+.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live validate-kgram-differential validate-learned-dictionary
 
-validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis validate-kgram-differential
+validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis validate-kgram-differential validate-learned-dictionary
 	@echo "OK: validate"
 
 validate-ingestion-pipeline-examples:
@@ -26,6 +26,10 @@ validate-kgram-differential:
 	python3 -m pip install --user numpy >/dev/null
 	python3 tools/validate_kgram_differential.py
 
+validate-learned-dictionary:
+	python3 -m pip install --user numpy >/dev/null
+	python3 tools/validate_learned_dictionary.py
+
 validate-stopword-analysis:
 	python3 tools/validate_stopword_analysis.py
 

fixtures/learned-dictionary/corpus/gov1.json

@@ -0,0 +1,17 @@
+{
+  "domain": "gov1",
+  "documents": [
+    {
+      "text": "the sealed receipt recorded the governance decision and the receipt was signed"
+    },
+    {
+      "text": "the release gate verified the attestation and the gate was signed and verified"
+    },
+    {
+      "text": "the receipt carried an epistemiclevel proved not speculative on the signed attestation"
+    },
+    {
+      "text": "a signed attestation kept the epistemiclevel proved the receipt sealed the governance decision"
+    }
+  ]
+}

fixtures/learned-dictionary/corpus/gov2.json

@@ -0,0 +1,17 @@
+{
+  "domain": "gov2",
+  "documents": [
+    {
+      "text": "the gate verified the signed attestation the receipt sealed the governance decision proved"
+    },
+    {
+      "text": "the sealed reciept recorded the governance decision proved not speculative"
+    },
+    {
+      "text": "the receipt proved the epistemiclevel speculative until the attestation was signed and verified"
+    },
+    {
+      "text": "the governance decision recorded qwzptl and the gate verified the signed receipt"
+    }
+  ]
+}

specs/learned-dictionary.md

@@ -0,0 +1,24 @@
+# Learned spell-correction + user dictionary (v0.1) — from context, not dictionary matching
+
+A static wordlist flags every domain term (`epistemiclevel`, `srcos`, `governedloop`) as a
+misspelling and corrects it away — the estate's own vocabulary treated as errors. So what counts as
+a "correct" word must be LEARNED from context, not matched against a list. For each UNKNOWN token
+`tools/learned_dictionary.py` decides from a skip-gram word-sense predictor:
+
+- **learn** (add to the user dictionary) — the token RECURS with a COHERENT context (its context
+  windows cluster into one stable word-sense); a real term the dictionary simply hadn't seen.
+- **correct** (to a known word) — the token is RARE and both spelling-near (small edit distance)
+  AND sense-near (high skip-gram cosine) to a known word. **Sense — not edit distance alone —
+  picks the target**, so a token spelled near a known word but used in a different sense is not
+  auto-corrected.
+- **unknown** — neither coherent enough to learn nor sense-close to a known word: left for a human.
+
+The predictor is a count-based skip-gram — PPMI over a co-occurrence window + truncated SVD
+(Levy-Goldberg: SGNS factorises shifted PPMI, so this is the same word-sense family). No wordlist
+decides correctness; context does. **Fail-closed:** every decision is a PROPOSAL (add / correct-to),
+never a silent rewrite — a human or the superconscious admits it.
+
+`make validate-learned-dictionary` proves it: a novel domain term (`epistemiclevel`) is learned, a
+typo (`reciept`) is corrected to `receipt` by sense, garbage (`qwzptl`) is left unknown, and a
+learned term is never auto-corrected away. Same doctrine as the stopword analysis and glossary
+currency: replace static membership tests with learned, context-driven predictors.