feat(vocab): k-gram TF-IDF/LSA differential (orders 3-7) — confirm stopword candidates (task #13)

CHANGELOG.md

@@ -10,6 +10,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). Thi
 
 ### Added
 - Stopword deviation analysis v0.1 — the dropped words are governed vocabulary too (task #13): `tools/stopword_analysis.py` audits the loop's stoplist ACROSS domains using two signals — cross-domain deviation (concentration) AND compositional density (repeated-collocation rate) — because frequency alone can't tell a domain term from a stylistic quirk. Surfaces `term-candidate` (concentrated + compositional = a domain term hiding in the stoplist, propose un-stoplisting) vs `stylistic` (concentrated by style only) vs `noise` (uniform). `validate-stopword-analysis` teeth: domain terms surfaced; a stylistically-concentrated word ('and') is NOT wrongly promoted; a uniform word ('the') is noise. `stopword-analysis-live` audits the shipped stoplist over specs/*.md. Compositional density is the bigram floor of the k-gram TF-IDF/LSA differential (orders 3..7) to follow.
+- k-gram TF-IDF/LSA differential v0.1 — confirm stopword candidates by compositional scale (task #13): `tools/kgram_tfidf_differential.py` measures a candidate word's domain-specificity across n-gram ORDERS 3..7 (TF-IDF over domains + LSA/truncated-SVD top component) and takes the differential. Signal is discounted by intrinsic unigram specificity so a stopword embedded in a domain phrase ('the state machine') can't borrow the phrase's specificity. A true term PERSISTS across orders (confirmed-term = strongest un-stoplist proposal); a concentrated-but-diffuse word ('and') or a borrowed-specificity word ('the') stays unconfirmed. `validate-kgram-differential` teeth: set/class/state confirmed across 3..7; 'and' unconfirmed; 'the' stripped by the unigram discount. Closes the two-stage design (stopword deviation -> k-gram confirmation). Uses numpy.
 - Agreement test v0.1 — glossary relations vs the blast-radius graph (task #13, neurosymbolic): `tools/agreement_test.py` projects the glossary's composition relations (`has-a`/`has-member`) onto the estate via `alignment.estateBinding` and compares them to a consumed blast-radius/dependency graph (GBRG owns that graph). Fail-closed on OVERCLAIM (a declared dependency with no observed edge = governance hole); reports DRIFT (an observed edge no relation names) as a remediation candidate (a proposed `has-a` relation), like the vocab-currency loop's candidate terms. `validate-agreement` teeth: aligned agrees; overclaim refused; drift surfaced as candidate.
 - Glossary promotion v0.1 — the governed draft→approved alignment pass (task #13, closes the vocabulary lifecycle): `tools/promote_glossary_term.py` promotes a `draft` GlossaryTerm to `approved` ONLY when the 3-method alignment holds as a fail-closed MEET — capture (`ontologyClassRef`) ∧ vector-align (`vectorLink`, pinned to the sovereign nomic/768 space AND reciprocated by the named peer) ∧ implement (`estateBinding` entity/service/action). It recomputes the meet (never trusts a flag), refuses on any missing/non-reciprocal/off-space method (term stays draft, no governance hole), and its approved output PASSES the #250 alignment drift-guard by construction. `validate-glossary-promotion` teeth: promotes on full alignment; refuses missing-implement / non-reciprocal / off-space; promoted output passes the #250 guard.
 - Vocab-currency loop — LIVE dogfood on the estate's own vocab (`make vocab-currency-dogfood`): `tools/dogfood_vocab_currency.py` runs the governed loop over this repo's approved `GlossaryTerm`s (fixed/LSA) vs `specs/*.md` (open/LDA). Today only a handful of terms are approved against the full spec corpus, so the governed loop escalates-human (can't reach currency in its bound), writing real draft `GlossaryTerm` proposals to `build/vocab-currency-proposals/` — the artifacts ontogenesis ingests. Reports the outcome (does not gate on it, exit 0); Markdown-cleaned + generic-word stoplisted so domain terms surface. Enforcement stays on fixtures in `validate-vocab-currency-loop`.

Makefile

@@ -1,6 +1,6 @@
-.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live
+.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live validate-kgram-differential
 
-validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis
+validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis validate-kgram-differential
 	@echo "OK: validate"
 
 validate-ingestion-pipeline-examples:
@@ -22,6 +22,10 @@ validate-glossary-promotion:
 validate-agreement:
 	python3 tools/validate_agreement.py
 
+validate-kgram-differential:
+	python3 -m pip install --user numpy >/dev/null
+	python3 tools/validate_kgram_differential.py
+
 validate-stopword-analysis:
 	python3 tools/validate_stopword_analysis.py
 

fixtures/kgram-differential/candidates.json

@@ -0,0 +1,9 @@
+{
+  "candidates": [
+    "set",
+    "class",
+    "state",
+    "and",
+    "the"
+  ]
+}

fixtures/kgram-differential/domains/formal.json

@@ -0,0 +1,14 @@
+{
+  "domain": "formal",
+  "documents": [
+    {
+      "text": "the finite state machine reads the set of states. a base class defines the state of the set. the empty set of states in the state machine. the finite state machine reads the set of states. a base class defines the state of the set."
+    },
+    {
+      "text": "the empty set of states in the state machine. the finite state machine reads the set of states. a base class defines the state of the set. the finite state machine reads the set of states again. the empty set of states in the state machine."
+    },
+    {
+      "text": "a base class defines the state of the set. the empty set of states in the state machine. the finite state machine reads the set of states. a base class defines the state of the set once more. the finite state machine reads the set of states."
+    }
+  ]
+}

fixtures/kgram-differential/domains/narrative.json

@@ -0,0 +1,14 @@
+{
+  "domain": "narrative",
+  "documents": [
+    {
+      "text": "the dog and the cat ran across the field and the sun was warm and the children laughed and the day was long and the evening came and the stars and the moon rose over the hills"
+    },
+    {
+      "text": "and so the story went and the people gathered and the music played and the night was full and the wind and the rain and the memory of the summer and the sea and the distant shore"
+    },
+    {
+      "text": "the morning and the mist and the river and the birds and the song and the quiet and the town and the market and the voices and the laughter and the bread and the wine and the road"
+    }
+  ]
+}

specs/kgram-tfidf-differential.md

@@ -0,0 +1,30 @@
+# k-gram TF-IDF / LSA differential (v0.1) — confirm stopword candidates by compositional scale
+
+The stopword deviation analysis flags a dropped word as a term-candidate when it is concentrated in
+a domain AND recurs in repeated bigram collocations. That is the **bigram floor**. This is the
+confirmation: it measures the candidate's domain-specificity across n-gram **orders 3..7** and takes
+the **differential**.
+
+Per order `n`, over the domain corpora:
+1. build the n-gram × domain count matrix (n-grams over RAW tokens, so a stopword that is part of a
+   real phrase — "held to maturity", "empty set of states" — is captured);
+2. weight it **TF-IDF** (documents = domains): a phrase frequent in ONE domain and rare across the
+   rest scores high — it is domain-specific;
+3. take the **LSA** (truncated SVD): the top singular component is the dominant axis of cross-domain
+   variation, and a domain-specific n-gram loads heavily on it.
+
+A candidate's per-order signal is the strongest domain-specific TF-IDF among the n-grams containing
+it, **discounted by its intrinsic unigram specificity** — a stopword embedded in a domain phrase
+("the state machine") would otherwise borrow the phrase's specificity; the discount strips that so
+only intrinsically domain-specific words persist.
+
+**The differential across 3..7 is the discriminator:**
+- a **true domain term PERSISTS** — it keeps heading domain-specific n-grams as `n` grows, so its
+  signal stays high across orders → **confirmed-term** (the strongest un-stoplist proposal);
+- a **stylistic / noise word DECAYS or is discounted** — its longer n-grams become unique and
+  diffuse, or its signal was borrowed → **unconfirmed** (stays stoplisted).
+
+`make validate-kgram-differential` proves it: real terms (`set`/`class`/`state`) are confirmed
+across all orders; a concentrated-but-diffuse word (`and`) is not; a word that only borrows phrase
+specificity (`the`) is stripped by the unigram discount. This closes the two-stage design — stopword
+deviation (candidates) → k-gram differential (confirmation).