feat(vocab): stopword deviation analysis — audit the words we DROP (task #13)

CHANGELOG.md

@@ -9,6 +9,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). Thi
 ## [Unreleased]
 
 ### Added
+- Stopword deviation analysis v0.1 — the dropped words are governed vocabulary too (task #13): `tools/stopword_analysis.py` audits the loop's stoplist ACROSS domains using two signals — cross-domain deviation (concentration) AND compositional density (repeated-collocation rate) — because frequency alone can't tell a domain term from a stylistic quirk. Surfaces `term-candidate` (concentrated + compositional = a domain term hiding in the stoplist, propose un-stoplisting) vs `stylistic` (concentrated by style only) vs `noise` (uniform). `validate-stopword-analysis` teeth: domain terms surfaced; a stylistically-concentrated word ('and') is NOT wrongly promoted; a uniform word ('the') is noise. `stopword-analysis-live` audits the shipped stoplist over specs/*.md. Compositional density is the bigram floor of the k-gram TF-IDF/LSA differential (orders 3..7) to follow.
 - Agreement test v0.1 — glossary relations vs the blast-radius graph (task #13, neurosymbolic): `tools/agreement_test.py` projects the glossary's composition relations (`has-a`/`has-member`) onto the estate via `alignment.estateBinding` and compares them to a consumed blast-radius/dependency graph (GBRG owns that graph). Fail-closed on OVERCLAIM (a declared dependency with no observed edge = governance hole); reports DRIFT (an observed edge no relation names) as a remediation candidate (a proposed `has-a` relation), like the vocab-currency loop's candidate terms. `validate-agreement` teeth: aligned agrees; overclaim refused; drift surfaced as candidate.
 - Glossary promotion v0.1 — the governed draft→approved alignment pass (task #13, closes the vocabulary lifecycle): `tools/promote_glossary_term.py` promotes a `draft` GlossaryTerm to `approved` ONLY when the 3-method alignment holds as a fail-closed MEET — capture (`ontologyClassRef`) ∧ vector-align (`vectorLink`, pinned to the sovereign nomic/768 space AND reciprocated by the named peer) ∧ implement (`estateBinding` entity/service/action). It recomputes the meet (never trusts a flag), refuses on any missing/non-reciprocal/off-space method (term stays draft, no governance hole), and its approved output PASSES the #250 alignment drift-guard by construction. `validate-glossary-promotion` teeth: promotes on full alignment; refuses missing-implement / non-reciprocal / off-space; promoted output passes the #250 guard.
 - Vocab-currency loop — LIVE dogfood on the estate's own vocab (`make vocab-currency-dogfood`): `tools/dogfood_vocab_currency.py` runs the governed loop over this repo's approved `GlossaryTerm`s (fixed/LSA) vs `specs/*.md` (open/LDA). Today only a handful of terms are approved against the full spec corpus, so the governed loop escalates-human (can't reach currency in its bound), writing real draft `GlossaryTerm` proposals to `build/vocab-currency-proposals/` — the artifacts ontogenesis ingests. Reports the outcome (does not gate on it, exit 0); Markdown-cleaned + generic-word stoplisted so domain terms surface. Enforcement stays on fixtures in `validate-vocab-currency-loop`.

Makefile

@@ -1,6 +1,6 @@
-.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement
+.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live
 
-validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement
+validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis
 	@echo "OK: validate"
 
 validate-ingestion-pipeline-examples:
@@ -22,6 +22,13 @@ validate-glossary-promotion:
 validate-agreement:
 	python3 tools/validate_agreement.py
 
+validate-stopword-analysis:
+	python3 tools/validate_stopword_analysis.py
+
+# On-demand audit of the shipped stoplist over specs/*.md (informational).
+stopword-analysis-live:
+	python3 tools/stopword_analysis.py --live
+
 validate-table-keys:
 	python3 -m pip install --user jsonschema referencing >/dev/null
 	python3 tools/validate_key_bipartite.py

fixtures/stopword-analysis/domains/formal.json

@@ -0,0 +1,14 @@
+{
+  "domain": "formal",
+  "documents": [
+    {
+      "text": "An empty set is a set with no elements. The empty set is unique. A finite state machine has a start state. The state machine transitions on input. A base class defines behaviour; a derived class extends the base class. Each required field is validated; a required field with no value fails. The set of states of the state machine forms a state space."
+    },
+    {
+      "text": "The empty set contains nothing. A state machine has states. The base class is abstract; the base class defines the interface. A required field must be present; the required field is checked. The state machine reads the set and updates the state."
+    },
+    {
+      "text": "Set theory studies the empty set and every set. The state machine and its state set. The base class and each required field. A required field maps to a state in the state machine."
+    }
+  ]
+}

fixtures/stopword-analysis/domains/narrative.json

@@ -0,0 +1,11 @@
+{
+  "domain": "narrative",
+  "documents": [
+    {
+      "text": "The dog and the cat ran across the field and the road, and the sun was warm and bright. The children laughed and played, and the day was long and the evening came and the stars and the moon rose over the hills and the valley and the quiet town below."
+    },
+    {
+      "text": "And so the story went, and the people gathered, and the music played, and the night was full of song and the wind and the rain and the memory of the summer and the sea."
+    }
+  ]
+}

fixtures/stopword-analysis/stoplist.json

@@ -0,0 +1,12 @@
+{
+  "stoplist": [
+    "the",
+    "and",
+    "set",
+    "class",
+    "state",
+    "required",
+    "field",
+    "with"
+  ]
+}

specs/stopword-analysis.md

@@ -0,0 +1,29 @@
+# Stopword deviation analysis (v0.1) — the words we DROP are governed vocabulary too
+
+The vocab-currency loop drops a fixed stoplist so domain terms surface. But that list is itself an
+ungoverned governance decision: a word that is filler in one domain ("set", "class", "state",
+"required", "value") is a real TERM in another (math, OOP, state machines, config). Dropping it
+universally erases signal where it matters. So we audit the dropped words the same way we audit the
+kept ones — frequency, connections, compositional density — ACROSS domains, using TWO signals
+because frequency alone lies:
+
+- **cross-domain deviation** (`concentration`) — is the word's relative-frequency mass concentrated
+  in a subset of domains?
+- **compositional density** — of the word's content-neighbour adjacencies in that domain, what
+  fraction are REPEATED collocations? A term recurs in fixed phrases ("empty set", "state machine");
+  a stylistic word ("and") glues arbitrary, unique content.
+
+Verdicts: **term-candidate** (concentrated AND compositional → a domain term hiding in the stoplist,
+propose un-stoplisting it there), **stylistic** (concentrated by style only → keep, but flagged),
+**noise** (uniform → true stopword everywhere). The stoplist becomes an auditable, per-domain
+artifact instead of a hard-coded assumption; term-candidates are a remediation signal like the
+currency loop's candidate terms and the agreement test's drift.
+
+`make validate-stopword-analysis` proves the two-signal discrimination on fixtures (real domain
+terms surfaced; a stylistically-concentrated word is NOT wrongly promoted — the trap frequency alone
+falls into; a uniform word is noise). `make stopword-analysis-live` audits the shipped stoplist over
+`specs/*.md`.
+
+Compositional density here is the **bigram floor** of the deeper follow-on: the **k-gram TF-IDF/LSA
+differential** over orders 3..7, which confirms a candidate by showing its signal GROWS with n-gram
+order (it participates in domain-specific higher-order collocations) rather than staying diffuse.