From 63d0f02ca0fc1fcfee8d37bfa5c9a13560464916 Mon Sep 17 00:00:00 2001 From: Michael Heller <21163552+mdheller@users.noreply.github.com> Date: Sun, 2 Aug 2026 21:01:27 -0400 Subject: [PATCH] =?UTF-8?q?feat(vocab):=20learned=20spell-correction=20+?= =?UTF-8?q?=20user=20dictionary=20=E2=80=94=20context,=20not=20dictionary?= =?UTF-8?q?=20matching?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A static wordlist flags every domain term (epistemiclevel, srcos) as a misspelling. What counts as a correct word must be LEARNED from context. tools/learned_dictionary.py decides each unknown token from a count-based skip-gram word-sense predictor (PPMI + truncated SVD; Levy-Goldberg SGNS≈PPMI-SVD): LEARN a token that recurs with a coherent word-sense (a real term), CORRECT a rare token to the known word its SENSE matches (skip-gram cosine picks the target, not edit distance alone — so a near-spelled but different-sense token is not auto-corrected), leave the rest UNKNOWN (fail-closed). Every decision is a proposal, never a silent rewrite (human/superconscious admits it). validate-learned-dictionary teeth: epistemiclevel learned; reciept->receipt by sense; qwzptl unknown; a learned term is never auto-corrected away. Uses numpy (make recipe). Same doctrine as the stopword analysis + glossary currency: learned context predictors, not static membership tests. --- CHANGELOG.md | 1 + Makefile | 8 +- fixtures/learned-dictionary/corpus/gov1.json | 17 ++ fixtures/learned-dictionary/corpus/gov2.json | 17 ++ specs/learned-dictionary.md | 24 +++ tools/learned_dictionary.py | 172 +++++++++++++++++++ tools/validate_learned_dictionary.py | 65 +++++++ 7 files changed, 302 insertions(+), 2 deletions(-) create mode 100644 fixtures/learned-dictionary/corpus/gov1.json create mode 100644 fixtures/learned-dictionary/corpus/gov2.json create mode 100644 specs/learned-dictionary.md create mode 100644 tools/learned_dictionary.py create mode 100644 tools/validate_learned_dictionary.py diff --git a/CHANGELOG.md b/CHANGELOG.md index d06e31d..ada160f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). Thi ## [Unreleased] ### Added +- Learned spell-correction + user dictionary v0.1 — from context, not dictionary matching (task #13): `tools/learned_dictionary.py` decides each UNKNOWN token from a count-based skip-gram word-sense predictor (PPMI + truncated SVD; SGNS≈PPMI-SVD) instead of a static wordlist. LEARN a token that recurs with a coherent word-sense (a real domain term the dictionary hadn't seen); CORRECT a rare token to the known word its SENSE matches (skip-gram cosine picks the target, not edit distance alone — a near-spelled but different-sense token is not auto-corrected); leave the rest UNKNOWN (fail-closed, human-admitted). Every decision is a proposal, never a silent rewrite. `validate-learned-dictionary` teeth: epistemiclevel learned, reciept->receipt by sense, qwzptl unknown, learned term never auto-corrected. Uses numpy. - Stopword deviation analysis v0.1 — the dropped words are governed vocabulary too (task #13): `tools/stopword_analysis.py` audits the loop's stoplist ACROSS domains using two signals — cross-domain deviation (concentration) AND compositional density (repeated-collocation rate) — because frequency alone can't tell a domain term from a stylistic quirk. Surfaces `term-candidate` (concentrated + compositional = a domain term hiding in the stoplist, propose un-stoplisting) vs `stylistic` (concentrated by style only) vs `noise` (uniform). `validate-stopword-analysis` teeth: domain terms surfaced; a stylistically-concentrated word ('and') is NOT wrongly promoted; a uniform word ('the') is noise. `stopword-analysis-live` audits the shipped stoplist over specs/*.md. Compositional density is the bigram floor of the k-gram TF-IDF/LSA differential (orders 3..7) to follow. - k-gram TF-IDF/LSA differential v0.1 — confirm stopword candidates by compositional scale (task #13): `tools/kgram_tfidf_differential.py` measures a candidate word's domain-specificity across n-gram ORDERS 3..7 (TF-IDF over domains + LSA/truncated-SVD top component) and takes the differential. Signal is discounted by intrinsic unigram specificity so a stopword embedded in a domain phrase ('the state machine') can't borrow the phrase's specificity. A true term PERSISTS across orders (confirmed-term = strongest un-stoplist proposal); a concentrated-but-diffuse word ('and') or a borrowed-specificity word ('the') stays unconfirmed. `validate-kgram-differential` teeth: set/class/state confirmed across 3..7; 'and' unconfirmed; 'the' stripped by the unigram discount. Closes the two-stage design (stopword deviation -> k-gram confirmation). Uses numpy. - Agreement test v0.1 — glossary relations vs the blast-radius graph (task #13, neurosymbolic): `tools/agreement_test.py` projects the glossary's composition relations (`has-a`/`has-member`) onto the estate via `alignment.estateBinding` and compares them to a consumed blast-radius/dependency graph (GBRG owns that graph). Fail-closed on OVERCLAIM (a declared dependency with no observed edge = governance hole); reports DRIFT (an observed edge no relation names) as a remediation candidate (a proposed `has-a` relation), like the vocab-currency loop's candidate terms. `validate-agreement` teeth: aligned agrees; overclaim refused; drift surfaced as candidate. diff --git a/Makefile b/Makefile index 43212bb..f55c34f 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ -.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live validate-kgram-differential +.PHONY: validate validate-resource-contract validate-measurement validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-ingestion-pipeline-examples validate-data-acquisition-examples validate-glossary-alignment-examples validate-data-class-examples validate-table-keys validate-dag-loop validate-vocab-currency-loop vocab-currency-dogfood validate-glossary-promotion validate-agreement validate-stopword-analysis stopword-analysis-live validate-kgram-differential validate-learned-dictionary -validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis validate-kgram-differential +validate: validate-data-class-examples validate-glossary-alignment-examples validate-data-acquisition-examples validate-ingestion-pipeline-examples validate-control-plane-examples validate-nlboot-examples validate-lattice-data-governai-examples validate-ops-history-examples validate-runtime-observability-examples validate-interpretability-examples validate-lifecycle-boundary-examples validate-svf-contracts validate-sync-cycle-receipts validate-onboarding-examples validate-runtime-causality-examples validate-agentic-os-examples validate-triparty-examples validate-labor-market-examples validate-supply-chain-risk-examples validate-reasoning-examples validate-mpcc-event-examples validate-knowledge-nugget-examples validate-semantic-action-examples validate-epistemic-kernel-examples validate-ab-update-examples validate-device-service-examples validate-duplicate-schema-ids validate-value-type validate-source-locator validate-sourceos-repo-manifest validate-mesh-action-registry validate-lawful-dispatch-receipt validate-architectural-building-block validate-agent-passport-examples validate-seam-definition-examples validate-agent-system-vocabulary validate-genesis-inception-examples validate-measurement validate-world-model-examples validate-eval-item-examples validate-resource-contract validate-table-keys validate-dag-loop validate-vocab-currency-loop validate-glossary-promotion validate-agreement validate-stopword-analysis validate-kgram-differential validate-learned-dictionary @echo "OK: validate" validate-ingestion-pipeline-examples: @@ -26,6 +26,10 @@ validate-kgram-differential: python3 -m pip install --user numpy >/dev/null python3 tools/validate_kgram_differential.py +validate-learned-dictionary: + python3 -m pip install --user numpy >/dev/null + python3 tools/validate_learned_dictionary.py + validate-stopword-analysis: python3 tools/validate_stopword_analysis.py diff --git a/fixtures/learned-dictionary/corpus/gov1.json b/fixtures/learned-dictionary/corpus/gov1.json new file mode 100644 index 0000000..3597e9d --- /dev/null +++ b/fixtures/learned-dictionary/corpus/gov1.json @@ -0,0 +1,17 @@ +{ + "domain": "gov1", + "documents": [ + { + "text": "the sealed receipt recorded the governance decision and the receipt was signed" + }, + { + "text": "the release gate verified the attestation and the gate was signed and verified" + }, + { + "text": "the receipt carried an epistemiclevel proved not speculative on the signed attestation" + }, + { + "text": "a signed attestation kept the epistemiclevel proved the receipt sealed the governance decision" + } + ] +} \ No newline at end of file diff --git a/fixtures/learned-dictionary/corpus/gov2.json b/fixtures/learned-dictionary/corpus/gov2.json new file mode 100644 index 0000000..c4a4323 --- /dev/null +++ b/fixtures/learned-dictionary/corpus/gov2.json @@ -0,0 +1,17 @@ +{ + "domain": "gov2", + "documents": [ + { + "text": "the gate verified the signed attestation the receipt sealed the governance decision proved" + }, + { + "text": "the sealed reciept recorded the governance decision proved not speculative" + }, + { + "text": "the receipt proved the epistemiclevel speculative until the attestation was signed and verified" + }, + { + "text": "the governance decision recorded qwzptl and the gate verified the signed receipt" + } + ] +} \ No newline at end of file diff --git a/specs/learned-dictionary.md b/specs/learned-dictionary.md new file mode 100644 index 0000000..565d29b --- /dev/null +++ b/specs/learned-dictionary.md @@ -0,0 +1,24 @@ +# Learned spell-correction + user dictionary (v0.1) — from context, not dictionary matching + +A static wordlist flags every domain term (`epistemiclevel`, `srcos`, `governedloop`) as a +misspelling and corrects it away — the estate's own vocabulary treated as errors. So what counts as +a "correct" word must be LEARNED from context, not matched against a list. For each UNKNOWN token +`tools/learned_dictionary.py` decides from a skip-gram word-sense predictor: + +- **learn** (add to the user dictionary) — the token RECURS with a COHERENT context (its context + windows cluster into one stable word-sense); a real term the dictionary simply hadn't seen. +- **correct** (to a known word) — the token is RARE and both spelling-near (small edit distance) + AND sense-near (high skip-gram cosine) to a known word. **Sense — not edit distance alone — + picks the target**, so a token spelled near a known word but used in a different sense is not + auto-corrected. +- **unknown** — neither coherent enough to learn nor sense-close to a known word: left for a human. + +The predictor is a count-based skip-gram — PPMI over a co-occurrence window + truncated SVD +(Levy-Goldberg: SGNS factorises shifted PPMI, so this is the same word-sense family). No wordlist +decides correctness; context does. **Fail-closed:** every decision is a PROPOSAL (add / correct-to), +never a silent rewrite — a human or the superconscious admits it. + +`make validate-learned-dictionary` proves it: a novel domain term (`epistemiclevel`) is learned, a +typo (`reciept`) is corrected to `receipt` by sense, garbage (`qwzptl`) is left unknown, and a +learned term is never auto-corrected away. Same doctrine as the stopword analysis and glossary +currency: replace static membership tests with learned, context-driven predictors. diff --git a/tools/learned_dictionary.py b/tools/learned_dictionary.py new file mode 100644 index 0000000..d9e767d --- /dev/null +++ b/tools/learned_dictionary.py @@ -0,0 +1,172 @@ +#!/usr/bin/env python3 +"""Learned spell-correction + user dictionary — from context, NOT dictionary matching (task #13). + +A static wordlist flags every domain term ("epistemiclevel", "srcos", "governedloop") as a +misspelling and "corrects" it away. The estate's own vocabulary can't be a list of known-good +strings — it must be LEARNED from context. So for each UNKNOWN token this decides, from a skip-gram +word-sense predictor rather than membership in a dictionary: + + * LEARN (add to the user dictionary) — the token RECURS with a COHERENT context (its context + windows cluster = one stable word-sense), and it is not merely a spelling variant of a known + word. A real term the dictionary simply hadn't seen yet. + * CORRECT (to a known word w') — the token is RARE and both close in spelling (small edit + distance) AND close in learned SENSE (high cosine between its skip-gram vector and w') to a + known word. Sense — not edit distance alone — picks the target, so a token spelled near a known + word but used in a DIFFERENT sense is NOT auto-corrected. + * UNKNOWN — neither coherent enough to learn nor sense-close to a known word (leave for a human). + +The predictor is a count-based skip-gram: PPMI over a co-occurrence window, then truncated SVD +(Levy-Goldberg: SGNS factorises shifted PPMI, so PPMI-SVD is the same word-sense family). No wordlist +decides correctness — context does. Fail-closed: every decision is a PROPOSAL (add / correct-to), +never a silent rewrite; a human or the superconscious admits it. +""" +from __future__ import annotations + +import argparse +import json +import re +from pathlib import Path + +import numpy as np + +ROOT = Path(__file__).resolve().parents[1] +WINDOW = 3 +EMBED_DIM = 12 +KNOWN_MIN_COUNT = 4 # a word seen this often is treated as "known" (the learned base lexicon) +# A base common-English lexicon (in production this is a frequency list). It is NOT the domain +# dictionary — it just keeps generic English out of consideration so the learner judges DOMAIN +# unknowns. The point of the tool is that DOMAIN terms need no such list; they are learned. +COMMON_ENGLISH = { + "the", "and", "was", "were", "not", "but", "for", "with", "from", "that", "this", "then", + "until", "kept", "carried", "recorded", "held", "left", "over", "each", "any", "are", "has", + "had", "have", "been", "will", "can", "may", "its", "our", "their", "when", "where", "which", +} +LEARN_MIN_DF = 2 # a term must recur across >= this many documents to be learned +COHERENCE_LEARN = 0.35 # context-window coherence needed to call it a stable sense +EDIT_MAX = 2 # max edit distance to consider a token a spelling variant +SENSE_SIM = 0.30 # min skip-gram cosine to a known word to accept a correction target + + +def tokenize(text: str) -> list[str]: + return [t for t in re.findall(r"[a-z][a-z0-9]+", text.lower()) if len(t) >= 3] + + +def edit_distance(a: str, b: str) -> int: + if abs(len(a) - len(b)) > EDIT_MAX: + return EDIT_MAX + 1 + prev = list(range(len(b) + 1)) + for i, ca in enumerate(a, 1): + cur = [i] + for j, cb in enumerate(b, 1): + cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb))) + prev = cur + return prev[-1] + + +def skipgram_embeddings(docs: list[list[str]]) -> tuple[dict[str, int], np.ndarray]: + vocab = sorted({t for d in docs for t in d}) + idx = {w: i for i, w in enumerate(vocab)} + n = len(vocab) + C = np.zeros((n, n)) + for d in docs: + for i, w in enumerate(d): + for j in range(max(0, i - WINDOW), min(len(d), i + WINDOW + 1)): + if j != i: + C[idx[w], idx[d[j]]] += 1 + total = C.sum() or 1.0 + row = C.sum(axis=1, keepdims=True) + col = C.sum(axis=0, keepdims=True) + with np.errstate(divide="ignore", invalid="ignore"): + ppmi = np.maximum(0.0, np.log((C * total) / (row * col + 1e-12) + 1e-12)) + k = min(EMBED_DIM, n) + U, S, _ = np.linalg.svd(ppmi, full_matrices=False) + emb = U[:, :k] * np.sqrt(S[:k]) + norms = np.linalg.norm(emb, axis=1, keepdims=True) + emb = emb / np.where(norms == 0, 1.0, norms) # unit vectors -> dot product = cosine + return idx, emb + + +def _context_coherence(docs: list[list[str]], w: str, idx: dict, emb: np.ndarray) -> float: + # Each occurrence of w has a context vector (mean of its window's word vectors). Coherence = + # how tightly those context vectors agree (a real term has one stable sense -> tight cluster). + ctx_vecs = [] + for d in docs: + for i, tok in enumerate(d): + if tok != w: + continue + neigh = [emb[idx[d[j]]] for j in range(max(0, i - WINDOW), min(len(d), i + WINDOW + 1)) + if j != i] + if neigh: + v = np.mean(neigh, axis=0) + nv = np.linalg.norm(v) + if nv: + ctx_vecs.append(v / nv) + if len(ctx_vecs) < 2: + return 0.0 + centroid = np.mean(ctx_vecs, axis=0) + centroid /= (np.linalg.norm(centroid) or 1.0) + return float(np.mean([c @ centroid for c in ctx_vecs])) + + +def learn(docs: list[list[str]]) -> dict: + idx, emb = skipgram_embeddings(docs) + counts = {w: sum(d.count(w) for d in docs) for w in idx} + df = {w: sum(1 for d in docs if w in d) for w in idx} + known = {w for w, c in counts.items() if c >= KNOWN_MIN_COUNT} | (COMMON_ENGLISH & set(idx)) + + decisions = [] + for w in sorted(idx): + if w in known: + continue + # nearest known word by SENSE among the spelling-near ones (sense, not edit distance, decides) + near = [(kw, edit_distance(w, kw)) for kw in known if edit_distance(w, kw) <= EDIT_MAX] + best_kw, best_sim = None, -1.0 + for kw, _ed in near: + sim = float(emb[idx[w]] @ emb[idx[kw]]) + if sim > best_sim: + best_kw, best_sim = kw, sim + coherence = _context_coherence(docs, w, idx, emb) + + if best_kw is not None and best_sim >= SENSE_SIM and df[w] < LEARN_MIN_DF: + decisions.append({"token": w, "decision": "correct", "correctTo": best_kw, + "senseSim": round(best_sim, 3), "editDistance": edit_distance(w, best_kw), + "why": "rare + spelling-near + sense matches a known word"}) + elif df[w] >= LEARN_MIN_DF and coherence >= COHERENCE_LEARN: + decisions.append({"token": w, "decision": "learn", "coherence": round(coherence, 3), + "documentFreq": df[w], + "why": "recurs with a coherent word-sense — a real term, add to user dictionary"}) + else: + decisions.append({"token": w, "decision": "unknown", "coherence": round(coherence, 3), + "documentFreq": df[w], "why": "neither coherent enough nor sense-close to a known word"}) + return { + "knownCount": len(known), + "learn": [d for d in decisions if d["decision"] == "learn"], + "correct": [d for d in decisions if d["decision"] == "correct"], + "unknown": [d for d in decisions if d["decision"] == "unknown"], + } + + +def _load_fixture() -> list[list[str]]: + FIX = ROOT / "fixtures" / "learned-dictionary" + docs = [] + for p in sorted((FIX / "corpus").glob("*.json")): + for d in json.loads(p.read_text())["documents"]: + docs.append(tokenize(d["text"])) + return docs + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--corpus-dir") + args = ap.parse_args() + if args.corpus_dir: + docs = [tokenize(d["text"]) for p in sorted(Path(args.corpus_dir).glob("*.json")) + for d in json.loads(Path(p).read_text())["documents"]] + else: + docs = _load_fixture() + print(json.dumps(learn(docs), indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/validate_learned_dictionary.py b/tools/validate_learned_dictionary.py new file mode 100644 index 0000000..004a399 --- /dev/null +++ b/tools/validate_learned_dictionary.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +"""CI teeth for the learned spell-correction + user dictionary (task #13). + +Asserts the decisions are made from the LEARNED skip-gram word-sense predictor, not a wordlist: +an unknown token that recurs coherently is LEARNED (not corrected away), a rare token whose SENSE +(not just spelling) matches a known word is CORRECTED to it, and a garbage token is left UNKNOWN +(fail-closed — never silently rewritten). +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import learned_dictionary as L # noqa: E402 + +FAILURES: list[str] = [] +CHECKS: dict[str, bool] = {} + + +def main() -> int: + r = L.learn(L._load_fixture()) + learn = {d["token"] for d in r["learn"]} + correct = {d["token"]: d for d in r["correct"]} + unknown = {d["token"] for d in r["unknown"]} + + # 1. A novel domain term the dictionary never saw is LEARNED (not corrected away). + if "epistemiclevel" not in learn: + FAILURES.append("'epistemiclevel' recurs coherently and must be LEARNED, not corrected/dropped") + else: + CHECKS["novel-term:learned"] = True + + # 2. A typo is CORRECTED to the known word its SENSE matches (skip-gram cosine drives the target). + c = correct.get("reciept") + if not c or c["correctTo"] != "receipt": + FAILURES.append("'reciept' must be corrected to 'receipt' by learned sense") + elif "senseSim" not in c: + FAILURES.append("a correction must be sense-driven (carry senseSim), not edit-distance-only") + else: + CHECKS["typo:corrected-by-sense"] = True + + # 3. A garbage token is left UNKNOWN — fail-closed, never silently rewritten. + if "qwzptl" not in unknown: + FAILURES.append("garbage 'qwzptl' must be UNKNOWN (not learned, not corrected)") + else: + CHECKS["garbage:unknown-fail-closed"] = True + + # 4. The learned term is NOT in the correct set (a real term must not be auto-corrected away). + if "epistemiclevel" in correct: + FAILURES.append("a learned term must never be auto-corrected away") + else: + CHECKS["learned-term:not-corrected"] = True + + for m in FAILURES: + print(f"FAIL: {m}", file=sys.stderr) + ok = not FAILURES and all(CHECKS.values()) + print(json.dumps({"ok": ok, "checks": CHECKS, + "learn": sorted(learn), "correct": {k: v["correctTo"] for k, v in correct.items()}, + "unknown": sorted(unknown)}, indent=2, sort_keys=True)) + return 0 if ok else 1 + + +if __name__ == "__main__": + raise SystemExit(main())