Skip to content

feat(vocab): learned spell-correction + user dictionary — context, not dictionary matching (task #13) - #263

Merged
mdheller merged 1 commit into
mainfrom
feat/learned-dictionary
Aug 3, 2026
Merged

feat(vocab): learned spell-correction + user dictionary — context, not dictionary matching (task #13)#263
mdheller merged 1 commit into
mainfrom
feat/learned-dictionary

Conversation

@mdheller

@mdheller mdheller commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

The dictionary should be learned, not matched

A static wordlist flags every domain term (epistemiclevel, srcos, governedloop) as a misspelling and corrects it away — the estate's own vocabulary treated as errors. So what counts as a correct word must be learned from context. For each UNKNOWN token tools/learned_dictionary.py decides from a skip-gram word-sense predictor:

  • learn (add to user dictionary) — recurs with a COHERENT context (context windows cluster into one stable sense): a real term the dictionary hadn't seen.
  • correct (to a known word) — rare AND spelling-near AND sense-near (skip-gram cosine). Sense, not edit distance alone, picks the target — a token spelled near a known word but used in a different sense is not auto-corrected.
  • unknown — neither coherent nor sense-close: left for a human.

Predictor = count-based skip-gram: PPMI over a co-occurrence window + truncated SVD (Levy-Goldberg: SGNS factorises shifted PPMI — same word-sense family). No wordlist decides correctness; context does. Fail-closed: every decision is a proposal (add / correct-to), never a silent rewrite.

Teeth (make validate-learned-dictionary)

  • epistemiclevel (novel, recurs coherently) → learned, not corrected away
  • recieptcorrected to receipt by learned sense (carries senseSim)
  • qwzptl (garbage) → unknown, fail-closed
  • a learned term is never auto-corrected away

Same doctrine as the stopword analysis (#260/#261) and glossary currency (#255): replace static membership tests with learned, context-driven predictors. Uses numpy (installed in the make recipe).

…t dictionary matching

A static wordlist flags every domain term (epistemiclevel, srcos) as a misspelling. What counts as
a correct word must be LEARNED from context. tools/learned_dictionary.py decides each unknown token
from a count-based skip-gram word-sense predictor (PPMI + truncated SVD; Levy-Goldberg SGNS≈PPMI-SVD):
LEARN a token that recurs with a coherent word-sense (a real term), CORRECT a rare token to the
known word its SENSE matches (skip-gram cosine picks the target, not edit distance alone — so a
near-spelled but different-sense token is not auto-corrected), leave the rest UNKNOWN (fail-closed).
Every decision is a proposal, never a silent rewrite (human/superconscious admits it).

validate-learned-dictionary teeth: epistemiclevel learned; reciept->receipt by sense; qwzptl
unknown; a learned term is never auto-corrected away. Uses numpy (make recipe). Same doctrine as
the stopword analysis + glossary currency: learned context predictors, not static membership tests.
@mdheller
mdheller merged commit c55c53a into main Aug 3, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant