feat(vocab): learned spell-correction + user dictionary — context, not dictionary matching (task #13) - #263
Merged
Merged
Conversation
…t dictionary matching A static wordlist flags every domain term (epistemiclevel, srcos) as a misspelling. What counts as a correct word must be LEARNED from context. tools/learned_dictionary.py decides each unknown token from a count-based skip-gram word-sense predictor (PPMI + truncated SVD; Levy-Goldberg SGNS≈PPMI-SVD): LEARN a token that recurs with a coherent word-sense (a real term), CORRECT a rare token to the known word its SENSE matches (skip-gram cosine picks the target, not edit distance alone — so a near-spelled but different-sense token is not auto-corrected), leave the rest UNKNOWN (fail-closed). Every decision is a proposal, never a silent rewrite (human/superconscious admits it). validate-learned-dictionary teeth: epistemiclevel learned; reciept->receipt by sense; qwzptl unknown; a learned term is never auto-corrected away. Uses numpy (make recipe). Same doctrine as the stopword analysis + glossary currency: learned context predictors, not static membership tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The dictionary should be learned, not matched
A static wordlist flags every domain term (
epistemiclevel,srcos,governedloop) as a misspelling and corrects it away — the estate's own vocabulary treated as errors. So what counts as a correct word must be learned from context. For each UNKNOWN tokentools/learned_dictionary.pydecides from a skip-gram word-sense predictor:Predictor = count-based skip-gram: PPMI over a co-occurrence window + truncated SVD (Levy-Goldberg: SGNS factorises shifted PPMI — same word-sense family). No wordlist decides correctness; context does. Fail-closed: every decision is a proposal (add / correct-to), never a silent rewrite.
Teeth (
make validate-learned-dictionary)epistemiclevel(novel, recurs coherently) → learned, not corrected awayreciept→ corrected toreceiptby learned sense (carriessenseSim)qwzptl(garbage) → unknown, fail-closedSame doctrine as the stopword analysis (#260/#261) and glossary currency (#255): replace static membership tests with learned, context-driven predictors. Uses numpy (installed in the make recipe).