Repository navigation
Extra lexicon from your own documents (mirror of #33, with QA hardening) - #40
Merged
Merged
Conversation
`magiah extra --name NAME --from PATH...` counts the words of .txt (UTF-8/UTF-16/Windows-1255), .html, .docx and .epub files (standard library only; PDF is refused or skipped) into extra_<name>.pkl, tokenized like the corpus. A scan that selects it with --extra-lexicon NAME never flags a word it holds at least extra_min (3) times: suppression only, like the whitelist, never a correction candidate. Remembered in run_config.json like --whitelist; --no-extra-lexicon clears it. The web UI scan panel gets a list field for it. With nothing selected the output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # README.md # magiah/cli.py
- zip/text size cap (200 MB per file, total over members) with a Hebrew error naming the file; linear-time html script/style/head/comment drop - per-file warning for a file with no Hebrew words; no lexicon is written when the whole build has 0 tokens - names are case-insensitive (lower-cased) so they cannot collide on Windows - html/epub decoded as declared (xml encoding / meta charset), UTF-8 fallback - web UI scans refuse a selected extra lexicon that does not exist up front - README: extra_min masking, trusting the output folder (pickle), limits - tests for each, incl. book command and UI book scan wiring Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirror of #33 (fork PR; CodeQL does not run on forks) plus QA hardening: size caps for docx/epub, linear html stripping, zero-token warnings, case-insensitive names, declared encodings, UI validation, README notes.
🤖 Generated with Claude Code