Skip to content

Extra lexicon from your own documents (mirror of #33, with QA hardening) - #40

Merged
Nathaniel-260 merged 3 commits into
mainfrom
mirror/pr33-extra-lexicon
Nov 16, 2026
Merged

Nathaniel-260 merged 3 commits into
mainfrom
mirror/pr33-extra-lexicon

Conversation

@Nathaniel-260

Copy link
Copy Markdown
Owner

Mirror of #33 (fork PR; CodeQL does not run on forks) plus QA hardening: size caps for docx/epub, linear html stripping, zero-token warnings, case-insensitive names, declared encodings, UI validation, README notes.

🤖 Generated with Claude Code

mynim and others added 3 commits November 15, 2026 21:13
`magiah extra --name NAME --from PATH...` counts the words of .txt
(UTF-8/UTF-16/Windows-1255), .html, .docx and .epub files (standard
library only; PDF is refused or skipped) into extra_<name>.pkl, tokenized
like the corpus. A scan that selects it with --extra-lexicon NAME never
flags a word it holds at least extra_min (3) times: suppression only, like
the whitelist, never a correction candidate. Remembered in run_config.json
like --whitelist; --no-extra-lexicon clears it. The web UI scan panel gets
a list field for it. With nothing selected the output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	README.md
#	magiah/cli.py
- zip/text size cap (200 MB per file, total over members) with a Hebrew
  error naming the file; linear-time html script/style/head/comment drop
- per-file warning for a file with no Hebrew words; no lexicon is written
  when the whole build has 0 tokens
- names are case-insensitive (lower-cased) so they cannot collide on Windows
- html/epub decoded as declared (xml encoding / meta charset), UTF-8 fallback
- web UI scans refuse a selected extra lexicon that does not exist up front
- README: extra_min masking, trusting the output folder (pickle), limits
- tests for each, incl. book command and UI book scan wiring

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@Nathaniel-260
Nathaniel-260 merged commit 76364cc into main Nov 16, 2026
13 checks passed
@Nathaniel-260
Nathaniel-260 deleted the mirror/pr33-extra-lexicon branch November 16, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants