Corpus-based typo detection for Hebrew and Aramaic texts — no dictionaries, no AI.
Magiah finds typing and OCR errors in large Hebrew/Aramaic corpora such as the Otzaria Torah library (7,000+ books, 400M+ words). It uses no external dictionaries and no machine learning — which is exactly why it works on rabbinic Hebrew, Aramaic, acronyms and abbreviations that no dictionary covers.
In a corpus of hundreds of millions of words, every real word — Hebrew, Aramaic, or abbreviation — appears many times. A typo is almost always a very rare word that is a small perturbation of a very frequent word. Magiah exploits a second property of large religious corpora: massive internal redundancy. The same phrases recur across books, editions and quotations, so a suspected correction can be verified against the corpus itself.
| Class | Example | Verification |
|---|---|---|
| Missing space | אתהשמים → את השמים |
every plausible segmentation (all 2-part splits, the best 3-part ones) is checked; a split is confirmed when its exact word sequence occurs with spaces in the corpus (at least split_obs_min times; a split with a 2-letter part also needs twice the count expected by chance, and a final letter marking the break needs one sighting), and among the confirmed splits the chosen one has the best score and the strongest association (observed vs. expected by chance). A split never displaces an edit-1 correction of the same word unless it is seen at least split_obs_min times and 10x above chance; otherwise it stays an alternative (alternatives and their counts are kept in report.db → split_alternatives) |
| Extra space | הימ נו → הימנו |
the joined form is frequent while a fragment is rare |
| Wrong / missing / extra / swapped letter | היעמנו → הימנו |
Damerau-Levenshtein distance 1 from a word ≥50× more frequent, boosted by a learned confusion matrix (see Calibration), then context-verified: the corrected word must appear next to the same neighboring words elsewhere |
| Final letter mid-word | שלוםעליכם |
deterministic rule of Hebrew orthography (ם ן ץ ף ך), applied up to struct_max occurrences (default 10), not only to rare words; abbreviations (geresh/gershayim) are exempt, and with no corpus word to propose it is reported as a suspicion with an empty suggestion |
| Non-final letter at word end | אדמ → אדם |
the final-form variant must be ≥50× more frequent |
| Abbreviation that lost its gershayim | רמבם → רמב"ם |
the quoted form must be a frequent abbreviation in the corpus |
Geresh for gershayim, or the reverse (--swapped-quotes 0 to disable) |
רמב'ם → רמב"ם |
the form with that one mark swapped must be a frequent abbreviation in the corpus, on the same bar as a lost gershayim; on by default, --swapped-quotes 0 turns it off |
| Book-specific OCR errors | ד↔ר confusion throughout one scanned book | per-book OCR profiles learned by calibration allow a sensitized rescan of books with a proven systematic confusion |
| Spelling variant (reported separately) | חבותיו ↔ חובותיו |
an extra/missing ו or י is usually ktiv male/chaser variation, not a typo — exported to its own file so you decide the policy |
| Deviation from Tanach (self-validating) | a verse quoted with one word off | only verified Bible books take part (see Tanach evidence below); a quotation must align with one verse position, and the verse reading must be backed by 2+ independent sources — it is stored as an alternative next to the detector's suggestion, and ranks higher only when the two agree |
- Which texts are editions. A book counts only when it sits under
תנ"ך → תורה/נביאים/כתובים, its title is one of the 39 books, and its
lines carry
<book>, <chapter>, <verse>heRefs.hasTeamimis not a criterion (siddurim, haftara collections and commentaries carry cantillation too). Each book row and eachbook_versionwith its own reading is an edition; editions are grouped into independent sources by provenance (the host ofversionSource), so several renderings of one upload count once. - Quotations. The context must align with a single verse position: at least 5 consecutive context tokens (one on each side of the word) and at least 2 distinctive ones (3+ letters, fewer than 200 occurrences in the Tanach), so a run of common words cannot pin a rabbinic phrase to a verse. Qere/ketiv slots, a single inner ו/י (male/haser) and readings backed by one source are labelled, never counted.
- Ranking.
tanach = 3(rank +4) only when the verse reading equals the detector's suggestion.tanach = 4means the verse reads otherwise: both candidates are kept (alternatives,tanach_reading) for a human to decide, with no bonus.tanach = 2marks findings of the old trigram heuristic: no bonus, flaggedtanach_legacyfor re-check in the review UI. - Edition errors need three independent sources. A reading is reported as
an edition error (
tanach_edition_variant) only when one source stands against at least two independent sources that agree. With two sources a disagreement is one witness against another and nobody can be outvoted, so it is reported astanach_edition_unresolved(rank 0, with areason: one_against_one, intra_source, plene, qere_ketiv, …), as are disagreements between renderings of the same source. The current Otzaria database has two independent Tanach sources (Wikisource's Miqra al pi ha-Masorah and tanach.us), so it yields unresolved rows only.
Every finding gets a confidence score; reports are sorted so genuine errors
concentrate at the top, and each class also gets a high-precision
*_verified.csv subset (context-verified or repeated in the same book).
Rarity is statistical evidence, not a verdict: each row also carries an
evidence column naming the support actually found — context (correction
seen beside the same neighbours corpus-wide), book_local (correction used
3+ times in the same book), split_observed (the spaced sequence occurs),
tanach, structural (a deterministic orthographic rule with a corpus suggestion), ocr_profile (the book's reviewed systematic confusion), suspicion (structural rule, nothing to propose) or none. A
single-book scan that looked only inside the book reports context_book /
split_observed_book instead.
Beyond the statistical verification above, Magiah automatically avoids the common failure modes of naive edit-distance flagging:
- Stacked prefixes —
דלאליעזר(= ד+ל+אליעזר) is legitimate morphology, so an "extra first letter" that is a valid prefix (ו ה ב ל מ ש כ ד א) is never proposed as an error; likewise an "extra last letter" that is a valid inflection suffix, and rare inflections of frequent stems. - Abbreviations — tokens with gershayim (
רמב"ם) and words followed by a geresh (וכוננ'= וכוננה) are recognized and skipped. - Foreign-language passages — lines where too many words are uncommon
(e.g. Judeo-Arabic in תפסיר רס"ג) are skipped entirely (
foreign_ratio). - Names — a rare word right after שם/רב/רבי/הרב… is likely a proper name
and is suppressed. The lexicon stage also counts how often each word stands
in a name position (
names.pkl); withnames_suggset (100 is a good value; default 0 = off) a word in the name (up to two words after the title) is still reported when the corpus never had it in a name position and its suggestion is a name seen there at leastnames_suggtimes — a known name misspelled. The option only ever adds findings, never hides one. - Book-local conventions — a "typo" repeated many times in the same book is probably that book's spelling convention; it is down-ranked, not flagged.
- Optional whitelist (
--whitelist words.txt, repeatable) — words listed are never flagged. Suppression only: the whitelist never creates findings, so an incomplete dictionary can't hurt Aramaic or rabbinic vocabulary. Works well with the hspell-derived inflected word lists from hebrew_wordlists (AGPL-licensed data, hence downloaded separately rather than bundled). - Optional extra lexicon (
magiah extra, then--extra-lexicon NAME; off by default) — a word count built from your own documents, for a book whose register the corpus does not cover (modern Hebrew, a technical field). Suppression only, like the whitelist; see Extra lexicons. - Your review decisions — words you rejected in the review interface are automatically excluded from every future scan.
1. lexicon count every word's frequency (one corpus pass)
2. detect flag rare words close to frequent ones,
verify missing-space splits against bigrams (one corpus pass)
3. locate find occurrences, detect extra spaces,
context-verify corrections, check Tanach quotes (two corpus passes)
4. report ranked CSVs + SQLite report, split per source
5. calibrate (optional, after reviewing findings) learn a letter-confusion
matrix and per-book OCR profiles from HUMAN-REVIEWED findings
(approved/fixed in the review UI), then rerun detect+locate
6. review local web interface for accepting/rejecting findings
On a 4-core machine the full pipeline over 417M tokens (5.9M lines, 7,293 books) runs in roughly 45 minutes using only the Python standard library.
Requires Python ≥3.9. Otzaria databases from schema 6 on store each line
as a zstd frame; Python 3.14+ decodes them with the standard library, and
older interpreters install zstandard automatically (pip install .).
pip install .
# or just run from the source tree:
python -m magiah --helpOn Windows always run with python -X utf8 -m magiah … to avoid console
encoding problems.
Otzaria library (auto-detects the seforim.db of the folder recorded in
%APPDATA%\otzaria\library_path.txt, falling back to
C:\ProgramData\otzaria\books\seforim.db). Both the old layout
(line.content) and schema 6 (line_content, zstd with a stored
dictionary) are read. Alternative editions in version_line are not
scanned and are counted as skipped. Every stage writes
coverage_<stage>.json, and a stage that could not read some rows
stops with an error instead of reporting a partial pass as complete (see
Unreadable rows below):
magiah all --otzaria --out resultsStages can be run separately (lexicon, detect, locate, report) — the
corpus source and thresholds are remembered in results/run_config.json, so
after tuning thresholds you can rerun from detect without recounting the
lexicon.
Updating the lexicon (lexicon --update). When the library changed only
a little (a new Otzaria release), magiah lexicon --update recounts only the
books that were added, changed or removed, and writes the same lexicon a full
magiah lexicon would. The first update counts everything and keeps each
book's counts in results/lexicon_books.pkl; later ones compare each book's
stamp (a seforim.db book: its contentHash and number of lines; a text file:
its size and modification time) and subtract/add only those books. Nothing
changed: nothing is written. The files it replaces are kept in
results/lexicon_prev/, and magiah lexicon --rollback swaps them back (run
it again to swap forward). magiah lexicon --status says what an update
would do. The side file is used only with the lexicon it was written with;
after a plain magiah lexicon, a corpus change or a partial lexicon, the
update counts everything again and says why. A plain --sqlite table has no
books to compare, so its update is a full count. As after magiah lexicon,
rerun detect and locate to bring the results up to date.
A failed stage never replaces its previous output, so after a failed run the
folder still holds the last good results. Every run therefore records itself
in results/run_state/ — when it started, the stage it reached, and how it
ended (done, failed, partial, cancelled). A run that never got to say how it
ended (killed, window closed, power cut) is recognized as interrupted: it
holds an OS file lock for as long as it lives. The review UI (magiah ui)
reads this record and, until a scan that rebuilds the results succeeds, shows
a warning above the findings that they come from the previous complete scan.
Two pipeline runs on one folder at a time are refused — they would overwrite
each other's files.
Unreadable rows (--allow-unread). seforim.db is used as downloaded,
and it may hold a row that cannot be read — a corrupt zstd frame, or a
line without its line_content row. By default any such row stops the
stage before it writes anything; the error lists the rows (book, reference,
line id) and says how to go on. If the database cannot be repaired, let the
scan skip them:
magiah all --otzaria --out results --allow-unread 3Nis an absolute number of distinct rows (a row met by several passes counts once). The default,0, skips none, and there is no "unlimited". With more than N unreadable rows the stage stops as before, so a database that degrades further is not let through.- With a folder of text files (
--textdir, or the library of a hybrid scan) an unreadable file counts as one row, and the review UI's notice speaks of files, not of the database. - Skipped rows are not scanned: errors in them are not found, and their
words are missing from the lexicon frequencies. Every output built this way
is marked partial, never complete:
coverage_<stage>.jsonkeepscomplete: falseand addsaccepted: true,allow_unread,unread_rows,unread_units(the row ids) andunread_refs(where the first 20 are); a stage built on a partial output of an earlier stage records it underinherited.report.dbcarries the same record, and the review UI shows a persistent notice listing the rows. Such a run still counts as completed: its results are the latest, so the warning about a scan that did not finish does not appear — unless a later scan fails, and then both do. - The option applies only to the run that names it and is not remembered in
run_config.json. A later run —detect,report, a single-book scan — refuses partial outputs unless it is given an--allow-unreadthat covers as many rows, and its error says which value to give. A complete run (for example on a repaired database, without the option) replaces the partial outputs and clears every mark. - A single-book scan follows the same rule: a book with at most N unreadable rows is scanned without them and marked partial; with more, it is refused.
- In the UI scan panel the option is the advanced setting «שורות לא קריאות מותרות»; it opens at 0 and is cleared after each scan.
- Deleting
coverage_*.jsonis not a workaround: on a fresh folder it unblocks nothing, and on an old one it passes stale results off as current.
Second, sharper pass (recommended, after reviewing some findings):
magiah calibrate --out results # learn confusion matrix + OCR book profiles
# from your reviewed decisions (ui_review.db)
magiah detect --out results
magiah locate --out results
magiah report --out resultsCalibration never learns from the machine's own unreviewed output by
default (that would reinforce its own guesses). Without reviewed findings it
learns nothing and says so; --calibrate-from-machine learns from
report.db anyway and labels the result machine_unreviewed in
calibration_meta.json. Learned files without that provenance (written by
older versions), or whose hash no longer matches it, are ignored, and a
calibrate run with nothing to learn disables the previous files. python -m magiah.eval --out results measures
the ranking's precision on reviewed books held out from calibration, per
error class and evidence kind, what each rank cut-off would lose, a
calibration table and the effect of removing each rank term. Reviewed
findings are mostly the top of the ranking, so --sample-out sample.csv
exports a stratified random sample (error class x rank band, with stratum
weights) to label, and --sample-labels sample.csv turns it into unbiased
estimates.
Rank signals measured against chance. report.db stores, per
occurrence, ctx_exp (the context pairs independence would predict) and
ctx_assoc (log10 of hits against it): the rank bonus grows with it, and
no hits cost a penalty only where hits were expected. Two optional signals
cost a longer context pass and are off by default: --rw-corpus 1
(rw_lr: the real-word language model, original vs. correction between the
same neighbours, the original's n-grams counted outside its own book) and
--par-corpus 1 (par_sugg / par_word / par_vote: the two-words-each-
side window seen with the correction vs. with the original in other books).
All of them move findings up or down, never in or out. With
--rank-model 1, calibrate fits a logistic ranking on the review labels
(rank_model.json, provenance-checked like the confusion matrix) once
there are --rank-model-min (200) of them; the reports then rank by its
log-odds and fill a prob column. With fewer labels the hand formula
stays.
Scan a single book (seconds instead of an hour):
magiah book --book-list "Bartenura" --out results # find a book
magiah book --book 593 --out results # a seforim.db book id
magiah book --book "books/halacha/x.txt" --out results # a library relpath
magiah book --book "C:\new book.txt" --out results # any .txt on diskScans one book against the already-built lexicon — the lexicon is never rebuilt, which is what makes this cost seconds. Every correction candidate is still scored against whole-corpus frequencies, so detection quality matches a full scan.
Findings are merged additively into ui_review.db. If the book was scanned
before, its rows are replaced in place (no duplicates) and your review
decisions on them are preserved; other books are untouched.
--book-verify-ctx additionally verifies each correction and each split
against the entire corpus (adds ~10 min); the findings are then the ones a
full scan gives for this book. Without it, verification is book-local and is
labelled so (ctx_scope/split_scope = book, evidence context_book /
split_observed_book).
--book-real-word (off by default) also looks for real-word errors: a
one-letter slip that produced another existing word, which no rarity rule can
see. Each common word of the book is weighed against its confusable
one-letter neighbours by how often each occurs between the same two words in
the corpus (bigrams and trigrams, a fixed substitution penalty), and is
reported as real_word only when the log10 odds against it reach
real_word_odds (5.49 — tuned on OCR output from outside this corpus, so
treat it as a starting point). A word the book uses at least real_word_rep
(3) times, and more often than the suggestion, is taken as the book's own
spelling. Tokens already reported as an extra space or an OCR-profile hit,
and words one or two words after a title (a name position), are not judged.
The findings carry evidence context_lm and are never counted as verified.
The counts come from the --book-verify-ctx pass (same pass, the
book's own rows excluded) and are kept in real_word/ for later book-scope
scans of the same book; a word whose neighbours changed since then is not
judged until the next corpus-scope scan. real_word/ holds one file per book
scanned this way and is never pruned; deleting it only means the next
book-scope scan needs a --book-verify-ctx rescan first.
--alternatives N (off by default) lists, for each edit-1 / spelling-variant
finding, up to N of the corrections the edit-1 rule accepted for the word
(alternatives in the finding; in the review UI, the drawer's "הצעות
חלופיות"). With --book-verify-ctx they are ranked per occurrence by the
same bigram/trigram model as --book-real-word, counted in the same pass
without the book's own rows (by: context); with no counts, no neighbour,
or no candidate ever seen beside these neighbours, they keep the detector's
order (by: frequency). The finding's suggestion does not change unless
--alt-rerank 1 is also given: the suggestion then becomes the alternative
the context ranks first, but only where the context prefers it to the
detector's own choice by at least 2 (log10 odds, alternatives.MARGIN);
otherwise the suggestion stays, and a finding whose candidates do not include
its suggestion gets no alternatives. The reranked suggestion is verified like
any other, so --alt-rerank also asks the context pass to count the
neighbour pairs of every candidate (on a synthetic book about 1.9x the
pairs of a plain scan; the pass reads the corpus once either way, so the
added time was within noise there).
--book-domain-max SHARE (off by default; 0.05 is a measured value) is a
domain gate for the whole book. The lexicon knows the language most of
the library is written in; a book in modern Hebrew, another language or another spelling
system is full of words it has never seen, and the rarity rules then flag
correct words. Before reporting, the scan counts the book's words of 3+
letters (abbreviations aside) that the lexicon has at most rare_max times
apart from the book itself. When more than SHARE of them are unknown (and
the book has at least book_domain_min_words, 200, such word occurrences
and at least book_domain_min_types, 100, different ones: a short text
that repeats a few words is not judged), the
findings that rest on a word being rare are left out — edit1_*,
spelling_variant, missing_space, lost_quotes and real_word — and
the scan says so in its log and in the result's domain entry (the share
and how many findings of each kind were left out). Final-letter findings,
the book's OCR profile and extra spaces stay. The foreign-line skip
(foreign_ratio) still works line by line inside any book; the gate is
the book-level decision on top of it. On the Otzaria library lexicon,
99% of the books of halacha, Talmud, responsa and Mishnah measure under 2%;
the 0.6% of library books above 5% are mostly Aramaic targums, plus the
Judeo-Arabic and Syriac translations and a few piyyut and midrash collections;
digital modern-Hebrew books measure 6-16%.
The gate refuses to apply, says so in the log and keeps every finding
(domain.refused is small or partial, out is false) when the lexicon
cannot serve as a yardstick: it counted fewer than book_domain_min_lexicon
(1,000,000) words in all, or it was built partial under --allow-unread.
A near-empty lexicon knows no word, so every ordinary book would look
foreign. A book read from the corpus (db/library) has its own
occurrences taken away from the lexicon's count first, the same rule as the
book-repeat and real-word checks; a file has none taken away. When the gate
acts, split alternatives of left-out missing_space findings are left out
too, and real_word.hints counts what is still shown.
Requires an existing
lexicon.pkl— i.e. one priormagiah lexicon(orall) run.
The corpus is the dictionary, so a book in modern Hebrew (or in a field the library does not cover) gets many ordinary words flagged as rare. An extra lexicon counts the words of documents you choose:
magiah extra --name modern --from "C:\modern texts" --out results # a folder (with subfolders)
magiah extra --name modern --from a.docx --from b.epub --out results # or files
magiah extra --out results # list the extra lexicons
magiah extra --remove modern --out results # delete one
magiah book --book "C:\new book.txt" --extra-lexicon modern --out resultsReadable: .txt (UTF-8, UTF-16 with a BOM, or Windows-1255), .html /
.htm / .xhtml, .docx, .epub — standard library only. PDF is not
read (that needs a third-party library): a PDF named on the command line is
refused, PDFs inside a folder are skipped and counted. The text is tokenized
exactly like the corpus. Building a name again replaces it; the lexicon is
extra_<name>.pkl in the output folder, with the list of what it was built
from.
A scan that selects it (--extra-lexicon NAME, repeatable; detect, all
and book) never flags a word the extra lexicon holds at least
--extra-min (3) times — fewer could be a typo in the documents. It is
suppression only, like --whitelist: it never creates a finding and is
never a correction candidate, and the real-word check is unchanged. The
selection is remembered in run_config.json like the whitelist;
--no-extra-lexicon clears it. In the web UI it is the "מילונים נוספים"
field of the scan panel (one name per line).
Select it for the books that need it, not always. Every modern word it
adds is a word an error can turn into without being seen: measured on
rabbinic text with injected errors, an extra lexicon built from 1,000 modern
Hebrew books hid 2.2% of the errors that were caught without it (one built
from 100 books: 0.2%), while on other modern books it cut the findings per
1,000 words from 29.3 to 4.3 (11.3 with 100 books). The larger the extra
lexicon, the more it hides.
Check a correction after the fact (magiah guard):
magiah guard --before page.ocr.txt --after page.fixed.txt --out resultsLines the two texts up word by word and judges every change by the corpus
frequencies of --out's lexicon: a common word (>= common_min) that
became a rare or unknown one (<= rare_max) is suspect — the corrector
(a person, an OCR post-corrector, a language model) probably broke a good
word; the reverse is supported; anything else neutral. A split, join,
insertion or deletion is one unit, judged by its least frequent word; but
inside such a unit, words that clearly correspond (the same word with a
letter or two changed) are paired first and judged one by one, so a real
common -> rare slip is not hidden by a split or join next to it.
Prints the suspect changes (--guard-all: every change) as tab-separated
line, verdict, old, new, f_old, f_new, then a count. Writes nothing (a
--out that does not exist is not created). The exit code is 0 even when
suspects are found: it is 1 only when the check could not run (a missing,
empty, corrupt or partial lexicon, an unreadable file, or two texts so
different and repetitive that comparing them would take unreasonably
long -- split them). Read the printed list, not the exit code. It
sees only one kind of damage: a common word turned into another common
word (a real-word slip) is neutral, and supported does not mean
correct.
Two more things to know:
extra_minmasking. Because a word counts as known once it appears--extra-min(3) times, a typo that appears three or more times in your own documents is hidden too: the extra lexicon cannot tell a repeated typo from a real word. Raise--extra-min(or clean the documents) if the documents are not carefully proofread. Names that differ only by case are one lexicon (Modernismodern), because Windows file names are case-insensitive.- Trust the output folder.
extra_<name>.pkl(likelexicon.pkland the other results) is a Python pickle, and loading a pickle can run code. Only use an output folder, and only copy extra lexicons, that come from you or from someone you trust.
Build limits: a file that yields no Hebrew words (for example a .docx
without word/document.xml) is reported with a warning and counted in the
summary; a build in which no file yields any word writes nothing. A .docx
or .epub whose text expands past 200 MB (a decompression bomb) stops the
build with an error naming the file; nothing is written. .epub and .html
content is read in the encoding it declares (encoding= / charset=),
falling back to UTF-8 and then Windows-1255.
Review the findings in your browser:
magiah review --out results # opens http://127.0.0.1:8765/Findings are shown one by one (filter by error type and source repository;
order by score or randomly shuffled). Keyboard: י accept, נ reject
this occurrence, ד reject the word everywhere, ת type your own
correction, ע ignore forever, space skip for now. Every decision is
stored immediately in decisions.db; the export button writes a
to_send/ folder with one ready-to-send CSV of approved fixes per source
repository, plus rejected_words.txt (which future scans use as a
whitelist automatically).
Magiah is not tied to Otzaria. Any Hebrew/Aramaic text collection works, and the whole pipeline needs just one thing: a way to iterate over your lines of text. There are three built-in adapters, from simplest to most capable.
Put your books in a directory tree of UTF-8 .txt files (subfolders are fine,
one book per file is best):
magiah all --textdir path/to/books --out resultsEach file is treated as a document, which enables book-local verification (is this correction already used elsewhere in the same book?) and per-book OCR profiles.
If your corpus is in any other format — Word documents, PDFs with a text
layer, JSON, CSV, HTML — the simplest route is to convert it to a folder of
.txt files. For example, from JSON:
import json, os, pathlib
data = json.load(open('mybooks.json', encoding='utf-8'))
out = pathlib.Path('books_txt'); out.mkdir(exist_ok=True)
for book in data:
safe = ''.join(c if c.isalnum() else '_' for c in book['title'])
(out / f"{safe}.txt").write_text('\n'.join(book['lines']), encoding='utf-8')Magiah's tokenizer already strips nikud, cantillation marks, HTML tags and Unicode presentation forms — you don't need to clean the text first.
If your corpus is already a SQLite database with one row per line/paragraph:
magiah all --sqlite mycorpus.db \
--table line --id-col id --text-col content --doc-col book_id \
--out results| Flag | Meaning | Default |
|---|---|---|
--table |
the table holding your text rows | line |
--id-col |
integer primary key of that table (used to chunk work across processes and to reference findings) | id |
--text-col |
the column with the actual text | content |
--doc-col |
optional but recommended: a column grouping rows into books/documents. Enables book-local verification and OCR profiles | none |
Any schema works as long as those columns exist. If your text lives in multiple tables, create a view:
CREATE VIEW all_lines AS
SELECT id, book_id, text AS content FROM mishna
UNION ALL
SELECT id + 1000000, book_id, text FROM talmud;then --table all_lines.
--otzaria is just a preset of Option 2 (table=line, id-col=id, text-col=content, doc-col=bookId) plus report enrichment that joins book
titles, references and source-repository names from Otzaria's schema. Use
--db path if your seforim.db lives elsewhere.
- The corpus source is saved in
results/run_config.jsonafter the first command — subsequent stages don't need the flags again. - Corpus size matters. The statistics need volume: below ~5M words,
raise thresholds (
--common-min 10 --ed1-ratio 20) and expect lower precision; below ~1M words the "corpus as dictionary" premise gets weak — consider adding more text of the same genre to the corpus (findings are reported per book anyway, so extra background text costs nothing). - The review interface and per-type reports work identically for every adapter. Otzaria-specific extras (per-source-repository folders, heRef references) simply stay empty for other corpora.
| File | Contents |
|---|---|
errors_<type>.csv |
one ranked CSV per error class: word, suggested correction, the aligned verse's reading (tanach_reading, when 2+ sources back it), confidence rank, context-verification hits, book, snippet |
errors_<type>_verified.csv |
high-precision subset (context-verified or correction already used in the same book) |
spelling_variants.csv |
ktiv male/chaser ו/י differences — policy decisions, not typos |
space_errors.csv |
extra-space findings |
tanach_matches.csv / tanach_edition_errors.csv |
quotations confirmed by 2+ independent sources / edition disagreements (evidence JSON: variant vs. unresolved, witnesses) |
by_source/<origin>/… |
the same reports split per source repository (Otzaria corpora) |
report.db |
everything as a queryable SQLite database |
coverage_<stage>.json |
what each stage actually read — a partial read stops the stage and is recorded here; for a partial output accepted under --allow-unread also accepted, the rows not read and where they are |
extra_<name>.pkl |
an extra lexicon built by magiah extra (word counts + what they were built from) |
run_state/ |
how the latest runs ended (scan.json, book.json) — read by the review UI |
to_send/ |
written by the review interface: approved fixes per source repository, ready to send upstream |
-- highest-confidence findings
SELECT * FROM occurrences_full ORDER BY score DESC LIMIT 100;
-- findings in one book
SELECT * FROM occurrences_full WHERE source LIKE '%תוספתא%';All thresholds are CLI flags (see magiah --help). The important ones:
| Flag | Default | Meaning |
|---|---|---|
--rare-max |
2 | a word is suspect if it occurs ≤N times in the whole corpus. Raise to 3–5 to find more errors at the cost of more false positives |
--common-min |
30 | minimum frequency for a proposed correction |
--ed1-ratio |
50 | how many times more frequent the correction must be |
--workers |
3 | parallel processes |
- Dictionaries fail on rabbinic Hebrew, Aramaic, Yiddish loanwords, acronyms, and the thousand spelling conventions of 1,000 years of printing. The corpus's own frequency distribution is the right dictionary for the corpus.
- LLMs over 400M words are slow, expensive, and hallucinate corrections. Statistical evidence (frequency ratios + bigram/context verification) is reproducible, explainable, and runs on a laptop.
MIT