Skip to content
Nathaniel-260Public

About

Corpus-based typo & OCR error detection for Hebrew/Aramaic texts - no dictionaries, no AI

Resources

Security policy

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

Magiah (מַגִּיהַּ)

Corpus-based typo detection for Hebrew and Aramaic texts — no dictionaries, no AI.

עברית

Magiah finds typing and OCR errors in large Hebrew/Aramaic corpora such as the Otzaria Torah library (7,000+ books, 400M+ words). It uses no external dictionaries and no machine learning — which is exactly why it works on rabbinic Hebrew, Aramaic, acronyms and abbreviations that no dictionary covers.

The idea: the corpus is its own dictionary

In a corpus of hundreds of millions of words, every real word — Hebrew, Aramaic, or abbreviation — appears many times. A typo is almost always a very rare word that is a small perturbation of a very frequent word. Magiah exploits a second property of large religious corpora: massive internal redundancy. The same phrases recur across books, editions and quotations, so a suspected correction can be verified against the corpus itself.

Error classes detected

Class Example Verification
Missing space אתהשמים → את השמים every plausible segmentation (all 2-part splits, the best 3-part ones) is checked; a split is confirmed when its exact word sequence occurs with spaces in the corpus (at least split_obs_min times; a split with a 2-letter part also needs twice the count expected by chance, and a final letter marking the break needs one sighting), and among the confirmed splits the chosen one has the best score and the strongest association (observed vs. expected by chance). A split never displaces an edit-1 correction of the same word unless it is seen at least split_obs_min times and 10x above chance; otherwise it stays an alternative (alternatives and their counts are kept in report.db → split_alternatives)
Extra space הימ נו → הימנו the joined form is frequent while a fragment is rare
Wrong / missing / extra / swapped letter היעמנו → הימנו Damerau-Levenshtein distance 1 from a word ≥50× more frequent, boosted by a learned confusion matrix (see Calibration), then context-verified: the corrected word must appear next to the same neighboring words elsewhere
Final letter mid-word שלוםעליכם deterministic rule of Hebrew orthography (ם ן ץ ף ך), applied up to struct_max occurrences (default 10), not only to rare words; abbreviations (geresh/gershayim) are exempt, and with no corpus word to propose it is reported as a suspicion with an empty suggestion
Non-final letter at word end אדמ → אדם the final-form variant must be ≥50× more frequent
Abbreviation that lost its gershayim רמבם → רמב"ם the quoted form must be a frequent abbreviation in the corpus
Geresh for gershayim, or the reverse (--swapped-quotes 0 to disable) רמב'ם → רמב"ם the form with that one mark swapped must be a frequent abbreviation in the corpus, on the same bar as a lost gershayim; on by default, --swapped-quotes 0 turns it off
Book-specific OCR errors ד↔ר confusion throughout one scanned book per-book OCR profiles learned by calibration allow a sensitized rescan of books with a proven systematic confusion
Spelling variant (reported separately) חבותיו ↔ חובותיו an extra/missing ו or י is usually ktiv male/chaser variation, not a typo — exported to its own file so you decide the policy
Deviation from Tanach (self-validating) a verse quoted with one word off only verified Bible books take part (see Tanach evidence below); a quotation must align with one verse position, and the verse reading must be backed by 2+ independent sources — it is stored as an alternative next to the detector's suggestion, and ranks higher only when the two agree

Tanach evidence

  • Which texts are editions. A book counts only when it sits under תנ"ך → תורה/נביאים/כתובים, its title is one of the 39 books, and its lines carry <book>, <chapter>, <verse> heRefs. hasTeamim is not a criterion (siddurim, haftara collections and commentaries carry cantillation too). Each book row and each book_version with its own reading is an edition; editions are grouped into independent sources by provenance (the host of versionSource), so several renderings of one upload count once.
  • Quotations. The context must align with a single verse position: at least 5 consecutive context tokens (one on each side of the word) and at least 2 distinctive ones (3+ letters, fewer than 200 occurrences in the Tanach), so a run of common words cannot pin a rabbinic phrase to a verse. Qere/ketiv slots, a single inner ו/י (male/haser) and readings backed by one source are labelled, never counted.
  • Ranking. tanach = 3 (rank +4) only when the verse reading equals the detector's suggestion. tanach = 4 means the verse reads otherwise: both candidates are kept (alternatives, tanach_reading) for a human to decide, with no bonus. tanach = 2 marks findings of the old trigram heuristic: no bonus, flagged tanach_legacy for re-check in the review UI.
  • Edition errors need three independent sources. A reading is reported as an edition error (tanach_edition_variant) only when one source stands against at least two independent sources that agree. With two sources a disagreement is one witness against another and nobody can be outvoted, so it is reported as tanach_edition_unresolved (rank 0, with a reason: one_against_one, intra_source, plene, qere_ketiv, …), as are disagreements between renderings of the same source. The current Otzaria database has two independent Tanach sources (Wikisource's Miqra al pi ha-Masorah and tanach.us), so it yields unresolved rows only.

Every finding gets a confidence score; reports are sorted so genuine errors concentrate at the top, and each class also gets a high-precision *_verified.csv subset (context-verified or repeated in the same book). Rarity is statistical evidence, not a verdict: each row also carries an evidence column naming the support actually found — context (correction seen beside the same neighbours corpus-wide), book_local (correction used 3+ times in the same book), split_observed (the spaced sequence occurs), tanach, structural (a deterministic orthographic rule with a corpus suggestion), ocr_profile (the book's reviewed systematic confusion), suspicion (structural rule, nothing to propose) or none. A single-book scan that looked only inside the book reports context_book / split_observed_book instead.

False-positive suppression

Beyond the statistical verification above, Magiah automatically avoids the common failure modes of naive edit-distance flagging:

  • Stacked prefixes — דלאליעזר (= ד+ל+אליעזר) is legitimate morphology, so an "extra first letter" that is a valid prefix (ו ה ב ל מ ש כ ד א) is never proposed as an error; likewise an "extra last letter" that is a valid inflection suffix, and rare inflections of frequent stems.
  • Abbreviations — tokens with gershayim (רמב"ם) and words followed by a geresh (וכוננ' = וכוננה) are recognized and skipped.
  • Foreign-language passages — lines where too many words are uncommon (e.g. Judeo-Arabic in תפסיר רס"ג) are skipped entirely (foreign_ratio).
  • Names — a rare word right after שם/רב/רבי/הרב… is likely a proper name and is suppressed. The lexicon stage also counts how often each word stands in a name position (names.pkl); with names_sugg set (100 is a good value; default 0 = off) a word in the name (up to two words after the title) is still reported when the corpus never had it in a name position and its suggestion is a name seen there at least names_sugg times — a known name misspelled. The option only ever adds findings, never hides one.
  • Book-local conventions — a "typo" repeated many times in the same book is probably that book's spelling convention; it is down-ranked, not flagged.
  • Optional whitelist (--whitelist words.txt, repeatable) — words listed are never flagged. Suppression only: the whitelist never creates findings, so an incomplete dictionary can't hurt Aramaic or rabbinic vocabulary. Works well with the hspell-derived inflected word lists from hebrew_wordlists (AGPL-licensed data, hence downloaded separately rather than bundled).
  • Optional extra lexicon (magiah extra, then --extra-lexicon NAME; off by default) — a word count built from your own documents, for a book whose register the corpus does not cover (modern Hebrew, a technical field). Suppression only, like the whitelist; see Extra lexicons.
  • Your review decisions — words you rejected in the review interface are automatically excluded from every future scan.

Pipeline

1. lexicon    count every word's frequency                    (one corpus pass)
2. detect     flag rare words close to frequent ones,
              verify missing-space splits against bigrams     (one corpus pass)
3. locate     find occurrences, detect extra spaces,
              context-verify corrections, check Tanach quotes (two corpus passes)
4. report     ranked CSVs + SQLite report, split per source
5. calibrate  (optional, after reviewing findings) learn a letter-confusion
              matrix and per-book OCR profiles from HUMAN-REVIEWED findings
              (approved/fixed in the review UI), then rerun detect+locate
6. review     local web interface for accepting/rejecting findings

On a 4-core machine the full pipeline over 417M tokens (5.9M lines, 7,293 books) runs in roughly 45 minutes using only the Python standard library.

Installation

Requires Python ≥3.9. Otzaria databases from schema 6 on store each line as a zstd frame; Python 3.14+ decodes them with the standard library, and older interpreters install zstandard automatically (pip install .).

pip install .
# or just run from the source tree:
python -m magiah --help

On Windows always run with python -X utf8 -m magiah … to avoid console encoding problems.

Usage

Otzaria library (auto-detects the seforim.db of the folder recorded in %APPDATA%\otzaria\library_path.txt, falling back to C:\ProgramData\otzaria\books\seforim.db). Both the old layout (line.content) and schema 6 (line_content, zstd with a stored dictionary) are read. Alternative editions in version_line are not scanned and are counted as skipped. Every stage writes coverage_<stage>.json, and a stage that could not read some rows stops with an error instead of reporting a partial pass as complete (see Unreadable rows below):

magiah all --otzaria --out results

Stages can be run separately (lexicon, detect, locate, report) — the corpus source and thresholds are remembered in results/run_config.json, so after tuning thresholds you can rerun from detect without recounting the lexicon.

Updating the lexicon (lexicon --update). When the library changed only a little (a new Otzaria release), magiah lexicon --update recounts only the books that were added, changed or removed, and writes the same lexicon a full magiah lexicon would. The first update counts everything and keeps each book's counts in results/lexicon_books.pkl; later ones compare each book's stamp (a seforim.db book: its contentHash and number of lines; a text file: its size and modification time) and subtract/add only those books. Nothing changed: nothing is written. The files it replaces are kept in results/lexicon_prev/, and magiah lexicon --rollback swaps them back (run it again to swap forward). magiah lexicon --status says what an update would do. The side file is used only with the lexicon it was written with; after a plain magiah lexicon, a corpus change or a partial lexicon, the update counts everything again and says why. A plain --sqlite table has no books to compare, so its update is a full count. As after magiah lexicon, rerun detect and locate to bring the results up to date.

A failed stage never replaces its previous output, so after a failed run the folder still holds the last good results. Every run therefore records itself in results/run_state/ — when it started, the stage it reached, and how it ended (done, failed, partial, cancelled). A run that never got to say how it ended (killed, window closed, power cut) is recognized as interrupted: it holds an OS file lock for as long as it lives. The review UI (magiah ui) reads this record and, until a scan that rebuilds the results succeeds, shows a warning above the findings that they come from the previous complete scan. Two pipeline runs on one folder at a time are refused — they would overwrite each other's files.

Unreadable rows (--allow-unread). seforim.db is used as downloaded, and it may hold a row that cannot be read — a corrupt zstd frame, or a line without its line_content row. By default any such row stops the stage before it writes anything; the error lists the rows (book, reference, line id) and says how to go on. If the database cannot be repaired, let the scan skip them:

magiah all --otzaria --out results --allow-unread 3
  • N is an absolute number of distinct rows (a row met by several passes counts once). The default, 0, skips none, and there is no "unlimited". With more than N unreadable rows the stage stops as before, so a database that degrades further is not let through.
  • With a folder of text files (--textdir, or the library of a hybrid scan) an unreadable file counts as one row, and the review UI's notice speaks of files, not of the database.
  • Skipped rows are not scanned: errors in them are not found, and their words are missing from the lexicon frequencies. Every output built this way is marked partial, never complete: coverage_<stage>.json keeps complete: false and adds accepted: true, allow_unread, unread_rows, unread_units (the row ids) and unread_refs (where the first 20 are); a stage built on a partial output of an earlier stage records it under inherited. report.db carries the same record, and the review UI shows a persistent notice listing the rows. Such a run still counts as completed: its results are the latest, so the warning about a scan that did not finish does not appear — unless a later scan fails, and then both do.
  • The option applies only to the run that names it and is not remembered in run_config.json. A later run — detect, report, a single-book scan — refuses partial outputs unless it is given an --allow-unread that covers as many rows, and its error says which value to give. A complete run (for example on a repaired database, without the option) replaces the partial outputs and clears every mark.
  • A single-book scan follows the same rule: a book with at most N unreadable rows is scanned without them and marked partial; with more, it is refused.
  • In the UI scan panel the option is the advanced setting «שורות לא קריאות מותרות»; it opens at 0 and is cleared after each scan.
  • Deleting coverage_*.json is not a workaround: on a fresh folder it unblocks nothing, and on an old one it passes stale results off as current.

Second, sharper pass (recommended, after reviewing some findings):

magiah calibrate --out results   # learn confusion matrix + OCR book profiles
                                 # from your reviewed decisions (ui_review.db)
magiah detect    --out results
magiah locate    --out results
magiah report    --out results

Calibration never learns from the machine's own unreviewed output by default (that would reinforce its own guesses). Without reviewed findings it learns nothing and says so; --calibrate-from-machine learns from report.db anyway and labels the result machine_unreviewed in calibration_meta.json. Learned files without that provenance (written by older versions), or whose hash no longer matches it, are ignored, and a calibrate run with nothing to learn disables the previous files. python -m magiah.eval --out results measures the ranking's precision on reviewed books held out from calibration, per error class and evidence kind, what each rank cut-off would lose, a calibration table and the effect of removing each rank term. Reviewed findings are mostly the top of the ranking, so --sample-out sample.csv exports a stratified random sample (error class x rank band, with stratum weights) to label, and --sample-labels sample.csv turns it into unbiased estimates.

Rank signals measured against chance. report.db stores, per occurrence, ctx_exp (the context pairs independence would predict) and ctx_assoc (log10 of hits against it): the rank bonus grows with it, and no hits cost a penalty only where hits were expected. Two optional signals cost a longer context pass and are off by default: --rw-corpus 1 (rw_lr: the real-word language model, original vs. correction between the same neighbours, the original's n-grams counted outside its own book) and --par-corpus 1 (par_sugg / par_word / par_vote: the two-words-each- side window seen with the correction vs. with the original in other books). All of them move findings up or down, never in or out. With --rank-model 1, calibrate fits a logistic ranking on the review labels (rank_model.json, provenance-checked like the confusion matrix) once there are --rank-model-min (200) of them; the reports then rank by its log-odds and fill a prob column. With fewer labels the hand formula stays.

Scan a single book (seconds instead of an hour):

magiah book --book-list "Bartenura" --out results  # find a book
magiah book --book 593 --out results               # a seforim.db book id
magiah book --book "books/halacha/x.txt" --out results  # a library relpath
magiah book --book "C:\new book.txt" --out results      # any .txt on disk

Scans one book against the already-built lexicon — the lexicon is never rebuilt, which is what makes this cost seconds. Every correction candidate is still scored against whole-corpus frequencies, so detection quality matches a full scan.

Findings are merged additively into ui_review.db. If the book was scanned before, its rows are replaced in place (no duplicates) and your review decisions on them are preserved; other books are untouched.

--book-verify-ctx additionally verifies each correction and each split against the entire corpus (adds ~10 min); the findings are then the ones a full scan gives for this book. Without it, verification is book-local and is labelled so (ctx_scope/split_scope = book, evidence context_book / split_observed_book).

--book-real-word (off by default) also looks for real-word errors: a one-letter slip that produced another existing word, which no rarity rule can see. Each common word of the book is weighed against its confusable one-letter neighbours by how often each occurs between the same two words in the corpus (bigrams and trigrams, a fixed substitution penalty), and is reported as real_word only when the log10 odds against it reach real_word_odds (5.49 — tuned on OCR output from outside this corpus, so treat it as a starting point). A word the book uses at least real_word_rep (3) times, and more often than the suggestion, is taken as the book's own spelling. Tokens already reported as an extra space or an OCR-profile hit, and words one or two words after a title (a name position), are not judged. The findings carry evidence context_lm and are never counted as verified. The counts come from the --book-verify-ctx pass (same pass, the book's own rows excluded) and are kept in real_word/ for later book-scope scans of the same book; a word whose neighbours changed since then is not judged until the next corpus-scope scan. real_word/ holds one file per book scanned this way and is never pruned; deleting it only means the next book-scope scan needs a --book-verify-ctx rescan first.

--alternatives N (off by default) lists, for each edit-1 / spelling-variant finding, up to N of the corrections the edit-1 rule accepted for the word (alternatives in the finding; in the review UI, the drawer's "הצעות חלופיות"). With --book-verify-ctx they are ranked per occurrence by the same bigram/trigram model as --book-real-word, counted in the same pass without the book's own rows (by: context); with no counts, no neighbour, or no candidate ever seen beside these neighbours, they keep the detector's order (by: frequency). The finding's suggestion does not change unless --alt-rerank 1 is also given: the suggestion then becomes the alternative the context ranks first, but only where the context prefers it to the detector's own choice by at least 2 (log10 odds, alternatives.MARGIN); otherwise the suggestion stays, and a finding whose candidates do not include its suggestion gets no alternatives. The reranked suggestion is verified like any other, so --alt-rerank also asks the context pass to count the neighbour pairs of every candidate (on a synthetic book about 1.9x the pairs of a plain scan; the pass reads the corpus once either way, so the added time was within noise there).

--book-domain-max SHARE (off by default; 0.05 is a measured value) is a domain gate for the whole book. The lexicon knows the language most of the library is written in; a book in modern Hebrew, another language or another spelling system is full of words it has never seen, and the rarity rules then flag correct words. Before reporting, the scan counts the book's words of 3+ letters (abbreviations aside) that the lexicon has at most rare_max times apart from the book itself. When more than SHARE of them are unknown (and the book has at least book_domain_min_words, 200, such word occurrences and at least book_domain_min_types, 100, different ones: a short text that repeats a few words is not judged), the findings that rest on a word being rare are left out — edit1_*, spelling_variant, missing_space, lost_quotes and real_word — and the scan says so in its log and in the result's domain entry (the share and how many findings of each kind were left out). Final-letter findings, the book's OCR profile and extra spaces stay. The foreign-line skip (foreign_ratio) still works line by line inside any book; the gate is the book-level decision on top of it. On the Otzaria library lexicon, 99% of the books of halacha, Talmud, responsa and Mishnah measure under 2%; the 0.6% of library books above 5% are mostly Aramaic targums, plus the Judeo-Arabic and Syriac translations and a few piyyut and midrash collections; digital modern-Hebrew books measure 6-16%.

The gate refuses to apply, says so in the log and keeps every finding (domain.refused is small or partial, out is false) when the lexicon cannot serve as a yardstick: it counted fewer than book_domain_min_lexicon (1,000,000) words in all, or it was built partial under --allow-unread. A near-empty lexicon knows no word, so every ordinary book would look foreign. A book read from the corpus (db/library) has its own occurrences taken away from the lexicon's count first, the same rule as the book-repeat and real-word checks; a file has none taken away. When the gate acts, split alternatives of left-out missing_space findings are left out too, and real_word.hints counts what is still shown.

Requires an existing lexicon.pkl — i.e. one prior magiah lexicon (or all) run.

Extra lexicons from your own documents

The corpus is the dictionary, so a book in modern Hebrew (or in a field the library does not cover) gets many ordinary words flagged as rare. An extra lexicon counts the words of documents you choose:

magiah extra --name modern --from "C:\modern texts" --out results  # a folder (with subfolders)
magiah extra --name modern --from a.docx --from b.epub --out results  # or files
magiah extra --out results                    # list the extra lexicons
magiah extra --remove modern --out results    # delete one
magiah book --book "C:\new book.txt" --extra-lexicon modern --out results

Readable: .txt (UTF-8, UTF-16 with a BOM, or Windows-1255), .html / .htm / .xhtml, .docx, .epub — standard library only. PDF is not read (that needs a third-party library): a PDF named on the command line is refused, PDFs inside a folder are skipped and counted. The text is tokenized exactly like the corpus. Building a name again replaces it; the lexicon is extra_<name>.pkl in the output folder, with the list of what it was built from.

A scan that selects it (--extra-lexicon NAME, repeatable; detect, all and book) never flags a word the extra lexicon holds at least --extra-min (3) times — fewer could be a typo in the documents. It is suppression only, like --whitelist: it never creates a finding and is never a correction candidate, and the real-word check is unchanged. The selection is remembered in run_config.json like the whitelist; --no-extra-lexicon clears it. In the web UI it is the "מילונים נוספים" field of the scan panel (one name per line).

Select it for the books that need it, not always. Every modern word it adds is a word an error can turn into without being seen: measured on rabbinic text with injected errors, an extra lexicon built from 1,000 modern Hebrew books hid 2.2% of the errors that were caught without it (one built from 100 books: 0.2%), while on other modern books it cut the findings per 1,000 words from 29.3 to 4.3 (11.3 with 100 books). The larger the extra lexicon, the more it hides. Check a correction after the fact (magiah guard):

magiah guard --before page.ocr.txt --after page.fixed.txt --out results

Lines the two texts up word by word and judges every change by the corpus frequencies of --out's lexicon: a common word (>= common_min) that became a rare or unknown one (<= rare_max) is suspect — the corrector (a person, an OCR post-corrector, a language model) probably broke a good word; the reverse is supported; anything else neutral. A split, join, insertion or deletion is one unit, judged by its least frequent word; but inside such a unit, words that clearly correspond (the same word with a letter or two changed) are paired first and judged one by one, so a real common -> rare slip is not hidden by a split or join next to it. Prints the suspect changes (--guard-all: every change) as tab-separated line, verdict, old, new, f_old, f_new, then a count. Writes nothing (a --out that does not exist is not created). The exit code is 0 even when suspects are found: it is 1 only when the check could not run (a missing, empty, corrupt or partial lexicon, an unreadable file, or two texts so different and repetitive that comparing them would take unreasonably long -- split them). Read the printed list, not the exit code. It sees only one kind of damage: a common word turned into another common word (a real-word slip) is neutral, and supported does not mean correct.

Two more things to know:

  • extra_min masking. Because a word counts as known once it appears --extra-min (3) times, a typo that appears three or more times in your own documents is hidden too: the extra lexicon cannot tell a repeated typo from a real word. Raise --extra-min (or clean the documents) if the documents are not carefully proofread. Names that differ only by case are one lexicon (Modern is modern), because Windows file names are case-insensitive.
  • Trust the output folder. extra_<name>.pkl (like lexicon.pkl and the other results) is a Python pickle, and loading a pickle can run code. Only use an output folder, and only copy extra lexicons, that come from you or from someone you trust.

Build limits: a file that yields no Hebrew words (for example a .docx without word/document.xml) is reported with a warning and counted in the summary; a build in which no file yields any word writes nothing. A .docx or .epub whose text expands past 200 MB (a decompression bomb) stops the build with an error naming the file; nothing is written. .epub and .html content is read in the encoding it declares (encoding= / charset=), falling back to UTF-8 and then Windows-1255.

Review the findings in your browser:

magiah review --out results      # opens http://127.0.0.1:8765/

Findings are shown one by one (filter by error type and source repository; order by score or randomly shuffled). Keyboard: י accept, נ reject this occurrence, ד reject the word everywhere, ת type your own correction, ע ignore forever, space skip for now. Every decision is stored immediately in decisions.db; the export button writes a to_send/ folder with one ready-to-send CSV of approved fixes per source repository, plus rejected_words.txt (which future scans use as a whitelist automatically).

Running Magiah on your own corpus

Magiah is not tied to Otzaria. Any Hebrew/Aramaic text collection works, and the whole pipeline needs just one thing: a way to iterate over your lines of text. There are three built-in adapters, from simplest to most capable.

Option 1 — a folder of text files (works for everything)

Put your books in a directory tree of UTF-8 .txt files (subfolders are fine, one book per file is best):

magiah all --textdir path/to/books --out results

Each file is treated as a document, which enables book-local verification (is this correction already used elsewhere in the same book?) and per-book OCR profiles.

If your corpus is in any other format — Word documents, PDFs with a text layer, JSON, CSV, HTML — the simplest route is to convert it to a folder of .txt files. For example, from JSON:

import json, os, pathlib
data = json.load(open('mybooks.json', encoding='utf-8'))
out = pathlib.Path('books_txt'); out.mkdir(exist_ok=True)
for book in data:
    safe = ''.join(c if c.isalnum() else '_' for c in book['title'])
    (out / f"{safe}.txt").write_text('\n'.join(book['lines']), encoding='utf-8')

Magiah's tokenizer already strips nikud, cantillation marks, HTML tags and Unicode presentation forms — you don't need to clean the text first.

Option 2 — any SQLite database

If your corpus is already a SQLite database with one row per line/paragraph:

magiah all --sqlite mycorpus.db \
    --table line --id-col id --text-col content --doc-col book_id \
    --out results
Flag Meaning Default
--table the table holding your text rows line
--id-col integer primary key of that table (used to chunk work across processes and to reference findings) id
--text-col the column with the actual text content
--doc-col optional but recommended: a column grouping rows into books/documents. Enables book-local verification and OCR profiles none

Any schema works as long as those columns exist. If your text lives in multiple tables, create a view:

CREATE VIEW all_lines AS
  SELECT id, book_id, text AS content FROM mishna
  UNION ALL
  SELECT id + 1000000, book_id, text FROM talmud;

then --table all_lines.

Option 3 — the Otzaria preset

--otzaria is just a preset of Option 2 (table=line, id-col=id, text-col=content, doc-col=bookId) plus report enrichment that joins book titles, references and source-repository names from Otzaria's schema. Use --db path if your seforim.db lives elsewhere.

Notes for custom corpora

  • The corpus source is saved in results/run_config.json after the first command — subsequent stages don't need the flags again.
  • Corpus size matters. The statistics need volume: below ~5M words, raise thresholds (--common-min 10 --ed1-ratio 20) and expect lower precision; below ~1M words the "corpus as dictionary" premise gets weak — consider adding more text of the same genre to the corpus (findings are reported per book anyway, so extra background text costs nothing).
  • The review interface and per-type reports work identically for every adapter. Otzaria-specific extras (per-source-repository folders, heRef references) simply stay empty for other corpora.

Outputs (in the --out directory)

File Contents
errors_<type>.csv one ranked CSV per error class: word, suggested correction, the aligned verse's reading (tanach_reading, when 2+ sources back it), confidence rank, context-verification hits, book, snippet
errors_<type>_verified.csv high-precision subset (context-verified or correction already used in the same book)
spelling_variants.csv ktiv male/chaser ו/י differences — policy decisions, not typos
space_errors.csv extra-space findings
tanach_matches.csv / tanach_edition_errors.csv quotations confirmed by 2+ independent sources / edition disagreements (evidence JSON: variant vs. unresolved, witnesses)
by_source/<origin>/… the same reports split per source repository (Otzaria corpora)
report.db everything as a queryable SQLite database
coverage_<stage>.json what each stage actually read — a partial read stops the stage and is recorded here; for a partial output accepted under --allow-unread also accepted, the rows not read and where they are
extra_<name>.pkl an extra lexicon built by magiah extra (word counts + what they were built from)
run_state/ how the latest runs ended (scan.json, book.json) — read by the review UI
to_send/ written by the review interface: approved fixes per source repository, ready to send upstream
-- highest-confidence findings
SELECT * FROM occurrences_full ORDER BY score DESC LIMIT 100;
-- findings in one book
SELECT * FROM occurrences_full WHERE source LIKE '%תוספתא%';

Tuning

All thresholds are CLI flags (see magiah --help). The important ones:

Flag Default Meaning
--rare-max 2 a word is suspect if it occurs ≤N times in the whole corpus. Raise to 3–5 to find more errors at the cost of more false positives
--common-min 30 minimum frequency for a proposed correction
--ed1-ratio 50 how many times more frequent the correction must be
--workers 3 parallel processes

Why not a dictionary? Why not AI?

  • Dictionaries fail on rabbinic Hebrew, Aramaic, Yiddish loanwords, acronyms, and the thousand spelling conventions of 1,000 years of printing. The corpus's own frequency distribution is the right dictionary for the corpus.
  • LLMs over 400M words are slow, expensive, and hallucinate corrections. Statistical evidence (frequency ratios + bigram/context verification) is reproducible, explainable, and runs on a laptop.

License

MIT

About

Corpus-based typo & OCR error detection for Hebrew/Aramaic texts - no dictionaries, no AI

Resources

Security policy

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages