Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions README.he.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,31 @@ magiah book --book "C:\ספר חדש.txt" --out results # קובץ מהמ
הבאה מול הקורפוס. בתיקייה `real_word/` נשמר קובץ אחד לכל ספר שנסרק כך, והיא
אינה מתנקה מעצמה; מחיקתה רק מחייבת סריקה חוזרת עם `--book-verify-ctx`.

`--book-domain-max SHARE` (כבוי כברירת מחדל; 0.05 הוא הערך שנמדד) הוא **שער
תחום** לספר כולו. המילון מכיר את הלשון שבה כתובה רוב הספרייה; ספר בעברית
מודרנית, בשפה אחרת או בשיטת כתיב אחרת מלא מילים שהמילון לא ראה, וכללי הנדירות
מסמנים בו מילים נכונות. לפני הדיווח הסריקה סופרת את מילות הספר בנות 3 אותיות
ומעלה (בלי ראשי תיבות) שהמילון מכיר לכל היותר `rare_max` פעמים מחוץ לספר
עצמו. כשיותר מ-SHARE מהן אינן מוכרות (ובספר יש לפחות `book_domain_min_words`,
200, מופעים של מילים כאלה, ולפחות `book_domain_min_types`, 100, מילים שונות — טקסט קצר החוזר על מילים מעטות אינו נמדד), הממצאים שנשענים על נדירות המילה מושמטים — `edit1_*`,
`spelling_variant`, `missing_space`, `lost_quotes` ו-`real_word` — והסריקה
אומרת זאת ביומן ובפריט `domain` של התוצאה (השיעור, וכמה ממצאים מכל סוג
הושמטו). ממצאי אותיות סופיות, פרופיל ה-OCR של הספר ורווחים מיותרים נשארים. הדילוג על שורה
זרה (`foreign_ratio`) ממשיך לפעול שורה-שורה בכל ספר; השער הוא ההחלטה ברמת
הספר שמעליו. במילון של ספריית אוצריא 99% מספרי ההלכה, התלמוד, השו"ת והמשנה
מתחת ל-2%; ה-0.6% מספרי הספרייה שמעל 5% הם בעיקר תרגומים ארמיים, ועוד תרגומים
בערבית-יהודית ובסורית וכמה קובצי פיוט ומדרש; ספרים דיגיטליים בעברית מודרנית
מגיעים ל-6–16%.

השער מסרב לפעול, אומר זאת ביומן ומשאיר את כל הממצאים (`domain.refused` הוא
`small` או `partial`, ו-`out` שקר) כשהמילון אינו יכול לשמש קנה מידה: הוא מכיל
פחות מ-`book_domain_min_lexicon` (1,000,000) מילים בסך הכול, או שנבנה חלקית
עם `--allow-unread`. מילון כמעט ריק אינו מכיר שום מילה, ולכן כל ספר רגיל היה
נראה זר. בספר שנקרא מהמאגר (`db`/`library`) מופעיו שלו מופחתים תחילה מספירת
המילון, כבכלל של חזרת הספר ובדיקת המילים האמיתיות; מקובץ (`file`) לא מופחת
דבר. כשהשער פועל, גם חלופות הפיצול של ממצאי `missing_space` שהושמטו מושמטות,
ו-`real_word.hints` סופר את מה שעוד מוצג.

> דורש `lexicon.pkl` קיים — כלומר הרצה אחת של `magiah lexicon` (או `all`).

**בדיקת תיקון בדיעבד** (`magiah guard`):
Expand Down
33 changes: 33 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,39 @@ judged until the next corpus-scope scan. `real_word/` holds one file per book
scanned this way and is never pruned; deleting it only means the next
book-scope scan needs a `--book-verify-ctx` rescan first.

`--book-domain-max SHARE` (off by default; 0.05 is a measured value) is a
**domain gate** for the whole book. The lexicon knows the language most of
the library is written in; a book in modern Hebrew, another language or another spelling
system is full of words it has never seen, and the rarity rules then flag
correct words. Before reporting, the scan counts the book's words of 3+
letters (abbreviations aside) that the lexicon has at most `rare_max` times
apart from the book itself. When more than SHARE of them are unknown (and
the book has at least `book_domain_min_words`, 200, such word occurrences
**and** at least `book_domain_min_types`, 100, different ones: a short text
that repeats a few words is not judged), the
findings that rest on a word being rare are left out — `edit1_*`,
`spelling_variant`, `missing_space`, `lost_quotes` and `real_word` — and
the scan says so in its log and in the result's `domain` entry (the share
and how many findings of each kind were left out). Final-letter findings,
the book's OCR profile and extra spaces stay. The foreign-line skip
(`foreign_ratio`) still works line by line inside any book; the gate is
the book-level decision on top of it. On the Otzaria library lexicon,
99% of the books of halacha, Talmud, responsa and Mishnah measure under 2%;
the 0.6% of library books above 5% are mostly Aramaic targums, plus the
Judeo-Arabic and Syriac translations and a few piyyut and midrash collections;
digital modern-Hebrew books measure 6-16%.

The gate refuses to apply, says so in the log and keeps every finding
(`domain.refused` is `small` or `partial`, `out` is false) when the lexicon
cannot serve as a yardstick: it counted fewer than `book_domain_min_lexicon`
(1,000,000) words in all, or it was built partial under `--allow-unread`.
A near-empty lexicon knows no word, so every ordinary book would look
foreign. A book read from the corpus (`db`/`library`) has its own
occurrences taken away from the lexicon's count first, the same rule as the
book-repeat and real-word checks; a `file` has none taken away. When the gate
acts, split alternatives of left-out `missing_space` findings are left out
too, and `real_word.hints` counts what is still shown.

> Requires an existing `lexicon.pkl` — i.e. one prior `magiah lexicon` (or
> `all`) run.

Expand Down
149 changes: 146 additions & 3 deletions magiah/book_scan.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
from .book_source import _rel_within, load_book
from .config import Config
from .corpus import OTZARIA_DB
from .normalize import tokenize
from .normalize import is_abbrev, tokenize
from .textsource import ReadStats

LOCAL_TYPES = core.LOCAL_TYPES
Expand Down Expand Up @@ -139,6 +139,87 @@ def detect_in_book(book, freq, cfg, out_dir, progress=None):
return lex, errors, split_cands, present, lsource


# Findings the domain gate (Config.book_domain_max) leaves out: the rules
# that call a word wrong because the lexicon finds it rare and knows a
# similar, a split or an abbreviated common form. In a book outside the
# lexicon's domain most rare words are simply words the lexicon never saw.
# Final-letter rules, the book's reviewed OCR profile and extra spaces do
# not rest on the word being rare, and stay.
DOMAIN_TYPES = ('edit1_sub', 'edit1_ins', 'edit1_del', 'edit1_swap',
'spelling_variant', 'missing_space', 'lost_quotes',
realword.ERRTYPE)


def domain_on(cfg):
"""Is the domain gate on (``Config.book_domain_max`` above 0)?"""
try:
return float(cfg.book_domain_max or 0) > 0
except (TypeError, ValueError):
return False


def book_domain(book, freq, present, cfg):
"""How much of `book` the lexicon does not know, and whether the book is
outside the lexicon's domain.

Counted over the book's words of 3+ letters that are not abbreviations;
a word is unknown when the lexicon has it at most ``rare_max`` times
apart from this book (a file from outside the corpus was never counted
in it, so nothing is taken away). Returns a dict: ``words`` (their
occurrences), ``types`` (how many different ones), ``unknown``
(occurrences), ``share``, ``max`` and ``out`` (True when the share is
above ``book_domain_max``; never for a book with fewer than
``book_domain_min_words`` such occurrences or fewer than
``book_domain_min_types`` different words: the minimum is both, so a
short text that repeats a few words is not judged).

Own count: a book read from the corpus (kind 'db' or 'library') is part
of what the lexicon counted, once, tokenized as ``_book_tokens`` does
(the lexicon's counting reads text the same way), so its own
occurrences are taken away before asking whether the lexicon knows the
word. This is the same assumption as the book-repeat rule
(``freq <= present`` is "only this book", ``core.book_repeat_value``)
and the real-word check (``own = present`` unless kind is 'file'). A
'file' is never taken away from, even one that happens to sit inside
the corpus library: that can only understate the unknown share (the
safe side for a gate that drops findings).
"""
own = book.kind != 'file'
words = types = unknown = 0
for w, c in present.items():
if len(w) < 3 or is_abbrev(w):
continue
words += c
types += 1
if freq.get(w, 0) - (c if own else 0) <= cfg.rare_max:
unknown += c
share = unknown / words if words else 0.0
limit = float(cfg.book_domain_max)
return {'words': words, 'types': types, 'unknown': unknown,
'share': round(share, 4), 'max': limit,
'out': (words >= cfg.book_domain_min_words
and types >= cfg.book_domain_min_types
and share > limit)}


def domain_lexicon_problem(freq, cfg, out_dir):
"""Why the lexicon cannot tell what is outside its domain, or None.

The share of unknown words is only meaningful against a lexicon that
knows a language. Two cases make nearly every word of a normal book
look unknown, so the gate would drop the findings of an ordinary book:
a lexicon with too few words in all (``book_domain_min_lexicon``;
'small'), and one built partial under ``--allow-unread``
('partial': a missing part of the library is missing words). Returns
``'small'`` / ``'partial'``; the gate then does not apply.
"""
if core.accepted_gaps(out_dir, 'lexicon'):
return 'partial'
if sum(freq.values()) < cfg.book_domain_min_lexicon:
return 'small'
return None


def _resolve_splits(errors, split_cands, freq, lex, seq_counts, scope):
"""Record each word's best-evidenced split; return all alternatives."""
alts_out, idx = {}, 0
Expand Down Expand Up @@ -471,16 +552,47 @@ def scan_loaded_book(book, freq, cfg, out_dir, verify_ctx=False, spec=None,
_covered(joins, ocr))
rows.extend(real_word.pop('rows'))

domain = None
dropped_words = set() # words whose every finding the gate left out
if domain_on(cfg):
domain = book_domain(book, freq, present, cfg)
refused = domain_lexicon_problem(freq, cfg, out_dir)
if refused: # measured, but not enough of a yardstick to act on
domain['out'] = False
domain['refused'] = refused
domain['lexicon_words'] = sum(freq.values())
domain['lexicon_min'] = cfg.book_domain_min_lexicon
left_out = Counter()
if domain['out']:
kept = []
for r in rows:
if r['errtype'] in DOMAIN_TYPES:
left_out[r['errtype']] += 1
dropped_words.add(r['word'])
else:
kept.append(r)
rows = kept
dropped_words -= {r['word'] for r in rows}
if real_word is not None: # its summary counts what is shown
n_rw = left_out.get(realword.ERRTYPE, 0)
real_word['hints'] -= n_rw
real_word['domain_left_out'] = n_rw
domain['left_out'] = dict(left_out)
say(book_domain_message(book.title, domain))

space_rows = [{
'part1': p1, 'part2': p2, 'joined': j, 'join_freq': jf,
'origin': book.origin, 'source': book.title, 'ref': ref,
'unit': unit, 'doc': book.doc, 'snippet': snip,
} for unit, ref, p1, p2, j, jf, snip in joins]

# a split of a word whose finding the gate left out is not shown either,
# and no word is primary for a missing_space finding that is gone
gone = 'missing_space' in left_out_types(domain)
split_alts = [{'word': w, 'parts': p, 'observed': o, 'expected': e,
'chosen': bool(ch),
'is_primary': errors[w][1] == 'missing_space'}
for w, lst in alts.items()
'is_primary': errors[w][1] == 'missing_space' and not gone}
for w, lst in alts.items() if w not in dropped_words
for p, o, e, ch in lst]

# same rule as a full scan: accepted is not complete
Expand Down Expand Up @@ -518,9 +630,40 @@ def scan_loaded_book(book, freq, cfg, out_dir, verify_ctx=False, spec=None,
}
if real_word is not None: # off: the result is as it always was
out['real_word'] = real_word
if domain is not None: # likewise
out['domain'] = domain
return out


def left_out_types(domain):
"""The finding types the domain gate left out (empty: it did not act)."""
return set((domain or {}).get('left_out') or ())


def book_domain_message(title, domain):
"""The Hebrew line that reports the domain gate's verdict on a book."""
share, limit = domain['share'], domain['max']
if domain.get('refused'):
if domain['refused'] == 'partial':
why = 'המילון נבנה חלקית (--allow-unread)'
else:
why = (f'המילון קטן מדי: {domain["lexicon_words"]:,} מילים, '
f'נדרשים לפחות {domain["lexicon_min"]:,} (book_domain_min_lexicon)')
return (f'[book] שער התחום (book_domain_max) לא הופעל: {why}; '
f'כל הממצאים נשמרים (השיעור המדוד: {share:.1%})')
if not domain['out']:
why = (f'מעט מדי מילים למדידה ({domain["words"]:,} מופעים, {domain["types"]:,} שונות)'
if share > limit else 'בתוך תחום המילון')
return (f'[book] מילים שהמילון אינו מכיר: {share:.1%} '
f'(סף {limit:.0%}) — {why}')
n = sum(domain['left_out'].values())
return (f'[book] «{title}» מחוץ לתחום המילון: {share:.1%} ממילותיו '
f'(3 אותיות ומעלה) כמעט אינן במאגר, יותר מ-{limit:.0%} '
f'(book_domain_max). {n:,} ממצאים שנשענים על נדירות המילה '
f'הושמטו; ממצאי אותיות סופיות, פרופיל OCR ורווחים מיותרים '
f'נשארו')


# ---------------------------------------------------------------------------
# optional: real-word errors judged by the neighbouring words
# ---------------------------------------------------------------------------
Expand Down
9 changes: 9 additions & 0 deletions magiah/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,8 @@ def _run_book_cmd(args, spec, cfg, out_dir):
source = args.book_source or _guess_book_source(args.book)
if args.book_real_word:
cfg.real_word = 1
if args.book_domain_max is not None:
cfg.book_domain_max = args.book_domain_max
from .webui import db as uidb
# recorded apart from the pipeline's runs: a book scan never touches the
# full scan's results, so it must neither raise nor clear their warning
Expand Down Expand Up @@ -208,6 +210,13 @@ def main(argv=None):
'another existing word, judged by the neighbouring '
'words; the corpus counts are made by a '
'--book-verify-ctx scan and kept for later scans')
bk.add_argument('--book-domain-max', type=float, default=None,
metavar='SHARE',
help='leave out the rarity-based findings of a book in '
'which more than this share of the words (e.g. '
'0.05) is unknown to the lexicon: modern Hebrew, '
'another language or spelling (default 0: off)')

gd = ap.add_argument_group(
'guard: check a corrected text against the lexicon (changes nothing)')
gd.add_argument('--before', metavar='FILE',
Expand Down
18 changes: 18 additions & 0 deletions magiah/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,24 @@ class Config:
real_word_rep: int = 3 # ... a word the book uses this often (>= 1),
# and more often than the suggestion, is its
# own spelling
book_domain_max: float = 0.0 # book scan: a book with more than this
# share of its words (3+ letters, not
# abbreviations) rare in the lexicon apart from
# the book itself (<= rare_max) is outside the
# lexicon's domain (modern Hebrew, another
# language or spelling): its frequency-based
# findings (book_scan.DOMAIN_TYPES) are left out
# the result says so. 0 (or less): off
book_domain_min_words: int = 200 # ... judged only on books with at
# least this many such word OCCURRENCES ...
book_domain_min_types: int = 100 # ... and this many DIFFERENT such
# words (a short repetitive text reaches the
# first without being a sample of a language)
book_domain_min_lexicon: int = 1000000 # the gate applies only when the
# lexicon counted at least this many words in
# all (a near-empty lexicon knows no word: every
# book would look foreign) and was not built
# partial (--allow-unread); 0: no size floor
rw_corpus: int = 0 # corpus scan (1 = on): weigh each edit-1 /
# spelling-variant correction against the
# original between the same neighbours
Expand Down
Loading
Loading