Skip to content

Lexicon: incremental update and rollback (mirror of #36, crash-safe swap) - #43

Merged
Nathaniel-260 merged 4 commits into
mainfrom
mirror/pr36-lexicon-incremental
Nov 16, 2026
Merged

Nathaniel-260 merged 4 commits into
mainfrom
mirror/pr36-lexicon-incremental

Conversation

@Nathaniel-260

Copy link
Copy Markdown
Owner

Mirror of #36 (fork PR; CodeQL does not run on forks) plus QA fixes: crash-safe transactional swap with a journal and recovery, file errors reported as StageError (Hebrew message, exit 1), leftover lexicon_new cleaned up. Merged with main; the cli.py conflict kept both the lexicon-update and the extra/guard options.

🤖 Generated with Claude Code

mynim and others added 4 commits November 15, 2026 20:48
…-rollback)

magiah lexicon --update recounts only the books added, changed or removed
since the last update (seforim.db: contentHash + line count; files: size +
mtime), keeping per-book counts in a side file, lexicon_books.pkl. The
result equals a full count. --rollback swaps in the version the last update
replaced; --status reports what an update would do. Plain magiah lexicon is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
40% smaller side file on the Otzaria library than a compressed pickle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The swap moved files one by one, so a failure midway could leave
lexicon.pkl missing or mismatched with names.pkl and lexicon_prev half
filled. Now one transaction (_exchange) serves --update and --rollback:
current files go to lexicon_old/, new ones into place, a journal
(lexicon_txn.json) is removed as the commit, then lexicon_old/ becomes
lexicon_prev/. Any failure before the commit puts everything back; a
killed process is repaired by recover() on the next update, rollback or
plain lexicon run. OSError during an update or rollback is now a Hebrew
StageError (exit 1); a leftover lexicon_new is cleaned up, or reported
clearly when it cannot be removed.

Tests inject OSError at every os.replace of update and rollback (single
failure and killed process), leftover lexicon_new, write errors, and the
CLI output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@Nathaniel-260
Nathaniel-260 merged commit e2103a0 into main Nov 16, 2026
13 checks passed
@Nathaniel-260
Nathaniel-260 deleted the mirror/pr36-lexicon-incremental branch November 16, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants