Add local MD5 hash calculation for ProgressSync (closes #150) - #160
Open
vishae wants to merge 1 commit into
Open
Add local MD5 hash calculation for ProgressSync (closes #150)#160vishae wants to merge 1 commit into
vishae wants to merge 1 commit into
Conversation
ProgressSync only works for books that already have their KOReader MD5 hash stored in Calibre's mapped column, which normally only gets populated by reading a real KOReader sidecar file. Devices that push straight to a ProgressSync server without ever running actual KOReader software (e.g. custom/alternative firmware) never give Calibre a chance to learn the hash, so those books get silently skipped forever (kyxap#150, also the root confusion in kyxap#142 - the hash isn't a plain file md5sum). Adds koreader_hash.py, a from-scratch Python port of KOReader's own partial-content hashing algorithm (util.partialMD5 in koreader/koreader's frontend/util.lua), verified directly against that source rather than derived secondhand - including its i = -1 special case, which relies on a 32-bit left-shift overflow landing on offset 0. Wires this into a new "Calculate Missing MD5 Hashes" menu action that computes the hash locally from each book's EPUB file and fills in the mapped MD5 column, but only for books that don't already have one - existing values (e.g. from a genuine KOReader sidecar) are left alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #150: local (in-Calibre) calculation of the KOReader MD5 hash, so
ProgressSyncworks for books that were only ever synced from a device that pushes straight to a ProgressSync server without running actual KOReader software (the exact scenario #150 describes, and part of what #142 ran into).Adds
koreader_hash.py, a Python port of KOReader's own document-hashing algorithm, and wires it into a new "Calculate Missing MD5 Hashes" menu action. For every book in the library that doesn't already have a value in the mapped MD5 column, it computes the hash directly from the book's EPUB file and fills it in. Books that already have an MD5 (e.g. learned from a genuine KOReader sidecar) are left untouched.Why this hash, and why it's correct
As
moozhubfound in #142's comments, the hash KOReader writes isn't a plain filemd5sum— it's a partial-content hash, read from a fixed set of chunks at specific offsets rather than the whole file (this is why it stays fast even on huge books).Rather than reverse-engineer this from scratch, I verified it directly against KOReader's own reference implementation,
util.partialMD5()inkoreader/koreader'sfrontend/util.lua:Offsets:
1024 << (2*i)fori = -1..10(12 offsets total:0, 1024, 4096, 16384, 65536, 262144, 1048576, 4194304, 16777216, 67108864, 268435456, 1073741824). Thei = -1case is a neat quirk -lshift(1024, -2)overflows a 32-bit int (the negative shift count gets masked to 5 bits, i.e.30, and1024 << 30wraps to exactly0) - whichkoreader_hash.pyreplicates via an expliciti < 0 → 0case instead of relying on the same overflow behavior in Python.Test plan
tests/unit/test_koreader_hash.py- small file, exact-offset-boundary file, multi-offset file, ~5MB file, empty file, missing file, determinism, digest shape. Each test computes its own independent expected hash (not by re-deriving from the module under test), so a bug in the offset table itself would still be caught.make test→ 22/24 pass. The 2 failures (test_version_match,test_version_tuple_match) are pre-existing, unrelated version-string sync checks that also fail on unmodifieddevelopHEAD.make lint→ 9.70/10 (threshold 9.5), no new warnings;koreader_hash.pyitself is clean.dummy_libraryfixture (which conveniently already has a#ko_md5column mapped, with one book missing a value and one already populated). Copied to an isolated scratch location and verified:calculate_koreader_md5()outside of Calibre.Scope notes
EPUBformat for each book (matches what devices like the one that prompted [FEATURE] Local (in-Calibre) calculation of MD5 hash #150 actually read).