Skip to content

Add local MD5 hash calculation for ProgressSync (closes #150) - #160

Open
vishae wants to merge 1 commit into
kyxap:developfrom
vishae:feature/md5-hash-auto-calc
Open

Add local MD5 hash calculation for ProgressSync (closes #150)#160
vishae wants to merge 1 commit into
kyxap:developfrom
vishae:feature/md5-hash-auto-calc

Conversation

@vishae

@vishae vishae commented Jul 11, 2026

Copy link
Copy Markdown

Summary

Implements #150: local (in-Calibre) calculation of the KOReader MD5 hash, so ProgressSync works for books that were only ever synced from a device that pushes straight to a ProgressSync server without running actual KOReader software (the exact scenario #150 describes, and part of what #142 ran into).

Adds koreader_hash.py, a Python port of KOReader's own document-hashing algorithm, and wires it into a new "Calculate Missing MD5 Hashes" menu action. For every book in the library that doesn't already have a value in the mapped MD5 column, it computes the hash directly from the book's EPUB file and fills it in. Books that already have an MD5 (e.g. learned from a genuine KOReader sidecar) are left untouched.

Why this hash, and why it's correct

As moozhub found in #142's comments, the hash KOReader writes isn't a plain file md5sum — it's a partial-content hash, read from a fixed set of chunks at specific offsets rather than the whole file (this is why it stays fast even on huge books).

Rather than reverse-engineer this from scratch, I verified it directly against KOReader's own reference implementation, util.partialMD5() in koreader/koreader's frontend/util.lua:

function util.partialMD5(filepath)
    local step, size = 1024, 1024
    local update = md5()
    for i = -1, 10 do
        file:seek("set", lshift(step, 2*i))
        local sample = file:read(size)
        if sample then update(sample) else break end
    end
    return update()
end

Offsets: 1024 << (2*i) for i = -1..10 (12 offsets total: 0, 1024, 4096, 16384, 65536, 262144, 1048576, 4194304, 16777216, 67108864, 268435456, 1073741824). The i = -1 case is a neat quirk - lshift(1024, -2) overflows a 32-bit int (the negative shift count gets masked to 5 bits, i.e. 30, and 1024 << 30 wraps to exactly 0) - which koreader_hash.py replicates via an explicit i < 0 → 0 case instead of relying on the same overflow behavior in Python.

Test plan

  • Algorithm correctness: 9 new unit tests in tests/unit/test_koreader_hash.py - small file, exact-offset-boundary file, multi-offset file, ~5MB file, empty file, missing file, determinism, digest shape. Each test computes its own independent expected hash (not by re-deriving from the module under test), so a bug in the offset table itself would still be caught.
  • Full suite: make test → 22/24 pass. The 2 failures (test_version_match, test_version_tuple_match) are pre-existing, unrelated version-string sync checks that also fail on unmodified develop HEAD.
  • Lint: make lint → 9.70/10 (threshold 9.5), no new warnings; koreader_hash.py itself is clean.
  • Calibre DB integration (the part with no existing test coverage in this repo): ran the actual logic against a real Calibre 9.11 database, using this repo's own bundled dummy_library fixture (which conveniently already has a #ko_md5 column mapped, with one book missing a value and one already populated). Copied to an isolated scratch location and verified:
    • The book missing an MD5 (Walden) got the exact hash independently precomputed via calculate_koreader_md5() outside of Calibre.
    • The book that already had an MD5 (Alice's Adventures in Wonderland) was left untouched.
    • Reopened the database on a fresh connection afterward and confirmed the write actually persisted to disk.

Scope notes

  • Only fills in books missing an MD5 - never overwrites an existing value, even if it looks wrong.
  • Only checks the EPUB format for each book (matches what devices like the one that prompted [FEATURE] Local (in-Calibre) calculation of MD5 hash #150 actually read).
  • New menu action, run on demand - doesn't change any existing sync behavior automatically.

ProgressSync only works for books that already have their KOReader MD5
hash stored in Calibre's mapped column, which normally only gets
populated by reading a real KOReader sidecar file. Devices that push
straight to a ProgressSync server without ever running actual KOReader
software (e.g. custom/alternative firmware) never give Calibre a chance
to learn the hash, so those books get silently skipped forever (kyxap#150,
also the root confusion in kyxap#142 - the hash isn't a plain file md5sum).

Adds koreader_hash.py, a from-scratch Python port of KOReader's own
partial-content hashing algorithm (util.partialMD5 in
koreader/koreader's frontend/util.lua), verified directly against that
source rather than derived secondhand - including its i = -1 special
case, which relies on a 32-bit left-shift overflow landing on offset 0.

Wires this into a new "Calculate Missing MD5 Hashes" menu action that
computes the hash locally from each book's EPUB file and fills in the
mapped MD5 column, but only for books that don't already have one -
existing values (e.g. from a genuine KOReader sidecar) are left alone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant