Detect a torn packed-cache read instead of serving wrong rows - #81
Merged
Merged
Conversation
`_save_packed` writes matrix.npy, ids.json, metas.json and sig.json as four independent calls. Writing sig last defends against a crash mid-write, but not against a *second* writer: two rebuilds of a same-size corpus leave the matrix from writer A beside ids/metas from writer B, and `_load_packed` validated only `len(ids) == mat.shape[0] and len(metas) == len(ids)`, so the mixture passed every check while row i no longer belonged to ids[i]. Searches then answered confidently wrong, with nothing raised and nothing logged. sig.json now also carries a sha256 `content_sig` over the exact ids.json / metas.json bytes written with that matrix, plus the matrix `shape`. On load, a disagreement reads as a cache miss and the matrix is rebuilt from records -- the same path already taken for a missing sig, a bad format, or an OSError. This is option 2 of the issue (detect cheaply and loudly), not option 1 (the atomic directory-pointer swap); a store-level lock and true write atomicity remain open there. Backward compatible in both directions: the new checks are guarded on the fields being present, so a cache written before them still loads on the length checks alone, and an older `ir` reading a new cache ignores the extra keys (it only reads `format`). `_PACKED_FORMAT` stays at 1 -- no cache is invalidated. No public name, signature or return type changes; every packed helper is module-private, and stores opened without `packed_dir` never reach this code. Claude-Session: https://claude.ai/code/session_01L1aQPB34n7PU7jmbztSjBe
thorwhalen
added a commit
that referenced
this pull request
Sep 22, 2026
#83) The content_sig added in #81 hashes ids/metas only. The interleaving "writer A saves matrix.npy, writer B saves a whole set, A then writes ids/metas/sig" leaves B's same-shape matrix under A's ids and a sig that vouches for them, so rows were still served under the wrong ids. Each save now writes matrix/ids/metas under a fresh generation token that only that writer touches, and publishes sig.json last via os.replace; a reader follows the sig to exactly one writer's complete set. Older generations are swept after publishing. Legacy flat-layout caches still load. Regression test drives the real interleaving through _save_packed. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Member
Author
|
Post-merge refute review: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detects a torn packed-cache read instead of silently serving mismatched rows.
CorpusStore._save_packedwritesmatrix.npy,ids.json,metas.jsonandsig.jsonas four independent calls. Two concurrent rebuilds of a same-sizecorpus can leave the matrix from writer A beside ids/metas from writer B; the
old
_load_packedonly checkedlen(ids) == mat.shape[0], so the mixturepassed every check while row
ino longer belonged toids[i]— searchesanswered confidently wrong, nothing raised or logged.
sig.jsonnow also carries a sha256content_sigover the exact ids/metasbytes written with that matrix, plus the matrix
shape. On load, adisagreement is treated as a cache miss and the matrix rebuilds from records
— the same fallback path already used for a missing sig, bad format, or
OSError.
This is option 2 of #77 (detect cheaply and loudly), not option 1 (atomic
directory-pointer swap); a store-level lock / true write atomicity remains
open there, so this closes #77 for the corruption-detection half of it.
Backward/forward compatible: checks are guarded on the new fields being
present, so a cache written before this loads on the length checks alone, and
_PACKED_FORMATstays at 1 (no cache invalidation). No public name,signature or return type changes — every packed helper is module-private,
and stores opened without
packed_dir(e.g.CorpusStore.memory()) neverreach this code.
What I did (rebase + gate)
master(2 commits it was missing: Replace argh with cw, preserving the BY_NAME_IF_KWONLY grammar #79, feat: arecordscorpus source whose records another package owns #80) — no conflicts.wads ci-local: ruff format + lint, pytest on py3.10 and py3.12 (556passed, 5 skipped), build — all green.
Dependents check
fleet_dependents.jsonlistsraglabandtruffleas importers ofir.raglab(present on this box) only callsCorpusStore.memory()and thepublic
CorpusStoresurface in its tests/source — it never setspacked_dirand never touches_load_packed/_save_packed, so it cannotreach this code path at all (confirmed by grep, matching the commit's own
claim that stores without
packed_dirnever reach this code).truffleis not cloned on this box, so it could not be checked directly.Given the change touches only module-private packed-cache helpers with no
public signature/return-type/name change, this is not treated as a
breaking change requiring
truffle's tests — noted here for the record.Closes #77
🤖 Generated with Claude Code