Support Binary Versioning - #191
Open
ZwFink wants to merge 11 commits into
Open
Conversation
The bytes and diff writers each re-typed the global-variable and blob record prefixes, so a layout change had to be made in four places at once. Move each prefix into a small value type with write/read/size and route every writer, reader and size estimator through it. The metadata size formula moves next to metadata::serialize for the same reason. Metadata itself stays outside BlobHeader: the bytes layout puts it after the blob data and the diff layout puts it before the range count, so one contiguous codec cannot cover both without changing a layout. No on-disk change; the bytes written are identical.
Which format a snapshot file is, and what that implies, was re-derived by hand in six unrelated places: two magic sniffs, two reader entry points whose names promised a format the code did not enforce, a hard-coded 13-byte string in the unit test, and a Python-side rule that an epilogue always needs a base prologue. SnapshotHeader::parse is now the only code that knows any magic string, including both legacy layouts. It returns the kind, the version and the offset the payload starts at, so a reader never has to know which prefix it skipped. SnapshotFormatRegistry maps that pair onto one reader per (kind, version) through a table; an unknown pair is one fatal that names what it found. Self-containment is now a property of the reader rather than a guess at the call site. A prologue goes through BaseSnapshotSource::load, which rejects a diff by name instead of failing later on an empty base path, and a bytes epilogue no longer needs a base at all. SnapshotHeader::write still emits the legacy prefixes, so this writes byte-identical files.
The record side branched on EpilogueSnapshotType inside the epilogue lambda, over three static writers with three mismatched signatures, and "best" mode compared them through computeMnemeBytesSnapshotSize, a by-hand copy of the bytes layout that had to be edited in lockstep with the writer. SnapshotInput collapses the three signatures into one argument. FormatWriter owns the shape every format shares, a header followed by a payload, so a format supplies only header() and writePayload(). "Best" is now a composite writer that measures its two members and delegates to the smaller, rather than a third policy that re-derives both sizes. BytesWriter::measure stays analytic, because the default counting pass would copy every blob device-to-host just to size it. ValidateMeasure in the unit test now pins both writers to the size they actually produced, which is the check the hand-maintained estimate never had. makeEpilogueWriter is the only remaining switch on EpilogueSnapshotType. MnemeSnapshot is gone. No on-disk change.
A bytes snapshot had no header at all, so it was recognised only as "anything that is not a diff" and could never be versioned. Every new file now begins with the 16-byte SnapshotHeader, bytes and diff alike, which is the whole point of having a header: the next layout change is a version bump and one reader subclass rather than a new magic string. Only SnapshotHeader::write changes. No reader or registry row is needed, because the version names the payload layout, which is unchanged, and whether the container prefix is present is already handled by parse returning a different offset. Recordings made before this keep loading through the two aliases in parse; verified by reading legacy-prefix copies of both formats back through the registry. Old Mneme builds cannot read files written after this, which is the intended break. Two test-side parsers of the bytes layout, tests/record/read-record.py and python/tests/test_record_annotations.py, learn to skip the prefix. They keep the offset-0 fallback, so they still read older recordings.
The only caller asks whether a file can serve as a base, which it cannot if it depends on a base of its own. Naming that dependency directly reads better than negating "self-contained" at every use, and matches the Base parameter of read() and BaseSnapshotSource.
Nothing modified the argument, so the non-const reference only forced the bytes reader to copy the name into a mutable local before constructing.
Same fix as the std::string constructor; the name is only copied.
Add globalVarHeader and readGlobalFromDevice so the bytes and diff writers share one device-to-host path for globals, and drop the stray std::cout debugging from BytesWriter. Replace countChangedRanges with a scratch-buffer pass so ranges are scanned once, shared through emitRanges. Collapse the diff reader's count checks into expectCount and use StringRef::starts_with for the magic comparisons.
Each reader now owns its (kind, version) pair in a Layout member. The registry table and the paired writer's header() reference that member, so bumping a format means typing the version number in exactly one place.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We are evolving Mneme's binary format in different ways (#188) is one example. At the same time, we are building persistent recording databases and adding new features to Mneme. This portends binary compatibility hell, which this branch diverts us from.
We create an interface for reading/writing Mneme's binary formats, with versioned readers.