feat: Add Write-Ahead Log for Crash Durability - #19
Merged
Conversation
Append-only log with length-prefixed records, a CRC32 on each payload, and fsync before append returns. Open replays complete records and truncates a torn or checksum-mismatched tail. DurableWorkQueue logs enqueue, ack, and drop before the in-memory mutation so a restart redelivers unacked work and does not resurrect acked work.
Checksum the length prefix with the payload so eight zero bytes are not a valid empty record. Honor short read and write counts. Mark in-flight durable deliveries settled on unsubscribe without writing ack or drop.
ThomasHartDev
force-pushed
the
thomas/feat/wal-durability
branch
from
September 4, 2026 19:32
a2d74bf to
3a72703
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kafka, Postgres, and SQLite all append to a log before they treat a write as durable. This adds that piece: a length-prefixed WAL with IEEE CRC32 over each length prefix and payload, fsync before append returns, and recovery that replays complete records then truncates at the first bad CRC or incomplete header. Eight zero bytes at the tail are not a valid record, and nothing after that hole is replayed.
DurableWorkQueue writes enqueue, ack, and drop records before it touches the in-memory WorkQueue. A restart redelivers unacked work and does not resurrect acked work. Unsubscribing requeues in-flight work without writing ack or drop, so a stale ack or nack on the old Delivery cannot erase it from the log. checkpoint() rewrites the file to the live enqueue records via a temp file, fsync, and rename. The package barrel also exports the dead-letter types again, which the work-queue merge dropped.
Closes #18