Repository navigation
Sync upstream v0.5.16 + mattn/SQLCipher CGO driver, per-DB keys, SkipJournalPatch, and hardening - #1
Merged
Merged
Conversation
Replace modernc.org/sqlite (pure Go) with mattn/go-sqlite3 (CGO) across the entire codebase. This enables SQLCipher page-level encryption via the 0xCarbon/go-sqlite3 fork's EncryptionKey field. Modernc is pure Go and cannot use SQLCipher (a C library). Alore requires CGO anyway (libtss, sqlite-vec, llama.cpp), so there is no build penalty. Changes: - litestream.go: Import mattn, register "litestream-sqlite3" driver with ConnectHook for PERSIST_WAL. Expose litestreamDriver for EncryptionKey. - db.go: EncryptionKey field on DB. Set litestreamDriver.EncryptionKey before sql.Open. DSN uses mattn format (?_busy_timeout=). Add PRAGMA wal_autocheckpoint=0 as SQL (was in modernc DSN pragma). setPersistWAL uses mattn's SetFileControlInt. - replica.go: Thread EncryptionKey through checkpointV3, checkIntegrity. Swap driver name "sqlite" → "litestream-sqlite3". - cmd/litestream/main.go, examples, tests: Swap imports and driver name. - go.mod: Remove modernc deps, add 0xCarbon/go-sqlite3 replace. All V3 protocol, VFS compaction, follow mode, crash recovery, integrity validation, and other main-branch features are preserved unchanged. PoC validated: SQLCipher 4.14.0 + FTS5 + sqlite-vec + Litestream replicate/restore. Performance: 2-17% overhead (AES-NI).
… encryption key Upstream disables SQLite automatic WAL checkpoints via modernc's _pragma=wal_autocheckpoint(0) DSN syntax, which mattn/go-sqlite3 does not support; set the pragma in the litestream-sqlite3 ConnectHook instead so Litestream remains the sole checkpoint owner. Restores can construct a Replica without a DB instance (NewReplicaWithClient(nil, client)); the SQLCipher swap introduced r.db.EncryptionKey derefs on those paths. Add an accessor that treats a DB-less replica as unencrypted.
Convert the remaining test and example connections from the modernc "sqlite" driver to mattn "sqlite3" and drop the modernc dependency. The fork's production driver is mattn/go-sqlite3 with SQLCipher; keeping a second engine in tests exercised different behavior than production (modernc and mattn also disagree on WAL retention across mixed connections, which broke the upstream WAL-chunking tests).
…page-1 reads VFSFile.ReadAt rewrites bytes 18-19 of page 1 (and randomizes the change counter) to emulate a journal-mode header. That patch corrupts SQLCipher databases, where page 1 is ciphertext protected by an HMAC. Add VFS.SkipJournalPatch / VFSFile.SkipJournalPatch to disable the patch. The flag propagates from the VFS to files opened through it and into the background hydrator. All four patch sites (dirty buffer, page cache, storage read, hydrator) honor the flag. Also guard len(p) >= 28 at the previously unguarded cache and storage sites: a read shorter than the patch window from offset 0 used to panic. The dirty-buffer and hydrator sites already had the guard. Additionally, fix the vfs-tagged root test build: the test-only replica clients did not implement ReplicaClient.SetLogger, so 'go test -tags vfs .' failed to compile at upstream main.
Replace the package-global litestreamDriver (whose EncryptionKey field was mutated before each sql.Open) with one registered driver instance per DB: litestream-sqlite3-<seq>. Each pool now opens every connection with its own database's key, so concurrent DBs with different keys — amplified by upstream's concurrent sync executor — can no longer race the key onto the wrong connection. Add DB.EncryptionKeyBytes ([]byte, takes precedence over the EncryptionKey SQL-literal string) for consumers that hold raw keys. checkpointV3/checkIntegrity take a driver name; Replica resolves it from the owning DB and registers an unencrypted fallback driver once for DB-less restores. Tests: encrypted replicate->restore round-trip keyed by raw bytes with a wrong-key negative check; four concurrent differently-keyed DBs under -race.
…ix stale pure-Go docs The mattn/go-sqlite3 fork builds SQLCipher (SQLITE_HAS_CODEC + SQLCIPHER_CRYPTO_OPENSSL), so CGO and libcrypto are required: - pr-metrics: build with CGO_ENABLED=1 after installing gcc + libssl-dev - commit: drop build-windows (mingw-w64 has no libcrypto; Windows is unsupported until a cross-CGO toolchain is wired up) - goreleaser: CGO_ENABLED=1, linux amd64+arm64 only; release workflow installs libssl-dev (+arm64 multiarch) - Dockerfile: install libssl-dev zlib1g-dev libzstd-dev and pass CGO_LDFLAGS='-lz -lzstd' for the static build (dynamic libcrypto pulls zlib/zstd via DT_NEEDED; archives do not) Docs: replace 'pure Go, no CGO' claims (AGENTS.md, docs/PATTERNS.md, docs/SQLITE_INTERNALS.md, skills) with the CGO/SQLCipher reality and add a Fork Deltas section documenting the driver scheme, encryption key fields, SkipJournalPatch, and the pin-branch policy.
The c-archive embeds SQLCipher (SQLCIPHER_CRYPTO_OPENSSL), so the extension link needs -lcrypto; without it the .so builds but dlopen fails on the first EVP_* symbol. gcc -shared permits undefined symbols, so 'file' checks stayed green while the artifact was unloadable. Also strip debug info from the extension links (84MB -> 46MB) and add a CI step that loads the built .so into the sqlite3 CLI — the oracle that actually catches this failure class.
…crypted DBs
applyLTXFile rewrote page-1 bytes 18-19/24-27 on every applied LTX file.
For SQLCipher databases page 1 is ciphertext; the rewrite breaks the
HMAC and follow-mode restore (Restore{Follow:true}, fillFollowGap,
crash-recovery resume) produces unrecoverable databases. Gate the patch
on the replica not being encrypted, mirroring VFS.SkipJournalPatch.
Plaintext behavior is unchanged and covered by the control case.
The mattn fork driver forces PRAGMA synchronous=NORMAL on every connection; upstream (modernc) ran FULL. For a disaster-recovery tool that silently weakens durability, so pin _sync=FULL in the DSN used for litestream's pool. Document the consumer-side NORMAL default and the encryption-blind CLI in AGENTS.md Fork Deltas.
The driver interpolates EncryptionKey into PRAGMA key = %s;. SQLite's PRAGMA grammar accepts only string literals on the right-hand side, so a bare x'…' blob literal (the form go-alore passes today) was a syntax error and only the double-quoted "x'…'" form worked. Normalize the bare blob form to the quoted SQLCipher convention at driver construction; quoted values pass through. Document the accepted forms and cover them with round-trip tests (bare blob, quoted blob, quoted passphrase).
…stry Replace sql.Register-based per-DB drivers with a database/sql Connector wrapper (sqliteConnector) opened via sql.OpenDB. Registrations are never removed, so the registry retained SQLCipher key material for the process lifetime and leaked one keyed entry per open/close cycle; the connector pool is garbage-collected with the DB. EncryptionKeyBytes is cloned into the driver so callers may zero their slice after Open. Also drop the now-redundant setPersistWAL Raw call and the duplicate wal_autocheckpoint ExecContext: the driver's ConnectHook already applies both to every pooled connection.
The library examples still built modernc's _pragma=busy_timeout(...) DSN syntax, which the mattn driver silently ignores — the pragmas were no-ops. Use the mattn parameters (_busy_timeout, _journal_mode). The directory-watcher stress test still opened the removed "sqlite" driver, failing at runtime under -tags stress.
…path checkpointV3 closes its pool with PERSIST_WAL set, leaving a 0-byte -wal/-shm next to restore outputs; remove them (restore outputs are single-file databases; WAL rotation recreates sidecars as needed). RestoreV3's failure path now also removes the tmp -wal/-shm sidecars. Add the cache-hit variant of the SkipJournalPatch test: a second 28+-byte read of page 1 is served from the page cache and must leave the header untouched with the flag set.
… cross-CC - commit.yml: new VFS Tests job (go test -tags=vfs .) — the SkipJournal Patch/short-read suite was never executed in CI. - Drop the macOS vfs build job, the release darwin VFS build/packaging legs, and the Makefile darwin targets: the fork's unconditional SQLCIPHER_CRYPTO_OPENSSL needs OpenSSL headers the stock runners don't provide, and the fork ships Linux-only. - goreleaser: set CC=aarch64-linux-gnu-gcc for the linux/arm64 build (cross-CGO otherwise assembles host-arch objects and fails), and fix the stale supported-platforms header.
mustExec reports with Errorf (FailNow is invalid off the test goroutine); VFS.SkipJournalPatch doc names both patched ranges; AGENTS.md formatting.
PR Build Metrics✅ All clear — no issues detected
Binary Size
Dependency ChangesAdded:
Removed:
govulncheck OutputBuild Info
History (2 previous)
🤖 Updated on each push. |
… paths External review findings: - release: vfs-build-linux arm64 leg installs multiarch libssl-dev:arm64 (the extension link now needs -lcrypto; cross-gcc alone cannot find arm64 OpenSSL headers/libs) - release: drop the remaining darwin legs and metadata — publish-gem matrix entries (downloaded artifacts that no longer exist), goreleaser release-notes platform rows, npm darwin package dirs + optionalDependencies + index.js platform map (empty packages would publish and getLoadablePath would resolve missing files on macOS), docs/VFS.md darwin make targets - vfs extension: LITESTREAM_SKIP_JOURNAL_PATCH env toggle (SQLCipher replicas through the distributed .so must skip the page-1 patch or the HMAC breaks); shared envFlag helper + unit test; document env vars incl. the previously undocumented hydration toggles - AGENTS.md: document the mandatory consumer-side go-sqlite3 replace (Go ignores this module's replaces when imported; without it the build fails on EncryptionKey/EncryptionKeyBytes)
canesin
marked this pull request as ready for review
August 17, 2026 07:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Syncs the fork to upstream
main(v0.5.16+2, 97d2dfa — already fast-forwarded plus tags v0.5.11..v0.5.16) and lands our CGO/SQLCipher work on top as the fork's delta, with the fixes and hardening identified along the way.This PR contains only our delta (16 commits); the upstream sync itself was a pure fast-forward on
main.What's here
modernc.org/sqlite→mattn/go-sqlite3via the 0xCarbon fork (replace aligned with consumers,c8f057756220), SQLCipher keys applied as the first statement of every connection.sql.OpenDB(connector)per database — kills the shared-driver key race (amplified by upstream's concurrent sync executor), never registers names, never retains keys pastClose;EncryptionKeyBytesis cloned into the driver.DB.EncryptionKeyBytes(3a22103/2c72181): raw-byte keys (takes precedence over the string form) — unblocks go-alore TODO(fix(test): resolve flaky TestStore_CompactDB/L1 timing issue benbjohnson/litestream#920).VFS/VFSFile.SkipJournalPatch(b65e198): gates all four page-1 journal-emulation patch sites (dirty/cache/storage/hydrator); adds the missinglen(p)>=28guards that panicked on short reads at offset 0 (oca worked around this).applyLTXFilerewrote page-1 bytes on every applied LTX — unrecoverable HMAC corruption for encrypted DBs; now gated on!encryptedwith regression test (control case keeps plaintext behavior).synchronous=NORMAL; litestream's own connections pin_sync=FULL(upstream parity).x'…'blob keys were a PRAGMA syntax error (only the quoted form worked — what go-alore passes today); auto-quoted now, round-trip tested for bare/quoted blob + quoted passphrase forms..soembedded SQLCipher but linked without-lcrypto— built fine, faileddlopenon the firstEVP_*symbol; fixed, stripped (84→46MB), and a CI step now loads it into thesqlite3CLI.SetLoggeron test clients).Verification
go test ./...,go test -race -count=1 ./...,go test -tags vfs . -count=1— all greenstaticcheck ./...,go vet ./..., goimports cleanmake vfs-linux-amd64+ load oracle: extension loads into a real SQLite host (rc=0)docker builddefault + hardened targets passgo build -tags vfs ./...+go test -tags vfs ./gofer/...; go-alorego build/vet -tags vfs ./internal/adapter/litestream/...— all green-raceKnown issues (not introduced here)
TestVFS_BlobData(cmd/litestream-vfs) is order-dependent and fails identically at upstreammain— pre-existing upstream defect.Follow-ups (separate waves)
main(lockstep, their repos)pin/<merged-sha>branch will be created after merge;pin/7a606e384ee9stays untouched_keyDSN param (oca builds?_key=today, silently ignored)