Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .goreleaser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ builds:
- id: litestream
main: ./cmd/litestream
binary: litestream
# The fork's mattn/go-sqlite3 driver builds SQLCipher (CGO, links
# The 0xCarbon/go-sqlite3 fork builds SQLCipher (CGO, links
# libcrypto); pure-Go builds are no longer possible. Windows, macOS, and
# 32-bit ARM targets are dropped until cross-CGO toolchains with libcrypto
# are wired up; fork consumers are Linux.
Expand Down
28 changes: 15 additions & 13 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# AGENTS.md - Litestream AI Agent Guide

Litestream is a disaster recovery tool for SQLite that runs as a background process, monitors the WAL, converts changes to immutable LTX files, and replicates them to cloud storage. This fork uses `mattn/go-sqlite3` (via the 0xCarbon fork with bundled SQLCipher) — CGO is required and libcrypto is linked.
Litestream is a disaster recovery tool for SQLite that runs as a background process, monitors the WAL, converts changes to immutable LTX files, and replicates them to cloud storage. This fork uses `github.com/0xCarbon/go-sqlite3` (the mattn-lineage fork with bundled SQLCipher) — CGO is required and libcrypto is linked.

## Before You Start

Expand Down Expand Up @@ -76,22 +76,24 @@ Before submitting changes:

This fork tracks upstream `benbjohnson/litestream` and adds:

- **Driver**: per-database `mattn/go-sqlite3` driver instances (registered as
`litestream-sqlite3-<seq>`), each carrying its database's SQLCipher key;
see `registerSQLiteDriver` in `litestream.go`.
- **Driver**: per-database `github.com/0xCarbon/go-sqlite3` driver instances
(private `sql.OpenDB` connectors, see `newSQLiteDriver`/`newSQLitePool` in
`litestream.go`), each carrying its database's SQLCipher key.
- **Encryption**: `DB.EncryptionKey` (string: bare `x'…'` raw-key blob is
auto-quoted; quoted literals `'passphrase'`/`"x'…'"` pass through) and
`DB.EncryptionKeyBytes` (raw bytes; takes precedence). Keys are applied as
the first statement of every pooled connection; set before Open.
`DB.EncryptionKeyBytes` (raw bytes; takes precedence; must be exactly 32,
48, or 80 bytes — other lengths are rejected at the first connection).
Keys are applied as the first statement of every pooled connection; set
before Open.
- **VFS**: `VFS.SkipJournalPatch` / `VFSFile.SkipJournalPatch` disable the
page-1 journal-mode emulation patch, required for SQLCipher databases
(plaintext bytes 18-19/24-28 must not be rewritten on ciphertext pages).
- **Mandatory consumer replace**: Go ignores this module's `replace`
directives when litestream is imported as a dependency. Every consuming
module MUST carry its own replace or the build fails with
`EncryptionKey`/`EncryptionKeyBytes` undefined on the upstream driver:

replace github.com/mattn/go-sqlite3 => github.com/0xCarbon/go-sqlite3 v1.14.38-0.20260515201505-c8f057756220
- **Driver dependency**: litestream requires the 0xCarbon SQLCipher fork
directly (`github.com/0xCarbon/go-sqlite3`) — no `replace` directive is
needed or used. Consumers must use the same module path for their own
direct sqlite use: importing both `github.com/mattn/go-sqlite3` and
`github.com/0xCarbon/go-sqlite3` in one binary registers the "sqlite3"
driver twice and panics at init.

- **Pin policy**: consumer modules pin this fork via the `pin/<sha>` branches
(e.g. `pin/7a606e384ee9`); never force-push or delete them.
Expand All @@ -102,7 +104,7 @@ This fork tracks upstream `benbjohnson/litestream` and adds:
stock runners do not provide; fork consumers are Linux.
- **Durability**: litestream's own connections run `synchronous=FULL` (DSN
`_sync=FULL`), matching upstream's modernc default. Consumer connections
through the mattn fork default to `synchronous=NORMAL` — pass `_sync=FULL`
through the 0xCarbon fork (mattn-lineage) default to `synchronous=NORMAL` — pass `_sync=FULL`
in your DSN if you need the stricter behavior.
- **CLI**: the `litestream` binary is encryption-blind — there is no key
configuration; SQLCipher is a library-only feature via `DB.EncryptionKey`
Expand Down
2 changes: 1 addition & 1 deletion _examples/library/basic/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import (
"syscall"
"time"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"

"github.com/benbjohnson/litestream"
"github.com/benbjohnson/litestream/file"
Expand Down
2 changes: 1 addition & 1 deletion _examples/library/library_example_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
"testing"
"time"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"

"github.com/benbjohnson/litestream"
"github.com/benbjohnson/litestream/file"
Expand Down
2 changes: 1 addition & 1 deletion _examples/library/s3/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import (
"syscall"
"time"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"

"github.com/benbjohnson/litestream"
"github.com/benbjohnson/litestream/s3"
Expand Down
2 changes: 1 addition & 1 deletion cmd/litestream-test/load.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
"syscall"
"time"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"
)

type LoadCommand struct {
Expand Down
2 changes: 1 addition & 1 deletion cmd/litestream-test/populate.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import (
"os"
"time"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"
)

type PopulateCommand struct {
Expand Down
2 changes: 1 addition & 1 deletion cmd/litestream-test/shrink.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
"os"
"time"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"
)

type ShrinkCommand struct {
Expand Down
2 changes: 1 addition & 1 deletion cmd/litestream-test/validate.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import (
"strings"
"time"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"
)

type ValidateCommand struct {
Expand Down
2 changes: 1 addition & 1 deletion cmd/litestream-vfs/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import (
"testing"
"time"

sqlite3 "github.com/mattn/go-sqlite3"
sqlite3 "github.com/0xCarbon/go-sqlite3"
"github.com/psanford/sqlite3vfs"
"github.com/stretchr/testify/require"

Expand Down
2 changes: 1 addition & 1 deletion cmd/litestream-vfs/time_travel_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import (
"testing"
"time"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"
"github.com/psanford/sqlite3vfs"
"github.com/stretchr/testify/require"

Expand Down
2 changes: 1 addition & 1 deletion cmd/litestream-vfs/vfs_soak_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import (
"testing"
"time"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"

"github.com/benbjohnson/litestream/file"
"github.com/benbjohnson/litestream/internal/testingutil"
Expand Down
2 changes: 1 addition & 1 deletion cmd/litestream/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ import (
"strings"
"time"

_ "github.com/0xCarbon/go-sqlite3"
"github.com/dustin/go-humanize"
_ "github.com/mattn/go-sqlite3"
"github.com/superfly/ltx"
_ "golang.org/x/crypto/x509roots/fallback"
"gopkg.in/yaml.v2"
Expand Down
11 changes: 7 additions & 4 deletions db.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import (
"syscall"
"time"

"github.com/mattn/go-sqlite3"
"github.com/0xCarbon/go-sqlite3"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/superfly/ltx"
Expand Down Expand Up @@ -198,8 +198,11 @@ type DB struct {
EncryptionKey string

// EncryptionKeyBytes is the raw SQLCipher key for this database. When
// non-empty it takes precedence over EncryptionKey. Applied to every
// connection of this DB's registered driver.
// non-empty it takes precedence over EncryptionKey. Must be exactly 32,
// 48, or 80 bytes (SQLCipher raw-key forms: key / key+salt /
// key+HMAC+salt); other lengths are rejected at the first connection by
// the 0xCarbon fork. Applied to every connection of this DB's driver;
// set before Open.
EncryptionKeyBytes []byte

// sqliteDriver is this DB's own driver instance, carrying its SQLCipher
Expand Down Expand Up @@ -1037,7 +1040,7 @@ func (db *DB) init(ctx context.Context) (err error) {

dsn := db.path
// _sync=FULL restores the durability litestream had with modernc: the
// mattn driver defaults every connection to synchronous=NORMAL, which
// go-sqlite3 fork defaults every connection to synchronous=NORMAL, which
// can lose the newest commits on power loss in WAL mode.
dsn += fmt.Sprintf("?_busy_timeout=%d&_sync=FULL", db.BusyTimeout.Milliseconds())

Expand Down
38 changes: 37 additions & 1 deletion db_internal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import (
"github.com/superfly/ltx"
"golang.org/x/sync/semaphore"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"

"github.com/benbjohnson/litestream/internal"
)
Expand Down Expand Up @@ -4689,3 +4689,39 @@ func TestDB_InitSetsSyncFull(t *testing.T) {
t.Fatalf("litestream pool connections must run synchronous=FULL (2), got %d", sync)
}
}

// TestDB_EncryptionKeyBytesLengthRejected asserts the driver's raw-key
// length contract surfaces through litestream's own pool: SQLCipher honors
// raw keys only at 32/48/80 bytes; the 0xCarbon fork rejects other lengths
// at connection open instead of silently KDF-falling-back.
func TestDB_EncryptionKeyBytesLengthRejected(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), "db")
setup, err := sql.Open("sqlite3", dbPath)
if err != nil {
t.Fatal(err)
}
if _, err := setup.Exec(`CREATE TABLE t (v TEXT);`); err != nil {
t.Fatal(err)
}
setup.Close()

db := NewDB(dbPath)
db.EncryptionKeyBytes = make([]byte, 16) // invalid: not 32/48/80
client := &testReplicaClient{dir: t.TempDir()}
r := NewReplicaWithClient(db, client)
r.MonitorEnabled = false
db.Replica = r
if err := db.Open(); err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close(context.Background()) }()

// db.Open does not connect; the first pooled connection (init) fails fast.
err = db.init(context.Background())
if err == nil {
t.Fatal("expected invalid raw key length to be rejected")
}
if !strings.Contains(err.Error(), "invalid raw key length") || !strings.Contains(err.Error(), "must be 32, 48, or 80 bytes") {
t.Fatalf("unexpected error: %v", err)
}
}
2 changes: 1 addition & 1 deletion docs/PATTERNS.md
Original file line number Diff line number Diff line change
Expand Up @@ -439,7 +439,7 @@ Lock page numbers by page size:

**Responsibilities:**

- Manages SQLite database connection (via `mattn/go-sqlite3`; CGO required, SQLCipher supported)
- Manages SQLite database connection (via `0xCarbon/go-sqlite3`; CGO required, SQLCipher supported)
- Monitors WAL for changes
- Performs checkpoints
- Maintains long-running read transaction
Expand Down
2 changes: 1 addition & 1 deletion docs/SQLITE_INTERNALS.md
Original file line number Diff line number Diff line change
Expand Up @@ -437,7 +437,7 @@ Litestream uses both database/sql and direct SQLite APIs:
// Using database/sql for queries
db, err := sql.Open("sqlite3", "database.db")

// Using mattn/go-sqlite3 for low-level access (CGO)
// Using 0xCarbon/go-sqlite3 for low-level access (CGO)
conn, err := sqlite3.Open("database.db")

// Direct page access (requires special builds)
Expand Down
2 changes: 1 addition & 1 deletion docs/VFS.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ import (
"database/sql"
"os"

_ "github.com/mattn/go-sqlite3"
_ "github.com/0xCarbon/go-sqlite3"
)

func main() {
Expand Down
66 changes: 65 additions & 1 deletion encryption_test.go
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
package litestream_test

import (
"bytes"
"context"
"database/sql"
"encoding/hex"
"fmt"
"os"
"path/filepath"
"sync"
"sync/atomic"
"testing"

"github.com/mattn/go-sqlite3"
"github.com/0xCarbon/go-sqlite3"

"github.com/benbjohnson/litestream"
"github.com/benbjohnson/litestream/file"
Expand Down Expand Up @@ -186,6 +188,68 @@ func TestDB_ConcurrentDifferentKeys(t *testing.T) {
wg.Wait()
}

// sqlcipher414FixtureKey is the raw key of testdata/sqlcipher-v4.14.0/fixture.db.
func sqlcipher414FixtureKey() []byte {
key := make([]byte, 32)
for i := range key {
key[i] = byte(i*7 + 3)
}
return key
}

// TestSQLCipher414FixtureDecrypts pins the cross-version ciphertext
// contract: databases written by the previous driver pin (0xCarbon fork
// c8f057756220, SQLCipher 4.14.0 / SQLite 3.51.3) must keep opening under
// the current pin (SQLCipher 4.17.0) with unchanged kdf/HMAC defaults.
// Regenerate the fixture only with a deliberate compatibility decision.
func TestSQLCipher414FixtureDecrypts(t *testing.T) {
fixturePath := filepath.Join("testdata", "sqlcipher-v4.14.0", "fixture.db")
fixture, err := os.ReadFile(fixturePath)
if err != nil {
t.Fatalf("read fixture: %v", err)
}
// Guard against an accidentally-plaintext fixture.
if bytes.Equal(fixture[:15], []byte("SQLite format 3")) {
t.Fatal("fixture is plaintext; regenerate with the recorded generator")
}

dbPath := filepath.Join(t.TempDir(), "db")
if err := os.WriteFile(dbPath, fixture, 0o600); err != nil {
t.Fatal(err)
}

// Litestream's own path: init reads the database with the key.
db := litestream.NewDB(dbPath)
db.EncryptionKeyBytes = sqlcipher414FixtureKey()
client := file.NewReplicaClient(t.TempDir())
r := litestream.NewReplicaWithClient(db, client)
r.MonitorEnabled = false
db.Replica = r
if err := db.Open(); err != nil {
t.Fatal(err)
}
if err := db.Sync(context.Background()); err != nil {
t.Fatalf("sync over 4.14-written database: %v", err)
}
if err := db.Close(context.Background()); err != nil {
t.Fatal(err)
}

// And the data survives intact through the current driver.
verifier, err := sql.Open(registerKeyedSQLite(t, sqlcipher414FixtureKey()), dbPath)
if err != nil {
t.Fatal(err)
}
defer verifier.Close()
var payload string
if err := verifier.QueryRow(`SELECT payload FROM fixture ORDER BY id LIMIT 1`).Scan(&payload); err != nil {
t.Fatalf("read fixture rows: %v", err)
}
if payload != "sqlcipher-4.14-compat" {
t.Fatalf("payload = %q", payload)
}
}

// TestDB_EncryptionKeyStringForms verifies the accepted string key forms
// through a full replicate/restore round trip. The driver interpolates the
// string into `PRAGMA key = %s;`, so the bare blob form must be auto-quoted
Expand Down
4 changes: 1 addition & 3 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ require (
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/mark3labs/mcp-go v0.32.0
github.com/mattn/go-shellwords v1.0.12
github.com/mattn/go-sqlite3 v1.14.24
github.com/nats-io/nats.go v1.44.0
github.com/pkg/sftp v1.13.6
github.com/prometheus/client_golang v1.17.0
Expand Down Expand Up @@ -51,6 +50,7 @@ require (
)

require (
github.com/0xCarbon/go-sqlite3 v1.14.38-0.20260817152354-d1c4d3294fa8
github.com/lmittmann/tint v1.1.3
github.com/mattn/go-isatty v0.0.20
github.com/pierrec/lz4/v4 v4.1.23
Expand Down Expand Up @@ -131,5 +131,3 @@ require (
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

replace github.com/mattn/go-sqlite3 => github.com/0xCarbon/go-sqlite3 v1.14.38-0.20260515201505-c8f057756220
Loading
Loading