diff --git a/README.md b/README.md index a9d1f89..966082f 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ Most libraries that call themselves *lite* got there by leaving features out. Li - **🧩 Two paradigms, one core β€” no library lock-in.** The `query` builder (typed, low-magic) and the `orm` (declarative, tag-driven) share one `Session`, one dialect, one scanner, and one set of normalized errors. A row fetched via `orm` feeds `query` on the *same* transaction. Most stacks make you pick a camp; LiteORM lets each part of your app pick the right tool. -- **⚑ Modern baseline, lean core.** Generics-first: typed result rows and typed `Col[V]` predicates, with no reflection on the hot path. The core module pulls in **no** database drivers β€” each backend (`dialect/sqlite`, `dialect/postgres`, `dialect/mysql`, `dialect/mssql`) is its own module, so your build only carries the driver you use. SQLite is pure Go: cross-compile with plain `go build`, ship static distroless/alpine binaries, no C toolchain. `iter.Seq2` streaming, `log/slog` logging, and `gopls modernize` enforced in CI keep it from drifting backward. +- **⚑ Modern baseline, lean core.** Generics-first: typed result rows and typed `Col[V]` predicates, with no reflection on the hot path. The core module pulls in **no** database drivers β€” each backend (`dialect/sqlite`, `dialect/postgres`, `dialect/mysql`, `dialect/mssql`) is its own module, so your build only carries the driver you use. SQLite is pure Go: cross-compile with plain `go build`, ship static distroless/alpine binaries, no C toolchain β€” and open it **[encrypted at rest](docs/guides/encryption.md)** with a key, transparent page-level encryption (Adiantum) without a CGo sqlcipher build. `iter.Seq2` streaming, `log/slog` logging, and `gopls modernize` enforced in CI keep it from drifting backward. - **πŸ€– Built for agents β€” AI-native two ways.** The embedded [**studio**](docs/guides/studio.md) turns plain English into SQL through one server-side `WithAI` hook to any model, so your API key never reaches the browser β€” no other Go ORM ships this. *Separately*, the repo ships [**Agent Skills**](skills/) and an [`AGENTS.md`](AGENTS.md) so an AI coding assistant writes correct LiteORM without guessing. AI for the people *using* your database and the people *building* against it. @@ -65,7 +65,8 @@ Most libraries that call themselves *lite* got there by leaving features out. Li - **[Migrations](docs/guides/migrations.md)** β€” additive `AutoMigrate` plus a two-track model: destructive changes become a *reviewable* migration you apply through a thin runner that reads golang-migrate / goose / plain SQL files. - **[Normalized errors](docs/guides/errors.md)** β€” `ErrUniqueViolation`, `ErrForeignKey`, `ErrNotNull`, `ErrCheck`, `ErrNoRows`, `ErrDeadlock`, `ErrSerialization` β€” the same on SQLite, Postgres, MySQL, and SQL Server. - **[Code generation](docs/guides/codegen.md)** β€” typed `Column[V]` constants for compile-time column safety, models from a live DB, SQLβ†’typed-Go from annotated queries, a sqlc process plugin, and a gormβ†’LiteORM tag porter. -- **[SQLite vector / full-text / hybrid search](docs/guides/sqlite-search.md)** β€” typed vector (sqlite-vec) and FTS5 search, plus reciprocal-rank-fusion **hybrid** search, keyed by your model's primary key; encryption at rest. +- **[SQLite vector / full-text / hybrid search](docs/guides/sqlite-search.md)** β€” typed vector (sqlite-vec) and FTS5 search, plus reciprocal-rank-fusion **hybrid** search, keyed by your model's primary key. +- **[At-rest encryption](docs/guides/encryption.md)** β€” open an encrypted SQLite database with a 32-byte key: transparent, page-level encryption (Adiantum), **pure Go β€” no CGo sqlcipher**. The on-disk file is ciphertext; the `query` builder, `orm`, and migrations all work unchanged above it. - **[SQLite changesets](docs/guides/sqlite-changeset.md)** β€” capture, apply, invert, and concat changesets for audit logs, one-way replication, and undo. - **[Postgres extras](docs/guides/postgres.md)** β€” LISTEN/NOTIFY and typed JSONB (`->`, `->>`, `@>`) and array (`@>`, `&&`, `= ANY`) operators. @@ -78,6 +79,7 @@ Read top-down: who LiteORM is, what it's built for, then the capability depth be | Core runtime model | generics, no reflection | reflection | reflection | generated code | generated code | | Explicit query builder **and** declarative ORM in one lib | βœ“ | βœ— (ORM) | βœ“ builder (light ORM) | βœ— (SQLβ†’Go) | βœ— (generated ORM) | | CGo-free SQLite (pure Go, no C toolchain) | βœ“ | driver of choice | driver of choice | driver of choice | driver of choice | +| At-rest encrypted SQLite, **pure Go** (no CGo sqlcipher) | βœ“ | βœ— | βœ— | βœ— | βœ— | | Ships an **embedded database studio** (admin GUI) | βœ“ | βœ— | βœ— | βœ— | βœ— | | Studio with **built-in AI** (NLβ†’SQL, English filters, result charts) | βœ“ | βœ— | βœ— | βœ— | βœ— | | Ships AI Agent Skills + task-oriented docs | βœ“ | βœ— | βœ— | βœ— | βœ— | @@ -149,7 +151,7 @@ Full walkthrough in **[Getting started](docs/getting-started.md)**. ## Examples -Runnable, smoke-tested programs under [`examples/`](examples/): `blog` (a small end-to-end blog engine β€” models, associations, nested eager loading, an aggregate query, and a transaction), `query` and `orm` feature showcases, `logging` (statement tracing), `search` (vector + full-text + hybrid), `queries` (SQLβ†’Go codegen), `gormport` (the gorm porter), and `codegen`. `just example ` runs one; `just examples` runs them all. +Runnable, smoke-tested programs under [`examples/`](examples/): `blog` (a small end-to-end blog engine β€” models, associations, nested eager loading, an aggregate query, and a transaction), `query` and `orm` feature showcases, `logging` (statement tracing), `search` (vector + full-text + hybrid), `encryption` (at-rest encrypted SQLite), `queries` (SQLβ†’Go codegen), `gormport` (the gorm porter), and `codegen`. `just example ` runs one; `just examples` runs them all. ## Development & contributing diff --git a/cmd/sqlc-gen-liteorm/generate.go b/cmd/sqlc-gen-liteorm/generate.go index ea26c5e..78e1e66 100644 --- a/cmd/sqlc-gen-liteorm/generate.go +++ b/cmd/sqlc-gen-liteorm/generate.go @@ -54,7 +54,7 @@ func emit(pkg string, queries []*plugin.Query) (string, error) { needQuery, needErrors, needTime := false, false, false for _, q := range queries { - cmd := gen.QueryCmd(normCmd(q.GetCmd())) + cmd := gen.QueryCmd(strings.TrimPrefix(q.GetCmd(), ":")) // sqlc cmds are ":one"/":many"/… // Result type: a generated row struct for multi-column reads, a scalar // for one. A zero-column :one/:many is degenerate β€” treat it as :exec. @@ -125,8 +125,6 @@ func emitRowStruct(b *strings.Builder, name string, cols []*plugin.Column, needT b.WriteString("}\n\n") } -func normCmd(cmd string) string { return strings.TrimPrefix(cmd, ":") } - func argName(p *plugin.Parameter, i int) string { if c := p.GetColumn(); c != nil && c.GetName() != "" { return lowerCamel(c.GetName()) diff --git a/conformance/orm_test.go b/conformance/orm_test.go index e7be22c..e2a28e7 100644 --- a/conformance/orm_test.go +++ b/conformance/orm_test.go @@ -120,8 +120,8 @@ type fkChild struct { func (fkChild) TableName() string { return "fk_children" } -func (a *Account) BeforeCreate(_ context.Context, op *orm.Op[Account]) error { - op.Model.Note = "hooked" +func (a *Account) BeforeCreate(_ context.Context, ev *orm.Event[Account]) error { + ev.Model.Note = "hooked" return nil } @@ -820,7 +820,7 @@ func ormScenarios(t *testing.T, db *liteorm.DB) { } // Idempotent: a second migrate of the same model must not error (no duplicate). if err := orm.AutoMigrate[widgetIndexed](ctx, db); err != nil { - t.Errorf("re-migrate should be a no-op, got %v", err) + t.Errorf("re-migrate should be a no-ev, got %v", err) } }) diff --git a/conformance/ormsuite/hooks_test.go b/conformance/ormsuite/hooks_test.go index 8c77d6f..57d21e8 100644 --- a/conformance/ormsuite/hooks_test.go +++ b/conformance/ormsuite/hooks_test.go @@ -18,12 +18,12 @@ type Hooked struct { func (Hooked) TableName() string { return "hooked" } // BeforeCreate derives a slug; returning an error from a hook aborts the write. -func (h *Hooked) BeforeCreate(_ context.Context, op *orm.Op[Hooked]) error { - if op.Model.Name == "boom" { +func (h *Hooked) BeforeCreate(_ context.Context, ev *orm.Event[Hooked]) error { + if ev.Model.Name == "boom" { return errors.New("hook abort") } - if op.Model.Slug == "" { - op.Model.Slug = "slug-" + op.Model.Name + if ev.Model.Slug == "" { + ev.Model.Slug = "slug-" + ev.Model.Name } return nil } diff --git a/conformance/ormsuite/multitenant_test.go b/conformance/ormsuite/multitenant_test.go index c041479..3658771 100644 --- a/conformance/ormsuite/multitenant_test.go +++ b/conformance/ormsuite/multitenant_test.go @@ -23,10 +23,10 @@ type Doc struct { func (Doc) TableName() string { return "docs" } // BeforeCreate stamps the tenant from context when the caller left it unset. -func (d *Doc) BeforeCreate(ctx context.Context, op *orm.Op[Doc]) error { - if op.Model.TenantID == 0 { +func (d *Doc) BeforeCreate(ctx context.Context, ev *orm.Event[Doc]) error { + if ev.Model.TenantID == 0 { if tid, ok := tenantFrom(ctx); ok { - op.Model.TenantID = tid + ev.Model.TenantID = tid } } return nil diff --git a/dialect/dialect.go b/dialect/dialect.go index 8056325..f5e325a 100644 --- a/dialect/dialect.go +++ b/dialect/dialect.go @@ -141,6 +141,81 @@ type ForeignKeyIntrospector interface { ForeignKeysQuery() string } +// SearchKind distinguishes the sidecar kinds a [SearchProvisioner] builds. +type SearchKind int + +const ( + // SearchFullText is a full-text sidecar (e.g. SQLite FTS5). + SearchFullText SearchKind = iota + // SearchVector is a nearest-neighbour vector sidecar (e.g. sqlite-vec vec0). + SearchVector +) + +// SearchSpec is a dialect-neutral description of one search sidecar, resolved +// from a model's declared index (columns already mapped to their SQL names). It +// is the hand-off type between the orm front-end and a [SearchProvisioner] +// backend, so the dialect package needs no dependency on orm. +type SearchSpec struct { + Kind SearchKind + Name string // sidecar table name + Table string // base table the sidecar mirrors + Columns []string // indexed columns (text columns for full-text; the embedding column for vector) + PKColumn string // base-table primary-key column the sidecar is keyed by + PKType string // canonical Go type of the PK ("int64", "string", …) + Sync string // resolved sync strategy: "triggers" or "hooks" + + // Full-text options. + Tokenizer string + Prefix []int + Detail string // "full" | "column" | "none" + Content string // "external" (default) | "intable" | "contentless" + + // Vector options. + Dim int + Metric string // distance token: "l2" | "cosine" | "l1" | "hamming" + Encoding string // "float32" (default) | "int8" | "bit" +} + +// RowidKeyed reports whether the sidecar is keyed by the table's implicit integer +// rowid β€” true for any integer primary key (int, int32, uint, …, not just int64), +// since SQLite aliases an integer PK to the rowid. A non-integer PK (e.g. string) +// is keyed by an explicit key column instead. +func (s SearchSpec) RowidKeyed() bool { + switch s.PKType { + case "int", "int8", "int16", "int32", "int64", + "uint", "uint8", "uint16", "uint32", "uint64": + return true + } + return false +} + +// SearchProvisioner is an optional capability: a backend that supports search +// sidecars (full-text / vector) returns the DDL to create or drop one. The +// methods only build SQL β€” the caller (orm's AutoMigrate) executes it β€” so the +// dialect stays free of any session or orm dependency. Creation DDL must be +// idempotent (IF NOT EXISTS); drop DDL must tolerate a missing sidecar. +type SearchProvisioner interface { + ProvisionSearchSQL(spec SearchSpec) ([]string, error) + DropSearchSQL(spec SearchSpec) ([]string, error) +} + +// SearchStmt is one SQL statement plus its bind arguments, produced by a +// [SearchRowSyncer] to maintain a sidecar for a single row. +type SearchStmt struct { + SQL string + Args []any +} + +// SearchRowSyncer is an optional capability for keeping a hook-synced sidecar in +// step from the ORM write path β€” used when a sidecar is not trigger-synced (e.g. +// a vector whose embedding is sidecar-only, not a stored base column). value +// carries the indexed payload for the row: a vector embedding as []float32, or a +// pre-encoded []byte. +type SearchRowSyncer interface { + UpsertSearchRowSQL(spec SearchSpec, key any, value any) ([]SearchStmt, error) + DeleteSearchRowSQL(spec SearchSpec, key any) ([]SearchStmt, error) +} + // Dialect is the contract each backend implements. AppendPlaceholder is an // explicit, position-aware bind-var writer (sqlite/mysql -> "?", pg -> "$n", // mssql -> "@pN"), so a generics-first builder emits the right placeholder diff --git a/dialect/sqlite/plural_test.go b/dialect/sqlite/plural_test.go new file mode 100644 index 0000000..71e738f --- /dev/null +++ b/dialect/sqlite/plural_test.go @@ -0,0 +1,49 @@ +package sqlite_test + +import ( + "context" + "path/filepath" + "testing" + + "liteorm.org/dialect/sqlite" + "liteorm.org/orm" + "liteorm.org/query" +) + +type plainWidget struct { + ID int64 + Name string +} + +// TestPluralTableNames_OrmQueryAgree locks the fix that the plural-table-name +// setting applies to BOTH front-ends: orm.AutoMigrate creates the pluralized +// table and query.Select must read from the same one (not the singular). +func TestPluralTableNames_OrmQueryAgree(t *testing.T) { + ctx := context.Background() + db, err := sqlite.Open(filepath.Join(t.TempDir(), "plural.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + + orm.UsePluralTableNames(true) + t.Cleanup(func() { orm.UsePluralTableNames(false) }) + + // orm side: provisions "plain_widgets". + if err := orm.AutoMigrate[plainWidget](ctx, db); err != nil { + t.Fatalf("AutoMigrate: %v", err) + } + if err := orm.NewRepo[plainWidget](db).Create(ctx, &plainWidget{Name: "gear"}); err != nil { + t.Fatalf("create: %v", err) + } + + // query side: if it targeted the singular "plain_widget" it would error with + // "no such table"; agreement means it reads the row orm wrote. + got, err := query.Select[plainWidget](db).All(ctx) + if err != nil { + t.Fatalf("query.Select under plural naming: %v (orm and query disagree on the table name)", err) + } + if len(got) != 1 || got[0].Name != "gear" { + t.Fatalf("query.Select returned %v, want one widget 'gear'", got) + } +} diff --git a/dialect/sqlite/regexp.go b/dialect/sqlite/regexp.go new file mode 100644 index 0000000..1f62687 --- /dev/null +++ b/dialect/sqlite/regexp.go @@ -0,0 +1,23 @@ +package sqlite + +import "gosqlite.org/ext/regexp" + +// WhereRegex returns a query-builder WHERE fragment (and its bind args) that +// matches column against an RE2 regular expression. When pattern is left-anchored +// it prepends a GLOB prefix so SQLite can range-scan an index on column and run +// REGEXP only on the survivors; an unanchored pattern falls back to a plain +// REGEXP filter. Pass the result straight to the query builder: +// +// frag, args := sqlite.WhereRegex("title", `^Intro to .* with Go$`) +// rows, err := query.Select[Doc](db).Where(frag, args...).All(ctx) +// +// The REGEXP operator must be registered on the connection β€” blank-import +// gosqlite.org/ext/regexp/auto (gosqlite registers it globally, so it then works +// through liteorm with no further wiring). +func WhereRegex(column, pattern string) (sql string, args []any) { + prefix := regexp.GlobPrefix(pattern) + if prefix == "" || prefix == "*" { // unanchored or unusable: plain REGEXP + return column + " REGEXP ?", []any{pattern} + } + return column + " GLOB ? AND " + column + " REGEXP ?", []any{prefix, pattern} +} diff --git a/dialect/sqlite/regexp_test.go b/dialect/sqlite/regexp_test.go new file mode 100644 index 0000000..91fb052 --- /dev/null +++ b/dialect/sqlite/regexp_test.go @@ -0,0 +1,95 @@ +package sqlite_test + +import ( + "context" + "path/filepath" + "strings" + "testing" + + _ "gosqlite.org/ext/regexp/auto" // global: registers the RE2 REGEXP operator on every connection + "liteorm.org/dialect/sqlite" + "liteorm.org/query" +) + +type account struct { + ID int64 + Email string +} + +func (account) TableName() string { return "accounts" } + +// gosqlite registers scalar functions globally (via a connect hook), so a +// function like REGEXP flows through liteorm with no liteorm-specific glue β€” this +// is what makes gosqlite's gorm-specific regexp wiring unnecessary here. +func TestRegexp_WorksInQueryPredicate(t *testing.T) { + ctx := context.Background() + db, err := sqlite.Open(filepath.Join(t.TempDir(), "rx.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + + if _, err := db.ExecContext(ctx, `CREATE TABLE accounts (id INTEGER PRIMARY KEY, email TEXT)`); err != nil { + t.Fatal(err) + } + for _, e := range []string{"a@example.com", "b@example.org", "c@example.com"} { + if _, err := db.ExecContext(ctx, `INSERT INTO accounts(email) VALUES (?)`, e); err != nil { + t.Fatal(err) + } + } + + got, err := query.Select[account](db). + Where("email REGEXP ?", `^.*@example\.com$`). + All(ctx) + if err != nil { + t.Fatalf("REGEXP query: %v", err) + } + if len(got) != 2 { + t.Fatalf("REGEXP matched %d rows, want 2 (the .com addresses)", len(got)) + } + for _, a := range got { + if a.Email[len(a.Email)-4:] != ".com" { + t.Errorf("unexpected match %q", a.Email) + } + } +} + +func TestWhereRegex_GlobOptimization(t *testing.T) { + // Anchored pattern -> GLOB prefix + REGEXP residual (index-friendly). + frag, args := sqlite.WhereRegex("email", `^a@`) + if !strings.Contains(frag, "GLOB") || !strings.Contains(frag, "REGEXP") { + t.Errorf("anchored pattern should emit GLOB+REGEXP, got %q", frag) + } + if len(args) != 2 { + t.Errorf("anchored pattern should bind prefix+pattern, got %v", args) + } + // Unanchored -> plain REGEXP, single bind. + frag, args = sqlite.WhereRegex("email", `example`) + if strings.Contains(frag, "GLOB") || len(args) != 1 { + t.Errorf("unanchored pattern should be a plain REGEXP, got %q %v", frag, args) + } + + // End-to-end: the optimized clause selects the same rows as plain REGEXP. + ctx := context.Background() + db, err := sqlite.Open(filepath.Join(t.TempDir(), "rx2.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + if _, err := db.ExecContext(ctx, `CREATE TABLE accounts (id INTEGER PRIMARY KEY, email TEXT)`); err != nil { + t.Fatal(err) + } + for _, e := range []string{"a@example.com", "ab@example.com", "b@example.com"} { + if _, err := db.ExecContext(ctx, `INSERT INTO accounts(email) VALUES (?)`, e); err != nil { + t.Fatal(err) + } + } + frag, args = sqlite.WhereRegex("email", `^a@`) + got, err := query.Select[account](db).Where(frag, args...).All(ctx) + if err != nil { + t.Fatalf("WhereRegex query: %v", err) + } + if len(got) != 1 || got[0].Email != "a@example.com" { + t.Errorf("WhereRegex `^a@` matched %d rows, want only a@example.com", len(got)) + } +} diff --git a/dialect/sqlite/search/intpk_test.go b/dialect/sqlite/search/intpk_test.go new file mode 100644 index 0000000..f2dd233 --- /dev/null +++ b/dialect/sqlite/search/intpk_test.go @@ -0,0 +1,56 @@ +package search_test + +import ( + "context" + "path/filepath" + "testing" + + _ "gosqlite.org/vec" + "liteorm.org/dialect/sqlite" + "liteorm.org/dialect/sqlite/search" + "liteorm.org/orm" +) + +// intDoc has a plain `int` primary key (a valid SQLite rowid alias), not int64 β€” +// the case the earlier PKType=="int64" check misclassified as a string key. +type intDoc struct { + ID int + Title string + Embedding []float32 `orm:"-"` +} + +func (intDoc) TableName() string { return "int_docs" } + +func (intDoc) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{orm.Vector("Embedding", 3).WithMetric(orm.L2)} +} + +// TestSearch_IntPKVector locks the fix for the misclassification of non-int64 +// integer PKs: an `int` PK must use the rowid-keyed vec0 path end to end. +func TestSearch_IntPKVector(t *testing.T) { + ctx := context.Background() + db, err := sqlite.Open(filepath.Join(t.TempDir(), "intpk.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + if err := orm.AutoMigrate[intDoc](ctx, db); err != nil { + t.Fatalf("AutoMigrate: %v", err) + } + repo := orm.NewRepo[intDoc](db) + for _, d := range []*intDoc{ + {Title: "fox", Embedding: []float32{1, 0, 0}}, + {Title: "dog", Embedding: []float32{0, 1, 0}}, + } { + if err := repo.Create(ctx, d); err != nil { + t.Fatalf("create %s: %v", d.Title, err) + } + } + near, err := search.For[intDoc](db).Vector(ctx, []float32{1, 0, 0}, 1) + if err != nil { + t.Fatalf("Vector: %v", err) + } + if len(near) != 1 || near[0].Model.Title != "fox" { + t.Fatalf("int-PK vector nearest = %+v, want title 'fox'", near) + } +} diff --git a/dialect/sqlite/search/search.go b/dialect/sqlite/search/search.go index b453107..91bb559 100644 --- a/dialect/sqlite/search/search.go +++ b/dialect/sqlite/search/search.go @@ -8,7 +8,7 @@ // Indexes are keyed by your model's int64 primary key β€” the canonical recipe is // "your table owns the rows, the sidecar index owns the embeddings/terms, the // primary key ties them together." A search returns ranked keys (or scored -// keys); Load fetches the model rows by key, preserving rank order. Hybrid runs +// keys); Fetch fetches the model rows by key, preserving rank order. Hybrid runs // a vector KNN and a full-text query and fuses the two rankings with // reciprocal-rank-fusion. package search @@ -172,12 +172,12 @@ func WithK(k float64) Option { return fusion.WithK(k) } // WithWeights weights the vector and full-text rankings, in that order. func WithWeights(weights ...float64) Option { return fusion.WithWeights(weights...) } -// Load fetches model rows of type T by primary key, preserving the ranked order +// Fetch fetches model rows of type T by primary key, preserving the ranked order // of the supplied keys. Missing keys are skipped. T must be a model the query // front-end can address (a TableName and an int64 primary key). It issues one // Get per key, which is the right shape for the small top-k slices search // returns. -func Load[T any](ctx context.Context, sess liteorm.Session, keys []int64) ([]T, error) { +func Fetch[T any](ctx context.Context, sess liteorm.Session, keys []int64) ([]T, error) { repo := query.NewRepo[T](sess) out := make([]T, 0, len(keys)) for _, key := range keys { @@ -193,10 +193,10 @@ func Load[T any](ctx context.Context, sess liteorm.Session, keys []int64) ([]T, return out, nil } -// LoadScored is Load over the keys of a scored result set (Vector.SearchScored +// FetchScored is Fetch over the keys of a scored result set (Vector.SearchScored // or Hybrid), preserving order. -func LoadScored[T any](ctx context.Context, sess liteorm.Session, scored []Scored) ([]T, error) { - return Load[T](ctx, sess, keys(scored)) +func FetchScored[T any](ctx context.Context, sess liteorm.Session, scored []Scored) ([]T, error) { + return Fetch[T](ctx, sess, keys(scored)) } func keys(s []Scored) []int64 { diff --git a/dialect/sqlite/search/search_test.go b/dialect/sqlite/search/search_test.go index c9488bf..659edfd 100644 --- a/dialect/sqlite/search/search_test.go +++ b/dialect/sqlite/search/search_test.go @@ -133,7 +133,7 @@ func TestHybridFusion(t *testing.T) { } // Load fetches the model rows in fused order. - rows, err := search.LoadScored[Doc](ctx, db, fused) + rows, err := search.FetchScored[Doc](ctx, db, fused) if err != nil { t.Fatal(err) } diff --git a/dialect/sqlite/search/typed.go b/dialect/sqlite/search/typed.go new file mode 100644 index 0000000..02fc3d0 --- /dev/null +++ b/dialect/sqlite/search/typed.go @@ -0,0 +1,255 @@ +package search + +import ( + "context" + "errors" + "fmt" + "slices" + + "gosqlite.org/fts" + "gosqlite.org/vec" + liteorm "liteorm.org" + "liteorm.org/dialect" + "liteorm.org/dialect/sqlite" + "liteorm.org/orm" +) + +// Hit is one search result: the loaded model and its relevance score. +type Hit[T any] struct { + Model T + // Score's meaning depends on the search: vector distance (smaller is nearer) + // for Vector, BM25 rank for FullText, and the reciprocal-rank-fusion score + // (larger is better) for Hybrid. + Score float64 +} + +// Searcher runs schema-aware searches for model T against its declared indexes, +// returning loaded models in ranked order. Build one with [For]. +type Searcher[T any] struct { + sess liteorm.Session + field string // optional: which field's index to use when a model declares >1 of a kind +} + +// For begins a search for model T over sess. The sidecar tables, dimension, +// metric, and columns all come from T's declared search indexes: +// +// hits, err := search.For[Article](db).Vector(ctx, queryVec, 5) +func For[T any](sess liteorm.Session) *Searcher[T] { return &Searcher[T]{sess: sess} } + +// Field selects which index to use when the model declares more than one of the +// same kind, naming the model field the index covers. Returns a new Searcher. +func (s *Searcher[T]) Field(name string) *Searcher[T] { c := *s; c.field = name; return &c } + +// Vector runs a nearest-neighbour search over T's vector index, nearest first. +// Soft-deleted models are excluded. +func (s *Searcher[T]) Vector(ctx context.Context, query []float32, k int) ([]Hit[T], error) { + spec, err := targetSpec[T](dialect.SearchVector, "vector", s.field) + if err != nil { + return nil, err + } + if !spec.RowidKeyed() { + return knnKeyed[T](ctx, s.sess, spec, query, k) // non-integer (e.g. string) key + } + v, err := OpenVector(s.sess, spec.Name, spec.Dim, metricToken(spec.Metric)) + if err != nil { + return nil, err + } + scored, err := v.SearchScored(ctx, query, k) + if err != nil { + return nil, err + } + return loadHits[T](ctx, s.sess, scoredKeys(scored)) +} + +// FullText runs a full-text search over T's FTS index, best (BM25) rank first. +// Soft-deleted models are excluded. +func (s *Searcher[T]) FullText(ctx context.Context, q Query, k int) ([]Hit[T], error) { + spec, err := targetSpec[T](dialect.SearchFullText, "full-text", s.field) + if err != nil { + return nil, err + } + f, err := OpenFullText(s.sess, spec.Name, spec.Columns...) + if err != nil { + return nil, err + } + scored, err := f.SearchScored(ctx, q, k) + if err != nil { + return nil, err + } + return loadHits[T](ctx, s.sess, scoredKeys(scored)) +} + +// Hybrid runs a vector and a full-text search over T's indexes and fuses the +// rankings with reciprocal rank fusion (larger score is better). Soft-deleted +// models are excluded. The RRF tuning options (WithK, WithWeights) pass via fuse. +func (s *Searcher[T]) Hybrid(ctx context.Context, query []float32, q Query, k int, fuse ...Option) ([]Hit[T], error) { + vspec, err := targetSpec[T](dialect.SearchVector, "vector", s.field) + if err != nil { + return nil, err + } + fspec, err := targetSpec[T](dialect.SearchFullText, "full-text", s.field) + if err != nil { + return nil, err + } + v, err := OpenVector(s.sess, vspec.Name, vspec.Dim, metricToken(vspec.Metric)) + if err != nil { + return nil, err + } + f, err := OpenFullText(s.sess, fspec.Name, fspec.Columns...) + if err != nil { + return nil, err + } + scored, err := Hybrid(ctx, v, f, query, q, k, fuse...) + if err != nil { + return nil, err + } + return loadHits[T](ctx, s.sess, scoredKeys(scored)) +} + +// OpenVector attaches to an existing vec0 sidecar (no CREATE), the shape +// AutoMigrate provisions β€” the read-path counterpart to NewVector. +func OpenVector(sess liteorm.Session, name string, dim int, metric Metric) (*Vector, error) { + g, ok := sqlite.Conn(sess) + if !ok { + return nil, ErrUnsupportedBackend + } + tbl, err := vec.Open(g.DB, name, dim, vec.Options{Metric: metric}) + if err != nil { + return nil, err + } + return &Vector{tbl: tbl}, nil +} + +// OpenFullText attaches to an existing FTS5 sidecar (no CREATE) given its indexed +// columns β€” the read-path counterpart to NewFullText for the external-content +// tables AutoMigrate provisions. +func OpenFullText(sess liteorm.Session, name string, cols ...string) (*FullText, error) { + g, ok := sqlite.Conn(sess) + if !ok { + return nil, ErrUnsupportedBackend + } + idx, err := fts.Open[int64, string](g.DB, name, cols...) + if err != nil { + return nil, err + } + return &FullText{idx: idx}, nil +} + +// knnKeyed runs a vector search against a non-integer-keyed (e.g. string-PK) +// vec0 sidecar, loading models by their string key. +func knnKeyed[T any](ctx context.Context, sess liteorm.Session, spec dialect.SearchSpec, query []float32, k int) ([]Hit[T], error) { + g, ok := sqlite.Conn(sess) + if !ok { + return nil, ErrUnsupportedBackend + } + // The sidecar's key column is named after the model's PK (not vec0's default + // "id"), matching how it was provisioned. + tbl, err := vec.OpenKeyed[string](g.DB, spec.Name, spec.Dim, + vec.Options{Metric: metricToken(spec.Metric)}, vec.WithKeyColumn(spec.PKColumn)) + if err != nil { + return nil, err + } + ns, err := tbl.KNNSlice(ctx, query, k) + if err != nil { + return nil, err + } + items := make([]scoredKey, len(ns)) + for i, n := range ns { + items[i] = scoredKey{key: n.Key, score: n.Distance} + } + return loadHits[T](ctx, sess, items) +} + +// SearchScored runs a full-text query and returns each matching key with its BM25 +// rank (the scored counterpart of [FullText.Search]). +func (f *FullText) SearchScored(ctx context.Context, q Query, k int) ([]Scored, error) { + hits, err := f.idx.SearchSlice(ctx, q, fts.WithRanking()) + if err != nil { + return nil, err + } + if k > 0 && len(hits) > k { + hits = hits[:k] + } + out := make([]Scored, len(hits)) + for i, h := range hits { + out[i] = Scored{Key: h.Key, Score: h.Rank} + } + return out, nil +} + +// scoredKey is a ranked primary key (int64 or string) with its score. +type scoredKey struct { + key any + score float64 +} + +func scoredKeys(s []Scored) []scoredKey { + out := make([]scoredKey, len(s)) + for i, x := range s { + out[i] = scoredKey{key: x.Key, score: x.Score} + } + return out +} + +// loadHits loads each scored key's model through the orm Repo (so soft-deleted +// rows are excluded and scopes apply), preserving rank order. Missing rows are +// skipped. Keys are int64 or string, so Repo.Get addresses either PK shape. +func loadHits[T any](ctx context.Context, sess liteorm.Session, items []scoredKey) ([]Hit[T], error) { + repo := orm.NewRepo[T](sess) + out := make([]Hit[T], 0, len(items)) + for _, it := range items { + row, err := repo.Get(ctx, it.key) + if errors.Is(err, liteorm.ErrNoRows) { + continue + } + if err != nil { + return nil, err + } + out = append(out, Hit[T]{Model: row, Score: it.score}) + } + return out, nil +} + +// targetSpec resolves the one search spec of the given kind for T, using the +// optional field name to disambiguate when the model declares more than one. +func targetSpec[T any](kind dialect.SearchKind, label, field string) (dialect.SearchSpec, error) { + s, err := orm.SchemaOf[T]() + if err != nil { + return dialect.SearchSpec{}, err + } + specs, err := orm.SearchSpecs[T]() // index-aligned with s.SearchIndexes + if err != nil { + return dialect.SearchSpec{}, err + } + var found *dialect.SearchSpec + for i, ix := range s.SearchIndexes { + if ix.Kind != kind { + continue + } + if field != "" && !slices.Contains(ix.Fields, field) { + continue + } + if found != nil { + return dialect.SearchSpec{}, fmt.Errorf("search: %T declares more than one %s index; use Field(...) to choose", *new(T), label) + } + sp := specs[i] + found = &sp + } + if found == nil { + return dialect.SearchSpec{}, fmt.Errorf("search: %T has no %s index matching the request", *new(T), label) + } + return *found, nil +} + +func metricToken(s string) Metric { + switch s { + case "cosine": + return Cosine + case "l1": + return L1 + case "hamming": + return Hamming + default: + return L2 + } +} diff --git a/dialect/sqlite/search/typed_test.go b/dialect/sqlite/search/typed_test.go new file mode 100644 index 0000000..90383a7 --- /dev/null +++ b/dialect/sqlite/search/typed_test.go @@ -0,0 +1,157 @@ +package search_test + +import ( + "context" + "database/sql" + "path/filepath" + "slices" + "testing" + + _ "gosqlite.org/vec" + "liteorm.org/dialect/sqlite" + "liteorm.org/dialect/sqlite/search" + "liteorm.org/orm" +) + +type article struct { + ID int64 + Title string + Body string + Embedding []float32 `orm:"-"` // sidecar-only -> hook-synced vector + DeletedAt sql.NullTime `orm:"deleted_at,soft_delete"` +} + +func (article) TableName() string { return "articles" } + +func (article) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{ + orm.FullText("Title", "Body"), // text columns -> external-content + triggers + orm.Vector("Embedding", 3).WithMetric(orm.L2), + } +} + +func ids(hits []search.Hit[article]) []int64 { + out := make([]int64, len(hits)) + for i, h := range hits { + out[i] = h.Model.ID + } + return out +} + +func TestTypedSearch_KNN_Match_Fuse(t *testing.T) { + ctx := context.Background() + db, err := sqlite.Open(filepath.Join(t.TempDir(), "typed.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + + if err := orm.AutoMigrate[article](ctx, db); err != nil { + t.Fatalf("AutoMigrate: %v", err) + } + repo := orm.NewRepo[article](db) + for _, a := range []*article{ + {Title: "quick brown fox", Body: "the fox jumps", Embedding: []float32{1, 0, 0}}, + {Title: "lazy dog", Body: "the dog sleeps", Embedding: []float32{0, 1, 0}}, + {Title: "fox meets dog", Body: "a fox and a dog", Embedding: []float32{0.7, 0.7, 0}}, + } { + if err := repo.Create(ctx, a); err != nil { + t.Fatalf("create: %v", err) + } + } + + // KNN returns models nearest-first, with the loaded model populated. + near, err := search.For[article](db).Vector(ctx, []float32{1, 0, 0}, 2) + if err != nil { + t.Fatalf("KNN: %v", err) + } + if len(near) != 2 || near[0].Model.ID != 1 { + t.Fatalf("KNN nearest = %v, want first id 1", ids(near)) + } + if near[0].Model.Title != "quick brown fox" { + t.Errorf("KNN top model not loaded: title = %q", near[0].Model.Title) + } + + // Match returns the full-text hit's model. + hits, err := search.For[article](db).FullText(ctx, search.Term("lazy"), 5) + if err != nil { + t.Fatalf("Match: %v", err) + } + if len(hits) != 1 || hits[0].Model.ID != 2 { + t.Fatalf("Match 'lazy' = %v, want [2]", ids(hits)) + } + + // Fuse blends both rankings; a doc strong in either surfaces. + fused, err := search.For[article](db).Hybrid(ctx, []float32{0, 1, 0}, search.Term("dog"), 5) + if err != nil { + t.Fatalf("Fuse: %v", err) + } + if len(fused) == 0 || !slices.Contains(ids(fused), 2) { + t.Errorf("Fuse results = %v, want to include id 2", ids(fused)) + } + + // Soft delete excludes the row from typed results (loaded through the orm Repo, + // which honors the soft-delete scope) even though it lingers in the sidecars. + two, _ := repo.Get(ctx, 2) + if err := repo.Delete(ctx, &two); err != nil { + t.Fatalf("soft delete: %v", err) + } + hits, err = search.For[article](db).FullText(ctx, search.Term("lazy"), 5) + if err != nil { + t.Fatalf("Match after delete: %v", err) + } + if len(hits) != 0 { + t.Errorf("Match 'lazy' after soft delete = %v, want none", ids(hits)) + } + near, err = search.For[article](db).Vector(ctx, []float32{0, 1, 0}, 3) + if err != nil { + t.Fatalf("KNN after delete: %v", err) + } + if slices.Contains(ids(near), 2) { + t.Errorf("KNN after soft delete returned the deleted id 2: %v", ids(near)) + } +} + +// kbDoc has a STRING primary key; its vector sidecar is keyed by that string. +type kbDoc struct { + Slug string `orm:"slug,pk"` + Title string + Embedding []float32 `orm:"-"` +} + +func (kbDoc) TableName() string { return "kb_docs" } + +func (kbDoc) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{orm.Vector("Embedding", 3).WithMetric(orm.L2)} +} + +func TestTypedSearch_StringKeyVector(t *testing.T) { + ctx := context.Background() + db, err := sqlite.Open(filepath.Join(t.TempDir(), "kb.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + if err := orm.AutoMigrate[kbDoc](ctx, db); err != nil { + t.Fatalf("AutoMigrate: %v", err) + } + repo := orm.NewRepo[kbDoc](db) + for _, d := range []*kbDoc{ + {Slug: "fox", Title: "Fox", Embedding: []float32{1, 0, 0}}, + {Slug: "dog", Title: "Dog", Embedding: []float32{0, 1, 0}}, + } { + if err := repo.Create(ctx, d); err != nil { + t.Fatalf("create %s: %v", d.Slug, err) + } + } + near, err := search.For[kbDoc](db).Vector(ctx, []float32{1, 0, 0}, 1) + if err != nil { + t.Fatalf("KNN: %v", err) + } + if len(near) != 1 || near[0].Model.Slug != "fox" { + t.Fatalf("string-key KNN nearest = %+v, want slug 'fox'", near) + } + if near[0].Model.Title != "Fox" { + t.Errorf("loaded model not populated: %+v", near[0].Model) + } +} diff --git a/dialect/sqlite/search_hooks_test.go b/dialect/sqlite/search_hooks_test.go new file mode 100644 index 0000000..3bb44f0 --- /dev/null +++ b/dialect/sqlite/search_hooks_test.go @@ -0,0 +1,106 @@ +package sqlite_test + +import ( + "context" + "path/filepath" + "testing" + + _ "gosqlite.org/vec" + "liteorm.org/dialect/sqlite" + "liteorm.org/orm" +) + +// hookDoc keeps the embedding sidecar-only (orm:"-"), so the vector index is +// hook-synced from the ORM write path rather than by a trigger. +type hookDoc struct { + ID int64 + Title string + Embedding []float32 `orm:"-"` +} + +func (hookDoc) TableName() string { return "hook_docs" } + +func (hookDoc) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{orm.Vector("Embedding", 3).WithMetric(orm.L2)} +} + +func TestSearchHooks_VectorSyncsThroughRepo(t *testing.T) { + ctx := context.Background() + db, err := sqlite.Open(filepath.Join(t.TempDir(), "hook.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + + if err := orm.AutoMigrate[hookDoc](ctx, db); err != nil { + t.Fatalf("AutoMigrate: %v", err) + } + repo := orm.NewRepo[hookDoc](db) + + vecCount := func() int { + rows, err := db.QueryContext(ctx, `SELECT rowid FROM hook_docs_vec`) + if err != nil { + t.Fatalf("count: %v", err) + } + defer rows.Close() + n := 0 + for rows.Next() { + n++ + } + return n + } + nearest := func(v ...float32) int64 { + rows, err := db.QueryContext(ctx, + `SELECT rowid FROM hook_docs_vec WHERE embedding MATCH ? ORDER BY distance LIMIT 1`, vblob(v...)) + if err != nil { + t.Fatalf("knn: %v", err) + } + defer rows.Close() + var id int64 + if rows.Next() { + _ = rows.Scan(&id) + } + return id + } + + // Create through the Repo: the AfterCreate choke point must upsert the sidecar. + a := &hookDoc{Title: "fox", Embedding: []float32{1, 0, 0}} + b := &hookDoc{Title: "dog", Embedding: []float32{0, 1, 0}} + for _, d := range []*hookDoc{a, b} { + if err := repo.Create(ctx, d); err != nil { + t.Fatalf("create: %v", err) + } + } + if n := vecCount(); n != 2 { + t.Fatalf("after Create, vec rows = %d, want 2", n) + } + if got := nearest(1, 0, 0); got != a.ID { + t.Errorf("KNN nearest (1,0,0) = %d, want %d", got, a.ID) + } + + // Update through the Repo: AfterUpdate re-syncs the new embedding. + a.Embedding = []float32{0, 0, 1} + if err := repo.Update(ctx, a); err != nil { + t.Fatalf("update: %v", err) + } + if got := nearest(0, 0, 1); got != a.ID { + t.Errorf("after Update, KNN nearest (0,0,1) = %d, want %d", got, a.ID) + } + + // A partial update without the embedding must NOT clobber the sidecar. + a.Title, a.Embedding = "renamed", nil + if err := repo.Update(ctx, a); err != nil { + t.Fatalf("partial update: %v", err) + } + if got := nearest(0, 0, 1); got != a.ID { + t.Errorf("after empty-embedding update, KNN nearest (0,0,1) = %d, want %d (sidecar should be untouched)", got, a.ID) + } + + // Hard delete through the Repo removes the row from the sidecar. + if err := repo.Delete(ctx, b); err != nil { // hookDoc has no soft-delete column -> hard delete + t.Fatalf("delete: %v", err) + } + if n := vecCount(); n != 1 { + t.Errorf("after Delete, vec rows = %d, want 1", n) + } +} diff --git a/dialect/sqlite/search_provision_test.go b/dialect/sqlite/search_provision_test.go new file mode 100644 index 0000000..0930257 --- /dev/null +++ b/dialect/sqlite/search_provision_test.go @@ -0,0 +1,139 @@ +package sqlite_test + +import ( + "context" + "encoding/binary" + "math" + "path/filepath" + "testing" + + _ "gosqlite.org/vec" // side-effect: registers the vec0 extension + "liteorm.org/dialect/sqlite" + "liteorm.org/orm" +) + +// provDoc declares both a full-text and a vector sidecar via the SearchIndexes +// method; the embedding is sidecar-only (orm:"-"), the text columns are real. +type provDoc struct { + ID int64 + Title string + Body string + Embedding []float32 `orm:"-"` +} + +func (provDoc) TableName() string { return "prov_docs" } + +func (provDoc) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{ + orm.FullText("Title", "Body").WithTokenizer("unicode61"), + orm.Vector("Embedding", 3).WithMetric(orm.Cosine), + } +} + +func vblob(v ...float32) []byte { + b := make([]byte, len(v)*4) + for i, x := range v { + binary.LittleEndian.PutUint32(b[i*4:], math.Float32bits(x)) + } + return b +} + +func TestAutoMigrate_ProvisionsSearchSidecars(t *testing.T) { + ctx := context.Background() + db, err := sqlite.Open(filepath.Join(t.TempDir(), "prov.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + + exists := func(name string) bool { + rows, err := db.QueryContext(ctx, `SELECT 1 FROM sqlite_master WHERE name = ?`, name) + if err != nil { + t.Fatalf("sqlite_master: %v", err) + } + defer rows.Close() + return rows.Next() + } + + // AutoMigrate should create the base table AND both sidecars. + if err := orm.AutoMigrate[provDoc](ctx, db); err != nil { + t.Fatalf("AutoMigrate: %v", err) + } + for _, name := range []string{"prov_docs", "prov_docs_fts", "prov_docs_vec"} { + if !exists(name) { + t.Errorf("expected table %q to exist after AutoMigrate", name) + } + } + + // Idempotent: a second run must not error (CREATE ... IF NOT EXISTS). + if err := orm.AutoMigrate[provDoc](ctx, db); err != nil { + t.Fatalf("second AutoMigrate (should be a no-op): %v", err) + } + + // Seed the base table. + corpus := []struct { + id int64 + title string + emb []byte + }{ + {1, "quick brown fox", vblob(1, 0, 0)}, + {2, "lazy dog sleeps", vblob(0, 1, 0)}, + {3, "a fox and a dog", vblob(0.7, 0.7, 0)}, + } + for _, c := range corpus { + if _, err := db.ExecContext(ctx, `INSERT INTO prov_docs(id, title, body) VALUES (?, ?, '')`, c.id, c.title); err != nil { + t.Fatalf("seed base row: %v", err) + } + } + + // The vec0 sidecar is a usable vec0 table: insert embeddings, KNN-search them. + for _, c := range corpus { + if _, err := db.ExecContext(ctx, `INSERT INTO prov_docs_vec(rowid, embedding) VALUES (?, ?)`, c.id, c.emb); err != nil { + t.Fatalf("insert into vec sidecar: %v", err) + } + } + var nearest int64 + rows, err := db.QueryContext(ctx, + `SELECT rowid FROM prov_docs_vec WHERE embedding MATCH ? ORDER BY distance LIMIT 1`, vblob(1, 0, 0)) + if err != nil { + t.Fatalf("vec KNN: %v", err) + } + if rows.Next() { + _ = rows.Scan(&nearest) + } + rows.Close() + if nearest != 1 { + t.Errorf("vec KNN nearest to (1,0,0) = %d, want 1", nearest) + } + + // The FTS5 sidecar is external-content (content='prov_docs'): backfill from the + // base table via 'rebuild', then MATCH resolves base rowids. + if _, err := db.ExecContext(ctx, `INSERT INTO prov_docs_fts(prov_docs_fts) VALUES('rebuild')`); err != nil { + t.Fatalf("fts rebuild: %v", err) + } + var hit int64 + frows, err := db.QueryContext(ctx, `SELECT rowid FROM prov_docs_fts WHERE prov_docs_fts MATCH ? LIMIT 1`, "quick") + if err != nil { + t.Fatalf("fts MATCH: %v", err) + } + if frows.Next() { + _ = frows.Scan(&hit) + } + frows.Close() + if hit != 1 { + t.Errorf("fts MATCH 'quick' = %d, want 1", hit) + } + + // DropSearchIndexes removes both sidecars, leaving the base table. + if err := orm.DropSearchIndexes[provDoc](ctx, db); err != nil { + t.Fatalf("DropSearchIndexes: %v", err) + } + if !exists("prov_docs") { + t.Error("base table should survive DropSearchIndexes") + } + for _, name := range []string{"prov_docs_fts", "prov_docs_vec"} { + if exists(name) { + t.Errorf("sidecar %q should be gone after DropSearchIndexes", name) + } + } +} diff --git a/dialect/sqlite/search_triggers_test.go b/dialect/sqlite/search_triggers_test.go new file mode 100644 index 0000000..b8b7530 --- /dev/null +++ b/dialect/sqlite/search_triggers_test.go @@ -0,0 +1,129 @@ +package sqlite_test + +import ( + "context" + "path/filepath" + "testing" + + _ "gosqlite.org/vec" + "liteorm.org/dialect/sqlite" + "liteorm.org/orm" +) + +// trigDoc stores the embedding as a real column, so the vector index is +// trigger-synced (along with the full-text index over Title). +type trigDoc struct { + ID int64 + Title string + Embedding []byte // stored blob -> vector trigger mode +} + +func (trigDoc) TableName() string { return "trig_docs" } + +func (trigDoc) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{ + orm.FullText("Title"), + orm.Vector("Embedding", 3).WithMetric(orm.L2), + } +} + +func TestSearchTriggers_AutoSyncOnRawWrites(t *testing.T) { + ctx := context.Background() + db, err := sqlite.Open(filepath.Join(t.TempDir(), "trig.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = db.Close() }) + + if err := orm.AutoMigrate[trigDoc](ctx, db); err != nil { + t.Fatalf("AutoMigrate: %v", err) + } + + exec := func(q string, args ...any) { + if _, err := db.ExecContext(ctx, q, args...); err != nil { + t.Fatalf("exec %q: %v", q, err) + } + } + scalarIDs := func(q string, args ...any) []int64 { + rows, err := db.QueryContext(ctx, q, args...) + if err != nil { + t.Fatalf("query %q: %v", q, err) + } + defer rows.Close() + var ids []int64 + for rows.Next() { + var id int64 + _ = rows.Scan(&id) + ids = append(ids, id) + } + return ids + } + count := func(table string) int { + ids := scalarIDs(`SELECT rowid FROM ` + table) + return len(ids) + } + ftsMatch := func(q string) []int64 { + return scalarIDs(`SELECT rowid FROM trig_docs_fts WHERE trig_docs_fts MATCH ? ORDER BY rowid`, q) + } + knn := func(v ...float32) int64 { + ids := scalarIDs(`SELECT rowid FROM trig_docs_vec WHERE embedding MATCH ? ORDER BY distance LIMIT 1`, vblob(v...)) + if len(ids) == 0 { + return 0 + } + return ids[0] + } + first := func(ids []int64) int64 { + if len(ids) == 0 { + return 0 + } + return ids[0] + } + + // Plain raw INSERTs β€” no manual sidecar maintenance. Triggers must populate + // BOTH sidecars. + exec(`INSERT INTO trig_docs(id, title, embedding) VALUES (1, 'quick brown fox', ?)`, vblob(1, 0, 0)) + exec(`INSERT INTO trig_docs(id, title, embedding) VALUES (2, 'lazy dog sleeps', ?)`, vblob(0, 1, 0)) + exec(`INSERT INTO trig_docs(id, title, embedding) VALUES (3, 'fox and dog', ?)`, vblob(0.7, 0.7, 0)) + + if got := first(ftsMatch("quick")); got != 1 { + t.Errorf("after insert, FTS MATCH 'quick' = %d, want 1", got) + } + if got := knn(1, 0, 0); got != 1 { + t.Errorf("after insert, vec KNN nearest (1,0,0) = %d, want 1", got) + } + if n := count("trig_docs_vec"); n != 3 { + t.Errorf("after insert, vec rows = %d, want 3", n) + } + + // UPDATE re-syncs both indexes. + exec(`UPDATE trig_docs SET title = 'zebra stripes', embedding = ? WHERE id = 1`, vblob(0, 0, 1)) + if got := ftsMatch("quick"); len(got) != 0 { + t.Errorf("after update, FTS MATCH 'quick' = %v, want none", got) + } + if got := first(ftsMatch("zebra")); got != 1 { + t.Errorf("after update, FTS MATCH 'zebra' = %d, want 1", got) + } + if got := knn(0, 0, 1); got != 1 { + t.Errorf("after update, vec KNN nearest (0,0,1) = %d, want 1", got) + } + + // DELETE removes from both. + exec(`DELETE FROM trig_docs WHERE id = 2`) + if got := ftsMatch("lazy"); len(got) != 0 { + t.Errorf("after delete, FTS MATCH 'lazy' = %v, want none", got) + } + if n := count("trig_docs_vec"); n != 2 { + t.Errorf("after delete, vec rows = %d, want 2", n) + } + + // Multi-row INSERT fires the row triggers per row (the bulk/raw case gorm + // callbacks can't cover). + exec(`INSERT INTO trig_docs(id, title, embedding) VALUES (10, 'alpha', ?), (11, 'beta', ?)`, + vblob(1, 1, 1), vblob(0.2, 0.2, 0.2)) + if n := count("trig_docs_vec"); n != 4 { + t.Errorf("after bulk insert, vec rows = %d, want 4", n) + } + if got := first(ftsMatch("alpha")); got != 10 { + t.Errorf("after bulk insert, FTS MATCH 'alpha' = %d, want 10", got) + } +} diff --git a/docs/guides/encryption.md b/docs/guides/encryption.md new file mode 100644 index 0000000..cbcdd09 --- /dev/null +++ b/docs/guides/encryption.md @@ -0,0 +1,42 @@ +# At-rest encryption + +LiteORM opens an encrypted SQLite database through gosqlite's transparent, page-level cipher: you pass a key when opening, and every page written to disk is ciphertext. Encryption is an open-time concern, orthogonal to how you use the data β€” once the database is open, the `query` builder, the `orm`, migrations, and search all work exactly as on an unencrypted database. + +## Opening + +`sqlite.OpenEncrypted(path, key)` opens (or creates) an encrypted database with a 32-byte key, using the default Adiantum cipher. The returned `*liteorm.DB` is used exactly like an unencrypted one: + +```go +db, err := sqlite.OpenEncrypted("app.db", key) // key is a 32-byte []byte +if err != nil { + return err +} +orm.AutoMigrate[Note](ctx, db) +orm.NewRepo[Note](db).Create(ctx, &Note{Text: "…"}) +``` + +The on-disk file is ciphertext; reopening requires the same key, and a wrong key fails rather than returning garbage. For full control over the cipher, pragmas, or pool sizing, open from a `gosqlite.Config`: + +```go +db, err := sqlite.OpenConfig(gosqlite.Config{ + Path: "app.db", + Pragmas: gosqlite.RecommendedPragmas(), + Encryption: &gosqlite.Encryption{Key: key, Cipher: gosqlite.Adiantum}, +}) +``` + +## Key handling + +The key is a 32-byte secret β€” source it from a key-management service or secret store, never a literal in source. Losing the key means losing the data; there is no recovery path. Rotating a key means re-encrypting: open the database with the old key and copy it into a new database opened with the new key. + +## Constraints + +- Encryption needs an on-disk path; `:memory:` is rejected, since there is nothing to encrypt at rest. +- It is mutually exclusive with a custom VFS β€” the cipher is itself a VFS layer. +- It encrypts the whole database file, set at open time; there is no per-table encryption. + +## See also + +- `examples/encryption` β€” write encrypted, verify the on-disk bytes are ciphertext, reopen with the key, and watch the wrong key fail. +- [Backends reference](../reference/backends.md) β€” opening the SQLite backend and its options. +- [SQLite search](sqlite-search.md) β€” the SQLite-specific query capabilities (independent of encryption). diff --git a/docs/guides/from-gorm.md b/docs/guides/from-gorm.md index 713ef60..284280a 100644 --- a/docs/guides/from-gorm.md +++ b/docs/guides/from-gorm.md @@ -30,7 +30,7 @@ LiteORM's `orm` front-end is deliberately close to gorm in spirit β€” declarativ ## What's different (and why) -- **No table-name pluralization.** `User` maps to table `user`, not `users`. Add `func (User) TableName() string { return "users" }` to keep gorm's names. (Your `gorm:"..."` field tags need no change.) +- **Pluralization is opt-in, not the default.** Out of the box `User` maps to table `user`, not `users`. To keep gorm's plural names everywhere, call `orm.UsePluralTableNames(true)` once at startup (`User` β†’ `users`) and register irregulars with `orm.RegisterPlural("person", "people")`; otherwise pin a single table with `func (User) TableName() string { return "users" }`. (Your `gorm:"..."` field tags need no change either way.) - **`gorm:"..."` tags are read as-is.** A ported model behaves identically; you don't have to convert tags to `orm:"..."`. The one type change: a `gorm.DeletedAt` field becomes a `sql.NullTime` tagged `soft_delete` β€” the porter below does this for you. - **Eager loading is explicit and N+1-safe.** There's no lazy loading; you call `orm.Load`/`LoadPath` for exactly the relations you want, and each is one batched query (the test suite asserts the query count). Touching an unloaded relation field gives the zero value, never a hidden query. - **Keyed `Update`/`Delete` return `ErrNoRows` on no match.** Updating a missing or out-of-scope (soft-deleted) row is `liteorm.ErrNoRows`, not a silent success β€” so a no-op is something you can detect. (gorm reports `RowsAffected` instead.) Reach a soft-deleted row on purpose with `IncludeDeleted()`. diff --git a/docs/guides/hooks.md b/docs/guides/hooks.md index 3540f1e..5e2c9c3 100644 --- a/docs/guides/hooks.md +++ b/docs/guides/hooks.md @@ -14,12 +14,12 @@ There are six hook points, one before and one after each write operation: | `Update` | `BeforeUpdate` | `AfterUpdate` | | `Delete` | `BeforeDelete` | `AfterDelete` | -Every hook has the same signature, taking a `context.Context` and a typed `*orm.Op[T]`, and returning an `error`: +Every hook has the same signature, taking a `context.Context` and a typed `*orm.Event[T]`, and returning an `error`: ```go -func (p *Post) BeforeCreate(ctx context.Context, op *orm.Op[Post]) error { - if op.Model.Slug == "" { - op.Model.Slug = slugify(op.Model.Title) +func (p *Post) BeforeCreate(ctx context.Context, ev *orm.Event[Post]) error { + if ev.Model.Slug == "" { + ev.Model.Slug = slugify(ev.Model.Title) } return nil } @@ -29,16 +29,16 @@ Implement only the hooks you need. The repository checks once per type which hoo ## The Op handle -`orm.Op[T]` is the narrow, explicit handle passed to every hook. It carries exactly two things: +`orm.Event[T]` is the narrow, explicit handle passed to every hook. It carries exactly two things: -- `op.Model` β€” the typed `*T` being written. Mutate it in a `Before*` hook to change what gets persisted (the slug example above), or read it in an `After*` hook. -- `op.Sess` β€” the executing `liteorm.Session`. When the write runs inside a transaction, this is that transaction, so anything you do through it commits or rolls back atomically with the write β€” see [transactions](transactions.md). +- `ev.Model` β€” the typed `*T` being written. Mutate it in a `Before*` hook to change what gets persisted (the slug example above), or read it in an `After*` hook. +- `ev.Sess` β€” the executing `liteorm.Session`. When the write runs inside a transaction, this is that transaction, so anything you do through it commits or rolls back atomically with the write β€” see [transactions](transactions.md). ```go -func (p *Post) AfterCreate(ctx context.Context, op *orm.Op[Post]) error { - // op.Sess is the same session (or tx) the Create ran on. - return orm.NewRepo[AuditEntry](op.Sess). - Create(ctx, &AuditEntry{Action: "post.created", PostID: op.Model.ID}) +func (p *Post) AfterCreate(ctx context.Context, ev *orm.Event[Post]) error { + // ev.Sess is the same session (or tx) the Create ran on. + return orm.NewRepo[AuditEntry](ev.Sess). + Create(ctx, &AuditEntry{Action: "post.created", PostID: ev.Model.ID}) } ``` @@ -49,8 +49,8 @@ In an `AfterCreate` hook the model's generated primary key is already populated, A hook that returns a non-nil error stops the write. A `Before*` error aborts before any SQL runs; an `After*` error surfaces from the repository call after the row was written β€” inside a transaction you'd roll back. Errors are never swallowed, so a failed validation in `BeforeCreate` reliably prevents the insert: ```go -func (u *User) BeforeCreate(ctx context.Context, op *orm.Op[User]) error { - if op.Model.Email == "" { +func (u *User) BeforeCreate(ctx context.Context, ev *orm.Event[User]) error { + if ev.Model.Email == "" { return fmt.Errorf("user: email is required") } return nil @@ -76,5 +76,5 @@ This is the recommended pattern for every hook you implement. ## Where to next - [The orm front-end](orm.md) β€” the repository whose writes fire these hooks. -- [Transactions](transactions.md) β€” `op.Sess` is the tx your hook runs in. +- [Transactions](transactions.md) β€” `ev.Sess` is the tx your hook runs in. - [Soft delete](soft-delete.md) β€” `BeforeDelete` / `AfterDelete` fire on soft deletes too. diff --git a/docs/guides/orm.md b/docs/guides/orm.md index f3a8957..9bc8a2b 100644 --- a/docs/guides/orm.md +++ b/docs/guides/orm.md @@ -230,7 +230,7 @@ err := posts.Where("views > ?", 0).FindInBatches(ctx, 500, func(batch []Post) er LiteORM's declarative layer leans on convention, but every convention is the kind you can reason about: -- **No pluralization.** The table name is your `TableName()` or the snake_case of the type β€” `Post` maps to `post`, not `posts`, unless you say so. Make it explicit with `TableName()`. +- **No silent pluralization.** By default the table name is your `TableName()` or the snake_case of the type β€” `Post` maps to `post`, not `posts`. Want gorm-style plurals? Opt in once with `orm.UsePluralTableNames(true)` (`Post` β†’ `posts`) and register any irregulars with `orm.RegisterPlural`; a per-type `TableName()` still wins over both. - **No lazy loading.** Associations are never fetched implicitly when you touch a field. You load them with one explicit, N+1-safe call β€” see [associations](associations.md). - **Hard errors over silent guesses.** A foreign key that can't be inferred, an unknown column, an ambiguous relation β€” these are returned errors, not best-effort SQL. - **Soft delete is opt-in and scoped.** A `soft_delete` field turns deletes into updates and excludes deleted rows from reads by default, with explicit scopes to see them β€” see [soft delete](soft-delete.md). diff --git a/docs/guides/query.md b/docs/guides/query.md index af96063..568cd6d 100644 --- a/docs/guides/query.md +++ b/docs/guides/query.md @@ -27,7 +27,7 @@ Other backends open the same way and return the same `*liteorm.DB`, so nothing b ## Your model type -A model is a plain struct. The table name comes from a `TableName() string` method if you define one, otherwise it's the snake_case of the type name (no pluralization β€” explicit over implicit). +A model is a plain struct. The table name comes from a `TableName() string` method if you define one, otherwise it's the snake_case of the type name β€” singular by default, or pluralized if you opt in with `orm.UsePluralTableNames(true)` (both front-ends share that one setting). ```go type Product struct { diff --git a/docs/guides/sqlite-search.md b/docs/guides/sqlite-search.md index 3ae264d..bfae3a7 100644 --- a/docs/guides/sqlite-search.md +++ b/docs/guides/sqlite-search.md @@ -1,46 +1,72 @@ # SQLite search -The `liteorm.org/dialect/sqlite/search` package adds vector nearest-neighbour, full-text, and hybrid search to LiteORM's SQLite backend. These features are SQLite-only and capability-gated: every constructor takes a `liteorm.Session` opened by `liteorm.org/dialect/sqlite` and returns `search.ErrUnsupportedBackend` for any other dialect. +The `liteorm.org/dialect/sqlite/search` package adds vector nearest-neighbour, full-text, and hybrid search to LiteORM's SQLite backend. These features are SQLite-only and capability-gated: the typed helpers and constructors take a `liteorm.Session` opened by `liteorm.org/dialect/sqlite` and return `search.ErrUnsupportedBackend` for any other dialect. -The model is a sidecar index. Your table owns the rows; a vector or full-text index owns the embeddings or terms; your model's `int64` primary key ties them together. A search returns ranked keys (optionally with scores); `search.Load` fetches the model rows by key, preserving rank order. The recipe is the same whether you search by vector, by text, or by both. +Every index is a *sidecar*: your table owns the rows, an FTS5 or vec0 table owns the terms or embeddings, and your model's `int64` primary key ties them together. There are two ways to drive that sidecar. The **declarative** layer (recommended) declares the index on the model and lets `AutoMigrate` provision it and keep it in sync; the **low-level** layer drives the sidecar by hand for callers that own its lifecycle. -## Vector search +## Declarative search (recommended) -`search.NewVector` creates (idempotently) or opens a fixed-dimension vector table with a distance metric. Add embeddings keyed by your primary key, then query for the nearest neighbours. +Declare the indexes on the model and let `orm.AutoMigrate` own the rest: it creates the FTS5/vec0 sidecar tables and the triggers (or ORM hooks) that keep them current, so ordinary `Repo.Create`/`Update`/`Delete` need no index bookkeeping. ```go -import "liteorm.org/dialect/sqlite/search" +type Article struct { + ID int64 + Title string + Body string + Embedding []float32 `orm:"-"` // sidecar-only (not a base-table column) +} -v, err := search.NewVector(ctx, db, "doc_vecs", dim, search.Cosine) -if err != nil { - return err +func (Article) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{ + orm.FullText("Title", "Body"), + orm.Vector("Embedding", 384).WithMetric(orm.Cosine), + } } -v.Add(ctx, doc.ID, embedding) // embedding is []float32; re-adding a key replaces it +``` -keys, err := v.Search(ctx, queryEmbedding, 5) // 5 nearest keys, nearest first -docs, err := search.Load[Doc](ctx, db, keys) // fetch rows in ranked order +`AutoMigrate[Article]` then provisions `articles`, `articles_fts`, and `articles_vec`; from there a plain write keeps every index current: + +```go +orm.AutoMigrate[Article](ctx, db) +repo := orm.NewRepo[Article](db) +repo.Create(ctx, &Article{Title: "…", Body: "…", Embedding: vec}) // both sidecars sync automatically ``` -The metrics are `search.Cosine` (the usual choice for normalized text embeddings), `search.L2` (the default), `search.L1`, and `search.Hamming` (bit vectors). To inspect distances, use `SearchScored`, which reports each neighbour's raw distance (smaller is nearer): +Search with the typed searcher `search.For[T](db)`, whose `.Vector` / `.FullText` / `.Hybrid` methods return your models in ranked order: ```go -scored, err := v.SearchScored(ctx, queryEmbedding, 5) // []search.Scored{Key, Score} -docs, err := search.LoadScored[Doc](ctx, db, scored) +near, _ := search.For[Article](db).Vector(ctx, queryVec, 5) +hits, _ := search.For[Article](db).FullText(ctx, search.Term("rocket"), 5) +fused, _ := search.For[Article](db).Hybrid(ctx, queryVec, search.Term("rocket"), 5) + +for _, h := range near { + // h.Score is the vector distance for .Vector, the BM25 rank for .FullText, and + // the reciprocal-rank-fusion score for .Hybrid. + fmt.Println(h.Model.Title, h.Score) +} ``` -## Full-text search +Soft-deleted rows drop out of results automatically β€” the searcher loads through the ORM, which honors the soft-delete scope. When a model declares more than one index of the same kind, pick one with `search.For[Article](db).Field("FieldName")`. -`search.NewFullText` creates or opens an FTS5 index over a text column, with the default tokenizer and BM25 ranking. Index text under a key, then query with the builder API. +### Declaring indexes: method or tags -```go -f, err := search.NewFullText(ctx, db, "doc_fts") -f.Add(ctx, doc.ID, doc.Title+" "+doc.Body) // re-adding a key replaces it +Both front-ends lower to the same `orm.SearchIndex`: -keys, err := f.Search(ctx, search.Term("rocket"), 5) // best (BM25) rank first -docs, err := search.Load[Doc](ctx, db, keys) -``` +- A `SearchIndexes() []orm.SearchIndex` method is the typed, full-power form β€” multi-column full-text, per-column BM25 weights (`WithWeights`), and the tokenizer/prefix/detail options. +- Struct tags cover the common single-field case: `vec:"dim=384;metric=cosine"` on the embedding field, or `fts:"tokenize=porter unicode61"` on a text field. (`fts5:` is accepted as an alias.) + +When both are present, the method wins on a sidecar-name collision. -Queries are built compositionally β€” there is no raw match-string parsing to get wrong: +### How writes stay in sync + +Each index syncs one of two ways, set with `.WithSync(...)` or left to the default: + +- **Triggers** β€” SQL `AFTER INSERT/UPDATE/DELETE` triggers maintain the sidecar, so *every* write stays indexed: bulk inserts and raw `query` writes that never touch the ORM included. This is the default for full-text (the indexed text already lives on the base table, so it costs nothing) and for a vector whose embedding is a stored column. +- **Hooks** β€” the ORM write path maintains the sidecar. This is the default for a vector whose embedding is sidecar-only (`orm:"-"`, so the vector is not duplicated on the base table); writes that bypass the ORM are not indexed. + +## Query builders + +Full-text queries are built compositionally β€” there is no raw match-string parsing to get wrong. The same builders feed both the searcher's `.FullText` and the low-level `FullText.Search`: ```go search.Term("rocket") // a single term @@ -52,23 +78,30 @@ search.Not(search.Term("space"), search.Term("opera")) // "space" but not "oper search.Near(3, "rocket", "engine") // terms within 3 of each other ``` -## Hybrid search +The vector metrics are `orm.Cosine` (the usual choice for normalized embeddings), `orm.L2` (the default), `orm.L1`, and `orm.Hamming` (bit vectors). -`search.Hybrid` runs a vector KNN *and* a full-text query, then fuses the two rankings with reciprocal rank fusion. A key that ranks well in either modality surfaces; one that ranks well in both rises highest β€” without tuning a brittle score-scale blend. The result is ordered by descending fusion score (larger is better). +## Low-level building blocks + +When you manage the index lifecycle yourself β€” no model, or a sidecar you provision and backfill on your own schedule β€” the constructors give you direct handles. `NewVector` and `NewFullText` create (idempotently) or open a sidecar; `Add` upserts a row keyed by your primary key; `Search` returns ranked keys and `Load` fetches the model rows in that order. ```go -fused, err := search.Hybrid(ctx, v, f, queryEmbedding, search.Term("software"), 5) -docs, err := search.LoadScored[Doc](ctx, db, fused) -for i, d := range docs { - fmt.Printf("%.4f %s\n", fused[i].Score, d.Title) -} +v, _ := search.NewVector(ctx, db, "doc_vecs", dim, search.Cosine) +v.Add(ctx, doc.ID, embedding) // []float32; re-adding a key replaces it +keys, _ := v.Search(ctx, queryEmbedding, 5) // 5 nearest keys, nearest first +docs, _ := search.Fetch[Doc](ctx, db, keys) // rows in ranked order + +f, _ := search.NewFullText(ctx, db, "doc_fts") +f.Add(ctx, doc.ID, doc.Title+" "+doc.Body) +keys, _ = f.Search(ctx, search.Term("rocket"), 5) ``` -`Hybrid` takes optional fusion knobs β€” `search.WithK` (the RRF damping constant) and `search.WithWeights` (weighting the vector and full-text rankings, in that order). +`SearchScored` reports each vector neighbour's raw distance (smaller is nearer), and `search.Hybrid` fuses an explicit `Vector` and `FullText` with reciprocal rank fusion β€” the same fusion `Fuse` runs, on handles you hold yourself. `Hybrid` and `Fuse` take optional knobs: `search.WithK` (the RRF damping constant) and `search.WithWeights` (weighting the vector and full-text rankings, in that order). + +`OpenVector` and `OpenFullText` attach to an already-provisioned sidecar (the shape `AutoMigrate` creates) without re-creating it β€” the read-path counterparts to the `New*` constructors. ## See also -- `examples/search` β€” vector, full-text, and hybrid search end to end. +- `examples/search` β€” the declarative path and the low-level building blocks, end to end. - [SQLite changeset](sqlite-changeset.md) β€” the other SQLite-only extension. - [Backends reference](../reference/backends.md) β€” the SQLite backend, how to open it, and at-rest encryption. - Full API: [`liteorm.org/dialect/sqlite/search`](https://pkg.go.dev/liteorm.org/dialect/sqlite/search) and [`liteorm.org/dialect/sqlite`](https://pkg.go.dev/liteorm.org/dialect/sqlite). diff --git a/docs/index.md b/docs/index.md index dae545b..cae430d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -30,7 +30,8 @@ New here? Start with **[Getting started](getting-started.md)**, then pick the fr **Backend-specific** -- [SQLite search](guides/sqlite-search.md) β€” vector, full-text, and hybrid (reciprocal-rank-fusion) search; encryption at rest. +- [SQLite search](guides/sqlite-search.md) β€” vector, full-text, and hybrid (reciprocal-rank-fusion) search. +- [At-rest encryption](guides/encryption.md) β€” open an encrypted SQLite database; the on-disk file is ciphertext, readable only with the key. - [SQLite changesets](guides/sqlite-changeset.md) β€” capture, apply, invert, and concat changesets for audit and replication. - [Postgres](guides/postgres.md) β€” LISTEN/NOTIFY and the typed JSONB / array operators. diff --git a/examples/blog/main.go b/examples/blog/main.go index 8ad02ef..1f2d9fe 100644 --- a/examples/blog/main.go +++ b/examples/blog/main.go @@ -58,9 +58,9 @@ type Post struct { func (Post) TableName() string { return "posts" } // BeforeCreate derives the URL slug from the title when one isn't set. -func (p *Post) BeforeCreate(_ context.Context, op *orm.Op[Post]) error { - if op.Model.Slug == "" { - op.Model.Slug = slugify(op.Model.Title) +func (p *Post) BeforeCreate(_ context.Context, ev *orm.Event[Post]) error { + if ev.Model.Slug == "" { + ev.Model.Slug = slugify(ev.Model.Title) } return nil } diff --git a/examples/encryption/go.mod b/examples/encryption/go.mod new file mode 100644 index 0000000..6c90aea --- /dev/null +++ b/examples/encryption/go.mod @@ -0,0 +1,28 @@ +module liteorm.org/examples/encryption + +go 1.25.7 + +require ( + liteorm.org v0.8.0 + liteorm.org/dialect/sqlite v0.8.0 +) + +require ( + github.com/dustin/go-humanize v1.0.1 // indirect + github.com/google/uuid v1.6.0 // indirect + github.com/mattn/go-isatty v0.0.20 // indirect + github.com/ncruces/go-strftime v1.0.0 // indirect + github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect + golang.org/x/crypto v0.53.0 // indirect + golang.org/x/sys v0.46.0 // indirect + gosqlite.org v0.7.1 // indirect + lukechampine.com/adiantum v1.1.1 // indirect + modernc.org/libc v1.72.3 // indirect + modernc.org/mathutil v1.7.1 // indirect + modernc.org/memory v1.11.0 // indirect + modernc.org/sqlite v1.52.0 // indirect +) + +replace liteorm.org => ../.. + +replace liteorm.org/dialect/sqlite => ../../dialect/sqlite diff --git a/examples/encryption/go.sum b/examples/encryption/go.sum new file mode 100644 index 0000000..35e20ec --- /dev/null +++ b/examples/encryption/go.sum @@ -0,0 +1,55 @@ +github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= +github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= +github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= +github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= +github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w= +github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls= +github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= +github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= +golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto= +golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio= +golang.org/x/mod v0.33.0 h1:tHFzIWbBifEmbwtGz65eaWyGiGZatSrT9prnU8DbVL8= +golang.org/x/mod v0.33.0/go.mod h1:swjeQEj+6r7fODbD2cqrnje9PnziFuw4bmLbBZFrQ5w= +golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM= +golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw= +golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8= +golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0= +gosqlite.org v0.7.1 h1:1ObtUU4Da/RzhgjfK3WkPJgEzcAESgH5fcCNkaF9oxw= +gosqlite.org v0.7.1/go.mod h1:516gDHZeDXlkUZut4ubiM8iCPNyanJbcf3Wr+pn1hNQ= +lukechampine.com/adiantum v1.1.1 h1:4fp6gTxWCqpEbLy40ExiYDDED3oUNWx5cTqBCtPdZqA= +lukechampine.com/adiantum v1.1.1/go.mod h1:LrAYVnTYLnUtE/yMp5bQr0HstAf060YUF8nM0B6+rUw= +modernc.org/cc/v4 v4.28.2 h1:3tQ0lf2ADtoby2EtSP+J7IE2SHwEJdP8ioR59wx7XpY= +modernc.org/cc/v4 v4.28.2/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI= +modernc.org/ccgo/v4 v4.34.0 h1:yRLPFZieg532OT4rp4JFNIVcquwalMX26G95WQDqwCQ= +modernc.org/ccgo/v4 v4.34.0/go.mod h1:AS5WYMyBakQ+fhsHhtP8mWB82KTGPkNNJDGfGQCe0/A= +modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM= +modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU= +modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI= +modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito= +modernc.org/gc/v3 v3.1.2 h1:ZtDCnhonXSZexk/AYsegNRV1lJGgaNZJuKjJSWKyEqo= +modernc.org/gc/v3 v3.1.2/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY= +modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks= +modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI= +modernc.org/libc v1.72.3 h1:ZnDF4tXn4NBXFutMMQC4vtbTFSXhhKzR73fv0beZEAU= +modernc.org/libc v1.72.3/go.mod h1:dn0dZNnnn1clLyvRxLxYExxiKRZIRENOfqQ8XEeg4Qs= +modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU= +modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg= +modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI= +modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw= +modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg= +modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns= +modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w= +modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE= +modernc.org/sqlite v1.52.0 h1:p4dhYh2tXZCiyaqHwRVJDjIGKWyXayiQpThxgDzJaxo= +modernc.org/sqlite v1.52.0/go.mod h1:tcNzv5p84E0skkmJn038y+hWJbLQXQqEnQfeh5r2JLM= +modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0= +modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A= +modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y= +modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM= diff --git a/examples/encryption/main.go b/examples/encryption/main.go new file mode 100644 index 0000000..048321b --- /dev/null +++ b/examples/encryption/main.go @@ -0,0 +1,107 @@ +// Command encryption demonstrates liteorm's at-rest encryption for SQLite (via +// gosqlite's transparent page-level cipher): open the database with a 32-byte key, +// use the ORM exactly as you would unencrypted, and observe that the on-disk file +// is ciphertext β€” readable only by reopening with the same key. Encryption is an +// open/connection concern, orthogonal to what you do with the data afterward. +package main + +import ( + "bytes" + "context" + "crypto/rand" + "fmt" + "log" + "os" + "path/filepath" + + "liteorm.org/dialect/sqlite" + "liteorm.org/orm" +) + +type Note struct { + ID int64 + Text string +} + +func (Note) TableName() string { return "notes" } + +const secret = "the launch codes are 0000" + +func main() { + if err := run(); err != nil { + log.Fatal(err) + } +} + +func run() error { + ctx := context.Background() + dir, err := os.MkdirTemp("", "liteorm-encryption-*") + if err != nil { + return err + } + defer os.RemoveAll(dir) + path := filepath.Join(dir, "secret.db") + + // A 32-byte key. In production this comes from a KMS / secret store, never a + // literal or a value you can lose β€” losing the key means losing the data. + key := make([]byte, 32) + if _, err := rand.Read(key); err != nil { + return err + } + + // 1. Open encrypted and use the ORM exactly as you would on a plain database. + db, err := sqlite.OpenEncrypted(path, key) + if err != nil { + return err + } + if err := orm.AutoMigrate[Note](ctx, db); err != nil { + return err + } + notes := orm.NewRepo[Note](db) + n := &Note{Text: secret} + if err := notes.Create(ctx, n); err != nil { + return err + } + _ = db.Close() + fmt.Printf("wrote %s with an encryption key\n", filepath.Base(path)) + + // 2. The raw file is ciphertext β€” the plaintext must not appear on disk. + raw, err := os.ReadFile(path) + if err != nil { + return err + } + if bytes.Contains(raw, []byte(secret)) { + return fmt.Errorf("plaintext leaked into the on-disk file") + } + fmt.Println("on-disk bytes do NOT contain the plaintext β€” encrypted at rest βœ“") + + // 3. Reopen with the SAME key and read it back through the ORM. + reopened, err := sqlite.OpenEncrypted(path, key) + if err != nil { + return err + } + defer reopened.Close() + got, err := orm.NewRepo[Note](reopened).Get(ctx, n.ID) + if err != nil { + return err + } + fmt.Printf("reopened with the key β†’ %q\n", got.Text) + + // 4. The WRONG key cannot read it (decryption fails). + wrong := make([]byte, 32) + if _, err := rand.Read(wrong); err != nil { + return err + } + switch bad, err := sqlite.OpenEncrypted(path, wrong); { + case err != nil: + fmt.Println("opening with the wrong key fails βœ“") + default: + _, err = orm.NewRepo[Note](bad).Get(ctx, n.ID) + _ = bad.Close() + if err == nil { + return fmt.Errorf("the wrong key was able to read the data") + } + fmt.Println("the wrong key cannot read the data βœ“") + } + return nil +} diff --git a/examples/fixtures/main.go b/examples/fixtures/main.go index 82ba830..f0ba18f 100644 --- a/examples/fixtures/main.go +++ b/examples/fixtures/main.go @@ -37,10 +37,10 @@ type Member struct { func (Member) TableName() string { return "members" } // BeforeCreate stamps the tenant (org) from context when the caller left it unset. -func (m *Member) BeforeCreate(ctx context.Context, op *orm.Op[Member]) error { - if op.Model.OrgID == 0 { +func (m *Member) BeforeCreate(ctx context.Context, ev *orm.Event[Member]) error { + if ev.Model.OrgID == 0 { if org, ok := orgFrom(ctx); ok { - op.Model.OrgID = org + ev.Model.OrgID = org } } return nil @@ -54,10 +54,10 @@ type Project struct { func (Project) TableName() string { return "projects" } -func (p *Project) BeforeCreate(ctx context.Context, op *orm.Op[Project]) error { - if op.Model.OrgID == 0 { +func (p *Project) BeforeCreate(ctx context.Context, ev *orm.Event[Project]) error { + if ev.Model.OrgID == 0 { if org, ok := orgFrom(ctx); ok { - op.Model.OrgID = org + ev.Model.OrgID = org } } return nil diff --git a/examples/orm/main.go b/examples/orm/main.go index cb29cb7..e2f20f7 100644 --- a/examples/orm/main.go +++ b/examples/orm/main.go @@ -116,9 +116,9 @@ type Post struct { func (Post) TableName() string { return "posts" } // BeforeCreate is a ctx-first hook: derive the slug from the title when unset. -func (p *Post) BeforeCreate(_ context.Context, op *orm.Op[Post]) error { - if op.Model.Slug == "" { - op.Model.Slug = slugify(op.Model.Title) +func (p *Post) BeforeCreate(_ context.Context, ev *orm.Event[Post]) error { + if ev.Model.Slug == "" { + ev.Model.Slug = slugify(ev.Model.Title) } return nil } diff --git a/examples/search/main.go b/examples/search/main.go index 48ea282..4748da3 100644 --- a/examples/search/main.go +++ b/examples/search/main.go @@ -1,8 +1,13 @@ // Command search is a tour of liteorm's SQLite-only advanced search: vector // (sqlite-vec) nearest-neighbour, full-text (FTS5) keyword/phrase queries, and -// the hybrid reciprocal-rank-fusion that combines them β€” plus the at-rest -// encryption passthrough. The vec/fts indexes are sidecars keyed by the Doc -// model's primary key; a search returns ranked keys and Load fetches the rows. +// the hybrid reciprocal-rank-fusion that combines them. +// +// It shows two layers. The DECLARATIVE layer is the recommended path: a model +// declares its indexes (a SearchIndexes method, or `vec:`/`fts:` struct tags), +// AutoMigrate provisions the sidecars and keeps them in sync on every write, and +// the typed searcher β€” search.For[T](db).Vector / .FullText / .Hybrid β€” returns +// ranked models. The LOW-LEVEL layer drives the sidecars by hand +// (NewVector/NewFullText + Add) for callers that manage the index lifecycle. package main import ( @@ -15,6 +20,7 @@ import ( liteorm "liteorm.org" "liteorm.org/dialect/sqlite" "liteorm.org/dialect/sqlite/search" + "liteorm.org/orm" ) type Doc struct { @@ -25,6 +31,28 @@ type Doc struct { func (Doc) TableName() string { return "docs" } +// Article is the DECLARATIVE model: it declares a full-text index over its text +// columns and a vector index over the embedding. AutoMigrate creates the base +// table, the FTS5 and vec0 sidecars, and the triggers/hooks that keep them +// current β€” so plain Repo.Create/Update/Delete need no manual index calls. +type Article struct { + ID int64 + Title string + Body string + Embedding []float32 `orm:"-"` // sidecar-only; synced from the ORM write path +} + +func (Article) TableName() string { return "articles" } + +func (Article) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{ + orm.FullText("Title", "Body"), + orm.Vector("Embedding", 5).WithMetric(orm.Cosine), + } + // Equivalently, declare per-field tags: Title/Body with `fts:"..."` and the + // embedding with `vec:"dim=5;metric=cosine"`. +} + // Each doc carries a toy 5-dimensional "topic" embedding over // [animals, space, cooking, tech, music]. Real systems use a model; the shape // is the same: an int64 key and a []float32. @@ -64,6 +92,15 @@ func run() error { return err } defer db.Close() + + // ===== Declarative layer (recommended): declare β†’ migrate β†’ write β†’ search. + if err := declarative(ctx, db); err != nil { + return err + } + + // ===== Low-level building blocks: drive the vec/fts sidecars by hand, for + // callers that own the index lifecycle. + section("Low-level: manage the sidecars yourself") if _, err := db.ExecContext(ctx, `CREATE TABLE docs ( id INTEGER PRIMARY KEY, title TEXT NOT NULL, @@ -128,7 +165,7 @@ func run() error { if err != nil { return err } - docs, err := search.LoadScored[Doc](ctx, db, fused) + docs, err := search.FetchScored[Doc](ctx, db, fused) if err != nil { return err } @@ -138,41 +175,64 @@ func run() error { fmt.Println(" β†’ 'The Software Behind Spaceflight' tops the list: it ranks in BOTH") fmt.Println(" the vector neighbourhood and the text query, which neither alone ranks first.") - // ---- At-rest encryption ---- - section("Encryption: write encrypted, reopen with the key") - encPath := filepath.Join(dir, "secret.db") - key := make([]byte, 32) - for i := range key { - key[i] = byte(i * 7) + fmt.Println() + return nil +} + +// declarative is the recommended path: declare the indexes on the model, +// AutoMigrate, write through the Repo (the sidecars stay in sync automatically), +// and search with the typed helpers that return models in ranked order. +func declarative(ctx context.Context, db *liteorm.DB) error { + if err := orm.AutoMigrate[Article](ctx, db); err != nil { + return err + } + repo := orm.NewRepo[Article](db) + for _, c := range corpus { + a := &Article{Title: c.doc.Title, Body: c.doc.Body, Embedding: c.emb} + if err := repo.Create(ctx, a); err != nil { // inserts the row AND syncs both sidecars + return err + } } - enc, err := sqlite.OpenEncrypted(encPath, key) + + section("Declarative: nearest to the 'space' topic (search.For[T].Vector returns models)") + near, err := search.For[Article](db).Vector(ctx, spaceQuery, 3) if err != nil { return err } - if _, err := enc.ExecContext(ctx, `CREATE TABLE notes (id INTEGER PRIMARY KEY, text TEXT)`); err != nil { - return err + for _, h := range near { + fmt.Printf(" %.4f %s\n", h.Score, h.Model.Title) } - if _, err := enc.ExecContext(ctx, `INSERT INTO notes (text) VALUES (?)`, "encrypted at rest"); err != nil { + + section("Declarative: full-text 'software' AND 'flight'") + hits, err := search.For[Article](db).FullText(ctx, search.And(search.Term("software"), search.Term("flight")), 5) + if err != nil { return err } - _ = enc.Close() - reopened, err := sqlite.OpenEncrypted(encPath, key) + printHits(hits) + + section("Declarative: hybrid (RRF) vector 'space' βŠ• text 'software'") + fused, err := search.For[Article](db).Hybrid(ctx, spaceQuery, search.Term("software"), 4) if err != nil { return err } - defer reopened.Close() - var text string - if err := scanOne(ctx, reopened, &text, `SELECT text FROM notes WHERE id = 1`); err != nil { - return err + for _, h := range fused { + fmt.Printf(" %.4f %s\n", h.Score, h.Model.Title) } - fmt.Printf(" reopened with key β†’ %q (the on-disk file is ciphertext)\n", text) - - fmt.Println() + fmt.Println(" β†’ 'The Software Behind Spaceflight' tops the hybrid: strong in BOTH modalities.") return nil } +func printHits(hits []search.Hit[Article]) { + if len(hits) == 0 { + fmt.Println(" (no matches)") + } + for _, h := range hits { + fmt.Printf(" #%d %s\n", h.Model.ID, h.Model.Title) + } +} + func printDocs(ctx context.Context, db *liteorm.DB, keys []int64) error { - docs, err := search.Load[Doc](ctx, db, keys) + docs, err := search.Fetch[Doc](ctx, db, keys) if err != nil { return err } @@ -184,15 +244,3 @@ func printDocs(ctx context.Context, db *liteorm.DB, keys []int64) error { } return nil } - -func scanOne(ctx context.Context, db *liteorm.DB, dst *string, q string) error { - rows, err := db.QueryContext(ctx, q) - if err != nil { - return err - } - defer rows.Close() - if !rows.Next() { - return rows.Err() - } - return rows.Scan(dst) -} diff --git a/gen/gen.go b/gen/gen.go index c6d06fd..8f4954d 100644 --- a/gen/gen.go +++ b/gen/gen.go @@ -52,7 +52,7 @@ func FromType[T any]() Model { } func walkFields(t reflect.Type, colPrefix string, m *Model) { - for i := 0; i < t.NumField(); i++ { + for i := range t.NumField() { sf := t.Field(i) if !sf.IsExported() { continue diff --git a/go.work b/go.work index 6d96b16..90bd01b 100644 --- a/go.work +++ b/go.work @@ -10,6 +10,7 @@ use ( ./dialect/sqlite ./examples/blog ./examples/codegen + ./examples/encryption ./examples/fixtures ./examples/gormport ./examples/logging diff --git a/internal/scan/inflect.go b/internal/scan/inflect.go new file mode 100644 index 0000000..8ce96df --- /dev/null +++ b/internal/scan/inflect.go @@ -0,0 +1,88 @@ +package scan + +import ( + "strings" + "sync" + "sync/atomic" +) + +// pluralTables, when set, makes TableNameOf pluralize the default (no-TableName) +// table name. Off by default; toggled via orm.UsePluralTableNames. +var pluralTables atomic.Bool + +// irregularPlurals overrides the rule-based pluralizer for words English doesn't +// inflect regularly. Seeded with common ones; extended via RegisterPlural. +var irregularPlurals sync.Map // snake-case singular word -> plural + +// SetPluralTableNames toggles the pluralized default-table-name convention. This +// is the internal switch; the user-facing entry point is orm.UsePluralTableNames. +func SetPluralTableNames(on bool) { pluralTables.Store(on) } + +// RegisterPlural registers an irregular plural for a single snake-case word, e.g. +// RegisterPlural("person", "people") β€” for names the rule-based pluralizer gets +// wrong. The user-facing entry point is orm.RegisterPlural. +func RegisterPlural(singular, plural string) { irregularPlurals.Store(singular, plural) } + +// uncountable words pluralize to themselves. +var uncountable = map[string]bool{ + "equipment": true, "information": true, "rice": true, "money": true, "news": true, + "species": true, "series": true, "fish": true, "sheep": true, "deer": true, +} + +func init() { + for s, p := range map[string]string{ + "person": "people", "child": "children", "man": "men", "woman": "women", + "tooth": "teeth", "foot": "feet", "mouse": "mice", "goose": "geese", + } { + irregularPlurals.Store(s, p) + } +} + +// pluralizeTable pluralizes a snake_case table name by inflecting its last +// underscore segment: user -> users, category -> categories, user_profile -> +// user_profiles. It covers standard English rules plus a small set of irregulars; +// register exceptions with RegisterPlural. +func pluralizeTable(snake string) string { + head, word := "", snake + if i := strings.LastIndexByte(snake, '_'); i >= 0 { + head, word = snake[:i+1], snake[i+1:] + } + return head + pluralizeWord(word) +} + +func pluralizeWord(w string) string { + if w == "" { + return w + } + if p, ok := irregularPlurals.Load(w); ok { + return p.(string) + } + if uncountable[w] { + return w + } + switch { + case hasAnySuffix(w, "s", "x", "z", "ch", "sh"): + return w + "es" // bus->buses, box->boxes, dish->dishes + case strings.HasSuffix(w, "y") && len(w) >= 2 && !isVowel(w[len(w)-2]): + return w[:len(w)-1] + "ies" // category->categories (but day->days) + default: + return w + "s" + } +} + +func hasAnySuffix(s string, suffixes ...string) bool { + for _, x := range suffixes { + if strings.HasSuffix(s, x) { + return true + } + } + return false +} + +func isVowel(b byte) bool { + switch b { + case 'a', 'e', 'i', 'o', 'u': + return true + } + return false +} diff --git a/internal/scan/reflecttype.go b/internal/scan/reflecttype.go index 8a39646..a9c08c1 100644 --- a/internal/scan/reflecttype.go +++ b/internal/scan/reflecttype.go @@ -28,10 +28,15 @@ func GoTypeName(t reflect.Type) string { } // TableNameOf returns t's TableName() method result if it has one (value or -// pointer receiver), else the snake_case of the type name. +// pointer receiver), else the snake_case of the type name β€” pluralized when the +// plural-table-names convention is enabled (orm.UsePluralTableNames). func TableNameOf(t reflect.Type) string { if tn, ok := reflect.New(t).Interface().(interface{ TableName() string }); ok { return tn.TableName() } - return Snake(t.Name()) + name := Snake(t.Name()) + if pluralTables.Load() { + return pluralizeTable(name) + } + return name } diff --git a/internal/scan/scan.go b/internal/scan/scan.go index adbfce4..7f6f3dd 100644 --- a/internal/scan/scan.go +++ b/internal/scan/scan.go @@ -60,13 +60,13 @@ func buildPlan(t reflect.Type) *plan { p := &plan{typ: t, byDB: map[string]int{}} var walk func(rt reflect.Type, prefix []int, colPrefix string) walk = func(rt reflect.Type, prefix []int, colPrefix string) { - for i := 0; i < rt.NumField(); i++ { + for i := range rt.NumField() { sf := rt.Field(i) if !sf.IsExported() { continue } ft := sf.Type - idx := append(append([]int{}, prefix...), i) + idx := append(slices.Clone(prefix), i) // Flatten embedded structs (anonymous, or named+`embedded`) with prefix. if ep, emb := EmbeddedInfo(sf); emb { et := ft diff --git a/internal/sqlgen/dialects.go b/internal/sqlgen/dialects.go index 9932fac..be3b53a 100644 --- a/internal/sqlgen/dialects.go +++ b/internal/sqlgen/dialects.go @@ -1,6 +1,9 @@ package sqlgen import ( + "encoding/binary" + "fmt" + "math" "strconv" "strings" @@ -79,6 +82,211 @@ FROM sqlite_master m JOIN pragma_foreign_key_list(m.name) fk WHERE m.type = 'table' AND m.name NOT LIKE 'sqlite_%'` } +// ProvisionSearchSQL builds the CREATE VIRTUAL TABLE for a search sidecar β€” an +// FTS5 table for full-text, a sqlite-vec vec0 table for vector β€” keyed by the +// base table's primary key, plus the AFTER INSERT/UPDATE/DELETE triggers that +// keep it in sync when Sync is "triggers" (FTS external-content always; vectors +// when the embedding is a stored column). Idempotent via IF NOT EXISTS. +func (d sqliteDialect) ProvisionSearchSQL(spec dialect.SearchSpec) ([]string, error) { + switch spec.Kind { + case dialect.SearchVector: + stmts := []string{d.vec0CreateSQL(spec)} + if spec.Sync == "triggers" { + stmts = append(stmts, d.vecTriggerSQL(spec)...) + } + return stmts, nil + case dialect.SearchFullText: + if !d.ftsExternal(spec) { + // Only external-content FTS5 is sync-wired (its triggers mirror the base + // table). Refuse other content modes rather than create a sidecar that + // silently never updates. + return nil, fmt.Errorf("sqlite: full-text content=%q is not supported yet β€” only external content is kept in sync", spec.Content) + } + return append([]string{d.fts5CreateSQL(spec)}, d.ftsTriggerSQL(spec)...), nil + } + return nil, fmt.Errorf("sqlite: unknown search kind %d", spec.Kind) +} + +// DropSearchSQL drops a search sidecar and its sync triggers (idempotent). +func (d sqliteDialect) DropSearchSQL(spec dialect.SearchSpec) ([]string, error) { + var stmts []string + for _, suffix := range []string{"_ai", "_au", "_ad"} { + stmts = append(stmts, "DROP TRIGGER IF EXISTS "+string(d.QuoteIdent(nil, spec.Name+suffix))) + } + stmts = append(stmts, "DROP TABLE IF EXISTS "+string(d.QuoteIdent(nil, spec.Name))) + return stmts, nil +} + +func (d sqliteDialect) ftsExternal(spec dialect.SearchSpec) bool { + return spec.Content == "" || spec.Content == "external" +} + +// vecKeyToken is the SQL token a vec0 sidecar is keyed by: the implicit "rowid" +// for an int64 primary key, else the explicit (quoted) key column. +func (d sqliteDialect) vecKeyToken(spec dialect.SearchSpec) string { + if !spec.RowidKeyed() { + return string(d.QuoteIdent(nil, spec.PKColumn)) + } + return "rowid" +} + +// UpsertSearchRowSQL replaces one row's embedding in a hook-synced vec0 sidecar +// (DELETE then INSERT β€” vec0's portable replace). DeleteSearchRowSQL removes it. +func (d sqliteDialect) UpsertSearchRowSQL(spec dialect.SearchSpec, key any, value any) ([]dialect.SearchStmt, error) { + if spec.Kind != dialect.SearchVector { + return nil, fmt.Errorf("sqlite: hook-mode sync is implemented for vector sidecars only") + } + blob, err := vecBlob(value) + if err != nil { + return nil, err + } + name, keyCol := string(d.QuoteIdent(nil, spec.Name)), d.vecKeyToken(spec) + return []dialect.SearchStmt{ + {SQL: "DELETE FROM " + name + " WHERE " + keyCol + " = ?", Args: []any{key}}, + {SQL: "INSERT INTO " + name + "(" + keyCol + ", embedding) VALUES (?, ?)", Args: []any{key, blob}}, + }, nil +} + +func (d sqliteDialect) DeleteSearchRowSQL(spec dialect.SearchSpec, key any) ([]dialect.SearchStmt, error) { + name, keyCol := string(d.QuoteIdent(nil, spec.Name)), d.vecKeyToken(spec) + return []dialect.SearchStmt{ + {SQL: "DELETE FROM " + name + " WHERE " + keyCol + " = ?", Args: []any{key}}, + }, nil +} + +// vecBlob encodes an embedding into sqlite-vec's compact little-endian float32 +// blob, or passes a pre-encoded []byte through unchanged. +func vecBlob(value any) ([]byte, error) { + switch v := value.(type) { + case []byte: + return v, nil + case []float32: + b := make([]byte, len(v)*4) + for i, x := range v { + binary.LittleEndian.PutUint32(b[i*4:], math.Float32bits(x)) + } + return b, nil + } + return nil, fmt.Errorf("sqlite: vector embedding must be []float32 or []byte, got %T", value) +} + +// ftsTriggerSQL builds the standard FTS5 external-content sync triggers: insert +// mirrors the new row, delete emits the FTS5 'delete' command, update does both. +func (d sqliteDialect) ftsTriggerSQL(spec dialect.SearchSpec) []string { + qi := func(s string) string { return string(d.QuoteIdent(nil, s)) } + fts, base, pk := qi(spec.Name), qi(spec.Table), qi(spec.PKColumn) + cols := make([]string, len(spec.Columns)) + for i, c := range spec.Columns { + cols[i] = qi(c) + } + colList := strings.Join(cols, ", ") + newParts := []string{"new." + pk} + oldParts := []string{"old." + pk} + for _, c := range cols { + newParts = append(newParts, "new."+c) + oldParts = append(oldParts, "old."+c) + } + newVals := strings.Join(newParts, ", ") + oldVals := strings.Join(oldParts, ", ") + ins := func(prefixCol bool, vals string) string { + if prefixCol { // FTS5 'delete' command form: INSERT INTO fts(fts, rowid, cols) + return "INSERT INTO " + fts + "(" + fts + ", rowid, " + colList + ") VALUES('delete', " + vals + ")" + } + return "INSERT INTO " + fts + "(rowid, " + colList + ") VALUES (" + vals + ")" + } + return []string{ + "CREATE TRIGGER IF NOT EXISTS " + qi(spec.Name+"_ai") + " AFTER INSERT ON " + base + + " BEGIN " + ins(false, newVals) + "; END", + "CREATE TRIGGER IF NOT EXISTS " + qi(spec.Name+"_ad") + " AFTER DELETE ON " + base + + " BEGIN " + ins(true, oldVals) + "; END", + "CREATE TRIGGER IF NOT EXISTS " + qi(spec.Name+"_au") + " AFTER UPDATE ON " + base + + " BEGIN " + ins(true, oldVals) + "; " + ins(false, newVals) + "; END", + } +} + +// vecTriggerSQL builds the vec0 sync triggers for trigger mode (the embedding is +// a stored base column copied into the sidecar). int64 PKs key the sidecar by +// rowid; other PKs by an explicit key column. +func (d sqliteDialect) vecTriggerSQL(spec dialect.SearchSpec) []string { + qi := func(s string) string { return string(d.QuoteIdent(nil, s)) } + vec, base, pk, emb := qi(spec.Name), qi(spec.Table), qi(spec.PKColumn), qi(spec.Columns[0]) + keyCol := d.vecKeyToken(spec) + return []string{ + "CREATE TRIGGER IF NOT EXISTS " + qi(spec.Name+"_ai") + " AFTER INSERT ON " + base + + " WHEN new." + emb + " IS NOT NULL BEGIN " + + "INSERT INTO " + vec + "(" + keyCol + ", embedding) VALUES (new." + pk + ", new." + emb + "); END", + "CREATE TRIGGER IF NOT EXISTS " + qi(spec.Name+"_ad") + " AFTER DELETE ON " + base + + " BEGIN DELETE FROM " + vec + " WHERE " + keyCol + " = old." + pk + "; END", + "CREATE TRIGGER IF NOT EXISTS " + qi(spec.Name+"_au") + " AFTER UPDATE ON " + base + + " BEGIN DELETE FROM " + vec + " WHERE " + keyCol + " = old." + pk + "; " + + "INSERT INTO " + vec + "(" + keyCol + ", embedding) SELECT new." + pk + ", new." + emb + + " WHERE new." + emb + " IS NOT NULL; END", + } +} + +func (d sqliteDialect) vec0CreateSQL(spec dialect.SearchSpec) string { + colType := "float" + switch spec.Encoding { + case "int8": + colType = "int8" + case "bit": + colType = "bit" + } + embed := "embedding " + colType + "[" + strconv.Itoa(spec.Dim) + "]" + if colType != "bit" { // bit[] ranks by Hamming implicitly and rejects a distance= clause + metric := spec.Metric + if metric == "" { + metric = "l2" + } + embed += " distance=" + metric + } + var b strings.Builder + b.WriteString("CREATE VIRTUAL TABLE IF NOT EXISTS ") + b.Write(d.QuoteIdent(nil, spec.Name)) + b.WriteString(" USING vec0(") + if !spec.RowidKeyed() { + // A non-integer key (e.g. string PK) needs an explicit primary-key column; + // an integer PK maps onto vec0's implicit rowid. vec0's constructor parser + // rejects a quoted identifier here, so the key-column name is written raw. + b.WriteString(spec.PKColumn + " text primary key, ") + } + b.WriteString(embed) + b.WriteByte(')') + return b.String() +} + +func (d sqliteDialect) fts5CreateSQL(spec dialect.SearchSpec) string { + var b strings.Builder + b.WriteString("CREATE VIRTUAL TABLE IF NOT EXISTS ") + b.Write(d.QuoteIdent(nil, spec.Name)) + b.WriteString(" USING fts5(") + for i, c := range spec.Columns { + if i > 0 { + b.WriteString(", ") + } + b.Write(d.QuoteIdent(nil, c)) + } + // External content (the only sync-wired mode; ProvisionSearchSQL rejects others): + // the index reads the text from the base table by rowid, kept current by triggers. + b.WriteString(", content=" + sqlLit(spec.Table)) + b.WriteString(", content_rowid=" + sqlLit(spec.PKColumn)) + if spec.Tokenizer != "" { + b.WriteString(", tokenize=" + sqlLit(spec.Tokenizer)) + } + if len(spec.Prefix) > 0 { + parts := make([]string, len(spec.Prefix)) + for i, p := range spec.Prefix { + parts[i] = strconv.Itoa(p) + } + b.WriteString(", prefix=" + sqlLit(strings.Join(parts, " "))) + } + if spec.Detail != "" { + b.WriteString(", detail=" + spec.Detail) + } + b.WriteByte(')') + return b.String() +} + func sqliteType(goType string) string { switch goType { case "bool", "int", "int8", "int16", "int32", "int64", "uint", "uint8", "uint16", "uint32", "uint64": diff --git a/log/handler.go b/log/handler.go index 145a57c..ccd33f0 100644 --- a/log/handler.go +++ b/log/handler.go @@ -18,6 +18,7 @@ import ( "log/slog" "os" "path/filepath" + "slices" "strconv" "strings" "sync" @@ -65,7 +66,7 @@ func (h *handler) Enabled(_ context.Context, l slog.Level) bool { return l >= h. func (h *handler) WithAttrs(as []slog.Attr) slog.Handler { c := *h - c.attrs = append(append([]slog.Attr{}, h.attrs...), as...) + c.attrs = slices.Concat(h.attrs, as) return &c } diff --git a/orm/hooks.go b/orm/hooks.go index ef78612..b10bad8 100644 --- a/orm/hooks.go +++ b/orm/hooks.go @@ -8,37 +8,37 @@ import ( liteorm "liteorm.org" ) -// Op is the narrow, explicit handle passed to a hook β€” the executing session and +// Event is the narrow, explicit handle passed to a hook β€” the executing session and // the typed model. It is not a mutable shared *DB: control flow and capabilities // are visible at the signature, and hook errors propagate and abort (they are // never swallowed). Wrong hook signatures are a compile error, not a // silently-dead hook, because the interfaces are typed on T. -type Op[T any] struct { +type Event[T any] struct { Sess liteorm.Session Model *T } // Hook interfaces. A model opts in by implementing the ones it needs on *T: // -// func (u *User) BeforeCreate(ctx context.Context, op *orm.Op[User]) error { ... } +// func (u *User) BeforeCreate(ctx context.Context, ev *orm.Event[User]) error { ... } type ( BeforeCreateHook[T any] interface { - BeforeCreate(ctx context.Context, op *Op[T]) error + BeforeCreate(ctx context.Context, ev *Event[T]) error } AfterCreateHook[T any] interface { - AfterCreate(ctx context.Context, op *Op[T]) error + AfterCreate(ctx context.Context, ev *Event[T]) error } BeforeUpdateHook[T any] interface { - BeforeUpdate(ctx context.Context, op *Op[T]) error + BeforeUpdate(ctx context.Context, ev *Event[T]) error } AfterUpdateHook[T any] interface { - AfterUpdate(ctx context.Context, op *Op[T]) error + AfterUpdate(ctx context.Context, ev *Event[T]) error } BeforeDeleteHook[T any] interface { - BeforeDelete(ctx context.Context, op *Op[T]) error + BeforeDelete(ctx context.Context, ev *Event[T]) error } AfterDeleteHook[T any] interface { - AfterDelete(ctx context.Context, op *Op[T]) error + AfterDelete(ctx context.Context, ev *Event[T]) error } ) @@ -71,39 +71,43 @@ func hooksFor[T any]() hookFlags { func impl[I any](z any) bool { _, ok := z.(I); return ok } -func fireBeforeCreate[T any](ctx context.Context, op *Op[T]) error { +func fireBeforeCreate[T any](ctx context.Context, ev *Event[T]) error { if !hooksFor[T]().beforeCreate { return nil } - return any(op.Model).(BeforeCreateHook[T]).BeforeCreate(ctx, op) + return any(ev.Model).(BeforeCreateHook[T]).BeforeCreate(ctx, ev) } -func fireAfterCreate[T any](ctx context.Context, op *Op[T]) error { - if !hooksFor[T]().afterCreate { - return nil +func fireAfterCreate[T any](ctx context.Context, ev *Event[T]) error { + if hooksFor[T]().afterCreate { + if err := any(ev.Model).(AfterCreateHook[T]).AfterCreate(ctx, ev); err != nil { + return err + } } - return any(op.Model).(AfterCreateHook[T]).AfterCreate(ctx, op) + return syncSearchUpsert(ctx, ev) } -func fireBeforeUpdate[T any](ctx context.Context, op *Op[T]) error { +func fireBeforeUpdate[T any](ctx context.Context, ev *Event[T]) error { if !hooksFor[T]().beforeUpdate { return nil } - return any(op.Model).(BeforeUpdateHook[T]).BeforeUpdate(ctx, op) + return any(ev.Model).(BeforeUpdateHook[T]).BeforeUpdate(ctx, ev) } -func fireAfterUpdate[T any](ctx context.Context, op *Op[T]) error { - if !hooksFor[T]().afterUpdate { - return nil +func fireAfterUpdate[T any](ctx context.Context, ev *Event[T]) error { + if hooksFor[T]().afterUpdate { + if err := any(ev.Model).(AfterUpdateHook[T]).AfterUpdate(ctx, ev); err != nil { + return err + } } - return any(op.Model).(AfterUpdateHook[T]).AfterUpdate(ctx, op) + return syncSearchUpsert(ctx, ev) } -func fireBeforeDelete[T any](ctx context.Context, op *Op[T]) error { +func fireBeforeDelete[T any](ctx context.Context, ev *Event[T]) error { if !hooksFor[T]().beforeDelete { return nil } - return any(op.Model).(BeforeDeleteHook[T]).BeforeDelete(ctx, op) + return any(ev.Model).(BeforeDeleteHook[T]).BeforeDelete(ctx, ev) } -func fireAfterDelete[T any](ctx context.Context, op *Op[T]) error { +func fireAfterDelete[T any](ctx context.Context, ev *Event[T]) error { if !hooksFor[T]().afterDelete { return nil } - return any(op.Model).(AfterDeleteHook[T]).AfterDelete(ctx, op) + return any(ev.Model).(AfterDeleteHook[T]).AfterDelete(ctx, ev) } diff --git a/orm/loadpath.go b/orm/loadpath.go index 9723672..2852d20 100644 --- a/orm/loadpath.go +++ b/orm/loadpath.go @@ -139,7 +139,7 @@ func loadRelation(ctx context.Context, sess liteorm.Session, parents []reflect.V } perParent := make([][]reflect.Value, len(parents)) - for ci := 0; ci < children.Len(); ci++ { + for ci := range children.Len() { cv := children.Index(ci) key := normalizeKey(cv.FieldByIndex(targetField.Index).Interface()) for _, oi := range byKey[key] { @@ -191,7 +191,7 @@ func loadRelationM2M(ctx context.Context, sess liteorm.Session, parents []reflec return nil, err } perParent := make([][]reflect.Value, len(parents)) - for i := 0; i < items.Len(); i++ { + for i := range items.Len() { cv := items.Index(i) for _, oi := range byKey[keys[i]] { perParent[oi] = append(perParent[oi], cv) @@ -239,7 +239,7 @@ func scatter(parents []reflect.Value, rel *Relation, perParent [][]reflect.Value sl = reflect.Append(sl, k) } fv.Set(sl) - for j := 0; j < fv.Len(); j++ { + for j := range fv.Len() { handles = append(handles, fv.Index(j)) } } else if fv.Kind() == reflect.Pointer { diff --git a/orm/migrate.go b/orm/migrate.go index c6715b2..b9d3557 100644 --- a/orm/migrate.go +++ b/orm/migrate.go @@ -2,6 +2,7 @@ package orm import ( "context" + "fmt" "reflect" "slices" "strings" @@ -118,6 +119,154 @@ func migrateSchema(ctx context.Context, sess liteorm.Session, s *Schema, cfg mig } } } + // Provision any declared search sidecars (full-text / vector). No-op on a + // backend without dialect.SearchProvisioner β€” the capability is optional, like + // the index introspection above. + if err := provisionSearch(ctx, sess, s, d); err != nil { + return err + } + return nil +} + +// provisionSearch creates the model's declared search sidecars, idempotently, on +// a backend that implements dialect.SearchProvisioner; otherwise it is a no-op. +func provisionSearch(ctx context.Context, sess liteorm.Session, s *Schema, d dialect.Dialect) error { + if len(s.SearchIndexes) == 0 { + return nil + } + sp, ok := d.(dialect.SearchProvisioner) + if !ok { + return nil + } + for _, ix := range s.SearchIndexes { + spec, err := searchSpecOf(s, ix) + if err != nil { + return err + } + stmts, err := sp.ProvisionSearchSQL(spec) + if err != nil { + return err + } + for _, stmt := range stmts { + if _, err := sess.ExecContext(ctx, stmt); err != nil { + return err + } + } + } + return nil +} + +// searchSpecOf lowers a SearchIndex into the dialect-neutral dialect.SearchSpec, +// resolving Go field names to columns and supplying the base table's primary key +// (which keys the sidecar). +func searchSpecOf(s *Schema, ix SearchIndex) (dialect.SearchSpec, error) { + if s.PK == nil { + return dialect.SearchSpec{}, fmt.Errorf("orm: search index %q on %q needs a single-column primary key", ix.Name, s.Table) + } + colOf := make(map[string]string, len(s.Fields)) + for _, f := range s.Fields { + colOf[f.GoName] = f.Column + } + spec := dialect.SearchSpec{ + Name: ix.Name, + Table: s.Table, + PKColumn: s.PK.Column, + PKType: s.PK.dialField.GoType, + } + switch ix.Kind { + case dialect.SearchVector: + spec.Kind = dialect.SearchVector + spec.Dim, spec.Metric, spec.Encoding = ix.Dim, ix.Metric.String(), ix.Encoding + col, stored := colOf[ix.Fields[0]] // stored == the embedding is a real base column + if !stored { + col = ix.Fields[0] + } + spec.Columns = []string{col} + // Trigger mode copies a stored base column into the sidecar; hook mode keeps + // the embedding sidecar-only. Auto picks triggers when it can (the embedding + // is stored), else hooks. + switch ix.Sync { + case SyncTriggers: + if !stored { + return dialect.SearchSpec{}, fmt.Errorf("orm: vector index %q is trigger-synced but field %q is not a stored column (remove orm:%q to store it, or use hook sync)", ix.Name, ix.Fields[0], "-") + } + spec.Sync = "triggers" + case SyncHooks: + spec.Sync = "hooks" + default: + if stored { + spec.Sync = "triggers" + } else { + spec.Sync = "hooks" + } + } + case dialect.SearchFullText: + // FTS5 external content is keyed by the integer rowid, so a non-integer PK + // can't drive it (the trigger would assign a TEXT key to rowid and fail at + // the first write). Reject it at migrate time, not as a datatype mismatch later. + if !spec.RowidKeyed() { + return dialect.SearchSpec{}, fmt.Errorf("orm: full-text index %q requires an integer primary key β€” FTS5 is keyed by integer rowid, but %q has a %q key", ix.Name, s.Table, spec.PKType) + } + spec.Kind = dialect.SearchFullText + spec.Tokenizer, spec.Prefix, spec.Detail, spec.Content = ix.Tokenizer, ix.Prefix, ix.Detail, ix.Content + spec.Sync = "triggers" // FTS external-content is kept in sync by triggers + for _, fn := range ix.Fields { + c, ok := colOf[fn] + if !ok { + return dialect.SearchSpec{}, fmt.Errorf("orm: full-text index %q field %q is not a column", ix.Name, fn) + } + spec.Columns = append(spec.Columns, c) + } + } + return spec, nil +} + +// SearchSpecs returns the resolved, dialect-neutral search specifications for T β€” +// sidecar names defaulted, field names resolved to columns, sync mode resolved. +// The typed search helpers use it to locate a model's sidecars without +// re-deriving them. The result is in the same order as Schema.SearchIndexes. +func SearchSpecs[T any]() ([]dialect.SearchSpec, error) { + s, err := SchemaOf[T]() + if err != nil { + return nil, err + } + out := make([]dialect.SearchSpec, 0, len(s.SearchIndexes)) + for _, ix := range s.SearchIndexes { + spec, err := searchSpecOf(s, ix) + if err != nil { + return nil, err + } + out = append(out, spec) + } + return out, nil +} + +// DropSearchIndexes drops the search sidecars declared by T (idempotent). It is a +// no-op on a backend without dialect.SearchProvisioner. +func DropSearchIndexes[T any](ctx context.Context, sess liteorm.Session) error { + s, err := SchemaOf[T]() + if err != nil { + return err + } + sp, ok := sess.Dialect().(dialect.SearchProvisioner) + if !ok { + return nil + } + for _, ix := range s.SearchIndexes { + spec, err := searchSpecOf(s, ix) + if err != nil { + return err + } + stmts, err := sp.DropSearchSQL(spec) + if err != nil { + return err + } + for _, stmt := range stmts { + if _, err := sess.ExecContext(ctx, stmt); err != nil { + return err + } + } + } return nil } diff --git a/orm/naming.go b/orm/naming.go new file mode 100644 index 0000000..b5c1888 --- /dev/null +++ b/orm/naming.go @@ -0,0 +1,27 @@ +package orm + +import "liteorm.org/internal/scan" + +// UsePluralTableNames switches the default table-name convention to pluralized +// snake_case β€” User -> "users", Category -> "categories", UserProfile -> +// "user_profiles" β€” matching gorm's default, to ease porting. It is off by +// default: liteorm otherwise uses the exact snake_case of the type name. A model's +// own TableName() method always wins, regardless of this setting. +// +// It applies to both the orm and query front-ends (they share the naming +// resolver). Call it once at startup, before any schema is resolved; it resets +// the orm schema cache so a toggle takes effect, but other components may have +// already captured a name. Register irregular plurals the rules miss with +// RegisterPlural. +func UsePluralTableNames(on bool) { + scan.SetPluralTableNames(on) + schemaCache.Clear() +} + +// RegisterPlural overrides the pluralization of a single irregular word for the +// pluralized table-name convention β€” e.g. RegisterPlural("quiz", "quizzes"). Only +// the last snake_case segment of a table name is inflected, so register the bare +// word, not a compound (register "person", which also covers "blog_person"). +// Common irregulars (person, child, man, …) are built in; use this for the rest, +// or for domain words the rule-based pluralizer gets wrong. +func RegisterPlural(singular, plural string) { scan.RegisterPlural(singular, plural) } diff --git a/orm/naming_test.go b/orm/naming_test.go new file mode 100644 index 0000000..64a7f53 --- /dev/null +++ b/orm/naming_test.go @@ -0,0 +1,60 @@ +package orm_test + +import ( + "testing" + + "liteorm.org/orm" +) + +type plUser struct{ ID int64 } +type plCategory struct{ ID int64 } +type plBox struct{ ID int64 } +type plUserProfile struct{ ID int64 } +type plPerson struct{ ID int64 } +type plQuiz struct{ ID int64 } + +type plExplicit struct{ ID int64 } + +func (plExplicit) TableName() string { return "my_explicit" } + +func tableOf[T any](t *testing.T) string { + t.Helper() + s, err := orm.SchemaOf[T]() + if err != nil { + t.Fatal(err) + } + return s.Table +} + +func TestUsePluralTableNames(t *testing.T) { + // Default (off): exact snake_case singular. + if got := tableOf[plUser](t); got != "pl_user" { + t.Fatalf("default table name = %q, want pl_user", got) + } + + orm.UsePluralTableNames(true) + t.Cleanup(func() { orm.UsePluralTableNames(false) }) + // The default z->es rule gives the wrong plural for "quiz"; register the real + // one. (Keyed on the bare last segment, so it applies to the pl_quiz table.) + orm.RegisterPlural("quiz", "quizzes") + + for _, tc := range []struct{ got, want string }{ + {tableOf[plUser](t), "pl_users"}, + {tableOf[plCategory](t), "pl_categories"}, + {tableOf[plBox](t), "pl_boxes"}, + {tableOf[plUserProfile](t), "pl_user_profiles"}, + {tableOf[plPerson](t), "pl_people"}, // built-in irregular (person -> people) + {tableOf[plQuiz](t), "pl_quizzes"}, // RegisterPlural override of the rule + {tableOf[plExplicit](t), "my_explicit"}, // explicit TableName always wins + } { + if tc.got != tc.want { + t.Errorf("table name = %q, want %q", tc.got, tc.want) + } + } + + // Toggling back restores the singular default (the cache was reset). + orm.UsePluralTableNames(false) + if got := tableOf[plUser](t); got != "pl_user" { + t.Errorf("after toggle off, table name = %q, want pl_user", got) + } +} diff --git a/orm/read.go b/orm/read.go index 6fe137e..95fbe5e 100644 --- a/orm/read.go +++ b/orm/read.go @@ -3,6 +3,7 @@ package orm import ( "context" "fmt" + "slices" "liteorm.org/internal/scan" "liteorm.org/query" @@ -18,7 +19,7 @@ type Scope[T any] func(*query.SelectBuilder[T]) *query.SelectBuilder[T] // copied so sibling Repo views never alias each other's scope chain. func (r *Repo[T]) withScope(sc Scope[T]) *Repo[T] { c := *r - c.readScopes = append(append([]Scope[T](nil), r.readScopes...), sc) + c.readScopes = append(slices.Clone(r.readScopes), sc) return &c } @@ -63,7 +64,7 @@ func (r *Repo[T]) Offset(n int) *Repo[T] { // over the query builder, so teams can package and share common filters. func (r *Repo[T]) Scopes(scopes ...Scope[T]) *Repo[T] { c := *r - c.readScopes = append(append([]Scope[T](nil), r.readScopes...), scopes...) + c.readScopes = slices.Concat(r.readScopes, scopes) return &c } diff --git a/orm/repo.go b/orm/repo.go index f648297..00b7e46 100644 --- a/orm/repo.go +++ b/orm/repo.go @@ -168,8 +168,8 @@ func (r *Repo[T]) Create(ctx context.Context, v *T) error { if err != nil { return err } - op := &Op[T]{Sess: r.sess, Model: v} - if err := fireBeforeCreate(ctx, op); err != nil { + ev := &Event[T]{Sess: r.sess, Model: v} + if err := fireBeforeCreate(ctx, ev); err != nil { return err } setAutoTimes(s, v, true) @@ -178,7 +178,7 @@ func (r *Repo[T]) Create(ctx context.Context, v *T) error { if err := query.InsertCapturingPK(ctx, r.sess, ins, v); err != nil { return err } - return fireAfterCreate(ctx, op) + return fireAfterCreate(ctx, ev) } // Get fetches the row whose primary key equals the given key (honoring the @@ -271,8 +271,8 @@ func (r *Repo[T]) Update(ctx context.Context, v *T) error { if len(s.PKs) == 0 { return fmt.Errorf("orm: type %T has no primary key", *v) } - op := &Op[T]{Sess: r.sess, Model: v} - if err := fireBeforeUpdate(ctx, op); err != nil { + ev := &Event[T]{Sess: r.sess, Model: v} + if err := fireBeforeUpdate(ctx, ev); err != nil { return err } setAutoTimes(s, v, false) @@ -296,11 +296,11 @@ func (r *Repo[T]) Update(ctx context.Context, v *T) error { return err } // A keyed update that matched no row (a wrong PK, or a soft-deleted row that is - // out of the current scope) is ErrNoRows, not a silent no-op. + // out of the current scope) is ErrNoRows, not a silent no-ev. if res.RowsAffected() == 0 { return liteorm.ErrNoRows } - return fireAfterUpdate(ctx, op) + return fireAfterUpdate(ctx, ev) } // Delete removes v: a soft delete (UPDATE deleted_at = now) when the model has a @@ -323,8 +323,8 @@ func (r *Repo[T]) delete(ctx context.Context, v *T, force bool) error { if len(s.PKs) == 0 { return fmt.Errorf("orm: type %T has no primary key", *v) } - op := &Op[T]{Sess: r.sess, Model: v} - if err := fireBeforeDelete(ctx, op); err != nil { + ev := &Event[T]{Sess: r.sess, Model: v} + if err := fireBeforeDelete(ctx, ev); err != nil { return err } where := r.pkWhere(s, v) @@ -360,9 +360,14 @@ func (r *Repo[T]) delete(ctx context.Context, v *T, force bool) error { return err } // A keyed delete that matched no row (a wrong PK, or an already-deleted row out - // of the current scope) is ErrNoRows, not a silent no-op. + // of the current scope) is ErrNoRows, not a silent no-ev. if res.RowsAffected() == 0 { return liteorm.ErrNoRows } - return fireAfterDelete(ctx, op) + if force || s.SoftDelete == nil { // a hard delete removes the row from hook-synced sidecars too + if err := syncSearchDelete(ctx, r.sess, s, v); err != nil { + return err + } + } + return fireAfterDelete(ctx, ev) } diff --git a/orm/schema.go b/orm/schema.go index 0a0d69f..ae63abb 100644 --- a/orm/schema.go +++ b/orm/schema.go @@ -11,6 +11,7 @@ package orm import ( "fmt" "reflect" + "slices" "strconv" "strings" "sync" @@ -96,6 +97,10 @@ type Schema struct { PK *Field // the single PK when there is exactly one; nil for a composite key SoftDelete *Field Relations map[string]*Relation + // SearchIndexes are the model's full-text / vector sidecars, collected from + // `vector`/`fts` struct tags and the optional SearchIndexes method. Empty for + // a model with no search indexes. + SearchIndexes []SearchIndex } // WriteColumns returns the columns to write: writable, non-auto-increment, and @@ -175,17 +180,22 @@ func buildSchema(t reflect.Type) (*Schema, error) { if err := walkRelations(t, nil, s); err != nil { return nil, err } + ix, err := resolveSearchIndexes(t, s) + if err != nil { + return nil, err + } + s.SearchIndexes = ix return s, nil } func walkColumns(t reflect.Type, prefix []int, colPrefix string, s *Schema) { - for i := 0; i < t.NumField(); i++ { + for i := range t.NumField() { sf := t.Field(i) if !sf.IsExported() { continue } ft := sf.Type - idx := append(append([]int{}, prefix...), i) + idx := append(slices.Clone(prefix), i) if ep, emb := scan.EmbeddedInfo(sf); emb { et := ft for et.Kind() == reflect.Pointer { @@ -232,13 +242,13 @@ func addColumn(sf reflect.StructField, idx []int, colPrefix string, ci scan.Colu } func walkRelations(t reflect.Type, prefix []int, s *Schema) error { - for i := 0; i < t.NumField(); i++ { + for i := range t.NumField() { sf := t.Field(i) if !sf.IsExported() { continue } ft := sf.Type - idx := append(append([]int{}, prefix...), i) + idx := append(slices.Clone(prefix), i) if _, emb := scan.EmbeddedInfo(sf); emb { et := ft for et.Kind() == reflect.Pointer { diff --git a/orm/search.go b/orm/search.go new file mode 100644 index 0000000..900f44a --- /dev/null +++ b/orm/search.go @@ -0,0 +1,458 @@ +package orm + +import ( + "fmt" + "reflect" + "strconv" + "strings" + + "liteorm.org/dialect" + "liteorm.org/internal/scan" +) + +// SyncMode chooses how a sidecar index is kept current with its base table. +type SyncMode int + +const ( + // SyncAuto lets the backend pick: triggers for full-text (the indexed text + // already lives on the base table, so it is free and catches every write) and + // hooks for vectors (the embedding need not be duplicated on the base table). + SyncAuto SyncMode = iota + // SyncTriggers keeps the sidecar current with SQL triggers, so bulk and raw + // (non-ORM) writes stay in sync. A vector index in trigger mode stores the + // embedding as a column on the base table. + SyncTriggers + // SyncHooks keeps the sidecar current from the ORM write path only (no base + // column duplication for vectors); writes that bypass the ORM are not indexed. + SyncHooks +) + +// Metric is a vector distance function. +type Metric int + +const ( + // L2 is Euclidean distance (the vec0 default). + L2 Metric = iota + // Cosine is cosine distance β€” the usual choice for normalized embeddings. + Cosine + // L1 is Manhattan distance. + L1 + // Hamming is bit-vector Hamming distance. + Hamming +) + +// SearchIndex declares a search sidecar (full-text or vector) attached to a +// model. It is the single representation that both the struct-tag sugar and the +// SearchIndexes method lower into; the migrator, the sync layer, and the typed +// search helpers all read it. The zero value is not useful β€” construct one with +// [FullText] or [Vector]. +type SearchIndex struct { + Kind dialect.SearchKind + // Name is the sidecar table name. Empty means the default: _fts for a + // full-text index,
_vec for a vector index. With more than one index of + // the same kind on a model, give each an explicit name. + Name string + // Fields are the model's Go field names the index covers β€” the text columns + // for a full-text index, the single embedding field for a vector index. + Fields []string + // Sync selects the synchronization strategy (see [SyncMode]). + Sync SyncMode + + // Full-text options. + Tokenizer string // FTS5 tokenizer, e.g. "porter unicode61" (default "unicode61") + Prefix []int // FTS5 prefix-index lengths + Detail string // FTS5 detail level: "full" (default), "column", or "none" + Content string // FTS5 content mode; only external (the default, "") is supported today + Weights []float64 // per-column BM25 weights; len must equal len(Fields) when set + + // Vector options. + Dim int // embedding dimension (required, > 0) + Metric Metric // distance function (default Cosine) + Encoding string // "float32" (default), "int8", or "bit" +} + +// FullText declares a full-text (FTS5) index over the named model fields. Refine +// it with the With* methods; for the common single-field case the defaults +// (unicode61 tokenizer, external content, trigger sync) are usually right. +func FullText(fields ...string) SearchIndex { + return SearchIndex{Kind: dialect.SearchFullText, Fields: fields} +} + +// Vector declares a vector (sqlite-vec) index over the named embedding field +// with the given dimension. The default metric is [Cosine]; override with +// [SearchIndex.WithMetric]. +func Vector(field string, dim int) SearchIndex { + return SearchIndex{Kind: dialect.SearchVector, Fields: []string{field}, Dim: dim, Metric: Cosine} +} + +// Named overrides the sidecar table name. +func (i SearchIndex) Named(name string) SearchIndex { i.Name = name; return i } + +// WithSync sets the synchronization strategy. +func (i SearchIndex) WithSync(m SyncMode) SearchIndex { i.Sync = m; return i } + +// WithTokenizer sets the FTS5 tokenizer (full-text only). +func (i SearchIndex) WithTokenizer(t string) SearchIndex { i.Tokenizer = t; return i } + +// WithPrefix sets FTS5 prefix-index lengths (full-text only). +func (i SearchIndex) WithPrefix(lengths ...int) SearchIndex { i.Prefix = lengths; return i } + +// WithDetail sets the FTS5 detail level (full-text only). +func (i SearchIndex) WithDetail(d string) SearchIndex { i.Detail = d; return i } + +// WithWeights sets per-column BM25 weights; the count must match the field count +// (full-text only). +func (i SearchIndex) WithWeights(w ...float64) SearchIndex { i.Weights = w; return i } + +// WithMetric sets the distance function (vector only). +func (i SearchIndex) WithMetric(m Metric) SearchIndex { i.Metric = m; return i } + +// WithEncoding sets the vector storage encoding (vector only). +func (i SearchIndex) WithEncoding(e string) SearchIndex { i.Encoding = e; return i } + +func (m Metric) String() string { + switch m { + case L2: + return "l2" + case Cosine: + return "cosine" + case L1: + return "l1" + case Hamming: + return "hamming" + } + return "unknown" +} + +func parseMetric(s string) (Metric, error) { + switch strings.ToLower(strings.TrimSpace(s)) { + case "", "cosine": + return Cosine, nil + case "l2", "euclidean": + return L2, nil + case "l1", "manhattan": + return L1, nil + case "hamming": + return Hamming, nil + } + return 0, fmt.Errorf("unknown metric %q (want l2, cosine, l1, or hamming)", s) +} + +func parseSyncMode(s string) (SyncMode, error) { + switch strings.ToLower(strings.TrimSpace(s)) { + case "", "auto": + return SyncAuto, nil + case "triggers", "trigger": + return SyncTriggers, nil + case "hooks", "hook": + return SyncHooks, nil + } + return 0, fmt.Errorf("unknown sync mode %q (want triggers or hooks)", s) +} + +// SearchIndexer is the optional interface a model implements to declare its +// search indexes in typed Go rather than struct tags. Tag-declared and +// method-declared indexes are merged; on a sidecar-name collision the +// method-declared one wins. +type SearchIndexer interface { + SearchIndexes() []SearchIndex +} + +// resolveSearchIndexes collects a model's search indexes from struct tags and +// from the optional SearchIndexes method, validates them, fills default sidecar +// names, and returns the merged set. Returns nil (no error) for a model with no +// search indexes. +func resolveSearchIndexes(t reflect.Type, s *Schema) ([]SearchIndex, error) { + valid := structFieldNames(t) + + tagged, err := searchIndexesFromTags(t) + if err != nil { + return nil, err + } + + var method []SearchIndex + if v, ok := reflect.New(t).Interface().(SearchIndexer); ok { + method = v.SearchIndexes() + } + + // Validate + default-name every index, keyed by sidecar name. The method's + // indexes are applied last so they win on a name collision. + byName := map[string]SearchIndex{} + order := []string{} + add := func(ix SearchIndex, fromMethod bool) error { + if err := validateSearchIndex(&ix, s.Table, valid); err != nil { + return err + } + if _, dup := byName[ix.Name]; dup && !fromMethod { + return fmt.Errorf("orm: model %s has two search indexes named %q β€” give each an explicit name", t.Name(), ix.Name) + } + if _, seen := byName[ix.Name]; !seen { + order = append(order, ix.Name) + } + byName[ix.Name] = ix + return nil + } + for _, ix := range tagged { + if err := add(ix, false); err != nil { + return nil, err + } + } + for _, ix := range method { + if err := add(ix, true); err != nil { + return nil, err + } + } + if len(order) == 0 { + return nil, nil + } + out := make([]SearchIndex, 0, len(order)) + for _, name := range order { + out = append(out, byName[name]) + } + return out, nil +} + +// validateSearchIndex checks an index and fills its default sidecar name. table +// is the model's table; valid is the set of the model's Go field names. +func validateSearchIndex(ix *SearchIndex, table string, valid map[string]bool) error { + if len(ix.Fields) == 0 { + return fmt.Errorf("orm: search index on %q declares no fields", table) + } + for _, f := range ix.Fields { + if !valid[f] { + return fmt.Errorf("orm: search index on %q references unknown field %q", table, f) + } + } + switch ix.Kind { + case dialect.SearchVector: + if len(ix.Fields) != 1 { + return fmt.Errorf("orm: vector index on %q must cover exactly one field, got %d", table, len(ix.Fields)) + } + if ix.Dim <= 0 { + return fmt.Errorf("orm: vector index on %q.%s needs a positive dim", table, ix.Fields[0]) + } + switch ix.Encoding { + case "", "float32", "int8", "bit": + default: + return fmt.Errorf("orm: vector index on %q has unknown encoding %q (want float32, int8, or bit)", table, ix.Encoding) + } + if ix.Name == "" { + ix.Name = table + "_vec" + } + case dialect.SearchFullText: + if len(ix.Weights) != 0 && len(ix.Weights) != len(ix.Fields) { + return fmt.Errorf("orm: full-text index on %q has %d weights for %d fields", table, len(ix.Weights), len(ix.Fields)) + } + switch ix.Detail { + case "", "full", "column", "none": + default: + return fmt.Errorf("orm: full-text index on %q has unknown detail %q (want full, column, or none)", table, ix.Detail) + } + if ix.Name == "" { + ix.Name = table + "_fts" + } + default: + return fmt.Errorf("orm: search index on %q has unknown kind %d", table, ix.Kind) + } + return nil +} + +// structFieldNames returns the set of exported, non-relation Go field names of t, +// recursing through embedded structs (matching how columns are walked). +func structFieldNames(t reflect.Type) map[string]bool { + out := map[string]bool{} + var walk func(reflect.Type) + walk = func(t reflect.Type) { + for i := range t.NumField() { + sf := t.Field(i) + if !sf.IsExported() { + continue + } + if _, emb := scan.EmbeddedInfo(sf); emb { + et := sf.Type + for et.Kind() == reflect.Pointer { + et = et.Elem() + } + if et.Kind() == reflect.Struct { + walk(et) + } + continue + } + out[sf.Name] = true + } + } + walk(t) + return out +} + +// searchIndexesFromTags walks a struct's fields and builds one SearchIndex per +// field tagged with a `vector` or `fts` marker. Tags express the common +// single-field case; combined multi-field indexes use the SearchIndexes method. +func searchIndexesFromTags(t reflect.Type) ([]SearchIndex, error) { + var out []SearchIndex + var walk func(reflect.Type) error + walk = func(t reflect.Type) error { + for i := range t.NumField() { + sf := t.Field(i) + if !sf.IsExported() { + continue + } + if _, emb := scan.EmbeddedInfo(sf); emb { + et := sf.Type + for et.Kind() == reflect.Pointer { + et = et.Elem() + } + if et.Kind() == reflect.Struct { + if err := walk(et); err != nil { + return err + } + } + continue + } + ix, ok, err := searchTagOf(sf) + if err != nil { + return err + } + if ok { + out = append(out, ix) + } + } + return nil + } + if err := walk(t); err != nil { + return nil, err + } + return out, nil +} + +// searchTagOf reads a dedicated `vec:` or `fts:` (alias `fts5:`) struct tag whose +// value is a ;-separated list of key=value options β€” the same grammar the sibling +// gosqlite gorm plugins use, so a model ports across with no tag changes. ok is +// false when the field carries no search tag. A combined multi-column index uses +// the SearchIndexes method instead. +func searchTagOf(sf reflect.StructField) (SearchIndex, bool, error) { + vecTag, hasVec := sf.Tag.Lookup("vec") + ftsTag, hasFTS := sf.Tag.Lookup("fts") + if !hasFTS { + ftsTag, hasFTS = sf.Tag.Lookup("fts5") // gosqlite-compatible alias + } + switch { + case hasVec && hasFTS: + return SearchIndex{}, false, fmt.Errorf("orm: field %q is tagged both vec and fts", sf.Name) + case hasVec: + return parseVecTag(sf.Name, vecTag) + case hasFTS: + return parseFTSTag(sf.Name, ftsTag) + } + return SearchIndex{}, false, nil +} + +func parseVecTag(field, tag string) (SearchIndex, bool, error) { + opts := parseSubOpts(tag) + dim, err := strconv.Atoi(strings.TrimSpace(opts["dim"])) + if err != nil { + return SearchIndex{}, false, fmt.Errorf("orm: field %q vec tag needs dim=N, got %q", field, opts["dim"]) + } + ix := Vector(field, dim) + if m, ok := opts["metric"]; ok { + if ix.Metric, err = parseMetric(m); err != nil { + return SearchIndex{}, false, fmt.Errorf("orm: field %q: %w", field, err) + } + } + if e := opts["encoding"]; e != "" { + ix.Encoding = e + } + if err := applyCommonSearchOpts(&ix, opts); err != nil { + return SearchIndex{}, false, fmt.Errorf("orm: field %q: %w", field, err) + } + return ix, true, nil +} + +func parseFTSTag(field, tag string) (SearchIndex, bool, error) { + opts := parseSubOpts(tag) + ix := FullText(field) + if tk := opts["tokenize"]; tk != "" { + ix.Tokenizer = strings.ReplaceAll(tk, "+", " ") // gosqlite "porter+unicode61" -> "porter unicode61" + } + if d := opts["detail"]; d != "" { + ix.Detail = d + } + if p := opts["prefix"]; p != "" { + lengths, err := parseIntList(p) + if err != nil { + return SearchIndex{}, false, fmt.Errorf("orm: field %q prefix: %w", field, err) + } + ix.Prefix = lengths + } + if w := opts["weights"]; w != "" { + ws, err := parseFloatList(w) + if err != nil { + return SearchIndex{}, false, fmt.Errorf("orm: field %q weights: %w", field, err) + } + ix.Weights = ws + } + if err := applyCommonSearchOpts(&ix, opts); err != nil { + return SearchIndex{}, false, fmt.Errorf("orm: field %q: %w", field, err) + } + return ix, true, nil +} + +// parseSubOpts parses a ;-separated list of key=value (or bare-key) options into +// a map with lowercased keys. +func parseSubOpts(s string) map[string]string { + out := map[string]string{} + for part := range strings.SplitSeq(s, ";") { + part = strings.TrimSpace(part) + if part == "" { + continue + } + k, v, _ := strings.Cut(part, "=") + out[strings.ToLower(strings.TrimSpace(k))] = strings.TrimSpace(v) + } + return out +} + +func parseFloatList(s string) ([]float64, error) { + fields := strings.FieldsFunc(s, func(r rune) bool { return r == ' ' || r == ',' }) + out := make([]float64, 0, len(fields)) + for _, f := range fields { + n, err := strconv.ParseFloat(f, 64) + if err != nil { + return nil, fmt.Errorf("%q is not a number", f) + } + out = append(out, n) + } + return out, nil +} + +// applyCommonSearchOpts reads option keys shared by both index kinds (table name +// and sync mode). +func applyCommonSearchOpts(ix *SearchIndex, opts map[string]string) error { + if name := opts["table"]; name != "" { + ix.Name = name + } + if s, ok := opts["sync"]; ok { + m, err := parseSyncMode(s) + if err != nil { + return err + } + ix.Sync = m + } + return nil +} + +// parseIntList parses a list of ints separated by spaces or commas (so a prefix +// list reads the same under the comma-delimited orm tag and the +// semicolon-delimited gorm tag). +func parseIntList(s string) ([]int, error) { + fields := strings.FieldsFunc(s, func(r rune) bool { return r == ' ' || r == ',' }) + out := make([]int, 0, len(fields)) + for _, f := range fields { + n, err := strconv.Atoi(f) + if err != nil { + return nil, fmt.Errorf("%q is not an integer", f) + } + out = append(out, n) + } + return out, nil +} diff --git a/orm/search_test.go b/orm/search_test.go new file mode 100644 index 0000000..1a3901a --- /dev/null +++ b/orm/search_test.go @@ -0,0 +1,315 @@ +package orm_test + +import ( + "slices" + "strings" + "testing" + + "liteorm.org/dialect" + "liteorm.org/orm" +) + +func indexByName(s *orm.Schema, name string) (orm.SearchIndex, bool) { + for _, ix := range s.SearchIndexes { + if ix.Name == name { + return ix, true + } + } + return orm.SearchIndex{}, false +} + +// --- tag-declared indexes (dedicated vec:/fts: tags; gosqlite-compatible) --- + +type vecTagModel struct { + ID int64 + Title string + Embedding []float32 `vec:"dim=8;metric=cosine"` +} + +func (vecTagModel) TableName() string { return "vec_t" } + +func TestSearchTags_Vector(t *testing.T) { + s, err := orm.SchemaOf[vecTagModel]() + if err != nil { + t.Fatal(err) + } + if len(s.SearchIndexes) != 1 { + t.Fatalf("want 1 index, got %d", len(s.SearchIndexes)) + } + ix := s.SearchIndexes[0] + if ix.Kind != dialect.SearchVector { + t.Errorf("kind = %v, want SearchVector", ix.Kind) + } + if ix.Name != "vec_t_vec" { + t.Errorf("name = %q, want vec_t_vec", ix.Name) + } + if len(ix.Fields) != 1 || ix.Fields[0] != "Embedding" { + t.Errorf("fields = %v, want [Embedding]", ix.Fields) + } + if ix.Dim != 8 { + t.Errorf("dim = %d, want 8", ix.Dim) + } + if ix.Metric != orm.Cosine { + t.Errorf("metric = %v, want Cosine", ix.Metric) + } +} + +// gosqlite used the `fts5:` tag with `+`-joined tokenizers; liteorm accepts that +// form unchanged so a model ports across. +type fts5AliasModel struct { + ID int64 + Body string `fts5:"tokenize=porter+unicode61"` +} + +func (fts5AliasModel) TableName() string { return "fts5_alias" } + +func TestSearchTags_FTS5Alias(t *testing.T) { + s, err := orm.SchemaOf[fts5AliasModel]() + if err != nil { + t.Fatal(err) + } + if len(s.SearchIndexes) != 1 || s.SearchIndexes[0].Kind != dialect.SearchFullText { + t.Fatalf("want 1 full-text index, got %v", s.SearchIndexes) + } + if got := s.SearchIndexes[0].Tokenizer; got != "porter unicode61" { + t.Errorf("tokenizer = %q, want %q (the + should normalize to a space)", got, "porter unicode61") + } +} + +type ftsTagModel struct { + ID int64 + Body string `fts:"tokenize=porter unicode61;prefix=2 3 4"` +} + +func (ftsTagModel) TableName() string { return "fts_tag" } + +func TestSearchTags_FullText(t *testing.T) { + s, err := orm.SchemaOf[ftsTagModel]() + if err != nil { + t.Fatal(err) + } + if len(s.SearchIndexes) != 1 { + t.Fatalf("want 1 index, got %d", len(s.SearchIndexes)) + } + ix := s.SearchIndexes[0] + if ix.Kind != dialect.SearchFullText || ix.Name != "fts_tag_fts" { + t.Errorf("got kind=%v name=%q", ix.Kind, ix.Name) + } + if ix.Tokenizer != "porter unicode61" { + t.Errorf("tokenizer = %q, want %q", ix.Tokenizer, "porter unicode61") + } + if want := []int{2, 3, 4}; !slices.Equal(ix.Prefix, want) { + t.Errorf("prefix = %v, want %v", ix.Prefix, want) + } +} + +func TestSearchTags_VectorMetricDefaultsCosine(t *testing.T) { + type m struct { + ID int64 + Emb []float32 `vec:"dim=4"` + } + s, err := orm.SchemaOf[m]() + if err != nil { + t.Fatal(err) + } + if s.SearchIndexes[0].Metric != orm.Cosine { + t.Errorf("default metric = %v, want Cosine", s.SearchIndexes[0].Metric) + } +} + +// --- method-declared indexes (full power: multi-field FTS, weights, sync) --- + +type methodModel struct { + ID int64 + Title string + Body string + Embedding []float32 `orm:"-"` // sidecar-only; not a base column +} + +func (methodModel) TableName() string { return "mm" } + +func (methodModel) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{ + orm.FullText("Title", "Body").WithTokenizer("porter unicode61").WithWeights(2, 1), + orm.Vector("Embedding", 8).WithMetric(orm.Cosine).WithSync(orm.SyncHooks), + } +} + +func TestSearchMethod(t *testing.T) { + s, err := orm.SchemaOf[methodModel]() + if err != nil { + t.Fatal(err) + } + if len(s.SearchIndexes) != 2 { + t.Fatalf("want 2 indexes, got %d", len(s.SearchIndexes)) + } + fts, ok := indexByName(s, "mm_fts") + if !ok { + t.Fatal("missing mm_fts") + } + if !slices.Equal(fts.Fields, []string{"Title", "Body"}) { + t.Errorf("fts fields = %v", fts.Fields) + } + if !slices.Equal(fts.Weights, []float64{2, 1}) { + t.Errorf("fts weights = %v", fts.Weights) + } + vec, ok := indexByName(s, "mm_vec") + if !ok { + t.Fatal("missing mm_vec") + } + if vec.Sync != orm.SyncHooks { + t.Errorf("vec sync = %v, want SyncHooks", vec.Sync) + } +} + +// --- merge: method wins on a sidecar-name collision --- + +type mergeModel struct { + ID int64 + Body string `fts:""` + Embedding []float32 `vec:"dim=4"` // default name mm2_vec, dim 4 +} + +func (mergeModel) TableName() string { return "mm2" } + +func (mergeModel) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{orm.Vector("Embedding", 16).Named("mm2_vec")} // overrides the tag's dim 4 +} + +func TestSearchMerge_MethodWins(t *testing.T) { + s, err := orm.SchemaOf[mergeModel]() + if err != nil { + t.Fatal(err) + } + if len(s.SearchIndexes) != 2 { + t.Fatalf("want 2 indexes (fts from tag + vec from method), got %d", len(s.SearchIndexes)) + } + if _, ok := indexByName(s, "mm2_fts"); !ok { + t.Error("tag-declared mm2_fts should survive the merge") + } + vec, ok := indexByName(s, "mm2_vec") + if !ok { + t.Fatal("missing mm2_vec") + } + if vec.Dim != 16 { + t.Errorf("dim = %d, want 16 (method should win over the tag's 4)", vec.Dim) + } +} + +func TestSearch_NoIndexes(t *testing.T) { + type plain struct { + ID int64 + Name string + } + s, err := orm.SchemaOf[plain]() + if err != nil { + t.Fatal(err) + } + if s.SearchIndexes != nil { + t.Errorf("want nil SearchIndexes, got %v", s.SearchIndexes) + } +} + +// --- validation failures --- + +func TestSearchValidation(t *testing.T) { + t.Run("missing dim", func(t *testing.T) { + type m struct { + ID int64 + Emb []float32 `vec:""` + } + mustSchemaErr(t, func() (*orm.Schema, error) { return orm.SchemaOf[m]() }, "dim") + }) + t.Run("unknown metric", func(t *testing.T) { + type m struct { + ID int64 + Emb []float32 `vec:"dim=4;metric=bogus"` + } + mustSchemaErr(t, func() (*orm.Schema, error) { return orm.SchemaOf[m]() }, "metric") + }) + t.Run("vec and fts on one field", func(t *testing.T) { + type m struct { + ID int64 + Emb []float32 `vec:"dim=4" fts:""` + } + mustSchemaErr(t, func() (*orm.Schema, error) { return orm.SchemaOf[m]() }, "both") + }) + t.Run("two unnamed fts tags collide", func(t *testing.T) { + type m struct { + ID int64 + Title string `fts:""` + Body string `fts:""` + } + mustSchemaErr(t, func() (*orm.Schema, error) { return orm.SchemaOf[m]() }, "explicit name") + }) +} + +type badFieldModel struct { + ID int64 + Body string +} + +func (badFieldModel) TableName() string { return "bad_field" } +func (badFieldModel) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{orm.FullText("Nope")} +} + +type badWeightsModel struct { + ID int64 + Title string + Body string +} + +func (badWeightsModel) TableName() string { return "bad_weights" } +func (badWeightsModel) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{orm.FullText("Title", "Body").WithWeights(1, 2, 3)} +} + +type badVecFieldsModel struct { + ID int64 + A []float32 `orm:"-"` + B []float32 `orm:"-"` +} + +func (badVecFieldsModel) TableName() string { return "bad_vec" } +func (badVecFieldsModel) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{{Kind: dialect.SearchVector, Fields: []string{"A", "B"}, Dim: 4}} +} + +type strPKFullText struct { + Code string `orm:"code,pk"` + Title string `fts:""` +} + +func (strPKFullText) TableName() string { return "str_pk_fts" } + +func TestSearch_FullTextRequiresInt64PK(t *testing.T) { + // A string-PK model with full-text must fail when its spec is resolved (at + // AutoMigrate), not silently provision a sidecar that breaks on first write. + _, err := orm.SearchSpecs[strPKFullText]() + if err == nil { + t.Fatal("want an error for string-PK full-text, got nil") + } + if !strings.Contains(err.Error(), "integer primary key") { + t.Errorf("error %q should explain the integer-PK requirement", err.Error()) + } +} + +func TestSearchMethodValidation(t *testing.T) { + mustSchemaErr(t, func() (*orm.Schema, error) { return orm.SchemaOf[badFieldModel]() }, "unknown field") + mustSchemaErr(t, func() (*orm.Schema, error) { return orm.SchemaOf[badWeightsModel]() }, "weights") + mustSchemaErr(t, func() (*orm.Schema, error) { return orm.SchemaOf[badVecFieldsModel]() }, "exactly one field") +} + +// --- helpers --- + +func mustSchemaErr(t *testing.T, fetch func() (*orm.Schema, error), want string) { + t.Helper() + _, err := fetch() + if err == nil { + t.Fatalf("want error containing %q, got nil", want) + } + if !strings.Contains(err.Error(), want) { + t.Fatalf("error %q does not contain %q", err.Error(), want) + } +} diff --git a/orm/searchsync.go b/orm/searchsync.go new file mode 100644 index 0000000..da1e10d --- /dev/null +++ b/orm/searchsync.go @@ -0,0 +1,89 @@ +package orm + +import ( + "context" + "reflect" + + liteorm "liteorm.org" + "liteorm.org/dialect" +) + +// syncSearchUpsert keeps hook-synced vector sidecars current after an ORM create +// or update, upserting the model's embedding into each hook-mode vector sidecar. +// It is a no-op when the model declares no search indexes, the backend cannot +// sync rows, the key is composite, or the embedding is empty (a partial write +// that did not set it) β€” matching the "skip empty embeddings" behavior so a +// keyless update never clobbers the sidecar with a zero vector. +func syncSearchUpsert[T any](ctx context.Context, ev *Event[T]) error { + s, err := SchemaOf[T]() + if err != nil || len(s.SearchIndexes) == 0 || s.PK == nil { + return err + } + rs, ok := ev.Sess.Dialect().(dialect.SearchRowSyncer) + if !ok { + return nil + } + mv := reflect.ValueOf(ev.Model).Elem() + for _, ix := range s.SearchIndexes { + spec, err := searchSpecOf(s, ix) + if err != nil { + return err + } + if spec.Kind != dialect.SearchVector || spec.Sync != "hooks" { + continue // trigger-synced (or non-vector) indexes maintain themselves + } + emb := mv.FieldByName(ix.Fields[0]) + if !emb.IsValid() || emb.Kind() != reflect.Slice || emb.Len() == 0 { + continue + } + key := mv.FieldByIndex(s.PK.Index).Interface() + stmts, err := rs.UpsertSearchRowSQL(spec, key, emb.Interface()) + if err != nil { + return err + } + if err := execSearchStmts(ctx, ev.Sess, stmts); err != nil { + return err + } + } + return nil +} + +// syncSearchDelete removes a row from its hook-mode vector sidecars on a hard +// delete (a soft delete leaves the sidecar row in place; the search helpers +// exclude soft-deleted rows at query time). +func syncSearchDelete[T any](ctx context.Context, sess liteorm.Session, s *Schema, v *T) error { + if len(s.SearchIndexes) == 0 || s.PK == nil { + return nil + } + rs, ok := sess.Dialect().(dialect.SearchRowSyncer) + if !ok { + return nil + } + key := reflect.ValueOf(v).Elem().FieldByIndex(s.PK.Index).Interface() + for _, ix := range s.SearchIndexes { + spec, err := searchSpecOf(s, ix) + if err != nil { + return err + } + if spec.Kind != dialect.SearchVector || spec.Sync != "hooks" { + continue + } + stmts, err := rs.DeleteSearchRowSQL(spec, key) + if err != nil { + return err + } + if err := execSearchStmts(ctx, sess, stmts); err != nil { + return err + } + } + return nil +} + +func execSearchStmts(ctx context.Context, sess liteorm.Session, stmts []dialect.SearchStmt) error { + for _, st := range stmts { + if _, err := sess.ExecContext(ctx, st.SQL, st.Args...); err != nil { + return err + } + } + return nil +} diff --git a/orm/write.go b/orm/write.go index efc0efb..f04f2cb 100644 --- a/orm/write.go +++ b/orm/write.go @@ -97,15 +97,15 @@ func (r *Repo[T]) Upsert(ctx context.Context, v *T, oc query.OnConflictSpec) err if err != nil { return err } - op := &Op[T]{Sess: r.sess, Model: v} - if err := fireBeforeCreate(ctx, op); err != nil { + ev := &Event[T]{Sess: r.sess, Model: v} + if err := fireBeforeCreate(ctx, ev); err != nil { return err } setAutoTimes(s, v, true) if err := query.NewRepo[T](r.sess).Upsert(ctx, v, oc); err != nil { return err } - return fireAfterCreate(ctx, op) + return fireAfterCreate(ctx, ev) } // Restore clears the soft-delete timestamp of v's row, bringing a soft-deleted row @@ -124,8 +124,8 @@ func (r *Repo[T]) Restore(ctx context.Context, v *T) error { if len(s.PKs) == 0 { return fmt.Errorf("orm: type %T has no primary key", *v) } - op := &Op[T]{Sess: r.sess, Model: v} - if err := fireBeforeUpdate(ctx, op); err != nil { + ev := &Event[T]{Sess: r.sess, Model: v} + if err := fireBeforeUpdate(ctx, ev); err != nil { return err } up := sqlgen.Update{ @@ -145,7 +145,7 @@ func (r *Repo[T]) Restore(ctx context.Context, v *T) error { return liteorm.ErrNoRows } reflect.ValueOf(v).Elem().FieldByIndex(s.SoftDelete.Index).SetZero() // clear in memory too - return fireAfterUpdate(ctx, op) + return fireAfterUpdate(ctx, ev) } // Updates writes only the named columns (matched by column or Go field name) of @@ -181,14 +181,14 @@ func (r *Repo[T]) CreateInBatches(ctx context.Context, vs []*T, batchSize int) e batch := vs[start:end] rows := make([][]any, len(batch)) - ops := make([]*Op[T], len(batch)) + ops := make([]*Event[T], len(batch)) for i, v := range batch { - op := &Op[T]{Sess: r.sess, Model: v} - if err := fireBeforeCreate(ctx, op); err != nil { + ev := &Event[T]{Sess: r.sess, Model: v} + if err := fireBeforeCreate(ctx, ev); err != nil { return err } setAutoTimes(s, v, true) - ops[i] = op + ops[i] = ev rows[i] = scan.Values(v, cols) } diff --git a/querier.go b/querier.go index 77ba380..40d142b 100644 --- a/querier.go +++ b/querier.go @@ -60,11 +60,6 @@ type Tx interface { Rollback(ctx context.Context) error } -// Queryer is the minimal read surface, so helpers can stay decoupled from *DB. -type Queryer interface { - QueryContext(ctx context.Context, query string, args ...any) (Rows, error) -} - // Session is a query-executing handle that knows its dialect and logger: both // *DB and *BoundTx satisfy it, so the front-ends work identically on a // connection or inside a transaction. diff --git a/query/cte.go b/query/cte.go index e6e23a5..e31c32c 100644 --- a/query/cte.go +++ b/query/cte.go @@ -1,6 +1,8 @@ package query import ( + "slices" + liteorm "liteorm.org" "liteorm.org/dialect" "liteorm.org/internal/sqlgen" @@ -99,7 +101,7 @@ func (b *SelectBuilder[T]) joinSub(kind, alias string, sub Subquery, on string, lat = "LATERAL " } clause := kind + " " + lat + "(" + frag + ") AS " + string(d.QuoteIdent(nil, alias)) + " ON " + on - allArgs := append(append([]any{}, subArgs...), args...) + allArgs := slices.Concat(subArgs, args) b.sel.Joins = append(b.sel.Joins, sqlgen.Expr{SQL: clause, Args: allArgs}) return b } diff --git a/query/jsonarray.go b/query/jsonarray.go index fa10ac9..f2558cb 100644 --- a/query/jsonarray.go +++ b/query/jsonarray.go @@ -2,6 +2,7 @@ package query import ( "encoding/json" + "slices" "strings" "liteorm.org/dialect" @@ -28,7 +29,7 @@ type JSONPath struct { // Key drills one level into an object. Chain Key calls to walk nested objects: // JSON("data").Key("address").Key("city").Eq("Paris"). func (p JSONPath) Key(k string) JSONPath { - return JSONPath{col: p.col, keys: append(append([]string{}, p.keys...), k)} + return JSONPath{col: p.col, keys: append(slices.Clone(p.keys), k)} } // extractText renders the column walked to the path as TEXT: the inner keys use diff --git a/query/query.go b/query/query.go index 08dbd37..e926373 100644 --- a/query/query.go +++ b/query/query.go @@ -25,19 +25,11 @@ import ( "liteorm.org/internal/sqlgen" ) -type tableNamer interface{ TableName() string } - -// tableName derives T's table name: a TableName() method if present, else the -// snake_case of the type name (no pluralization β€” explicit over implicit). +// tableName derives T's table name via the shared resolver: a TableName() method +// if present, else the snake_case of the type name β€” pluralized when +// orm.UsePluralTableNames is enabled, so the query and orm front-ends agree. func tableName[T any]() string { - var v T - if tn, ok := any(v).(tableNamer); ok { - return tn.TableName() - } - if tn, ok := any(&v).(tableNamer); ok { - return tn.TableName() - } - return scan.Snake(reflect.TypeFor[T]().Name()) + return scan.TableNameOf(reflect.TypeFor[T]()) } // colListKey caches a table-qualified column list per (type, table) β€” table diff --git a/query/query_test.go b/query/query_test.go index 24046c0..c69437c 100644 --- a/query/query_test.go +++ b/query/query_test.go @@ -123,7 +123,7 @@ func TestJoinsUnionSubquery(t *testing.T) { if err != nil { t.Fatal(err) } - if !contains(q, `FROM "users" LEFT JOIN "orders" ON orders.user_id = users.id WHERE "age" > ?`) { + if !strings.Contains(q, `FROM "users" LEFT JOIN "orders" ON orders.user_id = users.id WHERE "age" > ?`) { t.Errorf("join sql: %s", q) } }) @@ -136,7 +136,7 @@ func TestJoinsUnionSubquery(t *testing.T) { t.Fatal(err) } want := `WHERE "age" < $1 UNION SELECT "users"."id", "users"."name", "users"."age", "users"."email" FROM "users" WHERE "age" > $2 ORDER BY name` - if !contains(q, want) { + if !strings.Contains(q, want) { t.Errorf("union sql: %s", q) } if len(args) != 2 || args[0] != 18 || args[1] != 65 { @@ -151,7 +151,7 @@ func TestJoinsUnionSubquery(t *testing.T) { if err != nil { t.Fatal(err) } - if !contains(q, "UNION ALL SELECT") { + if !strings.Contains(q, "UNION ALL SELECT") { t.Errorf("union all sql: %s", q) } }) @@ -164,7 +164,7 @@ func TestJoinsUnionSubquery(t *testing.T) { t.Fatal(err) } want := `WHERE "id" IN (SELECT id FROM "docs" WHERE "data" = $1)` - if !contains(q, want) { + if !strings.Contains(q, want) { t.Errorf("in-subquery sql: %s", q) } if len(args) != 1 || args[0] != "x" { @@ -180,7 +180,7 @@ func TestJoinsUnionSubquery(t *testing.T) { t.Fatal(err) } // the outer name=$1 must precede the (argument-less) EXISTS body - if !contains(q, `WHERE "name" = $1 AND EXISTS (SELECT 1 FROM "docs" WHERE docs.id = users.id)`) { + if !strings.Contains(q, `WHERE "name" = $1 AND EXISTS (SELECT 1 FROM "docs" WHERE docs.id = users.id)`) { t.Errorf("exists sql: %s", q) } if len(args) != 1 || args[0] != "ada" { @@ -205,8 +205,6 @@ func TestJoinsUnionSubquery(t *testing.T) { }) } -func contains(s, sub string) bool { return strings.Contains(s, sub) } - func TestEmptyPredicateEdgeCases(t *testing.T) { sess := mockSession{d: sqlgen.SQLite} diff --git a/query/window.go b/query/window.go index a848c02..16dee5c 100644 --- a/query/window.go +++ b/query/window.go @@ -1,6 +1,7 @@ package query import ( + "slices" "strconv" "strings" @@ -19,13 +20,13 @@ func Over() Window { return Window{} } // PartitionBy adds PARTITION BY columns to the window. func (w Window) PartitionBy(cols ...Field) Window { - w.partition = append(append([]Field{}, w.partition...), cols...) + w.partition = slices.Concat(w.partition, cols) return w } // OrderBy adds ORDER BY terms to the window (use Asc/Desc). func (w Window) OrderBy(terms ...OrderTerm) Window { - w.order = append(append([]OrderTerm{}, w.order...), terms...) + w.order = slices.Concat(w.order, terms) return w } @@ -72,7 +73,7 @@ type WindowFunc struct { // Over attaches the OVER clause and the result alias, yielding a projection field: // OVER () AS . func (wf WindowFunc) Over(w Window, alias string) Field { - cols := append(append([]string{}, wf.cols...), w.cols()...) + cols := slices.Concat(wf.cols, w.cols()) f := plainField(cols, func(d dialect.Dialect) string { return wf.call(d) + " OVER (" + w.render(d) + ") AS " + quoteCol(d, alias) }) diff --git a/skills/README.md b/skills/README.md index 91c3b85..1c33bfa 100644 --- a/skills/README.md +++ b/skills/README.md @@ -15,7 +15,8 @@ Each `SKILL.md` begins with YAML frontmatter (`name` + a one-sentence `descripti | [orm-models](orm-models/SKILL.md) | Declarative models: structs + tags, AutoMigrate, the `orm.Repo`, associations (Load/Attach), hooks, soft delete. | | [migrations](migrations/SKILL.md) | Evolving a schema: AutoMigrate (additive), GenerateMigration (reviewable), the migrate runner, WritePair. | | [codegen](codegen/SKILL.md) | Generating typed columns/models/queries: `liteorm gen`, the sqlc plugin, the gorm porter. | -| [sqlite-search](sqlite-search/SKILL.md) | SQLite vector (sqlite-vec), full-text (FTS5), hybrid RRF search, and encryption at rest. | +| [sqlite-search](sqlite-search/SKILL.md) | SQLite vector (sqlite-vec), full-text (FTS5), and hybrid RRF search. | +| [encryption](encryption/SKILL.md) | Opening a SQLite database with at-rest (transparent page-level) encryption: keys, reopening, constraints. | | [postgres-advanced](postgres-advanced/SKILL.md) | Postgres LISTEN/NOTIFY, and JSONB / array typed operators. | | [porting-from-gorm](porting-from-gorm/SKILL.md) | Migrating a gorm codebase: native gorm-tag reading and rewriting to native `orm` tags; what differs. | | [logging](logging/SKILL.md) | Seeing/tracing executed SQL while developing: debug logging via slog or the colored handler, traced to your code. | diff --git a/skills/encryption/SKILL.md b/skills/encryption/SKILL.md new file mode 100644 index 0000000..da8c930 --- /dev/null +++ b/skills/encryption/SKILL.md @@ -0,0 +1,44 @@ +--- +name: encryption +description: Use when opening a SQLite database with at-rest (transparent page-level) encryption β€” writing/reading an encrypted file, key handling, reopening, and constraints. +--- + +# At-rest encryption (SQLite) + +liteorm opens an encrypted SQLite database through gosqlite's transparent, page-level cipher. Encryption is an **open-time concern**, orthogonal to queries: once the database is open, `query`, `orm`, migrations, and search all work exactly as on an unencrypted database. + +## Open with a key + +```go +import "liteorm.org/dialect/sqlite" + +db, err := sqlite.OpenEncrypted(path, key) // key is 32 bytes; default Adiantum cipher +// use db as a normal *liteorm.DB: orm.AutoMigrate, orm.NewRepo, query.Select, … +``` + +- `key` is a 32-byte secret β€” source it from a KMS/secret store, never a literal. Losing it loses the data (no recovery). +- The on-disk file is ciphertext. Reopen with the SAME key to read; a wrong key fails (open or first query), it does not return garbage. + +## Full control (cipher / pragmas / pool) via OpenConfig + +```go +import gosqlite "gosqlite.org" + +db, err := sqlite.OpenConfig(gosqlite.Config{ + Path: path, + Pragmas: gosqlite.RecommendedPragmas(), + Encryption: &gosqlite.Encryption{Key: key, Cipher: gosqlite.Adiantum}, +}) +``` + +## Constraints + +- Needs an on-disk path; `:memory:` is rejected (nothing to encrypt at rest). +- Mutually exclusive with a custom VFS (the cipher is itself a VFS layer). +- Per-database-file, set at open β€” there is no per-table encryption. +- Rotating a key = re-encrypt: open with the old key, copy into a new database opened with the new key. + +## Deeper + +- Example: `examples/encryption` (write encrypted, verify ciphertext on disk, reopen, reject the wrong key). +- API: https://pkg.go.dev/liteorm.org/dialect/sqlite (`OpenEncrypted`, `OpenConfig`). diff --git a/skills/orm-models/SKILL.md b/skills/orm-models/SKILL.md index 0cbfd17..00c155a 100644 --- a/skills/orm-models/SKILL.md +++ b/skills/orm-models/SKILL.md @@ -132,17 +132,17 @@ For has-many/has-one, `Append` sets each target's FK to the owner; `Delete`/`Cle ## Hooks -Implement the hook method on `*T`; a wrong signature is a compile error, not a dead hook. The `*orm.Op[T]` carries `Sess` (the executing session) and `Model` (`*T`). Returning an error aborts the operation. +Implement the hook method on `*T`; a wrong signature is a compile error, not a dead hook. The `*orm.Event[T]` carries `Sess` (the executing session) and `Model` (`*T`). Returning an error aborts the operation. ```go -func (p *Post) BeforeCreate(ctx context.Context, op *orm.Op[Post]) error { - if op.Model.Slug == "" { op.Model.Slug = slugify(op.Model.Title) } +func (p *Post) BeforeCreate(ctx context.Context, ev *orm.Event[Post]) error { + if ev.Model.Slug == "" { ev.Model.Slug = slugify(ev.Model.Title) } return nil } var _ orm.BeforeCreateHook[Post] = (*Post)(nil) // optional compile-time assert ``` -Available: `BeforeCreate` / `AfterCreate`, `BeforeUpdate` / `AfterUpdate`, `BeforeDelete` / `AfterDelete` β€” each `(ctx, *orm.Op[T]) error`. +Available: `BeforeCreate` / `AfterCreate`, `BeforeUpdate` / `AfterUpdate`, `BeforeDelete` / `AfterDelete` β€” each `(ctx, *orm.Event[T]) error`. ## Soft delete diff --git a/skills/pitfalls/SKILL.md b/skills/pitfalls/SKILL.md index a96776a..9ed2cfd 100644 --- a/skills/pitfalls/SKILL.md +++ b/skills/pitfalls/SKILL.md @@ -46,7 +46,7 @@ The gotchas that trip people up, with the fix. ## Hooks -- **A mis-signed hook is a compile error, not a silent no-op** β€” hooks are typed on T. The signature is `func (t *T) BeforeCreate(ctx context.Context, op *orm.Op[T]) error`. Returning an error aborts the operation. +- **A mis-signed hook is a compile error, not a silent no-ev** β€” hooks are typed on T. The signature is `func (t *T) BeforeCreate(ctx context.Context, ev *orm.Event[T]) error`. Returning an error aborts the operation. ## Deeper diff --git a/skills/porting-from-gorm/SKILL.md b/skills/porting-from-gorm/SKILL.md index 378628c..7314109 100644 --- a/skills/porting-from-gorm/SKILL.md +++ b/skills/porting-from-gorm/SKILL.md @@ -58,7 +58,7 @@ Tag translation (a sample): | `Preload("A").Preload("B")` | Separate `orm.Load` calls; nested = chain one level at a time. | | Soft delete hidden automatically + `Unscoped()` | Default reads also hide deleted rows, but the opt-outs are explicit: `IncludeDeleted()` / `OnlyDeleted()` / `ForceDelete`. | | `gorm.DeletedAt` field type | `sql.NullTime` with the `soft_delete` tag. | -| Pluralized table names by default | None β€” `TableName()` or `snake_case(TypeName)` with no pluralization. Keep your `TableName()`. | +| Pluralized table names by default | Opt-in: `orm.UsePluralTableNames(true)` restores gorm-style plurals globally (`orm.RegisterPlural` for irregulars). Otherwise `snake_case(TypeName)` singular, or pin with `TableName()`. | | Hooks as methods (silent if mis-signed) | Hooks are typed on T β€” a wrong signature is a compile error (see orm-models skill). | ## Pitfalls diff --git a/skills/sqlite-search/SKILL.md b/skills/sqlite-search/SKILL.md index 45e3880..77dde59 100644 --- a/skills/sqlite-search/SKILL.md +++ b/skills/sqlite-search/SKILL.md @@ -1,74 +1,86 @@ --- name: sqlite-search -description: Use when adding vector (sqlite-vec), full-text (FTS5), or hybrid RRF search to liteorm's SQLite backend, or opening an encrypted-at-rest SQLite database. +description: Use when adding vector (sqlite-vec), full-text (FTS5), or hybrid RRF search to liteorm's SQLite backend. --- # SQLite search -Import `liteorm.org/dialect/sqlite/search`. These features are SQLite-only and capability-gated: every constructor takes a session opened by `liteorm.org/dialect/sqlite` and returns `search.ErrUnsupportedBackend` for any other dialect. +Import `liteorm.org/dialect/sqlite/search`. These features are SQLite-only and capability-gated: the helpers take a session opened by `liteorm.org/dialect/sqlite` and return `search.ErrUnsupportedBackend` for any other dialect. Every index is a sidecar table keyed by the model's primary key. -The recipe: your table owns the rows, a sidecar index owns the embeddings/terms, and the model's `int64` primary key ties them together. A search returns ranked keys; `Load` fetches the model rows by key, preserving rank order. +## Declarative (recommended) -## Vector (sqlite-vec) +Declare the indexes on the model; `orm.AutoMigrate` creates the sidecars and the triggers/hooks that keep them in sync, so plain `Repo.Create`/`Update`/`Delete` need no index calls. Search with the typed helpers, which return models in ranked order. ```go -v, err := search.NewVector(ctx, db, "doc_vecs", 5 /* dim */, search.Cosine) -_ = v.Add(ctx, doc.ID /* int64 key */, embedding /* []float32 */) // re-Add replaces - -keys, _ := v.Search(ctx, queryEmb, 3) // []int64, nearest first -scored, _ := v.SearchScored(ctx, queryEmb, 3) // []Scored{Key, Score}; Score = raw distance, smaller is nearer +type Article struct { + ID int64 + Title string + Body string + Embedding []float32 `orm:"-"` // sidecar-only (not a base column) +} + +func (Article) SearchIndexes() []orm.SearchIndex { + return []orm.SearchIndex{ + orm.FullText("Title", "Body"), + orm.Vector("Embedding", 384).WithMetric(orm.Cosine), + } +} + +orm.AutoMigrate[Article](ctx, db) // creates articles + articles_fts + articles_vec (+sync) +repo := orm.NewRepo[Article](db) +repo.Create(ctx, &Article{Title: "…", Body: "…", Embedding: vec}) // sidecars sync automatically + +s := search.For[Article](db) // typed searcher +near, _ := s.Vector(ctx, queryVec, 5) // vector +hits, _ := s.FullText(ctx, search.Term("rocket"), 5) // full-text +fused, _ := s.Hybrid(ctx, queryVec, search.Term("rocket"), 5) // hybrid (RRF) +for _, h := range near { use(h.Model, h.Score) } ``` -Metrics: `search.L2` (default), `search.Cosine` (normalized text embeddings), `search.L1`, `search.Hamming`. - -## Full-text (FTS5) +- `h.Score`: vector distance for `.Vector` (smaller nearer), BM25 rank for `.FullText`, RRF score for `.Hybrid` (larger better). +- Soft-deleted rows are excluded automatically (loaded through the orm Repo). +- More than one index of a kind β†’ disambiguate with `search.For[Article](db).Field("FieldName")`. -```go -f, err := search.NewFullText(ctx, db, "doc_fts") // unicode61 tokenizer, BM25 ranking -_ = f.Add(ctx, doc.ID, doc.Title+" "+doc.Body) // re-Add replaces +### Declaring: method or tags -keys, _ := f.Search(ctx, search.Term("rocket"), 5) // []int64, best BM25 first -``` +- `SearchIndexes() []orm.SearchIndex` β€” full power: multi-column full-text, `WithWeights`, tokenizer/prefix/detail. +- Tags for the common case: `vec:"dim=384;metric=cosine"` on the embedding, `fts:"tokenize=porter unicode61"` on a text field (`fts5:` is an alias). -Query builders (re-exported, no need to import gosqlite/fts): `search.Term(s)`, `search.Phrase(tokens...)`, `search.Prefix(s)`, `search.And(qs...)`, `search.Or(qs...)`, `search.Not(pos, negs...)`, `search.Near(dist, terms...)`, `search.Column(name, q)`, `search.Raw(s)`. +### Sync mode (`.WithSync(...)`) -```go -f.Search(ctx, search.And(search.Term("software"), search.Term("flight")), 5) -``` +- **Triggers** (default for full-text, and for a vector whose embedding is a stored column): SQL triggers keep the sidecar current on *every* write β€” bulk and raw `query` writes included. +- **Hooks** (default for a sidecar-only `orm:"-"` vector embedding β€” no column duplication): synced from the orm write path only; writes bypassing the orm are not indexed. -## Hybrid (reciprocal rank fusion) +## Low-level building blocks -Runs a vector KNN and a full-text query, then fuses the two rankings with RRF β€” a key that ranks well in both surfaces highest, without tuning a score-scale blend. +Drive a sidecar by hand when there is no model, or you provision/backfill on your own schedule. ```go -fused, _ := search.Hybrid(ctx, v, f, queryEmb, search.Term("software"), 4) -// []Scored{Key, Score}; Score = RRF score, LARGER is better; ordered desc, capped at k -``` - -Tuning options: `search.WithK(60)` (RRF damping), `search.WithWeights(wVec, wText)`. +v, _ := search.NewVector(ctx, db, "doc_vecs", 5 /* dim */, search.Cosine) // or OpenVector to attach +_ = v.Add(ctx, id /* int64 */, emb /* []float32 */) // re-Add replaces +keys, _ := v.Search(ctx, queryEmb, 3) // []int64, nearest first +scored, _ := v.SearchScored(ctx, queryEmb, 3) // []Scored{Key, Score=distance} -## Load model rows by key (preserves order) +f, _ := search.NewFullText(ctx, db, "doc_fts") // or OpenFullText(name, cols...) +_ = f.Add(ctx, id, title+" "+body) +keys, _ = f.Search(ctx, search.And(search.Term("software"), search.Term("flight")), 5) -```go -docs, _ := search.Load[Doc](ctx, db, keys) // from a []int64 -docs, _ := search.LoadScored[Doc](ctx, db, scored) // from []Scored (Vector/Hybrid) +fused, _ := search.Hybrid(ctx, v, f, queryEmb, search.Term("software"), 4) // []Scored{Key, Score=RRF} +docs, _ := search.Fetch[Doc](ctx, db, keys) // or FetchScored from []Scored; preserves order ``` -T must be a model the query front-end can address: a `TableName()` and an `int64` primary key. Missing keys are skipped. `Load` issues one `Get` per key β€” the right shape for small top-k slices. +Query builders (re-exported, no gosqlite import): `Term`, `Phrase`, `Prefix`, `And`, `Or`, `Not`, `Near`, `Column`, `Raw`. Metrics: `orm.L2` (default), `orm.Cosine`, `orm.L1`, `orm.Hamming`. RRF tuning: `search.WithK(60)`, `search.WithWeights(wVec, wText)`. -## Encryption at rest +## Custom SQL functions / REGEXP -```go -db, err := sqlite.OpenEncrypted(path, key /* []byte */) -// write normally; the on-disk file is ciphertext. Reopen with the same key to read. -``` +gosqlite registers scalar functions globally, so they work through liteorm with no glue: blank-import `gosqlite.org/ext/regexp/auto`, then `query.Select[T](db).Where("col REGEXP ?", pattern)`. ## Pitfalls -- Indexes are keyed by the model's `int64` PK β€” that key is the contract between table and sidecar; keep them in sync on insert/delete. -- A constructor on a non-SQLite (or non-`dialect/sqlite`-opened) session returns `ErrUnsupportedBackend`. -- Hybrid `Score` is larger-is-better (RRF); `SearchScored` distance is smaller-is-nearer. Don't compare across them. -- Re-`Add`ing an existing key replaces it; there is no separate update call. +- Declarative trigger-mode keeps the index current on all writes; **hook-mode** indexes only sync through the orm Repo β€” a raw `query` insert won't be indexed. +- `Score` is larger-is-better for `.Hybrid` (RRF) but smaller-is-nearer for `.Vector`/`SearchScored` (distance). Don't compare across them. +- Full-text requires an `int64` primary key (FTS5 is keyed by the integer rowid; a string-PK model errors at migrate). Vector search supports both `int64` and string PKs. +- A helper on a non-`dialect/sqlite` session returns `ErrUnsupportedBackend`. ## Deeper diff --git a/skills/using-liteorm/SKILL.md b/skills/using-liteorm/SKILL.md index 57776f5..8190cdc 100644 --- a/skills/using-liteorm/SKILL.md +++ b/skills/using-liteorm/SKILL.md @@ -67,7 +67,7 @@ _ = repo.Create(ctx, &u) got, _ := repo.Get(ctx, u.ID) ``` -A model is just a struct with a `TableName() string` method. No method β‡’ the table name is `snake_case(TypeName)` with no pluralization. +A model is just a struct with a `TableName() string` method. No method β‡’ the table name is `snake_case(TypeName)`, singular by default (opt into plurals globally with `orm.UsePluralTableNames(true)`). ## Session & transactions