diff --git a/docs/plans/2026-10-04-plan-builder/README.md b/docs/plans/2026-10-04-plan-builder/README.md index b67641fe4..2d5c42fd5 100644 --- a/docs/plans/2026-10-04-plan-builder/README.md +++ b/docs/plans/2026-10-04-plan-builder/README.md @@ -6,6 +6,8 @@ The module path is `example.com/app`. ## What each file shows +Each directory has a `README.md` that walks through its code. + | File | What it shows | |---|---| | [`main.go`](main.go) | A client, one cipher for each tenant, and a batch of two types in one request | @@ -18,7 +20,7 @@ The module path is `example.com/app`. | [`accounts/account.go`](accounts/account.go) | An embedded `gorm.Model`, another library's tags, an unexported field and `-redact` | | [`contacts/contacts.go`](contacts/contacts.go) | [`crm.Contact`](crm/contact.go), a type in another package, in separate columns, with a model | | [`documents/documents.go`](documents/documents.go) | An `opaque` struct, sealed as one value | -| [`proto/`](proto/) | A protobuf message whose fields carry data categories, which generates [`internal/pb/`](internal/pb/) | +| [`proto/`](proto/) | A protobuf message whose fields have data categories, which generates [`internal/pb/`](internal/pb/) | | [`rules/rules.go`](rules/rules.go) | Rules that decide what to encrypt from each field's data categories | | [`cmd/genencrypt/main.go`](cmd/genencrypt/main.go) | The generate program that runs the rules | | [`individuals/`](individuals/) | The file the rules give for the protobuf message, and a store that uses it | diff --git a/docs/plans/2026-10-04-plan-builder/accounts/README.md b/docs/plans/2026-10-04-plan-builder/accounts/README.md new file mode 100644 index 000000000..32daa0802 --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/accounts/README.md @@ -0,0 +1,143 @@ +# Preserve GORM fields and redact an account + +This example shows an embedded GORM model, copied tags, and generated print methods. +It also distinguishes an unexported field from an omitted field. +The Golang SDK for CipherStash Stack is planned and does not exist yet, so this code does not build in this repository. +The [Go example index](../README.md) lists every example and what was checked. + +`account_stash.go` is written by hand. +It shows what `stashgen` is planned to write. + +## What you will learn + +- Tag an embedded GORM model and distinguish two fields that are not stored. +- Inspect the encrypted type, copied tags, warnings, and declaration. +- Add redacted print methods to the tagged struct. +- Give the encrypted type to GORM. + +## Tag the account fields + +- **Purpose:** Understand how one tag covers an embedded struct from another package. +- **Related:** [The plan's embedded struct design](../../2026-10-04-plan-builder.md#embedded-structs-unexported-fields-and-other-tags), [the plan's printing design](../../2026-10-04-plan-builder.md#printing) + +The `stash:",passthrough"` tag stores every `gorm.Model` field as a passthrough field. +`Email` becomes one Encrypt Query Language (EQL) column with the `TextEq` type. +`stashgen` copies the `gorm` and `json` tags of `Email` to the same field of `EncryptedAccount`. + +[`account.go`, lines 12 to 21](account.go#L12-L21) + +```go +//go:generate go tool stashgen -type Account -redact + +type Account struct { + _ struct{} `stash:"context=accounts"` + + // One tag decides for every field of gorm.Model, which cannot carry tags. + gorm.Model `stash:",passthrough"` + + // stashgen copies the gorm and json tags onto EncryptedAccount. + Email string `stash:"email,encrypt_into=TextEq" gorm:"uniqueIndex" json:"email"` +``` + +The unexported `cache` field has no tag, so the generator ignores it with three notices: + +- `stashgen` prints a warning. +- The generated file names `cache` in a comment. +- The program prints a warning to stderr once for each type, on first use. + +The `token` field uses `stash:"-"`, so it is an omitted field and produces no warning. + +[`account.go`, lines 23 to 29](account.go#L23-L29) + +```go + // An unexported field with no stash tag is ignored, and stashgen and the + // running program both warn about it. + cache string + + // The tag acknowledges the skip, so nothing warns. + token string `stash:"-"` +} +``` + +## Inspect the encrypted account + +- **Purpose:** Follow copied tags and fields that are not stored into the generated file. +- **Related:** [The plan's generated file design](../../2026-10-04-plan-builder.md#what-stashgen-writes), [the plan's embedded struct design](../../2026-10-04-plan-builder.md#embedded-structs-unexported-fields-and-other-tags) + +The generated comment names `cache`, and `EncryptedAccount` embeds the same `gorm.Model`. +The encrypted `Email` field retains its GORM and JSON tags. + +[`account_stash.go`, lines 19 to 25](account_stash.go#L19-L25) + +```go +// Not encrypted and not stored: the unexported field "cache". Tag it +// `stash:"-"` to confirm that. + +type EncryptedAccount struct { + gorm.Model + Email eql.TextEq `gorm:"uniqueIndex" json:"email"` +} +``` + +The codec's `Unexported` list enables the planned runtime warning for `cache`. +The declaration omits `token` and does not include `cache`. + +[`account_stash.go`, lines 55 to 66](account_stash.go#L55-L66) + +```go +var declaration = gensupport.Declare("accounts"). + Passthrough("id"). + Passthrough("created_at"). + Passthrough("updated_at"). + Passthrough("deleted_at"). + EncryptInto("email", "TextEq"). + Omit("token") + +var codec = gensupport.New(gensupport.Generated[Account, EncryptedAccount]{ + TypeName: "Account", + Declaration: declaration, + Unexported: []string{"cache"}, +``` + +## Redact the tagged struct + +- **Purpose:** See what the `-redact` flag adds to `Account`. +- **Related:** [The plan's printing design](../../2026-10-04-plan-builder.md#printing) + +`-redact` asks `stashgen` to add `String` and `LogValue` methods to `Account`. +These methods show `gorm.Model` and hide `Email`. + +[`account_stash.go`, lines 35 to 42](account_stash.go#L35-L42) + +```go +// Written because of -redact: Account no longer prints its sealed fields. +func (a Account) String() string { + return gensupport.Redacted("Account", map[string]any{"Model": a.Model}, "Email") +} + +func (a Account) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"Model": a.Model}, "Email") +} +``` + +## Give encrypted accounts to GORM + +- **Purpose:** Follow a slice through encryption and into GORM. +- **Related:** [The plan's database design](../../2026-10-04-plan-builder.md#databases) + +`EncryptedAccount.TableName` maps the encrypted type to the `accounts` table. +`Create` encrypts the whole slice before GORM receives it. + +[`account.go`, lines 31 to 39](account.go#L31-L39) + +```go +func (EncryptedAccount) TableName() string { return "accounts" } + +func Create(ctx context.Context, db *gorm.DB, cipher *encrypt.Cipher, accounts []Account) error { + encrypted, err := Encrypt(ctx, cipher, accounts) + if err != nil { + return err + } + return db.WithContext(ctx).Create(&encrypted).Error +} +``` diff --git a/docs/plans/2026-10-04-plan-builder/cmd/genencrypt/README.md b/docs/plans/2026-10-04-plan-builder/cmd/genencrypt/README.md new file mode 100644 index 000000000..c19a84236 --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/cmd/genencrypt/README.md @@ -0,0 +1,46 @@ +# Generate encryption code from protobuf rules + +This example contains the command that is planned to turn protobuf rules into a generated file. +It connects the protobuf source, `rules.Individuals`, and the requested output path. +The Golang SDK for CipherStash Stack is planned and does not exist yet, so this code does not build in this repository. +The [Go example index](../../README.md) lists every example and what was checked. + +## What you will learn + +- Run `stashgen.Generate` with a protobuf source and policy. +- Review rule changes in the generated file. + +## Connect the source, rules, and output + +- **Purpose:** Follow the three inputs to the planned library generator. +- **Related:** [The plan's policy declarations](../../../2026-10-04-plan-builder.md#declarations-from-a-policy), [the rules walkthrough](../../rules/README.md), [the individuals walkthrough](../../individuals/README.md) + +`protosource.New()` supplies protobuf descriptors and field options. +`rules.Individuals` supplies the field decisions. +`stashgen.Output` selects the generated file in the `individuals` package. +The output goes in your own `individuals` package, not `internal/pb`, which contains the generated type. +This separation stops the generate program from importing a file that it writes. + +[`main.go`, lines 12 to 18](main.go#L12-L18) + +```go +func main() { + err := stashgen.Generate(protosource.New(), rules.Individuals, + stashgen.Output("../individuals/individual_stash.go")) + if err != nil { + log.Fatal(err) + } +} +``` + +The `stashgen`, `protosource`, and policy packages do not exist yet. + +## Review rule changes in the generated file + +- **Purpose:** See how reviewers inspect policy changes without running rules in the application. +- **Related:** [The generated individuals walkthrough](../../individuals/README.md), [the rules walkthrough](../../rules/README.md) + +The [`rules` package](../../rules/README.md) runs this command through `go generate`. +The application calls the resulting `individuals.Encrypt` and `individuals.Decrypt` functions. +A rule change changes `individual_stash.go`, which gives a reviewer a concrete generated diff. +The application never runs the rules. diff --git a/docs/plans/2026-10-04-plan-builder/contacts/README.md b/docs/plans/2026-10-04-plan-builder/contacts/README.md new file mode 100644 index 000000000..2acccdbdb --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/contacts/README.md @@ -0,0 +1,237 @@ +# Encrypt a contact type from another package into separate columns + +This example declares encryption for `crm.Contact`, a type from another package. +It stores ciphertexts and search terms in separate columns, with a model for GORM. +An output is a ciphertext or one search term. +The Golang SDK for CipherStash Stack is planned and does not exist yet, so this code does not build in this repository. +The [Go example index](../README.md) lists every example and what was checked. + +`contactstash_stash.go` is written by hand. +It shows what `stashgen` is planned to write. + +## What you will learn + +- Declare tags for a type in another package. +- Read the nested encrypted field types and available indexes. +- Map separate outputs into a model with a shape check. +- Insert contacts through `database/sql` or GORM. +- Search a stored search term by phone number. + +## Declare tags for a type in another package + +- **Purpose:** Connect a tag declaration in your own package to `crm.Contact`. +- **Related:** [The `crm.Contact` type in another package](../crm/README.md), [the plan's design for types in another package](../../2026-10-04-plan-builder.md#types-in-another-package) + +The `-for crm.Contact` flag makes `contactStash` declare tags for a type in another package. +The generator is planned to match fields by name and type. + +[`contacts.go`, lines 14 to 25](contacts.go#L14-L25) + +```go +//go:generate go tool stashgen -type contactStash -for crm.Contact -model Rows=ContactRow + +// contactStash declares the tags for crm.Contact, which cannot carry them. +// stashgen matches each field to the crm.Contact field with the same name and +// type, and refuses a crm.Contact field this struct does not name. +type contactStash struct { + _ struct{} `stash:"context=contacts"` + ID int64 `stash:"id,passthrough"` + Email string `stash:"email,encrypt,index=equality;match"` + PhoneNumber string `stash:"phone_number,encrypt,index=equality"` + Internal string `stash:"-"` +} +``` + +`Email` declares the `equality` and `match` indexes. +`PhoneNumber` declares only the `equality` index. + +The engine produces only the `TextEq` Encrypt Query Language (EQL) type today. +No EQL type is built from match search terms yet. +The plan therefore uses separate columns for `Email`, which needs a match search. +One EQL column for each field is the layout to use when a suitable EQL type exists. + +The generated file uses a shape check for `crm.Contact`. +A change to `crm.Contact` stops the contacts package from building. + +[`contactstash_stash.go`, lines 45 to 53](contactstash_stash.go#L45-L53) + +```go +// Stops compiling when crm.Contact gains, loses, reorders or retypes a field. +var _ = contactShape(crm.Contact{}) + +type contactShape struct { + ID int64 + Email string + PhoneNumber string + Internal string +} +``` + +## Read the separate encrypted outputs + +- **Purpose:** See that each encrypted field contains a ciphertext and its declared search terms. +- **Related:** [The plan's column layouts](../../2026-10-04-plan-builder.md#columns), [the plan's generated file design](../../2026-10-04-plan-builder.md#what-stashgen-writes) + +`EncryptedContact` nests one encrypted type for each encrypted field. + +[`contactstash_stash.go`, lines 20 to 30](contactstash_stash.go#L20-L30) + +```go +type EncryptedContact struct { + ID int64 + Email EncryptedContactEmail + PhoneNumber EncryptedContactPhoneNumber +} + +type EncryptedContactEmail struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm +} +``` + +[`contactstash_stash.go`, lines 32 to 35](contactstash_stash.go#L32-L35) + +```go +type EncryptedContactPhoneNumber struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm +} +``` + +The declaration uses `EncryptIndex` to request those separate outputs. + +[`contactstash_stash.go`, lines 55 to 59](contactstash_stash.go#L55-L59) + +```go +var declaration = gensupport.Declare("contacts"). + Passthrough("id"). + EncryptIndex("email", encrypt.Equality, encrypt.Match()). + EncryptIndex("phone_number", encrypt.Equality). + Omit("internal") +``` + +The generated file can add print methods to `EncryptedContact`. +It cannot add methods to `crm.Contact` because Go requires methods to belong to the type's package. + +[`contactstash_stash.go`, lines 17 to 18](contactstash_stash.go#L17-L18) + +```go +// crm.Contact prints its sealed fields in the clear, and stashgen cannot add +// print methods to a type from another package. +``` + +## Map outputs into a model + +- **Purpose:** Flatten nested encrypted outputs into one field for each database column. +- **Related:** [The plan's model design](../../2026-10-04-plan-builder.md#models) + +`-model Rows=ContactRow` asks for `EncryptRows` and `DecryptRows`. +`ContactRow` is a model with one tagged field for each column. +`EncryptedContact` nests structs, but GORM maps one struct field to one column. +This separate-column layout therefore needs `ContactRow` for GORM. + +[`contacts.go`, lines 27 to 36](contacts.go#L27-L36) + +```go +// ContactRow is a model: one field for each column. Each tag names the output +// the field holds. +type ContactRow struct { + ID int64 `stash:"id"` + Email encrypt.Ciphertext `stash:"email"` + EmailEq encrypt.EqualityTerm `stash:"email,equality"` + EmailMatch encrypt.MatchTerm `stash:"email,match"` + PhoneNumber encrypt.Ciphertext `stash:"phone_number"` + PhoneNumberEq encrypt.EqualityTerm `stash:"phone_number,equality"` +} +``` + +`rowsShape` is the model's shape check. +The conversions compile only while the fields, types, and order match. + +[`contactstash_stash.go`, lines 155 to 164](contactstash_stash.go#L155-L164) + +```go +// The -model flag: ContactRow converts to and from its shape only while the +// two have the same fields, with the same types, in the same order. +type rowsShape struct { + ID int64 + Email encrypt.Ciphertext + EmailEq encrypt.EqualityTerm + EmailMatch encrypt.MatchTerm + PhoneNumber encrypt.Ciphertext + PhoneNumberEq encrypt.EqualityTerm +} +``` + +[`contactstash_stash.go`, lines 187 to 193](contactstash_stash.go#L187-L193) + +```go +func EncryptRows(ctx context.Context, cipher *encrypt.Cipher, contacts []crm.Contact) ([]ContactRow, error) { + return rowsCodec.Encrypt(ctx, cipher, contacts) +} + +func DecryptRows(ctx context.Context, d encrypt.Decrypter, rows []ContactRow) ([]crm.Contact, error) { + return rowsCodec.Decrypt(ctx, d, rows) +} +``` + +## Insert contacts through two database APIs + +- **Purpose:** Compare direct column writes with a model-based GORM write. +- **Related:** [The plan's database design](../../2026-10-04-plan-builder.md#databases) + +`Create` passes each nested output to `database/sql`. + +[`contacts.go`, lines 41 to 51](contacts.go#L41-L51) + +```go +func Create(ctx context.Context, db *sql.DB, cipher *encrypt.Cipher, list []crm.Contact) error { + encrypted, err := Encrypt(ctx, cipher, list) + if err != nil { + return err + } + for _, e := range encrypted { + _, err := db.ExecContext(ctx, ` + INSERT INTO contacts (id, email, email_eq, email_match, phone_number, phone_number_eq) + VALUES ($1, $2, $3, $4, $5, $6)`, + e.ID, e.Email.Ciphertext, e.Email.Equality, e.Email.Match, + e.PhoneNumber.Ciphertext, e.PhoneNumber.Equality) +``` + +`CreateWithGORM` asks `EncryptRows` for the flat model before GORM receives it. + +[`contacts.go`, lines 59 to 66](contacts.go#L59-L66) + +```go +// CreateWithGORM writes the model, which GORM maps one field to one column. +func CreateWithGORM(ctx context.Context, db *gorm.DB, cipher *encrypt.Cipher, list []crm.Contact) error { + rows, err := EncryptRows(ctx, cipher, list) + if err != nil { + return err + } + return db.WithContext(ctx).Create(&rows).Error +} +``` + +## Search by the phone search term + +- **Purpose:** Derive and use the equality search term for a separate column. +- **Related:** [The plan's call design](../../2026-10-04-plan-builder.md#calls), [the plan's terminology](../../2026-10-04-plan-builder.md#terminology) + +`Fields.PhoneNumber.Equality` derives an `encrypt.EqualityTerm` search term. +The query compares that search term with the `phone_number_eq` column. + +[`contacts.go`, lines 68 to 76](contacts.go#L68-L76) + +```go +func IDByPhone(ctx context.Context, db *sql.DB, cipher *encrypt.Cipher, phone string) (int64, error) { + term, err := Fields.PhoneNumber.Equality(ctx, cipher, phone) + if err != nil { + return 0, err + } + var id int64 + err = db.QueryRowContext(ctx, `SELECT id FROM contacts WHERE phone_number_eq = $1`, term).Scan(&id) + return id, err +} +``` diff --git a/docs/plans/2026-10-04-plan-builder/crm/README.md b/docs/plans/2026-10-04-plan-builder/crm/README.md new file mode 100644 index 000000000..5ed583424 --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/crm/README.md @@ -0,0 +1,72 @@ +# Stand in for a type in another package + +This example stands in for a package that the program does not own. +Its `crm.Contact` type has no `stash` tags because another package controls its source. +`crm` imports no code from the Golang SDK for CipherStash Stack. + +The SDK is planned and does not exist yet. +The [`contacts` package](../contacts/README.md) uses SDK code, so that package does not build in this repository. +The [Go example index](../README.md) lists every example and what was checked. + +## What you will learn + +- Declare encryption tags for a type in another package from your own package. +- Check changes to `crm.Contact` at compile time. + +## Declare tags in your own package + +- **Purpose:** Understand why `crm.Contact` stays free of encryption tags. +- **Related:** [The plan's design for types in another package](../../2026-10-04-plan-builder.md#types-in-another-package), [the contacts walkthrough](../contacts/README.md) + +`crm.Contact` contains only the fields supplied by the stand-in package. + +[`contact.go`, lines 1 to 9](contact.go#L1-L9) + +```go +// Package crm stands in for a package this program does not own. +package crm + +type Contact struct { + ID int64 + Email string + PhoneNumber string + Internal string +} +``` + +The [`contacts` package](../contacts/README.md) uses `contactStash` to declare tags for `crm.Contact`. +`contactStash` leaves out `Internal` with `stash:"-"`. + +[`contacts.go`, lines 19 to 25](../contacts/contacts.go#L19-L25) + +```go +type contactStash struct { + _ struct{} `stash:"context=contacts"` + ID int64 `stash:"id,passthrough"` + Email string `stash:"email,encrypt,index=equality;match"` + PhoneNumber string `stash:"phone_number,encrypt,index=equality"` + Internal string `stash:"-"` +} +``` + +## Check the type's shape at compile time + +- **Purpose:** See how a change to `crm.Contact` stops the contacts package from building. +- **Related:** [The contacts walkthrough](../contacts/README.md), [the plan's design for types in another package](../../2026-10-04-plan-builder.md#types-in-another-package) + +The generated file converts `crm.Contact` to `contactShape` at compile time. +The conversion fails when `crm.Contact` gains, loses, reorders, or retypes a field. + +[`contactstash_stash.go`, lines 45 to 53](../contacts/contactstash_stash.go#L45-L53) + +```go +// Stops compiling when crm.Contact gains, loses, reorders or retypes a field. +var _ = contactShape(crm.Contact{}) + +type contactShape struct { + ID int64 + Email string + PhoneNumber string + Internal string +} +``` diff --git a/docs/plans/2026-10-04-plan-builder/documents/README.md b/docs/plans/2026-10-04-plan-builder/documents/README.md new file mode 100644 index 000000000..22cfcb3e9 --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/documents/README.md @@ -0,0 +1,123 @@ +# Encrypt a document as one opaque value + +This example encrypts each `Document` as one ciphertext column. +Its fields have no tags because the opaque layout does not expose or index them separately. +The Golang SDK for CipherStash Stack is planned and does not exist yet, so this code does not build in this repository. +The [Go example index](../README.md) lists every example and what was checked. + +`document_stash.go` is written by hand. +It shows what `stashgen` is planned to write. + +## What you will learn + +- Declare an opaque tagged struct and its encryption context. +- Read the opaque declaration and single ciphertext field. +- Save with a cipher and load with the client. + +## Declare one opaque value + +- **Purpose:** Understand why the document fields have no individual tags. +- **Related:** [The plan's struct tags](../../2026-10-04-plan-builder.md#struct-tags), [the plan's column layouts](../../2026-10-04-plan-builder.md#columns) + +The `opaque` option is part of the encryption context tag. +It tells the planned generator to encrypt the complete `Document` as one value. +`Title`, `Body`, and `Tags` therefore have no `stash` tags. + +[`documents.go`, lines 12 to 19](documents.go#L12-L19) + +```go +//go:generate go tool stashgen -type Document + +type Document struct { + _ struct{} `stash:"context=documents/v2/body,opaque"` + Title string + Body string + Tags []string +} +``` + +Nothing inside the document can be read or searched independently through this layout. + +## Read the single ciphertext layout + +- **Purpose:** Follow the opaque tag into the encrypted type and declaration. +- **Related:** [The plan's generated file design](../../2026-10-04-plan-builder.md#what-stashgen-writes) + +`EncryptedDocument` has one `Sealed` ciphertext field. + +[`document_stash.go`, lines 19 to 29](document_stash.go#L19-L29) + +```go +type EncryptedDocument struct { + Sealed encrypt.Ciphertext +} + +func (e EncryptedDocument) String() string { + return gensupport.Redacted("EncryptedDocument", nil, "Sealed") +} + +func (e EncryptedDocument) LogValue() slog.Value { + return gensupport.RedactedLog(nil, "Sealed") +} +``` + +`DeclareOpaque` builds the declaration for the complete value. +The codec maps all three document fields into `gensupport.OpaqueField`. + +[`document_stash.go`, lines 41 to 49](document_stash.go#L41-L49) + +```go +var declaration = gensupport.DeclareOpaque("documents/v2/body") + +var codec = gensupport.New(gensupport.Generated[Document, EncryptedDocument]{ + TypeName: "Document", + Declaration: declaration, + PrintsPlaintext: true, + Source: func(v Document) gensupport.Values { + return gensupport.Values{gensupport.OpaqueField: map[string]any{"title": v.Title, "body": v.Body, "tags": v.Tags}} + }, +``` + +## Save with a cipher and load with the client + +- **Purpose:** See why `Save` uses a cipher and `Load` uses the client. +- **Related:** [The plan's call design](../../2026-10-04-plan-builder.md#calls), [the root example program](../main.go) + +`Save` encrypts one document by passing a one-element slice. +It stores `encrypted[0].Sealed` in the `body` column. + +[`documents.go`, lines 21 to 28](documents.go#L21-L28) + +```go +func Save(ctx context.Context, db *sql.DB, cipher *encrypt.Cipher, id int64, doc Document) error { + encrypted, err := Encrypt(ctx, cipher, []Document{doc}) + if err != nil { + return err + } + _, err = db.ExecContext(ctx, `INSERT INTO documents (id, body) VALUES ($1, $2)`, id, encrypted[0].Sealed) + return err +} +``` + +`Load` decrypts with the client, not the cipher. +A `*Client` decrypts each value under the keyset that encrypted it. +A `*Cipher` refuses a value from another keyset with `ErrForeignKeyset`. + +[`documents.go`, lines 30 to 43](documents.go#L30-L43) + +```go +// Load decrypts with the client, which opens each value under the keyset that +// sealed it. A *encrypt.Cipher would also refuse a value from another +// keyset. +func Load(ctx context.Context, db *sql.DB, client *encrypt.Client, id int64) (Document, error) { + var encrypted EncryptedDocument + if err := db.QueryRowContext(ctx, `SELECT body FROM documents WHERE id = $1`, id).Scan(&encrypted.Sealed); err != nil { + return Document{}, err + } + docs, err := Decrypt(ctx, client, []EncryptedDocument{encrypted}) + if err != nil { + return Document{}, err + } + return docs[0], nil +} +``` diff --git a/docs/plans/2026-10-04-plan-builder/individuals/README.md b/docs/plans/2026-10-04-plan-builder/individuals/README.md new file mode 100644 index 000000000..35dffde4f --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/individuals/README.md @@ -0,0 +1,189 @@ +# Store a protobuf message with mixed encrypted layouts + +This example shows the generated file planned from the `Individual` policy rules. +One message combines passthrough fields, separate columns, and one Encrypt Query Language (EQL) column. +The Golang SDK for CipherStash Stack is planned and does not exist yet, so this code does not build in this repository. +The [Go example index](../README.md) lists every example and what was checked. + +`individual_stash.go` is written by hand. +It shows what `stashgen.Generate` is planned to write. + +## What you will learn + +- Read the mixed storage layouts chosen by the policy rules. +- Follow pointer messages through field-by-name reads instead of a shape check. +- Use the generated helpers and let the compiler reject an undeclared query. +- Insert encrypted fields and search the Medicare EQL column. + +## Read the mixed storage layouts + +- **Purpose:** Connect policy decisions to the encrypted type and declaration. +- **Related:** [The rules walkthrough](../rules/README.md), [the plan's policy declarations](../../2026-10-04-plan-builder.md#declarations-from-a-policy) + +`Id` and `Nickname` are passthrough fields. +`Name` has one ciphertext, while `Email` has a ciphertext and two search terms. +`MedicareNo` uses one `TextEq` EQL column named `medicare_number` in the declaration. + +[`individual_stash.go`, lines 25 to 31](individual_stash.go#L25-L31) + +```go +type EncryptedIndividual struct { + Id int64 + Name EncryptedIndividualName + Email EncryptedIndividualEmail + MedicareNo eql.TextEq + Nickname string +} +``` + +[`individual_stash.go`, lines 33 to 41](individual_stash.go#L33-L41) + +```go +type EncryptedIndividualName struct { + Ciphertext encrypt.Ciphertext +} + +type EncryptedIndividualEmail struct { + Ciphertext encrypt.Ciphertext + Equality encrypt.EqualityTerm + Match encrypt.MatchTerm +} +``` + +[`individual_stash.go`, lines 51 to 56](individual_stash.go#L51-L56) + +```go +var declaration = gensupport.Declare("individuals"). + Passthrough("id"). + Encrypt("name"). + EncryptIndex("email", encrypt.Equality, encrypt.Match()). + EncryptInto("medicare_number", "TextEq"). + Passthrough("nickname") +``` + +## Read pointer messages by field name + +- **Purpose:** Understand why protobuf messages do not use the usual shape check. +- **Related:** [The generated protobuf walkthrough](../internal/pb/README.md), [the plan's design for types in another package](../../2026-10-04-plan-builder.md#types-in-another-package) + +The codec takes `*pb.Individual` values. +It reads each field through a generated getter because direct struct conversion cannot include unexported fields. + +[`individual_stash.go`, lines 58 to 69](individual_stash.go#L58-L69) + +```go +var codec = gensupport.New(gensupport.Generated[*pb.Individual, EncryptedIndividual]{ + TypeName: "pb.Individual", + Declaration: declaration, + PrintsPlaintext: true, + Source: func(v *pb.Individual) gensupport.Values { + return gensupport.Values{ + "id": v.GetId(), + "name": v.GetName(), + "email": v.GetEmail(), + "medicare_number": v.GetMedicareNo(), + "nickname": v.GetNickname(), + } +``` + +The compiler finds a removed or retyped field through these named reads. +Adding a field to `Individual` does not stop the build. +CI runs `go generate ./...` and fails when a generated file differs from the committed file. + +> **Take care** +> +> The [`String` method of `pb.Individual`](../internal/pb/individual.pb.go#L44-L46) prints every field in plaintext. +> `stashgen` cannot replace that method, because `pb.Individual` is a type in another package. +> Do not print or log `pb.Individual` values. + +## Use the generated helpers + +- **Purpose:** Follow protobuf pointers and the Medicare field through the generated API. +- **Related:** [The plan's call design](../../2026-10-04-plan-builder.md#calls), [the generation command walkthrough](../cmd/genencrypt/README.md) + +`Encrypt` accepts a slice of `*pb.Individual` pointers. +`Decrypt` returns the same pointer form. + +[`individual_stash.go`, lines 117 to 123](individual_stash.go#L117-L123) + +```go +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, individuals []*pb.Individual) ([]EncryptedIndividual, error) { + return codec.Encrypt(ctx, cipher, individuals) +} + +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedIndividual) ([]*pb.Individual, error) { + return codec.Decrypt(ctx, d, encrypted) +} +``` + +`Name` declares encryption without an index, so `NameField` has only an `Encrypt` method. + +[`individual_stash.go`, lines 135 to 142](individual_stash.go#L135-L142) + +```go +type NameField struct { + field gensupport.Field[string] +} + +func (f NameField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (EncryptedIndividualName, error) { + out, err := f.field.Encrypt(ctx, c, v) + return EncryptedIndividualName{Ciphertext: out.Ciphertext}, err +} +``` + +`Fields.Name.Equality` does not compile because `NameField` does not declare that method. + +`Fields.MedicareNo.Query` returns an `eql.TextEqQuery` query value. + +[`individual_stash.go`, lines 168 to 176](individual_stash.go#L168-L176) + +```go +func (f MedicareNoField) Encrypt(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEq, error) { + out, err := f.field.Encrypt(ctx, c, v) + return eql.TextEq(out.EQL), err +} + +func (f MedicareNoField) Query(ctx context.Context, c *encrypt.Cipher, v string) (eql.TextEqQuery, error) { + out, err := f.field.Query(ctx, c, v) + return eql.TextEqQuery(out.EQL), err +} +``` + +## Insert fields and search Medicare numbers + +- **Purpose:** Map each storage layout to its database columns and query. +- **Related:** [The plan's database design](../../2026-10-04-plan-builder.md#databases) + +`Create` passes the passthrough fields, ciphertexts, search terms, and EQL value to their columns. + +[`store.go`, lines 14 to 23](store.go#L14-L23) + +```go +func Create(ctx context.Context, db *sql.DB, cipher *encrypt.Cipher, people []*pb.Individual) error { + encrypted, err := Encrypt(ctx, cipher, people) + if err != nil { + return err + } + for _, e := range encrypted { + _, err := db.ExecContext(ctx, ` + INSERT INTO individuals (id, nickname, name, email, email_eq, email_match, medicare_number) + VALUES ($1, $2, $3, $4, $5, $6, $7)`, + e.Id, e.Nickname, e.Name.Ciphertext, e.Email.Ciphertext, e.Email.Equality, e.Email.Match, e.MedicareNo) +``` + +`IDByMedicare` creates a query value and casts it to `eql_v3.query_text_eq` in SQL. + +[`store.go`, lines 31 to 40](store.go#L31-L40) + +```go +func IDByMedicare(ctx context.Context, db *sql.DB, cipher *encrypt.Cipher, medicareNo string) (int64, error) { + query, err := Fields.MedicareNo.Query(ctx, cipher, medicareNo) + if err != nil { + return 0, err + } + var id int64 + err = db.QueryRowContext(ctx, + `SELECT id FROM individuals WHERE medicare_number = $1::eql_v3.query_text_eq`, query).Scan(&id) + return id, err +} +``` diff --git a/docs/plans/2026-10-04-plan-builder/internal/pb/README.md b/docs/plans/2026-10-04-plan-builder/internal/pb/README.md new file mode 100644 index 000000000..77ee577c7 --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/internal/pb/README.md @@ -0,0 +1,105 @@ +# Use generated protobuf messages and field options + +This example contains real `protoc-gen-go` output for the `Individual` message and its field option. +The rules use `pb.Individual`, while the individuals code uses its getters and exported fields. +The planned `protosource` package reads the descriptors and `E_DataCategories`. +The Golang SDK for CipherStash Stack is planned and does not exist yet, so those examples do not build in this repository. +The [Go example index](../../README.md) lists every example and what was checked. + +## What you will learn + +- Generate the protobuf package again without editing generated code. +- Use `pb.Individual` and its getters from generated encryption code. +- Read `E_DataCategories` through the planned protobuf source. + +## Generate the protobuf package again + +- **Purpose:** Identify the generators, versions, and the command that generates the package again. +- **Related:** [The protobuf source walkthrough](../../proto/README.md) + +buf v1.50.0 ran `protoc-gen-go` v1.26.0 to write these files. +The generated header reports the `protoc` version as unknown. +Run `buf generate` in [`proto/`](../../proto/) to generate the files again. +A person must not edit them. + +Adding a field to `Individual` does not stop the build. +CI runs `go generate ./...` and fails when a generated file differs from the committed file. + +[`individual.pb.go`, lines 1 to 7](individual.pb.go#L1-L7) + +```go +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.26.0 +// protoc (unknown) +// source: individual.proto + +package pb +``` + +## Use the message and getters + +- **Purpose:** Identify the generated values that the individuals example reads. +- **Related:** [The generated individuals walkthrough](../../individuals/README.md), [the plan's design for types in another package](../../../2026-10-04-plan-builder.md#types-in-another-package) + +`pb.Individual` includes three unexported protobuf fields before its exported data fields. + +[`individual.pb.go`, lines 23 to 33](individual.pb.go#L23-L33) + +```go +type Individual struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Id int64 `protobuf:"varint,1,opt,name=id,proto3" json:"id,omitempty"` + Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"` + Email string `protobuf:"bytes,3,opt,name=email,proto3" json:"email,omitempty"` + MedicareNo string `protobuf:"bytes,4,opt,name=medicare_no,json=medicareNo,proto3" json:"medicare_no,omitempty"` + Nickname string `protobuf:"bytes,5,opt,name=nickname,proto3" json:"nickname,omitempty"` +} +``` + +Those unexported fields prevent a shape check through direct struct conversion. +`individuals/individual_stash.go` reads each field through `GetId`, `GetName`, `GetEmail`, `GetMedicareNo`, or `GetNickname`. + +[`individual.pb.go`, lines 88 to 100](individual.pb.go#L88-L100) + +```go +func (x *Individual) GetMedicareNo() string { + if x != nil { + return x.MedicareNo + } + return "" +} + +func (x *Individual) GetNickname() string { + if x != nil { + return x.Nickname + } + return "" +} +``` + +The examples do not use protobuf internals such as `file_individual_proto_rawDesc`. + +## Read the field option identifier + +- **Purpose:** Connect the protobuf option to the planned policy source. +- **Related:** [The rules walkthrough](../../rules/README.md), [the plan's policy declarations](../../../2026-10-04-plan-builder.md#declarations-from-a-policy) + +`classification.pb.go` exports `E_DataCategories` for the repeated field option. + +[`classification.pb.go`, lines 34 to 40](classification.pb.go#L34-L40) + +```go +// Extension fields to descriptorpb.FieldOptions. +var ( + // The Fideslang data categories of a field. + // + // repeated string data_categories = 50001; + E_DataCategories = &file_classification_proto_extTypes[0] +) +``` + +The planned `protosource` package reads this option through protobuf descriptors. diff --git a/docs/plans/2026-10-04-plan-builder/internal/userdb/README.md b/docs/plans/2026-10-04-plan-builder/internal/userdb/README.md new file mode 100644 index 000000000..2393ce3a3 --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/internal/userdb/README.md @@ -0,0 +1,94 @@ +# Use the generated sqlc package with the encrypted type + +This example is the database package that [sqlc](https://github.com/sqlc-dev/sqlc) generates for the [`users` example](../../users/README.md). +Its structs match `users.EncryptedUser`, so the store uses Go conversions instead of field copies. +The generated fields use planned Encrypt Query Language (EQL) types. +The Golang SDK for CipherStash Stack is planned and does not exist yet, so this code does not build in this repository. +The [Go example index](../../README.md) lists every example and what was checked. + +## What you will learn + +- Generate the sqlc package again without editing generated code. +- Convert sqlc's row struct and parameters to the encrypted type. +- Use the generated methods from the users store. + +## Generate the sqlc package again + +- **Purpose:** Identify the generator, its version, and the command that generates the package again. +- **Related:** [The sqlc inputs](../../sqlc/README.md) + +sqlc v1.31.1 generated `db.go`, `models.go`, and `query.sql.go`. +Run `sqlc generate` in [`sqlc/`](../../sqlc/) to generate them again. +A person must not edit these files. + +[`db.go`, lines 1 to 5](db.go#L1-L5) + +```go +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 + +package userdb +``` + +## Convert generated structs by shape + +- **Purpose:** See why direct Go conversions connect sqlc to the encrypted type. +- **Related:** [The plan's model design](../../../2026-10-04-plan-builder.md#models), [the users sqlc store](../../users/README.md) + +sqlc's row struct `userdb.User` has the same fields as `users.EncryptedUser`. + +[`models.go`, lines 11 to 15](models.go#L11-L15) + +```go +type User struct { + ID int64 + Email eql.TextEq + Name eql.TextEq +} +``` + +`CreateUserParams` has the same shape too. + +[`query.sql.go`, lines 18 to 22](query.sql.go#L18-L22) + +```go +type CreateUserParams struct { + ID int64 + Email eql.TextEq + Name eql.TextEq +} +``` + +Go ignores struct tags during struct conversion in Go 1.8 and later. +The `db` and `gorm` tags on `users.EncryptedUser` therefore do not stop either conversion. +The users store converts with `EncryptedUser(row)` and `userdb.CreateUserParams(e)`. +A field change stops either conversion from compiling until both shapes agree. + +## Use the generated methods + +- **Purpose:** Connect generated identifiers to the handwritten users store. +- **Related:** [The users sqlc store](../../users/README.md), [the sqlc queries](../../sqlc/query.sql) + +`userdb.New` creates `Queries`, and `WithTx` binds the methods to a transaction. +The users store calls `CreateUser`, `GetUser`, and `FindUsersByEmail`. + +[`db.go`, lines 19 to 31](db.go#L19-L31) + +```go +func New(db DBTX) *Queries { + return &Queries{db: db} +} + +type Queries struct { + db DBTX +} + +func (q *Queries) WithTx(tx *sql.Tx) *Queries { + return &Queries{ + db: tx, + } +} +``` + +The examples do not use the generated `ListUsers` method. diff --git a/docs/plans/2026-10-04-plan-builder/proto/README.md b/docs/plans/2026-10-04-plan-builder/proto/README.md new file mode 100644 index 000000000..cc641a8fe --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/proto/README.md @@ -0,0 +1,78 @@ +# Annotate protobuf fields for encryption rules + +This example defines a protobuf field option and applies it to an `Individual` message. +The option lists [data categories](https://ethyca.github.io/fideslang/taxonomy/data_categories/) from [Fideslang](https://ethyca.github.io/fideslang/), a published taxonomy of personal data. +`buf generate` works for these protobuf files. + +The Golang SDK for CipherStash Stack is planned and does not exist yet. +The rules that read the field options belong to the planned SDK. +The [Go example index](../README.md) lists every example and what was checked. + +## What you will learn + +- Define the `data_categories` field option. +- Annotate message fields while leaving one field uncategorised. +- Generate the Go protobuf package in `internal/pb`. + +## Define the data category option + +- **Purpose:** See how a field option labels protobuf fields for encryption rules. +- **Related:** [The plan's policy declarations](../../2026-10-04-plan-builder.md#declarations-from-a-policy), [the rules walkthrough](../rules/README.md) + +`classification.proto` extends `google.protobuf.FieldOptions` with `data_categories`. +Each value names one Fideslang data category, such as `user.contact.email`. + +[`classification.proto`, lines 5 to 12](classification.proto#L5-L12) + +```proto +import "google/protobuf/descriptor.proto"; + +option go_package = "example.com/app/internal/pb"; + +extend google.protobuf.FieldOptions { + // The Fideslang data categories of a field. + repeated string data_categories = 50001; +} +``` + +## Annotate the message fields + +- **Purpose:** Follow each protobuf field into the policy rules. +- **Related:** [The rules walkthrough](../rules/README.md), [the generated individuals walkthrough](../individuals/README.md) + +`name`, `email`, and `medicare_no` each have a category. +`id` and `nickname` have no category, so message-specific rules must decide them. + +[`individual.proto`, lines 9 to 15](individual.proto#L9-L15) + +```proto +message Individual { + int64 id = 1; + string name = 2 [(classification.data_categories) = "user.name"]; + string email = 3 [(classification.data_categories) = "user.contact.email"]; + string medicare_no = 4 [(classification.data_categories) = "user.government_id"]; + string nickname = 5; +} +``` + +## Generate the Go protobuf package + +- **Purpose:** Understand where buf writes the generated Go files. +- **Related:** [The generated protobuf walkthrough](../internal/pb/README.md) + +`buf.gen.yaml` writes source-relative Go files into `../internal/pb`. + +[`buf.gen.yaml`, lines 1 to 5](buf.gen.yaml#L1-L5) + +```yaml +version: v2 +plugins: + - local: protoc-gen-go + out: ../internal/pb + opt: paths=source_relative +``` + +Run `buf generate` in this directory to generate the package again. +The committed files came from buf v1.50.0 and `protoc-gen-go` v1.26.0. +Adding a field to `Individual` does not stop the build. +CI runs `go generate ./...` and fails when a generated file differs from the committed file. diff --git a/docs/plans/2026-10-04-plan-builder/rules/README.md b/docs/plans/2026-10-04-plan-builder/rules/README.md new file mode 100644 index 000000000..895edff7f --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/rules/README.md @@ -0,0 +1,98 @@ +# Decide protobuf encryption from data categories + +This example defines planned policy rules for `pb.Individual` fields. +The first matching rule chooses each field's storage layout. +The Golang SDK for CipherStash Stack is planned and does not exist yet, so this code does not build in this repository. +The [Go example index](../README.md) lists every example and what was checked. + +## What you will learn + +- Match Fideslang data categories with shared rules. +- Add message-specific decisions and a column name. +- Run the generation command without using rules at application runtime. + +## Match data categories with shared rules + +- **Purpose:** Understand how `Base` decides fields from protobuf annotations. +- **Related:** [The plan's policy declarations](../../2026-10-04-plan-builder.md#declarations-from-a-policy), [the protobuf source walkthrough](../proto/README.md) + +A protobuf field can name [data categories](https://ethyca.github.io/fideslang/taxonomy/data_categories/) from [Fideslang](https://ethyca.github.io/fideslang/), such as the category `user.contact.email`. +`policy.Key` names the field option by its full name, `classification.data_categories`. +`policy.FirstOf` evaluates each `policy.When` in order, and the first match decides. + +[`rules.go`, lines 13 to 21](rules.go#L13-L21) + +```go +// The key is the full name of the field option in proto/classification.proto. +var category = policy.Key("classification.data_categories") + +// Base applies to every message. The first rule that matches a field decides it. +var Base = policy.FirstOf( + policy.When(category.Under("user.government_id"), policy.EncryptInto("TextEq")), + policy.When(category.Under("user.contact.email"), policy.EncryptIndex(encrypt.Equality, encrypt.Match())), + policy.When(category.Under("user"), policy.Encrypt()), +) +``` + +Each rule in `Base` makes one decision: + +- `user.government_id` uses `EncryptInto("TextEq")` and stores one Encrypt Query Language (EQL) column. +- `user.contact.email` uses `EncryptIndex` and stores a ciphertext with equality and match search terms in separate columns. +- `user` uses `Encrypt()` and stores a ciphertext without an index. + +`user.contact.email` is under `user`, so its rule must come before the broader `user` rule. +If the `user` rule came first, email would use `Encrypt()` without an index. +The policy API also defines `Index`, `Omit`, and `Fail` decisions. + +## Add message-specific decisions + +- **Purpose:** See how `Individuals` handles fields before it falls back to `Base`. +- **Related:** [The generated individuals walkthrough](../individuals/README.md), [the protobuf source walkthrough](../proto/README.md) + +`policy.ForMessage` binds the rules to `pb.Individual` and the `individuals` encryption context. +`policy.Field` handles uncategorised fields and overrides a categorised field. +`policy.Name("medicare_number")` changes the database column name for `medicare_no`. + +[`rules.go`, lines 23 to 31](rules.go#L23-L31) + +```go +// Individuals adds the rules for one message. A field with no data category +// needs a rule too: with none, the generator stops. +var Individuals = policy.ForMessage(&pb.Individual{}, policy.Context("individuals"), + policy.FirstOf( + policy.When(policy.Field("medicare_no"), policy.EncryptInto("TextEq"), policy.Name("medicare_number")), + policy.When(policy.Field("id"), policy.Passthrough()), + policy.When(policy.Field("nickname"), policy.Passthrough()), + ).OrElse(Base), +) +``` + +`OrElse(Base)` applies the shared rules when no message-specific rule matches. +A field with no matching rule stops the planned generator. +Adding a field to `Individual` does not stop the build. +CI runs `go generate ./...` and fails when a generated file differs from the committed file. + +## Run rules during generation + +- **Purpose:** Keep policy evaluation out of the application runtime. +- **Related:** [The generation command walkthrough](../cmd/genencrypt/README.md), [the plan's policy declarations](../../2026-10-04-plan-builder.md#declarations-from-a-policy) + +The `go:generate` line runs `cmd/genencrypt`. + +[`rules.go`, lines 1 to 11](rules.go#L1-L11) + +```go +// Package rules decides what to encrypt from the data categories that the +// protobuf schema gives each field. Only the generate program runs it. +package rules + +import ( + "example.com/app/internal/pb" + "github.com/cipherstash/stack/languages/golang/encrypt" + "github.com/cipherstash/stack/languages/golang/encrypt/policy" +) + +//go:generate go run ../cmd/genencrypt +``` + +The application uses the generated functions and never runs these rules. diff --git a/docs/plans/2026-10-04-plan-builder/sqlc/README.md b/docs/plans/2026-10-04-plan-builder/sqlc/README.md new file mode 100644 index 000000000..7117ab0b6 --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/sqlc/README.md @@ -0,0 +1,99 @@ +# Generate sqlc code for Encrypt Query Language columns + +This example configures [sqlc](https://github.com/sqlc-dev/sqlc) for a `users` table with two Encrypt Query Language (EQL) columns. +It also defines the queries that generate the `userdb` package. +`sqlc generate` ran for these SQL and YAML files. + +The Golang SDK for CipherStash Stack is planned and does not exist yet. +The generated package names the planned `encrypt/eql` package, so it does not build in this repository. +The [Go example index](../README.md) lists every example and what was checked. + +## What you will learn + +- Define the EQL domains and table columns that sqlc reads. +- Map each EQL domain spelling to its planned Go type. +- Generate typed methods from queries that use EQL columns. + +## Define the EQL domains and table columns + +- **Purpose:** Understand the schema files that sqlc reads. +- **Related:** [The plan's database design](../../2026-10-04-plan-builder.md#databases), [the generated `userdb` package](../internal/userdb/README.md) + +`eql-domains.sql` gives sqlc small domain declarations because sqlc cannot parse the EQL installation bundle. +The domain spelling must match `schema.sql` and the matching override. + +[`eql-domains.sql`, lines 1 to 6](eql-domains.sql#L1-L6) + +```sql +-- For sqlc only: sqlc cannot parse the EQL install bundle. Spell each domain +-- exactly as schema.sql and query.sql do, or its db_type override never matches. +CREATE SCHEMA eql_v3; + +CREATE DOMAIN public.eql_v3_text_eq AS jsonb; +CREATE DOMAIN eql_v3.query_text_eq AS jsonb; +``` + +`schema.sql` uses one `public.eql_v3_text_eq` EQL column for each encrypted field. + +[`schema.sql`, lines 1 to 5](schema.sql#L1-L5) + +```sql +CREATE TABLE users ( + id bigint PRIMARY KEY, + email public.eql_v3_text_eq NOT NULL, + name public.eql_v3_text_eq NOT NULL +); +``` + +Read the index's [three rules for sqlc with EQL columns](../README.md#use-sqlc-with-eql-columns) before you change these spellings. + +## Map each domain to its Go type + +- **Purpose:** Follow the sqlc inputs and type overrides into the generated package. +- **Related:** [The plan's model design](../../2026-10-04-plan-builder.md#models), [the generated `userdb` package](../internal/userdb/README.md) + +`sqlc.yaml` reads the domain file before the table schema. +It writes package `userdb` into `../internal/userdb`. + +[`sqlc.yaml`, lines 3 to 16](sqlc.yaml#L3-L16) + +```yaml + - engine: postgresql + schema: + - eql-domains.sql + - schema.sql + queries: query.sql + gen: + go: + package: userdb + out: ../internal/userdb + overrides: + - db_type: "public.eql_v3_text_eq" + go_type: { import: github.com/cipherstash/stack/languages/golang/encrypt/eql, type: TextEq } + - db_type: "eql_v3.query_text_eq" + go_type: { import: github.com/cipherstash/stack/languages/golang/encrypt/eql, type: TextEqQuery } +``` + +Run `sqlc generate` in this directory to generate the package again with sqlc v1.31.1. + +## Generate typed query methods + +- **Purpose:** See how the SQL produces typed create, read, list, and search methods. +- **Related:** [The users sqlc store](../users/README.md), [the generated query methods](../internal/userdb/README.md) + +`query.sql` defines four methods through sqlc's query annotations. +The email search casts its parameter directly to `eql_v3.query_text_eq`. + +[`query.sql`, lines 7 to 13](query.sql#L7-L13) + +```sql +-- name: ListUsers :many +SELECT * FROM users ORDER BY id; + +-- One cast, straight to the query domain: sqlc types a parameter by its first +-- cast, so ::jsonb::eql_v3.query_text_eq would generate json.RawMessage. +-- name: FindUsersByEmail :many +SELECT * FROM users WHERE email = sqlc.arg(email)::eql_v3.query_text_eq; +``` + +The direct cast lets the override generate an `eql.TextEqQuery` parameter. diff --git a/docs/plans/2026-10-04-plan-builder/users/README.md b/docs/plans/2026-10-04-plan-builder/users/README.md new file mode 100644 index 000000000..9f6bc6d48 --- /dev/null +++ b/docs/plans/2026-10-04-plan-builder/users/README.md @@ -0,0 +1,322 @@ +# Encrypt and store users with generated helpers + +This example shows the planned path from a tagged struct to three database stores: `database/sql`, GORM and [sqlc](https://github.com/sqlc-dev/sqlc). +Each encrypted field uses one Encrypt Query Language (EQL) column. +The Golang SDK for CipherStash Stack is planned and does not exist yet, so this code does not build in this repository. +The [Go example index](../README.md) lists every example and what was checked. + +`user_stash.go` is written by hand. +It shows what `stashgen` is planned to write. + +## What you will learn + +- Declare stored fields and create a tenant cipher. +- Read what the generated file contains. +- Insert and search users with `database/sql`. +- Batch encryption before GORM receives values. +- Convert between sqlc structs and update one EQL column. + +## Declare fields and create a tenant cipher + +- **Purpose:** Connect the tagged struct to its generator command and runtime cipher. +- **Related:** [The plan's struct tags](../../2026-10-04-plan-builder.md#struct-tags), [the root example program](../main.go) + +The `go:generate` line asks `stashgen` to read `User`. +The encryption context is `users`, and every exported field has a `stash` tag. +`ID` is a passthrough field, while `Internal` is an omitted field. +`Email` and `Name` each become one `TextEq` EQL column. + +[`model.go`, lines 3 to 13](model.go#L3-L13) + +```go +//go:generate go tool stashgen -type User + +// Every exported field needs a stash tag, so a new field cannot reach the +// database unencrypted by accident. +type User struct { + _ struct{} `stash:"context=users"` + ID int64 `stash:"id,passthrough" db:"id" gorm:"primaryKey"` + Email string `stash:"email,encrypt_into=TextEq" db:"email"` + Name string `stash:"name,encrypt_into=TextEq" db:"name"` + Internal string `stash:"-"` +} +``` + +The root program creates the client and derives one cipher for tenant `tenant-42`. +The cipher has the tenant's keyset and its extension to each encryption context. +`Extend` adds the tenant segment to each field's declared encryption context. +No call takes a keyset or an encryption context, so the write, query, and read cannot use different ones. + +[`../main.go`, lines 25 to 30](../main.go#L25-L30) + +```go +func run(ctx context.Context) error { + client, err := encrypt.NewClient(ctx, encrypt.WithCredentials(encrypt.AutoCredentials())) + if err != nil { + return err + } + defer client.Close() +``` + +[`../main.go`, lines 38 to 41](../main.go#L38-L41) + +```go + // One cipher for each tenant: its keyset, and its part of every field's + // context. Every call through this cipher carries both. + cipher := client.Keyset(encrypt.KeysetName("tenant-42")).Extend("tenant-42") + store := users.NewSQLStore(db) +``` + +## Read what the generated file contains + +- **Purpose:** Understand the main parts that `stashgen` is planned to write. +- **Related:** [The plan's generated file design](../../2026-10-04-plan-builder.md#what-stashgen-writes), [the plan's call design](../../2026-10-04-plan-builder.md#calls) + +`EncryptedUser` keeps `ID` and replaces `Email` and `Name` with `eql.TextEq`. +Its `String` and `LogValue` methods hide the encrypted fields when code prints the encrypted type. + +[`user_stash.go`, lines 20 to 32](user_stash.go#L20-L32) + +```go +type EncryptedUser struct { + ID int64 `db:"id" gorm:"primaryKey"` + Email eql.TextEq `db:"email"` + Name eql.TextEq `db:"name"` +} + +func (e EncryptedUser) String() string { + return gensupport.Redacted("EncryptedUser", map[string]any{"ID": e.ID}, "Email", "Name") +} + +func (e EncryptedUser) LogValue() slog.Value { + return gensupport.RedactedLog(map[string]any{"ID": e.ID}, "Email", "Name") +} +``` + +The `userShape(User{})` conversion is the shape check. +It stops compilation when the tagged struct and generated file have different fields. +The declaration crosses to the engine with the struct. +It names each field's encryption context, indexes, and EQL type. + +[`user_stash.go`, lines 35 to 49](user_stash.go#L35-L49) + +```go +var _ = userShape(User{}) + +type userShape struct { + _ struct{} + ID int64 + Email string + Name string + Internal string +} + +var declaration = gensupport.Declare("users"). + Passthrough("id"). + EncryptInto("email", "TextEq"). + EncryptInto("name", "TextEq"). + Omit("internal") +``` + +The codec maps the tagged struct to named values before the engine encrypts them. + +[`user_stash.go`, lines 51 to 57](user_stash.go#L51-L57) + +```go +var codec = gensupport.New(gensupport.Generated[User, EncryptedUser]{ + TypeName: "User", + Declaration: declaration, + PrintsPlaintext: true, + Source: func(v User) gensupport.Values { + return gensupport.Values{"id": v.ID, "email": v.Email, "name": v.Name} + }, +``` + +`Encrypt` and `Decrypt` process slices through that codec. +ZeroKMS is the CipherStash key service. +`Encrypt` and `Decrypt` send one ZeroKMS request for the whole slice. +`Fields.Email.Query` creates the query value for a `WHERE` clause. + +[`user_stash.go`, lines 88 to 97](user_stash.go#L88-L97) + +```go +// Encrypt seals every user in one ZeroKMS request. The result has one element +// for each user, in the same order. +func Encrypt(ctx context.Context, cipher *encrypt.Cipher, users []User) ([]EncryptedUser, error) { + return codec.Encrypt(ctx, cipher, users) +} + +// Decrypt opens every value in one ZeroKMS request. +func Decrypt(ctx context.Context, d encrypt.Decrypter, encrypted []EncryptedUser) ([]User, error) { + return codec.Decrypt(ctx, d, encrypted) +} +``` + +[`user_stash.go`, lines 99 to 105](user_stash.go#L99-L105) + +```go +var Fields = struct { + Email EmailField + Name NameField +}{ + Email: EmailField{gensupport.NewField[string](declaration, "email")}, + Name: NameField{gensupport.NewField[string](declaration, "name")}, +} +``` + +> **Take care** +> +> `User` has no `String` or `LogValue` method, so a program that prints or logs it shows the plaintext of `Email` and `Name`. +> [`main.go`, line 76](../main.go#L76) prints only IDs and counts for that reason. +> Write both methods, or run `stashgen` with `-redact`. +> The [accounts walkthrough](../accounts/README.md) shows `-redact`. + +## Insert and search with database/sql + +- **Purpose:** Follow encrypted values through a transaction and an EQL equality query. +- **Related:** [The plan's database design](../../2026-10-04-plan-builder.md#databases), [the sqlc schema walkthrough](../sqlc/README.md) + +`SQLStore.Import` encrypts the whole input slice before it starts inserting rows. +It passes each `EncryptedUser` field directly to `ExecContext`. + +[`sqlstore.go`, lines 27 to 39](sqlstore.go#L27-L39) + +```go +// Import encrypts every user in one ZeroKMS request, then inserts them in one +// transaction. +func (s *SQLStore) Import(ctx context.Context, cipher *encrypt.Cipher, people []User) error { + encrypted, err := Encrypt(ctx, cipher, people) + if err != nil { + return fmt.Errorf("encrypt %d users: %w", len(people), err) + } + + tx, err := s.db.BeginTx(ctx, nil) + if err != nil { + return err + } + defer tx.Rollback() +``` + +[`sqlstore.go`, lines 47 to 52](sqlstore.go#L47-L52) + +```go + for _, e := range encrypted { + if _, err := stmt.ExecContext(ctx, e.ID, e.Email, e.Name); err != nil { + return fmt.Errorf("insert user %d: %w", e.ID, err) + } + } + return tx.Commit() +``` + +`FindByEmail` creates an `eql.TextEqQuery` query value through `Fields.Email.Query`. +The SQL casts `$1` directly to `eql_v3.query_text_eq`. + +[`sqlstore.go`, lines 59 to 70](sqlstore.go#L59-L70) + +```go +// FindByEmail matches the whole address. Postgres compares the encrypted +// column through EQL's = operator. +func (s *SQLStore) FindByEmail(ctx context.Context, cipher *encrypt.Cipher, email string) ([]User, error) { + query, err := Fields.Email.Query(ctx, cipher, email) + if err != nil { + return nil, err + } + rs, err := s.db.QueryContext(ctx, selectUser+` WHERE email = $1::eql_v3.query_text_eq`, query) + if err != nil { + return nil, err + } + defer rs.Close() +``` + +## Batch before GORM receives values + +- **Purpose:** See why encryption happens before GORM maps the encrypted type. +- **Related:** [The plan's database design](../../2026-10-04-plan-builder.md#databases), [the plan's call design](../../2026-10-04-plan-builder.md#calls) + +`EncryptedUser.TableName` sends GORM to `users` instead of `encrypted_users`. +A `driver.Valuer` hook cannot batch because it receives one field and no `context.Context`. +It also cannot observe cancellation through `ctx`. + +[`gormstore.go`, lines 10 to 19](gormstore.go#L10-L19) + +```go +// TableName lets GORM use the generated type as the model. Without it GORM +// would look for a table named encrypted_users. +func (EncryptedUser) TableName() string { return "users" } + +// GormStore encrypts before GORM sees a value. driver.Valuer gets no +// context.Context and runs one field at a time, so it cannot batch a ZeroKMS +// request or stop when the request is cancelled. +type GormStore struct { + db *gorm.DB +} +``` + +`GormStore.Create` encrypts the slice first, then gives GORM the encrypted values. + +[`gormstore.go`, lines 25 to 31](gormstore.go#L25-L31) + +```go +func (s *GormStore) Create(ctx context.Context, cipher *encrypt.Cipher, people ...User) error { + encrypted, err := Encrypt(ctx, cipher, people) + if err != nil { + return err + } + return s.db.WithContext(ctx).CreateInBatches(encrypted, 500).Error +} +``` + +## Convert sqlc structs and update one column + +- **Purpose:** Connect generated sqlc shapes and field-level updates to the encrypted type. +- **Related:** [The generated `userdb` walkthrough](../internal/userdb/README.md), [the sqlc inputs](../sqlc/README.md) + +`userdb.User` and `userdb.CreateUserParams` have the same fields as `EncryptedUser`. +Go therefore converts these structs directly, and a change to either shape stops the build. +The store converts `EncryptedUser` to `userdb.CreateUserParams` before each insert. + +[`sqlcstore.go`, lines 39 to 45](sqlcstore.go#L39-L45) + +```go + queries := s.queries.WithTx(tx) + for _, e := range encrypted { + if err := queries.CreateUser(ctx, userdb.CreateUserParams(e)); err != nil { + return err + } + } + return tx.Commit() +``` + +It converts each sqlc row back with `EncryptedUser(row)` before decryption. + +[`sqlcstore.go`, lines 68 to 76](sqlcstore.go#L68-L76) + +```go + rows, err := s.queries.FindUsersByEmail(ctx, query) + if err != nil { + return nil, err + } + encrypted := make([]EncryptedUser, len(rows)) + for i, row := range rows { + encrypted[i] = EncryptedUser(row) + } + return Decrypt(ctx, cipher, encrypted) +``` + +`ChangeEmail` encrypts one field and updates the single EQL column. +That column stores the ciphertext and its search term together. + +[`sqlcstore.go`, lines 79 to 88](sqlcstore.go#L79-L88) + +```go +// ChangeEmail rewrites one field. One EQL column holds the ciphertext and +// its term, so they cannot go out of step. +func (s *SQLCStore) ChangeEmail(ctx context.Context, cipher *encrypt.Cipher, id int64, email string) error { + sealed, err := Fields.Email.Encrypt(ctx, cipher, email) + if err != nil { + return err + } + _, err = s.db.ExecContext(ctx, `UPDATE users SET email = $2 WHERE id = $1`, id, sealed) + return err +} +```