Repository navigation
stack-encrypt: a plan cannot take its context from the caller or a field, cannot lay a field out as another type, and silently drops match options on the wire #1079
Description
Background
stack-encrypt (packages/stack-encrypt) is our Rust library for field-level encryption: each value is sealed under its own data key from ZeroKMS, our key management service, and can carry search terms ("indexes": equality, match, ORE, OPE) beside the ciphertext so an encrypted column can still be queried. Every sealed value is bound to a context, a label such as users/email that ties the data to where it belongs and that ZeroKMS sees as a descriptor.
A plan (the plan builder, #1057, built in #1071) is a saved, checked description of how a value is encrypted: its context and, field by field, what each field becomes (a ciphertext, search terms, or a value carried through unencrypted). The plan is meant to be the single front door: #1058 makes #[derive(EncryptFrom)] emit a plan, and #1062 makes EQL domains (the column shapes of EQL, our PostgreSQL encryption library) into plan field targets, including from a generated Go package. For either to work, a plan has to be able to say everything the derive says.
Problem
Mechanism. Even after the derive is narrowed to what a plan could reasonably express (#1078), the plan as built in #1071 is missing these pieces. Line numbers are against the #1073 branch, which the work stacks on:
- A plan must be given its context when it is built. The only start is
Plan::context(c)(src/plan/build.rs:200). The derive'splaintext = Trecords and its generic form take their context from the caller, at run time. A plan has no way to say "the caller supplies the context". - A plan cannot take its context from a field of the value. The derive's
#[stash(context_field)], which EQL's Rust types use, seals every other field under the value of one field (for example a tenant id) and checks it on decrypt. A plan has no equivalent. - A plan field cannot be laid out as another target type. The derive composes a field whose type is itself a derived record, and a
plaintext = Stringrecord is a tuple of targets over one value. A plan can only use its own verbs (encrypt,encrypt_index,index,passthrough), so it cannot express either, nor an EQL domain as a field target (Go and Rust plans cannot target EQL domains, so Go writes rows EQL cannot query — make domains plan field targets #1062). - A plan reads a field only by name, through the value's
Field<F>impl (src/plan/build.rs:388). Code that generates a plan (the derive, or a Go generator) would need every value type to implement those traits, and needs a turbofish per field. There is no way to hand the plan an accessor. - A field name that is not a plain label segment cannot be rescued by a pinned identity. A tuple struct's field
0cannot be keyed asvalue, which the derive needs to lower tuple-struct records. - Match options have no wire form, and are dropped silently. When a plan is written out as data (for the Go binding or a saved plan), a match index with non-default options (a different tokenizer or filter size) is written as plain
"match"(src/target/index.rs:91), and reading"match"back always yields the default options (src/dynamic/term.rs:80). The query side then derives terms under different options than the stored rows, so every query returns zero rows with no error. This was found in review of feat(stack-encrypt): passthrough, Index/Indexes, Encrypted<Terms> and field types for the plan builder #1069.
Impact. #1058 cannot emit a plan for every derive shape with identical bytes, and #1062 cannot make domains plan field targets, until these exist. Item 6 is a silent data-correctness bug: wrong search results, no error.
Why nothing catches it today. No test compares a derive's output with an equivalent plan, and nothing round-trips an index through its data form.
Proposal
- Two starts, each with an optional
.context(c):Plan::fields()for a field-by-field plan, andPlan::value::<S>()for a one-value plan.Plan::context(c)keeps working. - Exactly one context source, from one of three places: the plan (
.context(c)when built), the call that runs it (.context(c)on encrypt, query and open), or a field of the value (.context_field(field), mirroring the derive'scontext_field, with the same check on open before any key is requested). Two sources is an error at build or at the call; none is an error when the plan runs, before any key request. encrypt_into::<T>: lay a field (or a one-value plan) out as target typeT, throughT's ownEncryptFrom, under<context>/<field>. A tuple of targets is itself a target. A field is either verbs or one target, never both. Queries on a typed field answer only the indexes the target names (a defaultedEncryptFrom::indexes()); anything else is refused, never derived.- Pickers: every field verb also takes a name plus an accessor,
pick("email", |u: &User| &u.email), which needs noField<F>impl. - A pinned
identityrescues a field name that is not a plain segment: only the identity enters the label. - Match options on the wire: an explicit conversion from an index to its data form that round-trips every kind and refuses a match index with non-default options, rather than dropping them. A loud refusal beats a silent zero-row miss. The
"json"key stays reserved for the JSON index (stack-encrypt has no searchable JSON — add aJsonindex that seals a document's entries under one data key #1060). - Proof: tests that show, byte for byte, that a plan using each addition writes what the corresponding derive writes (same ZeroKMS descriptors, same term bytes, each record opens through the other).
Verified against the code (on the #1073 branch): items 1, 2, 4 and 6 of the Problem, at the cited lines. From the fix's PR: item 5, and that the Go binding's data form of a plan is otherwise unchanged.
Relationship to other work
- Blocked by stack-encrypt-derive:
#[derive(EncryptFrom)]accepts five shapes a plan cannot express, so it can never emit the plan with identical bytes #1078 (the derive narrowing): the plan grows to meet the narrowed derive, not the old one. - Blocks stack-encrypt-derive:
#[derive(EncryptFrom)]hand-writes combinators — emit the plan builder instead #1058 (the derive emits the plan) and Go and Rust plans cannot target EQL domains, so Go writes rows EQL cannot query — make domains plan field targets #1062 (EQL domains as plan field targets). - Resolved by feat(stack-encrypt): plan grammar gains two starts, run-time and field contexts, encrypt_into and pickers #1074.
Activity