Go SDK: stashgen refuses a pointer field such as *encrypt.Date, so an optional value cannot be encrypted #1298

Background

The Go SDK (languages/golang) encrypts fields of a Go struct. A developer tags each field with a stash:"…" struct tag, and stashgen, a code generator, writes the code that encrypts and decrypts the struct. A field tagged encrypt is stored as one ciphertext column. A field tagged encrypt_into=<Type> is stored as an EQL type. EQL (Encrypt Query Language) is the SQL library we install into the customer's PostgreSQL database. Each EQL type is a PostgreSQL domain (public.eql_v4_date, for example) that holds an encrypted payload as JSON. Some EQL types carry index terms, so the database can compare or order values without decrypting them. "Storage-only" types such as Date carry no index terms. They only store the ciphertext.

Encryption itself is done by Stack Encrypt, a Rust engine. Go runs it as a WebAssembly guest. The engine seals each value as a "leaf": a type tag plus the value's bytes, encrypted with AES-GCM.

Problem

There is no way to encrypt an optional value. A struct field of type *encrypt.Date (or any *T where T is a scalar) is refused by stashgen:

a *encrypt.Date seals only as part of an opaque struct

#1293 reproduces this with a test. The two inputs are identical except for the field type, and both are tagged stash:"dob,encrypt_into=Date". DOB encrypt.Date generates. DOB *encrypt.Date is refused.

How it happens:

1. classify (languages/golang/stashgen/read.go) gives a pointer field KindPointer.
2. buildFields refuses any sealed field whose kind is not a scalar (read.go:697), and KindPointer is not a scalar.
3. Nothing further down could take a nil either. gensupport.Codec.split requires every declared field to be in every row. The record plan sent to the engine has no way to mark a field optional. eql.*.Scan returns an error when it reads SQL NULL.

The limitation is documented (encrypt/values.go:26). The workarounds it suggests are to put the field in an opaque struct (the whole struct becomes one encrypted JSON document) or to store a sentinel date the program treats as "absent". Neither fits a common case: an optional date of birth on a person record. The opaque struct gives up the per-column EQL type. A sentinel date in the plaintext is a value the application has to remember never to show or compare.

Who hits it: anyone with a nullable column they want encrypted, which covers most optional personal data. Today they have to restructure their model or invent a sentinel.

Proposal

Allow *T for a sealed scalar field, and give the developer two ways to store nil:

• Sealed null. nil is encrypted. The column holds a normal ciphertext (or EQL payload) whose sealed plaintext is a null marker. A reader of the database cannot tell an absent value from a present one. This works only for fields with no index terms. An order term (ORE) or a match filter has no meaningful value for "absent", and an equality term (HMAC) of a fixed null marker would make every absent row equal to every other, which reveals the same thing SQL NULL does.
• SQL NULL. nil never reaches the engine. The column is NULL, with no ciphertext and no terms. Whether the value is present can be seen. This matches the TypeScript SDK, which passes null through as null (Runtime null guards stripped from stack encryption operations cause silent ciphertext for null input #494).

Steps:

1. Accept *T in stashgen where T is one of today's sealable scalars, for encrypt, encrypt_into and encrypt,index=… fields. Keep refusing **T, pointers to slices, maps or structs, and a pointer context field or identity field. Each refusal names the field.
2. Add a per-field tag that chooses how absence is stored, e.g. null=sealed or null=sql (name to be settled). Fixed-width and variable-width types are handled separately.
   • Fixed-width types (bool, integers, floats, encrypt.Date, time.Time, encrypt.Decimal, encrypt.Int128, encrypt.Uint128), with no index (storage-only EQL types and plain encrypt): the default is null=sealed, and null=sql opts out.
   • Fixed-width types with an index: null=sql for now. A typed null could later get its own order and equality terms (see aead: a sealed null is shorter than the value it replaces, so ciphertext length shows which optional values are absent — add a typed null padded to its kind's width vitaminc#493, item 5), which would let indexed fields use null=sealed too.
   • Variable-width types (strings, []byte, and types defined over them): null=sql only. A present value's length already shows. Hiding absence there needs a padding length chosen to match real values, and a match index (Bloom filter) for an absent value would be empty unless it were filled with noise. That is a separate, harder problem and out of scope here.
   • Open question: where null=sql is the only option, should the tag be required, so that choosing a visible NULL is deliberate, or should it be the default?
3. Make a sealed null the same length as a present value. This is a hard requirement, not an optimisation, and it is blocked on aead: a sealed null is shorter than the value it replaces, so ciphertext length shows which optional values are absent — add a typed null padded to its kind's width vitaminc#493:
   • A null leaf is 1 byte of plaintext ([0x00]). A date leaf is 5 bytes (tag 0x12 plus an i32).
   • AES-GCM keeps the plaintext's length, so the two ciphertexts differ in length by 4 bytes. Anyone reading the table could pick out the absent rows, and the sealed null would hide nothing.
   • vitaminc's verify_absent accepts exactly [0x00]. aead: a sealed null is shorter than the value it replaces, so ciphertext length shows which optional values are absent — add a typed null padded to its kind's width vitaminc#493 proposes a typed null leaf padded to its kind's width.
   • bool is the exception: its leaf is already 1 byte.
4. Engine and wire changes.
   • Mark a plan field as optional.
   • Carry a null through the Go record model.
   • Have the guest seal a padded null for null=sealed, and skip the field for null=sql.
   • For EQL types, accept and produce a null leaf in the storage-only domains' c.
5. Generated code.
   • null=sql fields get pointer outputs (*eql.Date). database/sql scans NULL into a nil pointer and leaves eql.Date.Scan's refusal of NULL in place for required fields.
   • null=sealed fields keep non-pointer outputs, since the column is never NULL.
   • Decrypt gives nil for either form.
   • A null leaf found in a non-pointer field is a decrypt error that names the field, not a zero value.
6. Tests:
   • The test from test(stashgen): reproduce nullable DOB generation failure #1293. It should skip when the guest is not built, as TestRunAsksTheEmbeddedEngine does.
   • Refusal cases for the forms that stay unsupported.
   • An Encrypt and Decrypt round trip over one batch that mixes nil and present values, in both modes.
   • A test that a sealed null and a present value of the same type have equal ciphertext lengths.
   • A PostgreSQL test that writes and reads both forms.
7. Docs. Update encrypt/values.go and encrypt/ciphertext.go, and say plainly that null=sql shows whether a value is present.

Interim step that could ship first: steps 1 and 5 with null=sql only. While sealed nulls are unavailable, a nullable field without the tag is refused, with a message telling the developer to write null=sql. This needs no vitaminc change. It never silently changes later: when sealed nulls land, they become the default only for fields that never had a tag, and those fields were refused until then.

Evidence

Verified:

• The refusal is at languages/golang/stashgen/read.go:697, and KindPointer is not in Kind.Scalar() (stashgen/declaration.go:268).
• split requires every field (encrypt/gensupport/codec.go, "the generated Source gave no field").
• eql.scan errors on nil (encrypt/eql/eql.go:97). Value() already writes NULL for an empty value.
• Leaf widths, from vitaminc-aead 0.6.1 src/tags.rs: NULL = 0x00, and a DATE = 0x12 leaf is an i32 of days. vitaminc-encrypt 0.6.1 verify_absent requires the plaintext to be exactly the null leaf (cipher_static.rs, verify_absent_requires_the_null_leaf).
• Stack Encrypt already treats an absent value as a sealed null leaf for Option<T> (packages/stack-encrypt/src/cipher.rs, "An absent value (None) is a null leaf like any other scalar").

Not yet verified:

• Whether the EQL v4 storage-only domain CHECKs accept a c whose sealed plaintext is a null leaf. Likely, because c is opaque to SQL.
• Whether the guest's dynamic seal path refuses a null for a field that declares a kind. scalar_kind returns None for a null, which suggests it does.

Relationship to other work

• aead: a sealed null is shorter than the value it replaces, so ciphertext length shows which optional values are absent — add a typed null padded to its kind's width vitaminc#493: the typed, padded null leaf that the sealed-null mode needs. The interim null=sql step does not depend on it.
• test(stashgen): reproduce nullable DOB generation failure #1293: the reproduction. It will be closed by the PR that fixes this issue, and its test folded in.
• Runtime null guards stripped from stack encryption operations cause silent ciphertext for null input #494: the TypeScript SDK's null handling (null in, null out). The null=sql mode matches it. Sealed nulls would be new to the Go SDK. Whether the TypeScript SDK should offer them too is a separate decision.