From 65ece530934d367e30ea99d1f149f76117191739 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 24 Oct 2026 12:16:13 +1100 Subject: [PATCH 1/3] docs(plans): the json_tree capability in the Go SDK Plans how a Go program declares, stores and queries an encrypted JSON document (EQL v4 json_tree). Most of the work is in Rust: the engine refuses the JSON index in a data plan, and the family is missing from the EQL catalog, so Go has no types to generate. Five steps, one issue each, with the decisions the Go surface needs. Refs #1347 Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3 --- docs/plans/2026-10-24-go-json-tree.md | 253 ++++++++++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 docs/plans/2026-10-24-go-json-tree.md diff --git a/docs/plans/2026-10-24-go-json-tree.md b/docs/plans/2026-10-24-go-json-tree.md new file mode 100644 index 000000000..b420edc46 --- /dev/null +++ b/docs/plans/2026-10-24-go-json-tree.md @@ -0,0 +1,253 @@ +# The json_tree capability in the Go SDK + +Status: plan, not a spec. Tracking issue: #1347. Written against the json_tree stack +(#1288 → #1325 → #1329 → #1336, with #1345 on top) and `next` (#1229). +Where this disagrees with the code, the code wins. + +## Background + +`json_tree` is the EQL v4 domain family for an encrypted JSON document. The +Rust engine (`packages/stack-encrypt/src/json/`) seals a +`serde_json::Value` as one entry per node, plus a containment projection +`q`. The SQL (`packages/eql/src/v3/json_tree/`, v4 only) answers: + +- containment `@>` (GIN over `eql_v4.json_tree_query(col)`) and `<@`; +- scalar equality at a path, sent as a containment needle; +- `->` / `->>` extraction by an encrypted path selector; +- `< <= > >=`, ranges and `ORDER BY eql_v4.ord_term(col -> sel)`, at paths + declared `ordered` (CLLW OPE); +- walks over a subtree by token (`json_tree_subtree`, `_children`), with + a fallback through the sealed root. + +The design is `docs/plans/2026-10-17-json-tree-index.md`, and the leakage +write-up is `packages/eql/docs/reference/json-tree-leakage.md`. + +Today none of this reaches Go, and that is deliberate. The design doc puts +Go out of scope (lines 336-338, 377). The pieces that exist are +placeholders that refuse: + +- `encrypt.JSON()` and `encrypt.JSONTerm` (`term.go:73-78, 157`); +- stashgen's `index=json`, which parses and is then refused + (`engine.go:265-268`); +- the `eql.SteVecDocument` and `eql.JSON` rows, marked not producible + (`eql_gen.go:61-62`). + +## Constraints + +The design principles (`docs/sdk-design-principles.md`, ADR-0007 and +ADR-0008) settle most of the shape before any Go is written. + +1. **The engine does the work (G1, ADR-0007).** Go reaches the engine only + through a data plan run in the WASI guest. The engine refuses the JSON + index in a data plan today: `IndexSpec::Json` crosses the wire and is + refused with `Reason::JsonIndexPending` (`dynamic/kind.rs:106`). The + json_tree family is also missing from the EQL catalog, from `v4::all()` + and from the target table, so `eql-codegen` emits no Go type for it. + **Most of the work is in Rust, and it comes first.** +2. **One EQL column per field (Go-11).** A json_tree field is an + `encrypt_into=` field. It binds and scans as a generated + `eql.JSONTree`, which holds the EQL JSON exactly as PostgreSQL stores + it. Go never builds or reads that JSON. +3. **Checks run at the earliest stage (G3, Go-1).** Options and ordered + paths belong in the struct tag, so they are checked at `go generate` by + asking the engine (`se_plan_check`). Go keeps no copy of the path + grammar. +4. **Named concrete types (Go-2).** Every generated query method returns a + named `eql.*Query` type, never `[]byte` or `any`. +5. **Nulls.** A null json_tree document cannot be sealed at a value's + length, because documents have no single length. So a json_tree field + needs `null=sql`, exactly as text does. stashgen's `nullMode` already + gives this answer once the type reports that it is not fixed-width. + +## The proposed Go surface + +### Declaration + +```go +type Patient struct { + _ struct{} `stash:"context=patients"` + ID int64 `stash:"id,passthrough"` + Profile json.RawMessage `stash:"profile,encrypt_into=JSONTree(ordered=$.age|$.visits[*].date)"` + Notes *Notes `stash:"notes,encrypt_into=JSONTree,null=sql"` +} +``` + +- `encrypt_into=JSONTree` takes the engine's `JsonOptions`: + - `ordered=` is a `|`-separated list of paths; + - `sealed_root=false` opts out of the sealed root, which defaults to on, + as in the engine. + + stashgen hands these to the engine unparsed, as the wire form of the + `json` index (`{"json":{"sealed_root":…,"ordered":[…]}}`). The engine + parses and checks them, so a bad path or a position in an ordered path + (`JsonOrderingAtPosition`) fails `go generate`. +- **The field's Go type** (decision 1) is `json.RawMessage`, `[]byte`, or + any type `encoding/json` marshals (a struct, `map[string]any`, `any`). +- **The value crosses the ABI as JSON text** (decision 2): a new record + kind `json` whose source value is UTF-8 JSON. The engine parses it with + `serde_json`, so number canonicalisation (`2 == 2.0`) and key handling + happen in Rust, identically for every language (G1). + + On decrypt the engine returns JSON text, and the generated code + unmarshals it into the field's type. `encoding/json` is already how + opaque structs cross (`gensupport/codec.go:345`), so this adds no new + reflection to the SDK. + +### Writes and reads + +These are unchanged in shape. `patients.Encrypt(ctx, cipher, []Patient)` +returns `[]EncryptedPatient`, with `Profile eql.JSONTree`; +`patients.Decrypt` reverses it. The field's own +`Fields.Profile.Encrypt(ctx, c, v)` returns `eql.JSONTree`. + +### Queries + +These are generated per field from what the engine reports +(`se_operators`), like every other query method: + +| Generated method | Operand | Returns | SQL | +|---|---|---|---| +| `Contains(ctx, c, needle)` | a JSON document (decision 3) | `eql.JSONTreeQuery` | `profile @> $1::eql_v4.query_json_tree` | +| `ContainedBy(ctx, c, doc)` | a JSON document | `eql.JSONTreeQuery` | `profile <@ $1::eql_v4.query_json_tree` | +| `EqAt(ctx, c, path, v)` | `string` path, scalar `v` | `eql.JSONTreeQuery` | `profile @> $1::eql_v4.query_json_tree` | +| `At.Age.Lt/Lte/Gt/Gte(ctx, c, v)` | the scalar | `eql.JSONTreeLeafQuery` | `(profile -> $2::text) < $1::eql_v4.query_json_tree_leaf` | +| `At.Age.Between(ctx, c, lo, hi)` | two scalars | `encrypt.Range[eql.JSONTreeLeafQuery]` | two predicates | +| `At.Age.Selector(ctx, c)` | none | `eql.JSONTreeSelector` (text) | `profile -> $1::text`, `ORDER BY eql_v4.ord_term(profile -> $1::text)` | +| `Selector(ctx, c, path)` | `string` path | `eql.JSONTreeSelector` | `->` / `->>` on any path | + +Ordered paths get a generated accessor under `At` (decision 4), named from +the path (`$.age` becomes `Age`, or the name in an `as=`), so a typo in an +ordered path fails to compile. Equality at a path and extraction take a +runtime path, because they work at any path. The engine refuses a bad path +with `ErrTerm` and its reason, and never echoes the path text. + +Refusals come from the engine, with its reasons: + +- `JsonValueNotScalar`, for equality with a container; +- `JsonLeafNotOrdered`, for an ordering value that is not a number or + string; +- `JsonPathNotOrdered`; +- `JsonTooDeep`, for nesting deeper than 127. + +### Extraction and walks (phase 5, decision 5) + +- **Decrypting an extracted node.** A `profile -> sel` result scans into + `eql.JSONTreeEntry`. `Fields.Profile.OpenEntry(ctx, c, []eql.JSONTreeEntry)` + returns the scalars as JSON text, and a container is refused, as the + engine refuses it (`OpenedType`). +- **Subtree walks.** `Fields.Profile.SubtreeKey(ctx, c, path)` returns + `eql.JSONTreeSubtreeKey`, which binds as the token and gates for + `json_tree_subtree(_at)`. `key.Open(ctx, c, rows)` then rebuilds the + value, including the fallback through the sealed root. + + This needs a new guest entry point, because the key's path is not on the + wire and the open has to see it. It is the least settled part, so it + ships last. + +## Work, in PR-sized steps + +Each step is one PR closing one issue. Steps 1-3 are Rust and unblock +everything after them. + +1. **Catalog and Go types (eql-bindings, eql-codegen), #1348.** + - Add the json_tree family to the catalog, to `v4::all()` and to the + target table. `JSONTree`, `JSONTreeQuery`, `JSONTreeLeafQuery` and + `JSONTreeEntry` then generate into `eql_gen.go`, with a not-producible + reason that points at step 2. `JSONTreeSelector` is a text type, so + check whether the generator can emit it or it needs hand-writing in + `encrypt/eql`. + - Report the family as not fixed-width, so stashgen requires + `null=sql`. + - Gates: `types:check`, `go_eql_parity`, the codegen tests. +2. **The data-plan form of the JSON index (stack-encrypt `dynamic`), #1349.** + - Admit `IndexSpec::Json` with its options. + - Accept the `json` source kind as JSON text, seal it to a + `JsonDocument` through the same `Layout` the typed path uses, and open + it back to JSON text. + - Teach the guest's `EqlTargets` the family. `Producible` flips to true. + - Tests: + - a dynamic-versus-typed parity test, so the same document gives the + same ciphertext under the same key and the same terms; + - the mutants gate on the diff. +3. **Query by operator for json_tree (stack-encrypt, guest), #1350.** + - Add `contains` and `contained_by`. + - Add the path operators, taking a `{path, value}` input: + `eq_at`, `lt/lte/gt/gte/between` at a path, and `selector`. + - `se_operators` reports them per field, from the field's options. + - `se_query_term` returns the query twin's JSON and domain, as for other + EQL types. + - Tests: Rust parity between the dynamic and typed paths. +4. **Go: stashgen, gensupport, the checker and the record model, #1351.** + - Add the `JSONTree(...)` options on `encrypt_into`, a `json` `Kind`, + the operator names in `parseOperatorName`, and `json` in + `parseOperatorIndex`. + - Generate `Contains`, `ContainedBy`, `EqAt`, `Selector` and the `At.*` + accessors. + - Retire the `encrypt.JSON()` / `JSONTerm` / `index=json` placeholders, + which were the SteVec shape, or repoint them, so that a single + `json` index name means json_tree. This needs a decision: see Open + questions. + - Tests: + - stashgen golden output; + - refusal tests (bad path, `null=sealed`, a position in an ordered + path); + - a fixture test that the Go-sealed document and terms equal the + Rust-sealed ones under the deterministic guest; + - a live `eql_postgres_test.go` case: round trip, `@>`, `<@`, + equality at a path, ordering and `ORDER BY` at an ordered path, and + `->`. + - Docs: the Go README, the stashgen README, `doc.go` usage, the + CHANGELOG. +5. **Extraction and walks in Go, #1352.** `OpenEntry`, `SubtreeKey` and `Open`, + behind a new guest export. Live tests for walks, the sealed-root + fallback and a node that is not a scalar. + +The semantic oracle stays in Rust: #1345 holds encrypted containment to +`jsonb`. The Go tests check wiring and byte parity, not containment +semantics again. + +## Decisions needed + +1. **Which Go types a json_tree field may have.** Recommended: + `json.RawMessage`, `[]byte`, and anything `encoding/json` marshals. + - Alternative: `json.RawMessage` only. It is the strictest option and + involves no reflection, but users would unmarshal by hand. +2. **How the value crosses the ABI.** Recommended: JSON text, so the + engine parses it and the bytes match every language. + - Alternative: a vcvalue map/sequence tree. vcffi can carry one, but + Go's `encoding/json` decodes numbers to `float64`, which loses + integers above 2^53 before the engine sees them. +3. **The type of a containment needle.** Recommended: `any` marshalled + with `encoding/json`, documented. A needle built from the field's + struct type carries zero values (`"age":0`), which silently narrows the + match. + - Alternative: the field's type `T`, which is concrete (Go-2) but has + that trap whenever `T` is a struct without `omitempty`. +4. **How ordered paths are named.** Recommended: a generated accessor per + ordered path under `At`, so a typo fails to compile. + - Alternative: a runtime path string for everything, with fewer + generated methods and errors found at run time. +5. **Scope of the first release.** Recommended: steps 1-4, which cover + write, read, containment, equality at a path, ordering and + `->` selectors. Walks and opening entries (step 5) follow. + +## Open questions + +- **`index=json` and `encrypt.JSON()`.** They were reserved for the EQL v3 + SteVec index, which is also `"json"` in `checker.go:179`. Is json_tree + the only JSON index Go will ever offer? If so, delete the placeholders. + If not, give json_tree its own name. +- **"Query building in Go"** is listed as not yet decided in the + principles (line 213). The `At.*` accessors and the runtime-path methods + are a first answer, and it should be recorded there. +- **The design doc's open items reach Go:** + - The client does not yet check that a returned entry is the one at the + path asked for (DOC:495). + - The walk budget's default (DOC:391). + - The stored entry order (DOC:494). + + None blocks steps 1-4. Step 5 should wait for the selector check. +- **Ordering.** The json_tree stack is 101 commits behind `next`, which + carries the Go `null=` work, `eql_postgres_test.go` and the + `eql-postgres` CI job. Step 4 needs both, so the stack lands on `next` + first, or step 4 rebases onto it. From 42c38f4e79e40606bd7ba39564de726a0b9757db Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 22:55:58 +1100 Subject: [PATCH 2/3] docs: revise Go JSON dynamic contract plan --- docs/plans/2026-10-24-go-json-tree.md | 629 +++++++++++------- .../2026-10-25-go-json-engine-contract.md | 206 ++++++ 2 files changed, 586 insertions(+), 249 deletions(-) create mode 100644 docs/plans/2026-10-25-go-json-engine-contract.md diff --git a/docs/plans/2026-10-24-go-json-tree.md b/docs/plans/2026-10-24-go-json-tree.md index b420edc46..38f127a0a 100644 --- a/docs/plans/2026-10-24-go-json-tree.md +++ b/docs/plans/2026-10-24-go-json-tree.md @@ -1,253 +1,384 @@ -# The json_tree capability in the Go SDK - -Status: plan, not a spec. Tracking issue: #1347. Written against the json_tree stack -(#1288 → #1325 → #1329 → #1336, with #1345 on top) and `next` (#1229). -Where this disagrees with the code, the code wins. - -## Background - -`json_tree` is the EQL v4 domain family for an encrypted JSON document. The -Rust engine (`packages/stack-encrypt/src/json/`) seals a -`serde_json::Value` as one entry per node, plus a containment projection -`q`. The SQL (`packages/eql/src/v3/json_tree/`, v4 only) answers: - -- containment `@>` (GIN over `eql_v4.json_tree_query(col)`) and `<@`; -- scalar equality at a path, sent as a containment needle; -- `->` / `->>` extraction by an encrypted path selector; -- `< <= > >=`, ranges and `ORDER BY eql_v4.ord_term(col -> sel)`, at paths - declared `ordered` (CLLW OPE); -- walks over a subtree by token (`json_tree_subtree`, `_children`), with - a fallback through the sealed root. - -The design is `docs/plans/2026-10-17-json-tree-index.md`, and the leakage -write-up is `packages/eql/docs/reference/json-tree-leakage.md`. - -Today none of this reaches Go, and that is deliberate. The design doc puts -Go out of scope (lines 336-338, 377). The pieces that exist are -placeholders that refuse: - -- `encrypt.JSON()` and `encrypt.JSONTerm` (`term.go:73-78, 157`); -- stashgen's `index=json`, which parses and is then refused - (`engine.go:265-268`); -- the `eql.SteVecDocument` and `eql.JSON` rows, marked not producible - (`eql_gen.go:61-62`). - -## Constraints - -The design principles (`docs/sdk-design-principles.md`, ADR-0007 and -ADR-0008) settle most of the shape before any Go is written. - -1. **The engine does the work (G1, ADR-0007).** Go reaches the engine only - through a data plan run in the WASI guest. The engine refuses the JSON - index in a data plan today: `IndexSpec::Json` crosses the wire and is - refused with `Reason::JsonIndexPending` (`dynamic/kind.rs:106`). The - json_tree family is also missing from the EQL catalog, from `v4::all()` - and from the target table, so `eql-codegen` emits no Go type for it. - **Most of the work is in Rust, and it comes first.** -2. **One EQL column per field (Go-11).** A json_tree field is an - `encrypt_into=` field. It binds and scans as a generated - `eql.JSONTree`, which holds the EQL JSON exactly as PostgreSQL stores - it. Go never builds or reads that JSON. -3. **Checks run at the earliest stage (G3, Go-1).** Options and ordered - paths belong in the struct tag, so they are checked at `go generate` by - asking the engine (`se_plan_check`). Go keeps no copy of the path - grammar. -4. **Named concrete types (Go-2).** Every generated query method returns a - named `eql.*Query` type, never `[]byte` or `any`. -5. **Nulls.** A null json_tree document cannot be sealed at a value's - length, because documents have no single length. So a json_tree field - needs `null=sql`, exactly as text does. stashgen's `nullMode` already - gives this answer once the type reports that it is not fixed-width. - -## The proposed Go surface - -### Declaration +# Go JSON: revised dynamic contract and customer implementation plan + +Date: 2026-10-25. Status: implementation direction agreed in API review; +contract spellings and examples remain proposals, not implemented API. +Tracking issue: [#1347](https://github.com/cipherstash/stack/issues/1347). +This plan under [#1353](https://github.com/cipherstash/stack/pull/1353) uses +merged `next` at `ebdb5de0a12bbec06842851eece283fb7faea30b` and the +[API redesign](https://github.com/cipherstash/stack/blob/c5efc48837d3cb48bc7e168f780fd616a4f4845e/docs/plans/2026-10-24-stack-encrypt-api-plan.md) +tracked in [#1365](https://github.com/cipherstash/stack/pull/1365). +The [engine findings](2026-10-25-go-json-engine-contract.md) cite the source +for the current behavior described below. The examples are proposed contract +shapes, not runnable instructions for the current SDK. + +## Recommendation + +Deliver Go JSON through the existing Rust engine and EQL guest. Implement +configured target definitions, operation-specific results, and request-bound +reads now. These are the parts of the API redesign that affect the customer +contract. The public Rust traits, derives, method renames and crate moves +can follow without changing this contract. + +Keep the full JSON operation inventory in the first release, including +extraction, subtree reads, children and counts. Opening extracted values +is part of that release, alongside full-document writes and queries. The +merged engine already provides the main read machinery; adapting it safely is part of the work. + +Adopt the forthcoming Vitamin C release separately for opaque blobs. Its +storage profile does not replace JSON tree's custom encrypted nodes or EQL +envelope. Freeze and test those existing JSON formats independently. + +This recommendation makes no short-ngram mechanism decision. JSON's result +contract must be extensible without pretending that ngram Boolean-query +requirements have already been settled. + +## Changes from the first-cut plan + +| Original proposal | Revised requirement | +| --- | --- | +| JSON options travel as a special index configuration | A registered target owns typed options, validation and lowering; the same resolved definition serves writes, queries and reads | +| One query twin/domain per target; one or two byte terms | Describe each operation's input and output shape; containment, ordering, selector and subtree preparation have different results | +| `OpenEntry(cipher, rows)` without a request | A prepared selector/read object opens its own rows; expected path, definition and keyset come from the caller, never the database | +| Subtree token plus gates, `_subtree_at` | Current token lists, self tags, client wildcard expansion and probes; `json_tree_subtree(document, tokens, budget)` | +| Selector checks and number correctness are pending | Checks have landed; #1366 fixes #1364. Preserve their tests through the binding | +| Walks ship later | All required operations ship in the customer release | +| Nullable JSON field | SQL NULL, present JSON null and missing selection are distinct states | +| JSON implicitly means the old SteVec index | JSON is a plaintext kind; the target explicitly identifies the EQL JSON-tree representation | +| Reuse current EQL context plumbing | Current EQL targets are column-only and reject context extensions; do not advertise the redesign's row binding as implemented | + +The catalog, guest dispatch, generator and Rust parity work in #1353 remain +valid. The proposal needs revised contracts rather than a second SDK or +crypto implementation. Its five tracking issues remain useful work items. + +## 1. One configured target definition + +Make the public interoperability unit a versioned definition, lowered to +internal plans. It contains field identity, source kind, target, resolved +options and declared context roles. Bind requests to a field of that +definition. Do not pass another options object to a query or decrypt call. + +Proposed minimal JSON authoring form, for the existing column-only profile: + +```json +{ + "version": 1, + "fields": { + "profile": { + "type": "json", + "context": ["users", "profile"], + "target": { + "name": "EncryptedJson", + "options": {"sealed_root": true, "ordered": ["$.salary"]} + } + } + } +} +``` + +`EncryptedJson` is the recommended role-based target name, mapping to the +current Rust `JsonTreeDocument` and SQL `public.eql_v4_json_tree`. Go spells +the generated ciphertext type `eql.EncryptedJSON`. These names are proposals; +the existing SQL domain does not need a rename. Target identifiers are +contract identifiers, not names obtained from Rust type reflection. + +A string target remains shorthand for default options. Unknown keys, +unknown targets, unsupported versions and unsupported context capabilities +fail closed. Normalize defaults once. Resolve options into the same native +`JsonOptions` used by typed Rust, including its path validation. A new +configured definition is a new immutable instance, not mutation of one used +by an in-flight query. + +Extend discovery with option schema/profile identity and per-operation +input, result and capability metadata. The existing target descriptor's +single `query`/`query_sql_domain` pair cannot describe JSON accurately. +Generate schemas and Go types from the maintained registry where possible; +keep semantic validation in Rust. Do not add an independent Go path parser. + +The guest should cache validated definitions under their complete resolved +content and contract version. Generated field accessors share that immutable +content. Guest-local cache identifiers remain internal and can be recreated; +they must not become SQL values or customer persistence formats. + +The SDK keeps its idiomatic generated `Encrypt`, `Decrypt` and `Fields` +entry points. It does not expose a Rust plan or require a Go `.using()` +method. Both tags and the existing generator policy path lower to this same +definition. Simple target options can use the tag grammar; complex quoted +paths need a structured options form, not a naive `|` split. Validate both +through the engine at generation time. + +## 2. Explicit JSON values and three separate codecs + +Use UTF-8 JSON text for plaintext documents and JSON query inputs at the +host/guest boundary. In Go, use `json.RawMessage` as the fundamental value. +A typed struct adapter may explicitly call `encoding/json` and decode into +a declared type. Never decode first into `map[string]any` with default +`float64` numbers just to send the tree to Rust. + +Do not implicitly treat an ordinary `[]byte` as JSON: Go's JSON encoder +normally treats it as a byte string. Require an explicit `json.RawMessage` +conversion for raw JSON bytes. Containment accepts an explicit JSON needle, +not necessarily the field's full struct type, whose zero-valued fields can +accidentally narrow the query. + +Across the binary ABI, the declared `json` input is a UTF-8 text/byte payload +in the ordinary transport frame. In JSON authoring/request examples below, +`json` is a string carrying that text. Define this mapping explicitly; +a JSON document's `null` is not the frame's absent-field/null marker. + +| Boundary | Representation | +| --- | --- | +| Dynamic definitions, requests and responses | Versioned control schema carried by the existing Vitamin C transport frame | +| Searchable JSON plaintext | UTF-8 JSON parsed by Rust into the existing JSON value model | +| Stored searchable document | Existing EQL v4 JSON-tree envelope, header, encrypted nodes and search terms | +| Stored opaque whole struct | Vitamin C versioned storage blob from #520/#526, sealed as one value | + +Use direct typed EQL/engine conversion when implementing the adapter; remove +the internal serialize-to-Value/deserialize bridge without changing stored +bytes. This is distinct from parsing user JSON text at ingress. + +Pin the numeric model: signed/unsigned 64-bit integers and finite binary64 +values, not arbitrary-precision PostgreSQL numeric. Preserve integer tokens +without first casting to float. Add ingress checks for unsupported integer +literals instead of accepting a parser's lossy fallback. Fractional/exponent +numbers follow the documented binary64 interpretation; JSON text, key order +and decimal spelling are not promised to round trip byte for byte. The +existing `float_roundtrip` fix must remain enabled. Separately document that +JSON numeric ordering converts to `f64`, so large distinct integers can tie. + +Specify duplicate-member handling and invalid Unicode rather than inheriting +an accidental Go/Rust difference. Recommend rejecting duplicate object keys +and invalid Unicode at JSON ingress in Rust, before key I/O. This validation +is additional adapter work, not a claim about the current parser. A typed +Go encoder may already have transformed the caller's input; its explicit +adapter contract must describe that transformation. + +For an optional document, a nil `*json.RawMessage` is SQL NULL. A non-nil +pointer containing `null` is a present encrypted JSON null. A nonoptional +`json.RawMessage` is a required document; empty bytes are invalid JSON, not +an absent document. `null=sql` is required for nullable fields, not every +JSON field. Length-padded sealed null is not supported for this target. + +## 3. Operation-specific requests and results + +Use operator names with distinct operand schemas, all bound to the same +field definition. For example: + +```json +{"operation":"contains","field":"profile","input":{"json":"{\"active\":true}"}} +``` + +```json +{"operation":"eq_at","field":"profile","input":{"path":"$.name","json":"\"Alice\""}} +``` + +```json +{"operation":"subtree","field":"profile","input":{"path":"$.visits[*]","wildcard_bound":64}} +``` + +These fragments are operation bodies; the request envelope additionally +identifies protocol version, definition and execution/keyset inputs. + +Replace the assumption that every operation yields one or two anonymous +byte strings with a tagged result model. Predicate operands, ordered ranges, +selectors and prepared reads have separate variants and named Go types. +Each operand states its wire encoding and SQL parameter type. A selector +is text; subtree tokens are `text[]`; query envelopes are EQL JSON. A token +list is not a BETWEEN range or a generic Boolean OR expression. + +| Operation | Required Go/dynamic behavior | SQL use | +| --- | --- | --- | +| Encrypt/decrypt, including batches | Full document, both root policies, scalar/array/object/null | One `public.eql_v4_json_tree` column | +| Contains / ContainedBy | JSON needle and named containment query | `@>` / `<@` with `eql_v4.query_json_tree` | +| EqAt | Scalar JSON operand and validated selector-class path | Containment predicate; never ORE/OPE equality | +| LtAt/LteAt/GtAt/GteAt | Ordered path plus number/string | Extract entry, compare with `eql_v4.query_json_tree_leaf` | +| BetweenAt | Two ordered bounds plus selector | Inclusive lower/upper predicates | +| Selector | Prepared selection with named SQL selector | `->`, `->>`, first/all path query, path exists, sorting | +| Open selection | Original selection plus entries grouped by database document | Scalars; sealed root at `$`; ordinary container marker refused | +| Subtree | Prepared concrete or wildcard read with token list | `json_tree_subtree(document, tokens, budget)` | +| Open subtree | Original read plus grouped rows | Requested JSON only, including fallback | +| Children | Concrete node request and encrypted child entries | `json_tree_children(document, token)` | +| Open child scalars | Request-bound child validation in Rust | Additional adapter/helper work; never generic entry decryption | +| ChildCount | Concrete node token; nullable count | `json_tree_child_count(document, token)` | +| Stored-value comparisons | Same column/scope and compatible profile | Existing document containment and entry ordering operators | + +Generate ordered-path accessors from validated options, with explicit +aliases or a generation error on name collisions. Runtime path methods use +the same native validator. Reject positions for selector-class operations +including EqAt; those selectors erase `[n]` to `[*]`. Exact positional reads +use `subtree`. Wildcard equality is any-match; wildcard ordering through +`->` is first-match. Multiple independent EqAt predicates need not match the +same array element; use one containment needle for correlated conditions. + +For children, the SQL rows have depth one and are not complete subtrees. +Add a parent-bound opening helper that verifies child membership and opens +scalars; container children require a subtree read. Implement this in Rust +and reuse it from Go. Counts remain database observations: zero for a scalar +or empty container, NULL for absence or the sealed root. Neither counts nor +absence become authenticated merely because returned ciphertexts open. + +## 4. A read object retains the verification request + +Recommend an immutable host-owned read descriptor and guest rederivation. +Store the trusted definition/profile, resolved keyset identity, path, +wildcard bound and applicable scope inputs. Do not export PRF keys or +serialize a native `JsonSubtreeKey` expecting to recover its private state. +Its existing serialized token list is insufficient for opening checks. + +The SQL parameter is a separate value derived from the read. Opening sends +the original descriptor and untrusted rows to Rust, rederives the native +selector/subtree key and uses its existing verifier. Enforce definition, +keyset and scope agreement. Caller `context.Context` is passed per call, +never retained in the read. The current Go client has one guest instance; +rederivation also avoids imposing instance affinity on future execution. + +Proposed usage fragment, with database plumbing abbreviated deliberately: ```go -type Patient struct { - _ struct{} `stash:"context=patients"` - ID int64 `stash:"id,passthrough"` - Profile json.RawMessage `stash:"profile,encrypt_into=JSONTree(ordered=$.age|$.visits[*].date)"` - Notes *Notes `stash:"notes,encrypt_into=JSONTree,null=sql"` -} +read, err := users.Fields.Profile.Subtree(ctx, cipher, "$.visits[*]", + encrypt.JSONReadOptions{WildcardBound: 64}) +if err != nil { return err } + +// Bind read.Tokens() as PostgreSQL text[], using the driver's array adapter: +// SELECT e FROM eql_v4.json_tree_subtree($1::public.eql_v4_json_tree, +// $2::text[], $3::integer) AS e +// $1 is the stored document; $3 is the SQL walk budget, e.g. 64. + +selected, err := read.Open(ctx, cipher, encryptedRows) +if err != nil { return err } +// selected is an explicit list of JSON values, in document order. ``` -- `encrypt_into=JSONTree` takes the engine's `JsonOptions`: - - `ordered=` is a `|`-separated list of paths; - - `sealed_root=false` opts out of the sealed root, which defaults to on, - as in the engine. - - stashgen hands these to the engine unparsed, as the wire form of the - `json` index (`{"json":{"sealed_root":…,"ordered":[…]}}`). The engine - parses and checks them, so a bad path or a position in an ordered path - (`JsonOrderingAtPosition`) fails `go generate`. -- **The field's Go type** (decision 1) is `json.RawMessage`, `[]byte`, or - any type `encoding/json` marshals (a struct, `map[string]any`, `any`). -- **The value crosses the ABI as JSON text** (decision 2): a new record - kind `json` whose source value is UTF-8 JSON. The engine parses it with - `serde_json`, so number canonicalisation (`2 == 2.0`) and key handling - happen in Rust, identically for every language (G1). - - On decrypt the engine returns JSON text, and the generated code - unmarshals it into the field's type. `encoding/json` is already how - opaque structs cross (`gensupport/codec.go:345`), so this adds no new - reflection to the SDK. - -### Writes and reads - -These are unchanged in shape. `patients.Encrypt(ctx, cipher, []Patient)` -returns `[]EncryptedPatient`, with `Profile eql.JSONTree`; -`patients.Decrypt` reverses it. The field's own -`Fields.Profile.Encrypt(ctx, c, v)` returns `eql.JSONTree`. - -### Queries - -These are generated per field from what the engine reports -(`se_operators`), like every other query method: - -| Generated method | Operand | Returns | SQL | -|---|---|---|---| -| `Contains(ctx, c, needle)` | a JSON document (decision 3) | `eql.JSONTreeQuery` | `profile @> $1::eql_v4.query_json_tree` | -| `ContainedBy(ctx, c, doc)` | a JSON document | `eql.JSONTreeQuery` | `profile <@ $1::eql_v4.query_json_tree` | -| `EqAt(ctx, c, path, v)` | `string` path, scalar `v` | `eql.JSONTreeQuery` | `profile @> $1::eql_v4.query_json_tree` | -| `At.Age.Lt/Lte/Gt/Gte(ctx, c, v)` | the scalar | `eql.JSONTreeLeafQuery` | `(profile -> $2::text) < $1::eql_v4.query_json_tree_leaf` | -| `At.Age.Between(ctx, c, lo, hi)` | two scalars | `encrypt.Range[eql.JSONTreeLeafQuery]` | two predicates | -| `At.Age.Selector(ctx, c)` | none | `eql.JSONTreeSelector` (text) | `profile -> $1::text`, `ORDER BY eql_v4.ord_term(profile -> $1::text)` | -| `Selector(ctx, c, path)` | `string` path | `eql.JSONTreeSelector` | `->` / `->>` on any path | - -Ordered paths get a generated accessor under `At` (decision 4), named from -the path (`$.age` becomes `Age`, or the name in an `as=`), so a typo in an -ordered path fails to compile. Equality at a path and extraction take a -runtime path, because they work at any path. The engine refuses a bad path -with `ErrTerm` and its reason, and never echoes the path text. - -Refusals come from the engine, with its reasons: - -- `JsonValueNotScalar`, for equality with a container; -- `JsonLeafNotOrdered`, for an ordering value that is not a number or - string; -- `JsonPathNotOrdered`; -- `JsonTooDeep`, for nesting deeper than 127. - -### Extraction and walks (phase 5, decision 5) - -- **Decrypting an extracted node.** A `profile -> sel` result scans into - `eql.JSONTreeEntry`. `Fields.Profile.OpenEntry(ctx, c, []eql.JSONTreeEntry)` - returns the scalars as JSON text, and a container is refused, as the - engine refuses it (`OpenedType`). -- **Subtree walks.** `Fields.Profile.SubtreeKey(ctx, c, path)` returns - `eql.JSONTreeSubtreeKey`, which binds as the token and gates for - `json_tree_subtree(_at)`. `key.Open(ctx, c, rows)` then rebuilds the - value, including the fallback through the sealed root. - - This needs a new guest entry point, because the key's path is not on the - wire and the open has to see it. It is the least settled part, so it - ships last. - -## Work, in PR-sized steps - -Each step is one PR closing one issue. Steps 1-3 are Rust and unblock -everything after them. - -1. **Catalog and Go types (eql-bindings, eql-codegen), #1348.** - - Add the json_tree family to the catalog, to `v4::all()` and to the - target table. `JSONTree`, `JSONTreeQuery`, `JSONTreeLeafQuery` and - `JSONTreeEntry` then generate into `eql_gen.go`, with a not-producible - reason that points at step 2. `JSONTreeSelector` is a text type, so - check whether the generator can emit it or it needs hand-writing in - `encrypt/eql`. - - Report the family as not fixed-width, so stashgen requires - `null=sql`. - - Gates: `types:check`, `go_eql_parity`, the codegen tests. -2. **The data-plan form of the JSON index (stack-encrypt `dynamic`), #1349.** - - Admit `IndexSpec::Json` with its options. - - Accept the `json` source kind as JSON text, seal it to a - `JsonDocument` through the same `Layout` the typed path uses, and open - it back to JSON text. - - Teach the guest's `EqlTargets` the family. `Producible` flips to true. - - Tests: - - a dynamic-versus-typed parity test, so the same document gives the - same ciphertext under the same key and the same terms; - - the mutants gate on the diff. -3. **Query by operator for json_tree (stack-encrypt, guest), #1350.** - - Add `contains` and `contained_by`. - - Add the path operators, taking a `{path, value}` input: - `eq_at`, `lt/lte/gt/gte/between` at a path, and `selector`. - - `se_operators` reports them per field, from the field's options. - - `se_query_term` returns the query twin's JSON and domain, as for other - EQL types. - - Tests: Rust parity between the dynamic and typed paths. -4. **Go: stashgen, gensupport, the checker and the record model, #1351.** - - Add the `JSONTree(...)` options on `encrypt_into`, a `json` `Kind`, - the operator names in `parseOperatorName`, and `json` in - `parseOperatorIndex`. - - Generate `Contains`, `ContainedBy`, `EqAt`, `Selector` and the `At.*` - accessors. - - Retire the `encrypt.JSON()` / `JSONTerm` / `index=json` placeholders, - which were the SteVec shape, or repoint them, so that a single - `json` index name means json_tree. This needs a decision: see Open - questions. - - Tests: - - stashgen golden output; - - refusal tests (bad path, `null=sealed`, a position in an ordered - path); - - a fixture test that the Go-sealed document and terms equal the - Rust-sealed ones under the deterministic guest; - - a live `eql_postgres_test.go` case: round trip, `@>`, `<@`, - equality at a path, ordering and `ORDER BY` at an ordered path, and - `->`. - - Docs: the Go README, the stashgen README, `doc.go` usage, the - CHANGELOG. -5. **Extraction and walks in Go, #1352.** `OpenEntry`, `SubtreeKey` and `Open`, - behind a new guest export. Live tests for walks, the sealed-root - fallback and a node that is not a scalar. - -The semantic oracle stays in Rust: #1345 holds encrypted containment to -`jsonb`. The Go tests check wiring and byte parity, not containment -semantics again. - -## Decisions needed - -1. **Which Go types a json_tree field may have.** Recommended: - `json.RawMessage`, `[]byte`, and anything `encoding/json` marshals. - - Alternative: `json.RawMessage` only. It is the strictest option and - involves no reflection, but users would unmarshal by hand. -2. **How the value crosses the ABI.** Recommended: JSON text, so the - engine parses it and the bytes match every language. - - Alternative: a vcvalue map/sequence tree. vcffi can carry one, but - Go's `encoding/json` decodes numbers to `float64`, which loses - integers above 2^53 before the engine sees them. -3. **The type of a containment needle.** Recommended: `any` marshalled - with `encoding/json`, documented. A needle built from the field's - struct type carries zero values (`"age":0`), which silently narrows the - match. - - Alternative: the field's type `T`, which is concrete (Go-2) but has - that trap whenever `T` is a struct without `omitempty`. -4. **How ordered paths are named.** Recommended: a generated accessor per - ordered path under `At`, so a typo fails to compile. - - Alternative: a runtime path string for everything, with fewer - generated methods and errors found at run time. -5. **Scope of the first release.** Recommended: steps 1-4, which cover - write, read, containment, equality at a path, ordering and - `->` selectors. Walks and opening entries (step 5) follow. - -## Open questions - -- **`index=json` and `encrypt.JSON()`.** They were reserved for the EQL v3 - SteVec index, which is also `"json"` in `checker.go:179`. Is json_tree - the only JSON index Go will ever offer? If so, delete the placeholders. - If not, give json_tree its own name. -- **"Query building in Go"** is listed as not yet decided in the - principles (line 213). The `At.*` accessors and the runtime-path methods - are a first answer, and it should be recorded there. -- **The design doc's open items reach Go:** - - The client does not yet check that a returned entry is the one at the - path asked for (DOC:495). - - The walk budget's default (DOC:391). - - The stored entry order (DOC:494). - - None blocks steps 1-4. Step 5 should wait for the selector check. -- **Ordering.** The json_tree stack is 101 commits behind `next`, which - carries the Go `null=` work, `eql_postgres_test.go` and the - `eql-postgres` CI job. Step 4 needs both, so the stack lands on `next` - first, or step 4 rebases onto it. +Provide batch opening with groups keyed by the caller's database row +association. Never concatenate different documents' entries into one walk. +Go groups transport values; Rust still gathers and executes all key requests +in one logical batch. Do not promise one HTTP request for unlimited batches: +existing provider chunk limits and distinct keysets still apply. + +Use a singular result with `Found` plus raw JSON, and a plural result with a +JSON-value list. Present JSON null is `Found=true` with text `null`; no match +is `Found=false` or an empty plural list. A singular read with multiple +results fails rather than silently choosing one. SQL first-match selection +is a separately explicit operation. + +Name the two limits separately: + +- `wildcard_bound`: defaults to 64 per wildcard, with boundary probes; + total token expansion is capped at 8,192. A found boundary probe fails + with `JsonWildcardTruncated`; excessive expansion fails before key I/O. +- `walk_budget`: SQL defaults to 64; selects sealed-root fallback. Zero + disables fallback. With an unsealed root it does not cap response size. + +Always apply the original selection after sealed-root fallback. Do not +return the whole decrypted document from a subtree request. Preserve the +engine's rejection of forged, spliced and malformed returned walks. +Document the existing limits: the server can omit whole matches/probes, +and successful opening is not proof of result completeness or database row +identity. Do not expose a `complete=true` claim from this protocol. + +## 5. Context and provider boundary: freeze only implemented guarantees + +This is the main redesign issue that cannot be solved by Go wrappers. +JSON currently derives index context and ciphertext AAD from the same +context piece. The EQL layout accepts a column identifier; dynamic targets +reject extensions and context-field plans. + +Keep column identity, search scope, ciphertext binding and provider options +as distinct roles in the new contract. Validate source roles against the +trusted definition: a primary-key field cannot supply search scope; a +scope key is only used when explicitly selected. Keyset selection and +ZeroKMS lock policy remain execution inputs, not JSON target options. + +**Delivery recommendation:** implement the complete JSON operation surface +first under the existing column-and-keyset context profile. Advertise that +profile explicitly and reject additional scope/row-binding declarations +until their lowering is implemented. This does not claim the accepted +context redesign is complete or defer JSON operations. The customer has +not stated a requirement for new row-bound JSON profiles in this task. + +Before enabling row binding, split the native PRF and AEAD inputs and test +writes, all queries and all read paths together. Search uses column plus +scope; AEAD includes the additional row binding. Decrypt requests then need +the expected row values, including when SQL returns extracted entries. +Never append a row ID to today's shared context as a shortcut. + +Pin the current context derivation as part of the storage/search profile. +Introducing the separated profile must be explicit and retain the reader +and query behavior of released profiles; it must not reinterpret old data +or silently reindex it. Decide the profile identifier and its association +with a column/definition before release. Schema version, EQL envelope/header +version and Vitamin C blob version are different version boundaries. + +## 6. Specification and acceptance gates + +Publish the first supported subset of the dynamic interoperability spec +alongside this implementation: JSON Schema, normative semantics, transport +mapping and conformance vectors. Use the general target model, not a +JSON-only replacement protocol. Unsupported future features fail explicitly; +reserved fields are not accepted and ignored. + +The schema covers definitions, target options, operations, tagged results, +read descriptors, presence states and stable errors. The semantic contract +covers number normalization, path classes, contexts, limits and validation +before key I/O. Return machine-readable codes and safe locations, without +JSON values, paths or tokens in diagnostic text. Definition/field identifiers +can identify the failing operation without echoing user values. + +Customer release gates: + +1. Catalog, options and capability discovery match Rust and generated Go; + schema/semantic fixtures reject unknown options and unsupported profiles. +2. Go writes open in Rust and Rust writes open in Go, with identical search + terms under the same inputs. Use fixed randomness only in test fixtures; + cross-opening and SQL behavior are the production compatibility criteria. +3. All operation rows above run through the real guest and PostgreSQL. + Retain the engine's jsonb containment/read oracles; add Go binding coverage. +4. Test both root policies, nested arrays, quoted keys, scalar/null roots, + SQL NULL, absence, empty containers, large integers and the #1366 float. +5. Test wildcard N-1/N/N+1, nested expansion caps, budget zero/default, + fallback equivalence, malicious row substitutions, duplicates, mixed + documents, truncated subtrees and the documented omission limitations. +6. Test actual SQL driver encodings: `driver.Valuer`/`Scanner`, pgx arrays, + explicit casts, sqlc/GORM integration where the SDK promises support. +7. Freeze exact storage/header/search vectors and context derivation; + verify guest/engine/EQL bundle compatibility. Record the accepted leakage + profile. Future Rust/package changes must retain these fixtures. + +## Implementation order and issue amendments + +| Slice | Work | Existing issue to revise | +| --- | --- | --- | +| A. Contract and registry | Versioned configured targets, operation/result schemas, defaults, profile identity, catalog and generated types | #1348 plus contract portion of #1349 | +| B. Full-document path | JSON ingress/egress, typed target adapter, definition resolution, batched writes/decrypts | #1349 | +| C. Queries and reads in Rust/guest | JSON operators, selector preparation, token-list preparation, request-bound opening, child helper, stable errors | #1350 and #1352 together | +| D. Go surface | Tags/policies, checker, generated field methods, named SQL values, read objects and examples | #1351 | +| E. Customer release proof | Go/guest/PostgreSQL tests, conformance and storage vectors, supported-profile docs | #1347 acceptance checklist | + +Start with A and a thin B/C vertical slice: one configured JSON field, batch +round trip, containment and a request-bound subtree read through the guest. +That proves the difficult seams before expanding the generator. It is an +implementation milestone, not permission to release a reduced operation set. + +Coordinate opaque storage adoption with Vitamin C #520/#526 once reviewed +and released. Keep Rust facade/package refactors off this critical path. +Align tracking issue #1347 and work items #1348–#1352 with the slices above +before assigning implementation. Their external issue descriptions have not +yet been amended by this documentation change. + +## Evidence and verification performed + +Read #1353 and its six tracking issues; inspected merged engine, EQL SQL, +Go generator/runtime, guest dispatch and the API redesign documents. Read +Vitamin C #520/#526 descriptions to distinguish their storage deliverable +from Stack integration. The accompanying engine findings link immutable +source locations. Only documentation checks ran here; no new implementation, +Go/Rust build, live database test or performance claim is reported. + +Additional primary sources: + +- [SDK principles](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/docs/sdk-design-principles.md). +- [Dynamic target resolver and descriptor](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/target.rs). +- [Current Go query response](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/languages/golang/encrypt/query.go). +- [Guest target dispatch](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/languages/golang/encrypt/guest/src/targets.rs). +- [Current Go codec and opaque JSON storage](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/languages/golang/encrypt/gensupport/codec.go). +- [Vitamin C Rust storage profile #520](https://github.com/cipherstash/vitaminc/pull/520) + and [Go storage profile #526](https://github.com/cipherstash/vitaminc/pull/526). diff --git a/docs/plans/2026-10-25-go-json-engine-contract.md b/docs/plans/2026-10-25-go-json-engine-contract.md new file mode 100644 index 000000000..82ba99c30 --- /dev/null +++ b/docs/plans/2026-10-25-go-json-engine-contract.md @@ -0,0 +1,206 @@ +# Merged JSON engine: obligations for the Go contract + +Supporting evidence for the [Go JSON implementation plan](2026-10-24-go-json-tree.md). + +Research baseline: `origin/next` at +[`ebdb5de0a12bbec06842851eece283fb7faea30b`](https://github.com/cipherstash/stack/commit/ebdb5de0a12bbec06842851eece283fb7faea30b), +after JSON-tree #1363, leakage documentation/tests #1362 and float-roundtrip +fix #1366. This is source analysis, not a rerun of those changes' tests. +The Go API recommendations below are proposals; the named Rust/SQL behavior +is implemented at that baseline. No ngram decisions are made here. + +## What the merged implementation actually provides + +| Capability | Engine / SQL contract | Go consequence | +| --- | --- | --- | +| Store and open a document | `JsonDocument`; EQL `JsonTreeDocument` adds the version and column envelope | One document field is one JSON value, including scalar roots, arrays and JSON null; do not restrict it to a Go object/map | +| Containment | `JsonQuery`, exact element-nested projection; SQL `@>` and `<@` | Keep operand direction explicit; array-of-object containment must not combine predicates from different elements | +| Equality at a path | `AtPath` constructs a scalar containment needle | Use this equality capability, never infer equality from the ordering term | +| Ordering at a path | `JsonLeafQuery`; `<`, `<=`, `>`, `>=`, inclusive `Between`; only number/string leaves at configured paths | Configuration belongs to the shared target definition used for writes and queries | +| Selector extraction | `JsonSelector`; SQL `->`, `->>`, path query, first and exists | Selector queries collapse positions to wildcards; first means first matching selector, not a named array position | +| Subtree read | `JsonSubtreeKey`; SQL `json_tree_subtree(document, tokens, budget)` | Preserve the trusted read request through to opening its rows | +| Children / count | SQL `json_tree_children(document, token)` and `json_tree_child_count(document, token)` | These take one node token, not an arbitrary wildcard token list; returned child entries are not complete subtrees | + +Sources: [engine query types and operators](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/query.rs), +[SQL functions](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/functions.sql), +[SQL operators](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/operators.sql). + +`sealed_root` defaults to true; `ordered` defaults to empty. Ordered paths +may include wildcards, but may not name a position such as `$.scores[2]`. +A query at an unordered path fails before loading keys. Ordering through a +wildcard selector compares the first matching entry; it is not an existential +or universal comparison over all elements. Ordering numbers uses `f64`, so +different large integers can tie; text ordering folds case and accents. +These are storage/search semantics, not Go conversion choices. +[Options](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/mod.rs), +[ordering derivation](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs), +[documented ordering profile](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/docs/reference/json-tree-leakage.md). + +## Replace the old read recipe + +The revised design derives tokens directly from paths, stores a self tag +`x` on every entry, and expands wildcards on the client. A subtree request +sends an ordered token list (`{"t":["…"]}`), not a single token plus the +old gates. A concrete path sends one token; no ancestor token must be sent. +Tokens are column-wide structural capabilities, not decryption keys, and +must not appear in routine logs. +[Read-key implementation](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/query.rs). + +Two independent controls must be named separately: + +- **Wildcard bound:** default 64 elements at each wildcard. Each expansion + includes a probe for the element at the bound. The probe names that + element itself, even if the rest of the requested path does not exist + inside it. A matching probe makes opening fail with + `JsonWildcardTruncated`. A total expansion above **8,192 tokens** fails + with `JsonWildcardTooWide`; two wildcard levels send `N² + N + 1` tokens. +- **Walk budget:** SQL default 64 returned entries across the request. + Exceeding it with a sealed root returns the root entry alone. The client + opens the whole document and evaluates the same paths and probes locally. + Zero disables fallback; an unsealed root cannot fall back and returns the + whole requested walk. This budget is not a hard response-size cap. + +[Expansion and opening](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs), +[SQL budget behavior](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/functions.sql). + +An honest server returns the named node at depth zero plus descendants. +Consequently `{}`, `[]`, a scalar, and an absent path remain distinct. +`DecryptInto>` gives `Some(null)` for JSON null and `None` for +absence; `Vec` gives all wildcard results in document order; singular +`Value` fails on zero or multiple results. In Go use an explicit presence +indicator for a singular read and an explicit result list for a wildcard +read. A nullable Go pointer alone must not collapse JSON null into absence. +[Result implementations](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs). + +## Opening requires trusted request state + +An extracted `JsonNode` / `JsonTreeEntry` does **not** implement standalone +decryption. The selector opens a scalar extraction; the original subtree +key opens walk rows. The subtree key retains concrete paths, bound, probe +positions and the keyset/column machinery needed to recompute selectors; +its JSON serialization contains only the SQL tokens, so it is insufficient +to reconstruct the verification object. +[Engine wrappers](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/mod.rs), +[EQL wrappers](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs). + +The dynamic layer must retain an opaque guest handle or rederive the read +key from a trusted application request (definition, keyset, column context, +path, bound). Database rows must never supply the purported expected path +or context. A stateless rederivation avoids pinning requests to one WASM +instance, but the guest must still reconstruct the same native key and run +the existing opening checks. Do not duplicate these checks in Go. + +The checks reject wrong selectors, inconsistent token/self-tag relationships, +wrong positions or labels, duplicate entries, mixed headers, malformed walks +and incomplete returned subtrees. They do not authenticate the result set: +another document's entries with its own header can open, a field under one +array element can substitute for the same field under another in the cases +documented by the engine, and the database can omit entire requested nodes +or probes. Therefore “verified read” must not promise authenticated absence, +completeness, or row identity. Root fallback must return only the requested +result after client selection, never accidentally hand the whole plaintext +document to a field-read caller. +[Authentication limits](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/docs/reference/json-tree-leakage.md). + +Selector extraction opens scalars, or a sealed root explicitly requested at +`$`; container entries fail because their descendants hold their contents. +The children function returns immediate entries at depth one, not the +subtree result shape. The existing subtree opener cannot simply be applied +to those child rows. A binding should expose them as encrypted entries for +SQL use, or add a specifically request-bound opening operation if plaintext +child enumeration is required. A full subtree read and local child +enumeration is an alternative with different transfer/leakage costs. +[Extraction implementation](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs), +[children SQL](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/functions.sql). + +Child count returns zero for scalars and empty containers, NULL for a missing +node, and NULL for the sealed root. The latter's top-level fields do not +chain from the root token. Do not turn either NULL case into zero. + +## Remaining Go/dynamic work is real + +At this source baseline, the dynamic wire grammar can parse +`{"json":{"sealed_root":true,"ordered":["$.salary"]}}`, but runtime +admission still refuses every JSON index with `json_index_pending`. +`ValueKind` admission assumes scalar indexes. The EQL JSON family is +handwritten and explicitly absent from the catalog/target table. Dynamic +query operands currently distinguish only one term from a range, and +their operator model lacks the JSON-specific path/read inputs. Merely +generating Go types does not make this capability usable. +[Dynamic index grammar](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/term.rs), +[admission](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/kind.rs), +[dynamic queries](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/query.rs), +[EQL family](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs). + +Keep three contracts separate: plaintext JSON transport across the guest, +the existing EQL document/query/read wire forms, and the Vitamin C storage +profile for opaque blobs. A new storage codec for opaque user structs does +not replace `json_tree`'s node encoding or its EQL JSON envelope. The current +EQL/engine conversion still serializes through `serde_json::Value`; direct +typed conversion is a worthwhile format-preserving implementation change. +[Conversions and layouts](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs). + +## Context roles must not be conflated + +The existing `seal_pending` takes one `CallerContext`, obtains its ZeroKMS +descriptor, then uses the same context piece for both the index PRF context +and the AEAD context. The EQL layout accepts only a nonempty `Identifier` +for the column. Dynamic EQL target fields explicitly reject caller +extensions (`TargetError::Extended`) and context-field plans +(`TargetError::ContextField`). Separate row AAD and search scope are therefore +not an existing JSON capability. +[Sealing](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs#L755), +[EQL layout](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs), +[dynamic restrictions](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/record.rs#L73). + +Before freezing a profile that offers row binding, lower distinct column +identity, ciphertext context and search scope into the engine. Never append +a row ID to the present shared context as a shortcut: that also changes +selectors, value terms and read tokens per row. An initial column-only +profile can explicitly refuse those extensions, but must not advertise +unsupported row AAD or reinterpret the same profile later. Tenant keyset +selection is another separate input; it does not provide a replacement for +the context-role distinction. + +For the current client and any later pooled WASI execution, prefer a +host-owned immutable read descriptor +with definition/profile identity, resolved keyset identity, search-scope +inputs, path and wildcard bound. Each open call sends that trusted descriptor +plus untrusted rows, and the guest rederives `JsonSubtreeKey` before opening. +Ciphertext-only row context is supplied separately per document. This avoids +requiring a guest-local handle to survive pool scheduling. The descriptor +must never be reconstructed from database metadata; operation parameters +and column expectations remain application-owned. Reusing a definition ID +for different options/context encodings must be forbidden or versioned. + +One stale source comment in `JsonOptions::sealed_root` says there is no root +token. After #1363 the root has a token; with a sealed root it finds only +the root entry and cannot enumerate top-level members. The module-level +description, implementation and SQL establish the current behavior. + +## Numbers and release fixtures + +#1366 enables `serde_json/float_roundtrip` on Stack Encrypt and tests the +reported one-ULP failure through document, scalar extraction and subtree +opening. This fixes #1364; do not retain it as an unresolved blocker. +It does not make PostgreSQL arbitrary-precision numeric values supported or +make the `f64` ordering encoding lossless for all integers. +[Merged fix](https://github.com/cipherstash/stack/commit/ff287a32887d6a6c07c51d64e61b00deb46eed16). + +The Go contract must preserve signed/unsigned integers through 64 bits and +finite binary64 values without first decoding every JSON number as Go +`float64`. Declare the supported number model and reject unsupported values +instead of silently rounding them. Test Rust-written/Go-opened and +Go-written/Rust-opened documents, not just same-language round trips. + +Release fixtures should cover both root settings; null/absent/empty +containers; nested arrays and quoted keys; containment grouping; equality +versus lossy ordering; concrete and wildcard reads; bound minus one, bound +and bound plus one; token-cap refusal; fallback equivalence; grouped rows +from multiple database records; malicious wrong entries and dropped probes; +and PostgreSQL parameter casts. In particular `->` with an untyped literal +can select native `jsonb` behavior: examples should use typed parameters or +explicit `::text` casts. +[Existing engine tests](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/tests/json_index.rs), +[PostgreSQL oracle](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/tests/encryption/tests/json_tree.rs), +[operator resolution warning](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/operators.sql). From fb4b9e7b9f13413545e1d88f0d3130c80b2a50b1 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 27 Oct 2026 13:04:09 +1100 Subject: [PATCH 3/3] docs: reconcile Go JSON plan with implementation --- docs/plans/2026-10-24-go-json-tree.md | 607 +++++++----------- .../2026-10-25-go-json-engine-contract.md | 422 ++++++------ ...026-10-27-go-json-implementation-status.md | 184 ++++++ 3 files changed, 647 insertions(+), 566 deletions(-) create mode 100644 docs/plans/2026-10-27-go-json-implementation-status.md diff --git a/docs/plans/2026-10-24-go-json-tree.md b/docs/plans/2026-10-24-go-json-tree.md index 38f127a0a..da96ad3be 100644 --- a/docs/plans/2026-10-24-go-json-tree.md +++ b/docs/plans/2026-10-24-go-json-tree.md @@ -1,384 +1,261 @@ -# Go JSON: revised dynamic contract and customer implementation plan - -Date: 2026-10-25. Status: implementation direction agreed in API review; -contract spellings and examples remain proposals, not implemented API. -Tracking issue: [#1347](https://github.com/cipherstash/stack/issues/1347). -This plan under [#1353](https://github.com/cipherstash/stack/pull/1353) uses -merged `next` at `ebdb5de0a12bbec06842851eece283fb7faea30b` and the -[API redesign](https://github.com/cipherstash/stack/blob/c5efc48837d3cb48bc7e168f780fd616a4f4845e/docs/plans/2026-10-24-stack-encrypt-api-plan.md) -tracked in [#1365](https://github.com/cipherstash/stack/pull/1365). -The [engine findings](2026-10-25-go-json-engine-contract.md) cite the source -for the current behavior described below. The examples are proposed contract -shapes, not runnable instructions for the current SDK. - -## Recommendation - -Deliver Go JSON through the existing Rust engine and EQL guest. Implement -configured target definitions, operation-specific results, and request-bound -reads now. These are the parts of the API redesign that affect the customer -contract. The public Rust traits, derives, method renames and crate moves -can follow without changing this contract. - -Keep the full JSON operation inventory in the first release, including -extraction, subtree reads, children and counts. Opening extracted values -is part of that release, alongside full-document writes and queries. The -merged engine already provides the main read machinery; adapting it safely is part of the work. - -Adopt the forthcoming Vitamin C release separately for opaque blobs. Its -storage profile does not replace JSON tree's custom encrypted nodes or EQL -envelope. Freeze and test those existing JSON formats independently. - -This recommendation makes no short-ngram mechanism decision. JSON's result -contract must be extensible without pretending that ngram Boolean-query -requirements have already been settled. - -## Changes from the first-cut plan - -| Original proposal | Revised requirement | -| --- | --- | -| JSON options travel as a special index configuration | A registered target owns typed options, validation and lowering; the same resolved definition serves writes, queries and reads | -| One query twin/domain per target; one or two byte terms | Describe each operation's input and output shape; containment, ordering, selector and subtree preparation have different results | -| `OpenEntry(cipher, rows)` without a request | A prepared selector/read object opens its own rows; expected path, definition and keyset come from the caller, never the database | -| Subtree token plus gates, `_subtree_at` | Current token lists, self tags, client wildcard expansion and probes; `json_tree_subtree(document, tokens, budget)` | -| Selector checks and number correctness are pending | Checks have landed; #1366 fixes #1364. Preserve their tests through the binding | -| Walks ship later | All required operations ship in the customer release | -| Nullable JSON field | SQL NULL, present JSON null and missing selection are distinct states | -| JSON implicitly means the old SteVec index | JSON is a plaintext kind; the target explicitly identifies the EQL JSON-tree representation | -| Reuse current EQL context plumbing | Current EQL targets are column-only and reject context extensions; do not advertise the redesign's row binding as implemented | +# Go JSON: implementation status and remaining delivery plan -The catalog, guest dispatch, generator and Rust parity work in #1353 remain -valid. The proposal needs revised contracts rather than a second SDK or -crypto implementation. Its five tracking issues remain useful work items. +Updated: 2026-10-27. Reviewed `next` at +[`ffeed7adb3f7d8739cda53261046147852bffc82`](https://github.com/cipherstash/stack/commit/ffeed7adb3f7d8739cda53261046147852bffc82), +against the previous `ebdb5de0a` baseline. This is the plan in +[#1353](https://github.com/cipherstash/stack/pull/1353). +Tracking issue #1347 and implementation issues #1348–#1352 are now closed. -## 1. One configured target definition +Go JSON is implemented on `next`. The remaining task is release readiness +and explicit closure of the narrower gaps below, not rebuilding the original +catalog/guest/generator sequence. Broader Rust API and packaging work remains +in [#1365](https://github.com/cipherstash/stack/pull/1365). -Make the public interoperability unit a versioned definition, lowered to -internal plans. It contains field identity, source kind, target, resolved -options and declared context roles. Bind requests to a field of that -definition. Do not pass another options object to a query or decrypt call. +Source-backed details are in the [engine findings](2026-10-25-go-json-engine-contract.md) +and [Go implementation status](2026-10-27-go-json-implementation-status.md). +These distinguish inspected tests from tests rerun during this review. -Proposed minimal JSON authoring form, for the existing column-only profile: +## What has landed + +| PR | Merged behavior | Change to the plan | +| --- | --- | --- | +| [#1381](https://github.com/cipherstash/stack/pull/1381) | Mutation CI computes the diff against the actual PR base | Verification infrastructure; no new public API task | +| [#1379](https://github.com/cipherstash/stack/pull/1379) | Cascade authentication and randomized JSON entry/object/projection order | Update read guarantees, wildcard SQL and storage fixtures | +| [#1369](https://github.com/cipherstash/stack/pull/1369) | Go Vitamin C storage-capable pseudo-version pin; final-release guard | Adoption is done; replace the temporary Go pin before a final release | +| [#1370](https://github.com/cipherstash/stack/pull/1370) | Opaque Go structs use the versioned storage profile | Remove codec adoption from future implementation work | +| [#1372](https://github.com/cipherstash/stack/pull/1372) | JSON catalog, configured targets, parser and full-document dynamic execution | Mark the registry/document path implemented | +| [#1373](https://github.com/cipherstash/stack/pull/1373) | JSON operators, operand encodings and request-bound guest reads | Mark query/read plumbing implemented | +| [#1378](https://github.com/cipherstash/stack/pull/1378) | Generated Go JSON fields, typed document adapters and typed query inputs | Mark the generated frontend implemented | +| [#1397](https://github.com/cipherstash/stack/pull/1397) | Short whole-word terms in EQL v4 match filters | Record current behavior separately from unresolved configurable-match/privacy work | +| [#1402](https://github.com/cipherstash/stack/pull/1402) | Go typed JSON docs, runnable example and documentation drift check | Use the shipped examples instead of speculative API recipes | + +## The implemented public contract + +Use the landed names for customer-facing code. Do not make a cosmetic rename +or a replacement Go builder a release dependency. + +| Role | Current spelling | +| --- | --- | +| Registered target / tag | `JsonSearch`, `encrypt_into=JsonSearch` | +| Stored Go value | `eql.JSONSearch` | +| Containment operand | `eql.JSONSearchQuery` | +| Ordering operand | `eql.JSONSearchLeafQuery` | +| Returned encrypted entry | `eql.JSONEntry` | +| Prepared selector / subtree read | `encrypt.JSONSelection`, `encrypt.JSONRead` | +| SQL scalar / list parameter | `encrypt.JSONSelector`, `JSONToken`, `JSONTokens` | +| Singular plaintext result | `encrypt.JSONValue{Found, JSON}` | +| Stored SQL domain | `public.eql_v4_json_search` | +| Query SQL domains | `eql_v4.query_json`, `eql_v4.query_json_leaf` | +| Entry SQL domain | `public.eql_v4_json_entry` | + +The lower-level engine still uses `JsonDocument`, and handwritten EQL Rust +adapters retain `JsonTree*` names. `eql.JSON` remains storage-only. EQL v4's +old SteVec entries have been removed; v3 remains a separate contract. + +Configured target objects already work. The current data-plan grammar is a +field map, not the proposed `version`/`fields` envelope: ```json { - "version": 1, - "fields": { - "profile": { - "type": "json", - "context": ["users", "profile"], - "target": { - "name": "EncryptedJson", - "options": {"sealed_root": true, "ordered": ["$.salary"]} - } + "profile": { + "type": "json", + "context": ["users", "profile"], + "target": { + "name": "JsonSearch", + "options": {"sealed_root": true, "ordered": ["$.salary"]} } } } ``` -`EncryptedJson` is the recommended role-based target name, mapping to the -current Rust `JsonTreeDocument` and SQL `public.eql_v4_json_tree`. Go spells -the generated ciphertext type `eql.EncryptedJSON`. These names are proposals; -the existing SQL domain does not need a rename. Target identifiers are -contract identifiers, not names obtained from Rust type reflection. - -A string target remains shorthand for default options. Unknown keys, -unknown targets, unsupported versions and unsupported context capabilities -fail closed. Normalize defaults once. Resolve options into the same native -`JsonOptions` used by typed Rust, including its path validation. A new -configured definition is a new immutable instance, not mutation of one used -by an in-flight query. - -Extend discovery with option schema/profile identity and per-operation -input, result and capability metadata. The existing target descriptor's -single `query`/`query_sql_domain` pair cannot describe JSON accurately. -Generate schemas and Go types from the maintained registry where possible; -keep semantic validation in Rust. Do not add an independent Go path parser. - -The guest should cache validated definitions under their complete resolved -content and contract version. Generated field accessors share that immutable -content. Guest-local cache identifiers remain internal and can be recreated; -they must not become SQL values or customer persistence formats. - -The SDK keeps its idiomatic generated `Encrypt`, `Decrypt` and `Fields` -entry points. It does not expose a Rust plan or require a Go `.using()` -method. Both tags and the existing generator policy path lower to this same -definition. Simple target options can use the tag grammar; complex quoted -paths need a structured options form, not a naive `|` split. Validate both -through the engine at generation time. - -## 2. Explicit JSON values and three separate codecs - -Use UTF-8 JSON text for plaintext documents and JSON query inputs at the -host/guest boundary. In Go, use `json.RawMessage` as the fundamental value. -A typed struct adapter may explicitly call `encoding/json` and decode into -a declared type. Never decode first into `map[string]any` with default -`float64` numbers just to send the tree to Rust. - -Do not implicitly treat an ordinary `[]byte` as JSON: Go's JSON encoder -normally treats it as a byte string. Require an explicit `json.RawMessage` -conversion for raw JSON bytes. Containment accepts an explicit JSON needle, -not necessarily the field's full struct type, whose zero-valued fields can -accidentally narrow the query. - -Across the binary ABI, the declared `json` input is a UTF-8 text/byte payload -in the ordinary transport frame. In JSON authoring/request examples below, -`json` is a string carrying that text. Define this mapping explicitly; -a JSON document's `null` is not the frame's absent-field/null marker. - -| Boundary | Representation | -| --- | --- | -| Dynamic definitions, requests and responses | Versioned control schema carried by the existing Vitamin C transport frame | -| Searchable JSON plaintext | UTF-8 JSON parsed by Rust into the existing JSON value model | -| Stored searchable document | Existing EQL v4 JSON-tree envelope, header, encrypted nodes and search terms | -| Stored opaque whole struct | Vitamin C versioned storage blob from #520/#526, sealed as one value | - -Use direct typed EQL/engine conversion when implementing the adapter; remove -the internal serialize-to-Value/deserialize bridge without changing stored -bytes. This is distinct from parsing user JSON text at ingress. - -Pin the numeric model: signed/unsigned 64-bit integers and finite binary64 -values, not arbitrary-precision PostgreSQL numeric. Preserve integer tokens -without first casting to float. Add ingress checks for unsupported integer -literals instead of accepting a parser's lossy fallback. Fractional/exponent -numbers follow the documented binary64 interpretation; JSON text, key order -and decimal spelling are not promised to round trip byte for byte. The -existing `float_roundtrip` fix must remain enabled. Separately document that -JSON numeric ordering converts to `f64`, so large distinct integers can tie. - -Specify duplicate-member handling and invalid Unicode rather than inheriting -an accidental Go/Rust difference. Recommend rejecting duplicate object keys -and invalid Unicode at JSON ingress in Rust, before key I/O. This validation -is additional adapter work, not a claim about the current parser. A typed -Go encoder may already have transformed the caller's input; its explicit -adapter contract must describe that transformation. - -For an optional document, a nil `*json.RawMessage` is SQL NULL. A non-nil -pointer containing `null` is a present encrypted JSON null. A nonoptional -`json.RawMessage` is a required document; empty bytes are invalid JSON, not -an absent document. `null=sql` is required for nullable fields, not every -JSON field. Length-padded sealed null is not supported for this target. - -## 3. Operation-specific requests and results - -Use operator names with distinct operand schemas, all bound to the same -field definition. For example: - -```json -{"operation":"contains","field":"profile","input":{"json":"{\"active\":true}"}} -``` - -```json -{"operation":"eq_at","field":"profile","input":{"path":"$.name","json":"\"Alice\""}} -``` - -```json -{"operation":"subtree","field":"profile","input":{"path":"$.visits[*]","wildcard_bound":64}} -``` +A string target selects defaults. Native options are resolved in the plan; +queries and reads use that field's options. Unsupported options and malformed +JSON/path inputs fail through the engine. This implements the configured +JSON target seam; it does not implement the full general definition cache, +role system or versioned interoperability specification from the API review. +The guest still parses submitted plans per operation. -These fragments are operation bodies; the request envelope additionally -identifies protocol version, definition and execution/keyset inputs. +Query results now identify `encoding` (`bytes`, `eql`, `text`, `text_array`), +and capability entries identify the input shape. Go's `QueryTerm` carries +byte terms or text values, and generated methods check the expected result. +That meets JSON's operand distinction; do not require a new sum-type API +solely because the earlier plan illustrated one. -Replace the assumption that every operation yields one or two anonymous -byte strings with a tagged result model. Predicate operands, ordered ranges, -selectors and prepared reads have separate variants and named Go types. -Each operand states its wire encoding and SQL parameter type. A selector -is text; subtree tokens are `text[]`; query envelopes are EQL JSON. A token -list is not a BETWEEN range or a generic Boolean OR expression. +## Go usage and input boundaries -| Operation | Required Go/dynamic behavior | SQL use | -| --- | --- | --- | -| Encrypt/decrypt, including batches | Full document, both root policies, scalar/array/object/null | One `public.eql_v4_json_tree` column | -| Contains / ContainedBy | JSON needle and named containment query | `@>` / `<@` with `eql_v4.query_json_tree` | -| EqAt | Scalar JSON operand and validated selector-class path | Containment predicate; never ORE/OPE equality | -| LtAt/LteAt/GtAt/GteAt | Ordered path plus number/string | Extract entry, compare with `eql_v4.query_json_tree_leaf` | -| BetweenAt | Two ordered bounds plus selector | Inclusive lower/upper predicates | -| Selector | Prepared selection with named SQL selector | `->`, `->>`, first/all path query, path exists, sorting | -| Open selection | Original selection plus entries grouped by database document | Scalars; sealed root at `$`; ordinary container marker refused | -| Subtree | Prepared concrete or wildcard read with token list | `json_tree_subtree(document, tokens, budget)` | -| Open subtree | Original read plus grouped rows | Requested JSON only, including fallback | -| Children | Concrete node request and encrypted child entries | `json_tree_children(document, token)` | -| Open child scalars | Request-bound child validation in Rust | Additional adapter/helper work; never generic entry decryption | -| ChildCount | Concrete node token; nullable count | `json_tree_child_count(document, token)` | -| Stored-value comparisons | Same column/scope and compatible profile | Existing document containment and entry ordering operators | - -Generate ordered-path accessors from validated options, with explicit -aliases or a generation error on name collisions. Runtime path methods use -the same native validator. Reject positions for selector-class operations -including EqAt; those selectors erase `[n]` to `[*]`. Exact positional reads -use `subtree`. Wildcard equality is any-match; wildcard ordering through -`->` is first-match. Multiple independent EqAt predicates need not match the -same array element; use one containment needle for correlated conditions. - -For children, the SQL rows have depth one and are not complete subtrees. -Add a parent-bound opening helper that verifies child membership and opens -scalars; container children require a subtree read. Implement this in Rust -and reuse it from Go. Counts remain database observations: zero for a scalar -or empty container, NULL for absence or the sealed root. Neither counts nor -absence become authenticated merely because returned ciphertexts open. - -## 4. A read object retains the verification request - -Recommend an immutable host-owned read descriptor and guest rederivation. -Store the trusted definition/profile, resolved keyset identity, path, -wildcard bound and applicable scope inputs. Do not export PRF keys or -serialize a native `JsonSubtreeKey` expecting to recover its private state. -Its existing serialized token list is insufficient for opening checks. - -The SQL parameter is a separate value derived from the read. Opening sends -the original descriptor and untrusted rows to Rust, rederives the native -selector/subtree key and uses its existing verifier. Enforce definition, -keyset and scope agreement. Caller `context.Context` is passed per call, -never retained in the read. The current Go client has one guest instance; -rederivation also avoids imposing instance affinity on future execution. - -Proposed usage fragment, with database plumbing abbreviated deliberately: +The existing generated call shape remains `Encrypt`, `Decrypt` and `Fields`. +A representative declaration, using implemented syntax: ```go -read, err := users.Fields.Profile.Subtree(ctx, cipher, "$.visits[*]", - encrypt.JSONReadOptions{WildcardBound: 64}) -if err != nil { return err } - -// Bind read.Tokens() as PostgreSQL text[], using the driver's array adapter: -// SELECT e FROM eql_v4.json_tree_subtree($1::public.eql_v4_json_tree, -// $2::text[], $3::integer) AS e -// $1 is the stored document; $3 is the SQL walk budget, e.g. 64. - -selected, err := read.Open(ctx, cipher, encryptedRows) -if err != nil { return err } -// selected is an explicit list of JSON values, in document order. +type Profile struct { + Name string `json:"name"` + Salary int64 `json:"salary"` +} + +type User struct { + _ struct{} `stash:"context=users"` + ID int64 `stash:"id,passthrough"` + Profile Profile `stash:"profile,encrypt_into=JsonSearch(ordered=$.salary)"` +} ``` -Provide batch opening with groups keyed by the caller's database row -association. Never concatenate different documents' entries into one walk. -Go groups transport values; Rust still gathers and executes all key requests -in one logical batch. Do not promise one HTTP request for unlimited batches: -existing provider chunk limits and distinct keysets still apply. - -Use a singular result with `Found` plus raw JSON, and a plural result with a -JSON-value list. Present JSON null is `Found=true` with text `null`; no match -is `Found=false` or an empty plural list. A singular read with multiple -results fails rather than silently choosing one. SQL first-match selection -is a separately explicit operation. - -Name the two limits separately: - -- `wildcard_bound`: defaults to 64 per wildcard, with boundary probes; - total token expansion is capped at 8,192. A found boundary probe fails - with `JsonWildcardTruncated`; excessive expansion fails before key I/O. -- `walk_budget`: SQL defaults to 64; selects sealed-root fallback. Zero - disables fallback. With an unsealed root it does not cap response size. - -Always apply the original selection after sealed-root fallback. Do not -return the whole decrypted document from a subtree request. Preserve the -engine's rejection of forged, spliced and malformed returned walks. -Document the existing limits: the server can omit whole matches/probes, -and successful opening is not proof of result completeness or database row -identity. Do not expose a `complete=true` claim from this protocol. - -## 5. Context and provider boundary: freeze only implemented guarantees - -This is the main redesign issue that cannot be solved by Go wrappers. -JSON currently derives index context and ciphertext AAD from the same -context piece. The EQL layout accepts a column identifier; dynamic targets -reject extensions and context-field plans. - -Keep column identity, search scope, ciphertext binding and provider options -as distinct roles in the new contract. Validate source roles against the -trusted definition: a primary-key field cannot supply search scope; a -scope key is only used when explicitly selected. Keyset selection and -ZeroKMS lock policy remain execution inputs, not JSON target options. - -**Delivery recommendation:** implement the complete JSON operation surface -first under the existing column-and-keyset context profile. Advertise that -profile explicitly and reject additional scope/row-binding declarations -until their lowering is implemented. This does not claim the accepted -context redesign is complete or defer JSON operations. The customer has -not stated a requirement for new row-bound JSON profiles in this task. - -Before enabling row binding, split the native PRF and AEAD inputs and test -writes, all queries and all read paths together. Search uses column plus -scope; AEAD includes the additional row binding. Decrypt requests then need -the expected row values, including when SQL returns extracted entries. -Never append a row ID to today's shared context as a shortcut. - -Pin the current context derivation as part of the storage/search profile. -Introducing the separated profile must be explicit and retain the reader -and query behavior of released profiles; it must not reinterpret old data -or silently reindex it. Decide the profile identifier and its association -with a column/definition before release. Schema version, EQL envelope/header -version and Vitamin C blob version are different version boundaries. - -## 6. Specification and acceptance gates - -Publish the first supported subset of the dynamic interoperability spec -alongside this implementation: JSON Schema, normative semantics, transport -mapping and conformance vectors. Use the general target model, not a -JSON-only replacement protocol. Unsupported future features fail explicitly; -reserved fields are not accepted and ignored. - -The schema covers definitions, target options, operations, tagged results, -read descriptors, presence states and stable errors. The semantic contract -covers number normalization, path classes, contexts, limits and validation -before key I/O. Return machine-readable codes and safe locations, without -JSON values, paths or tokens in diagnostic text. Definition/field identifiers -can identify the failing operation without echoing user values. - -Customer release gates: - -1. Catalog, options and capability discovery match Rust and generated Go; - schema/semantic fixtures reject unknown options and unsupported profiles. -2. Go writes open in Rust and Rust writes open in Go, with identical search - terms under the same inputs. Use fixed randomness only in test fixtures; - cross-opening and SQL behavior are the production compatibility criteria. -3. All operation rows above run through the real guest and PostgreSQL. - Retain the engine's jsonb containment/read oracles; add Go binding coverage. -4. Test both root policies, nested arrays, quoted keys, scalar/null roots, - SQL NULL, absence, empty containers, large integers and the #1366 float. -5. Test wildcard N-1/N/N+1, nested expansion caps, budget zero/default, - fallback equivalence, malicious row substitutions, duplicates, mixed - documents, truncated subtrees and the documented omission limitations. -6. Test actual SQL driver encodings: `driver.Valuer`/`Scanner`, pgx arrays, - explicit casts, sqlc/GORM integration where the SDK promises support. -7. Freeze exact storage/header/search vectors and context derivation; - verify guest/engine/EQL bundle compatibility. Record the accepted leakage - profile. Future Rust/package changes must retain these fixtures. - -## Implementation order and issue amendments - -| Slice | Work | Existing issue to revise | -| --- | --- | --- | -| A. Contract and registry | Versioned configured targets, operation/result schemas, defaults, profile identity, catalog and generated types | #1348 plus contract portion of #1349 | -| B. Full-document path | JSON ingress/egress, typed target adapter, definition resolution, batched writes/decrypts | #1349 | -| C. Queries and reads in Rust/guest | JSON operators, selector preparation, token-list preparation, request-bound opening, child helper, stable errors | #1350 and #1352 together | -| D. Go surface | Tags/policies, checker, generated field methods, named SQL values, read objects and examples | #1351 | -| E. Customer release proof | Go/guest/PostgreSQL tests, conformance and storage vectors, supported-profile docs | #1347 acceptance checklist | - -Start with A and a thin B/C vertical slice: one configured JSON field, batch -round trip, containment and a request-bound subtree read through the guest. -That proves the difficult seams before expanding the generator. It is an -implementation milestone, not permission to release a reduced operation set. - -Coordinate opaque storage adoption with Vitamin C #520/#526 once reviewed -and released. Keep Rust facade/package refactors off this critical path. -Align tracking issue #1347 and work items #1348–#1352 with the slices above -before assigning implementation. Their external issue descriptions have not -yet been amended by this documentation change. - -## Evidence and verification performed - -Read #1353 and its six tracking issues; inspected merged engine, EQL SQL, -Go generator/runtime, guest dispatch and the API redesign documents. Read -Vitamin C #520/#526 descriptions to distinguish their storage deliverable -from Stack integration. The accompanying engine findings link immutable -source locations. Only documentation checks ran here; no new implementation, -Go/Rust build, live database test or performance claim is reported. - -Additional primary sources: - -- [SDK principles](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/docs/sdk-design-principles.md). -- [Dynamic target resolver and descriptor](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/target.rs). -- [Current Go query response](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/languages/golang/encrypt/query.go). -- [Guest target dispatch](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/languages/golang/encrypt/guest/src/targets.rs). -- [Current Go codec and opaque JSON storage](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/languages/golang/encrypt/gensupport/codec.go). -- [Vitamin C Rust storage profile #520](https://github.com/cipherstash/vitaminc/pull/520) - and [Go storage profile #526](https://github.com/cipherstash/vitaminc/pull/526). +`json.RawMessage` supports arbitrary JSON roots, including scalar and null. +Structs, supported maps and slices have explicit `encoding/json` adapters. +Pointer fields use `null=sql`; a nil pointer is absent, while a non-nil raw +message containing `null` is present JSON null. Typed nil maps/slices encode +as JSON null. The generator refuses unsupported typed shapes; use raw JSON +for roots not expressible by a supported typed document declaration. + +Query methods accept Go values and explicit raw JSON needles. They guard +against plain byte slices, accidental string/null containment inputs and a +full field struct used as a containment needle with zero-valued members. +`DecodeJSON[T]` reads selected JSON into the requested type and uses +`json.Number` inside interface values. Typed adapters retain Go's own JSON +semantics, including replacement of invalid UTF-8 before Rust sees it. + +The shared Rust parser rejects duplicate keys, invalid JSON/Unicode and +integer literals outside `i64::MIN..=u64::MAX`; finite fractional/exponent +numbers follow binary64 semantics. This validation is implemented, not a +future parser project. Arbitrary-precision PostgreSQL numbers remain outside +the contract. Numeric ordering still uses `f64`, so large integers can tie. + +Ordered paths in tags are semicolon-separated. Generated `At.` +accessors and runtime path methods exist. Quoted-key paths work in the native +path grammar but not in the current simple tag spelling; define a structured +Go declaration route if that is required for the customer's ordered keys. + +The complete executable example is maintained in +[languages/golang/encrypt/example/json](https://github.com/cipherstash/stack/tree/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/example/json). +Its current read shape is `read.Open(ctx, groupedEntries)`: the prepared read +already retains its cipher and declaration. The old proposed +`read.Open(ctx, cipher, rows)` is not the implemented API. + +## Corrected operation and read semantics + +| Capability | Implemented surface / current limit | +| --- | --- | +| Full-document writes/reads and batches | Generated `Encrypt`/`Decrypt`, including raw and typed documents | +| Contains / contained-by / scalar equality at path | Generated `Contains`, `ContainedBy`, `EqAt`; equality uses value terms, never ordering terms | +| Ordering and ranges | Declared-path accessors and runtime methods; `Between` returns named bounds | +| Existence / sorting / `->` / `->>` | Selector parameter plus application SQL; `->>` still returns encrypted text | +| Open selected entry | `JSONSelection.Open`, one optional entry per database row | +| Positional / wildcard subtree | `Subtree`, token list, `JSONRead.Open` / `OpenOne`; rederivation and verification happen in Rust | +| Enumerate / count children | `Children` / `ChildCount` prepare a concrete node token for SQL | +| Parent-bound plaintext child opening | Not implemented; do not pass child rows to the subtree or selector opener as if they were its requested result | +| All selector matches | SQL can enumerate them; the dynamic selector opener accepts at most one entry per document group. Use wildcard subtree reads for supported all-match plaintext retrieval | + +A selector addresses an array path class; positional selector queries are +rejected. Use subtree reads for exact positions. Wildcard `->` returns an +**unspecified element**, because stored entry order is randomized. It is +not the first element in document order. For an existential ordering predicate, +use `EXISTS` over `json_tree_path_query`; define an explicit aggregate for +sorting a multi-valued path. Equality at a wildcard is any-match. Independent +EqAt predicates need not match the same array element; one containment +needle preserves correlation. + +Cascade authentication binds a container's children into its ciphertext. +A nonempty container entry cannot open alone; use a subtree read. The +`JsonNotALeaf` error also covers alteration, since the client cannot reliably +distinguish those causes before opening. Empty ordinary containers instead +fail `OpenedType`; they also require subtree reads. Sealed-root fallback applies +the original requested path after decryption. + +Reads retain the original trusted path/definition/keyset. SQL rows cannot +choose their own expected request. Entries are grouped per database document +and opened through the common batch engine. The protocol still does not +prove that the database returned every match or the intended database row; +whole results or probes may be withheld. Do not claim authenticated absence +or result-set completeness. + +The two limits remain separate: wildcard bound defaults to 64 with probes +and an 8,192-token expansion cap; SQL walk budget defaults to 64 and selects +sealed-root fallback. A zero walk budget disables fallback; an open root +cannot use that fallback. `OpenOne` rejects wildcard reads up front. + +## Storage and release compatibility + +Opaque Go writes already call `vcffi.MarshalStorage`; readers recognize +storage blobs and retain the legacy JSON fallback. Future storage versions +fail as `ErrStorageVersion`, not JSON decode. Generated code now requires +`GeneratedVersion2` and `vc` tags. Record the implemented behavior: regenerate +code, upgrade readers first if earlier writers were deployed, nil containers +round-trip empty, nonfinite floats are allowed, and some formerly JSON-only +opaque types are refused. No new migration system is requested here. + +Go currently pins Vitamin C at `06568ba93bd9` using pseudo-versions. Rust +workspaces remain on crates.io Vitamin C 0.8.0. The final Go release guard +rejects the pseudo-version pins; a prerelease may carry them. Replace the +Go module pins with released tags before a final release rather than blocking +all JSON work on a Rust dependency move. + +JSON-tree storage remains separate. #1379 changed authentication and order +without changing the header version from 1. The old and new profiles are not +interchangeable merely because their header/envelope versions match. The +prototype compatibility exemption permits this pre-release change; freeze +the post-cascade vectors and context/search semantics for customer data. + +Containment projections and query projections are now randomized too. +Compare canonical/set semantics and cross-opening/query behavior in parity +tests. Raw byte equality requires controlled randomness; never require two +ordinary query calls to serialize identical arrays. + +## Context and broader API redesign + +The shipping JSON profile still uses column identity and selected keyset. +EQL target extensions and context-field plans are refused. Separate row AAD, +search scope and role eligibility are not implemented by the configured-target +work. Preserve the accepted direction, but do not advertise those capabilities +or block the existing profile's release on implementing the whole redesign. + +Before enabling row-bound JSON, split PRF and AEAD input roles and test all +read/query paths. Appending row IDs to the current shared context would make +search terms row-specific. Provider policy and keyset selection remain +separate from JSON options. Open-root JSON now also needs index-key loading +for verification; generic execution/provider bounds must account for both +`DataKeySource` and `IndexKeySource`. + +## Remaining work, in delivery order + +1. **Finish the supported-contract specification.** Describe the implemented + field-map grammar, options, operator inputs, `encoding` results, grouped + reads, error codes and supported context profile. Add the promised schema + and shared conformance corpus; allocate versioning explicitly. Do not + describe the earlier illustrative versioned envelope as already accepted + by the runtime. General definition caches and Rust factories can follow. +2. **Close narrow coverage gaps.** Add parent-bound child-scalar opening + to meet the comprehensive inventory requirement. SQL children + and counts are present, but are not this opener. For all-match reads, use + the existing subtree API; do not invent an unsafe selector-group shortcut. + Confirm whether complex ordered-key declarations need a structured Go + route. Complete direct typed EQL conversions as a format-preserving cleanup. +3. **Finish configurable match support for the customer.** #1397 supplies + whole two-character words in the existing EQL filter, enabled by the + pinned EQL match profile. Native Go `index=match` still refuses options + (#1112); configurable tokenizers/filter pipelines are not thereby done. + The implementation uses deterministic domain-separated word terms, not + random variants. This review records that fact without settling the + previously raised privacy concern or changing the uncommitted research. +4. **Validate and freeze the candidate.** Use the existing Rust, guest, Go + and PostgreSQL tests and runnable example; verify supported operations, + numeric/null behavior, both root policies, bounds/fallback, tampering and + the known omission limits on the exact release build. Freeze post-cascade + JSON and short-word search semantics. Rebuilding old Bloom rows is needed + for reliable short-word queries if pre-change prototype data is retained. +5. **Release dependencies and artifacts.** Replace the temporary Go module + pins for a final release; build production guests and the matching EQL v4 + bundle using the existing release workflow. Keep Rust facade/package moves + behind this customer delivery. + +The original thin vertical slice is complete and is no longer the next +implementation task. Closed issues #1347–#1352 should not remain labelled +unimplemented in the plan. Any residual work above needs its own scoped +acceptance item rather than reopening the whole JSON feature. + +## Evidence and verification + +This refresh inspected the nine first-parent merges, current source, +changelogs, tests and documentation, and checked the tracking issue states. +No new runtime code was written. Markdown/link/JSON checks are run on the +plan changes; runtime and database suites were not rerun for this review. +Merged PR verification reports and existing test files are evidence of +coverage, not a fresh certification of the customer release build. diff --git a/docs/plans/2026-10-25-go-json-engine-contract.md b/docs/plans/2026-10-25-go-json-engine-contract.md index 82ba99c30..fcca06c08 100644 --- a/docs/plans/2026-10-25-go-json-engine-contract.md +++ b/docs/plans/2026-10-25-go-json-engine-contract.md @@ -2,205 +2,225 @@ Supporting evidence for the [Go JSON implementation plan](2026-10-24-go-json-tree.md). -Research baseline: `origin/next` at -[`ebdb5de0a12bbec06842851eece283fb7faea30b`](https://github.com/cipherstash/stack/commit/ebdb5de0a12bbec06842851eece283fb7faea30b), -after JSON-tree #1363, leakage documentation/tests #1362 and float-roundtrip -fix #1366. This is source analysis, not a rerun of those changes' tests. -The Go API recommendations below are proposals; the named Rust/SQL behavior -is implemented at that baseline. No ngram decisions are made here. - -## What the merged implementation actually provides - -| Capability | Engine / SQL contract | Go consequence | +Updated: 2026-10-27. Source baseline: merged `next` at +[`ffeed7adb3f7d8739cda53261046147852bffc82`](https://github.com/cipherstash/stack/commit/ffeed7adb3f7d8739cda53261046147852bffc82). +This replaces the earlier `ebdb5de0a` assessment. The relevant new merges +are #1379 (cascade and random order), #1372 (configured document targets) +and #1373 (dynamic queries/reads); #1378 and #1402 add the typed Go surface +and its documentation. This is source reconciliation, not a rerun of their +builds, tests or security analysis. No ngram recommendation is made here. + +## Capabilities that are implemented now + +The previous catalog and dynamic-admission blockers are resolved for the +configured JSON target. The current target identifier is **`JsonSearch`**, +and the generated Go stored type is **`eql.JSONSearch`**. The proposed +`EncryptedJson` spelling is not the implemented interface. + +| Role | Current catalog / Go name | SQL domain | +| --- | --- | --- | +| Stored document | `JsonSearch` / `JSONSearch` | `public.eql_v4_json_search` | +| Containment or path-equality operand | `JsonSearchQuery` / `JSONSearchQuery` | `eql_v4.query_json` | +| Ordering operand | `JsonSearchLeafQuery` / `JSONSearchLeafQuery` | `eql_v4.query_json_leaf` | +| Extracted entry | `JsonEntry` / `JSONEntry` | `public.eql_v4_json_entry` | + +The engine's handwritten Rust `JsonTreeDocument`, `JsonTreeQuery`, +`JsonTreeLeafQuery` and `JsonTreeEntry` remain implementation names. SQL +functions retain `json_tree_*` names. Update examples and fixtures to these +current domain spellings rather than the former `_json_tree` domains. +[Catalog](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/eql/crates/eql-domains/src/lib.rs#L621), +[Rust domain mapping](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/eql/crates/eql-bindings/src/v4/json_tree.rs#L286). + +A dynamic field declares `"type":"json"` and either `"target":"JsonSearch"` +or `"target":{"name":"JsonSearch","options":{"sealed_root":true, +"ordered":["$.salary"]}}`. Options resolve once per parsed plan into +`FieldPlan::target_options` / `TargetOptions::Json`. Writes, queries and +reads consult those resolved options. The default remains sealed root and +no ordered paths. Unknown/incompatible options, `options:null`, JSON without +a target, and sealed field-null mode are refused. Legacy bare JSON index +admission is not the supported document route; the configured target is. +[Target options and resolver](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/dynamic/target.rs), +[plan resolution and grammar](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/dynamic/record.rs). + +This implements the concrete configured-target seam. It does not by itself +implement the whole proposed versioned definition protocol, JSON Schema, +provider capability schema or general cached `TargetDefinition` lifecycle. +The guest still parses submitted plans in its operations; do not describe a +persistent loaded-definition cache as completed. +[Guest operations](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/guest/src/ops.rs). + +## JSON transport and validation are no longer pending + +Documents enter as UTF-8 JSON in a text or bytes transport leaf and return +as JSON text in a bytes leaf. The new Rust parser rejects duplicate object +keys, malformed JSON/Unicode, and integer literals outside +`i64::MIN..=u64::MAX`. Decimal/exponent values use the engine's finite binary64 +model, not arbitrary-precision PostgreSQL numeric. Errors carry codes and +positions without echoing document text. Source checks invoke the same +parser before key requests. The existing `float_roundtrip` fix remains. +[Dynamic JSON adapter](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/dynamic/json.rs), +[parser and writer](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/json/text.rs). + +JSON text `null` is a present document. A transport null/missing optional +field with `null=sql` represents SQL NULL. A selection with no match is a +third state. Required JSON fields do not need a field-null mode. Whole +objects, arrays, scalars, null and empty containers are supported documents. + +Keep the transport, searchable JSON storage and opaque-blob storage profiles +separate. The EQL/engine adapter still uses `serde_json::to_value` / +`from_value` in `transcode`, `envelope` and `document`; direct typed conversion +remains format-preserving follow-up work. Vitamin C opaque storage does not +replace the tree's node encoding or EQL envelope. +[Current conversions](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/eql/crates/eql-bindings/src/v4/json_tree.rs#L293). + +## Dynamic operations and request-bound opening have landed + +`OpSpec` now covers containment, path equality, path ordering/range, +selector, subtree, children and child count. Operation discovery includes +an input-shape discriminator. Query responses include `encoding` (`bytes`, +`eql`, `text`, `text_array`), and operands include text and text lists as +well as one/range byte operands. The ordering SQL domain is represented +separately from the containment domain. These are implemented extensions to +the current wire model, not the entire proposed future tagged-result schema. +[Dynamic query model](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/dynamic/query.rs), +[JSON operation grammar](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/dynamic/json_query.rs). + +| Operation | Input | Derived operand | | --- | --- | --- | -| Store and open a document | `JsonDocument`; EQL `JsonTreeDocument` adds the version and column envelope | One document field is one JSON value, including scalar roots, arrays and JSON null; do not restrict it to a Go object/map | -| Containment | `JsonQuery`, exact element-nested projection; SQL `@>` and `<@` | Keep operand direction explicit; array-of-object containment must not combine predicates from different elements | -| Equality at a path | `AtPath` constructs a scalar containment needle | Use this equality capability, never infer equality from the ordering term | -| Ordering at a path | `JsonLeafQuery`; `<`, `<=`, `>`, `>=`, inclusive `Between`; only number/string leaves at configured paths | Configuration belongs to the shared target definition used for writes and queries | -| Selector extraction | `JsonSelector`; SQL `->`, `->>`, path query, first and exists | Selector queries collapse positions to wildcards; first means first matching selector, not a named array position | -| Subtree read | `JsonSubtreeKey`; SQL `json_tree_subtree(document, tokens, budget)` | Preserve the trusted read request through to opening its rows | -| Children / count | SQL `json_tree_children(document, token)` and `json_tree_child_count(document, token)` | These take one node token, not an arbitrary wildcard token list; returned child entries are not complete subtrees | - -Sources: [engine query types and operators](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/query.rs), -[SQL functions](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/functions.sql), -[SQL operators](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/operators.sql). - -`sealed_root` defaults to true; `ordered` defaults to empty. Ordered paths -may include wildcards, but may not name a position such as `$.scores[2]`. -A query at an unordered path fails before loading keys. Ordering through a -wildcard selector compares the first matching entry; it is not an existential -or universal comparison over all elements. Ordering numbers uses `f64`, so -different large integers can tie; text ordering folds case and accents. -These are storage/search semantics, not Go conversion choices. -[Options](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/mod.rs), -[ordering derivation](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs), -[documented ordering profile](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/docs/reference/json-tree-leakage.md). - -## Replace the old read recipe - -The revised design derives tokens directly from paths, stores a self tag -`x` on every entry, and expands wildcards on the client. A subtree request -sends an ordered token list (`{"t":["…"]}`), not a single token plus the -old gates. A concrete path sends one token; no ancestor token must be sent. -Tokens are column-wide structural capabilities, not decryption keys, and -must not appear in routine logs. -[Read-key implementation](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/query.rs). - -Two independent controls must be named separately: - -- **Wildcard bound:** default 64 elements at each wildcard. Each expansion - includes a probe for the element at the bound. The probe names that - element itself, even if the rest of the requested path does not exist - inside it. A matching probe makes opening fail with - `JsonWildcardTruncated`. A total expansion above **8,192 tokens** fails - with `JsonWildcardTooWide`; two wildcard levels send `N² + N + 1` tokens. -- **Walk budget:** SQL default 64 returned entries across the request. - Exceeding it with a sealed root returns the root entry alone. The client - opens the whole document and evaluates the same paths and probes locally. - Zero disables fallback; an unsealed root cannot fall back and returns the - whole requested walk. This budget is not a hard response-size cap. - -[Expansion and opening](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs), -[SQL budget behavior](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/functions.sql). - -An honest server returns the named node at depth zero plus descendants. -Consequently `{}`, `[]`, a scalar, and an absent path remain distinct. -`DecryptInto>` gives `Some(null)` for JSON null and `None` for -absence; `Vec` gives all wildcard results in document order; singular -`Value` fails on zero or multiple results. In Go use an explicit presence -indicator for a singular read and an explicit result list for a wildcard -read. A nullable Go pointer alone must not collapse JSON null into absence. -[Result implementations](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs). - -## Opening requires trusted request state - -An extracted `JsonNode` / `JsonTreeEntry` does **not** implement standalone -decryption. The selector opens a scalar extraction; the original subtree -key opens walk rows. The subtree key retains concrete paths, bound, probe -positions and the keyset/column machinery needed to recompute selectors; -its JSON serialization contains only the SQL tokens, so it is insufficient -to reconstruct the verification object. -[Engine wrappers](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/mod.rs), -[EQL wrappers](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs). - -The dynamic layer must retain an opaque guest handle or rederive the read -key from a trusted application request (definition, keyset, column context, -path, bound). Database rows must never supply the purported expected path -or context. A stateless rederivation avoids pinning requests to one WASM -instance, but the guest must still reconstruct the same native key and run -the existing opening checks. Do not duplicate these checks in Go. - -The checks reject wrong selectors, inconsistent token/self-tag relationships, -wrong positions or labels, duplicate entries, mixed headers, malformed walks -and incomplete returned subtrees. They do not authenticate the result set: -another document's entries with its own header can open, a field under one -array element can substitute for the same field under another in the cases -documented by the engine, and the database can omit entire requested nodes -or probes. Therefore “verified read” must not promise authenticated absence, -completeness, or row identity. Root fallback must return only the requested -result after client selection, never accidentally hand the whole plaintext -document to a field-read caller. -[Authentication limits](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/docs/reference/json-tree-leakage.md). - -Selector extraction opens scalars, or a sealed root explicitly requested at -`$`; container entries fail because their descendants hold their contents. -The children function returns immediate entries at depth one, not the -subtree result shape. The existing subtree opener cannot simply be applied -to those child rows. A binding should expose them as encrypted entries for -SQL use, or add a specifically request-bound opening operation if plaintext -child enumeration is required. A full subtree read and local child -enumeration is an alternative with different transfer/leakage costs. -[Extraction implementation](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs), -[children SQL](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/functions.sql). - -Child count returns zero for scalars and empty containers, NULL for a missing -node, and NULL for the sealed root. The latter's top-level fields do not -chain from the root token. Do not turn either NULL case into zero. - -## Remaining Go/dynamic work is real - -At this source baseline, the dynamic wire grammar can parse -`{"json":{"sealed_root":true,"ordered":["$.salary"]}}`, but runtime -admission still refuses every JSON index with `json_index_pending`. -`ValueKind` admission assumes scalar indexes. The EQL JSON family is -handwritten and explicitly absent from the catalog/target table. Dynamic -query operands currently distinguish only one term from a range, and -their operator model lacks the JSON-specific path/read inputs. Merely -generating Go types does not make this capability usable. -[Dynamic index grammar](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/term.rs), -[admission](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/kind.rs), -[dynamic queries](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/query.rs), -[EQL family](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs). - -Keep three contracts separate: plaintext JSON transport across the guest, -the existing EQL document/query/read wire forms, and the Vitamin C storage -profile for opaque blobs. A new storage codec for opaque user structs does -not replace `json_tree`'s node encoding or its EQL JSON envelope. The current -EQL/engine conversion still serializes through `serde_json::Value`; direct -typed conversion is a worthwhile format-preserving implementation change. -[Conversions and layouts](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs). - -## Context roles must not be conflated - -The existing `seal_pending` takes one `CallerContext`, obtains its ZeroKMS -descriptor, then uses the same context piece for both the index PRF context -and the AEAD context. The EQL layout accepts only a nonempty `Identifier` -for the column. Dynamic EQL target fields explicitly reject caller -extensions (`TargetError::Extended`) and context-field plans -(`TargetError::ContextField`). Separate row AAD and search scope are therefore -not an existing JSON capability. -[Sealing](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs#L755), -[EQL layout](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs), -[dynamic restrictions](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/record.rs#L73). - -Before freezing a profile that offers row binding, lower distinct column -identity, ciphertext context and search scope into the engine. Never append -a row ID to the present shared context as a shortcut: that also changes -selectors, value terms and read tokens per row. An initial column-only -profile can explicitly refuse those extensions, but must not advertise -unsupported row AAD or reinterpret the same profile later. Tenant keyset -selection is another separate input; it does not provide a replacement for -the context-role distinction. - -For the current client and any later pooled WASI execution, prefer a -host-owned immutable read descriptor -with definition/profile identity, resolved keyset identity, search-scope -inputs, path and wildcard bound. Each open call sends that trusted descriptor -plus untrusted rows, and the guest rederives `JsonSubtreeKey` before opening. -Ciphertext-only row context is supplied separately per document. This avoids -requiring a guest-local handle to survive pool scheduling. The descriptor -must never be reconstructed from database metadata; operation parameters -and column expectations remain application-owned. Reusing a definition ID -for different options/context encodings must be forbidden or versioned. - -One stale source comment in `JsonOptions::sealed_root` says there is no root -token. After #1363 the root has a token; with a sealed root it finds only -the root entry and cannot enumerate top-level members. The module-level -description, implementation and SQL establish the current behavior. - -## Numbers and release fixtures - -#1366 enables `serde_json/float_roundtrip` on Stack Encrypt and tests the -reported one-ULP failure through document, scalar extraction and subtree -opening. This fixes #1364; do not retain it as an unresolved blocker. -It does not make PostgreSQL arbitrary-precision numeric values supported or -make the `f64` ordering encoding lossless for all integers. -[Merged fix](https://github.com/cipherstash/stack/commit/ff287a32887d6a6c07c51d64e61b00deb46eed16). - -The Go contract must preserve signed/unsigned integers through 64 bits and -finite binary64 values without first decoding every JSON number as Go -`float64`. Declare the supported number model and reject unsupported values -instead of silently rounding them. Test Rust-written/Go-opened and -Go-written/Rust-opened documents, not just same-language round trips. - -Release fixtures should cover both root settings; null/absent/empty -containers; nested arrays and quoted keys; containment grouping; equality -versus lossy ordering; concrete and wildcard reads; bound minus one, bound -and bound plus one; token-cap refusal; fallback equivalence; grouped rows -from multiple database records; malicious wrong entries and dropped probes; -and PostgreSQL parameter casts. In particular `->` with an untyped literal -can select native `jsonb` behavior: examples should use typed parameters or -explicit `::text` casts. -[Existing engine tests](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/tests/json_index.rs), -[PostgreSQL oracle](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/tests/encryption/tests/json_tree.rs), -[operator resolution warning](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/operators.sql). +| `contains`, `contained_by` | `{json}` | Containment EQL JSON | +| `eq_at` | `{path,json}` | Containment EQL JSON for a scalar | +| `lt_at`, `lte_at`, `gt_at`, `gte_at` | `{path,json}` | Ordered-leaf EQL JSON | +| `between_at` | `{path,low,high}` | Two ordered-leaf operands | +| `selector` | `{path}` | `text` | +| `subtree` | `{path,wildcard_bound?}` | `text[]` tokens | +| `children`, `child_count` | `{path}` | One node token as `text` | + +Every JSON-valued operand is JSON text. Selector-class operations reject +explicit array positions rather than broadening `[n]` to `[*]`. Use a +subtree for positional reads. Children/count reject wildcard paths. Ordering +requires a configured path and a number/string operand. Equality remains +independent of lossy ordering; numeric ordering can tie large integers and +text ordering folds case and accents. + +`dynamic::json_open` and guest `se_json_open` now rederive the native read +from the original operation/path/bound and the field's options. They open +through the deriving keyset only. A different keyset is refused before data +key retrieval. Returned rows remain grouped by database document; each group +opens independently in the shared batch. The result is a list of JSON texts +per group, preserving missing versus JSON null. +[Opening implementation](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/dynamic/json_query.rs#L419), +[guest ABI](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/guest/src/abi.rs#L574). + +Two boundaries must remain explicit in the release plan: + +- The opener accepts **selector or subtree**, not children/count. A selector + group has at most one entry. All-match selector SQL results can be opened + as separate groups, but that does not verify their joint completeness or + restore document order. Subtree reads handle structured ordered results. +- SQL children/count preparation is available. A parent-bound child-scalar + opening helper from the previous plan is **not implemented** by + `json_open`; child entries are not complete subtree rows. If that convenience + remains required, implement and test it separately rather than marking it + complete with token preparation. A full subtree read can supply plaintext + children with different transfer and leakage costs. + +## Cascade and random storage order change the read contract + +#1379 adds a digest of children's AEAD tags, in sibling ordinal order, to +each entry's AAD. A returned subtree is authenticated from its root through +all descendants. Child omission, substitution or movement beneath a returned +root breaks an ancestor's authentication. The sealed whole-document root +is a separate ciphertext and does not commit to the redundant tree entries. +[Exact cascade](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/json/cascade.rs), +[AAD construction](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/json/derive.rs#L830). + +Stored `sv` slots are randomly permuted, except the root remains slot zero. +Object member ordinals and projection/query-list orders are random too. +The permutation is fresh randomness, not a deterministic encoding of the +plaintext or key. Consequently wildcard `->`, `->>` and ordering on +`ord_term(col -> selector)` select an **unspecified matching element** in +random stored order, not element zero or the first in document order. It +stays the same for that stored row until rewritten. For all matches use +`json_tree_path_query`; for ordered plaintext results use a subtree read. +[Shuffle implementation](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/json/shuffle.rs), +[public leakage/read semantics](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/eql/docs/reference/json-tree-leakage.md). + +A nonempty container no longer opens from `->` because its children's tags +are absent. It returns `JsonNotALeaf`, which also covers an altered entry; +the client cannot reliably distinguish those causes before opening. Empty +container entries authenticate but still return `OpenedType`, since the +extraction interface returns a scalar or sealed root. Use a subtree for +containers. Decrypting a complete document with an open root now additionally +loads the index keyset to reconstruct the chain; settling `Pending` therefore +requires `IndexKeySource` as well as `DataKeySource`. Preserve this requirement +when designing future provider capabilities and no-network test sources. +[Extraction and full-document opening](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/json/derive.rs), +[provider-bound change](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/CHANGELOG.md#changed). + +The cascade does not authenticate the complete query result. The server can +omit an entire requested root or a wildcard probe, answer with another row's +valid header and data, or substitute an equivalent-selector node at the top +of a read in the documented cases. Distinguish **complete descendants below +a returned root** from **complete query matches**. Do not expose an +unqualified authenticated-absence/completeness guarantee. +[Authentication scope](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/eql/docs/reference/json-tree-leakage.md#what-is-authenticated). + +## Limits and SQL behavior retained + +Wildcard expansion still defaults to 64 and is capped at 8,192 total tokens. +Boundary probes cause `JsonWildcardTruncated` when returned; excessive +expansion yields `JsonWildcardTooWide`. Two wildcard levels use +`N² + N + 1` tokens. SQL walk budget defaults to 64 and controls sealed-root +fallback, not a hard response cap. Budget zero disables fallback; an open +root cannot fall back. The client reapplies the original paths and probes +after opening a sealed-root fallback. + +Subtree SQL returns each named root at depth zero and its descendants. +Returned values distinguish missing, JSON null and empty containers. +Children SQL returns immediate entries, while child count is zero for a +scalar/empty container and NULL for absence or a sealed root. Treat the +count as a database observation, not an authenticated cardinality. +[Subtree expansion](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/json/derive.rs), +[SQL signatures and semantics](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/eql/src/v3/json_tree/functions.sql). + +## Remaining profile and context decisions + +The JSON header layout is still version 1, and the entry AAD domain still +ends in `/v1`, despite the pre-release cascade change. Identical envelope +fields do not imply that pre-cascade ciphertext opens under the new AAD. +Freeze the post-cascade vectors and semantics as the customer baseline; +do not claim compatibility from the unchanged header alone. The prototype +compatibility exemption permits this prior change, not future silent changes +to a released profile. +[Header encoding](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/json/mod.rs#L268), +[entry AAD](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/json/derive.rs#L830). + +Ciphertext context and search scope are still not separate JSON capabilities. +`seal_pending` derives both from one caller context; the EQL layout accepts +an `Identifier`; dynamic targets reject extensions and context-field plans. +Keep the column-and-keyset profile explicit. Before enabling row binding, +introduce separate native PRF/AEAD inputs and the corresponding definition +roles. Never append a row ID to the current shared context. Keyset routing +and provider lock policy remain separate concerns. +[Sealing](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/json/derive.rs#L779), +[EQL target restrictions](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/src/dynamic/record.rs). + +## Evidence inspected and release follow-up + +Existing tests now cover configured-target parity and refusals, dynamic +operator shapes, request-bound opening, grouped rows, cascade tampering, +wildcard bounds/fallback and JSON ingress. Their presence resolves the old +claim that these paths have no implementation or tests. It does not replace +running the assembled customer release against the matching guest and SQL. +[Configured target tests](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/eql/tests/encryption/tests/json_tree_data_plan.rs), +[dynamic operators/read tests](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/tests/json_operators.rs), +[read authentication tests](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/tests/json_read_checks.rs), +[JSON vectors](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/packages/stack-encrypt/tests/fixtures/json_tree_vectors.json). + +Update cross-language assertions: ciphertexts, stored slot order and query +projection order are randomized. Compare cross-opening, canonical selector +sets and SQL behavior; exact bytes require the test-only seeded RNG. Remaining +work is release conformance, formalizing the supported dynamic subset and +profile version boundary, deciding the child-opening gap, and the deferred +general API/context design. Reimplementing configured targets and JSON reads +from scratch is no longer an appropriate next step. diff --git a/docs/plans/2026-10-27-go-json-implementation-status.md b/docs/plans/2026-10-27-go-json-implementation-status.md new file mode 100644 index 000000000..bd1721669 --- /dev/null +++ b/docs/plans/2026-10-27-go-json-implementation-status.md @@ -0,0 +1,184 @@ +# Go JSON implementation status after the October 27 merges + +Source baseline: merged `next` at +[`ffeed7adb3f7d8739cda53261046147852bffc82`](https://github.com/cipherstash/stack/commit/ffeed7adb3f7d8739cda53261046147852bffc82). +This reconciles the [Go JSON plan](2026-10-24-go-json-tree.md) with the +implementation. It is source and test inspection, not a fresh test run or a +claim that every release gate has passed. + +## Work that has landed + +The customer path is implemented through the existing Rust engine and guest: +full-document writes and reads; JSON query operators; prepared selector and +subtree reads; generated Go field methods; typed document serialization and +number-preserving decoding; and a documented executable PostgreSQL example. +The plan should move from building its first vertical slice to validating +and closing concrete release gaps. + +The target's registered name is **`JsonSearch`**. Go uses +`eql.JSONSearch`, `eql.JSONSearchQuery`, `eql.JSONSearchLeafQuery` and +`eql.JSONEntry`; SQL uses `public.eql_v4_json_search`, `eql_v4.query_json`, +`eql_v4.query_json_leaf` and `public.eql_v4_json_entry`. `eql.JSON` is still +the storage-only type. The former v4 SteVec names were removed. The proposed +`EncryptedJson` / `eql.EncryptedJSON` names are not the implemented surface. +Do not make another cosmetic naming migration a customer prerequisite. +[Generated EQL types](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/eql/eql_gen.go), +[Go changelog](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/CHANGELOG.md). + +## Declarations and options + +An implemented declaration is: + +```go +type Customer struct { + _ struct{} `stash:"context=customers"` + ID int64 `stash:"id,passthrough"` + Profile Profile `stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"` +} +``` + +The generator emits one shared declaration with `EncryptInto(..., +"JsonSearch")` and `Options(..., gensupport.TargetOptions{...})`. Record +transport writes `target: {name, options}` when options are present and a +string target otherwise. Options currently contain `sealed_root` and +`ordered`; omitted values use native defaults. The native engine validates +the target and paths at generation/runtime boundaries. +[Example model](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/example/json/model.go), +[generated declaration](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/internal/testusers/customer_stash.go), +[record transport](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/internal/record/record.go). + +The tag supports `ordered=$.age;$.visits[*].at,sealed_root=false`. Its path +spelling is deliberately limited to plain member keys and array steps; +quoted keys and keys containing delimiters have no tag spelling. The Go +parser checks that restricted spelling, while the engine validates path +semantics. The structured internal options type can carry a string list, +but that alone does not establish a supported arbitrary-path generator +policy API. A general options schema/catalog, versioned definition wrapper +and cached immutable resolved-definition abstraction remain broader work. +[Tag parser](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/stashgen/tag.go), +[policy API](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/policy/policy.go). + +## Actual query and read surface + +| Surface | Implemented result / behavior | +| --- | --- | +| `Fields.Profile.Contains`, `ContainedBy` | `eql.JSONSearchQuery`; caller supplies an explicit needle | +| `EqAt(ctx, cipher, path, value)` | `eql.JSONSearchQuery`; positional selector paths refused | +| `At.Age.Lt/Lte/Gt/Gte` | `eql.JSONSearchLeafQuery`; declared ordered accessor | +| `At.Age.Between` | `encrypt.JSONRange[eql.JSONSearchLeafQuery]`, `Low` and `High` | +| `Selector(ctx, cipher, path)` | `encrypt.JSONSelection[eql.JSONEntry]` with `Selector()` and `Open(ctx, []*eql.JSONEntry)` | +| `Subtree(ctx, cipher, path, opts)` | `encrypt.JSONRead[eql.JSONEntry]` with `Tokens()`, `Open(ctx, [][]eql.JSONEntry)` and `OpenOne(ctx, []eql.JSONEntry)` | +| `Children`, `ChildCount` | `encrypt.JSONToken`; derives the concrete node token for the SQL function | +| `encrypt.DecodeJSON[T]` | Converts opened raw JSON to an application type, using `json.Number` for interface destinations | + +Prepared reads retain the cipher and original field/path/options request. +Their `Open` method therefore does **not** take the cipher argument used in +the earlier proposed example. Opening sends the trusted request and grouped +untrusted rows through `se_json_open`; Rust rederives verification state. +`JSONSelection.Open` reports `JSONValue{Found, JSON}`. `JSONRead.Open` returns +one list per database row; `OpenOne` refuses a wildcard read before opening, +rather than accepting it whenever a particular result happens to be singular. +The Go wrappers retain no `context.Context`. +[Read types and opening](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/json.go), +[generated query methods](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/internal/testusers/customer_stash.go). + +`JSONSelector` and `JSONToken` implement `driver.Valuer` for text. +`JSONTokens` itself binds a PostgreSQL text-array literal, so the earlier +instruction to add a driver-specific array adapter is unnecessary for this +API. Zero selectors/tokens/lists fail when bound. Token and prepared-read +formatters redact capabilities and paths. Opened `JSONValue` holds plaintext +and does not redact it. + +The low-level query response has been extended, rather than replaced by a +new public sum type: `QueryTerm.Encoding` is `bytes`, `eql`, `text` or +`text_array`; `Terms` carries byte operands and `Text` carries selectors or +tokens. Operation discovery adds an input shape. Generated helpers validate +the encoding, cardinality and fidelity they expect. This implements the +needed distinction; the plan should not require a wholesale response rewrite +solely to match its illustrative tagged-union spelling. +[Query response](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/query.go), +[generated helper checks](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/gensupport/json.go). + +Two limitations must remain visible: + +- `Children` and `ChildCount` prepare SQL operands. They do not provide the + earlier plan's parent-bound plaintext child opener. Immediate child rows + are not a complete subtree and must not be passed to a subtree opener as + though they were. Decide whether that extra helper is required for the + customer; do not mark it implemented merely because child tokens exist. +- After randomized stored entry ordering, a wildcard selector's `->` result + is an **unspecified matching element**, not the first array element. + Generated documentation directs wildcard comparisons through SQL + `json_tree_path_query` with explicit existential semantics and sorting + through the least/greatest matching order term. Subtree opening reconstructs + document order. Update old first-match examples accordingly. + +## Typed JSON and input guarantees + +A JSON field may be `json.RawMessage` (or its alias), or an accepted struct, +map or slice type, optionally behind a pointer with `null=sql`. Native scalar +field types are not accepted as documents; raw JSON can still hold scalar +roots. Plain bytes and a newly defined type over `json.RawMessage` are +refused, avoiding implicit base64 serialization. + +The generator checks whether a typed document can be written and read with +`encoding/json`. It rejects interfaces within document types; dropped or +colliding fields; unsupported channel/function types; unmatched custom +marshal/unmarshal methods; and `json:",string"`. Explicit JSON adapters use +`encoding/json` once, without a preliminary `map[string]any` conversion. +`DecodeJSON[T]` uses `UseNumber`, rejects trailing values, and sanitizes errors. +An interface destination through this explicit decoder is supported even +though an interface-containing generated document declaration is refused. +[Generator documentation and rules](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/cmd/stashgen/README.md), +[typed field admission](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/stashgen/read.go). + +Preserve these documented qualifications in the plan: + +- `encoding/json` replaces invalid UTF-8 in Go strings before Rust sees it. + Rust's strict raw-JSON ingress cannot recover or reject the original bytes. +- Nil maps/slices become JSON null; empty ones become `{}`/`[]`. A nil optional + field pointer becomes SQL NULL. A present raw `null` remains encrypted JSON + null. An empty raw document is invalid. +- Nested `time.Time` is RFC3339 text in its own offset, not a temporal search + domain. Equal instants can have different string terms, and text ordering + is not chronological ordering. Application representations must reflect + the intended query semantics. +- Containment helpers refuse plain Go string needles and needles serialized + as JSON null. For typed fields holding structs, `Contains` refuses a needle + of the field's own type (or pointer), preventing unintended zero-valued + constraints. Maps or dedicated optional-field needle types are supported; + `EqAt` accepts scalar values, including nil as JSON null. + +## Storage and release work + +Opaque generated structs now use `vcffi.MarshalStorage` and +`UnmarshalStorage`, with an old-JSON read fallback. The generated format +marker is `GeneratedVersion2`; generated shape fields use `vc:"name,default"`. +This integration is no longer prospective. The Go module still pins Vitamin +C pseudo-versions. The actual release guard refuses a final Go version +while those pins remain; prereleases may carry them. Regenerate application +code and replace them with released module tags before a final release. Existing +compatibility fallbacks need not be redesigned in this plan refresh. +[Opaque codec](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/gensupport/codec.go), +[module pins](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/go.mod), +[release caveats](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/CHANGELOG.md), +[enforced release guard](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/scripts/go-release-version.mjs). + +## Evidence of coverage, and what was not rerun + +The tree contains guest round-trip and refusal tests (`json_tree_test.go`), +generated raw/typed field tests (`json_fields_test.go`, `json_typed_test.go`), +number-preserving decoder tests, generator admission/refusal tests, and real +PostgreSQL suites (`json_postgres_test.go`, `json_typed_postgres_test.go`). +The PostgreSQL suites skip when `DATABASE_URL` is absent. The Go workflow +builds guests and installs EQL before its PostgreSQL test task. The typed +example includes generated code and a complete database flow. +[Guest tests](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/json_tree_test.go), +[typed PostgreSQL tests](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/json_typed_postgres_test.go), +[CI wiring](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/.github/workflows/tests-golang.yml), +[executable example](https://github.com/cipherstash/stack/blob/ffeed7adb3f7d8739cda53261046147852bffc82/languages/golang/encrypt/example/json/main.go). + +This review did not run those tests, inspect every CI execution result, or +prove all proposed cross-language/storage release vectors are covered. Keep +release verification, formal interoperability specification, supported-driver +coverage and the remaining context/child-opening questions as explicit tasks.