Repository navigation
Conversation
… to queries on its plan A typed plan answers an operator through its target type's `op::Answers` impl, but the chain required `Typed<T>: ValueShape<S>`: a label that converts into `T`'s context. An EQL v4 type is sealed under its column (`NonEmpty<Identifier>`), so no plan for one could be queried, and eql-bindings had nowhere to implement `Answers` (#1238). `op::LabelledTarget` is the opt-in: a target type says its stored terms were derived under the context a plan's label names. For such a type, `Chooses` asks only `ValueLayout<S>` (the indexes `T` declares), since a query derives through the index and never runs `T`'s encryption. Every other type keeps needing `ValueShape`, so a type sealed under, say, a tenant id cannot be queried under a label that would match nothing. This crate's term targets implement the trait, which keeps every query that compiled before compiling. `Chooses` now resolves the plan itself (`choose`) rather than through `Resolves`, which keeps `ValueShape`: a named index (`.via(..)`) on a plan whose type only opts in would send a term the type does not send. An EQL query value stores the column it queries, so its index needs the context its terms were derived under, and must refuse one it cannot record. `QueryToken`, which only the chain makes, now carries the label and whether the call extended it (`label`, `is_extended`): with `.extend(..)` the terms are derived under the label and the extension, so a term that records the label alone would match nothing. The defaulted `Supports::check_label` refuses such a context, and the chain runs it in `check`, where the label is resolved, before any keyset is loaded or term derived (and again in `prepare`, as for `check`). `Supports::query` stays infallible: the chain calls it only after `check_label` accepted the token's context. Every index here takes any context. Tests: a labelled target is queried under the plan's or the call's label, and its term reads whether the call extended it; a label its index refuses loads no keyset; and a UI case pins that a type with an `Answers` impl but no `LabelledTarget` answers no query on its plan.
…query chain Rust could store an EQL v4 type but query it only through a data plan: no v4 type implemented `op::Answers`, so `cipher.query(op::Eq, v).on(&plan)` did not compile on a plan for `TextEq`, while the Go SDK already had an `Eq` method per equality type and `Matches` with `Verify` on `TextMatch` (#1216). This gives Rust the same operators, with the same bytes (#1238). eql-codegen writes into `v4/answers.rs`, for each producible type with a query twin: `op::LabelledTarget` (it is sealed under the column a two-segment label names), the twin's assembly, and one `op::Answers` impl per operator. The operators come from the type's index keys, the input the data path's `TargetDescriptor::operators` reads: `"eq"` answers `Eq`, `"match"` answers `Matches`. It stops on any other index, or more than one, so making an ordered type producible has to extend the rule. The same rows give `ANSWERS`, which the `typed_queries` test holds to `TargetDescriptor::operators()` for every type. The file sits beside the target table rather than in it: `v4/targets.rs` is built on Stack Encrypt's `dynamic` layer and needs the `targets` feature, while a typed query needs neither, so `answers.rs` needs only `stack-encrypt`. The index each impl names is `TwinIndex<I, Q>`: Stack Encrypt's own index `I` (the same spec and derivation), whose query term is written into the twin `Q` with the column the query's label names, by the encodings the stored values use. It answers only for the twin's own plaintext (`QueryTwin::Plaintext`), so no plan can send one type's twin for another type's value, and its `check_label` refuses a label that names no column and any extension before any keyset is loaded: an extended query would be derived under more than the column the twin stores, and match nothing. Both refusals are a `QueryLabelError` in `Error::Other`, so a caller matches the variant rather than the message. `TextMatch` returns `MatchTwin<Q>`, which serializes as the twin and keeps the query's tokens for `verify` and `target_limit` under `EqlMatch`. `Hmac256` and `BloomFilter` now declare their indexes, so a type holding one declares the index a query is checked against, and `EqlMatch` implements `MatchCase`. Tests: every typed query is the data path's `query_term` bytes and domain (the float values the semantic baseline folds included), with the data path's inputs built as frames of Rust values; `TextMatch`'s `verify` and `target_limit` agree with the data path's; short text, a non-column label and an extended query are refused before any keyset is loaded; and compile-fail doctests for `op::Lt` on `TextEq`, `op::Matches` on `IntegerEq`, `.via(Equality)` and another type's `TwinIndex` on a `TextEq` plan, and `verify` on an exact term. The new module's example runs as a test in the encryption crate.
…nd pin the twin's plaintext tie `op::Answers` and the stack-encrypt CHANGELOG read as if only a type sealed under a context no label names needed `op::LabelledTarget`. Any type queried on a typed plan needs both, whatever its context; both now say so. The cross-type `TwinIndex` compile-fail doctest is rejected by `Resolves`' `ValueShape` bound before the plaintext tie is reached, so it would still pass if `QueryTwin::Plaintext` were removed. A doctest pair now checks the tie alone, through a `Supports<op::Eq, String>` bound: `TwinIndex<Equality, TextEqQuery>` meets it, and `TwinIndex<Equality, IntegerEqQuery>` fails with `<IntegerEqQuery as QueryTwin<Hmac256>>::Plaintext == String`.
feat(eql-bindings): query EQL v4 types through Stack Encrypt's typed query chain
…perator reads
eql_v4.query_text_search required hm, op and bf on every query value. bf
is the bloom filter built from 3-grams, so text shorter than one 3-gram has
none, and `col = 'Al'` or `col < 'Al'` could not be written against an
eql_v4_text_search column. That is why Stack Encrypt could not produce the
type.
In v4 the search query domains (query_text_search and its _ore twin) now
require at least one of their terms. Each extractor over them reads its
term through eql_v4_internal.require_query_term, which raises
invalid_parameter_value naming the domain, the term and the operators that
compare it when the term is absent, so `=` on a value with no hm, `<` with
no op and `@@` with no bf are refused rather than reading NULL and
matching nothing. The extractors stay LANGUAGE sql, so the wrappers still
inline and the eq_term/ord_term/match_term indexes still engage.
v3 is cipherstash-client's format and does not change. The SQL source is
rendered once per version and had no way to say "only in v4", so the
renderer gains version blocks: lines between `-- {{#v3}}`/`-- {{/v3}}`
or `-- {{#v4}}`/`-- {{/v4}}` render for that version only, and the
directive lines for neither. The generator emits them for the two search
query domains, and the helper sits in a v4 block of
src/v3/scalars/functions.sql. release/cipherstash-encrypt.sql and its
uninstaller are byte-identical to the build before this change.
ADR-0002 gains a dated amendment recording the decision.
…s operator TargetResolver::query took a type name, a label and a value, so a resolver had to build the type's whole query twin, every term of it, for every operator. EQL v4's TextSearch needs to emit only the term its operator reads: a match term cannot be made from text shorter than one 3-gram, and `=` or `<` on such text must not be refused for want of one. query_term already knows the answering index (OpSpec::answered_by over the type's indexes). It now passes it through record::query to TargetResolver::query as an &IndexSpec after the type's name. A resolver whose twins carry every term ignores it. eql-bindings' encryption::targets::query takes it the same way and refuses an index the type does not carry (TargetError::NoIndex) before deriving anything. The Go guest forwards it; an index the type lacks can only come from an engine bug, so the guest reports it as internal. BREAKING CHANGE: TargetResolver::query and dynamic::record::query take an &IndexSpec; eql_bindings::encryption::targets::query takes one after the name.
…perator's term TextSearch was unproducible because its query twin carried hm, op and bf on every query, and text shorter than one 3-gram has no bf: `=` or `<` on a two-letter name would have been refused. EQL v4's search query domain now takes any one term, so the twin can carry just the one its operator reads. In v4, TextSearchQuery (and the unproducible TextSearchOreQuery) hold each term as an Option, skipped in the JSON when None. The generator emits one projection per term, TextSearchEqualityQuery, TextSearchOrderQuery and TextSearchMatchQuery, each deriving EncryptFrom<String> over the envelope and its one term and converting into TextSearchQuery. The by-name query dispatch, which now receives the answering index, runs the projection for that index, so a match term is derived only for `matches`. The twin itself carries no derive. The stored TextSearch carries all three terms, the same bytes TextEq, TextOrd and TextMatch store, so text too short for a 3-gram is refused on write with EmptyTermText, exactly as TextMatch refuses it: a stored row with an empty filter would be one no match query finds. TextSearch joins ENCRYPTION_DOMAINS, the regenerated target table marks it producible, and the Go encrypt/eql table follows. The engine answers eq, lt, lte, gt, gte and matches on it. Tests: each index's query carries its term alone and equal to the stored one; "Al" is queried by eq and ope and refused for match; the typed projections serialize to what the dispatch writes; and, against an EQL v4 database, `=`, `<>`, `<`, `>=`, `@@` and the function forms find the plaintexts' rows, a one-term operand is a query_text_search and a termless one is not, and each operator and extractor refuses an operand missing its term with invalid_parameter_value naming the term and never the operand.
…ng one term A directory Entry model holds one TextSearch field. Generated over the deterministic EQL guest, its stored value carries bf, c, hm, i, op and v and opens back; Eq, Lt and Gte return a query with only hm or op, the stored term, for "Alice" and for the two-letter "Al"; Matches returns one with only bf, which Verify checks; and Matches on "Al", like storing "Qz" or "", is refused with ErrTerm without quoting the value. stashgen gets a golden case for the same field over its static engine, which now lists TextSearch with its three indexes, so the generated methods (Eq, Lt, Lte, Gt, Gte, Matches) are pinned and compile against the stub SDK.
…index `IndexSpec::OreBlock` (key `ore_block`) derives ore-rs 0.9.0's 6-bit block ORE under its own key domain, over the order keys the CLLW indexes use: `ore_rs::vitaminc::Bit6BlockOre` for targets of at most ten bytes (every integer to 64 bits, both floats) and `ChainedBlockOre` for text, bytes and the 128-bit integers. The legacy 8-bit scheme is not used. `sem::OreBlockTerm` is the whole ciphertext. It carries a random nonce, so it compares by the scheme's comparator, never by its bytes, and `from_bytes` checks a stored term's structure. A query is the full ciphertext too: EQL v4's comparator reads the right half of the operand a btree index passes second. Text whose fold is longer than 1023 bytes or ends in NUL is refused, on the query path before any key is touched. The CLLW `Ore` and `Ope` indexes are unchanged, byte for byte. BREAKING CHANGE: `IndexSpec` is exhaustive and gains `OreBlock`, so a `match` over it needs an arm for it.
… chain and exact containment Stack Encrypt had no JSON index: `op::Contains`, `op::ContainedBy` and `op::AtPath` were declared (#1209) and answered by nothing, and the v4 JSON wire format was unconstrained by anything shipped. This is Phase 1 of the spike in docs/plans/2026-10-17-json-index-stack-encrypt-spike.md (#1249). `Json` is an `Index<serde_json::Value>` whose term, `JsonTerms`, is the document's one-way terms: a path selector and a value selector per node, a token per node below the chain's heads, the grouping by container array element, and a leaf ordering term on numbers and strings. Sealing is the separate operation `target::json_document`, which flattens the document, requests one data key under the column's descriptor and seals every node's own label, kind and scalar under it, so a document's ciphertext is linear in its size. The stored `JsonDocument` carries the design's chain fields: position-keyed parent tags, wrapped tokens, and group tags salted by the document's iv, so nothing links across rows until a query supplies a token. By default the root is sealed: the root entry is the whole document and the top-level fields are the chain heads, so no token exists for the root. Queries go through the engine's own operators. `Supports<Contains>` and `ContainedBy` send a `JsonQuery`, the element-nested projection of the needle's selectors, which native `jsonb @>` compares exactly across array elements (the v4 half of #1231). `AtPath<Eq>` sends the projection of the needle document that holds the value at the path; the orderings at a path send a `JsonLeafQuery`, the selector and the leaf's OPE term; `Between` a `RangeQuery` of two. `Input<Pathed<One>>` and `Input<Pathed<Pair>>` are implemented for `(path, value)` and `(path, (lo, hi))`, which #1209 left open until the index existed. `JsonDocument` is an `EncryptFrom<Value>` target and a `DecryptInto<Value>`, and an extracted node (`JsonNode`) or the rows of a subtree walk (`JsonSubtree`) open on their own. `IndexSpec::Json(JsonOptions)` is the data form, key `"json"`, with the object form for a non-default `sealed_root`; a data plan reads it and refuses to run it with `Reason::JsonIndexPending`, since the binding's half is not written. The data form, the dynamic arms and the Go guest's and fuzz crate's lockfiles move with it (stack-encrypt gains `sha2`, already in the graph through vitaminc-hmac). Two deviations from the design are recorded in the module docs and the plan: nonces are the AEAD's own random ones rather than derived, so the crate keeps one leaf cipher, and the ordinal field is `o`, since `i` is the EQL envelope's identifier. Refs #1249, #1231, #1209, #1060
…, EQL v4 only A new family beside `public.eql_v4_json`, not a port of the SteVec surface: `public.eql_v4_json_tree` (the document), `_json_tree_entry` (one node, as the extractors return it), `eql_v4.query_json_tree` (a containment needle) and `eql_v4.query_json_tree_leaf` (an ordering operand at a path). Phase 2 of the spike in docs/plans/2026-10-17-json-index-stack-encrypt-spike.md (#1249). Confined to v4 with `-- ONLY EQL VERSION: 4`, so the v3 bundle is unchanged; the build's ordering, completeness, symbol-order and isolation gates pass as they are. Everything the server does is core PostgreSQL. Containment projects a stored document from its group tags into the element-nested shape the needle has (`eql_v4.json_tree_query`, a recursive plpgsql group builder) and compares the two with native `jsonb @>`, so each needle array element must fit inside one stored element: PostgreSQL's own rule, which closes the cross-element false positives of selector-set containment. A GIN index on that expression serves `@>`; the function is deliberately not STRICT, because a STRICT SQL function whose body is an `OR` is not inlined and the index never engaged until it was changed. The token chain is walked by a recursive CTE from one client-derived token, with `sha256`, `set_byte` and a sixteen-byte XOR: `json_tree_subtree`, `json_tree_subtree_at` (gated on a selector for a wildcard below the anchor), `json_tree_children` and `json_tree_child_count`. `->` by selector grafts the header and the entry's ordinal, so the entry opens on its own; `ord_term` serves a functional btree for the orderings. Every other native jsonb operator on the column is blocked, as on the other domains. `eql-bindings` gains the hand-written `v4::json_tree` types, with Stack Encrypt's `EncryptFrom<Value>` / `DecryptInto<Value>` for the document, the entry and a subtree's rows, under an `Identifier`; the family is not in the catalog yet, so it is not in the target table or `v4::all()`. The oracle is `tests/encryption/tests/json_tree.rs`, run by `mise run test:encryption:postgres`: the design's corpus and random documents stored encrypted beside their plaintexts, with `@>` and `<@` over the encrypted column checked against PostgreSQL's own `jsonb` operators over the plaintext column for every needle, the GIN index shown in `EXPLAIN`, extraction, the walk, the array functions and the orderings. This is Phase 0's oracle folded into Phase 2, against the real SQL and real encryption under the fake key source, so it runs on a fork PR. Refs #1249, #1231
…its design The implementation notes for Phases 1 and 2: where the code is, which of the ten decisions it took, and the deviations a reviewer reading the design first should know before the code (random nonces, the `o` ordinal, the walk excluding its anchor, the non-STRICT containment, the `<@` gap for a stored root scalar, byte-for-byte string selectors, and what the data plan and the catalog still lack). The agent guides point at the family's four homes. Refs #1249
🦋 Changeset detectedLatest commit: 7d65ad9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
…ional-terms feat(eql)!: EQL v4 text search queries carry only the term their operator reads; produce TextSearch
EQL v4's `ob` is one ore-rs ciphertext: the fixed-width 6-bit scheme
(header `02 02`) or the chained one (`02 03`, then the padding policy).
v3's is cipherstash-client's 8-bit blocks, which need another comparator,
so the v4 SQL holds its own in a version block:
- `{{ore_block}}_hash_bit`: ore-rs's sigma-MMO hash, the GF(2^128)
doubling on two bigint halves with no branch on the data, then AES under
the public key `ORE-rs.v2.H-pi.1` through pgcrypto.
- `{{ore_block}}_scheme`: the shape checks ore-rs's comparators make
(header, block count, exact length, padding policy and the block counts
it can produce, every symbol below 64). A malformed term raises, naming
its length and never its bytes.
- `compare_{{ore_block}}_term`: raises on terms of different schemes,
fixed-width terms of different block counts and chained terms padded
under different policies. For terms of one shape it runs a constant
operation count: one statement per block evaluates both comparisons and
latches the first difference arithmetically, the hash runs once on every
path, and the result is selected without a branch. Chained terms compare
lexicographically, the shorter first when its blocks all match.
- `compare_{{ore_block}}_terms` over an array requires exactly one term.
The v4 bundle does not reuse the 8-bit format's name. The renderer gains a
placeholder, `{{ore_block}}`, rendered `ore_block_256` in v3 and
`ore_block_v2` (ore-rs's wire version 2) in v4, and refuses a literal
`ore_block_256` outside the source directory's path. The hand-written SQL,
the pin script, the generated extractors and the ORE fallback take it.
`release/cipherstash-encrypt.sql` and its uninstaller are byte-identical
before and after.
`tests/timing/ore_block_v2/` is a hand-run timing harness for the v4
comparator, with a detached ciphertext generator. ADR-0002 is amended.
An EQL type whose `ob` term is block ORE lists the index `ore_block` (`record.OreBlock`) in `se_targets`, and the engine answers its orderings through it. The checker accepts it in a type's indexes and as an operator's index, and stashgen maps it to `IndexOreBlock`. It is not a tag word: a field reaches it only by naming such a type with `encrypt_into=`, so it has no entry in `indexNames` or `outputOfIndex`, and a plan's own outputs still refuse it. The static test engine produces `TextOrdOre`, `TextSearchOre` and `IntegerOrdOre`, so the refusal test names `BigintOrdOre` as the type it does not produce.
`IntegerOrdOre`, `SmallintOrdOre`, `BigintOrdOre`, `RealOrdOre`, `DoubleOrdOre`, `TextOrdOre` and `TextSearchOre` are producible. Their `ob` term is Stack Encrypt's block ORE term, one hex element holding the whole ore-rs ciphertext: `OreBlock256` implements `EncryptFrom` for any orderable plaintext and never decrypts. `TextSearchOre` follows `TextSearch`: its stored value takes the empty filter, and its query carries only the term its operator reads. In the target table an `ob` term is the index `ore_block`, not `ore`, which is CLLW ORE and has no EQL domain, and the block-ORE reason in `encryption_gap` is gone. The unproducible types are the date, timestamp and numeric families, which have no Rust plaintext type yet, and the two JSON types. `tests/encryption/tests/v4_ore_block.rs` compares real ore-rs ciphertexts in PostgreSQL, both ways round, against the plaintext order and ore-rs's own comparator: equal values, a first difference in every block, shared prefixes and text in three padding buckets. It checks the hash bit against ore.rs's vectors, that malformed and mismatched terms raise, and that a btree index answers `<`, `>=`, `=` and `ORDER BY`. `test:encryption:postgres` runs it. The Go catalog, the guest's tests and the Go operator test follow. ADR-0009 is amended. The EQL v4 range tests (tests/encryption/tests/v4_range.rs) now derive their order terms from the engine. They arrived from the range domains with cipherstash-client's ORE and OPE indexers writing the `ob` and `op` of every row and bound, which was the only source of block ORE terms while EQL's ordered types were not producible. The comparator this series adds refuses that 8-bit layout, and this commit is where the `OrdOre` types become producible, so the rows are real `IntegerOrdOre`, `TextOrdOpe`, ... values and each bound is the term the single-bound query of that type carries. The smoke test ranges over the families whose ordered types are producible and records that the date, timestamp and numeric ones are not.
…tence test `v3_and_v4_install_side_by_side` asserted that eql_v3 and eql_v4 own the same number of public domains. That held while the two bundles were one SQL tree rendered under two names. The JSON tree family under `src/v3/json_tree/` is confined to v4 with `-- ONLY EQL VERSION: 4`, so v4 now owns `public.eql_v4_json_tree` and `public.eql_v4_json_tree_entry` with no v3 counterpart, and the count assertion failed in CI (52 vs 54). The test now compares the domain sets by name: every v3 public domain must exist in v4 under the v4 prefix, and the domains v4 owns that v3 does not must be exactly the JSON tree family. A future v4-only domain fails the test until it is named here, which is the point: EQL v3's SQL must not change, and a domain leaking from v4 into v3, or a v3 domain missing from v4, is a build regression rather than a design decision. Verified by installing both built bundles into a scratch PostgreSQL 17 and diffing the sets with the same query: 52 v3 domains, 54 v4, nothing missing from v4, and the two JSON tree domains as the only surplus. Refs #1288
Contributor
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 304 | 0 | 83 | 0 |
Every mutant in the changed lines was caught by a test.
feat(eql)!: produce the block ORE EQL v4 types through ore-rs 6-bit block ORE
This comment has been minimized.
This comment has been minimized.
The Mutants gate on this branch reported 53 surviving and 6 timed-out mutants across stack-encrypt's json module. Each is now caught, removed or excluded: - The path parser and the tree rebuild indexed their input by hand, so a `+= 1` mutated to `*= 1` never advanced and the test run hung. The parser now walks a shrinking &str and the rebuild pulls children from an iterator, so every step consumes input and no such mutant exists. - New tests pin what the survivors showed was unchecked: refusal offsets and reasons for malformed paths, JSON escapes in quoted names, the integer/float boundary in number encoding, every truncation of a header and of a node label, the key tag's u16 length limit, the header and selector Debug forms (a subtree key never prints its token), stored wrapped tokens differing from the token, opening each stored form as a record field, documents without a sealed root, and accessor queries refused before any keyset load. - Four mutants are equivalent and are excluded in .cargo/mutants.toml with the reason beside each, matched on the specific replacement. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
This was referenced Oct 21, 2026
Open
coderdan
added this pull request to stack #1333
October 22, 2026 10:15
coderdan
commented
Oct 22, 2026
Comment on lines
+257
to
+267
| /// The keyset and the layout; the key material's identifiers are not | ||
| /// secret, but a header prints as what it is for. | ||
| impl fmt::Debug for JsonHeader { | ||
| fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { | ||
| f.debug_struct("JsonHeader") | ||
| .field("keyset_id", &self.keyset_id) | ||
| .field("sealed_root", &self.sealed_root) | ||
| .field("epoch", &self.epoch) | ||
| .finish_non_exhaustive() | ||
| } | ||
| } |
Contributor
Author
There was a problem hiding this comment.
Can OpaqueDebug not be used?
Contributor
Author
There was a problem hiding this comment.
Yes. vitaminc-protected 0.6.1 supports #[non_sensitive] per field, so the derive covers it: keyset_id, sealed_root and epoch are marked non-sensitive and print, and the ZeroKMS iv and tag print as ***. Switched to #[derive(OpaqueDebug)] in 7d65ad9, which also gives the header the OpaqueDebug marker. The printed text changes (full type path, masked fields listed), so the unit test that pins it now asserts the masking too.
coderdan
commented
Oct 22, 2026
| /// # Errors | ||
| /// | ||
| /// [`JsonPathError`] for text that is not one. | ||
| pub fn parse(text: &str) -> Result<Self, JsonPathError> { |
Contributor
Author
There was a problem hiding this comment.
This would fail a CRAP gate - but maybe there isn't one that runs for this code?
Contributor
Author
There was a problem hiding this comment.
The gate does run on this PR: crap-crates.yml triggers on any packages/stack-encrypt/** change whatever the base, and "CRAP (stack-encrypt)" passed on it. Measured locally with mise run crap:stack-encrypt on 7d65ad9: JsonPath::parse scores CRAP 10.0, cyclomatic complexity 10 at 98% coverage, against the threshold of 30. Untested it would score about 110, so the path tests are what keep it under. No function in the crate exceeds 30, so I have left it unchanged. Splitting the [ step into a helper would still be a fair readability change if you want it.
The hand-written impl printed the keyset id, the root flag and the epoch and left the ZeroKMS iv and tag out. vitaminc's OpaqueDebug derive does the same job declaratively: the three layout fields are marked `#[non_sensitive]`, and the iv and tag, which retrieve the data key, print as the mask. The masking is now part of the type's declaration rather than a convention a later edit to the impl could drop, and the header carries the OpaqueDebug marker like the crate's other secret-adjacent types. The printed text changes (the derive names the type by its full path and lists the masked fields), so the test that pins it is updated to the new text and now asserts the masking too. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The JSON index for Stack Encrypt (our Rust encryption engine), and the EQL v4 domain family that stores and queries what it writes. EQL is the SQL library we install into a customer's PostgreSQL so it can search encrypted values. Before this, Stack Encrypt could not encrypt a JSON document searchably at all: the JSON operators existed but nothing answered them. After it, a JSON document encrypts into one small ciphertext per node (so the ciphertext grows with the document, not with the document times its depth), the server can walk the subtree under a node from one token the client derives from a path, and containment (
@>) agrees with PostgreSQL's ownjsonb @>across array elements, which the v3 design gets wrong (#1231).This is Phases 1 and 2 of the spike #1249, built against the design in #1250 (this PR is stacked on it: read the design first, then this). Phase 0's oracle is folded into Phase 2. Phases 3 (measure) and 4 (decide the wire) are not here.
Changes
Engine (
packages/stack-encrypt)src/json/: theJsonindex (Index<serde_json::Value>), the sealing operationtarget::json_document, the storedJsonDocument(anEncryptFrom<Value>/DecryptInto<Value>target), the query terms (JsonQueryfor containment andAtPath<Eq>,JsonLeafQueryfor the orderings at a path), the extraction keys (JsonSelector,JsonSubtreeKey, derived on the keyset cipher), and what an extraction opens (JsonNode,JsonSubtree). The module docs are the reference for the wire form and the derivations.op:Supportsimpls forContains,ContainedBy,AtPath<Eq | Lt | Lte | Gt | Gte | Between>, and theInput<Pathed<One>>/Input<Pathed<Pair>>impls for(path, value)and(path, (lo, hi))that feat(stack-encrypt): JSON operators Contains, ContainedBy and AtPath #1209 left open.JsonDocumentanswers them, and they default toJson.IndexSpec::Json(JsonOptions), key"json", with the wire form{"json": {"sealed_root": bool}}. The data plan reads it and refuses to run it with a new reason,json_index_pending.TermErrorvariants for a bad path, a document too deep or wide, the sealed root's missing token, and an ordering on a non-orderable leaf.sha2, already compiled in the graph throughvitaminc-hmac; the four lockfiles that record the edge move with it.EQL (
packages/eql)src/v3/json_tree/{types,functions,operators,blockers}.sql, confined to v4 with-- ONLY EQL VERSION: 4: the four domains, the containment projection and operators,->, the subtree walk, the array functions, the ordering extractors, and the native-jsonb firewall. The v3 bundle is byte-for-byte unchanged.crates/eql-bindings/src/v4/json_tree.rs: the hand-written types with Stack Encrypt's targets. Not in the catalog yet, so not in the target table orv4::all().tests/encryption/tests/json_tree.rs, added tomise run test:encryption:postgres..changeset/eql-json-tree-domain.md(@cipherstash/eqlminor).Docs
Verification
cargo nextest run -p stack-encrypt --all-features: 716 tests pass, including the 14 new ones intests/json_index.rs: round trips under both root options, the chain unwrapping every child from its parent's token and nothing from a wrong one, tamper evidence (reordered, regrafted, truncated, a flipped byte, another column, a lifted node), and containment over the projection agreeing with a Rust model ofjsonb @>on the design's corpus and 300 random pairs. The model is pinned to the 23 answers injsonb-semantics.sql. Clippy clean with all features and with none; rustdoc clean with warnings as errors; the doc examples pass; the trybuild snapshots were regenerated and differ only by the new implementors they list.mise run test:encryption:postgresagainst PostgreSQL 17, with the v4 bundle built from the tree: all six suites pass. The new one stores 72 documents encrypted beside their plaintexts and, for 89 needles, checks that@>and<@over the encrypted column select exactly the rows PostgreSQL's ownjsonboperators select over the plaintext column, thatEXPLAINshows aBitmap Index Scanon the GIN index, that->opens to the leaf, that the walk from a client-derived token returns the six entries under$.nestedand reassembles them, that a wrong token unlocks nothing in any row, that a wildcard path walks each element's subtree, that the array functions count and list elements, that the orderings at a path select and sort as the plaintexts do, and that a native operator on the column raises.mise run test:encryption(both feature sets, clippy with warnings as errors),check:encryption:wasi,test:crates,build,codegen:parity,types:check,docs:validate:coverage,docs:validate:required-tags, anddocs:validate:documented-sql(the whole v4 install order in one transaction): all pass. The Go guest checks and lints with itseqlfeature under--locked. The three EQL CI-derivation tests underscripts/__tests__pass.STRICTcontainment function with anORbody is not inlined by PostgreSQL, so the GIN index never engaged until it was made non-STRICT (recorded in the SQL and the design notes); and an untyped literal on the firewall test resolved to the native operator, as the domain docs warn.stashgen, and the TypeScript export, which the design keeps out of the spike.Related
Refs #1249 (the spike; Phases 1 and 2)
Refs #1231 (the v4 half: exact containment across array elements)
Refs #1209, #1060
Stacked on #1250
Review notes
Read the design's new "Implementation notes" section first; it lists the deviations in one place. Then
packages/stack-encrypt/src/json/mod.rsfor the wire form,derive.rsfor the walk and the sealing,query.rsfor the projection, andpackages/eql/src/v3/json_tree/functions.sqlfor the server side of the same three things. The oracle intests/encryption/tests/json_tree.rsis what holds the two sides together.Deliberately deferred, with the reason in the design notes: a self tag per node (so a walk could return its anchor), the data-plan form, the catalog entry, and wiping the plaintext copies the walk makes.