Repository navigation
feat(eql): one-parameter jsonpath predicates on json_tree documents (json_tree_match, @@, match) - #1446
feat(eql): one-parameter jsonpath predicates on json_tree documents (json_tree_match, @@, match)#1446coderdan wants to merge 3 commits into
Conversation
🦋 Changeset detectedLatest commit: 6109e38 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 226 | 0 | 51 | 0 |
Every mutant in the changed lines was caught by a test.
| //! The grammar and the compilation are documented with the operator, in | ||
| //! the [`json_query`](super::json_query) module docs. This module holds the | ||
| //! parser ([`parse`]), which reads the text into an [`Ast`] and refuses | ||
| //! what the grammar does not read at the byte where it stops, and the | ||
| //! compiler ([`compile`]), which binds the values and maps each comparison | ||
| //! onto the inputs `eq_at`, `selector`, the orderings and `element_match` | ||
| //! take, so that every leaf is derived by that operator's own code | ||
| //! ([`derive`]). |
There was a problem hiding this comment.
Please include examples (including doctests) of the syntax as well as a link to the relevant postgres docs.
There was a problem hiding this comment.
Done in 6a31ad7.
- The
json_predicate.rsmodule docs now open with worked examples of the accepted syntax and a link to PostgreSQL's SQL/JSON path language. OpSpec::Match's rustdoc carries the same link and a doctest. The doctest runs three accepted predicates throughcheck_queryon a document field: a variable, a quoted key with!= null, and an element filter. It also shows two refusals,like_regexby reason and arithmetic by byte offset.
94db05b to
6a3419d
Compare
|
Force-pushed after the mutation run: the stack-encrypt commit now also carries the parser tests that kill the 15 mutants the first run missed (nesting depth, a second filter, |
| descriptor = descriptor.with_elements_query_sql_domain(domain); | ||
| } | ||
| if let Some(domain) = target.predicate_query_sql_domain() { | ||
| descriptor = descriptor.with_predicate_query_sql_domain(domain); |
There was a problem hiding this comment.
Go guest never wires predicate_query_sql_domain (or elements_query_sql_domain). This test resolver names all three domains, but the production resolver the Go SDK uses — languages/golang/encrypt/guest/src/targets.rs describe() (line ~118) — only calls with_leaf_query_sql_domain. So through the Go path json_query::domain(target, Binds::Predicate) is None, query_term returns TargetError::NoQuery, and match (and element_match) is refused and absent from operators(), even though eql.JSONSearchPredicateQuery was generated for Go in this PR. The module doc here says this is "the path the Go guest takes", but it is not the same resolver. Add the two with_* calls in the guest describe().
There was a problem hiding this comment.
The comment is fixed. Wiring the guest is deliberately left to #1442/#1443, per the milestone's order: Rust and EQL first, then Go on top.
- Comment: the
elements.rsandpredicate.rsmodule docs no longer claim this is the Go guest's path. They now say the test's resolver names all three domains, and that the Go guest's resolver names only the leaf domain. - Why not wire it here: naming the domains in the guest's
describe()would putelement_matchandmatchinto the Go SDK's operator catalog. The Go side does not parse those operators yet (parseOperatorName, stashgen'sjsonOperators, the input shapes inchecker.go), so every catalog read would fail withunknown operator. - The gap is now visible instead of silent: a new guest test,
targets::tests::a_document_names_the_leaf_domain_and_not_the_predicate_domains, pins thatJsonSearchnames the leaf domain and neither predicate domain, and that neither operator is listed. Go SDK: no way to test an equality and a range on the same array element of a JsonSearch field — add an element-scopedMatchon top of #1440 #1442/Go SDK: addMatch(ctx, cipher, predicate, vars)on a JsonSearch field, binding one jsonpath-style predicate as one parameter, on top of #1441 #1443 must update it in the same change that teaches the Go side the operators.
Done in 6a31ad7.
| at | ||
| }, | ||
| ); | ||
| if full.len() > MAX_DEPTH { |
There was a problem hiding this comment.
A single-equality filter whose base + member path exceeds MAX_DEPTH returns term(TermError::JsonTooDeep) (an Error::Cipher), while the PR/module docs say a path deeper than a document is Error::Predicate { reason: JsonPredicateTooDeep, at }. member_of has the same shape for the multi-condition filter. A client mirroring the refusals (#1443) gets a different error type with no offset for this one case; use the Refusal { JsonPredicateTooDeep, C_NESTING, at: path.at } form as steps() does.
There was a problem hiding this comment.
Fixed in 6a31ad7.
- The compiler checks the depth of base + member for every condition of a filter, before anything is derived.
- A filter's joined path deeper than a document is now
Error::Predicate { reason: JsonPredicateTooDeep, construct: "nesting", at: <the filter's path> }. This holds for the single-equality filter and the multi-condition (element_match) filter alike, so neither surfacesTermError::JsonTooDeepany more. - Tests cover both forms, and the path exactly as deep as a document still compiles.
| })) | ||
| } | ||
| Some("last") => Err(Refusal::unsupported(C_LAST, at)), | ||
| Some("strict" | "lax") => Err(Refusal::unsupported(C_MODE, at)), |
There was a problem hiding this comment.
Refusal classification diverges from how PostgreSQL reads keywords. PG matches strict, lax, last, like_regex, starts, is case-insensitively (only true/false/null must be lowercase), but only exists goes through keyword(); STRICT $.a == 1 or $.a LIKE_REGEX "x" therefore surfaces as json_predicate_syntax / "text the grammar does not read" instead of json_predicate_unsupported with the named construct. Same for exists(...): after_operand() is not called there, so exists($.a like_regex "x") / exists($.a + 1) fall to expect(")") and are reported as syntax errors. Not a wrong answer, but a client mirroring the reasons/constructs will classify these differently.
There was a problem hiding this comment.
Fixed in 6a31ad7.
strict,lax,last,like_regex,startsandisare matched case-insensitively, as PostgreSQL's scanner reads them.true,falseandnullstay lowercase-only, as PostgreSQL requires.- The
exists(...)operand now gets the same check for the constructs that follow an operand, both after a bare path and after a filter. - New cases, each asserted as
json_predicate_unsupportedwith its construct and offset:STRICT $.a == 1$.a LIKE_REGEX "x"$.a Starts with "x"$.a IS unknown$.a[LAST]$.a == Laxexists($.a like_regex "x")exists($.a + 1)exists($.a[*] ? (@ == 1) starts with "x")
| Each leaf is checked by the CHECK of its own domain, and `and` and `or` take at least one child. | ||
|
|
||
| - **The answer is the hand-written WHERE's, NULL included.** `and`, `or` and `not` follow SQL's three-valued logic, and a `cmp` leaf is NULL where the path is absent or its node has no ordering term, as `(col -> s) < $1` is. Stack Encrypt compiles a jsonpath comparison so that the tree answers as `jsonb_path_match` does over the plaintext: `$.age < $n` is a node at `$.age`, not a JSON `null`, whose leaf compares; `$.age != $n` is a node there that is not the value. So an absent path is false under every comparison and `null` false under an ordering, as jsonpath has them, and an ordering of a boolean is unknown (NULL), as jsonpath's comparison of two kinds is. | ||
| - **Where the tree and jsonpath part.** A path is read as written, so an array where the predicate names no `[*]` is not unwrapped as jsonpath's lax mode unwraps it. A comparison of a string with a number, which jsonpath answers unknown, is false under `==` and true under `!=`, since the terms do not record a value's kind. An ordering compares the ordering terms, so text is case- and accent-folded, as every ordering at a path is; so does each condition of a filter with more than one, which compiles to an element leaf. |
There was a problem hiding this comment.
The deviation list covers lax unwrapping but not lax wrapping: in lax mode [*] on a non-array wraps it, so $.tags[*] == "a" over {"tags": "a"}, exists($.tags[*]) over a scalar tags, and exists($.items[*] ? (@.sku == "x")) over {"items": {"sku": "x"}} are all true for jsonb_path_match and false here (the stored selector for a scalar tags is not $.tags[*]). Worth a clause here and in the json_query.rs module docs so "three documented cases" stays accurate.
There was a problem hiding this comment.
Fixed in 6a31ad7. Lax-mode wrapping is now the second of four documented differences, with your three examples. It is in eql-functions.md, in the json_query.rs module docs, and in OpSpec::Match's rustdoc.
The PR description now says four documented cases.
| iv bytea; | ||
| tags jsonb; | ||
| BEGIN | ||
| IF node @? '$.**.contains' THEN |
There was a problem hiding this comment.
Efficiency: both @? probes ($.**.contains, $.**.elements) are recursive-descent walks over the whole predicate, re-run for every row (the function is called per row, the predicate is the same constant each time). Gating the elements index build is worth it, but gating q := d -> q — a single key read on an already-decompressed jsonb — behind a .** walk likely costs more than it saves. Read q unconditionally, or test node @? $.**.elements only.
There was a problem hiding this comment.
Fixed in 6a31ad7.
q := d -> 'q'is read unconditionally.- Only the element-index build is gated, by a single
$.**.elementswalk per row. - I kept that one walk instead of adding a flag to the operand, because a flag would change the operand shape that Go SDK: add
Match(ctx, cipher, predicate, vars)on a JsonSearch field, binding one jsonpath-style predicate as one parameter, on top of #1441 #1443 mirrors. The function's doc says what each row pays.
| bounds := node -> 'cmp'; | ||
| entry := {{prefix}}.json_tree_path_query_first( | ||
| d, COALESCE(bounds -> 'gt', bounds -> 'gte', bounds -> 'lt', bounds -> 'lte') ->> 's'); | ||
| RETURN (NOT bounds ? 'gt' OR {{prefix}}.gt(entry, (bounds -> 'gt')::{{prefix}}.query_json_leaf)) |
There was a problem hiding this comment.
Reuse: this four-way NOT bounds ? gt OR gt(entry, …) AND … block is a copy of the satisfied CTE predicate in elements.sql (json_tree_element_match_opened). An inlinable eql_v4_internal.json_tree_within_bounds(entry public.eql_v4_json_entry, bounds jsonb) RETURNS boolean (LANGUAGE sql) would let both call one definition, so a change to how bounds are evaluated cannot diverge between cmp leaves and element members.
There was a problem hiding this comment.
Fixed in 6a31ad7. The new eql_v4_internal.json_tree_within_bounds(entry public.eql_v4_json_entry, bounds jsonb) is LANGUAGE sql, not STRICT, and inlinable; the pin pass leaves it unpinned.
- Both the
cmpleaf and thesatisfiedCTE injson_tree_element_match_openedcall it. The CTE passes the member object, and itstkey is not read. - Re-run after the change, all passing:
v4_json_tree_element_match_tests: 5v4_json_tree_match_tests, including the parity suite: 6test:encryption:postgres, json_tree: 17
6a3419d to
6a31ad7
Compare
8c14324 to
15e9c56
Compare
A filter compiler that targets `jsonb_path_match(col, $1, $2)` binds one predicate and its values. On `public.eql_v4_json_search` each question is its own operator, so the same filter had to be assembled per leaf. `eql_v4.query_json_predicate` is a boolean tree, one key per node (`and`, `or`, `not`, `contains`, `exists`, `cmp`, `elements`), whose leaves are the operands the family's operators already take. Its CHECK validates each leaf with that operand's own validator: the needle's, a new `json_tree_is_valid_bounds` that the element member's CHECK now calls too, and the elements predicate's. `eql_v4.json_tree_match` evaluates the tree with SQL three-valued logic, so it answers what the hand-written WHERE of those operators answers, NULL included, and `@@` on `(eql_v4_json_search, eql_v4.query_json_predicate)` is the operator form. The jsonpath `@@` blocker stays; an untyped operand now matches both and is refused as ambiguous rather than reaching native jsonb. The document is decompressed once per row and indexed once when the tree has an element leaf: the element match body moves into `json_tree_element_match_opened`, which both functions call. A `contains` leaf applies `json_tree_contains`'s rule to the projection read once. Children are read in order and the read stops at the first that decides the answer, so an element leaf the answer does not depend on cannot raise. The domain joins the catalog as the json_tree family's PredicateQuery role, so the eql-bindings v4 inventory and the Go type `eql.JSONSearchPredicateQuery` come from the generator. Refs #1441 Claude-Session: https://claude.ai/code/session_01MrxG724qYKcAWJdcABBFNY
6a31ad7 to
500aa58
Compare
|
Rebased onto #1445's new head (15e9c56); new head 500aa58. Why: #1444 and #1445 changed under review (#1444's tests moved to real Stack Encrypt documents and gained upgrade note U-004; #1445 got the same test move and the clippy fix that gates This PR's SQLx file |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
cipherstash-agent
left a comment
There was a problem hiding this comment.
Review attestation: automated code review (medium) plus cross-check against existing PR feedback, at head 500aa581.
Result: no critical or blocking issues. No new findings.
What was reviewed, read at head 500aa581:
packages/eql/src/v3/json_tree/predicate.sql, with thetypes.sqlandelements.sqlrefactors. I checked the three-valuedand/or/not, the short-circuit that keeps an unread element leaf from raising, thecmpleaf routed throughjson_tree_within_bounds, thecontainsrule againstjson_tree_contains, and the@@overload.- The compiler half of
packages/stack-encrypt/src/dynamic/json_predicate.rs:comparison,filter,memberandsimplify. The!=and ordering guards, thenot(not x)and dedupe rewrites, and the depth checks preserve the stated answers. - The
eql-bindings,TargetDescriptorand Go-guest wiring diffs.
Cross-check:
- The 7 existing review threads (doctests and docs, guest wiring, the
JsonTooDeeprefusal, keyword case, lax wrapping, the per-row@?walk,json_tree_within_boundsreuse) have all been answered with a fix or a stated deferral, and the head still matches those replies. Nothing is re-raised here. - The changeset covers
@cipherstash/eql. Noskills/*file mentions the json_tree operators, so no skill drift.
Not verified: I did not run the Rust or PostgreSQL suites, and read the ~1.8k-line parser and the test files only selectively. The PR description's results and the cargo-mutants report are the evidence for those.
Non-blocking observation: json_tree_is_valid_predicate_payload recurses without a depth bound. The engine caps nesting at 32, but a hand-built operand nested thousands deep would fail with a stack-depth error (54001) rather than a CHECK violation. It fails loudly, so this is optional hardening.
Generated by Claude Code
…ent operand
`match` takes what `jsonb_path_match` takes, `{"predicate", "vars"?,
"wildcard_bound"?}`, and derives the `eql_v4.query_json_predicate`
operand. A hand-written parser reads the subset of SQL/JSON path
predicates the terms can answer: comparisons of a path (`$.a.b`,
`$.a[*].b`, `$."a b"`) with a variable or a literal, `exists(path)`,
`exists(path ? (@.x == $v && @.y > $w))`, `&&`, `||`, `!` and
parentheses. Everything else is refused at the byte it starts, as
`Error::Predicate`, naming the construct in the grammar's words and
never quoting the text, whose keys and values can be customer data.
Each comparison is compiled onto an existing operator's input and
derived by that operator's code: `==` to `eq_at`'s needle, `!=` to a
`selector` and not the needle, an ordering to a `selector`, not the
`null` needle, and the ordering's leaf, so an absent path and a JSON
null answer false as jsonpath has them; an ordering at a wildcard and
a filter of more than one condition to `element_match`, a filter of one
equality to `eq_at`. No new derivation. `!=` at a wildcard, which asks
whether some element differs, is refused.
New reasons: `json_predicate_syntax`, `json_predicate_unsupported`,
`json_predicate_unknown_variable`, `json_predicate_value_type`,
`json_predicate_not_equal_at_wildcard` and `json_predicate_too_deep`.
The operator is listed only on a type that names the domain
(`TargetDescriptor::with_predicate_query_sql_domain`), so a host that
does not, such as the Go guest today, sees no new operator.
Refs #1441
Claude-Session: https://claude.ai/code/session_01MrxG73fnX1p7oiizh3GRG5
…stgreSQL Documents sealed by Stack Encrypt's JSON index, predicates compiled by `match` through a data plan over eql-bindings' target table, and `eql_v4.json_tree_match` and `@@` over the column, row by row against PostgreSQL's `jsonb_path_match` over the plaintexts, NULL included, and against the WHERE written from the operands each leaf's own operator derives. The rest of the SQL's contract is pinned over the same documents and derived operands: the domain's CHECK node kind by node kind, over a compiled tree and its parts; for every predicate, the WHERE written from the compiled tree's own leaves; three-valued logic under `and`, `or` and `not`, and an ordering leaf alone; a `contains` leaf as `@>`, the root scalar's `alt` included; an `elements` leaf as the element match, a leaf the answer does not depend on left unread; and `@@` with a typed predicate, a blocked jsonpath, an untyped operand and NULLs. EQL tests run over real encrypted data, and the SQLx suite has no way to produce a JSON tree document, so these live here. Refs #1441 Claude-Session: https://claude.ai/code/session_01MrxG724qYKcAWJdcABBFNY
500aa58 to
6109e38
Compare
|
New head: 6109e38. The stack-encrypt commit now refuses a path step past |
Summary
EQL v4 stores an encrypted JSON document in a
public.eql_v4_json_searchcolumn. On a plaintextjsonbcolumn, an application that compiles a filter language into SQL (an AIP-160 filter engine, for example) emits one call,jsonb_path_match(col, $1, $2): a jsonpath predicate and its variables, two parameters. On the encrypted column the database cannot evaluate a path or a value, so each question is its own operator (@>,json_tree_path_exists,(col -> $sel) < $leaf,json_tree_element_match), and the same filter had to be assembled by hand, one parameter per leaf.This PR gives the encrypted column a one-parameter form again. Stack Encrypt's new query operator
matchtakes{"predicate", "vars"}, the same two thingsjsonb_path_matchtakes, parses the jsonpath subset the encrypted terms can answer, and compiles it into one operand: a boolean tree whose leaves are the operands the existing operators already take. The new SQL functioneql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate), orcol @@ $1::eql_v4.query_json_predicate, evaluates the tree. Over the same documents it answers whatjsonb_path_matchanswers over the plaintexts, row by row, NULL included, apart from four documented cases. No stored format changes.The trade-off is in the plan: the tree is evaluated inside a function, so
json_tree_matchfilters a scan and engages no index. The per-operator form stays the fast path.Changes
EQL (
packages/eql)src/v3/json_tree/predicate.sql(new, v4 only):eql_v4.query_json_predicateand its recursive CHECK. Each node is an object with one key:and,or,not,contains(a needle),exists(a selector),cmp(one or two ordering bounds at one selector) orelements(an element-scoped predicate). Each leaf is validated by its own domain's validator: the needle's, the element predicate's, and a new shared bounds validator;eql_v4.json_tree_match(e, p) RETURNS boolean, which follows SQL three-valued logic. Each leaf answers as its operator does on the column;@@overload on(public.eql_v4_json_search, eql_v4.query_json_predicate). Thejsonpathblocker stays.src/v3/json_tree/types.sql:eql_v4_internal.json_tree_is_valid_bounds. The element member's CHECK now calls it too, so one function defines the bounds shape.src/v3/json_tree/elements.sql:json_tree_element_matchmoved intoeql_v4_internal.json_tree_element_match_opened(iv, sv, tags, p). Both functions call it, so a tree with several element leaves opens and indexes the row once.eql_v4_internal.json_tree_within_bounds(entry, bounds), an inlinable SQL function that tests an entry againstgt/gte/lt/lte. An element member and acmpleaf both call it, so one function defines how bounds are evaluated.json_tree_element_match's answers are unchanged, and its tests pass as before.eql_domains::JSON_TREEgains thePredicateQueryrole (JsonSearchPredicateQuery,eql_v4.query_json_predicate). Generated from it: the eql-bindings v4 inventory, and the Go typeeql.JSONSearchPredicateQueryinlanguages/golang/encrypt/eql/eql_gen.go. Hand-written:JsonTreePredicateQueryandJsonTreeBoundsineql-bindings, andTarget::predicate_query_sql_domain().docs/reference/eql-functions.md: the node table with the per-operator equivalent of each node, the planner cost, the cast, and where the answer and jsonpath's differ. Also a leakage bullet injson-tree-leakage.md, and the file and domain inpackages/eql/AGENTS.md.@cipherstash/eqlminor (.changeset/eql-json-tree-match.md).Engine (
packages/stack-encrypt)dynamic/json_predicate.rs(new): a hand-written recursive-descent parser and a compiler. No new dependency.dynamic/json_query.rs:matchin the operator table, and a "Compiled predicates" module section (grammar, mapping, deviations, refusals).deriveis split into reading andderive_input, so every leaf is derived by its own operator's existing code.dynamic/query.rs,target.rs,mod.rs:OpSpec::Match(wire namematch, inputpredicate, fidelityexact),TargetDescriptor::with_predicate_query_sql_domain, a new error variantError::Predicate { field, reason, construct, at }, and six new reasons.json/path.rs:JsonPath::of(segments)(crate-private).The grammar and the compiled shape
Input:
{"predicate": "<jsonpath>", "vars": "<JSON text of an object>"?, "wildcard_bound": N?}.varsis the third argument ofjsonb_path_match, verbatim, so a compiler passes the same two values it passes today.Accepted, as PostgreSQL reads them:
$,.key,."quoted key"(JSON escapes),[*];$v,$"v") or a JSON literal (a string, a number,true,false,null);==,!=(or<>),<,<=,>,>=between a path and a value, either side first;exists(path), andexists(path ? (cond && …)), each condition a comparison on@or@.member;&&,||,!(…)and parentheses.How each comparison compiles:
P == v,P == nullcontains:eq_at's needleP != v,P != nulland[exists(P), not(contains(P:v))]P < v…P >= vand[exists(P), not(contains(P:null)), cmp(P < v)]P < vwith a wildcard inPelements:element_matchon the array throughP's last[*]exists(P)exists: the selectorexists(P ? (@.m == v))containsatP.m(exact)exists(P ? (@.m == v && @.n > w))elementsThe guards make an absent path and a JSON
nullanswer as jsonpath answers them. Without them,!($.age < 3)on a row with noagewould be NULL, where jsonpath says true.Refusals #1443 must mirror. Each is reported as
Error::Predicate, whose payload isreason,construct(a fixed string from the grammar, never text from the predicate) andat(the byte offset in the caller's predicate text):json_predicate_syntax: text the grammar does not read;json_predicate_unsupported: arithmetic,like_regex,starts with,is unknown, methods,strict/lax,.*,.**, slices,last, a comparison of two paths or two values, a filter outsideexists, and||,!,!=orexistsinside a filter;json_predicate_unknown_variable;json_predicate_value_type: an object or array anywhere; under an ordering, or in a filter of more than one condition, anything but a number or a string;json_predicate_not_equal_at_wildcard;json_predicate_too_deep: more than 32 levels of nesting, or a path deeper than a document;json_path_position([2]),json_path_no_wildcard(a filter on a path with no[*]),json_path_wildcard(a[*]in a filter's member).Unchanged, from the leaves' own operators:
TermError::JsonPathNotOrdered, for a range on a path the column does not order, andjson_wildcard_bound_zero,json_inputandjson_syntaxfor the input object itself.Verification
All of this ran locally on macOS, against a private PostgreSQL 17 container on port 7641, after
mise run buildinpackages/eql.Engine (
packages/stack-encrypt)mise x --env test -- cargo nextest run -p stack-encrypt --all-features: 1028 passed. New tests:dynamic/json_predicate.rs. They cover each accepted form and what it compiles to, and every refusal by reason, construct and byte offset (about 90 cases). One test checks that a refusal never quotes text from the predicate, and another that a compiled tree'sDebugshows only its operators.tests/json_operators.rs(compiled_match):json_tree_matchover sealed documents, compared by a 64-case proptest with an independent jsonpath oracle over the plaintexts (three-valued, over absent fields, nulls, wildcards and filters);check_query, with the field and the offset;operators().match, with a marked literal and a marked variable.cargo clippy --workspace --all-targets -D warningsunder default features and--all-features,mise run wasm:wasi-check,mise run wasm:no-http-test,cargo fmt --all --check, doctests (91 across the workspace, including a new example onOpSpec::Matchthat runs predicates throughcheck_query),mise run doc:stack-encrypt, and stack-encrypt nextest--all-features(1028 passed): clean.cargo mutants -p stack-encrypt --in-diff <diff against #1445's head>: 274 mutants, 222 caught, 51 unviable, 1 timeout, 0 missed.lastand mode-word operands, a quoted key after., number literals with a fraction or a signed exponent, the depth limit on a path and on a filter's joined path, andNode'sDebug.+=replaced by-=in the parser's cursor. That loops forever, which the run stops.Default::default()replacements of types that have noDefault.--in-diffagainst the previous head, 6a3419d): 14 caught, 5 unviable, 0 missed.--in-diffrun was stopped part way on the owner's call to leave mutants to CI: 118 of 278 tested, 100 caught, 18 unviable, 0 missed.EQL (
packages/eql)SQLx
lint_tests(12),build_validation_tests(5),selector_literal_guard(10) andv3_public_surface_tests(7): passed.mise run test:encryption:postgreson a fresh database: all suites passed.json_treeis 36 tests, 8 of them new here, every one over real Stack Encrypt documents and predicates compiled bymatchthrough eql-bindings' target table. EQL tests must run over real encrypted data and the SQLx suite has no way to produce a JSON tree document, so the SQL's contract, which a SQLx file with a SQL stand-in for the JSON index pinned before, is pinned here. Six of the new tests cover:json_tree_matchequals the WHERE written from the compiled tree's own leaves (@>,json_tree_path_exists,->with the orderings,json_tree_element_match),@@equalsjson_tree_match, and all equaljsonb_path_matchover the plaintext, row by row, NULL included;and,orandnot, and an ordering leaf alone (NULL where the path is absent ornull);containsequals@>, including the root scalar'salt;elementsleaf equalsjson_tree_element_match; a leaf the answer does not depend on is not read and does not raise, one it does depend on raises22023;@@with a typed operand or a bound parameter works,@@ '…'::jsonpathstill raises, an untyped literal is refused with42725, and a NULL argument gives NULL.The other two:
!,||, and!=/== null/absent cases):json_tree_matchand@@equaljsonb_path_match(plain, predicate, vars)row by row, NULL included;json_tree_path_exists,@>,->with>=, andjson_tree_element_match), and also equalsjsonb_path_match;mise run test:crates(498 tests),test:encryption(543),types:check,typescript:checkandcodegen:parity: passed.docs:validate:required-tags,docs:validate:coverage,docs:validate:documented-sql(against my own container),test:doc-anchorsandtest:docs_v3_grep: passed.The v3 and v4 bundles install side by side, and
eql_v3.lints()andeql_v4.lints()both return 0 rows. The v3 bundle contains none of the new SQL.Go
mise run wasm:guest:test: passed, including the new guest test above.wasm:guest:build,wasm:auth-guest:build,wasm:guest:build:deterministic,wasm:guest:build:eqlandwasm:guest:build:eql:deterministic. Thenmise run go:testpassed for every package.Not run
v3_v4_coexistence_testsdoes not compile here. It reads the fixtures that need CipherStash credentials, and this environment has none (as on feat(eql): test equality and ranges on one array element of a json_tree document #1445). I checked its lint assertion by hand, as above.pnpm run code:check: no JS or TS changed.Related
Closes #1441
Stacked on #1445 (which is on #1444). Refs #1443 (the Go
Matchcall on top).Review notes
The answer matches jsonpath except in four places. All four come from the stored document or the ordering terms, not from the predicate, and all four are documented in
eql-functions.mdand the module docs:[*]) is not applied.[*]and the document holds a value that is not an array, jsonpath wraps it, and the tree finds nothing. For example,$.tags[*] == "a"over{"tags": "a"}is true to jsonpath and false here, because$.tags[*]and$.tagshave different selectors.==, true under!=, and decided by the terms under an ordering.element_match), compare the folded ordering terms.The parity tests use documents where none of these apply, and pass row for row, NULL included.
Leakage cost of exactness. An ordering carries a
nullneedle beside its bound. That makesnullanswer false and!(… < v)answer as jsonpath does, but it also tells the database which rows holdnullat that path. This is said injson-tree-leakage.md. Dropping it would make a negated ordering over anullanswer NULL instead of true. Say if you would rather have that.Refusals for an undeclared path carry no offset. A range on a path the column does not order still fails with the existing
TermError::JsonPathNotOrdered, as the brief asked. That error carries neither the path text (the crate keeps key names out of errors) nor an offset. Grammar refusals carry an offset. If you want the index-level refusals to carry the offset too, that is a small follow-up: wrap them inError::Predicatewith the path's offset.TargetDescriptor::operators()now filters JSON operators by named domain. Without that, a type's own list showedmatcheven when the type named no predicate domain. The Go guest reads that list, and failed withunknown operator "match"in four Go tests.element_matchnever hit this, because it also needs an ordered path. Now the type list and the field list agree, and a test pins it. This is the one change outside the new operator.The untyped
@@operand. Before this PR,col @@ '…'with an untyped literal reached the jsonpath blocker and raised. Now it raises42725(operator is not unique), still loudly. The docs say to cast.json_tree_matchopens the row once. It decompresses the document once and readsqonce. It builds the tag index only when the tree has an element leaf, which it checks with one$.**.elementswalk of the predicate per row (the predicate is small). A flag in the operand would remove that walk, but would change the operand shape; I left the shape as it is. Acontainsleaf appliesjson_tree_contains's two-line rule to the projection it has already read. Callingjson_tree_containsitself would re-read the TOASTed column for every leaf. Wrapping the rule in a shared SQL helper would stop the planner inlining@>into the GIN-indexed form, because the argument is costly and used twice.What Go SDK: add
Match(ctx, cipher, predicate, vars)on a JsonSearch field, binding one jsonpath-style predicate as one parameter, on top of #1441 #1443 needs (the Go guest does not name the domain, so the Go SDK sees no new operator today):languages/golang/encrypt/guest/src/targets.rs:with_predicate_query_sql_domain(target.predicate_query_sql_domain()), andwith_elements_query_sql_domain(...)for Go SDK: no way to test an equality and a range on the same array element of a JsonSearch field — add an element-scopedMatchon top of #1440 #1442. A new guest test,a_document_names_the_leaf_domain_and_not_the_predicate_domains, pins that the guest names neither domain today, so it fails until Go SDK: no way to test an equality and a range on the same array element of a JsonSearch field — add an element-scopedMatchon top of #1440 #1442/Go SDK: addMatch(ctx, cipher, predicate, vars)on a JsonSearch field, binding one jsonpath-style predicate as one parameter, on top of #1441 #1443 update it alongside the Go side;"match"inparseOperatorName(encrypt/checker.go) and in stashgen'sjsonOperators;"predicate"inchecker.go's list of shapes;Error::Predicatein the guest'sstatus_for_dynamic. Its catch-all maps it toSTATUS_INTERNALuntil then;reason,constructandat.JsonSubtreeKey::below(from feat(eql): test equality and ranges on one array element of a json_tree document #1445) is now gated ondynamicthere, and the one helper this PR adds (JsonPath::of) is gated the same way, so clippy passes with default features and with--all-features.https://claude.ai/code/session_01MrxG724qYKcAWJdcABBFNY