Skip to content

feat(eql): one-parameter jsonpath predicates on json_tree documents (json_tree_match, @@, match) - #1446

Open
coderdan wants to merge 3 commits into
feat/eql-v4-json-tree-element-matchfrom
feat/eql-v4-json-tree-match
Open

coderdan wants to merge 3 commits into
feat/eql-v4-json-tree-element-matchfrom
feat/eql-v4-json-tree-match

Conversation

@coderdan

@coderdan coderdan commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

EQL v4 stores an encrypted JSON document in a public.eql_v4_json_search column. On a plaintext jsonb column, an application that compiles a filter language into SQL (an AIP-160 filter engine, for example) emits one call, jsonb_path_match(col, $1, $2): a jsonpath predicate and its variables, two parameters. On the encrypted column the database cannot evaluate a path or a value, so each question is its own operator (@>, json_tree_path_exists, (col -> $sel) < $leaf, json_tree_element_match), and the same filter had to be assembled by hand, one parameter per leaf.

This PR gives the encrypted column a one-parameter form again. Stack Encrypt's new query operator match takes {"predicate", "vars"}, the same two things jsonb_path_match takes, parses the jsonpath subset the encrypted terms can answer, and compiles it into one operand: a boolean tree whose leaves are the operands the existing operators already take. The new SQL function eql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate), or col @@ $1::eql_v4.query_json_predicate, evaluates the tree. Over the same documents it answers what jsonb_path_match answers over the plaintexts, row by row, NULL included, apart from four documented cases. No stored format changes.

The trade-off is in the plan: the tree is evaluated inside a function, so json_tree_match filters a scan and engages no index. The per-operator form stays the fast path.

Changes

EQL (packages/eql)

  • src/v3/json_tree/predicate.sql (new, v4 only):
    • the domain eql_v4.query_json_predicate and its recursive CHECK. Each node is an object with one key: and, or, not, contains (a needle), exists (a selector), cmp (one or two ordering bounds at one selector) or elements (an element-scoped predicate). Each leaf is validated by its own domain's validator: the needle's, the element predicate's, and a new shared bounds validator;
    • eql_v4.json_tree_match(e, p) RETURNS boolean, which follows SQL three-valued logic. Each leaf answers as its operator does on the column;
    • the @@ overload on (public.eql_v4_json_search, eql_v4.query_json_predicate). The jsonpath blocker stays.
  • src/v3/json_tree/types.sql: eql_v4_internal.json_tree_is_valid_bounds. The element member's CHECK now calls it too, so one function defines the bounds shape.
  • src/v3/json_tree/elements.sql:
    • The body of json_tree_element_match moved into eql_v4_internal.json_tree_element_match_opened(iv, sv, tags, p). Both functions call it, so a tree with several element leaves opens and indexes the row once.
    • New: eql_v4_internal.json_tree_within_bounds(entry, bounds), an inlinable SQL function that tests an entry against gt/gte/lt/lte. An element member and a cmp leaf both call it, so one function defines how bounds are evaluated.
    • json_tree_element_match's answers are unchanged, and its tests pass as before.
  • Catalog: eql_domains::JSON_TREE gains the PredicateQuery role (JsonSearchPredicateQuery, eql_v4.query_json_predicate). Generated from it: the eql-bindings v4 inventory, and the Go type eql.JSONSearchPredicateQuery in languages/golang/encrypt/eql/eql_gen.go. Hand-written: JsonTreePredicateQuery and JsonTreeBounds in eql-bindings, and Target::predicate_query_sql_domain().
  • Docs: a new "A compiled predicate in EQL v4" section in docs/reference/eql-functions.md: the node table with the per-operator equivalent of each node, the planner cost, the cast, and where the answer and jsonpath's differ. Also a leakage bullet in json-tree-leakage.md, and the file and domain in packages/eql/AGENTS.md.
  • Changeset: @cipherstash/eql minor (.changeset/eql-json-tree-match.md).

Engine (packages/stack-encrypt)

  • dynamic/json_predicate.rs (new): a hand-written recursive-descent parser and a compiler. No new dependency.
  • dynamic/json_query.rs: match in the operator table, and a "Compiled predicates" module section (grammar, mapping, deviations, refusals). derive is split into reading and derive_input, so every leaf is derived by its own operator's existing code.
  • dynamic/query.rs, target.rs, mod.rs: OpSpec::Match (wire name match, input predicate, fidelity exact), TargetDescriptor::with_predicate_query_sql_domain, a new error variant Error::Predicate { field, reason, construct, at }, and six new reasons.
  • json/path.rs: JsonPath::of(segments) (crate-private).

The grammar and the compiled shape

Input: {"predicate": "<jsonpath>", "vars": "<JSON text of an object>"?, "wildcard_bound": N?}. vars is the third argument of jsonb_path_match, verbatim, so a compiler passes the same two values it passes today.

Accepted, as PostgreSQL reads them:

  • paths $, .key, ."quoted key" (JSON escapes), [*];
  • values: a variable ($v, $"v") or a JSON literal (a string, a number, true, false, null);
  • comparisons ==, != (or <>), <, <=, >, >= between a path and a value, either side first;
  • exists(path), and exists(path ? (cond && …)), each condition a comparison on @ or @.member;
  • &&, ||, !(…) and parentheses.

How each comparison compiles:

jsonpath compiled to
P == v, P == null contains: eq_at's needle
P != v, P != null and[exists(P), not(contains(P:v))]
P < v … P >= v and[exists(P), not(contains(P:null)), cmp(P < v)]
P < v with a wildcard in P elements: element_match on the array through P's last [*]
exists(P) exists: the selector
exists(P ? (@.m == v)) contains at P.m (exact)
exists(P ? (@.m == v && @.n > w)) elements

The guards make an absent path and a JSON null answer as jsonpath answers them. Without them, !($.age < 3) on a row with no age would be NULL, where jsonpath says true.

Refusals #1443 must mirror. Each is reported as Error::Predicate, whose payload is reason, construct (a fixed string from the grammar, never text from the predicate) and at (the byte offset in the caller's predicate text):

  • json_predicate_syntax: text the grammar does not read;
  • json_predicate_unsupported: arithmetic, like_regex, starts with, is unknown, methods, strict/lax, .*, .**, slices, last, a comparison of two paths or two values, a filter outside exists, and ||, !, != or exists inside a filter;
  • json_predicate_unknown_variable;
  • json_predicate_value_type: an object or array anywhere; under an ordering, or in a filter of more than one condition, anything but a number or a string;
  • json_predicate_not_equal_at_wildcard;
  • json_predicate_too_deep: more than 32 levels of nesting, or a path deeper than a document;
  • existing reasons, now with an offset: json_path_position ([2]), json_path_no_wildcard (a filter on a path with no [*]), json_path_wildcard (a [*] in a filter's member).

Unchanged, from the leaves' own operators: TermError::JsonPathNotOrdered, for a range on a path the column does not order, and json_wildcard_bound_zero, json_input and json_syntax for the input object itself.

Verification

All of this ran locally on macOS, against a private PostgreSQL 17 container on port 7641, after mise run build in packages/eql.

Engine (packages/stack-encrypt)

  • mise x --env test -- cargo nextest run -p stack-encrypt --all-features: 1028 passed. New tests:
    • 12 parser and compiler unit tests in dynamic/json_predicate.rs. They cover each accepted form and what it compiles to, and every refusal by reason, construct and byte offset (about 90 cases). One test checks that a refusal never quotes text from the predicate, and another that a compiled tree's Debug shows only its operators.
    • 9 tests in tests/json_operators.rs (compiled_match):
      • each leaf equals the operand its own operator derives;
      • the boolean structure survives compilation;
      • a Rust model of json_tree_match over sealed documents, compared by a 64-case proptest with an independent jsonpath oracle over the plaintexts (three-valued, over absent fields, nulls, wildcards and filters);
      • fixed three-valued cases;
      • refusals through check_query, with the field and the offset;
      • the catalog listing, both a field's and a type's own operators().
    • The heap-hygiene watch covers match, with a marked literal and a marked variable.
  • Root workspace, after the rebase onto feat(eql): test equality and ranges on one array element of a json_tree document #1445's new head: cargo clippy --workspace --all-targets -D warnings under default features and --all-features, mise run wasm:wasi-check, mise run wasm:no-http-test, cargo fmt --all --check, doctests (91 across the workspace, including a new example on OpSpec::Match that runs predicates through check_query), mise run doc:stack-encrypt, and stack-encrypt nextest --all-features (1028 passed): clean.
  • cargo-mutants, cargo mutants -p stack-encrypt --in-diff <diff against #1445's head>: 274 mutants, 222 caught, 51 unviable, 1 timeout, 0 missed.
    • The first full run left 15 missed, all in the parser: nesting depth that only grows, a second filter after a filter, the last and mode-word operands, a quoted key after . , number literals with a fraction or a signed exponent, the depth limit on a path and on a filter's joined path, and Node's Debug.
    • For each, I added a test that pins it. A filtered re-run of those 15 against the new tests: 15 caught.
    • The timeout is += replaced by -= in the parser's cursor. That loops forever, which the run stops.
    • The unviable mutants are Default::default() replacements of types that have no Default.
    • After the review fixes: the 19 mutants on the lines the fixes change (--in-diff against the previous head, 6a3419d): 14 caught, 5 unviable, 0 missed.
    • After the rebase onto feat(eql): test equality and ranges on one array element of a json_tree document #1445's new head, a fresh --in-diff run was stopped part way on the owner's call to leave mutants to CI: 118 of 278 tested, 100 caught, 18 unviable, 0 missed.

EQL (packages/eql)

  • SQLx lint_tests (12), build_validation_tests (5), selector_literal_guard (10) and v3_public_surface_tests (7): passed.

  • mise run test:encryption:postgres on a fresh database: all suites passed. json_tree is 36 tests, 8 of them new here, every one over real Stack Encrypt documents and predicates compiled by match through eql-bindings' target table. EQL tests must run over real encrypted data and the SQLx suite has no way to produce a JSON tree document, so the SQL's contract, which a SQLx file with a SQL stand-in for the JSON index pinned before, is pinned here. Six of the new tests cover:

    • the domain CHECK, node kind by node kind (9 valid, 21 invalid, over a compiled tree and its parts), plus the element member CHECK through the shared bounds validator;
    • for 18 predicates over 7 documents: json_tree_match equals the WHERE written from the compiled tree's own leaves (@>, json_tree_path_exists, -> with the orderings, json_tree_element_match), @@ equals json_tree_match, and all equal jsonb_path_match over the plaintext, row by row, NULL included;
    • NULL leaves under and, or and not, and an ordering leaf alone (NULL where the path is absent or null);
    • contains equals @>, including the root scalar's alt;
    • an elements leaf equals json_tree_element_match; a leaf the answer does not depend on is not read and does not raise, one it does depend on raises 22023;
    • @@ with a typed operand or a bound parameter works, @@ '…'::jsonpath still raises, an untyped literal is refused with 42725, and a NULL argument gives NULL.

    The other two:

    • for 24 predicates (with variables, wildcards, filters, !, ||, and !=/== null/absent cases): json_tree_match and @@ equal jsonb_path_match(plain, predicate, vars) row by row, NULL included;
    • a mixed predicate equals the WHERE written from each leaf operator's own operand (json_tree_path_exists, @>, -> with >=, and json_tree_element_match), and also equals jsonb_path_match;
    • refusals carry their offsets through the data plan.
  • mise run test:crates (498 tests), test:encryption (543), types:check, typescript:check and codegen:parity: passed.

  • docs:validate:required-tags, docs:validate:coverage, docs:validate:documented-sql (against my own container), test:doc-anchors and test:docs_v3_grep: passed.

  • The v3 and v4 bundles install side by side, and eql_v3.lints() and eql_v4.lints() both return 0 rows. The v3 bundle contains none of the new SQL.

Go

  • mise run wasm:guest:test: passed, including the new guest test above.
  • All five guests rebuilt: wasm:guest:build, wasm:auth-guest:build, wasm:guest:build:deterministic, wasm:guest:build:eql and wasm:guest:build:eql:deterministic. Then mise run go:test passed for every package.

Not run

Related

Closes #1441

Stacked on #1445 (which is on #1444). Refs #1443 (the Go Match call on top).

Review notes

  • The answer matches jsonpath except in four places. All four come from the stored document or the ordering terms, not from the predicate, and all four are documented in eql-functions.md and the module docs:

    1. A path is read as written, so lax mode's array unwrapping (an array where the predicate has no [*]) is not applied.
    2. Lax mode's wrapping is not applied either. Where the predicate names [*] and the document holds a value that is not an array, jsonpath wraps it, and the tree finds nothing. For example, $.tags[*] == "a" over {"tags": "a"} is true to jsonpath and false here, because $.tags[*] and $.tags have different selectors.
    3. A comparison of a string with a number is unknown (NULL) to jsonpath, but the terms do not record a value's kind. So it is false under ==, true under !=, and decided by the terms under an ordering.
    4. Orderings, and filters with more than one condition (which compile to element_match), compare the folded ordering terms.

    The parity tests use documents where none of these apply, and pass row for row, NULL included.

  • Leakage cost of exactness. An ordering carries a null needle beside its bound. That makes null answer false and !(… < v) answer as jsonpath does, but it also tells the database which rows hold null at that path. This is said in json-tree-leakage.md. Dropping it would make a negated ordering over a null answer NULL instead of true. Say if you would rather have that.

  • Refusals for an undeclared path carry no offset. A range on a path the column does not order still fails with the existing TermError::JsonPathNotOrdered, as the brief asked. That error carries neither the path text (the crate keeps key names out of errors) nor an offset. Grammar refusals carry an offset. If you want the index-level refusals to carry the offset too, that is a small follow-up: wrap them in Error::Predicate with the path's offset.

  • TargetDescriptor::operators() now filters JSON operators by named domain. Without that, a type's own list showed match even when the type named no predicate domain. The Go guest reads that list, and failed with unknown operator "match" in four Go tests. element_match never hit this, because it also needs an ordered path. Now the type list and the field list agree, and a test pins it. This is the one change outside the new operator.

  • The untyped @@ operand. Before this PR, col @@ '…' with an untyped literal reached the jsonpath blocker and raised. Now it raises 42725 (operator is not unique), still loudly. The docs say to cast.

  • json_tree_match opens the row once. It decompresses the document once and reads q once. It builds the tag index only when the tree has an element leaf, which it checks with one $.**.elements walk of the predicate per row (the predicate is small). A flag in the operand would remove that walk, but would change the operand shape; I left the shape as it is. A contains leaf applies json_tree_contains's two-line rule to the projection it has already read. Calling json_tree_contains itself would re-read the TOASTed column for every leaf. Wrapping the rule in a shared SQL helper would stop the planner inlining @> into the GIN-indexed form, because the argument is costly and used twice.

  • What Go SDK: add Match(ctx, cipher, predicate, vars) on a JsonSearch field, binding one jsonpath-style predicate as one parameter, on top of #1441 #1443 needs (the Go guest does not name the domain, so the Go SDK sees no new operator today):

  • JsonSubtreeKey::below (from feat(eql): test equality and ranges on one array element of a json_tree document #1445) is now gated on dynamic there, and the one helper this PR adds (JsonPath::of) is gated the same way, so clippy passes with default features and with --all-features.

https://claude.ai/code/session_01MrxG724qYKcAWJdcABBFNY

@changeset-bot

changeset-bot Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 6109e38

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages
Name Type
@cipherstash/eql Minor
stash Patch
@cipherstash/stack-prisma Patch
@cipherstash/basic-example Patch
@cipherstash/e2e Patch
@cipherstash/prisma-example Patch
@cipherstash/stack-drizzle Patch
@cipherstash/stack-supabase Patch
@cipherstash/stack Patch
@cipherstash/wizard Patch
@cipherstash/bench Patch
@cipherstash/test-kit Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI; Enforced overrides: Organization
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 00e40e9b-9a8b-445c-8b4e-2424e0a261f6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught missed unviable timeout
226 0 51 0

Every mutant in the changed lines was caught by a test.

@coderdan
coderdan added this pull request to stack #1449 October 30, 2026 00:51
Comment on lines +5 to +12
//! The grammar and the compilation are documented with the operator, in
//! the [`json_query`](super::json_query) module docs. This module holds the
//! parser ([`parse`]), which reads the text into an [`Ast`] and refuses
//! what the grammar does not read at the byte where it stops, and the
//! compiler ([`compile`]), which binds the values and maps each comparison
//! onto the inputs `eq_at`, `selector`, the orderings and `element_match`
//! take, so that every leaf is derived by that operator's own code
//! ([`derive`]).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include examples (including doctests) of the syntax as well as a link to the relevant postgres docs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6a31ad7.

  • The json_predicate.rs module docs now open with worked examples of the accepted syntax and a link to PostgreSQL's SQL/JSON path language.
  • OpSpec::Match's rustdoc carries the same link and a doctest. The doctest runs three accepted predicates through check_query on a document field: a variable, a quoted key with != null, and an element filter. It also shows two refusals, like_regex by reason and arithmetic by byte offset.

@coderdan
coderdan force-pushed the feat/eql-v4-json-tree-match branch from 94db05b to 6a3419d Compare October 30, 2026 00:58
@coderdan

Copy link
Copy Markdown
Contributor Author

Force-pushed after the mutation run: the stack-encrypt commit now also carries the parser tests that kill the 15 mutants the first run missed (nesting depth, a second filter, last and mode words as operands, a quoted key after . , fractional and signed-exponent numbers, the depth limits, and Node's Debug). No behaviour changed. The description has the final counts: 274 mutants, 222 caught, 51 unviable, 1 timeout, 0 missed.

descriptor = descriptor.with_elements_query_sql_domain(domain);
}
if let Some(domain) = target.predicate_query_sql_domain() {
descriptor = descriptor.with_predicate_query_sql_domain(domain);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go guest never wires predicate_query_sql_domain (or elements_query_sql_domain). This test resolver names all three domains, but the production resolver the Go SDK uses — languages/golang/encrypt/guest/src/targets.rs describe() (line ~118) — only calls with_leaf_query_sql_domain. So through the Go path json_query::domain(target, Binds::Predicate) is None, query_term returns TargetError::NoQuery, and match (and element_match) is refused and absent from operators(), even though eql.JSONSearchPredicateQuery was generated for Go in this PR. The module doc here says this is "the path the Go guest takes", but it is not the same resolver. Add the two with_* calls in the guest describe().

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is fixed. Wiring the guest is deliberately left to #1442/#1443, per the milestone's order: Rust and EQL first, then Go on top.

Done in 6a31ad7.

at
},
);
if full.len() > MAX_DEPTH {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A single-equality filter whose base + member path exceeds MAX_DEPTH returns term(TermError::JsonTooDeep) (an Error::Cipher), while the PR/module docs say a path deeper than a document is Error::Predicate { reason: JsonPredicateTooDeep, at }. member_of has the same shape for the multi-condition filter. A client mirroring the refusals (#1443) gets a different error type with no offset for this one case; use the Refusal { JsonPredicateTooDeep, C_NESTING, at: path.at } form as steps() does.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6a31ad7.

  • The compiler checks the depth of base + member for every condition of a filter, before anything is derived.
  • A filter's joined path deeper than a document is now Error::Predicate { reason: JsonPredicateTooDeep, construct: "nesting", at: <the filter's path> }. This holds for the single-equality filter and the multi-condition (element_match) filter alike, so neither surfaces TermError::JsonTooDeep any more.
  • Tests cover both forms, and the path exactly as deep as a document still compiles.

}))
}
Some("last") => Err(Refusal::unsupported(C_LAST, at)),
Some("strict" | "lax") => Err(Refusal::unsupported(C_MODE, at)),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refusal classification diverges from how PostgreSQL reads keywords. PG matches strict, lax, last, like_regex, starts, is case-insensitively (only true/false/null must be lowercase), but only exists goes through keyword(); STRICT $.a == 1 or $.a LIKE_REGEX "x" therefore surfaces as json_predicate_syntax / "text the grammar does not read" instead of json_predicate_unsupported with the named construct. Same for exists(...): after_operand() is not called there, so exists($.a like_regex "x") / exists($.a + 1) fall to expect(")") and are reported as syntax errors. Not a wrong answer, but a client mirroring the reasons/constructs will classify these differently.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6a31ad7.

  • strict, lax, last, like_regex, starts and is are matched case-insensitively, as PostgreSQL's scanner reads them. true, false and null stay lowercase-only, as PostgreSQL requires.
  • The exists(...) operand now gets the same check for the constructs that follow an operand, both after a bare path and after a filter.
  • New cases, each asserted as json_predicate_unsupported with its construct and offset:
    • STRICT $.a == 1
    • $.a LIKE_REGEX "x"
    • $.a Starts with "x"
    • $.a IS unknown
    • $.a[LAST]
    • $.a == Lax
    • exists($.a like_regex "x")
    • exists($.a + 1)
    • exists($.a[*] ? (@ == 1) starts with "x")

Each leaf is checked by the CHECK of its own domain, and `and` and `or` take at least one child.

- **The answer is the hand-written WHERE's, NULL included.** `and`, `or` and `not` follow SQL's three-valued logic, and a `cmp` leaf is NULL where the path is absent or its node has no ordering term, as `(col -> s) < $1` is. Stack Encrypt compiles a jsonpath comparison so that the tree answers as `jsonb_path_match` does over the plaintext: `$.age < $n` is a node at `$.age`, not a JSON `null`, whose leaf compares; `$.age != $n` is a node there that is not the value. So an absent path is false under every comparison and `null` false under an ordering, as jsonpath has them, and an ordering of a boolean is unknown (NULL), as jsonpath's comparison of two kinds is.
- **Where the tree and jsonpath part.** A path is read as written, so an array where the predicate names no `[*]` is not unwrapped as jsonpath's lax mode unwraps it. A comparison of a string with a number, which jsonpath answers unknown, is false under `==` and true under `!=`, since the terms do not record a value's kind. An ordering compares the ordering terms, so text is case- and accent-folded, as every ordering at a path is; so does each condition of a filter with more than one, which compiles to an element leaf.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The deviation list covers lax unwrapping but not lax wrapping: in lax mode [*] on a non-array wraps it, so $.tags[*] == "a" over {"tags": "a"}, exists($.tags[*]) over a scalar tags, and exists($.items[*] ? (@.sku == "x")) over {"items": {"sku": "x"}} are all true for jsonb_path_match and false here (the stored selector for a scalar tags is not $.tags[*]). Worth a clause here and in the json_query.rs module docs so "three documented cases" stays accurate.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6a31ad7. Lax-mode wrapping is now the second of four documented differences, with your three examples. It is in eql-functions.md, in the json_query.rs module docs, and in OpSpec::Match's rustdoc.

The PR description now says four documented cases.

iv bytea;
tags jsonb;
BEGIN
IF node @? '$.**.contains' THEN

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Efficiency: both @? probes ($.**.contains, $.**.elements) are recursive-descent walks over the whole predicate, re-run for every row (the function is called per row, the predicate is the same constant each time). Gating the elements index build is worth it, but gating q := d -> q — a single key read on an already-decompressed jsonb — behind a .** walk likely costs more than it saves. Read q unconditionally, or test node @? $.**.elements only.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6a31ad7.

bounds := node -> 'cmp';
entry := {{prefix}}.json_tree_path_query_first(
d, COALESCE(bounds -> 'gt', bounds -> 'gte', bounds -> 'lt', bounds -> 'lte') ->> 's');
RETURN (NOT bounds ? 'gt' OR {{prefix}}.gt(entry, (bounds -> 'gt')::{{prefix}}.query_json_leaf))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse: this four-way NOT bounds ? gt OR gt(entry, …) AND … block is a copy of the satisfied CTE predicate in elements.sql (json_tree_element_match_opened). An inlinable eql_v4_internal.json_tree_within_bounds(entry public.eql_v4_json_entry, bounds jsonb) RETURNS boolean (LANGUAGE sql) would let both call one definition, so a change to how bounds are evaluated cannot diverge between cmp leaves and element members.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6a31ad7. The new eql_v4_internal.json_tree_within_bounds(entry public.eql_v4_json_entry, bounds jsonb) is LANGUAGE sql, not STRICT, and inlinable; the pin pass leaves it unpinned.

  • Both the cmp leaf and the satisfied CTE in json_tree_element_match_opened call it. The CTE passes the member object, and its t key is not read.
  • Re-run after the change, all passing:
    • v4_json_tree_element_match_tests: 5
    • v4_json_tree_match_tests, including the parity suite: 6
    • test:encryption:postgres, json_tree: 17

A filter compiler that targets `jsonb_path_match(col, $1, $2)` binds one
predicate and its values. On `public.eql_v4_json_search` each question
is its own operator, so the same filter had to be assembled per leaf.

`eql_v4.query_json_predicate` is a boolean tree, one key per node
(`and`, `or`, `not`, `contains`, `exists`, `cmp`, `elements`), whose
leaves are the operands the family's operators already take. Its CHECK
validates each leaf with that operand's own validator: the needle's, a
new `json_tree_is_valid_bounds` that the element member's CHECK now
calls too, and the elements predicate's. `eql_v4.json_tree_match`
evaluates the tree with SQL three-valued logic, so it answers what the
hand-written WHERE of those operators answers, NULL included, and `@@`
on `(eql_v4_json_search, eql_v4.query_json_predicate)` is the operator
form. The jsonpath `@@` blocker stays; an untyped operand now matches
both and is refused as ambiguous rather than reaching native jsonb.

The document is decompressed once per row and indexed once when the
tree has an element leaf: the element match body moves into
`json_tree_element_match_opened`, which both functions call. A
`contains` leaf applies `json_tree_contains`'s rule to the projection
read once. Children are read in order and the read stops at the first
that decides the answer, so an element leaf the answer does not depend
on cannot raise.

The domain joins the catalog as the json_tree family's PredicateQuery
role, so the eql-bindings v4 inventory and the Go type
`eql.JSONSearchPredicateQuery` come from the generator.

Refs #1441

Claude-Session: https://claude.ai/code/session_01MrxG724qYKcAWJdcABBFNY
@coderdan
coderdan force-pushed the feat/eql-v4-json-tree-match branch from 6a31ad7 to 500aa58 Compare October 30, 2026 10:06
@coderdan

Copy link
Copy Markdown
Contributor Author

Rebased onto #1445's new head (15e9c56); new head 500aa58.

Why: #1444 and #1445 changed under review (#1444's tests moved to real Stack Encrypt documents and gained upgrade note U-004; #1445 got the same test move and the clippy fix that gates JsonSubtreeKey::below on dynamic).

This PR's SQLx file v4_json_tree_match_tests.rs sealed its documents with a SQL stand-in for the JSON index, which the rule on real encrypted data does not allow, so its six tests moved into packages/eql/tests/encryption/tests/json_tree/predicate.rs, over documents Stack Encrypt seals and trees match compiles: the domain CHECK node by node, the WHERE written from each compiled tree's own leaves for every predicate, three-valued logic, contains with alt, elements leaves, and @@. The SQL commit no longer adds the SQLx file; the test commit carries them. JsonPath::of was already gated on dynamic, and clippy passes under default features and --all-features.

@coderdan coderdan closed this Oct 30, 2026
@coderdan coderdan reopened this Oct 30, 2026
@coderdan
coderdan removed this pull request from stack #1449 October 30, 2026 10:32
@coderdan
coderdan marked this pull request as ready for review October 30, 2026 10:43
@coderdan
coderdan requested a review from a team as a code owner October 30, 2026 10:43
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T10:45:55.702803Z 500aa58 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-30T10:48:31.744744Z 500aa58 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cipherstash-agent cipherstash-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review attestation: automated code review (medium) plus cross-check against existing PR feedback, at head 500aa581.

Result: no critical or blocking issues. No new findings.

What was reviewed, read at head 500aa581:

  • packages/eql/src/v3/json_tree/predicate.sql, with the types.sql and elements.sql refactors. I checked the three-valued and / or / not, the short-circuit that keeps an unread element leaf from raising, the cmp leaf routed through json_tree_within_bounds, the contains rule against json_tree_contains, and the @@ overload.
  • The compiler half of packages/stack-encrypt/src/dynamic/json_predicate.rs: comparison, filter, member and simplify. The != and ordering guards, the not(not x) and dedupe rewrites, and the depth checks preserve the stated answers.
  • The eql-bindings, TargetDescriptor and Go-guest wiring diffs.

Cross-check:

  • The 7 existing review threads (doctests and docs, guest wiring, the JsonTooDeep refusal, keyword case, lax wrapping, the per-row @? walk, json_tree_within_bounds reuse) have all been answered with a fix or a stated deferral, and the head still matches those replies. Nothing is re-raised here.
  • The changeset covers @cipherstash/eql. No skills/* file mentions the json_tree operators, so no skill drift.

Not verified: I did not run the Rust or PostgreSQL suites, and read the ~1.8k-line parser and the test files only selectively. The PR description's results and the cargo-mutants report are the evidence for those.

Non-blocking observation: json_tree_is_valid_predicate_payload recurses without a depth bound. The engine caps nesting at 32, but a hand-built operand nested thousands deep would fail with a stack-depth error (54001) rather than a CHECK violation. It fails loudly, so this is optional hardening.


Generated by Claude Code

…ent operand

`match` takes what `jsonb_path_match` takes, `{"predicate", "vars"?,
"wildcard_bound"?}`, and derives the `eql_v4.query_json_predicate`
operand. A hand-written parser reads the subset of SQL/JSON path
predicates the terms can answer: comparisons of a path (`$.a.b`,
`$.a[*].b`, `$."a b"`) with a variable or a literal, `exists(path)`,
`exists(path ? (@.x == $v && @.y > $w))`, `&&`, `||`, `!` and
parentheses. Everything else is refused at the byte it starts, as
`Error::Predicate`, naming the construct in the grammar's words and
never quoting the text, whose keys and values can be customer data.

Each comparison is compiled onto an existing operator's input and
derived by that operator's code: `==` to `eq_at`'s needle, `!=` to a
`selector` and not the needle, an ordering to a `selector`, not the
`null` needle, and the ordering's leaf, so an absent path and a JSON
null answer false as jsonpath has them; an ordering at a wildcard and
a filter of more than one condition to `element_match`, a filter of one
equality to `eq_at`. No new derivation. `!=` at a wildcard, which asks
whether some element differs, is refused.

New reasons: `json_predicate_syntax`, `json_predicate_unsupported`,
`json_predicate_unknown_variable`, `json_predicate_value_type`,
`json_predicate_not_equal_at_wildcard` and `json_predicate_too_deep`.
The operator is listed only on a type that names the domain
(`TargetDescriptor::with_predicate_query_sql_domain`), so a host that
does not, such as the Go guest today, sees no new operator.

Refs #1441

Claude-Session: https://claude.ai/code/session_01MrxG73fnX1p7oiizh3GRG5
…stgreSQL

Documents sealed by Stack Encrypt's JSON index, predicates compiled by
`match` through a data plan over eql-bindings' target table, and
`eql_v4.json_tree_match` and `@@` over the column, row by row against
PostgreSQL's `jsonb_path_match` over the plaintexts, NULL included, and
against the WHERE written from the operands each leaf's own operator
derives.

The rest of the SQL's contract is pinned over the same documents and
derived operands: the domain's CHECK node kind by node kind, over a
compiled tree and its parts; for every predicate, the WHERE written
from the compiled tree's own leaves; three-valued logic under `and`,
`or` and `not`, and an ordering leaf alone; a `contains` leaf as `@>`,
the root scalar's `alt` included; an `elements` leaf as the element
match, a leaf the answer does not depend on left unread; and `@@` with
a typed predicate, a blocked jsonpath, an untyped operand and NULLs.
EQL tests run over real encrypted data, and the SQLx suite has no way
to produce a JSON tree document, so these live here.

Refs #1441

Claude-Session: https://claude.ai/code/session_01MrxG724qYKcAWJdcABBFNY
@coderdan
coderdan force-pushed the feat/eql-v4-json-tree-match branch from 500aa58 to 6109e38 Compare October 30, 2026 13:17
@coderdan

Copy link
Copy Markdown
Contributor Author

New head: 6109e38. The stack-encrypt commit now refuses a path step past MAX_DEPTH inside the step loop, before the step is read. That turns the steps mutant that used to time out (+= replaced with -= on the name length, which made the parser re-read one key forever) into a refusal that the tests catch. It also adds a test for that refusal. A targeted local cargo-mutants run over the earlier survivors (ascend, the exists ? arm, the operand guards, steps, number_len, Debug for Node, Compiler::filter) caught all 73 mutants: 71 caught, 2 unviable, 0 missed, 0 timeouts.

https://claude.ai/code/session_01MrxG73fnX1p7oiizh3GRG5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

json_tree: a compiled predicate operand and eql_v4.json_tree_match, so a filter engine binds one predicate as one parameter

2 participants