Repository navigation
feat(stack-encrypt,eql): store a JSON document through a data plan as JsonSearch - #1372
Merged
Merged
Conversation
… EncryptedJson
The JSON index could only be reached from typed Rust: a data plan parsed
`{"json": ...}` and refused it (json_index_pending), and the json_tree
family was missing from the EQL catalog, so no binding could name it.
The catalog now lists the family once, as a document family
(eql_domains::DOCUMENT_FAMILIES): its stored type is the target
`EncryptedJson` (public.eql_v4_json_tree), produced from the new `json`
value kind and carrying the `json` index, with EncryptedJsonQuery,
EncryptedJsonLeafQuery and EncryptedJsonEntry beside it. eql-codegen
renders the target table row, v4::all()/all_query() and the Go types from
it; the scalar SQL and struct materializers never see it, since its SQL
and Rust bodies are hand-written and v4 only.
A plan field names the target with the JSON index's options,
`"target": {"name": "EncryptedJson", "options": {...}}` (the bare name
for the defaults). The plan resolves the options once, when it is built,
so the field's writes, queries and reads run under one definition, and
refuses options on a target that takes none, a json field with no target,
and a sealed null on a document. TargetResolver::encrypt takes the
options; eql-bindings' encrypt_with seals the document through the same
JsonTree layout typed Rust uses.
The document crosses the binding as UTF-8 JSON text and is parsed once,
by json::parse_text: an object that names a member twice and an integer
literal outside the 64-bit range are refused rather than resolved or
rounded, and every refusal carries a code and a position, never the text
(TargetError::Json). It opens back to its text in a bytes leaf.
tests/json_tree_data_plan.rs holds the data plan to the typed layout: the
same selectors, ordering terms and containment projection for the same
document under the same column, each path opening what the other sealed,
and a batch of documents in one key request.
Refs #1347, #1348, #1349
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
encrypt/eql gains the generated EncryptedJSON, EncryptedJSONQuery,
EncryptedJSONLeafQuery and EncryptedJSONEntry. The Go-name rule, here and
in eql-codegen, spells the word Json as JSON wherever it is a word, not
only as a prefix (EncryptedJson is EncryptedJSON).
internal/record carries the `json` kind and a target's options
(Field.TargetOptions), written as the `{"name", "options"}` target the
engine reads. A hand-built plan can store and open a document through
the guest with the EQL types: json_tree_test.go round-trips scalar roots,
arrays, null and a u64, and shows a refused document is ErrEncoding with
the engine's code and none of the text. stashgen support follows.
Refs #1347, #1348, #1349
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
🦋 Changeset detectedLatest commit: db6dca0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
| Layer / File(s) | Summary |
|---|---|
JSON text and dynamic value handling packages/stack-encrypt/src/json/*, packages/stack-encrypt/src/dynamic/*, packages/stack-encrypt/fuzz/* |
Adds JSON parsing, serialization, wiping, JSON value-kind handling, target options, and JSON-specific errors. |
Document-family catalog and SQL contracts packages/eql/crates/eql-domains/*, packages/eql/crates/eql-codegen/*, packages/eql/crates/eql-bindings/src/v4/*, packages/eql/src/v3/* |
Adds document-family metadata and inventories, renames JSON domains and query types, and confines selected SQL surfaces to EQL v3. |
JSON data-plan and target encryption packages/eql/crates/eql-bindings/src/encryption/targets.rs, packages/eql/tests/encryption/tests/json_tree_data_plan.rs, packages/eql/tests/encryption/tests/typed_queries.rs |
Routes JSON document targets through option-aware encryption and decryption. Data-plan validation checks JSON input, target options, context, and document-query refusals. |
Go record and generated bindings languages/golang/encrypt/eql/*, languages/golang/internal/record/*, languages/golang/stashgen/*, languages/golang/encrypt/json_tree_test.go |
Adds JSON record and target-option wiring, generated searchable-JSON types, updated Go name conversion, and explicit stashgen rejection for unsupported JSON fields. |
Priority:
Estimated code review effort: 5 (Critical) | ~100 minutes
Change: Feature
Sequence Diagram(s)
sequenceDiagram
participant DataPlan
participant read_json
participant TargetResolver
participant encrypt_with
participant KeysetCipher
DataPlan->>read_json: validate JSON source
DataPlan->>TargetResolver: resolve target and options
TargetResolver->>encrypt_with: encrypt document with options
encrypt_with->>KeysetCipher: seal JSON-tree document
Possibly related issues
- cipherstash/stack#1349: The PR admits JSON indexes in data plans and adds JSON source handling, directly addressing the issue’s stated refusal path.
- cipherstash/stack#1348: The PR adds document-family catalog entries and generated searchable-JSON types.
- cipherstash/stack#1351: The PR updates the shared refusal path for unsupported generated JSON fields, while field generation remains unsupported.
- cipherstash/stack#1347: The PR implements the Rust data-plan and EQL catalog portion of encrypted JSON support; Go field generation and reads remain outside this change.
Suggested reviewers: auxesis
Merge Risk: 
c22b6
-
No concrete regression in JSON document storage is established here; the change is mergeable after normal checks.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 57.48% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 214 functions across 46 files. (1 skipped… Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely describes storing a JSON document through a data plan with the JsonSearch target. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 57.48% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 214 functions across 46 files. (1 skipped: 1 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
This comment has been minimized.
This comment has been minimized.
Contributor
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 152 | 0 | 20 | 0 |
Every mutant in the changed lines was caught by a test.
This was referenced Oct 25, 2026
Merged
…son in the per-type PostgreSQL test The fuzz harness's TargetResolver still had encrypt's old signature, so fuzz regression (check-record) did not compile. The per-type PostgreSQL test failed on EncryptedJson, the first producible type with no plaintexts of its own: a document has operators of its own and is run in PostgreSQL with the generated fields. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…ions::json The mutants gate found the integer pre-scan's index arithmetic unpinned (a scan that left a string early passed while the number it reached fitted, and several replacements hung). The scan now walks slices with no arithmetic to mutate, and tests pin a long number inside a string, a number after an escaped quote, an unterminated string, a lone sign and a malformed token. The visitor drops visit_string, visit_none and visit_some, which serde_json never calls. wipe_value and TargetOptions::json get unit tests. Two survivors are excluded with their reason: the strict visitor's expecting and visit_unit (equivalent), and JsonPlaintext's Drop (no safe code can read a value after it is dropped; what it runs is wipe_value, now pinned). Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Automated code review of this pull request at
|
- json::parse_text is the engine's own strict parser, held to serde_json by a differential property test and a new fuzz:json-text target, so no heap block it frees holds the document (serde_json's scratch buffer and partial documents on refusal did); to_text sizes its buffer first. A heap-spy test watches the boundary. - check_source reads a json field's document by the engine's rules, so a malformed one is refused before any key, as encrypt refuses it. - A present "options": null is refused rather than read as defaults. - eql-bindings' query() refuses a document by name (DocumentQuery); a test pins the engine's and eql-bindings' rules for which targets take JSON options to agree. - Go: the per-type PostgreSQL test no longer counts EncryptedJson; opened JSON numbers compare as text; EQLGoName decodes a rune; stashgen names the missing document support instead of a kind mismatch. - Docs: ValueKind's break under Breaking, the Go CHANGELOG no longer points at internal/record, the generated eql doc sentence reads. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Review response to the findings listed above (fixed in ffd4212 unless noted):
Found while fixing 2 and 3, and not fixed here: the JSON index's own seal path ( CI failures on the previous push (the |
This comment has been minimized.
This comment has been minimized.
…its wipes A local mutants run timed out on the parser's index arithmetic: every `at += 1` replaced by `at *= 1` loops forever, which the gate counts as a survivor. The parser now holds the unread text as a slice and moves by re-slicing, so a mutation changes what it reads rather than hanging. Tests pin a string's buffer to its raw length, an array's growth from four by doubling, and a container closed by the other bracket. JsonPlaintext's Drop is no longer excluded from mutation: heap_hygiene's new a_json_document_read_from_a_frame_leaves_no_plaintext_on_the_heap fails when it is a no-op. The strict-visitor exclusion goes with the visitor. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The local mutants run timed out on `len == capacity` replaced by `!=`: every push then doubled the buffer and wiped its ever larger spare capacity. Asking the buffer for its spare capacity leaves nothing there to mutate; the growth test still pins the sizes. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Follow-up to the response above. A local
|
…arbage CI's Linux run of EQL's heap_hygiene flagged a freed array buffer that held the marker in its unused slots: bytes another allocation freed before the watch began (the test's own format! growth, or the text-target test running beside it in the same process). The inputs are now built into strings sized before they are written, and EQL's boundary test, whose binary runs its tests as threads of one process, parses strings only; stack-encrypt's, run one test to a process, keeps containers. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
coderdan
added this pull request to stack #1377
October 25, 2026 22:47
EQL v3 stores a searchable JSON document as `public.eql_v3_json_search`, queries it with `eql_v3.query_json` and returns `public.eql_v3_json_entry` from `->`. In v4 the JSON tree scheme implements that document, so its domains take the same names: `public.eql_v4_json_tree` is now `public.eql_v4_json_search`, `public.eql_v4_json_tree_entry` is `public.eql_v4_json_entry`, and the two query operands are `eql_v4.query_json` and `eql_v4.query_json_leaf`. `json_tree` stays the implementation's name: the SQL directory, the Rust structs and the SQL functions keep it. Those names were already taken in the v4 bundle. The SteVec SQL under src/v3/json/ carried no version line, so it rendered into both bundles and gave v4 a SteVec `json_search`, `json_entry` and `query_json` that nothing writes. The five files, and the generated json_entry_<T> cross files that bind the SteVec entry to each scalar family's query operands, now carry `-- ONLY EQL VERSION: 3`; eql-codegen writes that line into the cross files. The v3 bundle changes only by those 23 comment lines. With the SteVec SQL gone from v4, the v4 target table drops its SteVecDocument row, which was never producible, and the generated Go package drops the SteVecDocument and SteVecQuery types with it. Both bundles still install side by side: every public domain each owns, the other owns under its own prefix, and both lints report no error. Refs #1347, #1348 Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
…s v3 does The previous commit gave the JSON tree domains v3's names; this gives the contract names that point at them the same treatment. A data plan, a Go `encrypt_into` tag and the target table named the stored document `EncryptedJson`, a role name chosen before the domain had one. It is now `JsonSearch`, over `public.eql_v4_json_search`, with `JsonSearchQuery` (`eql_v4.query_json`), `JsonSearchLeafQuery` (`eql_v4.query_json_leaf`) and `JsonEntry` (`public.eql_v4_json_entry`). `Json` stays the storage-only type over `public.eql_v4_json`, so the two v3 words mean the same thing in v4. The catalog's document family (`eql_domains::JSON_TREE`) is the only place the names are written; the target table, the stack-encrypt descriptors the tests pin, the guest's producible list and the generated Go package (`eql.JSONSearch`, `JSONSearchQuery`, `JSONSearchLeafQuery`, `JSONEntry`) follow from it, regenerated with `mise run types:generate`. The Rust structs keep the implementation's name (`JsonTreeDocument` and its companions), as v3's `SteVecDocument` does. Refs #1348, #1349 Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
The changesets and CHANGELOGs for the unreleased JSON tree family named the target `EncryptedJson` over `public.eql_v4_json_tree`, and the target changeset said no SQL changed. They now give the names the family ships under, say that the v4 bundle no longer carries the SteVec domains and that the v3 bundle gains only the version-directive comment lines, and the Go CHANGELOG lists the removal of `eql.SteVecDocument` and `eql.SteVecQuery` under Breaking. AGENTS.md and packages/eql/AGENTS.md record which surfaces are confined to which version; the design plan records the decision and keeps its earlier sections' placeholder names as history. Refs #1348, #1349 Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
The list of the four domains in src/v3/json_tree/types.sql was aligned for the `json_tree` names; with the shorter and longer v3 names the dashes and continuation lines no longer lined up. Comment only. Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
The target descriptor's table said a type's indexes include `json` for a SteVec document. The v4 target table no longer has a SteVec row; the one type carrying the JSON index is the document family's stored type, `JsonSearch`, as eql-bindings' own table says. Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
coderdan
changed the title
coderdan
marked this pull request as ready for review
October 26, 2026 04:54
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
cipherstash-agent
left a comment
cipherstash-agent
left a comment
Contributor
There was a problem hiding this comment.
Review attestation — automated /code-review medium plus cross-check-review-findings, run against head 5fef296. No critical or blocking issues found.
Seven findings were raised, checked against head, then cross-checked against the existing PR feedback. 4 dropped (1 refuted: float_roundtrip is enabled in stack-encrypt/Cargo.toml:82; 3 unconfirmed or style-only). 2 are new and minor:
- minor
packages/stack-encrypt/src/dynamic/record.rs:2391:check_fieldcallsread_jsonand discards the result, thenencrypt_document(packages/eql/crates/eql-bindings/src/encryption/targets.rs:427) parses and wipes the same document again, so each row pays two full parses. Keep the parsedJsonPlaintextfrom the check, or accept the cost knowingly for large documents. - minor
languages/golang/internal/record/record.go:332:Plan.ValidaterejectsTargetOptionswithout a target, but not aJSON-kind field with no target. That fails late in the engine (json_needs_target) without the field context the Go validator adds.
Not verified here: the live-credential and PostgreSQL suites, and the failing TestEQLTypesRoundTripAndQueryInPostgres/EncryptedJson reported by Blacksmith on an earlier commit. Re-check CI on the current head.
Generated by Claude Code
coderdan
force-pushed
the
feat/json-tree-data-plan
branch
from
October 26, 2026 05:03
5fef296 to
43faafc
Compare
Contributor
Author
|
History rewritten on this branch, authorship only. Twelve commits pushed here this morning by a cloud session were unsigned and authored |
coderdan
added a commit
that referenced
this pull request
Oct 26, 2026
…keys before freeing them The data-plan work in #1372 found that sealing a JSON document freed key names, scalar values and the data key unwiped, and left it for the index work. A heap spy over this branch's seal, open, read, extraction and needle paths, under the deterministic key source, found 42 blocks for one sealed-root seal (43 with the root open), 4 for each whole open and each `->`, 7 for each subtree read, 40 for a containment needle and 20 for an equality needle. Each holds a marker the document carries or the data key the seal used. After this change every one of those paths frees none. Where they came from, and what holds them now: - JsonPath carries key names, and the walk builds one per node: a path now wipes its key names when it drops, and `selector_bytes` and `token_bytes` are sized before they are written (a growing buffer freed a copy) and come back in `Zeroizing`. - The value selector framed the scalar, and the rank framed the key name, through `Context::pae`, whose buffer is not wiped: both now frame into a sized `Zeroizing` buffer with the same bytes (a unit test holds the equality), and the canonical value is `Zeroizing` too. The rank map no longer clones key names. - The document `seal_pending` owns (a clone, in borrowed mode) is now wiped when sealing returns (`wipe_value`); a query's needle, owned by the terms operation, is wiped through a new `SourceMode::release`; the copies `AtPath` checks make are wiped after the check. - The data key was moved out of the response buffers (`pop_front`, `drain`, `into_iter`), leaving its bytes in memory freed later. Each response is now handed out as a clone and the original drops in place, where `DataKey` wipes itself; `dispatch` clones retrieved keys the same way. This is the shared settling path, so every decryption benefits. `tests/heap_hygiene.rs` gains the JSON index's tests, with a key source that records each data key so the spy looks for it as well as the marker, and a control showing a freed key copy fails the watch. Each fix was checked by removing it and watching a test fail. One case is left out and written down: the path an `AtPath` caller passes as an owned `String` is cloned by the query builder and dropped by the plan chain, which knows only `P: AsRef<str>`. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
freshtonic
reviewed
Oct 26, 2026
freshtonic
left a comment
freshtonic
left a comment
Contributor
There was a problem hiding this comment.
This slice is well built, and I found no blocking defect.
- Single source. The
DOCUMENT_FAMILIEScatalogue is the single source for the target row,v4::all()/all_query()and the Go types.the_tableandv4_inventoryhold the hand-writtenjson_tree_domain_type!SQL names to the catalogue. - Agreement. The guest
targets.rs,record.goandeql_gen.goagree onJsonSearch/json_tree/json/public.eql_v4_json_search/eql_v4.query_json. - Context binding.
encrypt_documentseals underIdentifier::from_label(context), anddecrypt_documentopens withExpectedContextfrom the same column. This matches the typed path. Error payloads carry only a code and a position, never text. - Validation. Options are resolved once, when the plan is built. A document that is not valid JSON is refused in
check_sourcebefore any key request. - The SQL rename is safe for installed databases. The json_tree SQL exists only on
nextand the stack under it, and the published@cipherstash/eql@3.0.6has noeql_v4object. The rename is mechanical, and each new name is unique in its bundle, so the v3 bundle gains comment lines only, as the changeset says. - Changesets. There is one
@cipherstash/eqlminor change in the root.changeset/, and the two pending json_tree changesets are reworded.eql-bindingsis in lockstep and has a CHANGELOG entry. The breaking changes in stack-encrypt and Go are under Breaking. No skill needs a change, because noskills/*file nameseql_v4.fuzz:json-textis in bothfuzz.ymlmatrices, indocs/fuzzing.mdand in the corpus.
Checks that ran: cargo test -p eql-codegen -p eql-domains and mise run types:check (pass, no drift in generated files); the stack-encrypt json, kind and record-target tests (102 pass); and the encryption tests heap_hygiene, json_tree_data_plan and targets (pass). The Go tests and PostgreSQL were not run here.
Two small points that have no line in the diff:
- Nit: no SQL
NULLtest. No test inlanguages/golang/encrypt/json_tree_test.goorjson_tree_data_plan.rsseals an absent value (nil) on thenull: "sql"field. A row with no value is the common case for*json.RawMessage. Please add one row. - Nit:
packages/eql/tasks/build.sh:93-95. This comment still says that only the range surface is limited to one EQL version. The json_tree family (v4) and now the SteVec SQL (v3) are limited too.packages/eql/AGENTS.mdnow describes all three surfaces correctly.
Brings in the vitaminc d7de535c4 pin for vcffi/vcvalue and the go-release guard that refuses a final version while go.mod names a pseudo-version, with the opaque storage profile's two fixes beneath them. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings in the mutants gate fix from #1381, which reads the pull request's real base from the API and checks out the head, so the gate no longer diffs all of next and times out on this stack. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings in next's JSON tree cascade and random storage order (#1379). Conflicts resolved to #1379's names and wording, keeping this branch's data plan target, its JSON text parser and the JsonSearch rename: JsonProjection is JsonSelectorSet in the fuzz table and the JsonQuery docs, the leakage page takes #1379's opening under the eql_v4_json_search name, json_tree_path_query's comment says stored order under the json_entry domain, and the oracle's hostile-row updates find each entry's slot (the order is random) and cast to eql_v4_json_search. The changelog keeps both sides' entries. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
… as sets A sealed document now stores its entries in a random order (root first) and its containment selector set shuffled, so two seals of one document differ in order. The parity test compared entries by position and q by bytes, which only held while both seals walked the document the same way. It now compares the root entry, the entries as a sorted multiset of (selector, ordering term, parent tag's place), and the two selector sets through JsonSelectorSet's set equality. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The task's description still listed JsonProjection, the selector set's former name, among the decoders the target fuzzes. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The parser's rustdoc and the changelog said no heap block it frees holds plaintext. That holds for strings and array buffers, but an object's members go into serde_json's BTreeMap, which splits a full node by copying half its slots and does not clear the ones it leaves: a number or boolean member of an object with twelve or more members can stay in a freed node. Strings move by pointer, so their bytes never sit there. Both now say so, and that numbers and booleans are wiped best-effort. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
json_frame copied the text into the Protected value and wiped the original. Taking the buffer moves it, so the plaintext exists once. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…t options are refused check_field parses a json field's document and drops it, and the resolver parses it again when it seals: the check runs before any key and needs no resolver, and a resolver takes the field's value as a frame. Options are read in the JSON index's grammar whatever the target, so options outside it are an invalid target while well-formed options on a target that takes none are target_options; the rustdoc on TargetWire now says both. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…ta plan Nothing opened a JsonSearch value under another column. The data plan test now seals under users/profile and opens under users/other, which is refused as a context mismatch before any key is retrieved; the Go guest test does the same and gets ErrContextMismatch. A row whose document is a frame null, or absent, under null=sql stores no value and opens as a null, beside a JSON null that round-trips as a document: the doc comment that promised this had no test. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The engine refuses a json field that names no target (json_needs_target) after the plan crosses the guest. Validate already refused target options without a target; it now refuses a JSON document with no target too, and names the field before the guest is asked. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…ores documents build.sh's comment named only the range surface as confined to one EQL version; the JSON tree family (v4) and the SteVec SQL (v3) are confined too. The per-type PostgreSQL test skips JsonSearch, and its comment now names TestJSONDocumentsQueryAndReadInPostgres, which stores documents through the generated fields into an eql_v4_json_search column. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Review-body findings, at c22b688:
This branch also merged #1379 (the cascade and the random stored order) through #1370; b944419 compares the data plan's stored document with the typed one as sets, since each seal draws its own order. |
Carries the vitaminc Go bindings pin to 06568ba93 on vitaminc main. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
coderdan
removed this pull request from stack #1377
October 26, 2026 12:41
coderdan
added a commit
that referenced
this pull request
Oct 26, 2026
Brings in the Go json_tree stack's first pull requests (#1369, #1370, #1372). The only conflicts were adjacent CHANGELOG entries in the Go module and stack-encrypt, resolved by keeping both. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This lets a language SDK store an encrypted JSON document (EQL's
json_treetype) through a data plan. A data plan is the declaration a non-Rust SDK sends the Rust engine (Stack Encrypt) as data, field by field; until now the engine refused the JSON index there, and thejson_treefamily was missing from the EQL catalog, the one list every generated type comes from. EQL is the SQL library installed in the customer's PostgreSQL.A field now names the target
JsonSearch(stored inpublic.eql_v4_json_search), declares the newjsontype, gives the JSON index's options, and hands each document over as UTF-8 JSON text. The engine parses the text by one set of rules and seals it through exactly the layout typed Rust uses. This is slice A plus the full-document half of slice B of the plan in #1353; queries and reads follow in the next pull request.Changes
Catalog and generated code (
packages/eql)eql-domains: aDocumentFamilylist,DOCUMENT_FAMILIES = [JSON_TREE], naming the four domains and their roles (stored, query, leaf query, entry). Kept apart fromCATALOGso the scalar SQL and struct generators never see it: this family's SQL and Rust bodies are hand-written and EQL v4 only.eql-codegen: a target-table row forJsonSearch(plaintextjson, indexjson, needleJsonSearchQuery);v4::all()/v4::all_query()list the document and its two query operands; the Go typesJSONSearch,JSONSearchQuery,JSONSearchLeafQuery,JSONEntry. The Go-name rule spells the wordJsonasJSONwherever it is a word.eql-bindings:encryption::targets::encrypt_with(name, options, …)sealsJsonSearchunder the field's options;decryptopens it to its text. New errorsTargetError::Json,TargetError::OptionsandTargetError::DocumentQuery(a one-plaintextqueryof a document).Engine (
packages/stack-encrypt)jsonvalue kind (ValueKind::Json): a document crosses as text or bytes and opens back to its text as bytes (dynamic::read_json,dynamic::json_frame).json::parse_text, the one JSON parser at the boundary, and the engine's own (not serde_json's deserializer), so no heap block it frees holds the document: refuses an object that names a member twice, an integer literal outsidei64::MIN ..= u64::MAX, and text that is not JSON, each with a stable code and a position, never a byte of the text. Held to serde_json by a differential property test and a new fuzz target,fuzz:json-text.json::to_textsizes its buffer before it writes.check_sourcereads a document by these rules too, so a malformed one is refused before any key."target": {"name": "JsonSearch", "options": {"sealed_root": …, "ordered": […]}}(or the bare name). Options are resolved once when the plan is built (TargetOptions::Json), so one definition serves writes, queries and reads. Refused: options on a target that takes none, ajsonfield with no target, a sealed null on a document.TargetResolver::encrypttakes the field's options (breaking for an outside resolver; CHANGELOG).Go (
languages/golang)internal/record: thejsonkind andField.TargetOptions.encrypt/json_tree_test.go: a document round trip (numbers compared as text) and refusals through the real WebAssembly guest (the Rust engine compiled for Go) with the EQL types.stashgennames the missing support for a field ofJsonSearch(the next pull requests add it) instead of a kind mismatch; the per-type PostgreSQL test skips the document type rather than counting it.Docs: changeset for
@cipherstash/eql, CHANGELOG entries forstack-encrypt,eql-bindingsand the Go module, and a line indocs/plans/2026-10-17-json-tree-index.md.Assumptions
The five owner decisions in #1347 are not made; this stack takes the plan's recommended answer to each:
json.RawMessage(required) and*json.RawMessage(optional,null=sql). Here only the wire is fixed: a document crosses as text or bytes.json::parse_text); Go never builds a tree.Also assumed:
Jsonstays the storage-only type, andJsonSearch(Goeql.JSONSearch) is the searchable document atpublic.eql_v4_json_search, queried witheql_v4.query_jsonandeql_v4.query_json_leaf, with entriespublic.eql_v4_json_entry. The SQL domains were renamed fromjson_treeto match (theeql_v4.json_tree_*functions keep their names), and the v4 bundle no longer renders the SteVecjsonsurface: those SQL files are confined to EQL v3.{"json": …}beside"c"); a document is reached only through the target, as the plan says.[]byteit can wipe), and is accepted in a text or bytes leaf.table/column, and extended or row-bound contexts stay refused (plan section 5).Verification
After merging #1370, which brings in next's JSON tree cascade and random stored order (#1379), and the review fixes, at c22b688, on macOS arm64, Rust 1.94.1 (the root
mise.tomlpin), Go 1.26.10, PostgreSQL 17.10 (a fresh database per run):cargo fmt --all --checkandRUSTFLAGS=-D warnings cargo clippy --workspace --all-features --tests: clean.mise x --env test -- cargo nextest run --workspace --all-features: 1,526 passed, 1 skipped (includingtests/heap_hygiene.rsand the trybuilduibinary).packages/eql:cargo fmt --check, clippy (-D warnings) andcargo nextest run --workspace --exclude eql_tests(625 passed);mise run types:generatethentypes:check(no drift);mise run test:encryption(534 passed, including the newa_document_does_not_open_under_another_columnanda_sql_null_stores_no_document);mise run test:encryption:postgreswith the v4 bundle built from the tree: every suite passed, thejson_treeoracle 14 of 14.wasm:guest:build,:eql, both deterministic builds,wasm:auth-guest:build),mise run wasm:guest:test,mise run go:test(every packageok),mise run go:lint(0 issues),gofmt -l(clean) andgo generate ./...(no drift), includingTestJSONDocumentsDoNotOpenUnderAnotherColumn.cargo metadata --lockedin the root workspace, both guest workspaces andpackages/eql: clean.Related
Stack
Merge order is bottom to top; each pull request is based on the one below it.
d7de535c4(the docs: refresh root README around the three Stack pillars #526 merge) by git reference (temporary)JsonSearchstored through a data plan (engine, EQL catalog, guest, Go runtime) (this pull request)https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Summary by CodeRabbit