feat(eql-bindings): query EQL v4 types through Stack Encrypt's typed query chain

.changeset/eql-v4-bindings.md

@@ -2,7 +2,7 @@
 '@cipherstash/eql': minor
 ---
 
-**The `eql-bindings` crate has an `eql_bindings::v4` module, with the `stack-encrypt` feature: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt, so the module exists only with that feature and its types derive Stack Encrypt's traits without a further gate. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, and `TextMatch` (3-grams, `k = 6`, a 2048-bit filter, pinned as `eql_bindings::encryption::EqlMatch`; text shorter than three characters is refused, stored or queried, and only `stored @@ query` matches until #1171 is fixed), from the Rust primitives `i32`, `i64`, `bool`, `f32` and `f64`; the Go SDK's `encrypt/eql` table marks them producible. Stack Encrypt also produces the number equality types `IntegerEq`, `BigintEq`, `RealEq` and `DoubleEq`, whose `=` compares as PostgreSQL compares the plain types: an integer has one term whatever its width, and for floats `-0.0 = 0.0` and `NaN = NaN` (ADR-0002's semantic baseline). Every type it cannot produce yet names the issue that tracks its blocker. The by-name target entry points (`eql_bindings::encryption::targets::encrypt`, `decrypt` and `query`) exchange the plaintext as Stack Encrypt's `dynamic::ValueFrame`, one value as its transport frame, rather than a value tree. They, and the generated `v4::targets` table, need the new `targets` feature, which turns on `stack-encrypt` and Stack Encrypt's `dynamic` feature. `stack-encrypt` alone no longer builds Stack Encrypt's `dynamic` layer, so a Rust program using the typed v4 types through `encrypt_as` doesn't compile it.
+**The `eql-bindings` crate has an `eql_bindings::v4` module, with the `stack-encrypt` feature: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt, so the module exists only with that feature and its types derive Stack Encrypt's traits without a further gate. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, and `TextMatch` (3-grams, `k = 6`, a 2048-bit filter, pinned as `eql_bindings::encryption::EqlMatch`; text shorter than three characters is refused, stored or queried, and only `stored @@ query` matches until #1171 is fixed), from the Rust primitives `i32`, `i64`, `bool`, `f32` and `f64`; the Go SDK's `encrypt/eql` table marks them producible. Stack Encrypt also produces the number equality types `IntegerEq`, `BigintEq`, `RealEq` and `DoubleEq`, whose `=` compares as PostgreSQL compares the plain types: an integer has one term whatever its width, and for floats `-0.0 = 0.0` and `NaN = NaN` (ADR-0002's semantic baseline). Every type it cannot produce yet names the issue that tracks its blocker. A Rust caller queries those types through Stack Encrypt's typed query chain: `cipher.query(op::Eq, v).on(&plan)` on a plan for `TextEq` or a number equality type returns the type's query twin, and `cipher.query(op::Matches, text).on(&plan)` on a plan for `TextMatch` returns a `MatchTwin<TextMatchQuery>` with `verify` and `target_limit`, the same bytes and answers as the data path the Go SDK uses (`eql_bindings::encryption::query`; an operator a type does not answer does not compile). The by-name target entry points (`eql_bindings::encryption::targets::encrypt`, `decrypt` and `query`) exchange the plaintext as Stack Encrypt's `dynamic::ValueFrame`, one value as its transport frame, rather than a value tree. They, and the generated `v4::targets` table, need the new `targets` feature, which turns on `stack-encrypt` and Stack Encrypt's `dynamic` feature. `stack-encrypt` alone no longer builds Stack Encrypt's `dynamic` layer, so a Rust program using the typed v4 types through `encrypt_as` doesn't compile it.
 
 **The EQL v4 SQL bundle has range query domains and `eql_v4.between`, so a `BETWEEN` query is one operand.** The v4 bundle (`cipherstash-encrypt-v4.sql`) is built from this source and reaches users as the `eql-v4-install.sql` attached to each Go release; neither the npm package nor the crate ships it, and the v3 SQL they ship does not change. Every ordered family has one range query domain per order term, beside its single-bound ones: `eql_v4.query_integer_range_ore` beside `eql_v4.query_integer_ord_ore`, `eql_v4.query_integer_range_ope` beside `eql_v4.query_integer_ord_ope`, and the same for `smallint`, `bigint`, `date`, `timestamp`, `numeric`, `text`, `real` and `double`. A range is `{"v": 4, "i": …, "lo": <term>, "hi": <term>}`: each bound is the single-bound domain's `ob` (ORE) or `op` (OPE) value, in a `{v, i, lo, hi}` envelope. Stack Encrypt does not build that envelope yet; it will need to when the ordered types become producible. `eql_v4.between(column, range)` is true when the column's order term lies between the bounds, both included, and has an overload for every ordered column domain with the range of the term it stores (`_ord_ore` and `_search_ore` take the ORE range; `_ord`, `_ord_ope` and `_search` the OPE one). It is an inlinable SQL function, so a btree on the column's order term (`eql_v4.ord_term_ore(col)` or `eql_v4.ord_term(col)`) serves it as one index range scan. Write it schema-qualified (`BETWEEN` is an SQL keyword, so a bare `between(...)` does not parse), and cast the range to its exact domain (`$1::eql_v4.query_integer_range_ore`): an uncast `$1`, a `jsonb` parameter, or a range of another term or family fails with SQLSTATE 42725 ("function … is not unique"), unlike the single-bound operators, which resolve an untyped `$1`. An ORE range scan needs the `ore_block_256` btree operator class, which the installer creates only as a superuser; on an install without it (Supabase and other non-superuser roles), the ORE range domains are disabled like the single-bound ORE domains, and a value cast to one raises `feature_not_supported` naming the family's OPE range. In the crate, `eql_bindings::v4` (`stack-encrypt` feature) has a type for each range domain (`IntegerRangeOreQuery`, `TextRangeOpeQuery`, …, listed by `v4::all_range_query()`). EQL v3 has no range domains: its SQL and types are unchanged (ADR-0009, decision 7).
 

packages/eql/crates/eql-bindings/CHANGELOG.md

@@ -22,6 +22,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `eql_v4.query_<family>_range_<ore|ope>` domains in the v4 SQL bundle. No
   range type derives Stack Encrypt's traits yet, because the ordered types
   they range over are not producible. There are no v3 range types.
+- **Rust queries an EQL v4 field through Stack Encrypt's typed query chain**
+  (`stack-encrypt` feature, #1238). `cipher.query(op::Eq, v).on(&plan)` on a
+  plan for `TextEq`, `SmallintEq`, `IntegerEq`, `BigintEq`, `RealEq`,
+  `DoubleEq`, `DateEq`, `TimestampEq` or `NumericEq`
+  (`Plan::value::<S>().context("table/column").encrypt_into::<T>()`)
+  returns the type's query twin (`TextEqQuery`, …), and
+  `cipher.query(op::Matches, text).on(&plan)` on a plan for `TextMatch`
+  returns a `MatchTwin<TextMatchQuery>`: the twin, which it serializes as,
+  with `verify` and `target_limit` under `EqlMatch`. The query is the bytes
+  the data path (`stack_encrypt::dynamic::query_term`) derives for the same
+  field, and `verify` and `target_limit` answer as the data path's do.
+  eql-codegen writes the `op::Answers` impls into `v4::answers`, which needs
+  no `dynamic` layer and so not the `targets` feature, from each type's
+  indexes, by the rule the data path answers an EQL type's operators
+  with. A producible type whose twin carries an order term (`IntegerOrd`,
+  `TextOrd`, …) answers nothing in the typed chain yet: the chain sends one
+  index's term as the twin, and those twins need an order term, or several
+  terms at once. The encryption tests hold that gap to exactly the ordered
+  types, so query them through the data path for now. An operator a type
+  does not answer, or another index named with `.via(..)`, does not
+  compile. Each answering type is an
+  `op::LabelledTarget`. The label must name the column, table then column,
+  and the query must not be extended, or the awaited query is refused
+  before any keyset is resolved, with a `QueryLabelError` in
+  `Error::Other`. `eql_bindings::encryption::query` holds `TwinIndex`
+  (Stack Encrypt's index, its query sent as the twin, for the twin's own
+  plaintext only), `MatchTwin`, `QueryTwin` and `QueryLabelError`.
+  `Hmac256` and `BloomFilter` now declare their indexes (`EncryptFrom::indexes`),
+  so a type holding one declares the equality or match index, and `EqlMatch`
+  implements `op::MatchCase` (case-insensitive).
 - **`eql_bindings::v4` needs the `stack-encrypt` feature.** Only Stack
   Encrypt produces an EQL v4 value (ADR-0002), so the module is compiled with
   that feature alone, and its producible types derive `EncryptFrom` and

packages/eql/crates/eql-bindings/README.md

@@ -229,6 +229,71 @@ descriptor is `<table>/<column>`, two parts, the same context and descriptor as
 a two-segment `stack_encrypt::Label`. A table or column containing `/` is
 escaped in the descriptor rather than read as two names.
 
+### Querying an EQL v4 field from Rust
+
+A query names an operator, and a plan for the column's EQL type finds the
+index that answers it (Stack Encrypt's ADR-0009). The term that comes back
+is the type's query twin, the value the `eql_v4.query_*` domain takes:
+
+```rust,ignore
+use eql_bindings::v4::text::{TextEq, TextMatch};
+use stack_encrypt::op::{self, Probabilistic, Stats};
+use stack_encrypt::Plan;
+
+// A plan for the column's type, under the label that names the column:
+// "users/email" is Identifier::for_column("users", "email").
+let email_plan = Plan::value::<String>().context("users/email").encrypt_into::<TextEq>().build()?;
+let query = cipher.query(op::Eq, "alice@example.com").on(&email_plan).await?; // a TextEqQuery
+// WHERE email = $1::eql_v4.query_text_eq, binding sqlx::types::Json(&query)
+
+let name_plan = Plan::value::<String>().context("users/name").encrypt_into::<TextMatch>().build()?;
+let query = cipher.query(op::Matches, "ali").on(&name_plan).await?; // a MatchTwin<TextMatchQuery>
+let fetch = query.target_limit(10, &Stats::declared(1_000_000)).fetch_or(50);
+// WHERE name @@ $1::eql_v4.query_text_match LIMIT $2, then keep the
+// decrypted rows for which query.verify(&name) is true.
+```
+
+| v4 type | Operator | Query term |
+| --- | --- | --- |
+| `TextEq` | `op::Eq` | `TextEqQuery` |
+| `SmallintEq`, `IntegerEq`, `BigintEq`, `RealEq`, `DoubleEq` | `op::Eq` | `SmallintEqQuery`, `IntegerEqQuery`, `BigintEqQuery`, `RealEqQuery`, `DoubleEqQuery` |
+| `DateEq`, `TimestampEq`, `NumericEq` | `op::Eq` | `DateEqQuery`, `TimestampEqQuery`, `NumericEqQuery` (from `v4::plaintext::Date`, `Timestamp`, `Decimal`) |
+| `TextMatch` | `op::Matches` | `MatchTwin<TextMatchQuery>` |
+
+`MatchTwin` serializes as the twin and adds `verify` and `target_limit`
+under `EqlMatch`: a filter can match a row that does not hold the query's
+text, so keep only the decrypted rows that verify. An exact query has no
+`verify`. An operator a type does not answer does not compile: `op::Lt` on
+`TextEq`, or `op::Matches` on `IntegerEq`. Nor does naming another index
+with `.via(..)`, such as the bare `Equality`, whose term the type's SQL does
+not take. The ordered types (`IntegerOrd`, `TextOrd`, …) answer nothing in
+the typed chain yet: their twins need an order term, or several terms at
+once, and the chain sends one index's term. Query them through the data
+path (`stack_encrypt::dynamic::query_term`) for now.
+
+The label must name the column the value was written under, as two
+segments, table then column. A label of any other length, and a query
+extended with `.extend(..)` (whose terms would be derived under more than
+the column the twin stores), are refused when the query is awaited, before
+any keyset is resolved, with an `encryption::query::QueryLabelError` in
+`Error::Other`. A label's segments are plain, so a table or column name
+that is not a plain label segment cannot be named on a plan, and such a
+column, which `Identifier::for_column` accepts, cannot be queried this way:
+for instance a name that starts with a digit, `-` or `b64:`, or one holding
+`/`, `(`, `)`, or a control or invisible format character.
+The plan is for queries: write with
+`keyset.encrypt_as(&value, column)` and open with `cipher.decrypt_as`, since
+an EQL type is sealed under its column rather than the plan's label. The
+query is the same bytes the data path derives for a field that names the
+type (`stack_encrypt::dynamic::query_term`), which is what the Go SDK sends,
+and `tests/encryption/tests/typed_queries.rs` checks that for every type and
+operator. The `op::Answers` impls are generated into `src/v4/answers.rs`
+from the catalog by the rule the data path uses, and need only the
+`stack-encrypt` feature, not `targets`; the supporting types are in
+`eql_bindings::encryption::query`, whose
+[executable example](src/encryption/query_example.rs) runs with
+`cargo test -p eql-encryption-tests --test query_example`.
+
 ### Interoperability matrix
 
 The plan asked for the three producer/reader pairings to be proven rather than

packages/eql/crates/eql-bindings/src/encryption.rs

@@ -89,6 +89,8 @@
 //! require::<TextEq>();
 //! ```
 
+pub mod query;
+
 // The by-name dispatch over the EQL v4 types (`targets` feature). A plain
 // comment: a doc comment here would resolve the module's own links from
 // this scope.
@@ -102,7 +104,7 @@ use stack_encrypt::sem::{
     EqualityTerm, MatchConfig, MatchOptions, OpeTerm, Orderable, TermError, Tokenizer,
 };
 use stack_encrypt::target::transcode::{Reader, Transcode, Visitor};
-use stack_encrypt::target::{self, AeadContext, CallerContext};
+use stack_encrypt::target::{self, AeadContext, CallerContext, IndexSpec};
 use stack_encrypt::{
     CipherText, ContextPiece, Decrypt, DecryptField, DecryptInto, Decryptable, Decryption,
     Describe, Description, Encrypt, EncryptFrom, Encryption, Error, IntoContext, MaybeEmpty,
@@ -134,6 +136,20 @@ impl Identifier {
             c: column.into(),
         })
     }
+
+    /// The column a two-segment label names, table then column, or `None`
+    /// for a label of any other length. One reading of a label for the
+    /// by-name dispatch (`targets::Identifier::from_label`) and the typed
+    /// query chain ([`query`]).
+    pub(crate) fn column_of(label: &stack_encrypt::Label) -> Option<NonEmpty<Self>> {
+        let mut segments = label.segments();
+        let (Some(table), Some(column), None) = (segments.next(), segments.next(), segments.next())
+        else {
+            return None;
+        };
+        // A label segment is plain, so never empty.
+        Self::for_column(table, column).ok()
+    }
 }
 
 impl MaybeEmpty for Identifier {
@@ -242,9 +258,15 @@ pub struct EqualityVisitor;
 impl Visitor for EqualityVisitor {
     type Value = Hmac256;
     fn equality(self, term: EqualityTerm) -> Result<Self::Value, Error> {
-        Ok(Hmac256(hex::encode(term.as_bytes())))
+        Ok(hmac256(&term))
     }
 }
+
+/// EQL's `hm` for an equality term: the HMAC, hex encoded. One encoding for
+/// stored values and query twins alike.
+fn hmac256(term: &EqualityTerm) -> Hmac256 {
+    Hmac256(hex::encode(term.as_bytes()))
+}
 impl Transcode for Hmac256 {
     type Visitor = EqualityVisitor;
     fn visitor() -> Self::Visitor {
@@ -259,6 +281,11 @@ impl<S: Encrypt + Clone> EncryptFrom<S> for Hmac256 {
     {
         target::equality().transcode()
     }
+    /// The equality index: a plan for a type holding an `hm` answers
+    /// `op::Eq` through it.
+    fn indexes() -> Vec<IndexSpec> {
+        vec![IndexSpec::Equality]
+    }
 }
 impl Decryptable for Hmac256 {
     const DECRYPTABLE: bool = false;
@@ -321,22 +348,27 @@ impl Visitor for MatchVisitor {
         if terms.positions().is_empty() {
             return Err(Error::Term(TermError::EmptyTermText));
         }
-        terms
-            .positions()
-            .iter()
-            .map(|&position| {
-                i16::try_from(position).map_err(|_| {
-                    Error::Other(
-                        "a match position does not fit a smallint; EQL's bloom filter stores \
-                         filters of at most 32768 bits"
-                            .into(),
-                    )
-                })
-            })
-            .collect::<Result<_, _>>()
-            .map(BloomFilter)
+        bloom_filter(terms.positions())
     }
 }
+
+/// EQL's `bf` for a filter's set positions, each as a `smallint`. One
+/// encoding for stored values and query twins alike.
+fn bloom_filter(positions: &[u16]) -> Result<BloomFilter, Error> {
+    positions
+        .iter()
+        .map(|&position| {
+            i16::try_from(position).map_err(|_| {
+                Error::Other(
+                    "a match position does not fit a smallint; EQL's bloom filter stores \
+                     filters of at most 32768 bits"
+                        .into(),
+                )
+            })
+        })
+        .collect::<Result<_, _>>()
+        .map(BloomFilter)
+}
 impl Transcode for BloomFilter {
     type Visitor = MatchVisitor;
     fn visitor() -> Self::Visitor {
@@ -357,6 +389,11 @@ impl<S: AsRef<str>> EncryptFrom<S> for BloomFilter {
         let terms: Encryption<'s, S, MatchTerm<EqlMatch>, K, CallerContext> = target::matching();
         terms.transcode()
     }
+    /// The match index under [`EqlMatch`]: a plan for a type holding a
+    /// `bf` answers `op::Matches` through it.
+    fn indexes() -> Vec<IndexSpec> {
+        vec![IndexSpec::Match(EqlMatch::options())]
+    }
 }
 impl Decryptable for BloomFilter {
     const DECRYPTABLE: bool = false;