diff --git a/.changeset/eql-v4-match-short-words.md b/.changeset/eql-v4-match-short-words.md new file mode 100644 index 000000000..858f489de --- /dev/null +++ b/.changeset/eql-v4-match-short-words.md @@ -0,0 +1,11 @@ +--- +'@cipherstash/eql': minor +--- + +**EQL v4's match types index each two-letter word whole, so a two-letter surname such as "Ng" is searchable** (#1309). `eql-bindings`' `EqlMatch`, the match configuration of `TextMatch`, `TextSearch` and `TextSearchOre`, turns on Stack Encrypt's `MatchOptions::short_words`: beside the 3-grams of the whole text, every word of exactly two characters (words split on space, comma, semicolon, colon and exclamation mark) is one more token, the word itself, hashed under a domain of its own and folded into the same 2048-bit filter by the same 6 positions. "Ng" now stores into `TextMatch` (it was refused, cutting no 3-gram), a `TextSearch` holding "Ng" stores a filter rather than the empty one, and `@@` with the query "ng" finds the rows holding the word "Ng" ("Ng", "Mike Ng", "NG, Anna"), not "Nguyen". A query's two-letter words must be whole words of the row, so "Mike Ng" does not find "Mike Nguyen". Text that still cuts no token (one letter, or none) is refused by `TextMatch` and stored with the empty filter by the search types, as before. + +**The v4 match format now includes short-word tokens.** A filter of text holding a two-letter word gains that word's bits; text with none derives the filter it did. EQL v4 is unreleased, so this is the format it ships with: a v4 value written by an earlier build that holds a two-letter word lacks the word's bits, and a query naming that word does not find it until the value is re-encrypted. + +**What it reveals.** One more token per two-letter word, under the same PRF, in the same filter, by the same containment rule, so no new class of leakage. The token is specific to the word: someone who clusters tokens across rows can tell the rows holding the word "Ng" from those holding "Nguyen", where before only the boundary trigram `" ng"` they share was visible. No bigram of a longer word is indexed. + +No SQL function changes (one doc comment in the bundle is reworded), and the TypeScript types do not change. diff --git a/languages/golang/CHANGELOG.md b/languages/golang/CHANGELOG.md index c1b577e63..f7763a213 100644 --- a/languages/golang/CHANGELOG.md +++ b/languages/golang/CHANGELOG.md @@ -12,6 +12,20 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`. ### Added +- **A two-letter word is searchable in `TextMatch`, `TextSearch` and + `TextSearchOre` (#1309).** The EQL v4 match types index each two-letter + word whole, beside the 3-grams of the text, so a two-letter surname such + as "Ng" stores into a `TextMatch` field (it was refused with `ErrTerm`), + and `Matches` with "ng" is a query (it was refused) that finds the rows + holding the word "Ng", not "Nguyen". A query's two-letter words must be + whole words of the row: "Mike Ng" does not find "Mike Nguyen". Text that + cuts no token, an empty string or one letter, is still refused by + `TextMatch` and stored with the empty filter by the search types. The + word token is one more token per two-letter word, in the same filter, so + someone who clusters tokens across rows can tell "Ng" from "Nguyen". + A native `index=match` field takes no options yet (#1112), so this + reaches Go through the EQL types only. + - **Encrypted JSON documents of Go types** (#1351). A struct, map or slice field tagged `encrypt_into=JsonSearch` is a document: the generated code marshals it with `encoding/json` when it seals it, and reads the opened diff --git a/languages/golang/cmd/stashgen/README.md b/languages/golang/cmd/stashgen/README.md index 60e6f0aeb..bd8f6775f 100644 --- a/languages/golang/cmd/stashgen/README.md +++ b/languages/golang/cmd/stashgen/README.md @@ -297,7 +297,8 @@ The ordered date, timestamp and numeric types are refused: the engine derives no An ordered type has no `Between` method yet, because a `BETWEEN` operand needs EQL v4's range query domains. Any built-in Go integer field takes an integer type (ADR-0002) and seals at the type's width: an `int16` field always fits an `Integer`, and an `int64` field is range checked value by value, `Encrypt` refusing a value that does not fit with `encrypt.ErrEncoding`. `encrypt.Int128` and `encrypt.Uint128` are not converted. A float's width must match: a `float64` field is not a `Real`. -A `TextMatch` field refuses text shorter than three characters, such as an empty string or a two-letter code, because it has no 3-grams to index; its query refuses such text too. +A `TextMatch` field indexes the 3-grams of its text and each two-letter word whole, so a two-letter surname such as "Ng" is stored and found by the query "ng"; a query's two-letter words must be whole words of the row. +A `TextMatch` field refuses text that cuts no token, such as an empty string or one letter; its query refuses such text too. A `TextSearch` field stores such text with an empty match filter, so its `Eq`, `Lt`, `Lte`, `Gt` and `Gte` queries find it and no `Matches` query does; those queries take such text too, and its `Matches` query refuses it. Put the column on the left of `@@` (`WHERE name @@ $1::eql_v4.query_text_match`): the other operand order finds nothing until #1171 is fixed. `encrypt_into` with any other type is refused with the type's name and the reason it is not produced yet. diff --git a/languages/golang/encrypt/README.md b/languages/golang/encrypt/README.md index 15355cc76..61572d8f6 100644 --- a/languages/golang/encrypt/README.md +++ b/languages/golang/encrypt/README.md @@ -490,7 +490,7 @@ The EQL types the engine produces, and the Go field types each takes: | EQL type | Go field | Queries | |---|---|---| | `TextEq` | `string` | `=`, `<>` | -| `TextMatch` | `string` | `@@` (substring match, 3-grams, case-insensitive) | +| `TextMatch` | `string` | `@@` (3-grams and whole two-letter words, case-insensitive; see below) | | `TextOrd`, `TextOrdOpe`, `TextOrdOre` | `string` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` | | `TextSearch`, `TextSearchOre` | `string` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY`, `@@` | | `Text` | `string` | none: stored and read only | @@ -527,13 +527,19 @@ with `ErrEncoding`, naming the field's kind and not the value. `encrypt.Int128` and `encrypt.Uint128` are not converted, and floats take exactly their own width. -A `TextMatch` value must be at least three characters long: shorter text has -no 3-grams to index and is refused. A `TextSearch` value of such text, an -empty string included, is stored with an empty match filter: `Eq`, `Lt`, -`Lte`, `Gt` and `Gte` find it through its other terms, and no `Matches` -query does. A `TextSearch` query carries only the term its operator reads, -so `Eq` and `Lt` on two-letter text work and `Matches` on it is refused; EQL -v4 raises if a query lacks the term its operator compares. The number +A `TextMatch` value is indexed by its 3-grams and by each two-letter word +whole, so a two-letter surname such as "Ng" is stored, and `Matches` with +"ng" finds the rows holding the word "Ng" ("Ng", "Mike Ng"), not "Nguyen". +A query's two-letter words must be whole words of the row, so "Mike Ng" does +not find "Mike Nguyen". The word token reveals one more token per two-letter +word, in the same filter: someone who clusters tokens across rows can tell +"Ng" from "Nguyen", where before only the trigram they share was visible. +Text that cuts no token, such as an empty string or one letter, is refused. A +`TextSearch` value of such text is stored with an empty match filter: `Eq`, +`Lt`, `Lte`, `Gt` and `Gte` find it through its other terms, and no +`Matches` query does. A `TextSearch` query carries only the term its +operator reads, so `Eq` and `Lt` on one-letter text work and `Matches` on it +is refused; EQL v4 raises if a query lacks the term its operator compares. The number equality types compare as PostgreSQL compares the plain types: an integer equals itself whatever its width, and for floats `-0.0 = 0.0` and `NaN = NaN`; the stored value still diff --git a/languages/golang/encrypt/eql_v4_test.go b/languages/golang/encrypt/eql_v4_test.go index c6c457a23..00a13c5ae 100644 --- a/languages/golang/encrypt/eql_v4_test.go +++ b/languages/golang/encrypt/eql_v4_test.go @@ -279,13 +279,24 @@ func TestV4QueriesMatchTheRustFixture(t *testing.T) { t.Errorf("TargetLimit with a histogram = %d (bounded %v), want at least 10", rows, bounded) } - // A needle too short for a 3-gram has no filter bits, and an empty - // filter would be a subset of every row's: the engine refuses it. - for _, short := range []string{"", "al"} { + // A needle that cuts no token (one letter, or none) has no filter bits, + // and an empty filter would be a subset of every row's: the engine + // refuses it. + for _, short := range []string{"", "a"} { if q, err := testusers.ProfileFields.Name.Matches(ctx, cipher, short); err == nil { t.Errorf("Name.Matches(%q) = %s, want a refusal", short, q) } } + // A two-letter word is a token of its own in EQL v4's match types, so a + // two-letter surname is a query, and Verify keeps the rows holding it as + // a word. + ng, err := testusers.ProfileFields.Name.Matches(ctx, cipher, "Ng") + if err != nil { + t.Fatalf("Name.Matches(%q): %v", "Ng", err) + } + if ok, err := ng.Verify(ctx, []string{"Ng", "Mike Ng", "Nguyen"}); err != nil || !ok[0] || !ok[1] || ok[2] { + t.Errorf("Verify(Ng, Mike Ng, Nguyen) = %v, %v, want true, true, false", ok, err) + } } // rowsFixture is v4_rows.json: the profiles above as the generated code seals diff --git a/languages/golang/encrypt/eql_v4_text_search_test.go b/languages/golang/encrypt/eql_v4_text_search_test.go index 8f23f64d8..8bdac8f96 100644 --- a/languages/golang/encrypt/eql_v4_text_search_test.go +++ b/languages/golang/encrypt/eql_v4_text_search_test.go @@ -14,9 +14,10 @@ import ( // EQL v4's TextSearch, sealed and queried through generated code over the // deterministic guest build with the EQL types. A stored value carries all -// three terms, short text with the empty filter; a query carries only the -// one its operator reads (ADR-0002, amended 2026-10-19), so Eq and Lt take a -// two-letter name and Matches refuses one. +// three terms, text that cuts no match token with the empty filter; a query +// carries only the one its operator reads (ADR-0002, amended 2026-10-19), so +// Eq and Lt take a one-letter name and Matches refuses one. A two-letter +// word is a match token of its own, so a two-letter name is matched too. // packages/eql/tests/encryption/tests/v4_text_search.rs holds the same rules // against PostgreSQL. @@ -82,14 +83,14 @@ func TestV4TextSearchStoresEveryTermAndQueriesCarryOne(t *testing.T) { if string(terms[tc.term]) != string(stored[tc.term]) { t.Errorf("%s query's %s is not the stored one", tc.method, tc.term) } - // Text too short for a 3-gram still has an equality and an order + // Text that cuts no match token still has an equality and an order // term. - short, err := tc.query("Al") + short, err := tc.query("A") if err != nil { - t.Fatalf("%s on two-letter text: %v", tc.method, err) + t.Fatalf("%s on one-letter text: %v", tc.method, err) } if keys, _ := keysAndTerms(t, short); keys != strings.Join(sortedWith("i", "v", tc.term), ",") { - t.Errorf("%s query keys on two-letter text %s", tc.method, keys) + t.Errorf("%s query keys on one-letter text %s", tc.method, keys) } } @@ -108,19 +109,36 @@ func TestV4TextSearchStoresEveryTermAndQueriesCarryOne(t *testing.T) { t.Errorf("Verify = %v, %v", ok, err) } - // A match query on text too short for a 3-gram is refused, never sent - // as an empty filter. - if q, err := f.Matches(ctx, cipher, "Al"); !errors.Is(err, encrypt.ErrTerm) { - t.Errorf("Matches on two-letter text = %v, %v, want ErrTerm", q, err) + // A match query on text that cuts no token is refused, never sent as an + // empty filter. + if q, err := f.Matches(ctx, cipher, "A"); !errors.Is(err, encrypt.ErrTerm) { + t.Errorf("Matches on one-letter text = %v, %v, want ErrTerm", q, err) + } + // A two-letter word is one token: "Ng" stores a filter, and the query of + // "ng" carries the same bits. + ng, err := testusers.EncryptEntry(ctx, cipher, []testusers.Entry{{ID: 4, Name: "Ng"}}) + if err != nil { + t.Fatalf("storing a two-letter name in a TextSearch: %v", err) + } + _, ngTerms := keysAndTerms(t, ng[0].Name) + if string(ngTerms["bf"]) == "[]" { + t.Error(`"Ng" stored the empty filter, want its word token`) + } + ngQuery, err := f.Matches(ctx, cipher, "ng") + if err != nil { + t.Fatalf("Matches(%q): %v", "ng", err) + } + if _, q := keysAndTerms(t, ngQuery.Query()); string(q["bf"]) != string(ngTerms["bf"]) { + t.Errorf(`the query "ng"'s bf %s is not the stored "Ng"'s %s`, q["bf"], ngTerms["bf"]) } // Storing such text, an empty string included, keeps every term, the // match term as the empty filter (where TextMatch refuses the text): Eq // and Lt find the row through hm and op, and no Matches query does. - short, err := testusers.EncryptEntry(ctx, cipher, []testusers.Entry{{ID: 2, Name: "Al"}, {ID: 3, Name: ""}}) + short, err := testusers.EncryptEntry(ctx, cipher, []testusers.Entry{{ID: 2, Name: "A"}, {ID: 3, Name: ""}}) if err != nil { t.Fatalf("storing short text in a TextSearch: %v", err) } - for i, name := range []string{"Al", ""} { + for i, name := range []string{"A", ""} { keys, terms := keysAndTerms(t, short[i].Name) if keys != "bf,c,hm,i,op,v" { t.Errorf("stored keys for %q %s, want bf,c,hm,i,op,v", name, keys) @@ -147,7 +165,7 @@ func TestV4TextSearchStoresEveryTermAndQueriesCarryOne(t *testing.T) { if err != nil { t.Fatal(err) } - if opened[0].Name != "Al" || opened[1].Name != "" { + if opened[0].Name != "A" || opened[1].Name != "" { t.Errorf("opened %q and %q", opened[0].Name, opened[1].Name) } } diff --git a/languages/golang/encrypt/internal/testusers/directory.go b/languages/golang/encrypt/internal/testusers/directory.go index 7850d7197..cbf61ae1c 100644 --- a/languages/golang/encrypt/internal/testusers/directory.go +++ b/languages/golang/encrypt/internal/testusers/directory.go @@ -4,7 +4,7 @@ package testusers // Entry stores a name as EQL v4's TextSearch: one column that answers `=`, // the orderings and `@@`. Its query carries only the term its operator -// reads, so `=` and `<` work on a two-letter name while `@@` on one is +// reads, so `=` and `<` work on a one-letter name while `@@` on one is // refused (ADR-0002, amended 2026-10-19). type Entry struct { _ struct{} `stash:"context=directory"` diff --git a/packages/eql/crates/eql-bindings/README.md b/packages/eql/crates/eql-bindings/README.md index abc4526d5..8a35f0c1e 100644 --- a/packages/eql/crates/eql-bindings/README.md +++ b/packages/eql/crates/eql-bindings/README.md @@ -162,17 +162,22 @@ order term and is refused, stored or queried. A family's `Ord` and `OrdOpe` types store the same term under one column. The query value carries one bound: `BETWEEN` needs EQL v4's range query domains and is not produced yet. `TextMatch` builds its filter under a configuration pinned in this crate -(`EqlMatch`: 3-grams of the lower-cased text, `k = 6`, a 2048-bit filter) for -stored values and queries alike. The configuration is part of the stored +(`EqlMatch`: 3-grams of the lower-cased text, plus each two-letter word whole +as one more token, `k = 6`, a 2048-bit filter) for stored values and queries +alike. The word token makes a two-letter surname such as "Ng" storable and +searchable: `@@` with "ng" finds "Ng" and "Mike Ng", not "Nguyen". A query's +two-letter words must be whole words of the row (words split on space, +comma, semicolon, colon and exclamation mark), so "Mike Ng" does not find +"Mike Nguyen", and a query is not a plain substring test. The configuration is part of the stored format, so changing it means a new domain or payload version. A filter can match a row that does not contain the query, so re-check after decrypting where that matters. Write the stored column on the left, `col @@ query`: the -other order finds nothing until #1171 is fixed. Text too short to make a -3-gram (empty strings, two-letter codes) cannot be stored in a `TextMatch` -column yet, and a `TextMatchQuery` for it is refused too: one filter type -builds both, and an empty filter as a query would match every row. +other order finds nothing until #1171 is fixed. Text that cuts no token +(empty strings, one letter) cannot be stored in a `TextMatch` column, and a +`TextMatchQuery` for it is refused too: one filter type builds both, and an +empty filter as a query would match every row. `TextSearch` stores all three terms (`TextEq`'s `hm`, `TextOrd`'s `op` and -`TextMatch`'s `bf`). Text too short for a 3-gram is stored with the empty +`TextMatch`'s `bf`). Text that cuts no token is stored with the empty filter (`"bf": []`, the `SearchBloomFilter` field type): `=` and `<` find it through `hm` and `op`, and no `@@` query does, because a query's filter is never empty. (A `@@` between two stored values, which no producer builds, is @@ -180,7 +185,7 @@ not a query: two stored empty filters match each other there, as in v3.) Its que through `TextSearchEqualityQuery` (`=`, `<>`), `TextSearchOrderQuery` (`<`, `<=`, `>`, `>=`) or `TextSearchMatchQuery` (`@@`) and convert into `TextSearchQuery`, and the by-name dispatch picks the one for the index that -answers the operator. So `=` and `<` work on short text, a match query on it +answers the operator. So `=` and `<` work on such text, a match query on it is refused, and EQL v4 raises on an operator whose term the query lacks (or holds as `null`, or as an empty filter). The ORE types (`*OrdOre`, `TextSearchOre`) store a block ORE term (`ob`): diff --git a/packages/eql/crates/eql-bindings/src/encryption.rs b/packages/eql/crates/eql-bindings/src/encryption.rs index 83a72338f..968043866 100644 --- a/packages/eql/crates/eql-bindings/src/encryption.rs +++ b/packages/eql/crates/eql-bindings/src/encryption.rs @@ -295,8 +295,23 @@ impl DecryptField for Hmac256 { } } -/// The match configuration of EQL v4's `TextMatch`: 3-grams of the -/// lower-cased text, `k = 6` positions per token in a 2048-bit filter. +/// The match configuration of EQL v4's match types (`TextMatch`, +/// `TextSearch`, `TextSearchOre`): 3-grams of the lower-cased text, plus each +/// two-letter word whole as one more token (Stack Encrypt's +/// `MatchOptions::short_words`), `k = 6` positions per token in a 2048-bit +/// filter. +/// +/// The short-word tokens are what make a two-letter surname searchable: "Ng" +/// cuts no 3-gram, so without them it would be no `TextMatch` value at all, +/// and the query "ng" would be refused. With them "Ng" stores one token, and +/// `@@` with the query "ng" finds the rows holding the word "Ng" ("Ng", +/// "Mike Ng", "Ng, Mike"), not "Nguyen". A query's two-letter words must be +/// whole words of the row, so "Mike Ng" does not find "Mike Nguyen". What it +/// reveals: one more token per two-letter word, under the same PRF (in a +/// domain of its own), in the same filter, by the same containment rule, so +/// no new class of leakage; the token is specific to the word, so someone who +/// clusters tokens across rows can tell "Ng" from "Nguyen", where before only +/// the boundary trigram `" ng"` they share was visible. /// /// Pinned here rather than taken from Stack Encrypt's default, because it is /// part of the stored format: a stored filter and a query filter agree only @@ -316,6 +331,7 @@ impl MatchConfig for EqlMatch { MatchOptions { tokenizer: Tokenizer::Ngram { length: 3 }, downcase: true, + short_words: true, k: 6, m: 2048, } @@ -328,8 +344,9 @@ impl MatchConfig for EqlMatch { /// cipherstash-client's parameters is not a goal, because cipherstash-client /// never writes an `eql_v4` column. /// -/// Text that cuts no token (empty, separator-only, or shorter than a 3-gram) -/// is refused with [`TermError::EmptyTermText`], stored or queried. Stack +/// Text that cuts no token (empty, separator-only, or shorter than a 3-gram +/// with no two-letter word, such as one letter) is refused with +/// [`TermError::EmptyTermText`], stored or queried. Stack /// Encrypt stores such a value as the empty filter, but this type is the /// filter of every match query, and a query's filter must never be empty; /// it is also `TextMatch`'s stored filter, and a `TextMatch` value with an @@ -462,6 +479,10 @@ mod tests { let options = EqlMatch::options(); assert!(matches!(options.tokenizer, Tokenizer::Ngram { length: 3 })); assert!(options.downcase); + assert!( + options.short_words, + "the v4 match types index each two-letter word whole" + ); assert_eq!(options.k, 6); assert_eq!(options.m, 2048); assert!( diff --git a/packages/eql/crates/eql-bindings/src/v4/mod.rs b/packages/eql/crates/eql-bindings/src/v4/mod.rs index 2b8a9dae5..0c90f88ba 100644 --- a/packages/eql/crates/eql-bindings/src/v4/mod.rs +++ b/packages/eql/crates/eql-bindings/src/v4/mod.rs @@ -98,8 +98,9 @@ pub mod terms { /// The `bf` term of a stored search value (`TextSearch`, /// `TextSearchOre`): a [`BloomFilter`] that may be empty. /// - /// Text too short to cut a 3-gram (empty, separator-only, or shorter than - /// three characters) has no match tokens. A search column stores it with + /// Text that cuts no match token (empty, separator-only, or shorter than + /// three characters with no two-letter word, such as one letter) has no + /// match tokens. A search column stores it with /// the empty filter, Stack Encrypt's answer for a stored value, so `=` /// and `<` still find it through its `hm` and `op`, and no `@@` query /// does: EQL's `matches` finds a stored filter only when it contains the diff --git a/packages/eql/src/v3/scalars/functions.sql b/packages/eql/src/v3/scalars/functions.sql index affe0d755..70476face 100644 --- a/packages/eql/src/v3/scalars/functions.sql +++ b/packages/eql/src/v3/scalars/functions.sql @@ -70,9 +70,10 @@ $$ LANGUAGE plpgsql; --! as no row. --! --! A term whose value is JSON `null` is missing, and so is an empty array: a ---! `bf` with no positions is the filter of text too short for a 3-gram, which ---! a stored filter of the same text would contain, so a query carrying it ---! would match exactly the stored values no match query should find. +--! `bf` with no positions is the filter of text that cuts no match token (one +--! letter, or none), which a stored filter of the same text would contain, so +--! a query carrying it would match exactly the stored values no match query +--! should find. --! --! The raise happens where the operand is evaluated, not where the statement --! is planned. A literal operand is folded at plan time and raises even over diff --git a/packages/eql/tests/encryption/tests/typed_queries.rs b/packages/eql/tests/encryption/tests/typed_queries.rs index fbce6687f..b4e5a10d1 100644 --- a/packages/eql/tests/encryption/tests/typed_queries.rs +++ b/packages/eql/tests/encryption/tests/typed_queries.rs @@ -469,7 +469,7 @@ async fn a_typed_query_refuses_what_the_data_path_refuses() { .build() .unwrap(); let short = cipher - .query(op::Matches, "al") + .query(op::Matches, "a") .keyset("tenant") .on(&plan) .await; diff --git a/packages/eql/tests/encryption/tests/v4_text_search.rs b/packages/eql/tests/encryption/tests/v4_text_search.rs index bc90bb75e..586324f59 100644 --- a/packages/eql/tests/encryption/tests/v4_text_search.rs +++ b/packages/eql/tests/encryption/tests/v4_text_search.rs @@ -4,8 +4,9 @@ //! //! A stored value carries all three terms, each the one the single-index //! type stores (`TextEq`'s `hm`, `TextOrd`'s `op`, `TextMatch`'s `bf`). -//! Text too short to cut a 3-gram stores the empty filter, where `TextMatch` -//! refuses it, so `=` and `<` find it and no `@@` query does. A query is built +//! Text that cuts no match token (one letter, or none: a two-letter word is +//! a token of its own) stores the empty filter, where `TextMatch` refuses +//! it, so `=` and `<` find it and no `@@` query does. A query is built //! through the projection of the index that answers its operator, so `=` and //! `<` on such text work and a match query on it is refused. Against an EQL //! v4 database: a query with any one term is a `query_text_search`, one with @@ -106,7 +107,8 @@ async fn a_stored_value_carries_the_three_single_index_terms() { assert_eq!(opened.risky_ref(), frame("Alice").risky_ref()); } -/// Text too short to cut a 3-gram has no match tokens. `TextSearch` stores +/// Text that cuts no match token (one letter, or none) has no filter. +/// `TextSearch` stores /// it with the empty filter and the same `hm` and `op` as `TextEq` and /// `TextOrd`, so `=` and `<` find it and no match query does. `TextMatch`, /// which has no other term to find it by, still refuses it. @@ -114,7 +116,7 @@ async fn a_stored_value_carries_the_three_single_index_terms() { async fn short_text_stores_the_empty_filter_where_text_match_refuses_it() { let (cipher, _) = common::cipher().await; let keyset = cipher.default_keyset(); - for text in ["Al", "", " "] { + for text in ["A", "", " "] { let search = stored(&keyset, "TextSearch", text).await; assert_eq!(keys(&search), ["bf", "c", "hm", "i", "op", "v"], "{text:?}"); assert_eq!(search["bf"], serde_json::json!([]), "{text:?}"); @@ -186,24 +188,24 @@ async fn a_query_carries_only_the_term_its_operator_reads() { ); } -/// `=` and `<` on text shorter than one 3-gram: the query carries `hm` or +/// `=` and `<` on text that cuts no match token: the query carries `hm` or /// `op`, never an empty filter. A match query on it is refused, as a /// `TextMatchQuery` is, rather than sent as a filter that matches every row. #[tokio::test] async fn short_text_is_queried_by_equality_and_order_and_refused_for_match() { let (cipher, _) = common::cipher().await; let keyset = cipher.default_keyset(); - let eq = query(&keyset, "eq", "Al").await.unwrap(); + let eq = query(&keyset, "eq", "A").await.unwrap(); assert_eq!(keys(&eq), ["hm", "i", "v"]); assert!(eq["hm"].as_str().is_some_and(|hm| !hm.is_empty())); - let ord = query(&keyset, "ope", "Al").await.unwrap(); + let ord = query(&keyset, "ope", "A").await.unwrap(); assert_eq!(keys(&ord), ["i", "op", "v"]); assert!(ord["op"].as_str().is_some_and(|op| !op.is_empty())); - // The order term of "Al" sorts before "Alice"'s: a prefix first. + // The order term of "A" sorts before "Alice"'s: a prefix first. let alice = stored(&keyset, "TextSearch", "Alice").await; assert!(ord["op"].as_str().unwrap() < alice["op"].as_str().unwrap()); - let refused = query(&keyset, "match", "Al").await.unwrap_err(); + let refused = query(&keyset, "match", "A").await.unwrap_err(); assert!( matches!( refused, @@ -219,7 +221,7 @@ async fn short_text_is_queried_by_equality_and_order_and_refused_for_match() { async fn the_typed_projections_are_what_the_dispatch_writes() { let (cipher, _) = common::cipher().await; let keyset = cipher.default_keyset(); - let text = "Al".to_owned(); + let text = "A".to_owned(); let eq: TextSearchEqualityQuery = keyset.encrypt_as(&text, column()).await.unwrap(); let ord: TextSearchOrderQuery = keyset.encrypt_as(&text, column()).await.unwrap(); for (typed, key) in [ @@ -228,7 +230,7 @@ async fn the_typed_projections_are_what_the_dispatch_writes() { ] { assert_eq!( serde_json::to_value(&typed).unwrap(), - query(&keyset, key, "Al").await.unwrap(), + query(&keyset, key, "A").await.unwrap(), "{key}" ); } @@ -266,7 +268,7 @@ async fn the_typed_projections_are_what_the_dispatch_writes() { serde_json::json!([]) ); let opened: String = cipher.decrypt_as(short, column().into()).await.unwrap(); - assert_eq!(opened, "Al"); + assert_eq!(opened, "A"); } // ---- PostgreSQL ------------------------------------------------------------- @@ -311,7 +313,7 @@ fn sqlstate(error: &postgres::Error) -> &str { /// Needs an EQL v4 installation. Rows written through the by-name dispatch, /// queried with operands that carry one term each: `=`, `<`, `>` and `@@` -/// find the rows the plaintexts say, two-letter and empty text, stored with +/// find the rows the plaintexts say, one-letter and empty text, stored with /// the empty filter, included on `=` and `<` and found by no `@@`; a /// `query_text_search` holds any one term and refuses none; and each /// operator refuses an operand without its term, naming the term, never the @@ -330,9 +332,9 @@ fn postgres_answers_each_operator_through_its_own_term() { "CREATE TEMP TABLE v4_search (id int PRIMARY KEY, name public.eql_v4_text_search);", ) .unwrap(); - // 4 and 5 are too short for a 3-gram: stored with `"bf": []`, which the + // 4 and 5 cut no match token: stored with `"bf": []`, which the // domain's CHECK accepts. - let names = [(1, "alice"), (2, "Bob"), (3, "carol"), (4, "Al"), (5, "")]; + let names = [(1, "alice"), (2, "Bob"), (3, "carol"), (4, "A"), (5, "")]; for (id, name) in names { let value = runtime.block_on(stored(&keyset, "TextSearch", name)); db.execute( @@ -386,13 +388,13 @@ fn postgres_answers_each_operator_through_its_own_term() { ), [3] ); - // Two-letter text, which has no match term, on `=` and `<`/`>`: stored + // One-letter text, which has no match term, on `=` and `<`/`>`: stored // and queried. assert_eq!( ids( &mut db, &select(&format!("name = {operand}")), - &q("eq", "Al") + &q("eq", "A") ), [4] ); diff --git a/packages/eql/tests/encryption/tests/v4_types.rs b/packages/eql/tests/encryption/tests/v4_types.rs index ca1c86395..0e69f11a4 100644 --- a/packages/eql/tests/encryption/tests/v4_types.rs +++ b/packages/eql/tests/encryption/tests/v4_types.rs @@ -590,17 +590,74 @@ async fn text_match_filters_contain_their_substrings() { let named: text::TextMatch = by_name(&keyset, "TextMatch", frame("Alice@Example.com")).await; assert_eq!(named.bf, stored.bf); - // Text shorter than a 3-gram has no tokens. eql-bindings refuses it for a - // stored value as well as a query: one filter type builds both, and an - // empty filter as a query would match every row. So a short value cannot - // be written to a TextMatch column yet. - let short: Result = keyset.encrypt_as(&"ab".to_owned(), column()).await; - assert!(short.is_err()); - let short_query: Result = - keyset.encrypt_as(&"ab".to_owned(), column()).await; - assert!(short_query.is_err(), "a short query would match every row"); + // Text that cuts no token (one letter, or none) has no filter. + // eql-bindings refuses it for a stored value as well as a query: one + // filter type builds both, and an empty filter as a query would match + // every row. A two-letter word is a token of its own, so it stores. + for short in ["a", "", " "] { + let stored: Result = + keyset.encrypt_as(&short.to_owned(), column()).await; + assert!(stored.is_err(), "{short:?} cuts no token"); + let query: Result = + keyset.encrypt_as(&short.to_owned(), column()).await; + assert!( + query.is_err(), + "{short:?}: a query with no token would match every row" + ); + } +} + +/// Two-letter surnames, which cut no 3-gram: each is one short-word token +/// (`EqlMatch` turns `short_words` on), so it stores into `TextMatch`, and the +/// query of the surname finds the bare value and a full name holding it, +/// whatever its case, but not a longer name it begins. +#[tokio::test] +async fn two_letter_surnames_store_and_are_found() { + let (cipher, _) = common::cipher().await; + let keyset = cipher.default_keyset(); + for &surname in SURNAMES { + let bare: text::TextMatch = keyset + .encrypt_as(&surname.to_owned(), column()) + .await + .unwrap_or_else(|error| panic!("{surname:?} stores: {error}")); + assert!(!bare.bf.0.is_empty(), "{surname:?} stores a token"); + let query: text::TextMatchQuery = keyset + .encrypt_as(&surname.to_lowercase(), column()) + .await + .unwrap_or_else(|error| panic!("{surname:?} is a query: {error}")); + assert_eq!( + query.bf, bare.bf, + "{surname:?}: the query's filter is the stored one's" + ); + let full: text::TextMatch = keyset + .encrypt_as(&format!("Mike {surname}"), column()) + .await + .unwrap(); + assert!( + positions(&query.bf).is_subset(&positions(&full.bf)), + "{surname:?} is found in \"Mike {surname}\"" + ); + let longer: text::TextMatch = keyset + .encrypt_as(&format!("Mike {surname}ward"), column()) + .await + .unwrap(); + assert!( + !positions(&query.bf).is_subset(&positions(&longer.bf)), + "{surname:?} is not found in \"Mike {surname}ward\"" + ); + let named: text::TextMatch = by_name(&keyset, "TextMatch", frame(surname)).await; + assert_eq!( + named.bf, bare.bf, + "{surname:?}: the dispatch derives the same filter" + ); + } } +/// Common two-letter surnames. +const SURNAMES: &[&str] = &[ + "Ng", "Li", "Wu", "Le", "Do", "Ho", "Yu", "Lu", "Ma", "Xu", "Hu", "Su", "Oh", "Ko", "An", "Ye", +]; + /// The filter bits for one query under the test key source. The /// configuration is part of the stored format (`EqlMatch`): if these move, /// every stored filter stops agreeing with new queries, so a change here must @@ -689,6 +746,102 @@ async fn a_value_of_another_kind_is_refused_not_converted() { assert!(opened.is_err(), "a bool is not an integer"); } +/// Needs an EQL v4 installation. Two-letter surnames in a `TextMatch` and a +/// `TextSearch` column: `@@` with the query of the surname finds the rows +/// holding it as a word, bare or in a full name, and not a longer name it +/// begins; a full-name query needs its two-letter word whole. +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_finds_two_letter_surnames() { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let (cipher, _) = runtime.block_on(common::cipher()); + let keyset = cipher.default_keyset(); + let mut db = postgres::Client::connect( + &std::env::var("EQL_TEST_DATABASE_URL").expect("EQL_TEST_DATABASE_URL"), + postgres::NoTls, + ) + .unwrap(); + db.batch_execute( + "CREATE TEMP TABLE v4_surnames ( + id int PRIMARY KEY, + name public.eql_v4_text_match, + search public.eql_v4_text_search + );", + ) + .unwrap(); + let names = [ + (1, "Ng"), + (2, "Mike Ng"), + (3, "Nguyen"), + (4, "Mike Nguyen"), + (5, "Li Wei"), + (6, "NG, Anna"), + ]; + for (id, name) in names { + let (matched, search) = runtime.block_on(async { + let matched: text::TextMatch = + keyset.encrypt_as(&name.to_owned(), column()).await.unwrap(); + let search: text::TextSearch = + keyset.encrypt_as(&name.to_owned(), column()).await.unwrap(); + (matched, search) + }); + db.execute( + "INSERT INTO v4_surnames VALUES ($1, + $2::text::jsonb::public.eql_v4_text_match, + $3::text::jsonb::public.eql_v4_text_search)", + &[ + &id, + &serde_json::to_string(&matched).unwrap(), + &serde_json::to_string(&search).unwrap(), + ], + ) + .unwrap(); + } + let mut found = |needle: &str| -> (Vec, Vec) { + let (matched, search) = runtime.block_on(async { + let matched: text::TextMatchQuery = keyset + .encrypt_as(&needle.to_owned(), column()) + .await + .unwrap(); + let search: text::TextSearchMatchQuery = keyset + .encrypt_as(&needle.to_owned(), column()) + .await + .unwrap(); + (matched, text::TextSearchQuery::from(search)) + }); + let ids = |rows: Vec| rows.iter().map(|r| r.get(0)).collect(); + ( + ids(db + .query( + "SELECT id FROM v4_surnames WHERE name @@ $1::text::jsonb::eql_v4.query_text_match ORDER BY id", + &[&serde_json::to_string(&matched).unwrap()], + ) + .unwrap()), + ids(db + .query( + "SELECT id FROM v4_surnames WHERE search @@ $1::text::jsonb::eql_v4.query_text_search ORDER BY id", + &[&serde_json::to_string(&search).unwrap()], + ) + .unwrap()), + ) + }; + for (needle, want) in [ + ("ng", vec![1, 2, 6]), + ("Ng", vec![1, 2, 6]), + ("Mike Ng", vec![2]), + ("Nguyen", vec![3, 4]), + ("li", vec![5]), + ("Wei", vec![5]), + ] { + let (matched, search) = found(needle); + assert_eq!(matched, want, "TextMatch @@ {needle:?}"); + assert_eq!(search, want, "TextSearch @@ {needle:?}"); + } +} + /// Needs an EQL v4 installation; uses fresh real ciphertext with a fake key /// source. #[test] diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index f84f411d5..6155397d3 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -65,6 +65,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **A match index can search words too short for its n-grams: + `MatchOptions::short_words`** (#1309). Off by default. With it on, every + word of the normalized, case-folded text that is at least 2 characters long + and shorter than the n-gram length (two-letter words under the default + 3-grams) is one more token, the word itself, beside the n-grams of the whole + text. So the surname "Ng" stores one token rather than the empty filter, + and the query "Ng" is accepted and finds rows holding the word "Ng" or + "Mike Ng", but not "Nguyen". A query's short words must be whole words of + the row: "Mike Ng" does not find "Mike Nguyen" with the option on. A word + is a run of characters between the separators `Tokenizer::Standard` splits + on (space, comma, semicolon, colon, exclamation mark). The word token is + PRF'd under its own domain (`stack-encrypt/sem/match/word/v1`), so it never + sets the bits of an n-gram of the same characters. The option is part of + the index's identity, as `k` is (`PlanError::IndexOptions`), is written as + `"short_words": true` in a data plan's `{"match": {...}}` (and only when on, + so other plans serialize as before), and is refused beside the standard + tokenizer or an n-gram length under 3 (`TermError::InvalidOptions`). What + it reveals: one more token per short word, under the same PRF, in the same + filter, by the same containment rule, so no new class of leakage; the token + is specific to the word, so clustering tokens across rows can tell "Ng" + from "Nguyen", where before only their shared boundary trigram `" ng"` was + visible. `MatchOptions` gains the public field, so a struct literal naming + every field names it too. The `sem` module docs carry the match index's + leakage in full. + - **A document field's queries and reads through a data plan** (#1350, #1352). A field naming `JsonSearch` answers `contains`, `contained_by`, `eq_at`, `selector`, `subtree`, `children` and diff --git a/packages/stack-encrypt/src/dynamic/kind.rs b/packages/stack-encrypt/src/dynamic/kind.rs index da917dc93..633a56538 100644 --- a/packages/stack-encrypt/src/dynamic/kind.rs +++ b/packages/stack-encrypt/src/dynamic/kind.rs @@ -744,6 +744,7 @@ mod tests { let wide = IndexSpec::Match(MatchOptions { tokenizer: Tokenizer::Standard, downcase: false, + short_words: false, k: 6, m: 1024, }); diff --git a/packages/stack-encrypt/src/dynamic/query.rs b/packages/stack-encrypt/src/dynamic/query.rs index 84aaa0b4c..c08895572 100644 --- a/packages/stack-encrypt/src/dynamic/query.rs +++ b/packages/stack-encrypt/src/dynamic/query.rs @@ -3067,6 +3067,7 @@ mod wire_bytes { IndexSpec::Match(MatchOptions { tokenizer: Tokenizer::Standard, downcase: false, + short_words: false, k: 4, m: 512, }), diff --git a/packages/stack-encrypt/src/dynamic/record.rs b/packages/stack-encrypt/src/dynamic/record.rs index 451bcf03c..0cea0891a 100644 --- a/packages/stack-encrypt/src/dynamic/record.rs +++ b/packages/stack-encrypt/src/dynamic/record.rs @@ -1131,15 +1131,18 @@ pub fn plan(frame: &[u8]) -> Result { /// /// ```text /// { "match": { "tokenizer": "standard" | { "ngram": }, -/// "downcase": , "k": , "m": } } +/// "downcase": , "short_words": , +/// "k": , "m": } } /// ``` /// /// A bare `"match"` is the default options ([`MatchOptions::default`]: -/// 3-grams, downcased, `k = 3`, `m = 256`), so a plan written before options -/// had a wire form means what it always meant. In the object form each -/// option is optional and defaults the same way; an unknown or repeated -/// option, or a set the match scheme refuses (`k` outside `3..=16`, `m` not -/// a power of two in `32..=65536`, a zero n-gram length), is refused. The +/// 3-grams, downcased, no short-word tokens, `k = 3`, `m = 256`), so a plan +/// written before options had a wire form means what it always meant. In +/// the object form each option is optional and defaults the same way; an +/// unknown or repeated option, or a set the match scheme refuses (`k` +/// outside `3..=16`, `m` not a power of two in `32..=65536`, a zero n-gram +/// length, `short_words` beside the standard tokenizer or an n-gram length +/// under 3), is refused. The /// other three indexes have no options and no object form. A field names /// each output key at most once, so it carries at most one match index. /// [`IndexSpec`] serializes to this form and [`IndexSpec::from_frame`] @@ -3497,6 +3500,7 @@ mod tests { let options = crate::sem::MatchOptions { tokenizer: crate::sem::Tokenizer::Standard, downcase: false, + short_words: false, k: 6, m: 1024, }; diff --git a/packages/stack-encrypt/src/dynamic/term.rs b/packages/stack-encrypt/src/dynamic/term.rs index 0735b5b38..756cd567b 100644 --- a/packages/stack-encrypt/src/dynamic/term.rs +++ b/packages/stack-encrypt/src/dynamic/term.rs @@ -69,13 +69,15 @@ use crate::{Encrypt, IntoPrfContext, KeysetCipher, NonEmpty}; /// /// ```text /// { "match": { "tokenizer": "standard" | { "ngram": }, -/// "downcase": , "k": , "m": } } +/// "downcase": , "short_words": , +/// "k": , "m": } } /// ``` /// /// Every option is optional and defaults to [`MatchOptions::default`]'s /// value; an unknown option, an option given twice, or options that fail /// the match scheme's bounds (`k` in `3..=16`, `m` a power of two in -/// `[32, 65536]`, a non-zero n-gram length) are refused. +/// `[32, 65536]`, a non-zero n-gram length, `short_words` only beside an +/// n-gram length of at least 3) are refused. /// /// A JSON index is `"json"`, or an object mapping `"json"` to its options: /// @@ -94,9 +96,10 @@ use crate::{Encrypt, IntoPrfContext, KeysetCipher, NonEmpty}; /// indexes have no options and no object form. /// /// Serialized (it is `Serialize`), an index writes the string whenever the -/// options are the defaults and the object, with every option, otherwise, -/// so a default plan is byte-for-byte what it was before options had a -/// wire form. +/// options are the defaults and the object otherwise, with every option but +/// `short_words`, which it writes only when it is on. So a default plan is +/// byte-for-byte what it was before options had a wire form, and a plan with +/// other options is what it was before `short_words` existed. impl IndexSpec { /// The index a bare key names, under default options, or `None` for a /// string that is not an index key. @@ -202,8 +205,8 @@ fn ordered(ty: Option, value: &Physical<'_>) -> bool { } /// An index in its wire form: its key, or for a match index under options -/// other than the defaults, a one-entry object mapping `"match"` to all four -/// of them (see [`IndexSpec`'s docs](IndexSpec#wire-form)). +/// other than the defaults, a one-entry object mapping `"match"` to them, +/// `short_words` only when it is on (see [`IndexSpec`'s docs](IndexSpec#wire-form)). impl Serialize for IndexSpec { fn serialize(&self, s: S) -> Result { #[derive(Serialize)] @@ -230,6 +233,8 @@ impl Serialize for IndexSpec { struct MatchOut { tokenizer: TokenizerOut, downcase: bool, + #[serde(skip_serializing_if = "std::ops::Not::not")] + short_words: bool, k: u64, m: u64, } @@ -248,6 +253,7 @@ impl Serialize for IndexSpec { Tokenizer::Ngram { length } => TokenizerOut::Ngram(length as u64), }, downcase: options.downcase, + short_words: options.short_words, k: options.k as u64, m: u64::from(options.m), }), @@ -343,6 +349,7 @@ pub(super) struct MatchWire(pub(super) MatchOptions); struct MatchOptionsWire { tokenizer: TokenizerWire, downcase: bool, + short_words: bool, k: usize, m: u32, } @@ -352,6 +359,7 @@ impl Default for MatchOptionsWire { let MatchOptions { tokenizer, downcase, + short_words, k, m, } = MatchOptions::default(); @@ -361,6 +369,7 @@ impl Default for MatchOptionsWire { Tokenizer::Ngram { length } => TokenizerWire::Ngram(length), }, downcase, + short_words, k, m, } @@ -385,6 +394,7 @@ impl TryFrom for MatchWire { TokenizerWire::Ngram(length) => Tokenizer::Ngram { length }, }, downcase: wire.downcase, + short_words: wire.short_words, k: wire.k, m: wire.m, }; @@ -1640,6 +1650,7 @@ mod tests { MatchOptions { tokenizer: crate::sem::Tokenizer::Standard, downcase: false, + short_words: false, k: 6, m: 1024, } @@ -1721,6 +1732,15 @@ mod tests { m: 512, ..MatchOptions::default() }, + MatchOptions { + short_words: true, + ..MatchOptions::default() + }, + MatchOptions { + tokenizer: crate::sem::Tokenizer::Ngram { length: 4 }, + short_words: true, + ..MatchOptions::default() + }, ]; for options in cases { let spec = IndexSpec::Match(options.clone()); @@ -1737,6 +1757,64 @@ mod tests { } } + /// `short_words` is written only when it is on, so a plan under + /// other options is the bytes it was before the option existed, and + /// read as `"short_words": true`, with every other option defaulted. + #[test] + fn short_words_is_written_only_when_on() { + let obj = |entries: Vec<(&str, Tree)>| { + Tree::Object( + entries + .into_iter() + .map(|(k, v)| (k.to_string(), v)) + .collect(), + ) + }; + let keys = |spec: &IndexSpec| match crate::dynamic::test_frame::value_of(spec) { + Tree::Object(mut entries) => match entries.pop() { + Some((_, Tree::Object(options))) => { + options.into_iter().map(|(k, _)| k).collect::>() + } + other => panic!("a match options object, got {other:?}"), + }, + other => panic!("an object, got {other:?}"), + }; + assert_eq!( + keys(&IndexSpec::Match(non_default_match())), + ["tokenizer", "downcase", "k", "m"], + "off, the option is not written" + ); + assert_eq!( + keys(&IndexSpec::Match(MatchOptions { + short_words: true, + ..MatchOptions::default() + })), + ["tokenizer", "downcase", "short_words", "k", "m"], + "on, it is written beside the others" + ); + assert_eq!( + index_of(&obj(vec![( + "match", + obj(vec![("short_words", Tree::Bool(true))]) + )])) + .expect("reads"), + IndexSpec::Match(MatchOptions { + short_words: true, + ..MatchOptions::default() + }), + "the option alone is the defaults with short words" + ); + assert_eq!( + index_of(&obj(vec![( + "match", + obj(vec![("short_words", Tree::Bool(false))]) + )])) + .expect("reads"), + default_match(), + "the option off is the default index" + ); + } + /// The object form spells each option by name and defaults the /// rest; the exact shape here is what a binding writes. #[test] @@ -1834,6 +1912,24 @@ mod tests { opts(vec![("tokenizer", obj(vec![("ngram", Tree::UInt32(0))]))]), ), ("a mistyped downcase", opts(vec![("downcase", s("yes"))])), + ( + "a mistyped short_words", + opts(vec![("short_words", Tree::UInt32(1))]), + ), + ( + "short_words beside the standard tokenizer", + opts(vec![ + ("tokenizer", s("standard")), + ("short_words", Tree::Bool(true)), + ]), + ), + ( + "short_words beside 2-grams", + opts(vec![ + ("tokenizer", obj(vec![("ngram", Tree::UInt32(2))])), + ("short_words", Tree::Bool(true)), + ]), + ), ("a negative k", opts(vec![("k", Tree::Int32(-3))])), ("a float k", opts(vec![("k", Tree::Float64(3.0))])), ("k out of bounds", opts(vec![("k", Tree::UInt32(17))])), @@ -1867,6 +1963,7 @@ mod tests { MatchOptions { tokenizer: crate::sem::Tokenizer::Standard, downcase: false, + short_words: false, k: 6, m: 1024, } @@ -1894,6 +1991,48 @@ mod tests { .expect("default"); assert_ne!(wide, default, "other options derive other terms"); } + + /// The data path derives a short-word index's term as the typed + /// path does, and "Ng", which cuts nothing under the default + /// options, is one token under them. + #[tokio::test] + async fn short_words_drive_the_derivation() { + struct ShortWords; + impl crate::sem::MatchConfig for ShortWords { + fn options() -> MatchOptions { + MatchOptions { + short_words: true, + ..MatchOptions::default() + } + } + } + let options = ::options(); + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + for text in ["Ng", "Mike Ng"] { + let data = term( + &keyset, + &s(text), + &IndexSpec::Match(options.clone()), + ctx("users/x"), + ) + .await + .expect("a short word is a query"); + let typed = keyset + .match_terms::(text, nonempty!("users/x")) + .await + .expect("typed"); + assert_eq!(data, typed.to_bytes(), "{text:?}: the typed config's bytes"); + } + let refused = term(&keyset, &s("Ng"), &default_match(), ctx("users/x")).await; + assert!( + matches!( + refused, + Err(Error::Cipher(crate::Error::Term(TermError::EmptyTermText))) + ), + "without the option, \"Ng\" is no query: {refused:?}" + ); + } } } @@ -1917,6 +2056,7 @@ mod wire_form { MatchOptions { tokenizer: Tokenizer::Standard, downcase: false, + short_words: false, k: 4, m: 512, } diff --git a/packages/stack-encrypt/src/plan/error.rs b/packages/stack-encrypt/src/plan/error.rs index bb5893848..79c2f657b 100644 --- a/packages/stack-encrypt/src/plan/error.rs +++ b/packages/stack-encrypt/src/plan/error.rs @@ -295,8 +295,9 @@ impl crate::ErrorPayload for PlanError { } /// An index as a plan writes it: its key, or for a match index the object of -/// all four options (`{"match": {"tokenizer": "standard", "downcase": true, -/// "k": 3, "m": 256}}`, an n-gram tokenizer as `{"ngram": 3}`), and for a +/// its options (`{"match": {"tokenizer": "standard", "downcase": true, +/// "k": 3, "m": 256}}`, an n-gram tokenizer as `{"ngram": 3}`, and +/// `"short_words": true` only when that option is on), and for a /// JSON index the object of its options (`{"json": {"sealed_root": /// true, "ordered": ["$.age"]}}`). A caller in another language can read /// it, and it does not move when a field is added to the Rust type, as @@ -309,14 +310,16 @@ fn index_value(index: &IndexSpec) -> serde_json::Value { Tokenizer::Standard => serde_json::Value::from("standard"), Tokenizer::Ngram { length } => serde_json::json!({ "ngram": length }), }; - serde_json::json!({ - "match": { - "tokenizer": tokenizer, - "downcase": options.downcase, - "k": options.k, - "m": options.m, - } - }) + let mut value = serde_json::json!({ + "tokenizer": tokenizer, + "downcase": options.downcase, + "k": options.k, + "m": options.m, + }); + if options.short_words { + value["short_words"] = true.into(); + } + serde_json::json!({ "match": value }) } IndexSpec::Json(options) => serde_json::json!({ "json": { diff --git a/packages/stack-encrypt/src/sem/mod.rs b/packages/stack-encrypt/src/sem/mod.rs index c544e0df4..954260a4f 100644 --- a/packages/stack-encrypt/src/sem/mod.rs +++ b/packages/stack-encrypt/src/sem/mod.rs @@ -74,6 +74,29 @@ //! caller's context, so the same value indexed as an equality term, a match //! token, or an ORE key can never produce colliding PRF outputs. //! +//! # What a match index reveals +//! +//! A stored [`MatchTerms`] is a set of Bloom-filter bit positions, `k` per +//! token, under a key the server does not hold. Someone holding the column +//! sees, per row, which bits are set and how many: the fill tracks the number +//! of distinct tokens, and so the text's approximate length. Tokens are +//! deterministic per column, so the same token sets the same bits in every +//! row, and bits that recur together across rows can be clustered into +//! tokens and their frequency counted; a query shows which bits it asks for +//! and which rows hold them. A value that cuts no token stores the empty +//! filter, which shows that it is empty, separator-only or too short for +//! any token. +//! +//! With [`MatchOptions::short_words`] on, each word shorter than the n-gram +//! length (and at least 2 characters) adds one token: the word itself, under +//! the same PRF and a domain of its own, folded into the same filter by the +//! same `k` positions and matched by the same containment rule. That is one +//! extra token per short word and no new class of leakage. The token is +//! specific to the word, so someone who clusters tokens across rows can tell +//! the rows holding the word "Ng" from those holding "Nguyen", where before +//! only the boundary trigram `" ng"` they share was visible. No bigram of a +//! longer word is ever indexed. +//! //! This is a fresh (v2) term format: PRF inputs are framed with vitaminc's PAE //! context encoding, so terms are intentionally **not** byte-compatible with //! `cipherstash-client`'s existing `IndexTerm` values. @@ -141,7 +164,6 @@ mod tokenize; pub use tokenize::Tokenizer; -use std::collections::HashSet; use std::fmt; use std::marker::PhantomData; @@ -184,6 +206,12 @@ const EQUALITY_DOMAIN: &[u8] = b"stack-encrypt/sem/equality/v2"; /// `v2`: the text is tokenized in Unicode 16 NFC, where `v1` tokenized it as /// written. const MATCH_DOMAIN: &[u8] = b"stack-encrypt/sem/match/v2"; +/// PAE domain for a match index's short-word tokens +/// ([`MatchOptions::short_words`]). Distinct from [`MATCH_DOMAIN`] so a word +/// token and a tokenizer token of the same characters derive different PRF +/// blocks, under any tokenizer: `ng` as a word never sets the bits of `ng` +/// as a 2-gram. +const MATCH_WORD_DOMAIN: &[u8] = b"stack-encrypt/sem/match/word/v1"; /// PAE domain for ORE key derivation. /// /// `v2`: the key keys clww-ore's vitaminc ORE scheme, which encrypts the @@ -631,13 +659,58 @@ where /// Options controlling match-term generation. The defaults mirror the existing /// match indexer: 3-gram tokens, downcased, `k = 3` hash slices into an -/// `m = 256`-bit filter. +/// `m = 256`-bit filter, no short-word tokens. +/// +/// Every option is part of the index's identity: a stored filter and a query +/// filter agree only when both were cut under the same options, so a query +/// asked under options other than the index's is refused +/// ([`PlanError::IndexOptions`](crate::plan::PlanError::IndexOptions)). #[derive(Debug, Clone, PartialEq, Eq, Hash)] pub struct MatchOptions { /// How text splits into tokens. pub tokenizer: Tokenizer, /// Lower-case the text before tokenization (case-insensitive matching). pub downcase: bool, + /// Also index every short word whole, so a word too short to hold an + /// n-gram can be searched: "Ng" under 3-grams. Off by default. + /// + /// With it on, every word of the normalized (and, with + /// [`downcase`](Self::downcase), case-folded) text that is at least 2 + /// characters long and shorter than the n-gram length is one more + /// token, the word itself, beside the n-grams of the whole text. Under + /// the default 3-grams that is every two-letter word; under 4-grams, + /// every word of 2 or 3. A one-letter word adds nothing, and a word as + /// long as an n-gram is found by its n-grams as before. A word is a run + /// of characters between the separators [`Tokenizer::Standard`] splits + /// on (space, comma, semicolon, colon, exclamation mark), so "Ng, Mike" + /// holds the word `ng`, and "Ng." or "Ng-Smith" does not (a full stop, a + /// hyphen, a tab or a newline is part of a word); length is counted in + /// characters of the case-folded text. + /// + /// The word token is PRF'd under a domain of its own, so it never + /// derives the bits of an n-gram of the same characters, and folds into + /// the same filter by the same `k` positions. Stored, "Ng" is one token + /// rather than the empty filter, and "Mike Ng" carries its trigrams and + /// `ng`. Queried, "Ng" is one token and is accepted, and "Mike Ng" asks + /// for its trigrams and `ng`: every short word of a query must be a + /// whole word of the row, so "Mike Ng" does not find "Mike Nguyen", + /// which it would with the option off. A query that cuts nothing at all, + /// such as one letter, is still refused + /// ([`TermError::EmptyTermText`]). + /// + /// **What it reveals.** One more token per short word, under the same + /// PRF, in the same filter, matched by the same containment rule, so + /// there is no new class of leakage. The token is specific to the word: + /// someone who clusters tokens across rows can tell the rows holding the + /// word "Ng" from those holding "Nguyen", where before only the boundary + /// trigram `" ng"` they share was visible. Only words shorter than an + /// n-gram get one, so no bigram of a longer word is ever indexed. + /// + /// It takes an n-gram tokenizer of length at least 3: the + /// [`Standard`](Tokenizer::Standard) tokenizer already emits whole words, + /// and under shorter n-grams no word is short enough. Either pairing is + /// refused ([`TermError::InvalidOptions`]). + pub short_words: bool, /// Number of bit positions derived per token, `3..=16` (the v1 match /// indexer's bounds; the upper bound is also the PRF block size — each /// position consumes 2 bytes of the 32-byte block). @@ -652,6 +725,7 @@ impl Default for MatchOptions { Self { tokenizer: Tokenizer::default(), downcase: true, + short_words: false, k: 3, m: 256, } @@ -668,6 +742,21 @@ impl MatchOptions { "n-gram length must be at least 1", )); } + if self.short_words { + match self.tokenizer { + Tokenizer::Standard => { + return Err(TermError::InvalidOptions( + "short_words takes an n-gram tokenizer: the standard tokenizer already indexes whole words", + )) + } + Tokenizer::Ngram { length } if length < 3 => { + return Err(TermError::InvalidOptions( + "short_words takes an n-gram length of at least 3: no word of two characters or more is shorter than the n-grams", + )) + } + Tokenizer::Ngram { .. } => {} + } + } if !(3..=16).contains(&self.k) { return Err(TermError::InvalidOptions("k must be in 3..=16")); } @@ -689,7 +778,9 @@ pub trait MatchConfig: Send + Sync + 'static { fn options() -> MatchOptions; /// The typical number of distinct tokens in a value this index stores: - /// 20 for a name column, 200 for a notes column. A prior, used to size a query's + /// 20 for a name column, 200 for a notes column, counting a value's + /// short-word tokens when [`MatchOptions::short_words`] is on (each sets + /// `k` bits, as an n-gram does). A prior, used to size a query's /// fetch ([`Stats::declared`](crate::op::Stats::declared)) before the /// column's filters have been measured. It is not part of the index's /// identity and changes no term. `None`, the default, declares no @@ -1033,7 +1124,7 @@ impl RangeQuery { /// can make), so one query's tokens never verify rows for another value's /// filter. #[derive(Clone, Default)] -pub struct QueryTokens(Zeroizing>); +pub struct QueryTokens(Zeroizing); impl QueryTokens { /// The tokens of `text` under `O`'s tokenizer and case folding. @@ -1052,11 +1143,7 @@ impl QueryTokens { /// /// As for [`of`](Self::of). pub(crate) fn under(text: &str, options: &MatchOptions) -> Result { - Ok(Self(Zeroizing::new(tokenize::tokens( - text, - options.tokenizer, - options.downcase, - )?))) + Ok(Self(Zeroizing::new(tokenize::tokens(text, options)?))) } /// The tokens of a query's `text` under `options`, or why the query is @@ -1079,15 +1166,16 @@ impl QueryTokens { /// Whether `value` holds every one of these tokens, cut under the same /// `options` the tokens were: what [`MatchQuery`]'s `verify` asks, with /// the options a value rather than a type, so a match index declared as - /// data verifies by the same code. No tokens verify nothing, and a value - /// the index cannot normalize holds no token of any query. + /// data verifies by the same code. Each kind is held by its own kind: a + /// short-word token of the query must be a short word of the value. No + /// tokens verify nothing, and a value the index cannot normalize holds + /// no token of any query. pub(crate) fn verify(&self, value: &str, options: &MatchOptions) -> bool { - let Ok(tokens) = tokenize::tokens(value, options.tokenizer, options.downcase) else { + let Ok(tokens) = tokenize::tokens(value, options) else { return false; }; let tokens = Zeroizing::new(tokens); - let have: HashSet<&str> = tokens.iter().map(String::as_str).collect(); - !self.0.is_empty() && self.0.iter().all(|t| have.contains(t.as_str())) + !self.0.is_empty() && tokens.hold(&self.0) } } @@ -1158,9 +1246,13 @@ impl PrfVisitor<[u8; 32], P> for BloomVisitor { /// Derive a match term. Synchronous — see [`equality`]: validation and /// tokenization run before the PRF, the visitor folds blocks into positions. +/// The tokenizer's tokens derive under [`MATCH_DOMAIN`] and the short-word +/// tokens under [`MATCH_WORD_DOMAIN`], each bound to `context`, and both +/// fold into the one filter. /// /// Text that cuts no token (empty, separator-only, or shorter than the -/// n-gram length) is the empty filter: no position set, no PRF call. Stored, +/// n-gram length with no short word) is the empty filter: no position set, +/// no PRF call. Stored, /// it is a value no query matches, since a query always has at least one /// token and the empty filter holds none of its positions. This is the /// *stored* side's answer; a query of such text is refused by the query @@ -1172,25 +1264,54 @@ fn match_term( options: MatchOptions, ) -> Result, TermError> { let mask = options.validate()?; - let tokens = tokenize::tokens(text, options.tokenizer, options.downcase)?; - if tokens.is_empty() { - return Ok(MatchTerms::normalised(Vec::new())); - } - let context = Context::pae(&[MATCH_DOMAIN, context.as_bytes()]); - // The PRF takes the tokens by value and moves each `String`'s buffer // into a `Protected`, which zeroizes it on drop: the tokens are wiped - // there, not here. The `Vec` itself holds only their pointers. - - let positions = tokens - .prf_visit_with_context(prf, context, BloomVisitor { k: options.k, mask }) - .into_result() - .map_err(TermError::from_prf)?; + // there, not here. The `Vec`s themselves hold only their pointers. + let tokenize::Tokens { + tokenizer, + short_words, + } = tokenize::tokens(text, &options)?; + // Held wiped-on-drop until the PRF takes them, so a failure deriving + // the tokenizer's tokens does not free the words unwiped. + let mut short_words = Zeroizing::new(short_words); + let mut positions = bloom_positions(prf, tokenizer, MATCH_DOMAIN, &context, options.k, mask)?; + positions.extend(bloom_positions( + prf, + std::mem::take(&mut *short_words), + MATCH_WORD_DOMAIN, + &context, + options.k, + mask, + )?); // Every position came out of the visitor masked to `m - 1`, so the range // check `from_positions` applies is already satisfied by construction. Ok(MatchTerms::normalised(positions)) } +/// The Bloom positions of `tokens`, each PRF'd under the PAE-framed +/// `[domain, context]` and folded by [`BloomVisitor`]. No tokens is no +/// positions and no PRF call. +fn bloom_positions( + prf: &HmacSha256Prf, + tokens: Vec, + domain: &[u8], + context: &Context<'_>, + k: usize, + mask: u16, +) -> Result, TermError> { + if tokens.is_empty() { + return Ok(Vec::new()); + } + tokens + .prf_visit_with_context( + prf, + Context::pae(&[domain, context.as_bytes()]), + BloomVisitor { k, mask }, + ) + .into_result() + .map_err(TermError::from_prf) +} + /// A match term of any text source, generated under `O`'s options. Under /// the local HMAC backend, derived during the synchronous build — the /// returned [`Pending`] carries no requests (tokenize makes the one @@ -2134,7 +2255,7 @@ mod tests { // one built any other way, which must never match every row. let query = MatchQuery::::new( MatchTerms::normalised(vec![1, 2, 3]), - QueryTokens(Zeroizing::new(Vec::new())), + QueryTokens::default(), ); assert!(!crate::op::Probabilistic::verify(&query, "anything at all")); assert!(!crate::op::Probabilistic::verify(&query, "")); @@ -2215,14 +2336,202 @@ mod tests { )); let tokens = QueryTokens::of_query("ALICE", &options).expect("cuts tokens"); assert_eq!( - tokens.0.as_slice(), + tokens.0.tokenizer, QueryTokens::under("alice", &options) .expect("normalizes") .0 - .as_slice() + .tokenizer ); } + mod given_short_words { + use super::*; + use proptest::prelude::*; + + fn options() -> MatchOptions { + MatchOptions { + short_words: true, + ..MatchOptions::default() + } + } + + /// The positions `token` sets as one token of the kind `domain` + /// names, under the widest filter and the most slices, so two + /// blocks that differ anywhere in their first 32 bytes show it. + fn positions(prf: &HmacSha256Prf, token: &str, domain: &[u8], context: &[u8]) -> Vec { + bloom_positions( + prf, + vec![token.to_owned()], + domain, + &Context::from_encoded(context), + 16, + u16::MAX, + ) + .expect("derives") + } + + #[test] + fn the_option_is_valid_beside_ngrams_of_three_or_more() { + for length in [3, 4, 8] { + let options = MatchOptions { + tokenizer: Tokenizer::Ngram { length }, + ..options() + }; + assert!( + options.validate().is_ok(), + "{length}-grams take short words" + ); + } + } + + #[test] + fn the_option_is_refused_beside_the_standard_tokenizer_or_shorter_ngrams() { + for tokenizer in [ + Tokenizer::Standard, + Tokenizer::Ngram { length: 1 }, + Tokenizer::Ngram { length: 2 }, + ] { + let refused = MatchOptions { + tokenizer, + ..options() + } + .validate(); + assert!( + matches!(refused, Err(TermError::InvalidOptions(_))), + "{tokenizer:?}: {refused:?}" + ); + assert!( + MatchOptions { + tokenizer, + ..MatchOptions::default() + } + .validate() + .is_ok(), + "{tokenizer:?} is valid without the option" + ); + } + } + + #[test] + fn a_query_of_one_short_word_is_accepted_and_of_one_letter_refused() { + let tokens = QueryTokens::of_query("Ng", &options()).expect("one word token"); + assert_eq!(tokens.0.short_words, ["ng"], "the word, folded"); + assert!(tokens.0.tokenizer.is_empty(), "no trigram in two letters"); + for text in ["N", "", " ", "n "] { + assert!( + matches!( + QueryTokens::of_query(text, &options()), + Err(TermError::EmptyTermText) + ), + "{text:?} still cuts nothing" + ); + } + assert!( + matches!( + QueryTokens::of_query( + "Ng", + &MatchOptions { + tokenizer: Tokenizer::Standard, + ..options() + } + ), + Err(TermError::InvalidOptions(_)) + ), + "options are checked before the text" + ); + } + + /// A word token is framed under its own domain, so it never sets + /// the bits an n-gram of the same characters sets: pinned for + /// `ng`, under a fixed key and the widest filter, where the two + /// would share all sixteen positions if the domains did not part + /// them. + #[test] + fn a_word_token_and_an_ngram_of_the_same_characters_share_no_position() { + let prf = HmacSha256Prf::new(Protected::new([7; 32])); + let as_gram = positions(&prf, "ng", MATCH_DOMAIN, b"users/name"); + let as_word = positions(&prf, "ng", MATCH_WORD_DOMAIN, b"users/name"); + assert_eq!(as_gram.len(), 16, "sixteen distinct positions: {as_gram:?}"); + assert!( + as_gram.iter().all(|p| as_word.binary_search(p).is_err()), + "no position in common: {as_gram:?} and {as_word:?}" + ); + } + + /// The stored filter of a value is the union of its two kinds, + /// each under its own domain: the n-grams' positions under the + /// match domain, the words' under the word domain. + #[test] + fn a_stored_filter_is_the_union_of_both_kinds() { + let prf = HmacSha256Prf::new(Protected::new([7; 32])); + let context = Context::from_encoded(&b"users/name"[..]); + let stored: MatchTerms = + match_term(&prf, "Mike Ng", context.clone(), options()).expect("derives"); + let mask = options().validate().expect("valid"); + let mut union = bloom_positions( + &prf, + vec![ + "mik".into(), + "ike".into(), + "ke ".into(), + "e n".into(), + " ng".into(), + ], + MATCH_DOMAIN, + &context, + 3, + mask, + ) + .expect("grams"); + union.extend( + bloom_positions( + &prf, + vec!["ng".into()], + MATCH_WORD_DOMAIN, + &context, + 3, + mask, + ) + .expect("word"), + ); + assert_eq!( + stored, + MatchTerms::normalised(union), + "the trigrams of the whole text and the word" + ); + let alone: MatchTerms = + match_term(&prf, "Ng", context.clone(), options()).expect("derives"); + assert!( + !alone.positions().is_empty() && stored.contains(&alone), + "\"Ng\" stores one token, which \"Mike Ng\" holds: {alone:?}" + ); + let without: MatchTerms = + match_term(&prf, "Ng", context, MatchOptions::default()).expect("derives"); + assert!( + without.positions().is_empty(), + "without the option, \"Ng\" is the empty filter" + ); + } + + proptest! { + /// For any token, key and context, the word domain and the + /// match domain derive different blocks: the positions differ. + #[test] + fn the_two_domains_never_derive_one_block( + token in "\\PC{1,4}", + key in any::<[u8; 32]>(), + context in prop::collection::vec(any::(), 1..24), + ) { + let prf = HmacSha256Prf::new(Protected::new(key)); + prop_assert_ne!( + positions(&prf, &token, MATCH_DOMAIN, &context), + positions(&prf, &token, MATCH_WORD_DOMAIN, &context), + "a word token and an n-gram of the same characters derive apart" + ); + } + } + } + /// The Bloom fold reads a token *stream*: handed a map-shaped PRF input /// it refuses, rather than folding the entries' blocks as if they were /// tokens. diff --git a/packages/stack-encrypt/src/sem/tokenize.rs b/packages/stack-encrypt/src/sem/tokenize.rs index 419ae6c89..7d02675b5 100644 --- a/packages/stack-encrypt/src/sem/tokenize.rs +++ b/packages/stack-encrypt/src/sem/tokenize.rs @@ -9,10 +9,19 @@ //! `Standard` splits on the same separator set. The one deliberate divergence //! is that empty tokens are dropped (v1 keeps the empty strings its separator //! split produces, which only add noise bits to every filter). +//! +//! Beside the tokenizer's tokens, an n-gram index with +//! [`MatchOptions::short_words`] on cuts one more kind: each word too short to +//! hold an n-gram, whole. The two kinds are kept apart ([`Tokens`]) because +//! each is run through the PRF under a domain of its own. + +use std::collections::HashSet; use vitaminc_normalize::NormalizeError; use vitaminc_protected::Controlled; -use zeroize::Zeroizing; +use zeroize::{Zeroize, Zeroizing}; + +use super::MatchOptions; /// How text is split into tokens before each token is run through the PRF. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] @@ -25,6 +34,12 @@ pub enum Tokenizer { /// rather than silently never matching, and a stored value shorter than /// it is the empty filter, which no query matches. This is the default, /// matching the existing match indexer's 3-gram configuration. + /// + /// The n-grams are cut over the whole text, so a word shorter than + /// `length` is only ever seen inside the n-grams that cross its edges: + /// a query of a two-letter word cuts nothing under 3-grams. With + /// [`MatchOptions::short_words`] on, each such word is also one token of + /// its own. Ngram { length: usize }, /// Split on the v1 match indexer's separator set — space, comma, /// semicolon, colon and exclamation mark — one token per run of @@ -33,9 +48,65 @@ pub enum Tokenizer { } /// The separators `Tokenizer::Standard` splits on, as in the v1 match -/// indexer's `process_standard`. +/// indexer's `process_standard`. The short-word filter splits words on the +/// same set (see [`Tokens::short_words`]). const STANDARD_SEPARATORS: [char; 5] = [' ', ',', ';', ':', '!']; +/// The shortest word the short-word filter indexes, in characters. A +/// one-character word would be one of a few dozen tokens per alphabet, which +/// says more about a row than it finds. +const SHORTEST_WORD: usize = 2; + +/// The tokens a match index cuts from one text, by kind. Each kind is run +/// through the PRF under a domain of its own, so a word token can never +/// derive the bits of an n-gram of the same characters. +/// +/// Plaintext: the caller zeroizes it, through [`Zeroize`] (wrap it in +/// [`Zeroizing`]), or moves the strings on to something that does. +#[derive(Clone, Default)] +pub(crate) struct Tokens { + /// The tokenizer's tokens: the n-grams of the whole text, or the words + /// [`Tokenizer::Standard`] splits it into. + pub(crate) tokenizer: Vec, + /// The short-word tokens, empty unless [`MatchOptions::short_words`] is + /// on: every word of the text at least 2 characters long and shorter + /// than the n-gram length, whole, in the order the text holds them. + /// + /// A word is a run of characters between the separators + /// [`Tokenizer::Standard`] splits on (space, comma, semicolon, colon and + /// exclamation mark), so a short word is exactly a token the standard + /// tokenizer would cut, of that length: "Ng, Mike" holds the word `ng`. + /// Length is counted in characters of the NFC text after case folding + /// (the text the n-grams are cut from), so `李明` is a word of two, and + /// `İz`, which folds to three characters, is not. + pub(crate) short_words: Vec, +} + +impl Tokens { + /// Whether no token of either kind was cut. + pub(crate) fn is_empty(&self) -> bool { + self.tokenizer.is_empty() && self.short_words.is_empty() + } + + /// Whether `self` holds every token of `query`, kind by kind: what a + /// match query's verifier asks of a row. A token of one kind never + /// stands in for a token of the other. + pub(crate) fn hold(&self, query: &Tokens) -> bool { + fn holds(have: &[String], want: &[String]) -> bool { + let have: HashSet<&str> = have.iter().map(String::as_str).collect(); + want.iter().all(|token| have.contains(token.as_str())) + } + holds(&self.tokenizer, &query.tokenizer) && holds(&self.short_words, &query.short_words) + } +} + +impl Zeroize for Tokens { + fn zeroize(&mut self) { + self.tokenizer.zeroize(); + self.short_words.zeroize(); + } +} + impl Default for Tokenizer { fn default() -> Self { Self::Ngram { length: 3 } @@ -51,26 +122,23 @@ impl Default for Tokenizer { /// /// The normalized copy is held in [`Protected`](vitaminc_protected::Protected) /// and wiped on drop. -pub(crate) fn tokens( - text: &str, - tokenizer: Tokenizer, - downcase: bool, -) -> Result, NormalizeError> { +pub(crate) fn tokens(text: &str, options: &MatchOptions) -> Result { let normalized = vitaminc_normalize::text(text)?; // `text` returns the UTF-8 of a `str`, so this does not fail; a failure // would be a bug there, refused rather than tokenized. let normalized = std::str::from_utf8(normalized.risky_ref()).map_err(|_| NormalizeError::Invalid)?; - Ok(tokenize(normalized, tokenizer, downcase)) + Ok(tokenize(normalized, options)) } -/// Split `text` into tokens. `downcase` lower-cases the text first so matches -/// are case-insensitive. +/// Split `text` into tokens under `options`' tokenizer, case folding and +/// short-word filter. `downcase` lower-cases the text first so matches are +/// case-insensitive. /// /// May return no tokens (empty text, separator-only text, or text shorter -/// than the n-gram length). A stored value with none is the empty filter; a -/// query with none is refused before a term exists, because its filter would -/// match every row (ADR-0009). +/// than the n-gram length that holds no short word). A stored value with +/// none is the empty filter; a query with none is refused before a term +/// exists, because its filter would match every row (ADR-0009). /// /// The tokenizer must be valid: an n-gram length of zero panics. Callers /// validate the [`MatchOptions`](super::MatchOptions) first. @@ -84,23 +152,50 @@ pub(crate) fn tokens( /// into a buffer sized up front, and each n-gram, which is built at its /// final size. Neither reallocates, so no freed buffer holds plaintext. The /// one exception is inside `str::to_lowercase`, which grows its output when -/// folding lengthens the text (as `İ` does). The tokens returned are the -/// caller's to zeroize. +/// folding lengthens the text (as `İ` does). A short word is copied out of +/// the folded text at its final size. The tokens returned are the caller's +/// to zeroize. /// /// [`MatchOptions::validate`]: super::MatchOptions -pub(crate) fn tokenize(text: &str, tokenizer: Tokenizer, downcase: bool) -> Vec { - let text = Zeroizing::new(if downcase { +pub(crate) fn tokenize(text: &str, options: &MatchOptions) -> Tokens { + let text = Zeroizing::new(if options.downcase { text.to_lowercase() } else { text.to_string() }); + let short_words = match options.tokenizer { + Tokenizer::Ngram { length } if options.short_words => short_words(&text, length), + // `MatchOptions::validate` refuses the option beside the standard + // tokenizer, whose tokens already are the words. + _ => Vec::new(), + }; + Tokens { + tokenizer: tokenizer_tokens(&text, options.tokenizer), + short_words, + } +} + +/// Every word of `text` at least [`SHORTEST_WORD`] characters long and +/// shorter than `length`, whole, in order (see [`Tokens::short_words`]). +fn short_words(text: &str, length: usize) -> Vec { + words(text) + .filter(|word| (SHORTEST_WORD..length).contains(&word.chars().count())) + .map(str::to_string) + .collect() +} + +/// The words of `text`: the runs between [`STANDARD_SEPARATORS`], empty runs +/// dropped. The standard tokenizer's tokens and the short-word filter's +/// words are both these, so a short word is exactly a standard token. +fn words(text: &str) -> impl Iterator { + text.split(STANDARD_SEPARATORS) + .filter(|word| !word.is_empty()) +} +/// The tokens `tokenizer` cuts from `text`, already case folded. +fn tokenizer_tokens(text: &str, tokenizer: Tokenizer) -> Vec { match tokenizer { - Tokenizer::Standard => text - .split(STANDARD_SEPARATORS) - .filter(|token| !token.is_empty()) - .map(str::to_string) - .collect(), + Tokenizer::Standard => words(text).map(str::to_string).collect(), Tokenizer::Ngram { length } => { // `collect` would size the buffer from `Chars::size_hint`, about // a quarter of the text, and grow it, leaving plaintext in each @@ -132,9 +227,40 @@ fn gram(chars: &[char]) -> String { mod tests { use super::*; + /// The tokenizer's tokens of `text` under `tokenizer`, with or without + /// case folding and no short words: the tokens the index cut before the + /// short-word filter existed. + fn cut(text: &str, tokenizer: Tokenizer, downcase: bool) -> Vec { + let tokens = tokenize( + text, + &MatchOptions { + tokenizer, + downcase, + ..MatchOptions::default() + }, + ); + assert!( + tokens.short_words.is_empty(), + "no short word is cut with the option off: {:?}", + tokens.short_words + ); + tokens.tokenizer + } + #[test] fn tokens_are_cut_from_the_nfc_form() { - let standard = |text| tokens(text, Tokenizer::Standard, false).expect("tokens"); + let standard = |text| { + tokens( + text, + &MatchOptions { + tokenizer: Tokenizer::Standard, + downcase: false, + ..MatchOptions::default() + }, + ) + .expect("tokens") + .tokenizer + }; assert_eq!( standard("cafe\u{301} au lait"), standard("caf\u{e9} au lait"), @@ -145,29 +271,38 @@ mod tests { #[test] fn tokens_refuse_a_code_point_unassigned_in_unicode_16() { - assert_eq!( - tokens("a\u{378}b", Tokenizer::Standard, true), - Err(NormalizeError::UnassignedCodePoint) + assert!( + matches!( + tokens( + "a\u{378}b", + &MatchOptions { + tokenizer: Tokenizer::Standard, + ..MatchOptions::default() + } + ), + Err(NormalizeError::UnassignedCodePoint) + ), + "an unassigned code point has no normal form to tokenize" ); } #[test] fn ngram_tokenizes_sliding_windows() { assert_eq!( - tokenize("hello", Tokenizer::Ngram { length: 3 }, true), + cut("hello", Tokenizer::Ngram { length: 3 }, true), vec!["hel", "ell", "llo"] ); } #[test] fn a_zero_ngram_length_yields_no_tokens_rather_than_panicking() { - assert!(tokenize("hello", Tokenizer::Ngram { length: 0 }, true).is_empty()); + assert!(cut("hello", Tokenizer::Ngram { length: 0 }, true).is_empty()); } #[test] fn ngram_gram_length_text_yields_one_token() { assert_eq!( - tokenize("hey", Tokenizer::Ngram { length: 3 }, true), + cut("hey", Tokenizer::Ngram { length: 3 }, true), vec!["hey"] ); } @@ -176,36 +311,313 @@ mod tests { fn ngram_short_text_yields_no_tokens() { // Mirrors the v1 indexer: a sub-gram-length probe can never match a // stored gram, so it must not produce a token at all. - assert!(tokenize("hi", Tokenizer::Ngram { length: 3 }, true).is_empty()); + assert!(cut("hi", Tokenizer::Ngram { length: 3 }, true).is_empty()); } #[test] fn ngram_empty_text_yields_no_tokens() { - assert!(tokenize("", Tokenizer::Ngram { length: 3 }, true).is_empty()); + assert!(cut("", Tokenizer::Ngram { length: 3 }, true).is_empty()); } #[test] fn standard_splits_on_the_v1_separator_set() { assert_eq!( - tokenize("Hello, World! again", Tokenizer::Standard, true), + cut("Hello, World! again", Tokenizer::Standard, true), vec!["hello", "world", "again"] ); } #[test] fn standard_drops_empty_tokens() { - assert!(tokenize(" ,;:! ", Tokenizer::Standard, true).is_empty()); + assert!(cut(" ,;:! ", Tokenizer::Standard, true).is_empty()); } #[test] fn standard_does_not_split_on_other_whitespace() { // The v1 separator set is exactly ' ', ',', ';', ':', '!' — tabs and // newlines are part of the token, as in v1. - assert_eq!(tokenize("a\tb", Tokenizer::Standard, true), vec!["a\tb"]); + assert_eq!(cut("a\tb", Tokenizer::Standard, true), vec!["a\tb"]); } #[test] fn downcase_can_be_disabled() { - assert_eq!(tokenize("Hi", Tokenizer::Standard, false), vec!["Hi"]); + assert_eq!(cut("Hi", Tokenizer::Standard, false), vec!["Hi"]); + } + + /// Tokens are plaintext: zeroizing them wipes and drops both kinds, + /// which is what a `Zeroizing` does when a query is dropped. + #[test] + fn zeroizing_tokens_clears_both_kinds() { + let mut tokens = tokenize( + "Mike Ng", + &MatchOptions { + short_words: true, + ..MatchOptions::default() + }, + ); + assert!( + !tokens.tokenizer.is_empty() && !tokens.short_words.is_empty(), + "both kinds are cut before" + ); + tokens.zeroize(); + assert!(tokens.is_empty(), "no token of either kind is left after"); + } + + mod given_short_words { + use super::*; + + /// The tokens of `text` under `length`-grams, downcased, with the + /// short-word filter on. + fn cut(text: &str, length: usize) -> Tokens { + tokenize( + text, + &MatchOptions { + tokenizer: Tokenizer::Ngram { length }, + short_words: true, + ..MatchOptions::default() + }, + ) + } + + #[test] + fn a_two_letter_word_is_one_token_at_the_start_middle_and_end() { + for (text, words) in [ + ("Ng Mike", vec!["ng"]), + ("Mike Ng Smith", vec!["ng"]), + ("Mike Ng", vec!["ng"]), + ("Ng", vec!["ng"]), + ("Li Wu Ho", vec!["li", "wu", "ho"]), + ] { + assert_eq!( + cut(text, 3).short_words, + words, + "the short words of {text:?}" + ); + } + } + + #[test] + fn the_ngrams_of_the_whole_text_are_cut_as_without_the_option() { + for text in ["Mike Ng", "Ng", "a b", "", "Nguyen", "Ng, Mike!"] { + assert_eq!( + cut(text, 3).tokenizer, + super::cut(text, Tokenizer::Ngram { length: 3 }, true), + "the n-grams of {text:?} do not move" + ); + } + } + + #[test] + fn a_one_letter_word_is_no_token() { + let tokens = cut("J R R Tolkien", 3); + assert!( + tokens.short_words.is_empty(), + "one-letter words are ignored: {:?}", + tokens.short_words + ); + assert!(cut("a", 3).is_empty(), "a one-letter text cuts nothing"); + } + + #[test] + fn a_word_as_long_as_an_ngram_is_no_word_token() { + let tokens = cut("Ngo Nguyen", 3); + assert!( + tokens.short_words.is_empty(), + "a word of three letters or more is found by its trigrams: {:?}", + tokens.short_words + ); + } + + #[test] + fn a_four_gram_index_takes_two_and_three_letter_words() { + let tokens = cut("Le Ngo Nguyen x", 4); + assert_eq!( + tokens.short_words, + vec!["le", "ngo"], + "under 4-grams, words of 2 and 3 characters are short" + ); + } + + #[test] + fn words_split_on_the_standard_separators() { + assert_eq!( + cut("Ng,Mike;Li:Wu!Ho Yu", 3).short_words, + vec!["ng", "li", "wu", "ho", "yu"], + "space, comma, semicolon, colon and exclamation mark end a word" + ); + assert_eq!( + cut("ng\tli", 3).short_words, + Vec::::new(), + "a tab does not end a word, as it does not for the standard tokenizer" + ); + assert_eq!( + cut("ng.", 3).short_words, + Vec::::new(), + "a full stop is part of the word, so `ng.` is three characters" + ); + } + + #[test] + fn a_word_is_counted_in_characters_not_bytes() { + // Six bytes of UTF-8 each, two characters. + assert_eq!( + cut("\u{674e}\u{660e} \u{e9}\u{e9}", 3).short_words, + vec!["\u{674e}\u{660e}", "\u{e9}\u{e9}"], + "two-character words outside ASCII are short" + ); + assert_eq!( + tokens( + "e\u{301}e\u{301}", + &MatchOptions { + short_words: true, + ..MatchOptions::default() + } + ) + .expect("normalizes") + .short_words, + vec!["\u{e9}\u{e9}"], + "the characters are counted in NFC: `e` and a combining accent are one" + ); + } + + #[test] + fn the_word_is_case_folded_as_the_ngrams_are() { + let folded = cut("NG", 3); + assert_eq!(folded.short_words, vec!["ng"], "downcased"); + let kept = tokenize( + "NG", + &MatchOptions { + downcase: false, + short_words: true, + ..MatchOptions::default() + }, + ); + assert_eq!(kept.short_words, vec!["NG"], "kept as written"); + } + + #[test] + fn the_standard_tokenizer_cuts_no_word_token() { + // `MatchOptions::validate` refuses the pair; tokenizing before + // the check cuts no second copy of the words. + let tokens = tokenize( + "Mike Ng", + &MatchOptions { + tokenizer: Tokenizer::Standard, + short_words: true, + ..MatchOptions::default() + }, + ); + assert!( + tokens.short_words.is_empty(), + "the standard tokenizer's words are its own tokens: {:?}", + tokens.short_words + ); + } + + mod given_random_text { + use super::*; + use proptest::prelude::*; + + /// A word of one to six characters, Latin and not, so words + /// fall either side of every short-word length and length is + /// counted in characters. + fn word() -> impl Strategy { + prop::collection::vec( + prop::sample::select(vec!['a', 'n', 'g', 'é', 'ü', '李', '明']), + 1..=6, + ) + .prop_map(|chars| chars.into_iter().collect()) + } + + /// Words and the separators between them: runs of the standard + /// separators, and empty runs at either end. + fn text() -> impl Strategy, String)> { + let gap = prop::collection::vec( + prop::sample::select(STANDARD_SEPARATORS.to_vec()), + 1..=2, + ) + .prop_map(|gap| gap.into_iter().collect::()); + ( + prop::collection::vec((word(), gap), 0..10), + prop::sample::select(vec!["", " ", ","]), + ) + .prop_map(|(words, lead)| { + let mut text = lead.to_owned(); + for (word, gap) in &words { + text.push_str(word); + text.push_str(gap); + } + (words.into_iter().map(|(word, _)| word).collect(), text) + }) + } + + proptest! { + /// Every word of 2 characters up to the n-gram length is + /// exactly one word token, in order, and nothing else is. + #[test] + fn every_short_word_is_one_token_and_no_other_word_is( + (words, text) in text(), + length in 3usize..=5, + ) { + let want: Vec<&str> = words + .iter() + .map(String::as_str) + .filter(|word| (2..length).contains(&word.chars().count())) + .collect(); + let tokens = cut(&text, length); + prop_assert_eq!( + &tokens.short_words, + &want, + "the short words of {:?} under {}-grams", + text, + length + ); + for word in &tokens.short_words { + let chars = word.chars().count(); + prop_assert!( + (2..length).contains(&chars), + "{:?} is {} characters, outside 2..{}", + word, + chars, + length + ); + } + } + + /// The n-grams do not depend on the option: it adds word + /// tokens and moves nothing else. + #[test] + fn the_option_adds_word_tokens_and_moves_no_ngram( + (_, text) in text(), + length in 3usize..=5, + ) { + prop_assert_eq!( + cut(&text, length).tokenizer, + super::super::cut(&text, Tokenizer::Ngram { length }, true), + "the n-grams of {:?}", + text + ); + } + } + } + + #[test] + fn a_row_holds_a_query_kind_by_kind() { + let row = cut("Mike Ng", 3); + assert!(row.hold(&cut("Ng", 3)), "the word query is held"); + assert!(row.hold(&cut("Mike Ng", 3)), "the whole query is held"); + assert!( + !cut("Nguyen", 3).hold(&cut("Ng", 3)), + "Nguyen has no word ng" + ); + let as_gram = Tokens { + tokenizer: vec!["ng".into()], + short_words: Vec::new(), + }; + assert!( + !row.hold(&as_gram), + "a word token does not stand in for a token of the tokenizer's" + ); + } } } diff --git a/packages/stack-encrypt/tests/index.rs b/packages/stack-encrypt/tests/index.rs index 2b240598b..bf90b1f2d 100644 --- a/packages/stack-encrypt/tests/index.rs +++ b/packages/stack-encrypt/tests/index.rs @@ -226,6 +226,7 @@ mod given_a_set_of_indexes { MatchOptions { tokenizer: Tokenizer::Standard, downcase: false, + short_words: false, k: 4, m: 512, } @@ -611,6 +612,7 @@ mod given_encrypted_terms_as_a_target { MatchOptions { tokenizer: Tokenizer::Standard, downcase: false, + short_words: false, k: 4, m: 512, } diff --git a/packages/stack-encrypt/tests/match_short_words.rs b/packages/stack-encrypt/tests/match_short_words.rs new file mode 100644 index 000000000..bb154932f --- /dev/null +++ b/packages/stack-encrypt/tests/match_short_words.rs @@ -0,0 +1,522 @@ +//! A match index with short-word tokens (`MatchOptions::short_words`): a +//! word too short to hold an n-gram, such as the surname "Ng" under +//! 3-grams, is one token of its own, stored and queried. +//! +//! - **Round trip.** "Ng" stored is found by the query "Ng"; "Mike Ng" is +//! found by "Ng" and by "Mike Ng"; "Nguyen" is not found by "Ng". +//! - **Identity.** The option is part of the index: a query asked under +//! other options than the field declares is refused, and options the +//! index cannot use are refused on both sides. +//! - **Containment.** For random text, the stored filter holds the filter of +//! any run of its whole words. +//! - **Calibration.** A one-word query sets as many bits as a one-trigram +//! query, and a row's chance of matching either falsely is the same. + +mod common; + +use common::stack_cipher; +use proptest::prelude::*; +use proptest::strategy::ValueTree; +use proptest::test_runner::{Config, RngAlgorithm, RngSeed, TestRng, TestRunner}; +use stack_encrypt::kms::FakeDataKeySource; +use stack_encrypt::op::{self, MatchCase, Probabilistic}; +use stack_encrypt::plan::{Field, FieldSchema, Fields, Indexed, PlanError, ValuePlan}; +use stack_encrypt::sem::{MatchConfig, MatchOptions, MatchQuery, MatchTerms, TermError, Tokenizer}; +use stack_encrypt::target::{Encrypted, IndexSpec}; +use stack_encrypt::{nonempty, Error, Match, Plan, StackCipher}; + +/// A one-value plan of text under one match index of the configuration `C`. +type MatchPlan = ValuePlan>>; + +/// 3-grams, downcased, with short-word tokens: the default index with the +/// option on. +struct ShortWords; +impl MatchConfig for ShortWords { + fn options() -> MatchOptions { + MatchOptions { + short_words: true, + ..MatchOptions::default() + } + } +} +impl MatchCase for ShortWords { + type Case = op::CaseInsensitive; +} + +/// Short words beside the standard tokenizer, which already indexes whole +/// words: options the index refuses. +struct StandardShortWords; +impl MatchConfig for StandardShortWords { + fn options() -> MatchOptions { + MatchOptions { + tokenizer: Tokenizer::Standard, + short_words: true, + ..MatchOptions::default() + } + } +} +impl MatchCase for StandardShortWords { + type Case = op::CaseInsensitive; +} + +type Cipher = StackCipher; + +fn plan() -> MatchPlan { + Plan::context("people/name") + .with::(Match::::new()) + .build() + .expect("a valid plan") +} + +async fn store( + cipher: &Cipher, + plan: &MatchPlan, + text: &str, +) -> MatchTerms { + let stored: Encrypted> = cipher + .encrypt(&text.to_owned()) + .using(plan) + .await + .unwrap_or_else(|error| panic!("{text:?} stores: {error}")); + stored.terms +} + +async fn query( + cipher: &Cipher, + plan: &MatchPlan, + text: &str, +) -> Result, Error> { + cipher + .query(op::Matches, text) + .via(Match::::new()) + .on(plan) + .await +} + +fn plan_error(result: Result) -> PlanError { + match result { + Err(Error::Plan(error)) => error, + Err(other) => panic!("expected a plan error, got {other:?}"), + Ok(_) => panic!("expected a plan error, got a value"), + } +} + +mod given_short_words { + use super::*; + + #[tokio::test] + async fn a_two_letter_value_stores_one_token_and_the_two_letter_query_finds_it() { + let cipher = stack_cipher().await; + let plan = plan(); + let stored = store(&cipher, &plan, "Ng").await; + assert!( + !stored.positions().is_empty(), + "\"Ng\" stores a token, not the empty filter" + ); + let ng = query(&cipher, &plan, "Ng") + .await + .expect("\"Ng\" is a query"); + assert!(stored.matches(&ng), "the query \"Ng\" finds \"Ng\""); + assert!(ng.verify("Ng"), "and the row verifies"); + assert_eq!( + ng.positions(), + stored.positions(), + "a query's filter is the filter the index stores for the same text" + ); + } + + #[tokio::test] + async fn a_two_letter_query_finds_the_word_and_not_a_longer_word_it_begins() { + let cipher = stack_cipher().await; + let plan = plan(); + let ng = query(&cipher, &plan, "ng").await.expect("a query"); + for (text, found) in [ + ("Ng", true), + ("Mike Ng", true), + ("Ng, Mike", true), + ("NG Mike Smith", true), + ("Nguyen", false), + ("Mike Nguyen", false), + ("Long", false), + ] { + let stored = store(&cipher, &plan, text).await; + // The filter alone: a row without the word could hold its bits + // by chance, but under the test key and these rows none does, + // so a stored side that set the word's bits for a longer word + // fails here. + assert_eq!( + stored.matches(&ng), + found, + "the filter of the query \"ng\" on {text:?}" + ); + assert_eq!( + ng.verify(text), + found, + "verify agrees with the filter on {text:?}" + ); + } + } + + #[tokio::test] + async fn a_query_of_words_and_ngrams_needs_every_short_word_whole() { + let cipher = stack_cipher().await; + let plan = plan(); + let mike_ng = query(&cipher, &plan, "Mike Ng").await.expect("a query"); + let stored = store(&cipher, &plan, "Mike Ng").await; + assert!(stored.matches(&mike_ng), "\"Mike Ng\" finds itself"); + assert!(mike_ng.verify("mike ng"), "case folded"); + let longer = store(&cipher, &plan, "Mike Nguyen").await; + assert!( + !mike_ng.verify("Mike Nguyen"), + "the query's short word must be a whole word of the row" + ); + assert!( + !longer.matches(&mike_ng), + "and the filter says so: \"Mike Nguyen\" has no word token ng" + ); + let mike = query(&cipher, &plan, "Mike").await.expect("a query"); + assert!( + longer.matches(&mike) && stored.matches(&mike), + "a query of trigrams alone finds both" + ); + } + + #[tokio::test] + async fn a_query_that_cuts_nothing_is_still_refused() { + let cipher = stack_cipher().await; + let plan = plan(); + for text in ["N", "", " ", "n "] { + let refused = query(&cipher, &plan, text).await; + assert!( + matches!(refused, Err(Error::Term(TermError::EmptyTermText))), + "{text:?}: {refused:?}" + ); + } + let stored = store(&cipher, &plan, "N").await; + assert!( + stored.positions().is_empty(), + "a one-letter value is still the empty filter" + ); + } + + /// A person with one name under each index: the short-word index on + /// `name`, the default on `alias`. + struct Person { + name: String, + alias: String, + } + impl Fields for Person { + fn field_names(&self) -> Vec<&str> { + vec!["name", "alias"] + } + fn schema() -> Option> { + Some(vec![ + FieldSchema::of::("name"), + FieldSchema::of::("alias"), + ]) + } + } + impl Field for Person { + fn field(&self, name: &str) -> Option<&String> { + match name { + "name" => Some(&self.name), + "alias" => Some(&self.alias), + _ => None, + } + } + } + + /// The option is part of the index: asking a field for a match index + /// under the other setting is refused, as a mismatched `k` is. + #[tokio::test] + async fn a_query_under_other_options_than_the_field_declares_is_refused() { + let cipher = stack_cipher().await; + let plan: Plan = Plan::context("people") + .fields() + .encrypt_index::("name", Match::::new()) + .encrypt_index::("alias", Match) + .build() + .expect("a valid plan"); + let name = plan.field("name").expect("declared"); + let alias = plan.field("alias").expect("declared"); + assert_eq!( + plan_error(cipher.query(op::Matches, "Mike").on(&name).await), + PlanError::IndexOptions { + field: "name".into(), + declared: IndexSpec::Match(ShortWords::options()), + asked: IndexSpec::Match(MatchOptions::default()), + }, + "the default index is not the field's" + ); + assert_eq!( + plan_error( + cipher + .query(op::Matches, "Mike") + .via(Match::::new()) + .on(&alias) + .await + ), + PlanError::IndexOptions { + field: "alias".into(), + declared: IndexSpec::Match(MatchOptions::default()), + asked: IndexSpec::Match(ShortWords::options()), + }, + "nor is the short-word index a default field's" + ); + let found: MatchQuery = cipher + .query(op::Matches, "Ng") + .via(Match::::new()) + .on(&name) + .await + .expect("the field's own options answer"); + let mut row = cipher + .encrypt(&Person { + name: "Mike Ng".into(), + alias: "Mike Ng".into(), + }) + .using(&plan) + .await + .expect("stores"); + let stored: Encrypted> = row.take("name").expect("a name"); + assert!(stored.terms.matches(&found), "and find the row"); + } + + #[tokio::test] + async fn options_the_index_cannot_use_are_refused_stored_and_queried() { + let cipher = stack_cipher().await; + let plan: MatchPlan = Plan::context("people/name") + .with::(Match::::new()) + .build() + .expect("the plan does not read the options"); + let stored = cipher.encrypt(&"Mike Ng".to_owned()).using(&plan).await; + assert!( + matches!(stored, Err(Error::Term(TermError::InvalidOptions(_)))), + "stored: {stored:?}" + ); + let queried = cipher + .query(op::Matches, "Mike Ng") + .via(Match::::new()) + .on(&plan) + .await; + assert!( + matches!(queried, Err(Error::Term(TermError::InvalidOptions(_)))), + "queried: {queried:?}" + ); + } +} + +/// proptest's defaults, which read `PROPTEST_CASES` and +/// `PROPTEST_RNG_SEED`, with a fixed seed when none is set, so CI runs the +/// same cases every time. +fn config() -> Config { + let mut config = Config { + cases: 64, + ..Config::default() + }; + if config.rng_seed == RngSeed::Random { + config.rng_seed = RngSeed::Fixed(0x5d0f_1396); + } + config.rng_algorithm = RngAlgorithm::ChaCha; + config +} + +/// `proptest!` runs each case through a plain closure, so a property +/// builds one runtime and `block_on`s its case. +fn runtime() -> tokio::runtime::Runtime { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() +} + +/// A word of one to five characters, Latin and not, so words fall either +/// side of the short-word length and length is counted in characters. +fn word() -> impl Strategy { + prop::collection::vec( + prop::sample::select(vec!['a', 'b', 'n', 'g', 'é', '李']), + 1..=5, + ) + .prop_map(|chars| chars.into_iter().collect()) +} + +/// A separator the short-word filter splits on. +fn separator() -> impl Strategy { + prop::sample::select(vec![' ', ',', ';', ':', '!']) +} + +mod given_random_text { + use super::*; + + proptest! { + #![proptest_config(config())] + + /// The stored filter of a text holds the query filter of any run + /// of its whole words: every n-gram of the run is an n-gram of the + /// text, and every short word of the run a word of it. + #[test] + fn the_stored_filter_holds_any_run_of_its_whole_words( + words in prop::collection::vec(word(), 1..8), + separators in prop::collection::vec(separator(), 8), + run in (0usize..8, 0usize..8), + ) { + let mut text = String::new(); + let mut spans = Vec::with_capacity(words.len()); + for (i, word) in words.iter().enumerate() { + if i > 0 { + text.push(separators[i - 1]); + } + spans.push((text.len(), text.len() + word.len())); + text.push_str(word); + } + let (first, last) = (run.0 % words.len(), run.1 % words.len()); + let (first, last) = (first.min(last), first.max(last)); + let asked = &text[spans[first].0..spans[last].1]; + + runtime().block_on(async { + let cipher = stack_cipher().await; + let plan = plan(); + let stored = store(&cipher, &plan, &text).await; + match query(&cipher, &plan, asked).await { + Ok(query) => { + prop_assert!( + stored.matches(&query), + "{text:?} holds the filter of {asked:?}" + ); + prop_assert!(query.verify(&text), "{text:?} verifies {asked:?}"); + } + // A run of one one-letter word cuts nothing. + Err(Error::Term(TermError::EmptyTermText)) => { + prop_assert!( + asked.chars().count() < 2, + "only a run too short for any token cuts nothing: {asked:?}" + ); + } + Err(error) => prop_assert!(false, "{asked:?}: {error}"), + } + Ok(()) + })?; + } + } +} + +/// The false positives of a one-token query: a row whose filter holds the +/// query's bits without holding its token. +mod calibration { + use super::*; + + /// Rows of three words of four to six letters: a filter of about + /// fifteen trigrams and no short word, under `k = 3`, `m = 256`. + const ROWS: usize = 3000; + /// Queries of each kind. + const QUERIES: usize = 60; + + fn draw(runner: &mut TestRunner, strategy: S) -> S::Value { + strategy.new_tree(runner).expect("draws a value").current() + } + + /// The share of `rows` whose filter matches `query` falsely, over the + /// rows that do not hold its text. + fn false_positive_rate( + rows: &[(String, MatchTerms)], + query: &MatchTerms, + holds: impl Fn(&str) -> bool, + ) -> (usize, usize) { + let negatives = rows.iter().filter(|(text, _)| !holds(text)); + let mut seen = 0; + let mut false_positives = 0; + for (_, filter) in negatives { + seen += 1; + if filter.contains(query) { + false_positives += 1; + } + } + (false_positives, seen) + } + + /// A one-word query is one token, as a one-trigram query is: it sets + /// the same number of bits by the same fold of a PRF block, so a row + /// holds it falsely as often. Measured over a fixed random column: the + /// two rates agree, and both agree with the rate the filters' fill + /// predicts. A query of five letters, "ab cd", is the control: three + /// trigrams and two words set five times the bits, its rate is far + /// lower, so the measurement can tell a different profile apart. + #[tokio::test] + async fn a_one_word_query_has_the_false_positive_profile_of_a_one_trigram_query() { + let cipher = stack_cipher().await; + let keyset = cipher.default_keyset(); + let mut runner = TestRunner::new_with_rng( + Config::default(), + TestRng::from_seed(RngAlgorithm::ChaCha, &[7; 32]), + ); + let derive = |text: String| { + let keyset = &keyset; + async move { + let filter = keyset + .match_terms::(&text, nonempty!("people/name")) + .await + .expect("derives"); + (text, filter) + } + }; + + let mut rows = Vec::with_capacity(ROWS); + for _ in 0..ROWS { + let words = draw(&mut runner, prop::collection::vec("[a-z]{4,6}", 3)); + rows.push(derive(words.join(" ")).await); + } + let fill: f64 = rows + .iter() + .map(|(_, filter)| (filter.positions().len() as f64 / 256.0).powi(3)) + .sum::() + / ROWS as f64; + + let mut queries = Vec::new(); + for _ in 0..QUERIES { + queries.push(("word", draw(&mut runner, "[a-z]{2}"))); + queries.push(("trigram", draw(&mut runner, "[a-z]{3}"))); + queries.push(("five letters", draw(&mut runner, "[a-z]{2} [a-z]{2}"))); + } + + let mut totals = std::collections::BTreeMap::<&str, (usize, usize, usize)>::new(); + for (kind, text) in queries { + let (_, filter) = derive(text.clone()).await; + // No row holds a short word, so every match of a query with one + // is false; a trigram is held by the rows that contain it. + let (false_positives, seen) = match kind { + "trigram" => false_positive_rate(&rows, &filter, |row| row.contains(text.as_str())), + _ => false_positive_rate(&rows, &filter, |_| false), + }; + let total = totals.entry(kind).or_default(); + total.0 += false_positives; + total.1 += seen; + total.2 += filter.positions().len(); + } + let rate_of = |kind: &str| { + let (false_positives, seen, _) = totals[kind]; + false_positives as f64 / seen as f64 + }; + let bits_of = |kind: &str| totals[kind].2 as f64 / QUERIES as f64; + + let (word, trigram, five) = (rate_of("word"), rate_of("trigram"), rate_of("five letters")); + assert!( + (bits_of("word") - bits_of("trigram")).abs() < 0.2, + "a word and a trigram set as many bits: {} and {}", + bits_of("word"), + bits_of("trigram") + ); + assert!( + (word - trigram).abs() <= 0.25 * trigram, + "a word query's false-positive rate {word:.5} is a trigram query's {trigram:.5}" + ); + for (kind, rate) in [("word", word), ("trigram", trigram)] { + assert!( + (rate - fill).abs() <= 0.25 * fill, + "the {kind} rate {rate:.5} is the rate the fill predicts, {fill:.5}" + ); + } + assert!( + five < word / 5.0, + "the control: five letters, three trigrams and two words, match falsely far less ({five:.5} against {word:.5})" + ); + } +} diff --git a/packages/stack-encrypt/tests/op.rs b/packages/stack-encrypt/tests/op.rs index 2a682d160..7aff4da88 100644 --- a/packages/stack-encrypt/tests/op.rs +++ b/packages/stack-encrypt/tests/op.rs @@ -944,7 +944,9 @@ async fn verify_is_a_token_test_not_a_substring_test() { ); } -/// EQL v4's match parameters, with a notes column's prior. +/// EQL v4's filter size and slices (`k = 6`, `m = 2048`; its match types +/// also turn on short words, which this prior's arithmetic does not need), +/// with a notes column's prior. struct Notes; impl MatchConfig for Notes { fn options() -> MatchOptions { diff --git a/packages/stack-encrypt/tests/plan_grammar.rs b/packages/stack-encrypt/tests/plan_grammar.rs index 1fc6067f7..25889f4bb 100644 --- a/packages/stack-encrypt/tests/plan_grammar.rs +++ b/packages/stack-encrypt/tests/plan_grammar.rs @@ -1036,6 +1036,7 @@ impl MatchConfig for Words { MatchOptions { tokenizer: Tokenizer::Standard, downcase: false, + short_words: false, k: 4, m: 512, } diff --git a/packages/stack-encrypt/tests/term_bytes.rs b/packages/stack-encrypt/tests/term_bytes.rs index 98d72cb82..ae5f5729e 100644 --- a/packages/stack-encrypt/tests/term_bytes.rs +++ b/packages/stack-encrypt/tests/term_bytes.rs @@ -400,3 +400,44 @@ async fn every_integer_width_orders_by_its_own_key() { } width!(u16, u32, u64, u128, i16, i32, i64, i128); } + +/// Short-word tokens under a 65536-bit filter: the word domain +/// (`stack-encrypt/sem/match/word/v1`) and its framing, pinned. "ng" alone is +/// the word's token and nothing else; "Mike Ng" is its trigrams and the same +/// word, so it holds every one of "ng"'s positions. +struct WideShortWords; + +impl MatchConfig for WideShortWords { + fn options() -> MatchOptions { + MatchOptions { + short_words: true, + m: 65536, + ..Default::default() + } + } +} + +#[tokio::test] +async fn short_word_positions_are_pinned() { + let cipher = cipher().await; + let keyset = cipher.default_keyset(); + let word = keyset + .match_terms::("Ng", nonempty!("users/name")) + .await + .unwrap(); + assert_eq!( + word.positions(), + PINNED_NG, + "the word token's positions are frozen" + ); + let whole = keyset + .match_terms::("Mike Ng", nonempty!("users/name")) + .await + .unwrap(); + assert!( + whole.contains(&word), + "\"Mike Ng\" carries the word token of \"Ng\"" + ); +} + +const PINNED_NG: [u16; 3] = [47881, 55268, 60379];