Repository navigation
How should I handle one- and two-character substring queries with EQL v4 TextSearch? #1299
Description
Background
Hi! I'm after some guidance. Do you plan to support case-insensitive substring search over encrypted text, including one- and two-character queries? And if not, what would you recommend I do instead?
For context, I'm building on EQL v4 (Encrypt Query Language) through the Go SDK, using TextSearch columns for names.
Problem
What I need is the same rows PostgreSQL gives for value ILIKE '%' || q || '%'. So a search for al should find Al, Alice, Malin, Opal and ALICE. Short searches like this are common for us: I checked our production data (read-only) and one- and two-character matches turn up at the start, middle and end of display names, given names, family names and form names. Exact-value equality doesn't help, since it misses all the longer values.
On next (at a05affa), Matches refuses anything shorter than a 3-gram (three characters) and returns ErrTerm:
cipherstash: term derivation failed: the query's text produces no match tokens (empty, separator-only, or shorter than the n-gram length)
I know this is on purpose. A query with no tokens would match every row (the refusal says so), and an existing Go test checks for it.
As far as I can tell, there are two more things in the way:
- Short stored values get an empty Bloom filter (
"bf": []), and no@@query ever finds them (design record). So allowing short queries wouldn't be enough on its own: rows likeAlandAstill wouldn't come back. Verifychecks that the query's 3-grams all appear in the value, ignoring order and repetition (here). As Rename contains() → matches(): it is fuzzy n-gram token matching, not containment #617 points out, that's not quite a substring test, even for longer queries.
Our backup plan
If short substring search isn't on the cards, our fallback is: when the query is shorter than three characters, use TextSearch's equality query (Eq) instead of Matches. That works today, because = finds short stored values through their equality term. Roughly, with people as the generated code for a struct whose Name is stored as TextSearch:
if utf8.RuneCountInString(q) < 3 {
// Fallback: exact, case-sensitive equality on the whole value.
eq, err := people.Fields.Name.Eq(ctx, cipher, q)
// ...
rows, err = db.QueryContext(ctx,
`SELECT id, name FROM people WHERE name = $1::eql_v4.query_text_search`, eq)
} else {
// Match query: decrypt the rows, then keep those m.Verify accepts.
m, err := people.Fields.Name.Matches(ctx, cipher, q)
// ...
rows, err = db.QueryContext(ctx,
`SELECT id, name FROM people WHERE name @@ $1::eql_v4.query_text_search`, m)
}It's a big step down, though. Equality matches the whole value, and text equality isn't case-folded (design record). With the values above:
| Query | ILIKE returns |
Fallback returns |
|---|---|---|
a |
Al, Alice, Malin, Opal, ALICE, A |
nothing |
A |
same as a |
A |
al |
Al, Alice, Malin, Opal, ALICE |
nothing |
Al |
same as al |
Al |
So it gets users past an error, but most short searches will come back empty or nearly empty.
Proposal
I'm not proposing a fix, just hoping for your take on a couple of things:
- Is substring search like this something you'd like
TextSearchto cover, short queries included? If so, is there a plan or an issue I should follow? - If not, is the equality fallback above the pattern you'd recommend, or is there something better? For example, is a case-insensitive equality type planned (the design record mentions one as its own domain), and what leakage risks should I keep in mind with any fallback? (I've read the every-row-matching concern in core encryptQuery mints every-row-matching terms for 1–2 char free-text match needles (Supabase adapter guards, core doesn't) #697 and the decisions in stack-encrypt: store short TextMatch text as an empty filter, refuse only short queries #1185.)
Activity
coderdan commentedon Oct 26, 2026
Thanks for the detailed write-up, and sorry for the slow reply. Short answer: two-letter words are covered in the next release candidate,
v0.1.0-rc.2. Arbitrary one- and two-character substrings are not, and we don't plan to index them. Details below.What changes in rc.2
#1397 (open, going into rc.2; it closes #1309) adds short-word tokens to the match index, and turns them on for the EQL v4 text types,
TextSearchincluded. Every word that is at least two characters long and shorter than the 3-gram length gets one extra token, the word itself, case-folded like the rest of the index. The 3-grams of the whole value are unchanged.For your two blockers that means:
Alnow stores a filter holding the wordal, so it is no longer"bf": []and@@can find it. So is a longer value containing a two-letter word, such asMike Ng.Matches(ctx, cipher, "al")returns a term instead ofErrTerm, and finds every row that containsalas a whole word, in any case:Al,AL,Al Smith.What it does not do, using your table:
ILIKEreturns@@returnsaAl,Alice,Malin,Opal,ALICE,AErrTerm): a one-letter word gets no tokenalAl,Alice,Malin,Opal,ALICEAlonly: the whole word, not the substring insideAliceorMalinWords are split at the same separators as the standard tokenizer (space, comma, semicolon, colon and exclamation mark). So
Ng, Mikeholds the wordng, butNg.andNg-Smithdon't yet. That is #1399.Why not substrings of one or two characters
To answer
%al%, the index would need a token for every 1- and 2-character run of every value. That is a different class of leakage from what the match index accepts today. With an alphabet that small, the tokens behave like encrypted letters and letter pairs, and their frequencies across a table are close to the plaintext's letter statistics. A whole-word token reveals much less, because it repeats only where the same short word repeats. We looked at general bigrams as part of #1397 and rejected them for that reason.What we'd recommend
Matches, then confirm on the decrypted values.@@narrows the rows; it is token containment, not a substring test (your point from Rename contains() → matches(): it is fuzzy n-gram token matching, not containment #617). If you need exactILIKEsemantics, keep the rows wherestrings.Contains(strings.ToLower(value), strings.ToLower(q))holds after decryption.Matchesas well from rc.2. It finds the whole-word hits, which for names is usually what people mean byNg,LiorAl. If you also want the substring hits, the honest UI is to ask for a third character.Eqfallback works, and you can make it case-insensitive today by also storing the value lowercased in aTextEqcolumn, then querying it with the lowercased input. The leakage to keep in mind is equality's: anyone with the table can see which rows share a value, and for short values that repetition is more telling. Only add it if exact lookup is something your users actually do.We'll comment here when rc.2 is published.
https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
coderdan commentedon Oct 26, 2026
v0.1.0-rc.2is out with the two-letter word tokens described above: https://github.com/cipherstash/stack/releases/tag/languages/golang/v0.1.0-rc.2Two things to know when you move to it:
TextSearchrows need encrypting again to be found by a two-letter query. A row written by rc.1 has no word token, soMatches(ctx, cipher, "al")will not find anAlstored before the upgrade. Three-character and longer queries are unaffected.stashgenagain and re-encrypting columns with order indexes. The release notes list them under "Upgrading from v0.1.0-rc.1".https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu