Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions packages/stack-encrypt/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
copies of the text. An exact term has no `verify`, and none of this
crate's indexes can answer `op::Matches` but with a probabilistic term
(`op::TermFor`).
- **`target_limit` says how many rows a match query should fetch.**
`query.target_limit(want, &stats)` is `want` plus the false positives
expected at the confidence `stats` names (99% by default), as an
`op::Limit`. `op::Stats::declared(rows)` uses the new
`MatchConfig::expected_tokens()` prior; `op::Stats::measured(rows,
histogram)` uses a histogram of the stored filters' set-bit counts. With
neither, the limit is `Limit::Unbounded`, and the caller pages with a
cursor. A limit never exceeds `op::MAX_ROWS` (`i64::MAX`), the largest
SQL `LIMIT`. The bound assumes a non-matching row shares no token with
the query, so on natural text its confidence is nominal; `verify` still
decides every row.

- **Errors now expose stable codes, help text, and structured details.**
All error types implement `miette::Diagnostic` with codes such as
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -234,8 +234,7 @@ pub trait QueryTerm {}
pub trait Probabilistic: QueryTerm {
type Plaintext: ?Sized;
fn verify(&self, value: &Self::Plaintext) -> bool;
// `target_limit(&self, want, &Stats) -> Limit` joins it with decision 5
// (#1184).
fn target_limit(&self, want: u64, stats: &Stats) -> Limit;
}

pub trait Ranked: QueryTerm {
Expand Down Expand Up @@ -336,6 +335,15 @@ positives); the mean fill, 0.25, raised to 6 predicts 245. The number the
bound needs is therefore `p = mean(f_i^b)`, and the statistics the client
holds must let it compute that for the `b` of each query.

The model assumes a non-matching row shares no token with the query. The
same token sets the same bits in every filter, so a row holding part of the
query has fewer bits left to chance: "police" already holds two of
"alice"'s three 3-grams, and passes at `f^6` rather than `f^18`. On natural
text the stated confidence is therefore nominal, and a measured histogram
cannot correct it, because it measures fill, not overlap. Correctness does
not depend on it: `verify` decides every row, and the candidate count says
whether a fetch was complete.

False positives are then `Poisson(λ = p·N)`, and the worst case that still
satisfies the caller is exactly `want` true matches preceded by every false
positive, so
Expand All @@ -362,9 +370,10 @@ index declaration. The library takes either as numbers in `Stats` and does
not care which it was given. A column has the prior from the day it is
declared.

**Declared.** A match index declares `expected_tokens`, the typical token
count of the values it will hold, beside `m` and `k`: 20 for a name column,
200 for a notes column. From it `Stats::declared(rows)` takes
**Declared.** A match configuration declares `expected_tokens()`, the typical
token count of the values it will hold, beside the `m` and `k` of its
options: 20 for a name column, 200 for a notes column. It is not part of the
index's identity and changes no term. From it `Stats::declared(rows)` takes
`f = 1 − e^(−k·expected_tokens/m)` and `p = f^b`. This is a prior, and
`rows` is a planning estimate too. It is enough to warn before a query runs
(a three-letter query against a notes column) and to size a first fetch; it
Expand All @@ -378,6 +387,7 @@ fills. One aggregate gives it, at most `m` buckets:
```sql
SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*)
FROM users TABLESAMPLE SYSTEM (1)
WHERE name IS NOT NULL
GROUP BY 1
```

Expand All @@ -387,8 +397,18 @@ query has. The sample bounds the cost on a large table; `TABLESAMPLE` is a
page sample, which is fine for a distribution and unsuited to an exact count.
It is also a sample of pages, so on a table of a few thousand rows a 1%
sample can select none; below that size the clause is dropped and the whole
table is read, which costs nothing worth saving. `rows` comes from `SELECT reltuples FROM pg_class WHERE oid =
'users'::regclass`, a catalog read.
table is read, which costs nothing worth saving. `rows` comes from
`SELECT greatest(reltuples, 0) FROM pg_class WHERE oid = 'users'::regclass`,
a catalog read; `reltuples` is −1 for a table that has never been vacuumed
or analysed, hence the `greatest`. A `Limit` never exceeds `i64::MAX`, the
largest `LIMIT` PostgreSQL takes. An empty histogram falls back to the declared prior.

`target_limit` returns a `Limit`: `Rows(n)`, or `Unbounded` when the
statistics say nothing about the stored filters (no prior and no
histogram) or the confidence is not below 1. `fetch_or(page)` reads either
as a fetch size, for a caller that pages anyway. Past ten million expected
false positives the quantile is the conservative `λ + 8√λ`, since the query
is hopeless at that point.

Where the numbers live and who keeps them in step was asked on the review,
so it is written down. They are read from the server by the application, at
Expand Down Expand Up @@ -454,7 +474,7 @@ shot, with a bound, and the completeness check from decision 5:

```rust
let term = cipher.query(op::Matches, "ali").on(&name_plan).await?;
let fetch = term.target_limit(10, &stats).fetch();
let fetch = term.target_limit(10, &stats).fetch_or(50);

let rows: Vec<UserRow> = sqlx::query_as(
"SELECT id, name FROM users WHERE name @@ $1 ORDER BY id LIMIT $2",
Expand Down Expand Up @@ -715,7 +735,7 @@ the operator is `@@`, column on the left. The query selects the ciphertext

```rust
let term = cipher.query(op::Matches, "ali").on(&name_plan).await?;
let fetch = term.target_limit(10, &stats).fetch();
let fetch = term.target_limit(10, &stats).fetch_or(50);

let rows: Vec<UserRow> = sqlx::query_as(
"SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2",
Expand All @@ -735,14 +755,14 @@ let hits: Vec<User> = cipher.open(rows).using(&users_plan).await?
The `Stats` it used were read once, at startup, by the application:

```rust
let rows: i64 = sqlx::query_scalar("SELECT reltuples::bigint FROM pg_class WHERE oid = 'users'::regclass")
let rows: i64 = sqlx::query_scalar("SELECT greatest(reltuples, 0)::bigint FROM pg_class WHERE oid = 'users'::regclass")
.fetch_one(&pool).await?;
let histogram: Vec<(i16, i64)> = sqlx::query_as(
let histogram: Vec<(i32, i64)> = sqlx::query_as(
"SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*)
FROM users TABLESAMPLE SYSTEM (1) GROUP BY 1",
FROM users TABLESAMPLE SYSTEM (1) WHERE name IS NOT NULL GROUP BY 1",
)
.fetch_all(&pool).await?;
let stats = Stats::measured(rows as u64, histogram.into_iter().map(|(b, n)| (b as u16, n as u64)));
let stats = Stats::measured(rows as u64, histogram.into_iter().map(|(b, n)| (b as u32, n as u64)));
```

or, before anyone has measured, `Stats::declared(1_000_000)`, which uses the
Expand All @@ -762,7 +782,7 @@ let rows: Vec<UserRow> = sqlx::query_as(
)
.bind(sqlx::types::Json(&name))
.bind(sqlx::types::Json(&age))
.bind(name.target_limit(10, &stats).fetch() as i64)
.bind(name.target_limit(10, &stats).fetch_or(50) as i64)
.fetch_all(&pool).await?;
let hits: Vec<User> = cipher.open(rows).using(&users_plan).await?
.into_iter().filter(|u| name.verify(&u.name)).take(10).collect();
Expand Down Expand Up @@ -903,7 +923,7 @@ it binds as a parameter in any library built on `database/sql` or pgx:

```go
q, _ := users.Fields.Name.Matches(ctx, cipher, "ali")
limit := q.TargetLimit(10, stats).Fetch()
limit := q.TargetLimit(10, stats).FetchOr(50)

rows, err := db.QueryContext(ctx,
`SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2`,
Expand Down Expand Up @@ -1009,7 +1029,7 @@ TypeScript programs use:

```ts
// pg
const limit = name.targetLimit(10, stats).fetch()
const limit = name.targetLimit(10, stats).fetchOr(50)
const { rows } = await pool.query(
'SELECT id, name FROM users WHERE name @@ $1::eql_v4.query_text_match ORDER BY id LIMIT $2',
[name, limit],
Expand Down Expand Up @@ -1053,7 +1073,7 @@ const q = await ops.matches(users.name, 'ali') // { sql, verify, target
const rows = await db.select().from(users)
.where(q.sql) // name @@ '…'::eql_v4.query_text_match
.orderBy(users.id)
.limit(q.targetLimit(10, stats).fetch())
.limit(q.targetLimit(10, stats).fetchOr(50))
const hits = (await cipher.decrypt(users, rows)).filter((u) => q.verify(u.name)).slice(0, 10)
```

Expand All @@ -1067,8 +1087,8 @@ application, or from the declared prior:

```ts
const stats = Stats.measured(
(await pool.query("SELECT reltuples::bigint AS rows FROM pg_class WHERE oid = 'users'::regclass")).rows[0].rows,
(await pool.query('SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) FROM users TABLESAMPLE SYSTEM (1) GROUP BY 1')).rows,
(await pool.query("SELECT greatest(reltuples, 0)::bigint AS rows FROM pg_class WHERE oid = 'users'::regclass")).rows[0].rows,
(await pool.query('SELECT cardinality(eql_v4.match_term(name)) AS bits, count(*) FROM users TABLESAMPLE SYSTEM (1) WHERE name IS NOT NULL GROUP BY 1')).rows,
)
const prior = Stats.declared(1_000_000) // uses users.name's expectedTokens
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,9 @@ use std::marker::PhantomData;

use stack_kms::MaybeSend;

pub(crate) mod stats;
pub use stats::{Limit, Stats, DEFAULT_CONFIDENCE, MAX_ROWS};

use crate::sem::{DefaultMatch, EqualityTerm, MatchConfig, MatchQuery, MatchTerms, QueryTokens};
use crate::target::{Equality, Index, Match};
use crate::PlanError;
Expand Down Expand Up @@ -223,6 +226,21 @@ pub trait Probabilistic: QueryTerm {
/// the index was declared over, so every outcome other than a match is
/// `false`, never an error.
fn verify(&self, value: &Self::Plaintext) -> bool;

/// How many candidate rows to fetch so that, at the confidence `stats`
/// names, `want` of them [`verify`](Self::verify), when the table holds
/// that many matches.
///
/// A probabilistic bound, not a guarantee: at 99% confidence one query
/// in a hundred draws more false positives than it allows for, and the
/// model assumes a non-matching row shares no token with the query,
/// which natural text does not honour, so the confidence is nominal.
/// The candidate count says which case a fetch is in. If the server
/// returned fewer rows than the limit, the predicate is exhausted, and
/// fewer than `want` verified rows means fewer than `want` matches. If
/// it returned exactly the limit and fewer than `want` verified, the
/// result is a page: carry on from its last row with a cursor.
fn target_limit(&self, want: u64, stats: &Stats) -> Limit;
}

/// The query terms that may answer the operator `Op`: what
Expand Down
Loading
Loading