feat(stack-encrypt): size a match query's fetch with target_limit

[coderdan]

Summary

Stacked on #1191. This PR implements ADR-0009 decision 5: a probabilistic query says how many rows to fetch to expect want real matches.

use stack_encrypt::op::{self, Limit, Probabilistic, Stats};

let stats = Stats::measured(reltuples, histogram);        // or Stats::declared(reltuples)
let query = cipher.query("ali").using(&name_plan).op(op::Matches::new()).await?;
let fetch = query.target_limit(10, &stats).fetch_or(50);  // LIMIT for the SQL

• The method. Probabilistic::target_limit(&self, want, &Stats) -> Limit returns want + Poisson_quantile(λ = p·N, confidence). Here p is the mean over stored filters of f^b, and b is the number of bits the query sets.
• Two sources for the fill.
  • Stats::declared(rows) uses a new MatchConfig::expected_tokens() prior. It is a default method returning None, so existing configurations compile unchanged. It is not part of the index's identity and changes no term.
  • Stats::measured(rows, histogram) takes (set bits, rows) buckets, which one sampled aggregate over the column gives. It takes the mean of the power, never the power of the mean fill. That was the review finding on docs(adr): ADR-0009, operators select the index and a query term is not the stored term #1177, and a test pins it. An empty histogram falls back to the prior.
• Limit. It is Rows(n), or Unbounded when nothing is known of the stored filters or the confidence is not in (0, 1). fetch_or(page) reads either as a fetch size.
• The Poisson quantile. It is a bounded, normalised sum over [λ − 10√λ, λ + 12√λ + 50], which holds all but about 1e-23 of the mass. Each term is kept as a log relative to the first, and the sums run from the top, so it matches an exact reference to within one up to a confidence of 1 − 1e-12. That costs about 22√λ terms. Past λ = 1e7 it returns the conservative λ + 8√λ.
• Limits fit SQL. A Limit never exceeds op::MAX_ROWS (i64::MAX).
• The independence assumption is stated. A row sharing tokens with the needle passes more often, so the confidence is nominal on natural text.

Tests

• Unit tests in op::stats:
  • The quantile matches reference values computed with math.lgamma, from λ = 0.034 to 1e6.
  • ADR-0009's worked table reproduces: want + 1, want + 7805 and want + 3.
  • The mixed-length column gives λ ≈ 3,800, where the mean fill would give under 300.
  • The edge cases are covered: no prior, an empty histogram, out-of-range confidence, a count above m, and overflow of want.
• Integration tests in tests/op.rs:
  • A three-letter needle against a 200-token prior asks for thousands of rows, while a long needle asks for under 20.
  • A stricter confidence asks for more rows.
  • With no prior, target_limit is Unbounded until the column is measured from real stored filters.

Test plan

• cargo nextest run -p stack-encrypt --all-features: 575 passed. The review fixes are in a second commit; see the review comment.
• Doctests, clippy with -D warnings with and without --all-features, and rustdoc with warnings as errors.
• CI, including wasm32 clippy and mutants.

The ADR examples now use fetch_or(page). ADR-0009 stays proposed. #1178 says to mark it accepted once this pins Stats and Limit, and that call is a human's.

Closes #1184

https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB

[changeset-bot]

⚠️ No Changeset found

Latest commit: 850e345

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught	missed	unviable	timeout
89	0	5	0

Every mutant in the changed lines was caught by a test.

[coderdan]

Review (high effort): findings and how each was addressed

An independent review checked 5,220 (λ, confidence) pairs against an exact mpmath reference, plus a port of the mutants. The fixes are in b3dd1f78f, rebased onto #1191's review fix.

1. Some mutants would hang poisson_quantile, and CI counts a hang as a survivor. This was confirmed locally: || → && timed out, and the λ = 1e7 threshold's > → >= survived. Fixed:
   • The quantile is now a bounded sum over [λ − 10√λ, λ + 12√λ + 50], so no mutation can loop.
   • A test at exactly λ = 1e7 kills the threshold mutant.
2. "Exact" was false at high confidence for large λ. Cancellation in −λ + s·ln λ − ln s! cost about 1e-8 in the log. At λ = 4.75e6 and 1 − 1e-12 the answer was low by about 3,000. Fixed:
   • Each term is kept relative to the window's first term, and the sums are normalised.
   • Sums run from the top, where terms are smallest, so the tail a high confidence compares against is not lost to rounding.
   • It now matches the exact reference to within one at 1 − 1e-12, at 0.9999 for λ = 3.4e6, and at λ = 1e7.
   • ln_factorial is gone.
3. A Limit could exceed any SQL LIMIT. reltuples is −1 for a table never analysed, which reads as u64::MAX. Fixed:
   • Limit is capped at op::MAX_ROWS (i64::MAX).
   • The ADR's statements use greatest(reltuples, 0).
4. The model assumes a non-matching row shares no token with the needle. "police" holds two of "alice"'s 3-grams, so it passes at f^6 rather than f^18. Documented in decision 5 and on target_limit: the confidence is nominal on natural text, and verify and the candidate count still decide correctness.
5. Doc fixes.
   • The stats SQL now filters NULL values.
   • target_limit says a fetch that returned the limit is a page only when fewer than want rows verified.
   • expected_tokens counts distinct tokens.
6. API.
   • Histogram counts are summed as floats, so huge counts cannot overflow.
   • The measured-column test now bounds the fetch against the λ computed from its own histogram. The old 1..=4 bound was weak.
   • 15 runs with random keys did not flake.

The reviewer also confirmed these were fine:

• The skipped mass below λ − 10√λ is under 1e-23.
• The conservative branch past 1e7 holds.
• The worked table reproduces, and the mixed-length column gives 3,800 where the mean fill gives 245.
• Edge cases (rows = 0, expected_tokens = 0, a count above m) behave.
• The integration thresholds cannot flake whatever the keys.

575 tests pass locally, with doctests, clippy and rustdoc clean. cargo mutants --in-diff is running over the whole stack.

[coderdan]

Follow-up: the CI mutants gate on b3dd1f78f found one survivor and three timeouts in poisson_quantile. They are fixed in 850e345c8.

• The survivor. Shifting each log term by the window's peak was never needed: relative to the window's first term no term exceeds about e^100. That made - → + there equivalent, so the shift is gone.
• The timeouts. A mutated threshold or window bound sent the MAX_ROWS test's λ of about 9e18 into a window of billions of terms. The window is now 22√λ + 50 wide, capped at 2^20 terms whatever the input.
• A new test pins the window's end as the answer when the confidence allows no mass.

A focused cargo mutants run over op/stats.rs caught everything else. That run left one survivor, the initial answer, and the new test is aimed at it; it was written after the run, so that run has not checked it.

[coderdan]

Another survivor came up from the CI mutants gate on 850e345c8: start..=start + width → start * width. The loop's range bypassed the width cap, so that mutation ran for billions of terms instead of failing.

Fixed in 026759a57: the loop now runs over the capped width and works out each k from its offset.

A focused cargo mutants run over op/stats.rs tested 91 mutants: 87 caught, 4 unviable, none missed, no timeouts.