ORE v2: variable-length & string ORE, v2 wire format, SIMD + constant-time hardening (integration branch)

.github/workflows/test.yml

@@ -3,9 +3,10 @@ on:
   push:
     branches:
       - main
+  # Run on every pull request regardless of base branch, so stacked PRs that
+  # target a feature/integration branch (e.g. the ore-v2 stack) get CI too —
+  # not only PRs into `main`.
   pull_request:
-    branches:
-      - main
   workflow_dispatch:
 
 defaults:

docs/reviews/2026-06-16-constant-time-analysis-pr79.md

@@ -0,0 +1,100 @@
+# Constant-Time Analysis — ore-rs (PR #79 sweep, with v2 applicability)
+
+**Date:** 2026-06-16
+**Skill:** Trail of Bits `constant-time-testing` (Phase 1 — static analysis)
+**Primary scope:** `feat/ore-v2-core-refactor` (#79) — the legacy `bit2`/`Bit8`
+scheme (`KnuthShufflePRP` + `Aes128Prng`).
+**Cross-checked against:** `feat/ore-v2-chained` (#83) — the new v2 schemes
+(`bit2_w6`/Bit6 + chained), which use `LemireFyPrp<64>`.
+
+> Phase 2 (dudect statistical) and Phase 3 (timecop/Valgrind dynamic) were **not**
+> run: Valgrind has no Apple-Silicon support, and dudect would require adding a
+> dev-dependency + a CI-pinned core for stable signal. The static pass below is
+> conclusive for the structural channels; dudect would empirically confirm CT-1.
+
+## Method
+
+Manual review of the comparator and permutation paths for the three classic
+constant-time violation classes: secret-dependent branches, secret-indexed
+array/memory accesses, and data-dependent loop bounds. Targets: `scheme/bit2.rs`
+(comparator + encrypt), `primitives/prp.rs`, `primitives/prp/prng.rs`,
+`scheme/width.rs`.
+
+## What is correct (legacy and v2)
+
+The comparator's first-differing-block **prefix scan is genuinely constant-time**
+(`bit2.rs:240-247` on #79): it runs all `num_blocks` with `ct_eq` +
+`conditional_assign`, **no early exit**, updating `l`/`is_equal` obliviously. The
+branch *after* the scan (`if is_equal { return Equal }`) is on the comparison
+**result**, which ORE reveals by design — not a secret-dependent timing branch.
+
+## Findings (legacy `bit2`/Bit8 on #79)
+
+| ID | Location (#79) | Class | Secret leaked | Severity |
+|----|----------------|-------|---------------|----------|
+| **CT-1** | `prp/prng.rs:54` `gen_range` rejection loop, driving the shuffle at `prp.rs:43` | data-dependent loop count | PRP seed = `PRF₂(x[0..n])` → **plaintext via encrypt timing** | High |
+| **CT-2** | `prp.rs:64` `permute` → `inverse.get(input)` (and `invert` → `permutation.get`) | secret-indexed table read | plaintext byte `x[n]` | Med-High |
+| **CT-3** | `prp.rs:44` `permutation.swap(i, j)` (j from `gen_range`) | secret-indexed array write | PRP seed (via keystream `j`) | Medium |
+| **CT-4** | `bit2.rs:261` & `:323` `get_bit(block, xt[l])` | data-dependent array index | `xt[l]` (compare-side) | Medium |
+| **CT-5** | `bit2.rs:258,260,261` accesses indexed by `l` | index by first-differing block | common-prefix length `l` | Low / by-design |
+
+Notes:
+- **CT-1** is the "shuffle that leaked time": `gen_range` loops `next_byte()` until
+  `candidate <= max`, and `next_byte` triggers a fresh 16-block AES `generate()`
+  every 256 draws — so rejection count *and* re-keying frequency depend on the
+  key/plaintext-derived keystream.
+- **CT-2** is mislabeled in source: `permute`/`invert` carry an "in constant time"
+  comment (`prp.rs:58,70`) but perform plaintext-indexed lookups into a 256-byte
+  (= 4 cache-line) table.
+- **CT-5** is the inherent ORE *online* leakage (common-prefix length), not a defect.
+
+## Applicability to the new v2 schemes (Bit6 + chained, `LemireFyPrp<64>`)
+
+Verified against `feat/ore-v2-chained` (#83).
+
+| Finding | Applies to v2? | Evidence |
+|---------|----------------|----------|
+| **CT-1** rejection-sampling encrypt-timing | **No — fixed** | `LemireFyPrp` uses fixed-draw Fisher-Yates + Lemire multiply-high; **no rejection loop, no `%`** (`prp.rs:120-130`, `from_stream`). Still present in Bit8/`KnuthShufflePRP` (wire-frozen legacy). |
+| **CT-4** compare-side cache-line leak | **No — fixed** | All comparator byte reads route through the oblivious `width::ct_select_byte` (`width.rs:77`): the free `get_bit` in `compare_raw_slices` for bit2/bit2_w6/chained (`bit2.rs:285`, `bit2_w6.rs:294`, `chained.rs:325`) **and** `RightBlock32::get_bit` used by typed `Ord::cmp` (`block_types.rs:45`). The `>> (bit%8)` is a constant-time variable shift. |
+| **CT-2** plaintext-indexed `permute` lookup | **Partially — mitigated** | `permute` still does `inverse[input]` (`prp.rs:255`), but `#[repr(C, align(64))]` + `N=64` places `inverse` on exactly one 64-byte cache line → cache-line-constant access. Residual: MemJam 4-byte sub-line only. |
+| **CT-3** secret-indexed FY swap (keygen) | **Partially — mitigated** | `permutation.swap(i, j)` (secret `j`) and `inverse[val]=…` (secret `val`) still exist (`prp.rs:211-215`); each `[u8;64]` table is one cache line under `align(64)`, so writes stay line-uniform. Same MemJam sub-line residual. |
+| **CT-5** index by first-differing block | **Yes — by design** | Inherent ORE online leakage; applies to every scheme. Not a defect. |
+
+> **Superseded (2026-10-09):** the CT-2 and CT-3 residuals above are closed. Table reads are oblivious, and a timing study found the Fisher–Yates swap leaking through timing even without a co-tenant, so key generation is now register-resident with no secret address. See the crypto review brief, A4, "Status (2026-10-09)".
+
+## Bottom line
+
+- The two **genuinely actionable** legacy findings — **CT-1** (encrypt-timing) and
+  **CT-4** (compare-side cache leak) — **do not apply to the new v2 schemes.** They
+  are precisely what the `LemireFyPrp` switch and the `ct_select_byte` hardening
+  were built to close.
+- What remains in v2 (**CT-2 / CT-3**) is **structural-but-mitigated**: the PRP still
+  performs secret-indexed table accesses, reduced from cache-line leaks to the
+  **MemJam sub-cache-line** residual (Intel-SMT co-resident attacker only;
+  ARM/AMD/non-SMT unaffected). This is the documented, deliberate default, with
+  **oblivious-swap-FY** as the wire-compatible high-assurance opt-in.
+
+## Caveats / follow-ups
+
+1. **The mitigation is load-bearing on `LemireFyPrp` staying at `N ≤ 64`.** If a
+   future width pushed it past 64 — or a `repr` change reordered the
+   `permutation`/`inverse` fields — CT-2/CT-3 would silently regress from
+   MemJam-only back to full cache-line leaks. The `const _: () = assert!(domain <= 64)`
+   (`prp.rs:176`) guards the width; **nothing guards against a `repr` change.**
+   Consider a comment / test pinning the field layout assumption.
+2. **Correct the legacy "constant time" comment.** The
+   "performs the inverse permutation in constant time" comment is fair for
+   `LemireFyPrp<64>` at cache-line granularity, but the **identical comment on
+   `KnuthShufflePRP<256>` (`prp.rs:83` on #83 / `:70` on #79) is inaccurate** — the
+   256-byte, unaligned table is a real 4-line lookup. Qualify or remove it so it
+   doesn't imply a guarantee the legacy type doesn't provide.
+3. **dudect (Phase 2)** would empirically confirm CT-1 on the legacy scheme
+   (fixed-vs-random plaintext encrypt timing, Welch's t-test). Optional; needs a
+   `dudect-bencher` dev-dep and a pinned core.
+
+## Related
+
+- Zeroize audit of the same crate: `docs/reviews/2026-06-16-zeroize-audit-pr79.md`
+  (ZA-0001: `Aes128Prng` keystream not wiped on drop — a memory-hygiene, not
+  timing, issue, but in the same `prp/prng.rs`).
+- v2 architecture / leakage model: `docs/plans/2026-06-12-ore-v2-architecture.md`.

docs/reviews/2026-06-16-mmo-plpgsql-validation.md

@@ -0,0 +1,84 @@
+# Validation: v2 σ-MMO comparison in pl/pgSQL + pgcrypto
+
+**Date:** 2026-06-16
+**Question:** can the v2 (Bit6/chained) σ-MMO comparison function be implemented
+in pl/pgSQL / SQL using only the `pgcrypto` extension, as an interim before the
+Rust TLE lands? (The legacy bit2 comparator already lives in
+`cipherstash/encrypt-query-language`; domain types separate the operators, so
+this work targets the **new v2/MMO scheme**.)
+**Answer: the σ-MMO 1-bit hash, the one primitive the comparator needs beyond
+plain SQL, is implementable and validated empirically (12/12 vectors match
+Rust). The full comparator is not implemented or tested here** (see Scope of
+the result).
+
+## Key finding: the comparison needs no CMAC / PRP / key derivation
+
+Those are all **encryption-side** (client/Rust). The comparator (`compare_views`
+in `scheme/chained.rs`; the Bit6/bit2 `compare_raw_slices`) uses only:
+
+1. a byte-equality prefix scan over `xt` + the 16-byte `f` tags,
+2. the σ-MMO 1-bit hash of the first differing block's left tag `f[l]` under the
+   stored nonce, and
+3. an oblivious bit read of the right block at index `xt[l]`.
+
+So `pgcrypto` not providing CMAC is irrelevant — CMAC only derives per-block
+secrets during encryption.
+
+## Primitive mapping
+
+| Need | Implementation |
+|---|---|
+| π (fixed public-key AES-128) | `encrypt(m, 'ORE-rs.v2.H-pi.1'::bytea, 'aes-ecb/pad:none')` (pgcrypto) — `pad:none` for a raw single block; the `PI_KEY` is public, so it's embedded |
+| σ = GF(2¹²⁸) doubling | ~10 lines of pl/pgSQL: byte-wise shift-left-1 + conditional `# 0x87` (`gf128_double` below) — the only non-pgcrypto-native op |
+| `m = σ(f) ⊕ nonce`, feedforward, output bit | byte XOR (`get_byte`/`set_byte`/`#`) + `get_byte(·, 0) & 1` |
+| prefix scan, bit read | plain SQL (`substring`, `get_bit`) |
+
+## Empirical result
+
+The pl/pgSQL `mmo_hash_bit(f, nonce)` (see
+`2026-06-16-mmo-plpgsql-validation.sql`) was checked against ground-truth from
+the Rust `FixedPiZ2Hash::hash` on 12 `(f, nonce)` vectors (H mix
+`1,0,0,0,0,1,0,0,0,0,1,0`):
+
+```
+ total | matches | mismatches
+    12 |      12 |          0
+```
+
+Byte-for-byte agreement. (Postgres 15.12 + pgcrypto.)
+
+**The output bit is a fixed coordinate, not the integer LSB.** The construction
+is written `LSB(π(m) ⊕ m)`, but both Rust (`block[0] & 1`) and this SQL take
+bit 0 of byte 0 of the 16-byte block, which for the big-endian field element is
+bit 120. They agree with each other and with the frozen legacy bit2 hash, which
+uses the same coordinate; any fixed output bit is equivalent for security. The
+spec now names the coordinate rather than calling it the LSB.
+
+## Scope of the result
+
+Only `mmo_hash_bit` was implemented and checked. The rest of a comparator —
+header and length handling, the prefix scan over `xt` and the `f` tags, the
+shorter-sorts-first rule for prefixes, and the right-block bit read at
+`xt[l]` — is plain SQL in principle (`substring`, `get_bit`, byte equality),
+but none of it is implemented or executed here. A comparator built from these
+pieces needs end-to-end comparison vectors (pairs of ciphertexts with their
+expected order, including equal values, shared prefixes and different
+lengths) before it can be called correct.
+
+## Caveat — constant-time
+
+A pl/pgSQL comparator built on this would not be constant-time: pl/pgSQL is
+**not a constant-time environment**, and the cache-line/sub-line obliviousness the Rust
+comparator gets from `ct_select_byte`/`ct_bit` and the no-early-exit prefix scan
+is not realistically achievable in SQL. The interim SQL comparator therefore
+trades that hardening away; the Rust TLE restores it. Whether that's acceptable
+is a threat-model call for the interim (server-side comparison over stored
+ciphertexts) — flagged, not decided here.
+
+## Conclusion
+
+The blocking question is answered: the only cryptographic primitive the v2
+comparator needs, the σ-MMO 1-bit hash, is implementable in pl/pgSQL with
+pgcrypto and matches Rust. The comparator itself remains to be written and
+validated end to end. The encryption-side work (single-key derivation, CMAC accumulator, PRP)
+is unaffected by it. Safe to resume the single-key change + Bit6 vector regen.

docs/reviews/2026-06-16-mmo-plpgsql-validation.sql

@@ -0,0 +1,55 @@
+-- GF(2^128) doubling σ(x) = 2·x, reduction poly x^128+x^7+x^2+x+1 (0x87),
+-- big-endian 16-byte field element. byte-wise shift-left-1 + conditional XOR.
+CREATE OR REPLACE FUNCTION gf128_double(x bytea) RETURNS bytea AS $$
+DECLARE r bytea := x; carry int := 0; nc int; i int; topbit int;
+BEGIN
+  topbit := (get_byte(x,0) >> 7) & 1;
+  FOR i IN REVERSE 15..0 LOOP
+    nc := (get_byte(r,i) >> 7) & 1;
+    r := set_byte(r, i, ((get_byte(r,i) << 1) | carry) & 255);
+    carry := nc;
+  END LOOP;
+  IF topbit = 1 THEN
+    r := set_byte(r, 15, get_byte(r,15) # 135);  -- ^ 0x87
+  END IF;
+  RETURN r;
+END; $$ LANGUAGE plpgsql IMMUTABLE;
+
+CREATE OR REPLACE FUNCTION xor16(a bytea, b bytea) RETURNS bytea AS $$
+DECLARE r bytea := a; i int;
+BEGIN
+  FOR i IN 0..15 LOOP r := set_byte(r, i, get_byte(a,i) # get_byte(b,i)); END LOOP;
+  RETURN r;
+END; $$ LANGUAGE plpgsql IMMUTABLE;
+
+-- BHKR σ-MMO 1-bit hash: m = σ(f) ⊕ nonce; H = lsb(π(m)) ⊕ lsb(m).
+-- "lsb" is a fixed output coordinate, not the integer LSB: bit 0 of byte 0
+-- of the 16-byte block, i.e. bit 120 of the big-endian field element. This
+-- matches the Rust `block[0] & 1` (and the frozen legacy bit2 hash).
+-- π = AES-128-ECB under the fixed public key "ORE-rs.v2.H-pi.1".
+CREATE OR REPLACE FUNCTION mmo_hash_bit(f bytea, nonce bytea) RETURNS int AS $$
+DECLARE m bytea; pim bytea;
+BEGIN
+  m := xor16(gf128_double(f), nonce);
+  pim := encrypt(m, 'ORE-rs.v2.H-pi.1'::bytea, 'aes-ecb/pad:none');
+  RETURN (get_byte(pim,0) & 1) # (get_byte(m,0) & 1);
+END; $$ LANGUAGE plpgsql IMMUTABLE;
+
+WITH v(data, nonce, expected) AS (VALUES
+  ('\x00070e151c232a31383f464d545b6269'::bytea,'\x010e1b2835424f5c697683909daab7c4'::bytea,1),
+  ('\x1f262d343b424950575e656c737a8188'::bytea,'\x121f2c394653606d7a8794a1aebbc8d5'::bytea,0),
+  ('\x3e454c535a61686f767d848b9299a0a7'::bytea,'\x23303d4a5764717e8b98a5b2bfccd9e6'::bytea,0),
+  ('\x5d646b727980878e959ca3aab1b8bfc6'::bytea,'\x34414e5b6875828f9ca9b6c3d0ddeaf7'::bytea,0),
+  ('\x7c838a91989fa6adb4bbc2c9d0d7dee5'::bytea,'\x45525f6c798693a0adbac7d4e1eefb08'::bytea,0),
+  ('\x9ba2a9b0b7bec5ccd3dae1e8eff6fd04'::bytea,'\x5663707d8a97a4b1becbd8e5f2ff0c19'::bytea,1),
+  ('\xbac1c8cfd6dde4ebf2f900070e151c23'::bytea,'\x6774818e9ba8b5c2cfdce9f603101d2a'::bytea,0),
+  ('\xd9e0e7eef5fc030a11181f262d343b42'::bytea,'\x7885929facb9c6d3e0edfa0714212e3b'::bytea,0),
+  ('\xf8ff060d141b222930373e454c535a61'::bytea,'\x8996a3b0bdcad7e4f1fe0b1825323f4c'::bytea,0),
+  ('\x171e252c333a41484f565d646b727980'::bytea,'\x9aa7b4c1cedbe8f5020f1c293643505d'::bytea,0),
+  ('\x363d444b525960676e757c838a91989f'::bytea,'\xabb8c5d2dfecf90613202d3a4754616e'::bytea,1),
+  ('\x555c636a71787f868d949ba2a9b0b7be'::bytea,'\xbcc9d6e3f0fd0a1724313e4b5865727f'::bytea,0)
+)
+SELECT count(*) AS total,
+       count(*) FILTER (WHERE mmo_hash_bit(data,nonce) = expected) AS matches,
+       count(*) FILTER (WHERE mmo_hash_bit(data,nonce) <> expected) AS mismatches
+FROM v;