Skip to content

feat(f5): the masking prototype (Gate 0) - #213

Merged
SauravYadav12 merged 1 commit into
mainfrom
feat/f5-mask-proto
Nov 5, 2026
Merged

SauravYadav12 merged 1 commit into
mainfrom
feat/f5-mask-proto

Conversation

@SauravYadav12

Copy link
Copy Markdown
Collaborator

F5 Gate 0: masking's prototype, proving the claims of the design agreed in #212. The findings are in docs/FOX_F5_Masking_Design.md, section 17. There is no product code yet; Gate 1 follows.

What it adds

  • bb_identity 0.2: bbid_mask_hmac(domain, value). It is HMAC-SHA-256 under a key domain's key.
    • The key is a 32-byte file at /run/bb-mask/<domain>.key.
      • It must be a regular file of the server's own, readable by nobody else, or it is refused.
      • It is never in a table or a message.
      • It is read again when the file is replaced, so a rotation reaches sessions that read the old key.
    • The domain name is checked as a file name: 1 to 32 of a-z 0-9 _ -.
    • Never granted to PUBLIC, whether the extension is made at 0.2 or updated to it from 0.1. A masked reader able to call it could hash their own guesses.
  • make integration-f5 (scripts/integration_f5.sh). It builds the image under its own tag, and runs throwaway containers plus a ZFS pool of its own made from a file. The shared stack is never touched.
  • docker/postgres/f5_test.go: the function is never public, the key is read from its one fixed place with its checks, and no message can carry key bytes.

What it proves, on PostgreSQL 18 and 16 (57/57)

  • Each rule's mask:

    • a fake name, the same for the same input;
    • a 64-hex keyed hash;
    • partial phone and Aadhaar;
    • an empty column;
    • a JSON path partial, with the rest of the document as it was.
  • Out of a masked reader's reach: the real table, the shadows, the hash, the fake function and the key file.

  • Works on masks: filtering, grouping and joining (a customer's orders by the hashed email), and EXPLAIN. pg_stats is empty for the real table, and COPY of a query copies masks.

  • Writes: none to marked tables. Whatever a masked reader makes runs as them, so their own definer function reading the real table is refused.

  • The lookup: returns masked rows, allows 30 a minute, and records every call without the value.

  • The application: writes, key changes, deletes and TRUNCATE all work as before, and the shadows keep up.

  • The key:

    • two domains hash the same value apart;
    • a missing key, a bad name or a loose key file is refused;
    • the key is in no log;
    • a rotation is seen by a session that read the old key.
  • A masked branch on a real ZFS clone, after:

    • masking in place;
    • VACUUM FULL of the table and of pg_statistic;
    • WAL recycling off, and three switches and checkpoints.

    The branch has no real value left in any of its files, tables, statistics and WAL alike. A fresh cluster of only the masked rows also has none, in 0.3 s for 20,000 rows.

What it found and fixed (now in the design)

  1. A refused lookup put its value in the server log, through log_min_error_statement. Every masked login now gets log_min_error_statement = panic, which only a superuser can set or undo.
  2. VACUUM FULL pg_statistic sent with another statement runs in one transaction and is refused, silently leaving the old statistics in place. Each masked-branch step now runs as its own statement.

Measured (PostgreSQL 18, 16 clients, medians of three)

Without masks With masks
One 10,000-row UPDATE 55 ms 307 ms a row each; 198 ms a statement each
Single-row writes 34,566 tps 27,297 tps (21% fewer; 30% in a first run)
Indexed reads 207,798 tps 155,810 tps through the view (25% fewer)
100 documents of 250 KB, a JSON path masked 1.4 ms 544 ms

For 1,000,000 rows:

  • Filling an existing table's masks: 14 s, without ever holding up a writer.
  • Rotating the key: 21 s.

Over the 10% budget. The design's decision 9 recommends accepting the cost, as for the audit log:

  • it falls on marked tables only, and the read cost on masked readers only;
  • the cost is said before a rule is set;
  • masks are kept with a trigger a statement;
  • a JSON path in documents over 64 KB is warned about;
  • a C trigger is the way to cut the cost further.

That decision is the founders'.

Verified

Suite Result
integration-f5 (new) 57/57
integration-f1 70/70
integration-f6 71/71
integration 765/765

Those last three ran on the image with bb_identity 0.2. go test ./docker/postgres and the licence-header test pass.

bb_identity 0.2 adds bbid_mask_hmac(domain, value): HMAC-SHA-256 under
a key domain's key, a 32-byte file at /run/bb-mask/<domain>.key that
must be the server's own and readable by nobody else; never in a table
or a message; read again when the file is replaced; never granted to
PUBLIC. An upgrade from 0.1 is included.

make integration-f5 builds the image and, on throwaway containers and a
ZFS pool of its own, proves the design's claims on PostgreSQL 18 and 16
(57/57): each rule's mask; what a masked reader can and cannot reach;
the lookup; the application unaffected; key domains and rotation; and a
masked branch on a ZFS clone left with no real value in its files.

It found and fixed two things, now in the design (section 17): a
refused lookup put its value in the server log, so masked logins get
log_min_error_statement = panic; and a masked branch's VACUUM steps must
each run alone. It measured the cost over the 10% budget: writes to a
marked table 21-30% fewer, bulk updates 3.6x with a statement trigger,
masked reads 25% fewer.
@vercel

vercel Bot commented Nov 5, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
foxbyte-get Ready Ready Preview Nov 5, 2026 11:18am UTC

@SauravYadav12
SauravYadav12 merged commit 4acf4b6 into main Nov 5, 2026
8 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 5, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant