Skip to content

About

Research POC for masked private inference of public-weight LLMs without trusted hardware

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Veil: private inference for public-weight LLMs

Tests License Python

Veil is a research proof-of-concept for outsourcing the expensive linear layers of an open-weight LLM while keeping prompts, activations, KV state, and generated values private from the server—without trusted hardware.

The key observation is simple: the model weights are public. The client can therefore keep every nonlinearity, attention operation, KV cache, and sampling step locally, and outsource only public linear maps y = W x.

client                              untrusted server
------                              ----------------
private activation x
fresh random pad r
x_masked = x - r  ----------------> public GEMM: W · x_masked
                  <---------------- masked result
recover W·x with precomputed W·r
attention / norms / activation / KV stay local

For a fresh uniform pad, x - r mod q is exactly uniform and independent of x. Online value privacy is information-theoretic; see the threat model.

Warning

Research code, not a production secure inference service. The default end-to-end demos use TrustedDealer, an explicitly insecure preprocessing placeholder that sees the masks. The online path is value-private, but the published CPU HE replacement hit its compute kill criterion. Length/timing also leak outside the tested constant-shape setting. Read Security status before using or citing the system.

Results

All headline results were measured on one Apple-Silicon host with 128 GiB unified memory and 16 logical CPUs. Model load is excluded from token timing. Veil numbers use F12 public projections and int32 online ring tensors.

Plaintext versus private inference

Model Mode tok/s TTFT TTFT incl. preprocessing Online MB/token Client model state Real quality accuracy¹ Empirical privacy²
Gemma 4 12B Local BF16 0.03480 23.67 s — 0 23.95 GB 24/24 (100%) 3.38% unmasked control
Gemma 4 12B Veil masked 0.01629 61.39 s 135.17 s 16.39 2.05 GB 24/24 (100%) 97.74%
Gemma 4 31B Local BF16 0.01353 63.98 s — 0 62.58 GB 12/12 (100%) 5.63% unmasked control
Gemma 4 31B Veil masked 0.003496 286.03 s 654.00 s 30.61 2.85 GB 12/12 (100%) 97.10%

¹ gsm8k-rationale-final-v1: 36 real GSM8K cases with 4,699 gold-rationale tokens (40–202 tokens each). Given the gold rationale and shared number prefix, the model scores the correct first differing final-number token against answer + 1. BF16 and exact F12 agree on 36/36 decisions; aggregate 95% Wilson interval is 90.4–100%. This is a real long-rationale quality test, not standard free-generation GSM8K exact match.

² Empirical score is 1 - classifier advantage on 480 fixed-shape real-activation sessions per model, with 24 MMLU and 24 GSM8K prompts and 100 prompt-grouped holdouts. Masked attack accuracy is 51.13% at 12B and 51.45% at 31B. The exact fresh-OTP value-privacy theorem remains 100%; the empirical score is classifier- and shape-specific and does not include metadata leakage.

Full methodology, confidence intervals, memory measurements, raw data, and caveats:

What is implemented

Capability Status
Exact ring masking and unmasking Implemented and bit-exact
Tiny, 1.1B, 7B W8, Gemma 4 12B/31B model splits Implemented
Binary TCP transport and Docker isolation Implemented
Server-side uniformity/input-independence audit Implemented
Client-secret integrity checks Implemented; zero server/wire overhead
Speculative decoding Implemented; 1.121× modeled speedup at 50 ms RTT in selected 7B test
Constant-shape trace red team Implemented for the tiny harness
Secure BFV preprocessing semantics Implemented at small scale; CPU full-model path killed on compute
Production GPU kernels / secure 31B preprocessing Not implemented

Quick start

1. Install

Python 3.11+ is required. The model extra installs PyTorch, Transformers, and Safetensors; no model weights are downloaded during installation.

git clone https://github.com/FrancoisChastel/secure-inference.git
cd secure-inference
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev,model]"

2. Run the test suite

pytest -q

Large real-model tests are opt-in and skip unless their local artifacts and environment flags are present.

3. Run the no-download demo

This creates a deterministic tiny random-weight Llama, executes both ordinary and masked split inference, and checks exact greedy-token agreement:

python -m scripts.run_poc

Expected shape of the output:

plaintext tokens: [...]
masked tokens:    [...]
match: 4/4

Running a real Gemma 4 model

The repository does not redistribute Google model weights. Download an official public checkpoint from Hugging Face and deterministically prepare the split artifacts locally.

Download Gemma 4 12B

HF_HUB_DISABLE_IMPLICIT_TOKEN=1 python - <<'PY'
from huggingface_hub import snapshot_download

snapshot_download(
    "google/gemma-4-12B-it",
    local_dir="models/Gemma-4-12B-it",
    token=False,
    allow_patterns=[
        "*.json", "*.jinja", "*.safetensors", "*.safetensors.index.json"
    ],
)
PY

Prepare F12 client/server artifacts

python -m scripts.prepare_model \
  --source models/Gemma-4-12B-it \
  --output models/veil-gemma4-12b-f12

F12 means round(weight × 2^12): 12 fractional bits, not 12-bit weight storage. The prepared artifact contains a small client side and public projection matrices for the server.

Run the explicit real-model parity gate

VEIL_RUN_GEMMA4_12B=1 pytest tests/test_gemma4_real.py -k 12B -q

The 31B workflow is identical with Gemma-4-31B-it, but needs approximately 63 GB for the source checkpoint, 61 GB for prepared artifacts, 128 GiB host memory, and substantial CPU time. See M8_REPORT.md before running it.

Reproducing the published charts

The raw benchmark JSON is committed, so regenerating the report does not require model weights:

python -m scripts.render_privacy_comparison

This writes:

  • docs/results/m8_privacy_comparison.json
  • docs/results/m8_privacy_comparison.svg

To rerun the underlying measurements, see the exact commands in M8_PRIVACY_COMPARISON.md. The 12B/31B quality runs are intentionally expensive on CPU.

Docker isolation demo

The Docker demo keeps client, untrusted server, and insecure POC dealer in separate containers on an internal network. The server image has public weights only—no prompt, tokenizer, keys, masks, or client mount.

After preparing the TinyLlama artifact as documented in the experiment log:

scripts/run_m3.sh

For an interactive view of client plaintext beside actual server-side masked values:

python -m scripts.visual_demo

The Docker demo still uses the insecure trusted dealer. It demonstrates online blindness and isolation, not secure preprocessing.

Security status

Protected from the online server

  • prompt and generated values;
  • intermediate activations;
  • client-local KV cache;
  • client integrity challenges and masks.

Declared leakage and limitations

  • model architecture and public matrix shapes;
  • token/bucket count and timing outside the tested constant-shape mode;
  • service-use/network metadata;
  • denial of service;
  • insecure masks in the default TrustedDealer preprocessing path.

The M4 BFV experiments keep the secret key client-side and pass exact semantic checks, but the available CPU backend measured roughly 80,000× plaintext cost on a small map. The repository therefore makes no secure full-model HE throughput claim. See THREAT_MODEL.md and PROTOCOL.md.

Weight and artifact policy

Local checkpoints, F12 artifacts, audit logs, integrity secrets, and .env files are git-ignored. Prepared Gemma artifacts are large (about 24 GB at 12B and 61 GB at 31B) and are reproducible from official public checkpoints, so they are not uploaded to this repository or a separate Hugging Face repository.

Do not publish personalized M5 integrity material: it is a client secret.

Repository layout

veil/          protocol, client runtime, server, dealers, and audits
scripts/       demos, preparation, benchmarks, and report renderers
tests/         unit, adversarial, isolation, and opt-in real-model gates
docker/        isolated client/server/dealer images
docs/          math, protocol, threat model, experiment reports, raw results
benchmarks/    committed public benchmark subsets and provenance
validate/      standalone numerical validation

Documentation

Citation

If this repository is useful in your research, cite the metadata in CITATION.cff. Results should be cited with their trust boundary: “masked online path with trusted preprocessing placeholder,” unless you have replaced the dealer with a secure implementation.

License

Code is licensed under the Apache License 2.0. Model weights and benchmark data remain subject to their respective upstream licenses; source and license metadata are recorded in benchmarks/ and the model cards.

About

Research POC for masked private inference of public-weight LLMs without trusted hardware

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages