Veil is a research proof-of-concept for outsourcing the expensive linear layers of an open-weight LLM while keeping prompts, activations, KV state, and generated values private from the server—without trusted hardware.
The key observation is simple: the model weights are public. The client can
therefore keep every nonlinearity, attention operation, KV cache, and sampling step
locally, and outsource only public linear maps y = W x.
client untrusted server
------ ----------------
private activation x
fresh random pad r
x_masked = x - r ----------------> public GEMM: W · x_masked
<---------------- masked result
recover W·x with precomputed W·r
attention / norms / activation / KV stay local
For a fresh uniform pad, x - r mod q is exactly uniform and independent of x.
Online value privacy is information-theoretic; see the threat model.
Warning
Research code, not a production secure inference service. The default
end-to-end demos use TrustedDealer, an explicitly insecure preprocessing
placeholder that sees the masks. The online path is value-private, but the
published CPU HE replacement hit its compute kill criterion. Length/timing also
leak outside the tested constant-shape setting. Read Security status
before using or citing the system.
All headline results were measured on one Apple-Silicon host with 128 GiB unified memory and 16 logical CPUs. Model load is excluded from token timing. Veil numbers use F12 public projections and int32 online ring tensors.
| Model | Mode | tok/s | TTFT | TTFT incl. preprocessing | Online MB/token | Client model state | Real quality accuracy¹ | Empirical privacy² |
|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B | Local BF16 | 0.03480 | 23.67 s | — | 0 | 23.95 GB | 24/24 (100%) | 3.38% unmasked control |
| Gemma 4 12B | Veil masked | 0.01629 | 61.39 s | 135.17 s | 16.39 | 2.05 GB | 24/24 (100%) | 97.74% |
| Gemma 4 31B | Local BF16 | 0.01353 | 63.98 s | — | 0 | 62.58 GB | 12/12 (100%) | 5.63% unmasked control |
| Gemma 4 31B | Veil masked | 0.003496 | 286.03 s | 654.00 s | 30.61 | 2.85 GB | 12/12 (100%) | 97.10% |
¹ gsm8k-rationale-final-v1: 36 real GSM8K cases with 4,699 gold-rationale
tokens (40–202 tokens each). Given the gold rationale and shared number prefix, the
model scores the correct first differing final-number token against answer + 1.
BF16 and exact F12 agree on 36/36 decisions; aggregate 95% Wilson interval is
90.4–100%. This is a real long-rationale quality test, not standard free-generation
GSM8K exact match.
² Empirical score is 1 - classifier advantage on 480 fixed-shape real-activation
sessions per model, with 24 MMLU and 24 GSM8K prompts and 100 prompt-grouped
holdouts. Masked attack accuracy is 51.13% at 12B and 51.45% at 31B. The exact
fresh-OTP value-privacy theorem remains 100%; the empirical score is classifier-
and shape-specific and does not include metadata leakage.
Full methodology, confidence intervals, memory measurements, raw data, and caveats:
| Capability | Status |
|---|---|
| Exact ring masking and unmasking | Implemented and bit-exact |
| Tiny, 1.1B, 7B W8, Gemma 4 12B/31B model splits | Implemented |
| Binary TCP transport and Docker isolation | Implemented |
| Server-side uniformity/input-independence audit | Implemented |
| Client-secret integrity checks | Implemented; zero server/wire overhead |
| Speculative decoding | Implemented; 1.121× modeled speedup at 50 ms RTT in selected 7B test |
| Constant-shape trace red team | Implemented for the tiny harness |
| Secure BFV preprocessing semantics | Implemented at small scale; CPU full-model path killed on compute |
| Production GPU kernels / secure 31B preprocessing | Not implemented |
Python 3.11+ is required. The model extra installs PyTorch, Transformers, and Safetensors; no model weights are downloaded during installation.
git clone https://github.com/FrancoisChastel/secure-inference.git
cd secure-inference
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev,model]"pytest -qLarge real-model tests are opt-in and skip unless their local artifacts and environment flags are present.
This creates a deterministic tiny random-weight Llama, executes both ordinary and masked split inference, and checks exact greedy-token agreement:
python -m scripts.run_pocExpected shape of the output:
plaintext tokens: [...]
masked tokens: [...]
match: 4/4
The repository does not redistribute Google model weights. Download an official public checkpoint from Hugging Face and deterministically prepare the split artifacts locally.
HF_HUB_DISABLE_IMPLICIT_TOKEN=1 python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(
"google/gemma-4-12B-it",
local_dir="models/Gemma-4-12B-it",
token=False,
allow_patterns=[
"*.json", "*.jinja", "*.safetensors", "*.safetensors.index.json"
],
)
PYpython -m scripts.prepare_model \
--source models/Gemma-4-12B-it \
--output models/veil-gemma4-12b-f12F12 means round(weight × 2^12): 12 fractional bits, not 12-bit weight
storage. The prepared artifact contains a small client side and public projection
matrices for the server.
VEIL_RUN_GEMMA4_12B=1 pytest tests/test_gemma4_real.py -k 12B -qThe 31B workflow is identical with Gemma-4-31B-it, but needs approximately
63 GB for the source checkpoint, 61 GB for prepared artifacts, 128 GiB host memory,
and substantial CPU time. See M8_REPORT.md before running it.
The raw benchmark JSON is committed, so regenerating the report does not require model weights:
python -m scripts.render_privacy_comparisonThis writes:
docs/results/m8_privacy_comparison.jsondocs/results/m8_privacy_comparison.svg
To rerun the underlying measurements, see the exact commands in M8_PRIVACY_COMPARISON.md. The 12B/31B quality runs are intentionally expensive on CPU.
The Docker demo keeps client, untrusted server, and insecure POC dealer in separate containers on an internal network. The server image has public weights only—no prompt, tokenizer, keys, masks, or client mount.
After preparing the TinyLlama artifact as documented in the experiment log:
scripts/run_m3.shFor an interactive view of client plaintext beside actual server-side masked values:
python -m scripts.visual_demoThe Docker demo still uses the insecure trusted dealer. It demonstrates online blindness and isolation, not secure preprocessing.
- prompt and generated values;
- intermediate activations;
- client-local KV cache;
- client integrity challenges and masks.
- model architecture and public matrix shapes;
- token/bucket count and timing outside the tested constant-shape mode;
- service-use/network metadata;
- denial of service;
- insecure masks in the default
TrustedDealerpreprocessing path.
The M4 BFV experiments keep the secret key client-side and pass exact semantic checks, but the available CPU backend measured roughly 80,000× plaintext cost on a small map. The repository therefore makes no secure full-model HE throughput claim. See THREAT_MODEL.md and PROTOCOL.md.
Local checkpoints, F12 artifacts, audit logs, integrity secrets, and .env files
are git-ignored. Prepared Gemma artifacts are large (about 24 GB at 12B and 61 GB at
31B) and are reproducible from official public checkpoints, so they are not uploaded
to this repository or a separate Hugging Face repository.
Do not publish personalized M5 integrity material: it is a client secret.
veil/ protocol, client runtime, server, dealers, and audits
scripts/ demos, preparation, benchmarks, and report renderers
tests/ unit, adversarial, isolation, and opt-in real-model gates
docker/ isolated client/server/dealer images
docs/ math, protocol, threat model, experiment reports, raw results
benchmarks/ committed public benchmark subsets and provenance
validate/ standalone numerical validation
- Protocol math and cost model
- Wire protocol
- Threat model
- 31B scale report
- Privacy/quality comparison
- Speculative decoding
- Constant-shape trace red team
- Dealer-free PCG feasibility
- Complete experiment log
If this repository is useful in your research, cite the metadata in
CITATION.cff. Results should be cited with their trust boundary:
“masked online path with trusted preprocessing placeholder,” unless you have
replaced the dealer with a secure implementation.
Code is licensed under the Apache License 2.0. Model weights and benchmark
data remain subject to their respective upstream licenses; source and license
metadata are recorded in benchmarks/ and the model cards.