Skip to content

Verify Tinfoil model enclaves directly - #242

Draft
amiller wants to merge 7 commits into
13rac1:mainfrom
amiller:tinfoil-direct-2026-11
Draft

amiller wants to merge 7 commits into
13rac1:mainfrom
amiller:tinfoil-direct-2026-11

Conversation

@amiller

@amiller amiller commented Nov 21, 2026 •

Copy link
Copy Markdown

Stacked on #241 (and so on #239): the first five commits are theirs; this PR adds the last two.

Problem

tinfoil_v3_direct has been refused at startup since e732070, because on 2026-11-12 Tinfoil's enclave hosts did not answer attestation. On 2026-11-20 three of the six enclaves of deepseek-v4-1-flash answer. With e732070 reverted, teep still cannot verify any of them:

Enclave Platform Release it runs What teep did
deepseek-v4-1-flash-inf16-0 TDX v0.0.10+b200 refused the document: "enclave reports 4 device evidence items (formats https://tinfoil.sh/format/nvidia-gpu-evidence/v1) that teep does not read yet"
-inf22-0 SEV-SNP v0.0.10+b300 the same
-inf19-throughput-0 SEV-SNP v0.0.10+b300-throughput the same
-inf15-0, -inf15-1, -inf28-0 TDX, TDX, SEV-SNP no answer in 20 s

With the GPU check removed, three more things fail:

  • teep compares every enclave with the repo's latest release (v0.0.10+b200), so both SEV-SNP enclaves fail tee_measurement. The repo publishes one release line per hardware target.
  • MRTD and RTMR0 of inf16-0 are not in tinfoilsh/hardware-measurements, which stopped at v0.0.47 (2026-10-29). The enclave names a newer source in its platform collateral: tinfoilsh/cvmimage platform-v0.1.15, whose platform-endorsements/v1 predicate lists the shape b200_32cpu_512gb_4gpu_4disk with these values.
  • Proxy discovery lists all six enclaves. With no prompt_cache_key, the resolver picks inf15-0, which does not answer.

Change

  • Revert e732070. TINFOIL_API_KEY configures tinfoil_v3_direct again.
  • GPU evidence. readDeviceEvidence reads each device_evidence item of format nvidia-gpu-evidence/v1, kind gpu, vendor nvidia, whose evidence has exactly arch, certificate, evidence and nonce. Each item's nonce must be the challenge nonce, and all items must have one arch. Any other item, or another field, fails the document as before. The items become an NVIDIA EAT with the client nonce:
    • nvidia_signature checks each GPU's certificate chain and SPDM signature against the client nonce. The enclave passes it to the GPUs as the SPDM requester nonce unchanged (the first SPDM bytes of the inf16-0 capture are 11 e0 01 ff followed by the nonce).
    • nvidia_nras_verified sends the same EAT to NRAS.
    • VerifyReportData reports gpu_bound=true when the REPORTDATA-bound section carries GPUs, so cpu_gpu_chain passes.
    • nvidia_nonce_client_bound and nvidia_nras_verified are no longer inapplicable for Tinfoil.
  • Code release. The enclave names its release in the code collateral (repo, tag, digest). These are hints:
    • The repo must be the expected one: the router's, or the one proxy discovery returns.
    • teep fetches the bundle for the digest from GitHub itself and verifies it with the repo's release workflow identity. The certificate SAN must end in @refs/tags/<tag>, which binds the digest to that tag.
    • The release must be current. The enclave serves a freshness witness (code-freshness), a bundle signed by tinfoilsh/freshness-witness (freshness.yml@refs/heads/main) that endorses repo, tag and digest. teep verifies it with that exact identity, checks that it endorses the same release, and requires its earliest verified timestamp to be under seven days old, the window Tinfoil's verifier uses. The GitHub proxy serves no release list, so teep cannot check that the tag is the newest of its line any other way.
    • This replaces the comparison with the latest release, for both providers.
  • Platform release (TDX). MRTD and RTMR0 are matched against the measurements of the platform-endorsements/v1 predicate in the tinfoilsh/cvmimage release the platform collateral names, checked the same way. tinfoilsh/cvmimage replaces tinfoilsh/hardware-measurements in the supply chain policies.
  • enclaves = [...] in [providers.tinfoil_v3_direct] restricts requests to the listed enclave domains. A model with none of them fails to resolve, naming the configured and the listed enclaves. teep refuses the key on other providers.
  • nvswitch_binding fails with "the enclave serves no NVSwitch evidence; teep cannot check whether an NVSwitch fabric connects its GPUs" when GPUs are bound and no NVSwitch evidence is present. It used to pass as "verified absent", which nothing checked. It stays in TinfoilDirectDefaultAllowFail.
  • tinfoil_v3_cloud. With the two NVIDIA factors applicable, they fail on the router, which serves no GPU evidence. TinfoilCloudDefaultAllowFail lists them with the other core factors.
  • Inapplicable factors. compose_binding, sigstore_verification and event_log_integrity stay inapplicable. Their reasons now name the checks that cover them (sigstore_code_verified, tee_measurement, and RTMR3 = 0 in tee_hardware_config).
  • tinfoil.VerifySupplyChain replaces the two copies in verify and proxy.
  • Sigstore trusted root (second commit). The TUF client runs with WithDisableLocalCache, and SigstoreVerifier fetches the root once per check, not once per bundle. Under a read-only root (only /tmp writable, as Souk's CVM runs teep) the default cache under $HOME/.sigstore can't be created. That failed sigstore_code_verified and tee_boot_config before this commit.
  • Fixtures and docs. A new tinfoil_v3_direct_tinfoil capture (inf16-0) and a re-captured tinfoil_v3_cloud_tinfoil (the old one has no freshness or platform traffic). Captured documents v3_direct_tdx_b200.json and v3_direct_snp_b300.json. tinfoil_support.md: Steps 1 and 3a, platform release, GPU evidence, "SEV-SNP on Turin".

Effect

teep verify tinfoil_v3_direct --model deepseek-v4-1-flash on 2026-11-20, each enclave selected with enclaves:

Enclave allow_fail = [] Failures
inf16-0 (TDX) 29/32, e2ee_usable not probed (blocked) nvswitch_binding, cpu_id_registry
inf16-0, allow_fail = ["nvswitch_binding", "cpu_id_registry"] exit 0, 30/32 both allowed; e2ee_usable passes, EHBP to the enclave's quote-bound key
inf22-0 (SEV-SNP) 22/31 tee_cert_chain, tee_quote_signature, tee_hardware_config, tee_tcb_current, tee_tcb_not_revoked, intel_pcs_collateral, nvswitch_binding, cpu_id_registry
inf19-throughput-0 (SEV-SNP) 22/31 the same

On both SEV-SNP enclaves the code release (v0.0.10+b300 and v0.0.10+b300-throughput), REPORTDATA and GPU evidence pass. They run on AMD Turin, and go-sev-guest v0.15.0 decodes the Turin TCB with the Genoa layout. It builds a VCEK URL with a 64-byte hardware ID and no fmcSPL, which kds-proxy.tinfoil.sh answers with 404, and compares the misdecoded TCB with Genoa minimums. This PR does not fix that. A go-sev-guest with Turin TCB support, and the amd-turin-prod minimums from the same platform predicate, would.

phalacloud, nearcloud and chutes verify with this build as before (exit 0 against the tables they were tested with). All four, inf16-0 included, also exit 0 when built with golang:1.26.4 and run as uid 10001 in a --read-only container.

To verify: make check; TestVerifyNamedRelease_Rejects, TestParseV3Document_CapturedGPUEnclaves, TestParseV3Document_RejectsDeviceEvidence, TestCheckWitness, TestDirectResolver_RestrictEnclaves, TestLoadTOMLEnclaves, TestIntegration_TinfoilDirect_Fixture.

Points of discussion

  • The freshness window lets an enclave keep serving a release for up to seven days after Tinfoil stops witnessing it. Listing releases would be tighter, but github-proxy.tinfoil.sh serves no release list.
  • teep reads only the measurements of the platform predicate. Its per-machine policies (TDX mr_seam, minimum_tee_tcb_svn, SEV-SNP minimums) duplicate checks teep hard-codes; reading them would fix the Turin minimums.

Posted with Claude.

Phala Cloud's model CVMs run dstack-0.6.3. On 2026-11-19 the session of
deepseek/deepseek-v4.1-flash-1017 reported MRTD 83baf004... and MRSEAM
eddfa0c1..., and the session audit refused it at the MRTD check.

- MRTD 83baf004... is the TDX MRTD that dstack publishes for dstack-0.6.3,
  0.6.4 and 0.6.5 (image-hashes.txt of mkosi-os-v0.6.3 and v0.6.4, release
  notes of mkosi-os-v0.6.5).
- MRSEAM eddfa0c1... is the sha384 of Intel's signed TDX module 2.0.20
  binary and the hash in its sigstruct (tdx-module.binaries 73b2a66e).
  Intel's hash table gives 57a1a66f... for 2.0.20; the same two methods
  give the table value for 2.0.08.

Both lists are shared by every dstack provider.
dstack 0.6 sets MR_CONFIG_ID version 3 for a launch with a key provider:
0x03, sha256("dstack-mr-config-v3:" || 0x00 || document), 15 zero bytes.
VerifyComposeBinding accepts only version 1, so the session audit refused
every Phala model CVM on dstack 0.6 at the compose binding.

VerifyComposeBindingV3 builds the document that the dstack VMM builds
(prepare_mr_config_v3): compose_hash from sha256(app_compose),
key_provider, key_provider_id and the init script hashes from app_compose,
app_id, instance_id and gpu_policy_hash from the RTMR3 events. The audit
now replays the event log first, so the values it reads are the ones the
quote binds.

TestSessionAuditPhalaModelCVMDstack063 runs the captured session of
2026-11-19 through the audit; it fails on main at the MRTD check, and with
the measurements only at the compose binding.

Fixture: With dstack 0.6 sessions verified, three Phala model CVM sessions of the
2026-11-13 capture pass the audit, and the pinned set no longer matches the
captured probe request. The new capture of qwen/qwen3.8-27b
(2026-11-19T20:08:56Z) replaces it. The receipt fixture, citedSession and
liveReceiptID come from the same capture; the receipt cites the Phala
model CVM session 2fe1110a....
The canonical ACI/1 report of Phala Cloud carries no quote of the
inference host: that evidence is in the attested sessions, which the
enforced aci_session_verified audits. The core factors failed with "no TEE
quote", so the default allow_fail waived them, and a check that the audit
does not make looked the same as one it does.

SessionAuditor.Covers names the factors that the audit checks for a
session: quote structure, certificate chain, signature, debug bit,
MRTD/MRSEAM, REPORTDATA, the NVIDIA payload, signature and claims, online
the collateral and TCB, and for the Phala model CVM the event-log replay
and the compose binding. AuditSessions keeps the factors that every pinned
session covers, and BuildReport passes those with aci_session_verified
when the response has no quote of its own. Every other factor keeps its
own result: the RTMR policy, the client nonce of the GPU evidence, NRAS,
cpu_gpu_chain, the weights and the components still fail.

PhalaCloudACIDefaultAllowFail no longer waives the factors that every
session covers online. event_log_integrity and compose_binding stay
allowed, because a Chutes session does not cover them.
gateway_compose_binding checks that sha256(app_compose) matches the
gateway MRConfigID, but any compose passes: a gateway that redeploys with
a new source pin or new sidecar images still verifies.

gateway_compose_hash_allow lists the sha256(app_compose) values that the
gateway may run, 32 bytes of hex. When it is set, gateway_compose_binding
fails unless the binding holds and the hash is in the list; with no
app_compose in the response it fails instead of skipping. When it is
absent, nothing changes. It merges like the gateway measurement
allowlists, so a provider policy that sets only this key is kept.
UpdateConfig decodes the file into updatePolicy and writes it back, so a
key the struct does not name is dropped. updatePolicy now carries
gateway_compose_hash_allow, which --update-config never adds to.
e732070 disabled tinfoil_v3_direct because the enclave hosts did not
answer attestation (2026-11-12). On 2026-11-20 three of the six
deepseek-v4-1-flash enclaves answer: inf16-0 (TDX, release
v0.0.10+b200), inf22-0 (SEV-SNP, v0.0.10+b300) and inf19-throughput-0
(SEV-SNP, v0.0.10+b300-throughput). Each reports four Blackwell GPUs,
which the parser refused, and two of the three run a release that is
not the repo's latest. This commit reverts e732070 and verifies what
these enclaves serve.

- GPU evidence. readDeviceEvidence reads each device_evidence item of
  format nvidia-gpu-evidence/v1 (kind gpu, vendor nvidia; exactly arch,
  certificate, evidence, nonce). Each item's nonce must be the challenge
  nonce and all items one arch; anything else fails the document. The
  items become an NVIDIA EAT with the client nonce, so nvidia_signature
  checks each certificate chain and SPDM signature against the client
  nonce (the enclave sends it to the GPUs unchanged) and NRAS appraises
  the same EAT. VerifyReportData reports gpu_bound=true when the
  REPORTDATA-bound section carries GPUs, so cpu_gpu_chain passes.
  nvidia_nonce_client_bound and nvidia_nras_verified are no longer
  inapplicable.
- Code release. The enclave names its release in the "code" collateral.
  Its repo must be the expected one. teep fetches the bundle for the
  digest from GitHub, verifies it with the repo's release workflow
  identity, and requires the SAN to end in @refs/tags/<tag>. The release
  must be current: the "code-freshness" witness, signed by
  tinfoilsh/freshness-witness and endorsing the same repo, tag and
  digest, must verify and be under seven days old (Tinfoil's window).
  This replaces "latest release", which is one of three release lines.
- Platform release. MRTD and RTMR0 are matched against the
  platform-endorsements/v1 predicate of the tinfoilsh/cvmimage release
  the "platform" collateral names, checked the same way.
  tinfoilsh/hardware-measurements stopped at v0.0.47 and has no B200
  shape.
- enclaves = [...] in [providers.tinfoil_v3_direct] restricts requests
  to those enclave domains; a model with none of them fails to resolve.
  Proxy discovery lists enclaves that do not answer.
- nvswitch_binding fails with "the enclave serves no NVSwitch evidence"
  instead of passing as "verified absent".
- tinfoil_v3_cloud: nvidia_nonce_client_bound and nvidia_nras_verified
  are now applicable and fail (the router serves no GPU evidence), so
  TinfoilCloudDefaultAllowFail lists them with the other core factors.
- compose_binding, sigstore_verification and event_log_integrity stay
  inapplicable; their reasons name the checks that cover them.
- tinfoil.VerifySupplyChain replaces the two copies in verify and proxy.

Effect: teep verify tinfoil_v3_direct --model deepseek-v4-1-flash on
inf16-0, with allow_fail = ["nvswitch_binding", "cpu_id_registry"],
exits 0: 30/32 passed, e2ee_usable passes (EHBP to the enclave's key).
On the SEV-SNP enclaves the code release, GPU evidence and REPORTDATA
pass, and five factors fail because go-sev-guest v0.15.0 decodes the
Turin TCB with the Genoa layout (VCEK 404, TCB below Genoa minimums);
tinfoil_support.md, "SEV-SNP on Turin".

To verify: make check. New fixtures: tinfoil_v3_direct_tinfoil (inf16-0)
and a re-captured tinfoil_v3_cloud_tinfoil; testdata
v3_direct_tdx_b200.json and v3_direct_snp_b300.json.
teep verify tinfoil_v3_direct failed sigstore_code_verified and
tee_boot_config with a read-only HOME: "fetch Sigstore trusted root:
failed to create TUF client failed to create initial TUF updater: mkdir
.../.sigstore". The TUF client of sigstore-go caches under
$HOME/.sigstore by default. A container that runs teep with a read-only
root and only /tmp writable, as Souk's CVM does, cannot create it.

- The TUF client runs with WithDisableLocalCache, the sigstore-go option
  for read-only filesystems.
- SigstoreVerifier fetches the trusted root once and keeps it, so one
  supply chain check (code bundle, platform bundle and their freshness
  witnesses) fetches it once instead of four times.
- TestVerifyNamedRelease_Rejects: a release in another repo, or none.

Effect: with HOME a read-only directory and no TMPDIR, teep verify
tinfoil_v3_direct --model deepseek-v4-1-flash (enclave inf16-0) exits
0, 30/32 passed; before this commit it exited 1 with both factors
failing.

To verify: make check; TestVerifyNamedRelease_Rejects.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant