Repository navigation
Conversation
Phala Cloud's model CVMs run dstack-0.6.3. On 2026-11-19 the session of deepseek/deepseek-v4.1-flash-1017 reported MRTD 83baf004... and MRSEAM eddfa0c1..., and the session audit refused it at the MRTD check. - MRTD 83baf004... is the TDX MRTD that dstack publishes for dstack-0.6.3, 0.6.4 and 0.6.5 (image-hashes.txt of mkosi-os-v0.6.3 and v0.6.4, release notes of mkosi-os-v0.6.5). - MRSEAM eddfa0c1... is the sha384 of Intel's signed TDX module 2.0.20 binary and the hash in its sigstruct (tdx-module.binaries 73b2a66e). Intel's hash table gives 57a1a66f... for 2.0.20; the same two methods give the table value for 2.0.08. Both lists are shared by every dstack provider.
dstack 0.6 sets MR_CONFIG_ID version 3 for a launch with a key provider:
0x03, sha256("dstack-mr-config-v3:" || 0x00 || document), 15 zero bytes.
VerifyComposeBinding accepts only version 1, so the session audit refused
every Phala model CVM on dstack 0.6 at the compose binding.
VerifyComposeBindingV3 builds the document that the dstack VMM builds
(prepare_mr_config_v3): compose_hash from sha256(app_compose),
key_provider, key_provider_id and the init script hashes from app_compose,
app_id, instance_id and gpu_policy_hash from the RTMR3 events. The audit
now replays the event log first, so the values it reads are the ones the
quote binds.
TestSessionAuditPhalaModelCVMDstack063 runs the captured session of
2026-11-19 through the audit; it fails on main at the MRTD check, and with
the measurements only at the compose binding.
Fixture: With dstack 0.6 sessions verified, three Phala model CVM sessions of the
2026-11-13 capture pass the audit, and the pinned set no longer matches the
captured probe request. The new capture of qwen/qwen3.8-27b
(2026-11-19T20:08:56Z) replaces it. The receipt fixture, citedSession and
liveReceiptID come from the same capture; the receipt cites the Phala
model CVM session 2fe1110a....
The canonical ACI/1 report of Phala Cloud carries no quote of the inference host: that evidence is in the attested sessions, which the enforced aci_session_verified audits. The core factors failed with "no TEE quote", so the default allow_fail waived them, and a check that the audit does not make looked the same as one it does. SessionAuditor.Covers names the factors that the audit checks for a session: quote structure, certificate chain, signature, debug bit, MRTD/MRSEAM, REPORTDATA, the NVIDIA payload, signature and claims, online the collateral and TCB, and for the Phala model CVM the event-log replay and the compose binding. AuditSessions keeps the factors that every pinned session covers, and BuildReport passes those with aci_session_verified when the response has no quote of its own. Every other factor keeps its own result: the RTMR policy, the client nonce of the GPU evidence, NRAS, cpu_gpu_chain, the weights and the components still fail. PhalaCloudACIDefaultAllowFail no longer waives the factors that every session covers online. event_log_integrity and compose_binding stay allowed, because a Chutes session does not cover them.
gateway_compose_binding checks that sha256(app_compose) matches the gateway MRConfigID, but any compose passes: a gateway that redeploys with a new source pin or new sidecar images still verifies. gateway_compose_hash_allow lists the sha256(app_compose) values that the gateway may run, 32 bytes of hex. When it is set, gateway_compose_binding fails unless the binding holds and the hash is in the list; with no app_compose in the response it fails instead of skipping. When it is absent, nothing changes. It merges like the gateway measurement allowlists, so a provider policy that sets only this key is kept.
UpdateConfig decodes the file into updatePolicy and writes it back, so a key the struct does not name is dropped. updatePolicy now carries gateway_compose_hash_allow, which --update-config never adds to.
e732070 disabled tinfoil_v3_direct because the enclave hosts did not answer attestation (2026-11-12). On 2026-11-20 three of the six deepseek-v4-1-flash enclaves answer: inf16-0 (TDX, release v0.0.10+b200), inf22-0 (SEV-SNP, v0.0.10+b300) and inf19-throughput-0 (SEV-SNP, v0.0.10+b300-throughput). Each reports four Blackwell GPUs, which the parser refused, and two of the three run a release that is not the repo's latest. This commit reverts e732070 and verifies what these enclaves serve. - GPU evidence. readDeviceEvidence reads each device_evidence item of format nvidia-gpu-evidence/v1 (kind gpu, vendor nvidia; exactly arch, certificate, evidence, nonce). Each item's nonce must be the challenge nonce and all items one arch; anything else fails the document. The items become an NVIDIA EAT with the client nonce, so nvidia_signature checks each certificate chain and SPDM signature against the client nonce (the enclave sends it to the GPUs unchanged) and NRAS appraises the same EAT. VerifyReportData reports gpu_bound=true when the REPORTDATA-bound section carries GPUs, so cpu_gpu_chain passes. nvidia_nonce_client_bound and nvidia_nras_verified are no longer inapplicable. - Code release. The enclave names its release in the "code" collateral. Its repo must be the expected one. teep fetches the bundle for the digest from GitHub, verifies it with the repo's release workflow identity, and requires the SAN to end in @refs/tags/<tag>. The release must be current: the "code-freshness" witness, signed by tinfoilsh/freshness-witness and endorsing the same repo, tag and digest, must verify and be under seven days old (Tinfoil's window). This replaces "latest release", which is one of three release lines. - Platform release. MRTD and RTMR0 are matched against the platform-endorsements/v1 predicate of the tinfoilsh/cvmimage release the "platform" collateral names, checked the same way. tinfoilsh/hardware-measurements stopped at v0.0.47 and has no B200 shape. - enclaves = [...] in [providers.tinfoil_v3_direct] restricts requests to those enclave domains; a model with none of them fails to resolve. Proxy discovery lists enclaves that do not answer. - nvswitch_binding fails with "the enclave serves no NVSwitch evidence" instead of passing as "verified absent". - tinfoil_v3_cloud: nvidia_nonce_client_bound and nvidia_nras_verified are now applicable and fail (the router serves no GPU evidence), so TinfoilCloudDefaultAllowFail lists them with the other core factors. - compose_binding, sigstore_verification and event_log_integrity stay inapplicable; their reasons name the checks that cover them. - tinfoil.VerifySupplyChain replaces the two copies in verify and proxy. Effect: teep verify tinfoil_v3_direct --model deepseek-v4-1-flash on inf16-0, with allow_fail = ["nvswitch_binding", "cpu_id_registry"], exits 0: 30/32 passed, e2ee_usable passes (EHBP to the enclave's key). On the SEV-SNP enclaves the code release, GPU evidence and REPORTDATA pass, and five factors fail because go-sev-guest v0.15.0 decodes the Turin TCB with the Genoa layout (VCEK 404, TCB below Genoa minimums); tinfoil_support.md, "SEV-SNP on Turin". To verify: make check. New fixtures: tinfoil_v3_direct_tinfoil (inf16-0) and a re-captured tinfoil_v3_cloud_tinfoil; testdata v3_direct_tdx_b200.json and v3_direct_snp_b300.json.
teep verify tinfoil_v3_direct failed sigstore_code_verified and tee_boot_config with a read-only HOME: "fetch Sigstore trusted root: failed to create TUF client failed to create initial TUF updater: mkdir .../.sigstore". The TUF client of sigstore-go caches under $HOME/.sigstore by default. A container that runs teep with a read-only root and only /tmp writable, as Souk's CVM does, cannot create it. - The TUF client runs with WithDisableLocalCache, the sigstore-go option for read-only filesystems. - SigstoreVerifier fetches the trusted root once and keeps it, so one supply chain check (code bundle, platform bundle and their freshness witnesses) fetches it once instead of four times. - TestVerifyNamedRelease_Rejects: a release in another repo, or none. Effect: with HOME a read-only directory and no TMPDIR, teep verify tinfoil_v3_direct --model deepseek-v4-1-flash (enclave inf16-0) exits 0, 30/32 passed; before this commit it exited 1 with both factors failing. To verify: make check; TestVerifyNamedRelease_Rejects.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #241 (and so on #239): the first five commits are theirs; this PR adds the last two.
Problem
tinfoil_v3_directhas been refused at startup since e732070, because on 2026-11-12 Tinfoil's enclave hosts did not answer attestation. On 2026-11-20 three of the six enclaves ofdeepseek-v4-1-flashanswer. With e732070 reverted, teep still cannot verify any of them:deepseek-v4-1-flash-inf16-0v0.0.10+b200https://tinfoil.sh/format/nvidia-gpu-evidence/v1) that teep does not read yet"-inf22-0v0.0.10+b300-inf19-throughput-0v0.0.10+b300-throughput-inf15-0,-inf15-1,-inf28-0With the GPU check removed, three more things fail:
v0.0.10+b200), so both SEV-SNP enclaves failtee_measurement. The repo publishes one release line per hardware target.tinfoilsh/hardware-measurements, which stopped at v0.0.47 (2026-10-29). The enclave names a newer source in itsplatformcollateral:tinfoilsh/cvmimageplatform-v0.1.15, whoseplatform-endorsements/v1predicate lists the shapeb200_32cpu_512gb_4gpu_4diskwith these values.prompt_cache_key, the resolver picksinf15-0, which does not answer.Change
TINFOIL_API_KEYconfigurestinfoil_v3_directagain.readDeviceEvidencereads eachdevice_evidenceitem of formatnvidia-gpu-evidence/v1, kindgpu, vendornvidia, whose evidence has exactlyarch,certificate,evidenceandnonce. Each item'snoncemust be the challenge nonce, and all items must have onearch. Any other item, or another field, fails the document as before. The items become an NVIDIA EAT with the client nonce:nvidia_signaturechecks each GPU's certificate chain and SPDM signature against the client nonce. The enclave passes it to the GPUs as the SPDM requester nonce unchanged (the first SPDM bytes of the inf16-0 capture are11 e0 01 fffollowed by the nonce).nvidia_nras_verifiedsends the same EAT to NRAS.VerifyReportDatareportsgpu_bound=truewhen the REPORTDATA-bound section carries GPUs, socpu_gpu_chainpasses.nvidia_nonce_client_boundandnvidia_nras_verifiedare no longer inapplicable for Tinfoil.codecollateral (repo,tag,digest). These are hints:@refs/tags/<tag>, which binds the digest to that tag.code-freshness), a bundle signed bytinfoilsh/freshness-witness(freshness.yml@refs/heads/main) that endorses repo, tag and digest. teep verifies it with that exact identity, checks that it endorses the same release, and requires its earliest verified timestamp to be under seven days old, the window Tinfoil's verifier uses. The GitHub proxy serves no release list, so teep cannot check that the tag is the newest of its line any other way.measurementsof theplatform-endorsements/v1predicate in thetinfoilsh/cvmimagerelease theplatformcollateral names, checked the same way.tinfoilsh/cvmimagereplacestinfoilsh/hardware-measurementsin the supply chain policies.enclaves = [...]in[providers.tinfoil_v3_direct]restricts requests to the listed enclave domains. A model with none of them fails to resolve, naming the configured and the listed enclaves. teep refuses the key on other providers.nvswitch_bindingfails with "the enclave serves no NVSwitch evidence; teep cannot check whether an NVSwitch fabric connects its GPUs" when GPUs are bound and no NVSwitch evidence is present. It used to pass as "verified absent", which nothing checked. It stays inTinfoilDirectDefaultAllowFail.tinfoil_v3_cloud. With the two NVIDIA factors applicable, they fail on the router, which serves no GPU evidence.TinfoilCloudDefaultAllowFaillists them with the other core factors.compose_binding,sigstore_verificationandevent_log_integritystay inapplicable. Their reasons now name the checks that cover them (sigstore_code_verified,tee_measurement, and RTMR3 = 0 intee_hardware_config).tinfoil.VerifySupplyChainreplaces the two copies inverifyandproxy.WithDisableLocalCache, andSigstoreVerifierfetches the root once per check, not once per bundle. Under a read-only root (only/tmpwritable, as Souk's CVM runs teep) the default cache under$HOME/.sigstorecan't be created. That failedsigstore_code_verifiedandtee_boot_configbefore this commit.tinfoil_v3_direct_tinfoilcapture (inf16-0) and a re-capturedtinfoil_v3_cloud_tinfoil(the old one has no freshness or platform traffic). Captured documentsv3_direct_tdx_b200.jsonandv3_direct_snp_b300.json.tinfoil_support.md: Steps 1 and 3a, platform release, GPU evidence, "SEV-SNP on Turin".Effect
teep verify tinfoil_v3_direct --model deepseek-v4-1-flashon 2026-11-20, each enclave selected withenclaves:allow_fail = []e2ee_usablenot probed (blocked)nvswitch_binding,cpu_id_registryallow_fail = ["nvswitch_binding", "cpu_id_registry"]e2ee_usablepasses, EHBP to the enclave's quote-bound keytee_cert_chain,tee_quote_signature,tee_hardware_config,tee_tcb_current,tee_tcb_not_revoked,intel_pcs_collateral,nvswitch_binding,cpu_id_registryOn both SEV-SNP enclaves the code release (
v0.0.10+b300andv0.0.10+b300-throughput), REPORTDATA and GPU evidence pass. They run on AMD Turin, and go-sev-guest v0.15.0 decodes the Turin TCB with the Genoa layout. It builds a VCEK URL with a 64-byte hardware ID and nofmcSPL, whichkds-proxy.tinfoil.shanswers with 404, and compares the misdecoded TCB with Genoa minimums. This PR does not fix that. A go-sev-guest with Turin TCB support, and theamd-turin-prodminimums from the same platform predicate, would.phalacloud,nearcloudandchutesverify with this build as before (exit 0 against the tables they were tested with). All four, inf16-0 included, also exit 0 when built with golang:1.26.4 and run as uid 10001 in a--read-onlycontainer.To verify:
make check;TestVerifyNamedRelease_Rejects,TestParseV3Document_CapturedGPUEnclaves,TestParseV3Document_RejectsDeviceEvidence,TestCheckWitness,TestDirectResolver_RestrictEnclaves,TestLoadTOMLEnclaves,TestIntegration_TinfoilDirect_Fixture.Points of discussion
github-proxy.tinfoil.shserves no release list.measurementsof the platform predicate. Its per-machinepolicies(TDXmr_seam,minimum_tee_tcb_svn, SEV-SNP minimums) duplicate checks teep hard-codes; reading them would fix the Turin minimums.Posted with Claude.