Skip to content

Verify Phala Cloud model CVMs on dstack 0.6 - #239

Open
amiller wants to merge 2 commits into
13rac1:mainfrom
amiller:phala-dstack-0.6-2026-11
Open

amiller wants to merge 2 commits into
13rac1:mainfrom
amiller:phala-dstack-0.6-2026-11

Conversation

@amiller

@amiller amiller commented Nov 19, 2026 •

Copy link
Copy Markdown

Problem

teep verify phalacloud --model deepseek/deepseek-v4.1-flash-1017 fails the enforced aci_session_verified, so teep serve blocks every request to the model. Observed on 2026-11-19 at 6548996.

The gateway lists three attested sessions for the model. teep has no verifier for two of them (private-ai-verifier/tinfoil/v1, private-ai-verifier/near-ai-gateway/v1). The third is Phala's own model CVM (private-ai-verifier/phala-direct/v1), and its audit fails at three checks in turn:

Check Value in the session Why teep refuses it
MRTD 83baf004a06c83f6… not in dstackMRTDAllow
MRSEAM eddfa0c1750b39ba… not in dstackMRSEAMAllow
compose binding MRConfigID 03cbc87e… VerifyComposeBinding accepts only version 1 (0x01 ‖ sha256(app_compose))

This is not the first dstack 0.6 CVM at Phala. The current phalacloud_aci fixture (qwen/qwen3.8-27b, captured 2026-11-13) holds three Phala model CVM sessions (qwen38-27b-{0,1,2}.usc2.phala.com) with the same MRTD and a version 3 MRConfigID, on TDX module 1.5.24. teep refused all three.

The model CVM runs dstack-0.6.3 (vm_config image, os_image_hash 38a6f97b…) on a 4-GPU B300 host. dstack 0.6 sets MR_CONFIG_ID version 3 for a launch with a key provider (mr_config_version in dstack/vmm/src/app/mr_config.rs).

The session audit reads the hard-coded dstack lists (sessionMeasurementPolicy, internal/provider/phalacloud/session_audit.go), not mrtd_allow or mrseam_allow from the config, so a user cannot add the values. #190 plans signed allowlist updates; until then I followed #183 and added the values to the built-in lists with their sources.

The report shows only the first refusal (refusalSummary, internal/attestation/report.go:2267). For this model that is the Tinfoil session, so the report never showed why the Phala session failed. I found the three reasons with a local build that prints every refusal, adding one value at a time. I did not include that change here. Listing every refusal reason, or at least the reason of each session that teep has a verifier for, would have shown the cause directly.

Change

  • dstackMRTDAllow: add 83baf004…, the TDX MRTD that dstack publishes for dstack-0.6.3, 0.6.4 and 0.6.5: image-hashes.txt of the mkosi-os-v0.6.3 and mkosi-os-v0.6.4 releases, and the release notes of mkosi-os-v0.6.5. The three images share one MRTD. The live session's os_image_hash is the 0.6.3 value in that file.
  • dstackMRSEAMAllow: add eddfa0c1… for TDX module 2.0.20, from Intel's signed binary. See the point of discussion.
  • attestation.VerifyComposeBindingV3: checks an MRConfigID of version 3. It is 0x03 ‖ sha256("dstack-mr-config-v3:" ‖ 0x00 ‖ document) ‖ 15 zero bytes (MrConfigV3 in dstack-types/src/mr_config.rs). The quote does not carry the document, so teep builds the one that the VMM builds (prepare_mr_config_v3): compose_hash = sha256(app_compose); key_provider, key_provider_id and, from manifest version 3, the sha256 of each init script come from app_compose; app_id, instance_id and gpu_policy_hash come from the RTMR3 events. teep builds one document and does not try variants.
  • checkModelCVMIntegrity replays the event log first, so the RTMR3 values it reads are the ones the quote binds. It then checks the compose binding: version 3 with VerifyComposeBindingV3, otherwise VerifyComposeBinding as before.
  • Tests:
    • New fixture internal/provider/aci/testdata/session_phala_direct_0_6_3.json, the captured session record.
    • TestSessionAuditPhalaModelCVMDstack063 runs it through SessionAuditor.Audit. With only this test and the fixture applied to 6548996, it fails with MRTD 83baf004… is not allowed. On the first commit it fails with compose binding: MRConfigID does not match app_compose hash. (The full test file does not build on main, because TestComposeBindingV3Refuses calls the new function.)
    • TestComposeBindingV3Refuses changes each input of the document (app_compose, the GPU flag, the app-id and instance-id events, a second app-id event) and requires a refusal.
    • The entry counts in dstack_defaults_test.go are updated.
  • phalacloud_aci fixture: with this change the three 2026-11-13 Phala sessions above pass. The captured probe request had pinned only four sessions, so TestVerifyRun_PhalaCloudCanonical_Fixture failed. I replaced the fixture with a new capture of qwen/qwen3.8-27b (2026-11-19T20:08:56Z), as docs/transport/testing.md describes, and updated fixtureSessionIDs. The receipt fixture (internal/provider/aci/testdata/receipt.json), citedSession and liveReceiptID now come from the same capture. The receipt cites the Phala model CVM session 2fe1110a… (qwen3-8-27b.usw7-d9.phala.com), which runs dstack 0.6.4 on TDX module 2.0.20, with a version 3 MRConfigID. The refreshed fixture therefore needs both new values and the version 3 check, including the MRSEAM in the point of discussion.
  • Docs: the MRTD and MRSEAM tables and their sources in docs/measurement_allowlists.md, and check 6 of the session audit and its test list in docs/providers/phalacloud/phalacloud_support.md. teep requires app_compose to name key_provider and to give manifest_version as a string. dstack also accepts a compose without key_provider (it derives the provider from kms_enabled) or with a numeric version. teep refuses those composes, and the doc says so.

Effect

teep verify phalacloud --model deepseek/deepseek-v4.1-flash-1017 pins the Phala model CVM session, and the probe passes with E2EE v2 and a verified receipt. The Tinfoil and NEAR sessions are still refused.

Two commits: the measurements, then the version 3 check with the fixture refresh. Without the MRSEAM, the deepseek/deepseek-v4.1-flash-1017 session and the refreshed fixture's cited session stay refused, so dropping it also needs another fixture.

The MRTD and MRSEAM lists are shared, so the two values also reach Venice, NanoGPT, NEAR Direct, NEAR Cloud (model and gateway), Tinfoil and Chutes. Only the Phala session audit reads a version 3 MRConfigID. The other dstack compose checks (verify.go:267, attest.go:197, proxy.go:1279, proxy.go:1424) still accept version 1 only and will refuse a dstack 0.6 CVM with a key provider.

To verify

go test ./internal/provider/phalacloud/ -run 'SessionAuditPhalaModelCVMDstack063|ComposeBindingV3' -v
# --- PASS: TestSessionAuditPhalaModelCVMDstack063
# --- PASS: TestComposeBindingV3Refuses
make check     # lint: 0 issues; tests: ok for every package
make coverage

Live, 2026-11-19, with PHALA_API_KEY set:

teep verify phalacloud --model deepseek/deepseek-v4.1-flash-1017
  ✓ e2ee_usable            E2EE v2 secp256k1: … encrypted fields decrypted …  [ALLOWED]
  ✓ aci_session_verified   1 attested sessions pinned: … (2 refused; first: teep does not verify the evidence of private-ai-verifier/tinfoil/v1)  [ENFORCED]
  ✓ aci_receipt_verified   receipt verified, including the request hash  [ENFORCED]

On main the same command shows ✗ aci_session_verified … (3 refused; first: teep does not verify the evidence of private-ai-verifier/tinfoil/v1) and no probe: the report blocks inference requests.

To check the sources:

  • MRTD: tdx_mrtd in https://github.com/Dstack-TEE/dstack/releases/download/mkosi-os-v0.6.3/image-hashes.txt, the same in mkosi-os-v0.6.4, and the TDX MRTD row of the mkosi-os-v0.6.5 release notes.
  • MRSEAM: sha384sum separate_files/2.0/2.0.20/tdx_module_2.0.20.bin in intel/confidential-computing.tdx.tdx-module.binaries at 73b2a66e, and bytes 900-947 of tdx_module_2.0.20.sigstruct. Both methods give the table value for 2.0.08 (476a2997…).

The new capture's teep verify exited 1 at its capture self-check: --reverify reports the recorded receipt as a deferred skip (#211), so the report differs on aci_receipt_verified. The 2026-11-13 fixture shows the same difference with main's binary, so I did not treat it as a regression. The capture holds no API key (scanned).

With this change, a chat request with tools to this model is still refused while E2EE is on: E2EE v2 has no field path for tool fields. That is #240.

Point of discussion

MRSEAM of TDX module 2.0.20. Intel publishes two different values:

  • Intel's hash table (README of intel/confidential-computing.tdx.tdx-module, commit 859989ac, 2026-11-12) and the TDX_MODULE_2.0.20 release notes give 57a1a66f3704887a….
  • The SHA-384 of the signed 2.0.20 binary, and the MRSEAM field of its sigstruct, give eddfa0c1750b39ba6c9b039c579a5039baea307f8290fc5e3636d70e6001749dc95a368b9d8a9c50c84d1c22b5642d3c. Phala's quote reports this value. The quote's TEE_TCB_SVN (0903…) matches the 2.0.20 release notes, but 2.0.18 has the same SVNs, so it does not separate the two.

1.5.36, released the same day, has the same mismatch: the table gives 06b9e658… and the binary hashes to 28254b2a…. I added only the 2.0.20 value, because it is the one in use, labeled with where it comes from. Do you want to accept a value that comes from Intel's signed binary and not from its table, or wait until Intel corrects one of the two? #139 (move MRSEAM to tracking) would make this question smaller.

Checklist

  • I added unit tests for new code and regression tests for fixed bugs.
  • make check passes (golangci-lint: 0 issues; every package ok), and make coverage passes (86.5% total).
  • For a major feature: make integration and make reports pass. Not run; teep verify phalacloud was run live for this model only.
  • I updated the documentation that this change affects.

Posted with Claude.

Phala Cloud's model CVMs run dstack-0.6.3. On 2026-11-19 the session of
deepseek/deepseek-v4.1-flash-1017 reported MRTD 83baf004... and MRSEAM
eddfa0c1..., and the session audit refused it at the MRTD check.

- MRTD 83baf004... is the TDX MRTD that dstack publishes for dstack-0.6.3,
  0.6.4 and 0.6.5 (image-hashes.txt of mkosi-os-v0.6.3 and v0.6.4, release
  notes of mkosi-os-v0.6.5).
- MRSEAM eddfa0c1... is the sha384 of Intel's signed TDX module 2.0.20
  binary and the hash in its sigstruct (tdx-module.binaries 73b2a66e).
  Intel's hash table gives 57a1a66f... for 2.0.20; the same two methods
  give the table value for 2.0.08.

Both lists are shared by every dstack provider.
dstack 0.6 sets MR_CONFIG_ID version 3 for a launch with a key provider:
0x03, sha256("dstack-mr-config-v3:" || 0x00 || document), 15 zero bytes.
VerifyComposeBinding accepts only version 1, so the session audit refused
every Phala model CVM on dstack 0.6 at the compose binding.

VerifyComposeBindingV3 builds the document that the dstack VMM builds
(prepare_mr_config_v3): compose_hash from sha256(app_compose),
key_provider, key_provider_id and the init script hashes from app_compose,
app_id, instance_id and gpu_policy_hash from the RTMR3 events. The audit
now replays the event log first, so the values it reads are the ones the
quote binds.

TestSessionAuditPhalaModelCVMDstack063 runs the captured session of
2026-11-19 through the audit; it fails on main at the MRTD check, and with
the measurements only at the compose binding.

Fixture: With dstack 0.6 sessions verified, three Phala model CVM sessions of the
2026-11-13 capture pass the audit, and the pinned set no longer matches the
captured probe request. The new capture of qwen/qwen3.8-27b
(2026-11-19T20:08:56Z) replaces it. The receipt fixture, citedSession and
liveReceiptID come from the same capture; the receipt cites the Phala
model CVM session 2fe1110a....
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant