Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions docs/measurement_allowlists.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,11 @@ in the policy section of each provider that uses the gateway (or in the
global policy), and refreshes them when the gateway redeploys.
SEE: docs/attestation_gaps/aci_gateway.md.

`gateway_compose_hash_allow` lists the sha256 of the gateway app_compose
values the gateway may run (32 bytes of hex, not 48). When it is set,
`gateway_compose_binding` fails unless the compose binds to MRConfigID and its
hash is in the list. When it is absent, the compose is not pinned.

### Merge order

Each allowlist field is resolved independently using:
Expand Down Expand Up @@ -206,10 +211,21 @@ gateway), Tinfoil and Chutes.
| `476a2997c62bccc7...` | 2.0.08 | Granite Rapids | shared list |
| `685f891ea5c20e8f...` | 2.0.02 | Granite Rapids | shared list |
| `2d2de102461684f1...` | 2.0.18 | Granite Rapids / Clearwater Forest | shared list |
| `eddfa0c1750b39ba...` | 2.0.20 | Granite Rapids / Clearwater Forest | shared list (see below) |
| `5b38e33a6487958b...` | 1.5.06 | Sapphire/Emerald Rapids | Chutes only |
| `346bc77a1846cac2...` | 1.5.28 | Sapphire/Emerald Rapids | Chutes only |
| `ab62561a173acbd1...` | 1.5.34 | Sapphire/Emerald Rapids | Chutes only |

For 2.0.20, Intel's hash table and the release notes of `TDX_MODULE_2.0.20`
give `57a1a66f3704887a...`. The SHA-384 of the signed binary
`separate_files/2.0/2.0.20/tdx_module_2.0.20.bin` in
`intel/confidential-computing.tdx.tdx-module.binaries` (commit `73b2a66e`),
and the MRSEAM in its sigstruct at offset 900, are `eddfa0c1750b39ba...`.
The same two methods give the table's value for 2.0.08. Phala Cloud's
`deepseek/deepseek-v4.1-flash-1017` model CVM reported `eddfa0c1...` on
2026-11-19. The shared list has the value of the binary. Intel's table is
the source for the other rows.

A reasonable policy is to allow the set of non-deprecated module versions for
your target platform family. This does not require provider cooperation.

Expand Down Expand Up @@ -245,6 +261,12 @@ dstack-mr measure
| `b24d3b24e9e3c160...` | dstack-nvidia-0.5.4.1 |
| `f06dfda6dce1cf90...` | dstack-nvidia-0.5.5 |
| `df4d77c857de652b...` | dstack-nvidia-0.5.11 and 0.5.12 |
| `83baf004a06c83f6...` | dstack-0.6.3, 0.6.4 and 0.6.5 |

The dstack 0.6 value is the TDX `MRTD` that dstack publishes for each
image: `image-hashes.txt` of the `mkosi-os-v0.6.3` and `mkosi-os-v0.6.4`
releases, and the release notes of `mkosi-os-v0.6.5`, at
`github.com/Dstack-TEE/dstack/releases`.

For dstack-nvidia-0.5.12, RTMR1 and RTMR2 are not allowed until they are
computed with `mk-image-mr` from meta-dstack on the release files: no
Expand Down
41 changes: 35 additions & 6 deletions docs/providers/phalacloud/phalacloud_support.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,17 @@ one is not pinned:
5. The GPU evidence, verified locally with the session nonce, without NRAS:
the certificate chain and SPDM signature of every GPU must verify, as for
`nvidia_signature`.
6. For the Phala model CVM, the compose binding (MRConfigID) and the
event-log replay of RTMR0-3.
6. For the Phala model CVM, the event-log replay of RTMR0-3 and the
compose binding. An MRConfigID of version 1 (dstack 0.5) is 0x01 and
sha256(app_compose). One of version 3 (dstack 0.6) is 0x03 and the hash
of a launch document that the quote does not carry. teep builds the
document that the dstack VMM builds: compose_hash, key_provider,
key_provider_id and the init script hashes from app_compose; app_id,
instance_id and gpu_policy_hash from the RTMR3 events of the replayed
log (`attestation.VerifyComposeBindingV3`). app_compose must name
`key_provider` and give `manifest_version` as a string. dstack also
reads a compose without `key_provider` (from `kms_enabled`) or with a
numeric version; teep refuses those.

The parts of a session's evidence must have exactly the members that teep
reads. For a Chutes bundle these include the newer members of the Chutes
Expand Down Expand Up @@ -404,11 +413,30 @@ image, so a dstack-format response fails `component_recognition`.

## Factor enforcement

The report of `/v1/aci/attestation` carries no quote of the inference host;
that evidence is in the attested sessions. teep reports the factors of the
inference host from the session audit (ACI mode): a factor that the audit of
every pinned session checked passes with the detail "covered by
aci_session_verified" (`SessionAuditor.Covers`, `applyACISessionCoverage`).
These are the quote structure, certificate chain, signature and debug bit,
`tee_measurement` (the dstack or Chutes MRTD and MRSEAM list, not
`mrtd_allow`), `tee_reportdata_binding` (the session key and the gateway's
evidence nonce), the NVIDIA payload, signature and claims, online
`intel_pcs_collateral`, `tee_tcb_current` and `tee_tcb_not_revoked`, and,
when every pinned session is a Phala model CVM, `event_log_integrity` and
`compose_binding`. Every other factor keeps its own result. The audit
checks no RTMR value (`tee_hardware_config`, `tee_boot_config`), no client
nonce in the GPU evidence (`nvidia_nonce_client_bound`), no NRAS, and no
CPU-GPU binding, weights or components, so those fail.

The ACI/1 report selects `PhalaCloudACIDefaultAllowFail`: the list shared with
Venice and NanoGPT (`ACIDefaultAllowFail`) without `tls_key_binding`, which
is enforced, and with the GPU signature and nonce factors, because the
report carries no GPU evidence, and `nonce_match` and
`gateway_nonce_match`, because the report echoes no nonce.
is enforced, and without the factors that the audit checks for every session
online, which are enforced; with `nvidia_nonce_client_bound`, because the
sessions are a snapshot that the client nonce does not reach, and
`nonce_match` and `gateway_nonce_match`, because the report echoes no nonce.
`event_log_integrity` and `compose_binding` stay allowed, because the audit
does not check them for a Chutes session.
`aci_session_verified` and `aci_receipt_verified` are enforced.
`aci_receipt_verified` is a deferred skip, which does not block, when there
is no probe receipt to check: in `teep serve`, which checks the receipt of
Expand Down Expand Up @@ -501,7 +529,8 @@ stream failed.
[session tests](../../../internal/provider/aci/session_test.go), and
`TestPinSessions` and `TestPinSessionsRefuses` in
[pin tests](../../../internal/provider/aci/pin_test.go).
- `TestSessionAuditPhalaModelCVM`, `TestSessionAuditChutes` and
- `TestSessionAuditPhalaModelCVM`, `TestSessionAuditPhalaModelCVMDstack063`,
`TestComposeBindingV3Refuses`, `TestSessionAuditChutes` and
`TestSessionAuditRefuses` in
[session audit tests](../../../internal/provider/phalacloud/session_audit_test.go);
`TestFetchSessionsSelectsBoundedBundles` and `TestSelectBundles` in
Expand Down
66 changes: 66 additions & 0 deletions internal/attestation/aci_session_coverage_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package attestation

import (
"errors"
"slices"
"strings"
"testing"
)

// In ACI mode a factor that the audit of every pinned session checked passes
// with aci_session_verified, and is enforced by the Phala Cloud policy. A
// factor that the audit does not check keeps its own result, and with no
// pinned session every factor keeps it.
func TestACISessionCoverage(t *testing.T) {
nonce := NewNonce()
pinned := []ACIPinnedSession{{ID: "s1"}}
build := func(sessions *ACISessionResult, quote string) *VerificationReport {
raw := buildMinimalRaw(nonce, validSigningKey(t))
raw.ACICanonicalReport = true
raw.IntelQuote = quote
return BuildReport(&ReportInput{Provider: "phalacloud", Model: "m", Raw: raw, Nonce: nonce,
AllowFail: PhalaCloudACIDefaultAllowFail, ACISessions: sessions})
}

covered := build(&ACISessionResult{Pinned: pinned, Covered: []string{FactorTEECertChain, FactorComposeBinding}}, "")
for _, name := range []string{FactorTEECertChain, FactorComposeBinding} {
f := findFactor(t, covered, name)
if f.Status != Pass || !strings.Contains(f.Detail, "covered by aci_session_verified") {
t.Errorf("%s: %+v, want a pass covered by aci_session_verified", name, f)
}
}
if f := findFactor(t, covered, FactorTEECertChain); !f.Enforced {
t.Errorf("tee_cert_chain: %+v, want enforced for Phala Cloud", f)
}
for _, name := range []string{FactorTEEQuoteSignature, FactorTEEHardwareConfig, FactorNvidiaClientNonce, FactorCPUGPUChain} {
if f := findFactor(t, covered, name); f.Status == Pass {
t.Errorf("%s is not covered and passed: %+v", name, f)
}
}
if f := findFactor(t, covered, FactorTEEQuoteSignature); f.Status != Fail || !f.Enforced || !covered.Blocked() {
t.Errorf("tee_quote_signature not covered: %+v, want an enforced failure that blocks", f)
}

all := slices.Clone(phalaCloudSessionFactors)
for name, sessions := range map[string]*ACISessionResult{
"no pinned session": {Covered: all},
"sessions failed": {Err: errors.New("fetch"), Pinned: pinned, Covered: all},
} {
r := build(sessions, "")
if f := findFactor(t, r, FactorTEECertChain); f.Status != Fail {
t.Errorf("%s: tee_cert_chain %+v, want a failure", name, f)
}
}

direct := build(&ACISessionResult{Pinned: pinned, Covered: all}, "dGVzdA==")
if f := findFactor(t, direct, FactorTEECertChain); strings.Contains(f.Detail, "aci_session_verified") {
t.Errorf("a response with its own quote is not in ACI mode: %+v", f)
}

full := build(&ACISessionResult{Pinned: pinned, Covered: all}, "")
for _, name := range all {
if f := findFactor(t, full, name); f.Status != Pass || !f.Enforced {
t.Errorf("%s: %+v, want an enforced pass", name, f)
}
}
}
108 changes: 108 additions & 0 deletions internal/attestation/compose.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@ import (
"crypto/sha256"
"crypto/subtle"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"log/slog"
"maps"
"regexp"
"strconv"
"strings"

"github.com/13rac1/teep/internal/jsonstrict"
Expand Down Expand Up @@ -40,6 +42,12 @@ func VerifyComposeBinding(appCompose string, mrConfigID []byte) error {
hex.EncodeToString(mrConfigID),
)
}
return checkAppComposeObject(appCompose)
}

// checkAppComposeObject requires appCompose, whose hash the quote binds, to
// be read the same way by teep and dstack.
func checkAppComposeObject(appCompose string) error {
// The quote binds these bytes, but teep reads the images from the
// docker_compose_file member. An app_compose object that teep and
// dstack read differently (a duplicate member, or a member that
Expand Down Expand Up @@ -259,3 +267,103 @@ func MergeComposeDigests(model, gateway ComposeDigests) (allDigests []string, di
}
return allDigests, digestToRepo
}

// mrConfigV3 is the launch document that a dstack 0.6 VMM binds into a TDX
// MR_CONFIG_ID of version 3 (MrConfigV3 in dstack-types mr_config.rs). The
// members are in sorted order and every value is ASCII, so json.Marshal
// writes the RFC 8785 form that dstack hashes.
type mrConfigV3 struct {
AppID string `json:"app_id"`
ComposeHash string `json:"compose_hash"`
GPUPolicyHash string `json:"gpu_policy_hash,omitempty"`
InitScriptHashes *[]string `json:"init_script_hashes,omitempty"`
InstanceID string `json:"instance_id,omitempty"`
KeyProvider string `json:"key_provider"`
KeyProviderID string `json:"key_provider_id,omitempty"`
Version int `json:"version"`
}

// appComposeV3 holds the app_compose members that the VMM copies into the
// MrConfigV3 document.
type appComposeV3 struct {
KeyProvider string `json:"key_provider"`
KeyProviderID string `json:"key_provider_id,omitempty"`
ManifestVersion string `json:"manifest_version"`
InitScript []string `json:"init_script,omitempty"`
}

// VerifyComposeBindingV3 checks a TDX MRConfigID of version 3: 0x03, then
// sha256("dstack-mr-config-v3:" || 0x00 || document), then 15 zero bytes.
// The quote carries no document, so teep builds the one that the dstack
// VMM builds (prepare_mr_config_v3 in dstack vmm/src/app/mr_config.rs):
// compose_hash is sha256(appCompose); key_provider, key_provider_id and,
// from manifest version 3, the sha256 of each init script come from
// appCompose; app_id, instance_id and, for a launch with GPUs,
// gpu_policy_hash come from the RTMR3 events. The caller must have checked
// that events reproduce the RTMRs of the quote.
func VerifyComposeBindingV3(appCompose string, events []EventLogEntry, gpu bool, mrConfigID []byte) error {
var ac appComposeV3
_, missing, err := jsonstrict.Unmarshal([]byte(appCompose), &ac)
if err != nil {
return fmt.Errorf("app_compose: %w", err)
}
if len(missing) > 0 {
return fmt.Errorf("app_compose: missing members %q", missing)
}
manifest, err := strconv.Atoi(ac.ManifestVersion)
if err != nil {
return fmt.Errorf("app_compose manifest_version: %w", err)
}
composeHash := sha256.Sum256([]byte(appCompose))
doc := mrConfigV3{ComposeHash: hex.EncodeToString(composeHash[:]), KeyProvider: ac.KeyProvider, KeyProviderID: ac.KeyProviderID, Version: 3}
if manifest >= 3 {
hashes := make([]string, len(ac.InitScript))
for i, script := range ac.InitScript {
sum := sha256.Sum256([]byte(script))
hashes[i] = hex.EncodeToString(sum[:])
}
doc.InitScriptHashes = &hashes
}
if doc.AppID, err = runtimeEventPayload(events, "app-id"); err != nil {
return err
}
if doc.InstanceID, err = runtimeEventPayload(events, "instance-id"); err != nil {
return err
}
if gpu {
if doc.GPUPolicyHash, err = runtimeEventPayload(events, "gpu-policy-hash"); err != nil {
return err
}
}
document, err := json.Marshal(doc)
if err != nil {
return fmt.Errorf("MrConfigV3 document: %w", err)
}
digest := sha256.Sum256(append([]byte("dstack-mr-config-v3:\x00"), document...))
expected := make([]byte, 48)
expected[0] = 3
copy(expected[1:], digest[:])
if subtle.ConstantTimeCompare(mrConfigID, expected) != 1 {
return fmt.Errorf("MRConfigID %s does not bind the MrConfigV3 document %s", hex.EncodeToString(mrConfigID), document)
}
return checkAppComposeObject(appCompose)
}

// runtimeEventPayload returns the payload of the one RTMR3 dstack runtime
// event with this name, as lowercase hex.
func runtimeEventPayload(events []EventLogEntry, name string) (string, error) {
var found [][]byte
for _, e := range events {
if e.IMR == 3 && e.EventType == DstackRuntimeEventType && e.Event == name {
payload, err := hex.DecodeString(e.EventPayload)
if err != nil {
return "", fmt.Errorf("%s event payload: %w", name, err)
}
found = append(found, payload)
}
}
if len(found) != 1 {
return "", fmt.Errorf("event log has %d %s events, want 1", len(found), name)
}
return hex.EncodeToString(found[0]), nil
}
10 changes: 9 additions & 1 deletion internal/attestation/dstack_defaults.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,17 +24,25 @@ var dstackMRSEAMAllow = map[string]struct{}{
"489e585f1c54bc5a02066c8c6ec21619ff0334ec6f21e07e2a35202c59183789c8057e7d97dd591bb08314b185819e72": {},
// TDX module 2.0.18 — Granite Rapids / Clearwater Forest (Intel, TDX Module Releases and Hashes)
"2d2de102461684f14c8d0984a09d895e3e9e15944ce020a03b977e1f114d5e1ed32ef666a47fd19a5851b3800eda3afa": {},
// TDX module 2.0.20 — Granite Rapids / Clearwater Forest: sha384 of the
// signed binary and the hash in its sigstruct (Intel, tdx-module.binaries
// 73b2a66e). Intel's hash table lists 57a1a66f... for 2.0.20.
"eddfa0c1750b39ba6c9b039c579a5039baea307f8290fc5e3636d70e6001749dc95a368b9d8a9c50c84d1c22b5642d3c": {},
}

// dstackMRTDAllow contains the TD virtual firmware measurement hashes for
// known dstack-nvidia image versions.
// known dstack and dstack-nvidia image versions.
var dstackMRTDAllow = map[string]struct{}{
// dstack-nvidia-0.5.4.1 (os_image_hash 9b69bb...)
"b24d3b24e9e3c16012376b52362ca09856c4adecb709d5fac33addf1c47e193da075b125b6c364115771390a5461e217": {},
// dstack-nvidia-0.5.5 (os_image_hash da9a3d...)
"f06dfda6dce1cf904d4e2bab1dc370634cf95cefa2ceb2de2eee127c9382698090d7a4a13e14c536ec6c9c3c8fa87077": {},
// dstack-nvidia-0.5.11 (os_image_hash a6eafc...); 0.5.12 has the same MRTD
"df4d77c857de652bd0fafa343a2026b51071ee03773d076d283abf1a7d36ff91ffa279ee4a617ac664f7e2a2c8d5891b": {},
// dstack-0.6.3, 0.6.4 and 0.6.5 (os_image_hash 38a6f9..., feff36...,
// 4771a2...): the TDX MRTD that the mkosi-os-v0.6.3, v0.6.4 and v0.6.5
// releases publish
"83baf004a06c83f6371d4d9e19c8bb1ead734ba7696e56d6b52271011394a6f24efecc27af3b5c1da41594dbfa706e97": {},
}

// DstackBaseMeasurementPolicy returns a MeasurementPolicy with the shared
Expand Down
8 changes: 4 additions & 4 deletions internal/attestation/dstack_defaults_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ import (
)

func TestDstackMRSEAMAllow(t *testing.T) {
if len(dstackMRSEAMAllow) != 7 {
t.Errorf("dstackMRSEAMAllow has %d entries, want 7", len(dstackMRSEAMAllow))
if len(dstackMRSEAMAllow) != 8 {
t.Errorf("dstackMRSEAMAllow has %d entries, want 8", len(dstackMRSEAMAllow))
}
// TDX modules 1.5.24 and 2.0.18, from Intel's table of module hashes.
for _, tc := range []struct{ name, value string }{
Expand All @@ -32,8 +32,8 @@ func TestDstackMRSEAMAllow(t *testing.T) {
}

func TestDstackMRTDAllow(t *testing.T) {
if len(dstackMRTDAllow) != 3 {
t.Errorf("dstackMRTDAllow has %d entries, want 3", len(dstackMRTDAllow))
if len(dstackMRTDAllow) != 4 {
t.Errorf("dstackMRTDAllow has %d entries, want 4", len(dstackMRTDAllow))
}
for h := range dstackMRTDAllow {
b, err := hex.DecodeString(h)
Expand Down
8 changes: 8 additions & 0 deletions internal/attestation/measurement_policy.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@ type MeasurementPolicy struct {
MRTDAllow map[string]struct{}
MRSeamAllow map[string]struct{}
RTMRAllow [4]map[string]struct{}
// ComposeHashAllow lists sha256(app_compose) values; only the gateway
// policy sets it.
ComposeHashAllow map[string]struct{}
}

// HasComposeHashPolicy reports whether a compose hash allowlist is configured.
func (p MeasurementPolicy) HasComposeHashPolicy() bool {
return len(p.ComposeHashAllow) > 0
}

// HasMRTDPolicy reports whether an MRTD allowlist is configured.
Expand Down
Loading