diff --git a/README.md b/README.md index 082c948f..b3b138b3 100644 --- a/README.md +++ b/README.md @@ -230,16 +230,16 @@ serving inference exposes none, and the core tee_* factors state that. See | 39 | `gateway_tee_quote_structure` | Gateway quote parses as valid QuoteV4 or SEV-SNP report | | 40 | `gateway_tee_cert_chain` | Gateway cert chain verifies against Intel or AMD root CA | | 41 | `gateway_tee_quote_signature` | Gateway quote signature valid | -| 42 | `gateway_tee_debug_disabled` | Gateway debug bit is 0 (production enclave) | +| 42 | `gateway_tee_debug_disabled` | Gateway debug bit is 0 (TDX TD_ATTRIBUTES or SEV-SNP guest policy) | | 43 | `gateway_tee_measurement` | Gateway measurements match the gateway policy allowlists | -| 44 | `gateway_tee_hardware_config` | Gateway RTMR[0] matches hardware config allowlist | +| 44 | `gateway_tee_hardware_config` | Gateway RTMR[0] matches hardware config allowlist; for an SEV-SNP gateway, the guest policy forbids debug, the migration agent and single-socket mode, allows SMT, and the firmware build and version meet the minimums | | 45 | `gateway_tee_boot_config` | Gateway RTMR[1] and RTMR[2] match boot config allowlists; not applicable to an SEV-SNP gateway, whose launch measurement (`gateway_tee_measurement`) covers the boot | | 46 | `gateway_tee_reportdata_binding` | Gateway REPORTDATA binding verified | -| 47 | `gateway_compose_binding` | Gateway sha256(app_compose) matches TDX MRConfigID | -| 48 | `gateway_cpu_id_registry` | Gateway CPU PPID verified in Proof of Cloud registry | -| 49 | `gateway_event_log_integrity` | Gateway event log replayed; all 4 RTMRs match quote | -| 50 | `gateway_tee_tcb_current` | Gateway TCB SVN meets minimum threshold (SEV-SNP) | -| 51 | `gateway_tee_tcb_not_revoked` | Gateway TCB SVN is not revoked (SEV-SNP) | +| 47 | `gateway_compose_binding` | Gateway sha256(app_compose) matches TDX MRConfigID; not applicable to an SEV-SNP gateway | +| 48 | `gateway_cpu_id_registry` | Gateway CPU PPID verified in Proof of Cloud registry; not applicable to an SEV-SNP gateway | +| 49 | `gateway_event_log_integrity` | Gateway event log replayed; all 4 RTMRs match quote; not applicable to an SEV-SNP gateway, which exposes no event log | +| 50 | `gateway_tee_tcb_current` | Gateway TCB SVNs (bootloader, TEE, SNP, microcode) meet the minimum thresholds (SEV-SNP) | +| 51 | `gateway_tee_tcb_not_revoked` | Same check as `gateway_tee_tcb_current` (SEV-SNP); teep does not read the AMD CRL, so this factor does not test revocation | diff --git a/README_ADVANCED.md b/README_ADVANCED.md index ad1267c4..b8a33d0b 100644 --- a/README_ADVANCED.md +++ b/README_ADVANCED.md @@ -134,14 +134,14 @@ This provides true end-to-end encryption: the shared secret is derived between t | Provider | Attestation | Channel Security | REPORTDATA Binding | |----------|-------------|------------------|--------------------| -| Venice AI | TDX + NVIDIA | E2EE (secp256k1 ECDH + AES-256-GCM) | `keccak256(enclave_pubkey)` + nonce | +| Venice AI | dstack: TDX + NVIDIA; ACI/1: gateway TDX, plus relayed upstream GPU evidence (absent from some responses) | E2EE (secp256k1 ECDH + AES-256-GCM): dstack to the model enclave; ACI/1 to the gateway | dstack: `keccak256(enclave_pubkey)` + nonce; ACI/1: `keccak256(gateway signing key)` + nonce in the gateway quote | | NEAR AI Direct | TDX + NVIDIA | TLS pinning (model subdomain) | `sha256(signing_address ‖ tls_fingerprint)` + nonce | | NEAR AI Cloud | TDX + NVIDIA | TLS pinning (gateway) | `sha256(signing_address ‖ tls_fingerprint)` + nonce | -| NanoGPT | TDX (dStack), or ACI/1 gateway TDX + NVIDIA | None by default (plaintext over TLS to NanoGPT); with `e2ee = true`, ACI/1: E2EE v2 (secp256k1 ECDH + AES-256-GCM) to the gateway | None (dStack format); ACI/1: `keccak256(gateway key)` + nonce in the gateway quote | +| NanoGPT | Chutes evidence relayed by NanoGPT: TDX + GPU evidence; dstack: TDX + NVIDIA; ACI/1: gateway TDX, plus relayed upstream GPU evidence | None (plaintext over TLS to NanoGPT). With `e2ee = true`, ACI/1 requires E2EE v2 (secp256k1 ECDH + AES-256-GCM) to the gateway; NanoGPT does not forward it, so no model is served | Chutes evidence: not verifiable (the relayed response carries no `e2e_pubkey`); dstack: `keccak256(signing key)` + nonce; ACI/1: `keccak256(gateway signing key)` + nonce in the gateway quote | | Chutes | TDX + GPU evidence | E2EE (ML-KEM-768 + ChaCha20-Poly1305) | `sha256(nonce_hex + e2e_pubkey_base64)` | | Phala Cloud | ACI/1 gateway TDX | E2EE v2 (secp256k1 ECDH + AES-256-GCM) to the gateway | `sha256` of the ACI/1 statement over the keyset digest and the nonce, in the gateway quote | -| Tinfoil Cloud | SEV-SNP | E2EE (HPKE X25519 + AES-256-GCM) | `sha256(nonce ‖ hpke_key ‖ tls_fp ‖ gpu_hash)` in REPORTDATA | -| Tinfoil Direct | SEV-SNP | E2EE (HPKE X25519 + AES-256-GCM) | `sha256(nonce ‖ hpke_key ‖ tls_fp ‖ gpu_hash)` in REPORTDATA | +| Tinfoil Cloud | SEV-SNP | E2EE (HPKE X25519 + AES-256-GCM) | `sha256(algorithm URI ‖ nonce ‖ sha256(crypto_material) ‖ sha256(device_evidence))` in REPORTDATA[0:32], zeros in [32:64]; `crypto_material` holds the HPKE key and the TLS key fingerprint | +| Tinfoil Direct (disabled since 2026-11-12) | SEV-SNP | E2EE (HPKE X25519 + AES-256-GCM) | `sha256(algorithm URI ‖ nonce ‖ sha256(crypto_material) ‖ sha256(device_evidence))` in REPORTDATA[0:32], zeros in [32:64]; `crypto_material` holds the HPKE key and the TLS key fingerprint | ## Verification Factor Reference @@ -215,14 +215,14 @@ exposes none, and the core `tee_*` factors state that. | 40 | `gateway_tee_quote_signature` | Signature over the gateway quote body is valid. | | 41 | `gateway_tee_debug_disabled` | Gateway debug bit is 0 (production enclave). | | 42 | `gateway_tee_measurement` | Gateway measurements match the gateway policy allowlists. Allowed to fail by default for NEAR Cloud and for the ACI/1 routes of Venice, Phala Cloud and NanoGPT: the allowlist changes only with a teep release, so a gateway release fails this factor, and teep reports the failure. | -| 43 | `gateway_tee_hardware_config` | Gateway RTMR[0] matches the hardware config allowlist. | +| 43 | `gateway_tee_hardware_config` | Gateway RTMR[0] matches the hardware config allowlist. For an SEV-SNP gateway, the guest policy forbids debug, the migration agent and single-socket mode, allows SMT, and the firmware build and version meet the minimums. | | 44 | `gateway_tee_boot_config` | Gateway RTMR[1] and RTMR[2] match the boot config allowlists. Not applicable to an SEV-SNP gateway, whose launch measurement (`gateway_tee_measurement`) covers the boot. | | 45 | `gateway_tee_reportdata_binding` | Gateway REPORTDATA binding verified: nearcloud binds `sha256(signing_address ‖ tls_fingerprint)`; Venice ACI/1 binds `keccak256(signing key)` + nonce; Tinfoil binds the HPKE key and endorsed section hashes. | | 46 | `gateway_compose_binding` | Gateway `sha256(app_compose)` matches TDX MRConfigID, with the `app_compose` rules of `compose_binding`. Enforced for Venice ACI/1 — the gateway publishes its manifest with digest-pinned images. | | 47 | `gateway_cpu_id_registry` | Gateway CPU PPID verified against the Proof of Cloud registry. | | 48 | `gateway_event_log_integrity` | Gateway event log replayed; all 4 RTMRs match the gateway TDX quote. | | 49 | `gateway_tee_tcb_current` | Gateway TCB SVN meets minimum threshold (SEV-SNP; TDX defers to Intel PCS collateral). | -| 50 | `gateway_tee_tcb_not_revoked` | Gateway TCB SVN is not revoked (SEV-SNP; TDX defers to Intel PCS collateral). | +| 50 | `gateway_tee_tcb_not_revoked` | Same check as `gateway_tee_tcb_current` for an SEV-SNP gateway; teep does not read the AMD CRL, so this factor does not test revocation. TDX defers to Intel PCS collateral. | ## TOML Configuration diff --git a/docs/attestation_gaps/aci_gateway.md b/docs/attestation_gaps/aci_gateway.md index 86e29ed4..95982868 100644 --- a/docs/attestation_gaps/aci_gateway.md +++ b/docs/attestation_gaps/aci_gateway.md @@ -13,10 +13,9 @@ A user who sends a prompt to one of these models gets a hardware-backed guarantee about the gateway that forwards the prompt, not about the machine that processes it. The gateway receives the user's traffic and forwards it over its own connection to an inference service. The GPUs that answer -prove their identity fresh for each request, but nothing given to the -client proves which host they are in or what software drives them. Since -2026-11-12 the Venice response carries no GPU evidence at all (residual -exposure 11). +can prove their identity fresh for each request, but nothing given to the +client proves which host they are in or what software drives them. For some +models the provider sends no GPU evidence at all (residual exposure 11). The three providers reach the gateway in different ways. Phala Cloud's clients connect to the gateway directly. Venice's and NanoGPT's clients @@ -87,14 +86,15 @@ field, so the document is ambiguous, not extended. The Chutes evidence inside a Phala Cloud session does not have this check yet. There, a different reading of `e2e_pubkey` fails, because the key must hash to the channel binding that REPORTDATA binds. -The specification and its limits are in -[ACI/1 Specification: Assurance Limits and Deployment Requirements](aci_specification.md). -**Two response variants.** Venice relays the attestation with its own echo -fields (`nonce`, `model`, `upstream_model`, `verified` and others). Phala -Cloud and NanoGPT return the gateway's own form: no echo fields, and an -`all_attestations` list with one copy of the attestation. teep accepts that -list only when it holds exactly one entry equal to the top-level fields. +**Three response variants.** Venice relays the attestation with its own echo +fields (`nonce`, `model`, `upstream_model`, `verified` and others). NanoGPT +returns the gateway's own form: no echo fields, and an `all_attestations` +list with one copy of the attestation. teep accepts that list only when it +holds exactly one entry equal to the top-level fields. Phala Cloud returns +the canonical report of `/v1/aci/attestation`: `api_version`, `attestation`, +`service_capabilities` and `workload_keyset_digest`, with no echo fields and +no GPU evidence. **Host-selected TLS binding.** The gateway terminates TLS for each of its public hostnames inside the gateway CVM. Its keyset lists one TLS key for @@ -149,7 +149,13 @@ NanoGPT relay the compatibility route. ### Enforced and verified - The gateway TDX quote: structure, Intel cert chain, signature, debug bit, - event log replay against all four RTMRs, Proof of Cloud registration. + event log replay against all four RTMRs. Proof of Cloud registration + (`gateway_cpu_id_registry`) is checked and allowed to fail; see below. +- For Phala Cloud, `aci_session_verified` and `aci_receipt_verified` are + enforced. teep sends a request only through attested sessions that passed + its audit, and `teep serve` checks the receipt of every successful + response. A stream reaches the client before that check (residual + exposure 12). - `gateway_tee_reportdata_binding`: for Venice and NanoGPT the quote's REPORTDATA binds the gateway's signing key (keccak256 address scheme) and the client nonce; for Phala Cloud it is sha256 of the §3.2 statement over @@ -193,20 +199,24 @@ NanoGPT relay the compatibility route. `dstacktee/dstack-ingress`. That image is not in the policy, so their replay reports it as unrecognized in `provider_signer_recognition` and `component_signature_recognition`, which are allowed failures. -- The relayed NVIDIA evidence of NanoGPT, and of Venice when the response - carries it: per-GPU cert chains and SPDM signatures verify, and the EAT - nonce matches the client nonce. Since 2026-11-12 the Venice response - carries none (residual exposure 11). The Phala Cloud report has none; its - GPU factors fail and are allowed (`PhalaCloudACIDefaultAllowFail`). +- The relayed NVIDIA evidence of NanoGPT: per-GPU cert chains and SPDM + signatures verify, and the EAT nonce matches the client nonce. On + 2026-11-14, 2 of 7 NanoGPT ACI/1 models (TEE/gemma-4-31b-it, + TEE/gpt-oss-120b) carried an empty EAT `evidence_list`; their enforced + `nvidia_signature` and `nvidia_nonce_client_bound` failed, and teep blocked + them. Venice evidence gets the same checks, but these two factors are + allowed to fail for Venice (residual exposure 11). The Phala Cloud report + carries no GPU evidence; its GPU factors are allowed to fail + (`PhalaCloudACIDefaultAllowFail`). - For Phala Cloud, `tls_key_binding`: the certificate key of `tee.redpill.ai` on the connection that carries the report must equal the keyset entry for that host, and `downstream_tls_binding` must name the same entry. Inference requests use connections pinned to that key; a connection of another key sends no request bytes. -- For Phala Cloud, every inference response must carry - `X-ACI-Keyset-Digest`; a response without it fails. A response of another - keyset digest is relayed (ACI/1 §5.2) and the cached report ends, so the - next request verifies the served keyset again (§3.1). +- For Phala Cloud, every successful inference response must carry + `X-ACI-Keyset-Digest`. A response without it, or with another keyset + digest, fails before teep relays it. The authorization ends, so the next + request verifies the served keyset again (§3.1). - Venice and NanoGPT: teep does not check `X-ACI-Keyset-Digest`. On the compatibility route the gateway quote binds the signing key, not the keyset digest (residual exposure 7), so the report has no hardware-bound @@ -216,9 +226,15 @@ NanoGPT relay the compatibility route. ### Failed and allowed (`ACIDefaultAllowFail`) -Every core factor that describes the inference host: the twelve `tee_*` -factors, `measured_model_weights`, `event_log_integrity`, -`cpu_id_registry`, and the model-tier supply-chain factors. `cpu_gpu_chain` +Every core factor that describes the inference host fails or is skipped, and +is allowed: the eleven `tee_*` factors and `intel_pcs_collateral`, +`measured_model_weights`, `event_log_integrity`, `cpu_id_registry`, and the +model-tier supply-chain factors. On 2026-11-14, of the supply-chain factors +only `compose_binding` and `component_recognition` failed; the others passed +or did not apply. `gateway_cpu_id_registry` is in the list too: a Proof of +Cloud outage must not block the gateway. On 2026-11-14 a Proof of Cloud peer +served an expired certificate, so the factor was skipped on every ACI/1 +report. `cpu_gpu_chain` fails because the GPUs cannot be bound to any attested CPU: the only attested CPU is the gateway's, and the GPUs are not in it. `aci_backend_named` fails because no attested evidence binds a backend for @@ -229,8 +245,8 @@ gateway MRTD/MRSEAM against the dstack base allowlist (`aci.GatewayMeasurementPolicy`, shared by the three providers), and a gateway MRTD that is not in it fails the factor without blocking. `nvidia_signature` and `nvidia_nonce_client_bound` are in the Venice list -(`VeniceACIDefaultAllowFail`), not in the shared ACI/1 list, because the -Venice response carries no GPU evidence (residual exposure 11). These +(`VeniceACIDefaultAllowFail`), not in the shared ACI/1 list, because some +Venice responses carry no GPU evidence (residual exposure 11). These render as failed but allowed, so the gap stays visible in every report; removing an entry from the list blocks every ACI/1 model of the three providers. @@ -256,9 +272,6 @@ the three providers. `TestVerifyRun_ConcurrentACIProviders` ([gateway ACI/1 replay tests](../../internal/integration/gateway_aci_test.go)) verifies the three providers' captures against the one pin set. -- **Venice, 2026-10-30:** the ACI/1 response for `e2ee-gpt-oss-120b-p` - carried no NVIDIA payload, so `nvidia_signature` and - `nvidia_nonce_client_bound` (enforced) failed and teep blocked the model. - In every capture the attested VM reports `num_gpus: 0` while the response carries GPU attestation entries (relayed evidence), `service_capabilities.serving` is `"aggregator"`, and @@ -305,9 +318,15 @@ date. Cloud or NRAS request for a session; and the bounded set narrows routing to the pinned upstreams. On 2026-11-03, for `qwen/qwen-2.5-7b-instruct` (one of four models that list only Phala), the gateway replaced the - session at each re-verification and refused the pin with HTTP 412 - `session_not_accepted`, so teep fails closed for that model; the other - three were not sent a request. See the + session at each re-verification and refused the pin with HTTP 412; on + 2026-11-14 that model passed `aci_session_verified` and + `aci_receipt_verified`. On 2026-11-14, 9 of 23 Phala Cloud models failed + the enforced `aci_session_verified` and were blocked. For 4, the sessions + came from `private-ai-verifier/near-ai-gateway/v1`, and for 2 from + `private-ai-verifier/tinfoil/v1`; teep does not verify the evidence of + either verifier. 1 had an MRTD that is not allowed. 2 had a + `proxy_image_sha256` channel binding, which is not a channel binding of + ACI/1. See the [Phala Cloud provider document](../providers/phalacloud/phalacloud_support.md#attested-sessions). 2. **The KMS root and gateway app id are trust on first use.** The accepted dstack-KMS root and gateway app id in `internal/provider/aci/keyset.go` @@ -331,8 +350,10 @@ date. report it gateway-only. A third party cannot do this: passing the ACI/1 list requires the real gateway's quote with the client nonce. Closing it needs a per-model expected-format pin. -4. **Gateway measurements change with each redeploy.** The gateway image is - a dev channel (`dstack-dev-*`); RTMR0-2 change with each redeploy, so +4. **Gateway measurements change with each redeploy.** On 2026-11-14 the + gateway's `vm_config`, which teep does not verify, named the image + `dstack-0.5.12`. RTMR0-2 change when the gateway image or its VM + configuration changes, and teep has no published list of the values, so `gateway_tee_hardware_config` and `gateway_tee_boot_config` are allowed to fail. MRTD/MRSEAM come from the shared dstack base list, which changes only with a teep release; `gateway_tee_measurement` is allowed @@ -388,16 +409,17 @@ date. request. For Phala Cloud, teep holds that key and can fetch the receipt; for NanoGPT, NanoGPT made the gateway request with its own key, so teep cannot fetch it. -11. **The Venice response carries no GPU evidence.** The attackers are the +11. **Some Venice responses carry no GPU evidence.** The attackers are the operator of the downstream inference host and Venice, which builds the - response. Since 2026-11-12 the Venice response leaves out the GPU - evidence, so a prompt can be served on any machine, with or without + response. Some Venice responses leave out the GPU evidence (2 of the 6 + Venice ACI/1 models that answered on 2026-11-14: `e2ee-glm-5-2-p` and + `e2ee-gpt-oss-120b-p`), so a prompt to those models can be served on any machine, with or without confidential GPUs, and nothing in the evidence shows it. The user keeps the guarantee about the gateway and the key that teep encrypts to, and loses the proof that real GPUs answered the request. Nothing stops it - today: teep reports the missing evidence on every Venice ACI/1 report - and does not block. The live response of 2026-11-12 names a member - `gpu_evidence_omitted` and has no `nvidia_payload`. `nvidia_signature` + today: teep reports the missing evidence on those Venice ACI/1 reports + and does not block. On 2026-11-14 those two responses set + `gpu_evidence_omitted` to `true` and had no `nvidia_payload`. `nvidia_signature` and `nvidia_nonce_client_bound` fail and are allowed to fail for Venice ACI/1 (`VeniceACIDefaultAllowFail`). They return to enforced when the response carries GPU evidence again. Phala Cloud and NanoGPT are not @@ -515,7 +537,6 @@ in `internal/provider/aci/policy.go`. Addresses residual exposure 6. ## References - **ACI/1 specification:** [`spec/aci.md` at `dd333b24`](https://github.com/Dstack-TEE/private-ai-gateway/blob/dd333b24324a96468bc19137fbcf6c45aeecc366/spec/aci.md); receipt timing (residual exposure 12): [`spec/aci.md`](https://github.com/Dstack-TEE/private-ai-gateway/blob/8d0018f81e8dbadfe7b733bb294a0350f82b338c/spec/aci.md) and [`spec/e2ee-v2.md`](https://github.com/Dstack-TEE/private-ai-gateway/blob/8d0018f81e8dbadfe7b733bb294a0350f82b338c/spec/e2ee-v2.md) at `8d0018f8` -- **Specification limits:** [ACI/1 Specification: Assurance Limits and Deployment Requirements](aci_specification.md) - **Related teep assessment:** [dstack_integrity.md](dstack_integrity.md) - **Provider documents:** [Phala Cloud](../providers/phalacloud/phalacloud_support.md), [NanoGPT](../providers/nanogpt/nanogpt_support.md) - **ACI/1 parsing and custody verification:** `internal/provider/aci/aci.go`, `internal/provider/aci/keyset.go` @@ -533,13 +554,18 @@ ACI/1 response with an empty `evidence.quote` is rejected at the parser, and a report with no verified gateway TDX result fails `evidence_verified`, which no `allow_fail` entry can suppress. -**Host evidence:** The twelve core `tee_*` factors, -`measured_model_weights`, `event_log_integrity`, `cpu_id_registry`, -`cpu_gpu_chain`, `aci_backend_named` and the model-tier supply-chain factors -are in `ACIDefaultAllowFail`, which an ACI/1 response selects for Venice, +**Host evidence:** The eleven core `tee_*` factors and +`intel_pcs_collateral`, `measured_model_weights`, `event_log_integrity`, +`cpu_id_registry`, `cpu_gpu_chain`, +`aci_backend_named` and the model-tier supply-chain factors (of which +`compose_binding` and `component_recognition` failed on 2026-11-14) are in +`ACIDefaultAllowFail`, which an ACI/1 response selects for Venice, Phala Cloud and NanoGPT; the Venice list adds `nvidia_signature` and `nvidia_nonce_client_bound` (residual exposure 11). Their other formats -keep their own lists. +keep their own lists. `gateway_cpu_id_registry` is in the same list, so a +Proof of Cloud outage does not block the gateway. For Phala Cloud, `aci_session_verified` and +`aci_receipt_verified` are not in `PhalaCloudACIDefaultAllowFail`, so they +are enforced. **E2EE:** `provider.GatewayBindsE2EEKey` returns true for an ACI/1 response of Venice, Phala Cloud and NanoGPT, so their E2EE authorization @@ -554,8 +580,7 @@ prediction, a JSON schema) is refused before it is sent. `aci_key_custody` requi signing key to be the keyset entry of that suite. The field coverage is in [API support](../api_support.md#phala-cloud-and-nanogpt-e2ee-v2-of-the-private-ai-gateway). -**Pending:** the live TLS key binding for Phala Cloud (residual exposure -8) in the authorization-store migration, receipt verification on `teep +**Pending:** receipt verification on `teep verify --reverify` (residual exposure 10), and NanoGPT forwarding E2EE v2, which it did not do on 2026-10-30 (residual exposure 9). If a provider passes on attested sessions, teep should verify them and move the core factors out of diff --git a/docs/attestation_gaps/dstack_integrity.md b/docs/attestation_gaps/dstack_integrity.md index c820dcbf..8c3be19a 100644 --- a/docs/attestation_gaps/dstack_integrity.md +++ b/docs/attestation_gaps/dstack_integrity.md @@ -151,6 +151,10 @@ Intel publishes MRSEAM hashes directly in their official TDX module release note | `685f891ea5c20e8f...` | **2.0.02** | Granite Rapids | (not yet observed) | | `d0d80c085166ba78...` | **1.5.20** | Sapphire/Emerald Rapids | tinfoil_v3_direct (glm-5-2 on B200) | | `5b38e33a64879589...` | **1.5.05** | Sapphire/Emerald Rapids | (Phala docs, older) | +| `489e585f1c54bc5a...` | **1.5.24** | Sapphire/Emerald Rapids | NEAR model CVMs (2026-11-14) | +| `2d2de102461684f1...` | **2.0.18** | Granite Rapids / Clearwater Forest | nearcloud gateway (2026-11-14) | + +The "Observed In" column records the capture that first showed each value. On 2026-11-14 the NEAR and Venice dstack model CVMs showed `489e585f` (1.5.24) and `7bf06328` (1.5.16), the private-ai-gateway `7bf06328`, and the nearcloud gateway `2d2de102` (2.0.18); no capture showed `49b66faa` (1.5.08). The complete list of MRSEAM-to-version mappings is available from Intel's release pages. Tinfoil (`tinfoilsh/tinfoil-python`) also maintains a curated list of accepted MRSEAM values with version labels. @@ -158,7 +162,7 @@ The complete list of MRSEAM-to-version mappings is available from Intel's releas #### MRTD: Deterministic Per Dstack Image Version -All captured attestation data across neardirect, venice, and nanogpt (nearcloud) share a single MRTD value: +The captures before 2026-11-14 across neardirect, venice, and nanogpt (nearcloud) shared a single MRTD value: ``` b24d3b24e9e3c16012376b52362ca09856c4adecb709d5fac33addf1c47e193da075b125b6c364115771390a5461e217 @@ -168,11 +172,15 @@ This value appears in 34 GitHub repositories, including dstack's own verifier fi MRTD is deterministic for a given dstack OS image version because it measures only the virtual firmware (OVMF) binary — it does not vary with CPU count, memory, or GPU configuration. -**Implication:** MRTD can be computed from the reproducibly-built dstack image using the `dstack-mr` tool from the dstack repository. The current value can also be pinned by observation since it is consistent across all providers sharing the same dstack version. When the dstack image is updated, the new MRTD can be re-derived. +On 2026-11-14 no capture had this value: the NEAR and Venice dstack model CVMs had `df4d77c8…` (dstack-nvidia-0.5.11 and 0.5.12), and the gateways had `7fe60787…`, which is in no teep allowlist. The NanoGPT model backends ran sek8s, not dstack (MRTD `261ce538…`). + +**Implication:** MRTD can be computed from the reproducibly-built dstack image using the `dstack-mr` tool from the dstack repository. Because all providers that run the same dstack version have the same MRTD, an operator can also pin the served value by observation. When the dstack image is updated, the new MRTD can be re-derived. #### RTMR0: Per-Deployment-Class (CPU/RAM/GPU Configuration) -RTMR0 measures the virtual hardware configuration (vCPU count, memory size, PCI hole, GPU count, NVSwitch count, hotplug, QEMU version). Three distinct values were observed across all captures: +The RTMR0–RTMR2 tables below record the captures before 2026-11-14. On 2026-11-14 every NEAR and Venice dstack model CVM had the same three values: RTMR0 `bc122d14…`, RTMR1 `4ae60980…` and RTMR2 `bd67ab75…`. + +RTMR0 measures the virtual hardware configuration (vCPU count, memory size, PCI hole, GPU count, NVSwitch count, hotplug, QEMU version). The captures before 2026-11-14 showed three values: | RTMR0 | Observed In | |-------|-------------| @@ -186,7 +194,7 @@ The Atlas `BOOTCHAIN-VERIFICATION.md` explains that RTMR0 varies with hardware c #### RTMR1: Per Kernel Version (2-3 Variants) -RTMR1 measures the Linux kernel binary. Three distinct values were observed: +RTMR1 measures the Linux kernel binary. The captures before 2026-11-14 showed three values: | RTMR1 | Observed In | |-------|-------------| @@ -200,7 +208,7 @@ The multiple values suggest either different dstack image builds or different ke #### RTMR2: Per Image Metadata (5 Variants) -RTMR2 measures the kernel command line, initrd, and rootfs. Five distinct values were observed, making it the most variable register. This variation likely reflects different rootfs configurations for different model deployments. +RTMR2 measures the kernel command line, initrd, and rootfs. The captures before 2026-11-14 showed five values, making it the most variable register. This variation likely reflects different rootfs configurations for different model deployments. **Implication:** Like RTMR0, the variation means RTMR2 values must be either provider-published or observed and pinned per deployment class. @@ -346,10 +354,17 @@ When a provider supplies a quote, event log, and compose manifest, teep can chec - verify the TDX quote structure and PCS collateral - verify caller binding through `REPORTDATA` where the provider-specific protocol supports it -- enforce `MRSEAM` and `MRTD` allowlists when configured (pinnable from Intel and dstack sources respectively) -- enforce `RTMR0`–`RTMR2` allowlists when configured (pinnable from observed reports via `--update-config`) -- verify `MRCONFIGID` against the published compose manifest -- replay the event log and check `RTMR3` consistency +- check `MRSEAM` and `MRTD` against allowlists under `tee_measurement` +- check `RTMR0` under `tee_hardware_config` and `RTMR1`–`RTMR2` under `tee_boot_config` +- verify `MRCONFIGID` against the published compose manifest (`compose_binding`) +- replay the event log and check `RTMR3` consistency (`event_log_integrity`) +- check the gateway's `MRSEAM` and `MRTD` under `gateway_tee_measurement`, and its `RTMR0`–`RTMR2` under `gateway_tee_hardware_config` and `gateway_tee_boot_config` (NEAR Cloud and the ACI/1 providers) + +teep ships default allowlists: `MRSEAM` and `MRTD` in `internal/attestation/dstack_defaults.go`, and `RTMR0`–`RTMR2` per provider in `internal/provider/{neardirect,nearcloud,venice,nanogpt}/policy.go`. Whether a failure blocks comes from each provider's default `allow_fail` list: + +- `tee_measurement`, `compose_binding` and `event_log_integrity` are enforced for neardirect, nearcloud and venice (dstack), and allowed for nanogpt and phalacloud. +- `tee_hardware_config` and `tee_boot_config` are allowed for every dstack provider. On 2026-11-14 they failed without blocking: `tee_boot_config` on every NEAR and Venice dstack report, and `tee_hardware_config` on every Venice dstack report. +- `gateway_tee_measurement` is allowed for nearcloud and for the ACI/1 providers. Boot-chain checks depend on the effective measurement policy. Verification of the bound compose does not establish approval of its application images; that @@ -357,38 +372,24 @@ requires separate publisher evidence. Because providers do not publish authenticated measurement baselines in-band, teep operators must either derive values from source (feasible for `MRSEAM` and `MRTD`, difficult for `RTMR0`–`RTMR2` without hardware specs) or bootstrap them from observation (`teep verify --update-config`). Either way, the operator bears the burden of detecting and responding to legitimate infrastructure changes, with no automated mechanism to distinguish a routine dstack upgrade from a compromised lower stack. -### Recommended Policy Configuration +### Policy Configuration -Based on the research findings, teep can immediately configure the following without provider cooperation: +teep ships the default allowlists named above. A list configured in `[providers..policy]` replaces the default for that provider and field; it is not added to it. A list in `[policy]` replaces the default of every provider, Chutes included. A configured list must therefore hold every value that the provider serves, or the factor fails. neardirect, nearcloud and venice enforce `tee_measurement`. On 2026-11-14 they served MRTD `df4d77c8…`. A configured `mrtd_allow` without that value blocks every request to them. The configuration format is TOML: -**mrseam_allow** — The set of Intel-published MRSEAM values for non-deprecated TDX module versions: -``` -mrseam_allow: - - "49b66faa451d19ebbdbe89371b8daf2b65aa3984ec90110343e9e2eec116af08850fa20e3b1aa9a874d77a65380ee7e6" # TDX 1.5.08 - - "7bf063280e94fb051f5dd7b1fc59ce9aac42bb961df8d44b709c9b0ff87a7b4df648657ba6d1189589feab1d5a3c9a9d" # TDX 1.5.16 - - "476a2997c62bccc78370913d0a80b956e3721b24272bc66c4d6307ced4be2865c40e26afac75f12df3425b03eb59ea7c" # TDX 2.0.08 - - "685f891ea5c20e8fa27b151bf34bf3b50fbaf7143cc53662727cbdb167c0ad8385f1f6f3571539a91e104a1c96d75e04" # TDX 2.0.02 +```toml +[providers.neardirect.policy] +mrtd_allow = [ + "df4d77c857de652bd0fafa343a2026b51071ee03773d076d283abf1a7d36ff91ffa279ee4a617ac664f7e2a2c8d5891b", # dstack-nvidia-0.5.11 and 0.5.12 +] ``` -**mrtd_allow** — The MRTD for the current dstack image version, derivable from reproducible build or observed: -``` -mrtd_allow: - - "b24d3b24e9e3c16012376b52362ca09856c4adecb709d5fac33addf1c47e193da075b125b6c364115771390a5461e217" # dstack-nvidia-0.5.4.1 -``` - -**rtmr1_allow** — Observed kernel measurements (stopgap until derivable): -``` -rtmr1_allow: - - "c0445b704e4c48139496ae337423ddb1dcee3a673fd5fb60a53d562f127d235f11de471a7b4ee12c9027c829786757dc" - - "6e1afb7464ed0b941e8f5bf5b725cf1df9425e8105e3348dca52502f27c453f3018a28b90749cf05199d5a17820101a7" - - "920eb831509b58bf83a554b5377dd5ce26d3f5182f14d33622ac24c1d343a0fa3c7bde746e55098ca30baf784dfd2556" -``` +`teep verify --update-config` writes only the values of the reports it reads; it does not copy the default list. For neardirect, `mrseam_allow` and `mrtd_allow` then hold only the observed values. On 2026-11-14 neardirect served two MRSEAM values (`489e585f` and `7bf06328`), so a list made from one report blocks the instances that have the other value. -**rtmr0_allow** and **rtmr2_allow** — Must be populated per deployment class; pin observed values as a drift-detection measure until providers publish. +`rtmr0_allow`–`rtmr2_allow` must be populated per deployment class; pin observed values as a drift-detection measure until providers publish. -### Diagnostic Change +### Measurement Values in Reports -The `buildMetadata` function in `report.go` now includes full hex values for `mrseam`, `mrtd`, and `rtmr0`-`rtmr3` in every verification report's metadata section. This makes it straightforward to extract the values needed for policy configuration from any teep verification run. +`buildMetadata` in `report.go` stores the full hex values of `mrseam`, `mrtd` and `rtmr0`–`rtmr3` in the report metadata. `teep serve` returns them in the JSON report at `GET /v1/tee/report`. The text report of `teep verify` does not show them; `teep verify --update-config` writes them to the configuration file. ### With Published Manifests @@ -405,4 +406,4 @@ With authenticated measurement manifests in place, teep could: That is the end state teep needs: fail-closed verification of both the **boot image** and the **application payload**, with measurement baselines that update reliably when providers roll out infrastructure changes. -For the Tinfoil V3 model specifically, the `sigstore_code_verified` factor covers the entire boot chain through a single verification step, replacing manual `mrseam_allow`, `mrtd_allow`, and `rtmr0_allow`–`rtmr2_allow` configuration. Dstack providers that adopt a similar Sigstore-based publication model would close the same gap. +For Tinfoil V3 on TDX, teep still checks `MRSEAM` against the dstack list, and checks `MRTD` and `RTMR0` under `tee_boot_config` against Tinfoil's hardware-measurements registry; `sigstore_code_verified` covers the code measurements. Dstack providers that adopt a similar Sigstore-based publication model would close the same gap. diff --git a/docs/attestation_gaps/gpu_cpu_binding.md b/docs/attestation_gaps/gpu_cpu_binding.md index eed61b52..ef09cdca 100644 --- a/docs/attestation_gaps/gpu_cpu_binding.md +++ b/docs/attestation_gaps/gpu_cpu_binding.md @@ -271,16 +271,24 @@ of Cloud registry via a multi-peer quorum protocol: side). Requires quorum (currently 3 peers). 2. **Stage 2:** Chains opaque partial signatures through peers sequentially to produce a signed JWT confirming registration status. -3. **Verification:** Validates the JWT (EdDSA signature + claims when a - signing key is configured; claims-only otherwise) and reports `Pass` if - the PPID is found in the registry, `Fail` if explicitly not found, `Skip` - if the registry query was attempted but failed (or in equivalent - skip-classified cases such as offline mode with a PPID / a device ID with - no registry), and `Fail` when no PoC result and no PPID/device ID exist, - i.e. no CPU ID registry check is available/configured. +3. **Verification:** Validates the JWT claims: the quote hash, the + timestamp, the expiry when present, and the machine ID reported by the + stage-1 peers. teep does not verify the JWT signature. + `cpu_id_registry` reports: + - `Pass` when the quorum returns a JWT. + - `Fail` when a peer answers HTTP 403 (not registered). + - `Skip` when the query fails, when teep is offline and has a PPID, or + when the evidence has a device ID but no registry. + - `Fail` when the evidence has no PPID and no device ID. + + `gateway_cpu_id_registry` follows the same rules, but reports `Skip` in + the last case. It is not applicable to a SEV-SNP gateway. ##### Limitations +- **Unverified JWT signature:** teep does not verify the JWT signature. + The last peer in the chain can therefore return a registered answer that + the other peers did not sign, and teep reports `Pass`. - **Registry freshness:** Point-in-time verification. Hardware could be physically moved after the last verification. Level 3 (continuous monitoring) mitigates this but is not universally deployed. @@ -304,7 +312,14 @@ of Cloud registry via a multi-peer quorum protocol: ##### Deployment status Deployed in teep today. `cpu_id_registry` is evaluated for inference CVM -attestation; `gateway_cpu_id_registry` for gateway CVM attestation. +attestation; `gateway_cpu_id_registry` for gateway CVM attestation. On +2026-11-14 neither factor passed for any model of any provider. In 50 +`cpu_id_registry` results, the trust-server.nillion.network peer answered +HTTP 403 "Machine is not whitelisted", which teep reports as not +registered; in the other 50, the evidence carried no PPID to query. In 36 +`gateway_cpu_id_registry` results, the trust-server.scrtlabs.com peer +served a TLS certificate that expired on 2026-09-07, so the 3-of-3 quorum +could not complete and the factor reported `Skip`. #### Option 2 — GPU Evidence Hash in TDX REPORTDATA (Software-Only) @@ -368,7 +383,7 @@ with a Gap 1 mitigation (Option 1, 3, 4, or 5). 2. **Reconstruct the expected REPORTDATA** using the same derivation formula. 3. **Compare** the reconstructed REPORTDATA against the TDX quote's REPORTDATA field using constant-time comparison. -4. **Promote `cpu_gpu_chain`** from hardcoded `Fail` to a computed factor: +4. **Promote `cpu_gpu_chain`** from its current `Fail` to a computed factor: - `Pass` if the REPORTDATA includes and matches the GPU evidence hash. - `Fail` if the GPU evidence hash does not match. - `Skip` if the provider does not implement this scheme. @@ -377,8 +392,8 @@ with a Gap 1 mitigation (Option 1, 3, 4, or 5). - **Backwards compatibility:** Providers that have not adopted the new REPORTDATA scheme will continue to produce attestations where `cpu_gpu_chain` - is `Skip` or `Fail`. The factor should remain outside `DefaultEnforced` - initially and be promoted once providers adopt the scheme. + is `Skip` or `Fail`. The factor should stay in the default `allow_fail` + lists until providers adopt the scheme, and then be enforced. - **Ordering constraint:** GPU evidence must be collected before the TDX report is generated. This is already the natural order in most attestation @@ -392,24 +407,26 @@ with a Gap 1 mitigation (Option 1, 3, 4, or 5). - **No hardware changes required.** This can be implemented entirely in the CVM application layer and the client verifier. -##### Tinfoil V3: Existing implementation of Option 2 +##### Tinfoil V3: Option 2 in Tinfoil's format, not yet verified by teep -Tinfoil's V3 attestation format is the first concrete implementation of Option 2. V3 extends the basic approach to include both GPU and NVSwitch evidence hashes in REPORTDATA: +Tinfoil's V3 attestation format binds device evidence into REPORTDATA. The current format carries the evidence in endorsed sections (`crypto_material`, `device_evidence`) and derives: ``` -REPORTDATA[0:32] = SHA-256(tls_key_fp || hpke_key || nonce || gpu_evidence_hash || nvswitch_evidence_hash) -REPORTDATA[32:64] = zeros +crypto_material_hash = SHA-256(crypto_material section bytes) +device_evidence_hash = SHA-256(device_evidence section bytes) +REPORTDATA[0:32] = SHA-256(algorithm URI || nonce || crypto_material_hash || device_evidence_hash) +REPORTDATA[32:64] = zeros ``` -Where `gpu_evidence_hash = SHA-256(raw_gpu_spdm_json)` and `nvswitch_evidence_hash = SHA-256(raw_nvswitch_spdm_json)`. The CPU hardware signs REPORTDATA as part of the TDX or SEV-SNP quote, so both hashes are hardware-authenticated. +`crypto_material` holds the TLS key fingerprint and the HPKE key, so REPORTDATA binds both keys, the client nonce and the device evidence. -The V3 attestation endpoint (`GET /.well-known/tinfoil-attestation?nonce=<64hex>`) collects fresh SPDM evidence from all GPUs and NVSwitches via NVML APIs with the client-supplied nonce passed through to the GPU hardware. Evidence is returned in the response as `gpu` and `nvswitch` JSON fields. For 8-GPU HGX systems, PCIe topology validation (8 GPUs + 4 NVSwitches mesh integrity) is enforced at boot time. +The CPU hardware signs REPORTDATA as part of the TDX or SEV-SNP report, so the device evidence hash is hardware-authenticated. -GPU attestation is boot-time fail-closed: if `nvattest` local verification fails, the CVM aborts boot. This is enforced by CVM code that is itself verified via Sigstore supply chain attestation. +teep does not read device evidence yet. It refuses a document whose `device_evidence` section carries items, which blocks the provider. Its verifier reports `gpu_bound=false`, so `cpu_gpu_chain` does not pass for Tinfoil. teep attests only the Tinfoil router: on 2026-11-14 the router served an empty `device_evidence` item list in all 14 `tinfoil_v3_cloud` runs, and `cpu_gpu_chain` failed, allowed by default. -Teep verifies the binding by extracting `gpu` and `nvswitch` as `json.RawMessage` (preserving raw bytes — re-serialization would break the hash), recomputing the evidence hashes, reconstructing expected REPORTDATA, and constant-time comparing against the verified CPU quote. When the binding verifies, `cpu_gpu_chain` is promoted to `Pass`. +Tinfoil states that its enclaves check the PCIe topology of 8-GPU HGX systems and abort boot when `nvattest` local verification fails. teep does not verify this. -Tinfoil V3's residual risks are the same as Option 2 in general: it does **not** address Gap 1 (TEE.fail). The GPU evidence relay attack described in [Stage 2 analysis](#gpu-evidence-relay-attack-stage-2-residual-risk) applies. Tinfoil does not currently implement Proof of Cloud (Option 1), DCEA (Option 3), or TDX Connect (Option 5). +When teep verifies Tinfoil's device evidence, Tinfoil V3 will have the residual risks of Option 2: it does **not** address Gap 1 (TEE.fail), and the GPU evidence relay attack described in [Stage 2 analysis](#gpu-evidence-relay-attack-stage-2-residual-risk) applies. Until then, Gap 2 stays open for Tinfoil. Tinfoil does not currently implement Proof of Cloud (Option 1), DCEA (Option 3), or TDX Connect (Option 5). #### Option 3 — vTPM / DCEA Platform-Mediated Binding (CPU Identity + GPU) @@ -923,7 +940,7 @@ infrastructure changes. Gap 2 is independently exploitable *without* TEE.fail — any attacker controlling two legitimate machines can splice GPU evidence from one into the -attestation response of the other. Option 1 closes this gap at the +attestation response of the other. Option 2 closes this gap at the application layer, which has standalone value. **Security posture after Stage 2:** Gaps 1 and 2 are each mitigated by @@ -1002,7 +1019,7 @@ The GPU device identity is genuine and matches the registry. **Options:** 1 + 2 + 3   **Effort:** vTPM + CVM agent + verifier   **Gap 1:** Full ✅   **Gap 2:** Full ✅ Prioritize Option 3 ([DCEA](https://arxiv.org/html/2510.12469v1) protocol). -NearAI / dstack / tinfoil should: +NearAI / dstack (and other TDX providers) should: - Expose a vTPM to TD guests. - Extend GPU SPDM device identity into vTPM PCRs. - Include the AK public key hash in TDX report_data. @@ -1063,9 +1080,10 @@ Authority already supports [Azure vTPM + TDX](https://docs.trustauthority.intel.com/main/articles/tutorial-azure-vtpm.html) attestation, so Azure support is plausible but unvalidated for DCEA. -For teep's current providers (NearAI, dstack), neither runs on a major cloud -provider's managed vTPM infrastructure — they would need to provision vTPM -support themselves regardless of cloud platform. This is the primary +Of teep's current TDX providers, none runs on a major cloud provider's +managed vTPM infrastructure; they would need to provision vTPM support +themselves regardless of cloud platform. teep attests only Tinfoil's +SEV-SNP router, to which DCEA does not apply. This is the primary deployment cost of Stage 3. Option 4 ([CCxTrust](https://arxiv.org/html/2412.03842v2) / CNCF) provides @@ -1249,21 +1267,22 @@ CPU identity binding: CPU-to-GPU binding: -- **`cpu_gpu_chain`** — hardcoded `Fail` in - [`report.go` `evalCPUGPUChain`](../../internal/attestation/report.go#L1024-L1032) - -GPU nonce binding: - -- **`nvidia_nonce_client_bound`** — spurious failures on nearcloud +- **`cpu_gpu_chain`** — `Fail` for every provider in + [`report.go` `evalCPUGPUChain`](../../internal/attestation/report.go#L2355-L2361); + its `Pass` branch needs a Tinfoil GPU hash binding, which teep does not + verify yet ### Current behavior `cpu_id_registry` and `gateway_cpu_id_registry` are evaluated via the Proof of -Cloud multi-peer quorum protocol (Option 1, deployed). `cpu_gpu_chain` is -hardcoded to `Fail` because no provider currently implements CPU-to-GPU -binding (Options 2–5). +Cloud multi-peer quorum protocol (Option 1, deployed). `cpu_gpu_chain` +fails for every provider. Tinfoil binds a device evidence section into +REPORTDATA, but the router that teep attests carries no GPU evidence, and +teep does not read device evidence yet. No other provider binds GPU +evidence to the CPU quote (Options 2–5). Once providers adopt Option 2 (GPU evidence hash in REPORTDATA), teep will -promote `cpu_gpu_chain` from hardcoded `Fail` to a computed factor. The -factor should remain outside `DefaultEnforced` initially and be promoted to -enforced once providers deploy the scheme. +verify the binding in `cpu_gpu_chain`. The factor is in the default +`allow_fail` list of every provider except `tinfoil_v3_direct`, which +enforces it and is disabled. It should stay allowed until providers deploy +the scheme, and then be enforced. diff --git a/docs/attestation_gaps/model_weights.md b/docs/attestation_gaps/model_weights.md index 2d03d0ae..98f4fe3c 100644 --- a/docs/attestation_gaps/model_weights.md +++ b/docs/attestation_gaps/model_weights.md @@ -1,6 +1,6 @@ # Model Weight Authentication: Runtime Weights Not Covered by TEE Measurements -TEE attestation proves the correctness of an inference provider's software stack — firmware, kernel, and application code — but does not prove which model weights the inference engine loaded. A provider could pass all hardware attestation checks while serving inference from a cheaper, quantized, or backdoored model. No compensating controls that are independently verifiable by external clients currently exist across the industry, except for Tinfoil's dm-verity attested model volumes. +TEE attestation proves the correctness of an inference provider's software stack — firmware, kernel, and application code — but does not prove which model weights the inference engine loaded. A provider could pass all hardware attestation checks while serving inference from a cheaper, quantized, or backdoored model. No compensating controls that are independently verifiable by external clients currently exist across the industry. Tinfoil's dm-verity attested model volumes would close the gap, but teep can check them only through `tinfoil_v3_direct`, which has been disabled since 2026-11-12. ## The Problem @@ -8,7 +8,7 @@ A TEE attestation proves the *software stack* is correct (firmware, kernel, appl This creates a gap that undermines the trust model: a compromised or dishonest provider could boot the correct software stack (passing all TEE attestation), then load a cheaper, quantized, or backdoored model. The TEE measurements would still verify, but the inference output would come from the wrong model. Users and downstream applications relying on a specific model's capabilities, safety alignment, or licensing terms have no way to independently verify which model is actually generating their responses. -This is an industry-wide limitation of the current TEE attestation technology stack, not a failure of any individual provider. No current provider — across any TEE framework — exposes model weight hashes or per-token output binding to external clients in a way that enables independent verification. The one exception is Tinfoil, which closes this gap by design using dm-verity attested model volumes. +This is an industry-wide limitation of the current TEE attestation technology stack, not a failure of any individual provider. No current provider — across any TEE framework — exposes model weight hashes or per-token output binding to external clients in a way that enables independent verification. Tinfoil closes this gap by design using dm-verity attested model volumes; a client can check them only with a quote from the enclave that runs the model, not through Tinfoil's router. ## Impact @@ -60,7 +60,7 @@ The gap manifests differently across provider architectures: ### Dstack providers: compose binding does not cover model data -For dstack-based providers, the attestation chain from TDX hardware through compose binding to Sigstore-verified container images is complete and independently verifiable. However, this chain proves which *container image* is deployed — not which model weights the inference engine loaded. The inference engine downloads weights from HuggingFace (or another source) at runtime, inside the already-measured container. No current dstack provider includes a model-specific image or model identity in the Docker Compose manifest. +For dstack-based providers, teep checks the chain from TDX hardware through compose binding to Sigstore-logged container images; two of its links (the RTMR1–2 allowlist and the Rekor lookup) are allowed to fail by default. However, this chain proves which *container image* is deployed — not which model weights the inference engine loaded. The inference engine downloads weights from HuggingFace (or another source) at runtime, inside the already-measured container. No current dstack provider includes a model-specific image or model identity in the Docker Compose manifest. ### Chutes: existing verification mechanisms exclude TEE instances @@ -147,20 +147,23 @@ For TEE instances, cllmv failures are logged but not enforced. TEE chutes rely o **Could cllmv ever be useful to external clients?** In theory, if Chutes published the cllmv hash algorithm specification (or open-sourced the native libraries), the session key or a client-derivable equivalent, and the salt/sub construction from publicly available data, then external clients could add a verification factor that validates per-token proofs. This would provide post-attestation runtime verification — proof that the inference engine is actually using the claimed model, not just that the correct software was booted. However, this would require Chutes to change their TEE trust model to include cllmv enforcement for TEE instances. -### Tinfoil: gap closed by design +### Tinfoil: gap closed by design, verifiable only in direct mode -Tinfoil closes the model weight gap using dm-verity attested model volumes (see [dm-verity attested model volumes (Tinfoil V3)](#dm-verity-attested-model-volumes-tinfoil-v3) in Remediation). Model weights are transitively authenticated through the Sigstore → code measurements → RTMR2 → kernel cmdline → config hash → dm-verity root hash chain. +Tinfoil closes the model weight gap using dm-verity attested model volumes (see [dm-verity attested model volumes (Tinfoil V3)](#dm-verity-attested-model-volumes-tinfoil-v3) in Remediation). The chain is Sigstore → code measurements → RTMR2 → kernel cmdline → config hash → dm-verity root hash. A client can follow it only with a quote from the enclave that runs the model; the `tinfoil_v3_cloud` router does not carry that quote. ### Summary of per-provider model weight evidence | Provider | Weight auth mechanism | Available to external clients? | |----------|----------------------|-------------------------------| -| Venice (dstack) | None | N/A | +| Venice (dstack and ACI/1) | None | N/A | | Nearcloud (dstack) | None | N/A | | Neardirect (dstack) | None | N/A | | Chutes (sek8s) — watchtower | SHA256 weight file probing, random byte range spot-checks | No (server-side only, TEE-excluded) | | Chutes (sek8s) — cllmv | Per-token HMAC binding model+revision | No (closed-source, TEE-exempt) | -| Tinfoil | dm-verity root hash in attested config → Sigstore bundle | Yes (transitive via Sigstore verification) | +| Phala Cloud | None | N/A | +| NanoGPT | None | N/A | +| Tinfoil (`tinfoil_v3_cloud`) | dm-verity root hash in attested config → Sigstore bundle | No: the router carries no quote of the enclave that runs the model | +| Tinfoil (`tinfoil_v3_direct`) | dm-verity root hash in attested config → Sigstore bundle | Yes, with that quote; disabled since 2026-11-12 | --- @@ -170,7 +173,7 @@ Three approaches are viable paths to model weight authentication with current or 1. **[IMA manifest verification](#ima-manifest-verification-requires-provider-api-enrichment)** provides per-file runtime hash verification of model weight files anchored to TDX hardware, but requires providers to enable IMA in their CVM kernels and expose the measurement log through their attestation evidence API. -2. **[Compose-attested model image identity](#compose-attested-model-image-identity-dstack-attestation-chain)** is the fastest path for dstack. It requires only that providers include Sigstore-signed model weight images in the Docker Compose manifest — infrastructure that teep already fully verifies today. Tinfoil's [`modelwrap`](https://github.com/tinfoilsh/modelwrap) tool can create such images. No kernel changes, no new APIs, no new attestation endpoints. The missing piece is provider adoption. +2. **[Compose-attested model image identity](#compose-attested-model-image-identity-dstack-attestation-chain)** is the fastest path for dstack. It requires only that providers include Sigstore-signed model weight images in the Docker Compose manifest — infrastructure that teep already checks today. Tinfoil's [`modelwrap`](https://github.com/tinfoilsh/modelwrap) tool can create such images. No kernel changes, no new APIs, no new attestation endpoints. The missing piece is provider adoption. 3. **[dm-verity attested model volumes (Tinfoil V3)](#dm-verity-attested-model-volumes-tinfoil-v3)** is the strongest approach. It provides block-level runtime integrity enforcement via dm-verity, with root hashes pinned in a measured configuration and authenticated through Sigstore. Already implemented by Tinfoil; the pattern is reusable by dstack providers. @@ -253,30 +256,30 @@ Intel TDX Hardware ├── RTMR1 ← Linux kernel measurement ├── RTMR2 ← Kernel cmdline + initrd + rootfs hash │ - └── RTMR3 → Event log (replayable) - │ - └── MRCONFIGID = 0x01 || SHA256(app_compose) + ├── RTMR3 → Event log (replayable) + │ + └── MRCONFIGID = 0x01 || SHA256(app_compose) + │ + └── app_compose contains Docker Compose manifest │ - └── app_compose contains Docker Compose manifest - │ - ├── image: registry/inference-engine@sha256: - │ │ - │ └── Sigstore/Rekor verification: - │ ├── Rekor transparency log entry exists - │ ├── DSSE signature verifies against Fulcio cert - │ ├── SET verifies against Rekor public key - │ ├── Merkle inclusion proof verifies - │ └── Fulcio OIDC provenance (source repo, commit, builder) + ├── image: registry/inference-engine@sha256: + │ │ + │ └── Sigstore/Rekor verification: + │ ├── Rekor transparency log entry exists + │ ├── DSSE signature verifies against Fulcio cert + │ ├── SET verifies against Rekor public key + │ ├── Merkle inclusion proof verifies + │ └── Fulcio OIDC provenance (source repo, commit, builder) + │ + └── image: registry/model-weights@sha256: │ - └── image: registry/model-weights@sha256: - │ - └── Same Sigstore/Rekor chain as above + └── Same Sigstore/Rekor chain as above ``` -Each link in this chain is already verified by teep: +Teep checks each link in this chain. Two checks are allowed to fail by default: the RTMR0–2 allowlists (`tee_hardware_config`, `tee_boot_config`) and the Rekor lookup of image digests (`sigstore_verification`). See the table below: 1. **TDX quote verification**: The quote signature is verified against Intel PCS collateral, proving it was generated by genuine TDX hardware. -2. **Base image authentication**: `MRTD` is deterministic for a given dstack OS image version and can be derived from the [reproducible build](https://github.com/Dstack-TEE/meta-dstack) using the [`dstack-mr` tool](https://github.com/Dstack-TEE/dstack/tree/master/dstack-mr). This proves the virtual firmware is the expected open-source dstack firmware. `RTMR1` and `RTMR2` prove the kernel and rootfs match the expected dstack image. See the [dstack attestation guide](https://github.com/Dstack-TEE/dstack/blob/master/attestation.md) for the full measurement derivation process. +2. **Base image authentication**: `MRTD` is deterministic for a given dstack OS image version and can be derived from the [reproducible build](https://github.com/Dstack-TEE/meta-dstack) using the [`dstack-mr` tool](https://github.com/Dstack-TEE/dstack/tree/master/dstack-mr). This proves the virtual firmware is the expected open-source dstack firmware. `RTMR1` and `RTMR2` would prove the kernel and rootfs match the expected dstack image; on 2026-11-14 they did not match the teep allowlist for any Nearcloud, Neardirect or Venice dstack model. See the [dstack attestation guide](https://github.com/Dstack-TEE/dstack/blob/master/attestation.md) for the full measurement derivation process. 3. **Compose binding**: `MRCONFIGID` is compared (using `subtle.ConstantTimeCompare`) against `0x01 || SHA256(app_compose)`, proving the Docker Compose manifest running in the CVM is exactly the one declared in the attestation evidence. 4. **Image digest verification**: Image digests (`@sha256:...`) are extracted from the compose manifest and looked up in the [Rekor transparency log](https://docs.sigstore.dev/logging/overview/). Each entry is verified: DSSE signature against the Fulcio certificate, Signed Entry Timestamp (SET) against the Rekor public key, and Merkle inclusion proof against the append-only log. [Fulcio](https://docs.sigstore.dev/certificate_authority/overview/) OIDC provenance metadata (source repository, commit SHA, CI/CD builder identity) is extracted where available. @@ -342,9 +345,9 @@ For this approach to work, providers must: | Verification step | Status | |------------------|--------| | TDX quote authenticity | **Verified today** | -| Base image measurements (MRTD, RTMR0–2) | **Verified today** (with allowlists) | +| Base image measurements (MRTD, RTMR0–2) | **Checked today**: MRTD under `tee_measurement`; RTMR0–2 under `tee_hardware_config` and `tee_boot_config`, allowed to fail by default. On 2026-11-14 `tee_boot_config` failed on every Nearcloud, Neardirect and Venice dstack model | | Compose binding (MRCONFIGID) | **Verified today** | -| Image digests in Sigstore/Rekor | **Verified today** | +| Image digests in Sigstore/Rekor | **Checked today**; `sigstore_verification` is allowed to fail by default for every dstack provider. On 2026-11-14 it failed on every Nearcloud, Neardirect and Venice dstack model | | DSSE signature + SET + inclusion proof | **Verified today** | | Fulcio OIDC provenance extraction | **Verified today** | | **Model weights image in compose** | **Not yet available** — no provider currently includes a model-specific image or model identity in the compose manifest | @@ -358,7 +361,7 @@ Tinfoil's V3 attestation uses [`modelwrap`](https://github.com/tinfoilsh/modelwr At runtime, the CVM mounts the model volume read-only with dm-verity enabled. The kernel validates every block read against the Merkle tree root hash. A tampered block causes an I/O error and inference fails. All disks are mounted read-only with ramdisk for ephemeral data, so runtime weight substitution is not possible. -The full authentication chain is: Sigstore bundle → code measurements → RTMR2 → kernel cmdline → config hash → dm-verity root hash → model weight volume. Teep verifies the Sigstore bundle and compares attestation registers against it; when `sigstore_code_verified` passes, the model weights are transitively authenticated, and `measured_model_weights` can be set to `Pass`. +The full authentication chain is: Sigstore bundle → code measurements → RTMR2 → kernel cmdline → config hash → dm-verity root hash → model weight volume. Teep verifies the Sigstore bundle and compares attestation registers against it. `measured_model_weights` passes when three things verify: a quote from the enclave that runs the model (the backend quote), the Sigstore bundle, and the code measurements. Only `tinfoil_v3_direct` gets a backend quote, and that provider has been disabled since 2026-11-12. The `tinfoil_v3_cloud` router carries no backend quote, so the factor fails there and is allowed to fail by default. This approach is strictly stronger than compose-attested images because it provides block-level runtime integrity enforcement — not just proof that an image was deployed, but cryptographic verification of every block of model data as it is read. The `modelwrap` tool and dm-verity volume scheme are not Tinfoil-specific; dstack providers could adopt the same pattern by building dm-verity model volumes, pinning their root hashes in the compose manifest or a measured configuration file, and mounting them read-only inside the CVM. @@ -446,7 +449,7 @@ Key capabilities and limitations: Despite the promise, several fundamental obstacles prevent teep from implementing reproducible inference verification today: -1. **Providers do not enable deterministic mode**. No current teep provider (Venice, Nearcloud, Neardirect, Chutes) is known to launch its inference engine with `--enable-deterministic-inference` (SGLang) or `VLLM_BATCH_INVARIANT=1` (vLLM). Without these flags, the `seed` parameter only controls the sampling random state — the forward pass itself remains non-deterministic due to varying batch sizes, making cross-instance output comparison unreliable. +1. **Providers do not enable deterministic mode**. No current teep provider is known to launch its inference engine with `--enable-deterministic-inference` (SGLang) or `VLLM_BATCH_INVARIANT=1` (vLLM). Without these flags, the `seed` parameter only controls the sampling random state — the forward pass itself remains non-deterministic due to varying batch sizes, making cross-instance output comparison unreliable. 2. **Same-hardware requirement**. Both SGLang and vLLM only guarantee determinism on identical hardware (same GPU model, same driver version, same CUDA version) and the same engine version. Teep's known-good reference and the target provider must run on the same GPU SKU. If Venice runs vLLM on H100s and teep's reference was generated on an A100, the outputs will differ even with deterministic mode enabled and the same model. This means teep would need to maintain per-hardware-configuration, per-engine-version reference baselines — a combinatorial explosion. @@ -482,7 +485,7 @@ Even without complete provider support, teep could implement a partial verificat | 2 (strongest) | dm-verity attested model volumes | Moderate — build dm-verity volumes, pin root hash | Block-level runtime enforcement | Implemented by Tinfoil; pattern reusable | | 3 (most impactful) | IMA manifest verification | Significant — enable IMA, expose log, coordinate RTMR | Per-file runtime hashes, hardware-anchored | Kernel infrastructure exists; needs provider enablement + API | -For dstack and sek8s providers, `measured_model_weights` will remain `Fail` until one of the viable approaches is adopted: +For every provider except `tinfoil_v3_direct`, `measured_model_weights` will remain `Fail` until one of the viable approaches is adopted: 1. **[Compose-attested model image identity](#compose-attested-model-image-identity-dstack-attestation-chain)** (fastest path for dstack): Include Sigstore-signed model weight images in the Docker Compose manifest (Tinfoil's [`modelwrap`](https://github.com/tinfoilsh/modelwrap) can create such images). Teep's existing verification chain already covers every link from TDX hardware to image identity. The only missing piece is provider adoption. @@ -539,13 +542,16 @@ The remaining approaches documented above (HuggingFace baselines, per-token outp ## Teep Status -Teep currently returns `Fail` for `measured_model_weights` with detail "no model weight hashes" for all providers except Tinfoil. No provider currently exposes model weight hashes or per-token output binding to clients in a way that teep can independently verify. +Teep returns `Fail` for `measured_model_weights`, with detail "no model weight hashes", for every provider that teep can use today. On 2026-11-14 the factor failed, allowed by default, on all 100 models that returned a report, across seven providers; five more models returned an attestation error and no report. No provider currently exposes model weight hashes or per-token output binding to clients in a way that teep can independently verify. | Provider | Weight auth mechanism | Available to teep? | Status | |----------|----------------------|-------------------|--------| -| Venice (dstack) | None | N/A | `Fail`: "no model weight hashes" | +| Venice (dstack and ACI/1) | None | N/A | `Fail`: "no model weight hashes" | | Nearcloud (dstack) | None | N/A | `Fail`: "no model weight hashes" | | Neardirect (dstack) | None | N/A | `Fail`: "no model weight hashes" | | Chutes (sek8s) — watchtower | SHA256 weight file probing, random byte range spot-checks | No (server-side only, TEE-excluded) | `Fail`: "no model weight hashes" | | Chutes (sek8s) — cllmv | Per-token HMAC binding model+revision | No (closed-source, TEE-exempt) | `Fail`: "no model weight hashes" | -| Tinfoil | dm-verity root hash in attested config → Sigstore bundle | Yes (transitive via `sigstore_code_verified`) | `Pass`: model weights authenticated via dm-verity + Sigstore chain | +| Phala Cloud | None | N/A | `Fail`: "no model weight hashes" | +| NanoGPT | None | N/A | `Fail`: "no model weight hashes" | +| Tinfoil (`tinfoil_v3_cloud`) | dm-verity root hash in attested config → Sigstore bundle | No: the router carries no backend quote | `Fail`: "no model weight hashes" | +| Tinfoil (`tinfoil_v3_direct`) | dm-verity root hash in attested config → Sigstore bundle | Yes, with a backend quote | Disabled since 2026-11-12. When enabled, the factor is enforced and passes only when the backend quote, the Sigstore bundle and the code measurements all verify | diff --git a/docs/attestation_gaps/nanogpt_chutes_evidence.md b/docs/attestation_gaps/nanogpt_chutes_evidence.md index 87fa3c94..0619e46d 100644 --- a/docs/attestation_gaps/nanogpt_chutes_evidence.md +++ b/docs/attestation_gaps/nanogpt_chutes_evidence.md @@ -99,6 +99,12 @@ evidence response. For each of the 10: `failed_instance_ids` was empty in 9 and held 2 ids in one response; the 2 ids had no entry in `evidence`. +On 2026-11-14, in the last of three capture runs that day, the same 10 +models answered in this format. No response carried an `e2e_pubkey`. The +responses listed 5 to 41 instances, and the bodies were 609,673 to +2,463,332 bytes. `failed_instance_ids` and `excluded_instance_ids` were +empty in all 10. + The meaning of `failed_instance_ids` and `excluded_instance_ids` is not documented. Their names suggest instances whose evidence could not be collected, and instances left out of the evidence. @@ -134,6 +140,8 @@ without network access, gave: the GPU evidence did not verify. All three are allowed. - The report header showed `Candidates: 1/6 evaluated`: the response listed 6 instances. +- The 2026-11-14 report of the same model gave the same factor results, + with `Candidates: 1/13 evaluated`. --- diff --git a/docs/attestation_gaps/near_image_provenance.md b/docs/attestation_gaps/near_image_provenance.md index 72f31c65..b2a9a0d5 100644 --- a/docs/attestation_gaps/near_image_provenance.md +++ b/docs/attestation_gaps/near_image_provenance.md @@ -1,7 +1,9 @@ # NEAR DeepSeek Image Provenance: Platform Digest Discovery Gap -NEAR's DeepSeek-V4.1-Flash-1017 deployments do not provide a reliable way to -find NEAR's signed build records for a deployed Docker image. Clients must +NEAR's model deployments do not provide a reliable way to find NEAR's signed +build records for the deployed launcher image. On 2026-11-14 every captured +NEAR model had this problem, and so did every captured Venice model that runs +the same launcher. Clients must either block requests when those records cannot be found or accept the image without verifying that NEAR's approved build process produced it. Accepting an unverified image can expose customer data to unapproved container code. @@ -15,7 +17,7 @@ authentication and encrypted connections can protect the connection to that code without establishing that NEAR's approved GitHub Actions workflow built it. NEAR publishes signed build records for its launcher image in Sigstore, but -these DeepSeek deployments identify the launcher using a different Docker +these deployments identify the launcher using a different Docker digest from the one that NEAR signed and logged. A client cannot find the signed record by looking up the digest in the deployed Docker Compose file. It must first find a Docker image index that contains the deployed image and @@ -137,12 +139,13 @@ checks that a compose with this pin passes the Rekor lookup, so `sigstore_verification` passes with the factor enforced; the Rekor answer is built in the test. -The DeepSeek deployments observed on 2026-11-11 pin a different digest of the -same build: the `linux/amd64` child manifest of that index. +The deployments captured on 2026-11-14 (and the DeepSeek deployments first +observed on 2026-11-11) pin a different digest of the same build: the +`linux/amd64` child manifest of that index. ```text nearaidev/compose-manager-launcher@sha256:91fdff3c… OCI image index; NEAR's CI signs and logs this digest -├── sha256:079a14b4… linux/amd64 manifest; the DeepSeek compose pins this digest +├── sha256:079a14b4… linux/amd64 manifest; the 2026-11-14 composes pin this digest └── sha256:c3e4bbb1… unknown/unknown build-attestation manifest ``` @@ -179,11 +182,16 @@ workflows move them. The lookup failure starts in the compose's launcher image reference. This report does not describe a change in the attestation response schema or the -Rekor record format. The reported affected models are: +Rekor record format. The first reported affected models were: - `deepseek-ai/DeepSeek-V4.1-Flash-1017` - `deepseek-ai/DeepSeek-V4.1-Flash-1017-abliterated` +On 2026-11-14 all 12 NearCloud and all 12 NearDirect model composes, +including `z-ai/glm-5.3-flash`, pinned the same launcher manifest +`compose-manager-launcher@sha256:079a14b4…`. Rekor returned no entries for +that digest (teep's report shows this as "HTTP 404"). + | Evidence | Launcher digest named by compose | Rekor lookup for the pinned digest | | --- | --- | --- | | `z-ai/glm-5.3-flash`, NearDirect and NearCloud captures of 2026-09-09 | Index `91fdff3c…` | Direct lookup finds Rekor entries; `sigstore_verification`, which checks that an entry exists, passes with enforcement enabled. | @@ -200,10 +208,10 @@ manifest `079a14b4…`: `sigstore_verification` fails for them and is allowed by default. This comparison establishes lookup behavior for the two image-reference forms. -These observations describe specific deployments and dates, not the whole -NEAR fleet or the present state of every backend. The GLM captures establish -the prior format; they do not prove that a live GLM backend still supplies -the same compose today. +These observations describe the backends that answered on the dates given, +not every backend in the NEAR or Venice fleets. The GLM captures of +2026-09-09 establish the prior index pin; the GLM captures of 2026-11-14 pin +the platform manifest. ### Finding the containing index through registry tags @@ -291,9 +299,9 @@ signature check. ### Publish signed records for the digest in Docker Compose -For the affected DeepSeek deployments, NEAR can sign the platform manifest -that the compose pins, or pin the signed index in the measured compose again. -Either choice removes reverse discovery. +NEAR can sign the platform manifest that its composes pin, or pin the +signed index in the measured compose again. Venice can pin the signed index +in its dstack composes. Either choice removes reverse discovery. Alternatively, publish a stable parent-index hint with the deployment evidence and retain the index by digest. The hint need not be trusted if the client checks the index hash, exact child membership and the index digest named in @@ -330,19 +338,18 @@ separate [dstack boot-measurement gap](dstack_integrity.md). ## Teep Status -The platform-manifest lookup failure is reported for the two DeepSeek -models listed in [Scope and evidence](#scope-and-evidence). The older signed -GLM captures pass that lookup, which checks only that a Rekor entry exists, -with enforcement enabled. Teep currently reports -`sigstore_verification` failures as allowed for both NEAR providers; the -exception therefore affects models that did not require it too. The -parent-index resolver described above is not implemented. +The platform-manifest lookup failure is reported for every NEAR model +observed on 2026-11-14 (see [Scope and evidence](#scope-and-evidence)): all +24 NEAR reports showed `sigstore_verification` failed, allowed, for +`sha256:079a14b4…`. The 5 Venice dstack reports pin the same digest, and all 5 +failed the factor the same way. Teep reports `sigstore_verification` +failures as allowed for both NEAR providers and for Venice. The parent-index +resolver described above is not implemented. ### Temporary teep policy -Both NEAR defaults allow `sigstore_verification` to fail to accommodate the -reported DeepSeek deployments. The older GLM captures pass without this -exception, but the default policy also relaxes their enforcement. This permits +The NEAR defaults and the Venice dstack default allow `sigstore_verification` +to fail to accommodate this launcher manifest. This permits absent entries and lookup errors for every model image, plus gateway images for NearCloud. The exception is not restricted to the affected launcher digest. Restoring default enforcement is a separate policy change and requires live diff --git a/docs/attestation_gaps/near_model_endpoints.md b/docs/attestation_gaps/near_model_endpoints.md index 768f1d56..b0f7a7a8 100644 --- a/docs/attestation_gaps/near_model_endpoints.md +++ b/docs/attestation_gaps/near_model_endpoints.md @@ -1,10 +1,11 @@ # NEAR Model Endpoints: No Endpoint in the Model List -NEAR's model list does not state which endpoint each model serves. A client -that checks end-to-end encryption with a test request cannot choose the right -request for a model that does not serve chat, so the encryption of five NEAR -models is not tested by teep's verifier today. The models still pass every -other check, and a real request confirms their encryption. +NEAR's model list does not state which endpoint each model serves. Teep's +verifier tests end-to-end encryption with a chat request, so it does not test +five NEAR models that serve other endpoints. A real request confirms the +encryption of three of them. The other two serve endpoints that NEAR does not +encrypt end to end, and teep refuses those endpoints while encryption is on. +No enforced check fails for any of the five. ## The Problem @@ -26,10 +27,10 @@ that NEAR does not document. test. - **What stops it today:** for the embeddings, reranking and image models, the first real request in `teep serve` checks the encryption, and a - decryption failure blocks the model. For the audio model with encryption on - and for `openai/privacy-filter`, `teep serve` refuses or has no route, so no - request is sent unencrypted, but nothing confirms the encryption. In - `teep verify`, nothing: the factor stays skipped. + decryption failure blocks the model. NEAR does not encrypt audio + transcription or the privacy endpoints end to end; while encryption is on, + `teep serve` refuses them and sends nothing. In `teep verify`, nothing: + the factor stays skipped. ## Technical Background @@ -65,8 +66,10 @@ set them apart, which is not a contract. - **NEAR:** add the served endpoint (or endpoints) to each entry of the model list, and document the endpoint of `openai/privacy-filter`. -- **teep:** test each model on the endpoint it serves, with the encryption - that teep already applies to embeddings, images and reranking. +- **teep:** test the embeddings, reranking and image models on the endpoint + each serves, with the encryption that teep already applies to those + endpoints. Teep has nothing to test for the audio model and + `openai/privacy-filter` until NEAR encrypts their endpoints. ## References @@ -80,10 +83,11 @@ set them apart, which is not a contract. warning. `e2ee_usable` stays skipped, enforced and not probed, with the detail "not probed: the model serves ; the verify probe sends a chat completion". It counts as skipped and does not block. A model that is not in -the list gets the chat test request, so a new model of another kind fails the -factor and is reported. - -`teep serve` does not change. A real embeddings, reranking or image request -confirms the encryption of those three models, as for any other model. `teep -serve` refuses audio transcription while encryption is on, and has no route -for `openai/privacy-filter`, so nothing confirms the encryption of those two. +the list gets the chat test request. A new model of another kind therefore +fails `e2ee_usable`, which is enforced, and teep blocks that model. + +In `teep serve`, a real embeddings, reranking or image request confirms the +encryption of those three models, as for any other model. NEAR does not +encrypt audio transcription or the privacy endpoints end to end; while +encryption is on, `teep serve` refuses them, so the other two models cannot +be used with encryption. diff --git a/docs/attestation_gaps/phalacloud_model_endpoints.md b/docs/attestation_gaps/phalacloud_model_endpoints.md index 1d17a9ce..96881588 100644 --- a/docs/attestation_gaps/phalacloud_model_endpoints.md +++ b/docs/attestation_gaps/phalacloud_model_endpoints.md @@ -28,9 +28,13 @@ a chat request with an error. test. They read no request through teep: teep sends these models no request that they serve. - **What stops it today:** `teep serve` has no Phala Cloud path for these - endpoints. It sends these models only chat completions, encrypted when E2EE - is on, and the gateway refuses them. Nothing confirms their encryption. In - `teep verify`, nothing: the factor stays skipped. + endpoints. For Phala Cloud it routes only chat completions and embeddings, + and it refuses embeddings while E2EE is on. On 2026-11-14 the gateway + answered a chat completion for `jaredpalmer/kev-4b` with HTTP 404, and + teep blocked the reports of `black-forest-labs/flux.2-klein-4b` and + `openai/whisper-large-v3`, so it sent them no request. Nothing confirms + the encryption of the four models. In `teep verify`, nothing: no probe + runs. ## Technical Background @@ -53,14 +57,15 @@ is not text: | `mistralai/voxtral-small-24b-2507` | audio → transcription | audio transcription | | `jaredpalmer/kev-4b` | text → decisions | `/v1/systemone` (named in the description) | -On 2026-11-14, `jaredpalmer/kev-4b` got the chat probe, and the gateway -answered "upstream returned HTTP 404: no route available for model -jaredpalmer/kev-4b". `e2ee_usable`, which Phala Cloud's policy allows to -fail, failed, and the report did not block. The reports of -`black-forest-labs/flux.2-klein-4b` and `openai/whisper-large-v3` were -blocked by `aci_session_verified`, because teep does not verify the evidence -of their gateway session, so they got no probe. The run of 2026-11-14 made -no report for `mistralai/voxtral-small-24b-2507`. +Two runs of `teep verify` on 2026-11-14 tested these models. In the first, +made before teep skipped the probe for them, `jaredpalmer/kev-4b` got the +chat probe, and the gateway answered "upstream returned HTTP 404: no route +available for model jaredpalmer/kev-4b". In both runs, +`aci_session_verified` blocked the reports of +`black-forest-labs/flux.2-klein-4b` and `openai/whisper-large-v3`, because +teep does not verify the evidence of their gateway session; a blocked report +gets no probe. Neither run made a report for +`mistralai/voxtral-small-24b-2507`. ## Remediation @@ -78,14 +83,18 @@ no report for `mistralai/voxtral-small-24b-2507`. ## Teep Status -`teep verify` sends no chat probe to the four models above and logs a -warning. `e2ee_usable`, which the policy allows to fail, stays skipped with +`teep verify` sends no chat probe to the four models above while E2EE is +on, the default. When the report does not block, teep logs a warning and two +factors stay skipped: `e2ee_usable`, which the policy allows to fail, gets the detail "not probed: the model serves ; the verify probe sends a -chat completion". `aci_receipt_verified`, which is enforced, stays skipped -with the detail "no probe: the model serves ; …". Neither blocks. -Because `e2ee_usable` is allowed to fail, a failed probe did not block -either: the skip changes the detail in the report, not whether the report -blocks. A model that is not in the table gets the chat probe, so a new model +chat completion", and `aci_receipt_verified`, which is enforced, gets the +detail "no probe: the model serves ; …". Neither blocks. A blocked +report gets no probe either and keeps its own details: `e2ee_usable` reads +"E2EE configured; pending live test" and `aci_receipt_verified` reads "no +probe: the report blocks inference requests". On 2026-11-14 the reports of +`black-forest-labs/flux.2-klein-4b` and `openai/whisper-large-v3` blocked, +so only `jaredpalmer/kev-4b` got the "not probed" details. Because +`e2ee_usable` is allowed to fail, a failed probe would not block either. A model that is not in the table gets the chat probe, so a new model without chat completions fails the factor and is reported. `teep serve` has no Phala Cloud path for these endpoints, so no request diff --git a/docs/attestation_gaps/sek8s_integrity.md b/docs/attestation_gaps/sek8s_integrity.md index e604c151..0d2c91f4 100644 --- a/docs/attestation_gaps/sek8s_integrity.md +++ b/docs/attestation_gaps/sek8s_integrity.md @@ -121,9 +121,9 @@ Verified using the client-generated nonce and the ML-KEM-768 public key from the ### Nonce echo -The evidence response echoes no nonce. `nonce_match` fails with "no echoed nonce; freshness rests on tee_reportdata_binding" and is allowed (`ChutesDefaultAllowFail`). Freshness rests on the enforced `tee_reportdata_binding`, which checks `SHA256(client_nonce_hex + e2e_pubkey)`. The GPU evidence is bound by the SPDM requester nonce derived from the same values (`chutes.GPURequesterNonce`), which the enforced `nvidia_nonce_client_bound` checks. +No verified member of the evidence response states the nonce. Only `attested_body` states it, under a signature whose key Chutes does not publish. On the direct Chutes path and in Phala Cloud sessions, `nonce_match` fails with "no echoed nonce; freshness rests on tee_reportdata_binding" and is allowed (`ChutesDefaultAllowFail`). Freshness rests on the enforced `tee_reportdata_binding`, which checks `SHA256(client_nonce_hex + e2e_pubkey)`. The GPU evidence is bound by the SPDM requester nonce derived from the same values (`chutes.GPURequesterNonce`), which the enforced `nvidia_nonce_client_bound` checks. -**Status:** Allowed to fail; the client nonce is checked by two enforced factors. +**Status:** Allowed to fail. On the direct path and in Phala Cloud sessions, two enforced factors check the client nonce. ### Other enforced factors @@ -133,13 +133,17 @@ The evidence response echoes no nonce. `nonce_match` fails with "no echoed nonce ### Chutes' signed evidence entries -The Chutes evidence response now gives each entry `tee_type`, a -`signature` and the `attested_body` that it signs (base64 JSON with the TDX -quote and the request nonce). Chutes publishes no key for the signature, -so teep does not verify it. In the attested sessions of Phala Cloud, teep -requires that `attested_body` carries the entry's quote and the nonce of -the evidence request; the quote itself is verified as above. The direct -Chutes path accepts these members without a check. +Each entry of the Chutes evidence response has a `tee_type`, a `signature` +and the `attested_body` that it signs (base64 JSON with the TDX quote, the GPU +evidence and the request nonce). Chutes publishes no key for the signature, +so teep does not verify it. In the attested sessions of Phala Cloud and in +NanoGPT's relay of Chutes evidence, teep requires that `attested_body` +carries the entry's quote and the nonce of the evidence request; the quote +itself is verified as above. This check does not show that the evidence is +fresh, because the signature is not verified. The direct Chutes path accepts +these members without a check. NanoGPT's relay names no instance key, so the +nonce checks above fail for it and teep blocks its requests (see +[nanogpt_chutes_evidence.md](nanogpt_chutes_evidence.md)). ### Server-side-only security mechanisms @@ -197,7 +201,7 @@ The event log and runtime IMA measurements are not exposed in the client-facing ### Gap 5: GPU-to-measurement binding not independently verifiable -Chutes' `TeeMeasurementConfig` includes `expected_gpus` and `gpu_count` fields, allowing the validator to reject a VM whose GPU inventory does not match the expected deployment class. Remote verifiers can verify GPU evidence independently via NVIDIA EAT tokens, but cannot verify that the Chutes validator applied the GPU-to-measurement binding correctly. +Chutes' `TeeMeasurementConfig` includes `expected_gpus` and `gpu_count` fields, allowing the validator to reject a VM whose GPU inventory does not match the expected deployment class. Remote verifiers can verify the SPDM evidence of each GPU on their own (`nvidia_signature`: certificate chain and signature) and through NVIDIA's NRAS service (`nvidia_nras_verified`), but cannot verify that the Chutes validator applied the GPU-to-measurement binding correctly. ### Gap 6: IMA manifest not exposed despite documentation claims @@ -219,16 +223,16 @@ The sek8s attestation model requires trusting Chutes to correctly implement seve ### Teep Factor Behavior -Teep correctly returns `Skip` for factors where the sek8s architecture does not expose verifiable data: +Teep reports these factors as not applicable (`N/A`), because the sek8s architecture exposes no data to verify them: -- **`build_transparency_log`**: "chutes attestation does not include container image metadata; supply chain verification is validator-side only" -- **`compose_binding`**: "chutes uses cosign image admission + IMA, not docker-compose; compose binding not applicable" -- **`sigstore_verification`**: "chutes attestation does not include container image digests; cosign verification is validator-side only" -- **`event_log_integrity`**: "chutes performs RTMR verification validator-side against a reference baseline; event log not exposed to clients" +- **`build_transparency_log`**, **`component_recognition`**, **`provider_signer_recognition`** and **`component_signature_recognition`**: "Chutes supply chain verification is validator-side only" +- **`compose_binding`**: "Chutes uses cosign image admission + IMA, not docker-compose" +- **`sigstore_verification`**: "Chutes cosign verification is validator-side only" +- **`event_log_integrity`**: "Chutes RTMR verification is validator-side; event log not exposed" Teep returns `Fail` for `measured_model_weights` with detail "no model weight hashes" because neither weight verification mechanism exposes verifiable data to clients. -Teep returns `Fail` for `nonce_match`, allowed, with detail "no echoed nonce; freshness rests on tee_reportdata_binding": the evidence response states no nonce (see [Nonce echo](#nonce-echo)). +Teep returns `Fail` for `nonce_match`, allowed, with detail "no echoed nonce; freshness rests on tee_reportdata_binding": no verified member of the evidence response states the nonce (see [Nonce echo](#nonce-echo)). --- @@ -237,7 +241,7 @@ Teep returns `Fail` for `nonce_match`, allowed, with detail "no echoed nonce; fr ### Short-term: Pin observed measurements (done) Sek8s measurement defaults have been captured and coded in `internal/provider/chutes/policy.go`: -- MRTD, RTMR0, RTMR1, RTMR2 pinned from captured Chutes quotes and from NEAR's reviewed mirror of Chutes' unsigned measurement feed (sek8s 1.3.1, 1.4.0 and 1.4.1; RTMR0 rows for the 8×H200, 8×B200, 8×B300 and 8×RTX PRO 6000 classes) +- MRTD, RTMR0, RTMR1, RTMR2 pinned from captured Chutes quotes and from NEAR's reviewed mirror of Chutes' unsigned measurement feed (sek8s 1.3.0, 1.4.0 and 1.4.1, and MRTD also for 1.3.1; RTMR0 rows for the 8×H200, 8×B200, 8×B300 and 8×RTX PRO 6000 classes) - MRSEAM allowlist extended for sek8s fleet TDX module versions (1.5.06, 1.5.28, 1.5.34) - Registered in `internal/defaults/defaults.go` @@ -296,7 +300,9 @@ This would eliminate residual trust assumption 3 (reference measurement correctn **Hardware measurement enforcement:** Partially enforced by default. MRTD and MRSEAM are pinned in `internal/provider/chutes/policy.go` and enforced fail-closed via `tee_measurement`. RTMR0, RTMR1, and RTMR2 are also pinned and registered in Teep defaults, but their corresponding checks (`tee_hardware_config` / `tee_boot_config`) are allow-fail by default for Chutes. REPORTDATA binding is enforced via `ReportDataVerifier` in `internal/provider/chutes/reportdata.go`. -**Supply chain factors:** Correctly return `Skip` for `build_transparency_log`, `compose_binding`, `sigstore_verification`, and `event_log_integrity` because the sek8s architecture does not expose the required evidence to clients. +**Fleet results (2026-11-14, 11 models):** `tee_boot_config` failed on 8 of 11 models, all on the same RTMR2 value (`e1de99043052f3e1…`). `tee_hardware_config` failed on 2 of 11 models. Both factors are allowed to fail, so teep reported these hosts and did not block them. All 11 report MRTD `261ce538…`, which sek8s 1.3.1, 1.4.0 and 1.4.1 share; the pins hold no 1.3.1 RTMR rows. + +**Supply chain factors:** Not applicable (`N/A`) for `build_transparency_log`, `component_recognition`, `provider_signer_recognition`, `component_signature_recognition`, `compose_binding`, `sigstore_verification`, and `event_log_integrity` because the sek8s architecture does not expose the required evidence to clients. **Model weights:** Returns `Fail` for `measured_model_weights`. This is the correct behavior — neither Chutes weight verification mechanism exposes verifiable data to clients. diff --git a/docs/attestation_gaps/tinfoil_cloud_integrity.md b/docs/attestation_gaps/tinfoil_cloud_integrity.md index 7bf039ae..15a84dbf 100644 --- a/docs/attestation_gaps/tinfoil_cloud_integrity.md +++ b/docs/attestation_gaps/tinfoil_cloud_integrity.md @@ -352,14 +352,16 @@ Until stages 2 and 3 exist, cloud mode should be described as router-attested bo ## Teep Status -Teep exposes two Tinfoil provider aliases with materially different verification subjects. Both use the fail-closed V3 attestation path, but a passing factor means only that the endpoint named in the corresponding column was verified. +Teep defines two Tinfoil provider aliases with materially different verification subjects. Both use the fail-closed V3 attestation path, but a passing factor means only that the endpoint named in the corresponding column was verified. `tinfoil_v3_direct` has been disabled since 2026-11-12, because Tinfoil's enclave hosts did not answer attestation requests. Teep therefore has no Tinfoil route that excludes the router. To enable direct mode again, Teep must also read GPU and NVSwitch evidence: today it refuses an enclave that serves device evidence, and without device evidence the enforced `nvidia_payload_present` and `cpu_gpu_chain` factors fail. The `tinfoil_v3_direct` column below describes the direct-mode design, not a route that serves requests. + +For `tinfoil_v3_cloud`, Teep accepts only a SEV-SNP router report and reports it in the Tier 4 `gateway_*` factors. Every gateway factor that applies to SEV-SNP is enforced, except `gateway_tee_cert_chain` and `gateway_tee_quote_signature`, which need a request to AMD KDS and are allowed to fail by default. The router answer carries no backend evidence, so the backend `tee_*` factors, `nvidia_payload_present`, `nvidia_signature`, `nvidia_claims`, `cpu_gpu_chain`, `nvswitch_binding` and `measured_model_weights` fail or skip, allowed by default. `cpu_id_registry` also fails, allowed by default. `nvidia_nonce_client_bound` and `nvidia_nras_verified` are not applicable. On 2026-11-14 the reports of all 14 `tinfoil_v3_cloud` models showed this pattern, and no enforced factor failed. | Property | `tinfoil_v3_cloud` | `tinfoil_v3_direct` | |---|---|---| | Endpoint Teep attests | Confidential model router | Selected per-model inference enclave | -| Fresh hardware evidence Teep verifies | Router CPU evidence and any hardware evidence belonging to the router boundary | Selected backend's CPU, GPU, and, when applicable, NVSwitch evidence and topology | +| Fresh hardware evidence Teep verifies | Router CPU evidence (SEV-SNP) | Selected backend's CPU evidence only (see above) | | Supply-chain release Teep verifies | `tinfoilsh/confidential-model-router` | Per-model deployment repository returned by discovery | -| Meaning of `measured_model_weights=Pass` | Router-mediated trust: the verified router release contains the backend-admission policy; Teep does not compare the selected backend to its per-model release | Direct trust: the selected backend's measured configuration matches its per-model Sigstore release and commits to the model volume's dm-verity root | +| `measured_model_weights` result | Always `Fail` (allowed by default): no backend quote | `Pass` means direct trust: the selected backend's measured configuration matches its per-model Sigstore release and commits to the model volume's dm-verity root | | Attested TLS and HPKE keys | Router keys; EHBP plaintext is available to the router workload | Selected backend keys; EHBP terminates in the inference enclave | | Data-plane authentication visible to Teep | Request and response are AEAD-authenticated to the router boundary | Request and response are AEAD-authenticated to the selected inference-enclave key | @@ -369,12 +371,12 @@ Teep currently applies Tinfoil EHBP to the complete request body and does not in ### Teep Boundary Selection for Router Risk -Teep's architectural mitigation is `tinfoil_v3_direct`: it addresses both TLS and EHBP to the selected inference endpoint and bypasses the confidential model router, including its plaintext parsers, rewriting logic, owned-tool loop, and helper-service fan-out. If Tinfoil later supports opaque routed EHBP, Teep could retain a cloud entry point while binding the body to the backend key; until then, cloud and direct mode are distinct trust-boundary choices. +The architectural mitigation is direct mode, which `tinfoil_v3_direct` implements and which is disabled (see above): it addresses both TLS and EHBP to the selected inference endpoint and bypasses the confidential model router, including its plaintext parsers, rewriting logic, owned-tool loop, and helper-service fan-out. If Tinfoil later supports opaque routed EHBP, Teep could retain a cloud entry point while binding the body to the backend key; until then, cloud and direct mode are distinct trust-boundary choices. A `tinfoil_v3_cloud` deployment may still reject router-specific tool controls before EHBP encapsulation and isolate authenticated request construction from model-controlled configuration. Such filtering can prevent unwanted use of known router profiles, but it is only defense in depth: it neither removes the router from the plaintext path nor eliminates vulnerabilities in the rest of its request-processing surface. Teep should therefore present direct mode, not tool-capability policy, as the remedy for clients that do not accept the router boundary. -`tinfoil_v3_direct` performs those additional checks at the inference boundary. Teep resolves the requested model to a backend domain and per-model deployment repository, obtains a client-nonce-bound attestation from that backend, validates its CPU and GPU evidence and applicable NVSwitch evidence, and compares its measured configuration with the authenticated Sigstore release for that model deployment. The release-authenticated configuration commits to a particular read-only model-weight volume through its dm-verity root; the measured boot chain authenticates that configuration, and dm-verity validates blocks as the inference runtime reads the weights. Teep then pins the live TLS connection and EHBP encryption to keys from the same backend attestation. Taken together, these checks establish that the selected enclave authorized to decrypt and serve the request is running the specific model deployment release and committed model-weight volume that Teep verified, rather than merely establishing that an acceptable backend exists somewhere behind an attested router. +In direct mode, Teep checks the inference boundary itself. It resolves the requested model to a backend domain and per-model deployment repository, obtains a client-nonce-bound attestation from that backend, validates its CPU evidence, and compares its measured configuration with the authenticated Sigstore release for that model deployment. The release-authenticated configuration commits to a particular read-only model-weight volume through its dm-verity root; the measured boot chain authenticates that configuration, and dm-verity validates blocks as the inference runtime reads the weights. Teep then pins the live TLS connection and EHBP encryption to keys from the same backend attestation. Taken together, these checks are designed to establish that the selected enclave authorized to decrypt and serve the request is running the specific model deployment release and committed model-weight volume that Teep verified, rather than merely establishing that an acceptable backend exists somewhere behind an attested router. The model-weight conclusion is deliberately transitive: Teep does not download and hash every weight file itself. It verifies `Sigstore release -> measured configuration -> dm-verity root`, while the attested kernel enforces the final block-level checks. A failure or skip anywhere in that chain must prevent Teep from reporting the model weights as verified. -Direct mode closes the router-boundary gaps, but it does not produce a durable backend-signed receipt over the request and response. Users requiring end-to-inference body encryption, direct verification of fresh backend hardware evidence, or verification of the specific model-weight release used must reject `tinfoil_v3_cloud` and use `tinfoil_v3_direct`. Users requiring backend-authenticated request attribution for later audit must wait until Tinfoil adds a request-bound backend signature or equivalent protocol evidence, or use NEAR AI. +Direct mode closes the router-boundary gaps, but it does not produce a durable backend-signed receipt over the request and response. Users requiring end-to-inference body encryption, direct verification of fresh backend hardware evidence, or verification of the specific model-weight release used must reject `tinfoil_v3_cloud`; Teep has no Tinfoil route for them today. Users requiring backend-authenticated request attribution for later audit must wait until Tinfoil adds a request-bound backend signature or equivalent protocol evidence, or use NEAR AI. diff --git a/docs/attestation_gaps/tinfoil_nvswitch_json.md b/docs/attestation_gaps/tinfoil_nvswitch_json.md index 410ceff3..663490dc 100644 --- a/docs/attestation_gaps/tinfoil_nvswitch_json.md +++ b/docs/attestation_gaps/tinfoil_nvswitch_json.md @@ -10,9 +10,13 @@ bytes. Because the NVSwitch evidence hash is part of the REPORTDATA preimage, this mismatch prevents the entire REPORTDATA hash from being verified — meaning the TLS SPKI, HPKE key, nonce, and GPU evidence hash cannot be authenticated via the hardware-signed attestation report. -Teep implements a workaround that injects the reported hash value into -the preimage, restoring REPORTDATA authentication for all components -except NVSwitch evidence binding. + +This document describes Tinfoil's superseded attestation format. teep +refuses a document in that format. The current format hashes the +base64-decoded bytes of each endorsed section, so this re-encoding +mismatch cannot occur in it. The teep workaround that this document +described was removed when teep moved to the current format. The +[Teep Status](#teep-status) section gives the current results. ## The Problem @@ -55,8 +59,9 @@ from another would not be detected by the REPORTDATA check. ## Impact -**Security impact:** Without the REPORTDATA workaround, the NVSwitch -hash mismatch prevents hardware-signed authentication of the TLS SPKI, +**Security impact (superseded format):** For a verifier without a +workaround, the NVSwitch hash mismatch prevented hardware-signed +authentication of the TLS SPKI, HPKE key, nonce, and GPU evidence hash for all 8-GPU Hopper Tinfoil inference enclaves. While each component is independently verified by separate factor checks (TLS SPKI by `tls_key_binding`, HPKE key by @@ -68,13 +73,10 @@ key, and GPU evidence. Without it, a relay attack that substitutes keys or evidence from a different enclave would not be detected by the hardware binding check. -With the workaround, the TLS SPKI, HPKE key, nonce, and GPU evidence -hash are authenticated via the hardware-signed REPORTDATA. Only the -NVSwitch evidence hash binding remains unverified — an attacker who -could substitute NVSwitch evidence in transit would not be detected by -the REPORTDATA hash check. However, the NVSwitch evidence itself (SPDM -reports, certificate chains) is still present and independently -verifiable, and the `nvswitch_binding` factor fails closed. +A verifier that used the reported hash in the preimage, as teep did +before it moved to the current format, authenticated the TLS SPKI, HPKE +key, nonce, and GPU evidence hash via the hardware-signed REPORTDATA. +Only the NVSwitch evidence hash binding remained unverified. The GPU evidence hash (for the `gpu` field) is NOT affected by this bug — GPU evidence is collected via `json.Marshal` in Go, which @@ -82,20 +84,18 @@ produces compact JSON from the start, so the hash matches the response bytes. Only NVSwitch evidence, collected via the external `nvattest` CLI, is affected. -**Operational impact:** All 8-GPU Hopper Tinfoil inference enclaves -(e.g. `glm-5-2`) have the `nvswitch_binding` factor fail closed, -blocking requests to these enclaves. Single-GPU enclaves -(e.g. `gemma4-31b`) are unaffected because they do not serve NVSwitch -evidence. +**Operational impact:** none in the current format, which hashes the +bytes that the enclave sends. --- ## Technical Background -### V3 Attestation Document Structure +### V3 Attestation Document Structure (superseded format) -The Tinfoil V3 attestation response is a JSON document with the -following structure: +The superseded Tinfoil V3 attestation response was a JSON document with +the following structure. teep refuses a document that carries any of +the `report_data`, `certificate`, `signature` or `cpu` members: ```json { @@ -185,9 +185,10 @@ data due to: ### Server Source Code Analysis -The bug is in -[`CollectNVSwitchEvidence`](https://github.com/tinfoilsh/confidential-cvmimage/blob/main/tinfoil/internal/attestation/gpu.go) -in the `cvmimage` repository: +The bug was in `CollectNVSwitchEvidence` +(`tinfoil/internal/attestation/gpu.go`) in the +[`tinfoilsh/cvmimage`](https://github.com/tinfoilsh/cvmimage) repository, +in the version that served the superseded format: ```go func CollectNVSwitchEvidence(nonce [32]byte) (json.RawMessage, error) { @@ -224,13 +225,16 @@ without normalizing it. This raw output is then: The GPU evidence path does not have this issue because `CollectGPUEvidence` returns a Go struct (`*GPUEvidenceCollection`), which is marshaled with `json.Marshal(gpuEvidence)` at the call site -(line 335 of `api.go`), producing compact Go-native JSON. The same +in `api.go`, producing compact Go-native JSON. The same compact bytes are used for both the hash and the response. ### Empirical Verification -Two attestation responses were fetched from +Two attestation responses in the superseded format were fetched from `glm-5-2.tinfoil.containers.tinfoil.dev` (8-GPU Hopper TDX enclave). +These results were not reproduced on 2026-11-15: teep refuses the +superseded format, and `tinfoil_v3_direct` has been disabled since +2026-11-12. For each response, the NVSwitch JSON bytes were extracted from the raw HTTP response body and hashed with SHA-256. The computed hash was compared against the `report_data.nvswitch_evidence_hash` field. @@ -251,29 +255,10 @@ collection path. ### Teep Report Factor Behavior -Teep implements a workaround that preserves REPORTDATA authentication -for all components except NVSwitch: - -1. `tee_reportdata_binding` → **Pass**: When `verifyNVSwitchEvidenceHash` - fails, `VerifyReportData` does not return an error immediately. - Instead, it proceeds to verify the REPORTDATA hash using the - **reported** `nvswitch_evidence_hash` from `report_data` (which is - what the server bound into REPORTDATA). If the REPORTDATA hash - matches, the TLS SPKI, HPKE key, nonce, and GPU evidence hash are - all authenticated by the hardware signature. The detail string - includes `nvswitch_bound=false` to indicate the NVSwitch hash - binding is broken. - -2. `cpu_gpu_chain` → **Pass**: Because `GPUHashBound` is true (the GPU - evidence hash was verified and the REPORTDATA hash matches), - `evalCPUGPUChain` returns Pass. +teep no longer evaluates the superseded format. The current results are +in [Teep Status](#teep-status). -3. `nvswitch_binding` → **Fail**: Because `NVSwitchExpected` is true - (8-GPU Hopper topology) and `NVSwitchHashBound` is false, - `evalNVSwitchBinding` returns Fail with "NVSwitch evidence hash - mismatch (server-side JSON re-encoding bug)". - -### Why the Workaround Is Necessary +### Why the Workaround Was Necessary (superseded format) The REPORTDATA hash authenticates ALL of these components as a group: @@ -311,13 +296,16 @@ safe because: The only component that is NOT authenticated is the NVSwitch evidence hash binding — we cannot confirm that the NVSwitch evidence in the -response matches what the server hashed. The `nvswitch_binding` factor -correctly fails closed for this. +response matches what the server hashed. --- ## Remediation +The current format resolves this mismatch: each endorsed section is +hashed over the bytes that the enclave sends. The fix below applied to +the superseded format. + ### Server-Side Fix (Required) The fix is in `CollectNVSwitchEvidence` in the `cvmimage` repository. @@ -370,44 +358,27 @@ return json.RawMessage(compacted), nil This guarantees the hash is computed over Go-native JSON bytes that `json.Marshal` will reproduce exactly in the response. -### Client-Side Workaround (Implemented) - -Teep implements a workaround that preserves REPORTDATA authentication -for all components except NVSwitch. When `verifyNVSwitchEvidenceHash` -fails (the computed hash doesn't match the reported hash), teep does -NOT fail `VerifyReportData` immediately. Instead, it proceeds to verify -the REPORTDATA hash using the **reported** `nvswitch_evidence_hash` -from `report_data`. If the REPORTDATA hash matches, the TLS SPKI, HPKE -key, nonce, and GPU evidence hash are all authenticated by the hardware -signature. - -The `nvswitch_binding` factor still fails closed, but `tee_reportdata_binding` -and `cpu_gpu_chain` pass. This is a significant security improvement over -failing all three factors, which would leave the TLS SPKI, HPKE key, and -GPU evidence unauthenticated via REPORTDATA. - -This workaround does NOT attempt to reproduce the server's hash -computation (which would require re-implementing the `nvattest` CLI's -C++ `nlohmann/json` serialization in Go). Instead, it trusts the -reported hash value for the REPORTDATA preimage, which is safe because -the REPORTDATA hash itself is hardware-signed and the other components -are independently verified. +### Client-Side Workaround (removed) + +teep removed the workaround, which used the reported +`nvswitch_evidence_hash` in the REPORTDATA preimage, when it moved to +the current format. teep refuses the superseded format. ### Deployment Priority -1. **Server-side fix** (required, blocks NVSwitch binding for all 8-GPU Hopper enclaves) -2. **Client-side workaround** (implemented — preserves REPORTDATA authentication for all components except NVSwitch) +1. **Server-side fix** (the current format, which hashes the + base64-decoded section bytes, does not have this mismatch) +2. **teep:** read GPU and NVSwitch evidence in the current format --- ## References -- **Server source code:** - [`CollectNVSwitchEvidence`](https://github.com/tinfoilsh/confidential-cvmimage/blob/main/tinfoil/internal/attestation/gpu.go) - in `tinfoilsh/confidential-cvmimage` -- **Server attestation builder:** - [`BuildAttestation`](https://github.com/tinfoilsh/confidential-cvmimage/blob/main/tinfoil/internal/attestation/attestation.go) - in `tinfoilsh/confidential-cvmimage` +- **Server source code:** `CollectNVSwitchEvidence` + (`tinfoil/internal/attestation/gpu.go`) and `BuildAttestation` + (`tinfoil/internal/attestation/attestation.go`) in + [`tinfoilsh/cvmimage`](https://github.com/tinfoilsh/cvmimage), in the + version that served the superseded format (not pinned to a commit) - **nvattest CLI:** Built from [NVIDIA/attestation-sdk](https://github.com/NVIDIA/attestation-sdk) using `nlohmann/json` for serialization @@ -419,27 +390,20 @@ are independently verified. ## Teep Status -**Affected factors (with workaround):** -- `tee_reportdata_binding` — **Pass** (REPORTDATA hash verified using - reported NVSwitch hash; TLS SPKI, HPKE key, nonce, and GPU evidence - hash all authenticated by hardware signature) -- `cpu_gpu_chain` — **Pass** (GPUHashBound=true; GPU evidence hash - verified and REPORTDATA hash matches) -- `nvswitch_binding` — **Fail** (NVSwitchExpected=true but - NVSwitchHashBound=false; NVSwitch evidence hash does not match raw - JSON bytes due to server-side JSON re-encoding bug) - -**Teep behavior:** When the NVSwitch evidence hash doesn't match the -raw JSON bytes, teep logs a warning and proceeds to verify the -REPORTDATA hash using the reported `nvswitch_evidence_hash`. This -preserves hardware-signed authentication of the TLS SPKI, HPKE key, -nonce, and GPU evidence hash. The `nvswitch_binding` factor fails -closed with a clear message indicating the server-side bug. - -**Post-fix enforcement:** Once the server-side fix is deployed, teep -will automatically verify the NVSwitch evidence hash correctly (the -`verifyNVSwitchEvidenceHash` check will pass, setting -`nvswitch_bound=true` in the detail string). The `nvswitch_binding` -factor will pass for 8-GPU Hopper enclaves. No teep code changes are -required — the workaround automatically detects when the hash matches -and sets `nvswitch_bound=true`. +teep refuses the superseded format described above, and the workaround +is removed. On 2026-11-14, every `tinfoil_v3_cloud` model (14 of 14) +gave: + +- `tee_reportdata_binding` — **Fail**, allowed: the router answer carries + no backend TEE quote or report. +- `cpu_gpu_chain` — **Fail**, allowed: the router, the only attested + machine, serves an empty device evidence section, so no GPU is bound to + the CPU report. +- `nvswitch_binding` — **Skip**, allowed: no GPU evidence. +- `gateway_tee_reportdata_binding` — **Pass**, enforced: + `v3: reportdata_hash verified, nonce_bound=true, gpu_bound=false`. + +`nvswitch_binding` is in the default allow_fail lists of both Tinfoil +providers. teep does not yet read GPU or NVSwitch evidence in the +current format, and refuses an enclave that serves device evidence. +`tinfoil_v3_direct` has been disabled since 2026-11-12.