diff --git a/docs/attestation_gaps/README.md b/docs/attestation_gaps/README.md index c11cb588..22381896 100644 --- a/docs/attestation_gaps/README.md +++ b/docs/attestation_gaps/README.md @@ -18,6 +18,8 @@ New reports should follow [TEMPLATE.md](TEMPLATE.md). - [Model Weight Authentication](model_weights.md) - [Proof of Cloud Registry: JWT Validation Failure](poc_jwt_validation.md) - [Chutes/sek8s Integrity Chain](sek8s_integrity.md) +- [Tinfoil Model Endpoints: Verify Probe Covers Chat Only](tinfoil_model_endpoints.md) +- [Phala Cloud Model Endpoints: No Endpoint Field in the Model List](phalacloud_model_endpoints.md) - [ACI/1 Gateway: Attested Gateway, Unattested Inference Host](aci_gateway.md) - [NanoGPT Chutes Evidence: No Nonce Binding, No Serving Instance](nanogpt_chutes_evidence.md) diff --git a/docs/attestation_gaps/phalacloud_model_endpoints.md b/docs/attestation_gaps/phalacloud_model_endpoints.md new file mode 100644 index 00000000..1d17a9ce --- /dev/null +++ b/docs/attestation_gaps/phalacloud_model_endpoints.md @@ -0,0 +1,92 @@ +# Phala Cloud Model Endpoints: No Endpoint Field in the Model List + +`teep verify` tests end-to-end encryption with one chat completion, called +the probe. Four Phala Cloud models have no text output modality, so teep's +verifier does not test their encryption today. Phala Cloud's model list has +no endpoint field. Only modalities, pricing keys and, for one model, the +free-text description show which endpoint a model serves. `teep serve` has +no Phala Cloud path for image generation, audio transcription or +`/v1/systemone`, so it serves none of these models. + +## The Problem + +Anyone who checks a Phala Cloud model before use, such as a user's client or +an auditor, needs to send it a real request to see that encryption works. The +request must go to an endpoint that the model serves. The model list has no +field that states that endpoint, and a model without chat completions answers +a chat request with an error. + +## Impact + +**The encryption of models without text output is not tested before use.** + +- **Who:** Phala Cloud, or a party that controls a Phala Cloud model + deployment or the gateway. +- **What they do:** serve a model without chat completions whose encryption + does not work. +- **What they get:** a model that passes verification without an encryption + test. They read no request through teep: teep sends these models no request + that they serve. +- **What stops it today:** `teep serve` has no Phala Cloud path for these + endpoints. It sends these models only chat completions, encrypted when E2EE + is on, and the gateway refuses them. Nothing confirms their encryption. In + `teep verify`, nothing: the factor stays skipped. + +## Technical Background + +Phala Cloud's model list gives, for each model, `input_modalities`, +`output_modalities`, `pricing` and a `description`. It has no endpoint field. + +The probe of `teep verify` is one streaming chat completion with E2EE v2. +The model must answer with encrypted fields that teep decrypts, and the +response must carry a receipt that teep checks. + +## Detailed Gap Analysis + +Phala Cloud's model list of 2026-11-14 has four models whose output modality +is not text: + +| Model | Modalities in the model list | Endpoint kind (derived by teep) | +| --- | --- | --- | +| `black-forest-labs/flux.2-klein-4b` | text → image | image generation | +| `openai/whisper-large-v3` | audio → transcription | audio transcription | +| `mistralai/voxtral-small-24b-2507` | audio → transcription | audio transcription | +| `jaredpalmer/kev-4b` | text → decisions | `/v1/systemone` (named in the description) | + +On 2026-11-14, `jaredpalmer/kev-4b` got the chat probe, and the gateway +answered "upstream returned HTTP 404: no route available for model +jaredpalmer/kev-4b". `e2ee_usable`, which Phala Cloud's policy allows to +fail, failed, and the report did not block. The reports of +`black-forest-labs/flux.2-klein-4b` and `openai/whisper-large-v3` were +blocked by `aci_session_verified`, because teep does not verify the evidence +of their gateway session, so they got no probe. The run of 2026-11-14 made +no report for `mistralai/voxtral-small-24b-2507`. + +## Remediation + +- **Phala Cloud:** add the served endpoint (or endpoints) to each entry of the + model list, as a field. +- **teep:** probe each model on an endpoint that it serves, once E2EE v2 + covers that endpoint. + +## References + +- [Phala Cloud provider support: known limitations](../providers/phalacloud/phalacloud_support.md#known-limitations) +- [Tinfoil Model Endpoints: Verify Probe Covers Chat Only](tinfoil_model_endpoints.md) +- [NEAR Model Endpoints: No Endpoint in the Model List](near_model_endpoints.md) +- [The table of unprobed Phala Cloud models in teep](../../internal/provider/phalacloud/kinds.go) + +## Teep Status + +`teep verify` sends no chat probe to the four models above and logs a +warning. `e2ee_usable`, which the policy allows to fail, stays skipped with +the detail "not probed: the model serves ; the verify probe sends a +chat completion". `aci_receipt_verified`, which is enforced, stays skipped +with the detail "no probe: the model serves ; …". Neither blocks. +Because `e2ee_usable` is allowed to fail, a failed probe did not block +either: the skip changes the detail in the report, not whether the report +blocks. A model that is not in the table gets the chat probe, so a new model +without chat completions fails the factor and is reported. + +`teep serve` has no Phala Cloud path for these endpoints, so no request +confirms the encryption of these four models. diff --git a/docs/attestation_gaps/tinfoil_model_endpoints.md b/docs/attestation_gaps/tinfoil_model_endpoints.md new file mode 100644 index 00000000..a5b3dcf7 --- /dev/null +++ b/docs/attestation_gaps/tinfoil_model_endpoints.md @@ -0,0 +1,98 @@ +# Tinfoil Model Endpoints: Verify Probe Covers Chat Only + +`teep verify` tests end-to-end encryption with one chat completion, the +probe. Ten Tinfoil models serve no chat completions endpoint, so teep's +verifier does not test their encryption today. Their other factors give the +same results as Tinfoil's chat models. Tinfoil's model list states the +endpoint of each model, so the gap is in teep, not in Tinfoil's +documentation. + +## The Problem + +Anyone who checks a Tinfoil model before use, such as a user's client or an +auditor, needs to send it a real request to see that encryption works. The +request must go to an endpoint that the model serves. A model without chat +completions answers a chat request with HTTP 404, so a check that sends only +chat cannot test it. Each request body stays encrypted to the attested key, +but no check confirms a full encrypted exchange before use. + +## Impact + +**The encryption of models without chat completions is not tested before +use.** + +- **Who:** Tinfoil, or a party that controls a Tinfoil model enclave. +- **What they do:** serve a model without chat completions whose encryption + does not work. +- **What they get:** a model that passes verification without an encryption + test. They read no request: teep encrypts each request body to the attested + key. +- **What stops it today:** EHBP (HPKE X25519 with AES-256-GCM, full-body + encryption) encrypts the whole body of every request that teep sends with + E2EE on, and the response must decrypt under the same session. For + `nomic-embed-text`, `qwen3-tts` and `voxtral-tts`, the first real request in + `teep serve` checks the encryption, and a decryption failure blocks the + request. `teep serve` refuses audio transcription while encryption is on, + and has no route for `/redact`, `/v1/convert/file`, `/v1/realtime` or + `/v1/decisions`. No request to these models is sent unencrypted, but nothing + confirms their encryption. In `teep verify`, nothing: the factor stays + skipped. + +## Technical Background + +Tinfoil's model list (`inference.tinfoil.sh/v1/models`) gives, for each model, +a `type` and an `endpoints` list of the paths that the model serves. Teep does +not read this list during verification. + +The probe of `teep verify` is one streaming chat completion through EHBP. The +model must answer with a body that teep decrypts. + +## Detailed Gap Analysis + +Tinfoil's model list of 2026-11-14 has ten models whose `endpoints` have no +`/v1/chat/completions`: + +| Model | `type` | `endpoints` | +| --- | --- | --- | +| `nomic-embed-text` | embedding | `/v1/embeddings` | +| `pii-filter` | safety | `/redact` | +| `doc-upload` | document | `/v1/convert/file` | +| `whisper-large-v3-turbo` | audio | `/v1/audio/transcriptions` | +| `qwen3-tts` | tts | `/v1/audio/speech` | +| `voxtral-tts` | tts | `/v1/audio/speech` | +| `voxtral-mini-4b-realtime` | audio | `/v1/realtime` | +| `decision-large` | decision | `/v1/decisions` | +| `decision-medium` | decision | `/v1/decisions` | +| `decision-small` | decision | `/v1/decisions` | + +On 2026-11-14, six of them got the chat probe: `nomic-embed-text`, +`pii-filter`, `qwen3-tts`, `voxtral-mini-4b-realtime`, `voxtral-tts` and +`whisper-large-v3-turbo`. Each answered "upstream returned HTTP 404: Not +found.", and `e2ee_usable` failed and blocked the model. + +`voxtral-small-24b` serves audio transcription and also chat completions, so +it gets the chat probe. + +## Remediation + +- **teep:** read the `endpoints` field of Tinfoil's model list during + verification in place of a fixed table, and probe each model on an endpoint + that it serves, such as embeddings or speech, which EHBP already encrypts. +- **Tinfoil:** none. The model list states the endpoints. + +## References + +- [Tinfoil provider support: models the probe does not cover](../providers/tinfoil/tinfoil_support.md#models-the-probe-does-not-cover) +- [NEAR Model Endpoints: No Endpoint in the Model List](near_model_endpoints.md) +- [The table of unprobed Tinfoil models in teep](../../internal/provider/tinfoil/kinds.go) + +## Teep Status + +`teep verify` sends no chat probe to the ten models above and logs a warning. +`e2ee_usable` stays skipped, enforced and not probed, with the detail "not +probed: the model serves ; the verify probe sends a chat +completion". It counts as skipped and does not block. A model that is not in +the table gets the chat probe, so a new model without chat completions fails +the factor and is reported. + +`teep serve` does not change. diff --git a/docs/providers/phalacloud/phalacloud_support.md b/docs/providers/phalacloud/phalacloud_support.md index f9f1c607..d8e5a333 100644 --- a/docs/providers/phalacloud/phalacloud_support.md +++ b/docs/providers/phalacloud/phalacloud_support.md @@ -414,11 +414,23 @@ report carries no GPU evidence, and `nonce_match` and is no probe receipt to check: in `teep serve`, which checks the receipt of every response instead, in `teep verify` when the report blocks inference requests (teep sends no probe then), without an API key, with `--offline` -(an offline run sends no probe), and after a probe whose +(an offline run sends no probe), for a model that the probe does not cover +(see [Known limitations](#known-limitations)), and after a probe whose stream failed. ## Known limitations +- `teep verify` tests E2EE with one chat completion, called the probe. It + sends no probe to the four Phala Cloud models whose output modality is not + text, and it logs a warning. Those models are + `black-forest-labs/flux.2-klein-4b`, `openai/whisper-large-v3`, + `mistralai/voxtral-small-24b-2507` and `jaredpalmer/kev-4b`, listed in + `unprobedKinds` in + [`internal/provider/phalacloud/kinds.go`](../../../internal/provider/phalacloud/kinds.go). + `e2ee_usable` (allowed to fail) and `aci_receipt_verified` (enforced) stay + skipped and do not block. A model that is not in that table gets the + probe. `teep serve` has no path for the endpoints of these models. See + [Phala Cloud Model Endpoints: No Endpoint Field in the Model List](../../attestation_gaps/phalacloud_model_endpoints.md). - For a model whose gateway session is replaced at each re-verification, the pin is refused and teep serves no request (observed on 2026-11-03; see [Attested sessions](#attested-sessions)). diff --git a/docs/providers/tinfoil/tinfoil_support.md b/docs/providers/tinfoil/tinfoil_support.md index 27b88022..0927bfd9 100644 --- a/docs/providers/tinfoil/tinfoil_support.md +++ b/docs/providers/tinfoil/tinfoil_support.md @@ -169,6 +169,24 @@ as a single AEAD stream. There are **no field-level gaps** for those encrypted exchanges. Bodyless endpoints (for example `GET /v1/models`) are plaintext at the HTTP-body layer by design. +### Models the probe does not cover + +`teep verify` tests E2EE with one streaming chat completion, the probe. Ten +models in Tinfoil's model list of 2026-11-14 serve no `/v1/chat/completions`: +`nomic-embed-text`, `pii-filter`, `doc-upload`, `whisper-large-v3-turbo`, +`qwen3-tts`, `voxtral-tts`, `voxtral-mini-4b-realtime`, and +`decision-large`, `-medium` and `-small`. On 2026-11-14, the six of them that +got the probe answered it with HTTP 404. `teep verify` sends these ten models +no probe and logs a warning. `e2ee_usable` stays enforced with the status +Skip and the detail "not probed: the model serves ; the verify +probe sends a chat completion", and does not block. A model that is not in +the table gets the probe, so a new model without chat completions fails +`e2ee_usable` and is reported. The table is `unprobedKinds` in +[`internal/provider/tinfoil/kinds.go`](../../../internal/provider/tinfoil/kinds.go), +copied from the `endpoints` field of Tinfoil's model list of 2026-11-14. The +gap is recorded in +[Tinfoil Model Endpoints: Verify Probe Covers Chat Only](../../attestation_gaps/tinfoil_model_endpoints.md). + ### Endpoint Routing and Request-Shape Mechanics To produce a compatible implementation without depending on upstream source diff --git a/internal/provider/nearroute/kinds.go b/internal/provider/nearroute/kinds.go index 1abe6eb7..8161c3ed 100644 --- a/internal/provider/nearroute/kinds.go +++ b/internal/provider/nearroute/kinds.go @@ -19,12 +19,9 @@ var unprobedKinds = map[string]string{ "openai/privacy-filter": "an endpoint other than chat completions (unknown)", } -// UnprobedDetail returns the e2ee_usable detail for a NEAR model whose -// endpoint the verify probe does not cover, and false for any other model. -func UnprobedDetail(model string) (string, bool) { +// UnprobedKind returns the endpoint kind of a NEAR model that serves an +// endpoint other than chat completions, and false for any other model. +func UnprobedKind(model string) (string, bool) { kind, ok := unprobedKinds[model] - if !ok { - return "", false - } - return "not probed: the model serves " + kind + "; the verify probe sends a chat completion", true + return kind, ok } diff --git a/internal/provider/nearroute/kinds_test.go b/internal/provider/nearroute/kinds_test.go index e4568dae..076e22eb 100644 --- a/internal/provider/nearroute/kinds_test.go +++ b/internal/provider/nearroute/kinds_test.go @@ -2,25 +2,23 @@ package nearroute import "testing" -// The five non-chat models of 2026-11-14 get a full-sentence detail; any -// other model gets the chat probe. -func TestUnprobedDetail(t *testing.T) { - for model, kind := range map[string]string{ +// The five non-chat models of 2026-11-14 get their endpoint kind; any other +// model gets the chat probe. +func TestUnprobedKind(t *testing.T) { + for model, want := range map[string]string{ "black-forest-labs/FLUX.2-klein-4B": "image generation", "Qwen/Qwen3-Embedding-0.6B": "embeddings", "Qwen/Qwen3-Reranker-0.6B": "reranking", "openai/whisper-large-v3": "audio transcription", "openai/privacy-filter": "an endpoint other than chat completions (unknown)", } { - detail, ok := UnprobedDetail(model) - want := "not probed: the model serves " + kind + "; the verify probe sends a chat completion" - if !ok || detail != want { - t.Errorf("%s: detail %q, %v; want %q", model, detail, ok, want) + if kind, ok := UnprobedKind(model); !ok || kind != want { + t.Errorf("%s: kind %q, %v; want %q", model, kind, ok, want) } } for _, model := range []string{"z-ai/glm-5.3-flash", "qwen/qwen3-embedding-0.6b", "Qwen/Qwen3-Embedding-0.6B-v2", ""} { - if detail, ok := UnprobedDetail(model); ok { - t.Errorf("%q is not listed but got %q", model, detail) + if kind, ok := UnprobedKind(model); ok { + t.Errorf("%q is not listed but got %q", model, kind) } } } diff --git a/internal/provider/phalacloud/kinds.go b/internal/provider/phalacloud/kinds.go new file mode 100644 index 00000000..147ef81b --- /dev/null +++ b/internal/provider/phalacloud/kinds.go @@ -0,0 +1,26 @@ +package phalacloud + +// unprobedKinds lists, by exact model id, the Phala Cloud models whose output +// is not text, with the endpoint kind that each serves. The E2EE probe of +// teep verify sends a chat completion; on 2026-11-14 the gateway answered it +// for jaredpalmer/kev-4b with HTTP 404 "no route available for model". A +// model that is not listed gets the chat probe, so a new model without chat +// completions fails e2ee_usable and is reported. +// +// Read from Phala Cloud's model list of 2026-11-14: every model whose output +// modality is not text. The list has no endpoint field, so the kind follows +// from the modalities, except for jaredpalmer/kev-4b, whose description names +// its endpoint. +var unprobedKinds = map[string]string{ + "black-forest-labs/flux.2-klein-4b": "image generation", // text → image + "openai/whisper-large-v3": "audio transcription", // audio → transcription + "mistralai/voxtral-small-24b-2507": "audio transcription", // audio → transcription + "jaredpalmer/kev-4b": "/v1/systemone", // text → decisions; named in the description +} + +// UnprobedKind returns the endpoint kind of a Phala Cloud model whose output +// is not text, and false for any other model. +func UnprobedKind(model string) (string, bool) { + kind, ok := unprobedKinds[model] + return kind, ok +} diff --git a/internal/provider/phalacloud/kinds_test.go b/internal/provider/phalacloud/kinds_test.go new file mode 100644 index 00000000..c4adfe02 --- /dev/null +++ b/internal/provider/phalacloud/kinds_test.go @@ -0,0 +1,23 @@ +package phalacloud + +import "testing" + +// The four models without text output in Phala Cloud's model list of +// 2026-11-14 get their kind; any other model gets the chat probe. +func TestUnprobedKind(t *testing.T) { + for model, want := range map[string]string{ + "black-forest-labs/flux.2-klein-4b": "image generation", + "openai/whisper-large-v3": "audio transcription", + "mistralai/voxtral-small-24b-2507": "audio transcription", + "jaredpalmer/kev-4b": "/v1/systemone", + } { + if kind, ok := UnprobedKind(model); !ok || kind != want { + t.Errorf("%s: kind %q, %v; want %q", model, kind, ok, want) + } + } + for _, model := range []string{"z-ai/glm-5.3", "openai/whisper-large-v3-turbo", "Jaredpalmer/kev-4b", ""} { + if kind, ok := UnprobedKind(model); ok { + t.Errorf("%q is not listed but got %q", model, kind) + } + } +} diff --git a/internal/provider/tinfoil/kinds.go b/internal/provider/tinfoil/kinds.go new file mode 100644 index 00000000..8098230a --- /dev/null +++ b/internal/provider/tinfoil/kinds.go @@ -0,0 +1,30 @@ +package tinfoil + +// unprobedKinds lists, by exact model id, the Tinfoil models that serve no +// chat completions endpoint, with the endpoint that each serves. The E2EE +// probe of teep verify sends a chat completion, which these models answer +// with HTTP 404. A model that is not listed gets the chat probe, so a new +// model without chat completions fails e2ee_usable and is reported. +// +// Read from the `endpoints` field of Tinfoil's model list of 2026-11-14: +// every model whose list has no /v1/chat/completions. Reading that field at +// verification time, in place of this table, is a follow-up change. +var unprobedKinds = map[string]string{ + "nomic-embed-text": "/v1/embeddings", + "pii-filter": "/redact", + "doc-upload": "/v1/convert/file", + "whisper-large-v3-turbo": "/v1/audio/transcriptions", + "qwen3-tts": "/v1/audio/speech", + "voxtral-tts": "/v1/audio/speech", + "voxtral-mini-4b-realtime": "/v1/realtime", + "decision-large": "/v1/decisions", + "decision-medium": "/v1/decisions", + "decision-small": "/v1/decisions", +} + +// UnprobedKind returns the endpoint of a Tinfoil model that serves no chat +// completions endpoint, and false for any other model. +func UnprobedKind(model string) (string, bool) { + kind, ok := unprobedKinds[model] + return kind, ok +} diff --git a/internal/provider/tinfoil/kinds_test.go b/internal/provider/tinfoil/kinds_test.go new file mode 100644 index 00000000..65d97433 --- /dev/null +++ b/internal/provider/tinfoil/kinds_test.go @@ -0,0 +1,29 @@ +package tinfoil + +import "testing" + +// The ten models without chat completions in Tinfoil's model list of +// 2026-11-14 get their endpoint; any other model gets the chat probe. +func TestUnprobedKind(t *testing.T) { + for model, want := range map[string]string{ + "nomic-embed-text": "/v1/embeddings", + "pii-filter": "/redact", + "doc-upload": "/v1/convert/file", + "whisper-large-v3-turbo": "/v1/audio/transcriptions", + "qwen3-tts": "/v1/audio/speech", + "voxtral-tts": "/v1/audio/speech", + "voxtral-mini-4b-realtime": "/v1/realtime", + "decision-large": "/v1/decisions", + "decision-medium": "/v1/decisions", + "decision-small": "/v1/decisions", + } { + if kind, ok := UnprobedKind(model); !ok || kind != want { + t.Errorf("%s: kind %q, %v; want %q", model, kind, ok, want) + } + } + for _, model := range []string{"gpt-oss-120b", "voxtral-small-24b", "websearch", "Nomic-Embed-Text", ""} { + if kind, ok := UnprobedKind(model); ok { + t.Errorf("%q is not listed but got %q", model, kind) + } + } +} diff --git a/internal/verify/factory.go b/internal/verify/factory.go index f40dfac8..aca2eb52 100644 --- a/internal/verify/factory.go +++ b/internal/verify/factory.go @@ -171,6 +171,22 @@ func supplyChainPolicy(name string) (*attestation.SupplyChainPolicy, error) { } } +// unprobedModels returns the provider's table of models that serve an +// endpoint the chat probe of teep verify does not cover, or nil for a +// provider without one. The table maps a model id to its endpoint kind. +func unprobedModels(name string) func(model string) (string, bool) { + switch name { + case "neardirect", "nearcloud": + return nearroute.UnprobedKind + case "tinfoil_v3_cloud", "tinfoil_v3_direct": + return tinfoil.UnprobedKind + case "phalacloud": + return phalacloud.UnprobedKind + default: + return nil + } +} + func inapplicableFactors(providerName string) attestation.InapplicableFactors { switch providerName { case "venice", "neardirect", "nearcloud", "nanogpt", "phalacloud": diff --git a/internal/verify/tls_inference.go b/internal/verify/tls_inference.go index e0c17f65..95a7019c 100644 --- a/internal/verify/tls_inference.go +++ b/internal/verify/tls_inference.go @@ -18,7 +18,6 @@ import ( "github.com/13rac1/teep/internal/provider/aci" "github.com/13rac1/teep/internal/provider/nearcloud" "github.com/13rac1/teep/internal/provider/neardirect" - "github.com/13rac1/teep/internal/provider/nearroute" "github.com/13rac1/teep/internal/provider/phalacloud" "github.com/13rac1/teep/internal/provider/tinfoil" "github.com/13rac1/teep/internal/tlsct" @@ -165,22 +164,37 @@ func markBlockedProbe(report *attestation.VerificationReport) { } } -// markUnprobedModel states in the deferred e2ee_usable Skip why no probe -// runs, and reports true, for a NEAR model that serves an endpoint the chat -// probe does not cover. A blocked report keeps its result: no probe runs for -// it either way. A replay with a captured probe outcome keeps that outcome. +// unprobedDetail is the e2ee_usable detail for a model that serves an +// endpoint of the given kind, which the chat probe does not cover. +func unprobedDetail(kind string) string { + return "not probed: the model serves " + kind + "; the verify probe sends a chat completion" +} + +// markUnprobedModel states in the deferred e2ee_usable Skip, and in the +// deferred aci_receipt_verified Skip, why no probe runs, and reports true, for a model that the provider's unprobed-model +// table lists: a model that serves an endpoint the chat probe does not cover. +// A blocked report keeps its result: no probe runs for it either way. With +// E2EE off, and in a replay with a captured probe outcome, the path does not +// change. func markUnprobedModel(opts *Options, report *attestation.VerificationReport) bool { - if report == nil || report.Blocked() || nearTLSOnly(opts) || opts.CapturedE2EE != nil || - (opts.ProviderName != "neardirect" && opts.ProviderName != "nearcloud") { + if report == nil || report.Blocked() || !opts.Provider.E2EE || opts.CapturedE2EE != nil { + return false + } + kindOf := unprobedModels(opts.ProviderName) + if kindOf == nil { return false } - detail, ok := nearroute.UnprobedDetail(opts.ModelName) + kind, ok := kindOf(opts.ModelName) if !ok { return false } + detail := unprobedDetail(kind) slog.Warn("E2EE probe skipped: the model serves an endpoint the chat probe does not cover", "provider", opts.ProviderName, "model", opts.ModelName, "detail", detail) report.MarkE2EENotProbed(detail) + // The receipt factor of an ACI/1 report checks the probe's receipt; no + // probe means no receipt. A report without the factor does not change. + report.MarkACIReceipt(nil, "no probe: the model serves "+kind+"; the verify probe sends a chat completion") return true } diff --git a/internal/verify/unprobed_test.go b/internal/verify/unprobed_test.go index 397fd65a..074f4b11 100644 --- a/internal/verify/unprobed_test.go +++ b/internal/verify/unprobed_test.go @@ -26,21 +26,31 @@ func e2eeFactor(r *attestation.VerificationReport) attestation.FactorResult { return attestation.FactorResult{} } -// A listed non-chat NEAR model keeps e2ee_usable as a deferred, enforced -// Skip with the full detail, and gets no probe; any other model, a blocked -// report, the TLS-only mode, another provider and a replay with a captured -// probe outcome keep their path. +// A model in its provider's unprobed-model table keeps e2ee_usable as a +// deferred, enforced Skip with the full detail, and gets no probe; any other +// model, a blocked report, E2EE off, a provider without a table and a replay +// with a captured probe outcome keep their path. func TestMarkUnprobedModel(t *testing.T) { opts := func(provider, model string, e2ee bool) *Options { return &Options{ProviderName: provider, ModelName: model, Provider: &config.Provider{Name: provider, E2EE: e2ee}} } - for _, provider := range []string{"neardirect", "nearcloud"} { + for _, tc := range []struct{ provider, model, kind string }{ + {"neardirect", "Qwen/Qwen3-Embedding-0.6B", "embeddings"}, + {"nearcloud", "Qwen/Qwen3-Embedding-0.6B", "embeddings"}, + {"tinfoil_v3_cloud", "nomic-embed-text", "/v1/embeddings"}, + {"tinfoil_v3_cloud", "voxtral-mini-4b-realtime", "/v1/realtime"}, + {"phalacloud", "jaredpalmer/kev-4b", "/v1/systemone"}, + } { + provider := tc.provider r := unprobedReport(false) - if !markUnprobedModel(opts(provider, "Qwen/Qwen3-Embedding-0.6B", true), r) { - t.Fatalf("%s: listed model was probed", provider) + if !markUnprobedModel(opts(provider, tc.model, true), r) { + t.Fatalf("%s: listed model %s was probed", provider, tc.model) } f := e2eeFactor(r) - want := "not probed: the model serves embeddings; the verify probe sends a chat completion" + want := unprobedDetail(tc.kind) + if want != "not probed: the model serves "+tc.kind+"; the verify probe sends a chat completion" { + t.Fatalf("detail %q", want) + } if f.Status != attestation.Skip || !f.Deferred || !f.Enforced || f.Detail != want { t.Errorf("%s: e2ee_usable %v deferred=%v enforced=%v %q, want deferred enforced Skip %q", provider, f.Status, f.Deferred, f.Enforced, f.Detail, want) @@ -57,7 +67,11 @@ func TestMarkUnprobedModel(t *testing.T) { {"chat model", opts("nearcloud", "z-ai/glm-5.3-flash", true), false}, {"blocked report", opts("nearcloud", "Qwen/Qwen3-Embedding-0.6B", true), true}, {"TLS-only neardirect", opts("neardirect", "Qwen/Qwen3-Embedding-0.6B", false), false}, - {"other provider", opts("phalacloud", "Qwen/Qwen3-Embedding-0.6B", true), false}, + {"tinfoil with E2EE off", opts("tinfoil_v3_cloud", "nomic-embed-text", false), false}, + {"tinfoil chat model", opts("tinfoil_v3_cloud", "gpt-oss-120b", true), false}, + {"model of another provider's table", opts("tinfoil_v3_cloud", "Qwen/Qwen3-Embedding-0.6B", true), false}, + {"phalacloud chat model", opts("phalacloud", "z-ai/glm-5.3", true), false}, + {"provider without a table", opts("chutes", "Qwen/Qwen3-Embedding-0.6B", true), false}, {"replay with a captured probe outcome", func() *Options { o := opts("nearcloud", "Qwen/Qwen3-Embedding-0.6B", true) o.CapturedE2EE = &attestation.E2EETestResult{Attempted: true} @@ -75,3 +89,27 @@ func TestMarkUnprobedModel(t *testing.T) { }) } } + +// For an ACI/1 report, a skipped probe also states in the deferred +// aci_receipt_verified Skip that no probe ran and why. +func TestMarkUnprobedModelReceipt(t *testing.T) { + r := unprobedReport(false) + r.Factors = append(r.Factors, attestation.FactorResult{ + Name: attestation.FactorACIReceiptVerified, Status: attestation.Skip, Enforced: true, Deferred: true, + Detail: "no probe: teep serve checks the receipt of every response", + }) + opts := &Options{ProviderName: "phalacloud", ModelName: "jaredpalmer/kev-4b", Provider: &config.Provider{Name: "phalacloud", E2EE: true}} + if !markUnprobedModel(opts, r) { + t.Fatal("listed model was probed") + } + want := "no probe: the model serves /v1/systemone; the verify probe sends a chat completion" + for _, f := range r.Factors { + if f.Name == attestation.FactorACIReceiptVerified && + (f.Status != attestation.Skip || !f.Deferred || !f.Enforced || f.Detail != want) { + t.Errorf("aci_receipt_verified = %+v, want deferred enforced Skip %q", f, want) + } + } + if r.Blocked() { + t.Error("report blocked") + } +}