Repository navigation
Skip the verify probe for every provider's models that serve another endpoint - #231
Merged
Merged
Conversation
…ovider teep verify tests E2EE with one chat completion. Six Tinfoil models that serve no chat completions endpoint (nomic-embed-text, pii-filter, qwen3-tts, voxtral-mini-4b-realtime, voxtral-tts and whisper-large-v3-turbo) answered that probe with HTTP 404 on 2026-11-14, and e2ee_usable failed and blocked each of them. Phala Cloud's gateway answered it for jaredpalmer/kev-4b with HTTP 404. The skip for such models existed only for NEAR, and the decision compared the provider name. Each provider package now holds a table of the models that serve an endpoint the chat probe does not cover. The verify factory, where teep already selects code by provider, returns the table for a provider; the decision reads the table and compares no provider name. - Tinfoil: the ten models whose `endpoints` in Tinfoil's model list of 2026-11-14 have no /v1/chat/completions, each with its endpoint. - Phala Cloud: the four models whose output modality in Phala Cloud's model list of 2026-11-14 is not text, each with the kind that its modalities give, or the endpoint that its description names. - NEAR: the five models do not change. For a listed model, e2ee_usable stays a deferred Skip, with the enforcement that the provider's policy sets, and the detail "not probed: the model serves <endpoint kind>; the verify probe sends a chat completion", aci_receipt_verified (on an ACI/1 report) states that no probe ran, and teep logs a warning. A model that is not listed gets the chat probe, so a new model without chat completions fails the factor and is reported. With E2EE off the path does not change. The gaps are recorded in docs/attestation_gaps/tinfoil_model_endpoints.md and docs/attestation_gaps/phalacloud_model_endpoints.md. To verify: go test ./internal/verify ./internal/provider/tinfoil ./internal/provider/phalacloud ./internal/provider/nearroute -run Unprobed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
teep verifytests E2EE with one probe: a streaming chat completion. A model that serves no chat completions endpoint cannot answer it. On 2026-11-14, such models on three providers failed it:nomic-embed-text,pii-filter,qwen3-tts,voxtral-mini-4b-realtime,voxtral-tts,whisper-large-v3-turbo.e2ee_usableis enforced for Tinfoil, so each report blocked. Tinfoil's model list names ten models whoseendpointshave no/v1/chat/completions.jaredpalmer/kev-4bwith "upstream returned HTTP 404: no route available for model jaredpalmer/kev-4b". Phala Cloud's model list has four models whose output modality is not text.internal/provider/nearroute/kinds.go. That decision compared the provider name, so the rule applied to NEAR only.Change
internal/provider/tinfoil/kinds.go: the ten Tinfoil models, each with the endpoint from Tinfoil's model list of 2026-11-14.internal/provider/phalacloud/kinds.go: the four Phala Cloud models, each with the kind that its modalities give, or forjaredpalmer/kev-4bthe endpoint that its description names (/v1/systemone).internal/provider/nearroute/kinds.go: the five NEAR models, unchanged.unprobedModelsininternal/verify/factory.goreturns the table for a provider. The factory is where teep already selects code by provider.markUnprobedModelreads the table and compares no provider name.e2ee_usablestays a deferred Skip, with the enforcement that the provider's policy sets, and the detail "not probed: the model serves ; the verify probe sends a chat completion".aci_receipt_verifiedsays "no probe: the model serves ; …". Before, it said "teep serve checks the receipt of every response", which is wrong inteep verify.e2ee_usableand is reported. With E2EE off, the path does not change.docs/attestation_gaps/tinfoil_model_endpoints.mdanddocs/attestation_gaps/phalacloud_model_endpoints.md, and sections in the Tinfoil and Phala Cloud support documents.Effect
teep verify.e2ee_usableis a skipped, enforced factor.e2ee_usableis in Phala Cloud'sallow_faillist, so a failed probe did not block before either.teep verifymeans that E2EE was not tested. It does not mean that E2EE works. The gap documents record this. The fix is a probe on each model's own endpoint.teep servedoes not change.To verify
TestMarkUnprobedModelcovers a listed model of each provider, and the cases that keep the probe: a chat model, a blocked report, E2EE off, a model of another provider's table, a provider without a table, and a replay with a captured probe outcome.TestMarkUnprobedModelReceiptcovers theaci_receipt_verifieddetail.Checklist
make checkpasses.make integrationandmake reportspass.