Skip to content

Skip the verify probe for every provider's models that serve another endpoint - #231

Merged
13rac1 merged 1 commit into
13rac1:mainfrom
ai-13rac1:e2ee-probe-kind-skip-2
Nov 14, 2026
Merged

13rac1 merged 1 commit into
13rac1:mainfrom
ai-13rac1:e2ee-probe-kind-skip-2

Conversation

@ai-13rac1

Copy link
Copy Markdown
Contributor

Problem

teep verify tests E2EE with one probe: a streaming chat completion. A model that serves no chat completions endpoint cannot answer it. On 2026-11-14, such models on three providers failed it:

  • Tinfoil: six models answered the probe with "upstream returned HTTP 404: Not found.": nomic-embed-text, pii-filter, qwen3-tts, voxtral-mini-4b-realtime, voxtral-tts, whisper-large-v3-turbo. e2ee_usable is enforced for Tinfoil, so each report blocked. Tinfoil's model list names ten models whose endpoints have no /v1/chat/completions.
  • Phala Cloud: the gateway answered the probe for jaredpalmer/kev-4b with "upstream returned HTTP 404: no route available for model jaredpalmer/kev-4b". Phala Cloud's model list has four models whose output modality is not text.
  • NEAR: five models, which an earlier change handles with a table in internal/provider/nearroute/kinds.go. That decision compared the provider name, so the rule applied to NEAR only.

Change

  • Each provider package holds a table of the models that serve an endpoint the chat probe does not cover, with the endpoint kind of each:
    • internal/provider/tinfoil/kinds.go: the ten Tinfoil models, each with the endpoint from Tinfoil's model list of 2026-11-14.
    • internal/provider/phalacloud/kinds.go: the four Phala Cloud models, each with the kind that its modalities give, or for jaredpalmer/kev-4b the endpoint that its description names (/v1/systemone).
    • internal/provider/nearroute/kinds.go: the five NEAR models, unchanged.
  • unprobedModels in internal/verify/factory.go returns the table for a provider. The factory is where teep already selects code by provider. markUnprobedModel reads the table and compares no provider name.
  • For a listed model, teep sends no probe and logs a warning:
    • e2ee_usable stays a deferred Skip, with the enforcement that the provider's policy sets, and the detail "not probed: the model serves ; the verify probe sends a chat completion".
    • On an ACI/1 report, aci_receipt_verified says "no probe: the model serves ; …". Before, it said "teep serve checks the receipt of every response", which is wrong in teep verify.
  • A model that is not in its provider's table gets the probe. A new model without chat completions therefore fails e2ee_usable and is reported. With E2EE off, the path does not change.
  • New gap documents docs/attestation_gaps/tinfoil_model_endpoints.md and docs/attestation_gaps/phalacloud_model_endpoints.md, and sections in the Tinfoil and Phala Cloud support documents.

Effect

  • Tinfoil: the ten models no longer block in teep verify. e2ee_usable is a skipped, enforced factor.
  • Phala Cloud: only the detail in the report changes. e2ee_usable is in Phala Cloud's allow_fail list, so a failed probe did not block before either.
  • NEAR: no change.
  • Not tested is not the same as working. For a listed model, a zero exit from teep verify means that E2EE was not tested. It does not mean that E2EE works. The gap documents record this. The fix is a probe on each model's own endpoint.
  • teep serve does not change.

To verify

go test ./internal/verify ./internal/provider/tinfoil ./internal/provider/phalacloud ./internal/provider/nearroute -run Unprobed
make check
  • TestMarkUnprobedModel covers a listed model of each provider, and the cases that keep the probe: a chat model, a blocked report, E2EE off, a model of another provider's table, a provider without a table, and a replay with a captured probe outcome.
  • TestMarkUnprobedModelReceipt covers the aci_receipt_verified detail.

Checklist

  • I added unit tests for new code and regression tests for fixed bugs.
  • make check passes.
  • For a major feature: make integration and make reports pass.
  • I updated the documentation that this change affects.

…ovider

teep verify tests E2EE with one chat completion. Six Tinfoil models
that serve no chat completions endpoint (nomic-embed-text, pii-filter,
qwen3-tts, voxtral-mini-4b-realtime, voxtral-tts and
whisper-large-v3-turbo) answered that probe with HTTP 404 on
2026-11-14, and e2ee_usable failed and blocked each of them. Phala
Cloud's gateway answered it for jaredpalmer/kev-4b with HTTP 404. The
skip for such models existed only for NEAR, and the decision compared
the provider name.

Each provider package now holds a table of the models that serve an
endpoint the chat probe does not cover. The verify factory, where teep
already selects code by provider, returns the table for a provider;
the decision reads the table and compares no provider name.

- Tinfoil: the ten models whose `endpoints` in Tinfoil's model list of
  2026-11-14 have no /v1/chat/completions, each with its endpoint.
- Phala Cloud: the four models whose output modality in Phala Cloud's
  model list of 2026-11-14 is not text, each with the kind that its
  modalities give, or the endpoint that its description names.
- NEAR: the five models do not change.

For a listed model, e2ee_usable stays a deferred Skip, with the
enforcement that the provider's policy sets, and
the detail "not probed: the model serves <endpoint kind>; the verify
probe sends a chat completion", aci_receipt_verified (on an ACI/1
report) states that no probe ran, and teep logs a warning. A model that
is not listed gets the chat probe, so a new model without chat
completions fails the factor and is reported. With E2EE off the path
does not change.

The gaps are recorded in docs/attestation_gaps/tinfoil_model_endpoints.md
and docs/attestation_gaps/phalacloud_model_endpoints.md.

To verify: go test ./internal/verify ./internal/provider/tinfoil
./internal/provider/phalacloud ./internal/provider/nearroute -run
Unprobed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@13rac1
13rac1 merged commit ad7ddd0 into 13rac1:main Nov 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants