feat: per-call response verification at the proxy - #14
Merged
Conversation
Every forwarded tools/call response is verified against a per-tool reference:
declared outputSchema, else the core fingerprint (intersection of baselined
probe fingerprints). Output-side BREAKING classes (missing field, structural
retype, forbidden null, pinned value mismatch) block the response and
response-quarantine the tool; scalar retypes record DEGRADED and forward.
- covenant/verify.py: pure verifier (Layer 0) - references, tiers, pins
- quarantine: two buckets; sync() owns schema only, refresh clears response
- proxy: JSON + SSE interception (only the matching result frame is held),
1 MiB verify cap, verifier exceptions forward original bytes
- metrics: covenant_response_verifications_total{tool, outcome}
- cli: covenant proxy --observe; snapshot embeds expect pins into the lock
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ILS amounts still labeled balance_usd: schema, shape, and field name all identical, only the value lies. The committed pin blocks it per-call.
The proxy reads only the lock, so a pin must live there to be enforceable per-call. Only change: the expect block on the get_account probe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the v0.2.0 design spec (#13). Stacked on the spec branch - merge #13 first, then retarget or merge this.
What
The proxy now verifies every
tools/callresponse it forwards, not just schemas at list time. The window between two checks - where a server can start lying and every call sails through - is closed.outputSchema-> core fingerprint (intersection of the tool''s baselined probe fingerprints) -> unverifiable, counted asunverified.additionalProperties: falsedeliberately not enforced).sync()owns the schema bucket only, so the in-bandtools/listsync on client connect cannot release a response-caught liar.POST /covenant/refreshclears the response bucket;/covenant/statusshows each entry''s source.tools/callstreams are frame-parsed; only the frame with the matching JSON-RPC id is held (one frame''s latency), notifications stream through; 1 MiB cap ->skipped_large; the GET listen stream is never touched.--observe: detect, record, count - never block. WAF monitor/prevention rollout pattern.snapshotnow embedsexpectpins into the lock (the proxy reads only the baseline); lock re-snapshotted, diff is exactly the pin.covenant_response_verifications_total{tool, outcome}.Demo
Start the proxy, restart the upstream with
COVENANT_BEHAVIOR_DRIFT=1(or the newCOVENANT_SHEKEL_DRIFT=1- ILS values still labeledbalance_usd, schema and shape identical). The very next call is blocked and the tool quarantined - no refresh, no tools/list, no CI run.Tests
37 new tests (verifier unit, quarantine buckets, proxy JSON + SSE paths, observe mode, refresh release). Full suite green; ruff and strict mypy clean. Three pre-existing tests had mock upstreams returning bare text against a declared object outputSchema - under this feature that is a genuine violation, so those mocks now honor their own schema.
🤖 Generated with Claude Code