docs: spec for per-call response verification (v0.2.0 design) - #13
Conversation
The roadmap item from the launch post: verify every tools/call response against a per-tool reference (outputSchema, else the intersection of baselined probe fingerprints, else unverifiable), block deterministic BREAKING-class violations, quarantine in a new response bucket that in-band tools/list syncs cannot release, and intercept SSE frames so the default FastMCP transport is actually covered. Enforce by default, --observe for rollout. Targets v0.2.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 16 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds a design specification for per-call Covenant response verification at the proxy, covering reference selection, verification outcomes, quarantine behavior, SSE interception, failure handling, metrics, implementation placement, and out-of-scope boundaries. ChangesCovenant verification design
Estimated code review effort: 1 (Trivial) | ~3 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/specs/2026-07-26-covenant-per-call-response-verification-design.md`:
- Around line 83-86: Update the plain-JSON response design to enforce
_VERIFY_MAX_BYTES without reading the entire body via up_resp.aread(). Specify a
bounded passthrough behavior for oversized JSON responses—streaming the
remainder unverified or an equivalent bounded strategy—and define precisely when
skipped_large is incremented, while preserving verification for responses within
the cap.
- Around line 22-42: Clarify the output verification policy around the
`outputSchema` rule and the “Extra fields” violation class: explicitly state
whether additive fields are allowed to override `additionalProperties: false`,
including for nested objects, or whether the schema remains authoritative.
Ensure the chosen policy is applied consistently wherever schema validation and
extra-field classification are described.
- Around line 58-69: Define separate persistence and restoration APIs for the
schema and response quarantine buckets, including source labels in status
output; ensure response quarantines are saved and restored independently of
Quarantine.sync(). Add an explicit response-bucket clear operation and update
POST /covenant/refresh to invoke it, while preserving sync() as schema-bucket
replacement only.
- Around line 77-85: Update the per-call response verification design to
explicitly define the outcome when the SSE stream closes, is cancelled, or
contains no frame matching the request JSON-RPC id. Assign the resulting
verification counter classification and state whether this outcome contributes
to response-bucket quarantine, keeping the behavior consistent across all
no-match termination paths.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 31a958ae-1cd0-46c2-8ccb-c13a110fb284
📒 Files selected for processing (1)
docs/specs/2026-07-26-covenant-per-call-response-verification-design.md
| - Per-frame buffer cap: **1 MiB** (`_VERIFY_MAX_BYTES`, module constant, tunable). Over | ||
| the cap, the remainder of the stream forwards unverified and `skipped_large` is | ||
| counted — a bounded verifier, never an unbounded buffer. The same cap applies to | ||
| plain-JSON bodies. |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
Make the 1 MiB cap enforceable for plain JSON.
The current plain-JSON proxy path reads the entire upstream body with up_resp.aread() before returning it. This design only explains bounded streaming for SSE, so an implementation that retains the existing JSON path can still buffer arbitrarily large responses. Specify whether oversized JSON responses are streamed unverified or handled through another bounded passthrough strategy, and when skipped_large is recorded.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/2026-07-26-covenant-per-call-response-verification-design.md`
around lines 83 - 86, Update the plain-JSON response design to enforce
_VERIFY_MAX_BYTES without reading the entire body via up_resp.aread(). Specify a
bounded passthrough behavior for oversized JSON responses—streaming the
remainder unverified or an equivalent bounded strategy—and define precisely when
skipped_large is incremented, while preserving verification for responses within
the cap.
additionalProperties deliberately not enforced (classifier parity); two-bucket lifecycle vs sync()/persistence made explicit (response bucket in-memory only); SSE stream ending without the matching frame counts unverified; the 1 MiB cap bounds verification cost, not proxy buffering (JSON path already buffers today). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Design spec for the roadmap item from the launch post: the proxy verifies every
tools/callresponse it forwards, not just schemas at list time.Key decisions locked in the spec:
outputSchema→ intersection of the tool's probe fingerprints (core shape; variance across probed args can't false-positive) → unverifiable (counted, not hidden)--observeis the rollout escape hatch (WAF monitor/prevention pattern)tools/listin-band sync that runs on every client connect;POST /covenant/refreshis the release valvetools/callonly — without it the feature covers ~nothing on default FastMCP; GET listen streams stay untouchedcovenant/verify.py, no new dependencies, one new CLI flag, targets v0.2.0Spec-only PR — implementation follows after review.
🤖 Generated with Claude Code
Summary by CodeRabbit