AI gateway: SOTA differentiation (verifiable ledger, policy plane, guardrail mesh, outcome-aware routing, predictive budgets, LLM-aware resilience)

[rickcrawford]

Six SOTA AI-gateway capabilities that move sbproxy beyond LiteLLM parity: governance you can prove, resilience that understands LLMs, and a closed loop that self-optimizes spend. Every feature ships behind a backward-compatible, default-off config block, so the gateway's current behavior is unchanged unless an operator opts in. Each lands with unit tests, a flat docs page, and a runnable example.

What's in it

Verifiable usage ledger. A ledger usage sink hash-chains every completed-call event (SHA-256 over prev_hash || seq || recorded_at || event) into a JSONL write-ahead log, optionally Ed25519-signs each entry, and dedups by request_id (exactly-once on replay). sbproxy ai ledger verify <path> re-derives the chain and signatures and reports the first broken link. This also wires the usage-sink seam into the request path for the first time, so completed calls actually reach the configured sinks.

Unified CEL policy plane. One sandboxed CEL expression over an ai.* namespace (guardrail verdicts, budget state, routing candidate, principal context) that emits a closed, typed action set: block, redact, route_to, set_sink_tag, audit. Evaluated after guardrail/budget evaluation and before provider selection. Fails open on a policy error so a mistake can't take the gateway down.

Guardrail mesh. Evolves the input guardrails from a serial block-on-any chain into a cascade that collects the full verdict set and fuses it under a configurable rule (block on a quorum, redact-and-continue below it), with a cheap-first latency cascade and a content-addressed verdict cache. The label set feeds the policy plane.

Closed-loop outcome-aware routing. A new outcome_aware strategy that scores providers by the realized cost-per-success fed back from completed requests (an EWMA of realized cost, success rate, refusal rate, latency), demoting providers whose refusal or error rate is climbing. Round-robins while providers warm up, so it's safe to enable with no other change.

Predictive budgets with soft-landing. Degrades gracefully as a scope approaches its cap (warn, then downgrade to a cheaper model) before the hard block at 100%, instead of a cliff. The downgrade is tagged on the usage record and the live fraction is published to the policy plane.

LLM-aware resilience. Classifies each upstream failure into a typed cause (timeout, rate-limit, context-window overflow, content-policy refusal, auth, server error, bad request), maps each to its fallback list, and applies per-error-class retry counts in the failover loop, so a 429 can be retried while a refused or malformed request is not.

Composition

The features build on each other: the mesh feeds the policy plane's ai.guardrails.*, predictive budgets feed ai.budget.*, the ledger captures policy set_sink_tag decisions, and the failure classifier is the seam for compress-and-retry and redact-and-retry.

Tests and docs

Per-feature unit tests (ledger tamper/forgery/backpressure, policy action parsing and CEL evaluation, mesh fusion and cache, routing-feedback scoring, budget soft-landing thresholds, failure classification and retry policy). New docs pages under docs/ and runnable examples under examples/, all validated by the example-compile sweep. Default-off everywhere; the full local gate (fmt, build, nextest, doctest, clippy, docs, config-schema) is green.