diff --git a/docs/AI_GATEWAY_STRATEGY.md b/docs/AI_GATEWAY_STRATEGY.md new file mode 100644 index 0000000..c085ed6 --- /dev/null +++ b/docs/AI_GATEWAY_STRATEGY.md @@ -0,0 +1,180 @@ +# Post-Beta AI Gateway Strategy + +Status: accepted product decision framework; target capabilities remain proposed +unless another maintained document marks them implemented. + +Last reviewed: 2026-08-24. + +## Decision + +ModelPort will mature as a governed, self-hosted hybrid AI gateway and model/GPU +control plane. It will pursue parity in the capability families that make a +small team's routes safe, operable, and explainable, not parity by feature or +Provider count. The v0.1.x Small-Team Beta contract and single-process Rust data +path remain the baseline. + +This strategy uses four status labels: + +- **Shipped**: implemented in the current supported product. +- **Hardening**: implemented in scope, but the next work improves activation, + evidence, correctness, or operator confidence. +- **Evidence-gated**: a candidate only after the stated design-partner and + operational evidence exists. +- **Deferred**: outside the post-Beta sequence; it requires a new decision and + must not be implied by current APIs or UI. + +## Market evidence and interpretation + +Official documentation from [LiteLLM](https://docs.litellm.ai/docs/simple_proxy), +[Portkey](https://portkey.ai/docs/product/ai-gateway), +[Kong AI Gateway](https://docs.konghq.com/gateway/latest/ai-gateway/), +[Envoy AI Gateway](https://aigateway.envoyproxy.io/docs/), +[OpenRouter](https://openrouter.ai/docs/guides/routing/provider-selection), +[Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/), and +[New API](https://docs.newapi.pro/en/) establish recurring families: normalized +ingress, centralized upstream credentials and models, routing/retries/fallback, +usage governance, observability, caching, and policy extensions. These sources +are comparison inputs, not compatibility claims. Product names, breadth, and +deployment models differ, and ModelPort does not embed or depend on them. + +Two standards constrain the direction. OpenTelemetry's +[generative AI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) +provide an interoperable vocabulary, while the +[W3C Trace Context recommendation](https://www.w3.org/TR/trace-context/) +defines propagation. Neither permits ModelPort to export prompt, response, tool, +credential, or unbounded attribute content by default. + +## Capability parity map + +| Capability family | Current evidence | Status and next boundary | +| --- | --- | --- | +| Unified protocol ingress | Anthropic Messages and scoped OpenAI Chat Completions share a typed Exchange IR, bounded SSE, Tool Use conversion, and fail-closed fidelity checks. | **Hardening.** Preserve native semantics and publish capability evidence per client/Provider/model tuple. OpenAI Responses, realtime, embeddings, and broad multimodal ingress are **deferred** until a design partner has a blocking workload and the IR can reject loss rather than silently downgrade. | +| Typed Provider, credential, and model management | Provider configuration, environment-backed credentials, pools, health/cooldown, aliases, model inventory, and logical routing ship. | **Hardening.** Replace relay-style “channels” with distinct Provider, credential reference, Model, and Route resources. A credential belongs to an upstream trust/account boundary; it is never a Client profile or a Model. Secret-manager integrations are **evidence-gated**. | +| Routing and resilience | Deterministic aliases, bounded retry/fallback, cooldown, capability/policy gates, shadow/canary smart routing, and durable decision evidence ship. | **Hardening.** Make eligibility, attempt order, exclusions, fallback reason, and terminal outcome explainable. Persist trustworthy latency/reliability evidence before distributed routing. Semantic routing and online weight learning are **deferred**. | +| Identity, quota, and budget governance | Scoped keys, users, teams, IP/model/Provider policy, quotas, spend controls, atomic reservations, and approvals ship for the single-host profile. | **Hardening.** Prioritize exact ownership and exact cost provenance from Provider usage plus versioned rate cards. Additional identity integrations and resource policy are **evidence-gated**. Hosted/public tenancy, payments, and reseller billing are **deferred**. | +| Content-minimized observability | Request/attempt IDs, route and policy evidence, token/cost provenance, latency, metrics, retained logs, audit rows, and operations diagnostics ship without intentional prompt/response persistence. | **Hardening.** Add content-free, OpenTelemetry-compatible spans/metrics and trace-context propagation with bounded cardinality, tenant access controls, sampling, and retention. “Provider-reported” is not invoice-exact; unknown or estimated costs stay labelled. Prompt-content telemetry is **deferred**. | +| Operator UX | Dashboard flows cover onboarding, Providers, routing, governance, requests, budgets, incidents, backup, and operations. | **Hardening.** Optimize first governed request, Provider credential test state, capability evidence, route explanation, and copyable Client profiles. Compute and Deployment views follow the dependency chain below. | +| Cache and guardrail extensions | No general response cache or pluggable content guardrail is claimed. Existing size, protocol, policy, and egress checks are gateway safety controls, not that product category. | **Evidence-gated.** See the extension contract below. Exact and semantic cache are separate proposals; semantic cache follows exact-cache evidence. Guardrails remain opt-in policy components, not an implicit inspection layer. | +| Hybrid model/GPU control plane | Versioned Runtime Adapter contracts and read-only schemas exist; first-class persisted Compute and Deployment APIs do not yet ship. | **Hardening** for the immediate inventory chain, then **evidence-gated** lifecycle and placement. Hosted Providers remain first-class and no GPU is required. | + +## Identity boundaries + +A **Client/Harness profile** explains how a caller reaches ModelPort: base URL, +supported ingress protocol, key placement, logical model, and known compatibility. +Claude Code, Codex CLI, Qwen Code, and SDK snippets belong here. Profiles do not +own upstream credentials, health, billing, or route eligibility. + +A **Provider** identifies the upstream connectivity, credential, account, +commercial, and trust boundary. A **Model** identifies capabilities and pricing +within or across Providers. DeepSeek therefore remains a Provider/model family; +it becomes a Harness only if a distinct DeepSeek caller contract is introduced +and verified. Runtime Adapters, Compute, and Deployments are operational +resources and must not be encoded as Provider channels or Harness metadata. + +## Plane ownership and deployment boundary + +The current data plane remains one Rust process. It owns request parsing, +authentication, policy/quota admission, model resolution, credential selection, +routing, Provider calls, bounded streaming, fallback, and attempt finalization. +The control plane owns typed desired configuration, identities, credentials by +reference, catalog/capability facts, budgets, Runtime Adapter registration, +observed Compute, desired/observed Deployments, audit, and operator UX. +PostgreSQL is shared persistence, not a second routing authority; the Dashboard +is a client of the control plane. + +No service split follows from these logical ownership boundaries. A future +control-plane/data-plane split requires an ADR with measured contention or +availability need, configuration distribution and staleness semantics, secret +delivery, failure isolation, migration/rollback, and equivalent acceptance +evidence. The default must continue to support the single-host deployment. + +## Ordered delivery and dependencies + +```text +#25/#26 snapshot persistence + -> #29 bounded authenticated collection + -> #30 read-only admin API + -> #31 Compute/GPU dashboard + -> desired/observed Deployment + manual reconciliation + -> policy-bounded placement +``` + +1. **Snapshot persistence (#25/#26):** atomically retain the latest accepted + adapter observation with stable identity, provenance, collection time, and + schema version; restart must not fabricate freshness. +2. **Bounded collection (#29):** authenticate adapters, bound concurrency, + duration and payload size, validate before acceptance, and retain the last + good snapshot plus explicit error/stale state. +3. **Read-only admin API (#30):** expose server-derived freshness and provenance + under existing admin authorization. It cannot trigger collection or runtime + mutation. +4. **Compute/GPU dashboard (#31):** render empty, unavailable, stale, partial, + and fresh states from the API without becoming an inventory authority. +5. **Deployment lifecycle:** only after inventory acceptance, introduce typed + desired/observed state, idempotent manual reconciliation, bounded operations, + conflict handling, rollback, and durable evidence. +6. **Policy-bounded placement:** only after lifecycle gates pass, evaluate a + recommendation first. Mutation must remain constrained by tenant, model, + capacity, egress, budget, maintenance, and approval policy. + +## Priorities and measurable gates + +| Milestone | Exit evidence required before advancing | +| --- | --- | +| Activation and Provider onboarding | Beta activation gates in the Roadmap pass; at least 80% of clean Tier 1 installs reach a governed request within 30 minutes. Every enabled production route has a resolved credential test state and dated non-stream/stream/Tool Use capability evidence for the combinations it claims. | +| Explainable resilience and exact cost | Every sent attempt records candidate, exclusion/fallback reason, outcome, usage source, rate-card version when used, and cost confidence. No estimated amount is labelled actual/billable. Mock acceptance covers each retry class; paid upstream checks remain explicit. | +| Content-free telemetry | A schema review shows no prompt, response, tool argument/result, authorization, credential, or raw body fields. Trace propagation, bounded attributes/cardinality, sampling, tenant authorization, retention, and exporter failure behavior pass local acceptance with export off by default. | +| Compute inventory | Restart, stale/unavailable, oversized/invalid observation, adapter timeout, identity conflict, and last-good-snapshot cases pass. Two design-partner environments produce stable read-only observations for two weeks without runtime mutation. | +| Deployment lifecycle | Repeated reconcile is idempotent; crash/restart, conflict, timeout, partial failure, manual rollback, and audit evidence pass. At least two design partners complete 20 manual lifecycle operations each with no orphaned allocation or unexplained state. | +| Placement experiment | 100 shadow recommendations across at least two environments satisfy policy and capacity constraints and are explainable; operators accept at least 90%, and zero recommendation mutates a runtime. Enabling bounded mutation requires a separate reviewed decision and rollback drill. | + +Activation, Provider evidence, explainable resilience, content-free telemetry, +and exact cost provenance precede semantic routing, semantic cache, broad +protocol expansion, or online learning even if those features are available in +other gateways. + +## Cache and guardrail extension contract + +Caching and guardrails may be added only as explicit, opt-in policy components. +Each proposal must define: + +- tenant/project isolation and authorization for configuration and stored data; +- which request/response fields are inspected or retained, encryption, maximum + size, TTL, deletion, legal hold interaction, and backup behavior; +- a deterministic per-request bypass that is recorded in route evidence; +- ordering relative to protocol conversion, policy, routing, retries, streaming, + accounting, and response delivery; +- timeout and overload behavior plus an explicit fail-open or fail-closed mode; +- cache key canonicalization, model/Provider/policy/version partitioning, + invalidation, usage/cost attribution, and prevention of cross-tenant hits; and +- guardrail version, verdict provenance, mutation/deny semantics, false-positive + rollback, and whether inspected content can leave the trusted host. + +No cache may make a stale policy decision reusable. No asynchronous guardrail +may be described as enforcement. Semantic cache additionally requires a threat +model for embeddings, similarity thresholds, poisoning, nondeterminism, and +false-match measurement. Until those contracts and evidence exist, ModelPort +does not claim either capability ships. + +## Deliberate non-goals + +- Public relay/reseller billing, channel resale, recharge codes, payment + processing, opaque group/model multipliers, or presenting ModelPort estimates + as a Provider invoice. +- A ModelPort-hosted service, public multi-tenancy, enterprise/HA readiness, or + competition based on Provider count. +- Conflating Client/Harness profiles, Providers, credentials, Models, Runtime + Adapters, Compute, Deployments, or Routes. +- Silent protocol or capability downgrade, generic pass-through that bypasses + policy, or compatibility claims without dated evidence. +- Prompt, response, tool argument/result, or credential content in telemetry. +- Semantic routing, semantic cache, broad protocol expansion, or online + learning before the earlier gates. +- Automatic GPU/runtime mutation before inventory and reconciliation evidence; + model inference inside the gateway; or requiring a particular runtime. + +## Decision rule + +A parity proposal must identify the design-partner problem, typed resource +owner, plane owner, privacy and failure contract, cost provenance, measurable +gate, rollback, and displaced roadmap work. If it cannot, it remains deferred. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index c61c2d3..c6b910e 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -40,6 +40,9 @@ from that governed gateway into an independent hybrid model and GPU control plane, but documentation and APIs must not present target resources as shipped behavior. [ADR-0007](adr/0007-independent-model-and-gpu-control-plane.md) defines the boundary and delivery order. +The [Post-Beta AI Gateway Strategy](AI_GATEWAY_STRATEGY.md) maps gateway parity, +control/data-plane ownership, evidence gates, and cache/guardrail boundaries +onto that resource model. The target keeps inference engines outside ModelPort: @@ -65,7 +68,7 @@ talks directly to a runtime or becomes another source of desired state. | Resource | Owner and relationship | | --- | --- | -| Client/Harness | Calls a supported ModelPort protocol edge. Claude, Codex, DeepSeek tooling, SDKs, and internal applications are clients, not Provider types. | +| Client/Harness | Calls a supported ModelPort protocol edge. Claude Code, Codex CLI, Qwen Code, SDKs, and internal applications are clients, not Provider types. DeepSeek remains a Provider/model family unless a distinct caller contract is introduced and verified. | | Provider | Governs an upstream connectivity, credential, trust, and commercial boundary. It can reference a hosted API or a Deployment-backed endpoint. | | Model | Describes a catalog identity, capabilities, limits, compatibility, and optional pricing. Catalog presence does not imply deployment or route eligibility. | | Runtime Adapter | Implements a versioned contract for inventory and bounded lifecycle operations against an external inference runtime. | @@ -103,8 +106,11 @@ Backend ownership and Dashboard navigation converge on the same eight domains: This mapping is an information architecture contract, not a requirement to create eight services. The backend remains a modular monolith until a separate -ADR proves a deployment boundary. UI migrations should preserve deep links and -API compatibility or document an explicit replacement. +ADR proves a deployment boundary, including configuration distribution, +staleness, secret delivery, failure isolation, and rollback. The supported +default remains the single-process Rust data path on one host. UI migrations +should preserve deep links and API compatibility or document an explicit +replacement. CLIProxyAPI (CPA) can be inserted only as an internal Provider boundary: diff --git a/docs/README.md b/docs/README.md index 448cc3e..60cab5c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -66,7 +66,10 @@ read every document: choose the path that matches your task. ## Project Direction [Roadmap](ROADMAP.md) defines the Small-Team Beta productization freeze and the -staged control-plane direction. [ADR-0007](adr/0007-independent-model-and-gpu-control-plane.md) +staged control-plane direction. The +[Post-Beta AI Gateway Strategy](AI_GATEWAY_STRATEGY.md) maps industry capability +families to shipped, hardening, evidence-gated, and deferred work. +[ADR-0007](adr/0007-independent-model-and-gpu-control-plane.md) defines independent model, Runtime Adapter, Compute Node/GPU, and Deployment ownership. Deferred work must not be described as shipped. @@ -83,4 +86,4 @@ ownership. Deferred work must not be described as shipped. 6. Run `node scripts/check-doc-links.mjs` and the relevant checks from [Development](DEVELOPMENT.md) before merging. -Last reviewed: 2026-08-23. +Last reviewed: 2026-08-24. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 9e372c3..4c045a6 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -3,7 +3,12 @@ Status: accepted Small-Team Beta release contract with an approved post-Beta control-plane direction. -Last reviewed: 2026-08-23. +Last reviewed: 2026-08-24. + +The ordered post-Beta capability matrix, evidence gates, extension boundaries, +and non-goals are maintained in the +[Post-Beta AI Gateway Strategy](AI_GATEWAY_STRATEGY.md). This roadmap remains +the release sequence; the strategy is the decision framework for parity work. ## Product Contract @@ -58,7 +63,14 @@ at every step: These are sequential product boundaries, not one large implementation PR. Hosted-only installations remain supported throughout. -The next focused Issues after the adapter wire contracts are: +The immediate dependency chain after the adapter wire contracts is snapshot +persistence ([#25](https://github.com/tiammomo/ModelPort/issues/25) and +[#26](https://github.com/tiammomo/ModelPort/issues/26)), bounded collection +([#29](https://github.com/tiammomo/ModelPort/issues/29)), a read-only admin API +([#30](https://github.com/tiammomo/ModelPort/issues/30)), and the Compute/GPU +dashboard ([#31](https://github.com/tiammomo/ModelPort/issues/31)). Deployment +desired/observed lifecycle follows; policy-bounded placement comes only after +reconciliation evidence. In product terms, those slices: - collect authenticated Compute Node/GPU observations, persist the latest accepted snapshots, and expose an admin read API with server-derived