fix(gateway): one config object per request; drop LM cache

docs/ext-proc-gateway.md

@@ -168,6 +168,7 @@ and the pass-through fallback — is identical.
 - **Per-request configuration.** In Approach 1, all ITS parameters travel in `X-ITS-*` headers; no prior
   configuration is required. In Approach 2, the upstream LLM is configured once via `POST /configure`
   and each request supplies only its `budget` in the request body; `X-ITS-*` headers are also accepted
-  as per-request overrides (header > body > `/configure` default).
+  as per-request overrides. A field is applied if it is not `None`; otherwise the next tier down
+  (header > body > `/configure` default) supplies the value.
 - **Ports.** The values shown are defaults and may be changed: `:8108` (Envoy), `:50051` (ext_proc
   gRPC), `:8109` (IaaS service), and `:8100` (the upstream LLM / `llm_upstream`).

docs/iaas-service.md

@@ -34,8 +34,9 @@ used.
 | Request body | `budget` | Compute budget | 2 |
 | `/configure` | `budget`, `endpoint`, `api_key` | Service defaults | 3 (lowest) |
 
-Priority chain: **header > body > service default**. Headers are intended for Envoy
-ext_proc routing but are also accepted on the standalone IaaS endpoint.
+Priority chain: **header > body > service default**. A field is applied if it is
+not `None`; otherwise the next tier down supplies the value. Headers are intended for
+Envoy ext_proc routing but are also accepted on the standalone IaaS endpoint.
 
 ### Algorithm Selection
 
@@ -57,6 +58,12 @@ tool calls using the `tool_hierarchical` strategy, and text responses fall back
 exact-content matching. Supply `regex_patterns` to vote on extracted text answers, or
 set `tool_vote` to pick a different tool-voting strategy.
 
+> **Note on clearing optional fields.** Because `None` means "use the tier below," a
+> `/configure` call cannot explicitly reset `tool_vote` back to its default once set.
+> To switch voting strategies, supply the new value; to revert to the built-in default
+> (`tool_hierarchical`), pass `"tool_hierarchical"` explicitly. `regex_patterns` and
+> `exclude_tool_args` can be cleared by passing an empty list (`[]`).
+
 ## API Key Handling
 
 API keys can enter the system through three paths:
@@ -69,11 +76,12 @@ API keys can enter the system through three paths:
 **Security properties:**
 
 - Keys are **never logged** — the gateway logs endpoint and model but not credentials
-- Keys are **hashed** (SHA-256, truncated to 16 hex chars) in LM cache keys to prevent
-  credential cross-contamination between requests using different API keys
-- Keys are **not persisted** to disk — they exist only in memory for the lifetime of the
-  service process
-- On shutdown, all cached LM clients (and their associated keys) are cleared
+- Keys are **not persisted** to disk. A key supplied via the `X-ITS-API-Key` header is
+  request-scoped: it lives only for that request's LM client, which is closed when the
+  request completes. A key set via `/configure` remains in memory as a service default
+  (`ITSGateway._default_config`) until replaced or the process restarts.
+- Keys are **never shared between requests** — each request builds its own LM client, so
+  credentials supplied via header or `/configure` cannot cross-contaminate
 
 ## Prerequisites
 
@@ -181,7 +189,7 @@ All configurations support:
 - `endpoint`: OpenAI-compatible API endpoint URL
 - `api_key`: API key for the provider
 - `model`: Model identifier
-- `alg`: Algorithm name - `"self-consistency"` or `"best-of-n"`
+- `alg`: Algorithm name - `"self-consistency"`, `"adaptive-self-consistency"`, or `"beta-self-consistency"`
 
 ## Usage Examples
 
@@ -429,15 +437,15 @@ arrives as a burst rather than incrementally.
 
 ## Restart and Scaling
 
-- **State**: The service holds an in-memory LM client cache (LRU, default 64 entries),
-  a gateway instance, and the service config set via `/configure`. No state is persisted
+- **State**: The service holds a gateway instance and the service config set via
+  `/configure`. LM clients are created and discarded per request. No state is persisted
   to disk.
-- **Restart**: Restarting clears all cached LM clients and the service config.
-  `/configure` must be called again after restart.
+- **Restart**: Restarting clears the service config. `/configure` must be called again
+  after restart.
 - **Horizontal scaling**: Multiple IaaS instances can run independently. Each maintains
-  its own LM client cache, config, and gateway. There is no shared state between
-  instances. A load balancer must route `/configure` to all instances or each instance
-  must be configured independently.
+  its own config and gateway. There is no shared state between instances. A load
+  balancer must route `/configure` to all instances or each instance must be configured
+  independently.
 
 ## API Endpoints
 
@@ -450,7 +458,6 @@ arrives as a burst rather than incrementally.
 
 ### Health Check
 - `GET /docs` - API documentation
-- `GET /health` - Service health (if available)
 
 ## Troubleshooting
 
@@ -489,7 +496,7 @@ curl -X GET http://localhost:8109/docs
 **5. Slow Responses**
 - This is expected behavior for inference-time scaling
 - Reduce `budget` parameter for faster responses
-- Best-of-N with budget=4 typically takes 30-60 seconds
+- Self-consistency with budget=4 typically takes 30-60 seconds
 
 ### Log Files
 

its_hub/api/__init__.py

@@ -24,7 +24,15 @@
 from .orchestrator import AbstractOrchestrator
 from .reward_models.orm import AbstractOutcomeRewardModel
 from .reward_models.prm import AbstractProcessRewardModel
-from .types import ChatMessage, ChatMessages, GenerationUsage, ITSRequestConfig
+from .types import (
+    SUPPORTED_ALGORITHMS,
+    VALID_TOOL_VOTE_OPTIONS,
+    ChatMessage,
+    ChatMessages,
+    GenerationUsage,
+    ITSRequestConfig,
+    ITSRequestConfigUpdate,
+)
 
 __all__ = [  # noqa: RUF022
     # Algorithm abstractions
@@ -44,6 +52,9 @@
     "ChatMessages",
     "GenerationUsage",
     "ITSRequestConfig",
+    "ITSRequestConfigUpdate",
+    "SUPPORTED_ALGORITHMS",
+    "VALID_TOOL_VOTE_OPTIONS",
     # Error types
     "APIError",
     "RateLimitError",

its_hub/api/gateway.py

@@ -6,7 +6,7 @@
 from abc import ABC, abstractmethod
 from typing import Any
 
-from its_hub.api.types import ITSRequestConfig
+from its_hub.api.types import ITSRequestConfigUpdate
 
 
 class AbstractGateway(ABC):
@@ -21,7 +21,7 @@ class AbstractGateway(ABC):
     @abstractmethod
     async def arun_chat_completion(
         self,
-        config: ITSRequestConfig,
+        config: ITSRequestConfigUpdate,
         messages: list[dict[str, Any]],
         tools: list[dict[str, Any]] | None = None,
         tool_choice: str | dict[str, Any] | None = None,
@@ -31,7 +31,8 @@ async def arun_chat_completion(
         """Run chat completion with ITS algorithm asynchronously.
 
         Args:
-            config: Per-request ITS configuration
+            config: Per-request ITS configuration overlay (merged over the
+                gateway's service default at run time)
             messages: OpenAI-format conversation messages
             tools: Optional tool definitions for function calling
             tool_choice: Optional tool choice strategy
@@ -46,14 +47,9 @@ async def arun_chat_completion(
 
     def run_chat_completion(
         self,
-        config: ITSRequestConfig,
+        config: ITSRequestConfigUpdate,
         messages: list[dict[str, Any]],
         **kwargs,
     ) -> dict[str, Any]:
         """Synchronous wrapper for arun_chat_completion."""
         return asyncio.run(self.arun_chat_completion(config, messages, **kwargs))
-
-    @abstractmethod
-    async def ashutdown(self) -> None:
-        """Cleanup resources on service shutdown."""
-        pass