From a8d79133b6225af8392baa9a7ef255b52b46a989 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 12:20:01 -0400 Subject: [PATCH 01/34] K0: exercise the kubeai backend against a real cluster A GPU-less check on k3s: KubeAI's cpu profile runs real vLLM on CPU, and kubeai_k3s.sh drives doctor, `run` (acquire, one chat completion, release) and checks the released Model is pruned. First real run of the backend since it landed: it passed unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- dev/e2e_tests/kubeai-cpu-values.yaml | 9 +++ dev/e2e_tests/kubeai_k3s.sh | 64 +++++++++++++++++++ .../plan-backend-unification-2026-09-24.md | 13 +++- 3 files changed, 84 insertions(+), 2 deletions(-) create mode 100644 dev/e2e_tests/kubeai-cpu-values.yaml create mode 100755 dev/e2e_tests/kubeai_k3s.sh diff --git a/dev/e2e_tests/kubeai-cpu-values.yaml b/dev/e2e_tests/kubeai-cpu-values.yaml new file mode 100644 index 00000000..20300543 --- /dev/null +++ b/dev/e2e_tests/kubeai-cpu-values.yaml @@ -0,0 +1,9 @@ +# KubeAI on a GPU-less machine, for exercising the kubeai backend for real. +# The chart's `cpu` profile runs vLLM's CPU image (needs AVX-512). Sized for +# a sub-1B model; raise the requests for anything larger. +resourceProfiles: + cpu: + imageName: "cpu" + requests: + cpu: 8 + memory: "16Gi" diff --git a/dev/e2e_tests/kubeai_k3s.sh b/dev/e2e_tests/kubeai_k3s.sh new file mode 100755 index 00000000..e80d9c5f --- /dev/null +++ b/dev/e2e_tests/kubeai_k3s.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# End-to-end check of the kubeai backend against a real (GPU-less) cluster. +# +# Needs: k3s (or any cluster) with KubeAI installed from +# dev/e2e_tests/kubeai-cpu-values.yaml, KUBECONFIG pointing at it, and the +# KubeAI gateway reachable at KUBEAI_BASE_URL (default: a port-forward, +# `kubectl -n kubeai port-forward svc/kubeai 8000:80`). The CPU vLLM image +# needs AVX-512. +# +# Runs in a throwaway config/data dir, so it never touches a real setup: +# bash dev/e2e_tests/kubeai_k3s.sh +set -euo pipefail + +WORK=${WORK:-$(mktemp -d -t infer-stack-kubeai-XXXX)} +export INFER_STACK_CONFIG_DIR=$WORK/cfg INFER_STACK_DATA_DIR=$WORK/data +MODEL=${MODEL:-Qwen/Qwen2.5-0.5B-Instruct} +ENDPOINT=${ENDPOINT:-qwen-tiny} +KUBEAI_BASE_URL=${KUBEAI_BASE_URL:-http://127.0.0.1:8000/openai/v1} +echo "[kubeai-e2e] work dir: $WORK" + +infer-stack config init --backend kubeai --yes >/dev/null +infer-stack config set kubeai_resource_profile cpu >/dev/null +infer-stack config set kubeai_base_url "$KUBEAI_BASE_URL" >/dev/null +cat > "$INFER_STACK_CONFIG_DIR/catalog.yaml" <&2 + exit 1 +fi +echo "[kubeai-e2e] PASS" diff --git a/dev/tmp/plan-backend-unification-2026-09-24.md b/dev/tmp/plan-backend-unification-2026-09-24.md index 0c8b4e8f..24121cce 100644 --- a/dev/tmp/plan-backend-unification-2026-09-24.md +++ b/dev/tmp/plan-backend-unification-2026-09-24.md @@ -80,8 +80,17 @@ No GPU needed. Add `dev/e2e_tests/kubeai_kind.sh`: acquire, `run` a request, release, and verify the Model is pruned. Without it every step below is fake-verified only, which is how KubeAI drifted. -Open: installing `kind`/`kubectl`/`helm` on the guest needs network access to -their release downloads. +**Done 2026-09-24**, with k3s instead of kind (k3s runs as a systemd service +on the guest; `--disable traefik --disable servicelb` keeps ports 80/443 +free). KubeAI 0.23.4's `cpu` profile runs real vLLM 0.11.2 on CPU (needs +AVX-512), which is better than the simulator: KubeAI builds vLLM's own +command line, which the simulator's CLI would reject. Setup: +`dev/e2e_tests/kubeai-cpu-values.yaml`; check: +`dev/e2e_tests/kubeai_k3s.sh`, which passed. `doctor` passed its four checks; +acquire → ready took 143 s with an image pull, the full `run` took 86 s with +the image cached, a real completion came back, and release pruned the Model. +A Docker container on the node reaches KubeAI's gateway at its cluster IP, +which K1 relies on. ### K1. One client contract: the LiteLLM gateway fronts both backends From 711048b28fb46e2f7cb13b27b2efd6a7282e04cf Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 12:39:14 -0400 Subject: [PATCH 02/34] K1: front the kubeai backend with the LiteLLM gateway On KubeAI a client had to name a model by its Model name, a DNS slug of the served name; cards send the endpoint alias and got HTTP 404 (verified on k3s). The kubeai backend now owns a gateway-only ComposeBackend (own state dir, project infer-stack-gateway) and feeds it a new generic `upstream` route row per alias: the KubeAI URL plus the Model name. Clients get the same base_url, managed key and alias as on compose; readiness is judged through the gateway by alias; `secrets rotate` works unchanged. The gateway reaches KubeAI at the Service's cluster IP by default (kubeai_gateway_upstream overrides). --no-litellm keeps direct access. dev/kubeai_e2e.sh (the existing script; a duplicate I had started is folded in) sends the alias as a card does, gains GATEWAY=0 and a CPU profile, and now fails on a failed generation: the old check sat in a `&&` list where `set -e` does not apply, so a 404 printed PASS. Verified on k3s + KubeAI 0.23.4 (real vLLM on CPU): GATEWAY=1 passes, GATEWAY=0 fails with the 404, rotation accepts the new key and rejects the old, and no Model is left behind either way. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 16 +++ dev/e2e_tests/kubeai_k3s.sh | 64 ----------- dev/kubeai_e2e.sh | 31 ++++-- .../plan-backend-unification-2026-09-24.md | 9 ++ docs/kubeai-backend.md | 37 +++++-- infer_stack/backends/kubeai.py | 83 +++++++++++++- infer_stack/cli/commands_leasing.py | 16 +++ infer_stack/cli/commands_meta.py | 4 + infer_stack/leasing/compose.py | 14 +++ tests/test_leasing_kubeai.py | 102 ++++++++++++++++++ 10 files changed, 291 insertions(+), 85 deletions(-) delete mode 100755 dev/e2e_tests/kubeai_k3s.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d9819b8..95c6bdd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,22 @@ scheduler with its own GPU accounting: a lease held outside it, manual or keep-warm, occupies a GPU the scheduler believes is free, and the job it places there cannot start. +### The kubeai backend puts the LiteLLM gateway in front of the cluster + +On the kubeai backend a client had to name a model by its KubeAI Model name +(a DNS slug of the served name), not the endpoint alias. Cards send the +alias, so a card that ran on the compose backend got HTTP 404 on a cluster +(verified on k3s). The kubeai backend now runs the same LiteLLM gateway, +routing each alias to its Model: one `OPENAI_BASE_URL`, the managed key, and +the alias as the model name on both backends. `secrets rotate` works on it. +`--no-litellm` keeps the old direct access. New setting: +`kubeai_gateway_upstream`, for a gateway that cannot reach the cluster +Service's IP. + +`dev/kubeai_e2e.sh` now sends the alias as a card does, and fails when the +request fails. Before, a failed generation fell through to PASS: the check +sat in a `&&` list, where `set -e` does not apply. + ### Custom container launches are catalog data, not recipes `runtime.serve_recipe` is gone. An endpoint whose image has its own launcher diff --git a/dev/e2e_tests/kubeai_k3s.sh b/dev/e2e_tests/kubeai_k3s.sh deleted file mode 100755 index e80d9c5f..00000000 --- a/dev/e2e_tests/kubeai_k3s.sh +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env bash -# End-to-end check of the kubeai backend against a real (GPU-less) cluster. -# -# Needs: k3s (or any cluster) with KubeAI installed from -# dev/e2e_tests/kubeai-cpu-values.yaml, KUBECONFIG pointing at it, and the -# KubeAI gateway reachable at KUBEAI_BASE_URL (default: a port-forward, -# `kubectl -n kubeai port-forward svc/kubeai 8000:80`). The CPU vLLM image -# needs AVX-512. -# -# Runs in a throwaway config/data dir, so it never touches a real setup: -# bash dev/e2e_tests/kubeai_k3s.sh -set -euo pipefail - -WORK=${WORK:-$(mktemp -d -t infer-stack-kubeai-XXXX)} -export INFER_STACK_CONFIG_DIR=$WORK/cfg INFER_STACK_DATA_DIR=$WORK/data -MODEL=${MODEL:-Qwen/Qwen2.5-0.5B-Instruct} -ENDPOINT=${ENDPOINT:-qwen-tiny} -KUBEAI_BASE_URL=${KUBEAI_BASE_URL:-http://127.0.0.1:8000/openai/v1} -echo "[kubeai-e2e] work dir: $WORK" - -infer-stack config init --backend kubeai --yes >/dev/null -infer-stack config set kubeai_resource_profile cpu >/dev/null -infer-stack config set kubeai_base_url "$KUBEAI_BASE_URL" >/dev/null -cat > "$INFER_STACK_CONFIG_DIR/catalog.yaml" <&2 - exit 1 -fi -echo "[kubeai-e2e] PASS" diff --git a/dev/kubeai_e2e.sh b/dev/kubeai_e2e.sh index 8d21d1f7..cec829dd 100755 --- a/dev/kubeai_e2e.sh +++ b/dev/kubeai_e2e.sh @@ -8,6 +8,10 @@ # ./scripts/install_kubeai.sh /tmp/kubeai-values.yaml kubeai # kubectl -n kubeai port-forward svc/kubeai 8000:80 & # +# Without a GPU (verified on k3s): install the chart with +# dev/e2e_tests/kubeai-cpu-values.yaml, whose `cpu` profile runs real vLLM on +# CPU (needs AVX-512), and run with E2E_RESOURCE_PROFILE=cpu. +# # Then: ./dev/kubeai_e2e.sh # # Knobs (env): @@ -17,6 +21,9 @@ # E2E_BASE_URL gateway url (default http://127.0.0.1:8000/openai/v1) # E2E_TIMEOUT acquire readiness budget seconds (default 900 — # first run pulls model weights into the cluster) +# GATEWAY 1 (default): clients go through infer-stack's LiteLLM +# gateway, as on the compose backend. 0: straight to +# KubeAI, where the alias below does NOT route (404). set -euo pipefail MODEL="${E2E_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}" @@ -24,6 +31,7 @@ PROFILE="${E2E_RESOURCE_PROFILE:-nvidia-gpu}" NAMESPACE="${E2E_NAMESPACE:-kubeai}" BASE_URL="${E2E_BASE_URL:-http://127.0.0.1:8000/openai/v1}" TIMEOUT="${E2E_TIMEOUT:-900}" +ALIAS="$MODEL" # Isolated config/data roots, baked INLINE on every command (never rely on # exported env reaching subprocesses — see dev/audit notes on tmux env loss). @@ -56,7 +64,8 @@ models: e2e-tiny: source: hf://$MODEL endpoints: - e2e-tiny: + # Named like real catalogs, unlike its KubeAI Model name: a card sends this. + $ALIAS: engine: vllm model: e2e-tiny reclaim: {policy: stop} @@ -68,23 +77,31 @@ EOF run_is config set backend kubeai run_is config set kubeai_namespace "$NAMESPACE" run_is config set kubeai_base_url "$BASE_URL" +if [ "${GATEWAY:-1}" = 0 ]; then run_is config set litellm false; fi echo '== doctor (preflight)' run_is doctor echo '== acquire (readiness = a real generation through the gateway)' -run_is acquire e2e-tiny --yes --ttl 30m --timeout "$TIMEOUT" \ +run_is acquire "$ALIAS" --yes --ttl 30m --timeout "$TIMEOUT" \ --env-file "$WORK/lease.env" -echo '== the descriptor works for a plain OpenAI client' +echo '== a client that knows only the env file and the alias (like a card)' # shellcheck disable=SC1090 source "$WORK/lease.env" -REQUEST_NAME="$INFER_STACK_ENDPOINT_E2E_TINY" -curl -sf "$OPENAI_BASE_URL/chat/completions" \ +# An explicit `if`: in a `curl | grep && echo` list `set -e` does not fire, so +# a 404 used to fall through to PASS. +if curl -sS --fail-with-body "$OPENAI_BASE_URL/chat/completions" \ + -H "Authorization: Bearer ${OPENAI_API_KEY:-EMPTY}" \ -H 'Content-Type: application/json' \ - -d "{\"model\": \"$REQUEST_NAME\", \"max_tokens\": 8, + -d "{\"model\": \"$ALIAS\", \"max_tokens\": 8, \"messages\": [{\"role\": \"user\", \"content\": \"say ok\"}]}" \ - | grep -q 'choices' && echo ' generation ok' + | grep -q 'choices'; then + echo " generation ok via $OPENAI_BASE_URL" +else + echo "!! no generation for model=$ALIAS via $OPENAI_BASE_URL" >&2 + exit 1 +fi echo '== release prunes the Model (reclaim: stop)' run_is release --env-file "$WORK/lease.env" --yes diff --git a/dev/tmp/plan-backend-unification-2026-09-24.md b/dev/tmp/plan-backend-unification-2026-09-24.md index 24121cce..85b63c4b 100644 --- a/dev/tmp/plan-backend-unification-2026-09-24.md +++ b/dev/tmp/plan-backend-unification-2026-09-24.md @@ -115,6 +115,15 @@ Recommendation: (1) first, since it reuses everything and unblocks cards now; (2) when a real multi-node run needs it. Record (1)'s single host as a known limitation. +**Done 2026-09-24 as (1).** The kubeai backend owns a gateway-only +`ComposeBackend` (its own state dir and project, `infer-stack-gateway`) and +feeds it one generic `upstream` route row per alias (`api_base` + the Model +name). The same row type is what external endpoints need. Verified on k3s +with `dev/kubeai_e2e.sh`: without the gateway, the alias returned 404; with +it, the same request answered, and `secrets rotate` accepted the new key and +rejected the old one. Static routes only; dynamic routing stays +compose-only. + ### K2. One liveness authority Generalise `residency()` into a backend-neutral snapshot: per deployment, diff --git a/docs/kubeai-backend.md b/docs/kubeai-backend.md index 374f1aa5..9b068555 100644 --- a/docs/kubeai-backend.md +++ b/docs/kubeai-backend.md @@ -10,10 +10,19 @@ only the realization layer changes: |---|---|---| | unit of serving | docker compose service | KubeAI `Model` CR | | GPU placement | planned locally (`plan_placement`) | cluster-scheduled via `resourceProfile` | -| front door | LiteLLM gateway | KubeAI's own OpenAI-compatible gateway | -| request name | endpoint alias (LiteLLM route) | Model CR name (dns-slug of the served name) | -| auth | managed `LITELLM_MASTER_KEY` | none (`api_key: EMPTY`) | -| state dir | `/leasing/compose/` | `/leasing/kubeai/` (`models.yaml` + sidecar) | +| front door | LiteLLM gateway | the same LiteLLM gateway, on this host, routing to KubeAI | +| request name | endpoint alias | endpoint alias | +| auth | managed `LITELLM_MASTER_KEY` | the same | +| state dir | `/leasing/compose/` | `/leasing/kubeai/` (`models.yaml` + sidecar), `/leasing/kubeai-gateway/` | + +Clients see the same contract on both backends: one `OPENAI_BASE_URL`, the +managed key, and the endpoint alias as the model name, so a card runs +unchanged. The gateway is a one-service Compose project +(`infer-stack-gateway`) that routes each alias to KubeAI under the Model's +name. `secrets rotate` works as on compose. With `--no-litellm` (or `config +set litellm false`) there is no gateway: clients talk to KubeAI directly and +must use the Model name from the env file (`INFER_STACK_ENDPOINT_*`); the +alias gets HTTP 404. A deployment in the desired set renders as one `Model` CR labeled `infer-stack/managed=true` + `infer-stack/deployment=`; `apply` is @@ -54,6 +63,9 @@ infer-stack config set kubeai_namespace kubeai infer-stack config set kubeai_base_url http://127.0.0.1:8000/openai/v1 # fallback profile for endpoints whose runtime omits resource_profile: infer-stack config set kubeai_resource_profile nvidia-gpu-rtx-4090 +# how the gateway reaches KubeAI (default: the kubeai Service's cluster IP, +# reachable from a cluster node; set an ingress URL when this host is not one): +infer-stack config set kubeai_gateway_upstream http://kubeai.example/openai/v1 ``` Catalog endpoints opt into a specific profile per endpoint; the GPU count is @@ -82,9 +94,8 @@ Then the normal verbs just work: ```bash infer-stack acquire qwen-coder --ttl 2h --env-file lease.env --yes -source lease.env # OPENAI_BASE_URL -> the KubeAI gateway -# note: request the model by its CR name (the env-file's endpoint mapping -# carries it), not the endpoint alias — KubeAI has no alias layer. +source lease.env # OPENAI_BASE_URL + OPENAI_API_KEY -> the gateway +# request model=qwen-coder, the endpoint alias, as on the compose backend infer-stack release --env-file lease.env ``` @@ -105,6 +116,12 @@ infer-stack release --env-file lease.env (default 1/1 — the lease lifecycle, not the autoscaler, decides residency). - The legacy profile-era renderer (`infer_stack/backends/kubeai_renderer.py`, `kubeai_ops.py`) is superseded by this backend and kept only for reference. -- `dev/kubeai_e2e.sh` runs the full lifecycle (doctor → acquire → generation - through the gateway → release → prune-verified) against a real cluster with - an isolated config/data root; use it as the first smoke test on new setups. +- `dev/kubeai_e2e.sh` runs the full lifecycle (doctor → acquire → a request + by alias, as a card makes it → release → prune verified) against a real + cluster with an isolated config/data root; use it as the first smoke test + on new setups. Without a GPU, install the chart with + `dev/e2e_tests/kubeai-cpu-values.yaml` (real vLLM on CPU; needs AVX-512) + and run it with `E2E_RESOURCE_PROFILE=cpu`. Verified on k3s 2026-09-24. +- The gateway runs on the host running infer-stack, so that host is in every + request's path. Dynamic routing (`dynamic_routing`) is compose-only; the + KubeAI gateway uses static routes. diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index 88e5c348..c407d9b5 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -268,6 +268,8 @@ def __init__( run: Callable[[list[str]], str] | None = None, http: Any = None, assume_yes: bool = True, + gateway: Any = None, + gateway_upstream: str | None = None, ): self.state_dir = Path(state_dir) self.state_dir.mkdir(parents=True, exist_ok=True) @@ -285,6 +287,15 @@ def __init__( self.last_unplaced: set[str] = set() # KubeAI/k8s schedules; there are no host GPU indices to report. self.last_assignments: dict[str, list[int]] = {} + # The LiteLLM gateway in front of the cluster: a gateway-only + # ComposeBackend. With it, clients use the same front door, key and + # request names (the endpoint aliases) as on the compose backend. + # Without it, clients talk to KubeAI directly under its Model names. + self.gateway = gateway + # How that gateway reaches KubeAI. Unset: the KubeAI Service's cluster + # IP, which containers on a cluster node can reach; a gateway off the + # cluster needs an ingress URL here. + self.gateway_upstream = gateway_upstream # -- state-dir plumbing -------------------------------------------------- @@ -327,6 +338,51 @@ def _cluster_models(self) -> dict[str, str | None]: models[name] = (meta.get('labels') or {}).get(DEPLOYMENT_LABEL) return models + # -- the gateway ----------------------------------------------------------- + + @property + def litellm(self) -> bool: + return self.gateway is not None + + @property + def litellm_port(self) -> int | None: + return self.gateway.litellm_port if self.gateway is not None else None + + def master_key(self) -> str: + return self.gateway.master_key() + + def rotate_master_key(self) -> dict[str, str]: + return self.gateway.rotate_master_key() + + def restore_env(self, values) -> None: + self.gateway.restore_env(values) + + def gateway_accepts(self, key: str, *, wait: float = 0.0): + return self.gateway.gateway_accepts(key, wait=wait) + + def _upstream_url(self) -> str: + """Where the gateway sends requests for this cluster's Models.""" + if not self.gateway_upstream: + ip = self._kubectl(['get', 'service', 'kubeai', '-o', + 'jsonpath={.spec.clusterIP}']).strip() + if not ip: + raise RuntimeError('the kubeai Service has no cluster IP; set ' + '`config set kubeai_gateway_upstream `') + self.gateway_upstream = f'http://{ip}/openai/v1' + return self.gateway_upstream.rstrip('/') + + def _render_gateway(self, rendered: RenderedModels) -> None: + """Route each endpoint alias through the gateway to its Model.""" + from ..leasing.compose import UPSTREAM_ROUTE + + base = self._upstream_url() + self.gateway.upstream_routes = { + alias: {'engine': UPSTREAM_ROUTE, 'served': name, 'api_base': base} + for alias, name in rendered.request_names.items() + } + # Gateway only: no engines on this host, so nothing to place. + self.gateway.converge([], apply=False) + # -- converge-style surface ------------------------------------------------ def converge(self, desired: list[Deployment], *, apply: bool = True): @@ -371,6 +427,8 @@ def converge(self, desired: list[Deployment], *, apply: bool = True): 'request_names': rendered.request_names, } ) + if self.gateway is not None: + self._render_gateway(rendered) if not apply: logger.info( 'rendered {} Model(s) to {} (not applied; ' @@ -395,6 +453,8 @@ def render_profile(self) -> dict: 'namespace': self.namespace, 'base_url': self.base_url, 'resource_profile': self.default_resource_profile, + 'gateway': self.gateway is not None, + 'gateway_upstream': self.gateway_upstream, 'catalogs': catalog_sources(self.catalog), } @@ -416,6 +476,7 @@ def use_profile(self, profile: dict) -> None: self.namespace = profile['namespace'] self.base_url = profile['base_url'] self.default_resource_profile = profile['resource_profile'] + self.gateway_upstream = profile.get('gateway_upstream') or self.gateway_upstream sources = profile.get('catalogs') or [] self.catalog = CatalogUnion.from_sources(sources) if sources else None @@ -457,6 +518,8 @@ def apply(self) -> None: self._kubectl( ['delete', 'models.kubeai.org', name, '--ignore-not-found'] ) + if self.gateway is not None: + self.gateway.apply() def observe(self) -> set[str]: """Deployment ids with a managed Model CR on the cluster (best-effort).""" @@ -475,12 +538,20 @@ def probe_ready(self, deployment: Deployment, endpoint: str) -> Readiness: """ if deployment.id not in self.observe(): return Readiness(False, 'Model CR not on the cluster') - name = model_name_for(_served_name(deployment)) served = deployment.served.get(endpoint) or {} protocol = served.get('protocol') or 'chat' + if self.gateway is not None: + # Ready means ready the way a client sees it: the alias, through + # the gateway, with its key. + base, model = f'{self.gateway._gateway_base()}/v1', endpoint + headers = self.gateway._auth_headers() + else: + base, model = self.base_url, model_name_for(_served_name(deployment)) + headers = None ok, reason = openai_ready( - base_url=self.base_url, - model=name, + base_url=base, + headers=headers, + model=model, protocol=protocol, require_listed=True, require_generation=True, @@ -496,6 +567,8 @@ def access(self, endpoints: list[str]) -> dict[str, Any] | None: no alias layer. The gateway is unauthenticated, so the api key is the literal ``EMPTY`` placeholder and no key env var is advertised. """ + if self.gateway is not None: + return self.gateway.access(endpoints) request_names = self._load_sidecar().get('request_names') or {} return { 'base_url': self.base_url, @@ -523,11 +596,13 @@ def teardown(self, deployment: Deployment) -> None: ) def down(self) -> None: - """Delete every infer-stack-managed Model (explicit stop).""" + """Delete every infer-stack-managed Model (explicit stop), and the gateway.""" for name in sorted(self._cluster_models()): self._kubectl( ['delete', 'models.kubeai.org', name, '--ignore-not-found'] ) + if self.gateway is not None: + self.gateway.down() # -- preflight ------------------------------------------------------------- diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index e3f66355..8167ad50 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -276,6 +276,20 @@ def _make_backend(config, *, interactive: bool = False): if name == 'kubeai': from ..backends.kubeai import KubeaiBackend + gateway = None + if _resolve_litellm(config): + # The same LiteLLM front door as the compose backend, fronting the + # cluster: one base_url, the managed key, and endpoint aliases as + # request names. Its own state dir and compose project, so it can + # never touch a compose stack's containers on the same host. + gateway = ComposeBackend( + state_dir=data_root() / 'leasing' / 'kubeai-gateway', + inventory={'gpu_count': 0, 'gpus': []}, + project='infer-stack-gateway', + litellm=True, + ui=False, + assume_yes=_resolve_assume_yes(config, interactive=interactive), + ) backend = KubeaiBackend( state_dir=data_root() / 'leasing' / 'kubeai', namespace=get_setting('kubeai_namespace') or 'kubeai', @@ -283,6 +297,8 @@ def _make_backend(config, *, interactive: bool = False): default_resource_profile=get_setting('kubeai_resource_profile') or None, assume_yes=_resolve_assume_yes(config, interactive=interactive), + gateway=gateway, + gateway_upstream=get_setting('kubeai_gateway_upstream') or None, ) try: backend.catalog = _load_catalog(config) # frozen into the profile diff --git a/infer_stack/cli/commands_meta.py b/infer_stack/cli/commands_meta.py index fa43990b..8d382139 100644 --- a/infer_stack/cli/commands_meta.py +++ b/infer_stack/cli/commands_meta.py @@ -362,6 +362,10 @@ class _Setting: 'kubeai_resource_profile': 'Fallback KubeAI resourceProfiles name for catalog endpoints whose ' 'runtime omits resource_profile.', + 'kubeai_gateway_upstream': + 'URL the LiteLLM gateway uses to reach KubeAI (default: the kubeai ' + "Service's cluster IP, reachable from a cluster node; set an ingress " + 'URL when the gateway runs off the cluster).', } diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index 3fafe135..5a7b346f 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -97,6 +97,9 @@ # ComposeBackend._update_route_registry. LITELLM_REGISTRY_FILENAME = 'litellm_registry.json' LITELLM_REGISTRY_VERSION = 1 +# A route-registry row for a server this project does not run: +# ``{'engine': UPSTREAM_ROUTE, 'served': , 'api_base': }``. +UPSTREAM_ROUTE = 'upstream' POSTGRES_SERVICE = 'postgres-litellm' POSTGRES_CONTAINER_PORT = 5432 POSTGRES_DB_NAME = 'litellm' @@ -928,6 +931,12 @@ def _litellm_model_list_from_registry( f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' ) entries.append(_ollama_route_entry(name, tag, api_base)) + elif engine == UPSTREAM_ROUTE and row.get('api_base'): + # An OpenAI-compatible server this project does not run (a KubeAI + # cluster's gateway): the row carries its address and the name it + # serves the model under. + entries.append(_vllm_route_entry(name, row.get('served') or name, + str(row['api_base']))) return entries @@ -1914,6 +1923,9 @@ def __init__( # Called with a one-line message during long steps (image pulls); the # TUI sets it. Always also logged. self.progress: Callable[[str], None] | None = None + # alias -> UPSTREAM_ROUTE row, set by a backend that uses this one only + # as its gateway (the kubeai backend). Merged into the route registry. + self.upstream_routes: dict[str, dict[str, Any]] = {} self.images = {**PINNED_IMAGES, **(images or {})} self.ports = {**DEFAULT_PORTS, **(ports or {})} # Merge over the defaults (not replace) so a caller-supplied partial @@ -2608,6 +2620,8 @@ def _merged_route_registry( # live cross-runbook deployment routable (and, via persistence, routable # past release). incoming.update(_registry_incoming_from_deployments(desired, assignments)) + # Routes to servers another backend runs (see UPSTREAM_ROUTE). + incoming.update(self.upstream_routes) merged, warnings = _merge_route_registry(existing, incoming) for w in warnings: logger.warning(' route registry: {}', w) diff --git a/tests/test_leasing_kubeai.py b/tests/test_leasing_kubeai.py index 899c7875..f5414eb4 100644 --- a/tests/test_leasing_kubeai.py +++ b/tests/test_leasing_kubeai.py @@ -532,3 +532,105 @@ def test_doctor_cli_backends_without_preflight(monkeypatch, capsys): ) assert DoctorCLI.main(argv=[]) == 0 assert 'nothing to verify' in capsys.readouterr().out + + +# -- the LiteLLM gateway in front of the cluster (plan step K1) --------------- +# +# Real-cluster evidence: dev/e2e_tests/kubeai_k3s.sh. Without the gateway a +# card's alias got HTTP 404 from KubeAI; with it the same request answered. + + +UPSTREAM = 'http://10.43.0.9/openai/v1' + + +class GatewayHttp(FakeHttp): + """The LiteLLM gateway on 127.0.0.1, routing by its rendered model_list.""" + + def __init__(self, kubectl, gateway_dir): + super().__init__(kubectl) + self.gateway_dir = Path(gateway_dir) + + def _routes(self): + cfg = self.gateway_dir / 'litellm_config.yaml' + doc = yaml.safe_load(cfg.read_text()) if cfg.exists() else {} + return {e['model_name']: e['litellm_params'] for e in doc.get('model_list') or []} + + def get(self, url, **kw): + if url.startswith('http://127.0.0.1:') and url.endswith('/models'): + return self._Resp(200, {'data': [{'id': a} for a in self._routes()]}) + return super().get(url, **kw) + + def post(self, url, **kw): + if url.startswith('http://127.0.0.1:'): + route = self._routes().get((kw.get('json') or {}).get('model')) + if route is None or route['api_base'] != UPSTREAM: + return self._Resp(404, {'detail': 'no such alias'}) + kw = {**kw, 'json': {**kw['json'], 'model': route['model'].split('/', 1)[1]}} + return super().post(url, **kw) + + +def make_gateway_backend(tmp_path): + from infer_stack.leasing.compose import ComposeBackend + from test_leasing_compose import FakeDocker + + kubectl = FakeKubectl() + gateway = ComposeBackend( + state_dir=tmp_path / 'gateway', inventory={'gpu_count': 0, 'gpus': []}, + run=FakeDocker(), http=GatewayHttp(kubectl, tmp_path / 'gateway'), + project='infer-stack-gateway', litellm=True, ui=False, + images={'litellm': 'litellm:test'}, + ) + be = KubeaiBackend(state_dir=tmp_path / 'kubeai', run=kubectl, + http=gateway.http, gateway=gateway, gateway_upstream=UPSTREAM) + return be, kubectl + + +def test_gateway_routes_each_alias_to_its_model(tmp_path): + be, _ = make_gateway_backend(tmp_path) + dep = vllm('grp-a', served='Qwen/Qwen2.5-0.5B') + dep.served = {'Qwen/Qwen2.5-0.5B': {'served_model_name': 'Qwen/Qwen2.5-0.5B', + 'protocol': 'chat'}} + be.converge([dep]) + route = GatewayHttp._routes(be.gateway.http)['Qwen/Qwen2.5-0.5B'] + assert route == {'model': 'openai/qwen-qwen2-5-0-5b', 'api_base': UPSTREAM, + 'api_key': 'EMPTY'} + + +def test_with_a_gateway_clients_use_the_alias_and_the_managed_key(tmp_path): + be, _ = make_gateway_backend(tmp_path) + dep = vllm('grp-a', served='Qwen/Qwen2.5-0.5B') + dep.served = {'tiny': {'served_model_name': 'Qwen/Qwen2.5-0.5B', 'protocol': 'chat'}} + be.converge([dep]) + info = be.access(['tiny']) + assert info['request_names'] == {'tiny': 'tiny'} # the alias, as on compose + assert info['base_url'].startswith('http://127.0.0.1:') + assert info['api_key'] == be.gateway.master_key() + # Ready is judged the way a client sees it: the alias, through the gateway. + assert be.probe_ready(dep, 'tiny').ready + + +def test_without_a_gateway_clients_still_see_kubeai_directly(tmp_path): + be, _ = make_backend(tmp_path) + be.converge([vllm('grp-a', served='qwen-32b')]) + assert be.litellm is False + assert be.access(['grp-a'])['request_names'] == {'grp-a': 'qwen-32b'} + + +def test_rotating_the_key_recreates_the_gateway_in_front_of_the_cluster(tmp_path): + from infer_stack.leasing.residency import FINGERPRINT_LABEL + + be, _ = make_gateway_backend(tmp_path) + ledger = Ledger(SqliteStore(str(tmp_path / 'ledger.db'))) + ctl = Controller(ledger, be) + out = ctl.acquire('alice', [_req('qwen', profile='cpu')], wait=False) + ctl.release(out.lease.id) + old = be.master_key() + + def fingerprint(): + doc = yaml.safe_load(be.gateway.compose_file.read_text()) + return doc['services']['litellm']['labels'][FINGERPRINT_LABEL] + + before = fingerprint() + ctl.rotate_gateway_key() + assert be.master_key() != old + assert fingerprint() != before From b7bc3e8ca78ef3ac4bdae0960f4d713ebdcd9d36 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 12:40:50 -0400 Subject: [PATCH 03/34] Remove the unused profile-era KubeAI renderer render_kubeai_artifacts had no callers once the profile path was excised; it was a second KubeAI renderer beside the backend. The doc claimed it and a kubeai_ops.py (already gone) were kept for reference. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 4 + docs/kubeai-backend.md | 2 - infer_stack/backends/__init__.py | 3 +- infer_stack/backends/kubeai_renderer.py | 204 ------------------------ 4 files changed, 5 insertions(+), 208 deletions(-) delete mode 100644 infer_stack/backends/kubeai_renderer.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 95c6bdd0..bac4ff49 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,10 @@ the alias as the model name on both backends. `secrets rotate` works on it. `kubeai_gateway_upstream`, for a gateway that cannot reach the cluster Service's IP. +The profile-era KubeAI renderer (`infer_stack.backends.render_kubeai_artifacts`) +is removed. Nothing called it since the profile path was excised; the kubeai +backend is the one renderer for KubeAI. + `dev/kubeai_e2e.sh` now sends the alias as a card does, and fails when the request fails. Before, a failed generation fell through to PASS: the check sat in a `&&` list, where `set -e` does not apply. diff --git a/docs/kubeai-backend.md b/docs/kubeai-backend.md index 9b068555..b6255345 100644 --- a/docs/kubeai-backend.md +++ b/docs/kubeai-backend.md @@ -114,8 +114,6 @@ infer-stack release --env-file lease.env `--backend compose` for those. - `min_replicas` / `max_replicas` runtime keys pass through to the CR (default 1/1 — the lease lifecycle, not the autoscaler, decides residency). -- The legacy profile-era renderer (`infer_stack/backends/kubeai_renderer.py`, - `kubeai_ops.py`) is superseded by this backend and kept only for reference. - `dev/kubeai_e2e.sh` runs the full lifecycle (doctor → acquire → a request by alias, as a card makes it → release → prune verified) against a real cluster with an isolated config/data root; use it as the first smoke test diff --git a/infer_stack/backends/__init__.py b/infer_stack/backends/__init__.py index 065b27bc..aabed088 100644 --- a/infer_stack/backends/__init__.py +++ b/infer_stack/backends/__init__.py @@ -1,4 +1,3 @@ from .kubeai import KubeaiBackend -from .kubeai_renderer import render_kubeai_artifacts # legacy profile-era path -__all__ = ['KubeaiBackend', 'render_kubeai_artifacts'] +__all__ = ['KubeaiBackend'] diff --git a/infer_stack/backends/kubeai_renderer.py b/infer_stack/backends/kubeai_renderer.py deleted file mode 100644 index 21512310..00000000 --- a/infer_stack/backends/kubeai_renderer.py +++ /dev/null @@ -1,204 +0,0 @@ -from __future__ import annotations - -from pathlib import Path -from typing import Any - -import yaml - -from ..config import KUBEAI_GENERATED_SUBDIR, normalized_output -from ..diff_prompt import confirm_writes -from ..profile_runtime import vllm_args - - -def _resource_profile_values(plan: dict[str, Any]) -> dict[str, Any]: - values_doc = plan.get('deployment', {}).get('resource_profiles_values', {}) - return values_doc or {'resourceProfiles': {}} - - -def _kubeai_resource_profile(service: dict[str, Any]) -> str: - profile = str(service.get('resource_profile', '')) - if not profile or ':' in profile: - return profile - gpu_count = max( - 1, - len(service.get('gpu_indices', [])), - int(service.get('tensor_parallel_size', 1) or 1) - * int(service.get('pipeline_parallel_size', 1) or 1) - * int(service.get('data_parallel_size', 1) or 1), - ) - return f'{profile}:{gpu_count}' - - -def _kubeai_args(service: dict[str, Any]) -> list[str]: - kubeai_service = dict(service) - kubeai_service['served_model_name'] = service['profile_public_name'] - return vllm_args(kubeai_service) - - -def _model_doc(service: dict[str, Any]) -> dict[str, Any]: - doc = { - 'apiVersion': 'kubeai.org/v1', - 'kind': 'Model', - 'metadata': { - 'name': service['kubernetes_name'], - 'annotations': { - 'infer-stack/profile-name': service['profile_name'], - 'infer-stack/public-name': service['profile_public_name'], - 'infer-stack/logical-model-name': service['logical_model_name'], - 'infer-stack/protocol-mode': service['protocol_mode'], - }, - }, - 'spec': { - 'features': service.get('features', ['TextGeneration']), - 'url': service['model_url'], - 'engine': service.get('engine', 'VLLM'), - 'resourceProfile': _kubeai_resource_profile(service), - 'minReplicas': int(service.get('min_replicas', 0)), - 'maxReplicas': int(service.get('max_replicas', 1)), - 'args': _kubeai_args(service), - }, - } - if service.get('priority_class_name'): - doc['spec']['priorityClassName'] = service['priority_class_name'] - return doc - - -def render_kubeai_artifacts( - lock_data: dict, *, assume_yes: bool = True -) -> None: - """Render the KubeAI backend artifacts. - - When ``assume_yes`` is False, all rendered YAML files are diffed against - their on-disk versions and the user is prompted via Rich before any file - is written. - """ - deployment = lock_data.get('deployment', {}) - cluster = deployment.get('cluster', {}) - namespace = cluster.get('namespace', 'kubeai') - output_root = Path( - normalized_output(deployment.get('output'))['generated_dir'] - ) - generated = output_root / KUBEAI_GENERATED_SUBDIR - generated.mkdir(parents=True, exist_ok=True) - - namespace_doc = { - 'apiVersion': 'v1', - 'kind': 'Namespace', - 'metadata': {'name': namespace}, - } - namespace_text = yaml.safe_dump(namespace_doc, sort_keys=False) - - values_doc = _resource_profile_values(lock_data) - values_text = yaml.safe_dump(values_doc, sort_keys=False) - - model_docs = [ - _model_doc(service) - for service in ( - deployment.get('providers', {}).get('vllm', {}).get('runtimes', {}) - or {} - ).values() - ] - model_text = '---\n'.join( - yaml.safe_dump(doc, sort_keys=False) for doc in model_docs - ) - - ingress = cluster.get('ingress', {}) or {} - ingress_path = generated / 'ingress.yaml' - ingress_text: str | None = None - if ingress.get('enabled'): - path_prefix = ingress.get('path_prefix', '/') or '/' - ingress_doc: dict[str, Any] = { - 'apiVersion': 'networking.k8s.io/v1', - 'kind': 'Ingress', - 'metadata': { - 'name': cluster.get('service_name', 'kubeai'), - 'namespace': namespace, - }, - 'spec': { - 'ingressClassName': ingress.get('class_name', 'traefik'), - 'rules': [ - { - 'http': { - 'paths': [ - { - 'path': path_prefix, - 'pathType': 'Prefix', - 'backend': { - 'service': { - 'name': cluster.get( - 'service_name', 'kubeai' - ), - 'port': {'number': 80}, - } - }, - } - ] - } - } - ], - }, - } - if ingress.get('host'): - ingress_doc['spec']['rules'][0]['host'] = ingress['host'] - if ingress.get('tls_secret_name') and ingress.get('host'): - ingress_doc['spec']['tls'] = [ - { - 'hosts': [ingress['host']], - 'secretName': ingress['tls_secret_name'], - } - ] - ingress_text = yaml.safe_dump(ingress_doc, sort_keys=False) - - readme = f"""# Generated KubeAI artifacts - -Namespace: `{namespace}` -Release: `{cluster.get('kubeai_release_name', 'kubeai')}` -Chart: `{cluster.get('kubeai_chart', 'kubeai/kubeai')}` - -Files: -- `namespace.yaml`: namespace to apply before the chart and models -- `kubeai-values.yaml`: custom resource profiles for the KubeAI chart -- `models.yaml`: KubeAI `Model` objects derived intentionally from the selected serving profile(s) -- `ingress.yaml`: optional ingress for one stable hostname - -Typical flow: - -```bash -kubectl apply -f {generated}/namespace.yaml -helm repo add kubeai https://www.kubeai.org --force-update -helm repo update -helm upgrade --install {cluster.get('kubeai_release_name', 'kubeai')} {cluster.get('kubeai_chart', 'kubeai/kubeai')} \ - -n {namespace} --create-namespace \ - -f {generated}/kubeai-values.yaml \ - --wait -kubectl apply -f {generated}/models.yaml -``` -""" - - namespace_path = generated / 'namespace.yaml' - values_path = generated / 'kubeai-values.yaml' - models_path = generated / 'models.yaml' - readme_path = generated / 'README.md' - - planned: dict[Path, str] = { - namespace_path: namespace_text, - values_path: values_text, - models_path: model_text, - readme_path: readme, - } - if ingress_text is not None: - planned[ingress_path] = ingress_text - - if not confirm_writes( - planned, assume_yes=assume_yes, title='Pending KubeAI render' - ): - raise SystemExit('Aborted by user; no files were written.') - - namespace_path.write_text(namespace_text, encoding='utf-8') - values_path.write_text(values_text, encoding='utf-8') - models_path.write_text(model_text, encoding='utf-8') - readme_path.write_text(readme, encoding='utf-8') - if ingress_text is not None: - ingress_path.write_text(ingress_text, encoding='utf-8') - elif ingress_path.exists(): - ingress_path.unlink() From e84c131e8f04c65730625105a4e3cfbca90197fb Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 12:51:40 -0400 Subject: [PATCH 04/34] K2+K4: strict pod residency and shared crash diagnosis for kubeai K2: KubeaiBackend.residency() builds the same Residency as compose from `kubectl get pods` (KubeAI copies the Model's infer-stack labels onto its pods), mapping pod states onto the Docker vocabulary and keeping the Kubernetes reason; a kubectl failure raises ResidencyUnknown. `gc --orphans` and `network migrate` stop using "has residency" to mean compose. K4: the crash diagnosis moves to leasing/diagnosis.py (diagnose_startup(instances, read_logs)); compose re-exports the old names. The kubeai backend feeds it pods and pod logs (the previous run last, where the crash is), probes fail fast with the engine's words, and a not-ready wait names the pod reason (Unschedulable, ImagePullBackOff). `error: unrecognized arguments` is now a fatal signature on both backends. Verified on k3s + KubeAI: a Model whose vLLM rejected a flag failed its acquire in 52 s (timeout 800 s), quoting the error and its likely cause; live residency read running, starting and crashed pods correctly. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 16 ++ .../plan-backend-unification-2026-09-24.md | 30 +++- docs/kubeai-backend.md | 6 + infer_stack/backends/kubeai.py | 75 ++++++++- infer_stack/cli/commands_leasing.py | 8 +- infer_stack/leasing/compose.py | 129 ++------------- infer_stack/leasing/diagnosis.py | 150 ++++++++++++++++++ infer_stack/leasing/residency.py | 97 +++++++++++ tests/test_leasing_kubeai.py | 108 +++++++++++++ tests/test_leasing_startup_failure.py | 2 + 10 files changed, 494 insertions(+), 127 deletions(-) create mode 100644 infer_stack/leasing/diagnosis.py diff --git a/CHANGELOG.md b/CHANGELOG.md index bac4ff49..5781eaed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,22 @@ scheduler with its own GPU accounting: a lease held outside it, manual or keep-warm, occupies a GPU the scheduler believes is free, and the job it places there cannot start. +### The kubeai backend fails fast on an engine that cannot start + +The crash diagnosis (restart count, exit code, and the engine log classified +as fatal, transient or unknown) was Docker-only, so on a cluster a model that +could never start held its lease for the whole timeout. It now lives in +`infer_stack/leasing/diagnosis.py` and both backends use it. The kubeai +backend reads pods through a new strict `residency()` (a kubectl failure +raises instead of reading as "nothing running") and quotes the crashed run's +log (`kubectl logs --previous`). A not-ready wait names the pod's reason, such +as `Unschedulable` or `ImagePullBackOff`. vLLM rejecting a flag (`error: +unrecognized arguments`) is now recognised as fatal on both backends; before, +it waited for two restarts. + +`gc --orphans` and `network migrate` now refuse a non-compose backend +explicitly. They had used "the backend has `residency`" to mean compose. + ### The kubeai backend puts the LiteLLM gateway in front of the cluster On the kubeai backend a client had to name a model by its KubeAI Model name diff --git a/dev/tmp/plan-backend-unification-2026-09-24.md b/dev/tmp/plan-backend-unification-2026-09-24.md index 85b63c4b..350eb231 100644 --- a/dev/tmp/plan-backend-unification-2026-09-24.md +++ b/dev/tmp/plan-backend-unification-2026-09-24.md @@ -38,12 +38,18 @@ path carries the keep-warm fix, atomic acquire and serialised publication review; the other path is the pre-September one. In production only KubeAI takes it, and nothing has run it at scale. -### C. Two liveness authorities +### C. No strict liveness view on KubeAI -Compose answers "what is running" with `residency()`: strict, and it raises -`ResidencyUnknown` rather than guess. KubeAI answers with `observe()`, which -returns the **empty set** when `kubectl` fails. Compose had that exact bug: -an unreadable Docker looked like "nothing running" and led to wrong decisions. +Both backends have a lenient `observe()` that returns an empty set when the +runtime cannot be read. That is deliberate: it is for reporting. Compose also +has `residency()`, a strict view that raises `ResidencyUnknown` rather than +guess, and every decision that stops, evicts or hands over a GPU uses it. +KubeAI had only the lenient view. (Corrected 2026-09-24: the first draft +called KubeAI's `observe()` a bug; it matches Compose's contract.) + +Three CLI commands (`gc --orphans`, its orphan listing, `network migrate`) +used "has `residency`" to mean "is the Compose backend", which is a duplicate +authority of its own. ### D. Failure diagnosis only exists on Compose @@ -132,6 +138,14 @@ reason, waiting reason, and GPUs when the backend knows them. Build KubeAI's from `kubectl get pods -l model= -o json`, raising `ResidencyUnknown` when kubectl fails. `observe()` becomes a thin view of it on both backends. +**Done 2026-09-24.** `residency_from_pods` builds the same `Residency` +from `kubectl get pods`, selecting on the `infer-stack/managed` label KubeAI +copies onto pods. Pod states map onto the Docker vocabulary, with the +Kubernetes reason kept in `Container.reason`, and a kubectl failure raises. +With it, `_never_ran`, the `config publish` quiescence check and the status +view use strict residency on KubeAI too. The three CLI commands now check for +the Compose backend explicitly. + ### K3. One acquire path Give placement to the backend: @@ -155,6 +169,12 @@ The fail-fast wait, the "likely cause" text and the TUI error path then work on both. Pull progress stays Compose-only; Kubernetes pulls by itself, and `ImagePullBackOff` is reported as a failure. +**Done 2026-09-24.** The diagnosis moved to `leasing/diagnosis.py` +(`diagnose_startup(instances, read_logs)`), and both backends call it. On k3s, +a Model whose vLLM rejected a flag failed its acquire in 52 s with the +engine's error quoted, instead of waiting out the 800 s timeout. Adding +`error: unrecognized arguments` as a fatal signature helps compose as well. + ### K5. Small cleanups - one served-name function diff --git a/docs/kubeai-backend.md b/docs/kubeai-backend.md index b6255345..df1f4e8f 100644 --- a/docs/kubeai-backend.md +++ b/docs/kubeai-backend.md @@ -107,6 +107,12 @@ infer-stack release --env-file lease.env - **Placement can only fail at admission.** The render never rejects for capacity (the cluster schedules); a Model the cluster cannot place sits not-ready until the acquire's `--timeout`, which then rolls the lease back. + The wait says why (`pod: Unschedulable`, `pod: ImagePullBackOff`). +- **An engine that cannot start fails the acquire at once**, as on compose: + the crash diagnosis reads the pods (`kubectl get pods`, restart count, last + exit) and the previous run's log, and quotes the engine's error with a + likely cause. Verified on k3s: a rejected vLLM flag failed in 52 s instead + of the 800 s timeout. Loud render failures do exist for: a missing `resource_profile` (no invalid CR is ever written), served-name collisions between simultaneously live deployments, and **ollama endpoints** — KubeAI serves models, not daemons, diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index c407d9b5..aa0e0a0e 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -521,6 +521,71 @@ def apply(self) -> None: if self.gateway is not None: self.gateway.apply() + def residency(self): + """The managed Models' pods, strictly: raises rather than guess. + + Unlike :meth:`observe`, a kubectl failure is never "nothing running"; + see :mod:`infer_stack.leasing.residency`. + """ + from ..leasing.residency import ( + POD_MANAGED_LABEL, + ResidencyUnknown, + residency_from_pods, + ) + + try: + raw = self._kubectl(['get', 'pods', '-l', f'{POD_MANAGED_LABEL}=true', + '-o', 'json']) + except Exception as ex: # noqa: BLE001 - any failure is "unknown" + raise ResidencyUnknown(f'kubectl get pods failed: {ex}') from ex + return residency_from_pods(raw) + + def deployment_logs(self, deployment: Deployment, *, tail: int = 400) -> str: + """Recent engine logs: each pod's current run, then its previous one. + + After a crash the kubelet has already restarted the container, so the + cause is in the previous run's log; it goes last, where the diagnosis + quotes from. Fail-open to ``''``. + """ + from ..leasing.residency import ResidencyUnknown + + try: + pods = self.residency().containers(deployment.id) + except ResidencyUnknown: + return '' + parts: list[str] = [] + for pod in pods: + runs = [[]] + ([['--previous']] if pod.restart_count else []) + for extra in runs: + try: + parts.append(self._kubectl(['logs', pod.container_id, '--tail', + str(tail), *extra])) + except Exception: # noqa: BLE001 - gone, or no previous run + pass + return '\n'.join(p for p in parts if p) + + def startup_failure(self, deployment: Deployment) -> str | None: + """Why this Model's engine cannot start, or ``None`` if it may still load.""" + from ..leasing.diagnosis import diagnose_startup + from ..leasing.residency import ResidencyUnknown + + try: + pods = self.residency().containers(deployment.id) + except ResidencyUnknown: + return None # cannot read the cluster: say nothing + return diagnose_startup(pods, lambda: self.deployment_logs(deployment, tail=200)) + + def _waiting_reason(self, deployment: Deployment) -> str: + """The pod's own reason for not running yet (``ImagePullBackOff``...).""" + from ..leasing.residency import ResidencyUnknown + + try: + pods = self.residency().containers(deployment.id) + except ResidencyUnknown: + return '' + reasons = sorted({p.reason for p in pods if p.reason and p.state != 'running'}) + return ', '.join(reasons) + def observe(self) -> set[str]: """Deployment ids with a managed Model CR on the cluster (best-effort).""" try: @@ -557,7 +622,15 @@ def probe_ready(self, deployment: Deployment, endpoint: str) -> Readiness: require_generation=True, http=self.http, ) - return Readiness(ok, reason) + if ok: + return Readiness(True, reason) + # A crash-looping engine keeps its pod "Running" between restarts, so + # check every not-ready probe, not only when the Model is missing. + failure = self.startup_failure(deployment) + if failure is not None: + return Readiness(False, failure, fatal=True) + waiting = self._waiting_reason(deployment) + return Readiness(False, f'{reason} (pod: {waiting})' if waiting else reason) def access(self, endpoints: list[str]) -> dict[str, Any] | None: """Where a client reaches these endpoints (env-file descriptor). diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index 8167ad50..56ad31d5 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -1270,7 +1270,7 @@ def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) controller = _open_controller(config, interactive=True) if config.orphans: - if not callable(getattr(controller.backend, 'residency', None)): + if not isinstance(controller.backend, ComposeBackend): raise SystemExit('gc --orphans needs the compose backend') def confirm(found): @@ -1359,8 +1359,8 @@ def main(cls, argv=True, **kwargs): held = [g for g in deployments if g.state in (DeploymentState.LIVE, DeploymentState.IDLE)] observed, assignments = _placement_view(controller) - can_orphan = bool(config.orphans) and callable( - getattr(controller.backend, 'residency', None)) + can_orphan = bool(config.orphans) and isinstance( + controller.backend, ComposeBackend) found: list = [] @@ -2819,7 +2819,7 @@ def main(cls, argv=True, **kwargs): if not config.subnet: raise SystemExit('network migrate: --subnet is required') controller = _open_controller(config, interactive=True) - if not callable(getattr(controller.backend, 'residency', None)): + if not isinstance(controller.backend, ComposeBackend): raise SystemExit('network migrate needs the compose backend') try: rec = controller.network_migrate(config.subnet, force=bool(config.force)) diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index 5a7b346f..27306411 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -260,93 +260,17 @@ def profile_images(profile: dict[str, Any]) -> list[str]: return sorted(wanted) -#: Restarts after which Docker's own bookkeeping says an engine is looping, -#: not loading. Two is already conclusive: a model that loads does not exit. -CRASH_LOOP_RESTARTS = 2 - -#: Engine log lines worth quoting verbatim, and what they mean. Each is -#: UNRECOVERABLE: the same container, restarted, fails the same way, so one -#: crash is already conclusive and there is nothing to wait for. -_ENGINE_ERROR_HINTS = ( - ('trust_remote_code', 'the model needs trust_remote_code=True ' - '(set `runtime.trust_remote_code: true` on the endpoint)'), - ('does not recognize this architecture', 'this engine cannot read the ' - "model's architecture (it may need trust_remote_code=True)"), - ('are not supported for now', 'this vLLM build does not implement the ' - "model's architecture"), - ('is not supported', 'this vLLM build does not implement the ' - "model's architecture"), - ('No supported config format', 'the model repository has no config this ' - 'engine can read'), - ('ValidationError', 'the engine rejected its own configuration'), - ('401 Client Error', 'the model is gated: set HF_TOKEN with ' - '`infer-stack env HF_TOKEN=...`'), - ('403 Client Error', 'the model is gated: set HF_TOKEN with ' - '`infer-stack env HF_TOKEN=...`'), +# Crash diagnosis is backend-neutral (the kubeai backend uses it too); these +# names stay importable from here. +from .diagnosis import ( # noqa: E402,F401 + CRASH_LOOP_RESTARTS, + _ENGINE_ERROR_HINTS, + _TRANSIENT_ENGINE_SIGNATURES, + _engine_error_summary, + classify_engine_log, + diagnose_startup, ) -#: Failures a RESTART CAN FIX: the hub was unreachable, a download was cut off, -#: a port was still held by the container we just replaced. `restart: -#: unless-stopped` exists for exactly these, so a restart count alone must not -#: condemn an engine -- the whole point of the policy is that the next attempt -#: succeeds. Weigh a new entry by one question: would running the same container -#: again plausibly work? If yes it belongs here; if no it belongs above. -_TRANSIENT_ENGINE_SIGNATURES = ( - 'Max retries exceeded', - 'Connection reset by peer', - 'Connection refused', - 'Temporary failure in name resolution', - 'Failed to resolve', - 'Read timed out', - 'ReadTimeoutError', - 'ConnectionError', - 'IncompleteRead', - 'Consistency check failed', # a truncated HF download - '429 Client Error', # hub rate limit - '500 Server Error', - '502 Server Error', - '503 Server Error', - '504 Server Error', - 'Address already in use', -) - - -def classify_engine_log(logs: str) -> str | None: - """``'fatal'``, ``'transient'``, or ``None`` when the log says neither. - - Order matters: an unrecoverable signature wins over a transient one, because - a hub timeout earlier in the same log does not make a rejected config - loadable. CUDA OOM counts as fatal — the allocation is deterministic, so the - restart repeats it — and it is the one class with a documented remedy - (:mod:`infer_stack.leasing.vram`). - """ - from .vram import looks_like_cuda_oom - - if not logs: - return None - if any(needle in logs for needle, _ in _ENGINE_ERROR_HINTS): - return 'fatal' - if looks_like_cuda_oom(logs): - return 'fatal' - if any(needle in logs for needle in _TRANSIENT_ENGINE_SIGNATURES): - return 'transient' - return None - - -def _engine_error_summary(logs: str) -> str: - """The engine's own error, quoted, with a hint when we recognise it.""" - from .vram import looks_like_cuda_oom - - if not logs.strip(): - return '; no engine log available (`infer-stack logs` for more)' - hint = next((note for needle, note in _ENGINE_ERROR_HINTS if needle in logs), None) - if hint is None and looks_like_cuda_oom(logs): - hint = 'the GPU ran out of memory for this configuration' - lines = [line.strip() for line in logs.splitlines() if line.strip()] - quoted = ' | '.join(lines[-3:])[:400] - summary = f'; last log: {quoted}' - return f'{summary}; likely cause: {hint}' if hint else summary - def _network_name() -> str: from .network import NETWORK_NAME @@ -2493,39 +2417,10 @@ def startup_failure(self, deployment: Deployment) -> str | None: residency = self.residency() except ResidencyUnknown: return None # cannot read Docker: say nothing - containers = residency.containers(deployment.id) - if len(containers) != 1: - return None # absent (not created yet) or ambiguous - container = containers[0] - exited_for_good = ( - container.state in {'exited', 'dead'} - and (container.exit_code or 0) != 0 + return diagnose_startup( + residency.containers(deployment.id), + lambda: self.deployment_logs(deployment, tail=200), ) - crashed = exited_for_good or container.restart_count >= 1 - if not crashed: - return None # created, starting, or healthy - # The log decides, not the restart count alone. `restart: - # unless-stopped` is there so a hub timeout or a port still held by the - # container we replaced resolves itself; condemning those would make the - # policy pointless. An unrecoverable error, by contrast, repeats - # identically, so waiting for a second restart only wastes the GPU. - logs = self.deployment_logs(deployment, tail=200) - verdict = classify_engine_log(logs) - if verdict == 'transient' and container.will_be_restarted: - return None # a retry is coming, and may work - if verdict == 'transient': - # Transient, but nothing will run it again: waiting is as pointless - # as for an unrecoverable error, and the cause still belongs in the - # message. - return (f'engine is not starting (exited with code ' - f'{container.exit_code} and will not be restarted)' - f'{_engine_error_summary(logs)}') - looping = container.restart_count >= CRASH_LOOP_RESTARTS - if verdict != 'fatal' and not (exited_for_good or looping): - return None # unrecognised: keep today's budget - why = (f'restarted {container.restart_count} time(s)' if container.restart_count - else f'exited with code {container.exit_code}') - return f'engine is not starting ({why}){_engine_error_summary(logs)}' def deployment_logs(self, deployment: Deployment, *, tail: int = 400) -> str: """Recent engine logs for a deployment's compose service. diff --git a/infer_stack/leasing/diagnosis.py b/infer_stack/leasing/diagnosis.py new file mode 100644 index 00000000..7a7b8ac7 --- /dev/null +++ b/infer_stack/leasing/diagnosis.py @@ -0,0 +1,150 @@ +"""Why an engine is not starting: shared by every backend. + +A backend supplies the instances of one deployment (containers, or pods) as +:class:`~infer_stack.leasing.residency.Container` records, and a way to read +their recent logs. :func:`diagnose_startup` decides whether waiting could +still help, so the fail-fast wait, the "likely cause" text and the TUI's +error path behave the same on every backend. +""" + +from __future__ import annotations + +from typing import Callable, Sequence + +#: Restarts after which the runtime's own bookkeeping says an engine is +#: looping, not loading. Two is already conclusive: a model that loads does not exit. +CRASH_LOOP_RESTARTS = 2 + +#: Engine log lines worth quoting verbatim, and what they mean. Each is +#: UNRECOVERABLE: the same container, restarted, fails the same way, so one +#: crash is already conclusive and there is nothing to wait for. +_ENGINE_ERROR_HINTS = ( + ('trust_remote_code', 'the model needs trust_remote_code=True ' + '(set `runtime.trust_remote_code: true` on the endpoint)'), + ('does not recognize this architecture', 'this engine cannot read the ' + "model's architecture (it may need trust_remote_code=True)"), + ('are not supported for now', 'this vLLM build does not implement the ' + "model's architecture"), + ('is not supported', 'this vLLM build does not implement the ' + "model's architecture"), + ('No supported config format', 'the model repository has no config this ' + 'engine can read'), + ('ValidationError', 'the engine rejected its own configuration'), + ('error: unrecognized arguments', 'the engine rejected a command-line flag ' + '(check `runtime.extra_args` against this vLLM version)'), + ('401 Client Error', 'the model is gated: set HF_TOKEN with ' + '`infer-stack env HF_TOKEN=...`'), + ('403 Client Error', 'the model is gated: set HF_TOKEN with ' + '`infer-stack env HF_TOKEN=...`'), +) + +#: Failures a RESTART CAN FIX: the hub was unreachable, a download was cut off, +#: a port was still held by the container we just replaced. `restart: +#: unless-stopped` exists for exactly these, so a restart count alone must not +#: condemn an engine -- the whole point of the policy is that the next attempt +#: succeeds. Weigh a new entry by one question: would running the same container +#: again plausibly work? If yes it belongs here; if no it belongs above. +_TRANSIENT_ENGINE_SIGNATURES = ( + 'Max retries exceeded', + 'Connection reset by peer', + 'Connection refused', + 'Temporary failure in name resolution', + 'Failed to resolve', + 'Read timed out', + 'ReadTimeoutError', + 'ConnectionError', + 'IncompleteRead', + 'Consistency check failed', # a truncated HF download + '429 Client Error', # hub rate limit + '500 Server Error', + '502 Server Error', + '503 Server Error', + '504 Server Error', + 'Address already in use', +) + + +def classify_engine_log(logs: str) -> str | None: + """``'fatal'``, ``'transient'``, or ``None`` when the log says neither. + + Order matters: an unrecoverable signature wins over a transient one, because + a hub timeout earlier in the same log does not make a rejected config + loadable. CUDA OOM counts as fatal — the allocation is deterministic, so the + restart repeats it — and it is the one class with a documented remedy + (:mod:`infer_stack.leasing.vram`). + """ + from .vram import looks_like_cuda_oom + + if not logs: + return None + if any(needle in logs for needle, _ in _ENGINE_ERROR_HINTS): + return 'fatal' + if looks_like_cuda_oom(logs): + return 'fatal' + if any(needle in logs for needle in _TRANSIENT_ENGINE_SIGNATURES): + return 'transient' + return None + + +def _engine_error_summary(logs: str) -> str: + """The engine's own error, quoted, with a hint when we recognise it.""" + from .vram import looks_like_cuda_oom + + if not logs.strip(): + return '; no engine log available (`infer-stack logs` for more)' + hint = next((note for needle, note in _ENGINE_ERROR_HINTS if needle in logs), None) + if hint is None and looks_like_cuda_oom(logs): + hint = 'the GPU ran out of memory for this configuration' + lines = [line.strip() for line in logs.splitlines() if line.strip()] + quoted = ' | '.join(lines[-3:])[:400] + summary = f'; last log: {quoted}' + return f'{summary}; likely cause: {hint}' if hint else summary + + + + +def diagnose_startup(instances: Sequence, read_logs: Callable[[], str]) -> str | None: + """Diagnose an engine that cannot start, or ``None`` if it may still load. + + A restart policy makes an engine that exits immediately restart forever, + so a model that can never start looks exactly like one that is still + loading, and an acquire waits out its whole timeout. The runtime's + bookkeeping says an instance crashed; its LOG says whether another attempt + could ever work (:func:`classify_engine_log`): + + * an unrecoverable error -- a rejected config or flag, an architecture + this build does not implement, a gated repo, a CUDA OOM -- is fatal on + the FIRST crash, because the restart reproduces it exactly; + * a transient one -- an unreachable hub, a truncated download, a port + still held -- is never fatal while something will retry it; + * an unrecognised crash keeps the blunt budget of + :data:`CRASH_LOOP_RESTARTS` restarts. + + ``instances`` must be exactly one (absent or ambiguous: ``None``). The + returned string carries the engine's last words, because the cause is in + its log and nowhere else. + """ + if len(instances) != 1: + return None # absent (not created yet) or ambiguous + instance = instances[0] + exited_for_good = ( + instance.state in {'exited', 'dead'} + and (instance.exit_code or 0) != 0 + ) + crashed = exited_for_good or instance.restart_count >= 1 + if not crashed: + return None # created, starting, or healthy + logs = read_logs() + verdict = classify_engine_log(logs) + if verdict == 'transient' and instance.will_be_restarted: + return None # a retry is coming, and may work + if verdict == 'transient': + return (f'engine is not starting (exited with code ' + f'{instance.exit_code} and will not be restarted)' + f'{_engine_error_summary(logs)}') + looping = instance.restart_count >= CRASH_LOOP_RESTARTS + if verdict != 'fatal' and not (exited_for_good or looping): + return None # unrecognised: keep the restart budget + why = (f'restarted {instance.restart_count} time(s)' if instance.restart_count + else f'exited with code {instance.exit_code}') + return f'engine is not starting ({why}){_engine_error_summary(logs)}' diff --git a/infer_stack/leasing/residency.py b/infer_stack/leasing/residency.py index e768d8a3..1dadb691 100644 --- a/infer_stack/leasing/residency.py +++ b/infer_stack/leasing/residency.py @@ -106,6 +106,10 @@ class Container: #: will try to start the container again. restart_policy: str = '' restart_max: int = 0 + #: Why the instance is not running, when its runtime says: + #: ``CrashLoopBackOff``, ``ImagePullBackOff``, ``OOMKilled`` (Kubernetes). + #: Empty when there is nothing to say, or the runtime does not say. + reason: str = '' @property def warm(self) -> bool: @@ -272,3 +276,96 @@ def residency_from_inspect(raw: str, *, project: str) -> Residency: }, tuple(sorted(others, key=lambda c: c.container_id)), ) + + +#: Label a managed Kubernetes pod carries (copied by KubeAI from its Model). +POD_DEPLOYMENT_LABEL = 'infer-stack/deployment' +POD_MANAGED_LABEL = 'infer-stack/managed' + + +def residency_from_pods(raw: str) -> Residency: + """Build a :class:`Residency` from ``kubectl get pods -o json`` output. + + One pod is one instance (a :class:`Container`), keyed by the + ``infer-stack/deployment`` label KubeAI copies from the Model. The state + is mapped to the Docker vocabulary the rest of infer-stack reads: + + * running container -> ``running`` + * ``CrashLoopBackOff`` -> ``restarting`` (still warm: the kubelet retries) + * any other waiting reason, or a pod not started yet -> ``created`` + * terminated -> ``exited``; a pod being deleted -> ``removing`` + + The waiting or last-termination reason is kept in :attr:`Container.reason`. + GPUs are left empty: the cluster, not this host, owns placement. Raises + :class:`ResidencyUnknown` on output that is not a pod list. + + Example: + >>> raw = json.dumps({'items': [{ + ... 'metadata': {'name': 'model-q-1', 'labels': { + ... 'infer-stack/deployment': 'grp-a', 'infer-stack/managed': 'true', + ... 'model': 'q'}}, + ... 'status': {'phase': 'Running', 'containerStatuses': [{ + ... 'state': {'waiting': {'reason': 'CrashLoopBackOff'}}, + ... 'lastState': {'terminated': {'exitCode': 1, 'reason': 'Error'}}, + ... 'restartCount': 3}]}}]}) + >>> c = residency_from_pods(raw).containers('grp-a')[0] + >>> (c.state, c.reason, c.restart_count, c.exit_code, c.warm) + ('restarting', 'CrashLoopBackOff', 3, 1, True) + """ + try: + data = json.loads(raw or '{}') + except json.JSONDecodeError as ex: + raise ResidencyUnknown(f'kubectl get pods output is not JSON: {ex}') from ex + items = data.get('items') if isinstance(data, dict) else None + if not isinstance(items, list): + raise ResidencyUnknown('kubectl get pods output has no items list') + grouped: dict[str, list[Container]] = {} + others: list[Container] = [] + for pod in items: + meta = pod.get('metadata') or {} + labels = meta.get('labels') or {} + status = pod.get('status') or {} + statuses = status.get('containerStatuses') or [] + first = statuses[0] if statuses else {} + current = first.get('state') or {} + last = (first.get('lastState') or {}).get('terminated') or {} + waiting = (current.get('waiting') or {}).get('reason') or '' + if meta.get('deletionTimestamp'): + state = 'removing' + elif 'running' in current: + state = 'running' + elif waiting == 'CrashLoopBackOff': + state = 'restarting' + elif 'terminated' in current: + state = 'exited' + else: + state = 'created' + ended = current.get('terminated') or last + if not waiting and not statuses: + # Not started at all: the pod's own condition says why, e.g. the + # scheduler found no node with the resources ("Unschedulable"). + waiting = next((str(c.get('reason') or '') for c in status.get('conditions') or [] + if c.get('status') == 'False' and c.get('reason')), '') + ready = any(c.get('type') == 'Ready' and c.get('status') == 'True' + for c in status.get('conditions') or []) + container = Container( + container_id=str(meta.get('name') or ''), + deployment_id=str(labels.get(POD_DEPLOYMENT_LABEL) or ''), + state=state, + service=str(labels.get('model') or ''), + labelled=labels.get(POD_MANAGED_LABEL) == 'true', + health='healthy' if ready else ('starting' if state == 'running' else ''), + restart_count=int(first.get('restartCount') or 0), + exit_code=(None if ended.get('exitCode') is None else int(ended['exitCode'])), + # A Deployment's pods are always restarted by the kubelet. + restart_policy='always', + reason=waiting or str(ended.get('reason') or ''), + ) + if container.deployment_id: + grouped.setdefault(container.deployment_id, []).append(container) + else: + others.append(container) + return Residency( + by_deployment={gid: tuple(found) for gid, found in grouped.items()}, + others=tuple(others), + ) diff --git a/tests/test_leasing_kubeai.py b/tests/test_leasing_kubeai.py index f5414eb4..03b32cd7 100644 --- a/tests/test_leasing_kubeai.py +++ b/tests/test_leasing_kubeai.py @@ -634,3 +634,111 @@ def fingerprint(): ctl.rotate_gateway_key() assert be.master_key() != old assert fingerprint() != before + + +# -- strict residency from pods, and crash diagnosis (plan steps K2, K4) ------ +# +# Pod shapes follow real `kubectl get pods -o json` output captured on k3s + +# KubeAI 0.23.4: KubeAI copies the Model's labels onto its pods, and a pod +# whose engine rejected a flag reads Running, restartCount 1, lastState +# terminated {exitCode: 2, reason: Error}. + + +def _pod(name, gid, *, state=None, restarts=0, last=None, ready=False, + conditions=None, statuses=True): + status = {'phase': 'Running', + 'conditions': conditions or [{'type': 'Ready', + 'status': 'True' if ready else 'False'}]} + if statuses: + status['containerStatuses'] = [{ + 'name': 'server', 'restartCount': restarts, + 'state': state or {'running': {'startedAt': 't'}}, + 'lastState': {'terminated': last} if last else {}, + }] + return {'metadata': {'name': name, 'labels': { + 'infer-stack/deployment': gid, 'infer-stack/managed': 'true', + 'model': name.split('-')[1], 'app': 'model'}}, + 'status': status} + + +class PodKubectl(FakeKubectl): + """FakeKubectl plus pods and per-pod logs (current, and --previous).""" + + def __init__(self): + super().__init__() + self.pods: list[dict] = [] + self.logs: dict[tuple[str, bool], str] = {} + self.fail_pods = False + + def __call__(self, args): + if len(args) > 4 and args[3] == 'get' and args[4] == 'pods': + self.calls.append(args) + if self.fail_pods: + raise RuntimeError('connection refused') + return json.dumps({'items': self.pods}) + if len(args) > 3 and args[3] == 'logs': + self.calls.append(args) + return self.logs.get((args[4], '--previous' in args), '') + return super().__call__(args) + + +def make_pod_backend(tmp_path): + kubectl = PodKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + return be, kubectl + + +def test_residency_reads_pods_and_never_guesses(tmp_path): + from infer_stack.leasing.residency import ResidencyUnknown + + be, kubectl = make_pod_backend(tmp_path) + kubectl.pods = [_pod('model-qwen-1', 'grp-a', ready=True)] + pod = be.residency().resident('grp-a') + assert (pod.state, pod.health, pod.labelled) == ('running', 'healthy', True) + kubectl.fail_pods = True + with pytest.raises(ResidencyUnknown): + be.residency() # a failed look is never "nothing running" + + +def test_an_unschedulable_pod_says_why(tmp_path): + be, kubectl = make_pod_backend(tmp_path) + kubectl.pods = [_pod('model-big-1', 'grp-b', statuses=False, conditions=[ + {'type': 'PodScheduled', 'status': 'False', 'reason': 'Unschedulable'}])] + (pod,) = be.residency().containers('grp-b') + assert (pod.state, pod.reason, pod.warm) == ('created', 'Unschedulable', False) + + +def test_a_rejected_flag_fails_the_wait_with_the_engines_words(tmp_path): + be, kubectl = make_pod_backend(tmp_path) + dep = vllm('grp-x', served='broken') + be.converge([dep]) + kubectl.pods = [_pod('model-broken-1', 'grp-x', restarts=1, + last={'exitCode': 2, 'reason': 'Error'})] + kubectl.logs[('model-broken-1', True)] = ( + 'usage: ...\napi_server.py: error: unrecognized arguments: --bogus\n') + be.http.post = lambda url, **kw: FakeHttp._Resp(503, {'detail': 'not ready'}) + probe = be.probe_ready(dep, 'grp-x') + assert probe.fatal and not probe.ready + assert 'unrecognized arguments: --bogus' in probe.detail # the previous run's log + assert 'rejected a command-line flag' in probe.detail + + +def test_a_slow_start_is_not_a_failure(tmp_path): + be, kubectl = make_pod_backend(tmp_path) + dep = vllm('grp-y', served='slow') + be.converge([dep]) + kubectl.pods = [_pod('model-slow-1', 'grp-y')] # running, not ready + be.http.post = lambda url, **kw: FakeHttp._Resp(503, {'detail': 'not ready'}) + probe = be.probe_ready(dep, 'grp-y') + assert not probe.ready and not probe.fatal + + +def test_compose_only_commands_refuse_kubeai_explicitly(tmp_path, monkeypatch): + """`gc --orphans` used "has residency" to mean compose; kubeai has it now.""" + from infer_stack.cli import commands_leasing + + be, _ = make_pod_backend(tmp_path) + ctl = Controller(Ledger(SqliteStore(str(tmp_path / 'l.db'))), be) + monkeypatch.setattr(commands_leasing, '_open_controller', lambda *a, **k: ctl) + with pytest.raises(SystemExit, match='needs the compose backend'): + commands_leasing.GcCLI.main(argv=['--orphans', '--yes']) diff --git a/tests/test_leasing_startup_failure.py b/tests/test_leasing_startup_failure.py index 88fbc6b8..9f0068e4 100644 --- a/tests/test_leasing_startup_failure.py +++ b/tests/test_leasing_startup_failure.py @@ -150,6 +150,8 @@ def test_an_unrecoverable_error_is_fatal_on_the_very_first_crash(tmp_path): (CRASH_LOG, 'fatal'), ('ValueError: quantization fp8 is not supported on this device', 'fatal'), ('torch.OutOfMemoryError: CUDA out of memory', 'fatal'), + # vLLM rejecting a flag (e.g. from runtime.extra_args) repeats every restart. + ('api_server.py: error: unrecognized arguments: --bogus', 'fatal'), (HUB_TIMEOUT_LOG, 'transient'), ('OSError: [Errno 98] Address already in use', 'transient'), ('INFO starting\nSegmentation fault', None), From 791f4c8ea73c0af960ef1c7e2d411ad59c754c10 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 12:53:59 -0400 Subject: [PATCH 05/34] K5: one served-name rule, shared GPU count, env on kubeai leasing.models.served_name replaces five copies of the fallback chain. Two LiteLLM route builders had used `or deployment.id` where the engine used the first served alias, so without served_model_name the route named a model the engine did not serve. KubeAI's resource-profile units come from placement.required_gpu_count, and runtime.env reaches the Model's spec.env (templates filled, reserved names enforced at catalog load); command and mounts stay compose-only. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 7 ++++ README.md | 4 ++- .../plan-backend-unification-2026-09-24.md | 5 +++ infer_stack/backends/kubeai.py | 35 +++++++++++-------- infer_stack/leasing/compose.py | 18 ++++------ infer_stack/leasing/models.py | 22 ++++++++++++ tests/test_leasing_kubeai.py | 13 ++++++- 7 files changed, 75 insertions(+), 29 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5781eaed..218fa5f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,13 @@ as `Unschedulable` or `ImagePullBackOff`. vLLM rejecting a flag (`error: unrecognized arguments`) is now recognised as fatal on both backends; before, it waited for two restarts. +`runtime.env` now works on the kubeai backend (the Model's `spec.env`, with +the same templates and reserved names). The served-name rule lives in one +place, `leasing.models.served_name`. Two gateway route builders fell back to +the deployment id where the engine used the first served alias, so a +deployment without `served_model_name` routed to a name its engine did not +serve. + `gc --orphans` and `network migrate` now refuse a non-compose backend explicitly. They had used "the backend has `residency`" to mean compose. diff --git a/README.md b/README.md index 41a2877a..642c5ee3 100644 --- a/README.md +++ b/README.md @@ -650,7 +650,9 @@ never silently retunes an endpoint after the fact. launcher instead. - All of these are deployment identity: endpoints that launch differently never share a process. -- Compose only; KubeAI refuses an endpoint with `command`, `env` or `mounts`. +- `env` works on both backends (on KubeAI it becomes the Model's `spec.env`). + `command` and `mounts` are Compose only; KubeAI refuses an endpoint with + either rather than serve stock vLLM in its place. ### Reasoning / thinking models diff --git a/dev/tmp/plan-backend-unification-2026-09-24.md b/dev/tmp/plan-backend-unification-2026-09-24.md index 350eb231..4cdac436 100644 --- a/dev/tmp/plan-backend-unification-2026-09-24.md +++ b/dev/tmp/plan-backend-unification-2026-09-24.md @@ -183,6 +183,11 @@ engine's error quoted, instead of waiting out the 800 s timeout. Adding - TUI engine logs through a backend `deployment_logs` stream, with `kubectl logs -f` on KubeAI +**Done 2026-09-24** except TUI logs: there is one `served_name` (it fixed a +real disagreement, noted in the changelog), KubeAI's GPU units come from +`placement.required_gpu_count`, and KubeAI takes `runtime.env`. The unused +profile-era KubeAI renderer is deleted. TUI engine logs on KubeAI remain open. + ## Order and size K0, then K1, is the minimum for a card to run unchanged on a cluster. K2 and diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index aa0e0a0e..fdf03484 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -64,17 +64,16 @@ def model_name_for(served: str) -> str: def _served_name(deployment: Deployment) -> str: - return deployment.spec.get('served_model_name') or ( - sorted(deployment.served)[0] if deployment.served else deployment.id - ) + from ..leasing.models import served_name + + return served_name(deployment) def _gpu_count(deployment: Deployment) -> int: - runtime = deployment.spec.get('runtime', {}) or {} - tp = int(runtime.get('tensor_parallel_size', 1) or 1) - pp = int(runtime.get('pipeline_parallel_size', 1) or 1) - dp = int(runtime.get('data_parallel_size', 1) or 1) - return max(1, tp * pp * dp) + """Resource-profile units for a Model: the planner's GPU count, at least 1.""" + from ..leasing.placement import required_gpu_count + + return max(1, required_gpu_count(deployment)) def _model_doc( @@ -125,10 +124,16 @@ def _model_doc( }, 'spec': spec, } - # Attention backend is a vLLM env var, not a CLI arg (see compose._vllm_service); - # forward it through the KubeAI Model's env map for parity across backends. + # Container environment through the Model's env map, as compose renders + # it: runtime.env (templates filled, values as strings), plus the attention + # backend, which is a vLLM env var rather than a flag. + from ..leasing.launch import env_string, fill + + env = {str(k): fill(env_string(v), svc) for k, v in svc['env'].items()} if svc.get('attention_backend'): - spec['env'] = {'VLLM_ATTENTION_BACKEND': str(svc['attention_backend'])} + env['VLLM_ATTENTION_BACKEND'] = str(svc['attention_backend']) + if env: + spec['env'] = env return doc @@ -178,11 +183,11 @@ def render_models( from ..leasing.launch import translate_legacy runtime = translate_legacy(deployment.spec.get('runtime', {}) or {}) - custom = [k for k in ('command', 'env', 'mounts') if runtime.get(k)] + custom = [k for k in ('command', 'mounts') if runtime.get(k)] if custom: - # A KubeAI Model runs stock vLLM; it has no place for a container - # command, environment or host mounts. Fail closed, never silently - # serve the stock engine instead. + # A KubeAI Model runs stock vLLM: it has no place for a container + # command or host mounts (env maps onto spec.env). Fail closed, + # never silently serve the stock engine instead. out.unrenderable.add(deployment.id) out.errors.append( f"{deployment.id}: runtime.{custom[0]} describes a custom container " diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index 27306411..a9c1cb71 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -54,7 +54,7 @@ from ..profile_runtime import simulator_args, vllm_args from .backend import ConvergeScaffold, Readiness from .launch import env_string, fill, translate_legacy -from .models import Deployment, is_reservation +from .models import Deployment, is_reservation, served_name from .placement import plan_placement from .residency import ( # labels live beside the code that reads them back FINGERPRINT_LABEL, @@ -180,9 +180,7 @@ def vllm_service_name(deployment: Deployment, *, unique: bool = False) -> str: :meth:`ComposeBackend.observe` still maps service names through the render sidecar; it is for reporting, not for decisions that touch a GPU.) """ - served = deployment.spec.get('served_model_name') or ( - sorted(deployment.served)[0] if deployment.served else deployment.id - ) + served = served_name(deployment) if unique: return _unique_vllm_service_name(served, deployment.id) return vllm_service_name_for(served) @@ -429,9 +427,7 @@ def vllm_service_dict(deployment: Deployment) -> dict[str, Any]: # A deployment recorded before the generic launch fields may still carry # `serve_recipe`; read it as the fields it meant. runtime = translate_legacy(deployment.spec.get('runtime', {}) or {}) - served = deployment.spec.get('served_model_name') or ( - sorted(deployment.served)[0] if deployment.served else deployment.id - ) + served = served_name(deployment) return { 'served_model_name': served, 'tensor_parallel_size': int(runtime.get('tensor_parallel_size', 1) or 1), @@ -699,7 +695,7 @@ def _litellm_model_list( if deployment.id not in assignments: continue if deployment.engine == 'vllm': - served = deployment.spec.get('served_model_name') or deployment.id + served = served_name(deployment) api_base = f'http://{vllm_service_name(deployment)}:8000/v1' for endpoint in sorted(deployment.served): entries.append(_vllm_route_entry(endpoint, served, api_base)) @@ -810,9 +806,7 @@ def _registry_incoming_from_deployments( if deployment.id not in assignments: continue if deployment.engine == 'vllm': - served = deployment.spec.get('served_model_name') or ( - sorted(deployment.served)[0] if deployment.served else deployment.id - ) + served = served_name(deployment) for endpoint in sorted(deployment.served): incoming[endpoint] = {'engine': 'vllm', 'served': served} elif deployment.engine == 'ollama': @@ -970,7 +964,7 @@ def _litellm_routes( if deployment.id not in assignments: continue if deployment.engine == 'vllm': - served = deployment.spec.get('served_model_name') or deployment.id + served = served_name(deployment) api_base = ( f'http://{vllm_service_name(deployment, unique=True)}' f':{VLLM_CONTAINER_PORT}/v1' diff --git a/infer_stack/leasing/models.py b/infer_stack/leasing/models.py index c6fc922d..3677c5fd 100644 --- a/infer_stack/leasing/models.py +++ b/infer_stack/leasing/models.py @@ -73,6 +73,28 @@ class Sharing: # Synthetic endpoint/claim name for a reservation (it serves nothing). RESERVED_ENDPOINT = 'reserved-gpu' +def served_name(deployment: Any) -> str: + """The model name a deployment's engine serves under: the one rule. + + The spec's ``served_model_name``, else its first served alias, else its + id. The engine's ``--served-model-name``, the gateway route's upstream + model, the Compose service name and the KubeAI Model name all derive + from this, so they cannot disagree. + + Example: + >>> from types import SimpleNamespace as NS + >>> served_name(NS(spec={'served_model_name': 'q'}, served={'a': {}}, id='g')) + 'q' + >>> served_name(NS(spec={}, served={'b': {}, 'a': {}}, id='g')) + 'a' + >>> served_name(NS(spec={}, served={}, id='g')) + 'g' + """ + return deployment.spec.get('served_model_name') or ( + sorted(deployment.served)[0] if deployment.served else deployment.id + ) + + def is_reservation(obj: Any) -> bool: """True if a :class:`Deployment` / :class:`EndpointRequest` is a GPU reservation.""" return getattr(obj, 'engine', None) == RESERVED_ENGINE diff --git a/tests/test_leasing_kubeai.py b/tests/test_leasing_kubeai.py index 03b32cd7..de85cbc9 100644 --- a/tests/test_leasing_kubeai.py +++ b/tests/test_leasing_kubeai.py @@ -185,7 +185,7 @@ def test_render_attention_backend_reaches_cr_env(): @pytest.mark.parametrize('launch', [ {'serve_recipe': 'hyperqwen-3090-single'}, # a legacy entry, translated {'command': ['single']}, - {'env': {'SPEC': 'mtp'}}, + {'mounts': {'/cache': 'x/cache'}}, ]) def test_render_refuses_a_custom_container_launch(launch): rendered = render_models( @@ -742,3 +742,14 @@ def test_compose_only_commands_refuse_kubeai_explicitly(tmp_path, monkeypatch): monkeypatch.setattr(commands_leasing, '_open_controller', lambda *a, **k: ctl) with pytest.raises(SystemExit, match='needs the compose backend'): commands_leasing.GcCLI.main(argv=['--orphans', '--yes']) + + +def test_runtime_env_reaches_the_model_like_it_reaches_a_container(): + rendered = render_models( + [vllm('grp-e', served='q', env={'MODE': 'fast', 'CTX': '{max_model_len}', 'ON': True}, + attention_backend='TORCH_SDPA')], + namespace='kubeai', default_resource_profile='cpu', + ) + (doc,) = rendered.docs + assert doc['spec']['env'] == {'MODE': 'fast', 'CTX': '4096', 'ON': 'true', + 'VLLM_ATTENTION_BACKEND': 'TORCH_SDPA'} From bbbe61067ba4a00d906217b90388fd49ac377ad3 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 12:54:30 -0400 Subject: [PATCH 06/34] Journal: backend unification K0-K2, K4, K5 Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- dev/journals/claude.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/dev/journals/claude.md b/dev/journals/claude.md index caadd793..9bb84781 100644 --- a/dev/journals/claude.md +++ b/dev/journals/claude.md @@ -3180,3 +3180,40 @@ labels are stable enough to key residency on. **Takeaway.** Before unifying internals, check what callers see. Two backends that share 90% of their code but answer to different names are two products to the people using them. + +## 2026-09-24 12:54:30 -0400 + +**Intent.** The user approved the backend-unification plan, asked for the +work on a branch (`dev/backend-unification`), and authorised installing k3s +on the guest to test KubeAI for real. Model: Claude Opus 5.5 (1M context). + +**What happened.** K0: k3s plus KubeAI 0.23.4, using the chart's `cpu` +profile. It runs real vLLM on CPU, which beats the simulator: KubeAI builds +vLLM's own command line, and the simulator's CLI would have refused it. The +backend passed unchanged on its first real run since July. K1: the kubeai +backend now owns a gateway-only ComposeBackend and feeds it generic +`upstream` route rows. Before, a card's alias got 404 from KubeAI; after, +the same request answered. K2 and K4: strict pod residency, and the crash +diagnosis moved into a backend-neutral module. A Model whose vLLM rejected a +flag failed in 52 s instead of the 800 s timeout. K5: one served-name rule. +It fixed a real disagreement between the engine and two route builders. +Removed the dead profile-era KubeAI renderer. + +**What I got wrong along the way.** I wrote a second e2e script before +finding `dev/kubeai_e2e.sh`, a duplicate authority created in the middle of +a de-duplication task, and folded it back in. That script's generation check +could never fail (a `curl | grep && echo` list, where `set -e` does not +fire), so it "passed" on a 404. The plan's claim that KubeAI's lenient +`observe()` was a bug was wrong: Compose's is lenient by the same contract. +I corrected the plan. + +**Open: K3 (one acquire path).** It is feasible: only 12 controller tests and +the null backend depend on the old path. It carries a semantic decision the +user should make. On a cluster nothing displaces an idle keep-warm Model, so +the September starvation incident can recur on KubeAI unless admission +evicts idle keep-warm Models when a new one cannot be scheduled. + +**Takeaways.** (1) A test that has never failed is not evidence. Make it +fail on purpose first; GATEWAY=0 is that switch here. (2) "Has method X" is +a poor stand-in for "is backend Y": when a second backend gains X, every +such check silently changes meaning. From 2c513760add95ab4d594397bd1e87ead092a2de3 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 14:10:22 -0400 Subject: [PATCH 07/34] G-A/B: move naming rules and the gateway's module code out of compose.py compose.py mixed engine containers with the front door. Two new modules: - leasing/naming.py: dns_slug, the vLLM/Ollama service-name rules and the engine container ports. Engines, gateway routes and the kubeai backend all derive names and upstream addresses from these. - leasing/gateway.py: route rows and the LiteLLM model_list renderers, the route registry functions, the LiteLLM/Postgres/Open WebUI/nginx service renderers, the dynamic-routing constants, and set_master_key. Moved unchanged. ENGINE_LABEL joins the other labels in residency.py. compose keeps importing what it uses (so `from infer_stack.leasing.compose import dns_slug` still works); importers of gateway names now import them from gateway. No behaviour change: suite 898 passed, ty and flake8 clean. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- .../plan-backend-unification-2026-09-24.md | 28 + infer_stack/backends/kubeai.py | 2 +- infer_stack/leasing/compose.py | 826 +----------------- infer_stack/leasing/gateway.py | 732 ++++++++++++++++ infer_stack/leasing/naming.py | 99 +++ infer_stack/leasing/residency.py | 2 + tests/test_leasing_compose.py | 2 +- tests/test_leasing_dynamic_routing.py | 6 +- tests/test_leasing_route_registry.py | 2 +- tests/test_leasing_secrets.py | 7 +- 10 files changed, 917 insertions(+), 789 deletions(-) create mode 100644 infer_stack/leasing/gateway.py create mode 100644 infer_stack/leasing/naming.py diff --git a/dev/tmp/plan-backend-unification-2026-09-24.md b/dev/tmp/plan-backend-unification-2026-09-24.md index 4cdac436..ac31001b 100644 --- a/dev/tmp/plan-backend-unification-2026-09-24.md +++ b/dev/tmp/plan-backend-unification-2026-09-24.md @@ -202,3 +202,31 @@ to the kind script. - multi-cluster, or scheduling across a mix of Compose hosts and a cluster - KubeAI autoscaling policy beyond today's `min_replicas` / `max_replicas` - custom container launches on KubeAI + +## G. Pull the gateway out of compose.py (added 2026-09-24) + +`compose.py` mixes two jobs: engine containers (placement, render, +selective apply, pulls, diagnosis) and the front door (LiteLLM config and +service, route registry, dynamic-route reconciliation, keys, Open WebUI, +nginx). K1 reused the whole `ComposeBackend` as a gateway-only instance for +KubeAI. That added a mode rather than naming the seam. The user asked for +the tool to be more elegant, architecture first and the TUI after. + +The gateway methods depend only on the state dir, ports, the HTTP client, +the clock, the catalog and the flags (measured by AST), not on engine state. +So the split is: + +- **A. `leasing/naming.py`**: `dns_slug` and the vLLM/Ollama service-name + rules. Engines and gateway routes both use them, and so does KubeAI. +- **B. `leasing/gateway.py`**: the route functions, front-door service + renderers and gateway constants, moved as-is. +- **C. `Gateway`**: keys, route registry and dynamic-route reconciliation. + `ComposeBackend` owns one and keeps thin public delegations (`master_key`, + `access`, ...). +- **D. `render_front_door`**: `render_compose`'s second half. + +`compose` re-exports nothing private; importers (mostly tests) move to the +new modules. Each step leaves the full suite, `ty` and flake8 green. The +gateway still runs as a Compose project (it is a container), so KubeAI keeps +a `ComposeBackend` with no engines: that is now simply "a Compose project +with only the front door", not a special mode. diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index fdf03484..f57e2559 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -378,7 +378,7 @@ def _upstream_url(self) -> str: def _render_gateway(self, rendered: RenderedModels) -> None: """Route each endpoint alias through the gateway to its Model.""" - from ..leasing.compose import UPSTREAM_ROUTE + from ..leasing.gateway import UPSTREAM_ROUTE base = self._upstream_url() self.gateway.upstream_routes = { diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index ce980fc2..b012a327 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -40,7 +40,6 @@ import hashlib import json -import re import time from dataclasses import dataclass, field from pathlib import Path @@ -53,10 +52,55 @@ from ..probe import openai_ready from ..profile_runtime import simulator_args, vllm_args from .backend import ConvergeScaffold, Readiness +from .gateway import ( + API_KEY_ENV, + DB_PASSWORD_ENV, + _dump_route_registry, + LITELLM_CONFIG_FILENAME, + LITELLM_CONTAINER_PORT, + _litellm_model_list, + _litellm_model_list_from_catalog, + _litellm_model_list_from_registry, + LITELLM_REGISTRY_FILENAME, + LITELLM_REGISTRY_VERSION, + _litellm_routes, + LITELLM_ROUTES_FILENAME, + _litellm_service, + LITELLM_SERVICE, + _merge_route_registry, + _nginx_conf, + NGINX_CONFIG_FILENAME, + _nginx_service, + NGINX_SERVICE, + _open_webui_service, + OPEN_WEBUI_SERVICE, + _postgres_service, + POSTGRES_SERVICE, + _registry_incoming_from_catalog, + _registry_incoming_from_deployments, + ROUTE_ID_PREFIX, + ROUTE_RECONCILE_BOOTSTRAP_S, + ROUTE_RECONCILE_STEADY_S, + SALT_KEY_ENV, + _seed_registry_from_litellm_config, + set_master_key, +) from .launch import env_string, fill, translate_legacy from .models import Deployment, is_reservation, served_name +from .naming import ( # noqa: F401 (public names re-exported for callers) + OLLAMA_CONTAINER_PORT, + VLLM_CONTAINER_PORT, + _dns_slug, + _unique_vllm_service_name, + dns_slug, + ollama_service_name, + ollama_service_name_for, + vllm_service_name, + vllm_service_name_for, +) from .placement import plan_placement from .residency import ( # labels live beside the code that reads them back + ENGINE_LABEL, FINGERPRINT_LABEL, SERVICE_LABEL, COMPOSE_PROJECT_LABEL, @@ -68,46 +112,8 @@ LEASING_PROJECT = 'infer-stack' # docker compose project name for leased stacks VLLM_HOST_PORT_BASE = 18000 -VLLM_CONTAINER_PORT = 8000 -OLLAMA_CONTAINER_PORT = 11434 -LITELLM_CONTAINER_PORT = 4000 STATE_FILENAME = 'leasing-compose-state.json' COMPOSE_FILENAME = 'docker-compose.yml' -LITELLM_CONFIG_FILENAME = 'litellm_config.yaml' -LITELLM_SERVICE = 'litellm' -API_KEY_ENV = 'LITELLM_MASTER_KEY' -# The key LiteLLM encrypts credentials stored in its database with. Unset, it -# uses the master key -- so rotating the master key would make every -# DB-stored route undecryptable. `set_master_key` pins it to the pre-rotation -# master key the first time the key changes; it must never change after that. -SALT_KEY_ENV = 'LITELLM_SALT_KEY' - -# Dynamic-routing (admin-API) extras. When dynamic routing is on, the gateway's -# route table is managed live via LiteLLM's admin API against a Postgres-backed -# model store, instead of a static config file. See render_compose + -# ComposeBackend._reconcile_routes and docs/litellm-gateway-routing.md. -LITELLM_ROUTES_FILENAME = 'litellm_routes.json' # rendered desired route set -# Append-only route registry for static-superset mode: accumulates the semantic -# route inputs (served name / engine / host) of every catalog *and* every live -# deployment ever merged, across all runbooks sharing this state dir. The gateway -# `model_list` is rendered from the whole registry, so a converge under one -# runbook's catalog can no longer strip another's still-live routes, and once -# every catalog has been merged once the rendered config is byte-stable (the -# gateway is never recreated). See docs/litellm-gateway-routing.md and -# ComposeBackend._update_route_registry. -LITELLM_REGISTRY_FILENAME = 'litellm_registry.json' -LITELLM_REGISTRY_VERSION = 1 -# A route-registry row for a server this project does not run: -# ``{'engine': UPSTREAM_ROUTE, 'served': , 'api_base': }``. -UPSTREAM_ROUTE = 'upstream' -POSTGRES_SERVICE = 'postgres-litellm' -POSTGRES_CONTAINER_PORT = 5432 -POSTGRES_DB_NAME = 'litellm' -POSTGRES_DB_USER = 'litellm' -DB_PASSWORD_ENV = 'LITELLM_DB_PASSWORD' # managed secret in the sidecar .env -# Marks a LiteLLM route as infer-stack-managed, so reconcile only ever deletes -# routes it created (never a model added by hand through the UI/admin API). -ROUTE_ID_PREFIX = 'isr-' VLLM_DEFAULTS = { 'gpu_memory_utilization': 0.9, @@ -116,90 +122,8 @@ 'max_num_seqs': 256, } -ENGINE_LABEL = 'infer-stack.engine' - - -def dns_slug(text: str) -> str: - """A lowercase ``[a-z0-9-]`` label safe as a compose service / DNS / - Kubernetes object name (shared by the compose and kubeai backends).""" - out = re.sub(r'[^a-z0-9]+', '-', str(text).lower()).strip('-') - return out or 'model' - - -_dns_slug = dns_slug # historical internal name - - -def vllm_service_name_for(served: str) -> str: - """Deterministic compose/DNS service name for a vLLM upstream: ``vllm-``. - - Derived purely from the served model name (vLLM's ``--served-model-name``, - the Open WebUI label, the alias the user chose), so it is identical whether - computed from a live :class:`Deployment` or from a catalog endpoint. That - stability is what lets the LiteLLM gateway carry a *static* route table (one - per catalog endpoint) whose upstream hosts match the containers when they - come up — so adding/removing models does not rewrite the gateway's config and - the gateway is never recreated (no "blip"); see :func:`_litellm_model_list`. - """ - return f'vllm-{_dns_slug(served)}' - - -def _unique_vllm_service_name(served: str, deployment_id: str) -> str: - """Per-deployment vLLM service/DNS name: ``vllm--``. - - The static-superset gateway needs a name derivable from the served model - *alone* (so a catalog route can address it without knowing the live - deployment) — but that deliberately drops the deployment id, which - **collapses every** ``--dedicated`` **deployment of one model onto a single - container** (hence one GPU). Dynamic routing manages the gateway's routes - live via the admin API, so the upstream host no longer has to be predictable - from the catalog. That frees us to give each deployment its **own** service, - so N dedicated deployments of one model become N containers on N GPUs. The - suffix is the deployment id's hex tail, keeping the name short and DNS-safe. - """ - tail = deployment_id.rsplit('-', 1)[-1][:8] or 'x' - return f'{vllm_service_name_for(served)}-{_dns_slug(tail)}' - - -def vllm_service_name(deployment: Deployment, *, unique: bool = False) -> str: - """Compose service name for a vLLM deployment (see :func:`vllm_service_name_for`). - - Default (``unique=False``, static-superset mode): deterministic from the - served model name only — *no* deployment-id suffix — so it matches the - gateway's pre-rendered route for that endpoint. Trade-off: two - *simultaneously desired* deployments that share a served name collide on this - name; under the static gateway the catalog endpoint is the unit, so that case - (including same-model ``--dedicated``) is unsupported. - - ``unique=True`` (dynamic-routing mode): append the deployment-id tail - (:func:`_unique_vllm_service_name`) so same-model dedicated deployments get - distinct containers/GPUs; the admin-API route table addresses each by name. - - Either way the container carries the ``infer-stack.deployment`` label. - :meth:`ComposeBackend.residency` correlates containers to deployments by that - label, so the choice of suffix does not affect it. (The lenient - :meth:`ComposeBackend.observe` still maps service names through the render - sidecar; it is for reporting, not for decisions that touch a GPU.) - """ - served = served_name(deployment) - if unique: - return _unique_vllm_service_name(served, deployment.id) - return vllm_service_name_for(served) -def ollama_service_name_for(host: str) -> str: - """Deterministic service name for an Ollama daemon: ``ollama-``. - - One daemon per host (Ollama coalesces tags onto it), so the host is the - stable key — matching :func:`vllm_service_name_for`'s role for vLLM so the - gateway's static route table addresses it regardless of which tags are live. - """ - return f'ollama-{_dns_slug(host)}' - - -def ollama_service_name(deployment: Deployment) -> str: - host = deployment.spec.get('host') or deployment.id - return ollama_service_name_for(host) - class ApplyAborted(RuntimeError): """Selective apply refused to act; the change stays pending.""" @@ -654,639 +578,6 @@ def _ollama_service( return service -def _vllm_route_entry( - model_name: str, served: str, api_base: str -) -> dict[str, Any]: - """One LiteLLM ``model_list`` entry routing ``model_name`` to a vLLM upstream. - - Shared by every render path (legacy per-deployment, catalog-superset, and - the route registry) so a registry-rendered entry can never drift from what - the catalog/deployment paths produce for the same endpoint.""" - return { - 'model_name': model_name, - 'litellm_params': { - 'model': f'openai/{served}', - 'api_base': api_base, - 'api_key': 'EMPTY', - }, - } - - -def _ollama_route_entry( - model_name: str, tag: str, api_base: str -) -> dict[str, Any]: - """One LiteLLM ``model_list`` entry routing ``model_name`` to an Ollama tag - (see :func:`_vllm_route_entry` for why this is factored out).""" - return { - 'model_name': model_name, - 'litellm_params': { - 'model': f'ollama/{tag}', - 'api_base': api_base, - }, - } - - -def _litellm_model_list( - deployments: list[Deployment], assignments: dict[str, list[int]] -) -> list[dict[str, Any]]: - """One LiteLLM ``model_list`` entry per served endpoint alias.""" - entries: list[dict[str, Any]] = [] - for deployment in sorted(deployments, key=lambda g: (g.created_at, g.id)): - if deployment.id not in assignments: - continue - if deployment.engine == 'vllm': - served = served_name(deployment) - api_base = f'http://{vllm_service_name(deployment)}:8000/v1' - for endpoint in sorted(deployment.served): - entries.append(_vllm_route_entry(endpoint, served, api_base)) - elif deployment.engine == 'ollama': - api_base = f'http://{ollama_service_name(deployment)}:{OLLAMA_CONTAINER_PORT}' - for endpoint, payload in sorted(deployment.served.items()): - tag = payload.get('model', endpoint) - entries.append(_ollama_route_entry(endpoint, tag, api_base)) - return entries - - -def _litellm_model_list_from_catalog(catalog: Any) -> list[dict[str, Any]]: - """A *static superset* ``model_list``: one route per catalog endpoint. - - Unlike :func:`_litellm_model_list` (which routes only the currently-placed - deployments), this routes *every* catalog endpoint to its deterministic - upstream host (:func:`vllm_service_name_for` / :func:`ollama_service_name_for`). - The resulting config therefore depends only on the catalog, not on which - models happen to be up — so acquiring/releasing a model leaves the gateway's - config (and its container) untouched (no blip). A route whose upstream is not - currently running simply errors/cools-down until it comes up; the - ``router_settings`` below make that warmup self-healing. ``/v1/models`` lists - the whole catalog (some upstreams down) rather than only the live set. - - This static-superset path is the default. Its one limitation — it cannot give - same-model ``--dedicated`` deployments distinct upstreams, and cannot route - non-catalog acquires without a config change — is addressed by the opt-in - *dynamic routing* mode (``dynamic_routing=True``), which manages routes live - via LiteLLM's admin API against a Postgres model store (see - :func:`_litellm_routes`, :meth:`ComposeBackend._reconcile_routes`, and - ``docs/litellm-gateway-routing.md``). The two are mutually exclusive per - converge; this function is used only when dynamic routing is off. - """ - entries: list[dict[str, Any]] = [] - for name in sorted(getattr(catalog, 'endpoints', {})): - try: - req = catalog.resolve_endpoint(name) - except Exception: # noqa: BLE001 - a bad endpoint must not break the gateway - continue - if req.engine == 'vllm': - served = req.served.get('served_model_name') or name - api_base = ( - f'http://{vllm_service_name_for(served)}:{VLLM_CONTAINER_PORT}/v1' - ) - entries.append(_vllm_route_entry(name, served, api_base)) - elif req.engine == 'ollama': - host = req.spec.get('host') or req.host - tag = req.served.get('model') or name - api_base = ( - f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' - ) - entries.append(_ollama_route_entry(name, tag, api_base)) - return entries - - -# -- Route registry (static-superset persistence) -------------------------- -# -# The registry stores *semantic* route inputs (served name / engine / host), -# never rendered LiteLLM entries — render derives entries through the same -# helpers the catalog/deployment paths use (:func:`_litellm_model_list_from_registry`), -# so a future renderer change propagates to old registry rows automatically. -# All functions here are pure; the backend owns the file I/O and locking. - - -def _registry_incoming_from_catalog(catalog: Any) -> dict[str, dict[str, Any]]: - """Semantic route rows for every resolvable endpoint of ``catalog``. - - Mirrors :func:`_litellm_model_list_from_catalog`'s iteration (unresolvable - endpoints skipped) but emits registry rows keyed by endpoint name. A vLLM - row carries only ``served`` (the upstream host is re-derived at render via - :func:`vllm_service_name_for`); an Ollama row carries ``model`` (tag) + - ``host``.""" - incoming: dict[str, dict[str, Any]] = {} - for name in sorted(getattr(catalog, 'endpoints', {})): - try: - req = catalog.resolve_endpoint(name) - except Exception: # noqa: BLE001 - a bad endpoint must not break the gateway - continue - if req.engine == 'vllm': - served = req.served.get('served_model_name') or name - incoming[name] = {'engine': 'vllm', 'served': served} - elif req.engine == 'ollama': - host = req.spec.get('host') or req.host - tag = req.served.get('model') or name - incoming[name] = {'engine': 'ollama', 'model': tag, 'host': host} - return incoming - - -def _registry_incoming_from_deployments( - deployments: list[Deployment], assignments: dict[str, list[int]] -) -> dict[str, dict[str, Any]]: - """Semantic route rows for every *placed* deployment in ``assignments``. - - ``deployments`` is the full ``desired`` set (which spans all runbooks via - the shared ledger), so this keeps non-catalog / dedicated acquires routable - and — because the registry persists — routable past release. One row per key - of ``deployment.served`` (a coalesced deployment can back several endpoint - aliases). Only ``vllm``/``ollama`` engines contribute; ``RESERVED_ENGINE`` - and unknown engines render no service, so they contribute no row — exactly - as :func:`render_compose`'s service loop skips them. - - The vLLM ``served`` uses the same fallback chain as :func:`vllm_service_name` - (``spec['served_model_name'] or sorted(served)[0] or id``), so a - catalog-listed endpoint acquired live reduces to the identical row a catalog - merge produces — live-vs-released status never moves the rendered bytes.""" - incoming: dict[str, dict[str, Any]] = {} - for deployment in deployments: - if deployment.id not in assignments: - continue - if deployment.engine == 'vllm': - served = served_name(deployment) - for endpoint in sorted(deployment.served): - incoming[endpoint] = {'engine': 'vllm', 'served': served} - elif deployment.engine == 'ollama': - host = deployment.spec.get('host') or deployment.id - for endpoint, payload in sorted(deployment.served.items()): - tag = payload.get('model', endpoint) - incoming[endpoint] = { - 'engine': 'ollama', - 'model': tag, - 'host': host, - } - return incoming - - -def _litellm_model_list_from_registry( - registry: dict[str, Any] -) -> list[dict[str, Any]]: - """Render the gateway ``model_list`` from the whole accumulated registry. - - Iterates ``sorted(entries)`` (determinism, §8) and derives each upstream - ``api_base`` through the live naming helpers, so the registry never becomes - a rendered-config parse surface.""" - entries: list[dict[str, Any]] = [] - rows = registry.get('entries', {}) if isinstance(registry, dict) else {} - for name in sorted(rows): - row = rows[name] - if not isinstance(row, dict): - continue - engine = row.get('engine') - if engine == 'vllm': - served = row.get('served') or name - api_base = ( - f'http://{vllm_service_name_for(served)}:{VLLM_CONTAINER_PORT}/v1' - ) - entries.append(_vllm_route_entry(name, served, api_base)) - elif engine == 'ollama': - tag = row.get('model') or name - host = row.get('host') or name - api_base = ( - f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' - ) - entries.append(_ollama_route_entry(name, tag, api_base)) - elif engine == UPSTREAM_ROUTE and row.get('api_base'): - # An OpenAI-compatible server this project does not run (a KubeAI - # cluster's gateway): the row carries its address and the name it - # serves the model under. - entries.append(_vllm_route_entry(name, row.get('served') or name, - str(row['api_base']))) - return entries - - -def _merge_route_registry( - existing: dict[str, Any], incoming: dict[str, dict[str, Any]] -) -> tuple[dict[str, Any], list[str]]: - """Merge ``incoming`` semantic rows into ``existing`` (append-only). - - Idempotent (merging identical rows is a no-op) and additive (never removes a - row). On a conflict — same key, different row — *incoming wins* and a warning - naming both definitions is emitted; the changed definition changes the - rendered bytes, which is the one justified recreate. The existing ``version`` - is preserved (an unknown version merged under is not silently rewritten to - the current schema; see :meth:`ComposeBackend._load_route_registry`).""" - version = LITELLM_REGISTRY_VERSION - entries: dict[str, dict[str, Any]] = {} - if isinstance(existing, dict): - version = existing.get('version', LITELLM_REGISTRY_VERSION) - prior = existing.get('entries') - if isinstance(prior, dict): - entries = {k: v for k, v in prior.items()} - warnings: list[str] = [] - for name in sorted(incoming): - row = incoming[name] - if name in entries and entries[name] != row: - warnings.append( - f"route {name!r} redefined: {entries[name]} -> {row} " - '(incoming wins; gateway will be recreated once)' - ) - entries[name] = row - return {'version': version, 'entries': entries}, warnings - - -def _seed_registry_from_litellm_config( - config_text: str, -) -> tuple[dict[str, Any], list[str]]: - """One-shot upgrade seed: recover registry rows from a rendered - ``litellm_config.yaml`` so the first post-upgrade converge does not strip - the other runbooks' routes. - - Single-format, migration-time only (no cross-version promise): ``openai/`` - inverts *exactly* to ``{engine: vllm, served}``. Ollama rows are skipped with - a warning — the host survives only as a non-invertible ``dns_slug`` inside - ``api_base`` — and re-enter the registry at the next converge that has them - in its catalog or live set. Anything else unparseable is likewise skipped.""" - entries: dict[str, dict[str, Any]] = {} - warnings: list[str] = [] - try: - data = yaml.safe_load(config_text) or {} - except Exception: # noqa: BLE001 - a torn file must not brick seeding - return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}}, [ - 'seed: litellm_config.yaml is unparseable; starting an empty registry' - ] - for entry in data.get('model_list', []) or []: - name = entry.get('model_name') - model = (entry.get('litellm_params') or {}).get('model', '') - if not name: - continue - if isinstance(model, str) and model.startswith('openai/'): - entries[name] = {'engine': 'vllm', 'served': model[len('openai/'):]} - elif isinstance(model, str) and model.startswith('ollama/'): - warnings.append( - f"seed: skipping Ollama route {name!r} (host not recoverable " - 'from the rendered api_base; it re-enters at its next converge)' - ) - else: - warnings.append(f'seed: skipping unparseable route {name!r}') - return {'version': LITELLM_REGISTRY_VERSION, 'entries': entries}, warnings - - -def _dump_route_registry(registry: dict[str, Any]) -> str: - """Canonical, byte-stable serialization (§3): sorted keys + trailing - newline. A nondeterministic dump would manufacture phantom hash changes.""" - return json.dumps(registry, sort_keys=True, indent=2) + '\n' - - -def _route_id(deployment_id: str, endpoint: str) -> str: - """Deterministic LiteLLM model id for one (deployment, endpoint) route. - - Stable across converges, so route reconcile (:meth:`ComposeBackend. - _reconcile_routes`) can identify one logical route across renders. A route - whose id disappears is deleted by exactly this id; a route whose id remains - but whose observable routing semantics drifted is replaced under the same id. - The ``isr-`` prefix marks it - infer-stack-managed so reconcile never deletes a model someone added by hand. - """ - digest = hashlib.sha256(f'{deployment_id}|{endpoint}'.encode()).hexdigest() - return f'{ROUTE_ID_PREFIX}{digest[:32]}' - - -def _litellm_routes( - deployments: list[Deployment], assignments: dict[str, list[int]] -) -> list[dict[str, Any]]: - """Desired LiteLLM route set for the *live* deployments (dynamic routing). - - One entry per (placed deployment, served endpoint), addressing the - deployment's **own** unique upstream service (:func:`vllm_service_name` with - ``unique=True``). Several dedicated deployments of the same model therefore - yield several entries that share one public ``model_name`` but point at - distinct upstreams — LiteLLM load-balances the alias across them, so each - runs on its own GPU while clients still ask for the single name. Each entry - carries a deterministic ``model_info.id`` (:func:`_route_id`) so applying the - set via the admin API is an idempotent diff, not fire-and-forget calls. - """ - entries: list[dict[str, Any]] = [] - for deployment in sorted(deployments, key=lambda g: (g.created_at, g.id)): - if deployment.id not in assignments: - continue - if deployment.engine == 'vllm': - served = served_name(deployment) - api_base = ( - f'http://{vllm_service_name(deployment, unique=True)}' - f':{VLLM_CONTAINER_PORT}/v1' - ) - for endpoint in sorted(deployment.served): - entries.append( - { - 'model_name': endpoint, - 'litellm_params': { - 'model': f'openai/{served}', - 'api_base': api_base, - 'api_key': 'EMPTY', - }, - 'model_info': {'id': _route_id(deployment.id, endpoint)}, - } - ) - elif deployment.engine == 'ollama': - api_base = ( - f'http://{ollama_service_name(deployment)}:{OLLAMA_CONTAINER_PORT}' - ) - for endpoint, payload in sorted(deployment.served.items()): - tag = payload.get('model', endpoint) - entries.append( - { - 'model_name': endpoint, - 'litellm_params': { - 'model': f'ollama/{tag}', - 'api_base': api_base, - }, - 'model_info': {'id': _route_id(deployment.id, endpoint)}, - } - ) - return entries - - -CONFIG_HASH_LABEL = 'infer-stack.config-hash' - - -def _postgres_service( - images: dict[str, str], state: dict[str, str] -) -> dict[str, Any]: - """Postgres backing LiteLLM's runtime model store (dynamic routing only). - - LiteLLM's admin API (``/model/new`` / ``/model/delete``) only functions with - ``STORE_MODEL_IN_DB=true`` + a database, so dynamic routing needs a DB. This - is an **internal** service (no published host port); LiteLLM reaches it on - the compose network at ``postgres-litellm:5432``. The password is the managed - :data:`DB_PASSWORD_ENV` secret in the sidecar ``.env`` (interpolated by - ``docker compose --env-file``), so it never appears literally in the YAML. - The healthcheck lets the litellm service ``depends_on`` it (condition: - service_healthy) so the gateway only starts once the DB can accept queries. - """ - data_path = state.get('postgres_litellm') or str( - Path(next(iter(state.values()), '.')).parent / 'postgres-litellm' - ) - return { - 'image': images.get('postgres', PINNED_IMAGES['postgres']), - 'environment': { - 'POSTGRES_USER': POSTGRES_DB_USER, - 'POSTGRES_PASSWORD': '${' + DB_PASSWORD_ENV + '}', - 'POSTGRES_DB': POSTGRES_DB_NAME, - }, - 'volumes': [f'{data_path}:/var/lib/postgresql/data'], - 'restart': 'unless-stopped', - 'labels': {ENGINE_LABEL: 'postgres'}, - 'healthcheck': { - 'test': [ - 'CMD-SHELL', - f'pg_isready -U {POSTGRES_DB_USER} -d {POSTGRES_DB_NAME}', - ], - 'interval': '5s', - 'timeout': '5s', - 'retries': 30, - 'start_period': '30s', - }, - } - - -def _litellm_service( - service_names: list[str], - host_port: int, - images: dict[str, str], - aux_dir: str, - master_key: str | None = None, - config_hash: str | None = None, - *, - dynamic_routing: bool = False, - salt_key: bool = False, -) -> dict[str, Any]: - # Reference the managed key via ${...} rather than baking the literal secret - # into the compose YAML. Its value lives in the sidecar .env next to the - # compose file (written by master_key()), which `docker compose --env-file` - # loads for interpolation — so the container and the readiness probe (which - # reads the same .env) still agree regardless of the caller's shell env. - key_value = ( - '${' + API_KEY_ENV + '}' - if master_key is not None - else '${' + API_KEY_ENV + ':-sk-local}' - ) - environment = {API_KEY_ENV: key_value} - if salt_key: - # Only when the .env has one: LiteLLM treats an EMPTY salt as a key, - # so a `${...:-}` default would silently change the encryption key. - environment[SALT_KEY_ENV] = '${' + SALT_KEY_ENV + '}' - if dynamic_routing: - # DB-backed runtime model store so the admin API (/model/new, - # /model/delete) works; the gateway then never needs recreating to learn - # a route. Both are read from the env by LiteLLM. The password is - # interpolated from the sidecar .env, so no secret lands in the YAML. - environment['DATABASE_URL'] = ( - f'postgresql://{POSTGRES_DB_USER}:${{{DB_PASSWORD_ENV}}}' - f'@{POSTGRES_SERVICE}:{POSTGRES_CONTAINER_PORT}/{POSTGRES_DB_NAME}' - ) - environment['STORE_MODEL_IN_DB'] = 'True' - labels = {ENGINE_LABEL: 'litellm'} - if config_hash is not None: - # LiteLLM reads its routing config once at startup; the file is bind- - # mounted, so a config change alone does NOT change this service's spec - # and `docker compose up -d` would leave the old container (and old - # routes) running. Stamping the config hash onto a label makes the spec - # change exactly when the config does, so converge recreates LiteLLM and - # it picks up new/removed aliases. Without this, coalescing a second - # alias onto a live deployment never becomes routable (readiness times out). - labels[CONFIG_HASH_LABEL] = config_hash - service: dict[str, Any] = { - 'image': images['litellm'], - 'command': [ - '--config', - '/etc/litellm/config.yaml', - '--port', - str(LITELLM_CONTAINER_PORT), - ], - 'ports': [f'{host_port}:{LITELLM_CONTAINER_PORT}'], - 'volumes': [f'{aux_dir}/{LITELLM_CONFIG_FILENAME}:/etc/litellm/config.yaml:ro'], - 'environment': environment, - 'restart': 'unless-stopped', - 'labels': labels, - } - if dynamic_routing: - # Wait for the DB to accept queries before the gateway boots; do NOT add - # per-model depends_on (that would churn the spec, i.e. blip, on every - # model change). The route table is filled in afterward via the API. - service['depends_on'] = { - POSTGRES_SERVICE: {'condition': 'service_healthy'} - } - elif service_names: - # Only wait on upstreams when there are any (zero models -> empty gateway). - service['depends_on'] = sorted(service_names) - return service - - -OPEN_WEBUI_SERVICE = 'open-webui' -OPEN_WEBUI_CONTAINER_PORT = 8080 - -NGINX_SERVICE = 'reverse-proxy' -NGINX_CONTAINER_PORT = 80 -NGINX_CONFIG_FILENAME = 'nginx.conf' - - -def _open_webui_service( - host_port: int, - images: dict[str, str], - state: dict[str, str], - master_key: str | None, - *, - openai_urls: list[str] | None = None, - ollama_urls: list[str] | None = None, - depends_on: list[str] | None = None, -) -> dict[str, Any]: - """A managed Open WebUI pointed at whatever front door is available. - - Open WebUI holds two independent kinds of connection, wired here from the - rendered services: - - * **OpenAI** (``openai_urls``) — the chat/completions front door. This is the - LiteLLM gateway when it is enabled (so every declared endpoint alias is - reachable at one URL); with LiteLLM off it falls back to the rendered - upstreams' own ``/v1`` (a single vLLM/Ollama service, or several joined as - ``OPENAI_API_BASE_URLS``). With nothing to point at, the OpenAI API is - disabled rather than left dangling. - * **Ollama** (``ollama_urls``) — the *native* Ollama API of any rendered - Ollama daemon. This is what lets you pull/run/delete models from the UI - and have the daemon load them on demand, independent of LiteLLM — i.e. a - true drop-in for a hand-run ``ollama`` + Open WebUI stack. - - The spec is kept as independent of which models are live as it can be: the - LiteLLM URL is fixed, and the Ollama daemon's service name is its stable - structural id, so adding/removing other models does not rewrite this service - and ``docker compose up -d`` leaves the UI running (the legacy "the UI never - blinks" behavior). Chat history persists under the data dir. - """ - # Reference the managed key via ${...} (resolved from the sidecar .env, see - # _litellm_service) instead of inlining the secret into the compose YAML. - key_value = ( - '${' + API_KEY_ENV + '}' - if master_key is not None - else '${' + API_KEY_ENV + ':-sk-local}' - ) - data_path = state.get('open_webui') or str( - Path(next(iter(state.values()), '.')).parent / 'open-webui' - ) - openai_urls = list(openai_urls or []) - ollama_urls = list(ollama_urls or []) - env: dict[str, str] = { - # Single-user workstation default; the port shouldn't be exposed - # publicly. Tracked as a knob in dev/leasing-followups.md. - 'WEBUI_AUTH': 'False', - } - if openai_urls: - env['ENABLE_OPENAI_API'] = 'True' - if len(openai_urls) == 1: - env['OPENAI_API_BASE_URL'] = openai_urls[0] - else: - env['OPENAI_API_BASE_URLS'] = ';'.join(openai_urls) - env['OPENAI_API_KEY'] = key_value - else: - env['ENABLE_OPENAI_API'] = 'False' - if ollama_urls: - env['ENABLE_OLLAMA_API'] = 'True' - if len(ollama_urls) == 1: - env['OLLAMA_BASE_URL'] = ollama_urls[0] - else: - env['OLLAMA_BASE_URLS'] = ';'.join(ollama_urls) - else: - env['ENABLE_OLLAMA_API'] = 'False' - service: dict[str, Any] = { - 'image': images['open_webui'], - 'ports': [f'{host_port}:{OPEN_WEBUI_CONTAINER_PORT}'], - 'environment': env, - 'volumes': [f'{data_path}:/app/backend/data'], - 'restart': 'unless-stopped', - 'labels': {ENGINE_LABEL: 'open-webui'}, - } - if depends_on: - service['depends_on'] = sorted(depends_on) - return service - - -def _nginx_conf(*, litellm: bool, ui: bool) -> str: - """A minimal HTTP reverse-proxy conf: one origin, path-routed. - - ``/v1/`` -> the LiteLLM gateway (the OpenAI API), ``/`` -> Open WebUI (or the - gateway when there's no UI). Plain HTTP — no TLS, no auth — so the value is - "one port, nothing to remember", not security. The ``map`` is valid here - because a ``conf.d/*.conf`` file is included in nginx's ``http`` context. - """ - api = f'http://{LITELLM_SERVICE}:{LITELLM_CONTAINER_PORT}' - locations = '' - if litellm: - locations += ( - ' location /v1/ {\n' - f' proxy_pass {api}/v1/;\n' - ' proxy_set_header Host $host;\n' - ' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n' - ' proxy_set_header X-Forwarded-Proto $scheme;\n' - ' proxy_read_timeout 600s;\n' - ' }\n' - ) - # `/` serves the UI when present, else the gateway (so hitting the host root - # still lands somewhere useful). Upgrade headers keep Open WebUI's websockets - # working; client_max_body_size 0 allows large uploads. - if ui: - root = f'http://{OPEN_WEBUI_SERVICE}:{OPEN_WEBUI_CONTAINER_PORT}' - elif litellm: - root = api - else: - root = '' - if root: - locations += ( - ' location / {\n' - f' proxy_pass {root};\n' - ' proxy_http_version 1.1;\n' - ' proxy_set_header Upgrade $http_upgrade;\n' - ' proxy_set_header Connection $connection_upgrade;\n' - ' proxy_set_header Host $host;\n' - ' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n' - ' proxy_set_header X-Forwarded-Proto $scheme;\n' - ' }\n' - ) - return ( - 'map $http_upgrade $connection_upgrade {\n' - ' default upgrade;\n' - " '' close;\n" - '}\n\n' - 'server {\n' - f' listen {NGINX_CONTAINER_PORT};\n' - ' server_name _;\n' - ' client_max_body_size 0;\n' - f'{locations}' - '}\n' - ) - - -def _nginx_service( - host_port: int, - images: dict[str, str], - *, - aux_dir: str, - depends_on: list[str], - config_path: str | None = None, - config_hash: str | None = None, -) -> dict[str, Any]: - # BYO config (config_path) is mounted verbatim; otherwise the generated - # nginx.conf in the state dir is used. - mount = config_path or f'{aux_dir}/{NGINX_CONFIG_FILENAME}' - labels = {ENGINE_LABEL: 'nginx'} - if config_hash is not None: - # Same trick as LiteLLM: the conf is bind-mounted, so stamp its hash on a - # label to force a recreate when the routing changes. - labels[CONFIG_HASH_LABEL] = config_hash - service: dict[str, Any] = { - 'image': images['nginx'], - 'ports': [f'{host_port}:{NGINX_CONTAINER_PORT}'], - 'volumes': [f'{mount}:/etc/nginx/conf.d/default.conf:ro'], - 'restart': 'unless-stopped', - 'labels': labels, - } - if depends_on: - service['depends_on'] = sorted(depends_on) - return service - - def render_compose( deployments: list[Deployment], assignments: dict[str, list[int]], @@ -1534,15 +825,6 @@ def render_compose( DOCKER_TIMEOUT_CONVERGE = 1800.0 # compose up / down DOCKER_TIMEOUT_PULL = 3600.0 # pull, manifest inspect, in-container model pulls -#: Budget for reconciling dynamic routes against the gateway: listing, POSTs -#: and verification together. This default is the bootstrap budget (a fresh -#: gateway waits on Postgres health and runs DB migrations); it preserves the -#: previous 90 x 2 s listing retry. -ROUTE_RECONCILE_BOOTSTRAP_S = 180.0 -#: Budget when the gateway was already running before this apply (steady state). -#: Short, because the controller holds its host-wide lock while applying; on -#: expiry the change stays pending and the next applying operation retries. -ROUTE_RECONCILE_STEADY_S = 20.0 def _docker_timeout(args: list[str]) -> float: @@ -1761,24 +1043,6 @@ def _parse_ps(out: str) -> set[str]: return running -def set_master_key(env_path: Path, key: str) -> None: - """Replace the LiteLLM master key in ``env_path`` without losing DB routes. - - The first time the key changes, the old one is pinned as - ``LITELLM_SALT_KEY`` -- the value LiteLLM has been encrypting stored - credentials with. After that the salt stays put and only the key moves. - """ - if not key.startswith('sk-'): - # master_key() would silently replace it on the next render. - raise ValueError(f'{API_KEY_ENV} must start with "sk-" (LiteLLM rejects others)') - existing = parse_env_file(env_path) - values = {API_KEY_ENV: key} - old = existing.get(API_KEY_ENV, '').strip() - if old and old != key and not existing.get(SALT_KEY_ENV, '').strip(): - values[SALT_KEY_ENV] = old - write_env_file(env_path, values) - - class ComposeBackend(ConvergeScaffold): """Single-host docker compose backend (converge-style). diff --git a/infer_stack/leasing/gateway.py b/infer_stack/leasing/gateway.py new file mode 100644 index 00000000..a15d174c --- /dev/null +++ b/infer_stack/leasing/gateway.py @@ -0,0 +1,732 @@ +"""The front door: the LiteLLM gateway, its routes and keys, and the UI and +proxy beside it. + +Engines (vLLM, Ollama) are the backend's business; everything a client +talks to is here. A backend hands the gateway where its models are (route +rows), and the gateway renders its Compose services and config, keeps the +route registry, reconciles dynamic routes through LiteLLM's admin API, and +manages the master key. The compose backend runs it beside its engines; the +kubeai backend runs it alone, in front of a cluster. +""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +from typing import Any + +import yaml + +from ..config import PINNED_IMAGES +from ..env_utils import parse_env_file, write_env_file +from .models import Deployment, served_name +from .naming import ( + OLLAMA_CONTAINER_PORT, + VLLM_CONTAINER_PORT, + ollama_service_name, + ollama_service_name_for, + vllm_service_name, + vllm_service_name_for, +) +from .residency import ENGINE_LABEL + +LITELLM_CONTAINER_PORT = 4000 +LITELLM_CONFIG_FILENAME = 'litellm_config.yaml' +LITELLM_SERVICE = 'litellm' +API_KEY_ENV = 'LITELLM_MASTER_KEY' +# The key LiteLLM encrypts credentials stored in its database with. Unset, it +# uses the master key -- so rotating the master key would make every +# DB-stored route undecryptable. `set_master_key` pins it to the pre-rotation +# master key the first time the key changes; it must never change after that. +SALT_KEY_ENV = 'LITELLM_SALT_KEY' + +# Dynamic-routing (admin-API) extras. When dynamic routing is on, the gateway's +# route table is managed live via LiteLLM's admin API against a Postgres-backed +# model store, instead of a static config file. See render_compose + +# ComposeBackend._reconcile_routes and docs/litellm-gateway-routing.md. +LITELLM_ROUTES_FILENAME = 'litellm_routes.json' # rendered desired route set +# Append-only route registry for static-superset mode: accumulates the semantic +# route inputs (served name / engine / host) of every catalog *and* every live +# deployment ever merged, across all runbooks sharing this state dir. The gateway +# `model_list` is rendered from the whole registry, so a converge under one +# runbook's catalog can no longer strip another's still-live routes, and once +# every catalog has been merged once the rendered config is byte-stable (the +# gateway is never recreated). See docs/litellm-gateway-routing.md and +# ComposeBackend._update_route_registry. +LITELLM_REGISTRY_FILENAME = 'litellm_registry.json' +LITELLM_REGISTRY_VERSION = 1 +# A route-registry row for a server this project does not run: +# ``{'engine': UPSTREAM_ROUTE, 'served': , 'api_base': }``. +UPSTREAM_ROUTE = 'upstream' +POSTGRES_SERVICE = 'postgres-litellm' +POSTGRES_CONTAINER_PORT = 5432 +POSTGRES_DB_NAME = 'litellm' +POSTGRES_DB_USER = 'litellm' +DB_PASSWORD_ENV = 'LITELLM_DB_PASSWORD' # managed secret in the sidecar .env +# Marks a LiteLLM route as infer-stack-managed, so reconcile only ever deletes +# routes it created (never a model added by hand through the UI/admin API). +ROUTE_ID_PREFIX = 'isr-' + +#: Budget for reconciling dynamic routes against the gateway: listing, POSTs +#: and verification together. This default is the bootstrap budget (a fresh +#: gateway waits on Postgres health and runs DB migrations); it preserves the +#: previous 90 x 2 s listing retry. +ROUTE_RECONCILE_BOOTSTRAP_S = 180.0 +#: Budget when the gateway was already running before this apply (steady state). +#: Short, because the controller holds its host-wide lock while applying; on +#: expiry the change stays pending and the next applying operation retries. +ROUTE_RECONCILE_STEADY_S = 20.0 + + +def _vllm_route_entry( + model_name: str, served: str, api_base: str +) -> dict[str, Any]: + """One LiteLLM ``model_list`` entry routing ``model_name`` to a vLLM upstream. + + Shared by every render path (legacy per-deployment, catalog-superset, and + the route registry) so a registry-rendered entry can never drift from what + the catalog/deployment paths produce for the same endpoint.""" + return { + 'model_name': model_name, + 'litellm_params': { + 'model': f'openai/{served}', + 'api_base': api_base, + 'api_key': 'EMPTY', + }, + } + + +def _ollama_route_entry( + model_name: str, tag: str, api_base: str +) -> dict[str, Any]: + """One LiteLLM ``model_list`` entry routing ``model_name`` to an Ollama tag + (see :func:`_vllm_route_entry` for why this is factored out).""" + return { + 'model_name': model_name, + 'litellm_params': { + 'model': f'ollama/{tag}', + 'api_base': api_base, + }, + } + + +def _litellm_model_list( + deployments: list[Deployment], assignments: dict[str, list[int]] +) -> list[dict[str, Any]]: + """One LiteLLM ``model_list`` entry per served endpoint alias.""" + entries: list[dict[str, Any]] = [] + for deployment in sorted(deployments, key=lambda g: (g.created_at, g.id)): + if deployment.id not in assignments: + continue + if deployment.engine == 'vllm': + served = served_name(deployment) + api_base = f'http://{vllm_service_name(deployment)}:8000/v1' + for endpoint in sorted(deployment.served): + entries.append(_vllm_route_entry(endpoint, served, api_base)) + elif deployment.engine == 'ollama': + api_base = f'http://{ollama_service_name(deployment)}:{OLLAMA_CONTAINER_PORT}' + for endpoint, payload in sorted(deployment.served.items()): + tag = payload.get('model', endpoint) + entries.append(_ollama_route_entry(endpoint, tag, api_base)) + return entries + + +def _litellm_model_list_from_catalog(catalog: Any) -> list[dict[str, Any]]: + """A *static superset* ``model_list``: one route per catalog endpoint. + + Unlike :func:`_litellm_model_list` (which routes only the currently-placed + deployments), this routes *every* catalog endpoint to its deterministic + upstream host (:func:`vllm_service_name_for` / :func:`ollama_service_name_for`). + The resulting config therefore depends only on the catalog, not on which + models happen to be up — so acquiring/releasing a model leaves the gateway's + config (and its container) untouched (no blip). A route whose upstream is not + currently running simply errors/cools-down until it comes up; the + ``router_settings`` below make that warmup self-healing. ``/v1/models`` lists + the whole catalog (some upstreams down) rather than only the live set. + + This static-superset path is the default. Its one limitation — it cannot give + same-model ``--dedicated`` deployments distinct upstreams, and cannot route + non-catalog acquires without a config change — is addressed by the opt-in + *dynamic routing* mode (``dynamic_routing=True``), which manages routes live + via LiteLLM's admin API against a Postgres model store (see + :func:`_litellm_routes`, :meth:`ComposeBackend._reconcile_routes`, and + ``docs/litellm-gateway-routing.md``). The two are mutually exclusive per + converge; this function is used only when dynamic routing is off. + """ + entries: list[dict[str, Any]] = [] + for name in sorted(getattr(catalog, 'endpoints', {})): + try: + req = catalog.resolve_endpoint(name) + except Exception: # noqa: BLE001 - a bad endpoint must not break the gateway + continue + if req.engine == 'vllm': + served = req.served.get('served_model_name') or name + api_base = ( + f'http://{vllm_service_name_for(served)}:{VLLM_CONTAINER_PORT}/v1' + ) + entries.append(_vllm_route_entry(name, served, api_base)) + elif req.engine == 'ollama': + host = req.spec.get('host') or req.host + tag = req.served.get('model') or name + api_base = ( + f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' + ) + entries.append(_ollama_route_entry(name, tag, api_base)) + return entries + + +# -- Route registry (static-superset persistence) -------------------------- +# +# The registry stores *semantic* route inputs (served name / engine / host), +# never rendered LiteLLM entries — render derives entries through the same +# helpers the catalog/deployment paths use (:func:`_litellm_model_list_from_registry`), +# so a future renderer change propagates to old registry rows automatically. +# All functions here are pure; the backend owns the file I/O and locking. + + +def _registry_incoming_from_catalog(catalog: Any) -> dict[str, dict[str, Any]]: + """Semantic route rows for every resolvable endpoint of ``catalog``. + + Mirrors :func:`_litellm_model_list_from_catalog`'s iteration (unresolvable + endpoints skipped) but emits registry rows keyed by endpoint name. A vLLM + row carries only ``served`` (the upstream host is re-derived at render via + :func:`vllm_service_name_for`); an Ollama row carries ``model`` (tag) + + ``host``.""" + incoming: dict[str, dict[str, Any]] = {} + for name in sorted(getattr(catalog, 'endpoints', {})): + try: + req = catalog.resolve_endpoint(name) + except Exception: # noqa: BLE001 - a bad endpoint must not break the gateway + continue + if req.engine == 'vllm': + served = req.served.get('served_model_name') or name + incoming[name] = {'engine': 'vllm', 'served': served} + elif req.engine == 'ollama': + host = req.spec.get('host') or req.host + tag = req.served.get('model') or name + incoming[name] = {'engine': 'ollama', 'model': tag, 'host': host} + return incoming + + +def _registry_incoming_from_deployments( + deployments: list[Deployment], assignments: dict[str, list[int]] +) -> dict[str, dict[str, Any]]: + """Semantic route rows for every *placed* deployment in ``assignments``. + + ``deployments`` is the full ``desired`` set (which spans all runbooks via + the shared ledger), so this keeps non-catalog / dedicated acquires routable + and — because the registry persists — routable past release. One row per key + of ``deployment.served`` (a coalesced deployment can back several endpoint + aliases). Only ``vllm``/``ollama`` engines contribute; ``RESERVED_ENGINE`` + and unknown engines render no service, so they contribute no row — exactly + as :func:`render_compose`'s service loop skips them. + + The vLLM ``served`` uses the same fallback chain as :func:`vllm_service_name` + (``spec['served_model_name'] or sorted(served)[0] or id``), so a + catalog-listed endpoint acquired live reduces to the identical row a catalog + merge produces — live-vs-released status never moves the rendered bytes.""" + incoming: dict[str, dict[str, Any]] = {} + for deployment in deployments: + if deployment.id not in assignments: + continue + if deployment.engine == 'vllm': + served = served_name(deployment) + for endpoint in sorted(deployment.served): + incoming[endpoint] = {'engine': 'vllm', 'served': served} + elif deployment.engine == 'ollama': + host = deployment.spec.get('host') or deployment.id + for endpoint, payload in sorted(deployment.served.items()): + tag = payload.get('model', endpoint) + incoming[endpoint] = { + 'engine': 'ollama', + 'model': tag, + 'host': host, + } + return incoming + + +def _litellm_model_list_from_registry( + registry: dict[str, Any] +) -> list[dict[str, Any]]: + """Render the gateway ``model_list`` from the whole accumulated registry. + + Iterates ``sorted(entries)`` (determinism, §8) and derives each upstream + ``api_base`` through the live naming helpers, so the registry never becomes + a rendered-config parse surface.""" + entries: list[dict[str, Any]] = [] + rows = registry.get('entries', {}) if isinstance(registry, dict) else {} + for name in sorted(rows): + row = rows[name] + if not isinstance(row, dict): + continue + engine = row.get('engine') + if engine == 'vllm': + served = row.get('served') or name + api_base = ( + f'http://{vllm_service_name_for(served)}:{VLLM_CONTAINER_PORT}/v1' + ) + entries.append(_vllm_route_entry(name, served, api_base)) + elif engine == 'ollama': + tag = row.get('model') or name + host = row.get('host') or name + api_base = ( + f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' + ) + entries.append(_ollama_route_entry(name, tag, api_base)) + elif engine == UPSTREAM_ROUTE and row.get('api_base'): + # An OpenAI-compatible server this project does not run (a KubeAI + # cluster's gateway): the row carries its address and the name it + # serves the model under. + entries.append(_vllm_route_entry(name, row.get('served') or name, + str(row['api_base']))) + return entries + + +def _merge_route_registry( + existing: dict[str, Any], incoming: dict[str, dict[str, Any]] +) -> tuple[dict[str, Any], list[str]]: + """Merge ``incoming`` semantic rows into ``existing`` (append-only). + + Idempotent (merging identical rows is a no-op) and additive (never removes a + row). On a conflict — same key, different row — *incoming wins* and a warning + naming both definitions is emitted; the changed definition changes the + rendered bytes, which is the one justified recreate. The existing ``version`` + is preserved (an unknown version merged under is not silently rewritten to + the current schema; see :meth:`ComposeBackend._load_route_registry`).""" + version = LITELLM_REGISTRY_VERSION + entries: dict[str, dict[str, Any]] = {} + if isinstance(existing, dict): + version = existing.get('version', LITELLM_REGISTRY_VERSION) + prior = existing.get('entries') + if isinstance(prior, dict): + entries = {k: v for k, v in prior.items()} + warnings: list[str] = [] + for name in sorted(incoming): + row = incoming[name] + if name in entries and entries[name] != row: + warnings.append( + f"route {name!r} redefined: {entries[name]} -> {row} " + '(incoming wins; gateway will be recreated once)' + ) + entries[name] = row + return {'version': version, 'entries': entries}, warnings + + +def _seed_registry_from_litellm_config( + config_text: str, +) -> tuple[dict[str, Any], list[str]]: + """One-shot upgrade seed: recover registry rows from a rendered + ``litellm_config.yaml`` so the first post-upgrade converge does not strip + the other runbooks' routes. + + Single-format, migration-time only (no cross-version promise): ``openai/`` + inverts *exactly* to ``{engine: vllm, served}``. Ollama rows are skipped with + a warning — the host survives only as a non-invertible ``dns_slug`` inside + ``api_base`` — and re-enter the registry at the next converge that has them + in its catalog or live set. Anything else unparseable is likewise skipped.""" + entries: dict[str, dict[str, Any]] = {} + warnings: list[str] = [] + try: + data = yaml.safe_load(config_text) or {} + except Exception: # noqa: BLE001 - a torn file must not brick seeding + return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}}, [ + 'seed: litellm_config.yaml is unparseable; starting an empty registry' + ] + for entry in data.get('model_list', []) or []: + name = entry.get('model_name') + model = (entry.get('litellm_params') or {}).get('model', '') + if not name: + continue + if isinstance(model, str) and model.startswith('openai/'): + entries[name] = {'engine': 'vllm', 'served': model[len('openai/'):]} + elif isinstance(model, str) and model.startswith('ollama/'): + warnings.append( + f"seed: skipping Ollama route {name!r} (host not recoverable " + 'from the rendered api_base; it re-enters at its next converge)' + ) + else: + warnings.append(f'seed: skipping unparseable route {name!r}') + return {'version': LITELLM_REGISTRY_VERSION, 'entries': entries}, warnings + + +def _dump_route_registry(registry: dict[str, Any]) -> str: + """Canonical, byte-stable serialization (§3): sorted keys + trailing + newline. A nondeterministic dump would manufacture phantom hash changes.""" + return json.dumps(registry, sort_keys=True, indent=2) + '\n' + + +def _route_id(deployment_id: str, endpoint: str) -> str: + """Deterministic LiteLLM model id for one (deployment, endpoint) route. + + Stable across converges, so route reconcile (:meth:`ComposeBackend. + _reconcile_routes`) can identify one logical route across renders. A route + whose id disappears is deleted by exactly this id; a route whose id remains + but whose observable routing semantics drifted is replaced under the same id. + The ``isr-`` prefix marks it + infer-stack-managed so reconcile never deletes a model someone added by hand. + """ + digest = hashlib.sha256(f'{deployment_id}|{endpoint}'.encode()).hexdigest() + return f'{ROUTE_ID_PREFIX}{digest[:32]}' + + +def _litellm_routes( + deployments: list[Deployment], assignments: dict[str, list[int]] +) -> list[dict[str, Any]]: + """Desired LiteLLM route set for the *live* deployments (dynamic routing). + + One entry per (placed deployment, served endpoint), addressing the + deployment's **own** unique upstream service (:func:`vllm_service_name` with + ``unique=True``). Several dedicated deployments of the same model therefore + yield several entries that share one public ``model_name`` but point at + distinct upstreams — LiteLLM load-balances the alias across them, so each + runs on its own GPU while clients still ask for the single name. Each entry + carries a deterministic ``model_info.id`` (:func:`_route_id`) so applying the + set via the admin API is an idempotent diff, not fire-and-forget calls. + """ + entries: list[dict[str, Any]] = [] + for deployment in sorted(deployments, key=lambda g: (g.created_at, g.id)): + if deployment.id not in assignments: + continue + if deployment.engine == 'vllm': + served = served_name(deployment) + api_base = ( + f'http://{vllm_service_name(deployment, unique=True)}' + f':{VLLM_CONTAINER_PORT}/v1' + ) + for endpoint in sorted(deployment.served): + entries.append( + { + 'model_name': endpoint, + 'litellm_params': { + 'model': f'openai/{served}', + 'api_base': api_base, + 'api_key': 'EMPTY', + }, + 'model_info': {'id': _route_id(deployment.id, endpoint)}, + } + ) + elif deployment.engine == 'ollama': + api_base = ( + f'http://{ollama_service_name(deployment)}:{OLLAMA_CONTAINER_PORT}' + ) + for endpoint, payload in sorted(deployment.served.items()): + tag = payload.get('model', endpoint) + entries.append( + { + 'model_name': endpoint, + 'litellm_params': { + 'model': f'ollama/{tag}', + 'api_base': api_base, + }, + 'model_info': {'id': _route_id(deployment.id, endpoint)}, + } + ) + return entries + + +CONFIG_HASH_LABEL = 'infer-stack.config-hash' + + +def _postgres_service( + images: dict[str, str], state: dict[str, str] +) -> dict[str, Any]: + """Postgres backing LiteLLM's runtime model store (dynamic routing only). + + LiteLLM's admin API (``/model/new`` / ``/model/delete``) only functions with + ``STORE_MODEL_IN_DB=true`` + a database, so dynamic routing needs a DB. This + is an **internal** service (no published host port); LiteLLM reaches it on + the compose network at ``postgres-litellm:5432``. The password is the managed + :data:`DB_PASSWORD_ENV` secret in the sidecar ``.env`` (interpolated by + ``docker compose --env-file``), so it never appears literally in the YAML. + The healthcheck lets the litellm service ``depends_on`` it (condition: + service_healthy) so the gateway only starts once the DB can accept queries. + """ + data_path = state.get('postgres_litellm') or str( + Path(next(iter(state.values()), '.')).parent / 'postgres-litellm' + ) + return { + 'image': images.get('postgres', PINNED_IMAGES['postgres']), + 'environment': { + 'POSTGRES_USER': POSTGRES_DB_USER, + 'POSTGRES_PASSWORD': '${' + DB_PASSWORD_ENV + '}', + 'POSTGRES_DB': POSTGRES_DB_NAME, + }, + 'volumes': [f'{data_path}:/var/lib/postgresql/data'], + 'restart': 'unless-stopped', + 'labels': {ENGINE_LABEL: 'postgres'}, + 'healthcheck': { + 'test': [ + 'CMD-SHELL', + f'pg_isready -U {POSTGRES_DB_USER} -d {POSTGRES_DB_NAME}', + ], + 'interval': '5s', + 'timeout': '5s', + 'retries': 30, + 'start_period': '30s', + }, + } + + +def _litellm_service( + service_names: list[str], + host_port: int, + images: dict[str, str], + aux_dir: str, + master_key: str | None = None, + config_hash: str | None = None, + *, + dynamic_routing: bool = False, + salt_key: bool = False, +) -> dict[str, Any]: + # Reference the managed key via ${...} rather than baking the literal secret + # into the compose YAML. Its value lives in the sidecar .env next to the + # compose file (written by master_key()), which `docker compose --env-file` + # loads for interpolation — so the container and the readiness probe (which + # reads the same .env) still agree regardless of the caller's shell env. + key_value = ( + '${' + API_KEY_ENV + '}' + if master_key is not None + else '${' + API_KEY_ENV + ':-sk-local}' + ) + environment = {API_KEY_ENV: key_value} + if salt_key: + # Only when the .env has one: LiteLLM treats an EMPTY salt as a key, + # so a `${...:-}` default would silently change the encryption key. + environment[SALT_KEY_ENV] = '${' + SALT_KEY_ENV + '}' + if dynamic_routing: + # DB-backed runtime model store so the admin API (/model/new, + # /model/delete) works; the gateway then never needs recreating to learn + # a route. Both are read from the env by LiteLLM. The password is + # interpolated from the sidecar .env, so no secret lands in the YAML. + environment['DATABASE_URL'] = ( + f'postgresql://{POSTGRES_DB_USER}:${{{DB_PASSWORD_ENV}}}' + f'@{POSTGRES_SERVICE}:{POSTGRES_CONTAINER_PORT}/{POSTGRES_DB_NAME}' + ) + environment['STORE_MODEL_IN_DB'] = 'True' + labels = {ENGINE_LABEL: 'litellm'} + if config_hash is not None: + # LiteLLM reads its routing config once at startup; the file is bind- + # mounted, so a config change alone does NOT change this service's spec + # and `docker compose up -d` would leave the old container (and old + # routes) running. Stamping the config hash onto a label makes the spec + # change exactly when the config does, so converge recreates LiteLLM and + # it picks up new/removed aliases. Without this, coalescing a second + # alias onto a live deployment never becomes routable (readiness times out). + labels[CONFIG_HASH_LABEL] = config_hash + service: dict[str, Any] = { + 'image': images['litellm'], + 'command': [ + '--config', + '/etc/litellm/config.yaml', + '--port', + str(LITELLM_CONTAINER_PORT), + ], + 'ports': [f'{host_port}:{LITELLM_CONTAINER_PORT}'], + 'volumes': [f'{aux_dir}/{LITELLM_CONFIG_FILENAME}:/etc/litellm/config.yaml:ro'], + 'environment': environment, + 'restart': 'unless-stopped', + 'labels': labels, + } + if dynamic_routing: + # Wait for the DB to accept queries before the gateway boots; do NOT add + # per-model depends_on (that would churn the spec, i.e. blip, on every + # model change). The route table is filled in afterward via the API. + service['depends_on'] = { + POSTGRES_SERVICE: {'condition': 'service_healthy'} + } + elif service_names: + # Only wait on upstreams when there are any (zero models -> empty gateway). + service['depends_on'] = sorted(service_names) + return service + + +OPEN_WEBUI_SERVICE = 'open-webui' +OPEN_WEBUI_CONTAINER_PORT = 8080 + +NGINX_SERVICE = 'reverse-proxy' +NGINX_CONTAINER_PORT = 80 +NGINX_CONFIG_FILENAME = 'nginx.conf' + + +def _open_webui_service( + host_port: int, + images: dict[str, str], + state: dict[str, str], + master_key: str | None, + *, + openai_urls: list[str] | None = None, + ollama_urls: list[str] | None = None, + depends_on: list[str] | None = None, +) -> dict[str, Any]: + """A managed Open WebUI pointed at whatever front door is available. + + Open WebUI holds two independent kinds of connection, wired here from the + rendered services: + + * **OpenAI** (``openai_urls``) — the chat/completions front door. This is the + LiteLLM gateway when it is enabled (so every declared endpoint alias is + reachable at one URL); with LiteLLM off it falls back to the rendered + upstreams' own ``/v1`` (a single vLLM/Ollama service, or several joined as + ``OPENAI_API_BASE_URLS``). With nothing to point at, the OpenAI API is + disabled rather than left dangling. + * **Ollama** (``ollama_urls``) — the *native* Ollama API of any rendered + Ollama daemon. This is what lets you pull/run/delete models from the UI + and have the daemon load them on demand, independent of LiteLLM — i.e. a + true drop-in for a hand-run ``ollama`` + Open WebUI stack. + + The spec is kept as independent of which models are live as it can be: the + LiteLLM URL is fixed, and the Ollama daemon's service name is its stable + structural id, so adding/removing other models does not rewrite this service + and ``docker compose up -d`` leaves the UI running (the legacy "the UI never + blinks" behavior). Chat history persists under the data dir. + """ + # Reference the managed key via ${...} (resolved from the sidecar .env, see + # _litellm_service) instead of inlining the secret into the compose YAML. + key_value = ( + '${' + API_KEY_ENV + '}' + if master_key is not None + else '${' + API_KEY_ENV + ':-sk-local}' + ) + data_path = state.get('open_webui') or str( + Path(next(iter(state.values()), '.')).parent / 'open-webui' + ) + openai_urls = list(openai_urls or []) + ollama_urls = list(ollama_urls or []) + env: dict[str, str] = { + # Single-user workstation default; the port shouldn't be exposed + # publicly. Tracked as a knob in dev/leasing-followups.md. + 'WEBUI_AUTH': 'False', + } + if openai_urls: + env['ENABLE_OPENAI_API'] = 'True' + if len(openai_urls) == 1: + env['OPENAI_API_BASE_URL'] = openai_urls[0] + else: + env['OPENAI_API_BASE_URLS'] = ';'.join(openai_urls) + env['OPENAI_API_KEY'] = key_value + else: + env['ENABLE_OPENAI_API'] = 'False' + if ollama_urls: + env['ENABLE_OLLAMA_API'] = 'True' + if len(ollama_urls) == 1: + env['OLLAMA_BASE_URL'] = ollama_urls[0] + else: + env['OLLAMA_BASE_URLS'] = ';'.join(ollama_urls) + else: + env['ENABLE_OLLAMA_API'] = 'False' + service: dict[str, Any] = { + 'image': images['open_webui'], + 'ports': [f'{host_port}:{OPEN_WEBUI_CONTAINER_PORT}'], + 'environment': env, + 'volumes': [f'{data_path}:/app/backend/data'], + 'restart': 'unless-stopped', + 'labels': {ENGINE_LABEL: 'open-webui'}, + } + if depends_on: + service['depends_on'] = sorted(depends_on) + return service + + +def _nginx_conf(*, litellm: bool, ui: bool) -> str: + """A minimal HTTP reverse-proxy conf: one origin, path-routed. + + ``/v1/`` -> the LiteLLM gateway (the OpenAI API), ``/`` -> Open WebUI (or the + gateway when there's no UI). Plain HTTP — no TLS, no auth — so the value is + "one port, nothing to remember", not security. The ``map`` is valid here + because a ``conf.d/*.conf`` file is included in nginx's ``http`` context. + """ + api = f'http://{LITELLM_SERVICE}:{LITELLM_CONTAINER_PORT}' + locations = '' + if litellm: + locations += ( + ' location /v1/ {\n' + f' proxy_pass {api}/v1/;\n' + ' proxy_set_header Host $host;\n' + ' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n' + ' proxy_set_header X-Forwarded-Proto $scheme;\n' + ' proxy_read_timeout 600s;\n' + ' }\n' + ) + # `/` serves the UI when present, else the gateway (so hitting the host root + # still lands somewhere useful). Upgrade headers keep Open WebUI's websockets + # working; client_max_body_size 0 allows large uploads. + if ui: + root = f'http://{OPEN_WEBUI_SERVICE}:{OPEN_WEBUI_CONTAINER_PORT}' + elif litellm: + root = api + else: + root = '' + if root: + locations += ( + ' location / {\n' + f' proxy_pass {root};\n' + ' proxy_http_version 1.1;\n' + ' proxy_set_header Upgrade $http_upgrade;\n' + ' proxy_set_header Connection $connection_upgrade;\n' + ' proxy_set_header Host $host;\n' + ' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n' + ' proxy_set_header X-Forwarded-Proto $scheme;\n' + ' }\n' + ) + return ( + 'map $http_upgrade $connection_upgrade {\n' + ' default upgrade;\n' + " '' close;\n" + '}\n\n' + 'server {\n' + f' listen {NGINX_CONTAINER_PORT};\n' + ' server_name _;\n' + ' client_max_body_size 0;\n' + f'{locations}' + '}\n' + ) + + +def _nginx_service( + host_port: int, + images: dict[str, str], + *, + aux_dir: str, + depends_on: list[str], + config_path: str | None = None, + config_hash: str | None = None, +) -> dict[str, Any]: + # BYO config (config_path) is mounted verbatim; otherwise the generated + # nginx.conf in the state dir is used. + mount = config_path or f'{aux_dir}/{NGINX_CONFIG_FILENAME}' + labels = {ENGINE_LABEL: 'nginx'} + if config_hash is not None: + # Same trick as LiteLLM: the conf is bind-mounted, so stamp its hash on a + # label to force a recreate when the routing changes. + labels[CONFIG_HASH_LABEL] = config_hash + service: dict[str, Any] = { + 'image': images['nginx'], + 'ports': [f'{host_port}:{NGINX_CONTAINER_PORT}'], + 'volumes': [f'{mount}:/etc/nginx/conf.d/default.conf:ro'], + 'restart': 'unless-stopped', + 'labels': labels, + } + if depends_on: + service['depends_on'] = sorted(depends_on) + return service + + +def set_master_key(env_path: Path, key: str) -> None: + """Replace the LiteLLM master key in ``env_path`` without losing DB routes. + + The first time the key changes, the old one is pinned as + ``LITELLM_SALT_KEY`` -- the value LiteLLM has been encrypting stored + credentials with. After that the salt stays put and only the key moves. + """ + if not key.startswith('sk-'): + # master_key() would silently replace it on the next render. + raise ValueError(f'{API_KEY_ENV} must start with "sk-" (LiteLLM rejects others)') + existing = parse_env_file(env_path) + values = {API_KEY_ENV: key} + old = existing.get(API_KEY_ENV, '').strip() + if old and old != key and not existing.get(SALT_KEY_ENV, '').strip(): + values[SALT_KEY_ENV] = old + write_env_file(env_path, values) + + diff --git a/infer_stack/leasing/naming.py b/infer_stack/leasing/naming.py new file mode 100644 index 00000000..c2d58c82 --- /dev/null +++ b/infer_stack/leasing/naming.py @@ -0,0 +1,99 @@ +"""The names infer-stack gives the things it runs, in one place. + +A compose service, a Kubernetes object, and the upstream host a gateway +route points at are all derived from the served model name (or an Ollama +host). The engine side and the gateway side must agree on these exactly, so +both import them from here. +""" + +from __future__ import annotations + +import re + +from .models import Deployment, served_name + +#: The port every engine container serves on inside the Compose network. +VLLM_CONTAINER_PORT = 8000 +OLLAMA_CONTAINER_PORT = 11434 + + +def dns_slug(text: str) -> str: + """A lowercase ``[a-z0-9-]`` label safe as a compose service / DNS / + Kubernetes object name (shared by the compose and kubeai backends).""" + out = re.sub(r'[^a-z0-9]+', '-', str(text).lower()).strip('-') + return out or 'model' + + +_dns_slug = dns_slug # historical internal name + + +def vllm_service_name_for(served: str) -> str: + """Deterministic compose/DNS service name for a vLLM upstream: ``vllm-``. + + Derived purely from the served model name (vLLM's ``--served-model-name``, + the Open WebUI label, the alias the user chose), so it is identical whether + computed from a live :class:`Deployment` or from a catalog endpoint. That + stability is what lets the LiteLLM gateway carry a *static* route table (one + per catalog endpoint) whose upstream hosts match the containers when they + come up — so adding/removing models does not rewrite the gateway's config and + the gateway is never recreated (no "blip"); see :func:`_litellm_model_list`. + """ + return f'vllm-{_dns_slug(served)}' + + +def _unique_vllm_service_name(served: str, deployment_id: str) -> str: + """Per-deployment vLLM service/DNS name: ``vllm--``. + + The static-superset gateway needs a name derivable from the served model + *alone* (so a catalog route can address it without knowing the live + deployment) — but that deliberately drops the deployment id, which + **collapses every** ``--dedicated`` **deployment of one model onto a single + container** (hence one GPU). Dynamic routing manages the gateway's routes + live via the admin API, so the upstream host no longer has to be predictable + from the catalog. That frees us to give each deployment its **own** service, + so N dedicated deployments of one model become N containers on N GPUs. The + suffix is the deployment id's hex tail, keeping the name short and DNS-safe. + """ + tail = deployment_id.rsplit('-', 1)[-1][:8] or 'x' + return f'{vllm_service_name_for(served)}-{_dns_slug(tail)}' + + +def vllm_service_name(deployment: Deployment, *, unique: bool = False) -> str: + """Compose service name for a vLLM deployment (see :func:`vllm_service_name_for`). + + Default (``unique=False``, static-superset mode): deterministic from the + served model name only — *no* deployment-id suffix — so it matches the + gateway's pre-rendered route for that endpoint. Trade-off: two + *simultaneously desired* deployments that share a served name collide on this + name; under the static gateway the catalog endpoint is the unit, so that case + (including same-model ``--dedicated``) is unsupported. + + ``unique=True`` (dynamic-routing mode): append the deployment-id tail + (:func:`_unique_vllm_service_name`) so same-model dedicated deployments get + distinct containers/GPUs; the admin-API route table addresses each by name. + + Either way the container carries the ``infer-stack.deployment`` label. + :meth:`ComposeBackend.residency` correlates containers to deployments by that + label, so the choice of suffix does not affect it. (The lenient + :meth:`ComposeBackend.observe` still maps service names through the render + sidecar; it is for reporting, not for decisions that touch a GPU.) + """ + served = served_name(deployment) + if unique: + return _unique_vllm_service_name(served, deployment.id) + return vllm_service_name_for(served) + + +def ollama_service_name_for(host: str) -> str: + """Deterministic service name for an Ollama daemon: ``ollama-``. + + One daemon per host (Ollama coalesces tags onto it), so the host is the + stable key — matching :func:`vllm_service_name_for`'s role for vLLM so the + gateway's static route table addresses it regardless of which tags are live. + """ + return f'ollama-{_dns_slug(host)}' + + +def ollama_service_name(deployment: Deployment) -> str: + host = deployment.spec.get('host') or deployment.id + return ollama_service_name_for(host) diff --git a/infer_stack/leasing/residency.py b/infer_stack/leasing/residency.py index 1dadb691..bc5e9446 100644 --- a/infer_stack/leasing/residency.py +++ b/infer_stack/leasing/residency.py @@ -51,6 +51,8 @@ #: Labels every rendered service carries: its service name, and a behavioural #: fingerprint that changes only when the service's behaviour does. SERVICE_LABEL = 'infer-stack.service' +#: Which engine a service runs (``vllm``, ``ollama``, ``litellm``...). +ENGINE_LABEL = 'infer-stack.engine' FINGERPRINT_LABEL = 'infer-stack.fingerprint' #: Labels Docker Compose puts on every container of a project. COMPOSE_PROJECT_LABEL = 'com.docker.compose.project' diff --git a/tests/test_leasing_compose.py b/tests/test_leasing_compose.py index b7426f59..232c7537 100644 --- a/tests/test_leasing_compose.py +++ b/tests/test_leasing_compose.py @@ -929,7 +929,7 @@ def test_litellm_config_hash_label_tracks_model_list(tmp_path): (spec unchanged), so the new alias never became routable. Stamping the config hash onto a label makes converge recreate litellm on a config change. """ - from infer_stack.leasing.compose import CONFIG_HASH_LABEL + from infer_stack.leasing.gateway import CONFIG_HASH_LABEL def label(deployment): rc = render_compose( diff --git a/tests/test_leasing_dynamic_routing.py b/tests/test_leasing_dynamic_routing.py index 4235143c..036aa610 100644 --- a/tests/test_leasing_dynamic_routing.py +++ b/tests/test_leasing_dynamic_routing.py @@ -23,12 +23,12 @@ from infer_stack.hardware import simulate_inventory from infer_stack.leasing import ComposeBackend, render_compose -from infer_stack.leasing.compose import ( +from infer_stack.leasing.compose import vllm_service_name +from infer_stack.leasing.gateway import ( POSTGRES_SERVICE, ROUTE_ID_PREFIX, _litellm_routes, _route_id, - vllm_service_name, ) from infer_stack.leasing.models import Deployment, DeploymentState @@ -517,7 +517,7 @@ def test_apply_returns_true_once_routes_verify(tmp_path): def test_apply_uses_the_short_deadline_when_the_gateway_was_already_up(tmp_path): - from infer_stack.leasing.compose import ( + from infer_stack.leasing.gateway import ( ROUTE_RECONCILE_BOOTSTRAP_S, ROUTE_RECONCILE_STEADY_S, ) diff --git a/tests/test_leasing_route_registry.py b/tests/test_leasing_route_registry.py index a911243f..b29d8574 100644 --- a/tests/test_leasing_route_registry.py +++ b/tests/test_leasing_route_registry.py @@ -21,7 +21,7 @@ from infer_stack.hardware import simulate_inventory from infer_stack.leasing import Catalog, ComposeBackend, render_compose -from infer_stack.leasing.compose import ( +from infer_stack.leasing.gateway import ( CONFIG_HASH_LABEL, LITELLM_CONFIG_FILENAME, LITELLM_REGISTRY_FILENAME, diff --git a/tests/test_leasing_secrets.py b/tests/test_leasing_secrets.py index e8d1dc3b..08a2f2d8 100644 --- a/tests/test_leasing_secrets.py +++ b/tests/test_leasing_secrets.py @@ -14,8 +14,11 @@ from infer_stack.env_utils import parse_env_file from infer_stack.hardware import simulate_inventory from infer_stack.leasing import Controller, Ledger, SqliteStore -from infer_stack.leasing.compose import ( - API_KEY_ENV, SALT_KEY_ENV, ComposeBackend, set_master_key, +from infer_stack.leasing.compose import ComposeBackend +from infer_stack.leasing.gateway import ( + API_KEY_ENV, + SALT_KEY_ENV, + set_master_key, ) from infer_stack.leasing.profile import ProfileMismatch from infer_stack.leasing.residency import FINGERPRINT_LABEL From ba93ad952c22eb711a083ac6fc3ab45bbcfdfe50 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 14:22:33 -0400 Subject: [PATCH 08/34] G-C: a Gateway class owns the front door's state The gateway's methods move off ComposeBackend into leasing.gateway.Gateway: keys (master_key, rotate, restore_env, gateway_accepts, db_password), the route registry (load, merge, save), dynamic-route reconciliation (_reconcile_routes, _list_managed_routes, _post_route), and access(). ComposeBackend owns one (`backend.gateway`), keeps thin public delegations (master_key, access, litellm_port, ...), and stores the settings the gateway reads (litellm, ui, dynamic_routing, reverse proxy, ports, http, clock, sleep) on it through properties, so there is one copy of each. The registry merge no longer reads backend state: each backend supplies its rows (compose from its catalog and deployments; kubeai persists its upstream rows with merge_route_registry, the `routes seed` path), and the gateway merges, persists and renders them. The upstream_routes side-channel is gone. compose.py 2818 -> 2483 lines. Suite 898 passed; ty and flake8 clean. Live on the guest: a gateway-only compose apply answered 200 with its key, and `secrets rotate` accepted the new key and rejected the old. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- infer_stack/backends/kubeai.py | 11 +- infer_stack/cli/commands_leasing.py | 14 +- infer_stack/leasing/compose.py | 559 ++++++-------------------- infer_stack/leasing/gateway.py | 476 +++++++++++++++++++++- tests/test_leasing_admission.py | 4 +- tests/test_leasing_dynamic_routing.py | 24 +- tests/test_leasing_profile.py | 2 +- tests/test_leasing_secrets.py | 2 +- 8 files changed, 616 insertions(+), 476 deletions(-) diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index f57e2559..2385bd10 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -381,10 +381,12 @@ def _render_gateway(self, rendered: RenderedModels) -> None: from ..leasing.gateway import UPSTREAM_ROUTE base = self._upstream_url() - self.gateway.upstream_routes = { + # This backend's route rows, persisted like any other (the registry is + # append-only, so a released Model stays routable, as on compose). + self.gateway.merge_route_registry({ alias: {'engine': UPSTREAM_ROUTE, 'served': name, 'api_base': base} for alias, name in rendered.request_names.items() - } + }) # Gateway only: no engines on this host, so nothing to place. self.gateway.converge([], apply=False) @@ -613,8 +615,9 @@ def probe_ready(self, deployment: Deployment, endpoint: str) -> Readiness: if self.gateway is not None: # Ready means ready the way a client sees it: the alias, through # the gateway, with its key. - base, model = f'{self.gateway._gateway_base()}/v1', endpoint - headers = self.gateway._auth_headers() + front = self.gateway.gateway # the runner's leasing.gateway.Gateway + base, model = f'{front._gateway_base()}/v1', endpoint + headers = front._auth_headers() else: base, model = self.base_url, model_name_for(_served_name(deployment)) headers = None diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index a97f64c2..e5bbc64c 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -2384,7 +2384,7 @@ def main(cls, argv=True, **kwargs): 'the gateway out of its database' ) if key == 'LITELLM_MASTER_KEY': - from ..leasing.compose import set_master_key + from ..leasing.gateway import set_master_key try: # Pins the salt first, so DB-stored routes stay readable. set_master_key(env_path, value) @@ -2485,7 +2485,7 @@ def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) controller = _open_controller(config) backend = _require_compose_backend(controller) - registry = backend._load_route_registry() + registry = backend.gateway._load_route_registry() entries = registry.get('entries', {}) live = _live_endpoints(controller) @@ -2558,7 +2558,7 @@ class RoutesPruneCLI(_ApprovalMixin): def main(cls, argv=True, **kwargs): from ..diff_prompt import confirm_writes from ..leasing.backend import ConvergeAborted - from ..leasing.compose import ( + from ..leasing.gateway import ( LITELLM_REGISTRY_VERSION, _dump_route_registry, _registry_incoming_from_catalog, @@ -2578,7 +2578,7 @@ def prune_plan() -> tuple[dict, dict, list[str]]: if backend.catalog is not None: keep.update(_registry_incoming_from_catalog(backend.catalog)) keep.update(_registry_incoming_from_deployments(desired, plan.assignments)) - current = backend._load_route_registry().get('entries', {}) + current = backend.gateway._load_route_registry().get('entries', {}) return current, keep, sorted(set(current) - set(keep)) # Preview outside the lock (the prompt must not hold it); the change @@ -2595,7 +2595,7 @@ def prune_plan() -> tuple[dict, dict, list[str]]: for name in dropped: print(f' - {name}') ok = confirm_writes( - {backend._registry_file: _dump_route_registry(pruned)}, + {backend.gateway._registry_file: _dump_route_registry(pruned)}, assume_yes=False, title='infer-stack routes prune', ) @@ -2613,7 +2613,7 @@ def change(): entries = {k: v for k, v in current.items() if k not in drop} with backend._converge_lock(): backend._atomic_write( - backend._registry_file, + backend.gateway._registry_file, _dump_route_registry( {'version': LITELLM_REGISTRY_VERSION, 'entries': entries}), ) @@ -2686,7 +2686,7 @@ def main(cls, argv=True, **kwargs): ) def change(): - before = set(backend._load_route_registry().get('entries', {})) + before = set(backend.gateway._load_route_registry().get('entries', {})) backend.merge_route_registry(incoming) return sorted(set(incoming) - before) diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index b012a327..c3fdac03 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -48,26 +48,21 @@ import yaml from ..config import DEFAULT_PORTS, PINNED_IMAGES, default_state_paths -from ..env_utils import ensure_secret, parse_env_file, write_env_file +from ..env_utils import parse_env_file from ..probe import openai_ready from ..profile_runtime import simulator_args, vllm_args from .backend import ConvergeScaffold, Readiness from .gateway import ( + Gateway, API_KEY_ENV, - DB_PASSWORD_ENV, - _dump_route_registry, LITELLM_CONFIG_FILENAME, LITELLM_CONTAINER_PORT, _litellm_model_list, _litellm_model_list_from_catalog, _litellm_model_list_from_registry, - LITELLM_REGISTRY_FILENAME, - LITELLM_REGISTRY_VERSION, _litellm_routes, - LITELLM_ROUTES_FILENAME, _litellm_service, LITELLM_SERVICE, - _merge_route_registry, _nginx_conf, NGINX_CONFIG_FILENAME, _nginx_service, @@ -78,12 +73,9 @@ POSTGRES_SERVICE, _registry_incoming_from_catalog, _registry_incoming_from_deployments, - ROUTE_ID_PREFIX, ROUTE_RECONCILE_BOOTSTRAP_S, ROUTE_RECONCILE_STEADY_S, SALT_KEY_ENV, - _seed_registry_from_litellm_config, - set_master_key, ) from .launch import env_string, fill, translate_legacy from .models import Deployment, is_reservation, served_name @@ -1101,15 +1093,19 @@ def __init__( if http is None: import requests http = requests - self.http = http + # The front door: gateway settings, keys, routes. Created before the + # settings below, which are stored on it. + self.gateway = Gateway( + self.state_dir, + ports={**DEFAULT_PORTS, **(ports or {})}, + litellm=litellm, ui=ui, reverse_proxy=reverse_proxy, + reverse_proxy_port=reverse_proxy_port, dynamic_routing=dynamic_routing, + http=http, sleep=sleep, clock=clock, + ) # Called with a one-line message during long steps (image pulls); the # TUI sets it. Always also logged. self.progress: Callable[[str], None] | None = None - # alias -> UPSTREAM_ROUTE row, set by a backend that uses this one only - # as its gateway (the kubeai backend). Merged into the route registry. - self.upstream_routes: dict[str, dict[str, Any]] = {} self.images = {**PINNED_IMAGES, **(images or {})} - self.ports = {**DEFAULT_PORTS, **(ports or {})} # Merge over the defaults (not replace) so a caller-supplied partial # state dict — tests, embedders — still resolves every cache-mount key. self.state = {**default_state_paths(), **(state or {})} @@ -1117,10 +1113,6 @@ def __init__( self.reserved = tuple(reserved) self.project = project self.skip_display = skip_display - self.litellm = litellm - self.ui = ui - self.reverse_proxy = reverse_proxy - self.reverse_proxy_port = reverse_proxy_port self.reverse_proxy_config = reverse_proxy_config # Set by use_profile(): the published proxy config content. self._profile_proxy_text: str | None = None @@ -1136,9 +1128,6 @@ def __init__( # against a Postgres-backed model store, instead of a static config file. # Gives each deployment its own upstream (so same-model --dedicated # deployments land on distinct GPUs) with no gateway recreation/blip. - self.dynamic_routing = dynamic_routing - self._sleep = sleep - self._clock = clock self.last_errors: list[str] = [] self.last_unplaced: set[str] = set() # desired deployment ids placement skipped self.last_assignments: dict[str, list[int]] = {} # deployment id -> GPU ids @@ -1178,104 +1167,112 @@ def inventory(self, value: dict[str, Any] | None) -> None: def compose_file(self) -> Path: return self.state_dir / COMPOSE_FILENAME + # -- the front door (leasing.gateway) ------------------------------------ + # Settings the gateway owns. Kept as attributes of the backend because + # profiles and callers set them here; they are stored in one place. + @property - def _state_file(self) -> Path: - return self.state_dir / STATE_FILENAME + def http(self) -> Any: + return self.gateway.http + + @http.setter + def http(self, value: Any) -> None: + self.gateway.http = value @property - def litellm_port(self) -> int: - return self.ports.get('litellm', DEFAULT_PORTS['litellm']) + def _clock(self) -> Callable[[], float]: + return self.gateway._clock + + @_clock.setter + def _clock(self, value: Callable[[], float]) -> None: + self.gateway._clock = value @property - def ui_port(self) -> int: - return self.ports.get('open_webui', DEFAULT_PORTS['open_webui']) + def _sleep(self) -> Callable[[float], None]: + return self.gateway._sleep + + @_sleep.setter + def _sleep(self, value: Callable[[float], None]) -> None: + self.gateway._sleep = value @property - def _env_path(self) -> Path: - return self.state_dir / '.env' + def litellm(self) -> bool: + return self.gateway.litellm + + @litellm.setter + def litellm(self, value: bool) -> None: + self.gateway.litellm = value @property - def _routes_file(self) -> Path: - return self.state_dir / LITELLM_ROUTES_FILENAME + def ui(self) -> bool: + return self.gateway.ui + + @ui.setter + def ui(self, value: bool) -> None: + self.gateway.ui = value @property - def _registry_file(self) -> Path: - return self.state_dir / LITELLM_REGISTRY_FILENAME + def dynamic_routing(self) -> bool: + return self.gateway.dynamic_routing - def master_key(self) -> str: - """The managed LiteLLM master key. + @dynamic_routing.setter + def dynamic_routing(self, value: bool) -> None: + self.gateway.dynamic_routing = value - infer-stack manages this secret in the state dir's ``.env``: reused if - already present (you may pin your own ``sk-`` key there), otherwise - generated and persisted. The caller doesn't need to invent or export it - — it is baked into the LiteLLM service, used by the readiness probe, and - shipped in the env-file descriptor (``infer-stack env KEY`` prints it). - """ - existing = parse_env_file(self._env_path) - key = ensure_secret(existing, API_KEY_ENV, prefix='sk-') - if key != existing.get(API_KEY_ENV): - write_env_file(self._env_path, {API_KEY_ENV: key}) - return key + @property + def reverse_proxy(self) -> bool: + return self.gateway.reverse_proxy - def rotate_master_key(self) -> dict[str, str | None]: - """Write a fresh master key to the ``.env``; return the values it replaced. + @reverse_proxy.setter + def reverse_proxy(self, value: bool) -> None: + self.gateway.reverse_proxy = value - Only the file: the gateway and Open WebUI pick it up when the next apply - recreates them (their fingerprints hash the key). The caller holds the - publication lock and passes the return value to - :meth:`restore_env` if that apply does not happen. - """ - before = parse_env_file(self._env_path) - set_master_key(self._env_path, ensure_secret({}, API_KEY_ENV, prefix='sk-')) - return {k: before.get(k) for k in (API_KEY_ENV, SALT_KEY_ENV)} + @property + def reverse_proxy_port(self) -> int: + return self.gateway.reverse_proxy_port - def restore_env(self, values: dict[str, str | None]) -> None: - """Put back what :meth:`rotate_master_key` replaced.""" - from ..env_utils import remove_env_keys + @reverse_proxy_port.setter + def reverse_proxy_port(self, value: int) -> None: + self.gateway.reverse_proxy_port = value + + @property + def ports(self) -> dict[str, int]: + return self.gateway.ports + + @ports.setter + def ports(self, value: dict[str, int]) -> None: + self.gateway.ports = value - write_env_file(self._env_path, {k: v for k, v in values.items() if v is not None}) - remove_env_keys(self._env_path, [k for k, v in values.items() if v is None]) + @property + def litellm_port(self) -> int: + return self.gateway.litellm_port + + @property + def ui_port(self) -> int: + return self.gateway.ui_port + + def master_key(self) -> str: + return self.gateway.master_key() + + def rotate_master_key(self) -> dict[str, str | None]: + return self.gateway.rotate_master_key() + + def restore_env(self, values: dict[str, str | None]) -> None: + self.gateway.restore_env(values) def gateway_accepts(self, key: str, *, wait: float = 0.0) -> bool | None: - """Does the gateway accept ``key``? ``None`` if it never answered. + return self.gateway.gateway_accepts(key, wait=wait) - Polls ``/v1/models`` for up to ``wait`` seconds while the gateway is - unreachable or still starting. A definite no is 401/403, or the 400 - LiteLLM actually answers a wrong key with (measured on the pinned image). - """ - deadline = self._clock() + wait - while True: - try: - resp = self.http.get( - f'{self._gateway_base()}/v1/models', - headers={'Authorization': f'Bearer {key}'}, timeout=10.0, - ) - status = getattr(resp, 'status_code', 0) - except Exception: # noqa: BLE001 - not up yet - status = 0 - if status == 200: - return True - if status in (400, 401, 403): - return False - if self._clock() >= deadline: - return None - self._sleep(2.0) - - def db_password(self) -> str: - """The managed Postgres password for LiteLLM's model store. - - Same managed-secret pattern as :meth:`master_key`: reused if already - present in the state-dir ``.env`` (you may pin your own), else generated - and persisted. ``docker compose --env-file`` interpolates it into the - postgres + litellm services, so it never appears literally in the YAML. - Only used when ``dynamic_routing`` is on. ``token_urlsafe`` output is safe - inside the ``postgresql://`` URL (no ``@ : /`` characters). - """ - existing = parse_env_file(self._env_path) - pw = ensure_secret(existing, DB_PASSWORD_ENV) - if pw != existing.get(DB_PASSWORD_ENV): - write_env_file(self._env_path, {DB_PASSWORD_ENV: pw}) - return pw + def merge_route_registry(self, incoming: dict[str, dict[str, Any]]) -> dict[str, Any]: + return self.gateway.merge_route_registry(incoming) + + def access(self, endpoints: list[str]) -> dict[str, Any] | None: + """Where a client reaches these endpoints: the front door.""" + return self.gateway.access(endpoints) + + @property + def _state_file(self) -> Path: + return self.state_dir / STATE_FILENAME def _ensure_state_dir(self) -> None: """Raise a legible error if the state dir could not be created. @@ -1421,8 +1418,8 @@ def _compose(self, args: list[str]) -> str: # directory* (wherever infer-stack was invoked), not the state dir, so we # point it explicitly. Only when present: a litellm-less stack never # writes one, and a missing --env-file path is a hard error. - if self._env_path.exists(): - cmd += ['--env-file', str(self._env_path)] + if self.gateway._env_path.exists(): + cmd += ['--env-file', str(self.gateway._env_path)] cmd += ['-p', self.project, '-f', str(self.compose_file)] return self.run([*cmd, *args]) @@ -1478,7 +1475,7 @@ def _render_documents(self, desired: list[Deployment], placement) -> dict[str, A if self.litellm and self.dynamic_routing: # The DB secret must exist before rendering, so docker compose # --env-file can interpolate ${LITELLM_DB_PASSWORD} at apply time. - self.db_password() + self.gateway.db_password() route_registry = None if self.litellm and not self.dynamic_routing: # Unconditional in static-superset mode: `self.catalog` may be None; @@ -1490,7 +1487,7 @@ def _render_documents(self, desired: list[Deployment], placement) -> dict[str, A desired, plan.assignments, images=self.images, ports=self.ports, state=self.state, litellm=self.litellm, litellm_port=self.litellm_port, litellm_master_key=self.master_key() if self.litellm else None, - litellm_salt_key=SALT_KEY_ENV in parse_env_file(self._env_path), + litellm_salt_key=SALT_KEY_ENV in parse_env_file(self.gateway._env_path), ui=self.ui, ui_port=self.ui_port, reverse_proxy=self.reverse_proxy, reverse_proxy_port=self.reverse_proxy_port, reverse_proxy_config=self.reverse_proxy_config, aux_dir=self.state_dir, @@ -1516,9 +1513,9 @@ def _render_documents(self, desired: list[Deployment], placement) -> dict[str, A if rendered.nginx_config is not None: planned[self.state_dir / NGINX_CONFIG_FILENAME] = rendered.nginx_config if rendered.litellm_routes is not None: - planned[self._routes_file] = json.dumps(rendered.litellm_routes, indent=2) + planned[self.gateway._routes_file] = json.dumps(rendered.litellm_routes, indent=2) fingerprints = stamp_fingerprints( - rendered.compose, files=planned, env_file=self._env_path, + rendered.compose, files=planned, env_file=self.gateway._env_path, ) planned[self.compose_file] = yaml.safe_dump(rendered.compose, sort_keys=False) return {'plan': plan, 'rendered': rendered, 'planned': planned, @@ -1702,132 +1699,27 @@ def deployment_logs(self, deployment: Deployment, *, tail: int = 400) -> str: except Exception: return '' - def _load_route_registry(self) -> dict[str, Any]: - """Read the route registry, tolerantly (fail-open — a broken registry - must never block a converge). - - Missing file → seed from the live ``litellm_config.yaml`` if present - (upgrade migration, §6), else an empty registry. An *unknown* schema - version whose ``entries`` still parses as a name→row map is preserved - as-is (render what's understood, warn, do NOT rewrite) rather than - reseeded, so a binary rollback doesn't discard the accumulated union. - Only a structurally unusable file (not a map / garbage JSON) falls back - to seeding.""" - from .._log import logger - - if not self._registry_file.exists(): - config = self.state_dir / LITELLM_CONFIG_FILENAME - if config.exists(): - seeded, warnings = _seed_registry_from_litellm_config( - config.read_text() - ) - for w in warnings: - logger.warning(' route registry: {}', w) - logger.info( - ' route registry: seeded {} vLLM route(s) from {}', - len(seeded['entries']), config.name, - ) - return seeded - return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}} - try: - data = json.loads(self._registry_file.read_text()) - except (OSError, json.JSONDecodeError) as exc: - logger.warning( - ' route registry: {} is unreadable ({}); rebuilding from seed', - self._registry_file.name, exc, - ) - data = None - if not isinstance(data, dict) or not isinstance( - data.get('entries'), dict - ): - config = self.state_dir / LITELLM_CONFIG_FILENAME - if config.exists(): - seeded, warnings = _seed_registry_from_litellm_config( - config.read_text() - ) - for w in warnings: - logger.warning(' route registry: {}', w) - return seeded - return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}} - version = data.get('version') - if version != LITELLM_REGISTRY_VERSION: - logger.warning( - ' route registry: unknown schema version {!r} in {} — ' - 'rendering as-is without rewrite (fields this renderer does ' - 'not understand are ignored)', - version, self._registry_file.name, - ) - return data - def _merged_route_registry( self, desired: list[Deployment], assignments: dict[str, list[int]] ) -> dict[str, Any]: - """The route registry merged with the catalog and ``desired``, in memory.""" - from .._log import logger + """The route registry merged with this backend's rows, in memory. - existing = self._load_route_registry() + The rows are this backend's to supply: every catalog endpoint, and every + placed deployment (``desired`` spans all runbooks via the shared ledger, + so a live cross-runbook deployment stays routable, and past release). + """ incoming: dict[str, dict[str, Any]] = {} if self.catalog is not None: incoming.update(_registry_incoming_from_catalog(self.catalog)) - # `desired` spans all runbooks via the shared ledger, so this keeps every - # live cross-runbook deployment routable (and, via persistence, routable - # past release). incoming.update(_registry_incoming_from_deployments(desired, assignments)) - # Routes to servers another backend runs (see UPSTREAM_ROUTE). - incoming.update(self.upstream_routes) - merged, warnings = _merge_route_registry(existing, incoming) - for w in warnings: - logger.warning(' route registry: {}', w) - return merged - - def _save_route_registry(self, merged: dict[str, Any]) -> None: - """Persist a merged registry if it changed (under the converge flock).""" - from .._log import logger - - existing = self._load_route_registry() - if merged == existing: - return - prior = existing.get('entries', {}) if isinstance(existing, dict) else {} - added = sorted(set(merged['entries']) - set(prior)) - updated = sorted( - k for k in merged['entries'] if k in prior and merged['entries'][k] != prior[k] - ) - if added: - logger.info(' route registry: +{} route(s): {}', len(added), ', '.join(added)) - if updated: - logger.info(' route registry: updated route(s): {}', ', '.join(updated)) - self._atomic_write(self._registry_file, _dump_route_registry(merged)) + return self.gateway.merged_route_registry(incoming) def _update_route_registry( self, desired: list[Deployment], assignments: dict[str, list[int]] ) -> dict[str, Any]: """Merge and persist the route registry; return it (kept for callers).""" merged = self._merged_route_registry(desired, assignments) - self._save_route_registry(merged) - return merged - - def merge_route_registry( - self, incoming: dict[str, dict[str, Any]] - ) -> dict[str, Any]: - """Public write path for out-of-converge registry seeds (``routes seed``). - - Takes the converge flock, read-merge-writes the registry, and returns the - merged dict. ``converge`` only ever merges the invoking process's own - catalog, so a standalone caller (seeding a *sibling* runbook's catalog) - needs this to fold extra rows in before the follow-up ``reconcile`` - renders+applies. The flock here and the one the subsequent converge takes - are sequential acquisitions, not nested — no reentrancy concern.""" - from .._log import logger - - with self._converge_lock(): - existing = self._load_route_registry() - merged, warnings = _merge_route_registry(existing, incoming) - for w in warnings: - logger.warning(' route registry: {}', w) - if merged != existing: - self._atomic_write( - self._registry_file, _dump_route_registry(merged) - ) + self.gateway._save_route_registry(merged) return merged def converge(self, desired: list[Deployment], *, apply: bool = True, placement=None): @@ -1883,7 +1775,7 @@ def converge(self, desired: list[Deployment], *, apply: bool = True, placement=N if docs['addresses'] is not None and self.on_addresses is not None: self.on_addresses(docs['addresses']) if docs['route_registry'] is not None: - self._save_route_registry(docs['route_registry']) + self.gateway._save_route_registry(docs['route_registry']) for path, text in planned.items(): if path != self.compose_file: self._atomic_write(path, text) @@ -1960,7 +1852,7 @@ def apply(self) -> bool: networks=doc.get('networks') or {}, ) if dynamic and 'litellm' in services: - return self._reconcile_routes( + return self.gateway._reconcile_routes( deadline_s=ROUTE_RECONCILE_STEADY_S if 'litellm' in outcome.kept_services else ROUTE_RECONCILE_BOOTSTRAP_S, ) @@ -2264,208 +2156,6 @@ def _service_running(self, service: str) -> bool: # -- dynamic routing (admin API) -------------------------------------- - def _gateway_base(self) -> str: - return f'http://127.0.0.1:{self.litellm_port}' - - def _auth_headers(self) -> dict[str, str]: - return {'Authorization': f'Bearer {self.master_key()}'} - - def _desired_routes(self) -> list[dict[str, Any]]: - """The rendered desired route set (litellm_routes.json), or empty.""" - try: - data = json.loads(self._routes_file.read_text()) - except (FileNotFoundError, json.JSONDecodeError): - return [] - return data if isinstance(data, list) else [] - - def _reconcile_routes( - self, *, deadline_s: float = ROUTE_RECONCILE_BOOTSTRAP_S, delay: float = 2.0, - ) -> bool: - """Make the live gateway's managed routes match the rendered route set. - - The render half wrote the desired routes (one per live deployment× - endpoint) to ``litellm_routes.json``; this is the apply half. List the - gateway's current models, add the missing routes and delete the ones no - longer desired -- through the admin API, with **no** container restart -- - then list again to verify both ids and routing semantics, re-diffing and - retrying failed calls until the table matches or the budget runs out. - - Properties this relies on: - - * **Idempotent.** A redundant apply re-diffs to the same set and does - nothing. - * **Drift-healing.** Routes lost to a gateway/DB restart reappear in the - diff and are re-added; stale routes from a prior run (still in the DB) - are deleted because they're no longer desired. - * **Co-existence.** Only routes infer-stack created (id prefix ``isr-``) - are ever deleted, so a model added by hand through the UI/API is left - alone. - - **Bounded and reported.** Everything -- listing retries while the gateway - starts, every POST, and the final verification -- shares one wall-clock - budget, ``deadline_s``. A retry count alone would not bound it: a listing - can take 10 s and a POST 30 s. Returns ``True`` only when the verified - managed route set equals the desired set; any failure is logged and - returns ``False`` rather than raising, so the caller decides whether an - unverified route set blocks anything. - """ - from .._log import logger - - deadline = self._clock() + max(0.0, deadline_s) - desired = { - r['model_info']['id']: r - for r in self._desired_routes() - if isinstance(r.get('model_info'), dict) and r['model_info'].get('id') - } - desired_semantics = {rid: self._route_semantics(route) - for rid, route in desired.items()} - rounds = 0 - while True: - current = self._list_managed_routes(deadline=deadline, delay=delay) - if current is None: - logger.warning( - 'dynamic routing: route set not reconciled and verified within ' - '{:g}s; leaving it for the next apply', deadline_s, - ) - return False - mismatched = sorted( - rid for rid in desired.keys() & current.keys() - if desired_semantics[rid] != current[rid] - ) - to_add_ids = sorted((desired.keys() - current.keys()) | set(mismatched)) - to_delete = sorted((current.keys() - desired.keys()) | set(mismatched)) - to_add = [desired[rid] for rid in to_add_ids] - if not (to_add or to_delete): - return True # this listing is the verification - if rounds: - logger.info('dynamic routing: route set still differs; retrying') - rounds += 1 - ok = True - # A same-id semantic drift must be removed before it can be re-added; - # model/new is not an update API on every LiteLLM release. - for rid in to_delete: - # ok_if_missing: with a shared gateway, another converge may have - # deleted this route already; "not found in db" means the desired - # end-state (route gone) is reached, so don't treat it as an error. - ok &= self._post_route( - '/model/delete', {'id': rid}, rid, ok_if_missing=True, - deadline=deadline, - ) - for route in to_add: - ok &= self._post_route( - '/model/new', route, route.get('model_name'), deadline=deadline, - ) - logger.info( - 'dynamic routing: +{} route(s), -{} route(s), ~{} replacement(s) ' - '(now {} desired)', - len(to_add), len(to_delete), len(mismatched), len(desired), - ) - if not ok: - # A transient admin-API failure: spend the rest of the budget - # re-diffing rather than giving up with most of it unused. - if deadline - self._clock() <= delay: - return False - self._sleep(delay) - - @staticmethod - def _route_semantics(route: dict[str, Any]) -> dict[str, Any]: - """Observable route fields infer-stack owns and must verify. - - LiteLLM's model-info response contains additional database/runtime fields - and may redact credentials. The public alias, upstream model, and - upstream base URL are the routing semantics infer-stack can both set and - reliably observe. A matching managed id with different values here is - drift and is replaced, not accepted as healthy. - """ - params = route.get('litellm_params') or {} - return { - 'model_name': route.get('model_name'), - 'model': params.get('model'), - 'api_base': params.get('api_base'), - } - - def _list_managed_routes( - self, *, deadline: float, delay: float - ) -> dict[str, dict[str, Any]] | None: - """Observable semantics of infer-stack-managed gateway routes. - - Retries while the gateway is unreachable, until ``deadline`` (a value of - ``self._clock``). Each request's own timeout is capped by the time left. - Returns ``None`` if no listing succeeded in time. - """ - while True: - remaining = deadline - self._clock() - if remaining <= 0: - return None - resp = None - try: - resp = self.http.get( - f'{self._gateway_base()}/v1/model/info', - headers=self._auth_headers(), - timeout=min(10.0, remaining), - ) - except Exception: # noqa: BLE001 - the gateway may still be starting - resp = None - if resp is not None and getattr(resp, 'status_code', 0) == 200: - routes: dict[str, dict[str, Any]] = {} - for m in (resp.json().get('data') or []): - rid = (m.get('model_info') or {}).get('id') - if isinstance(rid, str) and rid.startswith(ROUTE_ID_PREFIX): - routes[rid] = self._route_semantics(m) - return routes - if deadline - self._clock() <= delay: - return None - self._sleep(delay) - - def _post_route( - self, - path: str, - payload: dict[str, Any], - label: Any, - *, - ok_if_missing: bool = False, - deadline: float | None = None, - ) -> bool: - """POST one admin-API call (``/model/new`` or ``/model/delete``). - - Returns whether it reached its desired end state. A failure is logged, - not raised, so the remaining calls still run. ``ok_if_missing`` accepts a - "model not found" response (a delete whose target is already gone). - The request timeout is capped by the time left before ``deadline``. - """ - from .._log import logger - - timeout = 30.0 - if deadline is not None: - remaining = deadline - self._clock() - if remaining <= 0: - logger.warning( - 'dynamic routing: POST {} {} skipped: route deadline passed', - path, label, - ) - return False - timeout = min(timeout, remaining) - try: - resp = self.http.post( - f'{self._gateway_base()}{path}', - headers=self._auth_headers(), - json=payload, - timeout=timeout, - ) - except Exception as ex: # noqa: BLE001 - one bad call must not abort apply - logger.warning('dynamic routing: POST {} {} error: {}', path, label, ex) - return False - if getattr(resp, 'status_code', 0) >= 300: - body = str(getattr(resp, 'text', '')) - if ok_if_missing and 'not found' in body.lower(): - return True - logger.warning( - 'dynamic routing: POST {} {} -> {} {}', - path, label, resp.status_code, body[:200], - ) - return False - return True - # -- published profile (see leasing/profile.py) --------------------------- REVERSE_PROXY_SNAPSHOT = 'reverse-proxy.conf' @@ -2677,31 +2367,6 @@ def observe(self) -> set[str]: services = self._load_sidecar().get('services', {}) return {services[name] for name in running if name in services} - def access(self, endpoints: list[str]) -> dict[str, Any] | None: - """Where a client reaches these endpoints, for the env-file descriptor. - - With the LiteLLM front door, that is one ``base_url`` and the request - model name is the endpoint alias itself. With LiteLLM off there is no - single base URL, but a managed Open WebUI (if on) is still a useful - access point, so report just its URL rather than ``None``. - """ - if not self.litellm: - if self.ui: - return {'ui_url': f'http://127.0.0.1:{self.ui_port}'} - return None - info: dict[str, Any] = { - 'base_url': f'http://127.0.0.1:{self.litellm_port}/v1', - 'api_key_env': API_KEY_ENV, - 'api_key': self.master_key(), - 'request_names': {ep: ep for ep in endpoints}, - } - if self.ui: - info['ui_url'] = f'http://127.0.0.1:{self.ui_port}' - if self.reverse_proxy: - # The unified front door: one origin, UI at / and the API at /v1. - info['proxy_url'] = f'http://127.0.0.1:{self.reverse_proxy_port}' - return info - def _ensure_ollama_tag( self, deployment: Deployment, endpoint: str ) -> str | None: diff --git a/infer_stack/leasing/gateway.py b/infer_stack/leasing/gateway.py index a15d174c..0da21a7f 100644 --- a/infer_stack/leasing/gateway.py +++ b/infer_stack/leasing/gateway.py @@ -13,13 +13,16 @@ import hashlib import json +import time from pathlib import Path -from typing import Any +from typing import Any, Callable import yaml from ..config import PINNED_IMAGES -from ..env_utils import parse_env_file, write_env_file +from ..config import DEFAULT_PORTS +from ..env_utils import ensure_secret, parse_env_file, write_env_file +from .backend import ConvergeScaffold from .models import Deployment, served_name from .naming import ( OLLAMA_CONTAINER_PORT, @@ -730,3 +733,472 @@ def set_master_key(env_path: Path, key: str) -> None: write_env_file(env_path, values) +class Gateway(ConvergeScaffold): + """The front door's state: settings, the managed keys, the route registry. + + Owned by the backend that runs it (``ComposeBackend.gateway``). The + backend supplies route rows for what it serves; this merges, persists and + renders them, reconciles dynamic routes through LiteLLM's admin API, and + keeps the ``.env`` secrets. It shares the backend's state directory (and + so its converge lock), because the gateway's files live beside the + compose project that runs it. + """ + + def __init__( + self, + state_dir: str | Path, + *, + ports: dict[str, int], + litellm: bool = True, + ui: bool = True, + reverse_proxy: bool = False, + reverse_proxy_port: int = 80, + dynamic_routing: bool = False, + http: Any = None, + sleep: Callable[[float], None] = time.sleep, + clock: Callable[[], float] = time.monotonic, + ): + self.state_dir = Path(state_dir) + self.ports = ports + self.litellm = litellm + self.ui = ui + self.reverse_proxy = reverse_proxy + self.reverse_proxy_port = reverse_proxy_port + self.dynamic_routing = dynamic_routing + if http is None: + import requests + http = requests + self.http = http + self._sleep = sleep + self._clock = clock + self.assume_yes = True + + @property + def litellm_port(self) -> int: + return self.ports.get('litellm', DEFAULT_PORTS['litellm']) + + @property + def ui_port(self) -> int: + return self.ports.get('open_webui', DEFAULT_PORTS['open_webui']) + + @property + def _env_path(self) -> Path: + return self.state_dir / '.env' + + @property + def _routes_file(self) -> Path: + return self.state_dir / LITELLM_ROUTES_FILENAME + + @property + def _registry_file(self) -> Path: + return self.state_dir / LITELLM_REGISTRY_FILENAME + + def master_key(self) -> str: + """The managed LiteLLM master key. + + infer-stack manages this secret in the state dir's ``.env``: reused if + already present (you may pin your own ``sk-`` key there), otherwise + generated and persisted. The caller doesn't need to invent or export it + — it is baked into the LiteLLM service, used by the readiness probe, and + shipped in the env-file descriptor (``infer-stack env KEY`` prints it). + """ + existing = parse_env_file(self._env_path) + key = ensure_secret(existing, API_KEY_ENV, prefix='sk-') + if key != existing.get(API_KEY_ENV): + write_env_file(self._env_path, {API_KEY_ENV: key}) + return key + + def rotate_master_key(self) -> dict[str, str | None]: + """Write a fresh master key to the ``.env``; return the values it replaced. + + Only the file: the gateway and Open WebUI pick it up when the next apply + recreates them (their fingerprints hash the key). The caller holds the + publication lock and passes the return value to + :meth:`restore_env` if that apply does not happen. + """ + before = parse_env_file(self._env_path) + set_master_key(self._env_path, ensure_secret({}, API_KEY_ENV, prefix='sk-')) + return {k: before.get(k) for k in (API_KEY_ENV, SALT_KEY_ENV)} + + def restore_env(self, values: dict[str, str | None]) -> None: + """Put back what :meth:`rotate_master_key` replaced.""" + from ..env_utils import remove_env_keys + + write_env_file(self._env_path, {k: v for k, v in values.items() if v is not None}) + remove_env_keys(self._env_path, [k for k, v in values.items() if v is None]) + + def gateway_accepts(self, key: str, *, wait: float = 0.0) -> bool | None: + """Does the gateway accept ``key``? ``None`` if it never answered. + + Polls ``/v1/models`` for up to ``wait`` seconds while the gateway is + unreachable or still starting. A definite no is 401/403, or the 400 + LiteLLM actually answers a wrong key with (measured on the pinned image). + """ + deadline = self._clock() + wait + while True: + try: + resp = self.http.get( + f'{self._gateway_base()}/v1/models', + headers={'Authorization': f'Bearer {key}'}, timeout=10.0, + ) + status = getattr(resp, 'status_code', 0) + except Exception: # noqa: BLE001 - not up yet + status = 0 + if status == 200: + return True + if status in (400, 401, 403): + return False + if self._clock() >= deadline: + return None + self._sleep(2.0) + + def db_password(self) -> str: + """The managed Postgres password for LiteLLM's model store. + + Same managed-secret pattern as :meth:`master_key`: reused if already + present in the state-dir ``.env`` (you may pin your own), else generated + and persisted. ``docker compose --env-file`` interpolates it into the + postgres + litellm services, so it never appears literally in the YAML. + Only used when ``dynamic_routing`` is on. ``token_urlsafe`` output is safe + inside the ``postgresql://`` URL (no ``@ : /`` characters). + """ + existing = parse_env_file(self._env_path) + pw = ensure_secret(existing, DB_PASSWORD_ENV) + if pw != existing.get(DB_PASSWORD_ENV): + write_env_file(self._env_path, {DB_PASSWORD_ENV: pw}) + return pw + + def _load_route_registry(self) -> dict[str, Any]: + """Read the route registry, tolerantly (fail-open — a broken registry + must never block a converge). + + Missing file → seed from the live ``litellm_config.yaml`` if present + (upgrade migration, §6), else an empty registry. An *unknown* schema + version whose ``entries`` still parses as a name→row map is preserved + as-is (render what's understood, warn, do NOT rewrite) rather than + reseeded, so a binary rollback doesn't discard the accumulated union. + Only a structurally unusable file (not a map / garbage JSON) falls back + to seeding.""" + from .._log import logger + + if not self._registry_file.exists(): + config = self.state_dir / LITELLM_CONFIG_FILENAME + if config.exists(): + seeded, warnings = _seed_registry_from_litellm_config( + config.read_text() + ) + for w in warnings: + logger.warning(' route registry: {}', w) + logger.info( + ' route registry: seeded {} vLLM route(s) from {}', + len(seeded['entries']), config.name, + ) + return seeded + return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}} + try: + data = json.loads(self._registry_file.read_text()) + except (OSError, json.JSONDecodeError) as exc: + logger.warning( + ' route registry: {} is unreadable ({}); rebuilding from seed', + self._registry_file.name, exc, + ) + data = None + if not isinstance(data, dict) or not isinstance( + data.get('entries'), dict + ): + config = self.state_dir / LITELLM_CONFIG_FILENAME + if config.exists(): + seeded, warnings = _seed_registry_from_litellm_config( + config.read_text() + ) + for w in warnings: + logger.warning(' route registry: {}', w) + return seeded + return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}} + version = data.get('version') + if version != LITELLM_REGISTRY_VERSION: + logger.warning( + ' route registry: unknown schema version {!r} in {} — ' + 'rendering as-is without rewrite (fields this renderer does ' + 'not understand are ignored)', + version, self._registry_file.name, + ) + return data + + def _save_route_registry(self, merged: dict[str, Any]) -> None: + """Persist a merged registry if it changed (under the converge flock).""" + from .._log import logger + + existing = self._load_route_registry() + if merged == existing: + return + prior = existing.get('entries', {}) if isinstance(existing, dict) else {} + added = sorted(set(merged['entries']) - set(prior)) + updated = sorted( + k for k in merged['entries'] if k in prior and merged['entries'][k] != prior[k] + ) + if added: + logger.info(' route registry: +{} route(s): {}', len(added), ', '.join(added)) + if updated: + logger.info(' route registry: updated route(s): {}', ', '.join(updated)) + self._atomic_write(self._registry_file, _dump_route_registry(merged)) + + def merge_route_registry( + self, incoming: dict[str, dict[str, Any]] + ) -> dict[str, Any]: + """Public write path for out-of-converge registry seeds (``routes seed``). + + Takes the converge flock, read-merge-writes the registry, and returns the + merged dict. ``converge`` only ever merges the invoking process's own + catalog, so a standalone caller (seeding a *sibling* runbook's catalog) + needs this to fold extra rows in before the follow-up ``reconcile`` + renders+applies. The flock here and the one the subsequent converge takes + are sequential acquisitions, not nested — no reentrancy concern.""" + from .._log import logger + + with self._converge_lock(): + existing = self._load_route_registry() + merged, warnings = _merge_route_registry(existing, incoming) + for w in warnings: + logger.warning(' route registry: {}', w) + if merged != existing: + self._atomic_write( + self._registry_file, _dump_route_registry(merged) + ) + return merged + + def _gateway_base(self) -> str: + return f'http://127.0.0.1:{self.litellm_port}' + + def _auth_headers(self) -> dict[str, str]: + return {'Authorization': f'Bearer {self.master_key()}'} + + def _desired_routes(self) -> list[dict[str, Any]]: + """The rendered desired route set (litellm_routes.json), or empty.""" + try: + data = json.loads(self._routes_file.read_text()) + except (FileNotFoundError, json.JSONDecodeError): + return [] + return data if isinstance(data, list) else [] + + def _reconcile_routes( + self, *, deadline_s: float = ROUTE_RECONCILE_BOOTSTRAP_S, delay: float = 2.0, + ) -> bool: + """Make the live gateway's managed routes match the rendered route set. + + The render half wrote the desired routes (one per live deployment× + endpoint) to ``litellm_routes.json``; this is the apply half. List the + gateway's current models, add the missing routes and delete the ones no + longer desired -- through the admin API, with **no** container restart -- + then list again to verify both ids and routing semantics, re-diffing and + retrying failed calls until the table matches or the budget runs out. + + Properties this relies on: + + * **Idempotent.** A redundant apply re-diffs to the same set and does + nothing. + * **Drift-healing.** Routes lost to a gateway/DB restart reappear in the + diff and are re-added; stale routes from a prior run (still in the DB) + are deleted because they're no longer desired. + * **Co-existence.** Only routes infer-stack created (id prefix ``isr-``) + are ever deleted, so a model added by hand through the UI/API is left + alone. + + **Bounded and reported.** Everything -- listing retries while the gateway + starts, every POST, and the final verification -- shares one wall-clock + budget, ``deadline_s``. A retry count alone would not bound it: a listing + can take 10 s and a POST 30 s. Returns ``True`` only when the verified + managed route set equals the desired set; any failure is logged and + returns ``False`` rather than raising, so the caller decides whether an + unverified route set blocks anything. + """ + from .._log import logger + + deadline = self._clock() + max(0.0, deadline_s) + desired = { + r['model_info']['id']: r + for r in self._desired_routes() + if isinstance(r.get('model_info'), dict) and r['model_info'].get('id') + } + desired_semantics = {rid: self._route_semantics(route) + for rid, route in desired.items()} + rounds = 0 + while True: + current = self._list_managed_routes(deadline=deadline, delay=delay) + if current is None: + logger.warning( + 'dynamic routing: route set not reconciled and verified within ' + '{:g}s; leaving it for the next apply', deadline_s, + ) + return False + mismatched = sorted( + rid for rid in desired.keys() & current.keys() + if desired_semantics[rid] != current[rid] + ) + to_add_ids = sorted((desired.keys() - current.keys()) | set(mismatched)) + to_delete = sorted((current.keys() - desired.keys()) | set(mismatched)) + to_add = [desired[rid] for rid in to_add_ids] + if not (to_add or to_delete): + return True # this listing is the verification + if rounds: + logger.info('dynamic routing: route set still differs; retrying') + rounds += 1 + ok = True + # A same-id semantic drift must be removed before it can be re-added; + # model/new is not an update API on every LiteLLM release. + for rid in to_delete: + # ok_if_missing: with a shared gateway, another converge may have + # deleted this route already; "not found in db" means the desired + # end-state (route gone) is reached, so don't treat it as an error. + ok &= self._post_route( + '/model/delete', {'id': rid}, rid, ok_if_missing=True, + deadline=deadline, + ) + for route in to_add: + ok &= self._post_route( + '/model/new', route, route.get('model_name'), deadline=deadline, + ) + logger.info( + 'dynamic routing: +{} route(s), -{} route(s), ~{} replacement(s) ' + '(now {} desired)', + len(to_add), len(to_delete), len(mismatched), len(desired), + ) + if not ok: + # A transient admin-API failure: spend the rest of the budget + # re-diffing rather than giving up with most of it unused. + if deadline - self._clock() <= delay: + return False + self._sleep(delay) + + @staticmethod + def _route_semantics(route: dict[str, Any]) -> dict[str, Any]: + """Observable route fields infer-stack owns and must verify. + + LiteLLM's model-info response contains additional database/runtime fields + and may redact credentials. The public alias, upstream model, and + upstream base URL are the routing semantics infer-stack can both set and + reliably observe. A matching managed id with different values here is + drift and is replaced, not accepted as healthy. + """ + params = route.get('litellm_params') or {} + return { + 'model_name': route.get('model_name'), + 'model': params.get('model'), + 'api_base': params.get('api_base'), + } + + def _list_managed_routes( + self, *, deadline: float, delay: float + ) -> dict[str, dict[str, Any]] | None: + """Observable semantics of infer-stack-managed gateway routes. + + Retries while the gateway is unreachable, until ``deadline`` (a value of + ``self._clock``). Each request's own timeout is capped by the time left. + Returns ``None`` if no listing succeeded in time. + """ + while True: + remaining = deadline - self._clock() + if remaining <= 0: + return None + resp = None + try: + resp = self.http.get( + f'{self._gateway_base()}/v1/model/info', + headers=self._auth_headers(), + timeout=min(10.0, remaining), + ) + except Exception: # noqa: BLE001 - the gateway may still be starting + resp = None + if resp is not None and getattr(resp, 'status_code', 0) == 200: + routes: dict[str, dict[str, Any]] = {} + for m in (resp.json().get('data') or []): + rid = (m.get('model_info') or {}).get('id') + if isinstance(rid, str) and rid.startswith(ROUTE_ID_PREFIX): + routes[rid] = self._route_semantics(m) + return routes + if deadline - self._clock() <= delay: + return None + self._sleep(delay) + + def _post_route( + self, + path: str, + payload: dict[str, Any], + label: Any, + *, + ok_if_missing: bool = False, + deadline: float | None = None, + ) -> bool: + """POST one admin-API call (``/model/new`` or ``/model/delete``). + + Returns whether it reached its desired end state. A failure is logged, + not raised, so the remaining calls still run. ``ok_if_missing`` accepts a + "model not found" response (a delete whose target is already gone). + The request timeout is capped by the time left before ``deadline``. + """ + from .._log import logger + + timeout = 30.0 + if deadline is not None: + remaining = deadline - self._clock() + if remaining <= 0: + logger.warning( + 'dynamic routing: POST {} {} skipped: route deadline passed', + path, label, + ) + return False + timeout = min(timeout, remaining) + try: + resp = self.http.post( + f'{self._gateway_base()}{path}', + headers=self._auth_headers(), + json=payload, + timeout=timeout, + ) + except Exception as ex: # noqa: BLE001 - one bad call must not abort apply + logger.warning('dynamic routing: POST {} {} error: {}', path, label, ex) + return False + if getattr(resp, 'status_code', 0) >= 300: + body = str(getattr(resp, 'text', '')) + if ok_if_missing and 'not found' in body.lower(): + return True + logger.warning( + 'dynamic routing: POST {} {} -> {} {}', + path, label, resp.status_code, body[:200], + ) + return False + return True + + def access(self, endpoints: list[str]) -> dict[str, Any] | None: + """Where a client reaches these endpoints, for the env-file descriptor. + + With the LiteLLM front door, that is one ``base_url`` and the request + model name is the endpoint alias itself. With LiteLLM off there is no + single base URL, but a managed Open WebUI (if on) is still a useful + access point, so report just its URL rather than ``None``. + """ + if not self.litellm: + if self.ui: + return {'ui_url': f'http://127.0.0.1:{self.ui_port}'} + return None + info: dict[str, Any] = { + 'base_url': f'http://127.0.0.1:{self.litellm_port}/v1', + 'api_key_env': API_KEY_ENV, + 'api_key': self.master_key(), + 'request_names': {ep: ep for ep in endpoints}, + } + if self.ui: + info['ui_url'] = f'http://127.0.0.1:{self.ui_port}' + if self.reverse_proxy: + # The unified front door: one origin, UI at / and the API at /v1. + info['proxy_url'] = f'http://127.0.0.1:{self.reverse_proxy_port}' + return info + + def merged_route_registry(self, incoming: dict[str, dict[str, Any]]) -> dict[str, Any]: + """The registry with ``incoming`` rows merged in, in memory (no write).""" + from .._log import logger + + merged, warnings = _merge_route_registry(self._load_route_registry(), incoming) + for w in warnings: + logger.warning(' route registry: {}', w) + return merged diff --git a/tests/test_leasing_admission.py b/tests/test_leasing_admission.py index 75630b51..dbbb552f 100644 --- a/tests/test_leasing_admission.py +++ b/tests/test_leasing_admission.py @@ -317,7 +317,7 @@ def test_declined_approval_of_a_coalescing_acquire_changes_nothing(tmp_path): first = ctl.acquire('x', cat.resolve_names(['a']), wait=False) gid = first.deployments[0].id served = dict(ledger.get_deployment(gid).served) - registry = ctl.backend._registry_file.read_text() if ctl.backend._registry_file.exists() else None + registry = ctl.backend.gateway._registry_file.read_text() if ctl.backend.gateway._registry_file.exists() else None def decline(planned): raise ConvergeAborted('no') @@ -327,7 +327,7 @@ def decline(planned): ctl.acquire('y', cat.resolve_names(['b']), wait=False) # would add alias b assert len(ledger.status()[0]) == 1 assert ledger.get_deployment(gid).served == served - now = ctl.backend._registry_file.read_text() if ctl.backend._registry_file.exists() else None + now = ctl.backend.gateway._registry_file.read_text() if ctl.backend.gateway._registry_file.exists() else None assert now == registry diff --git a/tests/test_leasing_dynamic_routing.py b/tests/test_leasing_dynamic_routing.py index 036aa610..4000dd3b 100644 --- a/tests/test_leasing_dynamic_routing.py +++ b/tests/test_leasing_dynamic_routing.py @@ -238,7 +238,7 @@ def test_reconcile_replaces_same_id_route_with_wrong_semantics(tmp_path): # Simulate DB drift / an earlier endpoint definition with the same stable id. gw.models[rid]['litellm_params']['model'] = 'ollama/wrong-tag' before = len(gw.calls) - assert be._reconcile_routes() is True + assert be.gateway._reconcile_routes() is True assert gw.models[rid]['litellm_params']['model'] == 'openai/smol' assert gw.calls[before:] == [('delete', rid), ('new', rid)] @@ -300,12 +300,12 @@ def test_reconcile_leaves_unmanaged_models_alone(tmp_path): def test_db_password_is_persisted_and_reused(tmp_path): be = make_backend(tmp_path, RecordingGateway()) - pw1 = be.db_password() + pw1 = be.gateway.db_password() assert pw1 # rewritten only if missing -> stable across calls and a fresh backend - assert be.db_password() == pw1 + assert be.gateway.db_password() == pw1 be2 = make_backend(tmp_path, RecordingGateway()) - assert be2.db_password() == pw1 + assert be2.gateway.db_password() == pw1 # -- no blip on the UPSTREAMS too (the readiness-killing churn) ------------- @@ -380,9 +380,9 @@ def post(self, url, **kw): monkeypatch.setattr( _log.logger, 'warning', lambda *a, **k: warnings.append((a, k)) ) - be._post_route('/model/delete', {'id': 'isr-x'}, 'isr-x', ok_if_missing=True) + be.gateway._post_route('/model/delete', {'id': 'isr-x'}, 'isr-x', ok_if_missing=True) assert warnings == [] # already gone -> no warning - be._post_route('/model/delete', {'id': 'isr-x'}, 'isr-x') + be.gateway._post_route('/model/delete', {'id': 'isr-x'}, 'isr-x') assert warnings # same response without the flag -> warns @@ -431,9 +431,9 @@ def test_reconcile_reports_success_only_when_routes_verify(tmp_path): gw = RecordingGateway() be = make_backend(tmp_path, gw) be.converge([a], apply=False) - assert be._reconcile_routes() is True + assert be.gateway._reconcile_routes() is True assert _managed(gw) == {_route_id(a.id, 'smol')} - assert be._reconcile_routes() is True # idempotent: nothing to change, still verified + assert be.gateway._reconcile_routes() is True # idempotent: nothing to change, still verified def test_reconcile_against_unreachable_gateway_is_bounded_by_its_deadline(tmp_path): @@ -443,7 +443,7 @@ def test_reconcile_against_unreachable_gateway_is_bounded_by_its_deadline(tmp_pa gw = UnreachableGateway(time) be = _timed_backend(tmp_path, gw, time) be.converge([dep('grp-aaaaaa', served='smol')], apply=False) - assert be._reconcile_routes(deadline_s=25.0) is False + assert be.gateway._reconcile_routes(deadline_s=25.0) is False assert time.now <= 25.0 assert gw.gets >= 2 # it did retry within the budget @@ -458,7 +458,7 @@ def post(self, url, **kw): time = FakeTime() be = _timed_backend(tmp_path, RejectingGateway(), time) be.converge([dep('grp-aaaaaa', served='smol')], apply=False) - assert be._reconcile_routes(deadline_s=20.0) is False + assert be.gateway._reconcile_routes(deadline_s=20.0) is False assert time.now <= 20.0 @@ -477,7 +477,7 @@ def post(self, url, **kw): be = _timed_backend(tmp_path, gw, time) a = dep('grp-aaaaaa', served='smol') be.converge([a], apply=False) - assert be._reconcile_routes(deadline_s=20.0) is True + assert be.gateway._reconcile_routes(deadline_s=20.0) is True assert _managed(gw) == {_route_id(a.id, 'smol')} @@ -496,7 +496,7 @@ def post(self, url, **kw): be = _timed_backend(tmp_path, SlowGateway(), time) be.converge([dep('grp-aaaaaa', served='smol')], apply=False) - be._reconcile_routes(deadline_s=10.0) + be.gateway._reconcile_routes(deadline_s=10.0) assert seen and all(t <= 10.0 - 4.0 for t in seen) # never the default 30 s diff --git a/tests/test_leasing_profile.py b/tests/test_leasing_profile.py index 32b864a9..1a1f5f8a 100644 --- a/tests/test_leasing_profile.py +++ b/tests/test_leasing_profile.py @@ -528,7 +528,7 @@ def test_a_publish_that_never_commits_leaves_no_append_only_state(tmp_path): ledger, ctl = controller(tmp_path, catalog=a) ctl.gc() before = ledger.profile() - registry = ctl.backend._registry_file + registry = ctl.backend.gateway._registry_file registry_before = registry.read_text() if registry.exists() else None def crash(*args, **kw): diff --git a/tests/test_leasing_secrets.py b/tests/test_leasing_secrets.py index 08a2f2d8..f5a730c0 100644 --- a/tests/test_leasing_secrets.py +++ b/tests/test_leasing_secrets.py @@ -55,7 +55,7 @@ def make(tmp_path): def env(ctl): - return parse_env_file(ctl.backend._env_path) + return parse_env_file(ctl.backend.gateway._env_path) def gateway(ctl): From 96ffaf5e594ee67192f2da358cf7ab029859ec1a Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 14:38:49 -0400 Subject: [PATCH 09/34] G-D: render_compose's front door becomes gateway.render_front_door render_compose renders the engine services, then merges in what render_front_door returns: the LiteLLM service and config (static, registry or dynamic routes), Postgres for dynamic routing, Open WebUI and the reverse proxy. The engines pass only what the front door needs from them (their service names for the legacy depends_on, and in-network URLs for a UI without a gateway). Verified byte-identical: six configurations (bare, gateway, gateway+UI+ proxy, route registry with an upstream row, dynamic routing, UI only), 27 services, rendered with the code before and after. Suite 898 passed x3, ty and flake8 clean. compose.py 2483 -> 2360 lines. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- infer_stack/leasing/compose.py | 165 ++++--------------------------- infer_stack/leasing/gateway.py | 173 +++++++++++++++++++++++++++++++++ 2 files changed, 194 insertions(+), 144 deletions(-) diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index c3fdac03..d564b7f4 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -53,29 +53,16 @@ from ..profile_runtime import simulator_args, vllm_args from .backend import ConvergeScaffold, Readiness from .gateway import ( - Gateway, - API_KEY_ENV, LITELLM_CONFIG_FILENAME, - LITELLM_CONTAINER_PORT, - _litellm_model_list, - _litellm_model_list_from_catalog, - _litellm_model_list_from_registry, - _litellm_routes, - _litellm_service, LITELLM_SERVICE, - _nginx_conf, NGINX_CONFIG_FILENAME, - _nginx_service, - NGINX_SERVICE, - _open_webui_service, - OPEN_WEBUI_SERVICE, - _postgres_service, - POSTGRES_SERVICE, - _registry_incoming_from_catalog, - _registry_incoming_from_deployments, ROUTE_RECONCILE_BOOTSTRAP_S, ROUTE_RECONCILE_STEADY_S, SALT_KEY_ENV, + Gateway, + _registry_incoming_from_catalog, + _registry_incoming_from_deployments, + render_front_door, ) from .launch import env_string, fill, translate_legacy from .models import Deployment, is_reservation, served_name @@ -667,133 +654,23 @@ def render_compose( ollama_native_urls.append(f'http://{name}:{OLLAMA_CONTAINER_PORT}') service_map[name] = deployment.id - litellm_config = None - litellm_routes = None - # The front door (gateway + UI) is rendered whenever it's enabled, even with - # zero models — it's a standing entry point, not a per-model service. So - # releasing/evicting every model leaves an empty gateway (and an empty Open - # WebUI picker) up instead of tearing the whole stack down; only an explicit - # `stack down` removes it. With no models the model_list is simply empty. - if litellm: - # Three route-table strategies, in order of preference: - # * DYNAMIC ROUTING: the rendered config is a STATIC base (empty - # model_list); the real routes live in Postgres and are applied to the - # running gateway via the admin API (see _reconcile_routes). The config - # hash never changes as models come/go, so the gateway is never - # recreated — no blip, and per-deployment routing works (so same-model - # --dedicated deployments each get their own upstream). - # * ROUTE REGISTRY (static-superset default from ComposeBackend): render - # from the whole accumulated registry (every catalog + live deployment - # ever merged, across all runbooks). Byte-stable once seeded, so the - # gateway is never recreated and a cross-catalog converge can no longer - # strip another runbook's routes. The backend loads/merges/writes the - # registry and passes the merged dict in; this function stays pure. - # * STATIC SUPERSET (catalog): one route per catalog endpoint to a - # deterministic host; config depends only on the catalog, so the - # gateway is not recreated as models come/go (no blip) but same-model - # dedicated collapses to one upstream. Unreachable from ComposeBackend - # once the registry is wired; kept for direct callers/tests. - # * LEGACY (no catalog): route only the placed deployments; churns the - # config (and recreates the gateway) on every model change. - if dynamic_routing: - entries: list[dict[str, Any]] = [] - litellm_routes = _litellm_routes(deployments, assignments) - litellm_depends: list[str] = [] - elif route_registry is not None: - entries = _litellm_model_list_from_registry(route_registry) - litellm_depends = [] # no per-model depends_on -> no churn - elif catalog is not None: - entries = _litellm_model_list_from_catalog(catalog) - litellm_depends = [] # no per-model depends_on -> no churn - else: - entries = _litellm_model_list(deployments, assignments) - litellm_depends = list(service_map) - litellm_config = yaml.safe_dump( - { - 'model_list': entries, - 'general_settings': { - 'master_key': f'os.environ/{API_KEY_ENV}' - }, - # An upstream vLLM/Ollama is unreachable only briefly, while it - # loads its model (LiteLLM does not wait for upstream health to - # start). Retry transient connection errors and don't park a - # model in a long cooldown, so the warmup window is self-healing - # instead of surfacing as client 500s ("Connection error. - # Received Model Deployment=…"). - 'router_settings': { - 'num_retries': 3, - 'timeout': 600, - 'cooldown_time': 5, - 'allowed_fails': 100, - }, - }, - sort_keys=False, - ) - config_hash = hashlib.sha256( - litellm_config.encode('utf-8') - ).hexdigest()[:12] - if dynamic_routing: - services[POSTGRES_SERVICE] = _postgres_service(images, state) - services[LITELLM_SERVICE] = _litellm_service( - litellm_depends, - litellm_port, - images, - str(aux_dir or '.'), - master_key=litellm_master_key, - config_hash=config_hash, - dynamic_routing=dynamic_routing, - salt_key=litellm_salt_key, - ) - - # Open WebUI is its own standing front door, rendered whenever ``ui`` is set - # — it does NOT require LiteLLM. Its OpenAI connection prefers the gateway - # (one URL covers every alias) and falls back to the rendered vLLM upstreams' - # own /v1 when there is no gateway. Its native Ollama connection always - # points straight at any Ollama daemon, so you can pull/run models from the - # UI and have the daemon load them on demand — a true drop-in for a - # hand-run ollama + Open WebUI stack. depends_on lists only LiteLLM (the one - # service guaranteed present alongside the UI); the per-model upstreams come - # and go, so the UI tolerates them being absent rather than hard-depending. - # With a gateway the UI is a standing front door (renders even at zero - # models). Without one it is only meaningful pointed at a live upstream, so - # render it only when there is something to connect to — otherwise an empty - # desired set has nothing to run and converge tears the project down. - if ui and (litellm or vllm_v1_urls or ollama_native_urls): - if litellm: - openai_urls = [f'http://{LITELLM_SERVICE}:{LITELLM_CONTAINER_PORT}/v1'] - ui_depends = [LITELLM_SERVICE] - else: - openai_urls = list(vllm_v1_urls) - ui_depends = [] - services[OPEN_WEBUI_SERVICE] = _open_webui_service( - ui_port, - images, - state, - litellm_master_key, - openai_urls=openai_urls, - ollama_urls=ollama_native_urls, - depends_on=ui_depends, - ) - - # Optional single-port HTTP reverse proxy fronting the gateway (+ UI). Needs - # the gateway, so it's only rendered alongside litellm. - nginx_config = None - if reverse_proxy and litellm: - depends = [LITELLM_SERVICE] + ([OPEN_WEBUI_SERVICE] if ui else []) - if reverse_proxy_config: - services[NGINX_SERVICE] = _nginx_service( - reverse_proxy_port, images, aux_dir=str(aux_dir or '.'), - depends_on=depends, config_path=reverse_proxy_config, - ) - else: - nginx_config = _nginx_conf(litellm=litellm, ui=ui) - services[NGINX_SERVICE] = _nginx_service( - reverse_proxy_port, images, aux_dir=str(aux_dir or '.'), - depends_on=depends, - config_hash=hashlib.sha256( - nginx_config.encode('utf-8') - ).hexdigest()[:12], - ) + front = render_front_door( + deployments, assignments, + engine_services=list(service_map), + vllm_v1_urls=vllm_v1_urls, ollama_native_urls=ollama_native_urls, + images=images, state=state, + litellm=litellm, litellm_port=litellm_port, + litellm_master_key=litellm_master_key, litellm_salt_key=litellm_salt_key, + ui=ui, ui_port=ui_port, + reverse_proxy=reverse_proxy, reverse_proxy_port=reverse_proxy_port, + reverse_proxy_config=reverse_proxy_config, aux_dir=aux_dir, + catalog=catalog, route_registry=route_registry, + dynamic_routing=dynamic_routing, + ) + services.update(front.services) + litellm_config = front.litellm_config + nginx_config = front.nginx_config + litellm_routes = front.litellm_routes for name, svc in services.items(): svc.setdefault('labels', {})[SERVICE_LABEL] = name diff --git a/infer_stack/leasing/gateway.py b/infer_stack/leasing/gateway.py index 0da21a7f..cf549633 100644 --- a/infer_stack/leasing/gateway.py +++ b/infer_stack/leasing/gateway.py @@ -14,6 +14,7 @@ import hashlib import json import time +from dataclasses import dataclass from pathlib import Path from typing import Any, Callable @@ -733,6 +734,178 @@ def set_master_key(env_path: Path, key: str) -> None: write_env_file(env_path, values) +@dataclass +class FrontDoor: + """The rendered front door: its Compose services and config files.""" + + services: dict[str, Any] + litellm_config: str | None + nginx_config: str | None + litellm_routes: list[dict[str, Any]] | None + + +def render_front_door( + deployments: list[Deployment], + assignments: dict[str, list[int]], + *, + engine_services: list[str], + vllm_v1_urls: list[str], + ollama_native_urls: list[str], + images: dict[str, str], + state: dict[str, str], + litellm: bool, + litellm_port: int, + litellm_master_key: str | None, + litellm_salt_key: bool, + ui: bool, + ui_port: int, + reverse_proxy: bool, + reverse_proxy_port: int, + reverse_proxy_config: str | None, + aux_dir: str | Path | None, + catalog: Any, + route_registry: dict[str, Any] | None, + dynamic_routing: bool, +) -> FrontDoor: + """Render the gateway, its database, Open WebUI and the reverse proxy. + + The engines are the caller's: it passes the services it rendered + (``engine_services``, only for the legacy per-model ``depends_on``) and + the in-network URLs a UI with no gateway can talk to directly. + """ + services: dict[str, Any] = {} + litellm_config = None + litellm_routes = None + # The front door (gateway + UI) is rendered whenever it's enabled, even with + # zero models — it's a standing entry point, not a per-model service. So + # releasing/evicting every model leaves an empty gateway (and an empty Open + # WebUI picker) up instead of tearing the whole stack down; only an explicit + # `stack down` removes it. With no models the model_list is simply empty. + if litellm: + # Three route-table strategies, in order of preference: + # * DYNAMIC ROUTING: the rendered config is a STATIC base (empty + # model_list); the real routes live in Postgres and are applied to the + # running gateway via the admin API (see _reconcile_routes). The config + # hash never changes as models come/go, so the gateway is never + # recreated — no blip, and per-deployment routing works (so same-model + # --dedicated deployments each get their own upstream). + # * ROUTE REGISTRY (static-superset default from ComposeBackend): render + # from the whole accumulated registry (every catalog + live deployment + # ever merged, across all runbooks). Byte-stable once seeded, so the + # gateway is never recreated and a cross-catalog converge can no longer + # strip another runbook's routes. The backend loads/merges/writes the + # registry and passes the merged dict in; this function stays pure. + # * STATIC SUPERSET (catalog): one route per catalog endpoint to a + # deterministic host; config depends only on the catalog, so the + # gateway is not recreated as models come/go (no blip) but same-model + # dedicated collapses to one upstream. Unreachable from ComposeBackend + # once the registry is wired; kept for direct callers/tests. + # * LEGACY (no catalog): route only the placed deployments; churns the + # config (and recreates the gateway) on every model change. + if dynamic_routing: + entries: list[dict[str, Any]] = [] + litellm_routes = _litellm_routes(deployments, assignments) + litellm_depends: list[str] = [] + elif route_registry is not None: + entries = _litellm_model_list_from_registry(route_registry) + litellm_depends = [] # no per-model depends_on -> no churn + elif catalog is not None: + entries = _litellm_model_list_from_catalog(catalog) + litellm_depends = [] # no per-model depends_on -> no churn + else: + entries = _litellm_model_list(deployments, assignments) + litellm_depends = list(engine_services) + litellm_config = yaml.safe_dump( + { + 'model_list': entries, + 'general_settings': { + 'master_key': f'os.environ/{API_KEY_ENV}' + }, + # An upstream vLLM/Ollama is unreachable only briefly, while it + # loads its model (LiteLLM does not wait for upstream health to + # start). Retry transient connection errors and don't park a + # model in a long cooldown, so the warmup window is self-healing + # instead of surfacing as client 500s ("Connection error. + # Received Model Deployment=…"). + 'router_settings': { + 'num_retries': 3, + 'timeout': 600, + 'cooldown_time': 5, + 'allowed_fails': 100, + }, + }, + sort_keys=False, + ) + config_hash = hashlib.sha256( + litellm_config.encode('utf-8') + ).hexdigest()[:12] + if dynamic_routing: + services[POSTGRES_SERVICE] = _postgres_service(images, state) + services[LITELLM_SERVICE] = _litellm_service( + litellm_depends, + litellm_port, + images, + str(aux_dir or '.'), + master_key=litellm_master_key, + config_hash=config_hash, + dynamic_routing=dynamic_routing, + salt_key=litellm_salt_key, + ) + + # Open WebUI is its own standing front door, rendered whenever ``ui`` is set + # — it does NOT require LiteLLM. Its OpenAI connection prefers the gateway + # (one URL covers every alias) and falls back to the rendered vLLM upstreams' + # own /v1 when there is no gateway. Its native Ollama connection always + # points straight at any Ollama daemon, so you can pull/run models from the + # UI and have the daemon load them on demand — a true drop-in for a + # hand-run ollama + Open WebUI stack. depends_on lists only LiteLLM (the one + # service guaranteed present alongside the UI); the per-model upstreams come + # and go, so the UI tolerates them being absent rather than hard-depending. + # With a gateway the UI is a standing front door (renders even at zero + # models). Without one it is only meaningful pointed at a live upstream, so + # render it only when there is something to connect to — otherwise an empty + # desired set has nothing to run and converge tears the project down. + if ui and (litellm or vllm_v1_urls or ollama_native_urls): + if litellm: + openai_urls = [f'http://{LITELLM_SERVICE}:{LITELLM_CONTAINER_PORT}/v1'] + ui_depends = [LITELLM_SERVICE] + else: + openai_urls = list(vllm_v1_urls) + ui_depends = [] + services[OPEN_WEBUI_SERVICE] = _open_webui_service( + ui_port, + images, + state, + litellm_master_key, + openai_urls=openai_urls, + ollama_urls=ollama_native_urls, + depends_on=ui_depends, + ) + + # Optional single-port HTTP reverse proxy fronting the gateway (+ UI). Needs + # the gateway, so it's only rendered alongside litellm. + nginx_config = None + if reverse_proxy and litellm: + depends = [LITELLM_SERVICE] + ([OPEN_WEBUI_SERVICE] if ui else []) + if reverse_proxy_config: + services[NGINX_SERVICE] = _nginx_service( + reverse_proxy_port, images, aux_dir=str(aux_dir or '.'), + depends_on=depends, config_path=reverse_proxy_config, + ) + else: + nginx_config = _nginx_conf(litellm=litellm, ui=ui) + services[NGINX_SERVICE] = _nginx_service( + reverse_proxy_port, images, aux_dir=str(aux_dir or '.'), + depends_on=depends, + config_hash=hashlib.sha256( + nginx_config.encode('utf-8') + ).hexdigest()[:12], + ) + + return FrontDoor(services=services, litellm_config=litellm_config, + nginx_config=nginx_config, litellm_routes=litellm_routes) + + class Gateway(ConvergeScaffold): """The front door's state: settings, the managed keys, the route registry. From f010ccf728bfdaab724d8f423b20f5c94108611a Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 14:39:35 -0400 Subject: [PATCH 10/34] Record the gateway extraction; drop two dead names in the controller Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 13 ++++++++ dev/journals/claude.md | 32 +++++++++++++++++++ .../plan-backend-unification-2026-09-24.md | 12 +++++++ infer_stack/leasing/controller.py | 4 +-- 4 files changed, 59 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 218fa5f7..800fc834 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,19 @@ scheduler with its own GPU accounting: a lease held outside it, manual or keep-warm, occupies a GPU the scheduler believes is free, and the job it places there cannot start. +### The gateway is its own module + +The front door (LiteLLM config and service, route registry, dynamic-route +reconciliation, the managed keys, Open WebUI, the reverse proxy) moved out of +`leasing/compose.py` into `leasing/gateway.py`, with a `Gateway` object that +`ComposeBackend` owns (`backend.gateway`). The service naming rules moved to +`leasing/naming.py`. Backends now hand the gateway their route rows rather +than the gateway reading backend state. Rendered output is unchanged +(verified byte for byte), and `compose.py` is about a third smaller. Code +that imported gateway names from `infer_stack.leasing.compose` should import +them from `infer_stack.leasing.gateway`; the public naming helpers are still +importable from `compose`. + ### The kubeai backend fails fast on an engine that cannot start The crash diagnosis (restart count, exit code, and the engine log classified diff --git a/dev/journals/claude.md b/dev/journals/claude.md index 9bb84781..aeb38b4a 100644 --- a/dev/journals/claude.md +++ b/dev/journals/claude.md @@ -3217,3 +3217,35 @@ evicts idle keep-warm Models when a new one cannot be scheduled. fail on purpose first; GATEWAY=0 is that switch here. (2) "Has method X" is a poor stand-in for "is backend Y": when a second backend gains X, every such check silently changes meaning. + +## 2026-09-24 14:39:35 -0400 + +**Intent.** Make infer-stack more elegant, architecture first and the TUI +after. The user agreed that the Compose backend's real seam is gateway vs. +engines. Model: Claude Opus 5.5 (1M context). + +**What I did.** Four steps on `dev/backend-unification`, each green before +the next: naming rules into `naming.py`, the gateway's module code into +`gateway.py`, a `Gateway` class for its state, and `render_front_door`. +The one real design change: the registry merge no longer reads backend +state. Backends supply rows and the gateway only merges them, which removed +the `upstream_routes` side-channel I had added for KubeAI in K1. Settings the +gateway reads became backend properties onto the single `Gateway`, because +`use_profile` and tests set them on the backend after construction. Two +copies would drift. + +**How I checked a pure refactor.** Unit tests alone are weak evidence for +moved code, so I also rendered six stack configurations with the old and new +code and compared bytes (identical), then ran the real Compose gateway, +`secrets rotate` and the KubeAI e2e. + +**What went wrong.** My first byte comparison "passed" on two empty files, +because the script crashed in both runs. I caught it only because the byte +count was 0. More serious: the weight-floor doctest I wrote earlier wrote +4 GiB of real zeros into /tmp on every run and never cleaned up. A day of +test runs filled 49 GB, k3s went into disk pressure and evicted KubeAI. +Fixed with sparse files. + +**Takeaways.** (1) A comparison of outputs must check the outputs exist: +equality of two failures is not evidence. (2) Test fixtures that need large +files should be sparse. Size was the property under test, not the bytes. diff --git a/dev/tmp/plan-backend-unification-2026-09-24.md b/dev/tmp/plan-backend-unification-2026-09-24.md index ac31001b..ab31e838 100644 --- a/dev/tmp/plan-backend-unification-2026-09-24.md +++ b/dev/tmp/plan-backend-unification-2026-09-24.md @@ -230,3 +230,15 @@ new modules. Each step leaves the full suite, `ty` and flake8 green. The gateway still runs as a Compose project (it is a container), so KubeAI keeps a `ComposeBackend` with no engines: that is now simply "a Compose project with only the front door", not a special mode. + +**G done 2026-09-24** (on `dev/backend-unification`). A: `naming.py` +(99 lines). B: `gateway.py` module code. C: the `Gateway` class; backends +supply route rows, the gateway merges, persists and renders them (the +`upstream_routes` side-channel is gone), and the backend's gateway settings +are properties onto the one `Gateway`. D: `render_front_door`, verified +byte-identical across six configurations. `compose.py` went from 3651 +lines (on `main`) to 2360; `gateway.py` is 1377. Verified live after C: a +Compose gateway apply and `secrets rotate`, and the KubeAI e2e. + +Found on the way: the weight-floor doctest wrote 4 GiB to `/tmp` per run +(fixed on `main`, 21f5e09). diff --git a/infer_stack/leasing/controller.py b/infer_stack/leasing/controller.py index 6636c9b6..35c74d55 100644 --- a/infer_stack/leasing/controller.py +++ b/infer_stack/leasing/controller.py @@ -1086,7 +1086,7 @@ def _acquire_profile_candidate(self, *, residency=None) -> dict | None: candidate['catalogs'] = merge_catalog_sources( stored.get('catalogs') or [], invocation.get('catalogs') or [] ) - except CatalogConflict as ex: + except CatalogConflict: # Only definitions a resident workload is actually running have to # stay frozen. Redefining anything else -- the normal case while # iterating with `catalog endpoint add --force` -- drops the stale @@ -1224,7 +1224,7 @@ def _render_in_scope(self, context: dict) -> None: scope = getattr(self.backend, 'placement_scope', None) if scope is not None: with scope(context): - rec = self._render() + self._render() self.ledger.clear_placement_context() def _apply_pending(self, rec: ReconcileResult) -> ReconcileResult: From 0101a2d8edac9f6920ae30dfaaba5c4f5dfd523b Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 14:39:48 -0400 Subject: [PATCH 11/34] Lesson: size-only test files should be sparse and cleaned up Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- dev/lessons/lessons.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/dev/lessons/lessons.md b/dev/lessons/lessons.md index 9f616c7c..ecdeb16c 100644 --- a/dev/lessons/lessons.md +++ b/dev/lessons/lessons.md @@ -95,3 +95,15 @@ evidence; prefer append-only; supersede incorrect entries with a new one. absent. - **Applies when:** writing or reviewing any multi-threaded test that uses a barrier, latch, or queue rendezvous. + +- **Lesson:** A test that needs a large file for its *size* should create it + sparse (`file.truncate(n)`), and remove its temp dir. `os.path.getsize` + reports the full size, but no disk is used. Writing real bytes leaks gigabytes + per run unless the test cleans up. +- **Evidence / MWE:** 21f5e09. The `weight_floor_gib` doctest wrote two 2 GiB + files into `mkdtemp()` and never removed them. A day of suite runs left + 49 GB in `/tmp` on the guest, k3s tainted its node for disk pressure, and + KubeAI's pods were evicted. With sparse files the doctest takes 0.15 s and + leaves nothing behind. +- **Applies when:** a test exercises size-dependent logic (VRAM floors, disk + checks, download sizes). From af2e8fcccebefe5e8049f7fbf18a417f1a6d1862 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 15:12:35 -0400 Subject: [PATCH 12/34] Make room for leased demand: idle keep-warm models give way on KubeAI Rule (user decision): a keep-warm model without a lease is always a candidate for eviction when a model with a lease needs a resource. Compose admission already enforces it when placing. Where the runtime schedules (KubeAI), a probe now reports Readiness.needs_room for an Unschedulable pod, and Controller.wait_ready evicts the longest-idle deployment through the normal locked evict, one per ROOM_COOLDOWN_S (30 s), never a leased one. The step runs after the deadline check: a test caught it evicting a second model after the wait had already given up. Verified on k3s: with a profile only one Model fits (cpu-half, added to the test values), acquiring a leased Model evicted the idle keep-warm one within 10 s and was ready in 107 s; `E2E_MAKE_ROOM=1 dev/kubeai_e2e.sh` passes. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 11 ++ dev/e2e_tests/kubeai-cpu-values.yaml | 7 ++ dev/kubeai_e2e.sh | 28 +++++ .../plan-backend-unification-2026-09-24.md | 6 + docs/kubeai-backend.md | 5 + infer_stack/backends/kubeai.py | 3 +- infer_stack/leasing/backend.py | 4 + infer_stack/leasing/controller.py | 34 ++++++ tests/test_leasing_make_room.py | 112 ++++++++++++++++++ 9 files changed, 209 insertions(+), 1 deletion(-) create mode 100644 tests/test_leasing_make_room.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 800fc834..2a1cdba6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,17 @@ scheduler with its own GPU accounting: a lease held outside it, manual or keep-warm, occupies a GPU the scheduler believes is free, and the job it places there cannot start. +### An idle keep-warm model gives way to a leased one on KubeAI too + +The rule: a keep-warm model that no lease holds is always a candidate for +eviction when a model with a lease needs a resource. Compose admission already +applied it when placing. On KubeAI the cluster schedules and never evicts, so +a leased Model could sit `Unschedulable` behind idle warm ones until its +timeout. A readiness probe can now report `needs_room`; the wait then evicts +the longest-idle deployment, one per 30 s, never a leased one, and never +after the wait's deadline. Verified on k3s: a leased Model that did not fit +beside an idle one was ready 107 s later, with the idle one evicted. + ### The gateway is its own module The front door (LiteLLM config and service, route registry, dynamic-route diff --git a/dev/e2e_tests/kubeai-cpu-values.yaml b/dev/e2e_tests/kubeai-cpu-values.yaml index 20300543..5d308441 100644 --- a/dev/e2e_tests/kubeai-cpu-values.yaml +++ b/dev/e2e_tests/kubeai-cpu-values.yaml @@ -7,3 +7,10 @@ resourceProfiles: requests: cpu: 8 memory: "16Gi" + # Half the guest's 120 cores: only one such Model fits at a time, which is + # how dev/kubeai_e2e.sh's make-room check forces a scheduling conflict. + cpu-half: + imageName: "cpu" + requests: + cpu: 64 + memory: "16Gi" diff --git a/dev/kubeai_e2e.sh b/dev/kubeai_e2e.sh index cec829dd..aba5df1b 100755 --- a/dev/kubeai_e2e.sh +++ b/dev/kubeai_e2e.sh @@ -109,4 +109,32 @@ remaining=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ -l infer-stack/managed=true -o name | wc -l) [ "$remaining" = 0 ] || { echo "!! model not pruned"; exit 1; } +if [ "${E2E_MAKE_ROOM:-0}" = 1 ]; then + # Needs a profile only one Model fits at a time (`cpu-half` in + # dev/e2e_tests/kubeai-cpu-values.yaml): E2E_ROOM_PROFILE=cpu-half. + echo '== an idle keep-warm model gives way to a leased one' + cat >> "$WORK/config/catalog.yaml" <&1 | tee "$WORK/big.log" | grep -q 'ready: True'; then + echo '!! the leased model never became ready' >&2; exit 1 + fi + grep -q 'making room for leased demand' "$WORK/big.log" \ + || { echo '!! no idle model was evicted to make room' >&2; exit 1; } + echo ' the idle keep-warm model was evicted; the leased one is ready' + run_is release --env-file "$WORK/big.env" --yes +fi + echo 'PASS: kubeai backend end-to-end lifecycle' diff --git a/dev/tmp/plan-backend-unification-2026-09-24.md b/dev/tmp/plan-backend-unification-2026-09-24.md index ab31e838..130bf9dc 100644 --- a/dev/tmp/plan-backend-unification-2026-09-24.md +++ b/dev/tmp/plan-backend-unification-2026-09-24.md @@ -160,6 +160,12 @@ backends, `_admission_mode()` is always true for real backends. The non-admission branches are then deleted. Memory and Null backends get a trivial residency so tests use the same path. +**Decision (user, 2026-09-24):** "Keep warm models without a lease should +always be candidates for eviction if a model with a lease needs a resource." +Implemented backend-neutrally ahead of K3: `Readiness.needs_room`, and +`Controller._make_room` in the wait. The one acquire path itself (K3) is +still open. + ### K4. One failure diagnosis Move `startup_failure` and the log classifier behind the K2 snapshot. Compose diff --git a/docs/kubeai-backend.md b/docs/kubeai-backend.md index df1f4e8f..59adb851 100644 --- a/docs/kubeai-backend.md +++ b/docs/kubeai-backend.md @@ -108,6 +108,11 @@ infer-stack release --env-file lease.env capacity (the cluster schedules); a Model the cluster cannot place sits not-ready until the acquire's `--timeout`, which then rolls the lease back. The wait says why (`pod: Unschedulable`, `pod: ImagePullBackOff`). +- **A keep-warm model without a lease gives way to one with a lease.** When + a leased Model's pod is `Unschedulable`, the wait evicts the longest-idle + keep-warm deployment, one per 30 s, until it fits. Compose applies the same + rule when placing. A leased model is never evicted this way. Verified on + k3s: `E2E_MAKE_ROOM=1 dev/kubeai_e2e.sh` with the `cpu-half` profile. - **An engine that cannot start fails the acquire at once**, as on compose: the crash diagnosis reads the pods (`kubectl get pods`, restart count, last exit) and the previous run's log, and quotes the engine's error with a diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index 2385bd10..a65bdf0c 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -638,7 +638,8 @@ def probe_ready(self, deployment: Deployment, endpoint: str) -> Readiness: if failure is not None: return Readiness(False, failure, fatal=True) waiting = self._waiting_reason(deployment) - return Readiness(False, f'{reason} (pod: {waiting})' if waiting else reason) + return Readiness(False, f'{reason} (pod: {waiting})' if waiting else reason, + needs_room='Unschedulable' in waiting) def access(self, endpoints: list[str]) -> dict[str, Any] | None: """Where a client reaches these endpoints (env-file descriptor). diff --git a/infer_stack/leasing/backend.py b/infer_stack/leasing/backend.py index 59f7736b..0c1db433 100644 --- a/infer_stack/leasing/backend.py +++ b/infer_stack/leasing/backend.py @@ -33,6 +33,10 @@ class Readiness: ready: bool detail: str = '' fatal: bool = False + #: The endpoint is waiting for resources held by others (a Kubernetes pod + #: the scheduler cannot place). The controller then frees room by + #: evicting idle keep-warm deployments, which no lease holds. + needs_room: bool = False class BackendTimeout(RuntimeError): diff --git a/infer_stack/leasing/controller.py b/infer_stack/leasing/controller.py index 35c74d55..de969761 100644 --- a/infer_stack/leasing/controller.py +++ b/infer_stack/leasing/controller.py @@ -37,6 +37,9 @@ _T = TypeVar('_T') KEEP_WARM = 'keep-warm' +#: Between two evictions made to fit leased demand: long enough for the +#: freed instance to terminate and the scheduler to retry. +ROOM_COOLDOWN_S = 30.0 #: After an interrupted apply, how long to wait for the runtime to stop changing #: before applying again, and how often to sample it. Held under the lock. @@ -1393,14 +1396,17 @@ def wait_ready( if endpoints is None or ep in endpoints ] deadline = self.clock() + timeout + last_room = float('-inf') while True: pending = [] failures = [] + needs_room = False for (g, ep) in pairs: probe = self.backend.probe_ready(g, ep) if probe.ready: continue pending.append((g, ep)) + needs_room = needs_room or probe.needs_room if probe.fatal: failures.append((g.id, ep, probe.detail)) if not pending: @@ -1416,9 +1422,37 @@ def wait_ready( ready=False, pending=[(g.id, ep) for g, ep in pending], ) + # After the deadline check: a wait that has given up evicts nothing. + if needs_room and self.clock() - last_room >= ROOM_COOLDOWN_S: + # A leased model is waiting on resources: a model without a + # lease is always a candidate to give them up. + if self._make_room(): + last_room = self.clock() self.sleep(interval) pairs = pending + def _make_room(self) -> str | None: + """Evict the longest-idle deployment for leased demand; its id, or ``None``. + + Idle means no lease holds it (a keep-warm model left resident). One at + a time: the runtime cannot say how much room is needed, and every + warm model kept is a load avoided. Compose backends never ask, since + their admission already moves idle models aside; this serves backends + where the runtime schedules (KubeAI). + """ + from .._log import logger + + _, deployments = self.ledger.status(virtual_expiry=True) + idle = sorted((g for g in deployments if g.state == DeploymentState.IDLE), + key=lambda g: (g.updated_at, g.id)) + if not idle: + return None + victim = idle[0].id + logger.info('making room for leased demand: evicting idle keep-warm {} ({})', + victim, ', '.join(sorted(idle[0].served))) + self.evict([victim]) + return victim + def _never_ran(self, deployment_ids: list[str]) -> list[str]: """Which of these deployments definitely have no container at all. diff --git a/tests/test_leasing_make_room.py b/tests/test_leasing_make_room.py new file mode 100644 index 00000000..ec6aeb05 --- /dev/null +++ b/tests/test_leasing_make_room.py @@ -0,0 +1,112 @@ +"""A keep-warm model without a lease gives way to a model with one. + +The rule (decided 2026-09-24): an idle keep-warm deployment -- resident, but +held by no lease -- is always a candidate for eviction when a leased model +needs a resource. Compose admission enforces it when placing. Where the +runtime schedules instead (KubeAI), the probe says `needs_room` and the wait +evicts idle models, the longest idle first, one per cooldown. +""" + +from __future__ import annotations + +from infer_stack.leasing import ( + Controller, + DeploymentState, + Ledger, + MemoryBackend, + SqliteStore, +) +from infer_stack.leasing.backend import Readiness +from infer_stack.leasing.controller import ROOM_COOLDOWN_S +from test_leasing_controller import FakeClock, _id_factory, vreq + + +class OneSlotBackend(MemoryBackend): + """Room for `slots` models; a model that does not fit waits for room.""" + + def __init__(self, slots=1): + super().__init__(ready=True) + self.slots = slots + + def probe_ready(self, deployment, endpoint): + others = [gid for gid in self.realized if gid != deployment.id] + if deployment.id in self.realized and len(others) >= self.slots: + return Readiness(False, 'pod: Unschedulable', needs_room=True) + return super().probe_ready(deployment, endpoint) + + +def make(slots=1): + clock = FakeClock() + ledger = Ledger(SqliteStore(':memory:'), clock=clock, id_factory=_id_factory()) + backend = OneSlotBackend(slots) + ctl = Controller(ledger, backend, clock=clock, sleep=clock.advance) + return ctl, ledger, backend, clock + + +def state(ledger, gid): + return ledger.get_deployment(gid).state + + +def test_an_idle_keep_warm_model_gives_way_to_a_leased_one(): + ctl, ledger, backend, _ = make() + warm = ctl.acquire('a', [vreq('warm')], wait=False) + ctl.release(warm.lease.id) # keep-warm: still resident + warm_id = warm.deployments[0].id + assert state(ledger, warm_id) == DeploymentState.IDLE + + out = ctl.acquire('b', [vreq('big')], wait=True, timeout=600, interval=5) + + assert out.wait.ready + assert state(ledger, warm_id) == DeploymentState.STOPPED + assert warm_id not in backend.realized + + +def test_the_longest_idle_goes_first_and_one_at_a_time(): + ctl, ledger, backend, clock = make(slots=2) + older = ctl.acquire('a', [vreq('older')], wait=False) + ctl.release(older.lease.id) + clock.advance(60) + newer = ctl.acquire('a', [vreq('newer')], wait=False) + ctl.release(newer.lease.id) + + out = ctl.acquire('b', [vreq('big')], wait=True, timeout=600, interval=5) + + assert out.wait.ready + assert state(ledger, older.deployments[0].id) == DeploymentState.STOPPED + # One eviction freed the slot, so the more recently used model stays warm. + assert state(ledger, newer.deployments[0].id) == DeploymentState.IDLE + + +def test_evictions_wait_for_the_cooldown(): + ctl, ledger, backend, clock = make(slots=0) # nothing ever fits + for name in ('w1', 'w2'): + out = ctl.acquire('a', [vreq(name)], wait=False) + ctl.release(out.lease.id) + big = ctl.acquire('b', [vreq('big')], wait=False) + + ctl.wait_ready(big.deployments, timeout=ROOM_COOLDOWN_S - 1, interval=5) + stopped = [g for g in ledger.status()[1] if g.state == DeploymentState.STOPPED] + assert len(stopped) == 1 # not both within one cooldown + + +def test_a_leased_model_is_never_evicted_to_make_room(): + ctl, ledger, backend, _ = make() + held = ctl.acquire('a', [vreq('held')], wait=False) # still leased + out = ctl.acquire('b', [vreq('big')], wait=True, timeout=120, interval=5) + + assert not out.wait.ready # waits, then times out + assert state(ledger, held.deployments[0].id) == DeploymentState.LIVE + + +def test_an_unschedulable_kubeai_pod_asks_for_room(tmp_path): + from test_leasing_kubeai import FakeHttp, _pod, make_pod_backend, vllm + + be, kubectl = make_pod_backend(tmp_path) + dep = vllm('grp-big', served='big') + be.converge([dep]) + kubectl.pods = [_pod('model-big-1', 'grp-big', statuses=False, conditions=[ + {'type': 'PodScheduled', 'status': 'False', 'reason': 'Unschedulable'}])] + be.http.post = lambda url, **kw: FakeHttp._Resp(503, {'detail': 'not ready'}) + + probe = be.probe_ready(dep, 'grp-big') + assert probe.needs_room and not probe.fatal and not probe.ready From 08e61d52165f3eb502907c089a9d19b156c94934 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 17:25:22 -0400 Subject: [PATCH 13/34] Serialize every use of the shared ledger connection SqliteStore shares one sqlite3 connection across threads (check_same_thread=False) but only write transactions took its lock, on the belief that sqlite's serialized mode makes concurrent reads safe. It does not for a shared Python connection: two threads stepping the same cached statement interleave. The TUI reads the ledger from its refresh worker and the UI thread at once, and intermittently got a lease whose deployment ids included None (TypeError drawing the leases table; seen as a flaky TUI test). Every statement now goes through _SerializedConnection, which holds the store's re-entrant lock until the statement has run and its rows are fetched, so no cursor outlives the lock and transaction() nests. New test: eight threads reading one ledger 150 times each; without the fix it fails with InterfaceError('bad parameter or other API misuse') and more. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- infer_stack/leasing/store.py | 64 +++++++++++++++++++++++++---- tests/test_leasing_store_threads.py | 45 ++++++++++++++++++++ 2 files changed, 102 insertions(+), 7 deletions(-) create mode 100644 tests/test_leasing_store_threads.py diff --git a/infer_stack/leasing/store.py b/infer_stack/leasing/store.py index 052d352f..a0c393a1 100644 --- a/infer_stack/leasing/store.py +++ b/infer_stack/leasing/store.py @@ -93,6 +93,54 @@ def _dumps(value: Any) -> str: return json.dumps(value, separators=(',', ':'), sort_keys=True) +class _Result: + """A statement's rows, fetched while the connection lock was held.""" + + def __init__(self, rows: list, rowcount: int, lastrowid: int | None): + self._rows = rows + self.rowcount = rowcount + self.lastrowid = lastrowid + + def fetchone(self): + return self._rows[0] if self._rows else None + + def fetchall(self) -> list: + return list(self._rows) + + def __iter__(self): + return iter(self._rows) + + +class _SerializedConnection: + """One sqlite connection, safe to share between threads. + + Python's sqlite3 connection is not: two threads stepping the same cached + statement interleave and return garbage rows. The TUI hit this reading the + ledger from its refresh worker and the UI thread at once (a lease whose + deployment ids included ``None``). Each call here holds the store lock + until the statement has run and every row is fetched, so no cursor ever + outlives it. The lock is re-entrant, so :meth:`SqliteStore.transaction` + (which holds it for the whole transaction) nests. + """ + + def __init__(self, conn: sqlite3.Connection, lock): + self._conn = conn + self._lock = lock + + def execute(self, sql: str, params=()) -> _Result: + with self._lock: + cur = self._conn.execute(sql, params) + return _Result(cur.fetchall(), cur.rowcount, cur.lastrowid) + + def executescript(self, script: str) -> None: + with self._lock: + self._conn.executescript(script) + + def close(self) -> None: + with self._lock: + self._conn.close() + + class SqliteStore: """Thin sqlite wrapper exposing ledger row operations.""" @@ -101,17 +149,19 @@ def __init__(self, path: str | Path = ':memory:', *, busy_timeout_ms: int = 5000 if self.path != ':memory:': Path(self.path).expanduser().parent.mkdir(parents=True, exist_ok=True) # check_same_thread=False lets a long-running process (e.g. the TUI) use - # this connection from a worker thread for converge-while-monitoring; - # ``_lock`` serializes write transactions so two threads can't both - # ``BEGIN IMMEDIATE`` on the one connection. sqlite itself is built - # serialized, so individual reads across threads are safe. - self._conn = sqlite3.connect( + # this connection from worker threads. That is only safe because every + # statement runs under ``_lock`` (see _SerializedConnection): sqlite's + # own serialized mode does NOT make a shared Python connection safe for + # concurrent reads. + raw = sqlite3.connect( self.path, isolation_level=None, timeout=busy_timeout_ms / 1000, check_same_thread=False, ) + raw.row_factory = sqlite3.Row self._lock = threading.RLock() + # Every use of the connection holds the lock: see _SerializedConnection. + self._conn = _SerializedConnection(raw, self._lock) self._busy_timeout_ms = busy_timeout_ms - self._conn.row_factory = sqlite3.Row self._conn.execute('PRAGMA foreign_keys = ON') self._conn.execute(f'PRAGMA busy_timeout = {busy_timeout_ms}') if self.path != ':memory:': @@ -212,7 +262,7 @@ def __del__(self) -> None: # -- transactions ------------------------------------------------------ @contextlib.contextmanager - def transaction(self) -> Iterator[sqlite3.Connection]: + def transaction(self) -> Iterator[_SerializedConnection]: """Take the write lock up front and commit/rollback atomically. ``BEGIN IMMEDIATE`` is what makes the ledger's find-or-create-deployment diff --git a/tests/test_leasing_store_threads.py b/tests/test_leasing_store_threads.py new file mode 100644 index 00000000..3e926ce1 --- /dev/null +++ b/tests/test_leasing_store_threads.py @@ -0,0 +1,45 @@ +"""One ledger connection shared by threads must never return garbage rows. + +The TUI reads the ledger from its refresh worker and from the UI thread at +the same moment. Python's sqlite3 connection is not safe for that on its own: +two threads stepping the same cached statement interleave, and a lease came +back with ``None`` among its deployment ids (TypeError in the leases table). +""" + +from __future__ import annotations + +import threading + +from infer_stack.leasing import Catalog, Controller, Ledger, NullBackend, SqliteStore + + +def test_concurrent_readers_see_consistent_rows(tmp_path): + catalog = Catalog.from_dict({ + 'models': {'m': {'source': 'hf://org/m'}}, + 'endpoints': {f'ep-{i:02d}': {'engine': 'vllm', 'model': 'm'} for i in range(30)}, + }) + ledger = Ledger(SqliteStore(str(tmp_path / 'ledger.db'))) + controller = Controller(ledger, NullBackend()) + for i in range(30): + controller.acquire(f'u{i:02d}', catalog.resolve_names([f'ep-{i:02d}'])) + + problems: list[str] = [] + + def read(): + try: + for _ in range(150): + leases, deployments = ledger.status(virtual_expiry=True) + if len(leases) != 30 or len(deployments) != 30: + problems.append(f'{len(leases)} leases, {len(deployments)} deployments') + for le in leases: + if not le.deployment_ids or None in le.deployment_ids: + problems.append(f'{le.id}: deployment ids {le.deployment_ids}') + except Exception as ex: # noqa: BLE001 - any error is the failure + problems.append(repr(ex)) + + threads = [threading.Thread(target=read) for _ in range(8)] + for t in threads: + t.start() + for t in threads: + t.join() + assert problems == [] From f03f6e981b5ec4c2c22ec73fc2dbf6cccda0b416 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 17:25:22 -0400 Subject: [PATCH 14/34] TUI: stop repainting unchanged panes; batch log drawing; report stalls Measured with dev/profile_tui.py (headless and on a real terminal) and a real `infer-stack tui` session in tmux: - The lease and deployment panes were repainted every refresh tick: Textual repaints a widget whenever its border title is set, even to the same text. Titles are now set only on change. Idle terminal output fell from 6922 to 75 bytes/s (the header clock), and from 5223 to 442 with the docker pane open, which is what an SSH session feels. - Streamed log lines went to the UI one call_from_thread per line, capping a loading engine at ~1800 lines/s. Workers now append to a deque the UI drains every 100 ms, at most 200 lines per tick (2000 in one tick stalled the loop 218 ms), keeping the newest lines when a flood outruns the pane. The line mirror is trimmed instead of growing without bound. - The compose file's service list is parsed only when the file changes, not every tick. - A watchdog reports any UI-thread stall over 0.5 s in the TUI log, with the stack sampled while it was stuck written to the error log file, so a freeze on the host comes with what caused it. The scroll-offset test waits for the mount-time refresh; its intermittent failure was the ledger race fixed in the previous commit. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- dev/profile_tui.py | 151 ++++++++++++++++++++++++++++++++++++++++++ infer_stack/tui.py | 162 ++++++++++++++++++++++++++++++++++++++++----- tests/test_tui.py | 65 +++++++++++++++++- 3 files changed, 361 insertions(+), 17 deletions(-) create mode 100644 dev/profile_tui.py diff --git a/dev/profile_tui.py b/dev/profile_tui.py new file mode 100644 index 00000000..88c252f8 --- /dev/null +++ b/dev/profile_tui.py @@ -0,0 +1,151 @@ +"""Measure how responsive the TUI's event loop is, and what blocks it. + +Runs the real app headless against a real ComposeBackend (over the tests' +fake Docker, so the numbers are the TUI's own cost, not Docker's), with a +ledger holding a few leases and deployments. A 50 ms timer on the app's own +loop measures lag: how late it fires is how late a keypress would be +handled. Scenarios: + + idle the dashboard refreshing on its normal timer + docker the docker pane open on the Logs tab (service list, log stream) + flood a log stream emitting lines as fast as it can (an engine loading) + + python dev/profile_tui.py [--profile] # --profile: cProfile the UI thread +""" + +from __future__ import annotations + +import asyncio +import cProfile +import io +import pstats +import statistics +import sys +import tempfile +import time +from pathlib import Path + +HERE = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(HERE / 'tests')) + +from test_leasing_admission import CAT, acquire, make # noqa: E402 + +from infer_stack.tui import InferStackTUI # noqa: E402 + +FLOOD_LINES = 20000 + + +class FloodProc: + """A log process that writes FLOOD_LINES lines as fast as it can.""" + + def __init__(self, n): + self.n = n + + @property + def stdout(self): + for i in range(self.n): + yield f'vllm-one | INFO 09-24 12:00:00 loader.py:1 loading shard {i}\n' + + def terminate(self): + pass + + +async def lag_probe(seconds: float, period: float = 0.05) -> list[float]: + lags = [] + end = time.perf_counter() + seconds + while time.perf_counter() < end: + t0 = time.perf_counter() + await asyncio.sleep(period) + lags.append(time.perf_counter() - t0 - period) + return lags + + +def summary(name: str, lags: list[float]) -> str: + ms = sorted(x * 1000 for x in lags) + p95 = ms[int(len(ms) * 0.95) - 1] if ms else 0 + return (f'{name:8s} samples={len(ms):4d} median={statistics.median(ms):7.1f} ms ' + f'p95={p95:7.1f} ms max={max(ms):7.1f} ms') + + +def main() -> None: + do_profile = '--profile' in sys.argv + # --docker-latency=S: each docker call takes S seconds, like a loaded host. + latency = next((float(a.split('=', 1)[1]) for a in sys.argv + if a.startswith('--docker-latency=')), 0.0) + # --tty: run on a real terminal (not headless), so drawing is measured too. + tty = '--tty' in sys.argv + import threading + tmp = Path(tempfile.mkdtemp()) + ledger, ctl, docker = make(tmp) + for name in ('one', 'two'): + out = acquire(ctl, name) + if name == 'two': + ctl.release(out.lease.id) # an idle keep-warm resident + flood = {'on': False} + if latency: + real_run = ctl.backend.run + + def slow_run(args, **kw): + time.sleep(latency) + return real_run(args, **kw) + ctl.backend.run = slow_run + + def proc_factory(service): + return FloodProc(FLOOD_LINES) if flood['on'] else FloodProc(0) + + results = [] + profiler = cProfile.Profile() + + async def scenario(): + app = InferStackTUI(ctl, CAT, interval=1.0, proc_factory=proc_factory) + async with app.run_test(size=(200, 60)) as pilot: + await pilot.pause(1.0) + if do_profile: + profiler.enable() + results.append(summary('idle', await lag_probe(8))) + app.query_one('#docker').collapsed = False + await pilot.pause(0.5) + results.append(summary('docker', await lag_probe(8))) + flood['on'] = True + app._restart_logs(app._log_service) + t0 = time.perf_counter() + results.append(summary('flood', await lag_probe(8))) + results.append(f'flood: {len(app._log_lines)} of {FLOOD_LINES} lines shown ' + f'after {time.perf_counter() - t0:.1f}s') + results.append(f'threads alive at end: {threading.active_count()}') + if do_profile: + profiler.disable() + + if tty: + # The app on the real terminal, driven by its own timers. + app = InferStackTUI(ctl, CAT, interval=1.0, proc_factory=proc_factory) + + async def drive(): + await asyncio.sleep(1.5) + results.append(summary('idle', await lag_probe(8))) + app.query_one('#docker').collapsed = False + await asyncio.sleep(0.5) + results.append(summary('docker', await lag_probe(8))) + flood['on'] = True + app._restart_logs(app._log_service) + t0 = time.perf_counter() + results.append(summary('flood', await lag_probe(8))) + results.append(f'flood: {len(app._log_lines)} of {FLOOD_LINES} lines shown ' + f'after {time.perf_counter() - t0:.1f}s') + results.append(f'threads alive at end: {threading.active_count()}') + app.exit() + + app.call_later(lambda: asyncio.ensure_future(drive())) + app.run() + else: + asyncio.run(scenario()) + for line in results: + print(line) + if do_profile: + out = io.StringIO() + pstats.Stats(profiler, stream=out).sort_stats('cumulative').print_stats(35) + print(out.getvalue()) + + +if __name__ == '__main__': + main() diff --git a/infer_stack/tui.py b/infer_stack/tui.py index 51f45f92..3fc5b0de 100644 --- a/infer_stack/tui.py +++ b/infer_stack/tui.py @@ -96,6 +96,17 @@ #: Title of the tab that shows what the TUI itself did, and its errors. APP_LOG_TAB_TITLE = 'TUI log' +#: The docker log pane shows at most this many lines (RichLog ``max_lines``). +LOG_PANE_LINES = 2000 +#: How often streamed log lines are drawn: in one batch, not one UI message per +#: line (which capped a loading engine's log at ~1800 lines/s). +LOG_DRAIN_S = 0.1 +#: At most this many lines per drain: RichLog costs ~0.1 ms a line, and 2000 in +#: one tick stalled the UI for 218 ms. The rest waits for the next tick. +LOG_DRAIN_LINES = 200 +#: A UI-thread stall at least this long is reported, with what was running. +STALL_REPORT_S = 0.5 + SELECT_BLANK = next( ( candidate @@ -982,6 +993,19 @@ def __init__( # visible; only the current generation is allowed to append. self._log_generation = 0 self._log_lines: list[str] = [] # mirror of the docker log pane, for tests + # Lines streamed by the log worker, drawn by the UI every LOG_DRAIN_S. + # (generation, line): a line from a replaced stream is dropped. + import collections + self._log_pending: collections.deque = collections.deque() + # Lines taken from _log_pending, not yet drawn; only the newest + # LOG_PANE_LINES are kept, since the pane shows no more. + self._log_backlog: collections.deque = collections.deque(maxlen=LOG_PANE_LINES) + self._log_dropped = 0 + # Parsed compose service names, keyed by the file's (mtime, size). + self._service_names_cache: tuple[tuple[float, int], list[str]] | None = None + # The UI loop's heartbeat, read by the stall watchdog thread. + self._beat = time.monotonic() + self._watchdog_stop = None self._app_log_lines: list[str] = [] # mirror of the TUI log pane, for tests self._app_log_errors = 0 self._api_lines: list[str] = [] # mirror of the API output, for tests @@ -1110,7 +1134,7 @@ def _compose_dashboard(self) -> ComposeResult: id='logsvc', ) yield RichLog(id='logs', highlight=False, - markup=False, max_lines=2000, + markup=False, max_lines=LOG_PANE_LINES, wrap=False) with TabPane('Containers', id='tab-containers'): yield DataTable(id='ps', cursor_type='row', @@ -1289,10 +1313,76 @@ def on_mount(self) -> None: self._refresh_timer = self.set_interval( self.ledger_interval, self.action_refresh ) + self.set_interval(LOG_DRAIN_S, self._drain_logs) + self._start_stall_watchdog() self.query_one('#endpoints', DataTable).focus() def on_unmount(self) -> None: self._terminate_logs() + if self._watchdog_stop is not None: + self._watchdog_stop.set() + + # -- responsiveness ------------------------------------------------------ + + def _start_stall_watchdog(self) -> None: + """Report any UI-thread stall, with the stack that caused it. + + A 0.1 s timer on the UI loop moves a heartbeat; a watchdog thread notices + when it stops, samples the UI thread's stack while it is stuck, and + reports the stall once it ends: its length in the TUI log, the stack in + the error log file. "The TUI froze" then comes with what it was doing. + """ + import sys + import threading + + ui_thread = threading.get_ident() + stop = threading.Event() + self._watchdog_stop = stop + + def beat() -> None: + self._beat = time.monotonic() + + self.set_interval(0.1, beat) + + def watch() -> None: + import traceback + + stalled_since = None + stack: list[str] = [] + while not stop.wait(0.1): + behind = time.monotonic() - self._beat + if behind >= STALL_REPORT_S: + if stalled_since is None: + stalled_since = self._beat + # The first sample, half a second in, is inside whatever + # blocks; later ones can catch the loop catching up. + frame = sys._current_frames().get(ui_thread) + if frame is not None: + stack = traceback.format_stack(frame) + elif stalled_since is not None: + length = self._beat - stalled_since + try: + self.call_from_thread(self._report_stall, length, stack) + except Exception: # noqa: BLE001 - app shutting down + return + stalled_since, stack = None, [] + + threading.Thread(target=watch, name='tui-stall-watchdog', daemon=True).start() + + def _report_stall(self, length: float, stack: list[str]) -> None: + # The innermost frames in infer_stack say what to fix; the file keeps all. + ours = [f for f in stack if 'infer_stack' in f] or stack + where = ours[-1].strip().splitlines()[0] if ours else '(no stack)' + path = self.error_log_path() + try: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open('a', encoding='utf-8') as handle: + handle.write(f'\n=== {time.strftime("%Y-%m-%d %H:%M:%S")} ' + f'UI stalled {length:.1f}s\n') + handle.writelines(stack) + except OSError: + pass + self.app_log(f'UI stalled {length:.1f}s in {where} (stack: {path})', level='warn') # -- theming for docker output bleed ---------------------------------- @@ -1640,16 +1730,23 @@ def _update_summary(self, leases, deployments, observed) -> None: # "0 running" — say we're still observing instead of silently lying. observing = self._observed_at is None running_label = 'observing…' if observing else f'{running} running' + # Assign only on change: Textual repaints a pane whenever its border + # title is set, even to the same text, and these two panes hold the + # lease and deployment tables -- most of the screen, every tick + # (~7 KB/s of terminal output while nothing changed). try: - self.query_one('#docker', Collapsible).title = ( - f'docker — {running_label}' - ) - self.query_one('#leases-pane').border_title = ( - f'leases — {active} active / {len(leases)}' - ) - self.query_one('#deployments-pane').border_title = ( - f'deployments — {running_label} / {len(deployments)}' - ) + docker = self.query_one('#docker', Collapsible) + title = f'docker — {running_label}' + if docker.title != title: + docker.title = title + for pane, text in ( + ('#leases-pane', f'leases — {active} active / {len(leases)}'), + ('#deployments-pane', + f'deployments — {running_label} / {len(deployments)}'), + ): + widget = self.query_one(pane) + if widget.border_title != text: + widget.border_title = text except Exception: # noqa: BLE001 pass @@ -1895,9 +1992,16 @@ def _service_names(self) -> list[str]: if not path: return [] try: + stat = Path(path).stat() + key = (stat.st_mtime, stat.st_size) + cached = self._service_names_cache + if cached is not None and cached[0] == key: + return list(cached[1]) # checked every refresh; parse on change import yaml data = yaml.safe_load(Path(path).read_text()) or {} - return sorted((data.get('services') or {}).keys()) + names = sorted((data.get('services') or {}).keys()) + self._service_names_cache = (key, names) + return list(names) except Exception: # noqa: BLE001 return [] @@ -1973,6 +2077,8 @@ def _restart_logs(self, service: str) -> None: log = self.query_one('#logs', RichLog) log.clear() self._log_lines = [] + self._log_backlog.clear() + self._log_dropped = 0 target, label = self._resolve_log_target(service) if target is NO_LOG_TARGET: log.write(f'— {label} —') @@ -2033,9 +2139,10 @@ def _stream_logs(self, service, generation: int) -> None: self._log_proc = proc try: for line in compact_litellm_tracebacks(proc.stdout): - self.call_from_thread( - self._append_log_if_current, generation, line.rstrip('\n') - ) + if generation != self._log_generation: + break + # Drawn in batches by _drain_logs; deque.append is thread-safe. + self._log_pending.append((generation, line.rstrip('\n'))) except Exception: # noqa: BLE001 - stream ends when the proc dies pass finally: @@ -2048,8 +2155,31 @@ def _append_log_if_current(self, generation: int, line: str) -> None: self._append_log(line) def _append_log(self, line: str) -> None: - self._log_lines.append(line) - self.query_one('#logs', RichLog).write(line) + self._write_log_lines([line]) + + def _write_log_lines(self, lines: list[str]) -> None: + self._log_lines.extend(lines) + if len(self._log_lines) > 2 * LOG_PANE_LINES: + del self._log_lines[:-LOG_PANE_LINES] + self.query_one('#logs', RichLog).write('\n'.join(lines)) + + def _drain_logs(self) -> None: + """Draw streamed lines: a bounded batch per tick, newest lines kept.""" + pending, backlog = self._log_pending, self._log_backlog + while pending: + generation, line = pending.popleft() + if generation != self._log_generation: + continue + if len(backlog) == backlog.maxlen: + self._log_dropped += 1 # the oldest undrawn line falls off + backlog.append(line) + if not backlog: + return + batch = [backlog.popleft() for _ in range(min(LOG_DRAIN_LINES, len(backlog)))] + if self._log_dropped: + batch.insert(0, f'… {self._log_dropped} earlier line(s) not shown') + self._log_dropped = 0 + self._write_log_lines(batch) # -- the TUI's own log ------------------------------------------------- diff --git a/tests/test_tui.py b/tests/test_tui.py index 6e2e4620..ce742ddd 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -1407,7 +1407,8 @@ async def scenario(): async with app.run_test() as pilot: await pilot.pause() app._restart_logs(ALL_SERVICES) # the gateway's own lines - await app.workers.wait_for_complete() # drain the log stream + await app.workers.wait_for_complete() # the stream has ended + app._drain_logs() # lines are drawn in batches await pilot.pause() assert any('ready' in line for line in app._log_lines) @@ -1436,6 +1437,7 @@ async def scenario(): await pilot.pause() app._restart_logs(ALL_SERVICES) # the gateway's own lines await app.workers.wait_for_complete() + app._drain_logs() # lines are drawn in batches await pilot.pause() text = '\n'.join(app._log_lines) assert 'Traceback (most recent call last):' not in text @@ -1563,6 +1565,10 @@ async def scenario(): app = InferStackTUI(controller, catalog, interval=999, proc_factory=lambda svc: None) async with app.run_test() as pilot: + await pilot.pause() + # The mount-time background refresh must land first, or it can + # repaint all 40 rows after the rebuild below (an intermittent fail). + await app.workers.wait_for_complete() await pilot.pause() table = app.query_one('#leases', DataTable) assert table.row_count == 40 @@ -2003,3 +2009,60 @@ async def scenario(): _run(scenario) assert 'CLI: infer-stack release --all --yes' in seen['applog'] assert 'pulling img: 1 of 4 layers' in seen['applog'] + + +def test_a_ui_thread_stall_is_reported_with_where_it_happened(tmp_path, monkeypatch): + """"The TUI froze" must come with what it was doing.""" + import time + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + monkeypatch.setattr(InferStackTUI, 'error_log_path', lambda self: tmp_path / 'e.log') + seen = {} + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause(0.3) + original = app._update_summary + + def slow(*args, **kwargs): + time.sleep(0.9) # a blocking call on the UI thread + return original(*args, **kwargs) + + app._update_summary = slow + app._refresh_now() + app._update_summary = original + await pilot.pause(0.5) # the watchdog reports after it ends + seen['applog'] = '\n'.join(app._app_log_lines) + + _run(scenario) + assert 'UI stalled' in seen['applog'] + assert 'in slow' in seen['applog'] or 'tui.py' in seen['applog'] + assert 'time.sleep(0.9)' in (tmp_path / 'e.log').read_text() + + +def test_a_log_flood_is_drawn_in_bounded_batches(): + from infer_stack.tui import ALL_SERVICES, LOG_PANE_LINES, InferStackTUI + + controller, catalog = _ctx() + lines = [f'vllm-x | loading shard {i}\n' for i in range(5000)] + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: _FakeProc(lines)) + async with app.run_test() as pilot: + await pilot.pause() + app._restart_logs(ALL_SERVICES) + await app.workers.wait_for_complete() + for _ in range(20): # a bounded batch per tick + app._drain_logs() + await pilot.pause() + shown = list(app._log_lines) + assert len(shown) <= 2 * LOG_PANE_LINES + assert shown[-1].endswith('loading shard 4999') # newest kept + assert any('earlier line(s) not shown' in s for s in shown) + + _run(scenario) From da718469cb15fbbf1367f6f7aca22bf23e5ce837 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 17:29:30 -0400 Subject: [PATCH 15/34] Changelog: the TUI stops repainting what did not change Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ef0d357..640f0de6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,18 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### The TUI stops repainting what did not change + +Every refresh tick repainted the lease and deployment panes, because setting +a pane's border title repaints it even when the text is unchanged: about +7 KB/s of terminal output with nothing on screen changing, which is what an +SSH session feels. Titles are now set only on change (75 bytes/s idle). +Streamed engine logs are drawn in batches of up to 200 lines every 100 ms +instead of one UI message per line, and a flood keeps its newest lines. A +watchdog reports any UI-thread stall over 0.5 s in the TUI log, with the +stack it was stuck in written to the error log file. `dev/profile_tui.py` +measures event-loop lag for idle, docker-pane and log-flood cases. + ### Ledger reads are safe across threads The ledger shares one SQLite connection between threads, but only write From 4feff29dcc32e0d6c4f144bc1a3fdbb223b8c9d1 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 17:46:17 -0400 Subject: [PATCH 16/34] TUI: an activity line for running actions; faster launch and quit The user's report: launch felt laggy, and a slow action gave no sign it was working. Measured on a real terminal with the new `infer-stack tui --exit_after_paint` (quits once the first frame is drawn, then prints when that was after process start): - Activity line: while an action runs in a background worker (acquire, release, evict, apply, save endpoint, inspecting GPUs, API calls), a line above the status bar shows a spinner, what, for how long, and the newest backend progress (image-pull layers). Driven by worker state changes, so no action handler had to change; status hints cannot overwrite it. - `requests` is imported on first use instead of when the backend is built: opening the controller went from 127 ms to 33 ms. - The API, UI and Settings tabs are built right after the first frame (textual Lazy) instead of before it. - On exit, in-flight read-only docker queries (ps, inspect, logs, ...) are killed so quitting does not wait for the background refresh's `docker compose ps`. Mutating commands are never killed. Real terminal, guest: launch+quit 1.15 s -> ~1.0 s, first frame ~0.74 s -> ~0.69 s. The rest is imports (textual, the CLI) and Textual's own mount. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- infer_stack/cli/commands_leasing.py | 7 ++ infer_stack/leasing/compose.py | 55 ++++++++- infer_stack/leasing/gateway.py | 19 ++- infer_stack/tui.py | 173 +++++++++++++++++++++++++++- tests/test_tui.py | 38 ++++++ 5 files changed, 279 insertions(+), 13 deletions(-) diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index e5bbc64c..18d1f3ff 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -1702,6 +1702,12 @@ class TuiCLI(_LeasingCommonMixin): catalog = scfg.Value(None, type=str, help='Path to catalog.yaml.') interval = scfg.Value(3.0, type=float, help='Auto-refresh interval (s).') + exit_after_paint = scfg.Value( + False, isflag=True, + help='Quit as soon as the first frame is drawn, then print when that ' + 'was (for measuring startup: `time infer-stack tui ' + '--exit_after_paint`).', + ) @classmethod def main(cls, argv=True, **kwargs): @@ -1730,6 +1736,7 @@ def main(cls, argv=True, **kwargs): return run_tui( controller, catalog, interval=float(config.interval), catalog_path=str(catalog_path), + exit_after_paint=bool(config.exit_after_paint), ) diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index d564b7f4..ea44570f 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -746,6 +746,52 @@ def docker_environment(environ: dict[str, str] | None = None) -> dict[str, str]: return env +#: Read-only docker queries in flight (ps, inspect, ...). A long-running +#: process that is exiting (the TUI) kills these rather than wait for them; +#: nothing that changes state is ever in this set. +_RUNNING_QUERIES: set = set() + + +_READ_VERBS = frozenset({'ps', 'inspect', 'logs', 'version', 'images', 'ls', + 'info', 'port', 'top', 'config'}) +_MUTATING_VERBS = frozenset({'up', 'down', 'rm', 'create', 'pull', 'run', 'exec', + 'start', 'stop', 'kill', 'pause', 'unpause', 'connect', + 'disconnect', 'prune', 'build', 'push', 'tag', 'restart'}) + + +def _read_only(args: list[str]) -> bool: + """A docker command that only reads state, and so is safe to abandon. + + Example: + >>> _read_only(['docker', 'compose', '-p', 'x', 'ps', '--format', 'json']) + True + >>> _read_only(['docker', 'network', 'create', 'n']) + False + """ + words = set(args) + return bool(words & _READ_VERBS) and not words & _MUTATING_VERBS + + +def cancel_running_queries() -> int: + """Kill every in-flight read-only docker query; how many were killed. + + For a process that is exiting: its background refresh would otherwise + hold the exit until ``docker compose ps`` returns (measured 0.66 s on the + guest, more on a loaded host). Mutations (up, rm, pull) are never killed. + """ + import os + import signal + + killed = 0 + for proc in list(_RUNNING_QUERIES): + try: + os.killpg(proc.pid, signal.SIGKILL) + killed += 1 + except (ProcessLookupError, PermissionError): + pass + return killed + + def _default_docker_run( args: list[str], *, timeout: float | None = None, stderr_lines: Callable[[str], None] | None = None, @@ -771,6 +817,9 @@ def _default_docker_run( env=docker_environment(), stderr=subprocess.PIPE if stderr_lines is not None else None, ) + query = _read_only(args) + if query: + _RUNNING_QUERIES.add(proc) def kill_group(): try: os.killpg(proc.pid, signal.SIGKILL) @@ -794,6 +843,9 @@ def kill_group(): # session, so without this it would keep running unattended. kill_group() raise + finally: + if query: + _RUNNING_QUERIES.discard(proc) if stderr_lines is not None: for line in (err or '').splitlines(): if line.strip(): @@ -967,9 +1019,6 @@ def __init__( # before the first frame. self._inventory = inventory self.run = run or _default_docker_run - if http is None: - import requests - http = requests # The front door: gateway settings, keys, routes. Created before the # settings below, which are stored on it. self.gateway = Gateway( diff --git a/infer_stack/leasing/gateway.py b/infer_stack/leasing/gateway.py index cf549633..433ee108 100644 --- a/infer_stack/leasing/gateway.py +++ b/infer_stack/leasing/gateway.py @@ -938,14 +938,25 @@ def __init__( self.reverse_proxy = reverse_proxy self.reverse_proxy_port = reverse_proxy_port self.dynamic_routing = dynamic_routing - if http is None: - import requests - http = requests - self.http = http + # None: `requests`, imported on first use. It costs ~75 ms, and a TUI + # launch or a ledger-only command never makes a request. + self._http = http self._sleep = sleep self._clock = clock self.assume_yes = True + @property + def http(self) -> Any: + if self._http is None: + import requests + + self._http = requests + return self._http + + @http.setter + def http(self, value: Any) -> None: + self._http = value + @property def litellm_port(self) -> int: return self.ports.get('litellm', DEFAULT_PORTS['litellm']) diff --git a/infer_stack/tui.py b/infer_stack/tui.py index 3fc5b0de..68fe105e 100644 --- a/infer_stack/tui.py +++ b/infer_stack/tui.py @@ -38,6 +38,9 @@ from textual.binding import Binding from textual.coordinate import Coordinate from textual.containers import Horizontal, Vertical, VerticalScroll +from textual.css.query import NoMatches +from textual.lazy import Lazy +from textual.widget import Widget from textual.screen import ModalScreen from textual.theme import Theme from textual.widgets import ( @@ -107,6 +110,48 @@ #: A UI-thread stall at least this long is reported, with what was running. STALL_REPORT_S = 0.5 +#: What a running background action is doing, shown in the activity line, by +#: worker method. A worker not listed here (the refresh, log streams) is +#: routine and shows nothing. +ACTIVITY_VERBS = { + '_do_acquire': 'acquiring', + '_do_release': 'releasing', + '_do_release_all': 'releasing all leases', + '_do_evict': 'evicting', + '_do_evict_all': 'evicting idle deployments', + '_do_cleanup': 'cleaning up the ledger', + '_do_apply': 'applying', + '_do_compose': 'docker compose', + '_save_endpoint': 'saving endpoint', + '_do_suggest': 'inspecting GPUs for a suggestion', + '_prepare_endpoint_editor': 'inspecting GPUs', + '_do_api_send': 'waiting for the model', + '_do_api_test_all': 'testing every model', + '_do_api_list': 'listing models', +} +SPINNER = '⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏' + + +def activity_label(description: str) -> str | None: + """``'acquiring qwen'`` from a worker description like ``_do_acquire('qwen')``. + + Example: + >>> activity_label("_do_acquire('qwen')") + 'acquiring qwen' + >>> activity_label('_do_apply()') + 'applying' + >>> activity_label('_refresh_bg()') is None + True + """ + import re + + method, _, rest = str(description).partition('(') + verb = ACTIVITY_VERBS.get(method.strip()) + if verb is None: + return None + first = re.match(r"\s*'([^']*)'", rest) + return f'{verb} {first.group(1)}' if first else verb + SELECT_BLANK = next( ( candidate @@ -793,6 +838,30 @@ def on_button_pressed(self, event: Button.Pressed) -> None: self.dismiss(event.button.id == 'ok') +class _Section(Widget): + """A tab's content, built from ``compose_fn`` when it mounts. + + Wrapped in :class:`textual.lazy.Lazy` for tabs hidden at launch, so the + first frame does not wait for widgets nobody can see yet (CSS and layout + of the whole tree was most of the launch time). The app is told when the + content exists, to fill anything it set before then. + """ + + DEFAULT_CSS = '_Section { height: 1fr; }' + + def __init__(self, compose_fn: Callable[[], ComposeResult], *, id: str): + super().__init__(id=id) + self._compose_fn = compose_fn + + def compose(self) -> ComposeResult: + yield from self._compose_fn() + + def on_mount(self) -> None: + on_section = getattr(self.app, '_on_section_mounted', None) + if on_section is not None: + on_section(self.id) + + class InferStackTUI(App): """Monitor + control the leasing stack across panes.""" @@ -886,6 +955,7 @@ class InferStackTUI(App): #api-extra Button { margin: 0 1 0 0; min-width: 12; } #status { dock: bottom; height: 1; padding: 0 2; color: $text-muted; } + #activity { dock: bottom; height: 1; padding: 0 2; color: $warning; display: none; } #settings, #ui-settings { padding: 1 2; } #settings Label, #ui-settings Label { margin: 1 0 0 0; color: $text-muted; } @@ -936,8 +1006,13 @@ def __init__( proc_factory: Callable[[str | None], Any] | None = None, catalog_path: str | Path | None = None, http: Any = None, + exit_after_paint: bool = False, ) -> None: super().__init__() + #: Quit once the first frame is drawn (``tui --exit_after_paint``). + self._exit_after_paint = exit_after_paint + #: Seconds from process start to the first drawn frame, once known. + self.first_frame_s: float | None = None self.controller = controller self.catalog = catalog self.interval = interval @@ -1003,6 +1078,10 @@ def __init__( self._log_dropped = 0 # Parsed compose service names, keyed by the file's (mtime, size). self._service_names_cache: tuple[tuple[float, int], list[str]] | None = None + # Running background actions: worker -> (label, started). Drawn by + # _draw_activity, with the newest backend progress message. + self._activity: dict[Any, tuple[str, float]] = {} + self._activity_note = '' # The UI loop's heartbeat, read by the stall watchdog thread. self._beat = time.monotonic() self._watchdog_stop = None @@ -1037,15 +1116,17 @@ def compose(self) -> ComposeResult: with TabbedContent(id='top'): with TabPane('Dashboard', id='tab-dashboard'): yield from self._compose_dashboard() + # Hidden at launch: built right after the first frame, not before. with TabPane('API', id='tab-api'): - yield from self._compose_api() + yield Lazy(_Section(self._compose_api, id='section-api')) with TabPane('UI', id='tab-ui'): - yield from self._compose_ui_settings() + yield Lazy(_Section(self._compose_ui_settings, id='section-ui')) with TabPane('Settings', id='tab-settings'): - yield from self._compose_settings() + yield Lazy(_Section(self._compose_settings, id='section-settings')) with TabPane(APP_LOG_TAB_TITLE, id='tab-applog'): yield from self._compose_app_log() yield Static('', id='status') + yield Static('', id='activity') yield Footer() def _compose_app_log(self) -> ComposeResult: @@ -1314,13 +1395,28 @@ def on_mount(self) -> None: self.ledger_interval, self.action_refresh ) self.set_interval(LOG_DRAIN_S, self._drain_logs) + # Held directly: query_one searches the active screen, which is a + # dialog's while one is open. + self._activity_widget = self.query_one('#activity', Static) + self.set_interval(0.1, self._draw_activity) self._start_stall_watchdog() self.query_one('#endpoints', DataTable).focus() + self.call_after_refresh(self._first_frame_drawn) + + def _first_frame_drawn(self) -> None: + self.first_frame_s = _seconds_since_process_start() + if self._exit_after_paint: + self.exit() def on_unmount(self) -> None: self._terminate_logs() if self._watchdog_stop is not None: self._watchdog_stop.set() + # Quit must not wait for the background refresh's `docker compose ps`: + # Python joins worker threads at exit. Read-only queries only. + from .leasing.compose import cancel_running_queries + + cancel_running_queries() # -- responsiveness ------------------------------------------------------ @@ -1416,8 +1512,34 @@ def _backend_progress(self, message: str) -> None: except RuntimeError: # already on the UI thread self._show_progress(message) + def _on_section_mounted(self, section_id: str | None) -> None: + """A deferred tab now exists: fill what was set before it did.""" + if section_id == 'section-api': + self._update_api_urls() + self._update_api_curl() + self._sync_api_models(list(getattr(self, '_api_models_wanted', []))) + def _show_progress(self, message: str) -> None: self._status(message) # also recorded in the TUI log + self._activity_note = message # and beside the running action + + def _draw_activity(self) -> None: + """The activity line: what is running, for how long, and its progress.""" + widget = getattr(self, '_activity_widget', None) + if widget is None: + return + if not self._activity: + if widget.display: + widget.display = False + self._activity_note = '' + return + now = time.monotonic() + spin = SPINNER[int(now * 10) % len(SPINNER)] + parts = [f'{label} · {now - started:.0f}s' + for label, started in sorted(self._activity.values(), key=lambda v: v[1])] + note = f' — {self._activity_note}' if self._activity_note else '' + widget.update(f'{spin} ' + ' | '.join(parts) + note) + widget.display = True # -- resizable panes --------------------------------------------------- @@ -1534,10 +1656,14 @@ def _cached_label(source: str, hub: Path | None) -> str: def _sync_api_models(self, names: list[str]) -> None: """Point the API model selector at the currently-ready endpoints only.""" + self._api_models_wanted = names if names == self._ready_endpoints: return + try: + select = self.query_one('#api-model', Select) + except NoMatches: + return # the API tab is not built yet; filled when it is self._ready_endpoints = names - select = self.query_one('#api-model', Select) current = None if _select_is_blank(select.value) else select.value select.set_options([(n, n) for n in names]) if current in names: @@ -2292,6 +2418,14 @@ def on_worker_state_changed(self, event) -> None: worker = event.worker name = worker.name or worker.group or 'worker' + if event.state is WorkerState.RUNNING: + label = activity_label(worker.description) + if label is not None: + self._activity[worker] = (label, time.monotonic()) + self._draw_activity() + elif event.state in (WorkerState.SUCCESS, WorkerState.ERROR, + WorkerState.CANCELLED): + self._activity.pop(worker, None) if event.state is WorkerState.ERROR: error = getattr(worker, 'error', None) if error is not None: @@ -3220,7 +3354,10 @@ def _api_log(self, line: str) -> None: self.query_one('#api-out', RichLog).write(line) def _selected_api_model(self) -> str | None: - value = self.query_one('#api-model', Select).value + try: + value = self.query_one('#api-model', Select).value + except NoMatches: + return None # the API tab is built right after the first frame return None if _select_is_blank(value) else str(value) def action_api_send(self) -> None: @@ -3417,12 +3554,29 @@ def _fmt_ports(row: dict) -> str: return ', '.join(bits) +def _seconds_since_process_start() -> float | None: + """Wall time since this process started (Linux ``/proc``), else ``None``.""" + import os + + try: + with open('/proc/self/stat') as handle: + # Field 22 counts clock ticks since boot; the command name (field 2) + # may contain spaces, so split after its closing parenthesis. + start_ticks = int(handle.read().rsplit(')', 1)[1].split()[19]) + with open('/proc/uptime') as handle: + uptime = float(handle.read().split()[0]) + return uptime - start_ticks / os.sysconf('SC_CLK_TCK') + except (OSError, ValueError, IndexError): + return None + + def run_tui( controller, catalog, *, interval: float = 3.0, catalog_path: str | Path | None = None, + exit_after_paint: bool = False, ) -> int: """Run the TUI against a built controller + catalog. Returns an exit code.""" # The narration loguru sink writes to stderr, which would corrupt the @@ -3434,7 +3588,14 @@ def run_tui( except Exception: # noqa: BLE001 pass app: Any = InferStackTUI( - controller, catalog, interval=interval, catalog_path=catalog_path + controller, catalog, interval=interval, catalog_path=catalog_path, + exit_after_paint=exit_after_paint, ) app.run() + if exit_after_paint: + first = app.first_frame_s + exited = _seconds_since_process_start() + print(f'first frame {first:.2f}s after process start; exited at {exited:.2f}s' + if first is not None and exited is not None + else 'first frame drawn (process start time unavailable)') return 0 diff --git a/tests/test_tui.py b/tests/test_tui.py index ce742ddd..66a111cd 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -2066,3 +2066,41 @@ async def scenario(): assert any('earlier line(s) not shown' in s for s in shown) _run(scenario) + + +def test_a_running_action_shows_in_the_activity_line_until_it_ends(): + """A slow action is visibly working: spinner, what, and for how long.""" + import threading + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + name = next(iter(catalog.endpoints)) + gate = threading.Event() + real_acquire = controller.acquire + + def slow_acquire(*args, **kwargs): + gate.wait(5) + return real_acquire(*args, **kwargs) + + controller.acquire = slow_acquire + seen = {} + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + app._start_acquire(name) + await pilot.pause(0.3) + line = app.query_one('#activity') + seen['during'] = (line.display, str(line.render())) + gate.set() + await app.workers.wait_for_complete() + await pilot.pause(0.3) + seen['after'] = line.display + + _run(scenario) + shown, text = seen['during'] + assert shown and f'acquiring {name}' in text and 's' in text + assert seen['after'] is False From a3d18c10c0c678bb8968cb16d19c99c98401801f Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 17:57:39 -0400 Subject: [PATCH 17/34] Switch the CLI from scriptconfig to kwconf kwconf is scriptconfig's successor and what the sibling packages (aiq-magnet, cmd_queue, kwdagger) already use. Mechanical where the APIs match: `import kwconf as kw`, scfg.Value -> kw.Value, scfg.ModalCLI -> kw.ModalCLI, scfg.DataConfig -> kw.Config (233 uses in 9 files). Differences that needed a change: - kwconf's automatic conversion reads `null` as None, so `--backend null` failed its choices; the 8 options with string choices get type=str. - ModalCLI names the program after the class ("ManageCLI"); `__prog__` keeps `infer-stack` in usage and error messages. - The TUI's activity_label doctest needs textual, which CI does not install; it is marked `+REQUIRES(module:textual)`. Dependency: kwconf>=0.11.0. The repo's uv exclude-newer cutoff (2026-06-04) predates every kwconf release, so kwconf alone is admitted up to 2026-08-08 (0.11.0, the version tested) via exclude-newer-package; uv.lock and the exported test lock only gain kwconf and lose scriptconfig. Checked: full suite 909 passed; CI's checks (flake8, ty, the package plus tests with only the tests extra) pass on 3.14 and 3.10 (838 each); CLI smoke: help, --version, subcommands, --no-wait, unknown flags rejected. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 15 +++ README.md | 2 +- docs/source/conf.py | 2 +- infer_stack/cli/__init__.py | 16 ++- infer_stack/cli/commands_catalog.py | 128 +++++++++--------- infer_stack/cli/commands_leasing.py | 200 ++++++++++++++-------------- infer_stack/cli/commands_meta.py | 32 ++--- infer_stack/cli/commands_mock.py | 24 ++-- infer_stack/cli/commands_runtime.py | 40 +++--- infer_stack/cli/context.py | 2 +- infer_stack/cli/options.py | 54 ++++---- infer_stack/tui.py | 1 + pyproject.toml | 6 +- requirements/locks/tests.txt | 12 +- tests/test_cli_equivalent.py | 4 +- uv.lock | 31 +++-- 16 files changed, 294 insertions(+), 275 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 640f0de6..571b00d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,21 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### The CLI is built on kwconf instead of scriptconfig + +kwconf is scriptconfig's successor, which aiq-magnet, cmd_queue and kwdagger +already use. `Value`, `ModalCLI` and `__command__` carry over; `DataConfig` +is now `Config`. One behavioural difference needed care: kwconf reads the +strings `null`, `true` and numbers on the command line as values, so +`--backend null` arrived as `None`. Options whose choices are strings now +take `type=str`. The program name stays `infer-stack` in usage and errors +(`__prog__`). + +The repository's uv `exclude-newer` cutoff (2026-06-04) predates every kwconf +release, so kwconf alone gets an exception up to 0.11.0, the version tested. +`uv.lock` and `requirements/locks/tests.txt` were regenerated: kwconf 0.11.0 +added, scriptconfig removed. + ### The TUI stops repainting what did not change Every refresh tick repainted the lease and deployment panes, because setting diff --git a/README.md b/README.md index 642c5ee3..fc51de66 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ infer-stack version # print the installed version infer-stack config paths # show where config / artifacts / caches live ``` -The CLI is built on [`scriptconfig`](https://gitlab.kitware.com/utils/scriptconfig), +The CLI is built on [`kwconf`](https://github.com/Erotemic/kwconf), so every subcommand is also importable as a Python class — useful for notebooks, tests, and other scripts: diff --git a/docs/source/conf.py b/docs/source/conf.py index d28cf6dc..0818d132 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -235,7 +235,7 @@ def visit_Assign(self, node): 'ubelt': ('https://ubelt.readthedocs.io/en/latest/', None), 'xdoctest': ('https://xdoctest.readthedocs.io/en/latest/', None), 'networkx': ('https://networkx.org/documentation/stable/', None), - 'scriptconfig': ('https://scriptconfig.readthedocs.io/en/latest/', None), + 'kwconf': ('https://kwconf.readthedocs.io/en/latest/', None), 'kwconf': ('https://kwconf.readthedocs.io/en/latest/', None), 'rich': ('https://rich.readthedocs.io/en/latest/', None), 'numpy': ('https://numpy.org/doc/stable/', None), diff --git a/infer_stack/cli/__init__.py b/infer_stack/cli/__init__.py index 94c81a8d..eb00a120 100644 --- a/infer_stack/cli/__init__.py +++ b/infer_stack/cli/__init__.py @@ -1,9 +1,9 @@ #!/usr/bin/env python3 # PYTHON_ARGCOMPLETE_OK -"""scriptconfig-based CLI for infer-stack. +"""kwconf-based CLI for infer-stack. -Each subcommand is a ``scfg.DataConfig`` subclass; ``ManageCLI`` composes -them into a single ``scfg.ModalCLI`` exposed as the ``infer-stack`` entry +Each subcommand is a ``kw.Config`` subclass; ``ManageCLI`` composes +them into a single ``kw.ModalCLI`` exposed as the ``infer-stack`` entry point. Because every subcommand is a ``DataConfig``, the same class can be invoked from the shell (``infer-stack render --profile X``) or from Python (``RenderCLI.main(argv=False, profile='X')``). @@ -22,7 +22,7 @@ from __future__ import annotations -import scriptconfig as scfg +import kwconf as kw from .. import __version__ @@ -78,7 +78,11 @@ # --------------------------------------------------------------------------- -class ManageCLI(scfg.ModalCLI): +class ManageCLI(kw.ModalCLI): + # The program name in usage and error messages (kwconf would use the class + # name, "ManageCLI"). + __prog__ = 'infer-stack' + description = ( 'Lease, acquire, and run LLM endpoints. Primary workflow: ' 'catalog -> acquire/run. See `infer-stack help tree`.' @@ -104,7 +108,7 @@ class ManageCLI(scfg.ModalCLI): actual. Run `infer-stack help tree` for the whole command surface. """ - # Backs the modal ``--version`` flag (scriptconfig reads ``__version__``). + # Backs the modal ``--version`` flag (kwconf reads ``__version__``). # The ``version`` *subcommand* is registered below under a non-colliding # attribute name; its CLI name comes from ``VersionCLI.__command__``. __version__ = __version__ diff --git a/infer_stack/cli/commands_catalog.py b/infer_stack/cli/commands_catalog.py index 38a5abcc..d68e62e0 100644 --- a/infer_stack/cli/commands_catalog.py +++ b/infer_stack/cli/commands_catalog.py @@ -16,7 +16,7 @@ from pathlib import Path from typing import Any -import scriptconfig as scfg +import kwconf as kw import yaml from ..leasing import Catalog, CatalogError @@ -223,8 +223,8 @@ def _parse_kv(items) -> dict[str, Any]: class _CatalogCommon(_PathOverridesMixin): - catalog = scfg.Value(None, type=str, help='Catalog path (default: config dir).') - dry_run = scfg.Value( + catalog = kw.Value(None, type=str, help='Catalog path (default: config dir).') + dry_run = kw.Value( False, isflag=True, help='Print the resulting YAML, do not write.' ) @@ -233,7 +233,7 @@ class CatalogInitCLI(_CatalogCommon): """Write a starter catalog.yaml (empty sections) if none exists.""" __command__ = 'init' - force = scfg.Value(False, isflag=True, help='Overwrite an existing catalog.') + force = kw.Value(False, isflag=True, help='Overwrite an existing catalog.') @classmethod def main(cls, argv=True, **kwargs): @@ -275,12 +275,12 @@ class CatalogSuggestCLI( """ __command__ = 'suggest' - catalog = scfg.Value(None, type=str, help='Catalog path (default: config dir).') - apply = scfg.Value( + catalog = kw.Value(None, type=str, help='Catalog path (default: config dir).') + apply = kw.Value( False, isflag=True, help='Merge the suggestion into the catalog (default: render only).', ) - force = scfg.Value( + force = kw.Value( False, isflag=True, help='With --apply, overwrite catalog entries that already exist.', ) @@ -363,7 +363,7 @@ class CatalogPathCLI(_PathOverridesMixin): """Print the catalog path.""" __command__ = 'path' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -392,8 +392,8 @@ class CatalogShowCLI(_PathOverridesMixin): """Pretty-print the whole catalog (or one named entry across sections).""" __command__ = 'show' - catalog = scfg.Value(None, type=str) - name = scfg.Value(None, position=1, type=str, help='Optional entry name.') + catalog = kw.Value(None, type=str) + name = kw.Value(None, position=1, type=str, help='Optional entry name.') @classmethod def main(cls, argv=True, **kwargs): @@ -417,7 +417,7 @@ class CatalogValidateCLI(_PathOverridesMixin): """Parse + cross-reference check the catalog.""" __command__ = 'validate' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -440,7 +440,7 @@ class CatalogEditCLI(_PathOverridesMixin): """Open the catalog in $EDITOR (escape hatch), then validate it.""" __command__ = 'edit' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -470,12 +470,12 @@ class ModelAddCLI(_CatalogCommon): """Add (or --force overwrite) a model: a Hugging Face / local weight source.""" __command__ = 'add' - name = scfg.Value(None, position=1, type=str) - source = scfg.Value(None, type=str, help='e.g. hf://org/Model or a path.') - revision = scfg.Value(None, type=str) - quantization = scfg.Value(None, type=str) - dtype = scfg.Value(None, type=str) - force = scfg.Value(False, isflag=True) + name = kw.Value(None, position=1, type=str) + source = kw.Value(None, type=str, help='e.g. hf://org/Model or a path.') + revision = kw.Value(None, type=str) + quantization = kw.Value(None, type=str) + dtype = kw.Value(None, type=str) + force = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -503,7 +503,7 @@ def main(cls, argv=True, **kwargs): class ModelListCLI(_PathOverridesMixin): """List model names.""" __command__ = 'list' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -513,8 +513,8 @@ def main(cls, argv=True, **kwargs): class ModelShowCLI(_PathOverridesMixin): """Show a model entry, or all of them when no NAME is given.""" __command__ = 'show' - catalog = scfg.Value(None, type=str) - name = scfg.Value(None, position=1, type=str, + catalog = kw.Value(None, type=str) + name = kw.Value(None, position=1, type=str, help='Model name; omit to show every model.') @classmethod @@ -526,7 +526,7 @@ def main(cls, argv=True, **kwargs): class ModelRmCLI(_CatalogCommon): """Remove one or more models by name.""" __command__ = 'rm' - names = scfg.Value([], nargs='+', position=1, type=str, + names = kw.Value([], nargs='+', position=1, type=str, help='Model name(s) to remove.') @classmethod @@ -535,7 +535,7 @@ def main(cls, argv=True, **kwargs): return _rm(config, 'models', config.names) -class CatalogModelCLI(scfg.ModalCLI): +class CatalogModelCLI(kw.ModalCLI): """Manage catalog models.""" __command__ = 'model' add = ModelAddCLI @@ -562,55 +562,55 @@ class EndpointAddCLI(_CatalogCommon): """ __command__ = 'add' - name = scfg.Value( + name = kw.Value( None, position=1, type=str, help='Endpoint alias (default: {model}-N, auto-incrementing).', ) - engine = scfg.Value('vllm', choices=['vllm', 'ollama']) - model = scfg.Value(None, type=str, help='Model name (vllm) or tag (ollama).') - host = scfg.Value(None, type=str, help='Runtime host (ollama).') - public_name = scfg.Value( + engine = kw.Value('vllm', type=str, choices=['vllm', 'ollama']) + model = kw.Value(None, type=str, help='Model name (vllm) or tag (ollama).') + host = kw.Value(None, type=str, help='Runtime host (ollama).') + public_name = kw.Value( None, type=str, help='Served/public name (for coalescing aliases).' ) - reclaim = scfg.Value( - None, choices=['keep-warm', 'stop', 'scale-to-zero'], + reclaim = kw.Value( + None, type=str, choices=['keep-warm', 'stop', 'scale-to-zero'], help='Reclaim policy when idle.', ) - protocol = scfg.Value( - None, choices=['chat', 'completions'], + protocol = kw.Value( + None, type=str, choices=['chat', 'completions'], help='Which OpenAI surface this endpoint serves. Load-bearing twice: ' 'a base model has no chat template, and the readiness probe ' 'follows this — declaring chat for a completions-only serve ' 'blocks `acquire` until the TTL. Default (unset): chat.', ) - min_vram_gib = scfg.Value( + min_vram_gib = kw.Value( None, type=float, help='placement.min_vram_gib — the VRAM this endpoint needs, so the ' 'planner can pick any eligible free GPU. Declaring this is what ' 'lets one catalog be correct on every host.', ) - gpu = scfg.Value( + gpu = kw.Value( [], nargs='*', type=int, help='placement.gpu_indices — exact physical GPU index/indices. Omit ' 'for automatic VRAM-aware placement. This is a local operator ' 'override and is intentionally less portable than --min-vram-gib.', ) # vLLM runtime conveniences - max_model_len = scfg.Value(None, type=int) - gpu_mem = scfg.Value( + max_model_len = kw.Value(None, type=int) + gpu_mem = kw.Value( None, type=float, help='gpu_memory_utilization (0-1).' ) - tensor_parallel = scfg.Value(None, type=int) - extra_args = scfg.Value( + tensor_parallel = kw.Value(None, type=int) + extra_args = kw.Value( None, type=str, help="Raw vLLM flags as one string (shell-split), " "e.g. --extra-args='--dtype=half --enforce-eager'.", ) - runtime = scfg.Value( + runtime = kw.Value( [], nargs='*', type=str, help='Extra runtime KEY=VALUE pairs (YAML-typed).', ) - force = scfg.Value(False, isflag=True) + force = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -681,7 +681,7 @@ def main(cls, argv=True, **kwargs): class EndpointListCLI(_PathOverridesMixin): """List endpoint names.""" __command__ = 'list' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -691,8 +691,8 @@ def main(cls, argv=True, **kwargs): class EndpointShowCLI(_PathOverridesMixin): """Show an endpoint entry, or all of them when no NAME is given.""" __command__ = 'show' - catalog = scfg.Value(None, type=str) - name = scfg.Value(None, position=1, type=str, + catalog = kw.Value(None, type=str) + name = kw.Value(None, position=1, type=str, help='Endpoint name; omit to show every endpoint.') @classmethod @@ -704,7 +704,7 @@ def main(cls, argv=True, **kwargs): class EndpointRmCLI(_CatalogCommon): """Remove one or more endpoints by name.""" __command__ = 'rm' - names = scfg.Value([], nargs='+', position=1, type=str, + names = kw.Value([], nargs='+', position=1, type=str, help='Endpoint name(s) to remove.') @classmethod @@ -713,7 +713,7 @@ def main(cls, argv=True, **kwargs): return _rm(config, 'endpoints', config.names) -class CatalogEndpointCLI(scfg.ModalCLI): +class CatalogEndpointCLI(kw.ModalCLI): """Manage catalog endpoints.""" __command__ = 'endpoint' add = EndpointAddCLI @@ -731,15 +731,15 @@ class HostAddCLI(_CatalogCommon): """Add (or --force overwrite) a runtime host (e.g. an Ollama daemon).""" __command__ = 'add' - name = scfg.Value(None, position=1, type=str) - engine = scfg.Value('ollama', choices=['ollama']) - gpu = scfg.Value([], nargs='*', type=int, help='GPU index/indices.') - keep_alive = scfg.Value(None, type=str, help='Ollama keep_alive, e.g. 5m.') - num_parallel = scfg.Value(None, type=int) - max_loaded_models = scfg.Value(None, type=int) - context_length = scfg.Value(None, type=int) - image = scfg.Value(None, type=str) - force = scfg.Value(False, isflag=True) + name = kw.Value(None, position=1, type=str) + engine = kw.Value('ollama', type=str, choices=['ollama']) + gpu = kw.Value([], nargs='*', type=int, help='GPU index/indices.') + keep_alive = kw.Value(None, type=str, help='Ollama keep_alive, e.g. 5m.') + num_parallel = kw.Value(None, type=int) + max_loaded_models = kw.Value(None, type=int) + context_length = kw.Value(None, type=int) + image = kw.Value(None, type=str) + force = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -780,7 +780,7 @@ def main(cls, argv=True, **kwargs): class HostListCLI(_PathOverridesMixin): """List runtime-host names.""" __command__ = 'list' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -790,7 +790,7 @@ def main(cls, argv=True, **kwargs): class HostRmCLI(_CatalogCommon): """Remove one or more runtime hosts by name.""" __command__ = 'rm' - names = scfg.Value([], nargs='+', position=1, type=str, + names = kw.Value([], nargs='+', position=1, type=str, help='Runtime-host name(s) to remove.') @classmethod @@ -799,7 +799,7 @@ def main(cls, argv=True, **kwargs): return _rm(config, 'runtime_hosts', config.names) -class CatalogHostCLI(scfg.ModalCLI): +class CatalogHostCLI(kw.ModalCLI): """Manage runtime hosts (Ollama daemons / placement).""" __command__ = 'host' add = HostAddCLI @@ -816,9 +816,9 @@ class BundleAddCLI(_CatalogCommon): """Add (or --force overwrite) a bundle: a named group of endpoints.""" __command__ = 'add' - name = scfg.Value(None, position=1, type=str) - members = scfg.Value([], nargs='*', position=2, type=str) - force = scfg.Value(False, isflag=True) + name = kw.Value(None, position=1, type=str) + members = kw.Value([], nargs='*', position=2, type=str) + force = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -844,7 +844,7 @@ def main(cls, argv=True, **kwargs): class BundleListCLI(_PathOverridesMixin): """List bundle names.""" __command__ = 'list' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -854,7 +854,7 @@ def main(cls, argv=True, **kwargs): class BundleRmCLI(_CatalogCommon): """Remove one or more bundles by name.""" __command__ = 'rm' - names = scfg.Value([], nargs='+', position=1, type=str, + names = kw.Value([], nargs='+', position=1, type=str, help='Bundle name(s) to remove.') @classmethod @@ -863,7 +863,7 @@ def main(cls, argv=True, **kwargs): return _rm(config, 'bundles', config.names) -class CatalogBundleCLI(scfg.ModalCLI): +class CatalogBundleCLI(kw.ModalCLI): """Manage endpoint bundles.""" __command__ = 'bundle' add = BundleAddCLI @@ -876,7 +876,7 @@ class CatalogBundleCLI(scfg.ModalCLI): # --------------------------------------------------------------------------- -class CatalogModalCLI(scfg.ModalCLI): +class CatalogModalCLI(kw.ModalCLI): """Edit the user serving catalog (models / endpoints / hosts / bundles).""" __command__ = 'catalog' diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index 18d1f3ff..14bdb67d 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -26,7 +26,7 @@ from typing import Any from pathlib import Path -import scriptconfig as scfg +import kwconf as kw from ..env_utils import parse_env_file, write_env_file from ..leasing import ( @@ -726,24 +726,24 @@ def _do_acquire(config, *, owner: str, ttl_seconds: float | None) -> int: class _LeasingCommonMixin(_PathOverridesMixin, _AllowedGpusMixin, _DisplayGpuMixin): - backend = scfg.Value( + backend = kw.Value( None, - choices=['null', 'compose', 'kubeai'], + type=str, choices=['null', 'compose', 'kubeai'], help='Serving backend: "null" (dry-run), "compose" (single-host ' 'docker), or "kubeai" (cluster; see docs/kubeai-backend.md). ' 'Defaults to `config set backend …`, else "null".', ) - ledger = scfg.Value( + ledger = kw.Value( None, type=str, help='Path to the lease ledger sqlite db.' ) - require_generation = scfg.Value( + require_generation = kw.Value( False, isflag=True, help='Deprecated/no-op: readiness now ALWAYS verifies a real generation ' '(a listed alias or a running container is not proof the model serves). ' 'Accepted for compatibility.', ) - litellm = scfg.Value( + litellm = kw.Value( None, isflag=True, help='Render the LiteLLM gateway — one OpenAI base_url fronting every ' @@ -751,14 +751,14 @@ class _LeasingCommonMixin(_PathOverridesMixin, _AllowedGpusMixin, _DisplayGpuMix 'a lean stack where Open WebUI talks to the upstreams (e.g. an Ollama ' 'daemon) directly. Overrides `config set litellm …`.', ) - ui = scfg.Value( + ui = kw.Value( None, isflag=True, help='Render a managed Open WebUI in front of the gateway (compose ' 'backend). On by default; use --no-ui to skip. Overrides ' '`config set ui …`.', ) - reverse_proxy = scfg.Value( + reverse_proxy = kw.Value( None, isflag=True, alias=['reverse-proxy'], @@ -767,7 +767,7 @@ class _LeasingCommonMixin(_PathOverridesMixin, _AllowedGpusMixin, _DisplayGpuMix '(localhost / trusted networks only). Port + bring-your-own nginx.conf ' 'live in the `reverse_proxy` setting (`config set` / `config edit`).', ) - dynamic_routing = scfg.Value( + dynamic_routing = kw.Value( None, isflag=True, alias=['dynamic-routing'], @@ -787,14 +787,14 @@ class _ApprovalMixin(_LeasingCommonMixin): terminal; ``--yes`` (or a non-TTY) applies without prompting. """ - catalog = scfg.Value( + catalog = kw.Value( None, type=str, help='Path to catalog.yaml. release/gc/evict reconcile the gateway too, ' 'so pass the same catalog as acquire to keep the static superset route ' 'table (no gateway blip); omitted, it falls back to the default-path ' 'catalog, else legacy per-deployment routing.', ) - yes = scfg.Value( + yes = kw.Value( False, isflag=True, alias=['y'], help='Apply compose changes without showing the diff / prompting ' '(compose backend). Implied when stdout is not a terminal.', @@ -802,21 +802,21 @@ class _ApprovalMixin(_LeasingCommonMixin): class _AcquireFlagsMixin(_LeasingCommonMixin): - catalog = scfg.Value(None, type=str, help='Path to catalog.yaml.') - base_url = scfg.Value( + catalog = kw.Value(None, type=str, help='Path to catalog.yaml.') + base_url = kw.Value( 'http://127.0.0.1:14042/v1', type=str, help='Base URL written into the endpoint descriptor (dry-run placeholder).', ) - api_key_env = scfg.Value( + api_key_env = kw.Value( 'LITELLM_MASTER_KEY', type=str, help='Name of the env var holding the API key (kept out of artifacts).', ) - wait = scfg.Value( + wait = kw.Value( True, isflag=True, help='Block until ready (use --no-wait to skip).' ) - queue = scfg.Value( + queue = kw.Value( False, isflag=True, help='Admission queue: if every GPU is busy, WAIT for one to free ' '(up to --timeout) instead of failing fast. Each retry sweeps the ' @@ -824,7 +824,7 @@ class _AcquireFlagsMixin(_LeasingCommonMixin): 'waiting. Intended for batch/pipeline fan-out; interactive use ' 'defaults off (fail fast with a clear "no GPU" error).', ) - apply = scfg.Value( + apply = kw.Value( True, isflag=True, help='Apply the render (docker compose up). Use --no-apply to *stage* ' @@ -833,17 +833,17 @@ class _AcquireFlagsMixin(_LeasingCommonMixin): '(compose backend). --no-apply implies no readiness wait and no diff ' 'prompt; `release` discards a staged lease.', ) - timeout = scfg.Value(600, type=float, help='Readiness wait timeout (s).') - interval = scfg.Value(5, type=float, help='Readiness poll interval (s).') - env_file = scfg.Value( + timeout = kw.Value(600, type=float, help='Readiness wait timeout (s).') + interval = kw.Value(5, type=float, help='Readiness poll interval (s).') + env_file = kw.Value( None, type=str, help='Write the sourceable endpoint env-file here.' ) - yes = scfg.Value( + yes = kw.Value( False, isflag=True, alias=['y'], help='Apply compose changes without showing the diff / prompting ' '(compose backend). Implied when stdout is not a terminal.', ) - json = scfg.Value(False, isflag=True, help='Emit JSON instead of text.') + json = kw.Value(False, isflag=True, help='Emit JSON instead of text.') # --------------------------------------------------------------------------- @@ -901,19 +901,19 @@ class AcquireCLI(_AcquireFlagsMixin): infer-stack leases """ - names = scfg.Value( + names = kw.Value( [], nargs='*', position=1, type=str, help='Endpoint or bundle names.' ) - ttl = scfg.Value( + ttl = kw.Value( None, type=str, help='Soft TTL (e.g. 2h, 30m); default infinite.' ) - owner = scfg.Value(None, type=str, help='Lease owner (default: $USER).') - dedicated = scfg.Value( + owner = kw.Value(None, type=str, help='Lease owner (default: $USER).') + dedicated = kw.Value( False, isflag=True, help='Force a dedicated deployment instead of coalescing.', ) - reserve_gpus = scfg.Value( + reserve_gpus = kw.Value( 0, type=int, alias=['reserve-gpus'], @@ -949,7 +949,7 @@ class RenderCLI(_LeasingCommonMixin): __command__ = 'render' - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -990,12 +990,12 @@ class ApplyCLI(_ApprovalMixin): __command__ = 'apply' - wait = scfg.Value( + wait = kw.Value( False, isflag=True, help='Also block until ready after bringing it up.' ) - timeout = scfg.Value(600, type=float, help='Readiness wait timeout (s).') - interval = scfg.Value(5, type=float, help='Readiness poll interval (s).') - json = scfg.Value(False, isflag=True) + timeout = kw.Value(600, type=float, help='Readiness wait timeout (s).') + interval = kw.Value(5, type=float, help='Readiness poll interval (s).') + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1075,22 +1075,22 @@ class ReleaseCLI(_ApprovalMixin): __command__ = 'release' - lease = scfg.Value( + lease = kw.Value( None, position=1, type=str, help='Lease id (or use --env-file).' ) - env_file = scfg.Value( + env_file = kw.Value( None, type=str, help='Read the lease id from this env-file.' ) - all = scfg.Value( + all = kw.Value( False, isflag=True, help='Release every active lease (the whole stack idles/tears down).', ) - evict = scfg.Value( + evict = kw.Value( False, isflag=True, help='Also evict (tear down) the released deployment(s) now, even if their ' 'reclaim policy is keep-warm — frees the GPU immediately.', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1183,12 +1183,12 @@ class EvictCLI(_ApprovalMixin): __command__ = 'evict' - names = scfg.Value( + names = kw.Value( [], nargs='*', position=1, type=str, help='Endpoint alias or deployment id to evict.', ) - all = scfg.Value(False, isflag=True, help='Evict every idle deployment.') - json = scfg.Value(False, isflag=True) + all = kw.Value(False, isflag=True, help='Evict every idle deployment.') + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1252,17 +1252,17 @@ class GcCLI(_ApprovalMixin): __command__ = 'gc' - evict = scfg.Value( + evict = kw.Value( False, isflag=True, help='Also tear down idle keep-warm deployments (like `evict --all`), ' 'not just leaked/expired demand.', ) - orphans = scfg.Value( + orphans = kw.Value( False, isflag=True, help='Instead: remove containers in the project that infer-stack does not ' 'manage (listed and confirmed first; --yes skips the prompt).', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1337,16 +1337,16 @@ class CleanCLI(_LeasingCommonMixin): infer-stack clean -f --no-orphans # leave unmanaged containers alone """ - force = scfg.Value( + force = kw.Value( False, isflag=True, short_alias=['f'], help='Actually release and tear down. Without it, clean only reports.', ) - orphans = scfg.Value( + orphans = kw.Value( True, isflag=True, help='Also remove containers in the project that infer-stack does not ' 'manage (--no-orphans keeps them).', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1448,13 +1448,13 @@ class WaitCLI(_LeasingCommonMixin): __command__ = 'wait' - names = scfg.Value( + names = kw.Value( [], nargs='*', position=1, type=str, help='Endpoint names to wait for (default: every live deployment).', ) - timeout = scfg.Value(600, type=float, help='Overall wait timeout (s).') - interval = scfg.Value(5, type=float, help='Readiness poll interval (s).') - json = scfg.Value(False, isflag=True) + timeout = kw.Value(600, type=float, help='Overall wait timeout (s).') + interval = kw.Value(5, type=float, help='Readiness poll interval (s).') + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1523,35 +1523,35 @@ class MeasureCLI(_LeasingCommonMixin): __command__ = 'measure' - endpoint = scfg.Value( + endpoint = kw.Value( None, position=1, required=True, type=str, help='Catalog endpoint to measure.', ) - record = scfg.Value( + record = kw.Value( False, isflag=True, help='Record the result into the measurements overlay ' '(consulted automatically at plan time when the catalog declares ' 'nothing for this endpoint).', ) - kv_gib = scfg.Value( + kv_gib = kw.Value( 2.0, type=float, help='KV-cache budget (GiB) added on top of the non-KV profile. ' 'A serving choice (max_model_len / max_num_seqs), not a model fact.', ) - margin = scfg.Value( + margin = kw.Value( 0.05, type=float, help='Safety-margin fraction over the non-KV profile ' '(allocator fragmentation, engine drift).', ) - timeout = scfg.Value( + timeout = kw.Value( 900, type=float, help='Readiness timeout when the endpoint must be brought up first (s).', ) - catalog = scfg.Value( + catalog = kw.Value( None, type=str, help='Catalog path (default: /catalog.yaml).', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1700,9 +1700,9 @@ class TuiCLI(_LeasingCommonMixin): __command__ = 'tui' - catalog = scfg.Value(None, type=str, help='Path to catalog.yaml.') - interval = scfg.Value(3.0, type=float, help='Auto-refresh interval (s).') - exit_after_paint = scfg.Value( + catalog = kw.Value(None, type=str, help='Path to catalog.yaml.') + interval = kw.Value(3.0, type=float, help='Auto-refresh interval (s).') + exit_after_paint = kw.Value( False, isflag=True, help='Quit as soon as the first frame is drawn, then print when that ' 'was (for measuring startup: `time infer-stack tui ' @@ -1745,9 +1745,9 @@ class RenewCLI(_LeasingCommonMixin): __command__ = 'renew' - lease = scfg.Value(None, position=1, type=str, help='Lease id.') - env_file = scfg.Value(None, type=str) - ttl = scfg.Value(None, type=str, help='New soft TTL (e.g. 2h); empty=infinite.') + lease = kw.Value(None, position=1, type=str, help='Lease id.') + env_file = kw.Value(None, type=str) + ttl = kw.Value(None, type=str, help='New soft TTL (e.g. 2h); empty=infinite.') @classmethod def main(cls, argv=True, **kwargs): @@ -1791,26 +1791,26 @@ class RunCLI(_LeasingCommonMixin): __command__ = 'run' - catalog = scfg.Value(None, type=str, help='Path to catalog.yaml.') - endpoint = scfg.Value( + catalog = kw.Value(None, type=str, help='Path to catalog.yaml.') + endpoint = kw.Value( None, type=str, alias=['endpoints'], help='Comma-separated endpoint or bundle names.', ) - base_url = scfg.Value('http://127.0.0.1:14042/v1', type=str) - api_key_env = scfg.Value('LITELLM_MASTER_KEY', type=str) - owner = scfg.Value(None, type=str) - ttl = scfg.Value('2h', type=str, help='Soft TTL backstop (default 2h).') - timeout = scfg.Value(600, type=float) - interval = scfg.Value(5, type=float) - queue = scfg.Value( + base_url = kw.Value('http://127.0.0.1:14042/v1', type=str) + api_key_env = kw.Value('LITELLM_MASTER_KEY', type=str) + owner = kw.Value(None, type=str) + ttl = kw.Value('2h', type=str, help='Soft TTL backstop (default 2h).') + timeout = kw.Value(600, type=float) + interval = kw.Value(5, type=float) + queue = kw.Value( False, isflag=True, help='Admission queue: wait (up to --timeout) for a GPU to free ' 'instead of failing fast when the fleet is full. Recommended for ' 'pipeline fan-out, where many jobs contend for a few GPUs.', ) - command = scfg.Value( + command = kw.Value( [], nargs='*', position=1, type=str, help='Command to run (after --).' ) @@ -2028,7 +2028,7 @@ class LeasesCLI(_LeasingCommonMixin): infer-stack leases --json # JSON (adds running + gpus per deployment) """ - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -2186,27 +2186,27 @@ class TestCLI(_PathOverridesMixin): __command__ = 'test' - catalog = scfg.Value( + catalog = kw.Value( None, type=str, help='Catalog path, used only to look up the endpoint protocol.', ) - name = scfg.Value( + name = kw.Value( None, position=1, type=str, help='Endpoint alias to test (e.g. chat).' ) - prompt = scfg.Value( + prompt = kw.Value( 'Reply with the single word: ready.', type=str, help='Prompt to send.' ) - max_tokens = scfg.Value(32, type=int) - timeout = scfg.Value(60, type=float, help='Request timeout (s).') - base_url = scfg.Value( + max_tokens = kw.Value(32, type=int) + timeout = kw.Value(60, type=float, help='Request timeout (s).') + base_url = kw.Value( None, type=str, help='Override the gateway base URL (…/v1).' ) - port = scfg.Value( + port = kw.Value( None, type=int, help='Override the gateway port (default: 14042).' ) - json = scfg.Value(False, isflag=True, help='Emit JSON instead of text.') - protocol = scfg.Value( - None, choices=['chat', 'completions'], + json = kw.Value(False, isflag=True, help='Emit JSON instead of text.') + protocol = kw.Value( + None, type=str, choices=['chat', 'completions'], help="Which surface to hit. Default: the endpoint's declared " '`protocol` from the catalog, falling back to chat.', ) @@ -2348,11 +2348,11 @@ class EnvCLI(_PathOverridesMixin): __command__ = 'env' - arg = scfg.Value( + arg = kw.Value( None, position=1, type=str, help='KEY to read its value, or KEY=VALUE to set it. Empty = path.', ) - export = scfg.Value( + export = kw.Value( False, isflag=True, help='Print every entry as `export KEY=value`.' ) @@ -2478,7 +2478,7 @@ class RoutesListCLI(_LeasingCommonMixin): __command__ = 'list' - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -2559,7 +2559,7 @@ class RoutesPruneCLI(_ApprovalMixin): __command__ = 'prune' - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -2656,12 +2656,12 @@ class RoutesSeedCLI(_ApprovalMixin): __command__ = 'seed' - catalogs = scfg.Value( + catalogs = kw.Value( None, nargs='+', position=1, type=str, help='One or more catalog.yaml files whose endpoints to merge into the ' 'route registry.', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -2735,16 +2735,16 @@ class ConfigPublishCLI(_ApprovalMixin): __command__ = 'publish' - catalogs = scfg.Value( + catalogs = kw.Value( [], nargs='*', position=1, type=str, help='Catalog files to publish as one union (default: --catalog, or the ' 'default-path catalog).', ) - pull = scfg.Value( + pull = kw.Value( True, isflag=True, help='Pre-pull every image the profile references (default; --no-pull skips).', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -2815,8 +2815,8 @@ class NetworkMigrateCLI(_ApprovalMixin): __command__ = 'migrate' - subnet = scfg.Value(None, type=str, help='IPv4 subnet, e.g. 172.30.0.0/24 (required).') - force = scfg.Value(False, isflag=True, help='Migrate even with active leases.') + subnet = kw.Value(None, type=str, help='IPv4 subnet, e.g. 172.30.0.0/24 (required).') + force = kw.Value(False, isflag=True, help='Migrate even with active leases.') @classmethod def main(cls, argv=True, **kwargs): @@ -2851,7 +2851,7 @@ class NetworkCheckCLI(_LeasingCommonMixin): __command__ = 'check' - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -2881,7 +2881,7 @@ class SecretsRotateCLI(_ApprovalMixin): __command__ = 'rotate' - force = scfg.Value(False, isflag=True, help='Rotate even with active leases.') + force = kw.Value(False, isflag=True, help='Rotate even with active leases.') @classmethod def main(cls, argv=True, **kwargs): @@ -2921,7 +2921,7 @@ def main(cls, argv=True, **kwargs): return 0 -class SecretsModalCLI(scfg.ModalCLI): +class SecretsModalCLI(kw.ModalCLI): """Manage the gateway's secrets.""" __command__ = 'secrets' @@ -2929,7 +2929,7 @@ class SecretsModalCLI(scfg.ModalCLI): rotate = SecretsRotateCLI -class NetworkModalCLI(scfg.ModalCLI): +class NetworkModalCLI(kw.ModalCLI): """Stable per-service addressing (migrate) and the upstream routing check.""" __command__ = 'network' @@ -2938,7 +2938,7 @@ class NetworkModalCLI(scfg.ModalCLI): check = NetworkCheckCLI -class RoutesModalCLI(scfg.ModalCLI): +class RoutesModalCLI(kw.ModalCLI): """Inspect + manage the LiteLLM route registry (static-superset mode). The registry accumulates every catalog's and every live deployment's routes diff --git a/infer_stack/cli/commands_meta.py b/infer_stack/cli/commands_meta.py index 8d382139..a0986fc3 100644 --- a/infer_stack/cli/commands_meta.py +++ b/infer_stack/cli/commands_meta.py @@ -16,7 +16,7 @@ from pathlib import Path from typing import Any -import scriptconfig as scfg +import kwconf as kw from .. import __version__ from ..paths import config_root, data_root, settings_path @@ -24,7 +24,7 @@ from .options import _PathOverridesMixin -class VersionCLI(scfg.DataConfig): +class VersionCLI(kw.Config): """Print the installed infer-stack version.""" __command__ = 'version' @@ -48,7 +48,7 @@ def _iter_subcommands(modal: Any): if attr.startswith('_'): continue if isinstance(val, type) and issubclass( - val, (scfg.DataConfig, scfg.ModalCLI) + val, (kw.Config, kw.ModalCLI) ): name = getattr(val, '__command__', None) or attr.replace('_', '-') out[name] = val @@ -64,7 +64,7 @@ def _doc_one_line(cls: Any) -> str: def _is_group(sub: Any) -> bool: - return isinstance(sub, type) and issubclass(sub, scfg.ModalCLI) + return isinstance(sub, type) and issubclass(sub, kw.ModalCLI) def _build_tree(modal: Any, node: Any) -> None: @@ -86,7 +86,7 @@ def _build_tree(modal: Any, node: Any) -> None: _build_tree(sub, child) -class HelpTreeCLI(scfg.DataConfig): +class HelpTreeCLI(kw.Config): """Print the full nested command tree with one-line descriptions.""" __command__ = 'tree' @@ -106,7 +106,7 @@ def main(cls, argv=True, **kwargs): return 0 -class HelpModalCLI(scfg.ModalCLI): +class HelpModalCLI(kw.ModalCLI): """Help utilities (use ``infer-stack --help`` for per-command help).""" __command__ = 'help' @@ -207,12 +207,12 @@ class ConfigPathsCLI(_PathOverridesMixin): __command__ = 'paths' - target: Any = scfg.Value( + target: Any = kw.Value( 'all', position=1, help='Path group to show: all, config, data, or leasing.', ) - json: Any = scfg.Value( + json: Any = kw.Value( False, isflag=True, help='Emit the path groups as JSON instead of human-readable text.', @@ -392,17 +392,17 @@ class ConfigInitCLI(_PathOverridesMixin): """ __command__ = 'init' - yes = scfg.Value( + yes = kw.Value( False, isflag=True, alias=['y'], help='Non-interactive: write without prompting/confirming.', ) - fresh = scfg.Value( + fresh = kw.Value( False, isflag=True, help='Start over: ignore any existing config and write a clean one from ' 'defaults (discards other persisted settings too).', ) - backend = scfg.Value( - None, choices=['compose', 'kubeai', 'null'], + backend = kw.Value( + None, type=str, choices=['compose', 'kubeai', 'null'], help='Preset the default backend (skips that prompt).', ) @@ -493,8 +493,8 @@ class ConfigSetCLI(_PathOverridesMixin): """Persist a durable default, e.g. ``config set backend compose``.""" __command__ = 'set' - key = scfg.Value(None, position=1, type=str) - value = scfg.Value(None, position=2, type=str) + key = kw.Value(None, position=1, type=str) + value = kw.Value(None, position=2, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -521,7 +521,7 @@ class ConfigGetCLI(_PathOverridesMixin): """Print one setting's value (or all settings).""" __command__ = 'get' - key = scfg.Value(None, position=1, type=str) + key = kw.Value(None, position=1, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -588,7 +588,7 @@ def main(cls, argv=True, **kwargs): from .commands_leasing import ConfigPublishCLI # noqa: E402 -class ConfigModalCLI(scfg.ModalCLI): +class ConfigModalCLI(kw.ModalCLI): """Inspect + manage infer-stack configuration (paths + durable settings).""" __command__ = 'config' diff --git a/infer_stack/cli/commands_mock.py b/infer_stack/cli/commands_mock.py index c425fa5b..16732d42 100644 --- a/infer_stack/cli/commands_mock.py +++ b/infer_stack/cli/commands_mock.py @@ -10,18 +10,18 @@ from typing import Any -import scriptconfig as scfg +import kwconf as kw import ubelt as ub -class MockServeCLI(scfg.DataConfig): +class MockServeCLI(kw.Config): """ Serve a deterministic OpenAI-compatible mock inference endpoint. """ __command__ = 'serve' - config_fpath = scfg.Value( + config_fpath = kw.Value( None, position=1, help=ub.paragraph( @@ -35,14 +35,14 @@ class MockServeCLI(scfg.DataConfig): ), ) - host = scfg.Value('127.0.0.1', help='Bind address.') + host = kw.Value('127.0.0.1', help='Bind address.') - port = scfg.Value( + port = kw.Value( 8100, help='Bind port. Use 0 to pick a free port and print it.', ) - mode = scfg.Value( + mode = kw.Value( None, help=ub.paragraph( """ @@ -57,13 +57,13 @@ class MockServeCLI(scfg.DataConfig): ), ) - require_auth = scfg.Value( + require_auth = kw.Value( False, isflag=True, help='Reject requests without a valid bearer token.', ) - api_key = scfg.Value( + api_key = kw.Value( None, help=ub.paragraph( """ @@ -74,11 +74,11 @@ class MockServeCLI(scfg.DataConfig): ), ) - list_modes = scfg.Value( + list_modes = kw.Value( False, isflag=True, help='Print the available response modes and exit.', ) - seed = scfg.Value( + seed = kw.Value( None, help=ub.paragraph( """ @@ -88,7 +88,7 @@ class MockServeCLI(scfg.DataConfig): ), ) - print_url = scfg.Value( + print_url = kw.Value( True, isflag=True, help='Print the bound base URL on startup.', @@ -158,7 +158,7 @@ def main(cls, argv=None, **kwargs): server.stop() -class MockModalCLI(scfg.ModalCLI): +class MockModalCLI(kw.ModalCLI): """ Deterministic mock inference server for tests and dry runs. """ diff --git a/infer_stack/cli/commands_runtime.py b/infer_stack/cli/commands_runtime.py index 99fa46c6..c37d7336 100644 --- a/infer_stack/cli/commands_runtime.py +++ b/infer_stack/cli/commands_runtime.py @@ -14,7 +14,7 @@ from pathlib import Path from typing import Any -import scriptconfig as scfg +import kwconf as kw from ..log_filter import compact_litellm_tracebacks from ..paths import config_root, data_root, get_setting, settings_path @@ -365,7 +365,7 @@ class StatusCLI(_PathOverridesMixin): (active leases / live deployments), with pointers to dig deeper.""" __command__ = 'status' - catalog = scfg.Value( + catalog = kw.Value( None, type=str, help='Catalog path (default: config dir).' ) @@ -393,7 +393,7 @@ class _ComposeWrapperBase(_PathOverridesMixin): """Common fields for ``docker compose `` wrappers over the leasing Compose deployment.""" - services = scfg.Value( + services = kw.Value( None, nargs='*', position=1, @@ -435,17 +435,17 @@ class LogsCLI(_ComposeWrapperBase): __command__ = 'logs' - follow = scfg.Value( + follow = kw.Value( False, isflag=True, short_alias=['f'], help='Stream logs (docker compose logs -f).', ) - tail = scfg.Value( + tail = kw.Value( None, type=str, help="Tail the last N lines (default: all). Pass a number or 'all'.", ) - timestamps = scfg.Value(False, isflag=True) - no_color = scfg.Value(False, isflag=True) - raw = scfg.Value( + timestamps = kw.Value(False, isflag=True) + no_color = kw.Value(False, isflag=True) + raw = kw.Value( False, isflag=True, help='Show raw followed logs without known LiteLLM traceback compaction.', @@ -487,14 +487,14 @@ class PsCLI(_ComposeWrapperBase): __command__ = 'ps' - all = scfg.Value( + all = kw.Value( False, isflag=True, short_alias=['a'], help='Include stopped containers.' ) - services_only = scfg.Value( + services_only = kw.Value( False, isflag=True, help='Print only service names (passes --services to docker compose).', ) - quiet = scfg.Value( + quiet = kw.Value( False, isflag=True, short_alias=['q'], help='Print only container IDs.' ) @@ -515,7 +515,7 @@ def main(cls, argv=True, **kwargs): class RestartCLI(_ComposeWrapperBase): """``docker compose restart [services...]``.""" - timeout = scfg.Value(None, type=int, help='Stop timeout in seconds.') + timeout = kw.Value(None, type=int, help='Stop timeout in seconds.') @classmethod def main(cls, argv=True, **kwargs): @@ -530,8 +530,8 @@ def main(cls, argv=True, **kwargs): class PullCLI(_ComposeWrapperBase): """``docker compose pull [services...]``.""" - quiet = scfg.Value(False, isflag=True, short_alias=['q']) - ignore_pull_failures = scfg.Value(False, isflag=True) + quiet = kw.Value(False, isflag=True, short_alias=['q']) + ignore_pull_failures = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -559,7 +559,7 @@ def main(cls, argv=True, **kwargs): class StopCLI(_ComposeWrapperBase): """``docker compose stop [services...]``.""" - timeout = scfg.Value(None, type=int) + timeout = kw.Value(None, type=int) @classmethod def main(cls, argv=True, **kwargs): @@ -578,7 +578,7 @@ class StackDownCLI(_ComposeWrapperBase): automatically on release; this is the manual escape hatch. """ - volumes = scfg.Value( + volumes = kw.Value( False, isflag=True, help='Also remove named volumes (--volumes).' ) @@ -608,7 +608,7 @@ def main(cls, argv=True, **kwargs): return int(subprocess.run(cmd, env=_docker_env()).returncode) -class StackModalCLI(scfg.ModalCLI): +class StackModalCLI(kw.ModalCLI): """Day-2 ops on the running leasing deployment.""" __command__ = 'stack' @@ -645,13 +645,13 @@ class DoctorCLI(_PathOverridesMixin): __command__ = 'doctor' - backend = scfg.Value( + backend = kw.Value( None, type=str, help='Backend to check (default: the configured `backend` setting).', ) - gpu = scfg.Value(False, isflag=True, + gpu = kw.Value(False, isflag=True, help='Also check host GPUs (implied by --sudo).') - sudo = scfg.Value(False, isflag=True, + sudo = kw.Value(False, isflag=True, help='Run the GPU holder scan as root. Without it that ' 'check reports "not checked", never "clear".') diff --git a/infer_stack/cli/context.py b/infer_stack/cli/context.py index 3e01b14b..8d86ebcf 100644 --- a/infer_stack/cli/context.py +++ b/infer_stack/cli/context.py @@ -18,7 +18,7 @@ def _as_mapping(args: Any) -> dict[str, Any]: """Coerce a CLI args object into a plain dict. - Works for ``None``, ``argparse.Namespace``, and ``scfg.DataConfig`` + Works for ``None``, ``argparse.Namespace``, and ``kw.Config`` instances. Used to side-step name clashes between user-declared fields and ``DataConfig`` builtins. """ diff --git a/infer_stack/cli/options.py b/infer_stack/cli/options.py index ced620ff..f4a1c3ce 100644 --- a/infer_stack/cli/options.py +++ b/infer_stack/cli/options.py @@ -2,17 +2,17 @@ from ..paths import CONFIG_DIR_ENV from ..paths import DATA_DIR_ENV -import scriptconfig as scfg +import kwconf as kw # --------------------------------------------------------------------------- # DataConfig mixins for common override flags # --------------------------------------------------------------------------- -class _PathOverridesMixin(scfg.DataConfig): +class _PathOverridesMixin(kw.Config): """Adds global ``--config-dir`` / ``--data-dir`` to a subcommand.""" - config_dir = scfg.Value( + config_dir = kw.Value( None, type=str, help=( @@ -20,7 +20,7 @@ class _PathOverridesMixin(scfg.DataConfig): f'~/.config/infer_stack (XDG_CONFIG_HOME) or ${CONFIG_DIR_ENV} when set.' ), ) - data_dir = scfg.Value( + data_dir = kw.Value( None, type=str, help=( @@ -30,42 +30,42 @@ class _PathOverridesMixin(scfg.DataConfig): ) -class _BackendOverrideMixin(scfg.DataConfig): - backend = scfg.Value( - None, choices=['compose', 'kubeai'], help='Active backend override.' +class _BackendOverrideMixin(kw.Config): + backend = kw.Value( + None, type=str, choices=['compose', 'kubeai'], help='Active backend override.' ) -class _ComposeOverrideMixin(scfg.DataConfig): - compose_cmd = scfg.Value( +class _ComposeOverrideMixin(kw.Config): + compose_cmd = kw.Value( None, type=str, help="Docker compose command override (e.g. 'podman compose').", ) -class _ProfileOverrideMixin(scfg.DataConfig): - profile = scfg.Value( +class _ProfileOverrideMixin(kw.Config): + profile = kw.Value( None, type=str, help='Active profile override (sets config.active_profile).', ) -class _PortOverridesMixin(scfg.DataConfig): - litellm_port = scfg.Value(None, type=int) - open_webui_port = scfg.Value(None, type=int) - postgres_port = scfg.Value(None, type=int) +class _PortOverridesMixin(kw.Config): + litellm_port = kw.Value(None, type=int) + open_webui_port = kw.Value(None, type=int) + postgres_port = kw.Value(None, type=int) -class _ClusterOverridesMixin(scfg.DataConfig): - namespace = scfg.Value( +class _ClusterOverridesMixin(kw.Config): + namespace = kw.Value( None, type=str, help='Kubernetes namespace for kubeai deployments.' ) - ingress_host = scfg.Value( + ingress_host = kw.Value( None, type=str, help='Ingress host (kubeai only).' ) - ingress_enabled = scfg.Value( + ingress_enabled = kw.Value( None, isflag=True, alias=['ingress'], @@ -73,24 +73,24 @@ class _ClusterOverridesMixin(scfg.DataConfig): ) -class _AllowUnsupportedMixin(scfg.DataConfig): - allow_unsupported = scfg.Value( +class _AllowUnsupportedMixin(kw.Config): + allow_unsupported = kw.Value( False, isflag=True, help='Allow validation errors when planning/rendering.', ) -class _SimulateHardwareMixin(scfg.DataConfig): - simulate_hardware = scfg.Value( +class _SimulateHardwareMixin(kw.Config): + simulate_hardware = kw.Value( None, type=str, help='Simulate GPUs: comma-separated NxM or M entries (e.g. 4x96, 2x80, "48,16" for a heterogeneous host). Useful for planning on smaller machines.', ) -class _AllowedGpusMixin(scfg.DataConfig): - allowed_gpus = scfg.Value( +class _AllowedGpusMixin(kw.Config): + allowed_gpus = kw.Value( None, type=str, help=( @@ -102,8 +102,8 @@ class _AllowedGpusMixin(scfg.DataConfig): ) -class _DisplayGpuMixin(scfg.DataConfig): - skip_display_gpus = scfg.Value( +class _DisplayGpuMixin(kw.Config): + skip_display_gpus = kw.Value( None, isflag=True, alias=['skip-display-gpus'], diff --git a/infer_stack/tui.py b/infer_stack/tui.py index 68fe105e..82ad51e7 100644 --- a/infer_stack/tui.py +++ b/infer_stack/tui.py @@ -136,6 +136,7 @@ def activity_label(description: str) -> str | None: """``'acquiring qwen'`` from a worker description like ``_do_acquire('qwen')``. Example: + >>> # xdoctest: +REQUIRES(module:textual) >>> activity_label("_do_acquire('qwen')") 'acquiring qwen' >>> activity_label('_do_apply()') diff --git a/pyproject.toml b/pyproject.toml index a08506f8..1c47d3e8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,11 +33,11 @@ dynamic = [ ] dependencies = [ "jinja2>=3.1", + "kwconf>=0.11.0", "loguru>=0.7", "pyyaml>=6.0", "requests>=2.31", "rich>=13.0", - "scriptconfig>=0.9", "ubelt>=1.3", ] optional-dependencies.tests = [ @@ -72,6 +72,10 @@ packages.find.include = [ [tool.uv] exclude-newer = "2026-06-04T04:00:00Z" +# kwconf (the CLI library, successor to scriptconfig) has no release before +# the cutoff. Admit it up to 0.11.0 (2026-08-07), the version tested; the +# cutoff still holds for every other package. +exclude-newer-package = { kwconf = "2026-08-08T00:00:00Z" } [tool.ruff] target-version = "py310" diff --git a/requirements/locks/tests.txt b/requirements/locks/tests.txt index 03dd8137..f78fc55b 100644 --- a/requirements/locks/tests.txt +++ b/requirements/locks/tests.txt @@ -18,6 +18,8 @@ iniconfig==2.3.0 # via pytest jinja2==3.1.6 # via infer-stack +kwconf==0.11.0 + # via infer-stack loguru==0.7.3 # via infer-stack markdown-it-py==4.2.0 @@ -46,15 +48,11 @@ pytest-codeblocks==0.17.0 pytest-cov==7.1.0 # via infer-stack pyyaml==6.0.3 - # via - # infer-stack - # scriptconfig + # via infer-stack requests==2.34.2 # via infer-stack rich==15.0.0 # via infer-stack -scriptconfig==0.9.1 - # via infer-stack tomli==2.4.1 ; python_full_version <= '3.11' # via # coverage @@ -62,9 +60,7 @@ tomli==2.4.1 ; python_full_version <= '3.11' typing-extensions==4.15.0 ; python_full_version < '3.11' # via exceptiongroup ubelt==1.4.2 - # via - # infer-stack - # scriptconfig + # via infer-stack urllib3==2.7.0 # via requests win32-setctime==1.2.0 ; sys_platform == 'win32' diff --git a/tests/test_cli_equivalent.py b/tests/test_cli_equivalent.py index 9d0e2e79..c90aa8c5 100644 --- a/tests/test_cli_equivalent.py +++ b/tests/test_cli_equivalent.py @@ -9,7 +9,7 @@ import shlex import pytest -import scriptconfig as scfg +import kwconf as kw from infer_stack import cli_equivalent as cli from infer_stack.cli import ManageCLI @@ -20,7 +20,7 @@ def parse(text: str): words = shlex.split(text.split(' #', 1)[0]) assert words[0] == 'infer-stack' node, rest = ManageCLI, words[1:] - while isinstance(node, type) and issubclass(node, scfg.ModalCLI): + while isinstance(node, type) and issubclass(node, kw.ModalCLI): node = getattr(node, rest[0].replace('-', '_')) rest = rest[1:] return node, node.cli(argv=rest, strict=True) diff --git a/uv.lock b/uv.lock index 558a181a..c9a7262a 100644 --- a/uv.lock +++ b/uv.lock @@ -11,6 +11,9 @@ resolution-markers = [ [options] exclude-newer = "2026-06-04T04:00:00Z" +[options.exclude-newer-package] +kwconf = "2026-08-08T00:00:00Z" + [[package]] name = "certifi" version = "2026.5.20" @@ -257,7 +260,7 @@ name = "exceptiongroup" version = "1.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", marker = "python_full_version < '3.11'" }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/50/79/66800aadf48771f6b62f7eb014e352e5d06856655206165d775e675a02c9/exceptiongroup-1.3.1.tar.gz", hash = "sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219", size = 30371, upload-time = "2025-11-21T23:01:54.787Z" } wheels = [ @@ -278,11 +281,11 @@ name = "infer-stack" source = { editable = "." } dependencies = [ { name = "jinja2" }, + { name = "kwconf" }, { name = "loguru" }, { name = "pyyaml" }, { name = "requests" }, { name = "rich" }, - { name = "scriptconfig" }, { name = "ubelt" }, ] @@ -300,6 +303,7 @@ tui = [ [package.metadata] requires-dist = [ { name = "jinja2", specifier = ">=3.1" }, + { name = "kwconf", specifier = ">=0.11.0" }, { name = "loguru", specifier = ">=0.7" }, { name = "pytest", marker = "extra == 'tests'", specifier = ">=7.0" }, { name = "pytest-codeblocks", marker = "extra == 'tests'", specifier = ">=0.17" }, @@ -307,7 +311,6 @@ requires-dist = [ { name = "pyyaml", specifier = ">=6.0" }, { name = "requests", specifier = ">=2.31" }, { name = "rich", specifier = ">=13.0" }, - { name = "scriptconfig", specifier = ">=0.9" }, { name = "textual", marker = "extra == 'tui'", specifier = ">=0.50" }, { name = "ubelt", specifier = ">=1.3" }, { name = "xdoctest", marker = "extra == 'tests'", specifier = ">=1.1.5" }, @@ -335,6 +338,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, ] +[[package]] +name = "kwconf" +version = "0.11.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c6/77/20ce9cf65bc20e769db9a506e72e291f7a38ee766393110858a15e93ea1b/kwconf-0.11.0.tar.gz", hash = "sha256:cd8e9192a3c0e54750368980f9bf6b02676f75b76ac3830b656e977cc55f988f", size = 172077, upload-time = "2026-08-07T15:25:34.385Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fa/7a/4b4a1c503cc621eb2256e5d18385ccfb546f536008cdb4db0189426f5e77/kwconf-0.11.0-py3-none-any.whl", hash = "sha256:4691afdf7173ca0d42302a2cf17910917ac3bd02729e0b2f2cb2aa1e3b2ee18b", size = 110488, upload-time = "2026-08-07T15:25:32.964Z" }, +] + [[package]] name = "linkify-it-py" version = "2.1.0" @@ -655,19 +667,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, ] -[[package]] -name = "scriptconfig" -version = "0.9.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "pyyaml", marker = "python_full_version < '4'" }, - { name = "ubelt" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/a4/4c/6d0ecc5e29b498dc6b5402a0c561e97dd00ce1aeebdc5e494cc5fa8a0b6b/scriptconfig-0.9.1.tar.gz", hash = "sha256:9766c2ee601c4b8d97753ba13a914f8c0a212da4f6d2aec18ad628fd8b53d7ef", size = 112424, upload-time = "2026-03-02T17:33:34.405Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/57/00/2bf691dfc2b0170013d466fcdbbfc9d036e26d6049d74efaf7775401321d/scriptconfig-0.9.1-py3-none-any.whl", hash = "sha256:4b69e2f4eb681ec4480e9bb5d6c82d2f8c82a9badc703a261ee8940736101d86", size = 87103, upload-time = "2026-03-02T17:33:31.394Z" }, -] - [[package]] name = "textual" version = "8.2.7" From 4def7e7eac74cb4de89a7c343981a856ee25c7db Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Thu, 24 Sep 2026 21:19:49 -0400 Subject: [PATCH 18/34] Remove the uv exclude-newer cutoff The 2026-06-04 cutoff predated every kwconf release and needed a per-package exception. Removed at the user's request, with the exception. Relocked without --upgrade: uv.lock loses only the cutoff header, the locked versions and requirements/locks/tests.txt are unchanged. A fresh unpinned install now gets kwconf 0.12.0, and the CI-equivalent suite passes on it (838 passed, 3.14, tests extra only). Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 5 ++--- pyproject.toml | 7 ------- uv.lock | 6 ------ 3 files changed, 2 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 571b00d9..908ab1ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,10 +12,9 @@ strings `null`, `true` and numbers on the command line as values, so take `type=str`. The program name stays `infer-stack` in usage and errors (`__prog__`). -The repository's uv `exclude-newer` cutoff (2026-06-04) predates every kwconf -release, so kwconf alone gets an exception up to 0.11.0, the version tested. `uv.lock` and `requirements/locks/tests.txt` were regenerated: kwconf 0.11.0 -added, scriptconfig removed. +added, scriptconfig removed. The uv `exclude-newer` cutoff (2026-06-04), which +predated every kwconf release, is removed; the locked versions are unchanged. ### The TUI stops repainting what did not change diff --git a/pyproject.toml b/pyproject.toml index 1c47d3e8..906273c7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -70,13 +70,6 @@ packages.find.include = [ "infer_stack*", ] -[tool.uv] -exclude-newer = "2026-06-04T04:00:00Z" -# kwconf (the CLI library, successor to scriptconfig) has no release before -# the cutoff. Admit it up to 0.11.0 (2026-08-07), the version tested; the -# cutoff still holds for every other package. -exclude-newer-package = { kwconf = "2026-08-08T00:00:00Z" } - [tool.ruff] target-version = "py310" line-length = 80 diff --git a/uv.lock b/uv.lock index c9a7262a..0028f947 100644 --- a/uv.lock +++ b/uv.lock @@ -8,12 +8,6 @@ resolution-markers = [ "python_full_version >= '4'", ] -[options] -exclude-newer = "2026-06-04T04:00:00Z" - -[options.exclude-newer-package] -kwconf = "2026-08-08T00:00:00Z" - [[package]] name = "certifi" version = "2026.5.20" From 271d6458bb8eb739f24112d34c68a77a6f0d795d Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Fri, 25 Sep 2026 13:54:52 -0400 Subject: [PATCH 19/34] TUI: one Clean up key in the footer; gc --forget; log CLI equivalents Both Clean up buttons ran the same ledger prune. It is now a single x binding in the footer, backed by a new `infer-stack gc --forget`. The TUI log also names the command for refresh, settings save and the API tab. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 13 ++++++++++++ infer_stack/cli/commands_leasing.py | 15 ++++++++++++++ infer_stack/cli_equivalent.py | 5 +++++ infer_stack/tui.py | 31 +++++++++++++++++++---------- tests/test_cli_leasing.py | 16 +++++++++++++++ tests/test_tui.py | 30 ++++++++++++++++++++++++++++ 6 files changed, 100 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 908ab1ea..378a48e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### One Clean up, in the footer, with a CLI command behind it + +The leases and deployments panes each had a Clean up button, and both did the +same thing: forget released/expired leases and stopped deployments. Nothing +running changes. That is now one `x Clean up` in the footer, and the same +action on the CLI is `infer-stack gc --forget`. + +The TUI log now names the CLI command for the actions that were missing one: +`r` (`status`), Clean up (`gc --forget`), saving settings (`config set …`), +and the API tab's send, test-all (`test `) and model list (a `curl` +that reads the key through `infer-stack env` and never prints it). Poll +intervals are TUI-only, and the log says so. + ### The CLI is built on kwconf instead of scriptconfig kwconf is scriptconfig's successor, which aiq-magnet, cmd_queue and kwdagger diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index 14bdb67d..507b3ff9 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -1248,6 +1248,7 @@ class GcCLI(_ApprovalMixin): blocking ``acquire`` (``--queue``) already does this implicitly while it waits. ``--evict`` additionally tears down idle *keep-warm* deployments (like ``evict --all``). On a terminal the teardown is shown and confirmed (``--yes`` skips). + ``--forget`` only drops finished rows from the ledger (the TUI's Clean up). """ __command__ = 'gc' @@ -1262,6 +1263,11 @@ class GcCLI(_ApprovalMixin): help='Instead: remove containers in the project that infer-stack does not ' 'manage (listed and confirmed first; --yes skips the prompt).', ) + forget = kw.Value( + False, isflag=True, + help='Instead: forget released/expired leases and stopped deployments ' + 'from the ledger. History only; nothing running changes.', + ) json = kw.Value(False, isflag=True) @classmethod @@ -1270,6 +1276,15 @@ def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) controller = _open_controller(config, interactive=True) + if config.forget: + n_leases, n_deployments = controller.prune() + if config.json: + print(json.dumps({'leases': n_leases, + 'deployments': n_deployments}, indent=2)) + else: + print(f'gc --forget: forgot {n_leases} released/expired lease(s), ' + f'{n_deployments} stopped deployment(s)') + return 0 if config.orphans: if not isinstance(controller.backend, ComposeBackend): raise SystemExit('gc --orphans needs the compose backend') diff --git a/infer_stack/cli_equivalent.py b/infer_stack/cli_equivalent.py index 5d271d5e..a4b93448 100644 --- a/infer_stack/cli_equivalent.py +++ b/infer_stack/cli_equivalent.py @@ -26,6 +26,11 @@ def _kv_value(value: Any) -> str: return json.dumps(value) +#: What the gateway routes; the key comes from the CLI, never the log. +MODELS_CURL = ('curl -s "$(infer-stack env OPENAI_BASE_URL)/models" ' + '-H "Authorization: Bearer $(infer-stack env LITELLM_MASTER_KEY)"') + + #: Endpoint fields `catalog endpoint add` can express. _ENDPOINT_FIELDS = {'engine', 'model', 'host', 'public_name', 'reclaim', 'protocol', 'placement', 'runtime'} diff --git a/infer_stack/tui.py b/infer_stack/tui.py index 82ad51e7..96add27e 100644 --- a/infer_stack/tui.py +++ b/infer_stack/tui.py @@ -970,9 +970,11 @@ class InferStackTUI(App): BINDINGS = [ # Truly global controls stay in the footer. - ('r', 'refresh', 'Refresh'), + ('r', 'refresh_now', 'Refresh'), # Global on purpose: an error can happen while any tab is in front. ('l', 'show_app_log', 'TUI log'), + # Forgets finished history in both tables, so it belongs to neither pane. + ('x', 'cleanup', 'Clean up'), ('tab', 'focus_next', 'Next pane'), ('q', 'quit', 'Quit'), # Pane-scoped actions: keys still work, but they live as buttons under @@ -981,7 +983,6 @@ class InferStackTUI(App): Binding('d', 'release', 'Release', show=False), Binding('e', 'evict', 'Evict', show=False), Binding('a', 'release_all', 'Release all', show=False), - Binding('x', 'cleanup', 'Clean up', show=False), # Multi-select: space toggles the cursor row in the focused leases/ # deployments table; release/evict then act on every checked row. Binding('space', 'toggle_select', 'Select row', show=False), @@ -1188,7 +1189,6 @@ def _compose_dashboard(self) -> ComposeResult: with Horizontal(id='lease-actions'): yield Button('Release', id='btn-release') yield Button('Release all', id='btn-release-all') - yield Button('Clean up', id='btn-cleanup') yield _Divider('y', self._drag_tables, id='tsplit') with Vertical(id='deployments-pane'): yield Static( @@ -1196,15 +1196,15 @@ def _compose_dashboard(self) -> ComposeResult: "The 'leases' column is how many leases hold each. " 'Evict an idle one to free its GPU (cursor row, or ' 'rows checked with space / ctrl/shift-click); Evict ' - 'all idle clears every kept-warm one; Clean up forgets ' - 'stopped ones.', classes='desc', + 'all idle clears every kept-warm one. x (Clean up) ' + 'forgets stopped deployments and finished leases.', + classes='desc', ) yield DataTable(id='deployments', cursor_type='row', zebra_stripes=True) with Horizontal(id='deployment-actions'): yield Button('Evict', id='btn-evict') yield Button('Evict all idle', id='btn-evict-all') - yield Button('Clean up', id='btn-cleanup-deployments') yield _Divider('y', self._drag_logs, id='hsplit') with Collapsible(title='docker', collapsed=True, id='docker'): with TabbedContent(id='docker-tabs'): @@ -1845,6 +1845,12 @@ def _refresh_bg(self) -> None: data = self._collect() self.call_from_thread(self._render, data) + def action_refresh_now(self) -> None: + """The `r` key: a refresh the user asked for (the timer calls + ``action_refresh`` directly, and must not fill the TUI log).""" + self._cli(cli.command('status')) + self.action_refresh() + def action_refresh(self) -> None: self._sync_pane_state() # capture pane state on the UI thread first self._refresh_bg() @@ -2706,8 +2712,6 @@ def on_button_pressed(self, event: Button.Pressed) -> None: 'btn-release-all': self.action_release_all, 'btn-evict': self.action_evict, 'btn-evict-all': self.action_evict_all, - 'btn-cleanup': self.action_cleanup, - 'btn-cleanup-deployments': self.action_cleanup, 'btn-suggest': self.action_suggest, 'btn-add-model': self.action_add_model, 'btn-add-endpoint': self.action_add_endpoint, @@ -2748,6 +2752,7 @@ def _on_apply_ui_settings(self) -> None: return self.ledger_interval = ledger self.observe_interval = max(observe, ledger) # observe never beats ledger + self.app_log('CLI: none; poll intervals are a TUI-only preference') self._apply_poll_settings() prefs = load_tui_settings() prefs['ledger_interval'] = self.ledger_interval @@ -2777,6 +2782,10 @@ def _on_save_settings(self) -> None: ) path = save_settings(s) self._status(f'saved settings → {path}') + self._cli(*(cli.command('config', 'set', key, str(s[key]).lower() + if isinstance(s[key], bool) else s[key]) + for key in ('backend', 'data_dir', 'ui', 'reverse_proxy', + 'skip_display_gpus') if key in s)) except Exception as ex: # noqa: BLE001 self._status(f'save settings failed: {ex}') @@ -2814,8 +2823,7 @@ def action_evict_all(self) -> None: def action_cleanup(self) -> None: self._status('cleaning up released/expired leases + stopped deployments…') - self.app_log('CLI: none yet; this only forgets finished rows in ' - 'the ledger (nothing running changes)') + self._cli(cli.command('gc', '--forget')) self._do_cleanup() # -- docker compose control ------------------------------------------- @@ -3369,6 +3377,7 @@ def action_api_send(self) -> None: prompt = (self.query_one('#api-prompt', Input).value.strip() or 'Say hello in one short sentence.') self._api_log(f'> [{model}] {prompt}') + self._cli(cli.command('test', model, '--prompt', prompt)) self._do_api_send(model, prompt) def action_api_test_all(self) -> None: @@ -3377,10 +3386,12 @@ def action_api_test_all(self) -> None: self._refuse('no ready models to test (acquire one first)') return self._api_log(f'— testing {len(models)} ready model(s) —') + self._cli(*(cli.command('test', model) for model in models)) self._do_api_test_all(models) def action_api_list_models(self) -> None: self._api_log('> GET /v1/models (what the gateway routes)') + self._cli(cli.MODELS_CURL) self._do_api_list() def action_api_copy_curl(self) -> None: diff --git a/tests/test_cli_leasing.py b/tests/test_cli_leasing.py index 9f612adc..d007f7c9 100644 --- a/tests/test_cli_leasing.py +++ b/tests/test_cli_leasing.py @@ -225,6 +225,22 @@ def test_release_evict_tears_down_immediately(env, capsys): assert _leases_json(env, capsys)['deployments'][0]['state'] == 'stopped' +def test_gc_forget_drops_only_finished_rows(env, capsys): + """`gc --forget` is the TUI's Clean up: history goes, live rows stay.""" + from infer_stack.cli.commands_leasing import GcCLI + + AcquireCLI.main(argv=['qwen-coder', *_base(env), '--owner', 'a']) + ReleaseCLI.main(argv=['--ledger', env.db, '--all', '--evict']) + AcquireCLI.main(argv=['reranker', *_base(env), '--owner', 'b']) + capsys.readouterr() + assert GcCLI.main(argv=['--ledger', env.db, '--forget', '--json']) == 0 + assert json.loads(capsys.readouterr().out) == {'leases': 1, 'deployments': 1} + data = _leases_json(env, capsys) + assert [le['owner'] for le in data['leases']] == ['b'] + assert [g['state'] for g in data['deployments']] != ['stopped'] + assert len(data['deployments']) == 1 + + def test_acquire_without_ttl_is_standing_lease(env, capsys): # No --ttl -> an infinite (standing-service) lease owned by the caller. AcquireCLI.main(argv=['qwen-coder', *_base(env)]) diff --git a/tests/test_tui.py b/tests/test_tui.py index 66a111cd..aedd48c1 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -1231,6 +1231,36 @@ async def scenario(): assert not any(str(le.state) == 'released' for le in leases) +def test_clean_up_is_one_footer_key_that_logs_its_cli_command(): + """One Clean up, on `x` in the footer, and each action names its command.""" + from textual.widgets import Button + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + seen = {} + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + seen['buttons'] = [b.id for b in app.query(Button) + if 'cleanup' in (b.id or '')] + seen['footer'] = [b for b in app.BINDINGS if isinstance(b, tuple)] + await pilot.press('x') + await app.workers.wait_for_complete() + await pilot.press('r') + await pilot.pause() + seen['applog'] = '\n'.join(app._app_log_lines) + + _run(scenario) + assert seen['buttons'] == [] + assert ('x', 'cleanup', 'Clean up') in seen['footer'] # tuples show in the footer + assert 'CLI: infer-stack gc --forget' in seen['applog'] + assert 'CLI: infer-stack status' in seen['applog'] + + def test_tui_evict_all_idle_button(): from infer_stack.leasing import DeploymentState from infer_stack.tui import InferStackTUI From d672bfa05c46931ab4f696a71b696dd55f129932 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Fri, 25 Sep 2026 14:05:03 -0400 Subject: [PATCH 20/34] TUI: shorten the four pane descriptions Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- infer_stack/tui.py | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-) diff --git a/infer_stack/tui.py b/infer_stack/tui.py index 96add27e..a07a2440 100644 --- a/infer_stack/tui.py +++ b/infer_stack/tui.py @@ -1149,8 +1149,7 @@ def _compose_dashboard(self) -> ComposeResult: with Horizontal(id='body'): with Vertical(id='sidebar'): yield Static( - 'Endpoints — runnable model + engine configs. Acquire one to ' - 'serve it, or Suggest a set sized to your GPUs.', classes='desc', + 'Endpoints: acquirable configurations', classes='desc', ) yield Static('', id='catalog-help') yield _EndpointTable(id='endpoints', cursor_type='row', @@ -1165,8 +1164,7 @@ def _compose_dashboard(self) -> ComposeResult: yield Button('✨ Suggest from my GPUs', id='btn-suggest') yield _Divider('y', self._drag_models, id='csplit') yield Static( - 'Models — weights an endpoint can serve. Add models here, ' - 'then point an endpoint at one.', classes='desc', + 'Models: servable weights', classes='desc', ) yield DataTable(id='models', cursor_type='row', zebra_stripes=True) @@ -1178,10 +1176,8 @@ def _compose_dashboard(self) -> ComposeResult: with Vertical(id='tables'): with Vertical(id='leases-pane'): yield Static( - 'Reservations you hold. Each maps to one deployment ' - 'below (see the deployment column); many leases can ' - 'share one. Release acts on the cursor row, or on ' - 'every row you check (space, or ctrl/shift-click).', + 'Reservations that map to a deployment. ' + '(space or ctrl/shift-click to multiselect).', classes='desc', ) yield DataTable(id='leases', cursor_type='row', @@ -1192,13 +1188,7 @@ def _compose_dashboard(self) -> ComposeResult: yield _Divider('y', self._drag_tables, id='tsplit') with Vertical(id='deployments-pane'): yield Static( - 'Running model deployments and the GPUs they hold. ' - "The 'leases' column is how many leases hold each. " - 'Evict an idle one to free its GPU (cursor row, or ' - 'rows checked with space / ctrl/shift-click); Evict ' - 'all idle clears every kept-warm one. x (Clean up) ' - 'forgets stopped deployments and finished leases.', - classes='desc', + 'Models running.', classes='desc', ) yield DataTable(id='deployments', cursor_type='row', zebra_stripes=True) From 3fd3a3785162309e3b17b28b7f739f5cf7577818 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Fri, 25 Sep 2026 14:10:38 -0400 Subject: [PATCH 21/34] TUI: show endpoints|models and leases|deployments as tabs Each pair is laid out by one helper that builds either tabs or the old draggable split from the same pane functions, so either layout is one class switch. Tab labels carry the lease and deployment counts. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 9 +++ infer_stack/tui.py | 137 ++++++++++++++++++++++++++++++--------------- tests/test_tui.py | 62 ++++++++++++++------ 3 files changed, 145 insertions(+), 63 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 378a48e8..bd01f31a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,15 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### Endpoints|models and leases|deployments are tabs + +The sidebar shows Endpoints and Models as two tabs instead of two panes split +by a divider, so the endpoint list gets the whole height; the main area does +the same for Leases and Deployments. A tab's label carries its counts +(`Leases 1/3`), so the hidden one is still readable. Each pair is one switch +(`InferStackTUI.TABBED_CATALOG`, `TABBED_TABLES`); setting it to `False` +restores the draggable split, and both layouts are tested. + ### One Clean up, in the footer, with a CLI command behind it The leases and deployments panes each had a Clean up button, and both did the diff --git a/infer_stack/tui.py b/infer_stack/tui.py index a07a2440..17272da9 100644 --- a/infer_stack/tui.py +++ b/infer_stack/tui.py @@ -940,6 +940,10 @@ class InferStackTUI(App): #leases-pane { height: 14; min-height: 6; } /* height set via _apply_sizes */ #deployments-pane { height: 1fr; min-height: 6; } #leases, #deployments { height: 1fr; min-height: 3; } + /* tabbed pairs (_pair): the visible pane takes all the room */ + .pair, .pair ContentSwitcher, .pair TabPane { height: 1fr; } + .pair TabPane { padding: 0; } + .pair #models, .pair #leases-pane { height: 1fr; } #docker-tabs { height: 16; min-height: 8; } #logsvc { margin: 0 0 1 0; } #logs, #ps { height: 1fr; background: $surface; } @@ -968,6 +972,11 @@ class InferStackTUI(App): #compose-actions Button { margin: 0 1 0 0; } """ + #: Endpoints|models and leases|deployments as tabs (True), or as two panes + #: split by a draggable divider (False). Each is one switch to flip back. + TABBED_CATALOG = True + TABBED_TABLES = True + BINDINGS = [ # Truly global controls stay in the footer. ('r', 'refresh_now', 'Refresh'), @@ -1145,56 +1154,78 @@ def _compose_app_log(self) -> ComposeResult: yield RichLog(id='applog', highlight=False, markup=True, max_lines=4000, wrap=True) + def _pair(self, tabbed: bool, tabs_id: str, split_id: str, drag, + first: tuple[str, Any], second: tuple[str, Any]) -> ComposeResult: + """Two panes, as tabs or stacked around a draggable divider. + + ``first``/``second`` are ``(tab label, compose function)``. Only the + container differs between the layouts: the panes, and every id the + rest of the app queries, are the same either way. + """ + if tabbed: + with TabbedContent(id=tabs_id, classes='pair'): + for label, body in (first, second): + with TabPane(label, id=f'pane-{label.lower()}'): + yield from body() + else: + yield from first[1]() + yield _Divider('y', drag, id=split_id) + yield from second[1]() + + def _compose_endpoints(self) -> ComposeResult: + yield Static('Endpoints: acquirable configurations', classes='desc') + yield Static('', id='catalog-help') + yield _EndpointTable(id='endpoints', cursor_type='row', + zebra_stripes=True) + with Horizontal(id='endpoint-actions'): + yield Button('Acquire', id='btn-acquire', variant='primary', + action='app.acquire') + yield Button('Add', id='btn-add-endpoint') + yield Button('Edit', id='btn-edit-endpoint') + yield Button('Remove', id='btn-remove-endpoint') + with Horizontal(id='suggest-actions'): + yield Button('✨ Suggest from my GPUs', id='btn-suggest') + + def _compose_models(self) -> ComposeResult: + yield Static('Models: servable weights', classes='desc') + yield DataTable(id='models', cursor_type='row', zebra_stripes=True) + with Horizontal(id='model-actions'): + yield Button('Add', id='btn-add-model') + yield Button('Remove', id='btn-remove-model') + + def _compose_leases(self) -> ComposeResult: + with Vertical(id='leases-pane'): + yield Static('Reservations that map to a deployment. ' + '(space or ctrl/shift-click to multiselect).', + classes='desc') + yield DataTable(id='leases', cursor_type='row', zebra_stripes=True) + with Horizontal(id='lease-actions'): + yield Button('Release', id='btn-release') + yield Button('Release all', id='btn-release-all') + + def _compose_deployments(self) -> ComposeResult: + with Vertical(id='deployments-pane'): + yield Static('Models running.', classes='desc') + yield DataTable(id='deployments', cursor_type='row', + zebra_stripes=True) + with Horizontal(id='deployment-actions'): + yield Button('Evict', id='btn-evict') + yield Button('Evict all idle', id='btn-evict-all') + def _compose_dashboard(self) -> ComposeResult: with Horizontal(id='body'): with Vertical(id='sidebar'): - yield Static( - 'Endpoints: acquirable configurations', classes='desc', - ) - yield Static('', id='catalog-help') - yield _EndpointTable(id='endpoints', cursor_type='row', - zebra_stripes=True) - with Horizontal(id='endpoint-actions'): - yield Button('Acquire', id='btn-acquire', variant='primary', - action='app.acquire') - yield Button('Add', id='btn-add-endpoint') - yield Button('Edit', id='btn-edit-endpoint') - yield Button('Remove', id='btn-remove-endpoint') - with Horizontal(id='suggest-actions'): - yield Button('✨ Suggest from my GPUs', id='btn-suggest') - yield _Divider('y', self._drag_models, id='csplit') - yield Static( - 'Models: servable weights', classes='desc', - ) - yield DataTable(id='models', cursor_type='row', - zebra_stripes=True) - with Horizontal(id='model-actions'): - yield Button('Add', id='btn-add-model') - yield Button('Remove', id='btn-remove-model') + yield from self._pair( + self.TABBED_CATALOG, 'catalog-tabs', 'csplit', self._drag_models, + ('Endpoints', self._compose_endpoints), + ('Models', self._compose_models)) yield _Divider('x', self._drag_sidebar, id='vsplit') with Vertical(id='main'): with Vertical(id='tables'): - with Vertical(id='leases-pane'): - yield Static( - 'Reservations that map to a deployment. ' - '(space or ctrl/shift-click to multiselect).', - classes='desc', - ) - yield DataTable(id='leases', cursor_type='row', - zebra_stripes=True) - with Horizontal(id='lease-actions'): - yield Button('Release', id='btn-release') - yield Button('Release all', id='btn-release-all') - yield _Divider('y', self._drag_tables, id='tsplit') - with Vertical(id='deployments-pane'): - yield Static( - 'Models running.', classes='desc', - ) - yield DataTable(id='deployments', cursor_type='row', - zebra_stripes=True) - with Horizontal(id='deployment-actions'): - yield Button('Evict', id='btn-evict') - yield Button('Evict all idle', id='btn-evict-all') + yield from self._pair( + self.TABBED_TABLES, 'table-tabs', 'tsplit', self._drag_tables, + ('Leases', self._compose_leases), + ('Deployments', self._compose_deployments)) yield _Divider('y', self._drag_logs, id='hsplit') with Collapsible(title='docker', collapsed=True, id='docker'): with TabbedContent(id='docker-tabs'): @@ -1537,8 +1568,11 @@ def _draw_activity(self) -> None: def _apply_sizes(self) -> None: self.query_one('#sidebar').styles.width = self._sidebar_w self.query_one('#docker-tabs').styles.height = self._log_h - self.query_one('#models').styles.height = self._models_h - self.query_one('#leases-pane').styles.height = self._leases_h + # Fixed heights only matter beside a divider; a tab fills its pane. + if not self.TABBED_CATALOG: + self.query_one('#models').styles.height = self._models_h + if not self.TABBED_TABLES: + self.query_one('#leases-pane').styles.height = self._leases_h def _drag_sidebar(self, delta: int) -> None: # Allow the full width range (down to a sliver, up to nearly all of it), @@ -1870,6 +1904,17 @@ def _update_summary(self, leases, deployments, observed) -> None: widget = self.query_one(pane) if widget.border_title != text: widget.border_title = text + if self.TABBED_TABLES: + # A hidden tab's counts show on its label. + tabs = self.query_one('#table-tabs', TabbedContent) + for pane_id, text in ( + ('pane-leases', f'Leases {active}/{len(leases)}'), + ('pane-deployments', f'Deployments {"…" if observing else running}' + f'/{len(deployments)}'), + ): + tab = tabs.get_tab(pane_id) + if str(tab.label) != text: + tab.label = text except Exception: # noqa: BLE001 pass diff --git a/tests/test_tui.py b/tests/test_tui.py index aedd48c1..e1557e87 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -373,35 +373,65 @@ async def scenario(): _run(scenario) -def test_tui_leases_deployments_are_separate_panes(): +def _split_layout(): + """The TUI with both pane pairs stacked around dividers instead of tabbed.""" + from infer_stack.tui import InferStackTUI + + class Split(InferStackTUI): + TABBED_CATALOG = False + TABBED_TABLES = False + return Split + + +@pytest.mark.parametrize('tabbed', [True, False]) +def test_tui_pane_pairs_are_tabs_or_split(tabbed): + """Either layout has the same panes and ids; only the container differs.""" from textual.containers import Vertical + from textual.css.query import NoMatches + from textual.widgets import TabbedContent from infer_stack.tui import InferStackTUI controller, catalog = _ctx() + out = controller.acquire('bob', catalog.resolve_names(['qwen-coder'])) + assert out.lease + cls = InferStackTUI if tabbed else _split_layout() async def scenario(): - app = InferStackTUI(controller, catalog, interval=999, - proc_factory=lambda svc: None) - async with app.run_test() as pilot: + app = cls(controller, catalog, interval=999, proc_factory=lambda svc: None) + async with app.run_test(size=(140, 45)) as pilot: await pilot.pause() - # leases/deployments are their own panes split by a drag handle, - # not collapsibles. - assert isinstance(app.query_one('#leases-pane'), Vertical) - assert isinstance(app.query_one('#deployments-pane'), Vertical) - assert app.query_one('#tsplit') + app.action_refresh() + await app.workers.wait_for_complete() + await pilot.pause() + for pane in ('#leases-pane', '#deployments-pane'): + assert isinstance(app.query_one(pane), Vertical) + for table in ('#endpoints', '#models', '#leases', '#deployments'): + assert app.query_one(table) + dividers = [] + for divider in ('#csplit', '#tsplit'): + try: + dividers.append(app.query_one(divider)) + except NoMatches: + pass + if tabbed: + assert not dividers + tabs = app.query_one('#table-tabs', TabbedContent) + assert str(tabs.get_tab('pane-leases').label) == 'Leases 1/1' + # the pane in front gets the whole height, not a fixed 14 rows + assert app.query_one('#leases-pane').size.height > 14 + else: + assert len(dividers) == 2 _run(scenario) def test_tui_panes_drag_resize(): - from infer_stack.tui import InferStackTUI - controller, catalog = _ctx() async def scenario(): - app = InferStackTUI(controller, catalog, interval=999, - proc_factory=lambda svc: None) + app = _split_layout()(controller, catalog, interval=999, + proc_factory=lambda svc: None) async with app.run_test(size=(120, 40)) as pilot: await pilot.pause() w0, h0 = app._sidebar_w, app._log_h @@ -420,13 +450,11 @@ async def scenario(): def test_tui_dividers_have_a_grab_area(): - from infer_stack.tui import InferStackTUI - controller, catalog = _ctx() async def scenario(): - app = InferStackTUI(controller, catalog, interval=999, - proc_factory=lambda svc: None) + app = _split_layout()(controller, catalog, interval=999, + proc_factory=lambda svc: None) async with app.run_test(size=(120, 40)) as pilot: await pilot.pause() # A 0-size divider can't be grabbed; both must span their cross-axis. From fe88f740dd243fcdd3cfb697429d2dd771b6a0fc Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Fri, 25 Sep 2026 21:05:39 -0400 Subject: [PATCH 22/34] Docs: Compose/KubeAI parity matrix and a roadmap to close the gaps docs/backend-parity.md states the intended relationship (KubeAI is Compose plus a scheduler: more information supplied, more tools installed, the same catalog, verbs, env file and TUI) and records row by row where that holds, where it does not, and which differences are deliberate. docs/planning/backend-parity-roadmap.md plans the rest in phases with exit criteria: one acquire path (K3), day-2 verbs and the TUI through the backend seam, gateway feature parity, placement information, and the gateway inside the cluster for a multi-workstation run. The README's KubeAI section described the pre-leasing setup/deploy workflow and a live kubectl patch that is no longer needed; it now points at the current docs and keeps the cluster prerequisites and debugging checks. The readiness-diagnosis subsection used commands that no longer exist. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 13 + README.md | 425 ++++-------------- .../plan-backend-unification-2026-09-24.md | 5 + docs/backend-parity.md | 148 ++++++ docs/kubeai-backend.md | 4 + docs/planning/backend-parity-roadmap.md | 178 ++++++++ 6 files changed, 429 insertions(+), 344 deletions(-) create mode 100644 docs/backend-parity.md create mode 100644 docs/planning/backend-parity-roadmap.md diff --git a/CHANGELOG.md b/CHANGELOG.md index bd01f31a..819244cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### Compose and KubeAI: a parity matrix and a roadmap + +`docs/backend-parity.md` states the relationship the two backends are meant +to have (KubeAI is Compose plus a scheduler: more information supplied, more +tools installed, the same catalog, verbs, env file and TUI) and records, row +by row, where that holds today, where it does not, and which differences are +deliberate. `docs/planning/backend-parity-roadmap.md` plans the rest in five +phases with exit criteria, from one acquire path to the gateway inside the +cluster for a multi-workstation run. The README's KubeAI section, which still +described the pre-leasing `setup` / `deploy` workflow and a live patch that +is no longer needed, now points at the current docs and keeps only the +cluster prerequisites and the `kubectl` debugging checks. + ### Endpoints|models and leases|deployments are tabs The sidebar shows Endpoints and Models as two tabs instead of two panes split diff --git a/README.md b/README.md index fc51de66..6c76e9aa 100644 --- a/README.md +++ b/README.md @@ -702,27 +702,33 @@ chat settings. ## Backend 2: KubeAI -Use KubeAI when you want Kubernetes-managed serving. +`--backend kubeai` runs the same leasing verbs against a Kubernetes cluster +running [KubeAI](https://www.kubeai.org): the same catalog, ledger, TTLs, +env file and TUI, with `Model` custom resources in place of compose +services and the cluster scheduler in place of the local GPU planner. The +LiteLLM gateway still fronts everything, so a card sees one `OPENAI_BASE_URL`, +the managed key and the endpoint alias on either backend. -### Important rules +* Setup, settings and semantics: [docs/kubeai-backend.md](docs/kubeai-backend.md). +* What matches Compose, what does not yet, and what is deliberate: + [docs/backend-parity.md](docs/backend-parity.md); the plan to close the + rest: [docs/planning/backend-parity-roadmap.md](docs/planning/backend-parity-roadmap.md). -1. **Use the same namespace everywhere.** The namespace in `infer-stack setup --namespace ...` must match the namespace where the KubeAI Helm release already exists. -2. **Prefer the repo-driven path.** The normal path is `setup` -> `validate` -> `render` -> `deploy` -> `status`. -3. **`kubectl port-forward` stays in the foreground.** Leave it running in one terminal and send requests from another. -4. **The first request can take a while.** `/openai/v1/models` may work before chat completions work. The first completion may trigger pod creation, image pull, model load, and compile warmup. -5. **On the current repo version, KubeAI still needs a live workaround after deploy.** The renderer currently produces a `Model` spec that needs a small manual patch to work with the KubeAI version used in these notes. +The short version: -### KubeAI prerequisites - -You need: - -* a working Kubernetes cluster -* `kubectl` -* Helm +```bash +./scripts/bootstrap_k3s.sh # a one-host cluster (k3s + helm) +./scripts/install_kubeai.sh kubeai-values.yaml # the chart, with your resourceProfiles +kubectl -n kubeai port-forward svc/kubeai 8000:80 & +infer-stack config set backend kubeai +infer-stack doctor # cluster -> CRD -> namespace -> gateway +infer-stack acquire --ttl 2h --env-file lease.env --yes +``` -If you want a quick local single-node cluster, K3s is a good option. +### KubeAI prerequisites -Install K3s: +You need a Kubernetes cluster, `kubectl` and Helm. `scripts/bootstrap_k3s.sh` +installs k3s and helm on one host; the steps it runs, by hand: ```bash curl -sfL https://get.k3s.io | sh - @@ -773,332 +779,78 @@ kubectl get node "$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" kubectl get nodes --show-labels | tr ',' '\n' | grep 'nvidia.com/' || true ``` -You want to see a non-empty `nvidia.com/gpu` count and `nvidia.com/*` labels such as product and memory. - -### KubeAI Helm repository +You want a non-empty `nvidia.com/gpu` count and `nvidia.com/*` labels such as product and memory. -```bash -helm repo add kubeai https://www.kubeai.org -helm repo update -``` +### Resource profiles -## Determine which namespace to use - -Before doing anything else, discover whether a `kubeai` release already exists and which namespace it uses. - -```bash -KUBEAI_NAMESPACE="$(helm list -A | awk '$1=="kubeai"{print $2; exit}')" -if [ -z "${KUBEAI_NAMESPACE}" ]; then - KUBEAI_NAMESPACE=default -fi -echo "Using KubeAI namespace: ${KUBEAI_NAMESPACE}" -``` - -If a release already exists, **reuse that namespace**. - -Sanity-check the cluster: - -```bash -kubectl get nodes -kubectl get crd models.kubeai.org || true -helm list -A | grep kubeai || true -kubectl -n "${KUBEAI_NAMESPACE}" get pods || true -kubectl get node "$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" \ - -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}' -``` - ---- - -## Generate the local KubeAI resource-profile file - -Generate a local KubeAI resource-profile file from the labels on this machine. - -For the built-in serving profiles in this repo, keep these names aligned: - -* `gpu-single-default` -* `gpu-tp2-balanced` -* `gpu-tp2-maxctx` - -**Important:** include GPU `requests`, GPU `limits`, and `runtimeClassName: nvidia`. Without those, the model pod can land on the GPU node but still start without `libcuda.so.1` available inside the container. +A `resourceProfile` is what one "GPU unit" means on your cluster; the +catalog's `runtime.resource_profile` (or the `kubeai_resource_profile` +setting) names one, and infer-stack appends the GPU count. Include GPU +`requests`, GPU `limits` and `runtimeClassName: nvidia`: without them the +pod can land on the GPU node and still start without `libcuda.so.1`. ```bash PRODUCT="$(kubectl get nodes -o jsonpath='{.items[0].metadata.labels.nvidia\.com/gpu\.product}')" -MEMORY="$(kubectl get nodes -o jsonpath='{.items[0].metadata.labels.nvidia\.com/gpu\.memory}')" -cat > values-kubeai-local-gpu.yaml < kubeai-values.yaml <:` - * append `:1` in the live `Model` spec -* `libcuda.so.1: cannot open shared object file` - * the pod landed on the GPU node without actually requesting a GPU; fix the resource-profile file to include GPU requests, limits, and `runtimeClassName: nvidia` -* `/models` works but completions 404 with `The model ... does not exist.` - * the served model name does not match the public profile name; apply the live args patch above -* startup probe fails with `connection refused` - * the model pod may still be pulling the image, loading the model, or warming up +* `libcuda.so.1: cannot open shared object file`: the pod did not request a + GPU; fix the resource profile (requests, limits, `runtimeClassName`). +* startup probe fails with `connection refused`: still pulling the image, + loading the model or warming up; the acquire's wait reports which. +* `/models` works but completions 404: the request bypassed the gateway and + used the alias; through the gateway the alias is the model name, directly + against KubeAI the Model name is (`INFER_STACK_ENDPOINT_*` in the env file). --- ## Which backend should I start with? -Start with **Compose** if you want: - -* the fastest path to a working local server -* easy inspection of generated files -* simple single-host iteration - -Move to **KubeAI** when you want: - -* vLLM runtimes on Kubernetes -* KubeAI’s OpenAI-compatible front door -* profile deployment through Kubernetes artifacts - -KubeAI rendering is vLLM-only for now. Profiles that enable Ollama, LiteLLM, or Open WebUI are rejected for `--backend kubeai`. - -A good workflow is: - -1. inspect a profile with `describe-profile` -2. run it with Compose when you want the simplest local deployment -3. move to KubeAI when you want Kubernetes-backed serving - -Compose is the better fit when you already know which profile you want. KubeAI has more first-request overhead because it may need to create pods, pull images, load the model, and warm up the backend. +**Compose** when one workstation is enough: it is the fastest path to a +working server, everything it renders is a file you can read, and it needs +only Docker. +**KubeAI** when the models must run on more than one machine, or a cluster +already exists. It is Compose plus a scheduler: the same catalog, verbs and +env file, with a cluster and `resourceProfiles` supplied. Expect more +first-request overhead (pod creation, image pull, model load). +A catalog written for Compose runs on KubeAI, with two exceptions: ollama +endpoints and custom container launches (`runtime.command` / `mounts`), +which stay Compose-only. [docs/backend-parity.md](docs/backend-parity.md) +has the full matrix. ## vLLM startup caches @@ -1107,39 +859,24 @@ Triton, and CUDA JIT caches. Warm starts avoid redownloading and redoing many compile/JIT steps, but a vLLM model swap still creates a new engine process and must reload weights into GPU memory. -### Diagnosing profile switches and readiness - -`docker compose` health only means that a container-level healthcheck passed. It -is not the same thing as "the routed model can answer a request through the -active access surface." This matters most when switching between two vLLM -profiles that reuse the same runtime service name: the old vLLM process exits, -Docker starts the replacement process, and LiteLLM may remain up while returning -upstream connection errors until vLLM finishes loading the new model. - -Use the dedicated readiness and diagnostics commands after a switch: - -```bash -infer-stack switch gpt2-single --apply --yes -infer-stack wait-ready --model gpt2 -infer-stack smoke-test --model gpt2 -``` +### Diagnosing readiness -For debugging, use: +`docker compose` health only means a container-level healthcheck passed. It +is not "the routed model answers a request through the front door", which is +what `acquire` and `wait` check: a model swap starts a new engine process, +and LiteLLM stays up while returning upstream connection errors until vLLM +has loaded the weights. ```bash -infer-stack diagnose --model gpt2 --generation -infer-stack diagnose --logs --tail 80 +infer-stack acquire --yes # waits for a real generation +infer-stack wait # after acquire --no-wait +infer-stack test # one generation through the gateway +infer-stack status # desired vs running, per deployment +infer-stack logs --tail 80 # the engine or gateway log ``` -`diagnose` prints the resolved provider/gateway/frontend graph, rendered Compose -service state, LiteLLM route probes, direct provider probes, and optional recent -logs. It is intended to distinguish an actual LiteLLM outage from the more -common case where LiteLLM is running but its upstream vLLM runtime is still -booting. - -The Compose service-state diagnostics include Docker's exit code, OOM-killed -flag, restart count, and actual container name. This is important because -`litellm exited with code 137` usually means Docker sent SIGKILL, commonly from -an OOM kill or a forced container replacement, whereas LiteLLM returning HTTP -500 with `Cannot connect to host vllm-*` means LiteLLM is still running but the -upstream vLLM runtime is not ready yet. +A crash-looping engine fails the acquire at once with its error quoted, so +the timeout is only for a model that is loading. Reading Docker's own +signals: `litellm exited with code 137` is a SIGKILL (an OOM kill or a forced +replacement), whereas LiteLLM returning HTTP 500 with `Cannot connect to host +vllm-*` means LiteLLM is running and its upstream vLLM is not ready yet. diff --git a/dev/tmp/plan-backend-unification-2026-09-24.md b/dev/tmp/plan-backend-unification-2026-09-24.md index 130bf9dc..528d61a1 100644 --- a/dev/tmp/plan-backend-unification-2026-09-24.md +++ b/dev/tmp/plan-backend-unification-2026-09-24.md @@ -1,6 +1,11 @@ # Plan: one set of authorities for the Compose and KubeAI backends Status: draft for review, 2026-09-24. Nothing here is implemented. +Update 2026-09-25: K0, K1, K2, K4, K5 (except TUI logs) and G are done; +what remains (K3, TUI logs, dynamic routing on KubeAI, the in-cluster +gateway) is planned as phases in `docs/planning/backend-parity-roadmap.md`, +and the current state is `docs/backend-parity.md`. This file stays as the +audit record. ## Goal diff --git a/docs/backend-parity.md b/docs/backend-parity.md new file mode 100644 index 00000000..520b7044 --- /dev/null +++ b/docs/backend-parity.md @@ -0,0 +1,148 @@ +# Compose and KubeAI: parity and deviations + +infer-stack has one control plane and two realization layers. Everything +above the backend seam is shared: the catalog, the ledger (leases, +deployments, TTLs), the controller (`acquire` / `release` / `wait` / `evict` +/ `gc` / `clean` / `renew`, the admission queue, keep-warm reclaim), the +LiteLLM front door, the env-file contract, and the TUI. A backend only makes +the desired set real: Compose as containers on the local Docker daemon, +KubeAI as `Model` custom resources on a cluster. + +The intended relationship is **KubeAI = Compose + a scheduler**. A catalog +that runs on Compose runs on KubeAI once you say what a GPU unit means on +the cluster and install the cluster tooling. A card does not change: the same +`OPENAI_BASE_URL`, the same managed key, the endpoint alias as the model +name. + +This page records where that holds today (2026-09-25), where it does not, +and which gaps are deliberate. The plan for the rest is +[planning/backend-parity-roadmap.md](planning/backend-parity-roadmap.md); +cluster setup is [kubeai-backend.md](kubeai-backend.md). + +## What you add to move from Compose to KubeAI + +| | Compose | KubeAI adds | +|---|---|---| +| tools | Docker with the NVIDIA runtime | a cluster (`scripts/bootstrap_k3s.sh` for one host, `scripts/join_agent.sh` for another), `kubectl`, `helm`, the KubeAI chart (`scripts/install_kubeai.sh`), the NVIDIA device plugin on GPU nodes | +| information | none: GPUs are discovered with `nvidia-smi` | `resourceProfiles` in the chart values (what one GPU unit requests, and on which nodes), and per endpoint `runtime.resource_profile` or a default `kubeai_resource_profile` | +| settings | `backend compose` | `backend kubeai`; `kubeai_namespace`, `kubeai_base_url`, `kubeai_gateway_upstream` when the defaults (namespace `kubeai`, a port-forward on 8000, the Service's cluster IP) do not hold | +| preflight | none needed | `infer-stack doctor`: cluster → CRD → namespace → gateway | + +The catalog, `acquire … --env-file`, `release`, the TUI and the env file a +card sources are the same on both. + +## Layers + +``` +catalog ──► ledger ──► controller ──► gateway (LiteLLM: aliases, key, routes) + │ + ▼ backend seam: leasing/backend.py + ┌──────────┴───────────┐ + ComposeBackend KubeaiBackend + docker compose kubectl apply of Model CRs + local GPU planner the cluster scheduler + leasing/compose/ leasing/kubeai/ + leasing/kubeai-gateway/ +``` + +On KubeAI the gateway is a Compose project with no engines +(`infer-stack-gateway`) on the host running infer-stack. It routes each +alias to the cluster's KubeAI Service under the Model's name, so `secrets +rotate` and the static superset route table work unchanged. + +## Parity matrix + +**same**: one code path, or verified equivalent. **≈**: the same outcome by +a different mechanism. **gap**: missing on one side and on the roadmap. +**n/a**: does not apply there. **boundary**: deliberately unsupported +(see [planning/known-limitations.md](planning/known-limitations.md)). + +### Client contract + +| | Compose | KubeAI | +|---|---|---| +| one `OPENAI_BASE_URL`, the managed key, the alias as model name | same | same through the gateway; with `litellm false`, Model names and no key | +| `secrets rotate` | same | same | +| env file (`INFER_STACK_*`, `OPENAI_*`) | same | same | +| readiness is a real generation through the front door | same | same | + +### Lifecycle + +| | Compose | KubeAI | +|---|---|---| +| `acquire` / `release` / `wait` / `evict` / `gc` / `clean` / `renew` / `run` / `test` | same | same | +| admission: lease and GPUs committed atomically; a `--queue`d acquire holds nothing | yes | **gap** (roadmap P1): the lease is committed first, then rendered; a Model the cluster cannot place waits out `--timeout`, then rolls back | +| `config publish` | pure preview, then commit | render, then commit (same gap) | +| `--no-apply` / `apply` / `render` | same | same | +| unleased keep-warm yields to leased demand | at placement | ≈ during the wait: `needs_room` evicts the longest-idle, one per 30 s | +| crash-loop fail-fast, the engine's error quoted | same | same (pods, `kubectl logs --previous`) | +| image-pull progress | yes | n/a: the kubelet pulls; `ImagePullBackOff` is a reported wait reason | +| strict `residency()` for decisions, lenient `observe()` for reports | same | same | +| recovery after an interrupted apply (settle check) | yes | ≈ `kubectl apply` is idempotent; nothing is left half-created | + +### Placement + +| | Compose | KubeAI | +|---|---|---| +| where a deployment lands | local planner over `nvidia-smi` | the cluster scheduler, via `resource_profile:` | +| GPU count from TP × PP × DP | same | same | +| `placement.gpu_indices`, `allowed_gpus`, `skip_display_gpus` | yes | n/a: no host indices; node-scoped resource profiles are the equivalent | +| `placement.min_vram_gib`, `infer-stack measure` | yes | warned and ignored; `measure` refused (**gap**, P4) | +| GPU allocations recorded in the ledger | yes | none: the cluster owns them | +| more than one host | boundary | yes: the reason the backend exists | + +### Catalog features + +| | Compose | KubeAI | +|---|---|---| +| vLLM endpoints, `runtime.*` flags, `extra_args`, `env` | same | same (one argument pipeline) | +| served-name rules | same | same (`leasing/naming.py`) | +| `runtime.command`, `runtime.mounts` (custom launchers) | yes | boundary: a stock vLLM Model has no place for them; the render refuses loudly | +| ollama endpoints (`runtime_hosts`) | yes | boundary: daemon-shaped, not model-shaped | +| `min_replicas` / `max_replicas` | n/a | pass-through, default 1/1 | +| weight cache | the host HF cache, mounted | whatever the chart provides; cache profiles are not rendered | + +### Gateway + +| | Compose | KubeAI | +|---|---|---| +| static superset routes, no blip on model churn | same | same | +| `routes` inspect / seed / prune | yes | **gap** (P3): refused, although the registry exists | +| `dynamic_routing` (admin API + Postgres; distinct upstreams for same-model `--dedicated`) | yes | **gap** (P3) | +| Open WebUI (`ui`), reverse proxy | yes | **gap** (P3): the gateway project is rendered with `ui` off | +| `network migrate` / `network check` | yes | n/a: Service addresses are stable | +| where the gateway runs | this host | this host, so it is in every request's path (P5 moves it into the cluster) | + +### Day-2 operations and the TUI + +| | Compose | KubeAI | +|---|---|---| +| `doctor` | nothing to check | four checks | +| `logs`, `ps`, `stack up` / `stack down` | docker compose wrappers | **gap** (P2): they say so and point at `kubectl` | +| TUI: leases, deployments, catalog editing, acquire / release / evict, API tab, settings | same | same | +| TUI: engine log follow, the docker pane, the Up / Down buttons | `docker logs`, `docker compose ps` | **gap** (P2): empty, or "nothing rendered yet" | +| TUI GPU pane | `nvidia-smi` on this host | this host, not the cluster | +| `gc --orphans` (also inside `clean`) | yes | n/a: unlabeled Models are never touched | + +### Testing + +| | Compose | KubeAI | +|---|---|---| +| unit suite with fake runtimes | yes | yes | +| end to end | `dev/e2e_tests/run.sh` (tiers; `--gpu` for serving) | `dev/kubeai_e2e.sh` against a real cluster; k3s with CPU vLLM needs no GPU | + +## Deviations that stay + +- **Custom container launches and ollama daemons** do not run on KubeAI. A + KubeAI Model is stock vLLM. Use `--backend compose` for those endpoints. +- **GPU indices** have no cluster meaning. Kubernetes places by resource, not + by index; the per-node equivalent is a resource profile with a node + selector. +- **One control plane never spans a Compose host and a cluster.** Pick a + backend per data root. + +## Where the gaps close + +[planning/backend-parity-roadmap.md](planning/backend-parity-roadmap.md) +has the phases and their exit criteria. The audit that produced this page, +with what has already landed, is +`dev/tmp/plan-backend-unification-2026-09-24.md`. diff --git a/docs/kubeai-backend.md b/docs/kubeai-backend.md index 59adb851..1878dace 100644 --- a/docs/kubeai-backend.md +++ b/docs/kubeai-backend.md @@ -31,6 +31,10 @@ semantics carry over: an idle `keep-warm` deployment keeps its CR (model stays resident); `stop` deployments are pruned on release; `evict`/`gc` free the cluster. Hand-applied Models without the managed label are never touched. +Where the two backends match and where they still differ, row by row: +[backend-parity.md](backend-parity.md); the plan for the rest is +[planning/backend-parity-roadmap.md](planning/backend-parity-roadmap.md). + ## One-time cluster setup ```bash diff --git a/docs/planning/backend-parity-roadmap.md b/docs/planning/backend-parity-roadmap.md new file mode 100644 index 00000000..47373b75 --- /dev/null +++ b/docs/planning/backend-parity-roadmap.md @@ -0,0 +1,178 @@ +# Backend parity roadmap: KubeAI as a superset of Compose + +**Status:** proposed 2026-09-25 · **P0 done** 2026-09-24 on +`dev/backend-unification` · P1–P5 not started · P6 is ongoing. +**Current state:** [../backend-parity.md](../backend-parity.md). +**Origin:** the scale-up run needs more than one workstation, and the +KubeAI backend had drifted from Compose for three months before the +2026-09-24 audit (`dev/tmp/plan-backend-unification-2026-09-24.md`). + +--- + +## Objective (read this first; it outlives the plan) + +A card, a catalog and an operator's habits move from one workstation to a +cluster with one change: naming what a GPU unit is. Concretely, + +1. every row of the parity matrix reads *same*, *≈*, *n/a* or *boundary*; + none reads *gap*; +2. a feature is written once. No code outside the backend modules and the + CLI's construction of them branches on the backend kind. + +Compose stays the fastest local path and loses nothing. KubeAI is Compose +plus a scheduler, with more information supplied and more tools installed; +it is not a sibling with its own habits. + +## Principles + +- **Superset, not sibling.** KubeAI gains what Compose has. Where a Compose + feature has no cluster meaning (GPU indices, `network migrate`), the + matrix says *n/a* and the CLI says why; it does not grow a second design. +- **The seam is the only place kind matters.** `isinstance(…, ComposeBackend)` + in the CLI or TUI is a smell. Each phase removes some. The Compose-only + escape hatches (`stack up`/`down` as raw compose) stay, and say so. +- **Verified on a real cluster.** A phase is done when its step in + `dev/kubeai_e2e.sh` passes on k3s, not when its fakes pass. This is how + the backend drifted last time. +- **One reviewable change per phase**, the Compose suite green throughout. + +## Phases + +### P0. One contract, one liveness view, one diagnosis — done + +The 2026-09-24 audit landed: an e2e on k3s (K0); the LiteLLM gateway +fronting both backends, so the alias and key are the same everywhere (K1); +a strict `residency()` built from pods (K2); one startup diagnosis over +containers or pods (K4); one served-name rule, one GPU count, `runtime.env` +on KubeAI (K5); the gateway as its own module (G); and unleased keep-warm +yielding to leased demand on both backends. + +### P1. One acquire path + +**Closes:** the admission and `config publish` rows. + +Today `Controller._admission_mode()` is true only for a backend with +`residency` + `preview` + `converge`, which is Compose. KubeAI takes the +pre-September branch in five places (render, acquire, `observe_state`, +`config publish`, `renew`). + +- `KubeaiBackend.preview(desired, placement, approve)`: render the Models + to memory with a digest, write nothing. Placement inputs are accepted and + ignored; `unplaced` is exactly the unrenderable set (no profile, engine, + custom launch). +- Admission accepts a backend with no GPU accounting: Compose reports + assignments, KubeAI reports none, and the ledger's allocation table stays + empty for it. A Pending pod is a wait reason, never an unplaced error. +- `--queue` on KubeAI: admitted at once; the cluster is the queue. Say so + in the option's help. +- `MemoryBackend` / `NullBackend` get a trivial `residency` and `preview` + so the tests run the one path. +- Delete the non-admission branches and `_admission_mode()`. + +**Exit:** `_admission_mode` is gone; a render failure on KubeAI rolls the +lease back before any `kubectl`; e2e passes; `test_controller` runs its +acquire scenarios on all three fakes. +**Size:** medium. The controller is the most-changed module; start from a +green e2e and keep the diff to the five sites. + +### P2. Day-2 verbs and the TUI through the seam + +**Closes:** `logs` / `ps` / `stack`, the TUI's log follow, docker pane and +Up / Down, and `measure`. + +- Two backend methods: `instances()` (from `residency()`: deployment, unit + name, state, restarts, age) and `stream_logs(target, *, follow, tail)` + (`docker logs -f` / `kubectl logs -f`). +- `ps` prints `instances()`. `logs` streams. `stack up` is `apply`; `stack + down` is `backend.down()`, which both backends have. The raw compose form + moves under `stack compose …` and stays Compose-only by name. +- TUI: the docker pane becomes an instances pane; the log follower uses + `stream_logs`; Up / Down call `apply` / `down`. +- `measure`: lift the guard (`deployment_logs` already exists on KubeAI) + after checking vLLM's memory line reaches the pod log. + +**Exit:** on k3s the TUI follows a Model's log and lists its pod; `infer-stack +logs ` and `ps` work on both backends with the same output shape. +**Size:** medium, mostly plumbing; no controller change. + +### P3. Gateway feature parity + +**Closes:** `routes` on KubeAI, `dynamic_routing`, Open WebUI and the reverse +proxy in front of a cluster. + +- The `routes` commands and `secrets rotate` resolve `controller.backend.gateway` + on either backend instead of checking the backend kind. +- The KubeAI gateway project is rendered with the same `ui`, `reverse_proxy` + and `dynamic_routing` settings as Compose. Postgres for dynamic routing is + a gateway-project service, so nothing engine-side changes. + +**Exit:** `routes list` on KubeAI; Open WebUI in front of a cluster; an e2e +step that acquires the same model `--dedicated` twice under dynamic routing +and gets two Models and two routes. +**Size:** small to medium; the `Gateway` class already owns the pieces. + +### P4. Placement information parity (optional) + +**Closes:** `min_vram_gib` / `measure`; less hand-written cluster +information. + +- `catalog suggest` on KubeAI reads the device plugin's node labels + (`nvidia.com/gpu.product`, `.memory`) and proposes `resourceProfiles` + (today the README does this by hand). +- With several profile sizes, `min_vram_gib` picks the smallest profile + whose GPU memory fits when an endpoint names none; otherwise the + warn-and-ignore stays. + +**Exit:** a catalog with `min_vram_gib` and no `resource_profile` lands on +the right profile on a cluster with two sizes. +**Size:** small. Shrinks "the information you add"; skip if nobody runs a +mixed-GPU cluster. + +### P5. The multi-workstation shape + +**Closes:** the gateway on one host, the port-forward default. + +- Render the gateway in the cluster (a Deployment + Service behind an + ingress) as the second gateway target; the Compose-project gateway stays + the single-host default. `secrets rotate` becomes a Secret update and a + rollout; `doctor` checks the ingress; `kubeai_base_url` defaults to it. +- A second workstation joins with `scripts/join_agent.sh`; resource + profiles with node selectors say which GPUs live where. Write the "add a + workstation" runbook into `kubeai-backend.md`. + +**Exit:** on two nodes, a card on node A leases a model that lands on node +B and talks to it through the in-cluster gateway; `secrets rotate` works. +**Size:** medium to large. The only phase that adds a second renderer for +the gateway, and the only one that needs a second machine. Last. + +### P6. One test surface (ongoing) + +- Parametrize the controller tests over the Memory, fake-Compose and + fake-KubeAI backends; `tests/test_parity.py` runs each *same* row's + scenario on both fakes. +- `dev/kubeai_e2e.sh` gains one step per phase. +- Finishing a phase updates the matrix in `backend-parity.md`; the plan is + done when the matrix has no *gap* row. + +## Not in scope + +- more than one cluster, or scheduling across a Compose host and a cluster; +- KubeAI autoscaling beyond `min_replicas` / `max_replicas`: the lease + decides residency; +- custom container launches and ollama daemons on KubeAI (a boundary in + [known-limitations.md](known-limitations.md)); +- Slurm, which is its own track ([../slurm-compatibility.md](../slurm-compatibility.md)); +- moving the Compose backend onto Kubernetes primitives. + +## Order and size + +| phase | size | needs | gives | +|---|---|---|---| +| P1 | medium | a green e2e | one controller; atomic acquire on a cluster | +| P2 | medium | P1 (instances from residency) | the TUI and day-2 verbs on a cluster | +| P3 | small–medium | none | dynamic routing, Open WebUI on a cluster | +| P4 | small | a mixed-GPU cluster to verify | less hand-written cluster information | +| P5 | medium–large | a second machine | the gateway off the operator's host | + +P1 → P2 → P3 is single-host-cluster parity: k3s on one workstation, with +everything a Compose user has. P5 is the multi-workstation step. From d6a19c48058acf0dddccf307ad2e0d264ab50241 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Fri, 25 Sep 2026 21:27:45 -0400 Subject: [PATCH 23/34] P1a: KubeAI acquires go through admission, as Compose's do KubeAI gains an in-memory preview (and plan_on_idle_host) over one shared render, and converge accepts placement=, so Controller._admission_mode() is true for both real backends. An unrenderable acquire now writes no lease and runs no kubectl; a queued one fails at once. Duplicate authorities removed on the way: - "does this backend allocate GPUs" is allocates_gpus(), read by the controller's accounting and the CLI's status view; on KubeAI every deployment commits an empty allocation, so nothing is unresolved. - the approval digest and pre-approval move from ComposeBackend into ConvergeScaffold. Compose's stable-address network and container adoption are handed only to a backend that declares them. The roadmap splits P1 after review: P1b (delete the legacy branch, which only test fakes use) goes with P6; deferred duplicates are tabled there. Verified: full suite, and dev/kubeai_e2e.sh on k3s with a new admission step and the make-room step. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 17 ++++++ dev/kubeai_e2e.sh | 24 ++++++++ docs/backend-parity.md | 5 +- docs/kubeai-backend.md | 16 +++-- docs/planning/backend-parity-roadmap.md | 79 +++++++++++++++++++------ infer_stack/backends/kubeai.py | 59 +++++++++++++++--- infer_stack/cli/commands_leasing.py | 10 +++- infer_stack/leasing/backend.py | 56 ++++++++++++++++-- infer_stack/leasing/compose.py | 24 +------- infer_stack/leasing/controller.py | 43 +++++++++----- tests/test_leasing_kubeai.py | 49 +++++++++++++-- 11 files changed, 299 insertions(+), 83 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 819244cb..f7acb9fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,23 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### KubeAI acquires go through admission, as Compose's do + +An acquire on the KubeAI backend is now previewed in memory and commits its +lease only if every deployment renders, the same path Compose takes. A +refused acquire (no resource profile, an ollama endpoint, a served-name +collision) writes no lease and runs no `kubectl`; before, the lease was +committed, rendered, and then rolled back. A `--queue` acquire of such an +endpoint fails at once instead of waiting out its timeout. The cluster +schedules, so a KubeAI deployment commits an empty GPU allocation; `leases` +still shows no GPU for it, and a `--no-apply` acquire now reports it as +cluster-scheduled rather than unplaced. An idle +keep-warm Model is kept only while its pod has started, as a keep-warm +container is on Compose. The approval digest moved into the scaffold both +backends share, and one function now answers whether a backend allocates +GPUs. Removing the older acquire branch, which only test fakes still use, +is the next step (roadmap P1b). + ### Compose and KubeAI: a parity matrix and a roadmap `docs/backend-parity.md` states the relationship the two backends are meant diff --git a/dev/kubeai_e2e.sh b/dev/kubeai_e2e.sh index aba5df1b..f735e1e2 100755 --- a/dev/kubeai_e2e.sh +++ b/dev/kubeai_e2e.sh @@ -109,6 +109,30 @@ remaining=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ -l infer-stack/managed=true -o name | wc -l) [ "$remaining" = 0 ] || { echo "!! model not pruned"; exit 1; } +echo '== an unrenderable endpoint is refused before anything is written' +# No resource_profile and no kubeai_resource_profile default: admission's +# preview refuses it, so no lease is committed and kubectl applies nothing; +# --queue fails at once, since waiting cannot make it renderable. +cat >> "$WORK/config/catalog.yaml" < "$WORK/bad.log" 2>&1; then + echo '!! an endpoint with no resource profile was admitted' >&2; exit 1 +fi +grep -q 'resource profile' "$WORK/bad.log" \ + || { cat "$WORK/bad.log" >&2; echo '!! the refusal did not name the cause' >&2; exit 1; } +active=$(run_is leases --json | python3 -c \ + 'import json,sys; print(sum(le["state"] == "active" for le in json.load(sys.stdin)["leases"]))') +[ "$active" = 0 ] || { echo "!! $active lease(s) left active" >&2; exit 1; } +remaining=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ + -l infer-stack/managed=true -o name | wc -l) +[ "$remaining" = 0 ] || { echo '!! a Model was applied for it' >&2; exit 1; } +echo ' refused at admission: no lease, no Model' + if [ "${E2E_MAKE_ROOM:-0}" = 1 ]; then # Needs a profile only one Model fits at a time (`cpu-half` in # dev/e2e_tests/kubeai-cpu-values.yaml): E2E_ROOM_PROFILE=cpu-half. diff --git a/docs/backend-parity.md b/docs/backend-parity.md index 520b7044..f4741e44 100644 --- a/docs/backend-parity.md +++ b/docs/backend-parity.md @@ -70,8 +70,9 @@ a different mechanism. **gap**: missing on one side and on the roadmap. | | Compose | KubeAI | |---|---|---| | `acquire` / `release` / `wait` / `evict` / `gc` / `clean` / `renew` / `run` / `test` | same | same | -| admission: lease and GPUs committed atomically; a `--queue`d acquire holds nothing | yes | **gap** (roadmap P1): the lease is committed first, then rendered; a Model the cluster cannot place waits out `--timeout`, then rolls back | -| `config publish` | pure preview, then commit | render, then commit (same gap) | +| admission: previewed in memory, then the lease committed with its allocation; a refused acquire writes nothing | yes | same; the allocation is empty (the cluster places), so admission decides renderability, and a pod the cluster cannot place is a wait reason | +| `--queue` | waits for a free GPU | ≈ admitted at once, the cluster is the queue; an unrenderable endpoint fails at once | +| `config publish` | pure preview, then commit | same | | `--no-apply` / `apply` / `render` | same | same | | unleased keep-warm yields to leased demand | at placement | ≈ during the wait: `needs_room` evicts the longest-idle, one per 30 s | | crash-loop fail-fast, the engine's error quoted | same | same (pods, `kubectl logs --previous`) | diff --git a/docs/kubeai-backend.md b/docs/kubeai-backend.md index 1878dace..3be5b96e 100644 --- a/docs/kubeai-backend.md +++ b/docs/kubeai-backend.md @@ -108,10 +108,18 @@ infer-stack release --env-file lease.env - **Readiness is a real generation** through the gateway (same philosophy as compose): a Model CR existing — even with ready replicas — is not proof it can serve. -- **Placement can only fail at admission.** The render never rejects for - capacity (the cluster schedules); a Model the cluster cannot place sits - not-ready until the acquire's `--timeout`, which then rolls the lease back. - The wait says why (`pod: Unschedulable`, `pod: ImagePullBackOff`). +- **Admission is the same as on compose, minus GPU accounting.** An acquire + is previewed in memory and commits its lease only if every deployment + renders; a refused one (no resource profile, an ollama endpoint, a + served-name collision) writes no lease and runs no `kubectl`. The cluster + schedules, so capacity is never an admission reason and `--queue` is + admitted at once. A Model the cluster cannot place sits not-ready until + the acquire's `--timeout`, which rolls the lease back; the wait says why + (`pod: Unschedulable`, `pod: ImagePullBackOff`). +- **An idle keep-warm Model stays only while its pod has started**, as an + idle keep-warm container does on compose (a crash-looping pod counts, like + a restarting container). One whose pod is gone, still Pending or finished + is pruned at the next render. - **A keep-warm model without a lease gives way to one with a lease.** When a leased Model's pod is `Unschedulable`, the wait evicts the longest-idle keep-warm deployment, one per 30 s, until it fits. Compose applies the same diff --git a/docs/planning/backend-parity-roadmap.md b/docs/planning/backend-parity-roadmap.md index 47373b75..cc554423 100644 --- a/docs/planning/backend-parity-roadmap.md +++ b/docs/planning/backend-parity-roadmap.md @@ -1,7 +1,8 @@ # Backend parity roadmap: KubeAI as a superset of Compose **Status:** proposed 2026-09-25 · **P0 done** 2026-09-24 on -`dev/backend-unification` · P1–P5 not started · P6 is ongoing. +`dev/backend-unification` · **P1a done** 2026-09-25 · P1b–P5 not started · +P6 is ongoing. **Current state:** [../backend-parity.md](../backend-parity.md). **Origin:** the scale-up run needs more than one workstation, and the KubeAI backend had drifted from Compose for three months before the @@ -56,24 +57,50 @@ Today `Controller._admission_mode()` is true only for a backend with pre-September branch in five places (render, acquire, `observe_state`, `config publish`, `renew`). +Review 2026-09-25 against the code split this in two. The admission path +assumes GPU accounting in three places (`_backfill_allocations`, +`_unresolved_allocations`, `_admission_view` keeps a LIVE deployment only +with committed GPUs), so a `preview` alone would drop every KubeAI lease from +the render. And about fifteen test fakes drive the legacy path on purpose +(queue, lock, serialised publication), so deleting it is test work, not +controller work. + +**P1a. KubeAI on the admission path.** + +- One authority for "this backend allocates GPUs": `allocates_gpus` + (Compose true, KubeAI false). The controller's accounting reads it + through one helper; on KubeAI every deployment commits an empty + allocation, so nothing is ever unresolved. - `KubeaiBackend.preview(desired, placement, approve)`: render the Models to memory with a digest, write nothing. Placement inputs are accepted and - ignored; `unplaced` is exactly the unrenderable set (no profile, engine, - custom launch). -- Admission accepts a backend with no GPU accounting: Compose reports - assignments, KubeAI reports none, and the ledger's allocation table stays - empty for it. A Pending pod is a wait reason, never an unplaced error. -- `--queue` on KubeAI: admitted at once; the cluster is the queue. Say so - in the option's help. -- `MemoryBackend` / `NullBackend` get a trivial `residency` and `preview` - so the tests run the one path. -- Delete the non-admission branches and `_admission_mode()`. - -**Exit:** `_admission_mode` is gone; a render failure on KubeAI rolls the -lease back before any `kubectl`; e2e passes; `test_controller` runs its -acquire scenarios on all three fakes. -**Size:** medium. The controller is the most-changed module; start from a -green e2e and keep the diff to the five sites. + ignored; the plan assigns every renderable deployment no GPUs. + `plan_on_idle_host` is the same render, so a `--queue` acquire of an + unrenderable endpoint fails at once instead of waiting out the timeout. +- `converge(..., placement=)` on KubeAI. Without it the controller's + `TypeError` fallback would render **and apply** in one step. +- The approved-digest guard (`_planned_digest`, pre-approval, + `last_planned_digest` / `last_preview_digest`) moves from `ComposeBackend` + to `ConvergeScaffold`: one approval mechanism, not a copy. +- Compose's stable addresses and container adoption stay Compose-only by + capability (`network`, `adopted`), not by joining KubeAI to them. +- `--queue` on KubeAI: admitted at once; the cluster is the queue. + +**P1a done 2026-09-25.** Verified on k3s by `dev/kubeai_e2e.sh`, which +gained the step "an unrenderable endpoint is refused before anything is +written" (a `--queue` acquire, no lease, no Model), and still passes the +make-room step, which now runs on the admission path. + +**P1b. Delete the legacy branch** (with P6). +`MemoryBackend` / `NullBackend` and the test fakes get a trivial +`residency` and `preview`; then the non-admission branches and +`_admission_mode()` go. Until then the legacy path serves only test fakes +and `realize`/`teardown` backends, and no new code may depend on it. + +**Exit:** P1a: `_admission_mode()` is true for both real backends; a render +failure on KubeAI rolls the lease back before any `kubectl`; e2e passes. +P1b: `_admission_mode` is gone; `test_controller` runs its acquire scenarios +on all three fakes. +**Size:** P1a medium, P1b medium (mostly tests). ### P2. Day-2 verbs and the TUI through the seam @@ -154,6 +181,21 @@ the gateway, and the only one that needs a second machine. Last. - Finishing a phase updates the matrix in `backend-parity.md`; the plan is done when the matrix has no *gap* row. +## Duplicate authorities + +The rule while executing: a duplicate authority found on the way is +refactored when it is small, recorded here when it is not, and never made +worse. A blocker is fixed whatever its size. + +| authority | where it was | status | +|---|---|---| +| "this backend allocates GPUs" | inferred from which methods a backend has | **fixed** (P1a): `allocates_gpus()` in `leasing/backend.py`, read by the controller and the CLI | +| approval digest and pre-approval | a copy inside `ComposeBackend` | **fixed** (P1a): `ConvergeScaffold`, shared | +| KubeAI render vs its plan | `converge` rendered inline | **fixed** (P1a): one `_render_documents` behind `converge`, `preview` and `plan_on_idle_host` | +| the acquire path | admission and a legacy branch in five places | deferred to P1b | +| `_render`'s one-shot `converge(desired)` fallback, which applies | a second render contract for old backends | deferred to P1b: it goes with the legacy branch | +| the KubeAI gateway's approval | the gateway project asks its own diff approval at render, after the lease commits, not in the admission preview | deferred to P3. Same result under `--yes`; interactively, a declined gateway change rolls the lease back after the commit | + ## Not in scope - more than one cluster, or scheduling across a Compose host and a cluster; @@ -168,7 +210,8 @@ the gateway, and the only one that needs a second machine. Last. | phase | size | needs | gives | |---|---|---|---| -| P1 | medium | a green e2e | one controller; atomic acquire on a cluster | +| P1a | medium | a green e2e | atomic acquire on a cluster | +| P1b | medium | P1a; with P6 | one controller path | | P2 | medium | P1 (instances from residency) | the TUI and day-2 verbs on a cluster | | P3 | small–medium | none | dynamic routing, Open WebUI on a cluster | | P4 | small | a mixed-GPU cluster to verify | less hand-written cluster information | diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index a65bdf0c..94f0af2c 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -392,8 +392,52 @@ def _render_gateway(self, rendered: RenderedModels) -> None: # -- converge-style surface ------------------------------------------------ - def converge(self, desired: list[Deployment], *, apply: bool = True): - """Render the desired Model set, then optionally apply it.""" + #: The cluster schedules: no host GPU indices are allocated or recorded, + #: and admission commits an empty allocation for every deployment. + allocates_gpus = False + + def _render_documents(self, desired: list[Deployment]): + """``(plan, rendered, planned)`` in memory: the one KubeAI render. + + The plan assigns every renderable deployment no GPUs; the + unrenderable ones are left out, with the render's reasons. + """ + from ..leasing.placement import GpuPlan + + rendered = render_models( + list(desired), + namespace=self.namespace, + default_resource_profile=self.default_resource_profile, + ) + plan = GpuPlan( + assignments={g.id: [] for g in desired if g.id not in rendered.unrenderable}, + errors=list(rendered.errors), + ) + return plan, rendered, {self.models_file: rendered.text} + + def preview(self, desired: list[Deployment], placement=None, *, approve: bool = False): + """Render ``desired`` as :meth:`converge` would; write nothing. + + ``placement`` is accepted for the admission interface and ignored: + the cluster places. Returns ``(plan, rendered)``. + """ + plan, rendered, planned = self._render_documents(desired) + self._preview_approval(planned, approve=approve) + return plan, rendered + + def plan_on_idle_host(self, desired: list[Deployment]): + """Whether ``desired`` could ever be served here: renderable or not. + + Capacity is the cluster's to decide, so only a render failure is + permanent; that lets a queued acquire of one fail at once. + """ + return self._render_documents(desired)[0] + + def converge(self, desired: list[Deployment], *, apply: bool = True, placement=None): + """Render the desired Model set, then optionally apply it. + + ``placement`` (admission inputs) is ignored: the cluster places. + """ from .._log import logger desired = list(desired) @@ -416,17 +460,14 @@ def converge(self, desired: list[Deployment], *, apply: bool = True): 'requirement via the resource profile instead)', g.id, ) - rendered = render_models( - desired, - namespace=self.namespace, - default_resource_profile=self.default_resource_profile, - ) + plan, rendered, planned = self._render_documents(desired) self.last_errors = list(rendered.errors) self.last_unplaced = set(rendered.unrenderable) - self.last_assignments = {} + self.last_assignments = {} # the cluster places for err in rendered.errors: logger.warning(' render: {}', err) - self._approve_changes({self.models_file: rendered.text}) + self.last_planned_digest = self._planned_digest(planned) + self._approve_changes(planned) self._atomic_write(self.models_file, rendered.text) self._save_sidecar( { diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index 507b3ff9..cce3120d 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -511,9 +511,13 @@ def _emit_staged(config, controller, outcome) -> int: }, indent=2)) return 0 print(f'staged {outcome.lease.id} (owner={outcome.lease.owner}) — not applied') + from ..leasing.backend import allocates_gpus + + local = allocates_gpus(controller.backend) for g in outcome.deployments: eps = ', '.join(sorted(g.served)) or g.id - print(f' {eps}: {_gpu_where(assignments.get(g.id))} ({g.id})') + where = _gpu_where(assignments.get(g.id)) if local else 'cluster-scheduled' + print(f' {eps}: {where} ({g.id})') path = _compose_file_path(controller) if path: print(f' compose: {path}') @@ -1898,6 +1902,10 @@ def _placement_view(controller): except Exception: # noqa: BLE001 - status must never crash pass assignments: dict[str, list[int]] = {} + from ..leasing.backend import allocates_gpus + + if not allocates_gpus(backend): + return observed, assignments # the cluster places; no GPU indices if controller._admission_mode(): # Committed allocations, and idle residents' physical GPUs; the # legacy planner view would show placements admission would not make. diff --git a/infer_stack/leasing/backend.py b/infer_stack/leasing/backend.py index 0c1db433..28356ddc 100644 --- a/infer_stack/leasing/backend.py +++ b/infer_stack/leasing/backend.py @@ -85,6 +85,17 @@ def __init__(self, deployment_ids, reasons): ) +def allocates_gpus(backend) -> bool: + """Whether ``backend`` allocates host GPU indices itself. + + True for Compose, which places on this host and records the GPUs each + deployment holds. False for a backend whose cluster schedules (KubeAI): + admission commits an empty allocation, and no GPU is ever "unresolved". + The one place the controller and the CLI ask this. + """ + return bool(getattr(backend, 'allocates_gpus', True)) + + @runtime_checkable class Backend(Protocol): """What the :class:`Controller` needs from a serving backend. @@ -156,12 +167,14 @@ def apply(self) -> bool | None: class AdmissionBackend(ConvergeBackend, Protocol): """The surface admission mode uses (see ``Controller._admission_mode``). - Today only :class:`~infer_stack.leasing.compose.ComposeBackend` has it: - strict residency, an in-memory placement ``preview``, and the Compose - network and adoption state the controller hands it. The controller - reaches these through ``Controller._admitting``, only after the - capability check, so the type checker sees one named capability instead - of attributes a minimal :class:`Backend` does not have. + Both real backends have it: strict residency and an in-memory + ``preview``. Compose also takes the stable-address network and the + container-adoption table from the controller (``network``, + ``on_addresses``, ``adopted``); those are Compose-only, and the + controller hands them over only to a backend that declares them. The + controller reaches these through ``Controller._admitting``, only after + the capability check, so the type checker sees one named capability + instead of attributes a minimal :class:`Backend` does not have. """ network: dict[str, Any] | None @@ -254,9 +267,37 @@ def _save_sidecar(self, data: dict) -> None: self._atomic_write(self._state_file, json.dumps(data, indent=2)) + #: Digest of files an admission preview already had approved. + _preapproved: str | None = None + #: Digest of the files the last render produced (approved-digest guard). + last_planned_digest: str | None = None + #: Digest of the files the last preview produced. + last_preview_digest: str | None = None + + @staticmethod + def _planned_digest(planned: dict) -> str: + import hashlib + import json + + material = json.dumps({str(k): v for k, v in planned.items()}, sort_keys=True) + return hashlib.sha256(material.encode('utf-8')).hexdigest() + + def _preview_approval(self, planned: dict, *, approve: bool) -> None: + """Record a preview's digest; with ``approve``, ask now, not after commit. + + The render that follows the commit skips the prompt when it produces + the same files (see :meth:`_approve_changes`). + """ + self.last_preview_digest = self._planned_digest(planned) + if approve: + self._approve_changes(planned) + self._preapproved = self.last_preview_digest + def _approve_changes(self, planned: dict) -> None: """Show pending rendered-state changes and confirm them. + Files a preview already had approved (same digest) pass silently once. + ``planned`` maps target paths to their new content. When nothing actually changed, this is a quiet no-op. When ``assume_yes`` (scripts / non-interactive / ``--yes``), it applies after a one-line log. @@ -265,6 +306,9 @@ def _approve_changes(self, planned: dict) -> None: """ from .._log import logger + preapproved, self._preapproved = self._preapproved, None + if preapproved is not None and self._planned_digest(planned) == preapproved: + return changed = { p: text for p, text in planned.items() diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index ea44570f..ff209d2c 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -1389,10 +1389,7 @@ def preview(self, desired: list[Deployment], placement=None, *, approve: bool = then does not ask again as long as it produces the same files. """ docs = self._render_documents(list(desired), placement) - self.last_preview_digest = self._planned_digest(docs['planned']) - if approve: - self._approve_changes(docs['planned']) - self._preapproved = self.last_preview_digest + self._preview_approval(docs['planned'], approve=approve) return docs['plan'], docs['rendered'] def _render_documents(self, desired: list[Deployment], placement) -> dict[str, Any]: @@ -1455,18 +1452,6 @@ def _render_documents(self, desired: list[Deployment], placement) -> dict[str, A #: newly allocated addresses (append-only). on_addresses: Any = None - @staticmethod - def _planned_digest(planned: dict) -> str: - material = json.dumps({str(k): v for k, v in planned.items()}, sort_keys=True) - return hashlib.sha256(material.encode('utf-8')).hexdigest() - - #: Digest of files an admission preview already had approved. - _preapproved: str | None = None - #: Digest of the files the last render produced (approved-digest guard). - last_planned_digest: str | None = None - #: Digest of the files the last preview produced. - last_preview_digest: str | None = None - def pull_images(self, images) -> list[str]: """Pull ``images``; ``config publish`` passes :func:`profile_images`.""" from .._log import logger @@ -1476,13 +1461,6 @@ def pull_images(self, images) -> list[str]: self.run(['docker', 'pull', image]) return sorted(set(images)) - def _approve_changes(self, planned: dict) -> None: - if self._preapproved is not None and self._planned_digest(planned) == self._preapproved: - self._preapproved = None - return - self._preapproved = None - super()._approve_changes(planned) - def plan_on_idle_host(self, desired: list[Deployment]): """Placement for ``desired`` alone, as if nothing else were running. diff --git a/infer_stack/leasing/controller.py b/infer_stack/leasing/controller.py index de969761..7e0c29f6 100644 --- a/infer_stack/leasing/controller.py +++ b/infer_stack/leasing/controller.py @@ -485,7 +485,8 @@ def _render(self) -> ReconcileResult: self._backfill_allocations(residency) self._prepare_network() desired, placement = self._admission_view(residency) - self._admitting.adopted = self._prune_adopted(residency) + if hasattr(self.backend, 'adopted'): + self._admitting.adopted = self._prune_adopted(residency) else: desired = self.desired_deployments() # Bound once rather than probed with hasattr: the capability check is @@ -520,7 +521,7 @@ def _render(self) -> ReconcileResult: displaced=list(getattr(self.backend, 'last_displaced', ()) or ()), degraded=list(getattr(self.backend, 'last_degraded', ()) or ()), ) - if placement is not None: + if placement is not None and hasattr(self.backend, 'adopted'): self._adopt_existing(residency) return rec desired_ids = {g.id for g in desired} @@ -554,16 +555,19 @@ def _render(self) -> ReconcileResult: # -- admission (plan steps P5, P6, P9) ------------------------------------ # - # Backends with strict residency and an in-memory preview (Compose) get - # admission semantics: + # Backends with strict residency and an in-memory preview (Compose and + # KubeAI) get admission semantics: # * a LIVE deployment holds a committed allocation (assigned_gpus); # * an IDLE keep-warm deployment is only an optional candidate, and only # while it is uniquely resident; it yields its GPUs to demand and is # never started; # * an acquire is previewed in memory (placement AND render) and commits # its lease together with its allocations, or commits nothing. - # Other backends (KubeAI, where the cluster schedules; test fakes) keep the - # previous behaviour. + # Where the cluster schedules (KubeAI, ``allocates_gpus`` false) every + # deployment commits an empty allocation: admission then decides only + # renderability, and a Pending pod is a wait reason, not an unplaced + # error. Test fakes without residency/preview keep the previous path + # (roadmap P1b removes it). def _stored_state(self, lease_id: str): """A lease's state as stored (not virtually expired), or ``None``.""" @@ -628,7 +632,13 @@ def _admission_view( return [*required.values(), *optional], inputs def _prepare_network(self) -> None: - """Give the backend the stable-address table, once a network is migrated.""" + """Give the backend the stable-address table, once a network is migrated. + + Compose only: a backend without a ``network`` attribute has no + host network to stamp addresses on. + """ + if not hasattr(self.backend, 'network'): + return config = self.ledger.network_config() if config is None: self._admitting.network = None @@ -859,6 +869,13 @@ def remove_orphans(self, confirm: Callable[[list], bool]) -> list: self._admitting.run(['docker', 'rm', '-f', *[c.container_id for c in orphans]]) return orphans + def _gpu_units(self, deployment) -> int: + """GPUs admission must account for: none where the cluster schedules.""" + from .backend import allocates_gpus + from .placement import required_gpu_count + + return required_gpu_count(deployment) if allocates_gpus(self.backend) else 0 + def _backfill_allocations(self, residency) -> list[str]: """Adopt allocations for LIVE deployments that predate them. @@ -867,14 +884,12 @@ def _backfill_allocations(self, residency) -> list[str]: GPUs stays unresolved: it keeps running where it is, but no new GPU is allocated to anyone until it is released. Returns the unresolved ids. """ - from .placement import required_gpu_count - unresolved = [] _, deployments = self.ledger.status() for deployment in deployments: if deployment.state != DeploymentState.LIVE or deployment.assigned_gpus is not None: continue - if required_gpu_count(deployment) == 0: + if self._gpu_units(deployment) == 0: self.ledger.set_allocation(deployment.id, []) continue resident = residency.resident(deployment.id) @@ -891,15 +906,13 @@ def _admit(self, overlay, residency): commit for the candidate's new and revived deployments. Nothing is written here. """ - from .placement import required_gpu_count - adopted: dict[str, list[int]] = {} need: list[str] = [] for gid, deployment in overlay.deployments.items(): fresh = gid in overlay.created or gid in overlay.revived if not fresh: continue - if required_gpu_count(deployment) == 0: + if self._gpu_units(deployment) == 0: continue resident = residency.resident(gid) if ( residency is not None and gid in overlay.revived) else None @@ -980,13 +993,11 @@ def _gpu_holders(self, plan) -> list[str]: return out def _unresolved_allocations(self, *, exclude: AbstractSet[str] = frozenset()) -> list[str]: - from .placement import required_gpu_count - _, deployments = self.ledger.status() return [ g.id for g in deployments if g.state == DeploymentState.LIVE and g.assigned_gpus is None - and g.id not in exclude and required_gpu_count(g) > 0 + and g.id not in exclude and self._gpu_units(g) > 0 ] def set_invocation_catalog(self, catalog) -> None: diff --git a/tests/test_leasing_kubeai.py b/tests/test_leasing_kubeai.py index de85cbc9..19071644 100644 --- a/tests/test_leasing_kubeai.py +++ b/tests/test_leasing_kubeai.py @@ -70,6 +70,11 @@ def __call__(self, args: list[str]) -> str: if doc: self.applied[doc['metadata']['name']] = doc return '' + if verb == 'get' and args[4] == 'pods': + # KubeAI runs one pod per applied Model; here it is up at once. + return json.dumps({'items': [ + _pod(f'model-{name}-0', doc['metadata']['labels']['infer-stack/deployment']) + for name, doc in self.applied.items()]}) if verb == 'get': items = list(self.applied.values()) if '-l' in args: # emulate the label selector @@ -386,19 +391,55 @@ def test_acquire_release_lifecycle(tmp_path): assert rel.reconcile is not None -def test_acquire_missing_profile_rolls_back(tmp_path): - """An unrenderable deployment behaves exactly like a placement failure: - the acquire fails loudly and the lease is rolled back.""" +def test_acquire_missing_profile_commits_nothing(tmp_path): + """An unrenderable deployment fails admission, as an unplaceable one does + on compose: the preview refuses it before any lease is written or any + kubectl apply runs.""" from infer_stack.leasing.backend import PlacementError ctl, be, kubectl = make_controller(tmp_path) with pytest.raises(PlacementError, match='resource profile'): ctl.acquire('alice', [_req('qwen', profile=None)], wait=False) leases, deployments = ctl.ledger.status() - assert [le.state for le in leases] == [LeaseState.RELEASED] + assert leases == [] + assert not any('apply' in call for call in kubectl.calls) assert kubectl.applied == {} +def test_kubeai_takes_the_admission_path(tmp_path): + """One acquire path: KubeAI admits by preview and commits no GPUs.""" + ctl, be, kubectl = make_controller(tmp_path) + assert ctl._admission_mode() + out = ctl.acquire('alice', [_req('qwen')], wait=False) + (deployment,) = out.deployments + assert deployment.assigned_gpus == [] # committed, and empty + assert ctl._unresolved_allocations() == [] + + +def test_status_shows_no_gpu_for_a_cluster_scheduled_deployment(tmp_path): + """The committed allocation is empty, which must not read as "cpu".""" + from infer_stack.cli.commands_leasing import _gpu_label, _placement_view + + ctl, be, kubectl = make_controller(tmp_path) + out = ctl.acquire('alice', [_req('qwen')], wait=False) + observed, assignments = _placement_view(ctl) + (deployment,) = out.deployments + assert _gpu_label(deployment.id, observed, assignments) == '-' + + +def test_queued_acquire_of_an_unrenderable_endpoint_fails_at_once(tmp_path): + """The cluster is the queue, but it cannot queue what it cannot render.""" + from infer_stack.leasing.backend import PlacementError + + ctl, be, kubectl = make_controller(tmp_path) + slept = [] + ctl.sleep = slept.append + with pytest.raises(PlacementError, match='resource profile'): + ctl.acquire('alice', [_req('qwen', profile=None)], wait=False, + wait_for_placement=True, timeout=600) + assert slept == [] + + def test_keep_warm_stays_resident_after_release(tmp_path): ctl, be, kubectl = make_controller(tmp_path) out = ctl.acquire( From d3ee494107746f95c90c2901df546fd8e4b85bcc Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Fri, 25 Sep 2026 21:31:18 -0400 Subject: [PATCH 24/34] Docs: a work queue for backend parity, ending in a UX audit loop docs/queue.md orders what can be done and verified without a GPU or a second machine, with a done-when per item, a rule for adding unforeseen items, and a UX audit that must pass before the queue is finished. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- docs/planning/backend-parity-roadmap.md | 1 + docs/queue.md | 146 ++++++++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 docs/queue.md diff --git a/docs/planning/backend-parity-roadmap.md b/docs/planning/backend-parity-roadmap.md index cc554423..268f8d76 100644 --- a/docs/planning/backend-parity-roadmap.md +++ b/docs/planning/backend-parity-roadmap.md @@ -4,6 +4,7 @@ `dev/backend-unification` · **P1a done** 2026-09-25 · P1b–P5 not started · P6 is ongoing. **Current state:** [../backend-parity.md](../backend-parity.md). +**Execution order:** [../queue.md](../queue.md). **Origin:** the scale-up run needs more than one workstation, and the KubeAI backend had drifted from Compose for three months before the 2026-09-24 audit (`dev/tmp/plan-backend-unification-2026-09-24.md`). diff --git a/docs/queue.md b/docs/queue.md new file mode 100644 index 00000000..08f8fa05 --- /dev/null +++ b/docs/queue.md @@ -0,0 +1,146 @@ +# Work queue: backend parity + +The executable order for [planning/backend-parity-roadmap.md](planning/backend-parity-roadmap.md), +limited to what can be done and verified without a GPU or a second physical +machine: a guest VM running k3s with CPU vLLM. Items that end in a hardware +check produce a script to hand over rather than a claim of done. + +Work top to bottom. An item is done when its **Done when** holds and its +commit is pushed; mark it `[x]` with the date and commit. Update +[backend-parity.md](backend-parity.md) and the roadmap's status in the same +commit as the change they describe. + +## Rules + +- **Unforeseen work gets added here, not done silently.** When an item turns + out to need something the plan did not foresee, add it as a new item in + its place in the order, with a *Why* line naming what was found. A blocker + goes directly above the item it blocks. +- **Duplicate authorities** found on the way: refactor if small, otherwise + add to the roadmap's *Duplicate authorities* table as deferred. Never make + one worse. A blocker is fixed whatever its size. +- **Verified on k3s, not on fakes.** Every item that changes KubeAI + behaviour adds or extends a step in `dev/kubeai_e2e.sh` and passes it. +- **The queue is not done until item 9 passes.** Do not stop at the last + feature item. + +## Items + +### 1. [x] P1a: KubeAI acquires go through admission + +Done 2026-09-25, `d6a19c4`. + +### 2. [ ] P1b: delete the legacy acquire branch + +`MemoryBackend`, `NullBackend` and the test fakes (queue, lock, serialised +publication) get a trivial `residency` and `preview`. Then the +non-admission branches, `_render`'s one-shot `converge(desired)` fallback +and `_admission_mode()` go. + +**Done when:** `_admission_mode` does not exist; the full suite passes; +queue-semantics tests still assert the same behaviour. + +### 3. [ ] P2: day-2 commands and the TUI through the backend + +`instances()` and `stream_logs(target, *, follow, tail)` on both backends; +`ps`, `logs`, `stack up` / `stack down` use them, the raw compose form +moves under `stack compose …`; the TUI's docker pane, log follower and +Up / Down go through the backend; `measure` works on KubeAI if vLLM's +memory line reaches the pod log. + +**Done when:** on k3s, `infer-stack ps` and `infer-stack logs ` work +with the same output shape as on Compose, and the TUI follows a Model's log +and lists its pod (checked in a real terminal, not only in tests). + +### 4. [ ] P3: gateway feature parity + +`routes` and `secrets rotate` resolve the backend's gateway instead of +checking its kind; the KubeAI gateway project honours `ui`, +`reverse_proxy` and `dynamic_routing`. Move the KubeAI gateway's approval +into the admission preview (deferred duplicate). + +**Done when:** `routes list` works on KubeAI; Open WebUI answers in front of +the cluster; an e2e step acquires the same model `--dedicated` twice under +dynamic routing and gets two Models and two routes. + +### 5. [ ] P6: one test surface + +Parametrize the controller's acquire scenarios over the Memory, +fake-Compose and fake-KubeAI backends; `tests/test_parity.py` runs each +*same* row of the parity matrix on both fakes. + +**Done when:** every *same* row has a parity test, and the CI suite runs it. + +### 6. [ ] P4: placement from node labels (verified with faked labels) + +`catalog suggest` on KubeAI proposes `resourceProfiles` from +`nvidia.com/gpu.product` / `.memory` node labels; `min_vram_gib` picks the +smallest fitting profile when an endpoint names none. + +**Done when:** on k3s with hand-set labels for two GPU sizes (CPU-backed +profiles), a catalog with `min_vram_gib` and no `resource_profile` lands on +the right profile; `dev/handover/p4_gpu_labels.sh` exists for one run on a +real GPU node. + +### 7. [ ] P5: in-cluster gateway and a second node + +Render the gateway as a Deployment + Service behind an ingress; `secrets +rotate` becomes a Secret update and a rollout; `doctor` checks the ingress. +Try a second k3s agent in a Docker container on the guest; write the +"add a workstation" runbook into `kubeai-backend.md`. + +**Done when:** on one node, a card reaches a Model through the in-cluster +gateway and `secrets rotate` works. If the simulated second node fits in the +VM, a Model pinned to it by node selector is served through the same +gateway. `dev/handover/p5_two_hosts.sh` exists for one run across two real +machines. + +### 8. [ ] The README's Compose sections + +About 33 references to verbs that no longer exist (`setup`, `up -d`, +`switch`, `describe-profile`, `smoke-test`, `wait-ready`, `diagnose`). + +**Done when:** every command in the README runs as written against the +current CLI, and `grep` finds none of those verbs. + +### 9. [ ] UX audit loop: do not stop without a passing audit + +Polishing the UX can take hours; that is expected, and this item is the +reason the queue exists. Loop: + +1. Audit (below) on **both** backends, from a fresh data root. +2. Write every finding as a sub-item here, with the exact command and + output. +3. Fix them, one commit each, tests added where the behaviour is testable. +4. Repeat from 1. The audit passes when a full pass finds nothing new. + +The audit covers: + +- **First run.** Following the README and `kubeai-backend.md` literally + from nothing reaches a working request on each backend. +- **Every verb** (`acquire`, `release`, `wait`, `evict`, `gc`, `clean`, + `renew`, `run`, `test`, `status`, `leases`, `ps`, `logs`, `routes`, + `doctor`, `config`, `catalog`, `secrets`, `stack`): `--help` is accurate + and its examples run; output has the same shape on both backends. +- **Every failure.** Each error names the cause and the next command to run; + none mentions Docker or Compose on KubeAI, or `kubectl` on Compose; no + traceback reaches the user for an expected condition (port in use, no + cluster, unrenderable endpoint, refused approval). +- **The TUI**, in a real terminal at 80x24 and wide: every pane, tab, key + and button works on both backends; every action logs its CLI equivalent; + nothing is empty without saying why. +- **Consistency.** The same thing has the same name in the CLI, the TUI and + the docs. + +Seed findings, already known: + +- [ ] `r` does not reload catalogs, and edits to a catalog file are not + picked up until restart. +- [ ] `infer-stack logs` should accept a container name, a prefixed name or + an endpoint name. +- [ ] `status` shows STALE during an apply instead of "apply in progress". + +### 10. [ ] Handover + +Summarize for the operator: what was verified here, and the two handover +scripts (items 6 and 7) with what each run proves. From 184289cae8f03ff5a57d4737b1c2a717bc02c460 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Sat, 26 Sep 2026 11:30:21 -0400 Subject: [PATCH 25/34] P1b: one acquire path; the pre-admission branch is gone SimpleAdmission gives a realize/teardown backend (dry-run, tests) the admission surface: residency from observe(), a no-GPU preview, and converge/apply over realize/teardown. Fakes that emulate capacity override plan(); one that emulates a render failure overrides refuse(). With every backend on it, the controller loses _admission_mode(), the legacy acquire, render, renew and publish branches, the one-shot converge(desired) fallback, desired_deployments() and the placement-scope recovery render. Duplicate authorities fixed on the way: - unknown residency: _admit admitted requests needing no new GPU, whose render then failed after commit; now nothing is admitted. - the desired set: routes prune used desired_deployments() instead of the admission view the render uses. Rollback after a commit keeps its tests, on a fake whose render refuses what its preview admitted. Verified: full suite, compose e2e (non-GPU tiers) and the kubeai e2e on k3s. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 10 + docs/planning/backend-parity-roadmap.md | 25 +- docs/queue.md | 4 +- infer_stack/cli/commands_leasing.py | 38 +- infer_stack/leasing/backend.py | 116 ++++- infer_stack/leasing/compose.py | 22 - infer_stack/leasing/controller.py | 457 +++++-------------- infer_stack/leasing/models.py | 4 +- tests/test_cli_leasing.py | 2 +- tests/test_leasing_admission.py | 11 +- tests/test_leasing_compose.py | 13 +- tests/test_leasing_controller_lock.py | 9 +- tests/test_leasing_controller_queue.py | 39 +- tests/test_leasing_kubeai.py | 1 - tests/test_leasing_profile.py | 19 +- tests/test_leasing_serialised_publication.py | 103 +++-- 16 files changed, 385 insertions(+), 488 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f7acb9fd..f27c8d3c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### One acquire path + +The controller's pre-admission branch is gone: every backend, including the +dry-run backend, acquires, renders, renews and publishes through admission. +A backend with only `realize` / `teardown` gets the admission surface from +`SimpleAdmission`. Two behaviours changed with it. While residency cannot +be read, no acquire is admitted; before, a request needing no new GPU was +admitted and then failed at its render, after the lease was committed. And +`routes prune` computes the desired set with the same view the render uses. + ### KubeAI acquires go through admission, as Compose's do An acquire on the KubeAI backend is now previewed in memory and commits its diff --git a/docs/planning/backend-parity-roadmap.md b/docs/planning/backend-parity-roadmap.md index 268f8d76..f1aa3530 100644 --- a/docs/planning/backend-parity-roadmap.md +++ b/docs/planning/backend-parity-roadmap.md @@ -1,10 +1,9 @@ # Backend parity roadmap: KubeAI as a superset of Compose **Status:** proposed 2026-09-25 · **P0 done** 2026-09-24 on -`dev/backend-unification` · **P1a done** 2026-09-25 · P1b–P5 not started · -P6 is ongoing. +`dev/backend-unification` · **P1 done** 2026-09-26 · P2–P5 not started · +P6 is ongoing. Execution order: [../queue.md](../queue.md). **Current state:** [../backend-parity.md](../backend-parity.md). -**Execution order:** [../queue.md](../queue.md). **Origin:** the scale-up run needs more than one workstation, and the KubeAI backend had drifted from Compose for three months before the 2026-09-24 audit (`dev/tmp/plan-backend-unification-2026-09-24.md`). @@ -91,11 +90,18 @@ gained the step "an unrenderable endpoint is refused before anything is written" (a `--queue` acquire, no lease, no Model), and still passes the make-room step, which now runs on the admission path. -**P1b. Delete the legacy branch** (with P6). +**P1b. Delete the legacy branch.** `MemoryBackend` / `NullBackend` and the test fakes get a trivial `residency` and `preview`; then the non-admission branches and -`_admission_mode()` go. Until then the legacy path serves only test fakes -and `realize`/`teardown` backends, and no new code may depend on it. +`_admission_mode()` go. + +**P1b done 2026-09-26.** `SimpleAdmission` (in `leasing/backend.py`) gives +a backend with only `realize` / `teardown` the admission surface; a fake +that emulates capacity overrides its `plan`, one that emulates a render +failure its `refuse`. The controller has one acquire, render, renew and +publish path. Rollback after a commit is still reachable (the runtime can +change between preview and render) and keeps its tests, on a fake whose +render refuses what its preview admitted. **Exit:** P1a: `_admission_mode()` is true for both real backends; a render failure on KubeAI rolls the lease back before any `kubectl`; e2e passes. @@ -193,8 +199,11 @@ worse. A blocker is fixed whatever its size. | "this backend allocates GPUs" | inferred from which methods a backend has | **fixed** (P1a): `allocates_gpus()` in `leasing/backend.py`, read by the controller and the CLI | | approval digest and pre-approval | a copy inside `ComposeBackend` | **fixed** (P1a): `ConvergeScaffold`, shared | | KubeAI render vs its plan | `converge` rendered inline | **fixed** (P1a): one `_render_documents` behind `converge`, `preview` and `plan_on_idle_host` | -| the acquire path | admission and a legacy branch in five places | deferred to P1b | -| `_render`'s one-shot `converge(desired)` fallback, which applies | a second render contract for old backends | deferred to P1b: it goes with the legacy branch | +| the acquire path | admission and a legacy branch in five places | **fixed** (P1b) | +| `_render`'s one-shot `converge(desired)` fallback, which applies | a second render contract for old backends | **fixed** (P1b): removed | +| "can anything be admitted while residency is unknown" | `_admit` admitted requests needing no new GPU; the render after the commit then failed without residency | **fixed** (P1b): nothing is admitted, and nothing is committed | +| the desired set | `desired_deployments()` beside the admission view; `routes prune` used the former | **fixed** (P1b): one view, `_admission_view` | +| a crashed acquire's placement scope | recorded in the marker and re-applied by a recovery render, although admission never records one | **fixed** (P1b): a stale scope is dropped | | the KubeAI gateway's approval | the gateway project asks its own diff approval at render, after the lease commits, not in the admission preview | deferred to P3. Same result under `--yes`; interactively, a declined gateway change rolls the lease back after the commit | ## Not in scope diff --git a/docs/queue.md b/docs/queue.md index 08f8fa05..84ebcaed 100644 --- a/docs/queue.md +++ b/docs/queue.md @@ -30,7 +30,9 @@ commit as the change they describe. Done 2026-09-25, `d6a19c4`. -### 2. [ ] P1b: delete the legacy acquire branch +### 2. [x] P1b: delete the legacy acquire branch + +Done 2026-09-26. `MemoryBackend`, `NullBackend` and the test fakes (queue, lock, serialised publication) get a trivial `residency` and `preview`. Then the diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index cce3120d..71507813 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -1906,28 +1906,19 @@ def _placement_view(controller): if not allocates_gpus(backend): return observed, assignments # the cluster places; no GPU indices - if controller._admission_mode(): - # Committed allocations, and idle residents' physical GPUs; the - # legacy planner view would show placements admission would not make. - _, deployments = controller.ledger.status(virtual_expiry=True) + # Committed allocations, and idle residents' physical GPUs. + _, deployments = controller.ledger.status(virtual_expiry=True) + for g in deployments: + if g.assigned_gpus is not None: + assignments[g.id] = list(g.assigned_gpus) + try: + residency = backend.residency() for g in deployments: - if g.assigned_gpus is not None: - assignments[g.id] = list(g.assigned_gpus) - try: - residency = backend.residency() - for g in deployments: - c = residency.resident(g.id) - if g.id not in assignments and c is not None: - assignments[g.id] = list(c.gpus) - except Exception: # noqa: BLE001 - pass - return observed, assignments - plan = getattr(backend, 'plan', None) - if plan is not None: - try: - assignments = dict(plan(controller.desired_deployments()).assignments) - except Exception: # noqa: BLE001 - pass + c = residency.resident(g.id) + if g.id not in assignments and c is not None: + assignments[g.id] = list(c.gpus) + except Exception: # noqa: BLE001 + pass return observed, assignments @@ -2602,8 +2593,9 @@ def main(cls, argv=True, **kwargs): backend = _require_compose_backend(controller) def prune_plan() -> tuple[dict, dict, list[str]]: - desired = controller.desired_deployments() - plan = backend.plan(desired) + # The desired set exactly as the next render sees it. + desired, inputs = controller._admission_view(backend.residency()) + plan = backend.plan(desired, inputs) keep: dict = {} if backend.catalog is not None: keep.update(_registry_incoming_from_catalog(backend.catalog)) diff --git a/infer_stack/leasing/backend.py b/infer_stack/leasing/backend.py index 28356ddc..9755130c 100644 --- a/infer_stack/leasing/backend.py +++ b/infer_stack/leasing/backend.py @@ -141,15 +141,16 @@ class ConvergeBackend(Backend, Protocol): * ``last_assignments`` — deployment id -> GPU indices (empty for backends where the cluster schedules). - Backends without this surface fall back to the per-deployment - ``realize``/``teardown`` path in :meth:`Controller._render`. + A backend with only ``realize``/``teardown`` gets this surface from + :class:`SimpleAdmission`. """ last_unplaced: set[str] last_errors: list[str] last_assignments: dict[str, list[int]] - def converge(self, desired: list[Deployment], *, apply: bool = True): + def converge(self, desired: list[Deployment], *, apply: bool = True, + placement: Any = None): """Render the desired set to backend state; optionally apply it.""" ... @@ -165,10 +166,11 @@ def apply(self) -> bool | None: @runtime_checkable class AdmissionBackend(ConvergeBackend, Protocol): - """The surface admission mode uses (see ``Controller._admission_mode``). + """The surface the controller drives: every acquire goes through admission. Both real backends have it: strict residency and an in-memory - ``preview``. Compose also takes the stable-address network and the + ``preview``; :class:`SimpleAdmission` supplies it for backends that + neither place nor inspect (the dry-run and test backends). Compose also takes the stable-address network and the container-adoption table from the controller (``network``, ``on_addresses``, ``adopted``); those are Compose-only, and the controller hands them over only to a backend that declares them. The @@ -331,7 +333,107 @@ def _approve_changes(self, planned: dict) -> None: ) -class MemoryBackend: +@dataclass +class RenderPreview: + """The render half of a :meth:`SimpleAdmission.preview`: what it refused.""" + + unrenderable: set[str] + errors: list[str] + + +class SimpleAdmission: + """The admission surface for a backend that neither places nor inspects. + + For in-process backends (the dry-run and test backends): they allocate no + GPUs, their :meth:`residency` is what :meth:`observe` reports (each + deployment one warm instance), and converge/apply is ``realize`` / + ``teardown`` over the rendered set. A subclass that emulates capacity + overrides :meth:`plan`; one that emulates render failures overrides + :meth:`refuse`. Everything else goes through the one admission path the + real backends use. + """ + + allocates_gpus = False + last_preview_digest: str | None = None + + def plan(self, desired: list[Deployment], placement: Any = None): + """Which of ``desired`` fit; here, all of them, on no GPU.""" + from .placement import GpuPlan + + return GpuPlan(assignments={g.id: [] for g in desired}) + + def refuse(self, desired: list[Deployment]) -> dict[str, str]: + """``{deployment id: reason}`` for what cannot be rendered; none here.""" + return {} + + def residency(self): + from .residency import Container, Residency + + return Residency(by_deployment={ + gid: (Container(container_id=gid, deployment_id=gid, + state='running', labelled=True),) + for gid in sorted(self.observe()) + }) + + def _preview(self, desired: list[Deployment], placement: Any = None): + desired = list(desired) + plan = self.plan(desired, placement) + refused = self.refuse(desired) + return plan, RenderPreview(set(refused), [f'{g}: {why}' for g, why in refused.items()]) + + def preview(self, desired: list[Deployment], placement: Any = None, *, + approve: bool = False): + plan, rendered = self._preview(desired, placement) + self.last_preview_digest = repr(sorted( + (gid, list(gpus)) for gid, gpus in plan.assignments.items() + if gid not in rendered.unrenderable)) + return plan, rendered + + def converge(self, desired: list[Deployment], *, apply: bool = True, + placement: Any = None): + """Record the renderable, placed part of ``desired``; ``apply`` realizes it.""" + desired = list(desired) + plan, rendered = self._preview(desired, placement) + self.last_errors = list(plan.errors) + list(rendered.errors) + self.last_unplaced = { + g.id for g in desired + if g.id not in plan.assignments or g.id in rendered.unrenderable} + self.last_assignments = (dict(plan.assignments) + if self.allocates_gpus else {}) + self._rendered = {g.id: g for g in desired if g.id not in self.last_unplaced} + known = getattr(self, '_known', {}) + known.update({g.id: g for g in desired}) + self._known = known + if apply: + self.apply() + + def apply(self) -> None: + """Realize what the last converge rendered; tear down the rest.""" + rendered = getattr(self, '_rendered', {}) + for gid in sorted(self.observe() - set(rendered)): + deployment = getattr(self, '_known', {}).get(gid) + if deployment is not None: + self.teardown(deployment) + for gid, deployment in rendered.items(): + if gid not in self.observe(): + self.realize(deployment) + + # Set by converge (the controller reads them with a default before then). + last_unplaced: set[str] + last_errors: list[str] + last_assignments: dict[str, list[int]] + + def observe(self) -> set[str]: # pragma: no cover - provided by the backend + raise NotImplementedError + + def realize(self, deployment: Deployment) -> None: # pragma: no cover + raise NotImplementedError + + def teardown(self, deployment: Deployment) -> None: # pragma: no cover + raise NotImplementedError + + +class MemoryBackend(SimpleAdmission): """In-memory backend that records calls and has configurable readiness. Not a real serving backend — it never starts a process. It exists so the @@ -395,7 +497,7 @@ def set_ready( self.ready_overrides[(deployment_id, endpoint)] = ready -class NullBackend: +class NullBackend(SimpleAdmission): """A no-op backend that serves nothing — for ``--dry-run`` and ``leases``. It never starts a process. ``observe`` returns the empty set, so the diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index ff209d2c..1ea628bd 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -2141,28 +2141,6 @@ def use_profile(self, profile: dict[str, Any]) -> None: sources = profile.get('catalogs') or [] self.catalog = CatalogUnion.from_sources(sources) if sources else None - def placement_context(self) -> dict[str, Any] | None: - """This caller's admission scope, stored with a pending acquire.""" - if self.allowed_gpus is None: - return None - return {'allowed_gpus': list(self.allowed_gpus)} - - def placement_scope(self, context: dict[str, Any] | None): - """Temporarily render with another caller's admission scope.""" - import contextlib - - @contextlib.contextmanager - def scope(): - saved = self.allowed_gpus - if context and 'allowed_gpus' in context: - self.allowed_gpus = context['allowed_gpus'] - try: - yield - finally: - self.allowed_gpus = saved - - return scope() - def validate_requests(self, requests) -> None: """Refuse requests the published catalog union does not define identically.""" from .profile import validate_requests_against diff --git a/infer_stack/leasing/controller.py b/infer_stack/leasing/controller.py index 7e0c29f6..8d5c3a36 100644 --- a/infer_stack/leasing/controller.py +++ b/infer_stack/leasing/controller.py @@ -419,19 +419,6 @@ def _global_lock(self): # -- reconcile --------------------------------------------------------- - def desired_deployments(self) -> list[Deployment]: - """Deployments that should currently be running.""" - _, deployments = self.ledger.status() - desired: list[Deployment] = [] - for deployment in deployments: - if deployment.state == DeploymentState.LIVE: - desired.append(deployment) - elif deployment.state == DeploymentState.IDLE: - policy = deployment.spec.get('reclaim', self.reclaim_default) - if policy == KEEP_WARM: - desired.append(deployment) - return desired - def _infeasible_alone(self, deployments, requested: set) -> dict: """Which of this lease's deployments cannot be placed even on an idle host. @@ -469,77 +456,36 @@ def _infeasible_alone(self, deployments, requested: set) -> dict: def _render(self) -> ReconcileResult: """Render the desired state to disk WITHOUT the slow apply. - The caller MUST hold :meth:`_global_lock`. For a converge-style backend - (Compose) this writes the compose project (placement + files) but does - not ``docker compose up`` -- that is :meth:`_apply_pending`, under the - same lock hold. A per-deployment ``realize``/``teardown`` backend has no - such split, so it realizes here; those backends expose no ``apply``. + The caller MUST hold :meth:`_global_lock`. This writes the backend's + rendered state (the compose project, the Model manifests) but does not + bring it up -- that is :meth:`_apply_pending`, under the same lock hold. """ self.ledger.sweep() - placement = None - if self._admission_mode(): - # Residency decides which idle keep-warm deployments are candidates - # at all; unknown residency fails the render (the change stays - # pending) rather than guessing which warm models exist. - residency = self._admitting.residency() - self._backfill_allocations(residency) - self._prepare_network() - desired, placement = self._admission_view(residency) - if hasattr(self.backend, 'adopted'): - self._admitting.adopted = self._prune_adopted(residency) - else: - desired = self.desired_deployments() - # Bound once rather than probed with hasattr: the capability check is - # the same, and the bound method keeps its type instead of narrowing - # to `object` the way an attribute reached through hasattr does. - converge = getattr(self.backend, 'converge', None) - if converge is not None: - before = set(self.backend.observe()) - try: - if placement is not None: - converge(desired, apply=False, placement=placement) - else: - converge(desired, apply=False) - except TypeError: - # Legacy converge(desired) with no apply kwarg renders+applies - # in one shot (no separate apply()); accept that here. - converge(desired) - after = set(self.backend.observe()) - rec = ReconcileResult( - realized=sorted(after - before), - torn_down=sorted(before - after), - unplaced=sorted( - getattr(self.backend, 'last_unplaced', ()) or () - ), - placement_errors=list( - getattr(self.backend, 'last_errors', ()) or () - ), - assignments=dict( - getattr(self.backend, 'last_assignments', {}) or {} - ), - applied=False, - displaced=list(getattr(self.backend, 'last_displaced', ()) or ()), - degraded=list(getattr(self.backend, 'last_degraded', ()) or ()), - ) - if placement is not None and hasattr(self.backend, 'adopted'): - self._adopt_existing(residency) - return rec - desired_ids = {g.id for g in desired} - actual = self.backend.observe() - result = ReconcileResult() - for deployment in desired: - if deployment.id not in actual: - self.backend.realize(deployment) - result.realized.append(deployment.id) - stale = actual - desired_ids - if stale: - by_id = {g.id: g for g in self.ledger.status()[1]} - for gid in stale: - deployment = by_id.get(gid) - if deployment is not None: - self.backend.teardown(deployment) - result.torn_down.append(gid) - return result + # Residency decides which idle keep-warm deployments are candidates at + # all; unknown residency fails the render (the change stays pending) + # rather than guessing which warm models exist. + residency = self._admitting.residency() + self._backfill_allocations(residency) + self._prepare_network() + desired, placement = self._admission_view(residency) + if hasattr(self.backend, 'adopted'): + self._admitting.adopted = self._prune_adopted(residency) + before = set(self.backend.observe()) + self._admitting.converge(desired, apply=False, placement=placement) + after = set(self.backend.observe()) + rec = ReconcileResult( + realized=sorted(after - before), + torn_down=sorted(before - after), + unplaced=sorted(getattr(self.backend, 'last_unplaced', ()) or ()), + placement_errors=list(getattr(self.backend, 'last_errors', ()) or ()), + assignments=dict(getattr(self.backend, 'last_assignments', {}) or {}), + applied=False, + displaced=list(getattr(self.backend, 'last_displaced', ()) or ()), + degraded=list(getattr(self.backend, 'last_degraded', ()) or ()), + ) + if hasattr(self.backend, 'adopted'): + self._adopt_existing(residency) + return rec # -- serialised publication -------------------------------------------- # @@ -555,8 +501,7 @@ def _render(self) -> ReconcileResult: # -- admission (plan steps P5, P6, P9) ------------------------------------ # - # Backends with strict residency and an in-memory preview (Compose and - # KubeAI) get admission semantics: + # Every acquire, renew and publication goes through admission: # * a LIVE deployment holds a committed allocation (assigned_gpus); # * an IDLE keep-warm deployment is only an optional candidate, and only # while it is uniquely resident; it yields its GPUs to demand and is @@ -566,8 +511,8 @@ def _render(self) -> ReconcileResult: # Where the cluster schedules (KubeAI, ``allocates_gpus`` false) every # deployment commits an empty allocation: admission then decides only # renderability, and a Pending pod is a wait reason, not an unplaced - # error. Test fakes without residency/preview keep the previous path - # (roadmap P1b removes it). + # error. Backends that neither place nor inspect (dry-run, tests) get the + # same surface from SimpleAdmission. def _stored_state(self, lease_id: str): """A lease's state as stored (not virtually expired), or ``None``.""" @@ -578,17 +523,11 @@ def _stored_state(self, lease_id: str): def _admitting(self) -> AdmissionBackend: """``self.backend`` as the admission surface. - Only for code that runs after :meth:`_admission_mode` (or an - equivalent capability check) said the backend has it. + Every backend the controller drives has it (the real ones, and + :class:`~infer_stack.leasing.backend.SimpleAdmission` for the rest). """ return cast(AdmissionBackend, self.backend) - def _admission_mode(self) -> bool: - return all( - callable(getattr(self.backend, name, None)) - for name in ('residency', 'preview', 'converge') - ) - def _admission_view( self, residency, *, overlay=None, virtual_expiry: bool = False ): @@ -753,11 +692,10 @@ def observe_state(self) -> dict: except Exception: # noqa: BLE001 - a view must always render sidecar = {} residency, residency_error = None, None - if callable(getattr(self.backend, 'residency', None)): - try: - residency = self._admitting.residency() - except ResidencyUnknown as ex: - residency_error = str(ex) + try: + residency = self._admitting.residency() + except ResidencyUnknown as ex: + residency_error = str(ex) degraded = set(sidecar.get('degraded') or ()) displaced = set(sidecar.get('displaced') or ()) rows = [] @@ -773,7 +711,7 @@ def observe_state(self) -> dict: elif g.id in displaced and g.state == DeploymentState.IDLE: condition = 'displaced' elif (g.state == DeploymentState.LIVE and g.assigned_gpus is None - and self._admission_mode() and residency is not None + and residency is not None and residency.resident(g.id) is None): condition = 'unresolved' elif residency is not None: @@ -818,7 +756,7 @@ def _prune_adopted(self, residency) -> dict: def _adopt_existing(self, residency) -> None: """One-time migration: adopt project containers from before ownership labels. - Runs after the first admission-mode render on a ledger. A container + Runs after the first render on a ledger. A container without infer-stack's service and fingerprint labels is adopted, with the fingerprint of this render, if it is infrastructure the render still has, or a deployment container whose deployment is LIVE on the @@ -920,10 +858,13 @@ def _admit(self, overlay, residency): adopted[gid] = list(resident.gpus) # IDLE->LIVE keeps its GPUs else: need.append(gid) - if residency is None and (need or adopted): + if residency is None: + # The render after the commit needs residency too, so nothing can + # be admitted without it; refusing here commits nothing. return {}, [ - 'Docker residency is unknown, so only requests that need no new ' - 'GPU are admitted; retry when `docker ps` works' + 'what is running cannot be read right now (the runtime did not ' + 'answer), so nothing is admitted; retry when `infer-stack leases` ' + 'shows no residency error' ] if need: unresolved = self._unresolved_allocations(exclude=set(overlay.deployments)) @@ -1205,8 +1146,7 @@ def _sync_profile(self, *, create: bool) -> None: self._applied_profile = stored def _mark_pending( - self, *, apply: bool, placement_context: dict | None = None, - create_profile: bool = True, + self, *, apply: bool, create_profile: bool = True, ) -> dict: """Record that desired state is about to change (caller holds the lock). @@ -1215,31 +1155,16 @@ def _mark_pending( ever turns ``apply_requested`` on: a staged change never cancels an apply already requested (promotion, see the plan's D23). - If an earlier acquire died between committing and its first render, its - placement scope is still in the marker: render once with that scope - first, so its deployment is placed where that caller was allowed. + A placement scope left in the marker by pre-admission code is dropped: + a row that code committed without an allocation stays unresolved and is + never placed, so no render needs that caller's scope. """ if create_profile: self._sync_profile(create=True) current = self.ledger.publication_pending() if current and current.get('placement_context'): - self._render_in_scope(current['placement_context']) - return self.ledger.mark_publication_pending( - apply_requested=apply, placement_context=placement_context, - ) - - def _render_in_scope(self, context: dict) -> None: - """Render with another caller's admission scope, then forget the scope. - - The scope is cleared only after a render that succeeded (and so pinned - the placement). A declined or failed render propagates and keeps it, - so no later caller can place that deployment within its own scope. - """ - scope = getattr(self.backend, 'placement_scope', None) - if scope is not None: - with scope(context): - self._render() - self.ledger.clear_placement_context() + self.ledger.clear_placement_context() + return self.ledger.mark_publication_pending(apply_requested=apply) def _apply_pending(self, rec: ReconcileResult) -> ReconcileResult: """Apply the last render if the marker requests it; clear on success. @@ -1254,12 +1179,7 @@ def _apply_pending(self, rec: ReconcileResult) -> ReconcileResult: marker = self.ledger.publication_pending() if marker is None: return rec - apply_fn = getattr(self.backend, 'apply', None) - if apply_fn is None: - # realize/teardown backends applied during the render itself. - self.ledger.clear_publication_pending(marker['version']) - rec.publication_pending = False - return rec + apply_fn = self._admitting.apply if not marker['apply_requested']: rec.publication_pending = True # staged; never applied here return rec @@ -1467,19 +1387,14 @@ def _make_room(self) -> str | None: def _never_ran(self, deployment_ids: list[str]) -> list[str]: """Which of these deployments definitely have no container at all. - Uses strict residency where the backend has it: a deployment with any - container, in any state, is kept, and if Docker cannot be read nothing - is reported (so nothing warm is ever evicted on a failed look). Backends - without residency fall back to ``observe()``. + Uses strict residency: a deployment with any container, in any state, + is kept, and if the runtime cannot be read nothing is reported (so + nothing warm is ever evicted on a failed look). """ - residency = getattr(self.backend, 'residency', None) - if residency is None: - running = set(self.backend.observe()) - return [gid for gid in deployment_ids if gid not in running] from .residency import ResidencyUnknown try: - snap = residency() + snap = self._admitting.residency() except ResidencyUnknown: return [] return [gid for gid in deployment_ids if not snap.containers(gid)] @@ -1593,106 +1508,12 @@ def acquire( grabs. Head-of-line GPU reservation is a follow-up; for the small-fleet case (few GPUs, rare multi-GPU jobs) plain queueing is sufficient. """ - from .backend import ConvergeAborted, PlacementError - - if self._admission_mode(): - result, rec = self._acquire_by_admission( - owner, requests, ttl_seconds=ttl_seconds, apply=apply, - wait_for_placement=wait_for_placement, - placement_timeout=timeout if placement_timeout is None else placement_timeout, - placement_interval=interval if placement_interval is None else placement_interval, - ) - return self._finish_acquire(result, rec, apply=apply, wait=wait, - timeout=timeout, interval=interval) - - # Intent, ledger write, render and (once placed) apply all under one lock - # hold, so a second caller blocks before touching sqlite and no render - # can change the files this apply reads. The readiness wait and the - # admission-queue sleep stay OUTSIDE the lock. - with self._global_lock(): - self._sync_profile(create=True) - stored_profile = self.ledger.profile() - candidate_profile = self._acquire_profile_candidate() - if candidate_profile is not None: - self._use_profile_candidate(candidate_profile) - try: - validate = getattr(self.backend, 'validate_requests', None) - if validate is not None: - validate(requests) # before desired state is written - except BaseException: - if candidate_profile is not None: - self._restore_stored_profile(stored_profile) - raise - if candidate_profile is not None: - # The user catalog is authoritative. Persist the compatible - # recovery inputs now; the desired-state marker belongs to the - # acquire immediately below, not to this snapshot refresh. - self._commit_profile_candidate(candidate_profile) - context = getattr(self.backend, 'placement_context', lambda: None)() - self._mark_pending(apply=apply, placement_context=context) - result = self.ledger.acquire( - owner, requests, ttl_seconds=ttl_seconds - ) - try: - try: - rec = self._render() - finally: - # Rendered (placement pinned) or about to roll back: either - # way a recovery no longer needs this caller's scope. - if context is not None: - self.ledger.clear_placement_context() - except ConvergeAborted: - # The operator declined the compose changes -- don't leave the - # just-created lease dangling in the ledger. - self._rollback_acquire(result.lease.id, apply=apply) - raise - # If a deployment this lease just requested could not be placed (e.g. no - # free GPU), either queue for one (wait_for_placement) or -- the default -- - # roll the lease back and report the planner's reason, so the deployment - # never lingers as a phantom ``live`` with nothing behind it. - requested = {g.id for g in result.deployments} - unplaced = requested & set(rec.unplaced) - if not unplaced: - rec = self._apply_admitted(rec, result.lease.id, apply=apply) - if unplaced and wait_for_placement and apply: - # Never queue for capacity that cannot exist. Re-plan this lease's - # deployments ALONE on an idle host: if they do not fit there, no - # amount of waiting will help, and waiting is actively harmful -- - # the lease holds whatever it did place for the whole timeout, so a - # request that was never satisfiable can block ones that are. - # - # Only the aggregate case needs this. A single deployment too large - # for any card is already caught by the planner's permanent branch; - # what is missed is a lease whose deployments cannot fit TOGETHER, - # e.g. a 4-GPU model plus a 1-GPU extractor on a 4-GPU host. - infeasible = self._infeasible_alone(result.deployments, requested) - if infeasible: - self._rollback_acquire(result.lease.id, apply=apply) - raise PlacementError(sorted(infeasible.keys()), - sorted(infeasible.values())) - p_timeout = timeout if placement_timeout is None else placement_timeout - p_interval = ( - interval if placement_interval is None else placement_interval - ) - deadline = self.clock() + p_timeout - while unplaced and self.clock() < deadline: - self.sleep(p_interval) - # Re-render under the lock: each retry sweeps (reclaiming a crashed - # job's TTL-expired lease) and re-plans against the freed GPUs. - with self._global_lock(): - self._mark_pending(apply=True) # the render sweeps - rec = self._render() - unplaced = requested & set(rec.unplaced) - if not unplaced: - rec = self._apply_admitted(rec, result.lease.id, apply=True) - if unplaced: - self._rollback_acquire(result.lease.id, apply=apply) - reasons = [ - e - for e in rec.placement_errors - if any(e.startswith(gid) for gid in unplaced) - ] - raise PlacementError(sorted(unplaced), reasons) + result, rec = self._acquire_by_admission( + owner, requests, ttl_seconds=ttl_seconds, apply=apply, + wait_for_placement=wait_for_placement, + placement_timeout=timeout if placement_timeout is None else placement_timeout, + placement_interval=interval if placement_interval is None else placement_interval, + ) return self._finish_acquire(result, rec, apply=apply, wait=wait, timeout=timeout, interval=interval) @@ -1700,7 +1521,7 @@ def _acquire_by_admission( self, owner, requests, *, ttl_seconds, apply, wait_for_placement, placement_timeout, placement_interval, ): - """Admission-mode acquire: preview in memory, commit only if admissible. + """The acquire: preview in memory, commit only if admissible. Each attempt runs under the lock: sweep (itself a published mutation), observe residency, overlay the request on the ledger, and preview @@ -1750,7 +1571,6 @@ def _acquire_by_admission( self._commit_profile_candidate(candidate_profile) # Allocations are committed with the lease, so no placement # scope needs recording for recovery. - context = None self._mark_pending(apply=apply) try: result = self.ledger.acquire( @@ -1763,11 +1583,7 @@ def _acquire_by_admission( except AdmissionConflict: continue # the ledger moved; preview again try: - try: - rec = self._render() - finally: - if context is not None: - self.ledger.clear_placement_context() + rec = self._render() except ConvergeAborted: self._rollback_acquire(result.lease.id, apply=apply) raise @@ -1946,68 +1762,43 @@ def publish_profile(self, profile: dict) -> ReconcileResult: f'config publish needs a quiescent stack: {len(active)} active ' f'lease(s) ({", ".join(active[:3])}); release them first' ) - residency = getattr(self.backend, 'residency', None) - if residency is not None: - try: - snap = residency() - except ResidencyUnknown as ex: - raise ProfileMismatch( - f'config publish cannot confirm the stack is quiescent: {ex}' - ) from ex - running = sorted({c.deployment_id for c in snap.all_containers() - if c.deployment_id}) - if running: - raise ProfileMismatch( - 'config publish needs a quiescent stack: deployment ' - f'container(s) exist for {", ".join(running[:3])}; ' - '`infer-stack evict --all` first' - ) - if self._admission_mode(): - # A PURE preview first: the real render persists append-only - # state (route registry, addresses), which must not happen for - # a candidate whose publication has not committed. - previous = self._applied_profile - use(profile) - try: - residency = self._admitting.residency() - self._prepare_network() - desired, inputs = self._admission_view( - residency, virtual_expiry=True - ) - self._admitting.preview(desired, inputs, approve=True) - except BaseException: - if previous is not None: - use(previous) - raise - self.ledger.store.publish_profile( - profile, approved_digest=self._admitting.last_preview_digest) - self._profile_error = None - self._applied_profile = profile - self._invocation_profile = profile - self._profile_drift_warned = False - return self._publish() - # Backends without a preview (KubeAI): render, then commit. - # No implicit profile here: on a fresh ledger a declined preview - # must leave neither a profile nor a marker behind. - existed = self.ledger.publication_pending() is not None - marker = self._mark_pending(apply=True, create_profile=False) + try: + snap = self._admitting.residency() + except ResidencyUnknown as ex: + raise ProfileMismatch( + f'config publish cannot confirm the stack is quiescent: {ex}' + ) from ex + running = sorted({c.deployment_id for c in snap.all_containers() + if c.deployment_id}) + if running: + raise ProfileMismatch( + 'config publish needs a quiescent stack: deployment ' + f'container(s) exist for {", ".join(running[:3])}; ' + '`infer-stack evict --all` first' + ) + # A PURE preview first: the real render persists append-only + # state (route registry, addresses), which must not happen for + # a candidate whose publication has not committed. previous = self._applied_profile use(profile) try: - rec = self._render() + residency = self._admitting.residency() + self._prepare_network() + desired, inputs = self._admission_view( + residency, virtual_expiry=True + ) + self._admitting.preview(desired, inputs, approve=True) except BaseException: if previous is not None: use(previous) - if not existed: - self.ledger.clear_publication_pending(marker['version']) raise self.ledger.store.publish_profile( - profile, approved_digest=getattr(self.backend, 'last_planned_digest', None)) + profile, approved_digest=self._admitting.last_preview_digest) self._profile_error = None self._applied_profile = profile self._invocation_profile = profile self._profile_drift_warned = False - return self._apply_pending(rec) + return self._publish() def prune(self) -> tuple[int, int]: """Forget released/expired leases and stopped deployments, under the lock. @@ -2027,7 +1818,7 @@ def renew(self, lease_id: str, *, ttl_seconds: float | None) -> RenewOutcome: nothing. **Slow path, under the lock:** a deployment went IDLE, so the renew is a - desired-state change. In admission mode it is re-admitted: the IDLE + desired-state change, so it is re-admitted: the IDLE deployment adopts its resident GPUs, or is placed fresh; if neither is possible the renew fails with :class:`PlacementError` and writes nothing. The lease is re-validated as ACTIVE under the lock first. @@ -2038,52 +1829,36 @@ def renew(self, lease_id: str, *, ttl_seconds: float | None) -> RenewOutcome: if fast is not False: return RenewOutcome(fast, []) with self._global_lock(): - if self._admission_mode(): - lease = self.ledger.get_lease(lease_id) - if lease is None or lease.state != LeaseState.ACTIVE: - return RenewOutcome(None, []) - from .residency import ResidencyUnknown - - try: - residency = self._admitting.residency() - except ResidencyUnknown: - residency = None - overlay = self.ledger.plan_acquire([]) - for gid in dict.fromkeys(lease.deployment_ids): - deployment = self.ledger.get_deployment(gid) - if deployment is not None and deployment.state == DeploymentState.IDLE: - deployment.state = DeploymentState.LIVE - overlay.deployments[gid] = deployment - overlay.revived.append(gid) - if not overlay.revived: - return RenewOutcome( - self.ledger.renew(lease_id, ttl_seconds=ttl_seconds), []) - allocations, reasons = self._admit(overlay, residency) - if reasons: - raise PlacementError(list(overlay.revived), reasons) - self._mark_pending(apply=True) - if self._admission_digest: - self.ledger.mark_publication_pending( - apply_requested=True, approved_digest=self._admission_digest) - renewed = self.ledger.renew( - lease_id, ttl_seconds=ttl_seconds, allocations=allocations) - rec = self._publish() - return RenewOutcome(renewed, list(overlay.revived), rec) lease = self.ledger.get_lease(lease_id) - reviving = [] - if lease is not None and lease.state == LeaseState.ACTIVE: - for gid in dict.fromkeys(lease.deployment_ids): - deployment = self.ledger.get_deployment(gid) - if deployment is not None and deployment.state == DeploymentState.IDLE: - reviving.append(gid) - if not reviving: + if lease is None or lease.state != LeaseState.ACTIVE: + return RenewOutcome(None, []) + from .residency import ResidencyUnknown + + try: + residency = self._admitting.residency() + except ResidencyUnknown: + residency = None + overlay = self.ledger.plan_acquire([]) + for gid in dict.fromkeys(lease.deployment_ids): + deployment = self.ledger.get_deployment(gid) + if deployment is not None and deployment.state == DeploymentState.IDLE: + deployment.state = DeploymentState.LIVE + overlay.deployments[gid] = deployment + overlay.revived.append(gid) + if not overlay.revived: return RenewOutcome( - self.ledger.renew(lease_id, ttl_seconds=ttl_seconds), [], - ) + self.ledger.renew(lease_id, ttl_seconds=ttl_seconds), []) + allocations, reasons = self._admit(overlay, residency) + if reasons: + raise PlacementError(list(overlay.revived), reasons) self._mark_pending(apply=True) - renewed = self.ledger.renew(lease_id, ttl_seconds=ttl_seconds) + if self._admission_digest: + self.ledger.mark_publication_pending( + apply_requested=True, approved_digest=self._admission_digest) + renewed = self.ledger.renew( + lease_id, ttl_seconds=ttl_seconds, allocations=allocations) rec = self._publish() - return RenewOutcome(renewed, reviving, rec) + return RenewOutcome(renewed, list(overlay.revived), rec) def evict(self, deployment_ids: Iterable[str] | None = None) -> EvictOutcome: """Force-evict idle (released) deployments now, overriding keep-warm. diff --git a/infer_stack/leasing/models.py b/infer_stack/leasing/models.py index ab24d8c9..d7a98022 100644 --- a/infer_stack/leasing/models.py +++ b/infer_stack/leasing/models.py @@ -362,8 +362,8 @@ class Deployment: created_at: float updated_at: float demand: int = 0 - # The committed GPU allocation of a LIVE deployment (admission-mode - # backends). ``None`` when not LIVE, or LIVE but unresolved (a ledger from + # The committed GPU allocation of a LIVE deployment (empty where the + # cluster schedules). ``None`` when not LIVE, or LIVE but unresolved (a ledger from # before allocations existed). Cleared in the same transaction as any # transition out of LIVE. assigned_gpus: list[int] | None = None diff --git a/tests/test_cli_leasing.py b/tests/test_cli_leasing.py index d007f7c9..91bb3e03 100644 --- a/tests/test_cli_leasing.py +++ b/tests/test_cli_leasing.py @@ -985,7 +985,7 @@ def test_apply_exits_nonzero_while_publication_stays_pending(env, capsys, monkey from infer_stack.leasing.backend import MemoryBackend class RoutesNeverVerify(MemoryBackend): - def converge(self, desired, *, apply=True): + def converge(self, desired, *, apply=True, placement=None): self.last_unplaced, self.last_errors, self.last_assignments = [], [], {} def apply(self): diff --git a/tests/test_leasing_admission.py b/tests/test_leasing_admission.py index dbbb552f..2ebd2494 100644 --- a/tests/test_leasing_admission.py +++ b/tests/test_leasing_admission.py @@ -179,7 +179,9 @@ def preview(self, desired, placement=None, **kw): assert len(calls) == 2 and ledger.get_deployment(out.deployments[0].id).assigned_gpus == [0] -def test_unknown_residency_admits_only_resource_neutral_requests(tmp_path): # 35 +def test_unknown_residency_admits_nothing(tmp_path): # 35 + """The render after a commit needs residency, so admission refuses without + it, even for a request that needs no new GPU.""" from infer_stack.leasing.residency import ResidencyUnknown ledger, ctl, _ = make(tmp_path) @@ -189,12 +191,11 @@ def unknown(): raise ResidencyUnknown('docker ps failed') ctl.backend.residency = unknown - with pytest.raises(PlacementError, match='residency is unknown'): + with pytest.raises(PlacementError, match='cannot be read'): ctl.acquire('x', CAT.resolve_names(['two']), wait=False, apply=False) - # Coalescing onto the LIVE, allocated deployment needs no new GPU. Its render - # still needs residency, so stage it without rendering residency-dependent state. overlay = ledger.plan_acquire(CAT.resolve_names(['one'])) - assert ctl._admit(overlay, None) == ({}, []) + allocations, reasons = ctl._admit(overlay, None) + assert allocations == {} and reasons # -- P5: allocations and renew ---------------------------------------------------------- diff --git a/tests/test_leasing_compose.py b/tests/test_leasing_compose.py index 232c7537..25173891 100644 --- a/tests/test_leasing_compose.py +++ b/tests/test_leasing_compose.py @@ -1108,11 +1108,20 @@ def test_acquire_rolls_back_lease_on_decline(tmp_path): from infer_stack.leasing import EndpointRequest, LeaseState, vllm_structural from infer_stack.leasing.backend import ConvergeAborted - class DeclineBackend: + from infer_stack.leasing.backend import SimpleAdmission + + class DeclineBackend(SimpleAdmission): + """The operator declines the diff, which admission asks at preview.""" + def observe(self): return set() - def converge(self, desired): + def preview(self, desired, placement=None, *, approve=False): + if approve: + raise ConvergeAborted('declined') + return super().preview(desired, placement) + + def converge(self, desired, *, apply=True, placement=None): raise ConvergeAborted('declined') led = Ledger(SqliteStore(tmp_path / 'ledger.db')) diff --git a/tests/test_leasing_controller_lock.py b/tests/test_leasing_controller_lock.py index 264be200..74d10cb0 100644 --- a/tests/test_leasing_controller_lock.py +++ b/tests/test_leasing_controller_lock.py @@ -14,6 +14,7 @@ import pytest +from infer_stack.leasing.backend import SimpleAdmission from infer_stack.leasing import ( Controller, EndpointRequest, @@ -32,7 +33,7 @@ THREAD_TIMEOUT_S = 30 -class OverlapBackend: +class OverlapBackend(SimpleAdmission): """A converge-style backend that records max concurrent converges.""" def __init__(self) -> None: @@ -43,7 +44,7 @@ def __init__(self) -> None: self.last_errors: tuple = () self.last_assignments: dict = {} - def converge(self, desired, *, apply: bool = True) -> None: + def converge(self, desired, *, apply: bool = True, placement=None) -> None: with self._guard: self.active += 1 self.max_active = max(self.max_active, self.active) @@ -172,7 +173,7 @@ def test_created_lock_file_and_dir_are_group_writable(tmp_path): assert dmode & stat.S_ISGID, f'lock dir missing setgid: {oct(dmode)}' -class SharedOverlapBackend: +class SharedOverlapBackend(SimpleAdmission): """Like OverlapBackend but records overlap into a shared counter, so two *separate* controllers' converges can be compared.""" @@ -184,7 +185,7 @@ def __init__(self, shared: dict, guard: threading.Lock) -> None: self.last_assignments: dict = {} self._ids: set = set() - def converge(self, desired, *, apply: bool = True) -> None: + def converge(self, desired, *, apply: bool = True, placement=None) -> None: with self.guard: self.shared['active'] += 1 self.shared['max'] = max(self.shared['max'], self.shared['active']) diff --git a/tests/test_leasing_controller_queue.py b/tests/test_leasing_controller_queue.py index 833fb9df..2afa912e 100644 --- a/tests/test_leasing_controller_queue.py +++ b/tests/test_leasing_controller_queue.py @@ -19,7 +19,7 @@ SqliteStore, vllm_structural, ) -from infer_stack.leasing.backend import PlacementError, Readiness +from infer_stack.leasing.backend import PlacementError, Readiness, SimpleAdmission from infer_stack.leasing.placement import GpuPlan @@ -45,39 +45,32 @@ def factory(prefix: str) -> str: return factory -class BudgetBackend: +class BudgetBackend(SimpleAdmission): """Places up to ``budget`` deployments (1 slot each); the rest are unplaced. - A converge-style fake: ``converge`` recomputes the placed/unplaced split from - the desired set each call, so freeing demand (a release) lets a queued - deployment in on the next reconcile. + Capacity is its :meth:`plan`, which admission consults before committing + anything, so freeing demand (a release) lets a queued deployment in on + the next attempt. It allocates GPUs, so admission accounts for them. """ + allocates_gpus = True + def __init__(self, budget: int): self.budget = budget self.realized: dict[str, object] = {} # what is "running" (post-apply) - self._placed: dict[str, object] = {} # what the last render decided to run self.apply_calls = 0 - self.last_unplaced: list[str] = [] - self.last_errors: list[str] = [] - self.last_assignments: dict[str, list[int]] = {} - def converge(self, desired, apply: bool = True) -> None: + def plan(self, desired, placement=None): placed = list(desired)[: self.budget] - unplaced = list(desired)[self.budget :] - # Render always records the placed set; bringing it "up" is apply()'s job - # (the render/apply split). Legacy apply=True still realizes in one shot. - self._placed = {g.id: g for g in placed} - if apply: - self.realized = dict(self._placed) - self.last_assignments = {g.id: [i] for i, g in enumerate(placed)} - self.last_unplaced = [g.id for g in unplaced] - self.last_errors = [f'{g.id}: no free GPU' for g in unplaced] + return GpuPlan( + assignments={g.id: [i] for i, g in enumerate(placed)}, + errors=[f'{g.id}: no free GPU' for g in list(desired)[self.budget:]], + ) def apply(self) -> None: """Bring the last-rendered set 'up' (idempotent).""" self.apply_calls += 1 - self.realized = dict(self._placed) + super().apply() def observe(self) -> set[str]: return set(self.realized) @@ -85,11 +78,11 @@ def observe(self) -> set[str]: def probe_ready(self, deployment, endpoint) -> Readiness: return Readiness(deployment.id in self.realized, 'ok') - def realize(self, deployment) -> None: # unused by converge backends - pass + def realize(self, deployment) -> None: + self.realized[deployment.id] = deployment def teardown(self, deployment) -> None: - pass + self.realized.pop(deployment.id, None) def vreq(endpoint, *, reclaim='stop'): diff --git a/tests/test_leasing_kubeai.py b/tests/test_leasing_kubeai.py index 19071644..b162318e 100644 --- a/tests/test_leasing_kubeai.py +++ b/tests/test_leasing_kubeai.py @@ -409,7 +409,6 @@ def test_acquire_missing_profile_commits_nothing(tmp_path): def test_kubeai_takes_the_admission_path(tmp_path): """One acquire path: KubeAI admits by preview and commits no GPUs.""" ctl, be, kubectl = make_controller(tmp_path) - assert ctl._admission_mode() out = ctl.acquire('alice', [_req('qwen')], wait=False) (deployment,) = out.deployments assert deployment.assigned_gpus == [] # committed, and empty diff --git a/tests/test_leasing_profile.py b/tests/test_leasing_profile.py index 1a1f5f8a..231fa82c 100644 --- a/tests/test_leasing_profile.py +++ b/tests/test_leasing_profile.py @@ -281,25 +281,16 @@ def fail(planned): assert ledger.publication_pending() is None -def test_a_declined_recovery_render_keeps_the_crashed_acquires_scope(tmp_path): - from infer_stack.leasing.backend import ConvergeAborted - +def test_a_stale_scope_from_pre_admission_code_is_dropped(tmp_path): + """No render needs a crashed legacy acquire's scope: its row stays + unresolved (see above), so the next change simply clears it.""" a = Catalog.from_dict(cat('alpha')) ledger, ctl = controller(tmp_path, catalog=a) ctl.gc() ledger.mark_publication_pending(apply_requested=True, placement_context={'allowed_gpus': [3]}) - ledger.acquire('x', a.resolve_names(['alpha'])) - _, ctl2 = controller(tmp_path, catalog=a, allowed_gpus=[0]) - - def decline(planned): - raise ConvergeAborted('no') - - ctl2.backend._approve_changes = decline - with pytest.raises(ConvergeAborted): - ctl2.gc() - assert ledger.publication_pending()['placement_context'] == {'allowed_gpus': [3]} - + ctl.acquire('x', a.resolve_names(['alpha']), wait=False, apply=False) + assert ledger.publication_pending()['placement_context'] is None def test_cli_catalog_edit_then_acquire_needs_no_publish_step(tmp_path, monkeypatch): from infer_stack.cli import commands_leasing as cl diff --git a/tests/test_leasing_serialised_publication.py b/tests/test_leasing_serialised_publication.py index fb3a378a..945b8a4c 100644 --- a/tests/test_leasing_serialised_publication.py +++ b/tests/test_leasing_serialised_publication.py @@ -25,7 +25,7 @@ SqliteStore, vllm_structural, ) -from infer_stack.leasing.backend import Readiness +from infer_stack.leasing.backend import Readiness, SimpleAdmission @@ -46,7 +46,7 @@ def _vreq(endpoint: str) -> EndpointRequest: ) -class SharedStackBackend: +class SharedStackBackend(SimpleAdmission): """Models the one shared compose project across separate controllers. ``converge(apply=False)`` (render) writes the desired union to shared @@ -63,7 +63,7 @@ def __init__(self, shared: dict, guard: threading.Lock, apply_sleep: float = 0.0 self.last_errors: list[str] = [] self.last_assignments: dict[str, list[int]] = {} - def converge(self, desired, *, apply: bool = True) -> None: + def converge(self, desired, *, apply: bool = True, placement=None) -> None: ids = {g.id for g in desired} with self.guard: self.shared['rendered'] = set(ids) @@ -440,20 +440,38 @@ def test_no_apply_rollback_never_applies_over_an_older_pending_apply(tmp_path): class _NoRoomBackend(SharedStackBackend): - """Converge fake that can never place an endpoint named ``big``.""" + """Can never place an endpoint named ``big``; preview and render agree.""" - def converge(self, desired, *, apply: bool = True) -> None: - placeable = [g for g in desired if 'big' not in g.served] - self.last_unplaced = [g.id for g in desired if 'big' in g.served] - super().converge(placeable, apply=apply) + def refuse(self, desired): + return {g.id: 'no room' for g in desired if 'big' in g.served} + def converge(self, desired, *, apply: bool = True, placement=None) -> None: + refused = self.refuse(desired) + self.last_unplaced = list(refused) + super().converge([g for g in desired if g.id not in refused], apply=apply) -def test_placement_rollback_evicts_and_rerenders(tmp_path): - """Regression: a failed placement must fully roll back — lease released, the - never-ran deployments evicted (not left idle-keep-warm, which would pin them - in the desired set), the placed sibling removed from the on-disk render, and - the publication marker cleared.""" - from infer_stack.leasing import DeploymentState, LeaseState + +class _RenderRefusesBackend(_NoRoomBackend): + """Its preview admits ``big``; its render then refuses it. + + The one way an admitted acquire still fails after its lease is committed + (the runtime changed between preview and render), so the rollback tests + use it. + """ + + def refuse(self, desired): + return {} + + def converge(self, desired, *, apply: bool = True, placement=None) -> None: + refused = [g.id for g in desired if 'big' in g.served] + SharedStackBackend.converge( + self, [g for g in desired if g.id not in refused], apply=apply) + self.last_unplaced = refused + + +def test_a_refused_acquire_commits_nothing(tmp_path): + """Admission refuses before anything is written: no lease, no deployment, + nothing rendered, no marker.""" from infer_stack.leasing.backend import PlacementError db = str(tmp_path / 'ledger.db') @@ -461,13 +479,34 @@ def test_placement_rollback_evicts_and_rerenders(tmp_path): ledger = Ledger(SqliteStore(db)) ctl = Controller(ledger, _NoRoomBackend(shared, threading.Lock())) + with pytest.raises(PlacementError, match='no room'): + ctl.acquire('alice', [_vreq('ok'), _vreq('big')], wait=False) + + assert ledger.status() == ([], []) + assert shared['rendered'] == set() + assert shared['realized'] == set() + assert ledger.publication_pending() is None + + +def test_a_render_refusal_after_commit_rolls_back_and_rerenders(tmp_path): + """Regression: a render that refuses what admission admitted must fully + roll back: lease released, the never-ran deployments evicted (not left + idle keep-warm, which would pin them in the desired set), the placed + sibling removed from the render, and the publication marker cleared.""" + from infer_stack.leasing import DeploymentState, LeaseState + from infer_stack.leasing.backend import PlacementError + + db = str(tmp_path / 'ledger.db') + shared = {'rendered': set(), 'realized': set(), 'apply_calls': 0} + ledger = Ledger(SqliteStore(db)) + ctl = Controller(ledger, _RenderRefusesBackend(shared, threading.Lock())) + with pytest.raises(PlacementError): ctl.acquire('alice', [_vreq('ok'), _vreq('big')], wait=False) leases, deployments = ledger.status() assert [le.state for le in leases] == [LeaseState.RELEASED] assert {g.state for g in deployments} == {DeploymentState.STOPPED} - # 'ok' was rendered by the failed acquire; the rollback re-render removed it. assert shared['rendered'] == set() assert shared['realized'] == set() assert ledger.publication_pending() is None @@ -475,8 +514,8 @@ def test_placement_rollback_evicts_and_rerenders(tmp_path): def test_rollback_keeps_coalesced_warm_deployment_resident(tmp_path): """A failed acquire that coalesced onto a pre-existing warm (idle keep-warm) - deployment must roll that deployment back to IDLE — not evict the resident - model someone else may still want warm.""" + deployment must leave it IDLE — not evict the resident model someone else + may still want warm.""" from infer_stack.leasing import DeploymentState from infer_stack.leasing.backend import PlacementError @@ -514,7 +553,7 @@ def __init__(self, *args, **kwargs): self.converge_calls = 0 self.declined = False - def converge(self, desired, *, apply: bool = True) -> None: + def converge(self, desired, *, apply: bool = True, placement=None) -> None: self.converge_calls += 1 if desired and not self.declined: self.declined = True @@ -558,13 +597,13 @@ def _warm_then_failed_acquire(tmp_path, residency): shared = _shared() - class Backend(_NoRoomBackend): + class Backend(_RenderRefusesBackend): pass - Backend.residency = lambda self: residency(self) ledger, ctl = _fresh(str(tmp_path / 'ledger.db'), shared, Backend) warm = ctl.acquire('alice', [_vreq('m')], wait=False) ctl.release(warm.lease.id) + Backend.residency = lambda self: residency(self) shared['realized'] = set() # observe() now says "nothing" (as on a docker error) with pytest.raises(PlacementError): ctl.acquire('bob', [_vreq('m'), _vreq('big')], wait=False) @@ -743,16 +782,13 @@ def test_release_all_releases_every_active_lease_in_one_apply(tmp_path): # -- transitional: rollback under unknown residency keeps a phantom warm candidate -- -def test_transitional_unknown_residency_rollback_keeps_an_idle_candidate(tmp_path): - """Behaviour of backends WITHOUT admission mode (no preview), kept by design. +def test_unknown_residency_admits_nothing(tmp_path): + """Without a strict view of what runs, nothing is admitted or written. - A brand-new acquire fails; rollback cannot read residency, so it refuses to - evict (the safe direction). On such backends the desired set still contains - that IDLE deployment, so a later apply starts it with no lease behind it. - Admission-mode backends (Compose) never commit a failed acquire and never - start idle deployments: see tests/test_leasing_admission.py. + (Before one acquire path, a backend without admission committed the + lease, failed, and left an IDLE candidate a later apply started with no + lease behind it.) """ - from infer_stack.leasing import DeploymentState from infer_stack.leasing.backend import PlacementError from infer_stack.leasing.residency import ResidencyUnknown @@ -763,9 +799,8 @@ def residency(self): raise ResidencyUnknown('docker ps failed') ledger, ctl = _fresh(str(tmp_path / 'ledger.db'), shared, Backend) - with pytest.raises(PlacementError): - ctl.acquire('alice', [_vreq('fresh'), _vreq('big')], wait=False) - fresh = next(g for g in ledger.status()[1] if 'fresh' in g.served) - assert fresh.state == DeploymentState.IDLE # not evicted - ctl.apply_now() - assert fresh.id in shared['realized'] # started without a lease (P9 fixes) + with pytest.raises(PlacementError, match='cannot be read'): + ctl.acquire('alice', [_vreq('fresh')], wait=False) + assert ledger.status() == ([], []) + assert ledger.publication_pending() is None + assert shared['realized'] == set() From 64f1e595d4f71be8303588c6bd08e2e9984b5129 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Sat, 26 Sep 2026 11:30:21 -0400 Subject: [PATCH 26/34] Queue: record the P1b commit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- docs/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/queue.md b/docs/queue.md index 84ebcaed..f78b8b06 100644 --- a/docs/queue.md +++ b/docs/queue.md @@ -32,7 +32,7 @@ Done 2026-09-25, `d6a19c4`. ### 2. [x] P1b: delete the legacy acquire branch -Done 2026-09-26. +Done 2026-09-26, `184289c`. `MemoryBackend`, `NullBackend` and the test fakes (queue, lock, serialised publication) get a trivial `residency` and `preview`. Then the From c2d393b443d2df519c809711f7d02e67c1b5cb89 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Sat, 26 Sep 2026 11:55:41 -0400 Subject: [PATCH 27/34] P2: ps, logs, status and the TUI read the backend, on either backend leasing/instances.py: an Instance per container or pod, built by each backend's instances() from its strict residency, the log commands for each runtime, target resolution (alias, name, id prefix, deployment id), and one LogFollower shared by `logs -f` and the TUI. - ps: one table on both backends, naming what each instance serves. - logs: by alias/name/id; -f follows instances that appear or restart. - status: health from residency (and `starting` before ready). - stack up = apply; stack down = backend.down(); the raw Compose verbs (stack compose, restart, pull, start, stop) act on the backend's Compose project on this host, the gateway's on kubeai. - TUI: the runtime pane lists instances, follows pods, Apply/Down work. Duplicate authorities removed: status's own `docker compose ps` liveness, the TUI's name-hint engine detection (it counted Open WebUI and Postgres), and the day-2 verbs' hard-coded project path. Bugs fixed: the TUI replaced kubeai's kubectl runner with Docker's (no KUBECONFIG, every kubectl call failed), and a kwconf flag swallowed the positional after it on every command (`logs -f qwen`, `acquire --yes qwen`). Verified on k3s (CPU vLLM) and on compose (the simulator catalog): ps, logs, logs -f during a request, status, stack compose, and the TUI in a real terminal (instances, pod log follow, Down then Apply). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 20 + README.md | 6 +- docs/backend-parity.md | 8 +- docs/kubeai-backend.md | 4 + docs/planning/backend-parity-roadmap.md | 16 +- docs/queue.md | 30 +- infer_stack/backends/kubeai.py | 19 + infer_stack/cli/commands_runtime.py | 588 +++++++++++++++--------- infer_stack/cli/options.py | 48 +- infer_stack/leasing/backend.py | 13 + infer_stack/leasing/compose.py | 38 +- infer_stack/leasing/instances.py | 278 +++++++++++ infer_stack/leasing/residency.py | 23 + infer_stack/tui.py | 428 +++++------------ tests/test_cli_meta.py | 10 +- tests/test_day2.py | 204 ++++++++ tests/test_log_filter.py | 101 ++-- tests/test_tui.py | 120 +++-- 18 files changed, 1275 insertions(+), 679 deletions(-) create mode 100644 infer_stack/leasing/instances.py create mode 100644 tests/test_day2.py diff --git a/CHANGELOG.md b/CHANGELOG.md index f27c8d3c..aacb24f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,26 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### `ps`, `logs`, `status` and the TUI read the backend, on either backend + +`infer-stack ps` lists what the backend runs, containers or pods and the +gateway, in one table that says what each serves; `infer-stack logs` takes an +endpoint alias, an instance name, a container id prefix or a deployment id, +and `-f` keeps following instances that start or restart later. Both read the +same strict residency the controller decides with, so they work on KubeAI; +`status` does too, and says `starting` for an instance that is up but not yet +ready. `stack up` is now `apply` and `stack down` stops everything on either +backend; the raw Compose verbs (`stack compose -- …`, `restart`, `pull`, +`start`, `stop`) act on the Compose project on this host, which is the +gateway's on KubeAI. The TUI's runtime pane (formerly "docker") follows pod +logs, lists pods, and its Apply / Down buttons work on KubeAI. + +Fixed on the way: every kubectl call from the TUI failed, because the TUI +replaced the backend's command runner with Docker's, whose environment has no +`KUBECONFIG`; the TUI's "engines" log view included Open WebUI and Postgres; +and a flag before a positional ate it (`logs -f qwen` followed everything, +`acquire --yes qwen` named no endpoint), on every command. + ### One acquire path The controller's pre-admission branch is gone: every backend, including the diff --git a/README.md b/README.md index 6c76e9aa..b7fdfd56 100644 --- a/README.md +++ b/README.md @@ -812,8 +812,10 @@ If a `kubeai` release already exists, reuse its namespace ### Debugging checks `infer-stack acquire` reports pod-level failures itself (`ImagePullBackOff`, -`Unschedulable`, a crash with the engine's error quoted). For anything else, -with `NS` the namespace and `MODEL` the Model's name (`kubectl -n $NS get models`): +`Unschedulable`, a crash with the engine's error quoted). `infer-stack ps` lists +the pods and `infer-stack logs -f ` follows one, as on compose. For +anything else, with `NS` the namespace and `MODEL` the Model's name +(`kubectl -n $NS get models`): ```bash kubectl -n "$NS" describe model "$MODEL" diff --git a/docs/backend-parity.md b/docs/backend-parity.md index f4741e44..711ccd42 100644 --- a/docs/backend-parity.md +++ b/docs/backend-parity.md @@ -87,7 +87,7 @@ a different mechanism. **gap**: missing on one side and on the roadmap. | where a deployment lands | local planner over `nvidia-smi` | the cluster scheduler, via `resource_profile:` | | GPU count from TP × PP × DP | same | same | | `placement.gpu_indices`, `allowed_gpus`, `skip_display_gpus` | yes | n/a: no host indices; node-scoped resource profiles are the equivalent | -| `placement.min_vram_gib`, `infer-stack measure` | yes | warned and ignored; `measure` refused (**gap**, P4) | +| `placement.min_vram_gib`, `infer-stack measure` | yes | **gap** (P4): `min_vram_gib` is warned and ignored; `measure` runs but is unverified on a GPU, and `--record` writes an overlay KubeAI does not read | | GPU allocations recorded in the ledger | yes | none: the cluster owns them | | more than one host | boundary | yes: the reason the backend exists | @@ -118,9 +118,11 @@ a different mechanism. **gap**: missing on one side and on the roadmap. | | Compose | KubeAI | |---|---|---| | `doctor` | nothing to check | four checks | -| `logs`, `ps`, `stack up` / `stack down` | docker compose wrappers | **gap** (P2): they say so and point at `kubectl` | +| `ps`, `logs [-f] `, `status` health | containers | same shape; pods, and the gateway's containers | +| `stack up` (= `apply`), `stack down` | same | same | +| `stack compose …`, `restart` / `pull` / `start` / `stop` | the stack's Compose project | the gateway's Compose project (the engines are pods) | | TUI: leases, deployments, catalog editing, acquire / release / evict, API tab, settings | same | same | -| TUI: engine log follow, the docker pane, the Up / Down buttons | `docker logs`, `docker compose ps` | **gap** (P2): empty, or "nothing rendered yet" | +| TUI: log follow, the Instances tab, Apply / Down | containers | same, over pods | | TUI GPU pane | `nvidia-smi` on this host | this host, not the cluster | | `gc --orphans` (also inside `clean`) | yes | n/a: unlabeled Models are never touched | diff --git a/docs/kubeai-backend.md b/docs/kubeai-backend.md index 3be5b96e..48e418ef 100644 --- a/docs/kubeai-backend.md +++ b/docs/kubeai-backend.md @@ -143,6 +143,10 @@ infer-stack release --env-file lease.env on new setups. Without a GPU, install the chart with `dev/e2e_tests/kubeai-cpu-values.yaml` (real vLLM on CPU; needs AVX-512) and run it with `E2E_RESOURCE_PROFILE=cpu`. Verified on k3s 2026-09-24. +- `infer-stack ps`, `infer-stack logs [-f] `, `status` and the TUI's + runtime pane read the pods (and the gateway's containers), in the same shape + as on compose. `stack compose …` and `stack restart` act on the gateway's + Compose project; the engines are pods, restarted by the kubelet. - The gateway runs on the host running infer-stack, so that host is in every request's path. Dynamic routing (`dynamic_routing`) is compose-only; the KubeAI gateway uses static routes. diff --git a/docs/planning/backend-parity-roadmap.md b/docs/planning/backend-parity-roadmap.md index f1aa3530..4d05d473 100644 --- a/docs/planning/backend-parity-roadmap.md +++ b/docs/planning/backend-parity-roadmap.md @@ -1,7 +1,7 @@ # Backend parity roadmap: KubeAI as a superset of Compose **Status:** proposed 2026-09-25 · **P0 done** 2026-09-24 on -`dev/backend-unification` · **P1 done** 2026-09-26 · P2–P5 not started · +`dev/backend-unification` · **P1, P2 done** 2026-09-26 · P3–P5 not started · P6 is ongoing. Execution order: [../queue.md](../queue.md). **Current state:** [../backend-parity.md](../backend-parity.md). **Origin:** the scale-up run needs more than one workstation, and the @@ -129,6 +129,16 @@ Up / Down, and `measure`. logs ` and `ps` work on both backends with the same output shape. **Size:** medium, mostly plumbing; no controller change. +**P2 done 2026-09-26.** `leasing/instances.py`: an `Instance` per container +or pod, built from residency by each backend's `instances()`, and one +`LogFollower` for the CLI and the TUI. `stack up` is `apply`, `stack down` is +`backend.down()`, and the raw Compose verbs (`stack compose …`) act on +whichever Compose project the backend has on this host, the gateway's on +KubeAI. Verified on k3s and on compose (the simulator catalog), in a real +terminal. `measure` was never refused on KubeAI (its guard was +`deployment_logs`, which KubeAI has), but it reads GPU memory-profiling lines +that CPU vLLM does not print: it moves to P4 with its GPU handover. + ### P3. Gateway feature parity **Closes:** `routes` on KubeAI, `dynamic_routing`, Open WebUI and the reverse @@ -204,6 +214,10 @@ worse. A blocker is fixed whatever its size. | "can anything be admitted while residency is unknown" | `_admit` admitted requests needing no new GPU; the render after the commit then failed without residency | **fixed** (P1b): nothing is admitted, and nothing is committed | | the desired set | `desired_deployments()` beside the admission view; `routes prune` used the former | **fixed** (P1b): one view, `_admission_view` | | a crashed acquire's placement scope | recorded in the marker and re-applied by a recovery render, although admission never records one | **fixed** (P1b): a stale scope is dropped | +| "is it running" for `status` | `docker compose ps` service names beside residency (so KubeAI read `unverified`) | **fixed** (P2): residency | +| engine vs gateway in the TUI | a `litellm` name hint (Open WebUI and Postgres counted as engines) | **fixed** (P2): an instance serves a deployment or it does not | +| where the day-2 verbs find the Compose project | a hard-coded path and project name | **fixed** (P2): the backend's `compose_project()` and `compose_argv()` | +| how the TUI runs a runtime command | it replaced the backend's runner with Docker's, whose allowlisted environment has no `KUBECONFIG`: every kubectl call from the TUI failed | **fixed** (P2): only `docker` commands are wrapped | | the KubeAI gateway's approval | the gateway project asks its own diff approval at render, after the lease commits, not in the admission preview | deferred to P3. Same result under `--yes`; interactively, a declined gateway change rolls the lease back after the commit | ## Not in scope diff --git a/docs/queue.md b/docs/queue.md index f78b8b06..d5e57cb5 100644 --- a/docs/queue.md +++ b/docs/queue.md @@ -42,7 +42,9 @@ and `_admission_mode()` go. **Done when:** `_admission_mode` does not exist; the full suite passes; queue-semantics tests still assert the same behaviour. -### 3. [ ] P2: day-2 commands and the TUI through the backend +### 3. [x] P2: day-2 commands and the TUI through the backend + +Done 2026-09-26. `instances()` and `stream_logs(target, *, follow, tail)` on both backends; `ps`, `logs`, `stack up` / `stack down` use them, the raw compose form @@ -79,10 +81,15 @@ fake-Compose and fake-KubeAI backends; `tests/test_parity.py` runs each `nvidia.com/gpu.product` / `.memory` node labels; `min_vram_gib` picks the smallest fitting profile when an endpoint names none. +`measure` on KubeAI (moved here from P2): it already runs, but CPU vLLM prints +no GPU memory-profiling lines, and `--record` writes Compose's measurements +overlay, which KubeAI does not read. Decide where a cluster measurement is +recorded, and verify `measure` in the GPU handover. + **Done when:** on k3s with hand-set labels for two GPU sizes (CPU-backed profiles), a catalog with `min_vram_gib` and no `resource_profile` lands on the right profile; `dev/handover/p4_gpu_labels.sh` exists for one run on a -real GPU node. +real GPU node, and runs `measure` there. ### 7. [ ] P5: in-cluster gateway and a second node @@ -97,13 +104,17 @@ VM, a Model pinned to it by node selector is served through the same gateway. `dev/handover/p5_two_hosts.sh` exists for one run across two real machines. -### 8. [ ] The README's Compose sections +### 8. [ ] The README's Compose sections, and the other stale docs About 33 references to verbs that no longer exist (`setup`, `up -d`, -`switch`, `describe-profile`, `smoke-test`, `wait-ready`, `diagnose`). +`switch`, `describe-profile`, `smoke-test`, `wait-ready`, `diagnose`), and +top-level `restart` / `stop` / `start` / `pull` (they live under `stack`). +*Why widened (2026-09-26):* the same verbs appear in +`docs/persistent-caches-and-warm-restarts.md` and +`docs/stack-graph-profiles.md`. -**Done when:** every command in the README runs as written against the -current CLI, and `grep` finds none of those verbs. +**Done when:** every command in the README and `docs/` runs as written +against the current CLI, and `grep` finds none of those verbs. ### 9. [ ] UX audit loop: do not stop without a passing audit @@ -138,8 +149,11 @@ Seed findings, already known: - [ ] `r` does not reload catalogs, and edits to a catalog file are not picked up until restart. -- [ ] `infer-stack logs` should accept a container name, a prefixed name or - an endpoint name. +- [x] `infer-stack logs` should accept a container name, a prefixed name or + an endpoint name. Done with P2: an instance name, a container id + prefix, a deployment id or an endpoint alias. +- [x] `infer-stack logs -f qwen` followed every instance: a kwconf flag took + the next word as its value. Fixed for every command (P2). - [ ] `status` shows STALE during an apply instead of "apply in progress". ### 10. [ ] Handover diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index 94f0af2c..d3117e99 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -308,6 +308,13 @@ def __init__( def models_file(self) -> Path: return self.state_dir / MODELS_FILENAME + #: The file a render writes, whatever the backend (``status`` shows it). + rendered_file = models_file + + def compose_project(self): + """The Compose project on this host: the gateway's, or ``None``.""" + return self.gateway + @property def _state_file(self) -> Path: return self.state_dir / STATE_FILENAME @@ -588,6 +595,18 @@ def residency(self): raise ResidencyUnknown(f'kubectl get pods failed: {ex}') from ex return residency_from_pods(raw) + def instances(self): + """The managed Models' pods, then the gateway's containers. + + Raises :class:`~infer_stack.leasing.residency.ResidencyUnknown` when + the cluster cannot be read. + """ + from ..leasing.instances import KUBERNETES, from_residency + + pods = from_residency(self.residency(), runtime=KUBERNETES, + namespace=self.namespace) + return pods + (self.gateway.instances() if self.gateway is not None else []) + def deployment_logs(self, deployment: Deployment, *, tail: int = 400) -> str: """Recent engine logs: each pod's current run, then its previous one. diff --git a/infer_stack/cli/commands_runtime.py b/infer_stack/cli/commands_runtime.py index c37d7336..0063014d 100644 --- a/infer_stack/cli/commands_runtime.py +++ b/infer_stack/cli/commands_runtime.py @@ -18,11 +18,12 @@ from ..log_filter import compact_litellm_tracebacks from ..paths import config_root, data_root, get_setting, settings_path +from .commands_leasing import ApplyCLI from .context import _apply_path_overrides from .options import _PathOverridesMixin # --------------------------------------------------------------------------- -# leasing compose project helpers (the target of the day-2 wrappers) +# the backend behind the day-2 verbs # --------------------------------------------------------------------------- @@ -32,43 +33,61 @@ def _docker_env() -> dict[str, str]: return docker_environment() -def _leasing_compose_file() -> Path: - from ..leasing.compose import COMPOSE_FILENAME +def _day2_backend(config): + """The configured backend, built as the leasing verbs build it.""" + from .commands_leasing import _make_backend - return data_root() / 'leasing' / 'compose' / COMPOSE_FILENAME + _apply_path_overrides(config) + return _make_backend(config) -def _day2_compose_base(config, command: str) -> list[str]: - """``docker compose -p -f `` base for the wrappers. +def _served_by_deployment() -> dict[str, list[str]]: + """Deployment id -> the endpoint aliases it serves (read-only ledger).""" + from ..leasing import Ledger, SqliteStore, default_ledger_path - Targets the leasing Compose deployment (no ``config.yaml`` needed). Raises a - helpful error when nothing has been deployed yet — the file is written by - ``acquire`` / ``apply``. - """ - from ..leasing.compose import LEASING_PROJECT + path = default_ledger_path() + if not path.exists(): + return {} + try: + _, deployments = Ledger(SqliteStore(str(path))).status(virtual_expiry=True) + except Exception: # noqa: BLE001 - a view never fails on the ledger + return {} + return {d.id: sorted(d.served) for d in deployments} - _apply_path_overrides(config) - compose_file = _leasing_compose_file() - if not compose_file.exists(): - hint = '' - if (get_setting('backend') or '') == 'kubeai': - hint = ( - ' Note: the configured backend is kubeai — these verbs manage ' - 'the docker compose stack only; inspect the cluster with ' - '`infer-stack doctor` / `kubectl -n get models`.' - ) + +def _instances(backend) -> list: + """What the backend runs, or a clean exit when it cannot be read.""" + from ..leasing.residency import ResidencyUnknown + + try: + return list(backend.instances()) + except ResidencyUnknown as ex: + raise SystemExit(f'cannot read what is running: {ex}') + + +def _compose_argv(config) -> list[str]: + """``docker compose ...`` for the Compose project on this host. + + That is the stack itself on the compose backend, and the gateway on the + kubeai backend. Exits when there is none, or nothing is rendered yet. + """ + backend = _day2_backend(config) + project = getattr(backend, 'compose_project', lambda: None)() + if project is None: raise SystemExit( - f'nothing deployed yet (no {compose_file}). ' - f'Bring a model up first, e.g. `infer-stack acquire `.' - f'{hint}' - ) - base = ['docker', 'compose'] - # The same managed .env the backend passes, so `stack up` interpolates the - # master key, DB password and HF_TOKEN from it (never the caller's shell). - env_file = compose_file.parent / '.env' - if env_file.exists(): - base += ['--env-file', str(env_file)] - return [*base, '-p', LEASING_PROJECT, '-f', str(compose_file)] + f'the {_backend_name(backend)} backend has no Compose project on this ' + 'host; use `infer-stack ps` and `infer-stack logs`') + if not project.compose_file.exists(): + raise SystemExit( + f'nothing rendered yet (no {project.compose_file}); bring a model up ' + 'first, e.g. `infer-stack acquire `') + return project.compose_argv() + + +def _backend_name(backend) -> str: + name = type(backend).__name__ + return {'ComposeBackend': 'compose', 'KubeaiBackend': 'kubeai', + 'NullBackend': 'null (dry-run)'}.get(name, name) # --------------------------------------------------------------------------- @@ -122,82 +141,65 @@ def _leasing_status() -> dict[str, Any]: for d in deployments ] out['summary'] = (active, len(leases), live, len(deployments)) - out['served'] = _served_models(deployments) + out['live_deployments'] = [d for d in deployments if d.state == DeploymentState.LIVE] return out -def _running_services() -> set[str] | None: - """Compose service names currently running, or None if docker can't say. +def _served_models(deployments, backend=None) -> list[tuple[str, str, str, str]]: + """``(endpoint, model, engine, health)`` rows for what is serving. - A single cheap `docker compose ps`. It exists because the ledger's LIVE is - a *belief*: a converge that fails partway (an unpullable image, a daemon - hiccup) leaves deployments recorded LIVE with no container behind them, and - the only way to notice used to be an acquire that hung on readiness. - Distinguishing "recorded" from "running" is the whole point of showing this. - - None (not an empty set) when docker is unreachable, so the caller can say - "unverified" instead of falsely reporting everything down. - """ - from ..leasing.compose import LEASING_PROJECT - - compose_file = _leasing_compose_file() - if not compose_file.exists(): - return None - try: - proc = subprocess.run( - ['docker', 'compose', '-p', LEASING_PROJECT, '-f', str(compose_file), - 'ps', '--services', '--filter', 'status=running'], - capture_output=True, text=True, timeout=15, env=_docker_env(), - ) - except Exception: # noqa: BLE001 - status must never fail on this - return None - if proc.returncode != 0: - return None - return {ln.strip() for ln in proc.stdout.splitlines() if ln.strip()} - - -def _served_models(deployments) -> list[tuple[str, str, str, str]]: - """``(endpoint, model, gpus_or_engine, health)`` rows for what is serving. - - Reads the ledger for what *should* be up and reconciles against the - containers that actually are, so `status` answers "what can I send a - request to right now" without the user having to cross-read `leases` - against `stack ps`. + Reads the ledger for what *should* be up and the backend's strict + residency for what is, so `status` answers "what can I send a request to + right now" without cross-reading `leases` against `ps`. """ from ..leasing import DeploymentState - from ..leasing.compose import ollama_service_name_for, vllm_service_name_for + from ..leasing.residency import ResidencyUnknown live = [d for d in deployments if d.state == DeploymentState.LIVE] if not live: return [] - running = _running_services() + residency = None + if backend is not None: + try: + residency = backend.residency() + except (ResidencyUnknown, Exception): # noqa: BLE001 - status never fails here + residency = None rows: list[tuple[str, str, str, str]] = [] for d in live: + if residency is None: + health = 'unverified' + elif residency.resident(d.id) is not None: + # Warm; a health check that has not passed yet means still loading. + health = ('starting' if residency.resident(d.id).health == 'starting' + else 'up') + elif residency.containers(d.id): + # There, but not warm: starting, crashed, or ambiguous. + health = residency.containers(d.id)[0].state + else: + # The ledger says live and nothing exists. Almost always an apply + # that failed after the record was written. + health = 'STALE' for endpoint in sorted(d.served): payload = d.served.get(endpoint) or {} model = (payload.get('hf_model_id') or payload.get('model') or d.spec.get('hf_model_id') or '-') - if d.engine == 'ollama': - service = ollama_service_name_for(d.spec.get('host') or endpoint) - else: - service = vllm_service_name_for( - payload.get('served_model_name') or endpoint) - if running is None: - health = 'unverified' - elif service in running: - health = 'up' - else: - # The ledger says live and nothing is running. Almost always a - # converge that failed after the record was written. - health = 'STALE' rows.append((endpoint, model, d.engine, health)) return rows def _gather_status(config) -> dict[str, Any]: - compose_file = _leasing_compose_file() + try: + backend = _day2_backend(config) + except (SystemExit, Exception): # noqa: BLE001 - status never fails on this + backend = None + rendered = getattr(backend, 'rendered_file', None) + leasing = _leasing_status() + if leasing.get('live_deployments'): + leasing['served'] = _served_models(leasing.pop('live_deployments'), backend) + else: + leasing.pop('live_deployments', None) return { 'backend': str(get_setting('backend') or 'null'), 'data_dir': str(data_root()), @@ -206,16 +208,17 @@ def _gather_status(config) -> dict[str, Any]: 'settings': {'path': str(settings_path()), 'exists': settings_path().exists()}, 'catalog': _catalog_summary(config), - 'compose': {'path': str(compose_file), 'exists': compose_file.exists()}, - 'leasing': _leasing_status(), + 'rendered': {'path': str(rendered) if rendered else None, + 'exists': bool(rendered and rendered.exists())}, + 'leasing': leasing, } _DIG_DEEPER = ( ('infer-stack leases', 'full lease + deployment tables'), ('infer-stack tui', 'live dashboard (opt-in: infer-stack[tui])'), - ('infer-stack stack ps', 'running containers'), - ('infer-stack logs -f', 'tail service logs'), + ('infer-stack ps', 'what is running (containers or pods)'), + ('infer-stack logs -f ', 'follow an engine\'s log'), ('infer-stack catalog show', 'what you can serve'), ) @@ -237,11 +240,14 @@ def _served_lines(served: list[tuple[str, str, str, str]]) -> list[str]: for endpoint, model, _engine, health in served: out.append(f' {endpoint.ljust(w_ep)} {model.ljust(w_mo)} {health}') if any(r[3] == 'STALE' for r in served): - out.append(' STALE = the ledger records this live but no container is ' - 'running; `infer-stack apply` or `gc`') + out.append(' STALE = the ledger records this live but nothing is ' + 'running for it; `infer-stack apply` or `gc`') + if any(r[3] == 'starting' for r in served): + out.append(' starting = up, but not ready yet (loading the model); ' + '`infer-stack wait `') if any(r[3] == 'unverified' for r in served): - out.append(' unverified = could not reach docker to confirm; the ' - 'ledger says live') + out.append(' unverified = the runtime could not be read to confirm; ' + 'the ledger says live') return out @@ -262,8 +268,9 @@ def _print_status_plain(d: dict[str, Any]) -> None: f'{"" if d["configured"] else " (run infer-stack config init)"}') lz = d['leasing'] print(f' ledger: {lz["path"]}{"" if lz["exists"] else " (none yet)"}') - print(f' compose: {d["compose"]["path"]}' - f'{"" if d["compose"]["exists"] else " (not rendered yet)"}') + if d['rendered']['path']: + print(f' rendered: {d["rendered"]["path"]}' + f'{"" if d["rendered"]["exists"] else " (not rendered yet)"}') if lz['summary']: active, total_l, live, total_d = lz['summary'] print() @@ -309,10 +316,11 @@ def _print_status_rich(d: dict[str, Any], console) -> None: if not lz['exists']: ledger.append(' (none yet)', style='dim') table.add_row('ledger', ledger) - compose = Text(d['compose']['path'], style='cyan') - if not d['compose']['exists']: - compose.append(' (not rendered yet)', style='dim') - table.add_row('compose', compose) + if d['rendered']['path']: + rendered = Text(d['rendered']['path'], style='cyan') + if not d['rendered']['exists']: + rendered.append(' (not rendered yet)', style='dim') + table.add_row('rendered', rendered) console.print(table) if lz['summary']: @@ -332,7 +340,8 @@ def _print_status_rich(d: dict[str, Any], console) -> None: served_table.add_column('model', overflow='fold') served_table.add_column('engine', style='dim', no_wrap=True) served_table.add_column('health', no_wrap=True) - styles = {'up': 'green', 'STALE': 'red', 'unverified': 'yellow'} + styles = {'up': 'green', 'starting': 'yellow', 'STALE': 'red', + 'unverified': 'yellow'} for endpoint, model, engine, health in served: served_table.add_row( endpoint, model, engine, @@ -343,7 +352,7 @@ def _print_status_rich(d: dict[str, Any], console) -> None: console.print(served_table) if any(r[3] == 'STALE' for r in served): console.print(Text( - ' STALE = recorded live but no container is running; ' + ' STALE = recorded live but nothing is running for it; ' '`infer-stack apply` or `gc`', style='dim')) console.print() @@ -385,66 +394,161 @@ def main(cls, argv=True, **kwargs): # --------------------------------------------------------------------------- -# stack — docker compose day-2-ops wrappers over the leasing project +# ps / logs / stack: what the backend runs, on either backend # --------------------------------------------------------------------------- -class _ComposeWrapperBase(_PathOverridesMixin): - """Common fields for ``docker compose `` wrappers over the leasing - Compose deployment.""" +class _InstancesBase(_PathOverridesMixin): + """Options shared by the verbs that read the backend's instances.""" services = kw.Value( None, nargs='*', position=1, - help='Optional service names to filter (empty = all).', + help='Which instances: a service or pod name, a container id (prefix), ' + 'a deployment id, or an endpoint alias. Empty = all.', + ) + backend = kw.Value( + None, type=str, + help='Backend to read (default: the configured `backend` setting).', ) -def _run_compacted_follow(cmd: list[str]) -> int: - """Stream Compose logs through the conservative LiteLLM compactor.""" - proc = subprocess.Popen( - cmd, - stdout=subprocess.PIPE, - text=True, - errors='replace', - bufsize=1, - env=_docker_env(), +#: States `ps` hides unless --all: finished, or on the way out. +_PS_HIDDEN = frozenset({'exited', 'dead', 'removing'}) + + +def _ps_rows(instances, served) -> list[dict[str, Any]]: + rows = [] + for inst in instances: + rows.append({ + 'name': inst.name, + 'id': inst.id, + 'deployment': inst.deployment_id or None, + 'serves': served.get(inst.deployment_id, []) if inst.deployment_id else [], + 'state': inst.state, + 'status': inst.status, + 'restarts': inst.restarts, + 'gpus': list(inst.gpus), + 'started': inst.started or None, + 'ports': inst.ports or None, + 'runtime': inst.runtime, + }) + return rows + + +def _print_ps(rows) -> None: + def cell(row): + serves = ', '.join(row['serves']) or ('-' if row['deployment'] else '(front door)') + gpus = ','.join(map(str, row['gpus'])) or '-' + ident = row['id'][:12] if row['runtime'] == 'docker' else '-' + return (row['name'], row['status'], serves, gpus, + (row['started'] or '-')[:19], ident, row['ports'] or '-') + + head = ('NAME', 'STATUS', 'SERVES', 'GPUS', 'STARTED', 'ID', 'PORTS') + table = [head, *(cell(r) for r in rows)] + widths = [max(len(str(r[i])) for r in table) for i in range(len(head))] + for r in table: + print(' '.join(str(v).ljust(w) for v, w in zip(r, widths)).rstrip()) + + +class PsCLI(_InstancesBase): + """What the backend is running: engine containers or pods, and the gateway. + + One shape on every backend. An engine row names the endpoints it serves; + the gateway, UI and proxy show as the front door. Reads the same strict + residency the controller decides with, so a row here is what admission + sees. + """ + + __command__ = 'ps' + + all = kw.Value( + False, isflag=True, short_alias=['a'], + help='Include exited and dying instances.', ) - try: - if proc.stdout is None: # pragma: no cover - PIPE guarantees stdout - return int(proc.wait()) - for line in compact_litellm_tracebacks(proc.stdout): - sys.stdout.write(line) - return int(proc.wait()) - except KeyboardInterrupt: - # The child normally receives the same SIGINT. If it is still alive, - # make sure an interrupted follow does not leave Compose behind. - if proc.poll() is None: - proc.terminate() + services_only = kw.Value( + False, isflag=True, help='Print only instance names.', + ) + quiet = kw.Value( + False, isflag=True, short_alias=['q'], + help='Print only ids (container ids, or pod names).', + ) + json = kw.Value(False, isflag=True, help='Print the rows as JSON.') + + @classmethod + def main(cls, argv=True, **kwargs): + import json + + from ..leasing.instances import UnknownTarget, resolve + + config = cls.cli(argv=argv, data=kwargs) + backend = _day2_backend(config) + instances = _instances(backend) + served = _served_by_deployment() + if config.services: try: - proc.wait(timeout=2) - except subprocess.TimeoutExpired: - proc.kill() - proc.wait() - return 130 + instances = resolve(instances, config.services, served) + except UnknownTarget as ex: + raise SystemExit(str(ex)) + if not config.all: + instances = [i for i in instances if i.state not in _PS_HIDDEN] + rows = _ps_rows(instances, served) + if config.json: + print(json.dumps(rows, indent=2)) + elif config.quiet: + for row in rows: + print(row['id']) + elif config.services_only: + for row in rows: + print(row['name']) + elif not rows: + print(f'nothing running ({_backend_name(backend)} backend); ' + 'bring a model up with `infer-stack acquire `') + else: + _print_ps(rows) + return 0 + +#: Prefix colors for `logs` on a terminal, cycled per instance name. +_LOG_COLORS = ('36', '33', '32', '35', '34', '96', '93', '92', '95', '94') -class LogsCLI(_ComposeWrapperBase): - """Tail leasing Compose service logs without typing the full docker path.""" + +def _colorize(lines, *, enabled: bool): + """Color each ``name | `` prefix, one color per name, like Compose did.""" + if not enabled: + yield from lines + return + colors: dict[str, str] = {} + for line in lines: + name, sep, rest = line.partition(' | ') + if not sep: + yield line + continue + code = colors.setdefault(name, _LOG_COLORS[len(colors) % len(_LOG_COLORS)]) + yield f'\x1b[{code}m{name} |\x1b[0m {rest}' + + +class LogsCLI(_InstancesBase): + """Engine and gateway logs, by instance, deployment or endpoint alias. + + ``infer-stack logs qwen -f`` follows whatever serves the ``qwen`` + endpoint, container or pod; with no names, every instance. Following + picks up instances that start later and follows a restarted one again. + """ __command__ = 'logs' follow = kw.Value( False, isflag=True, short_alias=['f'], - help='Stream logs (docker compose logs -f).', + help='Keep streaming, including instances that start later.', ) tail = kw.Value( None, type=str, - help="Tail the last N lines (default: all). Pass a number or 'all'.", + help="Only the last N lines of each (default: all). A number or 'all'.", ) timestamps = kw.Value(False, isflag=True) - no_color = kw.Value(False, isflag=True) + no_color = kw.Value(False, isflag=True, help='Do not color the name prefixes.') raw = kw.Value( False, isflag=True, @@ -453,74 +557,96 @@ class LogsCLI(_ComposeWrapperBase): @classmethod def main(cls, argv=True, **kwargs): + from ..leasing.instances import ( + LogFollower, + UnknownTarget, + history_argv, + resolve, + runtime_env, + ) + config = cls.cli(argv=argv, data=kwargs) + backend = _day2_backend(config) + served = _served_by_deployment() + names = list(config.services or []) - # The compacted path pipes Compose stdout through Python. Without an - # explicit ANSI mode Compose sees a non-TTY pipe and drops the service - # colors before the compactor can preserve them. Force ANSI only for - # the human-facing compacted view; --no-color remains authoritative. - compact = config.follow and not config.raw and sys.stdout.isatty() - cmd = _day2_compose_base(config, 'logs') - if compact and not config.no_color: - cmd.extend(['--ansi', 'always']) - cmd.append('logs') - if config.follow: - cmd.append('--follow') - if config.tail is not None: - cmd.extend(['--tail', str(config.tail)]) - if config.no_color: - cmd.append('--no-color') - if config.timestamps: - cmd.append('--timestamps') - cmd.extend(config.services or []) + def pick(instances): + return resolve(instances, names, served) if names else instances - # Compact only the human-facing live view. Captures and pipelines keep - # Docker's exact bytes unless a future explicit compact-output mode is - # added; ``--raw`` is also available for interactive LiteLLM debugging. - if compact: - return _run_compacted_follow(cmd) - return int(subprocess.run(cmd, env=_docker_env()).returncode) + try: + chosen = pick(_instances(backend)) + except UnknownTarget as ex: + raise SystemExit(str(ex)) + color = sys.stdout.isatty() and not config.no_color + if config.follow: + def listing(): + try: + return pick(backend.instances()) + except UnknownTarget: + return [] # the named one is between restarts + follower = LogFollower(listing, history=config.tail or 'all', + timestamps=bool(config.timestamps)) + lines = follower.stdout + if sys.stdout.isatty() and not config.raw: + lines = compact_litellm_tracebacks(lines) + try: + for line in _colorize(lines, enabled=color): + sys.stdout.write(line) + sys.stdout.flush() + except KeyboardInterrupt: + return 130 + finally: + follower.terminate() + return 0 + if not chosen: + print(f'nothing running ({_backend_name(backend)} backend)') + return 0 + prefix = len(chosen) > 1 + status = 0 + for inst in chosen: + argv_ = history_argv(inst, tail=config.tail, + timestamps=bool(config.timestamps)) + if argv_ is None: + print(f'{inst.name}: the {_backend_name(backend)} backend keeps no logs') + continue + proc = subprocess.run(argv_, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, env=runtime_env(inst)) + text = proc.stdout.decode('utf-8', 'replace') + lines = [f'{inst.name} | {ln}\n' if prefix else f'{ln}\n' + for ln in text.splitlines()] + for line in _colorize(lines, enabled=color and prefix): + sys.stdout.write(line) + status = status or proc.returncode + return int(status) -class PsCLI(_ComposeWrapperBase): - """``docker compose ps`` for the leasing deployment.""" +class _ComposeWrapperBase(_PathOverridesMixin): + """``docker compose `` over the Compose project on this host. - __command__ = 'ps' + The stack itself on the compose backend; the gateway on kubeai. + """ - all = kw.Value( - False, isflag=True, short_alias=['a'], help='Include stopped containers.' - ) - services_only = kw.Value( - False, isflag=True, - help='Print only service names (passes --services to docker compose).', + services = kw.Value( + None, + nargs='*', + position=1, + help='Optional service names to filter (empty = all).', ) - quiet = kw.Value( - False, isflag=True, short_alias=['q'], help='Print only container IDs.' + backend = kw.Value( + None, type=str, + help='Backend (default: the configured `backend` setting).', ) - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'ps') + ['ps'] - if config.all: - cmd.append('--all') - if config.services_only: - cmd.append('--services') - if config.quiet: - cmd.append('--quiet') - cmd.extend(config.services or []) - return int(subprocess.run(cmd, env=_docker_env()).returncode) - class RestartCLI(_ComposeWrapperBase): - """``docker compose restart [services...]``.""" + """``docker compose restart [services...]`` (the Compose project on this host).""" timeout = kw.Value(None, type=int, help='Stop timeout in seconds.') @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'restart') + ['restart'] + cmd = _compose_argv(config) + ['restart'] if config.timeout is not None: cmd.extend(['--timeout', str(config.timeout)]) cmd.extend(config.services or []) @@ -528,7 +654,7 @@ def main(cls, argv=True, **kwargs): class PullCLI(_ComposeWrapperBase): - """``docker compose pull [services...]``.""" + """``docker compose pull [services...]`` (the Compose project on this host).""" quiet = kw.Value(False, isflag=True, short_alias=['q']) ignore_pull_failures = kw.Value(False, isflag=True) @@ -536,7 +662,7 @@ class PullCLI(_ComposeWrapperBase): @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'pull') + ['pull'] + cmd = _compose_argv(config) + ['pull'] if config.quiet: cmd.append('--quiet') if config.ignore_pull_failures: @@ -546,81 +672,119 @@ def main(cls, argv=True, **kwargs): class StartCLI(_ComposeWrapperBase): - """``docker compose start [services...]``.""" + """``docker compose start [services...]`` (the Compose project on this host).""" @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'start') + ['start'] + cmd = _compose_argv(config) + ['start'] cmd.extend(config.services or []) return int(subprocess.run(cmd, env=_docker_env()).returncode) class StopCLI(_ComposeWrapperBase): - """``docker compose stop [services...]``.""" + """``docker compose stop [services...]`` (the Compose project on this host).""" timeout = kw.Value(None, type=int) @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'stop') + ['stop'] + cmd = _compose_argv(config) + ['stop'] if config.timeout is not None: cmd.extend(['--timeout', str(config.timeout)]) cmd.extend(config.services or []) return int(subprocess.run(cmd, env=_docker_env()).returncode) -class StackDownCLI(_ComposeWrapperBase): - """``docker compose down`` the leasing deployment. +class StackComposeCLI(_PathOverridesMixin): + """Run any ``docker compose`` command on the Compose project on this host. - Tears the whole project down. Leasing's reconcile manages teardown - automatically on release; this is the manual escape hatch. + The raw escape hatch: the stack itself on the compose backend, the gateway + on kubeai. It bypasses the ledger and renders nothing, e.g. + ``infer-stack stack compose -- up -d litellm``. """ - volumes = kw.Value( - False, isflag=True, help='Also remove named volumes (--volumes).' + __command__ = 'compose' + + args = kw.Value(None, nargs='*', position=1, + help='Arguments for docker compose (put them after --).') + backend = kw.Value( + None, type=str, + help='Backend (default: the configured `backend` setting).', ) @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'down') + ['down', '--remove-orphans'] - if config.volumes: - cmd.append('--volumes') + cmd = _compose_argv(config) + list(config.args or []) return int(subprocess.run(cmd, env=_docker_env()).returncode) -class StackUpCLI(_ComposeWrapperBase): - """``docker compose up -d`` exactly what is on disk — the raw escape hatch. +class StackDownCLI(_PathOverridesMixin): + """Stop everything the backend runs, bypassing the ledger. - Brings up the on-disk compose file as-is, without touching the ledger or - re-rendering. Prefer ``infer-stack apply``, which re-renders from intent - first. Reach for ``stack up`` only to run a *hand-edited* compose file - verbatim. + The manual escape hatch: releases no lease, so a later publish brings + leased models back. Compose: ``docker compose down``; kubeai: deletes every + managed Model, then the gateway. ``--volumes`` (Compose only) also removes + named volumes. """ + __command__ = 'down' + + volumes = kw.Value( + False, isflag=True, help='Also remove named volumes (Compose only).' + ) + backend = kw.Value( + None, type=str, + help='Backend (default: the configured `backend` setting).', + ) + @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'up') + ['up', '-d', '--remove-orphans'] - cmd.extend(config.services or []) - return int(subprocess.run(cmd, env=_docker_env()).returncode) + if config.volumes: + cmd = _compose_argv(config) + ['down', '--remove-orphans', '--volumes'] + return int(subprocess.run(cmd, env=_docker_env()).returncode) + backend = _day2_backend(config) + down = getattr(backend, 'down', None) + if down is None: + print(f'the {_backend_name(backend)} backend runs nothing to bring down') + return 0 + down() + return 0 + + +class StackUpCLI(ApplyCLI): + """``apply``: bring up what the ledger says should run (both backends). + + The raw form, ``docker compose up`` of the file on disk, is + ``infer-stack stack compose -- up -d``. + """ + + __command__ = 'up' class StackModalCLI(kw.ModalCLI): - """Day-2 ops on the running leasing deployment.""" + """Day-2 ops on what the backend runs. + + ``up`` is ``apply`` and ``down`` stops everything, on either backend. The + ``docker compose`` verbs (restart, pull, start, stop, and ``compose`` for + anything else) act on the Compose project on this host: the stack itself + on compose, the gateway on kubeai. + """ __command__ = 'stack' up = StackUpCLI logs = LogsCLI ps = PsCLI + down = StackDownCLI + compose = StackComposeCLI restart = RestartCLI pull = PullCLI start = StartCLI stop = StopCLI - down = StackDownCLI class DoctorCLI(_PathOverridesMixin): diff --git a/infer_stack/cli/options.py b/infer_stack/cli/options.py index f4a1c3ce..1acb1413 100644 --- a/infer_stack/cli/options.py +++ b/infer_stack/cli/options.py @@ -9,7 +9,53 @@ # --------------------------------------------------------------------------- -class _PathOverridesMixin(kw.Config): +#: Values a bare flag may carry (``--yes false``); anything else was a positional. +_BOOL_WORDS = frozenset({'true', 'false', 'yes', 'no', 'on', 'off', '1', '0'}) + + +def reclaim_swallowed_positionals(config) -> None: + """Give back a positional argument that a preceding flag consumed. + + kwconf flags take an optional value, so ``logs -f qwen`` parsed as + ``follow='qwen'`` and no names: the command then acted on everything. + Every infer-stack flag is a boolean, so a flag holding any other string + was handed a positional. It becomes ``True``, and the string goes back to + the front of the command's positional list (or its single positional, + when that is still empty). + """ + defaults = type(config).__default__ + positional = sorted((k for k, v in defaults.items() if getattr(v, 'position', None)), + key=lambda k: defaults[k].position) + for key, value in defaults.items(): + if not getattr(value, 'isflag', False) or value.isflag == 'counter': + continue + got = config[key] + if not isinstance(got, str) or got.strip().lower() in _BOOL_WORDS: + continue + config[key] = True + if not positional: + raise SystemExit(f'--{key} takes no value (got {got!r})') + target = positional[0] + many = defaults[target].parsekw.get('nargs') in ('*', '+') + if many: + config[target] = [got, *(config[target] or [])] + elif config[target] in (None, ''): + config[target] = got + else: + raise SystemExit(f'--{key} takes no value (got {got!r})') + + +class _FlagSafeMixin(kw.Config): + """Parses ``--flag positional`` as a flag and a positional (see above).""" + + @classmethod + def cli(cls, *args, **kwargs): + config = super().cli(*args, **kwargs) + reclaim_swallowed_positionals(config) + return config + + +class _PathOverridesMixin(_FlagSafeMixin): """Adds global ``--config-dir`` / ``--data-dir`` to a subcommand.""" config_dir = kw.Value( diff --git a/infer_stack/leasing/backend.py b/infer_stack/leasing/backend.py index 9755130c..d50454d6 100644 --- a/infer_stack/leasing/backend.py +++ b/infer_stack/leasing/backend.py @@ -188,6 +188,13 @@ def residency(self) -> Any: """A strict :class:`~infer_stack.leasing.residency.Residency`, or raise.""" ... + def instances(self) -> list[Any]: + """Every running unit, as :class:`~infer_stack.leasing.instances.Instance`. + + Built from :meth:`residency`, so it raises when that does. + """ + ... + def preview(self, desired: list[Deployment], placement: Any = None, *, approve: bool = False) -> Any: """Place and render ``desired`` without writing; ``(plan, rendered)``.""" @@ -375,6 +382,12 @@ def residency(self): for gid in sorted(self.observe()) }) + def instances(self): + """One in-process instance per realized deployment; no log to read.""" + from .instances import MEMORY, from_residency + + return from_residency(self.residency(), runtime=MEMORY) + def _preview(self, desired: list[Deployment], placement: Any = None): desired = list(desired) plan = self.plan(desired, placement) diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index 1ea628bd..0470a36f 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -1093,6 +1093,26 @@ def inventory(self, value: dict[str, Any] | None) -> None: def compose_file(self) -> Path: return self.state_dir / COMPOSE_FILENAME + #: The file a render writes, whatever the backend (``status`` shows it). + rendered_file = compose_file + + def compose_project(self): + """The Compose project on this host: this one.""" + return self + + def compose_argv(self) -> list[str]: + """``docker compose [--env-file ...] -p -f ``: the one base. + + The managed ``.env`` beside the compose file resolves the master key + and the other managed secrets; docker compose's own ``.env`` discovery + keys off the caller's working directory, so it is passed explicitly, + and only when present (a missing ``--env-file`` is a hard error). + """ + cmd = ['docker', 'compose'] + if self.gateway._env_path.exists(): + cmd += ['--env-file', str(self.gateway._env_path)] + return [*cmd, '-p', self.project, '-f', str(self.compose_file)] + # -- the front door (leasing.gateway) ------------------------------------ # Settings the gateway owns. Kept as attributes of the backend because # profiles and callers set them here; they are stored in one place. @@ -1337,17 +1357,7 @@ def doctor(self) -> list[tuple[str, bool, str]]: def _compose(self, args: list[str]) -> str: self._ensure_state_dir() - cmd = ['docker', 'compose'] - # Resolve ${LITELLM_MASTER_KEY} (and any other managed secret) from the - # sidecar .env beside the compose file, so secrets stay out of the YAML. - # docker compose's default .env discovery keys off the *current working - # directory* (wherever infer-stack was invoked), not the state dir, so we - # point it explicitly. Only when present: a litellm-less stack never - # writes one, and a missing --env-file path is a hard error. - if self.gateway._env_path.exists(): - cmd += ['--env-file', str(self.gateway._env_path)] - cmd += ['-p', self.project, '-f', str(self.compose_file)] - return self.run([*cmd, *args]) + return self.run([*self.compose_argv(), *args]) def plan(self, desired: list[Deployment], placement=None): """Compute GPU placement for ``desired`` without writing or applying. @@ -2202,6 +2212,12 @@ def settle_snapshot(self) -> tuple[tuple[str, str], ...]: pairs.append((parts[0], parts[1].lower())) return tuple(sorted(pairs)) + def instances(self): + """Every container of this project, engines first (raises if unknown).""" + from .instances import DOCKER, from_residency + + return from_residency(self.residency(), runtime=DOCKER) + def residency(self) -> Residency: """Strict snapshot of this project's deployment containers and their GPUs. diff --git a/infer_stack/leasing/instances.py b/infer_stack/leasing/instances.py new file mode 100644 index 00000000..3b4f9df6 --- /dev/null +++ b/infer_stack/leasing/instances.py @@ -0,0 +1,278 @@ +"""What a backend is running, one row per unit, and how to read its log. + +An :class:`Instance` is a container (Compose, or the KubeAI gateway) or a pod +(KubeAI). Backends build them from their strict residency +(:meth:`instances`), so ``ps``, ``logs``, ``status`` and the TUI read the +same view the controller decides with. The runtime an instance lives in is +recorded on it, so reading its log needs no knowledge of which backend +produced it: :func:`history_argv` and :func:`follow_argv` are the only place +that knows ``docker logs`` from ``kubectl logs``. +""" + +from __future__ import annotations + +import subprocess +from dataclasses import dataclass +from typing import Callable, Iterable, Iterator, Mapping + +#: Runtimes an instance can live in. +DOCKER = 'docker' +KUBERNETES = 'kubernetes' +#: In-process backends (dry-run, tests): nothing to read a log from. +MEMORY = 'memory' + + +@dataclass(frozen=True) +class Instance: + """One running unit: an engine container or pod, or a front-door service.""" + + #: What a user types: the Compose service, or the pod name. + name: str + #: The runtime's own id: container id, or the pod name. + id: str + #: The deployment it serves; empty for the gateway, UI, database, proxy. + deployment_id: str + state: str + restarts: int = 0 + #: Why it is not running, when the runtime says (``CrashLoopBackOff``...). + reason: str = '' + #: ``healthy`` / ``starting`` / ``unhealthy``, when the runtime has a check. + health: str = '' + gpus: tuple[int, ...] = () + started: str = '' + ports: str = '' + runtime: str = DOCKER + #: Kubernetes only: where the pod lives and which container is the engine. + namespace: str = '' + container: str = '' + + @property + def is_engine(self) -> bool: + return bool(self.deployment_id) + + @property + def status(self) -> str: + """``running``, ``restarting (CrashLoopBackOff, 3 restarts)``, ...""" + bits = [b for b in (self.reason, self.health if self.health != 'healthy' else '') + if b] + if self.restarts: + bits.append(f'{self.restarts} restart{"s" if self.restarts != 1 else ""}') + return f'{self.state} ({", ".join(bits)})' if bits else self.state + + +def from_residency(residency, *, runtime: str = DOCKER, namespace: str = '', + container: str = '') -> list[Instance]: + """Every instance in a :class:`~infer_stack.leasing.residency.Residency`.""" + out = [] + for c in residency.all_containers(): + name = c.service if runtime == DOCKER and c.service else c.container_id + out.append(Instance( + name=name, id=c.container_id, deployment_id=c.deployment_id, + state=c.state, restarts=c.restart_count, reason=c.reason, + health=c.health, gpus=tuple(c.gpus), started=c.started, ports=c.ports, + runtime=runtime, namespace=namespace, container=container, + )) + return sorted(out, key=lambda i: (not i.is_engine, i.name, i.id)) + + +def history_argv(instance: Instance, *, tail: str | int | None = None, + timestamps: bool = False) -> list[str] | None: + """The command that prints ``instance``'s log so far, or ``None``.""" + flags = [] if tail is None else ['--tail', str(tail)] + if timestamps: + flags.append('--timestamps') + if instance.runtime == DOCKER: + return ['docker', 'logs', *flags, instance.id] + if instance.runtime == KUBERNETES: + return ['kubectl', '-n', instance.namespace, 'logs', *flags, instance.id, + *(['-c', instance.container] if instance.container else [])] + return None + + +def follow_argv(instance: Instance, *, timestamps: bool = False) -> list[str] | None: + r"""The command that streams ``instance``'s output from now on, or ``None``. + + Docker: ``docker attach`` with stdin closed and signals not forwarded, not + ``docker logs -f``: the log driver holds a partial line until its newline, + so a progress bar redrawn with ``\r`` shows nothing for minutes (measured). + Ending the attach never touches the container. Kubernetes has no such + driver in the way: ``kubectl logs -f --tail 0``. + """ + if instance.runtime == DOCKER: + return ['docker', 'attach', '--no-stdin', '--sig-proxy=false', instance.id] + if instance.runtime == KUBERNETES: + argv = history_argv(instance, tail=0, timestamps=timestamps) + assert argv is not None + return [*argv, '--follow'] + return None + + +def runtime_env(instance: Instance) -> dict[str, str] | None: + """The environment to run the instance's log commands in (None: inherit).""" + if instance.runtime == DOCKER: + from .compose import docker_environment + + return docker_environment() + return None + + +class UnknownTarget(ValueError): + """A name that matches no instance; the message lists what exists.""" + + +def resolve(instances: list[Instance], names: Iterable[str], + served: Mapping[str, Iterable[str]] | None = None) -> list[Instance]: + """The instances ``names`` refer to, in order, without repeats. + + A name matches an instance's name, its id (a prefix of at least four + characters, as Docker accepts), its deployment id, or an endpoint alias + the deployment serves (``served``: deployment id -> aliases). One alias + or deployment can match several instances (a pod and its restart). + + >>> a = Instance('vllm-qwen', 'c0ffee12', 'grp-1', 'running') + >>> b = Instance('litellm', 'deadbeef', '', 'running') + >>> [i.name for i in resolve([a, b], ['qwen', 'litellm'], {'grp-1': ['qwen']})] + ['vllm-qwen', 'litellm'] + >>> [i.name for i in resolve([a, b], ['c0ff'])] + ['vllm-qwen'] + >>> resolve([a, b], ['nope']) + Traceback (most recent call last): + ... + infer_stack.leasing.instances.UnknownTarget: no instance matches 'nope'; running: litellm, vllm-qwen + """ + served = {gid: set(aliases) for gid, aliases in (served or {}).items()} + picked: list[Instance] = [] + for name in names: + found = [ + i for i in instances + if name in (i.name, i.id, i.deployment_id) + or (len(name) >= 4 and i.id.startswith(name)) + or (i.deployment_id and name in served.get(i.deployment_id, ())) + ] + if not found: + known = ', '.join(sorted({i.name for i in instances})) or 'nothing' + raise UnknownTarget(f'no instance matches {name!r}; running: {known}') + picked.extend(i for i in found if i not in picked) + return picked + + +class LogFollower: + r"""Follow instances' logs: recent history, then live output. + + ``list_instances`` is called every ``poll`` seconds, so an instance that + appears later (a model starting, a recreate) is followed from its first + line, and one that restarted is followed again. ``stdout`` yields + ``name | line``. Output written between the history read and the live + stream (a few milliseconds) can be missed. + """ + + poll = 3.0 + + def __init__(self, list_instances: Callable[[], list[Instance]], *, + history: str | int = 200, timestamps: bool = False): + import queue + import threading + + self._list = list_instances + self._history = history + self._timestamps = timestamps + self._lines: queue.Queue = queue.Queue() + self._stop = threading.Event() + self._seen: set[str] = set() + self._live: dict[str, subprocess.Popen] = {} + threading.Thread(target=self._watch, daemon=True).start() + + def _watch(self) -> None: + import threading + + first = True + while not self._stop.is_set(): + try: + found = self._list() + except Exception: # noqa: BLE001 - runtime unreachable: try again + found = [] + for inst in found: + running = inst.state == 'running' + live = self._live.get(inst.id) + attached = live is not None and live.poll() is None + if inst.id not in self._seen: + # Existing instances: the recent tail. A new one: all of it. + history: str | int | None = self._history if first else 'all' + elif running and not attached: + history = None # restarted: follow again, no repeat + else: + continue + self._seen.add(inst.id) + threading.Thread(target=self._follow, args=(inst, history, running), + daemon=True).start() + first = False + self._stop.wait(self.poll) + + def _follow(self, inst: Instance, history, running: bool) -> None: + from ..log_filter import LogLineSplitter + + prefix = f'{inst.name} | ' + env = runtime_env(inst) + if history is not None: + argv = history_argv(inst, tail=history, timestamps=self._timestamps) + old = '' + if argv is not None: + try: + old = subprocess.run( + argv, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + timeout=60, env=env, + ).stdout.decode('utf-8', 'replace') + except Exception: # noqa: BLE001 + old = '' + split = LogLineSplitter(every=0.0) + for line in split.feed(old) + split.flush(): + self._lines.put(prefix + line) + argv = follow_argv(inst, timestamps=self._timestamps) + if not running or argv is None or self._stop.is_set(): + return + # Engines log to stderr, which docker attach passes out on its own stderr. + proc = subprocess.Popen(argv, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, env=env) + self._live[inst.id] = proc + self._pump(proc, prefix) + + def _pump(self, proc: subprocess.Popen, prefix: str) -> None: + import codecs + import os + + from ..log_filter import LogLineSplitter + + decode = codecs.getincrementaldecoder('utf-8')('replace').decode + split = LogLineSplitter() + assert proc.stdout is not None # Popen(stdout=PIPE) + fd = proc.stdout.fileno() + try: + while True: + chunk = os.read(fd, 65536) + if not chunk: + break + for line in split.feed(decode(chunk)): + self._lines.put(prefix + line) + finally: + proc.stdout.close() + for line in split.flush(): + self._lines.put(prefix + line) + + @property + def stdout(self) -> Iterator[str]: + import queue + + while not self._stop.is_set(): + try: + yield self._lines.get(timeout=0.5) + '\n' + except queue.Empty: + continue + + def terminate(self) -> None: + self._stop.set() + for proc in list(self._live.values()): + try: + proc.terminate() + proc.wait(timeout=2) + except Exception: # noqa: BLE001 + proc.kill() diff --git a/infer_stack/leasing/residency.py b/infer_stack/leasing/residency.py index bc5e9446..51ccbabf 100644 --- a/infer_stack/leasing/residency.py +++ b/infer_stack/leasing/residency.py @@ -58,6 +58,21 @@ COMPOSE_PROJECT_LABEL = 'com.docker.compose.project' COMPOSE_SERVICE_LABEL = 'com.docker.compose.service' +def _published_ports(ports) -> str: + """``14042->4000/tcp`` for each published port of a ``docker inspect``. + + >>> _published_ports({'4000/tcp': [{'HostIp': '0.0.0.0', 'HostPort': '14042'}, + ... {'HostIp': '::', 'HostPort': '14042'}], + ... '8000/tcp': None}) + '14042->4000/tcp' + """ + out = [] + for inner, bindings in sorted((ports or {}).items()): + for host in sorted({b.get('HostPort') for b in bindings or [] if b.get('HostPort')}): + out.append(f'{host}->{inner}') + return ', '.join(out) + + #: Container states that hold, or will reclaim on their own, a warm model: the #: process is up, is being restarted by Docker's restart policy, or is paused #: (a paused process keeps its GPU memory). ``created``, ``exited``, @@ -112,6 +127,10 @@ class Container: #: ``CrashLoopBackOff``, ``ImagePullBackOff``, ``OOMKilled`` (Kubernetes). #: Empty when there is nothing to say, or the runtime does not say. reason: str = '' + #: When the current run started (the runtime's timestamp), for display. + started: str = '' + #: Published ports, ``host->container/proto`` joined by ``, ``; display only. + ports: str = '' @property def warm(self) -> bool: @@ -261,6 +280,8 @@ def residency_from_inspect(raw: str, *, project: str) -> Residency: or {}).get('MaximumRetryCount')) or 0), exit_code=(None if (item.get('State') or {}).get('ExitCode') is None else int((item.get('State') or {})['ExitCode'])), + started=str((item.get('State') or {}).get('StartedAt') or ''), + ports=_published_ports((item.get('NetworkSettings') or {}).get('Ports')), ips=tuple(sorted( str(n.get('IPAddress')) for n in (((item.get('NetworkSettings') or {}).get('Networks')) or {}).values() @@ -362,6 +383,8 @@ def residency_from_pods(raw: str) -> Residency: # A Deployment's pods are always restarted by the kubelet. restart_policy='always', reason=waiting or str(ended.get('reason') or ''), + started=str((current.get('running') or {}).get('startedAt') + or status.get('startTime') or ''), ) if container.deployment_id: grouped.setdefault(container.deployment_id, []).append(container) diff --git a/infer_stack/tui.py b/infer_stack/tui.py index 17272da9..439afd00 100644 --- a/infer_stack/tui.py +++ b/infer_stack/tui.py @@ -12,8 +12,9 @@ it in Open WebUI. * **Leases** + **Deployments** (center) — the live ledger (desired *state* vs what's actually *running*, and which GPUs), with Release / Evict / Clean-up. -* **docker** — a collapsible pane with **Logs** and **Containers** (the - ``docker ps`` view: status/uptime, created, id, ports) tabs (collapsed by +* **runtime** — a collapsible pane with **Logs**, **Instances** (what + ``infer-stack ps`` shows: containers or pods, status, what each serves, + ports) and **Control** (Apply / Down) tabs, on either backend (collapsed by default; ``c`` toggles it). * **system** — live ``nvidia-smi`` GPUs + host CPU/mem (collapsed by default). * **api** — send a prompt to a *ready* model through the LiteLLM gateway @@ -31,7 +32,7 @@ import subprocess import time from pathlib import Path -from typing import Any, Callable, Iterable +from typing import Any, Callable from textual import events, work from textual.app import App, ComposeResult @@ -69,7 +70,7 @@ from rich.markup import escape as escape_markup from . import cli_equivalent as cli -from .log_filter import LogLineSplitter, compact_litellm_tracebacks +from .log_filter import compact_litellm_tracebacks ALL_SERVICES = '' # the Select value meaning "every service" # The Select value meaning "every service EXCEPT the gateway". This is the @@ -78,12 +79,8 @@ # pane before anyone can read it. The gateway's own logs are one selection away # when they are what you want. ENGINE_SERVICES = '\x00engines' -# Substring rather than equality: compose names the gateway service `litellm` -# today, but deployment naming has carried suffixes before and a missed match -# silently restores the noisy view. -GATEWAY_SERVICE_HINT = 'litellm' # What `_resolve_log_target` returns when a view has nothing to follow. Not -# None: None means "every service" to `docker compose logs`. +# None: None means "every instance". NO_LOG_TARGET = object() @@ -167,15 +164,6 @@ def _select_is_blank(value: object) -> bool: return value is SELECT_BLANK -def is_gateway_service(name: str) -> bool: - """Is this compose service the LiteLLM gateway rather than an engine?""" - return GATEWAY_SERVICE_HINT in str(name).lower() - - -def engine_services(names) -> list[str]: - """Every service that is not the gateway, in the given order.""" - return [n for n in names if not is_gateway_service(n)] - SELECT_MARK = '✓' # multi-select marker in the leases/deployments tables DEFAULT_THEME = 'textual-dark' @@ -257,151 +245,6 @@ def on_key(self, event: events.Key) -> None: ) -class _DockerLogProc: - r"""Follow the project's containers: recent history, then live output. - - Not ``docker compose logs -f``, and not ``docker logs -f`` either: Docker's - log driver stores output line by line and holds a partial line until its - newline, so a download bar redrawn with ``\r`` showed nothing for many - minutes and then arrived all at once (measured with both). ``docker - attach`` reads the container's output as it is written, so each container - gets its recent ``docker logs --tail`` (complete lines), then an attach - with stdin closed and signals not forwarded -- ending it never touches the - container. - - Containers are re-listed every ``poll`` seconds: one created later (a model - starting, a recreate) is followed from its first line, and one that - restarted is attached again. ``stdout`` yields ``service | line`` like - Compose. Output written between the history read and the attach (a few - milliseconds) can be missed. - """ - - poll = 3.0 - - def __init__(self, project: str, compose_file: str, service=None): - import queue - import threading - - del compose_file # the project label is enough to find them - self.project = project - if isinstance(service, (list, tuple)): - self.services = {str(s) for s in service} - else: - self.services = {str(service)} if service else None - self._lines: queue.Queue = queue.Queue() - self._stop = threading.Event() - self._seen: set[str] = set() - self._live: dict[str, subprocess.Popen] = {} - threading.Thread(target=self._watch, daemon=True).start() - - def _containers(self) -> list[tuple[str, str, bool]]: - """(id, service, running) for this project's containers in view.""" - from .leasing.compose import docker_environment - - out = subprocess.run( - ['docker', 'ps', '-a', '--filter', - f'label=com.docker.compose.project={self.project}', - '--format', '{{.ID}} {{.State}} {{.Label "com.docker.compose.service"}}'], - capture_output=True, text=True, timeout=30, env=docker_environment(), - ).stdout - found = [] - for row in out.splitlines(): - cid, _, rest = row.strip().partition(' ') - state, _, svc = rest.partition(' ') - if cid and (self.services is None or svc in self.services): - found.append((cid, svc, state == 'running')) - return sorted(found, key=lambda row: row[1]) - - def _watch(self) -> None: - import threading - - first = True - while not self._stop.is_set(): - try: - found = self._containers() - except Exception: # noqa: BLE001 - docker unreachable: try again - found = [] - for cid, svc, running in found: - attached = cid in self._live and self._live[cid].poll() is None - if cid not in self._seen: - # Existing containers: the recent tail. A new one: all of it. - history = '200' if first else 'all' - elif running and not attached: - history = None # restarted: attach again, no repeat - else: - continue - self._seen.add(cid) - threading.Thread(target=self._follow, args=(cid, svc, history, running), - daemon=True).start() - first = False - self._stop.wait(self.poll) - - def _follow(self, cid: str, service: str, history: str | None, running: bool) -> None: - from .leasing.compose import docker_environment - - prefix = f'{service} | ' - if history is not None: - try: - old = subprocess.run( - ['docker', 'logs', '--tail', history, cid], - stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=60, - env=docker_environment(), - ).stdout.decode('utf-8', 'replace') - except Exception: # noqa: BLE001 - old = '' - split = LogLineSplitter(every=0.0) - for line in split.feed(old) + split.flush(): - self._lines.put(prefix + line) - if not running or self._stop.is_set(): - return - proc = subprocess.Popen( - ['docker', 'attach', '--no-stdin', '--sig-proxy=false', cid], - # Engines log to stderr, which attach passes out on its own stderr. - stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=docker_environment(), - ) - self._live[cid] = proc - self._pump(proc, prefix) - - def _pump(self, proc: subprocess.Popen, prefix: str) -> None: - import codecs - import os - - decode = codecs.getincrementaldecoder('utf-8')('replace').decode - split = LogLineSplitter() - assert proc.stdout is not None # Popen(stdout=PIPE) - fd = proc.stdout.fileno() - try: - while True: - chunk = os.read(fd, 65536) - if not chunk: - break - for line in split.feed(decode(chunk)): - self._lines.put(prefix + line) - finally: - proc.stdout.close() - for line in split.flush(): - self._lines.put(prefix + line) - - @property - def stdout(self) -> Iterable[str]: - import queue - - while not self._stop.is_set(): - try: - yield self._lines.get(timeout=0.5) + '\n' - except queue.Empty: - continue - - def terminate(self) -> None: - self._stop.set() - for proc in list(self._live.values()): - try: - proc.terminate() - proc.wait(timeout=2) - except Exception: # noqa: BLE001 - proc.kill() - - class _Divider(Static): """A draggable splitter bar. @@ -1000,7 +843,7 @@ class InferStackTUI(App): Binding('n', 'add_endpoint', 'Add endpoint', show=False), Binding('o', 'open', 'Open in browser', show=False), Binding('y', 'copy_status', 'Copy status', show=False), - Binding('c', 'toggle_docker', 'Toggle docker', show=False), + Binding('c', 'toggle_docker', 'Toggle runtime', show=False), Binding('left_square_bracket', 'sidebar_narrower', 'sidebar -', show=False), Binding('right_square_bracket', 'sidebar_wider', 'sidebar +', show=False), Binding('minus', 'logs_shorter', 'logs -', show=False), @@ -1088,7 +931,7 @@ def __init__( self._log_backlog: collections.deque = collections.deque(maxlen=LOG_PANE_LINES) self._log_dropped = 0 # Parsed compose service names, keyed by the file's (mtime, size). - self._service_names_cache: tuple[tuple[float, int], list[str]] | None = None + self._last_instances: list = [] # Running background actions: worker -> (label, started). Drawn by # _draw_activity, with the newest backend progress message. self._activity: dict[Any, tuple[str, float]] = {} @@ -1227,31 +1070,32 @@ def _compose_dashboard(self) -> ComposeResult: ('Leases', self._compose_leases), ('Deployments', self._compose_deployments)) yield _Divider('y', self._drag_logs, id='hsplit') - with Collapsible(title='docker', collapsed=True, id='docker'): + with Collapsible(title='runtime', collapsed=True, id='docker'): with TabbedContent(id='docker-tabs'): with TabPane('Logs', id='tab-logs'): yield Select( - [('(engines — no litellm)', ENGINE_SERVICES), - ('(all services)', ALL_SERVICES)], + [('(engines)', ENGINE_SERVICES), + ('(everything)', ALL_SERVICES)], value=ENGINE_SERVICES, allow_blank=False, id='logsvc', ) yield RichLog(id='logs', highlight=False, markup=False, max_lines=LOG_PANE_LINES, wrap=False) - with TabPane('Containers', id='tab-containers'): + with TabPane('Instances', id='tab-containers'): yield DataTable(id='ps', cursor_type='row', zebra_stripes=True) with TabPane('Control', id='tab-control'): yield Static( - 'Bring the rendered compose project up or down. ' + 'Apply brings up what the ledger says should run. ' + 'Down stops everything and releases no lease. ' 'Output appears in the Logs tab.', classes='hint', ) yield Static('', id='compose-path') with Horizontal(id='compose-actions'): - yield Button('Compose up', id='btn-compose-up', + yield Button('Apply', id='btn-compose-up', variant='primary') - yield Button('Compose down', + yield Button('Down', id='btn-compose-down') with Collapsible(title='system', collapsed=True, id='system'): yield Static( @@ -1376,7 +1220,7 @@ def on_mount(self) -> None: 'leases', 'held by' ) self.query_one('#ps', DataTable).add_columns( - 'service', 'status (uptime)', 'created', 'container id', 'ports' + 'name', 'status', 'serves', 'started', 'ports' ) self.query_one('#gpus', DataTable).add_columns( 'gpu', 'name', 'util%', 'mem (used/total)', 'temp' @@ -1395,9 +1239,9 @@ def on_mount(self) -> None: '…', '(loading…)', '-', '-', '-' ) self._gpus_rows_cache = [('__loading__',) * 5] - compose_file = getattr(self.controller.backend, 'compose_file', None) + rendered = getattr(self.controller.backend, 'rendered_file', None) self.query_one('#compose-path', Static).update( - f'compose file: {compose_file or "(not rendered yet)"}' + f'rendered: {rendered or "(this backend renders no file)"}' ) # Capture docker's own chatter (up/down progress on stderr) into the # logs pane instead of letting it bleed onto the full-screen terminal. @@ -1505,25 +1349,40 @@ def _report_stall(self, length: float, stack: list[str]) -> None: # -- theming for docker output bleed ---------------------------------- def _install_quiet_docker(self) -> None: - """Route ``docker compose`` output to the logs pane, not the terminal.""" - backend = self.controller.backend - if not hasattr(backend, 'run'): - return + """Route ``docker`` output to the logs pane, not the terminal. + Only ``docker`` commands: the kubeai backend runs ``kubectl`` through + the same seam, and the Docker runner's allowlisted environment has no + ``KUBECONFIG`` (every kubectl call from the TUI used to fail). The + kubeai backend's gateway is a Compose project of its own, so it is + wrapped too. + """ from .leasing.compose import _default_docker_run - def quiet_run(args: list[str], **kwargs) -> str: - # Same bounded, explicit-environment runner as the CLI; only stderr - # is redirected to the logs pane, unless the caller takes it. - noisy = not any(a == 'ps' for a in args) - sink = ( - (lambda line: self.call_from_thread(self._append_log, line)) - if noisy else (lambda line: None) - ) - kwargs.setdefault('stderr_lines', sink) - return _default_docker_run(args, **kwargs) + def quiet(target) -> None: + original = getattr(target, 'run', None) + if original is None: + return + + def quiet_run(args: list[str], **kwargs) -> str: + if not args or args[0] != 'docker': + return original(args, **kwargs) + # Same bounded, explicit-environment runner as the CLI; only + # stderr is redirected to the logs pane, unless the caller + # takes it. + noisy = not any(a == 'ps' for a in args) + sink = ( + (lambda line: self.call_from_thread(self._append_log, line)) + if noisy else (lambda line: None) + ) + kwargs.setdefault('stderr_lines', sink) + return _default_docker_run(args, **kwargs) - backend.run = quiet_run + target.run = quiet_run + + backend = self.controller.backend + quiet(backend) + quiet(getattr(backend, 'gateway', None)) if hasattr(backend, 'progress'): backend.progress = self._backend_progress @@ -1750,8 +1609,8 @@ def _collect(self) -> dict[str, Any]: 'leases': leases, 'deployments': deployments, 'observed': self._observed, 'assignments': self._assignments, } - if not self._collapsed['docker'] and self._active_tab == 'tab-containers': - data['ps'] = self._compose_ps_rows() + if not self._collapsed['docker']: + data['instances'] = self._instance_list() if not self._collapsed['system']: data['gpus'] = self._gpu_rows() data['sysinfo'] = self._system_line() @@ -1769,8 +1628,11 @@ def _render(self, data: dict[str, Any]) -> None: self._fill_deployments( data['deployments'], data['observed'], data['assignments'], data['leases'] ) - if 'ps' in data: - self._fill_ps(data['ps']) + if 'instances' in data: + if data['instances'] is not None: + self._last_instances = data['instances'] + if self._active_tab == 'tab-containers': + self._fill_ps(self._ps_rows(data['instances'])) if 'gpus' in data: self._fill_gpus(data['gpus']) self.query_one('#sysinfo', Static).update(data.get('sysinfo', '')) @@ -1893,7 +1755,7 @@ def _update_summary(self, leases, deployments, observed) -> None: # (~7 KB/s of terminal output while nothing changed). try: docker = self.query_one('#docker', Collapsible) - title = f'docker — {running_label}' + title = f'runtime — {running_label}' if docker.title != title: docker.title = title for pane, text in ( @@ -1997,12 +1859,15 @@ def _fill_deployments(self, deployments, observed, assignments, leases) -> None: def _fill_ps(self, rows) -> None: table = self.query_one('#ps', DataTable) - new_rows = [ - (row['service'], row['status'], row['created'] or '-', - row['id'] or '-', row['ports'] or '-') - for row in rows - ] - if not rows: + if rows is None: + new_rows = [('(cannot read the runtime)', '-', '-', '-', '-')] + else: + new_rows = [ + (row['name'], row['status'], row['serves'] or '-', + row['started'] or '-', row['ports'] or '-') + for row in rows + ] + if rows is not None and not rows: new_rows = [('(nothing running)', '-', '-', '-', '-')] self._diff_fill(table, new_rows, '_ps_rows_cache', id_index=0) @@ -2103,75 +1968,34 @@ def _system_line(self) -> str: # -- docker ps --------------------------------------------------------- - def _compose_ps_rows(self) -> list[dict[str, str]]: - """Best-effort ``docker compose ps`` rows.""" - import json - - backend = self.controller.backend - path = getattr(backend, 'compose_file', None) - run = getattr(backend, 'run', None) - if not path or not run or not Path(path).exists(): - return [] - project = getattr(backend, 'project', 'infer-stack') - try: - out = run([ - 'docker', 'compose', '-p', str(project), '-f', str(path), - 'ps', '--format', 'json', - ]) - except Exception: # noqa: BLE001 - ps is best-effort - return [] - rows: list[dict[str, Any]] = [] - out = (out or '').strip() - if not out: - return [] + def _instance_list(self): + """What the backend runs (worker thread), or ``None`` if unreadable.""" try: - parsed = json.loads(out) - rows = parsed if isinstance(parsed, list) else [parsed] - except json.JSONDecodeError: - for line in out.splitlines(): - line = line.strip() - if line: - try: - rows.append(json.loads(line)) - except json.JSONDecodeError: - pass - result = [] - for row in rows: - ports = _fmt_ports(row) or str(row.get('Ports') or '') - cid = str(row.get('ID') or '')[:12] - # `Status` is the docker-ps STATUS column ("Up 3 minutes") — it - # carries the uptime; `CreatedAt`/`RunningFor` give the age. - created = str(row.get('CreatedAt') or row.get('RunningFor') or '') - result.append({ - 'service': str(row.get('Service') or row.get('Name') or '?'), - 'status': str(row.get('Status') or row.get('State') or '?'), - 'created': created, - 'id': cid, - 'ports': ports, - }) - return sorted(result, key=lambda r: r['service']) + return list(self.controller.backend.instances()) + except Exception: # noqa: BLE001 - a monitor must never crash + return None - # -- logs -------------------------------------------------------------- + def _ps_rows(self, instances) -> list[dict[str, str]] | None: + """Rows for the Instances table (same shape as ``infer-stack ps``).""" + if instances is None: + return None + served = {g.id: sorted(g.served) for g in (self._last_deployments or [])} + return [{ + 'name': i.name, + 'status': i.status, + 'serves': (', '.join(served.get(i.deployment_id, [])) + if i.deployment_id else '(front door)'), + 'started': (i.started or '')[:19].replace('T', ' '), + 'ports': i.ports, + } for i in instances] def _service_names(self) -> list[str]: - """Service names from the on-disk compose file (best-effort).""" - backend = self.controller.backend - path = getattr(backend, 'compose_file', None) - if not path: - return [] - try: - stat = Path(path).stat() - key = (stat.st_mtime, stat.st_size) - cached = self._service_names_cache - if cached is not None and cached[0] == key: - return list(cached[1]) # checked every refresh; parse on change - import yaml - data = yaml.safe_load(Path(path).read_text()) or {} - names = sorted((data.get('services') or {}).keys()) - self._service_names_cache = (key, names) - return list(names) - except Exception: # noqa: BLE001 - return [] + """Names of the instances last seen (refreshed by the worker).""" + return sorted({i.name for i in self._last_instances}) + + def _engine_names(self) -> list[str]: + """Names of the last-seen instances that serve a deployment.""" + return sorted({i.name for i in self._last_instances if i.is_engine}) def _sync_log_services(self) -> None: names = self._service_names() @@ -2180,8 +2004,8 @@ def _sync_log_services(self) -> None: self._service_options = names select = self.query_one('#logsvc', Select) options = [ - ('(engines — no litellm)', ENGINE_SERVICES), - ('(all services)', ALL_SERVICES), + ('(engines)', ENGINE_SERVICES), + ('(everything)', ALL_SERVICES), ] + [(n, n) for n in names] # Keep whatever is selected. The two sentinels are not service names, # so they have to be allowed through explicitly or refreshing the @@ -2224,13 +2048,20 @@ def on_input_changed(self, event: Input.Changed) -> None: self._update_api_curl() def _default_proc_factory(self) -> Callable[[str | None], Any]: - def factory(service: str | None): + def factory(service): + from .leasing.instances import LogFollower + backend = self.controller.backend - path = getattr(backend, 'compose_file', None) - project = getattr(backend, 'project', 'infer-stack') - if not path: - return None - return _DockerLogProc(str(project), str(path), service) + if isinstance(service, (list, tuple)): + wanted: set[str] | None = {str(s) for s in service} + else: + wanted = {str(service)} if service else None + + def listing(): + found = backend.instances() + return [i for i in found if wanted is None or i.name in wanted] + + return LogFollower(listing) return factory @@ -2264,14 +2095,14 @@ def _resolve_log_target(self, service: str): its absence, and it stays that way once engines do appear. """ if service == ENGINE_SERVICES: - names = engine_services(self._service_names()) + names = self._engine_names() if not names: return NO_LOG_TARGET, ( - 'no engine services yet; choose (all services) ' + 'no engines running yet; choose (everything) ' 'for the gateway') return names, f'engines: {", ".join(names)}' if not service: - return None, 'all services' + return None, 'everything' return service, service def _stop_log_proc(self) -> None: @@ -2295,7 +2126,7 @@ def _stream_logs(self, service, generation: int) -> None: if proc is None: self.call_from_thread( self._append_log_if_current, generation, - '(no compose project yet — acquire a model)' + '(nothing to follow — acquire a model)' ) return if generation != self._log_generation: @@ -2863,14 +2694,10 @@ def action_cleanup(self) -> None: # -- docker compose control ------------------------------------------- - def _compose_target(self) -> tuple[Any, str, str] | None: - """(run, project, compose_file) for the leasing project, or None.""" - backend = self.controller.backend - path = getattr(backend, 'compose_file', None) - run = getattr(backend, 'run', None) - if not path or not run or not Path(path).exists(): - return None - return run, str(getattr(backend, 'project', 'infer-stack')), str(path) + def _compose_target(self): + """The backend's rendered file, when a render exists; else ``None``.""" + rendered = getattr(self.controller.backend, 'rendered_file', None) + return rendered if rendered is not None and Path(rendered).exists() else None def action_compose_up(self) -> None: if self._compose_target() is None: @@ -2897,22 +2724,17 @@ def action_compose_down(self) -> None: if self._compose_target() is None: self._refuse('nothing rendered yet — nothing to bring down') return - self._status('docker compose down (raw: bypasses leases; releases nothing)…') + self._status('down (bypasses leases; releases nothing)…') self._cli(cli.command('stack', 'down')) - self._do_compose(['down', '--remove-orphans'], 'down') + self._do_down() @work(thread=True, exclusive=True, group='mutate') - def _do_compose(self, args: list[str], label: str) -> None: - target = self._compose_target() - if target is None: - self._after_mutation('nothing rendered yet') - return - run, project, path = target + def _do_down(self) -> None: try: - run(['docker', 'compose', '-p', project, '-f', path, *args]) - msg = f'compose {label} done' + self.controller.backend.down() + msg = 'down done' except Exception as ex: # noqa: BLE001 - msg = f'compose {label} failed: {ex}' + msg = f'down failed: {ex}' self._after_mutation(msg) # -- open in browser --------------------------------------------------- @@ -3589,18 +3411,6 @@ def _parse_kv_str(text: str) -> dict[str, Any]: return out -def _fmt_ports(row: dict) -> str: - """Compact published-ports string from a compose ps JSON row.""" - pubs = row.get('Publishers') or [] - bits = [] - for pub in pubs: - published = pub.get('PublishedPort') - target = pub.get('TargetPort') - if published: - bits.append(f'{published}->{target}') - return ', '.join(bits) - - def _seconds_since_process_start() -> float | None: """Wall time since this process started (Linux ``/proc``), else ``None``.""" import os diff --git a/tests/test_cli_meta.py b/tests/test_cli_meta.py index d843cb8e..08030a34 100644 --- a/tests/test_cli_meta.py +++ b/tests/test_cli_meta.py @@ -139,15 +139,17 @@ def test_render_rich_colorizes_status() -> None: assert '\x1b[' in out -def test_day2_compose_base_prefers_leasing(tmp_path: Path, monkeypatch) -> None: - from infer_stack.cli.commands_runtime import _day2_compose_base +def test_day2_compose_verbs_target_the_backends_project(tmp_path: Path, monkeypatch) -> None: + from types import SimpleNamespace + + from infer_stack.cli.commands_runtime import _compose_argv monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path)) compose_file = tmp_path / 'leasing' / 'compose' / 'docker-compose.yml' compose_file.parent.mkdir(parents=True) compose_file.write_text('services: {}\n') - # leasing compose present -> targets it without needing config.yaml - base = _day2_compose_base(None, 'logs') + # The compose backend's own project; no config.yaml needed. + base = _compose_argv(SimpleNamespace(backend='compose')) assert base == [ 'docker', 'compose', '-p', 'infer-stack', '-f', str(compose_file) ] diff --git a/tests/test_day2.py b/tests/test_day2.py new file mode 100644 index 00000000..92fbe895 --- /dev/null +++ b/tests/test_day2.py @@ -0,0 +1,204 @@ +"""Day-2 verbs (ps, logs, status, stack) read the backend, on either backend.""" + +from __future__ import annotations + +import json +from types import SimpleNamespace + +import pytest + +from infer_stack.leasing.instances import ( + DOCKER, + KUBERNETES, + MEMORY, + Instance, + UnknownTarget, + follow_argv, + from_residency, + history_argv, + resolve, +) +from infer_stack.leasing.residency import Container, Residency, ResidencyUnknown + +POD = Instance('model-qwen-0-abc', 'model-qwen-0-abc', 'grp-1', 'running', + runtime=KUBERNETES, namespace='kubeai') +GATEWAY = Instance('litellm', 'c0ffee1234', '', 'running', ports='14042->4000/tcp') + + +def test_log_commands_follow_the_runtime(): + assert history_argv(POD, tail=50) == [ + 'kubectl', '-n', 'kubeai', 'logs', '--tail', '50', 'model-qwen-0-abc'] + assert follow_argv(POD) == [ + 'kubectl', '-n', 'kubeai', 'logs', '--tail', '0', 'model-qwen-0-abc', '--follow'] + assert history_argv(GATEWAY, tail='all', timestamps=True) == [ + 'docker', 'logs', '--tail', 'all', '--timestamps', 'c0ffee1234'] + assert follow_argv(GATEWAY)[:2] == ['docker', 'attach'] + memory = Instance('grp-1', 'grp-1', 'grp-1', 'running', runtime=MEMORY) + assert history_argv(memory) is None and follow_argv(memory) is None + + +def test_instances_come_from_residency_engines_first(): + residency = Residency( + by_deployment={'grp-1': (Container('pod-1', 'grp-1', 'restarting', + restart_count=3, reason='CrashLoopBackOff'),)}, + others=(Container('gw', '', 'running', service='litellm'),), + ) + pods = from_residency(residency, runtime=KUBERNETES, namespace='kubeai') + assert [i.name for i in pods] == ['pod-1', 'gw'] # pods are named by pod + assert pods[0].status == 'restarting (CrashLoopBackOff, 3 restarts)' + containers = from_residency(residency, runtime=DOCKER) + assert [i.name for i in containers] == ['pod-1', 'litellm'] # service names + + +def test_a_target_resolves_by_alias_deployment_name_or_id(): + served = {'grp-1': ['qwen']} + both = [POD, GATEWAY] + assert resolve(both, ['qwen'], served) == [POD] + assert resolve(both, ['grp-1', 'litellm'], served) == [POD, GATEWAY] + assert resolve(both, ['c0ff'], served) == [GATEWAY] + with pytest.raises(UnknownTarget, match='running: litellm, model-qwen-0-abc'): + resolve(both, ['nope'], served) + + +class _Backend: + """A backend as the day-2 verbs see it.""" + + def __init__(self, instances, *, residency=None): + self._instances = instances + self._residency = residency + self.down_calls = 0 + + def instances(self): + if self._instances is None: + raise ResidencyUnknown('kubectl get pods failed') + return list(self._instances) + + def residency(self): + if self._residency is None: + raise ResidencyUnknown('kubectl get pods failed') + return self._residency + + def down(self): + self.down_calls += 1 + + +def _cli(monkeypatch, backend, served=None): + from infer_stack.cli import commands_runtime as rt + + monkeypatch.setattr(rt, '_day2_backend', lambda config: backend) + monkeypatch.setattr(rt, '_served_by_deployment', lambda: dict(served or {})) + return rt + + +def test_ps_has_one_shape_and_names_what_each_serves(monkeypatch, capsys): + rt = _cli(monkeypatch, _Backend([POD, GATEWAY]), {'grp-1': ['qwen']}) + assert rt.PsCLI.main(argv=[]) == 0 + out = capsys.readouterr().out.splitlines() + assert out[0].split() == ['NAME', 'STATUS', 'SERVES', 'GPUS', 'STARTED', 'ID', 'PORTS'] + assert 'qwen' in out[1] and 'model-qwen-0-abc' in out[1] + assert '(front door)' in out[2] and '14042->4000/tcp' in out[2] + + assert rt.PsCLI.main(argv=['qwen', '--json']) == 0 + rows = json.loads(capsys.readouterr().out) + assert [r['name'] for r in rows] == ['model-qwen-0-abc'] + assert rows[0]['serves'] == ['qwen'] and rows[0]['runtime'] == KUBERNETES + + +def test_ps_hides_finished_instances_unless_all(monkeypatch, capsys): + done = Instance('old', 'dead0000', 'grp-2', 'exited') + rt = _cli(monkeypatch, _Backend([POD, done])) + rt.PsCLI.main(argv=['-q']) + assert capsys.readouterr().out.split() == ['model-qwen-0-abc'] + rt.PsCLI.main(argv=['-q', '--all']) + assert capsys.readouterr().out.split() == ['model-qwen-0-abc', 'dead0000'] + + +def test_ps_says_why_when_the_runtime_cannot_be_read(monkeypatch): + rt = _cli(monkeypatch, _Backend(None)) + with pytest.raises(SystemExit, match='cannot read what is running'): + rt.PsCLI.main(argv=[]) + + +def test_logs_reads_the_named_endpoint_through_its_runtime(monkeypatch, capsys): + import subprocess + + rt = _cli(monkeypatch, _Backend([POD, GATEWAY]), {'grp-1': ['qwen']}) + ran = [] + + def fake_run(argv, **kw): + ran.append(argv) + return SimpleNamespace(stdout=b'INFO: loaded\n', returncode=0) + + monkeypatch.setattr(subprocess, 'run', fake_run) + assert rt.LogsCLI.main(argv=['qwen', '--tail', '5']) == 0 + assert ran == [['kubectl', '-n', 'kubeai', 'logs', '--tail', '5', 'model-qwen-0-abc']] + assert capsys.readouterr().out == 'INFO: loaded\n' # one instance: no prefix + with pytest.raises(SystemExit, match="no instance matches 'nope'"): + rt.LogsCLI.main(argv=['nope']) + + +def test_status_health_comes_from_residency(monkeypatch): + from infer_stack.cli.commands_runtime import _served_models + from infer_stack.leasing import DeploymentState + + def dep(gid): + return SimpleNamespace(id=gid, state=DeploymentState.LIVE, engine='vllm', + served={gid: {'hf_model_id': 'org/m'}}, spec={}) + + residency = Residency(by_deployment={ + 'up': (Container('c1', 'up', 'running'),), + 'loading': (Container('c3', 'loading', 'running', health='starting'),), + 'crashing': (Container('c2', 'crashing', 'exited'),), + }) + rows = _served_models([dep('up'), dep('loading'), dep('crashing'), dep('gone')], + _Backend([], residency=residency)) + assert {r[0]: r[3] for r in rows} == {'up': 'up', 'loading': 'starting', + 'crashing': 'exited', 'gone': 'STALE'} + rows = _served_models([dep('up')], _Backend([], residency=None)) + assert rows[0][3] == 'unverified' + + +def test_stack_down_stops_the_backend_on_either_backend(monkeypatch): + backend = _Backend([]) + rt = _cli(monkeypatch, backend) + assert rt.StackDownCLI.main(argv=[]) == 0 + assert backend.down_calls == 1 + + +def test_stack_compose_targets_the_compose_project_on_this_host(monkeypatch, tmp_path): + from infer_stack.cli import commands_runtime as rt + + compose_file = tmp_path / 'docker-compose.yml' + project = SimpleNamespace(compose_file=compose_file, + compose_argv=lambda: ['docker', 'compose', '-p', 'gw']) + kubeai = SimpleNamespace(compose_project=lambda: project) + monkeypatch.setattr(rt, '_day2_backend', lambda config: kubeai) + with pytest.raises(SystemExit, match='nothing rendered yet'): + rt._compose_argv(SimpleNamespace()) + compose_file.write_text('services: {}\n') + assert rt._compose_argv(SimpleNamespace()) == ['docker', 'compose', '-p', 'gw'] + + monkeypatch.setattr(rt, '_day2_backend', lambda config: SimpleNamespace()) + with pytest.raises(SystemExit, match='no Compose project on this host'): + rt._compose_argv(SimpleNamespace()) + + +@pytest.mark.parametrize(('argv', 'follow', 'names'), [ + (['-f', 'qwen'], True, ['qwen']), + (['qwen', '-f'], True, ['qwen']), + (['--follow', 'qwen', 'litellm'], True, ['qwen', 'litellm']), + (['-f', 'false'], False, []), +]) +def test_a_flag_never_swallows_the_positional_after_it(argv, follow, names): + """kwconf flags take an optional value; `logs -f qwen` followed everything.""" + from infer_stack.cli.commands_runtime import LogsCLI + + config = LogsCLI.cli(argv=argv) + assert config.follow is follow and list(config.services or []) == names + + +def test_acquire_yes_before_the_endpoint_still_names_it(): + from infer_stack.cli.commands_leasing import AcquireCLI + + config = AcquireCLI.cli(argv=['--yes', 'qwen']) + assert config.yes is True and config.names == ['qwen'] diff --git a/tests/test_log_filter.py b/tests/test_log_filter.py index 46e26760..281ace38 100644 --- a/tests/test_log_filter.py +++ b/tests/test_log_filter.py @@ -262,84 +262,51 @@ def test_eof_mid_traceback_fails_open(): assert list(compact_litellm_tracebacks(original)) == original -def test_cli_follow_compaction_helper_streams_known_trace(monkeypatch): +def _follow_logs(monkeypatch, lines, *, no_color, tty=True): + """Run `infer-stack logs -f` over a fake instance whose log is ``lines``.""" from infer_stack.cli import commands_runtime + from infer_stack.leasing import instances as instances_mod + from infer_stack.leasing.instances import Instance - source = _known_segments()[0] + config = SimpleNamespace(follow=True, raw=False, no_color=no_color, tail=None, + timestamps=False, services=None) - class FakeProc: - def __init__(self): - self.stdout = iter(source) + class Backend: + def instances(self): + return [Instance('litellm', 'c1', '', 'running')] - def wait(self, timeout=None): - return 0 - - def poll(self): - return 0 + class Follower: + def __init__(self, listing, **kw): + assert [i.name for i in listing()] == ['litellm'] + self.stdout = iter(lines) def terminate(self): - raise AssertionError('completed process should not be terminated') + pass - def kill(self): - raise AssertionError('completed process should not be killed') + class Out(io.StringIO): + def isatty(self): + return tty - monkeypatch.setattr(commands_runtime.subprocess, 'Popen', lambda *a, **kw: FakeProc()) - stream = io.StringIO() + monkeypatch.setattr(commands_runtime.LogsCLI, 'cli', lambda *a, **kw: config) + monkeypatch.setattr(commands_runtime, '_day2_backend', lambda config: Backend()) + monkeypatch.setattr(commands_runtime, '_served_by_deployment', lambda: {}) + monkeypatch.setattr(instances_mod, 'LogFollower', Follower) + stream = Out() monkeypatch.setattr(commands_runtime.sys, 'stdout', stream) + assert commands_runtime.LogsCLI.main(argv=False) == 0 + return stream.getvalue() - rc = commands_runtime._run_compacted_follow(['docker', 'compose', 'logs', '-f']) - assert rc == 0 - assert 'Traceback (most recent call last):' not in stream.getvalue() - assert 'ConnectionRefusedError: [Errno 111]' in stream.getvalue() - -@pytest.mark.parametrize(('no_color', 'expected_ansi'), [ - (False, True), - (True, False), -]) -def test_cli_compacted_follow_preserves_compose_color_mode( - monkeypatch, no_color, expected_ansi -): - from infer_stack.cli import commands_runtime - - config = SimpleNamespace( - follow=True, - raw=False, - no_color=no_color, - tail=None, - timestamps=False, - services=None, - ) - - class TTY(io.StringIO): - def isatty(self): - return True - - captured = {} - monkeypatch.setattr( - commands_runtime.LogsCLI, 'cli', lambda *args, **kwargs: config - ) - monkeypatch.setattr( - commands_runtime, - '_day2_compose_base', - lambda config, purpose: [ - 'docker', 'compose', '-p', 'infer-stack', '-f', 'compose.yml' - ], - ) - def fake_follow(cmd): - captured['cmd'] = cmd - return 0 +def test_cli_follow_compacts_a_known_trace(monkeypatch): + lines = [line.replace('litellm-1 | ', 'litellm | ') for line in _known_segments()[0]] + out = _follow_logs(monkeypatch, lines, no_color=True) + assert 'Traceback (most recent call last):' not in out + assert 'ConnectionRefusedError: [Errno 111]' in out - monkeypatch.setattr(commands_runtime, '_run_compacted_follow', fake_follow) - monkeypatch.setattr(commands_runtime.sys, 'stdout', TTY()) - assert commands_runtime.LogsCLI.main(argv=False) == 0 - cmd = captured['cmd'] - logs_index = cmd.index('logs') - assert ('--ansi' in cmd) is expected_ansi - if expected_ansi: - ansi_index = cmd.index('--ansi') - assert cmd[ansi_index + 1] == 'always' - assert ansi_index < logs_index - assert ('--no-color' in cmd) is no_color +@pytest.mark.parametrize('no_color', [False, True]) +def test_cli_follow_colors_prefixes_unless_asked_not_to(monkeypatch, no_color): + out = _follow_logs(monkeypatch, ['litellm | hello\n'], no_color=no_color) + assert ('\x1b[' in out) is (not no_color) + assert 'hello' in out diff --git a/tests/test_tui.py b/tests/test_tui.py index e1557e87..ecccb8b1 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -3,7 +3,6 @@ from __future__ import annotations import asyncio -import json import pytest @@ -253,7 +252,7 @@ async def scenario(): proc_factory=lambda svc: None) async with app.run_test() as pilot: await pilot.pause() - # docker is a collapsible pane with Logs/Containers/Control tabs; + # runtime is a collapsible pane with Logs/Instances/Control tabs; # system is its own collapsed pane; API is a top-level tab now. tabs = app.query_one('#docker-tabs', TabbedContent) assert {p.id for p in tabs.query('TabPane')} == { @@ -263,12 +262,10 @@ async def scenario(): assert app.query_one('#system', Collapsible).collapsed top = app.query_one('#top', TabbedContent) assert 'tab-api' in {p.id for p in top.query('TabPane')} - # the containers ps view carries the docker-ps columns + # the instances view carries what `infer-stack ps` shows ps = app.query_one('#ps', DataTable) labels = [str(c.label) for c in ps.columns.values()] - assert 'status (uptime)' in labels - assert 'created' in labels - assert 'container id' in labels + assert labels == ['name', 'status', 'serves', 'started', 'ports'] _run(scenario) @@ -915,35 +912,33 @@ async def scenario(): _run(scenario) -def test_tui_ps_rows_parse_status_created_and_id(): +def test_tui_instance_rows_say_what_each_serves(): + from types import SimpleNamespace + + from infer_stack.leasing.instances import Instance from infer_stack.tui import InferStackTUI controller, catalog = _ctx() - sample = json.dumps([ - {'Service': 'litellm', 'Status': 'Up 3 minutes', 'State': 'running', - 'CreatedAt': '2026-06-19 00:00:00 -0400', 'ID': 'abcdef1234567890', - 'Publishers': [{'PublishedPort': 14042, 'TargetPort': 4000}]}, - ]) + instances = [ + Instance('vllm-qwen', 'abcdef1234567890', 'grp-1', 'restarting', + restarts=2, reason='CrashLoopBackOff', + started='2026-06-19T00:00:00.123Z'), + Instance('litellm', 'fedcba', '', 'running', ports='14042->4000/tcp'), + ] async def scenario(): app = InferStackTUI(controller, catalog, interval=999, proc_factory=lambda svc: None) async with app.run_test() as pilot: await pilot.pause() - # stub the backend's compose seam - backend = controller.backend - import tempfile - f = tempfile.NamedTemporaryFile('w', suffix='.yml', delete=False) - f.write('services: {}\n') - f.close() - backend.compose_file = f.name - backend.project = 'infer-stack' - backend.run = lambda args: sample - rows = app._compose_ps_rows() - assert rows[0]['status'] == 'Up 3 minutes' - assert rows[0]['created'].startswith('2026-06-19') - assert rows[0]['id'] == 'abcdef123456' # truncated to 12 - assert '14042->4000' in rows[0]['ports'] + app._last_deployments = [SimpleNamespace(id='grp-1', served={'qwen': {}})] + engine, gateway = app._ps_rows(instances) + assert engine['serves'] == 'qwen' + assert engine['status'] == 'restarting (CrashLoopBackOff, 2 restarts)' + assert engine['started'] == '2026-06-19 00:00:00' + assert gateway['serves'] == '(front door)' + assert gateway['ports'] == '14042->4000/tcp' + assert app._ps_rows(None) is None # the runtime was unreadable _run(scenario) @@ -980,11 +975,11 @@ async def scenario(): await pilot.pause() app._active_tab = 'tab-containers' app._collapsed['docker'] = True - assert 'ps' not in app._collect() # collapsed -> no ps poll + assert 'instances' not in app._collect() # collapsed -> no poll app._collapsed['docker'] = False - assert 'ps' in app._collect() # visible -> polled + assert 'instances' in app._collect() # visible -> polled app._active_tab = 'tab-logs' - assert 'ps' not in app._collect() # other tab -> no ps poll + assert 'instances' in app._collect() # the log picker needs names app._collapsed['system'] = True assert 'gpus' not in app._collect() # system collapsed app._collapsed['system'] = False @@ -1441,8 +1436,7 @@ async def scenario(): async with app.run_test() as pilot: await pilot.pause() backend = controller.backend - backend.compose_file = compose_file - backend.project = 'infer-stack' + backend.rendered_file = compose_file backend.run = lambda args: calls.append(args) or '' app.action_compose_up() await app.workers.wait_for_complete() @@ -1704,27 +1698,22 @@ def test_gateway_services_are_excluded_from_the_default_log_view(): """The logs pane defaults to engines, not everything. LiteLLM logs a line per proxied request, so on a busy host it scrolls the - engine output -- where errors actually appear -- out of the pane. + engine output -- where errors actually appear -- out of the pane. An + engine is an instance that serves a deployment; the gateway, UI, database + and proxy serve none, whatever they are named (a name hint used to let + Open WebUI and Postgres into the engines view). """ - from infer_stack.tui import ( - ALL_SERVICES, - ENGINE_SERVICES, - engine_services, - is_gateway_service, - ) + from infer_stack.leasing.instances import Instance + from infer_stack.tui import ALL_SERVICES, ENGINE_SERVICES - assert is_gateway_service('litellm') - # Substring, not equality: a suffixed gateway service must still match, or - # the noisy view comes back silently. - assert is_gateway_service('infer-stack-litellm-1') - assert not is_gateway_service('vllm-qwen-qwen3-8-27b') - - names = ['litellm', 'vllm-a', 'vllm-b'] - assert engine_services(names) == ['vllm-a', 'vllm-b'] + engine = Instance('vllm-a', 'c1', 'grp-1', 'running') + assert engine.is_engine + for name in ('litellm', 'infer-stack-litellm-1', 'open-webui', 'postgres'): + assert not Instance(name, 'c2', '', 'running').is_engine # The two sentinels must stay distinguishable from each other and from any - # real service name. + # real instance name. assert ENGINE_SERVICES != ALL_SERVICES - assert ENGINE_SERVICES not in names + assert ENGINE_SERVICES not in ('litellm', 'vllm-a') def test_named_log_process_follows_only_that_service(monkeypatch): @@ -1733,7 +1722,8 @@ def test_named_log_process_follows_only_that_service(monkeypatch): import subprocess import time - from infer_stack.tui import _DockerLogProc + from infer_stack.leasing.instances import Instance + from infer_stack.tui import InferStackTUI service = 'vllm-qwen3-8-27b-dbirks-hyperqwen' touched = [] @@ -1743,8 +1733,6 @@ def __init__(self, stdout): self.stdout = stdout def fake_run(cmd, **kwargs): - if cmd[:2] == ['docker', 'ps']: - return _Done(f'c1 running {service}\nc2 running litellm\n') touched.append(cmd[-1]) # docker logs --tail N return _Done(b'') @@ -1764,9 +1752,15 @@ def terminate(self): def wait(self, timeout=None): return 0 + controller, catalog = _ctx() + controller.backend.instances = lambda: [ + Instance(service, 'c1', 'grp-1', 'running'), + Instance('litellm', 'c2', '', 'running'), + ] monkeypatch.setattr(subprocess, 'run', fake_run) monkeypatch.setattr(subprocess, 'Popen', lambda cmd, **kw: _Proc(cmd)) - proc = _DockerLogProc('infer-stack', '/tmp/docker-compose.yml', service) + app = InferStackTUI(controller, catalog, interval=999) + proc = app._default_proc_factory()(service) deadline = time.monotonic() + 5 while len(touched) < 2 and time.monotonic() < deadline: time.sleep(0.05) @@ -1811,7 +1805,10 @@ async def scenario(): proc_factory=lambda svc: None) async with app.run_test() as pilot: await pilot.pause() - app._service_names = lambda: ['litellm', 'vllm-a', 'vllm-b'] + from infer_stack.leasing.instances import Instance + app._last_instances = [Instance('litellm', 'c0', '', 'running'), + Instance('vllm-a', 'c1', 'g1', 'running'), + Instance('vllm-b', 'c2', 'g2', 'running')] target, label = app._resolve_log_target(ENGINE_SERVICES) assert target == ['vllm-a', 'vllm-b'] @@ -1820,16 +1817,16 @@ async def scenario(): # A named service is passed straight through. assert app._resolve_log_target('litellm') == ('litellm', 'litellm') - # "all" stays None so docker compose logs gets no service argument. + # "everything" stays None: the follower takes every instance. target, label = app._resolve_log_target(ALL_SERVICES) - assert target is None and label == 'all services' + assert target is None and label == 'everything' # With no engines there is nothing to follow. Falling back to - # every service would show the gateway under the engines label. + # every instance would show the gateway under the engines label. from infer_stack.tui import NO_LOG_TARGET - app._service_names = lambda: ['litellm'] + app._last_instances = [Instance('litellm', 'c0', '', 'running')] target, label = app._resolve_log_target(ENGINE_SERVICES) - assert target is NO_LOG_TARGET and 'no engine services' in label + assert target is NO_LOG_TARGET and 'no engines' in label _run(scenario) @@ -1855,15 +1852,16 @@ async def scenario(): proc_factory=factory) async with app.run_test() as pilot: await pilot.pause() - services = ['litellm'] - app._service_names = lambda: list(services) + from infer_stack.leasing.instances import Instance + app._last_instances = [Instance('litellm', 'c0', '', 'running')] app._collapsed['docker'] = False app._sync_log_services() app._restart_logs(app._log_service) # the pane opens await pilot.pause(0.2) assert started == [] # nothing to follow yet - services.append('vllm-a') # an engine is deployed + app._last_instances = [*app._last_instances, # an engine is deployed + Instance('vllm-a', 'c1', 'g1', 'running')] app._sync_log_services() await pilot.pause(0.2) await pilot.pause() From 46b49b7a2cd49adcb4a34068d2c7df26e19cf6ac Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Sat, 26 Sep 2026 11:55:41 -0400 Subject: [PATCH 28/34] Queue: record the P2 commit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- docs/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/queue.md b/docs/queue.md index d5e57cb5..cf6c75e7 100644 --- a/docs/queue.md +++ b/docs/queue.md @@ -44,7 +44,7 @@ queue-semantics tests still assert the same behaviour. ### 3. [x] P2: day-2 commands and the TUI through the backend -Done 2026-09-26. +Done 2026-09-26, `c2d393b`. `instances()` and `stream_logs(target, *, follow, tail)` on both backends; `ps`, `logs`, `stack up` / `stack down` use them, the raw compose form From 7a322c1fb1cfb842fc958569b5198f7fb74e4096 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Sat, 26 Sep 2026 12:19:37 -0400 Subject: [PATCH 29/34] P3: the gateway in front of a cluster is the compose gateway The kubeai gateway project takes the compose front-door settings (ui, reverse_proxy, dynamic_routing), recorded in the recovery profile as the gateway project's own profile. The kubeai backend hands the gateway explicit render inputs (registry rows for the catalog and every Model, or per-deployment dynamic routes via the shared upstream_route) instead of writing rows into the registry before approval, so the gateway's diff is previewed and approved with the acquire's. Under dynamic routing each deployment is its own Model, named with compose's deployment tail. Duplicate authorities removed: a route's upstream (render vs routes list), a deployment's Model name (five places), and isinstance(ComposeBackend) checks in routes, gc/clean --orphans, network migrate and secrets rotate, now the capability each needs. Verified: full suite; compose e2e non-GPU tiers; dev/kubeai_e2e.sh on k3s with new steps for routes list, Open WebUI, and two --dedicated Models behind one alias under dynamic routing. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 13 +++ dev/kubeai_e2e.sh | 53 +++++++++ docs/backend-parity.md | 7 +- docs/kubeai-backend.md | 7 +- docs/planning/backend-parity-roadmap.md | 20 +++- docs/queue.md | 4 +- infer_stack/backends/kubeai.py | 134 +++++++++++++++++++---- infer_stack/cli/commands_leasing.py | 102 +++++++++--------- infer_stack/leasing/compose.py | 28 ++++- infer_stack/leasing/gateway.py | 83 ++++++++------ infer_stack/leasing/naming.py | 12 ++- tests/test_leasing_kubeai.py | 138 +++++++++++++++++++++++- 12 files changed, 475 insertions(+), 126 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aacb24f9..1bde84cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### The gateway in front of a cluster is the compose gateway + +On the kubeai backend the gateway now takes the same settings as on compose: +Open WebUI (`ui`, on by default), the reverse proxy, and dynamic routing, +under which each deployment is its own Model (`-`), so the +same model acquired `--dedicated` twice runs as two Models behind one alias. +Catalog endpoints are routed before they run, as on compose, so a new Model +no longer recreates the gateway. The gateway's changes are shown and +approved with the acquire's, before its lease commits. `routes list / seed / +prune` work on kubeai and show a cluster route's upstream (they showed `?`), +and `gc --orphans` and `clean --orphans` run there (and find nothing: the +backend only reads its own labelled pods). + ### `ps`, `logs`, `status` and the TUI read the backend, on either backend `infer-stack ps` lists what the backend runs, containers or pods and the diff --git a/dev/kubeai_e2e.sh b/dev/kubeai_e2e.sh index f735e1e2..e106ee75 100755 --- a/dev/kubeai_e2e.sh +++ b/dev/kubeai_e2e.sh @@ -24,6 +24,9 @@ # GATEWAY 1 (default): clients go through infer-stack's LiteLLM # gateway, as on the compose backend. 0: straight to # KubeAI, where the alias below does NOT route (404). +# E2E_DYNAMIC 1 (default): finish with dynamic routing, two +# --dedicated leases of one model (two Models at once). +# E2E_UI_PORT Open WebUI's port on this host (default 13000). set -euo pipefail MODEL="${E2E_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}" @@ -45,6 +48,7 @@ cleanup() { set +e run_is release --all --yes >/dev/null 2>&1 run_is gc --evict --yes >/dev/null 2>&1 + run_is stack down >/dev/null 2>&1 # the gateway, UI and database too # nothing managed may remain on the cluster, pass or fail leftover=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ -l infer-stack/managed=true -o name 2>/dev/null | wc -l) @@ -103,6 +107,26 @@ else exit 1 fi +if [ "${GATEWAY:-1}" = 1 ]; then + echo '== the gateway fronts the cluster as on compose: routes, Open WebUI' + if ! run_is routes list --json | python3 -c ' +import json, sys +alias = sys.argv[1] +row = next(r for r in json.load(sys.stdin)["routes"] if r["name"] == alias) +assert row["engine"] == "upstream" and row["live"], row +print(" routes list:", alias, "->", row["target"], "at", row["upstream"])' "$ALIAS"; then + echo '!! routes list does not route the alias to the cluster' >&2; exit 1 + fi + ui='' + for _ in $(seq 60); do + ui=$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1:${E2E_UI_PORT:-13000}/" || true) + [ "$ui" = 200 ] && break + sleep 2 + done + [ "$ui" = 200 ] || { echo "!! Open WebUI did not answer (HTTP $ui)" >&2; exit 1; } + echo ' Open WebUI answers in front of the cluster' +fi + echo '== release prunes the Model (reclaim: stop)' run_is release --env-file "$WORK/lease.env" --yes remaining=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ @@ -161,4 +185,33 @@ EOF run_is release --env-file "$WORK/big.env" --yes fi +if [ "${GATEWAY:-1}" = 1 ] && [ "${E2E_DYNAMIC:-1}" = 1 ]; then + echo '== dynamic routing: two --dedicated leases on one model, two Models, one alias' + run_is stack down >/dev/null 2>&1 # the gateway comes back with Postgres + run_is config set dynamic_routing true + for n in 1 2; do + run_is acquire "$ALIAS" --dedicated --yes --ttl 30m --timeout "$TIMEOUT" \ + --env-file "$WORK/dyn$n.env" + done + models=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ + -l infer-stack/managed=true -o name | wc -l) + [ "$models" = 2 ] || { echo "!! $models Model(s), expected 2" >&2; exit 1; } + # shellcheck disable=SC1090 + source "$WORK/dyn1.env" + routes=$(curl -s "$OPENAI_BASE_URL/model/info" -H "Authorization: Bearer $OPENAI_API_KEY" \ + | python3 -c 'import json,sys; print(sum(m["model_name"] == sys.argv[1] for m in json.load(sys.stdin)["data"]))' "$ALIAS") + [ "$routes" = 2 ] || { echo "!! $routes route(s) for $ALIAS, expected 2" >&2; exit 1; } + if curl -sS --fail-with-body "$OPENAI_BASE_URL/chat/completions" \ + -H "Authorization: Bearer $OPENAI_API_KEY" -H 'Content-Type: application/json' \ + -d "{\"model\": \"$ALIAS\", \"max_tokens\": 4, + \"messages\": [{\"role\": \"user\", \"content\": \"say ok\"}]}" \ + | grep -q 'choices'; then + echo ' two Models, two routes under one alias, and it answers' + else + echo "!! no generation for $ALIAS under dynamic routing" >&2; exit 1 + fi + run_is release --env-file "$WORK/dyn1.env" --yes + run_is release --env-file "$WORK/dyn2.env" --yes +fi + echo 'PASS: kubeai backend end-to-end lifecycle' diff --git a/docs/backend-parity.md b/docs/backend-parity.md index 711ccd42..9f399cc4 100644 --- a/docs/backend-parity.md +++ b/docs/backend-parity.md @@ -107,9 +107,10 @@ a different mechanism. **gap**: missing on one side and on the roadmap. | | Compose | KubeAI | |---|---|---| | static superset routes, no blip on model churn | same | same | -| `routes` inspect / seed / prune | yes | **gap** (P3): refused, although the registry exists | -| `dynamic_routing` (admin API + Postgres; distinct upstreams for same-model `--dedicated`) | yes | **gap** (P3) | -| Open WebUI (`ui`), reverse proxy | yes | **gap** (P3): the gateway project is rendered with `ui` off | +| `routes` inspect / seed / prune | yes | same; a route points at the cluster under the Model's name | +| `dynamic_routing` (admin API + Postgres; distinct upstreams for same-model `--dedicated`) | yes | same: each deployment is its own Model (`-`) | +| Open WebUI (`ui`), reverse proxy | yes | same, in the gateway's project on this host | +| the gateway's changes approved with the acquire's, before the lease commits | yes | same | | `network migrate` / `network check` | yes | n/a: Service addresses are stable | | where the gateway runs | this host | this host, so it is in every request's path (P5 moves it into the cluster) | diff --git a/docs/kubeai-backend.md b/docs/kubeai-backend.md index 48e418ef..8cf548f1 100644 --- a/docs/kubeai-backend.md +++ b/docs/kubeai-backend.md @@ -148,5 +148,8 @@ infer-stack release --env-file lease.env as on compose. `stack compose …` and `stack restart` act on the gateway's Compose project; the engines are pods, restarted by the kubelet. - The gateway runs on the host running infer-stack, so that host is in every - request's path. Dynamic routing (`dynamic_routing`) is compose-only; the - KubeAI gateway uses static routes. + request's path. It takes the same settings as on compose: `ui` (Open WebUI, + on by default), `reverse_proxy`, and `dynamic_routing`, under which each + deployment is its own Model (`-`), so `--dedicated` twice + gives two Models behind one alias. The gateway's config changes are shown + and approved with the acquire's, before its lease commits. diff --git a/docs/planning/backend-parity-roadmap.md b/docs/planning/backend-parity-roadmap.md index 4d05d473..6eacb09a 100644 --- a/docs/planning/backend-parity-roadmap.md +++ b/docs/planning/backend-parity-roadmap.md @@ -1,7 +1,7 @@ # Backend parity roadmap: KubeAI as a superset of Compose **Status:** proposed 2026-09-25 · **P0 done** 2026-09-24 on -`dev/backend-unification` · **P1, P2 done** 2026-09-26 · P3–P5 not started · +`dev/backend-unification` · **P1–P3 done** 2026-09-26 · P4, P5 not started · P6 is ongoing. Execution order: [../queue.md](../queue.md). **Current state:** [../backend-parity.md](../backend-parity.md). **Origin:** the scale-up run needs more than one workstation, and the @@ -155,6 +155,18 @@ step that acquires the same model `--dedicated` twice under dynamic routing and gets two Models and two routes. **Size:** small to medium; the `Gateway` class already owns the pieces. +**P3 done 2026-09-26.** The KubeAI gateway takes the compose settings (`ui`, +`reverse_proxy`, `dynamic_routing`) and records them in the recovery profile +as the gateway project's own profile. The kubeai backend hands the gateway +explicit render inputs (registry rows for the catalog and every Model, or +per-deployment dynamic routes) instead of writing rows into the registry +before approval, so the gateway's changes are previewed and approved with +the acquire's. Under dynamic routing each deployment is its own Model, +named with the deployment tail compose uses for its services. The `routes` +commands resolve the backend's Compose project and ask the backend for its +rows. Verified by `dev/kubeai_e2e.sh` on k3s: routes, Open WebUI, and two +`--dedicated` Models behind one alias. + ### P4. Placement information parity (optional) **Closes:** `min_vram_gib` / `measure`; less hand-written cluster @@ -218,7 +230,11 @@ worse. A blocker is fixed whatever its size. | engine vs gateway in the TUI | a `litellm` name hint (Open WebUI and Postgres counted as engines) | **fixed** (P2): an instance serves a deployment or it does not | | where the day-2 verbs find the Compose project | a hard-coded path and project name | **fixed** (P2): the backend's `compose_project()` and `compose_argv()` | | how the TUI runs a runtime command | it replaced the backend's runner with Docker's, whose allowlisted environment has no `KUBECONFIG`: every kubectl call from the TUI failed | **fixed** (P2): only `docker` commands are wrapped | -| the KubeAI gateway's approval | the gateway project asks its own diff approval at render, after the lease commits, not in the admission preview | deferred to P3. Same result under `--yes`; interactively, a declined gateway change rolls the lease back after the commit | +| the KubeAI gateway's approval | the gateway project asked its own diff approval at render, after the lease committed | **fixed** (P3): previewed and approved with the acquire's | +| KubeAI's route rows | written into the registry before any approval, and only for live Models (so every new Model recreated the gateway) | **fixed** (P3): render inputs, persisted after approval; the catalog's rows too, so no blip | +| a route's upstream | derived twice, in the render and in `routes list` (which showed `?` for a cluster route) | **fixed** (P3): `registry_route_entry` | +| a deployment's Model name | derived in five places | **fixed** (P3): `model_name()` | +| "is this the compose backend" in the CLI | `isinstance(…, ComposeBackend)` for `routes`, `gc --orphans`, `clean`, `network migrate`, `secrets rotate` | **fixed** (P3): the capability each needs (`compose_project()`, residency labels, `network`, `litellm`) | ## Not in scope diff --git a/docs/queue.md b/docs/queue.md index cf6c75e7..8bdd3b36 100644 --- a/docs/queue.md +++ b/docs/queue.md @@ -56,7 +56,9 @@ memory line reaches the pod log. with the same output shape as on Compose, and the TUI follows a Model's log and lists its pod (checked in a real terminal, not only in tests). -### 4. [ ] P3: gateway feature parity +### 4. [x] P3: gateway feature parity + +Done 2026-09-26. `routes` and `secrets rotate` resolve the backend's gateway instead of checking its kind; the KubeAI gateway project honours `ui`, diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index d3117e99..1852860c 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -53,14 +53,32 @@ DEFAULT_BASE_URL = 'http://127.0.0.1:8000/openai/v1' -def model_name_for(served: str) -> str: +def model_name_for(served: str, deployment_id: str | None = None) -> str: """Deterministic Model CR name for a served model name: ````. Derived purely from the served name (like the compose service name), so it is identical across releases/re-acquires of the same endpoint — the request - name clients use through the KubeAI gateway never changes. + name clients use through the KubeAI gateway never changes. With + ``deployment_id`` (dynamic routing), the deployment's tail is appended, as + for a compose service, so same-model ``--dedicated`` deployments are + separate Models; the gateway addresses each by name. + + >>> model_name_for('Qwen/Qwen3-8B') + 'qwen-qwen3-8b' + >>> model_name_for('Qwen/Qwen3-8B', 'grp-0123456789ab') + 'qwen-qwen3-8b-01234567' """ - return dns_slug(served) + if deployment_id is None: + return dns_slug(served) + from ..leasing.naming import deployment_tail + + tail = deployment_tail(deployment_id) + return f'{dns_slug(served)[:62 - len(tail)].rstrip("-")}-{tail}' + + +def model_name(deployment: Deployment, *, unique: bool = False) -> str: + """The Model CR name for ``deployment``: the one place it is derived.""" + return model_name_for(_served_name(deployment), deployment.id if unique else None) def _served_name(deployment: Deployment) -> str: @@ -81,6 +99,7 @@ def _model_doc( *, namespace: str, resource_profile: str, + name: str | None = None, ) -> dict[str, Any]: """Build one KubeAI ``Model`` CR for a vLLM deployment. @@ -90,7 +109,7 @@ def _model_doc( engine here too. ``served_model_name`` is overridden to the CR name so the gateway's request name and vLLM's served name agree. """ - name = model_name_for(_served_name(deployment)) + name = name or model_name(deployment) svc = vllm_service_dict(deployment) svc['served_model_name'] = name profile = resource_profile @@ -159,6 +178,7 @@ def render_models( *, namespace: str, default_resource_profile: str | None, + unique_names: bool = False, ) -> RenderedModels: """Render the desired set into KubeAI ``Model`` docs (pure, no I/O). @@ -208,7 +228,7 @@ def render_models( 'kubeai_resource_profile ` as the default.' ) continue - name = model_name_for(_served_name(deployment)) + name = model_name(deployment, unique=unique_names) if name in out.models: out.unrenderable.add(deployment.id) out.errors.append( @@ -222,6 +242,7 @@ def render_models( deployment, namespace=namespace, resource_profile=str(profile), + name=name, ) ) out.models[name] = deployment.id @@ -383,19 +404,68 @@ def _upstream_url(self) -> str: self.gateway_upstream = f'http://{ip}/openai/v1' return self.gateway_upstream.rstrip('/') - def _render_gateway(self, rendered: RenderedModels) -> None: - """Route each endpoint alias through the gateway to its Model.""" - from ..leasing.gateway import UPSTREAM_ROUTE + @property + def dynamic_routing(self) -> bool: + """The gateway manages routes live, so each deployment is its own Model.""" + return bool(self.gateway is not None and getattr(self.gateway, 'dynamic_routing', False)) + + def catalog_route_rows(self, catalog) -> dict[str, dict[str, Any]]: + """Registry rows sending each vLLM catalog endpoint to its Model. + The static superset, as on compose: the gateway's config then stays + byte-stable as models come and go, so it is never recreated for one. + """ + from ..leasing.gateway import UPSTREAM_ROUTE, _registry_incoming_from_catalog + + if catalog is None or self.gateway is None: + return {} base = self._upstream_url() - # This backend's route rows, persisted like any other (the registry is - # append-only, so a released Model stays routable, as on compose). - self.gateway.merge_route_registry({ + return { + alias: {'engine': UPSTREAM_ROUTE, + 'served': model_name_for(row.get('served') or alias), + 'api_base': base} + for alias, row in _registry_incoming_from_catalog(catalog).items() + if row.get('engine') == 'vllm' + } + + def route_rows(self, desired: list[Deployment], placement=None) -> dict[str, dict[str, Any]]: + """The registry rows a render of ``desired`` merges (``routes prune`` + keeps exactly these): the catalog's, and every rendered Model's.""" + return self._front_door_inputs(desired, self._render_documents(desired)[1])[0] + + def _front_door_inputs(self, desired, rendered: RenderedModels): + """``(registry rows, dynamic routes)`` for the gateway, from a render. + + Static routing: one row per alias, to the Model serving it. Dynamic + routing: one route per (deployment, endpoint), each to its own Model, + so same-model ``--dedicated`` Models share the alias and LiteLLM + balances across them. + """ + from ..leasing.gateway import UPSTREAM_ROUTE, upstream_route + + if self.gateway is None: + return {}, [] + base = self._upstream_url() + if self.dynamic_routing: + by_id = {g.id: g for g in desired} + routes = [ + upstream_route(gid, endpoint, name, base) + for name, gid in sorted(rendered.models.items()) + for endpoint in sorted(by_id[gid].served) + ] + return {}, routes + rows = self.catalog_route_rows(self.catalog) + rows.update({ alias: {'engine': UPSTREAM_ROUTE, 'served': name, 'api_base': base} for alias, name in rendered.request_names.items() }) - # Gateway only: no engines on this host, so nothing to place. - self.gateway.converge([], apply=False) + return rows, [] + + def _set_front_door(self, desired, rendered: RenderedModels) -> None: + """Hand the gateway its inputs for the next render or preview.""" + rows, routes = self._front_door_inputs(desired, rendered) + self.gateway.upstream_rows = rows + self.gateway.upstream_routes = routes # -- converge-style surface ------------------------------------------------ @@ -415,6 +485,7 @@ def _render_documents(self, desired: list[Deployment]): list(desired), namespace=self.namespace, default_resource_profile=self.default_resource_profile, + unique_names=self.dynamic_routing, ) plan = GpuPlan( assignments={g.id: [] for g in desired if g.id not in rendered.unrenderable}, @@ -430,8 +501,21 @@ def preview(self, desired: list[Deployment], placement=None, *, approve: bool = """ plan, rendered, planned = self._render_documents(desired) self._preview_approval(planned, approve=approve) + if self.gateway is not None: + # The gateway's changes are part of the same approval: shown now, + # before the lease commits, and not asked again at the render. + self._set_front_door(desired, rendered) + self.gateway.preview([], None, approve=approve) + self.last_preview_digest = self._combined_digest( + self.last_preview_digest, self.gateway.last_preview_digest) return plan, rendered + def _combined_digest(self, models: str | None, gateway: str | None) -> str | None: + """One digest for the Models and the gateway's files together.""" + if gateway is None: + return models + return self._planned_digest({'models': models or '', 'gateway': gateway}) + def plan_on_idle_host(self, desired: list[Deployment]): """Whether ``desired`` could ever be served here: renderable or not. @@ -473,7 +557,8 @@ def converge(self, desired: list[Deployment], *, apply: bool = True, placement=N self.last_assignments = {} # the cluster places for err in rendered.errors: logger.warning(' render: {}', err) - self.last_planned_digest = self._planned_digest(planned) + models_digest = self._planned_digest(planned) + self.last_planned_digest = models_digest self._approve_changes(planned) self._atomic_write(self.models_file, rendered.text) self._save_sidecar( @@ -483,7 +568,13 @@ def converge(self, desired: list[Deployment], *, apply: bool = True, placement=N } ) if self.gateway is not None: - self._render_gateway(rendered) + # Gateway only: no engines on this host, so nothing to place. + # Its converge persists the merged route registry after its + # own approval (pre-approved by the preview when there was one). + self._set_front_door(desired, rendered) + self.gateway.converge([], apply=False) + self.last_planned_digest = self._combined_digest( + models_digest, self.gateway.last_planned_digest) if not apply: logger.info( 'rendered {} Model(s) to {} (not applied; ' @@ -508,7 +599,9 @@ def render_profile(self) -> dict: 'namespace': self.namespace, 'base_url': self.base_url, 'resource_profile': self.default_resource_profile, - 'gateway': self.gateway is not None, + # The front door's own settings (UI, proxy, routing mode), in the + # gateway project's profile; None without a gateway. + 'gateway': self.gateway.render_profile() if self.gateway is not None else None, 'gateway_upstream': self.gateway_upstream, 'catalogs': catalog_sources(self.catalog), } @@ -534,6 +627,11 @@ def use_profile(self, profile: dict) -> None: self.gateway_upstream = profile.get('gateway_upstream') or self.gateway_upstream sources = profile.get('catalogs') or [] self.catalog = CatalogUnion.from_sources(sources) if sources else None + front = profile.get('gateway') + # A profile from before the front door's settings were recorded holds + # only True/False here; it then keeps this process's settings. + if isinstance(front, dict) and self.gateway is not None: + self.gateway.use_profile(front) def apply(self) -> None: """Converge the cluster to the last render: apply + prune. @@ -679,7 +777,7 @@ def probe_ready(self, deployment: Deployment, endpoint: str) -> Readiness: base, model = f'{front._gateway_base()}/v1', endpoint headers = front._auth_headers() else: - base, model = self.base_url, model_name_for(_served_name(deployment)) + base, model = self.base_url, model_name(deployment) headers = None ok, reason = openai_ready( base_url=base, @@ -732,7 +830,7 @@ def realize(self, deployment: Deployment) -> None: # pragma: no cover pass def teardown(self, deployment: Deployment) -> None: - name = model_name_for(_served_name(deployment)) + name = model_name(deployment, unique=self.dynamic_routing) self._kubectl( ['delete', 'models.kubeai.org', name, '--ignore-not-found'] ) diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index 71507813..6acb5e48 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -283,12 +283,19 @@ def _make_backend(config, *, interactive: bool = False): # cluster: one base_url, the managed key, and endpoint aliases as # request names. Its own state dir and compose project, so it can # never touch a compose stack's containers on the same host. + # The front door takes the same settings as on compose (UI, + # reverse proxy, dynamic routing); only the engines are elsewhere. + rp_enabled, rp_port, rp_config = _resolve_reverse_proxy(config) gateway = ComposeBackend( state_dir=data_root() / 'leasing' / 'kubeai-gateway', inventory={'gpu_count': 0, 'gpus': []}, project='infer-stack-gateway', litellm=True, - ui=False, + ui=_resolve_ui(config), + reverse_proxy=rp_enabled, + reverse_proxy_port=rp_port, + reverse_proxy_config=rp_config, + dynamic_routing=_resolve_dynamic_routing(config), assume_yes=_resolve_assume_yes(config, interactive=interactive), ) backend = KubeaiBackend( @@ -1290,8 +1297,8 @@ def main(cls, argv=True, **kwargs): f'{n_deployments} stopped deployment(s)') return 0 if config.orphans: - if not isinstance(controller.backend, ComposeBackend): - raise SystemExit('gc --orphans needs the compose backend') + # Unlabelled units in residency. The kubeai backend reads only the + # pods carrying its label, so it never has any. def confirm(found): print(f'gc --orphans: {len(found)} unmanaged container(s):') @@ -1379,8 +1386,8 @@ def main(cls, argv=True, **kwargs): held = [g for g in deployments if g.state in (DeploymentState.LIVE, DeploymentState.IDLE)] observed, assignments = _placement_view(controller) - can_orphan = bool(config.orphans) and isinstance( - controller.backend, ComposeBackend) + # On any backend: one that selects its units by label (kubeai) has none. + can_orphan = bool(config.orphans) found: list = [] @@ -2456,17 +2463,17 @@ def main(cls, argv=True, **kwargs): def _require_compose_backend(controller): - """The controller's ComposeBackend, or a SystemExit for other backends. + """The Compose project holding the gateway's route registry. - The route registry is a compose-backend concept (it feeds the static-superset - LiteLLM gateway); ``--backend null``/``kubeai`` have no registry to touch.""" - backend = controller.backend - if not isinstance(backend, ComposeBackend): + The stack itself on the compose backend, the gateway's project on kubeai; + a SystemExit for a backend with no gateway (null, or ``litellm false``).""" + project = getattr(controller.backend, 'compose_project', lambda: None)() + if project is None or not getattr(project, 'litellm', False): raise SystemExit( - 'the `routes` commands require the compose backend ' - '(set `--backend compose` or `config set backend compose`)' + 'the `routes` commands need a LiteLLM gateway (the compose or kubeai ' + 'backend, with `litellm` on)' ) - return backend + return project def _live_endpoints(controller) -> set[str]: @@ -2484,7 +2491,8 @@ class RoutesListCLI(_LeasingCommonMixin): """Print the accumulated LiteLLM route registry (static-superset mode). One row per persisted route: its alias, engine, served-model/tag, the - upstream compose service it derives, and whether a live deployment is + upstream it routes to (a compose service, or a KubeAI cluster's gateway + under the Model's name), and whether a live deployment is currently backing it. Routes with no live backer still list (that is the point — a released endpoint stays routable/testable); their upstream simply errors until something serves it. @@ -2496,12 +2504,7 @@ class RoutesListCLI(_LeasingCommonMixin): @classmethod def main(cls, argv=True, **kwargs): - from ..leasing.compose import ( - OLLAMA_CONTAINER_PORT, - VLLM_CONTAINER_PORT, - ollama_service_name_for, - vllm_service_name_for, - ) + from ..leasing.gateway import registry_route_entry config = cls.cli(argv=argv, data=kwargs) controller = _open_controller(config) @@ -2514,22 +2517,13 @@ def main(cls, argv=True, **kwargs): for name in sorted(entries): row = entries[name] engine = row.get('engine') - if engine == 'vllm': - served = row.get('served') or name - upstream = ( - f'http://{vllm_service_name_for(served)}:' - f'{VLLM_CONTAINER_PORT}/v1' - ) - target = served - elif engine == 'ollama': - target = row.get('model') or name - host = row.get('host') or name - upstream = ( - f'http://{ollama_service_name_for(host)}:' - f'{OLLAMA_CONTAINER_PORT}' - ) - else: + entry = registry_route_entry(name, row) # what the gateway renders + if entry is None: target, upstream = '?', '?' + else: + params = entry['litellm_params'] + target = params['model'].split('/', 1)[-1] + upstream = params['api_base'] rows.append({ 'name': name, 'engine': engine, @@ -2579,12 +2573,7 @@ class RoutesPruneCLI(_ApprovalMixin): def main(cls, argv=True, **kwargs): from ..diff_prompt import confirm_writes from ..leasing.backend import ConvergeAborted - from ..leasing.gateway import ( - LITELLM_REGISTRY_VERSION, - _dump_route_registry, - _registry_incoming_from_catalog, - _registry_incoming_from_deployments, - ) + from ..leasing.gateway import LITELLM_REGISTRY_VERSION, _dump_route_registry config = cls.cli(argv=argv, data=kwargs) # interactive=False so reconcile auto-applies the compose diff; the @@ -2593,13 +2582,13 @@ def main(cls, argv=True, **kwargs): backend = _require_compose_backend(controller) def prune_plan() -> tuple[dict, dict, list[str]]: - # The desired set exactly as the next render sees it. - desired, inputs = controller._admission_view(backend.residency()) - plan = backend.plan(desired, inputs) - keep: dict = {} - if backend.catalog is not None: - keep.update(_registry_incoming_from_catalog(backend.catalog)) - keep.update(_registry_incoming_from_deployments(desired, plan.assignments)) + # The desired set exactly as the next render sees it, and the rows + # that render merges: the catalog's and the live deployments'. + # Any: _require_compose_backend above confirmed a gateway, and + # both backends that have one supply route rows. + engines: Any = controller.backend + desired, inputs = controller._admission_view(engines.residency()) + keep = dict(engines.route_rows(desired, inputs)) current = backend.gateway._load_route_registry().get('entries', {}) return current, keep, sorted(set(current) - set(keep)) @@ -2681,7 +2670,6 @@ class RoutesSeedCLI(_ApprovalMixin): @classmethod def main(cls, argv=True, **kwargs): from ..leasing.backend import ConvergeAborted - from ..leasing.compose import _registry_incoming_from_catalog config = cls.cli(argv=argv, data=kwargs) paths = _collect_names(config.catalogs) @@ -2691,6 +2679,8 @@ def main(cls, argv=True, **kwargs): # there is no destructive gate to confirm). controller = _open_controller(config, interactive=False) backend = _require_compose_backend(controller) + # Any: a gateway exists (checked above), and its backend supplies rows. + engines: Any = controller.backend incoming: dict = {} for raw in paths: @@ -2701,7 +2691,9 @@ def main(cls, argv=True, **kwargs): cat = Catalog.load(path) except CatalogError as ex: raise SystemExit(f'invalid catalog {path}: {ex}') - incoming.update(_registry_incoming_from_catalog(cat)) + # The engine backend's rows for it: compose upstreams on compose, + # the cluster's Models on kubeai. + incoming.update(engines.catalog_route_rows(cat)) if not incoming: raise SystemExit( 'routes seed: the named catalog(s) resolved no routable endpoints' @@ -2842,8 +2834,9 @@ def main(cls, argv=True, **kwargs): if not config.subnet: raise SystemExit('network migrate: --subnet is required') controller = _open_controller(config, interactive=True) - if not isinstance(controller.backend, ComposeBackend): - raise SystemExit('network migrate needs the compose backend') + if not hasattr(controller.backend, 'network'): + raise SystemExit('network migrate gives compose containers stable ' + 'addresses; this backend runs none (n/a on kubeai)') try: rec = controller.network_migrate(config.subnet, force=bool(config.force)) except ProfileMismatch as ex: @@ -2908,7 +2901,7 @@ def main(cls, argv=True, **kwargs): # Any: rotate_gateway_key below refuses a backend without a gateway, # so past it these gateway methods exist. backend: Any = controller.backend - old = backend.master_key() if isinstance(backend, ComposeBackend) else None + old = backend.master_key() if getattr(backend, 'litellm', False) else None try: rec = controller.rotate_gateway_key(force=bool(config.force)) except ProfileMismatch as ex: @@ -2930,7 +2923,8 @@ def main(cls, argv=True, **kwargs): 'rejects the OLD key') else: print(' gateway: new key accepted, old key rejected') - if getattr(backend, 'ui', False): + front = getattr(backend, 'compose_project', lambda: None)() + if getattr(front, 'ui', False): print(' Open WebUI may keep the old key in its own settings: ' 'update it under Admin > Settings > Connections') return 0 diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index 0470a36f..c44793cd 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -578,6 +578,7 @@ def render_compose( catalog: Any = None, route_registry: dict[str, Any] | None = None, dynamic_routing: bool = False, + upstream_routes: list[dict[str, Any]] | None = None, ) -> RenderedCompose: """Render a compose project for the placed deployments. @@ -665,7 +666,7 @@ def render_compose( reverse_proxy=reverse_proxy, reverse_proxy_port=reverse_proxy_port, reverse_proxy_config=reverse_proxy_config, aux_dir=aux_dir, catalog=catalog, route_registry=route_registry, - dynamic_routing=dynamic_routing, + dynamic_routing=dynamic_routing, upstream_routes=upstream_routes, ) services.update(front.services) litellm_config = front.litellm_config @@ -1426,6 +1427,7 @@ def _render_documents(self, desired: list[Deployment], placement) -> dict[str, A reverse_proxy_config=self.reverse_proxy_config, aux_dir=self.state_dir, project=self.project, catalog=self.catalog, route_registry=route_registry, dynamic_routing=self.dynamic_routing, + upstream_routes=self.upstream_routes, ) addresses = None if self.network is not None: @@ -1622,12 +1624,30 @@ def _merged_route_registry( placed deployment (``desired`` spans all runbooks via the shared ledger, so a live cross-runbook deployment stays routable, and past release). """ - incoming: dict[str, dict[str, Any]] = {} - if self.catalog is not None: - incoming.update(_registry_incoming_from_catalog(self.catalog)) + incoming = self.catalog_route_rows(self.catalog) incoming.update(_registry_incoming_from_deployments(desired, assignments)) + incoming.update(self.upstream_rows) return self.gateway.merged_route_registry(incoming) + #: Render inputs from the owner of engines this project does not run (the + #: kubeai backend, whose gateway this is): static route-registry rows and + #: dynamic routes, both pointing at its servers. Set before each render. + upstream_rows: dict[str, dict[str, Any]] = {} + upstream_routes: list[dict[str, Any]] = [] + + def catalog_route_rows(self, catalog) -> dict[str, dict[str, Any]]: + """Route-registry rows for every endpoint of ``catalog``.""" + return {} if catalog is None else _registry_incoming_from_catalog(catalog) + + def route_rows(self, desired: list[Deployment], placement=None) -> dict[str, dict[str, Any]]: + """The rows a render of ``desired`` merges: catalog, placed deployments, + and the owner's upstream rows (``routes prune`` keeps exactly these).""" + assignments = self.plan(desired, placement).assignments + rows = self.catalog_route_rows(self.catalog) + rows.update(_registry_incoming_from_deployments(desired, assignments)) + rows.update(self.upstream_rows) + return rows + def _update_route_registry( self, desired: list[Deployment], assignments: dict[str, list[int]] ) -> dict[str, Any]: diff --git a/infer_stack/leasing/gateway.py b/infer_stack/leasing/gateway.py index 433ee108..daaa48f6 100644 --- a/infer_stack/leasing/gateway.py +++ b/infer_stack/leasing/gateway.py @@ -261,32 +261,51 @@ def _litellm_model_list_from_registry( entries: list[dict[str, Any]] = [] rows = registry.get('entries', {}) if isinstance(registry, dict) else {} for name in sorted(rows): - row = rows[name] - if not isinstance(row, dict): - continue - engine = row.get('engine') - if engine == 'vllm': - served = row.get('served') or name - api_base = ( - f'http://{vllm_service_name_for(served)}:{VLLM_CONTAINER_PORT}/v1' - ) - entries.append(_vllm_route_entry(name, served, api_base)) - elif engine == 'ollama': - tag = row.get('model') or name - host = row.get('host') or name - api_base = ( - f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' - ) - entries.append(_ollama_route_entry(name, tag, api_base)) - elif engine == UPSTREAM_ROUTE and row.get('api_base'): - # An OpenAI-compatible server this project does not run (a KubeAI - # cluster's gateway): the row carries its address and the name it - # serves the model under. - entries.append(_vllm_route_entry(name, row.get('served') or name, - str(row['api_base']))) + entry = registry_route_entry(name, rows[name]) + if entry is not None: + entries.append(entry) return entries +def registry_route_entry(name: str, row: Any) -> dict[str, Any] | None: + """The LiteLLM entry one registry row renders to, or ``None`` if it cannot. + + The one derivation of a row's upstream: the render uses it, and so does + ``routes list``. Upstreams come from the live naming helpers, so the + registry never becomes a rendered-config parse surface. + """ + if not isinstance(row, dict): + return None + engine = row.get('engine') + if engine == 'vllm': + served = row.get('served') or name + api_base = f'http://{vllm_service_name_for(served)}:{VLLM_CONTAINER_PORT}/v1' + return _vllm_route_entry(name, served, api_base) + if engine == 'ollama': + tag = row.get('model') or name + host = row.get('host') or name + api_base = f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' + return _ollama_route_entry(name, tag, api_base) + if engine == UPSTREAM_ROUTE and row.get('api_base'): + # An OpenAI-compatible server this project does not run (a KubeAI + # cluster's gateway): the row carries its address and the name it + # serves the model under. + return _vllm_route_entry(name, row.get('served') or name, str(row['api_base'])) + return None + + +def upstream_route(deployment_id: str, endpoint: str, served: str, + api_base: str) -> dict[str, Any]: + """A dynamic route to a server this project does not run (a KubeAI Model). + + The same entry shape, and the same deterministic id, as a Compose engine's + dynamic route (:func:`_litellm_routes`). + """ + entry = _vllm_route_entry(endpoint, served, api_base) + entry['model_info'] = {'id': _route_id(deployment_id, endpoint)} + return entry + + def _merge_route_registry( existing: dict[str, Any], incoming: dict[str, dict[str, Any]] ) -> tuple[dict[str, Any], list[str]]: @@ -399,17 +418,7 @@ def _litellm_routes( f':{VLLM_CONTAINER_PORT}/v1' ) for endpoint in sorted(deployment.served): - entries.append( - { - 'model_name': endpoint, - 'litellm_params': { - 'model': f'openai/{served}', - 'api_base': api_base, - 'api_key': 'EMPTY', - }, - 'model_info': {'id': _route_id(deployment.id, endpoint)}, - } - ) + entries.append(upstream_route(deployment.id, endpoint, served, api_base)) elif deployment.engine == 'ollama': api_base = ( f'http://{ollama_service_name(deployment)}:{OLLAMA_CONTAINER_PORT}' @@ -766,12 +775,15 @@ def render_front_door( catalog: Any, route_registry: dict[str, Any] | None, dynamic_routing: bool, + upstream_routes: list[dict[str, Any]] | None = None, ) -> FrontDoor: """Render the gateway, its database, Open WebUI and the reverse proxy. The engines are the caller's: it passes the services it rendered (``engine_services``, only for the legacy per-model ``depends_on``) and the in-network URLs a UI with no gateway can talk to directly. + ``upstream_routes`` are dynamic routes to servers this project does not + run (see :func:`upstream_route`). """ services: dict[str, Any] = {} litellm_config = None @@ -804,7 +816,8 @@ def render_front_door( # config (and recreates the gateway) on every model change. if dynamic_routing: entries: list[dict[str, Any]] = [] - litellm_routes = _litellm_routes(deployments, assignments) + litellm_routes = (_litellm_routes(deployments, assignments) + + list(upstream_routes or [])) litellm_depends: list[str] = [] elif route_registry is not None: entries = _litellm_model_list_from_registry(route_registry) diff --git a/infer_stack/leasing/naming.py b/infer_stack/leasing/naming.py index c2d58c82..c89515f8 100644 --- a/infer_stack/leasing/naming.py +++ b/infer_stack/leasing/naming.py @@ -54,8 +54,16 @@ def _unique_vllm_service_name(served: str, deployment_id: str) -> str: so N dedicated deployments of one model become N containers on N GPUs. The suffix is the deployment id's hex tail, keeping the name short and DNS-safe. """ - tail = deployment_id.rsplit('-', 1)[-1][:8] or 'x' - return f'{vllm_service_name_for(served)}-{_dns_slug(tail)}' + return f'{vllm_service_name_for(served)}-{deployment_tail(deployment_id)}' + + +def deployment_tail(deployment_id: str) -> str: + """The short, DNS-safe suffix that makes a per-deployment name unique. + + >>> deployment_tail('grp-0123456789ab') + '01234567' + """ + return _dns_slug(deployment_id.rsplit('-', 1)[-1][:8] or 'x') def vllm_service_name(deployment: Deployment, *, unique: bool = False) -> str: diff --git a/tests/test_leasing_kubeai.py b/tests/test_leasing_kubeai.py index b162318e..91b1d950 100644 --- a/tests/test_leasing_kubeai.py +++ b/tests/test_leasing_kubeai.py @@ -773,15 +773,18 @@ def test_a_slow_start_is_not_a_failure(tmp_path): assert not probe.ready and not probe.fatal -def test_compose_only_commands_refuse_kubeai_explicitly(tmp_path, monkeypatch): - """`gc --orphans` used "has residency" to mean compose; kubeai has it now.""" +def test_gc_orphans_on_kubeai_finds_nothing_to_remove(tmp_path, monkeypatch, capsys): + """Residency selects pods by infer-stack's label, so none is an orphan: + the command runs on kubeai and never deletes anything.""" from infer_stack.cli import commands_leasing - be, _ = make_pod_backend(tmp_path) + be, kubectl = make_pod_backend(tmp_path) + kubectl.pods = [_pod('model-a-0', 'grp-1', ready=True)] ctl = Controller(Ledger(SqliteStore(str(tmp_path / 'l.db'))), be) monkeypatch.setattr(commands_leasing, '_open_controller', lambda *a, **k: ctl) - with pytest.raises(SystemExit, match='needs the compose backend'): - commands_leasing.GcCLI.main(argv=['--orphans', '--yes']) + assert commands_leasing.GcCLI.main(argv=['--orphans', '--yes']) == 0 + assert 'removed 0' in capsys.readouterr().out + assert not any('delete' in call or 'rm' in call for call in kubectl.calls) def test_runtime_env_reaches_the_model_like_it_reaches_a_container(): @@ -793,3 +796,128 @@ def test_runtime_env_reaches_the_model_like_it_reaches_a_container(): (doc,) = rendered.docs assert doc['spec']['env'] == {'MODE': 'fast', 'CTX': '4096', 'ON': 'true', 'VLLM_ATTENTION_BACKEND': 'TORCH_SDPA'} + + +# -- the front door: same settings, same approval, same routes (P3) ----------- + + +def make_front_door_backend(tmp_path, **gateway_kw): + from infer_stack.leasing.compose import ComposeBackend + from test_leasing_compose import FakeDocker + + kubectl = FakeKubectl() + gateway = ComposeBackend( + state_dir=tmp_path / 'gateway', inventory={'gpu_count': 0, 'gpus': []}, + run=FakeDocker(), http=GatewayHttp(kubectl, tmp_path / 'gateway'), + project='infer-stack-gateway', litellm=True, + images={'litellm': 'litellm:test', 'open-webui': 'owui:test', + 'nginx': 'nginx:test', 'postgres': 'pg:test'}, + **gateway_kw, + ) + be = KubeaiBackend(state_dir=tmp_path / 'kubeai', run=kubectl, + http=gateway.http, gateway=gateway, gateway_upstream=UPSTREAM) + return be, kubectl + + +def _services(be): + doc = yaml.safe_load(be.gateway.compose_file.read_text()) + return set(doc['services']) + + +def test_the_gateway_project_takes_the_front_door_settings(tmp_path): + be, _ = make_front_door_backend(tmp_path, ui=True, reverse_proxy=True) + from infer_stack.leasing.gateway import NGINX_SERVICE, OPEN_WEBUI_SERVICE + + be.converge([vllm('grp-a', served='tiny')], apply=False) + assert {'litellm', OPEN_WEBUI_SERVICE, NGINX_SERVICE} <= _services(be) + + +def test_dynamic_routing_gives_each_dedicated_deployment_its_own_model(tmp_path): + be, kubectl = make_front_door_backend(tmp_path, dynamic_routing=True) + a = vllm('grp-aaaaaaaa1111', served='tiny', t=1.0) + b = vllm('grp-bbbbbbbb2222', served='tiny', t=2.0) + for dep in (a, b): + dep.served = {'tiny': {'served_model_name': 'tiny', 'protocol': 'chat'}} + be.converge([a, b], apply=False) + models = yaml.safe_load_all(be.models_file.read_text()) + assert {m['metadata']['name'] for m in models} == {'tiny-aaaaaaaa', 'tiny-bbbbbbbb'} + routes = json.loads((be.gateway.state_dir / 'litellm_routes.json').read_text()) + assert sorted(r['litellm_params']['model'] for r in routes) == [ + 'openai/tiny-aaaaaaaa', 'openai/tiny-bbbbbbbb'] + assert {r['model_name'] for r in routes} == {'tiny'} # one alias, two upstreams + assert 'postgres-litellm' in _services(be) + + +def test_catalog_endpoints_are_routed_before_they_run(tmp_path): + """The static superset, as on compose: acquiring a catalog endpoint does + not change the gateway's config, so the gateway is not recreated.""" + from infer_stack.leasing import Catalog + + be, _ = make_front_door_backend(tmp_path) + be.catalog = Catalog.from_dict({ + 'models': {'m': {'source': 'hf://org/m'}}, + 'endpoints': {'tiny': {'engine': 'vllm', 'model': 'm'}}, + }) + be.converge([], apply=False) + before = (be.gateway.state_dir / 'litellm_config.yaml').read_text() + assert 'tiny' in before + dep = vllm('grp-a', served='tiny') + dep.served = {'tiny': {'served_model_name': 'tiny', 'protocol': 'chat'}} + be.converge([dep], apply=False) + assert (be.gateway.state_dir / 'litellm_config.yaml').read_text() == before + + +def test_the_gateways_changes_are_approved_in_the_preview(tmp_path, monkeypatch): + """One approval, before the lease commits: a decline writes neither the + Models nor the gateway's route registry.""" + from infer_stack import diff_prompt as dp + from infer_stack.leasing.backend import ConvergeAborted + + be, _ = make_front_door_backend(tmp_path) + be.assume_yes = be.gateway.assume_yes = False + asked = [] + monkeypatch.setattr(dp, 'confirm_writes', lambda changed, **kw: asked.append( + sorted(p.name for p in changed)) or False) + with pytest.raises(ConvergeAborted): + be.preview([vllm('grp-a', served='tiny')], approve=True) + assert not be.models_file.exists() + assert not be.gateway.gateway._registry_file.exists() + + asked.clear() + monkeypatch.setattr(dp, 'confirm_writes', lambda changed, **kw: asked.append( + sorted(p.name for p in changed)) or True) + desired = [vllm('grp-a', served='tiny')] + be.preview(desired, approve=True) + approved = be.last_preview_digest + assert any('litellm_config.yaml' in names for names in asked) + asked.clear() + be.converge(desired, apply=False) + assert asked == [] # not asked a second time + assert be.last_planned_digest == approved # the approved-digest guard holds + + +def test_routes_list_shows_kubeai_upstreams(tmp_path, monkeypatch, capsys): + from infer_stack.cli import commands_leasing + + be, _ = make_front_door_backend(tmp_path) + dep = vllm('grp-a', served='Qwen/Qwen2.5-0.5B') + dep.served = {'tiny': {'served_model_name': 'Qwen/Qwen2.5-0.5B', 'protocol': 'chat'}} + be.converge([dep], apply=False) + ctl = Controller(Ledger(SqliteStore(str(tmp_path / 'l.db'))), be) + monkeypatch.setattr(commands_leasing, '_open_controller', lambda *a, **k: ctl) + capsys.readouterr() # the render's own output + assert commands_leasing.RoutesListCLI.main(argv=['--json']) == 0 + (row,) = json.loads(capsys.readouterr().out)['routes'] + assert row['name'] == 'tiny' and row['engine'] == 'upstream' + assert row['target'] == 'qwen-qwen2-5-0-5b' and row['upstream'] == UPSTREAM + + +def test_the_front_doors_settings_are_in_the_recovery_profile(tmp_path): + be, _ = make_front_door_backend(tmp_path, ui=True, dynamic_routing=True) + profile = be.render_profile() + assert profile['gateway']['ui'] is True and profile['gateway']['dynamic_routing'] is True + other, _ = make_front_door_backend(tmp_path / 'other') + other.use_profile(profile) + assert other.gateway.ui is True and other.dynamic_routing is True + other.use_profile({**profile, 'gateway': True}) # a profile from before + assert other.dynamic_routing is True # keeps what it has From 487cab54124800bab4ca8c5a0dcda54aad824d35 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Sat, 26 Sep 2026 12:19:37 -0400 Subject: [PATCH 30/34] Queue: record the P3 commit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- docs/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/queue.md b/docs/queue.md index 8bdd3b36..c662ea27 100644 --- a/docs/queue.md +++ b/docs/queue.md @@ -58,7 +58,7 @@ and lists its pod (checked in a real terminal, not only in tests). ### 4. [x] P3: gateway feature parity -Done 2026-09-26. +Done 2026-09-26, `7a322c1`. `routes` and `secrets rotate` resolve the backend's gateway instead of checking its kind; the KubeAI gateway project honours `ui`, From 1a4e4b266b8645aa17e2a5e52183d45ec468f7de Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Sat, 26 Sep 2026 12:37:15 -0400 Subject: [PATCH 31/34] P6: a parity suite, one test per row the matrix marks same tests/test_parity.py builds the same stack over a fake Docker (compose) and a fake kubectl (kubeai, with its gateway) and runs one scenario per *same* row of docs/backend-parity.md on both: the client contract, the lifecycle verbs, admission, config publish, --no-apply, crash-loop diagnosis, strict residency, GPU count, the argument pipeline, served names, static and dynamic routes, routes list, the UI and proxy, approval, instances, stack down, and the TUI. Found on the way: the TUI replaced an injected Docker runner with the real one, so a TUI on a test backend talked to the host's Docker; it now wraps only the default runner. The matrix's gc --orphans row is corrected (it runs on kubeai and finds none). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 7 + docs/backend-parity.md | 6 +- docs/planning/backend-parity-roadmap.md | 14 +- docs/queue.md | 4 +- infer_stack/tui.py | 15 +- tests/test_parity.py | 449 ++++++++++++++++++++++++ 6 files changed, 482 insertions(+), 13 deletions(-) create mode 100644 tests/test_parity.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 1bde84cb..d1f6b58a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### A parity suite for the two backends + +`tests/test_parity.py` runs one scenario per row that `docs/backend-parity.md` +marks *same*, on the compose backend over a fake Docker and on the kubeai +backend over a fake kubectl, from the client contract to the TUI. The TUI no +longer swaps an injected Docker runner for the real one. + ### The gateway in front of a cluster is the compose gateway On the kubeai backend the gateway now takes the same settings as on compose: diff --git a/docs/backend-parity.md b/docs/backend-parity.md index 9f399cc4..de60913f 100644 --- a/docs/backend-parity.md +++ b/docs/backend-parity.md @@ -51,7 +51,8 @@ rotate` and the static superset route table work unchanged. ## Parity matrix -**same**: one code path, or verified equivalent. **≈**: the same outcome by +**same**: one code path, or verified equivalent; every *same* row has a test +in `tests/test_parity.py` that runs the same scenario on both backends. **≈**: the same outcome by a different mechanism. **gap**: missing on one side and on the roadmap. **n/a**: does not apply there. **boundary**: deliberately unsupported (see [planning/known-limitations.md](planning/known-limitations.md)). @@ -125,13 +126,14 @@ a different mechanism. **gap**: missing on one side and on the roadmap. | TUI: leases, deployments, catalog editing, acquire / release / evict, API tab, settings | same | same | | TUI: log follow, the Instances tab, Apply / Down | containers | same, over pods | | TUI GPU pane | `nvidia-smi` on this host | this host, not the cluster | -| `gc --orphans` (also inside `clean`) | yes | n/a: unlabeled Models are never touched | +| `gc --orphans` (also inside `clean`) | yes | runs, and finds none: pods are read by infer-stack's label, so unlabelled Models are never touched | ### Testing | | Compose | KubeAI | |---|---|---| | unit suite with fake runtimes | yes | yes | +| parity suite (`tests/test_parity.py`: each *same* row, both backends) | yes | yes | | end to end | `dev/e2e_tests/run.sh` (tiers; `--gpu` for serving) | `dev/kubeai_e2e.sh` against a real cluster; k3s with CPU vLLM needs no GPU | ## Deviations that stay diff --git a/docs/planning/backend-parity-roadmap.md b/docs/planning/backend-parity-roadmap.md index 6eacb09a..6e5aee23 100644 --- a/docs/planning/backend-parity-roadmap.md +++ b/docs/planning/backend-parity-roadmap.md @@ -1,8 +1,8 @@ # Backend parity roadmap: KubeAI as a superset of Compose **Status:** proposed 2026-09-25 · **P0 done** 2026-09-24 on -`dev/backend-unification` · **P1–P3 done** 2026-09-26 · P4, P5 not started · -P6 is ongoing. Execution order: [../queue.md](../queue.md). +`dev/backend-unification` · **P1–P3, P6 done** 2026-09-26 · P4, P5 not +started. Execution order: [../queue.md](../queue.md). **Current state:** [../backend-parity.md](../backend-parity.md). **Origin:** the scale-up run needs more than one workstation, and the KubeAI backend had drifted from Compose for three months before the @@ -210,6 +210,16 @@ the gateway, and the only one that needs a second machine. Last. - Finishing a phase updates the matrix in `backend-parity.md`; the plan is done when the matrix has no *gap* row. +**P6 done 2026-09-26** as a suite that stays open: `tests/test_parity.py` +builds the same stack over a fake Docker and a fake kubectl and runs one +test per *same* row on both (21 rows, the TUI included). A row turned *same* +gets its test in the same change. The Memory and Null backends run the +controller's one path through `SimpleAdmission` (P1b), so the controller +suites cover them without a third parametrization. Found on the way: the +TUI replaced an injected Docker runner with the real one, so a TUI started +on a test backend talked to the host's Docker; it now wraps only the +default runner. + ## Duplicate authorities The rule while executing: a duplicate authority found on the way is diff --git a/docs/queue.md b/docs/queue.md index c662ea27..7c89a151 100644 --- a/docs/queue.md +++ b/docs/queue.md @@ -69,7 +69,9 @@ into the admission preview (deferred duplicate). the cluster; an e2e step acquires the same model `--dedicated` twice under dynamic routing and gets two Models and two routes. -### 5. [ ] P6: one test surface +### 5. [x] P6: one test surface + +Done 2026-09-26. Parametrize the controller's acquire scenarios over the Memory, fake-Compose and fake-KubeAI backends; `tests/test_parity.py` runs each diff --git a/infer_stack/tui.py b/infer_stack/tui.py index 439afd00..dc1b6687 100644 --- a/infer_stack/tui.py +++ b/infer_stack/tui.py @@ -1351,22 +1351,21 @@ def _report_stall(self, length: float, stack: list[str]) -> None: def _install_quiet_docker(self) -> None: """Route ``docker`` output to the logs pane, not the terminal. - Only ``docker`` commands: the kubeai backend runs ``kubectl`` through - the same seam, and the Docker runner's allowlisted environment has no - ``KUBECONFIG`` (every kubectl call from the TUI used to fail). The - kubeai backend's gateway is a Compose project of its own, so it is - wrapped too. + Only where the backend runs Docker with the default runner: the kubeai + backend runs ``kubectl`` through the same seam, and the Docker + runner's allowlisted environment has no ``KUBECONFIG`` (every kubectl + call from the TUI used to fail); an injected runner is the caller's + and is left alone. The kubeai backend's gateway is a Compose project + of its own, so it is wrapped too. """ from .leasing.compose import _default_docker_run def quiet(target) -> None: original = getattr(target, 'run', None) - if original is None: + if original is not _default_docker_run: return def quiet_run(args: list[str], **kwargs) -> str: - if not args or args[0] != 'docker': - return original(args, **kwargs) # Same bounded, explicit-environment runner as the CLI; only # stderr is redirected to the logs pane, unless the caller # takes it. diff --git a/tests/test_parity.py b/tests/test_parity.py new file mode 100644 index 00000000..34087a31 --- /dev/null +++ b/tests/test_parity.py @@ -0,0 +1,449 @@ +"""Every row of docs/backend-parity.md that reads *same*, run on both backends. + +One harness builds the same stack twice: the compose backend over a fake +Docker, and the kubeai backend (with its LiteLLM gateway) over a fake +kubectl. Each test is one row of the matrix, named after it, and runs the +same scenario on both. A row whose test is here and passes on both is what +*same* means; a behaviour that only one backend can show belongs in that +backend's own tests. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from typing import Any + +import pytest +import yaml + +from infer_stack.leasing import Catalog, Controller, LeaseState, Ledger, SqliteStore +from infer_stack.leasing.backend import PlacementError +from infer_stack.leasing.residency import DEPLOYMENT_LABEL, ResidencyUnknown + +CATALOG = { + 'models': { + 'tiny': {'source': 'hf://org/tiny'}, + 'big': {'source': 'hf://org/big'}, + }, + 'endpoints': { + 'one': {'engine': 'vllm', 'model': 'tiny', 'reclaim': {'policy': 'stop'}, + 'runtime': {'max_model_len': 2048, 'extra_args': ['--seed', '7']}}, + 'wide': {'engine': 'vllm', 'model': 'big', + 'runtime': {'tensor_parallel_size': 2, 'max_model_len': 4096}}, + 'warm': {'engine': 'vllm', 'model': 'big', 'reclaim': {'policy': 'keep-warm'}, + 'runtime': {'max_model_len': 1024}}, + }, +} +CAT = Catalog.from_dict(CATALOG) +UPSTREAM = 'http://10.43.0.9/openai/v1' +CRASH_LOG = ('INFO vLLM API server version 0.25.1\n' + 'ValueError: The checkpoint has model type `example_moe_v1` but ' + 'Transformers does not recognize this architecture. If the model is ' + 'custom, set trust_remote_code=True.\n') + + +@dataclass +class Stack: + """One backend under test, and the handles a scenario needs.""" + + kind: str + ctl: Controller + backend: Any + runtime: Any # the fake Docker or the fake kubectl + front: Any # the ComposeBackend holding the gateway + + def acquire(self, *names, dedicated: bool = False, **kw): + from infer_stack.leasing import Sharing + + kw.setdefault('wait', False) + sharing = Sharing.DEDICATED if dedicated else None + requests = CAT.resolve_names(list(names), sharing=sharing) + return self.ctl.acquire('alice', requests, **kw) + + def running(self) -> set[str]: + return {gid for gid, found in self.backend.residency().by_deployment.items() + if any(c.warm for c in found)} + + def crash(self, gid: str) -> None: + """Make the deployment's engine crash-loop with CRASH_LOG.""" + if self.kind == 'compose': + from infer_stack.leasing.compose import CRASH_LOOP_RESTARTS + + for c in self.runtime.containers.values(): + if c['labels'].get(DEPLOYMENT_LABEL) == gid: + c.update(state='restarting', restart_count=CRASH_LOOP_RESTARTS, + exit_code=1) + else: + self.runtime.crash[gid] = CRASH_LOG + # A crash-looping pod answers nothing, as a restarting container. + post = self.backend.http.post + self.backend.http.post = lambda url, **kw: ( + self.backend.http._Resp(503, {'detail': 'upstream down'}) + if url.startswith('http://127.0.0.1:') else post(url, **kw)) + self.backend.deployment_logs = lambda deployment, tail=400: CRASH_LOG + + def break_runtime(self) -> None: + """The runtime stops answering (docker or kubectl down).""" + def broken(args, **kw): + raise RuntimeError('connection refused') + self.backend.run = broken + + def gateway_routes(self) -> dict[str, dict]: + cfg = self.front.state_dir / 'litellm_config.yaml' + doc = yaml.safe_load(cfg.read_text()) if cfg.exists() else {} + return {e['model_name']: e['litellm_params'] for e in doc.get('model_list') or []} + + +def _compose(tmp_path, **gateway_kw) -> Stack: + from infer_stack.hardware import simulate_inventory + from infer_stack.leasing.compose import ComposeBackend + from test_leasing_compose import IMAGES, PORTS, STATE, FakeDocker, FakeHttp + + docker = FakeDocker() + backend = ComposeBackend( + state_dir=tmp_path / 'compose', inventory=simulate_inventory('4x80'), + run=docker, http=FakeHttp(tmp_path / 'compose'), + images={**IMAGES, 'open-webui': 'owui:test', 'nginx': 'nginx:test', + 'postgres': 'pg:test'}, + ports=PORTS, state=STATE, litellm=True, catalog=CAT, **gateway_kw, + ) + ctl = Controller(Ledger(SqliteStore(str(tmp_path / 'ledger.db'))), backend) + return Stack('compose', ctl, backend, docker, backend) + + +class _Kubectl: + """kubectl over applied Model CRs; each Model runs one pod.""" + + def __init__(self): + from test_leasing_kubeai import FakeKubectl + + self.inner = FakeKubectl() + self.crash: dict[str, str] = {} + + @property + def applied(self): + return self.inner.applied + + def __call__(self, args: list[str]) -> str: + from test_leasing_kubeai import _pod + + if len(args) > 4 and args[3] == 'get' and args[4] == 'pods': + pods = [] + for name, doc in self.inner.applied.items(): + gid = doc['metadata']['labels']['infer-stack/deployment'] + if gid in self.crash: + pods.append(_pod(f'model-{name}-0', gid, restarts=3, + state={'waiting': {'reason': 'CrashLoopBackOff'}}, + last={'exitCode': 1, 'reason': 'Error'})) + else: + pods.append(_pod(f'model-{name}-0', gid, ready=True)) + return json.dumps({'items': pods}) + return self.inner(args) + + +def _kubeai(tmp_path, **gateway_kw) -> Stack: + from infer_stack.backends.kubeai import KubeaiBackend + from infer_stack.leasing.compose import ComposeBackend + from test_leasing_compose import FakeDocker + from test_leasing_kubeai import GatewayHttp + + kubectl = _Kubectl() + gateway = ComposeBackend( + state_dir=tmp_path / 'gateway', inventory={'gpu_count': 0, 'gpus': []}, + run=FakeDocker(), http=GatewayHttp(kubectl.inner, tmp_path / 'gateway'), + project='infer-stack-gateway', litellm=True, + images={'litellm': 'litellm:test', 'open-webui': 'owui:test', + 'nginx': 'nginx:test', 'postgres': 'pg:test'}, + **gateway_kw, + ) + backend = KubeaiBackend(state_dir=tmp_path / 'kubeai', run=kubectl, + http=gateway.http, gateway=gateway, + gateway_upstream=UPSTREAM, default_resource_profile='gpu') + backend.catalog = CAT + ctl = Controller(Ledger(SqliteStore(str(tmp_path / 'ledger.db'))), backend) + return Stack('kubeai', ctl, backend, kubectl, gateway) + + +BUILDERS = {'compose': _compose, 'kubeai': _kubeai} + + +@pytest.fixture(params=sorted(BUILDERS)) +def make_stack(request, tmp_path): + def make(**gateway_kw) -> Stack: + return BUILDERS[request.param](tmp_path, **gateway_kw) + return make + + +# -- client contract ----------------------------------------------------------- + + +def test_one_base_url_the_managed_key_and_the_alias(make_stack): + stack = make_stack() + stack.acquire('one') + info = stack.backend.access(['one']) + assert info['base_url'].startswith('http://127.0.0.1:') + assert info['api_key'] == stack.front.master_key() + assert info['request_names'] == {'one': 'one'} + + +def test_env_file_has_the_same_keys(make_stack): + from types import SimpleNamespace + + from infer_stack.cli.commands_leasing import _descriptor_for + from infer_stack.leasing.envfile import descriptor_env + + stack = make_stack() + out = stack.acquire('one') + config = SimpleNamespace(base_url='unused', api_key_env='LITELLM_MASTER_KEY') + env = descriptor_env(_descriptor_for(stack.ctl, out.lease, out.deployments, config)) + assert set(env) >= {'INFER_STACK_LEASE_ID', 'OPENAI_BASE_URL', + 'INFER_STACK_ENDPOINT_ONE', 'INFER_STACK_MODELS'} + assert env['INFER_STACK_ENDPOINT_ONE'] == 'one' # the alias, on both + + +def test_readiness_is_a_generation_through_the_front_door(make_stack): + stack = make_stack() + out = stack.acquire('one', wait=True, timeout=10, interval=1) + assert out.wait is not None and out.wait.ready + + +def test_secrets_rotate(make_stack): + stack = make_stack() + old = stack.front.master_key() + stack.ctl.rotate_gateway_key() + assert stack.front.master_key() != old + + +# -- lifecycle ------------------------------------------------------------------- + + +def test_acquire_release_evict_gc_renew(make_stack): + stack = make_stack() + one = stack.acquire('one') + warm = stack.acquire('warm') + gid_one, gid_warm = one.deployments[0].id, warm.deployments[0].id + assert {gid_one, gid_warm} <= stack.running() + stack.ctl.renew(one.lease.id, ttl_seconds=3600) + stack.ctl.release(one.lease.id) + stack.ctl.release(warm.lease.id) + assert gid_one not in stack.running() # reclaim: stop + assert gid_warm in stack.running() # keep-warm stays resident + stack.ctl.gc() + assert gid_warm in stack.running() # plain gc leaves it + stack.ctl.evict(None) + assert stack.running() == set() + + +def test_a_refused_acquire_writes_nothing(make_stack): + stack = make_stack() + # What each backend cannot serve: more GPUs than the host has, or a + # Model with no resource profile. + if stack.kind == 'compose': + stack.backend.inventory = {'gpu_count': 1, 'gpus': [ + {'index': 0, 'memory_total_mib': 81920}]} + else: + stack.backend.default_resource_profile = None + with pytest.raises(PlacementError): + stack.acquire('wide') + assert stack.ctl.ledger.status() == ([], []) + assert stack.running() == set() + + +def test_config_publish_previews_then_commits(make_stack): + stack = make_stack() + profile = stack.backend.render_profile() + stack.ctl.publish_profile(profile) + assert stack.ctl.ledger.profile() == profile + + +def test_no_apply_stages_and_apply_brings_it_up(make_stack): + stack = make_stack() + out = stack.acquire('one', apply=False) + gid = out.deployments[0].id + assert gid not in stack.running() + assert stack.ctl.ledger.publication_pending() is not None + stack.ctl.apply_now() + assert gid in stack.running() + + +def test_a_crash_looping_engine_fails_fast_with_its_error(make_stack): + stack = make_stack() + out = stack.acquire('one') + gid = out.deployments[0].id + stack.crash(gid) + probe = stack.backend.probe_ready(stack.ctl.ledger.get_deployment(gid), 'one') + assert probe.ready is False and probe.fatal is True + assert 'trust_remote_code' in probe.detail + + +def test_strict_residency_and_lenient_observe(make_stack): + stack = make_stack() + stack.acquire('one') + stack.break_runtime() + with pytest.raises(ResidencyUnknown): + stack.backend.residency() + assert stack.backend.observe() == set() + + +# -- placement and the catalog ----------------------------------------------------- + + +def test_gpu_count_from_tp_pp_dp(make_stack): + stack = make_stack() + out = stack.acquire('wide') + gid = out.deployments[0].id + if stack.kind == 'compose': + assert len(stack.ctl.ledger.get_deployment(gid).assigned_gpus) == 2 + else: + (doc,) = [d for d in stack.runtime.applied.values() + if d['metadata']['labels']['infer-stack/deployment'] == gid] + assert doc['spec']['resourceProfile'] == 'gpu:2' + + +def _engine_args(stack, gid) -> list[str]: + if stack.kind == 'compose': + doc = yaml.safe_load(stack.backend.compose_file.read_text()) + (svc,) = [s for s in doc['services'].values() + if s.get('labels', {}).get(DEPLOYMENT_LABEL) == gid] + return [str(a) for a in svc['command']] + (doc,) = [d for d in stack.runtime.applied.values() + if d['metadata']['labels']['infer-stack/deployment'] == gid] + return [str(a) for a in doc['spec']['args']] + + +def test_runtime_flags_and_extra_args_reach_the_engine(make_stack): + stack = make_stack() + out = stack.acquire('one') + args = ' '.join(_engine_args(stack, out.deployments[0].id)) + assert '--max-model-len=2048' in args + assert '--seed 7' in args + + +def test_served_names_come_from_one_rule(make_stack): + from infer_stack.leasing.naming import dns_slug + + stack = make_stack() + out = stack.acquire('one') + gid = out.deployments[0].id + if stack.kind == 'compose': + doc = yaml.safe_load(stack.backend.compose_file.read_text()) + names = [n for n, s in doc['services'].items() + if s.get('labels', {}).get(DEPLOYMENT_LABEL) == gid] + assert names == [f'vllm-{dns_slug("one")}'] + else: + assert f'{dns_slug("one")}' in stack.runtime.applied + + +# -- the gateway ------------------------------------------------------------------ + + +def test_catalog_routes_exist_before_models_run_and_do_not_churn(make_stack): + stack = make_stack() + stack.ctl.apply_now() + before = stack.gateway_routes() + assert {'one', 'wide', 'warm'} <= set(before) + stack.acquire('one') + assert stack.gateway_routes() == before + + +def test_routes_list(make_stack, monkeypatch, capsys): + from infer_stack.cli import commands_leasing + + stack = make_stack() + stack.acquire('one') + monkeypatch.setattr(commands_leasing, '_open_controller', lambda *a, **k: stack.ctl) + capsys.readouterr() + assert commands_leasing.RoutesListCLI.main(argv=['--json']) == 0 + rows = {r['name']: r for r in json.loads(capsys.readouterr().out)['routes']} + assert rows['one']['live'] is True and rows['one']['upstream'] != '?' + + +def test_dynamic_routing_gives_dedicated_deployments_their_own_upstreams(make_stack): + stack = make_stack(dynamic_routing=True) + first = stack.acquire('one', dedicated=True, apply=False) + second = stack.acquire('one', dedicated=True, apply=False) + assert first.deployments[0].id != second.deployments[0].id + routes = json.loads((stack.front.state_dir / 'litellm_routes.json').read_text()) + ones = [r for r in routes if r['model_name'] == 'one'] + assert len(ones) == 2 # one alias, two upstreams + assert len({(r['litellm_params']['model'], r['litellm_params']['api_base']) + for r in ones}) == 2 + +def test_open_webui_and_the_reverse_proxy(make_stack): + from infer_stack.leasing.gateway import NGINX_SERVICE, OPEN_WEBUI_SERVICE + + stack = make_stack(ui=True, reverse_proxy=True) + stack.acquire('one', apply=False) + doc = yaml.safe_load(stack.front.compose_file.read_text()) + assert {OPEN_WEBUI_SERVICE, NGINX_SERVICE} <= set(doc['services']) + + +def test_the_gateways_changes_are_approved_with_the_acquire(make_stack, monkeypatch): + from infer_stack import diff_prompt as dp + + stack = make_stack() + for b in {id(stack.backend): stack.backend, id(stack.front): stack.front}.values(): + b.assume_yes = False + asked = [] + monkeypatch.setattr(dp, 'confirm_writes', + lambda changed, **kw: asked.append(sorted(p.name for p in changed)) + or True) + out = stack.acquire('one') + assert out.lease.state == LeaseState.ACTIVE + shown = [name for names in asked for name in names] + assert 'litellm_config.yaml' in shown # the gateway's change too + # Each file was shown once, before the commit, never again at the render. + assert len(shown) == len(set(shown)) + + +# -- day-2 ------------------------------------------------------------------------- + + +def test_instances_have_one_shape(make_stack): + stack = make_stack() + out = stack.acquire('one') + gid = out.deployments[0].id + instances = stack.backend.instances() + engines = [i for i in instances if i.deployment_id == gid] + assert len(engines) == 1 and engines[0].state == 'running' + assert any(not i.is_engine for i in instances) # the gateway + + +def test_stack_down_stops_everything(make_stack): + stack = make_stack() + stack.acquire('one') + stack.backend.down() + assert stack.running() == set() + + +def test_the_tui_reads_either_backend(make_stack): + """Leases, deployments, the Instances tab and the engines log view.""" + pytest.importorskip('textual') + import asyncio + + from infer_stack.tui import ENGINE_SERVICES, InferStackTUI + + stack = make_stack() + out = stack.acquire('one') + gid = out.deployments[0].id + + async def scenario(): + app = InferStackTUI(stack.ctl, CAT, interval=999, proc_factory=lambda s: None) + async with app.run_test() as pilot: + await pilot.pause() + await app.workers.wait_for_complete() # the startup refresh + app._collapsed['docker'] = False + data = app._collect() + assert [le.id for le in data['leases']] == [out.lease.id] + assert gid in {g.id for g in data['deployments']} + # What _render keeps, without racing the app's own refresh. + app._last_deployments = data['deployments'] + app._last_instances = data['instances'] + (engine,) = [r for r in app._ps_rows(data['instances']) + if r['serves'] == 'one'] + assert engine['status'] == 'running' + target, _ = app._resolve_log_target(ENGINE_SERVICES) + assert target == [engine['name']] + + asyncio.run(scenario()) From c430cafa2da0e0a881b8746169e0f43f4dd939de Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Sat, 26 Sep 2026 12:37:15 -0400 Subject: [PATCH 32/34] Queue: record the P6 commit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- docs/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/queue.md b/docs/queue.md index 7c89a151..49d92e8d 100644 --- a/docs/queue.md +++ b/docs/queue.md @@ -71,7 +71,7 @@ dynamic routing and gets two Models and two routes. ### 5. [x] P6: one test surface -Done 2026-09-26. +Done 2026-09-26, `1a4e4b2`. Parametrize the controller's acquire scenarios over the Memory, fake-Compose and fake-KubeAI backends; `tests/test_parity.py` runs each From 94c263de5c96928dfe0afa874cd034465a895e5b Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Sat, 26 Sep 2026 13:08:28 -0400 Subject: [PATCH 33/34] P4: KubeAI picks a resource profile by GPU size An endpoint with placement.min_vram_gib and no resource_profile gets the smallest resource profile whose nodes' GPUs are that large: a profile's size is the nvidia.com/gpu.memory label (GPU Feature Discovery) of the nodes its nodeSelector selects. The chart's selector-less profiles have no size. KubeAI keeps its own measurements overlay, filled by `measure --record`, and a measurement feeds the same choice, as on compose. `catalog suggest --backend kubeai` sizes the catalog to the largest GPU node and proposes a resource profile per GPU product. dev/k3s_agent_container.sh adds a tainted second node on this host; the e2e gains E2E_SIZED=1 (two nodes, fake GPU labels): 40 GiB lands on the 80 GiB profile's node, 10 GiB on the 24 GiB one and answers. The real-GPU check is dev/handover/p4_gpu_labels.sh. The e2e's `grep -q` checks no longer race pipefail, and the dynamic phase waits for a quiescent stack. Verified: pytest infer_stack tests; ty; dev/kubeai_e2e.sh with every phase on k3s. Journal and two lessons added. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- CHANGELOG.md | 12 ++ dev/e2e_tests/kubeai-cpu-values.yaml | 22 +++ dev/handover/p4_gpu_labels.sh | 108 ++++++++++++ dev/journals/claude.md | 41 +++++ dev/k3s_agent_container.sh | 49 ++++++ dev/kubeai_e2e.sh | 84 ++++++++- dev/lessons/lessons.md | 18 ++ dev/lessons/mwe/grep_q_pipefail.sh | 8 + docs/backend-parity.md | 3 +- docs/kubeai-backend.md | 17 ++ docs/planning/backend-parity-roadmap.md | 15 +- docs/queue.md | 13 +- infer_stack/backends/kubeai.py | 223 ++++++++++++++++++++++-- infer_stack/cli/commands_catalog.py | 53 +++++- infer_stack/cli/commands_leasing.py | 7 +- tests/test_leasing_kubeai.py | 89 ++++++++++ 16 files changed, 732 insertions(+), 30 deletions(-) create mode 100755 dev/handover/p4_gpu_labels.sh create mode 100755 dev/k3s_agent_container.sh create mode 100644 dev/lessons/mwe/grep_q_pipefail.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index d1f6b58a..a5ce85a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,18 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### KubeAI picks a resource profile by GPU size + +On the kubeai backend, an endpoint with `placement.min_vram_gib` and no +`resource_profile` gets the smallest resource profile whose nodes' GPUs are +that large, read from GPU Feature Discovery's `nvidia.com/gpu.memory` label; +before, `min_vram_gib` was warned about and ignored. `infer-stack measure +--record` works there too, and a recorded measurement feeds the same choice. +`infer-stack catalog suggest` on kubeai sizes the catalog to the cluster's +largest GPU node and proposes a resource profile per GPU product for the +helm values. `dev/k3s_agent_container.sh` adds a second (simulated) node to +a development cluster. + ### A parity suite for the two backends `tests/test_parity.py` runs one scenario per row that `docs/backend-parity.md` diff --git a/dev/e2e_tests/kubeai-cpu-values.yaml b/dev/e2e_tests/kubeai-cpu-values.yaml index 5d308441..8d5fb354 100644 --- a/dev/e2e_tests/kubeai-cpu-values.yaml +++ b/dev/e2e_tests/kubeai-cpu-values.yaml @@ -14,3 +14,25 @@ resourceProfiles: requests: cpu: 64 memory: "16Gi" + # Two "GPU sizes" on CPU, for the sizing check (E2E_SIZED=1 in + # dev/kubeai_e2e.sh): each selects the nodes labelled with one fake GPU + # product, as a real profile selects nvidia.com/gpu.product. `sized-80g` + # tolerates dev/k3s_agent_container.sh's taint, so it lands on that node. + sized-24g: + imageName: "cpu" + requests: + cpu: 8 + memory: "16Gi" + nodeSelector: + nvidia.com/gpu.product: "FAKE-24G" + sized-80g: + imageName: "cpu" + requests: + cpu: 8 + memory: "16Gi" + nodeSelector: + nvidia.com/gpu.product: "FAKE-80G" + tolerations: + - key: infer-stack.test/simulated + operator: Exists + effect: NoSchedule diff --git a/dev/handover/p4_gpu_labels.sh b/dev/handover/p4_gpu_labels.sh new file mode 100755 index 00000000..e47e2cfe --- /dev/null +++ b/dev/handover/p4_gpu_labels.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# P4 on real GPUs: what the CPU-only development cluster cannot show. +# +# Run on a GPU machine that is a k3s node with the NVIDIA device plugin and +# GPU Feature Discovery (README: "NVIDIA GPU support"), with the KubeAI chart +# installed and `kubectl -n kubeai port-forward svc/kubeai 8000:80` running. +# It serves one small model on one GPU for a few minutes: run it when that +# GPU is free. +# +# It checks, in order: +# 1. GPU Feature Discovery labels each GPU node (product, memory) and the +# device plugin advertises nvidia.com/gpu: the facts sizing reads. +# 2. `catalog suggest --backend kubeai` proposes a resource profile for +# each GPU product, from those labels. +# 3. With the proposed profiles installed, an endpoint that declares only +# `min_vram_gib` gets the right profile and answers a request on a GPU. +# 4. `infer-stack measure --record` finds vLLM's memory-profiling +# lines in the pod's log and records a min_vram_gib. +# +# It uses its own config and data directories (never this host's infer-stack +# state). It leaves the proposed profiles in the chart's values: they +# describe this cluster's GPUs and are what min_vram_gib needs. +# +# dev/handover/p4_gpu_labels.sh 2>&1 | tee p4-gpu.log +# +# Knobs: E2E_MODEL (default Qwen/Qwen2.5-0.5B-Instruct), NAMESPACE (kubeai), +# E2E_TIMEOUT (900 s). +set -euo pipefail + +MODEL="${E2E_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}" +NAMESPACE="${NAMESPACE:-kubeai}" +TIMEOUT="${E2E_TIMEOUT:-900}" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/infer-stack-p4.XXXXXX")" +IS_ENV="INFER_STACK_CONFIG_DIR=$WORK/config INFER_STACK_DATA_DIR=$WORK/data" +run_is() { env $IS_ENV infer-stack "$@"; } +pass() { echo "PASS $*"; } +fail() { echo "FAIL $*" >&2; exit 1; } + +cleanup() { + set +e + run_is release --all --yes >/dev/null 2>&1 + run_is stack down >/dev/null 2>&1 + echo "work dir kept for the report: $WORK" +} +trap cleanup EXIT +mkdir -p "$WORK/config" + +echo '== 1. the GPU facts sizing reads' +kubectl get nodes -o json | python3 -c ' +import json, sys +found = 0 +for n in json.load(sys.stdin)["items"]: + labels = n["metadata"].get("labels", {}) + gpus = n["status"].get("allocatable", {}).get("nvidia.com/gpu") + product, memory = labels.get("nvidia.com/gpu.product"), labels.get("nvidia.com/gpu.memory") + print(f" {n[\"metadata\"][\"name\"]}: gpus={gpus} product={product} memory_mib={memory}") + found += bool(gpus and product and memory) +sys.exit(0 if found else 1)' || fail 'no node has nvidia.com/gpu and the GFD product/memory labels' +pass 'GPU nodes are labelled' + +echo '== 2. catalog suggest proposes profiles from the labels' +run_is config set backend kubeai >/dev/null +run_is config set kubeai_namespace "$NAMESPACE" >/dev/null +run_is catalog suggest --backend kubeai > "$WORK/suggest.yaml" 2> "$WORK/suggest.err" \ + || { cat "$WORK/suggest.err" >&2; fail 'catalog suggest failed'; } +python3 - "$WORK/suggest.err" "$WORK/profiles.yaml" <<'PY' || fail 'suggest printed no resourceProfiles' +import sys, yaml +text = open(sys.argv[1]).read() +block = yaml.safe_load(text[text.index('resourceProfiles:'):]) +open(sys.argv[2], 'w').write(yaml.safe_dump(block, sort_keys=False)) +print(yaml.safe_dump(block, sort_keys=False)) +PY +pass 'suggest proposed a profile per GPU product' + +echo '== 3. min_vram_gib picks the proposed profile and serves on a GPU' +version=$(helm list -n "$NAMESPACE" -o json | python3 -c \ + 'import json,sys; print(json.load(sys.stdin)[0]["chart"].rsplit("-", 1)[1])') +helm upgrade kubeai kubeai/kubeai -n "$NAMESPACE" --version "$version" \ + --reuse-values -f "$WORK/profiles.yaml" --wait >/dev/null +cat > "$WORK/config/catalog.yaml" < "$WORK/acquire.log" 2>&1 || true +grep -q 'ready: True' "$WORK/acquire.log" || fail 'p4-small never became ready' +got=$(kubectl -n "$NAMESPACE" get models.kubeai.org -l infer-stack/managed=true \ + -o jsonpath='{.items[0].spec.resourceProfile}') +echo " Model resourceProfile: $got (profiles proposed: $(tr '\n' ' ' < "$WORK/profiles.yaml" | head -c 200))" +case "$got" in nvidia-*:1) pass "min_vram_gib 8 -> $got, and it answers" ;; + *) fail "unexpected resource profile $got" ;; esac + +echo '== 4. measure reads the pod log on a GPU' +run_is measure p4-small --record | tee "$WORK/measure.log" +grep -q 'min_vram_gib' "$WORK/measure.log" || fail 'measure printed no min_vram_gib' +test -s "$WORK/data/leasing/kubeai/measurements.json" || fail 'measure --record wrote nothing' +pass 'measure recorded a min_vram_gib on kubeai' + +echo 'ALL PASS' diff --git a/dev/journals/claude.md b/dev/journals/claude.md index aeb38b4a..c2441faa 100644 --- a/dev/journals/claude.md +++ b/dev/journals/claude.md @@ -3249,3 +3249,44 @@ Fixed with sparse files. **Takeaways.** (1) A comparison of outputs must check the outputs exist: equality of two failures is not evidence. (2) Test fixtures that need large files should be sparse. Size was the property under test, not the bytes. + +## 2026-09-26 13:05:00 -0400 + +**Intent.** Execute the backend-parity queue (`docs/queue.md`): the roadmap +phases that can be built and verified without a GPU or a second machine, +refactoring duplicate authorities on the way, and not stopping before a +passing UX audit. Model: Claude Opus 5.5 (Claude Code). + +**What landed.** P1b deleted the pre-admission acquire branch: a +`SimpleAdmission` mixin gives the dry-run and test backends the admission +surface, so the controller has one path. P2 put `ps`, `logs`, `status` and +the TUI behind an `Instance` view built from residency, on both backends. +P3 made the gateway in front of a cluster the compose gateway (UI, proxy, +dynamic routing, one approval). P6 is a parity suite, one test per *same* +row. P4 picks a KubeAI resource profile by GPU size from node labels. + +**Decisions.** The review split P1: a `preview` alone would have dropped +every KubeAI lease, because the admission view assumed GPU accounting, so +"does this backend allocate GPUs" became one function. Under dynamic +routing a KubeAI Model is named per deployment with compose's own tail +rule, rather than a new scheme. A profile's size is what its node selector +selects; the chart's selector-less profiles deliberately have none, because +guessing a size for "anywhere" would pick wrong silently. + +**What surprised me.** Three bugs older than this work surfaced only in a +real terminal or a real cluster: the TUI replaced kubeai's kubectl runner +with Docker's (no KUBECONFIG, so every kubectl call failed), a kwconf flag +swallowed the positional after it on every command, and the TUI swapped an +injected runner for the real Docker. Fakes passed throughout; each was +found by running the thing. The e2e also failed once on its own +`grep -q` + pipefail pattern (lessons.md). + +**Risks.** The kubeai recovery profile now nests the gateway's profile; +old profiles (a bare boolean) keep this process's settings. P4 is verified +with fake labels only; `dev/handover/p4_gpu_labels.sh` is the real test. + +**Takeaways.** (1) When a refactor removes a branch, first run the whole +suite with the other branch forced on: the failures list exactly what to +migrate. (2) A seam that wraps a runner must wrap only what it owns; a +wrapper that replaces is a second authority. (3) Put every e2e check's +output in a file before testing it. diff --git a/dev/k3s_agent_container.sh b/dev/k3s_agent_container.sh new file mode 100755 index 00000000..379594af --- /dev/null +++ b/dev/k3s_agent_container.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# A second k3s node on this host, in a Docker container: enough to exercise +# anything that depends on the cluster having more than one node (node +# labels, node selectors, where a pod lands) without a second machine. +# +# dev/k3s_agent_container.sh up [NAME] # join, taint, print the node +# dev/k3s_agent_container.sh down [NAME] # drain it out of the cluster +# +# The node is tainted `infer-stack.test/simulated=true:NoSchedule`, so only +# pods that tolerate it land there: an ordinary Model never does (its image +# would be pulled inside the container's own containerd). Give a test +# resource profile that toleration to put a Model on it. +# +# Needs: this host running the k3s server (scripts/bootstrap_k3s.sh), sudo +# to read the node token, and Docker. The image tag follows the server's. +set -euo pipefail + +ACTION="${1:-up}" +NAME="${2:-k3s-agent-b}" +CONTAINER="infer-stack-${NAME}" + +case "$ACTION" in + up) + version="$(k3s --version | awk 'NR==1 {print $3}')" # v1.36.4+k3s1 + image="rancher/k3s:${version/+/-}" + server_ip="$(ip -4 route get 1.1.1.1 | awk '{for (i=1;i/dev/null 2>&1 || true + docker run -d --name "$CONTAINER" --privileged --hostname "$NAME" \ + -e K3S_URL="https://${server_ip}:6443" -e K3S_TOKEN="$token" \ + --tmpfs /run --tmpfs /var/run "$image" agent --node-name "$NAME" >/dev/null + for _ in $(seq 60); do + kubectl get node "$NAME" >/dev/null 2>&1 && break + sleep 2 + done + kubectl wait --for=condition=Ready "node/$NAME" --timeout=120s >/dev/null + kubectl taint node "$NAME" infer-stack.test/simulated=true:NoSchedule --overwrite >/dev/null + kubectl get node "$NAME" -o wide + ;; + down) + kubectl delete node "$NAME" --ignore-not-found >/dev/null + docker rm -f "$CONTAINER" >/dev/null 2>&1 || true + echo "removed $NAME" + ;; + *) + echo "usage: $0 up|down [NAME]" >&2 + exit 2 + ;; +esac diff --git a/dev/kubeai_e2e.sh b/dev/kubeai_e2e.sh index e106ee75..5999338d 100755 --- a/dev/kubeai_e2e.sh +++ b/dev/kubeai_e2e.sh @@ -27,6 +27,10 @@ # E2E_DYNAMIC 1 (default): finish with dynamic routing, two # --dedicated leases of one model (two Models at once). # E2E_UI_PORT Open WebUI's port on this host (default 13000). +# E2E_SIZED 1: check min_vram_gib picks a profile by GPU size, +# on two nodes with fake GPU labels. Needs +# dev/k3s_agent_container.sh up (E2E_SIZED_NODE names +# the node; default k3s-agent-b). set -euo pipefail MODEL="${E2E_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}" @@ -49,6 +53,13 @@ cleanup() { run_is release --all --yes >/dev/null 2>&1 run_is gc --evict --yes >/dev/null 2>&1 run_is stack down >/dev/null 2>&1 # the gateway, UI and database too + if [ "${SIZED_LABELLED:-0}" = 1 ]; then # the fake GPU labels, off again + for node in "$NODE_A" "$NODE_B"; do + kubectl label node "$node" nvidia.com/gpu.product- nvidia.com/gpu.memory- >/dev/null 2>&1 + done + kubectl patch node "$NODE_B" --subresource=status --type=json \ + -p '[{"op":"remove","path":"/status/capacity/nvidia.com~1gpu"}]' >/dev/null 2>&1 + fi # nothing managed may remain on the cluster, pass or fail leftover=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ -l infer-stack/managed=true -o name 2>/dev/null | wc -l) @@ -175,8 +186,11 @@ if [ "${E2E_MAKE_ROOM:-0}" = 1 ]; then EOF run_is acquire e2e-warm --yes --timeout "$TIMEOUT" --env-file "$WORK/warm.env" run_is release --env-file "$WORK/warm.env" --yes # idle, still resident - if ! run_is acquire e2e-big --yes --timeout "$TIMEOUT" \ - --env-file "$WORK/big.env" 2>&1 | tee "$WORK/big.log" | grep -q 'ready: True'; then + # Into a file first: `grep -q` quits at its match, and under pipefail the + # writer's SIGPIPE would fail the check. + run_is acquire e2e-big --yes --timeout "$TIMEOUT" \ + --env-file "$WORK/big.env" > "$WORK/big.log" 2>&1 || true + if ! grep -q 'ready: True' "$WORK/big.log"; then echo '!! the leased model never became ready' >&2; exit 1 fi grep -q 'making room for leased demand' "$WORK/big.log" \ @@ -185,9 +199,75 @@ EOF run_is release --env-file "$WORK/big.env" --yes fi +if [ "${E2E_SIZED:-0}" = 1 ]; then + # Needs a second node (dev/k3s_agent_container.sh up) and the sized-* + # profiles of dev/e2e_tests/kubeai-cpu-values.yaml. Fake GPU labels stand + # in for GPU Feature Discovery's; cleanup removes them. + echo '== min_vram_gib picks the smallest resource profile whose GPUs fit' + NODE_A=$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{.items[0].metadata.name}') + NODE_B="${E2E_SIZED_NODE:-k3s-agent-b}" + kubectl label node "$NODE_A" nvidia.com/gpu.product=FAKE-24G nvidia.com/gpu.memory=24576 --overwrite >/dev/null + kubectl label node "$NODE_B" nvidia.com/gpu.product=FAKE-80G nvidia.com/gpu.memory=81920 --overwrite >/dev/null + kubectl patch node "$NODE_B" --subresource=status --type=merge \ + -p '{"status":{"capacity":{"nvidia.com/gpu":"2"}}}' >/dev/null + SIZED_LABELLED=1 + cat >> "$WORK/config/catalog.yaml" < -> " " of its Model's pod + for _ in $(seq 60); do + out=$(kubectl -n "$NAMESPACE" get models.kubeai.org -l infer-stack/managed=true -o json \ + | python3 -c ' +import json, sys +for m in json.load(sys.stdin)["items"]: + if sys.argv[1].replace("/", "-").lower() in m["metadata"]["name"]: + print(m["metadata"]["name"], m["spec"]["resourceProfile"])' "$1") + name=${out%% *}; profile=${out##* } + node=$(kubectl -n "$NAMESPACE" get pods -l "model=$name" -o jsonpath='{.items[0].spec.nodeName}' 2>/dev/null || true) + [ -n "$node" ] && { echo "$profile $node"; return; } + sleep 2 + done + echo "$profile unscheduled" + } + run_is acquire e2e-sized-big --wait false --yes --env-file "$WORK/big-sized.env" >/dev/null + got=$(where e2e-sized-big) + [ "$got" = "sized-80g:1 $NODE_B" ] || { echo "!! 40 GiB went to: $got" >&2; exit 1; } + echo " min_vram_gib 40 -> $got" + run_is acquire e2e-sized-small --yes --timeout "$TIMEOUT" --env-file "$WORK/small-sized.env" \ + > "$WORK/small-sized.log" 2>&1 || true + grep -q 'ready: True' "$WORK/small-sized.log" \ + || { echo '!! the 10 GiB endpoint never became ready' >&2; exit 1; } + got=$(where e2e-sized-small) + [ "$got" = "sized-24g:1 $NODE_A" ] || { echo "!! 10 GiB went to: $got" >&2; exit 1; } + echo " min_vram_gib 10 -> $got, and it answers" + run_is catalog suggest --backend kubeai > "$WORK/suggest.yaml" 2> "$WORK/suggest.err" + grep -q 'nvidia-fake-80g' "$WORK/suggest.err" \ + || { cat "$WORK/suggest.err" >&2; echo '!! suggest proposed no profile for FAKE-80G' >&2; exit 1; } + echo ' catalog suggest proposes a resource profile per GPU product' + run_is release --env-file "$WORK/big-sized.env" --yes >/dev/null + run_is release --env-file "$WORK/small-sized.env" --yes >/dev/null +fi + if [ "${GATEWAY:-1}" = 1 ] && [ "${E2E_DYNAMIC:-1}" = 1 ]; then echo '== dynamic routing: two --dedicated leases on one model, two Models, one alias' run_is stack down >/dev/null 2>&1 # the gateway comes back with Postgres + # A routing-mode change is adopted only by a quiescent stack: wait for the + # last managed pod (a released Model's) to be gone. + for _ in $(seq 90); do + [ -z "$(kubectl -n "$NAMESPACE" get pods -l infer-stack/managed=true -o name)" ] && break + sleep 2 + done run_is config set dynamic_routing true for n in 1 2; do run_is acquire "$ALIAS" --dedicated --yes --ttl 30m --timeout "$TIMEOUT" \ diff --git a/dev/lessons/lessons.md b/dev/lessons/lessons.md index ecdeb16c..31b2bbfb 100644 --- a/dev/lessons/lessons.md +++ b/dev/lessons/lessons.md @@ -107,3 +107,21 @@ evidence; prefer append-only; supersede incorrect entries with a new one. leaves nothing behind. - **Applies when:** a test exercises size-dependent logic (VRAM floors, disk checks, download sizes). + +- **Lesson:** A kwconf flag (`isflag=True`) takes an optional value, so a + flag written before a positional swallows it: `logs -f qwen` parsed as + `follow='qwen'` with no names. infer-stack's flags are all boolean, so + `_FlagSafeMixin` hands a non-boolean string back to the positional list. +- **Evidence / MWE:** `tests/test_day2.py::test_a_flag_never_swallows_the_positional_after_it`; + `infer_stack/cli/options.py` (`reclaim_swallowed_positionals`). Found live + 2026-09-26: `logs -f ` followed every instance. +- **Applies when:** a kwconf command has both flags and positional + arguments. + +- **Lesson:** `producer | grep -q pattern` under `set -o pipefail` can fail + although the pattern matched: `grep -q` exits at the first match and the + producer's next write dies of SIGPIPE (141). Write to a file, then grep it. +- **Evidence / MWE:** `dev/lessons/mwe/grep_q_pipefail.sh` prints 141 for + the pipeline and 0 for the file. +- **Applies when:** an e2e or handover script checks a command's output with + `grep -q` under `set -euo pipefail`. diff --git a/dev/lessons/mwe/grep_q_pipefail.sh b/dev/lessons/mwe/grep_q_pipefail.sh new file mode 100644 index 00000000..d85acfd8 --- /dev/null +++ b/dev/lessons/mwe/grep_q_pipefail.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +# `producer | grep -q pattern` under `set -o pipefail` can report failure +# although the pattern matched: grep -q exits at its first match, the +# producer's next write gets SIGPIPE (exit 141), and pipefail reports that. +set -o pipefail +seq 1 1000000 | grep -q '^1$'; echo "pipeline: $? (141 = the producer's SIGPIPE, not a miss)" +seq 1 1000000 > /tmp/grep-q-mwe.txt; grep -q '^1$' /tmp/grep-q-mwe.txt; echo "file: $?" +rm -f /tmp/grep-q-mwe.txt diff --git a/docs/backend-parity.md b/docs/backend-parity.md index de60913f..e776a625 100644 --- a/docs/backend-parity.md +++ b/docs/backend-parity.md @@ -88,7 +88,8 @@ a different mechanism. **gap**: missing on one side and on the roadmap. | where a deployment lands | local planner over `nvidia-smi` | the cluster scheduler, via `resource_profile:` | | GPU count from TP × PP × DP | same | same | | `placement.gpu_indices`, `allowed_gpus`, `skip_display_gpus` | yes | n/a: no host indices; node-scoped resource profiles are the equivalent | -| `placement.min_vram_gib`, `infer-stack measure` | yes | **gap** (P4): `min_vram_gib` is warned and ignored; `measure` runs but is unverified on a GPU, and `--record` writes an overlay KubeAI does not read | +| `placement.min_vram_gib` (declared, or recorded by `measure --record`) | picks GPUs that large | ≈ picks the smallest resource profile whose nodes' GPUs are that large (GFD's `nvidia.com/gpu.memory` label); verified with fake labels, a real-GPU run is `dev/handover/p4_gpu_labels.sh` | +| `catalog suggest` | sized to this host's GPUs | ≈ sized to the largest GPU node, plus a `resourceProfiles` block per GPU product | | GPU allocations recorded in the ledger | yes | none: the cluster owns them | | more than one host | boundary | yes: the reason the backend exists | diff --git a/docs/kubeai-backend.md b/docs/kubeai-backend.md index 8cf548f1..7524f12f 100644 --- a/docs/kubeai-backend.md +++ b/docs/kubeai-backend.md @@ -87,6 +87,23 @@ endpoints: resource_profile: nvidia-gpu-rtx-4090 # -> nvidia-gpu-rtx-4090:2 ``` +### Sizing: `min_vram_gib` picks a profile + +An endpoint that names no `resource_profile` but declares +`placement.min_vram_gib` (or has one recorded by `infer-stack measure +--record`) gets the smallest resource profile whose GPUs are that large. A +profile's GPU size is the `nvidia.com/gpu.memory` label (set by GPU Feature +Discovery) of the nodes its `nodeSelector` selects; a profile without a +`nodeSelector`, like the chart's generic ones, has no size and is never +picked this way. With none large enough, the `kubeai_resource_profile` +default is used, or the acquire is refused with the sizes it found. +`infer-stack catalog suggest` proposes one sized profile per GPU product in +the cluster, ready for the helm values: + +```bash +infer-stack catalog suggest --backend kubeai # catalog on stdout, profiles on stderr +``` + Verify the setup before the first acquire — `doctor` checks the chain in dependency order (cluster reachable → CRD installed → namespace → gateway): diff --git a/docs/planning/backend-parity-roadmap.md b/docs/planning/backend-parity-roadmap.md index 6e5aee23..503f2975 100644 --- a/docs/planning/backend-parity-roadmap.md +++ b/docs/planning/backend-parity-roadmap.md @@ -1,8 +1,8 @@ # Backend parity roadmap: KubeAI as a superset of Compose **Status:** proposed 2026-09-25 · **P0 done** 2026-09-24 on -`dev/backend-unification` · **P1–P3, P6 done** 2026-09-26 · P4, P5 not -started. Execution order: [../queue.md](../queue.md). +`dev/backend-unification` · **P1–P4, P6 done** 2026-09-26 (P4's GPU run is +a handover) · P5 not started. Execution order: [../queue.md](../queue.md). **Current state:** [../backend-parity.md](../backend-parity.md). **Origin:** the scale-up run needs more than one workstation, and the KubeAI backend had drifted from Compose for three months before the @@ -184,6 +184,17 @@ the right profile on a cluster with two sizes. **Size:** small. Shrinks "the information you add"; skip if nobody runs a mixed-GPU cluster. +**P4 done 2026-09-26** on two k3s nodes (the second a container, +`dev/k3s_agent_container.sh`) with fake GPU labels: `min_vram_gib 40` got +the 80 GiB profile and scheduled on that node, `10` got the 24 GiB one and +answered, and `catalog suggest` proposed a profile per product +(`E2E_SIZED=1 dev/kubeai_e2e.sh`). A profile's size is its selected nodes' +`nvidia.com/gpu.memory`; the chart's generic profiles have no selector, so +no size. KubeAI keeps its own measurements overlay and fills a missing +`min_vram_gib` from it, as compose does, so `measure --record` feeds the +choice. Real GPU labels, a GPU serving the chosen profile, and `measure` +reading vLLM's memory lines are `dev/handover/p4_gpu_labels.sh`. + ### P5. The multi-workstation shape **Closes:** the gateway on one host, the port-forward default. diff --git a/docs/queue.md b/docs/queue.md index 49d92e8d..9bb99396 100644 --- a/docs/queue.md +++ b/docs/queue.md @@ -79,7 +79,12 @@ fake-Compose and fake-KubeAI backends; `tests/test_parity.py` runs each **Done when:** every *same* row has a parity test, and the CI suite runs it. -### 6. [ ] P4: placement from node labels (verified with faked labels) +### 6. [x] P4: placement from node labels (verified with faked labels) + +Done 2026-09-26; the GPU run is `dev/handover/p4_gpu_labels.sh`. +*Why reordered:* two GPU sizes need two nodes (a node has one +`nvidia.com/gpu.memory` label), so item 7's simulated second node +(`dev/k3s_agent_container.sh`) was built first, here. `catalog suggest` on KubeAI proposes `resourceProfiles` from `nvidia.com/gpu.product` / `.memory` node labels; `min_vram_gib` picks the @@ -159,6 +164,12 @@ Seed findings, already known: - [x] `infer-stack logs -f qwen` followed every instance: a kwconf flag took the next word as its value. Fixed for every command (P2). - [ ] `status` shows STALE during an apply instead of "apply in progress". +- [ ] Open WebUI writes its data directory as root: removing a data root + (the e2e's cleanup, or an operator's `rm -rf`) fails with permission + denied. Found by `dev/kubeai_e2e.sh` 2026-09-26. +- [ ] A refused acquire always ends with "free a GPU first — …", also when + the reason is a render refusal (a served-name collision) or the + backend is kubeai, where no GPU is ours to free. Found 2026-09-26. ### 10. [ ] Handover diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index 1852860c..c9382288 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -156,6 +156,103 @@ def _model_doc( return doc +#: GPU Feature Discovery's node labels: the GPU model, and per-GPU memory (MiB). +GPU_PRODUCT_LABEL = 'nvidia.com/gpu.product' +GPU_MEMORY_LABEL = 'nvidia.com/gpu.memory' +GPU_RESOURCE = 'nvidia.com/gpu' + + +def node_gpus(nodes: list[dict[str, Any]]) -> dict[str, dict[str, Any]]: + """Each node's GPUs, from its labels and allocatable: ``{node: facts}``. + + ``facts`` is ``{'product', 'memory_gib', 'count', 'labels'}``; a node with + no memory label has ``memory_gib`` None. + + >>> n = {'metadata': {'name': 'a', 'labels': {GPU_PRODUCT_LABEL: 'L4', + ... GPU_MEMORY_LABEL: '23034'}}, + ... 'status': {'allocatable': {GPU_RESOURCE: '2'}}} + >>> node_gpus([n])['a']['memory_gib'], node_gpus([n])['a']['count'] + (22.49, 2) + """ + out = {} + for node in nodes: + meta = node.get('metadata') or {} + labels = dict(meta.get('labels') or {}) + memory = labels.get(GPU_MEMORY_LABEL) + try: + memory_gib = round(int(memory) / 1024, 2) if memory else None + except ValueError: + memory_gib = None + count = str(((node.get('status') or {}).get('allocatable') or {}).get(GPU_RESOURCE) or '') + out[str(meta.get('name'))] = { + 'product': labels.get(GPU_PRODUCT_LABEL), + 'memory_gib': memory_gib, + 'count': int(count) if count.isdigit() else 0, + 'labels': labels, + } + return out + + +def sized_profiles(profiles: dict[str, Any], nodes: dict[str, dict[str, Any]]) -> dict[str, float]: + """Per-GPU memory (GiB) of each resource profile that says where it runs. + + A profile's size is the smallest ``nvidia.com/gpu.memory`` among the nodes + its ``nodeSelector`` selects. A profile without a ``nodeSelector`` (the + chart's generic ones) could land anywhere, so it has no size and is never + picked by size. + + >>> nodes = {'a': {'memory_gib': 24.0, 'labels': {'gpu': 'small'}}, + ... 'b': {'memory_gib': 80.0, 'labels': {'gpu': 'big'}}} + >>> sized_profiles({'small': {'nodeSelector': {'gpu': 'small'}}, + ... 'big': {'nodeSelector': {'gpu': 'big'}}, + ... 'anywhere': {}}, nodes) + {'big': 80.0, 'small': 24.0} + """ + sizes = {} + for name, profile in sorted((profiles or {}).items()): + selector = (profile or {}).get('nodeSelector') or {} + if not selector: + continue + found = [facts['memory_gib'] for facts in nodes.values() + if facts.get('memory_gib') + and all(facts['labels'].get(k) == str(v) for k, v in selector.items())] + if found: + sizes[name] = min(found) + return sizes + + +def pick_profile(explicit: str | None, min_vram_gib: float | None, + sized: dict[str, float], default: str | None) -> tuple[str | None, str]: + """``(profile, why)`` for one deployment. + + An endpoint's own ``resource_profile`` wins. Else, with ``min_vram_gib``, + the smallest sized profile whose GPUs have that much memory. Else the + ``kubeai_resource_profile`` default. ``why`` says which rule chose, or + why nothing could. + + >>> sized = {'small': 24.0, 'big': 80.0} + >>> pick_profile(None, 40, sized, 'cpu') + ('big', 'min_vram_gib 40 GiB -> big (80 GiB GPUs)') + >>> pick_profile(None, 10, sized, None)[0], pick_profile('mine', 40, sized, None)[0] + ('small', 'mine') + >>> pick_profile(None, 100, sized, None) + (None, 'min_vram_gib 100 GiB: no resource profile has GPUs that large (big 80 GiB, small 24 GiB)') + """ + if explicit: + return explicit, 'runtime.resource_profile' + if min_vram_gib: + fitting = sorted((size, name) for name, size in sized.items() + if size >= float(min_vram_gib)) + if fitting: + size, name = fitting[0] + return name, f'min_vram_gib {min_vram_gib:g} GiB -> {name} ({size:g} GiB GPUs)' + if not default: + known = ', '.join(f'{n} {g:g} GiB' for n, g in sorted(sized.items())) or 'none sized' + return None, (f'min_vram_gib {min_vram_gib:g} GiB: no resource profile has ' + f'GPUs that large ({known})') + return default, 'kubeai_resource_profile' + + class RenderedModels: """Output of the render half: manifest text + bookkeeping maps.""" @@ -165,6 +262,8 @@ def __init__(self) -> None: self.request_names: dict[str, str] = {} # endpoint -> CR name self.unrenderable: set[str] = set() self.errors: list[str] = [] + #: deployment id -> which rule chose its resource profile. + self.profile_reasons: dict[str, str] = {} @property def text(self) -> str: @@ -179,6 +278,7 @@ def render_models( namespace: str, default_resource_profile: str | None, unique_names: bool = False, + sized: dict[str, float] | None = None, ) -> RenderedModels: """Render the desired set into KubeAI ``Model`` docs (pure, no I/O). @@ -215,10 +315,10 @@ def render_models( 'VLLM Model does not. Use --backend compose for this endpoint.' ) continue - profile = ( - runtime.get('resource_profile') or default_resource_profile or '' - ) - if not str(profile).strip(): + min_vram = (deployment.spec.get('placement') or {}).get('min_vram_gib') + profile, why = pick_profile(runtime.get('resource_profile'), min_vram, + sized or {}, default_resource_profile) + if not str(profile or '').strip(): out.unrenderable.add(deployment.id) out.errors.append( f'{deployment.id}: no resource profile — set ' @@ -226,8 +326,10 @@ def render_models( 'resourceProfiles key from your KubeAI helm values, e.g. ' "'nvidia-gpu-rtx-4090') or `config set " 'kubeai_resource_profile ` as the default.' + + (f' ({why})' if min_vram else '') ) continue + out.profile_reasons[deployment.id] = why name = model_name(deployment, unique=unique_names) if name in out.models: out.unrenderable.add(deployment.id) @@ -322,6 +424,12 @@ def __init__( # IP, which containers on a cluster node can reach; a gateway off the # cluster needs an ingress URL here. self.gateway_upstream = gateway_upstream + # `infer-stack measure --record` writes here, as on compose; a + # measurement fills an endpoint's missing min_vram_gib, which picks + # its resource profile by size. + from ..leasing.vram import Measurements + + self.measurements = Measurements(self.state_dir / 'measurements.json') # -- state-dir plumbing -------------------------------------------------- @@ -473,6 +581,92 @@ def _set_front_door(self, desired, rendered: RenderedModels) -> None: #: and admission commits an empty allocation for every deployment. allocates_gpus = False + #: Seconds a read of the cluster's GPU facts is reused: resource profiles + #: and node labels change rarely, and a render reads them once. + GPU_FACTS_TTL = 60.0 + #: The KubeAI chart's configuration, which holds its resourceProfiles. + CONFIG_MAP = 'kubeai-config' + + def gpu_facts(self) -> tuple[dict[str, dict[str, Any]], dict[str, float]]: + """``(each node's GPUs, each sized profile's GPU GiB)`` from the cluster. + + Empty when the cluster cannot say (no GPU labels, no chart config): + sizing then falls back to the default profile, as before. + """ + import time + + cached = getattr(self, '_gpu_facts_cache', None) + if cached is not None and time.monotonic() - cached[0] < self.GPU_FACTS_TTL: + return cached[1] + try: + nodes = (json.loads(self._kubectl(['get', 'nodes', '-o', 'json']) or '{}') + .get('items') or []) + config = json.loads(self._kubectl( + ['get', 'configmap', self.CONFIG_MAP, '-o', 'json']) or '{}') + system = yaml.safe_load(((config.get('data') or {}).get('system.yaml')) or '') or {} + facts = node_gpus(nodes) + sized = sized_profiles(system.get('resourceProfiles') or {}, facts) + except Exception: # noqa: BLE001 - sizing is best-effort, never fatal + facts, sized = {}, {} + self._gpu_facts_cache = (time.monotonic(), (facts, sized)) + return facts, sized + + def suggestion_inventory(self) -> tuple[dict[str, Any], dict[str, Any]]: + """``(inventory, resourceProfiles)`` for ``catalog suggest``. + + The inventory is the GPUs of the largest GPU node (a Model runs on one + node, so that is the biggest thing it can use), shaped like a host + inventory. The profiles are one per GPU product in the cluster, each + selecting that product's nodes: what the helm values need for + ``min_vram_gib`` to choose by size. + """ + facts, _ = self.gpu_facts() + gpu_nodes = [f for f in facts.values() if f['count'] and f['memory_gib']] + if not gpu_nodes: + return {'gpu_count': 0, 'gpus': []}, {} + best = max(gpu_nodes, key=lambda f: (f['count'] * f['memory_gib'], f['memory_gib'])) + gpus = [{'index': i, 'name': best['product'] or 'GPU', + 'memory_gib': best['memory_gib'], + 'memory_mib': int(best['memory_gib'] * 1024), + 'display_active': False} for i in range(best['count'])] + profiles = {} + for f in sorted(gpu_nodes, key=lambda f: f['memory_gib']): + product = f['product'] or 'gpu' + profiles[f'nvidia-{dns_slug(product)}'] = { + 'imageName': 'nvidia-gpu', + 'runtimeClassName': 'nvidia', + 'requests': {GPU_RESOURCE: '1'}, + 'limits': {GPU_RESOURCE: '1'}, + 'nodeSelector': {GPU_PRODUCT_LABEL: product}, + } + return {'gpu_count': len(gpus), 'gpus': gpus}, profiles + + def _enrich_min_vram(self, desired) -> None: + """Fill a missing ``min_vram_gib`` from a recorded measurement (in memory). + + The compose resolution order: a catalog-declared value wins, else the + measurements overlay. Never persisted. + """ + from ..leasing.vram import measurement_key_for_spec + + for g in desired: + placement = dict(g.spec.get('placement') or {}) + if g.engine != 'vllm' or placement.get('min_vram_gib'): + continue + measured = self.measurements.get_min_vram_gib(measurement_key_for_spec(g.spec)) + if measured: + placement.update(min_vram_gib=measured, min_vram_source='measured') + g.spec['placement'] = placement + + def _needs_sizing(self, desired) -> bool: + """Whether any deployment asks for its profile by GPU memory.""" + for g in desired: + runtime = g.spec.get('runtime') or {} + if (g.spec.get('placement') or {}).get('min_vram_gib') and not runtime.get( + 'resource_profile'): + return True + return False + def _render_documents(self, desired: list[Deployment]): """``(plan, rendered, planned)`` in memory: the one KubeAI render. @@ -481,11 +675,14 @@ def _render_documents(self, desired: list[Deployment]): """ from ..leasing.placement import GpuPlan + desired = list(desired) + self._enrich_min_vram(desired) rendered = render_models( - list(desired), + desired, namespace=self.namespace, default_resource_profile=self.default_resource_profile, unique_names=self.dynamic_routing, + sized=self.gpu_facts()[1] if self._needs_sizing(desired) else None, ) plan = GpuPlan( assignments={g.id: [] for g in desired if g.id not in rendered.unrenderable}, @@ -539,19 +736,11 @@ def converge(self, desired: list[Deployment], *, apply: bool = True, placement=N self.namespace, ', '.join(sorted(g.id for g in desired)) or '(none)', ) - for g in desired: - # Warn-and-ignore by decision (vram-aware-placement.md, - # Resolutions #3): k8s owns placement on this backend; the - # equivalent mechanism is the resourceProfile / resource - # requests, not our single-host planner. - if (g.spec.get('placement') or {}).get('min_vram_gib'): - logger.warning( - ' {}: placement.min_vram_gib is ignored on the ' - 'kubeai backend (k8s owns placement — express the ' - 'requirement via the resource profile instead)', - g.id, - ) plan, rendered, planned = self._render_documents(desired) + for gid, why in sorted(rendered.profile_reasons.items()): + if why.startswith('min_vram_gib'): + # The cluster places, within the profile chosen by size. + logger.info(' {}: resource profile by {}', gid, why) self.last_errors = list(rendered.errors) self.last_unplaced = set(rendered.unrenderable) self.last_assignments = {} # the cluster places diff --git a/infer_stack/cli/commands_catalog.py b/infer_stack/cli/commands_catalog.py index d68e62e0..83b0c5e7 100644 --- a/infer_stack/cli/commands_catalog.py +++ b/infer_stack/cli/commands_catalog.py @@ -269,6 +269,12 @@ class CatalogSuggestCLI( Pure + offline: ``--simulate-hardware 2x80`` suggests for hardware you do not have in front of you. + On the kubeai backend the hardware is the cluster's: GPU Feature + Discovery's node labels (``nvidia.com/gpu.product`` / ``.memory``) and each + node's allocatable GPUs. The catalog is sized to the largest GPU node, and + a ``resourceProfiles`` block (one per GPU product, selecting its nodes) is + printed for the helm values, so ``min_vram_gib`` can pick a profile by size. + infer-stack catalog suggest # render only (no write) infer-stack catalog suggest --simulate-hardware 4x48 infer-stack catalog suggest --apply # merge into the catalog @@ -284,6 +290,11 @@ class CatalogSuggestCLI( False, isflag=True, help='With --apply, overwrite catalog entries that already exist.', ) + backend = kw.Value( + None, type=str, + help='Whose hardware (default: the configured `backend` setting): this ' + "host's GPUs, or on kubeai the cluster's.", + ) @classmethod def main(cls, argv=True, **kwargs): @@ -292,16 +303,40 @@ def main(cls, argv=True, **kwargs): migrate_known_suggestion_aliases, suggest_catalog, ) - from .commands_leasing import _resolve_skip_display + from ..paths import get_setting + from .commands_leasing import _make_backend, _resolve_skip_display from .context import effective_inventory config = cls.cli(argv=argv, data=kwargs) - inventory = effective_inventory(config) or detect_inventory() + profiles: dict = {} + cluster = (config.backend or get_setting('backend')) == 'kubeai' + inventory = effective_inventory(config) + if inventory is None and cluster: + backend = _make_backend(config) + inventory, profiles = backend.suggestion_inventory() + if not inventory['gpus']: + print( + 'no cluster node reports GPUs: suggest needs each GPU node\'s ' + 'allocatable nvidia.com/gpu and GPU Feature Discovery\'s ' + 'nvidia.com/gpu.product / .memory labels (the NVIDIA device ' + 'plugin with gfd.enabled=true; see the README). Pass ' + '--simulate-hardware NxM to plan without them.', + file=sys.stderr, + ) + return 1 + inventory = inventory or detect_inventory() gpus = inventory.get('gpus') or [] - skip_display = _resolve_skip_display(config) + skip_display = _resolve_skip_display(config) and not cluster frag = suggest_catalog( inventory, reserve_display_gpu='auto' if skip_display else False ) + if cluster: + # A cluster has no host GPU indices: the profile says where. + for endpoint in frag['endpoints'].values(): + placement = endpoint.get('placement') or {} + placement.pop('gpu_indices', None) + if not placement: + endpoint.pop('placement', None) max_mem = max((g.get('memory_gib') or 0 for g in gpus), default=0) n_display = sum(1 for g in gpus if g.get('display_active')) @@ -322,6 +357,10 @@ def main(cls, argv=True, **kwargs): text = yaml.safe_dump(frag, sort_keys=False, default_flow_style=False) + if cluster: + hw = f'the cluster\'s largest GPU node: {hw}' + values = (yaml.safe_dump({'resourceProfiles': profiles}, sort_keys=False) + if profiles else '') if not config.apply: print( f'# suggested for: {hw} ({usable} usable)\n' @@ -330,6 +369,10 @@ def main(cls, argv=True, **kwargs): file=sys.stderr, ) _print_yaml(text) + if values: + print('# helm values for the KubeAI chart (merge, then ' + '`scripts/install_kubeai.sh `):\n' + values, + file=sys.stderr) return 0 # --apply: additive merge into the catalog (keep existing entries). @@ -356,6 +399,10 @@ def main(cls, argv=True, **kwargs): f' kept existing (pass --force to overwrite): ' f'{", ".join(skipped)}' ) + if values: + print('resource profiles for the KubeAI chart (merge into your helm ' + 'values, then `scripts/install_kubeai.sh `):') + print(values, end='') return 0 diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index 6acb5e48..922a6ea0 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -1594,8 +1594,8 @@ def main(cls, argv=True, **kwargs): logs_fn = getattr(backend, 'deployment_logs', None) if logs_fn is None: raise SystemExit( - 'measure needs the compose backend ' - '(the engine container log is the measurement source).' + 'measure reads the engine\'s log, and this backend runs no engine ' + '(set `--backend compose` or `kubeai`).' ) catalog = _requests_catalog(controller, config) name = config.endpoint @@ -1669,8 +1669,7 @@ def main(cls, argv=True, **kwargs): store = getattr(backend, 'measurements', None) if store is None: raise SystemExit( - '--record needs the compose backend measurements ' - 'overlay.' + '--record: this backend keeps no measurements overlay.' ) store.record( key, value, endpoint=name, profile=profile, diff --git a/tests/test_leasing_kubeai.py b/tests/test_leasing_kubeai.py index 91b1d950..dbc16253 100644 --- a/tests/test_leasing_kubeai.py +++ b/tests/test_leasing_kubeai.py @@ -921,3 +921,92 @@ def test_the_front_doors_settings_are_in_the_recovery_profile(tmp_path): assert other.gateway.ui is True and other.dynamic_routing is True other.use_profile({**profile, 'gateway': True}) # a profile from before assert other.dynamic_routing is True # keeps what it has + + +# -- placement information: profiles chosen by GPU size (P4) ------------------ + +SIZED_PROFILES = { + 'small': {'nodeSelector': {'nvidia.com/gpu.product': 'FAKE-24G'}}, + 'big': {'nodeSelector': {'nvidia.com/gpu.product': 'FAKE-80G'}}, + 'generic': {'limits': {'nvidia.com/gpu': '1'}}, # the chart's kind +} + + +def _gpu_node(name, product, mib, count=2): + return {'metadata': {'name': name, 'labels': { + 'nvidia.com/gpu.product': product, 'nvidia.com/gpu.memory': str(mib)}}, + 'status': {'allocatable': {'nvidia.com/gpu': str(count)}}} + + +class SizedKubectl(FakeKubectl): + """A cluster with a 24 GiB node and an 80 GiB node, and the chart's config.""" + + def __call__(self, args): + if len(args) > 4 and args[3] == 'get' and args[4] == 'nodes': + return json.dumps({'items': [_gpu_node('a', 'FAKE-24G', 24576), + _gpu_node('b', 'FAKE-80G', 81920)]}) + if len(args) > 4 and args[3] == 'get' and args[4] == 'configmap': + system = yaml.safe_dump({'resourceProfiles': SIZED_PROFILES}) + return json.dumps({'data': {'system.yaml': system}}) + return super().__call__(args) + + +def _sized(gid, min_vram, **runtime): + dep = vllm(gid, profile=None, **runtime) + dep.spec['placement'] = {'min_vram_gib': min_vram} + return dep + + +def test_min_vram_gib_picks_the_smallest_profile_that_fits(tmp_path): + kubectl = SizedKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + be.converge([_sized('grp-a', 40), _sized('grp-b', 10)]) + profiles = {doc['metadata']['labels']['infer-stack/deployment']: doc['spec']['resourceProfile'] + for doc in kubectl.applied.values()} + assert profiles == {'grp-a': 'big:1', 'grp-b': 'small:1'} + + +def test_an_explicit_profile_wins_and_too_large_is_refused(tmp_path): + kubectl = SizedKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + explicit = _sized('grp-a', 40, resource_profile='generic') + be.converge([explicit, _sized('grp-b', 200)], apply=False) + assert be.last_unplaced == {'grp-b'} + assert any('no resource profile has GPUs that large' in e for e in be.last_errors) + (doc,) = yaml.safe_load_all(be.models_file.read_text()) + assert doc['spec']['resourceProfile'] == 'generic:1' + + +def test_sizing_reads_the_cluster_only_when_asked(tmp_path): + kubectl = SizedKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + be.converge([vllm('grp-a')], apply=False) + assert not any('nodes' in call or 'configmap' in call for call in kubectl.calls) + + +def test_catalog_suggest_on_kubeai_sizes_to_the_cluster(tmp_path, monkeypatch, capsys): + from infer_stack.cli import commands_catalog, commands_leasing + + kubectl = SizedKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + monkeypatch.setattr(commands_leasing, '_make_backend', lambda config, **kw: be) + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path / 'config')) + assert commands_catalog.CatalogSuggestCLI.main(argv=['--backend', 'kubeai']) == 0 + out, err = capsys.readouterr() + assert "largest GPU node" in err + values = yaml.safe_load(err[err.index('resourceProfiles:'):])['resourceProfiles'] + assert values['nvidia-fake-80g']['nodeSelector'] == {'nvidia.com/gpu.product': 'FAKE-80G'} + assert 'gpu_indices' not in out # no host indices on a cluster + + +def test_a_recorded_measurement_picks_the_profile(tmp_path): + """`measure --record` works on kubeai: the overlay fills min_vram_gib.""" + from infer_stack.leasing.vram import measurement_key_for_spec + + kubectl = SizedKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + dep = vllm('grp-a', profile=None) + be.measurements.record(measurement_key_for_spec(dep.spec), 50.0, endpoint='grp-a') + be.converge([dep], apply=False) + (doc,) = yaml.safe_load_all(be.models_file.read_text()) + assert doc['spec']['resourceProfile'] == 'big:1' From 08dccd95a727df9fd8ff2137f29f4f0f9cd7c907 Mon Sep 17 00:00:00 2001 From: Jon Crall Date: Sat, 26 Sep 2026 13:08:28 -0400 Subject: [PATCH 34/34] Queue: record the P4 commit Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe --- docs/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/queue.md b/docs/queue.md index 9bb99396..0426a973 100644 --- a/docs/queue.md +++ b/docs/queue.md @@ -81,7 +81,7 @@ fake-Compose and fake-KubeAI backends; `tests/test_parity.py` runs each ### 6. [x] P4: placement from node labels (verified with faked labels) -Done 2026-09-26; the GPU run is `dev/handover/p4_gpu_labels.sh`. +Done 2026-09-26, `94c263d`; the GPU run is `dev/handover/p4_gpu_labels.sh`. *Why reordered:* two GPU sizes need two nodes (a node has one `nvidia.com/gpu.memory` label), so item 7's simulated second node (`dev/k3s_agent_container.sh`) was built first, here.