diff --git a/CHANGELOG.md b/CHANGELOG.md index ebbc081b..57289221 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,158 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### The gateway can run inside the cluster + +`infer-stack config set kubeai_gateway cluster` runs the LiteLLM gateway as a +Deployment and a NodePort Service in the KubeAI namespace instead of on the +host running infer-stack, so no single host is in every request's path and +no `kubectl port-forward` is needed: an env file's `OPENAI_BASE_URL` is a +node's address on the NodePort (or `kubeai_gateway_url`), and every node +answers. It is the same gateway, key and route registry; `secrets rotate` +updates its Secret and rolls it, `doctor` checks it, and `stack down` +removes it. It takes static routes only. `docs/kubeai-backend.md` gains +"Add a workstation", the runbook for a second GPU machine. + +### KubeAI picks a resource profile by GPU size + +On the kubeai backend, an endpoint with `placement.min_vram_gib` and no +`resource_profile` gets the smallest resource profile whose nodes' GPUs are +that large, read from GPU Feature Discovery's `nvidia.com/gpu.memory` label; +before, `min_vram_gib` was warned about and ignored. `infer-stack measure +--record` works there too, and a recorded measurement feeds the same choice. +`infer-stack catalog suggest` on kubeai sizes the catalog to the cluster's +largest GPU node and proposes a resource profile per GPU product for the +helm values. `dev/k3s_agent_container.sh` adds a second (simulated) node to +a development cluster. + +### A parity suite for the two backends + +`tests/test_parity.py` runs one scenario per row that `docs/backend-parity.md` +marks *same*, on the compose backend over a fake Docker and on the kubeai +backend over a fake kubectl, from the client contract to the TUI. The TUI no +longer swaps an injected Docker runner for the real one. + +### The gateway in front of a cluster is the compose gateway + +On the kubeai backend the gateway now takes the same settings as on compose: +Open WebUI (`ui`, on by default), the reverse proxy, and dynamic routing, +under which each deployment is its own Model (`-`), so the +same model acquired `--dedicated` twice runs as two Models behind one alias. +Catalog endpoints are routed before they run, as on compose, so a new Model +no longer recreates the gateway. The gateway's changes are shown and +approved with the acquire's, before its lease commits. `routes list / seed / +prune` work on kubeai and show a cluster route's upstream (they showed `?`), +and `gc --orphans` and `clean --orphans` run there (and find nothing: the +backend only reads its own labelled pods). + +### `ps`, `logs`, `status` and the TUI read the backend, on either backend + +`infer-stack ps` lists what the backend runs, containers or pods and the +gateway, in one table that says what each serves; `infer-stack logs` takes an +endpoint alias, an instance name, a container id prefix or a deployment id, +and `-f` keeps following instances that start or restart later. Both read the +same strict residency the controller decides with, so they work on KubeAI; +`status` does too, and says `starting` for an instance that is up but not yet +ready. `stack up` is now `apply` and `stack down` stops everything on either +backend; the raw Compose verbs (`stack compose -- …`, `restart`, `pull`, +`start`, `stop`) act on the Compose project on this host, which is the +gateway's on KubeAI. The TUI's runtime pane (formerly "docker") follows pod +logs, lists pods, and its Apply / Down buttons work on KubeAI. + +Fixed on the way: every kubectl call from the TUI failed, because the TUI +replaced the backend's command runner with Docker's, whose environment has no +`KUBECONFIG`; the TUI's "engines" log view included Open WebUI and Postgres; +and a flag before a positional ate it (`logs -f qwen` followed everything, +`acquire --yes qwen` named no endpoint), on every command. + +### One acquire path + +The controller's pre-admission branch is gone: every backend, including the +dry-run backend, acquires, renders, renews and publishes through admission. +A backend with only `realize` / `teardown` gets the admission surface from +`SimpleAdmission`. Two behaviours changed with it. While residency cannot +be read, no acquire is admitted; before, a request needing no new GPU was +admitted and then failed at its render, after the lease was committed. And +`routes prune` computes the desired set with the same view the render uses. + +### KubeAI acquires go through admission, as Compose's do + +An acquire on the KubeAI backend is now previewed in memory and commits its +lease only if every deployment renders, the same path Compose takes. A +refused acquire (no resource profile, an ollama endpoint, a served-name +collision) writes no lease and runs no `kubectl`; before, the lease was +committed, rendered, and then rolled back. A `--queue` acquire of such an +endpoint fails at once instead of waiting out its timeout. The cluster +schedules, so a KubeAI deployment commits an empty GPU allocation; `leases` +still shows no GPU for it, and a `--no-apply` acquire now reports it as +cluster-scheduled rather than unplaced. An idle +keep-warm Model is kept only while its pod has started, as a keep-warm +container is on Compose. The approval digest moved into the scaffold both +backends share, and one function now answers whether a backend allocates +GPUs. Removing the older acquire branch, which only test fakes still use, +is the next step (roadmap P1b). + +### Compose and KubeAI: a parity matrix and a roadmap + +`docs/backend-parity.md` states the relationship the two backends are meant +to have (KubeAI is Compose plus a scheduler: more information supplied, more +tools installed, the same catalog, verbs, env file and TUI) and records, row +by row, where that holds today, where it does not, and which differences are +deliberate. `docs/planning/backend-parity-roadmap.md` plans the rest in five +phases with exit criteria, from one acquire path to the gateway inside the +cluster for a multi-workstation run. The README's KubeAI section, which still +described the pre-leasing `setup` / `deploy` workflow and a live patch that +is no longer needed, now points at the current docs and keeps only the +cluster prerequisites and the `kubectl` debugging checks. + +### Endpoints|models and leases|deployments are tabs + +The sidebar shows Endpoints and Models as two tabs instead of two panes split +by a divider, so the endpoint list gets the whole height; the main area does +the same for Leases and Deployments. A tab's label carries its counts +(`Leases 1/3`), so the hidden one is still readable. Each pair is one switch +(`InferStackTUI.TABBED_CATALOG`, `TABBED_TABLES`); setting it to `False` +restores the draggable split, and both layouts are tested. + +### One Clean up, in the footer, with a CLI command behind it + +The leases and deployments panes each had a Clean up button, and both did the +same thing: forget released/expired leases and stopped deployments. Nothing +running changes. That is now one `x Clean up` in the footer, and the same +action on the CLI is `infer-stack gc --forget`. + +The TUI log now names the CLI command for the actions that were missing one: +`r` (`status`), Clean up (`gc --forget`), saving settings (`config set …`), +and the API tab's send, test-all (`test `) and model list (a `curl` +that reads the key through `infer-stack env` and never prints it). Poll +intervals are TUI-only, and the log says so. + +### The CLI is built on kwconf instead of scriptconfig + +kwconf is scriptconfig's successor, which aiq-magnet, cmd_queue and kwdagger +already use. `Value`, `ModalCLI` and `__command__` carry over; `DataConfig` +is now `Config`. One behavioural difference needed care: kwconf reads the +strings `null`, `true` and numbers on the command line as values, so +`--backend null` arrived as `None`. Options whose choices are strings now +take `type=str`. The program name stays `infer-stack` in usage and errors +(`__prog__`). + +`uv.lock` and `requirements/locks/tests.txt` were regenerated: kwconf 0.11.0 +added, scriptconfig removed. The uv `exclude-newer` cutoff (2026-06-04), which +predated every kwconf release, is removed; the locked versions are unchanged. + +### The TUI stops repainting what did not change + +Every refresh tick repainted the lease and deployment panes, because setting +a pane's border title repaints it even when the text is unchanged: about +7 KB/s of terminal output with nothing on screen changing, which is what an +SSH session feels. Titles are now set only on change (75 bytes/s idle). +Streamed engine logs are drawn in batches of up to 200 lines every 100 ms +instead of one UI message per line, and a flood keeps its newest lines. A +watchdog reports any UI-thread stall over 0.5 s in the TUI log, with the +stack it was stuck in written to the error log file. `dev/profile_tui.py` +measures event-loop lag for idle, docker-pane and log-flood cases. + ### Ledger reads are safe across threads The ledger shares one SQLite connection between threads, but only write @@ -23,6 +175,73 @@ scheduler with its own GPU accounting: a lease held outside it, manual or keep-warm, occupies a GPU the scheduler believes is free, and the job it places there cannot start. +### An idle keep-warm model gives way to a leased one on KubeAI too + +The rule: a keep-warm model that no lease holds is always a candidate for +eviction when a model with a lease needs a resource. Compose admission already +applied it when placing. On KubeAI the cluster schedules and never evicts, so +a leased Model could sit `Unschedulable` behind idle warm ones until its +timeout. A readiness probe can now report `needs_room`; the wait then evicts +the longest-idle deployment, one per 30 s, never a leased one, and never +after the wait's deadline. Verified on k3s: a leased Model that did not fit +beside an idle one was ready 107 s later, with the idle one evicted. + +### The gateway is its own module + +The front door (LiteLLM config and service, route registry, dynamic-route +reconciliation, the managed keys, Open WebUI, the reverse proxy) moved out of +`leasing/compose.py` into `leasing/gateway.py`, with a `Gateway` object that +`ComposeBackend` owns (`backend.gateway`). The service naming rules moved to +`leasing/naming.py`. Backends now hand the gateway their route rows rather +than the gateway reading backend state. Rendered output is unchanged +(verified byte for byte), and `compose.py` is about a third smaller. Code +that imported gateway names from `infer_stack.leasing.compose` should import +them from `infer_stack.leasing.gateway`; the public naming helpers are still +importable from `compose`. + +### The kubeai backend fails fast on an engine that cannot start + +The crash diagnosis (restart count, exit code, and the engine log classified +as fatal, transient or unknown) was Docker-only, so on a cluster a model that +could never start held its lease for the whole timeout. It now lives in +`infer_stack/leasing/diagnosis.py` and both backends use it. The kubeai +backend reads pods through a new strict `residency()` (a kubectl failure +raises instead of reading as "nothing running") and quotes the crashed run's +log (`kubectl logs --previous`). A not-ready wait names the pod's reason, such +as `Unschedulable` or `ImagePullBackOff`. vLLM rejecting a flag (`error: +unrecognized arguments`) is now recognised as fatal on both backends; before, +it waited for two restarts. + +`runtime.env` now works on the kubeai backend (the Model's `spec.env`, with +the same templates and reserved names). The served-name rule lives in one +place, `leasing.models.served_name`. Two gateway route builders fell back to +the deployment id where the engine used the first served alias, so a +deployment without `served_model_name` routed to a name its engine did not +serve. + +`gc --orphans` and `network migrate` now refuse a non-compose backend +explicitly. They had used "the backend has `residency`" to mean compose. + +### The kubeai backend puts the LiteLLM gateway in front of the cluster + +On the kubeai backend a client had to name a model by its KubeAI Model name +(a DNS slug of the served name), not the endpoint alias. Cards send the +alias, so a card that ran on the compose backend got HTTP 404 on a cluster +(verified on k3s). The kubeai backend now runs the same LiteLLM gateway, +routing each alias to its Model: one `OPENAI_BASE_URL`, the managed key, and +the alias as the model name on both backends. `secrets rotate` works on it. +`--no-litellm` keeps the old direct access. New setting: +`kubeai_gateway_upstream`, for a gateway that cannot reach the cluster +Service's IP. + +The profile-era KubeAI renderer (`infer_stack.backends.render_kubeai_artifacts`) +is removed. Nothing called it since the profile path was excised; the kubeai +backend is the one renderer for KubeAI. + +`dev/kubeai_e2e.sh` now sends the alias as a card does, and fails when the +request fails. Before, a failed generation fell through to PASS: the check +sat in a `&&` list, where `set -e` does not apply. + ### Custom container launches are catalog data, not recipes `runtime.serve_recipe` is gone. An endpoint whose image has its own launcher diff --git a/Makefile b/Makefile index 68465960..4a94f8ec 100644 --- a/Makefile +++ b/Makefile @@ -1,34 +1,18 @@ PYTHON ?= python -EXAMPLE ?= single-node - -init: - $(PYTHON) manage.py init - -render: - $(PYTHON) manage.py render - -deploy: - $(PYTHON) manage.py deploy +# The KubeAI chart values: your resourceProfiles (see docs/kubeai-backend.md). +VALUES ?= kubeai-values.yaml status: $(PYTHON) manage.py status -smoke-test: - $(PYTHON) manage.py smoke-test - -example-single-node: - cp examples/single-node/config.yaml ./config.yaml - cp examples/single-node/models.yaml ./models.yaml - @echo "Copied single-node example into repo root. Edit hostname/model ids as needed." +render: + $(PYTHON) manage.py render bootstrap-k3s: bash scripts/bootstrap_k3s.sh install-kubeai: - bash scripts/install_kubeai.sh generated/kubeai/kubeai-values.yaml kubeai - -bootstrap-single-node: example-single-node bootstrap-k3s - @echo "Single-node example copied and K3s bootstrapped. Next: python manage.py render && python manage.py deploy" + bash scripts/install_kubeai.sh $(VALUES) kubeai port-forward-kubeai: kubectl -n kubeai port-forward svc/kubeai 8000:80 diff --git a/README.md b/README.md index 41a2877a..5c89183f 100644 --- a/README.md +++ b/README.md @@ -4,14 +4,10 @@ [![Python versions](https://img.shields.io/pypi/pyversions/infer-stack.svg)](https://pypi.org/project/infer-stack/) [![License](https://img.shields.io/pypi/l/infer-stack.svg)](https://github.com/AIQ-Kitware/infer_stack/blob/main/LICENSE) -> **Heads up — the leasing model is now the primary workflow.** Declare models -> in a catalog (`infer-stack catalog …`) and `acquire`/`run` endpoints -> on demand; see `docs/source/manual/` (the Ollama + Open WebUI tutorial and the -> leasing demo) and `infer-stack help tree`. The -> **named stack profiles** documented below (`setup`/`render`/`up`/`switch`/…) -> are the pre-leasing model and now live under **`infer-stack legacy `** -> (e.g. `infer-stack legacy render`). This README still describes that legacy -> flow; a leasing-oriented rewrite is pending. +Declare models in a catalog (`infer-stack catalog …`) and `acquire` or `run` +endpoints on demand. `infer-stack help tree` prints the whole command surface; +[docs/source/manual/](docs/source/manual/) has the Ollama + Open WebUI +tutorial and the leasing demo. ## Primary leasing workflow @@ -23,7 +19,12 @@ infer-stack catalog suggest --apply infer-stack acquire ``` -`config.yaml` and `catalog.yaml` are the user configuration. Leasing keeps an +No GPU here? `infer-stack catalog suggest --simulator --apply` adds +`mock-smol`, a simulator that answers like vLLM with random text, so the same +three steps run on any host with Docker ([docs/mock-endpoints.md](docs/mock-endpoints.md)). + +`settings.yaml` (`infer-stack config …`) and `catalog.yaml` are the user +configuration. Leasing keeps an internal frozen recovery snapshot so a crash cannot re-render committed state with different settings, but ordinary `acquire` advances that snapshot automatically. Compatible catalog additions can be acquired while other models @@ -62,49 +63,46 @@ Operational and security constraints that are accepted during the current planning-stage release are tracked in [docs/planning/known-limitations.md](docs/planning/known-limitations.md). -`infer_stack` manages **named stack profiles** for local and Kubernetes-backed inference. - -A stack profile is a small graph made from: - -* **providers** — inference runtimes such as vLLM and Ollama -* **gateways** — optional API routers such as LiteLLM -* **frontends** — optional UIs such as Open WebUI -* **routes** — optional public model aliases exposed through a gateway - -This repo can render those profiles through two backends: +`infer-stack` serves the endpoints declared in a catalog. `acquire` takes a +lease on an endpoint; the controller places its engine on free GPUs and +reconciles the backend to run it: -* **Compose** for local single-host serving. Compose supports vLLM, Ollama, optional LiteLLM, and optional Open WebUI. -* **KubeAI** for Kubernetes-backed vLLM serving. KubeAI support is vLLM-only for now. +* **engines**: vLLM (one container per deployment) and Ollama (one daemon per + `runtime_hosts` entry, serving many tags); +* **LiteLLM gateway**: one OpenAI base URL, `http://127.0.0.1:14042/v1`, in + front of every endpoint alias. On by default; `config set litellm false` + drops it; +* **Open WebUI**: on by default at `http://127.0.0.1:13000`; + `config set ui false` or `acquire --no-ui` drops it; +* **reverse proxy**: an optional single-port nginx in front of both. -The direct Ollama path can run without LiteLLM and without predeclaring models. vLLM profiles still use explicit runtimes, placement, and runtime settings. +Two backends run this: **Compose** (single host; vLLM and Ollama) and +**KubeAI** (a Kubernetes cluster; vLLM only). ## Main commands ```bash -infer-stack setup --backend compose --profile ollama-direct -# or: infer-stack setup --backend compose --profile qwen2-5-7b-instruct-turbo-default -infer-stack list-profiles -infer-stack describe-profile -infer-stack validate -infer-stack render -infer-stack up -d -infer-stack deploy -infer-stack switch --apply # re-render and converge; no separate up needed -infer-stack status -infer-stack smoke-test -infer-stack version # print the installed version -infer-stack config paths # show where config / artifacts / caches live +infer-stack config init # data dir + default backend -> settings.yaml +infer-stack catalog suggest --apply # seed catalog.yaml from this host's GPUs +infer-stack catalog show # what can be acquired +infer-stack acquire # lease, render, bring up, wait for a real generation +infer-stack test # one generation through the gateway +infer-stack leases # desired vs running, per deployment +infer-stack status # paths, backend and a lease summary +infer-stack release --all # drop every lease +infer-stack paths # where settings, catalog, ledger and caches live +infer-stack version +infer-stack help tree # the whole command surface ``` -The CLI is built on [`scriptconfig`](https://gitlab.kitware.com/utils/scriptconfig), -so every subcommand is also importable as a Python class — useful for -notebooks, tests, and other scripts: +The CLI is built on [`kwconf`](https://github.com/Erotemic/kwconf), +so every subcommand is also importable as a Python class: ```python -from infer_stack.cli import RenderCLI, SmokeTestCLI +from infer_stack.cli import AcquireCLI, TestCLI -RenderCLI.main(argv=False, profile="qwen2-5-7b-instruct-turbo-default", yes=True) -SmokeTestCLI.main(argv=False, model="qwen/qwen2.5-7b-instruct-turbo") +AcquireCLI.main(argv=False, names=['smol135-1'], yes=True) +TestCLI.main(argv=False, name='smol135-1') ``` `manage.py` and `infer-stack` are aliases for the same entry point; @@ -112,247 +110,216 @@ shell examples below use `infer-stack`. ## Operating the rendered Compose stack -Once the stack is up, common docker compose operations are available as -`infer-stack` subcommands so you don't have to `cd` into the rendered -output directory or repeat the `-f docker-compose.yml --env-file .env` -flags. They all resolve the rendered location via the same -`output.generated_dir` chain as the rest of the CLI. +`ps` and `logs` read the backend directly; `stack` wraps `docker compose` on +the rendered project, so you never `cd` into it or repeat `-f`/`--env-file`. ```bash -infer-stack ps # docker compose ps -infer-stack ps -a # include stopped -infer-stack logs -f open-webui # follow one service -infer-stack logs --tail=200 litellm vllm-* # tailored backlog -infer-stack logs -f --raw litellm # full LiteLLM tracebacks -infer-stack restart open-webui # restart specific services -infer-stack stop # stop everything (no remove) -infer-stack start # start back up -infer-stack pull # refresh images +infer-stack ps # engines, gateway, UI +infer-stack ps -a # include exited instances +infer-stack logs -f # follow whatever serves an endpoint +infer-stack logs --tail 200 litellm # the gateway's backlog +infer-stack logs -f --raw litellm # full LiteLLM tracebacks +infer-stack stack restart open-webui # docker compose restart +infer-stack stack stop # stop everything (no remove) +infer-stack stack start # start it back up +infer-stack stack pull # refresh images +infer-stack stack compose -- ps --format json # any other docker compose command ``` -Interactive ``infer-stack logs -f`` compacts only explicitly registered, known-noisy -LiteLLM traceback shapes; unknown tracebacks pass through unchanged. Redirected or -piped output stays raw, and ``--raw`` disables compaction in an interactive follow. -The compacted CLI path preserves Compose ANSI service colors when attached to a TTY; -``--no-color`` still disables them. The TUI uses the same conservative compactor when -LiteLLM logs are visible. +`logs` accepts a service or pod name, a container id prefix, a deployment id +or an endpoint alias. Interactive `infer-stack logs -f` compacts only +explicitly registered, known-noisy LiteLLM traceback shapes; unknown +tracebacks pass through unchanged. Redirected or piped output stays raw, and +`--raw` disables compaction in an interactive follow. `--no-color` drops the +name-prefix colors. The TUI uses the same compactor. -For Ollama model management inside the rendered Ollama service, prefer the -CLI wrappers: +Ollama tags are pulled into the daemon on the first `acquire` of an endpoint +that serves them. `stack compose` reaches the daemon for anything else, e.g. +`infer-stack stack compose -- exec ollama-local-ollama ollama list`. + +On the KubeAI backend `ps` and `logs` read pods, and the `stack` Compose verbs +act on the gateway's Compose project on this host. + +## Inspect an endpoint before running it ```bash -infer-stack ollama-pull smollm2:135m -infer-stack ollama-list -infer-stack ollama-ps +infer-stack catalog show +infer-stack acquire --no-apply # declare + write the compose project; start nothing +infer-stack paths leasing # where docker-compose.yml landed +infer-stack apply # start it (or `release --all` to discard) ``` -For other interactive one-shot commands inside a container, use -`infer-stack logs`, `infer-stack ps`, `infer-stack restart`, or fall back to raw -Compose only when no wrapper exists. - -On the KubeAI backend these wrappers raise ``NotImplementedError`` — -use the equivalent ``kubectl`` commands in the meantime. +## Catalog model -## Inspect a profile before running it +`catalog.yaml` is the one user-edited description of what can run. Its +sections: -```bash -infer-stack describe-profile qwen2-5-7b-instruct-turbo-default --format yaml -``` +* `models`: weight sources (`hf://org/name`, with optional `revision`, + `quantization`, `dtype`); +* `endpoints`: served API names. Each picks an `engine` (`vllm` or `ollama`), + a `model` (a `models` key, or an Ollama tag), and optionally `runtime` + (vLLM settings), `protocol`, `placement`, `sharing` and `reclaim`; +* `runtime_hosts`: Ollama daemons, each with its GPUs and daemon settings; +* `bundles`: named lists of endpoints to acquire together. -## Stack profile model +```yaml +models: + smol135: + source: hf://HuggingFaceTB/SmolLM2-135M-Instruct -Profiles are written as stack graphs. The main sections are `providers`, `gateways`, `frontends`, and `routes`. For details and examples, see [docs/stack-graph-profiles.md](docs/stack-graph-profiles.md). +endpoints: + smol135-1: + engine: vllm + model: smol135 + runtime: {max_model_len: 8192} + chat: + engine: ollama + host: local-ollama + model: qwen3.5:4b -Common shapes: +runtime_hosts: + local-ollama: + engine: ollama + placement: {gpu_indices: [0]} + settings: {keep_alive: 30m} -```text -Open WebUI -> Ollama # ollama-direct, no LiteLLM -Open WebUI -> LiteLLM -> vLLM # classic vLLM compose profiles -Open WebUI -> LiteLLM -> Ollama # Ollama with stable aliases -Open WebUI -> LiteLLM -> Ollama + vLLM # mixed migration / test stacks -Ollama API + vLLM API directly # raw backend profiles +bundles: + both: [smol135-1, chat] ``` -Custom provider models and custom profiles live in the configured `catalog.user_models_file`, which defaults to `~/.config/infer_stack/models.yaml`. New files should prefer provider-specific top-level keys: +Edit it with `infer-stack catalog model|endpoint|host|bundle add …`, or by +hand with `infer-stack catalog edit`; `infer-stack catalog validate` checks it. +The schema reference is the docstring of `infer_stack/leasing/catalog.py`. -```yaml -vllm_models: - my-vllm-model: - hf_model_id: org/model - -ollama_models: - my-ollama-model: - tag: qwen3.5:4b - -profiles: - my-stack: - providers: {} - gateways: {} - frontends: {} - routes: {} +Shapes the stack renders: + +```text +Open WebUI -> LiteLLM -> vLLM / Ollama # the default +Open WebUI -> vLLM / Ollama # config set litellm false ``` -`models:` is still interpreted as a vLLM model catalog for convenience, but new docs and recipes use `vllm_models:` / `ollama_models:`. +The named stack profiles of earlier releases (`setup`, `switch`, +`--profile`) are gone; see +[docs/stack-graph-profiles.md](docs/stack-graph-profiles.md). ## Where config and rendered artifacts live -`infer-stack` follows XDG basedir conventions, so where you invoke it -from never changes which config it reads or where it writes rendered -artifacts: - -There are exactly two path roots: +`infer-stack` follows XDG basedir conventions, so the directory you invoke it +from never changes which config it reads or where it writes. There are two +roots: | What | Default location | How to relocate | | --- | --- | --- | -| `config.yaml`, `models.yaml`, `kubeai-values.local.yaml` | `~/.config/infer_stack/` (resp. `$XDG_CONFIG_HOME`) | `--config-dir` (or `INFER_STACK_CONFIG_DIR`) | -| **Everything generated** — `generated/` (docker-compose.yml, .env, plan.yaml, kubeai/*) **and** `state/` (hf-cache, postgres volumes, Ollama store, runtime bind mounts) | `~/.local/share/infer_stack/` (resp. `$XDG_DATA_HOME`) | `--data-dir` (or `INFER_STACK_DATA_DIR`) | +| `settings.yaml`, `catalog.yaml` | `~/.config/infer_stack/` (resp. `$XDG_CONFIG_HOME`) | `--config-dir` or `INFER_STACK_CONFIG_DIR` | +| **Everything generated**: `leasing/` (the ledger, the compose project, its `.env`) and the bind-mounted state (`hf-cache/`, `vllm-cache/`, `open-webui/`, `ollama/`, …) | `~/.local/share/infer_stack/` (resp. `$XDG_DATA_HOME`) | `config set data_dir `, `--data-dir` or `INFER_STACK_DATA_DIR` | + +`infer-stack paths` prints every resolved path and whether it exists. -`--data-dir` is the single knob for "put everything I generate in one -directory." It **relocates the one infer-stack installation controlling a +The data dir **relocates the one infer-stack installation controlling a host/backend; it does not create an isolated second installation**. Do not run controllers from multiple config/data roots against the same Docker host or Kubernetes namespace. See the [single-owner limitation](docs/planning/known-limitations.md#one-control-plane-per-host-or-backend-namespace). -Set it once at `setup`; it is baked into the absolute -`state.*` and `output.generated_dir` paths written to `config.yaml`, so -later commands don't need it again: - ```bash -# All rendered artifacts and bind-mount state land under one directory. -infer-stack setup \ - --backend compose \ - --profile ollama-direct \ - --data-dir /data/service/docker/vllm-stack +# Persist it once; later commands read it from settings.yaml. +infer-stack config set data_dir /data/service/docker/infer-stack -infer-stack render --yes +# Or keep both roots in a checkout for an ad-hoc experiment. +export INFER_STACK_CONFIG_DIR=$PWD/cfg INFER_STACK_DATA_DIR=$PWD/stack +infer-stack paths ``` -```bash -# Keep config.yaml in a checkout for ad-hoc experiments. -infer-stack setup --config-dir $PWD --backend compose --profile

--data-dir $PWD/stack -``` - -`--config-dir` / `--data-dir` live on every subcommand, so they appear -**after** the subcommand name. For "set once for the whole shell" use the -env vars instead. For a bespoke split layout (e.g. big `state/` on a data -disk, artifacts elsewhere), edit `state.*` / `output.generated_dir` in -`config.yaml` directly. +`--config-dir` / `--data-dir` are accepted by every subcommand, after the +subcommand name. ## Constraining placement to specific GPUs -If some of your GPUs are tied up by other work, restrict the planner -(and the rendered ``device_ids``) to the subset you want it to use: - ```bash # Only place onto GPU 1 (e.g. GPU 0 is running a display). -infer-stack render --yes --profile test-single-11gb --allowed-gpus 1 +infer-stack acquire --allowed-gpus 1 -# Or pin a TP=2 profile to physical GPUs 1 and 3. -infer-stack render --yes --profile test-multi-gpu --allowed-gpus 1,3 +# Or confine a TP=2 endpoint to physical GPUs 1 and 3. +infer-stack acquire --allowed-gpus 1,3 ``` -``--allowed-gpus`` (or ``INFER_STACK_ALLOWED_GPUS=1,3``) filters the -detected inventory before placement — real indices are preserved, so -the rendered compose stack pins ``device_ids: ["1", "3"]`` to those -exact physical GPUs. Useful for integration tests that need to share a -host with other jobs. - -## Demos / integration recipes - -End-to-end examples under [docs/demos/](docs/demos/) are written as -markdown tutorials. The CI smoke test is runnable with pytest-codeblocks: - -```bash -pytest --codeblocks docs/demos/ci_smoke_test.md -``` +`--allowed-gpus` (or `INFER_STACK_ALLOWED_GPUS=1,3`) filters the detected +inventory before placement for that call only. Real indices are preserved, so +the rendered compose stack pins `device_ids` to those exact GPUs. The durable +forms live in data: -Each ``bash`` block is a self-contained shell snippet you can also -copy-paste into a terminal. See -[docs/demos/ci_smoke_test.md](docs/demos/ci_smoke_test.md) for the -``setup → describe → validate → render`` flow on the smallest test -profiles. +* `placement: {gpu_indices: [1]}` on a vLLM endpoint pins it exactly (the list + length must equal tp×pp×dp); Ollama daemons pin through their + `runtime_hosts` entry; +* `placement: {min_vram_gib: 24}` makes smaller GPUs ineligible; +* `config set skip_display_gpus true` (or `--skip-display-gpus`) leaves the + GPU driving a monitor free. -For a real running vLLM stack on a workstation, see -[docs/demos/quickstart.md](docs/demos/quickstart.md). For direct Ollama on a dual GTX 1080 Ti style host, see -[docs/demos/ollama_direct_quickstart.md](docs/demos/ollama_direct_quickstart.md). For a focused GPU-1 backend switch test, see [docs/demos/smollm2_gpu1_backend_switch.md](docs/demos/smollm2_gpu1_backend_switch.md). +## Demos / integration recipes -User-supplied paths on the CLI (`--file`, `--from-file`, -`--resource-profiles-file`, `--output-dir`) still resolve against the -current working directory — they're meant to behave as typed. +The user manual under [docs/source/manual/](docs/source/manual/) has two +walkthroughs on the current CLI: +[the Ollama + Open WebUI tutorial](docs/source/manual/ollama-openwebui-tutorial.md) +and [the leasing demo](docs/source/manual/leasing-demo.md) (standing service, +Open WebUI, several models side by side). --- ## Backend 1: Compose -Use Compose for local single-host deployments. It can render direct Ollama stacks, vLLM stacks, mixed Ollama+vLLM stacks, and raw backend-only stacks. +Use Compose for single-host serving. It runs vLLM and Ollama engines, mixed +freely, behind the optional gateway and UI. ### Getting started -Prerequisite: Docker and the `docker compose` plugin must be installed. +Prerequisite: Docker and the `docker compose` plugin. ```bash -# Direct Ollama, no LiteLLM and no predeclared models. -infer-stack setup --backend compose --profile ollama-direct -infer-stack validate --simulate-hardware 2x11 -infer-stack render --yes --simulate-hardware 2x11 -infer-stack up -d - -# Classic vLLM through LiteLLM/Open WebUI. -infer-stack setup --backend compose --profile qwen2-5-7b-instruct-turbo-default -infer-stack validate -infer-stack render -infer-stack up -d -``` - -### Test that it is responding +infer-stack config init --backend compose +infer-stack doctor --gpu +infer-stack catalog init -When LiteLLM is enabled, the default Compose front door is: +# A vLLM endpoint. +infer-stack catalog model add smol135 --source hf://HuggingFaceTB/SmolLM2-135M-Instruct +infer-stack catalog endpoint add --model smol135 # -> smol135-1 +infer-stack acquire smol135-1 -```text -http://127.0.0.1:14042/v1 +# An Ollama endpoint, on a daemon pinned to GPU 0. +infer-stack catalog host add local-ollama --engine ollama --gpu 0 +infer-stack catalog endpoint add chat --engine ollama --host local-ollama --model qwen3.5:4b +infer-stack acquire chat ``` -When using `ollama-direct`, Open WebUI talks to Ollama directly and the Ollama API is available at: - -```text -http://127.0.0.1:11434 -http://127.0.0.1:11434/v1 -``` +`infer-stack catalog suggest --apply` fills the catalog with endpoints sized +for the detected GPUs instead. -unless you changed the relevant ports in config. +### Test that it is responding -Wait until the active profile can serve a real request through its resolved default endpoint: +With LiteLLM enabled, every endpoint is reachable by its alias at: -```bash -infer-stack wait-ready +```text +http://127.0.0.1:14042/v1 ``` -`wait-ready` is stronger than Docker Compose health: it probes the user-facing -LiteLLM, Ollama, or direct vLLM access surface and, by default, requires a tiny -generation/completion to succeed. The smoke test runs this readiness probe by -default before issuing its normal test request: +`acquire` already blocks until the endpoint returns a real generation through +that front door, which is stronger than Docker's container health. After +`acquire --no-wait`, block separately; to check again later, send one request: ```bash -infer-stack smoke-test +infer-stack wait smol135-1 +infer-stack test smol135-1 +infer-stack test chat --prompt "Name three colors." --max-tokens 32 ``` -For direct Ollama profiles, pull a model first and then smoke-test that model: +Clients read the front door and the managed key from the env file: ```bash -infer-stack ollama-pull qwen3.5:4b -infer-stack ollama-list -infer-stack smoke-test --model qwen3.5:4b +export OPENAI_BASE_URL=$(infer-stack env OPENAI_BASE_URL) +export OPENAI_API_KEY=$(infer-stack env LITELLM_MASTER_KEY) ``` -For LiteLLM profiles, `smoke-test` reads the rendered `.env` automatically and -uses the active profile's resolved OpenAI-compatible front door. You can inspect -individual secrets when needed: - -```bash -infer-stack env LITELLM_MASTER_KEY -infer-stack env VLLM_BACKEND_API_KEY -``` +or get both, plus per-endpoint names, from +`infer-stack acquire --env-file lease.env`. To replace the gateway's master key (refused while leases are active; the gateway restarts, and clients must fetch the key again): @@ -361,244 +328,111 @@ gateway restarts, and clients must fetch the key again): infer-stack secrets rotate ``` -When you intentionally want the old quick behavior, skip the readiness wait: - -```bash -infer-stack smoke-test --no-wait --model gpt2 -``` - ### Stop it ```bash -infer-stack down +infer-stack release --all # drop every lease +infer-stack release --all --evict # ...and stop the engines now +infer-stack clean -f # no leases, nothing on a GPU; the gateway stays +infer-stack stack down # docker compose down, bypassing the ledger ``` -`down` never removes named volumes. The Postgres data directory and the -Open WebUI volume are preserved across `down`, `up`, `switch`, and `render`. +After a plain `release`, `keep-warm` endpoints (the default `reclaim` policy) +stay loaded until another lease needs their GPUs or `infer-stack evict` stops +them. `stack down` releases no lease, so the next `apply` or `acquire` brings +leased models back. -### Open WebUI authentication +All state is bind-mounted from the data dir, so none of these delete it, +including `stack down --volumes`. For a destructive reset, remove the +directories `infer-stack paths` lists. -By default Open WebUI runs with `WEBUI_AUTH=False` — no login screen, -anyone who can reach the port gets straight into the UI. This is the -expected behavior for a local dev box. To re-enable login/signup, set -in `config.yaml`: - -```yaml -open_webui: - auth: true -``` +### Open WebUI authentication -and re-render. Existing accounts stored in the `postgres-open-webui` -volume are preserved across the toggle. +Open WebUI runs with `WEBUI_AUTH=False`: no login screen, and anyone who can +reach port 13000 gets the UI. No setting changes that. Keep the host on a +trusted network, or run without the UI (`config set ui false`). -### Reverse proxy (TLS) and LDAP +### Reverse proxy -Open WebUI can be fronted by an opt-in nginx TLS reverse proxy, and its -login can be backed by an LDAP directory. Both are off by default and -configured as ordinary config fields. The built-in `openwebui-tls-ldap` -profile wires them together as a worked example (Ollama + Open WebUI -behind nginx, no public Open WebUI/Ollama ports); see -[`examples/openwebui-tls-ldap/`](examples/openwebui-tls-ldap/). +An optional nginx service publishes one HTTP port with the UI at `/` and the +API at `/v1`. It needs the LiteLLM gateway. ```bash -infer-stack setup --backend compose --profile openwebui-tls-ldap +infer-stack config set reverse_proxy true # port 80 +infer-stack config set reverse_proxy '{enabled: true, port: 8080}' ``` -**Reverse proxy.** Enable it under `frontends.reverse_proxy`. It renders -an nginx service plus a generated `state.runtime/nginx.conf`: +`acquire --reverse-proxy` turns it on for one call. Add `config_path: +/path/to/nginx.conf` to the block to mount your own config at +`/etc/nginx/conf.d/default.conf` instead of the generated one. -```yaml -frontends: - reverse_proxy: - enabled: true - target: open_webui # or litellm / ollama / a custom upstream - server_name: host.example.com - ssl: - enabled: true - certificate: ./certs/site.crt - certificate_key: ./certs/site.key - dhparam: ./dhparam.pem # optional -``` - -When `ssl.enabled` is true, port 80 redirects to HTTPS (`force_https`) -and the cert/key/dhparam host paths are bind-mounted read-only. When -`ssl.enabled` is false, only HTTP is published (HTTPS publishing is -gated on TLS so you never get a `:443` mapping with nothing listening). - -> **Path caveat.** Relative `certificate`/`certificate_key`/`dhparam`/ -> `config_path` values are written verbatim into the generated -> `docker-compose.yml`, so Docker Compose resolves them **relative to the -> generated directory** (where the compose file lives), not your CWD. -> `infer-stack render` warns when a referenced cert or config file is not -> found. Use absolute paths if you want to avoid the ambiguity. - -**LDAP.** Enable it under `frontends.open_webui.ldap`. The directory -settings render as Open WebUI `LDAP_*` environment variables, and -secrets/site-specific values are emitted as `.env` placeholders -(`LDAP_HOST`, `LDAP_PASSWD`, `LDAP_SEARCH_BASE`, …) so you can fill them -in after the first render without re-touching the compose YAML: - -```yaml -frontends: - open_webui: - ldap: - enabled: true - env_defaults: - LDAP_PORT: '636' - LDAP_USE_TLS: 'true' - LDAP_ATTRIBUTE_FOR_USERNAME: uid -``` - -**Manual escape hatches.** When the typed renderer is not enough, drop -to manual control without leaving infer-stack: - -* `frontends.reverse_proxy.config_path` — mount an existing nginx config - file instead of rendering one. -* `frontends.reverse_proxy.extra_config` — inject extra directives into - the rendered HTTPS `server` block. -* Every rendered service (`ollama`, vLLM runtimes, `litellm`, - `open_webui`, `reverse_proxy`) accepts generic overrides: - `extra_env`, `env_file`, `extra_volumes`, `extra_hosts`, `labels`, - `additional_ports`, and `gpus` (scalar `all`/count or a structured - device-request list). - -Field precedence (lowest to highest) is: top-level config section -(`reverse_proxy:` / `open_webui:` / `ollama:`) → the matching -`frontends.*` / `providers.*` / `gateways.*` section → the active -profile. Newer configs should prefer the `frontends.*` / `providers.*` -form shown above. +It does no TLS and no authentication. Only the one port is published and no +certificate is mounted, so terminate TLS in a proxy in front of it. The TLS +and LDAP settings of the pre-leasing profiles no longer exist. ### Persistent state and database layout -Compose renders stateful services only when their components are enabled: +Everything lives under the data dir (`infer-stack paths`): -* `postgres-open-webui` — rendered only when Open WebUI is enabled. It stores chats, accounts, and settings in `state.postgres_open_webui`. -* `postgres-litellm` — rendered only when LiteLLM is enabled. It stores router state in `state.postgres_litellm`. -* `ollama` — rendered only when the Ollama provider is enabled. Its model store is `state.ollama`, mounted at `/root/.ollama`. -* vLLM runtimes mount `state.hf_cache` for Hugging Face weights and `state.vllm_cache` for compiled artifacts. +* `open-webui/`: Open WebUI's data directory (accounts, chats, settings), + mounted at `/app/backend/data`; +* `postgres-litellm/`: LiteLLM's route store, rendered only with + `config set dynamic_routing true`; +* `ollama/`: the Ollama model store, mounted at `/root/.ollama`; +* `hf-cache/`, `vllm-cache/`, `torch-cache/`, `triton-cache/`, `cuda-cache/`: + vLLM weights and compile caches (see + [docs/persistent-caches-and-warm-restarts.md](docs/persistent-caches-and-warm-restarts.md)); +* `runtime/`: directories an endpoint's `runtime.mounts` asks for; +* `leasing/`: the ledger and the rendered compose project. -Each Postgres container has its own `POSTGRES_DB`, `POSTGRES_USER`, and -`POSTGRES_PASSWORD`, sourced from component-specific `.env` keys. There is no shared Postgres instance and no `postgres-init` bootstrap service. - -Open WebUI chat history is **not** tied to the model currently being served, so after a profile switch old chats may reference model IDs the current gateway no longer advertises — that is expected. - -### Operational tips - -Prefer scoping commands to specific services rather than relying on -container names. Use only the services rendered by the active profile: - -```bash -# LiteLLM gateway profile -infer-stack logs -f litellm - -# Direct Ollama profile -infer-stack logs -f ollama - -# Ollama model store helpers -infer-stack ollama-list -infer-stack ollama-ps -``` - -You do not need to delete any volume during normal operation. If you -ever want a destructive reset, do it explicitly with -`docker compose down -v` against `generated/docker-compose.yml` — the -toolchain itself never does this. +Open WebUI chat history is not tied to the models currently served, so old +chats may name aliases the gateway no longer advertises. That is expected. ### Custom .env values are preserved -`generated/.env` is rewritten non-destructively. Any `KEY=value` pair -you add manually (for example `VERBOSE=1`, `HF_HOME=/data/hf`, or any -key this program does not yet know about) is preserved across -`render`, `setup`, `switch`, `up`, and `deploy`. Comments and the order -of existing lines are preserved where practical. +The compose project's `.env` holds the managed secrets (`LITELLM_MASTER_KEY`, +`HF_TOKEN`, …). `infer-stack env KEY=VALUE` merges a value into it, and keys +you add are kept across renders. Compose uses the file for interpolation, so a +key reaches a container only when the rendered service references it (as +`HF_TOKEN` does for vLLM). Set `HF_TOKEN` before the first `acquire` of a +gated model. -### Switching profiles +### Switching models + +There is no single active model to switch. `acquire` another endpoint and it +runs beside the first; release the first when you are done: ```bash -infer-stack switch --apply +infer-stack acquire smol135-1 +infer-stack acquire chat # now both are served +infer-stack leases # find smol135-1's lease id +infer-stack release ``` -`switch --apply` re-renders from the updated `config.yaml`, then brings the -stack up convergently with `--remove-orphans` so a separate `infer-stack up` is -not needed. Components/runtimes that are no longer in the rendered compose file -are dropped. Compose preserves existing containers whose service definitions did -not change. For vLLM-to-vLLM profile switches, unchanged Open WebUI stays up; -LiteLLM is refreshed through its admin API when possible. The live refresh -path treats LiteLLM's "model not found in db" response for config-backed -models as non-fatal, so switching aliases can add the new route without -tearing LiteLLM down. That can temporarily leave stale config-backed aliases -in `/v1/models`; restart LiteLLM manually only when you want to clean those -up. If Compose already created or recreated LiteLLM while converging the new -stack, no extra router refresh is attempted because the new container has -already loaded the freshly rendered YAML. Profiles that do not render -LiteLLM, such as direct Ollama profiles, skip the router refresh path even if -an old `runtime/litellm_config.yaml` file remains from a previous profile. -Switches that change Open WebUI's provider wiring, such as -`Open WebUI -> LiteLLM` to `Open WebUI -> Ollama`, necessarily recreate -Open WebUI because its environment changes. Postgres volumes and provider -caches are left untouched. vLLM runtime containers are named after their -Compose service, for example `vllm-chat`, so `docker ps` and -`infer-stack logs vllm-chat` clearly identify them as vLLM containers. +The gateway carries a route for every catalog endpoint, so adding or removing +a model does not recreate LiteLLM, and Open WebUI stays up. When GPUs are +short, `acquire` fails fast; `--queue` waits for a GPU instead, and idle +`keep-warm` deployments are evicted to make room. vLLM containers are named +after the served alias (`vllm-`), so `docker ps` and +`infer-stack logs vllm-` identify them. ### Protocol modes for base vs. instruct models -Profiles declare a `protocol_mode` (`chat` or `completions`) that the -served model must support. Models also declare which protocols they -support via `supported_protocols`. Validation runs before render and -fails with an actionable message if a profile asks for `chat` on a -completions-only model. - -Practical guidance: - -* Instruct/chat models (with a chat template) can use either, but - default to `chat`. -* Base models like Pythia, Llama-2 base, Mistral-v0.1 base, and Falcon - base do not define a chat template. Their HELM profiles use - `protocol_mode: completions` and the `smoke-test` command will - exercise `/v1/completions` for them. -* The rendered LiteLLM config uses `text-completion-openai/` - as the upstream provider for completions-only services. That means - even chat-shaped requests sent through Open WebUI to a Pythia model - get translated by LiteLLM into upstream `/v1/completions` calls — no - second vLLM container is needed to support Open WebUI for a - completions-only model. -* Open WebUI is still a chat UI, so prompt formatting matters. - HELM/eval clients should call `/v1/completions` directly for exact - prompt control rather than going through the chat frontend. - -### Chat-shaped clients on top of completions models - -Some clients (e.g. InspectAI / Inspect Evals stock MMLU tasks) only -speak `/v1/chat/completions` and cannot be reconfigured. For those -cases, profiles can opt into a LiteLLM-only adapter: - -```yaml -chat_compat: - enabled: true - strategy: flat_messages -``` - -When set on a `protocol_mode: completions` service, the rendered -LiteLLM config keeps the `text-completion-openai/` upstream -and adds LiteLLM's documented prompt-template fields -(`initial_prompt_value` / `roles` / `final_prompt_value`) so chat -messages get flattened into a plain prompt — no role labels, messages -joined by `\n` — before being forwarded to vLLM `/v1/completions`. - -This is **not** a chat tune; the model is still a base model and -prompt formatting still matters for evaluation. Use it only when a -chat-shaped client cannot be changed. The vLLM container is not -restarted, no `--chat-template` is rendered, and the adapter takes -effect after a `litellm`-only restart: +An endpoint's `protocol` is `chat` (the default) or `completions`. It decides +which surface the readiness probe and `infer-stack test` use, so a base model +without a chat template must declare `completions` or its `acquire` never sees +a ready generation: ```bash -infer-stack render -infer-stack restart litellm +infer-stack catalog model add pythia-160m --source hf://EleutherAI/pythia-160m +infer-stack catalog endpoint add --model pythia-160m --protocol completions +infer-stack acquire pythia-160m-1 +infer-stack test pythia-160m-1 # hits /v1/completions ``` -The built-in `pythia-inspect-mmlu-compat` profile is a ready-made -example; see -[`recipies/compose_pythia_inspect_mmlu_compat.md`](recipies/compose_pythia_inspect_mmlu_compat.md). +The gateway forwards `/v1/completions` unchanged, so evaluation clients that +need exact prompt control should call it directly. Open WebUI is a chat UI and +sends chat requests, which a base model cannot answer. ### Images with their own launcher @@ -608,34 +442,25 @@ endpoint's `runtime`; infer-stack has no model-specific code for it: ```yaml runtime: - image: ghcr.io/syv-ai/hyperqwen:sha-684e927 + image: example.org/my-vllm-launcher:1.0 max_model_len: 65536 gpu_memory_utilization: 0.93 command: [single] # replaces `vllm serve MODEL ` env: # container environment PORT: '{port}' - SPEC: dflash2 - CTX: fast - PREFIX_CACHE: 1 MAX_LEN: '{max_model_len}' # filled from the field above GPU_UTIL: '{gpu_memory_utilization}' EXTRA_ARGS: '--served-model-name={served_model_name}' mounts: # persisted under the runtime data dir - /app/models: hyperqwen/qwen3.8-27b/models - /cache: hyperqwen/qwen3.8-27b/cache + /app/models: my-launcher/models ``` -Switching that image to another context profile is a data edit, in the catalog -or the TUI's endpoint editor. HyperQwen's measured 3090 profiles are: - -- fast/default: `max_model_len: 65536`, `SPEC: dflash2`, `CTX: fast`; -- long: `max_model_len: 150000`, `SPEC: mtp`, `CTX: long`; -- huge: `max_model_len: 245760`, `SPEC: dflash2`, `CTX: huge`. - -On an RTX 3090, `catalog suggest` emits the latter two as `-long` and `-huge` -endpoint variants. The hardware check happens only while suggesting: the -resulting catalog contains ordinary explicit runtime data, so `apply`/`acquire` -never silently retunes an endpoint after the fact. +Changing the launcher's mode is a data edit, in the catalog or the TUI's +endpoint editor. The HyperQwen suggestion (see "Related work") is a worked +example: on an RTX 3090, `catalog suggest` emits its measured context +profiles as separate `-long` and `-huge` endpoints. The hardware check +happens only while suggesting; the resulting catalog holds ordinary explicit +runtime data, so `apply`/`acquire` never retunes an endpoint after the fact. - `{max_model_len}`, `{gpu_memory_utilization}`, `{served_model_name}` and `{port}` are filled in from the endpoint, so a launcher that takes them @@ -650,46 +475,35 @@ never silently retunes an endpoint after the fact. launcher instead. - All of these are deployment identity: endpoints that launch differently never share a process. -- Compose only; KubeAI refuses an endpoint with `command`, `env` or `mounts`. +- `env` works on both backends (on KubeAI it becomes the Model's `spec.env`). + `command` and `mounts` are Compose only; KubeAI refuses an endpoint with + either rather than serve stock vLLM in its place. ### Reasoning / thinking models -Models can declare reasoning support in the catalog: +vLLM separates a reasoning trace from the answer when it is started with a +reasoning parser. Pass the flag through `runtime.extra_args`: ```yaml -reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true +endpoints: + qwen3-think: + engine: vllm + model: qwen3-0.6b + runtime: + extra_args: [--reasoning-parser=qwen3] ``` -Profiles can override or set the same field per service. When a -service has `reasoning.enabled: true` and a `parser`, the renderer -adds `--reasoning-parser ` to that vLLM container's command -line — that flag alone enables reasoning extraction in the current -vLLM CLI. You do not need to repeat it by hand in `extra_args`. - -Open WebUI sees reasoning content via two paths: - -1. Inline `...` tags emitted by the model. -2. Structured `reasoning_content` fields when LiteLLM normalizes them. - -The LiteLLM template keeps `merge_reasoning_content_in_choices: true` -on chat-mode entries so Open WebUI can display reasoning in the -streamed response. To test reasoning end-to-end: +The parser name depends on the model family and the vLLM version (`vllm serve +--help` lists them). To test end to end: ```bash -# Non-streaming CLI smoke test: -infer-stack smoke-test \ - --model qwen3.6-35b-a3b \ - --prompt "Think step by step: 17*23" - -# For streaming inspection, read the key with the CLI wrapper: -LITELLM_MASTER_KEY=$(infer-stack env LITELLM_MASTER_KEY) -curl -N http://127.0.0.1:14042/v1/chat/completions \ - -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ +infer-stack test qwen3-think --prompt "Think step by step: 17*23" --max-tokens 512 + +# Streaming, through the gateway: +curl -N "$(infer-stack env OPENAI_BASE_URL)/chat/completions" \ + -H "Authorization: Bearer $(infer-stack env LITELLM_MASTER_KEY)" \ -H 'Content-Type: application/json' \ - -d '{"model":"qwen3.6-35b-a3b","stream":true, + -d '{"model":"qwen3-think","stream":true, "messages":[{"role":"user","content":"Think step by step: 17*23"}]}' ``` @@ -700,27 +514,33 @@ chat settings. ## Backend 2: KubeAI -Use KubeAI when you want Kubernetes-managed serving. - -### Important rules +`--backend kubeai` runs the same leasing verbs against a Kubernetes cluster +running [KubeAI](https://www.kubeai.org): the same catalog, ledger, TTLs, +env file and TUI, with `Model` custom resources in place of compose +services and the cluster scheduler in place of the local GPU planner. The +LiteLLM gateway still fronts everything, so a card sees one `OPENAI_BASE_URL`, +the managed key and the endpoint alias on either backend. -1. **Use the same namespace everywhere.** The namespace in `infer-stack setup --namespace ...` must match the namespace where the KubeAI Helm release already exists. -2. **Prefer the repo-driven path.** The normal path is `setup` -> `validate` -> `render` -> `deploy` -> `status`. -3. **`kubectl port-forward` stays in the foreground.** Leave it running in one terminal and send requests from another. -4. **The first request can take a while.** `/openai/v1/models` may work before chat completions work. The first completion may trigger pod creation, image pull, model load, and compile warmup. -5. **On the current repo version, KubeAI still needs a live workaround after deploy.** The renderer currently produces a `Model` spec that needs a small manual patch to work with the KubeAI version used in these notes. - -### KubeAI prerequisites +* Setup, settings and semantics: [docs/kubeai-backend.md](docs/kubeai-backend.md). +* What matches Compose, what does not yet, and what is deliberate: + [docs/backend-parity.md](docs/backend-parity.md); the plan to close the + rest: [docs/planning/backend-parity-roadmap.md](docs/planning/backend-parity-roadmap.md). -You need: +The short version: -* a working Kubernetes cluster -* `kubectl` -* Helm +```bash +./scripts/bootstrap_k3s.sh # a one-host cluster (k3s + helm) +./scripts/install_kubeai.sh kubeai-values.yaml # the chart, with your resourceProfiles +kubectl -n kubeai port-forward svc/kubeai 8000:80 & +infer-stack config set backend kubeai +infer-stack doctor # cluster -> CRD -> namespace -> KubeAI's API +infer-stack acquire --ttl 2h --env-file lease.env --yes +``` -If you want a quick local single-node cluster, K3s is a good option. +### KubeAI prerequisites -Install K3s: +You need a Kubernetes cluster, `kubectl` and Helm. `scripts/bootstrap_k3s.sh` +installs k3s and helm on one host; the steps it runs, by hand: ```bash curl -sfL https://get.k3s.io | sh - @@ -771,332 +591,80 @@ kubectl get node "$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" kubectl get nodes --show-labels | tr ',' '\n' | grep 'nvidia.com/' || true ``` -You want to see a non-empty `nvidia.com/gpu` count and `nvidia.com/*` labels such as product and memory. - -### KubeAI Helm repository - -```bash -helm repo add kubeai https://www.kubeai.org -helm repo update -``` - -## Determine which namespace to use - -Before doing anything else, discover whether a `kubeai` release already exists and which namespace it uses. - -```bash -KUBEAI_NAMESPACE="$(helm list -A | awk '$1=="kubeai"{print $2; exit}')" -if [ -z "${KUBEAI_NAMESPACE}" ]; then - KUBEAI_NAMESPACE=default -fi -echo "Using KubeAI namespace: ${KUBEAI_NAMESPACE}" -``` - -If a release already exists, **reuse that namespace**. - -Sanity-check the cluster: - -```bash -kubectl get nodes -kubectl get crd models.kubeai.org || true -helm list -A | grep kubeai || true -kubectl -n "${KUBEAI_NAMESPACE}" get pods || true -kubectl get node "$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" \ - -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}' -``` - ---- - -## Generate the local KubeAI resource-profile file - -Generate a local KubeAI resource-profile file from the labels on this machine. - -For the built-in serving profiles in this repo, keep these names aligned: +You want a non-empty `nvidia.com/gpu` count and `nvidia.com/*` labels such as product and memory. -* `gpu-single-default` -* `gpu-tp2-balanced` -* `gpu-tp2-maxctx` +### Resource profiles -**Important:** include GPU `requests`, GPU `limits`, and `runtimeClassName: nvidia`. Without those, the model pod can land on the GPU node but still start without `libcuda.so.1` available inside the container. +A `resourceProfile` is what one "GPU unit" means on your cluster; the +catalog's `runtime.resource_profile` (or the `kubeai_resource_profile` +setting) names one, and infer-stack appends the GPU count. Include GPU +`requests`, GPU `limits` and `runtimeClassName: nvidia`: without them the +pod can land on the GPU node and still start without `libcuda.so.1`. ```bash PRODUCT="$(kubectl get nodes -o jsonpath='{.items[0].metadata.labels.nvidia\.com/gpu\.product}')" -MEMORY="$(kubectl get nodes -o jsonpath='{.items[0].metadata.labels.nvidia\.com/gpu\.memory}')" -cat > values-kubeai-local-gpu.yaml < kubeai-values.yaml <` follows one, as on compose. For +anything else, with `NS` the namespace and `MODEL` the Model's name +(`kubectl -n $NS get models`): ```bash -kubectl -n "${KUBEAI_NAMESPACE}" patch model qwen2-5-7b-instruct-turbo-default --type merge -p '{ - "spec": { - "minReplicas": 1, - "resourceProfile": "gpu-single-default:1", - "args": [ - "--served-model-name=qwen2-5-7b-instruct-turbo-default", - "--tensor-parallel-size=1", - "--data-parallel-size=1", - "--max-model-len=32768", - "--gpu-memory-utilization=0.9", - "--max-num-batched-tokens=8192", - "--max-num-seqs=16", - "--disable-log-requests", - "--enable-prefix-caching" - ] - } -}' - -kubectl -n "${KUBEAI_NAMESPACE}" delete pod -l model=qwen2-5-7b-instruct-turbo-default +kubectl -n "$NS" describe model "$MODEL" +kubectl -n "$NS" get pods -l model="$MODEL" +kubectl -n "$NS" logs -f -l model="$MODEL" -c server # the engine +kubectl -n "$NS" logs -l model="$MODEL" -c server --previous # after a restart +kubectl -n "$NS" logs deploy/kubeai --tail=200 -f # the KubeAI controller +kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -n 40 ``` -If you run `infer-stack render` or `infer-stack deploy` again on the current repo version, re-apply this live patch. - ---- - -## Example 2: four-GPU system - -On a 4-GPU host, do the same **single-GPU smoke test first** to verify the cluster, KubeAI, runtime class, and model plumbing. That exact sequence worked on a 4-GPU machine during bring-up. - -```bash -infer-stack setup \ - --backend kubeai \ - --profile qwen2-5-7b-instruct-turbo-default \ - --namespace "${KUBEAI_NAMESPACE}" - -infer-stack validate -infer-stack render -infer-stack deploy -infer-stack status - -kubectl -n "${KUBEAI_NAMESPACE}" patch model qwen2-5-7b-instruct-turbo-default --type merge -p '{ - "spec": { - "minReplicas": 1, - "resourceProfile": "gpu-single-default:1", - "args": [ - "--served-model-name=qwen2-5-7b-instruct-turbo-default", - "--tensor-parallel-size=1", - "--data-parallel-size=1", - "--max-model-len=32768", - "--gpu-memory-utilization=0.9", - "--max-num-batched-tokens=8192", - "--max-num-seqs=16", - "--disable-log-requests", - "--enable-prefix-caching" - ] - } -}' - -kubectl -n "${KUBEAI_NAMESPACE}" delete pod -l model=qwen2-5-7b-instruct-turbo-default -``` +Common bad states: -After the 7B smoke test works, move up to larger profiles such as `qwen2-72b-instruct-tp2-balanced`. On the current repo version, apply the same kind of live patch after deploy: keep `minReplicas: 1`, append `:1` to the chosen `resourceProfile`, and make the single effective `--served-model-name` match the public profile name. - ---- - -## Test that KubeAI is responding - -If you are not exposing ingress yet, port-forward the service. - -**This command stays in the foreground.** Run it in one terminal and leave it there: - -```bash -kubectl -n "${KUBEAI_NAMESPACE}" port-forward svc/kubeai 8000:80 -``` - -Then use another terminal for requests. - -### First check: `/models` - -```bash -curl http://127.0.0.1:8000/openai/v1/models -``` - -If that works, the KubeAI front door is alive. - -### Then try the smoke test - -```bash -infer-stack smoke-test \ - --base-url http://127.0.0.1:8000/openai/v1 \ - --model qwen2-5-7b-instruct-turbo-default -``` - -### Or test chat completions directly - -```bash -time curl http://127.0.0.1:8000/openai/v1/chat/completions \ - -H 'Content-Type: application/json' \ - -d '{ - "model": "qwen2-5-7b-instruct-turbo-default", - "messages": [{"role": "user", "content": "Say hello in one short sentence."}], - "max_tokens": 8 - }' -``` - -### What to expect on the first request - -Common first-request behavior: - -* `/openai/v1/models` works before completions work -* a completion request causes KubeAI to create a model-serving pod -* that pod may spend time in `ContainerCreating` while the image is pulled -* the model then spends more time loading and warming up -* the first completion can be much slower than later ones - -That is not automatically a failure. Watch the system state while the first request is happening: - -```bash -watch -n 1 'kubectl -n '"${KUBEAI_NAMESPACE}"' get pods; echo; kubectl -n '"${KUBEAI_NAMESPACE}"' get models' -``` - ---- - -## Debugging checks - -### Check the live Model object - -```bash -kubectl -n "${KUBEAI_NAMESPACE}" describe model qwen2-5-7b-instruct-turbo-default -kubectl -n "${KUBEAI_NAMESPACE}" get model qwen2-5-7b-instruct-turbo-default -o yaml | grep -E 'minReplicas|maxReplicas|resourceProfile' -``` - -### Check the current model pod - -```bash -kubectl -n "${KUBEAI_NAMESPACE}" describe pod "$(kubectl -n "${KUBEAI_NAMESPACE}" get pods -o name | grep 'model-qwen2-5-7b-instruct-turbo-default' | tail -n 1 | cut -d/ -f2)" -``` - -### Tail KubeAI controller logs - -```bash -kubectl -n "${KUBEAI_NAMESPACE}" logs deploy/kubeai --tail=200 -f -``` - -### Tail model-server logs - -```bash -kubectl -n "${KUBEAI_NAMESPACE}" logs -f "$(kubectl -n "${KUBEAI_NAMESPACE}" get pods -o name | grep 'model-qwen2-5-7b-instruct-turbo-default' | tail -n 1 | cut -d/ -f2)" -c server -``` - -If the model pod restarted, inspect the previous crash: - -```bash -kubectl -n "${KUBEAI_NAMESPACE}" logs "$(kubectl -n "${KUBEAI_NAMESPACE}" get pods -o name | grep 'model-qwen2-5-7b-instruct-turbo-default' | tail -n 1 | cut -d/ -f2)" -c server --previous -``` - -### Check recent events - -```bash -kubectl -n "${KUBEAI_NAMESPACE}" get events --sort-by=.lastTimestamp | tail -n 40 -``` - -### Common bad states and what they mean - -* `invalid resource profile: "gpu-single-default", should match :` - * append `:1` in the live `Model` spec -* `libcuda.so.1: cannot open shared object file` - * the pod landed on the GPU node without actually requesting a GPU; fix the resource-profile file to include GPU requests, limits, and `runtimeClassName: nvidia` -* `/models` works but completions 404 with `The model ... does not exist.` - * the served model name does not match the public profile name; apply the live args patch above -* startup probe fails with `connection refused` - * the model pod may still be pulling the image, loading the model, or warming up +* `libcuda.so.1: cannot open shared object file`: the pod did not request a + GPU; fix the resource profile (requests, limits, `runtimeClassName`). +* startup probe fails with `connection refused`: still pulling the image, + loading the model or warming up; the acquire's wait reports which. +* `/models` works but completions 404: the request bypassed the gateway and + used the alias; through the gateway the alias is the model name, directly + against KubeAI the Model name is (`INFER_STACK_ENDPOINT_*` in the env file). --- ## Which backend should I start with? -Start with **Compose** if you want: - -* the fastest path to a working local server -* easy inspection of generated files -* simple single-host iteration - -Move to **KubeAI** when you want: - -* vLLM runtimes on Kubernetes -* KubeAI’s OpenAI-compatible front door -* profile deployment through Kubernetes artifacts - -KubeAI rendering is vLLM-only for now. Profiles that enable Ollama, LiteLLM, or Open WebUI are rejected for `--backend kubeai`. - -A good workflow is: - -1. inspect a profile with `describe-profile` -2. run it with Compose when you want the simplest local deployment -3. move to KubeAI when you want Kubernetes-backed serving - -Compose is the better fit when you already know which profile you want. KubeAI has more first-request overhead because it may need to create pods, pull images, load the model, and warm up the backend. +**Compose** when one workstation is enough: it is the fastest path to a +working server, everything it renders is a file you can read, and it needs +only Docker. +**KubeAI** when the models must run on more than one machine, or a cluster +already exists. It is Compose plus a scheduler: the same catalog, verbs and +env file, with a cluster and `resourceProfiles` supplied. Expect more +first-request overhead (pod creation, image pull, model load). +A catalog written for Compose runs on KubeAI, with two exceptions: ollama +endpoints and custom container launches (`runtime.command` / `mounts`), +which stay Compose-only. [docs/backend-parity.md](docs/backend-parity.md) +has the full matrix. ## vLLM startup caches @@ -1105,39 +673,24 @@ Triton, and CUDA JIT caches. Warm starts avoid redownloading and redoing many compile/JIT steps, but a vLLM model swap still creates a new engine process and must reload weights into GPU memory. -### Diagnosing profile switches and readiness - -`docker compose` health only means that a container-level healthcheck passed. It -is not the same thing as "the routed model can answer a request through the -active access surface." This matters most when switching between two vLLM -profiles that reuse the same runtime service name: the old vLLM process exits, -Docker starts the replacement process, and LiteLLM may remain up while returning -upstream connection errors until vLLM finishes loading the new model. - -Use the dedicated readiness and diagnostics commands after a switch: - -```bash -infer-stack switch gpt2-single --apply --yes -infer-stack wait-ready --model gpt2 -infer-stack smoke-test --model gpt2 -``` +### Diagnosing readiness -For debugging, use: +`docker compose` health only means a container-level healthcheck passed. It +is not "the routed model answers a request through the front door", which is +what `acquire` and `wait` check: a model swap starts a new engine process, +and LiteLLM stays up while returning upstream connection errors until vLLM +has loaded the weights. ```bash -infer-stack diagnose --model gpt2 --generation -infer-stack diagnose --logs --tail 80 +infer-stack acquire --yes # waits for a real generation +infer-stack wait # after acquire --no-wait +infer-stack test # one generation through the gateway +infer-stack status # desired vs running, per deployment +infer-stack logs --tail 80 # the engine or gateway log ``` -`diagnose` prints the resolved provider/gateway/frontend graph, rendered Compose -service state, LiteLLM route probes, direct provider probes, and optional recent -logs. It is intended to distinguish an actual LiteLLM outage from the more -common case where LiteLLM is running but its upstream vLLM runtime is still -booting. - -The Compose service-state diagnostics include Docker's exit code, OOM-killed -flag, restart count, and actual container name. This is important because -`litellm exited with code 137` usually means Docker sent SIGKILL, commonly from -an OOM kill or a forced container replacement, whereas LiteLLM returning HTTP -500 with `Cannot connect to host vllm-*` means LiteLLM is still running but the -upstream vLLM runtime is not ready yet. +A crash-looping engine fails the acquire at once with its error quoted, so +the timeout is only for a model that is loading. Reading Docker's own +signals: `litellm exited with code 137` is a SIGKILL (an OOM kill or a forced +replacement), whereas LiteLLM returning HTTP 500 with `Cannot connect to host +vllm-*` means LiteLLM is running and its upstream vLLM is not ready yet. diff --git a/dev/e2e_tests/kubeai-cpu-values.yaml b/dev/e2e_tests/kubeai-cpu-values.yaml new file mode 100644 index 00000000..8d5fb354 --- /dev/null +++ b/dev/e2e_tests/kubeai-cpu-values.yaml @@ -0,0 +1,38 @@ +# KubeAI on a GPU-less machine, for exercising the kubeai backend for real. +# The chart's `cpu` profile runs vLLM's CPU image (needs AVX-512). Sized for +# a sub-1B model; raise the requests for anything larger. +resourceProfiles: + cpu: + imageName: "cpu" + requests: + cpu: 8 + memory: "16Gi" + # Half the guest's 120 cores: only one such Model fits at a time, which is + # how dev/kubeai_e2e.sh's make-room check forces a scheduling conflict. + cpu-half: + imageName: "cpu" + requests: + cpu: 64 + memory: "16Gi" + # Two "GPU sizes" on CPU, for the sizing check (E2E_SIZED=1 in + # dev/kubeai_e2e.sh): each selects the nodes labelled with one fake GPU + # product, as a real profile selects nvidia.com/gpu.product. `sized-80g` + # tolerates dev/k3s_agent_container.sh's taint, so it lands on that node. + sized-24g: + imageName: "cpu" + requests: + cpu: 8 + memory: "16Gi" + nodeSelector: + nvidia.com/gpu.product: "FAKE-24G" + sized-80g: + imageName: "cpu" + requests: + cpu: 8 + memory: "16Gi" + nodeSelector: + nvidia.com/gpu.product: "FAKE-80G" + tolerations: + - key: infer-stack.test/simulated + operator: Exists + effect: NoSchedule diff --git a/dev/handover/p4_gpu_labels.sh b/dev/handover/p4_gpu_labels.sh new file mode 100755 index 00000000..e47e2cfe --- /dev/null +++ b/dev/handover/p4_gpu_labels.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# P4 on real GPUs: what the CPU-only development cluster cannot show. +# +# Run on a GPU machine that is a k3s node with the NVIDIA device plugin and +# GPU Feature Discovery (README: "NVIDIA GPU support"), with the KubeAI chart +# installed and `kubectl -n kubeai port-forward svc/kubeai 8000:80` running. +# It serves one small model on one GPU for a few minutes: run it when that +# GPU is free. +# +# It checks, in order: +# 1. GPU Feature Discovery labels each GPU node (product, memory) and the +# device plugin advertises nvidia.com/gpu: the facts sizing reads. +# 2. `catalog suggest --backend kubeai` proposes a resource profile for +# each GPU product, from those labels. +# 3. With the proposed profiles installed, an endpoint that declares only +# `min_vram_gib` gets the right profile and answers a request on a GPU. +# 4. `infer-stack measure --record` finds vLLM's memory-profiling +# lines in the pod's log and records a min_vram_gib. +# +# It uses its own config and data directories (never this host's infer-stack +# state). It leaves the proposed profiles in the chart's values: they +# describe this cluster's GPUs and are what min_vram_gib needs. +# +# dev/handover/p4_gpu_labels.sh 2>&1 | tee p4-gpu.log +# +# Knobs: E2E_MODEL (default Qwen/Qwen2.5-0.5B-Instruct), NAMESPACE (kubeai), +# E2E_TIMEOUT (900 s). +set -euo pipefail + +MODEL="${E2E_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}" +NAMESPACE="${NAMESPACE:-kubeai}" +TIMEOUT="${E2E_TIMEOUT:-900}" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/infer-stack-p4.XXXXXX")" +IS_ENV="INFER_STACK_CONFIG_DIR=$WORK/config INFER_STACK_DATA_DIR=$WORK/data" +run_is() { env $IS_ENV infer-stack "$@"; } +pass() { echo "PASS $*"; } +fail() { echo "FAIL $*" >&2; exit 1; } + +cleanup() { + set +e + run_is release --all --yes >/dev/null 2>&1 + run_is stack down >/dev/null 2>&1 + echo "work dir kept for the report: $WORK" +} +trap cleanup EXIT +mkdir -p "$WORK/config" + +echo '== 1. the GPU facts sizing reads' +kubectl get nodes -o json | python3 -c ' +import json, sys +found = 0 +for n in json.load(sys.stdin)["items"]: + labels = n["metadata"].get("labels", {}) + gpus = n["status"].get("allocatable", {}).get("nvidia.com/gpu") + product, memory = labels.get("nvidia.com/gpu.product"), labels.get("nvidia.com/gpu.memory") + print(f" {n[\"metadata\"][\"name\"]}: gpus={gpus} product={product} memory_mib={memory}") + found += bool(gpus and product and memory) +sys.exit(0 if found else 1)' || fail 'no node has nvidia.com/gpu and the GFD product/memory labels' +pass 'GPU nodes are labelled' + +echo '== 2. catalog suggest proposes profiles from the labels' +run_is config set backend kubeai >/dev/null +run_is config set kubeai_namespace "$NAMESPACE" >/dev/null +run_is catalog suggest --backend kubeai > "$WORK/suggest.yaml" 2> "$WORK/suggest.err" \ + || { cat "$WORK/suggest.err" >&2; fail 'catalog suggest failed'; } +python3 - "$WORK/suggest.err" "$WORK/profiles.yaml" <<'PY' || fail 'suggest printed no resourceProfiles' +import sys, yaml +text = open(sys.argv[1]).read() +block = yaml.safe_load(text[text.index('resourceProfiles:'):]) +open(sys.argv[2], 'w').write(yaml.safe_dump(block, sort_keys=False)) +print(yaml.safe_dump(block, sort_keys=False)) +PY +pass 'suggest proposed a profile per GPU product' + +echo '== 3. min_vram_gib picks the proposed profile and serves on a GPU' +version=$(helm list -n "$NAMESPACE" -o json | python3 -c \ + 'import json,sys; print(json.load(sys.stdin)[0]["chart"].rsplit("-", 1)[1])') +helm upgrade kubeai kubeai/kubeai -n "$NAMESPACE" --version "$version" \ + --reuse-values -f "$WORK/profiles.yaml" --wait >/dev/null +cat > "$WORK/config/catalog.yaml" < "$WORK/acquire.log" 2>&1 || true +grep -q 'ready: True' "$WORK/acquire.log" || fail 'p4-small never became ready' +got=$(kubectl -n "$NAMESPACE" get models.kubeai.org -l infer-stack/managed=true \ + -o jsonpath='{.items[0].spec.resourceProfile}') +echo " Model resourceProfile: $got (profiles proposed: $(tr '\n' ' ' < "$WORK/profiles.yaml" | head -c 200))" +case "$got" in nvidia-*:1) pass "min_vram_gib 8 -> $got, and it answers" ;; + *) fail "unexpected resource profile $got" ;; esac + +echo '== 4. measure reads the pod log on a GPU' +run_is measure p4-small --record | tee "$WORK/measure.log" +grep -q 'min_vram_gib' "$WORK/measure.log" || fail 'measure printed no min_vram_gib' +test -s "$WORK/data/leasing/kubeai/measurements.json" || fail 'measure --record wrote nothing' +pass 'measure recorded a min_vram_gib on kubeai' + +echo 'ALL PASS' diff --git a/dev/handover/p5_two_hosts.sh b/dev/handover/p5_two_hosts.sh new file mode 100755 index 00000000..527e4b78 --- /dev/null +++ b/dev/handover/p5_two_hosts.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# P5 across two real machines: what a second node in a container cannot show. +# +# Run on the cluster's first node (kubeconfig working) once a second GPU +# workstation has joined it (docs/kubeai-backend.md, "Add a workstation"), +# with the NVIDIA device plugin and GPU Feature Discovery running and the +# KubeAI chart installed. It serves one small model on the second node's GPU +# for a few minutes: run it when that GPU is free. +# +# It checks, in order: +# 1. two Ready nodes, and the second reports GPUs with GFD's labels; +# 2. with the gateway in the cluster, an endpoint whose resource profile +# selects the second node's GPU product lands there and answers through +# the gateway: pod-to-pod traffic across the real network (flannel); +# 3. the NodePort answers on the second node's own address too, so a card +# on either workstation uses the same env file; +# 4. `secrets rotate` (a Secret and a rollout) with the Model still served. +# +# Own config and data directories; nothing of this host's infer-stack state +# is touched. It adds one resource profile for the second node's GPU product +# to the chart's values and leaves it: that is what the node needs. +# +# dev/handover/p5_two_hosts.sh [SECOND_NODE_NAME] 2>&1 | tee p5-two-hosts.log +set -euo pipefail + +NODE_B="${1:-}" +MODEL="${E2E_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}" +NAMESPACE="${NAMESPACE:-kubeai}" +TIMEOUT="${E2E_TIMEOUT:-900}" +PORT="${NODE_PORT:-30442}" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/infer-stack-p5.XXXXXX")" +IS_ENV="INFER_STACK_CONFIG_DIR=$WORK/config INFER_STACK_DATA_DIR=$WORK/data" +run_is() { env $IS_ENV infer-stack "$@"; } +pass() { echo "PASS $*"; } +fail() { echo "FAIL $*" >&2; exit 1; } + +cleanup() { + set +e + run_is release --all --yes >/dev/null 2>&1 + run_is stack down >/dev/null 2>&1 + echo "work dir kept for the report: $WORK" +} +trap cleanup EXIT +mkdir -p "$WORK/config" + +echo '== 1. two nodes, the second with GPUs' +if [ -z "$NODE_B" ]; then + NODE_B=$(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' \ + -o jsonpath='{.items[0].metadata.name}') +fi +[ -n "$NODE_B" ] || fail 'no second node (pass its name as the first argument)' +kubectl wait --for=condition=Ready "node/$NODE_B" --timeout=30s >/dev/null \ + || fail "$NODE_B is not Ready" +PRODUCT=$(kubectl get node "$NODE_B" -o jsonpath='{.metadata.labels.nvidia\.com/gpu\.product}') +GPUS=$(kubectl get node "$NODE_B" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}') +ADDR_B=$(kubectl get node "$NODE_B" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') +[ -n "$PRODUCT" ] && [ "${GPUS:-0}" -gt 0 ] \ + || fail "$NODE_B reports no GPUs or no nvidia.com/gpu.product label" +pass "$NODE_B at $ADDR_B: $GPUS x $PRODUCT" + +echo '== 2. a Model on the second node, through the in-cluster gateway' +profile="p5-$(echo "$PRODUCT" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9\n' '-')" +cat > "$WORK/profile.yaml" </dev/null +run_is config set backend kubeai >/dev/null +run_is config set kubeai_namespace "$NAMESPACE" >/dev/null +run_is config set kubeai_gateway cluster >/dev/null +cat > "$WORK/config/catalog.yaml" < "$WORK/acquire.log" 2>&1 || true +grep -q 'ready: True' "$WORK/acquire.log" || { tail -30 "$WORK/acquire.log"; fail 'not ready'; } +node=$(kubectl -n "$NAMESPACE" get pods -l infer-stack/managed=true \ + -o jsonpath='{.items[0].spec.nodeName}') +[ "$node" = "$NODE_B" ] || fail "the Model runs on $node, not $NODE_B" +# shellcheck disable=SC1090 +source "$WORK/lease.env" +ask() { # -> a generation for p5-remote + curl -sS --fail-with-body "$1/chat/completions" -H "Authorization: Bearer $OPENAI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{"model": "p5-remote", "max_tokens": 4, "messages": [{"role": "user", "content": "say ok"}]}' \ + > "$WORK/ask.json" && grep -q choices "$WORK/ask.json" +} +ask "$OPENAI_BASE_URL" || fail "no generation via $OPENAI_BASE_URL" +pass "the Model on $NODE_B answers via $OPENAI_BASE_URL" + +echo '== 3. the NodePort answers on the second node too' +ask "http://$ADDR_B:$PORT/v1" || fail "no generation via $ADDR_B:$PORT (firewall on the NodePort?)" +pass "a card on $NODE_B can use http://$ADDR_B:$PORT/v1" + +echo '== 4. secrets rotate across the cluster' +run_is secrets rotate --force > "$WORK/rotate.log" 2>&1 || { cat "$WORK/rotate.log"; fail 'rotate'; } +grep -q 'new key accepted, old key rejected' "$WORK/rotate.log" || { cat "$WORK/rotate.log"; fail 'rotate'; } +pass 'secrets rotate: new key accepted, old key rejected' + +echo 'ALL PASS' diff --git a/dev/journals/claude.md b/dev/journals/claude.md index caadd793..c2441faa 100644 --- a/dev/journals/claude.md +++ b/dev/journals/claude.md @@ -3180,3 +3180,113 @@ labels are stable enough to key residency on. **Takeaway.** Before unifying internals, check what callers see. Two backends that share 90% of their code but answer to different names are two products to the people using them. + +## 2026-09-24 12:54:30 -0400 + +**Intent.** The user approved the backend-unification plan, asked for the +work on a branch (`dev/backend-unification`), and authorised installing k3s +on the guest to test KubeAI for real. Model: Claude Opus 5.5 (1M context). + +**What happened.** K0: k3s plus KubeAI 0.23.4, using the chart's `cpu` +profile. It runs real vLLM on CPU, which beats the simulator: KubeAI builds +vLLM's own command line, and the simulator's CLI would have refused it. The +backend passed unchanged on its first real run since July. K1: the kubeai +backend now owns a gateway-only ComposeBackend and feeds it generic +`upstream` route rows. Before, a card's alias got 404 from KubeAI; after, +the same request answered. K2 and K4: strict pod residency, and the crash +diagnosis moved into a backend-neutral module. A Model whose vLLM rejected a +flag failed in 52 s instead of the 800 s timeout. K5: one served-name rule. +It fixed a real disagreement between the engine and two route builders. +Removed the dead profile-era KubeAI renderer. + +**What I got wrong along the way.** I wrote a second e2e script before +finding `dev/kubeai_e2e.sh`, a duplicate authority created in the middle of +a de-duplication task, and folded it back in. That script's generation check +could never fail (a `curl | grep && echo` list, where `set -e` does not +fire), so it "passed" on a 404. The plan's claim that KubeAI's lenient +`observe()` was a bug was wrong: Compose's is lenient by the same contract. +I corrected the plan. + +**Open: K3 (one acquire path).** It is feasible: only 12 controller tests and +the null backend depend on the old path. It carries a semantic decision the +user should make. On a cluster nothing displaces an idle keep-warm Model, so +the September starvation incident can recur on KubeAI unless admission +evicts idle keep-warm Models when a new one cannot be scheduled. + +**Takeaways.** (1) A test that has never failed is not evidence. Make it +fail on purpose first; GATEWAY=0 is that switch here. (2) "Has method X" is +a poor stand-in for "is backend Y": when a second backend gains X, every +such check silently changes meaning. + +## 2026-09-24 14:39:35 -0400 + +**Intent.** Make infer-stack more elegant, architecture first and the TUI +after. The user agreed that the Compose backend's real seam is gateway vs. +engines. Model: Claude Opus 5.5 (1M context). + +**What I did.** Four steps on `dev/backend-unification`, each green before +the next: naming rules into `naming.py`, the gateway's module code into +`gateway.py`, a `Gateway` class for its state, and `render_front_door`. +The one real design change: the registry merge no longer reads backend +state. Backends supply rows and the gateway only merges them, which removed +the `upstream_routes` side-channel I had added for KubeAI in K1. Settings the +gateway reads became backend properties onto the single `Gateway`, because +`use_profile` and tests set them on the backend after construction. Two +copies would drift. + +**How I checked a pure refactor.** Unit tests alone are weak evidence for +moved code, so I also rendered six stack configurations with the old and new +code and compared bytes (identical), then ran the real Compose gateway, +`secrets rotate` and the KubeAI e2e. + +**What went wrong.** My first byte comparison "passed" on two empty files, +because the script crashed in both runs. I caught it only because the byte +count was 0. More serious: the weight-floor doctest I wrote earlier wrote +4 GiB of real zeros into /tmp on every run and never cleaned up. A day of +test runs filled 49 GB, k3s went into disk pressure and evicted KubeAI. +Fixed with sparse files. + +**Takeaways.** (1) A comparison of outputs must check the outputs exist: +equality of two failures is not evidence. (2) Test fixtures that need large +files should be sparse. Size was the property under test, not the bytes. + +## 2026-09-26 13:05:00 -0400 + +**Intent.** Execute the backend-parity queue (`docs/queue.md`): the roadmap +phases that can be built and verified without a GPU or a second machine, +refactoring duplicate authorities on the way, and not stopping before a +passing UX audit. Model: Claude Opus 5.5 (Claude Code). + +**What landed.** P1b deleted the pre-admission acquire branch: a +`SimpleAdmission` mixin gives the dry-run and test backends the admission +surface, so the controller has one path. P2 put `ps`, `logs`, `status` and +the TUI behind an `Instance` view built from residency, on both backends. +P3 made the gateway in front of a cluster the compose gateway (UI, proxy, +dynamic routing, one approval). P6 is a parity suite, one test per *same* +row. P4 picks a KubeAI resource profile by GPU size from node labels. + +**Decisions.** The review split P1: a `preview` alone would have dropped +every KubeAI lease, because the admission view assumed GPU accounting, so +"does this backend allocate GPUs" became one function. Under dynamic +routing a KubeAI Model is named per deployment with compose's own tail +rule, rather than a new scheme. A profile's size is what its node selector +selects; the chart's selector-less profiles deliberately have none, because +guessing a size for "anywhere" would pick wrong silently. + +**What surprised me.** Three bugs older than this work surfaced only in a +real terminal or a real cluster: the TUI replaced kubeai's kubectl runner +with Docker's (no KUBECONFIG, so every kubectl call failed), a kwconf flag +swallowed the positional after it on every command, and the TUI swapped an +injected runner for the real Docker. Fakes passed throughout; each was +found by running the thing. The e2e also failed once on its own +`grep -q` + pipefail pattern (lessons.md). + +**Risks.** The kubeai recovery profile now nests the gateway's profile; +old profiles (a bare boolean) keep this process's settings. P4 is verified +with fake labels only; `dev/handover/p4_gpu_labels.sh` is the real test. + +**Takeaways.** (1) When a refactor removes a branch, first run the whole +suite with the other branch forced on: the failures list exactly what to +migrate. (2) A seam that wraps a runner must wrap only what it owns; a +wrapper that replaces is a second authority. (3) Put every e2e check's +output in a file before testing it. diff --git a/dev/k3s_agent_container.sh b/dev/k3s_agent_container.sh new file mode 100755 index 00000000..379594af --- /dev/null +++ b/dev/k3s_agent_container.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# A second k3s node on this host, in a Docker container: enough to exercise +# anything that depends on the cluster having more than one node (node +# labels, node selectors, where a pod lands) without a second machine. +# +# dev/k3s_agent_container.sh up [NAME] # join, taint, print the node +# dev/k3s_agent_container.sh down [NAME] # drain it out of the cluster +# +# The node is tainted `infer-stack.test/simulated=true:NoSchedule`, so only +# pods that tolerate it land there: an ordinary Model never does (its image +# would be pulled inside the container's own containerd). Give a test +# resource profile that toleration to put a Model on it. +# +# Needs: this host running the k3s server (scripts/bootstrap_k3s.sh), sudo +# to read the node token, and Docker. The image tag follows the server's. +set -euo pipefail + +ACTION="${1:-up}" +NAME="${2:-k3s-agent-b}" +CONTAINER="infer-stack-${NAME}" + +case "$ACTION" in + up) + version="$(k3s --version | awk 'NR==1 {print $3}')" # v1.36.4+k3s1 + image="rancher/k3s:${version/+/-}" + server_ip="$(ip -4 route get 1.1.1.1 | awk '{for (i=1;i/dev/null 2>&1 || true + docker run -d --name "$CONTAINER" --privileged --hostname "$NAME" \ + -e K3S_URL="https://${server_ip}:6443" -e K3S_TOKEN="$token" \ + --tmpfs /run --tmpfs /var/run "$image" agent --node-name "$NAME" >/dev/null + for _ in $(seq 60); do + kubectl get node "$NAME" >/dev/null 2>&1 && break + sleep 2 + done + kubectl wait --for=condition=Ready "node/$NAME" --timeout=120s >/dev/null + kubectl taint node "$NAME" infer-stack.test/simulated=true:NoSchedule --overwrite >/dev/null + kubectl get node "$NAME" -o wide + ;; + down) + kubectl delete node "$NAME" --ignore-not-found >/dev/null + docker rm -f "$CONTAINER" >/dev/null 2>&1 || true + echo "removed $NAME" + ;; + *) + echo "usage: $0 up|down [NAME]" >&2 + exit 2 + ;; +esac diff --git a/dev/kubeai_e2e.sh b/dev/kubeai_e2e.sh index 8d21d1f7..92dfd7cb 100755 --- a/dev/kubeai_e2e.sh +++ b/dev/kubeai_e2e.sh @@ -3,11 +3,15 @@ # # Prereqs (once): a cluster + the KubeAI chart. On a single GPU host: # ./scripts/bootstrap_k3s.sh -# printf 'resourceProfiles:\n %s:\n limits:\n nvidia.com/gpu: "1"\n' \ +# printf 'resourceProfiles:\n %s:\n runtimeClassName: nvidia\n requests: {nvidia.com/gpu: "1"}\n limits: {nvidia.com/gpu: "1"}\n' \ # "${E2E_RESOURCE_PROFILE:-nvidia-gpu}" > /tmp/kubeai-values.yaml # ./scripts/install_kubeai.sh /tmp/kubeai-values.yaml kubeai # kubectl -n kubeai port-forward svc/kubeai 8000:80 & # +# Without a GPU (verified on k3s): install the chart with +# dev/e2e_tests/kubeai-cpu-values.yaml, whose `cpu` profile runs real vLLM on +# CPU (needs AVX-512), and run with E2E_RESOURCE_PROFILE=cpu. +# # Then: ./dev/kubeai_e2e.sh # # Knobs (env): @@ -17,6 +21,22 @@ # E2E_BASE_URL gateway url (default http://127.0.0.1:8000/openai/v1) # E2E_TIMEOUT acquire readiness budget seconds (default 900 — # first run pulls model weights into the cluster) +# GATEWAY 1 (default): clients go through infer-stack's LiteLLM +# gateway, as on the compose backend. 0: straight to +# KubeAI, where the alias below does NOT route (404). +# E2E_DYNAMIC 1 (default): finish with dynamic routing, two +# --dedicated leases of one model (two Models at once). +# E2E_UI_PORT Open WebUI's port on this host (default 13000). +# E2E_SIZED 1: check min_vram_gib picks a profile by GPU size, +# on two nodes with fake GPU labels. Needs +# dev/k3s_agent_container.sh up (E2E_SIZED_NODE names +# the node; default k3s-agent-b). +# E2E_CLUSTER_GATEWAY 1 (default): finish with the gateway in the cluster +# (kubeai_gateway cluster): a NodePort, doctor, +# secrets rotate, stack down. +# E2E_REMOTE_NODE 1 (with E2E_SIZED=1): also serve a Model on the +# second node through that gateway (pulls the vLLM +# CPU image inside the node container once). set -euo pipefail MODEL="${E2E_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}" @@ -24,6 +44,7 @@ PROFILE="${E2E_RESOURCE_PROFILE:-nvidia-gpu}" NAMESPACE="${E2E_NAMESPACE:-kubeai}" BASE_URL="${E2E_BASE_URL:-http://127.0.0.1:8000/openai/v1}" TIMEOUT="${E2E_TIMEOUT:-900}" +ALIAS="$MODEL" # Isolated config/data roots, baked INLINE on every command (never rely on # exported env reaching subprocesses — see dev/audit notes on tmux env loss). @@ -37,6 +58,14 @@ cleanup() { set +e run_is release --all --yes >/dev/null 2>&1 run_is gc --evict --yes >/dev/null 2>&1 + run_is stack down >/dev/null 2>&1 # the gateway, UI and database too + if [ "${SIZED_LABELLED:-0}" = 1 ]; then # the fake GPU labels, off again + for node in "$NODE_A" "$NODE_B"; do + kubectl label node "$node" nvidia.com/gpu.product- nvidia.com/gpu.memory- >/dev/null 2>&1 + done + kubectl patch node "$NODE_B" --subresource=status --type=json \ + -p '[{"op":"remove","path":"/status/capacity/nvidia.com~1gpu"}]' >/dev/null 2>&1 + fi # nothing managed may remain on the cluster, pass or fail leftover=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ -l infer-stack/managed=true -o name 2>/dev/null | wc -l) @@ -56,7 +85,8 @@ models: e2e-tiny: source: hf://$MODEL endpoints: - e2e-tiny: + # Named like real catalogs, unlike its KubeAI Model name: a card sends this. + $ALIAS: engine: vllm model: e2e-tiny reclaim: {policy: stop} @@ -68,23 +98,51 @@ EOF run_is config set backend kubeai run_is config set kubeai_namespace "$NAMESPACE" run_is config set kubeai_base_url "$BASE_URL" +if [ "${GATEWAY:-1}" = 0 ]; then run_is config set litellm false; fi echo '== doctor (preflight)' run_is doctor echo '== acquire (readiness = a real generation through the gateway)' -run_is acquire e2e-tiny --yes --ttl 30m --timeout "$TIMEOUT" \ +run_is acquire "$ALIAS" --yes --ttl 30m --timeout "$TIMEOUT" \ --env-file "$WORK/lease.env" -echo '== the descriptor works for a plain OpenAI client' +echo '== a client that knows only the env file and the alias (like a card)' # shellcheck disable=SC1090 source "$WORK/lease.env" -REQUEST_NAME="$INFER_STACK_ENDPOINT_E2E_TINY" -curl -sf "$OPENAI_BASE_URL/chat/completions" \ +# An explicit `if`: in a `curl | grep && echo` list `set -e` does not fire, so +# a 404 used to fall through to PASS. +if curl -sS --fail-with-body "$OPENAI_BASE_URL/chat/completions" \ + -H "Authorization: Bearer ${OPENAI_API_KEY:-EMPTY}" \ -H 'Content-Type: application/json' \ - -d "{\"model\": \"$REQUEST_NAME\", \"max_tokens\": 8, + -d "{\"model\": \"$ALIAS\", \"max_tokens\": 8, \"messages\": [{\"role\": \"user\", \"content\": \"say ok\"}]}" \ - | grep -q 'choices' && echo ' generation ok' + | grep -q 'choices'; then + echo " generation ok via $OPENAI_BASE_URL" +else + echo "!! no generation for model=$ALIAS via $OPENAI_BASE_URL" >&2 + exit 1 +fi + +if [ "${GATEWAY:-1}" = 1 ]; then + echo '== the gateway fronts the cluster as on compose: routes, Open WebUI' + if ! run_is routes list --json | python3 -c ' +import json, sys +alias = sys.argv[1] +row = next(r for r in json.load(sys.stdin)["routes"] if r["name"] == alias) +assert row["engine"] == "upstream" and row["live"], row +print(" routes list:", alias, "->", row["target"], "at", row["upstream"])' "$ALIAS"; then + echo '!! routes list does not route the alias to the cluster' >&2; exit 1 + fi + ui='' + for _ in $(seq 60); do + ui=$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1:${E2E_UI_PORT:-13000}/" || true) + [ "$ui" = 200 ] && break + sleep 2 + done + [ "$ui" = 200 ] || { echo "!! Open WebUI did not answer (HTTP $ui)" >&2; exit 1; } + echo ' Open WebUI answers in front of the cluster' +fi echo '== release prunes the Model (reclaim: stop)' run_is release --env-file "$WORK/lease.env" --yes @@ -92,4 +150,210 @@ remaining=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ -l infer-stack/managed=true -o name | wc -l) [ "$remaining" = 0 ] || { echo "!! model not pruned"; exit 1; } +echo '== an unrenderable endpoint is refused before anything is written' +# No resource_profile and no kubeai_resource_profile default: admission's +# preview refuses it, so no lease is committed and kubectl applies nothing; +# --queue fails at once, since waiting cannot make it renderable. +cat >> "$WORK/config/catalog.yaml" < "$WORK/bad.log" 2>&1; then + echo '!! an endpoint with no resource profile was admitted' >&2; exit 1 +fi +grep -q 'resource profile' "$WORK/bad.log" \ + || { cat "$WORK/bad.log" >&2; echo '!! the refusal did not name the cause' >&2; exit 1; } +active=$(run_is leases --json | python3 -c \ + 'import json,sys; print(sum(le["state"] == "active" for le in json.load(sys.stdin)["leases"]))') +[ "$active" = 0 ] || { echo "!! $active lease(s) left active" >&2; exit 1; } +remaining=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ + -l infer-stack/managed=true -o name | wc -l) +[ "$remaining" = 0 ] || { echo '!! a Model was applied for it' >&2; exit 1; } +echo ' refused at admission: no lease, no Model' + +if [ "${E2E_MAKE_ROOM:-0}" = 1 ]; then + # Needs a profile only one Model fits at a time (`cpu-half` in + # dev/e2e_tests/kubeai-cpu-values.yaml): E2E_ROOM_PROFILE=cpu-half. + echo '== an idle keep-warm model gives way to a leased one' + cat >> "$WORK/config/catalog.yaml" < "$WORK/big.log" 2>&1 || true + if ! grep -q 'ready: True' "$WORK/big.log"; then + echo '!! the leased model never became ready' >&2; exit 1 + fi + grep -q 'making room for leased demand' "$WORK/big.log" \ + || { echo '!! no idle model was evicted to make room' >&2; exit 1; } + echo ' the idle keep-warm model was evicted; the leased one is ready' + run_is release --env-file "$WORK/big.env" --yes +fi + +if [ "${E2E_SIZED:-0}" = 1 ]; then + # Needs a second node (dev/k3s_agent_container.sh up) and the sized-* + # profiles of dev/e2e_tests/kubeai-cpu-values.yaml. Fake GPU labels stand + # in for GPU Feature Discovery's; cleanup removes them. + echo '== min_vram_gib picks the smallest resource profile whose GPUs fit' + NODE_A=$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{.items[0].metadata.name}') + NODE_B="${E2E_SIZED_NODE:-k3s-agent-b}" + kubectl label node "$NODE_A" nvidia.com/gpu.product=FAKE-24G nvidia.com/gpu.memory=24576 --overwrite >/dev/null + kubectl label node "$NODE_B" nvidia.com/gpu.product=FAKE-80G nvidia.com/gpu.memory=81920 --overwrite >/dev/null + kubectl patch node "$NODE_B" --subresource=status --type=merge \ + -p '{"status":{"capacity":{"nvidia.com/gpu":"2"}}}' >/dev/null + SIZED_LABELLED=1 + cat >> "$WORK/config/catalog.yaml" < -> " " of its Model's pod + for _ in $(seq 60); do + out=$(kubectl -n "$NAMESPACE" get models.kubeai.org -l infer-stack/managed=true -o json \ + | python3 -c ' +import json, sys +for m in json.load(sys.stdin)["items"]: + if sys.argv[1].replace("/", "-").lower() in m["metadata"]["name"]: + print(m["metadata"]["name"], m["spec"]["resourceProfile"])' "$1") + name=${out%% *}; profile=${out##* } + node=$(kubectl -n "$NAMESPACE" get pods -l "model=$name" -o jsonpath='{.items[0].spec.nodeName}' 2>/dev/null || true) + [ -n "$node" ] && { echo "$profile $node"; return; } + sleep 2 + done + echo "$profile unscheduled" + } + run_is acquire e2e-sized-big --wait false --yes --env-file "$WORK/big-sized.env" >/dev/null + got=$(where e2e-sized-big) + [ "$got" = "sized-80g:1 $NODE_B" ] || { echo "!! 40 GiB went to: $got" >&2; exit 1; } + echo " min_vram_gib 40 -> $got" + run_is acquire e2e-sized-small --yes --timeout "$TIMEOUT" --env-file "$WORK/small-sized.env" \ + > "$WORK/small-sized.log" 2>&1 || true + grep -q 'ready: True' "$WORK/small-sized.log" \ + || { echo '!! the 10 GiB endpoint never became ready' >&2; exit 1; } + got=$(where e2e-sized-small) + [ "$got" = "sized-24g:1 $NODE_A" ] || { echo "!! 10 GiB went to: $got" >&2; exit 1; } + echo " min_vram_gib 10 -> $got, and it answers" + run_is catalog suggest --backend kubeai > "$WORK/suggest.yaml" 2> "$WORK/suggest.err" + grep -q 'nvidia-fake-80g' "$WORK/suggest.err" \ + || { cat "$WORK/suggest.err" >&2; echo '!! suggest proposed no profile for FAKE-80G' >&2; exit 1; } + echo ' catalog suggest proposes a resource profile per GPU product' + run_is release --env-file "$WORK/big-sized.env" --yes >/dev/null + run_is release --env-file "$WORK/small-sized.env" --yes >/dev/null +fi + +if [ "${GATEWAY:-1}" = 1 ] && [ "${E2E_DYNAMIC:-1}" = 1 ]; then + echo '== dynamic routing: two --dedicated leases on one model, two Models, one alias' + run_is stack down >/dev/null 2>&1 # the gateway comes back with Postgres + # A routing-mode change is adopted only by a quiescent stack: wait for the + # last managed pod (a released Model's) to be gone. + for _ in $(seq 90); do + [ -z "$(kubectl -n "$NAMESPACE" get pods -l infer-stack/managed=true -o name)" ] && break + sleep 2 + done + run_is config set dynamic_routing true + for n in 1 2; do + run_is acquire "$ALIAS" --dedicated --yes --ttl 30m --timeout "$TIMEOUT" \ + --env-file "$WORK/dyn$n.env" + done + models=$(kubectl -n "$NAMESPACE" get models.kubeai.org \ + -l infer-stack/managed=true -o name | wc -l) + [ "$models" = 2 ] || { echo "!! $models Model(s), expected 2" >&2; exit 1; } + # shellcheck disable=SC1090 + source "$WORK/dyn1.env" + routes=$(curl -s "$OPENAI_BASE_URL/model/info" -H "Authorization: Bearer $OPENAI_API_KEY" \ + | python3 -c 'import json,sys; print(sum(m["model_name"] == sys.argv[1] for m in json.load(sys.stdin)["data"]))' "$ALIAS") + [ "$routes" = 2 ] || { echo "!! $routes route(s) for $ALIAS, expected 2" >&2; exit 1; } + if curl -sS --fail-with-body "$OPENAI_BASE_URL/chat/completions" \ + -H "Authorization: Bearer $OPENAI_API_KEY" -H 'Content-Type: application/json' \ + -d "{\"model\": \"$ALIAS\", \"max_tokens\": 4, + \"messages\": [{\"role\": \"user\", \"content\": \"say ok\"}]}" \ + | grep -q 'choices'; then + echo ' two Models, two routes under one alias, and it answers' + else + echo "!! no generation for $ALIAS under dynamic routing" >&2; exit 1 + fi + run_is release --env-file "$WORK/dyn1.env" --yes + run_is release --env-file "$WORK/dyn2.env" --yes +fi + +if [ "${GATEWAY:-1}" = 1 ] && [ "${E2E_CLUSTER_GATEWAY:-1}" = 1 ]; then + echo '== the gateway inside the cluster: a card reaches the Model on a NodePort' + run_is stack down >/dev/null 2>&1 # the host gateway, and any Models + for _ in $(seq 90); do + [ -z "$(kubectl -n "$NAMESPACE" get pods -l infer-stack/managed=true -o name)" ] && break + sleep 2 + done + run_is config set dynamic_routing false >/dev/null + run_is config set kubeai_gateway cluster + run_is acquire "$ALIAS" --yes --ttl 30m --timeout "$TIMEOUT" \ + --env-file "$WORK/cluster.env" > "$WORK/cluster.log" 2>&1 || true + grep -q 'ready: True' "$WORK/cluster.log" \ + || { tail -20 "$WORK/cluster.log" >&2; echo '!! not ready through the in-cluster gateway' >&2; exit 1; } + # shellcheck disable=SC1090 + source "$WORK/cluster.env" + case "$OPENAI_BASE_URL" in + *:"${E2E_NODE_PORT:-30442}"/v1) ;; + *) echo "!! the env file points at $OPENAI_BASE_URL, not a NodePort" >&2; exit 1 ;; + esac + ask() { # -> a generation through the env file's gateway + curl -sS --fail-with-body "$OPENAI_BASE_URL/chat/completions" \ + -H "Authorization: Bearer $OPENAI_API_KEY" -H 'Content-Type: application/json' \ + -d "{\"model\": \"$1\", \"max_tokens\": 4, + \"messages\": [{\"role\": \"user\", \"content\": \"say ok\"}]}" > "$WORK/ask.json" \ + && grep -q choices "$WORK/ask.json" + } + ask "$ALIAS" || { echo "!! no generation via $OPENAI_BASE_URL" >&2; exit 1; } + echo " generation ok via $OPENAI_BASE_URL (a node's NodePort, no port-forward)" + run_is doctor > "$WORK/doctor.log" 2>&1 || { cat "$WORK/doctor.log" >&2; exit 1; } + grep -q '\[ok \] in-cluster gateway' "$WORK/doctor.log" \ + || { cat "$WORK/doctor.log" >&2; echo '!! doctor did not check the gateway' >&2; exit 1; } + echo ' doctor checks the in-cluster gateway' + if [ "${E2E_SIZED:-0}" = 1 ] && [ "${E2E_REMOTE_NODE:-0}" = 1 ]; then + run_is acquire e2e-sized-big --yes --timeout "$TIMEOUT" \ + --env-file "$WORK/remote.env" > "$WORK/remote.log" 2>&1 || true + grep -q 'ready: True' "$WORK/remote.log" \ + || { tail -20 "$WORK/remote.log" >&2; echo '!! the Model on the second node never became ready' >&2; exit 1; } + got=$(where e2e-sized-big) + [ "$got" = "sized-80g:1 $NODE_B" ] || { echo "!! the remote Model is at: $got" >&2; exit 1; } + ask e2e-sized-big || { echo '!! no generation from the second node' >&2; exit 1; } + echo " a Model on $NODE_B answers through the same gateway" + run_is release --env-file "$WORK/remote.env" --yes >/dev/null + fi + run_is release --env-file "$WORK/cluster.env" --yes >/dev/null + run_is secrets rotate > "$WORK/rotate.log" 2>&1 \ + || { cat "$WORK/rotate.log" >&2; echo '!! secrets rotate failed' >&2; exit 1; } + grep -q 'new key accepted, old key rejected' "$WORK/rotate.log" \ + || { cat "$WORK/rotate.log" >&2; exit 1; } + echo ' secrets rotate: new key accepted, old key rejected (a Secret and a rollout)' + run_is stack down >/dev/null + if kubectl -n "$NAMESPACE" get deployment infer-stack-gateway >/dev/null 2>&1; then + echo '!! stack down left the gateway Deployment' >&2; exit 1 + fi + echo ' stack down removes it' +fi + echo 'PASS: kubeai backend end-to-end lifecycle' diff --git a/dev/lessons/lessons.md b/dev/lessons/lessons.md index 9f616c7c..31b2bbfb 100644 --- a/dev/lessons/lessons.md +++ b/dev/lessons/lessons.md @@ -95,3 +95,33 @@ evidence; prefer append-only; supersede incorrect entries with a new one. absent. - **Applies when:** writing or reviewing any multi-threaded test that uses a barrier, latch, or queue rendezvous. + +- **Lesson:** A test that needs a large file for its *size* should create it + sparse (`file.truncate(n)`), and remove its temp dir. `os.path.getsize` + reports the full size, but no disk is used. Writing real bytes leaks gigabytes + per run unless the test cleans up. +- **Evidence / MWE:** 21f5e09. The `weight_floor_gib` doctest wrote two 2 GiB + files into `mkdtemp()` and never removed them. A day of suite runs left + 49 GB in `/tmp` on the guest, k3s tainted its node for disk pressure, and + KubeAI's pods were evicted. With sparse files the doctest takes 0.15 s and + leaves nothing behind. +- **Applies when:** a test exercises size-dependent logic (VRAM floors, disk + checks, download sizes). + +- **Lesson:** A kwconf flag (`isflag=True`) takes an optional value, so a + flag written before a positional swallows it: `logs -f qwen` parsed as + `follow='qwen'` with no names. infer-stack's flags are all boolean, so + `_FlagSafeMixin` hands a non-boolean string back to the positional list. +- **Evidence / MWE:** `tests/test_day2.py::test_a_flag_never_swallows_the_positional_after_it`; + `infer_stack/cli/options.py` (`reclaim_swallowed_positionals`). Found live + 2026-09-26: `logs -f ` followed every instance. +- **Applies when:** a kwconf command has both flags and positional + arguments. + +- **Lesson:** `producer | grep -q pattern` under `set -o pipefail` can fail + although the pattern matched: `grep -q` exits at the first match and the + producer's next write dies of SIGPIPE (141). Write to a file, then grep it. +- **Evidence / MWE:** `dev/lessons/mwe/grep_q_pipefail.sh` prints 141 for + the pipeline and 0 for the file. +- **Applies when:** an e2e or handover script checks a command's output with + `grep -q` under `set -euo pipefail`. diff --git a/dev/lessons/mwe/grep_q_pipefail.sh b/dev/lessons/mwe/grep_q_pipefail.sh new file mode 100644 index 00000000..d85acfd8 --- /dev/null +++ b/dev/lessons/mwe/grep_q_pipefail.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +# `producer | grep -q pattern` under `set -o pipefail` can report failure +# although the pattern matched: grep -q exits at its first match, the +# producer's next write gets SIGPIPE (exit 141), and pipefail reports that. +set -o pipefail +seq 1 1000000 | grep -q '^1$'; echo "pipeline: $? (141 = the producer's SIGPIPE, not a miss)" +seq 1 1000000 > /tmp/grep-q-mwe.txt; grep -q '^1$' /tmp/grep-q-mwe.txt; echo "file: $?" +rm -f /tmp/grep-q-mwe.txt diff --git a/dev/profile_tui.py b/dev/profile_tui.py new file mode 100644 index 00000000..88c252f8 --- /dev/null +++ b/dev/profile_tui.py @@ -0,0 +1,151 @@ +"""Measure how responsive the TUI's event loop is, and what blocks it. + +Runs the real app headless against a real ComposeBackend (over the tests' +fake Docker, so the numbers are the TUI's own cost, not Docker's), with a +ledger holding a few leases and deployments. A 50 ms timer on the app's own +loop measures lag: how late it fires is how late a keypress would be +handled. Scenarios: + + idle the dashboard refreshing on its normal timer + docker the docker pane open on the Logs tab (service list, log stream) + flood a log stream emitting lines as fast as it can (an engine loading) + + python dev/profile_tui.py [--profile] # --profile: cProfile the UI thread +""" + +from __future__ import annotations + +import asyncio +import cProfile +import io +import pstats +import statistics +import sys +import tempfile +import time +from pathlib import Path + +HERE = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(HERE / 'tests')) + +from test_leasing_admission import CAT, acquire, make # noqa: E402 + +from infer_stack.tui import InferStackTUI # noqa: E402 + +FLOOD_LINES = 20000 + + +class FloodProc: + """A log process that writes FLOOD_LINES lines as fast as it can.""" + + def __init__(self, n): + self.n = n + + @property + def stdout(self): + for i in range(self.n): + yield f'vllm-one | INFO 09-24 12:00:00 loader.py:1 loading shard {i}\n' + + def terminate(self): + pass + + +async def lag_probe(seconds: float, period: float = 0.05) -> list[float]: + lags = [] + end = time.perf_counter() + seconds + while time.perf_counter() < end: + t0 = time.perf_counter() + await asyncio.sleep(period) + lags.append(time.perf_counter() - t0 - period) + return lags + + +def summary(name: str, lags: list[float]) -> str: + ms = sorted(x * 1000 for x in lags) + p95 = ms[int(len(ms) * 0.95) - 1] if ms else 0 + return (f'{name:8s} samples={len(ms):4d} median={statistics.median(ms):7.1f} ms ' + f'p95={p95:7.1f} ms max={max(ms):7.1f} ms') + + +def main() -> None: + do_profile = '--profile' in sys.argv + # --docker-latency=S: each docker call takes S seconds, like a loaded host. + latency = next((float(a.split('=', 1)[1]) for a in sys.argv + if a.startswith('--docker-latency=')), 0.0) + # --tty: run on a real terminal (not headless), so drawing is measured too. + tty = '--tty' in sys.argv + import threading + tmp = Path(tempfile.mkdtemp()) + ledger, ctl, docker = make(tmp) + for name in ('one', 'two'): + out = acquire(ctl, name) + if name == 'two': + ctl.release(out.lease.id) # an idle keep-warm resident + flood = {'on': False} + if latency: + real_run = ctl.backend.run + + def slow_run(args, **kw): + time.sleep(latency) + return real_run(args, **kw) + ctl.backend.run = slow_run + + def proc_factory(service): + return FloodProc(FLOOD_LINES) if flood['on'] else FloodProc(0) + + results = [] + profiler = cProfile.Profile() + + async def scenario(): + app = InferStackTUI(ctl, CAT, interval=1.0, proc_factory=proc_factory) + async with app.run_test(size=(200, 60)) as pilot: + await pilot.pause(1.0) + if do_profile: + profiler.enable() + results.append(summary('idle', await lag_probe(8))) + app.query_one('#docker').collapsed = False + await pilot.pause(0.5) + results.append(summary('docker', await lag_probe(8))) + flood['on'] = True + app._restart_logs(app._log_service) + t0 = time.perf_counter() + results.append(summary('flood', await lag_probe(8))) + results.append(f'flood: {len(app._log_lines)} of {FLOOD_LINES} lines shown ' + f'after {time.perf_counter() - t0:.1f}s') + results.append(f'threads alive at end: {threading.active_count()}') + if do_profile: + profiler.disable() + + if tty: + # The app on the real terminal, driven by its own timers. + app = InferStackTUI(ctl, CAT, interval=1.0, proc_factory=proc_factory) + + async def drive(): + await asyncio.sleep(1.5) + results.append(summary('idle', await lag_probe(8))) + app.query_one('#docker').collapsed = False + await asyncio.sleep(0.5) + results.append(summary('docker', await lag_probe(8))) + flood['on'] = True + app._restart_logs(app._log_service) + t0 = time.perf_counter() + results.append(summary('flood', await lag_probe(8))) + results.append(f'flood: {len(app._log_lines)} of {FLOOD_LINES} lines shown ' + f'after {time.perf_counter() - t0:.1f}s') + results.append(f'threads alive at end: {threading.active_count()}') + app.exit() + + app.call_later(lambda: asyncio.ensure_future(drive())) + app.run() + else: + asyncio.run(scenario()) + for line in results: + print(line) + if do_profile: + out = io.StringIO() + pstats.Stats(profiler, stream=out).sort_stats('cumulative').print_stats(35) + print(out.getvalue()) + + +if __name__ == '__main__': + main() diff --git a/dev/tmp/plan-backend-unification-2026-09-24.md b/dev/tmp/plan-backend-unification-2026-09-24.md index 0c8b4e8f..0b1097c4 100644 --- a/dev/tmp/plan-backend-unification-2026-09-24.md +++ b/dev/tmp/plan-backend-unification-2026-09-24.md @@ -1,6 +1,11 @@ # Plan: one set of authorities for the Compose and KubeAI backends Status: draft for review, 2026-09-24. Nothing here is implemented. +Update 2026-09-25: K0, K1, K2, K4, K5 (except TUI logs) and G are done; +what remains (K3, TUI logs, dynamic routing on KubeAI, the in-cluster +gateway) is planned as phases in `docs/planning/backend-parity-roadmap.md`, +and the current state is `docs/backend-parity.md`. This file stays as the +audit record. ## Goal @@ -20,7 +25,7 @@ launches (`runtime.command` / `mounts`) stay Compose-only. | | Compose | KubeAI | |---|---|---| | front door | LiteLLM gateway, one `base_url` | KubeAI's own gateway | -| request name | the **endpoint alias** | a DNS slug of the served name (`Qwen/Qwen3.8-27B` → `qwen-qwen3-8-27b`) | +| request name | the **endpoint alias** | a DNS slug of the served name (`Org/Model-7B` → `org-model-7b`) | | auth | managed master key | none (`EMPTY`) | Cards and the pipeline send the endpoint alias as `model=`. Only one magnet @@ -38,12 +43,18 @@ path carries the keep-warm fix, atomic acquire and serialised publication review; the other path is the pre-September one. In production only KubeAI takes it, and nothing has run it at scale. -### C. Two liveness authorities +### C. No strict liveness view on KubeAI -Compose answers "what is running" with `residency()`: strict, and it raises -`ResidencyUnknown` rather than guess. KubeAI answers with `observe()`, which -returns the **empty set** when `kubectl` fails. Compose had that exact bug: -an unreadable Docker looked like "nothing running" and led to wrong decisions. +Both backends have a lenient `observe()` that returns an empty set when the +runtime cannot be read. That is deliberate: it is for reporting. Compose also +has `residency()`, a strict view that raises `ResidencyUnknown` rather than +guess, and every decision that stops, evicts or hands over a GPU uses it. +KubeAI had only the lenient view. (Corrected 2026-09-24: the first draft +called KubeAI's `observe()` a bug; it matches Compose's contract.) + +Three CLI commands (`gc --orphans`, its orphan listing, `network migrate`) +used "has `residency`" to mean "is the Compose backend", which is a duplicate +authority of its own. ### D. Failure diagnosis only exists on Compose @@ -80,8 +91,17 @@ No GPU needed. Add `dev/e2e_tests/kubeai_kind.sh`: acquire, `run` a request, release, and verify the Model is pruned. Without it every step below is fake-verified only, which is how KubeAI drifted. -Open: installing `kind`/`kubectl`/`helm` on the guest needs network access to -their release downloads. +**Done 2026-09-24**, with k3s instead of kind (k3s runs as a systemd service +on the guest; `--disable traefik --disable servicelb` keeps ports 80/443 +free). KubeAI 0.23.4's `cpu` profile runs real vLLM 0.11.2 on CPU (needs +AVX-512), which is better than the simulator: KubeAI builds vLLM's own +command line, which the simulator's CLI would reject. Setup: +`dev/e2e_tests/kubeai-cpu-values.yaml`; check: +`dev/e2e_tests/kubeai_k3s.sh`, which passed. `doctor` passed its four checks; +acquire → ready took 143 s with an image pull, the full `run` took 86 s with +the image cached, a real completion came back, and release pruned the Model. +A Docker container on the node reaches KubeAI's gateway at its cluster IP, +which K1 relies on. ### K1. One client contract: the LiteLLM gateway fronts both backends @@ -106,6 +126,15 @@ Recommendation: (1) first, since it reuses everything and unblocks cards now; (2) when a real multi-node run needs it. Record (1)'s single host as a known limitation. +**Done 2026-09-24 as (1).** The kubeai backend owns a gateway-only +`ComposeBackend` (its own state dir and project, `infer-stack-gateway`) and +feeds it one generic `upstream` route row per alias (`api_base` + the Model +name). The same row type is what external endpoints need. Verified on k3s +with `dev/kubeai_e2e.sh`: without the gateway, the alias returned 404; with +it, the same request answered, and `secrets rotate` accepted the new key and +rejected the old one. Static routes only; dynamic routing stays +compose-only. + ### K2. One liveness authority Generalise `residency()` into a backend-neutral snapshot: per deployment, @@ -114,6 +143,14 @@ reason, waiting reason, and GPUs when the backend knows them. Build KubeAI's from `kubectl get pods -l model= -o json`, raising `ResidencyUnknown` when kubectl fails. `observe()` becomes a thin view of it on both backends. +**Done 2026-09-24.** `residency_from_pods` builds the same `Residency` +from `kubectl get pods`, selecting on the `infer-stack/managed` label KubeAI +copies onto pods. Pod states map onto the Docker vocabulary, with the +Kubernetes reason kept in `Container.reason`, and a kubectl failure raises. +With it, `_never_ran`, the `config publish` quiescence check and the status +view use strict residency on KubeAI too. The three CLI commands now check for +the Compose backend explicitly. + ### K3. One acquire path Give placement to the backend: @@ -128,6 +165,12 @@ backends, `_admission_mode()` is always true for real backends. The non-admission branches are then deleted. Memory and Null backends get a trivial residency so tests use the same path. +**Decision (user, 2026-09-24):** "Keep warm models without a lease should +always be candidates for eviction if a model with a lease needs a resource." +Implemented backend-neutrally ahead of K3: `Readiness.needs_room`, and +`Controller._make_room` in the wait. The one acquire path itself (K3) is +still open. + ### K4. One failure diagnosis Move `startup_failure` and the log classifier behind the K2 snapshot. Compose @@ -137,6 +180,12 @@ The fail-fast wait, the "likely cause" text and the TUI error path then work on both. Pull progress stays Compose-only; Kubernetes pulls by itself, and `ImagePullBackOff` is reported as a failure. +**Done 2026-09-24.** The diagnosis moved to `leasing/diagnosis.py` +(`diagnose_startup(instances, read_logs)`), and both backends call it. On k3s, +a Model whose vLLM rejected a flag failed its acquire in 52 s with the +engine's error quoted, instead of waiting out the 800 s timeout. Adding +`error: unrecognized arguments` as a fatal signature helps compose as well. + ### K5. Small cleanups - one served-name function @@ -145,6 +194,11 @@ on both. Pull progress stays Compose-only; Kubernetes pulls by itself, and - TUI engine logs through a backend `deployment_logs` stream, with `kubectl logs -f` on KubeAI +**Done 2026-09-24** except TUI logs: there is one `served_name` (it fixed a +real disagreement, noted in the changelog), KubeAI's GPU units come from +`placement.required_gpu_count`, and KubeAI takes `runtime.env`. The unused +profile-era KubeAI renderer is deleted. TUI engine logs on KubeAI remain open. + ## Order and size K0, then K1, is the minimum for a card to run unchanged on a cluster. K2 and @@ -159,3 +213,43 @@ to the kind script. - multi-cluster, or scheduling across a mix of Compose hosts and a cluster - KubeAI autoscaling policy beyond today's `min_replicas` / `max_replicas` - custom container launches on KubeAI + +## G. Pull the gateway out of compose.py (added 2026-09-24) + +`compose.py` mixes two jobs: engine containers (placement, render, +selective apply, pulls, diagnosis) and the front door (LiteLLM config and +service, route registry, dynamic-route reconciliation, keys, Open WebUI, +nginx). K1 reused the whole `ComposeBackend` as a gateway-only instance for +KubeAI. That added a mode rather than naming the seam. The user asked for +the tool to be more elegant, architecture first and the TUI after. + +The gateway methods depend only on the state dir, ports, the HTTP client, +the clock, the catalog and the flags (measured by AST), not on engine state. +So the split is: + +- **A. `leasing/naming.py`**: `dns_slug` and the vLLM/Ollama service-name + rules. Engines and gateway routes both use them, and so does KubeAI. +- **B. `leasing/gateway.py`**: the route functions, front-door service + renderers and gateway constants, moved as-is. +- **C. `Gateway`**: keys, route registry and dynamic-route reconciliation. + `ComposeBackend` owns one and keeps thin public delegations (`master_key`, + `access`, ...). +- **D. `render_front_door`**: `render_compose`'s second half. + +`compose` re-exports nothing private; importers (mostly tests) move to the +new modules. Each step leaves the full suite, `ty` and flake8 green. The +gateway still runs as a Compose project (it is a container), so KubeAI keeps +a `ComposeBackend` with no engines: that is now simply "a Compose project +with only the front door", not a special mode. + +**G done 2026-09-24** (on `dev/backend-unification`). A: `naming.py` +(99 lines). B: `gateway.py` module code. C: the `Gateway` class; backends +supply route rows, the gateway merges, persists and renders them (the +`upstream_routes` side-channel is gone), and the backend's gateway settings +are properties onto the one `Gateway`. D: `render_front_door`, verified +byte-identical across six configurations. `compose.py` went from 3651 +lines (on `main`) to 2360; `gateway.py` is 1377. Verified live after C: a +Compose gateway apply and `secrets rotate`, and the KubeAI e2e. + +Found on the way: the weight-floor doctest wrote 4 GiB to `/tmp` per run +(fixed on `main`, 21f5e09). diff --git a/dev/ux_audit.sh b/dev/ux_audit.sh new file mode 100755 index 00000000..144a4f3e --- /dev/null +++ b/dev/ux_audit.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# The automated part of the UX audit (docs/queue.md, item 9): every command's +# help, the examples in them, common mistakes, and the day-2 commands with a +# model up, on one backend. It prints what it ran and flags the shapes that +# have been bugs before: a traceback, compose/docker wording on kubeai, a raw +# timestamp, ANSI codes in piped output, stray "Write .env" lines. +# +# dev/ux_audit.sh compose # the simulator catalog, no GPU +# dev/ux_audit.sh kubeai # CPU vLLM on the dev cluster +# +# The TUI, the first run from the README and wording consistency stay manual: +# they need eyes (docs/queue.md says what to look at). Isolated config and +# data roots; everything is released and taken down at the end. +set -uo pipefail + +BACKEND="${1:-compose}" +HERE="$(cd "$(dirname "$0")/.." && pwd)" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/infer-stack-ux.XXXXXX")" +export INFER_STACK_CONFIG_DIR="$WORK/config" INFER_STACK_DATA_DIR="$WORK/data" +mkdir -p "$WORK/config" +REPORT="$WORK/report.txt" +FLAGS=0 + +flag() { echo "!! $*" | tee -a "$REPORT"; FLAGS=$((FLAGS + 1)); } +run() { # run a command, keep its output, and check it for known bad shapes + local out rc + out=$(timeout 300 infer-stack "$@" 2>&1); rc=$? + { echo "\$ infer-stack $* (rc=$rc)"; echo "$out" | head -20; echo; } >> "$REPORT" + echo "$out" | grep -q 'Traceback' && flag "traceback: infer-stack $*" + echo "$out" | grep -q 'Write .env' && flag "stray 'Write .env': infer-stack $*" + echo "$out" | grep -qE 'ttl=@[0-9]' && flag "raw timestamp: infer-stack $*" + echo "$out" | grep -q $'\x1b\[' && flag "ANSI codes in piped output: infer-stack $*" + if [ "$BACKEND" = kubeai ]; then + echo "$out" | grep -v '^\s*\(INFO\|[0-9:]* INFO\)' \ + | grep -iqE '\bcompose (project|backend)\b|docker compose' \ + && flag "compose wording on kubeai: infer-stack $*" + fi + return 0 +} + +cleanup() { + infer-stack clean -f >/dev/null 2>&1 + infer-stack stack down >/dev/null 2>&1 + echo "report: $REPORT" + echo "flags: $FLAGS" +} +trap cleanup EXIT + +case "$BACKEND" in + compose) + cp "$HERE/dev/e2e_tests/catalog-mock.yaml" "$WORK/config/catalog.yaml" + ENDPOINT=mock-smol + ;; + kubeai) + cat > "$WORK/config/catalog.yaml" <&2; exit 2 ;; +esac +infer-stack config set backend "$BACKEND" >/dev/null + +echo '== help: every one-line summary is a sentence' +COLUMNS=250 infer-stack help tree 2>/dev/null | sed -E 's/^[│ ├└─]+//' \ + | awk '{$1=""; print substr($0,2)}' | grep -vE '[.)?]$|^$' \ + | while read -r line; do flag "summary ends mid-sentence: $line"; done + +echo '== mistakes: each names its cause, none is a traceback' +run acquire nope --yes +run release lease-nope --yes +run wait nope --timeout 5 +run logs nope +run evict grp-nope +run renew lease-nope --ttl 1h +run test "$ENDPOINT" --timeout 5 + +echo "== day-2, with $ENDPOINT up" +run acquire "$ENDPOINT" --yes --ttl 1h --timeout 600 --env-file "$WORK/lease.env" +for cmd in leases status ps env "env OPENAI_BASE_URL" doctor clean gc \ + "renew --env-file $WORK/lease.env --ttl 2h" "routes list" \ + "wait $ENDPOINT" "test $ENDPOINT" "logs $ENDPOINT --tail 3"; do + # shellcheck disable=SC2086 + run $cmd +done +run release --env-file "$WORK/lease.env" --yes diff --git a/docs/backend-parity.md b/docs/backend-parity.md new file mode 100644 index 00000000..ee60b760 --- /dev/null +++ b/docs/backend-parity.md @@ -0,0 +1,155 @@ +# Compose and KubeAI: parity and deviations + +infer-stack has one control plane and two realization layers. Everything +above the backend seam is shared: the catalog, the ledger (leases, +deployments, TTLs), the controller (`acquire` / `release` / `wait` / `evict` +/ `gc` / `clean` / `renew`, the admission queue, keep-warm reclaim), the +LiteLLM front door, the env-file contract, and the TUI. A backend only makes +the desired set real: Compose as containers on the local Docker daemon, +KubeAI as `Model` custom resources on a cluster. + +The intended relationship is **KubeAI = Compose + a scheduler**. A catalog +that runs on Compose runs on KubeAI once you say what a GPU unit means on +the cluster and install the cluster tooling. A card does not change: the same +`OPENAI_BASE_URL`, the same managed key, the endpoint alias as the model +name. + +This page records where that holds today (2026-09-25), where it does not, +and which gaps are deliberate. The plan for the rest is +[planning/backend-parity-roadmap.md](planning/backend-parity-roadmap.md); +cluster setup is [kubeai-backend.md](kubeai-backend.md). + +## What you add to move from Compose to KubeAI + +| | Compose | KubeAI adds | +|---|---|---| +| tools | Docker with the NVIDIA runtime | a cluster (`scripts/bootstrap_k3s.sh` for one host, `scripts/join_agent.sh` for another), `kubectl`, `helm`, the KubeAI chart (`scripts/install_kubeai.sh`), the NVIDIA device plugin on GPU nodes | +| information | none: GPUs are discovered with `nvidia-smi` | `resourceProfiles` in the chart values (what one GPU unit requests, and on which nodes; `catalog suggest --backend kubeai` proposes them from the nodes' GPU labels), and per endpoint `runtime.resource_profile`, a `min_vram_gib` that picks one by size, or a default `kubeai_resource_profile` | +| settings | `backend compose` | `backend kubeai`; `kubeai_gateway cluster` for more than one workstation; `kubeai_namespace`, `kubeai_base_url`, `kubeai_gateway_upstream` when the defaults (namespace `kubeai`, a port-forward on 8000, the Service's cluster IP) do not hold | +| preflight | none needed | `infer-stack doctor`: cluster → CRD → namespace → KubeAI's API | + +The catalog, `acquire … --env-file`, `release`, the TUI and the env file a +card sources are the same on both. + +## Layers + +``` +catalog ──► ledger ──► controller ──► gateway (LiteLLM: aliases, key, routes) + │ + ▼ backend seam: leasing/backend.py + ┌──────────┴───────────┐ + ComposeBackend KubeaiBackend + docker compose kubectl apply of Model CRs + local GPU planner the cluster scheduler + leasing/compose/ leasing/kubeai/ + leasing/kubeai-gateway/ +``` + +On KubeAI the gateway is a Compose project with no engines +(`infer-stack-gateway`) on the host running infer-stack. It routes each +alias to the cluster's KubeAI Service under the Model's name, so `secrets +rotate` and the static superset route table work unchanged. + +## Parity matrix + +**same**: one code path, or verified equivalent; every *same* row has a test +in `tests/test_parity.py` that runs the same scenario on both backends. **≈**: the same outcome by +a different mechanism. **gap**: missing on one side and on the roadmap. +**n/a**: does not apply there. **boundary**: deliberately unsupported +(see [planning/known-limitations.md](planning/known-limitations.md)). + +### Client contract + +| | Compose | KubeAI | +|---|---|---| +| one `OPENAI_BASE_URL`, the managed key, the alias as model name | same | same through the gateway; with `litellm false`, Model names and no key | +| `secrets rotate` | same | same | +| env file (`INFER_STACK_*`, `OPENAI_*`) | same | same | +| readiness is a real generation through the front door | same | same | + +### Lifecycle + +| | Compose | KubeAI | +|---|---|---| +| `acquire` / `release` / `wait` / `evict` / `gc` / `clean` / `renew` / `run` / `test` | same | same | +| admission: previewed in memory, then the lease committed with its allocation; a refused acquire writes nothing | yes | same; the allocation is empty (the cluster places), so admission decides renderability, and a pod the cluster cannot place is a wait reason | +| `--queue` | waits for a free GPU | ≈ admitted at once, the cluster is the queue; an unrenderable endpoint fails at once | +| `config publish` | pure preview, then commit | same | +| `--no-apply` / `apply` / `render` | same | same | +| unleased keep-warm yields to leased demand | at placement | ≈ during the wait: `needs_room` evicts the longest-idle, one per 30 s | +| crash-loop fail-fast, the engine's error quoted | same | same (pods, `kubectl logs --previous`) | +| image-pull progress | yes | n/a: the kubelet pulls; `ImagePullBackOff` is a reported wait reason | +| strict `residency()` for decisions, lenient `observe()` for reports | same | same | +| recovery after an interrupted apply (settle check) | yes | ≈ `kubectl apply` is idempotent; nothing is left half-created | + +### Placement + +| | Compose | KubeAI | +|---|---|---| +| where a deployment lands | local planner over `nvidia-smi` | the cluster scheduler, via `resource_profile:` | +| GPU count from TP × PP × DP | same | same | +| `placement.gpu_indices`, `allowed_gpus`, `skip_display_gpus` | yes | n/a: no host indices; node-scoped resource profiles are the equivalent | +| `placement.min_vram_gib` (declared, or recorded by `measure --record`) | picks GPUs that large | ≈ picks the smallest resource profile whose nodes' GPUs are that large (GFD's `nvidia.com/gpu.memory` label); verified with fake labels, a real-GPU run is `dev/handover/p4_gpu_labels.sh` | +| `catalog suggest` | sized to this host's GPUs | ≈ sized to the largest GPU node, plus a `resourceProfiles` block per GPU product | +| GPU allocations recorded in the ledger | yes | none: the cluster owns them | +| more than one host | boundary | yes: the reason the backend exists | + +### Catalog features + +| | Compose | KubeAI | +|---|---|---| +| vLLM endpoints, `runtime.*` flags, `extra_args`, `env` | same | same (one argument pipeline) | +| served-name rules | same | same (`leasing/naming.py`) | +| `runtime.command`, `runtime.mounts` (custom launchers) | yes | boundary: a stock vLLM Model has no place for them; the render refuses loudly | +| ollama endpoints (`runtime_hosts`) | yes | boundary: daemon-shaped, not model-shaped | +| `min_replicas` / `max_replicas` | n/a | pass-through, default 1/1 | +| weight cache | the host HF cache, mounted | whatever the chart provides; cache profiles are not rendered | + +### Gateway + +| | Compose | KubeAI | +|---|---|---| +| static superset routes, no blip on model churn | same | same | +| `routes` inspect / seed / prune | yes | same; a route points at the cluster under the Model's name | +| `dynamic_routing` (admin API + Postgres; distinct upstreams for same-model `--dedicated`) | yes | same: each deployment is its own Model (`-`) | +| Open WebUI (`ui`), reverse proxy | yes | same, in the gateway's project on this host | +| the gateway's changes approved with the acquire's, before the lease commits | yes | same | +| `network migrate` / `network check` | yes | n/a: Service addresses are stable | +| where the gateway runs | this host | this host (default), or the cluster (`kubeai_gateway cluster`: a Deployment and a NodePort any node answers on; static routes, no UI) | + +### Day-2 operations and the TUI + +| | Compose | KubeAI | +|---|---|---| +| `doctor` | nothing to check | four checks | +| `ps`, `logs [-f] `, `status` health | containers | same shape; pods, and the gateway's containers | +| `stack up` (= `apply`), `stack down` | same | same | +| `stack compose …`, `restart` / `pull` / `start` / `stop` | the stack's Compose project | the gateway's Compose project (the engines are pods) | +| TUI: leases, deployments, catalog editing, acquire / release / evict, API tab, settings | same | same | +| TUI: log follow, the Instances tab, Apply / Down | containers | same, over pods | +| TUI GPU pane | `nvidia-smi` on this host | this host, not the cluster | +| `gc --orphans` (also inside `clean`) | yes | runs, and finds none: pods are read by infer-stack's label, so unlabelled Models are never touched | + +### Testing + +| | Compose | KubeAI | +|---|---|---| +| unit suite with fake runtimes | yes | yes | +| parity suite (`tests/test_parity.py`: each *same* row, both backends) | yes | yes | +| end to end | `dev/e2e_tests/run.sh` (tiers; `--gpu` for serving) | `dev/kubeai_e2e.sh` against a real cluster; k3s with CPU vLLM needs no GPU | + +## Deviations that stay + +- **Custom container launches and ollama daemons** do not run on KubeAI. A + KubeAI Model is stock vLLM. Use `--backend compose` for those endpoints. +- **GPU indices** have no cluster meaning. Kubernetes places by resource, not + by index; the per-node equivalent is a resource profile with a node + selector. +- **One control plane never spans a Compose host and a cluster.** Pick a + backend per data root. + +## Where the gaps close + +[planning/backend-parity-roadmap.md](planning/backend-parity-roadmap.md) +has the phases and their exit criteria. The audit that produced this page, +with what has already landed, is +`dev/tmp/plan-backend-unification-2026-09-24.md`. diff --git a/docs/demos/ci_smoke_test.md b/docs/demos/ci_smoke_test.md index a1b6457d..763c08e8 100644 --- a/docs/demos/ci_smoke_test.md +++ b/docs/demos/ci_smoke_test.md @@ -1,5 +1,7 @@ # CI smoke test +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../../README.md). + Walks through the canonical `infer-stack` workflow end-to-end: pick a profile, render the deployment, inspect the rendered artifacts, and exercise a few read-only inspection commands. diff --git a/docs/demos/ollama_direct_quickstart.md b/docs/demos/ollama_direct_quickstart.md index 3ec204a6..fd790ad8 100644 --- a/docs/demos/ollama_direct_quickstart.md +++ b/docs/demos/ollama_direct_quickstart.md @@ -1,5 +1,7 @@ # Quickstart — direct Ollama on a dual GTX 1080 Ti host +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../../README.md). + This quickstart brings up the simplest local stack for Pascal-era GPUs: ```text diff --git a/docs/demos/quickstart.md b/docs/demos/quickstart.md index 7ef85742..b9fe2169 100644 --- a/docs/demos/quickstart.md +++ b/docs/demos/quickstart.md @@ -1,5 +1,7 @@ # Quickstart — vLLM + LiteLLM on an RTX 3090 workstation +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../../README.md). + This guide brings up the traditional vLLM-backed stack on a single-GPU workstation where GPU 0 may be reserved for the desktop and GPU 1 is free for inference. It starts with the tiniest vLLM model to prove the plumbing works, diff --git a/docs/demos/qwen_3090_profiles.md b/docs/demos/qwen_3090_profiles.md index aebc875b..6cb591ca 100644 --- a/docs/demos/qwen_3090_profiles.md +++ b/docs/demos/qwen_3090_profiles.md @@ -1,5 +1,7 @@ # Qwen profiles for a 3090 workstation +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../../README.md). + These built-in profiles are intended for quick GPU-1 testing on a workstation-class RTX 3090 host. diff --git a/docs/demos/smollm2_gpu1_backend_switch.md b/docs/demos/smollm2_gpu1_backend_switch.md index b7db1e86..53641c8f 100644 --- a/docs/demos/smollm2_gpu1_backend_switch.md +++ b/docs/demos/smollm2_gpu1_backend_switch.md @@ -1,5 +1,7 @@ # SmolLM2 backend switch test on GPU 1 +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../../README.md). + This is a copy/paste smoke path for a workstation where GPU 1 is available. It uses only built-in profiles and `infer-stack` runtime commands. diff --git a/docs/kubeai-backend.md b/docs/kubeai-backend.md index 374f1aa5..93cad3b9 100644 --- a/docs/kubeai-backend.md +++ b/docs/kubeai-backend.md @@ -10,10 +10,19 @@ only the realization layer changes: |---|---|---| | unit of serving | docker compose service | KubeAI `Model` CR | | GPU placement | planned locally (`plan_placement`) | cluster-scheduled via `resourceProfile` | -| front door | LiteLLM gateway | KubeAI's own OpenAI-compatible gateway | -| request name | endpoint alias (LiteLLM route) | Model CR name (dns-slug of the served name) | -| auth | managed `LITELLM_MASTER_KEY` | none (`api_key: EMPTY`) | -| state dir | `/leasing/compose/` | `/leasing/kubeai/` (`models.yaml` + sidecar) | +| front door | LiteLLM gateway | the same LiteLLM gateway, on this host, routing to KubeAI | +| request name | endpoint alias | endpoint alias | +| auth | managed `LITELLM_MASTER_KEY` | the same | +| state dir | `/leasing/compose/` | `/leasing/kubeai/` (`models.yaml` + sidecar), `/leasing/kubeai-gateway/` | + +Clients see the same contract on both backends: one `OPENAI_BASE_URL`, the +managed key, and the endpoint alias as the model name, so a card runs +unchanged. The gateway is a one-service Compose project +(`infer-stack-gateway`) that routes each alias to KubeAI under the Model's +name. `secrets rotate` works as on compose. With `--no-litellm` (or `config +set litellm false`) there is no gateway: clients talk to KubeAI directly and +must use the Model name from the env file (`INFER_STACK_ENDPOINT_*`); the +alias gets HTTP 404. A deployment in the desired set renders as one `Model` CR labeled `infer-stack/managed=true` + `infer-stack/deployment=`; `apply` is @@ -22,6 +31,10 @@ semantics carry over: an idle `keep-warm` deployment keeps its CR (model stays resident); `stop` deployments are pruned on release; `evict`/`gc` free the cluster. Hand-applied Models without the managed label are never touched. +Where the two backends match and where they still differ, row by row: +[backend-parity.md](backend-parity.md); the plan for the rest is +[planning/backend-parity-roadmap.md](planning/backend-parity-roadmap.md). + ## One-time cluster setup ```bash @@ -30,12 +43,20 @@ cluster. Hand-applied Models without the managed label are never touched. # 2. Resource profiles: name -> the requests/limits/nodeSelector that one # "GPU unit" means on your cluster. These names are what the catalog's -# `runtime.resource_profile` refers to. +# `runtime.resource_profile` refers to. Without the GPU request and +# runtimeClassName the pod can land on a GPU node and still start without +# libcuda.so.1. `infer-stack catalog suggest --backend kubeai` proposes +# one per GPU product once the device plugin runs. cat > kubeai-values.yaml <<'EOF' resourceProfiles: nvidia-gpu-rtx-4090: + runtimeClassName: nvidia + requests: + nvidia.com/gpu: "1" limits: nvidia.com/gpu: "1" + nodeSelector: + nvidia.com/gpu.product: NVIDIA-GeForce-RTX-4090 EOF # 3. Install the chart (HF_TOKEN, if exported, is passed to the chart secret): @@ -54,6 +75,9 @@ infer-stack config set kubeai_namespace kubeai infer-stack config set kubeai_base_url http://127.0.0.1:8000/openai/v1 # fallback profile for endpoints whose runtime omits resource_profile: infer-stack config set kubeai_resource_profile nvidia-gpu-rtx-4090 +# how the gateway reaches KubeAI (default: the kubeai Service's cluster IP, +# reachable from a cluster node; set an ingress URL when this host is not one): +infer-stack config set kubeai_gateway_upstream http://kubeai.example/openai/v1 ``` Catalog endpoints opt into a specific profile per endpoint; the GPU count is @@ -71,8 +95,25 @@ endpoints: resource_profile: nvidia-gpu-rtx-4090 # -> nvidia-gpu-rtx-4090:2 ``` +### Sizing: `min_vram_gib` picks a profile + +An endpoint that names no `resource_profile` but declares +`placement.min_vram_gib` (or has one recorded by `infer-stack measure +--record`) gets the smallest resource profile whose GPUs are that large. A +profile's GPU size is the `nvidia.com/gpu.memory` label (set by GPU Feature +Discovery) of the nodes its `nodeSelector` selects; a profile without a +`nodeSelector`, like the chart's generic ones, has no size and is never +picked this way. With none large enough, the `kubeai_resource_profile` +default is used, or the acquire is refused with the sizes it found. +`infer-stack catalog suggest` proposes one sized profile per GPU product in +the cluster, ready for the helm values: + +```bash +infer-stack catalog suggest --backend kubeai # catalog on stdout, profiles on stderr +``` + Verify the setup before the first acquire — `doctor` checks the chain in -dependency order (cluster reachable → CRD installed → namespace → gateway): +dependency order (cluster reachable → CRD installed → namespace → KubeAI's API): ```bash infer-stack doctor @@ -82,20 +123,93 @@ Then the normal verbs just work: ```bash infer-stack acquire qwen-coder --ttl 2h --env-file lease.env --yes -source lease.env # OPENAI_BASE_URL -> the KubeAI gateway -# note: request the model by its CR name (the env-file's endpoint mapping -# carries it), not the endpoint alias — KubeAI has no alias layer. +source lease.env # OPENAI_BASE_URL + OPENAI_API_KEY -> the gateway +# request model=qwen-coder, the endpoint alias, as on the compose backend infer-stack release --env-file lease.env ``` +## The gateway inside the cluster + +By default the gateway is a Compose project on the host running infer-stack, +so that host is in every request's path. For more than one workstation, put +it in the cluster: + +```bash +infer-stack stack down # the host gateway (and any Models) +infer-stack config set kubeai_gateway cluster +infer-stack acquire --env-file lease.env --yes +# OPENAI_BASE_URL is now http://:30442/v1: any node answers +``` + +It is the same gateway (image, config, managed key, route registry) as a +Deployment and a NodePort Service in the KubeAI namespace, and it reaches +KubeAI by the Service's cluster DNS name, so no `kubectl port-forward` is +needed. `secrets rotate` updates its Secret and rolls it; `doctor` checks it; +`stack down` removes it. Settings: `kubeai_gateway_node_port` (30442) and +`kubeai_gateway_url` (an ingress URL clients should use instead). Static +routes only: dynamic routing and Open WebUI need the host placement. + +## Add a workstation + +The cluster's first node is the one `scripts/bootstrap_k3s.sh` set up. To add +a GPU workstation as a second node: + +1. On the first node, collect the join facts: + ```bash + sudo cat /var/lib/rancher/k3s/server/node-token # the token + k3s --version # join with the same version + ``` +2. On the new workstation (NVIDIA driver and container toolkit installed), + join with the server's version: + ```bash + INSTALL_K3S_VERSION='v1.36.4+k3s1' \ + scripts/join_agent.sh https://:6443 + ``` + Between the nodes, open 6443/tcp (to the first node), 8472/udp (flannel) + and 10250/tcp; for clients, the NodePort (30442/tcp). +3. Back on the first node: the NVIDIA device plugin is a DaemonSet, so it + starts on the new node by itself. Check the new node reports its GPUs and + labels, then add a sized profile for its GPU product: + ```bash + kubectl get node -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}' + infer-stack catalog suggest --backend kubeai # profiles per GPU product (stderr) + ``` + Merge the proposed `resourceProfiles` into your helm values and rerun + `scripts/install_kubeai.sh `. An endpoint that names that profile, + or declares a `min_vram_gib` only that GPU meets, lands on the new node. +4. With the gateway in the cluster (above), a card on either workstation uses + the env file as it is: the NodePort answers on every node. + +`dev/k3s_agent_container.sh` makes a second node out of a container on one +host, for development; `dev/handover/p5_two_hosts.sh` checks a real one. + ## Semantics + limitations - **Readiness is a real generation** through the gateway (same philosophy as compose): a Model CR existing — even with ready replicas — is not proof it can serve. -- **Placement can only fail at admission.** The render never rejects for - capacity (the cluster schedules); a Model the cluster cannot place sits - not-ready until the acquire's `--timeout`, which then rolls the lease back. +- **Admission is the same as on compose, minus GPU accounting.** An acquire + is previewed in memory and commits its lease only if every deployment + renders; a refused one (no resource profile, an ollama endpoint, a + served-name collision) writes no lease and runs no `kubectl`. The cluster + schedules, so capacity is never an admission reason and `--queue` is + admitted at once. A Model the cluster cannot place sits not-ready until + the acquire's `--timeout`, which rolls the lease back; the wait says why + (`pod: Unschedulable`, `pod: ImagePullBackOff`). +- **An idle keep-warm Model stays only while its pod has started**, as an + idle keep-warm container does on compose (a crash-looping pod counts, like + a restarting container). One whose pod is gone, still Pending or finished + is pruned at the next render. +- **A keep-warm model without a lease gives way to one with a lease.** When + a leased Model's pod is `Unschedulable`, the wait evicts the longest-idle + keep-warm deployment, one per 30 s, until it fits. Compose applies the same + rule when placing. A leased model is never evicted this way. Verified on + k3s: `E2E_MAKE_ROOM=1 dev/kubeai_e2e.sh` with the `cpu-half` profile. +- **An engine that cannot start fails the acquire at once**, as on compose: + the crash diagnosis reads the pods (`kubectl get pods`, restart count, last + exit) and the previous run's log, and quotes the engine's error with a + likely cause. Verified on k3s: a rejected vLLM flag failed in 52 s instead + of the 800 s timeout. Loud render failures do exist for: a missing `resource_profile` (no invalid CR is ever written), served-name collisions between simultaneously live deployments, and **ollama endpoints** — KubeAI serves models, not daemons, @@ -103,8 +217,20 @@ infer-stack release --env-file lease.env `--backend compose` for those. - `min_replicas` / `max_replicas` runtime keys pass through to the CR (default 1/1 — the lease lifecycle, not the autoscaler, decides residency). -- The legacy profile-era renderer (`infer_stack/backends/kubeai_renderer.py`, - `kubeai_ops.py`) is superseded by this backend and kept only for reference. -- `dev/kubeai_e2e.sh` runs the full lifecycle (doctor → acquire → generation - through the gateway → release → prune-verified) against a real cluster with - an isolated config/data root; use it as the first smoke test on new setups. +- `dev/kubeai_e2e.sh` runs the full lifecycle (doctor → acquire → a request + by alias, as a card makes it → release → prune verified) against a real + cluster with an isolated config/data root; use it as the first smoke test + on new setups. Without a GPU, install the chart with + `dev/e2e_tests/kubeai-cpu-values.yaml` (real vLLM on CPU; needs AVX-512) + and run it with `E2E_RESOURCE_PROFILE=cpu`. Verified on k3s 2026-09-24. +- `infer-stack ps`, `infer-stack logs [-f] `, `status` and the TUI's + runtime pane read the pods (and the gateway's containers), in the same shape + as on compose. `stack compose …` and `stack restart` act on the gateway's + Compose project; the engines are pods, restarted by the kubelet. +- By default the gateway runs on the host running infer-stack, so that host is + in every request's path (`kubeai_gateway cluster` moves it; see above). It + takes the same settings as on compose: `ui` (Open WebUI, + on by default), `reverse_proxy`, and `dynamic_routing`, under which each + deployment is its own Model (`-`), so `--dedicated` twice + gives two Models behind one alias. The gateway's config changes are shown + and approved with the acquire's, before its lease commits. diff --git a/docs/persistent-caches-and-warm-restarts.md b/docs/persistent-caches-and-warm-restarts.md index 9426297e..e5cd8f21 100644 --- a/docs/persistent-caches-and-warm-restarts.md +++ b/docs/persistent-caches-and-warm-restarts.md @@ -11,9 +11,8 @@ process loads a model. ### Ollama model store — `state.ollama -> /root/.ollama` Ollama downloads GGUF/model blobs into `/root/.ollama`. The Compose template -mounts `state.ollama` there whenever the Ollama provider is enabled. Direct -Ollama profiles therefore survive `down`, `up`, `switch`, and `render` without -re-pulling models. +mounts `/ollama` there for every Ollama daemon, so pulled tags +survive container replacement, `release`, and `infer-stack stack down`. Ollama model residency is controlled by daemon/request settings such as `keep_alive` / `OLLAMA_KEEP_ALIVE`. A short keep-alive can let a mostly idle @@ -40,10 +39,10 @@ dtype, etc.). On a cold container with an empty cache, vLLM has to re-compile; on a warm restart against the same configuration, those artifacts are reused. -Default host path: `/data/service/docker/infer-stack/vllm-cache`. Override via -`state.vllm_cache` in `config.yaml` or by editing the rendered deployment -plan. The path is created on first volume mount; no manual `mkdir` is -required. +Host path: `/vllm-cache/cfg-`, one subdirectory per serve +configuration (`infer-stack paths` shows the data dir). It moves with the data +dir (`infer-stack config set data_dir `); there is no separate setting. +The path is created on first volume mount; no manual `mkdir` is required. The cache key is keyed on the engine configuration. Changing `max_model_len`, `tensor_parallel_size`, `gpu_memory_utilization`, the optimization level, @@ -58,11 +57,13 @@ The Compose template also persists the other obvious startup caches that sit outside `VLLM_CACHE_ROOT`: - `state.torch_cache -> /root/.cache/torch` for PyTorch / TorchInductor - artifacts, with `TORCH_HOME` and `TORCHINDUCTOR_CACHE_DIR` set explicitly. -- `state.triton_cache -> /root/.cache/triton` for Triton kernels, with - `TRITON_CACHE_DIR` set explicitly. -- `state.cuda_cache -> /root/.cache/nvidia/ComputeCache` for NVIDIA driver - JIT artifacts, with `CUDA_CACHE_PATH` set explicitly. + artifacts. +- `state.triton_cache -> /root/.triton` for Triton kernels. +- `state.cuda_cache -> /root/.nv` for NVIDIA driver JIT artifacts. + +Each is mounted at the tool's default location; no cache environment +variables are set. Unlike the vLLM cache, these are shared across serve +configurations, because their entries are content-addressed. These caches reduce repeated compile/JIT work, but they do not make model switching instantaneous. A vLLM process still has to import Python modules, @@ -81,53 +82,44 @@ the vLLM engine process, because vLLM serves one model configuration per process in this stack. That is why even tiny models can take tens of seconds to come back healthy: the expensive work is not just downloading weights. -## Shared memory: `ipc: host` - -vLLM uses shared memory for tensor-parallel communication and worker IPC. -Docker's default `--shm-size` (64 MiB) is far too small. The Compose -template sets `ipc: host` on every vLLM service, matching the upstream vLLM -Docker guidance and giving the engine the host's shared-memory budget. +## Shared memory -If your environment forbids host IPC (multi-tenant cluster policy, etc.), -replace `ipc: host` in the rendered Compose with an explicit `shm_size` -sized for your tensor-parallel topology — typically a few GiB for TP > 1. +vLLM uses shared memory for tensor-parallel communication and worker IPC, and +upstream vLLM's Docker guidance is `ipc: host` or a `shm_size` of a few GiB +for TP > 1. The rendered Compose currently sets neither, so vLLM containers get +Docker's default 64 MiB `/dev/shm`. If an endpoint, most likely a +tensor-parallel one, fails on shared memory, this is the first thing to +check. ## Minimal-restart workflow -LiteLLM deliberately does **not** depend on provider health in the rendered -Compose file. That prevents Compose from restarting LiteLLM every time a vLLM -runtime container is replaced during a model swap. The CLI refreshes LiteLLM's -route table through its admin API when possible; smoke tests retry briefly -while the selected upstream model finishes loading. +LiteLLM deliberately does **not** depend on engine health in the rendered +Compose file, and its route table already names every catalog endpoint. So +replacing a vLLM container during a model swap neither restarts LiteLLM nor +rewrites its config. -When you are iterating on a single profile and don't want to bounce the whole -stack: +To refresh one engine without bouncing the gateway or Open WebUI: ```bash -# Pull refreshed images through the rendered Compose wrapper. -infer-stack pull vllm- +# Pull a refreshed image for one service (names from `infer-stack ps`). +infer-stack stack pull vllm- -# Restart just the one service through the wrapper. This avoids bouncing -# Postgres / LiteLLM / Open WebUI. -infer-stack restart vllm- +# Restart just that service. +infer-stack stack restart vllm- ``` -For a full stack restart (e.g. after `infer-stack render` against a new -profile), `infer-stack down && infer-stack up -d` is the safe path; persistent -volumes and bind mounts (Postgres, Open WebUI data, Ollama model store, and -vLLM caches) are not touched by `down`. - -For direct Ollama model pulls, operate on the shared daemon instead of replacing -the container: +`infer-stack apply` re-renders from the ledger and recreates only services +whose definition changed. `infer-stack stack down` followed by `infer-stack +apply` is a full restart; the bind-mounted state (Open WebUI data, Ollama +model store, vLLM caches) is not touched by `stack down`. -```bash -infer-stack ollama-pull qwen3.5:4b -``` +Ollama tags are pulled into the running daemon when an endpoint that serves +them is acquired, so adding a tag never replaces the container. ## Verifying the cache is being reused -After a warm restart, a populated `state.vllm_cache` directory will contain -subdirectories keyed by configuration hash. The vLLM startup logs print +After a warm restart, `/vllm-cache` contains one `cfg-` +subdirectory per serve configuration. The vLLM startup logs print "Using cached compiled graph" (or similar; exact wording varies by version) when the cache is hit. If you see a long compile pass on every restart, check that the volume mount is actually pointing at the persistent path and diff --git a/docs/planning/backend-parity-roadmap.md b/docs/planning/backend-parity-roadmap.md new file mode 100644 index 00000000..4c85f48f --- /dev/null +++ b/docs/planning/backend-parity-roadmap.md @@ -0,0 +1,298 @@ +# Backend parity roadmap: KubeAI as a superset of Compose + +**Status:** proposed 2026-09-25 · **P0 done** 2026-09-24 on +`dev/backend-unification` · **all phases done** 2026-09-26; P4 and P5 end in +a handover run on real hardware (`dev/handover/`). Execution order: [../queue.md](../queue.md). +**Current state:** [../backend-parity.md](../backend-parity.md). +**Origin:** the scale-up run needs more than one workstation, and the +KubeAI backend had drifted from Compose for three months before the +2026-09-24 audit (`dev/tmp/plan-backend-unification-2026-09-24.md`). + +--- + +## Objective (read this first; it outlives the plan) + +A card, a catalog and an operator's habits move from one workstation to a +cluster with one change: naming what a GPU unit is. Concretely, + +1. every row of the parity matrix reads *same*, *≈*, *n/a* or *boundary*; + none reads *gap*; +2. a feature is written once. No code outside the backend modules and the + CLI's construction of them branches on the backend kind. + +Compose stays the fastest local path and loses nothing. KubeAI is Compose +plus a scheduler, with more information supplied and more tools installed; +it is not a sibling with its own habits. + +## Principles + +- **Superset, not sibling.** KubeAI gains what Compose has. Where a Compose + feature has no cluster meaning (GPU indices, `network migrate`), the + matrix says *n/a* and the CLI says why; it does not grow a second design. +- **The seam is the only place kind matters.** `isinstance(…, ComposeBackend)` + in the CLI or TUI is a smell. Each phase removes some. The Compose-only + escape hatches (`stack up`/`down` as raw compose) stay, and say so. +- **Verified on a real cluster.** A phase is done when its step in + `dev/kubeai_e2e.sh` passes on k3s, not when its fakes pass. This is how + the backend drifted last time. +- **One reviewable change per phase**, the Compose suite green throughout. + +## Phases + +### P0. One contract, one liveness view, one diagnosis — done + +The 2026-09-24 audit landed: an e2e on k3s (K0); the LiteLLM gateway +fronting both backends, so the alias and key are the same everywhere (K1); +a strict `residency()` built from pods (K2); one startup diagnosis over +containers or pods (K4); one served-name rule, one GPU count, `runtime.env` +on KubeAI (K5); the gateway as its own module (G); and unleased keep-warm +yielding to leased demand on both backends. + +### P1. One acquire path + +**Closes:** the admission and `config publish` rows. + +Today `Controller._admission_mode()` is true only for a backend with +`residency` + `preview` + `converge`, which is Compose. KubeAI takes the +pre-September branch in five places (render, acquire, `observe_state`, +`config publish`, `renew`). + +Review 2026-09-25 against the code split this in two. The admission path +assumes GPU accounting in three places (`_backfill_allocations`, +`_unresolved_allocations`, `_admission_view` keeps a LIVE deployment only +with committed GPUs), so a `preview` alone would drop every KubeAI lease from +the render. And about fifteen test fakes drive the legacy path on purpose +(queue, lock, serialised publication), so deleting it is test work, not +controller work. + +**P1a. KubeAI on the admission path.** + +- One authority for "this backend allocates GPUs": `allocates_gpus` + (Compose true, KubeAI false). The controller's accounting reads it + through one helper; on KubeAI every deployment commits an empty + allocation, so nothing is ever unresolved. +- `KubeaiBackend.preview(desired, placement, approve)`: render the Models + to memory with a digest, write nothing. Placement inputs are accepted and + ignored; the plan assigns every renderable deployment no GPUs. + `plan_on_idle_host` is the same render, so a `--queue` acquire of an + unrenderable endpoint fails at once instead of waiting out the timeout. +- `converge(..., placement=)` on KubeAI. Without it the controller's + `TypeError` fallback would render **and apply** in one step. +- The approved-digest guard (`_planned_digest`, pre-approval, + `last_planned_digest` / `last_preview_digest`) moves from `ComposeBackend` + to `ConvergeScaffold`: one approval mechanism, not a copy. +- Compose's stable addresses and container adoption stay Compose-only by + capability (`network`, `adopted`), not by joining KubeAI to them. +- `--queue` on KubeAI: admitted at once; the cluster is the queue. + +**P1a done 2026-09-25.** Verified on k3s by `dev/kubeai_e2e.sh`, which +gained the step "an unrenderable endpoint is refused before anything is +written" (a `--queue` acquire, no lease, no Model), and still passes the +make-room step, which now runs on the admission path. + +**P1b. Delete the legacy branch.** +`MemoryBackend` / `NullBackend` and the test fakes get a trivial +`residency` and `preview`; then the non-admission branches and +`_admission_mode()` go. + +**P1b done 2026-09-26.** `SimpleAdmission` (in `leasing/backend.py`) gives +a backend with only `realize` / `teardown` the admission surface; a fake +that emulates capacity overrides its `plan`, one that emulates a render +failure its `refuse`. The controller has one acquire, render, renew and +publish path. Rollback after a commit is still reachable (the runtime can +change between preview and render) and keeps its tests, on a fake whose +render refuses what its preview admitted. + +**Exit:** P1a: `_admission_mode()` is true for both real backends; a render +failure on KubeAI rolls the lease back before any `kubectl`; e2e passes. +P1b: `_admission_mode` is gone; `test_controller` runs its acquire scenarios +on all three fakes. +**Size:** P1a medium, P1b medium (mostly tests). + +### P2. Day-2 verbs and the TUI through the seam + +**Closes:** `logs` / `ps` / `stack`, the TUI's log follow, docker pane and +Up / Down, and `measure`. + +- Two backend methods: `instances()` (from `residency()`: deployment, unit + name, state, restarts, age) and `stream_logs(target, *, follow, tail)` + (`docker logs -f` / `kubectl logs -f`). +- `ps` prints `instances()`. `logs` streams. `stack up` is `apply`; `stack + down` is `backend.down()`, which both backends have. The raw compose form + moves under `stack compose …` and stays Compose-only by name. +- TUI: the docker pane becomes an instances pane; the log follower uses + `stream_logs`; Up / Down call `apply` / `down`. +- `measure`: lift the guard (`deployment_logs` already exists on KubeAI) + after checking vLLM's memory line reaches the pod log. + +**Exit:** on k3s the TUI follows a Model's log and lists its pod; `infer-stack +logs ` and `ps` work on both backends with the same output shape. +**Size:** medium, mostly plumbing; no controller change. + +**P2 done 2026-09-26.** `leasing/instances.py`: an `Instance` per container +or pod, built from residency by each backend's `instances()`, and one +`LogFollower` for the CLI and the TUI. `stack up` is `apply`, `stack down` is +`backend.down()`, and the raw Compose verbs (`stack compose …`) act on +whichever Compose project the backend has on this host, the gateway's on +KubeAI. Verified on k3s and on compose (the simulator catalog), in a real +terminal. `measure` was never refused on KubeAI (its guard was +`deployment_logs`, which KubeAI has), but it reads GPU memory-profiling lines +that CPU vLLM does not print: it moves to P4 with its GPU handover. + +### P3. Gateway feature parity + +**Closes:** `routes` on KubeAI, `dynamic_routing`, Open WebUI and the reverse +proxy in front of a cluster. + +- The `routes` commands and `secrets rotate` resolve `controller.backend.gateway` + on either backend instead of checking the backend kind. +- The KubeAI gateway project is rendered with the same `ui`, `reverse_proxy` + and `dynamic_routing` settings as Compose. Postgres for dynamic routing is + a gateway-project service, so nothing engine-side changes. + +**Exit:** `routes list` on KubeAI; Open WebUI in front of a cluster; an e2e +step that acquires the same model `--dedicated` twice under dynamic routing +and gets two Models and two routes. +**Size:** small to medium; the `Gateway` class already owns the pieces. + +**P3 done 2026-09-26.** The KubeAI gateway takes the compose settings (`ui`, +`reverse_proxy`, `dynamic_routing`) and records them in the recovery profile +as the gateway project's own profile. The kubeai backend hands the gateway +explicit render inputs (registry rows for the catalog and every Model, or +per-deployment dynamic routes) instead of writing rows into the registry +before approval, so the gateway's changes are previewed and approved with +the acquire's. Under dynamic routing each deployment is its own Model, +named with the deployment tail compose uses for its services. The `routes` +commands resolve the backend's Compose project and ask the backend for its +rows. Verified by `dev/kubeai_e2e.sh` on k3s: routes, Open WebUI, and two +`--dedicated` Models behind one alias. + +### P4. Placement information parity (optional) + +**Closes:** `min_vram_gib` / `measure`; less hand-written cluster +information. + +- `catalog suggest` on KubeAI reads the device plugin's node labels + (`nvidia.com/gpu.product`, `.memory`) and proposes `resourceProfiles` + (today the README does this by hand). +- With several profile sizes, `min_vram_gib` picks the smallest profile + whose GPU memory fits when an endpoint names none; otherwise the + warn-and-ignore stays. + +**Exit:** a catalog with `min_vram_gib` and no `resource_profile` lands on +the right profile on a cluster with two sizes. +**Size:** small. Shrinks "the information you add"; skip if nobody runs a +mixed-GPU cluster. + +**P4 done 2026-09-26** on two k3s nodes (the second a container, +`dev/k3s_agent_container.sh`) with fake GPU labels: `min_vram_gib 40` got +the 80 GiB profile and scheduled on that node, `10` got the 24 GiB one and +answered, and `catalog suggest` proposed a profile per product +(`E2E_SIZED=1 dev/kubeai_e2e.sh`). A profile's size is its selected nodes' +`nvidia.com/gpu.memory`; the chart's generic profiles have no selector, so +no size. KubeAI keeps its own measurements overlay and fills a missing +`min_vram_gib` from it, as compose does, so `measure --record` feeds the +choice. Real GPU labels, a GPU serving the chosen profile, and `measure` +reading vLLM's memory lines are `dev/handover/p4_gpu_labels.sh`. + +### P5. The multi-workstation shape + +**Closes:** the gateway on one host, the port-forward default. + +- Render the gateway in the cluster (a Deployment + Service behind an + ingress) as the second gateway target; the Compose-project gateway stays + the single-host default. `secrets rotate` becomes a Secret update and a + rollout; `doctor` checks the ingress; `kubeai_base_url` defaults to it. +- A second workstation joins with `scripts/join_agent.sh`; resource + profiles with node selectors say which GPUs live where. Write the "add a + workstation" runbook into `kubeai-backend.md`. + +**Exit:** on two nodes, a card on node A leases a model that lands on node +B and talks to it through the in-cluster gateway; `secrets rotate` works. + +**P5 done 2026-09-26** with the second node a container on the same host +(`dev/k3s_agent_container.sh`): `kubeai_gateway cluster` renders the gateway +as a Deployment and a NodePort Service (`backends/kubeai_gateway.py`), +reaching KubeAI by its Service's DNS name, so no port-forward is needed; +the key is a Secret applied from the state dir, never shown in a diff, and +its hash rolls the pods; `doctor` checks it. The recovery profile says which +placement renders, so changing the setting is adopted like any other once +the stack is quiescent. `dev/kubeai_e2e.sh` (with `E2E_SIZED=1 +E2E_REMOTE_NODE=1`) served a Model on the second node through that gateway, +rotated the key, and removed it with `stack down`. Deferred: dynamic +routing and Open WebUI in the cluster (they need Postgres and a UI there), +and an Ingress (no controller on the development cluster; `kubeai_gateway_url` +takes one). The run across two real machines is +`dev/handover/p5_two_hosts.sh`, and the runbook is "Add a workstation" in +`kubeai-backend.md`. +**Size:** medium to large. The only phase that adds a second renderer for +the gateway, and the only one that needs a second machine. Last. + +### P6. One test surface (ongoing) + +- Parametrize the controller tests over the Memory, fake-Compose and + fake-KubeAI backends; `tests/test_parity.py` runs each *same* row's + scenario on both fakes. +- `dev/kubeai_e2e.sh` gains one step per phase. +- Finishing a phase updates the matrix in `backend-parity.md`; the plan is + done when the matrix has no *gap* row. + +**P6 done 2026-09-26** as a suite that stays open: `tests/test_parity.py` +builds the same stack over a fake Docker and a fake kubectl and runs one +test per *same* row on both (21 rows, the TUI included). A row turned *same* +gets its test in the same change. The Memory and Null backends run the +controller's one path through `SimpleAdmission` (P1b), so the controller +suites cover them without a third parametrization. Found on the way: the +TUI replaced an injected Docker runner with the real one, so a TUI started +on a test backend talked to the host's Docker; it now wraps only the +default runner. + +## Duplicate authorities + +The rule while executing: a duplicate authority found on the way is +refactored when it is small, recorded here when it is not, and never made +worse. A blocker is fixed whatever its size. + +| authority | where it was | status | +|---|---|---| +| "this backend allocates GPUs" | inferred from which methods a backend has | **fixed** (P1a): `allocates_gpus()` in `leasing/backend.py`, read by the controller and the CLI | +| approval digest and pre-approval | a copy inside `ComposeBackend` | **fixed** (P1a): `ConvergeScaffold`, shared | +| KubeAI render vs its plan | `converge` rendered inline | **fixed** (P1a): one `_render_documents` behind `converge`, `preview` and `plan_on_idle_host` | +| the acquire path | admission and a legacy branch in five places | **fixed** (P1b) | +| `_render`'s one-shot `converge(desired)` fallback, which applies | a second render contract for old backends | **fixed** (P1b): removed | +| "can anything be admitted while residency is unknown" | `_admit` admitted requests needing no new GPU; the render after the commit then failed without residency | **fixed** (P1b): nothing is admitted, and nothing is committed | +| the desired set | `desired_deployments()` beside the admission view; `routes prune` used the former | **fixed** (P1b): one view, `_admission_view` | +| a crashed acquire's placement scope | recorded in the marker and re-applied by a recovery render, although admission never records one | **fixed** (P1b): a stale scope is dropped | +| "is it running" for `status` | `docker compose ps` service names beside residency (so KubeAI read `unverified`) | **fixed** (P2): residency | +| engine vs gateway in the TUI | a `litellm` name hint (Open WebUI and Postgres counted as engines) | **fixed** (P2): an instance serves a deployment or it does not | +| where the day-2 verbs find the Compose project | a hard-coded path and project name | **fixed** (P2): the backend's `compose_project()` and `compose_argv()` | +| how the TUI runs a runtime command | it replaced the backend's runner with Docker's, whose allowlisted environment has no `KUBECONFIG`: every kubectl call from the TUI failed | **fixed** (P2): only `docker` commands are wrapped | +| the KubeAI gateway's approval | the gateway project asked its own diff approval at render, after the lease committed | **fixed** (P3): previewed and approved with the acquire's | +| KubeAI's route rows | written into the registry before any approval, and only for live Models (so every new Model recreated the gateway) | **fixed** (P3): render inputs, persisted after approval; the catalog's rows too, so no blip | +| a route's upstream | derived twice, in the render and in `routes list` (which showed `?` for a cluster route) | **fixed** (P3): `registry_route_entry` | +| a deployment's Model name | derived in five places | **fixed** (P3): `model_name()` | +| "is this the compose backend" in the CLI | `isinstance(…, ComposeBackend)` for `routes`, `gc --orphans`, `clean`, `network migrate`, `secrets rotate` | **fixed** (P3): the capability each needs (`compose_project()`, residency labels, `network`, `litellm`) | + +## Not in scope + +- more than one cluster, or scheduling across a Compose host and a cluster; +- KubeAI autoscaling beyond `min_replicas` / `max_replicas`: the lease + decides residency; +- custom container launches and ollama daemons on KubeAI (a boundary in + [known-limitations.md](known-limitations.md)); +- Slurm, which is its own track ([../slurm-compatibility.md](../slurm-compatibility.md)); +- moving the Compose backend onto Kubernetes primitives. + +## Order and size + +| phase | size | needs | gives | +|---|---|---|---| +| P1a | medium | a green e2e | atomic acquire on a cluster | +| P1b | medium | P1a; with P6 | one controller path | +| P2 | medium | P1 (instances from residency) | the TUI and day-2 verbs on a cluster | +| P3 | small–medium | none | dynamic routing, Open WebUI on a cluster | +| P4 | small | a mixed-GPU cluster to verify | less hand-written cluster information | +| P5 | medium–large | a second machine | the gateway off the operator's host | + +P1 → P2 → P3 is single-host-cluster parity: k3s on one workstation, with +everything a Compose user has. P5 is the multi-workstation step. diff --git a/docs/queue.md b/docs/queue.md new file mode 100644 index 00000000..46c4313a --- /dev/null +++ b/docs/queue.md @@ -0,0 +1,299 @@ +# Work queue: backend parity + +The executable order for [planning/backend-parity-roadmap.md](planning/backend-parity-roadmap.md), +limited to what can be done and verified without a GPU or a second physical +machine: a guest VM running k3s with CPU vLLM. Items that end in a hardware +check produce a script to hand over rather than a claim of done. + +Work top to bottom. An item is done when its **Done when** holds and its +commit is pushed; mark it `[x]` with the date and commit. Update +[backend-parity.md](backend-parity.md) and the roadmap's status in the same +commit as the change they describe. + +## Rules + +- **Unforeseen work gets added here, not done silently.** When an item turns + out to need something the plan did not foresee, add it as a new item in + its place in the order, with a *Why* line naming what was found. A blocker + goes directly above the item it blocks. +- **Duplicate authorities** found on the way: refactor if small, otherwise + add to the roadmap's *Duplicate authorities* table as deferred. Never make + one worse. A blocker is fixed whatever its size. +- **Verified on k3s, not on fakes.** Every item that changes KubeAI + behaviour adds or extends a step in `dev/kubeai_e2e.sh` and passes it. +- **The queue is not done until item 9 passes.** Do not stop at the last + feature item. + +## Items + +### 1. [x] P1a: KubeAI acquires go through admission + +Done 2026-09-25, `d6a19c4`. + +### 2. [x] P1b: delete the legacy acquire branch + +Done 2026-09-26, `184289c`. + +`MemoryBackend`, `NullBackend` and the test fakes (queue, lock, serialised +publication) get a trivial `residency` and `preview`. Then the +non-admission branches, `_render`'s one-shot `converge(desired)` fallback +and `_admission_mode()` go. + +**Done when:** `_admission_mode` does not exist; the full suite passes; +queue-semantics tests still assert the same behaviour. + +### 3. [x] P2: day-2 commands and the TUI through the backend + +Done 2026-09-26, `c2d393b`. + +`instances()` and `stream_logs(target, *, follow, tail)` on both backends; +`ps`, `logs`, `stack up` / `stack down` use them, the raw compose form +moves under `stack compose …`; the TUI's docker pane, log follower and +Up / Down go through the backend; `measure` works on KubeAI if vLLM's +memory line reaches the pod log. + +**Done when:** on k3s, `infer-stack ps` and `infer-stack logs ` work +with the same output shape as on Compose, and the TUI follows a Model's log +and lists its pod (checked in a real terminal, not only in tests). + +### 4. [x] P3: gateway feature parity + +Done 2026-09-26, `7a322c1`. + +`routes` and `secrets rotate` resolve the backend's gateway instead of +checking its kind; the KubeAI gateway project honours `ui`, +`reverse_proxy` and `dynamic_routing`. Move the KubeAI gateway's approval +into the admission preview (deferred duplicate). + +**Done when:** `routes list` works on KubeAI; Open WebUI answers in front of +the cluster; an e2e step acquires the same model `--dedicated` twice under +dynamic routing and gets two Models and two routes. + +### 5. [x] P6: one test surface + +Done 2026-09-26, `1a4e4b2`. + +Parametrize the controller's acquire scenarios over the Memory, +fake-Compose and fake-KubeAI backends; `tests/test_parity.py` runs each +*same* row of the parity matrix on both fakes. + +**Done when:** every *same* row has a parity test, and the CI suite runs it. + +### 6. [x] P4: placement from node labels (verified with faked labels) + +Done 2026-09-26, `94c263d`; the GPU run is `dev/handover/p4_gpu_labels.sh`. +*Why reordered:* two GPU sizes need two nodes (a node has one +`nvidia.com/gpu.memory` label), so item 7's simulated second node +(`dev/k3s_agent_container.sh`) was built first, here. + +`catalog suggest` on KubeAI proposes `resourceProfiles` from +`nvidia.com/gpu.product` / `.memory` node labels; `min_vram_gib` picks the +smallest fitting profile when an endpoint names none. + +`measure` on KubeAI (moved here from P2): it already runs, but CPU vLLM prints +no GPU memory-profiling lines, and `--record` writes Compose's measurements +overlay, which KubeAI does not read. Decide where a cluster measurement is +recorded, and verify `measure` in the GPU handover. + +**Done when:** on k3s with hand-set labels for two GPU sizes (CPU-backed +profiles), a catalog with `min_vram_gib` and no `resource_profile` lands on +the right profile; `dev/handover/p4_gpu_labels.sh` exists for one run on a +real GPU node, and runs `measure` there. + +### 7. [x] P5: in-cluster gateway and a second node + +Done 2026-09-26, `49e70dd`; the two-machine run is `dev/handover/p5_two_hosts.sh`. + +Render the gateway as a Deployment + Service behind an ingress; `secrets +rotate` becomes a Secret update and a rollout; `doctor` checks the ingress. +Try a second k3s agent in a Docker container on the guest; write the +"add a workstation" runbook into `kubeai-backend.md`. + +**Done when:** on one node, a card reaches a Model through the in-cluster +gateway and `secrets rotate` works. If the simulated second node fits in the +VM, a Model pinned to it by node selector is served through the same +gateway. `dev/handover/p5_two_hosts.sh` exists for one run across two real +machines. + +### 8. [x] The README's Compose sections, and the other stale docs + +Done 2026-09-26, `e1c68f3`. + +About 33 references to verbs that no longer exist (`setup`, `up -d`, +`switch`, `describe-profile`, `smoke-test`, `wait-ready`, `diagnose`), and +top-level `restart` / `stop` / `start` / `pull` (they live under `stack`). +*Why widened (2026-09-26):* the same verbs appear in +`docs/persistent-caches-and-warm-restarts.md` and +`docs/stack-graph-profiles.md`. + +**Done when:** every command in the README and `docs/` runs as written +against the current CLI, and `grep` finds none of those verbs. + +### 8a. [ ] Decide: rewrite or delete the pre-leasing recipes + +*Why added (2026-09-26):* `docs/demos/`, `recipies/` and `examples/` (about +2,000 lines) are hardware recipes for the removed profile CLI; the Makefile +targets that used them are gone. Each now opens with a "Pre-leasing" banner +saying its commands no longer run. Rewriting them as catalog entries needs +the hardware they describe; deleting them loses tuning notes. **The +operator's call.** + +### 8b. [ ] Decide: `/dev/shm` for vLLM containers + +*Why added (2026-09-26):* the leasing renderer sets neither `ipc: host` nor +`shm_size`, so a vLLM container gets Docker's 64 MiB `/dev/shm`; the +pre-leasing template had `ipc: host`, and vLLM's own Docker instructions use +it for tensor parallelism. Adding it changes every vLLM service's +fingerprint, so the next apply recreates running engines. Needs a TP=2 run +on a GPU host to confirm the symptom first. **The operator's call.** + +### 9. [ ] UX audit loop: do not stop without a passing audit + +Polishing the UX can take hours; that is expected, and this item is the +reason the queue exists. Loop: + +1. Audit (below) on **both** backends, from a fresh data root. +2. Write every finding as a sub-item here, with the exact command and + output. +3. Fix them, one commit each, tests added where the behaviour is testable. +4. Repeat from 1. The audit passes when a full pass finds nothing new. + +The audit covers: + +- **First run.** Following the README and `kubeai-backend.md` literally + from nothing reaches a working request on each backend. +- **Every verb** (`acquire`, `release`, `wait`, `evict`, `gc`, `clean`, + `renew`, `run`, `test`, `status`, `leases`, `ps`, `logs`, `routes`, + `doctor`, `config`, `catalog`, `secrets`, `stack`): `--help` is accurate + and its examples run; output has the same shape on both backends. +- **Every failure.** Each error names the cause and the next command to run; + none mentions Docker or Compose on KubeAI, or `kubectl` on Compose; no + traceback reaches the user for an expected condition (port in use, no + cluster, unrenderable endpoint, refused approval). +- **The TUI**, in a real terminal at 80x24 and wide: every pane, tab, key + and button works on both backends; every action logs its CLI equivalent; + nothing is empty without saying why. +- **Consistency.** The same thing has the same name in the CLI, the TUI and + the docs. + +Seed findings, already known: + +- [x] `r` does not reload catalogs, and edits to a catalog file are not + picked up until restart. Fixed: each refresh rereads a changed catalog + (a broken save is reported once), and `r` rereads it always. Checked + live: `catalog endpoint add` in another shell shows in the TUI. +- [x] `infer-stack logs` should accept a container name, a prefixed name or + an endpoint name. Done with P2: an instance name, a container id + prefix, a deployment id or an endpoint alias. +- [x] `logs --no-color` silently kept color: kwconf reads a leading `no-` as + negation, so a flag *named* `no_color` never saw it. Now `color`, whose + `--no-color` works. Found by the README rewrite 2026-09-26. +- [x] `infer-stack logs -f qwen` followed every instance: a kwconf flag took + the next word as its value. Fixed for every command (P2). +- [x] `status` shows STALE during an apply instead of "apply in progress". + Fixed: `pending` while the publication marker says the change is not + applied yet; STALE only when nothing is pending. +- [x] Open WebUI writes its data directory as root: removing a data root + (the e2e's cleanup, or an operator's `rm -rf`) fails with permission + denied. Found by `dev/kubeai_e2e.sh` 2026-09-26; fixed: it runs as the + directory's owner (a directory root already wrote keeps root, and the + log says how to `chown` it). Engine caches (vLLM as root) are the same + shape; not yet looked at. +- [x] A refused acquire always ends with "free a GPU first — …", also when + the reason is a render refusal (a served-name collision) or the + backend is kubeai, where no GPU is ours to free. Found 2026-09-26; + fixed: `PlacementError.capacity` says whether room would have helped. + +Audit pass 1 (2026-09-26): every command's `--help` collected +(`infer-stack help tree`, 65 leaves) and every example in them run on a +dry-run root. + +- [x] Five one-line summaries ended mid-sentence in `help tree` (`measure`, + `routes prune`, `status`, `tui`, `wait`); `leases` said "deployment + deployments". +- [x] `--litellm`, `--ui` and `--yes` said "(compose backend)", and + `acquire`, `apply`, `--apply` and `render` described only a compose + project; `render` on kubeai printed "(backend has no on-disk project)". + `apply`'s help still called `stack up` the raw hatch. +- [x] Every `.env` write printed `Write .env to …` on stdout, into `--json` + output too. +- [x] The render step logged "(not applied; `infer-stack apply` …)" in + every acquire, right before the apply it said had not happened. +- [x] A runtime refusal (the gateway's port taken, a daemon down) ended in + a Python traceback of `CalledProcessError`. Now the command, docker's + own last lines, and a hint (a port in use names the port); docker's + stderr is still shown live, through a pipe that keeps its tail. +- [x] `evict ` printed "no idle deployment for" then "nothing to + evict"; it now says whether each name is held by a lease or unknown. + `test` on a stopped gateway printed a urllib3 dump; now "nothing is + listening there". `logs` said "running: nothing". +- [x] On kubeai with no cluster, `acquire` refused with "the runtime did + not answer" and no cause; `ps` and `leases` printed kubectl's klog + retries (hundreds of characters). Now kubectl's own last sentence + ("The connection to the server … was refused"), in the refusal too, + with `infer-stack doctor` as the next step. +- [x] TUI at 80x24: opening the runtime pane took every row (the lease and + deployment tables vanished), and each log line spent ~20 of its ~33 + columns on the instance-name prefix. The pane is now capped near half + the screen (three log lines at 24 rows), pane descriptions hide below + 32 rows, and one followed instance gets no prefix. +- [x] The TUI's API tab showed the gateway's master key in clear text in + its example curl. The pane now reads it at run time + (`$(infer-stack env LITELLM_MASTER_KEY)`); Copy curl copies the + literal key. The UI tab said "docker observe interval". +- [x] TUI: a refused runtime command showed as a `CalledProcessError` + repr full of paths (same fix as the CLI's), and colored engine output + (vLLM's `(APIServer pid=1)`) was garbled in the logs pane. Checked on + kubeai at 80x24: acquire, the pod's log, release-all, from the TUI. +- [x] First run, README literally, on a host with no GPU: `catalog suggest` + found nothing and offered only simulated hardware (endpoints that + cannot run here), so step 3 had nothing to acquire; `config init` + pointed at `catalog init`, not the README's next step. Now `catalog + suggest --simulator` adds a simulator endpoint (named in suggest's + message when there is no GPU, and in the README), and `config init` + points at `suggest`. Checked: init, suggest --simulator --apply, + acquire, test, release, from nothing. +- [x] Consistency: the TUI's "Clean up" (`x`) runs `gc --forget`, which + only forgets finished rows, while `infer-stack clean` releases and + tears everything down. It is "Clear finished" now. The KubeAI setup + example's resource profile had no GPU request or runtime class, + which the README warns makes a pod start without CUDA. + +Audit pass 2 (2026-09-26): the day-2 commands swept on both backends with a +live model each (`leases status ps env doctor clean gc renew routes wait +test logs`), plus pass 1 again. + +- [x] `infer-stack test` failed on kubeai with HTTP 401, and `env + LITELLM_MASTER_KEY` read nothing there (and so did the TUI's curl): + both read a hard-coded compose `.env`. They now ask the configured + backend's gateway for its `.env` and base URL (the in-cluster + gateway's NodePort included). +- [x] `leases` showed a lease's TTL as `ttl=@1790450980`; now `ttl=1h59m` + (JSON keeps the timestamp). `clean`'s dry run said `gpus=[]` where + `leases` says `cpu`, and converge logged "on GPU(s) (cpu)". +- [x] `logs` into a pipe or file kept the engines' color codes; with color + off they are stripped. + +Audit pass 3 (2026-09-26): `dev/ux_audit.sh compose` and `kubeai` (help, +mistakes, the day-2 sweep; 0 flags on both), and the reports read by eye. + +- [x] Every kubeai acquire logged "Converging 0 deployment(s): (none)" right + after "Converging 1 deployment(s) onto kubeai": the gateway-only + project narrating itself. It says "Converging the gateway project". +- [x] `doctor` on kubeai called KubeAI's own API "gateway", beside + infer-stack's LiteLLM gateway; it is "KubeAI's API" now, here and in + the docs. + +Audit pass 4 (2026-09-26): `dev/ux_audit.sh` on both backends (0 flags; +the two reports differ only in backend facts), the reports read by eye. + +- [x] `doctor` on kubeai did not check Docker, though the gateway on this + host is a Compose project: a stopped daemon passed preflight and + failed the first acquire. The gateway project's checks (Docker, + compose, its images; no engine image or GPU check) now follow + KubeAI's. + +### 10. [ ] Handover + +Summarize for the operator: what was verified here, and the two handover +scripts (items 6 and 7) with what each run proves. diff --git a/docs/source/conf.py b/docs/source/conf.py index d28cf6dc..0818d132 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -235,7 +235,7 @@ def visit_Assign(self, node): 'ubelt': ('https://ubelt.readthedocs.io/en/latest/', None), 'xdoctest': ('https://xdoctest.readthedocs.io/en/latest/', None), 'networkx': ('https://networkx.org/documentation/stable/', None), - 'scriptconfig': ('https://scriptconfig.readthedocs.io/en/latest/', None), + 'kwconf': ('https://kwconf.readthedocs.io/en/latest/', None), 'kwconf': ('https://kwconf.readthedocs.io/en/latest/', None), 'rich': ('https://rich.readthedocs.io/en/latest/', None), 'numpy': ('https://numpy.org/doc/stable/', None), diff --git a/docs/source/manual/leasing-demo.md b/docs/source/manual/leasing-demo.md index c1799324..436bc185 100644 --- a/docs/source/manual/leasing-demo.md +++ b/docs/source/manual/leasing-demo.md @@ -168,15 +168,18 @@ something is still warming up vs truly live. > **Prefer a live dashboard?** `infer-stack tui` opens a multi-pane Textual UI > (run `infer-stack config init` once first): > a **catalog** pane to pick an endpoint and serve it (`s`/Enter), live -> **leases** + **deployments** tables, and three collapsible panes below: **docker** -> (a `logs -f` tail + a **Containers** `docker ps` view: status/uptime · created -> · container id · ports), **system** (`nvidia-smi` GPUs + host CPU/mem), and -> **api** (send a prompt to a *ready* model — only running models are listed). -> Each pane carries its own description and buttons; expanding a pane (or its -> tab) is what triggers its polling, so hidden data is never fetched. +> **leases** + **deployments** tables, and two collapsible panes below: +> **runtime** (`c`; the engines' logs, an **Instances** view of the +> containers or pods and what each serves, and **Apply** / **Down**, on +> either backend) and **system** (`nvidia-smi` GPUs + host CPU/mem). The +> **API** tab sends a prompt to a *ready* model (only running models are +> listed). Each pane carries its own description and buttons; expanding a +> pane (or its tab) is what triggers its polling, so hidden data is never +> fetched. > New here? The catalog buttons (or `g`) **Suggest** a set sized to your GPUs, -> and `m` / `n` open wizards to add a model / endpoint by hand. **Clean up** -> (`x`) forgets released/stopped entries. **Ctrl+click** a served endpoint +> and `m` / `n` open wizards to add a model / endpoint by hand. **Clear finished** +> (`x`) forgets released/stopped entries (`gc --forget`; unlike `infer-stack +> clean`, it stops nothing). **Ctrl+click** a served endpoint > (or `o`) opens it in Open WebUI. Resize panes by dragging the splitter bars > (or `[` `]` / `-` `+`). Opt-in extra: `pip install "infer-stack[tui]"`. diff --git a/docs/stack-graph-profiles.md b/docs/stack-graph-profiles.md index 4880e87f..333e47fd 100644 --- a/docs/stack-graph-profiles.md +++ b/docs/stack-graph-profiles.md @@ -1,276 +1,20 @@ -# Stack graph profiles - -Profiles describe an LLM deployment as a small graph rather than as a single -vLLM-shaped service list. - -The four graph sections are: - -- `providers`: inference servers that can answer model requests, currently - `vllm` and `ollama`. -- `gateways`: optional API routers/proxies, currently `litellm`. -- `frontends`: optional user-facing UIs, currently `open_webui`. -- `routes`: optional public model aliases exposed through a gateway. - -This separation keeps the simple cases simple: - -- Ollama can run as one direct daemon with no predeclared model list. -- vLLM can still run one or more explicit model runtimes. -- LiteLLM is only needed when you want a unified `/v1` model namespace. -- Open WebUI can point either at LiteLLM or directly at Ollama. - -## Mental model - -```text -providers -> raw inference endpoints -routes -> optional public model names -gateways -> optional route surface, usually LiteLLM -frontends -> optional UI, usually Open WebUI -``` - -A profile may have providers without routes. For example, `ollama-direct` -starts Ollama and Open WebUI directly, and you pull models with the Ollama CLI -or from Open WebUI. A mixed profile with both Ollama and vLLM usually enables -LiteLLM so clients have one stable OpenAI-compatible endpoint. - -## Direct Ollama - -`ollama-direct` starts one Ollama daemon and Open WebUI connected directly to -it. It does not render LiteLLM, does not render `postgres-litellm`, and does -not require model declarations. - -```bash -infer-stack setup --backend compose --profile ollama-direct -infer-stack render --yes --simulate-hardware 2x11 -infer-stack up -d - -infer-stack ollama-pull qwen3.5:4b -``` - -Rendered shape: - -```text -Open WebUI -> Ollama -``` - -The profile owns daemon settings such as `keep_alive`, `context_length`, and -`max_loaded_models`; the model store itself lives in `state.ollama` and is -mounted at `/root/.ollama`. - -## Ollama through LiteLLM - -Use this when you want a stable OpenAI-compatible route name that can later be -moved from Ollama to vLLM without changing clients. - -```yaml -profiles: - ollama-qwen3.5-4b-gateway: - providers: - ollama: - enabled: true - gpu_indices: [0, 1] - keep_alive: 2m - context_length: 4096 - gateways: - litellm: - enabled: true - frontends: - open_webui: - enabled: true - provider: litellm - routes: - home-assistant-local: - provider: ollama - model: qwen3.5:4b -``` - -LiteLLM renders that route as `ollama_chat/qwen3.5:4b` with -`api_base: http://ollama:11434`. - -Rendered shape: - -```text -Open WebUI -> LiteLLM -> Ollama -``` - -Routes may reference an entry in `ollama_models`, or a raw Ollama tag such as -`qwen3.5:4b` directly. - -## vLLM through LiteLLM - -vLLM runtimes are explicit because each runtime starts a model-serving process. - -```yaml -vllm_models: - smollm2-135m-instruct: - hf_model_id: HuggingFaceTB/SmolLM2-135M-Instruct - served_model_name: smollm2-135m - supported_protocols: [chat] - min_vram_gib_per_replica: 4 - preferred_gpu_count: 1 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.5 - -profiles: - smollm-vllm-compose: - providers: - vllm: - runtimes: - chat: - model: smollm2-135m-instruct - placement: - strategy: first_fit - gpu_count: 1 - gateways: - litellm: - enabled: true - frontends: - open_webui: - enabled: true - provider: litellm - routes: - smollm2: - provider: vllm - runtime: chat -``` - -Rendered shape: - -```text -Open WebUI -> LiteLLM -> vLLM runtime -``` - -## Mixed Ollama + vLLM - -`mixed-ollama-smollm` demonstrates one shared Ollama daemon plus one vLLM -runtime behind LiteLLM. - -```yaml -profiles: - mixed-local: - providers: - ollama: - enabled: true - gpu_indices: [0, 1] - vllm: - runtimes: - smollm: - model: smollm2-135m-instruct - placement: - strategy: first_fit - gpu_count: 1 - gateways: - litellm: - enabled: true - frontends: - open_webui: - enabled: true - provider: litellm - routes: - home-assistant-local: - provider: ollama - model: qwen3.5:4b - smollm2-135m: - provider: vllm - runtime: smollm -``` - -Rendered shape: - -```text - -> Ollama -Open WebUI -> LiteLLM - -> vLLM -``` - -Mixed routes need a gateway for one unified client namespace. Without LiteLLM, -Ollama and vLLM can still run as raw servers, but clients must address them -separately. - -## Raw servers - -`raw-ollama-vllm` starts backend servers without Open WebUI or LiteLLM. This is -useful for debugging direct provider endpoints. - -```yaml -profiles: - raw-ollama-vllm: - providers: - ollama: - enabled: true - publish_port: true - vllm: - runtimes: - smollm: - model: smollm2-135m-instruct - publish_port: true - gateways: - litellm: - enabled: false - frontends: - open_webui: - enabled: false - routes: {} -``` - -Rendered shape: - -```text -Ollama API directly -vLLM API directly -``` - -## Backend support - -Compose supports Ollama, vLLM, LiteLLM, and Open WebUI in valid combinations. -KubeAI currently supports only vLLM runtimes; profiles that enable Ollama, -LiteLLM, or Open WebUI are rejected for `--backend kubeai`. - -## Configuration files - -Custom provider models and stack profiles can come from the configured -`catalog.user_models_file`, which defaults to `~/.config/infer_stack/models.yaml`, -from additional paths in `catalog.model_path` / `catalog.model_paths`, or from -`INFER_STACK_MODEL_PATH` for per-shell overlays. Use provider-specific top-level -keys: - -```yaml -vllm_models: - my-vllm-model: - hf_model_id: org/model - -ollama_models: - my-ollama-model: - tag: qwen3.5:4b - -profiles: - my-stack: - providers: {} - gateways: {} - frontends: {} - routes: {} -``` - -`models:` is still interpreted as a vLLM model catalog for convenience, but new -examples should use `vllm_models:` and `ollama_models:`. - -`INFER_STACK_MODEL_PATH` behaves like a PATH variable: entries are separated by -`:` on POSIX systems and by `;` on Windows. Each entry can be a YAML file or a -directory. Directory entries are scanned non-recursively for `models.yaml` / `models.yml`, -`*.models.yaml` / `*.models.yml`, and `*.profiles.yaml` / `*.profiles.yml`. Later files override earlier files, so a -repo-local experiment overlay can temporarily override or extend the normal user -catalog without editing `config.yaml`: - -```bash -export INFER_STACK_MODEL_PATH="${INFER_STACK_MODEL_PATH:+$INFER_STACK_MODEL_PATH:}$PWD/infer_stack_profiles" -infer-stack describe-profile my-experiment-profile -``` - -For persistent extra catalogs, put the same directory or file path in -`config.yaml`: - -```yaml -catalog: - model_path: - - /srv/experiments/infer_stack_profiles -``` +# Stack graph profiles (removed) + +Named stack profiles (`providers` / `gateways` / `frontends` / `routes`, +selected with `--profile` or `active_profile`, applied with `setup`, `switch` +and `up`) no longer exist. The leasing catalog replaced them: `catalog.yaml` +declares `models`, `endpoints`, `runtime_hosts` and `bundles`, and +`infer-stack acquire ` leases an endpoint and brings up its engine +behind the LiteLLM gateway and Open WebUI. Gateway and UI are settings +(`infer-stack config set litellm|ui|reverse_proxy …`), not graph nodes. + +Start with the README's "Primary leasing workflow" and "Catalog model" +sections, then [the user manual](source/manual/index.md) and +[litellm-gateway-routing.md](litellm-gateway-routing.md). + +"Profile" survives in one place: the ledger's **recovery snapshot**, the +frozen copy of every non-ledger render input (settings, image pins, the +published catalog union) that recovery re-renders from. `acquire` advances it +automatically; `infer-stack config publish` pre-seeds or previews it +explicitly. See [ADR 0001](adr/0001-user-config-is-authoritative.md) and the +docstring of `infer_stack/leasing/profile.py`. diff --git a/examples/config.multi-model.yaml b/examples/config.multi-model.yaml index 512ce8f7..da0c81e3 100644 --- a/examples/config.multi-model.yaml +++ b/examples/config.multi-model.yaml @@ -1,3 +1,5 @@ +# Pre-leasing: a stack-profile config for the removed profile CLI; it is +# not read by infer-stack any more. See the README for the catalog. name: aiq-multi-model backend: compose active_profile: qwen-mixed diff --git a/examples/openwebui-tls-ldap/README.md b/examples/openwebui-tls-ldap/README.md index 39404028..753f329c 100644 --- a/examples/openwebui-tls-ldap/README.md +++ b/examples/openwebui-tls-ldap/README.md @@ -1,5 +1,7 @@ # Open WebUI with TLS reverse proxy and LDAP +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../../README.md). + This example runs the built-in `openwebui-tls-ldap` profile: an opt-in Compose stack with diff --git a/examples/single-node/README.md b/examples/single-node/README.md index 160b6527..feb685ef 100644 --- a/examples/single-node/README.md +++ b/examples/single-node/README.md @@ -1,5 +1,7 @@ # Single-node KubeAI example +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../../README.md). + This example uses the stack-graph schema, but KubeAI currently renders only the `providers.vllm.runtimes` section. Ollama, LiteLLM, and Open WebUI are disabled in this example because they are Compose-only for now. diff --git a/examples/single-node/config.yaml b/examples/single-node/config.yaml index 09a54427..93bac038 100644 --- a/examples/single-node/config.yaml +++ b/examples/single-node/config.yaml @@ -1,3 +1,5 @@ +# Pre-leasing: a stack-profile config for the removed profile CLI; it is +# not read by infer-stack any more. See the README for the catalog. name: single-node-kubeai backend: kubeai active_profile: qwen-single-node diff --git a/examples/single-node/models.yaml b/examples/single-node/models.yaml index c233e6a9..7ec5f4b9 100644 --- a/examples/single-node/models.yaml +++ b/examples/single-node/models.yaml @@ -1,3 +1,5 @@ +# Pre-leasing: a stack-profile config for the removed profile CLI; it is +# not read by infer-stack any more. See the README for the catalog. vllm_models: qwen-small: hf_model_id: Qwen/Qwen2.5-7B-Instruct diff --git a/infer_stack/backends/__init__.py b/infer_stack/backends/__init__.py index 065b27bc..aabed088 100644 --- a/infer_stack/backends/__init__.py +++ b/infer_stack/backends/__init__.py @@ -1,4 +1,3 @@ from .kubeai import KubeaiBackend -from .kubeai_renderer import render_kubeai_artifacts # legacy profile-era path -__all__ = ['KubeaiBackend', 'render_kubeai_artifacts'] +__all__ = ['KubeaiBackend'] diff --git a/infer_stack/backends/kubeai.py b/infer_stack/backends/kubeai.py index 88e5c348..53cd481a 100644 --- a/infer_stack/backends/kubeai.py +++ b/infer_stack/backends/kubeai.py @@ -53,28 +53,45 @@ DEFAULT_BASE_URL = 'http://127.0.0.1:8000/openai/v1' -def model_name_for(served: str) -> str: +def model_name_for(served: str, deployment_id: str | None = None) -> str: """Deterministic Model CR name for a served model name: ````. Derived purely from the served name (like the compose service name), so it is identical across releases/re-acquires of the same endpoint — the request - name clients use through the KubeAI gateway never changes. + name clients use through the KubeAI gateway never changes. With + ``deployment_id`` (dynamic routing), the deployment's tail is appended, as + for a compose service, so same-model ``--dedicated`` deployments are + separate Models; the gateway addresses each by name. + + >>> model_name_for('Qwen/Qwen3-8B') + 'qwen-qwen3-8b' + >>> model_name_for('Qwen/Qwen3-8B', 'grp-0123456789ab') + 'qwen-qwen3-8b-01234567' """ - return dns_slug(served) + if deployment_id is None: + return dns_slug(served) + from ..leasing.naming import deployment_tail + + tail = deployment_tail(deployment_id) + return f'{dns_slug(served)[:62 - len(tail)].rstrip("-")}-{tail}' + + +def model_name(deployment: Deployment, *, unique: bool = False) -> str: + """The Model CR name for ``deployment``: the one place it is derived.""" + return model_name_for(_served_name(deployment), deployment.id if unique else None) def _served_name(deployment: Deployment) -> str: - return deployment.spec.get('served_model_name') or ( - sorted(deployment.served)[0] if deployment.served else deployment.id - ) + from ..leasing.models import served_name + + return served_name(deployment) def _gpu_count(deployment: Deployment) -> int: - runtime = deployment.spec.get('runtime', {}) or {} - tp = int(runtime.get('tensor_parallel_size', 1) or 1) - pp = int(runtime.get('pipeline_parallel_size', 1) or 1) - dp = int(runtime.get('data_parallel_size', 1) or 1) - return max(1, tp * pp * dp) + """Resource-profile units for a Model: the planner's GPU count, at least 1.""" + from ..leasing.placement import required_gpu_count + + return max(1, required_gpu_count(deployment)) def _model_doc( @@ -82,6 +99,7 @@ def _model_doc( *, namespace: str, resource_profile: str, + name: str | None = None, ) -> dict[str, Any]: """Build one KubeAI ``Model`` CR for a vLLM deployment. @@ -91,7 +109,7 @@ def _model_doc( engine here too. ``served_model_name`` is overridden to the CR name so the gateway's request name and vLLM's served name agree. """ - name = model_name_for(_served_name(deployment)) + name = name or model_name(deployment) svc = vllm_service_dict(deployment) svc['served_model_name'] = name profile = resource_profile @@ -125,13 +143,116 @@ def _model_doc( }, 'spec': spec, } - # Attention backend is a vLLM env var, not a CLI arg (see compose._vllm_service); - # forward it through the KubeAI Model's env map for parity across backends. + # Container environment through the Model's env map, as compose renders + # it: runtime.env (templates filled, values as strings), plus the attention + # backend, which is a vLLM env var rather than a flag. + from ..leasing.launch import env_string, fill + + env = {str(k): fill(env_string(v), svc) for k, v in svc['env'].items()} if svc.get('attention_backend'): - spec['env'] = {'VLLM_ATTENTION_BACKEND': str(svc['attention_backend'])} + env['VLLM_ATTENTION_BACKEND'] = str(svc['attention_backend']) + if env: + spec['env'] = env return doc +#: GPU Feature Discovery's node labels: the GPU model, and per-GPU memory (MiB). +GPU_PRODUCT_LABEL = 'nvidia.com/gpu.product' +GPU_MEMORY_LABEL = 'nvidia.com/gpu.memory' +GPU_RESOURCE = 'nvidia.com/gpu' + + +def node_gpus(nodes: list[dict[str, Any]]) -> dict[str, dict[str, Any]]: + """Each node's GPUs, from its labels and allocatable: ``{node: facts}``. + + ``facts`` is ``{'product', 'memory_gib', 'count', 'labels'}``; a node with + no memory label has ``memory_gib`` None. + + >>> n = {'metadata': {'name': 'a', 'labels': {GPU_PRODUCT_LABEL: 'L4', + ... GPU_MEMORY_LABEL: '23034'}}, + ... 'status': {'allocatable': {GPU_RESOURCE: '2'}}} + >>> node_gpus([n])['a']['memory_gib'], node_gpus([n])['a']['count'] + (22.49, 2) + """ + out = {} + for node in nodes: + meta = node.get('metadata') or {} + labels = dict(meta.get('labels') or {}) + memory = labels.get(GPU_MEMORY_LABEL) + try: + memory_gib = round(int(memory) / 1024, 2) if memory else None + except ValueError: + memory_gib = None + count = str(((node.get('status') or {}).get('allocatable') or {}).get(GPU_RESOURCE) or '') + out[str(meta.get('name'))] = { + 'product': labels.get(GPU_PRODUCT_LABEL), + 'memory_gib': memory_gib, + 'count': int(count) if count.isdigit() else 0, + 'labels': labels, + } + return out + + +def sized_profiles(profiles: dict[str, Any], nodes: dict[str, dict[str, Any]]) -> dict[str, float]: + """Per-GPU memory (GiB) of each resource profile that says where it runs. + + A profile's size is the smallest ``nvidia.com/gpu.memory`` among the nodes + its ``nodeSelector`` selects. A profile without a ``nodeSelector`` (the + chart's generic ones) could land anywhere, so it has no size and is never + picked by size. + + >>> nodes = {'a': {'memory_gib': 24.0, 'labels': {'gpu': 'small'}}, + ... 'b': {'memory_gib': 80.0, 'labels': {'gpu': 'big'}}} + >>> sized_profiles({'small': {'nodeSelector': {'gpu': 'small'}}, + ... 'big': {'nodeSelector': {'gpu': 'big'}}, + ... 'anywhere': {}}, nodes) + {'big': 80.0, 'small': 24.0} + """ + sizes = {} + for name, profile in sorted((profiles or {}).items()): + selector = (profile or {}).get('nodeSelector') or {} + if not selector: + continue + found = [facts['memory_gib'] for facts in nodes.values() + if facts.get('memory_gib') + and all(facts['labels'].get(k) == str(v) for k, v in selector.items())] + if found: + sizes[name] = min(found) + return sizes + + +def pick_profile(explicit: str | None, min_vram_gib: float | None, + sized: dict[str, float], default: str | None) -> tuple[str | None, str]: + """``(profile, why)`` for one deployment. + + An endpoint's own ``resource_profile`` wins. Else, with ``min_vram_gib``, + the smallest sized profile whose GPUs have that much memory. Else the + ``kubeai_resource_profile`` default. ``why`` says which rule chose, or + why nothing could. + + >>> sized = {'small': 24.0, 'big': 80.0} + >>> pick_profile(None, 40, sized, 'cpu') + ('big', 'min_vram_gib 40 GiB -> big (80 GiB GPUs)') + >>> pick_profile(None, 10, sized, None)[0], pick_profile('mine', 40, sized, None)[0] + ('small', 'mine') + >>> pick_profile(None, 100, sized, None) + (None, 'min_vram_gib 100 GiB: no resource profile has GPUs that large (big 80 GiB, small 24 GiB)') + """ + if explicit: + return explicit, 'runtime.resource_profile' + if min_vram_gib: + fitting = sorted((size, name) for name, size in sized.items() + if size >= float(min_vram_gib)) + if fitting: + size, name = fitting[0] + return name, f'min_vram_gib {min_vram_gib:g} GiB -> {name} ({size:g} GiB GPUs)' + if not default: + known = ', '.join(f'{n} {g:g} GiB' for n, g in sorted(sized.items())) or 'none sized' + return None, (f'min_vram_gib {min_vram_gib:g} GiB: no resource profile has ' + f'GPUs that large ({known})') + return default, 'kubeai_resource_profile' + + class RenderedModels: """Output of the render half: manifest text + bookkeeping maps.""" @@ -141,6 +262,8 @@ def __init__(self) -> None: self.request_names: dict[str, str] = {} # endpoint -> CR name self.unrenderable: set[str] = set() self.errors: list[str] = [] + #: deployment id -> which rule chose its resource profile. + self.profile_reasons: dict[str, str] = {} @property def text(self) -> str: @@ -154,6 +277,8 @@ def render_models( *, namespace: str, default_resource_profile: str | None, + unique_names: bool = False, + sized: dict[str, float] | None = None, ) -> RenderedModels: """Render the desired set into KubeAI ``Model`` docs (pure, no I/O). @@ -178,11 +303,11 @@ def render_models( from ..leasing.launch import translate_legacy runtime = translate_legacy(deployment.spec.get('runtime', {}) or {}) - custom = [k for k in ('command', 'env', 'mounts') if runtime.get(k)] + custom = [k for k in ('command', 'mounts') if runtime.get(k)] if custom: - # A KubeAI Model runs stock vLLM; it has no place for a container - # command, environment or host mounts. Fail closed, never silently - # serve the stock engine instead. + # A KubeAI Model runs stock vLLM: it has no place for a container + # command or host mounts (env maps onto spec.env). Fail closed, + # never silently serve the stock engine instead. out.unrenderable.add(deployment.id) out.errors.append( f"{deployment.id}: runtime.{custom[0]} describes a custom container " @@ -190,10 +315,10 @@ def render_models( 'VLLM Model does not. Use --backend compose for this endpoint.' ) continue - profile = ( - runtime.get('resource_profile') or default_resource_profile or '' - ) - if not str(profile).strip(): + min_vram = (deployment.spec.get('placement') or {}).get('min_vram_gib') + profile, why = pick_profile(runtime.get('resource_profile'), min_vram, + sized or {}, default_resource_profile) + if not str(profile or '').strip(): out.unrenderable.add(deployment.id) out.errors.append( f'{deployment.id}: no resource profile — set ' @@ -201,9 +326,11 @@ def render_models( 'resourceProfiles key from your KubeAI helm values, e.g. ' "'nvidia-gpu-rtx-4090') or `config set " 'kubeai_resource_profile ` as the default.' + + (f' ({why})' if min_vram else '') ) continue - name = model_name_for(_served_name(deployment)) + out.profile_reasons[deployment.id] = why + name = model_name(deployment, unique=unique_names) if name in out.models: out.unrenderable.add(deployment.id) out.errors.append( @@ -217,6 +344,7 @@ def render_models( deployment, namespace=namespace, resource_profile=str(profile), + name=name, ) ) out.models[name] = deployment.id @@ -238,13 +366,36 @@ def _default_kubectl_run(args: list[str]) -> str: args, capture_output=True, text=True, timeout=120, ) if proc.returncode != 0: - detail = (proc.stderr or proc.stdout or '').strip() + detail = kubectl_complaint(proc.stderr or proc.stdout or '') raise RuntimeError( f'{" ".join(args)} failed ({proc.returncode}): {detail[:500]}' ) return proc.stdout +def kubectl_complaint(stderr: str) -> str: + r"""kubectl's own complaint, without the client library's log lines. + + kubectl prefixes its retries with klog lines (``E0926 14:05:52 ... + memcache.go:265] "Unhandled Error" err="..."``) and ends with the sentence + that says what is wrong; that sentence is the complaint. + + >>> kubectl_complaint('E0926 14:05:52.449219 841241 memcache.go:265] "Unhandled ' + ... 'Error" err="couldn\'t get current server API group list"\n' + ... 'The connection to the server localhost:8080 was refused - ' + ... 'did you specify the right host or port?\n') + 'The connection to the server localhost:8080 was refused - did you specify the right host or port?' + """ + import re + + lines = [ln.strip() for ln in str(stderr).splitlines() if ln.strip()] + plain = [ln for ln in lines if not re.match(r'^[EWIF]\d{4} \d\d:\d\d:\d\d', ln)] + if plain: + return plain[-1] + return lines[-1] if lines else '' + + + class KubeaiBackend(ConvergeScaffold): """Cluster KubeAI backend (converge-style). @@ -268,6 +419,9 @@ def __init__( run: Callable[[list[str]], str] | None = None, http: Any = None, assume_yes: bool = True, + gateway: Any = None, + gateway_upstream: str | None = None, + gateway_factory: Callable[[str], Any] | None = None, ): self.state_dir = Path(state_dir) self.state_dir.mkdir(parents=True, exist_ok=True) @@ -285,6 +439,26 @@ def __init__( self.last_unplaced: set[str] = set() # KubeAI/k8s schedules; there are no host GPU indices to report. self.last_assignments: dict[str, list[int]] = {} + # The LiteLLM gateway in front of the cluster: a gateway-only + # ComposeBackend. With it, clients use the same front door, key and + # request names (the endpoint aliases) as on the compose backend. + # Without it, clients talk to KubeAI directly under its Model names. + self.gateway = gateway + if gateway is not None: + gateway.fronts_elsewhere = True + # Builds the gateway for a placement ('host' or 'cluster'), so a + # recovery profile can say where it runs (see use_profile). + self.gateway_factory = gateway_factory + # How that gateway reaches KubeAI. Unset: the KubeAI Service's cluster + # IP, which containers on a cluster node can reach; a gateway off the + # cluster needs an ingress URL here. + self.gateway_upstream = gateway_upstream + # `infer-stack measure --record` writes here, as on compose; a + # measurement fills an endpoint's missing min_vram_gib, which picks + # its resource profile by size. + from ..leasing.vram import Measurements + + self.measurements = Measurements(self.state_dir / 'measurements.json') # -- state-dir plumbing -------------------------------------------------- @@ -292,6 +466,18 @@ def __init__( def models_file(self) -> Path: return self.state_dir / MODELS_FILENAME + #: The file a render writes, whatever the backend (``status`` shows it). + rendered_file = models_file + + def compose_project(self): + """The Compose project on this host: the gateway's, or ``None`` + (no gateway, or the gateway runs in the cluster).""" + return self.gateway.compose_project() if self.gateway is not None else None + + def front_door(self): + """What holds the gateway's keys and route registry: the gateway.""" + return self.gateway + @property def _state_file(self) -> Path: return self.state_dir / STATE_FILENAME @@ -327,10 +513,257 @@ def _cluster_models(self) -> dict[str, str | None]: models[name] = (meta.get('labels') or {}).get(DEPLOYMENT_LABEL) return models + # -- the gateway ----------------------------------------------------------- + + @property + def litellm(self) -> bool: + return self.gateway is not None + + @property + def litellm_port(self) -> int | None: + return self.gateway.litellm_port if self.gateway is not None else None + + def master_key(self) -> str: + return self.gateway.master_key() + + def rotate_master_key(self) -> dict[str, str]: + return self.gateway.rotate_master_key() + + def restore_env(self, values) -> None: + self.gateway.restore_env(values) + + def gateway_accepts(self, key: str, *, wait: float = 0.0): + return self.gateway.gateway_accepts(key, wait=wait) + + def _upstream_url(self) -> str: + """Where the gateway sends requests for this cluster's Models.""" + if not self.gateway_upstream and getattr(self.gateway, 'in_cluster', False): + # Inside the cluster: the Service's DNS name, stable across + # reinstalls, unlike its cluster IP. + return f'http://kubeai.{self.namespace}.svc.cluster.local/openai/v1' + if not self.gateway_upstream: + ip = self._kubectl(['get', 'service', 'kubeai', '-o', + 'jsonpath={.spec.clusterIP}']).strip() + if not ip: + raise RuntimeError('the kubeai Service has no cluster IP; set ' + '`config set kubeai_gateway_upstream `') + self.gateway_upstream = f'http://{ip}/openai/v1' + return self.gateway_upstream.rstrip('/') + + @property + def dynamic_routing(self) -> bool: + """The gateway manages routes live, so each deployment is its own Model.""" + return bool(self.gateway is not None and getattr(self.gateway, 'dynamic_routing', False)) + + def catalog_route_rows(self, catalog) -> dict[str, dict[str, Any]]: + """Registry rows sending each vLLM catalog endpoint to its Model. + + The static superset, as on compose: the gateway's config then stays + byte-stable as models come and go, so it is never recreated for one. + """ + from ..leasing.gateway import UPSTREAM_ROUTE, _registry_incoming_from_catalog + + if catalog is None or self.gateway is None: + return {} + base = self._upstream_url() + return { + alias: {'engine': UPSTREAM_ROUTE, + 'served': model_name_for(row.get('served') or alias), + 'api_base': base} + for alias, row in _registry_incoming_from_catalog(catalog).items() + if row.get('engine') == 'vllm' + } + + def route_rows(self, desired: list[Deployment], placement=None) -> dict[str, dict[str, Any]]: + """The registry rows a render of ``desired`` merges (``routes prune`` + keeps exactly these): the catalog's, and every rendered Model's.""" + return self._front_door_inputs(desired, self._render_documents(desired)[1])[0] + + def _front_door_inputs(self, desired, rendered: RenderedModels): + """``(registry rows, dynamic routes)`` for the gateway, from a render. + + Static routing: one row per alias, to the Model serving it. Dynamic + routing: one route per (deployment, endpoint), each to its own Model, + so same-model ``--dedicated`` Models share the alias and LiteLLM + balances across them. + """ + from ..leasing.gateway import UPSTREAM_ROUTE, upstream_route + + if self.gateway is None: + return {}, [] + base = self._upstream_url() + if self.dynamic_routing: + by_id = {g.id: g for g in desired} + routes = [ + upstream_route(gid, endpoint, name, base) + for name, gid in sorted(rendered.models.items()) + for endpoint in sorted(by_id[gid].served) + ] + return {}, routes + rows = self.catalog_route_rows(self.catalog) + rows.update({ + alias: {'engine': UPSTREAM_ROUTE, 'served': name, 'api_base': base} + for alias, name in rendered.request_names.items() + }) + return rows, [] + + def _set_front_door(self, desired, rendered: RenderedModels) -> None: + """Hand the gateway its inputs for the next render or preview.""" + rows, routes = self._front_door_inputs(desired, rendered) + self.gateway.upstream_rows = rows + self.gateway.upstream_routes = routes + # -- converge-style surface ------------------------------------------------ - def converge(self, desired: list[Deployment], *, apply: bool = True): - """Render the desired Model set, then optionally apply it.""" + #: The cluster schedules: no host GPU indices are allocated or recorded, + #: and admission commits an empty allocation for every deployment. + allocates_gpus = False + + #: Seconds a read of the cluster's GPU facts is reused: resource profiles + #: and node labels change rarely, and a render reads them once. + GPU_FACTS_TTL = 60.0 + #: The KubeAI chart's configuration, which holds its resourceProfiles. + CONFIG_MAP = 'kubeai-config' + + def gpu_facts(self) -> tuple[dict[str, dict[str, Any]], dict[str, float]]: + """``(each node's GPUs, each sized profile's GPU GiB)`` from the cluster. + + Empty when the cluster cannot say (no GPU labels, no chart config): + sizing then falls back to the default profile, as before. + """ + import time + + cached = getattr(self, '_gpu_facts_cache', None) + if cached is not None and time.monotonic() - cached[0] < self.GPU_FACTS_TTL: + return cached[1] + try: + nodes = (json.loads(self._kubectl(['get', 'nodes', '-o', 'json']) or '{}') + .get('items') or []) + config = json.loads(self._kubectl( + ['get', 'configmap', self.CONFIG_MAP, '-o', 'json']) or '{}') + system = yaml.safe_load(((config.get('data') or {}).get('system.yaml')) or '') or {} + facts = node_gpus(nodes) + sized = sized_profiles(system.get('resourceProfiles') or {}, facts) + except Exception: # noqa: BLE001 - sizing is best-effort, never fatal + facts, sized = {}, {} + self._gpu_facts_cache = (time.monotonic(), (facts, sized)) + return facts, sized + + def suggestion_inventory(self) -> tuple[dict[str, Any], dict[str, Any]]: + """``(inventory, resourceProfiles)`` for ``catalog suggest``. + + The inventory is the GPUs of the largest GPU node (a Model runs on one + node, so that is the biggest thing it can use), shaped like a host + inventory. The profiles are one per GPU product in the cluster, each + selecting that product's nodes: what the helm values need for + ``min_vram_gib`` to choose by size. + """ + facts, _ = self.gpu_facts() + gpu_nodes = [f for f in facts.values() if f['count'] and f['memory_gib']] + if not gpu_nodes: + return {'gpu_count': 0, 'gpus': []}, {} + best = max(gpu_nodes, key=lambda f: (f['count'] * f['memory_gib'], f['memory_gib'])) + gpus = [{'index': i, 'name': best['product'] or 'GPU', + 'memory_gib': best['memory_gib'], + 'memory_mib': int(best['memory_gib'] * 1024), + 'display_active': False} for i in range(best['count'])] + profiles = {} + for f in sorted(gpu_nodes, key=lambda f: f['memory_gib']): + product = f['product'] or 'gpu' + profiles[f'nvidia-{dns_slug(product)}'] = { + 'imageName': 'nvidia-gpu', + 'runtimeClassName': 'nvidia', + 'requests': {GPU_RESOURCE: '1'}, + 'limits': {GPU_RESOURCE: '1'}, + 'nodeSelector': {GPU_PRODUCT_LABEL: product}, + } + return {'gpu_count': len(gpus), 'gpus': gpus}, profiles + + def _enrich_min_vram(self, desired) -> None: + """Fill a missing ``min_vram_gib`` from a recorded measurement (in memory). + + The compose resolution order: a catalog-declared value wins, else the + measurements overlay. Never persisted. + """ + from ..leasing.vram import measurement_key_for_spec + + for g in desired: + placement = dict(g.spec.get('placement') or {}) + if g.engine != 'vllm' or placement.get('min_vram_gib'): + continue + measured = self.measurements.get_min_vram_gib(measurement_key_for_spec(g.spec)) + if measured: + placement.update(min_vram_gib=measured, min_vram_source='measured') + g.spec['placement'] = placement + + def _needs_sizing(self, desired) -> bool: + """Whether any deployment asks for its profile by GPU memory.""" + for g in desired: + runtime = g.spec.get('runtime') or {} + if (g.spec.get('placement') or {}).get('min_vram_gib') and not runtime.get( + 'resource_profile'): + return True + return False + + def _render_documents(self, desired: list[Deployment]): + """``(plan, rendered, planned)`` in memory: the one KubeAI render. + + The plan assigns every renderable deployment no GPUs; the + unrenderable ones are left out, with the render's reasons. + """ + from ..leasing.placement import GpuPlan + + desired = list(desired) + self._enrich_min_vram(desired) + rendered = render_models( + desired, + namespace=self.namespace, + default_resource_profile=self.default_resource_profile, + unique_names=self.dynamic_routing, + sized=self.gpu_facts()[1] if self._needs_sizing(desired) else None, + ) + plan = GpuPlan( + assignments={g.id: [] for g in desired if g.id not in rendered.unrenderable}, + errors=list(rendered.errors), + ) + return plan, rendered, {self.models_file: rendered.text} + + def preview(self, desired: list[Deployment], placement=None, *, approve: bool = False): + """Render ``desired`` as :meth:`converge` would; write nothing. + + ``placement`` is accepted for the admission interface and ignored: + the cluster places. Returns ``(plan, rendered)``. + """ + plan, rendered, planned = self._render_documents(desired) + self._preview_approval(planned, approve=approve) + if self.gateway is not None: + # The gateway's changes are part of the same approval: shown now, + # before the lease commits, and not asked again at the render. + self._set_front_door(desired, rendered) + self.gateway.preview([], None, approve=approve) + self.last_preview_digest = self._combined_digest( + self.last_preview_digest, self.gateway.last_preview_digest) + return plan, rendered + + def _combined_digest(self, models: str | None, gateway: str | None) -> str | None: + """One digest for the Models and the gateway's files together.""" + if gateway is None: + return models + return self._planned_digest({'models': models or '', 'gateway': gateway}) + + def plan_on_idle_host(self, desired: list[Deployment]): + """Whether ``desired`` could ever be served here: renderable or not. + + Capacity is the cluster's to decide, so only a render failure is + permanent; that lets a queued acquire of one fail at once. + """ + return self._render_documents(desired)[0] + + def converge(self, desired: list[Deployment], *, apply: bool = True, placement=None): + """Render the desired Model set, then optionally apply it. + + ``placement`` (admission inputs) is ignored: the cluster places. + """ from .._log import logger desired = list(desired) @@ -341,29 +774,19 @@ def converge(self, desired: list[Deployment], *, apply: bool = True): self.namespace, ', '.join(sorted(g.id for g in desired)) or '(none)', ) - for g in desired: - # Warn-and-ignore by decision (vram-aware-placement.md, - # Resolutions #3): k8s owns placement on this backend; the - # equivalent mechanism is the resourceProfile / resource - # requests, not our single-host planner. - if (g.spec.get('placement') or {}).get('min_vram_gib'): - logger.warning( - ' {}: placement.min_vram_gib is ignored on the ' - 'kubeai backend (k8s owns placement — express the ' - 'requirement via the resource profile instead)', - g.id, - ) - rendered = render_models( - desired, - namespace=self.namespace, - default_resource_profile=self.default_resource_profile, - ) + plan, rendered, planned = self._render_documents(desired) + for gid, why in sorted(rendered.profile_reasons.items()): + if why.startswith('min_vram_gib'): + # The cluster places, within the profile chosen by size. + logger.info(' {}: resource profile by {}', gid, why) self.last_errors = list(rendered.errors) self.last_unplaced = set(rendered.unrenderable) - self.last_assignments = {} + self.last_assignments = {} # the cluster places for err in rendered.errors: logger.warning(' render: {}', err) - self._approve_changes({self.models_file: rendered.text}) + models_digest = self._planned_digest(planned) + self.last_planned_digest = models_digest + self._approve_changes(planned) self._atomic_write(self.models_file, rendered.text) self._save_sidecar( { @@ -371,13 +794,18 @@ def converge(self, desired: list[Deployment], *, apply: bool = True): 'request_names': rendered.request_names, } ) + if self.gateway is not None: + # Gateway only: no engines on this host, so nothing to place. + # Its converge persists the merged route registry after its + # own approval (pre-approved by the preview when there was one). + self._set_front_door(desired, rendered) + self.gateway.converge([], apply=False) + self.last_planned_digest = self._combined_digest( + models_digest, self.gateway.last_planned_digest) if not apply: - logger.info( - 'rendered {} Model(s) to {} (not applied; ' - '`infer-stack apply` to converge the cluster)', - len(rendered.models), - self.models_file, - ) + # The caller applies next, or (--no-apply, render) says how. + logger.info('rendered {} Model(s) to {}', + len(rendered.models), self.models_file) return None self.apply() return None @@ -395,6 +823,10 @@ def render_profile(self) -> dict: 'namespace': self.namespace, 'base_url': self.base_url, 'resource_profile': self.default_resource_profile, + # The front door's own settings (UI, proxy, routing mode), in the + # gateway project's profile; None without a gateway. + 'gateway': self.gateway.render_profile() if self.gateway is not None else None, + 'gateway_upstream': self.gateway_upstream, 'catalogs': catalog_sources(self.catalog), } @@ -416,8 +848,26 @@ def use_profile(self, profile: dict) -> None: self.namespace = profile['namespace'] self.base_url = profile['base_url'] self.default_resource_profile = profile['resource_profile'] + self.gateway_upstream = profile.get('gateway_upstream') or self.gateway_upstream sources = profile.get('catalogs') or [] self.catalog = CatalogUnion.from_sources(sources) if sources else None + front = profile.get('gateway') + # A profile from before the front door's settings were recorded holds + # only True/False here; it then keeps this process's settings. + if isinstance(front, dict) and self.gateway is not None: + wanted = 'cluster' if front.get('placement') == 'cluster' else 'host' + have = 'cluster' if getattr(self.gateway, 'in_cluster', False) else 'host' + if wanted != have: + # The snapshot decides where the gateway runs, as it decides + # everything else it renders. + if self.gateway_factory is None: + from ..leasing.profile import ProfileMismatch + + raise ProfileMismatch( + f'the active recovery snapshot has the gateway on the {wanted}, ' + f'this process on the {have}') + self.gateway = self.gateway_factory(wanted) + self.gateway.use_profile(front) def apply(self) -> None: """Converge the cluster to the last render: apply + prune. @@ -457,6 +907,85 @@ def apply(self) -> None: self._kubectl( ['delete', 'models.kubeai.org', name, '--ignore-not-found'] ) + if self.gateway is not None: + self.gateway.apply() + + def residency(self): + """The managed Models' pods, strictly: raises rather than guess. + + Unlike :meth:`observe`, a kubectl failure is never "nothing running"; + see :mod:`infer_stack.leasing.residency`. + """ + from ..leasing.residency import ( + POD_MANAGED_LABEL, + ResidencyUnknown, + residency_from_pods, + ) + + try: + raw = self._kubectl(['get', 'pods', '-l', f'{POD_MANAGED_LABEL}=true', + '-o', 'json']) + except Exception as ex: # noqa: BLE001 - any failure is "unknown" + raise ResidencyUnknown(f'kubectl get pods failed: {ex}') from ex + return residency_from_pods(raw) + + def instances(self): + """The managed Models' pods, then the gateway's containers. + + Raises :class:`~infer_stack.leasing.residency.ResidencyUnknown` when + the cluster cannot be read. + """ + from ..leasing.instances import KUBERNETES, from_residency + + pods = from_residency(self.residency(), runtime=KUBERNETES, + namespace=self.namespace) + return pods + (self.gateway.instances() if self.gateway is not None else []) + + def deployment_logs(self, deployment: Deployment, *, tail: int = 400) -> str: + """Recent engine logs: each pod's current run, then its previous one. + + After a crash the kubelet has already restarted the container, so the + cause is in the previous run's log; it goes last, where the diagnosis + quotes from. Fail-open to ``''``. + """ + from ..leasing.residency import ResidencyUnknown + + try: + pods = self.residency().containers(deployment.id) + except ResidencyUnknown: + return '' + parts: list[str] = [] + for pod in pods: + runs = [[]] + ([['--previous']] if pod.restart_count else []) + for extra in runs: + try: + parts.append(self._kubectl(['logs', pod.container_id, '--tail', + str(tail), *extra])) + except Exception: # noqa: BLE001 - gone, or no previous run + pass + return '\n'.join(p for p in parts if p) + + def startup_failure(self, deployment: Deployment) -> str | None: + """Why this Model's engine cannot start, or ``None`` if it may still load.""" + from ..leasing.diagnosis import diagnose_startup + from ..leasing.residency import ResidencyUnknown + + try: + pods = self.residency().containers(deployment.id) + except ResidencyUnknown: + return None # cannot read the cluster: say nothing + return diagnose_startup(pods, lambda: self.deployment_logs(deployment, tail=200)) + + def _waiting_reason(self, deployment: Deployment) -> str: + """The pod's own reason for not running yet (``ImagePullBackOff``...).""" + from ..leasing.residency import ResidencyUnknown + + try: + pods = self.residency().containers(deployment.id) + except ResidencyUnknown: + return '' + reasons = sorted({p.reason for p in pods if p.reason and p.state != 'running'}) + return ', '.join(reasons) def observe(self) -> set[str]: """Deployment ids with a managed Model CR on the cluster (best-effort).""" @@ -475,18 +1004,36 @@ def probe_ready(self, deployment: Deployment, endpoint: str) -> Readiness: """ if deployment.id not in self.observe(): return Readiness(False, 'Model CR not on the cluster') - name = model_name_for(_served_name(deployment)) served = deployment.served.get(endpoint) or {} protocol = served.get('protocol') or 'chat' + if self.gateway is not None: + # Ready means ready the way a client sees it: the alias, through + # the gateway, with its key. + front = self.gateway.gateway # the runner's leasing.gateway.Gateway + base, model = f'{front._gateway_base()}/v1', endpoint + headers = front._auth_headers() + else: + base, model = self.base_url, model_name(deployment) + headers = None ok, reason = openai_ready( - base_url=self.base_url, - model=name, + base_url=base, + headers=headers, + model=model, protocol=protocol, require_listed=True, require_generation=True, http=self.http, ) - return Readiness(ok, reason) + if ok: + return Readiness(True, reason) + # A crash-looping engine keeps its pod "Running" between restarts, so + # check every not-ready probe, not only when the Model is missing. + failure = self.startup_failure(deployment) + if failure is not None: + return Readiness(False, failure, fatal=True) + waiting = self._waiting_reason(deployment) + return Readiness(False, f'{reason} (pod: {waiting})' if waiting else reason, + needs_room='Unschedulable' in waiting) def access(self, endpoints: list[str]) -> dict[str, Any] | None: """Where a client reaches these endpoints (env-file descriptor). @@ -496,6 +1043,8 @@ def access(self, endpoints: list[str]) -> dict[str, Any] | None: no alias layer. The gateway is unauthenticated, so the api key is the literal ``EMPTY`` placeholder and no key env var is advertised. """ + if self.gateway is not None: + return self.gateway.access(endpoints) request_names = self._load_sidecar().get('request_names') or {} return { 'base_url': self.base_url, @@ -517,17 +1066,19 @@ def realize(self, deployment: Deployment) -> None: # pragma: no cover pass def teardown(self, deployment: Deployment) -> None: - name = model_name_for(_served_name(deployment)) + name = model_name(deployment, unique=self.dynamic_routing) self._kubectl( ['delete', 'models.kubeai.org', name, '--ignore-not-found'] ) def down(self) -> None: - """Delete every infer-stack-managed Model (explicit stop).""" + """Delete every infer-stack-managed Model (explicit stop), and the gateway.""" for name in sorted(self._cluster_models()): self._kubectl( ['delete', 'models.kubeai.org', name, '--ignore-not-found'] ) + if self.gateway is not None: + self.gateway.down() # -- preflight ------------------------------------------------------------- @@ -537,7 +1088,7 @@ def doctor(self) -> list[tuple[str, bool, str]]: Everything ``acquire`` needs, checked cheaply and in dependency order, so a fresh setup fails as a checklist instead of a mid-acquire traceback: cluster reachable -> KubeAI CRD installed -> namespace - exists -> gateway answering at ``base_url``. Never raises. + exists -> KubeAI's API answering at ``base_url``. Never raises. """ checks: list[tuple[str, bool, str]] = [] @@ -568,6 +1119,10 @@ def _run_check(name: str, args: list[str], hint: str) -> bool: f'kubectl create namespace {self.namespace} (or install the ' 'chart there)', ) + if getattr(self.gateway, 'in_cluster', False): + # Clients use the gateway in the cluster; no port-forward needed. + checks.append(self.gateway.doctor_check()) + return checks try: resp = self.http.get(f'{self.base_url}/models', timeout=10) code = getattr(resp, 'status_code', 0) @@ -582,5 +1137,12 @@ def _run_check(name: str, args: list[str], hint: str) -> bool: f'`kubectl -n {self.namespace} port-forward svc/kubeai ' '8000:80` (or set kubeai_base_url)' ) - checks.append((f'gateway at {self.base_url}', ok, detail)) + # KubeAI's own API (what the port-forward reaches), not infer-stack's + # LiteLLM gateway, which starts with the first acquire. + checks.append((f"KubeAI's API at {self.base_url}", ok, detail)) + if self.gateway is not None and hasattr(self.gateway, 'doctor'): + # The gateway on this host is a Compose project: Docker and its + # images are part of what an acquire needs. + checks.extend((f'gateway: {name}', good, why) + for name, good, why in self.gateway.doctor()) return checks diff --git a/infer_stack/backends/kubeai_gateway.py b/infer_stack/backends/kubeai_gateway.py new file mode 100644 index 00000000..95d24b54 --- /dev/null +++ b/infer_stack/backends/kubeai_gateway.py @@ -0,0 +1,319 @@ +"""The LiteLLM gateway inside the cluster, for the kubeai backend. + +The default front door for a cluster is a one-service Compose project on the +host running infer-stack, so that host is in every request's path. This is +the other placement: the same gateway (the same image, config, key and route +registry, through :class:`~infer_stack.leasing.gateway.Gateway`) rendered as +Kubernetes objects in the KubeAI namespace and reached on a NodePort of any +node, or an ingress URL. + +It answers the calls the kubeai backend makes on its gateway (keys, routes, +preview / converge / apply, down, instances), so the backend does not know +which placement it has. Static routes only: dynamic routing and Open WebUI +need Postgres and a UI in the cluster, which stay with the host placement. + +Objects, all named ``infer-stack-gateway``: a ConfigMap (the LiteLLM config), +a Secret (the master key, applied from the state dir's ``.env`` and never +shown in a diff), a Deployment whose pod template carries the config's and +the key's hashes (so either change rolls the pods), and a NodePort Service. +""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +from typing import Any, Callable + +import yaml + +from ..config import PINNED_IMAGES +from ..leasing.backend import ConvergeScaffold +from ..leasing.gateway import ( + API_KEY_ENV, + LITELLM_CONTAINER_PORT, + Gateway, + render_front_door, +) + +NAME = 'infer-stack-gateway' +#: The pod label the Service selects and ``instances`` lists by. +APP_LABEL = 'app.kubernetes.io/name' +MANIFESTS_FILENAME = 'gateway.yaml' +SECRET_FILENAME = 'gateway-secret.yaml' # 0600, like the .env it copies +STATE_FILENAME = 'gateway-state.json' +DEFAULT_NODE_PORT = 30442 + + +class ClusterGateway(ConvergeScaffold): + """The LiteLLM gateway as a Deployment + NodePort Service in the cluster.""" + + _approve_title = 'infer-stack will update the in-cluster gateway' + _state_noun = 'gateway manifests' + #: Where this gateway runs, for the kubeai backend's routing decisions. + in_cluster = True + #: The settings a host gateway has and this one does not (see module doc). + litellm = True + ui = False + reverse_proxy = False + dynamic_routing = False + + def __init__( + self, + *, + state_dir: str | Path, + namespace: str, + run: Callable[[list[str]], str], + node_port: int = DEFAULT_NODE_PORT, + url: str | None = None, + images: dict[str, str] | None = None, + http: Any = None, + assume_yes: bool = True, + ): + self.state_dir = Path(state_dir) + self.state_dir.mkdir(parents=True, exist_ok=True) + self.namespace = namespace + self.run = run + self.node_port = int(node_port) + self.url = url + self.images = {**PINNED_IMAGES, **(images or {})} + self.assume_yes = assume_yes + self.gateway = Gateway(self.state_dir, ports={'litellm': self.node_port}, + litellm=True, ui=False, http=http, + base_url=self.base_url) + self.catalog = None + #: Render inputs from the kubeai backend (see ComposeBackend's). + self.upstream_rows: dict[str, dict[str, Any]] = {} + self.upstream_routes: list[dict[str, Any]] = [] + self._node_address: str | None = None + + # -- where it is --------------------------------------------------------- + + @property + def http(self) -> Any: + return self.gateway.http + + @http.setter + def http(self, value: Any) -> None: + self.gateway.http = value + + @property + def manifests_file(self) -> Path: + return self.state_dir / MANIFESTS_FILENAME + + rendered_file = manifests_file + + @property + def _state_file(self) -> Path: + return self.state_dir / STATE_FILENAME + + @property + def litellm_port(self) -> int: + return self.node_port + + def _kubectl(self, args: list[str]) -> str: + return self.run(['kubectl', '-n', self.namespace, *args]) + + def base_url(self) -> str: + """Where clients reach it: the ``url`` setting, else a node's NodePort. + + A NodePort answers on every node, so any node's address works; the + first node's InternalIP is used. + """ + if self.url: + return self.url.rstrip('/') + if self._node_address is None: + nodes = json.loads(self._kubectl(['get', 'nodes', '-o', 'json']) or '{}') + addresses = [a['address'] for n in nodes.get('items') or [] + for a in (n.get('status') or {}).get('addresses') or [] + if a.get('type') == 'InternalIP'] + self._node_address = addresses[0] if addresses else '127.0.0.1' + return f'http://{self._node_address}:{self.node_port}' + + def compose_project(self): + """No Compose project: this gateway is Kubernetes objects.""" + return None + + def front_door(self): + return self + + # -- keys and routes: the one Gateway ------------------------------------ + + def master_key(self) -> str: + return self.gateway.master_key() + + def rotate_master_key(self): + return self.gateway.rotate_master_key() + + def restore_env(self, values) -> None: + self.gateway.restore_env(values) + + def gateway_accepts(self, key: str, *, wait: float = 0.0): + return self.gateway.gateway_accepts(key, wait=wait) + + def access(self, endpoints: list[str]) -> dict[str, Any] | None: + return self.gateway.access(endpoints) + + def merge_route_registry(self, incoming): + return self.gateway.merge_route_registry(incoming) + + def catalog_route_rows(self, catalog) -> dict[str, dict[str, Any]]: + return {} # the kubeai backend supplies its own rows + + # -- render / apply ----------------------------------------------------- + + def _render_documents(self) -> tuple[dict[Path, str], dict[str, Any]]: + """``(planned files, merged registry)`` in memory: the one render.""" + registry = self.gateway.merged_route_registry(dict(self.upstream_rows)) + front = render_front_door( + [], {}, engine_services=[], vllm_v1_urls=[], ollama_native_urls=[], + images=self.images, state={}, litellm=True, + litellm_port=LITELLM_CONTAINER_PORT, litellm_master_key=None, + litellm_salt_key=False, ui=False, ui_port=0, reverse_proxy=False, + reverse_proxy_port=0, reverse_proxy_config=None, aux_dir=self.state_dir, + catalog=None, route_registry=registry, dynamic_routing=False, + ) + config = front.litellm_config or '' + key_hash = hashlib.sha256(self.master_key().encode()).hexdigest()[:12] + docs = gateway_manifests( + namespace=self.namespace, image=self.images['litellm'], config=config, + key_hash=key_hash, node_port=self.node_port) + text = '---\n'.join(yaml.safe_dump(d, sort_keys=False) for d in docs) + return {self.manifests_file: text}, registry + + def preview(self, desired=(), placement=None, *, approve: bool = False): + """Render without writing; with ``approve``, show the diff now.""" + planned, _ = self._render_documents() + self._preview_approval(planned, approve=approve) + return None, None + + def converge(self, desired=(), *, apply: bool = True, placement=None): + """Render the gateway's objects (and persist the registry after approval).""" + with self._converge_lock(): + planned, registry = self._render_documents() + self.last_planned_digest = self._planned_digest(planned) + self._approve_changes(planned) + self.gateway._save_route_registry(registry) + for path, text in planned.items(): + self._atomic_write(path, text) + if apply: + self.apply() + + def apply(self) -> None: + """Apply the key's Secret, then the objects, and wait for the rollout.""" + import os + + from .._log import logger + + if not self.manifests_file.exists(): + return + secret = yaml.safe_dump(secret_manifest(self.namespace, self.master_key())) + path = self.state_dir / SECRET_FILENAME + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) + with os.fdopen(fd, 'w') as handle: + handle.write(secret) + logger.info('kubectl apply (the in-cluster gateway)') + self._kubectl(['apply', '-f', str(path)]) + self._kubectl(['apply', '-f', str(self.manifests_file)]) + self._kubectl(['rollout', 'status', f'deployment/{NAME}', '--timeout=300s']) + + def down(self) -> None: + self._kubectl(['delete', 'deployment,service,configmap,secret', NAME, + '--ignore-not-found']) + + def instances(self): + from ..leasing.instances import KUBERNETES, from_residency + from ..leasing.residency import ResidencyUnknown, residency_from_pods + + try: + raw = self._kubectl(['get', 'pods', '-l', f'{APP_LABEL}={NAME}', '-o', 'json']) + except Exception as ex: # noqa: BLE001 - any failure is "unknown" + raise ResidencyUnknown(f'kubectl get pods failed: {ex}') from ex + return from_residency(residency_from_pods(raw), runtime=KUBERNETES, + namespace=self.namespace) + + def doctor_check(self) -> tuple[str, bool, str]: + """``(check, ok, detail)``: does the gateway answer with the managed key?""" + base = self.base_url() + accepted = self.gateway_accepts(self.master_key()) + if accepted: + return (f'in-cluster gateway at {base}/v1', True, '') + return (f'in-cluster gateway at {base}/v1', False, + 'not answering; `infer-stack apply` renders and starts it, and ' + '`kubectl -n get pods -l ' + f'{APP_LABEL}={NAME}` shows why it is not up') + + # -- the recovery profile ------------------------------------------------ + + def render_profile(self) -> dict[str, Any]: + return {'placement': 'cluster', 'node_port': self.node_port, 'url': self.url, + 'image': self.images['litellm']} + + def use_profile(self, profile: dict[str, Any]) -> None: + if profile.get('placement') != 'cluster': + from ..leasing.profile import ProfileMismatch + + raise ProfileMismatch( + 'the active recovery snapshot has the gateway on this host; ' + '`infer-stack stack down` it before moving it into the cluster') + self.node_port = int(profile.get('node_port') or self.node_port) + self.url = profile.get('url') or self.url + self.images['litellm'] = profile.get('image') or self.images['litellm'] + + +def secret_manifest(namespace: str, key: str) -> dict[str, Any]: + """The master key as a Secret (applied, never rendered into a diff).""" + return {'apiVersion': 'v1', 'kind': 'Secret', + 'metadata': {'name': NAME, 'namespace': namespace}, + 'type': 'Opaque', 'stringData': {API_KEY_ENV: key}} + + +def gateway_manifests(*, namespace: str, image: str, config: str, key_hash: str, + node_port: int) -> list[dict[str, Any]]: + """ConfigMap, Deployment and NodePort Service for the in-cluster gateway. + + >>> docs = gateway_manifests(namespace='kubeai', image='litellm:x', + ... config='model_list: []\\n', key_hash='abc', node_port=30442) + >>> [d['kind'] for d in docs] + ['ConfigMap', 'Deployment', 'Service'] + >>> docs[2]['spec']['ports'][0]['nodePort'] + 30442 + """ + config_hash = hashlib.sha256(config.encode()).hexdigest()[:12] + labels = {APP_LABEL: NAME} + meta = {'name': NAME, 'namespace': namespace, 'labels': labels} + return [ + {'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': meta, + 'data': {'config.yaml': config}}, + {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': meta, + 'spec': { + 'replicas': 1, + 'selector': {'matchLabels': labels}, + 'template': { + 'metadata': {'labels': labels, 'annotations': { + # LiteLLM reads both once at startup: a change must roll. + 'infer-stack/config-hash': config_hash, + 'infer-stack/key-hash': key_hash}}, + 'spec': {'containers': [{ + 'name': 'litellm', + 'image': image, + 'args': ['--config', '/etc/litellm/config.yaml', + '--port', str(LITELLM_CONTAINER_PORT)], + 'envFrom': [{'secretRef': {'name': NAME}}], + 'ports': [{'containerPort': LITELLM_CONTAINER_PORT}], + 'readinessProbe': { + 'httpGet': {'path': '/health/liveliness', + 'port': LITELLM_CONTAINER_PORT}, + 'periodSeconds': 5}, + 'volumeMounts': [{'name': 'config', 'mountPath': '/etc/litellm', + 'readOnly': True}], + }], + 'volumes': [{'name': 'config', 'configMap': {'name': NAME}}]}, + }, + }}, + {'apiVersion': 'v1', 'kind': 'Service', 'metadata': meta, + 'spec': {'type': 'NodePort', 'selector': labels, + 'ports': [{'port': LITELLM_CONTAINER_PORT, + 'targetPort': LITELLM_CONTAINER_PORT, + 'nodePort': node_port}]}}, + ] diff --git a/infer_stack/backends/kubeai_renderer.py b/infer_stack/backends/kubeai_renderer.py deleted file mode 100644 index 21512310..00000000 --- a/infer_stack/backends/kubeai_renderer.py +++ /dev/null @@ -1,204 +0,0 @@ -from __future__ import annotations - -from pathlib import Path -from typing import Any - -import yaml - -from ..config import KUBEAI_GENERATED_SUBDIR, normalized_output -from ..diff_prompt import confirm_writes -from ..profile_runtime import vllm_args - - -def _resource_profile_values(plan: dict[str, Any]) -> dict[str, Any]: - values_doc = plan.get('deployment', {}).get('resource_profiles_values', {}) - return values_doc or {'resourceProfiles': {}} - - -def _kubeai_resource_profile(service: dict[str, Any]) -> str: - profile = str(service.get('resource_profile', '')) - if not profile or ':' in profile: - return profile - gpu_count = max( - 1, - len(service.get('gpu_indices', [])), - int(service.get('tensor_parallel_size', 1) or 1) - * int(service.get('pipeline_parallel_size', 1) or 1) - * int(service.get('data_parallel_size', 1) or 1), - ) - return f'{profile}:{gpu_count}' - - -def _kubeai_args(service: dict[str, Any]) -> list[str]: - kubeai_service = dict(service) - kubeai_service['served_model_name'] = service['profile_public_name'] - return vllm_args(kubeai_service) - - -def _model_doc(service: dict[str, Any]) -> dict[str, Any]: - doc = { - 'apiVersion': 'kubeai.org/v1', - 'kind': 'Model', - 'metadata': { - 'name': service['kubernetes_name'], - 'annotations': { - 'infer-stack/profile-name': service['profile_name'], - 'infer-stack/public-name': service['profile_public_name'], - 'infer-stack/logical-model-name': service['logical_model_name'], - 'infer-stack/protocol-mode': service['protocol_mode'], - }, - }, - 'spec': { - 'features': service.get('features', ['TextGeneration']), - 'url': service['model_url'], - 'engine': service.get('engine', 'VLLM'), - 'resourceProfile': _kubeai_resource_profile(service), - 'minReplicas': int(service.get('min_replicas', 0)), - 'maxReplicas': int(service.get('max_replicas', 1)), - 'args': _kubeai_args(service), - }, - } - if service.get('priority_class_name'): - doc['spec']['priorityClassName'] = service['priority_class_name'] - return doc - - -def render_kubeai_artifacts( - lock_data: dict, *, assume_yes: bool = True -) -> None: - """Render the KubeAI backend artifacts. - - When ``assume_yes`` is False, all rendered YAML files are diffed against - their on-disk versions and the user is prompted via Rich before any file - is written. - """ - deployment = lock_data.get('deployment', {}) - cluster = deployment.get('cluster', {}) - namespace = cluster.get('namespace', 'kubeai') - output_root = Path( - normalized_output(deployment.get('output'))['generated_dir'] - ) - generated = output_root / KUBEAI_GENERATED_SUBDIR - generated.mkdir(parents=True, exist_ok=True) - - namespace_doc = { - 'apiVersion': 'v1', - 'kind': 'Namespace', - 'metadata': {'name': namespace}, - } - namespace_text = yaml.safe_dump(namespace_doc, sort_keys=False) - - values_doc = _resource_profile_values(lock_data) - values_text = yaml.safe_dump(values_doc, sort_keys=False) - - model_docs = [ - _model_doc(service) - for service in ( - deployment.get('providers', {}).get('vllm', {}).get('runtimes', {}) - or {} - ).values() - ] - model_text = '---\n'.join( - yaml.safe_dump(doc, sort_keys=False) for doc in model_docs - ) - - ingress = cluster.get('ingress', {}) or {} - ingress_path = generated / 'ingress.yaml' - ingress_text: str | None = None - if ingress.get('enabled'): - path_prefix = ingress.get('path_prefix', '/') or '/' - ingress_doc: dict[str, Any] = { - 'apiVersion': 'networking.k8s.io/v1', - 'kind': 'Ingress', - 'metadata': { - 'name': cluster.get('service_name', 'kubeai'), - 'namespace': namespace, - }, - 'spec': { - 'ingressClassName': ingress.get('class_name', 'traefik'), - 'rules': [ - { - 'http': { - 'paths': [ - { - 'path': path_prefix, - 'pathType': 'Prefix', - 'backend': { - 'service': { - 'name': cluster.get( - 'service_name', 'kubeai' - ), - 'port': {'number': 80}, - } - }, - } - ] - } - } - ], - }, - } - if ingress.get('host'): - ingress_doc['spec']['rules'][0]['host'] = ingress['host'] - if ingress.get('tls_secret_name') and ingress.get('host'): - ingress_doc['spec']['tls'] = [ - { - 'hosts': [ingress['host']], - 'secretName': ingress['tls_secret_name'], - } - ] - ingress_text = yaml.safe_dump(ingress_doc, sort_keys=False) - - readme = f"""# Generated KubeAI artifacts - -Namespace: `{namespace}` -Release: `{cluster.get('kubeai_release_name', 'kubeai')}` -Chart: `{cluster.get('kubeai_chart', 'kubeai/kubeai')}` - -Files: -- `namespace.yaml`: namespace to apply before the chart and models -- `kubeai-values.yaml`: custom resource profiles for the KubeAI chart -- `models.yaml`: KubeAI `Model` objects derived intentionally from the selected serving profile(s) -- `ingress.yaml`: optional ingress for one stable hostname - -Typical flow: - -```bash -kubectl apply -f {generated}/namespace.yaml -helm repo add kubeai https://www.kubeai.org --force-update -helm repo update -helm upgrade --install {cluster.get('kubeai_release_name', 'kubeai')} {cluster.get('kubeai_chart', 'kubeai/kubeai')} \ - -n {namespace} --create-namespace \ - -f {generated}/kubeai-values.yaml \ - --wait -kubectl apply -f {generated}/models.yaml -``` -""" - - namespace_path = generated / 'namespace.yaml' - values_path = generated / 'kubeai-values.yaml' - models_path = generated / 'models.yaml' - readme_path = generated / 'README.md' - - planned: dict[Path, str] = { - namespace_path: namespace_text, - values_path: values_text, - models_path: model_text, - readme_path: readme, - } - if ingress_text is not None: - planned[ingress_path] = ingress_text - - if not confirm_writes( - planned, assume_yes=assume_yes, title='Pending KubeAI render' - ): - raise SystemExit('Aborted by user; no files were written.') - - namespace_path.write_text(namespace_text, encoding='utf-8') - values_path.write_text(values_text, encoding='utf-8') - models_path.write_text(model_text, encoding='utf-8') - readme_path.write_text(readme, encoding='utf-8') - if ingress_text is not None: - ingress_path.write_text(ingress_text, encoding='utf-8') - elif ingress_path.exists(): - ingress_path.unlink() diff --git a/infer_stack/cli/__init__.py b/infer_stack/cli/__init__.py index 94c81a8d..5e3d44a3 100644 --- a/infer_stack/cli/__init__.py +++ b/infer_stack/cli/__init__.py @@ -1,9 +1,9 @@ #!/usr/bin/env python3 # PYTHON_ARGCOMPLETE_OK -"""scriptconfig-based CLI for infer-stack. +"""kwconf-based CLI for infer-stack. -Each subcommand is a ``scfg.DataConfig`` subclass; ``ManageCLI`` composes -them into a single ``scfg.ModalCLI`` exposed as the ``infer-stack`` entry +Each subcommand is a ``kw.Config`` subclass; ``ManageCLI`` composes +them into a single ``kw.ModalCLI`` exposed as the ``infer-stack`` entry point. Because every subcommand is a ``DataConfig``, the same class can be invoked from the shell (``infer-stack render --profile X``) or from Python (``RenderCLI.main(argv=False, profile='X')``). @@ -22,7 +22,7 @@ from __future__ import annotations -import scriptconfig as scfg +import kwconf as kw from .. import __version__ @@ -78,7 +78,11 @@ # --------------------------------------------------------------------------- -class ManageCLI(scfg.ModalCLI): +class ManageCLI(kw.ModalCLI): + # The program name in usage and error messages (kwconf would use the class + # name, "ManageCLI"). + __prog__ = 'infer-stack' + description = ( 'Lease, acquire, and run LLM endpoints. Primary workflow: ' 'catalog -> acquire/run. See `infer-stack help tree`.' @@ -104,7 +108,7 @@ class ManageCLI(scfg.ModalCLI): actual. Run `infer-stack help tree` for the whole command surface. """ - # Backs the modal ``--version`` flag (scriptconfig reads ``__version__``). + # Backs the modal ``--version`` flag (kwconf reads ``__version__``). # The ``version`` *subcommand* is registered below under a non-colliding # attribute name; its CLI name comes from ``VersionCLI.__command__``. __version__ = __version__ @@ -161,8 +165,55 @@ def __getattr__(name: str): raise AttributeError(f'module {__name__!r} has no attribute {name!r}') +def runtime_failure(exc) -> str: + """A failed ``docker`` / ``kubectl`` command, for a person, not a traceback. + + The command (without its long path arguments), the runtime's own last + words, and a hint for the failures with a known cause. + + >>> import subprocess + >>> ex = subprocess.CalledProcessError(1, ['docker', 'compose', '-p', 'x', 'up', '-d'], + ... stderr='Error: Bind for :::14042 failed: port is already allocated') + >>> print(runtime_failure(ex)) + `docker compose up -d` failed: + Error: Bind for :::14042 failed: port is already allocated + port 14042 is in use by something else (`ss -ltnp | grep :14042` shows what); stop it, then `infer-stack apply` + >>> ex.stderr = 'failed to bind host port 0.0.0.0:13000/tcp: address already in use' + >>> runtime_failure(ex).splitlines()[-1].split(' (')[0] + ' port 13000 is in use by something else' + """ + import re + + cmd = [str(a) for a in (exc.cmd if isinstance(exc.cmd, (list, tuple)) else [exc.cmd])] + # Drop option values that are paths or project names: the verb is what reads. + shown, skip = [], False + for arg in cmd: + if skip: + skip = False + continue + if arg in ('-f', '-p', '--env-file', '-n', '--namespace'): + skip = True + continue + shown.append(arg) + text = '\n'.join(filter(None, [exc.stderr or '', exc.output or ''])) + last = [ln.strip() for ln in str(text).splitlines() if ln.strip()][-3:] + lines = [f'`{" ".join(shown)}` failed:'] + [f' {ln}' for ln in last] + port = re.search(r'Bind for \S*?:(\d+) failed: port is already allocated' + r'|:(\d+)(?:/tcp)?: (?:bind: )?address already in use', str(text)) + if port: + number = port.group(1) or port.group(2) + lines.append(f' port {number} is in use by something else (`ss -ltnp | grep ' + f':{number}` shows what); stop it, then `infer-stack apply`') + elif shown and shown[0] in ('docker', 'kubectl'): + lines.append(' `infer-stack doctor` checks what this backend needs') + return '\n'.join(lines) + + def main(argv=None) -> int: + import subprocess + from ..leasing import LeaseLockError + from ..leasing.backend import BackendTimeout try: rv = ManageCLI.main(argv=argv) @@ -170,6 +221,13 @@ def main(argv=None) -> int: # A mutating verb (acquire/release/gc/evict/apply) could not get the # cross-process lock; surface the actionable diagnosis, not a traceback. raise SystemExit(str(exc)) + except subprocess.CalledProcessError as exc: + # The runtime refused (a port in use, a daemon down): its words and a + # hint, not our stack. The controller already rolled back what it had + # committed, or left the change pending for `infer-stack apply`. + raise SystemExit(runtime_failure(exc)) + except BackendTimeout as exc: + raise SystemExit(str(exc)) return int(rv) if rv is not None else 0 diff --git a/infer_stack/cli/commands_catalog.py b/infer_stack/cli/commands_catalog.py index 38a5abcc..24e90062 100644 --- a/infer_stack/cli/commands_catalog.py +++ b/infer_stack/cli/commands_catalog.py @@ -16,7 +16,7 @@ from pathlib import Path from typing import Any -import scriptconfig as scfg +import kwconf as kw import yaml from ..leasing import Catalog, CatalogError @@ -223,8 +223,8 @@ def _parse_kv(items) -> dict[str, Any]: class _CatalogCommon(_PathOverridesMixin): - catalog = scfg.Value(None, type=str, help='Catalog path (default: config dir).') - dry_run = scfg.Value( + catalog = kw.Value(None, type=str, help='Catalog path (default: config dir).') + dry_run = kw.Value( False, isflag=True, help='Print the resulting YAML, do not write.' ) @@ -233,7 +233,7 @@ class CatalogInitCLI(_CatalogCommon): """Write a starter catalog.yaml (empty sections) if none exists.""" __command__ = 'init' - force = scfg.Value(False, isflag=True, help='Overwrite an existing catalog.') + force = kw.Value(False, isflag=True, help='Overwrite an existing catalog.') @classmethod def main(cls, argv=True, **kwargs): @@ -269,21 +269,38 @@ class CatalogSuggestCLI( Pure + offline: ``--simulate-hardware 2x80`` suggests for hardware you do not have in front of you. + On the kubeai backend the hardware is the cluster's: GPU Feature + Discovery's node labels (``nvidia.com/gpu.product`` / ``.memory``) and each + node's allocatable GPUs. The catalog is sized to the largest GPU node, and + a ``resourceProfiles`` block (one per GPU product, selecting its nodes) is + printed for the helm values, so ``min_vram_gib`` can pick a profile by size. + infer-stack catalog suggest # render only (no write) infer-stack catalog suggest --simulate-hardware 4x48 infer-stack catalog suggest --apply # merge into the catalog """ __command__ = 'suggest' - catalog = scfg.Value(None, type=str, help='Catalog path (default: config dir).') - apply = scfg.Value( + catalog = kw.Value(None, type=str, help='Catalog path (default: config dir).') + apply = kw.Value( False, isflag=True, help='Merge the suggestion into the catalog (default: render only).', ) - force = scfg.Value( + force = kw.Value( False, isflag=True, help='With --apply, overwrite catalog entries that already exist.', ) + backend = kw.Value( + None, type=str, + help='Whose hardware (default: the configured `backend` setting): this ' + "host's GPUs, or on kubeai the cluster's.", + ) + simulator = kw.Value( + False, isflag=True, + help='Suggest a simulator endpoint instead (`mock-smol`): it answers ' + "like vLLM with random text and needs no GPU, to try infer-stack's " + 'workflow on any host. Never for results.', + ) @classmethod def main(cls, argv=True, **kwargs): @@ -292,16 +309,47 @@ def main(cls, argv=True, **kwargs): migrate_known_suggestion_aliases, suggest_catalog, ) - from .commands_leasing import _resolve_skip_display + from ..paths import get_setting + from .commands_leasing import _make_backend, _resolve_skip_display from .context import effective_inventory config = cls.cli(argv=argv, data=kwargs) - inventory = effective_inventory(config) or detect_inventory() + if config.simulator: + import copy + + from ..leasing.suggest import SIMULATOR_FRAGMENT + + return _merge_or_print(config, copy.deepcopy(SIMULATOR_FRAGMENT), + 'a simulator: no GPU needed, random text, never results') + profiles: dict = {} + cluster = (config.backend or get_setting('backend')) == 'kubeai' + inventory = effective_inventory(config) + if inventory is None and cluster: + backend = _make_backend(config) + inventory, profiles = backend.suggestion_inventory() + if not inventory['gpus']: + print( + 'no cluster node reports GPUs: suggest needs each GPU node\'s ' + 'allocatable nvidia.com/gpu and GPU Feature Discovery\'s ' + 'nvidia.com/gpu.product / .memory labels (the NVIDIA device ' + 'plugin with gfd.enabled=true; see the README). Pass ' + '--simulate-hardware NxM to plan without them.', + file=sys.stderr, + ) + return 1 + inventory = inventory or detect_inventory() gpus = inventory.get('gpus') or [] - skip_display = _resolve_skip_display(config) + skip_display = _resolve_skip_display(config) and not cluster frag = suggest_catalog( inventory, reserve_display_gpu='auto' if skip_display else False ) + if cluster: + # A cluster has no host GPU indices: the profile says where. + for endpoint in frag['endpoints'].values(): + placement = endpoint.get('placement') or {} + placement.pop('gpu_indices', None) + if not placement: + endpoint.pop('placement', None) max_mem = max((g.get('memory_gib') or 0 for g in gpus), default=0) n_display = sum(1 for g in gpus if g.get('display_active')) @@ -315,13 +363,20 @@ def main(cls, argv=True, **kwargs): print( f'no pooled model fits the detected hardware ({hw}). ' 'Pass --simulate-hardware NxM to plan for a bigger box, or add ' - 'models by hand with `catalog model add`.', + 'models by hand with `catalog model add`.' + + ('' if gpus else ' With no GPU here, `infer-stack catalog suggest ' + '--simulator --apply` adds an endpoint that answers like vLLM ' + '(random text), to try the workflow.'), file=sys.stderr, ) return 0 text = yaml.safe_dump(frag, sort_keys=False, default_flow_style=False) + if cluster: + hw = f'the cluster\'s largest GPU node: {hw}' + values = (yaml.safe_dump({'resourceProfiles': profiles}, sort_keys=False) + if profiles else '') if not config.apply: print( f'# suggested for: {hw} ({usable} usable)\n' @@ -330,6 +385,10 @@ def main(cls, argv=True, **kwargs): file=sys.stderr, ) _print_yaml(text) + if values: + print('# helm values for the KubeAI chart (merge, then ' + '`scripts/install_kubeai.sh `):\n' + values, + file=sys.stderr) return 0 # --apply: additive merge into the catalog (keep existing entries). @@ -356,14 +415,44 @@ def main(cls, argv=True, **kwargs): f' kept existing (pass --force to overwrite): ' f'{", ".join(skipped)}' ) + if values: + print('resource profiles for the KubeAI chart (merge into your helm ' + 'values, then `scripts/install_kubeai.sh `):') + print(values, end='') return 0 +def _merge_or_print(config, frag: dict, what: str) -> int: + """Print a suggested catalog fragment, or with ``--apply`` merge it.""" + text = yaml.safe_dump(frag, sort_keys=False, default_flow_style=False) + if not config.apply: + print(f'# suggested: {what}; re-run with --apply to merge', file=sys.stderr) + _print_yaml(text) + return 0 + path = _catalog_path(config) + data = _load_raw(path) + added, skipped = [], [] + for section in ('models', 'endpoints'): + for name, value in frag[section].items(): + if name in data[section] and not config.force: + skipped.append(f'{section[:-1]}:{name}') + continue + data[section][name] = value + added.append(f'{section[:-1]}:{name}') + _save_raw(path, data, dry_run=False) + print(f'merged suggestion into {path} ({what})') + if added: + print(f' added: {", ".join(added)}') + if skipped: + print(f' kept existing (pass --force to overwrite): {", ".join(skipped)}') + return 0 + + class CatalogPathCLI(_PathOverridesMixin): """Print the catalog path.""" __command__ = 'path' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -392,8 +481,8 @@ class CatalogShowCLI(_PathOverridesMixin): """Pretty-print the whole catalog (or one named entry across sections).""" __command__ = 'show' - catalog = scfg.Value(None, type=str) - name = scfg.Value(None, position=1, type=str, help='Optional entry name.') + catalog = kw.Value(None, type=str) + name = kw.Value(None, position=1, type=str, help='Optional entry name.') @classmethod def main(cls, argv=True, **kwargs): @@ -417,7 +506,7 @@ class CatalogValidateCLI(_PathOverridesMixin): """Parse + cross-reference check the catalog.""" __command__ = 'validate' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -440,7 +529,7 @@ class CatalogEditCLI(_PathOverridesMixin): """Open the catalog in $EDITOR (escape hatch), then validate it.""" __command__ = 'edit' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -470,12 +559,12 @@ class ModelAddCLI(_CatalogCommon): """Add (or --force overwrite) a model: a Hugging Face / local weight source.""" __command__ = 'add' - name = scfg.Value(None, position=1, type=str) - source = scfg.Value(None, type=str, help='e.g. hf://org/Model or a path.') - revision = scfg.Value(None, type=str) - quantization = scfg.Value(None, type=str) - dtype = scfg.Value(None, type=str) - force = scfg.Value(False, isflag=True) + name = kw.Value(None, position=1, type=str) + source = kw.Value(None, type=str, help='e.g. hf://org/Model or a path.') + revision = kw.Value(None, type=str) + quantization = kw.Value(None, type=str) + dtype = kw.Value(None, type=str) + force = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -503,7 +592,7 @@ def main(cls, argv=True, **kwargs): class ModelListCLI(_PathOverridesMixin): """List model names.""" __command__ = 'list' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -513,8 +602,8 @@ def main(cls, argv=True, **kwargs): class ModelShowCLI(_PathOverridesMixin): """Show a model entry, or all of them when no NAME is given.""" __command__ = 'show' - catalog = scfg.Value(None, type=str) - name = scfg.Value(None, position=1, type=str, + catalog = kw.Value(None, type=str) + name = kw.Value(None, position=1, type=str, help='Model name; omit to show every model.') @classmethod @@ -526,7 +615,7 @@ def main(cls, argv=True, **kwargs): class ModelRmCLI(_CatalogCommon): """Remove one or more models by name.""" __command__ = 'rm' - names = scfg.Value([], nargs='+', position=1, type=str, + names = kw.Value([], nargs='+', position=1, type=str, help='Model name(s) to remove.') @classmethod @@ -535,7 +624,7 @@ def main(cls, argv=True, **kwargs): return _rm(config, 'models', config.names) -class CatalogModelCLI(scfg.ModalCLI): +class CatalogModelCLI(kw.ModalCLI): """Manage catalog models.""" __command__ = 'model' add = ModelAddCLI @@ -562,55 +651,55 @@ class EndpointAddCLI(_CatalogCommon): """ __command__ = 'add' - name = scfg.Value( + name = kw.Value( None, position=1, type=str, help='Endpoint alias (default: {model}-N, auto-incrementing).', ) - engine = scfg.Value('vllm', choices=['vllm', 'ollama']) - model = scfg.Value(None, type=str, help='Model name (vllm) or tag (ollama).') - host = scfg.Value(None, type=str, help='Runtime host (ollama).') - public_name = scfg.Value( + engine = kw.Value('vllm', type=str, choices=['vllm', 'ollama']) + model = kw.Value(None, type=str, help='Model name (vllm) or tag (ollama).') + host = kw.Value(None, type=str, help='Runtime host (ollama).') + public_name = kw.Value( None, type=str, help='Served/public name (for coalescing aliases).' ) - reclaim = scfg.Value( - None, choices=['keep-warm', 'stop', 'scale-to-zero'], + reclaim = kw.Value( + None, type=str, choices=['keep-warm', 'stop', 'scale-to-zero'], help='Reclaim policy when idle.', ) - protocol = scfg.Value( - None, choices=['chat', 'completions'], + protocol = kw.Value( + None, type=str, choices=['chat', 'completions'], help='Which OpenAI surface this endpoint serves. Load-bearing twice: ' 'a base model has no chat template, and the readiness probe ' 'follows this — declaring chat for a completions-only serve ' 'blocks `acquire` until the TTL. Default (unset): chat.', ) - min_vram_gib = scfg.Value( + min_vram_gib = kw.Value( None, type=float, help='placement.min_vram_gib — the VRAM this endpoint needs, so the ' 'planner can pick any eligible free GPU. Declaring this is what ' 'lets one catalog be correct on every host.', ) - gpu = scfg.Value( + gpu = kw.Value( [], nargs='*', type=int, help='placement.gpu_indices — exact physical GPU index/indices. Omit ' 'for automatic VRAM-aware placement. This is a local operator ' 'override and is intentionally less portable than --min-vram-gib.', ) # vLLM runtime conveniences - max_model_len = scfg.Value(None, type=int) - gpu_mem = scfg.Value( + max_model_len = kw.Value(None, type=int) + gpu_mem = kw.Value( None, type=float, help='gpu_memory_utilization (0-1).' ) - tensor_parallel = scfg.Value(None, type=int) - extra_args = scfg.Value( + tensor_parallel = kw.Value(None, type=int) + extra_args = kw.Value( None, type=str, help="Raw vLLM flags as one string (shell-split), " "e.g. --extra-args='--dtype=half --enforce-eager'.", ) - runtime = scfg.Value( + runtime = kw.Value( [], nargs='*', type=str, help='Extra runtime KEY=VALUE pairs (YAML-typed).', ) - force = scfg.Value(False, isflag=True) + force = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -681,7 +770,7 @@ def main(cls, argv=True, **kwargs): class EndpointListCLI(_PathOverridesMixin): """List endpoint names.""" __command__ = 'list' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -691,8 +780,8 @@ def main(cls, argv=True, **kwargs): class EndpointShowCLI(_PathOverridesMixin): """Show an endpoint entry, or all of them when no NAME is given.""" __command__ = 'show' - catalog = scfg.Value(None, type=str) - name = scfg.Value(None, position=1, type=str, + catalog = kw.Value(None, type=str) + name = kw.Value(None, position=1, type=str, help='Endpoint name; omit to show every endpoint.') @classmethod @@ -704,7 +793,7 @@ def main(cls, argv=True, **kwargs): class EndpointRmCLI(_CatalogCommon): """Remove one or more endpoints by name.""" __command__ = 'rm' - names = scfg.Value([], nargs='+', position=1, type=str, + names = kw.Value([], nargs='+', position=1, type=str, help='Endpoint name(s) to remove.') @classmethod @@ -713,7 +802,7 @@ def main(cls, argv=True, **kwargs): return _rm(config, 'endpoints', config.names) -class CatalogEndpointCLI(scfg.ModalCLI): +class CatalogEndpointCLI(kw.ModalCLI): """Manage catalog endpoints.""" __command__ = 'endpoint' add = EndpointAddCLI @@ -731,15 +820,15 @@ class HostAddCLI(_CatalogCommon): """Add (or --force overwrite) a runtime host (e.g. an Ollama daemon).""" __command__ = 'add' - name = scfg.Value(None, position=1, type=str) - engine = scfg.Value('ollama', choices=['ollama']) - gpu = scfg.Value([], nargs='*', type=int, help='GPU index/indices.') - keep_alive = scfg.Value(None, type=str, help='Ollama keep_alive, e.g. 5m.') - num_parallel = scfg.Value(None, type=int) - max_loaded_models = scfg.Value(None, type=int) - context_length = scfg.Value(None, type=int) - image = scfg.Value(None, type=str) - force = scfg.Value(False, isflag=True) + name = kw.Value(None, position=1, type=str) + engine = kw.Value('ollama', type=str, choices=['ollama']) + gpu = kw.Value([], nargs='*', type=int, help='GPU index/indices.') + keep_alive = kw.Value(None, type=str, help='Ollama keep_alive, e.g. 5m.') + num_parallel = kw.Value(None, type=int) + max_loaded_models = kw.Value(None, type=int) + context_length = kw.Value(None, type=int) + image = kw.Value(None, type=str) + force = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -780,7 +869,7 @@ def main(cls, argv=True, **kwargs): class HostListCLI(_PathOverridesMixin): """List runtime-host names.""" __command__ = 'list' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -790,7 +879,7 @@ def main(cls, argv=True, **kwargs): class HostRmCLI(_CatalogCommon): """Remove one or more runtime hosts by name.""" __command__ = 'rm' - names = scfg.Value([], nargs='+', position=1, type=str, + names = kw.Value([], nargs='+', position=1, type=str, help='Runtime-host name(s) to remove.') @classmethod @@ -799,7 +888,7 @@ def main(cls, argv=True, **kwargs): return _rm(config, 'runtime_hosts', config.names) -class CatalogHostCLI(scfg.ModalCLI): +class CatalogHostCLI(kw.ModalCLI): """Manage runtime hosts (Ollama daemons / placement).""" __command__ = 'host' add = HostAddCLI @@ -816,9 +905,9 @@ class BundleAddCLI(_CatalogCommon): """Add (or --force overwrite) a bundle: a named group of endpoints.""" __command__ = 'add' - name = scfg.Value(None, position=1, type=str) - members = scfg.Value([], nargs='*', position=2, type=str) - force = scfg.Value(False, isflag=True) + name = kw.Value(None, position=1, type=str) + members = kw.Value([], nargs='*', position=2, type=str) + force = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -844,7 +933,7 @@ def main(cls, argv=True, **kwargs): class BundleListCLI(_PathOverridesMixin): """List bundle names.""" __command__ = 'list' - catalog = scfg.Value(None, type=str) + catalog = kw.Value(None, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -854,7 +943,7 @@ def main(cls, argv=True, **kwargs): class BundleRmCLI(_CatalogCommon): """Remove one or more bundles by name.""" __command__ = 'rm' - names = scfg.Value([], nargs='+', position=1, type=str, + names = kw.Value([], nargs='+', position=1, type=str, help='Bundle name(s) to remove.') @classmethod @@ -863,7 +952,7 @@ def main(cls, argv=True, **kwargs): return _rm(config, 'bundles', config.names) -class CatalogBundleCLI(scfg.ModalCLI): +class CatalogBundleCLI(kw.ModalCLI): """Manage endpoint bundles.""" __command__ = 'bundle' add = BundleAddCLI @@ -876,7 +965,7 @@ class CatalogBundleCLI(scfg.ModalCLI): # --------------------------------------------------------------------------- -class CatalogModalCLI(scfg.ModalCLI): +class CatalogModalCLI(kw.ModalCLI): """Edit the user serving catalog (models / endpoints / hosts / bundles).""" __command__ = 'catalog' diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py index f0d2d415..37d036a4 100644 --- a/infer_stack/cli/commands_leasing.py +++ b/infer_stack/cli/commands_leasing.py @@ -8,7 +8,7 @@ infer-stack run --endpoint qwen-coder -- python my_node.py infer-stack release --env-file is.env infer-stack acquire qwen-coder # standing service (no --ttl) - infer-stack leases # status of leases + deployment deployments + infer-stack leases # status of leases + deployments Until the Compose/KubeAI backends land, the default ``--backend null`` is a dry-run: the ledger does all the real bookkeeping (coalescing, demand, TTL) but @@ -26,7 +26,7 @@ from typing import Any from pathlib import Path -import scriptconfig as scfg +import kwconf as kw from ..env_utils import parse_env_file, write_env_file from ..leasing import ( @@ -277,6 +277,43 @@ def _make_backend(config, *, interactive: bool = False): if name == 'kubeai': from ..backends.kubeai import KubeaiBackend + def make_gateway(placement: str): + """The LiteLLM front door for a placement: 'host' or 'cluster'.""" + if placement == 'cluster': + from ..backends.kubeai import _default_kubectl_run + from ..backends.kubeai_gateway import DEFAULT_NODE_PORT, ClusterGateway + + return ClusterGateway( + state_dir=data_root() / 'leasing' / 'kubeai-cluster-gateway', + namespace=get_setting('kubeai_namespace') or 'kubeai', + run=_default_kubectl_run, + node_port=int(get_setting('kubeai_gateway_node_port') + or DEFAULT_NODE_PORT), + url=get_setting('kubeai_gateway_url') or None, + assume_yes=_resolve_assume_yes(config, interactive=interactive), + ) + # The same LiteLLM front door as the compose backend, fronting the + # cluster: one base_url, the managed key, and endpoint aliases as + # request names. Its own state dir and compose project, so it can + # never touch a compose stack's containers on the same host. It + # takes the same settings as on compose (UI, reverse proxy, + # dynamic routing); only the engines are elsewhere. + rp_enabled, rp_port, rp_config = _resolve_reverse_proxy(config) + return ComposeBackend( + state_dir=data_root() / 'leasing' / 'kubeai-gateway', + inventory={'gpu_count': 0, 'gpus': []}, + project='infer-stack-gateway', + litellm=True, + ui=_resolve_ui(config), + reverse_proxy=rp_enabled, + reverse_proxy_port=rp_port, + reverse_proxy_config=rp_config, + dynamic_routing=_resolve_dynamic_routing(config), + assume_yes=_resolve_assume_yes(config, interactive=interactive), + ) + + gateway = (make_gateway(get_setting('kubeai_gateway') or 'host') + if _resolve_litellm(config) else None) backend = KubeaiBackend( state_dir=data_root() / 'leasing' / 'kubeai', namespace=get_setting('kubeai_namespace') or 'kubeai', @@ -284,6 +321,9 @@ def _make_backend(config, *, interactive: bool = False): default_resource_profile=get_setting('kubeai_resource_profile') or None, assume_yes=_resolve_assume_yes(config, interactive=interactive), + gateway=gateway, + gateway_upstream=get_setting('kubeai_gateway_upstream') or None, + gateway_factory=make_gateway if gateway is not None else None, ) try: backend.catalog = _load_catalog(config) # frozen into the profile @@ -458,10 +498,17 @@ def _public_descriptor(descriptor: dict) -> dict: def _compose_file_path(controller) -> str | None: + """The compose file, on the compose backend only (JSON's ``compose_file``).""" path = getattr(controller.backend, 'compose_file', None) return str(path) if path else None +def _rendered_path(controller) -> str | None: + """The file a render writes, on any backend: compose project or Models.""" + path = getattr(controller.backend, 'rendered_file', None) + return str(path) if path else None + + def _gpu_where(gpus) -> str: """Human label for a deployment's GPU assignment (or its absence).""" if gpus is None: @@ -487,6 +534,7 @@ def _emit_staged(config, controller, outcome) -> int: 'applied': False, 'descriptor': _public_descriptor(descriptor), 'compose_file': _compose_file_path(controller), + 'rendered_file': _rendered_path(controller), 'placement': [ {'deployment': g.id, 'served': sorted(g.served), 'gpus': assignments.get(g.id)} @@ -495,13 +543,18 @@ def _emit_staged(config, controller, outcome) -> int: }, indent=2)) return 0 print(f'staged {outcome.lease.id} (owner={outcome.lease.owner}) — not applied') + from ..leasing.backend import allocates_gpus + + local = allocates_gpus(controller.backend) for g in outcome.deployments: eps = ', '.join(sorted(g.served)) or g.id - print(f' {eps}: {_gpu_where(assignments.get(g.id))} ({g.id})') - path = _compose_file_path(controller) - if path: - print(f' compose: {path}') + where = _gpu_where(assignments.get(g.id)) if local else 'cluster-scheduled' + print(f' {eps}: {where} ({g.id})') + rendered = _rendered_path(controller) + if rendered: + print(f' rendered: {rendered}') print(' apply: infer-stack apply # bring the staged set up') + path = _compose_file_path(controller) if path: # The file carries `name: infer-stack`, so plain docker works too. print(f' ...or: docker compose -f {path} up -d') @@ -695,11 +748,13 @@ def _do_acquire(config, *, owner: str, ttl_seconds: float | None) -> int: lines += [f' {r}' for r in ex.reasons] or [ f' {", ".join(ex.deployment_ids)}' ] - lines.append( - ' free a GPU first — `infer-stack leases` to see what holds them, ' - 'then `infer-stack release`/`evict`. (Every GPU, including any ' - 'display-attached one, is used unless you set --skip-display-gpus.)' - ) + if ex.capacity: + lines.append( + ' free a GPU first — `infer-stack leases` to see what holds them, ' + 'then `infer-stack release`/`evict`, or wait for one with --queue. ' + '(Every GPU, including any display-attached one, is used unless ' + 'you set --skip-display-gpus.)' + ) raise SystemExit('\n'.join(lines)) return _emit_acquire(config, controller, outcome) @@ -710,39 +765,39 @@ def _do_acquire(config, *, owner: str, ttl_seconds: float | None) -> int: class _LeasingCommonMixin(_PathOverridesMixin, _AllowedGpusMixin, _DisplayGpuMixin): - backend = scfg.Value( + backend = kw.Value( None, - choices=['null', 'compose', 'kubeai'], + type=str, choices=['null', 'compose', 'kubeai'], help='Serving backend: "null" (dry-run), "compose" (single-host ' 'docker), or "kubeai" (cluster; see docs/kubeai-backend.md). ' 'Defaults to `config set backend …`, else "null".', ) - ledger = scfg.Value( + ledger = kw.Value( None, type=str, help='Path to the lease ledger sqlite db.' ) - require_generation = scfg.Value( + require_generation = kw.Value( False, isflag=True, help='Deprecated/no-op: readiness now ALWAYS verifies a real generation ' '(a listed alias or a running container is not proof the model serves). ' 'Accepted for compatibility.', ) - litellm = scfg.Value( + litellm = kw.Value( None, isflag=True, help='Render the LiteLLM gateway — one OpenAI base_url fronting every ' - 'endpoint alias (compose backend). On by default; use --no-litellm for ' + 'endpoint alias. On by default; use --no-litellm for ' 'a lean stack where Open WebUI talks to the upstreams (e.g. an Ollama ' 'daemon) directly. Overrides `config set litellm …`.', ) - ui = scfg.Value( + ui = kw.Value( None, isflag=True, - help='Render a managed Open WebUI in front of the gateway (compose ' - 'backend). On by default; use --no-ui to skip. Overrides ' - '`config set ui …`.', + help='Render a managed Open WebUI in front of the gateway (on this host; ' + 'not with kubeai_gateway cluster). On by default; use --no-ui to skip. ' + 'Overrides `config set ui …`.', ) - reverse_proxy = scfg.Value( + reverse_proxy = kw.Value( None, isflag=True, alias=['reverse-proxy'], @@ -751,7 +806,7 @@ class _LeasingCommonMixin(_PathOverridesMixin, _AllowedGpusMixin, _DisplayGpuMix '(localhost / trusted networks only). Port + bring-your-own nginx.conf ' 'live in the `reverse_proxy` setting (`config set` / `config edit`).', ) - dynamic_routing = scfg.Value( + dynamic_routing = kw.Value( None, isflag=True, alias=['dynamic-routing'], @@ -771,63 +826,66 @@ class _ApprovalMixin(_LeasingCommonMixin): terminal; ``--yes`` (or a non-TTY) applies without prompting. """ - catalog = scfg.Value( + catalog = kw.Value( None, type=str, help='Path to catalog.yaml. release/gc/evict reconcile the gateway too, ' 'so pass the same catalog as acquire to keep the static superset route ' 'table (no gateway blip); omitted, it falls back to the default-path ' 'catalog, else legacy per-deployment routing.', ) - yes = scfg.Value( + yes = kw.Value( False, isflag=True, alias=['y'], - help='Apply compose changes without showing the diff / prompting ' - '(compose backend). Implied when stdout is not a terminal.', + help='Apply the rendered changes (the compose project, or the KubeAI ' + 'Models and gateway) without showing the diff / prompting. Implied when ' + 'stdout is not a terminal.', ) class _AcquireFlagsMixin(_LeasingCommonMixin): - catalog = scfg.Value(None, type=str, help='Path to catalog.yaml.') - base_url = scfg.Value( + catalog = kw.Value(None, type=str, help='Path to catalog.yaml.') + base_url = kw.Value( 'http://127.0.0.1:14042/v1', type=str, help='Base URL written into the endpoint descriptor (dry-run placeholder).', ) - api_key_env = scfg.Value( + api_key_env = kw.Value( 'LITELLM_MASTER_KEY', type=str, help='Name of the env var holding the API key (kept out of artifacts).', ) - wait = scfg.Value( + wait = kw.Value( True, isflag=True, help='Block until ready (use --no-wait to skip).' ) - queue = scfg.Value( + queue = kw.Value( False, isflag=True, help='Admission queue: if every GPU is busy, WAIT for one to free ' '(up to --timeout) instead of failing fast. Each retry sweeps the ' 'ledger, so a crashed job\'s TTL-expired lease is reclaimed while ' 'waiting. Intended for batch/pipeline fan-out; interactive use ' - 'defaults off (fail fast with a clear "no GPU" error).', + 'defaults off (fail fast with a clear "no GPU" error). On kubeai the ' + 'cluster is the queue: admitted at once, and a Pending pod waits.', ) - apply = scfg.Value( + apply = kw.Value( True, isflag=True, - help='Apply the render (docker compose up). Use --no-apply to *stage* ' - 'only: declare the lease and write the on-disk compose project + ' - 'placement WITHOUT starting it, then `infer-stack apply` to bring it up ' - '(compose backend). --no-apply implies no readiness wait and no diff ' - 'prompt; `release` discards a staged lease.', + help='Apply the render (bring it up). Use --no-apply to *stage* only: ' + 'declare the lease and write the rendered state (the compose project, ' + 'or the KubeAI Models) WITHOUT starting it, then `infer-stack apply` to ' + 'bring it up. --no-apply implies no readiness wait and no diff prompt; ' + '`release` discards a staged lease.', ) - timeout = scfg.Value(600, type=float, help='Readiness wait timeout (s).') - interval = scfg.Value(5, type=float, help='Readiness poll interval (s).') - env_file = scfg.Value( + timeout = kw.Value(600, type=float, help='Readiness wait timeout (s).') + interval = kw.Value(5, type=float, help='Readiness poll interval (s).') + env_file = kw.Value( None, type=str, help='Write the sourceable endpoint env-file here.' ) - yes = scfg.Value( + yes = kw.Value( False, isflag=True, alias=['y'], - help='Apply compose changes without showing the diff / prompting ' - '(compose backend). Implied when stdout is not a terminal.', + help='Apply the rendered changes (the compose project, or the KubeAI ' + 'Models and gateway) without showing the diff / prompting. Implied when ' + 'stdout is not a terminal.', ) - json = scfg.Value(False, isflag=True, help='Emit JSON instead of text.') + json = kw.Value(False, isflag=True, help='Emit JSON instead of text.') # --------------------------------------------------------------------------- @@ -839,11 +897,11 @@ class AcquireCLI(_AcquireFlagsMixin): """Acquire a lease on one or more endpoints/bundles, bring them up, wait. ``acquire NAME…`` is the everyday verb. It takes a lease on each endpoint or - bundle, renders the compose project (the LiteLLM gateway + one container per - model + a managed Open WebUI), brings it up, and blocks until every endpoint - is ready. Run it again with more names to add models side by side — the - gateway and UI stay put. Placement, ``docker compose``, and readiness are - narrated on stderr. + bundle, renders what serves it (the LiteLLM gateway, one engine per model: + a container on compose, a Model on kubeai, and a managed Open WebUI), + brings it up, and blocks until every endpoint is ready. Run it again with + more names to add models side by side — the gateway and UI stay put. + Placement, the apply, and readiness are narrated on stderr. With no ``--ttl`` the lease is infinite — a standing service you tear down explicitly (``release`` / ``evict``). Pass ``--ttl`` (e.g. ``2h``, ``30m``) @@ -854,13 +912,13 @@ class AcquireCLI(_AcquireFlagsMixin): when done. (For a one-shot "acquire, run a command, release", use ``infer-stack run`` instead.) - The work is render (write the on-disk compose project) then apply (``docker - compose up``). ``--no-apply`` does just the render so you can see what - would run before pulling the trigger (then ``infer-stack apply``). - ``--no-wait`` applies but returns immediately so several models load in - parallel (``wait`` for them later). ``--no-ui`` skips Open WebUI. On a - terminal you are shown the compose diff and asked before applying; ``--yes`` - skips that prompt (and it is skipped automatically off a TTY). + The work is render (write the rendered state to disk) then apply (bring + it up). ``--no-apply`` does just the render so you can see what would run + before pulling the trigger (then ``infer-stack apply``). ``--no-wait`` + applies but returns immediately so several models load in parallel + (``wait`` for them later). ``--no-ui`` skips Open WebUI. On a terminal you + are shown the diff and asked before anything is committed; ``--yes`` skips + that prompt (and it is skipped automatically off a TTY). """ __command__ = 'acquire' @@ -885,19 +943,19 @@ class AcquireCLI(_AcquireFlagsMixin): infer-stack leases """ - names = scfg.Value( + names = kw.Value( [], nargs='*', position=1, type=str, help='Endpoint or bundle names.' ) - ttl = scfg.Value( + ttl = kw.Value( None, type=str, help='Soft TTL (e.g. 2h, 30m); default infinite.' ) - owner = scfg.Value(None, type=str, help='Lease owner (default: $USER).') - dedicated = scfg.Value( + owner = kw.Value(None, type=str, help='Lease owner (default: $USER).') + dedicated = kw.Value( False, isflag=True, help='Force a dedicated deployment instead of coalescing.', ) - reserve_gpus = scfg.Value( + reserve_gpus = kw.Value( 0, type=int, alias=['reserve-gpus'], @@ -922,10 +980,11 @@ def main(cls, argv=True, **kwargs): class RenderCLI(_LeasingCommonMixin): - """Write the on-disk compose project for the current desired set — no up. + """Write the rendered state for the current desired set, without bringing it up. - Lease-free and idempotent: ``render`` re-materializes the manifest (compose - file + gateway config + GPU placement) from whatever is *already* declared, + Lease-free and idempotent: ``render`` re-materializes what would run (the + compose project and gateway config with GPU placement, or the KubeAI Models + and gateway) from whatever is *already* declared, WITHOUT starting anything — to inspect what would run, or refresh a file you touched by hand. It creates no lease; to stage a *new* endpoint use ``acquire --no-apply`` (which declares it too). Apply with ``infer-stack apply``. @@ -933,23 +992,24 @@ class RenderCLI(_LeasingCommonMixin): __command__ = 'render' - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) controller = _open_controller(config, interactive=False) rec = controller.reconcile(apply=False) - path = _compose_file_path(controller) + path = _rendered_path(controller) if config.json: print(json.dumps({ 'applied': False, - 'compose_file': path, + 'compose_file': _compose_file_path(controller), + 'rendered_file': path, 'placement': rec.assignments, 'unplaced': rec.unplaced, }, indent=2)) return 0 - print(f'rendered desired set -> {path or "(backend has no on-disk project)"}') + print(f'rendered desired set -> {path or "(this backend renders no file)"}') for gid, gpus in sorted(rec.assignments.items()): print(f' {gid}: {_gpu_where(gpus)}') for err in rec.placement_errors: @@ -965,21 +1025,20 @@ class ApplyCLI(_ApprovalMixin): declared (by ``acquire``) onto the backend. It is the *trigger* for a staged ``acquire --no-apply`` and the re-sync button after a manual edit or a backend hiccup; idempotent (a second apply with nothing changed is a - no-op). On a terminal it shows the compose diff and asks (``--yes`` skips). - The rendered file carries ``name: infer-stack``, so ``docker compose -f - up -d`` is an exact equivalent. (``infer-stack stack up`` is the - lower-level "run exactly what is on disk" hatch; ``apply`` re-renders from - intent first.) + no-op). On a terminal it shows the diff and asks (``--yes`` skips). + ``infer-stack stack up`` is the same command. On compose, ``infer-stack + stack compose -- up -d`` runs exactly what is on disk without re-rendering + from intent first. """ __command__ = 'apply' - wait = scfg.Value( + wait = kw.Value( False, isflag=True, help='Also block until ready after bringing it up.' ) - timeout = scfg.Value(600, type=float, help='Readiness wait timeout (s).') - interval = scfg.Value(5, type=float, help='Readiness poll interval (s).') - json = scfg.Value(False, isflag=True) + timeout = kw.Value(600, type=float, help='Readiness wait timeout (s).') + interval = kw.Value(5, type=float, help='Readiness poll interval (s).') + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1059,22 +1118,22 @@ class ReleaseCLI(_ApprovalMixin): __command__ = 'release' - lease = scfg.Value( + lease = kw.Value( None, position=1, type=str, help='Lease id (or use --env-file).' ) - env_file = scfg.Value( + env_file = kw.Value( None, type=str, help='Read the lease id from this env-file.' ) - all = scfg.Value( + all = kw.Value( False, isflag=True, help='Release every active lease (the whole stack idles/tears down).', ) - evict = scfg.Value( + evict = kw.Value( False, isflag=True, help='Also evict (tear down) the released deployment(s) now, even if their ' 'reclaim policy is keep-warm — frees the GPU immediately.', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1167,12 +1226,12 @@ class EvictCLI(_ApprovalMixin): __command__ = 'evict' - names = scfg.Value( + names = kw.Value( [], nargs='*', position=1, type=str, help='Endpoint alias or deployment id to evict.', ) - all = scfg.Value(False, isflag=True, help='Evict every idle deployment.') - json = scfg.Value(False, isflag=True) + all = kw.Value(False, isflag=True, help='Evict every idle deployment.') + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1192,17 +1251,22 @@ def main(cls, argv=True, **kwargs): if missing: # Diagnostic, not payload: stdout must stay pure JSON # under --json (and stay grep-able human output without). - print( - f'no idle deployment for: {", ".join(missing)}', - file=sys.stderr, - ) + _, rows = controller.ledger.status(virtual_expiry=True) + held = {name for g in rows if g.state == DeploymentState.LIVE + for name in (g.id, *g.served)} + for name in missing: + why = ('is live: a lease holds it; release it first' + if name in held else + 'is not a deployment or served endpoint ' + '(`infer-stack leases` lists them)') + print(f'evict: {name} {why}', file=sys.stderr) if not targets: if config.json: print(json.dumps( {'evicted': [], 'torn_down': [], 'missing': missing}, indent=2)) else: - print('nothing to evict') + print('nothing evicted') return 0 outcome = controller.evict(targets) except ConvergeAborted: @@ -1232,21 +1296,27 @@ class GcCLI(_ApprovalMixin): blocking ``acquire`` (``--queue``) already does this implicitly while it waits. ``--evict`` additionally tears down idle *keep-warm* deployments (like ``evict --all``). On a terminal the teardown is shown and confirmed (``--yes`` skips). + ``--forget`` only drops finished rows from the ledger (the TUI's Clean up). """ __command__ = 'gc' - evict = scfg.Value( + evict = kw.Value( False, isflag=True, help='Also tear down idle keep-warm deployments (like `evict --all`), ' 'not just leaked/expired demand.', ) - orphans = scfg.Value( + orphans = kw.Value( False, isflag=True, help='Instead: remove containers in the project that infer-stack does not ' 'manage (listed and confirmed first; --yes skips the prompt).', ) - json = scfg.Value(False, isflag=True) + forget = kw.Value( + False, isflag=True, + help='Instead: forget released/expired leases and stopped deployments ' + 'from the ledger. History only; nothing running changes.', + ) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1254,9 +1324,18 @@ def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) controller = _open_controller(config, interactive=True) + if config.forget: + n_leases, n_deployments = controller.prune() + if config.json: + print(json.dumps({'leases': n_leases, + 'deployments': n_deployments}, indent=2)) + else: + print(f'gc --forget: forgot {n_leases} released/expired lease(s), ' + f'{n_deployments} stopped deployment(s)') + return 0 if config.orphans: - if not callable(getattr(controller.backend, 'residency', None)): - raise SystemExit('gc --orphans needs the compose backend') + # Unlabelled units in residency. The kubeai backend reads only the + # pods carrying its label, so it never has any. def confirm(found): print(f'gc --orphans: {len(found)} unmanaged container(s):') @@ -1321,16 +1400,16 @@ class CleanCLI(_LeasingCommonMixin): infer-stack clean -f --no-orphans # leave unmanaged containers alone """ - force = scfg.Value( + force = kw.Value( False, isflag=True, short_alias=['f'], help='Actually release and tear down. Without it, clean only reports.', ) - orphans = scfg.Value( + orphans = kw.Value( True, isflag=True, help='Also remove containers in the project that infer-stack does not ' 'manage (--no-orphans keeps them).', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1344,8 +1423,8 @@ def main(cls, argv=True, **kwargs): held = [g for g in deployments if g.state in (DeploymentState.LIVE, DeploymentState.IDLE)] observed, assignments = _placement_view(controller) - can_orphan = bool(config.orphans) and callable( - getattr(controller.backend, 'residency', None)) + # On any backend: one that selects its units by label (kubeai) has none. + can_orphan = bool(config.orphans) found: list = [] @@ -1378,10 +1457,9 @@ def _list_only(orphans): print(f' release {le.id} owner={le.owner} ' f'{",".join(le.endpoints)}') for g in held: - gpus = assignments.get(g.id) print(f' tear down {g.id} {g.state}' f'{" running" if g.id in observed else ""}' - f' gpus={gpus if gpus is not None else "-"}' + f' gpus={_gpu_label(g.id, observed, assignments)}' f' {",".join(sorted(g.served))}') for c in found: print(f' remove {c.container_id[:12]} {c.service or "?"} ' @@ -1417,8 +1495,7 @@ def _list_only(orphans): class WaitCLI(_LeasingCommonMixin): - """Block until served endpoints are ready — the companion to ``acquire - --no-wait``. + """Block until served endpoints are ready (the companion to ``acquire --no-wait``). Fan out, then wait: ``acquire --no-wait smol17b-1`` + ``acquire --no-wait smol135-1`` kick both deployments off in parallel (each converges and starts @@ -1432,13 +1509,13 @@ class WaitCLI(_LeasingCommonMixin): __command__ = 'wait' - names = scfg.Value( + names = kw.Value( [], nargs='*', position=1, type=str, help='Endpoint names to wait for (default: every live deployment).', ) - timeout = scfg.Value(600, type=float, help='Overall wait timeout (s).') - interval = scfg.Value(5, type=float, help='Readiness poll interval (s).') - json = scfg.Value(False, isflag=True) + timeout = kw.Value(600, type=float, help='Overall wait timeout (s).') + interval = kw.Value(5, type=float, help='Readiness poll interval (s).') + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1487,8 +1564,10 @@ def main(cls, argv=True, **kwargs): class MeasureCLI(_LeasingCommonMixin): - """Measure an endpoint's real per-GPU VRAM requirement from the engine's - own memory-profiling log (docs/planning/vram-aware-placement.md §3). + """Measure an endpoint's real per-GPU VRAM requirement from its engine's log. + + Reads the engine's own memory-profiling log + (docs/planning/vram-aware-placement.md §3). Parses vLLM's profiling breakdown (weights + non-torch + activation peak) — deliberately NOT ``nvidia-smi memory.used``, which only reflects the @@ -1507,35 +1586,35 @@ class MeasureCLI(_LeasingCommonMixin): __command__ = 'measure' - endpoint = scfg.Value( + endpoint = kw.Value( None, position=1, required=True, type=str, help='Catalog endpoint to measure.', ) - record = scfg.Value( + record = kw.Value( False, isflag=True, help='Record the result into the measurements overlay ' '(consulted automatically at plan time when the catalog declares ' 'nothing for this endpoint).', ) - kv_gib = scfg.Value( + kv_gib = kw.Value( 2.0, type=float, help='KV-cache budget (GiB) added on top of the non-KV profile. ' 'A serving choice (max_model_len / max_num_seqs), not a model fact.', ) - margin = scfg.Value( + margin = kw.Value( 0.05, type=float, help='Safety-margin fraction over the non-KV profile ' '(allocator fragmentation, engine drift).', ) - timeout = scfg.Value( + timeout = kw.Value( 900, type=float, help='Readiness timeout when the endpoint must be brought up first (s).', ) - catalog = scfg.Value( + catalog = kw.Value( None, type=str, help='Catalog path (default: /catalog.yaml).', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -1552,8 +1631,8 @@ def main(cls, argv=True, **kwargs): logs_fn = getattr(backend, 'deployment_logs', None) if logs_fn is None: raise SystemExit( - 'measure needs the compose backend ' - '(the engine container log is the measurement source).' + 'measure reads the engine\'s log, and this backend runs no engine ' + '(set `--backend compose` or `kubeai`).' ) catalog = _requests_catalog(controller, config) name = config.endpoint @@ -1627,8 +1706,7 @@ def main(cls, argv=True, **kwargs): store = getattr(backend, 'measurements', None) if store is None: raise SystemExit( - '--record needs the compose backend measurements ' - 'overlay.' + '--record: this backend keeps no measurements overlay.' ) store.record( key, value, endpoint=name, profile=profile, @@ -1673,8 +1751,9 @@ def main(cls, argv=True, **kwargs): class TuiCLI(_LeasingCommonMixin): - """Launch the Textual TUI: a live monitor of the stack with controls to - serve / release / evict models. + """Launch the Textual TUI: a live monitor of the stack, with controls. + + Serve, release and evict models, and follow their logs. Mostly a monitor — the lease + deployment tables (desired state vs running, GPUs) refresh live — with key-bound controls: ``s`` serve, ``d`` release, ``a`` @@ -1684,8 +1763,14 @@ class TuiCLI(_LeasingCommonMixin): __command__ = 'tui' - catalog = scfg.Value(None, type=str, help='Path to catalog.yaml.') - interval = scfg.Value(3.0, type=float, help='Auto-refresh interval (s).') + catalog = kw.Value(None, type=str, help='Path to catalog.yaml.') + interval = kw.Value(3.0, type=float, help='Auto-refresh interval (s).') + exit_after_paint = kw.Value( + False, isflag=True, + help='Quit as soon as the first frame is drawn, then print when that ' + 'was (for measuring startup: `time infer-stack tui ' + '--exit_after_paint`).', + ) @classmethod def main(cls, argv=True, **kwargs): @@ -1714,6 +1799,7 @@ def main(cls, argv=True, **kwargs): return run_tui( controller, catalog, interval=float(config.interval), catalog_path=str(catalog_path), + exit_after_paint=bool(config.exit_after_paint), ) @@ -1722,9 +1808,9 @@ class RenewCLI(_LeasingCommonMixin): __command__ = 'renew' - lease = scfg.Value(None, position=1, type=str, help='Lease id.') - env_file = scfg.Value(None, type=str) - ttl = scfg.Value(None, type=str, help='New soft TTL (e.g. 2h); empty=infinite.') + lease = kw.Value(None, position=1, type=str, help='Lease id.') + env_file = kw.Value(None, type=str) + ttl = kw.Value(None, type=str, help='New soft TTL (e.g. 2h); empty=infinite.') @classmethod def main(cls, argv=True, **kwargs): @@ -1768,26 +1854,26 @@ class RunCLI(_LeasingCommonMixin): __command__ = 'run' - catalog = scfg.Value(None, type=str, help='Path to catalog.yaml.') - endpoint = scfg.Value( + catalog = kw.Value(None, type=str, help='Path to catalog.yaml.') + endpoint = kw.Value( None, type=str, alias=['endpoints'], help='Comma-separated endpoint or bundle names.', ) - base_url = scfg.Value('http://127.0.0.1:14042/v1', type=str) - api_key_env = scfg.Value('LITELLM_MASTER_KEY', type=str) - owner = scfg.Value(None, type=str) - ttl = scfg.Value('2h', type=str, help='Soft TTL backstop (default 2h).') - timeout = scfg.Value(600, type=float) - interval = scfg.Value(5, type=float) - queue = scfg.Value( + base_url = kw.Value('http://127.0.0.1:14042/v1', type=str) + api_key_env = kw.Value('LITELLM_MASTER_KEY', type=str) + owner = kw.Value(None, type=str) + ttl = kw.Value('2h', type=str, help='Soft TTL backstop (default 2h).') + timeout = kw.Value(600, type=float) + interval = kw.Value(5, type=float) + queue = kw.Value( False, isflag=True, help='Admission queue: wait (up to --timeout) for a GPU to free ' 'instead of failing fast when the fleet is full. Recommended for ' 'pipeline fan-out, where many jobs contend for a few GPUs.', ) - command = scfg.Value( + command = kw.Value( [], nargs='*', position=1, type=str, help='Command to run (after --).' ) @@ -1841,8 +1927,26 @@ def main(cls, argv=True, **kwargs): controller.release(outcome.lease.id) -def _lease_ttl(le) -> str: - return 'inf' if le.expires_at is None else f'@{le.expires_at:.0f}' +def _lease_ttl(le, now: float | None = None) -> str: + """Time left on a lease, for a person: ``1h59m``, ``12m``, ``expired``. + + >>> from types import SimpleNamespace + >>> [_lease_ttl(SimpleNamespace(expires_at=e), now=1000.0) + ... for e in (None, 8140.0, 1720.0, 1045.0, 900.0)] + ['inf', '1h59m', '12m', '45s', 'expired'] + """ + import time + + if le.expires_at is None: + return 'inf' + left = le.expires_at - (time.time() if now is None else now) + if left <= 0: + return 'expired' + if left >= 3600: + return f'{int(left // 3600)}h{int(left % 3600 // 60):02d}m' + if left >= 60: + return f'{int(left // 60)}m' + return f'{int(left)}s' def _placement_view(controller): @@ -1860,28 +1964,23 @@ def _placement_view(controller): except Exception: # noqa: BLE001 - status must never crash pass assignments: dict[str, list[int]] = {} - if controller._admission_mode(): - # Committed allocations, and idle residents' physical GPUs; the - # legacy planner view would show placements admission would not make. - _, deployments = controller.ledger.status(virtual_expiry=True) + from ..leasing.backend import allocates_gpus + + if not allocates_gpus(backend): + return observed, assignments # the cluster places; no GPU indices + # Committed allocations, and idle residents' physical GPUs. + _, deployments = controller.ledger.status(virtual_expiry=True) + for g in deployments: + if g.assigned_gpus is not None: + assignments[g.id] = list(g.assigned_gpus) + try: + residency = backend.residency() for g in deployments: - if g.assigned_gpus is not None: - assignments[g.id] = list(g.assigned_gpus) - try: - residency = backend.residency() - for g in deployments: - c = residency.resident(g.id) - if g.id not in assignments and c is not None: - assignments[g.id] = list(c.gpus) - except Exception: # noqa: BLE001 - pass - return observed, assignments - plan = getattr(backend, 'plan', None) - if plan is not None: - try: - assignments = dict(plan(controller.desired_deployments()).assignments) - except Exception: # noqa: BLE001 - pass + c = residency.resident(g.id) + if g.id not in assignments and c is not None: + assignments[g.id] = list(c.gpus) + except Exception: # noqa: BLE001 + pass return observed, assignments @@ -1978,7 +2077,7 @@ def state_style(state) -> str: class LeasesCLI(_LeasingCommonMixin): - """Show current leases and deployment deployments (the leasing-model status). + """Show current leases and deployments (the leasing-model status). Two tables. **leases** are who asked for what (id, owner, state, ttl, endpoints). **deployments** are the actual deployments behind them — one deployment is @@ -2005,7 +2104,7 @@ class LeasesCLI(_LeasingCommonMixin): infer-stack leases --json # JSON (adds running + gpus per deployment) """ - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -2094,9 +2193,29 @@ def _postgres_initialised() -> bool: return path.is_dir() and any(path.iterdir()) -def _secret_env_path() -> Path: - """The managed compose secrets file (.env that docker compose auto-loads).""" - return data_root() / 'leasing' / 'compose' / '.env' +def _gateway_state(config) -> tuple[Path, str]: + """``(managed .env, base URL)`` of the configured backend's gateway. + + The compose stack's own; on kubeai the gateway's, on this host or in the + cluster (whose base URL is a node's NodePort). A backend with no gateway + falls back to the compose location and the default port. One place, so + `env`, `test` and a script built from `env` cannot disagree. + """ + from ..config import DEFAULT_PORTS + + try: + front = getattr(_make_backend(config), 'front_door', lambda: None)() + except Exception: # noqa: BLE001 - a lookup must not fail on the backend + front = None + if front is not None and getattr(front, 'litellm', False): + return front.gateway._env_path, f'{front.gateway._gateway_base()}/v1' + return (data_root() / 'leasing' / 'compose' / '.env', + f'http://127.0.0.1:{DEFAULT_PORTS["litellm"]}/v1') + + +def _secret_env_path(config=None) -> Path: + """The gateway's managed secrets file (see :func:`_gateway_state`).""" + return _gateway_state(config)[0] def _front_door(config) -> tuple[str, str | None]: @@ -2106,20 +2225,23 @@ def _front_door(config) -> tuple[str, str | None]: `test` is cheap and doesn't need GPU detection or a backend object. An explicit ``--base-url`` overrides the derived URL. """ - from ..config import DEFAULT_PORTS + from urllib.parse import urlparse + env_path, derived = _gateway_state(config) base_url = getattr(config, 'base_url', None) if not base_url: - port = int(getattr(config, 'port', None) or DEFAULT_PORTS['litellm']) - base_url = f'http://127.0.0.1:{port}/v1' + base_url = derived + port = getattr(config, 'port', None) + if port: + base_url = urlparse(derived)._replace( + netloc=f'{urlparse(derived).hostname}:{int(port)}').geturl() key = None - env_path = _secret_env_path() if env_path.exists(): key = parse_env_file(env_path).get('LITELLM_MASTER_KEY') return base_url.rstrip('/'), key -def _front_door_env(stored: dict[str, str]) -> dict[str, str]: +def _front_door_env(stored: dict[str, str], config=None) -> dict[str, str]: """The front-door values ``env`` answers without them being stored. A base URL is not a secret and has nothing to be read *out* of: it is @@ -2144,7 +2266,7 @@ def _front_door_env(stored: dict[str, str]) -> dict[str, str]: base_url = stored.get('OPENAI_BASE_URL') if not base_url: - base_url, _ = _front_door(None) + base_url, _ = _front_door(config) entries = {'OPENAI_BASE_URL': base_url} port = urlparse(base_url).port if port is not None: @@ -2163,27 +2285,27 @@ class TestCLI(_PathOverridesMixin): __command__ = 'test' - catalog = scfg.Value( + catalog = kw.Value( None, type=str, help='Catalog path, used only to look up the endpoint protocol.', ) - name = scfg.Value( + name = kw.Value( None, position=1, type=str, help='Endpoint alias to test (e.g. chat).' ) - prompt = scfg.Value( + prompt = kw.Value( 'Reply with the single word: ready.', type=str, help='Prompt to send.' ) - max_tokens = scfg.Value(32, type=int) - timeout = scfg.Value(60, type=float, help='Request timeout (s).') - base_url = scfg.Value( + max_tokens = kw.Value(32, type=int) + timeout = kw.Value(60, type=float, help='Request timeout (s).') + base_url = kw.Value( None, type=str, help='Override the gateway base URL (…/v1).' ) - port = scfg.Value( + port = kw.Value( None, type=int, help='Override the gateway port (default: 14042).' ) - json = scfg.Value(False, isflag=True, help='Emit JSON instead of text.') - protocol = scfg.Value( - None, choices=['chat', 'completions'], + json = kw.Value(False, isflag=True, help='Emit JSON instead of text.') + protocol = kw.Value( + None, type=str, choices=['chat', 'completions'], help="Which surface to hit. Default: the endpoint's declared " '`protocol` from the catalog, falling back to chat.', ) @@ -2232,6 +2354,12 @@ def main(cls, argv=True, **kwargs): json=payload, timeout=float(config.timeout), ) + except requests.exceptions.ConnectionError: + return _test_fail(config, base_url, + 'nothing is listening there (the gateway is not up)') + except requests.exceptions.Timeout: + return _test_fail(config, base_url, + f'no answer within {float(config.timeout):g}s') except requests.exceptions.RequestException as ex: return _test_fail(config, base_url, f'not reachable: {ex}') dt = time.monotonic() - t0 @@ -2325,11 +2453,11 @@ class EnvCLI(_PathOverridesMixin): __command__ = 'env' - arg = scfg.Value( + arg = kw.Value( None, position=1, type=str, help='KEY to read its value, or KEY=VALUE to set it. Empty = path.', ) - export = scfg.Value( + export = kw.Value( False, isflag=True, help='Print every entry as `export KEY=value`.' ) @@ -2337,7 +2465,7 @@ class EnvCLI(_PathOverridesMixin): def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) _apply_path_overrides(config) - env_path = _secret_env_path() + env_path = _secret_env_path(config) # Write: `env KEY=VALUE` if config.arg and '=' in config.arg: @@ -2368,7 +2496,7 @@ def main(cls, argv=True, **kwargs): 'the gateway out of its database' ) if key == 'LITELLM_MASTER_KEY': - from ..leasing.compose import set_master_key + from ..leasing.gateway import set_master_key try: # Pins the salt first, so DB-stored routes stay readable. set_master_key(env_path, value) @@ -2387,7 +2515,7 @@ def main(cls, argv=True, **kwargs): # Read: `env KEY` / `env --export`. A stored value always beats the # derived one -- writing the key is how you override the front door. env = parse_env_file(env_path) if env_path.exists() else {} - derived = _front_door_env(env) + derived = _front_door_env(env, config) if config.arg: if config.arg in env: print(env[config.arg]) @@ -2419,17 +2547,18 @@ def main(cls, argv=True, **kwargs): def _require_compose_backend(controller): - """The controller's ComposeBackend, or a SystemExit for other backends. + """What holds the gateway's route registry (its front door). - The route registry is a compose-backend concept (it feeds the static-superset - LiteLLM gateway); ``--backend null``/``kubeai`` have no registry to touch.""" - backend = controller.backend - if not isinstance(backend, ComposeBackend): + The stack itself on the compose backend; on kubeai the gateway, on this + host or in the cluster. A SystemExit for a backend with no gateway (null, + or ``litellm false``).""" + project = getattr(controller.backend, 'front_door', lambda: None)() + if project is None or not getattr(project, 'litellm', False): raise SystemExit( - 'the `routes` commands require the compose backend ' - '(set `--backend compose` or `config set backend compose`)' + 'the `routes` commands need a LiteLLM gateway (the compose or kubeai ' + 'backend, with `litellm` on)' ) - return backend + return project def _live_endpoints(controller) -> set[str]: @@ -2447,7 +2576,8 @@ class RoutesListCLI(_LeasingCommonMixin): """Print the accumulated LiteLLM route registry (static-superset mode). One row per persisted route: its alias, engine, served-model/tag, the - upstream compose service it derives, and whether a live deployment is + upstream it routes to (a compose service, or a KubeAI cluster's gateway + under the Model's name), and whether a live deployment is currently backing it. Routes with no live backer still list (that is the point — a released endpoint stays routable/testable); their upstream simply errors until something serves it. @@ -2455,21 +2585,16 @@ class RoutesListCLI(_LeasingCommonMixin): __command__ = 'list' - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): - from ..leasing.compose import ( - OLLAMA_CONTAINER_PORT, - VLLM_CONTAINER_PORT, - ollama_service_name_for, - vllm_service_name_for, - ) + from ..leasing.gateway import registry_route_entry config = cls.cli(argv=argv, data=kwargs) controller = _open_controller(config) backend = _require_compose_backend(controller) - registry = backend._load_route_registry() + registry = backend.gateway._load_route_registry() entries = registry.get('entries', {}) live = _live_endpoints(controller) @@ -2477,22 +2602,13 @@ def main(cls, argv=True, **kwargs): for name in sorted(entries): row = entries[name] engine = row.get('engine') - if engine == 'vllm': - served = row.get('served') or name - upstream = ( - f'http://{vllm_service_name_for(served)}:' - f'{VLLM_CONTAINER_PORT}/v1' - ) - target = served - elif engine == 'ollama': - target = row.get('model') or name - host = row.get('host') or name - upstream = ( - f'http://{ollama_service_name_for(host)}:' - f'{OLLAMA_CONTAINER_PORT}' - ) - else: + entry = registry_route_entry(name, row) # what the gateway renders + if entry is None: target, upstream = '?', '?' + else: + params = entry['litellm_params'] + target = params['model'].split('/', 1)[-1] + upstream = params['api_base'] rows.append({ 'name': name, 'engine': engine, @@ -2520,8 +2636,10 @@ def main(cls, argv=True, **kwargs): class RoutesPruneCLI(_ApprovalMixin): - """Forget stale routes: rewrite the registry to *invoking catalog ∪ live*, - then converge (one accepted gateway recreate). + """Forget stale routes: keep only the invoking catalog's and the live ones. + + Rewrites the registry to *invoking catalog ∪ live*, then converges (one + accepted gateway recreate). The registry is append-only by design (that is what keeps the gateway config byte-stable), so pruning is the explicit, operator-driven "forget" verb — @@ -2536,18 +2654,13 @@ class RoutesPruneCLI(_ApprovalMixin): __command__ = 'prune' - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): from ..diff_prompt import confirm_writes from ..leasing.backend import ConvergeAborted - from ..leasing.compose import ( - LITELLM_REGISTRY_VERSION, - _dump_route_registry, - _registry_incoming_from_catalog, - _registry_incoming_from_deployments, - ) + from ..leasing.gateway import LITELLM_REGISTRY_VERSION, _dump_route_registry config = cls.cli(argv=argv, data=kwargs) # interactive=False so reconcile auto-applies the compose diff; the @@ -2556,13 +2669,14 @@ def main(cls, argv=True, **kwargs): backend = _require_compose_backend(controller) def prune_plan() -> tuple[dict, dict, list[str]]: - desired = controller.desired_deployments() - plan = backend.plan(desired) - keep: dict = {} - if backend.catalog is not None: - keep.update(_registry_incoming_from_catalog(backend.catalog)) - keep.update(_registry_incoming_from_deployments(desired, plan.assignments)) - current = backend._load_route_registry().get('entries', {}) + # The desired set exactly as the next render sees it, and the rows + # that render merges: the catalog's and the live deployments'. + # Any: _require_compose_backend above confirmed a gateway, and + # both backends that have one supply route rows. + engines: Any = controller.backend + desired, inputs = controller._admission_view(engines.residency()) + keep = dict(engines.route_rows(desired, inputs)) + current = backend.gateway._load_route_registry().get('entries', {}) return current, keep, sorted(set(current) - set(keep)) # Preview outside the lock (the prompt must not hold it); the change @@ -2579,7 +2693,7 @@ def prune_plan() -> tuple[dict, dict, list[str]]: for name in dropped: print(f' - {name}') ok = confirm_writes( - {backend._registry_file: _dump_route_registry(pruned)}, + {backend.gateway._registry_file: _dump_route_registry(pruned)}, assume_yes=False, title='infer-stack routes prune', ) @@ -2597,7 +2711,7 @@ def change(): entries = {k: v for k, v in current.items() if k not in drop} with backend._converge_lock(): backend._atomic_write( - backend._registry_file, + backend.gateway._registry_file, _dump_route_registry( {'version': LITELLM_REGISTRY_VERSION, 'entries': entries}), ) @@ -2633,17 +2747,16 @@ class RoutesSeedCLI(_ApprovalMixin): __command__ = 'seed' - catalogs = scfg.Value( + catalogs = kw.Value( None, nargs='+', position=1, type=str, help='One or more catalog.yaml files whose endpoints to merge into the ' 'route registry.', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): from ..leasing.backend import ConvergeAborted - from ..leasing.compose import _registry_incoming_from_catalog config = cls.cli(argv=argv, data=kwargs) paths = _collect_names(config.catalogs) @@ -2653,6 +2766,8 @@ def main(cls, argv=True, **kwargs): # there is no destructive gate to confirm). controller = _open_controller(config, interactive=False) backend = _require_compose_backend(controller) + # Any: a gateway exists (checked above), and its backend supplies rows. + engines: Any = controller.backend incoming: dict = {} for raw in paths: @@ -2663,14 +2778,16 @@ def main(cls, argv=True, **kwargs): cat = Catalog.load(path) except CatalogError as ex: raise SystemExit(f'invalid catalog {path}: {ex}') - incoming.update(_registry_incoming_from_catalog(cat)) + # The engine backend's rows for it: compose upstreams on compose, + # the cluster's Models on kubeai. + incoming.update(engines.catalog_route_rows(cat)) if not incoming: raise SystemExit( 'routes seed: the named catalog(s) resolved no routable endpoints' ) def change(): - before = set(backend._load_route_registry().get('entries', {})) + before = set(backend.gateway._load_route_registry().get('entries', {})) backend.merge_route_registry(incoming) return sorted(set(incoming) - before) @@ -2712,16 +2829,16 @@ class ConfigPublishCLI(_ApprovalMixin): __command__ = 'publish' - catalogs = scfg.Value( + catalogs = kw.Value( [], nargs='*', position=1, type=str, help='Catalog files to publish as one union (default: --catalog, or the ' 'default-path catalog).', ) - pull = scfg.Value( + pull = kw.Value( True, isflag=True, help='Pre-pull every image the profile references (default; --no-pull skips).', ) - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -2792,8 +2909,8 @@ class NetworkMigrateCLI(_ApprovalMixin): __command__ = 'migrate' - subnet = scfg.Value(None, type=str, help='IPv4 subnet, e.g. 172.30.0.0/24 (required).') - force = scfg.Value(False, isflag=True, help='Migrate even with active leases.') + subnet = kw.Value(None, type=str, help='IPv4 subnet, e.g. 172.30.0.0/24 (required).') + force = kw.Value(False, isflag=True, help='Migrate even with active leases.') @classmethod def main(cls, argv=True, **kwargs): @@ -2804,8 +2921,9 @@ def main(cls, argv=True, **kwargs): if not config.subnet: raise SystemExit('network migrate: --subnet is required') controller = _open_controller(config, interactive=True) - if not callable(getattr(controller.backend, 'residency', None)): - raise SystemExit('network migrate needs the compose backend') + if not hasattr(controller.backend, 'network'): + raise SystemExit('network migrate gives compose containers stable ' + 'addresses; this backend runs none (n/a on kubeai)') try: rec = controller.network_migrate(config.subnet, force=bool(config.force)) except ProfileMismatch as ex: @@ -2828,7 +2946,7 @@ class NetworkCheckCLI(_LeasingCommonMixin): __command__ = 'check' - json = scfg.Value(False, isflag=True) + json = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): @@ -2858,7 +2976,7 @@ class SecretsRotateCLI(_ApprovalMixin): __command__ = 'rotate' - force = scfg.Value(False, isflag=True, help='Rotate even with active leases.') + force = kw.Value(False, isflag=True, help='Rotate even with active leases.') @classmethod def main(cls, argv=True, **kwargs): @@ -2870,7 +2988,7 @@ def main(cls, argv=True, **kwargs): # Any: rotate_gateway_key below refuses a backend without a gateway, # so past it these gateway methods exist. backend: Any = controller.backend - old = backend.master_key() if isinstance(backend, ComposeBackend) else None + old = backend.master_key() if getattr(backend, 'litellm', False) else None try: rec = controller.rotate_gateway_key(force=bool(config.force)) except ProfileMismatch as ex: @@ -2892,13 +3010,14 @@ def main(cls, argv=True, **kwargs): 'rejects the OLD key') else: print(' gateway: new key accepted, old key rejected') - if getattr(backend, 'ui', False): + front = getattr(backend, 'compose_project', lambda: None)() + if getattr(front, 'ui', False): print(' Open WebUI may keep the old key in its own settings: ' 'update it under Admin > Settings > Connections') return 0 -class SecretsModalCLI(scfg.ModalCLI): +class SecretsModalCLI(kw.ModalCLI): """Manage the gateway's secrets.""" __command__ = 'secrets' @@ -2906,7 +3025,7 @@ class SecretsModalCLI(scfg.ModalCLI): rotate = SecretsRotateCLI -class NetworkModalCLI(scfg.ModalCLI): +class NetworkModalCLI(kw.ModalCLI): """Stable per-service addressing (migrate) and the upstream routing check.""" __command__ = 'network' @@ -2915,7 +3034,7 @@ class NetworkModalCLI(scfg.ModalCLI): check = NetworkCheckCLI -class RoutesModalCLI(scfg.ModalCLI): +class RoutesModalCLI(kw.ModalCLI): """Inspect + manage the LiteLLM route registry (static-superset mode). The registry accumulates every catalog's and every live deployment's routes diff --git a/infer_stack/cli/commands_meta.py b/infer_stack/cli/commands_meta.py index fa43990b..7fc3a50a 100644 --- a/infer_stack/cli/commands_meta.py +++ b/infer_stack/cli/commands_meta.py @@ -16,7 +16,7 @@ from pathlib import Path from typing import Any -import scriptconfig as scfg +import kwconf as kw from .. import __version__ from ..paths import config_root, data_root, settings_path @@ -24,7 +24,7 @@ from .options import _PathOverridesMixin -class VersionCLI(scfg.DataConfig): +class VersionCLI(kw.Config): """Print the installed infer-stack version.""" __command__ = 'version' @@ -48,7 +48,7 @@ def _iter_subcommands(modal: Any): if attr.startswith('_'): continue if isinstance(val, type) and issubclass( - val, (scfg.DataConfig, scfg.ModalCLI) + val, (kw.Config, kw.ModalCLI) ): name = getattr(val, '__command__', None) or attr.replace('_', '-') out[name] = val @@ -64,7 +64,7 @@ def _doc_one_line(cls: Any) -> str: def _is_group(sub: Any) -> bool: - return isinstance(sub, type) and issubclass(sub, scfg.ModalCLI) + return isinstance(sub, type) and issubclass(sub, kw.ModalCLI) def _build_tree(modal: Any, node: Any) -> None: @@ -86,7 +86,7 @@ def _build_tree(modal: Any, node: Any) -> None: _build_tree(sub, child) -class HelpTreeCLI(scfg.DataConfig): +class HelpTreeCLI(kw.Config): """Print the full nested command tree with one-line descriptions.""" __command__ = 'tree' @@ -106,7 +106,7 @@ def main(cls, argv=True, **kwargs): return 0 -class HelpModalCLI(scfg.ModalCLI): +class HelpModalCLI(kw.ModalCLI): """Help utilities (use ``infer-stack --help`` for per-command help).""" __command__ = 'help' @@ -207,12 +207,12 @@ class ConfigPathsCLI(_PathOverridesMixin): __command__ = 'paths' - target: Any = scfg.Value( + target: Any = kw.Value( 'all', position=1, help='Path group to show: all, config, data, or leasing.', ) - json: Any = scfg.Value( + json: Any = kw.Value( False, isflag=True, help='Emit the path groups as JSON instead of human-readable text.', @@ -362,6 +362,19 @@ class _Setting: 'kubeai_resource_profile': 'Fallback KubeAI resourceProfiles name for catalog endpoints whose ' 'runtime omits resource_profile.', + 'kubeai_gateway_upstream': + 'URL the LiteLLM gateway uses to reach KubeAI (default: the kubeai ' + "Service's cluster IP, reachable from a cluster node; set an ingress " + 'URL when the gateway runs off the cluster).', + 'kubeai_gateway': + 'Where the LiteLLM gateway runs: `host` (default; a Compose project on ' + 'this host) or `cluster` (a Deployment + NodePort Service in the KubeAI ' + 'namespace, reachable from every node).', + 'kubeai_gateway_node_port': + 'NodePort of the in-cluster gateway (default: 30442).', + 'kubeai_gateway_url': + 'Where clients reach the in-cluster gateway, e.g. an ingress URL ' + "(default: the first node's address on the NodePort).", } @@ -388,17 +401,17 @@ class ConfigInitCLI(_PathOverridesMixin): """ __command__ = 'init' - yes = scfg.Value( + yes = kw.Value( False, isflag=True, alias=['y'], help='Non-interactive: write without prompting/confirming.', ) - fresh = scfg.Value( + fresh = kw.Value( False, isflag=True, help='Start over: ignore any existing config and write a clean one from ' 'defaults (discards other persisted settings too).', ) - backend = scfg.Value( - None, choices=['compose', 'kubeai', 'null'], + backend = kw.Value( + None, type=str, choices=['compose', 'kubeai', 'null'], help='Preset the default backend (skips that prompt).', ) @@ -481,7 +494,8 @@ def _proposed(s: _Setting): base.update(values) save_settings(base) print(f'wrote settings -> {path}') - print('next: `infer-stack catalog init` to add models/endpoints') + print('next: `infer-stack catalog suggest --apply` (a catalog sized to this ' + "host's GPUs), then `infer-stack acquire `") return 0 @@ -489,8 +503,8 @@ class ConfigSetCLI(_PathOverridesMixin): """Persist a durable default, e.g. ``config set backend compose``.""" __command__ = 'set' - key = scfg.Value(None, position=1, type=str) - value = scfg.Value(None, position=2, type=str) + key = kw.Value(None, position=1, type=str) + value = kw.Value(None, position=2, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -517,7 +531,7 @@ class ConfigGetCLI(_PathOverridesMixin): """Print one setting's value (or all settings).""" __command__ = 'get' - key = scfg.Value(None, position=1, type=str) + key = kw.Value(None, position=1, type=str) @classmethod def main(cls, argv=True, **kwargs): @@ -584,7 +598,7 @@ def main(cls, argv=True, **kwargs): from .commands_leasing import ConfigPublishCLI # noqa: E402 -class ConfigModalCLI(scfg.ModalCLI): +class ConfigModalCLI(kw.ModalCLI): """Inspect + manage infer-stack configuration (paths + durable settings).""" __command__ = 'config' diff --git a/infer_stack/cli/commands_mock.py b/infer_stack/cli/commands_mock.py index c425fa5b..16732d42 100644 --- a/infer_stack/cli/commands_mock.py +++ b/infer_stack/cli/commands_mock.py @@ -10,18 +10,18 @@ from typing import Any -import scriptconfig as scfg +import kwconf as kw import ubelt as ub -class MockServeCLI(scfg.DataConfig): +class MockServeCLI(kw.Config): """ Serve a deterministic OpenAI-compatible mock inference endpoint. """ __command__ = 'serve' - config_fpath = scfg.Value( + config_fpath = kw.Value( None, position=1, help=ub.paragraph( @@ -35,14 +35,14 @@ class MockServeCLI(scfg.DataConfig): ), ) - host = scfg.Value('127.0.0.1', help='Bind address.') + host = kw.Value('127.0.0.1', help='Bind address.') - port = scfg.Value( + port = kw.Value( 8100, help='Bind port. Use 0 to pick a free port and print it.', ) - mode = scfg.Value( + mode = kw.Value( None, help=ub.paragraph( """ @@ -57,13 +57,13 @@ class MockServeCLI(scfg.DataConfig): ), ) - require_auth = scfg.Value( + require_auth = kw.Value( False, isflag=True, help='Reject requests without a valid bearer token.', ) - api_key = scfg.Value( + api_key = kw.Value( None, help=ub.paragraph( """ @@ -74,11 +74,11 @@ class MockServeCLI(scfg.DataConfig): ), ) - list_modes = scfg.Value( + list_modes = kw.Value( False, isflag=True, help='Print the available response modes and exit.', ) - seed = scfg.Value( + seed = kw.Value( None, help=ub.paragraph( """ @@ -88,7 +88,7 @@ class MockServeCLI(scfg.DataConfig): ), ) - print_url = scfg.Value( + print_url = kw.Value( True, isflag=True, help='Print the bound base URL on startup.', @@ -158,7 +158,7 @@ def main(cls, argv=None, **kwargs): server.stop() -class MockModalCLI(scfg.ModalCLI): +class MockModalCLI(kw.ModalCLI): """ Deterministic mock inference server for tests and dry runs. """ diff --git a/infer_stack/cli/commands_runtime.py b/infer_stack/cli/commands_runtime.py index 99fa46c6..34eb2f6e 100644 --- a/infer_stack/cli/commands_runtime.py +++ b/infer_stack/cli/commands_runtime.py @@ -14,15 +14,16 @@ from pathlib import Path from typing import Any -import scriptconfig as scfg +import kwconf as kw from ..log_filter import compact_litellm_tracebacks from ..paths import config_root, data_root, get_setting, settings_path +from .commands_leasing import ApplyCLI from .context import _apply_path_overrides from .options import _PathOverridesMixin # --------------------------------------------------------------------------- -# leasing compose project helpers (the target of the day-2 wrappers) +# the backend behind the day-2 verbs # --------------------------------------------------------------------------- @@ -32,43 +33,61 @@ def _docker_env() -> dict[str, str]: return docker_environment() -def _leasing_compose_file() -> Path: - from ..leasing.compose import COMPOSE_FILENAME +def _day2_backend(config): + """The configured backend, built as the leasing verbs build it.""" + from .commands_leasing import _make_backend - return data_root() / 'leasing' / 'compose' / COMPOSE_FILENAME + _apply_path_overrides(config) + return _make_backend(config) -def _day2_compose_base(config, command: str) -> list[str]: - """``docker compose -p -f `` base for the wrappers. +def _served_by_deployment() -> dict[str, list[str]]: + """Deployment id -> the endpoint aliases it serves (read-only ledger).""" + from ..leasing import Ledger, SqliteStore, default_ledger_path - Targets the leasing Compose deployment (no ``config.yaml`` needed). Raises a - helpful error when nothing has been deployed yet — the file is written by - ``acquire`` / ``apply``. - """ - from ..leasing.compose import LEASING_PROJECT + path = default_ledger_path() + if not path.exists(): + return {} + try: + _, deployments = Ledger(SqliteStore(str(path))).status(virtual_expiry=True) + except Exception: # noqa: BLE001 - a view never fails on the ledger + return {} + return {d.id: sorted(d.served) for d in deployments} - _apply_path_overrides(config) - compose_file = _leasing_compose_file() - if not compose_file.exists(): - hint = '' - if (get_setting('backend') or '') == 'kubeai': - hint = ( - ' Note: the configured backend is kubeai — these verbs manage ' - 'the docker compose stack only; inspect the cluster with ' - '`infer-stack doctor` / `kubectl -n get models`.' - ) + +def _instances(backend) -> list: + """What the backend runs, or a clean exit when it cannot be read.""" + from ..leasing.residency import ResidencyUnknown + + try: + return list(backend.instances()) + except ResidencyUnknown as ex: + raise SystemExit(f'cannot read what is running: {ex}') + + +def _compose_argv(config) -> list[str]: + """``docker compose ...`` for the Compose project on this host. + + That is the stack itself on the compose backend, and the gateway on the + kubeai backend. Exits when there is none, or nothing is rendered yet. + """ + backend = _day2_backend(config) + project = getattr(backend, 'compose_project', lambda: None)() + if project is None: raise SystemExit( - f'nothing deployed yet (no {compose_file}). ' - f'Bring a model up first, e.g. `infer-stack acquire `.' - f'{hint}' - ) - base = ['docker', 'compose'] - # The same managed .env the backend passes, so `stack up` interpolates the - # master key, DB password and HF_TOKEN from it (never the caller's shell). - env_file = compose_file.parent / '.env' - if env_file.exists(): - base += ['--env-file', str(env_file)] - return [*base, '-p', LEASING_PROJECT, '-f', str(compose_file)] + f'the {_backend_name(backend)} backend has no Compose project on this ' + 'host; use `infer-stack ps` and `infer-stack logs`') + if not project.compose_file.exists(): + raise SystemExit( + f'nothing rendered yet (no {project.compose_file}); bring a model up ' + 'first, e.g. `infer-stack acquire `') + return project.compose_argv() + + +def _backend_name(backend) -> str: + name = type(backend).__name__ + return {'ComposeBackend': 'compose', 'KubeaiBackend': 'kubeai', + 'NullBackend': 'null (dry-run)'}.get(name, name) # --------------------------------------------------------------------------- @@ -108,7 +127,9 @@ def _leasing_status() -> dict[str, Any]: if not path.exists(): return out try: - leases, deployments = Ledger(SqliteStore(str(path))).status(virtual_expiry=True) + ledger = Ledger(SqliteStore(str(path))) + leases, deployments = ledger.status(virtual_expiry=True) + out['pending'] = ledger.publication_pending() is not None except Exception: # noqa: BLE001 return out active = sum(1 for le in leases if le.state == LeaseState.ACTIVE) @@ -122,82 +143,70 @@ def _leasing_status() -> dict[str, Any]: for d in deployments ] out['summary'] = (active, len(leases), live, len(deployments)) - out['served'] = _served_models(deployments) + out['live_deployments'] = [d for d in deployments if d.state == DeploymentState.LIVE] return out -def _running_services() -> set[str] | None: - """Compose service names currently running, or None if docker can't say. +def _served_models(deployments, backend=None, *, + pending: bool = False) -> list[tuple[str, str, str, str]]: + """``(endpoint, model, engine, health)`` rows for what is serving. - A single cheap `docker compose ps`. It exists because the ledger's LIVE is - a *belief*: a converge that fails partway (an unpullable image, a daemon - hiccup) leaves deployments recorded LIVE with no container behind them, and - the only way to notice used to be an acquire that hung on readiness. - Distinguishing "recorded" from "running" is the whole point of showing this. - - None (not an empty set) when docker is unreachable, so the caller can say - "unverified" instead of falsely reporting everything down. - """ - from ..leasing.compose import LEASING_PROJECT - - compose_file = _leasing_compose_file() - if not compose_file.exists(): - return None - try: - proc = subprocess.run( - ['docker', 'compose', '-p', LEASING_PROJECT, '-f', str(compose_file), - 'ps', '--services', '--filter', 'status=running'], - capture_output=True, text=True, timeout=15, env=_docker_env(), - ) - except Exception: # noqa: BLE001 - status must never fail on this - return None - if proc.returncode != 0: - return None - return {ln.strip() for ln in proc.stdout.splitlines() if ln.strip()} - - -def _served_models(deployments) -> list[tuple[str, str, str, str]]: - """``(endpoint, model, gpus_or_engine, health)`` rows for what is serving. - - Reads the ledger for what *should* be up and reconciles against the - containers that actually are, so `status` answers "what can I send a - request to right now" without the user having to cross-read `leases` - against `stack ps`. + Reads the ledger for what *should* be up and the backend's strict + residency for what is, so `status` answers "what can I send a request to + right now" without cross-reading `leases` against `ps`. """ from ..leasing import DeploymentState - from ..leasing.compose import ollama_service_name_for, vllm_service_name_for + from ..leasing.residency import ResidencyUnknown live = [d for d in deployments if d.state == DeploymentState.LIVE] if not live: return [] - running = _running_services() + residency = None + if backend is not None: + try: + residency = backend.residency() + except (ResidencyUnknown, Exception): # noqa: BLE001 - status never fails here + residency = None rows: list[tuple[str, str, str, str]] = [] for d in live: + if residency is None: + health = 'unverified' + elif residency.resident(d.id) is not None: + # Warm; a health check that has not passed yet means still loading. + health = ('starting' if residency.resident(d.id).health == 'starting' + else 'up') + elif residency.containers(d.id): + # There, but not warm: starting, crashed, or ambiguous. + health = residency.containers(d.id)[0].state + elif pending: + # Recorded, not applied yet: an apply is running, or a failed one + # left the change for `infer-stack apply`. + health = 'pending' + else: + # The ledger says live, nothing exists, and no change is pending. + health = 'STALE' for endpoint in sorted(d.served): payload = d.served.get(endpoint) or {} model = (payload.get('hf_model_id') or payload.get('model') or d.spec.get('hf_model_id') or '-') - if d.engine == 'ollama': - service = ollama_service_name_for(d.spec.get('host') or endpoint) - else: - service = vllm_service_name_for( - payload.get('served_model_name') or endpoint) - if running is None: - health = 'unverified' - elif service in running: - health = 'up' - else: - # The ledger says live and nothing is running. Almost always a - # converge that failed after the record was written. - health = 'STALE' rows.append((endpoint, model, d.engine, health)) return rows def _gather_status(config) -> dict[str, Any]: - compose_file = _leasing_compose_file() + try: + backend = _day2_backend(config) + except (SystemExit, Exception): # noqa: BLE001 - status never fails on this + backend = None + rendered = getattr(backend, 'rendered_file', None) + leasing = _leasing_status() + if leasing.get('live_deployments'): + leasing['served'] = _served_models(leasing.pop('live_deployments'), backend, + pending=bool(leasing.get('pending'))) + else: + leasing.pop('live_deployments', None) return { 'backend': str(get_setting('backend') or 'null'), 'data_dir': str(data_root()), @@ -206,16 +215,17 @@ def _gather_status(config) -> dict[str, Any]: 'settings': {'path': str(settings_path()), 'exists': settings_path().exists()}, 'catalog': _catalog_summary(config), - 'compose': {'path': str(compose_file), 'exists': compose_file.exists()}, - 'leasing': _leasing_status(), + 'rendered': {'path': str(rendered) if rendered else None, + 'exists': bool(rendered and rendered.exists())}, + 'leasing': leasing, } _DIG_DEEPER = ( ('infer-stack leases', 'full lease + deployment tables'), ('infer-stack tui', 'live dashboard (opt-in: infer-stack[tui])'), - ('infer-stack stack ps', 'running containers'), - ('infer-stack logs -f', 'tail service logs'), + ('infer-stack ps', 'what is running (containers or pods)'), + ('infer-stack logs -f ', 'follow an engine\'s log'), ('infer-stack catalog show', 'what you can serve'), ) @@ -237,11 +247,17 @@ def _served_lines(served: list[tuple[str, str, str, str]]) -> list[str]: for endpoint, model, _engine, health in served: out.append(f' {endpoint.ljust(w_ep)} {model.ljust(w_mo)} {health}') if any(r[3] == 'STALE' for r in served): - out.append(' STALE = the ledger records this live but no container is ' - 'running; `infer-stack apply` or `gc`') + out.append(' STALE = the ledger records this live but nothing is ' + 'running for it; `infer-stack apply` or `gc`') + if any(r[3] == 'pending' for r in served): + out.append(' pending = recorded but not applied yet: an apply is running, ' + 'or `infer-stack apply` finishes it') + if any(r[3] == 'starting' for r in served): + out.append(' starting = up, but not ready yet (loading the model); ' + '`infer-stack wait `') if any(r[3] == 'unverified' for r in served): - out.append(' unverified = could not reach docker to confirm; the ' - 'ledger says live') + out.append(' unverified = the runtime could not be read to confirm; ' + 'the ledger says live') return out @@ -262,8 +278,9 @@ def _print_status_plain(d: dict[str, Any]) -> None: f'{"" if d["configured"] else " (run infer-stack config init)"}') lz = d['leasing'] print(f' ledger: {lz["path"]}{"" if lz["exists"] else " (none yet)"}') - print(f' compose: {d["compose"]["path"]}' - f'{"" if d["compose"]["exists"] else " (not rendered yet)"}') + if d['rendered']['path']: + print(f' rendered: {d["rendered"]["path"]}' + f'{"" if d["rendered"]["exists"] else " (not rendered yet)"}') if lz['summary']: active, total_l, live, total_d = lz['summary'] print() @@ -309,10 +326,11 @@ def _print_status_rich(d: dict[str, Any], console) -> None: if not lz['exists']: ledger.append(' (none yet)', style='dim') table.add_row('ledger', ledger) - compose = Text(d['compose']['path'], style='cyan') - if not d['compose']['exists']: - compose.append(' (not rendered yet)', style='dim') - table.add_row('compose', compose) + if d['rendered']['path']: + rendered = Text(d['rendered']['path'], style='cyan') + if not d['rendered']['exists']: + rendered.append(' (not rendered yet)', style='dim') + table.add_row('rendered', rendered) console.print(table) if lz['summary']: @@ -332,7 +350,8 @@ def _print_status_rich(d: dict[str, Any], console) -> None: served_table.add_column('model', overflow='fold') served_table.add_column('engine', style='dim', no_wrap=True) served_table.add_column('health', no_wrap=True) - styles = {'up': 'green', 'STALE': 'red', 'unverified': 'yellow'} + styles = {'up': 'green', 'starting': 'yellow', 'pending': 'yellow', + 'STALE': 'red', 'unverified': 'yellow'} for endpoint, model, engine, health in served: served_table.add_row( endpoint, model, engine, @@ -343,7 +362,7 @@ def _print_status_rich(d: dict[str, Any], console) -> None: console.print(served_table) if any(r[3] == 'STALE' for r in served): console.print(Text( - ' STALE = recorded live but no container is running; ' + ' STALE = recorded live but nothing is running for it; ' '`infer-stack apply` or `gc`', style='dim')) console.print() @@ -361,11 +380,13 @@ def _print_status_rich(d: dict[str, Any], console) -> None: class StatusCLI(_PathOverridesMixin): - """Holistic overview: where things live, the backend, and a leasing summary - (active leases / live deployments), with pointers to dig deeper.""" + """Holistic overview: where things live, the backend, and what is serving. + + A leasing summary (active leases, live deployments) with each endpoint's + health, and pointers to dig deeper.""" __command__ = 'status' - catalog = scfg.Value( + catalog = kw.Value( None, type=str, help='Catalog path (default: config dir).' ) @@ -385,67 +406,172 @@ def main(cls, argv=True, **kwargs): # --------------------------------------------------------------------------- -# stack — docker compose day-2-ops wrappers over the leasing project +# ps / logs / stack: what the backend runs, on either backend # --------------------------------------------------------------------------- -class _ComposeWrapperBase(_PathOverridesMixin): - """Common fields for ``docker compose `` wrappers over the leasing - Compose deployment.""" +class _InstancesBase(_PathOverridesMixin): + """Options shared by the verbs that read the backend's instances.""" - services = scfg.Value( + services = kw.Value( None, nargs='*', position=1, - help='Optional service names to filter (empty = all).', + help='Which instances: a service or pod name, a container id (prefix), ' + 'a deployment id, or an endpoint alias. Empty = all.', + ) + backend = kw.Value( + None, type=str, + help='Backend to read (default: the configured `backend` setting).', ) -def _run_compacted_follow(cmd: list[str]) -> int: - """Stream Compose logs through the conservative LiteLLM compactor.""" - proc = subprocess.Popen( - cmd, - stdout=subprocess.PIPE, - text=True, - errors='replace', - bufsize=1, - env=_docker_env(), +#: States `ps` hides unless --all: finished, or on the way out. +_PS_HIDDEN = frozenset({'exited', 'dead', 'removing'}) + + +def _ps_rows(instances, served) -> list[dict[str, Any]]: + rows = [] + for inst in instances: + rows.append({ + 'name': inst.name, + 'id': inst.id, + 'deployment': inst.deployment_id or None, + 'serves': served.get(inst.deployment_id, []) if inst.deployment_id else [], + 'state': inst.state, + 'status': inst.status, + 'restarts': inst.restarts, + 'gpus': list(inst.gpus), + 'started': inst.started or None, + 'ports': inst.ports or None, + 'runtime': inst.runtime, + }) + return rows + + +def _print_ps(rows) -> None: + def cell(row): + serves = ', '.join(row['serves']) or ('-' if row['deployment'] else '(front door)') + gpus = ','.join(map(str, row['gpus'])) or '-' + ident = row['id'][:12] if row['runtime'] == 'docker' else '-' + return (row['name'], row['status'], serves, gpus, + (row['started'] or '-')[:19], ident, row['ports'] or '-') + + head = ('NAME', 'STATUS', 'SERVES', 'GPUS', 'STARTED', 'ID', 'PORTS') + table = [head, *(cell(r) for r in rows)] + widths = [max(len(str(r[i])) for r in table) for i in range(len(head))] + for r in table: + print(' '.join(str(v).ljust(w) for v, w in zip(r, widths)).rstrip()) + + +class PsCLI(_InstancesBase): + """What the backend is running: engine containers or pods, and the gateway. + + One shape on every backend. An engine row names the endpoints it serves; + the gateway, UI and proxy show as the front door. Reads the same strict + residency the controller decides with, so a row here is what admission + sees. + """ + + __command__ = 'ps' + + all = kw.Value( + False, isflag=True, short_alias=['a'], + help='Include exited and dying instances.', ) - try: - if proc.stdout is None: # pragma: no cover - PIPE guarantees stdout - return int(proc.wait()) - for line in compact_litellm_tracebacks(proc.stdout): - sys.stdout.write(line) - return int(proc.wait()) - except KeyboardInterrupt: - # The child normally receives the same SIGINT. If it is still alive, - # make sure an interrupted follow does not leave Compose behind. - if proc.poll() is None: - proc.terminate() + services_only = kw.Value( + False, isflag=True, help='Print only instance names.', + ) + quiet = kw.Value( + False, isflag=True, short_alias=['q'], + help='Print only ids (container ids, or pod names).', + ) + json = kw.Value(False, isflag=True, help='Print the rows as JSON.') + + @classmethod + def main(cls, argv=True, **kwargs): + import json + + from ..leasing.instances import UnknownTarget, resolve + + config = cls.cli(argv=argv, data=kwargs) + backend = _day2_backend(config) + instances = _instances(backend) + served = _served_by_deployment() + if config.services: try: - proc.wait(timeout=2) - except subprocess.TimeoutExpired: - proc.kill() - proc.wait() - return 130 + instances = resolve(instances, config.services, served) + except UnknownTarget as ex: + raise SystemExit(str(ex)) + if not config.all: + instances = [i for i in instances if i.state not in _PS_HIDDEN] + rows = _ps_rows(instances, served) + if config.json: + print(json.dumps(rows, indent=2)) + elif config.quiet: + for row in rows: + print(row['id']) + elif config.services_only: + for row in rows: + print(row['name']) + elif not rows: + print(f'nothing running ({_backend_name(backend)} backend); ' + 'bring a model up with `infer-stack acquire `') + else: + _print_ps(rows) + return 0 + +#: Prefix colors for `logs` on a terminal, cycled per instance name. +_LOG_COLORS = ('36', '33', '32', '35', '34', '96', '93', '92', '95', '94') -class LogsCLI(_ComposeWrapperBase): - """Tail leasing Compose service logs without typing the full docker path.""" + +def _colorize(lines, *, enabled: bool): + """Color each ``name | `` prefix, one color per name, like Compose did. + + Disabled (a pipe, a file, ``--no-color``), the engines' own color codes + go too: they are noise in a file and break a grep. + """ + if not enabled: + from ..log_filter import _ANSI_ESCAPE_RE + + for line in lines: + yield _ANSI_ESCAPE_RE.sub('', line) + return + colors: dict[str, str] = {} + for line in lines: + name, sep, rest = line.partition(' | ') + if not sep: + yield line + continue + code = colors.setdefault(name, _LOG_COLORS[len(colors) % len(_LOG_COLORS)]) + yield f'\x1b[{code}m{name} |\x1b[0m {rest}' + + +class LogsCLI(_InstancesBase): + """Engine and gateway logs, by instance, deployment or endpoint alias. + + ``infer-stack logs qwen -f`` follows whatever serves the ``qwen`` + endpoint, container or pod; with no names, every instance. Following + picks up instances that start later and follows a restarted one again. + """ __command__ = 'logs' - follow = scfg.Value( + follow = kw.Value( False, isflag=True, short_alias=['f'], - help='Stream logs (docker compose logs -f).', + help='Keep streaming, including instances that start later.', ) - tail = scfg.Value( + tail = kw.Value( None, type=str, - help="Tail the last N lines (default: all). Pass a number or 'all'.", + help="Only the last N lines of each (default: all). A number or 'all'.", ) - timestamps = scfg.Value(False, isflag=True) - no_color = scfg.Value(False, isflag=True) - raw = scfg.Value( + timestamps = kw.Value(False, isflag=True) + # A positive flag: kwconf reads a leading `no-` as negation, so a flag + # named `no_color` silently ignored `--no-color`. + color = kw.Value(True, isflag=True, + help='Color the name prefixes on a terminal (`--no-color` to not).') + raw = kw.Value( False, isflag=True, help='Show raw followed logs without known LiteLLM traceback compaction.', @@ -453,74 +579,96 @@ class LogsCLI(_ComposeWrapperBase): @classmethod def main(cls, argv=True, **kwargs): + from ..leasing.instances import ( + LogFollower, + UnknownTarget, + history_argv, + resolve, + runtime_env, + ) + config = cls.cli(argv=argv, data=kwargs) + backend = _day2_backend(config) + served = _served_by_deployment() + names = list(config.services or []) - # The compacted path pipes Compose stdout through Python. Without an - # explicit ANSI mode Compose sees a non-TTY pipe and drops the service - # colors before the compactor can preserve them. Force ANSI only for - # the human-facing compacted view; --no-color remains authoritative. - compact = config.follow and not config.raw and sys.stdout.isatty() - cmd = _day2_compose_base(config, 'logs') - if compact and not config.no_color: - cmd.extend(['--ansi', 'always']) - cmd.append('logs') - if config.follow: - cmd.append('--follow') - if config.tail is not None: - cmd.extend(['--tail', str(config.tail)]) - if config.no_color: - cmd.append('--no-color') - if config.timestamps: - cmd.append('--timestamps') - cmd.extend(config.services or []) + def pick(instances): + return resolve(instances, names, served) if names else instances - # Compact only the human-facing live view. Captures and pipelines keep - # Docker's exact bytes unless a future explicit compact-output mode is - # added; ``--raw`` is also available for interactive LiteLLM debugging. - if compact: - return _run_compacted_follow(cmd) - return int(subprocess.run(cmd, env=_docker_env()).returncode) + try: + chosen = pick(_instances(backend)) + except UnknownTarget as ex: + raise SystemExit(str(ex)) + color = sys.stdout.isatty() and bool(config.color) + if config.follow: + def listing(): + try: + return pick(backend.instances()) + except UnknownTarget: + return [] # the named one is between restarts + follower = LogFollower(listing, history=config.tail or 'all', + timestamps=bool(config.timestamps)) + lines = follower.stdout + if sys.stdout.isatty() and not config.raw: + lines = compact_litellm_tracebacks(lines) + try: + for line in _colorize(lines, enabled=color): + sys.stdout.write(line) + sys.stdout.flush() + except KeyboardInterrupt: + return 130 + finally: + follower.terminate() + return 0 + if not chosen: + print(f'nothing running ({_backend_name(backend)} backend)') + return 0 + prefix = len(chosen) > 1 + status = 0 + for inst in chosen: + argv_ = history_argv(inst, tail=config.tail, + timestamps=bool(config.timestamps)) + if argv_ is None: + print(f'{inst.name}: the {_backend_name(backend)} backend keeps no logs') + continue + proc = subprocess.run(argv_, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, env=runtime_env(inst)) + text = proc.stdout.decode('utf-8', 'replace') + lines = [f'{inst.name} | {ln}\n' if prefix else f'{ln}\n' + for ln in text.splitlines()] + for line in _colorize(lines, enabled=color): + sys.stdout.write(line) + status = status or proc.returncode + return int(status) -class PsCLI(_ComposeWrapperBase): - """``docker compose ps`` for the leasing deployment.""" +class _ComposeWrapperBase(_PathOverridesMixin): + """``docker compose `` over the Compose project on this host. - __command__ = 'ps' + The stack itself on the compose backend; the gateway on kubeai. + """ - all = scfg.Value( - False, isflag=True, short_alias=['a'], help='Include stopped containers.' - ) - services_only = scfg.Value( - False, isflag=True, - help='Print only service names (passes --services to docker compose).', + services = kw.Value( + None, + nargs='*', + position=1, + help='Optional service names to filter (empty = all).', ) - quiet = scfg.Value( - False, isflag=True, short_alias=['q'], help='Print only container IDs.' + backend = kw.Value( + None, type=str, + help='Backend (default: the configured `backend` setting).', ) - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'ps') + ['ps'] - if config.all: - cmd.append('--all') - if config.services_only: - cmd.append('--services') - if config.quiet: - cmd.append('--quiet') - cmd.extend(config.services or []) - return int(subprocess.run(cmd, env=_docker_env()).returncode) - class RestartCLI(_ComposeWrapperBase): - """``docker compose restart [services...]``.""" + """``docker compose restart [services...]`` (the Compose project on this host).""" - timeout = scfg.Value(None, type=int, help='Stop timeout in seconds.') + timeout = kw.Value(None, type=int, help='Stop timeout in seconds.') @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'restart') + ['restart'] + cmd = _compose_argv(config) + ['restart'] if config.timeout is not None: cmd.extend(['--timeout', str(config.timeout)]) cmd.extend(config.services or []) @@ -528,15 +676,15 @@ def main(cls, argv=True, **kwargs): class PullCLI(_ComposeWrapperBase): - """``docker compose pull [services...]``.""" + """``docker compose pull [services...]`` (the Compose project on this host).""" - quiet = scfg.Value(False, isflag=True, short_alias=['q']) - ignore_pull_failures = scfg.Value(False, isflag=True) + quiet = kw.Value(False, isflag=True, short_alias=['q']) + ignore_pull_failures = kw.Value(False, isflag=True) @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'pull') + ['pull'] + cmd = _compose_argv(config) + ['pull'] if config.quiet: cmd.append('--quiet') if config.ignore_pull_failures: @@ -546,88 +694,126 @@ def main(cls, argv=True, **kwargs): class StartCLI(_ComposeWrapperBase): - """``docker compose start [services...]``.""" + """``docker compose start [services...]`` (the Compose project on this host).""" @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'start') + ['start'] + cmd = _compose_argv(config) + ['start'] cmd.extend(config.services or []) return int(subprocess.run(cmd, env=_docker_env()).returncode) class StopCLI(_ComposeWrapperBase): - """``docker compose stop [services...]``.""" + """``docker compose stop [services...]`` (the Compose project on this host).""" - timeout = scfg.Value(None, type=int) + timeout = kw.Value(None, type=int) @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'stop') + ['stop'] + cmd = _compose_argv(config) + ['stop'] if config.timeout is not None: cmd.extend(['--timeout', str(config.timeout)]) cmd.extend(config.services or []) return int(subprocess.run(cmd, env=_docker_env()).returncode) -class StackDownCLI(_ComposeWrapperBase): - """``docker compose down`` the leasing deployment. +class StackComposeCLI(_PathOverridesMixin): + """Run any ``docker compose`` command on the Compose project on this host. - Tears the whole project down. Leasing's reconcile manages teardown - automatically on release; this is the manual escape hatch. + The raw escape hatch: the stack itself on the compose backend, the gateway + on kubeai. It bypasses the ledger and renders nothing, e.g. + ``infer-stack stack compose -- up -d litellm``. """ - volumes = scfg.Value( - False, isflag=True, help='Also remove named volumes (--volumes).' + __command__ = 'compose' + + args = kw.Value(None, nargs='*', position=1, + help='Arguments for docker compose (put them after --).') + backend = kw.Value( + None, type=str, + help='Backend (default: the configured `backend` setting).', ) @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'down') + ['down', '--remove-orphans'] - if config.volumes: - cmd.append('--volumes') + cmd = _compose_argv(config) + list(config.args or []) return int(subprocess.run(cmd, env=_docker_env()).returncode) -class StackUpCLI(_ComposeWrapperBase): - """``docker compose up -d`` exactly what is on disk — the raw escape hatch. +class StackDownCLI(_PathOverridesMixin): + """Stop everything the backend runs, bypassing the ledger. - Brings up the on-disk compose file as-is, without touching the ledger or - re-rendering. Prefer ``infer-stack apply``, which re-renders from intent - first. Reach for ``stack up`` only to run a *hand-edited* compose file - verbatim. + The manual escape hatch: releases no lease, so a later publish brings + leased models back. Compose: ``docker compose down``; kubeai: deletes every + managed Model, then the gateway. ``--volumes`` (Compose only) also removes + named volumes. """ + __command__ = 'down' + + volumes = kw.Value( + False, isflag=True, help='Also remove named volumes (Compose only).' + ) + backend = kw.Value( + None, type=str, + help='Backend (default: the configured `backend` setting).', + ) + @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - cmd = _day2_compose_base(config, 'up') + ['up', '-d', '--remove-orphans'] - cmd.extend(config.services or []) - return int(subprocess.run(cmd, env=_docker_env()).returncode) + if config.volumes: + cmd = _compose_argv(config) + ['down', '--remove-orphans', '--volumes'] + return int(subprocess.run(cmd, env=_docker_env()).returncode) + backend = _day2_backend(config) + down = getattr(backend, 'down', None) + if down is None: + print(f'the {_backend_name(backend)} backend runs nothing to bring down') + return 0 + down() + return 0 -class StackModalCLI(scfg.ModalCLI): - """Day-2 ops on the running leasing deployment.""" +class StackUpCLI(ApplyCLI): + """``apply``: bring up what the ledger says should run (both backends). + + The raw form, ``docker compose up`` of the file on disk, is + ``infer-stack stack compose -- up -d``. + """ + + __command__ = 'up' + + +class StackModalCLI(kw.ModalCLI): + """Day-2 ops on what the backend runs. + + ``up`` is ``apply`` and ``down`` stops everything, on either backend. The + ``docker compose`` verbs (restart, pull, start, stop, and ``compose`` for + anything else) act on the Compose project on this host: the stack itself + on compose, the gateway on kubeai. + """ __command__ = 'stack' up = StackUpCLI logs = LogsCLI ps = PsCLI + down = StackDownCLI + compose = StackComposeCLI restart = RestartCLI pull = PullCLI start = StartCLI stop = StopCLI - down = StackDownCLI class DoctorCLI(_PathOverridesMixin): """Preflight the configured backend: is everything acquire needs in place? Runs the backend's cheap dependency-ordered checks (for kubeai: cluster - reachable -> KubeAI CRD installed -> namespace exists -> gateway + reachable -> KubeAI CRD installed -> namespace exists -> KubeAI's API answering) and prints a checklist. Exits nonzero if any check fails, so scripts can gate on it. Backends without a preflight (null/compose) report that there is nothing to check. @@ -645,13 +831,13 @@ class DoctorCLI(_PathOverridesMixin): __command__ = 'doctor' - backend = scfg.Value( + backend = kw.Value( None, type=str, help='Backend to check (default: the configured `backend` setting).', ) - gpu = scfg.Value(False, isflag=True, + gpu = kw.Value(False, isflag=True, help='Also check host GPUs (implied by --sudo).') - sudo = scfg.Value(False, isflag=True, + sudo = kw.Value(False, isflag=True, help='Run the GPU holder scan as root. Without it that ' 'check reports "not checked", never "clear".') diff --git a/infer_stack/cli/context.py b/infer_stack/cli/context.py index 3e01b14b..8d86ebcf 100644 --- a/infer_stack/cli/context.py +++ b/infer_stack/cli/context.py @@ -18,7 +18,7 @@ def _as_mapping(args: Any) -> dict[str, Any]: """Coerce a CLI args object into a plain dict. - Works for ``None``, ``argparse.Namespace``, and ``scfg.DataConfig`` + Works for ``None``, ``argparse.Namespace``, and ``kw.Config`` instances. Used to side-step name clashes between user-declared fields and ``DataConfig`` builtins. """ diff --git a/infer_stack/cli/options.py b/infer_stack/cli/options.py index ced620ff..1acb1413 100644 --- a/infer_stack/cli/options.py +++ b/infer_stack/cli/options.py @@ -2,17 +2,63 @@ from ..paths import CONFIG_DIR_ENV from ..paths import DATA_DIR_ENV -import scriptconfig as scfg +import kwconf as kw # --------------------------------------------------------------------------- # DataConfig mixins for common override flags # --------------------------------------------------------------------------- -class _PathOverridesMixin(scfg.DataConfig): +#: Values a bare flag may carry (``--yes false``); anything else was a positional. +_BOOL_WORDS = frozenset({'true', 'false', 'yes', 'no', 'on', 'off', '1', '0'}) + + +def reclaim_swallowed_positionals(config) -> None: + """Give back a positional argument that a preceding flag consumed. + + kwconf flags take an optional value, so ``logs -f qwen`` parsed as + ``follow='qwen'`` and no names: the command then acted on everything. + Every infer-stack flag is a boolean, so a flag holding any other string + was handed a positional. It becomes ``True``, and the string goes back to + the front of the command's positional list (or its single positional, + when that is still empty). + """ + defaults = type(config).__default__ + positional = sorted((k for k, v in defaults.items() if getattr(v, 'position', None)), + key=lambda k: defaults[k].position) + for key, value in defaults.items(): + if not getattr(value, 'isflag', False) or value.isflag == 'counter': + continue + got = config[key] + if not isinstance(got, str) or got.strip().lower() in _BOOL_WORDS: + continue + config[key] = True + if not positional: + raise SystemExit(f'--{key} takes no value (got {got!r})') + target = positional[0] + many = defaults[target].parsekw.get('nargs') in ('*', '+') + if many: + config[target] = [got, *(config[target] or [])] + elif config[target] in (None, ''): + config[target] = got + else: + raise SystemExit(f'--{key} takes no value (got {got!r})') + + +class _FlagSafeMixin(kw.Config): + """Parses ``--flag positional`` as a flag and a positional (see above).""" + + @classmethod + def cli(cls, *args, **kwargs): + config = super().cli(*args, **kwargs) + reclaim_swallowed_positionals(config) + return config + + +class _PathOverridesMixin(_FlagSafeMixin): """Adds global ``--config-dir`` / ``--data-dir`` to a subcommand.""" - config_dir = scfg.Value( + config_dir = kw.Value( None, type=str, help=( @@ -20,7 +66,7 @@ class _PathOverridesMixin(scfg.DataConfig): f'~/.config/infer_stack (XDG_CONFIG_HOME) or ${CONFIG_DIR_ENV} when set.' ), ) - data_dir = scfg.Value( + data_dir = kw.Value( None, type=str, help=( @@ -30,42 +76,42 @@ class _PathOverridesMixin(scfg.DataConfig): ) -class _BackendOverrideMixin(scfg.DataConfig): - backend = scfg.Value( - None, choices=['compose', 'kubeai'], help='Active backend override.' +class _BackendOverrideMixin(kw.Config): + backend = kw.Value( + None, type=str, choices=['compose', 'kubeai'], help='Active backend override.' ) -class _ComposeOverrideMixin(scfg.DataConfig): - compose_cmd = scfg.Value( +class _ComposeOverrideMixin(kw.Config): + compose_cmd = kw.Value( None, type=str, help="Docker compose command override (e.g. 'podman compose').", ) -class _ProfileOverrideMixin(scfg.DataConfig): - profile = scfg.Value( +class _ProfileOverrideMixin(kw.Config): + profile = kw.Value( None, type=str, help='Active profile override (sets config.active_profile).', ) -class _PortOverridesMixin(scfg.DataConfig): - litellm_port = scfg.Value(None, type=int) - open_webui_port = scfg.Value(None, type=int) - postgres_port = scfg.Value(None, type=int) +class _PortOverridesMixin(kw.Config): + litellm_port = kw.Value(None, type=int) + open_webui_port = kw.Value(None, type=int) + postgres_port = kw.Value(None, type=int) -class _ClusterOverridesMixin(scfg.DataConfig): - namespace = scfg.Value( +class _ClusterOverridesMixin(kw.Config): + namespace = kw.Value( None, type=str, help='Kubernetes namespace for kubeai deployments.' ) - ingress_host = scfg.Value( + ingress_host = kw.Value( None, type=str, help='Ingress host (kubeai only).' ) - ingress_enabled = scfg.Value( + ingress_enabled = kw.Value( None, isflag=True, alias=['ingress'], @@ -73,24 +119,24 @@ class _ClusterOverridesMixin(scfg.DataConfig): ) -class _AllowUnsupportedMixin(scfg.DataConfig): - allow_unsupported = scfg.Value( +class _AllowUnsupportedMixin(kw.Config): + allow_unsupported = kw.Value( False, isflag=True, help='Allow validation errors when planning/rendering.', ) -class _SimulateHardwareMixin(scfg.DataConfig): - simulate_hardware = scfg.Value( +class _SimulateHardwareMixin(kw.Config): + simulate_hardware = kw.Value( None, type=str, help='Simulate GPUs: comma-separated NxM or M entries (e.g. 4x96, 2x80, "48,16" for a heterogeneous host). Useful for planning on smaller machines.', ) -class _AllowedGpusMixin(scfg.DataConfig): - allowed_gpus = scfg.Value( +class _AllowedGpusMixin(kw.Config): + allowed_gpus = kw.Value( None, type=str, help=( @@ -102,8 +148,8 @@ class _AllowedGpusMixin(scfg.DataConfig): ) -class _DisplayGpuMixin(scfg.DataConfig): - skip_display_gpus = scfg.Value( +class _DisplayGpuMixin(kw.Config): + skip_display_gpus = kw.Value( None, isflag=True, alias=['skip-display-gpus'], diff --git a/infer_stack/cli_equivalent.py b/infer_stack/cli_equivalent.py index 5d271d5e..a4b93448 100644 --- a/infer_stack/cli_equivalent.py +++ b/infer_stack/cli_equivalent.py @@ -26,6 +26,11 @@ def _kv_value(value: Any) -> str: return json.dumps(value) +#: What the gateway routes; the key comes from the CLI, never the log. +MODELS_CURL = ('curl -s "$(infer-stack env OPENAI_BASE_URL)/models" ' + '-H "Authorization: Bearer $(infer-stack env LITELLM_MASTER_KEY)"') + + #: Endpoint fields `catalog endpoint add` can express. _ENDPOINT_FIELDS = {'engine', 'model', 'host', 'public_name', 'reclaim', 'protocol', 'placement', 'runtime'} diff --git a/infer_stack/env_utils.py b/infer_stack/env_utils.py index eb6eb580..89bb8f29 100644 --- a/infer_stack/env_utils.py +++ b/infer_stack/env_utils.py @@ -93,7 +93,11 @@ def write_env_file(path: Path, values: dict[str, str]) -> None: if text and not text.endswith('\n'): text += '\n' - print(f'Write .env to {path}') + from ._log import logger + + # Debug, not stdout: a command's output (its --json above all) is not the + # place for an internal file write; `env set` confirms its own. + logger.debug('wrote {}', path) # Atomic replace: Docker Compose (or a fingerprint) must never read a # half-written file. import os diff --git a/infer_stack/leasing/backend.py b/infer_stack/leasing/backend.py index 59f7736b..d7023a9f 100644 --- a/infer_stack/leasing/backend.py +++ b/infer_stack/leasing/backend.py @@ -33,6 +33,10 @@ class Readiness: ready: bool detail: str = '' fatal: bool = False + #: The endpoint is waiting for resources held by others (a Kubernetes pod + #: the scheduler cannot place). The controller then frees room by + #: evicting idle keep-warm deployments, which no lease holds. + needs_room: bool = False class BackendTimeout(RuntimeError): @@ -70,17 +74,33 @@ class PlacementError(Exception): controller rolls back the just-created lease before raising, so a request that cannot be satisfied does not linger as a phantom ``live`` deployment with no container behind it. ``reasons`` carries the planner's per-deployment messages. + + ``capacity`` says whether free GPUs would have let it in, as opposed to a + request no amount of room admits (a render refusal, an unreadable + runtime, a host too small): only then is "free a GPU" the advice. """ - def __init__(self, deployment_ids, reasons): + def __init__(self, deployment_ids, reasons, *, capacity: bool = True): self.deployment_ids = list(deployment_ids) self.reasons = list(reasons) + self.capacity = bool(capacity) super().__init__( '; '.join(self.reasons) or f'could not place: {", ".join(self.deployment_ids)}' ) +def allocates_gpus(backend) -> bool: + """Whether ``backend`` allocates host GPU indices itself. + + True for Compose, which places on this host and records the GPUs each + deployment holds. False for a backend whose cluster schedules (KubeAI): + admission commits an empty allocation, and no GPU is ever "unresolved". + The one place the controller and the CLI ask this. + """ + return bool(getattr(backend, 'allocates_gpus', True)) + + @runtime_checkable class Backend(Protocol): """What the :class:`Controller` needs from a serving backend. @@ -126,15 +146,16 @@ class ConvergeBackend(Backend, Protocol): * ``last_assignments`` — deployment id -> GPU indices (empty for backends where the cluster schedules). - Backends without this surface fall back to the per-deployment - ``realize``/``teardown`` path in :meth:`Controller._render`. + A backend with only ``realize``/``teardown`` gets this surface from + :class:`SimpleAdmission`. """ last_unplaced: set[str] last_errors: list[str] last_assignments: dict[str, list[int]] - def converge(self, desired: list[Deployment], *, apply: bool = True): + def converge(self, desired: list[Deployment], *, apply: bool = True, + placement: Any = None): """Render the desired set to backend state; optionally apply it.""" ... @@ -150,14 +171,17 @@ def apply(self) -> bool | None: @runtime_checkable class AdmissionBackend(ConvergeBackend, Protocol): - """The surface admission mode uses (see ``Controller._admission_mode``). - - Today only :class:`~infer_stack.leasing.compose.ComposeBackend` has it: - strict residency, an in-memory placement ``preview``, and the Compose - network and adoption state the controller hands it. The controller - reaches these through ``Controller._admitting``, only after the - capability check, so the type checker sees one named capability instead - of attributes a minimal :class:`Backend` does not have. + """The surface the controller drives: every acquire goes through admission. + + Both real backends have it: strict residency and an in-memory + ``preview``; :class:`SimpleAdmission` supplies it for backends that + neither place nor inspect (the dry-run and test backends). Compose also takes the stable-address network and the + container-adoption table from the controller (``network``, + ``on_addresses``, ``adopted``); those are Compose-only, and the + controller hands them over only to a backend that declares them. The + controller reaches these through ``Controller._admitting``, only after + the capability check, so the type checker sees one named capability + instead of attributes a minimal :class:`Backend` does not have. """ network: dict[str, Any] | None @@ -169,6 +193,13 @@ def residency(self) -> Any: """A strict :class:`~infer_stack.leasing.residency.Residency`, or raise.""" ... + def instances(self) -> list[Any]: + """Every running unit, as :class:`~infer_stack.leasing.instances.Instance`. + + Built from :meth:`residency`, so it raises when that does. + """ + ... + def preview(self, desired: list[Deployment], placement: Any = None, *, approve: bool = False) -> Any: """Place and render ``desired`` without writing; ``(plan, rendered)``.""" @@ -250,9 +281,37 @@ def _save_sidecar(self, data: dict) -> None: self._atomic_write(self._state_file, json.dumps(data, indent=2)) + #: Digest of files an admission preview already had approved. + _preapproved: str | None = None + #: Digest of the files the last render produced (approved-digest guard). + last_planned_digest: str | None = None + #: Digest of the files the last preview produced. + last_preview_digest: str | None = None + + @staticmethod + def _planned_digest(planned: dict) -> str: + import hashlib + import json + + material = json.dumps({str(k): v for k, v in planned.items()}, sort_keys=True) + return hashlib.sha256(material.encode('utf-8')).hexdigest() + + def _preview_approval(self, planned: dict, *, approve: bool) -> None: + """Record a preview's digest; with ``approve``, ask now, not after commit. + + The render that follows the commit skips the prompt when it produces + the same files (see :meth:`_approve_changes`). + """ + self.last_preview_digest = self._planned_digest(planned) + if approve: + self._approve_changes(planned) + self._preapproved = self.last_preview_digest + def _approve_changes(self, planned: dict) -> None: """Show pending rendered-state changes and confirm them. + Files a preview already had approved (same digest) pass silently once. + ``planned`` maps target paths to their new content. When nothing actually changed, this is a quiet no-op. When ``assume_yes`` (scripts / non-interactive / ``--yes``), it applies after a one-line log. @@ -261,6 +320,9 @@ def _approve_changes(self, planned: dict) -> None: """ from .._log import logger + preapproved, self._preapproved = self._preapproved, None + if preapproved is not None and self._planned_digest(planned) == preapproved: + return changed = { p: text for p, text in planned.items() @@ -283,7 +345,113 @@ def _approve_changes(self, planned: dict) -> None: ) -class MemoryBackend: +@dataclass +class RenderPreview: + """The render half of a :meth:`SimpleAdmission.preview`: what it refused.""" + + unrenderable: set[str] + errors: list[str] + + +class SimpleAdmission: + """The admission surface for a backend that neither places nor inspects. + + For in-process backends (the dry-run and test backends): they allocate no + GPUs, their :meth:`residency` is what :meth:`observe` reports (each + deployment one warm instance), and converge/apply is ``realize`` / + ``teardown`` over the rendered set. A subclass that emulates capacity + overrides :meth:`plan`; one that emulates render failures overrides + :meth:`refuse`. Everything else goes through the one admission path the + real backends use. + """ + + allocates_gpus = False + last_preview_digest: str | None = None + + def plan(self, desired: list[Deployment], placement: Any = None): + """Which of ``desired`` fit; here, all of them, on no GPU.""" + from .placement import GpuPlan + + return GpuPlan(assignments={g.id: [] for g in desired}) + + def refuse(self, desired: list[Deployment]) -> dict[str, str]: + """``{deployment id: reason}`` for what cannot be rendered; none here.""" + return {} + + def residency(self): + from .residency import Container, Residency + + return Residency(by_deployment={ + gid: (Container(container_id=gid, deployment_id=gid, + state='running', labelled=True),) + for gid in sorted(self.observe()) + }) + + def instances(self): + """One in-process instance per realized deployment; no log to read.""" + from .instances import MEMORY, from_residency + + return from_residency(self.residency(), runtime=MEMORY) + + def _preview(self, desired: list[Deployment], placement: Any = None): + desired = list(desired) + plan = self.plan(desired, placement) + refused = self.refuse(desired) + return plan, RenderPreview(set(refused), [f'{g}: {why}' for g, why in refused.items()]) + + def preview(self, desired: list[Deployment], placement: Any = None, *, + approve: bool = False): + plan, rendered = self._preview(desired, placement) + self.last_preview_digest = repr(sorted( + (gid, list(gpus)) for gid, gpus in plan.assignments.items() + if gid not in rendered.unrenderable)) + return plan, rendered + + def converge(self, desired: list[Deployment], *, apply: bool = True, + placement: Any = None): + """Record the renderable, placed part of ``desired``; ``apply`` realizes it.""" + desired = list(desired) + plan, rendered = self._preview(desired, placement) + self.last_errors = list(plan.errors) + list(rendered.errors) + self.last_unplaced = { + g.id for g in desired + if g.id not in plan.assignments or g.id in rendered.unrenderable} + self.last_assignments = (dict(plan.assignments) + if self.allocates_gpus else {}) + self._rendered = {g.id: g for g in desired if g.id not in self.last_unplaced} + known = getattr(self, '_known', {}) + known.update({g.id: g for g in desired}) + self._known = known + if apply: + self.apply() + + def apply(self) -> None: + """Realize what the last converge rendered; tear down the rest.""" + rendered = getattr(self, '_rendered', {}) + for gid in sorted(self.observe() - set(rendered)): + deployment = getattr(self, '_known', {}).get(gid) + if deployment is not None: + self.teardown(deployment) + for gid, deployment in rendered.items(): + if gid not in self.observe(): + self.realize(deployment) + + # Set by converge (the controller reads them with a default before then). + last_unplaced: set[str] + last_errors: list[str] + last_assignments: dict[str, list[int]] + + def observe(self) -> set[str]: # pragma: no cover - provided by the backend + raise NotImplementedError + + def realize(self, deployment: Deployment) -> None: # pragma: no cover + raise NotImplementedError + + def teardown(self, deployment: Deployment) -> None: # pragma: no cover + raise NotImplementedError + + +class MemoryBackend(SimpleAdmission): """In-memory backend that records calls and has configurable readiness. Not a real serving backend — it never starts a process. It exists so the @@ -347,7 +515,7 @@ def set_ready( self.ready_overrides[(deployment_id, endpoint)] = ready -class NullBackend: +class NullBackend(SimpleAdmission): """A no-op backend that serves nothing — for ``--dry-run`` and ``leases``. It never starts a process. ``observe`` returns the empty set, so the diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py index f1c36049..0d2cb526 100644 --- a/infer_stack/leasing/compose.py +++ b/infer_stack/leasing/compose.py @@ -40,7 +40,6 @@ import hashlib import json -import re import time from dataclasses import dataclass, field from pathlib import Path @@ -49,14 +48,38 @@ import yaml from ..config import DEFAULT_PORTS, PINNED_IMAGES, default_state_paths -from ..env_utils import ensure_secret, parse_env_file, write_env_file +from ..env_utils import parse_env_file from ..probe import openai_ready from ..profile_runtime import simulator_args, vllm_args from .backend import ConvergeScaffold, Readiness +from .gateway import ( + LITELLM_CONFIG_FILENAME, + LITELLM_SERVICE, + NGINX_CONFIG_FILENAME, + ROUTE_RECONCILE_BOOTSTRAP_S, + ROUTE_RECONCILE_STEADY_S, + SALT_KEY_ENV, + Gateway, + _registry_incoming_from_catalog, + _registry_incoming_from_deployments, + render_front_door, +) from .launch import env_string, fill, translate_legacy -from .models import Deployment, is_reservation +from .models import Deployment, is_reservation, served_name +from .naming import ( # noqa: F401 (public names re-exported for callers) + OLLAMA_CONTAINER_PORT, + VLLM_CONTAINER_PORT, + _dns_slug, + _unique_vllm_service_name, + dns_slug, + ollama_service_name, + ollama_service_name_for, + vllm_service_name, + vllm_service_name_for, +) from .placement import plan_placement from .residency import ( # labels live beside the code that reads them back + ENGINE_LABEL, FINGERPRINT_LABEL, SERVICE_LABEL, COMPOSE_PROJECT_LABEL, @@ -68,43 +91,8 @@ LEASING_PROJECT = 'infer-stack' # docker compose project name for leased stacks VLLM_HOST_PORT_BASE = 18000 -VLLM_CONTAINER_PORT = 8000 -OLLAMA_CONTAINER_PORT = 11434 -LITELLM_CONTAINER_PORT = 4000 STATE_FILENAME = 'leasing-compose-state.json' COMPOSE_FILENAME = 'docker-compose.yml' -LITELLM_CONFIG_FILENAME = 'litellm_config.yaml' -LITELLM_SERVICE = 'litellm' -API_KEY_ENV = 'LITELLM_MASTER_KEY' -# The key LiteLLM encrypts credentials stored in its database with. Unset, it -# uses the master key -- so rotating the master key would make every -# DB-stored route undecryptable. `set_master_key` pins it to the pre-rotation -# master key the first time the key changes; it must never change after that. -SALT_KEY_ENV = 'LITELLM_SALT_KEY' - -# Dynamic-routing (admin-API) extras. When dynamic routing is on, the gateway's -# route table is managed live via LiteLLM's admin API against a Postgres-backed -# model store, instead of a static config file. See render_compose + -# ComposeBackend._reconcile_routes and docs/litellm-gateway-routing.md. -LITELLM_ROUTES_FILENAME = 'litellm_routes.json' # rendered desired route set -# Append-only route registry for static-superset mode: accumulates the semantic -# route inputs (served name / engine / host) of every catalog *and* every live -# deployment ever merged, across all runbooks sharing this state dir. The gateway -# `model_list` is rendered from the whole registry, so a converge under one -# runbook's catalog can no longer strip another's still-live routes, and once -# every catalog has been merged once the rendered config is byte-stable (the -# gateway is never recreated). See docs/litellm-gateway-routing.md and -# ComposeBackend._update_route_registry. -LITELLM_REGISTRY_FILENAME = 'litellm_registry.json' -LITELLM_REGISTRY_VERSION = 1 -POSTGRES_SERVICE = 'postgres-litellm' -POSTGRES_CONTAINER_PORT = 5432 -POSTGRES_DB_NAME = 'litellm' -POSTGRES_DB_USER = 'litellm' -DB_PASSWORD_ENV = 'LITELLM_DB_PASSWORD' # managed secret in the sidecar .env -# Marks a LiteLLM route as infer-stack-managed, so reconcile only ever deletes -# routes it created (never a model added by hand through the UI/admin API). -ROUTE_ID_PREFIX = 'isr-' VLLM_DEFAULTS = { 'gpu_memory_utilization': 0.9, @@ -113,91 +101,7 @@ 'max_num_seqs': 256, } -ENGINE_LABEL = 'infer-stack.engine' - - -def dns_slug(text: str) -> str: - """A lowercase ``[a-z0-9-]`` label safe as a compose service / DNS / - Kubernetes object name (shared by the compose and kubeai backends).""" - out = re.sub(r'[^a-z0-9]+', '-', str(text).lower()).strip('-') - return out or 'model' - - -_dns_slug = dns_slug # historical internal name - - -def vllm_service_name_for(served: str) -> str: - """Deterministic compose/DNS service name for a vLLM upstream: ``vllm-``. - - Derived purely from the served model name (vLLM's ``--served-model-name``, - the Open WebUI label, the alias the user chose), so it is identical whether - computed from a live :class:`Deployment` or from a catalog endpoint. That - stability is what lets the LiteLLM gateway carry a *static* route table (one - per catalog endpoint) whose upstream hosts match the containers when they - come up — so adding/removing models does not rewrite the gateway's config and - the gateway is never recreated (no "blip"); see :func:`_litellm_model_list`. - """ - return f'vllm-{_dns_slug(served)}' - - -def _unique_vllm_service_name(served: str, deployment_id: str) -> str: - """Per-deployment vLLM service/DNS name: ``vllm--``. - - The static-superset gateway needs a name derivable from the served model - *alone* (so a catalog route can address it without knowing the live - deployment) — but that deliberately drops the deployment id, which - **collapses every** ``--dedicated`` **deployment of one model onto a single - container** (hence one GPU). Dynamic routing manages the gateway's routes - live via the admin API, so the upstream host no longer has to be predictable - from the catalog. That frees us to give each deployment its **own** service, - so N dedicated deployments of one model become N containers on N GPUs. The - suffix is the deployment id's hex tail, keeping the name short and DNS-safe. - """ - tail = deployment_id.rsplit('-', 1)[-1][:8] or 'x' - return f'{vllm_service_name_for(served)}-{_dns_slug(tail)}' - - -def vllm_service_name(deployment: Deployment, *, unique: bool = False) -> str: - """Compose service name for a vLLM deployment (see :func:`vllm_service_name_for`). - - Default (``unique=False``, static-superset mode): deterministic from the - served model name only — *no* deployment-id suffix — so it matches the - gateway's pre-rendered route for that endpoint. Trade-off: two - *simultaneously desired* deployments that share a served name collide on this - name; under the static gateway the catalog endpoint is the unit, so that case - (including same-model ``--dedicated``) is unsupported. - - ``unique=True`` (dynamic-routing mode): append the deployment-id tail - (:func:`_unique_vllm_service_name`) so same-model dedicated deployments get - distinct containers/GPUs; the admin-API route table addresses each by name. - - Either way the container carries the ``infer-stack.deployment`` label. - :meth:`ComposeBackend.residency` correlates containers to deployments by that - label, so the choice of suffix does not affect it. (The lenient - :meth:`ComposeBackend.observe` still maps service names through the render - sidecar; it is for reporting, not for decisions that touch a GPU.) - """ - served = deployment.spec.get('served_model_name') or ( - sorted(deployment.served)[0] if deployment.served else deployment.id - ) - if unique: - return _unique_vllm_service_name(served, deployment.id) - return vllm_service_name_for(served) - - -def ollama_service_name_for(host: str) -> str: - """Deterministic service name for an Ollama daemon: ``ollama-``. - One daemon per host (Ollama coalesces tags onto it), so the host is the - stable key — matching :func:`vllm_service_name_for`'s role for vLLM so the - gateway's static route table addresses it regardless of which tags are live. - """ - return f'ollama-{_dns_slug(host)}' - - -def ollama_service_name(deployment: Deployment) -> str: - host = deployment.spec.get('host') or deployment.id - return ollama_service_name_for(host) class ApplyAborted(RuntimeError): @@ -257,93 +161,17 @@ def profile_images(profile: dict[str, Any]) -> list[str]: return sorted(wanted) -#: Restarts after which Docker's own bookkeeping says an engine is looping, -#: not loading. Two is already conclusive: a model that loads does not exit. -CRASH_LOOP_RESTARTS = 2 - -#: Engine log lines worth quoting verbatim, and what they mean. Each is -#: UNRECOVERABLE: the same container, restarted, fails the same way, so one -#: crash is already conclusive and there is nothing to wait for. -_ENGINE_ERROR_HINTS = ( - ('trust_remote_code', 'the model needs trust_remote_code=True ' - '(set `runtime.trust_remote_code: true` on the endpoint)'), - ('does not recognize this architecture', 'this engine cannot read the ' - "model's architecture (it may need trust_remote_code=True)"), - ('are not supported for now', 'this vLLM build does not implement the ' - "model's architecture"), - ('is not supported', 'this vLLM build does not implement the ' - "model's architecture"), - ('No supported config format', 'the model repository has no config this ' - 'engine can read'), - ('ValidationError', 'the engine rejected its own configuration'), - ('401 Client Error', 'the model is gated: set HF_TOKEN with ' - '`infer-stack env HF_TOKEN=...`'), - ('403 Client Error', 'the model is gated: set HF_TOKEN with ' - '`infer-stack env HF_TOKEN=...`'), +# Crash diagnosis is backend-neutral (the kubeai backend uses it too); these +# names stay importable from here. +from .diagnosis import ( # noqa: E402,F401 + CRASH_LOOP_RESTARTS, + _ENGINE_ERROR_HINTS, + _TRANSIENT_ENGINE_SIGNATURES, + _engine_error_summary, + classify_engine_log, + diagnose_startup, ) -#: Failures a RESTART CAN FIX: the hub was unreachable, a download was cut off, -#: a port was still held by the container we just replaced. `restart: -#: unless-stopped` exists for exactly these, so a restart count alone must not -#: condemn an engine -- the whole point of the policy is that the next attempt -#: succeeds. Weigh a new entry by one question: would running the same container -#: again plausibly work? If yes it belongs here; if no it belongs above. -_TRANSIENT_ENGINE_SIGNATURES = ( - 'Max retries exceeded', - 'Connection reset by peer', - 'Connection refused', - 'Temporary failure in name resolution', - 'Failed to resolve', - 'Read timed out', - 'ReadTimeoutError', - 'ConnectionError', - 'IncompleteRead', - 'Consistency check failed', # a truncated HF download - '429 Client Error', # hub rate limit - '500 Server Error', - '502 Server Error', - '503 Server Error', - '504 Server Error', - 'Address already in use', -) - - -def classify_engine_log(logs: str) -> str | None: - """``'fatal'``, ``'transient'``, or ``None`` when the log says neither. - - Order matters: an unrecoverable signature wins over a transient one, because - a hub timeout earlier in the same log does not make a rejected config - loadable. CUDA OOM counts as fatal — the allocation is deterministic, so the - restart repeats it — and it is the one class with a documented remedy - (:mod:`infer_stack.leasing.vram`). - """ - from .vram import looks_like_cuda_oom - - if not logs: - return None - if any(needle in logs for needle, _ in _ENGINE_ERROR_HINTS): - return 'fatal' - if looks_like_cuda_oom(logs): - return 'fatal' - if any(needle in logs for needle in _TRANSIENT_ENGINE_SIGNATURES): - return 'transient' - return None - - -def _engine_error_summary(logs: str) -> str: - """The engine's own error, quoted, with a hint when we recognise it.""" - from .vram import looks_like_cuda_oom - - if not logs.strip(): - return '; no engine log available (`infer-stack logs` for more)' - hint = next((note for needle, note in _ENGINE_ERROR_HINTS if needle in logs), None) - if hint is None and looks_like_cuda_oom(logs): - hint = 'the GPU ran out of memory for this configuration' - lines = [line.strip() for line in logs.splitlines() if line.strip()] - quoted = ' | '.join(lines[-3:])[:400] - summary = f'; last log: {quoted}' - return f'{summary}; likely cause: {hint}' if hint else summary - def _network_name() -> str: from .network import NETWORK_NAME @@ -502,9 +330,7 @@ def vllm_service_dict(deployment: Deployment) -> dict[str, Any]: # A deployment recorded before the generic launch fields may still carry # `serve_recipe`; read it as the fields it meant. runtime = translate_legacy(deployment.spec.get('runtime', {}) or {}) - served = deployment.spec.get('served_model_name') or ( - sorted(deployment.served)[0] if deployment.served else deployment.id - ) + served = served_name(deployment) return { 'served_model_name': served, 'tensor_parallel_size': int(runtime.get('tensor_parallel_size', 1) or 1), @@ -731,635 +557,6 @@ def _ollama_service( return service -def _vllm_route_entry( - model_name: str, served: str, api_base: str -) -> dict[str, Any]: - """One LiteLLM ``model_list`` entry routing ``model_name`` to a vLLM upstream. - - Shared by every render path (legacy per-deployment, catalog-superset, and - the route registry) so a registry-rendered entry can never drift from what - the catalog/deployment paths produce for the same endpoint.""" - return { - 'model_name': model_name, - 'litellm_params': { - 'model': f'openai/{served}', - 'api_base': api_base, - 'api_key': 'EMPTY', - }, - } - - -def _ollama_route_entry( - model_name: str, tag: str, api_base: str -) -> dict[str, Any]: - """One LiteLLM ``model_list`` entry routing ``model_name`` to an Ollama tag - (see :func:`_vllm_route_entry` for why this is factored out).""" - return { - 'model_name': model_name, - 'litellm_params': { - 'model': f'ollama/{tag}', - 'api_base': api_base, - }, - } - - -def _litellm_model_list( - deployments: list[Deployment], assignments: dict[str, list[int]] -) -> list[dict[str, Any]]: - """One LiteLLM ``model_list`` entry per served endpoint alias.""" - entries: list[dict[str, Any]] = [] - for deployment in sorted(deployments, key=lambda g: (g.created_at, g.id)): - if deployment.id not in assignments: - continue - if deployment.engine == 'vllm': - served = deployment.spec.get('served_model_name') or deployment.id - api_base = f'http://{vllm_service_name(deployment)}:8000/v1' - for endpoint in sorted(deployment.served): - entries.append(_vllm_route_entry(endpoint, served, api_base)) - elif deployment.engine == 'ollama': - api_base = f'http://{ollama_service_name(deployment)}:{OLLAMA_CONTAINER_PORT}' - for endpoint, payload in sorted(deployment.served.items()): - tag = payload.get('model', endpoint) - entries.append(_ollama_route_entry(endpoint, tag, api_base)) - return entries - - -def _litellm_model_list_from_catalog(catalog: Any) -> list[dict[str, Any]]: - """A *static superset* ``model_list``: one route per catalog endpoint. - - Unlike :func:`_litellm_model_list` (which routes only the currently-placed - deployments), this routes *every* catalog endpoint to its deterministic - upstream host (:func:`vllm_service_name_for` / :func:`ollama_service_name_for`). - The resulting config therefore depends only on the catalog, not on which - models happen to be up — so acquiring/releasing a model leaves the gateway's - config (and its container) untouched (no blip). A route whose upstream is not - currently running simply errors/cools-down until it comes up; the - ``router_settings`` below make that warmup self-healing. ``/v1/models`` lists - the whole catalog (some upstreams down) rather than only the live set. - - This static-superset path is the default. Its one limitation — it cannot give - same-model ``--dedicated`` deployments distinct upstreams, and cannot route - non-catalog acquires without a config change — is addressed by the opt-in - *dynamic routing* mode (``dynamic_routing=True``), which manages routes live - via LiteLLM's admin API against a Postgres model store (see - :func:`_litellm_routes`, :meth:`ComposeBackend._reconcile_routes`, and - ``docs/litellm-gateway-routing.md``). The two are mutually exclusive per - converge; this function is used only when dynamic routing is off. - """ - entries: list[dict[str, Any]] = [] - for name in sorted(getattr(catalog, 'endpoints', {})): - try: - req = catalog.resolve_endpoint(name) - except Exception: # noqa: BLE001 - a bad endpoint must not break the gateway - continue - if req.engine == 'vllm': - served = req.served.get('served_model_name') or name - api_base = ( - f'http://{vllm_service_name_for(served)}:{VLLM_CONTAINER_PORT}/v1' - ) - entries.append(_vllm_route_entry(name, served, api_base)) - elif req.engine == 'ollama': - host = req.spec.get('host') or req.host - tag = req.served.get('model') or name - api_base = ( - f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' - ) - entries.append(_ollama_route_entry(name, tag, api_base)) - return entries - - -# -- Route registry (static-superset persistence) -------------------------- -# -# The registry stores *semantic* route inputs (served name / engine / host), -# never rendered LiteLLM entries — render derives entries through the same -# helpers the catalog/deployment paths use (:func:`_litellm_model_list_from_registry`), -# so a future renderer change propagates to old registry rows automatically. -# All functions here are pure; the backend owns the file I/O and locking. - - -def _registry_incoming_from_catalog(catalog: Any) -> dict[str, dict[str, Any]]: - """Semantic route rows for every resolvable endpoint of ``catalog``. - - Mirrors :func:`_litellm_model_list_from_catalog`'s iteration (unresolvable - endpoints skipped) but emits registry rows keyed by endpoint name. A vLLM - row carries only ``served`` (the upstream host is re-derived at render via - :func:`vllm_service_name_for`); an Ollama row carries ``model`` (tag) + - ``host``.""" - incoming: dict[str, dict[str, Any]] = {} - for name in sorted(getattr(catalog, 'endpoints', {})): - try: - req = catalog.resolve_endpoint(name) - except Exception: # noqa: BLE001 - a bad endpoint must not break the gateway - continue - if req.engine == 'vllm': - served = req.served.get('served_model_name') or name - incoming[name] = {'engine': 'vllm', 'served': served} - elif req.engine == 'ollama': - host = req.spec.get('host') or req.host - tag = req.served.get('model') or name - incoming[name] = {'engine': 'ollama', 'model': tag, 'host': host} - return incoming - - -def _registry_incoming_from_deployments( - deployments: list[Deployment], assignments: dict[str, list[int]] -) -> dict[str, dict[str, Any]]: - """Semantic route rows for every *placed* deployment in ``assignments``. - - ``deployments`` is the full ``desired`` set (which spans all runbooks via - the shared ledger), so this keeps non-catalog / dedicated acquires routable - and — because the registry persists — routable past release. One row per key - of ``deployment.served`` (a coalesced deployment can back several endpoint - aliases). Only ``vllm``/``ollama`` engines contribute; ``RESERVED_ENGINE`` - and unknown engines render no service, so they contribute no row — exactly - as :func:`render_compose`'s service loop skips them. - - The vLLM ``served`` uses the same fallback chain as :func:`vllm_service_name` - (``spec['served_model_name'] or sorted(served)[0] or id``), so a - catalog-listed endpoint acquired live reduces to the identical row a catalog - merge produces — live-vs-released status never moves the rendered bytes.""" - incoming: dict[str, dict[str, Any]] = {} - for deployment in deployments: - if deployment.id not in assignments: - continue - if deployment.engine == 'vllm': - served = deployment.spec.get('served_model_name') or ( - sorted(deployment.served)[0] if deployment.served else deployment.id - ) - for endpoint in sorted(deployment.served): - incoming[endpoint] = {'engine': 'vllm', 'served': served} - elif deployment.engine == 'ollama': - host = deployment.spec.get('host') or deployment.id - for endpoint, payload in sorted(deployment.served.items()): - tag = payload.get('model', endpoint) - incoming[endpoint] = { - 'engine': 'ollama', - 'model': tag, - 'host': host, - } - return incoming - - -def _litellm_model_list_from_registry( - registry: dict[str, Any] -) -> list[dict[str, Any]]: - """Render the gateway ``model_list`` from the whole accumulated registry. - - Iterates ``sorted(entries)`` (determinism, §8) and derives each upstream - ``api_base`` through the live naming helpers, so the registry never becomes - a rendered-config parse surface.""" - entries: list[dict[str, Any]] = [] - rows = registry.get('entries', {}) if isinstance(registry, dict) else {} - for name in sorted(rows): - row = rows[name] - if not isinstance(row, dict): - continue - engine = row.get('engine') - if engine == 'vllm': - served = row.get('served') or name - api_base = ( - f'http://{vllm_service_name_for(served)}:{VLLM_CONTAINER_PORT}/v1' - ) - entries.append(_vllm_route_entry(name, served, api_base)) - elif engine == 'ollama': - tag = row.get('model') or name - host = row.get('host') or name - api_base = ( - f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' - ) - entries.append(_ollama_route_entry(name, tag, api_base)) - return entries - - -def _merge_route_registry( - existing: dict[str, Any], incoming: dict[str, dict[str, Any]] -) -> tuple[dict[str, Any], list[str]]: - """Merge ``incoming`` semantic rows into ``existing`` (append-only). - - Idempotent (merging identical rows is a no-op) and additive (never removes a - row). On a conflict — same key, different row — *incoming wins* and a warning - naming both definitions is emitted; the changed definition changes the - rendered bytes, which is the one justified recreate. The existing ``version`` - is preserved (an unknown version merged under is not silently rewritten to - the current schema; see :meth:`ComposeBackend._load_route_registry`).""" - version = LITELLM_REGISTRY_VERSION - entries: dict[str, dict[str, Any]] = {} - if isinstance(existing, dict): - version = existing.get('version', LITELLM_REGISTRY_VERSION) - prior = existing.get('entries') - if isinstance(prior, dict): - entries = {k: v for k, v in prior.items()} - warnings: list[str] = [] - for name in sorted(incoming): - row = incoming[name] - if name in entries and entries[name] != row: - warnings.append( - f"route {name!r} redefined: {entries[name]} -> {row} " - '(incoming wins; gateway will be recreated once)' - ) - entries[name] = row - return {'version': version, 'entries': entries}, warnings - - -def _seed_registry_from_litellm_config( - config_text: str, -) -> tuple[dict[str, Any], list[str]]: - """One-shot upgrade seed: recover registry rows from a rendered - ``litellm_config.yaml`` so the first post-upgrade converge does not strip - the other runbooks' routes. - - Single-format, migration-time only (no cross-version promise): ``openai/`` - inverts *exactly* to ``{engine: vllm, served}``. Ollama rows are skipped with - a warning — the host survives only as a non-invertible ``dns_slug`` inside - ``api_base`` — and re-enter the registry at the next converge that has them - in its catalog or live set. Anything else unparseable is likewise skipped.""" - entries: dict[str, dict[str, Any]] = {} - warnings: list[str] = [] - try: - data = yaml.safe_load(config_text) or {} - except Exception: # noqa: BLE001 - a torn file must not brick seeding - return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}}, [ - 'seed: litellm_config.yaml is unparseable; starting an empty registry' - ] - for entry in data.get('model_list', []) or []: - name = entry.get('model_name') - model = (entry.get('litellm_params') or {}).get('model', '') - if not name: - continue - if isinstance(model, str) and model.startswith('openai/'): - entries[name] = {'engine': 'vllm', 'served': model[len('openai/'):]} - elif isinstance(model, str) and model.startswith('ollama/'): - warnings.append( - f"seed: skipping Ollama route {name!r} (host not recoverable " - 'from the rendered api_base; it re-enters at its next converge)' - ) - else: - warnings.append(f'seed: skipping unparseable route {name!r}') - return {'version': LITELLM_REGISTRY_VERSION, 'entries': entries}, warnings - - -def _dump_route_registry(registry: dict[str, Any]) -> str: - """Canonical, byte-stable serialization (§3): sorted keys + trailing - newline. A nondeterministic dump would manufacture phantom hash changes.""" - return json.dumps(registry, sort_keys=True, indent=2) + '\n' - - -def _route_id(deployment_id: str, endpoint: str) -> str: - """Deterministic LiteLLM model id for one (deployment, endpoint) route. - - Stable across converges, so route reconcile (:meth:`ComposeBackend. - _reconcile_routes`) can identify one logical route across renders. A route - whose id disappears is deleted by exactly this id; a route whose id remains - but whose observable routing semantics drifted is replaced under the same id. - The ``isr-`` prefix marks it - infer-stack-managed so reconcile never deletes a model someone added by hand. - """ - digest = hashlib.sha256(f'{deployment_id}|{endpoint}'.encode()).hexdigest() - return f'{ROUTE_ID_PREFIX}{digest[:32]}' - - -def _litellm_routes( - deployments: list[Deployment], assignments: dict[str, list[int]] -) -> list[dict[str, Any]]: - """Desired LiteLLM route set for the *live* deployments (dynamic routing). - - One entry per (placed deployment, served endpoint), addressing the - deployment's **own** unique upstream service (:func:`vllm_service_name` with - ``unique=True``). Several dedicated deployments of the same model therefore - yield several entries that share one public ``model_name`` but point at - distinct upstreams — LiteLLM load-balances the alias across them, so each - runs on its own GPU while clients still ask for the single name. Each entry - carries a deterministic ``model_info.id`` (:func:`_route_id`) so applying the - set via the admin API is an idempotent diff, not fire-and-forget calls. - """ - entries: list[dict[str, Any]] = [] - for deployment in sorted(deployments, key=lambda g: (g.created_at, g.id)): - if deployment.id not in assignments: - continue - if deployment.engine == 'vllm': - served = deployment.spec.get('served_model_name') or deployment.id - api_base = ( - f'http://{vllm_service_name(deployment, unique=True)}' - f':{VLLM_CONTAINER_PORT}/v1' - ) - for endpoint in sorted(deployment.served): - entries.append( - { - 'model_name': endpoint, - 'litellm_params': { - 'model': f'openai/{served}', - 'api_base': api_base, - 'api_key': 'EMPTY', - }, - 'model_info': {'id': _route_id(deployment.id, endpoint)}, - } - ) - elif deployment.engine == 'ollama': - api_base = ( - f'http://{ollama_service_name(deployment)}:{OLLAMA_CONTAINER_PORT}' - ) - for endpoint, payload in sorted(deployment.served.items()): - tag = payload.get('model', endpoint) - entries.append( - { - 'model_name': endpoint, - 'litellm_params': { - 'model': f'ollama/{tag}', - 'api_base': api_base, - }, - 'model_info': {'id': _route_id(deployment.id, endpoint)}, - } - ) - return entries - - -CONFIG_HASH_LABEL = 'infer-stack.config-hash' - - -def _postgres_service( - images: dict[str, str], state: dict[str, str] -) -> dict[str, Any]: - """Postgres backing LiteLLM's runtime model store (dynamic routing only). - - LiteLLM's admin API (``/model/new`` / ``/model/delete``) only functions with - ``STORE_MODEL_IN_DB=true`` + a database, so dynamic routing needs a DB. This - is an **internal** service (no published host port); LiteLLM reaches it on - the compose network at ``postgres-litellm:5432``. The password is the managed - :data:`DB_PASSWORD_ENV` secret in the sidecar ``.env`` (interpolated by - ``docker compose --env-file``), so it never appears literally in the YAML. - The healthcheck lets the litellm service ``depends_on`` it (condition: - service_healthy) so the gateway only starts once the DB can accept queries. - """ - data_path = state.get('postgres_litellm') or str( - Path(next(iter(state.values()), '.')).parent / 'postgres-litellm' - ) - return { - 'image': images.get('postgres', PINNED_IMAGES['postgres']), - 'environment': { - 'POSTGRES_USER': POSTGRES_DB_USER, - 'POSTGRES_PASSWORD': '${' + DB_PASSWORD_ENV + '}', - 'POSTGRES_DB': POSTGRES_DB_NAME, - }, - 'volumes': [f'{data_path}:/var/lib/postgresql/data'], - 'restart': 'unless-stopped', - 'labels': {ENGINE_LABEL: 'postgres'}, - 'healthcheck': { - 'test': [ - 'CMD-SHELL', - f'pg_isready -U {POSTGRES_DB_USER} -d {POSTGRES_DB_NAME}', - ], - 'interval': '5s', - 'timeout': '5s', - 'retries': 30, - 'start_period': '30s', - }, - } - - -def _litellm_service( - service_names: list[str], - host_port: int, - images: dict[str, str], - aux_dir: str, - master_key: str | None = None, - config_hash: str | None = None, - *, - dynamic_routing: bool = False, - salt_key: bool = False, -) -> dict[str, Any]: - # Reference the managed key via ${...} rather than baking the literal secret - # into the compose YAML. Its value lives in the sidecar .env next to the - # compose file (written by master_key()), which `docker compose --env-file` - # loads for interpolation — so the container and the readiness probe (which - # reads the same .env) still agree regardless of the caller's shell env. - key_value = ( - '${' + API_KEY_ENV + '}' - if master_key is not None - else '${' + API_KEY_ENV + ':-sk-local}' - ) - environment = {API_KEY_ENV: key_value} - if salt_key: - # Only when the .env has one: LiteLLM treats an EMPTY salt as a key, - # so a `${...:-}` default would silently change the encryption key. - environment[SALT_KEY_ENV] = '${' + SALT_KEY_ENV + '}' - if dynamic_routing: - # DB-backed runtime model store so the admin API (/model/new, - # /model/delete) works; the gateway then never needs recreating to learn - # a route. Both are read from the env by LiteLLM. The password is - # interpolated from the sidecar .env, so no secret lands in the YAML. - environment['DATABASE_URL'] = ( - f'postgresql://{POSTGRES_DB_USER}:${{{DB_PASSWORD_ENV}}}' - f'@{POSTGRES_SERVICE}:{POSTGRES_CONTAINER_PORT}/{POSTGRES_DB_NAME}' - ) - environment['STORE_MODEL_IN_DB'] = 'True' - labels = {ENGINE_LABEL: 'litellm'} - if config_hash is not None: - # LiteLLM reads its routing config once at startup; the file is bind- - # mounted, so a config change alone does NOT change this service's spec - # and `docker compose up -d` would leave the old container (and old - # routes) running. Stamping the config hash onto a label makes the spec - # change exactly when the config does, so converge recreates LiteLLM and - # it picks up new/removed aliases. Without this, coalescing a second - # alias onto a live deployment never becomes routable (readiness times out). - labels[CONFIG_HASH_LABEL] = config_hash - service: dict[str, Any] = { - 'image': images['litellm'], - 'command': [ - '--config', - '/etc/litellm/config.yaml', - '--port', - str(LITELLM_CONTAINER_PORT), - ], - 'ports': [f'{host_port}:{LITELLM_CONTAINER_PORT}'], - 'volumes': [f'{aux_dir}/{LITELLM_CONFIG_FILENAME}:/etc/litellm/config.yaml:ro'], - 'environment': environment, - 'restart': 'unless-stopped', - 'labels': labels, - } - if dynamic_routing: - # Wait for the DB to accept queries before the gateway boots; do NOT add - # per-model depends_on (that would churn the spec, i.e. blip, on every - # model change). The route table is filled in afterward via the API. - service['depends_on'] = { - POSTGRES_SERVICE: {'condition': 'service_healthy'} - } - elif service_names: - # Only wait on upstreams when there are any (zero models -> empty gateway). - service['depends_on'] = sorted(service_names) - return service - - -OPEN_WEBUI_SERVICE = 'open-webui' -OPEN_WEBUI_CONTAINER_PORT = 8080 - -NGINX_SERVICE = 'reverse-proxy' -NGINX_CONTAINER_PORT = 80 -NGINX_CONFIG_FILENAME = 'nginx.conf' - - -def _open_webui_service( - host_port: int, - images: dict[str, str], - state: dict[str, str], - master_key: str | None, - *, - openai_urls: list[str] | None = None, - ollama_urls: list[str] | None = None, - depends_on: list[str] | None = None, -) -> dict[str, Any]: - """A managed Open WebUI pointed at whatever front door is available. - - Open WebUI holds two independent kinds of connection, wired here from the - rendered services: - - * **OpenAI** (``openai_urls``) — the chat/completions front door. This is the - LiteLLM gateway when it is enabled (so every declared endpoint alias is - reachable at one URL); with LiteLLM off it falls back to the rendered - upstreams' own ``/v1`` (a single vLLM/Ollama service, or several joined as - ``OPENAI_API_BASE_URLS``). With nothing to point at, the OpenAI API is - disabled rather than left dangling. - * **Ollama** (``ollama_urls``) — the *native* Ollama API of any rendered - Ollama daemon. This is what lets you pull/run/delete models from the UI - and have the daemon load them on demand, independent of LiteLLM — i.e. a - true drop-in for a hand-run ``ollama`` + Open WebUI stack. - - The spec is kept as independent of which models are live as it can be: the - LiteLLM URL is fixed, and the Ollama daemon's service name is its stable - structural id, so adding/removing other models does not rewrite this service - and ``docker compose up -d`` leaves the UI running (the legacy "the UI never - blinks" behavior). Chat history persists under the data dir. - """ - # Reference the managed key via ${...} (resolved from the sidecar .env, see - # _litellm_service) instead of inlining the secret into the compose YAML. - key_value = ( - '${' + API_KEY_ENV + '}' - if master_key is not None - else '${' + API_KEY_ENV + ':-sk-local}' - ) - data_path = state.get('open_webui') or str( - Path(next(iter(state.values()), '.')).parent / 'open-webui' - ) - openai_urls = list(openai_urls or []) - ollama_urls = list(ollama_urls or []) - env: dict[str, str] = { - # Single-user workstation default; the port shouldn't be exposed - # publicly. Tracked as a knob in dev/leasing-followups.md. - 'WEBUI_AUTH': 'False', - } - if openai_urls: - env['ENABLE_OPENAI_API'] = 'True' - if len(openai_urls) == 1: - env['OPENAI_API_BASE_URL'] = openai_urls[0] - else: - env['OPENAI_API_BASE_URLS'] = ';'.join(openai_urls) - env['OPENAI_API_KEY'] = key_value - else: - env['ENABLE_OPENAI_API'] = 'False' - if ollama_urls: - env['ENABLE_OLLAMA_API'] = 'True' - if len(ollama_urls) == 1: - env['OLLAMA_BASE_URL'] = ollama_urls[0] - else: - env['OLLAMA_BASE_URLS'] = ';'.join(ollama_urls) - else: - env['ENABLE_OLLAMA_API'] = 'False' - service: dict[str, Any] = { - 'image': images['open_webui'], - 'ports': [f'{host_port}:{OPEN_WEBUI_CONTAINER_PORT}'], - 'environment': env, - 'volumes': [f'{data_path}:/app/backend/data'], - 'restart': 'unless-stopped', - 'labels': {ENGINE_LABEL: 'open-webui'}, - } - if depends_on: - service['depends_on'] = sorted(depends_on) - return service - - -def _nginx_conf(*, litellm: bool, ui: bool) -> str: - """A minimal HTTP reverse-proxy conf: one origin, path-routed. - - ``/v1/`` -> the LiteLLM gateway (the OpenAI API), ``/`` -> Open WebUI (or the - gateway when there's no UI). Plain HTTP — no TLS, no auth — so the value is - "one port, nothing to remember", not security. The ``map`` is valid here - because a ``conf.d/*.conf`` file is included in nginx's ``http`` context. - """ - api = f'http://{LITELLM_SERVICE}:{LITELLM_CONTAINER_PORT}' - locations = '' - if litellm: - locations += ( - ' location /v1/ {\n' - f' proxy_pass {api}/v1/;\n' - ' proxy_set_header Host $host;\n' - ' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n' - ' proxy_set_header X-Forwarded-Proto $scheme;\n' - ' proxy_read_timeout 600s;\n' - ' }\n' - ) - # `/` serves the UI when present, else the gateway (so hitting the host root - # still lands somewhere useful). Upgrade headers keep Open WebUI's websockets - # working; client_max_body_size 0 allows large uploads. - if ui: - root = f'http://{OPEN_WEBUI_SERVICE}:{OPEN_WEBUI_CONTAINER_PORT}' - elif litellm: - root = api - else: - root = '' - if root: - locations += ( - ' location / {\n' - f' proxy_pass {root};\n' - ' proxy_http_version 1.1;\n' - ' proxy_set_header Upgrade $http_upgrade;\n' - ' proxy_set_header Connection $connection_upgrade;\n' - ' proxy_set_header Host $host;\n' - ' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n' - ' proxy_set_header X-Forwarded-Proto $scheme;\n' - ' }\n' - ) - return ( - 'map $http_upgrade $connection_upgrade {\n' - ' default upgrade;\n' - " '' close;\n" - '}\n\n' - 'server {\n' - f' listen {NGINX_CONTAINER_PORT};\n' - ' server_name _;\n' - ' client_max_body_size 0;\n' - f'{locations}' - '}\n' - ) - - -def _nginx_service( - host_port: int, - images: dict[str, str], - *, - aux_dir: str, - depends_on: list[str], - config_path: str | None = None, - config_hash: str | None = None, -) -> dict[str, Any]: - # BYO config (config_path) is mounted verbatim; otherwise the generated - # nginx.conf in the state dir is used. - mount = config_path or f'{aux_dir}/{NGINX_CONFIG_FILENAME}' - labels = {ENGINE_LABEL: 'nginx'} - if config_hash is not None: - # Same trick as LiteLLM: the conf is bind-mounted, so stamp its hash on a - # label to force a recreate when the routing changes. - labels[CONFIG_HASH_LABEL] = config_hash - service: dict[str, Any] = { - 'image': images['nginx'], - 'ports': [f'{host_port}:{NGINX_CONTAINER_PORT}'], - 'volumes': [f'{mount}:/etc/nginx/conf.d/default.conf:ro'], - 'restart': 'unless-stopped', - 'labels': labels, - } - if depends_on: - service['depends_on'] = sorted(depends_on) - return service - - def render_compose( deployments: list[Deployment], assignments: dict[str, list[int]], @@ -1381,6 +578,8 @@ def render_compose( catalog: Any = None, route_registry: dict[str, Any] | None = None, dynamic_routing: bool = False, + upstream_routes: list[dict[str, Any]] | None = None, + ui_run_as: str | None = None, ) -> RenderedCompose: """Render a compose project for the placed deployments. @@ -1457,133 +656,24 @@ def render_compose( ollama_native_urls.append(f'http://{name}:{OLLAMA_CONTAINER_PORT}') service_map[name] = deployment.id - litellm_config = None - litellm_routes = None - # The front door (gateway + UI) is rendered whenever it's enabled, even with - # zero models — it's a standing entry point, not a per-model service. So - # releasing/evicting every model leaves an empty gateway (and an empty Open - # WebUI picker) up instead of tearing the whole stack down; only an explicit - # `stack down` removes it. With no models the model_list is simply empty. - if litellm: - # Three route-table strategies, in order of preference: - # * DYNAMIC ROUTING: the rendered config is a STATIC base (empty - # model_list); the real routes live in Postgres and are applied to the - # running gateway via the admin API (see _reconcile_routes). The config - # hash never changes as models come/go, so the gateway is never - # recreated — no blip, and per-deployment routing works (so same-model - # --dedicated deployments each get their own upstream). - # * ROUTE REGISTRY (static-superset default from ComposeBackend): render - # from the whole accumulated registry (every catalog + live deployment - # ever merged, across all runbooks). Byte-stable once seeded, so the - # gateway is never recreated and a cross-catalog converge can no longer - # strip another runbook's routes. The backend loads/merges/writes the - # registry and passes the merged dict in; this function stays pure. - # * STATIC SUPERSET (catalog): one route per catalog endpoint to a - # deterministic host; config depends only on the catalog, so the - # gateway is not recreated as models come/go (no blip) but same-model - # dedicated collapses to one upstream. Unreachable from ComposeBackend - # once the registry is wired; kept for direct callers/tests. - # * LEGACY (no catalog): route only the placed deployments; churns the - # config (and recreates the gateway) on every model change. - if dynamic_routing: - entries: list[dict[str, Any]] = [] - litellm_routes = _litellm_routes(deployments, assignments) - litellm_depends: list[str] = [] - elif route_registry is not None: - entries = _litellm_model_list_from_registry(route_registry) - litellm_depends = [] # no per-model depends_on -> no churn - elif catalog is not None: - entries = _litellm_model_list_from_catalog(catalog) - litellm_depends = [] # no per-model depends_on -> no churn - else: - entries = _litellm_model_list(deployments, assignments) - litellm_depends = list(service_map) - litellm_config = yaml.safe_dump( - { - 'model_list': entries, - 'general_settings': { - 'master_key': f'os.environ/{API_KEY_ENV}' - }, - # An upstream vLLM/Ollama is unreachable only briefly, while it - # loads its model (LiteLLM does not wait for upstream health to - # start). Retry transient connection errors and don't park a - # model in a long cooldown, so the warmup window is self-healing - # instead of surfacing as client 500s ("Connection error. - # Received Model Deployment=…"). - 'router_settings': { - 'num_retries': 3, - 'timeout': 600, - 'cooldown_time': 5, - 'allowed_fails': 100, - }, - }, - sort_keys=False, - ) - config_hash = hashlib.sha256( - litellm_config.encode('utf-8') - ).hexdigest()[:12] - if dynamic_routing: - services[POSTGRES_SERVICE] = _postgres_service(images, state) - services[LITELLM_SERVICE] = _litellm_service( - litellm_depends, - litellm_port, - images, - str(aux_dir or '.'), - master_key=litellm_master_key, - config_hash=config_hash, - dynamic_routing=dynamic_routing, - salt_key=litellm_salt_key, - ) - - # Open WebUI is its own standing front door, rendered whenever ``ui`` is set - # — it does NOT require LiteLLM. Its OpenAI connection prefers the gateway - # (one URL covers every alias) and falls back to the rendered vLLM upstreams' - # own /v1 when there is no gateway. Its native Ollama connection always - # points straight at any Ollama daemon, so you can pull/run models from the - # UI and have the daemon load them on demand — a true drop-in for a - # hand-run ollama + Open WebUI stack. depends_on lists only LiteLLM (the one - # service guaranteed present alongside the UI); the per-model upstreams come - # and go, so the UI tolerates them being absent rather than hard-depending. - # With a gateway the UI is a standing front door (renders even at zero - # models). Without one it is only meaningful pointed at a live upstream, so - # render it only when there is something to connect to — otherwise an empty - # desired set has nothing to run and converge tears the project down. - if ui and (litellm or vllm_v1_urls or ollama_native_urls): - if litellm: - openai_urls = [f'http://{LITELLM_SERVICE}:{LITELLM_CONTAINER_PORT}/v1'] - ui_depends = [LITELLM_SERVICE] - else: - openai_urls = list(vllm_v1_urls) - ui_depends = [] - services[OPEN_WEBUI_SERVICE] = _open_webui_service( - ui_port, - images, - state, - litellm_master_key, - openai_urls=openai_urls, - ollama_urls=ollama_native_urls, - depends_on=ui_depends, - ) - - # Optional single-port HTTP reverse proxy fronting the gateway (+ UI). Needs - # the gateway, so it's only rendered alongside litellm. - nginx_config = None - if reverse_proxy and litellm: - depends = [LITELLM_SERVICE] + ([OPEN_WEBUI_SERVICE] if ui else []) - if reverse_proxy_config: - services[NGINX_SERVICE] = _nginx_service( - reverse_proxy_port, images, aux_dir=str(aux_dir or '.'), - depends_on=depends, config_path=reverse_proxy_config, - ) - else: - nginx_config = _nginx_conf(litellm=litellm, ui=ui) - services[NGINX_SERVICE] = _nginx_service( - reverse_proxy_port, images, aux_dir=str(aux_dir or '.'), - depends_on=depends, - config_hash=hashlib.sha256( - nginx_config.encode('utf-8') - ).hexdigest()[:12], - ) + front = render_front_door( + deployments, assignments, + engine_services=list(service_map), + vllm_v1_urls=vllm_v1_urls, ollama_native_urls=ollama_native_urls, + images=images, state=state, + litellm=litellm, litellm_port=litellm_port, + litellm_master_key=litellm_master_key, litellm_salt_key=litellm_salt_key, + ui=ui, ui_port=ui_port, + reverse_proxy=reverse_proxy, reverse_proxy_port=reverse_proxy_port, + reverse_proxy_config=reverse_proxy_config, aux_dir=aux_dir, + catalog=catalog, route_registry=route_registry, + dynamic_routing=dynamic_routing, upstream_routes=upstream_routes, + ui_run_as=ui_run_as, + ) + services.update(front.services) + litellm_config = front.litellm_config + nginx_config = front.nginx_config + litellm_routes = front.litellm_routes for name, svc in services.items(): svc.setdefault('labels', {})[SERVICE_LABEL] = name @@ -1607,15 +697,6 @@ def render_compose( DOCKER_TIMEOUT_CONVERGE = 1800.0 # compose up / down DOCKER_TIMEOUT_PULL = 3600.0 # pull, manifest inspect, in-container model pulls -#: Budget for reconciling dynamic routes against the gateway: listing, POSTs -#: and verification together. This default is the bootstrap budget (a fresh -#: gateway waits on Postgres health and runs DB migrations); it preserves the -#: previous 90 x 2 s listing retry. -ROUTE_RECONCILE_BOOTSTRAP_S = 180.0 -#: Budget when the gateway was already running before this apply (steady state). -#: Short, because the controller holds its host-wide lock while applying; on -#: expiry the change stays pending and the next applying operation retries. -ROUTE_RECONCILE_STEADY_S = 20.0 def _docker_timeout(args: list[str]) -> float: @@ -1668,6 +749,52 @@ def docker_environment(environ: dict[str, str] | None = None) -> dict[str, str]: return env +#: Read-only docker queries in flight (ps, inspect, ...). A long-running +#: process that is exiting (the TUI) kills these rather than wait for them; +#: nothing that changes state is ever in this set. +_RUNNING_QUERIES: set = set() + + +_READ_VERBS = frozenset({'ps', 'inspect', 'logs', 'version', 'images', 'ls', + 'info', 'port', 'top', 'config'}) +_MUTATING_VERBS = frozenset({'up', 'down', 'rm', 'create', 'pull', 'run', 'exec', + 'start', 'stop', 'kill', 'pause', 'unpause', 'connect', + 'disconnect', 'prune', 'build', 'push', 'tag', 'restart'}) + + +def _read_only(args: list[str]) -> bool: + """A docker command that only reads state, and so is safe to abandon. + + Example: + >>> _read_only(['docker', 'compose', '-p', 'x', 'ps', '--format', 'json']) + True + >>> _read_only(['docker', 'network', 'create', 'n']) + False + """ + words = set(args) + return bool(words & _READ_VERBS) and not words & _MUTATING_VERBS + + +def cancel_running_queries() -> int: + """Kill every in-flight read-only docker query; how many were killed. + + For a process that is exiting: its background refresh would otherwise + hold the exit until ``docker compose ps`` returns (measured 0.66 s on the + guest, more on a loaded host). Mutations (up, rm, pull) are never killed. + """ + import os + import signal + + killed = 0 + for proc in list(_RUNNING_QUERIES): + try: + os.killpg(proc.pid, signal.SIGKILL) + killed += 1 + except (ProcessLookupError, PermissionError): + pass + return killed + + def _default_docker_run( args: list[str], *, timeout: float | None = None, stderr_lines: Callable[[str], None] | None = None, @@ -1688,11 +815,23 @@ def _default_docker_run( from .backend import BackendTimeout bound = _docker_timeout(args) if timeout is None else timeout + # With no stderr handler, stderr still reaches the terminal as it is + # written (docker's progress), through a pipe of our own so its last lines + # can also go into the error: "port is already allocated" belongs in the + # message, not only scrolled past above it. + tee = None + if stderr_lines is None: + tee = _StderrTee() proc = subprocess.Popen( args, stdout=subprocess.PIPE, text=True, start_new_session=True, env=docker_environment(), - stderr=subprocess.PIPE if stderr_lines is not None else None, + stderr=subprocess.PIPE if tee is None else tee.write_end, ) + if tee is not None: + tee.start() + query = _read_only(args) + if query: + _RUNNING_QUERIES.add(proc) def kill_group(): try: os.killpg(proc.pid, signal.SIGKILL) @@ -1716,15 +855,58 @@ def kill_group(): # session, so without this it would keep running unattended. kill_group() raise + finally: + if query: + _RUNNING_QUERIES.discard(proc) if stderr_lines is not None: for line in (err or '').splitlines(): if line.strip(): stderr_lines(line.rstrip()) + if tee is not None: + err = tee.finish() if proc.returncode != 0: raise subprocess.CalledProcessError(proc.returncode, args, output=out, stderr=err) return out +class _StderrTee: + """A child's stderr, echoed to ours as it arrives, with its last lines kept.""" + + def __init__(self, keep: int = 20): + import collections + import os + + self._read_end, self.write_end = os.pipe() + self._tail: collections.deque = collections.deque(maxlen=keep) + self._thread = None + + def start(self) -> None: + import os + import threading + + os.close(self.write_end) # the child holds its copy + + def pump(): + import sys + + with os.fdopen(self._read_end, 'r', errors='replace') as stream: + for line in stream: + self._tail.append(line) + try: + sys.stderr.write(line) + sys.stderr.flush() + except Exception: # noqa: BLE001 - never fail the command + pass + + self._thread = threading.Thread(target=pump, daemon=True) + self._thread.start() + + def finish(self) -> str: + if self._thread is not None: + self._thread.join(timeout=5.0) + return ''.join(self._tail) + + def _communicate_streaming(proc, bound: float, on_line) -> tuple[str, str | None]: """``proc.communicate(timeout=bound)``, handing each stdout line to ``on_line``. @@ -1834,24 +1016,6 @@ def _parse_ps(out: str) -> set[str]: return running -def set_master_key(env_path: Path, key: str) -> None: - """Replace the LiteLLM master key in ``env_path`` without losing DB routes. - - The first time the key changes, the old one is pinned as - ``LITELLM_SALT_KEY`` -- the value LiteLLM has been encrypting stored - credentials with. After that the salt stays put and only the key moves. - """ - if not key.startswith('sk-'): - # master_key() would silently replace it on the next render. - raise ValueError(f'{API_KEY_ENV} must start with "sk-" (LiteLLM rejects others)') - existing = parse_env_file(env_path) - values = {API_KEY_ENV: key} - old = existing.get(API_KEY_ENV, '').strip() - if old and old != key and not existing.get(SALT_KEY_ENV, '').strip(): - values[SALT_KEY_ENV] = old - write_env_file(env_path, values) - - class ComposeBackend(ConvergeScaffold): """Single-host docker compose backend (converge-style). @@ -1907,15 +1071,19 @@ def __init__( # before the first frame. self._inventory = inventory self.run = run or _default_docker_run - if http is None: - import requests - http = requests - self.http = http + # The front door: gateway settings, keys, routes. Created before the + # settings below, which are stored on it. + self.gateway = Gateway( + self.state_dir, + ports={**DEFAULT_PORTS, **(ports or {})}, + litellm=litellm, ui=ui, reverse_proxy=reverse_proxy, + reverse_proxy_port=reverse_proxy_port, dynamic_routing=dynamic_routing, + http=http, sleep=sleep, clock=clock, + ) # Called with a one-line message during long steps (image pulls); the # TUI sets it. Always also logged. self.progress: Callable[[str], None] | None = None self.images = {**PINNED_IMAGES, **(images or {})} - self.ports = {**DEFAULT_PORTS, **(ports or {})} # Merge over the defaults (not replace) so a caller-supplied partial # state dict — tests, embedders — still resolves every cache-mount key. self.state = {**default_state_paths(), **(state or {})} @@ -1923,10 +1091,6 @@ def __init__( self.reserved = tuple(reserved) self.project = project self.skip_display = skip_display - self.litellm = litellm - self.ui = ui - self.reverse_proxy = reverse_proxy - self.reverse_proxy_port = reverse_proxy_port self.reverse_proxy_config = reverse_proxy_config # Set by use_profile(): the published proxy config content. self._profile_proxy_text: str | None = None @@ -1942,9 +1106,6 @@ def __init__( # against a Postgres-backed model store, instead of a static config file. # Gives each deployment its own upstream (so same-model --dedicated # deployments land on distinct GPUs) with no gateway recreation/blip. - self.dynamic_routing = dynamic_routing - self._sleep = sleep - self._clock = clock self.last_errors: list[str] = [] self.last_unplaced: set[str] = set() # desired deployment ids placement skipped self.last_assignments: dict[str, list[int]] = {} # deployment id -> GPU ids @@ -1984,104 +1145,136 @@ def inventory(self, value: dict[str, Any] | None) -> None: def compose_file(self) -> Path: return self.state_dir / COMPOSE_FILENAME + #: The file a render writes, whatever the backend (``status`` shows it). + rendered_file = compose_file + + def compose_project(self): + """The Compose project on this host: this one.""" + return self + + def front_door(self): + """What holds the gateway's keys and route registry: this project.""" + return self + + def compose_argv(self) -> list[str]: + """``docker compose [--env-file ...] -p -f ``: the one base. + + The managed ``.env`` beside the compose file resolves the master key + and the other managed secrets; docker compose's own ``.env`` discovery + keys off the caller's working directory, so it is passed explicitly, + and only when present (a missing ``--env-file`` is a hard error). + """ + cmd = ['docker', 'compose'] + if self.gateway._env_path.exists(): + cmd += ['--env-file', str(self.gateway._env_path)] + return [*cmd, '-p', self.project, '-f', str(self.compose_file)] + + # -- the front door (leasing.gateway) ------------------------------------ + # Settings the gateway owns. Kept as attributes of the backend because + # profiles and callers set them here; they are stored in one place. + @property - def _state_file(self) -> Path: - return self.state_dir / STATE_FILENAME + def http(self) -> Any: + return self.gateway.http + + @http.setter + def http(self, value: Any) -> None: + self.gateway.http = value @property - def litellm_port(self) -> int: - return self.ports.get('litellm', DEFAULT_PORTS['litellm']) + def _clock(self) -> Callable[[], float]: + return self.gateway._clock + + @_clock.setter + def _clock(self, value: Callable[[], float]) -> None: + self.gateway._clock = value @property - def ui_port(self) -> int: - return self.ports.get('open_webui', DEFAULT_PORTS['open_webui']) + def _sleep(self) -> Callable[[float], None]: + return self.gateway._sleep + + @_sleep.setter + def _sleep(self, value: Callable[[float], None]) -> None: + self.gateway._sleep = value @property - def _env_path(self) -> Path: - return self.state_dir / '.env' + def litellm(self) -> bool: + return self.gateway.litellm + + @litellm.setter + def litellm(self, value: bool) -> None: + self.gateway.litellm = value @property - def _routes_file(self) -> Path: - return self.state_dir / LITELLM_ROUTES_FILENAME + def ui(self) -> bool: + return self.gateway.ui + + @ui.setter + def ui(self, value: bool) -> None: + self.gateway.ui = value @property - def _registry_file(self) -> Path: - return self.state_dir / LITELLM_REGISTRY_FILENAME + def dynamic_routing(self) -> bool: + return self.gateway.dynamic_routing - def master_key(self) -> str: - """The managed LiteLLM master key. + @dynamic_routing.setter + def dynamic_routing(self, value: bool) -> None: + self.gateway.dynamic_routing = value - infer-stack manages this secret in the state dir's ``.env``: reused if - already present (you may pin your own ``sk-`` key there), otherwise - generated and persisted. The caller doesn't need to invent or export it - — it is baked into the LiteLLM service, used by the readiness probe, and - shipped in the env-file descriptor (``infer-stack env KEY`` prints it). - """ - existing = parse_env_file(self._env_path) - key = ensure_secret(existing, API_KEY_ENV, prefix='sk-') - if key != existing.get(API_KEY_ENV): - write_env_file(self._env_path, {API_KEY_ENV: key}) - return key + @property + def reverse_proxy(self) -> bool: + return self.gateway.reverse_proxy - def rotate_master_key(self) -> dict[str, str | None]: - """Write a fresh master key to the ``.env``; return the values it replaced. + @reverse_proxy.setter + def reverse_proxy(self, value: bool) -> None: + self.gateway.reverse_proxy = value - Only the file: the gateway and Open WebUI pick it up when the next apply - recreates them (their fingerprints hash the key). The caller holds the - publication lock and passes the return value to - :meth:`restore_env` if that apply does not happen. - """ - before = parse_env_file(self._env_path) - set_master_key(self._env_path, ensure_secret({}, API_KEY_ENV, prefix='sk-')) - return {k: before.get(k) for k in (API_KEY_ENV, SALT_KEY_ENV)} + @property + def reverse_proxy_port(self) -> int: + return self.gateway.reverse_proxy_port - def restore_env(self, values: dict[str, str | None]) -> None: - """Put back what :meth:`rotate_master_key` replaced.""" - from ..env_utils import remove_env_keys + @reverse_proxy_port.setter + def reverse_proxy_port(self, value: int) -> None: + self.gateway.reverse_proxy_port = value + + @property + def ports(self) -> dict[str, int]: + return self.gateway.ports - write_env_file(self._env_path, {k: v for k, v in values.items() if v is not None}) - remove_env_keys(self._env_path, [k for k, v in values.items() if v is None]) + @ports.setter + def ports(self, value: dict[str, int]) -> None: + self.gateway.ports = value + + @property + def litellm_port(self) -> int: + return self.gateway.litellm_port + + @property + def ui_port(self) -> int: + return self.gateway.ui_port + + def master_key(self) -> str: + return self.gateway.master_key() + + def rotate_master_key(self) -> dict[str, str | None]: + return self.gateway.rotate_master_key() + + def restore_env(self, values: dict[str, str | None]) -> None: + self.gateway.restore_env(values) def gateway_accepts(self, key: str, *, wait: float = 0.0) -> bool | None: - """Does the gateway accept ``key``? ``None`` if it never answered. + return self.gateway.gateway_accepts(key, wait=wait) - Polls ``/v1/models`` for up to ``wait`` seconds while the gateway is - unreachable or still starting. A definite no is 401/403, or the 400 - LiteLLM actually answers a wrong key with (measured on the pinned image). - """ - deadline = self._clock() + wait - while True: - try: - resp = self.http.get( - f'{self._gateway_base()}/v1/models', - headers={'Authorization': f'Bearer {key}'}, timeout=10.0, - ) - status = getattr(resp, 'status_code', 0) - except Exception: # noqa: BLE001 - not up yet - status = 0 - if status == 200: - return True - if status in (400, 401, 403): - return False - if self._clock() >= deadline: - return None - self._sleep(2.0) - - def db_password(self) -> str: - """The managed Postgres password for LiteLLM's model store. - - Same managed-secret pattern as :meth:`master_key`: reused if already - present in the state-dir ``.env`` (you may pin your own), else generated - and persisted. ``docker compose --env-file`` interpolates it into the - postgres + litellm services, so it never appears literally in the YAML. - Only used when ``dynamic_routing`` is on. ``token_urlsafe`` output is safe - inside the ``postgresql://`` URL (no ``@ : /`` characters). - """ - existing = parse_env_file(self._env_path) - pw = ensure_secret(existing, DB_PASSWORD_ENV) - if pw != existing.get(DB_PASSWORD_ENV): - write_env_file(self._env_path, {DB_PASSWORD_ENV: pw}) - return pw + def merge_route_registry(self, incoming: dict[str, dict[str, Any]]) -> dict[str, Any]: + return self.gateway.merge_route_registry(incoming) + + def access(self, endpoints: list[str]) -> dict[str, Any] | None: + """Where a client reaches these endpoints: the front door.""" + return self.gateway.access(endpoints) + + @property + def _state_file(self) -> Path: + return self.state_dir / STATE_FILENAME def _ensure_state_dir(self) -> None: """Raise a legible error if the state dir could not be created. @@ -2164,7 +1357,9 @@ def doctor(self) -> list[tuple[str, bool, str]]: # 3. Every image this configuration would actually bring up. Checking # the full PINNED_IMAGES set would fail on services that are off. - wanted: dict[str, str] = {'vllm': self.images['vllm']} + # A gateway-only project (kubeai's) runs no engine here. + wanted: dict[str, str] = ({} if self.fronts_elsewhere + else {'vllm': self.images['vllm']}) if self.litellm: wanted['litellm'] = self.images['litellm'] if self.dynamic_routing: @@ -2203,6 +1398,9 @@ def doctor(self) -> list[tuple[str, bool, str]]: # 4. GPUs. Last because a CPU-only stack is legitimate (mock endpoints, # the null backend), so this is informational rather than fatal. + # Not for a gateway-only project: its engines are elsewhere. + if self.fronts_elsewhere: + return checks try: gpus = self.inventory.get('gpus') or [] if gpus: @@ -2220,17 +1418,7 @@ def doctor(self) -> list[tuple[str, bool, str]]: def _compose(self, args: list[str]) -> str: self._ensure_state_dir() - cmd = ['docker', 'compose'] - # Resolve ${LITELLM_MASTER_KEY} (and any other managed secret) from the - # sidecar .env beside the compose file, so secrets stay out of the YAML. - # docker compose's default .env discovery keys off the *current working - # directory* (wherever infer-stack was invoked), not the state dir, so we - # point it explicitly. Only when present: a litellm-less stack never - # writes one, and a missing --env-file path is a hard error. - if self._env_path.exists(): - cmd += ['--env-file', str(self._env_path)] - cmd += ['-p', self.project, '-f', str(self.compose_file)] - return self.run([*cmd, *args]) + return self.run([*self.compose_argv(), *args]) def plan(self, desired: list[Deployment], placement=None): """Compute GPU placement for ``desired`` without writing or applying. @@ -2272,10 +1460,7 @@ def preview(self, desired: list[Deployment], placement=None, *, approve: bool = then does not ask again as long as it produces the same files. """ docs = self._render_documents(list(desired), placement) - self.last_preview_digest = self._planned_digest(docs['planned']) - if approve: - self._approve_changes(docs['planned']) - self._preapproved = self.last_preview_digest + self._preview_approval(docs['planned'], approve=approve) return docs['plan'], docs['rendered'] def _render_documents(self, desired: list[Deployment], placement) -> dict[str, Any]: @@ -2284,7 +1469,18 @@ def _render_documents(self, desired: list[Deployment], placement) -> dict[str, A if self.litellm and self.dynamic_routing: # The DB secret must exist before rendering, so docker compose # --env-file can interpolate ${LITELLM_DB_PASSWORD} at apply time. - self.db_password() + self.gateway.db_password() + ui_run_as = None + if self.ui: + from .gateway import open_webui_run_as + + self.gateway.webui_secret() # interpolated at apply, likewise + ui_run_as, why = open_webui_run_as(self.state['open_webui']) + if ui_run_as is None and not getattr(self, '_ui_root_noted', False): + from .._log import logger + + self._ui_root_noted = True + logger.info('Open WebUI runs as root: {}', why) route_registry = None if self.litellm and not self.dynamic_routing: # Unconditional in static-superset mode: `self.catalog` may be None; @@ -2296,12 +1492,13 @@ def _render_documents(self, desired: list[Deployment], placement) -> dict[str, A desired, plan.assignments, images=self.images, ports=self.ports, state=self.state, litellm=self.litellm, litellm_port=self.litellm_port, litellm_master_key=self.master_key() if self.litellm else None, - litellm_salt_key=SALT_KEY_ENV in parse_env_file(self._env_path), + litellm_salt_key=SALT_KEY_ENV in parse_env_file(self.gateway._env_path), ui=self.ui, ui_port=self.ui_port, reverse_proxy=self.reverse_proxy, reverse_proxy_port=self.reverse_proxy_port, reverse_proxy_config=self.reverse_proxy_config, aux_dir=self.state_dir, project=self.project, catalog=self.catalog, route_registry=route_registry, dynamic_routing=self.dynamic_routing, + upstream_routes=self.upstream_routes, ui_run_as=ui_run_as, ) addresses = None if self.network is not None: @@ -2322,9 +1519,9 @@ def _render_documents(self, desired: list[Deployment], placement) -> dict[str, A if rendered.nginx_config is not None: planned[self.state_dir / NGINX_CONFIG_FILENAME] = rendered.nginx_config if rendered.litellm_routes is not None: - planned[self._routes_file] = json.dumps(rendered.litellm_routes, indent=2) + planned[self.gateway._routes_file] = json.dumps(rendered.litellm_routes, indent=2) fingerprints = stamp_fingerprints( - rendered.compose, files=planned, env_file=self._env_path, + rendered.compose, files=planned, env_file=self.gateway._env_path, ) planned[self.compose_file] = yaml.safe_dump(rendered.compose, sort_keys=False) return {'plan': plan, 'rendered': rendered, 'planned': planned, @@ -2338,18 +1535,6 @@ def _render_documents(self, desired: list[Deployment], placement) -> dict[str, A #: newly allocated addresses (append-only). on_addresses: Any = None - @staticmethod - def _planned_digest(planned: dict) -> str: - material = json.dumps({str(k): v for k, v in planned.items()}, sort_keys=True) - return hashlib.sha256(material.encode('utf-8')).hexdigest() - - #: Digest of files an admission preview already had approved. - _preapproved: str | None = None - #: Digest of the files the last render produced (approved-digest guard). - last_planned_digest: str | None = None - #: Digest of the files the last preview produced. - last_preview_digest: str | None = None - def pull_images(self, images) -> list[str]: """Pull ``images``; ``config publish`` passes :func:`profile_images`.""" from .._log import logger @@ -2359,13 +1544,6 @@ def pull_images(self, images) -> list[str]: self.run(['docker', 'pull', image]) return sorted(set(images)) - def _approve_changes(self, planned: dict) -> None: - if self._preapproved is not None and self._planned_digest(planned) == self._preapproved: - self._preapproved = None - return - self._preapproved = None - super()._approve_changes(planned) - def plan_on_idle_host(self, desired: list[Deployment]): """Placement for ``desired`` alone, as if nothing else were running. @@ -2481,39 +1659,10 @@ def startup_failure(self, deployment: Deployment) -> str | None: residency = self.residency() except ResidencyUnknown: return None # cannot read Docker: say nothing - containers = residency.containers(deployment.id) - if len(containers) != 1: - return None # absent (not created yet) or ambiguous - container = containers[0] - exited_for_good = ( - container.state in {'exited', 'dead'} - and (container.exit_code or 0) != 0 + return diagnose_startup( + residency.containers(deployment.id), + lambda: self.deployment_logs(deployment, tail=200), ) - crashed = exited_for_good or container.restart_count >= 1 - if not crashed: - return None # created, starting, or healthy - # The log decides, not the restart count alone. `restart: - # unless-stopped` is there so a hub timeout or a port still held by the - # container we replaced resolves itself; condemning those would make the - # policy pointless. An unrecoverable error, by contrast, repeats - # identically, so waiting for a second restart only wastes the GPU. - logs = self.deployment_logs(deployment, tail=200) - verdict = classify_engine_log(logs) - if verdict == 'transient' and container.will_be_restarted: - return None # a retry is coming, and may work - if verdict == 'transient': - # Transient, but nothing will run it again: waiting is as pointless - # as for an unrecoverable error, and the cause still belongs in the - # message. - return (f'engine is not starting (exited with code ' - f'{container.exit_code} and will not be restarted)' - f'{_engine_error_summary(logs)}') - looping = container.restart_count >= CRASH_LOOP_RESTARTS - if verdict != 'fatal' and not (exited_for_good or looping): - return None # unrecognised: keep today's budget - why = (f'restarted {container.restart_count} time(s)' if container.restart_count - else f'exited with code {container.exit_code}') - return f'engine is not starting ({why}){_engine_error_summary(logs)}' def deployment_logs(self, deployment: Deployment, *, tail: int = 400) -> str: """Recent engine logs for a deployment's compose service. @@ -2537,130 +1686,48 @@ def deployment_logs(self, deployment: Deployment, *, tail: int = 400) -> str: except Exception: return '' - def _load_route_registry(self) -> dict[str, Any]: - """Read the route registry, tolerantly (fail-open — a broken registry - must never block a converge). - - Missing file → seed from the live ``litellm_config.yaml`` if present - (upgrade migration, §6), else an empty registry. An *unknown* schema - version whose ``entries`` still parses as a name→row map is preserved - as-is (render what's understood, warn, do NOT rewrite) rather than - reseeded, so a binary rollback doesn't discard the accumulated union. - Only a structurally unusable file (not a map / garbage JSON) falls back - to seeding.""" - from .._log import logger - - if not self._registry_file.exists(): - config = self.state_dir / LITELLM_CONFIG_FILENAME - if config.exists(): - seeded, warnings = _seed_registry_from_litellm_config( - config.read_text() - ) - for w in warnings: - logger.warning(' route registry: {}', w) - logger.info( - ' route registry: seeded {} vLLM route(s) from {}', - len(seeded['entries']), config.name, - ) - return seeded - return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}} - try: - data = json.loads(self._registry_file.read_text()) - except (OSError, json.JSONDecodeError) as exc: - logger.warning( - ' route registry: {} is unreadable ({}); rebuilding from seed', - self._registry_file.name, exc, - ) - data = None - if not isinstance(data, dict) or not isinstance( - data.get('entries'), dict - ): - config = self.state_dir / LITELLM_CONFIG_FILENAME - if config.exists(): - seeded, warnings = _seed_registry_from_litellm_config( - config.read_text() - ) - for w in warnings: - logger.warning(' route registry: {}', w) - return seeded - return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}} - version = data.get('version') - if version != LITELLM_REGISTRY_VERSION: - logger.warning( - ' route registry: unknown schema version {!r} in {} — ' - 'rendering as-is without rewrite (fields this renderer does ' - 'not understand are ignored)', - version, self._registry_file.name, - ) - return data - def _merged_route_registry( self, desired: list[Deployment], assignments: dict[str, list[int]] ) -> dict[str, Any]: - """The route registry merged with the catalog and ``desired``, in memory.""" - from .._log import logger + """The route registry merged with this backend's rows, in memory. - existing = self._load_route_registry() - incoming: dict[str, dict[str, Any]] = {} - if self.catalog is not None: - incoming.update(_registry_incoming_from_catalog(self.catalog)) - # `desired` spans all runbooks via the shared ledger, so this keeps every - # live cross-runbook deployment routable (and, via persistence, routable - # past release). + The rows are this backend's to supply: every catalog endpoint, and every + placed deployment (``desired`` spans all runbooks via the shared ledger, + so a live cross-runbook deployment stays routable, and past release). + """ + incoming = self.catalog_route_rows(self.catalog) incoming.update(_registry_incoming_from_deployments(desired, assignments)) - merged, warnings = _merge_route_registry(existing, incoming) - for w in warnings: - logger.warning(' route registry: {}', w) - return merged - - def _save_route_registry(self, merged: dict[str, Any]) -> None: - """Persist a merged registry if it changed (under the converge flock).""" - from .._log import logger - - existing = self._load_route_registry() - if merged == existing: - return - prior = existing.get('entries', {}) if isinstance(existing, dict) else {} - added = sorted(set(merged['entries']) - set(prior)) - updated = sorted( - k for k in merged['entries'] if k in prior and merged['entries'][k] != prior[k] - ) - if added: - logger.info(' route registry: +{} route(s): {}', len(added), ', '.join(added)) - if updated: - logger.info(' route registry: updated route(s): {}', ', '.join(updated)) - self._atomic_write(self._registry_file, _dump_route_registry(merged)) + incoming.update(self.upstream_rows) + return self.gateway.merged_route_registry(incoming) + + #: Render inputs from the owner of engines this project does not run (the + #: kubeai backend, whose gateway this is): static route-registry rows and + #: dynamic routes, both pointing at its servers. Set before each render. + upstream_rows: dict[str, dict[str, Any]] = {} + upstream_routes: list[dict[str, Any]] = [] + #: Set by an owner whose engines run elsewhere (kubeai): this project is + #: only the front door. + fronts_elsewhere = False + + def catalog_route_rows(self, catalog) -> dict[str, dict[str, Any]]: + """Route-registry rows for every endpoint of ``catalog``.""" + return {} if catalog is None else _registry_incoming_from_catalog(catalog) + + def route_rows(self, desired: list[Deployment], placement=None) -> dict[str, dict[str, Any]]: + """The rows a render of ``desired`` merges: catalog, placed deployments, + and the owner's upstream rows (``routes prune`` keeps exactly these).""" + assignments = self.plan(desired, placement).assignments + rows = self.catalog_route_rows(self.catalog) + rows.update(_registry_incoming_from_deployments(desired, assignments)) + rows.update(self.upstream_rows) + return rows def _update_route_registry( self, desired: list[Deployment], assignments: dict[str, list[int]] ) -> dict[str, Any]: """Merge and persist the route registry; return it (kept for callers).""" merged = self._merged_route_registry(desired, assignments) - self._save_route_registry(merged) - return merged - - def merge_route_registry( - self, incoming: dict[str, dict[str, Any]] - ) -> dict[str, Any]: - """Public write path for out-of-converge registry seeds (``routes seed``). - - Takes the converge flock, read-merge-writes the registry, and returns the - merged dict. ``converge`` only ever merges the invoking process's own - catalog, so a standalone caller (seeding a *sibling* runbook's catalog) - needs this to fold extra rows in before the follow-up ``reconcile`` - renders+applies. The flock here and the one the subsequent converge takes - are sequential acquisitions, not nested — no reentrancy concern.""" - from .._log import logger - - with self._converge_lock(): - existing = self._load_route_registry() - merged, warnings = _merge_route_registry(existing, incoming) - for w in warnings: - logger.warning(' route registry: {}', w) - if merged != existing: - self._atomic_write( - self._registry_file, _dump_route_registry(merged) - ) + self.gateway._save_route_registry(merged) return merged def converge(self, desired: list[Deployment], *, apply: bool = True, placement=None): @@ -2678,11 +1745,16 @@ def converge(self, desired: list[Deployment], *, apply: bool = True, placement=N desired = list(desired) with self._converge_lock(): - logger.info( - 'Converging {} deployment(s): {}', - len(desired), - ', '.join(sorted(g.id for g in desired)) or '(none)', - ) + if self.fronts_elsewhere: + # A gateway-only project (kubeai's): its owner narrates the + # deployments, and "0 deployment(s)" read as a contradiction. + logger.info('Converging the gateway project ({})', self.project) + else: + logger.info( + 'Converging {} deployment(s): {}', + len(desired), + ', '.join(sorted(g.id for g in desired)) or '(none)', + ) docs = self._render_documents(desired, placement) plan, rendered, planned = docs['plan'], docs['rendered'], docs['planned'] fingerprints = docs['fingerprints'] @@ -2690,7 +1762,8 @@ def converge(self, desired: list[Deployment], *, apply: bool = True, placement=N self.last_displaced = list(plan.displaced) self.last_degraded = list(plan.degraded) for gid, gpus in sorted(plan.assignments.items()): - logger.info(' placed {} on GPU(s) {}', gid, gpus or '(cpu)') + logger.info(' placed {} {}', gid, + f'on GPU(s) {gpus}' if gpus else 'without a GPU') for err in plan.errors: logger.warning(' placement: {}', err) for note in plan.warnings: @@ -2716,7 +1789,7 @@ def converge(self, desired: list[Deployment], *, apply: bool = True, placement=N if docs['addresses'] is not None and self.on_addresses is not None: self.on_addresses(docs['addresses']) if docs['route_registry'] is not None: - self._save_route_registry(docs['route_registry']) + self.gateway._save_route_registry(docs['route_registry']) for path, text in planned.items(): if path != self.compose_file: self._atomic_write(path, text) @@ -2738,11 +1811,9 @@ def converge(self, desired: list[Deployment], *, apply: bool = True, placement=N }) services = rendered.compose.get('services') if not apply: - logger.info( - 'rendered {} service(s) to {} (not applied; ' - '`infer-stack apply` to bring it up)', - len(services or {}), self.compose_file, - ) + # The caller applies next, or (--no-apply, render) says how. + logger.info('rendered {} service(s) to {}', + len(services or {}), self.compose_file) return plan # Apply OUTSIDE the converge (render) lock: the controller coalesces and # serializes applies via its own apply-lock, so re-taking the render lock @@ -2785,6 +1856,15 @@ def apply(self) -> bool: # A render from before fingerprints: re-render (any mutation) first. logger.warning('apply: the render predates fingerprints; re-render, then apply') return False + # A service that runs as a user needs its bind-mount sources made by + # us, as that user: Docker would make a missing one as root. + for svc in services.values(): + if not svc.get('user'): + continue + for volume in svc.get('volumes') or []: + source = str(volume).split(':', 1)[0] + if source.startswith('/') and not Path(source).exists(): + Path(source).mkdir(parents=True, exist_ok=True) dynamic = bool(self.litellm and self.dynamic_routing) outcome = self.selective_apply( services, fingerprints, @@ -2793,7 +1873,7 @@ def apply(self) -> bool: networks=doc.get('networks') or {}, ) if dynamic and 'litellm' in services: - return self._reconcile_routes( + return self.gateway._reconcile_routes( deadline_s=ROUTE_RECONCILE_STEADY_S if 'litellm' in outcome.kept_services else ROUTE_RECONCILE_BOOTSTRAP_S, ) @@ -3097,208 +2177,6 @@ def _service_running(self, service: str) -> bool: # -- dynamic routing (admin API) -------------------------------------- - def _gateway_base(self) -> str: - return f'http://127.0.0.1:{self.litellm_port}' - - def _auth_headers(self) -> dict[str, str]: - return {'Authorization': f'Bearer {self.master_key()}'} - - def _desired_routes(self) -> list[dict[str, Any]]: - """The rendered desired route set (litellm_routes.json), or empty.""" - try: - data = json.loads(self._routes_file.read_text()) - except (FileNotFoundError, json.JSONDecodeError): - return [] - return data if isinstance(data, list) else [] - - def _reconcile_routes( - self, *, deadline_s: float = ROUTE_RECONCILE_BOOTSTRAP_S, delay: float = 2.0, - ) -> bool: - """Make the live gateway's managed routes match the rendered route set. - - The render half wrote the desired routes (one per live deployment× - endpoint) to ``litellm_routes.json``; this is the apply half. List the - gateway's current models, add the missing routes and delete the ones no - longer desired -- through the admin API, with **no** container restart -- - then list again to verify both ids and routing semantics, re-diffing and - retrying failed calls until the table matches or the budget runs out. - - Properties this relies on: - - * **Idempotent.** A redundant apply re-diffs to the same set and does - nothing. - * **Drift-healing.** Routes lost to a gateway/DB restart reappear in the - diff and are re-added; stale routes from a prior run (still in the DB) - are deleted because they're no longer desired. - * **Co-existence.** Only routes infer-stack created (id prefix ``isr-``) - are ever deleted, so a model added by hand through the UI/API is left - alone. - - **Bounded and reported.** Everything -- listing retries while the gateway - starts, every POST, and the final verification -- shares one wall-clock - budget, ``deadline_s``. A retry count alone would not bound it: a listing - can take 10 s and a POST 30 s. Returns ``True`` only when the verified - managed route set equals the desired set; any failure is logged and - returns ``False`` rather than raising, so the caller decides whether an - unverified route set blocks anything. - """ - from .._log import logger - - deadline = self._clock() + max(0.0, deadline_s) - desired = { - r['model_info']['id']: r - for r in self._desired_routes() - if isinstance(r.get('model_info'), dict) and r['model_info'].get('id') - } - desired_semantics = {rid: self._route_semantics(route) - for rid, route in desired.items()} - rounds = 0 - while True: - current = self._list_managed_routes(deadline=deadline, delay=delay) - if current is None: - logger.warning( - 'dynamic routing: route set not reconciled and verified within ' - '{:g}s; leaving it for the next apply', deadline_s, - ) - return False - mismatched = sorted( - rid for rid in desired.keys() & current.keys() - if desired_semantics[rid] != current[rid] - ) - to_add_ids = sorted((desired.keys() - current.keys()) | set(mismatched)) - to_delete = sorted((current.keys() - desired.keys()) | set(mismatched)) - to_add = [desired[rid] for rid in to_add_ids] - if not (to_add or to_delete): - return True # this listing is the verification - if rounds: - logger.info('dynamic routing: route set still differs; retrying') - rounds += 1 - ok = True - # A same-id semantic drift must be removed before it can be re-added; - # model/new is not an update API on every LiteLLM release. - for rid in to_delete: - # ok_if_missing: with a shared gateway, another converge may have - # deleted this route already; "not found in db" means the desired - # end-state (route gone) is reached, so don't treat it as an error. - ok &= self._post_route( - '/model/delete', {'id': rid}, rid, ok_if_missing=True, - deadline=deadline, - ) - for route in to_add: - ok &= self._post_route( - '/model/new', route, route.get('model_name'), deadline=deadline, - ) - logger.info( - 'dynamic routing: +{} route(s), -{} route(s), ~{} replacement(s) ' - '(now {} desired)', - len(to_add), len(to_delete), len(mismatched), len(desired), - ) - if not ok: - # A transient admin-API failure: spend the rest of the budget - # re-diffing rather than giving up with most of it unused. - if deadline - self._clock() <= delay: - return False - self._sleep(delay) - - @staticmethod - def _route_semantics(route: dict[str, Any]) -> dict[str, Any]: - """Observable route fields infer-stack owns and must verify. - - LiteLLM's model-info response contains additional database/runtime fields - and may redact credentials. The public alias, upstream model, and - upstream base URL are the routing semantics infer-stack can both set and - reliably observe. A matching managed id with different values here is - drift and is replaced, not accepted as healthy. - """ - params = route.get('litellm_params') or {} - return { - 'model_name': route.get('model_name'), - 'model': params.get('model'), - 'api_base': params.get('api_base'), - } - - def _list_managed_routes( - self, *, deadline: float, delay: float - ) -> dict[str, dict[str, Any]] | None: - """Observable semantics of infer-stack-managed gateway routes. - - Retries while the gateway is unreachable, until ``deadline`` (a value of - ``self._clock``). Each request's own timeout is capped by the time left. - Returns ``None`` if no listing succeeded in time. - """ - while True: - remaining = deadline - self._clock() - if remaining <= 0: - return None - resp = None - try: - resp = self.http.get( - f'{self._gateway_base()}/v1/model/info', - headers=self._auth_headers(), - timeout=min(10.0, remaining), - ) - except Exception: # noqa: BLE001 - the gateway may still be starting - resp = None - if resp is not None and getattr(resp, 'status_code', 0) == 200: - routes: dict[str, dict[str, Any]] = {} - for m in (resp.json().get('data') or []): - rid = (m.get('model_info') or {}).get('id') - if isinstance(rid, str) and rid.startswith(ROUTE_ID_PREFIX): - routes[rid] = self._route_semantics(m) - return routes - if deadline - self._clock() <= delay: - return None - self._sleep(delay) - - def _post_route( - self, - path: str, - payload: dict[str, Any], - label: Any, - *, - ok_if_missing: bool = False, - deadline: float | None = None, - ) -> bool: - """POST one admin-API call (``/model/new`` or ``/model/delete``). - - Returns whether it reached its desired end state. A failure is logged, - not raised, so the remaining calls still run. ``ok_if_missing`` accepts a - "model not found" response (a delete whose target is already gone). - The request timeout is capped by the time left before ``deadline``. - """ - from .._log import logger - - timeout = 30.0 - if deadline is not None: - remaining = deadline - self._clock() - if remaining <= 0: - logger.warning( - 'dynamic routing: POST {} {} skipped: route deadline passed', - path, label, - ) - return False - timeout = min(timeout, remaining) - try: - resp = self.http.post( - f'{self._gateway_base()}{path}', - headers=self._auth_headers(), - json=payload, - timeout=timeout, - ) - except Exception as ex: # noqa: BLE001 - one bad call must not abort apply - logger.warning('dynamic routing: POST {} {} error: {}', path, label, ex) - return False - if getattr(resp, 'status_code', 0) >= 300: - body = str(getattr(resp, 'text', '')) - if ok_if_missing and 'not found' in body.lower(): - return True - logger.warning( - 'dynamic routing: POST {} {} -> {} {}', - path, label, resp.status_code, body[:200], - ) - return False - return True - # -- published profile (see leasing/profile.py) --------------------------- REVERSE_PROXY_SNAPSHOT = 'reverse-proxy.conf' @@ -3380,28 +2258,6 @@ def use_profile(self, profile: dict[str, Any]) -> None: sources = profile.get('catalogs') or [] self.catalog = CatalogUnion.from_sources(sources) if sources else None - def placement_context(self) -> dict[str, Any] | None: - """This caller's admission scope, stored with a pending acquire.""" - if self.allowed_gpus is None: - return None - return {'allowed_gpus': list(self.allowed_gpus)} - - def placement_scope(self, context: dict[str, Any] | None): - """Temporarily render with another caller's admission scope.""" - import contextlib - - @contextlib.contextmanager - def scope(): - saved = self.allowed_gpus - if context and 'allowed_gpus' in context: - self.allowed_gpus = context['allowed_gpus'] - try: - yield - finally: - self.allowed_gpus = saved - - return scope() - def validate_requests(self, requests) -> None: """Refuse requests the published catalog union does not define identically.""" from .profile import validate_requests_against @@ -3463,6 +2319,12 @@ def settle_snapshot(self) -> tuple[tuple[str, str], ...]: pairs.append((parts[0], parts[1].lower())) return tuple(sorted(pairs)) + def instances(self): + """Every container of this project, engines first (raises if unknown).""" + from .instances import DOCKER, from_residency + + return from_residency(self.residency(), runtime=DOCKER) + def residency(self) -> Residency: """Strict snapshot of this project's deployment containers and their GPUs. @@ -3510,31 +2372,6 @@ def observe(self) -> set[str]: services = self._load_sidecar().get('services', {}) return {services[name] for name in running if name in services} - def access(self, endpoints: list[str]) -> dict[str, Any] | None: - """Where a client reaches these endpoints, for the env-file descriptor. - - With the LiteLLM front door, that is one ``base_url`` and the request - model name is the endpoint alias itself. With LiteLLM off there is no - single base URL, but a managed Open WebUI (if on) is still a useful - access point, so report just its URL rather than ``None``. - """ - if not self.litellm: - if self.ui: - return {'ui_url': f'http://127.0.0.1:{self.ui_port}'} - return None - info: dict[str, Any] = { - 'base_url': f'http://127.0.0.1:{self.litellm_port}/v1', - 'api_key_env': API_KEY_ENV, - 'api_key': self.master_key(), - 'request_names': {ep: ep for ep in endpoints}, - } - if self.ui: - info['ui_url'] = f'http://127.0.0.1:{self.ui_port}' - if self.reverse_proxy: - # The unified front door: one origin, UI at / and the API at /v1. - info['proxy_url'] = f'http://127.0.0.1:{self.reverse_proxy_port}' - return info - def _ensure_ollama_tag( self, deployment: Deployment, endpoint: str ) -> str | None: diff --git a/infer_stack/leasing/controller.py b/infer_stack/leasing/controller.py index 6636c9b6..1cb9429a 100644 --- a/infer_stack/leasing/controller.py +++ b/infer_stack/leasing/controller.py @@ -37,6 +37,9 @@ _T = TypeVar('_T') KEEP_WARM = 'keep-warm' +#: Between two evictions made to fit leased demand: long enough for the +#: freed instance to terminate and the scheduler to retry. +ROOM_COOLDOWN_S = 30.0 #: After an interrupted apply, how long to wait for the runtime to stop changing #: before applying again, and how often to sample it. Held under the lock. @@ -196,6 +199,10 @@ def __init__( # that differs from an earlier approved digest. self._explicit_apply = False self._admission_digest: str | None = None + #: Whether the last refused admission was for lack of GPUs. + self._admission_capacity = False + #: Why residency could not be read, for the refusal that follows. + self._residency_error: str | None = None from .profile import ProfileMismatch try: @@ -416,19 +423,6 @@ def _global_lock(self): # -- reconcile --------------------------------------------------------- - def desired_deployments(self) -> list[Deployment]: - """Deployments that should currently be running.""" - _, deployments = self.ledger.status() - desired: list[Deployment] = [] - for deployment in deployments: - if deployment.state == DeploymentState.LIVE: - desired.append(deployment) - elif deployment.state == DeploymentState.IDLE: - policy = deployment.spec.get('reclaim', self.reclaim_default) - if policy == KEEP_WARM: - desired.append(deployment) - return desired - def _infeasible_alone(self, deployments, requested: set) -> dict: """Which of this lease's deployments cannot be placed even on an idle host. @@ -466,76 +460,36 @@ def _infeasible_alone(self, deployments, requested: set) -> dict: def _render(self) -> ReconcileResult: """Render the desired state to disk WITHOUT the slow apply. - The caller MUST hold :meth:`_global_lock`. For a converge-style backend - (Compose) this writes the compose project (placement + files) but does - not ``docker compose up`` -- that is :meth:`_apply_pending`, under the - same lock hold. A per-deployment ``realize``/``teardown`` backend has no - such split, so it realizes here; those backends expose no ``apply``. + The caller MUST hold :meth:`_global_lock`. This writes the backend's + rendered state (the compose project, the Model manifests) but does not + bring it up -- that is :meth:`_apply_pending`, under the same lock hold. """ self.ledger.sweep() - placement = None - if self._admission_mode(): - # Residency decides which idle keep-warm deployments are candidates - # at all; unknown residency fails the render (the change stays - # pending) rather than guessing which warm models exist. - residency = self._admitting.residency() - self._backfill_allocations(residency) - self._prepare_network() - desired, placement = self._admission_view(residency) + # Residency decides which idle keep-warm deployments are candidates at + # all; unknown residency fails the render (the change stays pending) + # rather than guessing which warm models exist. + residency = self._admitting.residency() + self._backfill_allocations(residency) + self._prepare_network() + desired, placement = self._admission_view(residency) + if hasattr(self.backend, 'adopted'): self._admitting.adopted = self._prune_adopted(residency) - else: - desired = self.desired_deployments() - # Bound once rather than probed with hasattr: the capability check is - # the same, and the bound method keeps its type instead of narrowing - # to `object` the way an attribute reached through hasattr does. - converge = getattr(self.backend, 'converge', None) - if converge is not None: - before = set(self.backend.observe()) - try: - if placement is not None: - converge(desired, apply=False, placement=placement) - else: - converge(desired, apply=False) - except TypeError: - # Legacy converge(desired) with no apply kwarg renders+applies - # in one shot (no separate apply()); accept that here. - converge(desired) - after = set(self.backend.observe()) - rec = ReconcileResult( - realized=sorted(after - before), - torn_down=sorted(before - after), - unplaced=sorted( - getattr(self.backend, 'last_unplaced', ()) or () - ), - placement_errors=list( - getattr(self.backend, 'last_errors', ()) or () - ), - assignments=dict( - getattr(self.backend, 'last_assignments', {}) or {} - ), - applied=False, - displaced=list(getattr(self.backend, 'last_displaced', ()) or ()), - degraded=list(getattr(self.backend, 'last_degraded', ()) or ()), - ) - if placement is not None: - self._adopt_existing(residency) - return rec - desired_ids = {g.id for g in desired} - actual = self.backend.observe() - result = ReconcileResult() - for deployment in desired: - if deployment.id not in actual: - self.backend.realize(deployment) - result.realized.append(deployment.id) - stale = actual - desired_ids - if stale: - by_id = {g.id: g for g in self.ledger.status()[1]} - for gid in stale: - deployment = by_id.get(gid) - if deployment is not None: - self.backend.teardown(deployment) - result.torn_down.append(gid) - return result + before = set(self.backend.observe()) + self._admitting.converge(desired, apply=False, placement=placement) + after = set(self.backend.observe()) + rec = ReconcileResult( + realized=sorted(after - before), + torn_down=sorted(before - after), + unplaced=sorted(getattr(self.backend, 'last_unplaced', ()) or ()), + placement_errors=list(getattr(self.backend, 'last_errors', ()) or ()), + assignments=dict(getattr(self.backend, 'last_assignments', {}) or {}), + applied=False, + displaced=list(getattr(self.backend, 'last_displaced', ()) or ()), + degraded=list(getattr(self.backend, 'last_degraded', ()) or ()), + ) + if hasattr(self.backend, 'adopted'): + self._adopt_existing(residency) + return rec # -- serialised publication -------------------------------------------- # @@ -551,16 +505,18 @@ def _render(self) -> ReconcileResult: # -- admission (plan steps P5, P6, P9) ------------------------------------ # - # Backends with strict residency and an in-memory preview (Compose) get - # admission semantics: + # Every acquire, renew and publication goes through admission: # * a LIVE deployment holds a committed allocation (assigned_gpus); # * an IDLE keep-warm deployment is only an optional candidate, and only # while it is uniquely resident; it yields its GPUs to demand and is # never started; # * an acquire is previewed in memory (placement AND render) and commits # its lease together with its allocations, or commits nothing. - # Other backends (KubeAI, where the cluster schedules; test fakes) keep the - # previous behaviour. + # Where the cluster schedules (KubeAI, ``allocates_gpus`` false) every + # deployment commits an empty allocation: admission then decides only + # renderability, and a Pending pod is a wait reason, not an unplaced + # error. Backends that neither place nor inspect (dry-run, tests) get the + # same surface from SimpleAdmission. def _stored_state(self, lease_id: str): """A lease's state as stored (not virtually expired), or ``None``.""" @@ -571,17 +527,11 @@ def _stored_state(self, lease_id: str): def _admitting(self) -> AdmissionBackend: """``self.backend`` as the admission surface. - Only for code that runs after :meth:`_admission_mode` (or an - equivalent capability check) said the backend has it. + Every backend the controller drives has it (the real ones, and + :class:`~infer_stack.leasing.backend.SimpleAdmission` for the rest). """ return cast(AdmissionBackend, self.backend) - def _admission_mode(self) -> bool: - return all( - callable(getattr(self.backend, name, None)) - for name in ('residency', 'preview', 'converge') - ) - def _admission_view( self, residency, *, overlay=None, virtual_expiry: bool = False ): @@ -625,7 +575,13 @@ def _admission_view( return [*required.values(), *optional], inputs def _prepare_network(self) -> None: - """Give the backend the stable-address table, once a network is migrated.""" + """Give the backend the stable-address table, once a network is migrated. + + Compose only: a backend without a ``network`` attribute has no + host network to stamp addresses on. + """ + if not hasattr(self.backend, 'network'): + return config = self.ledger.network_config() if config is None: self._admitting.network = None @@ -740,11 +696,10 @@ def observe_state(self) -> dict: except Exception: # noqa: BLE001 - a view must always render sidecar = {} residency, residency_error = None, None - if callable(getattr(self.backend, 'residency', None)): - try: - residency = self._admitting.residency() - except ResidencyUnknown as ex: - residency_error = str(ex) + try: + residency = self._admitting.residency() + except ResidencyUnknown as ex: + residency_error = str(ex) degraded = set(sidecar.get('degraded') or ()) displaced = set(sidecar.get('displaced') or ()) rows = [] @@ -760,7 +715,7 @@ def observe_state(self) -> dict: elif g.id in displaced and g.state == DeploymentState.IDLE: condition = 'displaced' elif (g.state == DeploymentState.LIVE and g.assigned_gpus is None - and self._admission_mode() and residency is not None + and residency is not None and residency.resident(g.id) is None): condition = 'unresolved' elif residency is not None: @@ -805,7 +760,7 @@ def _prune_adopted(self, residency) -> dict: def _adopt_existing(self, residency) -> None: """One-time migration: adopt project containers from before ownership labels. - Runs after the first admission-mode render on a ledger. A container + Runs after the first render on a ledger. A container without infer-stack's service and fingerprint labels is adopted, with the fingerprint of this render, if it is infrastructure the render still has, or a deployment container whose deployment is LIVE on the @@ -856,6 +811,13 @@ def remove_orphans(self, confirm: Callable[[list], bool]) -> list: self._admitting.run(['docker', 'rm', '-f', *[c.container_id for c in orphans]]) return orphans + def _gpu_units(self, deployment) -> int: + """GPUs admission must account for: none where the cluster schedules.""" + from .backend import allocates_gpus + from .placement import required_gpu_count + + return required_gpu_count(deployment) if allocates_gpus(self.backend) else 0 + def _backfill_allocations(self, residency) -> list[str]: """Adopt allocations for LIVE deployments that predate them. @@ -864,14 +826,12 @@ def _backfill_allocations(self, residency) -> list[str]: GPUs stays unresolved: it keeps running where it is, but no new GPU is allocated to anyone until it is released. Returns the unresolved ids. """ - from .placement import required_gpu_count - unresolved = [] _, deployments = self.ledger.status() for deployment in deployments: if deployment.state != DeploymentState.LIVE or deployment.assigned_gpus is not None: continue - if required_gpu_count(deployment) == 0: + if self._gpu_units(deployment) == 0: self.ledger.set_allocation(deployment.id, []) continue resident = residency.resident(deployment.id) @@ -888,15 +848,14 @@ def _admit(self, overlay, residency): commit for the candidate's new and revived deployments. Nothing is written here. """ - from .placement import required_gpu_count - + self._admission_capacity = False adopted: dict[str, list[int]] = {} need: list[str] = [] for gid, deployment in overlay.deployments.items(): fresh = gid in overlay.created or gid in overlay.revived if not fresh: continue - if required_gpu_count(deployment) == 0: + if self._gpu_units(deployment) == 0: continue resident = residency.resident(gid) if ( residency is not None and gid in overlay.revived) else None @@ -904,10 +863,13 @@ def _admit(self, overlay, residency): adopted[gid] = list(resident.gpus) # IDLE->LIVE keeps its GPUs else: need.append(gid) - if residency is None and (need or adopted): + if residency is None: + # The render after the commit needs residency too, so nothing can + # be admitted without it; refusing here commits nothing. + why = self._residency_error or 'the runtime did not answer' return {}, [ - 'Docker residency is unknown, so only requests that need no new ' - 'GPU are admitted; retry when `docker ps` works' + f'what is running cannot be read right now ({why}), so nothing is ' + 'admitted; `infer-stack doctor` checks the runtime' ] if need: unresolved = self._unresolved_allocations(exclude=set(overlay.deployments)) @@ -922,7 +884,10 @@ def _admit(self, overlay, residency): self._prepare_network() desired, inputs = self._admission_view(residency, overlay=overlay) plan, rendered = self._admitting.preview(desired, inputs) + from .backend import allocates_gpus + reasons = [] + capacity = False # did any refusal come from GPU placement? unresolved = set(self._unresolved_allocations()) # EVERY deployment the candidate claims must be placed and renderable, # including an existing one it only coalesces onto (whose served @@ -936,9 +901,12 @@ def _admit(self, overlay, residency): continue if gid in plan.degraded: reasons.append(f'{gid}: its GPUs are no longer available') + capacity = True elif gid not in plan.assignments: why = [e for e in plan.errors if e.startswith(gid)] reasons.extend(why or [f'{gid}: could not be placed']) + # Where the cluster places, a plan without it is a refusal. + capacity = capacity or allocates_gpus(self.backend) elif gid in set(rendered.unrenderable): why = [e for e in rendered.errors if gid in e] or [ f'{gid}: could not be rendered ({e})' for e in rendered.errors] @@ -950,6 +918,7 @@ def _admit(self, overlay, residency): if holders: reasons.append('GPUs held by admitted demand: ' + '; '.join(holders)) self._admission_digest = None + self._admission_capacity = capacity if not reasons: # Approval happens now, before anything is committed; the render # after the commit produces the same files and does not ask again. @@ -977,13 +946,11 @@ def _gpu_holders(self, plan) -> list[str]: return out def _unresolved_allocations(self, *, exclude: AbstractSet[str] = frozenset()) -> list[str]: - from .placement import required_gpu_count - _, deployments = self.ledger.status() return [ g.id for g in deployments if g.state == DeploymentState.LIVE and g.assigned_gpus is None - and g.id not in exclude and required_gpu_count(g) > 0 + and g.id not in exclude and self._gpu_units(g) > 0 ] def set_invocation_catalog(self, catalog) -> None: @@ -1086,7 +1053,7 @@ def _acquire_profile_candidate(self, *, residency=None) -> dict | None: candidate['catalogs'] = merge_catalog_sources( stored.get('catalogs') or [], invocation.get('catalogs') or [] ) - except CatalogConflict as ex: + except CatalogConflict: # Only definitions a resident workload is actually running have to # stay frozen. Redefining anything else -- the normal case while # iterating with `catalog endpoint add --force` -- drops the stale @@ -1191,8 +1158,7 @@ def _sync_profile(self, *, create: bool) -> None: self._applied_profile = stored def _mark_pending( - self, *, apply: bool, placement_context: dict | None = None, - create_profile: bool = True, + self, *, apply: bool, create_profile: bool = True, ) -> dict: """Record that desired state is about to change (caller holds the lock). @@ -1201,31 +1167,16 @@ def _mark_pending( ever turns ``apply_requested`` on: a staged change never cancels an apply already requested (promotion, see the plan's D23). - If an earlier acquire died between committing and its first render, its - placement scope is still in the marker: render once with that scope - first, so its deployment is placed where that caller was allowed. + A placement scope left in the marker by pre-admission code is dropped: + a row that code committed without an allocation stays unresolved and is + never placed, so no render needs that caller's scope. """ if create_profile: self._sync_profile(create=True) current = self.ledger.publication_pending() if current and current.get('placement_context'): - self._render_in_scope(current['placement_context']) - return self.ledger.mark_publication_pending( - apply_requested=apply, placement_context=placement_context, - ) - - def _render_in_scope(self, context: dict) -> None: - """Render with another caller's admission scope, then forget the scope. - - The scope is cleared only after a render that succeeded (and so pinned - the placement). A declined or failed render propagates and keeps it, - so no later caller can place that deployment within its own scope. - """ - scope = getattr(self.backend, 'placement_scope', None) - if scope is not None: - with scope(context): - rec = self._render() - self.ledger.clear_placement_context() + self.ledger.clear_placement_context() + return self.ledger.mark_publication_pending(apply_requested=apply) def _apply_pending(self, rec: ReconcileResult) -> ReconcileResult: """Apply the last render if the marker requests it; clear on success. @@ -1240,12 +1191,7 @@ def _apply_pending(self, rec: ReconcileResult) -> ReconcileResult: marker = self.ledger.publication_pending() if marker is None: return rec - apply_fn = getattr(self.backend, 'apply', None) - if apply_fn is None: - # realize/teardown backends applied during the render itself. - self.ledger.clear_publication_pending(marker['version']) - rec.publication_pending = False - return rec + apply_fn = self._admitting.apply if not marker['apply_requested']: rec.publication_pending = True # staged; never applied here return rec @@ -1393,14 +1339,17 @@ def wait_ready( if endpoints is None or ep in endpoints ] deadline = self.clock() + timeout + last_room = float('-inf') while True: pending = [] failures = [] + needs_room = False for (g, ep) in pairs: probe = self.backend.probe_ready(g, ep) if probe.ready: continue pending.append((g, ep)) + needs_room = needs_room or probe.needs_room if probe.fatal: failures.append((g.id, ep, probe.detail)) if not pending: @@ -1416,25 +1365,48 @@ def wait_ready( ready=False, pending=[(g.id, ep) for g, ep in pending], ) + # After the deadline check: a wait that has given up evicts nothing. + if needs_room and self.clock() - last_room >= ROOM_COOLDOWN_S: + # A leased model is waiting on resources: a model without a + # lease is always a candidate to give them up. + if self._make_room(): + last_room = self.clock() self.sleep(interval) pairs = pending + def _make_room(self) -> str | None: + """Evict the longest-idle deployment for leased demand; its id, or ``None``. + + Idle means no lease holds it (a keep-warm model left resident). One at + a time: the runtime cannot say how much room is needed, and every + warm model kept is a load avoided. Compose backends never ask, since + their admission already moves idle models aside; this serves backends + where the runtime schedules (KubeAI). + """ + from .._log import logger + + _, deployments = self.ledger.status(virtual_expiry=True) + idle = sorted((g for g in deployments if g.state == DeploymentState.IDLE), + key=lambda g: (g.updated_at, g.id)) + if not idle: + return None + victim = idle[0].id + logger.info('making room for leased demand: evicting idle keep-warm {} ({})', + victim, ', '.join(sorted(idle[0].served))) + self.evict([victim]) + return victim + def _never_ran(self, deployment_ids: list[str]) -> list[str]: """Which of these deployments definitely have no container at all. - Uses strict residency where the backend has it: a deployment with any - container, in any state, is kept, and if Docker cannot be read nothing - is reported (so nothing warm is ever evicted on a failed look). Backends - without residency fall back to ``observe()``. + Uses strict residency: a deployment with any container, in any state, + is kept, and if the runtime cannot be read nothing is reported (so + nothing warm is ever evicted on a failed look). """ - residency = getattr(self.backend, 'residency', None) - if residency is None: - running = set(self.backend.observe()) - return [gid for gid in deployment_ids if gid not in running] from .residency import ResidencyUnknown try: - snap = residency() + snap = self._admitting.residency() except ResidencyUnknown: return [] return [gid for gid in deployment_ids if not snap.containers(gid)] @@ -1548,106 +1520,12 @@ def acquire( grabs. Head-of-line GPU reservation is a follow-up; for the small-fleet case (few GPUs, rare multi-GPU jobs) plain queueing is sufficient. """ - from .backend import ConvergeAborted, PlacementError - - if self._admission_mode(): - result, rec = self._acquire_by_admission( - owner, requests, ttl_seconds=ttl_seconds, apply=apply, - wait_for_placement=wait_for_placement, - placement_timeout=timeout if placement_timeout is None else placement_timeout, - placement_interval=interval if placement_interval is None else placement_interval, - ) - return self._finish_acquire(result, rec, apply=apply, wait=wait, - timeout=timeout, interval=interval) - - # Intent, ledger write, render and (once placed) apply all under one lock - # hold, so a second caller blocks before touching sqlite and no render - # can change the files this apply reads. The readiness wait and the - # admission-queue sleep stay OUTSIDE the lock. - with self._global_lock(): - self._sync_profile(create=True) - stored_profile = self.ledger.profile() - candidate_profile = self._acquire_profile_candidate() - if candidate_profile is not None: - self._use_profile_candidate(candidate_profile) - try: - validate = getattr(self.backend, 'validate_requests', None) - if validate is not None: - validate(requests) # before desired state is written - except BaseException: - if candidate_profile is not None: - self._restore_stored_profile(stored_profile) - raise - if candidate_profile is not None: - # The user catalog is authoritative. Persist the compatible - # recovery inputs now; the desired-state marker belongs to the - # acquire immediately below, not to this snapshot refresh. - self._commit_profile_candidate(candidate_profile) - context = getattr(self.backend, 'placement_context', lambda: None)() - self._mark_pending(apply=apply, placement_context=context) - result = self.ledger.acquire( - owner, requests, ttl_seconds=ttl_seconds - ) - try: - try: - rec = self._render() - finally: - # Rendered (placement pinned) or about to roll back: either - # way a recovery no longer needs this caller's scope. - if context is not None: - self.ledger.clear_placement_context() - except ConvergeAborted: - # The operator declined the compose changes -- don't leave the - # just-created lease dangling in the ledger. - self._rollback_acquire(result.lease.id, apply=apply) - raise - # If a deployment this lease just requested could not be placed (e.g. no - # free GPU), either queue for one (wait_for_placement) or -- the default -- - # roll the lease back and report the planner's reason, so the deployment - # never lingers as a phantom ``live`` with nothing behind it. - requested = {g.id for g in result.deployments} - unplaced = requested & set(rec.unplaced) - if not unplaced: - rec = self._apply_admitted(rec, result.lease.id, apply=apply) - if unplaced and wait_for_placement and apply: - # Never queue for capacity that cannot exist. Re-plan this lease's - # deployments ALONE on an idle host: if they do not fit there, no - # amount of waiting will help, and waiting is actively harmful -- - # the lease holds whatever it did place for the whole timeout, so a - # request that was never satisfiable can block ones that are. - # - # Only the aggregate case needs this. A single deployment too large - # for any card is already caught by the planner's permanent branch; - # what is missed is a lease whose deployments cannot fit TOGETHER, - # e.g. a 4-GPU model plus a 1-GPU extractor on a 4-GPU host. - infeasible = self._infeasible_alone(result.deployments, requested) - if infeasible: - self._rollback_acquire(result.lease.id, apply=apply) - raise PlacementError(sorted(infeasible.keys()), - sorted(infeasible.values())) - p_timeout = timeout if placement_timeout is None else placement_timeout - p_interval = ( - interval if placement_interval is None else placement_interval - ) - deadline = self.clock() + p_timeout - while unplaced and self.clock() < deadline: - self.sleep(p_interval) - # Re-render under the lock: each retry sweeps (reclaiming a crashed - # job's TTL-expired lease) and re-plans against the freed GPUs. - with self._global_lock(): - self._mark_pending(apply=True) # the render sweeps - rec = self._render() - unplaced = requested & set(rec.unplaced) - if not unplaced: - rec = self._apply_admitted(rec, result.lease.id, apply=True) - if unplaced: - self._rollback_acquire(result.lease.id, apply=apply) - reasons = [ - e - for e in rec.placement_errors - if any(e.startswith(gid) for gid in unplaced) - ] - raise PlacementError(sorted(unplaced), reasons) + result, rec = self._acquire_by_admission( + owner, requests, ttl_seconds=ttl_seconds, apply=apply, + wait_for_placement=wait_for_placement, + placement_timeout=timeout if placement_timeout is None else placement_timeout, + placement_interval=interval if placement_interval is None else placement_interval, + ) return self._finish_acquire(result, rec, apply=apply, wait=wait, timeout=timeout, interval=interval) @@ -1655,7 +1533,7 @@ def _acquire_by_admission( self, owner, requests, *, ttl_seconds, apply, wait_for_placement, placement_timeout, placement_interval, ): - """Admission-mode acquire: preview in memory, commit only if admissible. + """The acquire: preview in memory, commit only if admissible. Each attempt runs under the lock: sweep (itself a published mutation), observe residency, overlay the request on the ledger, and preview @@ -1683,8 +1561,10 @@ def _acquire_by_admission( self._publish() try: residency = self._admitting.residency() - except ResidencyUnknown: + self._residency_error = None + except ResidencyUnknown as ex: residency = None + self._residency_error = str(ex) candidate_profile = self._acquire_profile_candidate( residency=residency ) @@ -1705,7 +1585,6 @@ def _acquire_by_admission( self._commit_profile_candidate(candidate_profile) # Allocations are committed with the lease, so no placement # scope needs recording for recovery. - context = None self._mark_pending(apply=apply) try: result = self.ledger.acquire( @@ -1718,11 +1597,7 @@ def _acquire_by_admission( except AdmissionConflict: continue # the ledger moved; preview again try: - try: - rec = self._render() - finally: - if context is not None: - self.ledger.clear_placement_context() + rec = self._render() except ConvergeAborted: self._rollback_acquire(result.lease.id, apply=apply) raise @@ -1750,15 +1625,16 @@ def _acquire_by_admission( if gid in overlay.created or gid in overlay.revived ) if not (wait_for_placement and apply): - raise PlacementError(blocked, reasons) + raise PlacementError(blocked, reasons, capacity=self._admission_capacity) if not checked_feasible: checked_feasible = True infeasible = self._infeasible_alone( list(overlay.deployments.values()), set(blocked)) if infeasible: - raise PlacementError(sorted(infeasible), sorted(infeasible.values())) + raise PlacementError(sorted(infeasible), sorted(infeasible.values()), + capacity=False) if self.clock() + placement_interval > deadline: - raise PlacementError(blocked, reasons) + raise PlacementError(blocked, reasons, capacity=self._admission_capacity) self.sleep(placement_interval) def _finish_acquire(self, result, rec, *, apply, wait, timeout, interval) -> AcquireOutcome: @@ -1901,68 +1777,43 @@ def publish_profile(self, profile: dict) -> ReconcileResult: f'config publish needs a quiescent stack: {len(active)} active ' f'lease(s) ({", ".join(active[:3])}); release them first' ) - residency = getattr(self.backend, 'residency', None) - if residency is not None: - try: - snap = residency() - except ResidencyUnknown as ex: - raise ProfileMismatch( - f'config publish cannot confirm the stack is quiescent: {ex}' - ) from ex - running = sorted({c.deployment_id for c in snap.all_containers() - if c.deployment_id}) - if running: - raise ProfileMismatch( - 'config publish needs a quiescent stack: deployment ' - f'container(s) exist for {", ".join(running[:3])}; ' - '`infer-stack evict --all` first' - ) - if self._admission_mode(): - # A PURE preview first: the real render persists append-only - # state (route registry, addresses), which must not happen for - # a candidate whose publication has not committed. - previous = self._applied_profile - use(profile) - try: - residency = self._admitting.residency() - self._prepare_network() - desired, inputs = self._admission_view( - residency, virtual_expiry=True - ) - self._admitting.preview(desired, inputs, approve=True) - except BaseException: - if previous is not None: - use(previous) - raise - self.ledger.store.publish_profile( - profile, approved_digest=self._admitting.last_preview_digest) - self._profile_error = None - self._applied_profile = profile - self._invocation_profile = profile - self._profile_drift_warned = False - return self._publish() - # Backends without a preview (KubeAI): render, then commit. - # No implicit profile here: on a fresh ledger a declined preview - # must leave neither a profile nor a marker behind. - existed = self.ledger.publication_pending() is not None - marker = self._mark_pending(apply=True, create_profile=False) + try: + snap = self._admitting.residency() + except ResidencyUnknown as ex: + raise ProfileMismatch( + f'config publish cannot confirm the stack is quiescent: {ex}' + ) from ex + running = sorted({c.deployment_id for c in snap.all_containers() + if c.deployment_id}) + if running: + raise ProfileMismatch( + 'config publish needs a quiescent stack: deployment ' + f'container(s) exist for {", ".join(running[:3])}; ' + '`infer-stack evict --all` first' + ) + # A PURE preview first: the real render persists append-only + # state (route registry, addresses), which must not happen for + # a candidate whose publication has not committed. previous = self._applied_profile use(profile) try: - rec = self._render() + residency = self._admitting.residency() + self._prepare_network() + desired, inputs = self._admission_view( + residency, virtual_expiry=True + ) + self._admitting.preview(desired, inputs, approve=True) except BaseException: if previous is not None: use(previous) - if not existed: - self.ledger.clear_publication_pending(marker['version']) raise self.ledger.store.publish_profile( - profile, approved_digest=getattr(self.backend, 'last_planned_digest', None)) + profile, approved_digest=self._admitting.last_preview_digest) self._profile_error = None self._applied_profile = profile self._invocation_profile = profile self._profile_drift_warned = False - return self._apply_pending(rec) + return self._publish() def prune(self) -> tuple[int, int]: """Forget released/expired leases and stopped deployments, under the lock. @@ -1982,7 +1833,7 @@ def renew(self, lease_id: str, *, ttl_seconds: float | None) -> RenewOutcome: nothing. **Slow path, under the lock:** a deployment went IDLE, so the renew is a - desired-state change. In admission mode it is re-admitted: the IDLE + desired-state change, so it is re-admitted: the IDLE deployment adopts its resident GPUs, or is placed fresh; if neither is possible the renew fails with :class:`PlacementError` and writes nothing. The lease is re-validated as ACTIVE under the lock first. @@ -1993,52 +1844,39 @@ def renew(self, lease_id: str, *, ttl_seconds: float | None) -> RenewOutcome: if fast is not False: return RenewOutcome(fast, []) with self._global_lock(): - if self._admission_mode(): - lease = self.ledger.get_lease(lease_id) - if lease is None or lease.state != LeaseState.ACTIVE: - return RenewOutcome(None, []) - from .residency import ResidencyUnknown - - try: - residency = self._admitting.residency() - except ResidencyUnknown: - residency = None - overlay = self.ledger.plan_acquire([]) - for gid in dict.fromkeys(lease.deployment_ids): - deployment = self.ledger.get_deployment(gid) - if deployment is not None and deployment.state == DeploymentState.IDLE: - deployment.state = DeploymentState.LIVE - overlay.deployments[gid] = deployment - overlay.revived.append(gid) - if not overlay.revived: - return RenewOutcome( - self.ledger.renew(lease_id, ttl_seconds=ttl_seconds), []) - allocations, reasons = self._admit(overlay, residency) - if reasons: - raise PlacementError(list(overlay.revived), reasons) - self._mark_pending(apply=True) - if self._admission_digest: - self.ledger.mark_publication_pending( - apply_requested=True, approved_digest=self._admission_digest) - renewed = self.ledger.renew( - lease_id, ttl_seconds=ttl_seconds, allocations=allocations) - rec = self._publish() - return RenewOutcome(renewed, list(overlay.revived), rec) lease = self.ledger.get_lease(lease_id) - reviving = [] - if lease is not None and lease.state == LeaseState.ACTIVE: - for gid in dict.fromkeys(lease.deployment_ids): - deployment = self.ledger.get_deployment(gid) - if deployment is not None and deployment.state == DeploymentState.IDLE: - reviving.append(gid) - if not reviving: + if lease is None or lease.state != LeaseState.ACTIVE: + return RenewOutcome(None, []) + from .residency import ResidencyUnknown + + try: + residency = self._admitting.residency() + self._residency_error = None + except ResidencyUnknown as ex: + residency = None + self._residency_error = str(ex) + overlay = self.ledger.plan_acquire([]) + for gid in dict.fromkeys(lease.deployment_ids): + deployment = self.ledger.get_deployment(gid) + if deployment is not None and deployment.state == DeploymentState.IDLE: + deployment.state = DeploymentState.LIVE + overlay.deployments[gid] = deployment + overlay.revived.append(gid) + if not overlay.revived: return RenewOutcome( - self.ledger.renew(lease_id, ttl_seconds=ttl_seconds), [], - ) + self.ledger.renew(lease_id, ttl_seconds=ttl_seconds), []) + allocations, reasons = self._admit(overlay, residency) + if reasons: + raise PlacementError(list(overlay.revived), reasons, + capacity=self._admission_capacity) self._mark_pending(apply=True) - renewed = self.ledger.renew(lease_id, ttl_seconds=ttl_seconds) + if self._admission_digest: + self.ledger.mark_publication_pending( + apply_requested=True, approved_digest=self._admission_digest) + renewed = self.ledger.renew( + lease_id, ttl_seconds=ttl_seconds, allocations=allocations) rec = self._publish() - return RenewOutcome(renewed, reviving, rec) + return RenewOutcome(renewed, list(overlay.revived), rec) def evict(self, deployment_ids: Iterable[str] | None = None) -> EvictOutcome: """Force-evict idle (released) deployments now, overriding keep-warm. diff --git a/infer_stack/leasing/diagnosis.py b/infer_stack/leasing/diagnosis.py new file mode 100644 index 00000000..7a7b8ac7 --- /dev/null +++ b/infer_stack/leasing/diagnosis.py @@ -0,0 +1,150 @@ +"""Why an engine is not starting: shared by every backend. + +A backend supplies the instances of one deployment (containers, or pods) as +:class:`~infer_stack.leasing.residency.Container` records, and a way to read +their recent logs. :func:`diagnose_startup` decides whether waiting could +still help, so the fail-fast wait, the "likely cause" text and the TUI's +error path behave the same on every backend. +""" + +from __future__ import annotations + +from typing import Callable, Sequence + +#: Restarts after which the runtime's own bookkeeping says an engine is +#: looping, not loading. Two is already conclusive: a model that loads does not exit. +CRASH_LOOP_RESTARTS = 2 + +#: Engine log lines worth quoting verbatim, and what they mean. Each is +#: UNRECOVERABLE: the same container, restarted, fails the same way, so one +#: crash is already conclusive and there is nothing to wait for. +_ENGINE_ERROR_HINTS = ( + ('trust_remote_code', 'the model needs trust_remote_code=True ' + '(set `runtime.trust_remote_code: true` on the endpoint)'), + ('does not recognize this architecture', 'this engine cannot read the ' + "model's architecture (it may need trust_remote_code=True)"), + ('are not supported for now', 'this vLLM build does not implement the ' + "model's architecture"), + ('is not supported', 'this vLLM build does not implement the ' + "model's architecture"), + ('No supported config format', 'the model repository has no config this ' + 'engine can read'), + ('ValidationError', 'the engine rejected its own configuration'), + ('error: unrecognized arguments', 'the engine rejected a command-line flag ' + '(check `runtime.extra_args` against this vLLM version)'), + ('401 Client Error', 'the model is gated: set HF_TOKEN with ' + '`infer-stack env HF_TOKEN=...`'), + ('403 Client Error', 'the model is gated: set HF_TOKEN with ' + '`infer-stack env HF_TOKEN=...`'), +) + +#: Failures a RESTART CAN FIX: the hub was unreachable, a download was cut off, +#: a port was still held by the container we just replaced. `restart: +#: unless-stopped` exists for exactly these, so a restart count alone must not +#: condemn an engine -- the whole point of the policy is that the next attempt +#: succeeds. Weigh a new entry by one question: would running the same container +#: again plausibly work? If yes it belongs here; if no it belongs above. +_TRANSIENT_ENGINE_SIGNATURES = ( + 'Max retries exceeded', + 'Connection reset by peer', + 'Connection refused', + 'Temporary failure in name resolution', + 'Failed to resolve', + 'Read timed out', + 'ReadTimeoutError', + 'ConnectionError', + 'IncompleteRead', + 'Consistency check failed', # a truncated HF download + '429 Client Error', # hub rate limit + '500 Server Error', + '502 Server Error', + '503 Server Error', + '504 Server Error', + 'Address already in use', +) + + +def classify_engine_log(logs: str) -> str | None: + """``'fatal'``, ``'transient'``, or ``None`` when the log says neither. + + Order matters: an unrecoverable signature wins over a transient one, because + a hub timeout earlier in the same log does not make a rejected config + loadable. CUDA OOM counts as fatal — the allocation is deterministic, so the + restart repeats it — and it is the one class with a documented remedy + (:mod:`infer_stack.leasing.vram`). + """ + from .vram import looks_like_cuda_oom + + if not logs: + return None + if any(needle in logs for needle, _ in _ENGINE_ERROR_HINTS): + return 'fatal' + if looks_like_cuda_oom(logs): + return 'fatal' + if any(needle in logs for needle in _TRANSIENT_ENGINE_SIGNATURES): + return 'transient' + return None + + +def _engine_error_summary(logs: str) -> str: + """The engine's own error, quoted, with a hint when we recognise it.""" + from .vram import looks_like_cuda_oom + + if not logs.strip(): + return '; no engine log available (`infer-stack logs` for more)' + hint = next((note for needle, note in _ENGINE_ERROR_HINTS if needle in logs), None) + if hint is None and looks_like_cuda_oom(logs): + hint = 'the GPU ran out of memory for this configuration' + lines = [line.strip() for line in logs.splitlines() if line.strip()] + quoted = ' | '.join(lines[-3:])[:400] + summary = f'; last log: {quoted}' + return f'{summary}; likely cause: {hint}' if hint else summary + + + + +def diagnose_startup(instances: Sequence, read_logs: Callable[[], str]) -> str | None: + """Diagnose an engine that cannot start, or ``None`` if it may still load. + + A restart policy makes an engine that exits immediately restart forever, + so a model that can never start looks exactly like one that is still + loading, and an acquire waits out its whole timeout. The runtime's + bookkeeping says an instance crashed; its LOG says whether another attempt + could ever work (:func:`classify_engine_log`): + + * an unrecoverable error -- a rejected config or flag, an architecture + this build does not implement, a gated repo, a CUDA OOM -- is fatal on + the FIRST crash, because the restart reproduces it exactly; + * a transient one -- an unreachable hub, a truncated download, a port + still held -- is never fatal while something will retry it; + * an unrecognised crash keeps the blunt budget of + :data:`CRASH_LOOP_RESTARTS` restarts. + + ``instances`` must be exactly one (absent or ambiguous: ``None``). The + returned string carries the engine's last words, because the cause is in + its log and nowhere else. + """ + if len(instances) != 1: + return None # absent (not created yet) or ambiguous + instance = instances[0] + exited_for_good = ( + instance.state in {'exited', 'dead'} + and (instance.exit_code or 0) != 0 + ) + crashed = exited_for_good or instance.restart_count >= 1 + if not crashed: + return None # created, starting, or healthy + logs = read_logs() + verdict = classify_engine_log(logs) + if verdict == 'transient' and instance.will_be_restarted: + return None # a retry is coming, and may work + if verdict == 'transient': + return (f'engine is not starting (exited with code ' + f'{instance.exit_code} and will not be restarted)' + f'{_engine_error_summary(logs)}') + looping = instance.restart_count >= CRASH_LOOP_RESTARTS + if verdict != 'fatal' and not (exited_for_good or looping): + return None # unrecognised: keep the restart budget + why = (f'restarted {instance.restart_count} time(s)' if instance.restart_count + else f'exited with code {instance.exit_code}') + return f'engine is not starting ({why}){_engine_error_summary(logs)}' diff --git a/infer_stack/leasing/gateway.py b/infer_stack/leasing/gateway.py new file mode 100644 index 00000000..a73dd7de --- /dev/null +++ b/infer_stack/leasing/gateway.py @@ -0,0 +1,1476 @@ +"""The front door: the LiteLLM gateway, its routes and keys, and the UI and +proxy beside it. + +Engines (vLLM, Ollama) are the backend's business; everything a client +talks to is here. A backend hands the gateway where its models are (route +rows), and the gateway renders its Compose services and config, keeps the +route registry, reconciles dynamic routes through LiteLLM's admin API, and +manages the master key. The compose backend runs it beside its engines; the +kubeai backend runs it alone, in front of a cluster. +""" + +from __future__ import annotations + +import hashlib +import json +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Callable + +import yaml + +from ..config import PINNED_IMAGES +from ..config import DEFAULT_PORTS +from ..env_utils import ensure_secret, parse_env_file, write_env_file +from .backend import ConvergeScaffold +from .models import Deployment, served_name +from .naming import ( + OLLAMA_CONTAINER_PORT, + VLLM_CONTAINER_PORT, + ollama_service_name, + ollama_service_name_for, + vllm_service_name, + vllm_service_name_for, +) +from .residency import ENGINE_LABEL + +LITELLM_CONTAINER_PORT = 4000 +LITELLM_CONFIG_FILENAME = 'litellm_config.yaml' +LITELLM_SERVICE = 'litellm' +API_KEY_ENV = 'LITELLM_MASTER_KEY' +# The key LiteLLM encrypts credentials stored in its database with. Unset, it +# uses the master key -- so rotating the master key would make every +# DB-stored route undecryptable. `set_master_key` pins it to the pre-rotation +# master key the first time the key changes; it must never change after that. +SALT_KEY_ENV = 'LITELLM_SALT_KEY' + +# Dynamic-routing (admin-API) extras. When dynamic routing is on, the gateway's +# route table is managed live via LiteLLM's admin API against a Postgres-backed +# model store, instead of a static config file. See render_compose + +# ComposeBackend._reconcile_routes and docs/litellm-gateway-routing.md. +LITELLM_ROUTES_FILENAME = 'litellm_routes.json' # rendered desired route set +# Append-only route registry for static-superset mode: accumulates the semantic +# route inputs (served name / engine / host) of every catalog *and* every live +# deployment ever merged, across all runbooks sharing this state dir. The gateway +# `model_list` is rendered from the whole registry, so a converge under one +# runbook's catalog can no longer strip another's still-live routes, and once +# every catalog has been merged once the rendered config is byte-stable (the +# gateway is never recreated). See docs/litellm-gateway-routing.md and +# ComposeBackend._update_route_registry. +LITELLM_REGISTRY_FILENAME = 'litellm_registry.json' +LITELLM_REGISTRY_VERSION = 1 +# A route-registry row for a server this project does not run: +# ``{'engine': UPSTREAM_ROUTE, 'served': , 'api_base': }``. +UPSTREAM_ROUTE = 'upstream' +POSTGRES_SERVICE = 'postgres-litellm' +POSTGRES_CONTAINER_PORT = 5432 +POSTGRES_DB_NAME = 'litellm' +POSTGRES_DB_USER = 'litellm' +DB_PASSWORD_ENV = 'LITELLM_DB_PASSWORD' # managed secret in the sidecar .env +WEBUI_SECRET_ENV = 'WEBUI_SECRET_KEY' # Open WebUI's session key, likewise +# Marks a LiteLLM route as infer-stack-managed, so reconcile only ever deletes +# routes it created (never a model added by hand through the UI/admin API). +ROUTE_ID_PREFIX = 'isr-' + +#: Budget for reconciling dynamic routes against the gateway: listing, POSTs +#: and verification together. This default is the bootstrap budget (a fresh +#: gateway waits on Postgres health and runs DB migrations); it preserves the +#: previous 90 x 2 s listing retry. +ROUTE_RECONCILE_BOOTSTRAP_S = 180.0 +#: Budget when the gateway was already running before this apply (steady state). +#: Short, because the controller holds its host-wide lock while applying; on +#: expiry the change stays pending and the next applying operation retries. +ROUTE_RECONCILE_STEADY_S = 20.0 + + +def _vllm_route_entry( + model_name: str, served: str, api_base: str +) -> dict[str, Any]: + """One LiteLLM ``model_list`` entry routing ``model_name`` to a vLLM upstream. + + Shared by every render path (legacy per-deployment, catalog-superset, and + the route registry) so a registry-rendered entry can never drift from what + the catalog/deployment paths produce for the same endpoint.""" + return { + 'model_name': model_name, + 'litellm_params': { + 'model': f'openai/{served}', + 'api_base': api_base, + 'api_key': 'EMPTY', + }, + } + + +def _ollama_route_entry( + model_name: str, tag: str, api_base: str +) -> dict[str, Any]: + """One LiteLLM ``model_list`` entry routing ``model_name`` to an Ollama tag + (see :func:`_vllm_route_entry` for why this is factored out).""" + return { + 'model_name': model_name, + 'litellm_params': { + 'model': f'ollama/{tag}', + 'api_base': api_base, + }, + } + + +def _litellm_model_list( + deployments: list[Deployment], assignments: dict[str, list[int]] +) -> list[dict[str, Any]]: + """One LiteLLM ``model_list`` entry per served endpoint alias.""" + entries: list[dict[str, Any]] = [] + for deployment in sorted(deployments, key=lambda g: (g.created_at, g.id)): + if deployment.id not in assignments: + continue + if deployment.engine == 'vllm': + served = served_name(deployment) + api_base = f'http://{vllm_service_name(deployment)}:8000/v1' + for endpoint in sorted(deployment.served): + entries.append(_vllm_route_entry(endpoint, served, api_base)) + elif deployment.engine == 'ollama': + api_base = f'http://{ollama_service_name(deployment)}:{OLLAMA_CONTAINER_PORT}' + for endpoint, payload in sorted(deployment.served.items()): + tag = payload.get('model', endpoint) + entries.append(_ollama_route_entry(endpoint, tag, api_base)) + return entries + + +def _litellm_model_list_from_catalog(catalog: Any) -> list[dict[str, Any]]: + """A *static superset* ``model_list``: one route per catalog endpoint. + + Unlike :func:`_litellm_model_list` (which routes only the currently-placed + deployments), this routes *every* catalog endpoint to its deterministic + upstream host (:func:`vllm_service_name_for` / :func:`ollama_service_name_for`). + The resulting config therefore depends only on the catalog, not on which + models happen to be up — so acquiring/releasing a model leaves the gateway's + config (and its container) untouched (no blip). A route whose upstream is not + currently running simply errors/cools-down until it comes up; the + ``router_settings`` below make that warmup self-healing. ``/v1/models`` lists + the whole catalog (some upstreams down) rather than only the live set. + + This static-superset path is the default. Its one limitation — it cannot give + same-model ``--dedicated`` deployments distinct upstreams, and cannot route + non-catalog acquires without a config change — is addressed by the opt-in + *dynamic routing* mode (``dynamic_routing=True``), which manages routes live + via LiteLLM's admin API against a Postgres model store (see + :func:`_litellm_routes`, :meth:`ComposeBackend._reconcile_routes`, and + ``docs/litellm-gateway-routing.md``). The two are mutually exclusive per + converge; this function is used only when dynamic routing is off. + """ + entries: list[dict[str, Any]] = [] + for name in sorted(getattr(catalog, 'endpoints', {})): + try: + req = catalog.resolve_endpoint(name) + except Exception: # noqa: BLE001 - a bad endpoint must not break the gateway + continue + if req.engine == 'vllm': + served = req.served.get('served_model_name') or name + api_base = ( + f'http://{vllm_service_name_for(served)}:{VLLM_CONTAINER_PORT}/v1' + ) + entries.append(_vllm_route_entry(name, served, api_base)) + elif req.engine == 'ollama': + host = req.spec.get('host') or req.host + tag = req.served.get('model') or name + api_base = ( + f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' + ) + entries.append(_ollama_route_entry(name, tag, api_base)) + return entries + + +# -- Route registry (static-superset persistence) -------------------------- +# +# The registry stores *semantic* route inputs (served name / engine / host), +# never rendered LiteLLM entries — render derives entries through the same +# helpers the catalog/deployment paths use (:func:`_litellm_model_list_from_registry`), +# so a future renderer change propagates to old registry rows automatically. +# All functions here are pure; the backend owns the file I/O and locking. + + +def _registry_incoming_from_catalog(catalog: Any) -> dict[str, dict[str, Any]]: + """Semantic route rows for every resolvable endpoint of ``catalog``. + + Mirrors :func:`_litellm_model_list_from_catalog`'s iteration (unresolvable + endpoints skipped) but emits registry rows keyed by endpoint name. A vLLM + row carries only ``served`` (the upstream host is re-derived at render via + :func:`vllm_service_name_for`); an Ollama row carries ``model`` (tag) + + ``host``.""" + incoming: dict[str, dict[str, Any]] = {} + for name in sorted(getattr(catalog, 'endpoints', {})): + try: + req = catalog.resolve_endpoint(name) + except Exception: # noqa: BLE001 - a bad endpoint must not break the gateway + continue + if req.engine == 'vllm': + served = req.served.get('served_model_name') or name + incoming[name] = {'engine': 'vllm', 'served': served} + elif req.engine == 'ollama': + host = req.spec.get('host') or req.host + tag = req.served.get('model') or name + incoming[name] = {'engine': 'ollama', 'model': tag, 'host': host} + return incoming + + +def _registry_incoming_from_deployments( + deployments: list[Deployment], assignments: dict[str, list[int]] +) -> dict[str, dict[str, Any]]: + """Semantic route rows for every *placed* deployment in ``assignments``. + + ``deployments`` is the full ``desired`` set (which spans all runbooks via + the shared ledger), so this keeps non-catalog / dedicated acquires routable + and — because the registry persists — routable past release. One row per key + of ``deployment.served`` (a coalesced deployment can back several endpoint + aliases). Only ``vllm``/``ollama`` engines contribute; ``RESERVED_ENGINE`` + and unknown engines render no service, so they contribute no row — exactly + as :func:`render_compose`'s service loop skips them. + + The vLLM ``served`` uses the same fallback chain as :func:`vllm_service_name` + (``spec['served_model_name'] or sorted(served)[0] or id``), so a + catalog-listed endpoint acquired live reduces to the identical row a catalog + merge produces — live-vs-released status never moves the rendered bytes.""" + incoming: dict[str, dict[str, Any]] = {} + for deployment in deployments: + if deployment.id not in assignments: + continue + if deployment.engine == 'vllm': + served = served_name(deployment) + for endpoint in sorted(deployment.served): + incoming[endpoint] = {'engine': 'vllm', 'served': served} + elif deployment.engine == 'ollama': + host = deployment.spec.get('host') or deployment.id + for endpoint, payload in sorted(deployment.served.items()): + tag = payload.get('model', endpoint) + incoming[endpoint] = { + 'engine': 'ollama', + 'model': tag, + 'host': host, + } + return incoming + + +def _litellm_model_list_from_registry( + registry: dict[str, Any] +) -> list[dict[str, Any]]: + """Render the gateway ``model_list`` from the whole accumulated registry. + + Iterates ``sorted(entries)`` (determinism, §8) and derives each upstream + ``api_base`` through the live naming helpers, so the registry never becomes + a rendered-config parse surface.""" + entries: list[dict[str, Any]] = [] + rows = registry.get('entries', {}) if isinstance(registry, dict) else {} + for name in sorted(rows): + entry = registry_route_entry(name, rows[name]) + if entry is not None: + entries.append(entry) + return entries + + +def registry_route_entry(name: str, row: Any) -> dict[str, Any] | None: + """The LiteLLM entry one registry row renders to, or ``None`` if it cannot. + + The one derivation of a row's upstream: the render uses it, and so does + ``routes list``. Upstreams come from the live naming helpers, so the + registry never becomes a rendered-config parse surface. + """ + if not isinstance(row, dict): + return None + engine = row.get('engine') + if engine == 'vllm': + served = row.get('served') or name + api_base = f'http://{vllm_service_name_for(served)}:{VLLM_CONTAINER_PORT}/v1' + return _vllm_route_entry(name, served, api_base) + if engine == 'ollama': + tag = row.get('model') or name + host = row.get('host') or name + api_base = f'http://{ollama_service_name_for(host)}:{OLLAMA_CONTAINER_PORT}' + return _ollama_route_entry(name, tag, api_base) + if engine == UPSTREAM_ROUTE and row.get('api_base'): + # An OpenAI-compatible server this project does not run (a KubeAI + # cluster's gateway): the row carries its address and the name it + # serves the model under. + return _vllm_route_entry(name, row.get('served') or name, str(row['api_base'])) + return None + + +def upstream_route(deployment_id: str, endpoint: str, served: str, + api_base: str) -> dict[str, Any]: + """A dynamic route to a server this project does not run (a KubeAI Model). + + The same entry shape, and the same deterministic id, as a Compose engine's + dynamic route (:func:`_litellm_routes`). + """ + entry = _vllm_route_entry(endpoint, served, api_base) + entry['model_info'] = {'id': _route_id(deployment_id, endpoint)} + return entry + + +def _merge_route_registry( + existing: dict[str, Any], incoming: dict[str, dict[str, Any]] +) -> tuple[dict[str, Any], list[str]]: + """Merge ``incoming`` semantic rows into ``existing`` (append-only). + + Idempotent (merging identical rows is a no-op) and additive (never removes a + row). On a conflict — same key, different row — *incoming wins* and a warning + naming both definitions is emitted; the changed definition changes the + rendered bytes, which is the one justified recreate. The existing ``version`` + is preserved (an unknown version merged under is not silently rewritten to + the current schema; see :meth:`ComposeBackend._load_route_registry`).""" + version = LITELLM_REGISTRY_VERSION + entries: dict[str, dict[str, Any]] = {} + if isinstance(existing, dict): + version = existing.get('version', LITELLM_REGISTRY_VERSION) + prior = existing.get('entries') + if isinstance(prior, dict): + entries = {k: v for k, v in prior.items()} + warnings: list[str] = [] + for name in sorted(incoming): + row = incoming[name] + if name in entries and entries[name] != row: + warnings.append( + f"route {name!r} redefined: {entries[name]} -> {row} " + '(incoming wins; gateway will be recreated once)' + ) + entries[name] = row + return {'version': version, 'entries': entries}, warnings + + +def _seed_registry_from_litellm_config( + config_text: str, +) -> tuple[dict[str, Any], list[str]]: + """One-shot upgrade seed: recover registry rows from a rendered + ``litellm_config.yaml`` so the first post-upgrade converge does not strip + the other runbooks' routes. + + Single-format, migration-time only (no cross-version promise): ``openai/`` + inverts *exactly* to ``{engine: vllm, served}``. Ollama rows are skipped with + a warning — the host survives only as a non-invertible ``dns_slug`` inside + ``api_base`` — and re-enter the registry at the next converge that has them + in its catalog or live set. Anything else unparseable is likewise skipped.""" + entries: dict[str, dict[str, Any]] = {} + warnings: list[str] = [] + try: + data = yaml.safe_load(config_text) or {} + except Exception: # noqa: BLE001 - a torn file must not brick seeding + return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}}, [ + 'seed: litellm_config.yaml is unparseable; starting an empty registry' + ] + for entry in data.get('model_list', []) or []: + name = entry.get('model_name') + model = (entry.get('litellm_params') or {}).get('model', '') + if not name: + continue + if isinstance(model, str) and model.startswith('openai/'): + entries[name] = {'engine': 'vllm', 'served': model[len('openai/'):]} + elif isinstance(model, str) and model.startswith('ollama/'): + warnings.append( + f"seed: skipping Ollama route {name!r} (host not recoverable " + 'from the rendered api_base; it re-enters at its next converge)' + ) + else: + warnings.append(f'seed: skipping unparseable route {name!r}') + return {'version': LITELLM_REGISTRY_VERSION, 'entries': entries}, warnings + + +def _dump_route_registry(registry: dict[str, Any]) -> str: + """Canonical, byte-stable serialization (§3): sorted keys + trailing + newline. A nondeterministic dump would manufacture phantom hash changes.""" + return json.dumps(registry, sort_keys=True, indent=2) + '\n' + + +def _route_id(deployment_id: str, endpoint: str) -> str: + """Deterministic LiteLLM model id for one (deployment, endpoint) route. + + Stable across converges, so route reconcile (:meth:`ComposeBackend. + _reconcile_routes`) can identify one logical route across renders. A route + whose id disappears is deleted by exactly this id; a route whose id remains + but whose observable routing semantics drifted is replaced under the same id. + The ``isr-`` prefix marks it + infer-stack-managed so reconcile never deletes a model someone added by hand. + """ + digest = hashlib.sha256(f'{deployment_id}|{endpoint}'.encode()).hexdigest() + return f'{ROUTE_ID_PREFIX}{digest[:32]}' + + +def _litellm_routes( + deployments: list[Deployment], assignments: dict[str, list[int]] +) -> list[dict[str, Any]]: + """Desired LiteLLM route set for the *live* deployments (dynamic routing). + + One entry per (placed deployment, served endpoint), addressing the + deployment's **own** unique upstream service (:func:`vllm_service_name` with + ``unique=True``). Several dedicated deployments of the same model therefore + yield several entries that share one public ``model_name`` but point at + distinct upstreams — LiteLLM load-balances the alias across them, so each + runs on its own GPU while clients still ask for the single name. Each entry + carries a deterministic ``model_info.id`` (:func:`_route_id`) so applying the + set via the admin API is an idempotent diff, not fire-and-forget calls. + """ + entries: list[dict[str, Any]] = [] + for deployment in sorted(deployments, key=lambda g: (g.created_at, g.id)): + if deployment.id not in assignments: + continue + if deployment.engine == 'vllm': + served = served_name(deployment) + api_base = ( + f'http://{vllm_service_name(deployment, unique=True)}' + f':{VLLM_CONTAINER_PORT}/v1' + ) + for endpoint in sorted(deployment.served): + entries.append(upstream_route(deployment.id, endpoint, served, api_base)) + elif deployment.engine == 'ollama': + api_base = ( + f'http://{ollama_service_name(deployment)}:{OLLAMA_CONTAINER_PORT}' + ) + for endpoint, payload in sorted(deployment.served.items()): + tag = payload.get('model', endpoint) + entries.append( + { + 'model_name': endpoint, + 'litellm_params': { + 'model': f'ollama/{tag}', + 'api_base': api_base, + }, + 'model_info': {'id': _route_id(deployment.id, endpoint)}, + } + ) + return entries + + +CONFIG_HASH_LABEL = 'infer-stack.config-hash' + + +def _postgres_service( + images: dict[str, str], state: dict[str, str] +) -> dict[str, Any]: + """Postgres backing LiteLLM's runtime model store (dynamic routing only). + + LiteLLM's admin API (``/model/new`` / ``/model/delete``) only functions with + ``STORE_MODEL_IN_DB=true`` + a database, so dynamic routing needs a DB. This + is an **internal** service (no published host port); LiteLLM reaches it on + the compose network at ``postgres-litellm:5432``. The password is the managed + :data:`DB_PASSWORD_ENV` secret in the sidecar ``.env`` (interpolated by + ``docker compose --env-file``), so it never appears literally in the YAML. + The healthcheck lets the litellm service ``depends_on`` it (condition: + service_healthy) so the gateway only starts once the DB can accept queries. + """ + data_path = state.get('postgres_litellm') or str( + Path(next(iter(state.values()), '.')).parent / 'postgres-litellm' + ) + return { + 'image': images.get('postgres', PINNED_IMAGES['postgres']), + 'environment': { + 'POSTGRES_USER': POSTGRES_DB_USER, + 'POSTGRES_PASSWORD': '${' + DB_PASSWORD_ENV + '}', + 'POSTGRES_DB': POSTGRES_DB_NAME, + }, + 'volumes': [f'{data_path}:/var/lib/postgresql/data'], + 'restart': 'unless-stopped', + 'labels': {ENGINE_LABEL: 'postgres'}, + 'healthcheck': { + 'test': [ + 'CMD-SHELL', + f'pg_isready -U {POSTGRES_DB_USER} -d {POSTGRES_DB_NAME}', + ], + 'interval': '5s', + 'timeout': '5s', + 'retries': 30, + 'start_period': '30s', + }, + } + + +def _litellm_service( + service_names: list[str], + host_port: int, + images: dict[str, str], + aux_dir: str, + master_key: str | None = None, + config_hash: str | None = None, + *, + dynamic_routing: bool = False, + salt_key: bool = False, +) -> dict[str, Any]: + # Reference the managed key via ${...} rather than baking the literal secret + # into the compose YAML. Its value lives in the sidecar .env next to the + # compose file (written by master_key()), which `docker compose --env-file` + # loads for interpolation — so the container and the readiness probe (which + # reads the same .env) still agree regardless of the caller's shell env. + key_value = ( + '${' + API_KEY_ENV + '}' + if master_key is not None + else '${' + API_KEY_ENV + ':-sk-local}' + ) + environment = {API_KEY_ENV: key_value} + if salt_key: + # Only when the .env has one: LiteLLM treats an EMPTY salt as a key, + # so a `${...:-}` default would silently change the encryption key. + environment[SALT_KEY_ENV] = '${' + SALT_KEY_ENV + '}' + if dynamic_routing: + # DB-backed runtime model store so the admin API (/model/new, + # /model/delete) works; the gateway then never needs recreating to learn + # a route. Both are read from the env by LiteLLM. The password is + # interpolated from the sidecar .env, so no secret lands in the YAML. + environment['DATABASE_URL'] = ( + f'postgresql://{POSTGRES_DB_USER}:${{{DB_PASSWORD_ENV}}}' + f'@{POSTGRES_SERVICE}:{POSTGRES_CONTAINER_PORT}/{POSTGRES_DB_NAME}' + ) + environment['STORE_MODEL_IN_DB'] = 'True' + labels = {ENGINE_LABEL: 'litellm'} + if config_hash is not None: + # LiteLLM reads its routing config once at startup; the file is bind- + # mounted, so a config change alone does NOT change this service's spec + # and `docker compose up -d` would leave the old container (and old + # routes) running. Stamping the config hash onto a label makes the spec + # change exactly when the config does, so converge recreates LiteLLM and + # it picks up new/removed aliases. Without this, coalescing a second + # alias onto a live deployment never becomes routable (readiness times out). + labels[CONFIG_HASH_LABEL] = config_hash + service: dict[str, Any] = { + 'image': images['litellm'], + 'command': [ + '--config', + '/etc/litellm/config.yaml', + '--port', + str(LITELLM_CONTAINER_PORT), + ], + 'ports': [f'{host_port}:{LITELLM_CONTAINER_PORT}'], + 'volumes': [f'{aux_dir}/{LITELLM_CONFIG_FILENAME}:/etc/litellm/config.yaml:ro'], + 'environment': environment, + 'restart': 'unless-stopped', + 'labels': labels, + } + if dynamic_routing: + # Wait for the DB to accept queries before the gateway boots; do NOT add + # per-model depends_on (that would churn the spec, i.e. blip, on every + # model change). The route table is filled in afterward via the API. + service['depends_on'] = { + POSTGRES_SERVICE: {'condition': 'service_healthy'} + } + elif service_names: + # Only wait on upstreams when there are any (zero models -> empty gateway). + service['depends_on'] = sorted(service_names) + return service + + +OPEN_WEBUI_SERVICE = 'open-webui' +OPEN_WEBUI_CONTAINER_PORT = 8080 + +NGINX_SERVICE = 'reverse-proxy' +NGINX_CONTAINER_PORT = 80 +NGINX_CONFIG_FILENAME = 'nginx.conf' + + +def _open_webui_service( + host_port: int, + images: dict[str, str], + state: dict[str, str], + master_key: str | None, + *, + openai_urls: list[str] | None = None, + ollama_urls: list[str] | None = None, + depends_on: list[str] | None = None, + run_as: str | None = None, +) -> dict[str, Any]: + """A managed Open WebUI pointed at whatever front door is available. + + ``run_as`` (``uid:gid``, see :func:`open_webui_run_as`) runs it as the + owner of its data directory, so what it writes there stays that owner's; + None keeps the image's default user (root). + + + Open WebUI holds two independent kinds of connection, wired here from the + rendered services: + + * **OpenAI** (``openai_urls``) — the chat/completions front door. This is the + LiteLLM gateway when it is enabled (so every declared endpoint alias is + reachable at one URL); with LiteLLM off it falls back to the rendered + upstreams' own ``/v1`` (a single vLLM/Ollama service, or several joined as + ``OPENAI_API_BASE_URLS``). With nothing to point at, the OpenAI API is + disabled rather than left dangling. + * **Ollama** (``ollama_urls``) — the *native* Ollama API of any rendered + Ollama daemon. This is what lets you pull/run/delete models from the UI + and have the daemon load them on demand, independent of LiteLLM — i.e. a + true drop-in for a hand-run ``ollama`` + Open WebUI stack. + + The spec is kept as independent of which models are live as it can be: the + LiteLLM URL is fixed, and the Ollama daemon's service name is its stable + structural id, so adding/removing other models does not rewrite this service + and ``docker compose up -d`` leaves the UI running (the legacy "the UI never + blinks" behavior). Chat history persists under the data dir. + """ + # Reference the managed key via ${...} (resolved from the sidecar .env, see + # _litellm_service) instead of inlining the secret into the compose YAML. + key_value = ( + '${' + API_KEY_ENV + '}' + if master_key is not None + else '${' + API_KEY_ENV + ':-sk-local}' + ) + data_path = state.get('open_webui') or str( + Path(next(iter(state.values()), '.')).parent / 'open-webui' + ) + openai_urls = list(openai_urls or []) + ollama_urls = list(ollama_urls or []) + env: dict[str, str] = { + # Single-user workstation default; the port shouldn't be exposed + # publicly. Tracked as a knob in dev/leasing-followups.md. + 'WEBUI_AUTH': 'False', + # A managed secret (the .env, like the master key): without it the + # image writes a generated one into its own directory, which a + # non-root user cannot, and a recreate would sign everyone out. + WEBUI_SECRET_ENV: '${' + WEBUI_SECRET_ENV + '}', + } + if run_as: + # Its bundled static assets are copied at startup; into the data + # directory, which this user can write, not the image's own. + env['STATIC_DIR'] = '/app/backend/data/static' + env['HOME'] = '/tmp' + if openai_urls: + env['ENABLE_OPENAI_API'] = 'True' + if len(openai_urls) == 1: + env['OPENAI_API_BASE_URL'] = openai_urls[0] + else: + env['OPENAI_API_BASE_URLS'] = ';'.join(openai_urls) + env['OPENAI_API_KEY'] = key_value + else: + env['ENABLE_OPENAI_API'] = 'False' + if ollama_urls: + env['ENABLE_OLLAMA_API'] = 'True' + if len(ollama_urls) == 1: + env['OLLAMA_BASE_URL'] = ollama_urls[0] + else: + env['OLLAMA_BASE_URLS'] = ';'.join(ollama_urls) + else: + env['ENABLE_OLLAMA_API'] = 'False' + service: dict[str, Any] = { + 'image': images['open_webui'], + 'ports': [f'{host_port}:{OPEN_WEBUI_CONTAINER_PORT}'], + 'environment': env, + 'volumes': [f'{data_path}:/app/backend/data'], + 'restart': 'unless-stopped', + 'labels': {ENGINE_LABEL: 'open-webui'}, + } + if run_as: + service['user'] = run_as + if depends_on: + service['depends_on'] = sorted(depends_on) + return service + + +def open_webui_run_as(data_path: str | Path) -> tuple[str | None, str]: + """``(uid:gid or None, why)``: who Open WebUI should run as. + + Its data directory's owner, so a data root stays removable by whoever + owns it (as root, the container left files only root could delete). A + directory not made yet will be made by this process, so its user. None, + the image default (root), when the directory is root's, or when it or + anything directly in it belongs to someone else: a data directory written + by a root container before, whose files Open WebUI could no longer open. + + >>> import os, tempfile + >>> d = tempfile.mkdtemp() + >>> open_webui_run_as(os.path.join(d, 'open-webui'))[0] == f'{os.getuid()}:{os.stat(d).st_gid}' + True + """ + import os + + path = Path(data_path) + if not path.exists(): + parent = next((p for p in path.parents if p.exists()), Path('/')) + return f'{os.getuid()}:{parent.stat().st_gid}', 'new directory' + owner = path.stat() + if owner.st_uid == 0: + return None, f'{path} is root\'s' + try: + foreign = [p.name for p in path.iterdir() if p.lstat().st_uid != owner.st_uid] + except OSError: + foreign = ['?'] + if foreign: + return None, (f'{path} holds files another user owns ({", ".join(foreign[:3])}); ' + f'`sudo chown -R {owner.st_uid}:{owner.st_gid} {path}` lets Open ' + 'WebUI run as its owner') + return f'{owner.st_uid}:{owner.st_gid}', 'the directory\'s owner' + + +def _nginx_conf(*, litellm: bool, ui: bool) -> str: + """A minimal HTTP reverse-proxy conf: one origin, path-routed. + + ``/v1/`` -> the LiteLLM gateway (the OpenAI API), ``/`` -> Open WebUI (or the + gateway when there's no UI). Plain HTTP — no TLS, no auth — so the value is + "one port, nothing to remember", not security. The ``map`` is valid here + because a ``conf.d/*.conf`` file is included in nginx's ``http`` context. + """ + api = f'http://{LITELLM_SERVICE}:{LITELLM_CONTAINER_PORT}' + locations = '' + if litellm: + locations += ( + ' location /v1/ {\n' + f' proxy_pass {api}/v1/;\n' + ' proxy_set_header Host $host;\n' + ' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n' + ' proxy_set_header X-Forwarded-Proto $scheme;\n' + ' proxy_read_timeout 600s;\n' + ' }\n' + ) + # `/` serves the UI when present, else the gateway (so hitting the host root + # still lands somewhere useful). Upgrade headers keep Open WebUI's websockets + # working; client_max_body_size 0 allows large uploads. + if ui: + root = f'http://{OPEN_WEBUI_SERVICE}:{OPEN_WEBUI_CONTAINER_PORT}' + elif litellm: + root = api + else: + root = '' + if root: + locations += ( + ' location / {\n' + f' proxy_pass {root};\n' + ' proxy_http_version 1.1;\n' + ' proxy_set_header Upgrade $http_upgrade;\n' + ' proxy_set_header Connection $connection_upgrade;\n' + ' proxy_set_header Host $host;\n' + ' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n' + ' proxy_set_header X-Forwarded-Proto $scheme;\n' + ' }\n' + ) + return ( + 'map $http_upgrade $connection_upgrade {\n' + ' default upgrade;\n' + " '' close;\n" + '}\n\n' + 'server {\n' + f' listen {NGINX_CONTAINER_PORT};\n' + ' server_name _;\n' + ' client_max_body_size 0;\n' + f'{locations}' + '}\n' + ) + + +def _nginx_service( + host_port: int, + images: dict[str, str], + *, + aux_dir: str, + depends_on: list[str], + config_path: str | None = None, + config_hash: str | None = None, +) -> dict[str, Any]: + # BYO config (config_path) is mounted verbatim; otherwise the generated + # nginx.conf in the state dir is used. + mount = config_path or f'{aux_dir}/{NGINX_CONFIG_FILENAME}' + labels = {ENGINE_LABEL: 'nginx'} + if config_hash is not None: + # Same trick as LiteLLM: the conf is bind-mounted, so stamp its hash on a + # label to force a recreate when the routing changes. + labels[CONFIG_HASH_LABEL] = config_hash + service: dict[str, Any] = { + 'image': images['nginx'], + 'ports': [f'{host_port}:{NGINX_CONTAINER_PORT}'], + 'volumes': [f'{mount}:/etc/nginx/conf.d/default.conf:ro'], + 'restart': 'unless-stopped', + 'labels': labels, + } + if depends_on: + service['depends_on'] = sorted(depends_on) + return service + + +def set_master_key(env_path: Path, key: str) -> None: + """Replace the LiteLLM master key in ``env_path`` without losing DB routes. + + The first time the key changes, the old one is pinned as + ``LITELLM_SALT_KEY`` -- the value LiteLLM has been encrypting stored + credentials with. After that the salt stays put and only the key moves. + """ + if not key.startswith('sk-'): + # master_key() would silently replace it on the next render. + raise ValueError(f'{API_KEY_ENV} must start with "sk-" (LiteLLM rejects others)') + existing = parse_env_file(env_path) + values = {API_KEY_ENV: key} + old = existing.get(API_KEY_ENV, '').strip() + if old and old != key and not existing.get(SALT_KEY_ENV, '').strip(): + values[SALT_KEY_ENV] = old + write_env_file(env_path, values) + + +@dataclass +class FrontDoor: + """The rendered front door: its Compose services and config files.""" + + services: dict[str, Any] + litellm_config: str | None + nginx_config: str | None + litellm_routes: list[dict[str, Any]] | None + + +def render_front_door( + deployments: list[Deployment], + assignments: dict[str, list[int]], + *, + engine_services: list[str], + vllm_v1_urls: list[str], + ollama_native_urls: list[str], + images: dict[str, str], + state: dict[str, str], + litellm: bool, + litellm_port: int, + litellm_master_key: str | None, + litellm_salt_key: bool, + ui: bool, + ui_port: int, + reverse_proxy: bool, + reverse_proxy_port: int, + reverse_proxy_config: str | None, + aux_dir: str | Path | None, + catalog: Any, + route_registry: dict[str, Any] | None, + dynamic_routing: bool, + upstream_routes: list[dict[str, Any]] | None = None, + ui_run_as: str | None = None, +) -> FrontDoor: + """Render the gateway, its database, Open WebUI and the reverse proxy. + + The engines are the caller's: it passes the services it rendered + (``engine_services``, only for the legacy per-model ``depends_on``) and + the in-network URLs a UI with no gateway can talk to directly. + ``upstream_routes`` are dynamic routes to servers this project does not + run (see :func:`upstream_route`). + """ + services: dict[str, Any] = {} + litellm_config = None + litellm_routes = None + # The front door (gateway + UI) is rendered whenever it's enabled, even with + # zero models — it's a standing entry point, not a per-model service. So + # releasing/evicting every model leaves an empty gateway (and an empty Open + # WebUI picker) up instead of tearing the whole stack down; only an explicit + # `stack down` removes it. With no models the model_list is simply empty. + if litellm: + # Three route-table strategies, in order of preference: + # * DYNAMIC ROUTING: the rendered config is a STATIC base (empty + # model_list); the real routes live in Postgres and are applied to the + # running gateway via the admin API (see _reconcile_routes). The config + # hash never changes as models come/go, so the gateway is never + # recreated — no blip, and per-deployment routing works (so same-model + # --dedicated deployments each get their own upstream). + # * ROUTE REGISTRY (static-superset default from ComposeBackend): render + # from the whole accumulated registry (every catalog + live deployment + # ever merged, across all runbooks). Byte-stable once seeded, so the + # gateway is never recreated and a cross-catalog converge can no longer + # strip another runbook's routes. The backend loads/merges/writes the + # registry and passes the merged dict in; this function stays pure. + # * STATIC SUPERSET (catalog): one route per catalog endpoint to a + # deterministic host; config depends only on the catalog, so the + # gateway is not recreated as models come/go (no blip) but same-model + # dedicated collapses to one upstream. Unreachable from ComposeBackend + # once the registry is wired; kept for direct callers/tests. + # * LEGACY (no catalog): route only the placed deployments; churns the + # config (and recreates the gateway) on every model change. + if dynamic_routing: + entries: list[dict[str, Any]] = [] + litellm_routes = (_litellm_routes(deployments, assignments) + + list(upstream_routes or [])) + litellm_depends: list[str] = [] + elif route_registry is not None: + entries = _litellm_model_list_from_registry(route_registry) + litellm_depends = [] # no per-model depends_on -> no churn + elif catalog is not None: + entries = _litellm_model_list_from_catalog(catalog) + litellm_depends = [] # no per-model depends_on -> no churn + else: + entries = _litellm_model_list(deployments, assignments) + litellm_depends = list(engine_services) + litellm_config = yaml.safe_dump( + { + 'model_list': entries, + 'general_settings': { + 'master_key': f'os.environ/{API_KEY_ENV}' + }, + # An upstream vLLM/Ollama is unreachable only briefly, while it + # loads its model (LiteLLM does not wait for upstream health to + # start). Retry transient connection errors and don't park a + # model in a long cooldown, so the warmup window is self-healing + # instead of surfacing as client 500s ("Connection error. + # Received Model Deployment=…"). + 'router_settings': { + 'num_retries': 3, + 'timeout': 600, + 'cooldown_time': 5, + 'allowed_fails': 100, + }, + }, + sort_keys=False, + ) + config_hash = hashlib.sha256( + litellm_config.encode('utf-8') + ).hexdigest()[:12] + if dynamic_routing: + services[POSTGRES_SERVICE] = _postgres_service(images, state) + services[LITELLM_SERVICE] = _litellm_service( + litellm_depends, + litellm_port, + images, + str(aux_dir or '.'), + master_key=litellm_master_key, + config_hash=config_hash, + dynamic_routing=dynamic_routing, + salt_key=litellm_salt_key, + ) + + # Open WebUI is its own standing front door, rendered whenever ``ui`` is set + # — it does NOT require LiteLLM. Its OpenAI connection prefers the gateway + # (one URL covers every alias) and falls back to the rendered vLLM upstreams' + # own /v1 when there is no gateway. Its native Ollama connection always + # points straight at any Ollama daemon, so you can pull/run models from the + # UI and have the daemon load them on demand — a true drop-in for a + # hand-run ollama + Open WebUI stack. depends_on lists only LiteLLM (the one + # service guaranteed present alongside the UI); the per-model upstreams come + # and go, so the UI tolerates them being absent rather than hard-depending. + # With a gateway the UI is a standing front door (renders even at zero + # models). Without one it is only meaningful pointed at a live upstream, so + # render it only when there is something to connect to — otherwise an empty + # desired set has nothing to run and converge tears the project down. + if ui and (litellm or vllm_v1_urls or ollama_native_urls): + if litellm: + openai_urls = [f'http://{LITELLM_SERVICE}:{LITELLM_CONTAINER_PORT}/v1'] + ui_depends = [LITELLM_SERVICE] + else: + openai_urls = list(vllm_v1_urls) + ui_depends = [] + services[OPEN_WEBUI_SERVICE] = _open_webui_service( + ui_port, + images, + state, + litellm_master_key, + openai_urls=openai_urls, + ollama_urls=ollama_native_urls, + depends_on=ui_depends, + run_as=ui_run_as, + ) + + # Optional single-port HTTP reverse proxy fronting the gateway (+ UI). Needs + # the gateway, so it's only rendered alongside litellm. + nginx_config = None + if reverse_proxy and litellm: + depends = [LITELLM_SERVICE] + ([OPEN_WEBUI_SERVICE] if ui else []) + if reverse_proxy_config: + services[NGINX_SERVICE] = _nginx_service( + reverse_proxy_port, images, aux_dir=str(aux_dir or '.'), + depends_on=depends, config_path=reverse_proxy_config, + ) + else: + nginx_config = _nginx_conf(litellm=litellm, ui=ui) + services[NGINX_SERVICE] = _nginx_service( + reverse_proxy_port, images, aux_dir=str(aux_dir or '.'), + depends_on=depends, + config_hash=hashlib.sha256( + nginx_config.encode('utf-8') + ).hexdigest()[:12], + ) + + return FrontDoor(services=services, litellm_config=litellm_config, + nginx_config=nginx_config, litellm_routes=litellm_routes) + + +class Gateway(ConvergeScaffold): + """The front door's state: settings, the managed keys, the route registry. + + Owned by the backend that runs it (``ComposeBackend.gateway``). The + backend supplies route rows for what it serves; this merges, persists and + renders them, reconciles dynamic routes through LiteLLM's admin API, and + keeps the ``.env`` secrets. It shares the backend's state directory (and + so its converge lock), because the gateway's files live beside the + compose project that runs it. + """ + + def __init__( + self, + state_dir: str | Path, + *, + ports: dict[str, int], + litellm: bool = True, + ui: bool = True, + reverse_proxy: bool = False, + reverse_proxy_port: int = 80, + dynamic_routing: bool = False, + http: Any = None, + sleep: Callable[[float], None] = time.sleep, + clock: Callable[[], float] = time.monotonic, + base_url: str | Callable[[], str] | None = None, + ): + self.state_dir = Path(state_dir) + # Where clients reach the gateway (no ``/v1``). None: this host, on the + # published port. The in-cluster gateway passes a callable, so the + # node address is looked up only when something asks. + self.base_url: str | Callable[[], str] | None = base_url + self.ports = ports + self.litellm = litellm + self.ui = ui + self.reverse_proxy = reverse_proxy + self.reverse_proxy_port = reverse_proxy_port + self.dynamic_routing = dynamic_routing + # None: `requests`, imported on first use. It costs ~75 ms, and a TUI + # launch or a ledger-only command never makes a request. + self._http = http + self._sleep = sleep + self._clock = clock + self.assume_yes = True + + @property + def http(self) -> Any: + if self._http is None: + import requests + + self._http = requests + return self._http + + @http.setter + def http(self, value: Any) -> None: + self._http = value + + @property + def litellm_port(self) -> int: + return self.ports.get('litellm', DEFAULT_PORTS['litellm']) + + @property + def ui_port(self) -> int: + return self.ports.get('open_webui', DEFAULT_PORTS['open_webui']) + + @property + def _env_path(self) -> Path: + return self.state_dir / '.env' + + @property + def _routes_file(self) -> Path: + return self.state_dir / LITELLM_ROUTES_FILENAME + + @property + def _registry_file(self) -> Path: + return self.state_dir / LITELLM_REGISTRY_FILENAME + + def master_key(self) -> str: + """The managed LiteLLM master key. + + infer-stack manages this secret in the state dir's ``.env``: reused if + already present (you may pin your own ``sk-`` key there), otherwise + generated and persisted. The caller doesn't need to invent or export it + — it is baked into the LiteLLM service, used by the readiness probe, and + shipped in the env-file descriptor (``infer-stack env KEY`` prints it). + """ + existing = parse_env_file(self._env_path) + key = ensure_secret(existing, API_KEY_ENV, prefix='sk-') + if key != existing.get(API_KEY_ENV): + write_env_file(self._env_path, {API_KEY_ENV: key}) + return key + + def rotate_master_key(self) -> dict[str, str | None]: + """Write a fresh master key to the ``.env``; return the values it replaced. + + Only the file: the gateway and Open WebUI pick it up when the next apply + recreates them (their fingerprints hash the key). The caller holds the + publication lock and passes the return value to + :meth:`restore_env` if that apply does not happen. + """ + before = parse_env_file(self._env_path) + set_master_key(self._env_path, ensure_secret({}, API_KEY_ENV, prefix='sk-')) + return {k: before.get(k) for k in (API_KEY_ENV, SALT_KEY_ENV)} + + def restore_env(self, values: dict[str, str | None]) -> None: + """Put back what :meth:`rotate_master_key` replaced.""" + from ..env_utils import remove_env_keys + + write_env_file(self._env_path, {k: v for k, v in values.items() if v is not None}) + remove_env_keys(self._env_path, [k for k, v in values.items() if v is None]) + + def gateway_accepts(self, key: str, *, wait: float = 0.0) -> bool | None: + """Does the gateway accept ``key``? ``None`` if it never answered. + + Polls ``/v1/models`` for up to ``wait`` seconds while the gateway is + unreachable or still starting. A definite no is 401/403, or the 400 + LiteLLM actually answers a wrong key with (measured on the pinned image). + """ + deadline = self._clock() + wait + while True: + try: + resp = self.http.get( + f'{self._gateway_base()}/v1/models', + headers={'Authorization': f'Bearer {key}'}, timeout=10.0, + ) + status = getattr(resp, 'status_code', 0) + except Exception: # noqa: BLE001 - not up yet + status = 0 + if status == 200: + return True + if status in (400, 401, 403): + return False + if self._clock() >= deadline: + return None + self._sleep(2.0) + + def webui_secret(self) -> str: + """Open WebUI's managed session key (the .env), made on first use.""" + existing = parse_env_file(self._env_path) + key = ensure_secret(existing, WEBUI_SECRET_ENV) + if key != existing.get(WEBUI_SECRET_ENV): + write_env_file(self._env_path, {WEBUI_SECRET_ENV: key}) + return key + + def db_password(self) -> str: + """The managed Postgres password for LiteLLM's model store. + + Same managed-secret pattern as :meth:`master_key`: reused if already + present in the state-dir ``.env`` (you may pin your own), else generated + and persisted. ``docker compose --env-file`` interpolates it into the + postgres + litellm services, so it never appears literally in the YAML. + Only used when ``dynamic_routing`` is on. ``token_urlsafe`` output is safe + inside the ``postgresql://`` URL (no ``@ : /`` characters). + """ + existing = parse_env_file(self._env_path) + pw = ensure_secret(existing, DB_PASSWORD_ENV) + if pw != existing.get(DB_PASSWORD_ENV): + write_env_file(self._env_path, {DB_PASSWORD_ENV: pw}) + return pw + + def _load_route_registry(self) -> dict[str, Any]: + """Read the route registry, tolerantly (fail-open — a broken registry + must never block a converge). + + Missing file → seed from the live ``litellm_config.yaml`` if present + (upgrade migration, §6), else an empty registry. An *unknown* schema + version whose ``entries`` still parses as a name→row map is preserved + as-is (render what's understood, warn, do NOT rewrite) rather than + reseeded, so a binary rollback doesn't discard the accumulated union. + Only a structurally unusable file (not a map / garbage JSON) falls back + to seeding.""" + from .._log import logger + + if not self._registry_file.exists(): + config = self.state_dir / LITELLM_CONFIG_FILENAME + if config.exists(): + seeded, warnings = _seed_registry_from_litellm_config( + config.read_text() + ) + for w in warnings: + logger.warning(' route registry: {}', w) + logger.info( + ' route registry: seeded {} vLLM route(s) from {}', + len(seeded['entries']), config.name, + ) + return seeded + return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}} + try: + data = json.loads(self._registry_file.read_text()) + except (OSError, json.JSONDecodeError) as exc: + logger.warning( + ' route registry: {} is unreadable ({}); rebuilding from seed', + self._registry_file.name, exc, + ) + data = None + if not isinstance(data, dict) or not isinstance( + data.get('entries'), dict + ): + config = self.state_dir / LITELLM_CONFIG_FILENAME + if config.exists(): + seeded, warnings = _seed_registry_from_litellm_config( + config.read_text() + ) + for w in warnings: + logger.warning(' route registry: {}', w) + return seeded + return {'version': LITELLM_REGISTRY_VERSION, 'entries': {}} + version = data.get('version') + if version != LITELLM_REGISTRY_VERSION: + logger.warning( + ' route registry: unknown schema version {!r} in {} — ' + 'rendering as-is without rewrite (fields this renderer does ' + 'not understand are ignored)', + version, self._registry_file.name, + ) + return data + + def _save_route_registry(self, merged: dict[str, Any]) -> None: + """Persist a merged registry if it changed (under the converge flock).""" + from .._log import logger + + existing = self._load_route_registry() + if merged == existing: + return + prior = existing.get('entries', {}) if isinstance(existing, dict) else {} + added = sorted(set(merged['entries']) - set(prior)) + updated = sorted( + k for k in merged['entries'] if k in prior and merged['entries'][k] != prior[k] + ) + if added: + logger.info(' route registry: +{} route(s): {}', len(added), ', '.join(added)) + if updated: + logger.info(' route registry: updated route(s): {}', ', '.join(updated)) + self._atomic_write(self._registry_file, _dump_route_registry(merged)) + + def merge_route_registry( + self, incoming: dict[str, dict[str, Any]] + ) -> dict[str, Any]: + """Public write path for out-of-converge registry seeds (``routes seed``). + + Takes the converge flock, read-merge-writes the registry, and returns the + merged dict. ``converge`` only ever merges the invoking process's own + catalog, so a standalone caller (seeding a *sibling* runbook's catalog) + needs this to fold extra rows in before the follow-up ``reconcile`` + renders+applies. The flock here and the one the subsequent converge takes + are sequential acquisitions, not nested — no reentrancy concern.""" + from .._log import logger + + with self._converge_lock(): + existing = self._load_route_registry() + merged, warnings = _merge_route_registry(existing, incoming) + for w in warnings: + logger.warning(' route registry: {}', w) + if merged != existing: + self._atomic_write( + self._registry_file, _dump_route_registry(merged) + ) + return merged + + def _gateway_base(self) -> str: + where = self.base_url + if where is None: + base = f'http://127.0.0.1:{self.litellm_port}' + elif isinstance(where, str): + base = where + else: + base = where() + return base.rstrip('/') + + def _auth_headers(self) -> dict[str, str]: + return {'Authorization': f'Bearer {self.master_key()}'} + + def _desired_routes(self) -> list[dict[str, Any]]: + """The rendered desired route set (litellm_routes.json), or empty.""" + try: + data = json.loads(self._routes_file.read_text()) + except (FileNotFoundError, json.JSONDecodeError): + return [] + return data if isinstance(data, list) else [] + + def _reconcile_routes( + self, *, deadline_s: float = ROUTE_RECONCILE_BOOTSTRAP_S, delay: float = 2.0, + ) -> bool: + """Make the live gateway's managed routes match the rendered route set. + + The render half wrote the desired routes (one per live deployment× + endpoint) to ``litellm_routes.json``; this is the apply half. List the + gateway's current models, add the missing routes and delete the ones no + longer desired -- through the admin API, with **no** container restart -- + then list again to verify both ids and routing semantics, re-diffing and + retrying failed calls until the table matches or the budget runs out. + + Properties this relies on: + + * **Idempotent.** A redundant apply re-diffs to the same set and does + nothing. + * **Drift-healing.** Routes lost to a gateway/DB restart reappear in the + diff and are re-added; stale routes from a prior run (still in the DB) + are deleted because they're no longer desired. + * **Co-existence.** Only routes infer-stack created (id prefix ``isr-``) + are ever deleted, so a model added by hand through the UI/API is left + alone. + + **Bounded and reported.** Everything -- listing retries while the gateway + starts, every POST, and the final verification -- shares one wall-clock + budget, ``deadline_s``. A retry count alone would not bound it: a listing + can take 10 s and a POST 30 s. Returns ``True`` only when the verified + managed route set equals the desired set; any failure is logged and + returns ``False`` rather than raising, so the caller decides whether an + unverified route set blocks anything. + """ + from .._log import logger + + deadline = self._clock() + max(0.0, deadline_s) + desired = { + r['model_info']['id']: r + for r in self._desired_routes() + if isinstance(r.get('model_info'), dict) and r['model_info'].get('id') + } + desired_semantics = {rid: self._route_semantics(route) + for rid, route in desired.items()} + rounds = 0 + while True: + current = self._list_managed_routes(deadline=deadline, delay=delay) + if current is None: + logger.warning( + 'dynamic routing: route set not reconciled and verified within ' + '{:g}s; leaving it for the next apply', deadline_s, + ) + return False + mismatched = sorted( + rid for rid in desired.keys() & current.keys() + if desired_semantics[rid] != current[rid] + ) + to_add_ids = sorted((desired.keys() - current.keys()) | set(mismatched)) + to_delete = sorted((current.keys() - desired.keys()) | set(mismatched)) + to_add = [desired[rid] for rid in to_add_ids] + if not (to_add or to_delete): + return True # this listing is the verification + if rounds: + logger.info('dynamic routing: route set still differs; retrying') + rounds += 1 + ok = True + # A same-id semantic drift must be removed before it can be re-added; + # model/new is not an update API on every LiteLLM release. + for rid in to_delete: + # ok_if_missing: with a shared gateway, another converge may have + # deleted this route already; "not found in db" means the desired + # end-state (route gone) is reached, so don't treat it as an error. + ok &= self._post_route( + '/model/delete', {'id': rid}, rid, ok_if_missing=True, + deadline=deadline, + ) + for route in to_add: + ok &= self._post_route( + '/model/new', route, route.get('model_name'), deadline=deadline, + ) + logger.info( + 'dynamic routing: +{} route(s), -{} route(s), ~{} replacement(s) ' + '(now {} desired)', + len(to_add), len(to_delete), len(mismatched), len(desired), + ) + if not ok: + # A transient admin-API failure: spend the rest of the budget + # re-diffing rather than giving up with most of it unused. + if deadline - self._clock() <= delay: + return False + self._sleep(delay) + + @staticmethod + def _route_semantics(route: dict[str, Any]) -> dict[str, Any]: + """Observable route fields infer-stack owns and must verify. + + LiteLLM's model-info response contains additional database/runtime fields + and may redact credentials. The public alias, upstream model, and + upstream base URL are the routing semantics infer-stack can both set and + reliably observe. A matching managed id with different values here is + drift and is replaced, not accepted as healthy. + """ + params = route.get('litellm_params') or {} + return { + 'model_name': route.get('model_name'), + 'model': params.get('model'), + 'api_base': params.get('api_base'), + } + + def _list_managed_routes( + self, *, deadline: float, delay: float + ) -> dict[str, dict[str, Any]] | None: + """Observable semantics of infer-stack-managed gateway routes. + + Retries while the gateway is unreachable, until ``deadline`` (a value of + ``self._clock``). Each request's own timeout is capped by the time left. + Returns ``None`` if no listing succeeded in time. + """ + while True: + remaining = deadline - self._clock() + if remaining <= 0: + return None + resp = None + try: + resp = self.http.get( + f'{self._gateway_base()}/v1/model/info', + headers=self._auth_headers(), + timeout=min(10.0, remaining), + ) + except Exception: # noqa: BLE001 - the gateway may still be starting + resp = None + if resp is not None and getattr(resp, 'status_code', 0) == 200: + routes: dict[str, dict[str, Any]] = {} + for m in (resp.json().get('data') or []): + rid = (m.get('model_info') or {}).get('id') + if isinstance(rid, str) and rid.startswith(ROUTE_ID_PREFIX): + routes[rid] = self._route_semantics(m) + return routes + if deadline - self._clock() <= delay: + return None + self._sleep(delay) + + def _post_route( + self, + path: str, + payload: dict[str, Any], + label: Any, + *, + ok_if_missing: bool = False, + deadline: float | None = None, + ) -> bool: + """POST one admin-API call (``/model/new`` or ``/model/delete``). + + Returns whether it reached its desired end state. A failure is logged, + not raised, so the remaining calls still run. ``ok_if_missing`` accepts a + "model not found" response (a delete whose target is already gone). + The request timeout is capped by the time left before ``deadline``. + """ + from .._log import logger + + timeout = 30.0 + if deadline is not None: + remaining = deadline - self._clock() + if remaining <= 0: + logger.warning( + 'dynamic routing: POST {} {} skipped: route deadline passed', + path, label, + ) + return False + timeout = min(timeout, remaining) + try: + resp = self.http.post( + f'{self._gateway_base()}{path}', + headers=self._auth_headers(), + json=payload, + timeout=timeout, + ) + except Exception as ex: # noqa: BLE001 - one bad call must not abort apply + logger.warning('dynamic routing: POST {} {} error: {}', path, label, ex) + return False + if getattr(resp, 'status_code', 0) >= 300: + body = str(getattr(resp, 'text', '')) + if ok_if_missing and 'not found' in body.lower(): + return True + logger.warning( + 'dynamic routing: POST {} {} -> {} {}', + path, label, resp.status_code, body[:200], + ) + return False + return True + + def access(self, endpoints: list[str]) -> dict[str, Any] | None: + """Where a client reaches these endpoints, for the env-file descriptor. + + With the LiteLLM front door, that is one ``base_url`` and the request + model name is the endpoint alias itself. With LiteLLM off there is no + single base URL, but a managed Open WebUI (if on) is still a useful + access point, so report just its URL rather than ``None``. + """ + if not self.litellm: + if self.ui: + return {'ui_url': f'http://127.0.0.1:{self.ui_port}'} + return None + info: dict[str, Any] = { + 'base_url': f'{self._gateway_base()}/v1', + 'api_key_env': API_KEY_ENV, + 'api_key': self.master_key(), + 'request_names': {ep: ep for ep in endpoints}, + } + if self.ui: + info['ui_url'] = f'http://127.0.0.1:{self.ui_port}' + if self.reverse_proxy: + # The unified front door: one origin, UI at / and the API at /v1. + info['proxy_url'] = f'http://127.0.0.1:{self.reverse_proxy_port}' + return info + + def merged_route_registry(self, incoming: dict[str, dict[str, Any]]) -> dict[str, Any]: + """The registry with ``incoming`` rows merged in, in memory (no write).""" + from .._log import logger + + merged, warnings = _merge_route_registry(self._load_route_registry(), incoming) + for w in warnings: + logger.warning(' route registry: {}', w) + return merged diff --git a/infer_stack/leasing/instances.py b/infer_stack/leasing/instances.py new file mode 100644 index 00000000..6a9db195 --- /dev/null +++ b/infer_stack/leasing/instances.py @@ -0,0 +1,289 @@ +"""What a backend is running, one row per unit, and how to read its log. + +An :class:`Instance` is a container (Compose, or the KubeAI gateway) or a pod +(KubeAI). Backends build them from their strict residency +(:meth:`instances`), so ``ps``, ``logs``, ``status`` and the TUI read the +same view the controller decides with. The runtime an instance lives in is +recorded on it, so reading its log needs no knowledge of which backend +produced it: :func:`history_argv` and :func:`follow_argv` are the only place +that knows ``docker logs`` from ``kubectl logs``. +""" + +from __future__ import annotations + +import subprocess +from dataclasses import dataclass +from typing import Callable, Iterable, Iterator, Mapping + +#: Runtimes an instance can live in. +DOCKER = 'docker' +KUBERNETES = 'kubernetes' +#: In-process backends (dry-run, tests): nothing to read a log from. +MEMORY = 'memory' + + +@dataclass(frozen=True) +class Instance: + """One running unit: an engine container or pod, or a front-door service.""" + + #: What a user types: the Compose service, or the pod name. + name: str + #: The runtime's own id: container id, or the pod name. + id: str + #: The deployment it serves; empty for the gateway, UI, database, proxy. + deployment_id: str + state: str + restarts: int = 0 + #: Why it is not running, when the runtime says (``CrashLoopBackOff``...). + reason: str = '' + #: ``healthy`` / ``starting`` / ``unhealthy``, when the runtime has a check. + health: str = '' + gpus: tuple[int, ...] = () + started: str = '' + ports: str = '' + runtime: str = DOCKER + #: Kubernetes only: where the pod lives and which container is the engine. + namespace: str = '' + container: str = '' + + @property + def is_engine(self) -> bool: + return bool(self.deployment_id) + + @property + def status(self) -> str: + """``running``, ``restarting (CrashLoopBackOff, 3 restarts)``, ...""" + bits = [b for b in (self.reason, self.health if self.health != 'healthy' else '') + if b] + if self.restarts: + bits.append(f'{self.restarts} restart{"s" if self.restarts != 1 else ""}') + return f'{self.state} ({", ".join(bits)})' if bits else self.state + + +def from_residency(residency, *, runtime: str = DOCKER, namespace: str = '', + container: str = '') -> list[Instance]: + """Every instance in a :class:`~infer_stack.leasing.residency.Residency`.""" + out = [] + for c in residency.all_containers(): + name = c.service if runtime == DOCKER and c.service else c.container_id + out.append(Instance( + name=name, id=c.container_id, deployment_id=c.deployment_id, + state=c.state, restarts=c.restart_count, reason=c.reason, + health=c.health, gpus=tuple(c.gpus), started=c.started, ports=c.ports, + runtime=runtime, namespace=namespace, container=container, + )) + return sorted(out, key=lambda i: (not i.is_engine, i.name, i.id)) + + +def history_argv(instance: Instance, *, tail: str | int | None = None, + timestamps: bool = False) -> list[str] | None: + """The command that prints ``instance``'s log so far, or ``None``.""" + flags = [] if tail is None else ['--tail', str(tail)] + if timestamps: + flags.append('--timestamps') + if instance.runtime == DOCKER: + return ['docker', 'logs', *flags, instance.id] + if instance.runtime == KUBERNETES: + return ['kubectl', '-n', instance.namespace, 'logs', *flags, instance.id, + *(['-c', instance.container] if instance.container else [])] + return None + + +def follow_argv(instance: Instance, *, timestamps: bool = False) -> list[str] | None: + r"""The command that streams ``instance``'s output from now on, or ``None``. + + Docker: ``docker attach`` with stdin closed and signals not forwarded, not + ``docker logs -f``: the log driver holds a partial line until its newline, + so a progress bar redrawn with ``\r`` shows nothing for minutes (measured). + Ending the attach never touches the container. Kubernetes has no such + driver in the way: ``kubectl logs -f --tail 0``. + """ + if instance.runtime == DOCKER: + return ['docker', 'attach', '--no-stdin', '--sig-proxy=false', instance.id] + if instance.runtime == KUBERNETES: + argv = history_argv(instance, tail=0, timestamps=timestamps) + assert argv is not None + return [*argv, '--follow'] + return None + + +def runtime_env(instance: Instance) -> dict[str, str] | None: + """The environment to run the instance's log commands in (None: inherit).""" + if instance.runtime == DOCKER: + from .compose import docker_environment + + return docker_environment() + return None + + +class UnknownTarget(ValueError): + """A name that matches no instance; the message lists what exists.""" + + +def resolve(instances: list[Instance], names: Iterable[str], + served: Mapping[str, Iterable[str]] | None = None) -> list[Instance]: + """The instances ``names`` refer to, in order, without repeats. + + A name matches an instance's name, its id (a prefix of at least four + characters, as Docker accepts), its deployment id, or an endpoint alias + the deployment serves (``served``: deployment id -> aliases). One alias + or deployment can match several instances (a pod and its restart). + + >>> a = Instance('vllm-qwen', 'c0ffee12', 'grp-1', 'running') + >>> b = Instance('litellm', 'deadbeef', '', 'running') + >>> [i.name for i in resolve([a, b], ['qwen', 'litellm'], {'grp-1': ['qwen']})] + ['vllm-qwen', 'litellm'] + >>> [i.name for i in resolve([a, b], ['c0ff'])] + ['vllm-qwen'] + >>> resolve([a, b], ['nope']) + Traceback (most recent call last): + ... + infer_stack.leasing.instances.UnknownTarget: no instance matches 'nope'; running: litellm, vllm-qwen + """ + served = {gid: set(aliases) for gid, aliases in (served or {}).items()} + picked: list[Instance] = [] + for name in names: + found = [ + i for i in instances + if name in (i.name, i.id, i.deployment_id) + or (len(name) >= 4 and i.id.startswith(name)) + or (i.deployment_id and name in served.get(i.deployment_id, ())) + ] + if not found: + known = ', '.join(sorted({i.name for i in instances})) + raise UnknownTarget(f'no instance matches {name!r}; ' + + (f'running: {known}' if known else 'nothing is running')) + picked.extend(i for i in found if i not in picked) + return picked + + +class LogFollower: + r"""Follow instances' logs: recent history, then live output. + + ``list_instances`` is called every ``poll`` seconds, so an instance that + appears later (a model starting, a recreate) is followed from its first + line, and one that restarted is followed again. ``stdout`` yields + ``name | line``; with ``prefix='auto'`` the name is left off while only + one instance has been followed (it says nothing then, and costs a narrow + pane most of its width). Output written between the history read and the + live stream (a few milliseconds) can be missed. + """ + + poll = 3.0 + + def __init__(self, list_instances: Callable[[], list[Instance]], *, + history: str | int = 200, timestamps: bool = False, + prefix: str = 'always'): + import queue + import threading + + self._list = list_instances + self._history = history + self._timestamps = timestamps + self._prefix = prefix + self._names: set[str] = set() + self._lines: queue.Queue = queue.Queue() + self._stop = threading.Event() + self._seen: set[str] = set() + self._live: dict[str, subprocess.Popen] = {} + threading.Thread(target=self._watch, daemon=True).start() + + def _watch(self) -> None: + import threading + + first = True + while not self._stop.is_set(): + try: + found = self._list() + except Exception: # noqa: BLE001 - runtime unreachable: try again + found = [] + for inst in found: + running = inst.state == 'running' + live = self._live.get(inst.id) + attached = live is not None and live.poll() is None + if inst.id not in self._seen: + # Existing instances: the recent tail. A new one: all of it. + history: str | int | None = self._history if first else 'all' + elif running and not attached: + history = None # restarted: follow again, no repeat + else: + continue + self._seen.add(inst.id) + self._names.add(inst.name) + threading.Thread(target=self._follow, args=(inst, history, running), + daemon=True).start() + first = False + self._stop.wait(self.poll) + + def _follow(self, inst: Instance, history, running: bool) -> None: + from ..log_filter import LogLineSplitter + + prefix = inst.name + env = runtime_env(inst) + if history is not None: + argv = history_argv(inst, tail=history, timestamps=self._timestamps) + old = '' + if argv is not None: + try: + old = subprocess.run( + argv, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + timeout=60, env=env, + ).stdout.decode('utf-8', 'replace') + except Exception: # noqa: BLE001 + old = '' + split = LogLineSplitter(every=0.0) + for line in split.feed(old) + split.flush(): + self._lines.put((prefix, line)) + argv = follow_argv(inst, timestamps=self._timestamps) + if not running or argv is None or self._stop.is_set(): + return + # Engines log to stderr, which docker attach passes out on its own stderr. + proc = subprocess.Popen(argv, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, env=env) + self._live[inst.id] = proc + self._pump(proc, prefix) + + def _pump(self, proc: subprocess.Popen, prefix: str) -> None: + import codecs + import os + + from ..log_filter import LogLineSplitter + + decode = codecs.getincrementaldecoder('utf-8')('replace').decode + split = LogLineSplitter() + assert proc.stdout is not None # Popen(stdout=PIPE) + fd = proc.stdout.fileno() + try: + while True: + chunk = os.read(fd, 65536) + if not chunk: + break + for line in split.feed(decode(chunk)): + self._lines.put((prefix, line)) + finally: + proc.stdout.close() + for line in split.flush(): + self._lines.put((prefix, line)) + + @property + def stdout(self) -> Iterator[str]: + import queue + + while not self._stop.is_set(): + try: + name, line = self._lines.get(timeout=0.5) + except queue.Empty: + continue + if self._prefix == 'auto' and len(self._names) <= 1: + yield line + '\n' + else: + yield f'{name} | {line}\n' + + def terminate(self) -> None: + self._stop.set() + for proc in list(self._live.values()): + try: + proc.terminate() + proc.wait(timeout=2) + except Exception: # noqa: BLE001 + proc.kill() diff --git a/infer_stack/leasing/models.py b/infer_stack/leasing/models.py index beacd53a..d7a98022 100644 --- a/infer_stack/leasing/models.py +++ b/infer_stack/leasing/models.py @@ -73,6 +73,28 @@ class Sharing: # Synthetic endpoint/claim name for a reservation (it serves nothing). RESERVED_ENDPOINT = 'reserved-gpu' +def served_name(deployment: Any) -> str: + """The model name a deployment's engine serves under: the one rule. + + The spec's ``served_model_name``, else its first served alias, else its + id. The engine's ``--served-model-name``, the gateway route's upstream + model, the Compose service name and the KubeAI Model name all derive + from this, so they cannot disagree. + + Example: + >>> from types import SimpleNamespace as NS + >>> served_name(NS(spec={'served_model_name': 'q'}, served={'a': {}}, id='g')) + 'q' + >>> served_name(NS(spec={}, served={'b': {}, 'a': {}}, id='g')) + 'a' + >>> served_name(NS(spec={}, served={}, id='g')) + 'g' + """ + return deployment.spec.get('served_model_name') or ( + sorted(deployment.served)[0] if deployment.served else deployment.id + ) + + def is_reservation(obj: Any) -> bool: """True if a :class:`Deployment` / :class:`EndpointRequest` is a GPU reservation.""" return getattr(obj, 'engine', None) == RESERVED_ENGINE @@ -340,8 +362,8 @@ class Deployment: created_at: float updated_at: float demand: int = 0 - # The committed GPU allocation of a LIVE deployment (admission-mode - # backends). ``None`` when not LIVE, or LIVE but unresolved (a ledger from + # The committed GPU allocation of a LIVE deployment (empty where the + # cluster schedules). ``None`` when not LIVE, or LIVE but unresolved (a ledger from # before allocations existed). Cleared in the same transaction as any # transition out of LIVE. assigned_gpus: list[int] | None = None diff --git a/infer_stack/leasing/naming.py b/infer_stack/leasing/naming.py new file mode 100644 index 00000000..c89515f8 --- /dev/null +++ b/infer_stack/leasing/naming.py @@ -0,0 +1,107 @@ +"""The names infer-stack gives the things it runs, in one place. + +A compose service, a Kubernetes object, and the upstream host a gateway +route points at are all derived from the served model name (or an Ollama +host). The engine side and the gateway side must agree on these exactly, so +both import them from here. +""" + +from __future__ import annotations + +import re + +from .models import Deployment, served_name + +#: The port every engine container serves on inside the Compose network. +VLLM_CONTAINER_PORT = 8000 +OLLAMA_CONTAINER_PORT = 11434 + + +def dns_slug(text: str) -> str: + """A lowercase ``[a-z0-9-]`` label safe as a compose service / DNS / + Kubernetes object name (shared by the compose and kubeai backends).""" + out = re.sub(r'[^a-z0-9]+', '-', str(text).lower()).strip('-') + return out or 'model' + + +_dns_slug = dns_slug # historical internal name + + +def vllm_service_name_for(served: str) -> str: + """Deterministic compose/DNS service name for a vLLM upstream: ``vllm-``. + + Derived purely from the served model name (vLLM's ``--served-model-name``, + the Open WebUI label, the alias the user chose), so it is identical whether + computed from a live :class:`Deployment` or from a catalog endpoint. That + stability is what lets the LiteLLM gateway carry a *static* route table (one + per catalog endpoint) whose upstream hosts match the containers when they + come up — so adding/removing models does not rewrite the gateway's config and + the gateway is never recreated (no "blip"); see :func:`_litellm_model_list`. + """ + return f'vllm-{_dns_slug(served)}' + + +def _unique_vllm_service_name(served: str, deployment_id: str) -> str: + """Per-deployment vLLM service/DNS name: ``vllm--``. + + The static-superset gateway needs a name derivable from the served model + *alone* (so a catalog route can address it without knowing the live + deployment) — but that deliberately drops the deployment id, which + **collapses every** ``--dedicated`` **deployment of one model onto a single + container** (hence one GPU). Dynamic routing manages the gateway's routes + live via the admin API, so the upstream host no longer has to be predictable + from the catalog. That frees us to give each deployment its **own** service, + so N dedicated deployments of one model become N containers on N GPUs. The + suffix is the deployment id's hex tail, keeping the name short and DNS-safe. + """ + return f'{vllm_service_name_for(served)}-{deployment_tail(deployment_id)}' + + +def deployment_tail(deployment_id: str) -> str: + """The short, DNS-safe suffix that makes a per-deployment name unique. + + >>> deployment_tail('grp-0123456789ab') + '01234567' + """ + return _dns_slug(deployment_id.rsplit('-', 1)[-1][:8] or 'x') + + +def vllm_service_name(deployment: Deployment, *, unique: bool = False) -> str: + """Compose service name for a vLLM deployment (see :func:`vllm_service_name_for`). + + Default (``unique=False``, static-superset mode): deterministic from the + served model name only — *no* deployment-id suffix — so it matches the + gateway's pre-rendered route for that endpoint. Trade-off: two + *simultaneously desired* deployments that share a served name collide on this + name; under the static gateway the catalog endpoint is the unit, so that case + (including same-model ``--dedicated``) is unsupported. + + ``unique=True`` (dynamic-routing mode): append the deployment-id tail + (:func:`_unique_vllm_service_name`) so same-model dedicated deployments get + distinct containers/GPUs; the admin-API route table addresses each by name. + + Either way the container carries the ``infer-stack.deployment`` label. + :meth:`ComposeBackend.residency` correlates containers to deployments by that + label, so the choice of suffix does not affect it. (The lenient + :meth:`ComposeBackend.observe` still maps service names through the render + sidecar; it is for reporting, not for decisions that touch a GPU.) + """ + served = served_name(deployment) + if unique: + return _unique_vllm_service_name(served, deployment.id) + return vllm_service_name_for(served) + + +def ollama_service_name_for(host: str) -> str: + """Deterministic service name for an Ollama daemon: ``ollama-``. + + One daemon per host (Ollama coalesces tags onto it), so the host is the + stable key — matching :func:`vllm_service_name_for`'s role for vLLM so the + gateway's static route table addresses it regardless of which tags are live. + """ + return f'ollama-{_dns_slug(host)}' + + +def ollama_service_name(deployment: Deployment) -> str: + host = deployment.spec.get('host') or deployment.id + return ollama_service_name_for(host) diff --git a/infer_stack/leasing/residency.py b/infer_stack/leasing/residency.py index e768d8a3..51ccbabf 100644 --- a/infer_stack/leasing/residency.py +++ b/infer_stack/leasing/residency.py @@ -51,11 +51,28 @@ #: Labels every rendered service carries: its service name, and a behavioural #: fingerprint that changes only when the service's behaviour does. SERVICE_LABEL = 'infer-stack.service' +#: Which engine a service runs (``vllm``, ``ollama``, ``litellm``...). +ENGINE_LABEL = 'infer-stack.engine' FINGERPRINT_LABEL = 'infer-stack.fingerprint' #: Labels Docker Compose puts on every container of a project. COMPOSE_PROJECT_LABEL = 'com.docker.compose.project' COMPOSE_SERVICE_LABEL = 'com.docker.compose.service' +def _published_ports(ports) -> str: + """``14042->4000/tcp`` for each published port of a ``docker inspect``. + + >>> _published_ports({'4000/tcp': [{'HostIp': '0.0.0.0', 'HostPort': '14042'}, + ... {'HostIp': '::', 'HostPort': '14042'}], + ... '8000/tcp': None}) + '14042->4000/tcp' + """ + out = [] + for inner, bindings in sorted((ports or {}).items()): + for host in sorted({b.get('HostPort') for b in bindings or [] if b.get('HostPort')}): + out.append(f'{host}->{inner}') + return ', '.join(out) + + #: Container states that hold, or will reclaim on their own, a warm model: the #: process is up, is being restarted by Docker's restart policy, or is paused #: (a paused process keeps its GPU memory). ``created``, ``exited``, @@ -106,6 +123,14 @@ class Container: #: will try to start the container again. restart_policy: str = '' restart_max: int = 0 + #: Why the instance is not running, when its runtime says: + #: ``CrashLoopBackOff``, ``ImagePullBackOff``, ``OOMKilled`` (Kubernetes). + #: Empty when there is nothing to say, or the runtime does not say. + reason: str = '' + #: When the current run started (the runtime's timestamp), for display. + started: str = '' + #: Published ports, ``host->container/proto`` joined by ``, ``; display only. + ports: str = '' @property def warm(self) -> bool: @@ -255,6 +280,8 @@ def residency_from_inspect(raw: str, *, project: str) -> Residency: or {}).get('MaximumRetryCount')) or 0), exit_code=(None if (item.get('State') or {}).get('ExitCode') is None else int((item.get('State') or {})['ExitCode'])), + started=str((item.get('State') or {}).get('StartedAt') or ''), + ports=_published_ports((item.get('NetworkSettings') or {}).get('Ports')), ips=tuple(sorted( str(n.get('IPAddress')) for n in (((item.get('NetworkSettings') or {}).get('Networks')) or {}).values() @@ -272,3 +299,98 @@ def residency_from_inspect(raw: str, *, project: str) -> Residency: }, tuple(sorted(others, key=lambda c: c.container_id)), ) + + +#: Label a managed Kubernetes pod carries (copied by KubeAI from its Model). +POD_DEPLOYMENT_LABEL = 'infer-stack/deployment' +POD_MANAGED_LABEL = 'infer-stack/managed' + + +def residency_from_pods(raw: str) -> Residency: + """Build a :class:`Residency` from ``kubectl get pods -o json`` output. + + One pod is one instance (a :class:`Container`), keyed by the + ``infer-stack/deployment`` label KubeAI copies from the Model. The state + is mapped to the Docker vocabulary the rest of infer-stack reads: + + * running container -> ``running`` + * ``CrashLoopBackOff`` -> ``restarting`` (still warm: the kubelet retries) + * any other waiting reason, or a pod not started yet -> ``created`` + * terminated -> ``exited``; a pod being deleted -> ``removing`` + + The waiting or last-termination reason is kept in :attr:`Container.reason`. + GPUs are left empty: the cluster, not this host, owns placement. Raises + :class:`ResidencyUnknown` on output that is not a pod list. + + Example: + >>> raw = json.dumps({'items': [{ + ... 'metadata': {'name': 'model-q-1', 'labels': { + ... 'infer-stack/deployment': 'grp-a', 'infer-stack/managed': 'true', + ... 'model': 'q'}}, + ... 'status': {'phase': 'Running', 'containerStatuses': [{ + ... 'state': {'waiting': {'reason': 'CrashLoopBackOff'}}, + ... 'lastState': {'terminated': {'exitCode': 1, 'reason': 'Error'}}, + ... 'restartCount': 3}]}}]}) + >>> c = residency_from_pods(raw).containers('grp-a')[0] + >>> (c.state, c.reason, c.restart_count, c.exit_code, c.warm) + ('restarting', 'CrashLoopBackOff', 3, 1, True) + """ + try: + data = json.loads(raw or '{}') + except json.JSONDecodeError as ex: + raise ResidencyUnknown(f'kubectl get pods output is not JSON: {ex}') from ex + items = data.get('items') if isinstance(data, dict) else None + if not isinstance(items, list): + raise ResidencyUnknown('kubectl get pods output has no items list') + grouped: dict[str, list[Container]] = {} + others: list[Container] = [] + for pod in items: + meta = pod.get('metadata') or {} + labels = meta.get('labels') or {} + status = pod.get('status') or {} + statuses = status.get('containerStatuses') or [] + first = statuses[0] if statuses else {} + current = first.get('state') or {} + last = (first.get('lastState') or {}).get('terminated') or {} + waiting = (current.get('waiting') or {}).get('reason') or '' + if meta.get('deletionTimestamp'): + state = 'removing' + elif 'running' in current: + state = 'running' + elif waiting == 'CrashLoopBackOff': + state = 'restarting' + elif 'terminated' in current: + state = 'exited' + else: + state = 'created' + ended = current.get('terminated') or last + if not waiting and not statuses: + # Not started at all: the pod's own condition says why, e.g. the + # scheduler found no node with the resources ("Unschedulable"). + waiting = next((str(c.get('reason') or '') for c in status.get('conditions') or [] + if c.get('status') == 'False' and c.get('reason')), '') + ready = any(c.get('type') == 'Ready' and c.get('status') == 'True' + for c in status.get('conditions') or []) + container = Container( + container_id=str(meta.get('name') or ''), + deployment_id=str(labels.get(POD_DEPLOYMENT_LABEL) or ''), + state=state, + service=str(labels.get('model') or ''), + labelled=labels.get(POD_MANAGED_LABEL) == 'true', + health='healthy' if ready else ('starting' if state == 'running' else ''), + restart_count=int(first.get('restartCount') or 0), + exit_code=(None if ended.get('exitCode') is None else int(ended['exitCode'])), + # A Deployment's pods are always restarted by the kubelet. + restart_policy='always', + reason=waiting or str(ended.get('reason') or ''), + started=str((current.get('running') or {}).get('startedAt') + or status.get('startTime') or ''), + ) + if container.deployment_id: + grouped.setdefault(container.deployment_id, []).append(container) + else: + others.append(container) + return Residency( + by_deployment={gid: tuple(found) for gid, found in grouped.items()}, + others=tuple(others), + ) diff --git a/infer_stack/leasing/suggest.py b/infer_stack/leasing/suggest.py index 83edcacd..428cd978 100644 --- a/infer_stack/leasing/suggest.py +++ b/infer_stack/leasing/suggest.py @@ -379,6 +379,28 @@ def derive_runtime( return runtime +#: A simulator endpoint for a host without a GPU (``catalog suggest +#: --simulator``): llm-d-inference-sim answers like vLLM with random text, so +#: the whole workflow (acquire, the gateway, a request, release) runs here. +#: Never a result. The same entry as dev/e2e_tests/catalog-mock.yaml's. +SIMULATOR_FRAGMENT: dict[str, Any] = { + 'models': {'smol135': {'source': 'hf://HuggingFaceTB/SmolLM2-135M-Instruct'}}, + 'endpoints': {'mock-smol': { + 'engine': 'vllm', + 'model': 'smol135', + 'runtime': { + 'image': 'ghcr.io/llm-d/llm-d-inference-sim:v0.9.0', + 'max_model_len': 2048, + 'max_num_seqs': 8, + 'simulator': {'kind': 'llm-d-sim', 'mode': 'random', 'seed': 20260731, + 'time_to_first_token': '120ms', + 'inter_token_latency': '8ms', 'startup_duration': '10s'}, + }, + 'reclaim': {'policy': 'stop'}, + }}, +} + + # --------------------------------------------------------------------------- # the top-level pure function # --------------------------------------------------------------------------- diff --git a/infer_stack/tui.py b/infer_stack/tui.py index 51f45f92..987492a1 100644 --- a/infer_stack/tui.py +++ b/infer_stack/tui.py @@ -12,8 +12,9 @@ it in Open WebUI. * **Leases** + **Deployments** (center) — the live ledger (desired *state* vs what's actually *running*, and which GPUs), with Release / Evict / Clean-up. -* **docker** — a collapsible pane with **Logs** and **Containers** (the - ``docker ps`` view: status/uptime, created, id, ports) tabs (collapsed by +* **runtime** — a collapsible pane with **Logs**, **Instances** (what + ``infer-stack ps`` shows: containers or pods, status, what each serves, + ports) and **Control** (Apply / Down) tabs, on either backend (collapsed by default; ``c`` toggles it). * **system** — live ``nvidia-smi`` GPUs + host CPU/mem (collapsed by default). * **api** — send a prompt to a *ready* model through the LiteLLM gateway @@ -31,13 +32,16 @@ import subprocess import time from pathlib import Path -from typing import Any, Callable, Iterable +from typing import Any, Callable from textual import events, work from textual.app import App, ComposeResult from textual.binding import Binding from textual.coordinate import Coordinate from textual.containers import Horizontal, Vertical, VerticalScroll +from textual.css.query import NoMatches +from textual.lazy import Lazy +from textual.widget import Widget from textual.screen import ModalScreen from textual.theme import Theme from textual.widgets import ( @@ -66,7 +70,7 @@ from rich.markup import escape as escape_markup from . import cli_equivalent as cli -from .log_filter import LogLineSplitter, compact_litellm_tracebacks +from .log_filter import compact_litellm_tracebacks ALL_SERVICES = '' # the Select value meaning "every service" # The Select value meaning "every service EXCEPT the gateway". This is the @@ -75,12 +79,8 @@ # pane before anyone can read it. The gateway's own logs are one selection away # when they are what you want. ENGINE_SERVICES = '\x00engines' -# Substring rather than equality: compose names the gateway service `litellm` -# today, but deployment naming has carried suffixes before and a missed match -# silently restores the noisy view. -GATEWAY_SERVICE_HINT = 'litellm' # What `_resolve_log_target` returns when a view has nothing to follow. Not -# None: None means "every service" to `docker compose logs`. +# None: None means "every instance". NO_LOG_TARGET = object() @@ -96,6 +96,60 @@ #: Title of the tab that shows what the TUI itself did, and its errors. APP_LOG_TAB_TITLE = 'TUI log' +#: The docker log pane shows at most this many lines (RichLog ``max_lines``). +LOG_PANE_LINES = 2000 +#: How often streamed log lines are drawn: in one batch, not one UI message per +#: line (which capped a loading engine's log at ~1800 lines/s). +LOG_DRAIN_S = 0.1 +#: At most this many lines per drain: RichLog costs ~0.1 ms a line, and 2000 in +#: one tick stalled the UI for 218 ms. The rest waits for the next tick. +LOG_DRAIN_LINES = 200 +#: A UI-thread stall at least this long is reported, with what was running. +STALL_REPORT_S = 0.5 + +#: What a running background action is doing, shown in the activity line, by +#: worker method. A worker not listed here (the refresh, log streams) is +#: routine and shows nothing. +ACTIVITY_VERBS = { + '_do_acquire': 'acquiring', + '_do_release': 'releasing', + '_do_release_all': 'releasing all leases', + '_do_evict': 'evicting', + '_do_evict_all': 'evicting idle deployments', + '_do_cleanup': 'cleaning up the ledger', + '_do_apply': 'applying', + '_do_compose': 'docker compose', + '_save_endpoint': 'saving endpoint', + '_do_suggest': 'inspecting GPUs for a suggestion', + '_prepare_endpoint_editor': 'inspecting GPUs', + '_do_api_send': 'waiting for the model', + '_do_api_test_all': 'testing every model', + '_do_api_list': 'listing models', +} +SPINNER = '⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏' + + +def activity_label(description: str) -> str | None: + """``'acquiring qwen'`` from a worker description like ``_do_acquire('qwen')``. + + Example: + >>> # xdoctest: +REQUIRES(module:textual) + >>> activity_label("_do_acquire('qwen')") + 'acquiring qwen' + >>> activity_label('_do_apply()') + 'applying' + >>> activity_label('_refresh_bg()') is None + True + """ + import re + + method, _, rest = str(description).partition('(') + verb = ACTIVITY_VERBS.get(method.strip()) + if verb is None: + return None + first = re.match(r"\s*'([^']*)'", rest) + return f'{verb} {first.group(1)}' if first else verb + SELECT_BLANK = next( ( candidate @@ -106,18 +160,21 @@ ) -def _select_is_blank(value: object) -> bool: - return value is SELECT_BLANK +def _why(ex: BaseException) -> str: + """A failure for the status line: a refused runtime command the way the + CLI says it (its words, a hint), anything else as it reads.""" + import subprocess + if isinstance(ex, subprocess.CalledProcessError): + from .cli import runtime_failure -def is_gateway_service(name: str) -> bool: - """Is this compose service the LiteLLM gateway rather than an engine?""" - return GATEWAY_SERVICE_HINT in str(name).lower() + return ' — '.join(line.strip() for line in runtime_failure(ex).splitlines()) + return str(ex) -def engine_services(names) -> list[str]: - """Every service that is not the gateway, in the given order.""" - return [n for n in names if not is_gateway_service(n)] +def _select_is_blank(value: object) -> bool: + return value is SELECT_BLANK + SELECT_MARK = '✓' # multi-select marker in the leases/deployments tables DEFAULT_THEME = 'textual-dark' @@ -200,151 +257,6 @@ def on_key(self, event: events.Key) -> None: ) -class _DockerLogProc: - r"""Follow the project's containers: recent history, then live output. - - Not ``docker compose logs -f``, and not ``docker logs -f`` either: Docker's - log driver stores output line by line and holds a partial line until its - newline, so a download bar redrawn with ``\r`` showed nothing for many - minutes and then arrived all at once (measured with both). ``docker - attach`` reads the container's output as it is written, so each container - gets its recent ``docker logs --tail`` (complete lines), then an attach - with stdin closed and signals not forwarded -- ending it never touches the - container. - - Containers are re-listed every ``poll`` seconds: one created later (a model - starting, a recreate) is followed from its first line, and one that - restarted is attached again. ``stdout`` yields ``service | line`` like - Compose. Output written between the history read and the attach (a few - milliseconds) can be missed. - """ - - poll = 3.0 - - def __init__(self, project: str, compose_file: str, service=None): - import queue - import threading - - del compose_file # the project label is enough to find them - self.project = project - if isinstance(service, (list, tuple)): - self.services = {str(s) for s in service} - else: - self.services = {str(service)} if service else None - self._lines: queue.Queue = queue.Queue() - self._stop = threading.Event() - self._seen: set[str] = set() - self._live: dict[str, subprocess.Popen] = {} - threading.Thread(target=self._watch, daemon=True).start() - - def _containers(self) -> list[tuple[str, str, bool]]: - """(id, service, running) for this project's containers in view.""" - from .leasing.compose import docker_environment - - out = subprocess.run( - ['docker', 'ps', '-a', '--filter', - f'label=com.docker.compose.project={self.project}', - '--format', '{{.ID}} {{.State}} {{.Label "com.docker.compose.service"}}'], - capture_output=True, text=True, timeout=30, env=docker_environment(), - ).stdout - found = [] - for row in out.splitlines(): - cid, _, rest = row.strip().partition(' ') - state, _, svc = rest.partition(' ') - if cid and (self.services is None or svc in self.services): - found.append((cid, svc, state == 'running')) - return sorted(found, key=lambda row: row[1]) - - def _watch(self) -> None: - import threading - - first = True - while not self._stop.is_set(): - try: - found = self._containers() - except Exception: # noqa: BLE001 - docker unreachable: try again - found = [] - for cid, svc, running in found: - attached = cid in self._live and self._live[cid].poll() is None - if cid not in self._seen: - # Existing containers: the recent tail. A new one: all of it. - history = '200' if first else 'all' - elif running and not attached: - history = None # restarted: attach again, no repeat - else: - continue - self._seen.add(cid) - threading.Thread(target=self._follow, args=(cid, svc, history, running), - daemon=True).start() - first = False - self._stop.wait(self.poll) - - def _follow(self, cid: str, service: str, history: str | None, running: bool) -> None: - from .leasing.compose import docker_environment - - prefix = f'{service} | ' - if history is not None: - try: - old = subprocess.run( - ['docker', 'logs', '--tail', history, cid], - stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=60, - env=docker_environment(), - ).stdout.decode('utf-8', 'replace') - except Exception: # noqa: BLE001 - old = '' - split = LogLineSplitter(every=0.0) - for line in split.feed(old) + split.flush(): - self._lines.put(prefix + line) - if not running or self._stop.is_set(): - return - proc = subprocess.Popen( - ['docker', 'attach', '--no-stdin', '--sig-proxy=false', cid], - # Engines log to stderr, which attach passes out on its own stderr. - stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=docker_environment(), - ) - self._live[cid] = proc - self._pump(proc, prefix) - - def _pump(self, proc: subprocess.Popen, prefix: str) -> None: - import codecs - import os - - decode = codecs.getincrementaldecoder('utf-8')('replace').decode - split = LogLineSplitter() - assert proc.stdout is not None # Popen(stdout=PIPE) - fd = proc.stdout.fileno() - try: - while True: - chunk = os.read(fd, 65536) - if not chunk: - break - for line in split.feed(decode(chunk)): - self._lines.put(prefix + line) - finally: - proc.stdout.close() - for line in split.flush(): - self._lines.put(prefix + line) - - @property - def stdout(self) -> Iterable[str]: - import queue - - while not self._stop.is_set(): - try: - yield self._lines.get(timeout=0.5) + '\n' - except queue.Empty: - continue - - def terminate(self) -> None: - self._stop.set() - for proc in list(self._live.values()): - try: - proc.terminate() - proc.wait(timeout=2) - except Exception: # noqa: BLE001 - proc.kill() - - class _Divider(Static): """A draggable splitter bar. @@ -782,6 +694,30 @@ def on_button_pressed(self, event: Button.Pressed) -> None: self.dismiss(event.button.id == 'ok') +class _Section(Widget): + """A tab's content, built from ``compose_fn`` when it mounts. + + Wrapped in :class:`textual.lazy.Lazy` for tabs hidden at launch, so the + first frame does not wait for widgets nobody can see yet (CSS and layout + of the whole tree was most of the launch time). The app is told when the + content exists, to fill anything it set before then. + """ + + DEFAULT_CSS = '_Section { height: 1fr; }' + + def __init__(self, compose_fn: Callable[[], ComposeResult], *, id: str): + super().__init__(id=id) + self._compose_fn = compose_fn + + def compose(self) -> ComposeResult: + yield from self._compose_fn() + + def on_mount(self) -> None: + on_section = getattr(self.app, '_on_section_mounted', None) + if on_section is not None: + on_section(self.id) + + class InferStackTUI(App): """Monitor + control the leasing stack across panes.""" @@ -807,6 +743,9 @@ class InferStackTUI(App): /* one-line, per-pane descriptions (replaces the old global intro) */ .desc { height: auto; color: $text-muted; padding: 0 1; } + /* A short terminal (see on_resize): rows go to tables and logs. */ + .compact .desc { display: none; } + .compact #logsvc { margin: 0; } #catalog-help { height: auto; color: $text-muted; padding: 0 1; } #endpoint-actions, #lease-actions, #deployment-actions, #model-actions, @@ -859,7 +798,11 @@ class InferStackTUI(App): #leases-pane { height: 14; min-height: 6; } /* height set via _apply_sizes */ #deployments-pane { height: 1fr; min-height: 6; } #leases, #deployments { height: 1fr; min-height: 3; } - #docker-tabs { height: 16; min-height: 8; } + /* tabbed pairs (_pair): the visible pane takes all the room */ + .pair, .pair ContentSwitcher, .pair TabPane { height: 1fr; } + .pair TabPane { padding: 0; } + .pair #models, .pair #leases-pane { height: 1fr; } + #docker-tabs { height: 16; min-height: 8; } /* capped by _log_height */ #logsvc { margin: 0 0 1 0; } #logs, #ps { height: 1fr; background: $surface; } #gpus, #api-out { height: 8; background: $surface; } @@ -875,6 +818,7 @@ class InferStackTUI(App): #api-extra Button { margin: 0 1 0 0; min-width: 12; } #status { dock: bottom; height: 1; padding: 0 2; color: $text-muted; } + #activity { dock: bottom; height: 1; padding: 0 2; color: $warning; display: none; } #settings, #ui-settings { padding: 1 2; } #settings Label, #ui-settings Label { margin: 1 0 0 0; color: $text-muted; } @@ -886,11 +830,20 @@ class InferStackTUI(App): #compose-actions Button { margin: 0 1 0 0; } """ + #: Endpoints|models and leases|deployments as tabs (True), or as two panes + #: split by a draggable divider (False). Each is one switch to flip back. + TABBED_CATALOG = True + TABBED_TABLES = True + BINDINGS = [ # Truly global controls stay in the footer. - ('r', 'refresh', 'Refresh'), + ('r', 'refresh_now', 'Refresh'), # Global on purpose: an error can happen while any tab is in front. ('l', 'show_app_log', 'TUI log'), + # Forgets finished history in both tables, so it belongs to neither pane. + # Not "Clean up": `infer-stack clean` releases and tears everything + # down, and this only forgets finished rows (`gc --forget`). + ('x', 'cleanup', 'Clear finished'), ('tab', 'focus_next', 'Next pane'), ('q', 'quit', 'Quit'), # Pane-scoped actions: keys still work, but they live as buttons under @@ -899,7 +852,6 @@ class InferStackTUI(App): Binding('d', 'release', 'Release', show=False), Binding('e', 'evict', 'Evict', show=False), Binding('a', 'release_all', 'Release all', show=False), - Binding('x', 'cleanup', 'Clean up', show=False), # Multi-select: space toggles the cursor row in the focused leases/ # deployments table; release/evict then act on every checked row. Binding('space', 'toggle_select', 'Select row', show=False), @@ -908,7 +860,7 @@ class InferStackTUI(App): Binding('n', 'add_endpoint', 'Add endpoint', show=False), Binding('o', 'open', 'Open in browser', show=False), Binding('y', 'copy_status', 'Copy status', show=False), - Binding('c', 'toggle_docker', 'Toggle docker', show=False), + Binding('c', 'toggle_docker', 'Toggle runtime', show=False), Binding('left_square_bracket', 'sidebar_narrower', 'sidebar -', show=False), Binding('right_square_bracket', 'sidebar_wider', 'sidebar +', show=False), Binding('minus', 'logs_shorter', 'logs -', show=False), @@ -925,8 +877,13 @@ def __init__( proc_factory: Callable[[str | None], Any] | None = None, catalog_path: str | Path | None = None, http: Any = None, + exit_after_paint: bool = False, ) -> None: super().__init__() + #: Quit once the first frame is drawn (``tui --exit_after_paint``). + self._exit_after_paint = exit_after_paint + #: Seconds from process start to the first drawn frame, once known. + self.first_frame_s: float | None = None self.controller = controller self.catalog = catalog self.interval = interval @@ -955,6 +912,7 @@ def __init__( self._ps_rows_cache: list[tuple] = [] self._gpus_rows_cache: list[tuple] = [] self.catalog_path = Path(catalog_path).expanduser() if catalog_path else None + self._catalog_seen = self._catalog_stamp() # the catalog given is this one self._http = http self._proc_factory = proc_factory or self._default_proc_factory() self._endpoint_names: list[str] = [] @@ -982,6 +940,23 @@ def __init__( # visible; only the current generation is allowed to append. self._log_generation = 0 self._log_lines: list[str] = [] # mirror of the docker log pane, for tests + # Lines streamed by the log worker, drawn by the UI every LOG_DRAIN_S. + # (generation, line): a line from a replaced stream is dropped. + import collections + self._log_pending: collections.deque = collections.deque() + # Lines taken from _log_pending, not yet drawn; only the newest + # LOG_PANE_LINES are kept, since the pane shows no more. + self._log_backlog: collections.deque = collections.deque(maxlen=LOG_PANE_LINES) + self._log_dropped = 0 + # Parsed compose service names, keyed by the file's (mtime, size). + self._last_instances: list = [] + # Running background actions: worker -> (label, started). Drawn by + # _draw_activity, with the newest backend progress message. + self._activity: dict[Any, tuple[str, float]] = {} + self._activity_note = '' + # The UI loop's heartbeat, read by the stall watchdog thread. + self._beat = time.monotonic() + self._watchdog_stop = None self._app_log_lines: list[str] = [] # mirror of the TUI log pane, for tests self._app_log_errors = 0 self._api_lines: list[str] = [] # mirror of the API output, for tests @@ -1013,15 +988,17 @@ def compose(self) -> ComposeResult: with TabbedContent(id='top'): with TabPane('Dashboard', id='tab-dashboard'): yield from self._compose_dashboard() + # Hidden at launch: built right after the first frame, not before. with TabPane('API', id='tab-api'): - yield from self._compose_api() + yield Lazy(_Section(self._compose_api, id='section-api')) with TabPane('UI', id='tab-ui'): - yield from self._compose_ui_settings() + yield Lazy(_Section(self._compose_ui_settings, id='section-ui')) with TabPane('Settings', id='tab-settings'): - yield from self._compose_settings() + yield Lazy(_Section(self._compose_settings, id='section-settings')) with TabPane(APP_LOG_TAB_TITLE, id='tab-applog'): yield from self._compose_app_log() yield Static('', id='status') + yield Static('', id='activity') yield Footer() def _compose_app_log(self) -> ComposeResult: @@ -1038,93 +1015,105 @@ def _compose_app_log(self) -> ComposeResult: yield RichLog(id='applog', highlight=False, markup=True, max_lines=4000, wrap=True) + def _pair(self, tabbed: bool, tabs_id: str, split_id: str, drag, + first: tuple[str, Any], second: tuple[str, Any]) -> ComposeResult: + """Two panes, as tabs or stacked around a draggable divider. + + ``first``/``second`` are ``(tab label, compose function)``. Only the + container differs between the layouts: the panes, and every id the + rest of the app queries, are the same either way. + """ + if tabbed: + with TabbedContent(id=tabs_id, classes='pair'): + for label, body in (first, second): + with TabPane(label, id=f'pane-{label.lower()}'): + yield from body() + else: + yield from first[1]() + yield _Divider('y', drag, id=split_id) + yield from second[1]() + + def _compose_endpoints(self) -> ComposeResult: + yield Static('Endpoints: acquirable configurations', classes='desc') + yield Static('', id='catalog-help') + yield _EndpointTable(id='endpoints', cursor_type='row', + zebra_stripes=True) + with Horizontal(id='endpoint-actions'): + yield Button('Acquire', id='btn-acquire', variant='primary', + action='app.acquire') + yield Button('Add', id='btn-add-endpoint') + yield Button('Edit', id='btn-edit-endpoint') + yield Button('Remove', id='btn-remove-endpoint') + with Horizontal(id='suggest-actions'): + yield Button('✨ Suggest from my GPUs', id='btn-suggest') + + def _compose_models(self) -> ComposeResult: + yield Static('Models: servable weights', classes='desc') + yield DataTable(id='models', cursor_type='row', zebra_stripes=True) + with Horizontal(id='model-actions'): + yield Button('Add', id='btn-add-model') + yield Button('Remove', id='btn-remove-model') + + def _compose_leases(self) -> ComposeResult: + with Vertical(id='leases-pane'): + yield Static('Reservations that map to a deployment. ' + '(space or ctrl/shift-click to multiselect).', + classes='desc') + yield DataTable(id='leases', cursor_type='row', zebra_stripes=True) + with Horizontal(id='lease-actions'): + yield Button('Release', id='btn-release') + yield Button('Release all', id='btn-release-all') + + def _compose_deployments(self) -> ComposeResult: + with Vertical(id='deployments-pane'): + yield Static('Models running.', classes='desc') + yield DataTable(id='deployments', cursor_type='row', + zebra_stripes=True) + with Horizontal(id='deployment-actions'): + yield Button('Evict', id='btn-evict') + yield Button('Evict all idle', id='btn-evict-all') + def _compose_dashboard(self) -> ComposeResult: with Horizontal(id='body'): with Vertical(id='sidebar'): - yield Static( - 'Endpoints — runnable model + engine configs. Acquire one to ' - 'serve it, or Suggest a set sized to your GPUs.', classes='desc', - ) - yield Static('', id='catalog-help') - yield _EndpointTable(id='endpoints', cursor_type='row', - zebra_stripes=True) - with Horizontal(id='endpoint-actions'): - yield Button('Acquire', id='btn-acquire', variant='primary', - action='app.acquire') - yield Button('Add', id='btn-add-endpoint') - yield Button('Edit', id='btn-edit-endpoint') - yield Button('Remove', id='btn-remove-endpoint') - with Horizontal(id='suggest-actions'): - yield Button('✨ Suggest from my GPUs', id='btn-suggest') - yield _Divider('y', self._drag_models, id='csplit') - yield Static( - 'Models — weights an endpoint can serve. Add models here, ' - 'then point an endpoint at one.', classes='desc', - ) - yield DataTable(id='models', cursor_type='row', - zebra_stripes=True) - with Horizontal(id='model-actions'): - yield Button('Add', id='btn-add-model') - yield Button('Remove', id='btn-remove-model') + yield from self._pair( + self.TABBED_CATALOG, 'catalog-tabs', 'csplit', self._drag_models, + ('Endpoints', self._compose_endpoints), + ('Models', self._compose_models)) yield _Divider('x', self._drag_sidebar, id='vsplit') with Vertical(id='main'): with Vertical(id='tables'): - with Vertical(id='leases-pane'): - yield Static( - 'Reservations you hold. Each maps to one deployment ' - 'below (see the deployment column); many leases can ' - 'share one. Release acts on the cursor row, or on ' - 'every row you check (space, or ctrl/shift-click).', - classes='desc', - ) - yield DataTable(id='leases', cursor_type='row', - zebra_stripes=True) - with Horizontal(id='lease-actions'): - yield Button('Release', id='btn-release') - yield Button('Release all', id='btn-release-all') - yield Button('Clean up', id='btn-cleanup') - yield _Divider('y', self._drag_tables, id='tsplit') - with Vertical(id='deployments-pane'): - yield Static( - 'Running model deployments and the GPUs they hold. ' - "The 'leases' column is how many leases hold each. " - 'Evict an idle one to free its GPU (cursor row, or ' - 'rows checked with space / ctrl/shift-click); Evict ' - 'all idle clears every kept-warm one; Clean up forgets ' - 'stopped ones.', classes='desc', - ) - yield DataTable(id='deployments', cursor_type='row', - zebra_stripes=True) - with Horizontal(id='deployment-actions'): - yield Button('Evict', id='btn-evict') - yield Button('Evict all idle', id='btn-evict-all') - yield Button('Clean up', id='btn-cleanup-deployments') + yield from self._pair( + self.TABBED_TABLES, 'table-tabs', 'tsplit', self._drag_tables, + ('Leases', self._compose_leases), + ('Deployments', self._compose_deployments)) yield _Divider('y', self._drag_logs, id='hsplit') - with Collapsible(title='docker', collapsed=True, id='docker'): + with Collapsible(title='runtime', collapsed=True, id='docker'): with TabbedContent(id='docker-tabs'): with TabPane('Logs', id='tab-logs'): yield Select( - [('(engines — no litellm)', ENGINE_SERVICES), - ('(all services)', ALL_SERVICES)], + [('(engines)', ENGINE_SERVICES), + ('(everything)', ALL_SERVICES)], value=ENGINE_SERVICES, allow_blank=False, id='logsvc', ) yield RichLog(id='logs', highlight=False, - markup=False, max_lines=2000, + markup=False, max_lines=LOG_PANE_LINES, wrap=False) - with TabPane('Containers', id='tab-containers'): + with TabPane('Instances', id='tab-containers'): yield DataTable(id='ps', cursor_type='row', zebra_stripes=True) with TabPane('Control', id='tab-control'): yield Static( - 'Bring the rendered compose project up or down. ' + 'Apply brings up what the ledger says should run. ' + 'Down stops everything and releases no lease. ' 'Output appears in the Logs tab.', classes='hint', ) yield Static('', id='compose-path') with Horizontal(id='compose-actions'): - yield Button('Compose up', id='btn-compose-up', + yield Button('Apply', id='btn-compose-up', variant='primary') - yield Button('Compose down', + yield Button('Down', id='btn-compose-down') with Collapsible(title='system', collapsed=True, id='system'): yield Static( @@ -1172,8 +1161,8 @@ def _compose_ui_settings(self) -> ComposeResult: yield Input(value=f'{self.ledger_interval:g}', id='set-ledger-interval') yield Label( - 'docker observe interval (seconds) — the "running" / GPU-placement ' - 'columns; higher = fewer `docker compose ps` calls' + 'runtime observe interval (seconds) — the "running" / GPU columns ' + 'and the runtime pane; higher = fewer runtime queries' ) yield Input(value=f'{self.observe_interval:g}', id='set-observe-interval') @@ -1249,7 +1238,7 @@ def on_mount(self) -> None: 'leases', 'held by' ) self.query_one('#ps', DataTable).add_columns( - 'service', 'status (uptime)', 'created', 'container id', 'ports' + 'name', 'status', 'serves', 'started', 'ports' ) self.query_one('#gpus', DataTable).add_columns( 'gpu', 'name', 'util%', 'mem (used/total)', 'temp' @@ -1268,9 +1257,9 @@ def on_mount(self) -> None: '…', '(loading…)', '-', '-', '-' ) self._gpus_rows_cache = [('__loading__',) * 5] - compose_file = getattr(self.controller.backend, 'compose_file', None) + rendered = getattr(self.controller.backend, 'rendered_file', None) self.query_one('#compose-path', Static).update( - f'compose file: {compose_file or "(not rendered yet)"}' + f'rendered: {rendered or "(this backend renders no file)"}' ) # Capture docker's own chatter (up/down progress on stderr) into the # logs pane instead of letting it bleed onto the full-screen terminal. @@ -1289,33 +1278,128 @@ def on_mount(self) -> None: self._refresh_timer = self.set_interval( self.ledger_interval, self.action_refresh ) + self.set_interval(LOG_DRAIN_S, self._drain_logs) + # Held directly: query_one searches the active screen, which is a + # dialog's while one is open. + self._activity_widget = self.query_one('#activity', Static) + self.set_interval(0.1, self._draw_activity) + self._start_stall_watchdog() self.query_one('#endpoints', DataTable).focus() + self.call_after_refresh(self._first_frame_drawn) + + def _first_frame_drawn(self) -> None: + self.first_frame_s = _seconds_since_process_start() + if self._exit_after_paint: + self.exit() def on_unmount(self) -> None: self._terminate_logs() + if self._watchdog_stop is not None: + self._watchdog_stop.set() + # Quit must not wait for the background refresh's `docker compose ps`: + # Python joins worker threads at exit. Read-only queries only. + from .leasing.compose import cancel_running_queries + + cancel_running_queries() + + # -- responsiveness ------------------------------------------------------ + + def _start_stall_watchdog(self) -> None: + """Report any UI-thread stall, with the stack that caused it. + + A 0.1 s timer on the UI loop moves a heartbeat; a watchdog thread notices + when it stops, samples the UI thread's stack while it is stuck, and + reports the stall once it ends: its length in the TUI log, the stack in + the error log file. "The TUI froze" then comes with what it was doing. + """ + import sys + import threading + + ui_thread = threading.get_ident() + stop = threading.Event() + self._watchdog_stop = stop + + def beat() -> None: + self._beat = time.monotonic() + + self.set_interval(0.1, beat) + + def watch() -> None: + import traceback + + stalled_since = None + stack: list[str] = [] + while not stop.wait(0.1): + behind = time.monotonic() - self._beat + if behind >= STALL_REPORT_S: + if stalled_since is None: + stalled_since = self._beat + # The first sample, half a second in, is inside whatever + # blocks; later ones can catch the loop catching up. + frame = sys._current_frames().get(ui_thread) + if frame is not None: + stack = traceback.format_stack(frame) + elif stalled_since is not None: + length = self._beat - stalled_since + try: + self.call_from_thread(self._report_stall, length, stack) + except Exception: # noqa: BLE001 - app shutting down + return + stalled_since, stack = None, [] + + threading.Thread(target=watch, name='tui-stall-watchdog', daemon=True).start() + + def _report_stall(self, length: float, stack: list[str]) -> None: + # The innermost frames in infer_stack say what to fix; the file keeps all. + ours = [f for f in stack if 'infer_stack' in f] or stack + where = ours[-1].strip().splitlines()[0] if ours else '(no stack)' + path = self.error_log_path() + try: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open('a', encoding='utf-8') as handle: + handle.write(f'\n=== {time.strftime("%Y-%m-%d %H:%M:%S")} ' + f'UI stalled {length:.1f}s\n') + handle.writelines(stack) + except OSError: + pass + self.app_log(f'UI stalled {length:.1f}s in {where} (stack: {path})', level='warn') # -- theming for docker output bleed ---------------------------------- def _install_quiet_docker(self) -> None: - """Route ``docker compose`` output to the logs pane, not the terminal.""" - backend = self.controller.backend - if not hasattr(backend, 'run'): - return - + """Route ``docker`` output to the logs pane, not the terminal. + + Only where the backend runs Docker with the default runner: the kubeai + backend runs ``kubectl`` through the same seam, and the Docker + runner's allowlisted environment has no ``KUBECONFIG`` (every kubectl + call from the TUI used to fail); an injected runner is the caller's + and is left alone. The kubeai backend's gateway is a Compose project + of its own, so it is wrapped too. + """ from .leasing.compose import _default_docker_run - def quiet_run(args: list[str], **kwargs) -> str: - # Same bounded, explicit-environment runner as the CLI; only stderr - # is redirected to the logs pane, unless the caller takes it. - noisy = not any(a == 'ps' for a in args) - sink = ( - (lambda line: self.call_from_thread(self._append_log, line)) - if noisy else (lambda line: None) - ) - kwargs.setdefault('stderr_lines', sink) - return _default_docker_run(args, **kwargs) + def quiet(target) -> None: + original = getattr(target, 'run', None) + if original is not _default_docker_run: + return + + def quiet_run(args: list[str], **kwargs) -> str: + # Same bounded, explicit-environment runner as the CLI; only + # stderr is redirected to the logs pane, unless the caller + # takes it. + noisy = not any(a == 'ps' for a in args) + sink = ( + (lambda line: self.call_from_thread(self._append_log, line)) + if noisy else (lambda line: None) + ) + kwargs.setdefault('stderr_lines', sink) + return _default_docker_run(args, **kwargs) + + target.run = quiet_run - backend.run = quiet_run + backend = self.controller.backend + quiet(backend) + quiet(getattr(backend, 'gateway', None)) if hasattr(backend, 'progress'): backend.progress = self._backend_progress @@ -1326,16 +1410,64 @@ def _backend_progress(self, message: str) -> None: except RuntimeError: # already on the UI thread self._show_progress(message) + def _on_section_mounted(self, section_id: str | None) -> None: + """A deferred tab now exists: fill what was set before it did.""" + if section_id == 'section-api': + self._update_api_urls() + self._update_api_curl() + self._sync_api_models(list(getattr(self, '_api_models_wanted', []))) + def _show_progress(self, message: str) -> None: self._status(message) # also recorded in the TUI log + self._activity_note = message # and beside the running action + + def _draw_activity(self) -> None: + """The activity line: what is running, for how long, and its progress.""" + widget = getattr(self, '_activity_widget', None) + if widget is None: + return + if not self._activity: + if widget.display: + widget.display = False + self._activity_note = '' + return + now = time.monotonic() + spin = SPINNER[int(now * 10) % len(SPINNER)] + parts = [f'{label} · {now - started:.0f}s' + for label, started in sorted(self._activity.values(), key=lambda v: v[1])] + note = f' — {self._activity_note}' if self._activity_note else '' + widget.update(f'{spin} ' + ' | '.join(parts) + note) + widget.display = True # -- resizable panes --------------------------------------------------- + def _log_height(self) -> int: + """The runtime pane's log height: the chosen one, capped near half the + screen, so on a small terminal the lease and deployment tables keep + rows when the pane opens (its tabs and picker take ~9 rows already).""" + # The tabbed area holds the tab strip (2) and the source picker (3) + # before any log line: 8 is those and three lines. + rows = self.size.height or 50 + return max(8, min(self._log_h, rows // 2 - 4)) + + #: Below this many rows the pane descriptions give way to tables and logs. + COMPACT_ROWS = 32 + + def on_resize(self, event: events.Resize) -> None: + self.set_class(event.size.height < self.COMPACT_ROWS, 'compact') + try: + self._apply_sizes() + except Exception: # noqa: BLE001 - not mounted yet + pass + def _apply_sizes(self) -> None: self.query_one('#sidebar').styles.width = self._sidebar_w - self.query_one('#docker-tabs').styles.height = self._log_h - self.query_one('#models').styles.height = self._models_h - self.query_one('#leases-pane').styles.height = self._leases_h + self.query_one('#docker-tabs').styles.height = self._log_height() + # Fixed heights only matter beside a divider; a tab fills its pane. + if not self.TABBED_CATALOG: + self.query_one('#models').styles.height = self._models_h + if not self.TABBED_TABLES: + self.query_one('#leases-pane').styles.height = self._leases_h def _drag_sidebar(self, delta: int) -> None: # Allow the full width range (down to a sliver, up to nearly all of it), @@ -1345,7 +1477,7 @@ def _drag_sidebar(self, delta: int) -> None: self._apply_sizes() def _drag_logs(self, delta: int) -> None: - # Dragging the divider down (delta > 0) makes the docker pane shorter. + # Dragging the divider down (delta > 0) makes the runtime pane shorter. self._log_h = max(6, min(60, self._log_h - delta)) self._apply_sizes() @@ -1444,10 +1576,14 @@ def _cached_label(source: str, hub: Path | None) -> str: def _sync_api_models(self, names: list[str]) -> None: """Point the API model selector at the currently-ready endpoints only.""" + self._api_models_wanted = names if names == self._ready_endpoints: return + try: + select = self.query_one('#api-model', Select) + except NoMatches: + return # the API tab is not built yet; filled when it is self._ready_endpoints = names - select = self.query_one('#api-model', Select) current = None if _select_is_blank(select.value) else select.value select.set_options([(n, n) for n in names]) if current in names: @@ -1468,9 +1604,26 @@ def _update_catalog_help(self) -> None: 'open it in Open WebUI.' ) + def _catalog_stamp(self): + """The catalog file's (mtime, size), or None when there is none.""" + try: + st = self.catalog_path.stat() if self.catalog_path else None + except OSError: + return None + return None if st is None else (st.st_mtime_ns, st.st_size) + + def _reload_catalog_if_changed(self) -> None: + """Pick up an edit made outside the TUI (checked on each refresh).""" + stamp = self._catalog_stamp() + if stamp is not None and stamp != getattr(self, '_catalog_seen', None): + self._reload_catalog() + def _reload_catalog(self) -> None: if not self.catalog_path or not self.catalog_path.exists(): return + # Remember this version even if it fails to load: a half-saved edit is + # reported once, not on every refresh until it is fixed. + self._catalog_seen = self._catalog_stamp() try: from .leasing import Catalog self.catalog = Catalog.load(self.catalog_path) @@ -1509,8 +1662,8 @@ def _collect(self) -> dict[str, Any]: 'leases': leases, 'deployments': deployments, 'observed': self._observed, 'assignments': self._assignments, } - if not self._collapsed['docker'] and self._active_tab == 'tab-containers': - data['ps'] = self._compose_ps_rows() + if not self._collapsed['docker']: + data['instances'] = self._instance_list() if not self._collapsed['system']: data['gpus'] = self._gpu_rows() data['sysinfo'] = self._system_line() @@ -1528,8 +1681,11 @@ def _render(self, data: dict[str, Any]) -> None: self._fill_deployments( data['deployments'], data['observed'], data['assignments'], data['leases'] ) - if 'ps' in data: - self._fill_ps(data['ps']) + if 'instances' in data: + if data['instances'] is not None: + self._last_instances = data['instances'] + if self._active_tab == 'tab-containers': + self._fill_ps(self._ps_rows(data['instances'])) if 'gpus' in data: self._fill_gpus(data['gpus']) self.query_one('#sysinfo', Static).update(data.get('sysinfo', '')) @@ -1628,8 +1784,17 @@ def _refresh_bg(self) -> None: data = self._collect() self.call_from_thread(self._render, data) + def action_refresh_now(self) -> None: + """The `r` key: a refresh the user asked for (the timer calls + ``action_refresh`` directly, and must not fill the TUI log). It also + rereads the catalog, edited or not.""" + self._cli(cli.command('status')) + self._reload_catalog() + self.action_refresh() + def action_refresh(self) -> None: self._sync_pane_state() # capture pane state on the UI thread first + self._reload_catalog_if_changed() self._refresh_bg() def _update_summary(self, leases, deployments, observed) -> None: @@ -1640,16 +1805,34 @@ def _update_summary(self, leases, deployments, observed) -> None: # "0 running" — say we're still observing instead of silently lying. observing = self._observed_at is None running_label = 'observing…' if observing else f'{running} running' + # Assign only on change: Textual repaints a pane whenever its border + # title is set, even to the same text, and these two panes hold the + # lease and deployment tables -- most of the screen, every tick + # (~7 KB/s of terminal output while nothing changed). try: - self.query_one('#docker', Collapsible).title = ( - f'docker — {running_label}' - ) - self.query_one('#leases-pane').border_title = ( - f'leases — {active} active / {len(leases)}' - ) - self.query_one('#deployments-pane').border_title = ( - f'deployments — {running_label} / {len(deployments)}' - ) + docker = self.query_one('#docker', Collapsible) + title = f'runtime — {running_label}' + if docker.title != title: + docker.title = title + for pane, text in ( + ('#leases-pane', f'leases — {active} active / {len(leases)}'), + ('#deployments-pane', + f'deployments — {running_label} / {len(deployments)}'), + ): + widget = self.query_one(pane) + if widget.border_title != text: + widget.border_title = text + if self.TABBED_TABLES: + # A hidden tab's counts show on its label. + tabs = self.query_one('#table-tabs', TabbedContent) + for pane_id, text in ( + ('pane-leases', f'Leases {active}/{len(leases)}'), + ('pane-deployments', f'Deployments {"…" if observing else running}' + f'/{len(deployments)}'), + ): + tab = tabs.get_tab(pane_id) + if str(tab.label) != text: + tab.label = text except Exception: # noqa: BLE001 pass @@ -1732,12 +1915,15 @@ def _fill_deployments(self, deployments, observed, assignments, leases) -> None: def _fill_ps(self, rows) -> None: table = self.query_one('#ps', DataTable) - new_rows = [ - (row['service'], row['status'], row['created'] or '-', - row['id'] or '-', row['ports'] or '-') - for row in rows - ] - if not rows: + if rows is None: + new_rows = [('(cannot read the runtime)', '-', '-', '-', '-')] + else: + new_rows = [ + (row['name'], row['status'], row['serves'] or '-', + row['started'] or '-', row['ports'] or '-') + for row in rows + ] + if rows is not None and not rows: new_rows = [('(nothing running)', '-', '-', '-', '-')] self._diff_fill(table, new_rows, '_ps_rows_cache', id_index=0) @@ -1838,68 +2024,34 @@ def _system_line(self) -> str: # -- docker ps --------------------------------------------------------- - def _compose_ps_rows(self) -> list[dict[str, str]]: - """Best-effort ``docker compose ps`` rows.""" - import json - - backend = self.controller.backend - path = getattr(backend, 'compose_file', None) - run = getattr(backend, 'run', None) - if not path or not run or not Path(path).exists(): - return [] - project = getattr(backend, 'project', 'infer-stack') + def _instance_list(self): + """What the backend runs (worker thread), or ``None`` if unreadable.""" try: - out = run([ - 'docker', 'compose', '-p', str(project), '-f', str(path), - 'ps', '--format', 'json', - ]) - except Exception: # noqa: BLE001 - ps is best-effort - return [] - rows: list[dict[str, Any]] = [] - out = (out or '').strip() - if not out: - return [] - try: - parsed = json.loads(out) - rows = parsed if isinstance(parsed, list) else [parsed] - except json.JSONDecodeError: - for line in out.splitlines(): - line = line.strip() - if line: - try: - rows.append(json.loads(line)) - except json.JSONDecodeError: - pass - result = [] - for row in rows: - ports = _fmt_ports(row) or str(row.get('Ports') or '') - cid = str(row.get('ID') or '')[:12] - # `Status` is the docker-ps STATUS column ("Up 3 minutes") — it - # carries the uptime; `CreatedAt`/`RunningFor` give the age. - created = str(row.get('CreatedAt') or row.get('RunningFor') or '') - result.append({ - 'service': str(row.get('Service') or row.get('Name') or '?'), - 'status': str(row.get('Status') or row.get('State') or '?'), - 'created': created, - 'id': cid, - 'ports': ports, - }) - return sorted(result, key=lambda r: r['service']) + return list(self.controller.backend.instances()) + except Exception: # noqa: BLE001 - a monitor must never crash + return None - # -- logs -------------------------------------------------------------- + def _ps_rows(self, instances) -> list[dict[str, str]] | None: + """Rows for the Instances table (same shape as ``infer-stack ps``).""" + if instances is None: + return None + served = {g.id: sorted(g.served) for g in (self._last_deployments or [])} + return [{ + 'name': i.name, + 'status': i.status, + 'serves': (', '.join(served.get(i.deployment_id, [])) + if i.deployment_id else '(front door)'), + 'started': (i.started or '')[:19].replace('T', ' '), + 'ports': i.ports, + } for i in instances] def _service_names(self) -> list[str]: - """Service names from the on-disk compose file (best-effort).""" - backend = self.controller.backend - path = getattr(backend, 'compose_file', None) - if not path: - return [] - try: - import yaml - data = yaml.safe_load(Path(path).read_text()) or {} - return sorted((data.get('services') or {}).keys()) - except Exception: # noqa: BLE001 - return [] + """Names of the instances last seen (refreshed by the worker).""" + return sorted({i.name for i in self._last_instances}) + + def _engine_names(self) -> list[str]: + """Names of the last-seen instances that serve a deployment.""" + return sorted({i.name for i in self._last_instances if i.is_engine}) def _sync_log_services(self) -> None: names = self._service_names() @@ -1908,8 +2060,8 @@ def _sync_log_services(self) -> None: self._service_options = names select = self.query_one('#logsvc', Select) options = [ - ('(engines — no litellm)', ENGINE_SERVICES), - ('(all services)', ALL_SERVICES), + ('(engines)', ENGINE_SERVICES), + ('(everything)', ALL_SERVICES), ] + [(n, n) for n in names] # Keep whatever is selected. The two sentinels are not service names, # so they have to be allowed through explicitly or refreshing the @@ -1952,13 +2104,20 @@ def on_input_changed(self, event: Input.Changed) -> None: self._update_api_curl() def _default_proc_factory(self) -> Callable[[str | None], Any]: - def factory(service: str | None): + def factory(service): + from .leasing.instances import LogFollower + backend = self.controller.backend - path = getattr(backend, 'compose_file', None) - project = getattr(backend, 'project', 'infer-stack') - if not path: - return None - return _DockerLogProc(str(project), str(path), service) + if isinstance(service, (list, tuple)): + wanted: set[str] | None = {str(s) for s in service} + else: + wanted = {str(service)} if service else None + + def listing(): + found = backend.instances() + return [i for i in found if wanted is None or i.name in wanted] + + return LogFollower(listing, prefix='auto') return factory @@ -1973,6 +2132,8 @@ def _restart_logs(self, service: str) -> None: log = self.query_one('#logs', RichLog) log.clear() self._log_lines = [] + self._log_backlog.clear() + self._log_dropped = 0 target, label = self._resolve_log_target(service) if target is NO_LOG_TARGET: log.write(f'— {label} —') @@ -1990,14 +2151,14 @@ def _resolve_log_target(self, service: str): its absence, and it stays that way once engines do appear. """ if service == ENGINE_SERVICES: - names = engine_services(self._service_names()) + names = self._engine_names() if not names: return NO_LOG_TARGET, ( - 'no engine services yet; choose (all services) ' + 'no engines running; choose (everything) ' 'for the gateway') return names, f'engines: {", ".join(names)}' if not service: - return None, 'all services' + return None, 'everything' return service, service def _stop_log_proc(self) -> None: @@ -2021,7 +2182,7 @@ def _stream_logs(self, service, generation: int) -> None: if proc is None: self.call_from_thread( self._append_log_if_current, generation, - '(no compose project yet — acquire a model)' + '(nothing to follow — acquire a model)' ) return if generation != self._log_generation: @@ -2033,9 +2194,10 @@ def _stream_logs(self, service, generation: int) -> None: self._log_proc = proc try: for line in compact_litellm_tracebacks(proc.stdout): - self.call_from_thread( - self._append_log_if_current, generation, line.rstrip('\n') - ) + if generation != self._log_generation: + break + # Drawn in batches by _drain_logs; deque.append is thread-safe. + self._log_pending.append((generation, line.rstrip('\n'))) except Exception: # noqa: BLE001 - stream ends when the proc dies pass finally: @@ -2048,8 +2210,36 @@ def _append_log_if_current(self, generation: int, line: str) -> None: self._append_log(line) def _append_log(self, line: str) -> None: - self._log_lines.append(line) - self.query_one('#logs', RichLog).write(line) + self._write_log_lines([line]) + + def _write_log_lines(self, lines: list[str]) -> None: + self._log_lines.extend(lines) + if len(self._log_lines) > 2 * LOG_PANE_LINES: + del self._log_lines[:-LOG_PANE_LINES] + # Engines color their output (vLLM's "(APIServer pid=1)" prefix): as + # a plain string the escape codes garbled the line; as ANSI they are + # colors, and _log_lines keeps the raw text for search and copy. + from rich.text import Text + + self.query_one('#logs', RichLog).write(Text.from_ansi('\n'.join(lines))) + + def _drain_logs(self) -> None: + """Draw streamed lines: a bounded batch per tick, newest lines kept.""" + pending, backlog = self._log_pending, self._log_backlog + while pending: + generation, line = pending.popleft() + if generation != self._log_generation: + continue + if len(backlog) == backlog.maxlen: + self._log_dropped += 1 # the oldest undrawn line falls off + backlog.append(line) + if not backlog: + return + batch = [backlog.popleft() for _ in range(min(LOG_DRAIN_LINES, len(backlog)))] + if self._log_dropped: + batch.insert(0, f'… {self._log_dropped} earlier line(s) not shown') + self._log_dropped = 0 + self._write_log_lines(batch) # -- the TUI's own log ------------------------------------------------- @@ -2162,6 +2352,14 @@ def on_worker_state_changed(self, event) -> None: worker = event.worker name = worker.name or worker.group or 'worker' + if event.state is WorkerState.RUNNING: + label = activity_label(worker.description) + if label is not None: + self._activity[worker] = (label, time.monotonic()) + self._draw_activity() + elif event.state in (WorkerState.SUCCESS, WorkerState.ERROR, + WorkerState.CANCELLED): + self._activity.pop(worker, None) if event.state is WorkerState.ERROR: error = getattr(worker, 'error', None) if error is not None: @@ -2441,8 +2639,6 @@ def on_button_pressed(self, event: Button.Pressed) -> None: 'btn-release-all': self.action_release_all, 'btn-evict': self.action_evict, 'btn-evict-all': self.action_evict_all, - 'btn-cleanup': self.action_cleanup, - 'btn-cleanup-deployments': self.action_cleanup, 'btn-suggest': self.action_suggest, 'btn-add-model': self.action_add_model, 'btn-add-endpoint': self.action_add_endpoint, @@ -2483,6 +2679,7 @@ def _on_apply_ui_settings(self) -> None: return self.ledger_interval = ledger self.observe_interval = max(observe, ledger) # observe never beats ledger + self.app_log('CLI: none; poll intervals are a TUI-only preference') self._apply_poll_settings() prefs = load_tui_settings() prefs['ledger_interval'] = self.ledger_interval @@ -2512,6 +2709,10 @@ def _on_save_settings(self) -> None: ) path = save_settings(s) self._status(f'saved settings → {path}') + self._cli(*(cli.command('config', 'set', key, str(s[key]).lower() + if isinstance(s[key], bool) else s[key]) + for key in ('backend', 'data_dir', 'ui', 'reverse_proxy', + 'skip_display_gpus') if key in s)) except Exception as ex: # noqa: BLE001 self._status(f'save settings failed: {ex}') @@ -2548,21 +2749,16 @@ def action_evict_all(self) -> None: self._do_evict_all() def action_cleanup(self) -> None: - self._status('cleaning up released/expired leases + stopped deployments…') - self.app_log('CLI: none yet; this only forgets finished rows in ' - 'the ledger (nothing running changes)') + self._status('clearing finished rows: released/expired leases, stopped deployments…') + self._cli(cli.command('gc', '--forget')) self._do_cleanup() # -- docker compose control ------------------------------------------- - def _compose_target(self) -> tuple[Any, str, str] | None: - """(run, project, compose_file) for the leasing project, or None.""" - backend = self.controller.backend - path = getattr(backend, 'compose_file', None) - run = getattr(backend, 'run', None) - if not path or not run or not Path(path).exists(): - return None - return run, str(getattr(backend, 'project', 'infer-stack')), str(path) + def _compose_target(self): + """The backend's rendered file, when a render exists; else ``None``.""" + rendered = getattr(self.controller.backend, 'rendered_file', None) + return rendered if rendered is not None and Path(rendered).exists() else None def action_compose_up(self) -> None: if self._compose_target() is None: @@ -2582,29 +2778,24 @@ def _do_apply(self) -> None: msg = ('apply done' if not rec.publication_pending else 'apply did not fully take effect; still pending') except Exception as ex: # noqa: BLE001 - msg = f'apply failed: {ex}' + msg = f'apply failed: {_why(ex)}' self._after_mutation(msg) def action_compose_down(self) -> None: if self._compose_target() is None: self._refuse('nothing rendered yet — nothing to bring down') return - self._status('docker compose down (raw: bypasses leases; releases nothing)…') + self._status('down (bypasses leases; releases nothing)…') self._cli(cli.command('stack', 'down')) - self._do_compose(['down', '--remove-orphans'], 'down') + self._do_down() @work(thread=True, exclusive=True, group='mutate') - def _do_compose(self, args: list[str], label: str) -> None: - target = self._compose_target() - if target is None: - self._after_mutation('nothing rendered yet') - return - run, project, path = target + def _do_down(self) -> None: try: - run(['docker', 'compose', '-p', project, '-f', path, *args]) - msg = f'compose {label} done' + self.controller.backend.down() + msg = 'down done' except Exception as ex: # noqa: BLE001 - msg = f'compose {label} failed: {ex}' + msg = f'down failed: {_why(ex)}' self._after_mutation(msg) # -- open in browser --------------------------------------------------- @@ -3036,15 +3227,25 @@ def _api_chat(self, model: str, prompt: str) -> str: self._raise_for_body(resp) return self._completion_text(resp.json()) - def _curl_for(self, model: str, prompt: str) -> str: + def _curl_for(self, model: str, prompt: str, *, reveal_key: bool = False) -> str: """The equivalent ``curl`` for a chat- or text-completion, matching the - endpoint's served protocol, against the gateway.""" + endpoint's served protocol, against the gateway. + + The key is read at run time (``infer-stack env LITELLM_MASTER_KEY``) + rather than shown: the pane is on screen, and screens get shared. + ``reveal_key`` puts the literal key in, for the clipboard. + """ import json as _json base, key = self._litellm() if not base: return '# acquire a model first — no LiteLLM gateway yet' - auth = f" -H 'Authorization: Bearer {key}'" if key else '' + if key and reveal_key: + auth = f" -H 'Authorization: Bearer {key}'" + elif key: + auth = ' -H "Authorization: Bearer $(infer-stack env LITELLM_MASTER_KEY)"' + else: + auth = '' if self._protocol_for(model) == 'completions': path = '/v1/completions' body = _json.dumps({ @@ -3090,7 +3291,10 @@ def _api_log(self, line: str) -> None: self.query_one('#api-out', RichLog).write(line) def _selected_api_model(self) -> str | None: - value = self.query_one('#api-model', Select).value + try: + value = self.query_one('#api-model', Select).value + except NoMatches: + return None # the API tab is built right after the first frame return None if _select_is_blank(value) else str(value) def action_api_send(self) -> None: @@ -3101,6 +3305,7 @@ def action_api_send(self) -> None: prompt = (self.query_one('#api-prompt', Input).value.strip() or 'Say hello in one short sentence.') self._api_log(f'> [{model}] {prompt}') + self._cli(cli.command('test', model, '--prompt', prompt)) self._do_api_send(model, prompt) def action_api_test_all(self) -> None: @@ -3109,15 +3314,20 @@ def action_api_test_all(self) -> None: self._refuse('no ready models to test (acquire one first)') return self._api_log(f'— testing {len(models)} ready model(s) —') + self._cli(*(cli.command('test', model) for model in models)) self._do_api_test_all(models) def action_api_list_models(self) -> None: self._api_log('> GET /v1/models (what the gateway routes)') + self._cli(cli.MODELS_CURL) self._do_api_list() def action_api_copy_curl(self) -> None: - text = str(self.query_one('#api-curl', Static).render()) - ok = self._copy(text) + # The clipboard gets the literal key (works in any shell); the pane + # shows only how to read it. + model = self._selected_api_model() or '' + prompt = self.query_one('#api-prompt', Input).value.strip() or 'hello' + ok = self._copy(self._curl_for(model, prompt, reveal_key=True)) self._status('copied curl to clipboard' if ok else 'copy failed — install wl-copy/xclip, or enable OSC 52') @@ -3189,7 +3399,7 @@ def _do_acquire(self, name: str) -> None: 'engine may still be loading' ) except Exception as ex: # noqa: BLE001 - msg = f'acquire {name} failed: {ex}' + msg = f'acquire {name} failed: {_why(ex)}' self.call_from_thread(self._finish_acquire, name, msg) def _finish_acquire(self, name: str, message: str) -> None: @@ -3207,7 +3417,7 @@ def _do_release(self, ids: list[str]) -> None: self._lease_sel.clear() msg = f'released {len(ids)} lease(s)' except Exception as ex: # noqa: BLE001 - msg = f'release failed: {ex}' + msg = f'release failed: {_why(ex)}' self._after_mutation(msg) @work(thread=True, exclusive=True, group='mutate') @@ -3216,7 +3426,7 @@ def _do_release_all(self) -> None: out = self.controller.release_leases(None) msg = f'released {len(out.released_lease_ids)} lease(s)' except Exception as ex: # noqa: BLE001 - msg = f'release --all failed: {ex}' + msg = f'release --all failed: {_why(ex)}' self._after_mutation(msg) @work(thread=True, exclusive=True, group='mutate') @@ -3233,7 +3443,7 @@ def _do_evict(self, ids: list[str]) -> None: msg = (f'none of the {len(ids)} selected were idle — ' 'release their leases first') except Exception as ex: # noqa: BLE001 - msg = f'evict failed: {ex}' + msg = f'evict failed: {_why(ex)}' self._after_mutation(msg) @work(thread=True, exclusive=True, group='mutate') @@ -3245,17 +3455,17 @@ def _do_evict_all(self) -> None: msg = (f'evicted {n} idle deployment(s)' if n else 'no idle deployments to evict') except Exception as ex: # noqa: BLE001 - msg = f'evict all failed: {ex}' + msg = f'evict all failed: {_why(ex)}' self._after_mutation(msg) @work(thread=True, exclusive=True, group='mutate') def _do_cleanup(self) -> None: try: n_leases, n_deployments = self.controller.prune() - msg = (f'cleaned up {n_leases} released/expired lease(s) + ' + msg = (f'cleared {n_leases} released/expired lease(s) + ' f'{n_deployments} stopped deployment(s)') except Exception as ex: # noqa: BLE001 - msg = f'cleanup failed: {ex}' + msg = f'clear finished failed: {_why(ex)}' self._after_mutation(msg) def _after_mutation(self, message: str) -> None: @@ -3275,16 +3485,20 @@ def _parse_kv_str(text: str) -> dict[str, Any]: return out -def _fmt_ports(row: dict) -> str: - """Compact published-ports string from a compose ps JSON row.""" - pubs = row.get('Publishers') or [] - bits = [] - for pub in pubs: - published = pub.get('PublishedPort') - target = pub.get('TargetPort') - if published: - bits.append(f'{published}->{target}') - return ', '.join(bits) +def _seconds_since_process_start() -> float | None: + """Wall time since this process started (Linux ``/proc``), else ``None``.""" + import os + + try: + with open('/proc/self/stat') as handle: + # Field 22 counts clock ticks since boot; the command name (field 2) + # may contain spaces, so split after its closing parenthesis. + start_ticks = int(handle.read().rsplit(')', 1)[1].split()[19]) + with open('/proc/uptime') as handle: + uptime = float(handle.read().split()[0]) + return uptime - start_ticks / os.sysconf('SC_CLK_TCK') + except (OSError, ValueError, IndexError): + return None def run_tui( @@ -3293,6 +3507,7 @@ def run_tui( *, interval: float = 3.0, catalog_path: str | Path | None = None, + exit_after_paint: bool = False, ) -> int: """Run the TUI against a built controller + catalog. Returns an exit code.""" # The narration loguru sink writes to stderr, which would corrupt the @@ -3304,7 +3519,14 @@ def run_tui( except Exception: # noqa: BLE001 pass app: Any = InferStackTUI( - controller, catalog, interval=interval, catalog_path=catalog_path + controller, catalog, interval=interval, catalog_path=catalog_path, + exit_after_paint=exit_after_paint, ) app.run() + if exit_after_paint: + first = app.first_frame_s + exited = _seconds_since_process_start() + print(f'first frame {first:.2f}s after process start; exited at {exited:.2f}s' + if first is not None and exited is not None + else 'first frame drawn (process start time unavailable)') return 0 diff --git a/pyproject.toml b/pyproject.toml index a08506f8..906273c7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,11 +33,11 @@ dynamic = [ ] dependencies = [ "jinja2>=3.1", + "kwconf>=0.11.0", "loguru>=0.7", "pyyaml>=6.0", "requests>=2.31", "rich>=13.0", - "scriptconfig>=0.9", "ubelt>=1.3", ] optional-dependencies.tests = [ @@ -70,9 +70,6 @@ packages.find.include = [ "infer_stack*", ] -[tool.uv] -exclude-newer = "2026-06-04T04:00:00Z" - [tool.ruff] target-version = "py310" line-length = 80 diff --git a/recipies/compose_pythia_inspect_mmlu_compat.md b/recipies/compose_pythia_inspect_mmlu_compat.md index b7ca7131..8687a7bc 100644 --- a/recipies/compose_pythia_inspect_mmlu_compat.md +++ b/recipies/compose_pythia_inspect_mmlu_compat.md @@ -1,5 +1,6 @@ # Compose recipe: Pythia with LiteLLM-only chat compatibility (Inspect / MMLU) +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../README.md). > Schema note: built-in profiles now resolve to stack-graph profiles. vLLM runtimes live under `providers.vllm.runtimes`, LiteLLM routes live under `routes`, and direct Ollama profiles can run without LiteLLM. diff --git a/recipies/compose_pythia_qwen36_mixed_4x96GB.md b/recipies/compose_pythia_qwen36_mixed_4x96GB.md index 73d48bb9..464f67d0 100644 --- a/recipies/compose_pythia_qwen36_mixed_4x96GB.md +++ b/recipies/compose_pythia_qwen36_mixed_4x96GB.md @@ -1,5 +1,6 @@ # Compose recipe: Pythia 6.9B + Pythia 2.8B + Qwen3.6-35B-A3B on a 4x96GB host +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../README.md). > Schema note: built-in profiles now resolve to stack-graph profiles. vLLM runtimes live under `providers.vllm.runtimes`, LiteLLM routes live under `routes`, and direct Ollama profiles can run without LiteLLM. diff --git a/recipies/compose_qwen35_122b_4x96GB.md b/recipies/compose_qwen35_122b_4x96GB.md index ea747ca9..9a0fcc6e 100644 --- a/recipies/compose_qwen35_122b_4x96GB.md +++ b/recipies/compose_qwen35_122b_4x96GB.md @@ -1,5 +1,7 @@ # Compose recipe: Qwen3.5-122B-A10B on a 4x96GB host +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../README.md). + This is the shortest working end-to-end example for serving **Qwen/Qwen3.5-122B-A10B** on a machine with **4 x 96GB GPUs** using the **Compose** backend. This profile uses: diff --git a/recipies/compose_qwen35_122b_fp8_4x96GB.md b/recipies/compose_qwen35_122b_fp8_4x96GB.md index 334c02ca..b01417d5 100644 --- a/recipies/compose_qwen35_122b_fp8_4x96GB.md +++ b/recipies/compose_qwen35_122b_fp8_4x96GB.md @@ -1,5 +1,7 @@ # Compose recipe: Qwen3.5-122B-A10B-FP8 on a 4x96GB host +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../README.md). + This is the shortest working end-to-end example for serving **Qwen/Qwen3.5-122B-A10B-FP8** on a machine with **4 x 96GB GPUs** using the **Compose** backend. This profile uses: diff --git a/recipies/compose_qwen36_35b_a3b_4x96GB.md b/recipies/compose_qwen36_35b_a3b_4x96GB.md index a2c41f16..c9cbd4a2 100644 --- a/recipies/compose_qwen36_35b_a3b_4x96GB.md +++ b/recipies/compose_qwen36_35b_a3b_4x96GB.md @@ -1,5 +1,7 @@ # Compose recipe: Qwen3.6-35B-A3B on a 4x96GB host +> **Pre-leasing.** This page uses the stack-profile CLI (`setup`, `up`, `deploy`, `switch`, `smoke-test`, `config.yaml` profiles), which was removed. Its commands no longer run. The model and vLLM settings may still inform a catalog entry; the current workflow is in the [README](../README.md). + This is the shortest end-to-end example for serving **Qwen/Qwen3.6-35B-A3B** on a machine with **4 x 96GB GPUs** using the **Compose** backend. This profile uses: diff --git a/requirements/locks/tests.txt b/requirements/locks/tests.txt index 03dd8137..f78fc55b 100644 --- a/requirements/locks/tests.txt +++ b/requirements/locks/tests.txt @@ -18,6 +18,8 @@ iniconfig==2.3.0 # via pytest jinja2==3.1.6 # via infer-stack +kwconf==0.11.0 + # via infer-stack loguru==0.7.3 # via infer-stack markdown-it-py==4.2.0 @@ -46,15 +48,11 @@ pytest-codeblocks==0.17.0 pytest-cov==7.1.0 # via infer-stack pyyaml==6.0.3 - # via - # infer-stack - # scriptconfig + # via infer-stack requests==2.34.2 # via infer-stack rich==15.0.0 # via infer-stack -scriptconfig==0.9.1 - # via infer-stack tomli==2.4.1 ; python_full_version <= '3.11' # via # coverage @@ -62,9 +60,7 @@ tomli==2.4.1 ; python_full_version <= '3.11' typing-extensions==4.15.0 ; python_full_version < '3.11' # via exceptiongroup ubelt==1.4.2 - # via - # infer-stack - # scriptconfig + # via infer-stack urllib3==2.7.0 # via requests win32-setctime==1.2.0 ; sys_platform == 'win32' diff --git a/tests/test_cli_catalog.py b/tests/test_cli_catalog.py index 578e8344..d581083f 100644 --- a/tests/test_cli_catalog.py +++ b/tests/test_cli_catalog.py @@ -382,3 +382,13 @@ def test_suggest_no_fit_writes_nothing(tmp_path, capsys): err = capsys.readouterr().err assert 'no pooled model fits' in err assert not (tmp_path / 'catalog.yaml').exists() + + +def test_suggest_simulator_gives_a_gpu_free_first_run(tmp_path): + """On a host without a GPU the documented first run dead-ended; the + simulator endpoint makes the same steps run anywhere.""" + CatalogInitCLI.main(argv=_opts(tmp_path)) + assert CatalogSuggestCLI.main(argv=[*_opts(tmp_path), '--simulator', '--apply']) == 0 + cat = Catalog.load(cat_path(tmp_path)) + (request,) = cat.resolve_names(['mock-smol']) + assert request.spec['runtime']['simulator']['kind'] == 'llm-d-sim' diff --git a/tests/test_cli_equivalent.py b/tests/test_cli_equivalent.py index 9d0e2e79..c90aa8c5 100644 --- a/tests/test_cli_equivalent.py +++ b/tests/test_cli_equivalent.py @@ -9,7 +9,7 @@ import shlex import pytest -import scriptconfig as scfg +import kwconf as kw from infer_stack import cli_equivalent as cli from infer_stack.cli import ManageCLI @@ -20,7 +20,7 @@ def parse(text: str): words = shlex.split(text.split(' #', 1)[0]) assert words[0] == 'infer-stack' node, rest = ManageCLI, words[1:] - while isinstance(node, type) and issubclass(node, scfg.ModalCLI): + while isinstance(node, type) and issubclass(node, kw.ModalCLI): node = getattr(node, rest[0].replace('-', '_')) rest = rest[1:] return node, node.cli(argv=rest, strict=True) diff --git a/tests/test_cli_leasing.py b/tests/test_cli_leasing.py index 9f612adc..9dcb0095 100644 --- a/tests/test_cli_leasing.py +++ b/tests/test_cli_leasing.py @@ -225,6 +225,22 @@ def test_release_evict_tears_down_immediately(env, capsys): assert _leases_json(env, capsys)['deployments'][0]['state'] == 'stopped' +def test_gc_forget_drops_only_finished_rows(env, capsys): + """`gc --forget` is the TUI's Clean up: history goes, live rows stay.""" + from infer_stack.cli.commands_leasing import GcCLI + + AcquireCLI.main(argv=['qwen-coder', *_base(env), '--owner', 'a']) + ReleaseCLI.main(argv=['--ledger', env.db, '--all', '--evict']) + AcquireCLI.main(argv=['reranker', *_base(env), '--owner', 'b']) + capsys.readouterr() + assert GcCLI.main(argv=['--ledger', env.db, '--forget', '--json']) == 0 + assert json.loads(capsys.readouterr().out) == {'leases': 1, 'deployments': 1} + data = _leases_json(env, capsys) + assert [le['owner'] for le in data['leases']] == ['b'] + assert [g['state'] for g in data['deployments']] != ['stopped'] + assert len(data['deployments']) == 1 + + def test_acquire_without_ttl_is_standing_lease(env, capsys): # No --ttl -> an infinite (standing-service) lease owned by the caller. AcquireCLI.main(argv=['qwen-coder', *_base(env)]) @@ -793,7 +809,7 @@ def test_evict_json_stdout_is_pure_json(env, capsys): out, err = capsys.readouterr() data = json.loads(out) # must parse: no human text mixed into stdout assert data == {'evicted': [], 'torn_down': [], 'missing': ['ghost']} - assert 'no idle deployment for: ghost' in err + assert 'evict: ghost is not a deployment or served endpoint' in err assert rc == 0 @@ -951,8 +967,7 @@ def test_routes_prune_drops_stale(tmp_path, monkeypatch, capsys): capsys.readouterr() rc = RoutesPruneCLI.main(argv=['--ledger', db, '--yes', '--json']) assert rc == 0 - raw = capsys.readouterr().out # `Write .env to ...` may precede the JSON - out = json.loads(raw[raw.index('{'):]) + out = json.loads(capsys.readouterr().out) # --json output is only the JSON assert out['dropped'] == ['beta'] assert out['kept'] == ['alpha'] @@ -969,7 +984,7 @@ def test_apply_exits_nonzero_while_publication_stays_pending(env, capsys, monkey from infer_stack.leasing.backend import MemoryBackend class RoutesNeverVerify(MemoryBackend): - def converge(self, desired, *, apply=True): + def converge(self, desired, *, apply=True, placement=None): self.last_unplaced, self.last_errors, self.last_assignments = [], [], {} def apply(self): diff --git a/tests/test_cli_meta.py b/tests/test_cli_meta.py index d843cb8e..08030a34 100644 --- a/tests/test_cli_meta.py +++ b/tests/test_cli_meta.py @@ -139,15 +139,17 @@ def test_render_rich_colorizes_status() -> None: assert '\x1b[' in out -def test_day2_compose_base_prefers_leasing(tmp_path: Path, monkeypatch) -> None: - from infer_stack.cli.commands_runtime import _day2_compose_base +def test_day2_compose_verbs_target_the_backends_project(tmp_path: Path, monkeypatch) -> None: + from types import SimpleNamespace + + from infer_stack.cli.commands_runtime import _compose_argv monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path)) compose_file = tmp_path / 'leasing' / 'compose' / 'docker-compose.yml' compose_file.parent.mkdir(parents=True) compose_file.write_text('services: {}\n') - # leasing compose present -> targets it without needing config.yaml - base = _day2_compose_base(None, 'logs') + # The compose backend's own project; no config.yaml needed. + base = _compose_argv(SimpleNamespace(backend='compose')) assert base == [ 'docker', 'compose', '-p', 'infer-stack', '-f', str(compose_file) ] diff --git a/tests/test_day2.py b/tests/test_day2.py new file mode 100644 index 00000000..0e787b21 --- /dev/null +++ b/tests/test_day2.py @@ -0,0 +1,264 @@ +"""Day-2 verbs (ps, logs, status, stack) read the backend, on either backend.""" + +from __future__ import annotations + +import json +from types import SimpleNamespace + +import pytest + +from infer_stack.leasing.instances import ( + DOCKER, + KUBERNETES, + MEMORY, + Instance, + UnknownTarget, + follow_argv, + from_residency, + history_argv, + resolve, +) +from infer_stack.leasing.residency import Container, Residency, ResidencyUnknown + +POD = Instance('model-qwen-0-abc', 'model-qwen-0-abc', 'grp-1', 'running', + runtime=KUBERNETES, namespace='kubeai') +GATEWAY = Instance('litellm', 'c0ffee1234', '', 'running', ports='14042->4000/tcp') + + +def test_log_commands_follow_the_runtime(): + assert history_argv(POD, tail=50) == [ + 'kubectl', '-n', 'kubeai', 'logs', '--tail', '50', 'model-qwen-0-abc'] + assert follow_argv(POD) == [ + 'kubectl', '-n', 'kubeai', 'logs', '--tail', '0', 'model-qwen-0-abc', '--follow'] + assert history_argv(GATEWAY, tail='all', timestamps=True) == [ + 'docker', 'logs', '--tail', 'all', '--timestamps', 'c0ffee1234'] + assert follow_argv(GATEWAY)[:2] == ['docker', 'attach'] + memory = Instance('grp-1', 'grp-1', 'grp-1', 'running', runtime=MEMORY) + assert history_argv(memory) is None and follow_argv(memory) is None + + +def test_instances_come_from_residency_engines_first(): + residency = Residency( + by_deployment={'grp-1': (Container('pod-1', 'grp-1', 'restarting', + restart_count=3, reason='CrashLoopBackOff'),)}, + others=(Container('gw', '', 'running', service='litellm'),), + ) + pods = from_residency(residency, runtime=KUBERNETES, namespace='kubeai') + assert [i.name for i in pods] == ['pod-1', 'gw'] # pods are named by pod + assert pods[0].status == 'restarting (CrashLoopBackOff, 3 restarts)' + containers = from_residency(residency, runtime=DOCKER) + assert [i.name for i in containers] == ['pod-1', 'litellm'] # service names + + +def test_a_target_resolves_by_alias_deployment_name_or_id(): + served = {'grp-1': ['qwen']} + both = [POD, GATEWAY] + assert resolve(both, ['qwen'], served) == [POD] + assert resolve(both, ['grp-1', 'litellm'], served) == [POD, GATEWAY] + assert resolve(both, ['c0ff'], served) == [GATEWAY] + with pytest.raises(UnknownTarget, match='running: litellm, model-qwen-0-abc'): + resolve(both, ['nope'], served) + + +class _Backend: + """A backend as the day-2 verbs see it.""" + + def __init__(self, instances, *, residency=None): + self._instances = instances + self._residency = residency + self.down_calls = 0 + + def instances(self): + if self._instances is None: + raise ResidencyUnknown('kubectl get pods failed') + return list(self._instances) + + def residency(self): + if self._residency is None: + raise ResidencyUnknown('kubectl get pods failed') + return self._residency + + def down(self): + self.down_calls += 1 + + +def _cli(monkeypatch, backend, served=None): + from infer_stack.cli import commands_runtime as rt + + monkeypatch.setattr(rt, '_day2_backend', lambda config: backend) + monkeypatch.setattr(rt, '_served_by_deployment', lambda: dict(served or {})) + return rt + + +def test_ps_has_one_shape_and_names_what_each_serves(monkeypatch, capsys): + rt = _cli(monkeypatch, _Backend([POD, GATEWAY]), {'grp-1': ['qwen']}) + assert rt.PsCLI.main(argv=[]) == 0 + out = capsys.readouterr().out.splitlines() + assert out[0].split() == ['NAME', 'STATUS', 'SERVES', 'GPUS', 'STARTED', 'ID', 'PORTS'] + assert 'qwen' in out[1] and 'model-qwen-0-abc' in out[1] + assert '(front door)' in out[2] and '14042->4000/tcp' in out[2] + + assert rt.PsCLI.main(argv=['qwen', '--json']) == 0 + rows = json.loads(capsys.readouterr().out) + assert [r['name'] for r in rows] == ['model-qwen-0-abc'] + assert rows[0]['serves'] == ['qwen'] and rows[0]['runtime'] == KUBERNETES + + +def test_ps_hides_finished_instances_unless_all(monkeypatch, capsys): + done = Instance('old', 'dead0000', 'grp-2', 'exited') + rt = _cli(monkeypatch, _Backend([POD, done])) + rt.PsCLI.main(argv=['-q']) + assert capsys.readouterr().out.split() == ['model-qwen-0-abc'] + rt.PsCLI.main(argv=['-q', '--all']) + assert capsys.readouterr().out.split() == ['model-qwen-0-abc', 'dead0000'] + + +def test_ps_says_why_when_the_runtime_cannot_be_read(monkeypatch): + rt = _cli(monkeypatch, _Backend(None)) + with pytest.raises(SystemExit, match='cannot read what is running'): + rt.PsCLI.main(argv=[]) + + +def test_logs_reads_the_named_endpoint_through_its_runtime(monkeypatch, capsys): + import subprocess + + rt = _cli(monkeypatch, _Backend([POD, GATEWAY]), {'grp-1': ['qwen']}) + ran = [] + + def fake_run(argv, **kw): + ran.append(argv) + return SimpleNamespace(stdout=b'INFO: loaded\n', returncode=0) + + monkeypatch.setattr(subprocess, 'run', fake_run) + assert rt.LogsCLI.main(argv=['qwen', '--tail', '5']) == 0 + assert ran == [['kubectl', '-n', 'kubeai', 'logs', '--tail', '5', 'model-qwen-0-abc']] + assert capsys.readouterr().out == 'INFO: loaded\n' # one instance: no prefix + with pytest.raises(SystemExit, match="no instance matches 'nope'"): + rt.LogsCLI.main(argv=['nope']) + + +def test_status_health_comes_from_residency(monkeypatch): + from infer_stack.cli.commands_runtime import _served_models + from infer_stack.leasing import DeploymentState + + def dep(gid): + return SimpleNamespace(id=gid, state=DeploymentState.LIVE, engine='vllm', + served={gid: {'hf_model_id': 'org/m'}}, spec={}) + + residency = Residency(by_deployment={ + 'up': (Container('c1', 'up', 'running'),), + 'loading': (Container('c3', 'loading', 'running', health='starting'),), + 'crashing': (Container('c2', 'crashing', 'exited'),), + }) + rows = _served_models([dep('up'), dep('loading'), dep('crashing'), dep('gone')], + _Backend([], residency=residency)) + assert {r[0]: r[3] for r in rows} == {'up': 'up', 'loading': 'starting', + 'crashing': 'exited', 'gone': 'STALE'} + rows = _served_models([dep('up')], _Backend([], residency=None)) + assert rows[0][3] == 'unverified' + # Recorded but not applied yet (an apply is running): not STALE. + rows = _served_models([dep('gone')], _Backend([], residency=residency), pending=True) + assert rows[0][3] == 'pending' + + +def test_stack_down_stops_the_backend_on_either_backend(monkeypatch): + backend = _Backend([]) + rt = _cli(monkeypatch, backend) + assert rt.StackDownCLI.main(argv=[]) == 0 + assert backend.down_calls == 1 + + +def test_stack_compose_targets_the_compose_project_on_this_host(monkeypatch, tmp_path): + from infer_stack.cli import commands_runtime as rt + + compose_file = tmp_path / 'docker-compose.yml' + project = SimpleNamespace(compose_file=compose_file, + compose_argv=lambda: ['docker', 'compose', '-p', 'gw']) + kubeai = SimpleNamespace(compose_project=lambda: project) + monkeypatch.setattr(rt, '_day2_backend', lambda config: kubeai) + with pytest.raises(SystemExit, match='nothing rendered yet'): + rt._compose_argv(SimpleNamespace()) + compose_file.write_text('services: {}\n') + assert rt._compose_argv(SimpleNamespace()) == ['docker', 'compose', '-p', 'gw'] + + monkeypatch.setattr(rt, '_day2_backend', lambda config: SimpleNamespace()) + with pytest.raises(SystemExit, match='no Compose project on this host'): + rt._compose_argv(SimpleNamespace()) + + +@pytest.mark.parametrize(('argv', 'follow', 'names'), [ + (['-f', 'qwen'], True, ['qwen']), + (['qwen', '-f'], True, ['qwen']), + (['--follow', 'qwen', 'litellm'], True, ['qwen', 'litellm']), + (['-f', 'false'], False, []), +]) +def test_a_flag_never_swallows_the_positional_after_it(argv, follow, names): + """kwconf flags take an optional value; `logs -f qwen` followed everything.""" + from infer_stack.cli.commands_runtime import LogsCLI + + config = LogsCLI.cli(argv=argv) + assert config.follow is follow and list(config.services or []) == names + + +def test_acquire_yes_before_the_endpoint_still_names_it(): + from infer_stack.cli.commands_leasing import AcquireCLI + + config = AcquireCLI.cli(argv=['--yes', 'qwen']) + assert config.yes is True and config.names == ['qwen'] + + +def test_no_color_turns_color_off(): + """kwconf reads a leading `no-` as negation: `--no-color` must negate `color`.""" + from infer_stack.cli.commands_runtime import LogsCLI + + assert LogsCLI.cli(argv=[]).color is True + assert LogsCLI.cli(argv=['--no-color']).color is False + + +def test_one_followed_instance_needs_no_name_prefix(monkeypatch): + """The TUI's follower leaves the name off while it follows one instance: + on a narrow pane the prefix took most of every line.""" + import os + import subprocess + import time + + from infer_stack.leasing.instances import LogFollower + + class _Done: + stdout = b'first\nsecond\n' + + class _Proc: + def __init__(self, *a, **kw): + r, w = os.pipe() + os.close(w) + self.stdout = os.fdopen(r, 'rb') + + def poll(self): + return 0 + + def terminate(self): + pass + + def wait(self, timeout=None): + return 0 + + monkeypatch.setattr(subprocess, 'run', lambda *a, **kw: _Done()) + monkeypatch.setattr(subprocess, 'Popen', _Proc) + follower = LogFollower(lambda: [GATEWAY], prefix='auto') + lines, deadline = [], time.monotonic() + 5 + for line in follower.stdout: + lines.append(line) + if len(lines) == 2 or time.monotonic() > deadline: + break + follower.terminate() + assert lines == ['first\n', 'second\n'] + + +def test_logs_into_a_pipe_drop_the_engines_color_codes(monkeypatch, capsys): + import subprocess + + rt = _cli(monkeypatch, _Backend([POD]), {'grp-1': ['qwen']}) + monkeypatch.setattr(subprocess, 'run', lambda argv, **kw: SimpleNamespace( + stdout=b'\x1b[1;36m(APIServer pid=1)\x1b[0;0m INFO ready\n', returncode=0)) + assert rt.LogsCLI.main(argv=['qwen']) == 0 + assert capsys.readouterr().out == '(APIServer pid=1) INFO ready\n' diff --git a/tests/test_leasing_admission.py b/tests/test_leasing_admission.py index 75630b51..2ebd2494 100644 --- a/tests/test_leasing_admission.py +++ b/tests/test_leasing_admission.py @@ -179,7 +179,9 @@ def preview(self, desired, placement=None, **kw): assert len(calls) == 2 and ledger.get_deployment(out.deployments[0].id).assigned_gpus == [0] -def test_unknown_residency_admits_only_resource_neutral_requests(tmp_path): # 35 +def test_unknown_residency_admits_nothing(tmp_path): # 35 + """The render after a commit needs residency, so admission refuses without + it, even for a request that needs no new GPU.""" from infer_stack.leasing.residency import ResidencyUnknown ledger, ctl, _ = make(tmp_path) @@ -189,12 +191,11 @@ def unknown(): raise ResidencyUnknown('docker ps failed') ctl.backend.residency = unknown - with pytest.raises(PlacementError, match='residency is unknown'): + with pytest.raises(PlacementError, match='cannot be read'): ctl.acquire('x', CAT.resolve_names(['two']), wait=False, apply=False) - # Coalescing onto the LIVE, allocated deployment needs no new GPU. Its render - # still needs residency, so stage it without rendering residency-dependent state. overlay = ledger.plan_acquire(CAT.resolve_names(['one'])) - assert ctl._admit(overlay, None) == ({}, []) + allocations, reasons = ctl._admit(overlay, None) + assert allocations == {} and reasons # -- P5: allocations and renew ---------------------------------------------------------- @@ -317,7 +318,7 @@ def test_declined_approval_of_a_coalescing_acquire_changes_nothing(tmp_path): first = ctl.acquire('x', cat.resolve_names(['a']), wait=False) gid = first.deployments[0].id served = dict(ledger.get_deployment(gid).served) - registry = ctl.backend._registry_file.read_text() if ctl.backend._registry_file.exists() else None + registry = ctl.backend.gateway._registry_file.read_text() if ctl.backend.gateway._registry_file.exists() else None def decline(planned): raise ConvergeAborted('no') @@ -327,7 +328,7 @@ def decline(planned): ctl.acquire('y', cat.resolve_names(['b']), wait=False) # would add alias b assert len(ledger.status()[0]) == 1 assert ledger.get_deployment(gid).served == served - now = ctl.backend._registry_file.read_text() if ctl.backend._registry_file.exists() else None + now = ctl.backend.gateway._registry_file.read_text() if ctl.backend.gateway._registry_file.exists() else None assert now == registry diff --git a/tests/test_leasing_compose.py b/tests/test_leasing_compose.py index b7426f59..8231dc74 100644 --- a/tests/test_leasing_compose.py +++ b/tests/test_leasing_compose.py @@ -929,7 +929,7 @@ def test_litellm_config_hash_label_tracks_model_list(tmp_path): (spec unchanged), so the new alias never became routable. Stamping the config hash onto a label makes converge recreate litellm on a config change. """ - from infer_stack.leasing.compose import CONFIG_HASH_LABEL + from infer_stack.leasing.gateway import CONFIG_HASH_LABEL def label(deployment): rc = render_compose( @@ -1108,11 +1108,20 @@ def test_acquire_rolls_back_lease_on_decline(tmp_path): from infer_stack.leasing import EndpointRequest, LeaseState, vllm_structural from infer_stack.leasing.backend import ConvergeAborted - class DeclineBackend: + from infer_stack.leasing.backend import SimpleAdmission + + class DeclineBackend(SimpleAdmission): + """The operator declines the diff, which admission asks at preview.""" + def observe(self): return set() - def converge(self, desired): + def preview(self, desired, placement=None, *, approve=False): + if approve: + raise ConvergeAborted('declined') + return super().preview(desired, placement) + + def converge(self, desired, *, apply=True, placement=None): raise ConvergeAborted('declined') led = Ledger(SqliteStore(tmp_path / 'ledger.db')) @@ -1841,3 +1850,23 @@ def test_default_docker_run_can_redirect_stderr_lines(): out = _default_docker_run(['sh', '-c', 'echo out; echo err1 >&2; echo err2 >&2'], timeout=10, stderr_lines=seen.append) assert out.strip() == 'out' and seen == ['err1', 'err2'] + + +def test_open_webui_runs_as_its_data_directorys_owner(tmp_path): + """As root it left files only root could delete (the data root could not + be removed without sudo); now it runs as the directory's owner, with a + managed session key and its static assets inside its data.""" + import os + + be = ComposeBackend( + state_dir=tmp_path / 'state', inventory=simulate_inventory('1x80'), + run=FakeDocker(), http=FakeHttp(tmp_path / 'state'), litellm=True, ui=True, + images={**IMAGES, 'open_webui': 'owui:test'}, ports=PORTS, + state=dict(STATE, open_webui=str(tmp_path / 'open-webui')), + ) + be.converge([], apply=False) + svc = yaml.safe_load(be.compose_file.read_text())['services']['open-webui'] + assert svc['user'] == f'{os.getuid()}:{tmp_path.stat().st_gid}' + assert svc['environment']['STATIC_DIR'] == '/app/backend/data/static' + assert svc['environment']['WEBUI_SECRET_KEY'] == '${WEBUI_SECRET_KEY}' + assert 'WEBUI_SECRET_KEY=' in (tmp_path / 'state' / '.env').read_text() diff --git a/tests/test_leasing_controller_lock.py b/tests/test_leasing_controller_lock.py index 264be200..74d10cb0 100644 --- a/tests/test_leasing_controller_lock.py +++ b/tests/test_leasing_controller_lock.py @@ -14,6 +14,7 @@ import pytest +from infer_stack.leasing.backend import SimpleAdmission from infer_stack.leasing import ( Controller, EndpointRequest, @@ -32,7 +33,7 @@ THREAD_TIMEOUT_S = 30 -class OverlapBackend: +class OverlapBackend(SimpleAdmission): """A converge-style backend that records max concurrent converges.""" def __init__(self) -> None: @@ -43,7 +44,7 @@ def __init__(self) -> None: self.last_errors: tuple = () self.last_assignments: dict = {} - def converge(self, desired, *, apply: bool = True) -> None: + def converge(self, desired, *, apply: bool = True, placement=None) -> None: with self._guard: self.active += 1 self.max_active = max(self.max_active, self.active) @@ -172,7 +173,7 @@ def test_created_lock_file_and_dir_are_group_writable(tmp_path): assert dmode & stat.S_ISGID, f'lock dir missing setgid: {oct(dmode)}' -class SharedOverlapBackend: +class SharedOverlapBackend(SimpleAdmission): """Like OverlapBackend but records overlap into a shared counter, so two *separate* controllers' converges can be compared.""" @@ -184,7 +185,7 @@ def __init__(self, shared: dict, guard: threading.Lock) -> None: self.last_assignments: dict = {} self._ids: set = set() - def converge(self, desired, *, apply: bool = True) -> None: + def converge(self, desired, *, apply: bool = True, placement=None) -> None: with self.guard: self.shared['active'] += 1 self.shared['max'] = max(self.shared['max'], self.shared['active']) diff --git a/tests/test_leasing_controller_queue.py b/tests/test_leasing_controller_queue.py index 833fb9df..2f05038f 100644 --- a/tests/test_leasing_controller_queue.py +++ b/tests/test_leasing_controller_queue.py @@ -19,7 +19,7 @@ SqliteStore, vllm_structural, ) -from infer_stack.leasing.backend import PlacementError, Readiness +from infer_stack.leasing.backend import PlacementError, Readiness, SimpleAdmission from infer_stack.leasing.placement import GpuPlan @@ -45,39 +45,32 @@ def factory(prefix: str) -> str: return factory -class BudgetBackend: +class BudgetBackend(SimpleAdmission): """Places up to ``budget`` deployments (1 slot each); the rest are unplaced. - A converge-style fake: ``converge`` recomputes the placed/unplaced split from - the desired set each call, so freeing demand (a release) lets a queued - deployment in on the next reconcile. + Capacity is its :meth:`plan`, which admission consults before committing + anything, so freeing demand (a release) lets a queued deployment in on + the next attempt. It allocates GPUs, so admission accounts for them. """ + allocates_gpus = True + def __init__(self, budget: int): self.budget = budget self.realized: dict[str, object] = {} # what is "running" (post-apply) - self._placed: dict[str, object] = {} # what the last render decided to run self.apply_calls = 0 - self.last_unplaced: list[str] = [] - self.last_errors: list[str] = [] - self.last_assignments: dict[str, list[int]] = {} - def converge(self, desired, apply: bool = True) -> None: + def plan(self, desired, placement=None): placed = list(desired)[: self.budget] - unplaced = list(desired)[self.budget :] - # Render always records the placed set; bringing it "up" is apply()'s job - # (the render/apply split). Legacy apply=True still realizes in one shot. - self._placed = {g.id: g for g in placed} - if apply: - self.realized = dict(self._placed) - self.last_assignments = {g.id: [i] for i, g in enumerate(placed)} - self.last_unplaced = [g.id for g in unplaced] - self.last_errors = [f'{g.id}: no free GPU' for g in unplaced] + return GpuPlan( + assignments={g.id: [i] for i, g in enumerate(placed)}, + errors=[f'{g.id}: no free GPU' for g in list(desired)[self.budget:]], + ) def apply(self) -> None: """Bring the last-rendered set 'up' (idempotent).""" self.apply_calls += 1 - self.realized = dict(self._placed) + super().apply() def observe(self) -> set[str]: return set(self.realized) @@ -85,11 +78,11 @@ def observe(self) -> set[str]: def probe_ready(self, deployment, endpoint) -> Readiness: return Readiness(deployment.id in self.realized, 'ok') - def realize(self, deployment) -> None: # unused by converge backends - pass + def realize(self, deployment) -> None: + self.realized[deployment.id] = deployment def teardown(self, deployment) -> None: - pass + self.realized.pop(deployment.id, None) def vreq(endpoint, *, reclaim='stop'): @@ -122,8 +115,9 @@ def sleep(clock, ledger): ctl, backend, _, _ = _make(budget=1, sleep=sleep) ctl.acquire('alice', [vreq('A')]) # fills the single slot - with pytest.raises(PlacementError): + with pytest.raises(PlacementError) as ei: ctl.acquire('bob', [vreq('B')]) # no wait -> fail fast + assert ei.value.capacity # a free GPU would have let it in def test_acquire_queues_until_a_gpu_frees(): diff --git a/tests/test_leasing_dynamic_routing.py b/tests/test_leasing_dynamic_routing.py index 4235143c..4000dd3b 100644 --- a/tests/test_leasing_dynamic_routing.py +++ b/tests/test_leasing_dynamic_routing.py @@ -23,12 +23,12 @@ from infer_stack.hardware import simulate_inventory from infer_stack.leasing import ComposeBackend, render_compose -from infer_stack.leasing.compose import ( +from infer_stack.leasing.compose import vllm_service_name +from infer_stack.leasing.gateway import ( POSTGRES_SERVICE, ROUTE_ID_PREFIX, _litellm_routes, _route_id, - vllm_service_name, ) from infer_stack.leasing.models import Deployment, DeploymentState @@ -238,7 +238,7 @@ def test_reconcile_replaces_same_id_route_with_wrong_semantics(tmp_path): # Simulate DB drift / an earlier endpoint definition with the same stable id. gw.models[rid]['litellm_params']['model'] = 'ollama/wrong-tag' before = len(gw.calls) - assert be._reconcile_routes() is True + assert be.gateway._reconcile_routes() is True assert gw.models[rid]['litellm_params']['model'] == 'openai/smol' assert gw.calls[before:] == [('delete', rid), ('new', rid)] @@ -300,12 +300,12 @@ def test_reconcile_leaves_unmanaged_models_alone(tmp_path): def test_db_password_is_persisted_and_reused(tmp_path): be = make_backend(tmp_path, RecordingGateway()) - pw1 = be.db_password() + pw1 = be.gateway.db_password() assert pw1 # rewritten only if missing -> stable across calls and a fresh backend - assert be.db_password() == pw1 + assert be.gateway.db_password() == pw1 be2 = make_backend(tmp_path, RecordingGateway()) - assert be2.db_password() == pw1 + assert be2.gateway.db_password() == pw1 # -- no blip on the UPSTREAMS too (the readiness-killing churn) ------------- @@ -380,9 +380,9 @@ def post(self, url, **kw): monkeypatch.setattr( _log.logger, 'warning', lambda *a, **k: warnings.append((a, k)) ) - be._post_route('/model/delete', {'id': 'isr-x'}, 'isr-x', ok_if_missing=True) + be.gateway._post_route('/model/delete', {'id': 'isr-x'}, 'isr-x', ok_if_missing=True) assert warnings == [] # already gone -> no warning - be._post_route('/model/delete', {'id': 'isr-x'}, 'isr-x') + be.gateway._post_route('/model/delete', {'id': 'isr-x'}, 'isr-x') assert warnings # same response without the flag -> warns @@ -431,9 +431,9 @@ def test_reconcile_reports_success_only_when_routes_verify(tmp_path): gw = RecordingGateway() be = make_backend(tmp_path, gw) be.converge([a], apply=False) - assert be._reconcile_routes() is True + assert be.gateway._reconcile_routes() is True assert _managed(gw) == {_route_id(a.id, 'smol')} - assert be._reconcile_routes() is True # idempotent: nothing to change, still verified + assert be.gateway._reconcile_routes() is True # idempotent: nothing to change, still verified def test_reconcile_against_unreachable_gateway_is_bounded_by_its_deadline(tmp_path): @@ -443,7 +443,7 @@ def test_reconcile_against_unreachable_gateway_is_bounded_by_its_deadline(tmp_pa gw = UnreachableGateway(time) be = _timed_backend(tmp_path, gw, time) be.converge([dep('grp-aaaaaa', served='smol')], apply=False) - assert be._reconcile_routes(deadline_s=25.0) is False + assert be.gateway._reconcile_routes(deadline_s=25.0) is False assert time.now <= 25.0 assert gw.gets >= 2 # it did retry within the budget @@ -458,7 +458,7 @@ def post(self, url, **kw): time = FakeTime() be = _timed_backend(tmp_path, RejectingGateway(), time) be.converge([dep('grp-aaaaaa', served='smol')], apply=False) - assert be._reconcile_routes(deadline_s=20.0) is False + assert be.gateway._reconcile_routes(deadline_s=20.0) is False assert time.now <= 20.0 @@ -477,7 +477,7 @@ def post(self, url, **kw): be = _timed_backend(tmp_path, gw, time) a = dep('grp-aaaaaa', served='smol') be.converge([a], apply=False) - assert be._reconcile_routes(deadline_s=20.0) is True + assert be.gateway._reconcile_routes(deadline_s=20.0) is True assert _managed(gw) == {_route_id(a.id, 'smol')} @@ -496,7 +496,7 @@ def post(self, url, **kw): be = _timed_backend(tmp_path, SlowGateway(), time) be.converge([dep('grp-aaaaaa', served='smol')], apply=False) - be._reconcile_routes(deadline_s=10.0) + be.gateway._reconcile_routes(deadline_s=10.0) assert seen and all(t <= 10.0 - 4.0 for t in seen) # never the default 30 s @@ -517,7 +517,7 @@ def test_apply_returns_true_once_routes_verify(tmp_path): def test_apply_uses_the_short_deadline_when_the_gateway_was_already_up(tmp_path): - from infer_stack.leasing.compose import ( + from infer_stack.leasing.gateway import ( ROUTE_RECONCILE_BOOTSTRAP_S, ROUTE_RECONCILE_STEADY_S, ) diff --git a/tests/test_leasing_kubeai.py b/tests/test_leasing_kubeai.py index 899c7875..31d7b03f 100644 --- a/tests/test_leasing_kubeai.py +++ b/tests/test_leasing_kubeai.py @@ -70,6 +70,11 @@ def __call__(self, args: list[str]) -> str: if doc: self.applied[doc['metadata']['name']] = doc return '' + if verb == 'get' and args[4] == 'pods': + # KubeAI runs one pod per applied Model; here it is up at once. + return json.dumps({'items': [ + _pod(f'model-{name}-0', doc['metadata']['labels']['infer-stack/deployment']) + for name, doc in self.applied.items()]}) if verb == 'get': items = list(self.applied.values()) if '-l' in args: # emulate the label selector @@ -185,7 +190,7 @@ def test_render_attention_backend_reaches_cr_env(): @pytest.mark.parametrize('launch', [ {'serve_recipe': 'hyperqwen-3090-single'}, # a legacy entry, translated {'command': ['single']}, - {'env': {'SPEC': 'mtp'}}, + {'mounts': {'/cache': 'x/cache'}}, ]) def test_render_refuses_a_custom_container_launch(launch): rendered = render_models( @@ -386,19 +391,54 @@ def test_acquire_release_lifecycle(tmp_path): assert rel.reconcile is not None -def test_acquire_missing_profile_rolls_back(tmp_path): - """An unrenderable deployment behaves exactly like a placement failure: - the acquire fails loudly and the lease is rolled back.""" +def test_acquire_missing_profile_commits_nothing(tmp_path): + """An unrenderable deployment fails admission, as an unplaceable one does + on compose: the preview refuses it before any lease is written or any + kubectl apply runs.""" from infer_stack.leasing.backend import PlacementError ctl, be, kubectl = make_controller(tmp_path) with pytest.raises(PlacementError, match='resource profile'): ctl.acquire('alice', [_req('qwen', profile=None)], wait=False) leases, deployments = ctl.ledger.status() - assert [le.state for le in leases] == [LeaseState.RELEASED] + assert leases == [] + assert not any('apply' in call for call in kubectl.calls) assert kubectl.applied == {} +def test_kubeai_takes_the_admission_path(tmp_path): + """One acquire path: KubeAI admits by preview and commits no GPUs.""" + ctl, be, kubectl = make_controller(tmp_path) + out = ctl.acquire('alice', [_req('qwen')], wait=False) + (deployment,) = out.deployments + assert deployment.assigned_gpus == [] # committed, and empty + assert ctl._unresolved_allocations() == [] + + +def test_status_shows_no_gpu_for_a_cluster_scheduled_deployment(tmp_path): + """The committed allocation is empty, which must not read as "cpu".""" + from infer_stack.cli.commands_leasing import _gpu_label, _placement_view + + ctl, be, kubectl = make_controller(tmp_path) + out = ctl.acquire('alice', [_req('qwen')], wait=False) + observed, assignments = _placement_view(ctl) + (deployment,) = out.deployments + assert _gpu_label(deployment.id, observed, assignments) == '-' + + +def test_queued_acquire_of_an_unrenderable_endpoint_fails_at_once(tmp_path): + """The cluster is the queue, but it cannot queue what it cannot render.""" + from infer_stack.leasing.backend import PlacementError + + ctl, be, kubectl = make_controller(tmp_path) + slept = [] + ctl.sleep = slept.append + with pytest.raises(PlacementError, match='resource profile'): + ctl.acquire('alice', [_req('qwen', profile=None)], wait=False, + wait_for_placement=True, timeout=600) + assert slept == [] + + def test_keep_warm_stays_resident_after_release(tmp_path): ctl, be, kubectl = make_controller(tmp_path) out = ctl.acquire( @@ -477,7 +517,7 @@ def test_doctor_all_green(tmp_path): names = [c[0] for c in checks] assert any('cluster' in n for n in names) assert any('CRD' in n for n in names) - assert any('gateway' in n for n in names) + assert any("KubeAI's API" in n for n in names) def test_doctor_stops_at_first_missing_dependency(tmp_path): @@ -499,7 +539,7 @@ def get(self, url, **kw): kubectl = FakeKubectl() be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=DownHttp()) checks = be.doctor() - gateway = [c for c in checks if c[0].startswith('gateway')][0] + gateway = [c for c in checks if c[0].startswith("KubeAI's API")][0] assert gateway[1] is False assert 'port-forward' in gateway[2] @@ -532,3 +572,574 @@ def test_doctor_cli_backends_without_preflight(monkeypatch, capsys): ) assert DoctorCLI.main(argv=[]) == 0 assert 'nothing to verify' in capsys.readouterr().out + + +# -- the LiteLLM gateway in front of the cluster (plan step K1) --------------- +# +# Real-cluster evidence: dev/e2e_tests/kubeai_k3s.sh. Without the gateway a +# card's alias got HTTP 404 from KubeAI; with it the same request answered. + + +UPSTREAM = 'http://10.43.0.9/openai/v1' + + +class GatewayHttp(FakeHttp): + """The LiteLLM gateway on 127.0.0.1, routing by its rendered model_list.""" + + def __init__(self, kubectl, gateway_dir): + super().__init__(kubectl) + self.gateway_dir = Path(gateway_dir) + + def _routes(self): + cfg = self.gateway_dir / 'litellm_config.yaml' + doc = yaml.safe_load(cfg.read_text()) if cfg.exists() else {} + return {e['model_name']: e['litellm_params'] for e in doc.get('model_list') or []} + + def get(self, url, **kw): + if url.startswith('http://127.0.0.1:') and url.endswith('/models'): + return self._Resp(200, {'data': [{'id': a} for a in self._routes()]}) + return super().get(url, **kw) + + def post(self, url, **kw): + if url.startswith('http://127.0.0.1:'): + route = self._routes().get((kw.get('json') or {}).get('model')) + if route is None or route['api_base'] != UPSTREAM: + return self._Resp(404, {'detail': 'no such alias'}) + kw = {**kw, 'json': {**kw['json'], 'model': route['model'].split('/', 1)[1]}} + return super().post(url, **kw) + + +def make_gateway_backend(tmp_path): + from infer_stack.leasing.compose import ComposeBackend + from test_leasing_compose import FakeDocker + + kubectl = FakeKubectl() + gateway = ComposeBackend( + state_dir=tmp_path / 'gateway', inventory={'gpu_count': 0, 'gpus': []}, + run=FakeDocker(), http=GatewayHttp(kubectl, tmp_path / 'gateway'), + project='infer-stack-gateway', litellm=True, ui=False, + images={'litellm': 'litellm:test'}, + ) + be = KubeaiBackend(state_dir=tmp_path / 'kubeai', run=kubectl, + http=gateway.http, gateway=gateway, gateway_upstream=UPSTREAM) + return be, kubectl + + +def test_gateway_routes_each_alias_to_its_model(tmp_path): + be, _ = make_gateway_backend(tmp_path) + dep = vllm('grp-a', served='Qwen/Qwen2.5-0.5B') + dep.served = {'Qwen/Qwen2.5-0.5B': {'served_model_name': 'Qwen/Qwen2.5-0.5B', + 'protocol': 'chat'}} + be.converge([dep]) + route = GatewayHttp._routes(be.gateway.http)['Qwen/Qwen2.5-0.5B'] + assert route == {'model': 'openai/qwen-qwen2-5-0-5b', 'api_base': UPSTREAM, + 'api_key': 'EMPTY'} + + +def test_with_a_gateway_clients_use_the_alias_and_the_managed_key(tmp_path): + be, _ = make_gateway_backend(tmp_path) + dep = vllm('grp-a', served='Qwen/Qwen2.5-0.5B') + dep.served = {'tiny': {'served_model_name': 'Qwen/Qwen2.5-0.5B', 'protocol': 'chat'}} + be.converge([dep]) + info = be.access(['tiny']) + assert info['request_names'] == {'tiny': 'tiny'} # the alias, as on compose + assert info['base_url'].startswith('http://127.0.0.1:') + assert info['api_key'] == be.gateway.master_key() + # Ready is judged the way a client sees it: the alias, through the gateway. + assert be.probe_ready(dep, 'tiny').ready + + +def test_without_a_gateway_clients_still_see_kubeai_directly(tmp_path): + be, _ = make_backend(tmp_path) + be.converge([vllm('grp-a', served='qwen-32b')]) + assert be.litellm is False + assert be.access(['grp-a'])['request_names'] == {'grp-a': 'qwen-32b'} + + +def test_rotating_the_key_recreates_the_gateway_in_front_of_the_cluster(tmp_path): + from infer_stack.leasing.residency import FINGERPRINT_LABEL + + be, _ = make_gateway_backend(tmp_path) + ledger = Ledger(SqliteStore(str(tmp_path / 'ledger.db'))) + ctl = Controller(ledger, be) + out = ctl.acquire('alice', [_req('qwen', profile='cpu')], wait=False) + ctl.release(out.lease.id) + old = be.master_key() + + def fingerprint(): + doc = yaml.safe_load(be.gateway.compose_file.read_text()) + return doc['services']['litellm']['labels'][FINGERPRINT_LABEL] + + before = fingerprint() + ctl.rotate_gateway_key() + assert be.master_key() != old + assert fingerprint() != before + + +# -- strict residency from pods, and crash diagnosis (plan steps K2, K4) ------ +# +# Pod shapes follow real `kubectl get pods -o json` output captured on k3s + +# KubeAI 0.23.4: KubeAI copies the Model's labels onto its pods, and a pod +# whose engine rejected a flag reads Running, restartCount 1, lastState +# terminated {exitCode: 2, reason: Error}. + + +def _pod(name, gid, *, state=None, restarts=0, last=None, ready=False, + conditions=None, statuses=True): + status = {'phase': 'Running', + 'conditions': conditions or [{'type': 'Ready', + 'status': 'True' if ready else 'False'}]} + if statuses: + status['containerStatuses'] = [{ + 'name': 'server', 'restartCount': restarts, + 'state': state or {'running': {'startedAt': 't'}}, + 'lastState': {'terminated': last} if last else {}, + }] + return {'metadata': {'name': name, 'labels': { + 'infer-stack/deployment': gid, 'infer-stack/managed': 'true', + 'model': name.split('-')[1], 'app': 'model'}}, + 'status': status} + + +class PodKubectl(FakeKubectl): + """FakeKubectl plus pods and per-pod logs (current, and --previous).""" + + def __init__(self): + super().__init__() + self.pods: list[dict] = [] + self.logs: dict[tuple[str, bool], str] = {} + self.fail_pods = False + + def __call__(self, args): + if len(args) > 4 and args[3] == 'get' and args[4] == 'pods': + self.calls.append(args) + if self.fail_pods: + raise RuntimeError('connection refused') + return json.dumps({'items': self.pods}) + if len(args) > 3 and args[3] == 'logs': + self.calls.append(args) + return self.logs.get((args[4], '--previous' in args), '') + return super().__call__(args) + + +def make_pod_backend(tmp_path): + kubectl = PodKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + return be, kubectl + + +def test_residency_reads_pods_and_never_guesses(tmp_path): + from infer_stack.leasing.residency import ResidencyUnknown + + be, kubectl = make_pod_backend(tmp_path) + kubectl.pods = [_pod('model-qwen-1', 'grp-a', ready=True)] + pod = be.residency().resident('grp-a') + assert (pod.state, pod.health, pod.labelled) == ('running', 'healthy', True) + kubectl.fail_pods = True + with pytest.raises(ResidencyUnknown): + be.residency() # a failed look is never "nothing running" + + +def test_an_unschedulable_pod_says_why(tmp_path): + be, kubectl = make_pod_backend(tmp_path) + kubectl.pods = [_pod('model-big-1', 'grp-b', statuses=False, conditions=[ + {'type': 'PodScheduled', 'status': 'False', 'reason': 'Unschedulable'}])] + (pod,) = be.residency().containers('grp-b') + assert (pod.state, pod.reason, pod.warm) == ('created', 'Unschedulable', False) + + +def test_a_rejected_flag_fails_the_wait_with_the_engines_words(tmp_path): + be, kubectl = make_pod_backend(tmp_path) + dep = vllm('grp-x', served='broken') + be.converge([dep]) + kubectl.pods = [_pod('model-broken-1', 'grp-x', restarts=1, + last={'exitCode': 2, 'reason': 'Error'})] + kubectl.logs[('model-broken-1', True)] = ( + 'usage: ...\napi_server.py: error: unrecognized arguments: --bogus\n') + be.http.post = lambda url, **kw: FakeHttp._Resp(503, {'detail': 'not ready'}) + probe = be.probe_ready(dep, 'grp-x') + assert probe.fatal and not probe.ready + assert 'unrecognized arguments: --bogus' in probe.detail # the previous run's log + assert 'rejected a command-line flag' in probe.detail + + +def test_a_slow_start_is_not_a_failure(tmp_path): + be, kubectl = make_pod_backend(tmp_path) + dep = vllm('grp-y', served='slow') + be.converge([dep]) + kubectl.pods = [_pod('model-slow-1', 'grp-y')] # running, not ready + be.http.post = lambda url, **kw: FakeHttp._Resp(503, {'detail': 'not ready'}) + probe = be.probe_ready(dep, 'grp-y') + assert not probe.ready and not probe.fatal + + +def test_gc_orphans_on_kubeai_finds_nothing_to_remove(tmp_path, monkeypatch, capsys): + """Residency selects pods by infer-stack's label, so none is an orphan: + the command runs on kubeai and never deletes anything.""" + from infer_stack.cli import commands_leasing + + be, kubectl = make_pod_backend(tmp_path) + kubectl.pods = [_pod('model-a-0', 'grp-1', ready=True)] + ctl = Controller(Ledger(SqliteStore(str(tmp_path / 'l.db'))), be) + monkeypatch.setattr(commands_leasing, '_open_controller', lambda *a, **k: ctl) + assert commands_leasing.GcCLI.main(argv=['--orphans', '--yes']) == 0 + assert 'removed 0' in capsys.readouterr().out + assert not any('delete' in call or 'rm' in call for call in kubectl.calls) + + +def test_runtime_env_reaches_the_model_like_it_reaches_a_container(): + rendered = render_models( + [vllm('grp-e', served='q', env={'MODE': 'fast', 'CTX': '{max_model_len}', 'ON': True}, + attention_backend='TORCH_SDPA')], + namespace='kubeai', default_resource_profile='cpu', + ) + (doc,) = rendered.docs + assert doc['spec']['env'] == {'MODE': 'fast', 'CTX': '4096', 'ON': 'true', + 'VLLM_ATTENTION_BACKEND': 'TORCH_SDPA'} + + +# -- the front door: same settings, same approval, same routes (P3) ----------- + + +def make_front_door_backend(tmp_path, **gateway_kw): + from infer_stack.leasing.compose import ComposeBackend + from test_leasing_compose import FakeDocker + + kubectl = FakeKubectl() + gateway = ComposeBackend( + state_dir=tmp_path / 'gateway', inventory={'gpu_count': 0, 'gpus': []}, + run=FakeDocker(), http=GatewayHttp(kubectl, tmp_path / 'gateway'), + project='infer-stack-gateway', litellm=True, + images={'litellm': 'litellm:test', 'open-webui': 'owui:test', + 'nginx': 'nginx:test', 'postgres': 'pg:test'}, + **gateway_kw, + ) + be = KubeaiBackend(state_dir=tmp_path / 'kubeai', run=kubectl, + http=gateway.http, gateway=gateway, gateway_upstream=UPSTREAM) + return be, kubectl + + +def _services(be): + doc = yaml.safe_load(be.gateway.compose_file.read_text()) + return set(doc['services']) + + +def test_the_gateway_project_takes_the_front_door_settings(tmp_path): + be, _ = make_front_door_backend(tmp_path, ui=True, reverse_proxy=True) + from infer_stack.leasing.gateway import NGINX_SERVICE, OPEN_WEBUI_SERVICE + + be.converge([vllm('grp-a', served='tiny')], apply=False) + assert {'litellm', OPEN_WEBUI_SERVICE, NGINX_SERVICE} <= _services(be) + + +def test_dynamic_routing_gives_each_dedicated_deployment_its_own_model(tmp_path): + be, kubectl = make_front_door_backend(tmp_path, dynamic_routing=True) + a = vllm('grp-aaaaaaaa1111', served='tiny', t=1.0) + b = vllm('grp-bbbbbbbb2222', served='tiny', t=2.0) + for dep in (a, b): + dep.served = {'tiny': {'served_model_name': 'tiny', 'protocol': 'chat'}} + be.converge([a, b], apply=False) + models = yaml.safe_load_all(be.models_file.read_text()) + assert {m['metadata']['name'] for m in models} == {'tiny-aaaaaaaa', 'tiny-bbbbbbbb'} + routes = json.loads((be.gateway.state_dir / 'litellm_routes.json').read_text()) + assert sorted(r['litellm_params']['model'] for r in routes) == [ + 'openai/tiny-aaaaaaaa', 'openai/tiny-bbbbbbbb'] + assert {r['model_name'] for r in routes} == {'tiny'} # one alias, two upstreams + assert 'postgres-litellm' in _services(be) + + +def test_catalog_endpoints_are_routed_before_they_run(tmp_path): + """The static superset, as on compose: acquiring a catalog endpoint does + not change the gateway's config, so the gateway is not recreated.""" + from infer_stack.leasing import Catalog + + be, _ = make_front_door_backend(tmp_path) + be.catalog = Catalog.from_dict({ + 'models': {'m': {'source': 'hf://org/m'}}, + 'endpoints': {'tiny': {'engine': 'vllm', 'model': 'm'}}, + }) + be.converge([], apply=False) + before = (be.gateway.state_dir / 'litellm_config.yaml').read_text() + assert 'tiny' in before + dep = vllm('grp-a', served='tiny') + dep.served = {'tiny': {'served_model_name': 'tiny', 'protocol': 'chat'}} + be.converge([dep], apply=False) + assert (be.gateway.state_dir / 'litellm_config.yaml').read_text() == before + + +def test_the_gateways_changes_are_approved_in_the_preview(tmp_path, monkeypatch): + """One approval, before the lease commits: a decline writes neither the + Models nor the gateway's route registry.""" + from infer_stack import diff_prompt as dp + from infer_stack.leasing.backend import ConvergeAborted + + be, _ = make_front_door_backend(tmp_path) + be.assume_yes = be.gateway.assume_yes = False + asked = [] + monkeypatch.setattr(dp, 'confirm_writes', lambda changed, **kw: asked.append( + sorted(p.name for p in changed)) or False) + with pytest.raises(ConvergeAborted): + be.preview([vllm('grp-a', served='tiny')], approve=True) + assert not be.models_file.exists() + assert not be.gateway.gateway._registry_file.exists() + + asked.clear() + monkeypatch.setattr(dp, 'confirm_writes', lambda changed, **kw: asked.append( + sorted(p.name for p in changed)) or True) + desired = [vllm('grp-a', served='tiny')] + be.preview(desired, approve=True) + approved = be.last_preview_digest + assert any('litellm_config.yaml' in names for names in asked) + asked.clear() + be.converge(desired, apply=False) + assert asked == [] # not asked a second time + assert be.last_planned_digest == approved # the approved-digest guard holds + + +def test_routes_list_shows_kubeai_upstreams(tmp_path, monkeypatch, capsys): + from infer_stack.cli import commands_leasing + + be, _ = make_front_door_backend(tmp_path) + dep = vllm('grp-a', served='Qwen/Qwen2.5-0.5B') + dep.served = {'tiny': {'served_model_name': 'Qwen/Qwen2.5-0.5B', 'protocol': 'chat'}} + be.converge([dep], apply=False) + ctl = Controller(Ledger(SqliteStore(str(tmp_path / 'l.db'))), be) + monkeypatch.setattr(commands_leasing, '_open_controller', lambda *a, **k: ctl) + capsys.readouterr() # the render's own output + assert commands_leasing.RoutesListCLI.main(argv=['--json']) == 0 + (row,) = json.loads(capsys.readouterr().out)['routes'] + assert row['name'] == 'tiny' and row['engine'] == 'upstream' + assert row['target'] == 'qwen-qwen2-5-0-5b' and row['upstream'] == UPSTREAM + + +def test_the_front_doors_settings_are_in_the_recovery_profile(tmp_path): + be, _ = make_front_door_backend(tmp_path, ui=True, dynamic_routing=True) + profile = be.render_profile() + assert profile['gateway']['ui'] is True and profile['gateway']['dynamic_routing'] is True + other, _ = make_front_door_backend(tmp_path / 'other') + other.use_profile(profile) + assert other.gateway.ui is True and other.dynamic_routing is True + other.use_profile({**profile, 'gateway': True}) # a profile from before + assert other.dynamic_routing is True # keeps what it has + + +# -- placement information: profiles chosen by GPU size (P4) ------------------ + +SIZED_PROFILES = { + 'small': {'nodeSelector': {'nvidia.com/gpu.product': 'FAKE-24G'}}, + 'big': {'nodeSelector': {'nvidia.com/gpu.product': 'FAKE-80G'}}, + 'generic': {'limits': {'nvidia.com/gpu': '1'}}, # the chart's kind +} + + +def _gpu_node(name, product, mib, count=2): + return {'metadata': {'name': name, 'labels': { + 'nvidia.com/gpu.product': product, 'nvidia.com/gpu.memory': str(mib)}}, + 'status': {'allocatable': {'nvidia.com/gpu': str(count)}}} + + +class SizedKubectl(FakeKubectl): + """A cluster with a 24 GiB node and an 80 GiB node, and the chart's config.""" + + def __call__(self, args): + if len(args) > 4 and args[3] == 'get' and args[4] == 'nodes': + return json.dumps({'items': [_gpu_node('a', 'FAKE-24G', 24576), + _gpu_node('b', 'FAKE-80G', 81920)]}) + if len(args) > 4 and args[3] == 'get' and args[4] == 'configmap': + system = yaml.safe_dump({'resourceProfiles': SIZED_PROFILES}) + return json.dumps({'data': {'system.yaml': system}}) + return super().__call__(args) + + +def _sized(gid, min_vram, **runtime): + dep = vllm(gid, profile=None, **runtime) + dep.spec['placement'] = {'min_vram_gib': min_vram} + return dep + + +def test_min_vram_gib_picks_the_smallest_profile_that_fits(tmp_path): + kubectl = SizedKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + be.converge([_sized('grp-a', 40), _sized('grp-b', 10)]) + profiles = {doc['metadata']['labels']['infer-stack/deployment']: doc['spec']['resourceProfile'] + for doc in kubectl.applied.values()} + assert profiles == {'grp-a': 'big:1', 'grp-b': 'small:1'} + + +def test_an_explicit_profile_wins_and_too_large_is_refused(tmp_path): + kubectl = SizedKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + explicit = _sized('grp-a', 40, resource_profile='generic') + be.converge([explicit, _sized('grp-b', 200)], apply=False) + assert be.last_unplaced == {'grp-b'} + assert any('no resource profile has GPUs that large' in e for e in be.last_errors) + (doc,) = yaml.safe_load_all(be.models_file.read_text()) + assert doc['spec']['resourceProfile'] == 'generic:1' + + +def test_sizing_reads_the_cluster_only_when_asked(tmp_path): + kubectl = SizedKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + be.converge([vllm('grp-a')], apply=False) + assert not any('nodes' in call or 'configmap' in call for call in kubectl.calls) + + +def test_catalog_suggest_on_kubeai_sizes_to_the_cluster(tmp_path, monkeypatch, capsys): + from infer_stack.cli import commands_catalog, commands_leasing + + kubectl = SizedKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + monkeypatch.setattr(commands_leasing, '_make_backend', lambda config, **kw: be) + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path / 'config')) + assert commands_catalog.CatalogSuggestCLI.main(argv=['--backend', 'kubeai']) == 0 + out, err = capsys.readouterr() + assert "largest GPU node" in err + values = yaml.safe_load(err[err.index('resourceProfiles:'):])['resourceProfiles'] + assert values['nvidia-fake-80g']['nodeSelector'] == {'nvidia.com/gpu.product': 'FAKE-80G'} + assert 'gpu_indices' not in out # no host indices on a cluster + + +def test_a_recorded_measurement_picks_the_profile(tmp_path): + """`measure --record` works on kubeai: the overlay fills min_vram_gib.""" + from infer_stack.leasing.vram import measurement_key_for_spec + + kubectl = SizedKubectl() + be = KubeaiBackend(state_dir=tmp_path, run=kubectl, http=FakeHttp(kubectl)) + dep = vllm('grp-a', profile=None) + be.measurements.record(measurement_key_for_spec(dep.spec), 50.0, endpoint='grp-a') + be.converge([dep], apply=False) + (doc,) = yaml.safe_load_all(be.models_file.read_text()) + assert doc['spec']['resourceProfile'] == 'big:1' + + +# -- the gateway inside the cluster (P5) ---------------------------------------- + + +class ClusterKubectl(FakeKubectl): + """FakeKubectl that keeps the gateway's objects apart from the Models.""" + + def __init__(self): + super().__init__() + self.gateway_objects: dict[str, dict] = {} + self.rollouts = 0 + + def __call__(self, args): + if len(args) > 4 and args[3] == 'get' and args[4] == 'nodes': + return json.dumps({'items': [{'status': {'addresses': [ + {'type': 'InternalIP', 'address': '10.0.0.7'}]}}]}) + if len(args) > 3 and args[3] == 'rollout': + self.rollouts += 1 + return '' + if len(args) > 3 and args[3] == 'apply': + path = Path(args[args.index('-f') + 1]) + docs = [d for d in yaml.safe_load_all(path.read_text()) if d] + if all(d.get('kind') != 'Model' for d in docs): + for d in docs: + self.gateway_objects[d['kind']] = d + return '' + return super().__call__(args) + + +def make_cluster_gateway_backend(tmp_path): + from infer_stack.backends.kubeai_gateway import ClusterGateway + + kubectl = ClusterKubectl() + gateway = ClusterGateway(state_dir=tmp_path / 'gw', namespace='kubeai', run=kubectl, + images={'litellm': 'litellm:test'}) + be = KubeaiBackend(state_dir=tmp_path / 'kubeai', run=kubectl, http=FakeHttp(kubectl), + gateway=gateway) + return be, kubectl + + +def test_the_in_cluster_gateway_routes_by_cluster_dns(tmp_path): + be, kubectl = make_cluster_gateway_backend(tmp_path) + dep = vllm('grp-a', served='tiny') + dep.served = {'tiny': {'served_model_name': 'tiny', 'protocol': 'chat'}} + be.converge([dep]) + config = yaml.safe_load(kubectl.gateway_objects['ConfigMap']['data']['config.yaml']) + (route,) = config['model_list'] + assert route['model_name'] == 'tiny' + assert route['litellm_params']['api_base'] == \ + 'http://kubeai.kubeai.svc.cluster.local/openai/v1' + assert kubectl.gateway_objects['Service']['spec']['type'] == 'NodePort' + assert kubectl.rollouts == 1 + assert be.access(['tiny'])['base_url'] == 'http://10.0.0.7:30442/v1' + assert be.compose_project() is None and be.front_door() is be.gateway + + +def test_the_key_is_a_secret_never_a_diff_and_rotating_it_rolls_the_pods(tmp_path): + import stat + + be, kubectl = make_cluster_gateway_backend(tmp_path) + be.converge([]) + key = be.master_key() + assert key not in be.gateway.manifests_file.read_text() + secret = be.gateway.state_dir / 'gateway-secret.yaml' + assert stat.S_IMODE(secret.stat().st_mode) == 0o600 + assert kubectl.gateway_objects['Secret']['stringData']['LITELLM_MASTER_KEY'] == key + + def key_hash(): + tmpl = kubectl.gateway_objects['Deployment']['spec']['template'] + return tmpl['metadata']['annotations']['infer-stack/key-hash'] + + before = key_hash() + be.rotate_master_key() + be.converge([]) + assert key_hash() != before + + +def test_doctor_checks_the_in_cluster_gateway_not_a_port_forward(tmp_path): + be, _ = make_cluster_gateway_backend(tmp_path) + be.gateway.gateway_accepts = lambda key, wait=0.0: True + names = [name for name, ok, _ in be.doctor()] + assert names[-1] == 'in-cluster gateway at http://10.0.0.7:30442/v1' + assert not any('port-forward' in name for name in names) + + +def test_the_recovery_profile_says_where_the_gateway_runs(tmp_path): + """Changing `kubeai_gateway` is adopted like any setting: the snapshot + decides which gateway renders, host or cluster, and switches it.""" + from infer_stack.backends.kubeai_gateway import ClusterGateway + + cluster_be, _ = make_cluster_gateway_backend(tmp_path / 'c') + host_be, _ = make_front_door_backend(tmp_path / 'h') + host_gateway = host_be.gateway + host_be.gateway_factory = lambda placement: ( + cluster_be.gateway if placement == 'cluster' else host_gateway) + host_be.use_profile(cluster_be.render_profile()) + assert isinstance(host_be.gateway, ClusterGateway) + host_be.use_profile({**cluster_be.render_profile(), + 'gateway': host_gateway.render_profile()}) + assert host_be.gateway is host_gateway + + +def test_a_render_refusal_is_not_a_capacity_problem(tmp_path): + """"Free a GPU first" is advice only for a lack of GPUs.""" + from infer_stack.leasing.backend import PlacementError + + ctl, be, kubectl = make_controller(tmp_path) + with pytest.raises(PlacementError) as ei: + ctl.acquire('alice', [_req('qwen', profile=None)], wait=False) + assert ei.value.capacity is False + + +def test_env_and_test_use_the_kubeai_gateways_key(tmp_path, monkeypatch): + """They read a hard-coded compose .env, so `test` got 401 on kubeai and + `env LITELLM_MASTER_KEY` printed nothing.""" + from infer_stack.cli import commands_leasing + + be, _ = make_front_door_backend(tmp_path) + monkeypatch.setattr(commands_leasing, '_make_backend', lambda config, **kw: be) + key = be.master_key() + env_path, base = commands_leasing._gateway_state(None) + assert env_path == be.gateway.gateway._env_path + assert commands_leasing._front_door(None) == (base.rstrip('/'), key) + + +def test_doctor_checks_the_host_gateways_docker_too(tmp_path): + """The gateway on this host is a Compose project: without Docker the + first acquire failed after a clean preflight.""" + be, _ = make_front_door_backend(tmp_path) + names = [name for name, ok, _ in be.doctor()] + assert 'gateway: docker daemon reachable' in names + assert not any('image vllm' in n or 'GPUs visible' in n for n in names) diff --git a/tests/test_leasing_make_room.py b/tests/test_leasing_make_room.py new file mode 100644 index 00000000..ec6aeb05 --- /dev/null +++ b/tests/test_leasing_make_room.py @@ -0,0 +1,112 @@ +"""A keep-warm model without a lease gives way to a model with one. + +The rule (decided 2026-09-24): an idle keep-warm deployment -- resident, but +held by no lease -- is always a candidate for eviction when a leased model +needs a resource. Compose admission enforces it when placing. Where the +runtime schedules instead (KubeAI), the probe says `needs_room` and the wait +evicts idle models, the longest idle first, one per cooldown. +""" + +from __future__ import annotations + +from infer_stack.leasing import ( + Controller, + DeploymentState, + Ledger, + MemoryBackend, + SqliteStore, +) +from infer_stack.leasing.backend import Readiness +from infer_stack.leasing.controller import ROOM_COOLDOWN_S +from test_leasing_controller import FakeClock, _id_factory, vreq + + +class OneSlotBackend(MemoryBackend): + """Room for `slots` models; a model that does not fit waits for room.""" + + def __init__(self, slots=1): + super().__init__(ready=True) + self.slots = slots + + def probe_ready(self, deployment, endpoint): + others = [gid for gid in self.realized if gid != deployment.id] + if deployment.id in self.realized and len(others) >= self.slots: + return Readiness(False, 'pod: Unschedulable', needs_room=True) + return super().probe_ready(deployment, endpoint) + + +def make(slots=1): + clock = FakeClock() + ledger = Ledger(SqliteStore(':memory:'), clock=clock, id_factory=_id_factory()) + backend = OneSlotBackend(slots) + ctl = Controller(ledger, backend, clock=clock, sleep=clock.advance) + return ctl, ledger, backend, clock + + +def state(ledger, gid): + return ledger.get_deployment(gid).state + + +def test_an_idle_keep_warm_model_gives_way_to_a_leased_one(): + ctl, ledger, backend, _ = make() + warm = ctl.acquire('a', [vreq('warm')], wait=False) + ctl.release(warm.lease.id) # keep-warm: still resident + warm_id = warm.deployments[0].id + assert state(ledger, warm_id) == DeploymentState.IDLE + + out = ctl.acquire('b', [vreq('big')], wait=True, timeout=600, interval=5) + + assert out.wait.ready + assert state(ledger, warm_id) == DeploymentState.STOPPED + assert warm_id not in backend.realized + + +def test_the_longest_idle_goes_first_and_one_at_a_time(): + ctl, ledger, backend, clock = make(slots=2) + older = ctl.acquire('a', [vreq('older')], wait=False) + ctl.release(older.lease.id) + clock.advance(60) + newer = ctl.acquire('a', [vreq('newer')], wait=False) + ctl.release(newer.lease.id) + + out = ctl.acquire('b', [vreq('big')], wait=True, timeout=600, interval=5) + + assert out.wait.ready + assert state(ledger, older.deployments[0].id) == DeploymentState.STOPPED + # One eviction freed the slot, so the more recently used model stays warm. + assert state(ledger, newer.deployments[0].id) == DeploymentState.IDLE + + +def test_evictions_wait_for_the_cooldown(): + ctl, ledger, backend, clock = make(slots=0) # nothing ever fits + for name in ('w1', 'w2'): + out = ctl.acquire('a', [vreq(name)], wait=False) + ctl.release(out.lease.id) + big = ctl.acquire('b', [vreq('big')], wait=False) + + ctl.wait_ready(big.deployments, timeout=ROOM_COOLDOWN_S - 1, interval=5) + stopped = [g for g in ledger.status()[1] if g.state == DeploymentState.STOPPED] + assert len(stopped) == 1 # not both within one cooldown + + +def test_a_leased_model_is_never_evicted_to_make_room(): + ctl, ledger, backend, _ = make() + held = ctl.acquire('a', [vreq('held')], wait=False) # still leased + out = ctl.acquire('b', [vreq('big')], wait=True, timeout=120, interval=5) + + assert not out.wait.ready # waits, then times out + assert state(ledger, held.deployments[0].id) == DeploymentState.LIVE + + +def test_an_unschedulable_kubeai_pod_asks_for_room(tmp_path): + from test_leasing_kubeai import FakeHttp, _pod, make_pod_backend, vllm + + be, kubectl = make_pod_backend(tmp_path) + dep = vllm('grp-big', served='big') + be.converge([dep]) + kubectl.pods = [_pod('model-big-1', 'grp-big', statuses=False, conditions=[ + {'type': 'PodScheduled', 'status': 'False', 'reason': 'Unschedulable'}])] + be.http.post = lambda url, **kw: FakeHttp._Resp(503, {'detail': 'not ready'}) + + probe = be.probe_ready(dep, 'grp-big') + assert probe.needs_room and not probe.fatal and not probe.ready diff --git a/tests/test_leasing_profile.py b/tests/test_leasing_profile.py index 32b864a9..4e294100 100644 --- a/tests/test_leasing_profile.py +++ b/tests/test_leasing_profile.py @@ -281,25 +281,16 @@ def fail(planned): assert ledger.publication_pending() is None -def test_a_declined_recovery_render_keeps_the_crashed_acquires_scope(tmp_path): - from infer_stack.leasing.backend import ConvergeAborted - +def test_a_stale_scope_from_pre_admission_code_is_dropped(tmp_path): + """No render needs a crashed legacy acquire's scope: its row stays + unresolved (see above), so the next change simply clears it.""" a = Catalog.from_dict(cat('alpha')) ledger, ctl = controller(tmp_path, catalog=a) ctl.gc() ledger.mark_publication_pending(apply_requested=True, placement_context={'allowed_gpus': [3]}) - ledger.acquire('x', a.resolve_names(['alpha'])) - _, ctl2 = controller(tmp_path, catalog=a, allowed_gpus=[0]) - - def decline(planned): - raise ConvergeAborted('no') - - ctl2.backend._approve_changes = decline - with pytest.raises(ConvergeAborted): - ctl2.gc() - assert ledger.publication_pending()['placement_context'] == {'allowed_gpus': [3]} - + ctl.acquire('x', a.resolve_names(['alpha']), wait=False, apply=False) + assert ledger.publication_pending()['placement_context'] is None def test_cli_catalog_edit_then_acquire_needs_no_publish_step(tmp_path, monkeypatch): from infer_stack.cli import commands_leasing as cl @@ -528,7 +519,7 @@ def test_a_publish_that_never_commits_leaves_no_append_only_state(tmp_path): ledger, ctl = controller(tmp_path, catalog=a) ctl.gc() before = ledger.profile() - registry = ctl.backend._registry_file + registry = ctl.backend.gateway._registry_file registry_before = registry.read_text() if registry.exists() else None def crash(*args, **kw): @@ -579,7 +570,7 @@ def write(path, values): handle.close() monkeypatch.setattr(cl, 'write_env_file', write) - monkeypatch.setattr(cl, '_secret_env_path', lambda: tmp_path / '.env') + monkeypatch.setattr(cl, '_secret_env_path', lambda config=None: tmp_path / '.env') assert cl.EnvCLI.main(argv=['HF_TOKEN=x']) == 0 assert held == [True] diff --git a/tests/test_leasing_route_registry.py b/tests/test_leasing_route_registry.py index a911243f..b29d8574 100644 --- a/tests/test_leasing_route_registry.py +++ b/tests/test_leasing_route_registry.py @@ -21,7 +21,7 @@ from infer_stack.hardware import simulate_inventory from infer_stack.leasing import Catalog, ComposeBackend, render_compose -from infer_stack.leasing.compose import ( +from infer_stack.leasing.gateway import ( CONFIG_HASH_LABEL, LITELLM_CONFIG_FILENAME, LITELLM_REGISTRY_FILENAME, diff --git a/tests/test_leasing_secrets.py b/tests/test_leasing_secrets.py index e8d1dc3b..f5a730c0 100644 --- a/tests/test_leasing_secrets.py +++ b/tests/test_leasing_secrets.py @@ -14,8 +14,11 @@ from infer_stack.env_utils import parse_env_file from infer_stack.hardware import simulate_inventory from infer_stack.leasing import Controller, Ledger, SqliteStore -from infer_stack.leasing.compose import ( - API_KEY_ENV, SALT_KEY_ENV, ComposeBackend, set_master_key, +from infer_stack.leasing.compose import ComposeBackend +from infer_stack.leasing.gateway import ( + API_KEY_ENV, + SALT_KEY_ENV, + set_master_key, ) from infer_stack.leasing.profile import ProfileMismatch from infer_stack.leasing.residency import FINGERPRINT_LABEL @@ -52,7 +55,7 @@ def make(tmp_path): def env(ctl): - return parse_env_file(ctl.backend._env_path) + return parse_env_file(ctl.backend.gateway._env_path) def gateway(ctl): diff --git a/tests/test_leasing_serialised_publication.py b/tests/test_leasing_serialised_publication.py index fb3a378a..945b8a4c 100644 --- a/tests/test_leasing_serialised_publication.py +++ b/tests/test_leasing_serialised_publication.py @@ -25,7 +25,7 @@ SqliteStore, vllm_structural, ) -from infer_stack.leasing.backend import Readiness +from infer_stack.leasing.backend import Readiness, SimpleAdmission @@ -46,7 +46,7 @@ def _vreq(endpoint: str) -> EndpointRequest: ) -class SharedStackBackend: +class SharedStackBackend(SimpleAdmission): """Models the one shared compose project across separate controllers. ``converge(apply=False)`` (render) writes the desired union to shared @@ -63,7 +63,7 @@ def __init__(self, shared: dict, guard: threading.Lock, apply_sleep: float = 0.0 self.last_errors: list[str] = [] self.last_assignments: dict[str, list[int]] = {} - def converge(self, desired, *, apply: bool = True) -> None: + def converge(self, desired, *, apply: bool = True, placement=None) -> None: ids = {g.id for g in desired} with self.guard: self.shared['rendered'] = set(ids) @@ -440,20 +440,38 @@ def test_no_apply_rollback_never_applies_over_an_older_pending_apply(tmp_path): class _NoRoomBackend(SharedStackBackend): - """Converge fake that can never place an endpoint named ``big``.""" + """Can never place an endpoint named ``big``; preview and render agree.""" - def converge(self, desired, *, apply: bool = True) -> None: - placeable = [g for g in desired if 'big' not in g.served] - self.last_unplaced = [g.id for g in desired if 'big' in g.served] - super().converge(placeable, apply=apply) + def refuse(self, desired): + return {g.id: 'no room' for g in desired if 'big' in g.served} + def converge(self, desired, *, apply: bool = True, placement=None) -> None: + refused = self.refuse(desired) + self.last_unplaced = list(refused) + super().converge([g for g in desired if g.id not in refused], apply=apply) -def test_placement_rollback_evicts_and_rerenders(tmp_path): - """Regression: a failed placement must fully roll back — lease released, the - never-ran deployments evicted (not left idle-keep-warm, which would pin them - in the desired set), the placed sibling removed from the on-disk render, and - the publication marker cleared.""" - from infer_stack.leasing import DeploymentState, LeaseState + +class _RenderRefusesBackend(_NoRoomBackend): + """Its preview admits ``big``; its render then refuses it. + + The one way an admitted acquire still fails after its lease is committed + (the runtime changed between preview and render), so the rollback tests + use it. + """ + + def refuse(self, desired): + return {} + + def converge(self, desired, *, apply: bool = True, placement=None) -> None: + refused = [g.id for g in desired if 'big' in g.served] + SharedStackBackend.converge( + self, [g for g in desired if g.id not in refused], apply=apply) + self.last_unplaced = refused + + +def test_a_refused_acquire_commits_nothing(tmp_path): + """Admission refuses before anything is written: no lease, no deployment, + nothing rendered, no marker.""" from infer_stack.leasing.backend import PlacementError db = str(tmp_path / 'ledger.db') @@ -461,13 +479,34 @@ def test_placement_rollback_evicts_and_rerenders(tmp_path): ledger = Ledger(SqliteStore(db)) ctl = Controller(ledger, _NoRoomBackend(shared, threading.Lock())) + with pytest.raises(PlacementError, match='no room'): + ctl.acquire('alice', [_vreq('ok'), _vreq('big')], wait=False) + + assert ledger.status() == ([], []) + assert shared['rendered'] == set() + assert shared['realized'] == set() + assert ledger.publication_pending() is None + + +def test_a_render_refusal_after_commit_rolls_back_and_rerenders(tmp_path): + """Regression: a render that refuses what admission admitted must fully + roll back: lease released, the never-ran deployments evicted (not left + idle keep-warm, which would pin them in the desired set), the placed + sibling removed from the render, and the publication marker cleared.""" + from infer_stack.leasing import DeploymentState, LeaseState + from infer_stack.leasing.backend import PlacementError + + db = str(tmp_path / 'ledger.db') + shared = {'rendered': set(), 'realized': set(), 'apply_calls': 0} + ledger = Ledger(SqliteStore(db)) + ctl = Controller(ledger, _RenderRefusesBackend(shared, threading.Lock())) + with pytest.raises(PlacementError): ctl.acquire('alice', [_vreq('ok'), _vreq('big')], wait=False) leases, deployments = ledger.status() assert [le.state for le in leases] == [LeaseState.RELEASED] assert {g.state for g in deployments} == {DeploymentState.STOPPED} - # 'ok' was rendered by the failed acquire; the rollback re-render removed it. assert shared['rendered'] == set() assert shared['realized'] == set() assert ledger.publication_pending() is None @@ -475,8 +514,8 @@ def test_placement_rollback_evicts_and_rerenders(tmp_path): def test_rollback_keeps_coalesced_warm_deployment_resident(tmp_path): """A failed acquire that coalesced onto a pre-existing warm (idle keep-warm) - deployment must roll that deployment back to IDLE — not evict the resident - model someone else may still want warm.""" + deployment must leave it IDLE — not evict the resident model someone else + may still want warm.""" from infer_stack.leasing import DeploymentState from infer_stack.leasing.backend import PlacementError @@ -514,7 +553,7 @@ def __init__(self, *args, **kwargs): self.converge_calls = 0 self.declined = False - def converge(self, desired, *, apply: bool = True) -> None: + def converge(self, desired, *, apply: bool = True, placement=None) -> None: self.converge_calls += 1 if desired and not self.declined: self.declined = True @@ -558,13 +597,13 @@ def _warm_then_failed_acquire(tmp_path, residency): shared = _shared() - class Backend(_NoRoomBackend): + class Backend(_RenderRefusesBackend): pass - Backend.residency = lambda self: residency(self) ledger, ctl = _fresh(str(tmp_path / 'ledger.db'), shared, Backend) warm = ctl.acquire('alice', [_vreq('m')], wait=False) ctl.release(warm.lease.id) + Backend.residency = lambda self: residency(self) shared['realized'] = set() # observe() now says "nothing" (as on a docker error) with pytest.raises(PlacementError): ctl.acquire('bob', [_vreq('m'), _vreq('big')], wait=False) @@ -743,16 +782,13 @@ def test_release_all_releases_every_active_lease_in_one_apply(tmp_path): # -- transitional: rollback under unknown residency keeps a phantom warm candidate -- -def test_transitional_unknown_residency_rollback_keeps_an_idle_candidate(tmp_path): - """Behaviour of backends WITHOUT admission mode (no preview), kept by design. +def test_unknown_residency_admits_nothing(tmp_path): + """Without a strict view of what runs, nothing is admitted or written. - A brand-new acquire fails; rollback cannot read residency, so it refuses to - evict (the safe direction). On such backends the desired set still contains - that IDLE deployment, so a later apply starts it with no lease behind it. - Admission-mode backends (Compose) never commit a failed acquire and never - start idle deployments: see tests/test_leasing_admission.py. + (Before one acquire path, a backend without admission committed the + lease, failed, and left an IDLE candidate a later apply started with no + lease behind it.) """ - from infer_stack.leasing import DeploymentState from infer_stack.leasing.backend import PlacementError from infer_stack.leasing.residency import ResidencyUnknown @@ -763,9 +799,8 @@ def residency(self): raise ResidencyUnknown('docker ps failed') ledger, ctl = _fresh(str(tmp_path / 'ledger.db'), shared, Backend) - with pytest.raises(PlacementError): - ctl.acquire('alice', [_vreq('fresh'), _vreq('big')], wait=False) - fresh = next(g for g in ledger.status()[1] if 'fresh' in g.served) - assert fresh.state == DeploymentState.IDLE # not evicted - ctl.apply_now() - assert fresh.id in shared['realized'] # started without a lease (P9 fixes) + with pytest.raises(PlacementError, match='cannot be read'): + ctl.acquire('alice', [_vreq('fresh')], wait=False) + assert ledger.status() == ([], []) + assert ledger.publication_pending() is None + assert shared['realized'] == set() diff --git a/tests/test_leasing_startup_failure.py b/tests/test_leasing_startup_failure.py index 88fbc6b8..9f0068e4 100644 --- a/tests/test_leasing_startup_failure.py +++ b/tests/test_leasing_startup_failure.py @@ -150,6 +150,8 @@ def test_an_unrecoverable_error_is_fatal_on_the_very_first_crash(tmp_path): (CRASH_LOG, 'fatal'), ('ValueError: quantization fp8 is not supported on this device', 'fatal'), ('torch.OutOfMemoryError: CUDA out of memory', 'fatal'), + # vLLM rejecting a flag (e.g. from runtime.extra_args) repeats every restart. + ('api_server.py: error: unrecognized arguments: --bogus', 'fatal'), (HUB_TIMEOUT_LOG, 'transient'), ('OSError: [Errno 98] Address already in use', 'transient'), ('INFO starting\nSegmentation fault', None), diff --git a/tests/test_log_filter.py b/tests/test_log_filter.py index 46e26760..d2a06a51 100644 --- a/tests/test_log_filter.py +++ b/tests/test_log_filter.py @@ -262,84 +262,51 @@ def test_eof_mid_traceback_fails_open(): assert list(compact_litellm_tracebacks(original)) == original -def test_cli_follow_compaction_helper_streams_known_trace(monkeypatch): +def _follow_logs(monkeypatch, lines, *, no_color, tty=True): + """Run `infer-stack logs -f` over a fake instance whose log is ``lines``.""" from infer_stack.cli import commands_runtime + from infer_stack.leasing import instances as instances_mod + from infer_stack.leasing.instances import Instance - source = _known_segments()[0] + config = SimpleNamespace(follow=True, raw=False, color=not no_color, tail=None, + timestamps=False, services=None) - class FakeProc: - def __init__(self): - self.stdout = iter(source) + class Backend: + def instances(self): + return [Instance('litellm', 'c1', '', 'running')] - def wait(self, timeout=None): - return 0 - - def poll(self): - return 0 + class Follower: + def __init__(self, listing, **kw): + assert [i.name for i in listing()] == ['litellm'] + self.stdout = iter(lines) def terminate(self): - raise AssertionError('completed process should not be terminated') + pass - def kill(self): - raise AssertionError('completed process should not be killed') + class Out(io.StringIO): + def isatty(self): + return tty - monkeypatch.setattr(commands_runtime.subprocess, 'Popen', lambda *a, **kw: FakeProc()) - stream = io.StringIO() + monkeypatch.setattr(commands_runtime.LogsCLI, 'cli', lambda *a, **kw: config) + monkeypatch.setattr(commands_runtime, '_day2_backend', lambda config: Backend()) + monkeypatch.setattr(commands_runtime, '_served_by_deployment', lambda: {}) + monkeypatch.setattr(instances_mod, 'LogFollower', Follower) + stream = Out() monkeypatch.setattr(commands_runtime.sys, 'stdout', stream) + assert commands_runtime.LogsCLI.main(argv=False) == 0 + return stream.getvalue() - rc = commands_runtime._run_compacted_follow(['docker', 'compose', 'logs', '-f']) - assert rc == 0 - assert 'Traceback (most recent call last):' not in stream.getvalue() - assert 'ConnectionRefusedError: [Errno 111]' in stream.getvalue() - -@pytest.mark.parametrize(('no_color', 'expected_ansi'), [ - (False, True), - (True, False), -]) -def test_cli_compacted_follow_preserves_compose_color_mode( - monkeypatch, no_color, expected_ansi -): - from infer_stack.cli import commands_runtime - - config = SimpleNamespace( - follow=True, - raw=False, - no_color=no_color, - tail=None, - timestamps=False, - services=None, - ) - - class TTY(io.StringIO): - def isatty(self): - return True - - captured = {} - monkeypatch.setattr( - commands_runtime.LogsCLI, 'cli', lambda *args, **kwargs: config - ) - monkeypatch.setattr( - commands_runtime, - '_day2_compose_base', - lambda config, purpose: [ - 'docker', 'compose', '-p', 'infer-stack', '-f', 'compose.yml' - ], - ) - def fake_follow(cmd): - captured['cmd'] = cmd - return 0 +def test_cli_follow_compacts_a_known_trace(monkeypatch): + lines = [line.replace('litellm-1 | ', 'litellm | ') for line in _known_segments()[0]] + out = _follow_logs(monkeypatch, lines, no_color=True) + assert 'Traceback (most recent call last):' not in out + assert 'ConnectionRefusedError: [Errno 111]' in out - monkeypatch.setattr(commands_runtime, '_run_compacted_follow', fake_follow) - monkeypatch.setattr(commands_runtime.sys, 'stdout', TTY()) - assert commands_runtime.LogsCLI.main(argv=False) == 0 - cmd = captured['cmd'] - logs_index = cmd.index('logs') - assert ('--ansi' in cmd) is expected_ansi - if expected_ansi: - ansi_index = cmd.index('--ansi') - assert cmd[ansi_index + 1] == 'always' - assert ansi_index < logs_index - assert ('--no-color' in cmd) is no_color +@pytest.mark.parametrize('no_color', [False, True]) +def test_cli_follow_colors_prefixes_unless_asked_not_to(monkeypatch, no_color): + out = _follow_logs(monkeypatch, ['litellm | hello\n'], no_color=no_color) + assert ('\x1b[' in out) is (not no_color) + assert 'hello' in out diff --git a/tests/test_parity.py b/tests/test_parity.py new file mode 100644 index 00000000..34087a31 --- /dev/null +++ b/tests/test_parity.py @@ -0,0 +1,449 @@ +"""Every row of docs/backend-parity.md that reads *same*, run on both backends. + +One harness builds the same stack twice: the compose backend over a fake +Docker, and the kubeai backend (with its LiteLLM gateway) over a fake +kubectl. Each test is one row of the matrix, named after it, and runs the +same scenario on both. A row whose test is here and passes on both is what +*same* means; a behaviour that only one backend can show belongs in that +backend's own tests. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from typing import Any + +import pytest +import yaml + +from infer_stack.leasing import Catalog, Controller, LeaseState, Ledger, SqliteStore +from infer_stack.leasing.backend import PlacementError +from infer_stack.leasing.residency import DEPLOYMENT_LABEL, ResidencyUnknown + +CATALOG = { + 'models': { + 'tiny': {'source': 'hf://org/tiny'}, + 'big': {'source': 'hf://org/big'}, + }, + 'endpoints': { + 'one': {'engine': 'vllm', 'model': 'tiny', 'reclaim': {'policy': 'stop'}, + 'runtime': {'max_model_len': 2048, 'extra_args': ['--seed', '7']}}, + 'wide': {'engine': 'vllm', 'model': 'big', + 'runtime': {'tensor_parallel_size': 2, 'max_model_len': 4096}}, + 'warm': {'engine': 'vllm', 'model': 'big', 'reclaim': {'policy': 'keep-warm'}, + 'runtime': {'max_model_len': 1024}}, + }, +} +CAT = Catalog.from_dict(CATALOG) +UPSTREAM = 'http://10.43.0.9/openai/v1' +CRASH_LOG = ('INFO vLLM API server version 0.25.1\n' + 'ValueError: The checkpoint has model type `example_moe_v1` but ' + 'Transformers does not recognize this architecture. If the model is ' + 'custom, set trust_remote_code=True.\n') + + +@dataclass +class Stack: + """One backend under test, and the handles a scenario needs.""" + + kind: str + ctl: Controller + backend: Any + runtime: Any # the fake Docker or the fake kubectl + front: Any # the ComposeBackend holding the gateway + + def acquire(self, *names, dedicated: bool = False, **kw): + from infer_stack.leasing import Sharing + + kw.setdefault('wait', False) + sharing = Sharing.DEDICATED if dedicated else None + requests = CAT.resolve_names(list(names), sharing=sharing) + return self.ctl.acquire('alice', requests, **kw) + + def running(self) -> set[str]: + return {gid for gid, found in self.backend.residency().by_deployment.items() + if any(c.warm for c in found)} + + def crash(self, gid: str) -> None: + """Make the deployment's engine crash-loop with CRASH_LOG.""" + if self.kind == 'compose': + from infer_stack.leasing.compose import CRASH_LOOP_RESTARTS + + for c in self.runtime.containers.values(): + if c['labels'].get(DEPLOYMENT_LABEL) == gid: + c.update(state='restarting', restart_count=CRASH_LOOP_RESTARTS, + exit_code=1) + else: + self.runtime.crash[gid] = CRASH_LOG + # A crash-looping pod answers nothing, as a restarting container. + post = self.backend.http.post + self.backend.http.post = lambda url, **kw: ( + self.backend.http._Resp(503, {'detail': 'upstream down'}) + if url.startswith('http://127.0.0.1:') else post(url, **kw)) + self.backend.deployment_logs = lambda deployment, tail=400: CRASH_LOG + + def break_runtime(self) -> None: + """The runtime stops answering (docker or kubectl down).""" + def broken(args, **kw): + raise RuntimeError('connection refused') + self.backend.run = broken + + def gateway_routes(self) -> dict[str, dict]: + cfg = self.front.state_dir / 'litellm_config.yaml' + doc = yaml.safe_load(cfg.read_text()) if cfg.exists() else {} + return {e['model_name']: e['litellm_params'] for e in doc.get('model_list') or []} + + +def _compose(tmp_path, **gateway_kw) -> Stack: + from infer_stack.hardware import simulate_inventory + from infer_stack.leasing.compose import ComposeBackend + from test_leasing_compose import IMAGES, PORTS, STATE, FakeDocker, FakeHttp + + docker = FakeDocker() + backend = ComposeBackend( + state_dir=tmp_path / 'compose', inventory=simulate_inventory('4x80'), + run=docker, http=FakeHttp(tmp_path / 'compose'), + images={**IMAGES, 'open-webui': 'owui:test', 'nginx': 'nginx:test', + 'postgres': 'pg:test'}, + ports=PORTS, state=STATE, litellm=True, catalog=CAT, **gateway_kw, + ) + ctl = Controller(Ledger(SqliteStore(str(tmp_path / 'ledger.db'))), backend) + return Stack('compose', ctl, backend, docker, backend) + + +class _Kubectl: + """kubectl over applied Model CRs; each Model runs one pod.""" + + def __init__(self): + from test_leasing_kubeai import FakeKubectl + + self.inner = FakeKubectl() + self.crash: dict[str, str] = {} + + @property + def applied(self): + return self.inner.applied + + def __call__(self, args: list[str]) -> str: + from test_leasing_kubeai import _pod + + if len(args) > 4 and args[3] == 'get' and args[4] == 'pods': + pods = [] + for name, doc in self.inner.applied.items(): + gid = doc['metadata']['labels']['infer-stack/deployment'] + if gid in self.crash: + pods.append(_pod(f'model-{name}-0', gid, restarts=3, + state={'waiting': {'reason': 'CrashLoopBackOff'}}, + last={'exitCode': 1, 'reason': 'Error'})) + else: + pods.append(_pod(f'model-{name}-0', gid, ready=True)) + return json.dumps({'items': pods}) + return self.inner(args) + + +def _kubeai(tmp_path, **gateway_kw) -> Stack: + from infer_stack.backends.kubeai import KubeaiBackend + from infer_stack.leasing.compose import ComposeBackend + from test_leasing_compose import FakeDocker + from test_leasing_kubeai import GatewayHttp + + kubectl = _Kubectl() + gateway = ComposeBackend( + state_dir=tmp_path / 'gateway', inventory={'gpu_count': 0, 'gpus': []}, + run=FakeDocker(), http=GatewayHttp(kubectl.inner, tmp_path / 'gateway'), + project='infer-stack-gateway', litellm=True, + images={'litellm': 'litellm:test', 'open-webui': 'owui:test', + 'nginx': 'nginx:test', 'postgres': 'pg:test'}, + **gateway_kw, + ) + backend = KubeaiBackend(state_dir=tmp_path / 'kubeai', run=kubectl, + http=gateway.http, gateway=gateway, + gateway_upstream=UPSTREAM, default_resource_profile='gpu') + backend.catalog = CAT + ctl = Controller(Ledger(SqliteStore(str(tmp_path / 'ledger.db'))), backend) + return Stack('kubeai', ctl, backend, kubectl, gateway) + + +BUILDERS = {'compose': _compose, 'kubeai': _kubeai} + + +@pytest.fixture(params=sorted(BUILDERS)) +def make_stack(request, tmp_path): + def make(**gateway_kw) -> Stack: + return BUILDERS[request.param](tmp_path, **gateway_kw) + return make + + +# -- client contract ----------------------------------------------------------- + + +def test_one_base_url_the_managed_key_and_the_alias(make_stack): + stack = make_stack() + stack.acquire('one') + info = stack.backend.access(['one']) + assert info['base_url'].startswith('http://127.0.0.1:') + assert info['api_key'] == stack.front.master_key() + assert info['request_names'] == {'one': 'one'} + + +def test_env_file_has_the_same_keys(make_stack): + from types import SimpleNamespace + + from infer_stack.cli.commands_leasing import _descriptor_for + from infer_stack.leasing.envfile import descriptor_env + + stack = make_stack() + out = stack.acquire('one') + config = SimpleNamespace(base_url='unused', api_key_env='LITELLM_MASTER_KEY') + env = descriptor_env(_descriptor_for(stack.ctl, out.lease, out.deployments, config)) + assert set(env) >= {'INFER_STACK_LEASE_ID', 'OPENAI_BASE_URL', + 'INFER_STACK_ENDPOINT_ONE', 'INFER_STACK_MODELS'} + assert env['INFER_STACK_ENDPOINT_ONE'] == 'one' # the alias, on both + + +def test_readiness_is_a_generation_through_the_front_door(make_stack): + stack = make_stack() + out = stack.acquire('one', wait=True, timeout=10, interval=1) + assert out.wait is not None and out.wait.ready + + +def test_secrets_rotate(make_stack): + stack = make_stack() + old = stack.front.master_key() + stack.ctl.rotate_gateway_key() + assert stack.front.master_key() != old + + +# -- lifecycle ------------------------------------------------------------------- + + +def test_acquire_release_evict_gc_renew(make_stack): + stack = make_stack() + one = stack.acquire('one') + warm = stack.acquire('warm') + gid_one, gid_warm = one.deployments[0].id, warm.deployments[0].id + assert {gid_one, gid_warm} <= stack.running() + stack.ctl.renew(one.lease.id, ttl_seconds=3600) + stack.ctl.release(one.lease.id) + stack.ctl.release(warm.lease.id) + assert gid_one not in stack.running() # reclaim: stop + assert gid_warm in stack.running() # keep-warm stays resident + stack.ctl.gc() + assert gid_warm in stack.running() # plain gc leaves it + stack.ctl.evict(None) + assert stack.running() == set() + + +def test_a_refused_acquire_writes_nothing(make_stack): + stack = make_stack() + # What each backend cannot serve: more GPUs than the host has, or a + # Model with no resource profile. + if stack.kind == 'compose': + stack.backend.inventory = {'gpu_count': 1, 'gpus': [ + {'index': 0, 'memory_total_mib': 81920}]} + else: + stack.backend.default_resource_profile = None + with pytest.raises(PlacementError): + stack.acquire('wide') + assert stack.ctl.ledger.status() == ([], []) + assert stack.running() == set() + + +def test_config_publish_previews_then_commits(make_stack): + stack = make_stack() + profile = stack.backend.render_profile() + stack.ctl.publish_profile(profile) + assert stack.ctl.ledger.profile() == profile + + +def test_no_apply_stages_and_apply_brings_it_up(make_stack): + stack = make_stack() + out = stack.acquire('one', apply=False) + gid = out.deployments[0].id + assert gid not in stack.running() + assert stack.ctl.ledger.publication_pending() is not None + stack.ctl.apply_now() + assert gid in stack.running() + + +def test_a_crash_looping_engine_fails_fast_with_its_error(make_stack): + stack = make_stack() + out = stack.acquire('one') + gid = out.deployments[0].id + stack.crash(gid) + probe = stack.backend.probe_ready(stack.ctl.ledger.get_deployment(gid), 'one') + assert probe.ready is False and probe.fatal is True + assert 'trust_remote_code' in probe.detail + + +def test_strict_residency_and_lenient_observe(make_stack): + stack = make_stack() + stack.acquire('one') + stack.break_runtime() + with pytest.raises(ResidencyUnknown): + stack.backend.residency() + assert stack.backend.observe() == set() + + +# -- placement and the catalog ----------------------------------------------------- + + +def test_gpu_count_from_tp_pp_dp(make_stack): + stack = make_stack() + out = stack.acquire('wide') + gid = out.deployments[0].id + if stack.kind == 'compose': + assert len(stack.ctl.ledger.get_deployment(gid).assigned_gpus) == 2 + else: + (doc,) = [d for d in stack.runtime.applied.values() + if d['metadata']['labels']['infer-stack/deployment'] == gid] + assert doc['spec']['resourceProfile'] == 'gpu:2' + + +def _engine_args(stack, gid) -> list[str]: + if stack.kind == 'compose': + doc = yaml.safe_load(stack.backend.compose_file.read_text()) + (svc,) = [s for s in doc['services'].values() + if s.get('labels', {}).get(DEPLOYMENT_LABEL) == gid] + return [str(a) for a in svc['command']] + (doc,) = [d for d in stack.runtime.applied.values() + if d['metadata']['labels']['infer-stack/deployment'] == gid] + return [str(a) for a in doc['spec']['args']] + + +def test_runtime_flags_and_extra_args_reach_the_engine(make_stack): + stack = make_stack() + out = stack.acquire('one') + args = ' '.join(_engine_args(stack, out.deployments[0].id)) + assert '--max-model-len=2048' in args + assert '--seed 7' in args + + +def test_served_names_come_from_one_rule(make_stack): + from infer_stack.leasing.naming import dns_slug + + stack = make_stack() + out = stack.acquire('one') + gid = out.deployments[0].id + if stack.kind == 'compose': + doc = yaml.safe_load(stack.backend.compose_file.read_text()) + names = [n for n, s in doc['services'].items() + if s.get('labels', {}).get(DEPLOYMENT_LABEL) == gid] + assert names == [f'vllm-{dns_slug("one")}'] + else: + assert f'{dns_slug("one")}' in stack.runtime.applied + + +# -- the gateway ------------------------------------------------------------------ + + +def test_catalog_routes_exist_before_models_run_and_do_not_churn(make_stack): + stack = make_stack() + stack.ctl.apply_now() + before = stack.gateway_routes() + assert {'one', 'wide', 'warm'} <= set(before) + stack.acquire('one') + assert stack.gateway_routes() == before + + +def test_routes_list(make_stack, monkeypatch, capsys): + from infer_stack.cli import commands_leasing + + stack = make_stack() + stack.acquire('one') + monkeypatch.setattr(commands_leasing, '_open_controller', lambda *a, **k: stack.ctl) + capsys.readouterr() + assert commands_leasing.RoutesListCLI.main(argv=['--json']) == 0 + rows = {r['name']: r for r in json.loads(capsys.readouterr().out)['routes']} + assert rows['one']['live'] is True and rows['one']['upstream'] != '?' + + +def test_dynamic_routing_gives_dedicated_deployments_their_own_upstreams(make_stack): + stack = make_stack(dynamic_routing=True) + first = stack.acquire('one', dedicated=True, apply=False) + second = stack.acquire('one', dedicated=True, apply=False) + assert first.deployments[0].id != second.deployments[0].id + routes = json.loads((stack.front.state_dir / 'litellm_routes.json').read_text()) + ones = [r for r in routes if r['model_name'] == 'one'] + assert len(ones) == 2 # one alias, two upstreams + assert len({(r['litellm_params']['model'], r['litellm_params']['api_base']) + for r in ones}) == 2 + +def test_open_webui_and_the_reverse_proxy(make_stack): + from infer_stack.leasing.gateway import NGINX_SERVICE, OPEN_WEBUI_SERVICE + + stack = make_stack(ui=True, reverse_proxy=True) + stack.acquire('one', apply=False) + doc = yaml.safe_load(stack.front.compose_file.read_text()) + assert {OPEN_WEBUI_SERVICE, NGINX_SERVICE} <= set(doc['services']) + + +def test_the_gateways_changes_are_approved_with_the_acquire(make_stack, monkeypatch): + from infer_stack import diff_prompt as dp + + stack = make_stack() + for b in {id(stack.backend): stack.backend, id(stack.front): stack.front}.values(): + b.assume_yes = False + asked = [] + monkeypatch.setattr(dp, 'confirm_writes', + lambda changed, **kw: asked.append(sorted(p.name for p in changed)) + or True) + out = stack.acquire('one') + assert out.lease.state == LeaseState.ACTIVE + shown = [name for names in asked for name in names] + assert 'litellm_config.yaml' in shown # the gateway's change too + # Each file was shown once, before the commit, never again at the render. + assert len(shown) == len(set(shown)) + + +# -- day-2 ------------------------------------------------------------------------- + + +def test_instances_have_one_shape(make_stack): + stack = make_stack() + out = stack.acquire('one') + gid = out.deployments[0].id + instances = stack.backend.instances() + engines = [i for i in instances if i.deployment_id == gid] + assert len(engines) == 1 and engines[0].state == 'running' + assert any(not i.is_engine for i in instances) # the gateway + + +def test_stack_down_stops_everything(make_stack): + stack = make_stack() + stack.acquire('one') + stack.backend.down() + assert stack.running() == set() + + +def test_the_tui_reads_either_backend(make_stack): + """Leases, deployments, the Instances tab and the engines log view.""" + pytest.importorskip('textual') + import asyncio + + from infer_stack.tui import ENGINE_SERVICES, InferStackTUI + + stack = make_stack() + out = stack.acquire('one') + gid = out.deployments[0].id + + async def scenario(): + app = InferStackTUI(stack.ctl, CAT, interval=999, proc_factory=lambda s: None) + async with app.run_test() as pilot: + await pilot.pause() + await app.workers.wait_for_complete() # the startup refresh + app._collapsed['docker'] = False + data = app._collect() + assert [le.id for le in data['leases']] == [out.lease.id] + assert gid in {g.id for g in data['deployments']} + # What _render keeps, without racing the app's own refresh. + app._last_deployments = data['deployments'] + app._last_instances = data['instances'] + (engine,) = [r for r in app._ps_rows(data['instances']) + if r['serves'] == 'one'] + assert engine['status'] == 'running' + target, _ = app._resolve_log_target(ENGINE_SERVICES) + assert target == [engine['name']] + + asyncio.run(scenario()) diff --git a/tests/test_tui.py b/tests/test_tui.py index 6e2e4620..b9b14c90 100644 --- a/tests/test_tui.py +++ b/tests/test_tui.py @@ -3,7 +3,6 @@ from __future__ import annotations import asyncio -import json import pytest @@ -253,7 +252,7 @@ async def scenario(): proc_factory=lambda svc: None) async with app.run_test() as pilot: await pilot.pause() - # docker is a collapsible pane with Logs/Containers/Control tabs; + # runtime is a collapsible pane with Logs/Instances/Control tabs; # system is its own collapsed pane; API is a top-level tab now. tabs = app.query_one('#docker-tabs', TabbedContent) assert {p.id for p in tabs.query('TabPane')} == { @@ -263,12 +262,10 @@ async def scenario(): assert app.query_one('#system', Collapsible).collapsed top = app.query_one('#top', TabbedContent) assert 'tab-api' in {p.id for p in top.query('TabPane')} - # the containers ps view carries the docker-ps columns + # the instances view carries what `infer-stack ps` shows ps = app.query_one('#ps', DataTable) labels = [str(c.label) for c in ps.columns.values()] - assert 'status (uptime)' in labels - assert 'created' in labels - assert 'container id' in labels + assert labels == ['name', 'status', 'serves', 'started', 'ports'] _run(scenario) @@ -373,35 +370,65 @@ async def scenario(): _run(scenario) -def test_tui_leases_deployments_are_separate_panes(): +def _split_layout(): + """The TUI with both pane pairs stacked around dividers instead of tabbed.""" + from infer_stack.tui import InferStackTUI + + class Split(InferStackTUI): + TABBED_CATALOG = False + TABBED_TABLES = False + return Split + + +@pytest.mark.parametrize('tabbed', [True, False]) +def test_tui_pane_pairs_are_tabs_or_split(tabbed): + """Either layout has the same panes and ids; only the container differs.""" from textual.containers import Vertical + from textual.css.query import NoMatches + from textual.widgets import TabbedContent from infer_stack.tui import InferStackTUI controller, catalog = _ctx() + out = controller.acquire('bob', catalog.resolve_names(['qwen-coder'])) + assert out.lease + cls = InferStackTUI if tabbed else _split_layout() async def scenario(): - app = InferStackTUI(controller, catalog, interval=999, - proc_factory=lambda svc: None) - async with app.run_test() as pilot: + app = cls(controller, catalog, interval=999, proc_factory=lambda svc: None) + async with app.run_test(size=(140, 45)) as pilot: + await pilot.pause() + app.action_refresh() + await app.workers.wait_for_complete() await pilot.pause() - # leases/deployments are their own panes split by a drag handle, - # not collapsibles. - assert isinstance(app.query_one('#leases-pane'), Vertical) - assert isinstance(app.query_one('#deployments-pane'), Vertical) - assert app.query_one('#tsplit') + for pane in ('#leases-pane', '#deployments-pane'): + assert isinstance(app.query_one(pane), Vertical) + for table in ('#endpoints', '#models', '#leases', '#deployments'): + assert app.query_one(table) + dividers = [] + for divider in ('#csplit', '#tsplit'): + try: + dividers.append(app.query_one(divider)) + except NoMatches: + pass + if tabbed: + assert not dividers + tabs = app.query_one('#table-tabs', TabbedContent) + assert str(tabs.get_tab('pane-leases').label) == 'Leases 1/1' + # the pane in front gets the whole height, not a fixed 14 rows + assert app.query_one('#leases-pane').size.height > 14 + else: + assert len(dividers) == 2 _run(scenario) def test_tui_panes_drag_resize(): - from infer_stack.tui import InferStackTUI - controller, catalog = _ctx() async def scenario(): - app = InferStackTUI(controller, catalog, interval=999, - proc_factory=lambda svc: None) + app = _split_layout()(controller, catalog, interval=999, + proc_factory=lambda svc: None) async with app.run_test(size=(120, 40)) as pilot: await pilot.pause() w0, h0 = app._sidebar_w, app._log_h @@ -420,13 +447,11 @@ async def scenario(): def test_tui_dividers_have_a_grab_area(): - from infer_stack.tui import InferStackTUI - controller, catalog = _ctx() async def scenario(): - app = InferStackTUI(controller, catalog, interval=999, - proc_factory=lambda svc: None) + app = _split_layout()(controller, catalog, interval=999, + proc_factory=lambda svc: None) async with app.run_test(size=(120, 40)) as pilot: await pilot.pause() # A 0-size divider can't be grabbed; both must span their cross-axis. @@ -887,35 +912,33 @@ async def scenario(): _run(scenario) -def test_tui_ps_rows_parse_status_created_and_id(): +def test_tui_instance_rows_say_what_each_serves(): + from types import SimpleNamespace + + from infer_stack.leasing.instances import Instance from infer_stack.tui import InferStackTUI controller, catalog = _ctx() - sample = json.dumps([ - {'Service': 'litellm', 'Status': 'Up 3 minutes', 'State': 'running', - 'CreatedAt': '2026-06-19 00:00:00 -0400', 'ID': 'abcdef1234567890', - 'Publishers': [{'PublishedPort': 14042, 'TargetPort': 4000}]}, - ]) + instances = [ + Instance('vllm-qwen', 'abcdef1234567890', 'grp-1', 'restarting', + restarts=2, reason='CrashLoopBackOff', + started='2026-06-19T00:00:00.123Z'), + Instance('litellm', 'fedcba', '', 'running', ports='14042->4000/tcp'), + ] async def scenario(): app = InferStackTUI(controller, catalog, interval=999, proc_factory=lambda svc: None) async with app.run_test() as pilot: await pilot.pause() - # stub the backend's compose seam - backend = controller.backend - import tempfile - f = tempfile.NamedTemporaryFile('w', suffix='.yml', delete=False) - f.write('services: {}\n') - f.close() - backend.compose_file = f.name - backend.project = 'infer-stack' - backend.run = lambda args: sample - rows = app._compose_ps_rows() - assert rows[0]['status'] == 'Up 3 minutes' - assert rows[0]['created'].startswith('2026-06-19') - assert rows[0]['id'] == 'abcdef123456' # truncated to 12 - assert '14042->4000' in rows[0]['ports'] + app._last_deployments = [SimpleNamespace(id='grp-1', served={'qwen': {}})] + engine, gateway = app._ps_rows(instances) + assert engine['serves'] == 'qwen' + assert engine['status'] == 'restarting (CrashLoopBackOff, 2 restarts)' + assert engine['started'] == '2026-06-19 00:00:00' + assert gateway['serves'] == '(front door)' + assert gateway['ports'] == '14042->4000/tcp' + assert app._ps_rows(None) is None # the runtime was unreadable _run(scenario) @@ -952,11 +975,11 @@ async def scenario(): await pilot.pause() app._active_tab = 'tab-containers' app._collapsed['docker'] = True - assert 'ps' not in app._collect() # collapsed -> no ps poll + assert 'instances' not in app._collect() # collapsed -> no poll app._collapsed['docker'] = False - assert 'ps' in app._collect() # visible -> polled + assert 'instances' in app._collect() # visible -> polled app._active_tab = 'tab-logs' - assert 'ps' not in app._collect() # other tab -> no ps poll + assert 'instances' in app._collect() # the log picker needs names app._collapsed['system'] = True assert 'gpus' not in app._collect() # system collapsed app._collapsed['system'] = False @@ -1231,6 +1254,36 @@ async def scenario(): assert not any(str(le.state) == 'released' for le in leases) +def test_clean_up_is_one_footer_key_that_logs_its_cli_command(): + """One Clean up, on `x` in the footer, and each action names its command.""" + from textual.widgets import Button + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + seen = {} + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + seen['buttons'] = [b.id for b in app.query(Button) + if 'cleanup' in (b.id or '')] + seen['footer'] = [b for b in app.BINDINGS if isinstance(b, tuple)] + await pilot.press('x') + await app.workers.wait_for_complete() + await pilot.press('r') + await pilot.pause() + seen['applog'] = '\n'.join(app._app_log_lines) + + _run(scenario) + assert seen['buttons'] == [] + assert ('x', 'cleanup', 'Clear finished') in seen['footer'] # tuples show in the footer + assert 'CLI: infer-stack gc --forget' in seen['applog'] + assert 'CLI: infer-stack status' in seen['applog'] + + def test_tui_evict_all_idle_button(): from infer_stack.leasing import DeploymentState from infer_stack.tui import InferStackTUI @@ -1383,8 +1436,7 @@ async def scenario(): async with app.run_test() as pilot: await pilot.pause() backend = controller.backend - backend.compose_file = compose_file - backend.project = 'infer-stack' + backend.rendered_file = compose_file backend.run = lambda args: calls.append(args) or '' app.action_compose_up() await app.workers.wait_for_complete() @@ -1407,7 +1459,8 @@ async def scenario(): async with app.run_test() as pilot: await pilot.pause() app._restart_logs(ALL_SERVICES) # the gateway's own lines - await app.workers.wait_for_complete() # drain the log stream + await app.workers.wait_for_complete() # the stream has ended + app._drain_logs() # lines are drawn in batches await pilot.pause() assert any('ready' in line for line in app._log_lines) @@ -1436,6 +1489,7 @@ async def scenario(): await pilot.pause() app._restart_logs(ALL_SERVICES) # the gateway's own lines await app.workers.wait_for_complete() + app._drain_logs() # lines are drawn in batches await pilot.pause() text = '\n'.join(app._log_lines) assert 'Traceback (most recent call last):' not in text @@ -1563,6 +1617,10 @@ async def scenario(): app = InferStackTUI(controller, catalog, interval=999, proc_factory=lambda svc: None) async with app.run_test() as pilot: + await pilot.pause() + # The mount-time background refresh must land first, or it can + # repaint all 40 rows after the rebuild below (an intermittent fail). + await app.workers.wait_for_complete() await pilot.pause() table = app.query_one('#leases', DataTable) assert table.row_count == 40 @@ -1640,27 +1698,22 @@ def test_gateway_services_are_excluded_from_the_default_log_view(): """The logs pane defaults to engines, not everything. LiteLLM logs a line per proxied request, so on a busy host it scrolls the - engine output -- where errors actually appear -- out of the pane. + engine output -- where errors actually appear -- out of the pane. An + engine is an instance that serves a deployment; the gateway, UI, database + and proxy serve none, whatever they are named (a name hint used to let + Open WebUI and Postgres into the engines view). """ - from infer_stack.tui import ( - ALL_SERVICES, - ENGINE_SERVICES, - engine_services, - is_gateway_service, - ) - - assert is_gateway_service('litellm') - # Substring, not equality: a suffixed gateway service must still match, or - # the noisy view comes back silently. - assert is_gateway_service('infer-stack-litellm-1') - assert not is_gateway_service('vllm-qwen-qwen3-8-27b') + from infer_stack.leasing.instances import Instance + from infer_stack.tui import ALL_SERVICES, ENGINE_SERVICES - names = ['litellm', 'vllm-a', 'vllm-b'] - assert engine_services(names) == ['vllm-a', 'vllm-b'] + engine = Instance('vllm-a', 'c1', 'grp-1', 'running') + assert engine.is_engine + for name in ('litellm', 'infer-stack-litellm-1', 'open-webui', 'postgres'): + assert not Instance(name, 'c2', '', 'running').is_engine # The two sentinels must stay distinguishable from each other and from any - # real service name. + # real instance name. assert ENGINE_SERVICES != ALL_SERVICES - assert ENGINE_SERVICES not in names + assert ENGINE_SERVICES not in ('litellm', 'vllm-a') def test_named_log_process_follows_only_that_service(monkeypatch): @@ -1669,7 +1722,8 @@ def test_named_log_process_follows_only_that_service(monkeypatch): import subprocess import time - from infer_stack.tui import _DockerLogProc + from infer_stack.leasing.instances import Instance + from infer_stack.tui import InferStackTUI service = 'vllm-qwen3-8-27b-dbirks-hyperqwen' touched = [] @@ -1679,8 +1733,6 @@ def __init__(self, stdout): self.stdout = stdout def fake_run(cmd, **kwargs): - if cmd[:2] == ['docker', 'ps']: - return _Done(f'c1 running {service}\nc2 running litellm\n') touched.append(cmd[-1]) # docker logs --tail N return _Done(b'') @@ -1700,9 +1752,15 @@ def terminate(self): def wait(self, timeout=None): return 0 + controller, catalog = _ctx() + controller.backend.instances = lambda: [ + Instance(service, 'c1', 'grp-1', 'running'), + Instance('litellm', 'c2', '', 'running'), + ] monkeypatch.setattr(subprocess, 'run', fake_run) monkeypatch.setattr(subprocess, 'Popen', lambda cmd, **kw: _Proc(cmd)) - proc = _DockerLogProc('infer-stack', '/tmp/docker-compose.yml', service) + app = InferStackTUI(controller, catalog, interval=999) + proc = app._default_proc_factory()(service) deadline = time.monotonic() + 5 while len(touched) < 2 and time.monotonic() < deadline: time.sleep(0.05) @@ -1747,7 +1805,10 @@ async def scenario(): proc_factory=lambda svc: None) async with app.run_test() as pilot: await pilot.pause() - app._service_names = lambda: ['litellm', 'vllm-a', 'vllm-b'] + from infer_stack.leasing.instances import Instance + app._last_instances = [Instance('litellm', 'c0', '', 'running'), + Instance('vllm-a', 'c1', 'g1', 'running'), + Instance('vllm-b', 'c2', 'g2', 'running')] target, label = app._resolve_log_target(ENGINE_SERVICES) assert target == ['vllm-a', 'vllm-b'] @@ -1756,16 +1817,16 @@ async def scenario(): # A named service is passed straight through. assert app._resolve_log_target('litellm') == ('litellm', 'litellm') - # "all" stays None so docker compose logs gets no service argument. + # "everything" stays None: the follower takes every instance. target, label = app._resolve_log_target(ALL_SERVICES) - assert target is None and label == 'all services' + assert target is None and label == 'everything' # With no engines there is nothing to follow. Falling back to - # every service would show the gateway under the engines label. + # every instance would show the gateway under the engines label. from infer_stack.tui import NO_LOG_TARGET - app._service_names = lambda: ['litellm'] + app._last_instances = [Instance('litellm', 'c0', '', 'running')] target, label = app._resolve_log_target(ENGINE_SERVICES) - assert target is NO_LOG_TARGET and 'no engine services' in label + assert target is NO_LOG_TARGET and 'no engines running' in label _run(scenario) @@ -1791,15 +1852,16 @@ async def scenario(): proc_factory=factory) async with app.run_test() as pilot: await pilot.pause() - services = ['litellm'] - app._service_names = lambda: list(services) + from infer_stack.leasing.instances import Instance + app._last_instances = [Instance('litellm', 'c0', '', 'running')] app._collapsed['docker'] = False app._sync_log_services() app._restart_logs(app._log_service) # the pane opens await pilot.pause(0.2) assert started == [] # nothing to follow yet - services.append('vllm-a') # an engine is deployed + app._last_instances = [*app._last_instances, # an engine is deployed + Instance('vllm-a', 'c1', 'g1', 'running')] app._sync_log_services() await pilot.pause(0.2) await pilot.pause() @@ -2003,3 +2065,211 @@ async def scenario(): _run(scenario) assert 'CLI: infer-stack release --all --yes' in seen['applog'] assert 'pulling img: 1 of 4 layers' in seen['applog'] + + +def test_a_ui_thread_stall_is_reported_with_where_it_happened(tmp_path, monkeypatch): + """"The TUI froze" must come with what it was doing.""" + import time + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + monkeypatch.setattr(InferStackTUI, 'error_log_path', lambda self: tmp_path / 'e.log') + seen = {} + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause(0.3) + original = app._update_summary + + def slow(*args, **kwargs): + time.sleep(0.9) # a blocking call on the UI thread + return original(*args, **kwargs) + + app._update_summary = slow + app._refresh_now() + app._update_summary = original + await pilot.pause(0.5) # the watchdog reports after it ends + seen['applog'] = '\n'.join(app._app_log_lines) + + _run(scenario) + assert 'UI stalled' in seen['applog'] + assert 'in slow' in seen['applog'] or 'tui.py' in seen['applog'] + assert 'time.sleep(0.9)' in (tmp_path / 'e.log').read_text() + + +def test_a_log_flood_is_drawn_in_bounded_batches(): + from infer_stack.tui import ALL_SERVICES, LOG_PANE_LINES, InferStackTUI + + controller, catalog = _ctx() + lines = [f'vllm-x | loading shard {i}\n' for i in range(5000)] + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: _FakeProc(lines)) + async with app.run_test() as pilot: + await pilot.pause() + app._restart_logs(ALL_SERVICES) + await app.workers.wait_for_complete() + for _ in range(20): # a bounded batch per tick + app._drain_logs() + await pilot.pause() + shown = list(app._log_lines) + assert len(shown) <= 2 * LOG_PANE_LINES + assert shown[-1].endswith('loading shard 4999') # newest kept + assert any('earlier line(s) not shown' in s for s in shown) + + _run(scenario) + + +def test_a_running_action_shows_in_the_activity_line_until_it_ends(): + """A slow action is visibly working: spinner, what, and for how long.""" + import threading + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + name = next(iter(catalog.endpoints)) + gate = threading.Event() + real_acquire = controller.acquire + + def slow_acquire(*args, **kwargs): + gate.wait(5) + return real_acquire(*args, **kwargs) + + controller.acquire = slow_acquire + seen = {} + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + app._start_acquire(name) + await pilot.pause(0.3) + line = app.query_one('#activity') + seen['during'] = (line.display, str(line.render())) + gate.set() + await app.workers.wait_for_complete() + await pilot.pause(0.3) + seen['after'] = line.display + + _run(scenario) + shown, text = seen['during'] + assert shown and f'acquiring {name}' in text and 's' in text + assert seen['after'] is False + + +def test_an_edit_made_outside_the_tui_appears_on_the_next_refresh(tmp_path): + """The catalog file is reread when it changes (and on `r`); a broken save + is reported once, not on every refresh.""" + import copy + + import yaml + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + catalog_path = tmp_path / 'catalog.yaml' + catalog_path.write_text(yaml.safe_dump(CATALOG)) + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None, + catalog_path=str(catalog_path)) + async with app.run_test() as pilot: + await pilot.pause() + edited = copy.deepcopy(CATALOG) + edited['endpoints']['qwen-extra'] = dict(edited['endpoints']['qwen-fast']) + catalog_path.write_text(yaml.safe_dump(edited)) + app.action_refresh() + await pilot.pause() + assert 'qwen-extra' in app._endpoint_names + + refused = [] + app._refuse = lambda msg, **kw: refused.append(msg) + catalog_path.write_text('endpoints: [not, a, mapping\n') + app.action_refresh() + app.action_refresh() + assert len(refused) == 1 and 'catalog reload failed' in refused[0] + assert 'qwen-extra' in app._endpoint_names # the last good one stays + + _run(scenario) + + +def test_an_80x24_terminal_shows_logs_and_the_tables_when_the_runtime_opens(): + """At 80x24 the runtime pane used to take every row (the tables vanished), + and then, capped naively, left none for the log itself.""" + from textual.widgets import Collapsible + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test(size=(80, 24)) as pilot: + await pilot.pause() + assert app.has_class('compact') # descriptions give way + app.query_one('#docker', Collapsible).collapsed = False + await pilot.pause() + await pilot.pause() + assert app.query_one('#logs').region.height >= 3 + assert app.query_one('#tables').region.height >= 1 + + _run(scenario) + + +def test_the_api_tab_never_shows_the_master_key(): + """The curl on screen reads the key when run; the clipboard gets it.""" + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + app._litellm = lambda: ('http://localhost:14042', 'sk-secret-key') + app._sync_api_models(['qwen-coder']) + app._update_api_curl() + shown = str(app.query_one('#api-curl').render()) + assert 'sk-secret-key' not in shown + assert '$(infer-stack env LITELLM_MASTER_KEY)' in shown + copied = [] + app._copy = lambda text: copied.append(text) or True + app.action_api_copy_curl() + assert copied and 'sk-secret-key' in copied[0] + + _run(scenario) + + +def test_colored_engine_output_reads_cleanly_in_the_logs_pane(): + """vLLM colors its "(APIServer pid=1)" prefix; the escape codes used to + garble the line in the pane.""" + from textual.widgets import RichLog + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test(size=(160, 40)) as pilot: + await pilot.pause() + from textual.widgets import Collapsible + + app.query_one('#docker', Collapsible).collapsed = False + await pilot.pause() + app._write_log_lines(['\x1b[1;36m(APIServer pid=1)\x1b[0;0m INFO engines: model loaded']) + await pilot.pause() + text = '\n'.join(strip.text for strip in app.query_one('#logs', RichLog).lines) + assert '(APIServer pid=1) INFO engines: model loaded' in text + assert '\x1b' not in text and '[1;36m' not in text + + _run(scenario) diff --git a/uv.lock b/uv.lock index 558a181a..0028f947 100644 --- a/uv.lock +++ b/uv.lock @@ -8,9 +8,6 @@ resolution-markers = [ "python_full_version >= '4'", ] -[options] -exclude-newer = "2026-06-04T04:00:00Z" - [[package]] name = "certifi" version = "2026.5.20" @@ -257,7 +254,7 @@ name = "exceptiongroup" version = "1.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", marker = "python_full_version < '3.11'" }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/50/79/66800aadf48771f6b62f7eb014e352e5d06856655206165d775e675a02c9/exceptiongroup-1.3.1.tar.gz", hash = "sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219", size = 30371, upload-time = "2025-11-21T23:01:54.787Z" } wheels = [ @@ -278,11 +275,11 @@ name = "infer-stack" source = { editable = "." } dependencies = [ { name = "jinja2" }, + { name = "kwconf" }, { name = "loguru" }, { name = "pyyaml" }, { name = "requests" }, { name = "rich" }, - { name = "scriptconfig" }, { name = "ubelt" }, ] @@ -300,6 +297,7 @@ tui = [ [package.metadata] requires-dist = [ { name = "jinja2", specifier = ">=3.1" }, + { name = "kwconf", specifier = ">=0.11.0" }, { name = "loguru", specifier = ">=0.7" }, { name = "pytest", marker = "extra == 'tests'", specifier = ">=7.0" }, { name = "pytest-codeblocks", marker = "extra == 'tests'", specifier = ">=0.17" }, @@ -307,7 +305,6 @@ requires-dist = [ { name = "pyyaml", specifier = ">=6.0" }, { name = "requests", specifier = ">=2.31" }, { name = "rich", specifier = ">=13.0" }, - { name = "scriptconfig", specifier = ">=0.9" }, { name = "textual", marker = "extra == 'tui'", specifier = ">=0.50" }, { name = "ubelt", specifier = ">=1.3" }, { name = "xdoctest", marker = "extra == 'tests'", specifier = ">=1.1.5" }, @@ -335,6 +332,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, ] +[[package]] +name = "kwconf" +version = "0.11.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c6/77/20ce9cf65bc20e769db9a506e72e291f7a38ee766393110858a15e93ea1b/kwconf-0.11.0.tar.gz", hash = "sha256:cd8e9192a3c0e54750368980f9bf6b02676f75b76ac3830b656e977cc55f988f", size = 172077, upload-time = "2026-08-07T15:25:34.385Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fa/7a/4b4a1c503cc621eb2256e5d18385ccfb546f536008cdb4db0189426f5e77/kwconf-0.11.0-py3-none-any.whl", hash = "sha256:4691afdf7173ca0d42302a2cf17910917ac3bd02729e0b2f2cb2aa1e3b2ee18b", size = 110488, upload-time = "2026-08-07T15:25:32.964Z" }, +] + [[package]] name = "linkify-it-py" version = "2.1.0" @@ -655,19 +661,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, ] -[[package]] -name = "scriptconfig" -version = "0.9.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "pyyaml", marker = "python_full_version < '4'" }, - { name = "ubelt" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/a4/4c/6d0ecc5e29b498dc6b5402a0c561e97dd00ce1aeebdc5e494cc5fa8a0b6b/scriptconfig-0.9.1.tar.gz", hash = "sha256:9766c2ee601c4b8d97753ba13a914f8c0a212da4f6d2aec18ad628fd8b53d7ef", size = 112424, upload-time = "2026-03-02T17:33:34.405Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/57/00/2bf691dfc2b0170013d466fcdbbfc9d036e26d6049d74efaf7775401321d/scriptconfig-0.9.1-py3-none-any.whl", hash = "sha256:4b69e2f4eb681ec4480e9bb5d6c82d2f8c82a9badc703a261ee8940736101d86", size = 87103, upload-time = "2026-03-02T17:33:31.394Z" }, -] - [[package]] name = "textual" version = "8.2.7"