diff --git a/.gitignore b/.gitignore index 1eca01dd..93d85439 100644 --- a/.gitignore +++ b/.gitignore @@ -20,3 +20,6 @@ generated/kubeai/* !examples/single-node/ !examples/single-node/config.yaml !examples/single-node/models.yaml + +# rsync target for e2e harness reports (developer artifacts, not source) +e2e-report-*/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 6f177ce7..e20fe792 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,554 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +## [Version 0.7.0] - Unreleased + +### Added +* **Open WebUI can manage Ollama's own models.** Open WebUI is no longer locked + to the LiteLLM gateway with `ENABLE_OLLAMA_API=False`. It now holds two + connections: an **OpenAI** connection (the gateway when on, else a single + upstream's own `/v1`) for chat, and a **native Ollama** connection pointed + straight at any Ollama daemon, so you can pull/run/delete models from the UI + and have the daemon load them on demand — a true drop-in for a hand-run + `ollama` + Open WebUI stack. +* **LiteLLM gateway is now optional** — `--litellm` / `--no-litellm` (and + `config set litellm`). With it off, Open WebUI (still on by default) talks to + the rendered upstreams directly, and `access()` reports the UI URL. Open WebUI + also renders without a gateway whenever there is an upstream to point at. + Ollama tags are now pulled on `acquire` even with the gateway off. +* New tutorial: `docs/source/manual/ollama-openwebui-tutorial.md` — stand up a + self-managing Ollama + Open WebUI box (GPU-pinned) entirely from the CLI. Adds + a `docs/source/manual/` section to the Sphinx docs (the leasing demo moved + there too). +* TUI **API tab** is now a proper console: shows the gateway + Open WebUI URLs + (ctrl+click to open), **List models** (GET `/v1/models` on the gateway) beside + Send / Test-all, a live **curl** preview with a **Copy curl** button, and an + **Open WebUI** button. Clipboard support via Textual (OSC 52): Copy-curl, and + `y` copies the status line (handy for the open-URL it prints). +* TUI live-feedback polish: the **endpoint wizard is labeled and + engine-adaptive** — vLLM shows tensor-parallel / **data-parallel** / max-model-len + / GPU-mem / max-seqs / prefix-caching / extra-args; Ollama shows host + + free-form runtime — so the form is no longer a row of unlabeled inputs. The + **API tester moved to its own top-level tab** (more room for the monitor); + catalog **Add/Edit/Remove are localized** to the endpoints + models panels + (the bottom button stack is gone; Suggest moved to Settings); the vertical + splitter drags the **full width range** (not just the middle); and the + endpoints↔models drag direction is fixed. +* TUI **top-level tabs** — the multipane monitor is now a **Dashboard** tab, with + a new **Settings** tab to edit the durable settings (backend, data dir, Open + WebUI, reverse proxy, skip-display-GPUs) and save them to `settings.yaml` + without dropping to YAML. Textual's command palette (ctrl+p) exposes every + action for search. +* TUI docker pane gained a **Control** tab (compose up/down + the rendered + compose-file path), and the models table now shows **quant** + a **cached** + flag (a cheap existence check against the HF hub cache — no slow `du`). +* TUI catalog management — the add-endpoint wizard now exposes the runtime knobs + that matter for serving (**tensor-parallel size, max model len, GPU memory + fraction, raw extra vLLM args** — where data-parallel etc. go — and the + **reclaim policy**), mirroring `catalog endpoint add`. You can **edit** an + endpoint (blocked while it's actively served) and **remove** an endpoint or + model (with a confirm dialog; a model still referenced by an endpoint is + refused by the validating writer). CLI parity already exists via + `catalog endpoint add [--force]` / `catalog endpoint rm` / `catalog model rm`. + +### Fixed +* **Ollama GPU pinning to a non-zero GPU** silently fell back to CPU. The docker + device reservation (`device_ids`) already exposes only the pinned GPU and the + NVIDIA runtime renumbers it to `0` inside the container, but the service also + set `CUDA_VISIBLE_DEVICES` to the *host* index — so pinning to GPU 1 left the + container looking for device `1` that wasn't there. Now pinned by the + reservation alone (like vLLM). New e2e tier `88_gpu_pinning` exercises this on + a real 2nd GPU; new tier `86_ollama_lean` covers the `--no-litellm` stack, and + `85_ollama` now asserts the dual Open WebUI connection wiring. + +### Removed (breaking) +* **Collapsed `serve` into `acquire` (no compatibility alias).** `serve` was a + thin preset of `acquire` (an infinite, `manual`-owned lease) that routed + through the identical code path; the two verbs differed only by an owner label + and which flags they exposed. There is now a single verb: `infer-stack acquire` + — with no `--ttl` it is an infinite (standing-service) lease, and `--ttl 2h` + makes it a time-boxed reservation. `acquire` now carries the everyday-`serve` + help (render→apply→wait, `--no-apply`/`--no-wait`/`--no-ui`). Migration: + `serve X` → `acquire X`; the default lease owner is now `$USER` (was `manual` + for `serve`) — pass `--owner manual` to keep the old label. The TUI's "Serve" + control is unchanged (it calls the controller directly, not the CLI verb). +* Removed the pre-leasing **profile world** now superseded by catalog + leasing + (no back-compat — pre-release): the entire `infer-stack legacy …` command + group and its modules (`cli/commands_profile`, `cli/commands_smoke`, + `renderer`, `benchmark`, `verification`, `contracts`, the active-profile + `Up/Down/Purge/Deploy/Env/Ollama*` runtime verbs), plus their tests + (~5k LOC). The ollama and kubeai *concepts* are retained (catalog + `engine: ollama`, leasing coalescing, `kubeai_ops`/`backends/kubeai_renderer`) + for when those backends are implemented. Follow-up pass also removed the + pre-leasing `resolver`, `validator`, the `cli/compose` + `cli/probes` shims, + and the superseded `backends/compose_renderer` (replaced by + `leasing/compose`); carved `cli/context` down to `_apply_path_overrides` + + `effective_inventory`. (The old top-level `catalog.py` + the now-dead helpers + inside `config.py` are an internal-only follow-up — they no longer surface in + the CLI.) +* `infer-stack status` is now a **leasing-native holistic overview** — backend, + data/config dirs, catalog (with model/endpoint counts), settings, ledger, and + compose-project locations, plus a leasing summary (active leases / live + deployments) and "dig deeper" pointers (`leases`, `tui`, `stack ps`, `logs`). + It no longer reports on the old active-profile render. + +### Changed (breaking) +* Renamed the **deployment group concept → "deployment"** throughout (core + classes, CLI, TUI, docs): `DeploymentGroup` → `Deployment`, `GroupState` → + `DeploymentState`, `group_id(s)` → `deployment_id(s)`, ledger methods + (`get_group` → `get_deployment`, `list_groups` → `list_deployments`, …), the + `leases --json` key `groups` → `deployments`, and the SQLite `groups` table → + `deployments` (+ `claims.group_id` → `claims.deployment_id`). No DB migration — + delete any existing ledger DB and it will be recreated. Mental model: "many + leases → one deployment." +* Renamed the lease identifier **`session_id` → `lease_id`** everywhere (the + object is a `Lease`; the dual vocabulary was confusing). This is a hard rename: + the env-file key is now `INFER_STACK_LEASE_ID` (was `INFER_STACK_SESSION_ID`), + the `release`/`renew` positional/flag is `--lease` (was `--session`), JSON + output uses `lease_id`, `read_session_id()` → `read_lease_id()`, and generated + lease ids are prefixed `lease-` (was `sess-`). Update any scripts that sourced + the old env var or parsed `session_id`. + +### Added +* Begin the leasing/controller redesign (see + `dev/infer-stack-redesign-critique.md` in the aiq-eval-runner repo). New + `infer_stack.leasing` subpackage with a backend-agnostic, sqlite-backed lease + ledger: `acquire`/`release`/`renew` bookkeeping, demand reference-counting, + same-model coalescing (with capacity subsumption), per-daemon coalescing for + Ollama, soft-TTL expiry, and idle-deployment reclaim computation. This is the core + that later phases (reconciler, backend protocol, `acquire`/`run` CLI) build on. +* Serving catalog parser (`infer_stack.leasing.catalog`): the new declarative + `models` / `endpoints` / `runtime_hosts` / `bundles` schema (replacing + profiles as the primary unit), with cross-reference validation and + `resolve_endpoint` / `resolve_names` that turn endpoint and bundle names into + ledger `EndpointRequest`s (vLLM per-model, Ollama per-daemon). +* Backend protocol + controller (`infer_stack.leasing.backend` / + `.controller`): a 4-method `Backend` seam (`realize`/`teardown`/`observe`/ + `probe_ready`), a `MemoryBackend` for tests/dry-runs, and a `Controller` that + reconciles the ledger's desired state onto a backend (LIVE + keep-warm-idle + deployments), enforces TTL on every reconcile, scopes readiness waits to the + endpoints a lease requested, and exposes thin `acquire`/`release`. The Compose + and KubeAI backends will implement the same protocol. +* Leasing CLI verbs: `infer-stack acquire` / `release` / `renew` / `run` / + `serve` / `leases`, plus an endpoint-descriptor env-file + (`infer_stack.leasing.envfile`, aligned with the `contracts.py` shape) and a + dry-run `NullBackend`. `run -- ` acquires, injects the endpoint env into + the child, and releases on exit — the kwdagger pipeline-node seam. Until the + Compose/KubeAI backends land, `--backend null` (default) exercises the whole + surface without serving anything real. +* Single-host GPU placement planner (`infer_stack.leasing.placement`): assigns + GPUs across the whole live set of deployment deployments (reusing the resolver's + `_first_fit`), honoring `allowed_gpus`, `reserved` GPUs (for Phase-2 raw-GPU + reservations), display-GPU skipping, and `pinned` assignments so adding or + removing a deployment does not reshuffle already-running models. This is the placer + the Compose backend will use; multi-node/bin-packing stay out of scope. +* Compose backend (`infer_stack.leasing.compose`): a focused renderer that turns + the live set of deployment deployments directly into a docker-compose project + (reusing `profile_runtime.vllm_args`), and a `ComposeBackend` that converges + the whole union on each reconcile (`docker compose up -d --remove-orphans`), + persisting GPU assignments so reconciles don't reshuffle running models. + Docker is invoked through an injected `run` seam (unit-tested against a fake; + real docker/GPU path validated on a host). The controller now prefers a + backend's `converge(desired)` over per-deployment realize/teardown. `infer-stack + ... --backend compose` is wired up. +* Compose LiteLLM front door + readiness + converge lock: the Compose backend + now renders a LiteLLM gateway (default on) that routes each endpoint alias to + its upstream vLLM/Ollama service, giving one stable `base_url`. `ComposeBackend + .access()` supplies that real base_url + per-endpoint request name into the + env-file descriptor (the CLI prefers it over the `--base-url` placeholder). + `probe_ready` now checks the gateway's `/v1/models` listing (model is + routable) via an injected HTTP seam, and `converge` is serialized with a file + lock so concurrent processes don't clobber the shared compose file. (Ollama + tag pull/warmup is a remaining readiness follow-up.) + +### Added (continued) +* Managed **Open WebUI**, on by default. The Compose backend now renders an + `open-webui` service in front of the LiteLLM gateway whenever the front door + is up, so `infer-stack serve chat` brings up a working chat UI (default + `http://127.0.0.1:13000`) with no hand-run `docker run`. Its spec is + independent of which models are live (it talks to the `litellm` service at a + fixed URL), so `docker compose up -d` leaves it running across model + add/remove/switch — only the gateway is recreated on a routing change, so the + UI never blinks (the property the legacy stack worked to preserve). Chat + history persists under the data dir. Opt out per-call with `--no-ui` or + globally with `infer-stack config set ui false`; `serve`/`acquire` print the + UI URL when ready. +* `infer-stack test ` — a leasing-native smoke test. Sends one real + chat completion to the endpoint *alias* through the gateway (managed key + applied automatically) and prints latency + the reply, or an actionable + failure with a non-zero exit. The concise alternative to hand-rolled `curl` + (the demo still shows `curl` for the raw form). +* `infer-stack env` is now the single verb for the managed env-file, replacing + both the `secrets` alias and the `secret get|set|list` modal (removed — the + secrets live in a readable `.env`, so a separate "secret" surface earned its + keep only by adding `set`, which folds in cleanly): + - `infer-stack env` — print the env-file **path** (`source "$(infer-stack + env)"` to load everything); + - `infer-stack env LITELLM_MASTER_KEY` — print one value; + - `infer-stack env HF_TOKEN=hf_…` — set one value (merges non-destructively, + so a gated model's token can be set once before `serve`); + - `infer-stack env --export` — every entry as `export KEY=value` lines. + The argument is a `KEY` to read or `KEY=VALUE` to write — mirroring the legacy + `infer-stack env` ergonomic in one command. +* `config set ui true|false` — a durable default for the managed Open WebUI + (a new recognized setting alongside `backend` / `data_dir`). +* LiteLLM front-door config now carries `router_settings` + (`num_retries`/`timeout`/`cooldown_time`/`allowed_fails`) so the brief window + where an upstream vLLM/Ollama is still loading its model is retried and + self-heals instead of surfacing as client `500`s / loud + `InternalServerError: … Connection error. Received Model Group=…` logs. + (LiteLLM doesn't wait for upstream *health* to start, so it forwards early + requests to a not-yet-listening upstream during warmup.) +* `python -m infer_stack` now runs the CLI (a top-level `__main__.py` redirects + to `infer_stack.cli.main`), matching `python -m infer_stack.cli`. +* Faster CLI startup. Heavy third-party deps that only matter at *runtime* + (`requests`, `jinja2`, `rich.syntax`/`pygments`) are now imported lazily + inside the functions that use them instead of at module import, so a bare + `infer-stack --help` / tab-completion no longer pays for the HTTP, templating, + and syntax-highlighting stacks (cli import time roughly −30%). The + `cli_mod.requests` test seam is preserved via a module `__getattr__`. +* Reorganized the sprawly flat CLI (~38 top-level verbs) into noun submodals, + keeping the leasing hot path at the top level (see `dev/cli-redesign.md`; + `infer-stack help tree` prints the whole surface): + - `infer-stack catalog …` — a flag-driven editor for the user catalog + (`catalog model|endpoint|host|bundle add|list|show|rm`, plus + `init/path/show/validate/edit`) with a validating writer, so models/endpoints + are added without hand-editing YAML. + - `infer-stack config …` — `init/paths/show/set/get/edit` over a new durable + `settings.yaml`. `config init` is an interactive rich prompt (data dir + + default backend, with a confirmation) and takes `--yes` (and auto-detects a + non-TTY) for non-interactive scripting. `config set backend compose` and `config set data_dir

` + are honored (the leasing `--backend` default and `data_root()` consult them), + so the backend flag and storage location no longer have to be repeated/ + exported. + - `infer-stack release --all` — release every active lease in one shot (the + whole stack idles/tears down per each lease's reclaim policy). Makes teardown + a one-liner instead of scraping session ids out of `leases`. + - `infer-stack wait [NAME…]` — block until served endpoints are ready, the + leasing-native companion to `serve --no-wait` (the old `wait-ready` was + legacy/profile-only). Lets you fan out — `serve --no-wait a; serve --no-wait + b; wait a b` loads models in parallel instead of back-to-back. No names waits + for every live deployment; `--require-generation`/`--timeout`/`--interval` apply. + (Readiness has two orthogonal knobs: `--require-generation` is the + *criterion* — a real token vs a listed model — and `--wait`/`--no-wait` + + `wait` are the *blocking* control.) + - `infer-stack evict [NAME…|--all]` — force-tear-down released (idle) models + now, overriding `keep-warm`, to free their GPUs. A keep-warm model normally + stays resident after release (no cold-start next time) but holds a GPU; evict + drops it. Target by served endpoint alias or deployment id, or `--all` for every + idle deployment; live models (with an active lease) are never evicted. `release + --evict` does the release-then-evict in one step (composes with `--all`). + Mechanically: idle deployments are marked `stopped`, so the next reconcile + converges them away. + - `infer-stack leases` is rich-formatted on a terminal (lease/deployment tables + with state colors); piped/`--json` output is unchanged. + - `infer-stack leases` now shows **actual vs desired**, not just the ledger's + intent. Each deployment gains a `running` column (from `backend.observe()` — what + docker actually has up) and a `gpus` column (which GPU indices it is on, or + `→N` *slated* for a desired-but-not-yet-started deployment). So a `state=live` / + `running=—` row reads as "wanted, not up yet" (starting, staged, or + unplaceable) instead of looking like a phantom. Both fields are in `--json`. + Best-effort: a dry-run/docker-less host degrades to "unknown" rather than + erroring. + - Render and apply are now separate verbs, with `serve`/`acquire` as the + combined "declare + render + apply + wait". `infer-stack render` writes the + on-disk compose project (+ GPU placement) for the current desired set + **without** `docker compose up`; `infer-stack apply` brings the desired set + up (idempotent; re-renders from intent first). Both are **lease-free** — the + duplicate-lease trap from staging-then-re-serving is gone because applying a + staged lease no longer goes through a declare verb (so the refcount only + climbs when you genuinely declare again). Declares refcount + (`acquire`/`serve`); reconciles are idempotent (`render`/`apply`). + `serve|acquire --no-apply` *stages*: declare the lease + render, skip the up + (and the wait + diff prompt); `release` discards it. Placement still runs at + render time, so an unplaceable request fails fast either way. Fits the + existing ledger→controller→backend split: `converge(desired, apply=False)` + is the render half, `ComposeBackend.plan()` exposes read-only placement (also + what `leases`' `gpus` column uses). + - The rendered compose carries a top-level `name: infer-stack`, so a plain + `docker compose -f docker-compose.yml up -d` (infer-stack not involved) + lands in the *same* project — same container names, same network — as the + tool's own `-p` invocations. "Drop infer-stack and run docker yourself" is + now an exact equivalent of `apply` rather than a sibling project the tool + can no longer see. (`infer-stack stack up` remains the raw "run the on-disk + file verbatim" hatch, vs `apply` which re-renders from intent.) + - Optional Textual TUI (`infer-stack tui`): a multi-pane dashboard. A + **catalog** pane (left) lists your models + endpoints — select an endpoint + and press `s`/Enter to request a lease; **leases** + **deployments** panes show + the live ledger (desired state vs running, GPUs), auto-refreshing; a **logs** + pane tails `docker compose logs -f` and a dropdown points it at a specific + service (or all). Controls: `s` serve, `d` release, `a` release-all, `e` + evict, `r` refresh, `tab` cycle panes, `q` quit. Mutations converge off the + UI thread so the monitor stays responsive; narration is silenced while the + TUI owns the terminal. The log source is injectable, so the whole app is + exercised headless via Textual's pilot in the tests. Opt-in extra — + `pip install "infer-stack[tui]"`; without textual the command exits with an + install hint. (This also made `SqliteStore` thread-safe — + `check_same_thread=False` + a lock serializing write transactions — so the + same ledger connection is usable from a converge worker thread.) + - TUI, second pass — made it approachable and snappier for new users: + a warm **orange / white / dark-gray theme**; an intro line plus per-pane + help text; **add-model / add-endpoint wizards** (`m` / `n`, or the sidebar + buttons) and a **Suggest** button (`g`) that seeds a catalog sized to your + GPUs via `catalog suggest`; the logs pane became a **docker tab** (live + `logs -f` *and* a `ps` snapshot). Actions are now **scoped to the pane they + act on** — Serve sits under the catalog, Release/Release-all under the + leases table, Evict under the deployments table — so the global footer keeps only + truly-global controls (Refresh / Next-pane / Quit); the keys still work. + Panes are **drag-resizable** (grab the full-height/width splitter bars) in + addition to the `[` `]` / `-` `+` keys. Responsiveness: + the periodic refresh + `docker compose ps` now run on a worker thread (no UI + freeze), and docker's own `up`/`down` progress is captured into the logs + pane instead of bleeding onto the screen. The TUI now requires a one-time + `config init`, and tolerates a missing/empty catalog (it shows the + empty-state + Suggest button rather than erroring). + - TUI, third pass — the bottom pane became a **collapsible, tabbed console** + (click the title, or `c`, to collapse it). Tabs: **Logs**; **Status · ps** + (now with the `docker ps`-style status/uptime, created time, and container + id, plus service + ports); **System** (live `nvidia-smi` per-GPU + util/mem/temp, plus host load/mem/cpu from `/proc`); and an **API** tester + that sends a prompt to a served model — or pings *all* of them with a + latency report — through the LiteLLM gateway. Only the **visible** tab's + data is polled (and nothing when the console is collapsed), so `ps` / + nvidia-smi don't run when you can't see them. **Ctrl+click** a served + endpoint (or `o`) opens it in Open WebUI (`/?models=`). The HTTP + client is injectable for headless tests. + - TUI, fourth pass — pane-local clarity + housekeeping. The describe-everything + intro line is gone; each pane carries its own one-line description. The + bottom **console** is renamed **docker** and its **Status** tab is now + **Containers**; **System** and **API** are promoted to their own collapsed + panes (so they're only polled when you expand them). The **API** model + picker now lists *only models that are up and ready* (served by a running + deployment), not every catalog entry. New **Clean up** action (`x`, or the button + under leases/deployments) forgets released/expired leases and stopped deployments — + backed by a new `Ledger.prune()` / `SqliteStore.prune()`. Default theme is + now the stock **textual-dark** (the orange theme stays available from the + command palette). + - `Ledger.prune()` (+ `SqliteStore.prune()`): delete terminal ledger rows — + RELEASED/EXPIRED leases and STOPPED deployments (and their claims) — for callers + that want to forget history rather than keep it inspectable. + - TUI — made the lease↔deployment (many-to-one) relationship legible: the + leases pane gained a **deployment** column (the deployment id(s) a lease holds — + the same id shown in the deployments pane, so the join is visible), and the deployments + pane now shows **leases** (how many hold it) + **held by** (their owners) + instead of the opaque "demand". Moving the cursor spells the link out in the + status bar ("lease … → deployment …" / "deployment … ← held by N lease(s)"). + Action-bar buttons are compact (1 row). + - Optional single-port HTTP reverse proxy (`reverse_proxy`). Enable it + (`--reverse-proxy`, or `config set reverse_proxy true`) to front the gateway + + Open WebUI with one nginx origin — UI at `/`, the OpenAI API at `/v1` — so + there's one port to hit instead of remembering 13000 (UI) and 14042 (API). + Plain HTTP, no TLS/auth (localhost / trusted networks only); the value is + ergonomics, not security. The generated conf handles Open WebUI's websockets + + large uploads; a `{enabled, port, config_path}` block (via `config edit`) + sets the port or mounts a bring-your-own `nginx.conf`. Needs the gateway, so + it's rendered only alongside litellm; `access()` reports the unified + `proxy_url`. (TLS/LDAP — the legacy `frontends.reverse_proxy` features — and a + remote control surface are deliberately *not* included here, since those + require the auth this plain proxy can't provide.) + - The front door (LiteLLM gateway + Open WebUI) is now a standing service, + decoupled from model count. It was rendered only alongside ≥1 model, so + `release --all`/`evict --all` left an empty desired set and `converge` downed + the *whole* project — taking the CPU-only gateway/UI with it (the UI blinked, + and `evict`, whose job is freeing GPUs, tore down the front door as a side + effect). Now the gateway/UI render whenever enabled (empty `model_list` when + no models), so `release`/`evict` only stop model containers (freeing GPUs) + and the front door stays up — Open WebUI never blinks, even to zero models, + and reconnects as you serve again. `infer-stack stack down` is the way to + take the whole stack (gateway + UI included) down. The empty-set→`down` + converge path now only triggers with the gateway off (`litellm=False`). + - Better `--help`. Expanded the leasing verbs' docstrings (rendered as the + argparse description) and added `__epilog__` examples to `serve` and + `leases` plus a quickstart + mental-model epilog on the top-level + `infer-stack --help` (catalog → serve/acquire → reconcile; render vs apply; + desired vs running). The `leases` help now documents each deployment column. + - Friendlier "unknown endpoint" error. You serve/acquire *endpoints*, not + models — passing a model name (`serve qwen05`) now says so and lists the + endpoints that run it (`Endpoints for 'qwen05': qwen05-1, qwen05-2 …`), or, + if the model has none, points at `catalog endpoint add --model`. For a name + that's neither, it offers a did-you-mean over the known endpoints/bundles. + - Compose changes are now shown before they're applied, for **every** verb + that touches the compose project — `serve`/`acquire` *and* `release`/`evict`/ + `apply`. On a terminal each renders a diff of the compose project (and + LiteLLM routing) it's about to write and asks to confirm; `--yes`/`-y` skips + it, and it's skipped automatically off a terminal (scripts/CI). Nothing + mutates `docker-compose.yml` or runs docker without that gate — earlier, + `release`/`evict` converged silently, which (with the container rename) could + e.g. recreate a keep-warm model during a `release` with no preview. Each + command batches its ledger changes into a single converge, so you're asked at + most once. Declining an *acquire* rolls back the just-created lease; declining + a *release/evict* leaves the ledger change recorded but docker untouched — + `infer-stack apply` then applies it (consistent with the render/apply split). + - `config init` now prompts for **every** known setting (data dir, backend, + Open WebUI, display-GPU skipping), not just data dir + backend — driven by a + single settings registry, so a newly added setting is asked about + automatically. It also says which mode it's in ("initializing a new config + from scratch" vs "editing the existing config at (or hand-edit it with + `infer-stack config edit`)"). Re-running edits in place and preserves any + keys it doesn't manage; `--fresh` discards the existing config and resets to + defaults. + - `catalog model show` / `endpoint show` with no NAME now print every entry in + that section instead of failing with `endpoint 'None' not found`; an unknown + NAME's error now lists what's available (`… not found (have: e1, e2)`). + - Behind-the-scenes feedback via loguru (like `aivm`): the leasing verbs + narrate placement, `docker compose up/down`, and readiness waits to stderr + (INFO; `$INFER_STACK_LOG_LEVEL` to change). It's kept off the `--help` + import path (loguru is ~50ms) and silent for library/test use until the CLI + enables it, so stdout/JSON output is untouched. + - `infer-stack catalog endpoint add` — `NAME` is now optional and defaults to + `{model}-{N}` (the vLLM model name, or the Ollama tag, slugified, with an + auto-incrementing suffix). So `catalog endpoint add --model smol135` creates + `smol135-1`, and a repeated add for the same model gets `smol135-2` instead + of colliding — keeping the served name (what Open WebUI shows) tied to the + model. An explicit `NAME` is still there for a stable alias decoupled from + the model (e.g. `chat` you can re-point with `--force`). + - `infer-stack catalog rm` now takes **multiple + names** (`rm a b c`); removal is atomic — if any name is missing, nothing is + removed. + - `infer-stack env` — read/write the managed compose `.env` (path / `env KEY` + / `env KEY=VALUE` / `--export`); e.g. `env HF_TOKEN=…` sets a gated model's + token once before `serve`. (Supersedes the short-lived `secret` modal and + `secrets` alias — see the `infer-stack env` entry above.) + - `infer-stack stack …` — the day-2 compose wrappers (`logs/ps/restart/pull/ + start/stop/down`); `logs`/`ps` remain top-level aliases. + - `infer-stack legacy …` — the pre-leasing profile/active-profile commands + (`setup/init/render/switch/resolve/lock/validate/up/down/deploy/…/ollama-*`) + grouped into a holding pen, promoted out as they gain leasing-native + behavior and removed wholesale once empty. + - `infer-stack help tree` — the full nested command tree at a glance + (cf. `aivm help tree`). +* The day-2 compose wrappers (`logs`, `ps`, `restart`, `pull`, `start`, `stop`) + now target the **leasing** Compose deployment when one exists + (`data_root/leasing/compose`, project `infer-stack`) — so `infer-stack logs + -f` / `infer-stack ps` work for a leasing user with no `config.yaml`. They + fall back to the legacy rendered stack when there's no leasing deployment. + (The `ollama-*` wrappers remain legacy: they exec a fixed `ollama` service + that the leasing model names per-daemon.) +* Keep the legacy meta commands relevant post-refactor: `infer-stack config + paths` (also exposed top-level as `infer-stack paths`) gained a `leasing` + deployment showing the lease ledger, the compose state dir, and its rendered + artifacts (docker-compose.yml, litellm_config.yaml, the secrets `.env`, + sidecar). `infer-stack status` now prints a one-line leasing summary (active + leases / live deployments) pointing at `infer-stack leases`. +* Managed LiteLLM secret + `infer-stack env`. The Compose backend now owns + `LITELLM_MASTER_KEY` (reused from the state dir's `.env` if you pin one, else + generated via `ensure_secret`), bakes it into the LiteLLM service, uses it for + the readiness probe, and ships it in the `--env-file` descriptor as + `OPENAI_API_KEY` — so `source`-ing the env-file fully configures an OpenAI + client (no manual `export`). `infer-stack env [KEY]` prints the managed + secrets (`$(infer-stack env LITELLM_MASTER_KEY)`), restoring the legacy + `infer-stack env` ergonomic for the leasing model. +* Ollama pull/warmup readiness in the Compose backend: a daemon serves a tag + lazily, so `probe_ready` now pulls the endpoint's tag into its daemon + (`docker compose exec … ollama pull`, idempotent) and forces a generation + through the front door to warm it before reporting ready. A `--require-generation` + flag opts vLLM readiness into the same real-generation check. + +### Fixed +* `infer-stack status` is leasing-aware. It no longer tells a leasing user (who + has `catalog.yaml` / `settings.yaml`, maybe with active leases) that they are + "Not initialized — run `infer-stack setup …`". `config.yaml` is now reported + as `legacy config` (it belongs to the pre-leasing profile world), and the + summary leads with backend / data dir / catalog / settings + the leasing + one-liner. The setup hint only appears when *nothing* is set up, and then + points at the leasing getting-started (`config init` / `catalog init` / + `serve`); the legacy KubeAI status error now references `infer-stack legacy + setup` (the command's real path after the CLI reorg). On a terminal the + summary is now rich-formatted (bold labels, colored values, a styled leasing + line); piped/redirected output stays plain (`Console.is_terminal`), so scripts + and tests are unaffected. +* Converging to an empty desired set no longer crashes. Releasing the last + `reclaim:stop` lease leaves zero services to render, and `docker compose up -d` + errors with "no service selected" on a services-less file — so the release's + reconcile raised (and `infer-stack run` surfaced it as a non-zero exit even + though the job succeeded). Converge now tears the project `down` when there are + no services instead of `up`-ing an empty file. (Latent until deployments were + fully isolated; the GPU e2e `80_run_wrapper` tier caught it.) +* LiteLLM now reloads when its routing config changes. The gateway reads its + model_list once at startup from a bind-mounted file; converge rewrote that + file but `docker compose up -d` left the old container running (its service + spec was unchanged), so a newly added/removed alias never became routable. In + practice: coalescing a second alias onto a live deployment (e.g. an endpoint with a + `public_name`) added it to the rendered config but the running gateway never + picked it up, so that lease's readiness probe timed out. The LiteLLM service + now carries a `infer-stack.config-hash` label derived from the config content, + so converge recreates it exactly when the routing changes (and leaves it alone + otherwise). Found by the `50_coalescing` e2e tier on GPU hardware. This trades + a brief gateway blip on a model add/remove for correctness; keeping LiteLLM up + across switches (as the legacy stack did) is tracked in + `dev/leasing-followups.md`. +* Placement uses **every** GPU by default, including display-attached ones — + skipping the monitor's GPU is now opt-in. A single-GPU host (whose only GPU + drives the display) would otherwise place nothing at all, so the safe default + is "use it". Opt in to leaving a display GPU free with `--skip-display-gpus` + (per command) or `infer-stack config set skip_display_gpus true` (persisted). + This flips the earlier default — the leasing verbs' `--include-display-gpus` + flag is replaced by `--skip-display-gpus`, and `plan_placement`/`ComposeBackend` + default `skip_display=False`. Demoed by the `45_both_gpus` e2e tier; see + `dev/leasing-demo.md` and `dev/e2e_tests/`. +* `vllm_args` no longer emits `--disable-log-requests`, which vLLM v0.19.1 + rejects (`unrecognized arguments`) — it crashed vLLM, surfacing as a LiteLLM + "Connection error". Engine-version-specific flags can go in `extra_args`. +* Compose `observe()` is now resilient to a stale/invalid on-disk compose file. + `reconcile` observes (via `docker compose ps`, which validates the file) + before `converge` rewrites it, so a bad file left by an earlier run would + crash `acquire` before it could be fixed. `observe()` now returns "nothing + observed" on any docker/parse error and lets converge overwrite the file + (self-heal). +* Test isolation: the subprocess `run_cli` helpers in `tests/test_cli_meta.py` + and `tests/test_cli_setup.py` used `env.setdefault('INFER_STACK_CONFIG_DIR'/ + 'INFER_STACK_DATA_DIR', tmp_path)`, which let an ambient `INFER_STACK_*` + exported in the caller's shell leak in — the tests then read the real config/ + data dir and failed (e.g. on a box where those were exported for manual + testing). Force the vars to `tmp_path` instead. +* Compose GPU reservation emitted `capabilities: [["gpu"]]` (list-of-lists), + which `docker compose` rejects ("capabilities.0 must be a string"); now emits + `capabilities: ["gpu"]`. Found while testing the Compose backend on real + 2-GPU hardware (see `dev/leasing-test-plan.md`). Guarded against recurrence by + a test that runs `docker compose config -q` on the rendered project (skipped + where docker compose is unavailable) — it validates the artifact's schema, not + just the dict we build. +* `acquire`/`serve` now fail fast when a requested model can't be placed, + instead of hanging on readiness forever. Previously, if every GPU was already + taken (e.g. one model already serving on the only free GPU), serving a second + model placed *nothing* for it — placement was a silent `WARNING` — yet the + ledger had already marked the new deployment `LIVE`, so `infer-stack leases` showed + a phantom "live" deployment with no container behind it, the compose diff was empty + (nothing to approve, hence no prompt), and `serve` blocked on a readiness + probe for a container that would never start until Ctrl-C. The controller now + detects that a just-requested deployment landed unplaced, rolls the lease back + (matching the diff-declined path) and raises `PlacementError`; the CLI prints + the planner's reason ("need 1 GPUs but only 0 available") plus how to free a + GPU (`leases` → `release`/`evict`). Found running the `dev/leasing-demo.md` + walkthrough on yardrat (two models, one free GPU). + +### Changed +* vLLM compose service/container names now lead with the served model: + `vllm--` (e.g. `infer-stack-vllm-qwen05-1-grp-098e…`) instead + of the opaque `vllm-grp-098e…`. A vLLM deployment is exactly one model in one + container, so `docker ps` / `nvidia-smi` are now legible *without* infer-stack + — a stated goal: you can drop the tool and the running stack still makes sense. + The full deployment id is kept as a suffix so the name stays unique (two desired + deployments can share a served name when an endpoint is re-pointed at a new model) + and correlates 1:1 with the `id` column of `infer-stack leases`. The name is + also LiteLLM's on-network upstream host, so it is slugified to a DNS-safe + `[a-z0-9-]` label and derived from one helper used by both the service key and + the routing config. Ollama daemons keep their deployment-id name (one daemon can + host several models, so a model-led name would mislead). Upgrading recreates + already-running vLLM containers once (the service key changes). +* Consolidate shared machinery so the legacy and leasing code paths reuse one + implementation instead of duplicating it: + - GPU-pool placement primitives (`available_gpu_indices` / `first_fit` / + `resolve_gpu_indices`) moved to `infer_stack.hardware` and reused by both + the resolver and the leasing placement planner (no more importing the + resolver's private functions). + - HTTP readiness probes moved to a new layer-neutral `infer_stack.probe` + (`openai_ready` / `ollama_ready`) over an injectable HTTP client. `cli.probes` + re-exports them for the legacy `wait-ready`/`switch` callers; the leasing + Compose backend reuses the same probe (so its readiness gained the + advertised-alias and optional-generation checks). One probe implementation + instead of two. + ## [Version 0.0.1] - ### Added diff --git a/README.md b/README.md index 863b531f..6c95ee5b 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,15 @@ [![Python versions](https://img.shields.io/pypi/pyversions/infer-stack.svg)](https://pypi.org/project/infer-stack/) [![License](https://img.shields.io/pypi/l/infer-stack.svg)](https://github.com/AIQ-Kitware/infer_stack/blob/main/LICENSE) +> **Heads up — the leasing model is now the primary workflow.** Declare models +> in a catalog (`infer-stack catalog …`) and `acquire`/`run` endpoints +> on demand; see `docs/source/manual/` (the Ollama + Open WebUI tutorial and the +> leasing demo) and `infer-stack help tree`. The +> **named stack profiles** documented below (`setup`/`render`/`up`/`switch`/…) +> are the pre-leasing model and now live under **`infer-stack legacy `** +> (e.g. `infer-stack legacy render`). This README still describes that legacy +> flow; a leasing-oriented rewrite is pending. + `infer_stack` manages **named stack profiles** for local and Kubernetes-backed inference. A stack profile is a small graph made from: diff --git a/dev/cli-redesign.md b/dev/cli-redesign.md new file mode 100644 index 00000000..65bd2fab --- /dev/null +++ b/dev/cli-redesign.md @@ -0,0 +1,178 @@ +# infer-stack CLI redesign — submodals, a catalog editor, and `help tree` + +Status: **implemented** (decisions: noun-verb grammar, full phased reorg). +Shipped across phases: `catalog` editor + `help tree`; `config` submodal + +settings store (honors `backend`/`data_dir`); `secret` get/set/list; `stack` +day-2 group; the pre-leasing profile world grouped under `legacy`. Top-level is +now the leasing loop + submodals. Companion to the leasing redesign +(`dev/leasing-followups.md`, `dev/infer-stack-redesign-critique.md`). + +The tree and rationale below are kept as the design record; `infer-stack help +tree` prints the live surface. + +## What we have + +`infer-stack` is one flat `scfg.ModalCLI` (`infer_stack/cli/__init__.py`) with +~38 top-level subcommands spanning three eras mashed together: + +- **Leasing (the new primary model):** `acquire release renew run serve leases + secrets` +- **Day-2 ops on the running stack:** `logs ps restart pull start stop` +- **Meta:** `version config(+paths) paths status env` +- **Legacy profile/active-profile world:** `setup init resolve validate lock + render switch list-models list-profiles explain describe-profile + verify-profile kubeai-sync-resource-profiles up down purge deploy diagnose + wait-ready smoke-test benchmark ollama-pull ollama-list ollama-ps` + +Problems: +- **Sprawl** — 38 flat verbs; the hot path (acquire/run) is buried among + legacy profile machinery. +- **No ergonomic catalog editing** — adding a model/endpoint means hand-editing + `~/.config/infer_stack/catalog.yaml`. (The demo literally `cat >` a heredoc.) +- **`init` is mis-placed** — it's config bootstrap, not a top-level verb. +- **No way to see the whole surface** — nested help requires walking each group. + +Good news: `config` is already a nested `ModalCLI` (`config paths`), and +scriptconfig 0.9.1 supports **3-level** nesting (verified: `infer-stack catalog +model add NAME` works, and group-level `--help` renders). So everything below is +feasible with the framework we already use. + +## Principles + +1. **Hot paths stay at the top level.** The kwdagger/AIQ pipeline-node loop — + `acquire`, `run`, `release`, `leases`, `status` — and the two most + common day-2 verbs (`logs`, `ps`) remain one token deep. +2. **Everything else is grouped into noun submodals** (`catalog`, `config`, + `secret`, `stack`) so the surface reads as a small set of areas. +3. **`legacy` is a holding pen,** not a graveyard: the pre-leasing profile world + moves under `infer-stack legacy …` verbatim. We promote commands out as we + give them leasing-native behavior, and delete `legacy` wholesale once empty. +4. **`help tree` shows the totality** (ported from `aivm help tree`). + +## Proposed CLI tree + +``` +infer-stack +│ # ── hot path: the leasing loop (top-level) ── +├── acquire lease endpoints, block until ready +├── run -- acquire, run cmd with endpoint env, release +├── serve standing service (infinite lease) +├── release release a lease +├── renew extend a lease's TTL +├── leases show leases + deployment groups +├── status overall stack status (+ leasing summary) +├── logs / ps day-2 convenience aliases (-> stack logs/ps) +├── secrets [KEY] convenience alias (-> secret get) +├── version +├── help [cmd…] | help tree per-command help; `tree` prints the whole surface +│ +│ # ── catalog: edit the user catalog without raw YAML (NEW) ── +├── catalog +│ ├── init write a starter catalog.yaml +│ ├── path print the catalog path +│ ├── show [name] pretty-print the catalog (or one entry) +│ ├── validate parse + cross-reference check +│ ├── edit open $EDITOR (escape hatch) +│ ├── model add|list|show|rm models: source/revision/quantization +│ ├── endpoint add|list|show|rm endpoints: engine/model/runtime/reclaim +│ ├── host add|list|rm runtime_hosts: ollama daemons / placement +│ └── bundle add|list|rm bundles: named endpoint groups +│ +│ # ── config: where things live + durable settings ── +├── config +│ ├── init was top-level `init` +│ ├── paths (existing) resolved file/dir locations +│ ├── show effective config +│ ├── set persist defaults — e.g. backend=compose, +│ │ data_dir=/data/infer-stack (kills two smells) +│ └── edit open $EDITOR +│ +│ # ── secret: the managed LiteLLM key + friends ── +├── secret +│ ├── get [KEY] was `secrets`; $(infer-stack secret get …) +│ ├── set KEY=VALUE e.g. HF_TOKEN -> the managed .env +│ └── list export-style lines +│ +│ # ── stack: day-2 ops on the running (leased) compose stack ── +├── stack +│ ├── logs ├── ps ├── restart ├── start ├── stop ├── pull └── down +│ +│ # ── legacy: pre-leasing profile/active-profile world (promote out, then delete) ── +└── legacy + ├── setup ├── render ├── switch ├── resolve ├── lock ├── validate + ├── list-models ├── list-profiles ├── explain ├── describe-profile + ├── verify-profile ├── kubeai-sync-resource-profiles + ├── up ├── down ├── purge ├── deploy + ├── env ├── diagnose ├── wait-ready ├── smoke-test ├── benchmark + └── ollama-pull ├── ollama-list └── ollama-ps +``` + +### The `catalog` submodal (the headline new feature) + +A `CatalogEditor` loads the YAML, mutates the dict, validates via +`Catalog.from_dict`, and writes it back (atomic temp+rename). Flags map 1:1 to +the schema so the demo's heredoc becomes: + +```bash +infer-stack catalog init +infer-stack catalog model add smol17b --source hf://HuggingFaceTB/SmolLM2-1.7B-Instruct +infer-stack catalog endpoint add chat --engine vllm --model smol17b \ + --max-model-len 8192 --gpu-mem 0.4 --extra-arg --dtype=half \ + --reclaim keep-warm +infer-stack catalog endpoint add chat-fast --engine vllm --model smol135 --reclaim stop +infer-stack catalog host add local-ollama --engine ollama --gpu 0 --keep-alive 5m +infer-stack catalog bundle add pair chat chat-fast +infer-stack catalog validate +``` + +Notes / decisions for this group: +- **Round-tripping comments:** plain `yaml` drops comments. For a + tool-generated catalog that's acceptable; if we want to preserve hand-written + comments, use `ruamel.yaml`. Proposed: start with plain yaml, revisit. +- **`add` is idempotent-ish:** `add` errors if the name exists unless + `--force`/`--update`; `rm` is by name with a friendly "not found". +- **`--dry-run`** prints the resulting YAML without writing. +- Validation reuses the existing `Catalog` parser, so the editor can never write + a catalog the leasing path would reject. + +### `help tree` + +Walk `ManageCLI`'s registered subconfigs recursively (each is a `DataConfig` or +nested `ModalCLI`), printing an indented tree with the one-line description from +each class docstring. `infer-stack help ` dispatches to that command's +`--help`. Pure introspection over the modal registry — no per-command wiring. + +## Migration plan (phased; no hard back-compat required) + +The user has said back-compat isn't required, but we minimize churn to the +already-green e2e harness + demo, which only use top-level +`acquire/run/serve/release/renew/leases/secrets/status/logs/ps/paths`. + +1. **Add `catalog` + `help tree`** (purely additive). Unblocks the demo's + "add a model without YAML" and gives the totality view. Lowest risk. +2. **Add `config` subcommands** (`init`→`config init` with a thin top-level + `init` shim that warns; `config set/show/edit`). Wire `config set` to a real + settings store so `--backend compose` / `data_dir` stop being repeated + (leasing-followups smells #1/#2). +3. **Add `secret` group** (`secret get/set/list`); keep `secrets` as a top-level + alias for `secret get` (script compatibility for `$(infer-stack secrets …)`). +4. **Add `stack` group**; alias `logs`/`ps` at top level; the other day-2 verbs + move under `stack`. +5. **Introduce `legacy`**; move the profile-world commands under it. Keep + top-level shims for any that the e2e/consumers still call, emitting a + one-line "moved to `legacy …`" deprecation note, for one release. +6. **Prune:** once nothing calls the shims, delete them; promote any `legacy` + command we've made leasing-native; delete `legacy` when empty. + +Each phase is independently shippable and testable. Phases 1–3 are the high +ergonomic payoff; 4–6 are cleanup. + +## Open questions for review + +- `catalog model add` (noun-verb, 3-level) vs `catalog add-model` (verb-noun, + 2-level)? Proposal: **noun-verb** — scales better, reads like `aivm`/`kubectl`. +- Keep top-level `secrets`/`logs`/`ps` aliases, or force the grouped form? + Proposal: **keep the few hottest as aliases.** +- `config set` backing store: a new `settings.yaml` in the config dir vs folding + into the existing `config.yaml`. Proposal: a small `settings.yaml` owned by the + leasing world (config.yaml stays the legacy profile artifact). diff --git a/dev/demo-textual.py b/dev/demo-textual.py new file mode 100644 index 00000000..52131146 --- /dev/null +++ b/dev/demo-textual.py @@ -0,0 +1,4173 @@ +from __future__ import annotations + +""" +Single-file RPG architecture prototype. + +This version integrates the TUI with the game a little bit better and has some +visual art. + +Reading guide +------------- +v7 is v6 with one extra: an ASCII_ART_POLISHED dictionary (search the +file) that maps event keys -> multi-line ASCII art strings. The engines +look these up to render an illustration alongside text for some +encounters and NPCs. Everything else is the v4 RPG core + v5 Textual +front-end fused together — see v3, v4, v5 for the architecture comments. +""" + +from collections import Counter +from dataclasses import dataclass, field +import random +import time +from typing import Callable + + +# ============================================================ +# Small helpers / presentation +# ============================================================ + + +def typewriter_print(prefix: str, text: str, word_delay: float = 0.018) -> None: + print(prefix, end="") + for word in text.split(): + print(word, end=" ", flush=True) + time.sleep(word_delay) + print() + + +def say(name: str, text: str) -> None: + typewriter_print(f"{name}: ", text) + + +def divider(title: str | None = None) -> None: + print("\n" + "=" * 72) + if title: + print(title) + print("-" * 72) + + +def action_separator(label: str | None = None) -> None: + print("\n" + "-" * 72) + if label: + print(f"> {label}") + print("-" * 72) + + +def prompt_continue(prompt: str = "Press Enter to continue...") -> None: + input(f"\n{prompt}") + + +# ============================================================ +# Visual art (ASCII) +# ============================================================ +# Maps short string keys ("ruins_stumble", an NPC name, etc.) to +# multi-line ASCII art. r"""...""" is a *raw* triple-quoted string — +# the `r` prefix means backslashes are kept literal, which matters here +# because the art uses backslashes as drawing characters. +# Engines below look these keys up to render art when an NPC is +# inspected or an encounter fires. +ASCII_ART_POLISHED = { + "ruins_stumble": r"""\ + __ + __/ /__ + _/ _ _\_ +/__ /_\ /_\_\ + \/ / \/ + /_/\ + /_/ / +""", + "bramble_push": r"""\ + \\\ | //// + \\\ | //// +---\\ | ////--- +----\\|////---- +----////|\\---- +---//// | \\--- + //// | \\ +""", + "dark_cave_blocked": r"""\ + _________ + / _____ \ + / / \ \ + | | XXX | | + | | XXX | | + | |_______| | + \___________/ +""", + "broken_crossing": r"""\ + __ __ +_/ /________/ /_ +\_\ ____ /_/ + / / __ \ \ +_/__/ / \ \__\_ +\____/ \____/ +""", + "tower_gate_unlock": r"""\ + /\ + /[]\ + /_[]_\ + | || | + | || | + | || | + |_/\_| +""", + "forest_whisper_hint": r"""\ + ^^ ^^ + < \ / > + { o } + / | \ + / | \ + / \ +""", + "trail_reopened": r"""\ + _________ + __/ __ __ \__ +/_ / // / _\ + \ /_//_/ \ / + / __ \ + _/__/ \____\_ +\______________/ +""", + "serve_time": r"""\ + | | | | | | + | | | | | | + | | |o| | | + | | /|\ | | + | | / \ | | + |_|_____|_| +""", + "jail_release": r"""\ + | | | | | + | | | | | + | | |o | | + | | /|\ | + | | / \ | + |_|_______| +""", + "first_tower_visit": r"""\ + /\ + / \ + / ** \ + / **** \ + /______ \ + |||| + ~~~~~~~~ +""", + "bandit_surrenders": r"""\ + .------. + / @ @ \ + | ^ | + | --- | + \_/| |\_/ + | | + / \ +""", + "player_surrender_accepted": r"""\ + \o/ + | + / \ + /___\ + | + / \ +""", + "player_surrender_rejected": r"""\ + \o/ + | + / \ + X/ \X + /_____\ + / \ +""", + "guardian_trial_passed": r"""\ + .-***-. + .' *** '. + / ** ** \ + < *** > + \ ** ** / + '. *** .' + '-***-' +""", + "guardian_trial_failed": r"""\ + .--?--. + .' ! '. + / /_\ \ + < ! > + \ \_/ / + '. ! .' + '--.--' +""", + "spider_web_cleared": r"""\ + \ | | | / + \ | | | / +---\ \ () / /--- +---/ / /\/\ \ \--- + / | | | \ + / | | | \ +""", + "return_crystal": r"""\ + /\ + /**\ + /****\ + \__/ + || + .-========-. + / ________ \ + \_/________\_/ +""", + "fountain_restored": r"""\ + /\ + /**\ + /****\ + ~~~||~~~ + ~~~~~||~~~~~ + _||_ + _/____\_ +""", + "fountain_rest": r"""\ + _=====_ + _/ _____ \_ + / / \ \ + | | ~~~ | | + | | ~~~ | | + \ \_____/ / + \_________/ +""", + "quest_complete_cheer": r"""\ + \o/ \o/ + | | + / \ / \ + * * * * * * + \o/ \o/ + | | + / \ / \ +""", + "guard_arrest": r"""\ + [###] + /|\ + ___ | ___ + [___]/ \ [___] + | | + / \ / \ +""", + "item_shatter": r"""\ + __ + |++| + __|__|__ + / /\/\ \ + \_/\__/\\_/ + / \ +""", + "mercy_choice": r"""\ + \o + |\ + / \ + __/ \__ + / mercy \ + \________/ +""", + # Dedicated location icons + "loc::village": r"""\ + _/\_ _/\_ + / \ / \ + | [] | | [] | + |_____| |____| + \ ___ / + /~ ~ \ + /_____\ +""", + "loc::crossroads": r"""\ + || + =====++===== + || + ---++--- + || + / \ +""", + "loc::forest": r"""\ + &&& &&& + &&&&&&&&&&& + || || || + /| || ||\ + /_|_||_||_\ +""", + "loc::garden": r"""\ + .-*-. + .-( * )-. + / * /\ * \ + \___/ \___/ + /_/\_\ + <*> <*> +""", + "loc::lake": r"""\ + _ _ + .-(.)----(.)-. + ~~~/ \~~~ + / ~~~~~~~~ \ + /________________\ + /_/ \_\ +""", + "loc::cave_entrance": r"""\ + ________ + / ____ \ + / / \ \ + | | | | + | | __ | | + \ \____/ / + \________/ +""", + "loc::cave_depths": r"""\ + /\/\ + /_**_\ + /_****_\ + \ * * / + \_**_/ + /_/ \_\ +""", + "loc::ruins": r"""\ + _[]_ + | | + __|_ _|__ + / _ || _ \ + ||_|_||_|_|| + \__ __/ + |____| +""", + "loc::tower_gate": r"""\ + /^^\ + /_[]_\ + | || | + | || | + | || | + |_||_| + /_/\_\ +""", + "loc::tower_top": r"""\ + /\ + / \ + /_[]_\ + || + __||__ + /______\ + / \ +""", + "loc::jail": r"""\ + .------. + | |||| | + | |||| | + | |||| | + | |||| | + |______| +""", + # NPC-specific placeholder art + "npc::elder_mira::alive": r"""\ + .-====-. + .' .--. '. + / / \ \ + | | o o | | + | | -- | | + | | \__/ | | + \ \____/ / + '. ____ .' + '------' +""", + "npc::elder_mira::dead": r"""\ + .-====-. + .' .--. '. + / / \ \ + | | x x | | + | | -- | | + | | \__/ | | + \ \____/ / + '. ____ .' + '------' +""", + "npc::guard_halwen::alive": r"""\ + .------. + / ____ \ + | |o o| | + | | -- | | + | |____| | + | /|_||_|\ | + \_ || _/ + | || | + |_||_| +""", + "npc::guard_halwen::dead": r"""\ + .------. + / ____ \ + | |x x| | + | | -- | | + | |____| | + | /|_.._|\ | + \_ || _/ + | | + |_/\_| +""", + "npc::merchant_sella::alive": r"""\ + .------. + / o o \\ + | .. | + | \____/ | + | .-====-.| + \| $$$$ |/ + '------' + /||||\\ +""", + "npc::merchant_sella::dead": r"""\ + .------. + / x x \\ + | .. | + | \____/ | + | .-====-.| + \|_$$$$_|/ + '------' + / \\ +""", + "npc::fisher_rowan::alive": r"""\ + .------. + / o o \\ + | >< | + | \____/ | + \ || / + '._||_.' + ||~~~ + || +""", + "npc::fisher_rowan::dead": r"""\ + .------. + / x x \\ + | >< | + | \____/ | + \ || / + '._||_.' + || + ~~ +""", + "npc::bandit_nox::alive": r"""\ + .------. + / o o \ + | ^ | + | --- | + \_/| |\_/ + | | + / \ + +""", + "npc::bandit_nox::dead": r"""\ + .------. + / x x \ + | ^ | + | --- | + \_/| |\_/ + | | + / \ + +""", + "npc::crystal_spider::alive": r"""\ + \\ |\ /| // + \\ | \/ | // +-----\\| () |//----- +-----//|/==\\|\\----- + // |/ \\| \\ + // /_/\\_\\ \\ +""", + "npc::crystal_spider::dead": r"""\ + \\ |\ /| // + \\ | xx | // +-----\\|_==_|//----- +-----//|/--\\|\\----- + // |/ \\| \\ + // /_/\\_\\ \\ +""", + "npc::tower_guardian::alive": r"""\ + .-**-. + .' * * '. + / * /\\ * \\ + < * || * > + \\ * \/ * // + '. * * .' + '-**-' +""", + "npc::tower_guardian::dead": r"""\ + .-..-. + .' xx xx'. + / .. /\\ ..\\ + < .. || .. > + \\ .. \/ ..// + '. xx xx .' + '-..-' +""", + "npc::small_spider::alive": r"""\ + \\ /\ // +----\\(oo)//---- +----// || \\\---- + // _||_ \\ +""", + "npc::small_spider::dead": r"""\ + \\ /\ // +----\\(xx)//---- +----//_||_\\---- + // / \\ +""", +} + + +# ============================================================ +# Core data models +# ============================================================ + + +@dataclass +class Stats: + max_hp: int + attack_min: int + attack_max: int + defense: int = 0 + + +@dataclass +class Item: + item_id: str + name: str + description: str = "" + slot: str | None = None + power_bonus: int = 0 + defense_bonus: int = 0 + hp_bonus: int = 0 + healing: int = 0 + set_flags_on_pickup: set[str] = field(default_factory=set) + tags: set[str] = field(default_factory=set) + + +@dataclass +class TradeOffer: + title: str + wants_items: list[str] = field(default_factory=list) + gives_items: list[str] = field(default_factory=list) + wants_gold: int = 0 + gives_gold: int = 0 + repeatable: bool = False + required_flags: set[str] = field(default_factory=set) + blocked_flags: set[str] = field(default_factory=set) + + +@dataclass +class DialogueTopic: + key: str + title: str + lines: list[str] + required_flags: set[str] = field(default_factory=set) + blocked_flags: set[str] = field(default_factory=set) + required_items: list[str] = field(default_factory=list) + once: bool = False + outcome_effect: Effect | None = None + + +@dataclass +class Riddle: + question: str + answers: list[str] + intro_lines: list[str] = field(default_factory=list) + success_lines: list[str] = field(default_factory=list) + failure_lines: list[str] = field(default_factory=list) + repeat_lines: list[str] = field(default_factory=list) + damage_on_failure: int = 0 + set_flags_on_success: set[str] = field(default_factory=set) + + +@dataclass +class Exit: + direction: str + destination: str + requires_item: str | None = None + requires_flag: str | None = None + blocked: bool = False + blocked_text: str = "That path is blocked." + warning_text: str | None = None + on_attempt_effect: Effect | None = None + on_blocked_effect: Effect | None = None + on_success_effect: Effect | None = None + + +@dataclass +class Location: + key: str + name: str + description: str + exits: list[Exit] = field(default_factory=list) + items: list[str] = field(default_factory=list) + npcs: list[NPC] = field(default_factory=list) + features: list[Feature] = field(default_factory=list) + + +# ============================================================ +# Effects +# ============================================================ + + +class Effect: + def apply(self, game: Game) -> None: + raise NotImplementedError + + +class PrintEffect(Effect): + def __init__(self, lines: str | list[str]): + self.lines = [lines] if isinstance(lines, str) else lines + + def apply(self, game: Game) -> None: + for line in self.lines: + print(line) + + +class SetFlagEffect(Effect): + def __init__(self, *flags: str): + self.flags = list(flags) + + def apply(self, game: Game) -> None: + for flag in self.flags: + game.flags.add(flag) + + +class ClearFlagEffect(Effect): + def __init__(self, *flags: str): + self.flags = list(flags) + + def apply(self, game: Game) -> None: + for flag in self.flags: + game.flags.discard(flag) + + +class SetBountyEffect(Effect): + def __init__(self, amount: int): + self.amount = amount + + def apply(self, game: Game) -> None: + game.bounty = max(0, self.amount) + print(f"[Bounty] Set to {game.bounty}") + + +class ChangeBountyEffect(Effect): + def __init__(self, amount: int, reason: str | None = None): + self.amount = amount + self.reason = reason + + def apply(self, game: Game) -> None: + game.change_bounty(self.amount, self.reason) + + +class HealPlayerEffect(Effect): + def __init__( + self, amount: int, heal_text: str | None = None, full_text: str | None = None + ): + self.amount = amount + self.heal_text = heal_text or "You recover some health." + self.full_text = full_text or "You already feel fully rested." + + def apply(self, game: Game) -> None: + if game.player.health >= game.player.total_max_hp(game.world.item_db): + print(self.full_text) + return + print(self.heal_text) + healed = game.player.heal(self.amount, game.world.item_db) + print(f"Recovered {healed} health.") + + +class GivePlayerItemEffect(Effect): + def __init__(self, *item_ids: str): + self.item_ids = list(item_ids) + + def apply(self, game: Game) -> None: + for item_id in self.item_ids: + game.give_player_item(item_id) + + +class ChangeGoldEffect(Effect): + def __init__( + self, amount: int, *, target: str = "player", reason: str | None = None + ): + self.amount = amount + self.target = target + self.reason = reason + + def apply(self, game: Game) -> None: + actor = game.player if self.target == "player" else game.find_npc(self.target) + if actor is None: + return + actor.gold = max(0, actor.gold + self.amount) + label = f"[Gold] {actor.name} now has {actor.gold} gold" + if self.reason: + print(f"{self.reason}") + print(label) + elif self.target == "player": + if self.amount > 0: + print(f"You gain {self.amount} gold.") + elif self.amount < 0: + print(f"You lose {-self.amount} gold.") + print(label) + + +class RemoveItemEffect(Effect): + def __init__( + self, item_id: str, *, target: str = "player", text: str | None = None + ): + self.item_id = item_id + self.target = target + self.text = text + + def apply(self, game: Game) -> None: + actor = game.player if self.target == "player" else game.find_npc(self.target) + if actor is None: + return + if actor.remove_item(self.item_id) and self.text: + print(self.text) + + +class RemoveRandomItemEffect(Effect): + def __init__(self, *, target: str = "player", text_prefix: str = "You lose"): + self.target = target + self.text_prefix = text_prefix + + def apply(self, game: Game) -> None: + actor = game.player if self.target == "player" else game.find_npc(self.target) + if actor is None or not actor.inventory: + return + item_id = random.choice(actor.inventory) + actor.remove_item(item_id) + print(f"{self.text_prefix}: {game.item_name(item_id)}") + + +class DamageCharacterEffect(Effect): + def __init__(self, target: str, amount: int, text: str | None = None): + self.target = target + self.amount = amount + self.text = text + + def apply(self, game: Game) -> None: + actor: Character | None + if self.target == "player": + actor = game.player + else: + actor = game.find_npc(self.target) + if actor is None: + return + if self.text: + print(self.text) + actual = actor.take_damage(self.amount, game.world.item_db) + print(f"{actor.name} takes {actual} damage.") + if isinstance(actor, NPC) and not actor.is_alive() and not actor.defeated: + actor.on_defeat(game) + + +class MovePlayerEffect(Effect): + def __init__(self, destination: str, text: str | None = None): + self.destination = destination + self.text = text + + def apply(self, game: Game) -> None: + if self.text: + print(self.text) + game.player.location = self.destination + game.flags.add(f"visited:{self.destination}") + game.world.on_location_enter(game, game.current_location()) + + +class BlockPathEffect(Effect): + def __init__(self, location_key: str, direction: str, reason: str): + self.location_key = location_key + self.direction = direction + self.reason = reason + + def apply(self, game: Game) -> None: + game.world.block_path(self.location_key, self.direction, self.reason) + + +class UnblockPathEffect(Effect): + def __init__(self, location_key: str, direction: str): + self.location_key = location_key + self.direction = direction + + def apply(self, game: Game) -> None: + game.world.unblock_path(self.location_key, self.direction) + + +class ConditionalEffect(Effect): + def __init__( + self, + *, + required_flags: set[str] | None = None, + blocked_flags: set[str] | None = None, + required_items: list[str] | None = None, + success_effect: Effect, + failure_effect: Effect | None = None, + ): + self.required_flags = required_flags or set() + self.blocked_flags = blocked_flags or set() + self.required_items = required_items or [] + self.success_effect = success_effect + self.failure_effect = failure_effect + + def apply(self, game: Game) -> None: + ok = True + if not self.required_flags.issubset(game.flags): + ok = False + if self.blocked_flags.intersection(game.flags): + ok = False + if not game.player.has_items(self.required_items): + ok = False + if ok: + self.success_effect.apply(game) + elif self.failure_effect is not None: + self.failure_effect.apply(game) + + +class ChanceEffect(Effect): + def __init__( + self, + chance: float, + success_effect: Effect, + failure_effect: Effect | None = None, + ): + self.chance = chance + self.success_effect = success_effect + self.failure_effect = failure_effect + + def apply(self, game: Game) -> None: + if random.random() < self.chance: + self.success_effect.apply(game) + elif self.failure_effect is not None: + self.failure_effect.apply(game) + + +class CompositeEffect(Effect): + def __init__(self, *effects: Effect): + self.effects = list(effects) + + def apply(self, game: Game) -> None: + for effect in self.effects: + effect.apply(game) + + +# ============================================================ +# Controllers and characters +# ============================================================ + + +class Controller: + def choose_combat_action( + self, game: Game, actor: Character, opponent: Character + ) -> tuple[str, str | None]: + raise NotImplementedError + + +class PlayerController(Controller): + def choose_combat_action( + self, game: Game, actor: Character, opponent: Character + ) -> tuple[str, str | None]: + options = [ + {"kind": "attack", "text": "Attack", "value": None}, + {"kind": "defend", "text": "Defend", "value": None}, + {"kind": "item", "text": "Use consumable", "value": None}, + {"kind": "run", "text": "Run", "value": None}, + {"kind": "surrender", "text": "Surrender", "value": None}, + ] + choice = game.choose(options, prompt="Choose a combat action: ") + if choice["kind"] == "item": + consumables = actor.consumable_items(game.world.item_db) + if not consumables: + print("You have no consumables to use.") + return "none", None + item_options = [] + for item_id in consumables: + item = game.world.item(item_id) + item_options.append( + { + "kind": "use", + "text": f"Use {item.name} (+{item.healing} HP)", + "value": item_id, + } + ) + item_options.append({"kind": "back", "text": "Back", "value": None}) + item_choice = game.choose(item_options, prompt="Choose an item: ") + if item_choice["kind"] == "back": + return "none", None + return "item", item_choice["value"] + return choice["kind"], choice["value"] + + +class AIController(Controller): + def choose_combat_action( + self, game: Game, actor: Character, opponent: Character + ) -> tuple[str, str | None]: + consumables = actor.consumable_items(game.world.item_db) + hp_ratio = actor.health / max(1, actor.total_max_hp(game.world.item_db)) + if consumables and hp_ratio <= 0.35 and random.random() < 0.60: + return "item", consumables[0] + if isinstance(actor, NPC) and actor.aggression < 35 and random.random() < 0.25: + return "defend", None + return "attack", None + + +class Character: + EQUIPMENT_SLOTS = ("weapon", "armor", "charm") + + def __init__( + self, + name: str, + base_stats: Stats, + controller: Controller, + *, + location: str | None = None, + gold: int = 0, + ): + self.name = name + self.base_stats = base_stats + self.controller = controller + self.location = location + self.health = base_stats.max_hp + self.gold = gold + self.inventory: list[str] = [] + self.equipment: dict[str, str | None] = { + slot: None for slot in self.EQUIPMENT_SLOTS + } + + def is_alive(self) -> bool: + return self.health > 0 + + def item_counts(self) -> Counter: + return Counter(self.inventory) + + def has_item(self, item_id: str) -> bool: + return item_id in self.inventory + + def has_items(self, item_ids: list[str]) -> bool: + owned = self.item_counts() + need = Counter(item_ids) + return all(owned[item_id] >= count for item_id, count in need.items()) + + def add_item(self, item_id: str) -> None: + self.inventory.append(item_id) + + def add_items(self, item_ids: list[str]) -> None: + for item_id in item_ids: + self.add_item(item_id) + + def remove_item(self, item_id: str) -> bool: + if item_id in self.inventory: + self.inventory.remove(item_id) + for slot, equipped in self.equipment.items(): + if equipped == item_id: + self.equipment[slot] = None + return True + return False + + def remove_items(self, item_ids: list[str]) -> bool: + if not self.has_items(item_ids): + return False + for item_id in item_ids: + self.remove_item(item_id) + return True + + def equipped_items(self, item_db: dict[str, Item]) -> list[Item]: + return [ + item_db[item_id] + for item_id in self.equipment.values() + if item_id is not None + ] + + def total_max_hp(self, item_db: dict[str, Item]) -> int: + return self.base_stats.max_hp + sum( + item.hp_bonus for item in self.equipped_items(item_db) + ) + + def total_defense(self, item_db: dict[str, Item]) -> int: + return self.base_stats.defense + sum( + item.defense_bonus for item in self.equipped_items(item_db) + ) + + def attack_range(self, item_db: dict[str, Item]) -> tuple[int, int]: + bonus = sum(item.power_bonus for item in self.equipped_items(item_db)) + return self.base_stats.attack_min + bonus, self.base_stats.attack_max + bonus + + def roll_attack_damage(self, item_db: dict[str, Item]) -> int: + low, high = self.attack_range(item_db) + return random.randint(low, high) + + def take_damage(self, raw_damage: int, item_db: dict[str, Item]) -> int: + actual = max(1, raw_damage - self.total_defense(item_db)) + self.health = max(0, self.health - actual) + return actual + + def heal(self, amount: int, item_db: dict[str, Item]) -> int: + before = self.health + self.health = min(self.total_max_hp(item_db), self.health + amount) + return self.health - before + + def clamp_health(self, item_db: dict[str, Item]) -> None: + self.health = min(self.health, self.total_max_hp(item_db)) + + def equippable_items(self, item_db: dict[str, Item]) -> list[str]: + result = [] + seen = set() + for item_id in self.inventory: + if item_id in seen: + continue + seen.add(item_id) + if item_db[item_id].slot in self.EQUIPMENT_SLOTS: + result.append(item_id) + return result + + def consumable_items(self, item_db: dict[str, Item]) -> list[str]: + result = [] + seen = set() + for item_id in self.inventory: + if item_id in seen: + continue + seen.add(item_id) + if item_db[item_id].healing > 0: + result.append(item_id) + return result + + def equip(self, item_id: str, item_db: dict[str, Item]) -> tuple[bool, str]: + if item_id not in self.inventory: + return False, f"{self.name} does not have that item." + item = item_db[item_id] + if item.slot not in self.EQUIPMENT_SLOTS: + return False, f"{item.name} cannot be equipped." + previous = self.equipment[item.slot] + self.equipment[item.slot] = item_id + self.clamp_health(item_db) + if previous is None: + return True, f"Equipped {item.name} in {item.slot} slot." + return True, f"Equipped {item.name}, replacing {item_db[previous].name}." + + def unequip(self, slot: str, item_db: dict[str, Item]) -> tuple[bool, str]: + if slot not in self.EQUIPMENT_SLOTS: + return False, "Invalid equipment slot." + current = self.equipment[slot] + if current is None: + return False, f"Nothing is equipped in {slot}." + item_name = item_db[current].name + self.equipment[slot] = None + self.clamp_health(item_db) + return True, f"Unequipped {item_name}." + + def use_consumable( + self, item_id: str, item_db: dict[str, Item] + ) -> tuple[bool, str, int]: + if item_id not in self.inventory: + return False, "Item not in inventory.", 0 + item = item_db[item_id] + if item.healing <= 0: + return False, f"{item.name} is not a consumable healing item.", 0 + self.remove_item(item_id) + healed = self.heal(item.healing, item_db) + return True, f"Used {item.name}.", healed + + def show_status(self, item_db: dict[str, Item], *, bounty: int = 0) -> None: + low, high = self.attack_range(item_db) + divider("STATUS") + print(f"Name: {self.name}") + print(f"Health: {self.health}/{self.total_max_hp(item_db)}") + print(f"Attack: {low}-{high}") + print(f"Defense: {self.total_defense(item_db)}") + print(f"Gold: {self.gold}") + print(f"Bounty: {bounty}") + print("\nEquipment:") + for slot in self.EQUIPMENT_SLOTS: + item_id = self.equipment[slot] + label = item_db[item_id].name if item_id is not None else "empty" + print(f"- {slot}: {label}") + print("\nInventory:") + if not self.inventory: + print("- empty") + else: + counts = Counter(self.inventory) + seen = set() + for item_id in self.inventory: + if item_id in seen: + continue + seen.add(item_id) + suffix = f" x{counts[item_id]}" if counts[item_id] > 1 else "" + print(f"- {item_db[item_id].name}{suffix}") + + +# ============================================================ +# Shared NPC model +# ============================================================ + + +class NPC(Character): + def __init__( + self, + name: str, + base_stats: Stats, + *, + description: str = "", + tags: set[str] | None = None, + aggression: int = 0, + courage: int = 50, + willingness_to_trade: int = 0, + hostile: bool = False, + dialogue_topics: list[DialogueTopic] | None = None, + trade_offers: list[TradeOffer] | None = None, + riddle: Riddle | None = None, + surrender_at_ratio: float | None = None, + surrender_lines: list[str] | None = None, + surrender_trade_offers: list[TradeOffer] | None = None, + peaceful_after_surrender: bool = True, + surrender_accept_lines: list[str] | None = None, + surrender_reject_lines: list[str] | None = None, + surrender_accept_effect: Effect | None = None, + defeat_lines: list[str] | None = None, + reward_items: list[str] | None = None, + reward_flags: set[str] | None = None, + reward_gold: int = 0, + persistent: bool = True, + inventory: list[str] | None = None, + equipment: dict[str, str | None] | None = None, + gold: int = 0, + ): + super().__init__( + name=name, base_stats=base_stats, controller=AIController(), gold=gold + ) + self.description = description + self.tags = tags or set() + self.aggression = aggression + self.courage = courage + self.willingness_to_trade = willingness_to_trade + self.hostile = hostile + self.dialogue_topics = dialogue_topics or [] + self.trade_offers = trade_offers or [] + self.riddle = riddle + self.surrender_at_ratio = surrender_at_ratio + self.surrender_lines = surrender_lines or [] + self.surrender_trade_offers = surrender_trade_offers or [] + self.peaceful_after_surrender = peaceful_after_surrender + self.surrender_accept_lines = surrender_accept_lines or [] + self.surrender_reject_lines = surrender_reject_lines or [] + self.surrender_accept_effect = surrender_accept_effect + self.defeat_lines = defeat_lines or [] + self.reward_items = reward_items or [] + self.reward_flags = reward_flags or set() + self.reward_gold = reward_gold + self.persistent = persistent + self.used_topics: set[str] = set() + self.completed_trades: set[int] = set() + self.surrendered = False + self.riddle_solved = False + self.defeated = False + + for item_id in inventory or []: + self.add_item(item_id) + if equipment: + for slot, item_id in equipment.items(): + self.equipment[slot] = item_id + + def is_guard(self) -> bool: + return "guard" in self.tags or "law" in self.tags + + def mood_label(self) -> str: + if self.defeated: + return "defeated" + if self.surrendered: + return "surrendered" + if self.hostile: + return "hostile" + if self.willingness_to_trade >= 70: + return "open to trade" + if self.aggression >= 60: + return "dangerous" + return "neutral" + + def menu_text(self) -> str: + return f"Approach {self.name}" + + def available_topics(self, game: Game) -> list[DialogueTopic]: + results = [] + for topic in self.dialogue_topics: + if topic.once and topic.key in self.used_topics: + continue + if not topic.required_flags.issubset(game.flags): + continue + if topic.blocked_flags.intersection(game.flags): + continue + if not game.player.has_items(topic.required_items): + continue + results.append(topic) + return results + + def available_trade_offers(self, game: Game) -> list[tuple[int, TradeOffer]]: + results = [] + for index, offer in enumerate(self.trade_offers): + if not offer.repeatable and index in self.completed_trades: + continue + if not offer.required_flags.issubset(game.flags): + continue + if offer.blocked_flags.intersection(game.flags): + continue + if self.gold < offer.gives_gold: + continue + if not self.has_items(offer.gives_items): + continue + results.append((index, offer)) + return results + + def can_talk(self, game: Game) -> bool: + return self.is_alive() and bool(self.available_topics(game)) + + def can_trade(self, game: Game) -> bool: + if not self.is_alive(): + return False + if not self.available_trade_offers(game): + return False + if self.surrendered: + return True + return self.willingness_to_trade >= 30 and not self.hostile + + def can_riddle(self) -> bool: + return self.is_alive() and self.riddle is not None and not self.riddle_solved + + def can_attack(self) -> bool: + return self.is_alive() + + def can_loot(self) -> bool: + return self.defeated and (bool(self.inventory) or self.gold > 0) + + def on_player_attack(self, game: Game) -> None: + if not self.hostile and not self.defeated: + if self.is_guard(): + ChangeBountyEffect(75, f"Assaulting guard {self.name}").apply(game) + elif self.aggression < 50: + ChangeBountyEffect(25, f"Assaulting {self.name}").apply(game) + self.hostile = True + self.aggression = max(self.aggression, 70) + + def maybe_surrender(self, game: Game) -> bool: + if self.surrendered or self.surrender_at_ratio is None or not self.is_alive(): + return False + ratio = self.health / max(1, self.total_max_hp(game.world.item_db)) + if ratio > self.surrender_at_ratio: + return False + self.surrendered = True + if self.peaceful_after_surrender: + self.hostile = False + self.willingness_to_trade = max(self.willingness_to_trade, 80) + self.trade_offers.extend(self.surrender_trade_offers) + for line in self.surrender_lines: + say(self.name, line) + return True + + def will_accept_player_surrender(self, game: Game) -> bool: + if self.defeated or not self.is_alive(): + return False + if self.is_guard(): + return True + if self.surrendered: + return True + return self.aggression < 90 or self.willingness_to_trade > 40 + + def handle_player_surrender(self, game: Game) -> bool: + if not self.will_accept_player_surrender(game): + lines = self.surrender_reject_lines or ["No. This ends here."] + for line in lines: + say(self.name, line) + self.hostile = True + return False + + lines = self.surrender_accept_lines or ["Drop your guard and back away slowly."] + for line in lines: + say(self.name, line) + + self.hostile = False + if self.surrender_accept_effect is not None: + self.surrender_accept_effect.apply(game) + else: + if self.aggression >= 70: + if game.player.gold > 0: + stolen = min(game.player.gold, max(5, game.player.gold // 2)) + game.player.gold -= stolen + self.gold += stolen + print(f"{self.name} takes {stolen} gold from you.") + elif game.player.inventory: + RemoveRandomItemEffect(text_prefix=f"{self.name} takes").apply(game) + game.enter_mode("exploration") + if game.mode == "combat": + game.enter_mode("exploration") + return True + + def on_defeat(self, game: Game) -> None: + if self.defeated: + return + self.defeated = True + self.hostile = False + self.health = 0 + + if self.surrendered: + ChangeBountyEffect(100, f"Killing surrendering {self.name}").apply(game) + elif self.is_guard(): + ChangeBountyEffect(150, f"Killing guard {self.name}").apply(game) + elif self.aggression < 50: + ChangeBountyEffect(40, f"Killing non-hostile {self.name}").apply(game) + + for line in self.defeat_lines: + print(line) + for item_id in self.reward_items: + game.give_player_item(item_id) + if self.reward_gold: + game.player.gold += self.reward_gold + print(f"You gain {self.reward_gold} gold.") + for flag in self.reward_flags: + game.flags.add(flag) + game.world.on_npc_defeated(game, self) + + +class MerchantNPC(NPC): + def __init__(self, name: str, base_stats: Stats | None = None, **kwargs): + kwargs.setdefault("aggression", 10) + kwargs.setdefault("courage", 40) + kwargs.setdefault("willingness_to_trade", 90) + kwargs.setdefault("hostile", False) + super().__init__( + name=name, base_stats=base_stats or Stats(35, 4, 6, 1), **kwargs + ) + + +class BanditNPC(NPC): + def __init__(self, name: str, base_stats: Stats | None = None, **kwargs): + kwargs.setdefault("aggression", 70) + kwargs.setdefault("courage", 35) + kwargs.setdefault("willingness_to_trade", 10) + kwargs.setdefault("hostile", False) + kwargs.setdefault("surrender_at_ratio", 0.25) + super().__init__( + name=name, base_stats=base_stats or Stats(28, 5, 9, 1), **kwargs + ) + + +class GuardianNPC(NPC): + def __init__(self, name: str, base_stats: Stats | None = None, **kwargs): + kwargs.setdefault("aggression", 50) + kwargs.setdefault("courage", 90) + kwargs.setdefault("willingness_to_trade", 0) + kwargs.setdefault("hostile", False) + super().__init__( + name=name, base_stats=base_stats or Stats(38, 7, 11, 2), **kwargs + ) + + +class ElderNPC(NPC): + def __init__(self, name: str, base_stats: Stats | None = None, **kwargs): + kwargs.setdefault("aggression", 5) + kwargs.setdefault("courage", 20) + kwargs.setdefault("willingness_to_trade", 15) + kwargs.setdefault("hostile", False) + super().__init__( + name=name, base_stats=base_stats or Stats(40, 4, 7, 0), **kwargs + ) + + +class GuardNPC(NPC): + def __init__(self, name: str, base_stats: Stats | None = None, **kwargs): + kwargs.setdefault("tags", {"guard", "law"}) + kwargs.setdefault("aggression", 40) + kwargs.setdefault("courage", 85) + kwargs.setdefault("willingness_to_trade", 0) + kwargs.setdefault("hostile", False) + super().__init__( + name=name, base_stats=base_stats or Stats(45, 6, 9, 2), **kwargs + ) + + +# ============================================================ +# Features / world objects +# ============================================================ + + +class Feature: + def __init__(self, name: str, verb: str = "Inspect"): + self.name = name + self.verb = verb + + def menu_text(self) -> str: + return f"{self.verb} {self.name}" + + def interact(self, game: Game) -> None: + print(f"Nothing happens when you interact with {self.name}.") + + +class TextFeature(Feature): + def __init__(self, name: str, text: str, verb: str = "Inspect"): + super().__init__(name=name, verb=verb) + self.text = text + + def interact(self, game: Game) -> None: + print("\n" + self.text) + + +class ScriptedFeature(Feature): + def __init__( + self, + name: str, + *, + verb: str = "Use", + required_flags: set[str] | None = None, + blocked_text: str = "Nothing happens.", + once_flag: str | None = None, + first_effect: Effect | None = None, + repeat_effect: Effect | None = None, + ): + super().__init__(name=name, verb=verb) + self.required_flags = required_flags or set() + self.blocked_text = blocked_text + self.once_flag = once_flag + self.first_effect = first_effect + self.repeat_effect = repeat_effect + + def interact(self, game: Game) -> None: + if not self.required_flags.issubset(game.flags): + print(self.blocked_text) + return + if self.once_flag and self.once_flag in game.flags: + if self.repeat_effect is not None: + self.repeat_effect.apply(game) + else: + print("Nothing more happens.") + return + if self.first_effect is not None: + self.first_effect.apply(game) + if self.once_flag: + game.flags.add(self.once_flag) + if self.once_flag is None and self.repeat_effect is not None: + self.repeat_effect.apply(game) + + +# ============================================================ +# Encounters +# ============================================================ + + +class EncounterRule: + def __init__( + self, + name: str, + *, + locations: set[str], + chance: float, + handler: Callable[[Game], bool], + required_flags: set[str] | None = None, + blocked_flags: set[str] | None = None, + once_flag: str | None = None, + predicate: Callable[[Game], bool] | None = None, + ): + self.name = name + self.locations = locations + self.chance = chance + self.handler = handler + self.required_flags = required_flags or set() + self.blocked_flags = blocked_flags or set() + self.once_flag = once_flag + self.predicate = predicate + + def try_trigger(self, game: Game) -> bool: + location = game.current_location() + if location.key not in self.locations: + return False + if self.once_flag and self.once_flag in game.flags: + return False + if not self.required_flags.issubset(game.flags): + return False + if self.blocked_flags.intersection(game.flags): + return False + if self.predicate is not None and not self.predicate(game): + return False + if random.random() > self.chance: + return False + triggered = self.handler(game) + if triggered and self.once_flag: + game.flags.add(self.once_flag) + return triggered + + +# ============================================================ +# World base class +# ============================================================ + + +class World: + def __init__(self, name: str): + self.name = name + self.locations: dict[str, Location] = {} + self.item_db: dict[str, Item] = {} + self.encounter_rules: list[EncounterRule] = [] + + def define_item(self, item: Item) -> None: + self.item_db[item.item_id] = item + + def item(self, item_id: str) -> Item: + return self.item_db[item_id] + + def add_location(self, key: str, name: str, description: str) -> None: + self.locations[key] = Location(key=key, name=name, description=description) + + def get(self, key: str) -> Location: + return self.locations[key] + + def connect( + self, + source: str, + direction: str, + destination: str, + *, + requires_item: str | None = None, + requires_flag: str | None = None, + blocked: bool = False, + blocked_text: str = "That path is blocked.", + warning_text: str | None = None, + on_attempt_effect: Effect | None = None, + on_blocked_effect: Effect | None = None, + on_success_effect: Effect | None = None, + ) -> None: + self.locations[source].exits.append( + Exit( + direction=direction, + destination=destination, + requires_item=requires_item, + requires_flag=requires_flag, + blocked=blocked, + blocked_text=blocked_text, + warning_text=warning_text, + on_attempt_effect=on_attempt_effect, + on_blocked_effect=on_blocked_effect, + on_success_effect=on_success_effect, + ) + ) + + def connect_two_way( + self, a: str, dir_a_to_b: str, b: str, dir_b_to_a: str, **kwargs + ) -> None: + self.connect(a, dir_a_to_b, b, **kwargs) + self.connect(b, dir_b_to_a, a) + + def place_item(self, location_key: str, item_id: str) -> None: + self.locations[location_key].items.append(item_id) + + def place_feature(self, location_key: str, feature: Feature) -> None: + self.locations[location_key].features.append(feature) + + def place_npc(self, location_key: str, npc: NPC) -> None: + npc.location = location_key + self.locations[location_key].npcs.append(npc) + + def add_encounter_rule(self, rule: EncounterRule) -> None: + self.encounter_rules.append(rule) + + def find_exit(self, location_key: str, direction: str) -> Exit | None: + for exit_obj in self.locations[location_key].exits: + if exit_obj.direction == direction: + return exit_obj + return None + + def block_path(self, location_key: str, direction: str, reason: str) -> None: + exit_obj = self.find_exit(location_key, direction) + if exit_obj is not None: + exit_obj.blocked = True + exit_obj.blocked_text = reason + + def unblock_path(self, location_key: str, direction: str) -> None: + exit_obj = self.find_exit(location_key, direction) + if exit_obj is not None: + exit_obj.blocked = False + + def on_location_enter(self, game: Game, location: Location) -> None: + pass + + def on_item_picked_up(self, game: Game, item_id: str) -> None: + pass + + def on_npc_defeated(self, game: Game, npc: NPC) -> None: + pass + + def try_start_of_exploration_encounter(self, game: Game) -> bool: + for rule in self.encounter_rules: + if rule.try_trigger(game): + return True + return False + + +# ============================================================ +# Concrete world +# ============================================================ + + +class StarCrystalWorld(World): + def __init__(self): + super().__init__(name="The Star Crystal") + self.build_items() + self.build_locations() + self.build_paths() + self.build_features() + self.build_npcs() + self.scatter_items() + self.build_encounters() + + def build_items(self) -> None: + self.define_item( + Item("village_map", "Village Map", "A hand-drawn map of the valley.") + ) + self.define_item( + Item( + "moonleaf_herb", + "Moonleaf Herb", + "A silver-blue herb that smells sharp and fresh.", + ) + ) + self.define_item( + Item("rope", "Rope", "Strong rope left behind at an old campsite.") + ) + self.define_item( + Item("lantern", "Lantern", "A sturdy lantern. Essential in dark places.") + ) + self.define_item( + Item( + "silver_key", + "Silver Key", + "A polished key shaped like a crescent moon.", + ) + ) + self.define_item( + Item( + "star_crystal", + "Star Crystal", + "A warm crystal filled with light.", + set_flags_on_pickup={"has_star_crystal"}, + ) + ) + + self.define_item( + Item( + "iron_sword", + "Iron Sword", + "Rusty, but still dependable.", + slot="weapon", + power_bonus=4, + ) + ) + self.define_item( + Item( + "guard_coat", + "Guard Coat", + "Reinforced cloth armor.", + slot="armor", + defense_bonus=2, + hp_bonus=10, + ) + ) + self.define_item( + Item( + "lucky_coin", + "Lucky Coin", + "A coin etched with a star. It makes you strangely steady.", + slot="charm", + defense_bonus=1, + ) + ) + self.define_item( + Item( + "hunter_spear", + "Hunter Spear", + "A balanced spear made to keep biting creatures at a distance.", + slot="weapon", + power_bonus=6, + ) + ) + + self.define_item( + Item( + "health_tonic", + "Health Tonic", + "A red tonic that restores health.", + healing=20, + tags={"consumable"}, + ) + ) + self.define_item( + Item( + "greater_tonic", + "Greater Tonic", + "A stronger restorative draught.", + healing=35, + tags={"consumable"}, + ) + ) + + def build_locations(self) -> None: + self.add_location( + "village", + "Sunmeadow Village", + "A quiet village square surrounds a dry fountain. Traders, guards, and villagers drift across the square.", + ) + self.add_location( + "crossroads", + "Old Crossroads", + "Four dirt paths meet under a leaning wooden sign.", + ) + self.add_location( + "forest", + "Whispering Forest", + "Tall trees sway overhead. Moss, roots, and old camp supplies lie under the leaves.", + ) + self.add_location( + "garden", + "Hidden Garden", + "A secret clearing full of bright flowers and soft green light.", + ) + self.add_location( + "lake", + "Mirror Lake", + "A glassy lake reflects the sky. A weathered fisherman waits by the dock.", + ) + self.add_location( + "cave_entrance", + "Cave Entrance", + "A crack in the hillside opens into darkness. Cold air spills out.", + ) + self.add_location( + "cave_depths", + "Crystal Cave", + "Blue crystals glow faintly from the cave walls.", + ) + self.add_location( + "ruins", + "Moonstone Ruins", + "Broken pillars and ancient arches rise from a hilltop.", + ) + self.add_location( + "tower_gate", + "Tower Gate", + "An iron gate blocks the tower stairs. The keyhole is shaped like a crescent moon.", + ) + self.add_location( + "tower_top", + "Tower Summit", + "Wind curls around the broken top of the tower. A glowing crystal rests on a pedestal.", + ) + self.add_location( + "jail", + "Village Jail", + "A cramped stone cell. The door is locked from the outside, and there is no key within reach.", + ) + + def build_paths(self) -> None: + self.connect_two_way("village", "north", "crossroads", "south") + self.connect_two_way("crossroads", "west", "forest", "east") + self.connect_two_way("crossroads", "east", "lake", "west") + + self.connect( + "crossroads", + "north", + "ruins", + on_success_effect=ConditionalEffect( + blocked_flags={"ruins_ambush_triggered"}, + success_effect=CompositeEffect( + PrintEffect( + "As you head toward the ruins, a loose stone gives way underfoot and you stumble." + ), + DamageCharacterEffect("player", 4), + SetFlagEffect("ruins_ambush_triggered"), + ), + ), + ) + self.connect("ruins", "south", "crossroads") + + self.connect_two_way("forest", "west", "garden", "east") + + self.connect( + "forest", + "north", + "cave_entrance", + warning_text="The bramble path ahead looks painful. Push through anyway?", + on_success_effect=ChanceEffect( + 0.60, + CompositeEffect( + PrintEffect( + "Thorns rake your arms and tear at your clothes as you force your way through." + ), + DamageCharacterEffect("player", 6), + ), + PrintEffect( + "You find a careful route through the brambles and avoid the worst of them." + ), + ), + ) + self.connect("cave_entrance", "south", "forest") + + self.connect( + "cave_entrance", + "north", + "cave_depths", + requires_item="lantern", + blocked_text="It is too dark to enter safely. You need a Lantern.", + on_blocked_effect=PrintEffect( + "Shadows shift in the crack ahead. You reconsider." + ), + ) + self.connect("cave_depths", "south", "cave_entrance") + + self.connect( + "ruins", + "north", + "tower_gate", + requires_item="rope", + blocked_text="The path ahead is broken. You need a Rope to cross safely.", + warning_text="The crossing is narrow and unstable. Go on?", + on_success_effect=ChanceEffect( + 0.30, + CompositeEffect( + PrintEffect( + "The old crossing gives way beneath you for a moment. Your pack slams against the stone." + ), + DamageCharacterEffect("player", 5), + ConditionalEffect( + required_items=["health_tonic"], + success_effect=RemoveItemEffect( + "health_tonic", + text="One of your Health Tonics shatters on the rocks.", + ), + ), + ), + PrintEffect("You cross the broken path without incident."), + ), + on_blocked_effect=PrintEffect( + "The gap ahead looks fatal without proper support." + ), + ) + self.connect("tower_gate", "south", "ruins") + + self.connect( + "tower_gate", + "up", + "tower_top", + requires_item="silver_key", + blocked_text="The gate is locked. You need the Silver Key.", + on_success_effect=ConditionalEffect( + blocked_flags={"tower_entry_event"}, + success_effect=CompositeEffect( + PrintEffect( + "Ancient light flickers along the stairwell as the gate opens." + ), + SetFlagEffect("tower_entry_event"), + ), + ), + ) + self.connect("tower_top", "down", "tower_gate") + + def build_features(self) -> None: + self.place_feature( + "village", + ScriptedFeature( + name="fountain", + verb="Rest at", + first_effect=HealPlayerEffect( + amount=15, + heal_text="You sit beside the fountain and catch your breath.", + full_text="You already feel fully rested.", + ), + ), + ) + self.place_feature( + "crossroads", + TextFeature( + name="wooden sign", + text=( + "NORTH -> Moonstone Ruins\n" + "EAST -> Mirror Lake\n" + "WEST -> Whispering Forest\n" + "SOUTH -> Sunmeadow Village" + ), + verb="Read", + ), + ) + self.place_feature( + "forest", + ScriptedFeature( + name="whispering trees", + verb="Listen to", + once_flag="forest_hint_seen", + first_effect=PrintEffect( + "A cool breeze seems to whisper: 'Light opens the dark path.'" + ), + repeat_effect=PrintEffect( + "The leaves rustle softly. The forest has already shared its hint." + ), + ), + ) + self.place_feature( + "forest", + ScriptedFeature( + name="unstable hillside", + verb="Climb", + once_flag="landslide_happened", + first_effect=CompositeEffect( + PrintEffect( + [ + "You scramble up the unstable slope.", + "A deep crack echoes through the trees.", + "Rocks crash down and bury the garden trail!", + ] + ), + BlockPathEffect( + "forest", + "west", + "A landslide has buried the trail under loose rocks.", + ), + BlockPathEffect( + "garden", + "east", + "A landslide has buried the trail under loose rocks.", + ), + ), + repeat_effect=PrintEffect( + "The hillside already collapsed. Rocks still cover the trail." + ), + ), + ) + self.place_feature( + "ruins", + ScriptedFeature( + name="old winch", + verb="Turn", + required_flags={"landslide_happened"}, + blocked_text="The mechanism is tied to the collapsed trail, but nothing needs clearing yet.", + once_flag="garden_path_reopened", + first_effect=CompositeEffect( + PrintEffect( + [ + "You turn the ancient winch.", + "Some buried stones shift and part of the trail is cleared.", + ] + ), + UnblockPathEffect("forest", "west"), + UnblockPathEffect("garden", "east"), + ), + repeat_effect=PrintEffect("The old winch has already done all it can."), + ), + ) + self.place_feature( + "cave_depths", + TextFeature( + name="glowing crystals", + text="Their light feels ancient, calm, and strangely familiar. You begin to suspect the tower and village are connected by old magic.", + verb="Study", + ), + ) + self.place_feature( + "jail", + ScriptedFeature( + name="prison cot", + verb="Serve time on", + required_flags={"jailed"}, + blocked_text="There is no sentence to serve right now.", + first_effect=CompositeEffect( + PrintEffect( + [ + "You sit on the narrow cot and wait.", + "Hours pass. Your anger cools. The guard finally opens the cell.", + ] + ), + SetBountyEffect(0), + ClearFlagEffect("jailed"), + MovePlayerEffect( + "village", text="You are released back into Sunmeadow Village." + ), + HealPlayerEffect( + 999, + heal_text="A night's rest mends your wounds.", + full_text="You are already fully rested.", + ), + ), + ), + ) + + def build_npcs(self) -> None: + elder_topics = [ + DialogueTopic( + key="dry_fountain", + title="Ask about the dry fountain", + lines=[ + "Our fountain has run dry, and the tower's shadow grows longer every night.", + "Bring back the Star Crystal, and Sunmeadow may shine again.", + ], + blocked_flags={"quest_started"}, + once=True, + outcome_effect=CompositeEffect( + GivePlayerItemEffect("village_map"), + SetFlagEffect("quest_started"), + ), + ), + DialogueTopic( + key="advice", + title="Ask what to do next", + lines=[ + "Listen to the valley. The people, the ruins, and even the trees know more than they first reveal." + ], + required_flags={"quest_started"}, + blocked_flags={"has_star_crystal", "game_won"}, + ), + DialogueTopic( + key="return_crystal", + title="Return the Star Crystal", + lines=[ + "You found it... the Star Crystal!", + "You raise the crystal above the fountain.", + "Light pours across the square. Water bursts upward. Cheers echo through the village.", + "You saved Sunmeadow Village. You win!", + ], + required_flags={"has_star_crystal"}, + blocked_flags={"game_won"}, + once=True, + outcome_effect=SetFlagEffect("game_won"), + ), + ] + self.place_npc( + "village", + ElderNPC( + "Elder Mira", + description="The village elder watches the dry fountain with worried eyes.", + dialogue_topics=elder_topics, + inventory=["health_tonic"], + defeat_lines=["Elder Mira falls. The square goes silent."], + ), + ) + + self.place_npc( + "village", + GuardNPC( + "Guard Halwen", + description="A village guard in a faded coat watches the square with practiced patience.", + dialogue_topics=[ + DialogueTopic( + key="guard_law", + title="Ask about the law", + lines=[ + "Keep the peace, pay your debts, and do not test my patience.", + "If you draw steel on the innocent, I will answer it.", + ], + ) + ], + surrender_accept_lines=[ + "Drop your weapon and come quietly.", + "You can explain yourself from the cell.", + ], + surrender_accept_effect=CompositeEffect( + SetFlagEffect("jailed"), + MovePlayerEffect( + "jail", + text="Guard Halwen binds your hands and marches you to the village jail.", + ), + ), + inventory=["guard_coat", "health_tonic"], + equipment={"armor": "guard_coat"}, + defeat_lines=[ + "Guard Halwen crashes to the stones. A hush spreads across the square." + ], + gold=12, + ), + ) + + self.place_npc( + "village", + MerchantNPC( + "Merchant Sella", + description="A sharp-eyed merchant has laid out weapons, tonics, and travel gear on a thick canvas cloth.", + dialogue_topics=[ + DialogueTopic( + key="merchant_pitch", + title="Ask what she recommends", + lines=[ + "If you are heading for caves, buy reach and buy confidence.", + "Spiders hate a spear point more than a short blade.", + ], + ) + ], + trade_offers=[ + TradeOffer( + title="Buy Hunter Spear for 12 gold", + wants_gold=12, + gives_items=["hunter_spear"], + ), + TradeOffer( + title="Buy Health Tonic for 6 gold", + wants_gold=6, + gives_items=["health_tonic"], + repeatable=True, + ), + ], + inventory=["hunter_spear", "health_tonic", "health_tonic"], + gold=30, + ), + ) + + fisher_topics = [ + DialogueTopic( + key="lake_gossip", + title="Ask about the lake", + lines=[ + "The lake tells you things if you wait long enough.", + "Mostly warnings. Sometimes luck.", + ], + ), + DialogueTopic( + key="ask_tower", + title="Ask about the tower", + lines=[ + "Dark places do not stay dark forever. Sometimes they only wait for the right light." + ], + ), + ] + fisher_offers = [ + TradeOffer( + "Trade Moonleaf Herb for Lantern", + wants_items=["moonleaf_herb"], + gives_items=["lantern"], + ), + TradeOffer( + "Trade Lucky Coin for Guard Coat", + wants_items=["lucky_coin"], + gives_items=["guard_coat"], + ), + ] + self.place_npc( + "lake", + MerchantNPC( + "Fisher Rowan", + description="A patient fisherman sits on the dock, humming to himself.", + dialogue_topics=fisher_topics, + trade_offers=fisher_offers, + inventory=["lantern", "guard_coat", "health_tonic"], + equipment={"armor": "guard_coat"}, + defeat_lines=[ + "Fisher Rowan drops his rod and backs away from the dock." + ], + gold=8, + ), + ) + + bandit_topics = [ + DialogueTopic( + key="bandit_warning", + title="Ask why he is here", + lines=[ + "Because hidden places collect hidden things.", + "And hidden things can be sold.", + ], + blocked_flags={"bandit_spared"}, + ), + DialogueTopic( + key="bandit_after_surrender", + title="Ask what changed his mind", + lines=["Turns out survival is worth more than pride."], + required_flags={"bandit_spared"}, + ), + ] + bandit_surrender_offers = [ + TradeOffer( + "Trade Moonleaf Herb for Lucky Coin", + wants_items=["moonleaf_herb"], + gives_items=["lucky_coin"], + ), + TradeOffer( + "Trade Rope for Greater Tonic", + wants_items=["rope"], + gives_items=["greater_tonic"], + ), + ] + self.place_npc( + "garden", + BanditNPC( + "Bandit Nox", + description="A lean bandit lurks among the flowers, guarding a hidden stash.", + dialogue_topics=bandit_topics, + surrender_lines=[ + "Wait! Enough! I do not want to die here.", + "If you spare me, we can make a deal.", + ], + surrender_trade_offers=bandit_surrender_offers, + surrender_accept_lines=[ + "Fine. Take what you came for and let me live." + ], + peaceful_after_surrender=True, + inventory=["lucky_coin", "greater_tonic", "health_tonic"], + equipment={"weapon": "iron_sword"}, + defeat_lines=[ + "Bandit Nox drops his bag and limps away through the garden." + ], + reward_flags={"bandit_defeated"}, + reward_gold=10, + gold=15, + ), + ) + + self.place_npc( + "cave_depths", + NPC( + name="Crystal Spider", + base_stats=Stats(max_hp=24, attack_min=5, attack_max=8, defense=2), + description="A giant spider clings to a web of glowing crystal threads.", + tags={"monster"}, + aggression=90, + courage=100, + willingness_to_trade=0, + hostile=False, + surrender_reject_lines=[ + "The spider only clicks its mandibles and closes in." + ], + inventory=["silver_key", "health_tonic"], + defeat_lines=["The Crystal Spider curls up and stops moving."], + reward_flags={"spider_defeated"}, + reward_gold=14, + ), + ) + + guardian_topics = [ + DialogueTopic( + key="guardian_warning", + title="Ask who the crystal belongs to", + lines=[ + "The crystal does not belong to one hand. It belongs to balance." + ], + blocked_flags={"guardian_passed"}, + ), + DialogueTopic( + key="guardian_after_trial", + title="Ask if the trial is complete", + lines=["The trial is complete. The Star Crystal may now be taken."], + required_flags={"guardian_passed"}, + ), + ] + guardian_riddle = Riddle( + intro_lines=["Answer my riddle, and the crystal may be yours."], + question="I am always running, but I never walk. I often murmur, but I never talk. What am I?", + answers=["river", "a river", "stream", "a stream"], + success_lines=["Wisdom and patience walk beside you. Step forward."], + failure_lines=["Not quite. Think more carefully."], + repeat_lines=["The trial is complete. The Star Crystal may now be taken."], + damage_on_failure=12, + set_flags_on_success={"guardian_passed"}, + ) + self.place_npc( + "tower_top", + GuardianNPC( + "Tower Guardian", + description="A glowing spirit hovers beside the pedestal.", + dialogue_topics=guardian_topics, + riddle=guardian_riddle, + surrender_reject_lines=[ + "The trial cannot be escaped. Only answered or endured." + ], + inventory=["star_crystal"], + defeat_lines=["The guardian dissolves into drifting sparks of light."], + reward_flags={"guardian_passed"}, + ), + ) + + def scatter_items(self) -> None: + self.place_item("forest", "moonleaf_herb") + self.place_item("forest", "rope") + self.place_item("forest", "iron_sword") + self.place_item("forest", "health_tonic") + + def build_encounters(self) -> None: + self.add_encounter_rule( + EncounterRule( + "guard_confrontation", + locations={"village"}, + chance=1.0, + blocked_flags={"jailed", "game_won"}, + predicate=lambda game: game.bounty > 0 + and self.guard_alive("Guard Halwen"), + handler=self.handle_guard_confrontation, + ) + ) + self.add_encounter_rule( + EncounterRule( + "forest_small_spider", + locations={"forest"}, + chance=0.30, + blocked_flags={"jailed"}, + predicate=lambda game: game.player.is_alive(), + handler=self.handle_forest_spider_encounter, + ) + ) + + def guard_alive(self, name: str) -> bool: + guard = next( + (npc for npc in self.get("village").npcs if npc.name == name), None + ) + return guard is not None and guard.is_alive() and not guard.defeated + + def handle_guard_confrontation(self, game: Game) -> bool: + guard = next( + (npc for npc in self.get("village").npcs if npc.name == "Guard Halwen"), + None, + ) + if guard is None or guard.defeated or not guard.is_alive(): + return False + action_separator("A guard confronts you") + say( + guard.name, + f"You have a bounty of {game.bounty} gold on your head. Stand down.", + ) + guard.hostile = True + game.enter_mode("combat", npc=guard) + return True + + def handle_forest_spider_encounter(self, game: Game) -> bool: + spider = NPC( + name="Small Spider", + base_stats=Stats(max_hp=10, attack_min=3, attack_max=5, defense=0), + description="A skittering spider drops from a branch, startled by your movement.", + tags={"monster", "encounter"}, + aggression=80, + courage=20, + willingness_to_trade=0, + hostile=True, + surrender_reject_lines=["The spider only hisses and darts forward."], + defeat_lines=["The small spider curls up and goes still among the leaves."], + reward_gold=8, + persistent=False, + ) + action_separator("A forest encounter") + print("A small spider drops from the branches and rushes toward you!") + game.enter_mode("combat", npc=spider) + return True + + def on_item_picked_up(self, game: Game, item_id: str) -> None: + if item_id == "star_crystal": + game.flags.add("has_star_crystal") + + def on_npc_defeated(self, game: Game, npc: NPC) -> None: + if npc.name == "Crystal Spider": + print( + "The crystal web tears apart, revealing a safer route through the cave." + ) + + def on_location_enter(self, game: Game, location: Location) -> None: + if location.key == "tower_top" and "first_tower_visit" not in game.flags: + game.flags.add("first_tower_visit") + print( + "The air grows brighter and heavier at the same time. Something ancient is waiting here." + ) + + +# ============================================================ +# Engines +# ============================================================ + + +class Engine: + def run(self, game: Game) -> None: + raise NotImplementedError + + +class ExplorationEngine(Engine): + def run(self, game: Game) -> None: + if game.world.try_start_of_exploration_encounter(game): + return + + location = game.current_location() + divider(f"{game.world.name} - {location.name}") + print(location.description) + print( + f"\n[HP {game.player.health}/{game.player.total_max_hp(game.world.item_db)}] [Gold {game.player.gold}] [Bounty {game.bounty}]" + ) + + if location.items: + print("\nItems here:") + for item_id in location.items: + print(f"- {game.item_name(item_id)}") + + if location.npcs: + print("\nPeople / creatures here:") + for npc in location.npcs: + print(f"- {npc.name} ({npc.mood_label()})") + + if location.features: + print("\nNotable features:") + for feature in location.features: + print(f"- {feature.name}") + + game.show_goal() + + choices = [] + jailed = "jailed" in game.flags and location.key == "jail" + if not jailed: + for exit_obj in location.exits: + destination = game.world.get(exit_obj.destination) + text = f"Go {exit_obj.direction} to {destination.name}" + if game.exit_is_blocked(exit_obj): + text += " [blocked]" + elif exit_obj.warning_text: + text += " [risky]" + choices.append({"kind": "move", "text": text, "value": exit_obj}) + + for item_id in location.items: + choices.append( + { + "kind": "take_item", + "text": f"Take {game.item_name(item_id)}", + "value": item_id, + } + ) + + for npc in location.npcs: + choices.append({"kind": "npc", "text": npc.menu_text(), "value": npc}) + + for feature in location.features: + choices.append( + {"kind": "feature", "text": feature.menu_text(), "value": feature} + ) + + if not jailed: + choices.append({"kind": "menu", "text": "Open menu", "value": None}) + + choice = game.choose(choices, prompt="\nWhat do you want to do? ") + + if choice["kind"] == "move": + game.handle_move(choice["value"]) + elif choice["kind"] == "take_item": + game.handle_take_item(choice["value"]) + elif choice["kind"] == "npc": + game.enter_mode("npc", npc=choice["value"]) + elif choice["kind"] == "feature": + choice["value"].interact(game) + elif choice["kind"] == "menu": + game.enter_mode("menu") + + +class NPCInteractionEngine(Engine): + def run(self, game: Game) -> None: + npc: NPC = game.context["npc"] + + divider(f"Interacting with {npc.name}") + print(npc.description) + print(f"Mood: {npc.mood_label()}") + print(f"Health: {npc.health}/{npc.total_max_hp(game.world.item_db)}") + print(f"Gold: {npc.gold}") + + options = [] + if npc.can_talk(game): + options.append( + {"kind": "talk", "text": f"Talk to {npc.name}", "value": npc} + ) + if npc.can_trade(game): + options.append( + {"kind": "trade", "text": f"Trade with {npc.name}", "value": npc} + ) + if npc.can_riddle(): + options.append( + {"kind": "riddle", "text": f"Challenge {npc.name}", "value": npc} + ) + if npc.can_attack(): + options.append( + {"kind": "attack", "text": f"Attack {npc.name}", "value": npc} + ) + if npc.can_loot(): + options.append({"kind": "loot", "text": f"Loot {npc.name}", "value": npc}) + options.append({"kind": "back", "text": "Step away", "value": None}) + + choice = game.choose(options, prompt="\nChoose an interaction: ") + kind = choice["kind"] + if kind == "talk": + game.enter_mode("dialogue", npc=npc) + elif kind == "trade": + game.enter_mode("trade", npc=npc) + elif kind == "riddle": + game.enter_mode("riddle", npc=npc) + elif kind == "attack": + npc.on_player_attack(game) + game.enter_mode("combat", npc=npc) + elif kind == "loot": + game.loot_npc(npc) + game.enter_mode("npc", npc=npc) + elif kind == "back": + game.enter_mode("exploration") + + +class DialogueEngine(Engine): + def run(self, game: Game) -> None: + npc: NPC = game.context["npc"] + topics = npc.available_topics(game) + if not topics: + say(npc.name, "I have nothing more to say right now.") + game.enter_mode("npc", npc=npc) + return + + divider(f"Dialogue - {npc.name}") + options = [ + {"kind": "topic", "text": topic.title, "value": topic} for topic in topics + ] + options.append({"kind": "back", "text": "Back", "value": None}) + choice = game.choose(options, prompt="\nChoose a topic: ") + if choice["kind"] == "back": + game.enter_mode("npc", npc=npc) + return + + topic: DialogueTopic = choice["value"] + for line in topic.lines: + say(npc.name, line) + if topic.outcome_effect is not None: + topic.outcome_effect.apply(game) + if topic.once: + npc.used_topics.add(topic.key) + if "game_won" in game.flags: + game.running = False + return + game.enter_mode("npc", npc=npc) + + +class TradeEngine(Engine): + def run(self, game: Game) -> None: + npc: NPC = game.context["npc"] + offers = npc.available_trade_offers(game) + divider(f"Trade - {npc.name}") + print(f"Your gold: {game.player.gold}") + if not offers: + say(npc.name, "I have nothing I can trade right now.") + game.enter_mode("npc", npc=npc) + return + + options = [] + for offer_index, offer in offers: + wants_parts = [] + if offer.wants_gold: + wants_parts.append(f"{offer.wants_gold} gold") + if offer.wants_items: + wants_parts.extend( + game.item_name(item_id) for item_id in offer.wants_items + ) + gives_parts = [] + if offer.gives_gold: + gives_parts.append(f"{offer.gives_gold} gold") + if offer.gives_items: + gives_parts.extend( + game.item_name(item_id) for item_id in offer.gives_items + ) + wants = ", ".join(wants_parts) or "nothing" + gives = ", ".join(gives_parts) or "nothing" + options.append( + { + "kind": "offer", + "text": f"{offer.title} [{wants} -> {gives}]", + "value": (offer_index, offer), + } + ) + options.append({"kind": "back", "text": "Back", "value": None}) + + choice = game.choose(options, prompt="\nChoose a trade: ") + if choice["kind"] == "back": + game.enter_mode("npc", npc=npc) + return + + offer_index, offer = choice["value"] + if game.player.gold < offer.wants_gold: + say(npc.name, f"Come back when you have {offer.wants_gold} gold.") + game.enter_mode("npc", npc=npc) + return + if not game.player.has_items(offer.wants_items): + say( + npc.name, + f"Come back when you have: {', '.join(game.item_name(i) for i in offer.wants_items)}.", + ) + game.enter_mode("npc", npc=npc) + return + if npc.gold < offer.gives_gold or not npc.has_items(offer.gives_items): + say(npc.name, "I cannot complete that trade right now.") + game.enter_mode("npc", npc=npc) + return + + game.player.gold -= offer.wants_gold + npc.gold += offer.wants_gold + npc.gold -= offer.gives_gold + game.player.gold += offer.gives_gold + game.player.remove_items(offer.wants_items) + npc.add_items(offer.wants_items) + npc.remove_items(offer.gives_items) + for item_id in offer.gives_items: + game.give_player_item(item_id) + if not offer.repeatable: + npc.completed_trades.add(offer_index) + say(npc.name, "A fair trade.") + game.enter_mode("npc", npc=npc) + + +class RiddleEngine(Engine): + def run(self, game: Game) -> None: + npc: NPC = game.context["npc"] + riddle = npc.riddle + if riddle is None: + game.enter_mode("npc", npc=npc) + return + + divider(f"Challenge - {npc.name}") + if npc.riddle_solved: + for line in riddle.repeat_lines: + say(npc.name, line) + game.enter_mode("npc", npc=npc) + return + + for line in riddle.intro_lines: + say(npc.name, line) + say(npc.name, riddle.question) + answer = input("Your answer: ").strip().lower() + + if answer in [a.lower() for a in riddle.answers]: + for line in riddle.success_lines: + say(npc.name, line) + npc.riddle_solved = True + for flag in riddle.set_flags_on_success: + game.flags.add(flag) + else: + for line in riddle.failure_lines: + say(npc.name, line) + if riddle.damage_on_failure > 0: + actual = game.player.take_damage( + riddle.damage_on_failure, game.world.item_db + ) + print(f"You take {actual} damage.") + + game.enter_mode("npc", npc=npc) + + +class CombatEngine(Engine): + def run(self, game: Game) -> None: + enemy: NPC = game.context["npc"] + player_defending = False + enemy_defending = False + + divider(f"Combat - {enemy.name}") + print(f"{enemy.name} squares off against you.") + + while ( + game.running + and game.player.is_alive() + and enemy.is_alive() + and enemy.hostile + ): + print("\n--- Combat Round ---") + print( + f"Your HP: {game.player.health}/{game.player.total_max_hp(game.world.item_db)}" + ) + print( + f"{enemy.name} HP: {enemy.health}/{enemy.total_max_hp(game.world.item_db)}" + ) + print(f"Your gold: {game.player.gold} | Their gold: {enemy.gold}") + + action, payload = game.player.controller.choose_combat_action( + game, game.player, enemy + ) + if action == "none": + continue + + if action == "attack": + raw = game.player.roll_attack_damage(game.world.item_db) + if enemy_defending: + raw = max(1, raw // 2) + enemy_defending = False + actual = enemy.take_damage(raw, game.world.item_db) + print(f"You hit {enemy.name} for {actual} damage.") + + if enemy.maybe_surrender(game): + if enemy.name == "Bandit Nox": + game.flags.add("bandit_spared") + if game.resolve_enemy_surrender(enemy): + return + + if not enemy.is_alive(): + enemy.on_defeat(game) + if enemy.persistent: + game.enter_mode("npc", npc=enemy) + else: + game.enter_mode("exploration") + return + + elif action == "defend": + player_defending = True + print("You brace for the next attack.") + + elif action == "item": + ok, message, healed = game.player.use_consumable( + payload, game.world.item_db + ) + if ok: + print(f"{message} Restored {healed} HP.") + else: + print(message) + continue + + elif action == "run": + if random.random() < 0.5: + print("You escape successfully!") + enemy.hostile = False if enemy.surrendered else enemy.hostile + game.enter_mode("exploration") + return + print("You try to run, but the enemy cuts you off!") + + elif action == "surrender": + if enemy.handle_player_surrender(game): + return + continue + + if enemy.is_alive() and enemy.hostile: + enemy_action, enemy_payload = enemy.controller.choose_combat_action( + game, enemy, game.player + ) + if enemy_action == "item" and enemy_payload is not None: + ok, msg, healed = enemy.use_consumable( + enemy_payload, game.world.item_db + ) + if ok: + print( + f"{enemy.name} uses {game.item_name(enemy_payload)} and heals {healed} HP." + ) + elif enemy_action == "defend": + enemy_defending = True + print(f"{enemy.name} takes a guarded stance.") + else: + raw = enemy.roll_attack_damage(game.world.item_db) + if player_defending: + raw = max(1, raw // 2) + player_defending = False + actual = game.player.take_damage(raw, game.world.item_db) + print(f"{enemy.name} hits you for {actual} damage.") + if not game.player.is_alive(): + return + + if enemy.persistent: + game.enter_mode("npc", npc=enemy) + else: + game.enter_mode("exploration") + + +class MenuEngine(Engine): + def run(self, game: Game) -> None: + divider("MENU") + options = [ + {"kind": "status", "text": "View status", "value": None}, + {"kind": "loadout", "text": "Manage loadout", "value": None}, + {"kind": "journal", "text": "View world flags / journal", "value": None}, + {"kind": "back", "text": "Return to exploration", "value": None}, + {"kind": "quit", "text": "Quit game", "value": None}, + ] + choice = game.choose(options, prompt="Choose a menu option: ") + if choice["kind"] == "status": + game.player.show_status(game.world.item_db, bounty=game.bounty) + game.enter_mode("menu") + elif choice["kind"] == "loadout": + game.enter_mode("loadout") + elif choice["kind"] == "journal": + game.show_journal() + game.enter_mode("menu") + elif choice["kind"] == "back": + game.enter_mode("exploration") + elif choice["kind"] == "quit": + print("Thanks for playing!") + game.running = False + + +class LoadoutEngine(Engine): + def run(self, game: Game) -> None: + divider("LOADOUT") + player = game.player + item_db = game.world.item_db + + print("Current equipment:") + for slot in player.EQUIPMENT_SLOTS: + item_id = player.equipment[slot] + label = game.item_name(item_id) if item_id is not None else "empty" + print(f"- {slot}: {label}") + + options = [] + for item_id in player.equippable_items(item_db): + item = item_db[item_id] + options.append( + { + "kind": "equip", + "text": f"Equip {item.name} ({item.slot})", + "value": item_id, + } + ) + for slot in player.EQUIPMENT_SLOTS: + if player.equipment[slot] is not None: + options.append( + {"kind": "unequip", "text": f"Unequip {slot}", "value": slot} + ) + options.append({"kind": "back", "text": "Back", "value": None}) + + choice = game.choose(options, prompt="Choose a loadout action: ") + if choice["kind"] == "equip": + ok, message = player.equip(choice["value"], item_db) + print(message) + game.enter_mode("loadout") + elif choice["kind"] == "unequip": + ok, message = player.unequip(choice["value"], item_db) + print(message) + game.enter_mode("loadout") + elif choice["kind"] == "back": + game.enter_mode("menu") + + +# ============================================================ +# Game director / router +# ============================================================ + + +class Game: + def __init__(self, player_name: str = "Tav", world: World | None = None): + self.world = world or StarCrystalWorld() + self.player = Character( + name=player_name, + base_stats=Stats(max_hp=100, attack_min=6, attack_max=10, defense=1), + controller=PlayerController(), + location="village", + gold=5, + ) + self.flags: set[str] = {"game_started"} + self.bounty = 0 + self.running = True + self.mode = "exploration" + self.context: dict[str, object] = {} + self.engines: dict[str, Engine] = { + "exploration": ExplorationEngine(), + "npc": NPCInteractionEngine(), + "dialogue": DialogueEngine(), + "trade": TradeEngine(), + "riddle": RiddleEngine(), + "combat": CombatEngine(), + "menu": MenuEngine(), + "loadout": LoadoutEngine(), + } + self.world.on_location_enter(self, self.current_location()) + + def item_name(self, item_id: str | None) -> str: + if item_id is None: + return "empty" + return self.world.item(item_id).name + + def current_location(self) -> Location: + return self.world.get(self.player.location) + + def find_npc(self, npc_name: str) -> NPC | None: + for location in self.world.locations.values(): + for npc in location.npcs: + if npc.name == npc_name: + return npc + return None + + def enter_mode(self, mode: str, **context: object) -> None: + self.mode = mode + self.context = context + + def change_bounty(self, amount: int, reason: str | None = None) -> None: + self.bounty = max(0, self.bounty + amount) + if reason: + print(f"[Bounty] {reason}: {self.bounty}") + else: + print(f"[Bounty] {self.bounty}") + + def show_goal(self) -> None: + if "jailed" in self.flags: + print("\nGoal: Serve your time.") + elif "quest_started" not in self.flags: + print("\nGoal: Talk to Elder Mira.") + elif "has_star_crystal" in self.flags and "game_won" not in self.flags: + print("\nGoal: Return the Star Crystal to Elder Mira.") + elif "game_won" in self.flags: + print("\nGoal: The valley has been saved.") + else: + print("\nGoal: Explore the valley and recover the Star Crystal.") + + def show_journal(self) -> None: + divider("JOURNAL / WORLD FLAGS") + print(f"Gold: {self.player.gold}") + print(f"Bounty: {self.bounty}") + if not self.flags: + print("No flags set.") + return + for flag in sorted(self.flags): + print(f"- {flag}") + + def choose(self, options: list[dict], prompt: str = "Choose: ") -> dict: + for index, option in enumerate(options, start=1): + print(f"{index}. {option['text']}") + while True: + answer = input(prompt).strip() + try: + idx = int(answer) - 1 + selected = options[idx] + action_separator(selected.get("text")) + return selected + except (ValueError, IndexError): + print("Please enter a valid menu number.") + + def exit_is_blocked(self, exit_obj: Exit) -> bool: + if exit_obj.blocked: + return True + if exit_obj.requires_item and not self.player.has_item(exit_obj.requires_item): + return True + if exit_obj.requires_flag and exit_obj.requires_flag not in self.flags: + return True + return False + + def handle_move(self, exit_obj: Exit) -> None: + if exit_obj.on_attempt_effect is not None: + exit_obj.on_attempt_effect.apply(self) + + if self.exit_is_blocked(exit_obj): + print(exit_obj.blocked_text) + if exit_obj.on_blocked_effect is not None: + exit_obj.on_blocked_effect.apply(self) + return + + if exit_obj.warning_text: + print(exit_obj.warning_text) + confirm = input("Continue? [y/N] ").strip().lower() + if confirm not in {"y", "yes"}: + print("You decide not to risk it.") + return + + self.player.location = exit_obj.destination + self.flags.add(f"visited:{exit_obj.destination}") + if exit_obj.on_success_effect is not None: + exit_obj.on_success_effect.apply(self) + self.world.on_location_enter(self, self.current_location()) + + def give_player_item(self, item_id: str) -> None: + self.player.add_item(item_id) + item = self.world.item(item_id) + print(f"[Inventory] You got: {item.name}") + if item.description: + print(item.description) + for flag in item.set_flags_on_pickup: + self.flags.add(flag) + self.world.on_item_picked_up(self, item_id) + + def handle_take_item(self, item_id: str) -> None: + location = self.current_location() + if item_id not in location.items: + print("That item is no longer here.") + return + location.items.remove(item_id) + self.give_player_item(item_id) + + def loot_npc(self, npc: NPC) -> None: + if not npc.can_loot(): + print(f"There is nothing to loot from {npc.name}.") + return + divider(f"Looting {npc.name}") + if npc.gold > 0: + print(f"You take {npc.gold} gold.") + self.player.gold += npc.gold + npc.gold = 0 + while npc.inventory: + seen = set() + options = [] + for item_id in npc.inventory: + if item_id in seen: + continue + seen.add(item_id) + count = npc.inventory.count(item_id) + suffix = f" x{count}" if count > 1 else "" + options.append( + { + "kind": "take", + "text": f"Take {self.item_name(item_id)}{suffix}", + "value": item_id, + } + ) + options.append({"kind": "done", "text": "Done looting", "value": None}) + choice = self.choose(options, prompt="Choose loot: ") + if choice["kind"] == "done": + return + item_id = choice["value"] + npc.remove_item(item_id) + self.give_player_item(item_id) + print("Nothing remains.") + + def resolve_enemy_surrender(self, enemy: NPC) -> bool: + divider(f"{enemy.name} Surrenders") + options = [ + {"kind": "spare", "text": f"Spare {enemy.name}", "value": None}, + {"kind": "kill", "text": f"Kill {enemy.name}", "value": None}, + ] + choice = self.choose(options, prompt="Choose your response: ") + if choice["kind"] == "spare": + if enemy.name == "Bandit Nox": + self.flags.add("bandit_spared") + self.enter_mode("npc", npc=enemy) + return True + if choice["kind"] == "kill": + enemy.health = 0 + enemy.on_defeat(self) + if enemy.persistent: + self.enter_mode("npc", npc=enemy) + else: + self.enter_mode("exploration") + return True + return False + + def run(self) -> None: + divider(f"Welcome to {self.world.name}") + print( + "This prototype uses Character + Controller, effect-driven paths and features, gold-based trade, and encounter rules that can fire as you explore." + ) + while self.running: + if not self.player.is_alive(): + divider("Defeat") + print("You collapse from your injuries. The adventure ends here.") + break + engine = self.engines[self.mode] + engine.run(self) + if self.running and self.player.is_alive(): + prompt_continue() + + +# ============================================================ +# Textual front-end (v5 architecture, fused inline) +# ============================================================ +# Same structure as v6: QueueWriter + BackendBridge + RPGTextualApp. +# In this version the snapshots and log lines may also include ASCII +# art keys so the UI can render the matching illustration. + +"""Textual front end for rich_single_file_rpg.py. + +Development version that imports the core game from a sibling file. +A build script can merge both files into a single-file program by setting +EMBEDDED_CORE=True and prepending the core code. +""" + +from collections import Counter +from contextlib import redirect_stdout, redirect_stderr +import builtins +import importlib.util +from pathlib import Path +import queue +import sys +import threading +import traceback +from typing import Any + +from textual.app import App, ComposeResult +from textual.containers import Horizontal, Vertical +from textual.events import Key +from textual.widgets import Footer, Header, Input, OptionList, RichLog, Static + + +EMBEDDED_CORE = True +CORE_PATH = Path(__file__).with_name("rich_single_file_rpg.py") + + +def load_core_module() -> Any: + if EMBEDDED_CORE: + return sys.modules[__name__] + spec = importlib.util.spec_from_file_location("rpg_core", CORE_PATH) + if spec is None or spec.loader is None: + raise RuntimeError(f"Unable to load game core from {CORE_PATH}") + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +class QueueWriter: + """Capture stdout/stderr and emit complete lines to the UI queue.""" + + def __init__(self, bridge: "BackendBridge"): + self.bridge = bridge + self._buffer = "" + + def write(self, text: str) -> int: + self._buffer += text + while "\n" in self._buffer: + line, self._buffer = self._buffer.split("\n", 1) + self.bridge.emit("log", text=line) + return len(text) + + def flush(self) -> None: + if self._buffer: + self.bridge.emit("log", text=self._buffer) + self._buffer = "" + + +class BackendBridge: + def __init__(self, core: Any): + self.core = core + self.events: queue.Queue[dict[str, Any]] = queue.Queue() + self.responses: queue.Queue[Any] = queue.Queue() + self.thread: threading.Thread | None = None + self.game: Any | None = None + + def start(self) -> None: + self.thread = threading.Thread(target=self._run_backend, daemon=True) + self.thread.start() + + def emit(self, event_type: str, **payload: Any) -> None: + self.events.put({"type": event_type, **payload}) + + def snapshot(self) -> dict[str, Any]: + game = self.game + if game is None: + return { + "mode": "boot", + "goal": "", + "location": "Starting...", + "description": "", + "items": [], + "npcs": [], + "features": [], + "health": 0, + "max_health": 0, + "gold": 0, + "bounty": 0, + "attack": (0, 0), + "defense": 0, + "equipment": {}, + "inventory": [], + "flags": [], + } + location = game.current_location() + item_db = game.world.item_db + low, high = game.player.attack_range(item_db) + counts = Counter(game.player.inventory) + inventory = [] + seen = set() + for item_id in game.player.inventory: + if item_id in seen: + continue + seen.add(item_id) + item = item_db[item_id] + count = counts[item_id] + inventory.append((item.name, count)) + equipment = {} + for slot in game.player.EQUIPMENT_SLOTS: + item_id = game.player.equipment[slot] + equipment[slot] = item_db[item_id].name if item_id is not None else "empty" + focus_npc = game.context.get("npc") if isinstance(game.context, dict) else None + focus_npc_name = None + focus_npc_state = None + focus_npc_mood = None + if focus_npc is not None and hasattr(focus_npc, "name"): + focus_npc_name = focus_npc.name + if getattr(focus_npc, "defeated", False): + focus_npc_state = "dead" + elif getattr(focus_npc, "is_alive", lambda: False)(): + focus_npc_state = "alive" + else: + focus_npc_state = "unknown" + if hasattr(focus_npc, "mood_label"): + focus_npc_mood = focus_npc.mood_label() + return { + "mode": game.mode, + "goal": self.goal_text(game), + "location_key": location.key, + "location": location.name, + "description": location.description, + "items": [game.item_name(item_id) for item_id in location.items], + "npcs": [f"{npc.name} ({npc.mood_label()})" for npc in location.npcs], + "features": [feature.name for feature in location.features], + "health": game.player.health, + "max_health": game.player.total_max_hp(item_db), + "gold": game.player.gold, + "bounty": game.bounty, + "attack": (low, high), + "defense": game.player.total_defense(item_db), + "equipment": equipment, + "inventory": inventory, + "flags": sorted(game.flags), + "focus_npc_name": focus_npc_name, + "focus_npc_state": focus_npc_state, + "focus_npc_mood": focus_npc_mood, + } + + def goal_text(self, game: Any) -> str: + if "jailed" in game.flags: + return "Serve your time." + if "quest_started" not in game.flags: + return "Talk to Elder Mira." + if "has_star_crystal" in game.flags and "game_won" not in game.flags: + return "Return the Star Crystal to Elder Mira." + if "game_won" in game.flags: + return "The valley has been saved." + return "Explore the valley and recover the Star Crystal." + + def request_choice(self, options: list[dict[str, Any]], prompt: str) -> int: + self.emit("state", snapshot=self.snapshot()) + self.emit( + "choices", prompt=prompt, options=[option["text"] for option in options] + ) + return int(self.responses.get()) + + def request_text(self, prompt: str) -> str: + self.emit("state", snapshot=self.snapshot()) + self.emit("text_input", prompt=prompt) + response = self.responses.get() + return "" if response is None else str(response) + + def request_continue(self, prompt: str) -> None: + # In the TUI, the current outcome remains visible in its own pane, so + # an explicit continue step is unnecessary. Treat it as a no-op. + self.emit("state", snapshot=self.snapshot()) + return + + def _run_backend(self) -> None: + core = self.core + bridge = self + + class TuiExplorationEngine(core.Engine): + def run(self, game: Any) -> None: + if game.world.try_start_of_exploration_encounter(game): + return + + location = game.current_location() + choices = [] + jailed = "jailed" in game.flags and location.key == "jail" + if not jailed: + for exit_obj in location.exits: + destination = game.world.get(exit_obj.destination) + text = f"Go {exit_obj.direction} to {destination.name}" + if game.exit_is_blocked(exit_obj): + text += " [blocked]" + elif exit_obj.warning_text: + text += " [risky]" + choices.append( + {"kind": "move", "text": text, "value": exit_obj} + ) + for item_id in location.items: + choices.append( + { + "kind": "take_item", + "text": f"Take {game.item_name(item_id)}", + "value": item_id, + } + ) + for npc in location.npcs: + choices.append( + {"kind": "npc", "text": npc.menu_text(), "value": npc} + ) + for feature in location.features: + choices.append( + { + "kind": "feature", + "text": feature.menu_text(), + "value": feature, + } + ) + if not jailed: + choices.append({"kind": "menu", "text": "Open menu", "value": None}) + + choice = game.choose(choices, prompt="What do you want to do?") + if choice["kind"] == "move": + game.handle_move(choice["value"]) + elif choice["kind"] == "take_item": + game.handle_take_item(choice["value"]) + elif choice["kind"] == "npc": + game.enter_mode("npc", npc=choice["value"]) + elif choice["kind"] == "feature": + choice["value"].interact(game) + elif choice["kind"] == "menu": + game.enter_mode("menu") + + class TuiNPCInteractionEngine(core.Engine): + def run(self, game: Any) -> None: + npc = game.context["npc"] + if ( + npc.is_guard() + and game.bounty > 0 + and npc.is_alive() + and not npc.defeated + and not npc.hostile + ): + core.say( + npc.name, f"You have a bounty of {game.bounty}. Stand down." + ) + npc.hostile = True + game.enter_mode("combat", npc=npc) + return + + options = [] + if npc.can_talk(game): + options.append( + {"kind": "talk", "text": f"Talk to {npc.name}", "value": npc} + ) + if npc.can_trade(game): + options.append( + { + "kind": "trade", + "text": f"Trade with {npc.name}", + "value": npc, + } + ) + if npc.can_riddle(): + options.append( + { + "kind": "riddle", + "text": f"Challenge {npc.name}", + "value": npc, + } + ) + if npc.can_attack(): + options.append( + {"kind": "attack", "text": f"Attack {npc.name}", "value": npc} + ) + if npc.can_loot(): + options.append( + {"kind": "loot", "text": f"Loot {npc.name}", "value": npc} + ) + options.append({"kind": "back", "text": "Step away", "value": None}) + + choice = game.choose( + options, prompt=f"How do you want to interact with {npc.name}?" + ) + kind = choice["kind"] + if kind == "talk": + game.enter_mode("dialogue", npc=npc) + elif kind == "trade": + game.enter_mode("trade", npc=npc) + elif kind == "riddle": + game.enter_mode("riddle", npc=npc) + elif kind == "attack": + npc.on_player_attack(game) + game.enter_mode("combat", npc=npc) + elif kind == "loot": + game.loot_npc(npc) + game.enter_mode("npc", npc=npc) + elif kind == "back": + game.enter_mode("exploration") + + class TuiDialogueEngine(core.Engine): + def run(self, game: Any) -> None: + npc = game.context["npc"] + topics = npc.available_topics(game) + if not topics: + core.say(npc.name, "I have nothing more to say right now.") + game.enter_mode("npc", npc=npc) + return + + options = [ + {"kind": "topic", "text": topic.title, "value": topic} + for topic in topics + ] + options.append({"kind": "back", "text": "Back", "value": None}) + choice = game.choose( + options, prompt=f"What do you want to ask {npc.name}?" + ) + if choice["kind"] == "back": + game.enter_mode("npc", npc=npc) + return + + topic = choice["value"] + for line in topic.lines: + core.say(npc.name, line) + if topic.outcome_effect is not None: + topic.outcome_effect.apply(game) + if topic.once: + npc.used_topics.add(topic.key) + if "game_won" in game.flags: + game.running = False + return + game.enter_mode("npc", npc=npc) + + class TuiTradeEngine(core.Engine): + def run(self, game: Any) -> None: + npc = game.context["npc"] + offers = npc.available_trade_offers(game) + if not offers: + core.say(npc.name, "I have nothing I can trade right now.") + game.enter_mode("npc", npc=npc) + return + + options = [] + for offer_index, offer in offers: + wants_parts = [] + if offer.wants_gold: + wants_parts.append(f"{offer.wants_gold} gold") + if offer.wants_items: + wants_parts.extend( + game.item_name(item_id) for item_id in offer.wants_items + ) + gives_parts = [] + if offer.gives_gold: + gives_parts.append(f"{offer.gives_gold} gold") + if offer.gives_items: + gives_parts.extend( + game.item_name(item_id) for item_id in offer.gives_items + ) + wants = ", ".join(wants_parts) or "nothing" + gives = ", ".join(gives_parts) or "nothing" + options.append( + { + "kind": "offer", + "text": f"{offer.title} [{wants} -> {gives}]", + "value": (offer_index, offer), + } + ) + options.append({"kind": "back", "text": "Back", "value": None}) + + choice = game.choose(options, prompt=f"Choose a trade with {npc.name}") + if choice["kind"] == "back": + game.enter_mode("npc", npc=npc) + return + + offer_index, offer = choice["value"] + if game.player.gold < offer.wants_gold: + core.say( + npc.name, f"Come back when you have {offer.wants_gold} gold." + ) + game.enter_mode("npc", npc=npc) + return + if not game.player.has_items(offer.wants_items): + core.say( + npc.name, + f"Come back when you have: {', '.join(game.item_name(i) for i in offer.wants_items)}.", + ) + game.enter_mode("npc", npc=npc) + return + if npc.gold < offer.gives_gold or not npc.has_items(offer.gives_items): + core.say(npc.name, "I cannot complete that trade right now.") + game.enter_mode("npc", npc=npc) + return + + game.player.gold -= offer.wants_gold + npc.gold += offer.wants_gold + npc.gold -= offer.gives_gold + game.player.gold += offer.gives_gold + game.player.remove_items(offer.wants_items) + npc.add_items(offer.wants_items) + npc.remove_items(offer.gives_items) + for item_id in offer.gives_items: + game.give_player_item(item_id) + if not offer.repeatable: + npc.completed_trades.add(offer_index) + core.say(npc.name, "A fair trade.") + game.enter_mode("npc", npc=npc) + + class TuiRiddleEngine(core.Engine): + def run(self, game: Any) -> None: + npc = game.context["npc"] + riddle = npc.riddle + if riddle is None: + game.enter_mode("npc", npc=npc) + return + if npc.riddle_solved: + for line in riddle.repeat_lines: + core.say(npc.name, line) + game.enter_mode("npc", npc=npc) + return + + for line in riddle.intro_lines: + core.say(npc.name, line) + core.say(npc.name, riddle.question) + answer = input("Your answer: ").strip().lower() + if answer in [a.lower() for a in riddle.answers]: + for line in riddle.success_lines: + core.say(npc.name, line) + npc.riddle_solved = True + for flag in riddle.set_flags_on_success: + game.flags.add(flag) + else: + for line in riddle.failure_lines: + core.say(npc.name, line) + if riddle.damage_on_failure > 0: + actual = game.player.take_damage( + riddle.damage_on_failure, game.world.item_db + ) + print(f"You take {actual} damage.") + game.enter_mode("npc", npc=npc) + + class TuiMenuEngine(core.Engine): + def run(self, game: Any) -> None: + options = [ + {"kind": "status", "text": "View status", "value": None}, + {"kind": "loadout", "text": "Manage loadout", "value": None}, + { + "kind": "journal", + "text": "View world flags / journal", + "value": None, + }, + {"kind": "back", "text": "Return to exploration", "value": None}, + {"kind": "quit", "text": "Quit game", "value": None}, + ] + choice = game.choose(options, prompt="Choose a menu option") + if choice["kind"] == "status": + game.player.show_status(game.world.item_db, bounty=game.bounty) + game.enter_mode("menu") + elif choice["kind"] == "loadout": + game.enter_mode("loadout") + elif choice["kind"] == "journal": + game.show_journal() + game.enter_mode("menu") + elif choice["kind"] == "back": + game.enter_mode("exploration") + elif choice["kind"] == "quit": + print("Thanks for playing!") + game.running = False + + class TuiLoadoutEngine(core.Engine): + def run(self, game: Any) -> None: + player = game.player + item_db = game.world.item_db + options = [] + for item_id in player.equippable_items(item_db): + item = item_db[item_id] + current = player.equipment.get(item.slot) + suffix = " [equipped]" if current == item_id else "" + options.append( + { + "kind": "equip", + "text": f"Equip {item.name} ({item.slot}){suffix}", + "value": item_id, + } + ) + for slot in player.EQUIPMENT_SLOTS: + if player.equipment[slot] is not None: + options.append( + { + "kind": "unequip", + "text": f"Unequip {slot}", + "value": slot, + } + ) + options.append({"kind": "back", "text": "Back", "value": None}) + choice = game.choose(options, prompt="Choose a loadout action") + if choice["kind"] == "equip": + ok, message = player.equip(choice["value"], item_db) + print(message) + game.enter_mode("loadout") + elif choice["kind"] == "unequip": + ok, message = player.unequip(choice["value"], item_db) + print(message) + game.enter_mode("loadout") + elif choice["kind"] == "back": + game.enter_mode("menu") + + class TuiGame(core.Game): + def __init__(self, bridge: BackendBridge): + self._bridge = bridge + super().__init__(player_name="Tav", world=core.StarCrystalWorld()) + self.engines.update( + { + "exploration": TuiExplorationEngine(), + "npc": TuiNPCInteractionEngine(), + "dialogue": TuiDialogueEngine(), + "trade": TuiTradeEngine(), + "riddle": TuiRiddleEngine(), + "menu": TuiMenuEngine(), + "loadout": TuiLoadoutEngine(), + } + ) + + def choose(self, options: list[dict], prompt: str = "Choose: ") -> dict: + index = self._bridge.request_choice(options, prompt) + selected = options[index] + core.action_separator(selected.get("text")) + return selected + + old_input = builtins.input + old_typewriter = core.typewriter_print + old_continue = core.prompt_continue + + def queued_typewriter(prefix: str, text: str, word_delay: float = 0.02) -> None: + bridge.emit("stream_start", prefix=prefix) + words = text.split() + for index, word in enumerate(words): + suffix = " " if index < len(words) - 1 else "" + bridge.emit("stream_chunk", text=word + suffix) + if word_delay > 0: + time.sleep(word_delay) + bridge.emit("stream_end") + + def queued_input(prompt: str = "") -> str: + return bridge.request_text(prompt) + + def queued_continue(prompt: str = "Press Enter to continue...") -> None: + bridge.request_continue(prompt) + + writer = QueueWriter(bridge) + + try: + builtins.input = queued_input + core.typewriter_print = queued_typewriter + core.prompt_continue = queued_continue + with redirect_stdout(writer), redirect_stderr(writer): + bridge.game = TuiGame(bridge) + bridge.emit("state", snapshot=bridge.snapshot()) + bridge.game.run() + writer.flush() + bridge.emit("state", snapshot=bridge.snapshot()) + bridge.emit("game_over", message="Game over.") + except Exception: + writer.flush() + bridge.emit("log", text=traceback.format_exc()) + bridge.emit("game_over", message="The backend crashed.") + finally: + builtins.input = old_input + core.typewriter_print = old_typewriter + core.prompt_continue = old_continue + + +class RPGTextualApp(App): + CSS = """ + Screen { + layout: vertical; + } + + #body { + height: 1fr; + layout: horizontal; + } + + #sidebar { + width: 34; + min-width: 28; + border: round $accent; + padding: 1; + } + + #main { + width: 1fr; + border: round $accent; + padding: 1; + } + + #actions { + width: 40; + min-width: 34; + border: round $accent; + padding: 1; + } + + #goal, #location, #status { + margin-bottom: 1; + border: round $surface; + padding: 1; + } + + #ascii_art { + height: 14; + min-height: 10; + border: round $surface; + padding: 1; + margin-bottom: 1; + content-align: center middle; + } + + #current_event { + height: 1fr; + min-height: 8; + border: round $surface; + padding: 1; + margin-bottom: 1; + overflow-y: auto; + } + + #log_status { + height: auto; + margin-bottom: 1; + border: round $surface; + padding: 0 1; + } + + #history { + height: 6; + min-height: 4; + max-height: 12; + border: round $surface; + padding: 1; + } + + #prompt { + margin-bottom: 1; + min-height: 3; + border: round $surface; + padding: 1; + } + + #options { + height: 1fr; + margin-bottom: 1; + } + + #options:focus { + border: round $accent; + } + + #text_input { + margin-top: 1; + } + + + .hidden { + display: none; + } + """ + + BINDINGS = [ + ("q", "quit", "Quit"), + ("m", "menu", "Menu"), + ("c", "clear_history", "Clear history"), + ("l", "toggle_log", "Toggle log"), + ("L", "toggle_log", "Toggle log"), + ("escape", "focus_choices", "Focus choices"), + ] + + def __init__(self) -> None: + super().__init__() + self.core = load_core_module() + self.bridge = BackendBridge(self.core) + self.interaction_mode = "boot" + self.pending_options: list[str] = [] + self.current_event_text = "" + self.game_over = False + self.log_collapsed = True + self.history_line_count = 0 + + def compose(self) -> ComposeResult: + yield Header(show_clock=False) + with Horizontal(id="body"): + with Vertical(id="sidebar"): + yield Static("Goal", id="goal") + yield Static("Location", id="location") + yield Static("Status", id="status") + with Vertical(id="main"): + yield Static(self.make_ascii_art_placeholder({}), id="ascii_art") + yield Static("Latest outcome will appear here.", id="current_event") + yield Static("System log hidden. Press L to show.", id="log_status") + yield RichLog( + id="history", + wrap=True, + markup=False, + auto_scroll=True, + max_lines=600, + classes="hidden", + ) + with Vertical(id="actions"): + yield Static("Starting up...", id="prompt") + yield OptionList(id="options") + yield Input( + placeholder="Type response and press Enter", + id="text_input", + classes="hidden", + ) + yield Footer() + + def on_mount(self) -> None: + self.bridge.start() + self.set_interval(0.05, self.poll_backend) + + def action_menu(self) -> None: + if self.interaction_mode == "choice": + for i, label in enumerate(self.pending_options): + if label.lower().startswith("open menu"): + self.archive_current_event() + self.bridge.responses.put(i) + return + + def action_focus_choices(self) -> None: + if self.interaction_mode == "choice": + self.query_one("#options", OptionList).focus() + elif self.interaction_mode == "text": + self.query_one("#text_input", Input).focus() + + def action_clear_history(self) -> None: + self.query_one("#history", RichLog).clear() + self.history_line_count = 0 + self.update_history_size() + + def action_toggle_log(self) -> None: + self.log_collapsed = not self.log_collapsed + log = self.query_one("#history", RichLog) + status = self.query_one("#log_status", Static) + if self.log_collapsed: + log.add_class("hidden") + status.update("System log hidden. Press L to show.") + else: + log.remove_class("hidden") + status.update("System log visible. Press L to hide.") + self.update_history_size() + + def poll_backend(self) -> None: + while True: + try: + event = self.bridge.events.get_nowait() + except queue.Empty: + break + self.handle_backend_event(event) + + def handle_backend_event(self, event: dict[str, Any]) -> None: + event_type = event["type"] + if event_type == "log": + self.append_current_event_line(event.get("text", "")) + elif event_type == "stream_start": + self.prepare_for_stream_start() + self.append_current_event_chunk(event.get("prefix", "")) + elif event_type == "stream_chunk": + self.append_current_event_chunk(event.get("text", "")) + elif event_type == "stream_end": + self.append_current_event_chunk("\n") + elif event_type == "state": + self.update_state(event["snapshot"]) + elif event_type == "choices": + self.show_choices(event["prompt"], event["options"]) + elif event_type == "text_input": + self.show_text_input(event["prompt"]) + elif event_type == "game_over": + self.game_over = True + self.query_one("#prompt", Static).update(event.get("message", "Game over.")) + self.clear_options() + self.hide_text_input() + self.archive_current_event() + + def append_current_event_line(self, text: str) -> None: + if self.current_event_text and not self.current_event_text.endswith("\n"): + self.current_event_text += "\n" + self.current_event_text += text + self.render_current_event() + + def prepare_for_stream_start(self) -> None: + if not self.current_event_text: + return + if not self.current_event_text.endswith("\n"): + self.current_event_text += "\n" + if not self.current_event_text.endswith("\n\n"): + self.current_event_text += "\n" + self.render_current_event() + + def append_current_event_chunk(self, text: str) -> None: + self.current_event_text += text + self.render_current_event() + + def normalize_event_text(self) -> str: + if not self.current_event_text: + return "Latest outcome will appear here." + lines = self.current_event_text.split("\n") + while lines and lines[0] == "": + lines.pop(0) + while lines and lines[-1] == "": + lines.pop() + if not lines: + return "Latest outcome will appear here." + normalized = [] + blank_run = 0 + for line in lines: + if line == "": + blank_run += 1 + if blank_run <= 1: + normalized.append(line) + else: + blank_run = 0 + normalized.append(line) + return "\n".join(normalized) + + def render_current_event(self) -> None: + self.query_one("#current_event", Static).update(self.normalize_event_text()) + + def archive_current_event(self) -> None: + text = self.normalize_event_text() + if text and text != "Latest outcome will appear here.": + history = self.query_one("#history", RichLog) + history.write("─" * 56) + self.history_line_count += 1 + for line in text.splitlines(): + history.write(line) + self.history_line_count += 1 + self.update_history_size() + self.current_event_text = "" + self.render_current_event() + + def update_history_size(self) -> None: + history = self.query_one("#history", RichLog) + if self.log_collapsed: + return + desired = max(4, min(12, self.history_line_count + 2)) + history.styles.height = desired + + def choose_ascii_art_key(self, snapshot: dict[str, Any]) -> str: + flags = set(snapshot.get("flags", [])) + location = snapshot.get("location", "") + location_key = snapshot.get("location_key", "") + latest = self.normalize_event_text().lower() + focus_npc_name = snapshot.get("focus_npc_name") + focus_npc_state = snapshot.get("focus_npc_state") + + if focus_npc_name: + slug = focus_npc_name.lower().replace(" ", "_") + return f"npc::{slug}::{focus_npc_state or 'alive'}" + + if "game_won" in flags: + return "fountain_restored" + if ( + "return the star crystal" in latest + or "you found it... the star crystal" in latest + ): + return "return_crystal" + if "surrenders" in latest or "spare" in latest or "make a deal" in latest: + return "bandit_surrenders" + if "shatters" in latest or "shatters on the rocks" in latest: + return "item_shatter" + if ( + "marches you to the village jail" in latest + or "bounty" in latest + and "stand down" in latest + ): + return "guard_arrest" + if "the trial is complete" in latest or "wisdom and patience" in latest: + return "guardian_trial_passed" + if "not quite. think more carefully." in latest: + return "guardian_trial_failed" + if ( + "first_tower_visit" in flags + or location_key == "tower_top" + or location == "Tower Summit" + ): + return "first_tower_visit" + + location_icons = { + "village": "loc::village", + "crossroads": "loc::crossroads", + "forest": "loc::forest", + "garden": "loc::garden", + "lake": "loc::lake", + "cave_entrance": "loc::cave_entrance", + "cave_depths": "loc::cave_depths", + "ruins": "loc::ruins", + "tower_gate": "loc::tower_gate", + "tower_top": "loc::tower_top", + "jail": "loc::jail", + } + if location_key in location_icons: + return location_icons[location_key] + + # Backward-compatible fallback if the snapshot was created before location_key existed. + legacy_location_icons = { + "Sunmeadow Village": "loc::village", + "Old Crossroads": "loc::crossroads", + "Whispering Forest": "loc::forest", + "Hidden Garden": "loc::garden", + "Mirror Lake": "loc::lake", + "Cave Entrance": "loc::cave_entrance", + "Crystal Cave": "loc::cave_depths", + "Moonstone Ruins": "loc::ruins", + "Tower Gate": "loc::tower_gate", + "Tower Summit": "loc::tower_top", + "Village Jail": "loc::jail", + } + return legacy_location_icons.get(location, "loc::village") + + def make_npc_ascii_placeholder( + self, name: str, state: str, mood: str | None = None + ) -> str: + mood_text = mood or "unknown" + if state == "dead": + art = "\n".join( + [ + " .-------.", + " / x x \\", + " | - |", + r" | \\___// |", + r" \\_________/", + " / \\", + r" /_____\\", + ] + ) + title = f"{name} [defeated]" + else: + art = "\n".join( + [ + " .-------.", + " / o o \\", + " | ^ |", + r" | \\___// |", + r" \\_________/", + r" /| |\\", + r" /_| |_\\", + ] + ) + title = f"{name} [{mood_text}]" + return f"[b]{title}[/b]\n\n{art}" + + def make_ascii_art_placeholder(self, snapshot: dict[str, Any]) -> str: + key = self.choose_ascii_art_key(snapshot) + if key.startswith("npc::"): + art = ASCII_ART_POLISHED.get(key) + if art: + name = snapshot.get("focus_npc_name") or "Unknown NPC" + state = snapshot.get("focus_npc_state") or "alive" + mood = snapshot.get("focus_npc_mood") or "unknown" + title = f"{name} [{'defeated' if state == 'dead' else mood}]" + return f"[b]{title}[/b]\n\n{art}" + return self.make_npc_ascii_placeholder( + snapshot.get("focus_npc_name") or "Unknown NPC", + snapshot.get("focus_npc_state") or "alive", + snapshot.get("focus_npc_mood"), + ) + art = ASCII_ART_POLISHED.get(key, "") + title = snapshot.get("location", "Scene") or "Scene" + return f"[b]{title}[/b]\n\n{art}" + + def update_state(self, snapshot: dict[str, Any]) -> None: + goal = self.query_one("#goal", Static) + location = self.query_one("#location", Static) + status = self.query_one("#status", Static) + ascii_art = self.query_one("#ascii_art", Static) + + goal.update(f"[b]Goal[/b]\n{snapshot['goal']}") + ascii_art.update(self.make_ascii_art_placeholder(snapshot)) + + parts = [f"[b]{snapshot['location']}[/b]", "", snapshot["description"]] + if snapshot["items"]: + parts.extend( + ["", "[b]Items here[/b]"] + [f"- {item}" for item in snapshot["items"]] + ) + if snapshot["npcs"]: + parts.extend( + ["", "[b]People / creatures[/b]"] + + [f"- {npc}" for npc in snapshot["npcs"]] + ) + if snapshot["features"]: + parts.extend( + ["", "[b]Features[/b]"] + + [f"- {feature}" for feature in snapshot["features"]] + ) + location.update("\n".join(parts)) + + attack_low, attack_high = snapshot["attack"] + status_lines = [ + "[b]Player[/b]", + f"HP: {snapshot['health']}/{snapshot['max_health']}", + f"Attack: {attack_low}-{attack_high}", + f"Defense: {snapshot['defense']}", + f"Gold: {snapshot['gold']}", + f"Bounty: {snapshot['bounty']}", + "", + "[b]Equipment[/b]", + ] + for slot, item_name in snapshot["equipment"].items(): + status_lines.append(f"- {slot}: {item_name}") + status_lines.extend(["", "[b]Inventory[/b]"]) + if snapshot["inventory"]: + for item_name, count in snapshot["inventory"]: + suffix = f" x{count}" if count > 1 else "" + status_lines.append(f"- {item_name}{suffix}") + else: + status_lines.append("- empty") + status.update("\n".join(status_lines)) + + def clear_options(self) -> None: + option_list = self.query_one("#options", OptionList) + option_list.clear_options() + self.pending_options = [] + + def show_choices(self, prompt: str, options: list[str]) -> None: + self.interaction_mode = "choice" + self.pending_options = list(options) + self.query_one("#prompt", Static).update( + f"{prompt}\n\nUse ↑/↓ and Enter, number keys 1-9, or click." + ) + self.hide_text_input() + option_list = self.query_one("#options", OptionList) + option_list.clear_options() + display_options = [f"{i}. {label}" for i, label in enumerate(options, start=1)] + option_list.add_options(display_options) + option_list.highlighted = 0 if options else None + option_list.focus() + + def show_text_input(self, prompt: str) -> None: + self.interaction_mode = "text" + self.clear_options() + self.query_one("#prompt", Static).update(prompt) + input_widget = self.query_one("#text_input", Input) + input_widget.remove_class("hidden") + input_widget.value = "" + input_widget.placeholder = prompt or "Type response and press Enter" + input_widget.focus() + + def hide_text_input(self) -> None: + input_widget = self.query_one("#text_input", Input) + input_widget.add_class("hidden") + input_widget.value = "" + + def on_option_list_option_selected(self, event: OptionList.OptionSelected) -> None: + if self.interaction_mode != "choice": + return + self.archive_current_event() + self.bridge.responses.put(event.option_index) + self.clear_options() + self.query_one("#prompt", Static).update("Resolving action...") + + def on_key(self, event: Key) -> None: + if self.interaction_mode != "choice": + return + + option_list = self.query_one("#options", OptionList) + count = len(self.pending_options) + if count == 0: + return + + key = event.key + option_list_has_focus = self.focused is option_list + + # When the OptionList itself has focus, let it handle arrow keys and + # Enter so we do not double-advance the selection. + if option_list_has_focus and key in {"up", "down", "enter"}: + return + + if key == "up": + current = option_list.highlighted + if current is None: + option_list.highlighted = 0 + else: + option_list.highlighted = max(0, current - 1) + event.stop() + return + + if key == "down": + current = option_list.highlighted + if current is None: + option_list.highlighted = 0 + else: + option_list.highlighted = min(count - 1, current + 1) + event.stop() + return + + if key == "enter": + current = option_list.highlighted + if current is None: + current = 0 + self.archive_current_event() + self.bridge.responses.put(current) + self.clear_options() + self.query_one("#prompt", Static).update("Resolving action...") + event.stop() + return + + if len(key) == 1 and key.isdigit(): + index = int(key) - 1 + if 0 <= index < count: + option_list.highlighted = index + self.archive_current_event() + self.bridge.responses.put(index) + self.clear_options() + self.query_one("#prompt", Static).update("Resolving action...") + event.stop() + return + + def on_input_submitted(self, event: Input.Submitted) -> None: + if event.input.id != "text_input" or self.interaction_mode != "text": + return + self.archive_current_event() + self.bridge.responses.put(event.value) + self.hide_text_input() + self.query_one("#prompt", Static).update("Resolving input...") + + +if __name__ == "__main__": + RPGTextualApp().run() diff --git a/dev/e2e_tests/.gitignore b/dev/e2e_tests/.gitignore new file mode 100644 index 00000000..fbca2253 --- /dev/null +++ b/dev/e2e_tests/.gitignore @@ -0,0 +1 @@ +results/ diff --git a/dev/e2e_tests/README.md b/dev/e2e_tests/README.md new file mode 100644 index 00000000..92706403 --- /dev/null +++ b/dev/e2e_tests/README.md @@ -0,0 +1,110 @@ +# Leasing e2e harness (developer tests) + +**This is a developer harness, not a unit suite.** It lives in `dev/` (not +`tests/`) on purpose: it stands up real docker/GPU/LiteLLM and is meant to be run +by hand on a serving host (yardrat), produce a report, and let us see what +breaks — efficiently and ergonomically. The polished, always-run, no-side-effect +checks stay in `tests/`. + +It is the executable companion to `dev/leasing-test-plan.md` (the copy-paste +runbook). Same scenarios, but here the harness runs them, asserts the wiring, and +writes one self-contained report you can `rsync` back. + +## Run it + +```bash +# on yardrat, with the infer-stack venv active: +cd submodules/infer-stack/dev/e2e_tests + +./run.sh # non-serving tiers (no GPU, no image pulls) +./run.sh --gpu # + real serving on GPU 0 (vLLM + Ollama) +``` + +Then rsync the printed results dir back; I review `report.md`. + +### Before `--gpu` (one-time, so timeouts reflect logic not 10GB pulls) + +```bash +docker pull vllm/vllm-openai:v0.19.1 +docker pull ghcr.io/berriai/litellm:v1.82.3-stable +docker pull ollama/ollama:latest +``` + +GPU tiers assume **GPU 0 free**. Placement now uses **every** GPU by default — +including yardrat's display-attached **GPU 1** — so the both-GPUs / GPU-1-pin +tiers need no special flag (set `config set skip_display_gpus true` to exclude +the display GPU; `60_dedicated_f5` probes placement on a busy box). + +## What runs + +| tier | `--gpu`? | what it proves | +| --- | --- | --- | +| `01_environment` | always | infer-stack imports, docker compose v2, (gpu) nvidia runtime | +| `10_dryrun` | no | acquire→leases→env-file→release on the null backend; bundles | +| `20_ergonomics` | no | `paths`, `config paths leasing`, `status`, `secrets`, day-2 fallback | +| `30_negative` | no | unknown endpoint / kubeai / missing catalog fail *friendly* (no traceback) | +| `40_single_vllm` | yes | acquire a vLLM model, real chat completion, day-2 `ps`, release | +| `45_both_gpus` | yes | `--include-display-gpus` spreads two models across GPU 0 + GPU 1 | +| `50_coalescing` | yes | two leases on one model → one group (demand 2), one container | +| `60_dedicated_f5` | yes | a dedicated group needs its own GPU; placement on a busy box | +| `70_ttl_reclaim` | yes | short TTL expires; reclaim:stop group reclaimed | +| `80_run_wrapper` | yes | `run -- ` injects env, releases on exit, propagates exit code | +| `85_ollama` | yes | ollama daemon lazy pull/warmup + chat through the gateway; Open WebUI gets both the gateway (OpenAI) and the daemon (native Ollama) connections | +| `86_ollama_lean` | yes | `--no-litellm`: no gateway, Open WebUI wired straight at the daemon's native API, tag still pulled, chat hits the daemon directly, `status` shows the UI URL | +| `88_gpu_pinning` | yes | pin an ollama daemon to **GPU 1** (`--include-display-gpus`): device-reservation `device_ids=[1]`, no host-index `CUDA_VISIBLE_DEVICES`, and it runs ON the GPU (not a CPU fallback) | +| `90_concurrency` | yes | two racing acquires; file lock keeps the compose file valid | +| `99_cleanup` | always | release stragglers + `down` the project (never leak containers) | + +Run a subset: `./run.sh --gpu --only '40 50'`. `99_cleanup` always runs. + +GPU tiers are isolated: before each serving tier the runner tears down the +`infer-stack` compose project **and wipes the ledger**, so a leftover group from +one tier can't hold the only usable GPU and starve the next. Long-running steps +(model cold-starts) print a `… still running (Ns)` heartbeat every 20s so they +don't look hung. + +## Stuck run? `cleanup.sh` + +If you Ctrl-C mid-run (or a box is left with leftover containers), reclaim it: + +```bash +./cleanup.sh # down the 'infer-stack' compose project + stragglers +./cleanup.sh --wipe-ledger DIR # + wipe the ledger under a run's results//infer-stack-data +``` + +The compose project is always named `infer-stack`, so the teardown works without +needing the original run's data dir. + +## The report + +`run.sh` writes a timestamped dir under `results/` (gitignored): + +``` +results// + report.md # READ THIS — summary, wiring axes, failures w/ log tails + results.jsonl # one machine-readable record per step + environment.txt # host / docker / nvidia-smi capture + logs/

..log # full combined output of every step + infer-stack-data/ # the run's ledger + rendered compose/litellm/.env (gpu) +``` + +`report.md` groups results into the three axes the harness is checking: +**Correctness** (does it serve / coalesce / release), **Efficiency** (one +container per shared model, reclaim, concurrency), **Ergonomics** (`paths` / +`secrets` / `status` / env-file / day-2 wrappers). Failures are surfaced first +with their log tails so a single file tells you what broke and where. + +## Knobs + +| flag | effect | +| --- | --- | +| `--gpu` | enable the serving tiers | +| `--only '40 50'` | run only those numeric prefixes (+ cleanup) | +| `--keep-running` | don't `down` the compose stack at the end | +| `--keep-data` | (no-op placeholder; data dir is always kept inside results) | +| `--data-dir DIR` | use an explicit `INFER_STACK_DATA_DIR` | +| `--results DIR` | write the report somewhere other than `results/` | +| `--catalog PATH` | use a different catalog than `catalog.yaml` | + +Each run uses a fresh `INFER_STACK_DATA_DIR` inside its results dir, so runs are +independent and the ledger/compose artifacts travel with the report. diff --git a/dev/e2e_tests/catalog.yaml b/dev/e2e_tests/catalog.yaml new file mode 100644 index 00000000..122e9f75 --- /dev/null +++ b/dev/e2e_tests/catalog.yaml @@ -0,0 +1,51 @@ +# Catalog for the leasing e2e harness, tailored to yardrat: +# GPU 0 Quadro RTX 8000 (48 GiB, free), GPU 1 Quadro RTX 5000 (16 GiB, +# display-attached -> skipped unless --include-display-gpus). Both Turing +# (sm_75): fp16 only, no bf16/FP8, so every vLLM endpoint pins `--dtype=half`. +# +# SPEED: these are the smallest sane ungated chat models (SmolLM2 135M/360M) and +# every vLLM endpoint runs with `--enforce-eager` (skip torch.compile + CUDA +# graph capture, which dominate vLLM startup and are model-size-independent) and +# a small max_model_len (less KV alloc / smaller capture). We're validating the +# leasing plumbing, not model perf, so eager + tiny is the right trade. +# +# Endpoints are named for the model they serve (smol-135 / smol-360 = SmolLM2 +# 135M / 360M). The alias is what clients/tests request through the gateway. +models: + smol135: {source: hf://HuggingFaceTB/SmolLM2-135M-Instruct} + smol360: {source: hf://HuggingFaceTB/SmolLM2-360M-Instruct} +endpoints: + smol-135: + engine: vllm + model: smol135 + runtime: {max_model_len: 2048, gpu_memory_utilization: 0.2, extra_args: ['--dtype=half', '--enforce-eager']} + reclaim: {policy: stop} + smol-135-dup: # same model+runtime, aliased to smol-135 -> must coalesce + engine: vllm + model: smol135 + public_name: smol-135 + runtime: {max_model_len: 2048, gpu_memory_utilization: 0.2, extra_args: ['--dtype=half', '--enforce-eager']} + smol-360: # 2nd distinct model -> for the both-GPU / display-GPU tiers + engine: vllm + model: smol360 + runtime: {max_model_len: 2048, gpu_memory_utilization: 0.2, extra_args: ['--dtype=half', '--enforce-eager']} + reclaim: {policy: keep-warm} + smol-ollama: + engine: ollama + host: local-ollama + model: smollm2:135m + smol-ollama-gpu1: # same tag, daemon pinned to GPU 1 (display) + engine: ollama # -> exercises explicit GPU pinning on the + host: ollama-gpu1 # 2nd GPU (needs --include-display-gpus) + model: smollm2:135m +runtime_hosts: + local-ollama: + engine: ollama + placement: {gpu_indices: [0]} + settings: {keep_alive: 5m} + ollama-gpu1: # pins the daemon to GPU 1 — the device + engine: ollama # reservation must put it there (and NOT a + placement: {gpu_indices: [1]} # host-index CUDA_VISIBLE_DEVICES, which + settings: {keep_alive: 5m} # would CPU-fallback — see 88_gpu_pinning) +bundles: + pair: [smol-135, smol-360] diff --git a/dev/e2e_tests/cleanup.sh b/dev/e2e_tests/cleanup.sh new file mode 100755 index 00000000..f112a75a --- /dev/null +++ b/dev/e2e_tests/cleanup.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# Manually tear down the leasing stack and (optionally) wipe the ledger. +# +# Use after Ctrl-C'ing a run, or any time a dev box is left with leftover +# infer-stack containers / a wedged ledger. The leasing compose project is +# always named "infer-stack", so the container teardown works regardless of +# which run's data dir was in play. +# +# ./cleanup.sh # down the 'infer-stack' compose project +# ./cleanup.sh --wipe-ledger # + wipe the ledger at $INFER_STACK_DATA_DIR +# ./cleanup.sh --wipe-ledger DIR # + wipe the ledger under DIR (a run's +# # results//infer-stack-data) +set -uo pipefail + +WIPE=0 +DATA="${INFER_STACK_DATA_DIR:-}" +for arg in "$@"; do + case "$arg" in + --wipe-ledger) WIPE=1 ;; + *) DATA="$arg" ;; + esac +done + +echo "== tearing down the 'infer-stack' compose project ==" +# By project name first (works even if no compose file is around)... +docker compose -p infer-stack down --remove-orphans 2>&1 | sed 's/^/ /' || true +# ...and via the rendered file if we can find one (belt and suspenders). +if [ -n "$DATA" ]; then + cf="$DATA/leasing/compose/docker-compose.yml" + [ -f "$cf" ] && docker compose -p infer-stack -f "$cf" down \ + --remove-orphans 2>&1 | sed 's/^/ /' || true +fi + +# Any stragglers labeled as ours (e.g. left by a hard kill mid-converge). +stragglers="$(docker ps -aq --filter 'label=com.docker.compose.project=infer-stack' 2>/dev/null)" +if [ -n "$stragglers" ]; then + echo "== removing straggler containers ==" + echo "$stragglers" | xargs -r docker rm -f 2>&1 | sed 's/^/ /' || true +fi + +if [ "$WIPE" = 1 ]; then + if [ -z "$DATA" ]; then + echo "!! --wipe-ledger needs INFER_STACK_DATA_DIR set or a DIR argument" + exit 2 + fi + echo "== wiping ledger under $DATA/leasing ==" + rm -f "$DATA"/leasing/ledger.db \ + "$DATA"/leasing/ledger.db-wal \ + "$DATA"/leasing/ledger.db-shm 2>/dev/null || true +fi + +echo "done." diff --git a/dev/e2e_tests/inspect.py b/dev/e2e_tests/inspect.py new file mode 100644 index 00000000..810e1bf2 --- /dev/null +++ b/dev/e2e_tests/inspect.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +"""Tiny rendered-compose inspector for the e2e harness assertions. + +Usage: ``python3 inspect.py `` + +Prints flat ``key: value`` lines the shell tiers assert on with ``expect_out`` / +``expect_re`` — keeping the YAML parsing out of brittle bash. Covers the front +door wiring (which services exist, how Open WebUI is connected) and each Ollama +daemon's host port + GPU pinning, which is what the new leasing features and the +GPU-pin bug fix need to be checked on real hardware. +""" + +from __future__ import annotations + +import sys + +import yaml + + +def main() -> None: + d = yaml.safe_load(open(sys.argv[1])) or {} + svcs = d.get('services', {}) or {} + print('services:', ' '.join(sorted(svcs))) + print('has litellm:', 'litellm' in svcs) + print('has open-webui:', 'open-webui' in svcs) + + ow_svc = svcs.get('open-webui') or {} + ow_ports = ow_svc.get('ports') or [] + if ow_ports: + print('open-webui host_port:', str(ow_ports[0]).split(':')[0]) + ow = ow_svc.get('environment', {}) or {} + for k in ( + 'ENABLE_OPENAI_API', + 'OPENAI_API_BASE_URL', + 'ENABLE_OLLAMA_API', + 'OLLAMA_BASE_URL', + ): + print(f'open-webui {k}: {ow.get(k)}') + + for name, svc in sorted(svcs.items()): + if not name.startswith('ollama-'): + continue + ports = svc.get('ports') or ['?:?'] + host_port = str(ports[0]).split(':')[0] + env = svc.get('environment', {}) or {} + devices = ( + (((svc.get('deploy') or {}).get('resources') or {}).get( + 'reservations', {} + ) or {}).get('devices') + or [{}] + ) + device_ids = devices[0].get('device_ids') + print(f'{name} host_port: {host_port}') + print(f'{name} device_ids: {device_ids}') + print(f'{name} CUDA_VISIBLE_DEVICES: {env.get("CUDA_VISIBLE_DEVICES")}') + + +if __name__ == '__main__': + main() diff --git a/dev/e2e_tests/lib.sh b/dev/e2e_tests/lib.sh new file mode 100644 index 00000000..47eba282 --- /dev/null +++ b/dev/e2e_tests/lib.sh @@ -0,0 +1,165 @@ +# shellcheck shell=bash +# Helper library for the infer-stack leasing e2e harness. +# +# A test script sources this file and then describes its work as a sequence of +# steps: +# +# step single-vllm 'acquire a vllm model and serve a chat completion' +# run 'infer-stack acquire smol-135 --backend compose --catalog "$E2E_CAT"' +# expect_rc 0 +# expect_out smol-135 +# end_step +# +# Every step records a structured record (id, title, verdict, duration, the +# individual assertions, and a path to the captured combined-output log) as one +# JSON line in "$E2E_RESULTS/results.jsonl". render_report.py turns those lines +# into report.md. The harness never uses `set -e`: a failing assertion marks the +# step failed and the run keeps going so one report covers everything. +# +# Required exported env (set by run.sh): E2E_RESULTS, E2E_SECTION, E2E_CAT, +# INFER_STACK_DATA_DIR, E2E_ENABLE_GPU. + +: "${E2E_RESULTS:?lib.sh: E2E_RESULTS must be exported by run.sh}" +: "${E2E_SECTION:=misc}" + +E2E_LOGDIR="$E2E_RESULTS/logs" +# Transient per-step files (captured stdout, assertion/note tallies) live in a +# scratch dir, NOT in logs/, so the rsync'd report's logs/ holds only real +# .log files. rsync excludes .scratch/ (see run.sh). +E2E_SCRATCH="$E2E_RESULTS/.scratch" +mkdir -p "$E2E_LOGDIR" "$E2E_SCRATCH" + +# ANSI colors (only when stdout is a tty) +if [ -t 1 ]; then + C_PASS=$'\033[32m'; C_FAIL=$'\033[31m'; C_SKIP=$'\033[33m' + C_DIM=$'\033[2m'; C_RST=$'\033[0m' +else + C_PASS=''; C_FAIL=''; C_SKIP=''; C_DIM=''; C_RST='' +fi + +_now() { date +%s.%N; } + +# --- heartbeat lifecycle ---------------------------------------------------- +# The per-step progress heartbeat is a backgrounded subshell. In a +# non-interactive script bash makes async commands IGNORE SIGINT, so a Ctrl-C +# would otherwise leave the heartbeat looping and printing long after the run. +# We track its pid and kill it (and its in-flight `sleep` child) on every exit +# path of this test-script process — normal, error, INT, or TERM. +_HB_PID='' +_kill_hb() { + [ -n "$_HB_PID" ] || return 0 + kill "$_HB_PID" $(pgrep -P "$_HB_PID" 2>/dev/null) 2>/dev/null + wait "$_HB_PID" 2>/dev/null + _HB_PID='' +} +trap '_kill_hb' EXIT +trap '_kill_hb; exit 130' INT TERM + +# step +step() { + CUR_ID="$1"; shift + CUR_TITLE="$*" + CUR_VERDICT='pass' # flips to 'fail' on any failed assertion + CUR_SKIP='' + CUR_START="$(_now)" + CUR_LOG="$E2E_LOGDIR/${E2E_SECTION}.${CUR_ID}.log" + LAST_OUT_FILE="$E2E_SCRATCH/${E2E_SECTION}.${CUR_ID}.lastout" + CUR_ASSERT_FILE="$E2E_SCRATCH/${E2E_SECTION}.${CUR_ID}.asserts" + CUR_NOTE_FILE="$E2E_SCRATCH/${E2E_SECTION}.${CUR_ID}.notes" + : > "$CUR_LOG"; : > "$LAST_OUT_FILE" + : > "$CUR_ASSERT_FILE"; : > "$CUR_NOTE_FILE" + { + echo "### step: $CUR_ID — $CUR_TITLE" + echo "section: $E2E_SECTION started: $(date -Iseconds)" + echo "--------------------------------------------------------------" + } >> "$CUR_LOG" + printf '%s» %s%s %s\n' "$C_DIM" "$CUR_ID" "$C_RST" "$CUR_TITLE" +} + +# run '' — executes via `bash -c`, captures combined +# output to the step log and to LAST_OUT_FILE, records RC and the duration. +# A heartbeat prints elapsed seconds every 20s so a long acquire doesn't +# look hung (it writes to the console, not to the captured output). +run() { + local start end + printf '\n$ %s\n' "$1" >> "$CUR_LOG" + start="$(_now)" + # Heartbeat also self-terminates if this test-script process dies (kill -0 + # $$ — in a subshell $$ is the parent shell's pid), covering even SIGKILL or + # a missed trap. Belt-and-suspenders with the EXIT/INT/TERM trap above. + ( while true; do + sleep 20 + kill -0 "$$" 2>/dev/null || exit 0 + printf ' %s… still running (%ds)%s\n' "$C_DIM" \ + "$(awk -v a="$start" -v b="$(_now)" 'BEGIN{printf "%d", b-a}')" \ + "$C_RST" + done ) & + _HB_PID=$! + bash -c "$1" > "$LAST_OUT_FILE" 2>&1 + RC=$? + _kill_hb + end="$(_now)" + DUR="$(awk -v a="$start" -v b="$end" 'BEGIN{printf "%.3f", b-a}')" + cat "$LAST_OUT_FILE" >> "$CUR_LOG" + printf '[exit %s in %ss]\n' "$RC" "$DUR" >> "$CUR_LOG" + return "$RC" +} + +_assert() { # _assert + local ok="$1"; shift + if [ "$ok" -eq 0 ]; then + printf 'PASS|%s\n' "$*" >> "$CUR_ASSERT_FILE" + else + printf 'FAIL|%s\n' "$*" >> "$CUR_ASSERT_FILE" + CUR_VERDICT='fail' + fi +} + +expect_rc() { [ "${RC:-1}" -eq "$1" ]; _assert $? "exit code == $1 (was ${RC:-?})"; } +expect_rc_not() { [ "${RC:-1}" -ne "$1" ]; _assert $? "exit code != $1 (was ${RC:-?})"; } +expect_out() { grep -qF -- "$1" "$LAST_OUT_FILE"; _assert $? "output contains: $1"; } +expect_no_out() { ! grep -qF -- "$1" "$LAST_OUT_FILE"; _assert $? "output excludes: $1"; } +expect_re() { grep -qE -- "$1" "$LAST_OUT_FILE"; _assert $? "output matches /$1/"; } +expect_file() { [ -f "$1" ]; _assert $? "file exists: $1"; } +expect_file_has(){ [ -f "$1" ] && grep -qF -- "$2" "$1"; _assert $? "file $1 contains: $2"; } + +# count_out — exact occurrence count assertion +count_out() { + local n; n="$(grep -cF -- "$1" "$LAST_OUT_FILE")" + [ "$n" -eq "$2" ]; _assert $? "count('$1') == $2 (was $n)" +} + +note() { printf '%s\n' "$*" >> "$CUR_NOTE_FILE"; printf '%s note: %s%s\n' "$C_DIM" "$*" "$C_RST"; } + +# skip — emit a skipped record (e.g. GPU disabled) +skip() { + CUR_ID="$1"; shift + local reason="$*" + CUR_TITLE="$reason"; CUR_VERDICT='skip'; CUR_SKIP=1 + CUR_START="$(_now)" + CUR_LOG=""; CUR_ASSERT_FILE=""; CUR_NOTE_FILE="" + printf '%s» %s SKIP%s %s\n' "$C_SKIP" "$CUR_ID" "$C_RST" "$reason" + E2E_SKIP_REASON="$reason" python3 "$E2E_ROOT/render_report.py" --emit-skip \ + --section "$E2E_SECTION" --id "$CUR_ID" >> "$E2E_RESULTS/results.jsonl" +} + +end_step() { + local end dur logrel + end="$(_now)" + dur="$(awk -v a="$CUR_START" -v b="$end" 'BEGIN{printf "%.3f", b-a}')" + logrel="logs/$(basename "$CUR_LOG")" + case "$CUR_VERDICT" in + pass) printf ' %s✓ pass%s (%ss)\n' "$C_PASS" "$C_RST" "$dur" ;; + fail) printf ' %s✗ FAIL%s (%ss) see %s\n' "$C_FAIL" "$C_RST" "$dur" "$logrel" ;; + esac + E2E_DUR="$dur" E2E_LOGREL="$logrel" \ + python3 "$E2E_ROOT/render_report.py" --emit-step \ + --section "$E2E_SECTION" --id "$CUR_ID" --title "$CUR_TITLE" \ + --verdict "$CUR_VERDICT" \ + --asserts "$CUR_ASSERT_FILE" --notes "$CUR_NOTE_FILE" \ + >> "$E2E_RESULTS/results.jsonl" +} + +# require_gpu — call at the top of a GPU test body; returns 1 (so the caller +# `return`s) when GPU serving is disabled, after emitting one skip record. +gpu_enabled() { [ "${E2E_ENABLE_GPU:-0}" = "1" ]; } diff --git a/dev/e2e_tests/render_report.py b/dev/e2e_tests/render_report.py new file mode 100755 index 00000000..6c767a87 --- /dev/null +++ b/dev/e2e_tests/render_report.py @@ -0,0 +1,255 @@ +#!/usr/bin/env python3 +"""Emit/assemble the infer-stack leasing e2e report. + +Three modes (lib.sh uses the first two; run.sh the third): + + --emit-step read an assert-file + note-file and print ONE results.jsonl line + --emit-skip print one skipped results.jsonl line + --assemble read results.jsonl (+ run_meta.json, environment.txt) and write + report.md + +Kept dependency-free (stdlib only) so it runs under whatever python the host +has. This is a developer harness, not shipped code. +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +from pathlib import Path + + +def _read_asserts(path: str) -> list[dict]: + out: list[dict] = [] + if not path or not Path(path).exists(): + return out + for line in Path(path).read_text().splitlines(): + if not line.strip(): + continue + status, _, desc = line.partition('|') + out.append({'ok': status == 'PASS', 'desc': desc}) + return out + + +def _read_notes(path: str) -> list[str]: + if not path or not Path(path).exists(): + return [] + return [ln for ln in Path(path).read_text().splitlines() if ln.strip()] + + +def emit_step(args) -> None: + rec = { + 'kind': 'step', + 'section': args.section, + 'id': args.id, + 'title': args.title, + 'verdict': args.verdict, + 'duration': float(os.environ.get('E2E_DUR', '0') or 0), + 'log': os.environ.get('E2E_LOGREL', ''), + 'asserts': _read_asserts(args.asserts), + 'notes': _read_notes(args.notes), + } + print(json.dumps(rec)) + + +def emit_skip(args) -> None: + rec = { + 'kind': 'step', + 'section': args.section, + 'id': args.id, + 'title': os.environ.get('E2E_SKIP_REASON', 'skipped'), + 'verdict': 'skip', + 'duration': 0.0, + 'log': '', + 'asserts': [], + 'notes': [], + } + print(json.dumps(rec)) + + +# -------------------------------------------------------------------------- +# assembly +# -------------------------------------------------------------------------- + +# Which step ids feed each "wiring" axis the user cares about. A step counts +# toward an axis if its id contains any of these substrings. +AXES = { + 'Correctness': ['acquire', 'serve', 'chat', 'models', 'coalesce', + 'release', 'leases', 'bundle', 'ttl', 'run-', 'ollama', + 'unknown', 'kubeai', 'missing', 'no-names', 'version'], + 'Efficiency': ['coalesce', 'reclaim', 'concurrency', 'placement', 'ttl'], + 'Ergonomics': ['paths', 'secrets', 'status', 'envfile', 'env', 'day2', + 'docker'], +} + + +def _load(results: Path) -> list[dict]: + recs: list[dict] = [] + if not results.exists(): + return recs + for line in results.read_text().splitlines(): + line = line.strip() + if line: + try: + recs.append(json.loads(line)) + except json.JSONDecodeError: + pass + return recs + + +def _badge(verdict: str) -> str: + return {'pass': '✅ pass', 'fail': '❌ FAIL', + 'skip': '⏭️ skip'}.get(verdict, verdict) + + +def assemble(args) -> None: + rdir = Path(args.results) + recs = [r for r in _load(rdir / 'results.jsonl') if r.get('kind') == 'step'] + meta = {} + if (rdir / 'run_meta.json').exists(): + meta = json.loads((rdir / 'run_meta.json').read_text()) + + n_pass = sum(r['verdict'] == 'pass' for r in recs) + n_fail = sum(r['verdict'] == 'fail' for r in recs) + n_skip = sum(r['verdict'] == 'skip' for r in recs) + total_dur = sum(r.get('duration', 0) for r in recs) + wall = None + wf = rdir / 'wall_seconds.txt' + if wf.exists(): + try: + secs = int(wf.read_text().strip()) + wall = f'{secs // 60}m {secs % 60:02d}s' + except ValueError: + wall = None + + L: list[str] = [] + L.append('# infer-stack leasing — e2e report') + L.append('') + L.append(f"- host: `{meta.get('host', '?')}`") + L.append(f"- when: {meta.get('when', '?')}") + L.append(f"- infer-stack: `{meta.get('version', '?')}` " + f"git: `{meta.get('git', '?')}`") + L.append(f"- tiers: {meta.get('tiers', '?')} " + f"GPU serving: **{'on' if meta.get('gpu') else 'off'}**") + L.append(f"- data dir: `{meta.get('data_dir', '?')}`") + L.append('') + verdict = '❌ FAILURES' if n_fail else '✅ all green' + L.append(f"## Result: {verdict} — " + f"{n_pass} passed, {n_fail} failed, {n_skip} skipped") + L.append('') + if wall: + L.append(f"- total wall time: **{wall}** " + f"({total_dur:.0f}s in step bodies)") + else: + L.append(f"- time in step bodies: {total_dur:.0f}s") + L.append('') + + # Wiring axes rollup + L.append('### Wiring axes') + L.append('') + L.append('| axis | pass | fail | skip |') + L.append('| --- | --- | --- | --- |') + for axis, keys in AXES.items(): + sel = [r for r in recs if any(k in r['id'] for k in keys)] + p = sum(r['verdict'] == 'pass' for r in sel) + f = sum(r['verdict'] == 'fail' for r in sel) + s = sum(r['verdict'] == 'skip' for r in sel) + L.append(f'| {axis} | {p} | {f} | {s} |') + L.append('') + + # Failures first — the thing to read + fails = [r for r in recs if r['verdict'] == 'fail'] + if fails: + L.append('## ❌ Failures (read these)') + L.append('') + for r in fails: + L.append(f"### {r['section']} / {r['id']} — {r['title']}") + for a in r['asserts']: + if not a['ok']: + L.append(f"- **failed:** {a['desc']}") + for nt in r.get('notes', []): + L.append(f"- note: {nt}") + log = rdir / r['log'] if r.get('log') else None + if log and log.exists(): + tail = log.read_text().splitlines()[-40:] + L.append('') + L.append('
log tail') + L.append('') + L.append('```') + L.extend(tail) + L.append('```') + L.append('
') + L.append('') + + # Full table grouped by section + L.append('## All steps') + L.append('') + sections: dict[str, list[dict]] = {} + for r in recs: + sections.setdefault(r['section'], []).append(r) + for section, items in sections.items(): + L.append(f'### {section}') + L.append('') + L.append('| step | verdict | dur (s) | checks | title |') + L.append('| --- | --- | --- | --- | --- |') + for r in items: + n_ok = sum(a['ok'] for a in r['asserts']) + n_all = len(r['asserts']) + checks = f'{n_ok}/{n_all}' if n_all else '-' + L.append(f"| `{r['id']}` | {_badge(r['verdict'])} | " + f"{r.get('duration', 0):.2f} | {checks} | " + f"{r['title']} |") + L.append('') + + # Environment dump + env_txt = rdir / 'environment.txt' + if env_txt.exists(): + L.append('## Environment') + L.append('') + L.append('```') + L.append(env_txt.read_text().rstrip()) + L.append('```') + L.append('') + + L.append('## Artifacts in this results dir') + L.append('') + L.append('- `report.md` — this file') + L.append('- `results.jsonl` — machine-readable per-step records') + L.append('- `environment.txt` — host/docker/gpu capture') + L.append('- `logs/` — full combined output of every step') + L.append('- `infer-stack-data/leasing/compose/` — rendered compose, ' + 'litellm config, secrets `.env`, sidecar state (if GPU ran)') + L.append('') + + (rdir / 'report.md').write_text('\n'.join(L) + '\n') + sys.stderr.write(f'wrote {rdir / "report.md"}\n') + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument('--emit-step', action='store_true') + ap.add_argument('--emit-skip', action='store_true') + ap.add_argument('--assemble', action='store_true') + ap.add_argument('--section', default='') + ap.add_argument('--id', default='') + ap.add_argument('--title', default='') + ap.add_argument('--verdict', default='pass') + ap.add_argument('--asserts', default='') + ap.add_argument('--notes', default='') + ap.add_argument('--results', default=os.environ.get('E2E_RESULTS', '.')) + args = ap.parse_args() + if args.emit_step: + emit_step(args) + elif args.emit_skip: + emit_skip(args) + elif args.assemble: + assemble(args) + else: + ap.error('one of --emit-step/--emit-skip/--assemble required') + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/dev/e2e_tests/run.sh b/dev/e2e_tests/run.sh new file mode 100755 index 00000000..83dca978 --- /dev/null +++ b/dev/e2e_tests/run.sh @@ -0,0 +1,231 @@ +#!/usr/bin/env bash +# infer-stack leasing — developer e2e harness (run this on yardrat). +# +# Exercises the new leasing/compose CLI surface end to end, captures every +# command's exit code / duration / output, asserts the wiring, and writes a +# single self-contained results dir (report.md + logs + rendered artifacts) you +# can rsync back for review. +# +# ./run.sh # non-serving tiers only (no GPU/docker pulls) +# ./run.sh --gpu # + the real serving tiers (vLLM/Ollama on GPU 0) +# ./run.sh --gpu --only '40 50' # just single-vllm + coalescing +# ./run.sh --gpu --keep-running # leave the compose stack up afterward +# +# Tiers WITHOUT --gpu: environment, dry-run (null backend), ergonomics, +# negative cases. They need infer-stack + docker-compose-v2 present but stand up +# nothing. Tiers WITH --gpu actually serve models and need GPU 0 free + images +# pre-pulled (see README). +set -uo pipefail + +E2E_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +export E2E_ROOT + +GPU=0 +ONLY='' +KEEP_RUNNING=0 +KEEP_DATA=0 +RESULTS='' +DATA_DIR='' +CATALOG="$E2E_ROOT/catalog.yaml" + +usage() { sed -n '2,30p' "$E2E_ROOT/run.sh"; exit "${1:-0}"; } + +while [ $# -gt 0 ]; do + case "$1" in + --gpu) GPU=1 ;; + --only) ONLY="$2"; shift ;; + --results) RESULTS="$2"; shift ;; + --data-dir) DATA_DIR="$2"; shift ;; + --catalog) CATALOG="$2"; shift ;; + --keep-running) KEEP_RUNNING=1 ;; + --keep-data) KEEP_DATA=1 ;; + -h|--help) usage 0 ;; + *) echo "unknown arg: $1" >&2; usage 1 ;; + esac + shift +done + +command -v infer-stack >/dev/null 2>&1 || { + echo 'error: infer-stack not on PATH (activate the venv first)' >&2; exit 1 +} +command -v python3 >/dev/null 2>&1 || { + echo 'error: python3 not on PATH' >&2; exit 1 +} + +TS="$(date +%Y%m%dT%H%M%S)" +RUN_START="$(date +%s)" +[ -n "$RESULTS" ] || RESULTS="$E2E_ROOT/results/$TS" +mkdir -p "$RESULTS" +[ -n "$DATA_DIR" ] || DATA_DIR="$RESULTS/infer-stack-data" +mkdir -p "$DATA_DIR" +# Isolate the config dir too (not just data) so `status`/`paths` reflect the +# e2e run, not the user's real ~/.config/infer_stack. Catalog is always passed +# explicitly via --catalog, so an empty config dir is fine. +CONFIG_DIR="$RESULTS/config" +mkdir -p "$CONFIG_DIR" + +# Exported context every test script + lib.sh relies on. +export E2E_RESULTS="$RESULTS" +export E2E_CAT="$CATALOG" +export E2E_ENABLE_GPU="$GPU" +export E2E_KEEP_RUNNING="$KEEP_RUNNING" +export INFER_STACK_DATA_DIR="$DATA_DIR" +export INFER_STACK_CONFIG_DIR="$CONFIG_DIR" + +: > "$RESULTS/results.jsonl" + +# ---- run metadata + environment capture ------------------------------------- +VERSION="$(infer-stack version 2>/dev/null | head -n1)" +GITREV="$(git -C "$E2E_ROOT" rev-parse --short HEAD 2>/dev/null || echo '?')" +GITDIRTY="$(git -C "$E2E_ROOT" status --short 2>/dev/null | wc -l | tr -d ' ')" +[ "$GITDIRTY" != '0' ] && GITREV="$GITREV+$GITDIRTY dirty" +HOST="$(hostname 2>/dev/null || echo '?')" +WHEN="$(date -Iseconds)" +TIERS=$([ "$GPU" = 1 ] && echo 'non-serving + GPU serving' || echo 'non-serving only') + +WHEN="$WHEN" HOST="$HOST" VERSION="$VERSION" GITREV="$GITREV" \ +TIERS="$TIERS" GPU="$GPU" DATA_DIR="$DATA_DIR" \ +python3 - "$RESULTS/run_meta.json" <<'PY' +import json, os, sys +json.dump({ + 'host': os.environ['HOST'], 'when': os.environ['WHEN'], + 'version': os.environ['VERSION'], 'git': os.environ['GITREV'], + 'tiers': os.environ['TIERS'], 'gpu': os.environ['GPU'] == '1', + 'data_dir': os.environ['DATA_DIR'], +}, open(sys.argv[1], 'w'), indent=2) +PY + +{ + echo "host: $HOST" + echo "when: $WHEN" + echo "infer-stack: $VERSION" + echo "git: $GITREV" + echo "data dir: $DATA_DIR" + echo "catalog: $CATALOG" + echo + echo '== uname =='; uname -a 2>/dev/null + echo; echo '== python =='; python3 --version 2>&1 + echo; echo '== docker =='; docker version --format '{{.Server.Version}}' 2>&1 + echo; echo '== docker compose =='; docker compose version 2>&1 + echo; echo '== nvidia-smi ==' + nvidia-smi --query-gpu=index,name,memory.total,memory.used,display_active \ + --format=csv 2>&1 || echo '(no nvidia-smi)' + echo; echo '== mem/disk =='; free -g 2>/dev/null; df -h "$DATA_DIR" 2>/dev/null +} > "$RESULTS/environment.txt" 2>&1 + +# ---- finish handler (runs on normal exit AND on Ctrl-C) --------------------- +# The data dir lives inside the results dir, so it also holds the multi-GB HF +# weight cache + compiled kernel caches. Those must NOT be rsync'd back — only +# the report, logs, and the small leasing artifacts (ledger, compose, litellm +# config, .env). These excludes drop the heavy dirs; leasing/ is kept. +RSYNC_EXCLUDES=( + "--exclude=.scratch/" + "--exclude=infer-stack-data/*-cache/" # hf-cache, vllm-cache, torch/triton/cuda + "--exclude=infer-stack-data/ollama/" + "--exclude=infer-stack-data/open-webui/" + "--exclude=infer-stack-data/postgres-*/" + "--exclude=infer-stack-data/runtime/" +) +# Compute the pull command once so it can be shown at the START (for syncing +# partial results mid-run) and again at the end. +_exc='' +for e in "${RSYNC_EXCLUDES[@]}"; do _exc+=" $e"; done +RSYNC_LINE="rsync -av${_exc} '$HOST:$RESULTS/' './e2e-report-$TS/'" +{ echo '#!/usr/bin/env bash' + echo '# Pull this report back (excludes the heavy weight/kernel caches).' + echo '# Safe to run mid-run for partial results.' + echo "$RSYNC_LINE" +} > "$RESULTS/rsync-back.sh" 2>/dev/null || true + +_finished=0 +finish() { + [ "$_finished" = 1 ] && return + _finished=1 + local elapsed=$(( $(date +%s) - RUN_START )) + printf '%d\n' "$elapsed" > "$RESULTS/wall_seconds.txt" 2>/dev/null || true + echo + python3 "$E2E_ROOT/render_report.py" --assemble --results "$RESULTS" \ + 2>/dev/null || echo '(report assembly failed; results.jsonl + logs/ still usable)' + + local summary + summary="$(python3 -c " +import json +recs=[json.loads(l) for l in open('$RESULTS/results.jsonl') if l.strip()] +recs=[r for r in recs if r.get('kind')=='step'] +p=sum(r['verdict']=='pass' for r in recs) +f=sum(r['verdict']=='fail' for r in recs) +s=sum(r['verdict']=='skip' for r in recs) +print(f'{p} passed, {f} failed, {s} skipped') +" 2>/dev/null || echo 'run incomplete')" + + local wall + wall="$(printf '%dm %02ds' $((elapsed / 60)) $((elapsed % 60)))" + + echo + echo "================================================================" + echo " $summary" + echo " total wall time: $wall" + echo " report: $RESULTS/report.md" + echo + echo " rsync back (excludes weight/kernel caches):" + echo " $RSYNC_LINE" + echo "================================================================" +} +trap 'finish' EXIT +trap 'echo; echo "[interrupted — assembling partial report]"; exit 130' INT TERM + +# Isolate GPU tiers: tear down any running stack between tiers so a leftover +# group (especially a keep-warm one that survives `release`) can't hold the only +# usable GPU and starve the next tier. Cheap relative to a tier; weights stay +# cached so it's a reload, not a re-download. +reset_between_tiers() { + local cf="$INFER_STACK_DATA_DIR/leasing/compose/docker-compose.yml" + [ -f "$cf" ] && \ + docker compose -p infer-stack -f "$cf" down --remove-orphans \ + >/dev/null 2>&1 || true + # Also tear down by project name in case the file is gone/stale. + docker compose -p infer-stack down --remove-orphans >/dev/null 2>&1 || true + # Wipe the shared ledger so each tier starts with zero leases/groups. + # Otherwise a leftover LIVE group — e.g. an `acquire` that returned + # not-ready keeps its lease active, or a swallowed cleanup missed one — + # persists in the ledger, gets re-realized by the next tier's reconcile, + # and holds the only usable GPU (starving the tier). Per-tier isolation. + rm -f "$INFER_STACK_DATA_DIR"/leasing/ledger.db \ + "$INFER_STACK_DATA_DIR"/leasing/ledger.db-wal \ + "$INFER_STACK_DATA_DIR"/leasing/ledger.db-shm 2>/dev/null || true +} + +# ---- select + run test scripts ---------------------------------------------- +echo +echo "infer-stack leasing e2e — $TIERS" +echo "results: $RESULTS" +echo +echo "rsync back (run any time for partial results; also in $RESULTS/rsync-back.sh):" +echo " $RSYNC_LINE" +echo + +for f in "$E2E_ROOT"/tests/*.sh; do + name="$(basename "$f")" + prefix="${name%%_*}" + if [ -n "$ONLY" ]; then + # 99_cleanup always runs so we never leak containers + case " $ONLY " in + *" $prefix "*) ;; + *) [ "$prefix" = '99' ] || continue ;; + esac + fi + export E2E_SECTION="${name%.sh}" + echo "── $E2E_SECTION ──────────────────────────────────────────" + # Reset GPU state before each *serving* tier (prefix >= 40), not the + # non-serving ones (01/10/20/30) — those share a ledger the dry-run/status + # checks rely on — and not 99_cleanup (it downs the stack itself). + if [ "$GPU" = 1 ] && [ "$((10#$prefix))" -ge 40 ] && [ "$prefix" != '99' ] + then + reset_between_tiers + fi + bash "$f" +done + +# Normal completion: the EXIT trap (finish) assembles the report, writes +# rsync-back.sh, and prints the summary + cache-excluding rsync line. +exit 0 diff --git a/dev/e2e_tests/tests/01_environment.sh b/dev/e2e_tests/tests/01_environment.sh new file mode 100644 index 00000000..6ecd1ee7 --- /dev/null +++ b/dev/e2e_tests/tests/01_environment.sh @@ -0,0 +1,25 @@ +# shellcheck shell=bash source=../lib.sh +# Environment capture + the bare-minimum preconditions. Always runs. +source "$E2E_ROOT/lib.sh" + +step version 'infer-stack is importable and reports a version' +run 'infer-stack version' +expect_rc 0 +expect_re 'infer-stack.*[0-9]+\.[0-9]+\.[0-9]+' +end_step + +step docker-compose 'docker compose v2 is available' +run 'docker compose version' +expect_rc 0 +expect_re 'v?2\.' +end_step + +if gpu_enabled; then + step docker-gpu 'docker can see the GPUs (nvidia runtime)' + run 'docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi' + expect_rc 0 + expect_out 'NVIDIA-SMI' + end_step +else + skip docker-gpu 'GPU serving disabled (run with --gpu)' +fi diff --git a/dev/e2e_tests/tests/10_dryrun.sh b/dev/e2e_tests/tests/10_dryrun.sh new file mode 100644 index 00000000..bc089ce9 --- /dev/null +++ b/dev/e2e_tests/tests/10_dryrun.sh @@ -0,0 +1,50 @@ +# shellcheck shell=bash source=../lib.sh +# The whole acquire -> leases -> env-file -> release surface on the null backend. +# No docker, no GPU: this isolates the ledger/CLI wiring from serving. +source "$E2E_ROOT/lib.sh" + +ENVF="$E2E_RESULTS/dryrun.env" + +step acquire-nowait 'acquire smol-135 (null backend, --no-wait, JSON)' +run "infer-stack acquire smol-135 --catalog \"$E2E_CAT\" \ + --env-file \"$ENVF\" --no-wait --json" +expect_rc 0 +expect_out '"session_id"' +expect_out 'smol-135' +end_step + +step envfile 'env-file is sourceable and carries the endpoint contract' +run "cat \"$ENVF\"" +expect_rc 0 +expect_file_has "$ENVF" 'export INFER_STACK_SESSION_ID=' +expect_file_has "$ENVF" 'INFER_STACK_ENDPOINT_SMOL_135=smol-135' +expect_file_has "$ENVF" 'INFER_STACK_MODELS=smol-135' +end_step + +step leases-active 'leases shows one active lease and one live group (demand 1)' +run 'infer-stack leases --json' +expect_rc 0 +expect_out '"state": "active"' +expect_out '"demand": 1' +end_step + +step release 'release by env-file tears the lease down cleanly' +run "infer-stack release --env-file \"$ENVF\"" +expect_rc 0 +end_step + +step leases-empty 'after release no active lease remains' +run 'infer-stack leases --json' +expect_rc 0 +expect_no_out '"state": "active"' +end_step + +step bundle 'acquiring a bundle resolves all its endpoints' +run "infer-stack acquire pair --catalog \"$E2E_CAT\" --no-wait --json" +expect_rc 0 +expect_out 'smol-135' +expect_out 'smol-360' +end_step + +# tidy the dry-run leases so they don't pollute later GPU tiers' ledger view +run 'infer-stack leases --json | python3 -c "import json,sys;[print(l[\"id\"]) for l in json.load(sys.stdin)[\"leases\"]]" | xargs -r -n1 infer-stack release >/dev/null 2>&1; true' diff --git a/dev/e2e_tests/tests/20_ergonomics.sh b/dev/e2e_tests/tests/20_ergonomics.sh new file mode 100644 index 00000000..7549b0eb --- /dev/null +++ b/dev/e2e_tests/tests/20_ergonomics.sh @@ -0,0 +1,47 @@ +# shellcheck shell=bash source=../lib.sh +# The day-2 / discovery ergonomics that must survive the leasing refactor: +# paths, secrets, status, and the day-2 compose wrappers' graceful behavior. +source "$E2E_ROOT/lib.sh" + +step paths-toplevel 'infer-stack paths surfaces the leasing artifacts' +run 'infer-stack paths' +expect_rc 0 +expect_out 'config:' +expect_out 'leasing:' +end_step + +step paths-leasing 'config paths leasing points at ledger / compose / secrets' +run 'infer-stack config paths leasing' +expect_rc 0 +expect_out 'ledger' +expect_out 'docker-compose.yml' +expect_out 'env (secrets)' +end_step + +step status 'status runs cleanly and prints its (leasing-aware) summary' +run 'infer-stack status' +expect_rc 0 +expect_no_out 'Traceback (most recent call last)' +expect_out 'backend:' +# Leasing-first: config.yaml is now reported as `legacy config:`, and a user +# with a catalog/settings is not nagged to run the legacy setup. +expect_no_out 'Not initialized' +# The one-line leasing summary only appears when the ledger is non-empty +# (by design); that path is covered by the unit test +# test_status_summarizes_leases_when_present. Here we just smoke `status`. +end_step + +step env-missing 'env gives a friendly error before any compose acquire' +run 'infer-stack env LITELLM_MASTER_KEY' +expect_rc_not 0 +expect_out 'no managed env-file' +note 'expected on a fresh data dir; becomes populated after a compose acquire' +end_step + +step day2-fallback 'day-2 ps degrades gracefully with no deployment + no config' +run 'infer-stack ps' +# No leasing compose yet and no legacy config.yaml -> should fail, not crash +# with a traceback. We only assert it does not emit a Python traceback. +expect_no_out 'Traceback (most recent call last)' +note "rc=$RC (a non-zero 'no config' style error is expected here)" +end_step diff --git a/dev/e2e_tests/tests/30_negative.sh b/dev/e2e_tests/tests/30_negative.sh new file mode 100644 index 00000000..d6a0a17e --- /dev/null +++ b/dev/e2e_tests/tests/30_negative.sh @@ -0,0 +1,31 @@ +# shellcheck shell=bash source=../lib.sh +# Negative / edge cases — these must fail *cleanly* (friendly message, non-zero +# exit), never with a raw traceback. Dry-run friendly. +source "$E2E_ROOT/lib.sh" + +step unknown-endpoint 'acquiring an unknown endpoint fails friendly' +run "infer-stack acquire nope --catalog \"$E2E_CAT\" --no-wait" +expect_rc_not 0 +expect_no_out 'Traceback (most recent call last)' +expect_re 'unknown|not found|no such|nope' +end_step + +step kubeai-unimplemented 'kubeai backend reports not-implemented, not a crash' +run "infer-stack acquire smol-135 --backend kubeai --catalog \"$E2E_CAT\" --no-wait" +expect_rc_not 0 +expect_no_out 'Traceback (most recent call last)' +expect_re 'not implemented|kubeai' +end_step + +step missing-catalog 'a missing catalog path fails friendly' +run 'infer-stack acquire smol-135 --catalog /no/such/catalog.yaml --no-wait' +expect_rc_not 0 +expect_no_out 'Traceback (most recent call last)' +expect_re 'catalog not found|not found' +end_step + +step no-names 'acquire with no endpoint names is rejected' +run "infer-stack acquire --catalog \"$E2E_CAT\" --no-wait" +expect_rc_not 0 +expect_no_out 'Traceback (most recent call last)' +end_step diff --git a/dev/e2e_tests/tests/40_single_vllm.sh b/dev/e2e_tests/tests/40_single_vllm.sh new file mode 100644 index 00000000..cf66c8c3 --- /dev/null +++ b/dev/e2e_tests/tests/40_single_vllm.sh @@ -0,0 +1,62 @@ +# shellcheck shell=bash source=../lib.sh +# The core path: acquire a single vLLM model on the Compose backend, prove it +# serves a real chat completion through the LiteLLM front door, exercise the +# day-2 wrappers against the live stack, then release (reclaim:stop -> gone). +source "$E2E_ROOT/lib.sh" + +if ! gpu_enabled; then + skip single-vllm-acquire 'GPU serving disabled (run with --gpu)' + skip single-vllm-models 'GPU serving disabled (run with --gpu)' + skip single-vllm-chat 'GPU serving disabled (run with --gpu)' + skip single-vllm-day2-ps 'GPU serving disabled (run with --gpu)' + skip single-vllm-secrets 'GPU serving disabled (run with --gpu)' + skip single-vllm-release 'GPU serving disabled (run with --gpu)' + exit 0 +fi + +ENVF="$E2E_RESULTS/single.env" + +step single-vllm-acquire 'acquire smol-135 --backend compose --require-generation' +run "infer-stack acquire smol-135 --backend compose --catalog \"$E2E_CAT\" \ + --require-generation --env-file \"$ENVF\" --timeout 1200 --json" +expect_rc 0 +expect_out '"ready": true' +note 'duration is the cold-start serve latency (image assumed pre-pulled)' +end_step + +step single-vllm-secrets 'managed LITELLM_MASTER_KEY is now fetchable' +run 'infer-stack env LITELLM_MASTER_KEY' +expect_rc 0 +expect_re '^sk-' +end_step + +step single-vllm-models 'the gateway lists smol-135 at the descriptor base_url' +run "set -a; . \"$ENVF\"; set +a; \ + curl -s \"\$OPENAI_BASE_URL/models\" -H \"Authorization: Bearer \$OPENAI_API_KEY\"" +expect_rc 0 +expect_out 'smol-135' +end_step + +step single-vllm-chat 'a real chat completion returns content' +run "set -a; . \"$ENVF\"; set +a; \ + curl -s \"\$OPENAI_BASE_URL/chat/completions\" \ + -H \"Authorization: Bearer \$OPENAI_API_KEY\" \ + -H 'Content-Type: application/json' \ + -d '{\"model\":\"smol-135\",\"messages\":[{\"role\":\"user\",\"content\":\"say hi\"}],\"max_tokens\":8}'" +expect_rc 0 +expect_out '"choices"' +expect_no_out 'model=None' +end_step + +step single-vllm-day2-ps 'day-2 wrappers target the live leasing stack' +run 'infer-stack ps' +expect_rc 0 +expect_re 'vllm|litellm' +end_step + +step single-vllm-release 'release tears down a reclaim:stop group' +run "infer-stack release --env-file \"$ENVF\"" +expect_rc 0 +run 'infer-stack leases --json' +expect_no_out '"state": "active"' +end_step diff --git a/dev/e2e_tests/tests/45_both_gpus.sh b/dev/e2e_tests/tests/45_both_gpus.sh new file mode 100644 index 00000000..f3cb091a --- /dev/null +++ b/dev/e2e_tests/tests/45_both_gpus.sh @@ -0,0 +1,48 @@ +# shellcheck shell=bash source=../lib.sh +# Both GPUs at once. Placement uses every GPU by default — including yardrat's +# display-attached GPU 1 — so two distinct models land on two distinct GPUs with +# no extra flag (`config set skip_display_gpus true` would exclude the display +# GPU). Proves the multi-GPU spread. +source "$E2E_ROOT/lib.sh" + +if ! gpu_enabled; then + skip both-gpus-spread 'GPU serving disabled (run with --gpu)' + exit 0 +fi + +SIDECAR="$INFER_STACK_DATA_DIR/leasing/compose/leasing-compose-state.json" + +step both-gpus-spread 'two distinct models spread across GPU 0 and GPU 1' +run "infer-stack acquire smol-135 --backend compose --catalog \"$E2E_CAT\" \ + --owner g0 --require-generation --timeout 1200" +expect_rc 0 +run "infer-stack acquire smol-360 --backend compose --catalog \"$E2E_CAT\" \ + --owner g1 --require-generation --timeout 1200" +expect_rc 0 +run "python3 -c ' +import json, sys +d = json.load(open(sys.argv[1])) +a = d.get(\"assignments\", {}) +gpus = sorted({g for v in a.values() for g in v}) +print(\"assignments:\", a) +print(\"distinct_gpus:\", gpus) +sys.exit(0 if len(gpus) >= 2 else 1) +' \"$SIDECAR\"" +expect_rc 0 +expect_out 'distinct_gpus: [0, 1]' +note 'if this fails placement (only GPU 0 used) the display-GPU override regressed' +end_step + +step both-gpus-both-routable 'both endpoints answer through the one gateway' +run "key=\$(infer-stack env LITELLM_MASTER_KEY); base=http://127.0.0.1:14042/v1; \ + for m in smol-135 smol-360; do \ + echo \"== \$m ==\"; \ + curl -s \"\$base/chat/completions\" -H \"Authorization: Bearer \$key\" \ + -H 'Content-Type: application/json' \ + -d \"{\\\"model\\\":\\\"\$m\\\",\\\"messages\\\":[{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"hi\\\"}],\\\"max_tokens\\\":8}\"; \ + echo; done" +expect_rc 0 +count_out '"choices"' 2 +# cleanup +run "infer-stack leases --json | python3 -c 'import json,sys;[print(l[\"id\"]) for l in json.load(sys.stdin)[\"leases\"] if l[\"state\"]==\"active\"]' | xargs -r -n1 infer-stack release >/dev/null 2>&1; true" +end_step diff --git a/dev/e2e_tests/tests/50_coalescing.sh b/dev/e2e_tests/tests/50_coalescing.sh new file mode 100644 index 00000000..35cc34d3 --- /dev/null +++ b/dev/e2e_tests/tests/50_coalescing.sh @@ -0,0 +1,45 @@ +# shellcheck shell=bash source=../lib.sh +# Coalescing & demand: two leases on the same model (smol-135-dup aliases smol-135) +# must share ONE deployment group (demand 2) and ONE vllm container. This is the +# efficiency headline — no duplicate model per consumer. +source "$E2E_ROOT/lib.sh" + +if ! gpu_enabled; then + skip coalesce-acquire 'GPU serving disabled (run with --gpu)' + skip coalesce-demand 'GPU serving disabled (run with --gpu)' + skip coalesce-one-container 'GPU serving disabled (run with --gpu)' + skip coalesce-release 'GPU serving disabled (run with --gpu)' + exit 0 +fi + +step coalesce-acquire 'two owners acquire the same underlying model' +run "infer-stack acquire smol-135 --backend compose --catalog \"$E2E_CAT\" \ + --owner alice --require-generation --timeout 1200" +expect_rc 0 +run "infer-stack acquire smol-135-dup --backend compose --catalog \"$E2E_CAT\" \ + --owner bob --require-generation --timeout 1200" +expect_rc 0 +end_step + +step coalesce-demand 'one group with demand 2, two active leases' +run 'infer-stack leases --json' +expect_rc 0 +expect_out '"demand": 2' +count_out '"state": "active"' 2 +end_step + +step coalesce-one-container 'exactly one live vllm group / running container' +run "infer-stack ps | grep -c vllm || true" +note "running vllm containers (expect 1): $(cat "$LAST_OUT_FILE")" +# Count only LIVE vllm groups — a reclaim:stop group from an earlier tier lingers +# in the ledger as state=stopped and must not count toward 'one container'. +run "infer-stack leases --json | python3 -c 'import json,sys; g=json.load(sys.stdin)[\"groups\"]; print(len([x for x in g if x[\"engine\"]==\"vllm\" and x[\"state\"]==\"live\"]))'" +expect_out '1' +end_step + +step coalesce-release 'releasing both leases drops demand to zero' +run "infer-stack leases --json | python3 -c 'import json,sys;[print(l[\"id\"]) for l in json.load(sys.stdin)[\"leases\"] if l[\"state\"]==\"active\"]' | xargs -r -n1 infer-stack release" +expect_rc 0 +run 'infer-stack leases --json' +expect_no_out '"state": "active"' +end_step diff --git a/dev/e2e_tests/tests/60_dedicated_f5.sh b/dev/e2e_tests/tests/60_dedicated_f5.sh new file mode 100644 index 00000000..12b73902 --- /dev/null +++ b/dev/e2e_tests/tests/60_dedicated_f5.sh @@ -0,0 +1,25 @@ +# shellcheck shell=bash source=../lib.sh +# Dedicated placement — on yardrat this deliberately exercises the display-GPU +# limit (F5): with only GPU 0 placeable, a 2nd dedicated group can't place. We +# record the behavior so the report shows whether F5 still bites (and whether +# the failure is graceful). +source "$E2E_ROOT/lib.sh" + +if ! gpu_enabled; then + skip placement-dedicated 'GPU serving disabled (run with --gpu)' + exit 0 +fi + +step placement-dedicated 'a 2nd dedicated group on a 1-GPU box cannot place (F5)' +run "infer-stack acquire smol-135 --backend compose --catalog \"$E2E_CAT\" \ + --owner a --require-generation --timeout 1200" +expect_rc 0 +run "infer-stack acquire smol-135 --backend compose --catalog \"$E2E_CAT\" \ + --owner b --dedicated --require-generation --timeout 45 --json" +note "dedicated acquire rc=$RC (expected non-zero / not-ready on a 1-GPU box)" +run 'infer-stack leases --json' +expect_no_out 'Traceback (most recent call last)' +note 'inspect groups above: the dedicated group should be requested but not live' +# cleanup +run "infer-stack leases --json | python3 -c 'import json,sys;[print(l[\"id\"]) for l in json.load(sys.stdin)[\"leases\"] if l[\"state\"]==\"active\"]' | xargs -r -n1 infer-stack release >/dev/null 2>&1; true" +end_step diff --git a/dev/e2e_tests/tests/70_ttl_reclaim.sh b/dev/e2e_tests/tests/70_ttl_reclaim.sh new file mode 100644 index 00000000..7f3a8fae --- /dev/null +++ b/dev/e2e_tests/tests/70_ttl_reclaim.sh @@ -0,0 +1,23 @@ +# shellcheck shell=bash source=../lib.sh +# Soft-TTL expiry + reclaim policy: a short-TTL lease expires without an explicit +# release, and a reclaim:stop group is reclaimed once no lease protects it. +source "$E2E_ROOT/lib.sh" + +if ! gpu_enabled; then + skip ttl-expire 'GPU serving disabled (run with --gpu)' + exit 0 +fi + +step ttl-expire 'a 30s TTL lease expires and its reclaim:stop group is reclaimed' +run "infer-stack acquire smol-135 --backend compose --catalog \"$E2E_CAT\" \ + --ttl 30s --require-generation --env-file \"$E2E_RESULTS/ttl.env\" --timeout 1200" +expect_rc 0 +note 'sleeping 40s to cross the soft TTL...' +run 'sleep 40' +run 'infer-stack leases --json' +expect_rc 0 +expect_no_out '"state": "active"' +note 'lease should be expired/released; smol-135 (reclaim:stop) group reclaimed' +# belt-and-suspenders cleanup +run "infer-stack release --env-file \"$E2E_RESULTS/ttl.env\" >/dev/null 2>&1; true" +end_step diff --git a/dev/e2e_tests/tests/80_run_wrapper.sh b/dev/e2e_tests/tests/80_run_wrapper.sh new file mode 100644 index 00000000..84574507 --- /dev/null +++ b/dev/e2e_tests/tests/80_run_wrapper.sh @@ -0,0 +1,29 @@ +# shellcheck shell=bash source=../lib.sh +# The `run` wrapper — the kwdagger pipeline-node seam: acquire, inject endpoint +# env into a child, release on exit, propagate the child's exit code. +source "$E2E_ROOT/lib.sh" + +if ! gpu_enabled; then + skip run-injects-env 'GPU serving disabled (run with --gpu)' + skip run-exit-code 'GPU serving disabled (run with --gpu)' + exit 0 +fi + +step run-injects-env 'run -- sees a working OpenAI endpoint, then releases' +run "infer-stack run --endpoint smol-135 --backend compose --catalog \"$E2E_CAT\" \ + --require-generation --timeout 1200 -- \ + bash -c 'curl -s \"\$OPENAI_BASE_URL/chat/completions\" \ + -H \"Authorization: Bearer \$OPENAI_API_KEY\" -H \"Content-Type: application/json\" \ + -d \"{\\\"model\\\":\\\"\$INFER_STACK_ENDPOINT_SMOL_135\\\",\\\"messages\\\":[{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"hi\\\"}],\\\"max_tokens\\\":8}\"'" +expect_rc 0 +expect_out '"choices"' +run 'infer-stack leases --json' +expect_no_out '"state": "active"' +note 'lease auto-released on child exit' +end_step + +step run-exit-code 'run propagates the child exit code' +run "infer-stack run --endpoint smol-135 --backend compose --catalog \"$E2E_CAT\" \ + --require-generation --timeout 1200 -- bash -c 'exit 7'" +expect_rc 7 +end_step diff --git a/dev/e2e_tests/tests/85_ollama.sh b/dev/e2e_tests/tests/85_ollama.sh new file mode 100644 index 00000000..0e180581 --- /dev/null +++ b/dev/e2e_tests/tests/85_ollama.sh @@ -0,0 +1,44 @@ +# shellcheck shell=bash source=../lib.sh +# Ollama daemon + lazy tag pull/warmup readiness. Run with GPU 0 free. +source "$E2E_ROOT/lib.sh" + +if ! gpu_enabled; then + skip ollama-acquire 'GPU serving disabled (run with --gpu)' + skip ollama-webui-wiring 'GPU serving disabled (run with --gpu)' + skip ollama-chat 'GPU serving disabled (run with --gpu)' + exit 0 +fi + +ENVF="$E2E_RESULTS/ollama.env" +COMPOSE="$INFER_STACK_DATA_DIR/leasing/compose/docker-compose.yml" + +step ollama-acquire 'acquire an ollama endpoint -> daemon up, tag pulled+warmed' +run "infer-stack acquire smol-ollama --backend compose --catalog \"$E2E_CAT\" \ + --require-generation --timeout 300 --env-file \"$ENVF\" --json" +expect_rc 0 +expect_out '"ready": true' +run 'infer-stack logs --tail=120' +expect_re 'pull|smollm2' +end_step + +step ollama-webui-wiring 'with the gateway on, Open WebUI gets BOTH connections' +# OpenAI (chat) -> the litellm gateway; native Ollama (pull/manage) -> the daemon +run "python3 \"$E2E_ROOT/inspect.py\" \"$COMPOSE\"" +expect_rc 0 +expect_out 'has litellm: True' +expect_out 'has open-webui: True' +expect_out 'open-webui OPENAI_API_BASE_URL: http://litellm:4000/v1' +expect_out 'open-webui ENABLE_OLLAMA_API: True' +expect_re 'open-webui OLLAMA_BASE_URL: http://ollama-' +note 'declared models route via the gateway; ad-hoc pulls show via the ollama conn' +end_step + +step ollama-chat 'a chat completion routes to the ollama daemon' +run "set -a; . \"$ENVF\"; set +a; \ + curl -s \"\$OPENAI_BASE_URL/chat/completions\" \ + -H \"Authorization: Bearer \$OPENAI_API_KEY\" -H 'Content-Type: application/json' \ + -d '{\"model\":\"smol-ollama\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":8}'" +expect_rc 0 +expect_out '"choices"' +run "infer-stack release --env-file \"$ENVF\" >/dev/null 2>&1; true" +end_step diff --git a/dev/e2e_tests/tests/86_ollama_lean.sh b/dev/e2e_tests/tests/86_ollama_lean.sh new file mode 100644 index 00000000..d8f24118 --- /dev/null +++ b/dev/e2e_tests/tests/86_ollama_lean.sh @@ -0,0 +1,56 @@ +# shellcheck shell=bash source=../lib.sh +# Lean --no-litellm Ollama + Open WebUI stack: no gateway at all. Open WebUI +# talks to the daemon's NATIVE Ollama API directly (manage-your-own-models), the +# declared tag is still pulled even without a gateway (the probe_ready fix), and +# a chat hits the daemon's own OpenAI-compatible port. This is the "replace my +# ollama server" path from the manual tutorial. Run with GPU 0 free. +source "$E2E_ROOT/lib.sh" + +if ! gpu_enabled; then + skip ollama-lean-serve 'GPU serving disabled (run with --gpu)' + skip ollama-lean-chat 'GPU serving disabled (run with --gpu)' + exit 0 +fi + +COMPOSE="$INFER_STACK_DATA_DIR/leasing/compose/docker-compose.yml" + +step ollama-lean-serve '--no-litellm: daemon up + tag pulled, no gateway, UI wired direct' +run "infer-stack acquire smol-ollama --no-litellm --backend compose \ + --catalog \"$E2E_CAT\" --require-generation --timeout 300 --json" +expect_rc 0 +expect_out '"ready": true' +# the tag was pulled even with NO gateway (probe_ready pulls before the +# litellm-routability check it would otherwise short-circuit on) +run 'infer-stack logs --tail=160' +expect_re 'pull|smollm2' +# rendered compose: no litellm; Open WebUI present and pointed straight at the +# ollama daemon's native API, OpenAI connection off (nothing to point at) +run "python3 \"$E2E_ROOT/inspect.py\" \"$COMPOSE\"" +expect_rc 0 +expect_out 'has litellm: False' +expect_out 'has open-webui: True' +expect_out 'open-webui host_port: 13000' # UI front door is up even with no gateway +expect_out 'open-webui ENABLE_OLLAMA_API: True' +expect_out 'open-webui ENABLE_OPENAI_API: False' +expect_re 'open-webui OLLAMA_BASE_URL: http://ollama-' +note 'lean stack: Open WebUI pulls/runs models straight from the ollama daemon' +end_step + +step ollama-lean-chat 'chat hits the ollama daemon directly (no gateway)' +# pull the daemon host port out of the rendered compose +run "python3 \"$E2E_ROOT/inspect.py\" \"$COMPOSE\" | grep -E 'ollama-.* host_port'" +expect_rc 0 +OPORT="$(awk '{print $NF}' "$LAST_OUT_FILE" | head -n1)" +note "ollama daemon host port: $OPORT" +# the daemon's own OpenAI-compatible endpoint; model is the TAG, not the alias +run "curl -s \"http://127.0.0.1:$OPORT/v1/chat/completions\" \ + -H 'Content-Type: application/json' \ + -d '{\"model\":\"smollm2:135m\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":8}'" +expect_rc 0 +expect_out '"choices"' +note 'chat answered straight from the daemon — no gateway in the path' +# status still works with no gateway (holistic overview, not the UI URL) +run 'infer-stack status' +expect_rc 0 +run "infer-stack leases --json | python3 -c 'import json,sys;[print(l[\"id\"]) for l in json.load(sys.stdin)[\"leases\"] if l[\"state\"]==\"active\"]' | xargs -r -n1 infer-stack release >/dev/null 2>&1; true" +end_step diff --git a/dev/e2e_tests/tests/88_gpu_pinning.sh b/dev/e2e_tests/tests/88_gpu_pinning.sh new file mode 100644 index 00000000..9f0e6cf7 --- /dev/null +++ b/dev/e2e_tests/tests/88_gpu_pinning.sh @@ -0,0 +1,47 @@ +# shellcheck shell=bash source=../lib.sh +# Explicit GPU pinning of an Ollama daemon to GPU 1 (yardrat's display GPU), the +# 2nd-GPU capability we couldn't exercise before. The daemon must land on GPU 1 +# via the docker device reservation (device_ids) and run ON the GPU — NOT fall +# back to CPU, which the old host-index CUDA_VISIBLE_DEVICES did (the reserved +# GPU is renumbered to 0 inside the container, so CUDA_VISIBLE_DEVICES=1 pointed +# at nothing). Placement uses every GPU by default (including the display GPU 1), +# so no extra flag is needed; `config set skip_display_gpus true` would exclude +# it. +source "$E2E_ROOT/lib.sh" + +if ! gpu_enabled; then + skip ollama-gpu1-pin 'GPU serving disabled (run with --gpu)' + skip ollama-gpu1-on-gpu 'GPU serving disabled (run with --gpu)' + exit 0 +fi + +COMPOSE="$INFER_STACK_DATA_DIR/leasing/compose/docker-compose.yml" + +step ollama-gpu1-pin 'daemon pinned to GPU 1: device_ids=[1], no host-index CUDA var' +run "infer-stack acquire smol-ollama-gpu1 --backend compose \ + --catalog \"$E2E_CAT\" --require-generation --timeout 300 --json" +expect_rc 0 +expect_out '"ready": true' +run "python3 \"$E2E_ROOT/inspect.py\" \"$COMPOSE\"" +expect_rc 0 +expect_out "device_ids: ['1']" +# the fix: pin by reservation only; the host-index env var must NOT be set +expect_out 'CUDA_VISIBLE_DEVICES: None' +SVC="$(grep -oE 'ollama-[^ ]+ host_port' "$LAST_OUT_FILE" | head -n1 | cut -d' ' -f1)" +OPORT="$(grep -E 'ollama-.* host_port' "$LAST_OUT_FILE" | awk '{print $NF}' | head -n1)" +note "ollama service: $SVC host port: $OPORT" +end_step + +step ollama-gpu1-on-gpu 'the pinned daemon runs ON the GPU, not a CPU fallback' +# load the model fresh, then ask the daemon which processor it is using +run "curl -s \"http://127.0.0.1:$OPORT/api/generate\" \ + -d '{\"model\":\"smollm2:135m\",\"prompt\":\"hi\",\"stream\":false}' >/dev/null; \ + docker compose -p infer-stack -f \"$COMPOSE\" exec -T \"$SVC\" ollama ps" +expect_rc 0 +# `ollama ps` PROCESSOR column shows GPU usage when on-GPU; full CPU fallback +# (the old bug) shows "100% CPU". +expect_out 'GPU' +expect_no_out '100% CPU' +note 'with the old host-index CUDA_VISIBLE_DEVICES bug this read 100% CPU' +run "infer-stack leases --json | python3 -c 'import json,sys;[print(l[\"id\"]) for l in json.load(sys.stdin)[\"leases\"] if l[\"state\"]==\"active\"]' | xargs -r -n1 infer-stack release >/dev/null 2>&1; true" +end_step diff --git a/dev/e2e_tests/tests/90_concurrency.sh b/dev/e2e_tests/tests/90_concurrency.sh new file mode 100644 index 00000000..99368b60 --- /dev/null +++ b/dev/e2e_tests/tests/90_concurrency.sh @@ -0,0 +1,32 @@ +# shellcheck shell=bash source=../lib.sh +# Concurrency / "last render wins" guard: two acquires race to converge the same +# shared compose file. The file lock must serialize them — no corrupt/half- +# written docker-compose.yml, no dropped service. +source "$E2E_ROOT/lib.sh" + +if ! gpu_enabled; then + skip concurrency-race 'GPU serving disabled (run with --gpu)' + skip concurrency-compose-valid 'GPU serving disabled (run with --gpu)' + exit 0 +fi + +step concurrency-race 'two concurrent acquires both land, demand is correct' +run "( infer-stack acquire smol-135 --backend compose --catalog \"$E2E_CAT\" \ + --owner u1 --require-generation --timeout 1200 ) & \ + ( infer-stack acquire smol-135-dup --backend compose --catalog \"$E2E_CAT\" \ + --owner u2 --require-generation --timeout 1200 ) & \ + wait" +expect_rc 0 +run 'infer-stack leases --json' +expect_out '"demand": 2' +count_out '"state": "active"' 2 +end_step + +step concurrency-compose-valid 'the shared compose file is schema-valid (not half-written)' +COMPOSE="$INFER_STACK_DATA_DIR/leasing/compose/docker-compose.yml" +run "docker compose -p infer-stack -f \"$COMPOSE\" config -q" +expect_rc 0 +expect_no_out 'error' +# cleanup +run "infer-stack leases --json | python3 -c 'import json,sys;[print(l[\"id\"]) for l in json.load(sys.stdin)[\"leases\"] if l[\"state\"]==\"active\"]' | xargs -r -n1 infer-stack release >/dev/null 2>&1; true" +end_step diff --git a/dev/e2e_tests/tests/99_cleanup.sh b/dev/e2e_tests/tests/99_cleanup.sh new file mode 100644 index 00000000..0de5b95b --- /dev/null +++ b/dev/e2e_tests/tests/99_cleanup.sh @@ -0,0 +1,14 @@ +# shellcheck shell=bash source=../lib.sh +# Always-run teardown: release any stragglers and down the compose project so a +# run never leaks containers. Honors E2E_KEEP_RUNNING=1 to leave the stack up. +source "$E2E_ROOT/lib.sh" + +step cleanup 'release stragglers and down the leasing compose project' +run "infer-stack leases --json | python3 -c 'import json,sys;[print(l[\"id\"]) for l in json.load(sys.stdin).get(\"leases\",[]) if l[\"state\"]==\"active\"]' 2>/dev/null | xargs -r -n1 infer-stack release >/dev/null 2>&1; true" +if [ "${E2E_KEEP_RUNNING:-0}" = "1" ]; then + note 'E2E_KEEP_RUNNING=1 -> leaving any compose stack up' +else + COMPOSE="$INFER_STACK_DATA_DIR/leasing/compose/docker-compose.yml" + run "[ -f \"$COMPOSE\" ] && docker compose -p infer-stack -f \"$COMPOSE\" down --remove-orphans 2>&1 || echo 'no compose project to down'" +fi +end_step diff --git a/dev/journals/claude.md b/dev/journals/claude.md index 30cca5e3..cc9e52f2 100644 --- a/dev/journals/claude.md +++ b/dev/journals/claude.md @@ -300,3 +300,1516 @@ Model: claude-sonnet-4-6 (via eval_audit harness). **Confidence:** High. The rename is purely mechanical (string substitution + directory mv). All 22 Python files compile cleanly. No residual `vllm_service`/`vllm-stack`/`VLLM_SERVICE` references remain outside historical journal entries and the tarball artifact. **Next steps for the user:** Update `eval_audit` (the parent repo) to reference `infer_stack` instead of `vllm_service` in any imports, config references, or scripts that use the submodule. The GitHub remote rename is also handled separately by the user. + +## 2026-06-16 14:43:08 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. + +**User intent:** Begin the long-discussed `infer-stack` redesign toward a +leasing/controller model so multiple users running multiple `kwdagger` +pipelines can `acquire` the models a node needs, block until ready, and +`release` after — instead of the current single global "active profile" that is +last-render-wins under a shared data dir. The full critique + staged plan lives +in the *parent* repo at `aiq-eval-runner/dev/infer-stack-redesign-critique.md` +(grounded in the two real consumers: aiq-eval-runner's Incubilate/Contextual +Drag and eval_audit's HELM/kwdagger path). User explicitly asked to start on a +new branch and bump versions; chose sqlite for the store (open question #1 in +that doc). This is Phase 1's foundation. + +**What I built (branch `dev/leasing-controller`, version 0.6.1 -> 0.7.0):** +A new backend-agnostic `infer_stack/leasing/` subpackage — the *controller* +half, deliberately separate from the existing stateless compiler +(`resolver`/`validator`/`renderer`), which is kept intact. +- `models.py`: `EndpointRequest` (catalog-resolved input), `Lease`, + `DeploymentGroup`, the `compatibility_key` (pure structural identity hash), + `vllm_structural`/`ollama_structural` builders, and `capacity_satisfies` + (subsumption, not equality). +- `store.py`: `SqliteStore` — autocommit + WAL + `busy_timeout`, with a + `transaction()` context manager that issues `BEGIN IMMEDIATE` so the + find-or-create-group critical section is race-safe across processes. +- `ledger.py`: `Ledger.acquire/release/renew/sweep/reclaimable_groups/status`. + Demand = count of *protecting* leases (ACTIVE and not past TTL), computed by a + SQL `COUNT(DISTINCT lease_id)` join, not stored. Groups flip LIVE<->IDLE on + demand; teardown of IDLE groups is left to the future reconciler per reclaim + policy. +- 17 unit tests + 3 xdoctests, all passing; ruff clean; existing fast suite + unaffected. Ran via `uv run --extra tests --with xdoctest` (xdoctest is in + pyproject addopts but not the tests extra — had to add it; the system python + has no pytest). + +**Key design decisions / reflections:** +- The central reframe is compiler-vs-controller. I resisted bolting acquire/TTL + onto the resolver; the ledger is a separate stateful layer with its own store. + This is what makes "last render wins" go away: desired state becomes the union + of live leases, not one mutated profile. +- Coalescing grain = "the thing that gets a process": per-model for vLLM, + per-daemon for Ollama. Modeled uniformly by making the Ollama structural key + the *host config* (no tag), so many tag endpoints collapse to one group whose + `served` map accumulates the tags. This kept one code path for both engines. +- Compatibility is structural-equality + capacity-subsumption, NOT a single + equality key. A 32k deployment serves an 8k request; the reverse makes a new + group. Sharing policy is handled in the ledger (dedicated => always new), + deliberately kept OUT of the hash so the key stays a clean deployment identity. +- TTL is soft and is the crash backstop: protection lapses by time immediately + in the demand query; `sweep()` only *materializes* the EXPIRED state for + status/reclaim. A job killed by SIGKILL (no release) is recovered by TTL. + +**Uncertainties / risks / what might break:** +- The ledger is pure bookkeeping with NO backend yet — nothing realizes, + probes, or tears down deployments. `AcquireResult.groups` is the handoff + point for the next slice (reconciler + 4-method backend protocol + readiness + with the Ollama pull/warmup rung). +- `EndpointRequest` is hand-built in tests; the catalog.yaml parser that + produces them (new schema: models/endpoints/hosts/bundles) is not written. +- Cross-process concurrency relies on `BEGIN IMMEDIATE` + WAL + busy_timeout; I + reasoned it through and unit-tested the single-process invariants, but did not + write a true multi-process stress test. If contention shows up, that's where + to look first. +- Group `spec` is taken from the first creating request; coalesced requests' + specs are assumed compatible by construction. Correct given the compat key, + but worth remembering when the backend starts consuming `spec`. + +**Confident about:** the object model and the coalescing/TTL/demand semantics +(well covered by tests), the layering (controller cleanly separated from the +compiler, no edits to existing modules beyond the version bump), and that this +is the right minimal foundation for Phases 1-4. + +**Next steps:** catalog.yaml parser -> EndpointRequest; the Reconciler + Backend +protocol (`realize/teardown/observe/probe_ready`) with Compose first (reuse +`_first_fit`, `_wait_until_ready`, the live LiteLLM router refresh); then the +`acquire`/`release`/`run`/`status` CLI emitting the `contracts.py` env-file. + +## 2026-06-16 15:05:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. Continuation of +the same session; committed the ledger as `b1ebf17`, then built the next stage. + +**What I built (same branch `dev/leasing-controller`):** the serving-catalog +parser `infer_stack/leasing/catalog.py` — the declarative input side of the +controller and the new replacement for "profiles" as the primary unit. Schema: +`models` / `endpoints` / `runtime_hosts` / `bundles`. `Catalog.from_dict`/`load` +parse + validate (cross-reference checks: vllm endpoint -> model, ollama +endpoint -> host, bundle -> endpoints, known engines). `resolve_endpoint` and +`resolve_names` turn endpoint/bundle names into the ledger's `EndpointRequest`s, +so the catalog composes directly with what I built in the prior commit. 15 +catalog tests incl. ledger-integration (alias endpoints coalesce; an ollama +bundle collapses to one daemon group serving both tags), 32 leasing tests total, +4 xdoctests, ruff clean. + +**Decisions / reflections:** +- Kept the catalog in the `leasing/` subpackage (not the legacy top-level + `catalog.py`) — it is the *serving* catalog for the new model and should not + be conflated with the old vllm_models/profiles catalog. +- `served_name`/`public_name` is the endpoint's exposed name; the alias case + (two endpoint names, same model+runtime) intentionally produces the same + `compat_key`, so distinct catalog entries still coalesce onto one deployment. + This validates the "compat key = deployment identity, not endpoint name" + decision from the ledger stage. +- vLLM `EndpointRequest.spec` carries hf_model_id + runtime + reclaim; ollama + `spec` carries host/image/settings/gpu_indices + reclaim. These are the + payloads the (not-yet-written) backend will consume to realize a process. + +**Risks / unknowns:** still no backend — `spec` shapes are my best guess at what +compose/kubeai renderers will need and may shift when the reconciler lands. The +new catalog schema is not yet wired to any CLI or migrated from the existing +`config.yaml`/profiles (the legacy `_legacy_profile_to_stack` migration is +future work). No backwards-compat shim yet between the old and new catalogs. + +**Next:** the Reconciler + a 4-method Backend protocol +(`realize`/`teardown`/`observe`/`probe_ready`), testable first with a fake +backend, then a Compose implementation reusing the existing placement/readiness +machinery. + +## 2026-06-16 15:30:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. Same session; +committed the catalog as `1274470`, then built the reconcile layer. + +**What I built (same branch):** the backend seam + controller. +- `backend.py`: a tiny 4-method `Backend` Protocol + (`realize`/`teardown`/`observe`/`probe_ready`, all idempotent) + `Readiness` + + a `MemoryBackend` (records calls, configurable per-group/per-endpoint + readiness) for tests and dry-runs. This Protocol is the single thing a real + backend (Compose/KubeAI) must implement — and the deliberate line between + "infer-stack coordinates" and "the backend/k8s schedules". +- `controller.py`: `Controller(ledger, backend)` with the desired-vs-actual + reconcile loop, `wait_ready`, and thin `acquire`/`release`. 11 controller + tests (incl. catalog integration), 43 leasing tests total, 5 xdoctests, ruff + clean. + +**Decisions / reflections:** +- Named the orchestrator `Controller` (module `controller.py`) and kept the + reconcile *loop* as its `reconcile()` method. The design doc says "Reconciler"; + I went with Controller because the object also owns the thin acquire/release + orchestration, and "reconciler" reads odd as the home of `acquire`. The loop + is still literally a reconciler. +- `reconcile()` calls `ledger.sweep()` first. Without it, a group whose only + lease has expired by time still has `state == LIVE` in the row (nothing + flipped it), so it would stay "desired". Sweeping makes TTL self-enforcing on + every reconcile — the crash backstop actually fires. +- Desired set = LIVE groups + IDLE groups whose reclaim policy is keep-warm. + Teardown falls out of the diff (actual - desired). I deliberately did NOT add + a STOPPED group state: after teardown the backend's `observe()` no longer + returns the group, so it isn't re-torn-down, and a later acquire reuses the + IDLE row and re-realizes it. Fewer states, same behavior. +- `wait_ready` is scoped to the endpoints the *lease* requested, not all of a + coalesced group's served endpoints. Important for Ollama: a daemon may serve + several tags; a node acquiring one tag shouldn't block on a sibling tag's + health. Tested explicitly. + +**Risks / unknowns:** still no real backend — `MemoryBackend` proves the +control logic but not the docker/GPU reality. The reclaim model has no +*pressure* concept yet: keep-warm idle groups stay up forever (reclaimed only +when explicitly stop/scale-to-zero, or reused). That's intended for now but the +Compose backend will need a pressure-driven reaper when GPUs are contended. +`realize` is per-group; the Compose backend renders one compose file for the +union, so it will likely converge the whole desired set at once rather than +per-call — the Protocol allows either, but I haven't validated the batch shape. + +**Next:** the Compose backend implementing this Protocol (reuse `_first_fit`, +`_wait_until_ready` + Ollama pull/warmup, the live LiteLLM router refresh), then +the `acquire`/`release`/`run`/`status` CLI emitting the `contracts.py` env-file. + +## 2026-06-16 16:05:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. Same session; +committed the controller as `b512c62`, then built the CLI surface. + +**What I built (same branch):** the leasing CLI verbs + the env-file/descriptor +emitter + a dry-run backend. +- `leasing/envfile.py`: the endpoint descriptor (aligned with `contracts.py`: + base_url / api_key_env / models) and its sourceable shell env-file + (`INFER_STACK_SESSION_ID`, `OPENAI_BASE_URL`, `INFER_STACK_ENDPOINT_`, + `INFER_STACK_MODELS`, optional `CUDA_VISIBLE_DEVICES`). `read_session_id` + recovers the session from a written env-file so `release --env-file` works. +- `leasing/backend.py`: added `NullBackend` (dry-run) — `observe()` returns the + empty set so the controller no-op-realizes everything and never tears down; + readiness is immediate; no in-memory state, so it stays coherent across + separate CLI invocations (the persistent ledger is the only truth). +- `cli/commands_leasing.py`: `acquire` / `serve` / `release` / `renew` / `run` / + `leases`, registered on `ManageCLI`. `run -- ` acquires, injects the + endpoint env into the child, runs, releases on exit, propagates exit code. +- 13 CLI tests + a real end-to-end shell smoke through `python -m + infer_stack.cli`. 66 leasing/CLI tests total, 6 xdoctests, ruff clean. + +**Decisions / reflections:** +- Named the status verb `leases` (not `status`) because the legacy profile model + already owns `infer-stack status`. The two models coexist during the + transition; `leases` is unambiguously the new one. +- Default `--backend null`. The whole acquire/release/run/env-file flow is + exercisable without docker, which is the point of doing the CLI before the + Compose backend: it locks the user-facing contract (verbs + env-file shape) + that the consumer repos and the Compose backend then build against. +- Hit the scriptconfig smartcast trap: a `nargs='*'` positional smart-splits + string elements on commas, which mangled `run -- python -c "import os, ..."` + into a nested list. Fix: `type=str` on the positional/`command` values and do + comma-splitting myself in `_collect_names`. (Worth a dev/lesson if it recurs.) +- For Ollama the env-file's request-model name is the *tag* (e.g. `qwen3.5:4b`), + not the endpoint name — confirmed in the smoke (`INFER_STACK_MODELS=qwen3.5:4b`). + `build_descriptor` reads it from the group's per-endpoint served payload. + +**Risks / unknowns:** `base_url` in the descriptor is a dry-run placeholder +(`--base-url`, default the LiteLLM port) — the real URL must come from the +Compose/KubeAI backend once it exists, so the descriptor's `base_url` and +`api_key_env` will be backend-supplied later (likely a 5th backend method or an +access() call). `run` releases in a `finally`, but a hard kill (SIGKILL) skips +it — that's exactly why `run` defaults `--ttl 2h` as the backstop, and why a +periodic `sweep` (who runs it?) is still an open v1 question. Concurrency across +processes is still only single-process tested. + +**Next:** the Compose backend implementing the `Backend` Protocol — the first +time this touches docker/GPUs (renderer adapter from DeploymentGroups, GPU +placement over the union, cross-process compose-file lock). + +## 2026-06-16 16:35:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. Same session; +committed the CLI as `0100a52`. Starting the Compose backend, which I'm building +in fully-testable sub-slices because this sandbox has no GPUs/docker. + +**What I built (same branch):** sub-slice 1 — the single-host GPU placement +planner `leasing/placement.py`. `plan_placement(groups, inventory, *, +allowed_gpus, reserved, pinned, skip_display) -> GpuPlan`. Reuses the resolver's +`_first_fit` and `_available_gpu_indices`. 15 tests (simulated inventories), +71 leasing/CLI tests total, 7 xdoctests, ruff clean. + +**Decisions / reflections:** +- This is the placement-over-the-union the ledger deliberately deferred. I made + it a standalone module (not buried in the Compose backend) because it is pure, + fully testable without docker, and Phase-2 raw-GPU reservations need the same + `reserved` exclusion. +- Three-tier deterministic order — pinned, then explicit, then first-fit — with + groups sorted by (created_at, id). `pinned` is the stability mechanism: a + group already running on [0,1] keeps it when a sibling is added, so reconciles + don't reshuffle live models. An invalid pin (GPU gone) silently re-places. +- Ollama daemons pin explicitly (their host `gpu_indices`, possibly `[]` for + CPU); vLLM groups first-fit by `tp × dp`. One planner, both engines. +- Deferred VRAM-fit validation: the leasing catalog doesn't carry per-model + memory yet, so placement is by GPU *count* only. The legacy validator's + VRAM/headroom check is the reference to port when ModelSpec grows a + `min_vram_gib` field. + +**Risks / unknowns:** still no docker. The next sub-slices (render +DeploymentGroups -> compose artifacts + `docker compose up`; observe via +`docker compose ps`; teardown) will be built behind an injected runner seam and +unit-tested against fakes, but the real docker/GPU path can only be validated by +the user on the GPU host — I'll mark that explicitly. The big open design choice +for the next slice: reuse the legacy `resolve()` + compose renderer by +synthesizing a config from the groups, vs. write a focused renderer straight +from DeploymentGroups. Leaning toward a focused renderer to avoid forcing the +new model through the legacy config schema, but will decide after reading the +renderer's exact inputs. + +**Next:** Compose render/realize/observe/teardown behind a docker-runner seam. + +## 2026-06-16 17:10:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. Same session; +committed placement as `3f136a3`. Built Compose backend slice 2. + +**What I built (same branch):** `leasing/compose.py` — a focused compose +renderer + `ComposeBackend`, plus a controller `converge` branch. +- Decided **focused renderer over reusing legacy `resolve()`/`render_compose_artifacts`**: + the legacy renderer is welded to the resolved-deployment v5 schema + (providers/gateways/frontends, jinja templates, diff-gated writes); synthesizing + that from DeploymentGroups would be a large fragile mapping. The focused + renderer emits a compose dict straight from groups and *reuses + `profile_runtime.vllm_args`* for the genuinely tricky vLLM flag-building. +- `ComposeBackend` is converge-style: place (pinned from a persisted sidecar) -> + render -> write compose.yml + sidecar -> `docker compose up -d + --remove-orphans`. observe() parses `docker compose ps --format json` and maps + running services back to group ids via the sidecar. Docker is an injected + `run(args)->str` seam. +- Controller `reconcile` now prefers `backend.converge(desired)` when present + (whole-union convergence) and falls back to the per-group realize/teardown + loop for Memory/Null backends. Computed realized/torn_down from before/after + observe() diffs. +- Wired `--backend compose` into the CLI (`detect_inventory`, data_root state + dir, `--allowed-gpus`). 90 leasing/CLI tests, 7 xdoctests, ruff clean. Tested + with a *stateful* FakeDocker (`up` reflects the rendered compose file, `ps` + returns it, `--remove-orphans` drops gone services) so converge->observe is + coherent end to end without docker. + +**Decisions / reflections:** +- Per-group `realize`/`teardown` never fit compose's one-file model; rather than + contort the protocol I added an optional `converge` the controller detects via + `hasattr`. ComposeBackend therefore implements converge/observe/probe_ready + (not realize/teardown) — it duck-types into the controller's converge path. +- `--remove-orphans` *is* teardown: converge always renders only the desired + groups, so a dropped group's service becomes an orphan and is removed. No + separate teardown docker call needed. +- Placement stability via the persisted sidecar: converge feeds current + assignments as `pinned`, so adding/removing a group never migrates a running + model's GPUs. + +**Risks / unknowns (for when you test on the GPU host):** +- Readiness is only "container running" this slice. `acquire --backend compose` + returns once the vLLM *container* is up, NOT once the model has finished + loading/serving. Real generation-probe readiness + Ollama pull/warmup come in + slice 3. +- The env-file `base_url` is still the `--base-url` placeholder (default + :14042). With compose-and-no-LiteLLM the real endpoint is the per-vLLM direct + port (18000+i). The LiteLLM front door in slice 3 is what makes one stable + base_url correct; until then pass `--base-url` explicitly or hit the direct + port. +- Cross-process compose-file writes are not yet file-locked — concurrent + `acquire`s from two processes could race on the compose file. The ledger's + BEGIN IMMEDIATE covers the lease rows but not the file; a flock around + converge is on the slice-3/hardening list. +- No real docker/GPU run in this sandbox; the FakeDocker proves the logic. You + validate the real path. + +**Next (slice 3):** LiteLLM front door (alias routing -> served names, real +base_url) + real readiness (`_wait_until_ready` generation probe, Ollama +pull/warmup) + a converge file-lock. + +## 2026-06-16 17:45:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. Same session; +committed compose slice 2 as `301d4de`. Built slice 3. + +**What I built (same branch):** the LiteLLM front door + access/descriptor +wiring + converge file-lock + routability readiness, all in `compose.py` + +`envfile.py` + `commands_leasing.py`. +- LiteLLM service + config rendered from the group set (default on): one + `model_list` entry per served endpoint alias -> upstream `openai/` at + `http://vllm-:8000/v1` (or `ollama/`). This is what makes one stable + `base_url` possible and is why the alias (endpoint name), not the upstream + served name, is what a client requests. +- `ComposeBackend.access(endpoints)` returns the real `base_url` (LiteLLM port), + `api_key_env`, and per-endpoint request names. The CLI's new `_descriptor_for` + prefers `backend.access` over the `--base-url` placeholder; `build_descriptor` + gained a `request_names` override. NullBackend has no `access` -> CLI falls + back unchanged (verified by smoke). +- `probe_ready` upgraded: container-running (observe) AND the alias appears in + the gateway's `/v1/models`, via an injected `http_get` seam. Tested with a + fake HTTP that lists whatever the rendered LiteLLM config declares — realistic + and fully offline. +- `converge` wrapped in an `fcntl.flock` (the cross-process compose-file race I + flagged in slice 2). 94 leasing/CLI tests, 7 xdoctests, ruff clean. + +**Decisions / reflections:** +- Backend-supplied access (a 5th, optional method discovered via `getattr`) is + the clean fix for "who knows the real base_url": the backend does, not the + CLI. Same hasattr/duck-type pattern as `converge`. Keeps Null/Memory backends + untouched. +- Readiness = "alias listed by the gateway" is the honest, testable middle + between "container up" (too weak) and "generation succeeds" (needs a live + model). A real generation probe + Ollama pull/warmup is the remaining rung. +- Caught a test-helper bug while wiring this: my `vllm()` test helper had keyed + the group's `served` map by the *served name* instead of the endpoint alias, + so the LiteLLM alias didn't match the probed endpoint. Fixed the helper to key + by endpoint (group id) with served_model_name as the value — which is exactly + the endpoint-vs-served distinction the front door depends on. + +**Risks / unknowns (for host testing):** still no real docker/HTTP here — the +seams prove the logic; the LiteLLM container actually starting, vLLM upstreams +becoming healthy, and the gateway listing models is the host-validated part. +Ollama pull/warmup not implemented (a daemon serves a tag lazily; readiness +should pull + warm it). The flock only guards a single host's state dir. + +**Next:** the migration bridge (`migrate-config` from legacy profiles; `switch` +as a standing-lease shim) and Ollama pull/warmup readiness; then consumer +integration in eval_audit / aiq-eval-runner. + +## 2026-06-16 18:30:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. Same session. +User redirected: drop the migration-bridge / back-compat work (no backwards +compatibility needed) and instead make the code more maintainable. They chose +"consolidate shared machinery" — fold duplicated logic into one reused core, +without deleting the legacy CLI surface. + +**What I did (two tested commits):** +1. `9c8e05a` GPU primitives -> `hardware.py`: `leasing/placement` had been + importing the resolver's *private* `_first_fit`/`_available_gpu_indices`. + Moved those + `_resolve_gpu_indices` to `infer_stack.hardware` as public + functions; resolver imports them back under the old private aliases (zero + internal churn), leasing imports the public names. One home for GPU-pool + placement. +2. HTTP probes -> new `infer_stack/probe.py`: `cli/probes.py` had + `_ready_openai_probe`/`_ready_ollama_probe` (pure HTTP, but stuck in the cli + layer), while `leasing/compose` had its own weaker `/v1/models` check. Moved + the probes to a layer-neutral `probe.py` over an injectable `http` client; + `cli/probes` re-exports them under the old names for legacy callers; leasing's + `probe_ready` now calls `openai_ready(require_listed=True, + require_generation=...)`, so it gained the advertised-alias check and an + optional real-generation rung for free. + +**Decisions / reflections:** +- Verified the test patch mechanism before moving the probes: the suite does + `monkeypatch.setattr(cli_mod.requests, 'get', ...)` — patching the shared + `requests` *module object*'s attributes, which intercepts `requests.get` + everywhere regardless of which module calls it. So relocating the probe body + was safe; full suite (197) confirms it. +- Kept the schema-tied helpers (`_default_model_for_deployment`, + `_resolve_smoke_protocol_from_deployment`) in `cli/probes` — they read the + legacy v5 deployment dict and aren't shareable. Only the pure HTTP probes + moved. +- Layering: `probe.py` and the GPU primitives in `hardware.py` are below both + `cli` and `leasing`, so neither consolidation created an upward import. This + is the right shape — shared mechanism sinks to a common low layer. +- Standardized leasing's HTTP seam from a bespoke `http_get(url)->(status,body)` + callable to a requests-like `http` client (`.get`/`.post`), matching the probe + and the real `requests` — one seam shape across the codebase. + +**Risks:** behavior-preserving by construction; the only behavior *change* is +leasing readiness getting stricter (alias must be advertised), which matches +intent. Full suite green. Still no real docker/HTTP here — seams prove it. + +**Next:** Ollama pull/warmup readiness (could reuse `probe.ollama_ready`), then +consumer integration; further consolidation candidates if any surface (the +LiteLLM config template vs the leasing dict builder is the remaining duplicate, +but template-vs-dict makes it lower-value). + +## 2026-06-16 18:55:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. Same session. +Did the next functional step: Ollama pull/warmup readiness. + +**What I built:** `ComposeBackend.probe_ready` now handles the Ollama lazy-load +gap. An Ollama daemon being up (and the alias being in the LiteLLM config) does +NOT mean the tag is resident, so for ollama groups probe_ready: +1. pulls the endpoint's tag into the daemon via `docker compose exec -T + ollama- ollama pull ` (idempotent; tracked in `self._pulled` so + it's issued once per process; failures are retryable so the poll loop heals + once the daemon accepts the pull), then +2. forces `require_generation=True` so the front-door readiness check runs a + real chat through LiteLLM->ollama — which both confirms the tag loads and + warms it. +vLLM readiness is unchanged (alias-listed by default); a new `--require-generation` +CLI flag opts vLLM into the same real-generation check. + +**Decisions / reflections:** +- Kept everything through the LiteLLM front door (one base_url) rather than + reusing `probe.ollama_ready` against the daemon directly — the only + ollama-specific action is the *pull*, which needs no host-port tracking + (`exec` by service name). So I did NOT end up calling `probe.ollama_ready` + here; it stays for the legacy direct-ollama path. The unifying insight: with a + gateway, readiness is engine-agnostic (openai_ready) and the only divergence + is the side-effecting pull. +- Pull lives in probe_ready (not converge) on purpose: right after `up -d` the + daemon isn't ready to accept a pull, and the poll loop is exactly the retry + mechanism. Idempotence + `_pulled` keep it from re-pulling. +- "alias listed" is too weak for ollama (LiteLLM lists configured-but-unpulled + tags), which is why ollama forces generation — the honest readiness signal. + +**Risks:** as ever, no real docker/ollama here — FakeDocker handles `exec` and +the pull command is asserted; the real pull (which can download GBs and block +the first poll) is host-validated. 17 compose tests (3 new ollama), 88 +leasing/CLI + 7 xdoctests green. + +**Next:** consumer integration — wrap eval_audit's MaterializeHelmRunNode command +with `infer-stack run`, and switch aiq-eval-runner Incubilate to acquire/release. + +## 2026-06-17 12:20:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. User started +testing the Compose backend on real hardware (yardrat: 2 GPUs — RTX 8000 free, +RTX 5000 display-attached; both Turing/sm_75, fp16-only). + +**First real-hardware bug (F1), FIXED:** `docker compose up` rejected the +rendered file — `_gpu_reservation` emitted `capabilities: [['gpu']]` but the +Compose schema wants a list of *strings* (`capabilities: ['gpu']`). My nested +list was a python-on-whales-ism. Fixed + locked with a test asserting +`devs['capabilities'] == ['gpu']`. This is exactly the class of bug the unit +tests couldn't catch (they asserted device_ids, never the capabilities shape, +and no real `docker compose config` validation runs offline). + +**Wrote `dev/leasing-test-plan.md`** — a living runbook: setup, 9 staged test +blocks (each self-contained, re-exporting its own env so blocks don't depend on +each other), a findings/fixes log, debug-capture, and cleanup. Tailored to +yardrat: Turing `--dtype=half` in every vLLM endpoint, tiny ungated chat models, +and explicit calls-out of the predicted issues. + +**Predicted issues now documented as findings to confirm/fix:** +- F2 (setup): the readiness probe must send `LITELLM_MASTER_KEY`; export it or + readiness 401s. The container reads the same var via Compose interpolation. +- F3: vLLM without `--require-generation` is false-ready (alias listed before + vLLM loads; `depends_on` doesn't gate on health). +- F5 (OPEN): display-active GPU 1 is skipped and there's no CLI knob to include + it; `--allowed-gpus` filters after the skip. Blocks 2-distinct-model tests. +- F6 (OPEN): no first-class dtype/protocol/image knobs — dtype via extra_args, + readiness always chat (completions-only models fail), images pinned in config. + +**Reflection:** the offline seam tests (FakeDocker) gave false confidence on the +rendered artifact's *schema* — they checked the dict we built, not whether +docker accepts it. Added the guard: a parametrized test that writes the rendered +project and runs `docker compose config -q` (skipped where docker compose is +absent). It runs against real `docker compose` even in this sandbox (~sub-second +each) and now passes for vllm-single / vllm-tp2 / vllm+ollama / no-litellm — so +F1 is verified at the schema level, not just the dict. + +**Next:** keep fixing what yardrat surfaces (likely F2/F3 ergonomics next, maybe +the F5 skip_display knob so both GPUs are testable), updating the findings log. + +## 2026-06-17 13:05:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. yardrat reported +8 pytest failures — all in the *legacy* `test_cli_meta.py` / `test_cli_setup.py` +(setup/render/status/config-paths), none in the leasing code, and the full suite +passes clean here. Root cause (F7, FIXED): both files' subprocess `run_cli` +helpers anchored the tmp dirs with `env.setdefault('INFER_STACK_CONFIG_DIR'/ +'DATA_DIR', tmp_path)`. The user had `INFER_STACK_DATA_DIR=~/infer-stack-test/data` +exported from the test-plan blocks, so `setdefault` was a no-op and the tests +read the real data dir — which also held a live `infer-stack` compose project +(hence the `docker compose ps` table leaking into the status test). Reproduced +locally by exporting the same var (got the identical failure), fixed by forcing +the vars (`env[...] = tmp_path`), and verified the fix makes all of test_cli_meta ++ test_cli_setup pass *even with the ambient var set* (29 passed). + +**Takeaway:** `os.environ.copy()` + `setdefault` is a latent isolation hole for +any test that anchors via env vars — an ambient export silently wins. Force the +vars. (The in-process `_anchor_paths` used `monkeypatch.setenv`, which already +forces, so those tests were fine — the bug was only in the subprocess helpers.) +This is a test-isolation bug, not a regression from the leasing branch; worth a +dev/lesson if it recurs. + +## 2026-06-17 13:40:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. yardrat re-ran +acquire and hit the *same* capabilities error — but now during `ps`, not `up` +(F8). Root cause: the bad `docker-compose.yml` from run #1 (pre-fix, with +`capabilities: [[gpu]]`) was still on disk, and `reconcile`'s converge branch +calls `observe()` (→ `docker compose ps`, which validates the file) *before* +`converge()` rewrites it. So a stale/invalid file crashed acquire before the fix +could take effect. Fixed by making `observe()` best-effort: catch any +docker/parse error and return an empty set, so converge overwrites the file and +self-heals. Test added with a runner that raises on `ps`. + +**Takeaway / design lesson:** a reconcile step that *reads* actual state must be +tolerant — `docker compose ps` doubly so, because it validates the whole file, +not just lists containers. Read-side operations should degrade to "unknown -> +empty", never abort the write that would fix the very file they choke on. The +generic shape: in a desired-state reconciler, observe() must never be able to +prevent converge(). + +## 2026-06-17 14:30:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. Two more yardrat +findings + an ergonomics fix the user flagged. + +**F9 (FIXED):** vLLM v0.19.1 rejects `--disable-log-requests` (vllm_args emitted +it unconditionally) -> vLLM crashed -> LiteLLM "Connection error". Dropped the +flag (only the leasing + kubeai renderers used it; the legacy compose template +never did; no test asserted it). Engine-version-specific flags belong in +`extra_args`. + +**Ergonomics regression the user caught:** the leasing Compose backend had made +the *user* invent + export `LITELLM_MASTER_KEY` (probe read os.environ; +container got it via `${...}`). The legacy renderer instead *owned* the secret +(ensure_secret -> .env) and you fetched it with `infer-stack env`. Restored: +ComposeBackend.master_key() manages the key in the state dir's `.env` (reuse if +pinned, else generate), bakes the *literal* into the LiteLLM service (so +container + probe agree regardless of shell), uses it for the probe, and +access() returns it; the env-file descriptor now carries `OPENAI_API_KEY` (+ +OPENAI_BASE_URL) so `source is.env` configures an OpenAI client outright; new +`infer-stack secrets [KEY]` prints managed secrets. Baked the literal (not +`${...}`) on purpose: the user had the var exported, and compose lets shell env +override `.env`, which would desync container vs probe. + +**Wording:** user softened "never user-supplied" (a user *can* pin a key in +`.env`; ensure_secret reuses it). Aligned the master_key() docstring + test-plan +F2 to "managed: reused if pinned, else generated". + +208 tests pass; ruff clean. + +**Still TODO this turn:** verify `infer-stack config paths` and `infer-stack +status` are correct/relevant post-refactor (maybe elevate `config paths` -> +top-level `paths`). Doing that next. + +Did it: both worked but were legacy-only. Added a `leasing` group to `config +paths` (ledger, compose dir + docker-compose.yml/litellm_config.yaml/.env/ +sidecar), exposed a top-level `infer-stack paths` alias, and gave legacy +`status` a one-line leasing summary (active leases / live groups -> `infer-stack +leases`). 13 meta tests + the legacy status test pass. + +**Process scar (my mistake):** to check whether some `ruff` errors in cli/ were +pre-existing, I ran `git stash && ruff && git stash pop` in a bash one-liner. A +concurrent `uv run` touched `uv.lock`, the `pop` hit a conflict and aborted +SILENTLY (`>/dev/null`), and the working tree reverted to the last commit — +making it look like all my PART-2 edits had vanished. They were safe in +`stash@{0}`; `git checkout -- uv.lock` then `git stash pop` restored them. +Lesson: never `git stash` uncommitted work in a tree where a build tool mutates +a tracked file (uv.lock) — and never hide `git stash pop` output. To check +pre-existing lint, use `git show HEAD:file | ruff -` or a worktree, not stash. + +## 2026-06-17 15:30:00 -0400 + +Model: claude-opus-4-8[1m] (Opus 4.8, 1M context), Claude Code. User: `infer-stack +logs -f` failed for a leasing user ("No config.yaml found ..."). + +**Root cause:** the day-2 compose wrappers (logs/ps/restart/pull/start/stop) were +bound to the legacy model — `config_for_runtime(config)` (needs config.yaml) + +`_compose_base_cmd(cfg)` (targets generated/docker-compose.yml). The leasing +model has no config.yaml and its compose lives at +`data_root/leasing/compose/docker-compose.yml` (project `infer-stack`). + +**Fix:** added `_day2_compose_base(config, command)` — prefers the leasing +compose project when its file exists (no config.yaml needed), else the legacy +rendered stack (kubeai still stubs). Routed all six generic wrappers through it +(also collapsed their repeated 4-line config+kubeai+base preamble to one line). +Added `LEASING_PROJECT='infer-stack'` constant in leasing/compose.py so the +project name is shared, not duplicated. Smoke: `infer-stack ps`/`logs` now run +against the leasing compose with no config.yaml. Left ollama-* wrappers legacy +(they `exec` a fixed `ollama` service; leasing names daemons per-group — a +separate fix if needed). + +**Precedence decision:** prefer leasing-if-its-file-exists. A pure-legacy user +has no leasing file -> unchanged. A user who once leased keeps the (possibly +empty) leasing compose file, so their day-2 ops point at the leasing project — +acceptable given leasing is the model; documented in the wrapper docstring. + +## 2026-06-17 14:35 -0400 + +Model: claude-opus-4-8 (Claude Code, 1M context). + +User intent: "I thought we had an e2e test in dev — write a set of e2e tests +under `dev/e2e_tests` (a developer test, not a polished unit test) that +exercises the new leasing tools, so I can rsync back a report and you can check +everything is wired up correctly, efficiently, and ergonomically." + +What I found: there was no executable e2e test — only `dev/leasing-test-plan.md`, +a copy-paste runbook. The gap was real. + +What I built: `dev/e2e_tests/` — a self-contained bash harness. +- `lib.sh`: a step/assert DSL (`step` / `run` / `expect_rc|out|re|file*` / + `count_out` / `note` / `skip` / `end_step`). Every step appends one JSON + record (id, title, verdict, duration, per-assertion results, log path) to + `results.jsonl`; the harness never uses `set -e` so one report covers the + whole run. +- `render_report.py` (stdlib only): doubles as the per-step JSON emitter (so the + bash side never hand-rolls JSON) and the final assembler — `report.md` with a + pass/fail/skip summary, a Correctness/Efficiency/Ergonomics "wiring axes" + rollup, failures-first (with log tails), a full per-section table, and the + environment capture. +- `catalog.yaml`: committed (not heredoc'd — that bit us before with editor + merges mangling the runbook's `cat </dev/null 2>&1; true`), so they passed while silently failing +to tear down. + +Fix: converge tears the project `down` when there are no services, else `up`. +`down` targets the project, so it works on the empty file and is idempotent. Unit +test asserts converge([]) issues `down` and never `up`, and observe() is empty. +215 passed (was 214). CHANGELOG updated. + +State of mind: this is the second latent bug the e2e harness surfaced only after +state isolation (first was litellm-reload; both were masked by leftover state). +Reinforces that the reset was the right call — it makes each tier a real +clean-slate test. Confident in the fix; `down`-on-empty is standard compose. +Remaining unrun-on-GPU after the user pulls (speed + this fix): 90_concurrency +(was interrupted), and a full green pass. Group-id compat-key work still queued +behind a green baseline. + +Takeaway: "up the union" converge has an edge at the empty union — the teardown +path is not just "up with fewer services", it's a different verb. Test the +convergence to zero, not just to N-1. + +## 2026-06-17 21:40 -0400 + +Model: claude-opus-4-8 (Claude Code, 1M context). + +User intent: the demo must use the smol models; and the sprawly flat CLI (~38 +top-level verbs) should be reorganized into submodals (aivm-like, "but better"), +with an ergonomic `catalog` editor (no raw YAML), `help tree`, `init`->`config +init`, and a `legacy` holding pen for anything not carried forward. Decisions +(via AskUserQuestion): noun-verb grammar; full phased reorg now. + +Did it in 6 committed phases (each with tests + green suite): +1. `catalog` submodal (commands_catalog.py): model/endpoint/host/bundle + add|list|show|rm + init/path/show/validate/edit. A validating writer + (Catalog.from_dict) refuses to persist a catalog the leasing path would + reject. `help tree` walks the modal registry. (Verified scriptconfig 0.9.1 + does 3-level nesting first.) +2. `config` submodal + settings.yaml in paths.py (load/save/get_setting, no + import cycle). data_root() honors `data_dir`; leasing `--backend` default is + now None and resolves explicit > `config set backend` > null. Kills two + ergonomic smells. +3. `secret` get/set/list; `secret set HF_TOKEN=…` writes the managed .env + (merge-preserving), set before serve. `secrets` kept as alias. +4. `stack` day-2 group (+ new `stack down`); logs/ps kept top-level. +5. `legacy` modal — moved ~25 profile-world verbs under it. Top level is now the + leasing loop + submodals. Kept tests working by auto-prefixing `legacy ` in + the two subprocess run_cli helpers (one edit each, not per call-site). +6. Docs: rewrote leasing-demo.md to use catalog/config/secret (no heredoc, no + per-block exports), marked the resolved smells; cli-redesign.md -> + implemented; followups + CHANGELOG; README banner pointing at leasing + + `legacy` prefix (full README rewrite deferred). + +State of mind: confident — every phase kept the suite green (215 -> 230) and the +new code is ruff-clean (the only ruff hits are pre-existing I001 import-order in +paths.py/options.py that predate me). The catalog validating-writer is the piece +I'm happiest with: the editor literally cannot write a catalog acquire would +choke on. Risk/uncertainty: the GPU e2e hasn't run against the reorged CLI yet — +but the e2e/demo only use top-level verbs that stayed (acquire/serve/run/ +release/leases/secrets/status/logs/ps/paths) so it should be unaffected; worth a +yardrat pass to confirm. The README is now accurate-by-banner but not rewritten. + +Takeaway: when reorganizing a CLI built on a modal framework, a registry-walking +`help tree` + a `legacy` bucket let you move fast without a flag-day — and +honoring persisted settings (backend/data_dir) in the *resolution* layer (None +default -> setting -> fallback) avoids the explicit-vs-default ambiguity cleanly. + +## 2026-06-18 14:53:48 -0400 + +Model: claude-opus-4-8 (Claude Code, "fast"/Opus 4.8). Config: default tools, +running from the aiq-eval-runner superrepo against the infer-stack submodule on +branch dev/leasing-controller. + +User intent (one prompt, four threads): (1) Open WebUI should be on by default +again — bundled, and crucially NOT torn down when models switch (the legacy +stack worked to keep the UI from blinking); user judged this easier here than +the LiteLLM case. (2) Rename `infer-stack secrets` -> `infer-stack env`, +path-first with optional KEY — secrets live in a readable `.env` so no reason to +hide the path; mirror the legacy `env` ergonomic. (3) Bring back a concise smoke +test so the demo doesn't have to curl (but keep curl shown too). (4) Diagnose a +`litellm-1 InternalServerError: Connection error. Received Model Group=chat` +seen during the demo. + +Diagnosis of (4): startup noise, not a real failure. Compose `depends_on` for +litellm waits for the upstream to *start*, not be *healthy*, so during vLLM's +model-load window litellm forwards probe/early requests to a not-yet-listening +`vllm-…:8000` and logs the connection error; once vLLM is up the same route +200s (matches the user's trailing log lines). I deliberately did NOT switch +litellm to `condition: service_healthy` — that would couple every litellm +recreate (which happens on each routing change) to the slowest upstream's +health, delaying routing for already-up models. Instead I added +`router_settings` (num_retries/timeout/cooldown_time/allowed_fails) so the +warmup window is retried/self-healing rather than surfacing as client 500s. + +Design decisions worth recording: +- Open WebUI stability across switches falls out of making its compose service + spec *independent of the model set*: it points at the `litellm` service at a + fixed internal URL with the (stable) baked master key, so the rendered dict is + byte-identical every converge. `docker compose up -d` then leaves it running + while only litellm (config-hash label) is recreated. Verified by a test that + asserts `open-webui` is unchanged but `litellm` differs after adding a second + model. This is the same lever the config-hash fix used, applied in reverse: + put churn in the spec only where you *want* recreation. +- `ui` resolution mirrors the backend/data_dir pattern: tri-state flag + (`--ui/--no-ui` default None) -> `config set ui` -> default True, resolved in + `_resolve_ui`. Keeps explicit-vs-default unambiguous. +- `infer-stack test` reads the front door straight from DEFAULT_PORTS + the + managed `.env` rather than building a ComposeBackend, so it's cheap and needs + no GPU detection. +- Name collision caught: `EnvCLI` exists in both commands_runtime (legacy) and + now commands_leasing; imported the leasing one `as LeasingEnvCLI` so the + legacy modal's `env` keeps resolving to the runtime class. + +Risks/uncertainties: WEBUI_AUTH=False is fine for a single-user workstation but +must gain an auth/port knob before a shared host (noted in followups). The +managed UI binds host port 13000 by default now on every compose serve — a +behavior change a user could be surprised by, mitigated by --no-ui/config. +Tests pass (240, +8) and ruff is clean on touched files; the GPU e2e hasn't been +re-run against this on yardrat yet — the open-webui service is new on the serving +path and only validated via the fake-docker render/converge tests. + +Takeaway: to make one service in a converged compose project immune to churn +while another recreates on change, encode "what may change" exclusively in the +spec (labels/env) of the service you *want* recreated, and keep the stable +service's spec a pure function of inputs that don't change — recreation is then +a derived property, not a special case. + +Follow-up same session: user asked whether `secret` is needed at all given +`env`. It wasn't — `env` already covered get/path/list; `secret` only added +`set`. Folded `set` into `env` as a `KEY=VALUE` positional (`env KEY` reads, +`env KEY=VALUE` writes, merging non-destructively), removed the `secret` modal +and its three classes entirely. One verb, unambiguous by the presence of `=`. +Reinforces the rename's premise: secrets in a readable `.env` don't warrant a +separate "secret" surface. Updated demo/CHANGELOG/followups and tests (240 pass, +ruff clean). + +## 2026-06-18 17:31:31 -0400 + +Model: claude-opus-4-8 (Claude Code, Opus 4.8). Config: default tools, working +the infer-stack submodule from the aiq-eval-runner superrepo. + +User intent (one bundled prompt, five threads): (1) keep leasing-demo current; +(2) rich-format `infer-stack leases` (and "other CLIs"); (3) `release --all` to +make teardown concise; (4) restore the lost "show me the compose diff before you +change it, --yes to skip" approval; (5) add loguru narration (aivm-style) so the +behind-the-scenes process is legible. + +Design decisions worth recording: +- **loguru off the hot path.** loguru imports at ~50ms — we'd spent real effort + getting `infer-stack --help` to ~0.2s, so adding it at module scope would undo + that. Solution: a private `_log.py` that is imported only at the leasing + *runtime* chokepoint (`_open_controller`) and inside converge/reconcile, never + by the help/catalog/config surface. It also `logger.disable('infer_stack')` on + import, so library and test use is silent until the CLI calls + `configure_logging()`. Narration goes to **stderr** so stdout (JSON, + `$(infer-stack env KEY)`) stays clean. Verified loguru count == 0 in the + `--help` importtime trace. +- **Diff-approval lives in the backend's converge, but the policy lives in the + CLI.** ComposeBackend gained `assume_yes`; converge renders, computes the + changed files vs disk, and (when not assume_yes) calls the existing + `diff_prompt.confirm_writes`. The *decision* of when to prompt is the CLI's: + only the additive verbs (acquire/serve) prompt, and only on a TTY without + `--yes` (`_resolve_assume_yes(interactive=...)`). release/leases/run never + prompt — for a teardown, the action is the approval. This keeps `release --all` + and the `run` pipeline-node usable non-interactively without hanging. +- **Decline must not leak ledger state.** acquire = ledger.acquire (creates the + lease) → reconcile (converge). If the operator declines, converge raises a + neutral `ConvergeAborted` (defined in backend.py, not compose.py, to respect + the layer boundary), and controller.acquire rolls the lease back with a + *bookkeeping-only* `ledger.release` (not controller.release, which would + reconcile again). Tested with a tiny DeclineBackend rather than the real + renderer so the test doesn't depend on a fully-populated spec. +- **release --all** enumerates active leases from the ledger and releases each; + the reconciles collapse naturally (each release converges the shrinking union; + the last one downs the project). + +Risks/uncertainties: the diff prompt fires inside converge, which is reached +through the controller — fine for the single-shot CLI, but any future caller +that converges in a loop on a TTY would prompt repeatedly (mitigated: only +acquire/serve set interactive). loguru is a new hard dependency (added to +pyproject + uv.lock; clean 2-package diff). The compose diff/approval path is +unit-tested with a fake docker + monkeypatched confirm; the real interactive +prompt on GPU hardware hasn't been exercised yet. 250 tests pass, ruff clean. + +Takeaway: when adding cross-cutting feedback (logging) or a new dependency to a +CLI you've tuned for startup latency, gate it behind the runtime entry points +rather than module scope — "imported only when actually doing work" preserves +both the fast `--help`/completion path and library silence, and costs only a few +lazy imports. + +## 2026-06-18 20:46:06 -0400 + +Model: claude-opus-4-8 (Claude Code, Opus 4.8). Config: default tools, working +the infer-stack submodule from the aiq-eval-runner superrepo, branch +`dev/leasing-controller`. + +User intent (one exploratory session, several threads, mostly driven by running +the demo on yardrat): (1) add two Qwen models to the demo (cross-family); (2) +"why didn't `serve qwen05-1` prompt / why is it `live` when nothing's running?"; +(3) "show what's *actually* running + which GPUs"; (4) "separate render from +execution — set up on-disk state and see what would run before pulling the +trigger"; (5) expand `--help`; then a verb-shape brainstorm landing on +declares-refcount / reconciles-idempotent; (6) "do we ALWAYS preview before +modifying the compose file?" → gate every converge. Committed in two batches: +`fe90d2e` (1–5) and a follow-up (6). + +Design decisions worth recording: +- **Fail fast on unplaceable requests.** The yardrat hang was: acquire marks the + group LIVE in the ledger *before* reconcile, placement silently dropped it (no + free GPU → only a WARNING), and `serve` then blocked on readiness for a + container that never rendered. Fix: the backend records `last_unplaced`; + controller.acquire rolls the lease back and raises `PlacementError` if a + *just-requested* group landed unplaced. This also dissolved the "phantom live" + display — the bad lease never persists. Key realization: `live` is *desired* + (ledger), never *observed*; the two were conflated in `leases`. +- **leases shows desired vs actual.** Added `running` (backend.observe) and + `gpus` (`plan()`, read-only placement; `→N` = slated) columns. Backed by a new + `ComposeBackend.plan()` that the converge path now reuses. +- **Legible names.** vLLM service/container is `vllm--` + (was `vllm-`). The group-id suffix is load-bearing: two desired + groups can share a served name (an endpoint re-pointed at a new model), and it + keeps `docker ps` ↔ `leases` correlatable. Ollama keeps its id-name (one + daemon, many models). One-time container recreate on upgrade (service key + changed) — which is exactly what bit the user later (see below). +- **Render/apply as the separating seam.** `converge(desired, apply=False)` + renders the on-disk project without `docker compose up`. Surfaced as a flag on + the declare verbs (`serve|acquire --no-apply`, stage) plus lease-free `render` + / `apply` reconcile verbs. The crux we reasoned through with the user: `apply` + writes *no* infer-stack tracking state — all of it (ledger intent + sidecar + placement) is written at declare/stage time — so "drop the tool and `docker + compose up` yourself" is a true equivalent. Cemented that by baking a top-level + `name: infer-stack` into the rendered compose, so a bare `docker compose -f + up` lands in the same project (was the one real gap — default project + name would've been the dir name). +- **Verb shape: declares refcount, reconciles are idempotent.** The user found + that `serve --no-apply` then `serve` makes *two* leases (demand 2). We + explicitly rejected idempotent serve/acquire (serve *is* `acquire --ttl inf`; + a refcount that sometimes doesn't count is the confusing thing). The real fix + is structural: a lease-free `apply` means "apply a staged lease" no longer goes + through a declare verb, so the refcount only climbs when you genuinely declare. + Idempotency belongs on render/apply (pure functions of desired state), not on + the refcount verbs. +- **Gate every converge (supersedes the 2026-06-18 17:31 decision).** That entry + said teardown verbs don't prompt ("the action is the approval"). The user's + GPU run disproved it: `release --all` silently rewrote the compose file *and + recreated a keep-warm model* (the one-time rename migration happened to land on + a release). So now `release`/`evict`/`apply` also build `interactive=True` and + gate the diff (`--yes` skips). To keep it to one prompt, `release`/`evict` now + batch all ledger mutations then converge once (was: per-lease reconcile). New + asymmetry: declining an *acquire* rolls back; declining a *release/evict* + records the ledger change but leaves docker untouched (`apply` later) — which + is the honest plan/apply semantics, not a bug. + +Risks/uncertainties: none of this has run against real docker/GPUs since the +changes (only fake-docker units) — the new gating prompt, the one-time container +recreate, render/apply, and the `name:` field are unexercised on hardware. The +batched release changes decline semantics (ledger released even if docker +declined); defensible but a behavior change. The container rename will recreate +every running vLLM container once on upgrade (acceptable, noted in CHANGELOG). +267→ tests pass, ruff clean on changed files (pre-existing I001s elsewhere). + +Takeaways: (1) separate *desired* (ledger) from *observed* (backend) in both the +data model and the UI — conflating them produced both the "phantom live" bug and +the silent-placement hang; surfacing `running` vs `state` made the system +honest. (2) When a tool drives another tool, make the generated artifact +self-sufficient (`name:` in the compose file) so "bypass the tool" is a +first-class equivalent, not a divergent sibling — that property is also what let +us reason that `apply` hides no state. (3) Idempotency is a property of +reconcile operations, not of intent-declaring ones; put it where it's natural +rather than special-casing a refcount. + +## 2026-06-19 00:09:12 -0400 + +Model: claude-opus-4-8 (Claude Code, "fast" Opus). Continuation of the leasing +TUI work. + +User intent (compressed): make the TUI usable by a newcomer and stop it feeling +janky. Concretely: (1) an orange / white / very-dark-gray theme; (2) "feel +faster or more responsive"; (3) a real bug — `docker compose` progress was +overwriting the top-left of the screen for a moment when serving; (4) turn the +logs pane into a docker *tab* (logs **or** ps); (5) require `config init` before +launch, and if there are no models offer a button that runs the new `suggest` +tool to populate the catalog; (6) wizards to add models/endpoints; (7) more +explanatory text so new users can intuit what's going on; (8) drag-to-resize +panes. + +What I did (all in `infer_stack/tui.py`, `tests/test_tui.py`, plus a small +`TuiCLI` change in `cli/commands_leasing.py`): + +- Theme: registered a custom `textual.theme.Theme` (`infer-orange`) — dark-gray + canvas, white text, one warm orange accent that only shows on the focused + pane's border. Replaces the borrowed `nord`. +- Bleed bug: root cause was `_default_docker_run` = `check_output` (captures + stdout, lets **stderr** through), and `docker compose up -d`/`down` write all + progress to stderr → straight onto the full-screen terminal. Fix lives at the + TUI layer (didn't touch the CLI's runner, where stderr-to-terminal is + desirable): on mount I wrap `backend.run` with a `capture_output=True` runner + that routes the noisy verbs' stderr into the logs pane and swallows `ps` + polling. Felt cleaner than changing the shared default. +- Responsiveness: the periodic refresh used to call `backend.observe()` + (`docker compose ps`) **on the UI thread** every 3s — a real freeze. Split + into `_collect()` (thread-safe, no widgets) + `_render()` (UI thread); + `on_mount` does one synchronous `_refresh_now()` for an instant first paint + (and so the headless tests still see data after a single `pause()`), while the + interval + `r` + post-mutation refresh go through a `@work(thread=True)` + worker. The bleed fix doubles as responsiveness (no terminal corruption). +- Docker tab: logs pane is now a `TabbedContent` — "Logs" (the existing Select + + RichLog, IDs preserved) and "Status · ps" (a DataTable fed by a best-effort + compose-ps parse in the collect worker). +- Onboarding: `TuiCLI` now hard-requires `settings.yaml` (i.e. `config init`) + and *tolerates a missing/empty catalog* via a new `_load_catalog_for_tui` + (returns an empty Catalog + the path it would write). Empty state shows a + help line steering to Suggest/add. Suggest (`g`/button) calls the other + agent's now-committed `leasing.suggest.suggest_catalog` + `commands_catalog` + load/save helpers (imported, never edited), merges, reloads in place. +- Wizards: `_AddModelScreen` / `_AddEndpointScreen` ModalScreens with Inputs; + results written through `commands_catalog._load_raw/_save_raw` (which validate) + and the catalog reloaded. `m` / `n` or sidebar buttons. +- Drag-resize: a tiny `_Divider(Static)` that captures the mouse and reports + axis-delta to a callback (`_drag_sidebar` / `_drag_logs`), alongside the + existing `[` `]` / `-` `+` keys. Vertical bar between sidebar/main, horizontal + bar between tables/docker. + +State of mind / risks: as always, none of this has been *eyeballed on a real +terminal* — headless pilot proves it composes, tabs, resizes, serves, streams, +and writes the catalog, but the orange theme's actual feel, the drag ergonomics, +and (critically) whether the stderr-capture fully kills the corner-bleed are +only verifiable by running `infer-stack tui` on a GPU host (yardrat). The +suggest path in particular only runs under real hardware detection. Confident +about: the threading split (clear win, observe was blocking), the +backend.run-wrapping approach (localized, reversible), and that I stayed off the +other agent's files (their suggest CLI is committed at 4ef5147; I only import). +8 TUI tests + full prior suite (303) green; ruff clean on changed files. + +Takeaways: (1) when a full-screen UI drives a subprocess, capture *both* streams +at the UI boundary — stderr is the usual screen-corrupter, and the fix belongs +at the UI layer, not in the shared runner where stderr is wanted. (2) Keep a +synchronous first paint even after moving refresh to a worker: it makes the app +feel instant *and* keeps headless tests deterministic without `wait_for_complete` +gymnastics. (3) For a tool newcomers land in, downgrade "missing config" from an +error to an empty-state-with-a-button — the absent catalog is the start of the +funnel, not a failure. + +## 2026-06-19 00:40:00 -0400 + +Model: claude-opus-4-8 (Claude Code, fast Opus). Same TUI session, two follow-up +fixes from the user: (1) drag-to-resize still didn't work; (2) the global bottom +menu listed pane-specific actions (serve/release/evict) that don't apply +everywhere — wanted them scoped to their panes, keeping only truly-global items +global. + +Drag root cause (found by a headless harness, not guessing): the `_Divider` +`Static` had no explicit cross-axis size, so `Region` was `width=1,height=1` — a +1×1 dot. The drag *logic* was fine (posting a captured `MouseMove(delta_x=5)` +moved the width 38→43 in the harness), there was simply nothing to grab. Fix: +`#vsplit { height: 1fr }` and `#hsplit { width: 1fr }` so each bar spans its +cross axis (verified: vsplit now 1×36, hsplit 79×1). Lesson worth keeping: when +a Textual mouse interaction "doesn't fire," check the widget's `region` first — +an auto-sized helper widget can collapse to an un-hittable cell even though its +handlers are correct. + +Scoped actions: moved s/d/e/a/g/m/n bindings to `show=False` (keys still work) +and added per-pane Buttons — Serve under `#endpoints`, Release/Release-all under +`#leases`, Evict under `#groups`, plus the existing catalog buttons. Footer now +shows only Refresh / Next-pane / Quit. `on_button_pressed` became an id→action +map. Intro line updated to point at the per-pane buttons + drag bars. + +Tests: added grab-area, real-drag (mouse_down + posted MouseMove + mouse_up → +width changes), and scoped-button-serve cases. 11 TUI tests green, ruff clean. +Confident in the drag fix (measured). Still unverified on a real terminal: the +actual feel of grabbing a 1-cell bar in tmux (motion reporting under tmux can be +finicky) — if dragging is still flaky on yardrat, the next move is widening the +hit target or enabling a drag affordance glyph. + +## 2026-06-19 01:25:00 -0400 + +Model: claude-opus-4-8 (Claude Code, fast Opus). Third TUI pass. User asks: +(1) system info à la nvidia-smi/btop; (2) a small API-test tab (send a prompt to +a model, or test all models); (3) ctrl+click a served endpoint to open it in +Open WebUI; (4) click-to-collapse panes so hidden data isn't polled; and a +follow-up: (5) the `ps` view should carry uptime/status, created time, and +container id like `docker ps`. + +Approach: turned the bottom pane into a `Collapsible` wrapping a `TabbedContent` +with four tabs — Logs, Status·ps, System, API. The collapse is the user's +"click to collapse," and it does double duty as the polling gate: `_collect()` +now only fetches the *active* tab's expensive data (`docker compose ps` for the +ps tab, `nvidia-smi` for System) and nothing when collapsed. `_active_tab` is +tracked via `TabbedContent.TabActivated` and `_console_collapsed` via +`Collapsible.Toggled`; both are plain attrs read by the worker thread, set on the +UI thread — cheap and race-free enough for a monitor. Switching tabs fires a +refresh so the newly-shown tab fills immediately. + +System tab: `nvidia-smi --query-gpu=… --format=csv,noheader,nounits` parsed into +a table; returns `None` when nvidia-smi is absent so the UI shows a clear "not +found" row instead of looking broken (this dev box has no GPU — only yardrat +will populate it). Host line from `/proc/loadavg` + `/proc/meminfo` + cpu_count +(no psutil dep, which isn't installed). API tab: a model Select (from catalog +endpoints), a prompt Input, Send / Test-all buttons, and a RichLog; calls go to +`http://localhost:{litellm_port}/v1/chat/completions` with the backend master +key. Made the HTTP client injectable (`http=`) so a fake drives it headless — +otherwise this tab would be untestable without a live gateway. ps columns now: +service · status(uptime) · created · container-id · ports, pulled from the +compose-ps JSON (`Status`, `CreatedAt`/`RunningFor`, `ID[:12]`). + +Open-in-browser: `action_open` (key `o`) builds `…:{ui_port}/?models={endpoint}` +and `webbrowser.open`s it; ctrl+click routes through `on_click` → +`screen.get_widget_at` → walk to `#endpoints`. Over SSH/tmux `webbrowser.open` +will no-op, so the status line always prints the URL to copy. + +Risks/uncertainties: the big one remains *unverified on real hardware* — the +Open WebUI `?models=` deep-link param depends on the running Open WebUI version +(it may just land on the home page; acceptable fallback). nvidia-smi parsing +assumes the standard CSV columns. Collapsible + TabbedContent + my fixed +`#docker` height composes cleanly in pilot, but I haven't watched it animate on +a terminal. ctrl+click hit-testing via `get_widget_at` is the part I'm least +sure survives tmux mouse quirks; the `o` key is the reliable fallback and shares +all the logic. 16 TUI tests (added: ps-parse, system-no-smi, collapse-gating, +open-url, injected-http API send), full suite 316 green, ruff clean. + +Takeaways: (1) make the collapse state and active-tab the *same* signal that +gates polling — "don't poll what you can't see" falls out for free instead of +being a second mechanism. (2) Any pane that hits the network or a subprocess +needs an injection seam (proc_factory, http) or it's simply not testable +headless; build it in from the first line, not after. + +## 2026-06-19 02:40:00 -0400 + +Model: claude-opus-4-8 (Claude Code, fast Opus). Fourth TUI pass + a leasing-core +addition. User asks: (1) API tab should list only up-and-ready models; +(2) a way to clean up old released/evicted entries; (3) drop the global intro, +make descriptions pane-local; (4) rename console→docker, the ps tab→"containers"; +(5) promote System and API to their own panes; and a late one: (6) default theme +to stock "textual dark". + +Core add: `SqliteStore.prune(lease_states, group_states)` + `Ledger.prune()` +(deletes RELEASED/EXPIRED leases and STOPPED groups, claims first to dodge the +groups FK). This is the first ledger *deletion* path — sweep/evict only ever +transitioned state, leaving a growing tail. Kept it explicit (prune, not +auto-gc) so history stays inspectable until you choose to forget it; the TUI's +Clean-up button is the first caller. + +"Ready" models: I define ready = endpoints served by a group that is *observed +running* (g.id in the observe() set), computed in `_render` and pushed to the API +selector. Cheap (reuses the placement view the groups table already needs) and +matches the "running" column the user sees. Not a true vLLM readiness probe — +documented as such; a container can be Up but still loading. Good enough for "let +me poke a model that's actually there," and avoids per-refresh HTTP. + +Layout: System and API became their own `Collapsible` panes, collapsed by +default — which doubles as the polling gate (`_collapsed` dict + `_active_tab`): +nvidia-smi only runs when System is expanded, `docker ps` only when docker's +Containers tab is visible. leases/groups got wrapped in bordered Verticals so +each can hold a description + table + its own buttons. + +Sharp edge found + fixed: in Textual 8.2.7 the Select no-selection sentinel is +`Select.NULL`, and `Select.BLANK` is literally `False` — so my `value is +Select.BLANK` checks were always False (a latent bug, dormant only because the +other Selects always carry a real value). Swapped all four to `Select.NULL`. +Lesson candidate: don't assume a framework sentinel's name across versions — +verify with a one-liner. 18 TUI tests (added: docker/containers tabs, ready-only +API list, collapse-gates-system, cleanup-prunes-released), full suite 318 green, +ruff clean. + +Uncertainties: same standing one — unverified on real hardware (System/API/ps +need a live GPU + gateway). The ready-model definition may surprise someone who +expects a just-served-but-still-loading model to be immediately queryable; the +status line + "only ready models listed" desc set that expectation. + +## 2026-06-19 12:30:00 -0400 + +Model: claude-opus-4-8 (Claude Code, fast Opus). Large multi-request turn; Jon +explicitly OK'd staging. Locked four design decisions via a question prompt +(saved to agent memory): group→deployment, session_id→lease_id (HARD), TUI +top-level tabs (Dashboard/Settings/Catalog) + command palette, keep both +monitors; and the standing rule that the CLI must be a superset of TUI features. + +Shipped two commits: +1. (08353d3) Made the lease↔deployment many-to-one legible — the thing Jon most + wanted to "click." Leases pane gained a `deployment` column (the literal group + id, matching the deployments pane's id column, so the join is eyeball-able); + deployments pane shows `leases` (count) + `held by` (owners) instead of the + opaque `demand`. Cursor movement narrates the link in the status bar. Buttons + compacted to 1 row. +2. (0ab64db) session_id→lease_id hard rename: env key INFER_STACK_LEASE_ID, + --lease flag, JSON lease_id, read_lease_id(), id prefix lease-. Verified no + parent-repo consumers before breaking it. 319 tests green. + +Deliberately deferred the group→deployment rename: 558 code sites + a SQLite +`groups` table is too big to do safely alongside features. It's its own next +pass. Open sub-decision I'll default on unless redirected: keep the SQLite table +named `groups` (purely internal, avoids a ledger-DB migration) while renaming all +Python symbols + user-facing strings to deployment — i.e. rename the concept +everywhere a human sees it, leave one hidden DB identifier alone. + +State of mind: good progress, low regression risk so far (rename was contained, +full suite green twice). The remaining work is mostly additive TUI (tabs, +settings, catalog mgmt, edit/remove, docker control, model cache info, all-panes +collapsible + drag bars everywhere) plus the big mechanical group rename. Biggest +judgment call ahead is the reorg: top-level tabs change the whole compose() tree, +so I want to land the group rename first (so new UI is built on final vocabulary) +unless Jon prefers to see the reorg sooner. + +## 2026-06-19 12:50:00 -0400 + +Model: claude-opus-4-8 (Claude Code, fast Opus). Jon's directive: clean break, +no back-compat (pre-release, no users), delete-and-recreate the ledger DB rather +than migrate, and rip out superseded legacy later — but PRESERVE the ollama + +kubeai conceptual paths and any reusable logic. Names must agree everywhere; flag +any place a presentation name should legitimately differ from a backend name (I +found none in the leasing surface). + +Did the group→deployment deep rename. Approach that kept it safe despite 558 +sites: first proved the concept is isolated — `CacheGroupSpec`/`cache_groups` +(KV-cache, experimental/), `_is_group` (CLI command-group, commands_meta), and +regex `.group()` all live OUTSIDE the 12 leasing-concept files — so I scoped an +ordered sed (DeploymentGroup→Deployment, GroupState→DeploymentState, +group_id(s)→deployment_id(s), groups→deployments, group→deployment, +Group→Deployment) to exactly those 12 src + 6 test files, plus the SQLite table +`groups`→`deployments`. Two traps caught by the test suite: (1) the all-caps +`GROUP_LABEL` constant (my rules were case-specific) and (2) the rename clobbered +Textual's `@work(..., group='logs')` worker-group kwarg → `deployment='logs'`, +which `work()` rejects — reverted those 10 decorators. Also `ruff --fix infer_stack/ +tests/` over-reached and autofixed ~13 unrelated files' pre-existing lint; I +`git checkout`'d everything outside scope to keep the diff focused. test_cli_leasing ++ docs + CHANGELOG updated to the new vocabulary (protecting the LiteLLM "Model +Group=" error quote and the "grouped into a holding pen" verb). 319 tests green. + +State of mind: confident — the rename is mechanical and twice-verified green, the +DB table rename is fine since we're nuking state. Deliberately did NOT keep the +SQLite table name (Jon: no migration, delete the DB). Remaining: legacy-code +removal (profiles/old hookups, keeping ollama+kubeai concepts), then the TUI +top-level-tabs reorg + catalog mgmt/edit-remove + advanced endpoint params + +docker control + model cache info + all-panes-collapsible/drag. Those build on +the now-final vocabulary, so no rework. + +## 2026-06-19 13:30:00 -0400 + +Model: claude-opus-4-8 (Claude Code, fast Opus). Legacy removal, Phase 1, with +Jon's exception: keep + improve StatusCLI into a holistic `infer-stack status`. + +Mapped the import graph first (the only safe way to delete in a web of shared +modules). Found the pre-leasing profile world is gated behind an explicit +`legacy` command group whose own docstring said "removed wholesale" — so that +group + its exclusive modules were the target. Deleted (~5k LOC, no back-compat +per Jon — pre-release, delete-and-recreate state): the `legacy` group; +cli/commands_profile, cli/commands_smoke; renderer, benchmark, verification, +contracts; the active-profile runtime verbs (Up/Down/Purge/Deploy/Env/Ollama*) ++ the _ensure_rendered/RenderCLI hook inside commands_runtime; and 4 legacy test +files (serving_profiles, ollama_stack_graph, reverse_proxy_ldap, cli_setup) + +2 legacy tests in test_cli_meta. Suite 319→222 green, and 110s→15s — the deleted +legacy tests were the slow ones. + +Rewrote commands_runtime.py clean: the `stack` day-2 wrappers now target ONLY +the leasing compose project (dropped the legacy config.yaml fallback + +config_for_runtime/_compose_base_cmd/_kubeai_stub), and a new holistic StatusCLI +reports backend, data/config dirs, catalog (+counts), settings, ledger, compose +locations, a leasing summary (active leases / live deployments, read-only — no +sweep so status never mutates), and "dig deeper" pointers. Kept the ollama + +kubeai *concepts* (kubeai_ops.py, backends/kubeai_renderer.py, catalog +engine:ollama) per Jon, even though currently unwired. + +Deliberately deferred (Phase 2): resolver/validator/old top-level catalog.py + +the legacy plan helpers in cli/context.py (build_plan/resolve/validate/ +load_config/config_for_runtime) and cli/compose.py. These are load-bearing for +context.py (which kept code imports for _apply_path_overrides/effective_inventory), +so they need per-function surgery, not deletion — its own careful pass. Also +Phase 2: ConfigPathsCLI still lists legacy artifacts (config.yaml/models.yaml/ +kubeai_generated_dir); trim once config.py's legacy paths go. + +Risk/uncertainty: the new StatusCLI is unverified against a real populated stack +(only the leases-present unit test). The deferred cluster is where the remaining +breakage risk lives. Confident the deletion is clean (graph-driven, suite green). + +## 2026-06-19 14:30:00 -0400 + +Model: claude-opus-4-8 (Claude Code, fast Opus). Legacy sweep Phase 2. + +2a: deleted resolver.py (1053) + validator.py (314) — imported only by +cli/context — after carving context down to the two helpers the leasing surface +actually uses (_apply_path_overrides, effective_inventory). Also deleted the +cli/compose + cli/probes shims (only the deleted commands used them) and trimmed +cli/__init__'s context/compose/probes re-exports. Rewrote ConfigPathsCLI to +report current locations only (config_root, settings.yaml, catalog.yaml, +data_root, leasing ledger/compose) instead of the old config.yaml/models.yaml/ +generated_dir/state groups; updated its tests. ~2700 LOC. + +2b: deleted backends/compose_renderer.py (superseded by leasing/compose) and +slimmed backends/__init__ to just the kubeai renderer (kept as the kubeai +concept per Jon). The whole backends package is now unimported except itself — +kubeai_renderer is deliberately retained for when that backend lands. + +Deliberately deferred (internal-only, no CLI surface): the old top-level +catalog.py and the now-dead helper chain inside config.py (builtin_*_catalog, +the catalog loader, initial_config/deep_merge/load_yaml/etc. — all 0 external +callers). config.py stays for normalized_output/normalized_state/ +default_state_paths + DEFAULT_PORTS/PINNED_IMAGES/KUBEAI_GENERATED_SUBDIR; the +dead chain is interwoven with initial_config and would need careful +function-boundary excision. Judgment call: at ~7800 LOC removed across Phase 1+2 +with the suite green throughout, the risk of nicking a kept config function for +a maintainer-internal tidy wasn't worth doing at the tail of a long session. +The "two catalogs" (top-level catalog.py vs leasing/catalog.py) confusion is the +one remaining legacy-naming smell; flagged for a focused follow-up. + +Confident: every deletion was import-graph-driven and verified green. Next up is +the TUI top-level-tabs reorg + catalog management (edit/remove, advanced +endpoint params) + docker control + model cache, all on the clean base. + +## 2026-06-19 16:00:00 -0400 + +Model: claude-opus-4-8 (Claude Code, fast Opus). TUI feature sweep on the clean, +post-legacy, final-vocabulary base. + +- Advanced endpoint wizard: exposed tensor-parallel / max-model-len / gpu-mem / + extra-args / reclaim (mirrors `catalog endpoint add`), plus Edit (blocked while + served) and Remove for endpoints + models with a confirm dialog. The + validating writer refuses removing a model still referenced by an endpoint. +- Top-level tabs: wrapped the multipane in a TabbedContent → Dashboard + + Settings. Settings reads/writes settings.yaml (backend, data dir, Open WebUI, + reverse proxy, skip-display-GPUs). Key trick: extracted the dashboard body into + `_compose_dashboard()` and `yield from` it, so the big block kept its exact + indentation (no risky re-indent) — verified by the 26 existing tests staying + green after the wrap. +- docker Control tab (compose up/down + compose-file path) driving the captured + runner so output lands in Logs. Models table gained quant + a cheap 'cached' + flag (existence check vs state.hf_cache/hub — no du). +- Drag bars: added the two splitters Jon explicitly called out — endpoints|models + (#csplit) and leases|deployments (#lsplit) — reusing the fixed-height-neighbor + + 1fr-neighbor pattern (a divider that drags one pane's height while the other + flexes). All four splitters now grabbable. + +Remaining: (1) "every pane collapsible" — docker/system/api collapse; +leases/deployments/endpoints/models are now drag-resizable instead (collapse + +fixed-height-divider fight each other; drag covers the space-management need). +Full per-pane collapse is a further polish. (2) The internal config.py dead-chain +gut + old top-level catalog.py deletion (no CLI surface) — still deferred. + +229 tests green throughout; each TUI feature landed as its own committed, +test-backed increment. Confidence high on the mechanics; the on-terminal feel +(tab switching, drag ergonomics under tmux) remains the standing +unverified-without-a-real-terminal caveat. + +## 2026-06-19 18:30:00 -0400 + +Model: claude-opus-4-8 (Claude Code, fast Opus). Live-testing feedback round + +the two deferred remainders. Jon is eyeballing each change. + +TUI fixes from live testing: +- Endpoint wizard was an unlabeled blank form in edit mode (prefilled inputs hide + placeholders). Rebuilt it: a Label per field, and engine-adaptive groups — vLLM + shows tensor-parallel / data-parallel / max-model-len / gpu-mem / max-seqs / + prefix-caching / extra-args; Ollama shows host + free-form KEY=VALUE runtime. + Mapped to the canonical leasing runtime keys (verified against + leasing/compose._service_dict), so no CLI change needed — data-parallel etc. + were always reachable via runtime keys / --runtime / --extra-args; the wizard + just surfaces them. +- API tester promoted to a top-level tab (frees the monitor column). Catalog + Add/Edit/Remove localized to the endpoints + models panels; the bottom button + stack removed; Suggest moved to Settings. +- Fixed the endpoints|models drag inversion (fixed-pane-below ⇒ subtract delta). +- Vertical splitter now drags the full width range (clamped to terminal width), + not just the middle (min-width was 26; lowered to 10, clamp uses self.size). + +Per-pane collapse: made leases + deployments Collapsibles (docker/system already +were). Discovered + documented a real Textual constraint via a scratch: +setting an explicit height defeats Collapsible's collapse (height wins), and +clearing to None didn't restore shrink — so collapse and a fixed-height drag bar +can't share a pane. Chose collapse for the four monitor panes (dropped the +leases|deployments lsplit) and kept drag for the catalog (endpoints|models +csplit), since you act on the catalog constantly and want it sized, not hidden. + +config.py gut + old catalog.py deletion (the last "two catalogs" smell): traced +the call graph — the builtin_*_catalog → file/overlay/merge → normalized_catalogs +chain + initial_config were all dead externally (only normalized_output/ +normalized_state/default_state_paths + the port/image constants are used, by +leasing/compose + kubeai_renderer). Excised the chain (kept deep_merge, used by +normalized_cluster), dropped the `from .catalog import normalize_*`, deleted +infer_stack/catalog.py, and removed the now-orphaned profile templates +(default-*.yaml + the legacy .j2 renderer templates; kept suggestion-pool.yaml). + +232 tests green throughout; ruff clean. Standing caveat unchanged: the +on-terminal feel is Jon's to confirm (he is, live). + +## 2026-06-19 21:30:00 -0400 + +Model: claude-opus-4-8 (Claude Code, fast Opus). Live-test fixes + the suggest +sizing bug (Jon green-lit touching suggest.py — the other agent has been idle the +whole session). + +- API tab: added gateway + Open WebUI URLs (ctrl+click / Open button), a List + models check (GET /v1/models), a live curl preview + Copy curl, and clipboard + via Textual OSC 52 (Copy curl; `y` copies the status line). Caveat surfaced to + Jon: OSC 52 needs terminal/tmux `set-clipboard on`. +- Crash fix: the URL line used Textual `[link=http://...]` markup, which rejects + the ':' in the URL and threw MarkupError in get_content_height. Switched to + plain text (ctrl+click + button still open it). +- suggest.py sizing bug (the real one): vLLM OOM'd at startup for smollm2-1.7b + because suggest overrode the pool's hand-tuned gpu_memory_utilization (0.4, + sized for the 8192 KV cache) with a bare footprint ratio (min_vram*1.3/host = + 0.22 on a 24 GiB GPU) — below what the KV cache needs. Root insight: the pool + default is a *floor* (it encodes the KV requirement); the footprint estimate + should only *raise* it on smaller GPUs. Fix = max(footprint, pool_default), + still clamped [0.2, 0.92]. smollm2-1.7b now → 0.4. Added a regression test. + Note for Jon: existing catalogs already have the bad 0.22 baked in — re-run + `catalog suggest --apply --force`, or Edit the endpoint, to pick up 0.4. + +236 tests green. The suggest fix only changes future suggestions; the standing +on-terminal caveat (clipboard/drag under tmux) is Jon's to confirm live. diff --git a/dev/leasing-followups.md b/dev/leasing-followups.md new file mode 100644 index 00000000..aacf038e --- /dev/null +++ b/dev/leasing-followups.md @@ -0,0 +1,194 @@ +# Leasing model — tracked follow-ups + +Durable list of known design follow-ups for the `infer_stack.leasing` path. +Not bugs (those are fixed on the branch / in CHANGELOG) — these are deliberate +"do this later" decisions with enough context to pick up cold. + +--- + +## LiteLLM reload on model switch — keep the gateway alive + +**Status:** interim fix shipped; **#1 chosen as the real fix (not yet done)**; +**#2 to retry in the future.** + +### Background + +The legacy (pre-leasing) stack deliberately kept LiteLLM **up across model +switches** so in-flight requests to *other* models weren't interrupted. Evidence +— the comment still in `infer_stack/templates/docker-compose.yml.j2`: + +> "avoiding provider dependency edges prevents Compose from restarting LiteLLM on +> every vLLM model swap. Smoke tests and clients should retry until the selected +> upstream model is healthy." + +It worked because legacy LiteLLM had a **static, comprehensive route table** +(model → `vllm-:8000`) plus a `postgres-litellm` (`DATABASE_URL`). A +"switch" only started/stopped vLLM containers; LiteLLM's config never changed, so +it never restarted. + +### What the leasing model does instead (and the interim fix) + +The leasing Compose backend renders LiteLLM's `model_list` **dynamically, to +exactly the live deployment groups**. So adding/removing a served alias changes +the config file. LiteLLM reads that bind-mounted file only at startup, and +`docker compose up -d` won't recreate the litellm service if only the mounted +file changed — so a new route silently never appeared (coalescing readiness +timed out; see CHANGELOG + the `50_coalescing` e2e tier). + +**Interim fix (shipped, commit ~6efad9b):** stamp an `infer-stack.config-hash` +label (sha256 of the rendered config) on the litellm service, so converge +recreates it **iff** the routing changes. Correct, and idempotent (no restart on +repeat-acquires of an existing alias, or on GPU reshuffles that keep the same +groups). **Cost / regression vs. legacy:** a brief (~1–3s) gateway-wide blip on +each *genuine* model add/remove. The vLLM model containers are NOT touched +(compose recreates only the `litellm` service), so GPU-resident models stay +loaded and serving — only the shared proxy hiccups momentarily. + +### #1 — static superset route table (the chosen direction) + +Render LiteLLM with **all catalog endpoints** routed to **stable upstream +addresses**, independent of the live set — like legacy. Then serving/releasing a +model is pure vLLM container churn; LiteLLM's config (and the gateway) never +changes. The config only changes when the *catalog* changes (rare, user edit). + +Why it's a *medium* change, not a one-liner (scope notes for whoever picks it up): +- The Compose backend / controller only see the **live groups** at converge time + — they do **not** have the catalog. A superset table needs the full + endpoint→upstream map threaded CLI → controller → backend → `render_compose`. +- Upstream service names are currently `vllm-` where `group.id` is + **random per group**. A static table needs **deterministic** upstream targets + (e.g. a compose network alias = the served name) so a route is valid before / + across (re)creations. That ripples into the placement sidecar (keyed by group + id) and `observe()`'s service→group mapping. +- Philosophical shift: `/v1/models` would then advertise the whole catalog, + including models not currently deployed (legacy lived with this; clients retry + until the upstream is up). Mitigation: LiteLLM's `/health` (and + `/health/readiness`) probes each configured model's upstream, so "configured" + (`/v1/models`) and "actually deployed/up" (`/health`) are separable — a client + or UI can show undeployed models as down rather than guessing. Worth wiring + into the demo/docs if we go superset, so the side effect is explicit, not a + trap. + +### Deterministic group ids / upstream service names (prereq for #1) + +Today a `DeploymentGroup.id` is random (`grp-`) and the compose service / +LiteLLM upstream are derived from it (`vllm-`, `http://vllm-:8000`). For +#1 the upstream must be predictable. **Decision (maintainer): use the compat-key +hash (option 1) — after the GPU e2e dashboards are green.** Recommendation, in +order: + +- **Preferred — derive the id from the compatibility key:** `grp-`. Coalescing already keys on the compat_key, so the id becomes a + pure function of (model, runtime, served_name, …): same logical model → same + id → same service name → stable upstream. Not a footgun — the compat_key *is* + the identity that defines coalescing. Dedicated groups (which must not + coalesce) get the same hash **plus an owner/nonce salt** so they stay unique. +- **Lower-risk alternative — keep ids opaque, add a deterministic network + alias:** leave `group.id` random but give each vLLM service a compose network + alias equal to its served name, and have LiteLLM route to the alias. Decouples + container identity (random, collision-free) from network identity + (deterministic) without touching the sidecar/observe keying. Pairs naturally + with #1's static table. +- **Not recommended — per-user ids.** The leasing model coalesces *across* users + (two users wanting the same model share one deployment — the whole point), so a + per-user prefix either breaks coalescing or duplicates deployments. This is the + footgun the maintainer worried about; avoid it. Per-user isolation is what the + `--dedicated` flag is for, handled by the nonce salt above. + +### Keep-warm should yield its GPU to live demand (resolved policy) + +Surfaced by the GPU e2e suite: a `reclaim: keep-warm` group survives `release` +and keeps its GPU. On a host where few GPUs are usable (e.g. yardrat: only GPU 0, +GPU 1 is display), an idle keep-warm group starves later acquires — they can't +place and time out. + +**Resolved policy (maintainer):** the decision turns on what "idle" means, which +is unambiguous here because `demand` counts active, un-expired leases: + +- **LIVE (demand ≥ 1, a lease is held): never reclaim.** A held lease protects + the group whether or not requests are currently flowing — "I'm holding this + GPU, don't touch it." Respect it. +- **IDLE (demand == 0: every lease released, or TTL-expired and not renewed): + reclaimable.** No one is holding it; it's warm only as a courtesy. A new lease + with real demand may **evict it to free the GPU** instead of failing to place. + +So keep-warm means "stay resident *if there's room*; yield to real demand when +GPUs are contended" — not "pin a GPU forever." Implementation: when placement +fails for a LIVE-demand group, allow it to reclaim an IDLE group's GPU (tear the +idle one down, place the new one). The ledger already distinguishes LIVE vs IDLE +by demand, so this is a placement/reclaim policy change, not a new state. (The +e2e harness resets between tiers regardless, so this isn't blocking the suite.) + +**Manual eviction landed** (the *explicit* half of this): `infer-stack evict +[NAME…|--all]` and `release --evict` mark IDLE groups `stopped` so the next +reconcile tears them down, freeing the GPU on demand. The *automatic* +yield-under-pressure above (placement evicting an idle group when a live acquire +can't otherwise place) is still open — `evict_idle` / `Controller.evict` are the +reusable primitive it should call once the placer detects contention. + +### #2 — DB-backed live model management (retry later) + +Bring back `postgres-litellm` and use LiteLLM's admin API (`/model/new`, +`/model/delete`, `STORE_MODEL_IN_DB`) so converge updates routes on a **running** +gateway — zero blip, no restart at all. + +**We tried this before and hit issues** (per the maintainer; details to recover +from the legacy stack's history). Not the current direction, but **worth +retrying in the future** — it's strictly nicer than #1 (no blip, and no +advertising of undeployed models) if the earlier problems can be resolved. When +revisiting, capture *what* broke last time before re-committing to it. + +### Recommendation / sequencing + +Do #1 as a focused change after the GPU e2e baseline is fully green (so the +rendering redesign doesn't destabilize tiers still being validated). Keep the +config-hash label as the correctness backstop even after #1 (it's harmless when +the config is static — same hash → no restart). + +--- + +## Other noted smells (lower priority) + +See `docs/source/manual/leasing-demo.md` §"Ergonomic notes". RESOLVED by the CLI reorg +(`dev/cli-redesign.md`): persisted default backend (`config set backend`), +durable storage location (`config set data_dir`, honored by `data_root()`), and +a managed `HF_TOKEN` slot (`secret set HF_TOKEN=…`). ALSO RESOLVED: Open WebUI +is now managed — bundled into the leasing compose stack, on by default +(`--no-ui` / `config set ui false`), and rendered stable across model switches +so it isn't recreated when routing changes (see `infer_stack.leasing.compose +._open_webui_service`). STILL OPEN: no endpoint-addressed teardown for standing +`acquire` leases (want `release --endpoint `). + +Open WebUI sub-follow-ups: it currently runs with `WEBUI_AUTH=False` (single-user +workstation assumption) — expose an auth/port knob before anyone points it at a +shared host. And like LiteLLM, the UI's own recreation on key rotation isn't +specially handled (rare; the baked `OPENAI_API_KEY` changes the spec, so it +recreates — acceptable). + +Also: reclaim:stop deployment groups linger in the ledger as `state=stopped` and +still show up in `infer-stack leases` / its group list. Decide whether `leases` +should hide/prune stopped groups or keep them as history. (Surfaced by the +`50_coalescing` e2e tier needing to filter to `state==live`.) + +### Ledger accumulation / leaked-active-lease (real, found by GPU e2e) + +The GPU e2e suite surfaced a cluster of related product issues (worked around in +the harness by wiping the ledger between tiers; worth fixing in the product): + +- **A not-ready `acquire` keeps its lease ACTIVE with no auto-cleanup.** Unlike + `run` (which releases in a `finally`), a plain `acquire` that times out / never + becomes ready leaves the lease active, so its group stays LIVE and holds a GPU + indefinitely. Decide: should `acquire --timeout` that fails auto-release? At + minimum it's a footgun — a failed acquire silently pins a GPU. +- **Re-acquiring a model spawns a new group instead of reviving a compatible + idle one.** Repeated acquire/release of the same model accumulates distinct + group rows (different ids) rather than coalescing onto the existing idle group. + Fixing this is largely the deterministic-group-id work above (a compat-key id + makes "the qwen-small group" a single stable row). +- **Ledger vs. reality drift.** If a group's container is removed out-of-band + (here: the harness `compose down` between tiers), the ledger still believes it + is LIVE and the next reconcile re-realizes it. Reconcile trusts the ledger as + desired-state and never prunes a LIVE group whose lease leaked. Combined with + the first bullet, one leaked active lease re-spawns a GPU-hogging group on + every subsequent reconcile. An `observe()`-driven reconciliation of ledger + state (or just fixing the leak) would close this. diff --git a/dev/leasing-test-plan.md b/dev/leasing-test-plan.md new file mode 100644 index 00000000..a4a98a29 --- /dev/null +++ b/dev/leasing-test-plan.md @@ -0,0 +1,314 @@ +# Leasing / Compose backend — manual test plan (yardrat) + +Living runbook for exercising the `infer_stack.leasing` Compose path on real +hardware and recording what breaks. Validated against **yardrat**: 2 GPUs — +GPU 0 Quadro RTX 8000 (48 GiB, free), GPU 1 Quadro RTX 5000 (16 GiB, **display +attached**). Both are Turing (sm_75): **fp16 only, no bf16/FP8**. + +Every test block is self-contained: it re-exports the env it needs, so blocks do +not depend on each other (only on the one-time **Setup** that writes +`catalog.yaml`). Copy-paste a whole block. + +> **Prefer the executable harness.** `dev/e2e_tests/` runs these same scenarios, +> asserts the wiring, and writes one `report.md` (+ logs + rendered artifacts) +> you can `rsync` back: `cd dev/e2e_tests && ./run.sh --gpu`. This runbook stays +> as the hand-driven / debugging reference. + +Two hardware facts shape the whole plan: +- GPU 1 has the display, and the placer skips display-active GPUs by default, so + **only GPU 0 is usable** unless that's overridden. `--allowed-gpus` filters + *after* the skip and can't re-include GPU 1. +- Turing needs `extra_args: ['--dtype=half']` on every vLLM endpoint or vLLM + crashes trying bf16. + +--- + +## Findings / fixes log + +Newest first. `FIXED` = patched on the branch; `OPEN` = still to address; +`CONFIRM` = expected, verify on hardware. + +- **F10 — manual chat `curl` got `model=None`. DOC FIX (system is fine).** + The chat-completions `curl` omitted `-H "Content-Type: application/json"`, so + `curl -d` sent form-encoding and LiteLLM didn't parse the JSON body + (`Invalid model name passed in model=None`). Added the header. Confirms the + serving path works end-to-end: vLLM healthy, `/v1/models` lists the alias, and + `acquire --require-generation` already did a real chat through the gateway + (the probe posts via `requests(json=…)`, which sets the header). +- **F9 — vLLM rejects `--disable-log-requests`. FIXED.** `vllm_args` appended it + unconditionally; vLLM v0.19.1 removed it (`unrecognized arguments`), so vLLM + crashed and LiteLLM reported `Connection error … Model Group=qwen-small`. + Dropped the flag (put engine-version-specific flags in `extra_args` if needed). +- **F8 — a stale/invalid compose file bricked `acquire`. FIXED.** + `reconcile` calls `observe()` (which runs `docker compose ps`, validating the + on-disk file) *before* `converge()` rewrites it. A stale file from an earlier + (pre-fix) run made `ps` raise and crashed the whole acquire before converge + could overwrite it. `observe()` is now lenient (returns "nothing observed" on + any docker/parse error), so converge self-heals the file. If you hit this on + a box with an old compose file, the fix overwrites it on the next acquire; or + clean it: `rm -rf "$INFER_STACK_DATA_DIR/leasing/compose"`. +- **F7 — unit-test isolation: ambient `INFER_STACK_*` leaked into the legacy CLI + tests. FIXED.** The subprocess `run_cli` helpers used `setdefault`, so an + exported `INFER_STACK_DATA_DIR` (from these test-plan blocks) made the tests + read the real data dir. Forced to `tmp_path`. (Run unit tests in any shell + now; the leasing acquire/run commands still use the export intentionally.) +- **F1 — `capabilities: [[gpu]]` rejected by Compose schema. FIXED.** + `services.*.deploy.resources.reservations.devices.0.capabilities.0 must be a + string`. `_gpu_reservation` emitted a list-of-lists; Compose wants a list of + strings (`capabilities: [gpu]`). Fixed in `leasing/compose.py` + test. +- **F2 — LiteLLM auth: no manual key needed. FIXED.** infer-stack manages + `LITELLM_MASTER_KEY` in the compose state dir's `.env` (generated, or reused + if you pin your own `sk-` key there), bakes it into the LiteLLM service, uses + it for the probe, and ships it in the `--env-file` as `OPENAI_API_KEY`. So + `source` the env-file and your OpenAI client is fully configured — no `export + LITELLM_MASTER_KEY`. Fetch it ad-hoc with + `$(infer-stack secrets LITELLM_MASTER_KEY)`. +- **F3 — vLLM false-ready without `--require-generation`. CONFIRM.** + Default readiness is "alias listed by LiteLLM", true before vLLM finishes + loading. Use `--require-generation` for honest readiness. +- **F4 — Turing dtype. CONFIRM.** Needs `extra_args: ['--dtype=half']`. +- **F5 — Display GPU unusable, no override knob. FIXED.** The placer skips + display-attached GPUs by default (keeps a desktop responsive). Added + `--include-display-gpus` to the leasing verbs, wired to the Compose backend's + `skip_display`. With it, GPU 1 (the display GPU) is placeable, so two distinct + models spread across both GPUs. Exercised by e2e tier `45_both_gpus`. +- **F6 — No first-class dtype / protocol / image knobs. OPEN.** dtype only via + `extra_args`; readiness probe is always `protocol=chat` (completions-only + models fail it); images pinned in `config.py` with no leasing-CLI override. + +--- + +## Setup (one time) + +```bash +# infer-stack already installed editable on yardrat (uvpy3.11.2). Sanity: +infer-stack version # expect 0.7.0 + +# Docker GPU path must work BEFORE infer-stack: +docker compose version # need v2 +docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi + +# Pre-pull images so acquire timeouts reflect logic, not 10GB downloads: +docker pull vllm/vllm-openai:v0.19.1 # confirm this tag exists; else F6 blocks you +docker pull ghcr.io/berriai/litellm:v1.82.3-stable +docker pull ollama/ollama:latest + +# Scratch + catalog (tailored to yardrat: Turing dtype, tiny ungated chat models) +mkdir -p ~/infer-stack-test +cat > ~/infer-stack-test/catalog.yaml <<'YAML' +models: + qwen05b: {source: hf://Qwen/Qwen2.5-0.5B-Instruct} + qwen15b: {source: hf://Qwen/Qwen2.5-1.5B-Instruct} +endpoints: + qwen-small: + engine: vllm + model: qwen05b + runtime: {max_model_len: 8192, gpu_memory_utilization: 0.3, extra_args: ['--dtype=half']} + reclaim: {policy: stop} + qwen-dup: # same model+runtime, exposed as qwen-small -> must coalesce + engine: vllm + model: qwen05b + public_name: qwen-small + runtime: {max_model_len: 8192, gpu_memory_utilization: 0.3, extra_args: ['--dtype=half']} + qwen-15b: # 2nd distinct model -> exercises the display-GPU limit (F5) + engine: vllm + model: qwen15b + runtime: {max_model_len: 8192, gpu_memory_utilization: 0.3, extra_args: ['--dtype=half']} + reclaim: {policy: keep-warm} + qwen-ollama: + engine: ollama + host: local-ollama + model: qwen2.5:0.5b +runtime_hosts: + local-ollama: + engine: ollama + placement: {gpu_indices: [0]} # run only when no vLLM lease holds GPU 0 (or [] for CPU) + settings: {keep_alive: 5m} +bundles: + pair: [qwen-small, qwen-15b] +YAML +echo "catalog written to ~/infer-stack-test/catalog.yaml" +``` + +--- + +## 1. Dry-run sanity (null backend, no GPU) + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export CAT=~/infer-stack-test/catalog.yaml +infer-stack acquire qwen-small --catalog "$CAT" --env-file /tmp/is.env --no-wait --json +infer-stack leases --json +cat /tmp/is.env +infer-stack release --env-file /tmp/is.env +``` +Expect: JSON with a lease id, one active lease / one live deployment (demand 1), +env-file with `INFER_STACK_ENDPOINT_QWEN_SMALL=qwen-small`, clean release. No +containers (null backend). + +--- + +## 2. Single vLLM model on GPU 0 — the core path + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export CAT=~/infer-stack-test/catalog.yaml +# infer-stack manages LITELLM_MASTER_KEY for you — no export needed. +infer-stack acquire qwen-small --backend compose --catalog "$CAT" \ + --require-generation --env-file /tmp/is.env --timeout 1200 +``` +While it blocks, in a second shell watch: +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export C="$INFER_STACK_DATA_DIR/leasing/compose" +infer-stack ps +infer-stack logs --tail=80 | tail -80 +``` +On success, verify serving + descriptor, then release: +```bash +set -a; . /tmp/is.env; set +a # exports OPENAI_BASE_URL + OPENAI_API_KEY +curl -s "$OPENAI_BASE_URL/models" -H "Authorization: Bearer $OPENAI_API_KEY" | python -m json.tool +curl -s "$OPENAI_BASE_URL/chat/completions" -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" \ + -d '{"model":"qwen-small","messages":[{"role":"user","content":"hi"}],"max_tokens":8}' +infer-stack release --env-file /tmp/is.env +``` +Expect: `qwen-small` listed by `/models`; a chat completion returns text; +`qwen-small` is `reclaim: stop` so its container is gone after release. Watch +for F2 (401 → never ready), F3 (returns ready before vLLM loaded — shouldn't, +since `--require-generation`), F4 (dtype crash in vLLM logs). + +--- + +## 3. Coalescing & demand + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export CAT=~/infer-stack-test/catalog.yaml +export C="$INFER_STACK_DATA_DIR/leasing/compose" +infer-stack acquire qwen-small --backend compose --catalog "$CAT" --owner alice --require-generation --timeout 1200 +infer-stack acquire qwen-dup --backend compose --catalog "$CAT" --owner bob --require-generation --timeout 1200 +infer-stack leases --json # ONE deployment, demand 2; two leases +infer-stack ps # exactly one vllm-… service +``` +Cleanup: +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +infer-stack leases --json | python -c 'import json,sys; [print(l["id"]) for l in json.load(sys.stdin)["leases"]]' \ + | xargs -n1 infer-stack release +``` + +--- + +## 4. Dedicated (exercises the display-GPU limit, F5) + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export CAT=~/infer-stack-test/catalog.yaml +infer-stack acquire qwen-small --backend compose --catalog "$CAT" --owner a --require-generation --timeout 1200 +infer-stack acquire qwen-small --backend compose --catalog "$CAT" --owner b --dedicated --require-generation --timeout 300 +infer-stack leases --json # two deployments requested... +``` +Expected on yardrat: the dedicated 2nd deployment **can't place** (only GPU 0 free → +placement error, no container) so its acquire times out. Confirm via `leases` +(deployment with demand 1 but not running) and the compose `ps`. This is F5. + +--- + +## 5. TTL & reclaim + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export CAT=~/infer-stack-test/catalog.yaml +infer-stack acquire qwen-small --backend compose --catalog "$CAT" --ttl 90s --require-generation --env-file /tmp/is.env --timeout 1200 +sleep 100 +infer-stack leases # lease EXPIRED; qwen-small is reclaim:stop -> deployment reclaimed +``` +Repeat with `qwen-15b` (reclaim: keep-warm) to confirm it stays running after +TTL expiry. + +--- + +## 6. `run` wrapper (the kwdagger pipeline-node seam) + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export CAT=~/infer-stack-test/catalog.yaml +infer-stack run --endpoint qwen-small --backend compose --catalog "$CAT" --require-generation -- \ + bash -c 'curl -s "$OPENAI_BASE_URL/chat/completions" -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" \ + -d "{\"model\":\"$INFER_STACK_ENDPOINT_QWEN_SMALL\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":8}"' +echo "exit=$?" +infer-stack leases # the run lease should be released +# exit-code propagation: +infer-stack run --endpoint qwen-small --backend compose --catalog "$CAT" -- bash -c 'exit 7'; echo "exit=$? (expect 7)" +``` + +--- + +## 7. Ollama daemon + pull/warmup (run with GPU 0 free) + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export CAT=~/infer-stack-test/catalog.yaml +export C="$INFER_STACK_DATA_DIR/leasing/compose" +infer-stack acquire qwen-ollama --backend compose --catalog "$CAT" --require-generation --timeout 900 --env-file /tmp/o.env +infer-stack logs --tail=80 | grep -i pull # expect "ollama pull qwen2.5:0.5b" +set -a; . /tmp/o.env; set +a +curl -s "$OPENAI_BASE_URL/chat/completions" -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" \ + -d '{"model":"qwen-ollama","messages":[{"role":"user","content":"hi"}],"max_tokens":8}' +infer-stack release --env-file /tmp/o.env +``` + +--- + +## 8. Concurrency / "last render wins" fix (the headline) + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export CAT=~/infer-stack-test/catalog.yaml +( infer-stack acquire qwen-small --backend compose --catalog "$CAT" --owner u1 --require-generation --timeout 1200 ) & +( infer-stack acquire qwen-dup --backend compose --catalog "$CAT" --owner u2 --require-generation --timeout 1200 ) & +wait +infer-stack leases --json # both leases present, demand correct, compose file intact +``` +Watch for: the file lock serializing converge, no corrupt/half-written +`docker-compose.yml`, no dropped service. + +--- + +## 9. Negative / edge cases + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export CAT=~/infer-stack-test/catalog.yaml +infer-stack acquire nope --backend compose --catalog "$CAT" --no-wait # friendly "unknown endpoint" +infer-stack acquire qwen-small --backend kubeai --catalog "$CAT" --no-wait # "not implemented" +``` +Also: temporarily remove `--dtype=half` from `qwen-small` in the catalog and +acquire → confirm the vLLM bf16/Turing crash in the logs (F4). + +--- + +## Debugging — capture these when something breaks + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export C="$INFER_STACK_DATA_DIR/leasing/compose" +infer-stack ps +infer-stack logs --tail=120 +infer-stack leases --json +echo '--- rendered compose ---'; cat "$C/docker-compose.yml" +echo '--- litellm config ---'; cat "$C/litellm_config.yaml" +echo '--- backend sidecar ---'; cat "$C/leasing-compose-state.json" +``` +Those pin down whether a failure is placement, render, docker, or the probe. + +--- + +## Cleanup / reset + +```bash +export INFER_STACK_DATA_DIR=~/infer-stack-test/data +export C="$INFER_STACK_DATA_DIR/leasing/compose" +docker compose -p infer-stack -f "$C/docker-compose.yml" down --remove-orphans 2>/dev/null +rm -rf ~/infer-stack-test/data # ledger + compose state (catalog.yaml kept) +``` diff --git a/docs/source/index.rst b/docs/source/index.rst index 7a313b49..4b3218e8 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -13,8 +13,15 @@ Welcome to infer-stack's documentation! .. automodule:: infer_stack.__init__ :show-inheritance: +.. toctree:: + :maxdepth: 2 + :caption: User Manual + + manual/index + .. toctree:: :maxdepth: 5 + :caption: API Reference auto/modules diff --git a/docs/source/manual/index.md b/docs/source/manual/index.md new file mode 100644 index 00000000..9c267958 --- /dev/null +++ b/docs/source/manual/index.md @@ -0,0 +1,17 @@ +# User manual + +Task-oriented guides for running `infer-stack`'s leasing/controller stack. + +```{toctree} +:maxdepth: 1 + +ollama-openwebui-tutorial +leasing-demo +``` + +- **[Ollama + Open WebUI tutorial](ollama-openwebui-tutorial.md)** — stand up a + self-managing Ollama server with Open WebUI from the CLI (GPU-pinned; worked + example on a dual GTX 1080 Ti box). +- **[Leasing demo](leasing-demo.md)** — the happy-path walkthrough: configure + storage once, deploy a chat model as a standing service, and talk to it + through Open WebUI. diff --git a/docs/source/manual/leasing-demo.md b/docs/source/manual/leasing-demo.md new file mode 100644 index 00000000..c1799324 --- /dev/null +++ b/docs/source/manual/leasing-demo.md @@ -0,0 +1,373 @@ +# Leasing demo — deploy a model behind Open WebUI (yardrat) + +A walkthrough, not a test plan. Where `leasing-test-plan.md` pokes at edges, +this shows the **happy path a real user takes**: set up your config + storage +once, deploy a chat model as a standing service, talk to it from Open WebUI, +then run several models side by side — all through the leasing CLI. + +Validated against **yardrat**: GPU 0 Quadro RTX 8000 (48 GiB, free), GPU 1 +Quadro RTX 5000 (16 GiB, display-attached). Both Turing (sm_75): **fp16 only**, +so every vLLM endpoint pins `--dtype=half`. + +Setup is now durable config, not shell exports: `config set` persists your +storage location and default backend, and `catalog` edits the model list — so +the per-command flags stay short and every block is copy-paste-independent. + +> The point of infer-stack is that you switch freely between models and configs. +> "demo settings" and "real settings" are the same settings — we write to the +> real config + real storage and can re-point either at any time. + +Run `infer-stack help tree` any time to see the whole command surface. + +--- + +## 1. One-time setup + +### 1a. Choose where docker-mounted state lives, and the default backend + +The vLLM containers bind-mount a Hugging Face weight cache (a few GiB for these +small SmolLM2 / Qwen models; tens of GiB for a large model). That cache + Open +WebUI history live under your data dir. Persist it once — no shell export needed afterward — +and set Compose as the default backend so you never repeat `--backend`: + +```bash +# It is a good idea to setup the data-dir beforehand with appropriate permissions +export INFER_STACK_DATA_DIR=/data/service/docker/infer-stack +mkdir -p "$INFER_STACK_DATA_DIR" +chown "$USER" "$INFER_STACK_DATA_DIR" + +# Interactive: prompts for each setting (data dir, backend, Open WebUI, +# display-GPU skipping), shows them, confirms. Re-run to edit; --fresh to reset. +infer-stack config init +# ...or non-interactively (scripts / CI): +infer-stack config init --yes --data-dir "$HOME/infer-stack" --backend compose +infer-stack config show +``` + +(Point the data dir at a disk with room — `/data/infer-stack` etc. for large +models. You can also change either later: `infer-stack config set backend …`.) + +### 1b. Build your catalog (your durable model list) + +No YAML by hand — the `catalog` editor validates as it writes: + +```bash +infer-stack catalog init +# Two from the SmolLM2 family... +infer-stack catalog model add smol17b --source hf://HuggingFaceTB/SmolLM2-1.7B-Instruct +infer-stack catalog model add smol135 --source hf://HuggingFaceTB/SmolLM2-135M-Instruct +# ...and two from the Qwen family, so the catalog spans more than one model +# lineage and you can run them side by side (see §5): +infer-stack catalog model add qwen15 --source hf://Qwen/Qwen2.5-1.5B-Instruct +infer-stack catalog model add qwen05 --source hf://Qwen/Qwen2.5-0.5B-Instruct +# Omit the endpoint NAME and it defaults to the model, so what you serve and see +# in Open WebUI *is* the model (Turing => --dtype=half). The main one is kept warm: +infer-stack catalog endpoint add --model smol17b \ + --max-model-len 8192 --gpu-mem 0.4 --extra-args='--dtype=half' --reclaim keep-warm +# the small one frees its GPU on release: +infer-stack catalog endpoint add --model smol135 \ + --max-model-len 4096 --gpu-mem 0.2 --extra-args='--dtype=half' --reclaim stop +# the two Qwen endpoints, both reclaim:stop (they free their GPU on release): +infer-stack catalog endpoint add --model qwen15 \ + --max-model-len 8192 --gpu-mem 0.4 --extra-args='--dtype=half' --reclaim stop +infer-stack catalog endpoint add --model qwen05 \ + --max-model-len 4096 --gpu-mem 0.2 --extra-args='--dtype=half' --reclaim stop +infer-stack catalog show +``` + +A defaulted name gets an auto-incrementing `-N` suffix, so this gives four +endpoints named after their models — **`smol17b-1`** / **`smol135-1`** (SmolLM2) +and **`qwen15-1`** / **`qwen05-1`** (Qwen) — which is what we use throughout. (Add +another `--model smol17b` endpoint and it becomes `smol17b-2`, never clobbering +the first.) `infer-stack catalog show` lists them. + +**Or name endpoints explicitly.** Give a NAME positional when you want a stable +alias *decoupled* from the model — e.g. a `chat` you later re-point at a +different model, so clients and Open WebUI keep asking for `chat` while the model +behind it changes (see §5). The NAME is just the first argument: + +```bash +# explicit name `chat` -> smol17b (everything after NAME is the same flags) +infer-stack catalog endpoint add chat --model smol17b \ + --max-model-len 8192 --gpu-mem 0.4 --extra-args='--dtype=half' --reclaim keep-warm +# a couple more, to show the pattern: +infer-stack catalog endpoint add coder --model smol17b --extra-args='--dtype=half' +infer-stack catalog endpoint add fast-chat --model smol135 --extra-args='--dtype=half' --reclaim stop +infer-stack catalog show +``` + +Mix freely: model-centric names (`smol17b-1`) tell you *which model* in Open +WebUI; explicit names (`chat`) give you a stable handle that outlives the model. +The rest of this demo uses the model-centric ones. + +> Gated model? Set the token once (stored in the managed `.env` that Compose +> auto-loads), no shell export: `infer-stack env HF_TOKEN=hf_…`. + +--- + +## 2. Deploy the model as a standing service + +`acquire` with no `--ttl` is an infinite lease — "deploy this and keep it up" +(add `--ttl 2h` for a time-boxed reservation instead). +`--require-generation` makes readiness honest (waits for a real token). The first +run downloads the weights (a few GiB), so allow time. No `--backend` — it comes +from your settings. A managed **Open WebUI comes up alongside the gateway by +default** (disable with `--no-ui`, or globally `infer-stack config set ui false`). + +On a terminal, **every verb that changes the compose project** — `acquire`, +`release`, `evict`, `apply` — **shows you the diff** and asks you to +confirm before it touches `docker-compose.yml` or runs docker (so you see exactly +which services are added/removed/recreated); `--yes` (or `-y`) skips the prompt, +and it's skipped automatically when output isn't a terminal (scripts/CI). +Declining an `acquire` rolls the lease back; declining a `release`/`evict` +records the change but leaves docker as-is until you `infer-stack apply`. +infer-stack also narrates what it's doing (placement, `docker compose up`, +readiness) on stderr. + +```bash +infer-stack acquire smol17b-1 --require-generation --timeout 1200 +# shows the compose diff, asks to apply (or pass --yes), then: +# open webui: http://127.0.0.1:13000 +``` + +**Prefer to look before you leap?** `acquire` does two things — *render* the +on-disk compose project, then *apply* it (`docker compose up`). They're separate +verbs: `acquire … --no-apply` stages (declares the lease + writes the compose file ++ computes GPU placement, but starts nothing), and `infer-stack apply` is the +trigger. So you can read exactly what would run before committing: + +```bash +infer-stack acquire smol17b-1 --no-apply # stage: writes the project, no `up` +# smol17b-1: GPU 0 (grp-…) +# compose: …/leasing/compose/docker-compose.yml +cat "$(infer-stack config get data_dir)"/leasing/compose/docker-compose.yml # inspect +infer-stack apply # bring the staged set up (the trigger) +infer-stack wait smol17b-1 --require-generation # then block until ready +# (change your mind instead? `infer-stack release --all` discards the staged lease.) +``` + +The compose file carries `name: infer-stack`, so if you'd rather drop the tool +entirely, `docker compose -f up -d` brings up the **same** project — +identical container names and network to what `infer-stack apply` would do. +`infer-stack render` (lease-free) just re-writes that file for whatever's +currently declared, without starting anything. + +Watch it come up from another shell: + +```bash +infer-stack leases # leases + deployments: state (desired) vs running, and GPUs +infer-stack stack ps # vllm-… + litellm + open-webui (alias: infer-stack ps) +infer-stack stack logs -f # follow startup (Ctrl-C to stop) (alias: infer-stack logs -f) +``` + +`leases` now shows the **deployments** table with both what the ledger *wants* +(`state`) and what's *actually up* (`running`), plus the GPUs each model is on +(or `→N` for one that's slated but not started yet) — so it's obvious when +something is still warming up vs truly live. + +> **Prefer a live dashboard?** `infer-stack tui` opens a multi-pane Textual UI +> (run `infer-stack config init` once first): +> a **catalog** pane to pick an endpoint and serve it (`s`/Enter), live +> **leases** + **deployments** tables, and three collapsible panes below: **docker** +> (a `logs -f` tail + a **Containers** `docker ps` view: status/uptime · created +> · container id · ports), **system** (`nvidia-smi` GPUs + host CPU/mem), and +> **api** (send a prompt to a *ready* model — only running models are listed). +> Each pane carries its own description and buttons; expanding a pane (or its +> tab) is what triggers its polling, so hidden data is never fetched. +> New here? The catalog buttons (or `g`) **Suggest** a set sized to your GPUs, +> and `m` / `n` open wizards to add a model / endpoint by hand. **Clean up** +> (`x`) forgets released/stopped entries. **Ctrl+click** a served endpoint +> (or `o`) opens it in Open WebUI. Resize panes by dragging the splitter bars +> (or `[` `]` / `-` `+`). Opt-in extra: `pip install "infer-stack[tui]"`. + +--- + +## 3. Talk to it (the stable front door) + +The quickest check is the built-in smoke test — it sends a real generation to +the endpoint through the gateway and prints latency + the reply: + +```bash +infer-stack test smol17b-1 +# smol17b-1: ok (0.42s) 'ready' +``` + +Under the hood that is one HTTP call to a single base URL +(`http://127.0.0.1:14042/v1`), with infer-stack owning the API key. You can of +course do it by hand — fetch the key inline, never export it: + +```bash +KEY="$(infer-stack env LITELLM_MASTER_KEY)" +curl -s http://127.0.0.1:14042/v1/models \ + -H "Authorization: Bearer $KEY" | python -m json.tool +curl -s http://127.0.0.1:14042/v1/chat/completions \ + -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \ + -d '{"model":"smol17b-1","messages":[{"role":"user","content":"Say hi in one word."}],"max_tokens":16}' +``` + +You ask for the **endpoint name** (`smol17b-1`, your catalog's short name for the +model), not the full HF id — the gateway routes it. + +> **One port instead of two?** Turn on the reverse proxy and you hit a single +> origin — the UI at `/` and the API at `/v1` — so there's nothing to remember: +> ```bash +> infer-stack acquire smol17b-1 --reverse-proxy # or: config set reverse_proxy true +> # then: http://localhost/ -> Open WebUI +> # http://localhost/v1 -> the OpenAI API (test still hits :14042 directly) +> ``` +> It's plain HTTP — no TLS or auth — so it's for localhost / a trusted network, +> not public exposure. Set the port or mount your own `nginx.conf` via the +> `reverse_proxy` setting (`config edit`): `{enabled: true, port: 8080, +> config_path: /my/nginx.conf}`. + +--- + +## 4. Open WebUI (managed, already running) + +Because `acquire` brought Open WebUI up with the stack, just browse to it: + +```bash +echo "Open WebUI -> http://$(hostname):13000 (model: smol17b-1)" +``` + +`smol17b-1` is in the model picker. Its chat history persists under your data dir +(`$(infer-stack config get data_dir)/open-webui`), so it survives restarts and +model switches — and crucially the UI container is **not** recreated when you +add/remove/switch models (only the gateway is), so it never blinks. `WEBUI_AUTH` +is off for a single-user workstation — don't expose port 13000 publicly. + +> Don't want it? `infer-stack acquire smol17b-1 --no-ui` (once) or +> `infer-stack config set ui false` (always). Turning it back on re-renders it +> on the next `acquire`. + +--- + +## 5. Run several models at once (the whole point) + +The gateway stays put; add or drop models freely and Open WebUI's picker +follows. Bring the small model up alongside the big one — each is addressable by +its own name: + +> **One model per GPU.** Placement is whole-GPU: each model lands on its own GPU, +> and `--gpu-mem` is only vLLM's reservation *within* that GPU, not a knob to pack +> two models onto one. So the first model takes GPU 0 and the second takes GPU 1 +> — including yardrat's display-attached GPU 1, since placement uses **every** GPU +> by default (so single-GPU hosts work too). Want to keep the monitor's GPU free? +> `infer-stack config set skip_display_gpus true` (or `--skip-display-gpus` per +> command) — then a second model has nowhere to go and `acquire` fails fast. +> `infer-stack leases` shows which GPU each model is on. + +```bash +infer-stack acquire smol135-1 --require-generation --timeout 600 +infer-stack leases # two live deployments now — GPU 0 and GPU 1 +infer-stack test smol135-1 # confirm the new model serves +``` + +Both models now show in Open WebUI by name (`smol17b-1`, `smol135-1`); the UI +container itself is untouched by the change. + +**Bring several up in parallel — across families.** `acquire` blocks until ready +by default; pass `--no-wait` to kick a deployment off and return immediately, +then `wait` for them together (so models load at once instead of back-to-back). +Here we fan out a SmolLM2 and a Qwen endpoint so both lineages serve side by side: + +```bash +# two models -> two GPUs (GPU 1 is the display GPU, used by default; see note above) +infer-stack acquire smol17b-1 --no-wait --yes # SmolLM2 -> GPU 0 +infer-stack acquire qwen15-1 --no-wait --yes # Qwen -> GPU 1 +infer-stack wait smol17b-1 qwen15-1 --require-generation --timeout 1200 +# (bare `infer-stack wait` waits for every live model) +infer-stack test qwen15-1 # confirm the Qwen endpoint serves +``` + +Open WebUI's picker now lists both `smol17b-1` and `qwen15-1`, so you can A/B the +two families from the same UI. `infer-stack leases` shows each on its own GPU +(GPU 0 and GPU 1); the 16 GiB display GPU holds a small model like `qwen15-1` +comfortably at `--gpu-mem 0.4`. + +> `--require-generation` is the readiness *criterion* (ready == a real generated +> token, not just a listed model); `--no-wait` / `wait` are the *blocking* +> control. They're separate knobs: `acquire` already waits unless you say +> `--no-wait`, and `wait` re-blocks later. + +Drop a standing service when you're done. A standing `acquire` lease taken +without `--env-file` has none, so release it by its lease id (copy it from +`leases`): + +```bash +infer-stack leases # note the lease id of the lease to drop +infer-stack release +# smol17b-1 is reclaim:keep-warm (stays resident); smol135-1 is reclaim:stop +# (its container is torn down once no lease protects it). +``` + +A `keep-warm` model stays resident after release (no cold-start next time) — but +it holds its GPU. To free that GPU now, **evict** it (overrides keep-warm): + +```bash +infer-stack evict smol17b-1 # tear down the idle `smol17b-1` deployment now (by name) +infer-stack evict --all # ...or every idle/released model at once +# or do it in one step at release time: +infer-stack release --evict +``` + +`release`/`evict` only tear down **models** (freeing GPUs). The front door — the +gateway and Open WebUI — stays up even when zero models are served, so the UI +never blinks and reconnects as you serve again. To take the *whole* stack +(gateway + UI included) down, use `infer-stack stack down` (see §6). + +> The explicitly-named `chat` endpoint from §1b is the **stable handle** case: +> re-point it at a different model in place and clients/Open WebUI never change +> what they ask for — +> `infer-stack catalog endpoint add chat --model smol135 --force`, then +> `infer-stack acquire chat`. Same `chat` alias, new model behind it. + +--- + +## 6. Teardown + +```bash +# release frees the models/GPUs but leaves the front door (gateway + Open WebUI) +# up; `stack down` is what takes the whole project — gateway, UI, everything — +# down for a clean slate. +infer-stack release --all +infer-stack stack down +# weights cache + chat history under your data dir are kept (re-serving is then +# fast). Delete the data dir to reclaim disk. +echo "done" +``` + +--- + +## Ergonomic notes + +The earlier draft of this demo flagged five ergonomic smells; building the +`config`/`catalog`/`secret` submodals resolved the first cluster: + +- ✅ **Default backend** — `infer-stack config set backend compose`; the leasing + verbs default to it, so no more `--backend` on every call. +- ✅ **Storage in durable config** — `infer-stack config set data_dir `; + `data_root()` honors it (override > `$INFER_STACK_DATA_DIR` > setting > XDG), so + no `~/.bashrc` export. +- ✅ **Managed `HF_TOKEN`** — `infer-stack env HF_TOKEN=…` writes the managed + `.env` Compose auto-loads, set once before `acquire`. +- ✅ **Managed Open WebUI** — bundled into the leasing stack and on by default + (`--no-ui` / `config set ui false` to opt out). Stable across model switches + *and* down to zero models — the gateway + UI are a standing front door that + `release`/`evict` leave running (only `stack down` removes them), so it never + blinks and reconnects as you serve again. +- ✅ **Concise smoke test** — `infer-stack test smol17b-1` instead of + hand-rolled `curl`. + +Still open (tracked in `dev/leasing-followups.md`): + +- **No endpoint-addressed teardown for standing services.** Stopping a standing + `acquire` means copying a lease id out of `infer-stack leases`. Want + `infer-stack release --endpoint smol17b-1`. (`evict + smol17b-1` already tears the deployment down, but doesn't release the lease.) + + + +### JON TEST + +infer-stack acquire qwen05-1 diff --git a/docs/source/manual/ollama-openwebui-tutorial.md b/docs/source/manual/ollama-openwebui-tutorial.md new file mode 100644 index 00000000..56b6a397 --- /dev/null +++ b/docs/source/manual/ollama-openwebui-tutorial.md @@ -0,0 +1,197 @@ +# Tutorial: a self-managing Ollama + Open WebUI box + +Goal: stand up a plain **Ollama** server with **Open WebUI** in front of it, +driven entirely by the `infer-stack` CLI — a drop-in replacement for a +hand-rolled `ollama` + `open-webui` docker-compose. Ollama manages its own +models: you pull/run/delete them from the Open WebUI model panel and the daemon +loads them on demand. No per-model catalog entries required beyond one anchor. + +Worked example: a 2× GTX 1080 Ti box (Pascal, 11 GiB each). Ollama is the right +engine here — vLLM's bf16/FP8 paths are awkward on pre-Ampere cards, ollama +doesn't care. + +--- + +## TL;DR + +```bash +# 1. one-time setup +infer-stack config init # storage + pick backend = compose +infer-stack config set litellm false # lean: Open WebUI talks straight to ollama + +# 2. declare the daemon (pinned to both 1080 Tis) + one anchor model +infer-stack catalog init +infer-stack catalog host add local-ollama --engine ollama --gpu 0 1 --keep_alive 30m +infer-stack catalog endpoint add chat --engine ollama --host local-ollama --model llama3.2:3b + +# 3. bring it up +infer-stack acquire chat + +# 4. open http://localhost:13000 -> pull any other model from the UI +``` + +That's the whole thing. The rest of this doc explains each piece and the knobs. + +--- + +## What gets stood up + +`infer-stack acquire chat` renders a docker-compose project and brings it up. With +`litellm` off you get exactly two services: + +| service | what it is | port | +|----------------|---------------------------------------------|-----------------| +| `ollama-` | the Ollama daemon, pinned to your GPUs | 11434 (host) | +| `open-webui` | Open WebUI, wired to the daemon's native API | **13000** (host) | + +Open WebUI's **Ollama connection** points straight at the daemon, so its +*Settings → Models* panel can pull, run, and delete models, and the daemon loads +them on demand — just like talking to ollama directly. Chat history and pulled +models persist across restarts (docker volumes under your data dir). + +If you'd rather also have a unified OpenAI `base_url` for scripts/other clients, +**keep LiteLLM on** (skip `config set litellm false`). Then you get a third +`litellm` service; Open WebUI uses it for chat *and* still gets the native Ollama +connection for model management. See [Variants](#variants). + +--- + +## Step by step + +### 1. One-time setup + +```bash +infer-stack config init +``` + +Pick the **compose** backend and a data directory (where weights/state are +bind-mounted). Then make the stack lean: + +```bash +infer-stack config set litellm false +``` + +This is the persistent default; you can also decide per-invocation with +`acquire … --no-litellm` / `--litellm`. + +### 2. Declare the Ollama daemon + +A runtime host is the long-lived daemon (one daemon serves many tags): + +```bash +infer-stack catalog host add local-ollama \ + --engine ollama \ + --gpu 0 1 \ # pin to GPU 0 and 1 (both 1080 Tis) + --keep_alive 30m # how long a model stays resident after last use +``` + +`--gpu` is exactly the GPU restriction you wanted: the daemon only ever sees the +indices you list (it becomes the container's `CUDA_VISIBLE_DEVICES`). Use +`--gpu 0` to keep it off your display card, `--gpu 0 1` to let ollama use both +(it will split a model across them when one card can't hold it). + +Other optional knobs: `--num_parallel`, `--max_loaded_models`, +`--context_length`, `--image` (a custom ollama image). + +### 3. Declare one anchor model + +`acquire` needs at least one endpoint to start the daemon. Declare a small default +— it gets pulled automatically so the box is useful the moment it's up: + +```bash +infer-stack catalog endpoint add chat \ + --engine ollama --host local-ollama --model llama3.2:3b +``` + +For ollama, `--model` is just the **ollama tag** (`llama3.2:3b`, `qwen2.5:7b`, +…) — no separate `catalog model add` needed. The alias `chat` is what shows up +in Open WebUI. (Omit the name and it auto-names `llama3.2-3b-1`.) + +You can review what you built any time: + +```bash +infer-stack catalog show +``` + +### 4. Bring it up + +```bash +infer-stack acquire chat +``` + +This places the daemon on your GPUs, writes the compose project, runs +`docker compose up`, pulls the `llama3.2:3b` tag, and waits until it's serving. +On a terminal you'll see the compose diff and a confirm prompt first +(`--yes` to skip). + +### 5. Use it — and let ollama manage the rest + +Open **http://localhost:13000**. The `chat` model is already there. To add more, +go to **Settings → Admin → Models** (or the model picker's *Manage* / +*Pull a model* control) and pull any tag — `qwen2.5:14b`, `gemma2:9b`, whatever. +The daemon downloads it, keeps it on disk, and loads it on demand. You never +have to touch the catalog again for day-to-day model juggling. + +--- + +## Managing the stack + +```bash +infer-stack status # holistic overview: backend, GPUs, leases, URLs +infer-stack ps # docker compose ps for the stack +infer-stack logs # tail service logs +infer-stack stack down # stop everything +``` + +To change the daemon's GPUs or keep-alive later, re-run `catalog host add +local-ollama … --force` and `infer-stack acquire chat` again — the daemon (and its +on-disk models) is recreated with the new pinning; Open WebUI stays up. + +--- + +## Variants + +### Keep LiteLLM too (OpenAI base_url for scripts) + +Leave `litellm` on (the default). You then also get an OpenAI-compatible gateway: + +```bash +infer-stack config set litellm true # or just don't turn it off +infer-stack acquire chat +infer-stack env # prints base_url + API key for clients +``` + +Open WebUI uses the gateway for chat **and** keeps the native Ollama connection +for pull/manage — best of both. The trade-off: only models declared as endpoints +are routable through the gateway; ad-hoc models you pull from the UI are +reachable in the UI but not through the OpenAI `base_url` until you also +`catalog endpoint add` them. + +### Leave the display GPU free + +If one 1080 Ti drives your monitor and you don't want ollama touching it, pin to +the other: + +```bash +infer-stack catalog host add local-ollama --engine ollama --gpu 0 --keep_alive 30m --force +``` + +(Or globally: `infer-stack config set skip_display_gpus true`, which drops +display-attached GPUs from the placement pool.) + +### Change the Open WebUI port + +```bash +infer-stack config edit # set ports.open_webui +``` + +--- + +## Notes for Pascal (1080 Ti) + +* Ollama runs fine on Pascal; no special flags. (This is the main reason to pick + ollama over vLLM on this hardware.) +* 11 GiB per card: a ~13B Q4 model needs both cards — `--gpu 0 1` lets ollama + split it. Smaller models sit on one. +* `--keep_alive` trades VRAM for latency: longer keeps models resident (snappy, + but holds VRAM); `0` unloads immediately after each request. diff --git a/infer_stack/__init__.py b/infer_stack/__init__.py index 677eff3b..e4100e1a 100644 --- a/infer_stack/__init__.py +++ b/infer_stack/__init__.py @@ -1,2 +1,2 @@ __all__ = [] -__version__ = '0.6.1' +__version__ = '0.7.0' diff --git a/infer_stack/__main__.py b/infer_stack/__main__.py new file mode 100644 index 00000000..f0dd620c --- /dev/null +++ b/infer_stack/__main__.py @@ -0,0 +1,6 @@ +"""Enable ``python -m infer_stack`` to run the CLI.""" + +from .cli import main + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/infer_stack/_log.py b/infer_stack/_log.py new file mode 100644 index 00000000..1384f113 --- /dev/null +++ b/infer_stack/_log.py @@ -0,0 +1,51 @@ +"""loguru-based "behind the scenes" feedback for the leasing CLI. + +This is deliberately kept off the ``--help`` import path (loguru is ~50ms to +import): nothing at module scope in the CLI imports this. It is pulled in only +at runtime by the leasing verbs (``configure_logging``) and the converge/ +reconcile code that narrates what it is doing. + +The ``infer_stack`` package is ``disable``d here on import, so importing the +library (or running the test suite, which calls subcommands directly) stays +silent until the CLI explicitly calls :func:`configure_logging`. +""" + +from __future__ import annotations + +import os +import sys + +from loguru import logger + +__all__ = ['logger', 'configure_logging'] + +# Silent by default; the CLI entry points re-enable via configure_logging(). +logger.disable('infer_stack') +_CONFIGURED = False + + +def configure_logging(level: str | None = None, *, force: bool = False): + """Route infer-stack's narration to stderr (idempotent). + + Level resolves to ``level`` arg > ``$INFER_STACK_LOG_LEVEL`` > ``INFO``. + Narration goes to stderr so it never pollutes a command's stdout (JSON, + ``$(infer-stack env KEY)``, etc.). Colorized only on a real terminal. + """ + global _CONFIGURED + logger.enable('infer_stack') + if _CONFIGURED and not force: + return logger + level = (level or os.environ.get('INFER_STACK_LOG_LEVEL') or 'INFO').upper() + colorize = sys.stderr.isatty() and os.environ.get('NO_COLOR') is None + logger.remove() + logger.add( + sys.stderr, + level=level, + colorize=colorize, + format=( + '{time:HH:mm:ss} ' + '{level: <7} {message}' + ), + ) + _CONFIGURED = True + return logger diff --git a/infer_stack/backends/__init__.py b/infer_stack/backends/__init__.py index 2beff8d9..ad2cf5d0 100644 --- a/infer_stack/backends/__init__.py +++ b/infer_stack/backends/__init__.py @@ -1,7 +1,3 @@ -from .compose_renderer import render_compose_artifacts from .kubeai_renderer import render_kubeai_artifacts -__all__ = [ - 'render_compose_artifacts', - 'render_kubeai_artifacts', -] +__all__ = ['render_kubeai_artifacts'] diff --git a/infer_stack/backends/compose_renderer.py b/infer_stack/backends/compose_renderer.py deleted file mode 100644 index 992b46a7..00000000 --- a/infer_stack/backends/compose_renderer.py +++ /dev/null @@ -1,243 +0,0 @@ -from __future__ import annotations - -import json -from importlib.resources import files -from pathlib import Path - -import yaml -from jinja2 import BaseLoader, Environment - -from ..config import normalized_output, normalized_state, DEFAULT_PORTS -from ..diff_prompt import confirm_writes -from ..env_utils import ensure_secret, parse_env_file, write_env_file - - -def _template(name: str) -> str: - return ( - files('infer_stack') - .joinpath(f'templates/{name}') - .read_text(encoding='utf-8') - ) - - -def _compose_quote(value: object) -> str: - """Quote scalars for Compose YAML while preserving env interpolation text.""" - - return json.dumps('' if value is None else str(value)) - - -def _compose_gpus(value: object, indent: int = 4) -> str: - """Render the ``gpus:`` service key for either form Compose accepts. - - Compose understands both a scalar (``all``, ``-1``, or a device count) and - a structured list of device-request mappings. Scalars render inline and - quoted; lists/dicts are emitted as indented block YAML so the structured - "GPU settings" escape hatch round-trips faithfully instead of collapsing to - a quoted Python repr. - """ - - pad = ' ' * indent - if value is None or value == '': - return '' - if isinstance(value, bool): - value = 'true' if value else 'false' - if isinstance(value, (str, int, float)): - scalar = ( - _compose_quote(value) - if isinstance(value, str) - else json.dumps(value) - ) - return f'{pad}gpus: {scalar}' - dumped = yaml.safe_dump( - {'gpus': value}, default_flow_style=False, sort_keys=False - ).rstrip('\n') - return '\n'.join(pad + line for line in dumped.splitlines()) - - -def render_compose_artifacts( - lock_data: dict, *, assume_yes: bool = True -) -> None: - """Render component-aware Compose artifacts for the resolved stack.""" - deployment = dict(lock_data.get('deployment', {})) - deployment['state'] = normalized_state(deployment.get('state', {})) - deployment['output'] = normalized_output(deployment.get('output')) - generated = Path(deployment['output']['generated_dir']) - generated.mkdir(parents=True, exist_ok=True) - runtime_dir = Path(deployment['state']['runtime']) - runtime_dir.mkdir(parents=True, exist_ok=True) - - env_path = generated / '.env' - existing = parse_env_file(env_path) - env_values: dict[str, str] = {} - - frontends = deployment.get('frontends', {}) or {} - gateways = deployment.get('gateways', {}) or {} - providers = deployment.get('providers', {}) or {} - - if (frontends.get('open_webui') or {}).get('enabled'): - env_values.update( - { - 'OPENWEBUI_POSTGRES_DB': existing.get( - 'OPENWEBUI_POSTGRES_DB', 'openwebui' - ), - 'OPENWEBUI_POSTGRES_USER': existing.get( - 'OPENWEBUI_POSTGRES_USER', 'openwebui' - ), - 'OPENWEBUI_POSTGRES_PASSWORD': ensure_secret( - existing, 'OPENWEBUI_POSTGRES_PASSWORD' - ), - 'WEBUI_SECRET_KEY': ensure_secret(existing, 'WEBUI_SECRET_KEY'), - } - ) - ldap_defaults = ( - (frontends.get('open_webui') or {}).get('ldap') or {} - ).get('env_defaults') or {} - for key, default in ldap_defaults.items(): - env_values[key] = existing.get(key, str(default)) - - if (gateways.get('litellm') or {}).get('enabled'): - env_values.update( - { - 'LITELLM_POSTGRES_DB': existing.get( - 'LITELLM_POSTGRES_DB', 'litellm' - ), - 'LITELLM_POSTGRES_USER': existing.get( - 'LITELLM_POSTGRES_USER', 'litellm' - ), - 'LITELLM_POSTGRES_PASSWORD': ensure_secret( - existing, 'LITELLM_POSTGRES_PASSWORD' - ), - 'LITELLM_MASTER_KEY': ensure_secret( - existing, 'LITELLM_MASTER_KEY', prefix='sk-' - ), - } - ) - - if (providers.get('vllm') or {}).get('enabled'): - env_values.update( - { - 'VLLM_BACKEND_API_KEY': ensure_secret( - existing, 'VLLM_BACKEND_API_KEY' - ), - 'HF_TOKEN': existing.get('HF_TOKEN', ''), - } - ) - - # Ports: expose configured host ports via environment variables so - # docker-compose can reference them and we persist them into `.env`. - ports = deployment.get('ports', {}) or {} - - # LiteLLM port - if (gateways.get('litellm') or {}).get('enabled'): - litellm_port = ports.get('litellm') or DEFAULT_PORTS.get( - 'litellm', 14042 - ) - env_values['INFER_STACK_LITELLM_PORT'] = existing.get( - 'INFER_STACK_LITELLM_PORT', str(litellm_port) - ) - - # Open WebUI port - if (frontends.get('open_webui') or {}).get('enabled'): - open_webui_port = ports.get('open_webui') or DEFAULT_PORTS.get( - 'open_webui', 13000 - ) - env_values['INFER_STACK_OPEN_WEBUI_PORT'] = existing.get( - 'INFER_STACK_OPEN_WEBUI_PORT', str(open_webui_port) - ) - - # Reverse proxy ports - reverse_proxy = frontends.get('reverse_proxy') or {} - if reverse_proxy.get('enabled'): - http_port = ( - reverse_proxy.get('http_port') - or ports.get('reverse_proxy_http') - or DEFAULT_PORTS.get('reverse_proxy_http', 80) - ) - https_port = ( - reverse_proxy.get('https_port') - or ports.get('reverse_proxy_https') - or DEFAULT_PORTS.get('reverse_proxy_https', 443) - ) - env_values['INFER_STACK_REVERSE_PROXY_HTTP_PORT'] = existing.get( - 'INFER_STACK_REVERSE_PROXY_HTTP_PORT', str(http_port) - ) - env_values['INFER_STACK_REVERSE_PROXY_HTTPS_PORT'] = existing.get( - 'INFER_STACK_REVERSE_PROXY_HTTPS_PORT', str(https_port) - ) - - # Ollama port (if publish enabled) - if (providers.get('ollama') or {}).get('enabled'): - # Ollama host_port is resolved in the deployment; fall back to DEFAULT_PORTS - ollama_port = ( - (providers.get('ollama') or {}).get('host_port') - or ports.get('ollama') - or DEFAULT_PORTS.get('ollama', 11434) - ) - env_values['INFER_STACK_OLLAMA_PORT'] = existing.get( - 'INFER_STACK_OLLAMA_PORT', str(ollama_port) - ) - - # vLLM runtimes: enumerate and export per-runtime host ports (index-based) - vllm_runtimes = (providers.get('vllm') or {}).get('runtimes', {}) or {} - for idx, (name, svc) in enumerate(vllm_runtimes.items()): - host_port = svc.get('host_port') or ports.get('vllm') or (18000 + idx) - env_name = f'INFER_STACK_VLLM_{idx}_PORT' - env_values[env_name] = existing.get(env_name, str(host_port)) - - # Preserve unknown/user-supplied keys, but let managed keys above win. - for key, value in existing.items(): - env_values.setdefault(key, value) - - env = Environment( - loader=BaseLoader(), - autoescape=False, - trim_blocks=True, - lstrip_blocks=True, - ) - env.filters['compose_quote'] = _compose_quote - env.filters['compose_gpus'] = _compose_gpus - normalized_lock = dict(lock_data) - normalized_lock['deployment'] = deployment - - reverse_proxy = (deployment.get('frontends') or {}).get( - 'reverse_proxy' - ) or {} - if reverse_proxy.get('enabled'): - if reverse_proxy.get('config_path'): - reverse_proxy['nginx_config_path'] = reverse_proxy['config_path'] - else: - reverse_proxy['nginx_config_path'] = str(runtime_dir / 'nginx.conf') - deployment['frontends']['reverse_proxy'] = reverse_proxy - - ctx = {'lock': normalized_lock} - compose = ( - env.from_string(_template('docker-compose.yml.j2')).render(**ctx) + '\n' - ) - - compose_fpath = generated / 'docker-compose.yml' - planned: dict[Path, str] = {compose_fpath: compose} - - lite_llm_config_fpath = runtime_dir / 'litellm_config.yaml' - if (gateways.get('litellm') or {}).get('enabled'): - litellm_cfg = ( - env.from_string(_template('litellm_config.yaml.j2')).render(**ctx) - + '\n' - ) - planned[lite_llm_config_fpath] = litellm_cfg - - if reverse_proxy.get('enabled') and not reverse_proxy.get('config_path'): - nginx_ctx = {'rp': reverse_proxy} - planned[Path(reverse_proxy['nginx_config_path'])] = ( - env.from_string(_template('nginx.conf.j2')).render(**nginx_ctx) - + '\n' - ) - - if not confirm_writes( - planned, assume_yes=assume_yes, title='Pending compose render' - ): - raise SystemExit('Aborted by user; no files were written.') - - write_env_file(env_path, env_values) - for path, text in planned.items(): - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(text, encoding='utf-8') diff --git a/infer_stack/benchmark.py b/infer_stack/benchmark.py deleted file mode 100644 index a0c69bf7..00000000 --- a/infer_stack/benchmark.py +++ /dev/null @@ -1,38 +0,0 @@ -from __future__ import annotations - -import time -from typing import Any - -import requests - - -def run_benchmark( - base_url: str, api_key: str, model: str, prompts: list[str] -) -> dict[str, Any]: - headers = {'Content-Type': 'application/json'} - if api_key: - headers['Authorization'] = f'Bearer {api_key}' - results = [] - for prompt in prompts: - payload = { - 'model': model, - 'messages': [{'role': 'user', 'content': prompt}], - 'max_tokens': 128, - } - t0 = time.time() - resp = requests.post( - f'{base_url.rstrip("/")}/chat/completions', - headers=headers, - json=payload, - timeout=300, - ) - dt = time.time() - t0 - results.append( - { - 'prompt': prompt, - 'status_code': resp.status_code, - 'elapsed_s': dt, - 'ok': resp.ok, - } - ) - return {'results': results} diff --git a/infer_stack/catalog.py b/infer_stack/catalog.py deleted file mode 100644 index cd0981a4..00000000 --- a/infer_stack/catalog.py +++ /dev/null @@ -1,438 +0,0 @@ -from __future__ import annotations - -from copy import deepcopy -from typing import Any - - -PROFILE_NAME_ALIASES = { - 'helm-qwen2-72b-instruct': 'qwen2-72b-instruct-tp2-balanced', - 'helm-qwen2.5-7b-instruct': 'qwen2-5-7b-instruct-turbo-default', - 'helm-qwen2.5-72b-instruct': 'qwen2-5-72b-instruct-tp2-balanced', - 'helm-gpt-oss-20b': 'gpt-oss-20b-completions', - 'helm-vicuna-7b-v1.3': 'vicuna-7b-v1-3-no-chat-template', -} - - -def sanitize_name(value: str) -> str: - value = str(value).strip().lower() - out: list[str] = [] - prev_dash = False - for char in value: - if char.isalnum(): - out.append(char) - prev_dash = False - continue - if not prev_dash: - out.append('-') - prev_dash = True - sanitized = ''.join(out).strip('-') - return sanitized or 'profile' - - -def canonical_profile_name(name: str | None) -> str | None: - if name is None: - return None - return PROFILE_NAME_ALIASES.get(name, name) - - -def _list(value: Any, default: list[Any] | None = None) -> list[Any]: - if value is None: - return list(default or []) - if isinstance(value, list): - return deepcopy(value) - if isinstance(value, tuple): - return list(value) - return [value] - - -def normalize_vllm_models(catalog: dict[str, Any]) -> dict[str, Any]: - normalized: dict[str, Any] = {} - for key, raw in (catalog or {}).items(): - entry = deepcopy(raw) or {} - hf_model_id = entry.get('hf_model_id', '') - url = entry.get('url') or (f'hf://{hf_model_id}' if hf_model_id else '') - defaults = deepcopy(entry.get('defaults', {})) - supported_protocols = entry.get('supported_protocols') - if supported_protocols is None: - supported_protocols = ['chat', 'completions'] - normalized[key] = { - 'key': key, - 'provider': 'vllm', - 'canonical_key': sanitize_name(entry.get('canonical_key', key)), - 'hf_model_id': hf_model_id, - 'url': url, - 'family': entry.get('family', ''), - 'modalities': _list(entry.get('modalities'), ['text']), - 'supported_protocols': [str(p) for p in supported_protocols], - 'reasoning': deepcopy(entry.get('reasoning', {})), - 'tokenizer_name': entry.get('tokenizer_name') - or entry.get('tokenizer') - or entry.get('served_model_name') - or key, - 'logical_model_name': entry.get('logical_model_name') - or entry.get('served_model_name') - or key, - 'served_model_name': entry.get('served_model_name') - or entry.get('logical_model_name') - or key, - 'memory_class_gib': entry.get('memory_class_gib'), - 'min_vram_gib_per_replica': entry.get( - 'min_vram_gib_per_replica', 0 - ), - 'preferred_gpu_count': entry.get('preferred_gpu_count', 1), - 'context_window': entry.get('context_window'), - 'defaults': defaults, - 'engine': 'VLLM', - 'resource_profile': entry.get('resource_profile', ''), - 'priority_class_name': entry.get('priority_class_name'), - 'tool_calling': deepcopy(entry.get('tool_calling', {})), - 'thinking_history_policy': entry.get( - 'thinking_history_policy', 'keep_final_only' - ), - 'features': deepcopy(entry.get('features', ['TextGeneration'])), - 'safe_defaults': deepcopy(entry.get('safe_defaults', defaults)), - 'notes': _list(entry.get('notes')), - 'caveats': _list(entry.get('caveats')), - } - return normalized - - -def normalize_ollama_models(catalog: dict[str, Any]) -> dict[str, Any]: - normalized: dict[str, Any] = {} - for key, raw in (catalog or {}).items(): - entry = deepcopy(raw) or {} - tag = ( - entry.get('tag') - or entry.get('ollama_model') - or entry.get('model') - or key - ) - normalized[key] = { - 'key': key, - 'provider': 'ollama', - 'tag': str(tag), - 'served_model_name': entry.get('served_model_name') - or sanitize_name(str(tag)), - 'logical_model_name': entry.get('logical_model_name') - or entry.get('served_model_name') - or sanitize_name(str(tag)), - 'modalities': _list(entry.get('modalities'), ['text']), - 'supported_protocols': [ - str(p) - for p in _list(entry.get('supported_protocols'), ['chat']) - ], - 'context_window': entry.get('context_window'), - 'defaults': deepcopy(entry.get('defaults', {})), - 'notes': _list(entry.get('notes')), - 'caveats': _list(entry.get('caveats')), - } - return normalized - - -# Compatibility view used by older helper commands/tests. Prefer provider-specific -# catalogs in new code. -def normalize_model_catalog(catalog: dict[str, Any]) -> dict[str, Any]: - return normalize_vllm_models(catalog) - - -def _infer_protocol_mode( - profile_name: str, logical_model_name: str, raw: dict[str, Any] -) -> str: - explicit = raw.get('protocol_mode') or raw.get('protocol') - if explicit: - return str(explicit) - tags = {str(tag) for tag in raw.get('tags', [])} - if 'completions' in tags: - return 'completions' - if 'chat' in tags: - return 'chat' - name = f'{profile_name} {logical_model_name}'.lower() - if 'completion' in name: - return 'completions' - return 'chat' - - -def _normalize_bool_map( - raw: Any, default_enabled: bool = False -) -> dict[str, Any]: - if isinstance(raw, dict): - return deepcopy(raw) - if raw in (None, 'auto'): - return {'enabled': 'auto'} - return {'enabled': bool(raw) if raw is not None else default_enabled} - - -def _normalize_components( - profile: dict[str, Any], -) -> tuple[dict[str, Any], dict[str, Any], dict[str, Any]]: - providers = deepcopy(profile.get('providers', {})) - gateways = deepcopy(profile.get('gateways', {})) - frontends = deepcopy(profile.get('frontends', {})) - - # Convenience/older shape: components: {ollama: true, litellm: false, ...} - components = deepcopy(profile.get('components', {}) or {}) - for name in ['ollama', 'vllm']: - if name in components and name not in providers: - providers[name] = _normalize_bool_map(components[name]) - if 'litellm' in components and 'litellm' not in gateways: - gateways['litellm'] = _normalize_bool_map(components['litellm']) - if 'open_webui' in components and 'open_webui' not in frontends: - frontends['open_webui'] = _normalize_bool_map(components['open_webui']) - - providers.setdefault('vllm', {}) - providers.setdefault('ollama', {'enabled': 'auto'}) - gateways.setdefault('litellm', {'enabled': 'auto'}) - frontends.setdefault('open_webui', {'enabled': 'auto', 'provider': 'auto'}) - return providers, gateways, frontends - - -def _route_aliases(route_name: str, raw: dict[str, Any]) -> list[str]: - aliases = raw.get('aliases') - if aliases is None: - aliases = raw.get('served_aliases') - if aliases is None: - aliases = [route_name] - elif isinstance(aliases, str): - aliases = [aliases] - else: - aliases = list(aliases) - if route_name not in aliases: - aliases.insert(0, route_name) - ordered: list[str] = [] - for alias in aliases: - if alias and alias not in ordered: - ordered.append(str(alias)) - return ordered - - -def _legacy_profile_to_stack( - name: str, raw: dict[str, Any], vllm_models: dict[str, Any] -) -> dict[str, Any]: - services = deepcopy(raw.get('services', [])) - if not services and (raw.get('model') or raw.get('base_model')): - services = [deepcopy(raw)] - aliases = deepcopy(raw.get('router', {}).get('aliases', {})) - runtimes: dict[str, Any] = {} - routes: dict[str, Any] = {} - for index, service in enumerate(services): - runtime_name = sanitize_name( - service.get('service_name') - or service.get('runtime') - or service.get('name') - or (name if len(services) == 1 else f'runtime-{index + 1}') - ) - model_key = service.get('base_model') or service.get('model') - if model_key not in vllm_models: - raise KeyError(f'Unknown vLLM model: {model_key}') - model = vllm_models[model_key] - public_name = sanitize_name( - service.get('public_name') - or (name if len(services) == 1 else f'{name}-{runtime_name}') - ) - logical_model_name = ( - service.get('logical_model_name') - or model.get('logical_model_name') - or model.get('served_model_name') - or model_key - ) - served_model_name = ( - service.get('served_model_name') - or model.get('served_model_name') - or logical_model_name - ) - protocol_mode = _infer_protocol_mode( - public_name, logical_model_name, service - ) - runtimes[runtime_name] = { - 'model': model_key, - 'public_name': public_name, - 'logical_model_name': logical_model_name, - 'served_model_name': served_model_name, - 'protocol_mode': protocol_mode, - 'placement': deepcopy(service.get('placement', {})), - 'topology': deepcopy(service.get('topology', {})), - 'runtime': deepcopy(service.get('runtime', {})), - 'extra_args': deepcopy(service.get('extra_args', [])), - 'reasoning': deepcopy(service.get('reasoning', {})), - 'tool_calling': deepcopy(service.get('tool_calling', {})), - 'chat_compat': deepcopy(service.get('chat_compat', {})), - 'resource_profile': service.get( - 'resource_profile', model.get('resource_profile', '') - ), - 'min_replicas': int( - service.get( - 'min_replicas', - model.get('defaults', {}).get('min_replicas', 0), - ) - ), - 'max_replicas': int( - service.get( - 'max_replicas', - model.get('defaults', {}).get('max_replicas', 1), - ) - ), - 'priority_class_name': service.get( - 'priority_class_name', model.get('priority_class_name') - ), - 'tags': list(service.get('tags', raw.get('tags', [])) or []), - 'audit_notes': list( - service.get('audit_notes', raw.get('audit_notes', [])) or [] - ), - 'notes': list(service.get('notes', raw.get('notes', [])) or []), - 'benchmark_transport': deepcopy( - service.get( - 'benchmark_transport', - service.get( - 'transport', - raw.get( - 'benchmark_transport', raw.get('transport', {}) - ), - ), - ) - ), - 'publish_port': bool(service.get('publish_port', False)), - } - # Convert old router alias map to route map. - if aliases: - for alias, target in aliases.items(): - routes[str(alias)] = { - 'provider': 'vllm', - 'runtime': sanitize_name(str(target)), - } - else: - for runtime_name, rt in runtimes.items(): - route_name = str( - rt.get('public_name') - or rt.get('logical_model_name') - or rt.get('served_model_name') - or runtime_name - ) - routes[route_name] = {'provider': 'vllm', 'runtime': runtime_name} - return { - 'name': name, - 'description': raw.get('description', ''), - 'kind': 'stack', - 'providers': { - 'vllm': {'enabled': True, 'runtimes': runtimes}, - 'ollama': {'enabled': False}, - }, - 'gateways': {'litellm': {'enabled': True}}, - 'frontends': {'open_webui': {'enabled': True, 'provider': 'litellm'}}, - 'routes': routes, - 'policy': deepcopy(raw.get('policy', {})), - 'vllm': deepcopy(raw.get('vllm', {})), - 'tags': list(raw.get('tags', []) or []), - 'audit_notes': list(raw.get('audit_notes', []) or []), - 'notes': list(raw.get('notes', []) or []), - } - - -def _normalize_route_map(routes_raw: Any) -> dict[str, Any]: - if routes_raw is None: - return {} - if isinstance(routes_raw, list): - out: dict[str, Any] = {} - for item in routes_raw: - raw = deepcopy(item) - name = str( - raw.pop('name', raw.get('alias', raw.get('model', 'route'))) - ) - out[name] = raw - return out - return deepcopy(routes_raw) - - -def normalize_stack_profiles( - catalog: dict[str, Any], - vllm_models: dict[str, Any], - ollama_models: dict[str, Any], -) -> dict[str, Any]: - normalized: dict[str, Any] = {} - for original_name, raw in (catalog or {}).items(): - name = original_name - profile = deepcopy(raw) or {} - try: - if ( - 'providers' not in profile - and 'routes' not in profile - and 'components' not in profile - ): - stack = _legacy_profile_to_stack(name, profile, vllm_models) - else: - providers, gateways, frontends = _normalize_components(profile) - stack = { - 'name': name, - 'description': profile.get('description', ''), - 'kind': profile.get('kind', 'stack'), - 'providers': providers, - 'gateways': gateways, - 'frontends': frontends, - 'routes': _normalize_route_map(profile.get('routes', {})), - 'policy': deepcopy(profile.get('policy', {})), - 'vllm': deepcopy(profile.get('vllm', {})), - 'tags': list(profile.get('tags', []) or []), - 'audit_notes': list(profile.get('audit_notes', []) or []), - 'notes': list(profile.get('notes', []) or []), - } - normalized[name] = stack - except KeyError as ex: - normalized[name] = { - 'name': name, - 'description': profile.get('description', ''), - 'kind': 'invalid-profile', - 'catalog_error': str(ex), - 'providers': {}, - 'gateways': {}, - 'frontends': {}, - 'routes': {}, - 'policy': deepcopy(profile.get('policy', {})), - 'vllm': deepcopy(profile.get('vllm', {})), - 'tags': list(profile.get('tags', []) or []), - 'audit_notes': list(profile.get('audit_notes', []) or []), - 'notes': list(profile.get('notes', []) or []), - } - return normalized - - -# Compatibility function name used in a few imports. -def normalize_profile_catalog( - catalog: dict[str, Any], models: dict[str, Any] -) -> dict[str, Any]: - return normalize_stack_profiles(catalog, normalize_vllm_models(models), {}) - - -def profile_summary(profile: dict[str, Any]) -> dict[str, Any]: - providers = [] - p = profile.get('providers', {}) or {} - if (p.get('ollama') or {}).get('enabled') not in (False, 'false', None): - providers.append('ollama') - vllm_runtimes = (p.get('vllm') or {}).get('runtimes') or {} - if vllm_runtimes or (p.get('vllm') or {}).get('enabled') is True: - providers.append('vllm') - litellm = (profile.get('gateways', {}).get('litellm') or {}).get( - 'enabled', 'auto' - ) - open_webui = profile.get('frontends', {}).get('open_webui', {}) or {} - return { - 'name': profile['name'], - 'public_name': profile.get('name', ''), - 'kind': profile.get('kind', 'stack'), - 'providers': providers, - 'gateway': 'litellm' - if litellm is True - else ('auto' if litellm == 'auto' else 'none'), - 'frontend': 'open_webui' - if open_webui.get('enabled', 'auto') not in (False, 'false') - else 'none', - 'frontend_provider': open_webui.get('provider', 'auto'), - 'route_count': len(profile.get('routes', {}) or {}), - 'description': profile.get('description', ''), - # Old fields kept so older list formatting doesn't crash. - 'base_model': '', - 'logical_model_name': '', - 'served_model_name': '', - 'protocol_mode': 'stack', - 'engine': ','.join(providers) or 'none', - 'resource_profile': '', - 'tags': profile.get('tags', []), - } diff --git a/infer_stack/cli/__init__.py b/infer_stack/cli/__init__.py index 00a11bde..705b847e 100644 --- a/infer_stack/cli/__init__.py +++ b/infer_stack/cli/__init__.py @@ -8,23 +8,20 @@ be invoked from the shell (``infer-stack render --profile X``) or from Python (``RenderCLI.main(argv=False, profile='X')``). -This package was split out of a single ``cli.py`` module. The layers are: +The layers are: -* ``context`` — path/config/env/override/plan helpers (no other cli deps). +* ``context`` — path/config/env/override helpers (no other cli deps). * ``probes`` — pure readiness/model-selection helpers over a deployment. -* ``compose`` — compose + LiteLLM + preflight + diagnostics helpers. +* ``compose`` — compose + LiteLLM helpers. * ``options`` — shared ``DataConfig`` mixins for override flags. -* ``commands_profile`` — profile/config management subcommands. -* ``commands_runtime`` — up/down/deploy/status/env + day-2-ops wrappers. -* ``commands_smoke`` — diagnose/wait/smoke-test/benchmark subcommands. - -``infer_stack.cli`` re-exports the previously top-level names so existing -``from infer_stack.cli import ...`` imports keep working. +* ``commands_catalog`` — catalog editor (models/endpoints/hosts/bundles). +* ``commands_leasing`` — acquire/release/run/leases/test + reconcile. +* ``commands_runtime`` — ``status`` + ``stack`` day-2 compose wrappers. +* ``commands_meta`` — version/help/config introspection. """ from __future__ import annotations -import requests # noqa: F401 (cli_mod.requests is patched in tests) import scriptconfig as scfg from .. import __version__ @@ -32,92 +29,41 @@ # Keep submodules importable as attributes (e.g. ``infer_stack.cli.commands_runtime``) # so tests can patch seams where they are actually looked up. from . import ( # noqa: F401 + commands_catalog, + commands_leasing, commands_meta, - commands_profile, commands_runtime, - commands_smoke, - compose, context, options, - probes, -) -from ..kubeai_ops import ( # noqa: F401 - CommandError, - deploy_rendered_artifacts, - print_status as kubeai_print_status, -) -from .commands_profile import ( - DescribeProfileCLI, - ExplainCLI, - InitCLI, - KubeaiSyncResourceProfilesCLI, - ListModelsCLI, - ListProfilesCLI, - LockCLI, - RenderCLI, - ResolveCLI, - SetupCLI, - SwitchCLI, - ValidateCLI, - VerifyProfileCLI, ) -from .commands_runtime import ( - DeployCLI, - DownCLI, - EnvCLI, - LogsCLI, - OllamaListCLI, - OllamaPsCLI, - OllamaPullCLI, - PsCLI, - PullCLI, - PurgeCLI, - RestartCLI, - StartCLI, - StatusCLI, - StopCLI, - UpCLI, +from .commands_catalog import CatalogModalCLI +from .commands_leasing import ( + AcquireCLI, + ApplyCLI, + EvictCLI, + LeasesCLI, + ReleaseCLI, + RenewCLI, + RunCLI, + TestCLI, + TuiCLI, + WaitCLI, ) +from .commands_leasing import EnvCLI as LeasingEnvCLI +from .commands_leasing import RenderCLI as LeasingRenderCLI from .commands_meta import ( ConfigModalCLI, ConfigPathsCLI, + HelpModalCLI, VersionCLI, ) -from .commands_smoke import ( - BenchmarkCLI, - DiagnoseCLI, - SmokeTestCLI, - WaitReadyCLI, -) -from .compose import ( # noqa: F401 - _compose_has_service, - _litellm_delete_missed_config_model, -) -from .context import ( # noqa: F401 - apply_config_overrides, - backend_name, - build_plan, - config_for_runtime, - config_path, - effective_allow_unsupported, - effective_inventory, - ensure_renderable, - generated_dir, - has_runtime_overrides, - kubeai_generated_dir, - load_config, - models_path, - plan_path, - render_is_stale, - runtime_dir_for_config, - save_plan, -) -from .probes import ( # noqa: F401 - _default_model_for_deployment, - _resolve_smoke_protocol_from_deployment, +from .commands_runtime import ( + LogsCLI, + PsCLI, + StackModalCLI, + StatusCLI, ) - # --------------------------------------------------------------------------- # Modal CLI + entry point # --------------------------------------------------------------------------- @@ -125,10 +71,30 @@ class ManageCLI(scfg.ModalCLI): description = ( - 'Render and run vLLM serving profiles through the Compose or KubeAI ' - 'backends. Primary workflow: setup -> render -> up (or deploy).' + 'Lease, acquire, and run LLM endpoints. Primary workflow: ' + 'catalog -> acquire/run. See `infer-stack help tree`.' ) + __epilog__ = """ + Quickstart: + infer-stack config init # storage + default backend + infer-stack catalog init # start a model catalog + infer-stack catalog model add smol135 \\ + --source hf://HuggingFaceTB/SmolLM2-135M-Instruct + infer-stack catalog endpoint add --model smol135 # -> smol135-1 + infer-stack acquire smol135-1 # render + bring up + wait + infer-stack leases # what is desired vs running + infer-stack test smol135-1 # one real generation + infer-stack release --all # tear it back down + + Mental model: + catalog (what can run) -> acquire (declare intent, a lease; --ttl for a + soft TTL) -> the controller reconciles a compose project (gateway + + models + Open WebUI) onto your GPUs. `acquire --no-apply` writes that + project without starting it (then `apply`); `leases` shows desired vs + actual. Run `infer-stack help tree` for the whole command surface. + """ + # Backs the modal ``--version`` flag (scriptconfig reads ``__version__``). # The ``version`` *subcommand* is registered below under a non-colliding # attribute name; its CLI name comes from ``VersionCLI.__command__``. @@ -136,45 +102,44 @@ class ManageCLI(scfg.ModalCLI): # Meta / introspection version_command = VersionCLI + help = HelpModalCLI # `infer-stack help tree` — the whole surface at a glance config = ConfigModalCLI - - # Config / profile management - setup = SetupCLI - init = InitCLI - resolve = ResolveCLI - validate = ValidateCLI - lock = LockCLI - render = RenderCLI - switch = SwitchCLI - list_models = ListModelsCLI - list_profiles = ListProfilesCLI - explain = ExplainCLI - describe_profile = DescribeProfileCLI - verify_profile = VerifyProfileCLI - kubeai_sync_resource_profiles = KubeaiSyncResourceProfilesCLI - - # Runtime - up = UpCLI - down = DownCLI - purge = PurgeCLI - deploy = DeployCLI + paths = ConfigPathsCLI # top-level alias for `config paths` status = StatusCLI - env = EnvCLI - diagnose = DiagnoseCLI - wait_ready = WaitReadyCLI - smoke_test = SmokeTestCLI - benchmark = BenchmarkCLI - ollama_pull = OllamaPullCLI - ollama_list = OllamaListCLI - ollama_ps = OllamaPsCLI - - # Compose day-2-ops wrappers + + # Catalog editor (models / endpoints / hosts / bundles — no raw YAML) + catalog = CatalogModalCLI + + # Leasing model (acquire/release/run + status) + acquire = AcquireCLI # stand up endpoints: lease + up + wait (--ttl for soft TTL) + release = ReleaseCLI + evict = EvictCLI # force-tear-down released (idle) models to free GPUs + renew = RenewCLI + run = RunCLI + # Reconcile primitives (lease-free): render desired -> disk, apply disk -> up + render = LeasingRenderCLI # write the compose project for desired, no `up` + apply = ApplyCLI # bring the desired set up (the trigger for acquire --no-apply) + wait = WaitCLI # block until endpoints are ready (acquire --no-wait fan-out) + leases = LeasesCLI + tui = TuiCLI # live Textual monitor + controls (opt-in: infer-stack[tui]) + test = TestCLI # smoke-test a served endpoint through the front door + env = LeasingEnvCLI # managed env-file: path / read KEY / set KEY=VALUE + + # Day-2 ops on the running stack, grouped under `stack`; logs/ps also kept + # at the top level as the two hottest convenience aliases. + stack = StackModalCLI logs = LogsCLI ps = PsCLI - restart = RestartCLI - pull = PullCLI - start = StartCLI - stop = StopCLI + + +def __getattr__(name: str): + # ``requests`` is imported lazily so a bare ``infer-stack --help`` doesn't + # pay for it. Tests still patch the seam as ``cli_mod.requests`` — resolving + # the attribute imports the (shared) module so the patch is global. + if name == 'requests': + import requests + return requests + raise AttributeError(f'module {__name__!r} has no attribute {name!r}') def main(argv=None) -> int: diff --git a/infer_stack/cli/commands_catalog.py b/infer_stack/cli/commands_catalog.py new file mode 100644 index 00000000..01f01821 --- /dev/null +++ b/infer_stack/cli/commands_catalog.py @@ -0,0 +1,788 @@ +"""`infer-stack catalog …` — edit the user serving catalog without raw YAML. + +The catalog (``config_root()/catalog.yaml`` by default) is the durable list of +models / endpoints / runtime-hosts / bundles the leasing verbs (``acquire`` / +``run``) read. Hand-editing YAML is the current workflow; this +submodal adds a flag-driven editor with a validating writer, so a bad edit is +rejected before it can land (the same :class:`~infer_stack.leasing.catalog.Catalog` +parser the leasing path uses). + +Grammar is noun-verb: ``catalog model add``, ``catalog endpoint rm``, etc. +""" + +from __future__ import annotations + +import sys +from pathlib import Path +from typing import Any + +import scriptconfig as scfg +import yaml + +from ..leasing import Catalog, CatalogError +from ..paths import config_root +from .context import _apply_path_overrides +from .options import ( + _AllowedGpusMixin, + _DisplayGpuMixin, + _PathOverridesMixin, + _SimulateHardwareMixin, +) + +SECTIONS = ('models', 'endpoints', 'runtime_hosts', 'bundles') + + +def _print_yaml(text: str) -> None: + """Print YAML, syntax-highlighted when stdout is a terminal. + + Falls back to a plain ``print`` when piped/redirected so downstream + parsers and tests get clean, color-free output. + """ + if not sys.stdout.isatty(): + print(text, end='') + return + try: + from rich.console import Console + from rich.syntax import Syntax + except ImportError: + print(text, end='') + return + syntax = Syntax( + text.rstrip('\n'), 'yaml', theme='ansi_dark', background_color='default' + ) + Console().print(syntax) + + +# --------------------------------------------------------------------------- +# raw catalog load / mutate / save +# --------------------------------------------------------------------------- + + +def _catalog_path(config) -> Path: + _apply_path_overrides(config) + raw = getattr(config, 'catalog', None) + if raw: + return Path(raw).expanduser() + return config_root() / 'catalog.yaml' + + +def _load_raw(path: Path) -> dict[str, Any]: + data = yaml.safe_load(path.read_text()) if path.exists() else {} + data = data or {} + for section in SECTIONS: + data.setdefault(section, {}) + return data + + +def _validate(data: dict[str, Any]) -> None: + """Refuse to persist a catalog the leasing path would reject.""" + try: + Catalog.from_dict(data) + except CatalogError as ex: + raise SystemExit(f'refusing to write an invalid catalog: {ex}') + + +def _save_raw(path: Path, data: dict[str, Any], *, dry_run: bool = False) -> None: + # Drop empty sections for a tidy file. + out = {k: v for k, v in data.items() if v or k == 'models'} + text = yaml.safe_dump(out, sort_keys=False, default_flow_style=False) + if dry_run: + _print_yaml(text) + return + _validate(data) + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + '.tmp') + tmp.write_text(text) + tmp.replace(path) + + +def _exists_guard(data, section, name, force, hint='') -> None: + if name in data[section] and not force: + raise SystemExit( + f"{section[:-1]} '{name}' already exists; " + f'pass --force to overwrite{hint}' + ) + + +def _next_indexed_name(existing, base: str) -> str: + """First free ``{base}-{N}`` (N starting at 1) not already in ``existing``. + + Defaulted endpoint names get a numeric suffix so repeated ``endpoint add`` + for the same model don't collide — they accumulate as ``base-1``, ``base-2``, + … and the first is deterministically ``base-1``. + """ + n = 1 + while f'{base}-{n}' in existing: + n += 1 + return f'{base}-{n}' + + +def _slug_alias(text: str) -> str: + """Make ``text`` safe to use as an endpoint alias. + + An endpoint name doubles as the LiteLLM ``model_name`` (what clients ask for + and what Open WebUI shows) and as a CLI-typed token, so keep it shell/URL + friendly: collapse model/tag separators (``/``, ``:``) and any other + non-``[A-Za-z0-9._-]`` runs to a single ``-``. + """ + import re + + out = re.sub(r'[^A-Za-z0-9._-]+', '-', text).strip('-') + return out or text + + +def _rm(config, section, names) -> int: + names = [names] if isinstance(names, str) else list(names or []) + if not names: + raise SystemExit(f'{section[:-1]} rm: give at least one name') + path = _catalog_path(config) + data = _load_raw(path) + missing = [n for n in names if n not in data[section]] + if missing: + raise SystemExit( + f'{section[:-1]}(s) not found in {path}: {", ".join(missing)}' + ) + for name in names: + del data[section][name] + _save_raw(path, data, dry_run=getattr(config, 'dry_run', False)) + for name in names: + print(f"removed {section[:-1]} '{name}'") + return 0 + + +def _list(config, section) -> int: + data = _load_raw(_catalog_path(config)) + names = sorted(data[section]) + print('\n'.join(names) if names else f'(no {section})') + return 0 + + +def _show(config, section, name) -> int: + data = _load_raw(_catalog_path(config)) + entries = data.get(section) or {} + # No name -> show every entry in the section (the whole `endpoints:` block), + # rather than erroring on a `None` lookup. + if name is None: + if not entries: + print(f'(no {section})') + return 0 + _print_yaml(yaml.safe_dump({section: entries}, sort_keys=False)) + return 0 + if name not in entries: + have = f" (have: {', '.join(sorted(entries))})" if entries else '' + raise SystemExit(f"{section[:-1]} '{name}' not found{have}") + _print_yaml(yaml.safe_dump({name: entries[name]}, sort_keys=False)) + return 0 + + +def _parse_kv(items) -> dict[str, Any]: + """Parse ``KEY=VALUE`` pairs; values are YAML-typed (ints/floats/bools).""" + out: dict[str, Any] = {} + for item in items or []: + if '=' not in item: + raise SystemExit(f"expected KEY=VALUE, got {item!r}") + key, _, val = item.partition('=') + out[key.strip()] = yaml.safe_load(val) + return out + + +# --------------------------------------------------------------------------- +# catalog-level commands: init / path / show / validate / edit +# --------------------------------------------------------------------------- + + +class _CatalogCommon(_PathOverridesMixin): + catalog = scfg.Value(None, type=str, help='Catalog path (default: config dir).') + dry_run = scfg.Value( + False, isflag=True, help='Print the resulting YAML, do not write.' + ) + + +class CatalogInitCLI(_CatalogCommon): + """Write a starter catalog.yaml (empty sections) if none exists.""" + + __command__ = 'init' + force = scfg.Value(False, isflag=True, help='Overwrite an existing catalog.') + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + path = _catalog_path(config) + if path.exists() and not config.force: + raise SystemExit(f'{path} already exists; pass --force to reset') + _save_raw(path, {s: {} for s in SECTIONS}, dry_run=config.dry_run) + if not config.dry_run: + print(f'wrote starter catalog -> {path}') + return 0 + + +class CatalogSuggestCLI( + _PathOverridesMixin, + _SimulateHardwareMixin, + _AllowedGpusMixin, + _DisplayGpuMixin, +): + """Seed a catalog from server introspection (the suggestion pool). + + Introspects the GPU inventory and joins it against the curated suggestion + pool (``templates/suggestion-pool.yaml``) to derive a concrete, fits-this-box + set of models + endpoints — sized ``max_model_len`` / ``gpu_memory_utilization``, + ``--dtype=half`` on pre-Ampere GPUs, the largest model kept warm. It mirrors + the hand-tuning of the leasing demo, but derived rather than guessed. + + Render then apply (a plan/apply split): by default it *renders* the + suggestion as YAML to stdout and writes nothing — so you can review it, pipe + it (``> catalog.yaml``), or trim it. Pass ``--apply`` to merge it into your + catalog (additive: existing entries are kept unless ``--force``). + + Pure + offline: ``--simulate-hardware 2x80`` suggests for hardware you do not + have in front of you. + + infer-stack catalog suggest # render only (no write) + infer-stack catalog suggest --simulate-hardware 4x48 + infer-stack catalog suggest --apply # merge into the catalog + """ + + __command__ = 'suggest' + catalog = scfg.Value(None, type=str, help='Catalog path (default: config dir).') + apply = scfg.Value( + False, isflag=True, + help='Merge the suggestion into the catalog (default: render only).', + ) + force = scfg.Value( + False, isflag=True, + help='With --apply, overwrite catalog entries that already exist.', + ) + + @classmethod + def main(cls, argv=True, **kwargs): + from ..hardware import detect_inventory + from ..leasing.suggest import suggest_catalog + from .commands_leasing import _resolve_skip_display + from .context import effective_inventory + + config = cls.cli(argv=argv, data=kwargs) + inventory = effective_inventory(config) or detect_inventory() + gpus = inventory.get('gpus') or [] + skip_display = _resolve_skip_display(config) + frag = suggest_catalog( + inventory, reserve_display_gpu='auto' if skip_display else False + ) + + max_mem = max((g.get('memory_gib') or 0 for g in gpus), default=0) + n_display = sum(1 for g in gpus if g.get('display_active')) + usable = len(gpus) - (n_display if skip_display else 0) + hw = ( + f'{len(gpus)} GPU(s), max {max_mem:g} GiB' + + (f', skipping {n_display} display-attached' if skip_display and n_display else '') + ) + + if not frag['models']: + print( + f'no pooled model fits the detected hardware ({hw}). ' + 'Pass --simulate-hardware NxM to plan for a bigger box, or add ' + 'models by hand with `catalog model add`.', + file=sys.stderr, + ) + return 0 + + text = yaml.safe_dump(frag, sort_keys=False, default_flow_style=False) + + if not config.apply: + print( + f'# suggested for: {hw} ({usable} usable)\n' + f'# {len(frag["models"])} model(s); review, then re-run with ' + f'--apply to merge (or redirect this to a catalog.yaml).', + file=sys.stderr, + ) + _print_yaml(text) + return 0 + + # --apply: additive merge into the catalog (keep existing entries). + path = _catalog_path(config) + data = _load_raw(path) + added: list[str] = [] + skipped: list[str] = [] + for section in ('models', 'endpoints'): + for name, value in frag[section].items(): + if name in data[section] and not config.force: + skipped.append(f'{section[:-1]}:{name}') + continue + data[section][name] = value + added.append(f'{section[:-1]}:{name}') + _save_raw(path, data, dry_run=False) + print(f'merged suggestion into {path} ({hw})') + if added: + print(f' added: {", ".join(added)}') + if skipped: + print( + f' kept existing (pass --force to overwrite): ' + f'{", ".join(skipped)}' + ) + return 0 + + +class CatalogPathCLI(_PathOverridesMixin): + """Print the catalog path.""" + + __command__ = 'path' + catalog = scfg.Value(None, type=str) + + @classmethod + def main(cls, argv=True, **kwargs): + print(_catalog_path(cls.cli(argv=argv, data=kwargs))) + return 0 + + +class CatalogShowCLI(_PathOverridesMixin): + """Pretty-print the whole catalog (or one named entry across sections).""" + + __command__ = 'show' + catalog = scfg.Value(None, type=str) + name = scfg.Value(None, position=1, type=str, help='Optional entry name.') + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + data = _load_raw(_catalog_path(config)) + if config.name: + hits = { + s: data[s][config.name] + for s in SECTIONS + if config.name in data[s] + } + if not hits: + raise SystemExit(f"'{config.name}' not found in any section") + _print_yaml(yaml.safe_dump(hits, sort_keys=False)) + else: + _print_yaml(yaml.safe_dump(data, sort_keys=False)) + return 0 + + +class CatalogValidateCLI(_PathOverridesMixin): + """Parse + cross-reference check the catalog.""" + + __command__ = 'validate' + catalog = scfg.Value(None, type=str) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + path = _catalog_path(config) + if not path.exists(): + raise SystemExit(f'no catalog at {path}') + try: + cat = Catalog.load(path) + except CatalogError as ex: + raise SystemExit(f'invalid: {ex}') + print( + f'ok: {len(cat.models)} model(s), {len(cat.endpoints)} endpoint(s), ' + f'{len(cat.hosts)} host(s), {len(cat.bundles)} bundle(s)' + ) + return 0 + + +class CatalogEditCLI(_PathOverridesMixin): + """Open the catalog in $EDITOR (escape hatch), then validate it.""" + + __command__ = 'edit' + catalog = scfg.Value(None, type=str) + + @classmethod + def main(cls, argv=True, **kwargs): + import os + import subprocess + + config = cls.cli(argv=argv, data=kwargs) + path = _catalog_path(config) + if not path.exists(): + _save_raw(path, {s: {} for s in SECTIONS}) + editor = os.environ.get('EDITOR', 'vi') + subprocess.run([*editor.split(), str(path)], check=False) + try: + Catalog.load(path) + except CatalogError as ex: + print(f'warning: catalog is invalid after edit: {ex}') + return 1 + return 0 + + +# --------------------------------------------------------------------------- +# model {add,list,show,rm} +# --------------------------------------------------------------------------- + + +class ModelAddCLI(_CatalogCommon): + """Add (or --force overwrite) a model: a Hugging Face / local weight source.""" + + __command__ = 'add' + name = scfg.Value(None, position=1, type=str) + source = scfg.Value(None, type=str, help='e.g. hf://org/Model or a path.') + revision = scfg.Value(None, type=str) + quantization = scfg.Value(None, type=str) + dtype = scfg.Value(None, type=str) + force = scfg.Value(False, isflag=True) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + if not config.name or not config.source: + raise SystemExit('model add: NAME and --source are required') + path = _catalog_path(config) + data = _load_raw(path) + _exists_guard(data, 'models', config.name, config.force) + entry: dict[str, Any] = {'source': config.source} + for key in ('revision', 'quantization', 'dtype'): + if getattr(config, key) is not None: + entry[key] = getattr(config, key) + data['models'][config.name] = entry + _save_raw(path, data, dry_run=config.dry_run) + if not config.dry_run: + print(f"added model '{config.name}'") + return 0 + + +class ModelListCLI(_PathOverridesMixin): + """List model names.""" + __command__ = 'list' + catalog = scfg.Value(None, type=str) + + @classmethod + def main(cls, argv=True, **kwargs): + return _list(cls.cli(argv=argv, data=kwargs), 'models') + + +class ModelShowCLI(_PathOverridesMixin): + """Show a model entry, or all of them when no NAME is given.""" + __command__ = 'show' + catalog = scfg.Value(None, type=str) + name = scfg.Value(None, position=1, type=str, + help='Model name; omit to show every model.') + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + return _show(config, 'models', config.name) + + +class ModelRmCLI(_CatalogCommon): + """Remove one or more models by name.""" + __command__ = 'rm' + names = scfg.Value([], nargs='+', position=1, type=str, + help='Model name(s) to remove.') + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + return _rm(config, 'models', config.names) + + +class CatalogModelCLI(scfg.ModalCLI): + """Manage catalog models.""" + __command__ = 'model' + add = ModelAddCLI + list = ModelListCLI + show = ModelShowCLI + rm = ModelRmCLI + + +# --------------------------------------------------------------------------- +# endpoint {add,list,show,rm} +# --------------------------------------------------------------------------- + + +class EndpointAddCLI(_CatalogCommon): + """Add (or --force overwrite) an endpoint — the served API name. + + ``NAME`` is optional: when omitted it defaults to ``{model}-{N}`` (the vLLM + model name, or the Ollama tag, slugified, with an auto-incrementing suffix — + ``smol135-1``, ``smol135-2``, …). That keeps the served alias — what you ask + for and what Open WebUI shows — tied to the model, and a repeated add for one + model just gets the next index instead of colliding. Give an explicit + ``NAME`` when you want a stable alias decoupled from the model (e.g. ``chat`` + you can re-point). + """ + + __command__ = 'add' + name = scfg.Value( + None, position=1, type=str, + help='Endpoint alias (default: {model}-N, auto-incrementing).', + ) + engine = scfg.Value('vllm', choices=['vllm', 'ollama']) + model = scfg.Value(None, type=str, help='Model name (vllm) or tag (ollama).') + host = scfg.Value(None, type=str, help='Runtime host (ollama).') + public_name = scfg.Value( + None, type=str, help='Served/public name (for coalescing aliases).' + ) + reclaim = scfg.Value( + None, choices=['keep-warm', 'stop', 'scale-to-zero'], + help='Reclaim policy when idle.', + ) + # vLLM runtime conveniences + max_model_len = scfg.Value(None, type=int) + gpu_mem = scfg.Value( + None, type=float, help='gpu_memory_utilization (0-1).' + ) + tensor_parallel = scfg.Value(None, type=int) + extra_args = scfg.Value( + None, type=str, + help="Raw vLLM flags as one string (shell-split), " + "e.g. --extra-args='--dtype=half --enforce-eager'.", + ) + runtime = scfg.Value( + [], nargs='*', type=str, + help='Extra runtime KEY=VALUE pairs (YAML-typed).', + ) + force = scfg.Value(False, isflag=True) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + path = _catalog_path(config) + data = _load_raw(path) + if config.name: + name = config.name + _exists_guard(data, 'endpoints', name, config.force) + else: + if not config.model: + raise SystemExit( + 'endpoint add: give a NAME, or --model to derive the name ' + 'from it' + ) + # Default name = {model}-N, auto-incrementing so repeated adds for + # one model accumulate (smol135-1, smol135-2, …) instead of colliding. + name = _next_indexed_name( + data['endpoints'], _slug_alias(config.model) + ) + entry: dict[str, Any] = {'engine': config.engine} + if config.model: + entry['model'] = config.model + if config.host: + entry['host'] = config.host + if config.public_name: + entry['public_name'] = config.public_name + runtime: dict[str, Any] = _parse_kv(config.runtime) + if config.max_model_len is not None: + runtime['max_model_len'] = config.max_model_len + if config.gpu_mem is not None: + runtime['gpu_memory_utilization'] = config.gpu_mem + if config.tensor_parallel is not None: + runtime['tensor_parallel_size'] = config.tensor_parallel + if config.extra_args: + import shlex + runtime['extra_args'] = shlex.split(config.extra_args) + if runtime: + entry['runtime'] = runtime + if config.reclaim: + entry['reclaim'] = {'policy': config.reclaim} + data['endpoints'][name] = entry + _save_raw(path, data, dry_run=config.dry_run) + if not config.dry_run: + model_note = f' -> {config.model}' if config.model else '' + print(f"added endpoint '{name}'{model_note}") + return 0 + + +class EndpointListCLI(_PathOverridesMixin): + """List endpoint names.""" + __command__ = 'list' + catalog = scfg.Value(None, type=str) + + @classmethod + def main(cls, argv=True, **kwargs): + return _list(cls.cli(argv=argv, data=kwargs), 'endpoints') + + +class EndpointShowCLI(_PathOverridesMixin): + """Show an endpoint entry, or all of them when no NAME is given.""" + __command__ = 'show' + catalog = scfg.Value(None, type=str) + name = scfg.Value(None, position=1, type=str, + help='Endpoint name; omit to show every endpoint.') + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + return _show(config, 'endpoints', config.name) + + +class EndpointRmCLI(_CatalogCommon): + """Remove one or more endpoints by name.""" + __command__ = 'rm' + names = scfg.Value([], nargs='+', position=1, type=str, + help='Endpoint name(s) to remove.') + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + return _rm(config, 'endpoints', config.names) + + +class CatalogEndpointCLI(scfg.ModalCLI): + """Manage catalog endpoints.""" + __command__ = 'endpoint' + add = EndpointAddCLI + list = EndpointListCLI + show = EndpointShowCLI + rm = EndpointRmCLI + + +# --------------------------------------------------------------------------- +# host {add,list,rm} (runtime_hosts — ollama daemons) +# --------------------------------------------------------------------------- + + +class HostAddCLI(_CatalogCommon): + """Add (or --force overwrite) a runtime host (e.g. an Ollama daemon).""" + + __command__ = 'add' + name = scfg.Value(None, position=1, type=str) + engine = scfg.Value('ollama', choices=['ollama']) + gpu = scfg.Value([], nargs='*', type=int, help='GPU index/indices.') + keep_alive = scfg.Value(None, type=str, help='Ollama keep_alive, e.g. 5m.') + num_parallel = scfg.Value(None, type=int) + max_loaded_models = scfg.Value(None, type=int) + context_length = scfg.Value(None, type=int) + image = scfg.Value(None, type=str) + force = scfg.Value(False, isflag=True) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + if not config.name: + raise SystemExit('host add: NAME is required') + path = _catalog_path(config) + data = _load_raw(path) + _exists_guard(data, 'runtime_hosts', config.name, config.force) + entry: dict[str, Any] = {'engine': config.engine} + if config.gpu: + entry['placement'] = {'gpu_indices': list(config.gpu)} + settings: dict[str, Any] = {} + for flag, key in ( + ('keep_alive', 'keep_alive'), + ('num_parallel', 'num_parallel'), + ('max_loaded_models', 'max_loaded_models'), + ('context_length', 'context_length'), + ): + if getattr(config, flag) is not None: + settings[key] = getattr(config, flag) + if settings: + entry['settings'] = settings + if config.image: + entry['image'] = config.image + data['runtime_hosts'][config.name] = entry + _save_raw(path, data, dry_run=config.dry_run) + if not config.dry_run: + print(f"added host '{config.name}'") + return 0 + + +class HostListCLI(_PathOverridesMixin): + """List runtime-host names.""" + __command__ = 'list' + catalog = scfg.Value(None, type=str) + + @classmethod + def main(cls, argv=True, **kwargs): + return _list(cls.cli(argv=argv, data=kwargs), 'runtime_hosts') + + +class HostRmCLI(_CatalogCommon): + """Remove one or more runtime hosts by name.""" + __command__ = 'rm' + names = scfg.Value([], nargs='+', position=1, type=str, + help='Runtime-host name(s) to remove.') + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + return _rm(config, 'runtime_hosts', config.names) + + +class CatalogHostCLI(scfg.ModalCLI): + """Manage runtime hosts (Ollama daemons / placement).""" + __command__ = 'host' + add = HostAddCLI + list = HostListCLI + rm = HostRmCLI + + +# --------------------------------------------------------------------------- +# bundle {add,list,rm} +# --------------------------------------------------------------------------- + + +class BundleAddCLI(_CatalogCommon): + """Add (or --force overwrite) a bundle: a named group of endpoints.""" + + __command__ = 'add' + name = scfg.Value(None, position=1, type=str) + members = scfg.Value([], nargs='*', position=2, type=str) + force = scfg.Value(False, isflag=True) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + members = list(config.members or []) + if not config.name or not members: + raise SystemExit('bundle add: NAME and at least one endpoint required') + path = _catalog_path(config) + data = _load_raw(path) + _exists_guard(data, 'bundles', config.name, config.force) + data['bundles'][config.name] = members + _save_raw(path, data, dry_run=config.dry_run) + if not config.dry_run: + print(f"added bundle '{config.name}' -> {', '.join(members)}") + return 0 + + +class BundleListCLI(_PathOverridesMixin): + """List bundle names.""" + __command__ = 'list' + catalog = scfg.Value(None, type=str) + + @classmethod + def main(cls, argv=True, **kwargs): + return _list(cls.cli(argv=argv, data=kwargs), 'bundles') + + +class BundleRmCLI(_CatalogCommon): + """Remove one or more bundles by name.""" + __command__ = 'rm' + names = scfg.Value([], nargs='+', position=1, type=str, + help='Bundle name(s) to remove.') + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + return _rm(config, 'bundles', config.names) + + +class CatalogBundleCLI(scfg.ModalCLI): + """Manage endpoint bundles.""" + __command__ = 'bundle' + add = BundleAddCLI + list = BundleListCLI + rm = BundleRmCLI + + +# --------------------------------------------------------------------------- +# top-level catalog modal +# --------------------------------------------------------------------------- + + +class CatalogModalCLI(scfg.ModalCLI): + """Edit the user serving catalog (models / endpoints / hosts / bundles).""" + + __command__ = 'catalog' + + init = CatalogInitCLI + suggest = CatalogSuggestCLI + path = CatalogPathCLI + show = CatalogShowCLI + validate = CatalogValidateCLI + edit = CatalogEditCLI + model = CatalogModelCLI + endpoint = CatalogEndpointCLI + host = CatalogHostCLI + bundle = CatalogBundleCLI diff --git a/infer_stack/cli/commands_leasing.py b/infer_stack/cli/commands_leasing.py new file mode 100644 index 00000000..b5168e13 --- /dev/null +++ b/infer_stack/cli/commands_leasing.py @@ -0,0 +1,1478 @@ +"""Leasing subcommands: the acquire/release/run surface for the new model. + +These verbs drive the leasing controller (ledger + backend) instead of the +legacy "active profile". They are what a kwdagger pipeline node uses to request +the models it needs, block until ready, and release after: + + infer-stack acquire qwen-coder reranker --ttl 2h --env-file is.env + infer-stack run --endpoint qwen-coder -- python my_node.py + infer-stack release --env-file is.env + infer-stack acquire qwen-coder # standing service (no --ttl) + infer-stack leases # status of leases + deployment deployments + +Until the Compose/KubeAI backends land, the default ``--backend null`` is a +dry-run: the ledger does all the real bookkeeping (coalescing, demand, TTL) but +nothing is actually served. This lets the whole surface — including env-file +emission and the ``run`` wrapper — be exercised end to end without docker. +""" + +from __future__ import annotations + +import json +import os +import shlex +import subprocess +from pathlib import Path + +import scriptconfig as scfg + +from ..env_utils import parse_env_file, write_env_file +from ..leasing import ( + Catalog, + CatalogError, + ComposeBackend, + Controller, + DeploymentState, + LeaseState, + Ledger, + NullBackend, + Sharing, + SqliteStore, + default_ledger_path, +) +from ..leasing.envfile import ( + build_descriptor, + descriptor_env, + read_lease_id, + render_env_file, +) +from ..paths import config_root, data_root +from .context import _apply_path_overrides +from .options import _AllowedGpusMixin, _DisplayGpuMixin, _PathOverridesMixin + +# --------------------------------------------------------------------------- +# helpers +# --------------------------------------------------------------------------- + + +def _default_owner() -> str: + import getpass + + try: + return getpass.getuser() + except Exception: + return 'unknown' + + +def _parse_duration(text) -> float | None: + """Parse ``2h`` / ``30m`` / ``90s`` / ``1d`` / bare seconds; None = infinite.""" + if text is None: + return None + text = str(text).strip().lower() + if text in ('', 'none', 'inf', 'infinite', '0'): + return None + units = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400} + if text[-1] in units: + return float(text[:-1]) * units[text[-1]] + return float(text) + + +def _collect_names(value) -> list[str]: + """Normalize positional/flag names: accept a list or comma-separated str.""" + if value is None: + return [] + items = [value] if isinstance(value, str) else list(value) + out: list[str] = [] + for item in items: + for part in str(item).split(','): + part = part.strip() + if part: + out.append(part) + return out + + +def _parse_gpus(text) -> list[int] | None: + if text in (None, ''): + return None + return [int(p) for p in str(text).split(',') if p.strip() != ''] + + +def _coerce_bool(value, default: bool) -> bool: + """Interpret a setting/flag as a bool (None -> default).""" + if value is None: + return default + if isinstance(value, bool): + return value + return str(value).strip().lower() in ('1', 'true', 'yes', 'on') + + +def _resolve_ui(config) -> bool: + """Open WebUI on/off: explicit --ui/--no-ui wins, else `config set ui`, + else on by default.""" + from ..paths import get_setting + + flag = getattr(config, 'ui', None) + if flag is not None: + return bool(flag) + return _coerce_bool(get_setting('ui'), True) + + +def _resolve_litellm(config) -> bool: + """LiteLLM gateway on/off: explicit --litellm/--no-litellm wins, else + `config set litellm`, else on by default. + + With the gateway off there is no single OpenAI ``base_url`` fronting every + alias; Open WebUI (if on) then talks to the rendered upstreams directly — + handy for a lean ``ollama + Open WebUI`` stack that manages its own models. + """ + from ..paths import get_setting + + flag = getattr(config, 'litellm', None) + if flag is not None: + return bool(flag) + return _coerce_bool(get_setting('litellm'), True) + + +def _resolve_reverse_proxy(config) -> tuple[bool, int, str | None]: + """Resolve the reverse-proxy front door: (enabled, port, byo_config_path). + + The ``reverse_proxy`` setting may be a bool (enable with defaults) or a + ``{enabled, port, config_path}`` block (via `config edit`); the + ``--reverse-proxy`` flag overrides the enabled bit. + """ + from ..paths import get_setting + + raw = get_setting('reverse_proxy') + block = raw if isinstance(raw, dict) else {} + flag = getattr(config, 'reverse_proxy', None) + if flag is not None: + enabled = bool(flag) + elif isinstance(raw, dict): + enabled = _coerce_bool(block.get('enabled'), False) + else: + enabled = _coerce_bool(raw, False) + port = int(block.get('port') or 80) + return enabled, port, block.get('config_path') + + +def _resolve_skip_display(config) -> bool: + """Whether to skip display-attached GPUs during placement. + + Off by default — placement uses every GPU, so a single-GPU host (where the + only GPU drives the display) works without ceremony. Opt in with the + ``--skip-display-gpus`` flag (wins) or `config set skip_display_gpus true`. + """ + from ..paths import get_setting + + flag = getattr(config, 'skip_display_gpus', None) + if flag is not None: + return bool(flag) + return _coerce_bool(get_setting('skip_display_gpus'), False) + + +def _resolve_assume_yes(config, *, interactive: bool) -> bool: + """Whether to apply compose changes without the diff prompt. + + Only the additive verb (``acquire``) prompts, and only on a real terminal + without ``--yes``. Everything else (release/leases/run/non-TTY) auto-applies. + """ + import sys + + if not interactive: + return True + if getattr(config, 'yes', False): + return True + return not sys.stdout.isatty() + + +def _make_backend(config, *, interactive: bool = False): + from ..paths import get_setting + + # Resolve the backend: explicit --backend wins, else the persisted default + # (`config set backend compose`), else the dry-run null backend. + name = getattr(config, 'backend', None) or get_setting('backend') or 'null' + if name in (None, '', 'null', 'dry-run'): + return NullBackend() + if name == 'compose': + from ..hardware import detect_inventory + + rp_enabled, rp_port, rp_config = _resolve_reverse_proxy(config) + return ComposeBackend( + state_dir=data_root() / 'leasing' / 'compose', + inventory=detect_inventory(), + allowed_gpus=_parse_gpus(getattr(config, 'allowed_gpus', None)), + skip_display=_resolve_skip_display(config), + litellm=_resolve_litellm(config), + ui=_resolve_ui(config), + reverse_proxy=rp_enabled, + reverse_proxy_port=rp_port, + reverse_proxy_config=rp_config, + require_generation=bool(getattr(config, 'require_generation', False)), + assume_yes=_resolve_assume_yes(config, interactive=interactive), + ) + raise SystemExit( + f'backend {name!r} is not implemented in the leasing CLI yet ' + '(kubeai is a later stage). Use --backend null or compose.' + ) + + +def _open_controller(config, *, interactive: bool = False) -> Controller: + from .._log import configure_logging + + configure_logging() # narrate the leasing verbs (stderr); off the help path + _apply_path_overrides(config) + ledger_path = config.ledger or str(default_ledger_path()) + ledger = Ledger(SqliteStore(ledger_path)) + return Controller(ledger, _make_backend(config, interactive=interactive)) + + +def _load_catalog(config) -> Catalog: + raw = config.catalog or (config_root() / 'catalog.yaml') + path = Path(raw).expanduser() + if not path.exists(): + raise SystemExit( + f'catalog not found: {path} (pass --catalog or create catalog.yaml)' + ) + try: + return Catalog.load(path) + except CatalogError as ex: + raise SystemExit(f'invalid catalog {path}: {ex}') + + +def _load_catalog_for_tui(config) -> tuple[Catalog, Path]: + """Catalog + its path for the TUI, tolerating a missing/empty file. + + The TUI is the place a brand-new user lands, so an absent catalog is not a + hard error here: it loads an empty catalog (the dashboard then shows the + empty-state with a Suggest button) and returns the path it would write to. + """ + raw = config.catalog or (config_root() / 'catalog.yaml') + path = Path(raw).expanduser() + if not path.exists(): + return Catalog.from_dict({'models': {}, 'endpoints': {}}), path + try: + return Catalog.load(path), path + except CatalogError as ex: + raise SystemExit(f'invalid catalog {path}: {ex}') + + +def _resolve(catalog, names, *, sharing=None): + try: + return catalog.resolve_names(names, sharing=sharing) + except CatalogError as ex: + raise SystemExit(str(ex)) + + +def _resolve_lease(config) -> str | None: + sid = getattr(config, 'lease', None) + if not sid and getattr(config, 'env_file', None): + sid = read_lease_id(config.env_file) + return sid + + +def _descriptor_for(controller, lease, deployments, config): + """Build the descriptor, preferring backend-supplied access (real base_url).""" + base_url = config.base_url + api_key_env = config.api_key_env + api_key = None + request_names = None + access = getattr(controller.backend, 'access', None) + info = access(list(lease.endpoints)) if access else None + if info: + base_url = info.get('base_url', base_url) + api_key_env = info.get('api_key_env', api_key_env) + api_key = info.get('api_key') + request_names = info.get('request_names') + return build_descriptor( + lease, + deployments, + base_url=base_url, + api_key_env=api_key_env, + api_key=api_key, + request_names=request_names, + ) + + +def _compose_file_path(controller) -> str | None: + path = getattr(controller.backend, 'compose_file', None) + return str(path) if path else None + + +def _gpu_where(gpus) -> str: + """Human label for a deployment's GPU assignment (or its absence).""" + if gpus is None: + return 'unplaced' + if not gpus: + return 'cpu' + return 'GPU ' + ','.join(str(i) for i in gpus) + + +def _emit_staged(config, controller, outcome) -> int: + """Output for a ``--no-apply`` acquire: what was written + what would run.""" + descriptor = _descriptor_for( + controller, outcome.lease, outcome.deployments, config + ) + if config.env_file: + Path(config.env_file).expanduser().write_text(render_env_file(descriptor)) + assignments = outcome.reconcile.assignments + if config.json: + print(json.dumps({ + 'lease_id': outcome.lease.id, + 'owner': outcome.lease.owner, + 'applied': False, + 'descriptor': descriptor, + 'compose_file': _compose_file_path(controller), + 'placement': [ + {'deployment': g.id, 'served': sorted(g.served), + 'gpus': assignments.get(g.id)} + for g in outcome.deployments + ], + }, indent=2)) + return 0 + print(f'staged {outcome.lease.id} (owner={outcome.lease.owner}) — not applied') + for g in outcome.deployments: + eps = ', '.join(sorted(g.served)) or g.id + print(f' {eps}: {_gpu_where(assignments.get(g.id))} ({g.id})') + path = _compose_file_path(controller) + if path: + print(f' compose: {path}') + print(' apply: infer-stack apply # bring the staged set up') + if path: + # The file carries `name: infer-stack`, so plain docker works too. + print(f' ...or: docker compose -f {path} up -d') + print(f' discard: infer-stack release {outcome.lease.id}') + if config.env_file: + print(f' env-file: {config.env_file}') + return 0 + + +def _emit_acquire(config, controller, outcome) -> int: + if not outcome.applied: + return _emit_staged(config, controller, outcome) + descriptor = _descriptor_for( + controller, outcome.lease, outcome.deployments, config + ) + if config.env_file: + Path(config.env_file).expanduser().write_text( + render_env_file(descriptor) + ) + not_ready = outcome.wait is not None and not outcome.wait.ready + if config.json: + print( + json.dumps( + { + 'lease_id': outcome.lease.id, + 'owner': outcome.lease.owner, + 'descriptor': descriptor, + 'realized': outcome.reconcile.realized, + 'ready': None + if outcome.wait is None + else outcome.wait.ready, + 'pending': [] + if outcome.wait is None + else outcome.wait.pending, + }, + indent=2, + ) + ) + else: + print(f'acquired {outcome.lease.id} (owner={outcome.lease.owner})') + for endpoint, model in descriptor['endpoints'].items(): + print(f' endpoint {endpoint} -> {model}') + if outcome.wait is not None: + print(f' ready: {outcome.wait.ready}') + for gid, endpoint in outcome.wait.pending: + print(f' pending: {endpoint} ({gid})') + access = getattr(controller.backend, 'access', None) + info = access(list(outcome.lease.endpoints)) if access else None + if info and info.get('ui_url'): + print(f' open webui: {info["ui_url"]}') + if info and info.get('proxy_url'): + print(f' front door: {info["proxy_url"]} (UI: / API: /v1)') + if config.env_file: + print(f' env-file: {config.env_file}') + return 2 if not_ready else 0 + + +def _do_acquire(config, *, owner: str, ttl_seconds: float | None) -> int: + from .._log import logger + from ..leasing.backend import ConvergeAborted, PlacementError + + render_only = not bool(getattr(config, 'apply', True)) + # Staging (--no-apply) is non-destructive (no docker up), so don't gate it + # behind the diff prompt — the rendered file *is* the thing you asked to see. + controller = _open_controller(config, interactive=not render_only) + catalog = _load_catalog(config) + names = _collect_names(config.names) + if not names: + raise SystemExit('give at least one endpoint or bundle name') + sharing = Sharing.DEDICATED if getattr(config, 'dedicated', False) else None + requests = _resolve(catalog, names, sharing=sharing) + logger.info('Acquiring {} for {}', ', '.join(names), owner) + if render_only: + logger.info('Render-only: staging the compose project without applying') + elif config.wait: + logger.info( + 'Will wait up to {:.0f}s for readiness (poll {:.0f}s)', + float(config.timeout), float(config.interval), + ) + try: + outcome = controller.acquire( + owner, + requests, + ttl_seconds=ttl_seconds, + wait=bool(config.wait), + timeout=float(config.timeout), + interval=float(config.interval), + apply=not render_only, + ) + except ConvergeAborted: + raise SystemExit('aborted: compose changes not applied (no lease kept)') + except PlacementError as ex: + lines = ['could not place every requested endpoint (no lease kept):'] + lines += [f' {r}' for r in ex.reasons] or [ + f' {", ".join(ex.deployment_ids)}' + ] + lines.append( + ' free a GPU first — `infer-stack leases` to see what holds them, ' + 'then `infer-stack release`/`evict`. (Every GPU, including any ' + 'display-attached one, is used unless you set --skip-display-gpus.)' + ) + raise SystemExit('\n'.join(lines)) + return _emit_acquire(config, controller, outcome) + + +# --------------------------------------------------------------------------- +# shared flag mixins +# --------------------------------------------------------------------------- + + +class _LeasingCommonMixin(_PathOverridesMixin, _AllowedGpusMixin, _DisplayGpuMixin): + backend = scfg.Value( + None, + choices=['null', 'compose', 'kubeai'], + help='Serving backend. "null" (dry-run) and "compose" are implemented. ' + 'Defaults to `config set backend …`, else "null".', + ) + ledger = scfg.Value( + None, type=str, help='Path to the lease ledger sqlite db.' + ) + require_generation = scfg.Value( + False, + isflag=True, + help='Readiness requires a real generation, not just model listing ' + '(compose backend; Ollama always generates to warm the tag).', + ) + litellm = scfg.Value( + None, + isflag=True, + help='Render the LiteLLM gateway — one OpenAI base_url fronting every ' + 'endpoint alias (compose backend). On by default; use --no-litellm for ' + 'a lean stack where Open WebUI talks to the upstreams (e.g. an Ollama ' + 'daemon) directly. Overrides `config set litellm …`.', + ) + ui = scfg.Value( + None, + isflag=True, + help='Render a managed Open WebUI in front of the gateway (compose ' + 'backend). On by default; use --no-ui to skip. Overrides ' + '`config set ui …`.', + ) + reverse_proxy = scfg.Value( + None, + isflag=True, + alias=['reverse-proxy'], + help='Front the gateway + UI with a single-port HTTP reverse proxy, so ' + 'you hit one origin (UI at /, API at /v1) — off by default, no TLS/auth ' + '(localhost / trusted networks only). Port + bring-your-own nginx.conf ' + 'live in the `reverse_proxy` setting (`config set` / `config edit`).', + ) + + +class _ApprovalMixin(_LeasingCommonMixin): + """A verb that converges the compose project and so gates on a diff. + + Any change to the on-disk compose project is shown and confirmed on a + terminal; ``--yes`` (or a non-TTY) applies without prompting. + """ + + yes = scfg.Value( + False, isflag=True, alias=['y'], + help='Apply compose changes without showing the diff / prompting ' + '(compose backend). Implied when stdout is not a terminal.', + ) + + +class _AcquireFlagsMixin(_LeasingCommonMixin): + catalog = scfg.Value(None, type=str, help='Path to catalog.yaml.') + base_url = scfg.Value( + 'http://127.0.0.1:14042/v1', + type=str, + help='Base URL written into the endpoint descriptor (dry-run placeholder).', + ) + api_key_env = scfg.Value( + 'LITELLM_MASTER_KEY', + type=str, + help='Name of the env var holding the API key (kept out of artifacts).', + ) + wait = scfg.Value( + True, isflag=True, help='Block until ready (use --no-wait to skip).' + ) + apply = scfg.Value( + True, + isflag=True, + help='Apply the render (docker compose up). Use --no-apply to *stage* ' + 'only: declare the lease and write the on-disk compose project + ' + 'placement WITHOUT starting it, then `infer-stack apply` to bring it up ' + '(compose backend). --no-apply implies no readiness wait and no diff ' + 'prompt; `release` discards a staged lease.', + ) + timeout = scfg.Value(600, type=float, help='Readiness wait timeout (s).') + interval = scfg.Value(5, type=float, help='Readiness poll interval (s).') + env_file = scfg.Value( + None, type=str, help='Write the sourceable endpoint env-file here.' + ) + yes = scfg.Value( + False, isflag=True, alias=['y'], + help='Apply compose changes without showing the diff / prompting ' + '(compose backend). Implied when stdout is not a terminal.', + ) + json = scfg.Value(False, isflag=True, help='Emit JSON instead of text.') + + +# --------------------------------------------------------------------------- +# subcommands +# --------------------------------------------------------------------------- + + +class AcquireCLI(_AcquireFlagsMixin): + """Acquire a lease on one or more endpoints/bundles, bring them up, wait. + + ``acquire NAME…`` is the everyday verb. It takes a lease on each endpoint or + bundle, renders the compose project (the LiteLLM gateway + one container per + model + a managed Open WebUI), brings it up, and blocks until every endpoint + is ready. Run it again with more names to add models side by side — the + gateway and UI stay put. Placement, ``docker compose``, and readiness are + narrated on stderr. + + With no ``--ttl`` the lease is infinite — a standing service you tear down + explicitly (``release`` / ``evict``). Pass ``--ttl`` (e.g. ``2h``, ``30m``) + for a soft, time-boxed reservation: the lease protects its models while + held, and once it expires (or is released) and nothing else needs them they + can be reclaimed. This is also the programmatic seam — write a sourceable + endpoint env-file with ``--env-file`` and release by that file or lease id + when done. (For a one-shot "acquire, run a command, release", use + ``infer-stack run`` instead.) + + The work is render (write the on-disk compose project) then apply (``docker + compose up``). ``--no-apply`` does just the render so you can see what + would run before pulling the trigger (then ``infer-stack apply``). + ``--no-wait`` applies but returns immediately so several models load in + parallel (``wait`` for them later). ``--no-ui`` skips Open WebUI. On a + terminal you are shown the compose diff and asked before applying; ``--yes`` + skips that prompt (and it is skipped automatically off a TTY). + """ + + __command__ = 'acquire' + __epilog__ = """ + Examples: + # stand up one model: render + up + wait until it can generate + infer-stack acquire qwen05-1 --require-generation + + # a time-boxed reservation with a sourceable env-file, released by id + infer-stack acquire qwen-coder --ttl 2h --env-file is.env + + # stage only: write the compose project but don't start it, then apply + infer-stack acquire qwen05-1 --no-apply + infer-stack apply + + # fan out: start several without blocking, then wait together + infer-stack acquire smol17b-1 --no-wait --yes + infer-stack acquire qwen15-1 --no-wait --yes + infer-stack wait smol17b-1 qwen15-1 --require-generation + + # see what is actually running and on which GPUs + infer-stack leases + """ + + names = scfg.Value( + [], nargs='*', position=1, type=str, help='Endpoint or bundle names.' + ) + ttl = scfg.Value( + None, type=str, help='Soft TTL (e.g. 2h, 30m); default infinite.' + ) + owner = scfg.Value(None, type=str, help='Lease owner (default: $USER).') + dedicated = scfg.Value( + False, + isflag=True, + help='Force a dedicated deployment instead of coalescing.', + ) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + return _do_acquire( + config, + owner=config.owner or _default_owner(), + ttl_seconds=_parse_duration(config.ttl), + ) + + +class RenderCLI(_LeasingCommonMixin): + """Write the on-disk compose project for the current desired set — no up. + + Lease-free and idempotent: ``render`` re-materializes the manifest (compose + file + gateway config + GPU placement) from whatever is *already* declared, + WITHOUT starting anything — to inspect what would run, or refresh a file you + touched by hand. It creates no lease; to stage a *new* endpoint use ``acquire + --no-apply`` (which declares it too). Apply with ``infer-stack apply``. + """ + + __command__ = 'render' + + json = scfg.Value(False, isflag=True) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + controller = _open_controller(config, interactive=False) + rec = controller.reconcile(apply=False) + path = _compose_file_path(controller) + if config.json: + print(json.dumps({ + 'applied': False, + 'compose_file': path, + 'placement': rec.assignments, + 'unplaced': rec.unplaced, + }, indent=2)) + return 0 + print(f'rendered desired set -> {path or "(backend has no on-disk project)"}') + for gid, gpus in sorted(rec.assignments.items()): + print(f' {gid}: {_gpu_where(gpus)}') + for err in rec.placement_errors: + print(f' ! {err}') + print(' apply: infer-stack apply') + return 0 + + +class ApplyCLI(_ApprovalMixin): + """Bring the current desired set up to match the ledger (render + up). + + Lease-free: ``apply`` creates no lease — it converges whatever is already + declared (by ``acquire``) onto the backend. It is the *trigger* + for a staged ``acquire --no-apply`` and the re-sync button after a manual edit + or a backend hiccup; idempotent (a second apply with nothing changed is a + no-op). On a terminal it shows the compose diff and asks (``--yes`` skips). + The rendered file carries ``name: infer-stack``, so ``docker compose -f + up -d`` is an exact equivalent. (``infer-stack stack up`` is the + lower-level "run exactly what is on disk" hatch; ``apply`` re-renders from + intent first.) + """ + + __command__ = 'apply' + + wait = scfg.Value( + False, isflag=True, help='Also block until ready after bringing it up.' + ) + timeout = scfg.Value(600, type=float, help='Readiness wait timeout (s).') + interval = scfg.Value(5, type=float, help='Readiness poll interval (s).') + json = scfg.Value(False, isflag=True) + + @classmethod + def main(cls, argv=True, **kwargs): + from ..leasing.backend import ConvergeAborted + + config = cls.cli(argv=argv, data=kwargs) + controller = _open_controller(config, interactive=True) + try: + rec = controller.reconcile(apply=True) + except ConvergeAborted: + raise SystemExit('aborted: compose changes not applied') + result = None + if config.wait: + controller.ledger.sweep() + _, deployments = controller.ledger.status() + live = [g for g in deployments if g.state == DeploymentState.LIVE] + if live: + result = controller.wait_ready( + live, + timeout=float(config.timeout), + interval=float(config.interval), + ) + if config.json: + print(json.dumps({ + 'applied': True, + 'realized': rec.realized, + 'torn_down': rec.torn_down, + 'placement': rec.assignments, + 'unplaced': rec.unplaced, + 'ready': None if result is None else result.ready, + }, indent=2)) + return 0 if (result is None or result.ready) else 2 + print( + f'applied: {len(rec.realized)} started, {len(rec.torn_down)} stopped' + ) + for gid in rec.realized: + print(f' + {gid}') + for gid in rec.torn_down: + print(f' - {gid}') + for err in rec.placement_errors: + print(f' ! {err}') + if result is not None and not result.ready: + print(' not ready (timed out)') + for gid, ep in result.pending: + print(f' pending: {ep} ({gid})') + return 2 + return 0 + + +def _declined_exit() -> SystemExit: + return SystemExit( + 'declined: the ledger change stands but docker was not touched — ' + 'run `infer-stack apply` to apply it.' + ) + + +class ReleaseCLI(_ApprovalMixin): + """Release a lease; deployments idle/teardown per their reclaim policy. + + On a terminal the resulting compose change is shown and confirmed before + docker is touched (``--yes`` skips); declining leaves the lease released in + the ledger but the containers running — ``infer-stack apply`` then applies + it. All the lease's changes converge in a single step, so you are asked at + most once. + """ + + __command__ = 'release' + + lease = scfg.Value( + None, position=1, type=str, help='Lease id (or use --env-file).' + ) + env_file = scfg.Value( + None, type=str, help='Read the lease id from this env-file.' + ) + all = scfg.Value( + False, isflag=True, + help='Release every active lease (the whole stack idles/tears down).', + ) + evict = scfg.Value( + False, isflag=True, + help='Also evict (tear down) the released deployment(s) now, even if their ' + 'reclaim policy is keep-warm — frees the GPU immediately.', + ) + json = scfg.Value(False, isflag=True) + + @classmethod + def main(cls, argv=True, **kwargs): + from ..leasing.backend import ConvergeAborted + + config = cls.cli(argv=argv, data=kwargs) + controller = _open_controller(config, interactive=True) + + if config.all: + if config.lease or config.env_file: + raise SystemExit('release: --all takes no lease/--env-file') + controller.ledger.sweep() + leases, _ = controller.ledger.status() + released = [le.id for le in leases if le.state == LeaseState.ACTIVE] + for sid in released: + controller.ledger.release(sid) + evicted: list[str] = [] + if config.evict: + evicted = controller.ledger.evict_idle(None) # every idle deployment + else: + sid = _resolve_lease(config) + if not sid: + raise SystemExit( + 'release: give a lease id, --env-file, or --all' + ) + rel = controller.ledger.release(sid) + released = [sid] + evicted = [] + if config.evict: + evicted = controller.ledger.evict_idle(rel.idled_deployment_ids) + + # One converge for the whole command -> at most one diff prompt. + try: + rec = controller.reconcile() + except ConvergeAborted: + raise _declined_exit() + return _emit_release(config, released, sorted(rec.torn_down), evicted) + + +def _emit_release(config, released, torn_down, evicted) -> int: + if config.json: + print(json.dumps({ + 'released': released, + 'torn_down': torn_down, + 'evicted': evicted, + }, indent=2)) + return 0 + if not released: + print('no active leases to release') + else: + print(f'released {len(released)} lease(s)') + for sid in released: + print(f' {sid}') + for gid in torn_down: + print(f' torn down: {gid}') + return 0 + + +def _resolve_idle_targets(controller, names: list[str]) -> tuple[list[str], list[str]]: + """Map endpoint aliases / deployment ids to currently-idle deployment ids. + + Returns ``(target_deployment_ids, unmatched_names)``. + """ + controller.ledger.sweep() + _, deployments = controller.ledger.status() + idle = [g for g in deployments if g.state == DeploymentState.IDLE] + wanted = set(names) + targets, matched = [], set() + for g in idle: + hit = wanted & ({g.id} | set(g.served)) + if hit: + targets.append(g.id) + matched |= hit + return targets, sorted(wanted - matched) + + +class EvictCLI(_ApprovalMixin): + """Force-evict released (idle) models now, freeing their GPUs. + + A released ``keep-warm`` model stays resident (idle) to avoid cold-start + thrash — handy, but it holds a GPU. ``evict`` tears such deployments down now, + overriding keep-warm. Target by served endpoint alias or deployment id, or + ``--all`` for every idle deployment. (Live models — those with an active lease — + are never evicted; release them first.) On a terminal the teardown is shown + and confirmed before docker is touched (``--yes`` skips). + """ + + __command__ = 'evict' + + names = scfg.Value( + [], nargs='*', position=1, type=str, + help='Endpoint alias or deployment id to evict.', + ) + all = scfg.Value(False, isflag=True, help='Evict every idle deployment.') + json = scfg.Value(False, isflag=True) + + @classmethod + def main(cls, argv=True, **kwargs): + from ..leasing.backend import ConvergeAborted + + config = cls.cli(argv=argv, data=kwargs) + controller = _open_controller(config, interactive=True) + names = _collect_names(config.names) + if not names and not config.all: + raise SystemExit('evict: give an endpoint/deployment name or --all') + try: + if config.all: + outcome = controller.evict(None) + else: + targets, missing = _resolve_idle_targets(controller, names) + if missing: + print(f'no idle deployment for: {", ".join(missing)}') + if not targets: + if config.json: + print(json.dumps( + {'evicted': [], 'torn_down': []}, indent=2)) + else: + print('nothing to evict') + return 0 + outcome = controller.evict(targets) + except ConvergeAborted: + raise _declined_exit() + if config.json: + print(json.dumps({ + 'evicted': outcome.evicted_deployment_ids, + 'torn_down': outcome.reconcile.torn_down, + }, indent=2)) + elif not outcome.evicted_deployment_ids: + print('nothing to evict') + else: + print(f'evicted {len(outcome.evicted_deployment_ids)} deployment(s)') + for gid in outcome.evicted_deployment_ids: + print(f' {gid}') + return 0 + + +class WaitCLI(_LeasingCommonMixin): + """Block until served endpoints are ready — the companion to ``acquire + --no-wait``. + + Fan out, then wait: ``acquire --no-wait smol17b-1`` + ``acquire --no-wait + smol135-1`` kick both deployments off in parallel (each converges and starts + its container without blocking), then ``wait smol17b-1 smol135-1`` blocks + until they can actually serve. With no names it waits for every live deployment. + + ``--require-generation`` makes "ready" mean a real generated token (not just + a model that is listed) — the same readiness *criterion* the ``acquire`` + verb takes; this command is the *blocking* half, distinct from it. + """ + + __command__ = 'wait' + + names = scfg.Value( + [], nargs='*', position=1, type=str, + help='Endpoint names to wait for (default: every live deployment).', + ) + timeout = scfg.Value(600, type=float, help='Overall wait timeout (s).') + interval = scfg.Value(5, type=float, help='Readiness poll interval (s).') + json = scfg.Value(False, isflag=True) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + controller = _open_controller(config) + controller.ledger.sweep() + _, deployments = controller.ledger.status() + live = [g for g in deployments if g.state == DeploymentState.LIVE] + names = _collect_names(config.names) + if names: + wanted = set(names) + served = {ep for g in live for ep in g.served} + missing = sorted(wanted - served) + if missing: + raise SystemExit( + f'not served by any live deployment: {", ".join(missing)} ' + '(acquire it first, or check `infer-stack leases`)' + ) + targets = [g for g in live if wanted & set(g.served)] + endpoints = wanted + else: + targets, endpoints = live, None + if not targets: + print('nothing to wait for (no live deployments)') + return 0 + result = controller.wait_ready( + targets, + endpoints=endpoints, + timeout=float(config.timeout), + interval=float(config.interval), + ) + if config.json: + print(json.dumps({ + 'ready': result.ready, + 'pending': [ + {'deployment': gid, 'endpoint': ep} + for gid, ep in result.pending + ], + }, indent=2)) + elif result.ready: + print('ready') + else: + print('not ready (timed out)') + for gid, ep in result.pending: + print(f' pending: {ep} ({gid})') + return 0 if result.ready else 2 + + +class TuiCLI(_LeasingCommonMixin): + """Launch the Textual TUI: a live monitor of the stack with controls to + serve / release / evict models. + + Mostly a monitor — the lease + deployment tables (desired state vs running, GPUs) + refresh live — with key-bound controls: ``s`` serve, ``d`` release, ``a`` + release-all, ``e`` evict, ``r`` refresh, ``q`` quit. Opt-in extra: install + textual with ``pip install "infer-stack[tui]"``. + """ + + __command__ = 'tui' + + catalog = scfg.Value(None, type=str, help='Path to catalog.yaml.') + interval = scfg.Value(3.0, type=float, help='Auto-refresh interval (s).') + + @classmethod + def main(cls, argv=True, **kwargs): + from ..paths import settings_path + + config = cls.cli(argv=argv, data=kwargs) + try: + from ..tui import run_tui + except ImportError as ex: + raise SystemExit( + 'the TUI needs the optional `textual` dependency — install it ' + 'with `pip install "infer-stack[tui]"` (or `pip install textual`). ' + f'[{ex}]' + ) + # Require a one-time `config init` first: the TUI assumes a configured + # world (backend, data dir, …). Without it the dashboard would launch + # against defaults the user never chose. + _apply_path_overrides(config) + if not settings_path().exists(): + raise SystemExit( + 'no settings found — run `infer-stack config init` once before ' + f'launching the TUI (expected {settings_path()}).' + ) + controller = _open_controller(config) + catalog, catalog_path = _load_catalog_for_tui(config) + return run_tui( + controller, catalog, + interval=float(config.interval), catalog_path=str(catalog_path), + ) + + +class RenewCLI(_LeasingCommonMixin): + """Extend (or make infinite) a lease's protection window.""" + + __command__ = 'renew' + + lease = scfg.Value(None, position=1, type=str, help='Lease id.') + env_file = scfg.Value(None, type=str) + ttl = scfg.Value(None, type=str, help='New soft TTL (e.g. 2h); empty=infinite.') + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + controller = _open_controller(config) + sid = _resolve_lease(config) + if not sid: + raise SystemExit('renew: give a lease id or --env-file') + lease = controller.ledger.renew( + sid, ttl_seconds=_parse_duration(config.ttl) + ) + if lease is None: + raise SystemExit(f'renew: no such lease {sid}') + print(f'renewed {sid}') + return 0 + + +class RunCLI(_LeasingCommonMixin): + """Acquire endpoints, run a command with the endpoint env, then release. + + Everything after ``--`` is the command. The lease is always released on + exit; the TTL is the backstop if the process is hard-killed. + """ + + __command__ = 'run' + + catalog = scfg.Value(None, type=str, help='Path to catalog.yaml.') + endpoint = scfg.Value( + None, + type=str, + alias=['endpoints'], + help='Comma-separated endpoint or bundle names.', + ) + base_url = scfg.Value('http://127.0.0.1:14042/v1', type=str) + api_key_env = scfg.Value('LITELLM_MASTER_KEY', type=str) + owner = scfg.Value(None, type=str) + ttl = scfg.Value('2h', type=str, help='Soft TTL backstop (default 2h).') + timeout = scfg.Value(600, type=float) + interval = scfg.Value(5, type=float) + command = scfg.Value( + [], nargs='*', position=1, type=str, help='Command to run (after --).' + ) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + controller = _open_controller(config) + catalog = _load_catalog(config) + names = _collect_names(config.endpoint) + command = list(config.command or []) + if not names: + raise SystemExit('run: --endpoint is required') + if not command: + raise SystemExit('run: give a command after --') + requests = _resolve(catalog, names) + outcome = controller.acquire( + config.owner or _default_owner(), + requests, + ttl_seconds=_parse_duration(config.ttl), + wait=True, + timeout=float(config.timeout), + interval=float(config.interval), + ) + if outcome.wait is not None and not outcome.wait.ready: + controller.release(outcome.lease.id) + raise SystemExit( + f'run: endpoints not ready: {outcome.wait.pending}' + ) + descriptor = _descriptor_for( + controller, outcome.lease, outcome.deployments, config + ) + env = dict(os.environ) + env.update(descriptor_env(descriptor)) + try: + proc = subprocess.run(command, env=env) + return int(proc.returncode) + finally: + controller.release(outcome.lease.id) + + +def _lease_ttl(le) -> str: + return 'inf' if le.expires_at is None else f'@{le.expires_at:.0f}' + + +def _placement_view(controller): + """Best-effort (observed running gids, deployment id -> GPU indices). + + Reads what the backend *actually* has running (``observe``) and where each + desired deployment is/would be placed (``plan``). Both are guarded so a dry-run + (``NullBackend``) or a docker-less host degrades to "unknown" rather than + erroring — ``leases`` must always render. + """ + backend = controller.backend + observed: set[str] = set() + try: + observed = set(backend.observe()) + except Exception: # noqa: BLE001 - status must never crash + pass + assignments: dict[str, list[int]] = {} + plan = getattr(backend, 'plan', None) + if plan is not None: + try: + assignments = dict(plan(controller.desired_deployments()).assignments) + except Exception: # noqa: BLE001 + pass + return observed, assignments + + +def _running_label(gid, observed) -> str: + return 'running' if gid in observed else '—' + + +def _gpu_label(gid, observed, assignments) -> str: + """Where a deployment is (running) or is slated to be (desired, not yet up).""" + gpus = assignments.get(gid) + if gpus is None: + return '-' + where = ','.join(str(i) for i in gpus) if gpus else 'cpu' + return where if gid in observed else f'→{where}' # → = slated, not yet up + + +def _print_leases_plain(leases, deployments, observed, assignments) -> None: + print('leases:') + if not leases: + print(' (none)') + for le in leases: + print( + f' {le.id} owner={le.owner} state={le.state} ' + f'ttl={_lease_ttl(le)} endpoints={",".join(le.endpoints) or "-"}' + ) + print('deployments:') + if not deployments: + print(' (none)') + for g in deployments: + print( + f' {g.id} {g.engine} state={g.state} ' + f'running={_running_label(g.id, observed)} ' + f'gpus={_gpu_label(g.id, observed, assignments)} ' + f'demand={g.demand} served={",".join(sorted(g.served)) or "-"}' + ) + + +def _print_leases_rich(leases, deployments, observed, assignments, console) -> None: + from rich.table import Table + from rich.text import Text + + def state_style(state) -> str: + s = str(state).lower() + if 'active' in s or 'live' in s: + return 'green' + if 'stop' in s or 'expir' in s or 'idle' in s: + return 'yellow' + return 'dim' + + console.print(Text('leases', style='bold')) + if not leases: + console.print(' [dim](none)[/dim]') + else: + lt = Table(box=None, pad_edge=False, padding=(0, 2, 0, 0), + header_style='dim') + lt.add_column('id', style='cyan', no_wrap=True) + lt.add_column('owner') + lt.add_column('state') + lt.add_column('ttl', style='dim') + lt.add_column('endpoints', style='magenta', overflow='fold') + for le in leases: + lt.add_row( + le.id, le.owner, + Text(str(le.state), style=state_style(le.state)), + _lease_ttl(le), ','.join(le.endpoints) or '-', + ) + console.print(lt) + + console.print(Text('deployments', style='bold')) + if not deployments: + console.print(' [dim](none)[/dim]') + else: + gt = Table(box=None, pad_edge=False, padding=(0, 2, 0, 0), + header_style='dim') + gt.add_column('id', style='cyan', no_wrap=True) + gt.add_column('engine') + gt.add_column('state') # desired (ledger) + gt.add_column('running') # actual (backend.observe) + gt.add_column('gpus') # on / →slated + gt.add_column('demand', justify='right', style='dim') + gt.add_column('served', style='magenta', overflow='fold') + for g in deployments: + running = g.id in observed + gpus = _gpu_label(g.id, observed, assignments) + gt.add_row( + g.id, g.engine, + Text(str(g.state), style=state_style(g.state)), + Text('running' if running else '—', + style='green' if running else 'dim'), + Text(gpus, style='cyan' if running else 'yellow'), + str(g.demand), ','.join(sorted(g.served)) or '-', + ) + console.print(gt) + + +class LeasesCLI(_LeasingCommonMixin): + """Show current leases and deployment deployments (the leasing-model status). + + Two tables. **leases** are who asked for what (id, owner, state, ttl, + endpoints). **deployments** are the actual deployments behind them — one deployment is + one model in one container (it may serve several endpoint aliases). For each + deployment: + + \b + state what the ledger *wants* — live / idle / stopped + running what the backend *actually* has up right now + gpus the GPU indices it is on; a ``→`` prefix means slated (desired + but not yet started), ``-`` means no placement info + demand how many active leases are protecting it + served the endpoint aliases routed to it + + A row that is ``state=live`` but ``running=—`` is desired-but-not-up: either + still starting, staged via ``acquire --no-apply`` (run ``apply`` to bring it + up), or unplaceable (no free GPU — see ``acquire``'s error / ``evict``). + """ + + __command__ = 'leases' + __epilog__ = """ + Examples: + infer-stack leases # the two tables (rich on a terminal) + infer-stack leases --json # JSON (adds running + gpus per deployment) + """ + + json = scfg.Value(False, isflag=True) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + controller = _open_controller(config) + controller.ledger.sweep() # materialize TTL expiry for an accurate view + leases, deployments = controller.ledger.status() + observed, assignments = _placement_view(controller) + if config.json: + print( + json.dumps( + { + 'leases': [ + { + 'id': le.id, + 'owner': le.owner, + 'state': le.state, + 'endpoints': le.endpoints, + 'expires_at': le.expires_at, + } + for le in leases + ], + 'deployments': [ + { + 'id': g.id, + 'engine': g.engine, + 'state': g.state, # desired (ledger) + 'running': g.id in observed, # actual (backend) + 'gpus': assignments.get(g.id), + 'demand': g.demand, + 'served': sorted(g.served), + } + for g in deployments + ], + }, + indent=2, + ) + ) + return 0 + from rich.console import Console + + console = Console() + if console.is_terminal: + _print_leases_rich(leases, deployments, observed, assignments, console) + else: + _print_leases_plain(leases, deployments, observed, assignments) + return 0 + + +def _secret_env_path() -> Path: + """The managed compose secrets file (.env that docker compose auto-loads).""" + return data_root() / 'leasing' / 'compose' / '.env' + + +def _front_door(config) -> tuple[str, str | None]: + """Resolve the front-door base_url + master key for a smoke test. + + Reads them straight from the managed state (front-door port + ``.env``) so + `test` is cheap and doesn't need GPU detection or a backend object. An + explicit ``--base-url`` overrides the derived URL. + """ + from ..config import DEFAULT_PORTS + + base_url = getattr(config, 'base_url', None) + if not base_url: + port = int(getattr(config, 'port', None) or DEFAULT_PORTS['litellm']) + base_url = f'http://127.0.0.1:{port}/v1' + key = None + env_path = _secret_env_path() + if env_path.exists(): + key = parse_env_file(env_path).get('LITELLM_MASTER_KEY') + return base_url.rstrip('/'), key + + +class TestCLI(_PathOverridesMixin): + """Smoke-test a served endpoint through the front door (a real generation). + + The concise alternative to hand-rolling ``curl``: sends one chat completion + to the endpoint *alias* via the LiteLLM gateway, then prints latency and the + reply (or an actionable error). Exit code is non-zero on failure, so it is + usable in scripts/CI. + """ + + __command__ = 'test' + + name = scfg.Value( + None, position=1, type=str, help='Endpoint alias to test (e.g. chat).' + ) + prompt = scfg.Value( + 'Reply with the single word: ready.', type=str, help='Prompt to send.' + ) + max_tokens = scfg.Value(32, type=int) + timeout = scfg.Value(60, type=float, help='Request timeout (s).') + base_url = scfg.Value( + None, type=str, help='Override the gateway base URL (…/v1).' + ) + port = scfg.Value( + None, type=int, help='Override the gateway port (default: 14042).' + ) + json = scfg.Value(False, isflag=True, help='Emit JSON instead of text.') + + @classmethod + def main(cls, argv=True, **kwargs): + import time + + import requests + + config = cls.cli(argv=argv, data=kwargs) + _apply_path_overrides(config) + if not config.name: + raise SystemExit('test: give an endpoint alias (e.g. `test chat`)') + base_url, key = _front_door(config) + headers = {'Content-Type': 'application/json'} + if key: + headers['Authorization'] = f'Bearer {key}' + payload = { + 'model': config.name, + 'messages': [{'role': 'user', 'content': config.prompt}], + 'max_tokens': int(config.max_tokens), + } + t0 = time.monotonic() + try: + resp = requests.post( + f'{base_url}/chat/completions', + headers=headers, + json=payload, + timeout=float(config.timeout), + ) + except requests.exceptions.RequestException as ex: + return _test_fail(config, base_url, f'not reachable: {ex}') + dt = time.monotonic() - t0 + if resp.status_code >= 400: + body = (resp.text or '').strip()[:300] + return _test_fail( + config, base_url, f'HTTP {resp.status_code}: {body}' + ) + try: + reply = resp.json()['choices'][0]['message']['content'] + except (ValueError, KeyError, IndexError) as ex: + return _test_fail(config, base_url, f'unexpected response: {ex}') + reply = (reply or '').strip() + if config.json: + print(json.dumps( + {'endpoint': config.name, 'ok': True, + 'seconds': round(dt, 3), 'reply': reply}, indent=2)) + else: + print(f'{config.name}: ok ({dt:.2f}s) {reply!r}') + return 0 + + +def _test_fail(config, base_url: str, reason: str) -> int: + if config.json: + print(json.dumps( + {'endpoint': config.name, 'ok': False, + 'base_url': base_url, 'reason': reason}, indent=2)) + else: + print(f'{config.name}: FAILED via {base_url} — {reason}') + print(' is it served? infer-stack leases | ' + 'infer-stack acquire ' + str(config.name)) + return 1 + + +class EnvCLI(_PathOverridesMixin): + """The managed env-file: print its path, read a value, or set one. + + infer-stack keeps managed secrets — the LiteLLM master key, ``HF_TOKEN``, + … — in a ``.env`` that docker compose auto-loads. Anyone who can read it + already has the secrets, so there's nothing to hide behind a separate + ``secret`` verb; one ``env`` does it all: + + \b + infer-stack env # the .env path (source it to load) + infer-stack env LITELLM_MASTER_KEY # print one value + infer-stack env HF_TOKEN=hf_… # set one value (merges, before acquire) + infer-stack env --export # every entry as `export KEY=value` + + The argument is a KEY to read, or ``KEY=VALUE`` to write (writes merge + non-destructively, so the managed LiteLLM key is preserved). + """ + + __command__ = 'env' + + arg = scfg.Value( + None, position=1, type=str, + help='KEY to read its value, or KEY=VALUE to set it. Empty = path.', + ) + export = scfg.Value( + False, isflag=True, help='Print every entry as `export KEY=value`.' + ) + + @classmethod + def main(cls, argv=True, **kwargs): + config = cls.cli(argv=argv, data=kwargs) + _apply_path_overrides(config) + env_path = _secret_env_path() + + # Write: `env KEY=VALUE` + if config.arg and '=' in config.arg: + key, _, value = config.arg.partition('=') + key = key.strip() + if not key: + raise SystemExit('env: empty key in KEY=VALUE') + write_env_file(env_path, {key: value}) + print(f'set {key} ({env_path})') + return 0 + + # Path first and foremost (it may not exist yet — that's fine). + if not (config.arg or config.export): + print(env_path) + return 0 + + # Read: `env KEY` / `env --export` + if not env_path.exists(): + raise SystemExit( + f'no managed env-file at {env_path}; run an `acquire` ' + 'with --backend compose first (or `infer-stack env KEY=VALUE`)' + ) + env = parse_env_file(env_path) + if config.arg: + if config.arg not in env: + raise SystemExit(f'{config.arg!r} not found in {env_path}') + print(env[config.arg]) + return 0 + for name, value in env.items(): + print(f'export {name}={shlex.quote(value)}') + return 0 diff --git a/infer_stack/cli/commands_meta.py b/infer_stack/cli/commands_meta.py index 76ac4488..83765e56 100644 --- a/infer_stack/cli/commands_meta.py +++ b/infer_stack/cli/commands_meta.py @@ -12,25 +12,16 @@ from __future__ import annotations import json +from dataclasses import dataclass from pathlib import Path from typing import Any import scriptconfig as scfg from .. import __version__ -from ..config import kubeai_local_values_path -from ..config import normalized_state +from ..paths import config_root, data_root, settings_path from .context import _apply_path_overrides -from .context import _safe_load_config -from .context import backend_name -from .context import config_path -from .context import generated_dir -from .context import kubeai_generated_dir -from .context import models_path -from .context import plan_path -from .context import runtime_dir_for_config from .options import _PathOverridesMixin -from ..paths import config_root class VersionCLI(scfg.DataConfig): @@ -45,6 +36,84 @@ def main(cls, argv=True, **kwargs): return 0 +# --------------------------------------------------------------------------- +# help tree — the whole command surface at a glance (cf. `aivm help tree`) +# --------------------------------------------------------------------------- + + +def _iter_subcommands(modal: Any): + """(name, cls) for each registered subcommand of a ModalCLI class.""" + out: dict[str, Any] = {} + for attr, val in vars(modal).items(): + if attr.startswith('_'): + continue + if isinstance(val, type) and issubclass( + val, (scfg.DataConfig, scfg.ModalCLI) + ): + name = getattr(val, '__command__', None) or attr.replace('_', '-') + out[name] = val + return sorted(out.items()) + + +def _doc_one_line(cls: Any) -> str: + doc = (getattr(cls, '__doc__', None) or '').strip() + if not doc: + doc = (getattr(cls, 'description', '') or '').strip() + line = doc.splitlines()[0].strip() if doc else '' + return line.replace('``', '').replace('`', '') # drop rST backticks + + +def _is_group(sub: Any) -> bool: + return isinstance(sub, type) and issubclass(sub, scfg.ModalCLI) + + +def _build_tree(modal: Any, node: Any) -> None: + """Attach each subcommand of ``modal`` to a rich Tree ``node``.""" + from rich.text import Text + + for name, sub in _iter_subcommands(modal): + label = Text() + # Groups (submodals) in bold cyan, runnable leaves in green; the + # one-line description trails in dim. Built from Text segments (not + # markup) so brackets/backticks in docstrings can't break rendering. + label.append(name, style='bold cyan' if _is_group(sub) else 'green') + desc = _doc_one_line(sub) + if desc: + label.append(' ') + label.append(desc, style='dim') + child = node.add(label) + if _is_group(sub): + _build_tree(sub, child) + + +class HelpTreeCLI(scfg.DataConfig): + """Print the full nested command tree with one-line descriptions.""" + + __command__ = 'tree' + + @classmethod + def main(cls, argv=True, **kwargs): + cls.cli(argv=argv, data=kwargs) + from rich.console import Console + from rich.text import Text + from rich.tree import Tree + + from infer_stack.cli import ManageCLI + + tree = Tree(Text('infer-stack', style='bold')) + _build_tree(ManageCLI, tree) + Console().print(tree) + return 0 + + +class HelpModalCLI(scfg.ModalCLI): + """Help utilities (use ``infer-stack --help`` for per-command help).""" + + __command__ = 'help' + + tree = HelpTreeCLI + + def _path_status(path: Path) -> str: try: if path.exists(): @@ -75,9 +144,8 @@ def _status_style(status: str) -> str: return 'red' # error:* / permission-denied -def _render_plain(groups: dict, backend: str) -> None: +def _render_plain(groups: dict) -> None: print('infer-stack paths') - print(f'backend: {backend}') for group, entries in groups.items(): print(f'{group}:') for e in entries: @@ -86,12 +154,11 @@ def _render_plain(groups: dict, backend: str) -> None: ) -def _render_rich(groups: dict, backend: str, console) -> None: +def _render_rich(groups: dict, console) -> None: from rich.table import Table from rich.text import Text console.print('[bold]infer-stack paths[/bold]') - console.print(f'backend: [bold cyan]{backend}[/bold cyan]') for group, entries in groups.items(): table = Table( title=group, @@ -143,7 +210,7 @@ class ConfigPathsCLI(_PathOverridesMixin): target: Any = scfg.Value( 'all', position=1, - help='Path group to show: all, config, data, or state.', + help='Path group to show: all, config, data, or leasing.', ) json: Any = scfg.Value( False, @@ -157,53 +224,48 @@ def main(cls, argv=True, **kwargs): _apply_path_overrides(config) target = str(config.target or 'all').strip().lower() - valid = {'all', 'config', 'data', 'state'} + valid = {'all', 'config', 'data', 'leasing'} if target not in valid: raise SystemExit( f'Unknown path group {target!r}. Expected one of: ' + ', '.join(sorted(valid)) ) - # Resolve against the persisted config when present, else defaults, - # so paths reflect the user's actual state.* / generated_dir choices. - cfg = _safe_load_config() - backend = backend_name(cfg) - groups: dict[str, list[dict[str, str]]] = {} if target in {'all', 'config'}: - entries = [ + groups['config'] = [ _entry('config_root', config_root(), kind='dir'), - _entry('config.yaml', config_path(), kind='file'), - _entry('models.yaml', models_path(), kind='file'), + _entry('settings.yaml', settings_path(), kind='file'), + _entry('catalog.yaml', config_root() / 'catalog.yaml', + kind='file'), + ] + if target in {'all', 'data'}: + groups['data'] = [ + _entry('data_root', data_root(), kind='dir'), + ] + if target in {'all', 'leasing'}: + compose_dir = data_root() / 'leasing' / 'compose' + groups['leasing'] = [ _entry( - 'kubeai_values_local', - kubeai_local_values_path(), + 'ledger', data_root() / 'leasing' / 'ledger.db', kind='file' + ), + _entry('compose_dir', compose_dir, kind='dir'), + _entry( + 'docker-compose.yml', + compose_dir / 'docker-compose.yml', kind='file', ), - ] - groups['config'] = entries - if target in {'all', 'data'}: - entries = [ - _entry('generated_dir', generated_dir(cfg), kind='dir'), - _entry('plan.yaml', plan_path(cfg), kind='file'), _entry( - 'runtime_dir', runtime_dir_for_config(cfg), kind='dir' + 'litellm_config.yaml', + compose_dir / 'litellm_config.yaml', + kind='file', + ), + _entry('env (secrets)', compose_dir / '.env', kind='file'), + _entry( + 'compose_state', + compose_dir / 'leasing-compose-state.json', + kind='file', ), - ] - if backend == 'kubeai': - entries.append( - _entry( - 'kubeai_generated_dir', - kubeai_generated_dir(cfg), - kind='dir', - ) - ) - groups['data'] = entries - if target in {'all', 'state'}: - state = normalized_state(cfg.get('state')) - groups['state'] = [ - _entry(name, Path(value), kind='dir') - for name, value in state.items() ] if config.json: @@ -214,15 +276,294 @@ def main(cls, argv=True, **kwargs): console = Console() if console.is_terminal: - _render_rich(groups, backend, console) + _render_rich(groups, console) else: - _render_plain(groups, backend) + _render_plain(groups) + return 0 + + +# --------------------------------------------------------------------------- +# config settings: durable defaults (backend, data_dir) in settings.yaml +# --------------------------------------------------------------------------- + +# The settings the leasing world honors. One registry drives `config init`'s +# prompts, `config set`'s validation, and the help — so adding a setting here is +# enough for `config init` to ask about it (no separate prompt to wire up). +@dataclass(frozen=True) +class _Setting: + key: str + label: str # short prompt label shown by `config init` + help: str # one-line description (config set / KNOWN_SETTINGS) + kind: str # 'path' | 'choice' | 'bool' + default: object = None + choices: tuple = () + + +_SETTINGS: tuple[_Setting, ...] = ( + _Setting( + 'data_dir', 'Data dir (docker-mounted weight/state)', + 'Where docker-mounted state lives (overrides the XDG default).', + 'path', + ), + _Setting( + 'backend', 'Default backend', + 'Default serving backend (compose | kubeai | null).', + 'choice', 'compose', ('compose', 'kubeai', 'null'), + ), + _Setting( + 'litellm', 'Render the LiteLLM gateway (one OpenAI base_url for all aliases)', + 'Render the LiteLLM gateway with the compose stack (true | false). On by ' + 'default; turn off for a lean stack where Open WebUI talks to the ' + 'upstreams (e.g. an Ollama daemon) directly.', + 'bool', True, + ), + _Setting( + 'ui', 'Manage an Open WebUI alongside the stack', + 'Render a managed Open WebUI with the compose stack (true | false).', + 'bool', True, + ), + _Setting( + 'skip_display_gpus', 'Skip display-attached GPUs (leave the monitor GPU free)', + 'Skip display-attached GPUs during placement (true | false). Off by ' + 'default — every GPU is used; turn on to leave a monitor GPU free.', + 'bool', False, + ), + _Setting( + 'reverse_proxy', 'Front the stack with a single-port HTTP reverse proxy', + 'Front the gateway + UI with one HTTP reverse proxy — UI at /, API at ' + '/v1 (true | false, or a {enabled, port, config_path} block via ' + '`config edit`). No TLS/auth — localhost / trusted networks only.', + 'bool', False, + ), +) + +# Keys the leasing world actually honors (others are allowed but warned about). +KNOWN_SETTINGS = {s.key: s.help for s in _SETTINGS} + + +def _as_bool(value, default: bool = False) -> bool: + if isinstance(value, bool): + return value + if value is None: + return default + return str(value).strip().lower() in ('1', 'true', 'yes', 'on') + + +class ConfigInitCLI(_PathOverridesMixin): + """Set up the durable settings (data dir + default backend). + + Prompts for each setting and shows them for confirmation before writing. Use + ``--yes`` for non-interactive scripting (accepts the provided flags / + current values / defaults without prompting); the same non-interactive path + is taken automatically when stdin is not a TTY. + + Re-running edits the existing config in place — it keeps your other settings + and just re-confirms the data dir and backend (or hand-edit the file with + ``infer-stack config edit``). Pass ``--fresh`` to discard the existing config + and start from defaults. + """ + + __command__ = 'init' + yes = scfg.Value( + False, isflag=True, alias=['y'], + help='Non-interactive: write without prompting/confirming.', + ) + fresh = scfg.Value( + False, isflag=True, + help='Start over: ignore any existing config and write a clean one from ' + 'defaults (discards other persisted settings too).', + ) + backend = scfg.Value( + None, choices=['compose', 'kubeai', 'null'], + help='Preset the default backend (skips that prompt).', + ) + + @classmethod + def main(cls, argv=True, **kwargs): + import sys + + from ..paths import ( + data_root, + load_settings, + save_settings, + settings_path, + ) + + config = cls.cli(argv=argv, data=kwargs) + _apply_path_overrides(config) + path = settings_path() + existing = load_settings() + + # Tell the user, every time, which mode this is (and how else to edit). + if config.fresh and existing: + print(f'config init: starting fresh — replacing the config at {path}') + elif existing: + print(f'config init: editing the existing config at {path} ' + '(or hand-edit it with `infer-stack config edit`)') + else: + print(f'config init: initializing a new config from scratch ({path})') + + # On --fresh, ignore the old file entirely (clean slate); otherwise keep + # other settings and seed the proposals from the current values. + base: dict = {} if config.fresh else dict(existing) + seed: dict = {} if config.fresh else existing + # Flags `config init` exposes directly (the rest are prompt/default only). + flags = {'data_dir': config.data_dir, 'backend': config.backend} + + def _proposed(s: _Setting): + # explicit flag > current setting > sensible default. + if flags.get(s.key) is not None: + return flags[s.key] + if s.key in seed: + return seed[s.key] + # data_root() resolves --data-dir override / $env / setting / XDG. + return str(data_root()) if s.key == 'data_dir' else s.default + + values = {s.key: _proposed(s) for s in _SETTINGS} + + interactive = ( + not config.yes and sys.stdin.isatty() and sys.stdout.isatty() + ) + if interactive: + from rich.console import Console + from rich.prompt import Confirm, Prompt + from rich.table import Table + + console = Console() + for s in _SETTINGS: + cur = values[s.key] + if s.kind == 'bool': + values[s.key] = Confirm.ask(s.label, default=_as_bool(cur)) + elif s.kind == 'choice': + values[s.key] = Prompt.ask( + s.label, choices=list(s.choices), default=str(cur) + ) + else: + values[s.key] = Prompt.ask(s.label, default=str(cur)) + table = Table(show_header=True, header_style='bold') + table.add_column('setting') + table.add_column('value', style='green') + for s in _SETTINGS: + table.add_row(s.key, str(values[s.key])) + console.print(table) + if not Confirm.ask(f'Write these to {path}?', default=True): + console.print('[yellow]aborted — nothing written[/]') + return 0 + + # Normalize bools so they persist as YAML true/false (not strings). + for s in _SETTINGS: + if s.kind == 'bool': + values[s.key] = _as_bool(values[s.key]) + base.update(values) + save_settings(base) + print(f'wrote settings -> {path}') + print('next: `infer-stack catalog init` to add models/endpoints') + return 0 + + +class ConfigSetCLI(_PathOverridesMixin): + """Persist a durable default, e.g. ``config set backend compose``.""" + + __command__ = 'set' + key = scfg.Value(None, position=1, type=str) + value = scfg.Value(None, position=2, type=str) + + @classmethod + def main(cls, argv=True, **kwargs): + import yaml + + from ..paths import load_settings, save_settings + + config = cls.cli(argv=argv, data=kwargs) + _apply_path_overrides(config) + if not config.key or config.value is None: + raise SystemExit('config set: KEY and VALUE are required') + if config.key not in KNOWN_SETTINGS: + print(f"warning: '{config.key}' is not a recognized setting " + f'(known: {", ".join(sorted(KNOWN_SETTINGS))})') + settings = load_settings() + settings[config.key] = yaml.safe_load(config.value) + path = save_settings(settings) + print(f"set {config.key} = {settings[config.key]!r} ({path})") + return 0 + + +class ConfigGetCLI(_PathOverridesMixin): + """Print one setting's value (or all settings).""" + + __command__ = 'get' + key = scfg.Value(None, position=1, type=str) + + @classmethod + def main(cls, argv=True, **kwargs): + from ..paths import load_settings + + config = cls.cli(argv=argv, data=kwargs) + _apply_path_overrides(config) + settings = load_settings() + if config.key: + if config.key not in settings: + raise SystemExit(f"'{config.key}' is not set") + print(settings[config.key]) + else: + for k, v in settings.items(): + print(f'{k}={v}') + return 0 + + +class ConfigShowCLI(_PathOverridesMixin): + """Show the persisted settings and where they live.""" + + __command__ = 'show' + + @classmethod + def main(cls, argv=True, **kwargs): + import yaml + + from ..paths import load_settings, settings_path + from .commands_catalog import _print_yaml + + config = cls.cli(argv=argv, data=kwargs) + _apply_path_overrides(config) + settings = load_settings() + print(f'# {settings_path()}') + if settings: + _print_yaml(yaml.safe_dump(settings, sort_keys=False)) + else: + print('(no settings yet — `infer-stack config set …`)') + return 0 + + +class ConfigEditCLI(_PathOverridesMixin): + """Open settings.yaml in $EDITOR.""" + + __command__ = 'edit' + + @classmethod + def main(cls, argv=True, **kwargs): + import os + import subprocess + + from ..paths import save_settings, settings_path + + config = cls.cli(argv=argv, data=kwargs) + _apply_path_overrides(config) + path = settings_path() + if not path.exists(): + save_settings({}) + subprocess.run([*os.environ.get('EDITOR', 'vi').split(), str(path)], + check=False) return 0 class ConfigModalCLI(scfg.ModalCLI): - """Inspect infer-stack configuration.""" + """Inspect + manage infer-stack configuration (paths + durable settings).""" __command__ = 'config' + init = ConfigInitCLI paths = ConfigPathsCLI + show = ConfigShowCLI + set = ConfigSetCLI + get = ConfigGetCLI + edit = ConfigEditCLI diff --git a/infer_stack/cli/commands_profile.py b/infer_stack/cli/commands_profile.py deleted file mode 100644 index 8606fcd2..00000000 --- a/infer_stack/cli/commands_profile.py +++ /dev/null @@ -1,496 +0,0 @@ -from __future__ import annotations - -from ..catalog import PROFILE_NAME_ALIASES -from ..catalog import profile_summary -from ..config import dump_yaml -from ..config import initial_config -from ..config import load_kubeai_resource_profiles -from ..config import load_yaml -from ..config import normalized_catalogs -from ..config import save_kubeai_resource_profiles -from ..config import save_yaml -from ..contracts import load_profile_contract -from ..diff_prompt import confirm_and_write -from ..kubeai_ops import deploy_rendered_artifacts -from ..renderer import render_from_lock -from ..verification import verify_profile -from pathlib import Path -from typing import Any -import json -import scriptconfig as scfg - -from .context import ( - _apply_path_overrides, - _arg_or_env, - _as_mapping, - apply_config_overrides, - backend_name, - build_plan, - config_for_runtime, - config_path, - effective_allow_unsupported, - effective_inventory, - ensure_renderable, - generated_dir, - kubeai_generated_dir, - load_config, - models_path, - plan_path, - runtime_dir_for_config, - save_plan, -) -from .compose import _compose_up_with_router_recreate -from .options import ( - _AllowUnsupportedMixin, - _BackendOverrideMixin, - _ClusterOverridesMixin, - _ComposeOverrideMixin, - _PathOverridesMixin, - _PlanOverridesCLI, - _PortOverridesMixin, - _ProfileOverrideMixin, - _SimulateHardwareMixin, - _SwitchPathOverridesCLI, -) - -# --------------------------------------------------------------------------- -# Profile / config management commands -# --------------------------------------------------------------------------- - - -class InitCLI(_PathOverridesMixin): - """Write a fresh config.yaml + empty models.yaml under config_root().""" - - force = scfg.Value( - False, isflag=True, help='Overwrite an existing config.yaml.' - ) - yes = scfg.Value( - False, - isflag=True, - short_alias=['y'], - help='Apply the writes without prompting. Without this, init shows a ' - 'per-file diff and asks for confirmation.', - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg_path = config_path() - if cfg_path.exists() and not config.force: - raise SystemExit( - 'config.yaml already exists. Use --force to overwrite.' - ) - planned = {cfg_path: dump_yaml(initial_config())} - if not models_path().exists(): - planned[models_path()] = dump_yaml({'models': {}, 'profiles': {}}) - if not confirm_and_write( - planned, assume_yes=bool(config.yes), title='Pending init' - ): - raise SystemExit('Aborted by user; no files were written.') - print(f'Wrote {cfg_path}') - return 0 - - -class SetupCLI( - _PathOverridesMixin, - _ProfileOverrideMixin, - _BackendOverrideMixin, - _ComposeOverrideMixin, - _PortOverridesMixin, - _ClusterOverridesMixin, -): - """Create / update config.yaml with the requested overrides applied.""" - - reset = scfg.Value( - False, - isflag=True, - help='Start from default config values before applying overrides.', - ) - resource_profiles_file = scfg.Value( - None, - type=str, - help='For kubeai setups, sync a local Helm values file with resourceProfiles into kubeai-values.local.yaml.', - ) - yes = scfg.Value( - False, - isflag=True, - short_alias=['y'], - help='Apply the config writes without prompting. Without this, setup ' - 'shows a per-file diff and asks for confirmation.', - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg_path = config_path() - if cfg_path.exists() and not config.reset: - cfg = load_yaml(cfg_path) - else: - cfg = initial_config() - cfg = apply_config_overrides(cfg, config) - planned = {cfg_path: dump_yaml(cfg)} - if not models_path().exists(): - planned[models_path()] = dump_yaml({'models': {}, 'profiles': {}}) - if not confirm_and_write( - planned, assume_yes=bool(config.yes), title='Pending setup' - ): - raise SystemExit('Aborted by user; no files were written.') - if config.resource_profiles_file: - # User-supplied path: anchor on CWD so a typed relative path - # behaves as the user expects. - source = Path(config.resource_profiles_file) - if not source.is_absolute(): - source = Path.cwd() / source - values_doc = load_yaml(source) - if 'resourceProfiles' not in values_doc: - raise SystemExit( - f'{source} is missing a top-level resourceProfiles map' - ) - target = save_kubeai_resource_profiles(values_doc) - if plan_path(cfg).exists(): - plan_path(cfg).unlink() - print(f'Wrote {target}') - print(f'Wrote {cfg_path}') - print( - f'Configured backend={cfg.get("backend", "compose")} ' - f'active_profile={cfg.get("active_profile", "") or ""}' - ) - return 0 - - -class ResolveCLI(_PlanOverridesCLI): - """Resolve the active profile into a deployment dict and print it.""" - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - plan = build_plan( - cfg, - profile_name=config.profile, - allow_unsupported=effective_allow_unsupported(config, cfg), - inventory=effective_inventory(config), - ) - save_plan(plan, cfg) - print(json.dumps(plan['deployment'], indent=2)) - return 0 - - -class ValidateCLI(_PlanOverridesCLI): - """Resolve + validate the active profile; exit non-zero on validation errors.""" - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - plan = build_plan( - cfg, - profile_name=config.profile, - allow_unsupported=effective_allow_unsupported(config, cfg), - inventory=effective_inventory(config), - ) - save_plan(plan, cfg) - print(json.dumps(plan['validated'], indent=2)) - return 0 if plan['validated']['ok'] else 2 - - -class LockCLI(_PlanOverridesCLI): - """Write plan.yaml after resolving + validating the active profile.""" - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - plan = build_plan( - cfg, - profile_name=config.profile, - allow_unsupported=effective_allow_unsupported(config, cfg), - inventory=effective_inventory(config), - ) - if not plan['validated']['ok'] and not plan['allow_unsupported']: - raise SystemExit( - 'Refusing to write plan.yaml because validation failed. Use --allow-unsupported to override.' - ) - save_plan(plan, cfg) - print(json.dumps(plan, indent=2)) - return 0 - - -class RenderCLI(_PlanOverridesCLI): - """Render the active profile's deployment into compose/kubeai artifacts.""" - - yes = scfg.Value( - False, - isflag=True, - short_alias=['y'], - help='Apply rendered changes without prompting. Without this, render shows a per-file diff and asks for confirmation.', - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - plan = build_plan( - cfg, - profile_name=config.profile, - allow_unsupported=effective_allow_unsupported(config, cfg), - inventory=effective_inventory(config), - ) - ensure_renderable(plan) - save_plan(plan, cfg) - render_from_lock(plan, assume_yes=bool(config.yes)) - print(f'Wrote {plan_path(cfg)}') - if backend_name(cfg) == 'kubeai': - print(f'Rendered KubeAI artifacts into {kubeai_generated_dir(cfg)}') - else: - print(f'Rendered Compose into {generated_dir(cfg)}') - print( - f'Rendered mounted runtime files into {runtime_dir_for_config(cfg)}' - ) - return 0 - - -class SwitchCLI(_SwitchPathOverridesCLI): - """Persist a new active_profile and re-render (optionally re-applying).""" - - profile = scfg.Value( - None, type=str, position=1, help='Profile name to switch to.' - ) - apply = scfg.Value( - False, - isflag=True, - help='Also apply the new profile to the running stack.', - ) - yes = scfg.Value( - False, - isflag=True, - short_alias=['y'], - help='Apply rendered changes without prompting.', - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - if not config.profile: - raise SystemExit('switch: missing required profile name') - persisted_cfg = load_config() - persisted_cfg['active_profile'] = config.profile - if not confirm_and_write( - {config_path(): dump_yaml(persisted_cfg)}, - assume_yes=bool(config.yes), - title='Pending active_profile change', - ): - raise SystemExit('Aborted by user; no files were written.') - cfg = apply_config_overrides(persisted_cfg, config) - - plan = build_plan( - cfg, - profile_name=config.profile, - allow_unsupported=effective_allow_unsupported(config, cfg), - inventory=effective_inventory(config), - ) - ensure_renderable(plan) - save_plan(plan, cfg) - render_from_lock(plan, assume_yes=bool(config.yes)) - if config.apply: - if backend_name(cfg) == 'compose': - _compose_up_with_router_recreate(cfg, detach=True) - else: - deploy_rendered_artifacts(plan['deployment']) - print(f'Switched active_profile to {config.profile}') - return 0 - - -class ListModelsCLI(_PathOverridesMixin): - """Print every model in the merged catalog.""" - - __command__ = 'list-models' - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = load_config() if config_path().exists() else initial_config() - cats = normalized_catalogs(cfg) - for name, model in cats.get('models', {}).items(): - ref = model.get('hf_model_id') or model.get('url', '') - print(f'{name}: {ref}') - return 0 - - -class ListProfilesCLI(_PathOverridesMixin): - """Print every serving profile in the merged catalog.""" - - __command__ = 'list-profiles' - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = load_config() if config_path().exists() else initial_config() - cats = normalized_catalogs(cfg) - profiles = cats.get('profiles', {}) - hidden_legacy = set(PROFILE_NAME_ALIASES) - for name, profile in profiles.items(): - if name in hidden_legacy: - continue - if profile.get('kind') == 'invalid-profile': - continue - summary = profile_summary(profile) - providers = ','.join(summary.get('providers', [])) or 'none' - print( - f'{name}: providers={providers} gateway={summary["gateway"]} ' - f'frontend={summary["frontend"]} frontend_provider={summary["frontend_provider"]} ' - f'routes={summary["route_count"]}' - ) - return 0 - - -class ExplainCLI(_PathOverridesMixin): - """Pretty-print a YAML file (defaults to the current plan.yaml).""" - - __command__ = 'explain' - - file = scfg.Value( - None, type=str, help='Path to read (default: current plan.yaml).' - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - if config.file: - target = Path(config.file) - if not target.is_absolute(): - target = Path.cwd() / target - else: - target = plan_path() - if not target.exists(): - raise SystemExit(f'Missing file: {target}') - print(json.dumps(load_yaml(target), indent=2)) - return 0 - - -class DescribeProfileCLI( - _PathOverridesMixin, - _BackendOverrideMixin, - _AllowUnsupportedMixin, - _SimulateHardwareMixin, -): - """Print the profile contract for a given profile name.""" - - __command__ = 'describe-profile' - - profile = scfg.Value( - None, type=str, position=1, help='Profile name to describe.' - ) - format = scfg.Value('yaml', choices=['json', 'yaml']) - output = scfg.Value( - None, type=str, help='Write to this file instead of stdout.' - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - overrides = _as_mapping(config) - if not overrides.get('profile'): - raise SystemExit('describe-profile: missing required profile name') - contract = load_profile_contract( - overrides['profile'], - backend=_arg_or_env(overrides, 'backend', 'INFER_STACK_BACKEND'), - simulate_hardware_spec=overrides.get('simulate_hardware'), - ) - return _print_structured( - contract, overrides['format'], overrides.get('output') - ) - - -def _print_structured( - data: dict[str, Any], fmt: str, output: str | None -) -> int: - if fmt == 'yaml': - import yaml - - text = yaml.safe_dump(data, sort_keys=False) - else: - text = json.dumps(data, indent=2) - if output: - target = Path(output) - target.parent.mkdir(parents=True, exist_ok=True) - target.write_text( - text + ('' if text.endswith('\n') else '\n'), encoding='utf-8' - ) - print(f'Wrote {target}') - return 0 - print(text) - return 0 - - -class VerifyProfileCLI(_SwitchPathOverridesCLI): - """Sanity-check a resolved profile (post-render expectations).""" - - __command__ = 'verify-profile' - - profile = scfg.Value( - None, type=str, position=1, help='Profile name to verify.' - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - if not config.profile: - raise SystemExit('verify-profile: missing required profile name') - cfg = config_for_runtime(config) - plan = build_plan( - cfg, - profile_name=config.profile, - allow_unsupported=effective_allow_unsupported(config, cfg), - inventory=effective_inventory(config), - ) - result = verify_profile(plan['deployment']) - print(json.dumps(result, indent=2)) - return 0 if result['ok'] else 2 - - -class KubeaiSyncResourceProfilesCLI(_PathOverridesMixin): - """Pull a Helm ``resourceProfiles`` values file into kubeai-values.local.yaml.""" - - __command__ = 'kubeai-sync-resource-profiles' - - from_file = scfg.Value( - None, - type=str, - required=True, - help='Helm values file with a top-level resourceProfiles map.', - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config, allow_missing=True) - # User-supplied path: anchor on CWD. - source = Path(config.from_file) - if not source.is_absolute(): - source = Path.cwd() / source - values_doc = load_yaml(source) - if 'resourceProfiles' not in values_doc: - raise SystemExit( - f'{source} is missing a top-level resourceProfiles map' - ) - target = save_kubeai_resource_profiles(values_doc) - if plan_path(cfg).exists(): - plan_path(cfg).unlink() - profiles, _, _ = load_kubeai_resource_profiles() - print(f'Wrote {target}') - print(f'Synced {len(profiles)} KubeAI resource profile(s)') - return 0 diff --git a/infer_stack/cli/commands_runtime.py b/infer_stack/cli/commands_runtime.py index 716497a4..b0cf1295 100644 --- a/infer_stack/cli/commands_runtime.py +++ b/infer_stack/cli/commands_runtime.py @@ -1,494 +1,264 @@ +"""Day-2 ops on the running leasing stack, plus a holistic ``status`` command. + +The ``stack`` wrappers (``logs`` / ``ps`` / ``up`` / ``down`` / …) target the +leasing Compose deployment — the project rendered under the data dir by +``acquire`` / ``apply``. ``status`` is the one-glance overview: where +everything lives, the active backend, and a leasing summary (active leases / live +deployments), with pointers to dig deeper. +""" + from __future__ import annotations -from ..config import load_yaml -from ..config import normalized_state -from ..docker_utils import DockerCommandError -from ..docker_utils import check_docker_compose_version -from ..docker_utils import compose_down -from ..docker_utils import compose_up -from ..docker_utils import docker_rm_dirs -from ..env_utils import parse_env_file -from ..kubeai_ops import CommandError -from ..kubeai_ops import deploy_rendered_artifacts -from ..kubeai_ops import print_status as kubeai_print_status +import subprocess from pathlib import Path from typing import Any -import scriptconfig as scfg -import shlex -import subprocess - -from .context import ( - _apply_path_overrides, - _as_mapping, - backend_name, - config_for_runtime, - config_path, - config_root, - generated_dir, - has_runtime_overrides, - kubeai_generated_dir, - plan_path, - render_is_stale, - runtime_env_path, -) -from .compose import ( - _compose_base_cmd, - _compose_up_with_router_recreate, - _kubeai_stub, -) -from .options import ( - _BackendOverrideMixin, - _ClusterOverridesMixin, - _ComposeOverrideMixin, - _PathOverridesMixin, - _PlanOverridesCLI, - _PortOverridesMixin, -) -from .commands_profile import RenderCLI - - -def _maybe_rerender(config: Any, cfg: dict[str, Any]) -> None: - """Re-run RenderCLI if runtime overrides changed or rendered outputs are stale. - Both ``up`` and ``deploy`` need this so the rendered artifacts always - match the current config + overrides before any container action. - """ - if has_runtime_overrides(config) or render_is_stale(cfg): - overrides = _as_mapping(config) - RenderCLI.main( - argv=False, - profile=overrides.get('profile'), - backend=overrides.get('backend'), - compose_cmd=overrides.get('compose_cmd'), - litellm_port=overrides.get('litellm_port'), - open_webui_port=overrides.get('open_webui_port'), - postgres_port=overrides.get('postgres_port'), - namespace=overrides.get('namespace'), - ingress_host=overrides.get('ingress_host'), - ingress_enabled=overrides.get('ingress_enabled'), - allow_unsupported=bool(overrides.get('allow_unsupported')), - simulate_hardware=overrides.get('simulate_hardware'), - allowed_gpus=overrides.get('allowed_gpus'), - yes=bool(overrides.get('yes')), - ) +import scriptconfig as scfg +from ..paths import config_root, data_root, get_setting, settings_path +from .context import _apply_path_overrides +from .options import _PathOverridesMixin # --------------------------------------------------------------------------- -# Runtime commands +# leasing compose project helpers (the target of the day-2 wrappers) # --------------------------------------------------------------------------- -class UpCLI(_PlanOverridesCLI): - """Bring the rendered compose stack up. Re-renders first if anything changed.""" +def _leasing_compose_file() -> Path: + from ..leasing.compose import COMPOSE_FILENAME - detach = scfg.Value( - False, - isflag=True, - short_alias=['d'], - help='Run in background instead of attaching to logs.', - ) - yes = scfg.Value( - False, - isflag=True, - short_alias=['y'], - help='If `up` triggers a re-render, apply changes without prompting.', - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - raise SystemExit( - '`up` only supports the compose backend. Use `deploy` for kubeai.' - ) - compose_cmd = cfg['runtime']['compose_cmd'] - check_docker_compose_version(compose_cmd) - _maybe_rerender(config, cfg) - _compose_up_with_router_recreate(cfg, detach=bool(config.detach)) - return 0 - - -class DownCLI( - _PathOverridesMixin, - _BackendOverrideMixin, - _ComposeOverrideMixin, - _PortOverridesMixin, -): - """Bring the rendered compose stack down (does not touch volumes).""" - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - raise SystemExit('`down` only supports the compose backend.') - # Re-render before down so a stale/invalid compose file from an older - # package version does not strand containers. Compose's - # --remove-orphans still removes services from the previous profile - # when the project name / generated directory is unchanged. - _maybe_rerender(config, cfg) - compose_cmd = cfg['runtime']['compose_cmd'] - check_docker_compose_version(compose_cmd) - try: - compose_down( - compose_cmd, - generated_dir(cfg) / 'docker-compose.yml', - runtime_env_path(cfg), - ) - except DockerCommandError as ex: - raise SystemExit( - f'compose down failed after re-rendering the current stack: {ex}' - ) from ex - return 0 + return data_root() / 'leasing' / 'compose' / COMPOSE_FILENAME -class PurgeCLI( - _PathOverridesMixin, - _BackendOverrideMixin, - _ComposeOverrideMixin, -): - """Stop the stack and delete all Docker-written state directories. +def _day2_compose_base(config, command: str) -> list[str]: + """``docker compose -p -f `` base for the wrappers. - Uses a temporary Alpine container to remove directories that Docker wrote - as root, avoiding ``Permission denied`` errors from plain ``rm -rf``. + Targets the leasing Compose deployment (no ``config.yaml`` needed). Raises a + helpful error when nothing has been deployed yet — the file is written by + ``acquire`` / ``apply``. """ - - yes = scfg.Value( - False, isflag=True, short_alias=['y'], help='Skip confirmation prompt.' - ) - delete_cache = scfg.Value( - False, - isflag=True, - help='Also delete hf-cache and vllm-cache (model weights). By default those are preserved.', - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config, allow_missing=True) - - state = normalized_state(cfg.get('state', {})) - always_delete = [ - 'postgres_litellm', - 'postgres_open_webui', - 'open_webui', - 'ollama', - 'runtime', - ] - model_dirs = ['hf_cache', 'vllm_cache'] - keys = ( - always_delete + model_dirs if config.delete_cache else always_delete + from ..leasing.compose import LEASING_PROJECT + + _apply_path_overrides(config) + compose_file = _leasing_compose_file() + if not compose_file.exists(): + raise SystemExit( + f'nothing deployed yet (no {compose_file}). ' + 'Bring a model up first, e.g. `infer-stack acquire `.' ) - dirs_to_delete = [ - Path(state[k]) for k in keys if Path(state[k]).exists() - ] - - if not dirs_to_delete: - print('Nothing to purge — state directories do not exist.') - return 0 - - if not config.yes: - print( - 'The following directories will be deleted (via Docker to handle root-owned files):' - ) - for d in dirs_to_delete: - print(f' {d}') - answer = input('Proceed? [y/N] ').strip().lower() - if answer not in {'y', 'yes'}: - print('Aborted.') - return 1 - - compose_file = generated_dir(cfg) / 'docker-compose.yml' - if compose_file.exists() and backend_name(cfg) == 'compose': - try: - compose_down( - cfg['runtime']['compose_cmd'], - compose_file, - runtime_env_path(cfg), - ) - except Exception as ex: - print( - f'Warning: compose down failed (containers may already be stopped): {ex}' - ) - - compose_cmd = cfg.get('runtime', {}).get( - 'compose_cmd', 'docker compose' - ) - docker_cmd = compose_cmd.split()[0] - docker_rm_dirs(dirs_to_delete, docker_cmd=docker_cmd) - print('Purge complete.') - return 0 - + return [ + 'docker', 'compose', '-p', LEASING_PROJECT, '-f', str(compose_file) + ] -class DeployCLI(_PlanOverridesCLI): - """Apply the rendered stack to its backend (kubeai apply / compose up).""" - - detach = scfg.Value(False, isflag=True, short_alias=['d']) - yes = scfg.Value( - False, - isflag=True, - short_alias=['y'], - help='If `deploy` triggers a re-render, apply changes without prompting.', - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - _maybe_rerender(config, cfg) - if backend_name(cfg) == 'kubeai': - plan = load_yaml(plan_path(cfg)) - try: - deploy_rendered_artifacts(plan['deployment']) - except CommandError as ex: - namespace = cfg.get('cluster', {}).get('namespace', 'kubeai') - raise SystemExit( - f'Failed to deploy to namespace {namespace!r}. Confirm ' - f'`infer-stack setup --backend kubeai --namespace {namespace}` ' - 'matches the namespace where the KubeAI Helm release is installed.\n' - f'Original error: {ex}' - ) from ex - return 0 - compose_up( - cfg['runtime']['compose_cmd'], - generated_dir(cfg) / 'docker-compose.yml', - generated_dir(cfg) / '.env', - detach=bool(config.detach), - remove_orphans=True, - ) - return 0 +# --------------------------------------------------------------------------- +# status — holistic overview +# --------------------------------------------------------------------------- -class EnvCLI( - _PathOverridesMixin, - _BackendOverrideMixin, -): - """Inspect the rendered .env file (path, single value, or eval-friendly export).""" - key = scfg.Value( - None, - type=str, - position=1, - help="Print only this variable's value. Empty = all.", - ) - export = scfg.Value( - False, - isflag=True, - help='Print `export KEY=value` lines suitable for `eval`.', - ) - path = scfg.Value( - False, - isflag=True, - help='Print only the absolute path to .env (default if no flags).', +def _catalog_summary(config) -> dict[str, Any] | None: + raw = getattr(config, 'catalog', None) or (config_root() / 'catalog.yaml') + path = Path(raw).expanduser() + info: dict[str, Any] = {'path': str(path), 'exists': path.exists()} + if path.exists(): + try: + from ..leasing import Catalog + + cat = Catalog.load(path) + info['models'] = len(cat.models) + info['endpoints'] = len(cat.endpoints) + except Exception: # noqa: BLE001 - status must never crash + info['models'] = info['endpoints'] = None + return info + + +def _leasing_status() -> dict[str, Any]: + """Read-only ledger snapshot for ``status`` (never mutates).""" + from ..leasing import ( + DeploymentState, + LeaseState, + Ledger, + SqliteStore, + default_ledger_path, ) - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - raise SystemExit('`env` only applies to the compose backend.') - env_file = runtime_env_path(cfg) - if not env_file.exists(): - raise SystemExit( - f'No .env at {env_file}. Run `infer-stack render` first.' - ) - # Default with no flags and no key: print the path so users can do - # `source $(infer-stack env)` (with `set -a` if they want export semantics). - if config.key is None and not config.export: - print(env_file) - return 0 - env = parse_env_file(env_file) - if config.key: - if config.key not in env: - raise SystemExit(f'{config.key!r} not found in {env_file}') - print(env[config.key]) - return 0 - # --export: emit eval-friendly export lines. - for k, v in env.items(): - print(f'export {k}={shlex.quote(v)}') - return 0 - - -def _enabled_components(deployment: dict[str, Any]) -> list[str]: - """Names of the components the resolved plan turned on.""" - out: list[str] = [] - providers = deployment.get('providers', {}) or {} - if (providers.get('ollama') or {}).get('enabled'): - out.append('ollama') - vllm = providers.get('vllm') or {} - if vllm.get('enabled'): - runtimes = vllm.get('runtimes') or {} - out.append(f'vllm({len(runtimes)})' if runtimes else 'vllm') - if ((deployment.get('gateways', {}) or {}).get('litellm') or {}).get( - 'enabled' - ): - out.append('litellm') - frontends = deployment.get('frontends', {}) or {} - if (frontends.get('open_webui') or {}).get('enabled'): - out.append('open_webui') - if (frontends.get('reverse_proxy') or {}).get('enabled'): - out.append('reverse_proxy') + path = default_ledger_path() + out: dict[str, Any] = {'path': str(path), 'exists': path.exists(), + 'leases': [], 'deployments': [], 'summary': None} + if not path.exists(): + return out + try: + leases, deployments = Ledger(SqliteStore(str(path))).status() + except Exception: # noqa: BLE001 + return out + active = sum(1 for le in leases if le.state == LeaseState.ACTIVE) + live = sum(1 for d in deployments if d.state == DeploymentState.LIVE) + out['leases'] = [ + (le.id, le.owner, str(le.state), ','.join(le.deployment_ids) or '-') + for le in leases + ] + out['deployments'] = [ + (d.id, str(d.state), ','.join(sorted(d.served)) or '-', d.demand) + for d in deployments + ] + out['summary'] = (active, len(leases), live, len(deployments)) return out -def _access_endpoints(access: dict[str, Any]) -> list[tuple[str, str]]: - """(name, base_url) pairs from the resolved access map.""" - out: list[tuple[str, str]] = [] - for name, entry in (access or {}).items(): - if isinstance(entry, dict) and entry.get('base_url'): - out.append((name, str(entry['base_url']))) - return out +def _gather_status(config) -> dict[str, Any]: + compose_file = _leasing_compose_file() + return { + 'backend': str(get_setting('backend') or 'null'), + 'data_dir': str(data_root()), + 'config_dir': str(config_root()), + 'configured': settings_path().exists(), + 'settings': {'path': str(settings_path()), + 'exists': settings_path().exists()}, + 'catalog': _catalog_summary(config), + 'compose': {'path': str(compose_file), 'exists': compose_file.exists()}, + 'leasing': _leasing_status(), + } + + +_DIG_DEEPER = ( + ('infer-stack leases', 'full lease + deployment tables'), + ('infer-stack tui', 'live dashboard (opt-in: infer-stack[tui])'), + ('infer-stack stack ps', 'running containers'), + ('infer-stack logs -f', 'tail service logs'), + ('infer-stack catalog show', 'what you can serve'), +) +_GETTING_STARTED = ( + ('infer-stack config init', 'storage + default backend'), + ('infer-stack catalog suggest --apply', 'a catalog sized to your GPUs'), + ('infer-stack acquire ', 'bring a model up'), +) -def _print_status_summary(cfg: dict[str, Any], *, initialized: bool) -> None: - """Print cheap, no-network context about the current stack. - Everything here comes from config.yaml, on-disk artifact existence, and the - already-resolved plan.yaml, so it never touches Docker, the network, or - hardware detection. - """ - backend = backend_name(cfg) +def _print_status_plain(d: dict[str, Any]) -> None: print('infer-stack status') - print() - print( - f' initialized: {"yes" if initialized else "no"} ({config_path()})' - ) - print(f' backend: {backend}') - print(f' active profile: {cfg.get("active_profile") or ""}') - print(f' config dir: {config_root()}') - - out_dir = ( - kubeai_generated_dir(cfg) if backend == 'kubeai' else generated_dir(cfg) - ) - rendered_marker = out_dir / ( - 'models.yaml' if backend == 'kubeai' else 'docker-compose.yml' - ) - print(f' generated dir: {out_dir}') - - if not initialized: + print(f' backend: {d["backend"]}') + print(f' data dir: {d["data_dir"]}') + print(f' config dir: {d["config_dir"]}') + cat = d['catalog'] + if cat and cat['exists']: + counts = '' + if cat.get('models') is not None: + counts = f' ({cat["models"]} models, {cat["endpoints"]} endpoints)' + print(f' catalog: {cat["path"]}{counts}') + else: + print(' catalog: (none — infer-stack catalog init)') + print(f' settings: {d["settings"]["path"]}' + f'{"" if d["configured"] else " (run infer-stack config init)"}') + lz = d['leasing'] + print(f' ledger: {lz["path"]}{"" if lz["exists"] else " (none yet)"}') + print(f' compose: {d["compose"]["path"]}' + f'{"" if d["compose"]["exists"] else " (not rendered yet)"}') + if lz['summary']: + active, total_l, live, total_d = lz['summary'] print() - print( - ' Not initialized — run ' - '`infer-stack setup --backend compose --profile ` ' - 'to create config.yaml.' - ) - return - - if rendered_marker.exists(): - stale = render_is_stale(cfg) - print( - f' rendered: yes{" (stale — run `infer-stack render`)" if stale else ""}' - ) + print(f'leasing: {active} active / {total_l} lease(s), ' + f'{live} live / {total_d} deployment(s) (infer-stack leases)') + if not d['configured']: + print() + for cmd, comment in _GETTING_STARTED: + print(f' {cmd:<38}{comment}') + + +def _print_status_rich(d: dict[str, Any], console) -> None: + from rich.table import Table + from rich.text import Text + + console.print(Text('infer-stack status', style='bold')) + table = Table(box=None, show_header=False, pad_edge=False, + padding=(0, 2, 0, 0)) + table.add_column(style='bold', justify='left', no_wrap=True) + table.add_column(overflow='fold') + + table.add_row('backend', Text(d['backend'], style='bold cyan')) + table.add_row('data dir', Text(d['data_dir'], style='cyan')) + table.add_row('config dir', Text(d['config_dir'], style='cyan')) + cat = d['catalog'] + if cat and cat['exists']: + val = Text(cat['path'], style='cyan') + if cat.get('models') is not None: + val.append(f' ({cat["models"]} models · {cat["endpoints"]} endpoints)', + style='dim') + table.add_row('catalog', val) else: - print(' rendered: no (run `infer-stack render`)') - - # The resolved view comes straight from plan.yaml (cheap file read; no - # hardware probe or re-resolution). - plan_file = plan_path(cfg) - if not plan_file.exists(): - return - try: - plan = load_yaml(plan_file) or {} - except Exception: - return - deployment = plan.get('deployment', {}) or {} - validated = plan.get('validated', {}) or {} - - description = (deployment.get('serving_profile', {}) or {}).get( - 'description' + table.add_row('catalog', Text('(none — infer-stack catalog init)', + style='yellow')) + settings = Text(d['settings']['path'], style='cyan') + if not d['configured']: + settings = Text('(run infer-stack config init)', style='yellow') + table.add_row('settings', settings) + lz = d['leasing'] + ledger = Text(lz['path'], style='cyan') + if not lz['exists']: + ledger.append(' (none yet)', style='dim') + table.add_row('ledger', ledger) + compose = Text(d['compose']['path'], style='cyan') + if not d['compose']['exists']: + compose.append(' (not rendered yet)', style='dim') + table.add_row('compose', compose) + console.print(table) + + if lz['summary']: + active, total_l, live, total_d = lz['summary'] + line = Text('leasing ', style='bold') + line.append(f'{active} active', style='green' if active else 'dim') + line.append(f' / {total_l} lease(s) ') + line.append(f'{live} live', style='green' if live else 'dim') + line.append(f' / {total_d} deployment(s)') + console.print() + console.print(line) + + console.print() + if d['configured']: + console.print(Text('dig deeper', style='bold')) + rows = _DIG_DEEPER + else: + console.print(Text('getting started', style='bold')) + rows = _GETTING_STARTED + for cmd, comment in rows: + line = Text(' ') + line.append(cmd.ljust(30), style='cyan') + line.append(comment, style='dim') + console.print(line) + + +class StatusCLI(_PathOverridesMixin): + """Holistic overview: where things live, the backend, and a leasing summary + (active leases / live deployments), with pointers to dig deeper.""" + + __command__ = 'status' + catalog = scfg.Value( + None, type=str, help='Catalog path (default: config dir).' ) - if description: - print(f' description: {description}') - - if validated: - errors = validated.get('errors') or [] - warnings = validated.get('warnings') or [] - if errors: - vstate = f'{len(errors)} error(s)' + ( - f', {len(warnings)} warning(s)' if warnings else '' - ) - elif warnings: - vstate = f'ok, {len(warnings)} warning(s)' - else: - vstate = 'ok' - print(f' validation: {vstate}') - - components = _enabled_components(deployment) - if components: - print(f' components: {", ".join(components)}') - - endpoints = _access_endpoints(deployment.get('access', {}) or {}) - if endpoints: - print(' endpoints:') - for name, url in endpoints: - print(f' {name}: {url}') - - -class StatusCLI( - _PathOverridesMixin, - _BackendOverrideMixin, - _ComposeOverrideMixin, - _PortOverridesMixin, - _ClusterOverridesMixin, -): - """Show stack status: where config/artifacts live, the active profile, - whether it has been rendered, the resolved components/endpoints, and the - live container/cluster state.""" @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) _apply_path_overrides(config) - initialized = config_path().exists() - cfg = config_for_runtime(config, allow_missing=True) - _print_status_summary(cfg, initialized=initialized) - if not initialized: - return 0 - - if backend_name(cfg) == 'kubeai': - namespace = cfg.get('cluster', {}).get('namespace', 'kubeai') - print() - print(f'cluster resources (namespace {namespace!r}):') - try: - kubeai_print_status(namespace) - except CommandError as ex: - raise SystemExit( - f'Failed to query KubeAI resources in namespace {namespace!r}. Confirm ' - f'`infer-stack setup --backend kubeai --namespace {namespace}` ' - 'matches the namespace where the KubeAI Helm release is installed.\n' - f'Original error: {ex}' - ) from ex - return 0 - - compose_file = generated_dir(cfg) / 'docker-compose.yml' - if not compose_file.exists(): - return 0 - compose_cmd = cfg['runtime']['compose_cmd'] - check_docker_compose_version(compose_cmd) - print() - print('containers (docker compose ps):') - proc = subprocess.run(_compose_base_cmd(cfg) + ['ps']) - return int(proc.returncode) + data = _gather_status(config) + from rich.console import Console + + console = Console() + if console.is_terminal: + _print_status_rich(data, console) + else: + _print_status_plain(data) + return 0 # --------------------------------------------------------------------------- -# Compose day-2-ops wrappers (raise NotImplementedError on kubeai) +# stack — docker compose day-2-ops wrappers over the leasing project # --------------------------------------------------------------------------- -class _ComposeWrapperBase( - _PathOverridesMixin, - _BackendOverrideMixin, - _ComposeOverrideMixin, -): - """Common fields for ``docker compose `` wrappers.""" +class _ComposeWrapperBase(_PathOverridesMixin): + """Common fields for ``docker compose `` wrappers over the leasing + Compose deployment.""" services = scfg.Value( None, @@ -499,17 +269,16 @@ class _ComposeWrapperBase( class LogsCLI(_ComposeWrapperBase): - """Tail rendered Compose service logs without typing the full docker compose path.""" + """Tail leasing Compose service logs without typing the full docker path.""" + + __command__ = 'logs' follow = scfg.Value( - False, - isflag=True, - short_alias=['f'], + False, isflag=True, short_alias=['f'], help='Stream logs (docker compose logs -f).', ) tail = scfg.Value( - None, - type=str, + None, type=str, help="Tail the last N lines (default: all). Pass a number or 'all'.", ) timestamps = scfg.Value(False, isflag=True) @@ -518,11 +287,7 @@ class LogsCLI(_ComposeWrapperBase): @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - _kubeai_stub('logs') - cmd = _compose_base_cmd(cfg) + ['logs'] + cmd = _day2_compose_base(config, 'logs') + ['logs'] if config.follow: cmd.append('--follow') if config.tail is not None: @@ -532,22 +297,19 @@ def main(cls, argv=True, **kwargs): if config.timestamps: cmd.append('--timestamps') cmd.extend(config.services or []) - proc = subprocess.run(cmd) - return int(proc.returncode) + return int(subprocess.run(cmd).returncode) class PsCLI(_ComposeWrapperBase): - """``docker compose ps`` for the rendered stack.""" + """``docker compose ps`` for the leasing deployment.""" + + __command__ = 'ps' all = scfg.Value( - False, - isflag=True, - short_alias=['a'], - help='Include stopped containers.', + False, isflag=True, short_alias=['a'], help='Include stopped containers.' ) services_only = scfg.Value( - False, - isflag=True, + False, isflag=True, help='Print only service names (passes --services to docker compose).', ) quiet = scfg.Value( @@ -557,11 +319,7 @@ class PsCLI(_ComposeWrapperBase): @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - _kubeai_stub('ps') - cmd = _compose_base_cmd(cfg) + ['ps'] + cmd = _day2_compose_base(config, 'ps') + ['ps'] if config.all: cmd.append('--all') if config.services_only: @@ -569,8 +327,7 @@ def main(cls, argv=True, **kwargs): if config.quiet: cmd.append('--quiet') cmd.extend(config.services or []) - proc = subprocess.run(cmd) - return int(proc.returncode) + return int(subprocess.run(cmd).returncode) class RestartCLI(_ComposeWrapperBase): @@ -581,16 +338,11 @@ class RestartCLI(_ComposeWrapperBase): @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - _kubeai_stub('restart') - cmd = _compose_base_cmd(cfg) + ['restart'] + cmd = _day2_compose_base(config, 'restart') + ['restart'] if config.timeout is not None: cmd.extend(['--timeout', str(config.timeout)]) cmd.extend(config.services or []) - proc = subprocess.run(cmd) - return int(proc.returncode) + return int(subprocess.run(cmd).returncode) class PullCLI(_ComposeWrapperBase): @@ -602,18 +354,13 @@ class PullCLI(_ComposeWrapperBase): @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - _kubeai_stub('pull') - cmd = _compose_base_cmd(cfg) + ['pull'] + cmd = _day2_compose_base(config, 'pull') + ['pull'] if config.quiet: cmd.append('--quiet') if config.ignore_pull_failures: cmd.append('--ignore-pull-failures') cmd.extend(config.services or []) - proc = subprocess.run(cmd) - return int(proc.returncode) + return int(subprocess.run(cmd).returncode) class StartCLI(_ComposeWrapperBase): @@ -622,14 +369,9 @@ class StartCLI(_ComposeWrapperBase): @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - _kubeai_stub('start') - cmd = _compose_base_cmd(cfg) + ['start'] + cmd = _day2_compose_base(config, 'start') + ['start'] cmd.extend(config.services or []) - proc = subprocess.run(cmd) - return int(proc.returncode) + return int(subprocess.run(cmd).returncode) class StopCLI(_ComposeWrapperBase): @@ -640,87 +382,60 @@ class StopCLI(_ComposeWrapperBase): @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - _kubeai_stub('stop') - cmd = _compose_base_cmd(cfg) + ['stop'] + cmd = _day2_compose_base(config, 'stop') + ['stop'] if config.timeout is not None: cmd.extend(['--timeout', str(config.timeout)]) cmd.extend(config.services or []) - proc = subprocess.run(cmd) - return int(proc.returncode) + return int(subprocess.run(cmd).returncode) -class OllamaPullCLI( - _PathOverridesMixin, _BackendOverrideMixin, _ComposeOverrideMixin -): - """Pull an Ollama model into the rendered Ollama model store.""" +class StackDownCLI(_ComposeWrapperBase): + """``docker compose down`` the leasing deployment. - __command__ = 'ollama-pull' + Tears the whole project down. Leasing's reconcile manages teardown + automatically on release; this is the manual escape hatch. + """ - model = scfg.Value( - None, - type=str, - position=1, - help='Ollama model tag to pull, e.g. smollm2:135m.', + volumes = scfg.Value( + False, isflag=True, help='Also remove named volumes (--volumes).' ) @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - if not config.model: - raise SystemExit( - 'ollama-pull: missing required model tag, e.g. `infer-stack ollama-pull smollm2:135m`' - ) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - raise SystemExit('`ollama-pull` only supports the compose backend.') - cmd = _compose_base_cmd(cfg) + [ - 'exec', - 'ollama', - 'ollama', - 'pull', - str(config.model), - ] - proc = subprocess.run(cmd) - return int(proc.returncode) - - -class OllamaListCLI( - _PathOverridesMixin, _BackendOverrideMixin, _ComposeOverrideMixin -): - """List models installed in the rendered Ollama service.""" - - __command__ = 'ollama-list' + cmd = _day2_compose_base(config, 'down') + ['down', '--remove-orphans'] + if config.volumes: + cmd.append('--volumes') + return int(subprocess.run(cmd).returncode) + + +class StackUpCLI(_ComposeWrapperBase): + """``docker compose up -d`` exactly what is on disk — the raw escape hatch. + + Brings up the on-disk compose file as-is, without touching the ledger or + re-rendering. Prefer ``infer-stack apply``, which re-renders from intent + first. Reach for ``stack up`` only to run a *hand-edited* compose file + verbatim. + """ @classmethod def main(cls, argv=True, **kwargs): config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - raise SystemExit('`ollama-list` only supports the compose backend.') - cmd = _compose_base_cmd(cfg) + ['exec', 'ollama', 'ollama', 'list'] - proc = subprocess.run(cmd) - return int(proc.returncode) + cmd = _day2_compose_base(config, 'up') + ['up', '-d', '--remove-orphans'] + cmd.extend(config.services or []) + return int(subprocess.run(cmd).returncode) -class OllamaPsCLI( - _PathOverridesMixin, _BackendOverrideMixin, _ComposeOverrideMixin -): - """Show loaded Ollama models for the rendered Ollama service.""" +class StackModalCLI(scfg.ModalCLI): + """Day-2 ops on the running leasing deployment.""" - __command__ = 'ollama-ps' + __command__ = 'stack' - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - if backend_name(cfg) != 'compose': - raise SystemExit('`ollama-ps` only supports the compose backend.') - cmd = _compose_base_cmd(cfg) + ['exec', 'ollama', 'ollama', 'ps'] - proc = subprocess.run(cmd) - return int(proc.returncode) + up = StackUpCLI + logs = LogsCLI + ps = PsCLI + restart = RestartCLI + pull = PullCLI + start = StartCLI + stop = StopCLI + down = StackDownCLI diff --git a/infer_stack/cli/commands_smoke.py b/infer_stack/cli/commands_smoke.py deleted file mode 100644 index 8ae62554..00000000 --- a/infer_stack/cli/commands_smoke.py +++ /dev/null @@ -1,691 +0,0 @@ -from __future__ import annotations - -from ..benchmark import run_benchmark -from ..env_utils import parse_env_file -from ..kubeai_ops import print_status as kubeai_print_status -from ..paths import config_root -from ..profile_runtime import default_base_url -from pathlib import Path -from typing import Any -import json -import requests -import scriptconfig as scfg - -from .context import ( - _apply_path_overrides, - _as_mapping, - backend_name, - build_plan, - config_for_runtime, - effective_allow_unsupported, - effective_inventory, - plan_path, - runtime_env_path, -) -from .probes import ( - _default_model_for_deployment, - _ready_ollama_probe, - _ready_openai_probe, - _resolve_smoke_protocol_from_deployment, -) -from .compose import ( - _explain_readiness_message, - _print_compose_diagnostics, - _print_gateway_diagnostics, -) -from .options import ( - _AllowUnsupportedMixin, - _BackendOverrideMixin, - _ClusterOverridesMixin, - _ComposeOverrideMixin, - _PathOverridesMixin, - _PortOverridesMixin, - _ProfileOverrideMixin, - _SimulateHardwareMixin, -) - -# --------------------------------------------------------------------------- -# Smoke-test / benchmark commands -# --------------------------------------------------------------------------- - - -def _wait_until_ready( - cfg: dict[str, Any], - config: Any, - *, - model: str | None = None, - timeout: float = 600.0, - interval: float = 5.0, - prompt: str = 'Reply with ready.', - max_tokens: int = 1, - require_generation: bool = True, - quiet: bool = False, -) -> str: - """Wait until the active profile can serve a real request. - - Docker Compose health only tells us that a process/container passed its - healthcheck. For vLLM, the API can exist before the model path is fully - ready through LiteLLM. This probes the user-facing access surface and, by - default, requires a tiny generation/completion to succeed. - """ - import time - - plan = _smoke_plan(cfg, config) - deployment = plan.get('deployment', {}) - access = deployment.get('access', {}).get('default', {}) or {} - access_kind = str(access.get('kind') or 'openai-compatible') - base_url = _infer_default_base_url(cfg, config, deployment=deployment) - model_name = _default_model_for_deployment(deployment, explicit=model) - deadline = time.monotonic() + float(timeout) - last_message = 'not probed yet' - attempt = 0 - - env = ( - parse_env_file(runtime_env_path(cfg)) - if backend_name(cfg) == 'compose' - else {} - ) - headers = {'Content-Type': 'application/json'} - if access_kind != 'ollama-native': - auth_env_name = str(access.get('auth_env_name') or 'LITELLM_MASTER_KEY') - api_key = ( - getattr(config, 'api_key', None) - or env.get(auth_env_name, '') - or env.get('LITELLM_MASTER_KEY', '') - ) - if api_key: - headers['Authorization'] = f'Bearer {api_key}' - - protocol = _resolve_smoke_protocol_from_deployment(deployment, model_name) - while True: - attempt += 1 - if access_kind == 'ollama-native': - ok, message = _ready_ollama_probe( - base_url=base_url, - model=model_name, - prompt=prompt, - max_tokens=max_tokens, - require_generation=require_generation, - ) - else: - ok, message = _ready_openai_probe( - base_url=base_url, - headers=headers, - model=model_name, - protocol=protocol, - prompt=prompt, - max_tokens=max_tokens, - require_generation=require_generation, - ) - last_message = message - if ok: - if not quiet: - print(f'Ready: {message}') - return message - now = time.monotonic() - if now >= deadline: - raise SystemExit( - 'Timed out waiting for the active stack to serve requests.\n' - f'Last probe: {_explain_readiness_message(last_message)}\n' - 'Useful diagnostics:\n' - ' infer-stack diagnose --logs --tail 80\n' - ' infer-stack ps\n' - ' infer-stack logs vllm-* litellm open-webui' - ) - if not quiet and (attempt == 1 or attempt % 6 == 0): - print( - f'Waiting for readiness: {_explain_readiness_message(last_message)}' - ) - time.sleep(float(interval)) - - -def _smoke_plan(cfg: dict[str, Any], config: Any) -> dict[str, Any]: - overrides = _as_mapping(config) - return build_plan( - cfg, - profile_name=overrides.get('profile'), - allow_unsupported=effective_allow_unsupported(config, cfg), - inventory=effective_inventory(config), - ) - - -def _infer_default_base_url( - cfg: dict[str, Any], config: Any, deployment: dict[str, Any] | None = None -) -> str: - explicit = _as_mapping(config).get('base_url') - if explicit: - return str(explicit).rstrip('/') - if deployment is None: - try: - deployment = _smoke_plan(cfg, config).get('deployment', {}) - except Exception: - deployment = { - 'backend': backend_name(cfg), - 'cluster': cfg.get('cluster', {}), - 'ports': cfg.get('ports', {}), - } - return default_base_url(deployment) - - -def _smoke_request( - method: str, - url: str, - *, - headers: dict[str, str] | None = None, - json_body: Any = None, - timeout: float = 30, - retries: int = 1, - retry_delay: float = 2.0, -) -> requests.Response: - """Wrapper around ``requests.{get,post}`` that emits actionable errors. - - The smoke test runs against a stack that may be (a) not listening yet, - (b) listening but with an unhealthy upstream that resets connections, or - (c) returning HTTP errors during model load. Retry transient startup - failures so ``switch --apply && smoke-test`` is usable immediately after a - provider container was recreated. - """ - last_timeout: requests.exceptions.Timeout | None = None - last_conn: requests.exceptions.ConnectionError | None = None - for attempt in range(1, max(1, retries) + 1): - try: - if method.upper() == 'GET': - resp = requests.get(url, headers=headers, timeout=timeout) - else: - resp = requests.post( - url, headers=headers, json=json_body, timeout=timeout - ) - break - except requests.exceptions.Timeout as ex: - last_timeout = ex - if attempt < retries: - import time - - time.sleep(retry_delay) - continue - raise SystemExit( - f'Request to {url} timed out after {timeout}s.\n' - 'The model may still be loading, or the server is overloaded.\n' - ' infer-stack logs vllm-*' - ) from ex - except requests.exceptions.ConnectionError as ex: - last_conn = ex - if attempt < retries: - import time - - time.sleep(retry_delay) - continue - # Two distinct sub-cases inside ConnectionError that warrant different - # remediation: (a) nothing listening on the port, (b) something is - # listening but it closed the connection without responding (typical - # of LiteLLM up but a depended-on vLLM container still loading the - # model and failing the dependency health-check chain). - cause = ex.args[0] if ex.args else ex - cause_str = str(cause) - if ( - 'RemoteDisconnected' in cause_str - or 'Connection aborted' in cause_str - ): - raise SystemExit( - f'Connection to {url} was closed before a response arrived.\n' - 'The router is listening but an upstream service is not ready yet.\n' - 'Check container status and logs:\n' - ' infer-stack ps\n' - ' infer-stack logs vllm-*' - ) from ex - if ( - 'Connection refused' in cause_str - or 'Failed to establish a new connection' in cause_str - ): - raise SystemExit( - f'Could not connect to {url}: nothing is listening yet.\n' - 'If you just ran `infer-stack up`, give the router a few seconds.\n' - ' infer-stack ps # confirm the litellm container is running\n' - ' infer-stack logs litellm # check for startup errors' - ) from ex - raise SystemExit( - f'Connection error reaching {url}: {cause_str}' - ) from ex - else: # pragma: no cover - defensive; loop exits via break or raise - if last_timeout is not None: - raise last_timeout - if last_conn is not None: - raise last_conn - raise RuntimeError('smoke request failed without an exception') - status = getattr(resp, 'status_code', 200) - if status >= 400: - body = getattr(resp, 'text', '') or '' - body = body.strip() - if len(body) > 500: - body = body[:500] + '... [truncated]' - reason = getattr(resp, 'reason', '') or '' - if status in (401, 403): - raise SystemExit( - f'{status} {reason} from {url}.\n' - "The auth key didn't match what the running container expects.\n" - 'If you re-rendered after the container started, the key in .env ' - 'may have changed. Restart with:\n' - ' infer-stack down && infer-stack up -d\n' - f'Response: {body}' - ) - if status == 503: - raise SystemExit( - f'{status} {reason} from {url}.\n' - 'An upstream service is unavailable (commonly the vLLM engine is still loading).\n' - ' infer-stack logs vllm-*\n' - f'Response: {body}' - ) - raise SystemExit( - f'HTTP {status} {reason} from {url}.\nResponse: {body}' - ) - return resp - - -def _resolve_smoke_test_protocol( - cfg: dict[str, Any], - config: Any, - model_name: str, -) -> str: - """Pick the OpenAI route for smoke-test based on protocol resolution order. - - 1. ``--protocol`` CLI override (``chat`` or ``completions``). - 2. Resolved deployment: if the requested model maps to a service whose - protocol_mode is known, use that. - 3. Active profile's primary service protocol_mode. - 4. Fallback: ``chat``. - """ - overrides = _as_mapping(config) - explicit = overrides.get('protocol') - if explicit: - return str(explicit) - try: - plan = build_plan( - cfg, - profile_name=overrides.get('profile'), - allow_unsupported=effective_allow_unsupported(config, cfg), - inventory=effective_inventory(config), - ) - except Exception: - return 'chat' - deployment = plan.get('deployment', {}) - return _resolve_smoke_protocol_from_deployment(deployment, model_name) - - -def _ollama_smoke_test( - base_url: str, - *, - model: str | None, - prompt: str, - max_tokens: int, - skip_chat: bool, -) -> int: - """Smoke-test an Ollama-native endpoint without requiring LiteLLM.""" - tags_resp = _smoke_request( - 'GET', f'{base_url}/api/tags', timeout=30, retries=12, retry_delay=5 - ) - tags_doc = tags_resp.json() - print(json.dumps(tags_doc, indent=2)) - if skip_chat: - return 0 - models = tags_doc.get('models') or [] - model_name = model or (models[0].get('name') if models else None) - if not model_name: - raise SystemExit( - 'Ollama is reachable, but no models are installed in its model store.\n' - 'Pull one through the CLI wrapper, for example:\n' - ' infer-stack ollama-pull smollm2:135m\n' - 'Then rerun:\n' - ' infer-stack smoke-test --model smollm2:135m' - ) - payload = { - 'model': model_name, - 'messages': [{'role': 'user', 'content': prompt}], - 'stream': False, - 'options': {'num_predict': max_tokens}, - } - resp = _smoke_request( - 'POST', - f'{base_url}/api/chat', - json_body=payload, - timeout=120, - retries=3, - retry_delay=5, - ) - print(json.dumps(resp.json(), indent=2)) - return 0 - - -class DiagnoseCLI( - _PathOverridesMixin, - _ProfileOverrideMixin, - _BackendOverrideMixin, - _PortOverridesMixin, - _ClusterOverridesMixin, - _AllowUnsupportedMixin, - _SimulateHardwareMixin, -): - """Print targeted diagnostics for the active rendered stack. - - This command is intentionally more specific than ``ps`` or ``logs``. It - prints the resolved provider/gateway/frontend graph, rendered compose - service state, LiteLLM route probes, direct provider probes, and optional - recent logs. It helps distinguish these cases: - - * LiteLLM container is actually absent/down. - * LiteLLM is running but its upstream vLLM process is still booting. - * Open WebUI is polling a provider that is not present in the active - profile. - """ - - __command__ = 'diagnose' - - model = scfg.Value( - None, - type=str, - help='Model/alias to use for optional generation diagnostics.', - ) - logs = scfg.Value( - False, - isflag=True, - help='Include recent logs for litellm/open-webui/vllm/ollama services.', - ) - tail = scfg.Value( - 80, type=int, help='Number of log lines per service when --logs is set.' - ) - generation = scfg.Value( - False, - isflag=True, - help='Also run a tiny generation probe through the active access surface.', - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - plan = build_plan( - cfg, - profile_name=_as_mapping(config).get('profile'), - allow_unsupported=effective_allow_unsupported(config, cfg), - inventory=effective_inventory(config), - ) - deployment = plan.get('deployment', {}) or {} - print( - f'active_profile: {deployment.get("source", {}).get("active_profile") or cfg.get("active_profile")}' - ) - print(f'backend: {deployment.get("backend") or backend_name(cfg)}') - print(f'plan: {plan_path(cfg)}') - access = (deployment.get('access', {}) or {}).get('default', {}) or {} - if access: - print('default access:') - print(f' kind: {access.get("kind")}') - print(f' base_url: {access.get("base_url")}') - if access.get('auth_env_name'): - print(f' auth_env_name: {access.get("auth_env_name")}') - - providers = deployment.get('providers', {}) or {} - gateways = deployment.get('gateways', {}) or {} - frontends = deployment.get('frontends', {}) or {} - print('\nresolved graph:') - print( - f' providers: {", ".join(k for k, v in providers.items() if (v or {}).get("enabled") or (v or {}).get("runtimes")) or "none"}' - ) - print( - f' gateways: {", ".join(k for k, v in gateways.items() if (v or {}).get("enabled")) or "none"}' - ) - print( - f' frontends: {", ".join(k for k, v in frontends.items() if (v or {}).get("enabled")) or "none"}' - ) - litellm_routes = (gateways.get('litellm') or {}).get('routes') or {} - if litellm_routes: - print('\nLiteLLM routes:') - for alias, route in litellm_routes.items(): - print( - f' {alias}: provider={route.get("provider")} ' - f'runtime={route.get("runtime", "-")} upstream={route.get("upstream_model", route.get("model", "-"))} ' - f'protocol={route.get("protocol_mode", "chat")}' - ) - - if backend_name(cfg) == 'compose': - _print_compose_diagnostics( - cfg, tail=int(config.tail) if config.logs else 0 - ) - _print_gateway_diagnostics( - cfg, - deployment, - model=config.model, - require_generation=bool(config.generation), - ) - else: - namespace = cfg.get('cluster', {}).get('namespace', 'kubeai') - kubeai_print_status(namespace) - return 0 - - -class WaitReadyCLI( - _PathOverridesMixin, - _ProfileOverrideMixin, - _BackendOverrideMixin, - _PortOverridesMixin, - _ClusterOverridesMixin, - _AllowUnsupportedMixin, - _SimulateHardwareMixin, -): - """Wait until the active profile can serve a real request. - - This is stronger than Docker Compose health. It probes the same access - surface users will hit (LiteLLM, direct Ollama, or direct vLLM) and, by - default, requires a tiny generation/completion to succeed. - """ - - __command__ = 'wait-ready' - - base_url = scfg.Value( - None, type=str, help='Override the resolved base URL.' - ) - api_key = scfg.Value( - None, - type=str, - help='Override the auth key for OpenAI-compatible surfaces.', - ) - model = scfg.Value( - None, - type=str, - help='Model/alias to probe. Defaults to the first active route/runtime.', - ) - prompt = scfg.Value('Reply with ready.', type=str) - max_tokens = scfg.Value(1, type=int) - timeout = scfg.Value(600, type=float, help='Maximum seconds to wait.') - interval = scfg.Value(5, type=float, help='Seconds between probes.') - skip_generation = scfg.Value( - False, - isflag=True, - help='Only wait for the API model listing/tag endpoint, not generation.', - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - _wait_until_ready( - cfg, - config, - model=config.model, - timeout=float(config.timeout), - interval=float(config.interval), - prompt=config.prompt, - max_tokens=int(config.max_tokens), - require_generation=not bool(config.skip_generation), - quiet=False, - ) - return 0 - - -class SmokeTestCLI( - _PathOverridesMixin, - _ProfileOverrideMixin, - _BackendOverrideMixin, - _PortOverridesMixin, - _ClusterOverridesMixin, - _AllowUnsupportedMixin, - _SimulateHardwareMixin, -): - """Probe the running router with a single chat/completions request.""" - - __command__ = 'smoke-test' - - base_url = scfg.Value(None, type=str) - api_key = scfg.Value(None, type=str) - model = scfg.Value(None, type=str) - prompt = scfg.Value('Say hello in one sentence.', type=str) - max_tokens = scfg.Value(128, type=int) - skip_chat = scfg.Value(False, isflag=True) - no_wait = scfg.Value( - False, - isflag=True, - help='Do not wait for the active access surface to serve a real request before the smoke request.', - ) - wait_timeout = scfg.Value( - 600, - type=float, - help='Seconds to wait for readiness before the smoke request.', - ) - wait_interval = scfg.Value( - 5, type=float, help='Seconds between readiness probes.' - ) - protocol = scfg.Value( - None, - choices=['chat', 'completions'], - help="Force the smoke-test endpoint. Defaults to the resolved profile's protocol_mode.", - ) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - cfg = config_for_runtime(config) - plan = _smoke_plan(cfg, config) - deployment = plan.get('deployment', {}) - access = deployment.get('access', {}).get('default', {}) or {} - env = ( - parse_env_file(runtime_env_path(cfg)) - if backend_name(cfg) == 'compose' - else {} - ) - base_url = _infer_default_base_url(cfg, config, deployment=deployment) - - if not bool(config.no_wait): - _wait_until_ready( - cfg, - config, - model=config.model, - timeout=float(config.wait_timeout), - interval=float(config.wait_interval), - prompt=config.prompt, - max_tokens=1, - require_generation=not bool(config.skip_chat), - quiet=True, - ) - - access_kind = str(access.get('kind') or 'openai-compatible') - explicit_base_url = bool(_as_mapping(config).get('base_url')) - if access_kind == 'ollama-native' and not explicit_base_url: - return _ollama_smoke_test( - base_url, - model=config.model, - prompt=config.prompt, - max_tokens=int(config.max_tokens), - skip_chat=bool(config.skip_chat), - ) - - headers = {'Content-Type': 'application/json'} - auth_env_name = str(access.get('auth_env_name') or 'LITELLM_MASTER_KEY') - api_key = ( - config.api_key - or env.get(auth_env_name, '') - or env.get('LITELLM_MASTER_KEY', '') - ) - if api_key: - headers['Authorization'] = f'Bearer {api_key}' - - models_resp = _smoke_request( - 'GET', - f'{base_url}/models', - headers=headers, - timeout=30, - retries=12, - retry_delay=5, - ) - models = models_resp.json().get('data', []) - print(json.dumps(models_resp.json(), indent=2)) - if config.skip_chat: - return 0 - if not models: - raise SystemExit('No models returned from /models') - model_name = config.model or models[0]['id'] - protocol = _resolve_smoke_test_protocol(cfg, config, model_name) - if protocol == 'completions': - payload = { - 'model': model_name, - 'prompt': config.prompt, - 'max_tokens': config.max_tokens, - } - endpoint = f'{base_url}/completions' - else: - payload = { - 'model': model_name, - 'messages': [{'role': 'user', 'content': config.prompt}], - 'max_tokens': config.max_tokens, - } - endpoint = f'{base_url}/chat/completions' - resp = _smoke_request( - 'POST', - endpoint, - headers=headers, - json_body=payload, - timeout=120, - retries=3, - retry_delay=5, - ) - print(json.dumps(resp.json(), indent=2)) - return 0 - - -class BenchmarkCLI( - _PathOverridesMixin, - _BackendOverrideMixin, - _ComposeOverrideMixin, - _PortOverridesMixin, -): - """Run benchmark_prompts.json against the router.""" - - model = scfg.Value(None, type=str, required=True) - base_url = scfg.Value(None, type=str) - api_key = scfg.Value(None, type=str) - - @classmethod - def main(cls, argv=True, **kwargs): - config = cls.cli(argv=argv, data=kwargs) - _apply_path_overrides(config) - # benchmark_prompts.json is a user-supplied fixture. Look for it - # first in the config dir, then fall back to CWD so an ad-hoc - # invocation from a checkout still picks up a sibling file. - prompts_path = config_root() / 'benchmark_prompts.json' - if not prompts_path.exists(): - prompts_path = Path.cwd() / 'benchmark_prompts.json' - if not prompts_path.exists(): - raise SystemExit( - f'benchmark_prompts.json not found at {config_root() / "benchmark_prompts.json"} ' - f'or {Path.cwd() / "benchmark_prompts.json"}' - ) - prompts = json.loads(prompts_path.read_text(encoding='utf-8')) - cfg = config_for_runtime(config) - env = parse_env_file(runtime_env_path(cfg)) - base_url = ( - config.base_url or f'http://127.0.0.1:{cfg["ports"]["litellm"]}/v1' - ) - api_key = config.api_key or env.get('LITELLM_MASTER_KEY', '') - data = run_benchmark(base_url, api_key, config.model, prompts) - print(json.dumps(data, indent=2)) - return 0 diff --git a/infer_stack/cli/compose.py b/infer_stack/cli/compose.py deleted file mode 100644 index 8556ab6c..00000000 --- a/infer_stack/cli/compose.py +++ /dev/null @@ -1,755 +0,0 @@ -from __future__ import annotations - -from ..config import load_yaml -from ..docker_utils import PortInUseError -from ..docker_utils import check_ports_available -from ..docker_utils import compose_recreate_router -from ..docker_utils import compose_up -from ..docker_utils import our_published_ports -from ..env_utils import parse_env_file -from pathlib import Path -from typing import Any -import json -import requests -import subprocess - -from .context import ( - generated_dir, - plan_path, - runtime_env_path, - runtime_litellm_config_path, -) -from .probes import ( - _default_model_for_deployment, - _ready_openai_probe, - _resolve_smoke_protocol_from_deployment, -) - -# Backend-specific helpers -# --------------------------------------------------------------------------- - - -def _compose_base_cmd(cfg: dict[str, Any]) -> list[str]: - """Build the shared ``docker compose -f ... --env-file ...`` prefix. - - Used by every compose-wrapper subcommand so the user doesn't have to - cd into the rendered-artifacts directory just to run a one-shot - ``ps`` / ``restart`` / ``pull`` / ``logs``. - """ - compose_file = generated_dir(cfg) / 'docker-compose.yml' - env_file = generated_dir(cfg) / '.env' - return cfg['runtime']['compose_cmd'].split() + [ - '-f', - str(compose_file), - '--env-file', - str(env_file), - ] - - -def _kubeai_stub(cmd_name: str) -> None: - """Raise for a day-2-ops subcommand that has no kubeai implementation yet. - - These wrappers (logs/ps/restart/pull/start/stop) compose docker-compose - invocations and have no kubectl equivalent in this CLI. Until somebody - writes one, surface the gap as ``NotImplementedError`` so callers can - distinguish "kubeai doesn't do this yet" from a real failure. - """ - raise NotImplementedError( - f'`{cmd_name}` is not implemented for the kubeai backend yet. ' - f'Use the equivalent kubectl command in the meantime ' - f'(e.g. `kubectl -n ...`).' - ) - - -def _compose_up_with_router_recreate( - cfg: dict[str, Any], - *, - detach: bool, -) -> None: - """Run ``compose up`` and refresh LiteLLM's model list to match the new render. - - A compose ``up`` against the rendered stack only restarts vLLM services - whose specs changed. LiteLLM and Open WebUI keep their existing - containers and would therefore serve stale model lists until something - else refreshed them. Two ways to do that: - - 1. **Live refresh** (preferred): talk to LiteLLM's admin API - (``/model/new`` / ``/model/delete``) to diff and apply alias changes - in-process. LiteLLM and Open WebUI stay up — users hitting unchanged - models see no disruption. Skipped automatically on cold start or if - the admin API is unreachable. - 2. **Container recreate** (fallback): force-recreate only the LiteLLM - container so it reloads the rendered YAML on startup. Open WebUI stays - up; used only when live refresh fails and Compose did not already - reload LiteLLM during convergence. - """ - compose_file = generated_dir(cfg) / 'docker-compose.yml' - env_file = generated_dir(cfg) / '.env' - compose_cmd = cfg['runtime']['compose_cmd'] - - _preflight_check_ports(cfg) - - litellm_in_render = _compose_has_service(compose_file, 'litellm') - litellm_before = ( - _compose_service_state(compose_cmd, compose_file, env_file, 'litellm') - if litellm_in_render - else {} - ) - - compose_up( - compose_cmd, - compose_file, - env_file, - detach=detach, - remove_orphans=True, - ) - - # If `up` failed it would have already raised; only do the router refresh - # in detached mode (foreground `up` keeps the user attached to logs and - # leaves cycling decisions to compose). - if not detach: - return - - if not runtime_litellm_config_path(cfg).exists(): - return - - if not litellm_in_render: - # Direct Ollama / raw-server profiles intentionally do not render a - # LiteLLM service. A stale litellm_config.yaml may still exist in the - # runtime directory from a previous gateway profile, but that must not - # trigger a router refresh/recreate against a service that is no longer - # present in the active compose file. - return - - litellm_after = _compose_service_state( - compose_cmd, compose_file, env_file, 'litellm' - ) - litellm_reloaded_by_compose = bool(litellm_after) and ( - litellm_after.get('id') != litellm_before.get('id') - or litellm_after.get('started_at') != litellm_before.get('started_at') - or litellm_before.get('running') not in {'true', 'True'} - ) - if litellm_reloaded_by_compose: - # Compose already created or restarted LiteLLM while converging the - # stack. The process has read the freshly rendered YAML, so a second - # live refresh or forced recreate is redundant churn. - print( - 'LiteLLM was started/reloaded by compose; skipping extra router refresh.' - ) - return - - try: - _litellm_refresh_router_live(cfg) - return - except RouterRefreshError as ex: - print( - f'Live router refresh skipped ({ex}); ' - 'recreating litellm container to reload config from YAML ' - '(open-webui stays up)...' - ) - - compose_recreate_router( - compose_cmd, - compose_file, - env_file, - detach=True, - ) - - -def _compose_has_service(compose_file: Path, service_name: str) -> bool: - """Return true when a rendered compose file contains ``service_name``. - - This is intentionally based on the rendered compose file rather than the - presence of sidecar artifacts such as ``runtime/litellm_config.yaml``. - Runtime artifacts are persistent across profile switches, while the compose - service list is the active source of truth for what ``docker compose up`` - can recreate. - """ - try: - doc = load_yaml(compose_file) - except FileNotFoundError: - return False - services = doc.get('services') or {} - return service_name in services - - -def _compose_service_container_id( - compose_cmd: str, - compose_file: Path, - env_file: Path, - service_name: str, -) -> str: - """Return the current container id for a rendered compose service.""" - return _compose_service_state( - compose_cmd, compose_file, env_file, service_name - ).get('id', '') - - -def _compose_service_state( - compose_cmd: str, - compose_file: Path, - env_file: Path, - service_name: str, -) -> dict[str, str]: - """Return a robust best-effort state snapshot for a compose service. - - Use JSON ``docker inspect`` rather than a Go template. The template form is - brittle for containers without a healthcheck: missing ``State.Health`` can - make ``docker inspect --format`` fail, causing diagnostics to show only an - id and the convergence logic to misclassify a still-running LiteLLM - container as newly reloaded. - - The returned fields are also used to diagnose ``exit code 137`` cases: when - a container disappears or restarts, ``oom_killed`` / ``exit_code`` make it - clear whether Docker killed it, Compose recreated it, or the process exited - normally. - """ - if not compose_file.exists(): - return {} - ps_cmd = compose_cmd.split() + [ - '-f', - str(compose_file), - '--env-file', - str(env_file), - 'ps', - '-q', - service_name, - ] - try: - proc = subprocess.run( - ps_cmd, capture_output=True, text=True, check=False, timeout=10 - ) - except (subprocess.SubprocessError, OSError): - return {} - if proc.returncode != 0 or not proc.stdout.strip(): - return {} - container_id = proc.stdout.strip().splitlines()[0] - inspect_cmd = ['docker', 'inspect', container_id] - try: - proc = subprocess.run( - inspect_cmd, capture_output=True, text=True, check=False, timeout=10 - ) - except (subprocess.SubprocessError, OSError): - return {'id': container_id} - if proc.returncode != 0 or not proc.stdout.strip(): - return {'id': container_id} - try: - payload = json.loads(proc.stdout)[0] - except (json.JSONDecodeError, IndexError, TypeError): - return {'id': container_id} - - state = payload.get('State') or {} - health = state.get('Health') or {} - return { - 'id': payload.get('Id') or container_id, - 'name': str(payload.get('Name') or '').lstrip('/'), - 'running': str(bool(state.get('Running'))).lower(), - 'status': str(state.get('Status') or ''), - 'health': str(health.get('Status') or 'none'), - 'started_at': str(state.get('StartedAt') or ''), - 'finished_at': str(state.get('FinishedAt') or ''), - 'exit_code': str( - state.get('ExitCode') if state.get('ExitCode') is not None else '' - ), - 'oom_killed': str(bool(state.get('OOMKilled'))).lower(), - 'restart_count': str( - payload.get('RestartCount') - if payload.get('RestartCount') is not None - else '' - ), - } - - -def _compose_rendered_service_names(compose_file: Path) -> list[str]: - """Return service names from the rendered compose file.""" - try: - doc = load_yaml(compose_file) - except FileNotFoundError: - return [] - return sorted((doc.get('services') or {}).keys()) - - -def _short_id(value: str) -> str: - """Shorten a docker container id for human diagnostics.""" - return value[:12] if value else '-' - - -def _http_probe_summary( - method: str, - url: str, - *, - headers: dict[str, str] | None = None, - json_body: Any | None = None, - timeout: float = 8.0, -) -> str: - """Return a concise one-line summary for a diagnostic HTTP probe.""" - try: - resp = requests.request( - method, url, headers=headers, json=json_body, timeout=timeout - ) - except requests.exceptions.RequestException as ex: - return f'ERR {type(ex).__name__}: {ex}' - body = (resp.text or '').strip().replace('\n', ' ') - if len(body) > 220: - body = body[:220] + '...' - return ( - f'HTTP {resp.status_code}: {body}' - if resp.status_code >= 400 - else f'HTTP {resp.status_code}' - ) - - -def _print_gateway_diagnostics( - cfg: dict[str, Any], - deployment: dict[str, Any], - *, - model: str | None = None, - require_generation: bool = False, -) -> None: - """Print targeted probes for the active gateway/provider graph.""" - env = ( - parse_env_file(runtime_env_path(cfg)) - if runtime_env_path(cfg).exists() - else {} - ) - ports = cfg.get('ports', {}) or {} - gateways = deployment.get('gateways', {}) or {} - providers = deployment.get('providers', {}) or {} - - litellm = gateways.get('litellm') or {} - if litellm.get('enabled'): - litellm_port = ports.get('litellm') - base = f'http://127.0.0.1:{litellm_port}' - key = env.get('LITELLM_MASTER_KEY', '') - headers = {'Authorization': f'Bearer {key}'} if key else {} - print('\nLiteLLM probes:') - print( - f' GET {base}/model/info -> {_http_probe_summary("GET", base + "/model/info", headers=headers)}' - ) - print( - f' GET {base}/v1/models -> {_http_probe_summary("GET", base + "/v1/models", headers=headers)}' - ) - if require_generation: - probe_model = model or _default_model_for_deployment(deployment) - protocol = _resolve_smoke_protocol_from_deployment( - deployment, probe_model - ) - ok, msg = _ready_openai_probe( - base_url=f'{base}/v1', - headers=headers, - model=probe_model, - protocol=protocol, - prompt='Reply with ready.', - max_tokens=1, - require_generation=True, - ) - status = 'OK' if ok else 'WAIT' - print( - f' generation probe ({probe_model}, {protocol}) -> {status}: {msg}' - ) - - vllm = providers.get('vllm') or {} - runtimes = vllm.get('runtimes') or {} - if runtimes: - print('\nvLLM provider probes:') - for name, rt in runtimes.items(): - service = rt.get('compose_service_name') or f'vllm-{name}' - host_port = rt.get('host_port') or ports.get('vllm') or 18000 - print( - f' {service}: model={rt.get("served_model_name")} protocol={rt.get("protocol_mode")} gpu={rt.get("gpu_indices")}' - ) - if host_port: - url = f'http://127.0.0.1:{host_port}/health' - print(f' GET {url} -> {_http_probe_summary("GET", url)}') - - ollama = providers.get('ollama') or {} - if ollama.get('enabled') and ollama.get('publish_port'): - port = ports.get('ollama') or 11434 - base = f'http://127.0.0.1:{port}' - print('\nOllama probes:') - print( - f' GET {base}/api/tags -> {_http_probe_summary("GET", base + "/api/tags")}' - ) - - -def _print_compose_diagnostics(cfg: dict[str, Any], *, tail: int = 0) -> None: - """Print compose state and optionally recent logs for diagnostic purposes.""" - compose_file = generated_dir(cfg) / 'docker-compose.yml' - env_file = runtime_env_path(cfg) - compose_cmd = cfg['runtime']['compose_cmd'] - services = _compose_rendered_service_names(compose_file) - if not services: - print(f'No rendered compose services found at {compose_file}') - return - print('\nCompose services:') - for svc in services: - state = _compose_service_state(compose_cmd, compose_file, env_file, svc) - if not state: - print(f' {svc:22s} absent') - continue - print( - f' {svc:22s} id={_short_id(state.get("id", ""))} ' - f'name={state.get("name", "-")} ' - f'running={state.get("running", "-")} status={state.get("status", "-")} ' - f'health={state.get("health", "-")} exit={state.get("exit_code", "-")} ' - f'oom={state.get("oom_killed", "-")} restarts={state.get("restart_count", "-")} ' - f'started_at={state.get("started_at", "-")}' - ) - if tail: - log_services = [ - svc - for svc in services - if svc == 'litellm' - or svc == 'open-webui' - or svc.startswith('vllm-') - or svc == 'ollama' - ] - if log_services: - print(f'\nRecent logs (--tail {tail}):') - cmd = _compose_base_cmd(cfg) + [ - 'logs', - '--tail', - str(tail), - *log_services, - ] - subprocess.run(cmd, check=False) - - -def _explain_readiness_message(msg: str) -> str: - """Add operator-facing interpretation to common readiness failures.""" - lower = msg.lower() - if ( - 'cannot connect to host litellm' in lower - or 'name or service not known' in lower - ): - return ( - msg - + '\n hint: the frontend or caller cannot resolve/reach the LiteLLM service. ' - 'Run `infer-stack diagnose --logs --tail 80` to check whether the active profile renders LiteLLM and whether the container is running. ' - 'If diagnose shows `oom=true` or `exit=137`, Docker killed LiteLLM rather than merely waiting on a vLLM upstream.' - ) - if ( - 'connection error' in lower - or 'connection refused' in lower - or 'cannot connect to host vllm' in lower - ): - return ( - msg - + '\n hint: LiteLLM is responding, but the upstream vLLM process is not serving yet. ' - 'This is expected while a single vLLM runtime is being replaced; wait-ready will keep polling.' - ) - return msg - - -class RouterRefreshError(RuntimeError): - """Live LiteLLM router refresh did not complete; caller should fall back.""" - - -def _resolve_env_refs(obj: Any, env: dict[str, str]) -> Any: - """Recursively replace ``os.environ/VAR`` strings with the value from ``env``. - - Mirrors LiteLLM's YAML-load substitution so we can feed the admin API - literal credentials. If a referenced variable isn't in ``env`` the original - string is left alone — caller will get the upstream error to debug. - """ - if isinstance(obj, str): - if obj.startswith('os.environ/'): - var = obj.removeprefix('os.environ/') - return env.get(var, obj) - return obj - if isinstance(obj, dict): - return {k: _resolve_env_refs(v, env) for k, v in obj.items()} - if isinstance(obj, list): - return [_resolve_env_refs(v, env) for v in obj] - return obj - - -def _litellm_delete_missed_config_model(resp: requests.Response) -> bool: - """Return true for LiteLLM's config-model delete miss response. - - ``GET /model/info`` returns both models loaded from config.yaml and - models inserted into LiteLLM's DB via ``/model/new``. In current LiteLLM - releases, ``POST /model/delete`` only deletes DB-backed models. When the - reported id belongs to a config-backed model, the delete endpoint returns a - 400/404 response whose body says the model id was not found in the DB. - That is not a proxy availability failure, so it should not trigger the - fallback path that restarts the LiteLLM container. - """ - try: - payload = resp.json() - except ValueError: - payload = resp.text - text = str(payload).lower() - return 'not found' in text and 'db' in text - - -def _litellm_refresh_router_live(cfg: dict[str, Any]) -> None: - """Sync the running LiteLLM router's model list to match the rendered YAML. - - Diffs ``GET /model/info`` (current state in the running container) against - the rendered ``litellm_config.yaml`` (desired state), then applies - ``POST /model/delete`` and ``POST /model/new`` for the differences. Aliases - that didn't change keep serving traffic without interruption. - - Raises ``RouterRefreshError`` on any failure (admin API unreachable, - auth missing, individual call fails); the caller falls back to the - full container-recreate path. - """ - import yaml as _yaml - - litellm_port = cfg.get('ports', {}).get('litellm') - if not litellm_port: - raise RouterRefreshError('litellm port not configured in cfg') - base = f'http://127.0.0.1:{litellm_port}' - - env = parse_env_file(runtime_env_path(cfg)) - master_key = env.get('LITELLM_MASTER_KEY', '').strip() - if not master_key: - raise RouterRefreshError( - 'LITELLM_MASTER_KEY missing from rendered .env' - ) - headers = { - 'Authorization': f'Bearer {master_key}', - 'Content-Type': 'application/json', - } - - config_path_ = runtime_litellm_config_path(cfg) - if not config_path_.exists(): - raise RouterRefreshError( - f'rendered litellm config not found at {config_path_}' - ) - try: - desired_doc = ( - _yaml.safe_load(config_path_.read_text(encoding='utf-8')) or {} - ) - except _yaml.YAMLError as ex: - raise RouterRefreshError( - f'could not parse {config_path_}: {ex}' - ) from ex - desired_models = desired_doc.get('model_list') or [] - - # The rendered YAML keeps secrets as `os.environ/VAR` references so the - # file itself isn't sensitive. LiteLLM resolves these only at YAML-load - # time on container startup — the admin API takes literal values. Inline - # the actual env values now so /model/new gets a usable upstream. - desired_models = [_resolve_env_refs(m, env) for m in desired_models] - - last_ex: requests.exceptions.RequestException | None = None - resp = None - max_attempts = 20 - for attempt in range(1, max_attempts + 1): - try: - resp = requests.get( - f'{base}/model/info', headers=headers, timeout=5 - ) - resp.raise_for_status() - break - except requests.exceptions.RequestException as ex: - last_ex = ex - if attempt < max_attempts: - import time - - time.sleep(1.5) - continue - raise RouterRefreshError( - f'GET /model/info failed after {attempt} attempts: {ex}' - ) from ex - assert resp is not None - current_models = resp.json().get('data') or [] - - # Key by alias (model_name). Within an alias, the "upstream" identity is - # litellm_params.model (e.g. "openai/qwen3.5-9b"). If that changes, the - # alias points to a different service and must be re-added; if it matches, - # the alias is untouched and continues serving. - def upstream_of(entry): - return (entry.get('litellm_params') or {}).get('model') - - current_by_alias = {m['model_name']: m for m in current_models} - desired_by_alias = {m['model_name']: m for m in desired_models} - - to_delete: list[tuple[str, str]] = [] - to_add: list[dict] = [] - - for alias, current in current_by_alias.items(): - desired = desired_by_alias.get(alias) - if desired is None: - to_delete.append((alias, current['model_info']['id'])) - elif upstream_of(current) != upstream_of(desired): - to_delete.append((alias, current['model_info']['id'])) - to_add.append(desired) - - for alias, desired in desired_by_alias.items(): - if alias not in current_by_alias: - to_add.append(desired) - - if not to_delete and not to_add: - return - - # Delete-before-add so the same alias can transition to a new upstream - # without LiteLLM rejecting a duplicate model_name. LiteLLM distinguishes - # config-file models from DB-backed models: /model/delete only applies to - # DB-backed rows. When an existing model was loaded from config.yaml, - # LiteLLM may report it in /model/info but return "not found in db" from - # /model/delete. Treat that as a non-fatal stale-config alias instead of - # forcing a LiteLLM container restart; the desired new aliases can still be - # added live, and a later manual LiteLLM restart will clean up the stale - # config-backed aliases if the operator cares about /v1/models hygiene. - stale_config_aliases: set[str] = set() - deleted_count = 0 - for alias, model_id in to_delete: - try: - resp = requests.post( - f'{base}/model/delete', - headers=headers, - json={'id': model_id}, - timeout=5, - ) - if resp.status_code in { - 400, - 404, - } and _litellm_delete_missed_config_model(resp): - stale_config_aliases.add(alias) - continue - resp.raise_for_status() - deleted_count += 1 - except requests.exceptions.RequestException as ex: - raise RouterRefreshError( - f'DELETE alias={alias} id={model_id} failed: {ex}' - ) from ex - - skipped_add_aliases: set[str] = set() - for model in to_add: - alias = model.get('model_name', '') - if alias in stale_config_aliases: - # Same alias, changed upstream, and the old alias is config-backed. - # Adding would collide and deleting would require a container restart. - skipped_add_aliases.add(alias) - continue - try: - resp = requests.post( - f'{base}/model/new', - headers=headers, - json=model, - timeout=5, - ) - resp.raise_for_status() - except requests.exceptions.RequestException as ex: - raise RouterRefreshError( - f'POST /model/new alias={alias} failed: {ex}' - ) from ex - - summary_parts = [] - if deleted_count: - summary_parts.append(f'removed {deleted_count} alias(es)') - added_count = len(to_add) - len(skipped_add_aliases) - if added_count: - summary_parts.append(f'added {added_count} alias(es)') - if stale_config_aliases: - summary_parts.append( - 'left ' - f'{len(stale_config_aliases)} stale config-backed alias(es) live ' - 'because LiteLLM would not delete them without a restart' - ) - if skipped_add_aliases: - summary_parts.append( - 'skipped ' - f'{len(skipped_add_aliases)} same-name update(s); ' - 'restart LiteLLM to replace those aliases' - ) - print(f'Live LiteLLM router refresh: {", ".join(summary_parts)}.') - - -def _preflight_check_ports(cfg: dict[str, Any]) -> None: - """Verify only the host ports the current rendered stack will publish.""" - ports = cfg.get('ports', {}) - candidates: list[tuple[str, int, str]] = [] - deployment: dict[str, Any] = {} - try: - if plan_path(cfg).exists(): - deployment = load_yaml(plan_path(cfg)).get('deployment', {}) - except Exception: - deployment = {} - - frontends = deployment.get('frontends', {}) or {} - gateways = deployment.get('gateways', {}) or {} - providers = deployment.get('providers', {}) or {} - - if not deployment: - # Fallback for very old rendered states; keep this conservative. - if ports.get('litellm'): - candidates.append(('litellm', int(ports['litellm']), '0.0.0.0')) - if ports.get('open_webui'): - candidates.append( - ('open-webui', int(ports['open_webui']), '0.0.0.0') - ) - else: - if (gateways.get('litellm') or {}).get('enabled') and ports.get( - 'litellm' - ): - candidates.append(('litellm', int(ports['litellm']), '0.0.0.0')) - if ( - (frontends.get('open_webui') or {}).get('enabled') - and (frontends.get('open_webui') or {}).get('publish_port', True) - and ports.get('open_webui') - ): - candidates.append( - ('open-webui', int(ports['open_webui']), '0.0.0.0') - ) - reverse_proxy = frontends.get('reverse_proxy') or {} - if reverse_proxy.get('enabled'): - if reverse_proxy.get('publish_http', True): - candidates.append( - ( - 'reverse-proxy-http', - int( - reverse_proxy.get('http_port') - or ports.get('reverse_proxy_http') - or 80 - ), - reverse_proxy.get('http_bind_host') or '0.0.0.0', - ) - ) - if reverse_proxy.get('publish_https', True): - candidates.append( - ( - 'reverse-proxy-https', - int( - reverse_proxy.get('https_port') - or ports.get('reverse_proxy_https') - or 443 - ), - reverse_proxy.get('https_bind_host') or '0.0.0.0', - ) - ) - ollama = providers.get('ollama') or {} - if ( - ollama.get('enabled') - and ollama.get('publish_port') - and ports.get('ollama') - ): - candidates.append(('ollama', int(ports['ollama']), '127.0.0.1')) - for name, rt in ( - (providers.get('vllm') or {}).get('runtimes') or {} - ).items(): - if rt.get('publish_port'): - candidates.append( - ( - f'vllm-{name}', - int(rt.get('host_port') or 18000), - '127.0.0.1', - ) - ) - - owned = our_published_ports( - cfg['runtime']['compose_cmd'], - generated_dir(cfg) / 'docker-compose.yml', - runtime_env_path(cfg), - ) - to_check = [ - (svc, port, host) for svc, port, host in candidates if port not in owned - ] - - try: - check_ports_available(to_check) - except PortInUseError as ex: - raise SystemExit(str(ex)) from ex diff --git a/infer_stack/cli/context.py b/infer_stack/cli/context.py index 155c9c39..3e01b14b 100644 --- a/infer_stack/cli/context.py +++ b/infer_stack/cli/context.py @@ -1,156 +1,26 @@ -from __future__ import annotations - -from ..config import CONFIG_FILE -from ..config import MODELS_FILE -from ..config import deep_merge -from ..config import default_output_config -from ..config import default_state_paths -from ..config import generated_dir_for_config -from ..config import initial_config -from ..config import kubeai_generated_dir_for_config -from ..config import kubeai_local_values_path -from ..config import load_yaml -from ..config import plan_path_for_config -from ..config import save_yaml -from ..hardware import detect_inventory -from ..hardware import simulate_inventory -from ..paths import CONFIG_DIR_ENV -from ..paths import config_root -from ..paths import data_root -from ..paths import set_config_root -from ..paths import set_data_root -from ..resolver import resolve -from ..validator import validate_resolved -from copy import deepcopy -from pathlib import Path -from typing import Any -import os - -# --------------------------------------------------------------------------- -# Path / config helpers -# --------------------------------------------------------------------------- - - -def config_path() -> Path: - return config_root() / CONFIG_FILE - - -def models_path() -> Path: - return config_root() / MODELS_FILE - - -def generated_dir(cfg: dict[str, Any] | None = None) -> Path: - cfg = cfg if cfg is not None else _safe_load_config() - return generated_dir_for_config(cfg) - - -def kubeai_generated_dir(cfg: dict[str, Any] | None = None) -> Path: - cfg = cfg if cfg is not None else _safe_load_config() - return kubeai_generated_dir_for_config(cfg) - - -def plan_path(cfg: dict[str, Any] | None = None) -> Path: - cfg = cfg if cfg is not None else _safe_load_config() - return plan_path_for_config(cfg) - - -def _hydrate_config_defaults(cfg: dict[str, Any] | None) -> dict[str, Any]: - """Merge persisted config on top of current defaults. - - The stack schema is still moving quickly. Users may already have a - config.yaml that predates a newly introduced component such as Ollama. - Loading through this helper keeps those configs valid by filling in new - default images, ports, provider toggles, state paths, and frontend/gateway - defaults without rewriting the user's file. - """ - return deep_merge(initial_config(), cfg or {}) - - -def _safe_load_config() -> dict[str, Any]: - """Load config.yaml if present; otherwise return defaults.""" - path = config_path() - if path.exists(): - return _hydrate_config_defaults(load_yaml(path)) - return initial_config() - - -def load_config() -> dict[str, Any]: - path = config_path() - if not path.exists(): - raise SystemExit( - f'No config.yaml found at {path}. Run ' - '`infer-stack setup --backend compose --profile qwen2-5-7b-instruct-turbo-default` first, ' - f'or point ${CONFIG_DIR_ENV} / --config-dir at an existing config.' - ) - return _hydrate_config_defaults(load_yaml(path)) - - -def runtime_dir_for_config(cfg: dict[str, Any]) -> Path: - state = cfg.get('state', {}) - runtime = state.get('runtime') - if not runtime: - return data_root() / 'runtime' - p = Path(runtime) - if p.is_absolute(): - return p - return data_root() / p - - -def runtime_env_path(cfg: dict[str, Any]) -> Path: - return generated_dir(cfg) / '.env' +"""Shared CLI helpers: path overrides + inventory resolution. +Trimmed to what the leasing surface needs after the pre-leasing profile world +was removed: ``_apply_path_overrides`` (honour ``--config-dir`` / ``--data-dir``) +and ``effective_inventory`` (honour ``--simulate-hardware`` / ``--allowed-gpus`` +so placement/suggest can plan for hardware you don't have in front of you). +""" -def runtime_litellm_config_path(cfg: dict[str, Any]) -> Path: - return runtime_dir_for_config(cfg) / 'litellm_config.yaml' - - -def backend_name(cfg: dict[str, Any]) -> str: - return str(cfg.get('backend', 'compose')).lower() - - -# --------------------------------------------------------------------------- -# Env-var / override resolution -# --------------------------------------------------------------------------- - - -def _env_text(name: str) -> str | None: - value = os.environ.get(name) - if value is None: - return None - text = value.strip() - return text or None - - -def _env_bool(name: str) -> bool | None: - value = _env_text(name) - if value is None: - return None - lowered = value.lower() - if lowered in {'1', 'true', 'yes', 'on', 'enabled'}: - return True - if lowered in {'0', 'false', 'no', 'off', 'disabled'}: - return False - raise SystemExit(f'Invalid boolean value for {name}: {value!r}') +from __future__ import annotations +import os +from typing import Any -def _env_int(name: str) -> int | None: - value = _env_text(name) - if value is None: - return None - try: - return int(value) - except ValueError as ex: - raise SystemExit(f'Invalid integer value for {name}: {value!r}') from ex +from ..hardware import detect_inventory, simulate_inventory +from ..paths import set_config_root, set_data_root def _as_mapping(args: Any) -> dict[str, Any]: """Coerce a CLI args object into a plain dict. Works for ``None``, ``argparse.Namespace``, and ``scfg.DataConfig`` - instances. Used to side-step name clashes between user-declared fields - and ``DataConfig`` builtins (e.g. ``namespace`` is a property on the - base class; ``getattr(cfg, 'namespace')`` returns the property, not the - field value, while ``asdict()['namespace']`` returns the field value). + instances. Used to side-step name clashes between user-declared fields and + ``DataConfig`` builtins. """ if args is None: return {} @@ -161,173 +31,19 @@ def _as_mapping(args: Any) -> dict[str, Any]: return dict(args) -def _arg_or_env( - args_dict: dict[str, Any], attr: str, env_name: str, *, caster=None -): - """Look up ``attr`` in the args dict, falling back to env var ``env_name``.""" - value = args_dict.get(attr) - if value is not None: - return value - env_value = _env_text(env_name) - if env_value is None: +def _env_text(name: str) -> str | None: + value = os.environ.get(name) + if value is None: return None - if caster is None: - return env_value - try: - return caster(env_value) - except ValueError as ex: - raise SystemExit(f'Invalid value for {env_name}: {env_value!r}') from ex - - -def apply_config_overrides( - cfg: dict[str, Any], args: Any | None -) -> dict[str, Any]: - """Merge runtime overrides (CLI args + env vars) on top of ``cfg``. - - ``args`` may be an ``argparse.Namespace``, a ``scfg.DataConfig`` instance, - or any mapping; it is coerced to a plain dict via ``_as_mapping``. - """ - if args is None: - return deepcopy(cfg) - overrides = _as_mapping(args) - out = deepcopy(cfg) - out.setdefault('runtime', {}) - out.setdefault('ports', {}) - out.setdefault('state', {}) - out.setdefault('output', {}) - out.setdefault('cluster', {}) - out['cluster'].setdefault('ingress', {}) - - backend = _arg_or_env(overrides, 'backend', 'INFER_STACK_BACKEND') - if backend: - out['backend'] = backend - - profile = _arg_or_env(overrides, 'profile', 'INFER_STACK_PROFILE') - if profile: - out['active_profile'] = profile - - compose_cmd = _arg_or_env( - overrides, 'compose_cmd', 'INFER_STACK_COMPOSE_CMD' - ) - if compose_cmd: - out['runtime']['compose_cmd'] = compose_cmd - - litellm_port = overrides.get('litellm_port') - if litellm_port is None: - litellm_port = _env_int('INFER_STACK_LITELLM_PORT') - if litellm_port is not None: - out['ports']['litellm'] = litellm_port - - open_webui_port = overrides.get('open_webui_port') - if open_webui_port is None: - open_webui_port = _env_int('INFER_STACK_OPEN_WEBUI_PORT') - if open_webui_port is not None: - out['ports']['open_webui'] = open_webui_port - - postgres_port = overrides.get('postgres_port') - if postgres_port is None: - postgres_port = _env_int('INFER_STACK_POSTGRES_PORT') - if postgres_port is not None: - out['ports']['postgres'] = postgres_port - - # All rendered artifacts and bind-mount state live under a single root - # (``--data-dir`` / ``INFER_STACK_DATA_DIR``), which is baked into the - # absolute ``state.*`` and ``output.generated_dir`` paths at setup time. - # Granular per-knob path overrides were removed in favour of that one root; - # edit ``state.*`` / ``output.generated_dir`` in config.yaml directly for - # bespoke split layouts. - # - # When the data root is *explicitly* relocated via ``--data-dir``, - # re-anchor the managed state tree + generated dir onto the new root. - # ``set_data_root`` has already been applied by ``_apply_path_overrides``, - # so the ``default_*`` helpers reflect the new location. Without this, - # ``setup --data-dir X`` against an existing config would silently no-op, - # since these paths are stored absolute and baked once at first setup. - # - # Triggered on the CLI flag only, not ``INFER_STACK_DATA_DIR``: the env var - # is the ambient anchor a fresh config already picks up, and regenerating - # from defaults on every plain ``setup`` would clobber hand-edited custom - # ``state.*`` paths. The flag is the unambiguous "relocate now" signal. - if overrides.get('data_dir'): - out['state'] = default_state_paths() - out['output']['generated_dir'] = default_output_config()[ - 'generated_dir' - ] - elif not out['output'].get('generated_dir'): - out['output']['generated_dir'] = default_output_config()[ - 'generated_dir' - ] - - namespace = _arg_or_env(overrides, 'namespace', 'INFER_STACK_NAMESPACE') - if namespace: - out['cluster']['namespace'] = namespace - - ingress_host = _arg_or_env( - overrides, 'ingress_host', 'INFER_STACK_INGRESS_HOST' - ) - if ingress_host: - out['cluster']['ingress']['host'] = ingress_host - - ingress_enabled = overrides.get('ingress_enabled') - if ingress_enabled is None: - ingress_enabled = _env_bool('INFER_STACK_INGRESS_ENABLED') - if ingress_enabled is not None: - out['cluster']['ingress']['enabled'] = bool(ingress_enabled) - - return out - - -_OVERRIDE_ATTRS = ( - 'profile', - 'backend', - 'compose_cmd', - 'litellm_port', - 'open_webui_port', - 'postgres_port', - 'namespace', - 'ingress_host', - 'ingress_enabled', - 'simulate_hardware', - 'allowed_gpus', -) - -_OVERRIDE_ENVS = ( - 'INFER_STACK_BACKEND', - 'INFER_STACK_PROFILE', - 'INFER_STACK_COMPOSE_CMD', - 'INFER_STACK_LITELLM_PORT', - 'INFER_STACK_OPEN_WEBUI_PORT', - 'INFER_STACK_POSTGRES_PORT', - 'INFER_STACK_NAMESPACE', - 'INFER_STACK_INGRESS_HOST', - 'INFER_STACK_INGRESS_ENABLED', - 'INFER_STACK_ALLOWED_GPUS', -) - - -def has_runtime_overrides(args: Any | None) -> bool: - if args is None: - return False - overrides = _as_mapping(args) - if any(overrides.get(attr) is not None for attr in _OVERRIDE_ATTRS): - return True - return any(_env_text(name) is not None for name in _OVERRIDE_ENVS) - - -def effective_allow_unsupported(args: Any | None, cfg: dict[str, Any]) -> bool: - overrides = _as_mapping(args) - arg_value = bool(overrides.get('allow_unsupported')) - policy_value = bool( - cfg.get('policy', {}).get('allow_unsupported_render', False) - ) - return arg_value or policy_value + text = value.strip() + return text or None def _parse_allowed_gpus(raw: Any) -> list[int] | None: """Parse a comma-separated list of GPU indices, or ``None`` if unset. - Accepts ints (when the value comes from ``data=`` kwargs in the - programmatic API), as well as strings of the form ``"1"`` or ``"1,3"``. + Accepts ints (when the value comes from ``data=`` kwargs in the programmatic + API), as well as strings of the form ``"1"`` or ``"1,3"``. """ if raw is None or raw == '': return None @@ -347,11 +63,9 @@ def _parse_allowed_gpus(raw: Any) -> list[int] | None: def _filter_inventory_to_allowed( inventory: dict[str, Any], allowed: list[int] | None ) -> dict[str, Any]: - """Return a new inventory containing only the GPUs whose ``index`` is in ``allowed``. + """Return a new inventory containing only the GPUs whose ``index`` is allowed. - Real indices are preserved — there is no renumbering — so a profile - that says ``placement.gpu_indices: [1, 3]`` still pins to physical - GPUs 1 and 3 after filtering. + Real indices are preserved — there is no renumbering. """ if not allowed: return inventory @@ -363,10 +77,10 @@ def _filter_inventory_to_allowed( def effective_inventory(args: Any | None) -> dict[str, Any] | None: - """Build the inventory the resolver should see, honoring CLI / env overrides. + """Build the inventory to plan against, honoring CLI / env overrides. - Returns ``None`` when nothing is constraining the inventory, so the - resolver falls back to ``detect_inventory()`` at plan time. + Returns ``None`` when nothing is constraining the inventory, so the caller + falls back to ``detect_inventory()`` at plan time. """ overrides = _as_mapping(args) spec = overrides.get('simulate_hardware') @@ -379,107 +93,6 @@ def effective_inventory(args: Any | None) -> dict[str, Any] | None: return _filter_inventory_to_allowed(base, allowed) -def config_for_runtime( - args: Any | None, *, allow_missing: bool = False -) -> dict[str, Any]: - if config_path().exists(): - cfg = load_config() - elif allow_missing: - cfg = initial_config() - else: - raise SystemExit( - f'No config.yaml found at {config_path()}. Run ' - '`infer-stack setup --backend compose --profile qwen2-5-7b-instruct-turbo-default` first, ' - f'or point ${CONFIG_DIR_ENV} / --config-dir at an existing config.' - ) - return apply_config_overrides(cfg, args) - - -# --------------------------------------------------------------------------- -# Plan helpers -# --------------------------------------------------------------------------- - - -def build_plan( - cfg: dict[str, Any], - *, - profile_name: str | None = None, - allow_unsupported: bool = False, - inventory: dict[str, Any] | None = None, -) -> dict[str, Any]: - resolved = resolve(cfg, inventory=inventory, profile_name=profile_name) - report = validate_resolved(resolved) - return { - 'schema_version': 1, - 'allow_unsupported': bool(allow_unsupported), - 'validated': report, - 'deployment': resolved, - } - - -def save_plan(plan: dict[str, Any], cfg: dict[str, Any] | None = None) -> Path: - path = plan_path(cfg) - save_yaml(path, plan) - return path - - -def ensure_renderable(plan: dict[str, Any]) -> None: - validated = plan.get('validated', {}) or {} - if validated.get('errors') and not plan.get('allow_unsupported', False): - raise SystemExit( - 'Refusing to render because the resolved plan contains validation errors. ' - 'Use `--allow-unsupported` to override.' - ) - - -def render_is_stale(cfg: dict[str, Any] | None = None) -> bool: - cfg = load_config() if cfg is None else cfg - cfg_path = config_path() - current_plan = plan_path(cfg) - backend = backend_name(cfg) - - if backend == 'kubeai': - kubeai_root = kubeai_generated_dir(cfg) - required_outputs = [ - current_plan, - kubeai_root / 'namespace.yaml', - kubeai_root / 'kubeai-values.yaml', - kubeai_root / 'models.yaml', - ] - else: - required_outputs = [ - current_plan, - generated_dir(cfg) / 'docker-compose.yml', - runtime_env_path(cfg), - ] - # litellm_config.yaml is optional now; direct Ollama/raw-server profiles - # intentionally do not render it. - - if any(not p.exists() for p in required_outputs): - return True - - if cfg_path.exists(): - oldest_generated = min(p.stat().st_mtime for p in required_outputs) - if cfg_path.stat().st_mtime > oldest_generated: - return True - if backend == 'kubeai': - local_values_path = kubeai_local_values_path() - if ( - local_values_path.exists() - and local_values_path.stat().st_mtime > oldest_generated - ): - return True - - if any( - current_plan.stat().st_mtime > p.stat().st_mtime - for p in required_outputs - if p != current_plan - ): - return True - return False - - -# --------------------------------------------------------------------------- def _apply_path_overrides(config: Any) -> None: """Honour ``--config-dir`` / ``--data-dir`` from a parsed subcommand config.""" overrides = _as_mapping(config) diff --git a/infer_stack/cli/options.py b/infer_stack/cli/options.py index 795a1e63..a633f84e 100644 --- a/infer_stack/cli/options.py +++ b/infer_stack/cli/options.py @@ -103,6 +103,21 @@ class _AllowedGpusMixin(scfg.DataConfig): ) +class _DisplayGpuMixin(scfg.DataConfig): + skip_display_gpus = scfg.Value( + None, + isflag=True, + alias=['skip-display-gpus'], + help=( + 'Skip display-attached GPUs during placement, leaving the GPU ' + 'driving a monitor free. OFF by default — placement uses every ' + 'GPU, so single-GPU and workstation hosts work out of the box. ' + 'Opt in per-command with this flag, or persist it with ' + '`infer-stack config set skip_display_gpus true`.' + ), + ) + + class _PlanOverridesCLI( _PathOverridesMixin, _ProfileOverrideMixin, diff --git a/infer_stack/config.py b/infer_stack/config.py index 31fa047d..5233d2e2 100644 --- a/infer_stack/config.py +++ b/infer_stack/config.py @@ -1,19 +1,12 @@ from __future__ import annotations from copy import deepcopy -from importlib.resources import files from pathlib import Path from typing import Any -import os import yaml -from .catalog import ( - normalize_ollama_models, - normalize_stack_profiles, - normalize_vllm_models, -) -from .paths import MODEL_PATH_ENV, config_root, data_root +from .paths import config_root, data_root CONFIG_FILE = Path('config.yaml') MODELS_FILE = Path('models.yaml') @@ -238,30 +231,6 @@ def save_yaml(path: Path, data: dict[str, Any]) -> None: path.write_text(dump_yaml(data), encoding='utf-8') -def _load_template_yaml(name: str) -> dict[str, Any]: - text = ( - files('infer_stack') - .joinpath(f'templates/{name}') - .read_text(encoding='utf-8') - ) - return yaml.safe_load(text) or {} - - -def builtin_vllm_models_catalog() -> dict[str, Any]: - return _load_template_yaml('default-vllm-models.yaml') - - -def builtin_ollama_models_catalog() -> dict[str, Any]: - return _load_template_yaml('default-ollama-models.yaml') - - -def builtin_profiles_catalog() -> dict[str, Any]: - return _load_template_yaml('default-profiles.yaml') - - -# Backwards-compatible helper name for callers that only know about vLLM models. -def builtin_models_catalog() -> dict[str, Any]: - return builtin_vllm_models_catalog() def deep_merge(a: dict[str, Any], b: dict[str, Any]) -> dict[str, Any]: @@ -275,316 +244,5 @@ def deep_merge(a: dict[str, Any], b: dict[str, Any]) -> dict[str, Any]: -def _as_list(value: Any) -> list[Any]: - """Return ``value`` as a list while treating strings as one entry.""" - if value in (None, ''): - return [] - if isinstance(value, list): - return list(value) - if isinstance(value, tuple): - return list(value) - return [value] - - -def _resolve_catalog_path(raw: Any, *, anchor: Path) -> Path: - path = Path(str(raw)).expanduser() - if not path.is_absolute(): - path = anchor / path - return path - - -def _split_model_path_env(raw: str | None) -> list[str]: - if not raw: - return [] - return [part for part in raw.split(os.pathsep) if part] - - -def _catalog_files_from_entry(entry: Path) -> list[Path]: - """Expand one catalog search-path entry into model/profile YAML files. - - Explicit file entries are loaded regardless of name. Directory entries are - scanned non-recursively for the intentionally narrow catalog names used by - infer-stack overlays: ``models.yaml``, ``*.models.yaml``, - ``*.profiles.yaml`` and their ``.yml`` variants. This avoids accidentally - loading unrelated YAML files - such as ``config.yaml`` or generated runtime files. - """ - if entry.is_file(): - return [entry] - if not entry.is_dir(): - return [] - files: list[Path] = [] - for primary_name in (str(MODELS_FILE), 'models.yml'): - primary = entry / primary_name - if primary.exists() and primary.is_file() and primary not in files: - files.append(primary) - for pattern in ( - '*.models.yaml', - '*.models.yml', - '*.profiles.yaml', - '*.profiles.yml', - ): - for candidate in sorted(entry.glob(pattern)): - if candidate.is_file() and candidate not in files: - files.append(candidate) - return files - - -def _catalog_entries_from_config( - catalog_cfg: dict[str, Any], *, anchor: Path -) -> list[Path]: - """Return catalog entries declared in config.yaml merge order. - - ``user_models_file`` preserves the historical single-catalog behavior. - ``model_path`` / ``model_paths`` add PATH-like overlay entries without - replacing the user catalog. Relative config entries are anchored on - ``config_root()`` because they are part of the infer-stack configuration, - not the caller's shell working directory. - """ - entries: list[Path] = [] - - raw_user_models = catalog_cfg.get('user_models_file', str(MODELS_FILE)) - for raw in _as_list(raw_user_models): - entries.append(_resolve_catalog_path(raw, anchor=anchor)) - - for key in ('model_path', 'model_paths'): - for raw in _as_list(catalog_cfg.get(key)): - entries.append(_resolve_catalog_path(raw, anchor=anchor)) - - return entries - - -def _catalog_entries_from_env(raw_env: str | None, *, anchor: Path) -> list[Path]: - """Return catalog entries declared by ``INFER_STACK_MODEL_PATH``. - - Relative environment entries are anchored on the process working directory, - matching the ergonomics of shell variables such as ``PATH`` and - ``PYTHONPATH``. - """ - return [ - _resolve_catalog_path(raw, anchor=anchor) - for raw in _split_model_path_env(raw_env) - ] -def catalog_files(config: dict[str, Any]) -> list[Path]: - """Return model/profile catalog overlay files in merge order. - - Merge precedence is deterministic and backward compatible: - - 1. ``catalog.user_models_file`` (defaults to ``models.yaml`` under - ``config_root()``) - 2. optional ``catalog.model_path`` / ``catalog.model_paths`` entries - 3. optional ``INFER_STACK_MODEL_PATH`` entries, split like ``PATH`` - - Later files override earlier ones via ``deep_merge``. - """ - catalog_cfg = config.get('catalog', {}) or {} - entries: list[Path] = [] - entries.extend( - _catalog_entries_from_config(catalog_cfg, anchor=config_root()) - ) - entries.extend( - _catalog_entries_from_env( - os.environ.get(MODEL_PATH_ENV), anchor=Path.cwd() - ) - ) - - files: list[Path] = [] - seen: set[Path] = set() - for entry in entries: - for candidate in _catalog_files_from_entry(entry): - key = candidate.resolve() - if key in seen: - continue - seen.add(key) - files.append(candidate) - return files - -def load_catalog_overlays(config: dict[str, Any]) -> dict[str, Any]: - """Load and merge all configured model/profile catalog overlay files.""" - merged: dict[str, Any] = {} - for path in catalog_files(config): - merged = deep_merge(merged, load_yaml(path)) - return merged - -def merged_catalogs(config: dict[str, Any]) -> dict[str, Any]: - catalog_cfg = config.get('catalog', {}) - built_vllm = ( - builtin_vllm_models_catalog() - if catalog_cfg.get('builtin_models', True) - else {} - ) - built_ollama = ( - builtin_ollama_models_catalog() - if catalog_cfg.get('builtin_models', True) - else {} - ) - built_profiles = ( - builtin_profiles_catalog() - if catalog_cfg.get('builtin_profiles', True) - else {} - ) - user = load_catalog_overlays(config) - - # User files may use the new provider-specific keys or the old generic - # `models` key, which is interpreted as vLLM models. - vllm_models = deep_merge( - built_vllm.get('vllm_models', built_vllm.get('models', {})), - user.get('vllm_models', user.get('models', {})), - ) - ollama_models = deep_merge( - built_ollama.get('ollama_models', {}), user.get('ollama_models', {}) - ) - profiles = deep_merge( - built_profiles.get('profiles', {}), user.get('profiles', {}) - ) - profiles = deep_merge(profiles, config.get('profiles', {})) - return { - 'vllm_models': vllm_models, - 'ollama_models': ollama_models, - 'profiles': profiles, - # Compatibility view. - 'models': vllm_models, - } - - -def normalized_catalogs(config: dict[str, Any]) -> dict[str, Any]: - catalogs = merged_catalogs(config) - vllm_models = normalize_vllm_models(catalogs.get('vllm_models', {})) - ollama_models = normalize_ollama_models(catalogs.get('ollama_models', {})) - profiles = normalize_stack_profiles( - catalogs.get('profiles', {}), vllm_models, ollama_models - ) - return { - 'vllm_models': vllm_models, - 'ollama_models': ollama_models, - 'models': vllm_models, - 'profiles': profiles, - } - - -def initial_config() -> dict[str, Any]: - # SmolLM2-135M is tiny (~270 MB) and runs on a single small GPU, so it is a - # safe out-of-the-box default regardless of detected hardware. - return { - 'name': 'local-llm-stack', - 'backend': 'compose', - 'active_profile': 'smollm2-135m-single', - 'catalog': { - 'builtin_models': True, - 'builtin_profiles': True, - 'user_models_file': str(MODELS_FILE), - }, - 'policy': { - 'require_fit_validation': True, - 'reserve_display_gpu': 'auto', - 'forbid_reserved_gpu_use': False, - 'require_homogeneous_multi_gpu_groups': True, - 'minimum_vram_headroom_gib': 2, - 'allow_unsupported_render': False, - }, - 'runtime': { - 'compose_cmd': 'docker compose', - 'target_inventory': 'auto', - }, - 'providers': { - 'ollama': {'enabled': 'auto'}, - 'vllm': {'enabled': 'auto'}, - }, - 'gateways': { - 'litellm': {'enabled': 'auto'}, - }, - 'frontends': { - 'open_webui': {'enabled': 'auto'}, - 'reverse_proxy': {'enabled': False}, - }, - 'ollama': { - 'publish_port': False, - 'host': '0.0.0.0:11434', - 'keep_alive': '2m', - 'context_length': 4096, - 'num_parallel': 1, - 'max_loaded_models': 1, - 'max_queue': 8, - 'gpu_indices': 'auto', - 'extra_env': {}, - 'env_file': [], - 'extra_volumes': [], - 'extra_hosts': [], - 'labels': {}, - }, - 'ports': deepcopy(DEFAULT_PORTS), - 'images': deepcopy(PINNED_IMAGES), - 'state': default_state_paths(), - 'output': default_output_config(), - 'open_webui': { - 'auth': False, - 'provider': 'auto', - 'publish_port': True, - 'extra_env': {}, - 'env_file': [], - 'extra_volumes': [], - 'extra_hosts': [], - 'labels': {}, - 'ldap': {'enabled': False}, - }, - 'reverse_proxy': { - 'enabled': False, - 'image': '', - 'container_name': 'reverse-proxy', - 'target': 'open_webui', - 'server_name': 'localhost', - 'publish_http': True, - 'publish_https': True, - 'http_port': None, - 'https_port': None, - 'http_bind_host': '', - 'https_bind_host': '', - 'force_https': True, - 'client_max_body_size': '1G', - 'proxy_connect_timeout': '60s', - 'proxy_read_timeout': '600s', - 'proxy_send_timeout': '600s', - 'proxy_buffer_size': '128k', - 'proxy_buffers': '4 256k', - 'proxy_busy_buffers_size': '256k', - 'proxy_buffering': None, - 'proxy_cache': None, - 'resolver': [], - 'resolver_timeout': '5s', - 'hsts': { - 'enabled': True, - 'max_age': 63072000, - 'include_subdomains': True, - 'preload': False, - }, - 'ssl': { - 'enabled': True, - 'certificate': '', - 'certificate_key': '', - 'dhparam': '', - 'certificate_container_path': '/etc/ssl/certs/infer-stack-site.crt', - 'certificate_key_container_path': '/etc/ssl/private/infer-stack-site.key', - 'dhparam_container_path': '/etc/ssl/certs/dhparam.pem', - 'protocols': 'TLSv1.2 TLSv1.3', - 'ciphers': 'ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384', - 'prefer_server_ciphers': True, - 'session_cache': 'shared:SSL:10m', - 'ecdh_curve': 'secp384r1', - 'session_tickets': False, - 'stapling': True, - 'stapling_verify': True, - }, - 'config_path': '', - 'extra_config': '', - 'extra_env': {}, - 'env_file': [], - 'extra_volumes': [], - 'extra_hosts': [], - 'labels': {}, - }, - 'cluster': default_cluster_config(), - 'resource_profiles': default_resource_profiles(), - 'profiles': {}, - } diff --git a/infer_stack/contracts.py b/infer_stack/contracts.py deleted file mode 100644 index 96bca714..00000000 --- a/infer_stack/contracts.py +++ /dev/null @@ -1,223 +0,0 @@ -from __future__ import annotations - -from copy import deepcopy -from typing import Any - -from .config import CONFIG_FILE, initial_config, load_yaml -from .hardware import simulate_inventory -from .paths import config_root -from .resolver import resolve -from .profile_runtime import default_base_url - - -def _default_access( - service: dict[str, Any], deployment: dict[str, Any] -) -> dict[str, Any]: - backend = str(deployment.get('backend', 'compose')).lower() - if backend == 'kubeai': - return { - 'name': 'kubeai-frontdoor', - 'kind': 'openai-compatible', - 'base_url': default_base_url(deployment), - 'request_model_name': service['profile_public_name'], - 'auth_env_name': 'KUBEAI_OPENAI_API_KEY', - 'auth_placeholder': 'EMPTY', - 'auth_required': False, - 'notes': [ - 'Use the KubeAI OpenAI-compatible front door for routed requests.' - ], - } - return { - 'name': 'compose-router', - 'kind': 'openai-compatible', - 'base_url': default_base_url(deployment), - 'request_model_name': service['served_model_name'], - 'auth_env_name': 'LITELLM_MASTER_KEY', - 'auth_placeholder': 'SET_LITELLM_MASTER_KEY_IN_ENV', - 'auth_required': True, - 'notes': ['Use the LiteLLM router front door for routed requests.'], - } - - -def _additional_accesses( - service: dict[str, Any], deployment: dict[str, Any] -) -> list[dict[str, Any]]: - benchmark_transport = deepcopy(service.get('benchmark_transport', {})) - if not benchmark_transport: - return [] - default = _default_access(service, deployment) - default_auth_env = ( - 'VLLM_API_KEY' - if benchmark_transport.get('kind') == 'vllm-direct' - else default['auth_env_name'] - ) - default_auth_placeholder = ( - 'EMPTY' - if benchmark_transport.get('kind') == 'vllm-direct' - else default['auth_placeholder'] - ) - access = { - 'name': benchmark_transport.get('name') - or benchmark_transport.get('kind') - or 'compatibility-access', - 'kind': benchmark_transport.get('kind') or default['kind'], - 'base_url': benchmark_transport.get('base_url') or default['base_url'], - 'request_model_name': benchmark_transport.get('request_model_name') - or service.get('hf_model_id') - or service['served_model_name'], - 'auth_env_name': benchmark_transport.get('api_key_env') - or default_auth_env, - 'auth_placeholder': benchmark_transport.get('api_key_placeholder') - or default_auth_placeholder, - 'auth_required': bool( - benchmark_transport.get( - 'api_key_required', default_auth_placeholder != 'EMPTY' - ) - ), - 'notes': [ - 'Optional compatibility access hint retained for external integrations.' - ], - } - dedupe_key = ( - access['kind'], - access['base_url'], - access['request_model_name'], - ) - default_key = ( - default['kind'], - default['base_url'], - default['request_model_name'], - ) - if dedupe_key == default_key: - return [] - return [access] - - -def build_profile_contract(deployment: dict[str, Any]) -> dict[str, Any]: - services = [] - for service in deployment.get('services', []): - services.append( - { - 'profile_name': service['profile_name'], - 'public_name': service['profile_public_name'], - 'service_name': service['service_name'], - 'kubernetes_name': service['kubernetes_name'], - 'model': { - 'model_ref': service['model_ref'], - 'hf_model_id': service['hf_model_id'], - 'logical_model_name': service['logical_model_name'], - 'served_model_name': service['served_model_name'], - 'served_aliases': deepcopy( - service.get('served_aliases', []) - ), - 'tokenizer_name': service['tokenizer_name'], - 'modalities': deepcopy(service.get('modalities', [])), - }, - 'protocol': { - 'mode': service['protocol_mode'], - 'engine': service['engine'], - 'features': deepcopy(service.get('features', [])), - }, - 'access': { - 'default': _default_access(service, deployment), - 'additional': _additional_accesses(service, deployment), - }, - 'runtime': { - 'resource_profile': service['resource_profile'], - 'priority_class_name': service.get('priority_class_name'), - 'min_replicas': service['min_replicas'], - 'max_replicas': service['max_replicas'], - 'tensor_parallel_size': service['tensor_parallel_size'], - 'data_parallel_size': service['data_parallel_size'], - 'max_model_len': service['max_model_len'], - 'gpu_memory_utilization': service['gpu_memory_utilization'], - 'max_num_batched_tokens': service['max_num_batched_tokens'], - 'max_num_seqs': service['max_num_seqs'], - 'enable_prefix_caching': service['enable_prefix_caching'], - 'extra_args': deepcopy(service.get('extra_args', [])), - }, - 'placement': { - 'gpu_indices': deepcopy(service.get('gpu_indices', [])), - 'placement': deepcopy(service.get('placement', {})), - 'topology': deepcopy(service.get('topology', {})), - }, - 'notes': deepcopy(service.get('notes', [])), - 'caveats': deepcopy(service.get('audit_notes', [])), - } - ) - return { - 'schema_version': 1, - 'kind': 'serving-profile-contract', - 'backend': deployment['backend'], - 'profile': { - 'name': deployment['serving_profile']['name'], - 'public_name': deployment['serving_profile']['public_name'], - 'description': deployment['serving_profile'].get('description', ''), - 'kubernetes_name': deployment['serving_profile'].get( - 'kubernetes_name', '' - ), - 'service_name': deployment['serving_profile'].get( - 'service_name', '' - ), - 'protocol_mode': deployment['serving_profile'].get( - 'protocol_mode', '' - ), - 'engine': deployment['serving_profile'].get('engine', ''), - 'resource_profile': deployment['serving_profile'].get( - 'resource_profile', '' - ), - 'notes': deepcopy(deployment['serving_profile'].get('notes', [])), - 'caveats': deepcopy( - deployment['serving_profile'].get('audit_notes', []) - ), - }, - 'router': { - 'type': deployment.get('router', {}).get('type', ''), - 'aliases': deepcopy( - deployment.get('router', {}).get('aliases', {}) - ), - }, - 'services': services, - } - - -def describe_profile_contract( - config: dict[str, Any], - *, - resolve_fn, - profile_name: str | None = None, - inventory: dict[str, Any] | None = None, -) -> dict[str, Any]: - deployment = resolve_fn( - config, inventory=inventory, profile_name=profile_name - ) - return build_profile_contract(deployment) - - -def load_profile_contract( - profile_name: str, - *, - backend: str | None = None, - simulate_hardware_spec: str | None = None, -) -> dict[str, Any]: - config_path = config_root() / CONFIG_FILE - if config_path.exists(): - config = load_yaml(config_path) - else: - config = initial_config() - config.setdefault('catalog', {}) - config['catalog']['builtin_models'] = True - config['catalog']['builtin_profiles'] = True - if backend is not None: - config['backend'] = backend - inventory = ( - simulate_inventory(simulate_hardware_spec) - if simulate_hardware_spec - else None - ) - return describe_profile_contract( - config, - resolve_fn=resolve, - profile_name=profile_name, - inventory=inventory, - ) diff --git a/infer_stack/diff_prompt.py b/infer_stack/diff_prompt.py index 23044dfd..d164db08 100644 --- a/infer_stack/diff_prompt.py +++ b/infer_stack/diff_prompt.py @@ -14,7 +14,6 @@ from rich.console import Console, Group from rich.panel import Panel from rich.prompt import Confirm -from rich.syntax import Syntax from rich.text import Text @@ -49,6 +48,8 @@ def confirm_writes( A planned write whose new content matches the existing file is shown as "no changes" and never blocks acceptance on its own. """ + from rich.syntax import Syntax + console = console or Console() panels: list[Panel] = [] diff --git a/infer_stack/hardware.py b/infer_stack/hardware.py index 9111a0fd..5fd58928 100644 --- a/infer_stack/hardware.py +++ b/infer_stack/hardware.py @@ -2,6 +2,7 @@ import csv import subprocess +from copy import deepcopy from typing import Any @@ -68,3 +69,59 @@ def detect_inventory() -> dict[str, Any]: 'gpu_count': len(gpus), 'gpus': gpus, } + + +# --------------------------------------------------------------------------- +# Low-level GPU-pool placement primitives. +# +# Shared by the legacy resolver (single profile) and the leasing placement +# planner (the union of live deployment groups), so there is one home for +# "which GPUs are available" and "first-fit N of them". +# --------------------------------------------------------------------------- + + +def available_gpu_indices( + inventory: dict[str, Any], reserve_display_gpu: str | bool | None +) -> list[int]: + """GPU indices in the inventory, optionally skipping display-active ones.""" + gpus = deepcopy(inventory.get('gpus', [])) + if reserve_display_gpu in ('auto', True): + return [g['index'] for g in gpus if not g.get('display_active')] + return [g['index'] for g in gpus] + + +def first_fit(available: list[int], count: int) -> tuple[list[int], str | None]: + """Take the first ``count`` available indices, or report the shortfall.""" + if len(available) < count: + return ( + available[:], + f'need {count} GPUs but only {len(available)} available', + ) + return available[:count], None + + +def resolve_gpu_indices( + *, + name: str, + placement: dict[str, Any], + topology: dict[str, Any], + preferred_gpu_count: int, + available: list[int], +) -> tuple[list[int], str | None]: + """Resolve a single runtime's GPU indices from its placement/topology.""" + strategy = placement.get('strategy', 'first_fit') + if strategy in {'exact', 'multi_gpu', 'single_gpu'}: + gpu_indices = list(placement.get('gpu_indices', [])) + if not gpu_indices: + return ( + [], + f'{name} uses {strategy} placement but no gpu_indices were provided', + ) + return gpu_indices, None + gpu_count = int( + placement.get( + 'gpu_count', + topology.get('tensor_parallel_size', preferred_gpu_count) or 1, + ) + ) + return first_fit(available, gpu_count) diff --git a/infer_stack/leasing/__init__.py b/infer_stack/leasing/__init__.py new file mode 100644 index 00000000..a2d3c926 --- /dev/null +++ b/infer_stack/leasing/__init__.py @@ -0,0 +1,93 @@ +"""Lease ledger: the stateful controller half of infer-stack. + +See ``dev/infer-stack-redesign-critique.md`` (aiq-eval-runner) for the design. +This subpackage is backend-agnostic bookkeeping — ``acquire``/``release``/ +``renew`` over a shared sqlite store with demand reference-counting, same-model +coalescing, and soft-TTL reclaim. The reconciler/backend layer (compose, +kubeai) and the CLI verbs are built on top of this in later phases. +""" + +from __future__ import annotations + +from .backend import Backend, MemoryBackend, NullBackend, Readiness +from .catalog import ( + Catalog, + CatalogError, + EndpointSpec, + ModelSpec, + RuntimeHostSpec, +) +from .compose import ComposeBackend, RenderedCompose, render_compose +from .controller import ( + AcquireOutcome, + Controller, + ReconcileResult, + ReleaseOutcome, + WaitResult, +) +from .ledger import ( + AcquireResult, + Ledger, + ReleaseResult, + SweepResult, + default_ledger_path, +) +from .models import ( + Deployment, + DeploymentState, + EndpointRequest, + Lease, + LeaseState, + Sharing, + capacity_satisfies, + compatibility_key, + ollama_structural, + vllm_structural, +) +from .placement import ( + GpuPlan, + available_indices, + plan_placement, + required_gpu_count, +) +from .store import SqliteStore + +__all__ = [ + 'AcquireOutcome', + 'AcquireResult', + 'Backend', + 'Catalog', + 'CatalogError', + 'ComposeBackend', + 'Controller', + 'Deployment', + 'EndpointRequest', + 'EndpointSpec', + 'GpuPlan', + 'DeploymentState', + 'Ledger', + 'Lease', + 'LeaseState', + 'MemoryBackend', + 'ModelSpec', + 'NullBackend', + 'Readiness', + 'ReconcileResult', + 'ReleaseOutcome', + 'ReleaseResult', + 'RenderedCompose', + 'RuntimeHostSpec', + 'Sharing', + 'SqliteStore', + 'SweepResult', + 'WaitResult', + 'available_indices', + 'capacity_satisfies', + 'compatibility_key', + 'default_ledger_path', + 'ollama_structural', + 'plan_placement', + 'render_compose', + 'required_gpu_count', + 'vllm_structural', +] diff --git a/infer_stack/leasing/backend.py b/infer_stack/leasing/backend.py new file mode 100644 index 00000000..c65e5461 --- /dev/null +++ b/infer_stack/leasing/backend.py @@ -0,0 +1,172 @@ +"""Backend protocol: the seam between the ledger and real serving. + +The ledger decides *what should be running* (desired deployment deployments); a +backend makes it so. The :class:`Controller` reconciles between them through the +four methods below. Keeping this surface tiny is deliberate — it is the only +thing a new backend (Compose, KubeAI, ...) must implement, and it is where the +redesign draws the line between "infer-stack coordinates" and "the backend / +KubeAI / k8s schedules". + +All four methods MUST be idempotent: the reconciler may ``realize`` a deployment that +is already up, or ``teardown`` one that is already gone. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Protocol, runtime_checkable + +from .models import Deployment + + +@dataclass +class Readiness: + """Result of a single readiness probe for one served endpoint.""" + + ready: bool + detail: str = '' + + +class ConvergeAborted(Exception): + """A backend's ``converge`` was declined by the user (diff not approved). + + Raised by an interactive backend when the operator rejects the pending + compose changes. The controller rolls back the just-created lease so a + declined ``acquire`` doesn't leave dangling ledger state. + """ + + +class PlacementError(Exception): + """An ``acquire`` requested a deployment the backend could not place. + + Raised by the controller when reconcile leaves one of the just-requested + deployments unplaced (e.g. no free GPU). Like :class:`ConvergeAborted`, the + controller rolls back the just-created lease before raising, so a request + that cannot be satisfied does not linger as a phantom ``live`` deployment with no + container behind it. ``reasons`` carries the planner's per-deployment messages. + """ + + def __init__(self, deployment_ids, reasons): + self.deployment_ids = list(deployment_ids) + self.reasons = list(reasons) + super().__init__( + '; '.join(self.reasons) + or f'could not place: {", ".join(self.deployment_ids)}' + ) + + +@runtime_checkable +class Backend(Protocol): + """What the :class:`Controller` needs from a serving backend. + + Implementations: :class:`MemoryBackend` (here, for tests/dry-runs), and + later ``ComposeBackend`` / ``KubeAIBackend``. + """ + + def realize(self, deployment: Deployment) -> None: + """Ensure a deployment for ``deployment`` exists and is converging.""" + ... + + def teardown(self, deployment: Deployment) -> None: + """Ensure ``deployment``'s deployment is stopped/removed.""" + ... + + def observe(self) -> set[str]: + """Return the set of deployment ids currently realized in the backend.""" + ... + + def probe_ready( + self, deployment: Deployment, endpoint: str + ) -> Readiness: + """Report whether one served ``endpoint`` of ``deployment`` is ready.""" + ... + + +class MemoryBackend: + """In-memory backend that records calls and has configurable readiness. + + Not a real serving backend — it never starts a process. It exists so the + controller's reconcile/wait logic can be tested deterministically, and as a + ``--dry-run`` backend. + + Example: + >>> from infer_stack.leasing.models import Deployment, DeploymentState + >>> b = MemoryBackend(ready=True) + >>> g = Deployment('grp-1', 'ck', 'vllm', 'shared-compatible', + ... {}, {}, {'qwen': {}}, DeploymentState.LIVE, 0.0, 0.0) + >>> b.realize(g); sorted(b.observe()) + ['grp-1'] + >>> b.probe_ready(g, 'qwen').ready + True + >>> b.teardown(g); sorted(b.observe()) + [] + """ + + def __init__(self, *, ready: bool = True): + self.ready_default = ready + self.realized: dict[str, Deployment] = {} + self.ready_overrides: dict[object, bool] = {} + self.realize_calls: list[str] = [] + self.teardown_calls: list[str] = [] + + def realize(self, deployment: Deployment) -> None: + self.realized[deployment.id] = deployment + self.realize_calls.append(deployment.id) + + def teardown(self, deployment: Deployment) -> None: + self.realized.pop(deployment.id, None) + self.teardown_calls.append(deployment.id) + + def observe(self) -> set[str]: + return set(self.realized) + + def probe_ready( + self, deployment: Deployment, endpoint: str + ) -> Readiness: + if deployment.id not in self.realized: + return Readiness(False, 'not realized') + ready = self.ready_overrides.get((deployment.id, endpoint)) + if ready is None: + ready = self.ready_overrides.get(deployment.id, self.ready_default) + return Readiness(bool(ready), 'ok' if ready else 'warming up') + + def set_ready( + self, + ready: bool, + *, + deployment_id: str | None = None, + endpoint: str | None = None, + ) -> None: + """Override readiness globally, per deployment, or per (deployment, endpoint).""" + if deployment_id is None: + self.ready_default = ready + elif endpoint is None: + self.ready_overrides[deployment_id] = ready + else: + self.ready_overrides[(deployment_id, endpoint)] = ready + + +class NullBackend: + """A no-op backend that serves nothing — for ``--dry-run`` and ``leases``. + + It never starts a process. ``observe`` returns the empty set, so the + controller treats every desired deployment as freshly realized (a no-op) and + never tears anything down; readiness is immediate. Because it keeps no + in-memory state, behaviour stays coherent across separate CLI invocations: + the persistent ledger is the only source of truth. Use this to exercise + acquire/release/run plumbing before the Compose backend exists. + """ + + def realize(self, deployment: Deployment) -> None: + pass + + def teardown(self, deployment: Deployment) -> None: + pass + + def observe(self) -> set[str]: + return set() + + def probe_ready( + self, deployment: Deployment, endpoint: str + ) -> Readiness: + return Readiness(True, 'dry-run') diff --git a/infer_stack/leasing/catalog.py b/infer_stack/leasing/catalog.py new file mode 100644 index 00000000..d33e8f0d --- /dev/null +++ b/infer_stack/leasing/catalog.py @@ -0,0 +1,415 @@ +"""The serving catalog: the declarative input to the leasing controller. + +This is the new, version-controlled, lifecycle-free description of *what can be +served* (the redesign demotes the old "profiles" singleton; see +``dev/infer-stack-redesign-critique.md`` §9). It parses a ``catalog.yaml`` into +typed specs and resolves an endpoint or bundle name into the +:class:`~infer_stack.leasing.models.EndpointRequest` objects the ledger already +consumes — so the catalog never has to know about leases, demand, or backends. + +Schema (all sections optional except as referenced):: + + models: # vLLM models (what exists) + qwen-coder-32b: + source: hf://Qwen/Qwen2.5-Coder-32B-Instruct + revision: main + quantization: null + dtype: null + + endpoints: # served API names (what users ask for) + qwen-coder: + model: qwen-coder-32b + engine: vllm + runtime: {tensor_parallel_size: 1, max_model_len: 32768} + sharing: {mode: shared-compatible} + reclaim: {policy: keep-warm} + qwen-small: + engine: ollama + host: local-ollama + model: qwen3.5:4b + + runtime_hosts: # Ollama daemons (one daemon, many tags) + local-ollama: + engine: ollama + placement: {gpu_indices: [1]} + settings: {keep_alive: 2m, num_parallel: 1, max_loaded_models: 2} + storage: {model_store: shared-ollama-store} + + bundles: # convenience: a named list of endpoints + draft-and-verify: [draft-model, verifier-model] +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import yaml + +from .models import ( + EndpointRequest, + Sharing, + ollama_structural, + vllm_structural, +) + +VLLM = 'vllm' +OLLAMA = 'ollama' + +DEFAULT_RECLAIM = 'keep-warm' + + +class CatalogError(ValueError): + """Raised when a catalog is structurally invalid or has dangling refs.""" + + +@dataclass +class ModelSpec: + """A vLLM model: what exists, independent of how it is run.""" + + name: str + source: str + revision: str | None = None + quantization: str | None = None + dtype: str | None = None + + @property + def hf_model_id(self) -> str: + """The Hugging Face id (``source`` minus any ``hf://`` scheme).""" + if '://' in self.source: + return self.source.split('://', 1)[1] + return self.source + + +@dataclass +class RuntimeHostSpec: + """A long-running engine instance. Today only Ollama needs an explicit one + (one daemon serves many tags); vLLM hosts are synthesized per endpoint.""" + + name: str + engine: str + gpu_indices: list[int] = field(default_factory=list) + settings: dict[str, Any] = field(default_factory=dict) + model_store: str | None = None + image: str | None = None + + +@dataclass +class EndpointSpec: + """A served API name → (model/tag, engine, runtime, sharing policy).""" + + name: str + engine: str + model: str + host: str | None = None + runtime: dict[str, Any] = field(default_factory=dict) + sharing: str = Sharing.SHARED + reclaim: str = DEFAULT_RECLAIM + served_name: str | None = None + + +def _parse_sharing(value: Any) -> str: + if value is None: + return Sharing.SHARED + if isinstance(value, str): + return value + if isinstance(value, dict): + return value.get('mode', Sharing.SHARED) + raise CatalogError(f'invalid sharing spec: {value!r}') + + +def _parse_reclaim(value: Any) -> str: + if value is None: + return DEFAULT_RECLAIM + if isinstance(value, str): + return value + if isinstance(value, dict): + return value.get('policy', DEFAULT_RECLAIM) + raise CatalogError(f'invalid reclaim spec: {value!r}') + + +@dataclass +class Catalog: + """A parsed, validated serving catalog.""" + + models: dict[str, ModelSpec] = field(default_factory=dict) + endpoints: dict[str, EndpointSpec] = field(default_factory=dict) + hosts: dict[str, RuntimeHostSpec] = field(default_factory=dict) + bundles: dict[str, list[str]] = field(default_factory=dict) + + # -- construction ------------------------------------------------------ + + @classmethod + def from_dict(cls, data: dict[str, Any] | None) -> Catalog: + """Parse and validate a catalog mapping. + + Example: + >>> cat = Catalog.from_dict({ + ... 'models': {'m': {'source': 'hf://org/m'}}, + ... 'endpoints': { + ... 'e': {'engine': 'vllm', 'model': 'm', + ... 'runtime': {'max_model_len': 8192}}}, + ... }) + >>> req = cat.resolve_endpoint('e') + >>> req.engine, req.capacity + ('vllm', {'max_model_len': 8192}) + """ + data = data or {} + models = { + name: ModelSpec( + name=name, + source=spec['source'], + revision=spec.get('revision'), + quantization=spec.get('quantization'), + dtype=spec.get('dtype'), + ) + for name, spec in (data.get('models') or {}).items() + } + hosts = {} + for name, spec in (data.get('runtime_hosts') or {}).items(): + spec = spec or {} + hosts[name] = RuntimeHostSpec( + name=name, + engine=spec.get('engine', OLLAMA), + gpu_indices=list((spec.get('placement') or {}).get('gpu_indices', [])), + settings=dict(spec.get('settings') or {}), + model_store=(spec.get('storage') or {}).get('model_store'), + image=spec.get('image'), + ) + endpoints = {} + for name, spec in (data.get('endpoints') or {}).items(): + spec = spec or {} + endpoints[name] = EndpointSpec( + name=name, + engine=spec.get('engine', VLLM), + model=spec.get('model'), + host=spec.get('host'), + runtime=dict(spec.get('runtime') or {}), + sharing=_parse_sharing(spec.get('sharing')), + reclaim=_parse_reclaim(spec.get('reclaim')), + served_name=spec.get('public_name') or spec.get('served_name'), + ) + bundles = { + name: list(members or []) + for name, members in (data.get('bundles') or {}).items() + } + catalog = cls( + models=models, endpoints=endpoints, hosts=hosts, bundles=bundles + ) + catalog.validate() + return catalog + + @classmethod + def load(cls, path: str | Path) -> Catalog: + text = Path(path).expanduser().read_text() + return cls.from_dict(yaml.safe_load(text)) + + # -- validation -------------------------------------------------------- + + def validate(self) -> None: + """Check engine values and cross-references; raise if any are wrong.""" + errors = self.errors() + if errors: + raise CatalogError('; '.join(errors)) + + def errors(self) -> list[str]: + errors: list[str] = [] + for ep in self.endpoints.values(): + if ep.engine == VLLM: + if not ep.model: + errors.append( + f"endpoint '{ep.name}' (vllm) needs a 'model'" + ) + elif ep.model not in self.models: + errors.append( + f"endpoint '{ep.name}' (vllm) references unknown " + f"model '{ep.model}'" + ) + elif ep.engine == OLLAMA: + if not ep.host: + errors.append( + f"endpoint '{ep.name}' (ollama) needs a 'host'" + ) + elif ep.host not in self.hosts: + errors.append( + f"endpoint '{ep.name}' (ollama) references unknown " + f"host '{ep.host}'" + ) + if not ep.model: + errors.append( + f"endpoint '{ep.name}' (ollama) needs a 'model' tag" + ) + else: + errors.append( + f"endpoint '{ep.name}' has unknown engine '{ep.engine}'" + ) + for bundle, members in self.bundles.items(): + for member in members: + if member not in self.endpoints: + errors.append( + f"bundle '{bundle}' references unknown endpoint " + f"'{member}'" + ) + return errors + + # -- resolution -------------------------------------------------------- + + def _unknown_endpoint_error(self, name: str) -> CatalogError: + """A helpful error for a name that isn't an endpoint. + + You acquire *endpoints*, not models — a common slip is to pass a + model name (``qwen05``) instead of one of its endpoints (``qwen05-1``). + Recognize that case and point at the endpoints that run the model; + otherwise fall back to a did-you-mean over the known endpoints/bundles. + """ + if name in self.models: + serving = sorted( + n for n, ep in self.endpoints.items() if ep.model == name + ) + if serving: + return CatalogError( + f"'{name}' is a model, not an endpoint — you bring up an " + f"endpoint that runs it. Endpoints for '{name}': " + f"{', '.join(serving)} (e.g. `infer-stack acquire " + f"{serving[0]}`)." + ) + return CatalogError( + f"'{name}' is a model with no endpoints yet — add one with " + f"`infer-stack catalog endpoint add --model {name}` " + f"(it defaults to the endpoint name '{name}-1')." + ) + + import difflib + + pool = list(self.endpoints) + list(self.bundles) + close = difflib.get_close_matches(name, pool, n=3, cutoff=0.6) + if close: + return CatalogError( + f"unknown endpoint '{name}' — did you mean: " + f"{', '.join(close)}?" + ) + if self.endpoints: + return CatalogError( + f"unknown endpoint '{name}'. Available endpoints: " + f"{', '.join(sorted(self.endpoints))} " + f"(`infer-stack catalog show` for details)." + ) + return CatalogError( + f"unknown endpoint '{name}' — the catalog has no endpoints yet " + f"(add one with `infer-stack catalog endpoint add …`)." + ) + + def resolve_endpoint( + self, name: str, *, sharing: str | None = None + ) -> EndpointRequest: + """Resolve one endpoint name into a ledger :class:`EndpointRequest`. + + ``sharing`` overrides the catalog's declared policy (e.g. the CLI + ``--dedicated`` flag) when given. + """ + if name not in self.endpoints: + raise self._unknown_endpoint_error(name) + ep = self.endpoints[name] + share = sharing or ep.sharing + if ep.engine == VLLM: + return self._resolve_vllm(ep, share) + if ep.engine == OLLAMA: + return self._resolve_ollama(ep, share) + raise CatalogError( + f"endpoint '{name}' has unknown engine '{ep.engine}'" + ) + + def resolve_names( + self, names: list[str], *, sharing: str | None = None + ) -> list[EndpointRequest]: + """Expand a mix of endpoint and bundle names into requests. + + Bundles expand to their member endpoints; duplicates (e.g. an endpoint + named directly and also via a bundle) are de-duplicated, preserving + order. + """ + ordered: list[str] = [] + for name in names: + members = self.bundles.get(name, [name]) + for member in members: + if member not in ordered: + ordered.append(member) + return [self.resolve_endpoint(n, sharing=sharing) for n in ordered] + + def _resolve_vllm( + self, ep: EndpointSpec, sharing: str + ) -> EndpointRequest: + model = self.models[ep.model] + rt = ep.runtime + served_name = ep.served_name or ep.name + structural = vllm_structural( + model_ref=model.source, + revision=model.revision, + quantization=model.quantization, + dtype=model.dtype, + tensor_parallel_size=rt.get('tensor_parallel_size', 1), + pipeline_parallel_size=rt.get('pipeline_parallel_size', 1), + image=rt.get('image'), + chat_template=rt.get('chat_template'), + trust_remote_code=rt.get('trust_remote_code', False), + lora_adapters=rt.get('lora_adapters'), + served_name=served_name, + ) + capacity: dict[str, Any] = {} + if rt.get('max_model_len') is not None: + capacity['max_model_len'] = rt['max_model_len'] + spec = { + 'engine': VLLM, + 'hf_model_id': model.hf_model_id, + 'served_model_name': served_name, + 'runtime': dict(rt), + 'reclaim': ep.reclaim, + } + served = { + 'served_model_name': served_name, + 'hf_model_id': model.hf_model_id, + } + return EndpointRequest( + endpoint=ep.name, + engine=VLLM, + structural=structural, + capacity=capacity, + sharing=sharing, + spec=spec, + served=served, + ) + + def _resolve_ollama( + self, ep: EndpointSpec, sharing: str + ) -> EndpointRequest: + host = self.hosts[ep.host] + settings = host.settings + structural = ollama_structural( + host=host.name, + gpu_indices=host.gpu_indices, + keep_alive=settings.get('keep_alive'), + num_parallel=settings.get('num_parallel'), + max_loaded_models=settings.get('max_loaded_models'), + model_store=host.model_store, + ) + spec = { + 'engine': OLLAMA, + 'host': host.name, + 'image': host.image, + 'gpu_indices': host.gpu_indices, + 'settings': dict(settings), + 'model_store': host.model_store, + 'reclaim': ep.reclaim, + } + served = {'model': ep.model} + return EndpointRequest( + endpoint=ep.name, + engine=OLLAMA, + structural=structural, + capacity={}, + sharing=sharing, + spec=spec, + served=served, + host=host.name, + ) diff --git a/infer_stack/leasing/compose.py b/infer_stack/leasing/compose.py new file mode 100644 index 00000000..9b4e4663 --- /dev/null +++ b/infer_stack/leasing/compose.py @@ -0,0 +1,1052 @@ +"""Focused Compose backend for the leasing model. + +Renders a docker-compose project straight from the live set of +:class:`Deployment` s — not the legacy resolved-deployment schema — using +the placement planner for GPU assignment and reusing ``profile_runtime.vllm_args`` +for the vLLM CLI flags. It *converges the whole union* on every reconcile: +render the file, then ``docker compose up -d --remove-orphans``. Adding or +removing a deployment re-renders and converges; pinned placement (persisted in a +sidecar) keeps already-running models on their GPUs, and ``--remove-orphans`` +tears down services whose deployment is gone. + +A **LiteLLM front door** (default on) gives one stable ``base_url`` and routes +each endpoint *alias* to its upstream vLLM/Ollama service, so a client always +talks to ``http://host:/v1`` and asks for the public endpoint name. +That is what makes the endpoint descriptor's ``base_url`` correct (the backend +supplies it via :meth:`ComposeBackend.access`). + +Docker and HTTP are invoked through injected seams (``run`` / ``http_get``), so +all logic here is unit-testable without docker or a network. The real +docker/GPU path is validated on a GPU host. ``converge`` is serialized with a +file lock so concurrent processes don't clobber the shared compose file. + +Slice status: readiness probes the LiteLLM ``/v1/models`` listing (model is +routable). The Ollama tag pull/warmup rung is a follow-up. +""" + +from __future__ import annotations + +import contextlib +import fcntl +import hashlib +import json +import re +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable + +import yaml + +from ..config import DEFAULT_PORTS, PINNED_IMAGES, default_state_paths +from ..env_utils import ensure_secret, parse_env_file, write_env_file +from ..probe import openai_ready +from ..profile_runtime import vllm_args +from .backend import Readiness +from .models import Deployment +from .placement import plan_placement + +LEASING_PROJECT = 'infer-stack' # docker compose project name for leased stacks +VLLM_HOST_PORT_BASE = 18000 +VLLM_CONTAINER_PORT = 8000 +OLLAMA_CONTAINER_PORT = 11434 +LITELLM_CONTAINER_PORT = 4000 +STATE_FILENAME = 'leasing-compose-state.json' +COMPOSE_FILENAME = 'docker-compose.yml' +LITELLM_CONFIG_FILENAME = 'litellm_config.yaml' +LOCK_FILENAME = '.converge.lock' +LITELLM_SERVICE = 'litellm' +API_KEY_ENV = 'LITELLM_MASTER_KEY' + +VLLM_DEFAULTS = { + 'gpu_memory_utilization': 0.9, + 'max_model_len': 8192, + 'max_num_batched_tokens': 8192, + 'max_num_seqs': 256, +} + +DEPLOYMENT_LABEL = 'infer-stack.deployment' +ENGINE_LABEL = 'infer-stack.engine' + + +def _dns_slug(text: str) -> str: + """A lowercase ``[a-z0-9-]`` label safe as a compose service/DNS name.""" + out = re.sub(r'[^a-z0-9]+', '-', str(text).lower()).strip('-') + return out or 'model' + + +def vllm_service_name(deployment: Deployment) -> str: + """Compose service name for a vLLM deployment: ``vllm--``. + + A vLLM deployment is exactly one model in one container (possibly across several + GPUs), so we lead the service name with the served model name — the alias + the user chose, also vLLM's ``--served-model-name`` and the Open WebUI label + — to make ``docker ps`` / ``nvidia-smi`` legible *without* infer-stack. The + full deployment id is kept as a suffix so the name is unique (two desired deployments + can share a served name, e.g. an endpoint re-pointed at a new model) and so + it correlates 1:1 with the ``id`` column of ``infer-stack leases``. + + The service name is also the on-network DNS host LiteLLM routes to, so + :func:`_litellm_model_list` must derive the upstream from this same helper. + """ + served = deployment.spec.get('served_model_name') or ( + sorted(deployment.served)[0] if deployment.served else deployment.id + ) + return f'vllm-{_dns_slug(served)}-{deployment.id}' + + +@dataclass +class RenderedCompose: + compose: dict[str, Any] + services: dict[str, str] = field(default_factory=dict) # service -> deployment id + litellm_config: str | None = None + nginx_config: str | None = None + + +def _gpu_reservation(indices: list[int]) -> dict[str, Any]: + return { + 'resources': { + 'reservations': { + 'devices': [ + { + 'driver': 'nvidia', + 'device_ids': [str(i) for i in indices], + 'capabilities': ['gpu'], + } + ] + } + } + } + + +def _vllm_service_dict(deployment: Deployment) -> dict[str, Any]: + """Build the dict ``vllm_args`` consumes from a deployment's runtime spec.""" + runtime = deployment.spec.get('runtime', {}) or {} + served = deployment.spec.get('served_model_name') or ( + sorted(deployment.served)[0] if deployment.served else deployment.id + ) + return { + 'served_model_name': served, + 'tensor_parallel_size': int(runtime.get('tensor_parallel_size', 1) or 1), + 'data_parallel_size': int(runtime.get('data_parallel_size', 1) or 1), + 'max_model_len': runtime.get('max_model_len', VLLM_DEFAULTS['max_model_len']), + 'gpu_memory_utilization': runtime.get( + 'gpu_memory_utilization', VLLM_DEFAULTS['gpu_memory_utilization'] + ), + 'max_num_batched_tokens': runtime.get( + 'max_num_batched_tokens', VLLM_DEFAULTS['max_num_batched_tokens'] + ), + 'max_num_seqs': runtime.get( + 'max_num_seqs', VLLM_DEFAULTS['max_num_seqs'] + ), + 'enable_prefix_caching': bool(runtime.get('enable_prefix_caching', False)), + 'extra_args': list(runtime.get('extra_args', []) or []), + } + + +def _vllm_service( + deployment: Deployment, + gpus: list[int], + host_port: int, + images: dict[str, str], + state: dict[str, str], +) -> dict[str, Any]: + svc = _vllm_service_dict(deployment) + command = [ + deployment.spec['hf_model_id'], + '--host', + '0.0.0.0', + '--port', + '8000', + *vllm_args(svc), + ] + service: dict[str, Any] = { + 'image': images['vllm'], + 'command': command, + 'ports': [f'{host_port}:8000'], + 'environment': {'HF_TOKEN': '${HF_TOKEN:-}'}, + 'volumes': [f'{state["hf_cache"]}:/root/.cache/huggingface'], + 'restart': 'unless-stopped', + 'labels': {DEPLOYMENT_LABEL: deployment.id, ENGINE_LABEL: 'vllm'}, + 'healthcheck': { + 'test': ['CMD', 'curl', '-f', 'http://localhost:8000/health'], + 'interval': '30s', + 'timeout': '10s', + 'retries': 5, + 'start_period': '1800s', + }, + } + if gpus: + service['deploy'] = _gpu_reservation(gpus) + return service + + +def _ollama_service( + deployment: Deployment, + gpus: list[int], + host_port: int, + images: dict[str, str], + state: dict[str, str], +) -> dict[str, Any]: + settings = deployment.spec.get('settings', {}) or {} + env: dict[str, str] = {} + if settings.get('keep_alive'): + env['OLLAMA_KEEP_ALIVE'] = str(settings['keep_alive']) + if settings.get('num_parallel') is not None: + env['OLLAMA_NUM_PARALLEL'] = str(settings['num_parallel']) + if settings.get('max_loaded_models') is not None: + env['OLLAMA_MAX_LOADED_MODELS'] = str(settings['max_loaded_models']) + if settings.get('context_length') is not None: + env['OLLAMA_CONTEXT_LENGTH'] = str(settings['context_length']) + # GPU pinning is done by the device reservation below (``device_ids``), which + # exposes *only* those physical GPUs to the container — and the NVIDIA + # runtime renumbers them to 0..n-1 inside it. So we must NOT also set + # ``CUDA_VISIBLE_DEVICES`` to the host indices: pinning to host GPU 1 would + # leave the container seeing one GPU as device 0 while CUDA_VISIBLE_DEVICES=1 + # points at nothing, and ollama silently falls back to CPU. vLLM relies on + # the reservation alone; ollama does the same. + service: dict[str, Any] = { + 'image': deployment.spec.get('image') or images['ollama'], + 'ports': [f'{host_port}:11434'], + 'environment': env, + 'volumes': [f'{state["ollama"]}:/root/.ollama'], + 'restart': 'unless-stopped', + 'labels': {DEPLOYMENT_LABEL: deployment.id, ENGINE_LABEL: 'ollama'}, + 'healthcheck': { + 'test': ['CMD', 'ollama', 'list'], + 'interval': '30s', + 'timeout': '10s', + 'retries': 5, + }, + } + if gpus: + service['deploy'] = _gpu_reservation(gpus) + return service + + +def _litellm_model_list( + deployments: list[Deployment], assignments: dict[str, list[int]] +) -> list[dict[str, Any]]: + """One LiteLLM ``model_list`` entry per served endpoint alias.""" + entries: list[dict[str, Any]] = [] + for deployment in sorted(deployments, key=lambda g: (g.created_at, g.id)): + if deployment.id not in assignments: + continue + if deployment.engine == 'vllm': + served = deployment.spec.get('served_model_name') or deployment.id + api_base = f'http://{vllm_service_name(deployment)}:8000/v1' + for endpoint in sorted(deployment.served): + entries.append( + { + 'model_name': endpoint, + 'litellm_params': { + 'model': f'openai/{served}', + 'api_base': api_base, + 'api_key': 'EMPTY', + }, + } + ) + elif deployment.engine == 'ollama': + api_base = f'http://ollama-{deployment.id}:11434' + for endpoint, payload in sorted(deployment.served.items()): + tag = payload.get('model', endpoint) + entries.append( + { + 'model_name': endpoint, + 'litellm_params': { + 'model': f'ollama/{tag}', + 'api_base': api_base, + }, + } + ) + return entries + + +CONFIG_HASH_LABEL = 'infer-stack.config-hash' + + +def _litellm_service( + service_names: list[str], + host_port: int, + images: dict[str, str], + aux_dir: str, + master_key: str | None = None, + config_hash: str | None = None, +) -> dict[str, Any]: + # Bake the managed key in literally (not ${...}) so the container and the + # readiness probe always agree regardless of the caller's shell env. + key_value = ( + master_key + if master_key is not None + else '${' + API_KEY_ENV + ':-sk-local}' + ) + labels = {ENGINE_LABEL: 'litellm'} + if config_hash is not None: + # LiteLLM reads its routing config once at startup; the file is bind- + # mounted, so a config change alone does NOT change this service's spec + # and `docker compose up -d` would leave the old container (and old + # routes) running. Stamping the config hash onto a label makes the spec + # change exactly when the config does, so converge recreates LiteLLM and + # it picks up new/removed aliases. Without this, coalescing a second + # alias onto a live deployment never becomes routable (readiness times out). + labels[CONFIG_HASH_LABEL] = config_hash + service: dict[str, Any] = { + 'image': images['litellm'], + 'command': [ + '--config', + '/etc/litellm/config.yaml', + '--port', + str(LITELLM_CONTAINER_PORT), + ], + 'ports': [f'{host_port}:{LITELLM_CONTAINER_PORT}'], + 'volumes': [f'{aux_dir}/{LITELLM_CONFIG_FILENAME}:/etc/litellm/config.yaml:ro'], + 'environment': {API_KEY_ENV: key_value}, + 'restart': 'unless-stopped', + 'labels': labels, + } + # Only wait on upstreams when there are any (zero models -> empty gateway). + if service_names: + service['depends_on'] = sorted(service_names) + return service + + +OPEN_WEBUI_SERVICE = 'open-webui' +OPEN_WEBUI_CONTAINER_PORT = 8080 + +NGINX_SERVICE = 'reverse-proxy' +NGINX_CONTAINER_PORT = 80 +NGINX_CONFIG_FILENAME = 'nginx.conf' + + +def _open_webui_service( + host_port: int, + images: dict[str, str], + state: dict[str, str], + master_key: str | None, + *, + openai_urls: list[str] | None = None, + ollama_urls: list[str] | None = None, + depends_on: list[str] | None = None, +) -> dict[str, Any]: + """A managed Open WebUI pointed at whatever front door is available. + + Open WebUI holds two independent kinds of connection, wired here from the + rendered services: + + * **OpenAI** (``openai_urls``) — the chat/completions front door. This is the + LiteLLM gateway when it is enabled (so every declared endpoint alias is + reachable at one URL); with LiteLLM off it falls back to the rendered + upstreams' own ``/v1`` (a single vLLM/Ollama service, or several joined as + ``OPENAI_API_BASE_URLS``). With nothing to point at, the OpenAI API is + disabled rather than left dangling. + * **Ollama** (``ollama_urls``) — the *native* Ollama API of any rendered + Ollama daemon. This is what lets you pull/run/delete models from the UI + and have the daemon load them on demand, independent of LiteLLM — i.e. a + true drop-in for a hand-run ``ollama`` + Open WebUI stack. + + The spec is kept as independent of which models are live as it can be: the + LiteLLM URL is fixed, and the Ollama daemon's service name is its stable + structural id, so adding/removing other models does not rewrite this service + and ``docker compose up -d`` leaves the UI running (the legacy "the UI never + blinks" behavior). Chat history persists under the data dir. + """ + key_value = ( + master_key if master_key is not None else '${' + API_KEY_ENV + ':-sk-local}' + ) + data_path = state.get('open_webui') or str( + Path(next(iter(state.values()), '.')).parent / 'open-webui' + ) + openai_urls = list(openai_urls or []) + ollama_urls = list(ollama_urls or []) + env: dict[str, str] = { + # Single-user workstation default; the port shouldn't be exposed + # publicly. Tracked as a knob in dev/leasing-followups.md. + 'WEBUI_AUTH': 'False', + } + if openai_urls: + env['ENABLE_OPENAI_API'] = 'True' + if len(openai_urls) == 1: + env['OPENAI_API_BASE_URL'] = openai_urls[0] + else: + env['OPENAI_API_BASE_URLS'] = ';'.join(openai_urls) + env['OPENAI_API_KEY'] = key_value + else: + env['ENABLE_OPENAI_API'] = 'False' + if ollama_urls: + env['ENABLE_OLLAMA_API'] = 'True' + if len(ollama_urls) == 1: + env['OLLAMA_BASE_URL'] = ollama_urls[0] + else: + env['OLLAMA_BASE_URLS'] = ';'.join(ollama_urls) + else: + env['ENABLE_OLLAMA_API'] = 'False' + service: dict[str, Any] = { + 'image': images['open_webui'], + 'ports': [f'{host_port}:{OPEN_WEBUI_CONTAINER_PORT}'], + 'environment': env, + 'volumes': [f'{data_path}:/app/backend/data'], + 'restart': 'unless-stopped', + 'labels': {ENGINE_LABEL: 'open-webui'}, + } + if depends_on: + service['depends_on'] = sorted(depends_on) + return service + + +def _nginx_conf(*, litellm: bool, ui: bool) -> str: + """A minimal HTTP reverse-proxy conf: one origin, path-routed. + + ``/v1/`` -> the LiteLLM gateway (the OpenAI API), ``/`` -> Open WebUI (or the + gateway when there's no UI). Plain HTTP — no TLS, no auth — so the value is + "one port, nothing to remember", not security. The ``map`` is valid here + because a ``conf.d/*.conf`` file is included in nginx's ``http`` context. + """ + api = f'http://{LITELLM_SERVICE}:{LITELLM_CONTAINER_PORT}' + locations = '' + if litellm: + locations += ( + ' location /v1/ {\n' + f' proxy_pass {api}/v1/;\n' + ' proxy_set_header Host $host;\n' + ' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n' + ' proxy_set_header X-Forwarded-Proto $scheme;\n' + ' proxy_read_timeout 600s;\n' + ' }\n' + ) + # `/` serves the UI when present, else the gateway (so hitting the host root + # still lands somewhere useful). Upgrade headers keep Open WebUI's websockets + # working; client_max_body_size 0 allows large uploads. + if ui: + root = f'http://{OPEN_WEBUI_SERVICE}:{OPEN_WEBUI_CONTAINER_PORT}' + elif litellm: + root = api + else: + root = '' + if root: + locations += ( + ' location / {\n' + f' proxy_pass {root};\n' + ' proxy_http_version 1.1;\n' + ' proxy_set_header Upgrade $http_upgrade;\n' + ' proxy_set_header Connection $connection_upgrade;\n' + ' proxy_set_header Host $host;\n' + ' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n' + ' proxy_set_header X-Forwarded-Proto $scheme;\n' + ' }\n' + ) + return ( + 'map $http_upgrade $connection_upgrade {\n' + ' default upgrade;\n' + " '' close;\n" + '}\n\n' + 'server {\n' + f' listen {NGINX_CONTAINER_PORT};\n' + ' server_name _;\n' + ' client_max_body_size 0;\n' + f'{locations}' + '}\n' + ) + + +def _nginx_service( + host_port: int, + images: dict[str, str], + *, + aux_dir: str, + depends_on: list[str], + config_path: str | None = None, + config_hash: str | None = None, +) -> dict[str, Any]: + # BYO config (config_path) is mounted verbatim; otherwise the generated + # nginx.conf in the state dir is used. + mount = config_path or f'{aux_dir}/{NGINX_CONFIG_FILENAME}' + labels = {ENGINE_LABEL: 'nginx'} + if config_hash is not None: + # Same trick as LiteLLM: the conf is bind-mounted, so stamp its hash on a + # label to force a recreate when the routing changes. + labels[CONFIG_HASH_LABEL] = config_hash + service: dict[str, Any] = { + 'image': images['nginx'], + 'ports': [f'{host_port}:{NGINX_CONTAINER_PORT}'], + 'volumes': [f'{mount}:/etc/nginx/conf.d/default.conf:ro'], + 'restart': 'unless-stopped', + 'labels': labels, + } + if depends_on: + service['depends_on'] = sorted(depends_on) + return service + + +def render_compose( + deployments: list[Deployment], + assignments: dict[str, list[int]], + *, + images: dict[str, str], + ports: dict[str, int], + state: dict[str, str], + litellm: bool = False, + litellm_port: int = 14042, + litellm_master_key: str | None = None, + ui: bool = False, + ui_port: int = 13000, + reverse_proxy: bool = False, + reverse_proxy_port: int = 80, + reverse_proxy_config: str | None = None, + aux_dir: str | Path | None = None, + project: str = LEASING_PROJECT, +) -> RenderedCompose: + """Render a compose project for the placed deployments. + + Deployments absent from ``assignments`` (placement failures) are skipped. When + ``litellm`` is set, a front-door service + config is added so every endpoint + alias is reachable at one ``base_url``. When ``ui`` is also set, a managed + Open WebUI is rendered in front of that gateway. + + The project name is baked into the file as a top-level ``name:`` so a plain + ``docker compose -f docker-compose.yml up`` (infer-stack not involved) lands + in the *same* project — same container names, same network — as + ``infer-stack``'s own ``-p`` invocations. That makes "drop the tool and run + docker yourself" a true equivalent of ``apply`` rather than a sibling + project the tool can no longer see. + """ + services: dict[str, Any] = {} + service_map: dict[str, str] = {} + # In-network upstreams Open WebUI can connect to directly when there is no + # LiteLLM gateway (or, for Ollama, *in addition* to it — see below). + vllm_v1_urls: list[str] = [] # OpenAI /v1 of each vLLM process + ollama_native_urls: list[str] = [] # native Ollama API of each daemon + ordered = sorted(deployments, key=lambda g: (g.created_at, g.id)) + vllm_i = 0 + ollama_i = 0 + for deployment in ordered: + if deployment.id not in assignments: + continue + gpus = assignments[deployment.id] + if deployment.engine == 'vllm': + name = vllm_service_name(deployment) + port = VLLM_HOST_PORT_BASE + vllm_i + vllm_i += 1 + services[name] = _vllm_service(deployment, gpus, port, images, state) + vllm_v1_urls.append(f'http://{name}:{VLLM_CONTAINER_PORT}/v1') + elif deployment.engine == 'ollama': + name = f'ollama-{deployment.id}' + port = ports.get('ollama', DEFAULT_PORTS['ollama']) + ollama_i + ollama_i += 1 + services[name] = _ollama_service(deployment, gpus, port, images, state) + ollama_native_urls.append(f'http://{name}:{OLLAMA_CONTAINER_PORT}') + else: + continue + service_map[name] = deployment.id + + litellm_config = None + # The front door (gateway + UI) is rendered whenever it's enabled, even with + # zero models — it's a standing entry point, not a per-model service. So + # releasing/evicting every model leaves an empty gateway (and an empty Open + # WebUI picker) up instead of tearing the whole stack down; only an explicit + # `stack down` removes it. With no models the model_list is simply empty. + if litellm: + entries = _litellm_model_list(deployments, assignments) + litellm_config = yaml.safe_dump( + { + 'model_list': entries, + 'general_settings': { + 'master_key': f'os.environ/{API_KEY_ENV}' + }, + # An upstream vLLM/Ollama is unreachable only briefly, while it + # loads its model (LiteLLM does not wait for upstream health to + # start). Retry transient connection errors and don't park a + # model in a long cooldown, so the warmup window is self-healing + # instead of surfacing as client 500s ("Connection error. + # Received Model Deployment=…"). + 'router_settings': { + 'num_retries': 3, + 'timeout': 600, + 'cooldown_time': 5, + 'allowed_fails': 100, + }, + }, + sort_keys=False, + ) + config_hash = hashlib.sha256( + litellm_config.encode('utf-8') + ).hexdigest()[:12] + services[LITELLM_SERVICE] = _litellm_service( + list(service_map), + litellm_port, + images, + str(aux_dir or '.'), + master_key=litellm_master_key, + config_hash=config_hash, + ) + + # Open WebUI is its own standing front door, rendered whenever ``ui`` is set + # — it does NOT require LiteLLM. Its OpenAI connection prefers the gateway + # (one URL covers every alias) and falls back to the rendered vLLM upstreams' + # own /v1 when there is no gateway. Its native Ollama connection always + # points straight at any Ollama daemon, so you can pull/run models from the + # UI and have the daemon load them on demand — a true drop-in for a + # hand-run ollama + Open WebUI stack. depends_on lists only LiteLLM (the one + # service guaranteed present alongside the UI); the per-model upstreams come + # and go, so the UI tolerates them being absent rather than hard-depending. + # With a gateway the UI is a standing front door (renders even at zero + # models). Without one it is only meaningful pointed at a live upstream, so + # render it only when there is something to connect to — otherwise an empty + # desired set has nothing to run and converge tears the project down. + if ui and (litellm or vllm_v1_urls or ollama_native_urls): + if litellm: + openai_urls = [f'http://{LITELLM_SERVICE}:{LITELLM_CONTAINER_PORT}/v1'] + ui_depends = [LITELLM_SERVICE] + else: + openai_urls = list(vllm_v1_urls) + ui_depends = [] + services[OPEN_WEBUI_SERVICE] = _open_webui_service( + ui_port, + images, + state, + litellm_master_key, + openai_urls=openai_urls, + ollama_urls=ollama_native_urls, + depends_on=ui_depends, + ) + + # Optional single-port HTTP reverse proxy fronting the gateway (+ UI). Needs + # the gateway, so it's only rendered alongside litellm. + nginx_config = None + if reverse_proxy and litellm: + depends = [LITELLM_SERVICE] + ([OPEN_WEBUI_SERVICE] if ui else []) + if reverse_proxy_config: + services[NGINX_SERVICE] = _nginx_service( + reverse_proxy_port, images, aux_dir=str(aux_dir or '.'), + depends_on=depends, config_path=reverse_proxy_config, + ) + else: + nginx_config = _nginx_conf(litellm=litellm, ui=ui) + services[NGINX_SERVICE] = _nginx_service( + reverse_proxy_port, images, aux_dir=str(aux_dir or '.'), + depends_on=depends, + config_hash=hashlib.sha256( + nginx_config.encode('utf-8') + ).hexdigest()[:12], + ) + + return RenderedCompose( + compose={'name': project, 'services': services}, + services=service_map, + litellm_config=litellm_config, + nginx_config=nginx_config, + ) + + +def _default_docker_run(args: list[str]) -> str: + import subprocess + + return subprocess.check_output(args, text=True) + + +def _parse_ps(out: str) -> set[str]: + """Parse running service names from ``docker compose ps --format json``. + + Handles both a JSON array and newline-delimited JSON objects. + """ + out = (out or '').strip() + if not out: + return set() + rows: list[dict[str, Any]] = [] + try: + data = json.loads(out) + rows = data if isinstance(data, list) else [data] + except json.JSONDecodeError: + for line in out.splitlines(): + line = line.strip() + if not line: + continue + try: + rows.append(json.loads(line)) + except json.JSONDecodeError: + pass + running: set[str] = set() + for row in rows: + if str(row.get('State', '')).lower().startswith('running'): + name = row.get('Service') or row.get('Name') + if name: + running.add(str(name)) + return running + + +class ComposeBackend: + """Single-host docker compose backend (converge-style). + + Driven by the controller's ``converge`` path. ``run`` / ``http_get`` are the + injected docker/HTTP seams; defaults shell out to ``docker compose`` and + ``requests``. + """ + + def __init__( + self, + *, + state_dir: str | Path, + inventory: dict[str, Any], + run: Callable[[list[str]], str] | None = None, + http: Any = None, + images: dict[str, str] | None = None, + ports: dict[str, int] | None = None, + state: dict[str, str] | None = None, + allowed_gpus: list[int] | None = None, + reserved: list[int] | tuple[int, ...] = (), + project: str = LEASING_PROJECT, + skip_display: bool = False, + litellm: bool = True, + ui: bool = True, + reverse_proxy: bool = False, + reverse_proxy_port: int = 80, + reverse_proxy_config: str | None = None, + require_generation: bool = False, + assume_yes: bool = True, + ): + self.state_dir = Path(state_dir) + self.state_dir.mkdir(parents=True, exist_ok=True) + self.inventory = inventory + self.run = run or _default_docker_run + if http is None: + import requests + http = requests + self.http = http + self.images = {**PINNED_IMAGES, **(images or {})} + self.ports = {**DEFAULT_PORTS, **(ports or {})} + self.state = state or default_state_paths() + self.allowed_gpus = allowed_gpus + self.reserved = tuple(reserved) + self.project = project + self.skip_display = skip_display + self.litellm = litellm + self.ui = ui + self.reverse_proxy = reverse_proxy + self.reverse_proxy_port = reverse_proxy_port + self.reverse_proxy_config = reverse_proxy_config + self.require_generation = require_generation + self.assume_yes = assume_yes + self.last_errors: list[str] = [] + self.last_unplaced: set[str] = set() # desired deployment ids placement skipped + self.last_assignments: dict[str, list[int]] = {} # deployment id -> GPU ids + self._pulled: set[str] = set() # (deployment:tag) pulled this process + + @property + def compose_file(self) -> Path: + return self.state_dir / COMPOSE_FILENAME + + @property + def _state_file(self) -> Path: + return self.state_dir / STATE_FILENAME + + @property + def litellm_port(self) -> int: + return self.ports.get('litellm', DEFAULT_PORTS['litellm']) + + @property + def ui_port(self) -> int: + return self.ports.get('open_webui', DEFAULT_PORTS['open_webui']) + + @property + def _env_path(self) -> Path: + return self.state_dir / '.env' + + def master_key(self) -> str: + """The managed LiteLLM master key. + + infer-stack manages this secret in the state dir's ``.env``: reused if + already present (you may pin your own ``sk-`` key there), otherwise + generated and persisted. The caller doesn't need to invent or export it + — it is baked into the LiteLLM service, used by the readiness probe, and + shipped in the env-file descriptor (``infer-stack env KEY`` prints it). + """ + existing = parse_env_file(self._env_path) + key = ensure_secret(existing, API_KEY_ENV, prefix='sk-') + if key != existing.get(API_KEY_ENV): + write_env_file(self._env_path, {API_KEY_ENV: key}) + return key + + def _load_sidecar(self) -> dict[str, Any]: + if self._state_file.exists(): + return json.loads(self._state_file.read_text()) + return {} + + def _save_sidecar(self, data: dict[str, Any]) -> None: + self._state_file.write_text(json.dumps(data, indent=2)) + + def _compose(self, args: list[str]) -> str: + return self.run( + [ + 'docker', + 'compose', + '-p', + self.project, + '-f', + str(self.compose_file), + *args, + ] + ) + + @contextlib.contextmanager + def _converge_lock(self): + """Serialize converge across processes sharing this state dir.""" + handle = open(self.state_dir / LOCK_FILENAME, 'w') + try: + fcntl.flock(handle, fcntl.LOCK_EX) + yield + finally: + fcntl.flock(handle, fcntl.LOCK_UN) + handle.close() + + def _approve_changes(self, planned: dict[Path, str]) -> None: + """Show pending compose/litellm changes and confirm them. + + ``planned`` maps target paths to their new content. When nothing + actually changed, this is a quiet no-op. When ``assume_yes`` (scripts / + non-interactive / ``--yes``), it applies after a one-line log. Otherwise + it renders a per-file diff and prompts; a decline raises + :class:`ConvergeAborted` so the caller can roll back. + """ + from .._log import logger + from .backend import ConvergeAborted + + changed = { + p: text + for p, text in planned.items() + if (p.read_text() if p.exists() else '') != text + } + if not changed: + logger.debug('compose project already up to date') + return + names = ', '.join(p.name for p in changed) + if self.assume_yes: + logger.info('Updating compose project ({})', names) + return + from ..diff_prompt import confirm_writes + + if not confirm_writes( + changed, + assume_yes=False, + title='infer-stack will update the compose project', + ): + raise ConvergeAborted('compose changes were not approved') + + def plan(self, desired: list[Deployment]): + """Compute GPU placement for ``desired`` without writing or applying. + + Read-only and side-effect free: it honors the persisted pins, so the + result reflects where deployments are (for running ones) or *would* be (for + not-yet-started ones) placed. ``leases`` uses it to show actual/slated + GPUs; ``converge`` uses it as the first step of render. + """ + pinned = self._load_sidecar().get('assignments', {}) + return plan_placement( + list(desired), + self.inventory, + allowed_gpus=self.allowed_gpus, + reserved=self.reserved, + pinned=pinned, + skip_display=self.skip_display, + ) + + def converge(self, desired: list[Deployment], *, apply: bool = True): + """Place + render the desired union, then optionally apply it. + + The work splits into *render* (decide placement, write the + ``docker-compose.yml`` / LiteLLM config / placement sidecar to the state + dir) and *apply* (``docker compose up -d`` / ``down``). With + ``apply=False`` only the render half runs, so the on-disk project shows + exactly what *would* execute — inspect it, or bring it up yourself + (``infer-stack apply``). Either way the placement plan is returned and + ``last_errors`` / ``last_unplaced`` / ``last_assignments`` are updated. + """ + from .._log import logger + + desired = list(desired) + with self._converge_lock(): + logger.info( + 'Converging {} deployment(s): {}', + len(desired), + ', '.join(sorted(g.id for g in desired)) or '(none)', + ) + plan = self.plan(desired) + self.last_errors = plan.errors + self.last_unplaced = { + g.id for g in desired if g.id not in plan.assignments + } + self.last_assignments = dict(plan.assignments) + for gid, gpus in sorted(plan.assignments.items()): + logger.info(' placed {} on GPU(s) {}', gid, gpus or '(cpu)') + for err in plan.errors: + logger.warning(' placement: {}', err) + rendered = render_compose( + desired, + plan.assignments, + images=self.images, + ports=self.ports, + state=self.state, + litellm=self.litellm, + litellm_port=self.litellm_port, + litellm_master_key=self.master_key() if self.litellm else None, + ui=self.ui, + ui_port=self.ui_port, + reverse_proxy=self.reverse_proxy, + reverse_proxy_port=self.reverse_proxy_port, + reverse_proxy_config=self.reverse_proxy_config, + aux_dir=self.state_dir, + project=self.project, + ) + + compose_text = yaml.safe_dump(rendered.compose, sort_keys=False) + planned: dict[Path, str] = {self.compose_file: compose_text} + if rendered.litellm_config is not None: + planned[self.state_dir / LITELLM_CONFIG_FILENAME] = ( + rendered.litellm_config + ) + if rendered.nginx_config is not None: + planned[self.state_dir / NGINX_CONFIG_FILENAME] = ( + rendered.nginx_config + ) + self._approve_changes(planned) # may raise ConvergeAborted + + if rendered.litellm_config is not None: + (self.state_dir / LITELLM_CONFIG_FILENAME).write_text( + rendered.litellm_config + ) + if rendered.nginx_config is not None: + (self.state_dir / NGINX_CONFIG_FILENAME).write_text( + rendered.nginx_config + ) + self.compose_file.write_text(compose_text) + self._save_sidecar( + {'assignments': plan.assignments, 'services': rendered.services} + ) + services = rendered.compose.get('services') + if not apply: + logger.info( + 'rendered {} service(s) to {} (not applied; ' + '`infer-stack apply` to bring it up)', + len(services or {}), self.compose_file, + ) + return plan + if services: + logger.info( + 'docker compose up -d ({} service(s): {})', + len(services), ', '.join(sorted(services)), + ) + self._compose(['up', '-d', '--remove-orphans']) + else: + # Nothing at all to run — only reachable with the gateway off + # (litellm=False) and zero models, since the front door otherwise + # keeps the project non-empty. `docker compose up` errors with + # "no service selected" on a services-less file, so tear the + # project down instead (`down` works on the empty file). With the + # gateway on, releasing every model lands in the `up` branch above + # and leaves the front door standing; `stack down` is the way to + # take everything off. + logger.info('no services desired -> docker compose down') + self._compose(['down', '--remove-orphans']) + return plan + + def observe(self) -> set[str]: + if not self.compose_file.exists(): + return set() + try: + out = self._compose(['ps', '--format', 'json']) + except Exception: # noqa: BLE001 - observe is best-effort + # A stale/invalid compose file on disk (e.g. left by an older + # version) or a transient docker error must not brick acquire: + # `docker compose ps` validates the file, so a bad file would raise + # here *before* converge gets to overwrite it. Treat as "nothing + # observed" and let converge rewrite + reconcile. + return set() + running = _parse_ps(out) + services = self._load_sidecar().get('services', {}) + return {services[name] for name in running if name in services} + + def access(self, endpoints: list[str]) -> dict[str, Any] | None: + """Where a client reaches these endpoints, for the env-file descriptor. + + With the LiteLLM front door, that is one ``base_url`` and the request + model name is the endpoint alias itself. With LiteLLM off there is no + single base URL, but a managed Open WebUI (if on) is still a useful + access point, so report just its URL rather than ``None``. + """ + if not self.litellm: + if self.ui: + return {'ui_url': f'http://127.0.0.1:{self.ui_port}'} + return None + info: dict[str, Any] = { + 'base_url': f'http://127.0.0.1:{self.litellm_port}/v1', + 'api_key_env': API_KEY_ENV, + 'api_key': self.master_key(), + 'request_names': {ep: ep for ep in endpoints}, + } + if self.ui: + info['ui_url'] = f'http://127.0.0.1:{self.ui_port}' + if self.reverse_proxy: + # The unified front door: one origin, UI at / and the API at /v1. + info['proxy_url'] = f'http://127.0.0.1:{self.reverse_proxy_port}' + return info + + def _ensure_ollama_tag( + self, deployment: Deployment, endpoint: str + ) -> str | None: + """Pull the endpoint's Ollama tag into its daemon (idempotent). + + An Ollama daemon loads tags lazily, so a tag must be present before it + can serve. Returns an error reason if the pull failed (retry next poll), + else ``None``. + """ + tag = (deployment.served.get(endpoint) or {}).get('model') + if not tag: + return None + key = f'{deployment.id}:{tag}' + if key in self._pulled: + return None + try: + self._compose(['exec', '-T', f'ollama-{deployment.id}', 'ollama', 'pull', tag]) + except Exception as ex: # noqa: BLE001 - readiness is retryable + return f'pulling {tag}: {ex}' + self._pulled.add(key) + return None + + def probe_ready( + self, deployment: Deployment, endpoint: str + ) -> Readiness: + """Ready == container running and (with LiteLLM) the alias is routable. + + Delegates the HTTP check to the shared :func:`infer_stack.probe.openai_ready` + — ``require_listed`` confirms the alias is advertised by the gateway, and + ``require_generation`` additionally runs a tiny chat. For Ollama the tag + is pulled first and a generation is forced, so readiness means the tag is + actually loaded and serving (not just lazily configured). + """ + if deployment.id not in self.observe(): + return Readiness(False, 'container not running') + require_generation = self.require_generation + if deployment.engine == 'ollama': + # Pull the declared tag regardless of the gateway — a lean + # `--no-litellm` Ollama stack must still make its anchor model + # present, and the pull is what readiness means for that daemon. + error = self._ensure_ollama_tag(deployment, endpoint) + if error: + return Readiness(False, error) + require_generation = True # warm the tag so it is resident + if not self.litellm: + # No gateway to probe for routability; the container is up (and any + # Ollama tag has been pulled), so it is as ready as we can confirm. + return Readiness(True, 'container running') + headers = {'Authorization': f'Bearer {self.master_key()}'} + ok, reason = openai_ready( + base_url=f'http://127.0.0.1:{self.litellm_port}/v1', + headers=headers, + model=endpoint, + require_listed=True, + require_generation=require_generation, + http=self.http, + ) + return Readiness(ok, reason) + + def down(self) -> None: + """Tear the whole project down (for an explicit stop).""" + if self.compose_file.exists(): + self._compose(['down', '--remove-orphans']) diff --git a/infer_stack/leasing/controller.py b/infer_stack/leasing/controller.py new file mode 100644 index 00000000..fd93667c --- /dev/null +++ b/infer_stack/leasing/controller.py @@ -0,0 +1,288 @@ +"""The controller: reconcile the ledger's desired state onto a backend. + +This is the thin orchestration layer the design doc (§11) calls for. The ledger +is pure bookkeeping; a backend realizes deployments; the controller is what +turns ``acquire`` / ``release`` into "the right things are running and ready". + +The reconcile loop is the standard desired-vs-actual converge: + + desired = LIVE deployments + IDLE deployments whose reclaim policy is keep-warm + actual = backend.observe() + realize(desired - actual); teardown(actual - desired) + +``reclaim`` policy lives on each deployment's spec (from the catalog): ``keep-warm`` +(default — survive idle until pressure, avoids cold-start thrash) keeps an idle +deployment running; ``stop`` / ``scale-to-zero`` let it be torn down as soon as demand +hits zero. TTL expiry is enforced here too: every ``reconcile`` first +``sweep``s the ledger, so a crashed job's lease stops protecting its deployment once +its TTL elapses. +""" + +from __future__ import annotations + +import time +from dataclasses import dataclass, field +from typing import Callable, Iterable + +from .backend import Backend +from .ledger import Ledger +from .models import Deployment, DeploymentState, EndpointRequest, Lease + +KEEP_WARM = 'keep-warm' + + +@dataclass +class ReconcileResult: + realized: list[str] = field(default_factory=list) + torn_down: list[str] = field(default_factory=list) + # Desired deployments the backend could not place (e.g. no free GPU) plus the + # planner's per-deployment reasons; empty for backends without placement. + unplaced: list[str] = field(default_factory=list) + placement_errors: list[str] = field(default_factory=list) + # deployment id -> GPU indices it is on / slated for (placement backends only). + assignments: dict[str, list[int]] = field(default_factory=dict) + # False when reconcile only rendered the on-disk state (no docker up/down). + applied: bool = True + + +@dataclass +class WaitResult: + ready: bool + pending: list[tuple[str, str]] = field(default_factory=list) + + +@dataclass +class AcquireOutcome: + lease: Lease + deployments: list[Deployment] + reconcile: ReconcileResult + wait: WaitResult | None = None + applied: bool = True # False for a --no-apply (staged, not brought up) acquire + + +@dataclass +class ReleaseOutcome: + idled_deployment_ids: list[str] + reconcile: ReconcileResult + + +@dataclass +class EvictOutcome: + evicted_deployment_ids: list[str] + reconcile: ReconcileResult + + +class Controller: + """Ties a :class:`Ledger` to a :class:`Backend`. + + Args: + ledger: the lease bookkeeping store. + backend: realizes/observes/probes deployments. + clock: epoch-seconds source for wait timing; defaults to the ledger's. + sleep: how to wait between readiness polls; injectable for tests. + reclaim_default: policy for deployments whose spec omits one. + """ + + def __init__( + self, + ledger: Ledger, + backend: Backend, + *, + clock: Callable[[], float] | None = None, + sleep: Callable[[float], None] = time.sleep, + reclaim_default: str = KEEP_WARM, + ): + self.ledger = ledger + self.backend = backend + self.clock = clock or ledger.clock + self.sleep = sleep + self.reclaim_default = reclaim_default + + # -- reconcile --------------------------------------------------------- + + def desired_deployments(self) -> list[Deployment]: + """Deployments that should currently be running.""" + _, deployments = self.ledger.status() + desired: list[Deployment] = [] + for deployment in deployments: + if deployment.state == DeploymentState.LIVE: + desired.append(deployment) + elif deployment.state == DeploymentState.IDLE: + policy = deployment.spec.get('reclaim', self.reclaim_default) + if policy == KEEP_WARM: + desired.append(deployment) + return desired + + def reconcile(self, *, apply: bool = True) -> ReconcileResult: + """Converge the backend to the ledger's desired state. + + A backend may implement a single ``converge(desired)`` (whole-union + convergence — e.g. Compose renders one file and ``up``s it); otherwise + the per-deployment ``realize``/``teardown`` loop is used. + + ``apply=False`` asks a converge-style backend to render the on-disk + project *without* bringing it up — the "see what would execute" / staged + path. Backends without that capability ignore it (render and apply are + inseparable for them). + """ + self.ledger.sweep() + desired = self.desired_deployments() + if hasattr(self.backend, 'converge'): + before = set(self.backend.observe()) + # Only pass apply= when staging, so backends/fakes with the older + # ``converge(desired)`` signature keep working on the default path. + if apply: + self.backend.converge(desired) + else: + self.backend.converge(desired, apply=False) + after = set(self.backend.observe()) + return ReconcileResult( + realized=sorted(after - before), + torn_down=sorted(before - after), + unplaced=sorted(getattr(self.backend, 'last_unplaced', ()) or ()), + placement_errors=list( + getattr(self.backend, 'last_errors', ()) or () + ), + assignments=dict( + getattr(self.backend, 'last_assignments', {}) or {} + ), + applied=apply, + ) + desired_ids = {g.id for g in desired} + actual = self.backend.observe() + result = ReconcileResult() + for deployment in desired: + if deployment.id not in actual: + self.backend.realize(deployment) + result.realized.append(deployment.id) + stale = actual - desired_ids + if stale: + by_id = {g.id: g for g in self.ledger.status()[1]} + for gid in stale: + deployment = by_id.get(gid) + if deployment is not None: + self.backend.teardown(deployment) + result.torn_down.append(gid) + return result + + # -- readiness --------------------------------------------------------- + + def wait_ready( + self, + deployments: Iterable[Deployment], + *, + endpoints: set[str] | None = None, + timeout: float = 300.0, + interval: float = 2.0, + ) -> WaitResult: + """Block until the requested served endpoints are ready or ``timeout``. + + ``endpoints`` filters which served names to wait on (a coalesced deployment + may serve more than this caller asked for); ``None`` waits for all. + """ + pairs = [ + (deployment, ep) + for deployment in deployments + for ep in sorted(deployment.served) + if endpoints is None or ep in endpoints + ] + deadline = self.clock() + timeout + while True: + pending = [ + (g, ep) + for (g, ep) in pairs + if not self.backend.probe_ready(g, ep).ready + ] + if not pending: + return WaitResult(ready=True) + if self.clock() >= deadline: + return WaitResult( + ready=False, + pending=[(g.id, ep) for g, ep in pending], + ) + self.sleep(interval) + pairs = pending + + # -- thin acquire / release ------------------------------------------- + + def acquire( + self, + owner: str, + requests: list[EndpointRequest], + *, + ttl_seconds: float | None = None, + wait: bool = True, + timeout: float = 300.0, + interval: float = 2.0, + apply: bool = True, + ) -> AcquireOutcome: + """Create a lease, realize its deployments, and (optionally) block on ready. + + ``apply=False`` stages the lease and renders the on-disk project without + bringing it up (and skips the readiness wait, since nothing is running). + Placement is still computed, so an unplaceable request still fails fast. + """ + from .backend import ConvergeAborted, PlacementError + + result = self.ledger.acquire( + owner, requests, ttl_seconds=ttl_seconds + ) + try: + rec = self.reconcile(apply=apply) + except ConvergeAborted: + # The operator declined the compose changes — don't leave the + # just-created lease dangling in the ledger. + self.ledger.release(result.lease.id) + raise + # If a deployment this lease just requested could not be placed (e.g. no free + # GPU), don't block forever waiting on a container that will never come + # up — roll the lease back and report the planner's reason, so the deployment + # never lingers as a phantom ``live`` with nothing behind it. + requested = {g.id for g in result.deployments} + unplaced = requested & set(rec.unplaced) + if unplaced: + self.ledger.release(result.lease.id) + reasons = [ + e + for e in rec.placement_errors + if any(e.startswith(gid) for gid in unplaced) + ] + raise PlacementError(sorted(unplaced), reasons) + deployments = [self.ledger.get_deployment(g.id) for g in result.deployments] + deployments = [g for g in deployments if g is not None] + wait_result = None + if wait and apply: # nothing to wait on when we only staged the render + wait_result = self.wait_ready( + deployments, + endpoints=set(result.lease.endpoints), + timeout=timeout, + interval=interval, + ) + return AcquireOutcome( + lease=result.lease, + deployments=deployments, + reconcile=rec, + wait=wait_result, + applied=apply, + ) + + def release(self, lease_id: str) -> ReleaseOutcome: + """Release a lease and converge (tearing down per reclaim policy).""" + rel = self.ledger.release(lease_id) + rec = self.reconcile() + return ReleaseOutcome( + idled_deployment_ids=rel.idled_deployment_ids, reconcile=rec + ) + + def evict(self, deployment_ids: Iterable[str] | None = None) -> EvictOutcome: + """Force-evict idle (released) deployments now, overriding keep-warm. + + Marks the matching IDLE deployments STOPPED and reconciles, so a keep-warm + model that is merely resident gets torn down and its GPU freed. + ``deployment_ids=None`` evicts every idle deployment. + """ + ids = None if deployment_ids is None else list(deployment_ids) + self.ledger.sweep() + evicted = self.ledger.evict_idle(ids) + rec = self.reconcile() + return EvictOutcome(evicted_deployment_ids=evicted, reconcile=rec) diff --git a/infer_stack/leasing/envfile.py b/infer_stack/leasing/envfile.py new file mode 100644 index 00000000..3c262ae6 --- /dev/null +++ b/infer_stack/leasing/envfile.py @@ -0,0 +1,137 @@ +"""The endpoint descriptor + env-file a lease hands to a job. + +This is the standard "here is the endpoint and the models" artifact the +inference-endpoint-injection brainstorm asked for, aligned with the +``contracts.py`` serving-profile-contract (same ``base_url`` / ``api_key_env`` / +``models`` shape). ``acquire`` / ``run`` emit it so a pipeline node sources one +file and gets a ready OpenAI-compatible client, instead of every team inventing +its own env-var contract. + +For now ``base_url`` is supplied by the caller (the dry-run/null backend has no +real front door). Once the Compose/KubeAI backends land they will report the +realized base URL, and this same descriptor will carry it. +""" + +from __future__ import annotations + +import re +from pathlib import Path +from typing import Any + +from .models import Deployment, Lease + +LEASE_ENV = 'INFER_STACK_LEASE_ID' + + +def _request_model_name(payload: dict[str, Any], endpoint: str) -> str: + """The name a client must pass as ``model`` to reach this endpoint.""" + return ( + payload.get('served_model_name') + or payload.get('model') + or endpoint + ) + + +def _endpoint_var(endpoint: str) -> str: + slug = re.sub(r'[^A-Z0-9]+', '_', endpoint.upper()).strip('_') + return f'INFER_STACK_ENDPOINT_{slug}' + + +def build_descriptor( + lease: Lease, + deployments: list[Deployment], + *, + base_url: str, + api_key_env: str = 'LITELLM_MASTER_KEY', + api_key: str | None = None, + request_names: dict[str, str] | None = None, + cuda_visible_devices: str | None = None, +) -> dict[str, Any]: + """Build the endpoint descriptor for one lease. + + Only the endpoints this lease actually requested are included (a coalesced + deployment may serve more). ``request_names`` overrides the model name a client + must request per endpoint — e.g. behind a LiteLLM front door the client asks + for the endpoint *alias*, not the upstream served name. + """ + endpoints: dict[str, str] = {} + for deployment in deployments: + for endpoint, payload in deployment.served.items(): + if endpoint in lease.endpoints: + if request_names and endpoint in request_names: + endpoints[endpoint] = request_names[endpoint] + else: + endpoints[endpoint] = _request_model_name(payload, endpoint) + # preserve the lease's requested order where possible + ordered = {ep: endpoints[ep] for ep in lease.endpoints if ep in endpoints} + descriptor: dict[str, Any] = { + 'schema_version': 1, + 'kind': 'infer-stack-endpoint', + 'lease_id': lease.id, + 'base_url': base_url, + 'api_key_env': api_key_env, + 'protocol': 'openai', + 'endpoints': ordered, + 'models': list(ordered.values()), + } + if api_key: + descriptor['api_key'] = api_key + if cuda_visible_devices is not None: + descriptor['cuda_visible_devices'] = cuda_visible_devices + return descriptor + + +def descriptor_env(descriptor: dict[str, Any]) -> dict[str, str]: + """Flatten a descriptor into the env vars a job should see.""" + env: dict[str, str] = {LEASE_ENV: descriptor['lease_id']} + if descriptor.get('base_url'): + env['OPENAI_BASE_URL'] = descriptor['base_url'] + if descriptor.get('api_key'): + # The actual key, so `source ` configures an OpenAI client + # outright (OPENAI_BASE_URL + OPENAI_API_KEY) — no manual export. + env['OPENAI_API_KEY'] = descriptor['api_key'] + if descriptor.get('api_key_env'): + env['INFER_STACK_API_KEY_ENV'] = descriptor['api_key_env'] + for endpoint, model in descriptor.get('endpoints', {}).items(): + env[_endpoint_var(endpoint)] = model + if descriptor.get('models'): + env['INFER_STACK_MODELS'] = ','.join(descriptor['models']) + if descriptor.get('cuda_visible_devices') is not None: + env['CUDA_VISIBLE_DEVICES'] = str(descriptor['cuda_visible_devices']) + return env + + +def render_env_file(descriptor: dict[str, Any]) -> str: + """Render the descriptor as a sourceable shell env-file. + + Example: + >>> from infer_stack.leasing.models import Lease, Deployment, DeploymentState + >>> lease = Lease('lease-1', 'me', 'active', 0.0, None, None, 0.0, + ... endpoints=['qwen-coder']) + >>> grp = Deployment('g1', 'ck', 'vllm', 'shared-compatible', {}, + ... {}, {'qwen-coder': {'served_model_name': 'qwen-coder'}}, + ... DeploymentState.LIVE, 0.0, 0.0) + >>> d = build_descriptor(lease, [grp], base_url='http://h:1/v1') + >>> print(render_env_file(d), end='') + export INFER_STACK_LEASE_ID=lease-1 + export OPENAI_BASE_URL=http://h:1/v1 + export INFER_STACK_API_KEY_ENV=LITELLM_MASTER_KEY + export INFER_STACK_ENDPOINT_QWEN_CODER=qwen-coder + export INFER_STACK_MODELS=qwen-coder + """ + return ''.join( + f'export {key}={value}\n' + for key, value in descriptor_env(descriptor).items() + ) + + +def read_lease_id(env_file: str | Path) -> str | None: + """Recover the lease id from a previously written env-file.""" + prefix = f'export {LEASE_ENV}=' + for line in Path(env_file).expanduser().read_text().splitlines(): + line = line.strip() + if line.startswith(prefix): + return line[len(prefix):].strip().strip('"').strip("'") + if line.startswith(f'{LEASE_ENV}='): + return line.split('=', 1)[1].strip().strip('"').strip("'") + return None diff --git a/infer_stack/leasing/ledger.py b/infer_stack/leasing/ledger.py new file mode 100644 index 00000000..f044e51f --- /dev/null +++ b/infer_stack/leasing/ledger.py @@ -0,0 +1,313 @@ +"""High-level lease bookkeeping over the sqlite store. + +This is the backend-agnostic heart of the controller. It owns the *invariants*: + +* **Coalescing.** Two requests for the same compatible deployment share one + :class:`~infer_stack.leasing.models.Deployment` (demand reference-counted) + unless one asks for ``dedicated``. Compatibility is structural identity + (:func:`~infer_stack.leasing.models.compatibility_key`) plus capacity + subsumption (:func:`~infer_stack.leasing.models.capacity_satisfies`). +* **Soft TTL.** A lease protects its deployments while ACTIVE and not past its TTL. + Expiry does not kill anything; it merely stops protecting, so the deployment can be + reclaimed once nothing else needs it. TTL is the crash-recovery backstop for + leases whose explicit ``release`` never ran. +* **Demand-driven deployment state.** A deployment is LIVE while demand > 0 and IDLE when + it reaches 0. Actually tearing an IDLE deployment down is a reconciler/backend + concern (per reclaim policy); the ledger only computes the candidates. + +It does **not** talk to docker, GPUs, or KubeAI — that is the reconciler/backend +layer that consumes :class:`AcquireResult`. +""" + +from __future__ import annotations + +import time +import uuid +from dataclasses import dataclass, field +from pathlib import Path +from typing import Callable + +from ..paths import data_root +from .models import ( + Deployment, + DeploymentState, + EndpointRequest, + Lease, + LeaseState, + Sharing, + capacity_satisfies, +) +from .store import SqliteStore + + +def default_ledger_path() -> Path: + """Default shared ledger location under the infer-stack data root.""" + return data_root() / 'leasing' / 'ledger.db' + + +@dataclass +class AcquireResult: + """What :meth:`Ledger.acquire` returns to the reconciler. + + ``deployments`` are the deployments the caller must ensure are realized and ready + before handing the lease's env-file to the job. + """ + + lease: Lease + deployments: list[Deployment] = field(default_factory=list) + + +@dataclass +class ReleaseResult: + idled_deployment_ids: list[str] = field(default_factory=list) + + +@dataclass +class SweepResult: + expired_lease_ids: list[str] = field(default_factory=list) + idled_deployment_ids: list[str] = field(default_factory=list) + + +class Ledger: + """Lease/deployment bookkeeping. + + Args: + store: the sqlite store. + clock: returns epoch seconds; injectable for tests. + id_factory: ``(prefix) -> id``; injectable for deterministic tests. + + Example: + >>> from infer_stack.leasing.store import SqliteStore + >>> from infer_stack.leasing.models import EndpointRequest, vllm_structural + >>> led = Ledger(SqliteStore(':memory:')) + >>> req = EndpointRequest( + ... endpoint='qwen-coder', engine='vllm', + ... structural=vllm_structural(model_ref='qwen-coder-32b'), + ... capacity={'max_model_len': 32768}) + >>> a = led.acquire('alice', [req]) + >>> b = led.acquire('bob', [req]) # same model -> coalesces + >>> a.deployments[0].id == b.deployments[0].id + True + >>> led.get_deployment(a.deployments[0].id).demand # two protecting leases + 2 + >>> _ = led.release(a.lease.id) + >>> led.get_deployment(a.deployments[0].id).demand + 1 + >>> res = led.release(b.lease.id) # last one out -> idle + >>> res.idled_deployment_ids == [a.deployments[0].id] + True + """ + + def __init__( + self, + store: SqliteStore, + *, + clock: Callable[[], float] = time.time, + id_factory: Callable[[str], str] | None = None, + ): + self.store = store + self.clock = clock + self.id_factory = id_factory or self._default_id + + @staticmethod + def _default_id(prefix: str) -> str: + return f'{prefix}-{uuid.uuid4().hex[:12]}' + + # -- public API -------------------------------------------------------- + + def acquire( + self, + owner: str, + requests: list[EndpointRequest], + *, + ttl_seconds: float | None = None, + ) -> AcquireResult: + """Create a lease and coalesce its endpoints onto deployment deployments. + + ``ttl_seconds=None`` is an infinite (standing) lease — the shape the + legacy ``switch `` maps onto. + """ + now = self.clock() + lease_id = self.id_factory('lease') + expires_at = None if ttl_seconds is None else now + ttl_seconds + deployment_ids: list[str] = [] + with self.store.transaction(): + self.store.insert_lease( + lease_id=lease_id, + owner=owner, + created_at=now, + ttl_seconds=ttl_seconds, + expires_at=expires_at, + heartbeat_at=now, + ) + for req in requests: + deployment = self._find_or_create_deployment(req, now) + self.store.insert_claim( + lease_id=lease_id, endpoint=req.endpoint, deployment_id=deployment.id + ) + deployment_ids.append(deployment.id) + lease = self.store.get_lease(lease_id) + deployments = [ + self.store.get_deployment(gid, now=now) + for gid in dict.fromkeys(deployment_ids) + ] + return AcquireResult(lease=lease, deployments=[g for g in deployments if g]) + + def release(self, lease_id: str) -> ReleaseResult: + """Mark a lease released and idle any deployment whose demand hits zero.""" + now = self.clock() + idled: list[str] = [] + with self.store.transaction(): + lease = self.store.get_lease(lease_id) + if lease is None or lease.state == LeaseState.RELEASED: + return ReleaseResult() + self.store.set_lease_state(lease_id, LeaseState.RELEASED) + idled = self._idle_deployments(lease.deployment_ids, now) + return ReleaseResult(idled_deployment_ids=idled) + + def renew(self, lease_id: str, *, ttl_seconds: float | None) -> Lease | None: + """Extend (or make infinite) a lease's protection window.""" + now = self.clock() + expires_at = None if ttl_seconds is None else now + ttl_seconds + with self.store.transaction(): + if self.store.get_lease(lease_id) is None: + return None + self.store.renew_lease( + lease_id, + ttl_seconds=ttl_seconds, + expires_at=expires_at, + heartbeat_at=now, + ) + return self.store.get_lease(lease_id) + + def sweep(self) -> SweepResult: + """Materialize TTL expiry and recompute idle deployments. + + Safe to call periodically. Expiring a lease is the crash backstop: a job + that died without ``release`` stops protecting once its TTL elapses. + """ + now = self.clock() + expired: list[str] = [] + idled: list[str] = [] + with self.store.transaction(): + due = self.store.active_leases_past(now) + affected: list[str] = [] + for lease in due: + self.store.set_lease_state(lease.id, LeaseState.EXPIRED) + expired.append(lease.id) + affected.extend(lease.deployment_ids) + idled = self._idle_deployments(affected, now) + return SweepResult(expired_lease_ids=expired, idled_deployment_ids=idled) + + def reclaimable_deployments(self) -> list[Deployment]: + """IDLE deployments the reconciler may tear down per reclaim policy.""" + now = self.clock() + return [ + g + for g in self.store.list_deployments(now=now) + if g.state == DeploymentState.IDLE + ] + + def evict_idle(self, deployment_ids: list[str] | None = None) -> list[str]: + """Force IDLE deployments to STOPPED so the next reconcile tears them down. + + This overrides ``keep-warm``: a released deployment normally stays resident + (IDLE) to avoid cold-start thrash, but evicting it frees its GPU now. + ``deployment_ids=None`` evicts every idle deployment; a list restricts it (ids not + currently idle are skipped). Returns the ids actually evicted. + """ + now = self.clock() + wanted = None if deployment_ids is None else set(deployment_ids) + evicted: list[str] = [] + with self.store.transaction(): + for g in self.store.list_deployments(now=now): + if g.state != DeploymentState.IDLE: + continue + if wanted is not None and g.id not in wanted: + continue + self.store.set_deployment_state(g.id, DeploymentState.STOPPED, now) + evicted.append(g.id) + return evicted + + def prune(self) -> tuple[int, int]: + """Forget terminal entries: released/expired leases + stopped deployments. + + Sweep/evict leave a tail of RELEASED/EXPIRED leases and STOPPED deployments + in the ledger so history stays inspectable; this clears that tail once + you no longer care. Returns ``(n_leases, n_deployments)`` removed. + """ + return self.store.prune( + lease_states=(LeaseState.RELEASED, LeaseState.EXPIRED), + deployment_states=(DeploymentState.STOPPED,), + ) + + def status(self) -> tuple[list[Lease], list[Deployment]]: + """Snapshot for ``infer-stack status`` (leases, deployments-with-demand).""" + now = self.clock() + return self.store.list_leases(), self.store.list_deployments(now=now) + + def get_lease(self, lease_id: str) -> Lease | None: + return self.store.get_lease(lease_id) + + def get_deployment(self, deployment_id: str) -> Deployment | None: + return self.store.get_deployment(deployment_id, now=self.clock()) + + # -- internals --------------------------------------------------------- + + def _find_or_create_deployment( + self, req: EndpointRequest, now: float + ) -> Deployment: + if req.sharing == Sharing.DEDICATED: + return self._create_deployment(req, now) + candidates = self.store.deployments_by_compat( + req.compat_key, + sharing=Sharing.SHARED, + states=(DeploymentState.LIVE, DeploymentState.IDLE), + ) + for deployment in candidates: + if capacity_satisfies(deployment.capacity, req.capacity): + if deployment.state == DeploymentState.IDLE: + self.store.set_deployment_state( + deployment.id, DeploymentState.LIVE, now + ) + self._merge_served(deployment, req, now) + return self.store.get_deployment(deployment.id) or deployment + return self._create_deployment(req, now) + + def _create_deployment( + self, req: EndpointRequest, now: float + ) -> Deployment: + deployment = Deployment( + id=self.id_factory('grp'), + compat_key=req.compat_key, + engine=req.engine, + sharing=req.sharing, + capacity=dict(req.capacity), + spec=dict(req.spec), + served={req.endpoint: dict(req.served)}, + state=DeploymentState.LIVE, + created_at=now, + updated_at=now, + ) + self.store.insert_deployment(deployment) + return deployment + + def _merge_served( + self, deployment: Deployment, req: EndpointRequest, now: float + ) -> None: + """Add an endpoint (e.g. a new Ollama tag) to a coalesced deployment.""" + served = dict(deployment.served) + if served.get(req.endpoint) != req.served: + served[req.endpoint] = dict(req.served) + self.store.update_deployment_served(deployment.id, served, now) + + def _idle_deployments(self, deployment_ids: list[str], now: float) -> list[str]: + idled: list[str] = [] + for gid in dict.fromkeys(deployment_ids): + deployment = self.store.get_deployment(gid) + if deployment is None or deployment.state != DeploymentState.LIVE: + continue + if self.store.demand(gid, now) == 0: + self.store.set_deployment_state(gid, DeploymentState.IDLE, now) + idled.append(gid) + return idled diff --git a/infer_stack/leasing/models.py b/infer_stack/leasing/models.py new file mode 100644 index 00000000..3962fa1c --- /dev/null +++ b/infer_stack/leasing/models.py @@ -0,0 +1,300 @@ +"""Data model for the leasing ledger. + +This is the *controller* half of infer-stack (see the redesign doc +``dev/infer-stack-redesign-critique.md`` in the aiq-eval-runner repo). The +compiler half (``resolver``/``validator``/``renderer``) turns a desired +deployment into backend artifacts; the ledger tracks *who wants what now* so +that multiple users / pipeline nodes can ``acquire`` and ``release`` models +without clobbering each other. + +Three record kinds, in three layers: + +* :class:`EndpointRequest` — catalog-resolved input to the ledger. The + catalog/resolver layer builds these; the ledger consumes them and never needs + to know engine-specific details beyond what is packed in here. +* :class:`Lease` — who wants a set of endpoints, with a soft TTL. +* :class:`Deployment` — compatible requests coalesced into one realizable + backend deployment. ``demand`` is the number of *protecting* leases pointing + at it. + +The unit that gets reference-counted is **the thing that gets a process**: +one model for vLLM (one process per model), one daemon for Ollama (one daemon +serves many tags). Both are expressed uniformly through the +:func:`compatibility_key`: for vLLM the structural fields describe the model + +runtime; for Ollama they describe the *daemon* config, so many tag endpoints +coalesce onto one deployment. +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass, field +from typing import Any + + +class LeaseState: + """Lifecycle states for a :class:`Lease`.""" + + ACTIVE = 'active' + EXPIRED = 'expired' + RELEASED = 'released' + + +class DeploymentState: + """Lifecycle states for a :class:`Deployment`. + + ``LIVE``/``IDLE`` are driven by the ledger from demand. ``RECLAIMING``/ + ``STOPPED`` are owned by the (future) reconciler/backend and are reserved + here for forward compatibility. + """ + + LIVE = 'live' + IDLE = 'idle' + RECLAIMING = 'reclaiming' + STOPPED = 'stopped' + + +class Sharing: + """Sharing policy a requester can ask for.""" + + SHARED = 'shared-compatible' + DEDICATED = 'dedicated' + + +# Fields that must match *exactly* for two requests to share one deployment. +# Capacity fields (e.g. ``max_model_len``) are deliberately NOT here: they are +# handled by subsumption (existing >= requested) in +# :func:`capacity_satisfies`, so a 32k deployment can serve an 8k request. +VLLM_STRUCTURAL_FIELDS = ( + 'engine', + 'model_ref', + 'revision', + 'quantization', + 'dtype', + 'tensor_parallel_size', + 'pipeline_parallel_size', + 'image', + 'chat_template', + 'trust_remote_code', + 'lora_adapters', + 'served_name', +) + +# For Ollama the coalescing unit is the *daemon*, so the structural identity is +# the host config, not the model tag (tags load/unload inside the daemon). +OLLAMA_STRUCTURAL_FIELDS = ( + 'engine', + 'host', + 'gpu_indices', + 'keep_alive', + 'num_parallel', + 'max_loaded_models', + 'model_store', +) + + +def _canonical(value: Any) -> Any: + """Normalize a value so equal-meaning configs hash identically.""" + if isinstance(value, dict): + return {k: _canonical(value[k]) for k in sorted(value)} + if isinstance(value, (list, tuple)): + return [_canonical(v) for v in value] + return value + + +def compatibility_key(engine: str, structural: dict[str, Any]) -> str: + """Stable hash identifying deployments that may be shared. + + Two :class:`EndpointRequest` s with the same ``compatibility_key`` describe + the *same realizable deployment* and are eligible to coalesce (subject to + capacity subsumption and sharing policy, which are handled separately so the + key stays a pure structural identity). + + Args: + engine: the serving engine (``'vllm'`` / ``'ollama'``). + structural: the engine-appropriate structural fields (see + :func:`vllm_structural` / :func:`ollama_structural`). + + Returns: + A 16-hex-char digest. + + Example: + >>> a = vllm_structural(model_ref='qwen', tensor_parallel_size=1) + >>> b = vllm_structural(model_ref='qwen', tensor_parallel_size=1) + >>> c = vllm_structural(model_ref='qwen', tensor_parallel_size=2) + >>> compatibility_key('vllm', a) == compatibility_key('vllm', b) + True + >>> compatibility_key('vllm', a) == compatibility_key('vllm', c) + False + """ + payload = {'engine': engine, 'structural': _canonical(structural)} + blob = json.dumps(payload, sort_keys=True, separators=(',', ':')) + return hashlib.sha256(blob.encode('utf-8')).hexdigest()[:16] + + +def vllm_structural( + *, + model_ref: str, + revision: str | None = None, + quantization: str | None = None, + dtype: str | None = None, + tensor_parallel_size: int = 1, + pipeline_parallel_size: int = 1, + image: str | None = None, + chat_template: str | None = None, + trust_remote_code: bool = False, + lora_adapters: list[str] | None = None, + served_name: str | None = None, +) -> dict[str, Any]: + """Build the structural dict for a vLLM endpoint (one process per model).""" + return { + 'engine': 'vllm', + 'model_ref': model_ref, + 'revision': revision, + 'quantization': quantization, + 'dtype': dtype, + 'tensor_parallel_size': tensor_parallel_size, + 'pipeline_parallel_size': pipeline_parallel_size, + 'image': image, + 'chat_template': chat_template, + 'trust_remote_code': trust_remote_code, + 'lora_adapters': sorted(lora_adapters or []), + 'served_name': served_name or model_ref, + } + + +def ollama_structural( + *, + host: str, + gpu_indices: list[int] | None = None, + keep_alive: str | None = None, + num_parallel: int | None = None, + max_loaded_models: int | None = None, + model_store: str | None = None, +) -> dict[str, Any]: + """Build the structural dict for an Ollama daemon (one daemon, many tags). + + The model *tag* is intentionally absent — it is per-endpoint detail, not + part of the daemon's identity — so several tag endpoints sharing a host + config coalesce onto a single daemon deployment. + """ + return { + 'engine': 'ollama', + 'host': host, + 'gpu_indices': list(gpu_indices or []), + 'keep_alive': keep_alive, + 'num_parallel': num_parallel, + 'max_loaded_models': max_loaded_models, + 'model_store': model_store, + } + + +def capacity_satisfies( + existing: dict[str, Any], requested: dict[str, Any] +) -> bool: + """Whether an existing deployment's capacity covers a new request. + + Coalescing is *subsumption*, not equality: every capacity field the request + needs must be present on the existing deployment and at least as large + (e.g. a 32k ``max_model_len`` deployment serves an 8k request, but not the + reverse). An empty request (e.g. Ollama, which has no capacity fields) is + trivially satisfied. + + Example: + >>> capacity_satisfies({'max_model_len': 32768}, {'max_model_len': 8192}) + True + >>> capacity_satisfies({'max_model_len': 8192}, {'max_model_len': 32768}) + False + >>> capacity_satisfies({}, {'max_model_len': 8192}) + False + >>> capacity_satisfies({'max_model_len': 8192}, {}) + True + """ + for key, need in requested.items(): + have = existing.get(key) + if have is None or have < need: + return False + return True + + +@dataclass(frozen=True) +class EndpointRequest: + """A catalog-resolved request for one served endpoint. + + Produced by the catalog/resolver layer, consumed by the ledger. Carries + everything the ledger needs to coalesce without knowing engine specifics. + + Attributes: + endpoint: public served name the user asked for. + engine: ``'vllm'`` / ``'ollama'``. + structural: fields that must match exactly to coalesce (see + :func:`vllm_structural` / :func:`ollama_structural`). + capacity: subsumption fields where existing must be ``>=`` requested. + sharing: :class:`Sharing` policy. + spec: opaque host-realization payload the backend uses to start the + process/daemon (shared across coalesced requests). + served: opaque per-endpoint payload (e.g. the Ollama tag) merged into + the deployment's served map under ``endpoint``. + host: Ollama daemon name; ``None`` for auto-synthesized vLLM hosts. + """ + + endpoint: str + engine: str + structural: dict[str, Any] + capacity: dict[str, Any] = field(default_factory=dict) + sharing: str = Sharing.SHARED + spec: dict[str, Any] = field(default_factory=dict) + served: dict[str, Any] = field(default_factory=dict) + host: str | None = None + + @property + def compat_key(self) -> str: + return compatibility_key(self.engine, self.structural) + + +@dataclass +class Lease: + """Who wants a set of endpoints, with a soft TTL. + + A lease *protects* its deployment deployments while it is ``ACTIVE`` and not past + its TTL. Protection is what ``demand`` counts. + """ + + id: str + owner: str + state: str + created_at: float + ttl_seconds: float | None + expires_at: float | None + heartbeat_at: float + endpoints: list[str] = field(default_factory=list) + deployment_ids: list[str] = field(default_factory=list) + + def is_protecting(self, now: float) -> bool: + """True if this lease still protects its allocations at ``now``.""" + if self.state != LeaseState.ACTIVE: + return False + return self.expires_at is None or self.expires_at > now + + +@dataclass +class Deployment: + """Compatible requests coalesced into one realizable deployment. + + ``served`` maps endpoint name -> per-endpoint payload; for vLLM it has one + entry, for Ollama one per tag served by the daemon. ``demand`` is the count + of protecting leases and is computed on read, not stored. + """ + + id: str + compat_key: str + engine: str + sharing: str + capacity: dict[str, Any] + spec: dict[str, Any] + served: dict[str, Any] + state: str + created_at: float + updated_at: float + demand: int = 0 diff --git a/infer_stack/leasing/placement.py b/infer_stack/leasing/placement.py new file mode 100644 index 00000000..dfbacf5f --- /dev/null +++ b/infer_stack/leasing/placement.py @@ -0,0 +1,185 @@ +"""Single-host GPU placement over the union of live deployment deployments. + +The ledger deliberately does no GPU assignment — it only tracks demand. A +single-host backend (Compose) must place the *whole live set* of deployments onto the +GPU pool at once: this is the "minimal single-host placer because nothing else +will" from the redesign. Multi-node / bin-packing / preemption are explicitly +out of scope (that is KubeAI/k8s/Slurm territory). + +Placement rules, applied in a deterministic order so the result is stable across +reconciles: + +1. **pinned** deployments (already realized) keep their current GPUs when still valid, + so adding/removing a deployment does not reshuffle running models. +2. **explicit** deployments (an Ollama daemon's ``gpu_indices``, or a vLLM deployment with + an explicit placement) claim exactly those GPUs. +3. **first-fit** deployments (vLLM needing ``tensor_parallel_size × data_parallel_size`` + GPUs) fill the remaining pool in order. + +The pool is the inventory minus display GPUs (optional), minus anything not in +``allowed_gpus``, minus ``reserved`` (raw-GPU reservations, Phase 2). Real GPU +indices are preserved throughout. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +from ..hardware import available_gpu_indices as _available_gpu_indices +from ..hardware import first_fit as _first_fit +from .models import Deployment + +OLLAMA = 'ollama' + + +@dataclass +class GpuPlan: + """Result of placing a set of deployments: ``deployment_id -> gpu indices``.""" + + assignments: dict[str, list[int]] = field(default_factory=dict) + errors: list[str] = field(default_factory=list) + + @property + def ok(self) -> bool: + return not self.errors + + +def available_indices( + inventory: dict[str, Any], + *, + allowed_gpus: list[int] | None = None, + reserved: list[int] | tuple[int, ...] = (), + skip_display: bool = False, +) -> list[int]: + """The placeable GPU pool, in real-index order. + + Reuses the resolver's display-GPU handling, then applies the allow-list and + raw-GPU reservations. ``skip_display`` defaults to *off* — every GPU is + placeable, so a single-GPU host (whose only GPU drives the display) works; + set it to leave a monitor's GPU free on a multi-GPU workstation. + """ + indices = _available_gpu_indices( + inventory, 'auto' if skip_display else False + ) + reserved_set = set(reserved) + if allowed_gpus is not None: + allowed_set = set(allowed_gpus) + indices = [i for i in indices if i in allowed_set] + return [i for i in indices if i not in reserved_set] + + +def required_gpu_count(deployment: Deployment) -> int: + """GPUs a vLLM deployment needs = tensor_parallel × data_parallel.""" + runtime = deployment.spec.get('runtime', {}) or {} + tp = int(runtime.get('tensor_parallel_size', 1) or 1) + dp = int(runtime.get('data_parallel_size', 1) or 1) + return max(1, tp * dp) + + +def explicit_indices(deployment: Deployment) -> list[int] | None: + """Indices a deployment pins explicitly, or ``None`` if it wants first-fit. + + Ollama daemons always pin (possibly to ``[]`` for CPU); a vLLM deployment pins + only if its runtime carries ``gpu_indices``. + """ + spec = deployment.spec + if spec.get('engine') == OLLAMA: + return [int(i) for i in (spec.get('gpu_indices') or [])] + runtime = spec.get('runtime', {}) or {} + if runtime.get('gpu_indices'): + return [int(i) for i in runtime['gpu_indices']] + return None + + +def _sorted(deployments: list[Deployment]) -> list[Deployment]: + return sorted(deployments, key=lambda g: (g.created_at, g.id)) + + +def plan_placement( + deployments: list[Deployment], + inventory: dict[str, Any], + *, + allowed_gpus: list[int] | None = None, + reserved: list[int] | tuple[int, ...] = (), + pinned: dict[str, list[int]] | None = None, + skip_display: bool = False, +) -> GpuPlan: + """Assign GPUs to every deployment, or record per-deployment placement errors. + + Example: + >>> from infer_stack.hardware import simulate_inventory + >>> from infer_stack.leasing.models import Deployment, DeploymentState + >>> def vllm(gid, tp=1, t=0.0): + ... return Deployment(gid, 'ck', 'vllm', 'shared-compatible', + ... {}, {'engine': 'vllm', 'runtime': {'tensor_parallel_size': tp}}, + ... {}, DeploymentState.LIVE, t, t) + >>> plan = plan_placement([vllm('a', tp=2), vllm('b', t=1.0)], + ... simulate_inventory('4x80')) + >>> plan.assignments + {'a': [0, 1], 'b': [2]} + """ + pinned = pinned or {} + pool = available_indices( + inventory, + allowed_gpus=allowed_gpus, + reserved=reserved, + skip_display=skip_display, + ) + pool_set = set(pool) + used: set[int] = set() + plan = GpuPlan() + + ordered = _sorted(deployments) + + # 1) pinned deployments that are still fully placeable keep their GPUs. + deferred: list[Deployment] = [] + for deployment in ordered: + want = pinned.get(deployment.id) + if want is None: + deferred.append(deployment) + continue + want = [int(i) for i in want] + if all(i in pool_set for i in want) and not (used & set(want)): + plan.assignments[deployment.id] = want + used.update(want) + else: + deferred.append(deployment) # pin no longer valid; replace below + + # 2) explicit-placement deployments. + first_fit_deployments: list[Deployment] = [] + for deployment in deferred: + want = explicit_indices(deployment) + if want is None: + first_fit_deployments.append(deployment) + continue + if not want: # e.g. CPU-only Ollama daemon + plan.assignments[deployment.id] = [] + continue + invalid = [i for i in want if i not in pool_set] + clash = used & set(want) + if invalid or clash: + reason = [] + if invalid: + reason.append(f'gpus {invalid} not in pool {sorted(pool_set)}') + if clash: + reason.append(f'gpus {sorted(clash)} already in use') + plan.errors.append( + f'{deployment.id}: explicit placement failed ({"; ".join(reason)})' + ) + continue + plan.assignments[deployment.id] = want + used.update(want) + + # 3) first-fit deployments fill what remains. + for deployment in first_fit_deployments: + count = required_gpu_count(deployment) + remaining = [i for i in pool if i not in used] + indices, error = _first_fit(remaining, count) + if error: + plan.errors.append(f'{deployment.id}: {error}') + continue + plan.assignments[deployment.id] = indices + used.update(indices) + + return plan diff --git a/infer_stack/leasing/store.py b/infer_stack/leasing/store.py new file mode 100644 index 00000000..d697934d --- /dev/null +++ b/infer_stack/leasing/store.py @@ -0,0 +1,401 @@ +"""SQLite-backed persistence for the leasing ledger. + +A single sqlite database is the shared store that lets multiple processes / +users coordinate (the redesign replaces the old "everyone re-renders the same +compose file, last render wins" pattern). SQLite is chosen over a file-of-JSON +because it gives atomic reference-count updates and a real write lock for the +read-modify-write coalescing critical section, with zero extra dependencies. + +This module is intentionally *low level*: methods execute statements and map +rows to dataclasses, but the multi-step invariants (coalescing, demand-driven +state transitions) live in :mod:`infer_stack.leasing.ledger`, which wraps the +relevant calls in :meth:`SqliteStore.transaction`. + +Concurrency model: + +* ``isolation_level=None`` -> autocommit; transactions are explicit. +* WAL journal + ``busy_timeout`` so readers don't block writers and a contended + writer waits rather than failing immediately. +* The coalescing critical section uses ``BEGIN IMMEDIATE`` (via + :meth:`transaction`) to take the write lock *before* reading, so two + concurrent ``acquire`` calls cannot both decide "no deployment exists" and create + duplicates. +""" + +from __future__ import annotations + +import contextlib +import json +import sqlite3 +import threading +from pathlib import Path +from typing import Any, Iterator + +from .models import Deployment, Lease, LeaseState + +SCHEMA_VERSION = 1 + +_SCHEMA = """ +CREATE TABLE IF NOT EXISTS meta ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS leases ( + id TEXT PRIMARY KEY, + owner TEXT NOT NULL, + state TEXT NOT NULL, + created_at REAL NOT NULL, + ttl_seconds REAL, + expires_at REAL, + heartbeat_at REAL NOT NULL, + metadata TEXT NOT NULL DEFAULT '{}' +); + +CREATE TABLE IF NOT EXISTS deployments ( + id TEXT PRIMARY KEY, + compat_key TEXT NOT NULL, + engine TEXT NOT NULL, + sharing TEXT NOT NULL, + capacity TEXT NOT NULL DEFAULT '{}', + spec TEXT NOT NULL DEFAULT '{}', + served TEXT NOT NULL DEFAULT '{}', + state TEXT NOT NULL, + created_at REAL NOT NULL, + updated_at REAL NOT NULL +); + +CREATE TABLE IF NOT EXISTS claims ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + lease_id TEXT NOT NULL REFERENCES leases(id) ON DELETE CASCADE, + endpoint TEXT NOT NULL, + deployment_id TEXT NOT NULL REFERENCES deployments(id), + kind TEXT NOT NULL DEFAULT 'endpoint' +); + +CREATE INDEX IF NOT EXISTS idx_claims_lease ON claims(lease_id); +CREATE INDEX IF NOT EXISTS idx_claims_deployment ON claims(deployment_id); +CREATE INDEX IF NOT EXISTS idx_deployments_compat ON deployments(compat_key); +""" + + +def _loads(text: str) -> Any: + return json.loads(text) if text else None + + +def _dumps(value: Any) -> str: + return json.dumps(value, separators=(',', ':'), sort_keys=True) + + +class SqliteStore: + """Thin sqlite wrapper exposing ledger row operations.""" + + def __init__(self, path: str | Path = ':memory:', *, busy_timeout_ms: int = 5000): + self.path = str(path) + if self.path != ':memory:': + Path(self.path).expanduser().parent.mkdir(parents=True, exist_ok=True) + # check_same_thread=False lets a long-running process (e.g. the TUI) use + # this connection from a worker thread for converge-while-monitoring; + # ``_lock`` serializes write transactions so two threads can't both + # ``BEGIN IMMEDIATE`` on the one connection. sqlite itself is built + # serialized, so individual reads across threads are safe. + self._conn = sqlite3.connect( + self.path, isolation_level=None, timeout=busy_timeout_ms / 1000, + check_same_thread=False, + ) + self._lock = threading.RLock() + self._conn.row_factory = sqlite3.Row + self._conn.execute('PRAGMA foreign_keys = ON') + self._conn.execute(f'PRAGMA busy_timeout = {busy_timeout_ms}') + if self.path != ':memory:': + self._conn.execute('PRAGMA journal_mode = WAL') + self._ensure_schema() + + def _ensure_schema(self) -> None: + self._conn.executescript(_SCHEMA) + row = self._conn.execute( + "SELECT value FROM meta WHERE key = 'schema_version'" + ).fetchone() + if row is None: + self._conn.execute( + "INSERT INTO meta(key, value) VALUES ('schema_version', ?)", + (str(SCHEMA_VERSION),), + ) + + def close(self) -> None: + self._conn.close() + + # -- transactions ------------------------------------------------------ + + @contextlib.contextmanager + def transaction(self) -> Iterator[sqlite3.Connection]: + """Take the write lock up front and commit/rollback atomically. + + ``BEGIN IMMEDIATE`` is what makes the ledger's find-or-create-deployment + step race-safe across processes. ``_lock`` adds the same guarantee + across threads in one process (overlapping ``BEGIN IMMEDIATE`` on a + shared connection would otherwise raise). + """ + with self._lock: + self._conn.execute('BEGIN IMMEDIATE') + try: + yield self._conn + self._conn.execute('COMMIT') + except Exception: + self._conn.execute('ROLLBACK') + raise + + # -- leases ------------------------------------------------------------ + + def insert_lease( + self, + *, + lease_id: str, + owner: str, + created_at: float, + ttl_seconds: float | None, + expires_at: float | None, + heartbeat_at: float, + metadata: dict[str, Any] | None = None, + ) -> None: + self._conn.execute( + 'INSERT INTO leases(id, owner, state, created_at, ttl_seconds,' + ' expires_at, heartbeat_at, metadata)' + ' VALUES(?, ?, ?, ?, ?, ?, ?, ?)', + ( + lease_id, + owner, + LeaseState.ACTIVE, + created_at, + ttl_seconds, + expires_at, + heartbeat_at, + _dumps(metadata or {}), + ), + ) + + def set_lease_state(self, lease_id: str, state: str) -> None: + self._conn.execute( + 'UPDATE leases SET state = ? WHERE id = ?', (state, lease_id) + ) + + def renew_lease( + self, + lease_id: str, + *, + ttl_seconds: float | None, + expires_at: float | None, + heartbeat_at: float, + ) -> None: + self._conn.execute( + 'UPDATE leases SET state = ?, ttl_seconds = ?, expires_at = ?,' + ' heartbeat_at = ? WHERE id = ?', + ( + LeaseState.ACTIVE, + ttl_seconds, + expires_at, + heartbeat_at, + lease_id, + ), + ) + + def get_lease(self, lease_id: str) -> Lease | None: + row = self._conn.execute( + 'SELECT * FROM leases WHERE id = ?', (lease_id,) + ).fetchone() + return self._row_to_lease(row) if row else None + + def list_leases(self, *, states: tuple[str, ...] | None = None) -> list[Lease]: + if states: + placeholders = ','.join('?' for _ in states) + rows = self._conn.execute( + f'SELECT * FROM leases WHERE state IN ({placeholders})' + ' ORDER BY created_at', + states, + ).fetchall() + else: + rows = self._conn.execute( + 'SELECT * FROM leases ORDER BY created_at' + ).fetchall() + return [self._row_to_lease(r) for r in rows] + + def prune( + self, + *, + lease_states: tuple[str, ...] = (), + deployment_states: tuple[str, ...] = (), + ) -> tuple[int, int]: + """Delete terminal leases/deployments (and their claims) from the ledger. + + Claims are removed first so the ``deployments.id`` foreign key can't block a + deployment deletion; deleting leases also cascades their claims. Returns + ``(n_leases_deleted, n_deployments_deleted)``. + """ + n_leases = n_deployments = 0 + with self.transaction() as conn: + if lease_states: + lq = ','.join('?' for _ in lease_states) + conn.execute( + f'DELETE FROM claims WHERE lease_id IN ' + f'(SELECT id FROM leases WHERE state IN ({lq}))', + lease_states, + ) + if deployment_states: + gq = ','.join('?' for _ in deployment_states) + conn.execute( + f'DELETE FROM claims WHERE deployment_id IN ' + f'(SELECT id FROM deployments WHERE state IN ({gq}))', + deployment_states, + ) + if lease_states: + lq = ','.join('?' for _ in lease_states) + n_leases = conn.execute( + f'DELETE FROM leases WHERE state IN ({lq})', lease_states + ).rowcount + if deployment_states: + gq = ','.join('?' for _ in deployment_states) + n_deployments = conn.execute( + f'DELETE FROM deployments WHERE state IN ({gq})', deployment_states + ).rowcount + return n_leases, n_deployments + + def active_leases_past(self, now: float) -> list[Lease]: + """Active leases whose TTL has elapsed (candidates for expiry).""" + rows = self._conn.execute( + 'SELECT * FROM leases WHERE state = ? AND expires_at IS NOT NULL' + ' AND expires_at <= ?', + (LeaseState.ACTIVE, now), + ).fetchall() + return [self._row_to_lease(r) for r in rows] + + def _row_to_lease(self, row: sqlite3.Row) -> Lease: + claims = self._conn.execute( + 'SELECT endpoint, deployment_id FROM claims WHERE lease_id = ?' + ' ORDER BY id', + (row['id'],), + ).fetchall() + return Lease( + id=row['id'], + owner=row['owner'], + state=row['state'], + created_at=row['created_at'], + ttl_seconds=row['ttl_seconds'], + expires_at=row['expires_at'], + heartbeat_at=row['heartbeat_at'], + endpoints=[c['endpoint'] for c in claims], + deployment_ids=list(dict.fromkeys(c['deployment_id'] for c in claims)), + ) + + # -- claims ------------------------------------------------------------ + + def insert_claim( + self, *, lease_id: str, endpoint: str, deployment_id: str, kind: str = 'endpoint' + ) -> None: + self._conn.execute( + 'INSERT INTO claims(lease_id, endpoint, deployment_id, kind)' + ' VALUES(?, ?, ?, ?)', + (lease_id, endpoint, deployment_id, kind), + ) + + # -- deployments ------------------------------------------------------------ + + def insert_deployment(self, deployment: Deployment) -> None: + self._conn.execute( + 'INSERT INTO deployments(id, compat_key, engine, sharing, capacity,' + ' spec, served, state, created_at, updated_at)' + ' VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', + ( + deployment.id, + deployment.compat_key, + deployment.engine, + deployment.sharing, + _dumps(deployment.capacity), + _dumps(deployment.spec), + _dumps(deployment.served), + deployment.state, + deployment.created_at, + deployment.updated_at, + ), + ) + + def set_deployment_state(self, deployment_id: str, state: str, updated_at: float) -> None: + self._conn.execute( + 'UPDATE deployments SET state = ?, updated_at = ? WHERE id = ?', + (state, updated_at, deployment_id), + ) + + def update_deployment_served( + self, deployment_id: str, served: dict[str, Any], updated_at: float + ) -> None: + self._conn.execute( + 'UPDATE deployments SET served = ?, updated_at = ? WHERE id = ?', + (_dumps(served), updated_at, deployment_id), + ) + + def get_deployment(self, deployment_id: str, *, now: float | None = None) -> Deployment | None: + row = self._conn.execute( + 'SELECT * FROM deployments WHERE id = ?', (deployment_id,) + ).fetchone() + if row is None: + return None + deployment = self._row_to_deployment(row) + if now is not None: + deployment.demand = self.demand(deployment_id, now) + return deployment + + def deployments_by_compat( + self, compat_key: str, *, sharing: str, states: tuple[str, ...] + ) -> list[Deployment]: + placeholders = ','.join('?' for _ in states) + rows = self._conn.execute( + 'SELECT * FROM deployments WHERE compat_key = ? AND sharing = ?' + f' AND state IN ({placeholders}) ORDER BY created_at', + (compat_key, sharing, *states), + ).fetchall() + return [self._row_to_deployment(r) for r in rows] + + def list_deployments(self, *, now: float) -> list[Deployment]: + rows = self._conn.execute( + 'SELECT * FROM deployments ORDER BY created_at' + ).fetchall() + deployments = [self._row_to_deployment(r) for r in rows] + for deployment in deployments: + deployment.demand = self.demand(deployment.id, now) + return deployments + + def deployments_for_lease(self, lease_id: str) -> list[Deployment]: + rows = self._conn.execute( + 'SELECT DISTINCT g.* FROM deployments g JOIN claims c' + ' ON c.deployment_id = g.id WHERE c.lease_id = ?', + (lease_id,), + ).fetchall() + return [self._row_to_deployment(r) for r in rows] + + def demand(self, deployment_id: str, now: float) -> int: + """Number of *protecting* leases referencing ``deployment_id``. + + A lease protects iff it is ACTIVE and not past its TTL, matching + :meth:`infer_stack.leasing.models.Lease.is_protecting`. + """ + row = self._conn.execute( + 'SELECT COUNT(DISTINCT c.lease_id) AS n FROM claims c' + ' JOIN leases l ON c.lease_id = l.id' + ' WHERE c.deployment_id = ? AND l.state = ?' + ' AND (l.expires_at IS NULL OR l.expires_at > ?)', + (deployment_id, LeaseState.ACTIVE, now), + ).fetchone() + return int(row['n']) + + def _row_to_deployment(self, row: sqlite3.Row) -> Deployment: + return Deployment( + id=row['id'], + compat_key=row['compat_key'], + engine=row['engine'], + sharing=row['sharing'], + capacity=_loads(row['capacity']) or {}, + spec=_loads(row['spec']) or {}, + served=_loads(row['served']) or {}, + state=row['state'], + created_at=row['created_at'], + updated_at=row['updated_at'], + ) diff --git a/infer_stack/leasing/suggest.py b/infer_stack/leasing/suggest.py new file mode 100644 index 00000000..bb7333c0 --- /dev/null +++ b/infer_stack/leasing/suggest.py @@ -0,0 +1,274 @@ +"""Seed a catalog from server introspection — the suggestion pool + the join. + +The leasing controller reads a hand-built ``catalog.yaml``, but a fresh host +shouldn't start empty. This module turns *what the server is* (the detected GPU +inventory) plus *what's worth running* (a curated, hardware-independent pool of +models) into a concrete, fits-this-box catalog the user can review and merge. + +The design is that **seeding is a pure function** +``inventory × pool → suggested catalog`` — the same compiler/controller split +the rest of the leasing redesign rests on. Nothing here is +baked into the catalog; re-run it on a new box and you get that box's +suggestions. The pool lives in ``templates/suggestion-pool.yaml`` (lifted from +the real entries of the legacy ``default-vllm-models.yaml``). + +Two layers, kept apart on purpose: + +* :class:`SuggestionModel` — intrinsic, portable model facts (footprint, min + per-GPU VRAM, preferred GPU count, context window, sane vLLM defaults). +* :func:`suggest_catalog` — derives the *server-specific* layer (which models + fit, what ``max_model_len`` / ``gpu_memory_utilization`` / ``dtype`` to use, + which one to keep warm) by joining the pool against an inventory dict. + +Example: + >>> from infer_stack.hardware import simulate_inventory + >>> # a single 48 GiB GPU (yardrat's free Quadro RTX 8000) + >>> out = suggest_catalog(simulate_inventory('1x48')) + >>> 'qwen2.5-7b' in out['models'] and 'qwen2.5-72b' not in out['models'] + True + >>> out['endpoints']['qwen2.5-7b']['runtime']['max_model_len'] + 32768 + >>> # the 72B (needs 2 GPUs) only appears once there are two + >>> 'qwen2.5-72b' in suggest_catalog(simulate_inventory('2x80'))['models'] + True +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import yaml + +from ..hardware import available_gpu_indices + +__all__ = [ + 'SuggestionModel', + 'builtin_pool', + 'load_pool', + 'fits_on', + 'derive_runtime', + 'suggest_catalog', +] + + +@dataclass +class SuggestionModel: + """One pool entry: intrinsic facts about a model, independent of hardware.""" + + name: str + hf_model_id: str + served_model_name: str | None = None + family: str | None = None + modalities: list[str] = field(default_factory=lambda: ['text']) + memory_class_gib: float = 0.0 + min_vram_gib_per_replica: float = 0.0 + preferred_gpu_count: int = 1 + context_window: int | None = None + defaults: dict[str, Any] = field(default_factory=dict) + + @classmethod + def from_entry(cls, name: str, spec: dict[str, Any]) -> SuggestionModel: + return cls( + name=name, + hf_model_id=spec['hf_model_id'], + served_model_name=spec.get('served_model_name'), + family=spec.get('family'), + modalities=list(spec.get('modalities') or ['text']), + memory_class_gib=spec.get('memory_class_gib', 0) or 0, + min_vram_gib_per_replica=( + spec.get('min_vram_gib_per_replica') + or spec.get('memory_class_gib', 0) + or 0 + ), + preferred_gpu_count=int(spec.get('preferred_gpu_count', 1) or 1), + context_window=spec.get('context_window'), + defaults=dict(spec.get('defaults') or {}), + ) + + +# --------------------------------------------------------------------------- +# pool loading +# --------------------------------------------------------------------------- + + +def load_pool(path: str | Path) -> dict[str, SuggestionModel]: + """Parse a suggestion-pool YAML file into typed entries.""" + data = yaml.safe_load(Path(path).expanduser().read_text()) or {} + return _pool_from_dict(data) + + +def builtin_pool() -> dict[str, SuggestionModel]: + """The curated pool shipped in ``templates/suggestion-pool.yaml``.""" + from importlib.resources import files + + text = ( + files('infer_stack') + .joinpath('templates/suggestion-pool.yaml') + .read_text(encoding='utf-8') + ) + return _pool_from_dict(yaml.safe_load(text) or {}) + + +def _pool_from_dict(data: dict[str, Any]) -> dict[str, SuggestionModel]: + entries = data.get('vllm_models') or {} + return { + name: SuggestionModel.from_entry(name, spec or {}) + for name, spec in entries.items() + } + + +# --------------------------------------------------------------------------- +# the join: inventory × pool → derived runtime +# --------------------------------------------------------------------------- + +# GPUs that predate Ampere (compute capability < 8.0) have no bf16, so vLLM must +# be pinned to fp16 (``--dtype=half``). The inventory does not yet carry the +# compute capability (``detect_inventory`` queries name/memory/display only), so +# we sniff the GPU *name*. This is the one place that wants a real signal: adding +# ``compute_cap`` to the nvidia-smi query in hardware.py would make this exact. +_PRE_AMPERE_NAME_HINTS = ( + 'quadro rtx', # Turing Quadro (RTX 8000/6000/5000/4000) — e.g. yardrat + 'titan rtx', + 'titan v', + 'tesla t4', + ' t4', + 'tesla v100', + 'v100', + 'rtx 20', # GeForce RTX 2080 etc. (Turing) + 'gtx 16', # GTX 1660 (Turing, no tensor cores) + 'gtx 10', # Pascal + 'tesla p100', + 'tesla p40', +) + + +def _needs_fp16(gpu_name: str | None) -> bool: + name = (gpu_name or '').lower() + return any(h in name for h in _PRE_AMPERE_NAME_HINTS) + + +def _gpu_mem(gpu: dict[str, Any]) -> float: + return float(gpu.get('memory_gib') or 0.0) + + +def fits_on(model: SuggestionModel, gpus: list[dict[str, Any]]) -> bool: + """True iff ``preferred_gpu_count`` GPUs each hold one replica's VRAM.""" + big_enough = [g for g in gpus if _gpu_mem(g) >= model.min_vram_gib_per_replica] + return len(big_enough) >= model.preferred_gpu_count + + +def _host_gpus( + model: SuggestionModel, gpus: list[dict[str, Any]] +) -> list[dict[str, Any]]: + """The GPUs this model would most plausibly land on (smallest that fit). + + Picking the *smallest* viable GPUs (rather than the largest) makes the + derived ``gpu_memory_utilization`` honest: it must reserve enough on the + tightest GPU the placer might choose, not the roomiest. + """ + big_enough = sorted( + (g for g in gpus if _gpu_mem(g) >= model.min_vram_gib_per_replica), + key=_gpu_mem, + ) + return big_enough[: model.preferred_gpu_count] + + +def derive_runtime( + model: SuggestionModel, gpus: list[dict[str, Any]] +) -> dict[str, Any]: + """Derive a concrete vLLM runtime block for ``model`` on ``gpus``. + + * ``max_model_len`` — the pool default, clamped to the context window. + * ``gpu_memory_utilization`` — sized from footprint ÷ host-GPU VRAM (with KV + headroom) so a small model on a big GPU does not greedily claim it all, + matching the hand-tuned 0.2-0.4 values in the leasing demo. + * ``tensor_parallel_size`` — ``preferred_gpu_count`` when > 1. + * ``extra_args: [--dtype=half]`` — only on pre-Ampere GPUs (no bf16). + """ + host = _host_gpus(model, gpus) + host_mem = min((_gpu_mem(g) for g in host), default=0.0) + + runtime: dict[str, Any] = {} + + want_len = model.defaults.get('max_model_len') or model.context_window + if want_len and model.context_window: + want_len = min(want_len, model.context_window) + if want_len: + runtime['max_model_len'] = int(want_len) + + # Footprint over the (smallest) host GPU, padded ~30%, then bounded to a sane + # band. Crucially this is a *floor-raiser*, never a floor-lowerer: the pool's + # own ``gpu_memory_utilization`` default encodes the fraction the model needs + # for its context's KV cache, so sizing *below* it (as the bare footprint + # ratio can on a big GPU) starves the KV cache and the engine OOMs at + # startup. We therefore take the max of the footprint estimate and the pool + # default, so the computed value can only *raise* the reservation on a + # smaller GPU where the model needs a bigger slice. Fall back to the default + # when host GPU mem is unknown. + default_util = model.defaults.get('gpu_memory_utilization') + if host_mem > 0 and model.min_vram_gib_per_replica > 0: + footprint = (model.min_vram_gib_per_replica * 1.3) / host_mem + util = footprint if default_util is None else max(footprint, default_util) + util = max(0.2, min(0.92, round(util, 2))) + else: + util = default_util if default_util is not None else 0.9 + runtime['gpu_memory_utilization'] = util + + if model.preferred_gpu_count > 1: + runtime['tensor_parallel_size'] = model.preferred_gpu_count + + if model.defaults.get('enable_prefix_caching'): + runtime['enable_prefix_caching'] = True + + if any(_needs_fp16(g.get('name')) for g in host): + runtime['extra_args'] = ['--dtype=half'] + + return runtime + + +# --------------------------------------------------------------------------- +# the top-level pure function +# --------------------------------------------------------------------------- + + +def suggest_catalog( + inventory: dict[str, Any], + *, + pool: dict[str, SuggestionModel] | None = None, + reserve_display_gpu: str | bool | None = False, +) -> dict[str, Any]: + """Join an inventory against the pool into a mergeable catalog fragment. + + Returns ``{'models': {...}, 'endpoints': {...}}`` shaped exactly like a + ``catalog.yaml`` (so it round-trips through + :meth:`~infer_stack.leasing.catalog.Catalog.from_dict` and merges into an + existing catalog without rewriting it). The single largest fitting model is + marked ``reclaim: keep-warm`` (worth the resident GPU to avoid cold-start + thrash); the rest ``reclaim: stop`` — mirroring the leasing demo. + + Pure and offline: pass ``simulate_inventory('2x80')`` to suggest for + hardware you do not have in front of you. + """ + pool = builtin_pool() if pool is None else pool + all_gpus = inventory.get('gpus') or [] + allowed = set(available_gpu_indices(inventory, reserve_display_gpu)) + gpus = [g for g in all_gpus if g.get('index') in allowed] + + fitting = [m for m in pool.values() if fits_on(m, gpus)] + # Largest first, so the keep-warm pick is the biggest thing this box can run. + fitting.sort(key=lambda m: m.min_vram_gib_per_replica, reverse=True) + + models: dict[str, Any] = {} + endpoints: dict[str, Any] = {} + for rank, model in enumerate(fitting): + models[model.name] = {'source': f'hf://{model.hf_model_id}'} + endpoint: dict[str, Any] = {'engine': 'vllm', 'model': model.name} + runtime = derive_runtime(model, gpus) + if runtime: + endpoint['runtime'] = runtime + endpoint['reclaim'] = {'policy': 'keep-warm' if rank == 0 else 'stop'} + endpoints[model.name] = endpoint + + return {'models': models, 'endpoints': endpoints} diff --git a/infer_stack/paths.py b/infer_stack/paths.py index e023e85f..ded3d0b4 100644 --- a/infer_stack/paths.py +++ b/infer_stack/paths.py @@ -45,10 +45,45 @@ DATA_DIR_ENV = 'INFER_STACK_DATA_DIR' MODEL_PATH_ENV = 'INFER_STACK_MODEL_PATH' +SETTINGS_FILENAME = 'settings.yaml' + _config_root_override: Path | None = None _data_root_override: Path | None = None +# --------------------------------------------------------------------------- +# durable user settings (config dir / leasing world): default backend, data dir +# --------------------------------------------------------------------------- + + +def settings_path() -> Path: + """Where the leasing-world user settings live (``config set`` writes here).""" + return config_root() / SETTINGS_FILENAME + + +def load_settings() -> dict: + """Load ``settings.yaml`` (empty dict if absent).""" + path = settings_path() + if path.exists(): + import yaml + + return yaml.safe_load(path.read_text()) or {} + return {} + + +def save_settings(settings: dict) -> Path: + import yaml + + path = settings_path() + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(yaml.safe_dump(settings, sort_keys=False)) + return path + + +def get_setting(key: str, default=None): + return load_settings().get(key, default) + + def _default_config_root() -> Path: return Path(ub.Path.appdir('infer_stack', type='config')) @@ -72,6 +107,12 @@ def data_root() -> Path: env = os.environ.get(DATA_DIR_ENV) if env: return Path(env).expanduser() + # Honor a persisted `config set data_dir ` before the XDG default, so + # "where my docker-mounted state lives" can live in the durable config + # instead of being re-exported every shell. + configured = load_settings().get('data_dir') + if configured: + return Path(configured).expanduser() return _default_data_root() diff --git a/infer_stack/cli/probes.py b/infer_stack/probe.py similarity index 53% rename from infer_stack/cli/probes.py rename to infer_stack/probe.py index 2b559604..03ca05aa 100644 --- a/infer_stack/cli/probes.py +++ b/infer_stack/probe.py @@ -1,83 +1,42 @@ -from __future__ import annotations - -from typing import Any -import requests +"""Low-level readiness probes for OpenAI-compatible and Ollama surfaces. +One implementation of "is this endpoint actually serving", used by the leasing +Compose backend (readiness waits). Each probe is a pure function over an injected HTTP +client (``http``, defaulting to the ``requests`` module), so it works against a +real server or a fake in tests, and stays out of the CLI layer that depends on +it (no upward imports). -def _default_model_for_deployment( - deployment: dict[str, Any], explicit: str | None = None -) -> str | None: - """Pick a reasonable model name for readiness/smoke probes.""" - if explicit: - return str(explicit) - litellm_routes = ( - (deployment.get('gateways', {}) or {}).get('litellm', {}) or {} - ).get('routes', {}) or {} - if litellm_routes: - return str(next(iter(litellm_routes))) - vllm_runtimes = ( - (deployment.get('providers', {}) or {}).get('vllm', {}) or {} - ).get('runtimes', {}) or {} - if vllm_runtimes: - first = next(iter(vllm_runtimes.values())) - return ( - str( - first.get('served_model_name') - or first.get('logical_model_name') - or first.get('runtime_name') - or '' - ) - or None - ) - ollama_routes = ( - (deployment.get('providers', {}) or {}).get('ollama', {}) or {} - ).get('routes', {}) or {} - if ollama_routes: - first = next(iter(ollama_routes.values())) - return ( - str(first.get('upstream_model') or first.get('model_ref') or '') - or None - ) - return None +Returns ``(ready, reason)`` and never raises for ordinary network errors — a +not-yet-ready endpoint is a normal, retryable outcome. +""" +from __future__ import annotations -def _resolve_smoke_protocol_from_deployment( - deployment: dict[str, Any], model_name: str | None -) -> str: - """Resolve chat vs completions from schema-v5 routes/runtimes.""" - if model_name: - routes = ( - (deployment.get('gateways', {}) or {}).get('litellm', {}) or {} - ).get('routes', {}) or {} - route = routes.get(model_name) - if route: - return str(route.get('protocol_mode') or 'chat') - vllm_runtimes = ( - (deployment.get('providers', {}) or {}).get('vllm', {}) or {} - ).get('runtimes', {}) or {} - for rt in vllm_runtimes.values(): - aliases = set(rt.get('served_aliases') or []) - aliases.add(str(rt.get('served_model_name') or '')) - if model_name in aliases: - return str(rt.get('protocol_mode') or 'chat') - return 'chat' +from typing import Any -def _ready_openai_probe( +def openai_ready( *, base_url: str, - headers: dict[str, str], - model: str | None, - protocol: str, - prompt: str, - max_tokens: int, - require_generation: bool, + headers: dict[str, str] | None = None, + model: str | None = None, + protocol: str = 'chat', + prompt: str = 'Reply with ready.', + max_tokens: int = 1, + require_generation: bool = True, + require_listed: bool = False, + http: Any = None, ) -> tuple[bool, str]: - """Probe an OpenAI-compatible surface once without exiting.""" + """Probe an OpenAI-compatible surface once without exiting. + + ``require_listed`` additionally insists the requested ``model`` appears in + ``/models`` (used by the leasing front door to confirm an alias is routable). + """ + import requests + http = http if http is not None else requests + headers = headers or {} try: - models_resp = requests.get( - f'{base_url}/models', headers=headers, timeout=10 - ) + models_resp = http.get(f'{base_url}/models', headers=headers, timeout=10) except requests.exceptions.RequestException as ex: return False, f'/models not reachable yet: {ex}' if models_resp.status_code >= 400: @@ -91,13 +50,16 @@ def _ready_openai_probe( except ValueError: return False, '/models returned non-JSON response' models = models_doc.get('data') or [] + listed = {m.get('id') for m in models} model_name = model or (models[0].get('id') if models else None) if not model_name: return False, '/models is reachable but no models are advertised' + if require_listed and model is not None and listed and model not in listed: + return False, f'{model} is not advertised by the gateway yet' if not require_generation: return True, f'/models is ready; selected model {model_name}' if protocol == 'completions': - payload = { + payload: dict[str, Any] = { 'model': model_name, 'prompt': prompt, 'max_tokens': max_tokens, @@ -111,9 +73,7 @@ def _ready_openai_probe( } endpoint = f'{base_url}/chat/completions' try: - resp = requests.post( - endpoint, headers=headers, json=payload, timeout=45 - ) + resp = http.post(endpoint, headers=headers, json=payload, timeout=45) except requests.exceptions.RequestException as ex: return False, f'{endpoint} not serving yet: {ex}' if resp.status_code >= 400: @@ -125,17 +85,20 @@ def _ready_openai_probe( return True, f'{model_name} served a {protocol} probe' -def _ready_ollama_probe( +def ollama_ready( *, base_url: str, - model: str | None, - prompt: str, - max_tokens: int, - require_generation: bool, + model: str | None = None, + prompt: str = 'Reply with ready.', + max_tokens: int = 1, + require_generation: bool = True, + http: Any = None, ) -> tuple[bool, str]: """Probe an Ollama-native surface once without exiting.""" + import requests + http = http if http is not None else requests try: - tags_resp = requests.get(f'{base_url}/api/tags', timeout=10) + tags_resp = http.get(f'{base_url}/api/tags', timeout=10) except requests.exceptions.RequestException as ex: return False, f'/api/tags not reachable yet: {ex}' if tags_resp.status_code >= 400: @@ -166,7 +129,7 @@ def _ready_ollama_probe( 'options': {'num_predict': max_tokens}, } try: - resp = requests.post(f'{base_url}/api/chat', json=payload, timeout=45) + resp = http.post(f'{base_url}/api/chat', json=payload, timeout=45) except requests.exceptions.RequestException as ex: return False, f'/api/chat not serving yet: {ex}' if resp.status_code >= 400: diff --git a/infer_stack/profile_runtime.py b/infer_stack/profile_runtime.py index e7ce6ffe..135788f5 100644 --- a/infer_stack/profile_runtime.py +++ b/infer_stack/profile_runtime.py @@ -12,7 +12,6 @@ def vllm_args(service: dict[str, Any]) -> list[str]: f'--gpu-memory-utilization={service["gpu_memory_utilization"]}', f'--max-num-batched-tokens={service["max_num_batched_tokens"]}', f'--max-num-seqs={service["max_num_seqs"]}', - '--disable-log-requests', ] if service.get('enable_prefix_caching'): args.append('--enable-prefix-caching') diff --git a/infer_stack/renderer.py b/infer_stack/renderer.py deleted file mode 100644 index 154493cc..00000000 --- a/infer_stack/renderer.py +++ /dev/null @@ -1,19 +0,0 @@ -from __future__ import annotations - -from .backends import render_compose_artifacts, render_kubeai_artifacts - - -def render_from_lock(lock_data: dict, *, assume_yes: bool = True) -> None: - """Render backend artifacts. - - ``assume_yes`` defaults to True so programmatic callers and tests - are unaffected. CLI entry points pass ``assume_yes=False`` to surface - the per-file diff confirmation prompt. - """ - backend = str( - lock_data.get('deployment', {}).get('backend', 'compose') - ).lower() - if backend == 'kubeai': - render_kubeai_artifacts(lock_data, assume_yes=assume_yes) - return - render_compose_artifacts(lock_data, assume_yes=assume_yes) diff --git a/infer_stack/resolver.py b/infer_stack/resolver.py deleted file mode 100644 index 3406e311..00000000 --- a/infer_stack/resolver.py +++ /dev/null @@ -1,1092 +0,0 @@ -from __future__ import annotations - -from copy import deepcopy -from typing import Any - -from .catalog import ( - canonical_profile_name, - normalize_ollama_models, - normalize_stack_profiles, - normalize_vllm_models, - sanitize_name, -) -from .config import ( - DEFAULT_PORTS, - PINNED_IMAGES, - deep_merge, - load_kubeai_resource_profiles, - merged_catalogs, - normalized_cluster, - normalized_output, - normalized_state, - resource_profiles_to_kubeai_values, -) -from .hardware import detect_inventory - - -def _available_gpu_indices( - inventory: dict[str, Any], reserve_display_gpu: str | bool | None -) -> list[int]: - gpus = deepcopy(inventory.get('gpus', [])) - if reserve_display_gpu == 'auto': - return [g['index'] for g in gpus if not g.get('display_active')] - if reserve_display_gpu is True: - return [g['index'] for g in gpus if not g.get('display_active')] - return [g['index'] for g in gpus] - - -def _first_fit( - available: list[int], count: int -) -> tuple[list[int], str | None]: - if len(available) < count: - return available[ - : - ], f'need {count} GPUs but only {len(available)} available' - return available[:count], None - - -def _runtime_value( - runtime: dict[str, Any], model: dict[str, Any], key: str, default: Any -) -> Any: - runtime_cfg = runtime.get('runtime', {}) or {} - if key in runtime_cfg: - return runtime_cfg[key] - if key in runtime: - return runtime[key] - return model.get('defaults', {}).get(key, default) - - -def _enabled_value(value: Any, default: bool = False) -> bool: - if value == 'auto': - return default - if value is None: - return default - if isinstance(value, str): - return value.lower() in {'1', 'true', 'yes', 'on', 'enabled'} - return bool(value) - - -def _as_string_map(value: Any) -> dict[str, str]: - """Normalize user-supplied environment/label maps without dropping values. - - Compose service override hooks are intentionally permissive. Keeping the - values as strings lets users pass literal values, numeric-looking settings, - booleans, and ``${ENV_VAR}`` substitutions through to Docker Compose. - """ - - if not value: - return {} - if not isinstance(value, dict): - raise TypeError( - f'expected a mapping, got {type(value).__name__}: {value!r}' - ) - return {str(k): '' if v is None else str(v) for k, v in value.items()} - - -def _as_string_list(value: Any) -> list[str]: - if value in (None, ''): - return [] - if isinstance(value, str): - return [value] - return [str(v) for v in value] - - -def _bool_string(value: Any, default: bool = False) -> str: - return 'true' if _enabled_value(value, default=default) else 'false' - - -def _service_override_fields(raw: dict[str, Any]) -> dict[str, Any]: - """Common opt-in escape hatches for rendered Compose services.""" - - return { - 'extra_env': _as_string_map( - raw.get('extra_env', raw.get('environment', {})) or {} - ), - 'env_file': _as_string_list(raw.get('env_file', [])), - 'extra_volumes': _as_string_list( - raw.get('extra_volumes', raw.get('volumes', [])) - ), - 'extra_hosts': _as_string_list(raw.get('extra_hosts', [])), - 'labels': _as_string_map(raw.get('labels', {})), - 'additional_ports': _as_string_list(raw.get('additional_ports', [])), - 'gpus': raw.get('gpus'), - } - - -def _resolve_gpu_indices( - *, - name: str, - placement: dict[str, Any], - topology: dict[str, Any], - preferred_gpu_count: int, - available: list[int], -) -> tuple[list[int], str | None]: - strategy = placement.get('strategy', 'first_fit') - if strategy in {'exact', 'multi_gpu', 'single_gpu'}: - gpu_indices = list(placement.get('gpu_indices', [])) - if not gpu_indices: - return ( - [], - f'{name} uses {strategy} placement but no gpu_indices were provided', - ) - return gpu_indices, None - gpu_count = int( - placement.get( - 'gpu_count', - topology.get('tensor_parallel_size', preferred_gpu_count) or 1, - ) - ) - return _first_fit(available, gpu_count) - - -def _merge(a: dict[str, Any], b: dict[str, Any]) -> dict[str, Any]: - out = deepcopy(a or {}) - for k, v in (b or {}).items(): - if isinstance(v, dict) and isinstance(out.get(k), dict): - out[k] = _merge(out[k], v) - else: - out[k] = deepcopy(v) - return out - - -def _resolve_vllm_runtime( - *, - profile: dict[str, Any], - runtime_name: str, - runtime: dict[str, Any], - models: dict[str, Any], - inventory: dict[str, Any], - policy: dict[str, Any], - used: set[int], - backend: str, -) -> dict[str, Any]: - model_key = runtime.get('model') or runtime.get('base_model') - if not model_key and runtime.get('hf_model_id'): - model_key = sanitize_name(runtime['hf_model_id']) - models = dict(models) - models[model_key] = { - 'key': model_key, - 'hf_model_id': runtime['hf_model_id'], - 'url': f'hf://{runtime["hf_model_id"]}', - 'served_model_name': runtime.get('served_model_name') or model_key, - 'logical_model_name': runtime.get('logical_model_name') - or model_key, - 'tokenizer_name': runtime.get('tokenizer_name') or model_key, - 'supported_protocols': ['chat', 'completions'], - 'modalities': ['text'], - 'features': ['TextGeneration'], - 'defaults': {}, - 'preferred_gpu_count': 1, - 'min_vram_gib_per_replica': 0, - 'resource_profile': '', - 'notes': [], - 'caveats': [], - } - if model_key not in models: - raise KeyError(f'Unknown vLLM model: {model_key}') - model = deepcopy(models[model_key]) - placement = deepcopy(runtime.get('placement', {})) - topology = deepcopy(runtime.get('topology', {})) - available = [ - i - for i in _available_gpu_indices( - inventory, policy.get('reserve_display_gpu', 'auto') - ) - if i not in used - ] - if 'tp' in topology and 'tensor_parallel_size' not in topology: - topology['tensor_parallel_size'] = topology['tp'] - if 'dp' in topology and 'data_parallel_size' not in topology: - topology['data_parallel_size'] = topology['dp'] - gpu_indices, placement_error = _resolve_gpu_indices( - name=f'vLLM runtime {runtime_name}', - placement=placement, - topology=topology, - preferred_gpu_count=int(model.get('preferred_gpu_count', 1) or 1), - available=available, - ) - if backend == 'compose': - used.update(gpu_indices) - tp = int( - topology.get( - 'tensor_parallel_size', - max(1, len(gpu_indices) or placement.get('gpu_count', 1)), - ) - ) - dp = int(topology.get('data_parallel_size', 1)) - - tool_calling = _merge( - model.get('tool_calling', {}), runtime.get('tool_calling', {}) - ) - tool_call_parser = tool_calling.get('parser') - tool_calling_on = bool( - tool_calling.get('enabled', tool_calling.get('auto', False)) - ) - enable_auto_tool_choice = bool(tool_calling_on and tool_call_parser) - - reasoning = _merge(model.get('reasoning', {}), runtime.get('reasoning', {})) - reasoning_enabled = bool(reasoning.get('enabled', False)) - reasoning_parser = reasoning.get('parser') if reasoning_enabled else None - reasoning_expose_to_openwebui = bool( - reasoning.get('expose_to_openwebui', reasoning_enabled) - ) - - chat_compat = _merge( - model.get('chat_compat', {}), runtime.get('chat_compat', {}) - ) - chat_compat_enabled = bool(chat_compat.get('enabled', False)) - chat_compat_strategy = ( - str(chat_compat.get('strategy', 'flat_messages')) - if chat_compat_enabled - else None - ) - - hf_model_id = runtime.get('hf_model_id', model.get('hf_model_id', '')) - served_model_name = ( - runtime.get('served_model_name') - or model.get('served_model_name') - or runtime_name - ) - logical_model_name = ( - runtime.get('logical_model_name') - or model.get('logical_model_name') - or served_model_name - ) - public_name = runtime.get('public_name') or runtime_name - protocol_mode = ( - runtime.get('protocol_mode') or runtime.get('protocol') or 'chat' - ) - model_url = ( - runtime.get('url') - or model.get('url') - or (f'hf://{hf_model_id}' if hf_model_id else '') - ) - - return { - 'provider': 'vllm', - 'runtime_name': runtime_name, - 'service_name': sanitize_name(runtime_name), - 'compose_service_name': f'vllm-{sanitize_name(runtime_name)}', - 'container_name': f'vllm-{sanitize_name(runtime_name)}', - 'profile_name': profile['name'], - 'profile_public_name': public_name, - 'kubernetes_name': sanitize_name(public_name), - 'model_ref': model_key, - 'hf_model_id': hf_model_id, - 'model_url': model_url, - 'logical_model_name': logical_model_name, - 'served_model_name': served_model_name, - 'served_aliases': [], - 'protocol_mode': protocol_mode, - 'supported_protocols': list( - model.get('supported_protocols', ['chat', 'completions']) - ), - 'modalities': model.get('modalities', ['text']), - 'features': deepcopy(model.get('features', ['TextGeneration'])), - 'engine': 'VLLM', - 'memory_class_gib': model.get('memory_class_gib'), - 'min_vram_gib_per_replica': model.get('min_vram_gib_per_replica', 0), - 'context_window': model.get('context_window'), - 'tokenizer_name': runtime.get( - 'tokenizer_name', model.get('tokenizer_name', logical_model_name) - ), - 'notes': deepcopy(model.get('notes', [])) - + deepcopy(runtime.get('notes', [])), - 'audit_notes': deepcopy(runtime.get('audit_notes', [])) - + deepcopy(model.get('caveats', [])), - 'tags': deepcopy(runtime.get('tags', [])), - 'gpu_indices': gpu_indices, - 'tensor_parallel_size': tp, - 'data_parallel_size': dp, - 'resource_profile': runtime.get( - 'resource_profile', model.get('resource_profile', '') - ), - 'min_replicas': int( - runtime.get( - 'min_replicas', model.get('defaults', {}).get('min_replicas', 0) - ) - ), - 'max_replicas': int( - runtime.get( - 'max_replicas', model.get('defaults', {}).get('max_replicas', 1) - ) - ), - 'priority_class_name': runtime.get( - 'priority_class_name', model.get('priority_class_name') - ), - 'max_model_len': int( - _runtime_value(runtime, model, 'max_model_len', 32768) - ), - 'gpu_memory_utilization': float( - _runtime_value(runtime, model, 'gpu_memory_utilization', 0.9) - ), - 'enable_prefix_caching': bool( - _runtime_value(runtime, model, 'enable_prefix_caching', True) - ), - 'max_num_batched_tokens': int( - _runtime_value(runtime, model, 'max_num_batched_tokens', 8192) - ), - 'max_num_seqs': int(_runtime_value(runtime, model, 'max_num_seqs', 16)), - 'thinking_history_policy': model.get( - 'thinking_history_policy', 'keep_final_only' - ), - 'placement': placement, - 'topology': topology, - 'placement_error': placement_error, - 'enable_auto_tool_choice': enable_auto_tool_choice, - 'tool_call_parser': tool_call_parser, - 'extra_args': deepcopy( - runtime.get( - 'extra_args', model.get('defaults', {}).get('extra_args', []) - ) - ), - 'reasoning_enabled': reasoning_enabled, - 'reasoning_parser': reasoning_parser, - 'reasoning_expose_to_openwebui': reasoning_expose_to_openwebui, - 'chat_compat_enabled': chat_compat_enabled, - 'chat_compat_strategy': chat_compat_strategy, - 'benchmark_transport': deepcopy( - runtime.get('benchmark_transport', runtime.get('transport', {})) - ), - 'publish_port': bool(runtime.get('publish_port', False)), - 'host_port': runtime.get('host_port'), - **_service_override_fields(runtime), - } - - -def _resolve_ollama_model_tag( - model_ref: str, ollama_models: dict[str, Any] -) -> tuple[str, dict[str, Any]]: - if model_ref in ollama_models: - model = deepcopy(ollama_models[model_ref]) - return model['tag'], model - return str(model_ref), { - 'key': model_ref, - 'tag': str(model_ref), - 'served_model_name': sanitize_name(str(model_ref)), - 'defaults': {}, - } - - -def _collect_ollama_needed(profile: dict[str, Any]) -> bool: - p = profile.get('providers', {}).get('ollama', {}) or {} - if _enabled_value(p.get('enabled'), default=False): - return True - for route in (profile.get('routes', {}) or {}).values(): - if str(route.get('provider', '')).lower() == 'ollama': - return True - return False - - -def _resolve_ollama_provider( - profile: dict[str, Any], - config: dict[str, Any], - inventory: dict[str, Any], - policy: dict[str, Any], -) -> dict[str, Any]: - profile_ollama = deepcopy( - profile.get('providers', {}).get('ollama', {}) or {} - ) - # Historical configs used the top-level ``ollama`` section for provider - # details. Newer configs may keep those details under - # ``providers.ollama``. Support both so users can opt into advanced - # settings without having to rewrite old config files. - config_ollama = deepcopy(config.get('ollama', {}) or {}) - config_provider_ollama = deepcopy( - (config.get('providers', {}) or {}).get('ollama', {}) or {} - ) - merged = _merge( - _merge(config_ollama, config_provider_ollama), profile_ollama - ) - enabled = _enabled_value( - merged.get('enabled'), default=_collect_ollama_needed(profile) - ) - if not enabled: - return {'enabled': False, 'routes': {}} - - gpu_indices = merged.get('gpu_indices', 'auto') - placement_error = None - if gpu_indices == 'auto' or gpu_indices is None: - available = _available_gpu_indices( - inventory, policy.get('reserve_display_gpu', 'auto') - ) - gpu_indices = available - else: - gpu_indices = list(gpu_indices) - return { - 'enabled': True, - 'service_name': 'ollama', - 'base_url': 'http://ollama:11434', - 'host_port': int(config.get('ports', {}).get('ollama', 11434)), - 'gpu_indices': gpu_indices, - 'publish_port': bool(merged.get('publish_port', False)), - 'host': str(merged.get('host', '0.0.0.0:11434')), - 'keep_alive': str(merged.get('keep_alive', '2m')), - 'context_length': int(merged.get('context_length', 4096)), - 'num_parallel': int(merged.get('num_parallel', 1)), - 'max_loaded_models': int(merged.get('max_loaded_models', 1)), - 'max_queue': int(merged.get('max_queue', 8)), - 'placement_error': placement_error, - 'routes': {}, - **_service_override_fields(merged), - } - - -def _resolve_routes( - profile: dict[str, Any], - vllm_runtimes: dict[str, Any], - ollama_models: dict[str, Any], -) -> dict[str, Any]: - routes: dict[str, Any] = {} - for alias, raw in (profile.get('routes', {}) or {}).items(): - route = deepcopy(raw) or {} - provider = str(route.get('provider', 'vllm')).lower() - aliases = route.get('aliases') or [alias] - if isinstance(aliases, str): - aliases = [aliases] - aliases = [str(a) for a in aliases] - if alias not in aliases: - aliases.insert(0, alias) - for public_alias in aliases: - if provider == 'vllm': - runtime_name = ( - route.get('runtime') - or route.get('service') - or route.get('target') - ) - if runtime_name is None and len(vllm_runtimes) == 1: - runtime_name = next(iter(vllm_runtimes)) - if runtime_name not in vllm_runtimes: - raise KeyError( - f'route {alias!r} references unknown vLLM runtime {runtime_name!r}' - ) - rt = vllm_runtimes[runtime_name] - routes[public_alias] = { - 'alias': public_alias, - 'provider': 'vllm', - 'runtime': runtime_name, - 'service_name': rt['service_name'], - 'upstream_service_name': rt.get( - 'compose_service_name', rt['service_name'] - ), - 'served_model_name': rt['served_model_name'], - 'protocol_mode': rt['protocol_mode'], - 'max_model_len': rt['max_model_len'], - 'chat_compat_enabled': rt.get('chat_compat_enabled', False), - 'chat_compat_strategy': rt.get('chat_compat_strategy'), - } - elif provider == 'ollama': - model_ref = ( - route.get('model') or route.get('tag') or public_alias - ) - tag, model = _resolve_ollama_model_tag( - str(model_ref), ollama_models - ) - defaults = deepcopy(model.get('defaults', {})) - defaults.update(deepcopy(route.get('options', {}))) - max_model_len = int( - route.get('max_model_len') - or defaults.get('num_ctx') - or model.get('context_window') - or 4096 - ) - routes[public_alias] = { - 'alias': public_alias, - 'provider': 'ollama', - 'upstream_model': tag, - 'model_ref': str(model_ref), - 'protocol_mode': 'chat', - 'max_model_len': max_model_len, - 'options': defaults, - } - else: - raise KeyError( - f'route {alias!r} uses unknown provider {provider!r}' - ) - return routes - - -def _resolve_litellm( - profile: dict[str, Any], - routes: dict[str, Any], - providers: dict[str, Any], - backend: str, - config: dict[str, Any], -) -> dict[str, Any]: - if backend == 'kubeai': - return {'enabled': False, 'base_url': '', 'routes': {}} - cfg = deepcopy(config.get('litellm', {}) or {}) - cfg = _merge( - cfg, (config.get('gateways', {}) or {}).get('litellm', {}) or {} - ) - cfg = _merge(cfg, profile.get('gateways', {}).get('litellm', {}) or {}) - route_providers = {route.get('provider') for route in routes.values()} - needs_gateway = bool(routes) and (backend == 'compose') - if len(route_providers) > 1: - needs_gateway = True - enabled = _enabled_value(cfg.get('enabled'), default=needs_gateway) - return { - 'enabled': bool(enabled), - 'base_url': 'http://litellm:4000/v1' if enabled else '', - 'routes': routes if enabled else {}, - **_service_override_fields(cfg), - } - - -def _resolve_open_webui_ldap(raw: dict[str, Any]) -> dict[str, Any]: - ldap = deepcopy(raw.get('ldap', {}) or {}) - enabled = _enabled_value(ldap.get('enabled'), default=False) - if not enabled: - return {'enabled': False, 'env': {}, 'env_defaults': {}} - - # Defaults are intentionally expressed through .env placeholders so LDAP - # credentials and site-local directory settings can be edited after render - # without changing generated Compose YAML. - env = { - 'ENABLE_LDAP': _bool_string(True), - 'LDAP_SERVER_LABEL': str(ldap.get('server_label', 'OpenLDAP')), - 'LDAP_SERVER_HOST': str(ldap.get('server_host', '${LDAP_HOST}')), - 'LDAP_SERVER_PORT': str(ldap.get('server_port', '${LDAP_PORT}')), - 'LDAP_USE_TLS': str(ldap.get('use_tls', '${LDAP_USE_TLS}')), - 'LDAP_VALIDATE_CERT': str( - ldap.get( - 'validate_cert', - ldap.get('validate_certs', '${LDAP_VALIDATE_CERTS}'), - ) - ), - 'LDAP_APP_DN': str(ldap.get('app_dn', '${LDAP_BASEDN}')), - 'LDAP_APP_PASSWORD': str(ldap.get('app_password', '${LDAP_PASSWD}')), - 'LDAP_SEARCH_BASE': str(ldap.get('search_base', '${LDAP_SEARCH_BASE}')), - 'LDAP_ATTRIBUTE_FOR_USERNAME': str( - ldap.get('attribute_for_username', '${LDAP_ATTRIBUTE_FOR_USERNAME}') - ), - 'LDAP_ATTRIBUTE_FOR_MAIL': str(ldap.get('attribute_for_mail', 'mail')), - 'LDAP_SEARCH_FILTER': str( - ldap.get('search_filter', '${LDAP_SEARCH_FILTER}') - ), - } - env.update(_as_string_map(ldap.get('extra_env', {}))) - env_defaults = { - 'LDAP_HOST': str(ldap.get('env_defaults', {}).get('LDAP_HOST', '')), - 'LDAP_PORT': str(ldap.get('env_defaults', {}).get('LDAP_PORT', '636')), - 'LDAP_USE_TLS': str( - ldap.get('env_defaults', {}).get('LDAP_USE_TLS', 'true') - ), - 'LDAP_VALIDATE_CERTS': str( - ldap.get('env_defaults', {}).get('LDAP_VALIDATE_CERTS', 'true') - ), - 'LDAP_BASEDN': str(ldap.get('env_defaults', {}).get('LDAP_BASEDN', '')), - 'LDAP_PASSWD': str(ldap.get('env_defaults', {}).get('LDAP_PASSWD', '')), - 'LDAP_SEARCH_BASE': str( - ldap.get('env_defaults', {}).get('LDAP_SEARCH_BASE', '') - ), - 'LDAP_ATTRIBUTE_FOR_USERNAME': str( - ldap.get('env_defaults', {}).get( - 'LDAP_ATTRIBUTE_FOR_USERNAME', 'uid' - ) - ), - 'LDAP_SEARCH_FILTER': str( - ldap.get('env_defaults', {}).get('LDAP_SEARCH_FILTER', '') - ), - } - return {'enabled': True, 'env': env, 'env_defaults': env_defaults} - - -def _resolve_open_webui( - profile: dict[str, Any], - gateways: dict[str, Any], - providers: dict[str, Any], - backend: str, - config: dict[str, Any], -) -> dict[str, Any]: - if backend == 'kubeai': - return { - 'enabled': False, - 'auth': False, - 'provider': 'none', - 'publish_port': False, - } - raw = deepcopy(config.get('open_webui', {}) or {}) - raw = _merge( - raw, (config.get('frontends', {}) or {}).get('open_webui', {}) or {} - ) - raw = _merge(raw, profile.get('frontends', {}).get('open_webui', {}) or {}) - default_enabled = backend == 'compose' and ( - gateways.get('litellm', {}).get('enabled') - or providers.get('ollama', {}).get('enabled') - ) - enabled = _enabled_value(raw.get('enabled'), default=bool(default_enabled)) - provider = str(raw.get('provider', 'auto')) - if provider == 'auto': - if gateways.get('litellm', {}).get('enabled'): - provider = 'litellm' - elif providers.get('ollama', {}).get('enabled'): - provider = 'ollama' - else: - provider = 'none' - ldap = _resolve_open_webui_ldap(raw) - return { - 'enabled': enabled, - 'auth': bool(raw.get('auth', False)), - 'provider': provider, - 'publish_port': _enabled_value(raw.get('publish_port'), default=True), - 'webui_url': str(raw.get('webui_url', raw.get('WEBUI_URL', ''))), - 'cors_allow_origin': str( - raw.get('cors_allow_origin', raw.get('CORS_ALLOW_ORIGIN', '')) - ), - 'ldap': ldap, - **_service_override_fields(raw), - } - - -def _resolve_reverse_proxy( - profile: dict[str, Any], backend: str, config: dict[str, Any] -) -> dict[str, Any]: - if backend == 'kubeai': - return {'enabled': False} - - raw = deepcopy(config.get('reverse_proxy', {}) or {}) - raw = _merge( - raw, (config.get('frontends', {}) or {}).get('reverse_proxy', {}) or {} - ) - raw = _merge( - raw, profile.get('frontends', {}).get('reverse_proxy', {}) or {} - ) - enabled = _enabled_value(raw.get('enabled'), default=False) - if not enabled: - return {'enabled': False} - - ssl = deepcopy(raw.get('ssl', {}) or {}) - hsts = deepcopy(raw.get('hsts', {}) or {}) - ssl_enabled = _enabled_value(ssl.get('enabled'), default=True) - target = str(raw.get('target', 'open_webui')) - if target in {'open_webui', 'open-webui'}: - target_service = 'open-webui' - target_port = 8080 - target_scheme = 'http' - depends_on = ['open-webui'] - elif target == 'litellm': - target_service = 'litellm' - target_port = 4000 - target_scheme = 'http' - depends_on = ['litellm'] - elif target == 'ollama': - target_service = 'ollama' - target_port = 11434 - target_scheme = 'http' - depends_on = ['ollama'] - else: - target_service = str(raw.get('target_service', target)) - target_port = int(raw.get('target_port', 8080)) - target_scheme = str(raw.get('target_scheme', 'http')) - depends_on = _as_string_list(raw.get('depends_on', [])) - - service_overrides = _service_override_fields(raw) - ports = config.get('ports', {}) or {} - return { - 'enabled': True, - 'image': str( - raw.get('image') - or config.get('images', {}).get('nginx') - or PINNED_IMAGES.get('nginx', 'nginx:alpine') - ), - 'service_name': str(raw.get('service_name', 'reverse-proxy')), - 'container_name': str(raw.get('container_name', 'reverse-proxy')), - 'target': target, - 'target_service': target_service, - 'target_port': target_port, - 'target_scheme': target_scheme, - 'depends_on': depends_on, - 'server_name': str(raw.get('server_name', 'localhost')), - 'publish_http': _enabled_value(raw.get('publish_http'), default=True), - # Publishing :443 is only meaningful when there is a TLS server block to - # listen on it, so gate it on ssl regardless of how publish_https was - # defaulted (the top-level config template ships it as true). - 'publish_https': _enabled_value(raw.get('publish_https'), default=True) - and ssl_enabled, - 'http_port': int( - raw.get('http_port') or ports.get('reverse_proxy_http') or 80 - ), - 'https_port': int( - raw.get('https_port') or ports.get('reverse_proxy_https') or 443 - ), - 'http_bind_host': str(raw.get('http_bind_host', '')), - 'https_bind_host': str(raw.get('https_bind_host', '')), - 'force_https': _enabled_value( - raw.get('force_https'), default=ssl_enabled - ), - 'client_max_body_size': str(raw.get('client_max_body_size', '1G')), - 'proxy_connect_timeout': str(raw.get('proxy_connect_timeout', '60s')), - 'proxy_read_timeout': str(raw.get('proxy_read_timeout', '600s')), - 'proxy_send_timeout': str(raw.get('proxy_send_timeout', '600s')), - 'proxy_buffer_size': str(raw.get('proxy_buffer_size', '128k')), - 'proxy_buffers': str(raw.get('proxy_buffers', '4 256k')), - 'proxy_busy_buffers_size': str( - raw.get('proxy_busy_buffers_size', '256k') - ), - 'proxy_buffering': raw.get('proxy_buffering'), - 'proxy_cache': raw.get('proxy_cache'), - 'resolver': _as_string_list(raw.get('resolver', [])), - 'resolver_timeout': str(raw.get('resolver_timeout', '5s')), - 'hsts': { - 'enabled': _enabled_value(hsts.get('enabled'), default=True), - 'max_age': int(hsts.get('max_age', 63072000)), - 'include_subdomains': _enabled_value( - hsts.get('include_subdomains'), default=True - ), - 'preload': _enabled_value(hsts.get('preload'), default=False), - }, - 'ssl': { - 'enabled': ssl_enabled, - 'certificate': str(ssl.get('certificate', '')), - 'certificate_key': str(ssl.get('certificate_key', '')), - 'dhparam': str(ssl.get('dhparam', '')), - 'certificate_container_path': str( - ssl.get( - 'certificate_container_path', - '/etc/ssl/certs/infer-stack-site.crt', - ) - ), - 'certificate_key_container_path': str( - ssl.get( - 'certificate_key_container_path', - '/etc/ssl/private/infer-stack-site.key', - ) - ), - 'dhparam_container_path': str( - ssl.get('dhparam_container_path', '/etc/ssl/certs/dhparam.pem') - ), - 'protocols': str(ssl.get('protocols', 'TLSv1.2 TLSv1.3')), - 'ciphers': str( - ssl.get( - 'ciphers', - 'ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384', - ) - ), - 'prefer_server_ciphers': _enabled_value( - ssl.get('prefer_server_ciphers'), default=True - ), - 'session_cache': str(ssl.get('session_cache', 'shared:SSL:10m')), - 'ecdh_curve': str(ssl.get('ecdh_curve', 'secp384r1')), - 'session_tickets': _enabled_value( - ssl.get('session_tickets'), default=False - ), - 'stapling': _enabled_value(ssl.get('stapling'), default=True), - 'stapling_verify': _enabled_value( - ssl.get('stapling_verify'), default=True - ), - }, - 'config_path': str(raw.get('config_path', '')), - 'extra_config': str(raw.get('extra_config', '')), - **service_overrides, - } - - -def _serving_profile( - profile: dict[str, Any], - vllm_runtimes: dict[str, Any], - routes: dict[str, Any], -) -> dict[str, Any]: - first_rt = next(iter(vllm_runtimes.values()), {}) if vllm_runtimes else {} - first_route = next(iter(routes.values()), {}) if routes else {} - public = ( - first_route.get('alias') - or first_rt.get('profile_public_name') - or profile['name'] - ) - return { - 'name': profile['name'], - 'public_name': public, - 'kind': profile.get('kind', 'stack'), - 'description': profile.get('description', ''), - 'base_model': first_rt.get( - 'model_ref', first_route.get('model_ref', '') - ), - 'logical_model_name': first_rt.get( - 'logical_model_name', first_route.get('alias', '') - ), - 'served_model_name': first_rt.get( - 'served_model_name', first_route.get('upstream_model', '') - ), - 'served_aliases': list(routes.keys()), - 'protocol_mode': first_rt.get( - 'protocol_mode', first_route.get('protocol_mode', 'chat') - ), - 'engine': 'mixed' - if vllm_runtimes - and any(r.get('provider') == 'ollama' for r in routes.values()) - else ('VLLM' if vllm_runtimes else 'OLLAMA'), - 'resource_profile': first_rt.get('resource_profile', ''), - 'service_name': first_rt.get('service_name', ''), - 'kubernetes_name': first_rt.get( - 'kubernetes_name', sanitize_name(profile['name']) - ), - 'tags': deepcopy(profile.get('tags', [])), - 'audit_notes': deepcopy(profile.get('audit_notes', [])), - 'notes': deepcopy(profile.get('notes', [])), - 'benchmark_transport': {}, - } - - -def _resolve_access(deployment: dict[str, Any]) -> dict[str, Any]: - ports = deployment.get('ports', {}) - access: dict[str, Any] = {} - litellm = deployment['gateways']['litellm'] - ollama = deployment['providers']['ollama'] - vllm_runtimes = deployment['providers']['vllm'].get('runtimes', {}) - frontends = deployment.get('frontends', {}) or {} - open_webui = frontends.get('open_webui', {}) or {} - reverse_proxy = frontends.get('reverse_proxy', {}) or {} - if deployment.get('backend') == 'kubeai': - ingress = deployment.get('cluster', {}).get('ingress', {}) or {} - base = ( - f'http://{ingress["host"]}/openai/v1' - if ingress.get('enabled') and ingress.get('host') - else 'http://127.0.0.1:8000/openai/v1' - ) - access['default'] = { - 'kind': 'openai-compatible', - 'base_url': base, - 'auth_env_name': 'KUBEAI_OPENAI_API_KEY', - } - elif litellm.get('enabled'): - access['default'] = { - 'kind': 'openai-compatible', - 'base_url': f'http://127.0.0.1:{ports.get("litellm", 14042)}/v1', - 'auth_env_name': 'LITELLM_MASTER_KEY', - } - elif ollama.get('enabled'): - access['default'] = { - 'kind': 'ollama-native', - 'base_url': f'http://127.0.0.1:{ports.get("ollama", 11434)}', - 'auth_required': False, - } - access['ollama_openai'] = { - 'kind': 'openai-compatible', - 'base_url': f'http://127.0.0.1:{ports.get("ollama", 11434)}/v1', - 'auth_required': False, - } - elif vllm_runtimes: - name, rt = next(iter(vllm_runtimes.items())) - port = rt.get('host_port') or 18000 - access['default'] = { - 'kind': 'openai-compatible', - 'base_url': f'http://127.0.0.1:{port}/v1', - 'auth_env_name': 'VLLM_BACKEND_API_KEY', - } - else: - access['default'] = {'kind': 'none', 'base_url': ''} - if open_webui.get('enabled') and open_webui.get('publish_port'): - access['open_webui'] = { - 'kind': 'web-ui', - 'base_url': f'http://127.0.0.1:{ports.get("open_webui", 13000)}', - 'auth_required': bool(open_webui.get('auth')), - } - if reverse_proxy.get('enabled'): - scheme = ( - 'https' - if reverse_proxy.get('ssl', {}).get('enabled', True) - and reverse_proxy.get('publish_https', True) - else 'http' - ) - host = reverse_proxy.get('server_name') or '127.0.0.1' - port = ( - reverse_proxy.get('https_port') - if scheme == 'https' - else reverse_proxy.get('http_port') - ) - default_port = 443 if scheme == 'https' else 80 - port_suffix = ( - '' if int(port or default_port) == default_port else f':{port}' - ) - access['reverse_proxy'] = { - 'kind': 'web-ui', - 'base_url': f'{scheme}://{host}{port_suffix}', - 'auth_required': bool(open_webui.get('auth')), - } - for idx, (name, rt) in enumerate(vllm_runtimes.items()): - port = rt.get('host_port') or (18000 + idx) - access[f'vllm_{name}'] = { - 'kind': 'openai-compatible', - 'base_url': f'http://127.0.0.1:{port}/v1', - 'auth_env_name': 'VLLM_BACKEND_API_KEY', - } - return access - - -def resolve( - config: dict[str, Any], - inventory: dict[str, Any] | None = None, - profile_name: str | None = None, -) -> dict[str, Any]: - raw_catalogs = merged_catalogs(config) - vllm_models = normalize_vllm_models( - raw_catalogs.get('vllm_models', raw_catalogs.get('models', {})) - ) - ollama_models = normalize_ollama_models( - raw_catalogs.get('ollama_models', {}) - ) - profiles = normalize_stack_profiles( - raw_catalogs.get('profiles', {}), vllm_models, ollama_models - ) - inventory = ( - deepcopy(inventory) if inventory is not None else detect_inventory() - ) - effective_profile_name = canonical_profile_name( - profile_name or config.get('active_profile') - ) - if effective_profile_name not in profiles: - raise KeyError(f'Unknown profile: {effective_profile_name}') - profile = deepcopy(profiles[effective_profile_name]) - if profile.get('kind') == 'invalid-profile': - raise KeyError( - f'Profile {effective_profile_name!r} is invalid: {profile.get("catalog_error", "unknown error")}' - ) - - backend = str(config.get('backend', 'compose')).lower() - merged_policy = _merge(config.get('policy', {}), profile.get('policy', {})) - images = deep_merge(PINNED_IMAGES, config.get('images', {}) or {}) - ports = deep_merge(DEFAULT_PORTS, config.get('ports', {}) or {}) - state = normalized_state(config.get('state', {})) - output = normalized_output(config.get('output', {})) - - used: set[int] = set() - vllm_cfg = deepcopy(profile.get('providers', {}).get('vllm', {}) or {}) - runtime_cfgs = deepcopy(vllm_cfg.get('runtimes', {}) or {}) - vllm_runtimes: dict[str, Any] = {} - for runtime_name, runtime in runtime_cfgs.items(): - resolved_rt = _resolve_vllm_runtime( - profile=profile, - runtime_name=str(runtime_name), - runtime=runtime, - models=vllm_models, - inventory=inventory, - policy=merged_policy, - used=used, - backend=backend, - ) - vllm_runtimes[str(runtime_name)] = resolved_rt - - ollama_provider = _resolve_ollama_provider( - profile, config, inventory, merged_policy - ) - routes = _resolve_routes(profile, vllm_runtimes, ollama_models) - for alias, route in routes.items(): - if ( - route.get('provider') == 'vllm' - and route.get('runtime') in vllm_runtimes - ): - aliases = vllm_runtimes[route['runtime']].setdefault( - 'served_aliases', [] - ) - if alias not in aliases: - aliases.append(alias) - ollama_routes = { - k: v for k, v in routes.items() if v.get('provider') == 'ollama' - } - if ollama_routes and not ollama_provider.get('enabled'): - ollama_provider = _resolve_ollama_provider( - { - **profile, - 'providers': { - **profile.get('providers', {}), - 'ollama': {'enabled': True}, - }, - }, - config, - inventory, - merged_policy, - ) - ollama_provider['routes'] = ollama_routes - - providers = { - 'ollama': ollama_provider, - 'vllm': {'enabled': bool(vllm_runtimes), 'runtimes': vllm_runtimes}, - } - litellm = _resolve_litellm(profile, routes, providers, backend, config) - gateways = {'litellm': litellm} - frontends = { - 'open_webui': _resolve_open_webui( - profile, gateways, providers, backend, config - ) - } - frontends['reverse_proxy'] = _resolve_reverse_proxy( - profile, backend, config - ) - serving_profile = _serving_profile(profile, vllm_runtimes, routes) - - if backend == 'kubeai': - resource_profiles, resource_profiles_values, resource_profiles_path = ( - load_kubeai_resource_profiles() - ) - if resource_profiles: - resource_profile_source = str(resource_profiles_path) - else: - resource_profiles = deepcopy(config.get('resource_profiles', {})) - resource_profiles_values = deepcopy( - { - 'resourceProfiles': resource_profiles_to_kubeai_values( - resource_profiles - )['resourceProfiles'] - } - ) - resource_profile_source = 'config.yaml.resource_profiles' - else: - resource_profiles = deepcopy(config.get('resource_profiles', {})) - resource_profiles_values = deepcopy( - { - 'resourceProfiles': resource_profiles_to_kubeai_values( - resource_profiles - )['resourceProfiles'] - } - ) - resource_profile_source = 'config.yaml.resource_profiles' - - # Compatibility aliases. New templates should use providers/gateways/frontends. - services = list(vllm_runtimes.values()) - router_aliases = { - alias: route.get('service_name', route.get('upstream_model', '')) - for alias, route in routes.items() - } - - deployment = { - 'schema_version': 5, - 'source': { - 'config_file': 'config.yaml', - 'active_profile': effective_profile_name, - }, - 'backend': backend, - 'images': images, - 'ports': ports, - 'policy': merged_policy, - 'vllm': { - 'enable_responses_api_store': bool( - profile.get('vllm', {}).get('enable_responses_api_store', False) - ), - 'logging_level': str( - profile.get('vllm', {}).get('logging_level', 'INFO') - ), - }, - 'state': state, - 'output': output, - 'cluster': normalized_cluster(config.get('cluster', {})), - 'resource_profiles': resource_profiles, - 'resource_profiles_values': resource_profiles_values, - 'resource_profiles_source': resource_profile_source, - 'inventory': inventory, - 'providers': providers, - 'gateways': gateways, - 'frontends': frontends, - 'access': {}, - 'profile': serving_profile, - 'serving_profile': deepcopy(serving_profile), - 'services': services, - 'router': { - 'enabled': litellm.get('enabled'), - 'type': 'litellm' if litellm.get('enabled') else 'none', - 'aliases': router_aliases, - }, - 'open_webui': frontends['open_webui'], - } - deployment['access'] = _resolve_access(deployment) - return deployment diff --git a/infer_stack/templates/default-models.yaml b/infer_stack/templates/default-models.yaml deleted file mode 100644 index e0fa7dc6..00000000 --- a/infer_stack/templates/default-models.yaml +++ /dev/null @@ -1,674 +0,0 @@ -models: - # Smallest Qwen3.5 — intended for fast smoke tests, local development, - # and high-concurrency utility tasks. Fits on a single 8 GiB GPU at FP16 - # with room for activations and a modest KV cache. - qwen3.5-0.8b: - hf_model_id: Qwen/Qwen3.5-0.8B - served_model_name: qwen3.5-0.8b - family: qwen3.5 - modalities: [text] - memory_class_gib: 4 - min_vram_gib_per_replica: 4 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 32 - qwen3.5-2b: - hf_model_id: Qwen/Qwen3.5-2B - served_model_name: qwen3.5-2b - family: qwen3.5 - modalities: [text] - memory_class_gib: 8 - min_vram_gib_per_replica: 8 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 32 - qwen3.5-4b: - hf_model_id: Qwen/Qwen3.5-4B - served_model_name: qwen3.5-4b - family: qwen3.5 - modalities: [text] - memory_class_gib: 12 - min_vram_gib_per_replica: 12 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 24 - qwen3.5-9b: - hf_model_id: Qwen/Qwen3.5-9B - served_model_name: qwen3.5-9b - family: qwen3.5 - modalities: [text] - memory_class_gib: 20 - min_vram_gib_per_replica: 20 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - qwen3.5-27b: - hf_model_id: Qwen/Qwen3.5-27B - served_model_name: qwen3.5-27b - family: qwen3.5 - modalities: [text] - memory_class_gib: 48 - min_vram_gib_per_replica: 48 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - qwen3.5-35b-a3b: - hf_model_id: Qwen/Qwen3.5-35B-A3B - served_model_name: qwen3.5-35b-a3b - family: qwen3.5 - modalities: [text] - memory_class_gib: 48 - min_vram_gib_per_replica: 48 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - qwen3.5-122b-a10b: - hf_model_id: Qwen/Qwen3.5-122B-A10B - served_model_name: qwen3.5-122b-a10b - family: qwen3.5 - modalities: [text] - memory_class_gib: 80 - min_vram_gib_per_replica: 80 - preferred_gpu_count: 2 - context_window: 131072 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 8 - - qwen3.5-122b-a10b-fp8: - hf_model_id: Qwen/Qwen3.5-122B-A10B-FP8 - served_model_name: qwen3.5-122b-a10b-fp8 - family: qwen3.5 - modalities: [text] - memory_class_gib: 80 - min_vram_gib_per_replica: 80 - preferred_gpu_count: 2 - context_window: 262144 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 8 - tool_calling: - auto: true - parser: qwen3_xml - - qwen3.6-35b-a3b: - hf_model_id: Qwen/Qwen3.6-35B-A3B - served_model_name: qwen3.6-35b-a3b - tokenizer_name: Qwen/Qwen3.6-35B-A3B - family: qwen3.6 - modalities: [text] - memory_class_gib: 80 - min_vram_gib_per_replica: 24 - preferred_gpu_count: 2 - context_window: 262144 - supported_protocols: [chat, completions] - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 262144 - gpu_memory_utilization: 0.95 - enable_prefix_caching: false - max_num_batched_tokens: 8192 - max_num_seqs: 8 - - gemma4-e2b: - hf_model_id: google/gemma-4-E2B-it - served_model_name: gemma4-e2b - family: gemma4 - modalities: [text, image, audio] - memory_class_gib: 8 - min_vram_gib_per_replica: 8 - preferred_gpu_count: 1 - context_window: 131072 - thinking_history_policy: keep_final_only - notes: - - Historical turns should keep only final responses, not prior thinking content. - defaults: - max_model_len: 65536 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 16 - tool_calling: - auto: true - parser: gemma4 - gemma4-e4b: - hf_model_id: google/gemma-4-E4B-it - served_model_name: gemma4-e4b - family: gemma4 - modalities: [text, image, audio] - memory_class_gib: 10 - min_vram_gib_per_replica: 10 - preferred_gpu_count: 1 - context_window: 131072 - thinking_history_policy: keep_final_only - notes: - - Historical turns should keep only final responses, not prior thinking content. - defaults: - max_model_len: 65536 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 16 - tool_calling: - auto: true - parser: gemma4 - gemma4-26b: - hf_model_id: google/gemma-4-26B-A4B-it - served_model_name: gemma4-26b - family: gemma4 - modalities: [text, image] - memory_class_gib: 20 - min_vram_gib_per_replica: 20 - preferred_gpu_count: 1 - context_window: 262144 - thinking_history_policy: keep_final_only - notes: - - Historical turns should keep only final responses, not prior thinking content. - defaults: - max_model_len: 65536 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 12 - tool_calling: - auto: true - parser: gemma4 - gemma4-31b: - hf_model_id: google/gemma-4-31B-it - served_model_name: gemma4-31b - family: gemma4 - modalities: [text, image] - memory_class_gib: 22 - min_vram_gib_per_replica: 22 - preferred_gpu_count: 1 - context_window: 262144 - thinking_history_policy: keep_final_only - notes: - - Historical turns should keep only final responses, not prior thinking content. - defaults: - max_model_len: 65536 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 12 - tool_calling: - auto: true - parser: gemma4 - - # ── HELM evaluation models ──────────────────────────────────────────────── - # served_model_name matches the HELM side tag so HELM can route correctly. - - qwen2-72b-instruct: - hf_model_id: Qwen/Qwen2-72B-Instruct - served_model_name: qwen/qwen2-72b-instruct - logical_model_name: qwen/qwen2-72b-instruct - tokenizer_name: qwen/qwen2-72b-instruct - family: qwen2 - modalities: [text] - memory_class_gib: 144 - min_vram_gib_per_replica: 144 - preferred_gpu_count: 2 - context_window: 131072 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 8 - - # Smallest *real* instruct-tuned Qwen — usable for true end-to-end - # smoke tests that actually pull weights and exercise vLLM. ~1 GiB fp16. - qwen2.5-0.5b-instruct: - hf_model_id: Qwen/Qwen2.5-0.5B-Instruct - served_model_name: qwen/qwen2.5-0.5b-instruct - logical_model_name: qwen/qwen2.5-0.5b-instruct - tokenizer_name: Qwen/Qwen2.5-0.5B-Instruct - family: qwen2.5 - modalities: [text] - memory_class_gib: 2 - min_vram_gib_per_replica: 2 - preferred_gpu_count: 1 - context_window: 32768 - defaults: - max_model_len: 8192 - gpu_memory_utilization: 0.85 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 16 - - qwen2.5-7b-instruct: - hf_model_id: Qwen/Qwen2.5-7B-Instruct - served_model_name: qwen/qwen2.5-7b-instruct-turbo - logical_model_name: qwen/qwen2.5-7b-instruct-turbo - tokenizer_name: qwen/qwen2.5-7b-instruct-turbo - family: qwen2.5 - modalities: [text] - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 131072 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - - qwen2.5-72b-instruct: - hf_model_id: Qwen/Qwen2.5-72B-Instruct - served_model_name: qwen/qwen2.5-72b-instruct-turbo - logical_model_name: qwen/qwen2.5-72b-instruct-turbo - tokenizer_name: qwen/qwen2.5-72b-instruct-turbo - family: qwen2.5 - modalities: [text] - memory_class_gib: 144 - min_vram_gib_per_replica: 144 - preferred_gpu_count: 2 - context_window: 131072 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 8 - - gpt-oss-20b: - hf_model_id: openai/gpt-oss-20b - served_model_name: openai/gpt-oss-20b - logical_model_name: openai/gpt-oss-20b - tokenizer_name: openai/o200k_harmony - family: gpt-oss - modalities: [text] - memory_class_gib: 40 - min_vram_gib_per_replica: 40 - preferred_gpu_count: 1 - context_window: 131072 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - - llama-3.1-8b-instruct: - hf_model_id: meta-llama/Meta-Llama-3.1-8B-Instruct - served_model_name: meta/llama-3.1-8b-instruct-turbo - family: llama3.1 - modalities: [text] - memory_class_gib: 18 - min_vram_gib_per_replica: 18 - preferred_gpu_count: 1 - context_window: 131072 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - - llama-3-8b-instruct: - hf_model_id: meta-llama/Meta-Llama-3-8B-Instruct - served_model_name: meta/llama-3-8b-chat - family: llama3 - modalities: [text] - memory_class_gib: 18 - min_vram_gib_per_replica: 18 - preferred_gpu_count: 1 - context_window: 8192 - defaults: - max_model_len: 8192 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - - llama-2-7b: - hf_model_id: meta-llama/Llama-2-7b-hf - served_model_name: meta/llama-2-7b - family: llama2 - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 4096 - defaults: - max_model_len: 4096 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 16 - - llama-2-13b: - hf_model_id: meta-llama/Llama-2-13b-hf - served_model_name: meta/llama-2-13b - family: llama2 - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 28 - min_vram_gib_per_replica: 28 - preferred_gpu_count: 1 - context_window: 4096 - defaults: - max_model_len: 4096 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 16 - - # LLaMA-2-70B at fp16 needs ~140 GB; the default tp=2 layout fits on - # two 96 GB GPUs with headroom for KV cache. Public HELM ran fp16, - # so the audit recipe matches that — no quantization in this profile. - llama-2-70b: - hf_model_id: meta-llama/Llama-2-70b-hf - served_model_name: meta/llama-2-70b - family: llama2 - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 140 - min_vram_gib_per_replica: 70 - preferred_gpu_count: 2 - context_window: 4096 - defaults: - max_model_len: 4096 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 8 - - mistral-7b-instruct-v0.3: - hf_model_id: mistralai/Mistral-7B-Instruct-v0.3 - served_model_name: mistralai/mistral-7b-instruct-v0.3 - family: mistral - modalities: [text] - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 32768 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - - mistral-7b-v0.1: - hf_model_id: mistralai/Mistral-7B-v0.1 - served_model_name: mistralai/mistral-7b-v0.1 - family: mistral - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 32768 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - - # Falcon uses ALiBi attention: prefix caching must be disabled. - falcon-7b: - hf_model_id: tiiuae/falcon-7b - served_model_name: tiiuae/falcon-7b - family: falcon - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - - falcon-7b-instruct: - hf_model_id: tiiuae/falcon-7b-instruct - served_model_name: tiiuae/falcon-7b-instruct - family: falcon - modalities: [text] - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - - # GPT-2 base, 124M params (~250 MB fp16). Completions-only — no chat - # template. Useful as a near-zero-cost plumbing smoke test: download is - # fast, no HF_TOKEN required, and the request path covers the whole stack. - gpt2: - hf_model_id: openai-community/gpt2 - served_model_name: gpt2 - logical_model_name: gpt2 - tokenizer_name: openai-community/gpt2 - family: gpt2 - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 1 - min_vram_gib_per_replica: 1 - preferred_gpu_count: 1 - context_window: 1024 - defaults: - max_model_len: 1024 - gpu_memory_utilization: 0.3 - enable_prefix_caching: true - max_num_batched_tokens: 1024 - max_num_seqs: 16 - - # Smallest practical instruct-tuned model on HuggingFace. ~270 MB fp16. - # Ideal for CI smoke tests that need real chat-template output. - smollm2-135m-instruct: - hf_model_id: HuggingFaceTB/SmolLM2-135M-Instruct - served_model_name: huggingfacetb/smollm2-135m-instruct - logical_model_name: huggingfacetb/smollm2-135m-instruct - tokenizer_name: HuggingFaceTB/SmolLM2-135M-Instruct - family: smollm2 - modalities: [text] - memory_class_gib: 1 - min_vram_gib_per_replica: 1 - preferred_gpu_count: 1 - context_window: 8192 - defaults: - max_model_len: 4096 - gpu_memory_utilization: 0.6 - enable_prefix_caching: true - max_num_batched_tokens: 2048 - max_num_seqs: 16 - - # Pythia uses ALiBi attention: prefix caching must be disabled. - pythia-160m: - hf_model_id: EleutherAI/pythia-160m - served_model_name: eleutherai/pythia-160m - logical_model_name: eleutherai/pythia-160m - tokenizer_name: EleutherAI/pythia-160m - family: pythia - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 1 - min_vram_gib_per_replica: 1 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.6 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - - pythia-410m: - hf_model_id: EleutherAI/pythia-410m - served_model_name: eleutherai/pythia-410m - logical_model_name: eleutherai/pythia-410m - tokenizer_name: EleutherAI/pythia-410m - family: pythia - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 2 - min_vram_gib_per_replica: 2 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.7 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - - pythia-6.9b: - hf_model_id: EleutherAI/pythia-6.9b - served_model_name: eleutherai/pythia-6.9b - logical_model_name: eleutherai/pythia-6.9b - tokenizer_name: eleutherai/pythia-6.9b - family: pythia - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - - pythia-2.8b-v0: - hf_model_id: EleutherAI/pythia-2.8b-v0 - served_model_name: eleutherai/pythia-2.8b-v0 - logical_model_name: eleutherai/pythia-2.8b-v0 - tokenizer_name: eleutherai/pythia-2.8b-v0 - family: pythia - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 8 - min_vram_gib_per_replica: 8 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - - pythia-1b-v0: - hf_model_id: EleutherAI/pythia-1b-v0 - served_model_name: eleutherai/pythia-1b-v0 - logical_model_name: eleutherai/pythia-1b-v0 - tokenizer_name: eleutherai/pythia-1b-v0 - family: pythia - modalities: [text] - supported_protocols: [completions] - memory_class_gib: 4 - min_vram_gib_per_replica: 4 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - - vicuna-7b-v1.3: - hf_model_id: lmsys/vicuna-7b-v1.3 - served_model_name: lmsys/vicuna-7b-v1.3 - logical_model_name: lmsys/vicuna-7b-v1.3 - tokenizer_name: lmsys/vicuna-7b-v1.3 - family: vicuna - supported_protocols: [completions] - notes: - - Use a no-chat-template style profile when reproducing legacy completion-style prompting. - modalities: [text] - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 diff --git a/infer_stack/templates/default-ollama-models.yaml b/infer_stack/templates/default-ollama-models.yaml deleted file mode 100644 index 1cc8c5d7..00000000 --- a/infer_stack/templates/default-ollama-models.yaml +++ /dev/null @@ -1,31 +0,0 @@ -ollama_models: - qwen3.5-2b: - tag: qwen3.5:2b - served_model_name: qwen3.5-2b - defaults: - num_ctx: 4096 - temperature: 0.2 - qwen3.5-4b: - tag: qwen3.5:4b - served_model_name: qwen3.5-4b - defaults: - num_ctx: 4096 - temperature: 0.2 - qwen3.5-9b-q4: - tag: qwen3.5:9b-q4_K_M - served_model_name: qwen3.5-9b-q4 - defaults: - num_ctx: 4096 - temperature: 0.2 - gemma4-e2b: - tag: gemma4:e2b - served_model_name: gemma4-e2b - defaults: - num_ctx: 4096 - temperature: 0.2 - smollm2-135m: - tag: smollm2:135m - served_model_name: smollm2-135m - defaults: - num_ctx: 4096 - temperature: 0.2 diff --git a/infer_stack/templates/default-profiles.yaml b/infer_stack/templates/default-profiles.yaml deleted file mode 100644 index 603bfc84..00000000 --- a/infer_stack/templates/default-profiles.yaml +++ /dev/null @@ -1,1731 +0,0 @@ -profiles: - ollama-direct: - description: Local Ollama daemon with Open WebUI connected directly. No model - registry or LiteLLM required. - providers: - ollama: - enabled: true - gpu_indices: - - 0 - - 1 - publish_port: true - keep_alive: 2m - context_length: 4096 - num_parallel: 1 - max_loaded_models: 1 - max_queue: 8 - gateways: - litellm: - enabled: false - frontends: - open_webui: - enabled: true - provider: ollama - routes: {} - openwebui-tls-ldap: - description: Ollama + Open WebUI behind an nginx TLS reverse proxy with LDAP login, - and no public Ollama/Open WebUI ports. Fill in LDAP and TLS settings in - config.yaml/.env before using on a real host. - providers: - ollama: - enabled: true - gpu_indices: auto - gpus: all - publish_port: false - keep_alive: 2m - context_length: 4096 - num_parallel: 0 - max_loaded_models: 1 - max_queue: 8 - extra_env: - OLLAMA_FLASH_ATTENTION: '1' - OLLAMA_NO_CLOUD: 'true' - OLLAMA_SCHED_SPREAD: 'true' - gateways: - litellm: - enabled: false - frontends: - open_webui: - enabled: true - provider: ollama - auth: true - publish_port: false - gpus: all - ldap: - enabled: true - server_label: OpenLDAP - server_host: ${LDAP_HOST} - server_port: ${LDAP_PORT} - use_tls: ${LDAP_USE_TLS} - validate_cert: ${LDAP_VALIDATE_CERTS} - app_dn: ${LDAP_BASEDN} - app_password: ${LDAP_PASSWD} - search_base: ${LDAP_SEARCH_BASE} - attribute_for_username: ${LDAP_ATTRIBUTE_FOR_USERNAME} - attribute_for_mail: mail - search_filter: ${LDAP_SEARCH_FILTER} - env_defaults: - LDAP_PORT: '636' - LDAP_USE_TLS: 'true' - LDAP_VALIDATE_CERTS: 'true' - LDAP_ATTRIBUTE_FOR_USERNAME: uid - reverse_proxy: - enabled: true - image: nginx:1.29.7-alpine - target: open_webui - server_name: openwebui.example.com - publish_http: true - publish_https: true - force_https: true - client_max_body_size: 1G - ssl: - enabled: true - certificate: ./certs/site.crt - certificate_key: ./certs/site.key - dhparam: ./dhparam.pem - routes: {} - ollama-qwen3.5-4b-gateway: - description: Qwen3.5 4B through Ollama behind LiteLLM/Open WebUI. - providers: - ollama: - enabled: true - gpu_indices: - - 0 - - 1 - keep_alive: 2m - context_length: 4096 - num_parallel: 1 - max_loaded_models: 1 - max_queue: 8 - gateways: - litellm: - enabled: true - frontends: - open_webui: - enabled: true - provider: litellm - routes: - qwen3.5-4b: - provider: ollama - model: qwen3.5-4b - ollama-qwen3.5-2b-gateway: - description: Qwen3.5 2B through Ollama behind LiteLLM/Open WebUI. - providers: - ollama: - enabled: true - gpu_indices: - - 0 - - 1 - keep_alive: 2m - context_length: 4096 - num_parallel: 1 - max_loaded_models: 1 - max_queue: 8 - gateways: - litellm: - enabled: true - frontends: - open_webui: - enabled: true - provider: litellm - routes: - qwen3.5-2b: - provider: ollama - model: qwen3.5-2b - mixed-ollama-smollm: - description: 'Example mixed stack: Ollama local assistant plus tiny vLLM route - behind LiteLLM.' - providers: - ollama: - enabled: true - gpu_indices: - - 0 - - 1 - keep_alive: 2m - context_length: 4096 - num_parallel: 1 - max_loaded_models: 1 - max_queue: 8 - vllm: - runtimes: - smollm: - model: smollm2-135m-instruct - placement: - strategy: first_fit - gpu_count: 1 - gateways: - litellm: - enabled: true - frontends: - open_webui: - enabled: true - provider: litellm - routes: - home-assistant-local: - provider: ollama - model: qwen3.5-4b - smollm2-135m: - provider: vllm - runtime: smollm - raw-ollama-vllm: - description: 'Raw backend servers only: Ollama plus tiny vLLM, no gateway and - no UI.' - providers: - ollama: - enabled: true - gpu_indices: - - 0 - - 1 - publish_port: true - keep_alive: 2m - context_length: 4096 - vllm: - runtimes: - smollm: - model: smollm2-135m-instruct - placement: - strategy: first_fit - gpu_count: 1 - publish_port: true - gateways: - litellm: - enabled: false - frontends: - open_webui: - enabled: false - routes: {} - workstation-safe: - description: Single model on the first non-display GPU. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: chat - model: qwen3.5-9b - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - qwen3.5-9b: chat - qwen3.5-0.8b-single: - description: Tiny Qwen3.5 0.8B on a single GPU; smallest CLI-runnable profile. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: chat - model: qwen3.5-0.8b - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - qwen3.5-0.8b: chat - gpt2-single: - description: GPT-2 124M (completions-only) on one GPU; smallest plumbing test - (~250 MB, no HF_TOKEN needed). - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: chat - model: gpt2 - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - gpt2: chat - smollm2-135m-single: - description: SmolLM2 135M Instruct on one GPU; smallest real-weights profile (~270 - MB). - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: chat - model: smollm2-135m-instruct - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - smollm2-135m: chat - qwen2.5-0.5b-instruct-single: - description: Qwen2.5 0.5B Instruct on one GPU; smallest real Qwen for end-to-end - tests. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: chat - model: qwen2.5-0.5b-instruct - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - qwen2.5-0.5b: chat - pythia-160m-single: - description: Pythia 160M (completions-only) on one GPU; smallest real base-model - profile. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: chat - model: pythia-160m - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - pythia-160m: chat - pythia-410m-single: - description: Pythia 410M (completions-only) on one GPU. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: chat - model: pythia-410m - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - pythia-410m: chat - test-single-11gb: - description: Integration-test profile for a single ~11 GB GPU; uses SmolLM2 135M. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: chat - model: smollm2-135m-instruct - placement: - strategy: first_fit - gpu_count: 1 - runtime: - max_model_len: 2048 - gpu_memory_utilization: 0.5 - max_num_batched_tokens: 2048 - max_num_seqs: 8 - enable_prefix_caching: true - router: - aliases: - smollm2-135m: chat - test-multi-gpu: - description: Integration-test profile across 2 GPUs (TP=2); uses Qwen2.5 0.5B - Instruct. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: chat - model: qwen2.5-0.5b-instruct - placement: - strategy: first_fit - gpu_count: 2 - topology: - tensor_parallel_size: 2 - runtime: - max_model_len: 4096 - gpu_memory_utilization: 0.6 - max_num_batched_tokens: 4096 - max_num_seqs: 8 - enable_prefix_caching: true - router: - aliases: - qwen2.5-0.5b: chat - qwen-mixed: - description: Multiple Qwen models behind LiteLLM. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: qwen-27b - model: qwen3.5-27b - placement: - strategy: first_fit - gpu_count: 1 - - service_name: qwen-2b - model: qwen3.5-2b - placement: - strategy: first_fit - gpu_count: 1 - - service_name: qwen-122b - model: qwen3.5-122b-a10b - placement: - strategy: first_fit - gpu_count: 2 - topology: - tensor_parallel_size: 2 - router: - aliases: - qwen3.5-27b: qwen-27b - qwen3.5-2b: qwen-2b - qwen3.5-122b: qwen-122b - gemma-mixed: - description: Mix small and medium Gemma 4 models behind LiteLLM. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: gemma-e4b - model: gemma4-e4b - placement: - strategy: first_fit - gpu_count: 1 - - service_name: gemma-31b - model: gemma4-31b - placement: - strategy: first_fit - gpu_count: 1 - - service_name: gemma-26b - model: gemma4-26b - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - gemma4-e4b: gemma-e4b - gemma4-31b: gemma-31b - gemma4-26b: gemma-26b - pythia-qwen3.6-mixed-4x96: - description: Serve Pythia 6.9B, Pythia 2.8B, and Qwen3.6-35B-A3B on a 4x96GB host. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: qwen36-35b - model: qwen3.6-35b-a3b - protocol_mode: chat - placement: - strategy: exact - gpu_indices: - - 0 - - 1 - topology: - tensor_parallel_size: 2 - runtime: - max_model_len: 262144 - gpu_memory_utilization: 0.95 - max_num_batched_tokens: 8192 - max_num_seqs: 8 - enable_prefix_caching: false - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - tool_calling: - enabled: true - parser: qwen3_coder - extra_args: - - --language-model-only - - service_name: pythia-69b - model: pythia-6.9b - protocol_mode: completions - placement: - strategy: exact - gpu_indices: - - 2 - runtime: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 2048 - max_num_seqs: 16 - enable_prefix_caching: false - chat_compat: - enabled: true - strategy: flat_messages - - service_name: pythia-28b - model: pythia-2.8b-v0 - protocol_mode: completions - placement: - strategy: exact - gpu_indices: - - 3 - runtime: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 2048 - max_num_seqs: 16 - enable_prefix_caching: false - chat_compat: - enabled: true - strategy: flat_messages - router: - aliases: - qwen3.6-35b-a3b: qwen36-35b - eleutherai/pythia-6.9b: pythia-69b - eleutherai/pythia-2.8b-v0: pythia-28b - qwen3.6-35b-a3b-dual-tp2-4x96: - description: Two Qwen3.6-35B-A3B tp=2 services across 4x96GB GPUs at 262k context. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: qwen36-35b-gpu01 - model: qwen3.6-35b-a3b - served_model_name: qwen3.6-35b-a3b-gpu01 - protocol_mode: chat - placement: - strategy: exact - gpu_indices: - - 0 - - 1 - topology: - tensor_parallel_size: 2 - runtime: - max_model_len: 262144 - gpu_memory_utilization: 0.95 - max_num_batched_tokens: 8192 - max_num_seqs: 8 - enable_prefix_caching: false - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - extra_args: - - --language-model-only - - service_name: qwen36-35b-gpu23 - model: qwen3.6-35b-a3b - served_model_name: qwen3.6-35b-a3b-gpu23 - protocol_mode: chat - placement: - strategy: exact - gpu_indices: - - 2 - - 3 - topology: - tensor_parallel_size: 2 - runtime: - max_model_len: 262144 - gpu_memory_utilization: 0.95 - max_num_batched_tokens: 8192 - max_num_seqs: 8 - enable_prefix_caching: false - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - extra_args: - - --language-model-only - router: - aliases: - qwen3.6-35b-a3b-gpu01: qwen36-35b-gpu01 - qwen3.6-35b-a3b-gpu23: qwen36-35b-gpu23 - qwen3.5-122b-a10b-tp4-4x96: - description: Single Qwen3.5-122B-A10B service across 4x96GB GPUs (TP=4) at 128k - context. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: qwen-122b - model: qwen3.5-122b-a10b - served_model_name: qwen3.5-122b-a10b-128k - protocol_mode: chat - placement: - strategy: exact - gpu_indices: - - 0 - - 1 - - 2 - - 3 - topology: - tensor_parallel_size: 4 - runtime: - max_model_len: 131072 - gpu_memory_utilization: 0.95 - max_num_batched_tokens: 1024 - max_num_seqs: 1 - enable_prefix_caching: false - extra_args: - - --language-model-only - router: - aliases: - qwen3.5-122b-a10b-128k: qwen-122b - qwen3.5-122b-a10b-fp8-tp4-4x96: - description: Single Qwen3.5-122B-A10B-FP8 service across 4x96GB GPUs (TP=4) at - 262k context. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: qwen-122b-fp8 - model: qwen3.5-122b-a10b-fp8 - served_model_name: qwen3.5-122b-a10b-fp8-262k - protocol_mode: chat - placement: - strategy: exact - gpu_indices: - - 0 - - 1 - - 2 - - 3 - topology: - tensor_parallel_size: 4 - runtime: - max_model_len: 262144 - gpu_memory_utilization: 0.95 - max_num_batched_tokens: 1024 - max_num_seqs: 1 - enable_prefix_caching: false - extra_args: - - --language-model-only - router: - aliases: - qwen3.5-122b-a10b-fp8-262k: qwen-122b-fp8 - pythia-qwen25-gptoss-mixed-4x96: - description: Serve gpt-oss-20b + Qwen 2.5 7B Instruct + Pythia 6.9B + Pythia 2.8B - on a 4x96GB host for the finish_qwen25_gptoss audit batch. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: gpt-oss-20b - model: gpt-oss-20b - protocol_mode: completions - placement: - strategy: exact - gpu_indices: - - 0 - runtime: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 8192 - max_num_seqs: 16 - enable_prefix_caching: true - chat_compat: - enabled: true - strategy: flat_messages - - service_name: qwen25-7b - model: qwen2.5-7b-instruct - protocol_mode: chat - placement: - strategy: exact - gpu_indices: - - 1 - runtime: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 8192 - max_num_seqs: 16 - enable_prefix_caching: true - - service_name: pythia-69b - model: pythia-6.9b - protocol_mode: completions - placement: - strategy: exact - gpu_indices: - - 2 - runtime: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 2048 - max_num_seqs: 16 - enable_prefix_caching: false - chat_compat: - enabled: true - strategy: flat_messages - - service_name: pythia-28b - model: pythia-2.8b-v0 - protocol_mode: completions - placement: - strategy: exact - gpu_indices: - - 3 - runtime: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 2048 - max_num_seqs: 16 - enable_prefix_caching: false - chat_compat: - enabled: true - strategy: flat_messages - router: - aliases: - openai/gpt-oss-20b: gpt-oss-20b - qwen/qwen2.5-7b-instruct-turbo: qwen25-7b - eleutherai/pythia-6.9b: pythia-69b - eleutherai/pythia-2.8b-v0: pythia-28b - pythia-llama2-70b-mixed-4x96: - description: Serve LLaMA-2-70B (fp16, tp=2) + Pythia 6.9B + Pythia 2.8B on a 4x96GB - host for the LLaMA-2-70B audit batch. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: llama2-70b - model: llama-2-70b - protocol_mode: completions - placement: - strategy: exact - gpu_indices: - - 0 - - 1 - topology: - tensor_parallel_size: 2 - runtime: - max_model_len: 4096 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 4096 - max_num_seqs: 8 - enable_prefix_caching: true - chat_compat: - enabled: true - strategy: flat_messages - - service_name: pythia-69b - model: pythia-6.9b - protocol_mode: completions - placement: - strategy: exact - gpu_indices: - - 2 - runtime: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 2048 - max_num_seqs: 16 - enable_prefix_caching: false - chat_compat: - enabled: true - strategy: flat_messages - - service_name: pythia-28b - model: pythia-2.8b-v0 - protocol_mode: completions - placement: - strategy: exact - gpu_indices: - - 3 - runtime: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 2048 - max_num_seqs: 16 - enable_prefix_caching: false - chat_compat: - enabled: true - strategy: flat_messages - router: - aliases: - meta/llama-2-70b: llama2-70b - eleutherai/pythia-6.9b: pythia-69b - eleutherai/pythia-2.8b-v0: pythia-28b - gemma4-26b-workstation: - description: Efficient default for agentic coding and general chat on a 4x96GB - server. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - name: gemma-main - model: gemma4-26b - served_model_name: gemma4-26b - placement: - strategy: single_gpu - gpu_indices: - - 0 - prefer_homogeneous_group: true - topology: - tp: 1 - dp: 1 - runtime: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - max_num_seqs: 8 - max_num_batched_tokens: 8192 - enable_prefix_caching: true - enable_chunked_prefill: true - reasoning_history_policy: final_only - router: - aliases: - gemma4-26b: gemma-main - policy: - require_fit_validation: true - minimum_vram_headroom_gib: 8 - allow_unsupported_render: false - gemma4-31b-quality: - description: Higher-quality Gemma 4 profile for coding/reasoning on a 4x96GB server. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: gemma-main - model: gemma4-31b - served_model_name: gemma4-31b - placement: - strategy: single_gpu - gpu_indices: - - 0 - prefer_homogeneous_group: true - topology: - tp: 1 - dp: 1 - runtime: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - max_num_seqs: 8 - max_num_batched_tokens: 8192 - enable_prefix_caching: true - enable_chunked_prefill: true - reasoning_history_policy: final_only - router: - aliases: - gemma4-31b: gemma-main - policy: - require_fit_validation: true - minimum_vram_headroom_gib: 8 - allow_unsupported_render: false - gemma4-31b-max-context: - description: Maximize context window for Gemma 4 31B on a 4x96GB server. Sacrifices - concurrency and co-hosting. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: gemma-main - model: gemma4-31b - served_model_name: gemma4-31b - placement: - strategy: multi_gpu - gpu_indices: - - 0 - - 1 - - 2 - - 3 - prefer_homogeneous_group: true - topology: - tp: 4 - dp: 1 - runtime: - max_model_len: 262144 - gpu_memory_utilization: 0.95 - max_num_seqs: 1 - max_num_batched_tokens: 4096 - enable_prefix_caching: true - enable_chunked_prefill: true - reasoning_history_policy: final_only - router: - aliases: - gemma4-31b: gemma-main - policy: - require_fit_validation: true - minimum_vram_headroom_gib: 4 - allow_unsupported_render: false - gemma4-26b-max-context: - description: Maximize context window for Gemma 4 26B on a 4x96GB server. Sacrifices - concurrency and co-hosting. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: gemma-main - model: gemma4-26b - served_model_name: gemma4-26b - placement: - strategy: multi_gpu - gpu_indices: - - 0 - - 1 - - 2 - - 3 - prefer_homogeneous_group: true - topology: - tp: 4 - dp: 1 - runtime: - max_model_len: 262144 - gpu_memory_utilization: 0.95 - max_num_seqs: 1 - max_num_batched_tokens: 4096 - enable_prefix_caching: true - enable_chunked_prefill: true - reasoning_history_policy: final_only - router: - aliases: - gemma4-26b: gemma-main - policy: - require_fit_validation: true - minimum_vram_headroom_gib: 4 - allow_unsupported_render: false - gemma4-e4b-3090-workstation: - description: Single RTX 3090 workstation profile. Uses GPU 1 because GPU 0 is - reserved for graphics. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: gemma-main - model: gemma4-e4b - served_model_name: gemma4-e4b - placement: - strategy: single_gpu - gpu_indices: - - 1 - prefer_homogeneous_group: true - topology: - tp: 1 - dp: 1 - runtime: - max_model_len: 16384 - gpu_memory_utilization: 0.92 - max_num_seqs: 2 - max_num_batched_tokens: 4096 - enable_prefix_caching: true - enable_chunked_prefill: true - reasoning_history_policy: final_only - router: - aliases: - gemma4-e4b: gemma-main - policy: - require_fit_validation: true - minimum_vram_headroom_gib: 2 - allow_unsupported_render: false - qwen3.5-0.8b-4x96: - description: Tiny Qwen3.5 profile for utility tasks and high concurrency on a - 4x96GB host. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: qwen-main - model: qwen3.5-0.8b - served_model_name: qwen3.5-0.8b - placement: - strategy: multi_gpu - gpu_indices: - - 0 - - 1 - - 2 - - 3 - prefer_homogeneous_group: true - topology: - tp: 1 - dp: 4 - runtime: - max_model_len: 65536 - gpu_memory_utilization: 0.85 - max_num_seqs: 16 - max_num_batched_tokens: 8192 - enable_prefix_caching: true - enable_chunked_prefill: true - router: - aliases: - qwen3.5-0.8b: qwen-main - qwen3.5-2b-4x96: - description: Small Qwen3.5 profile for cheap agent loops and high concurrency - on a 4x96GB host. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: qwen-main - model: qwen3.5-2b - served_model_name: qwen3.5-2b - placement: - strategy: multi_gpu - gpu_indices: - - 0 - - 1 - - 2 - - 3 - prefer_homogeneous_group: true - topology: - tp: 1 - dp: 4 - runtime: - max_model_len: 65536 - gpu_memory_utilization: 0.88 - max_num_seqs: 12 - max_num_batched_tokens: 8192 - enable_prefix_caching: true - enable_chunked_prefill: true - router: - aliases: - qwen3.5-2b: qwen-main - qwen3.5-4b-4x96: - description: Small-mid Qwen3.5 profile for general use with high concurrency on - a 4x96GB host. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: qwen-main - model: qwen3.5-4b - served_model_name: qwen3.5-4b - placement: - strategy: multi_gpu - gpu_indices: - - 0 - - 1 - - 2 - - 3 - prefer_homogeneous_group: true - topology: - tp: 1 - dp: 4 - runtime: - max_model_len: 65536 - gpu_memory_utilization: 0.88 - max_num_seqs: 10 - max_num_batched_tokens: 8192 - enable_prefix_caching: true - enable_chunked_prefill: true - router: - aliases: - qwen3.5-4b: qwen-main - qwen3.5-9b-4x96: - description: General-purpose Qwen3.5 profile with 4-way data parallelism on a - 4x96GB host. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: qwen-main - model: qwen3.5-9b - served_model_name: qwen3.5-9b - placement: - strategy: multi_gpu - gpu_indices: - - 0 - - 1 - - 2 - - 3 - prefer_homogeneous_group: true - topology: - tp: 1 - dp: 4 - runtime: - max_model_len: 65536 - gpu_memory_utilization: 0.9 - max_num_seqs: 8 - max_num_batched_tokens: 8192 - enable_prefix_caching: true - enable_chunked_prefill: true - router: - aliases: - qwen3.5-9b: qwen-main - qwen2-72b-instruct-tp2-balanced: - description: Audit serving profile for qwen/qwen2-72b-instruct on tp=2. - base_model: qwen2-72b-instruct - public_name: qwen2-72b-instruct-tp2-balanced - logical_model_name: qwen/qwen2-72b-instruct - served_model_name: qwen/qwen2-72b-instruct - protocol_mode: chat - resource_profile: gpu-tp2-balanced - placement: - strategy: first_fit - gpu_count: 2 - topology: - tp: 2 - dp: 1 - tags: - - audit - - helm - - qwen - benchmark_transport: - kind: vllm-direct - deployment_name: vllm/qwen2-72b-instruct-local - request_model_name: Qwen/Qwen2-72B-Instruct - base_url: http://localhost:8000/v1 - audit_notes: - - Preserve the HELM logical model identity while serving locally through a named - profile. - vllm: - enable_responses_api_store: false - logging_level: INFO - qwen2-5-7b-instruct-turbo-default: - description: Audit serving profile for qwen/qwen2.5-7b-instruct-turbo. - base_model: qwen2.5-7b-instruct - public_name: qwen2-5-7b-instruct-turbo-default - logical_model_name: qwen/qwen2.5-7b-instruct-turbo - served_model_name: qwen/qwen2.5-7b-instruct-turbo - protocol_mode: chat - resource_profile: gpu-single-default - placement: - strategy: first_fit - gpu_count: 1 - tags: - - audit - - helm - - qwen - benchmark_transport: - kind: vllm-direct - deployment_name: vllm/qwen2-5-7b-instruct-turbo-local - request_model_name: Qwen/Qwen2.5-7B-Instruct - base_url: http://localhost:8000/v1 - vllm: - enable_responses_api_store: false - logging_level: INFO - qwen2-5-72b-instruct-tp2-balanced: - description: Audit serving profile for qwen/qwen2.5-72b-instruct-turbo on tp=2. - base_model: qwen2.5-72b-instruct - public_name: qwen2-5-72b-instruct-tp2-balanced - logical_model_name: qwen/qwen2.5-72b-instruct-turbo - served_model_name: qwen/qwen2.5-72b-instruct-turbo - protocol_mode: chat - resource_profile: gpu-tp2-balanced - placement: - strategy: first_fit - gpu_count: 2 - topology: - tp: 2 - dp: 1 - tags: - - audit - - helm - - qwen - benchmark_transport: - kind: vllm-direct - deployment_name: vllm/qwen2-5-72b-instruct-turbo-local - request_model_name: Qwen/Qwen2.5-72B-Instruct - base_url: http://localhost:8000/v1 - vllm: - enable_responses_api_store: false - logging_level: INFO - qwen3-5-122b-a10b-tp2-balanced: - description: Audit serving profile for Qwen/Qwen3.5-122B-A10B on tp=2. - base_model: qwen3.5-122b-a10b - public_name: qwen3-5-122b-a10b-tp2-balanced - logical_model_name: qwen/qwen3.5-122b-a10b - served_model_name: qwen/qwen3.5-122b-a10b - protocol_mode: chat - resource_profile: gpu-tp2-balanced - placement: - strategy: first_fit - gpu_count: 2 - topology: - tp: 2 - dp: 1 - tags: - - audit - - helm - - qwen - vllm: - enable_responses_api_store: false - logging_level: INFO - qwen3-5-122b-a10b-maxctx: - description: Audit serving profile for Qwen/Qwen3.5-122B-A10B with larger context - and lower concurrency. - base_model: qwen3.5-122b-a10b - public_name: qwen3-5-122b-a10b-maxctx - logical_model_name: qwen/qwen3.5-122b-a10b - served_model_name: qwen/qwen3.5-122b-a10b - protocol_mode: chat - resource_profile: gpu-tp2-maxctx - placement: - strategy: first_fit - gpu_count: 2 - topology: - tp: 2 - dp: 1 - runtime: - max_model_len: 65536 - max_num_seqs: 2 - max_num_batched_tokens: 4096 - gpu_memory_utilization: 0.95 - tags: - - audit - - helm - - qwen - - max-context - audit_notes: - - This profile intentionally trades concurrency for context length. - vllm: - enable_responses_api_store: false - logging_level: INFO - gpt-oss-20b-chat: - description: Audit serving profile for openai/gpt-oss-20b via chat-completions - semantics. - base_model: gpt-oss-20b - public_name: gpt-oss-20b-chat - logical_model_name: openai/gpt-oss-20b - served_model_name: openai/gpt-oss-20b - protocol_mode: chat - resource_profile: gpu-single-default - placement: - strategy: first_fit - gpu_count: 1 - tags: - - audit - - helm - - gpt-oss - - chat - benchmark_transport: - kind: openai-compatible - deployment_name: litellm/gpt-oss-20b-chat-local - request_model_name: openai/gpt-oss-20b - api_key_env: LITELLM_MASTER_KEY - api_key_placeholder: SET_LITELLM_MASTER_KEY_IN_RUNTIME_BUNDLE - base_url: http://localhost:14042/v1 - audit_notes: - - Exported HELM bundles should use a chat-capable client class for this profile. - vllm: - enable_responses_api_store: false - logging_level: INFO - gpt-oss-20b-completions: - description: Audit serving profile for openai/gpt-oss-20b via legacy completions - semantics. - base_model: gpt-oss-20b - public_name: gpt-oss-20b-completions - logical_model_name: openai/gpt-oss-20b - served_model_name: openai/gpt-oss-20b - protocol_mode: completions - resource_profile: gpu-single-default - placement: - strategy: first_fit - gpu_count: 1 - tags: - - audit - - helm - - gpt-oss - - completions - benchmark_transport: - kind: openai-compatible - deployment_name: litellm/gpt-oss-20b-local - request_model_name: openai/gpt-oss-20b - api_key_env: LITELLM_MASTER_KEY - api_key_placeholder: SET_LITELLM_MASTER_KEY_IN_RUNTIME_BUNDLE - base_url: http://localhost:14042/v1 - audit_notes: - - Exported HELM bundles should use a completions-capable client class for this - profile. - vllm: - enable_responses_api_store: false - logging_level: INFO - gpt-oss-20b-completions-dp4: - description: "4-replica data-parallel serving of openai/gpt-oss-20b via completions\ - \ protocol on a 4\xD796GB host. One replica per GPU, all sharing a single LiteLLM\ - \ endpoint." - protocol_mode: completions - benchmark_transport: - kind: openai-compatible - deployment_name: litellm/gpt-oss-20b-local - request_model_name: openai/gpt-oss-20b - api_key_env: LITELLM_MASTER_KEY - api_key_placeholder: SET_LITELLM_MASTER_KEY_IN_RUNTIME_BUNDLE - base_url: http://localhost:14042/v1 - audit_notes: - - Exported HELM bundles should use a completions-capable client class for this - profile. - - Supports 4 parallel HELM workers; vLLM dp=4 handles request distribution across - GPUs. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: gpt-oss-20b-completions - model: gpt-oss-20b - served_model_name: openai/gpt-oss-20b - placement: - strategy: multi_gpu - gpu_indices: - - 0 - - 1 - - 2 - - 3 - prefer_homogeneous_group: true - topology: - tp: 1 - dp: 4 - runtime: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - max_num_seqs: 16 - max_num_batched_tokens: 8192 - enable_prefix_caching: true - router: - aliases: - openai/gpt-oss-20b: gpt-oss-20b-completions - vicuna-7b-v1-3-no-chat-template: - description: Audit serving profile for legacy Vicuna prompting without relying - on a chat template. - base_model: vicuna-7b-v1.3 - public_name: vicuna-7b-v1-3-no-chat-template - logical_model_name: lmsys/vicuna-7b-v1.3 - served_model_name: lmsys/vicuna-7b-v1.3 - protocol_mode: completions - resource_profile: gpu-single-default - placement: - strategy: first_fit - gpu_count: 1 - extra_args: - - --disable-frontend-multiprocessing - tags: - - audit - - helm - - vicuna - - completions - audit_notes: - - Keep this profile separate from chat-oriented transports because prompt formatting - matters scientifically. - vllm: - enable_responses_api_store: false - logging_level: INFO - pythia-6-9b-default: - description: Audit serving profile for EleutherAI/pythia-6.9b. - base_model: pythia-6.9b - public_name: pythia-6-9b-default - logical_model_name: eleutherai/pythia-6.9b - served_model_name: eleutherai/pythia-6.9b - protocol_mode: completions - resource_profile: gpu-single-default - placement: - strategy: first_fit - gpu_count: 1 - tags: - - audit - - helm - - pythia - - completions - vllm: - enable_responses_api_store: false - logging_level: INFO - helm-qwen2-72b-instruct: - description: 'HELM: qwen/qwen2-72b-instruct (Qwen/Qwen2-72B-Instruct, tp=2)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: qwen2-72b-instruct - placement: - strategy: first_fit - gpu_count: 2 - topology: - tp: 2 - dp: 1 - router: - aliases: - qwen/qwen2-72b-instruct: helm-main - helm-qwen2.5-7b-instruct: - description: 'HELM: qwen/qwen2.5-7b-instruct-turbo (Qwen/Qwen2.5-7B-Instruct)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: qwen2.5-7b-instruct - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - qwen/qwen2.5-7b-instruct-turbo: helm-main - helm-qwen2.5-72b-instruct: - description: 'HELM: qwen/qwen2.5-72b-instruct-turbo (Qwen/Qwen2.5-72B-Instruct, - tp=2)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: qwen2.5-72b-instruct - placement: - strategy: first_fit - gpu_count: 2 - topology: - tp: 2 - dp: 1 - router: - aliases: - qwen/qwen2.5-72b-instruct-turbo: helm-main - helm-qwen3.5-9b: - description: 'HELM: qwen/qwen3.5-9b (Qwen/Qwen3.5-9B)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: qwen3.5-9b - served_model_name: qwen/qwen3.5-9b - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - qwen/qwen3.5-9b: helm-main - helm-gpt-oss-20b: - description: 'HELM: openai/gpt-oss-20b' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: gpt-oss-20b - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - openai/gpt-oss-20b: helm-main - helm-llama-3.1-8b-instruct: - description: 'HELM: meta/llama-3.1-8b-instruct-turbo (meta-llama/Meta-Llama-3.1-8B-Instruct)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: llama-3.1-8b-instruct - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - meta/llama-3.1-8b-instruct-turbo: helm-main - helm-llama-3-8b-chat: - description: 'HELM: meta/llama-3-8b-chat (meta-llama/Meta-Llama-3-8B-Instruct)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: llama-3-8b-instruct - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - meta/llama-3-8b-chat: helm-main - helm-llama-2-7b: - description: 'HELM: meta/llama-2-7b (meta-llama/Llama-2-7b-hf)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: llama-2-7b - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - meta/llama-2-7b: helm-main - helm-llama-2-13b: - description: 'HELM: meta/llama-2-13b (meta-llama/Llama-2-13b-hf)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: llama-2-13b - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - meta/llama-2-13b: helm-main - helm-llama-2-70b: - description: 'HELM: meta/llama-2-70b (meta-llama/Llama-2-70b-hf, tp=2)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: llama-2-70b - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 2 - topology: - tp: 2 - dp: 1 - router: - aliases: - meta/llama-2-70b: helm-main - helm-mistral-7b-instruct-v0.3: - description: 'HELM: mistralai/mistral-7b-instruct-v0.3 (mistralai/Mistral-7B-Instruct-v0.3)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: mistral-7b-instruct-v0.3 - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - mistralai/mistral-7b-instruct-v0.3: helm-main - helm-mistral-7b-v0.1: - description: 'HELM: mistralai/mistral-7b-v0.1 (mistralai/Mistral-7B-v0.1)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: mistral-7b-v0.1 - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - mistralai/mistral-7b-v0.1: helm-main - helm-falcon-7b: - description: 'HELM: tiiuae/falcon-7b' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: falcon-7b - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - tiiuae/falcon-7b: helm-main - helm-falcon-7b-instruct: - description: 'HELM: tiiuae/falcon-7b-instruct' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: falcon-7b-instruct - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - tiiuae/falcon-7b-instruct: helm-main - helm-pythia-6.9b: - description: 'HELM: eleutherai/pythia-6.9b (EleutherAI/pythia-6.9b)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: pythia-6.9b - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - eleutherai/pythia-6.9b: helm-main - helm-pythia-2.8b-v0: - description: 'HELM: eleutherai/pythia-2.8b-v0 (EleutherAI/pythia-2.8b-v0)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: pythia-2.8b-v0 - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - eleutherai/pythia-2.8b-v0: helm-main - helm-pythia-1b-v0: - description: 'HELM: eleutherai/pythia-1b-v0 (EleutherAI/pythia-1b-v0)' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: pythia-1b-v0 - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - eleutherai/pythia-1b-v0: helm-main - pythia-inspect-mmlu-compat: - description: Pythia 6.9B and 2.8B as completions models with a LiteLLM-only chat - compatibility adapter for chat-shaped clients (e.g. InspectAI MMLU). vLLM still - serves /v1/completions; LiteLLM flattens chat messages into a plain prompt. - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: pythia-69b - model: pythia-6.9b - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - runtime: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 2048 - max_num_seqs: 16 - enable_prefix_caching: false - chat_compat: - enabled: true - strategy: flat_messages - - service_name: pythia-28b - model: pythia-2.8b-v0 - protocol_mode: completions - placement: - strategy: first_fit - gpu_count: 1 - runtime: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - max_num_batched_tokens: 2048 - max_num_seqs: 16 - enable_prefix_caching: false - chat_compat: - enabled: true - strategy: flat_messages - router: - aliases: - eleutherai/pythia-6.9b: pythia-69b - eleutherai/pythia-2.8b-v0: pythia-28b - helm-vicuna-7b-v1.3: - description: 'HELM: lmsys/vicuna-7b-v1.3' - vllm: - enable_responses_api_store: false - logging_level: INFO - services: - - service_name: helm-main - model: vicuna-7b-v1.3 - placement: - strategy: first_fit - gpu_count: 1 - router: - aliases: - lmsys/vicuna-7b-v1.3: helm-main - qwen3.5-9b-vllm-gpu1-3090: - description: Qwen3.5 9B through vLLM on physical GPU 1, tuned for a 24 GB RTX - 3090 with a shorter context than the catalog default. - providers: - vllm: - runtimes: - chat: - model: qwen3.5-9b - served_model_name: qwen3.5-9b - logical_model_name: qwen3.5-9b - placement: - strategy: exact - gpu_indices: - - 1 - runtime: - max_model_len: 8192 - gpu_memory_utilization: 0.85 - max_num_batched_tokens: 4096 - max_num_seqs: 4 - enable_prefix_caching: true - ollama: - enabled: false - gateways: - litellm: - enabled: true - frontends: - open_webui: - enabled: true - provider: litellm - routes: - qwen3.5-9b: - provider: vllm - runtime: chat - qwen3.6-35b-a3b-vllm-gpu1-3090: - description: Experimental Qwen3.6 35B-A3B FP8 through vLLM on physical GPU 1. - This is a best-effort 3090 profile with short context; official FP8 guidance - targets data-center GPUs, so this may fail on 24 GB / non-FP8 hardware. - providers: - vllm: - runtimes: - chat: - model: qwen3.6-35b-a3b-fp8 - served_model_name: qwen3.6-35b-a3b - logical_model_name: qwen3.6-35b-a3b - placement: - strategy: exact - gpu_indices: - - 1 - runtime: - max_model_len: 4096 - gpu_memory_utilization: 0.92 - max_num_batched_tokens: 2048 - max_num_seqs: 1 - enable_prefix_caching: false - extra_args: - - --language-model-only - ollama: - enabled: false - gateways: - litellm: - enabled: true - frontends: - open_webui: - enabled: true - provider: litellm - routes: - qwen3.6-35b-a3b: - provider: vllm - runtime: chat - smollm2-135m-vllm-gpu1: - description: SmolLM2 135M Instruct through vLLM on physical GPU 1, routed through - LiteLLM and Open WebUI. - providers: - vllm: - runtimes: - chat: - model: smollm2-135m-instruct - served_model_name: smollm2-135m - logical_model_name: smollm2-135m - placement: - strategy: exact - gpu_indices: - - 1 - runtime: - max_model_len: 4096 - gpu_memory_utilization: 0.5 - max_num_batched_tokens: 2048 - max_num_seqs: 16 - enable_prefix_caching: true - ollama: - enabled: false - gateways: - litellm: - enabled: true - frontends: - open_webui: - enabled: true - provider: litellm - routes: - smollm2-135m: - provider: vllm - runtime: chat - smollm2-135m-ollama-gpu1: - description: Direct Ollama on physical GPU 1 with Open WebUI connected directly. - Pull smollm2:135m into Ollama before chat/smoke tests. - providers: - ollama: - enabled: true - gpu_indices: - - 1 - publish_port: true - keep_alive: 2m - context_length: 4096 - num_parallel: 1 - max_loaded_models: 1 - max_queue: 8 - vllm: - enabled: false - runtimes: {} - gateways: - litellm: - enabled: false - frontends: - open_webui: - enabled: true - provider: ollama - routes: {} diff --git a/infer_stack/templates/default-vllm-models.yaml b/infer_stack/templates/default-vllm-models.yaml deleted file mode 100644 index eee094f6..00000000 --- a/infer_stack/templates/default-vllm-models.yaml +++ /dev/null @@ -1,714 +0,0 @@ -vllm_models: - qwen3.5-0.8b: - hf_model_id: Qwen/Qwen3.5-0.8B - served_model_name: qwen3.5-0.8b - family: qwen3.5 - modalities: - - text - memory_class_gib: 4 - min_vram_gib_per_replica: 4 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 32 - qwen3.5-2b: - hf_model_id: Qwen/Qwen3.5-2B - served_model_name: qwen3.5-2b - family: qwen3.5 - modalities: - - text - memory_class_gib: 8 - min_vram_gib_per_replica: 8 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 32 - qwen3.5-4b: - hf_model_id: Qwen/Qwen3.5-4B - served_model_name: qwen3.5-4b - family: qwen3.5 - modalities: - - text - memory_class_gib: 12 - min_vram_gib_per_replica: 12 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 24 - qwen3.5-9b: - hf_model_id: Qwen/Qwen3.5-9B - served_model_name: qwen3.5-9b - family: qwen3.5 - modalities: - - text - memory_class_gib: 20 - min_vram_gib_per_replica: 20 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - qwen3.5-27b: - hf_model_id: Qwen/Qwen3.5-27B - served_model_name: qwen3.5-27b - family: qwen3.5 - modalities: - - text - memory_class_gib: 48 - min_vram_gib_per_replica: 48 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - qwen3.5-35b-a3b: - hf_model_id: Qwen/Qwen3.5-35B-A3B - served_model_name: qwen3.5-35b-a3b - family: qwen3.5 - modalities: - - text - memory_class_gib: 48 - min_vram_gib_per_replica: 48 - preferred_gpu_count: 1 - context_window: 32768 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - qwen3.5-122b-a10b: - hf_model_id: Qwen/Qwen3.5-122B-A10B - served_model_name: qwen3.5-122b-a10b - family: qwen3.5 - modalities: - - text - memory_class_gib: 80 - min_vram_gib_per_replica: 80 - preferred_gpu_count: 2 - context_window: 131072 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 8 - qwen3.5-122b-a10b-fp8: - hf_model_id: Qwen/Qwen3.5-122B-A10B-FP8 - served_model_name: qwen3.5-122b-a10b-fp8 - family: qwen3.5 - modalities: - - text - memory_class_gib: 80 - min_vram_gib_per_replica: 80 - preferred_gpu_count: 2 - context_window: 262144 - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 8 - tool_calling: - auto: true - parser: qwen3_xml - qwen3.6-35b-a3b: - hf_model_id: Qwen/Qwen3.6-35B-A3B - served_model_name: qwen3.6-35b-a3b - tokenizer_name: Qwen/Qwen3.6-35B-A3B - family: qwen3.6 - modalities: - - text - memory_class_gib: 80 - min_vram_gib_per_replica: 24 - preferred_gpu_count: 2 - context_window: 262144 - supported_protocols: - - chat - - completions - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 262144 - gpu_memory_utilization: 0.95 - enable_prefix_caching: false - max_num_batched_tokens: 8192 - max_num_seqs: 8 - qwen3.6-35b-a3b-fp8: - hf_model_id: Qwen/Qwen3.6-35B-A3B-FP8 - served_model_name: qwen3.6-35b-a3b - tokenizer_name: Qwen/Qwen3.6-35B-A3B - family: qwen3.6 - modalities: - - text - memory_class_gib: 40 - min_vram_gib_per_replica: 24 - preferred_gpu_count: 1 - context_window: 262144 - supported_protocols: - - chat - - completions - reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true - defaults: - max_model_len: 8192 - gpu_memory_utilization: 0.92 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 2 - tool_calling: - auto: true - parser: qwen3_coder - gemma4-e2b: - hf_model_id: google/gemma-4-E2B-it - served_model_name: gemma4-e2b - family: gemma4 - modalities: - - text - - image - - audio - memory_class_gib: 8 - min_vram_gib_per_replica: 8 - preferred_gpu_count: 1 - context_window: 131072 - thinking_history_policy: keep_final_only - notes: - - Historical turns should keep only final responses, not prior thinking content. - defaults: - max_model_len: 65536 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 16 - tool_calling: - auto: true - parser: gemma4 - gemma4-e4b: - hf_model_id: google/gemma-4-E4B-it - served_model_name: gemma4-e4b - family: gemma4 - modalities: - - text - - image - - audio - memory_class_gib: 10 - min_vram_gib_per_replica: 10 - preferred_gpu_count: 1 - context_window: 131072 - thinking_history_policy: keep_final_only - notes: - - Historical turns should keep only final responses, not prior thinking content. - defaults: - max_model_len: 65536 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 16 - tool_calling: - auto: true - parser: gemma4 - gemma4-26b: - hf_model_id: google/gemma-4-26B-A4B-it - served_model_name: gemma4-26b - family: gemma4 - modalities: - - text - - image - memory_class_gib: 20 - min_vram_gib_per_replica: 20 - preferred_gpu_count: 1 - context_window: 262144 - thinking_history_policy: keep_final_only - notes: - - Historical turns should keep only final responses, not prior thinking content. - defaults: - max_model_len: 65536 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 12 - tool_calling: - auto: true - parser: gemma4 - gemma4-31b: - hf_model_id: google/gemma-4-31B-it - served_model_name: gemma4-31b - family: gemma4 - modalities: - - text - - image - memory_class_gib: 22 - min_vram_gib_per_replica: 22 - preferred_gpu_count: 1 - context_window: 262144 - thinking_history_policy: keep_final_only - notes: - - Historical turns should keep only final responses, not prior thinking content. - defaults: - max_model_len: 65536 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 12 - tool_calling: - auto: true - parser: gemma4 - qwen2-72b-instruct: - hf_model_id: Qwen/Qwen2-72B-Instruct - served_model_name: qwen/qwen2-72b-instruct - logical_model_name: qwen/qwen2-72b-instruct - tokenizer_name: qwen/qwen2-72b-instruct - family: qwen2 - modalities: - - text - memory_class_gib: 144 - min_vram_gib_per_replica: 144 - preferred_gpu_count: 2 - context_window: 131072 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 8 - qwen2.5-0.5b-instruct: - hf_model_id: Qwen/Qwen2.5-0.5B-Instruct - served_model_name: qwen/qwen2.5-0.5b-instruct - logical_model_name: qwen/qwen2.5-0.5b-instruct - tokenizer_name: Qwen/Qwen2.5-0.5B-Instruct - family: qwen2.5 - modalities: - - text - memory_class_gib: 2 - min_vram_gib_per_replica: 2 - preferred_gpu_count: 1 - context_window: 32768 - defaults: - max_model_len: 8192 - gpu_memory_utilization: 0.85 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 16 - qwen2.5-7b-instruct: - hf_model_id: Qwen/Qwen2.5-7B-Instruct - served_model_name: qwen/qwen2.5-7b-instruct-turbo - logical_model_name: qwen/qwen2.5-7b-instruct-turbo - tokenizer_name: qwen/qwen2.5-7b-instruct-turbo - family: qwen2.5 - modalities: - - text - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 131072 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - qwen2.5-72b-instruct: - hf_model_id: Qwen/Qwen2.5-72B-Instruct - served_model_name: qwen/qwen2.5-72b-instruct-turbo - logical_model_name: qwen/qwen2.5-72b-instruct-turbo - tokenizer_name: qwen/qwen2.5-72b-instruct-turbo - family: qwen2.5 - modalities: - - text - memory_class_gib: 144 - min_vram_gib_per_replica: 144 - preferred_gpu_count: 2 - context_window: 131072 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 8 - gpt-oss-20b: - hf_model_id: openai/gpt-oss-20b - served_model_name: openai/gpt-oss-20b - logical_model_name: openai/gpt-oss-20b - tokenizer_name: openai/o200k_harmony - family: gpt-oss - modalities: - - text - memory_class_gib: 40 - min_vram_gib_per_replica: 40 - preferred_gpu_count: 1 - context_window: 131072 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - llama-3.1-8b-instruct: - hf_model_id: meta-llama/Meta-Llama-3.1-8B-Instruct - served_model_name: meta/llama-3.1-8b-instruct-turbo - family: llama3.1 - modalities: - - text - memory_class_gib: 18 - min_vram_gib_per_replica: 18 - preferred_gpu_count: 1 - context_window: 131072 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - llama-3-8b-instruct: - hf_model_id: meta-llama/Meta-Llama-3-8B-Instruct - served_model_name: meta/llama-3-8b-chat - family: llama3 - modalities: - - text - memory_class_gib: 18 - min_vram_gib_per_replica: 18 - preferred_gpu_count: 1 - context_window: 8192 - defaults: - max_model_len: 8192 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - llama-2-7b: - hf_model_id: meta-llama/Llama-2-7b-hf - served_model_name: meta/llama-2-7b - family: llama2 - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 4096 - defaults: - max_model_len: 4096 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 16 - llama-2-13b: - hf_model_id: meta-llama/Llama-2-13b-hf - served_model_name: meta/llama-2-13b - family: llama2 - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 28 - min_vram_gib_per_replica: 28 - preferred_gpu_count: 1 - context_window: 4096 - defaults: - max_model_len: 4096 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 16 - llama-2-70b: - hf_model_id: meta-llama/Llama-2-70b-hf - served_model_name: meta/llama-2-70b - family: llama2 - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 140 - min_vram_gib_per_replica: 70 - preferred_gpu_count: 2 - context_window: 4096 - defaults: - max_model_len: 4096 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 4096 - max_num_seqs: 8 - mistral-7b-instruct-v0.3: - hf_model_id: mistralai/Mistral-7B-Instruct-v0.3 - served_model_name: mistralai/mistral-7b-instruct-v0.3 - family: mistral - modalities: - - text - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 32768 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - mistral-7b-v0.1: - hf_model_id: mistralai/Mistral-7B-v0.1 - served_model_name: mistralai/mistral-7b-v0.1 - family: mistral - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 32768 - defaults: - max_model_len: 32768 - gpu_memory_utilization: 0.9 - enable_prefix_caching: true - max_num_batched_tokens: 8192 - max_num_seqs: 16 - falcon-7b: - hf_model_id: tiiuae/falcon-7b - served_model_name: tiiuae/falcon-7b - family: falcon - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - falcon-7b-instruct: - hf_model_id: tiiuae/falcon-7b-instruct - served_model_name: tiiuae/falcon-7b-instruct - family: falcon - modalities: - - text - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - gpt2: - hf_model_id: openai-community/gpt2 - served_model_name: gpt2 - logical_model_name: gpt2 - tokenizer_name: openai-community/gpt2 - family: gpt2 - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 1 - min_vram_gib_per_replica: 1 - preferred_gpu_count: 1 - context_window: 1024 - defaults: - max_model_len: 1024 - gpu_memory_utilization: 0.3 - enable_prefix_caching: true - max_num_batched_tokens: 1024 - max_num_seqs: 16 - smollm2-135m-instruct: - hf_model_id: HuggingFaceTB/SmolLM2-135M-Instruct - served_model_name: huggingfacetb/smollm2-135m-instruct - logical_model_name: huggingfacetb/smollm2-135m-instruct - tokenizer_name: HuggingFaceTB/SmolLM2-135M-Instruct - family: smollm2 - modalities: - - text - memory_class_gib: 1 - min_vram_gib_per_replica: 1 - preferred_gpu_count: 1 - context_window: 8192 - defaults: - max_model_len: 4096 - gpu_memory_utilization: 0.6 - enable_prefix_caching: true - max_num_batched_tokens: 2048 - max_num_seqs: 16 - pythia-160m: - hf_model_id: EleutherAI/pythia-160m - served_model_name: eleutherai/pythia-160m - logical_model_name: eleutherai/pythia-160m - tokenizer_name: EleutherAI/pythia-160m - family: pythia - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 1 - min_vram_gib_per_replica: 1 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.6 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - pythia-410m: - hf_model_id: EleutherAI/pythia-410m - served_model_name: eleutherai/pythia-410m - logical_model_name: eleutherai/pythia-410m - tokenizer_name: EleutherAI/pythia-410m - family: pythia - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 2 - min_vram_gib_per_replica: 2 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.7 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - pythia-6.9b: - hf_model_id: EleutherAI/pythia-6.9b - served_model_name: eleutherai/pythia-6.9b - logical_model_name: eleutherai/pythia-6.9b - tokenizer_name: eleutherai/pythia-6.9b - family: pythia - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - pythia-2.8b-v0: - hf_model_id: EleutherAI/pythia-2.8b-v0 - served_model_name: eleutherai/pythia-2.8b-v0 - logical_model_name: eleutherai/pythia-2.8b-v0 - tokenizer_name: eleutherai/pythia-2.8b-v0 - family: pythia - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 8 - min_vram_gib_per_replica: 8 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - pythia-1b-v0: - hf_model_id: EleutherAI/pythia-1b-v0 - served_model_name: eleutherai/pythia-1b-v0 - logical_model_name: eleutherai/pythia-1b-v0 - tokenizer_name: eleutherai/pythia-1b-v0 - family: pythia - modalities: - - text - supported_protocols: - - completions - memory_class_gib: 4 - min_vram_gib_per_replica: 4 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 - vicuna-7b-v1.3: - hf_model_id: lmsys/vicuna-7b-v1.3 - served_model_name: lmsys/vicuna-7b-v1.3 - logical_model_name: lmsys/vicuna-7b-v1.3 - tokenizer_name: lmsys/vicuna-7b-v1.3 - family: vicuna - supported_protocols: - - completions - notes: - - Use a no-chat-template style profile when reproducing legacy completion-style - prompting. - modalities: - - text - memory_class_gib: 16 - min_vram_gib_per_replica: 16 - preferred_gpu_count: 1 - context_window: 2048 - defaults: - max_model_len: 2048 - gpu_memory_utilization: 0.9 - enable_prefix_caching: false - max_num_batched_tokens: 2048 - max_num_seqs: 16 diff --git a/infer_stack/templates/docker-compose.yml.j2 b/infer_stack/templates/docker-compose.yml.j2 deleted file mode 100644 index 31511637..00000000 --- a/infer_stack/templates/docker-compose.yml.j2 +++ /dev/null @@ -1,430 +0,0 @@ -services: -{% if lock.deployment.frontends.open_webui.enabled %} - postgres-open-webui: - image: {{ lock.deployment.images.postgres }} - container_name: postgres-open-webui - restart: unless-stopped - environment: - POSTGRES_DB: ${OPENWEBUI_POSTGRES_DB} - POSTGRES_USER: ${OPENWEBUI_POSTGRES_USER} - POSTGRES_PASSWORD: ${OPENWEBUI_POSTGRES_PASSWORD} - volumes: - - {{ lock.deployment.state.postgres_open_webui }}:/var/lib/postgresql/data - healthcheck: - test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"] - interval: 10s - timeout: 5s - retries: 10 - -{% endif %} -{% if lock.deployment.gateways.litellm.enabled %} - postgres-litellm: - image: {{ lock.deployment.images.postgres }} - container_name: postgres-litellm - restart: unless-stopped - environment: - POSTGRES_DB: ${LITELLM_POSTGRES_DB} - POSTGRES_USER: ${LITELLM_POSTGRES_USER} - POSTGRES_PASSWORD: ${LITELLM_POSTGRES_PASSWORD} - volumes: - - {{ lock.deployment.state.postgres_litellm }}:/var/lib/postgresql/data - healthcheck: - test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"] - interval: 10s - timeout: 5s - retries: 10 - -{% endif %} -{% if lock.deployment.providers.ollama.enabled %} - ollama: - image: {{ lock.deployment.images.ollama }} - container_name: ollama - restart: unless-stopped -{% if lock.deployment.providers.ollama.labels %} - labels: -{% for key, value in lock.deployment.providers.ollama.labels.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} -{% endif %} -{% if lock.deployment.providers.ollama.env_file %} - env_file: -{% for env_file in lock.deployment.providers.ollama.env_file %} - - {{ env_file }} -{% endfor %} -{% endif %} -{% if lock.deployment.providers.ollama.extra_hosts %} - extra_hosts: -{% for extra_host in lock.deployment.providers.ollama.extra_hosts %} - - {{ extra_host|compose_quote }} -{% endfor %} -{% endif %} - environment: - NVIDIA_DRIVER_CAPABILITIES: compute,utility - CUDA_VISIBLE_DEVICES: {{ (lock.deployment.providers.ollama.gpu_indices | join(','))|compose_quote }} - OLLAMA_HOST: {{ lock.deployment.providers.ollama.host|compose_quote }} - OLLAMA_KEEP_ALIVE: {{ lock.deployment.providers.ollama.keep_alive|compose_quote }} - OLLAMA_CONTEXT_LENGTH: {{ lock.deployment.providers.ollama.context_length|compose_quote }} - OLLAMA_NUM_PARALLEL: {{ lock.deployment.providers.ollama.num_parallel|compose_quote }} - OLLAMA_MAX_LOADED_MODELS: {{ lock.deployment.providers.ollama.max_loaded_models|compose_quote }} - OLLAMA_MAX_QUEUE: {{ lock.deployment.providers.ollama.max_queue|compose_quote }} -{% for key, value in lock.deployment.providers.ollama.extra_env.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} - volumes: - - {{ lock.deployment.state.ollama }}:/root/.ollama -{% for volume in lock.deployment.providers.ollama.extra_volumes %} - - {{ volume }} -{% endfor %} -{% if lock.deployment.providers.ollama.publish_port or lock.deployment.providers.ollama.additional_ports %} - ports: -{% if lock.deployment.providers.ollama.publish_port %} - - "127.0.0.1:{{ lock.deployment.ports.ollama or 11434 }}:11434" -{% endif %} -{% for port in lock.deployment.providers.ollama.additional_ports %} - - {{ port|compose_quote }} -{% endfor %} -{% endif %} - healthcheck: - test: ["CMD-SHELL", "ollama list >/dev/null 2>&1 || exit 1"] - interval: 30s - timeout: 10s - retries: 30 - start_period: 60s -{% if lock.deployment.providers.ollama.gpus is not none %} -{{ lock.deployment.providers.ollama.gpus|compose_gpus }} -{% elif lock.deployment.providers.ollama.gpu_indices %} - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: [{% for idx in lock.deployment.providers.ollama.gpu_indices %}"{{ idx }}"{% if not loop.last %}, {% endif %}{% endfor %}] - capabilities: [gpu] -{% endif %} - -{% endif %} -{% for name, svc in lock.deployment.providers.vllm.runtimes.items() %} - {{ svc.compose_service_name }}: - image: {{ lock.deployment.images.vllm }} - container_name: {{ svc.container_name or svc.compose_service_name }} - restart: unless-stopped - ipc: host - labels: - infer_stack.profile_name: {{ svc.profile_name|compose_quote }} - infer_stack.public_name: {{ svc.profile_public_name|compose_quote }} - infer_stack.logical_model_name: {{ svc.logical_model_name|compose_quote }} - infer_stack.protocol_mode: {{ svc.protocol_mode|compose_quote }} -{% for key, value in svc.labels.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} -{% if svc.env_file %} - env_file: -{% for env_file in svc.env_file %} - - {{ env_file }} -{% endfor %} -{% endif %} -{% if svc.extra_hosts %} - extra_hosts: -{% for extra_host in svc.extra_hosts %} - - {{ extra_host|compose_quote }} -{% endfor %} -{% endif %} - environment: - NVIDIA_DRIVER_CAPABILITIES: compute,utility - HF_TOKEN: ${HF_TOKEN} - VLLM_API_KEY: ${VLLM_BACKEND_API_KEY} - HF_HOME: /root/.cache/huggingface - HF_HUB_CACHE: /root/.cache/huggingface/hub - HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface/hub - VLLM_CACHE_ROOT: /root/.cache/vllm - TORCH_HOME: /root/.cache/torch - TORCHINDUCTOR_CACHE_DIR: /root/.cache/torch/inductor - TRITON_CACHE_DIR: /root/.cache/triton - CUDA_CACHE_PATH: /root/.cache/nvidia/ComputeCache -{% if lock.deployment.vllm.enable_responses_api_store %} - VLLM_ENABLE_RESPONSES_API_STORE: "1" -{% endif %} - VLLM_LOGGING_LEVEL: {{ lock.deployment.vllm.logging_level|compose_quote }} -{% for key, value in svc.extra_env.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} - command: - - {{ svc.hf_model_id|compose_quote }} - - --host - - 0.0.0.0 - - --port - - "8000" - - --api-key - - ${VLLM_BACKEND_API_KEY} - - --served-model-name - - {{ svc.served_model_name|compose_quote }} - - --tensor-parallel-size - - {{ svc.tensor_parallel_size|compose_quote }} - - --data-parallel-size - - {{ svc.data_parallel_size|compose_quote }} - - --max-model-len - - {{ svc.max_model_len|compose_quote }} - - --gpu-memory-utilization - - {{ svc.gpu_memory_utilization|compose_quote }} - - --max-num-batched-tokens - - {{ svc.max_num_batched_tokens|compose_quote }} - - --max-num-seqs - - {{ svc.max_num_seqs|compose_quote }} -{% if svc.enable_prefix_caching %} - - --enable-prefix-caching -{% endif %} -{% if svc.enable_auto_tool_choice %} - - --enable-auto-tool-choice - - --tool-call-parser - - {{ svc.tool_call_parser|compose_quote }} -{% endif %} -{% if svc.reasoning_enabled and svc.reasoning_parser %} - - --reasoning-parser - - {{ svc.reasoning_parser|compose_quote }} -{% endif %} -{% for extra_arg in svc.extra_args %} - - {{ extra_arg|compose_quote }} -{% endfor %} - volumes: - - {{ lock.deployment.state.hf_cache }}:/root/.cache/huggingface - - {{ lock.deployment.state.vllm_cache }}:/root/.cache/vllm - - {{ lock.deployment.state.torch_cache }}:/root/.cache/torch - - {{ lock.deployment.state.triton_cache }}:/root/.cache/triton - - {{ lock.deployment.state.cuda_cache }}:/root/.cache/nvidia/ComputeCache -{% for volume in svc.extra_volumes %} - - {{ volume }} -{% endfor %} -{% if svc.publish_port or svc.additional_ports %} - ports: -{% if svc.publish_port %} - - "127.0.0.1:${INFER_STACK_VLLM_{{ loop.index0 }}_PORT:-{{ svc.host_port or (18000 + loop.index0) }}}:8000" -{% endif %} -{% for port in svc.additional_ports %} - - {{ port|compose_quote }} -{% endfor %} -{% endif %} - healthcheck: - test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:8000/health >/dev/null 2>&1 || exit 1"] - interval: 30s - timeout: 10s - retries: 30 - start_period: 1800s -{% if svc.gpus is not none %} -{{ svc.gpus|compose_gpus }} -{% else %} - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: [{% for idx in svc.gpu_indices %}"{{ idx }}"{% if not loop.last %}, {% endif %}{% endfor %}] - capabilities: [gpu] -{% endif %} - -{% endfor %} -{% if lock.deployment.gateways.litellm.enabled %} - litellm: - image: {{ lock.deployment.images.litellm }} - container_name: litellm - restart: unless-stopped -{% if lock.deployment.gateways.litellm.labels %} - labels: -{% for key, value in lock.deployment.gateways.litellm.labels.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} -{% endif %} - depends_on: - postgres-litellm: - condition: service_healthy - # Do not depend directly on provider health here. LiteLLM can load its - # route table before upstream providers are healthy, and avoiding provider - # dependency edges prevents Compose from restarting LiteLLM on every vLLM - # model swap. Smoke tests and clients should retry until the selected - # upstream model is healthy. -{% if lock.deployment.gateways.litellm.env_file %} - env_file: -{% for env_file in lock.deployment.gateways.litellm.env_file %} - - {{ env_file }} -{% endfor %} -{% endif %} -{% if lock.deployment.gateways.litellm.extra_hosts %} - extra_hosts: -{% for extra_host in lock.deployment.gateways.litellm.extra_hosts %} - - {{ extra_host|compose_quote }} -{% endfor %} -{% endif %} - environment: - LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY} - VLLM_BACKEND_API_KEY: ${VLLM_BACKEND_API_KEY} - DATABASE_URL: postgresql://${LITELLM_POSTGRES_USER}:${LITELLM_POSTGRES_PASSWORD}@postgres-litellm:5432/${LITELLM_POSTGRES_DB} -{% for key, value in lock.deployment.gateways.litellm.extra_env.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} - command: - - --config - - /app/config.yaml - - --port - - "4000" - - --num_workers - - "1" - volumes: - - {{ lock.deployment.state.runtime }}/litellm_config.yaml:/app/config.yaml:ro -{% for volume in lock.deployment.gateways.litellm.extra_volumes %} - - {{ volume }} -{% endfor %} -{% if lock.deployment.gateways.litellm.additional_ports or lock.deployment.ports.litellm %} - ports: - - "${INFER_STACK_LITELLM_PORT:-{{ lock.deployment.ports.litellm or 14042 }}}:4000" -{% for port in lock.deployment.gateways.litellm.additional_ports %} - - {{ port|compose_quote }} -{% endfor %} -{% endif %} -{% if lock.deployment.gateways.litellm.gpus is not none %} -{{ lock.deployment.gateways.litellm.gpus|compose_gpus }} -{% endif %} - -{% endif %} -{% if lock.deployment.frontends.open_webui.enabled %} - open-webui: - image: {{ lock.deployment.images.open_webui }} - container_name: open-webui - restart: unless-stopped -{% if lock.deployment.frontends.open_webui.labels %} - labels: -{% for key, value in lock.deployment.frontends.open_webui.labels.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} -{% endif %} - depends_on: - postgres-open-webui: - condition: service_healthy -{% if lock.deployment.frontends.open_webui.provider == 'ollama' %} - ollama: - condition: service_started -{% elif lock.deployment.frontends.open_webui.provider == 'litellm' %} - litellm: - condition: service_started -{% endif %} -{% if lock.deployment.frontends.open_webui.env_file %} - env_file: -{% for env_file in lock.deployment.frontends.open_webui.env_file %} - - {{ env_file }} -{% endfor %} -{% endif %} -{% if lock.deployment.frontends.open_webui.extra_hosts %} - extra_hosts: -{% for extra_host in lock.deployment.frontends.open_webui.extra_hosts %} - - {{ extra_host|compose_quote }} -{% endfor %} -{% endif %} - environment: - WEBUI_AUTH: {{ ('True' if lock.deployment.frontends.open_webui.auth else 'False')|compose_quote }} - ENABLE_PERSISTENT_CONFIG: "False" - DATABASE_URL: postgresql://${OPENWEBUI_POSTGRES_USER}:${OPENWEBUI_POSTGRES_PASSWORD}@postgres-open-webui:5432/${OPENWEBUI_POSTGRES_DB} -{% if lock.deployment.frontends.open_webui.webui_url %} - WEBUI_URL: {{ lock.deployment.frontends.open_webui.webui_url|compose_quote }} -{% endif %} -{% if lock.deployment.frontends.open_webui.cors_allow_origin %} - CORS_ALLOW_ORIGIN: {{ lock.deployment.frontends.open_webui.cors_allow_origin|compose_quote }} -{% endif %} -{% if lock.deployment.frontends.open_webui.provider == 'ollama' %} - ENABLE_OLLAMA_API: "True" - OLLAMA_BASE_URL: http://ollama:11434 -{% elif lock.deployment.frontends.open_webui.provider == 'litellm' %} - ENABLE_OLLAMA_API: "False" - OPENAI_API_BASE_URL: http://litellm:4000/v1 - OPENAI_API_KEY: ${LITELLM_MASTER_KEY} -{% endif %} - WEBUI_SECRET_KEY: ${WEBUI_SECRET_KEY} -{% if lock.deployment.frontends.open_webui.ldap.enabled %} -{% for key, value in lock.deployment.frontends.open_webui.ldap.env.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} -{% endif %} -{% for key, value in lock.deployment.frontends.open_webui.extra_env.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} - volumes: - - {{ lock.deployment.state.open_webui }}:/app/backend/data -{% for volume in lock.deployment.frontends.open_webui.extra_volumes %} - - {{ volume }} -{% endfor %} -{% if lock.deployment.frontends.open_webui.publish_port or lock.deployment.frontends.open_webui.additional_ports %} - ports: -{% if lock.deployment.frontends.open_webui.publish_port %} - - "${INFER_STACK_OPEN_WEBUI_PORT:-{{ lock.deployment.ports.open_webui or 13000 }}}:8080" -{% endif %} -{% for port in lock.deployment.frontends.open_webui.additional_ports %} - - {{ port|compose_quote }} -{% endfor %} -{% endif %} -{% if lock.deployment.frontends.open_webui.gpus is not none %} -{{ lock.deployment.frontends.open_webui.gpus|compose_gpus }} -{% endif %} - -{% endif %} -{% if lock.deployment.frontends.reverse_proxy.enabled %} - {{ lock.deployment.frontends.reverse_proxy.service_name }}: - image: {{ lock.deployment.frontends.reverse_proxy.image }} - container_name: {{ lock.deployment.frontends.reverse_proxy.container_name }} - restart: unless-stopped -{% if lock.deployment.frontends.reverse_proxy.labels %} - labels: -{% for key, value in lock.deployment.frontends.reverse_proxy.labels.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} -{% endif %} -{% if lock.deployment.frontends.reverse_proxy.depends_on %} - depends_on: -{% for service in lock.deployment.frontends.reverse_proxy.depends_on %} - - {{ service }} -{% endfor %} -{% endif %} -{% if lock.deployment.frontends.reverse_proxy.env_file %} - env_file: -{% for env_file in lock.deployment.frontends.reverse_proxy.env_file %} - - {{ env_file }} -{% endfor %} -{% endif %} -{% if lock.deployment.frontends.reverse_proxy.extra_env %} - environment: -{% for key, value in lock.deployment.frontends.reverse_proxy.extra_env.items() %} - {{ key }}: {{ value|compose_quote }} -{% endfor %} -{% endif %} -{% if lock.deployment.frontends.reverse_proxy.extra_hosts %} - extra_hosts: -{% for extra_host in lock.deployment.frontends.reverse_proxy.extra_hosts %} - - {{ extra_host|compose_quote }} -{% endfor %} -{% endif %} - volumes: - - {{ lock.deployment.frontends.reverse_proxy.nginx_config_path }}:/etc/nginx/conf.d/default.conf:ro -{% if lock.deployment.frontends.reverse_proxy.ssl.enabled and lock.deployment.frontends.reverse_proxy.ssl.certificate %} - - {{ lock.deployment.frontends.reverse_proxy.ssl.certificate }}:{{ lock.deployment.frontends.reverse_proxy.ssl.certificate_container_path }}:ro -{% endif %} -{% if lock.deployment.frontends.reverse_proxy.ssl.enabled and lock.deployment.frontends.reverse_proxy.ssl.certificate_key %} - - {{ lock.deployment.frontends.reverse_proxy.ssl.certificate_key }}:{{ lock.deployment.frontends.reverse_proxy.ssl.certificate_key_container_path }}:ro -{% endif %} -{% if lock.deployment.frontends.reverse_proxy.ssl.enabled and lock.deployment.frontends.reverse_proxy.ssl.dhparam %} - - {{ lock.deployment.frontends.reverse_proxy.ssl.dhparam }}:{{ lock.deployment.frontends.reverse_proxy.ssl.dhparam_container_path }}:ro -{% endif %} -{% for volume in lock.deployment.frontends.reverse_proxy.extra_volumes %} - - {{ volume }} -{% endfor %} -{% if lock.deployment.frontends.reverse_proxy.publish_http or lock.deployment.frontends.reverse_proxy.publish_https or lock.deployment.frontends.reverse_proxy.additional_ports %} - ports: -{% if lock.deployment.frontends.reverse_proxy.publish_http %} - - "{{ lock.deployment.frontends.reverse_proxy.http_bind_host }}{% if lock.deployment.frontends.reverse_proxy.http_bind_host %}:{% endif %}${INFER_STACK_REVERSE_PROXY_HTTP_PORT:-{{ lock.deployment.frontends.reverse_proxy.http_port }}}:80" -{% endif %} -{% if lock.deployment.frontends.reverse_proxy.publish_https %} - - "{{ lock.deployment.frontends.reverse_proxy.https_bind_host }}{% if lock.deployment.frontends.reverse_proxy.https_bind_host %}:{% endif %}${INFER_STACK_REVERSE_PROXY_HTTPS_PORT:-{{ lock.deployment.frontends.reverse_proxy.https_port }}}:443" -{% endif %} -{% for port in lock.deployment.frontends.reverse_proxy.additional_ports %} - - {{ port|compose_quote }} -{% endfor %} -{% endif %} -{% if lock.deployment.frontends.reverse_proxy.gpus is not none %} -{{ lock.deployment.frontends.reverse_proxy.gpus|compose_gpus }} -{% endif %} -{% endif %} diff --git a/infer_stack/templates/litellm_config.yaml.j2 b/infer_stack/templates/litellm_config.yaml.j2 deleted file mode 100644 index 289ba393..00000000 --- a/infer_stack/templates/litellm_config.yaml.j2 +++ /dev/null @@ -1,44 +0,0 @@ -general_settings: - master_key: os.environ/LITELLM_MASTER_KEY - store_model_in_db: True - -model_list: -{% for alias, route in lock.deployment.gateways.litellm.routes.items() %} -{% set max_model_len = route.max_model_len or 4096 %} -{% set max_output = max_model_len // 2 %} -{% set max_input = max_model_len - max_output %} - - model_name: {{ alias }} - litellm_params: -{% if route.provider == 'ollama' %} - model: ollama_chat/{{ route.upstream_model }} - api_base: http://ollama:11434 - drop_params: true -{% elif route.protocol_mode == 'completions' %} - model: text-completion-openai/{{ route.served_model_name }} - api_base: http://{{ route.upstream_service_name or route.service_name }}:8000/v1 - api_key: os.environ/VLLM_BACKEND_API_KEY -{% else %} - model: openai/{{ route.served_model_name }} - api_base: http://{{ route.upstream_service_name or route.service_name }}:8000/v1 - api_key: os.environ/VLLM_BACKEND_API_KEY - merge_reasoning_content_in_choices: true -{% endif %} -{% if route.chat_compat_enabled and route.chat_compat_strategy == 'flat_messages' %} - initial_prompt_value: "" - roles: - system: - pre_message: "" - post_message: "\n" - user: - pre_message: "" - post_message: "\n" - assistant: - pre_message: "" - post_message: "\n" - final_prompt_value: "" -{% endif %} - model_info: - max_tokens: {{ max_model_len }} - max_input_tokens: {{ max_input }} - max_output_tokens: {{ max_output }} -{% endfor %} diff --git a/infer_stack/templates/nginx.conf.j2 b/infer_stack/templates/nginx.conf.j2 deleted file mode 100644 index 8567e1eb..00000000 --- a/infer_stack/templates/nginx.conf.j2 +++ /dev/null @@ -1,84 +0,0 @@ -# Rendered by infer-stack. To take full manual control, set -# frontends.reverse_proxy.config_path to an existing nginx config file. -upstream infer_stack_target { - server {{ rp.target_service }}:{{ rp.target_port }}; -} - -{% macro proxy_location(rp) %} -location / { - proxy_pass {{ rp.target_scheme }}://infer_stack_target; - proxy_redirect off; - proxy_http_version 1.1; - proxy_cache_bypass $http_upgrade; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Forwarded-Host $server_name; - proxy_buffer_size {{ rp.proxy_buffer_size }}; - proxy_buffers {{ rp.proxy_buffers }}; - proxy_busy_buffers_size {{ rp.proxy_busy_buffers_size }}; - proxy_connect_timeout {{ rp.proxy_connect_timeout }}; - proxy_read_timeout {{ rp.proxy_read_timeout }}; - proxy_send_timeout {{ rp.proxy_send_timeout }}; -{% if rp.proxy_buffering is not none %} - proxy_buffering {{ 'on' if rp.proxy_buffering else 'off' }}; -{% endif %} -{% if rp.proxy_cache is not none %} - proxy_cache {{ rp.proxy_cache }}; -{% endif %} - client_max_body_size {{ rp.client_max_body_size }}; -} -{% endmacro %} - -server { - listen 80; - listen [::]:80; - server_name {{ rp.server_name }}; -{% if rp.ssl.enabled and rp.force_https %} - - return 301 https://$host$request_uri; -{% else %} - -{{ proxy_location(rp)|indent(4, true) }} -{% endif %} -} - -{% if rp.ssl.enabled %} -server { - listen 443 ssl; - listen [::]:443 ssl; - server_name {{ rp.server_name }}; - - ssl_certificate {{ rp.ssl.certificate_container_path }}; - ssl_certificate_key {{ rp.ssl.certificate_key_container_path }}; -{% if rp.ssl.dhparam %} - ssl_dhparam {{ rp.ssl.dhparam_container_path }}; -{% endif %} - ssl_protocols {{ rp.ssl.protocols }}; - ssl_ciphers {{ rp.ssl.ciphers }}; - ssl_prefer_server_ciphers {{ 'on' if rp.ssl.prefer_server_ciphers else 'off' }}; - ssl_session_cache {{ rp.ssl.session_cache }}; - ssl_ecdh_curve {{ rp.ssl.ecdh_curve }}; - ssl_session_tickets {{ 'on' if rp.ssl.session_tickets else 'off' }}; - ssl_stapling {{ 'on' if rp.ssl.stapling else 'off' }}; - ssl_stapling_verify {{ 'on' if rp.ssl.stapling_verify else 'off' }}; -{% if rp.resolver %} - resolver {{ rp.resolver | join(' ') }} valid=300s; - resolver_timeout {{ rp.resolver_timeout }}; -{% endif %} -{% if rp.hsts.enabled %} - add_header Strict-Transport-Security "max-age={{ rp.hsts.max_age }}{% if rp.hsts.include_subdomains %}; includeSubdomains{% endif %}{% if rp.hsts.preload %}; preload{% endif %}" always; -{% endif %} - add_header X-Frame-Options DENY; - add_header X-Content-Type-Options nosniff; - -{{ proxy_location(rp)|indent(4, true) }} -{% if rp.extra_config %} - -{{ rp.extra_config }} -{% endif %} -} -{% endif %} diff --git a/infer_stack/templates/suggestion-pool.yaml b/infer_stack/templates/suggestion-pool.yaml new file mode 100644 index 00000000..53cdaad5 --- /dev/null +++ b/infer_stack/templates/suggestion-pool.yaml @@ -0,0 +1,377 @@ +# suggestion-pool.yaml — the curated pool of "useful models" the leasing +# controller draws on when it seeds a catalog from server introspection. +# +# This is the intrinsic, *hardware-independent* model registry: each entry +# states facts that are true on any box (footprint, min per-GPU VRAM, preferred +# GPU count, context window, sane vLLM defaults). `infer_stack.leasing.suggest` +# JOINS this pool against the detected GPU inventory to derive a concrete, +# fits-this-server catalog ("seeding is a pure function +# inventory × registry → catalog"). +# +# Entries are lifted from the legacy `default-vllm-models.yaml`, spanning the +# current-generation families (Qwen3.5 / Qwen3.6 / Gemma 4 — all real, released +# Feb-Apr 2026) down through the prior-gen instruct workhorses (Qwen2.5, Llama 3, +# Mistral) and a couple of tiny smoke-test models, plus the two SmolLM2/Qwen2.5 +# models the leasing demo serves so the demo's set is reproducible. Big entries +# are kept even where they need more than one GPU — the pool is hardware-blind, +# and `suggest` filters per box. served_model_name is normalized to a clean slug +# (the legacy Together-style `org/model-turbo` aliases are dropped). +# +# This file is a *suggestion source*, never a live catalog — nothing here has a +# lifecycle. The legacy `default-vllm-models.yaml` is left untouched as the +# legacy compiler's fixture. + +vllm_models: + + # -- tiny: smoke-test / CI models, fit on any GPU ---------------------- + gpt2: + hf_model_id: openai-community/gpt2 + served_model_name: gpt2 + family: gpt2 + modalities: [text] + memory_class_gib: 1 + min_vram_gib_per_replica: 1 + preferred_gpu_count: 1 + context_window: 1024 + defaults: + max_model_len: 1024 + gpu_memory_utilization: 0.3 + pythia-410m: + hf_model_id: EleutherAI/pythia-410m + served_model_name: pythia-410m + family: pythia + modalities: [text] + memory_class_gib: 2 + min_vram_gib_per_replica: 2 + preferred_gpu_count: 1 + context_window: 2048 + defaults: + max_model_len: 2048 + gpu_memory_utilization: 0.3 + smollm2-135m: + hf_model_id: HuggingFaceTB/SmolLM2-135M-Instruct + served_model_name: smollm2-135m + family: smollm2 + modalities: [text] + memory_class_gib: 1 + min_vram_gib_per_replica: 1 + preferred_gpu_count: 1 + context_window: 8192 + defaults: + max_model_len: 8192 + gpu_memory_utilization: 0.3 + enable_prefix_caching: true + + # -- small instruct: comfortable on a single small/mid GPU ------------- + smollm2-1.7b: + hf_model_id: HuggingFaceTB/SmolLM2-1.7B-Instruct + served_model_name: smollm2-1.7b + family: smollm2 + modalities: [text] + memory_class_gib: 4 + min_vram_gib_per_replica: 4 + preferred_gpu_count: 1 + context_window: 8192 + defaults: + max_model_len: 8192 + gpu_memory_utilization: 0.4 + enable_prefix_caching: true + qwen2.5-0.5b: + hf_model_id: Qwen/Qwen2.5-0.5B-Instruct + served_model_name: qwen2.5-0.5b + family: qwen2.5 + modalities: [text] + memory_class_gib: 2 + min_vram_gib_per_replica: 2 + preferred_gpu_count: 1 + context_window: 32768 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.4 + enable_prefix_caching: true + qwen2.5-1.5b: + hf_model_id: Qwen/Qwen2.5-1.5B-Instruct + served_model_name: qwen2.5-1.5b + family: qwen2.5 + modalities: [text] + memory_class_gib: 4 + min_vram_gib_per_replica: 4 + preferred_gpu_count: 1 + context_window: 32768 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.4 + enable_prefix_caching: true + + # -- mid instruct: a single 16-24 GiB GPU ------------------------------ + qwen2.5-7b: + hf_model_id: Qwen/Qwen2.5-7B-Instruct + served_model_name: qwen2.5-7b + family: qwen2.5 + modalities: [text] + memory_class_gib: 16 + min_vram_gib_per_replica: 16 + preferred_gpu_count: 1 + context_window: 131072 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + mistral-7b-instruct: + hf_model_id: mistralai/Mistral-7B-Instruct-v0.3 + served_model_name: mistral-7b-instruct + family: mistral + modalities: [text] + memory_class_gib: 16 + min_vram_gib_per_replica: 16 + preferred_gpu_count: 1 + context_window: 32768 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + llama3-8b-instruct: + hf_model_id: meta-llama/Meta-Llama-3-8B-Instruct + served_model_name: llama3-8b-instruct + family: llama3 + modalities: [text] + memory_class_gib: 18 + min_vram_gib_per_replica: 18 + preferred_gpu_count: 1 + context_window: 8192 + defaults: + max_model_len: 8192 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + llama3.1-8b-instruct: + hf_model_id: meta-llama/Meta-Llama-3.1-8B-Instruct + served_model_name: llama3.1-8b-instruct + family: llama3.1 + modalities: [text] + memory_class_gib: 18 + min_vram_gib_per_replica: 18 + preferred_gpu_count: 1 + context_window: 131072 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + + # -- large: needs a big GPU, or tensor-parallel across two -------------- + gpt-oss-20b: + hf_model_id: openai/gpt-oss-20b + served_model_name: gpt-oss-20b + family: gpt-oss + modalities: [text] + memory_class_gib: 40 + min_vram_gib_per_replica: 40 + preferred_gpu_count: 1 + context_window: 131072 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + qwen2.5-72b: + hf_model_id: Qwen/Qwen2.5-72B-Instruct + served_model_name: qwen2.5-72b + family: qwen2.5 + modalities: [text] + memory_class_gib: 144 + min_vram_gib_per_replica: 72 + preferred_gpu_count: 2 + context_window: 131072 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + + # -- Qwen3.5 (current gen, reasoning) ---------------------------------- + # 0.8B-9B fit a single 24 GiB GPU; 27B+ need a 48 GiB card or more. + qwen3.5-0.8b: + hf_model_id: Qwen/Qwen3.5-0.8B + served_model_name: qwen3.5-0.8b + family: qwen3.5 + modalities: [text] + memory_class_gib: 4 + min_vram_gib_per_replica: 4 + preferred_gpu_count: 1 + context_window: 32768 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + qwen3.5-2b: + hf_model_id: Qwen/Qwen3.5-2B + served_model_name: qwen3.5-2b + family: qwen3.5 + modalities: [text] + memory_class_gib: 8 + min_vram_gib_per_replica: 8 + preferred_gpu_count: 1 + context_window: 32768 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + qwen3.5-4b: + hf_model_id: Qwen/Qwen3.5-4B + served_model_name: qwen3.5-4b + family: qwen3.5 + modalities: [text] + memory_class_gib: 12 + min_vram_gib_per_replica: 12 + preferred_gpu_count: 1 + context_window: 32768 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + qwen3.5-9b: + hf_model_id: Qwen/Qwen3.5-9B + served_model_name: qwen3.5-9b + family: qwen3.5 + modalities: [text] + memory_class_gib: 20 + min_vram_gib_per_replica: 20 + preferred_gpu_count: 1 + context_window: 32768 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + qwen3.5-27b: + hf_model_id: Qwen/Qwen3.5-27B + served_model_name: qwen3.5-27b + family: qwen3.5 + modalities: [text] + memory_class_gib: 48 + min_vram_gib_per_replica: 48 + preferred_gpu_count: 1 + context_window: 32768 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + qwen3.5-35b-a3b: + hf_model_id: Qwen/Qwen3.5-35B-A3B + served_model_name: qwen3.5-35b-a3b + family: qwen3.5 + modalities: [text] + memory_class_gib: 48 + min_vram_gib_per_replica: 48 + preferred_gpu_count: 1 + context_window: 32768 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + qwen3.5-122b-a10b: + hf_model_id: Qwen/Qwen3.5-122B-A10B + served_model_name: qwen3.5-122b-a10b + family: qwen3.5 + modalities: [text] + memory_class_gib: 80 + min_vram_gib_per_replica: 80 + preferred_gpu_count: 2 + context_window: 131072 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + qwen3.5-122b-a10b-fp8: + hf_model_id: Qwen/Qwen3.5-122B-A10B-FP8 + served_model_name: qwen3.5-122b-a10b-fp8 + family: qwen3.5 + modalities: [text] + memory_class_gib: 80 + min_vram_gib_per_replica: 80 + preferred_gpu_count: 2 + context_window: 262144 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + + # -- Qwen3.6 (current gen, reasoning, MoE) ----------------------------- + # The FP8 build squeezes onto one 24 GiB GPU; the bf16 build wants two. + qwen3.6-35b-a3b: + hf_model_id: Qwen/Qwen3.6-35B-A3B + served_model_name: qwen3.6-35b-a3b + family: qwen3.6 + modalities: [text] + memory_class_gib: 80 + min_vram_gib_per_replica: 24 + preferred_gpu_count: 2 + context_window: 262144 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: false + qwen3.6-35b-a3b-fp8: + hf_model_id: Qwen/Qwen3.6-35B-A3B-FP8 + served_model_name: qwen3.6-35b-a3b-fp8 + family: qwen3.6 + modalities: [text] + memory_class_gib: 40 + min_vram_gib_per_replica: 24 + preferred_gpu_count: 1 + context_window: 262144 + defaults: + max_model_len: 32768 + gpu_memory_utilization: 0.9 + enable_prefix_caching: false + + # -- Gemma 4 (current gen, multimodal, reasoning) ---------------------- + # All four fit a single 24 GiB GPU. + gemma4-e2b: + hf_model_id: google/gemma-4-E2B-it + served_model_name: gemma4-e2b + family: gemma4 + modalities: [text, image, audio] + memory_class_gib: 8 + min_vram_gib_per_replica: 8 + preferred_gpu_count: 1 + context_window: 131072 + defaults: + max_model_len: 65536 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + gemma4-e4b: + hf_model_id: google/gemma-4-E4B-it + served_model_name: gemma4-e4b + family: gemma4 + modalities: [text, image, audio] + memory_class_gib: 10 + min_vram_gib_per_replica: 10 + preferred_gpu_count: 1 + context_window: 131072 + defaults: + max_model_len: 65536 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + gemma4-26b: + hf_model_id: google/gemma-4-26B-A4B-it + served_model_name: gemma4-26b + family: gemma4 + modalities: [text, image] + memory_class_gib: 20 + min_vram_gib_per_replica: 20 + preferred_gpu_count: 1 + context_window: 262144 + defaults: + max_model_len: 65536 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true + gemma4-31b: + hf_model_id: google/gemma-4-31B-it + served_model_name: gemma4-31b + family: gemma4 + modalities: [text, image] + memory_class_gib: 22 + min_vram_gib_per_replica: 22 + preferred_gpu_count: 1 + context_window: 262144 + defaults: + max_model_len: 65536 + gpu_memory_utilization: 0.9 + enable_prefix_caching: true diff --git a/infer_stack/tui.py b/infer_stack/tui.py new file mode 100644 index 00000000..3335bb83 --- /dev/null +++ b/infer_stack/tui.py @@ -0,0 +1,1999 @@ +"""Optional Textual TUI to monitor and control the leasing stack. + +A multi-pane dashboard built to be approachable for someone who has never +touched infer-stack. Each pane carries its own one-line description, its own +action buttons, and (for the heavy ones) collapses with a click so it isn't +polled while hidden: + +* **Catalog** (left) — the models + endpoints you can run; Acquire the selected + endpoint, Suggest a set sized to your GPUs, or add one by hand. Ctrl+click a + served endpoint to open it in Open WebUI. +* **Leases** + **Deployments** (center) — the live ledger (desired *state* vs what's + actually *running*, and which GPUs), with Release / Evict / Clean-up. +* **docker** — a collapsible pane with **Logs** and **Containers** (the + ``docker ps`` view: status/uptime, created, id, ports) tabs. +* **system** — live ``nvidia-smi`` GPUs + host CPU/mem (collapsed by default). +* **api** — send a prompt to a *ready* model through the LiteLLM gateway + (collapsed by default). + +Responsiveness: refresh runs on a worker thread, only the visible tab / expanded +pane's expensive data is polled, and docker's own ``up``/``down`` output is +captured into the logs pane instead of bleeding onto the terminal. The log +source (``proc_factory``) and HTTP client (``http``) are injectable, so the app +is fully exercisable headless via Textual's pilot. +""" + +from __future__ import annotations + +import subprocess +from pathlib import Path +from typing import Any, Callable, Iterable + +from textual import events, work +from textual.app import App, ComposeResult +from textual.binding import Binding +from textual.containers import Horizontal, Vertical +from textual.screen import ModalScreen +from textual.theme import Theme +from textual.widgets import ( + Button, + Collapsible, + DataTable, + Footer, + Header, + Input, + Label, + RichLog, + Select, + Static, + TabbedContent, + TabPane, +) + +from .cli.commands_leasing import ( + _gpu_label, + _lease_ttl, + _placement_view, + _running_label, +) +from .leasing import DeploymentState, LeaseState + +ALL_SERVICES = '' # the Select value meaning "every service" +DEFAULT_THEME = 'textual-dark' + + +# A warm alternative palette, kept registered (selectable from the command +# palette) even though the default is the stock dark theme. +INFER_THEME = Theme( + name='infer-orange', + primary='#ff8c1a', + secondary='#ffb066', + accent='#ff8c1a', + foreground='#f2f2f2', + background='#141414', + surface='#1f1f1f', + panel='#272727', + success='#7bd88f', + warning='#ffb454', + error='#ff6b6b', + dark=True, +) + + +class _DockerLogProc: + """A live ``docker compose logs -f`` process for one (or all) service(s).""" + + def __init__(self, project: str, compose_file: str, service: str | None): + cmd = [ + 'docker', 'compose', '-p', project, '-f', compose_file, + 'logs', '-f', '--tail', '200', '--no-color', + ] + if service: + cmd.append(service) + self._proc = subprocess.Popen( + cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + text=True, bufsize=1, + ) + + @property + def stdout(self) -> Iterable[str]: + return self._proc.stdout or iter(()) + + def terminate(self) -> None: + self._proc.terminate() + try: + self._proc.wait(timeout=2) + except Exception: # noqa: BLE001 + self._proc.kill() + + +class _Divider(Static): + """A draggable splitter bar. + + Reports incremental drags to ``on_drag(delta)`` — ``delta`` is the pointer's + movement along the bar's resize axis since the last event. The bar must span + its full cross-axis (see CSS) or there is nothing to grab. + """ + + def __init__(self, axis: str, on_drag: Callable[[int], None], **kw: Any): + super().__init__('', **kw) + self._axis = axis # 'x' -> drag horizontally, 'y' -> drag vertically + self._on_drag = on_drag + self._dragging = False + + def on_mouse_down(self, event: events.MouseDown) -> None: + self._dragging = True + self.capture_mouse() + event.stop() + + def on_mouse_move(self, event: events.MouseMove) -> None: + if not self._dragging: + return + delta = event.delta_x if self._axis == 'x' else event.delta_y + if delta: + self._on_drag(int(delta)) + event.stop() + + def on_mouse_up(self, event: events.MouseUp) -> None: + self._dragging = False + self.release_mouse() + event.stop() + + +class _AddModelScreen(ModalScreen): + """Wizard: add a model (a weight source vLLM will serve).""" + + CSS = """ + _AddModelScreen { align: center middle; } + #dialog { + width: 70; height: auto; padding: 1 2; + border: round $accent; background: $surface; + } + #dialog .hint { color: $text-muted; margin: 0 0 1 0; } + #dialog Input { margin: 0 0 1 0; } + #dialog Horizontal { height: auto; align-horizontal: right; } + #dialog Button { margin: 0 0 0 2; } + """ + + def compose(self) -> ComposeResult: + with Vertical(id='dialog'): + yield Label('Add a model', classes='title') + yield Static( + 'A model is a set of weights (Hugging Face repo or local path). ' + 'Endpoints point at it.', + classes='hint', + ) + yield Input(placeholder='name (e.g. qwen-coder)', id='m-name') + yield Input( + placeholder='source (hf://Qwen/Qwen2.5-Coder-7B-Instruct or /path)', + id='m-source', + ) + with Horizontal(): + yield Button('Cancel', id='cancel') + yield Button('Add model', variant='primary', id='ok') + + def on_button_pressed(self, event: Button.Pressed) -> None: + if event.button.id == 'cancel': + self.dismiss(None) + return + name = self.query_one('#m-name', Input).value.strip() + source = self.query_one('#m-source', Input).value.strip() + if not name or not source: + self.query_one(Label).update('name and source are both required') + return + self.dismiss({'name': name, 'source': source}) + + +class _AddEndpointScreen(ModalScreen): + """Wizard: add or edit an endpoint (the served API name clients ask for). + + Exposes the runtime knobs that matter for serving — tensor-parallel size, + max model length, GPU memory fraction, raw extra vLLM args (where data + parallelism etc. go), and the reclaim policy — mirroring + ``catalog endpoint add``. Pass ``entry``/``name`` to edit an existing one. + """ + + CSS = """ + _AddEndpointScreen { align: center middle; } + #dialog { + width: 78; max-height: 90%; height: auto; padding: 1 2; + border: round $accent; background: $surface; overflow-y: auto; + } + #dialog .hint { color: $text-muted; margin: 0 0 1 0; } + #dialog Label { margin: 1 0 0 0; color: $text-muted; } + #dialog Input, #dialog Select { margin: 0 0 1 0; } + #dialog #buttons { height: auto; align-horizontal: right; margin: 1 0 0 0; } + #dialog #buttons Button { margin: 0 0 0 2; } + #vllm-opts, #ollama-opts { height: auto; } + """ + + def __init__(self, models: list[str], *, name: str | None = None, + entry: dict | None = None): + super().__init__() + self._models = models + self._edit_name = name + self._entry = entry or {} + + def _rt(self, key): + val = (self._entry.get('runtime') or {}).get(key) + return '' if val is None else str(val) + + def compose(self) -> ComposeResult: + editing = self._edit_name is not None + rt = self._entry.get('runtime') or {} + extra = rt.get('extra_args') or [] + extra_str = ' '.join(extra) if isinstance(extra, list) else str(extra) + reclaim = (self._entry.get('reclaim') or {}).get('policy', '') + cur_model = self._entry.get('model') + cur_engine = self._entry.get('engine', 'vllm') + with Vertical(id='dialog'): + yield Label('Edit endpoint' if editing else 'Add an endpoint', + classes='title') + yield Static( + 'The served name clients / Open WebUI request. Pick a model and ' + 'engine; the runtime knobs size it on the GPU(s).', classes='hint', + ) + yield Label('name') + yield Input(value=self._edit_name or '', + placeholder='optional — defaults to -N', + id='e-name', disabled=editing) + yield Label('model') + model_opts = [(m, m) for m in self._models] + if model_opts: + yield Select(model_opts, prompt='model…', id='e-model', + value=cur_model if cur_model in self._models + else Select.NULL) + else: + yield Input(value=cur_model or '', placeholder='model name', + id='e-model-text') + yield Label('engine') + yield Select([('vllm', 'vllm'), ('ollama', 'ollama')], + value=cur_engine, allow_blank=False, id='e-engine') + with Vertical(id='vllm-opts'): + yield Label('tensor-parallel size (GPUs per replica)') + yield Input(value=self._rt('tensor_parallel_size'), + placeholder='int, e.g. 2', id='e-tp') + yield Label('data-parallel size (replicas across GPUs)') + yield Input(value=self._rt('data_parallel_size'), + placeholder='int, e.g. 2', id='e-dp') + yield Label('max model len (context window, tokens)') + yield Input(value=self._rt('max_model_len'), + placeholder='int, e.g. 8192', id='e-mml') + yield Label('GPU memory utilization (0-1, per GPU)') + yield Input(value=self._rt('gpu_memory_utilization'), + placeholder='float, e.g. 0.9', id='e-gpu') + yield Label('max concurrent sequences') + yield Input(value=self._rt('max_num_seqs'), + placeholder='int, optional', id='e-seqs') + yield Label('prefix caching') + yield Select([('default', ''), ('on', 'on'), ('off', 'off')], + value='on' if rt.get('enable_prefix_caching') else '', + allow_blank=False, id='e-prefix') + yield Label('extra vLLM args (raw flags — dtype etc. go here)') + yield Input(value=extra_str, + placeholder='--dtype=half --enforce-eager', + id='e-extra') + with Vertical(id='ollama-opts'): + yield Label('host (runtime host name from the catalog)') + yield Input(value=self._entry.get('host', ''), + placeholder='e.g. ollama-local', id='e-host') + yield Label('extra runtime (KEY=VALUE, space-separated)') + yield Input(value=self._kv_str(rt), + placeholder='num_ctx=8192 keep_alive=5m', id='e-orun') + yield Label('reclaim policy (when idle)') + yield Select( + [('default', ''), ('keep-warm', 'keep-warm'), ('stop', 'stop'), + ('scale-to-zero', 'scale-to-zero')], + value=reclaim or '', allow_blank=False, id='e-reclaim', + ) + with Horizontal(id='buttons'): + yield Button('Cancel', id='cancel') + yield Button('Save' if editing else 'Add endpoint', + variant='primary', id='ok') + + def on_mount(self) -> None: + self._show_engine(self._entry.get('engine', 'vllm') or 'vllm') + + def on_select_changed(self, event: Select.Changed) -> None: + if event.select.id == 'e-engine': + engine = 'vllm' if event.value is Select.NULL else str(event.value) + self._show_engine(engine) + + def _show_engine(self, engine: str) -> None: + try: + self.query_one('#vllm-opts').display = engine == 'vllm' + self.query_one('#ollama-opts').display = engine == 'ollama' + except Exception: # noqa: BLE001 + pass + + @staticmethod + def _kv_str(runtime: dict) -> str: + return ' '.join( + f'{k}={v}' for k, v in (runtime or {}).items() if k != 'extra_args' + ) + + @staticmethod + def _num(raw: str, cast): + raw = raw.strip() + return cast(raw) if raw else None + + def on_button_pressed(self, event: Button.Pressed) -> None: + if event.button.id == 'cancel': + self.dismiss(None) + return + try: + value = self.query_one('#e-model', Select).value + model = '' if value is Select.NULL else str(value) + except Exception: # noqa: BLE001 - free-text fallback when no models yet + model = self.query_one('#e-model-text', Input).value.strip() + if not model: + self.query_one(Label).update('pick (or type) a model') + return + name = self.query_one('#e-name', Input).value.strip() + engine = str(self.query_one('#e-engine', Select).value or 'vllm') + reclaim = str(self.query_one('#e-reclaim', Select).value or '') + result: dict[str, Any] = { + 'name': self._edit_name or name or None, + 'model': model, 'engine': engine, 'reclaim': reclaim, + } + try: + if engine == 'vllm': + result.update({ + 'tensor_parallel': self._num(self._v('e-tp'), int), + 'data_parallel': self._num(self._v('e-dp'), int), + 'max_model_len': self._num(self._v('e-mml'), int), + 'gpu_mem': self._num(self._v('e-gpu'), float), + 'max_num_seqs': self._num(self._v('e-seqs'), int), + 'prefix_caching': str( + self.query_one('#e-prefix', Select).value or ''), + 'extra_args': self._v('e-extra'), + }) + else: + result.update({ + 'host': self._v('e-host'), + 'ollama_runtime': self._v('e-orun'), + }) + except ValueError: + self.query_one(Label).update( + 'parallel sizes / max-len / seqs must be ints, gpu-mem a float') + return + self.dismiss(result) + + def _v(self, wid: str) -> str: + return self.query_one(f'#{wid}', Input).value + + +class _ConfirmScreen(ModalScreen): + """A small yes/no confirmation for destructive actions.""" + + CSS = """ + _ConfirmScreen { align: center middle; } + #dialog { + width: 64; height: auto; padding: 1 2; + border: round $error; background: $surface; + } + #dialog Horizontal { height: auto; align-horizontal: right; } + #dialog Button { margin: 0 0 0 2; } + """ + + def __init__(self, message: str): + super().__init__() + self._message = message + + def compose(self) -> ComposeResult: + with Vertical(id='dialog'): + yield Label(self._message) + with Horizontal(): + yield Button('Cancel', id='cancel') + yield Button('Remove', variant='error', id='ok') + + def on_button_pressed(self, event: Button.Pressed) -> None: + self.dismiss(event.button.id == 'ok') + + +class InferStackTUI(App): + """Monitor + control the leasing stack across panes.""" + + TITLE = 'infer-stack' + SUB_TITLE = 'leasing dashboard' + + CSS = """ + Screen { layout: vertical; } + + #body { height: 1fr; padding: 0 1; } + #sidebar { width: 38; min-width: 10; } + #vsplit { width: 1; height: 1fr; background: $panel; } + #vsplit:hover { background: $accent; } + #main { width: 1fr; } + #tables { height: 1fr; overflow-y: auto; } + #hsplit, #csplit { + width: 1fr; height: 1; background: $panel; margin: 0 0 1 0; + } + #hsplit:hover, #csplit:hover { background: $accent; } + + /* one-line, per-pane descriptions (replaces the old global intro) */ + .desc { height: auto; color: $text-muted; padding: 0 1; } + #catalog-help { height: auto; color: $text-muted; padding: 0 1; } + + #endpoint-actions, #lease-actions, #deployment-actions, #model-actions { + height: auto; margin: 0 0 1 0; + } + /* width: 1fr + min-width: 0 so several buttons share a narrow sidebar row + instead of overflowing it (Button's default min-width is 16). */ + #endpoint-actions Button, #model-actions Button { width: 1fr; min-width: 0; } + #lease-actions Button, #deployment-actions Button { + margin: 0 1 0 0; min-width: 8; + } + /* compact buttons: 1 row, no border box, so action bars don't eat space. + text-wrap: nowrap is load-bearing, not cosmetic: the endpoint/model + buttons above are min-width: 0, so in a narrow sidebar their content box + can shrink to ~2 cells. Textual's Button carries line-pad: 1, and its + wrapping path folds the label at (width - line_pad*2). At width 2 that is + 0, and rich's chop_cells does range(0, n, 0) -> ValueError, crashing the + render. nowrap skips that fold path entirely. (We can't just set + line-pad: 0 -- Textual's integer CSS parser rejects 0.) */ + #endpoint-actions Button, #lease-actions Button, #deployment-actions Button, + #model-actions Button, #api-controls Button { + height: 1; border: none; padding: 0 1; text-wrap: nowrap; + } + #api { padding: 1 2; } + + #endpoints, #models, #leases-pane, #deployments-pane, #docker, #system, #api { + border: round $surface; + background: $boost; + padding: 0 1; + margin: 0 0 1 0; + border-title-color: $text-muted; + border-title-align: left; + border-subtitle-color: $text-muted; + border-subtitle-align: right; + } + #endpoints:focus, #models:focus, #leases-pane:focus-within, + #deployments-pane:focus-within, #docker:focus-within, #system:focus-within, + #api:focus-within { + border: round $accent; + border-title-color: $accent; + } + + #endpoints { height: 1fr; min-height: 5; } + #models { height: 8; min-height: 4; } /* height set via _apply_sizes */ + /* leases/deployments are collapsible panes; bound their tables so the + column sizes sanely and #tables scrolls when both are expanded. */ + #leases, #deployments { height: 10; min-height: 3; } + #docker-tabs { height: 16; min-height: 8; } + #logsvc { margin: 0 0 1 0; } + #logs, #ps { height: 1fr; background: $surface; } + #gpus, #api-out { height: 8; background: $surface; } + #sysinfo { height: auto; color: $text-muted; padding: 0 1; } + .hint { height: auto; color: $text-muted; padding: 0 1; } + #api-controls { height: auto; margin: 0 0 1 0; } + #api-controls Select { width: 1fr; } + #api-controls Button { margin: 0 0 0 1; min-width: 10; } + #api-prompt { margin: 0 0 1 0; } + #api-urls { height: auto; color: $text-muted; padding: 0 1; margin: 0 0 1 0; } + #api-curl { height: auto; color: $text-muted; padding: 0 1; } + #api-extra { height: auto; margin: 0 0 1 0; } + #api-extra Button { margin: 0 1 0 0; min-width: 12; } + + #status { dock: bottom; height: 1; padding: 0 2; color: $text-muted; } + + #settings { padding: 1 2; } + #settings Label { margin: 1 0 0 0; color: $text-muted; } + #settings Input, #settings Select { margin: 0 0 1 0; width: 64; } + #settings-actions { height: auto; margin: 1 0 0 0; } + #compose-path { height: auto; color: $text-muted; padding: 0 1; margin: 0 0 1 0; } + #compose-actions { height: auto; } + #compose-actions Button { margin: 0 1 0 0; } + """ + + BINDINGS = [ + # Truly global controls stay in the footer. + ('r', 'refresh', 'Refresh'), + ('tab', 'focus_next', 'Next pane'), + ('q', 'quit', 'Quit'), + # Pane-scoped actions: keys still work, but they live as buttons under + # the pane they act on rather than in the global menu. + Binding('s', 'acquire', 'Acquire', show=False), + Binding('d', 'release', 'Release', show=False), + Binding('e', 'evict', 'Evict', show=False), + Binding('a', 'release_all', 'Release all', show=False), + Binding('x', 'cleanup', 'Clean up', show=False), + Binding('g', 'suggest', 'Suggest', show=False), + Binding('m', 'add_model', 'Add model', show=False), + Binding('n', 'add_endpoint', 'Add endpoint', show=False), + Binding('o', 'open', 'Open in browser', show=False), + Binding('y', 'copy_status', 'Copy status', show=False), + Binding('c', 'toggle_docker', 'Collapse docker', show=False), + Binding('left_square_bracket', 'sidebar_narrower', 'sidebar -', show=False), + Binding('right_square_bracket', 'sidebar_wider', 'sidebar +', show=False), + Binding('minus', 'logs_shorter', 'logs -', show=False), + Binding('plus', 'logs_taller', 'logs +', show=False), + Binding('equals_sign', 'logs_taller', 'logs +', show=False), + ] + + def __init__( + self, + controller, + catalog, + *, + interval: float = 3.0, + proc_factory: Callable[[str | None], Any] | None = None, + catalog_path: str | Path | None = None, + http: Any = None, + ) -> None: + super().__init__() + self.controller = controller + self.catalog = catalog + self.interval = interval + self.catalog_path = Path(catalog_path).expanduser() if catalog_path else None + self._http = http + self._proc_factory = proc_factory or self._default_proc_factory() + self._endpoint_names: list[str] = [] + self._model_names: list[str] = [] + self._lease_ids: list[str] = [] + self._deployment_ids: list[str] = [] + self._last_leases: list[Any] = [] # row-aligned with the leases table + self._last_deployments: list[Any] = [] # row-aligned with the deployments table + self._service_options: list[str] = [] + self._log_service: str = ALL_SERVICES + self._log_proc: Any = None + self._log_lines: list[str] = [] # mirror of the log pane, for tests + self._api_lines: list[str] = [] # mirror of the API output, for tests + self._sidebar_w = 38 # resizable via [ ] or dragging #vsplit + self._log_h = 16 # resizable via - + or dragging #hsplit + self._models_h = 8 # resizable by dragging #csplit + self._active_tab = 'tab-logs' # which docker tab is visible + # heavy panes start collapsed (and therefore unpolled) + self._collapsed = {'docker': False, 'system': True} + self._ready_endpoints: list[str] = [] + + # -- layout ------------------------------------------------------------ + + def compose(self) -> ComposeResult: + yield Header(show_clock=True) + with TabbedContent(id='top'): + with TabPane('Dashboard', id='tab-dashboard'): + yield from self._compose_dashboard() + with TabPane('API', id='tab-api'): + yield from self._compose_api() + with TabPane('Settings', id='tab-settings'): + yield from self._compose_settings() + yield Static('', id='status') + yield Footer() + + def _compose_dashboard(self) -> ComposeResult: + with Horizontal(id='body'): + with Vertical(id='sidebar'): + yield Static( + 'Models & endpoints you can run. Acquire one, or Suggest a ' + 'set sized to your GPUs.', classes='desc', + ) + yield Static('', id='catalog-help') + yield DataTable(id='endpoints', cursor_type='row', + zebra_stripes=True) + with Horizontal(id='endpoint-actions'): + yield Button('Acquire', id='btn-acquire', variant='primary') + yield Button('Add', id='btn-add-endpoint') + yield Button('Edit', id='btn-edit-endpoint') + yield Button('Remove', id='btn-remove-endpoint') + yield _Divider('y', self._drag_models, id='csplit') + yield DataTable(id='models', cursor_type='row', + zebra_stripes=True) + with Horizontal(id='model-actions'): + yield Button('Add', id='btn-add-model') + yield Button('Remove', id='btn-remove-model') + yield _Divider('x', self._drag_sidebar, id='vsplit') + with Vertical(id='main'): + with Vertical(id='tables'): + with Collapsible(title='leases', collapsed=False, + id='leases-pane'): + yield Static( + 'Reservations you hold. Each maps to one deployment ' + 'below (see the deployment column); many leases can ' + 'share one. Select one → Release.', classes='desc', + ) + yield DataTable(id='leases', cursor_type='row', + zebra_stripes=True) + with Horizontal(id='lease-actions'): + yield Button('Release', id='btn-release') + yield Button('Release all', id='btn-release-all') + yield Button('Clean up', id='btn-cleanup') + with Collapsible(title='deployments', collapsed=False, + id='deployments-pane'): + yield Static( + 'Running model deployments and the GPUs they hold. ' + "The 'leases' column is how many leases hold each. " + 'Evict an idle one to free its GPU; Clean up forgets ' + 'stopped ones.', classes='desc', + ) + yield DataTable(id='deployments', cursor_type='row', + zebra_stripes=True) + with Horizontal(id='deployment-actions'): + yield Button('Evict', id='btn-evict') + yield Button('Clean up', id='btn-cleanup-deployments') + yield _Divider('y', self._drag_logs, id='hsplit') + with Collapsible(title='docker', collapsed=False, id='docker'): + with TabbedContent(id='docker-tabs'): + with TabPane('Logs', id='tab-logs'): + yield Select( + [('(all services)', ALL_SERVICES)], + value=ALL_SERVICES, allow_blank=False, + id='logsvc', + ) + yield RichLog(id='logs', highlight=False, + markup=False, max_lines=2000, + wrap=False) + with TabPane('Containers', id='tab-containers'): + yield DataTable(id='ps', cursor_type='row', + zebra_stripes=True) + with TabPane('Control', id='tab-control'): + yield Static( + 'Bring the rendered compose project up or down. ' + 'Output appears in the Logs tab.', classes='hint', + ) + yield Static('', id='compose-path') + with Horizontal(id='compose-actions'): + yield Button('Compose up', id='btn-compose-up', + variant='primary') + yield Button('Compose down', + id='btn-compose-down') + with Collapsible(title='system', collapsed=True, id='system'): + yield Static( + 'Live GPU utilization (nvidia-smi) and host CPU/memory.', + classes='desc', + ) + yield Static('', id='sysinfo') + yield DataTable(id='gpus', cursor_type='row', + zebra_stripes=True) + + def _compose_api(self) -> ComposeResult: + with Vertical(id='api'): + yield Static( + 'Talk to the LiteLLM gateway. Only models that are up and ready ' + 'are listed. Ctrl+click a URL to open it; buttons copy to your ' + 'clipboard.', classes='desc', + ) + yield Static('', id='api-urls', markup=False) + with Horizontal(id='api-controls'): + yield Select([], prompt='model…', id='api-model') + yield Button('Send', id='btn-api-send', variant='primary') + yield Button('Test all', id='btn-api-test-all') + yield Button('List models', id='btn-api-list') + yield Input(placeholder='prompt (default: a short hello)', + id='api-prompt') + yield Static('', id='api-curl') + with Horizontal(id='api-extra'): + yield Button('Copy curl', id='btn-api-copy-curl') + yield Button('Open WebUI', id='btn-open-webui') + yield RichLog(id='api-out', highlight=False, markup=False, + wrap=True, max_lines=500) + + def _compose_settings(self) -> ComposeResult: + from .cli.commands_leasing import _coerce_bool + from .paths import data_root, get_setting, settings_path + + def onoff(key, default): + return 'on' if _coerce_bool(get_setting(key), default) else 'off' + + with Vertical(id='settings'): + yield Static( + 'Durable settings (settings.yaml). Save writes them; backend / ' + 'data-dir / proxy take effect on the next acquire.', classes='desc', + ) + yield Static(f'file: {settings_path()}', classes='hint') + yield Label('backend') + yield Select( + [('null (dry-run)', 'null'), ('compose', 'compose'), + ('kubeai', 'kubeai')], + value=str(get_setting('backend') or 'null'), + allow_blank=False, id='set-backend', + ) + yield Label('data dir (where weights + state live)') + yield Input(value=str(get_setting('data_dir') or data_root()), + id='set-data-dir') + yield Label('Open WebUI') + yield Select([('on', 'on'), ('off', 'off')], value=onoff('ui', True), + allow_blank=False, id='set-ui') + yield Label('reverse proxy (single-port front door)') + yield Select([('off', 'off'), ('on', 'on')], + value=onoff('reverse_proxy', False), + allow_blank=False, id='set-rp') + yield Label('skip display GPUs during placement') + yield Select([('off', 'off'), ('on', 'on')], + value=onoff('skip_display_gpus', False), + allow_blank=False, id='set-skip') + with Horizontal(id='settings-actions'): + yield Button('Save settings', variant='primary', + id='btn-save-settings') + yield Static( + 'Catalog: seed a fits-your-GPUs set of models + endpoints from ' + 'the suggestion pool (merges into your catalog).', classes='desc', + ) + yield Button('✨ Suggest from my GPUs', id='btn-suggest') + + def on_mount(self) -> None: + try: + self.register_theme(INFER_THEME) + self.theme = DEFAULT_THEME + except Exception: # noqa: BLE001 + pass + titles = { + '#endpoints': 'catalog · endpoints', '#models': 'catalog · models', + } + for sel, title in titles.items(): + self.query_one(sel).border_title = title + self.query_one('#endpoints', DataTable).add_columns( + 'endpoint', 'model', 'engine', 'reclaim' + ) + self.query_one('#models', DataTable).add_columns( + 'model', 'source', 'quant', 'cached' + ) + self.query_one('#leases', DataTable).add_columns( + 'id', 'owner', 'state', 'ttl', 'endpoints', 'deployment' + ) + self.query_one('#deployments', DataTable).add_columns( + 'id', 'engine', 'state', 'running', 'gpus', 'served', + 'leases', 'held by' + ) + self.query_one('#ps', DataTable).add_columns( + 'service', 'status (uptime)', 'created', 'container id', 'ports' + ) + self.query_one('#gpus', DataTable).add_columns( + 'gpu', 'name', 'util%', 'mem (used/total)', 'temp' + ) + compose_file = getattr(self.controller.backend, 'compose_file', None) + self.query_one('#compose-path', Static).update( + f'compose file: {compose_file or "(not rendered yet)"}' + ) + # Capture docker's own chatter (up/down progress on stderr) into the + # logs pane instead of letting it bleed onto the full-screen terminal. + self._install_quiet_docker() + self._apply_sizes() + self._fill_catalog() + self._update_api_urls() + self._update_api_curl() + self._refresh_now() # synchronous first paint (snappy + testable) + self._restart_logs(self._log_service) + self.set_interval(self.interval, self.action_refresh) + self.query_one('#endpoints', DataTable).focus() + + def on_unmount(self) -> None: + self._terminate_logs() + + # -- theming for docker output bleed ---------------------------------- + + def _install_quiet_docker(self) -> None: + """Route ``docker compose`` output to the logs pane, not the terminal.""" + backend = self.controller.backend + if not hasattr(backend, 'run'): + return + + def quiet_run(args: list[str]) -> str: + proc = subprocess.run(args, capture_output=True, text=True) + noisy = not any(a == 'ps' for a in args) + if noisy: + for line in (proc.stderr or '').splitlines(): + if line.strip(): + self.call_from_thread(self._append_log, line.rstrip()) + if proc.returncode != 0: + raise subprocess.CalledProcessError( + proc.returncode, args, proc.stdout, proc.stderr + ) + return proc.stdout + + backend.run = quiet_run + + # -- resizable panes --------------------------------------------------- + + def _apply_sizes(self) -> None: + self.query_one('#sidebar').styles.width = self._sidebar_w + self.query_one('#docker-tabs').styles.height = self._log_h + self.query_one('#models').styles.height = self._models_h + + def _drag_sidebar(self, delta: int) -> None: + # Allow the full width range (down to a sliver, up to nearly all of it), + # not just the middle — clamp against the actual terminal width. + hi = max(20, self.size.width - 12) + self._sidebar_w = max(10, min(hi, self._sidebar_w + delta)) + self._apply_sizes() + + def _drag_logs(self, delta: int) -> None: + # Dragging the divider down (delta > 0) makes the docker pane shorter. + self._log_h = max(6, min(60, self._log_h - delta)) + self._apply_sizes() + + def _drag_models(self, delta: int) -> None: + # Divider above the models table (models fixed-height, endpoints flexes): + # drag down = bar follows the cursor down = models shorter. + self._models_h = max(3, min(40, self._models_h - delta)) + self._apply_sizes() + + def action_sidebar_narrower(self) -> None: + self._drag_sidebar(-4) + + def action_sidebar_wider(self) -> None: + self._drag_sidebar(4) + + def action_logs_shorter(self) -> None: + self._drag_logs(2) + + def action_logs_taller(self) -> None: + self._drag_logs(-2) + + def action_toggle_docker(self) -> None: + col = self.query_one('#docker', Collapsible) + col.collapsed = not col.collapsed + + def action_copy_status(self) -> None: + """Copy the status line (often a URL/command) to the clipboard.""" + text = str(self.query_one('#status', Static).render()) + if not self._copy(text): + self._status('copy failed — install wl-copy/xclip, or enable OSC 52') + + # -- catalog ----------------------------------------------------------- + + def _fill_catalog(self) -> None: + eps = self.query_one('#endpoints', DataTable) + eps.clear() + self._endpoint_names = [] + for name in sorted(self.catalog.endpoints): + ep = self.catalog.endpoints[name] + eps.add_row( + name, getattr(ep, 'model', '') or '-', + getattr(ep, 'engine', '') or '-', + str(getattr(ep, 'reclaim', '') or '-'), + ) + self._endpoint_names.append(name) + models = self.query_one('#models', DataTable) + models.clear() + self._model_names = [] + hub = self._hf_hub_dir() + for name in sorted(self.catalog.models): + m = self.catalog.models[name] + source = getattr(m, 'source', '') or '' + quant = getattr(m, 'quantization', None) or '-' + models.add_row(name, source, quant, self._cached_label(source, hub)) + self._model_names.append(name) + self._update_catalog_help() + + @staticmethod + def _hf_hub_dir() -> Path | None: + """Host Hugging Face hub cache dir (best-effort) for a 'cached?' check.""" + try: + from .config import default_state_paths + hf = default_state_paths().get('hf_cache') + return Path(hf).expanduser() / 'hub' if hf else None + except Exception: # noqa: BLE001 + return None + + @staticmethod + def _cached_label(source: str, hub: Path | None) -> str: + """'yes'/'no'/'-' whether a model's weights look present (no slow du).""" + if not source: + return '-' + if not source.startswith('hf://'): + try: + return 'yes' if Path(source).expanduser().exists() else 'no' + except Exception: # noqa: BLE001 + return '-' + if hub is None: + return '?' + repo = source[len('hf://'):].split('@', 1)[0] + try: + return 'yes' if (hub / f'models--{repo.replace("/", "--")}').exists() \ + else 'no' + except Exception: # noqa: BLE001 + return '-' + + def _sync_api_models(self, names: list[str]) -> None: + """Point the API model selector at the currently-ready endpoints only.""" + if names == self._ready_endpoints: + return + self._ready_endpoints = names + select = self.query_one('#api-model', Select) + current = None if select.value is Select.NULL else select.value + select.set_options([(n, n) for n in names]) + if current in names: + select.value = current + elif names: + select.value = names[0] + + def _update_catalog_help(self) -> None: + help_ = self.query_one('#catalog-help', Static) + if not self._endpoint_names: + help_.update( + 'No endpoints yet. Press [b]g[/b] to suggest a set sized to ' + 'your GPUs, or [b]m[/b]/[b]n[/b] to add a model/endpoint.' + ) + else: + help_.update( + 'Select an endpoint and Acquire it. Ctrl+click a served one to ' + 'open it in Open WebUI.' + ) + + def _reload_catalog(self) -> None: + if not self.catalog_path or not self.catalog_path.exists(): + return + try: + from .leasing import Catalog + self.catalog = Catalog.load(self.catalog_path) + except Exception as ex: # noqa: BLE001 + self._status(f'catalog reload failed: {ex}') + return + self._fill_catalog() + + # -- monitor ----------------------------------------------------------- + + def _collect(self) -> dict[str, Any]: + """Gather ledger + observed state. Safe to call off the UI thread. + + Only data for a *visible* (expanded, active) pane is polled — collapse + the docker / system panes (or sit on another docker tab) and the + expensive ``ps`` / nvidia-smi calls stop. + """ + try: + self.controller.ledger.sweep() + leases, deployments = self.controller.ledger.status() + observed, assignments = _placement_view(self.controller) + data: dict[str, Any] = { + 'leases': leases, 'deployments': deployments, 'observed': observed, + 'assignments': assignments, + } + if not self._collapsed['docker'] and self._active_tab == 'tab-containers': + data['ps'] = self._compose_ps_rows() + if not self._collapsed['system']: + data['gpus'] = self._gpu_rows() + data['sysinfo'] = self._system_line() + return data + except Exception as ex: # noqa: BLE001 - a monitor must never crash + return {'error': str(ex)} + + def _render(self, data: dict[str, Any]) -> None: + if 'error' in data: + self._status(f'refresh error: {data["error"]}') + return + self._last_leases = data['leases'] + self._last_deployments = data['deployments'] + self._fill_leases(data['leases']) + self._fill_deployments( + data['deployments'], data['observed'], data['assignments'], data['leases'] + ) + if 'ps' in data: + self._fill_ps(data['ps']) + if 'gpus' in data: + self._fill_gpus(data['gpus']) + self.query_one('#sysinfo', Static).update(data.get('sysinfo', '')) + # Ready = endpoints served by a deployment that is actually running. + ready: set[str] = set() + for g in data['deployments']: + if g.id in data['observed']: + ready.update(g.served) + self._sync_api_models(sorted(ready)) + self._update_summary(data['leases'], data['deployments'], data['observed']) + self._sync_log_services() + + def _sync_pane_state(self) -> None: + """Read live collapse + active-tab state (UI thread) so the polling gate + in ``_collect`` reflects reality — Collapsible.Toggled doesn't fire on + every path, so don't depend on it alone.""" + try: + self._collapsed['docker'] = \ + self.query_one('#docker', Collapsible).collapsed + self._collapsed['system'] = \ + self.query_one('#system', Collapsible).collapsed + self._active_tab = self.query_one('#docker-tabs', TabbedContent).active + except Exception: # noqa: BLE001 + pass + + def _refresh_now(self) -> None: + """Synchronous collect + render (first paint; also what tests rely on).""" + self._sync_pane_state() + self._render(self._collect()) + + @work(thread=True, exclusive=True, group='refresh') + def _refresh_bg(self) -> None: + data = self._collect() + self.call_from_thread(self._render, data) + + def action_refresh(self) -> None: + self._sync_pane_state() # capture pane state on the UI thread first + self._refresh_bg() + + def _update_summary(self, leases, deployments, observed) -> None: + active = sum(1 for le in leases if str(le.state) == 'active') + running = sum(1 for g in deployments if g.id in observed) + try: + self.query_one('#docker', Collapsible).title = ( + f'docker — {running} running' + ) + self.query_one('#leases-pane', Collapsible).title = ( + f'leases — {active} active / {len(leases)}' + ) + self.query_one('#deployments-pane', Collapsible).title = ( + f'deployments — {running} running / {len(deployments)}' + ) + except Exception: # noqa: BLE001 + pass + + def _fill_leases(self, leases) -> None: + table = self.query_one('#leases', DataTable) + cursor = table.cursor_row + table.clear() + self._lease_ids = [] + for le in leases: + # The 'deployment' column is the join key: it lists the same deployment + # id(s) shown in the deployments pane, so lease -> deployment is visible. + table.add_row( + le.id, le.owner, str(le.state), _lease_ttl(le), + ','.join(le.endpoints) or '-', + ','.join(le.deployment_ids) or '-', + ) + self._lease_ids.append(le.id) + self._restore_cursor(table, cursor) + + def _fill_deployments(self, deployments, observed, assignments, leases) -> None: + table = self.query_one('#deployments', DataTable) + cursor = table.cursor_row + table.clear() + self._deployment_ids = [] + # owners of the active leases holding each deployment (the "many" side) + owners: dict[str, list[str]] = {} + for le in leases: + if le.state == LeaseState.ACTIVE: + for gid in le.deployment_ids: + owners.setdefault(gid, []).append(le.owner) + for g in deployments: + table.add_row( + g.id, g.engine, str(g.state), + _running_label(g.id, observed), + _gpu_label(g.id, observed, assignments), + ','.join(sorted(g.served)) or '-', + str(g.demand), ','.join(owners.get(g.id, [])) or '-', + ) + self._deployment_ids.append(g.id) + self._restore_cursor(table, cursor) + + def _fill_ps(self, rows) -> None: + table = self.query_one('#ps', DataTable) + cursor = table.cursor_row + table.clear() + for row in rows: + table.add_row( + row['service'], row['status'], row['created'] or '-', + row['id'] or '-', row['ports'] or '-', + ) + if not rows: + table.add_row('(nothing running)', '-', '-', '-', '-') + self._restore_cursor(table, cursor) + + def _fill_gpus(self, rows) -> None: + table = self.query_one('#gpus', DataTable) + cursor = table.cursor_row + table.clear() + if rows is None: + table.add_row('—', 'nvidia-smi not found on this host', '-', '-', '-') + elif not rows: + table.add_row('—', '(no GPUs reported)', '-', '-', '-') + else: + for r in rows: + idx, name, util, used, total, temp = r + table.add_row(idx, name, f'{util}%', + f'{used}/{total} MiB', f'{temp}°C') + self._restore_cursor(table, cursor) + + @staticmethod + def _restore_cursor(table: DataTable, row: int) -> None: + if table.row_count: + table.move_cursor(row=min(max(row, 0), table.row_count - 1)) + + # -- system info ------------------------------------------------------- + + def _gpu_rows(self) -> list[list[str]] | None: + """nvidia-smi rows, or ``None`` when nvidia-smi is unavailable.""" + import shutil + + if not shutil.which('nvidia-smi'): + return None + try: + out = subprocess.run( + ['nvidia-smi', + '--query-gpu=index,name,utilization.gpu,memory.used,' + 'memory.total,temperature.gpu', + '--format=csv,noheader,nounits'], + capture_output=True, text=True, timeout=5, + ) + except Exception: # noqa: BLE001 + return [] + rows = [] + for line in out.stdout.splitlines(): + parts = [p.strip() for p in line.split(',')] + if len(parts) >= 6: + rows.append(parts[:6]) + return rows + + def _system_line(self) -> str: + """A compact host CPU/mem/load line from /proc (best-effort).""" + import os + + bits = [] + try: + la = Path('/proc/loadavg').read_text().split()[:3] + bits.append('load ' + ' '.join(la)) + except Exception: # noqa: BLE001 + pass + try: + mem = {} + for line in Path('/proc/meminfo').read_text().splitlines(): + key, _, val = line.partition(':') + mem[key] = val.strip() + total = int(mem['MemTotal'].split()[0]) # kB + avail = int(mem['MemAvailable'].split()[0]) + used_g = (total - avail) / 1024 / 1024 + total_g = total / 1024 / 1024 + bits.append(f'mem {used_g:.1f}/{total_g:.1f} GiB') + except Exception: # noqa: BLE001 + pass + bits.append(f'cpus {os.cpu_count()}') + return ' · '.join(bits) + + # -- docker ps --------------------------------------------------------- + + def _compose_ps_rows(self) -> list[dict[str, str]]: + """Best-effort ``docker compose ps`` rows.""" + import json + + backend = self.controller.backend + path = getattr(backend, 'compose_file', None) + run = getattr(backend, 'run', None) + if not path or not run or not Path(path).exists(): + return [] + project = getattr(backend, 'project', 'infer-stack') + try: + out = run([ + 'docker', 'compose', '-p', str(project), '-f', str(path), + 'ps', '--format', 'json', + ]) + except Exception: # noqa: BLE001 - ps is best-effort + return [] + rows: list[dict[str, Any]] = [] + out = (out or '').strip() + if not out: + return [] + try: + parsed = json.loads(out) + rows = parsed if isinstance(parsed, list) else [parsed] + except json.JSONDecodeError: + for line in out.splitlines(): + line = line.strip() + if line: + try: + rows.append(json.loads(line)) + except json.JSONDecodeError: + pass + result = [] + for row in rows: + ports = _fmt_ports(row) or str(row.get('Ports') or '') + cid = str(row.get('ID') or '')[:12] + # `Status` is the docker-ps STATUS column ("Up 3 minutes") — it + # carries the uptime; `CreatedAt`/`RunningFor` give the age. + created = str(row.get('CreatedAt') or row.get('RunningFor') or '') + result.append({ + 'service': str(row.get('Service') or row.get('Name') or '?'), + 'status': str(row.get('Status') or row.get('State') or '?'), + 'created': created, + 'id': cid, + 'ports': ports, + }) + return sorted(result, key=lambda r: r['service']) + + # -- logs -------------------------------------------------------------- + + def _service_names(self) -> list[str]: + """Service names from the on-disk compose file (best-effort).""" + backend = self.controller.backend + path = getattr(backend, 'compose_file', None) + if not path: + return [] + try: + import yaml + data = yaml.safe_load(Path(path).read_text()) or {} + return sorted((data.get('services') or {}).keys()) + except Exception: # noqa: BLE001 + return [] + + def _sync_log_services(self) -> None: + names = self._service_names() + if names == self._service_options: + return + self._service_options = names + select = self.query_one('#logsvc', Select) + options = [('(all services)', ALL_SERVICES)] + [(n, n) for n in names] + select.set_options(options) + select.value = ( + self._log_service if self._log_service in names else ALL_SERVICES + ) + + def on_select_changed(self, event: Select.Changed) -> None: + if event.select.id == 'api-model': + self._update_api_curl() + return + if event.select.id != 'logsvc': + return + service = '' if event.value is Select.NULL else str(event.value) + if service != self._log_service: + self._log_service = service + self._restart_logs(service) + + def on_input_changed(self, event: Input.Changed) -> None: + if event.input.id == 'api-prompt': + self._update_api_curl() + + def _default_proc_factory(self) -> Callable[[str | None], Any]: + def factory(service: str | None): + backend = self.controller.backend + path = getattr(backend, 'compose_file', None) + project = getattr(backend, 'project', 'infer-stack') + if not path: + return None + return _DockerLogProc(str(project), str(path), service) + + return factory + + def _restart_logs(self, service: str) -> None: + self._terminate_logs() + log = self.query_one('#logs', RichLog) + log.clear() + self._log_lines = [] + label = service or 'all services' + log.write(f'— following logs: {label} —') + self._stream_logs(service or None) + + def _terminate_logs(self) -> None: + proc, self._log_proc = self._log_proc, None + if proc is not None: + try: + proc.terminate() + except Exception: # noqa: BLE001 + pass + + @work(thread=True, exclusive=True, group='logs') + def _stream_logs(self, service: str | None) -> None: + proc = self._proc_factory(service) + if proc is None: + self.call_from_thread( + self._append_log, '(no compose project yet — acquire a model)' + ) + return + self._log_proc = proc + try: + for line in proc.stdout: + self.call_from_thread(self._append_log, line.rstrip('\n')) + except Exception: # noqa: BLE001 - stream ends when the proc dies + pass + + def _append_log(self, line: str) -> None: + self._log_lines.append(line) + self.query_one('#logs', RichLog).write(line) + + # -- docker tab / collapse plumbing ------------------------------------ + + def on_tabbed_content_tab_activated( + self, event: TabbedContent.TabActivated + ) -> None: + # Only the docker sub-tabs gate polling; the top-level Dashboard/Settings + # tabs fire this too — ignore those. + if getattr(event.tabbed_content, 'id', None) != 'docker-tabs': + return + self._active_tab = self.query_one('#docker-tabs', TabbedContent).active + self.action_refresh() # fill the newly-shown tab right away + + def on_collapsible_toggled(self, event: Collapsible.Toggled) -> None: + self._sync_pane_state() + if self._collapsed['docker']: + self._terminate_logs() + elif self._log_proc is None: + self._restart_logs(self._log_service) + self.action_refresh() + + # -- helpers + actions ------------------------------------------------- + + def _status(self, message: str) -> None: + self.query_one('#status', Static).update(message) + + def _copy(self, text: str) -> bool: + """Copy to the system clipboard. Prefer OS tools (reliable on a desktop: + wl-copy / xclip / xsel / pbcopy) since many terminals don't honor the + OSC 52 escape Textual uses; fall back to OSC 52.""" + import shutil + import subprocess + + for cmd in (['wl-copy'], ['xclip', '-selection', 'clipboard'], + ['xsel', '--clipboard', '--input'], ['pbcopy']): + if shutil.which(cmd[0]): + try: + subprocess.run(cmd, input=text.encode(), check=True, + timeout=5) + return True + except Exception: # noqa: BLE001 + continue + try: + self.copy_to_clipboard(text) # OSC 52 (terminal must allow it) + return True + except Exception: # noqa: BLE001 + return False + + def _selected(self, table_id: str, ids: list[str]) -> str | None: + row = self.query_one(f'#{table_id}', DataTable).cursor_row + return ids[row] if 0 <= row < len(ids) else None + + def action_acquire(self) -> None: + name = self._selected('endpoints', self._endpoint_names) + if not name: + self._status('select an endpoint in the catalog to acquire') + return + self._status(f'acquiring {name}… (docker output appears in the logs pane)') + self._do_acquire(name) + + def on_data_table_row_selected(self, event: DataTable.RowSelected) -> None: + # Enter on the endpoints table acquires that endpoint. + if event.data_table.id == 'endpoints': + self.action_acquire() + + def on_data_table_row_highlighted( + self, event: DataTable.RowHighlighted + ) -> None: + # Spell out the many-to-one link in the status bar as you move the + # cursor: a lease points at one deployment; a deployment is held by many. + tid = event.data_table.id + row = event.cursor_row + if tid == 'leases' and 0 <= row < len(self._last_leases): + le = self._last_leases[row] + deps = ', '.join(le.deployment_ids) or '—' + held = {g.id: g.demand for g in self._last_deployments} + n = max((held.get(gid, 0) for gid in le.deployment_ids), default=0) + others = f' (1 of {n} lease(s) on it)' if n > 1 else '' + self._status(f'lease {le.id} → deployment {deps}{others}') + elif tid == 'deployments' and 0 <= row < len(self._last_deployments): + g = self._last_deployments[row] + owners = [ + le.owner for le in self._last_leases + if g.id in le.deployment_ids and le.state == LeaseState.ACTIVE + ] + who = ', '.join(owners) or '—' + self._status( + f'deployment {g.id} ← held by {g.demand} lease(s): {who}' + ) + + def on_click(self, event: events.Click) -> None: + # Ctrl+click a served endpoint -> open it in Open WebUI. + if not getattr(event, 'ctrl', False): + return + try: + node, _ = self.screen.get_widget_at(event.screen_x, event.screen_y) + except Exception: # noqa: BLE001 + return + while node is not None: + nid = getattr(node, 'id', None) + if nid == 'endpoints': + self.action_open() + event.stop() + return + if nid == 'api-urls': + self.action_open_webui() + event.stop() + return + node = getattr(node, 'parent', None) + + def on_button_pressed(self, event: Button.Pressed) -> None: + handlers = { + 'btn-acquire': self.action_acquire, + 'btn-release': self.action_release, + 'btn-release-all': self.action_release_all, + 'btn-evict': self.action_evict, + 'btn-cleanup': self.action_cleanup, + 'btn-cleanup-deployments': self.action_cleanup, + 'btn-suggest': self.action_suggest, + 'btn-add-model': self.action_add_model, + 'btn-add-endpoint': self.action_add_endpoint, + 'btn-edit-endpoint': self.action_edit_endpoint, + 'btn-remove-endpoint': self.action_remove_endpoint, + 'btn-remove-model': self.action_remove_model, + 'btn-api-send': self.action_api_send, + 'btn-api-test-all': self.action_api_test_all, + 'btn-api-list': self.action_api_list_models, + 'btn-api-copy-curl': self.action_api_copy_curl, + 'btn-open-webui': self.action_open_webui, + 'btn-save-settings': self._on_save_settings, + 'btn-compose-up': self.action_compose_up, + 'btn-compose-down': self.action_compose_down, + } + handler = handlers.get(event.button.id or '') + if handler: + handler() + + def _on_save_settings(self) -> None: + from .paths import load_settings, save_settings + + try: + s = load_settings() + s['backend'] = str(self.query_one('#set-backend', Select).value) + data_dir = self.query_one('#set-data-dir', Input).value.strip() + if data_dir: + s['data_dir'] = data_dir + s['ui'] = self.query_one('#set-ui', Select).value == 'on' + s['reverse_proxy'] = self.query_one('#set-rp', Select).value == 'on' + s['skip_display_gpus'] = ( + self.query_one('#set-skip', Select).value == 'on' + ) + path = save_settings(s) + self._status(f'saved settings → {path}') + except Exception as ex: # noqa: BLE001 + self._status(f'save settings failed: {ex}') + + def action_release(self) -> None: + sid = self._selected('leases', self._lease_ids) + if not sid: + self._status('select a lease row to release') + return + self._status(f'releasing {sid}…') + self._do_release(sid) + + def action_release_all(self) -> None: + self._status('releasing all active leases…') + self._do_release_all() + + def action_evict(self) -> None: + gid = self._selected('deployments', self._deployment_ids) + if not gid: + self._status('select a deployment row to evict') + return + self._status(f'evicting {gid}…') + self._do_evict(gid) + + def action_cleanup(self) -> None: + self._status('cleaning up released/expired leases + stopped deployments…') + self._do_cleanup() + + # -- docker compose control ------------------------------------------- + + def _compose_target(self) -> tuple[Any, str, str] | None: + """(run, project, compose_file) for the leasing project, or None.""" + backend = self.controller.backend + path = getattr(backend, 'compose_file', None) + run = getattr(backend, 'run', None) + if not path or not run or not Path(path).exists(): + return None + return run, str(getattr(backend, 'project', 'infer-stack')), str(path) + + def action_compose_up(self) -> None: + if self._compose_target() is None: + self._status('nothing rendered yet — acquire a model first') + return + self._status('docker compose up… (output in the Logs tab)') + self._do_compose(['up', '-d', '--remove-orphans'], 'up') + + def action_compose_down(self) -> None: + if self._compose_target() is None: + self._status('nothing rendered yet — nothing to bring down') + return + self._status('docker compose down…') + self._do_compose(['down', '--remove-orphans'], 'down') + + @work(thread=True, exclusive=True, group='mutate') + def _do_compose(self, args: list[str], label: str) -> None: + target = self._compose_target() + if target is None: + self._after_mutation('nothing rendered yet') + return + run, project, path = target + try: + run(['docker', 'compose', '-p', project, '-f', path, *args]) + msg = f'compose {label} done' + except Exception as ex: # noqa: BLE001 + msg = f'compose {label} failed: {ex}' + self._after_mutation(msg) + + # -- open in browser --------------------------------------------------- + + def _ui_url(self, endpoint: str) -> str | None: + base = self._openwebui_url() + return f'{base}/?models={endpoint}' if base else None + + def _openwebui_url(self) -> str | None: + port = getattr(self.controller.backend, 'ui_port', None) + return f'http://localhost:{port}' if port else None + + def _served_endpoints(self) -> set[str]: + try: + leases, _ = self.controller.ledger.status() + except Exception: # noqa: BLE001 + return set() + served: set[str] = set() + for le in leases: + if le.state == LeaseState.ACTIVE: + served.update(le.endpoints) + return served + + def action_open(self) -> None: + name = self._selected('endpoints', self._endpoint_names) + if not name: + self._status('select an endpoint to open in the browser') + return + url = self._ui_url(name) + if not url: + self._status('no Open WebUI URL (compose backend only)') + return + served = name in self._served_endpoints() + opened = False + try: + import webbrowser + opened = webbrowser.open(url) + except Exception: # noqa: BLE001 + opened = False + note = '' if served else ' (not served yet — start it first)' + tail = '' if opened else ' [copy this URL into your browser]' + self._status(f'open {url}{note}{tail}') + + # -- catalog editing (suggest + wizards) ------------------------------- + + def action_add_model(self) -> None: + if not self.catalog_path: + self._status('no catalog path — launch the TUI with a catalog to edit') + return + self.push_screen(_AddModelScreen(), self._on_add_model) + + def _on_add_model(self, result: dict | None) -> None: + if not result: + return + try: + self._write_catalog('models', result['name'], { + 'source': result['source'], + }) + self._status(f'added model {result["name"]}') + except Exception as ex: # noqa: BLE001 + self._status(f'add model failed: {ex}') + return + self._reload_catalog() + + def action_add_endpoint(self) -> None: + if not self.catalog_path: + self._status('no catalog path — launch the TUI with a catalog to edit') + return + self.push_screen( + _AddEndpointScreen(sorted(self.catalog.models)), self._on_add_endpoint + ) + + def action_edit_endpoint(self) -> None: + name = self._selected('endpoints', self._endpoint_names) + if not name: + self._status('select an endpoint to edit') + return + if not self.catalog_path: + self._status('no catalog path — launch the TUI with a catalog to edit') + return + if name in self._served_endpoints(): + self._status(f'{name} is actively served — release it before editing') + return + from .cli.commands_catalog import _load_raw + entry = _load_raw(self.catalog_path)['endpoints'].get(name, {}) + self.push_screen( + _AddEndpointScreen(sorted(self.catalog.models), name=name, entry=entry), + self._on_add_endpoint, + ) + + @staticmethod + def _endpoint_entry(result: dict) -> dict: + """Build a catalog endpoint entry from a wizard result (mirrors the CLI). + + vLLM and Ollama expose different knobs: vLLM gets the parallelism / + context / memory runtime keys (dtype etc. via extra_args), Ollama gets a + host plus free-form KEY=VALUE runtime. + """ + import shlex + + entry: dict[str, Any] = {'engine': result['engine'], + 'model': result['model']} + runtime: dict[str, Any] = {} + if result['engine'] == 'vllm': + keymap = { + 'tensor_parallel': 'tensor_parallel_size', + 'data_parallel': 'data_parallel_size', + 'max_model_len': 'max_model_len', + 'max_num_seqs': 'max_num_seqs', + } + for rk, ck in keymap.items(): + if result.get(rk) is not None: + runtime[ck] = result[rk] + if result.get('gpu_mem') is not None: + runtime['gpu_memory_utilization'] = result['gpu_mem'] + if result.get('prefix_caching') in ('on', 'off'): + runtime['enable_prefix_caching'] = result['prefix_caching'] == 'on' + if result.get('extra_args'): + runtime['extra_args'] = shlex.split(result['extra_args']) + else: # ollama + if result.get('host'): + entry['host'] = result['host'] + runtime.update(_parse_kv_str(result.get('ollama_runtime', ''))) + if runtime: + entry['runtime'] = runtime + if result.get('reclaim'): + entry['reclaim'] = {'policy': result['reclaim']} + return entry + + def _on_add_endpoint(self, result: dict | None) -> None: + if not result: + return + try: + from .cli.commands_catalog import ( + _load_raw, + _next_indexed_name, + _slug_alias, + ) + data = _load_raw(self.catalog_path) + name = result['name'] + if not name: + name = _next_indexed_name( + data['endpoints'], _slug_alias(result['model']) + ) + self._write_catalog('endpoints', name, self._endpoint_entry(result)) + self._status(f'saved endpoint {name} -> {result["model"]}') + except Exception as ex: # noqa: BLE001 + self._status(f'save endpoint failed: {ex}') + return + self._reload_catalog() + + def action_remove_endpoint(self) -> None: + name = self._selected('endpoints', self._endpoint_names) + if not name or not self.catalog_path: + self._status('select an endpoint to remove') + return + if name in self._served_endpoints(): + self._status(f'{name} is actively served — release it before removing') + return + self.push_screen( + _ConfirmScreen(f"Remove endpoint '{name}' from the catalog?"), + lambda ok: self._do_remove('endpoints', name) if ok else None, + ) + + def action_remove_model(self) -> None: + name = self._selected('models', self._model_names) + if not name or not self.catalog_path: + self._status('select a model to remove') + return + self.push_screen( + _ConfirmScreen(f"Remove model '{name}'? (endpoints using it will " + 'block the write)'), + lambda ok: self._do_remove('models', name) if ok else None, + ) + + def _do_remove(self, section: str, name: str) -> None: + try: + from .cli.commands_catalog import _load_raw, _save_raw + data = _load_raw(self.catalog_path) + data[section].pop(name, None) + _save_raw(self.catalog_path, data) # validates cross-refs + self._status(f'removed {section[:-1]} {name}') + except Exception as ex: # noqa: BLE001 + self._status(f'remove failed: {ex}') + return + self._reload_catalog() + + def _write_catalog(self, section: str, name: str, entry: dict) -> None: + from .cli.commands_catalog import _load_raw, _save_raw + data = _load_raw(self.catalog_path) + data[section][name] = entry + _save_raw(self.catalog_path, data) # validates; raises on a bad write + + def action_suggest(self) -> None: + if not self.catalog_path: + self._status('no catalog path — launch the TUI with a catalog to edit') + return + self._status('inspecting GPUs and suggesting a catalog…') + self._do_suggest() + + @work(thread=True, exclusive=True, group='mutate') + def _do_suggest(self) -> None: + try: + from .hardware import detect_inventory + from .leasing.suggest import suggest_catalog + inventory = detect_inventory() + frag = suggest_catalog(inventory, reserve_display_gpu='auto') + if not frag.get('models'): + self._after_mutation( + 'no pooled model fits the detected GPUs — add one with m/n' + ) + return + from .cli.commands_catalog import _load_raw, _save_raw + data = _load_raw(self.catalog_path) + added = 0 + for sec in ('models', 'endpoints'): + for nm, val in frag.get(sec, {}).items(): + if nm not in data[sec]: + data[sec][nm] = val + added += 1 + _save_raw(self.catalog_path, data) + self.call_from_thread(self._reload_catalog) + self._after_mutation(f'suggested catalog merged ({added} new entries)') + except Exception as ex: # noqa: BLE001 + self._after_mutation(f'suggest failed: {ex}') + + # -- API tester -------------------------------------------------------- + + def _litellm(self) -> tuple[str | None, str | None]: + backend = self.controller.backend + port = getattr(backend, 'litellm_port', None) + if not port: + return None, None + key = None + mk = getattr(backend, 'master_key', None) + try: + key = mk() if callable(mk) else None + except Exception: # noqa: BLE001 + key = None + return f'http://localhost:{port}', key + + def _http_client(self) -> Any: + if self._http is not None: + return self._http + import requests + return requests + + def _api_chat(self, model: str, prompt: str) -> str: + base, key = self._litellm() + if not base: + raise RuntimeError('no LiteLLM gateway (needs the compose backend)') + headers = {'Authorization': f'Bearer {key}'} if key else {} + resp = self._http_client().post( + f'{base}/v1/chat/completions', + json={ + 'model': model, + 'messages': [{'role': 'user', 'content': prompt}], + 'max_tokens': 128, 'temperature': 0, + }, + headers=headers, timeout=120, + ) + resp.raise_for_status() + data = resp.json() + return data['choices'][0]['message']['content'] + + def _curl_for(self, model: str, prompt: str) -> str: + """The equivalent ``curl`` for a chat-completion against the gateway.""" + import json as _json + + base, key = self._litellm() + if not base: + return '# acquire a model first — no LiteLLM gateway yet' + auth = f" -H 'Authorization: Bearer {key}'" if key else '' + body = _json.dumps({ + 'model': model or '', + 'messages': [{'role': 'user', 'content': prompt or 'hello'}], + }) + return (f"curl -s {base}/v1/chat/completions{auth} " + f"-H 'Content-Type: application/json' -d '{body}'") + + def _update_api_urls(self) -> None: + # Plain text (no Textual [link] markup — it rejects the ':' in URLs); + # ctrl+click this line opens Open WebUI, or use the Open WebUI button. + base, _ = self._litellm() + ui = self._openwebui_url() + parts = [] + if base: + parts.append(f'gateway: {base}/v1') + if ui: + parts.append(f'open webui: {ui}') + text = ' · '.join(parts) or '(acquire a model to get a gateway URL)' + try: + self.query_one('#api-urls', Static).update(text) + except Exception: # noqa: BLE001 + pass + + def _update_api_curl(self) -> None: + model = self._selected_api_model() or '' + try: + prompt = self.query_one('#api-prompt', Input).value.strip() or 'hello' + self.query_one('#api-curl', Static).update( + self._curl_for(model, prompt)) + except Exception: # noqa: BLE001 + pass + + def _api_log(self, line: str) -> None: + self._api_lines.append(line) + self.query_one('#api-out', RichLog).write(line) + + def _selected_api_model(self) -> str | None: + value = self.query_one('#api-model', Select).value + return None if value is Select.NULL else str(value) + + def action_api_send(self) -> None: + model = self._selected_api_model() + if not model: + self._status('no ready models to query (acquire one first)') + return + prompt = (self.query_one('#api-prompt', Input).value.strip() + or 'Say hello in one short sentence.') + self._api_log(f'> [{model}] {prompt}') + self._do_api_send(model, prompt) + + def action_api_test_all(self) -> None: + models = list(self._ready_endpoints) + if not models: + self._status('no ready models to test (acquire one first)') + return + self._api_log(f'— testing {len(models)} ready model(s) —') + self._do_api_test_all(models) + + def action_api_list_models(self) -> None: + self._api_log('> GET /v1/models (what the gateway routes)') + self._do_api_list() + + def action_api_copy_curl(self) -> None: + text = str(self.query_one('#api-curl', Static).render()) + ok = self._copy(text) + self._status('copied curl to clipboard' if ok else + 'copy failed — install wl-copy/xclip, or enable OSC 52') + + def action_open_webui(self) -> None: + url = self._openwebui_url() + if not url: + self._status('no Open WebUI URL (compose backend only)') + return + opened = False + try: + import webbrowser + opened = webbrowser.open(url) + except Exception: # noqa: BLE001 + opened = False + self._status( + f'open {url}' + ('' if opened else ' [copy into your browser]')) + + @work(thread=True, group='api') + def _do_api_send(self, model: str, prompt: str) -> None: + try: + out = self._api_chat(model, prompt) + self.call_from_thread(self._api_log, f' [{model}] {out}') + except Exception as ex: # noqa: BLE001 + self.call_from_thread(self._api_log, f' [{model}] ERROR: {ex}') + + @work(thread=True, group='api') + def _do_api_list(self) -> None: + base, key = self._litellm() + if not base: + self.call_from_thread(self._api_log, ' ERROR: no LiteLLM gateway') + return + try: + headers = {'Authorization': f'Bearer {key}'} if key else {} + resp = self._http_client().get( + f'{base}/v1/models', headers=headers, timeout=30) + resp.raise_for_status() + ids = [m.get('id') for m in (resp.json().get('data') or [])] + self.call_from_thread( + self._api_log, f' routes: {", ".join(ids) or "(none)"}') + except Exception as ex: # noqa: BLE001 + self.call_from_thread(self._api_log, f' ERROR: {ex}') + + @work(thread=True, group='api') + def _do_api_test_all(self, models: list[str]) -> None: + import time + + for model in models: + start = time.perf_counter() + try: + self._api_chat(model, 'Reply with the single word: ok') + dt = time.perf_counter() - start + self.call_from_thread(self._api_log, f' ✓ {model} ({dt:.1f}s)') + except Exception as ex: # noqa: BLE001 + self.call_from_thread(self._api_log, f' ✗ {model} {ex}') + self.call_from_thread(self._api_log, '— done —') + + # Mutations converge the backend (docker up/down, possibly slow) off the UI + # thread; results + a refresh are marshalled back on. + + @work(thread=True, exclusive=True, group='mutate') + def _do_acquire(self, name: str) -> None: + try: + requests = self.catalog.resolve_names([name]) + self.controller.acquire( + 'manual', requests, ttl_seconds=None, wait=False, apply=True + ) + msg = f'acquiring {name}' + except Exception as ex: # noqa: BLE001 + msg = f'acquire {name} failed: {ex}' + self._after_mutation(msg) + + @work(thread=True, exclusive=True, group='mutate') + def _do_release(self, sid: str) -> None: + try: + self.controller.release(sid) + msg = f'released {sid}' + except Exception as ex: # noqa: BLE001 + msg = f'release {sid} failed: {ex}' + self._after_mutation(msg) + + @work(thread=True, exclusive=True, group='mutate') + def _do_release_all(self) -> None: + try: + self.controller.ledger.sweep() + leases, _ = self.controller.ledger.status() + active = [le.id for le in leases if le.state == LeaseState.ACTIVE] + for sid in active: + self.controller.ledger.release(sid) + self.controller.reconcile() + msg = f'released {len(active)} lease(s)' + except Exception as ex: # noqa: BLE001 + msg = f'release --all failed: {ex}' + self._after_mutation(msg) + + @work(thread=True, exclusive=True, group='mutate') + def _do_evict(self, gid: str) -> None: + try: + self.controller.ledger.sweep() + deployment = self.controller.ledger.get_deployment(gid) + if deployment is None or deployment.state != DeploymentState.IDLE: + msg = f'{gid} is not idle — release it first' + else: + self.controller.evict([gid]) + msg = f'evicted {gid}' + except Exception as ex: # noqa: BLE001 + msg = f'evict {gid} failed: {ex}' + self._after_mutation(msg) + + @work(thread=True, exclusive=True, group='mutate') + def _do_cleanup(self) -> None: + try: + n_leases, n_deployments = self.controller.ledger.prune() + msg = (f'cleaned up {n_leases} released/expired lease(s) + ' + f'{n_deployments} stopped deployment(s)') + except Exception as ex: # noqa: BLE001 + msg = f'cleanup failed: {ex}' + self._after_mutation(msg) + + def _after_mutation(self, message: str) -> None: + self.call_from_thread(self._status, message) + self.call_from_thread(self.action_refresh) + + +def _parse_kv_str(text: str) -> dict[str, Any]: + """Parse ``KEY=VALUE`` pairs (space-separated); values are YAML-typed.""" + import yaml + + out: dict[str, Any] = {} + for item in (text or '').split(): + if '=' in item: + key, _, val = item.partition('=') + out[key.strip()] = yaml.safe_load(val) + return out + + +def _fmt_ports(row: dict) -> str: + """Compact published-ports string from a compose ps JSON row.""" + pubs = row.get('Publishers') or [] + bits = [] + for pub in pubs: + published = pub.get('PublishedPort') + target = pub.get('TargetPort') + if published: + bits.append(f'{published}->{target}') + return ', '.join(bits) + + +def run_tui( + controller, + catalog, + *, + interval: float = 3.0, + catalog_path: str | Path | None = None, +) -> int: + """Run the TUI against a built controller + catalog. Returns an exit code.""" + # The narration loguru sink writes to stderr, which would corrupt the + # full-screen UI — silence it while the TUI owns the terminal. + try: + from ._log import logger + + logger.disable('infer_stack') + except Exception: # noqa: BLE001 + pass + app: Any = InferStackTUI( + controller, catalog, interval=interval, catalog_path=catalog_path + ) + app.run() + return 0 diff --git a/infer_stack/tuning.py b/infer_stack/tuning.py deleted file mode 100644 index 24d7179b..00000000 --- a/infer_stack/tuning.py +++ /dev/null @@ -1,3 +0,0 @@ -from __future__ import annotations - -# Placeholder for future tuning expansion. diff --git a/infer_stack/validator.py b/infer_stack/validator.py deleted file mode 100644 index 3f71e91c..00000000 --- a/infer_stack/validator.py +++ /dev/null @@ -1,314 +0,0 @@ -from __future__ import annotations - -from pathlib import Path -from typing import Any - - -def _host_path_exists(raw_path: str, generated_dir: str | None) -> bool: - """Mirror how Docker Compose resolves a bind-mount source path. - - Relative paths in the rendered ``docker-compose.yml`` resolve against the - directory that holds the compose file (the generated dir), not the CWD. - """ - - if not raw_path: - return False - p = Path(raw_path) - if not p.is_absolute() and generated_dir: - p = Path(generated_dir) / p - return p.exists() - - -def validate_resolved(resolved: dict[str, Any]) -> dict[str, Any]: - inventory = resolved.get('inventory', {}) - gpu_map = {g['index']: g for g in inventory.get('gpus', [])} - policy = resolved.get('policy', {}) - backend = resolved.get('backend', 'compose') - errors: list[str] = [] - warnings: list[str] = [] - used_ports: set[int] = set() - - providers = resolved.get('providers', {}) or {} - gateways = resolved.get('gateways', {}) or {} - frontends = resolved.get('frontends', {}) or {} - ollama = providers.get('ollama', {}) or {} - vllm = providers.get('vllm', {}) or {} - vllm_runtimes = vllm.get('runtimes', {}) or {} - litellm = gateways.get('litellm', {}) or {} - open_webui = frontends.get('open_webui', {}) or {} - reverse_proxy = frontends.get('reverse_proxy', {}) or {} - routes = (litellm.get('routes') or {}) if litellm.get('enabled') else {} - - if backend == 'kubeai': - if ollama.get('enabled'): - errors.append( - 'backend=kubeai does not support the Ollama provider yet' - ) - if litellm.get('enabled'): - errors.append( - 'backend=kubeai does not render the LiteLLM gateway yet' - ) - if open_webui.get('enabled'): - errors.append( - 'backend=kubeai does not render the Open WebUI frontend yet' - ) - if reverse_proxy.get('enabled'): - errors.append( - 'backend=kubeai does not render the Compose reverse proxy frontend' - ) - profiles = resolved.get('resource_profiles', {}) - if not profiles: - source = resolved.get( - 'resource_profiles_source', 'kubeai-values.local.yaml' - ) - errors.append( - 'No local KubeAI resource profiles were loaded for validation. ' - f'Expected them at {source}. ' - 'Run `python manage.py kubeai-sync-resource-profiles --from-file values-kubeai-local-gpu.yaml` first.' - ) - - if ( - not ollama.get('enabled') - and not vllm_runtimes - and not open_webui.get('enabled') - and not litellm.get('enabled') - ): - warnings.append( - 'resolved profile has no enabled providers, gateways, or frontends' - ) - - if litellm.get('enabled'): - route_providers = {route.get('provider') for route in routes.values()} - if not routes: - warnings.append('LiteLLM is enabled but has no routes') - if 'ollama' in route_providers and not ollama.get('enabled'): - errors.append( - 'LiteLLM has Ollama routes but the Ollama provider is disabled' - ) - if 'vllm' in route_providers and not vllm_runtimes: - errors.append( - 'LiteLLM has vLLM routes but no vLLM runtimes are enabled' - ) - - # Mixed direct backends with Open WebUI and no LiteLLM are intentionally not - # supported in this first graph rewrite. - if ( - open_webui.get('enabled') - and not litellm.get('enabled') - and ollama.get('enabled') - and vllm_runtimes - ): - if open_webui.get('provider') != 'ollama': - errors.append( - 'Open WebUI with mixed Ollama+vLLM direct providers requires LiteLLM' - ) - else: - warnings.append( - 'Open WebUI is connected only to Ollama; vLLM runtimes are raw/direct endpoints' - ) - - if open_webui.get('enabled'): - provider = open_webui.get('provider') - if provider == 'litellm' and not litellm.get('enabled'): - errors.append('Open WebUI provider=litellm but LiteLLM is disabled') - if provider == 'ollama' and not ollama.get('enabled'): - errors.append('Open WebUI provider=ollama but Ollama is disabled') - - if reverse_proxy.get('enabled'): - generated_dir = (resolved.get('output', {}) or {}).get('generated_dir') - target = reverse_proxy.get('target') - if target in {'open_webui', 'open-webui'} and not open_webui.get( - 'enabled' - ): - errors.append( - 'reverse_proxy target=open_webui but Open WebUI is disabled' - ) - if target == 'litellm' and not litellm.get('enabled'): - errors.append( - 'reverse_proxy target=litellm but LiteLLM is disabled' - ) - if target == 'ollama' and not ollama.get('enabled'): - errors.append('reverse_proxy target=ollama but Ollama is disabled') - - ssl = reverse_proxy.get('ssl', {}) or {} - ssl_enabled = ssl.get('enabled', True) - config_path = reverse_proxy.get('config_path') or '' - - # Publishing :443 with no TLS server block means nginx never listens on - # 443 and clients get connection refused. This only happens when a user - # explicitly overrides the publish_https default. - if reverse_proxy.get('publish_https', True) and not ssl_enabled: - errors.append( - 'reverse_proxy publishes HTTPS (:443) but ssl.enabled is false; ' - 'nginx will not listen on 443. Set ssl.enabled: true or publish_https: false.' - ) - - if config_path: - # A manual nginx config bypasses the typed SSL renderer entirely, so - # certificate fields below do not apply; just sanity-check the file. - if not _host_path_exists(config_path, generated_dir): - warnings.append( - f'reverse_proxy.config_path {config_path!r} was not found ' - '(Docker would create an empty directory mount and nginx would fail to start)' - ) - elif ssl_enabled: - for field in ('certificate', 'certificate_key'): - value = ssl.get(field) or '' - if not value: - warnings.append( - f'reverse_proxy SSL is enabled but ssl.{field} is empty' - ) - elif not _host_path_exists(value, generated_dir): - warnings.append( - f'reverse_proxy ssl.{field} {value!r} was not found ' - '(relative paths resolve against the generated compose directory)' - ) - dhparam = ssl.get('dhparam') or '' - if dhparam and not _host_path_exists(dhparam, generated_dir): - warnings.append( - f'reverse_proxy ssl.dhparam {dhparam!r} was not found ' - '(relative paths resolve against the generated compose directory)' - ) - - if ollama.get('enabled'): - if ollama.get('placement_error'): - errors.append( - f'ollama placement failed: {ollama["placement_error"]}' - ) - for idx in ollama.get('gpu_indices', []) or []: - if idx not in gpu_map: - errors.append(f'ollama references missing gpu index {idx}') - elif gpu_map[idx].get('display_active'): - warnings.append(f'ollama uses display-active GPU {idx}') - if ollama.get('publish_port'): - port = int(resolved.get('ports', {}).get('ollama', 11434)) - if port in used_ports: - errors.append(f'duplicate host port assignment: {port}') - used_ports.add(port) - - seen_service_names: set[str] = set() - for svc in vllm_runtimes.values(): - if svc['service_name'] in seen_service_names: - errors.append(f'duplicate service name: {svc["service_name"]}') - seen_service_names.add(svc['service_name']) - - if backend == 'kubeai': - resource_profile = str(svc.get('resource_profile', '')).strip() - if not resource_profile: - errors.append( - f'vLLM runtime {svc["runtime_name"]} is missing resource_profile for kubeai backend' - ) - else: - profile_name = resource_profile.split(':', 1)[0] - if profile_name not in resolved.get('resource_profiles', {}): - errors.append( - f'vLLM runtime {svc["runtime_name"]} references unknown resource profile {profile_name!r}' - ) - - if not svc.get('hf_model_id'): - errors.append( - f'vLLM runtime {svc["runtime_name"]} is missing hf_model_id' - ) - if not svc.get('served_model_name'): - errors.append( - f'vLLM runtime {svc["runtime_name"]} is missing served_model_name' - ) - protocol_mode = svc.get('protocol_mode') - supported = list(svc.get('supported_protocols') or []) - if supported and protocol_mode not in supported: - errors.append( - f'vLLM runtime {svc["runtime_name"]} requests protocol_mode={protocol_mode}, ' - f'but model {svc.get("model_ref")} supports only {supported}.' - ) - if svc.get('placement_error'): - errors.append( - f'vLLM runtime {svc["runtime_name"]} placement failed: {svc["placement_error"]}' - ) - gpu_indices = svc.get('gpu_indices', []) - if not gpu_indices: - warnings.append( - f'vLLM runtime {svc["runtime_name"]} has no concrete GPU assignment in the rendered plan' - ) - continue - if svc.get('tensor_parallel_size', 1) > len(gpu_indices): - errors.append( - f'vLLM runtime {svc["runtime_name"]} has tensor_parallel_size larger than assigned GPU count' - ) - tp = max(1, int(svc.get('tensor_parallel_size', 1))) - per_gpu_need = float(svc.get('min_vram_gib_per_replica', 0)) / tp - headroom = float(policy.get('minimum_vram_headroom_gib', 0)) - for idx in gpu_indices: - if idx not in gpu_map: - errors.append( - f'vLLM runtime {svc["runtime_name"]} references missing gpu index {idx}' - ) - continue - gpu = gpu_map[idx] - if ( - policy.get('reserve_display_gpu') == 'auto' - and gpu.get('display_active') - and policy.get('forbid_reserved_gpu_use') - ): - errors.append( - f'vLLM runtime {svc["runtime_name"]} uses display-active GPU {idx}' - ) - elif gpu.get('display_active'): - warnings.append( - f'vLLM runtime {svc["runtime_name"]} uses display-active GPU {idx}' - ) - if gpu.get('memory_gib', 0) < (per_gpu_need + headroom): - errors.append( - f'vLLM runtime {svc["runtime_name"]} estimates {per_gpu_need} GiB + {headroom} GiB headroom on GPU {idx}, ' - f'but only {gpu.get("memory_gib")} GiB is available' - ) - if len(gpu_indices) > 1 and policy.get( - 'require_homogeneous_multi_gpu_groups' - ): - names = { - gpu_map[idx]['name'] for idx in gpu_indices if idx in gpu_map - } - mems = { - gpu_map[idx]['memory_gib'] - for idx in gpu_indices - if idx in gpu_map - } - if len(names) > 1 or len(mems) > 1: - errors.append( - f'vLLM runtime {svc["runtime_name"]} uses a heterogeneous multi-GPU group' - ) - - if backend == 'compose' and litellm.get('enabled'): - port = int(resolved.get('ports', {}).get('litellm', 14042)) - if port in used_ports: - errors.append(f'duplicate host port assignment: {port}') - used_ports.add(port) - - if ( - backend == 'compose' - and open_webui.get('enabled') - and open_webui.get('publish_port', True) - ): - port = int(resolved.get('ports', {}).get('open_webui', 13000)) - if port in used_ports: - errors.append(f'duplicate host port assignment: {port}') - used_ports.add(port) - - if backend == 'compose' and reverse_proxy.get('enabled'): - if reverse_proxy.get('publish_http', True): - port = int( - reverse_proxy.get('http_port') - or resolved.get('ports', {}).get('reverse_proxy_http', 80) - ) - if port in used_ports: - errors.append(f'duplicate host port assignment: {port}') - used_ports.add(port) - if reverse_proxy.get('publish_https', True): - port = int( - reverse_proxy.get('https_port') - or resolved.get('ports', {}).get('reverse_proxy_https', 443) - ) - if port in used_ports: - errors.append(f'duplicate host port assignment: {port}') - used_ports.add(port) - - return {'ok': not errors, 'errors': errors, 'warnings': warnings} diff --git a/infer_stack/verification.py b/infer_stack/verification.py deleted file mode 100644 index 5b5a56ba..00000000 --- a/infer_stack/verification.py +++ /dev/null @@ -1,46 +0,0 @@ -from __future__ import annotations - -from pathlib import Path -from typing import Any - -from .config import KUBEAI_GENERATED_SUBDIR, normalized_output -from .profile_runtime import default_base_url - - -def verify_profile(deployment: dict[str, Any]) -> dict[str, Any]: - service = ( - deployment.get('services', [])[0] if deployment.get('services') else {} - ) - output_root = Path( - normalized_output(deployment.get('output'))['generated_dir'] - ) - expected = { - 'public_name': deployment['serving_profile']['public_name'], - 'logical_model_name': service.get('logical_model_name', ''), - 'protocol_mode': service.get( - 'protocol_mode', - deployment['serving_profile'].get('protocol_mode', ''), - ), - 'served_model_name': service.get( - 'served_model_name', - deployment['serving_profile'].get('served_model_name', ''), - ), - 'endpoint_base_url': default_base_url(deployment), - 'generated_artifacts': { - 'compose': str(output_root / 'docker-compose.yml'), - 'kubeai_models': str( - output_root / KUBEAI_GENERATED_SUBDIR / 'models.yaml' - ), - }, - } - checks = { - 'has_services': bool(deployment.get('services')), - 'has_public_name': bool(expected['public_name']), - 'has_logical_model_name': bool(expected['logical_model_name']), - 'has_protocol_mode': bool(expected['protocol_mode']), - } - return { - 'ok': all(checks.values()), - 'profile': expected, - 'checks': checks, - } diff --git a/pyproject.toml b/pyproject.toml index 44b9f97d..e12742ca 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -32,6 +32,7 @@ dynamic = [ ] dependencies = [ "jinja2>=3.1", + "loguru>=0.7", "pyyaml>=6.0", "requests>=2.31", "rich>=13.0", @@ -42,6 +43,9 @@ optional-dependencies.tests = [ "pytest>=7.0", "pytest-codeblocks>=0.17", ] +optional-dependencies.tui = [ + "textual>=0.50", +] urls.Homepage = "https://github.com/AIQ-Kitware/infer_stack" scripts.infer-stack = "infer_stack.cli:main" diff --git a/tests/test_cli_catalog.py b/tests/test_cli_catalog.py new file mode 100644 index 00000000..3330b12d --- /dev/null +++ b/tests/test_cli_catalog.py @@ -0,0 +1,264 @@ +"""Tests for the `infer-stack catalog …` editor and `help tree`.""" + +from __future__ import annotations + +import pytest +import yaml + +from infer_stack.cli.commands_catalog import ( + BundleAddCLI, + CatalogInitCLI, + CatalogSuggestCLI, + EndpointAddCLI, + EndpointRmCLI, + HostAddCLI, + ModelAddCLI, +) +from infer_stack.leasing import Catalog + + +def cat_path(tmp_path): + return str(tmp_path / 'catalog.yaml') + + +def _opts(tmp_path): + return ['--catalog', cat_path(tmp_path)] + + +def test_init_creates_sections(tmp_path): + CatalogInitCLI.main(argv=_opts(tmp_path)) + data = yaml.safe_load((tmp_path / 'catalog.yaml').read_text()) + assert set(data) >= {'models'} + # an empty catalog still parses + Catalog.load(cat_path(tmp_path)) + + +def test_add_model_endpoint_roundtrips_and_validates(tmp_path): + ModelAddCLI.main(argv=['smol17b', '--source', 'hf://org/SmolLM2', + *_opts(tmp_path)]) + EndpointAddCLI.main(argv=[ + 'chat', '--engine', 'vllm', '--model', 'smol17b', + '--max-model-len', '8192', '--gpu-mem', '0.4', + '--extra-args=--dtype=half --enforce-eager', '--reclaim', 'keep-warm', + *_opts(tmp_path), + ]) + cat = Catalog.load(cat_path(tmp_path)) + ep = cat.endpoints['chat'] + assert ep.model == 'smol17b' + assert ep.runtime['max_model_len'] == 8192 + assert ep.runtime['extra_args'] == ['--dtype=half', '--enforce-eager'] + assert ep.reclaim == 'keep-warm' + + +def test_endpoint_name_defaults_to_model(tmp_path): + # No NAME given -> the endpoint alias defaults to `{model}-1`, so the served + # alias / Open WebUI label is tied to the model. + ModelAddCLI.main(argv=['smol135', '--source', 'hf://org/Smol', + *_opts(tmp_path)]) + EndpointAddCLI.main(argv=['--model', 'smol135', '--max-model-len', '4096', + *_opts(tmp_path)]) + cat = Catalog.load(cat_path(tmp_path)) + assert cat.endpoints['smol135-1'].model == 'smol135' + + +def test_endpoint_default_name_slugs_ollama_tag(tmp_path): + # An Ollama tag (`llama3:8b`) is slugified for the alias but kept as the tag. + HostAddCLI.main(argv=['gpu0', '--engine', 'ollama', '--gpu', '0', + *_opts(tmp_path)]) + EndpointAddCLI.main(argv=['--engine', 'ollama', '--model', 'llama3:8b', + '--host', 'gpu0', *_opts(tmp_path)]) + cat = Catalog.load(cat_path(tmp_path)) + assert 'llama3-8b-1' in cat.endpoints + assert cat.endpoints['llama3-8b-1'].model == 'llama3:8b' + + +def test_endpoint_no_name_no_model_errors(tmp_path): + CatalogInitCLI.main(argv=_opts(tmp_path)) + with pytest.raises(SystemExit): + EndpointAddCLI.main(argv=_opts(tmp_path)) # neither NAME nor --model + + +def test_endpoint_default_name_autoincrements(tmp_path): + # Repeated default-named adds for one model accumulate as -1, -2 (no clobber). + ModelAddCLI.main(argv=['m', '--source', 'hf://org/M', *_opts(tmp_path)]) + EndpointAddCLI.main(argv=['--model', 'm', *_opts(tmp_path)]) + EndpointAddCLI.main(argv=['--model', 'm', *_opts(tmp_path)]) + cat = Catalog.load(cat_path(tmp_path)) + assert set(cat.endpoints) == {'m-1', 'm-2'} + + +def test_endpoint_referencing_missing_model_is_refused(tmp_path): + CatalogInitCLI.main(argv=_opts(tmp_path)) + with pytest.raises(SystemExit): + EndpointAddCLI.main(argv=['bad', '--engine', 'vllm', '--model', 'ghost', + *_opts(tmp_path)]) + # nothing persisted + cat = Catalog.load(cat_path(tmp_path)) + assert 'bad' not in cat.endpoints + + +def test_ollama_endpoint_needs_existing_host(tmp_path): + CatalogInitCLI.main(argv=_opts(tmp_path)) + with pytest.raises(SystemExit): + EndpointAddCLI.main(argv=['ot', '--engine', 'ollama', '--host', 'h', + '--model', 'm:1b', *_opts(tmp_path)]) + HostAddCLI.main(argv=['h', '--engine', 'ollama', '--gpu', '0', + '--keep-alive', '5m', *_opts(tmp_path)]) + EndpointAddCLI.main(argv=['ot', '--engine', 'ollama', '--host', 'h', + '--model', 'm:1b', *_opts(tmp_path)]) + cat = Catalog.load(cat_path(tmp_path)) + assert cat.hosts['h'].gpu_indices == [0] + assert cat.endpoints['ot'].host == 'h' + + +def test_add_existing_without_force_errors(tmp_path): + ModelAddCLI.main(argv=['m', '--source', 'hf://a', *_opts(tmp_path)]) + with pytest.raises(SystemExit): + ModelAddCLI.main(argv=['m', '--source', 'hf://b', *_opts(tmp_path)]) + ModelAddCLI.main(argv=['m', '--source', 'hf://b', '--force', *_opts(tmp_path)]) + cat = Catalog.load(cat_path(tmp_path)) + assert cat.models['m'].source == 'hf://b' + + +def test_rm_missing_errors_and_rm_removes(tmp_path): + ModelAddCLI.main(argv=['m', '--source', 'hf://a', *_opts(tmp_path)]) + EndpointAddCLI.main(argv=['e', '--engine', 'vllm', '--model', 'm', + *_opts(tmp_path)]) + with pytest.raises(SystemExit): + EndpointRmCLI.main(argv=['nope', *_opts(tmp_path)]) + EndpointRmCLI.main(argv=['e', *_opts(tmp_path)]) + cat = Catalog.load(cat_path(tmp_path)) + assert 'e' not in cat.endpoints + + +def test_rm_multiple_endpoints(tmp_path): + ModelAddCLI.main(argv=['m', '--source', 'hf://a', *_opts(tmp_path)]) + for nm in ('a', 'b', 'c'): + EndpointAddCLI.main(argv=[nm, '--model', 'm', *_opts(tmp_path)]) + # rm accepts several names at once + EndpointRmCLI.main(argv=['a', 'c', *_opts(tmp_path)]) + cat = Catalog.load(cat_path(tmp_path)) + assert set(cat.endpoints) == {'b'} + + # one missing name -> nothing removed (atomic) + with pytest.raises(SystemExit): + EndpointRmCLI.main(argv=['b', 'ghost', *_opts(tmp_path)]) + assert 'b' in Catalog.load(cat_path(tmp_path)).endpoints + + +def test_bundle_add_and_validate(tmp_path): + ModelAddCLI.main(argv=['m', '--source', 'hf://a', *_opts(tmp_path)]) + EndpointAddCLI.main(argv=['a', '--engine', 'vllm', '--model', 'm', *_opts(tmp_path)]) + EndpointAddCLI.main(argv=['b', '--engine', 'vllm', '--model', 'm', *_opts(tmp_path)]) + BundleAddCLI.main(argv=['pair', 'a', 'b', *_opts(tmp_path)]) + cat = Catalog.load(cat_path(tmp_path)) + assert cat.bundles['pair'] == ['a', 'b'] + + +def test_dry_run_does_not_write(tmp_path, capsys): + CatalogInitCLI.main(argv=_opts(tmp_path)) + ModelAddCLI.main(argv=['m', '--source', 'hf://a', '--dry-run', *_opts(tmp_path)]) + out = capsys.readouterr().out + assert 'hf://a' in out # printed + cat = Catalog.load(cat_path(tmp_path)) + assert 'm' not in cat.models # but not persisted + + +def test_endpoint_show_no_name_lists_all(tmp_path, capsys): + from infer_stack.cli.commands_catalog import EndpointShowCLI + + ModelAddCLI.main(argv=['m', '--source', 'hf://a', *_opts(tmp_path)]) + EndpointAddCLI.main(argv=['e1', '--engine', 'vllm', '--model', 'm', + *_opts(tmp_path)]) + EndpointAddCLI.main(argv=['e2', '--engine', 'vllm', '--model', 'm', + *_opts(tmp_path)]) + capsys.readouterr() + # no NAME -> show every endpoint (not "endpoint 'None' not found") + assert EndpointShowCLI.main(argv=_opts(tmp_path)) == 0 + shown = yaml.safe_load(capsys.readouterr().out)['endpoints'] + assert set(shown) == {'e1', 'e2'} + # a real name still shows just that one + capsys.readouterr() + EndpointShowCLI.main(argv=['e1', *_opts(tmp_path)]) + assert set(yaml.safe_load(capsys.readouterr().out)) == {'e1'} + # an unknown name errors and lists what's available + with pytest.raises(SystemExit) as exc: + EndpointShowCLI.main(argv=['ghost', *_opts(tmp_path)]) + assert 'have: e1, e2' in str(exc.value) + + +def test_show_piped_output_is_plain(tmp_path, capsys): + # Under capsys stdout is not a tty -> no ANSI escapes leak into pipes. + from infer_stack.cli.commands_catalog import CatalogShowCLI + + CatalogInitCLI.main(argv=_opts(tmp_path)) + ModelAddCLI.main(argv=['m', '--source', 'hf://a', *_opts(tmp_path)]) + capsys.readouterr() + CatalogShowCLI.main(argv=_opts(tmp_path)) + out = capsys.readouterr().out + assert '\x1b[' not in out # color-free when piped + assert yaml.safe_load(out)['models']['m']['source'] == 'hf://a' + + +def test_help_tree_lists_groups_and_leaves(capsys): + from infer_stack.cli.commands_meta import HelpTreeCLI + + HelpTreeCLI.main(argv=[]) + out = capsys.readouterr().out + assert 'catalog' in out + assert 'model' in out and 'endpoint' in out + assert 'acquire' in out # top-level leasing verb + assert 'tree' in out # itself, under help + + +# --------------------------------------------------------------------------- +# catalog suggest — seed from server introspection (simulated for determinism) +# --------------------------------------------------------------------------- + + +def test_suggest_render_only_writes_nothing(tmp_path, capsys): + # Default is render: prints a catalog fragment, but creates no file. + CatalogSuggestCLI.main( + argv=['--simulate-hardware', '2x80', *_opts(tmp_path)] + ) + out = capsys.readouterr().out + rendered = yaml.safe_load(out) + assert 'qwen2.5-72b' in rendered['models'] # 2 GPUs -> the 72B fits + assert 'qwen2.5-72b' in rendered['endpoints'] + assert not (tmp_path / 'catalog.yaml').exists() # nothing written + + +def test_suggest_apply_merges_and_validates(tmp_path): + CatalogSuggestCLI.main( + argv=['--simulate-hardware', '1x48', '--apply', *_opts(tmp_path)] + ) + cat = Catalog.load(cat_path(tmp_path)) # parses + cross-refs ok + assert 'qwen2.5-7b' in cat.endpoints + # the largest fitting model is kept warm, the rest stop + warm = [n for n, e in cat.endpoints.items() if e.reclaim == 'keep-warm'] + assert len(warm) == 1 + + +def test_suggest_apply_is_additive_and_idempotent(tmp_path): + # A hand-added entry survives; re-applying changes nothing without --force. + ModelAddCLI.main(argv=['mine', '--source', 'hf://me/Model', *_opts(tmp_path)]) + CatalogSuggestCLI.main( + argv=['--simulate-hardware', '1x24', '--apply', *_opts(tmp_path)] + ) + after_first = (tmp_path / 'catalog.yaml').read_text() + CatalogSuggestCLI.main( + argv=['--simulate-hardware', '1x24', '--apply', *_opts(tmp_path)] + ) + assert (tmp_path / 'catalog.yaml').read_text() == after_first + cat = Catalog.load(cat_path(tmp_path)) + assert 'mine' in cat.models # hand-added entry preserved + + +def test_suggest_no_fit_writes_nothing(tmp_path, capsys): + # A box too small for any pooled model: a friendly note, no file, no crash. + CatalogSuggestCLI.main( + argv=['--simulate-hardware', '1x0', '--apply', *_opts(tmp_path)] + ) + err = capsys.readouterr().err + assert 'no pooled model fits' in err + assert not (tmp_path / 'catalog.yaml').exists() diff --git a/tests/test_cli_config.py b/tests/test_cli_config.py new file mode 100644 index 00000000..b50f2baa --- /dev/null +++ b/tests/test_cli_config.py @@ -0,0 +1,140 @@ +"""Tests for `infer-stack config …` settings + how they're honored.""" + +from __future__ import annotations + + +def test_config_init_yes_is_noninteractive(tmp_path, monkeypatch): + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) + from infer_stack.cli.commands_meta import ConfigInitCLI + from infer_stack.paths import load_settings, set_data_root + + set_data_root(None) + # --yes must not prompt (no tty in tests) and must persist presets + ConfigInitCLI.main(argv=['--yes', '--data-dir', str(tmp_path / 's'), + '--backend', 'compose']) + settings = load_settings() + assert settings['backend'] == 'compose' + assert settings['data_dir'] == str(tmp_path / 's') + + +def test_config_init_announces_new_then_editing(tmp_path, monkeypatch, capsys): + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) + from infer_stack.cli.commands_meta import ConfigInitCLI + from infer_stack.paths import set_data_root + + set_data_root(None) + ConfigInitCLI.main(argv=['--yes', '--backend', 'compose']) + assert 'from scratch' in capsys.readouterr().out # first run = new + ConfigInitCLI.main(argv=['--yes', '--backend', 'compose']) + out = capsys.readouterr().out # second = editing + assert 'editing the existing config' in out + assert 'config edit' in out + + +def test_config_init_persists_all_known_settings(tmp_path, monkeypatch): + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) + from infer_stack.cli.commands_meta import KNOWN_SETTINGS, ConfigInitCLI + from infer_stack.paths import load_settings, set_data_root + + set_data_root(None) + ConfigInitCLI.main(argv=['--yes', '--data-dir', str(tmp_path / 's'), + '--backend', 'compose']) + settings = load_settings() + # init now writes every known setting (incl. ui + skip_display_gpus), so + # nothing the system honors is silently missing from a fresh config. + assert set(settings) == set(KNOWN_SETTINGS) + assert settings['skip_display_gpus'] is False # default + assert settings['ui'] is True # default + + +def test_config_init_preserves_other_settings(tmp_path, monkeypatch): + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) + from infer_stack.cli.commands_meta import ConfigInitCLI, ConfigSetCLI + from infer_stack.paths import load_settings, set_data_root + + set_data_root(None) + ConfigInitCLI.main(argv=['--yes', '--backend', 'compose']) + ConfigSetCLI.main(argv=['ui', 'false']) # an unrelated key + ConfigInitCLI.main(argv=['--yes', '--backend', 'null']) # re-init in place + settings = load_settings() + assert settings['ui'] is False # preserved + assert settings['backend'] == 'null' + + +def test_config_init_fresh_resets_to_defaults(tmp_path, monkeypatch, capsys): + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) + from infer_stack.cli.commands_meta import ConfigInitCLI, ConfigSetCLI + from infer_stack.paths import load_settings, set_data_root + + set_data_root(None) + ConfigInitCLI.main(argv=['--yes', '--backend', 'compose']) + ConfigSetCLI.main(argv=['ui', 'false']) # customize ui + ConfigInitCLI.main(argv=['--yes', '--fresh', '--backend', 'compose']) + assert 'starting fresh' in capsys.readouterr().out + settings = load_settings() + # --fresh discards the customization and resets to the default (ui defaults on) + assert settings['ui'] is True + assert settings['backend'] == 'compose' + + +def test_config_set_get_roundtrip(tmp_path, monkeypatch): + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) + from infer_stack.cli.commands_meta import ConfigGetCLI, ConfigSetCLI + from infer_stack.paths import load_settings + + ConfigSetCLI.main(argv=['backend', 'compose']) + assert load_settings()['backend'] == 'compose' + # get prints it (smoke; value asserted via load_settings) + ConfigGetCLI.main(argv=['backend']) + + +def test_data_dir_setting_honored_by_data_root(tmp_path, monkeypatch): + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) + monkeypatch.delenv('INFER_STACK_DATA_DIR', raising=False) + from infer_stack.cli.commands_meta import ConfigSetCLI + from infer_stack.paths import data_root, set_data_root + + set_data_root(None) # clear any process override + ConfigSetCLI.main(argv=['data_dir', str(tmp_path / 'state')]) + assert data_root() == tmp_path / 'state' + + +def test_env_data_dir_beats_setting(tmp_path, monkeypatch): + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) + from infer_stack.cli.commands_meta import ConfigSetCLI + from infer_stack.paths import data_root, set_data_root + + set_data_root(None) + ConfigSetCLI.main(argv=['data_dir', str(tmp_path / 'from-setting')]) + monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path / 'from-env')) + assert data_root() == tmp_path / 'from-env' # env wins over setting + + +def test_backend_setting_resolved_by_make_backend(tmp_path, monkeypatch): + monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) + import infer_stack.cli.commands_leasing as mod + import infer_stack.hardware as hw + from infer_stack.cli.commands_leasing import AcquireCLI + from infer_stack.cli.commands_meta import ConfigSetCLI + + seen = {} + + class FakeCompose: + def __init__(self, **kw): + seen.update(kw) + + monkeypatch.setattr(mod, 'ComposeBackend', FakeCompose) + monkeypatch.setattr(hw, 'detect_inventory', lambda: {}) + + # No persisted backend -> null (dry-run) + cfg = AcquireCLI.cli(argv=['e'], strict=False) + assert type(mod._make_backend(cfg)).__name__ == 'NullBackend' + + # Persisted backend -> compose, even without --backend + ConfigSetCLI.main(argv=['backend', 'compose']) + cfg = AcquireCLI.cli(argv=['e'], strict=False) + assert isinstance(mod._make_backend(cfg), FakeCompose) + + # Explicit --backend still wins (null overrides the setting) + cfg = AcquireCLI.cli(argv=['e', '--backend', 'null'], strict=False) + assert type(mod._make_backend(cfg)).__name__ == 'NullBackend' diff --git a/tests/test_cli_leasing.py b/tests/test_cli_leasing.py new file mode 100644 index 00000000..0ae7da18 --- /dev/null +++ b/tests/test_cli_leasing.py @@ -0,0 +1,494 @@ +"""Tests for the leasing CLI verbs (dry-run / NullBackend).""" + +from __future__ import annotations + +import json +import sys +from types import SimpleNamespace + +import pytest +import yaml + +from infer_stack.cli.commands_leasing import ( + AcquireCLI, + LeasesCLI, + ReleaseCLI, + RenewCLI, + RunCLI, + _default_owner, +) + +CATALOG = { + 'models': { + 'qc': {'source': 'hf://Qwen/Qwen2.5-Coder-32B-Instruct'}, + 'rr': {'source': 'hf://BAAI/bge-reranker-base'}, + }, + 'endpoints': { + 'qwen-coder': { + 'engine': 'vllm', + 'model': 'qc', + 'runtime': {'max_model_len': 32768}, + }, + 'reranker': {'engine': 'vllm', 'model': 'rr'}, + }, + 'bundles': {'both': ['qwen-coder', 'reranker']}, +} + + +@pytest.fixture +def env(tmp_path): + cat = tmp_path / 'catalog.yaml' + cat.write_text(yaml.safe_dump(CATALOG)) + return SimpleNamespace( + cat=str(cat), db=str(tmp_path / 'ledger.db'), tmp=tmp_path + ) + + +def _base(env): + return ['--ledger', env.db, '--catalog', env.cat] + + +def _leases_json(env, capsys): + capsys.readouterr() + LeasesCLI.main(argv=['--ledger', env.db, '--json']) + return json.loads(capsys.readouterr().out) + + +def test_acquire_writes_env_file(env): + envf = env.tmp / 'is.env' + rc = AcquireCLI.main( + argv=['qwen-coder', *_base(env), '--env-file', str(envf), + '--base-url', 'http://x:1/v1'] + ) + assert rc == 0 + text = envf.read_text() + assert 'INFER_STACK_LEASE_ID=lease-' in text + assert 'OPENAI_BASE_URL=http://x:1/v1' in text + assert 'INFER_STACK_ENDPOINT_QWEN_CODER=qwen-coder' in text + assert 'INFER_STACK_API_KEY_ENV=LITELLM_MASTER_KEY' in text + + +def test_acquire_coalesces_demand(env, capsys): + AcquireCLI.main(argv=['qwen-coder', *_base(env), '--owner', 'alice']) + AcquireCLI.main(argv=['qwen-coder', *_base(env), '--owner', 'bob']) + data = _leases_json(env, capsys) + assert len(data['leases']) == 2 + assert len(data['deployments']) == 1 + assert data['deployments'][0]['demand'] == 2 + + +def test_dedicated_makes_separate_deployment(env, capsys): + AcquireCLI.main(argv=['qwen-coder', *_base(env)]) + AcquireCLI.main(argv=['qwen-coder', '--dedicated', *_base(env)]) + data = _leases_json(env, capsys) + assert len(data['deployments']) == 2 + + +def test_bundle_acquire_expands(env, capsys): + capsys.readouterr() + AcquireCLI.main(argv=['both', *_base(env), '--json']) + data = json.loads(capsys.readouterr().out) + assert set(data['descriptor']['endpoints']) == {'qwen-coder', 'reranker'} + + +def test_release_via_env_file(env, capsys): + envf = env.tmp / 'is.env' + AcquireCLI.main(argv=['qwen-coder', *_base(env), '--env-file', str(envf)]) + rc = ReleaseCLI.main(argv=['--ledger', env.db, '--env-file', str(envf)]) + assert rc == 0 + data = _leases_json(env, capsys) + assert data['leases'][0]['state'] == 'released' + # default reclaim is keep-warm, so the deployment is idle but not torn down + assert data['deployments'][0]['state'] == 'idle' + assert data['deployments'][0]['demand'] == 0 + + +def test_release_all_releases_every_active_lease(env, capsys): + AcquireCLI.main(argv=['qwen-coder', *_base(env), '--owner', 'a']) + AcquireCLI.main(argv=['reranker', *_base(env), '--owner', 'b']) + rc = ReleaseCLI.main(argv=['--ledger', env.db, '--all']) + assert rc == 0 + data = _leases_json(env, capsys) + assert data['leases'] # leases still listed + assert all(le['state'] == 'released' for le in data['leases']) + + +def test_release_all_with_no_leases_is_friendly(env, capsys): + rc = ReleaseCLI.main(argv=['--ledger', env.db, '--all']) + assert rc == 0 + assert 'no active leases' in capsys.readouterr().out + + +def test_evict_idle_deployment_by_endpoint(env, capsys): + from infer_stack.cli.commands_leasing import EvictCLI + + # keep-warm: releasing leaves the deployment resident (idle), not stopped. + AcquireCLI.main(argv=['qwen-coder', *_base(env)]) + ReleaseCLI.main(argv=['--ledger', env.db, '--all']) + assert _leases_json(env, capsys)['deployments'][0]['state'] == 'idle' + + # evict by the served endpoint alias -> the deployment is stopped (GPU freed). + rc = EvictCLI.main(argv=['qwen-coder', '--ledger', env.db]) + assert rc == 0 + assert _leases_json(env, capsys)['deployments'][0]['state'] == 'stopped' + + +def test_evict_requires_target_or_all(env): + from infer_stack.cli.commands_leasing import EvictCLI + + with pytest.raises(SystemExit): + EvictCLI.main(argv=['--ledger', env.db]) # neither name nor --all + + +def test_release_evict_tears_down_immediately(env, capsys): + # --evict overrides keep-warm: the released deployment is stopped, not idle. + AcquireCLI.main(argv=['qwen-coder', *_base(env)]) + rc = ReleaseCLI.main(argv=['--ledger', env.db, '--all', '--evict']) + assert rc == 0 + assert _leases_json(env, capsys)['deployments'][0]['state'] == 'stopped' + + +def test_acquire_without_ttl_is_standing_lease(env, capsys): + # No --ttl -> an infinite (standing-service) lease owned by the caller. + AcquireCLI.main(argv=['qwen-coder', *_base(env)]) + data = _leases_json(env, capsys) + assert data['leases'][0]['owner'] == _default_owner() + assert data['leases'][0]['expires_at'] is None + + +def test_wait_after_parallel_no_wait_acquire(env, capsys): + from infer_stack.cli.commands_leasing import WaitCLI + + # fan out: kick both off without blocking, then wait for both + AcquireCLI.main(argv=['qwen-coder', *_base(env), '--no-wait']) + AcquireCLI.main(argv=['reranker', *_base(env), '--no-wait']) + capsys.readouterr() + rc = WaitCLI.main(argv=['qwen-coder', 'reranker', '--ledger', env.db]) + assert rc == 0 # null backend: ready now + assert 'ready' in capsys.readouterr().out + + +def test_wait_unknown_endpoint_errors(env): + from infer_stack.cli.commands_leasing import WaitCLI + + with pytest.raises(SystemExit): + WaitCLI.main(argv=['ghost', '--ledger', env.db]) # nothing serving it + + +def test_acquire_no_apply_stages_without_applying(env, capsys): + capsys.readouterr() + rc = AcquireCLI.main(argv=['qwen-coder', *_base(env), '--no-apply', '--json']) + assert rc == 0 + data = json.loads(capsys.readouterr().out) + assert data['applied'] is False # staged, not brought up + assert data['lease_id'].startswith('lease-') + # the lease is recorded (staged intent), so it shows up in `leases`... + ls = _leases_json(env, capsys) + assert ls['leases'][0]['state'] == 'active' + # ...and can be discarded + ReleaseCLI.main(argv=['--ledger', env.db, '--all']) + assert _leases_json(env, capsys)['leases'][0]['state'] == 'released' + + +def test_release_evict_apply_accept_yes_flag(env, capsys): + # release/evict/apply now gate the compose diff on a TTY; --yes skips it. + # (null backend never prompts, so this is the plumbing + batched-release path.) + from infer_stack.cli.commands_leasing import ApplyCLI, EvictCLI + + AcquireCLI.main(argv=['qwen-coder', *_base(env), '--owner', 'a']) + AcquireCLI.main(argv=['reranker', *_base(env), '--owner', 'b']) + assert ReleaseCLI.main(argv=['--ledger', env.db, '--all', '--yes']) == 0 + assert all(le['state'] == 'released' + for le in _leases_json(env, capsys)['leases']) + assert EvictCLI.main(argv=['--ledger', env.db, '--all', '--yes']) == 0 + assert ApplyCLI.main(argv=['--ledger', env.db, '--yes']) == 0 + + +def test_reverse_proxy_resolution_flag_bool_block(monkeypatch): + import infer_stack.cli.commands_leasing as mod + import infer_stack.paths as paths + + def resolve(argv, setting=None): + monkeypatch.setattr( + paths, 'get_setting', + lambda k: setting if k == 'reverse_proxy' else None, + ) + cfg = AcquireCLI.cli(argv=argv, strict=False) + return mod._resolve_reverse_proxy(cfg) + + base = ['qwen-coder'] + assert resolve(base) == (False, 80, None) # default off + assert resolve([*base, '--reverse-proxy']) == (True, 80, None) # flag + assert resolve(base, setting=True) == (True, 80, None) # bool setting + # a block carries port + BYO config; flag still overrides enabled + block = {'enabled': True, 'port': 8080, 'config_path': '/etc/n.conf'} + assert resolve(base, setting=block) == (True, 8080, '/etc/n.conf') + assert resolve([*base, '--no-reverse-proxy'], setting=block) == ( + False, 8080, '/etc/n.conf') + + +def test_render_and_apply_are_lease_free(env, capsys): + from infer_stack.cli.commands_leasing import ApplyCLI, RenderCLI + + # declare intent without applying, then the lease-free verbs operate on it + AcquireCLI.main(argv=['qwen-coder', *_base(env), '--no-apply']) + before = len(_leases_json(env, capsys)['leases']) + assert RenderCLI.main(argv=['--ledger', env.db]) == 0 # render: no `up` + assert ApplyCLI.main(argv=['--ledger', env.db]) == 0 # apply: brings up + after = _leases_json(env, capsys) + # neither render nor apply minted a lease — still exactly one + assert len(after['leases']) == before == 1 + + +def test_leases_reports_running_and_gpus(env, capsys): + # NullBackend serves nothing, so a leased deployment is desired-live but not + # actually running and has no GPU assignment — the view must say so. + AcquireCLI.main(argv=['qwen-coder', *_base(env)]) + data = _leases_json(env, capsys) + g = data['deployments'][0] + assert g['state'] == 'live' # desired (ledger) + assert g['running'] is False # actual (backend.observe) + assert g['gpus'] is None # no placement on the dry-run backend + + +def test_leases_gpu_and_running_labels(): + from infer_stack.cli.commands_leasing import _gpu_label, _running_label + + observed = {'g1'} + assignments = {'g1': [0, 1], 'g2': [2], 'g3': []} + assert _running_label('g1', observed) == 'running' + assert _running_label('g2', observed) == '—' + assert _gpu_label('g1', observed, assignments) == '0,1' # on these GPUs + assert _gpu_label('g2', observed, assignments) == '→2' # slated, not up + assert _gpu_label('g3', observed, assignments) == '→cpu' # cpu, slated + assert _gpu_label('gX', observed, assignments) == '-' # unknown + + +def test_renew_extends(env): + envf = env.tmp / 'is.env' + AcquireCLI.main( + argv=['qwen-coder', *_base(env), '--ttl', '1h', '--env-file', str(envf)] + ) + rc = RenewCLI.main( + argv=['--ledger', env.db, '--env-file', str(envf), '--ttl', '3h'] + ) + assert rc == 0 + + +def test_run_propagates_env_and_exit(env): + out = env.tmp / 'out.txt' + code = ( + 'import os, pathlib; ' + f'pathlib.Path({str(out)!r}).write_text(' + "os.environ.get('OPENAI_BASE_URL', ''))" + ) + rc = RunCLI.main( + argv=['--endpoint', 'qwen-coder', *_base(env), + '--base-url', 'http://run:9/v1', '--', sys.executable, '-c', code] + ) + assert rc == 0 + assert out.read_text() == 'http://run:9/v1' + + +def test_run_returns_child_exit_code(env): + rc = RunCLI.main( + argv=['--endpoint', 'qwen-coder', *_base(env), '--', + sys.executable, '-c', 'import sys; sys.exit(3)'] + ) + assert rc == 3 + + +def test_run_releases_on_exit(env, capsys): + RunCLI.main( + argv=['--endpoint', 'qwen-coder', *_base(env), '--', + sys.executable, '-c', 'pass'] + ) + data = _leases_json(env, capsys) + assert data['leases'][0]['state'] == 'released' + + +def test_unimplemented_backend_errors(env): + with pytest.raises(SystemExit): + AcquireCLI.main(argv=['qwen-coder', '--backend', 'kubeai', *_base(env)]) + + +def test_unknown_endpoint_errors(env): + with pytest.raises(SystemExit): + AcquireCLI.main(argv=['ghost', *_base(env)]) + + +def test_missing_catalog_errors(env): + with pytest.raises(SystemExit): + AcquireCLI.main( + argv=['qwen-coder', '--ledger', env.db, '--catalog', + str(env.tmp / 'nope.yaml')] + ) + + +def test_env_set_read_list_roundtrip(tmp_path, monkeypatch): + """`env` does it all: set KEY=VALUE, read KEY, --export, path.""" + import contextlib + import io + + monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path)) + from infer_stack.cli.commands_leasing import EnvCLI + + # `env KEY=VALUE` works before any serve (creates the .env), merges keys + EnvCLI.main(argv=['HF_TOKEN=hf_demo']) + EnvCLI.main(argv=['OTHER=x']) + env_file = tmp_path / 'leasing' / 'compose' / '.env' + text = env_file.read_text() + assert 'HF_TOKEN=hf_demo' in text and 'OTHER=x' in text # both preserved + + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + EnvCLI.main(argv=['HF_TOKEN']) # `env KEY` prints the value + assert buf.getvalue().strip() == 'hf_demo' + + # `--export` dumps every entry as shell-sourceable export lines + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + EnvCLI.main(argv=['--export']) + out = buf.getvalue() + assert 'export HF_TOKEN=hf_demo' in out and 'export OTHER=x' in out + + +def test_env_prints_path_first(tmp_path, monkeypatch): + import contextlib + import io + + monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path)) + from infer_stack.cli.commands_leasing import EnvCLI + + # `env` with no args prints the .env path even before it exists. + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + EnvCLI.main(argv=[]) + assert buf.getvalue().strip() == str( + tmp_path / 'leasing' / 'compose' / '.env' + ) + + +def test_env_read_missing_is_friendly(tmp_path, monkeypatch): + monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path)) + from infer_stack.cli.commands_leasing import EnvCLI + + with pytest.raises(SystemExit): + EnvCLI.main(argv=['NOPE']) # no .env yet + + +def test_skip_display_gpus_default_off_flag_and_setting(env, monkeypatch): + """Display GPUs are used by default; --skip-display-gpus / the setting opt in.""" + import infer_stack.cli.commands_leasing as mod + import infer_stack.hardware as hw + import infer_stack.paths as paths + + seen = {} + + class FakeCompose: + def __init__(self, **kw): + seen.update(kw) + + monkeypatch.setattr(mod, 'ComposeBackend', FakeCompose) + # _make_backend / _resolve_skip_display import these lazily from their home + # modules, so patch them there. + monkeypatch.setattr(hw, 'detect_inventory', lambda: {}) + + def skip_display_for(argv, setting=None): + seen.clear() + monkeypatch.setattr( + paths, 'get_setting', + lambda k: setting if k == 'skip_display_gpus' else None, + ) + cfg = AcquireCLI.cli(argv=argv, strict=False) + mod._make_backend(cfg) + return seen['skip_display'] + + base = ['qwen-coder', '--backend', 'compose'] + assert skip_display_for(base) is False # default: use all + assert skip_display_for([*base, '--skip-display-gpus']) is True # flag opts in + assert skip_display_for(base, setting='true') is True # setting opts in + # explicit flag wins over the setting + assert skip_display_for([*base, '--no-skip-display-gpus'], setting='true') is False + + +def test_ui_flag_and_setting_resolution(env, monkeypatch): + """Open WebUI: default on, --no-ui off, and `config set ui` honored.""" + import infer_stack.cli.commands_leasing as mod + import infer_stack.hardware as hw + + seen = {} + + class FakeCompose: + def __init__(self, **kw): + seen.update(kw) + + monkeypatch.setattr(mod, 'ComposeBackend', FakeCompose) + monkeypatch.setattr(hw, 'detect_inventory', lambda: {}) + + def ui_for(argv, setting=None): + # _make_backend / _resolve_ui both do `from ..paths import get_setting`. + monkeypatch.setattr( + 'infer_stack.paths.get_setting', + lambda k: {'backend': 'compose', 'ui': setting}.get(k), + ) + seen.clear() + mod._make_backend(mod.AcquireCLI.cli(argv=argv, strict=False)) + return seen['ui'] + + assert ui_for(['e', '--backend', 'compose']) is True # default on + assert ui_for(['e', '--backend', 'compose', '--no-ui']) is False + assert ui_for(['e', '--backend', 'compose'], setting=False) is False # setting + # explicit flag overrides the setting + assert ui_for(['e', '--backend', 'compose', '--ui'], setting=False) is True + + +def test_test_command_smokes_endpoint(tmp_path, monkeypatch, capsys): + """`infer-stack test ` posts a chat completion and prints the reply.""" + monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path)) + import requests + + from infer_stack.cli.commands_leasing import EnvCLI, TestCLI + + EnvCLI.main(argv=['LITELLM_MASTER_KEY=sk-test']) + capsys.readouterr() + + captured = {} + + class Resp: + status_code = 200 + text = '{}' + + def json(self): + return {'choices': [{'message': {'content': 'ready'}}]} + + def fake_post(url, headers=None, json=None, timeout=None): + captured['url'] = url + captured['auth'] = headers.get('Authorization') + captured['model'] = json['model'] + return Resp() + + monkeypatch.setattr(requests, 'post', fake_post) + rc = TestCLI.main(argv=['chat']) + out = capsys.readouterr().out + assert rc == 0 + assert captured['url'] == 'http://127.0.0.1:14042/v1/chat/completions' + assert captured['auth'] == 'Bearer sk-test' # managed key applied + assert captured['model'] == 'chat' # asks for the alias + assert 'ready' in out + + +def test_test_command_reports_failure(tmp_path, monkeypatch, capsys): + monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path)) + import requests + + from infer_stack.cli.commands_leasing import TestCLI + + def boom(*a, **k): + raise requests.exceptions.ConnectionError('refused') + + monkeypatch.setattr(requests, 'post', boom) + rc = TestCLI.main(argv=['chat']) + out = capsys.readouterr().out + assert rc == 1 + assert 'FAILED' in out and 'chat' in out diff --git a/tests/test_cli_meta.py b/tests/test_cli_meta.py index 07347770..d843cb8e 100644 --- a/tests/test_cli_meta.py +++ b/tests/test_cli_meta.py @@ -8,16 +8,29 @@ from infer_stack import __version__ - MANAGE_PY = Path(__file__).resolve().parents[1] / 'manage.py' +# Pre-leasing commands moved under `infer-stack legacy …`. +_LEGACY_CMDS = { + 'setup', 'init', 'resolve', 'validate', 'lock', 'render', 'switch', + 'list-models', 'list-profiles', 'explain', 'describe-profile', + 'verify-profile', 'kubeai-sync-resource-profiles', 'up', 'down', 'purge', + 'deploy', 'env', 'diagnose', 'wait-ready', 'smoke-test', 'benchmark', + 'ollama-pull', 'ollama-list', 'ollama-ps', +} + + def run_cli( tmp_path: Path, *args: str, check: bool = True ) -> subprocess.CompletedProcess[str]: + if args and args[0] in _LEGACY_CMDS: + args = ('legacy', *args) env = os.environ.copy() - env.setdefault('INFER_STACK_CONFIG_DIR', str(tmp_path)) - env.setdefault('INFER_STACK_DATA_DIR', str(tmp_path)) + # Force (not setdefault) so an ambient INFER_STACK_* in the caller's shell + # can't leak in and point the test at the real config/data dir. + env['INFER_STACK_CONFIG_DIR'] = str(tmp_path) + env['INFER_STACK_DATA_DIR'] = str(tmp_path) return subprocess.run( [sys.executable, str(MANAGE_PY), *args], env=env, @@ -33,65 +46,58 @@ def test_version_prints_package_version(tmp_path: Path) -> None: assert 'infer-stack' in out -def test_config_paths_reports_config_and_state(tmp_path: Path) -> None: +def test_config_paths_reports_groups(tmp_path: Path) -> None: out = run_cli(tmp_path, 'config', 'paths').stdout # Group headers are present. assert 'config:' in out assert 'data:' in out - assert 'state:' in out + assert 'leasing:' in out # Anchored under tmp_path, and nothing exists yet on a fresh root. - assert f'config.yaml (file, missing): {tmp_path / "config.yaml"}' in out - assert f'hf_cache (dir, missing): {tmp_path / "hf-cache"}' in out - - -def test_config_paths_status_tracks_existence(tmp_path: Path) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'compose', - '--profile', - 'qwen2-5-7b-instruct-turbo-default', - ) - out = run_cli(tmp_path, 'config', 'paths', 'config').stdout - assert f'config.yaml (file, exists): {tmp_path / "config.yaml"}' in out + assert f'settings.yaml (file, missing): {tmp_path / "settings.yaml"}' in out + assert f'catalog.yaml (file, missing): {tmp_path / "catalog.yaml"}' in out def test_config_paths_json_emits_structured_groups(tmp_path: Path) -> None: payload = json.loads(run_cli(tmp_path, 'config', 'paths', '--json').stdout) - assert set(payload) == {'config', 'data', 'state'} + assert set(payload) == {'config', 'data', 'leasing'} entry = payload['config'][0] assert set(entry) == {'label', 'kind', 'status', 'path'} -def test_config_paths_target_filters_to_single_group(tmp_path: Path) -> None: - out = run_cli(tmp_path, 'config', 'paths', 'state').stdout - assert 'state:' in out - assert 'config:' not in out - assert 'data:' not in out +def test_config_paths_includes_leasing_group(tmp_path: Path) -> None: + out = run_cli(tmp_path, 'config', 'paths', 'leasing').stdout + assert 'leasing:' in out + assert 'ledger' in out + assert 'docker-compose.yml' in out + assert 'env (secrets)' in out + + +def test_paths_top_level_alias_works(tmp_path: Path) -> None: + out = run_cli(tmp_path, 'paths').stdout + assert 'config:' in out + assert 'leasing:' in out -def test_config_paths_kubeai_includes_generated_dir(tmp_path: Path) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'kubeai', - '--profile', - 'qwen2-72b-instruct-tp2-balanced', +def test_status_summarizes_leases_when_present(tmp_path: Path) -> None: + from infer_stack.leasing import ( + EndpointRequest, + Ledger, + SqliteStore, + vllm_structural, ) - out = run_cli(tmp_path, 'config', 'paths', 'data').stdout - assert 'kubeai_generated_dir' in out + # A lease in the ledger at the anchored data dir (default_ledger_path()). + led = Ledger(SqliteStore(tmp_path / 'leasing' / 'ledger.db')) + led.acquire('me', [EndpointRequest('m', 'vllm', vllm_structural(model_ref='m'))]) + out = run_cli(tmp_path, 'status').stdout + assert 'leasing:' in out and '1 active' in out and 'lease(s)' in out -def test_config_paths_omits_data_root(tmp_path: Path) -> None: - # data_root is only a default anchor for relative paths; with absolute - # state/output paths it is unused, so it should not be reported. - out = run_cli(tmp_path, 'config', 'paths').stdout - assert 'data_root' not in out - assert 'generated_dir' in out + +def test_config_paths_target_filters_to_single_group(tmp_path: Path) -> None: + out = run_cli(tmp_path, 'config', 'paths', 'leasing').stdout + assert 'leasing:' in out + assert 'config:' not in out + assert 'data:' not in out def test_config_paths_rejects_unknown_target(tmp_path: Path) -> None: @@ -125,9 +131,23 @@ def test_render_rich_colorizes_status() -> None: } buf = io.StringIO() console = Console(file=buf, force_terminal=True, width=100) - _render_rich(groups, 'compose', console) + _render_rich(groups, console) out = buf.getvalue() assert 'config.yaml' in out assert 'models.yaml' in out # ANSI escape sequences are emitted on a (forced) terminal. assert '\x1b[' in out + + +def test_day2_compose_base_prefers_leasing(tmp_path: Path, monkeypatch) -> None: + from infer_stack.cli.commands_runtime import _day2_compose_base + + monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path)) + compose_file = tmp_path / 'leasing' / 'compose' / 'docker-compose.yml' + compose_file.parent.mkdir(parents=True) + compose_file.write_text('services: {}\n') + # leasing compose present -> targets it without needing config.yaml + base = _day2_compose_base(None, 'logs') + assert base == [ + 'docker', 'compose', '-p', 'infer-stack', '-f', str(compose_file) + ] diff --git a/tests/test_cli_setup.py b/tests/test_cli_setup.py deleted file mode 100644 index 02c42674..00000000 --- a/tests/test_cli_setup.py +++ /dev/null @@ -1,793 +0,0 @@ -from __future__ import annotations -from typing import Any - -import os -import subprocess -import sys -import time -from pathlib import Path - -import yaml - -from infer_stack import cli as cli_mod - - -MANAGE_PY = Path(__file__).resolve().parents[1] / 'manage.py' - - -def _anchor_paths(tmp_path: Path, monkeypatch) -> Path: - """Point both config_root() and data_root() at ``tmp_path``. - - ``config_root()`` / ``data_root()`` read these env vars fresh on every - call, so this is enough for in-process ``cli_mod.cmd_*`` invocations to - see the same layout as the subprocess ones launched via ``run_cli``. - ``monkeypatch.setenv`` auto-restores at test teardown. - """ - monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) - monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path)) - return tmp_path - - -def run_cli( - tmp_path: Path, *args: str, env: dict[str, str] | None = None -) -> subprocess.CompletedProcess[str]: - full_env = os.environ.copy() - # Anchor config + rendered artifacts under tmp_path so tests are - # independent of the user's ~/.config and ~/.cache directories. - full_env.setdefault('INFER_STACK_CONFIG_DIR', str(tmp_path)) - full_env.setdefault('INFER_STACK_DATA_DIR', str(tmp_path)) - if env: - full_env.update(env) - return subprocess.run( - [sys.executable, str(MANAGE_PY), *args], - env=full_env, - text=True, - capture_output=True, - check=True, - ) - - -def write_kubeai_values_file(path: Path) -> None: - path.write_text( - yaml.safe_dump( - { - 'resourceProfiles': { - 'gpu-single-default': { - 'nodeSelector': { - 'nvidia.com/gpu.product': 'NVIDIA_TEST_GPU', - 'nvidia.com/gpu.memory': '81920', - }, - 'extraField': 'keep-me', - }, - 'gpu-tp2-balanced': { - 'nodeSelector': { - 'nvidia.com/gpu.product': 'NVIDIA_TEST_GPU', - 'nvidia.com/gpu.memory': '81920', - } - }, - 'gpu-tp2-maxctx': { - 'nodeSelector': { - 'nvidia.com/gpu.product': 'NVIDIA_TEST_GPU', - 'nvidia.com/gpu.memory': '81920', - } - }, - } - }, - sort_keys=False, - ), - encoding='utf-8', - ) - - -def test_setup_compose_then_render_without_manual_file_edits( - tmp_path: Path, -) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'compose', - '--profile', - 'qwen2-5-7b-instruct-turbo-default', - ) - cfg = yaml.safe_load((tmp_path / 'config.yaml').read_text()) - assert cfg['backend'] == 'compose' - assert cfg['active_profile'] == 'qwen2-5-7b-instruct-turbo-default' - run_cli(tmp_path, 'render', '--yes', '--simulate-hardware', '1x96') - assert (tmp_path / 'generated' / 'docker-compose.yml').exists() - - -def test_setup_kubeai_then_render_without_manual_file_edits( - tmp_path: Path, -) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'kubeai', - '--profile', - 'qwen2-72b-instruct-tp2-balanced', - '--namespace', - 'demo-llm', - '--ingress', - '--ingress-host', - 'llm.example.test', - ) - cfg = yaml.safe_load((tmp_path / 'config.yaml').read_text()) - assert cfg['backend'] == 'kubeai' - assert cfg['cluster']['namespace'] == 'demo-llm' - assert cfg['cluster']['ingress']['enabled'] is True - run_cli(tmp_path, 'render', '--yes', '--simulate-hardware', '2x96') - assert (tmp_path / 'generated' / 'kubeai' / 'models.yaml').exists() - assert (tmp_path / 'generated' / 'kubeai' / 'ingress.yaml').exists() - - -def test_setup_kubeai_with_resource_profiles_file_syncs_canonical_values( - tmp_path: Path, -) -> None: - source = tmp_path / 'values-kubeai-local-gpu.yaml' - write_kubeai_values_file(source) - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'kubeai', - '--profile', - 'qwen2-5-7b-instruct-turbo-default', - '--namespace', - 'kubeai', - '--resource-profiles-file', - str(source), - ) - values_doc = yaml.safe_load( - (tmp_path / 'kubeai-values.local.yaml').read_text() - ) - assert 'gpu-single-default' in values_doc['resourceProfiles'] - assert ( - values_doc['resourceProfiles']['gpu-single-default']['extraField'] - == 'keep-me' - ) - - -def test_render_overrides_backend_and_profile_without_persisting_config( - tmp_path: Path, -) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'compose', - '--profile', - 'gpt-oss-20b-chat', - ) - run_cli( - tmp_path, - 'render', - '--yes', - '--backend', - 'kubeai', - '--profile', - 'qwen2-72b-instruct-tp2-balanced', - '--namespace', - 'override-ns', - '--simulate-hardware', - '2x96', - ) - cfg = yaml.safe_load((tmp_path / 'config.yaml').read_text()) - assert cfg['backend'] == 'compose' - assert cfg['active_profile'] == 'gpt-oss-20b-chat' - models = list( - yaml.safe_load_all( - (tmp_path / 'generated' / 'kubeai' / 'models.yaml').read_text() - ) - ) - namespace_doc = yaml.safe_load( - (tmp_path / 'generated' / 'kubeai' / 'namespace.yaml').read_text() - ) - assert models[0]['metadata']['name'] == 'qwen2-72b-instruct-tp2-balanced' - assert namespace_doc['metadata']['name'] == 'override-ns' - - -def test_setup_supports_environment_fallbacks(tmp_path: Path) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - env={ - 'INFER_STACK_BACKEND': 'kubeai', - 'INFER_STACK_PROFILE': 'gpt-oss-20b-chat', - 'INFER_STACK_NAMESPACE': 'env-ns', - 'INFER_STACK_INGRESS_ENABLED': 'true', - 'INFER_STACK_INGRESS_HOST': 'env.example.test', - }, - ) - cfg = yaml.safe_load((tmp_path / 'config.yaml').read_text()) - assert cfg['backend'] == 'kubeai' - assert cfg['active_profile'] == 'gpt-oss-20b-chat' - assert cfg['cluster']['namespace'] == 'env-ns' - assert cfg['cluster']['ingress']['enabled'] is True - assert cfg['cluster']['ingress']['host'] == 'env.example.test' - - -def test_kubeai_sync_resource_profiles_then_validate_without_config_duplication( - tmp_path: Path, -) -> None: - source = tmp_path / 'values-kubeai-local-gpu.yaml' - write_kubeai_values_file(source) - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'kubeai', - '--profile', - 'qwen2-5-7b-instruct-turbo-default', - '--namespace', - 'kubeai', - ) - cfg = yaml.safe_load((tmp_path / 'config.yaml').read_text()) - cfg.pop('resource_profiles', None) - (tmp_path / 'config.yaml').write_text( - yaml.safe_dump(cfg, sort_keys=False), encoding='utf-8' - ) - run_cli( - tmp_path, 'kubeai-sync-resource-profiles', '--from-file', str(source) - ) - run_cli(tmp_path, 'validate', '--simulate-hardware', '1x96') - - -def test_kubeai_config_fallback_still_works_before_and_after_render_without_sync( - tmp_path: Path, -) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'kubeai', - '--profile', - 'qwen2-5-7b-instruct-turbo-default', - '--namespace', - 'kubeai', - ) - run_cli(tmp_path, 'validate', '--simulate-hardware', '1x96') - run_cli(tmp_path, 'render', '--yes', '--simulate-hardware', '1x96') - assert not (tmp_path / 'kubeai-values.local.yaml').exists() - cfg = yaml.safe_load((tmp_path / 'config.yaml').read_text()) - cfg['resource_profiles']['gpu-single-default']['node_selector'] = { - 'example.com/profile-source': 'config-after-render' - } - (tmp_path / 'config.yaml').write_text( - yaml.safe_dump(cfg, sort_keys=False), encoding='utf-8' - ) - run_cli(tmp_path, 'validate', '--simulate-hardware', '1x96') - run_cli(tmp_path, 'render', '--yes', '--simulate-hardware', '1x96') - rendered_values = yaml.safe_load( - (tmp_path / 'generated' / 'kubeai' / 'kubeai-values.yaml').read_text() - ) - assert ( - rendered_values['resourceProfiles']['gpu-single-default'][ - 'nodeSelector' - ]['example.com/profile-source'] - == 'config-after-render' - ) - - -def test_kubeai_synced_source_is_preferred_over_config_and_preserves_unknown_fields( - tmp_path: Path, -) -> None: - source = tmp_path / 'values-kubeai-local-gpu.yaml' - write_kubeai_values_file(source) - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'kubeai', - '--profile', - 'qwen2-5-7b-instruct-turbo-default', - '--namespace', - 'kubeai', - ) - cfg = yaml.safe_load((tmp_path / 'config.yaml').read_text()) - cfg['resource_profiles']['gpu-single-default']['node_selector'] = { - 'example.com/profile-source': 'config' - } - (tmp_path / 'config.yaml').write_text( - yaml.safe_dump(cfg, sort_keys=False), encoding='utf-8' - ) - run_cli( - tmp_path, 'kubeai-sync-resource-profiles', '--from-file', str(source) - ) - run_cli(tmp_path, 'render', '--yes', '--simulate-hardware', '1x96') - generated_values = yaml.safe_load( - (tmp_path / 'generated' / 'kubeai' / 'kubeai-values.yaml').read_text() - ) - assert ( - generated_values['resourceProfiles']['gpu-single-default'][ - 'nodeSelector' - ]['nvidia.com/gpu.product'] - == 'NVIDIA_TEST_GPU' - ) - assert ( - generated_values['resourceProfiles']['gpu-single-default']['extraField'] - == 'keep-me' - ) - assert 'example.com/profile-source' not in generated_values[ - 'resourceProfiles' - ]['gpu-single-default'].get('nodeSelector', {}) - - -def test_kubeai_local_values_change_marks_render_stale( - tmp_path: Path, monkeypatch -) -> None: - source = tmp_path / 'values-kubeai-local-gpu.yaml' - write_kubeai_values_file(source) - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'kubeai', - '--profile', - 'qwen2-5-7b-instruct-turbo-default', - '--namespace', - 'kubeai', - '--resource-profiles-file', - str(source), - ) - run_cli(tmp_path, 'render', '--yes', '--simulate-hardware', '1x96') - _anchor_paths(tmp_path, monkeypatch) - assert cli_mod.render_is_stale() is False - time.sleep(1.1) - values_doc = yaml.safe_load( - (tmp_path / 'kubeai-values.local.yaml').read_text() - ) - values_doc['resourceProfiles']['gpu-single-default']['nodeSelector'][ - 'example.com/profile-source' - ] = 'updated-local' - (tmp_path / 'kubeai-values.local.yaml').write_text( - yaml.safe_dump(values_doc, sort_keys=False), encoding='utf-8' - ) - assert cli_mod.render_is_stale() is True - - -def test_kubeai_deploy_rerenders_from_updated_local_values( - tmp_path: Path, monkeypatch -) -> None: - source = tmp_path / 'values-kubeai-local-gpu.yaml' - write_kubeai_values_file(source) - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'kubeai', - '--profile', - 'qwen2-5-7b-instruct-turbo-default', - '--namespace', - 'kubeai', - '--resource-profiles-file', - str(source), - ) - run_cli(tmp_path, 'render', '--yes', '--simulate-hardware', '1x96') - _anchor_paths(tmp_path, monkeypatch) - - values_doc = yaml.safe_load( - (tmp_path / 'kubeai-values.local.yaml').read_text() - ) - time.sleep(1.1) - values_doc['resourceProfiles']['gpu-single-default']['nodeSelector'][ - 'example.com/profile-source' - ] = 'deploy-rerender' - (tmp_path / 'kubeai-values.local.yaml').write_text( - yaml.safe_dump(values_doc, sort_keys=False), encoding='utf-8' - ) - - observed: dict[str, object] = {} - - def fake_deploy_rendered_artifacts(deployment: dict) -> None: - observed['source'] = deployment['resource_profiles_source'] - - monkeypatch.setattr( - cli_mod.commands_runtime, - 'deploy_rendered_artifacts', - fake_deploy_rendered_artifacts, - ) - cli_mod.DeployCLI.main( - argv=False, - detach=False, - simulate_hardware='1x96', - yes=True, - ) - generated_values = yaml.safe_load( - (tmp_path / 'generated' / 'kubeai' / 'kubeai-values.yaml').read_text() - ) - assert observed['source'] == str((tmp_path / 'kubeai-values.local.yaml')) - assert ( - generated_values['resourceProfiles']['gpu-single-default'][ - 'nodeSelector' - ]['example.com/profile-source'] - == 'deploy-rerender' - ) - - -def test_switch_apply_persists_only_active_profile_and_uses_transient_overrides( - tmp_path: Path, monkeypatch -) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'compose', - '--profile', - 'gpt-oss-20b-chat', - '--compose-cmd', - 'docker compose', - ) - _anchor_paths(tmp_path, monkeypatch) - observed: dict[str, object] = {} - - def fake_deploy_rendered_artifacts(deployment: dict) -> None: - observed['backend'] = deployment['backend'] - observed['namespace'] = deployment['cluster']['namespace'] - observed['profile'] = deployment['serving_profile']['public_name'] - - monkeypatch.setattr( - cli_mod.commands_profile, - 'deploy_rendered_artifacts', - fake_deploy_rendered_artifacts, - ) - cli_mod.SwitchCLI.main( - argv=False, - profile='qwen2-5-7b-instruct-turbo-default', - backend='kubeai', - compose_cmd='podman compose', - namespace='override-ns', - apply=True, - simulate_hardware='1x96', - yes=True, - ) - - cfg = yaml.safe_load((tmp_path / 'config.yaml').read_text()) - assert cfg['active_profile'] == 'qwen2-5-7b-instruct-turbo-default' - assert cfg['backend'] == 'compose' - assert cfg['runtime']['compose_cmd'] == 'docker compose' - assert cfg['cluster']['namespace'] != 'override-ns' - - assert observed['backend'] == 'kubeai' - assert observed['namespace'] == 'override-ns' - assert observed['profile'] == 'qwen2-5-7b-instruct-turbo-default' - - -def test_switch_apply_compose_recreates_router_when_config_changes( - tmp_path: Path, monkeypatch -) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'compose', - '--profile', - 'gpt-oss-20b-chat', - ) - run_cli(tmp_path, 'render', '--yes', '--simulate-hardware', '1x96') - _anchor_paths(tmp_path, monkeypatch) - - invocations: list[list[str]] = [] - - def fake_run(cmd: list[str]) -> None: - invocations.append(list(cmd)) - - import infer_stack.docker_utils as du - - monkeypatch.setattr(du, 'run', fake_run) - - cli_mod.SwitchCLI.main( - argv=False, - profile='qwen2-5-7b-instruct-turbo-default', - apply=True, - simulate_hardware='1x96', - yes=True, - ) - - flat = [' '.join(c) for c in invocations] - assert not any('down -v' in c or '--volumes' in c for c in flat) - assert any(' up ' in c and '-d' in c.split() for c in flat) - # When live router refresh isn't reachable (no real LiteLLM under test), - # we fall back to force-recreating just the litellm container — open-webui - # is deliberately left running so chat sessions don't drop on a switch. - recreate_cmds = [c for c in flat if '--force-recreate' in c] - assert recreate_cmds, 'expected a force-recreate fallback to fire' - assert any('litellm' in c for c in recreate_cmds) - assert all('open-webui' not in c for c in recreate_cmds), ( - 'open-webui should not be force-recreated during a profile switch' - ) - - -def test_openwebui_state_paths_are_not_deleted_or_reinitialized_on_switch( - tmp_path: Path, monkeypatch -) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'compose', - '--profile', - 'gpt-oss-20b-chat', - ) - run_cli(tmp_path, 'render', '--yes', '--simulate-hardware', '1x96') - _anchor_paths(tmp_path, monkeypatch) - - state_root = tmp_path / 'state' - sentinel_open_webui = state_root / 'open-webui' / 'sentinel.txt' - sentinel_postgres_owui = state_root / 'postgres-open-webui' / 'sentinel.txt' - sentinel_postgres_litellm = state_root / 'postgres-litellm' / 'sentinel.txt' - for sentinel in ( - sentinel_open_webui, - sentinel_postgres_owui, - sentinel_postgres_litellm, - ): - sentinel.parent.mkdir(parents=True, exist_ok=True) - sentinel.write_text('keep', encoding='utf-8') - - invocations: list[list[str]] = [] - - def fake_run(cmd: list[str]) -> None: - invocations.append(list(cmd)) - - import infer_stack.docker_utils as du - - monkeypatch.setattr(du, 'run', fake_run) - - cli_mod.SwitchCLI.main( - argv=False, - profile='qwen2-5-7b-instruct-turbo-default', - apply=True, - simulate_hardware='1x96', - yes=True, - ) - - flat = [' '.join(c) for c in invocations] - assert not any('down -v' in c or '--volumes' in c for c in flat) - assert not any('rm -rf' in c for c in flat) - for sentinel in ( - sentinel_open_webui, - sentinel_postgres_owui, - sentinel_postgres_litellm, - ): - assert sentinel.exists() and sentinel.read_text() == 'keep' - - -def test_kubeai_status_namespace_error_is_actionable( - tmp_path: Path, monkeypatch -) -> None: - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'kubeai', - '--profile', - 'qwen2-5-7b-instruct-turbo-default', - '--namespace', - 'default', - ) - _anchor_paths(tmp_path, monkeypatch) - - def fake_status(namespace: str) -> None: - raise cli_mod.CommandError(f'Command failed in namespace {namespace}') - - monkeypatch.setattr( - cli_mod.commands_runtime, 'kubeai_print_status', fake_status - ) - try: - cli_mod.StatusCLI.main(argv=False) - except SystemExit as ex: - text = str(ex) - else: - raise AssertionError('expected StatusCLI.main to raise SystemExit') - assert "namespace 'default'" in text - assert 'setup --backend kubeai --namespace default' in text - - -def test_status_reports_initialization_and_profile(tmp_path: Path) -> None: - # Before setup: status reports it is uninitialized and exits cleanly - # (no Docker / network / hardware probe required). - pre = run_cli(tmp_path, 'status') - assert 'Not initialized' in pre.stdout - - # After setup: status reports the active profile and that it is not yet - # rendered, still without touching Docker. - run_cli( - tmp_path, - 'setup', - '--yes', - '--backend', - 'compose', - '--profile', - 'smollm2-135m-single', - ) - post = run_cli(tmp_path, 'status') - assert 'Not initialized' not in post.stdout - assert 'active profile: smollm2-135m-single' in post.stdout - assert 'rendered: no' in post.stdout - - -class _FakeResponse: - """Minimal stand-in matching what ``_ready_openai_probe`` reads from a response.""" - - def __init__( - self, - payload: dict[str, object], - *, - status_code: int = 200, - text: str = '', - ) -> None: - self._payload = payload - self.status_code = status_code - self.text = text - - def json(self) -> dict[str, object]: - return self._payload - - -def _smoke_test_kwargs(**overrides) -> dict: - base = dict( - base_url='http://127.0.0.1:14042/v1', - prompt='Say hello in one sentence.', - max_tokens=64, - simulate_hardware='1x96', - ) - base.update(overrides) - return base - - -def _setup_compose(tmp_path: Path, profile: str) -> None: - run_cli( - tmp_path, 'setup', '--yes', '--backend', 'compose', '--profile', profile - ) - run_cli(tmp_path, 'render', '--yes', '--simulate-hardware', '1x96') - - -def test_smoke_test_uses_chat_endpoint_for_chat_profiles( - tmp_path: Path, monkeypatch -) -> None: - _setup_compose(tmp_path, 'qwen2-5-7b-instruct-turbo-default') - _anchor_paths(tmp_path, monkeypatch) - posted: dict[str, Any] = {} - - def fake_get(url, **kwargs): - return _FakeResponse( - {'data': [{'id': 'qwen2-5-7b-instruct-turbo-default'}]} - ) - - def fake_post(url, headers=None, json=None, timeout=None): - posted['url'] = url - posted['json'] = json - return _FakeResponse({'choices': [{'message': {'content': 'hi'}}]}) - - monkeypatch.setattr(cli_mod.requests, 'get', fake_get) - monkeypatch.setattr(cli_mod.requests, 'post', fake_post) - cli_mod.SmokeTestCLI.main( - argv=False, - **_smoke_test_kwargs(model='qwen2-5-7b-instruct-turbo-default'), - ) - assert posted['url'].endswith('/chat/completions') - assert 'messages' in posted['json'] - assert 'prompt' not in posted['json'] - - -def test_smoke_test_uses_completions_endpoint_for_completions_profiles( - tmp_path: Path, monkeypatch -) -> None: - _setup_compose(tmp_path, 'helm-pythia-1b-v0') - _anchor_paths(tmp_path, monkeypatch) - posted: dict[str, Any] = {} - - def fake_get(url, **kwargs): - return _FakeResponse({'data': [{'id': 'eleutherai/pythia-1b-v0'}]}) - - def fake_post(url, headers=None, json=None, timeout=None): - posted['url'] = url - posted['json'] = json - return _FakeResponse({'choices': [{'text': 'hi'}]}) - - monkeypatch.setattr(cli_mod.requests, 'get', fake_get) - monkeypatch.setattr(cli_mod.requests, 'post', fake_post) - cli_mod.SmokeTestCLI.main( - argv=False, **_smoke_test_kwargs(model='eleutherai/pythia-1b-v0') - ) - assert posted['url'].endswith('/completions') - assert not posted['url'].endswith('/chat/completions') - assert 'prompt' in posted['json'] - assert 'messages' not in posted['json'] - - -def test_smoke_test_protocol_override_forces_completions( - tmp_path: Path, monkeypatch -) -> None: - _setup_compose(tmp_path, 'qwen2-5-7b-instruct-turbo-default') - _anchor_paths(tmp_path, monkeypatch) - posted: dict[str, Any] = {} - - def fake_get(url, **kwargs): - return _FakeResponse( - {'data': [{'id': 'qwen2-5-7b-instruct-turbo-default'}]} - ) - - def fake_post(url, headers=None, json=None, timeout=None): - posted['url'] = url - posted['json'] = json - return _FakeResponse({'choices': [{'text': 'hi'}]}) - - monkeypatch.setattr(cli_mod.requests, 'get', fake_get) - monkeypatch.setattr(cli_mod.requests, 'post', fake_post) - cli_mod.SmokeTestCLI.main( - argv=False, - **_smoke_test_kwargs( - model='qwen2-5-7b-instruct-turbo-default', protocol='completions' - ), - ) - assert posted['url'].endswith('/completions') - assert 'prompt' in posted['json'] - - -def test_setup_data_dir_relocates_state_and_preserves_other_config( - tmp_path: Path, monkeypatch -) -> None: - """``setup --data-dir`` against an existing config moves the managed data - tree to the new root while leaving every other field untouched.""" - old_root = tmp_path / 'old-data' - new_root = tmp_path / 'new-data' - monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) - monkeypatch.setenv('INFER_STACK_DATA_DIR', str(old_root)) - - cli_mod.SetupCLI.main( - argv=[ - '--backend', - 'compose', - '--profile', - 'qwen2-5-7b-instruct-turbo-default', - '--yes', - ] - ) - cfg = yaml.safe_load((tmp_path / 'config.yaml').read_text()) - assert cfg['state']['hf_cache'] == str(old_root / 'hf-cache') - - cli_mod.SetupCLI.main(argv=['--data-dir', str(new_root), '--yes']) - cfg2 = yaml.safe_load((tmp_path / 'config.yaml').read_text()) - # The data tree moved... - assert cfg2['state']['hf_cache'] == str(new_root / 'hf-cache') - assert cfg2['output']['generated_dir'] == str(new_root / 'generated') - # ...but nothing else did. - assert cfg2['backend'] == 'compose' - assert cfg2['active_profile'] == 'qwen2-5-7b-instruct-turbo-default' - - -def test_setup_declined_diff_writes_nothing( - tmp_path: Path, monkeypatch -) -> None: - """Without ``--yes``, declining the diff prompt leaves no files behind.""" - monkeypatch.setenv('INFER_STACK_CONFIG_DIR', str(tmp_path)) - monkeypatch.setenv('INFER_STACK_DATA_DIR', str(tmp_path)) - # Simulate the user answering "no" at the confirmation prompt. - monkeypatch.setattr( - 'infer_stack.diff_prompt.Confirm.ask', lambda *a, **k: False - ) - - raised = False - try: - cli_mod.SetupCLI.main(argv=['--backend', 'compose']) - except SystemExit: - raised = True - assert raised, 'declining the prompt should abort via SystemExit' - assert not (tmp_path / 'config.yaml').exists() - assert not (tmp_path / 'models.yaml').exists() diff --git a/tests/test_leasing_catalog.py b/tests/test_leasing_catalog.py new file mode 100644 index 00000000..9216d4be --- /dev/null +++ b/tests/test_leasing_catalog.py @@ -0,0 +1,194 @@ +"""Tests for the serving catalog parser and its handoff to the ledger.""" + +from __future__ import annotations + +import pytest + +from infer_stack.leasing import ( + Catalog, + CatalogError, + Ledger, + Sharing, + SqliteStore, +) + +SAMPLE = { + 'models': { + 'qwen-coder-32b': { + 'source': 'hf://Qwen/Qwen2.5-Coder-32B-Instruct', + 'revision': 'main', + }, + 'llama-small': {'source': 'hf://meta-llama/Llama-3.2-3B-Instruct'}, + }, + 'endpoints': { + 'qwen-coder': { + 'model': 'qwen-coder-32b', + 'engine': 'vllm', + 'runtime': {'tensor_parallel_size': 1, 'max_model_len': 32768}, + 'sharing': {'mode': 'shared-compatible'}, + 'reclaim': {'policy': 'keep-warm'}, + }, + # an alias of the same model + runtime: should coalesce with qwen-coder + 'qwen-coder-alias': { + 'model': 'qwen-coder-32b', + 'engine': 'vllm', + 'runtime': {'tensor_parallel_size': 1, 'max_model_len': 32768}, + 'public_name': 'qwen-coder', + }, + 'draft-model': {'model': 'llama-small', 'engine': 'vllm'}, + 'verifier-model': { + 'model': 'qwen-coder-32b', + 'engine': 'vllm', + 'runtime': {'tensor_parallel_size': 2}, + }, + 'qwen-small': { + 'engine': 'ollama', + 'host': 'local-ollama', + 'model': 'qwen3.5:4b', + }, + 'smollm': { + 'engine': 'ollama', + 'host': 'local-ollama', + 'model': 'smollm2:135m', + }, + }, + 'runtime_hosts': { + 'local-ollama': { + 'engine': 'ollama', + 'placement': {'gpu_indices': [1]}, + 'settings': {'keep_alive': '2m', 'max_loaded_models': 2}, + 'storage': {'model_store': 'shared-ollama-store'}, + }, + }, + 'bundles': { + 'draft-and-verify': ['draft-model', 'verifier-model'], + 'local-small-models': ['qwen-small', 'smollm'], + }, +} + + +@pytest.fixture +def catalog(): + return Catalog.from_dict(SAMPLE) + + +def test_resolve_vllm_endpoint(catalog): + req = catalog.resolve_endpoint('qwen-coder') + assert req.engine == 'vllm' + assert req.capacity == {'max_model_len': 32768} + assert req.sharing == Sharing.SHARED + assert req.served['served_model_name'] == 'qwen-coder' + assert req.spec['hf_model_id'] == 'Qwen/Qwen2.5-Coder-32B-Instruct' + + +def test_resolve_ollama_endpoint(catalog): + req = catalog.resolve_endpoint('qwen-small') + assert req.engine == 'ollama' + assert req.host == 'local-ollama' + assert req.served == {'model': 'qwen3.5:4b'} + assert req.capacity == {} + assert req.structural['gpu_indices'] == [1] + + +def test_alias_shares_compat_key(catalog): + a = catalog.resolve_endpoint('qwen-coder') + b = catalog.resolve_endpoint('qwen-coder-alias') + # different endpoint names, same model+runtime -> same deployment identity + assert a.endpoint != b.endpoint + assert a.compat_key == b.compat_key + + +def test_runtime_difference_splits_compat_key(catalog): + a = catalog.resolve_endpoint('qwen-coder') # tp=1 + b = catalog.resolve_endpoint('verifier-model') # tp=2, same model + assert a.compat_key != b.compat_key + + +def test_resolve_model_name_points_at_its_endpoints(catalog): + # passing a *model* name (a common slip) lists the endpoints that run it + with pytest.raises(CatalogError) as exc: + catalog.resolve_endpoint('qwen-coder-32b') + msg = str(exc.value) + assert 'is a model, not an endpoint' in msg + # all three endpoints on that model are suggested + for ep in ('qwen-coder', 'qwen-coder-alias', 'verifier-model'): + assert ep in msg + + +def test_resolve_model_without_endpoints_suggests_adding_one(): + cat = Catalog.from_dict({'models': {'solo': {'source': 'hf://x/y'}}, + 'endpoints': {}}) + with pytest.raises(CatalogError) as exc: + cat.resolve_endpoint('solo') + assert 'no endpoints yet' in str(exc.value) + assert 'catalog endpoint add --model solo' in str(exc.value) + + +def test_resolve_unknown_name_did_you_mean(catalog): + with pytest.raises(CatalogError) as exc: + catalog.resolve_endpoint('qwen-codr') # typo of qwen-coder + assert 'did you mean' in str(exc.value) + assert 'qwen-coder' in str(exc.value) + + +def test_sharing_override(catalog): + req = catalog.resolve_endpoint('qwen-coder', sharing=Sharing.DEDICATED) + assert req.sharing == Sharing.DEDICATED + + +def test_resolve_names_expands_bundles_and_dedups(catalog): + reqs = catalog.resolve_names(['draft-and-verify', 'draft-model']) + names = [r.endpoint for r in reqs] + assert names == ['draft-model', 'verifier-model'] # dedup, order kept + + +@pytest.mark.parametrize( + 'mutation, needle', + [ + ({'endpoints': {'x': {'engine': 'vllm', 'model': 'nope'}}}, 'unknown model'), + ({'endpoints': {'x': {'engine': 'ollama', 'model': 't', 'host': 'no'}}}, 'unknown host'), + ({'endpoints': {'x': {'engine': 'warp', 'model': 'm'}}, 'models': {'m': {'source': 's'}}}, 'unknown engine'), + ({'endpoints': {'x': {'engine': 'vllm'}}, 'models': {}}, "needs a 'model'"), + ({'bundles': {'b': ['ghost']}}, 'unknown endpoint'), + ], +) +def test_validation_errors(mutation, needle): + with pytest.raises(CatalogError) as exc: + Catalog.from_dict(mutation) + assert needle in str(exc.value) + + +def test_load_from_yaml(tmp_path): + import yaml + + path = tmp_path / 'catalog.yaml' + path.write_text(yaml.safe_dump(SAMPLE)) + catalog = Catalog.load(path) + assert set(catalog.endpoints) >= {'qwen-coder', 'qwen-small'} + + +# -- integration with the ledger ------------------------------------------ + + +def test_catalog_to_ledger_coalesces(catalog): + ledger = Ledger(SqliteStore(':memory:')) + a = ledger.acquire('alice', catalog.resolve_names(['qwen-coder'])) + b = ledger.acquire('bob', catalog.resolve_names(['qwen-coder-alias'])) + # alias resolves to the same deployment identity -> one deployment, demand 2 + assert a.deployments[0].id == b.deployments[0].id + assert ledger.get_deployment(a.deployments[0].id).demand == 2 + + +def test_catalog_ollama_bundle_one_daemon(catalog): + ledger = Ledger(SqliteStore(':memory:')) + res = ledger.acquire('alice', catalog.resolve_names(['local-small-models'])) + # two tags, one daemon -> a single deployment serving both endpoints + assert len(res.deployments) == 1 + deployment = ledger.get_deployment(res.deployments[0].id) + assert set(deployment.served) == {'qwen-small', 'smollm'} + + +def test_catalog_bundle_distinct_models(catalog): + ledger = Ledger(SqliteStore(':memory:')) + res = ledger.acquire('alice', catalog.resolve_names(['draft-and-verify'])) + assert len({g.id for g in res.deployments}) == 2 diff --git a/tests/test_leasing_compose.py b/tests/test_leasing_compose.py new file mode 100644 index 00000000..3f112124 --- /dev/null +++ b/tests/test_leasing_compose.py @@ -0,0 +1,938 @@ +"""Tests for the Compose backend, driven by a stateful fake docker seam. + +These exercise render + converge + observe + teardown logic without real docker +or GPUs. The real docker/GPU path is validated separately on a GPU host. +""" + +from __future__ import annotations + +import json +import shutil +import subprocess +from pathlib import Path + +import pytest +import yaml + +from infer_stack.hardware import simulate_inventory +from infer_stack.leasing import ( + Catalog, + ComposeBackend, + Controller, + Ledger, + SqliteStore, + plan_placement, + render_compose, +) +from infer_stack.leasing.compose import vllm_service_name +from infer_stack.leasing.models import Deployment, DeploymentState + +STATE = {'hf_cache': '/cache/hf', 'ollama': '/cache/ollama'} +IMAGES = { + 'vllm': 'vllm/vllm-openai:test', + 'ollama': 'ollama/ollama:test', + 'litellm': 'ghcr.io/berriai/litellm:test', +} +PORTS = {'ollama': 11434} + + +def vllm(gid, *, hf='org/model', served=None, tp=1, max_len=32768, reclaim='keep-warm', t=0.0): + # endpoint (public alias) is the deployment id; served_model_name is the upstream + served_name = served or gid + return Deployment( + gid, 'ck-' + gid, 'vllm', 'shared-compatible', {}, + { + 'engine': 'vllm', + 'hf_model_id': hf, + 'served_model_name': served_name, + 'runtime': {'tensor_parallel_size': tp, 'max_model_len': max_len}, + 'reclaim': reclaim, + }, + {gid: {'served_model_name': served_name}}, + DeploymentState.LIVE, t, t, + ) + + +def ollama(gid, *, tag='m:1b', t=0.0): + return Deployment( + gid, 'ck-' + gid, 'ollama', 'shared-compatible', {}, + {'engine': 'ollama', 'gpu_indices': [], 'settings': {'keep_alive': '2m'}}, + {gid: {'model': tag}}, DeploymentState.LIVE, t, t, + ) + + +class FakeDocker: + """Stateful docker compose stand-in: `up` reflects the compose file.""" + + def __init__(self): + self.running: list[str] = [] + self.calls: list[list[str]] = [] + + def __call__(self, args: list[str]) -> str: + self.calls.append(args) + compose_file = args[args.index('-f') + 1] if '-f' in args else None + if 'up' in args: + data = yaml.safe_load(Path(compose_file).read_text()) or {} + self.running = sorted((data.get('services') or {}).keys()) + return '' + if 'down' in args: + self.running = [] + return '' + if 'ps' in args: + return json.dumps( + [{'Service': s, 'State': 'running'} for s in self.running] + ) + return '' + + +class FakeResp: + def __init__(self, status, payload): + self.status_code = status + self._payload = payload + self.text = json.dumps(payload) + + def json(self): + return self._payload + + +class FakeHttp: + """Fake requests-like client whose /models lists the LiteLLM config aliases.""" + + def __init__(self, state_dir): + self.state_dir = Path(state_dir) + + def _model_names(self): + cfg = self.state_dir / 'litellm_config.yaml' + if not cfg.exists(): + return [] + data = yaml.safe_load(cfg.read_text()) or {} + return [e['model_name'] for e in data.get('model_list', [])] + + def get(self, url, **kw): + if url.endswith('/models'): + data = [{'id': n} for n in self._model_names()] + return FakeResp(200, {'data': data}) + return FakeResp(404, {'detail': 'not found'}) + + def post(self, url, **kw): + return FakeResp(200, {'choices': [{'message': {'content': 'ok'}}]}) + + +# -- pure render ----------------------------------------------------------- + + +def test_render_vllm_service(): + deployment = vllm('grp-a', hf='Qwen/Q', served='qwen', tp=2) + rc = render_compose( + [deployment], + {'grp-a': [0, 1]}, + images=IMAGES, ports=PORTS, state=STATE, + ) + # service name leads with the served model, suffixed by the deployment id + name = vllm_service_name(deployment) + assert name == 'vllm-qwen-grp-a' + svc = rc.compose['services'][name] + assert svc['image'] == 'vllm/vllm-openai:test' + assert svc['command'][0] == 'Qwen/Q' + assert '--tensor-parallel-size=2' in svc['command'] + assert '--served-model-name=qwen' in svc['command'] + assert svc['ports'] == ['18000:8000'] + devs = svc['deploy']['resources']['reservations']['devices'][0] + assert devs['device_ids'] == ['0', '1'] + # Compose schema: capabilities is a list of *strings* (not [["gpu"]]). + assert devs['capabilities'] == ['gpu'] + assert svc['labels']['infer-stack.deployment'] == 'grp-a' + assert rc.services == {name: 'grp-a'} + + +def test_render_two_vllm_distinct_ports(): + a, b = vllm('a', served='aa', t=0), vllm('b', served='bb', t=1) + rc = render_compose( + [a, b], + {'a': [0], 'b': [1]}, + images=IMAGES, ports=PORTS, state=STATE, + ) + assert rc.compose['services'][vllm_service_name(a)]['ports'] == ['18000:8000'] + assert rc.compose['services'][vllm_service_name(b)]['ports'] == ['18001:8000'] + + +def test_render_ollama_service(): + rc = render_compose( + [ollama('daemon')], {'daemon': [1]}, + images=IMAGES, ports=PORTS, state=STATE, + ) + svc = rc.compose['services']['ollama-daemon'] + assert svc['image'] == 'ollama/ollama:test' + assert svc['environment']['OLLAMA_KEEP_ALIVE'] == '2m' + assert svc['ports'] == ['11434:11434'] + # GPU pinning is via the device reservation, which renumbers the reserved + # GPU to 0 inside the container. Setting CUDA_VISIBLE_DEVICES to the *host* + # index (1) would point at a non-existent in-container device -> CPU + # fallback. So it must pin by device_ids and NOT set the host-index env var. + assert svc['deploy']['resources']['reservations']['devices'][0][ + 'device_ids' + ] == ['1'] + assert 'CUDA_VISIBLE_DEVICES' not in svc['environment'] + + +def test_render_skips_unplaced_deployments(): + a, b = vllm('a', served='aa'), vllm('b', served='bb') + rc = render_compose( + [a, b], {'a': [0]}, # b not placed + images=IMAGES, ports=PORTS, state=STATE, + ) + assert set(rc.compose['services']) == {vllm_service_name(a)} + + +def test_render_bakes_project_name(): + # a top-level `name:` makes a plain `docker compose -f file up` (no -p) land + # in the same project infer-stack uses, so the manual path == apply. + rc = render_compose( + [vllm('grp-a', served='qwen')], {'grp-a': [0]}, + images=IMAGES, ports=PORTS, state=STATE, + ) + assert rc.compose['name'] == 'infer-stack' # default project + rc2 = render_compose( + [vllm('grp-a', served='qwen')], {'grp-a': [0]}, + images=IMAGES, ports=PORTS, state=STATE, project='custom-proj', + ) + assert rc2.compose['name'] == 'custom-proj' + + +def test_vllm_service_name_is_model_led_and_dns_safe(): + import re as _re + + g = vllm('grp-098ed1', hf='Qwen/Qwen2.5-0.5B-Instruct', served='Qwen2.5-0.5B') + name = vllm_service_name(g) + # leads with the served model (slugified), suffixed by the full deployment id so + # it is unique and correlates with `infer-stack leases` + assert name == 'vllm-qwen2-5-0-5b-grp-098ed1' + # usable as a compose service *and* an on-network DNS host (LiteLLM routes + # to it), so it must be lowercase [a-z0-9-] + assert _re.fullmatch(r'[a-z0-9-]+', name) + + +# -- ComposeBackend converge/observe -------------------------------------- + + +def make_backend(tmp_path, *, spec='4x80', **kw): + return ComposeBackend( + state_dir=tmp_path, + inventory=simulate_inventory(spec), + run=FakeDocker(), + http=FakeHttp(tmp_path), + images=IMAGES, ports=PORTS, state=STATE, + **kw, + ) + + +def test_converge_writes_and_observes(tmp_path): + be = make_backend(tmp_path) + be.converge([vllm('grp-a', tp=2)]) + assert be.compose_file.exists() + sidecar = json.loads((tmp_path / 'leasing-compose-state.json').read_text()) + assert sidecar['assignments'] == {'grp-a': [0, 1]} + assert be.observe() == {'grp-a'} + + +def test_converge_is_idempotent_and_pins(tmp_path): + be = make_backend(tmp_path) + be.converge([vllm('a', tp=2, t=0)]) + be.converge([vllm('a', tp=2, t=0), vllm('b', t=1)]) # add b, keep a put + sidecar = json.loads((tmp_path / 'leasing-compose-state.json').read_text()) + assert sidecar['assignments']['a'] == [0, 1] # unchanged + assert sidecar['assignments']['b'] == [2] + assert be.observe() == {'a', 'b'} + + +def test_converge_removes_dropped_deployment(tmp_path): + be = make_backend(tmp_path) + be.converge([vllm('a', t=0), vllm('b', t=1)]) + assert be.observe() == {'a', 'b'} + be.converge([vllm('a', t=0)]) # b gone -> --remove-orphans + assert be.observe() == {'a'} + + +def test_converge_to_empty_keeps_the_front_door(tmp_path): + """Releasing the last model leaves the gateway + UI up (an empty front door). + + The front door is a standing entry point, not a per-model service: zero + models -> empty model_list, but litellm/open-webui keep running. Only + `stack down` takes everything off. + """ + be = make_backend(tmp_path) # litellm + ui on by default + be.converge([vllm('a', t=0)]) + assert be.observe() == {'a'} + fake = be.run + fake.calls.clear() + be.converge([]) # last model released + verbs = [c[c.index('-f') + 2] if '-f' in c else c[0] for c in fake.calls] + assert 'up' in verbs and 'down' not in verbs # front door stays up + # no model deployments running, but the compose project still has the gateway/UI + assert be.observe() == set() + compose = yaml.safe_load(be.compose_file.read_text()) + assert set(compose['services']) == {'litellm', 'open-webui'} + + +def test_converge_to_empty_downs_when_gateway_off(tmp_path): + """With no gateway (litellm=False), an empty desired set has nothing to run, + so converge `down`s rather than `up`-ing a services-less file.""" + be = make_backend(tmp_path, litellm=False) + be.converge([vllm('a', t=0)]) + fake = be.run + fake.calls.clear() + be.converge([]) + verbs = [c[c.index('-f') + 2] if '-f' in c else c[0] for c in fake.calls] + assert 'down' in verbs and 'up' not in verbs + + +def test_render_reverse_proxy(tmp_path): + images = {**IMAGES, 'open_webui': 'owui:test', 'nginx': 'nginx:test'} + rc = render_compose( + [vllm('grp-a', served='chat')], {'grp-a': [0]}, + images=images, ports=PORTS, state=STATE, + litellm=True, ui=True, reverse_proxy=True, reverse_proxy_port=8080, + aux_dir=tmp_path, + ) + svc = rc.compose['services']['reverse-proxy'] + assert svc['image'] == 'nginx:test' + assert svc['ports'] == ['8080:80'] + assert svc['depends_on'] == ['litellm', 'open-webui'] + # generated conf path-routes /v1 -> gateway, / -> UI + conf = rc.nginx_config + assert 'location /v1/' in conf and 'litellm:4000' in conf + assert 'location /' in conf and 'open-webui:8080' in conf + assert 'connection_upgrade' in conf # websockets for the UI + + +def test_render_reverse_proxy_byo_config(tmp_path): + images = {**IMAGES, 'open_webui': 'owui:test', 'nginx': 'nginx:test'} + rc = render_compose( + [vllm('grp-a', served='chat')], {'grp-a': [0]}, + images=images, ports=PORTS, state=STATE, + litellm=True, ui=True, reverse_proxy=True, + reverse_proxy_config='/etc/my/nginx.conf', aux_dir=tmp_path, + ) + svc = rc.compose['services']['reverse-proxy'] + # BYO: mount the operator's file verbatim, generate nothing + assert svc['volumes'] == ['/etc/my/nginx.conf:/etc/nginx/conf.d/default.conf:ro'] + assert rc.nginx_config is None + + +def test_reverse_proxy_needs_the_gateway(tmp_path): + images = {**IMAGES, 'nginx': 'nginx:test'} + rc = render_compose( + [vllm('grp-a', served='chat')], {'grp-a': [0]}, + images=images, ports=PORTS, state=STATE, + litellm=False, reverse_proxy=True, aux_dir=tmp_path, + ) + assert 'reverse-proxy' not in rc.compose['services'] # no gateway -> no proxy + + +def test_converge_writes_nginx_conf_and_access_reports_proxy(tmp_path): + be = make_backend(tmp_path, reverse_proxy=True, reverse_proxy_port=8080) + be.converge([vllm('a', served='aa')]) + assert (tmp_path / 'nginx.conf').exists() + assert 'reverse-proxy' in yaml.safe_load(be.compose_file.read_text())['services'] + info = be.access(['aa']) + assert info['proxy_url'] == 'http://127.0.0.1:8080' + + +def test_render_gateway_with_zero_models(tmp_path): + state = {**STATE, 'open_webui': '/cache/open-webui'} + images = {**IMAGES, 'open_webui': 'ghcr.io/open-webui/open-webui:test'} + rc = render_compose( + [], {}, images=images, ports=PORTS, state=state, + litellm=True, ui=True, aux_dir=tmp_path, + ) + # gateway + UI render even with no models; the model_list is just empty + assert set(rc.compose['services']) == {'litellm', 'open-webui'} + assert yaml.safe_load(rc.litellm_config)['model_list'] == [] + assert 'depends_on' not in rc.compose['services']['litellm'] + + +def test_observe_tolerates_unreadable_compose_file(tmp_path): + # A stale/invalid file makes `docker compose ps` raise; observe must not + # propagate it (else acquire bricks before converge can overwrite the file). + class RaisingPs(FakeDocker): + def __call__(self, args): + if 'ps' in args: + raise RuntimeError('compose schema error on a stale file') + return super().__call__(args) + + be = ComposeBackend( + state_dir=tmp_path, inventory=simulate_inventory('4x80'), + run=RaisingPs(), http=FakeHttp(tmp_path), + images=IMAGES, ports=PORTS, state=STATE, + ) + be.converge([vllm('a')]) # writes a good compose file + assert be.observe() == set() # ps raises -> lenient empty, no crash + + +def test_probe_ready_tracks_running(tmp_path): + be = make_backend(tmp_path) + g = vllm('a') + assert be.probe_ready(g, 'a').ready is False # nothing converged yet + be.converge([g]) + assert be.probe_ready(g, 'a').ready is True + + +def test_placement_error_skips_deployment(tmp_path): + be = make_backend(tmp_path, spec='2x80') + plan = be.converge([vllm('big', tp=4)]) # needs 4, only 2 + assert not plan.ok + assert be.last_errors + assert be.observe() == set() + + +def test_plan_is_read_only(tmp_path): + be = make_backend(tmp_path) + p = be.plan([vllm('a', served='aa'), vllm('b', served='bb')]) + assert p.assignments == {'a': [0], 'b': [1]} # placement computed + assert not be.compose_file.exists() # but nothing written + assert be.run.calls == [] # and docker never invoked + + +def test_converge_render_only_writes_without_applying(tmp_path): + be = make_backend(tmp_path) + plan = be.converge([vllm('a', served='aa')], apply=False) + assert be.compose_file.exists() # on-disk project rendered + assert plan.assignments == {'a': [0]} + assert be.last_assignments == {'a': [0]} + # neither `up` nor `down` ran — only render happened + verbs = [c[c.index('-f') + 2] for c in be.run.calls if '-f' in c] + assert 'up' not in verbs and 'down' not in verbs + # ...and applying afterwards (default) brings exactly that file up + be.converge([vllm('a', served='aa')], apply=True) + assert be.observe() == {'a'} + + +# -- controller + ledger + catalog integration ---------------------------- + + +CATALOG = { + 'models': { + 'qc': {'source': 'hf://Qwen/Qwen2.5-Coder-32B-Instruct'}, + 'rr': {'source': 'hf://BAAI/bge-reranker-base'}, + }, + 'endpoints': { + 'qwen-coder': {'engine': 'vllm', 'model': 'qc', + 'runtime': {'max_model_len': 32768}}, + 'reranker': {'engine': 'vllm', 'model': 'rr', + 'reclaim': {'policy': 'stop'}}, + }, +} + + +def test_controller_compose_end_to_end(tmp_path): + catalog = Catalog.from_dict(CATALOG) + ledger = Ledger(SqliteStore(':memory:')) + backend = make_backend(tmp_path) + ctl = Controller(ledger, backend) + + out = ctl.acquire('alice', catalog.resolve_names(['qwen-coder', 'reranker'])) + assert out.wait.ready is True + assert set(backend.observe()) == set(g.id for g in out.deployments) + # two vLLM services + the LiteLLM front door + compose = yaml.safe_load(backend.compose_file.read_text()) + deployment_services = [s for s in compose['services'] if s.startswith('vllm-')] + assert len(deployment_services) == 2 + assert 'litellm' in compose['services'] + + # release the lease: reranker is stop-policy -> torn down; qwen keep-warm stays + qwen_gid = next(g.id for g in out.deployments + if 'qwen-coder' in g.served) + ctl.release(out.lease.id) + assert backend.observe() == {qwen_gid} + + +def test_evict_tears_down_keep_warm_idle_deployment(tmp_path): + catalog = Catalog.from_dict(CATALOG) + ledger = Ledger(SqliteStore(':memory:')) + backend = make_backend(tmp_path) + ctl = Controller(ledger, backend) + + out = ctl.acquire('alice', catalog.resolve_names(['qwen-coder'])) + qwen_gid = out.deployments[0].id + ctl.release(out.lease.id) + assert backend.observe() == {qwen_gid} # keep-warm stays resident + + ev = ctl.evict(None) # force-evict idle deployments + assert qwen_gid in ev.evicted_deployment_ids + assert backend.observe() == set() # GPU freed: service gone + # ledger records it as stopped, not idle + _, deployments = ledger.status() + assert {g.state for g in deployments} == {DeploymentState.STOPPED} + + +# -- LiteLLM front door ---------------------------------------------------- + + +def test_render_litellm_front_door(tmp_path): + rc = render_compose( + [vllm('grp-a', served='qwen-served')], {'grp-a': [0]}, + images=IMAGES, ports=PORTS, state=STATE, + litellm=True, litellm_port=14042, aux_dir=tmp_path, + ) + assert 'litellm' in rc.compose['services'] + litellm = rc.compose['services']['litellm'] + assert litellm['ports'] == ['14042:4000'] + cfg = yaml.safe_load(rc.litellm_config) + entry = cfg['model_list'][0] + assert entry['model_name'] == 'grp-a' # alias = endpoint + assert entry['litellm_params']['model'] == 'openai/qwen-served' + # api_base host == the vLLM service name, so LiteLLM routes over the network + assert entry['litellm_params']['api_base'] == ( + 'http://vllm-qwen-served-grp-a:8000/v1' + ) + + +def test_litellm_config_hash_label_tracks_model_list(tmp_path): + """The litellm service must change when its routing config changes. + + Regression: a second alias coalesced onto a live deployment rewrote the config + file, but `docker compose up -d` left the old litellm container running + (spec unchanged), so the new alias never became routable. Stamping the + config hash onto a label makes converge recreate litellm on a config change. + """ + from infer_stack.leasing.compose import CONFIG_HASH_LABEL + + def label(deployment): + rc = render_compose( + [deployment], {deployment.id: [0]}, images=IMAGES, ports=PORTS, state=STATE, + litellm=True, litellm_port=14042, aux_dir=tmp_path, + ) + return rc.compose['services']['litellm']['labels'][CONFIG_HASH_LABEL] + + one = vllm('grp-a', served='qwen-served') + # same deployment serving two aliases (the coalesced case) + two = vllm('grp-a', served='qwen-served') + two.served['extra-alias'] = {'served_model_name': 'qwen-served'} + + h_one, h_two = label(one), label(two) + assert h_one and h_two + assert h_one != h_two # config changed -> spec (label) changed + assert label(one) == h_one # stable for identical input (idempotent) + + +def test_render_open_webui_default_and_stable(tmp_path): + """Open WebUI renders with the gateway and is byte-stable across switches.""" + state = {**STATE, 'open_webui': '/cache/open-webui'} + images = {**IMAGES, 'open_webui': 'ghcr.io/open-webui/open-webui:test'} + + def render(deployments, assigns): + return render_compose( + deployments, assigns, images=images, ports=PORTS, state=state, + litellm=True, ui=True, ui_port=13000, + litellm_master_key='sk-x', aux_dir=tmp_path, + ) + + one = render([vllm('grp-a', served='a')], {'grp-a': [0]}) + ow = one.compose['services']['open-webui'] + assert ow['ports'] == ['13000:8080'] + assert ow['environment']['OPENAI_API_BASE_URL'] == 'http://litellm:4000/v1' + assert ow['environment']['OPENAI_API_KEY'] == 'sk-x' + assert ow['depends_on'] == ['litellm'] + + # Adding a second model recreates litellm (routing changed) but must NOT + # touch open-webui — that is what keeps the UI from blinking on a switch. + two = render([vllm('grp-a', served='a'), vllm('grp-b', served='b', t=1.0)], + {'grp-a': [0], 'grp-b': [1]}) + assert two.compose['services']['open-webui'] == ow + assert two.compose['services']['litellm'] != one.compose['services']['litellm'] + + +def test_render_no_open_webui_when_ui_off(tmp_path): + state = {**STATE, 'open_webui': '/cache/open-webui'} + images = {**IMAGES, 'open_webui': 'ow:test'} + rc = render_compose( + [vllm('grp-a', served='a')], {'grp-a': [0]}, + images=images, ports=PORTS, state=state, + litellm=True, ui=False, aux_dir=tmp_path, + ) + assert 'open-webui' not in rc.compose['services'] + + +def _render_ui(deployments, assigns, tmp_path, **kw): + state = {**STATE, 'open_webui': '/cache/open-webui'} + images = {**IMAGES, 'open_webui': 'ow:test'} + return render_compose( + deployments, assigns, images=images, ports=PORTS, state=state, + ui=True, ui_port=13000, litellm_master_key='sk-x', aux_dir=tmp_path, + **kw, + ) + + +def test_open_webui_connects_ollama_daemon_directly(tmp_path): + """No gateway: the UI talks straight to the Ollama daemon's native API, so + you can pull/run models from the UI — a drop-in for hand-run ollama+webui.""" + rc = _render_ui([ollama('daemon')], {'daemon': [0]}, tmp_path, litellm=False) + assert 'litellm' not in rc.compose['services'] + env = rc.compose['services']['open-webui']['environment'] + assert env['ENABLE_OLLAMA_API'] == 'True' + assert env['OLLAMA_BASE_URL'] == 'http://ollama-daemon:11434' + # nothing OpenAI to point at -> that connection is off, not dangling + assert env['ENABLE_OPENAI_API'] == 'False' + assert 'OPENAI_API_BASE_URL' not in env + # the per-model upstream comes and goes, so the UI does not hard-depend on it + assert 'depends_on' not in rc.compose['services']['open-webui'] + + +def test_open_webui_enables_ollama_api_alongside_litellm(tmp_path): + """Gateway on AND an Ollama daemon: chat routes through LiteLLM (one base_url + for every alias) while the native Ollama connection still gives the UI its + pull/manage panel.""" + rc = _render_ui([ollama('daemon')], {'daemon': [0]}, tmp_path, litellm=True) + env = rc.compose['services']['open-webui']['environment'] + assert env['OPENAI_API_BASE_URL'] == 'http://litellm:4000/v1' + assert env['OPENAI_API_KEY'] == 'sk-x' + assert env['ENABLE_OLLAMA_API'] == 'True' + assert env['OLLAMA_BASE_URL'] == 'http://ollama-daemon:11434' + assert rc.compose['services']['open-webui']['depends_on'] == ['litellm'] + + +def test_open_webui_points_at_vllm_v1_without_litellm(tmp_path): + """No gateway, a single vLLM upstream: the UI's OpenAI connection points at + that process's own /v1.""" + rc = _render_ui([vllm('grp-a', served='a')], {'grp-a': [0]}, tmp_path, + litellm=False) + name = vllm_service_name(vllm('grp-a', served='a')) + env = rc.compose['services']['open-webui']['environment'] + assert env['OPENAI_API_BASE_URL'] == f'http://{name}:8000/v1' + assert env['ENABLE_OLLAMA_API'] == 'False' + + +def test_no_open_webui_without_litellm_or_upstreams(tmp_path): + """No gateway and nothing to point at -> no standing UI (an empty desired + set then has nothing to run and converge tears the project down).""" + rc = _render_ui([], {}, tmp_path, litellm=False) + assert 'open-webui' not in rc.compose['services'] + + +def test_litellm_router_settings_present(tmp_path): + rc = render_compose( + [vllm('grp-a', served='a')], {'grp-a': [0]}, + images=IMAGES, ports=PORTS, state=STATE, + litellm=True, aux_dir=tmp_path, + ) + cfg = yaml.safe_load(rc.litellm_config) + # transient upstream connection errors during warmup are retried, not 500s + assert cfg['router_settings']['num_retries'] >= 1 + + +def test_access_includes_ui_url_when_ui_on(tmp_path): + be = make_backend(tmp_path, ui=True) + assert be.access(['a'])['ui_url'] == 'http://127.0.0.1:13000' + be_noui = make_backend(tmp_path, ui=False) + assert 'ui_url' not in be_noui.access(['a']) + + +def test_converge_diff_decline_aborts(tmp_path, monkeypatch): + import infer_stack.diff_prompt as dp + from infer_stack.leasing.backend import ConvergeAborted + + monkeypatch.setattr(dp, 'confirm_writes', lambda *a, **k: False) + be = make_backend(tmp_path, assume_yes=False) + with pytest.raises(ConvergeAborted): + be.converge([vllm('grp-a', served='a')]) + # Declined -> nothing written, no `docker compose up`. + assert not be.compose_file.exists() + assert not any('up' in call for call in be.run.calls) + + +def test_converge_diff_accept_applies(tmp_path, monkeypatch): + import infer_stack.diff_prompt as dp + + monkeypatch.setattr(dp, 'confirm_writes', lambda *a, **k: True) + be = make_backend(tmp_path, assume_yes=False) + be.converge([vllm('grp-a', served='a')]) + assert be.compose_file.exists() + assert any('up' in call for call in be.run.calls) + + +def test_converge_no_change_does_not_prompt(tmp_path, monkeypatch): + import infer_stack.diff_prompt as dp + + be = make_backend(tmp_path, assume_yes=False) + monkeypatch.setattr(dp, 'confirm_writes', lambda *a, **k: True) + be.converge([vllm('grp-a', served='a')]) # first apply + + # An identical desired set renders byte-identical files -> no diff, no prompt. + seen = {'n': 0} + + def boom(*a, **k): + seen['n'] += 1 + return True + + monkeypatch.setattr(dp, 'confirm_writes', boom) + be.converge([vllm('grp-a', served='a')]) + assert seen['n'] == 0 + + +def test_acquire_rolls_back_lease_on_decline(tmp_path): + from infer_stack.leasing import EndpointRequest, LeaseState, vllm_structural + from infer_stack.leasing.backend import ConvergeAborted + + class DeclineBackend: + def observe(self): + return set() + + def converge(self, desired): + raise ConvergeAborted('declined') + + led = Ledger(SqliteStore(tmp_path / 'ledger.db')) + ctrl = Controller(led, DeclineBackend()) + with pytest.raises(ConvergeAborted): + ctrl.acquire( + 'me', [EndpointRequest('a', 'vllm', vllm_structural(model_ref='a'))] + ) + # The just-created lease must not linger as active after a decline. + leases, _ = led.status() + assert not [le for le in leases if le.state == LeaseState.ACTIVE] + + +def test_acquire_rolls_back_when_deployment_cannot_be_placed(tmp_path): + """An acquire whose deployment can't be placed must fail fast, not hang on ready. + + Regression: with the only GPU already held, `acquire ` left the new + deployment LIVE in the ledger (so `leases` showed a phantom "live" deployment) and + then blocked forever waiting on a container placement skipped. Acquire now + rolls the lease back and raises PlacementError. + """ + from infer_stack.leasing import LeaseState + from infer_stack.leasing.backend import PlacementError + + catalog = Catalog.from_dict(CATALOG) + ledger = Ledger(SqliteStore(tmp_path / 'ledger.db')) + backend = make_backend(tmp_path, spec='1x80') # exactly one GPU + ctl = Controller(ledger, backend) + + first = ctl.acquire('alice', catalog.resolve_names(['qwen-coder'])) + assert first.wait.ready # claims the only GPU + qwen_gid = first.deployments[0].id + + with pytest.raises(PlacementError) as ei: + ctl.acquire('bob', catalog.resolve_names(['reranker'])) + assert ei.value.reasons # carries the planner reason + + leases, deployments = ledger.status() + # the rolled-back request leaves no second active lease, and the reranker + # never lingers as a LIVE deployment with no container behind it + assert [le.owner for le in leases if le.state == LeaseState.ACTIVE] == ['alice'] + live_served = {ep for g in deployments + if g.state == DeploymentState.LIVE for ep in g.served} + assert 'reranker' not in live_served + assert backend.observe() == {qwen_gid} # only qwen is running + + +def test_controller_acquire_render_only_stages(tmp_path): + """`acquire --no-apply` records intent + writes on-disk state, no `up`.""" + from infer_stack.leasing import LeaseState + + catalog = Catalog.from_dict(CATALOG) + ledger = Ledger(SqliteStore(':memory:')) + backend = make_backend(tmp_path) + ctl = Controller(ledger, backend) + + out = ctl.acquire('alice', catalog.resolve_names(['qwen-coder']), apply=False) + assert out.applied is False + assert out.wait is None # never blocked on readiness + assert backend.compose_file.exists() # on-disk project written + assert backend.observe() == set() # but nothing brought up + assert out.reconcile.assignments # placement was computed + leases, _ = ledger.status() + assert [le.state for le in leases] == [LeaseState.ACTIVE] # staged intent + + # applying (the default) brings up exactly what was staged + ctl.reconcile() + assert backend.observe() == {out.deployments[0].id} + + +def test_access_reports_litellm_base_url(tmp_path): + be = make_backend(tmp_path) + info = be.access(['qwen-coder', 'reranker']) + assert info['base_url'] == 'http://127.0.0.1:14042/v1' + assert info['api_key_env'] == 'LITELLM_MASTER_KEY' + assert info['api_key'].startswith('sk-') # infer-stack manages the key + assert info['request_names'] == {'qwen-coder': 'qwen-coder', + 'reranker': 'reranker'} + + +def test_master_key_managed_stable_and_persisted(tmp_path): + be = make_backend(tmp_path) + k1 = be.master_key() + assert k1.startswith('sk-') + assert be.master_key() == k1 # reused, not regenerated + # a fresh backend over the same state dir recovers the same key + assert make_backend(tmp_path).master_key() == k1 + + +def test_converge_bakes_master_key_into_litellm(tmp_path): + be = make_backend(tmp_path) + be.converge([vllm('a')]) + compose = yaml.safe_load(be.compose_file.read_text()) + baked = compose['services']['litellm']['environment']['LITELLM_MASTER_KEY'] + assert baked == be.master_key() and baked.startswith('sk-') + + +def test_envfile_carries_managed_api_key(tmp_path): + from infer_stack.leasing.envfile import build_descriptor, render_env_file + from infer_stack.leasing.models import Lease + + be = make_backend(tmp_path) + info = be.access(['qwen-coder']) + lease = Lease('sess-x', 'me', 'active', 0.0, None, None, 0.0, + endpoints=['qwen-coder']) + g = Deployment('g', 'ck', 'vllm', 'shared-compatible', {}, {}, + {'qwen-coder': {'served_model_name': 'qwen-coder'}}, + DeploymentState.LIVE, 0.0, 0.0) + d = build_descriptor(lease, [g], base_url=info['base_url'], + api_key_env=info['api_key_env'], api_key=info['api_key'], + request_names=info['request_names']) + env = render_env_file(d) + assert 'OPENAI_API_KEY=sk-' in env # source-and-go: key is in the env-file + assert f"OPENAI_BASE_URL={info['base_url']}" in env + + +def test_access_none_without_litellm(tmp_path): + # No gateway and no UI -> no single access point. + be = make_backend(tmp_path, litellm=False, ui=False) + assert be.access(['x']) is None + + +def test_access_reports_ui_url_without_litellm(tmp_path): + # No gateway but a managed UI -> the UI is still a useful access point. + be = make_backend(tmp_path, litellm=False, ui=True) + info = be.access(['x']) + assert info == {'ui_url': 'http://127.0.0.1:13000'} + + +def test_probe_ready_requires_routable_alias(tmp_path): + be = make_backend(tmp_path) + g = vllm('a', served='a-served') + be.converge([g]) # writes litellm config listing 'a' + assert be.probe_ready(g, 'a').ready is True + # an endpoint the gateway doesn't list is not ready + assert be.probe_ready(g, 'ghost').ready is False + + +# -- Ollama pull / warmup readiness ---------------------------------------- + + +def test_ollama_probe_pulls_tag_then_ready(tmp_path): + be = make_backend(tmp_path) + g = ollama('daemon', tag='qwen3.5:4b') + be.converge([g]) + r = be.probe_ready(g, 'daemon') + assert r.ready is True + # the tag was pulled into the daemon via `docker compose exec ... ollama pull` + pulls = [c for c in be.run.calls if 'pull' in c] + assert pulls and 'qwen3.5:4b' in pulls[0] + + +def test_ollama_probe_pulls_tag_without_litellm(tmp_path): + # A lean --no-litellm Ollama stack must still pull its declared anchor tag; + # readiness for that daemon *is* the pull (no gateway to probe). + be = make_backend(tmp_path, litellm=False) + g = ollama('daemon', tag='qwen3.5:4b') + be.converge([g]) + r = be.probe_ready(g, 'daemon') + assert r.ready is True + pulls = [c for c in be.run.calls if 'pull' in c] + assert pulls and 'qwen3.5:4b' in pulls[0] + + +def test_ollama_pull_is_idempotent(tmp_path): + be = make_backend(tmp_path) + g = ollama('daemon', tag='m:1b') + be.converge([g]) + be.probe_ready(g, 'daemon') + be.probe_ready(g, 'daemon') + assert len([c for c in be.run.calls if 'pull' in c]) == 1 + + +class _FailingExecDocker(FakeDocker): + def __call__(self, args): + if 'exec' in args and 'pull' in args: + raise RuntimeError('daemon not ready') + return super().__call__(args) + + +def test_ollama_not_ready_when_pull_fails(tmp_path): + be = ComposeBackend( + state_dir=tmp_path, + inventory=simulate_inventory('4x80'), + run=_FailingExecDocker(), + http=FakeHttp(tmp_path), + images=IMAGES, ports=PORTS, state=STATE, + ) + g = ollama('daemon', tag='m:1b') + be.converge([g]) + r = be.probe_ready(g, 'daemon') + assert r.ready is False + assert 'pulling' in r.detail + + +# -- docker compose schema validation -------------------------------------- +# +# The fake-docker tests above check the dict we *build*, not whether docker +# accepts it (which is how `capabilities: [["gpu"]]` slipped through to real +# hardware). `docker compose config -q` runs Compose's schema validation +# without pulling images, starting containers, or needing a GPU, so it is fast +# and catches that whole class of bug. Skipped where docker compose is absent. + + +def _docker_compose_available() -> bool: + if shutil.which('docker') is None: + return False + try: + subprocess.run( + ['docker', 'compose', 'version'], + capture_output=True, + timeout=15, + check=True, + ) + return True + except Exception: + return False + + +requires_compose = pytest.mark.skipif( + not _docker_compose_available(), + reason='docker compose not available', +) + + +def _render(deployments, *, litellm, tmp_path): + plan = plan_placement(deployments, simulate_inventory('2x48')) + return render_compose( + deployments, plan.assignments, + images=IMAGES, ports=PORTS, state=STATE, + litellm=litellm, litellm_port=14042, aux_dir=tmp_path, + ) + + +@requires_compose +@pytest.mark.parametrize( + 'name, deployments, litellm', + [ + ('vllm-single', [vllm('grp-v', served='qwen')], True), + ('vllm-tp2', [vllm('grp-v', served='qwen', tp=2)], True), + ('vllm-and-ollama', + [vllm('grp-v', served='qwen', t=0), ollama('grp-o', tag='m:1b', t=1)], + True), + ('no-litellm', [vllm('grp-v', served='qwen')], False), + ], +) +def test_rendered_compose_passes_docker_schema(name, deployments, litellm, tmp_path): + rc = _render(deployments, litellm=litellm, tmp_path=tmp_path) + if rc.litellm_config is not None: + (tmp_path / 'litellm_config.yaml').write_text(rc.litellm_config) + compose_file = tmp_path / 'docker-compose.yml' + compose_file.write_text(yaml.safe_dump(rc.compose, sort_keys=False)) + result = subprocess.run( + ['docker', 'compose', '-p', 'infer-stack-validate', + '-f', str(compose_file), 'config', '-q'], + capture_output=True, text=True, + ) + assert result.returncode == 0, f'{name}: {result.stderr}' diff --git a/tests/test_leasing_controller.py b/tests/test_leasing_controller.py new file mode 100644 index 00000000..d5357269 --- /dev/null +++ b/tests/test_leasing_controller.py @@ -0,0 +1,208 @@ +"""Tests for the controller: reconcile + readiness against MemoryBackend.""" + +from __future__ import annotations + +from infer_stack.leasing import ( + Catalog, + Controller, + EndpointRequest, + Ledger, + MemoryBackend, + Sharing, + SqliteStore, + ollama_structural, + vllm_structural, +) + + +class FakeClock: + def __init__(self, t: float = 1000.0): + self.t = t + + def __call__(self) -> float: + return self.t + + def advance(self, dt: float) -> None: + self.t += dt + + +def _id_factory(): + counters: dict[str, int] = {} + + def factory(prefix: str) -> str: + n = counters.get(prefix, 0) + counters[prefix] = n + 1 + return f'{prefix}-{n}' + + return factory + + +def make_controller(*, ready=True, sleep=None): + clock = FakeClock() + ledger = Ledger( + SqliteStore(':memory:'), clock=clock, id_factory=_id_factory() + ) + backend = MemoryBackend(ready=ready) + ctl = Controller( + ledger, backend, sleep=sleep or (lambda dt: clock.advance(dt)) + ) + return ctl, backend, clock + + +def vreq(endpoint, *, reclaim='keep-warm', sharing=Sharing.SHARED, max_model_len=32768): + return EndpointRequest( + endpoint=endpoint, + engine='vllm', + structural=vllm_structural(model_ref=endpoint), + capacity={'max_model_len': max_model_len}, + sharing=sharing, + spec={'reclaim': reclaim}, + served={'served_model_name': endpoint}, + ) + + +def oreq(endpoint, tag, host='local-ollama'): + return EndpointRequest( + endpoint=endpoint, + engine='ollama', + structural=ollama_structural(host=host), + spec={'host': host, 'reclaim': 'keep-warm'}, + served={'model': tag}, + host=host, + ) + + +def test_acquire_realizes_and_ready(): + ctl, backend, _ = make_controller(ready=True) + out = ctl.acquire('alice', [vreq('qwen')]) + gid = out.deployments[0].id + assert backend.realize_calls == [gid] + assert backend.observe() == {gid} + assert out.wait.ready is True + + +def test_coalesced_realize_once(): + ctl, backend, _ = make_controller() + a = ctl.acquire('alice', [vreq('qwen')]) + b = ctl.acquire('bob', [vreq('qwen')]) + assert a.deployments[0].id == b.deployments[0].id + assert backend.realize_calls == [a.deployments[0].id] # realized only once + + +def test_release_one_keeps_running(): + ctl, backend, _ = make_controller() + a = ctl.acquire('alice', [vreq('qwen')]) + ctl.acquire('bob', [vreq('qwen')]) + out = ctl.release(a.lease.id) + assert out.reconcile.torn_down == [] + assert backend.teardown_calls == [] + assert backend.observe() == {a.deployments[0].id} + + +def test_release_last_keepwarm_stays(): + ctl, backend, _ = make_controller() + a = ctl.acquire('alice', [vreq('qwen', reclaim='keep-warm')]) + ctl.release(a.lease.id) + assert backend.teardown_calls == [] + assert backend.observe() == {a.deployments[0].id} # kept warm + + +def test_release_last_stop_tears_down(): + ctl, backend, _ = make_controller() + a = ctl.acquire('alice', [vreq('qwen', reclaim='stop')]) + gid = a.deployments[0].id + out = ctl.release(a.lease.id) + assert out.reconcile.torn_down == [gid] + assert backend.teardown_calls == [gid] + assert backend.observe() == set() + + +def test_wait_ready_timeout(): + ctl, backend, _ = make_controller(ready=False) + out = ctl.acquire('alice', [vreq('qwen')], timeout=10, interval=2) + assert out.wait.ready is False + assert out.wait.pending == [(out.deployments[0].id, 'qwen')] + + +def test_wait_ready_becomes_ready(): + clock = FakeClock() + ledger = Ledger( + SqliteStore(':memory:'), clock=clock, id_factory=_id_factory() + ) + backend = MemoryBackend(ready=False) + state = {'n': 0} + + def sleep(dt): + clock.advance(dt) + state['n'] += 1 + if state['n'] == 1: + backend.set_ready(True) + + ctl = Controller(ledger, backend, sleep=sleep) + out = ctl.acquire('alice', [vreq('qwen')], timeout=100, interval=2) + assert out.wait.ready is True + assert state['n'] == 1 # one poll, then ready + + +def test_ttl_expiry_reaps_stop_deployment(): + ctl, backend, clock = make_controller() + a = ctl.acquire( + 'alice', [vreq('qwen', reclaim='stop')], ttl_seconds=10, wait=False + ) + gid = a.deployments[0].id + assert backend.observe() == {gid} + clock.advance(11) + rec = ctl.reconcile() # sweep -> idle -> stop + assert gid in rec.torn_down + assert backend.observe() == set() + + +def test_ttl_expiry_keepwarm_survives(): + ctl, backend, clock = make_controller() + a = ctl.acquire( + 'alice', [vreq('qwen', reclaim='keep-warm')], ttl_seconds=10, wait=False + ) + gid = a.deployments[0].id + clock.advance(11) + rec = ctl.reconcile() + assert rec.torn_down == [] + assert backend.observe() == {gid} # keep-warm survives TTL + + +def test_wait_scoped_to_requested_endpoints(): + ctl, backend, _ = make_controller(ready=True) + a = ctl.acquire('alice', [oreq('qwen-small', 'qwen3.5:4b')]) + gid = a.deployments[0].id + # one tag on the daemon is unhealthy, but the next caller only needs the other + backend.set_ready(False, deployment_id=gid, endpoint='qwen-small') + b = ctl.acquire( + 'bob', [oreq('smollm', 'smollm2:135m')], timeout=10, interval=2 + ) + assert b.deployments[0].id == gid # same daemon + assert b.wait.ready is True # only waited for smollm + + +CATALOG = { + 'models': {'m': {'source': 'hf://org/m'}}, + 'endpoints': { + 'e1': {'engine': 'vllm', 'model': 'm', 'runtime': {'max_model_len': 8192}}, + 'e2': { + 'engine': 'vllm', + 'model': 'm', + 'runtime': {'max_model_len': 8192}, + 'public_name': 'e1', + }, + }, + 'bundles': {'both': ['e1', 'e2']}, +} + + +def test_controller_with_catalog(): + catalog = Catalog.from_dict(CATALOG) + ctl, backend, _ = make_controller(ready=True) + out = ctl.acquire('alice', catalog.resolve_names(['both'])) + # e1/e2 are the same deployment identity -> one realized deployment, both served + assert len(out.deployments) == 1 + assert backend.realize_calls == [out.deployments[0].id] + assert set(out.deployments[0].served) == {'e1', 'e2'} + assert out.wait.ready is True diff --git a/tests/test_leasing_ledger.py b/tests/test_leasing_ledger.py new file mode 100644 index 00000000..e4f977cf --- /dev/null +++ b/tests/test_leasing_ledger.py @@ -0,0 +1,250 @@ +"""Tests for the backend-agnostic lease ledger (Phase 1 core).""" + +from __future__ import annotations + +import pytest + +from infer_stack.leasing import ( + DeploymentState, + EndpointRequest, + LeaseState, + Ledger, + Sharing, + SqliteStore, + ollama_structural, + vllm_structural, +) + + +class FakeClock: + def __init__(self, t: float = 1000.0): + self.t = t + + def __call__(self) -> float: + return self.t + + def advance(self, dt: float) -> None: + self.t += dt + + +def _id_factory(): + counters: dict[str, int] = {} + + def factory(prefix: str) -> str: + n = counters.get(prefix, 0) + counters[prefix] = n + 1 + return f'{prefix}-{n}' + + return factory + + +@pytest.fixture +def ledger(): + return Ledger( + SqliteStore(':memory:'), + clock=FakeClock(), + id_factory=_id_factory(), + ) + + +def vllm_req( + endpoint, + *, + model_ref=None, + tp=1, + max_model_len=32768, + sharing=Sharing.SHARED, +): + model_ref = model_ref or endpoint + return EndpointRequest( + endpoint=endpoint, + engine='vllm', + structural=vllm_structural( + model_ref=model_ref, tensor_parallel_size=tp + ), + capacity={'max_model_len': max_model_len} if max_model_len else {}, + sharing=sharing, + spec={'hf_model_id': model_ref}, + ) + + +def ollama_req(endpoint, *, tag, host='local-ollama'): + return EndpointRequest( + endpoint=endpoint, + engine='ollama', + structural=ollama_structural(host=host), + spec={'host': host}, + served={'model': tag}, + host=host, + ) + + +def test_acquire_creates_live_deployment(ledger): + res = ledger.acquire('alice', [vllm_req('qwen-coder')]) + assert len(res.deployments) == 1 + deployment = ledger.get_deployment(res.deployments[0].id) + assert deployment.state == DeploymentState.LIVE + assert deployment.demand == 1 + assert res.lease.endpoints == ['qwen-coder'] + + +def test_same_model_coalesces(ledger): + a = ledger.acquire('alice', [vllm_req('qwen-coder')]) + b = ledger.acquire('bob', [vllm_req('qwen-coder')]) + assert a.deployments[0].id == b.deployments[0].id + assert ledger.get_deployment(a.deployments[0].id).demand == 2 + + +def test_release_decrements_then_idles(ledger): + a = ledger.acquire('alice', [vllm_req('qwen-coder')]) + b = ledger.acquire('bob', [vllm_req('qwen-coder')]) + gid = a.deployments[0].id + + r1 = ledger.release(a.lease.id) + assert r1.idled_deployment_ids == [] # bob still protects it + assert ledger.get_deployment(gid).demand == 1 + + r2 = ledger.release(b.lease.id) + assert r2.idled_deployment_ids == [gid] + assert ledger.get_deployment(gid).state == DeploymentState.IDLE + assert ledger.get_deployment(gid).demand == 0 + + +def test_double_release_is_noop(ledger): + a = ledger.acquire('alice', [vllm_req('qwen-coder')]) + ledger.release(a.lease.id) + assert ledger.release(a.lease.id).idled_deployment_ids == [] + + +def test_dedicated_does_not_coalesce(ledger): + shared = ledger.acquire('alice', [vllm_req('qwen-coder')]) + dedi = ledger.acquire( + 'bob', [vllm_req('qwen-coder', sharing=Sharing.DEDICATED)] + ) + assert shared.deployments[0].id != dedi.deployments[0].id + assert ledger.get_deployment(dedi.deployments[0].id).sharing == Sharing.DEDICATED + # both still demand 1 each + assert ledger.get_deployment(shared.deployments[0].id).demand == 1 + assert ledger.get_deployment(dedi.deployments[0].id).demand == 1 + + +def test_capacity_subsumption(ledger): + big = ledger.acquire('alice', [vllm_req('qwen-coder', max_model_len=32768)]) + # smaller request fits the bigger deployment -> coalesces + small = ledger.acquire( + 'bob', [vllm_req('qwen-coder', max_model_len=8192)] + ) + assert small.deployments[0].id == big.deployments[0].id + assert ledger.get_deployment(big.deployments[0].id).demand == 2 + + +def test_capacity_insufficient_makes_new_deployment(ledger): + small = ledger.acquire( + 'alice', [vllm_req('qwen-coder', max_model_len=8192)] + ) + # bigger request cannot be served by the smaller deployment -> new deployment + big = ledger.acquire( + 'bob', [vllm_req('qwen-coder', max_model_len=32768)] + ) + assert big.deployments[0].id != small.deployments[0].id + + +def test_structural_mismatch_separate_deployments(ledger): + a = ledger.acquire('alice', [vllm_req('qwen-coder', tp=1)]) + b = ledger.acquire('bob', [vllm_req('qwen-coder', tp=2)]) + assert a.deployments[0].id != b.deployments[0].id + + +def test_ollama_coalesces_per_daemon(ledger): + a = ledger.acquire('alice', [ollama_req('qwen-small', tag='qwen3.5:4b')]) + b = ledger.acquire('bob', [ollama_req('smollm', tag='smollm2:135m')]) + # different tags, same daemon config -> one deployment serving both endpoints + assert a.deployments[0].id == b.deployments[0].id + deployment = ledger.get_deployment(a.deployments[0].id) + assert deployment.demand == 2 + assert set(deployment.served) == {'qwen-small', 'smollm'} + + +def test_ollama_different_host_separate_deployment(ledger): + a = ledger.acquire('alice', [ollama_req('qwen-small', tag='q', host='h1')]) + b = ledger.acquire('bob', [ollama_req('qwen-small', tag='q', host='h2')]) + assert a.deployments[0].id != b.deployments[0].id + + +def test_ttl_expiry_stops_protecting(ledger): + a = ledger.acquire('alice', [vllm_req('qwen-coder')], ttl_seconds=3600) + gid = a.deployments[0].id + assert ledger.get_deployment(gid).demand == 1 + + ledger.clock.advance(3601) # past the TTL + # protection lapses immediately for demand purposes... + assert ledger.get_deployment(gid).demand == 0 + # ...and sweep() materializes the EXPIRED state + idles the deployment + res = ledger.sweep() + assert a.lease.id in res.expired_lease_ids + assert gid in res.idled_deployment_ids + assert ledger.get_lease(a.lease.id).state == LeaseState.EXPIRED + assert ledger.get_deployment(gid).state == DeploymentState.IDLE + + +def test_renew_extends_protection(ledger): + a = ledger.acquire('alice', [vllm_req('qwen-coder')], ttl_seconds=3600) + ledger.clock.advance(3000) + ledger.renew(a.lease.id, ttl_seconds=3600) + ledger.clock.advance(1000) # would have expired without renew + assert ledger.get_deployment(a.deployments[0].id).demand == 1 + assert ledger.sweep().expired_lease_ids == [] + + +def test_idle_deployment_is_reused_and_relit(ledger): + a = ledger.acquire('alice', [vllm_req('qwen-coder')]) + gid = a.deployments[0].id + ledger.release(a.lease.id) + assert ledger.get_deployment(gid).state == DeploymentState.IDLE + + b = ledger.acquire('bob', [vllm_req('qwen-coder')]) + assert b.deployments[0].id == gid # reused, not recreated + assert ledger.get_deployment(gid).state == DeploymentState.LIVE + assert ledger.get_deployment(gid).demand == 1 + + +def test_single_lease_multiple_endpoints(ledger): + res = ledger.acquire( + 'alice', [vllm_req('qwen-coder'), vllm_req('reranker')] + ) + assert len(res.deployments) == 2 + assert sorted(res.lease.endpoints) == ['qwen-coder', 'reranker'] + # one lease protects both -> each deployment demand 1 + for g in res.deployments: + assert ledger.get_deployment(g.id).demand == 1 + + +def test_reclaimable_deployments(ledger): + a = ledger.acquire('alice', [vllm_req('qwen-coder')]) + assert ledger.reclaimable_deployments() == [] + ledger.release(a.lease.id) + reclaimable = ledger.reclaimable_deployments() + assert [g.id for g in reclaimable] == [a.deployments[0].id] + + +def test_persistence_across_reopen(tmp_path): + db = tmp_path / 'ledger.db' + clock = FakeClock() + led = Ledger(SqliteStore(db), clock=clock, id_factory=_id_factory()) + a = led.acquire('alice', [vllm_req('qwen-coder')]) + led.store.close() + + led2 = Ledger(SqliteStore(db), clock=clock, id_factory=_id_factory()) + lease = led2.get_lease(a.lease.id) + assert lease is not None and lease.owner == 'alice' + deployment = led2.get_deployment(a.deployments[0].id) + assert deployment is not None and deployment.demand == 1 + + +def test_status_snapshot(ledger): + ledger.acquire('alice', [vllm_req('qwen-coder')]) + ledger.acquire('bob', [ollama_req('smollm', tag='smollm2:135m')]) + leases, deployments = ledger.status() + assert len(leases) == 2 + assert len(deployments) == 2 + assert {g.engine for g in deployments} == {'vllm', 'ollama'} diff --git a/tests/test_leasing_placement.py b/tests/test_leasing_placement.py new file mode 100644 index 00000000..7bf5868e --- /dev/null +++ b/tests/test_leasing_placement.py @@ -0,0 +1,130 @@ +"""Tests for single-host GPU placement over the union of deployments.""" + +from __future__ import annotations + +from infer_stack.hardware import simulate_inventory +from infer_stack.leasing import plan_placement +from infer_stack.leasing.models import Deployment, DeploymentState + + +def vllm(gid, *, tp=1, dp=1, gpu_indices=None, t=0.0): + runtime = {'tensor_parallel_size': tp, 'data_parallel_size': dp} + if gpu_indices is not None: + runtime['gpu_indices'] = gpu_indices + return Deployment( + gid, 'ck-' + gid, 'vllm', 'shared-compatible', {}, + {'engine': 'vllm', 'runtime': runtime}, {}, DeploymentState.LIVE, t, t, + ) + + +def ollama(gid, *, gpu_indices=(), t=0.0): + return Deployment( + gid, 'ck-' + gid, 'ollama', 'shared-compatible', {}, + {'engine': 'ollama', 'gpu_indices': list(gpu_indices)}, {}, + DeploymentState.LIVE, t, t, + ) + + +def inv(spec='4x80', *, display=()): + inventory = simulate_inventory(spec) + for i in display: + inventory['gpus'][i]['display_active'] = True + return inventory + + +def test_single_vllm_gets_gpu0(): + plan = plan_placement([vllm('a')], inv()) + assert plan.ok + assert plan.assignments == {'a': [0]} + + +def test_two_vllm_pack_sequentially(): + plan = plan_placement([vllm('a', t=0), vllm('b', t=1)], inv()) + assert plan.assignments == {'a': [0], 'b': [1]} + + +def test_tensor_parallel_spans_gpus(): + plan = plan_placement([vllm('big', tp=2, t=0), vllm('s', t=1)], inv()) + assert plan.assignments == {'big': [0, 1], 's': [2]} + + +def test_data_parallel_multiplies_count(): + plan = plan_placement([vllm('a', tp=2, dp=2)], inv()) + assert plan.assignments == {'a': [0, 1, 2, 3]} + + +def test_insufficient_gpus_errors(): + plan = plan_placement([vllm('a', tp=2), vllm('b', tp=2), vllm('c')], inv('2x80')) + assert not plan.ok + assert plan.assignments['a'] == [0, 1] + assert any('c' in e for e in plan.errors) + + +def test_allowed_gpus_preserves_real_indices(): + plan = plan_placement([vllm('a'), vllm('b', t=1)], inv(), allowed_gpus=[2, 3]) + assert plan.assignments == {'a': [2], 'b': [3]} + + +def test_reserved_gpus_are_avoided(): + plan = plan_placement([vllm('a'), vllm('b', t=1)], inv(), reserved=[0]) + assert plan.assignments == {'a': [1], 'b': [2]} + + +def test_display_gpu_used_by_default(): + # default is to use every GPU (so a single-GPU/display host works) + plan = plan_placement([vllm('a')], inv(display=[0])) + assert plan.assignments == {'a': [0]} + + +def test_display_gpu_skipped_when_opted_in(): + plan = plan_placement([vllm('a')], inv(display=[0]), skip_display=True) + assert plan.assignments == {'a': [1]} + + +def test_ollama_explicit_indices_claimed_and_avoided(): + plan = plan_placement( + [ollama('daemon', gpu_indices=[0], t=0), vllm('v', t=1)], inv() + ) + assert plan.assignments['daemon'] == [0] + assert plan.assignments['v'] == [1] # first-fit skips the claimed GPU + + +def test_ollama_cpu_gets_no_gpus(): + plan = plan_placement([ollama('cpu')], inv()) + assert plan.assignments == {'cpu': []} + + +def test_explicit_conflict_errors(): + plan = plan_placement( + [ollama('d1', gpu_indices=[0], t=0), ollama('d2', gpu_indices=[0], t=1)], + inv(), + ) + assert not plan.ok + assert any('already in use' in e for e in plan.errors) + + +def test_explicit_out_of_pool_errors(): + plan = plan_placement([ollama('d', gpu_indices=[7])], inv('4x80')) + assert not plan.ok + assert any('not in pool' in e for e in plan.errors) + + +def test_pinned_deployments_stay_put_when_new_arrives(): + # 'a' already runs on [0,1]; adding 'b' must not reshuffle 'a' + plan = plan_placement( + [vllm('a', tp=2, t=0), vllm('b', t=1)], inv(), pinned={'a': [0, 1]} + ) + assert plan.assignments == {'a': [0, 1], 'b': [2]} + + +def test_invalid_pin_is_replaced(): + # pinned GPU 9 doesn't exist -> 'a' is re-placed by first-fit + plan = plan_placement([vllm('a')], inv('4x80'), pinned={'a': [9]}) + assert plan.assignments == {'a': [0]} + + +def test_placement_is_deterministic_by_created_at(): + deployments = [vllm('z', t=1), vllm('a', t=0)] + plan = plan_placement(deployments, inv()) + # earlier created_at ('a', t=0) takes gpu 0 regardless of list order + assert plan.assignments == {'a': [0], 'z': [1]} diff --git a/tests/test_leasing_suggest.py b/tests/test_leasing_suggest.py new file mode 100644 index 00000000..0771c0c7 --- /dev/null +++ b/tests/test_leasing_suggest.py @@ -0,0 +1,132 @@ +"""Tests for the suggestion pool + the pure ``inventory × pool → catalog`` join.""" + +from __future__ import annotations + +from infer_stack.hardware import simulate_inventory +from infer_stack.leasing import Catalog +from infer_stack.leasing.suggest import ( + builtin_pool, + derive_runtime, + fits_on, + suggest_catalog, +) + + +def _gpu(index, mem, name='GPU', display=False): + return {'index': index, 'name': name, 'memory_gib': mem, 'display_active': display} + + +def test_builtin_pool_is_nonempty_and_real(): + pool = builtin_pool() + assert pool, 'the shipped suggestion pool should not be empty' + # the current-generation families (real, released 2026) are carried over + families = {m.family for m in pool.values()} + assert {'qwen3.5', 'qwen3.6', 'gemma4'} <= families + # ...with the real Hugging Face ids, not slugs + assert pool['qwen3.5-9b'].hf_model_id == 'Qwen/Qwen3.5-9B' + assert pool['gemma4-31b'].hf_model_id == 'google/gemma-4-31B-it' + # the demo's models are reproducible from the pool + assert {'smollm2-1.7b', 'qwen2.5-0.5b'} <= set(pool) + + +def test_derive_runtime_never_sizes_below_pool_default(): + # Regression: on a big GPU the bare footprint ratio for a small model is well + # below the pool's hand-tuned gpu_memory_utilization (sized for its KV cache + # at its context). Sizing below it starved the KV cache and vLLM OOM'd at + # startup. The computed util must only *raise* the default. + pool = builtin_pool() + model = pool['smollm2-1.7b'] + rt = derive_runtime(model, [_gpu(0, 24)]) + assert rt['gpu_memory_utilization'] >= model.defaults['gpu_memory_utilization'] + # a smaller GPU may need a bigger slice — the footprint estimate can raise it + rt_small = derive_runtime(model, [_gpu(0, 8)]) + assert rt_small['gpu_memory_utilization'] >= rt['gpu_memory_utilization'] + + +def test_rtx_3090_suggests_the_current_gen_models_that_fit(): + # A single 24 GiB RTX 3090: the current-gen models that fit a 24 GiB card + # should be suggested; the ones needing a bigger/second GPU should not. + inv = {'gpu_count': 1, 'gpus': [_gpu(0, 24, name='NVIDIA GeForce RTX 3090')]} + models = suggest_catalog(inv)['models'] + fits = {'qwen3.5-0.8b', 'qwen3.5-2b', 'qwen3.5-4b', 'qwen3.5-9b', + 'qwen3.6-35b-a3b-fp8', 'gemma4-e2b', 'gemma4-e4b', 'gemma4-26b', + 'gemma4-31b'} + too_big = {'qwen3.5-27b', 'qwen3.5-35b-a3b', 'qwen3.5-122b-a10b', + 'qwen3.6-35b-a3b'} # 35b-a3b needs 2 GPUs even at 24 GiB each + assert fits <= set(models) + assert too_big.isdisjoint(models) + + +def test_fits_on_respects_vram_and_gpu_count(): + pool = builtin_pool() + big = pool['qwen2.5-72b'] # needs 2 GPUs, 72 GiB each + assert not fits_on(big, [_gpu(0, 80)]) # one GPU: no + assert fits_on(big, [_gpu(0, 80), _gpu(1, 80)]) # two: yes + assert not fits_on(big, [_gpu(0, 48), _gpu(1, 48)]) # too small per GPU + + +def test_suggested_catalog_roundtrips_through_catalog(): + out = suggest_catalog(simulate_inventory('2x48')) + # The fragment is shaped like a catalog.yaml and parses/cross-refs cleanly. + cat = Catalog.from_dict(out) + assert cat.models and cat.endpoints + for ep in cat.endpoints.values(): + assert ep.model in cat.models + + +def test_fit_filter_tracks_gpu_size(): + one_small = suggest_catalog(simulate_inventory('1x16'))['models'] + assert 'qwen2.5-7b' in one_small # 16 GiB model fits a 16 GiB GPU + assert 'gpt-oss-20b' not in one_small # 40 GiB model does not + assert 'qwen2.5-72b' not in one_small # needs two GPUs + + +def test_derive_runtime_clamps_len_and_sizes_utilization(): + pool = builtin_pool() + # a tiny model on a big GPU should not greedily claim it (low utilization)... + rt_small = derive_runtime(pool['smollm2-135m'], [_gpu(0, 48)]) + assert rt_small['gpu_memory_utilization'] <= 0.3 + assert rt_small['max_model_len'] <= pool['smollm2-135m'].context_window + # ...and a snug model claims most of it. + rt_big = derive_runtime(pool['qwen2.5-7b'], [_gpu(0, 16)]) + assert rt_big['gpu_memory_utilization'] >= 0.8 + + +def test_pre_ampere_gpu_pins_fp16(): + pool = builtin_pool() + turing = [_gpu(0, 48, name='Quadro RTX 8000')] + ampere = [_gpu(0, 48, name='NVIDIA A40')] + assert derive_runtime(pool['qwen2.5-7b'], turing).get('extra_args') == ['--dtype=half'] + assert 'extra_args' not in derive_runtime(pool['qwen2.5-7b'], ampere) + + +def test_tensor_parallel_for_multi_gpu_model(): + pool = builtin_pool() + rt = derive_runtime(pool['qwen2.5-72b'], [_gpu(0, 80), _gpu(1, 80)]) + assert rt['tensor_parallel_size'] == 2 + + +def test_largest_fitting_model_is_kept_warm(): + out = suggest_catalog(simulate_inventory('1x48')) + warm = [n for n, e in out['endpoints'].items() + if e['reclaim']['policy'] == 'keep-warm'] + assert len(warm) == 1 + others = [e['reclaim']['policy'] for n, e in out['endpoints'].items() + if n not in warm] + assert set(others) == {'stop'} + + +def test_display_gpu_reservation_shrinks_the_pool(): + inv = {'gpu_count': 2, 'gpus': [ + _gpu(0, 80, name='A100'), + _gpu(1, 80, name='A100', display=True), + ]} + reserved = suggest_catalog(inv, reserve_display_gpu='auto')['models'] + used_all = suggest_catalog(inv, reserve_display_gpu=False)['models'] + assert 'qwen2.5-72b' not in reserved # only 1 usable GPU -> no 2-GPU model + assert 'qwen2.5-72b' in used_all # both GPUs -> it fits + + +def test_empty_inventory_yields_empty_suggestion(): + out = suggest_catalog({'gpu_count': 0, 'gpus': []}) + assert out == {'models': {}, 'endpoints': {}} diff --git a/tests/test_ollama_stack_graph.py b/tests/test_ollama_stack_graph.py deleted file mode 100644 index 31f79277..00000000 --- a/tests/test_ollama_stack_graph.py +++ /dev/null @@ -1,262 +0,0 @@ -from __future__ import annotations - -from pathlib import Path - -import yaml - -from infer_stack.backends.compose_renderer import render_compose_artifacts -from infer_stack.cli import _compose_has_service -from infer_stack.config import initial_config -from infer_stack.hardware import simulate_inventory -from infer_stack.resolver import resolve -from infer_stack.validator import validate_resolved - - -def _cfg(tmp_path: Path, profile: str, backend: str = 'compose') -> dict: - cfg = initial_config() - cfg['backend'] = backend - cfg['active_profile'] = profile - cfg['output']['generated_dir'] = str(tmp_path / 'generated') - for key in [ - 'hf_cache', - 'vllm_cache', - 'torch_cache', - 'triton_cache', - 'cuda_cache', - 'open_webui', - 'postgres_open_webui', - 'postgres_litellm', - 'ollama', - 'runtime', - ]: - cfg['state'][key] = str(tmp_path / 'state' / key) - return cfg - - -def test_ollama_direct_renders_no_litellm(tmp_path: Path) -> None: - dep = resolve( - _cfg(tmp_path, 'ollama-direct'), inventory=simulate_inventory('2x11') - ) - assert dep['providers']['ollama']['enabled'] is True - assert dep['gateways']['litellm']['enabled'] is False - assert dep['frontends']['open_webui']['provider'] == 'ollama' - assert dep['services'] == [] - assert validate_resolved(dep)['ok'] is True - render_compose_artifacts({'deployment': dep}) - compose = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - assert 'ollama:' in compose - assert 'litellm:' not in compose - assert 'postgres-litellm:' not in compose - assert 'OLLAMA_BASE_URL: http://ollama:11434' in compose - assert not (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').exists() - - -def test_ollama_gateway_routes_through_litellm(tmp_path: Path) -> None: - dep = resolve( - _cfg(tmp_path, 'ollama-qwen3.5-4b-gateway'), - inventory=simulate_inventory('2x11'), - ) - assert dep['gateways']['litellm']['enabled'] is True - assert ( - dep['gateways']['litellm']['routes']['qwen3.5-4b']['provider'] - == 'ollama' - ) - render_compose_artifacts({'deployment': dep}) - cfg_doc = yaml.safe_load( - (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').read_text() - ) - entry = cfg_doc['model_list'][0] - assert entry['model_name'] == 'qwen3.5-4b' - assert entry['litellm_params']['model'] == 'ollama_chat/qwen3.5:4b' - - -def test_mixed_gateway_renders_ollama_vllm_and_litellm(tmp_path: Path) -> None: - dep = resolve( - _cfg(tmp_path, 'mixed-ollama-smollm'), - inventory=simulate_inventory('3x11'), - ) - assert dep['providers']['ollama']['enabled'] is True - assert dep['providers']['vllm']['runtimes'] - assert dep['gateways']['litellm']['enabled'] is True - render_compose_artifacts({'deployment': dep}) - compose = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - assert 'ollama:' in compose - assert 'vllm-smollm:' in compose - assert 'litellm:' in compose - litellm = ( - tmp_path / 'state' / 'runtime' / 'litellm_config.yaml' - ).read_text() - assert 'ollama_chat/qwen3.5:4b' in litellm - assert 'openai/' in litellm - - -def test_kubeai_rejects_ollama_provider(tmp_path: Path) -> None: - dep = resolve( - _cfg(tmp_path, 'ollama-direct', backend='kubeai'), - inventory=simulate_inventory('2x11'), - ) - report = validate_resolved(dep) - assert report['ok'] is False - assert any('Ollama provider' in err for err in report['errors']) - - -def test_smollm_gpu1_vllm_profile_is_builtin_and_pins_gpu1( - tmp_path: Path, -) -> None: - dep = resolve( - _cfg(tmp_path, 'smollm2-135m-vllm-gpu1'), - inventory=simulate_inventory('2x24'), - ) - rt = dep['providers']['vllm']['runtimes']['chat'] - assert rt['gpu_indices'] == [1] - assert rt['compose_service_name'] == 'vllm-chat' - assert rt['container_name'] == 'vllm-chat' - assert dep['gateways']['litellm']['enabled'] is True - assert dep['frontends']['open_webui']['provider'] == 'litellm' - assert ( - dep['gateways']['litellm']['routes']['smollm2-135m']['provider'] - == 'vllm' - ) - render_compose_artifacts({'deployment': dep}) - compose = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - assert 'vllm-chat:' in compose - assert 'container_name: vllm-chat' in compose - assert 'container_name: chat' not in compose - - -def test_smollm_gpu1_ollama_profile_is_builtin_and_has_no_routes( - tmp_path: Path, -) -> None: - dep = resolve( - _cfg(tmp_path, 'smollm2-135m-ollama-gpu1'), - inventory=simulate_inventory('2x24'), - ) - assert dep['providers']['ollama']['enabled'] is True - assert dep['providers']['ollama']['gpu_indices'] == [1] - assert dep['providers']['ollama']['publish_port'] is True - assert dep['providers']['vllm']['runtimes'] == {} - assert dep['gateways']['litellm']['enabled'] is False - assert dep['frontends']['open_webui']['provider'] == 'ollama' - assert dep['gateways']['litellm']['routes'] == {} - render_compose_artifacts({'deployment': dep}) - compose = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - assert 'CUDA_VISIBLE_DEVICES: "1"' in compose - assert 'device_ids: ["1"]' in compose - assert 'litellm:' not in compose - - -def test_existing_config_without_new_ollama_defaults_is_hydrated( - tmp_path: Path, -) -> None: - cfg = _cfg(tmp_path, 'smollm2-135m-ollama-gpu1') - cfg['images'].pop('ollama', None) - cfg['ports'].pop('ollama', None) - dep = resolve(cfg, inventory=simulate_inventory('2x24')) - assert dep['images']['ollama'] == 'ollama/ollama:latest' - assert dep['ports']['ollama'] == 11434 - render_compose_artifacts({'deployment': dep}) - compose = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - assert 'image: ollama/ollama:latest' in compose - assert '"127.0.0.1:11434:11434"' in compose - - -def test_compose_has_service_ignores_stale_litellm_runtime_file( - tmp_path: Path, -) -> None: - compose_file = tmp_path / 'docker-compose.yml' - compose_file.write_text( - """ -services: - postgres-open-webui: - image: postgres:16.8 - ollama: - image: ollama/ollama:latest - open-webui: - image: ghcr.io/open-webui/open-webui:v0.8.6 -""".lstrip(), - encoding='utf-8', - ) - - assert _compose_has_service(compose_file, 'ollama') is True - assert _compose_has_service(compose_file, 'open-webui') is True - assert _compose_has_service(compose_file, 'litellm') is False - - -def test_vllm_compose_persists_obvious_startup_caches(tmp_path: Path) -> None: - dep = resolve( - _cfg(tmp_path, 'smollm2-135m-vllm-gpu1'), - inventory=simulate_inventory('2x24'), - ) - render_compose_artifacts({'deployment': dep}) - compose = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - assert 'HF_HOME: /root/.cache/huggingface' in compose - assert 'VLLM_CACHE_ROOT: /root/.cache/vllm' in compose - assert 'TORCHINDUCTOR_CACHE_DIR: /root/.cache/torch/inductor' in compose - assert 'TRITON_CACHE_DIR: /root/.cache/triton' in compose - assert 'CUDA_CACHE_PATH: /root/.cache/nvidia/ComputeCache' in compose - assert ':/root/.cache/huggingface' in compose - assert ':/root/.cache/vllm' in compose - assert ':/root/.cache/torch' in compose - assert ':/root/.cache/triton' in compose - assert ':/root/.cache/nvidia/ComputeCache' in compose - - -def test_litellm_does_not_depend_on_provider_health_for_model_swaps( - tmp_path: Path, -) -> None: - dep = resolve( - _cfg(tmp_path, 'mixed-ollama-smollm'), - inventory=simulate_inventory('3x11'), - ) - render_compose_artifacts({'deployment': dep}) - doc = yaml.safe_load( - (tmp_path / 'generated' / 'docker-compose.yml').read_text() - ) - depends_on = doc['services']['litellm'].get('depends_on', {}) - assert 'postgres-litellm' in depends_on - assert 'ollama' not in depends_on - assert 'vllm-smollm' not in depends_on - - -def test_litellm_config_model_delete_miss_is_nonfatal() -> None: - from infer_stack.cli import _litellm_delete_missed_config_model - - class DummyResponse: - text = '' - - def json(self): - return {'error': 'Model with id=abc not found in db'} - - assert _litellm_delete_missed_config_model(DummyResponse()) is True # type: ignore - - class OtherResponse: - text = 'permission denied' - - def json(self): - raise ValueError - - assert _litellm_delete_missed_config_model(OtherResponse()) is False # type: ignore - - -def test_schema_v5_default_model_and_protocol_helpers(tmp_path: Path) -> None: - from infer_stack.cli import ( - _default_model_for_deployment, - _resolve_smoke_protocol_from_deployment, - ) - - smol = resolve( - _cfg(tmp_path, 'smollm2-135m-vllm-gpu1'), - inventory=simulate_inventory('2x24'), - ) - assert _default_model_for_deployment(smol) == 'smollm2-135m' - assert ( - _resolve_smoke_protocol_from_deployment(smol, 'smollm2-135m') == 'chat' - ) - - gpt2 = resolve( - _cfg(tmp_path, 'gpt2-single'), inventory=simulate_inventory('2x24') - ) - assert _default_model_for_deployment(gpt2) == 'gpt2' - assert ( - _resolve_smoke_protocol_from_deployment(gpt2, 'gpt2') == 'completions' - ) diff --git a/tests/test_reverse_proxy_ldap.py b/tests/test_reverse_proxy_ldap.py deleted file mode 100644 index 50a4613a..00000000 --- a/tests/test_reverse_proxy_ldap.py +++ /dev/null @@ -1,392 +0,0 @@ -from __future__ import annotations - -from pathlib import Path - -import yaml - -from infer_stack.backends.compose_renderer import render_compose_artifacts -from infer_stack.config import initial_config -from infer_stack.hardware import simulate_inventory -from infer_stack.resolver import resolve -from infer_stack.validator import validate_resolved - - -def _cfg(tmp_path: Path, profile: str) -> dict: - cfg = initial_config() - cfg['backend'] = 'compose' - cfg['active_profile'] = profile - cfg['output']['generated_dir'] = str(tmp_path / 'generated') - for key in cfg['state']: - cfg['state'][key] = str(tmp_path / 'state' / key) - return cfg - - -def _cfg_with_profile(tmp_path: Path, name: str, profile: dict) -> dict: - """Build a config whose active profile is a caller-supplied custom stack.""" - cfg = _cfg(tmp_path, name) - cfg['profiles'][name] = profile - return cfg - - -def _render(cfg: dict, tmp_path: Path, hardware: str = '1x24') -> dict: - """Resolve + validate + render and return the parsed compose document.""" - dep = resolve(cfg, inventory=simulate_inventory(hardware)) - report = validate_resolved(dep) - assert report['ok'] is True, report['errors'] - render_compose_artifacts({'deployment': dep}) - return yaml.safe_load( - (tmp_path / 'generated' / 'docker-compose.yml').read_text() - ) - - -def test_openwebui_tls_ldap_profile_renders_ldap_and_reverse_proxy( - tmp_path: Path, -) -> None: - dep = resolve( - _cfg(tmp_path, 'openwebui-tls-ldap'), - inventory=simulate_inventory('2x24'), - ) - report = validate_resolved(dep) - assert report['ok'] is True - assert dep['frontends']['open_webui']['publish_port'] is False - assert dep['providers']['ollama']['publish_port'] is False - assert dep['frontends']['reverse_proxy']['enabled'] is True - - render_compose_artifacts({'deployment': dep}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - compose_doc = yaml.safe_load(compose_text) - services = compose_doc['services'] - - assert 'reverse-proxy' in services - assert 'ports' not in services['open-webui'] - assert 'ports' not in services['ollama'] - assert services['ollama']['environment']['OLLAMA_FLASH_ATTENTION'] == '1' - assert services['ollama']['environment']['OLLAMA_NO_CLOUD'] == 'true' - assert services['open-webui']['environment']['ENABLE_LDAP'] == 'true' - assert ( - services['open-webui']['environment']['LDAP_SERVER_HOST'] - == '${LDAP_HOST}' - ) - assert services['reverse-proxy']['ports'] == [ - '${INFER_STACK_REVERSE_PROXY_HTTP_PORT:-80}:80', - '${INFER_STACK_REVERSE_PROXY_HTTPS_PORT:-443}:443', - ] - - env_text = (tmp_path / 'generated' / '.env').read_text() - assert 'LDAP_PORT=636' in env_text - assert 'LDAP_ATTRIBUTE_FOR_USERNAME=uid' in env_text - - nginx_text = (tmp_path / 'state' / 'runtime' / 'nginx.conf').read_text() - assert 'server_name openwebui.example.com;' in nginx_text - assert 'proxy_pass http://infer_stack_target;' in nginx_text - assert 'client_max_body_size 1G;' in nginx_text - - -def test_reverse_proxy_can_use_manual_nginx_config_path(tmp_path: Path) -> None: - cfg = _cfg(tmp_path, 'ollama-direct') - cfg['profiles']['manual-nginx'] = { - 'description': 'manual nginx config path smoke test', - 'providers': { - 'ollama': { - 'enabled': True, - 'gpu_indices': [0], - 'publish_port': False, - } - }, - 'gateways': {'litellm': {'enabled': False}}, - 'frontends': { - 'open_webui': { - 'enabled': True, - 'provider': 'ollama', - 'publish_port': False, - }, - 'reverse_proxy': { - 'enabled': True, - 'target': 'open_webui', - 'ssl': {'enabled': False}, - 'config_path': str(tmp_path / 'manual-nginx.conf'), - }, - }, - 'routes': {}, - } - cfg['active_profile'] = 'manual-nginx' - (tmp_path / 'manual-nginx.conf').write_text( - 'server { listen 80; }\n', encoding='utf-8' - ) - - dep = resolve(cfg, inventory=simulate_inventory('1x24')) - assert validate_resolved(dep)['ok'] is True - render_compose_artifacts({'deployment': dep}) - compose_doc = yaml.safe_load( - (tmp_path / 'generated' / 'docker-compose.yml').read_text() - ) - volumes = compose_doc['services']['reverse-proxy']['volumes'] - assert ( - f'{tmp_path / "manual-nginx.conf"}:/etc/nginx/conf.d/default.conf:ro' - in volumes - ) - assert not (tmp_path / 'state' / 'runtime' / 'nginx.conf').exists() - - -# --- Base profiles reused by the focused tests below --------------------------- - - -def _ollama_openwebui_profile( - reverse_proxy: dict | None = None, open_webui: dict | None = None -) -> dict: - frontends: dict = { - 'open_webui': { - 'enabled': True, - 'provider': 'ollama', - 'publish_port': False, - **(open_webui or {}), - } - } - if reverse_proxy is not None: - frontends['reverse_proxy'] = reverse_proxy - return { - 'description': 'ollama + open-webui test stack', - 'providers': { - 'ollama': { - 'enabled': True, - 'gpu_indices': [0], - 'publish_port': False, - } - }, - 'gateways': {'litellm': {'enabled': False}}, - 'frontends': frontends, - 'routes': {}, - } - - -def test_http_only_reverse_proxy_has_no_tls_block(tmp_path: Path) -> None: - cfg = _cfg_with_profile( - tmp_path, - 'http-only', - _ollama_openwebui_profile( - reverse_proxy={ - 'enabled': True, - 'target': 'open_webui', - 'ssl': {'enabled': False}, - } - ), - ) - compose_doc = _render(cfg, tmp_path) - - # No TLS means HTTPS must not be published, and port 80 alone is mapped. - assert compose_doc['services']['reverse-proxy']['ports'] == [ - '${INFER_STACK_REVERSE_PROXY_HTTP_PORT:-80}:80' - ] - nginx_text = (tmp_path / 'state' / 'runtime' / 'nginx.conf').read_text() - assert 'listen 443' not in nginx_text - assert 'return 301 https' not in nginx_text # no force_https without TLS - assert 'proxy_pass http://infer_stack_target;' in nginx_text - - -def test_force_https_redirect_present_with_tls(tmp_path: Path) -> None: - rp = { - 'enabled': True, - 'target': 'open_webui', - 'server_name': 'host.example.com', - 'ssl': { - 'enabled': True, - 'certificate': 'cert.crt', - 'certificate_key': 'key.key', - }, - } - cfg = _cfg_with_profile( - tmp_path, 'tls', _ollama_openwebui_profile(reverse_proxy=rp) - ) - _render(cfg, tmp_path) - nginx_text = (tmp_path / 'state' / 'runtime' / 'nginx.conf').read_text() - assert 'return 301 https://$host$request_uri;' in nginx_text - assert 'listen 443 ssl;' in nginx_text - - -def test_ldap_disabled_omits_ldap_env(tmp_path: Path) -> None: - cfg = _cfg_with_profile(tmp_path, 'no-ldap', _ollama_openwebui_profile()) - compose_doc = _render(cfg, tmp_path) - env = compose_doc['services']['open-webui']['environment'] - assert not any( - key.startswith('LDAP_') or key == 'ENABLE_LDAP' for key in env - ) - - -def test_reverse_proxy_target_litellm(tmp_path: Path) -> None: - profile = { - 'description': 'litellm target', - 'providers': {'ollama': {'enabled': True, 'gpu_indices': [0]}}, - 'gateways': {'litellm': {'enabled': True}}, - 'frontends': { - 'open_webui': { - 'enabled': True, - 'provider': 'litellm', - 'publish_port': False, - }, - 'reverse_proxy': { - 'enabled': True, - 'target': 'litellm', - 'ssl': {'enabled': False}, - }, - }, - 'routes': {'chat': {'provider': 'ollama', 'model': 'llama3'}}, - } - cfg = _cfg_with_profile(tmp_path, 'rp-litellm', profile) - compose_doc = _render(cfg, tmp_path) - assert 'litellm' in compose_doc['services']['reverse-proxy']['depends_on'] - nginx_text = (tmp_path / 'state' / 'runtime' / 'nginx.conf').read_text() - assert 'server litellm:4000;' in nginx_text - - -def test_generic_service_escape_hatches_render(tmp_path: Path) -> None: - open_webui = { - 'extra_env': {'WEBUI_NAME': 'Demo', 'CUSTOM_FLAG': '1'}, - 'env_file': ['./openwebui.env'], - 'extra_volumes': ['./branding:/app/branding:ro'], - 'extra_hosts': ['registry.internal:10.0.0.5'], - 'labels': {'com.example.team': 'infra'}, - 'additional_ports': ['127.0.0.1:3001:8080'], - 'gpus': 'all', - } - cfg = _cfg_with_profile( - tmp_path, 'hatches', _ollama_openwebui_profile(open_webui=open_webui) - ) - compose_doc = _render(cfg, tmp_path) - svc = compose_doc['services']['open-webui'] - assert svc['environment']['WEBUI_NAME'] == 'Demo' - assert svc['environment']['CUSTOM_FLAG'] == '1' - assert svc['env_file'] == ['./openwebui.env'] - assert './branding:/app/branding:ro' in svc['volumes'] - assert svc['extra_hosts'] == ['registry.internal:10.0.0.5'] - assert svc['labels']['com.example.team'] == 'infra' - assert '127.0.0.1:3001:8080' in svc['ports'] - assert svc['gpus'] == 'all' - - -def test_structured_gpus_renders_as_yaml_list(tmp_path: Path) -> None: - device_request = [ - {'driver': 'nvidia', 'count': 'all', 'capabilities': ['gpu']} - ] - profile = _ollama_openwebui_profile() - profile['providers']['ollama']['gpus'] = device_request - cfg = _cfg_with_profile(tmp_path, 'structured-gpus', profile) - compose_doc = _render(cfg, tmp_path) - ollama = compose_doc['services']['ollama'] - # The structured form round-trips as real YAML, not a quoted repr, and the - # device-reservation deploy block is not also emitted. - assert ollama['gpus'] == device_request - assert 'deploy' not in ollama - - -def test_extra_config_injected_into_nginx(tmp_path: Path) -> None: - rp = { - 'enabled': True, - 'target': 'open_webui', - 'ssl': { - 'enabled': True, - 'certificate': 'cert.crt', - 'certificate_key': 'key.key', - }, - 'extra_config': "location /healthz { return 200 'ok'; }", - } - cfg = _cfg_with_profile( - tmp_path, 'extra-config', _ollama_openwebui_profile(reverse_proxy=rp) - ) - _render(cfg, tmp_path) - nginx_text = (tmp_path / 'state' / 'runtime' / 'nginx.conf').read_text() - assert "location /healthz { return 200 'ok'; }" in nginx_text - - -def test_no_watchtower_service_is_rendered(tmp_path: Path) -> None: - dep = resolve( - _cfg(tmp_path, 'openwebui-tls-ldap'), - inventory=simulate_inventory('2x24'), - ) - render_compose_artifacts({'deployment': dep}) - compose_doc = yaml.safe_load( - (tmp_path / 'generated' / 'docker-compose.yml').read_text() - ) - assert all('watchtower' not in name for name in compose_doc['services']) - assert not any( - (svc or {}).get('image', '').startswith('nickfedor/watchtower') - for svc in compose_doc['services'].values() - ) - - -# --- Validator guards ---------------------------------------------------------- - - -def test_validator_errors_when_target_disabled(tmp_path: Path) -> None: - profile = _ollama_openwebui_profile( - reverse_proxy={ - 'enabled': True, - 'target': 'litellm', - 'ssl': {'enabled': False}, - } - ) - cfg = _cfg_with_profile(tmp_path, 'bad-target', profile) - dep = resolve(cfg, inventory=simulate_inventory('1x24')) - report = validate_resolved(dep) - assert report['ok'] is False - assert any('target=litellm' in e for e in report['errors']) - - -def test_validator_warns_on_missing_certs(tmp_path: Path) -> None: - rp = { - 'enabled': True, - 'target': 'open_webui', - 'ssl': { - 'enabled': True, - 'certificate': '', - 'certificate_key': 'missing.key', - }, - } - cfg = _cfg_with_profile( - tmp_path, 'missing-certs', _ollama_openwebui_profile(reverse_proxy=rp) - ) - dep = resolve(cfg, inventory=simulate_inventory('1x24')) - report = validate_resolved(dep) - assert report['ok'] is True # warnings, not errors - joined = ' '.join(report['warnings']) - assert 'ssl.certificate is empty' in joined - assert 'ssl.certificate_key' in joined and 'not found' in joined - - -def test_validator_errors_on_publish_https_without_ssl_in_plan( - tmp_path: Path, -) -> None: - # Simulate a hand-edited plan where the resolver's gating was bypassed. - cfg = _cfg_with_profile( - tmp_path, - 'http-only', - _ollama_openwebui_profile( - reverse_proxy={ - 'enabled': True, - 'target': 'open_webui', - 'ssl': {'enabled': False}, - } - ), - ) - dep = resolve(cfg, inventory=simulate_inventory('1x24')) - dep['frontends']['reverse_proxy']['publish_https'] = True - report = validate_resolved(dep) - assert report['ok'] is False - assert any('publishes HTTPS' in e for e in report['errors']) - - -def test_validator_warns_on_missing_config_path(tmp_path: Path) -> None: - rp = { - 'enabled': True, - 'target': 'open_webui', - 'ssl': {'enabled': False}, - 'config_path': str(tmp_path / 'does-not-exist.conf'), - } - cfg = _cfg_with_profile( - tmp_path, 'missing-config', _ollama_openwebui_profile(reverse_proxy=rp) - ) - dep = resolve(cfg, inventory=simulate_inventory('1x24')) - report = validate_resolved(dep) - assert report['ok'] is True - assert any( - 'config_path' in w and 'not found' in w for w in report['warnings'] - ) diff --git a/tests/test_serving_profiles.py b/tests/test_serving_profiles.py deleted file mode 100644 index 5d1605f6..00000000 --- a/tests/test_serving_profiles.py +++ /dev/null @@ -1,1227 +0,0 @@ -from __future__ import annotations - -from pathlib import Path - -import pytest -import yaml - -from infer_stack.backends.compose_renderer import render_compose_artifacts -from infer_stack.backends.kubeai_renderer import render_kubeai_artifacts -from infer_stack.config import initial_config -from infer_stack.config import save_yaml -from infer_stack.contracts import build_profile_contract, load_profile_contract -from infer_stack.hardware import simulate_inventory -from infer_stack.paths import set_config_root, set_data_root -from infer_stack.resolver import resolve -from infer_stack.validator import validate_resolved - - -@pytest.fixture(autouse=True) -def _anchor_paths(tmp_path: Path): - """Anchor config_root() / data_root() at tmp_path for every test. - - The resolver and renderers read from data_root() (for generated/ - and state/), and load_profile_contract() reads from config_root(). - Pointing both at tmp_path keeps each test hermetic. - """ - set_config_root(tmp_path) - set_data_root(tmp_path) - yield - set_config_root(None) - set_data_root(None) - - -def _cfg(tmp_path: Path, *, backend: str = 'compose') -> dict: - cfg = initial_config() - cfg['backend'] = backend - cfg['state'] = { - 'hf_cache': 'state/hf-cache', - 'vllm_cache': 'state/vllm-cache', - 'open_webui': 'state/open-webui', - 'postgres': 'state/postgres', - 'runtime': 'state/runtime', - } - cfg['ports'] = {'litellm': 14042, 'open_webui': 13000, 'postgres': 15432} - # Relative paths in output.generated_dir / state.* anchor on data_root(), - # which the autouse fixture pins to tmp_path. - cfg['output'] = {'generated_dir': 'generated'} - return cfg - - -def _write_root_config(tmp_path: Path, *, backend: str = 'compose') -> Path: - cfg = _cfg(tmp_path, backend=backend) - save_yaml(tmp_path / 'config.yaml', cfg) - save_yaml(tmp_path / 'models.yaml', {'models': {}, 'profiles': {}}) - return tmp_path - - -def _deployment( - tmp_path: Path, - profile_name: str, - *, - backend: str = 'compose', - inventory: str = '4x96', -) -> dict: - cfg = _cfg(tmp_path, backend=backend) - deployment = resolve( - cfg, inventory=simulate_inventory(inventory), profile_name=profile_name - ) - validated = validate_resolved(deployment) - assert validated['ok'], validated - return deployment - - -def test_profile_resolution_uses_named_serving_profile(tmp_path: Path) -> None: - deployment = _deployment(tmp_path, 'qwen2-72b-instruct-tp2-balanced') - assert ( - deployment['serving_profile']['public_name'] - == 'qwen2-72b-instruct-tp2-balanced' - ) - assert ( - deployment['serving_profile']['logical_model_name'] - == 'qwen/qwen2-72b-instruct' - ) - assert deployment['services'][0]['tensor_parallel_size'] == 2 - assert 'qwen2-72b-instruct-tp2-balanced' in deployment['router']['aliases'] - - -def test_legacy_profile_alias_resolves_to_canonical_profile( - tmp_path: Path, -) -> None: - deployment = _deployment(tmp_path, 'helm-qwen2-72b-instruct') - assert ( - deployment['serving_profile']['name'] - == 'qwen2-72b-instruct-tp2-balanced' - ) - - -def test_kubeai_render_uses_profile_identity(tmp_path: Path) -> None: - deployment = _deployment( - tmp_path, 'qwen2-72b-instruct-tp2-balanced', backend='kubeai' - ) - render_kubeai_artifacts({'deployment': deployment}) - models = list( - yaml.safe_load_all( - (tmp_path / 'generated' / 'kubeai' / 'models.yaml').read_text() - ) - ) - assert models[0]['metadata']['name'] == 'qwen2-72b-instruct-tp2-balanced' - assert ( - models[0]['metadata']['annotations']['infer-stack/logical-model-name'] - == 'qwen/qwen2-72b-instruct' - ) - assert models[0]['spec']['resourceProfile'] == 'gpu-tp2-balanced:2' - assert '--tensor-parallel-size=2' in models[0]['spec']['args'] - assert ( - '--served-model-name=qwen2-72b-instruct-tp2-balanced' - in models[0]['spec']['args'] - ) - - -def test_compose_render_includes_profile_labels_and_aliases( - tmp_path: Path, -) -> None: - deployment = _deployment(tmp_path, 'gpt-oss-20b-chat') - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - litellm_text = ( - tmp_path / 'state' / 'runtime' / 'litellm_config.yaml' - ).read_text() - assert 'infer_stack.public_name: "gpt-oss-20b-chat"' in compose_text - assert 'openai/gpt-oss-20b' in litellm_text - assert 'gpt-oss-20b-chat' in litellm_text - - -def test_profile_contract_is_generic_and_backend_agnostic( - tmp_path: Path, -) -> None: - deployment = _deployment(tmp_path, 'qwen2-72b-instruct-tp2-balanced') - contract = build_profile_contract(deployment) - service = contract['services'][0] - assert contract['kind'] == 'serving-profile-contract' - assert ( - contract['profile']['public_name'] == 'qwen2-72b-instruct-tp2-balanced' - ) - assert service['access']['default']['kind'] == 'openai-compatible' - assert service['access']['additional'][0]['kind'] == 'vllm-direct' - assert service['access']['additional'][0]['auth_env_name'] == 'VLLM_API_KEY' - assert 'client_spec' not in str(contract) - assert 'model_deployments' not in str(contract) - - -def test_profile_contract_for_kubeai_uses_public_profile_name( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'qwen2-72b-instruct-tp2-balanced', backend='kubeai' - ) - contract = build_profile_contract(deployment) - access = contract['services'][0]['access']['default'] - assert access['kind'] == 'openai-compatible' - assert access['request_model_name'] == 'qwen2-72b-instruct-tp2-balanced' - assert access['base_url'].endswith('/openai/v1') - - -def test_load_profile_contract_uses_public_loader_for_qwen( - tmp_path: Path, -) -> None: - _write_root_config(tmp_path) - contract = load_profile_contract( - 'qwen2-72b-instruct-tp2-balanced', - simulate_hardware_spec='2x96', - ) - assert ( - contract['profile']['public_name'] == 'qwen2-72b-instruct-tp2-balanced' - ) - assert ( - contract['services'][0]['model']['logical_model_name'] - == 'qwen/qwen2-72b-instruct' - ) - - -def _split_compose_blocks(compose_text: str) -> dict[str, str]: - """Split a rendered docker-compose.yml into a {service_name: body} dict. - - Service blocks start at column 2 (two-space indent) followed by the name - and a colon. Splitting this way avoids substring confusion between e.g. - ``open-webui:`` and ``postgres-open-webui:``. - """ - import re - - pattern = re.compile(r'(?m)^ ([a-zA-Z0-9_-]+):\s*$') - matches = list(pattern.finditer(compose_text)) - blocks: dict[str, str] = {} - for i, m in enumerate(matches): - name = m.group(1) - start = m.end() - end = ( - matches[i + 1].start() - if i + 1 < len(matches) - else len(compose_text) - ) - blocks[name] = compose_text[start:end] - return blocks - - -def test_compose_uses_separate_postgres_services(tmp_path: Path) -> None: - deployment = _deployment(tmp_path, 'gpt-oss-20b-chat') - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - blocks = _split_compose_blocks(compose_text) - - assert 'postgres-open-webui' in blocks - assert 'postgres-litellm' in blocks - assert 'postgres-init' not in blocks - assert 'postgres-init' not in compose_text - assert 'shared-postgress' not in compose_text - - owui_pg = blocks['postgres-open-webui'] - ll_pg = blocks['postgres-litellm'] - assert '${OPENWEBUI_POSTGRES_DB}' in owui_pg - assert '${OPENWEBUI_POSTGRES_USER}' in owui_pg - assert '${OPENWEBUI_POSTGRES_PASSWORD}' in owui_pg - assert '${LITELLM_POSTGRES_DB}' not in owui_pg - assert '${LITELLM_POSTGRES_DB}' in ll_pg - assert '${LITELLM_POSTGRES_USER}' in ll_pg - assert '${LITELLM_POSTGRES_PASSWORD}' in ll_pg - assert '${OPENWEBUI_POSTGRES_DB}' not in ll_pg - - -def test_compose_router_and_ui_point_at_their_own_postgres( - tmp_path: Path, -) -> None: - deployment = _deployment(tmp_path, 'gpt-oss-20b-chat') - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - blocks = _split_compose_blocks(compose_text) - - litellm_block = blocks['litellm'] - openwebui_block = blocks['open-webui'] - assert '@postgres-litellm:5432/${LITELLM_POSTGRES_DB}' in litellm_block - assert '@postgres-open-webui:' not in litellm_block - assert ( - '@postgres-open-webui:5432/${OPENWEBUI_POSTGRES_DB}' in openwebui_block - ) - assert '@postgres-litellm:' not in openwebui_block - - -def test_compose_env_uses_distinct_db_names_users_and_passwords( - tmp_path: Path, -) -> None: - deployment = _deployment(tmp_path, 'gpt-oss-20b-chat') - render_compose_artifacts({'deployment': deployment}) - env_text = (tmp_path / 'generated' / '.env').read_text() - env_kv = dict( - line.split('=', 1) - for line in env_text.splitlines() - if line and '=' in line and not line.startswith('#') - ) - assert env_kv['OPENWEBUI_POSTGRES_DB'] == 'openwebui' - assert env_kv['LITELLM_POSTGRES_DB'] == 'litellm' - assert env_kv['OPENWEBUI_POSTGRES_USER'] == 'openwebui' - assert env_kv['LITELLM_POSTGRES_USER'] == 'litellm' - assert env_kv['OPENWEBUI_POSTGRES_PASSWORD'] - assert env_kv['LITELLM_POSTGRES_PASSWORD'] - assert ( - env_kv['OPENWEBUI_POSTGRES_PASSWORD'] - != env_kv['LITELLM_POSTGRES_PASSWORD'] - ) - # Old shared schema must not be emitted automatically. - assert 'POSTGRES_DB' not in env_kv - assert 'POSTGRES_USER' not in env_kv - assert 'POSTGRES_PASSWORD' not in env_kv - - -def test_env_rewrite_preserves_unknown_key_value_pairs(tmp_path: Path) -> None: - generated = tmp_path / 'generated' - generated.mkdir() - (generated / '.env').write_text( - '# user comment\n' - 'OPENWEBUI_POSTGRES_PASSWORD=keepme\n' - 'VERBOSE=1\n' - 'CUSTOM_THING=abc\n' - 'HTTP_PROXY=http://proxy:3128\n' - 'NO_PROXY=localhost,127.0.0.1\n', - encoding='utf-8', - ) - deployment = _deployment(tmp_path, 'gpt-oss-20b-chat') - render_compose_artifacts({'deployment': deployment}) - text = (generated / '.env').read_text() - assert 'VERBOSE=1' in text - assert 'CUSTOM_THING=abc' in text - assert 'HTTP_PROXY=http://proxy:3128' in text - assert 'NO_PROXY=localhost,127.0.0.1' in text - assert '# user comment' in text - env_kv = dict( - line.split('=', 1) - for line in text.splitlines() - if line and '=' in line and not line.startswith('#') - ) - # Existing managed values are preserved (password not rotated). - assert env_kv['OPENWEBUI_POSTGRES_PASSWORD'] == 'keepme' - - -def test_helm_pythia_profiles_render_as_completions(tmp_path: Path) -> None: - for profile_name in ( - 'helm-pythia-6.9b', - 'helm-pythia-2.8b-v0', - 'helm-pythia-1b-v0', - ): - deployment = _deployment(tmp_path, profile_name, inventory='1x96') - render_compose_artifacts({'deployment': deployment}) - compose_text = ( - tmp_path / 'generated' / 'docker-compose.yml' - ).read_text() - assert 'infer_stack.protocol_mode: "completions"' in compose_text, ( - profile_name - ) - assert deployment['services'][0]['protocol_mode'] == 'completions', ( - profile_name - ) - - -def test_helm_base_model_profiles_render_as_completions(tmp_path: Path) -> None: - for profile_name in ( - 'helm-llama-2-7b', - 'helm-mistral-7b-v0.1', - 'helm-falcon-7b', - ): - deployment = _deployment(tmp_path, profile_name, inventory='1x96') - assert deployment['services'][0]['protocol_mode'] == 'completions', ( - profile_name - ) - - -def test_validation_rejects_chat_protocol_for_completions_only_model( - tmp_path: Path, -) -> None: - from infer_stack.config import initial_config - - cfg = initial_config() - cfg['backend'] = 'compose' - cfg['state'] = { - 'hf_cache': 'state/hf-cache', - 'vllm_cache': 'state/vllm-cache', - 'open_webui': 'state/open-webui', - 'postgres_open_webui': 'state/postgres-open-webui', - 'postgres_litellm': 'state/postgres-litellm', - 'runtime': 'state/runtime', - } - cfg['ports'] = {'litellm': 14042, 'open_webui': 13000, 'postgres': 15432} - cfg['profiles'] = { - 'broken-pythia-chat': { - 'description': 'Synthetic invalid profile: chat on a completions-only model.', - 'base_model': 'pythia-1b-v0', - 'public_name': 'broken-pythia-chat', - 'protocol_mode': 'chat', - } - } - deployment = resolve( - cfg, - inventory=simulate_inventory('1x96'), - profile_name='broken-pythia-chat', - ) - report = validate_resolved(deployment) - assert report['ok'] is False - joined = ' | '.join(report['errors']) - assert 'broken-pythia-chat' in joined - assert 'completions' in joined - assert 'pythia-1b-v0' in joined - - -def test_validation_accepts_chat_for_chat_capable_profile( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'qwen2-5-7b-instruct-turbo-default', inventory='1x96' - ) - report = validate_resolved(deployment) - assert report['ok'] is True, report - assert deployment['services'][0]['protocol_mode'] == 'chat' - - -def test_litellm_completions_profile_uses_text_completion_provider( - tmp_path: Path, -) -> None: - deployment = _deployment(tmp_path, 'helm-pythia-6.9b', inventory='1x96') - render_compose_artifacts({'deployment': deployment}) - litellm_text = ( - tmp_path / 'state' / 'runtime' / 'litellm_config.yaml' - ).read_text() - assert 'text-completion-openai/eleutherai/pythia-6.9b' in litellm_text - # Make sure we're not also emitting the bare openai/ provider for the same model. - cfg_doc = yaml.safe_load(litellm_text) - pythia_entry = next( - m - for m in cfg_doc['model_list'] - if m['model_name'] == 'eleutherai/pythia-6.9b' - ) - assert ( - pythia_entry['litellm_params']['model'] - == 'text-completion-openai/eleutherai/pythia-6.9b' - ) - # Even though it's completions-only, the alias must remain in model_list - # so Open WebUI can still see/select it. - advertised = {m['model_name'] for m in cfg_doc['model_list']} - assert 'eleutherai/pythia-6.9b' in advertised - - -def test_litellm_chat_profile_does_not_use_text_completion_provider( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'qwen2-5-7b-instruct-turbo-default', inventory='1x96' - ) - render_compose_artifacts({'deployment': deployment}) - litellm_text = ( - tmp_path / 'state' / 'runtime' / 'litellm_config.yaml' - ).read_text() - assert 'text-completion-openai' not in litellm_text - assert 'openai/qwen/qwen2.5-7b-instruct-turbo' in litellm_text - # Chat models keep merge_reasoning_content_in_choices so Open WebUI can - # display reasoning when LiteLLM normalizes it into message content. - assert 'merge_reasoning_content_in_choices: true' in litellm_text - - -def test_pythia_routing_end_to_end_unit_check(tmp_path: Path) -> None: - """One-shot regression covering the Pythia routing contract. - - Compose label, LiteLLM provider, alias visibility, and the smoke-test - protocol picker should agree that this profile is completions. - """ - deployment = _deployment(tmp_path, 'helm-pythia-6.9b', inventory='1x96') - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - litellm_text = ( - tmp_path / 'state' / 'runtime' / 'litellm_config.yaml' - ).read_text() - cfg_doc = yaml.safe_load(litellm_text) - - # Compose label. - assert 'infer_stack.protocol_mode: "completions"' in compose_text - # LiteLLM provider. - pythia_entry = next( - m - for m in cfg_doc['model_list'] - if m['model_name'] == 'eleutherai/pythia-6.9b' - ) - assert ( - pythia_entry['litellm_params']['model'] - == 'text-completion-openai/eleutherai/pythia-6.9b' - ) - # Alias is still advertised (Open WebUI can see it). - assert any( - m['model_name'] == 'eleutherai/pythia-6.9b' - for m in cfg_doc['model_list'] - ) - # Resolved deployment service is completions and matches the alias target. - svc = deployment['services'][0] - assert svc['protocol_mode'] == 'completions' - assert 'eleutherai/pythia-6.9b' in svc['served_aliases'] - - -def test_qwen3_6_reasoning_profile_emits_reasoning_flags( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'pythia-qwen3.6-mixed-4x96', inventory='4x96' - ) - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - qwen_block = compose_text.split('vllm-qwen36-35b:', 1)[1].split( - 'vllm-pythia-69b:', 1 - )[0] - assert '--reasoning-parser' in qwen_block - assert '"qwen3"' in qwen_block - # vLLM CLI does not accept --enable-reasoning; the parser flag alone enables it. - assert '--enable-reasoning' not in qwen_block - - -def test_pythia_profile_does_not_emit_reasoning_flags(tmp_path: Path) -> None: - deployment = _deployment(tmp_path, 'helm-pythia-6.9b', inventory='1x96') - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - assert '--enable-reasoning' not in compose_text - assert '--reasoning-parser' not in compose_text - - -def test_mixed_profile_qwen_renders_tool_call_flags_pythia_does_not( - tmp_path: Path, -) -> None: - """Qwen3.6-35B-A3B model card requires: - - --reasoning-parser qwen3 - --enable-auto-tool-choice - --tool-call-parser qwen3_coder - --language-model-only - - Pythia services must not get any tool-call flags. - """ - deployment = _deployment( - tmp_path, 'pythia-qwen3.6-mixed-4x96', inventory='4x96' - ) - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - blocks = _split_compose_blocks(compose_text) - - qwen_block = blocks['vllm-qwen36-35b'] - # All four model-card-recommended flags are present. - assert '--enable-auto-tool-choice' in qwen_block - assert '--tool-call-parser' in qwen_block - assert '"qwen3_coder"' in qwen_block - assert '--reasoning-parser' in qwen_block - assert '"qwen3"' in qwen_block - assert '--language-model-only' in qwen_block - # qwen3_xml is only a manual fallback, never rendered for this profile. - assert 'qwen3_xml' not in qwen_block - - for pythia_name in ('vllm-pythia-69b', 'vllm-pythia-28b'): - block = blocks[pythia_name] - assert '--enable-auto-tool-choice' not in block, pythia_name - assert '--tool-call-parser' not in block, pythia_name - assert 'qwen3_coder' not in block, pythia_name - assert 'qwen3_xml' not in block, pythia_name - - -def test_mixed_profile_qwen_command_arg_order_matches_model_card( - tmp_path: Path, -) -> None: - """Lock down the exact command-line tokens for the Qwen service. - - The Qwen3.6-35B-A3B model card lists these together; we want each - parser flag's value to immediately follow the flag (vLLM positional - contract). This catches regressions where a value gets dropped or - reordered into another argument. - """ - deployment = _deployment( - tmp_path, 'pythia-qwen3.6-mixed-4x96', inventory='4x96' - ) - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - blocks = _split_compose_blocks(compose_text) - qwen_block = blocks['vllm-qwen36-35b'] - - # Each flag is immediately followed by its value on the next rendered line. - for flag, value in ( - ('--reasoning-parser', 'qwen3'), - ('--tool-call-parser', 'qwen3_coder'), - ): - assert flag in qwen_block - flag_idx = qwen_block.index(flag) - following = qwen_block[flag_idx:] - first_value_line = next( - line.strip() for line in following.splitlines()[1:] if line.strip() - ) - assert first_value_line == f'- "{value}"', (flag, first_value_line) - - -def test_mixed_profile_litellm_config_unchanged_by_tool_calling( - tmp_path: Path, -) -> None: - """Adding tool_calling on a service must not perturb the LiteLLM config.""" - deployment = _deployment( - tmp_path, 'pythia-qwen3.6-mixed-4x96', inventory='4x96' - ) - render_compose_artifacts({'deployment': deployment}) - cfg_doc = yaml.safe_load( - (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').read_text() - ) - by_alias = { - m['model_name']: m['litellm_params'] for m in cfg_doc['model_list'] - } - - qwen = by_alias['qwen3.6-35b-a3b'] - assert qwen['model'] == 'openai/qwen3.6-35b-a3b' - assert qwen.get('merge_reasoning_content_in_choices') is True - # No tool-calling-specific keys leak into LiteLLM params. - assert 'tools' not in qwen - assert 'tool_choice' not in qwen - assert 'enable_auto_tool_choice' not in qwen - - assert ( - by_alias['eleutherai/pythia-6.9b']['model'] - == 'text-completion-openai/eleutherai/pythia-6.9b' - ) - assert ( - by_alias['eleutherai/pythia-2.8b-v0']['model'] - == 'text-completion-openai/eleutherai/pythia-2.8b-v0' - ) - - -def test_mixed_profile_compose_service_names_are_stable(tmp_path: Path) -> None: - deployment = _deployment( - tmp_path, 'pythia-qwen3.6-mixed-4x96', inventory='4x96' - ) - render_compose_artifacts({'deployment': deployment}) - blocks = _split_compose_blocks( - (tmp_path / 'generated' / 'docker-compose.yml').read_text() - ) - infer_stacks = sorted(name for name in blocks if name.startswith('vllm-')) - assert infer_stacks == [ - 'vllm-pythia-28b', - 'vllm-pythia-69b', - 'vllm-qwen36-35b', - ] - - -def test_default_pythia_profiles_have_no_chat_compat(tmp_path: Path) -> None: - for profile_name in ( - 'helm-pythia-6.9b', - 'helm-pythia-2.8b-v0', - 'helm-pythia-1b-v0', - ): - deployment = _deployment(tmp_path, profile_name, inventory='1x96') - render_compose_artifacts({'deployment': deployment}) - cfg_doc = yaml.safe_load( - (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').read_text() - ) - for entry in cfg_doc['model_list']: - params = entry['litellm_params'] - assert 'roles' not in params, profile_name - assert 'initial_prompt_value' not in params, profile_name - assert 'final_prompt_value' not in params, profile_name - # Resolved service has the flag off. - for svc in deployment['services']: - assert svc['chat_compat_enabled'] is False, profile_name - - -def test_pythia_inspect_mmlu_compat_renders_completions_with_litellm_template( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'pythia-inspect-mmlu-compat', inventory='2x96' - ) - render_compose_artifacts({'deployment': deployment}) - - # Both Pythia services keep protocol_mode=completions. - for svc in deployment['services']: - assert svc['protocol_mode'] == 'completions', svc['service_name'] - assert svc['chat_compat_enabled'] is True, svc['service_name'] - assert svc['chat_compat_strategy'] == 'flat_messages', svc[ - 'service_name' - ] - - cfg_doc = yaml.safe_load( - (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').read_text() - ) - by_alias = { - m['model_name']: m['litellm_params'] for m in cfg_doc['model_list'] - } - - for alias, served in ( - ('eleutherai/pythia-6.9b', 'eleutherai/pythia-6.9b'), - ('eleutherai/pythia-2.8b-v0', 'eleutherai/pythia-2.8b-v0'), - ): - params = by_alias[alias] - # Still a text-completion upstream — chat_compat does not change that. - assert params['model'] == f'text-completion-openai/{served}' - # Flat-messages prompt template fields are present and use newline-only post_message. - assert params['initial_prompt_value'] == '' - assert params['final_prompt_value'] == '' - assert set(params['roles']) == {'system', 'user', 'assistant'} - for role, role_cfg in params['roles'].items(): - assert role_cfg['pre_message'] == '', (alias, role) - assert role_cfg['post_message'] == '\n', (alias, role) - # No reasoning passthrough on completions entries. - assert 'merge_reasoning_content_in_choices' not in params - - -def test_pythia_inspect_mmlu_compat_does_not_render_chat_template_flags_in_compose( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'pythia-inspect-mmlu-compat', inventory='2x96' - ) - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - assert '--chat-template' not in compose_text - assert '--chat-template-content-format' not in compose_text - - -def test_no_compose_service_renders_chat_template_flags(tmp_path: Path) -> None: - """chat_compat is LiteLLM-only; Compose must never render --chat-template.""" - for profile_name, inventory in ( - ('pythia-inspect-mmlu-compat', '2x96'), - ('pythia-qwen3.6-mixed-4x96', '4x96'), - ('qwen3.6-35b-a3b-dual-tp2-4x96', '4x96'), - ('helm-pythia-6.9b', '1x96'), - ('qwen2-5-7b-instruct-turbo-default', '1x96'), - ): - deployment = _deployment(tmp_path, profile_name, inventory=inventory) - render_compose_artifacts({'deployment': deployment}) - compose_text = ( - tmp_path / 'generated' / 'docker-compose.yml' - ).read_text() - assert '--chat-template' not in compose_text, profile_name - - -def test_mixed_profile_pythia_services_get_chat_compat_qwen_does_not( - tmp_path: Path, -) -> None: - """In `pythia-qwen3.6-mixed-4x96`, only the two Pythia LiteLLM entries - should carry the flat-messages prompt-template fields. The Qwen chat - entry must remain a plain `openai/...` route with no `roles` block.""" - deployment = _deployment( - tmp_path, 'pythia-qwen3.6-mixed-4x96', inventory='4x96' - ) - render_compose_artifacts({'deployment': deployment}) - - services = {s['service_name']: s for s in deployment['services']} - assert services['pythia-69b']['chat_compat_enabled'] is True - assert services['pythia-69b']['chat_compat_strategy'] == 'flat_messages' - assert services['pythia-28b']['chat_compat_enabled'] is True - assert services['pythia-28b']['chat_compat_strategy'] == 'flat_messages' - assert services['qwen36-35b']['chat_compat_enabled'] is False - # vLLM commands are unchanged: no --chat-template anywhere. - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - assert '--chat-template' not in compose_text - - cfg_doc = yaml.safe_load( - (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').read_text() - ) - by_alias = { - m['model_name']: m['litellm_params'] for m in cfg_doc['model_list'] - } - - for alias, served in ( - ('eleutherai/pythia-6.9b', 'eleutherai/pythia-6.9b'), - ('eleutherai/pythia-2.8b-v0', 'eleutherai/pythia-2.8b-v0'), - ): - params = by_alias[alias] - assert params['model'] == f'text-completion-openai/{served}' - assert params['initial_prompt_value'] == '' - assert params['final_prompt_value'] == '' - assert set(params['roles']) == {'system', 'user', 'assistant'} - for role_cfg in params['roles'].values(): - assert role_cfg['pre_message'] == '' - assert role_cfg['post_message'] == '\n' - - qwen_params = by_alias['qwen3.6-35b-a3b'] - assert qwen_params['model'] == 'openai/qwen3.6-35b-a3b' - assert 'roles' not in qwen_params - assert 'initial_prompt_value' not in qwen_params - assert 'final_prompt_value' not in qwen_params - - -def test_qwen_chat_profiles_are_unaffected_by_chat_compat( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'qwen2-5-7b-instruct-turbo-default', inventory='1x96' - ) - render_compose_artifacts({'deployment': deployment}) - cfg_doc = yaml.safe_load( - (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').read_text() - ) - for entry in cfg_doc['model_list']: - params = entry['litellm_params'] - assert params['model'].startswith('openai/') - assert 'roles' not in params - assert 'initial_prompt_value' not in params - assert 'final_prompt_value' not in params - assert deployment['services'][0]['chat_compat_enabled'] is False - - -def test_litellm_config_for_chat_compat_profile_is_valid_yaml( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'pythia-inspect-mmlu-compat', inventory='2x96' - ) - render_compose_artifacts({'deployment': deployment}) - text = (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').read_text() - cfg_doc = yaml.safe_load(text) - assert isinstance(cfg_doc, dict) - assert 'model_list' in cfg_doc - assert all('litellm_params' in m for m in cfg_doc['model_list']) - - -def test_no_profile_renders_unsupported_enable_reasoning_flag( - tmp_path: Path, -) -> None: - """Regression: vLLM CLI rejects --enable-reasoning.""" - for profile_name, inventory in ( - ('pythia-qwen3.6-mixed-4x96', '4x96'), - ('qwen3.6-35b-a3b-dual-tp2-4x96', '4x96'), - ('helm-pythia-6.9b', '1x96'), - ('qwen2-5-7b-instruct-turbo-default', '1x96'), - ): - deployment = _deployment(tmp_path, profile_name, inventory=inventory) - render_compose_artifacts({'deployment': deployment}) - compose_text = ( - tmp_path / 'generated' / 'docker-compose.yml' - ).read_text() - assert '--enable-reasoning' not in compose_text, profile_name - - -def test_litellm_keeps_merge_reasoning_for_reasoning_models( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'pythia-qwen3.6-mixed-4x96', inventory='4x96' - ) - render_compose_artifacts({'deployment': deployment}) - cfg_doc = yaml.safe_load( - (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').read_text() - ) - qwen_entry = next( - m for m in cfg_doc['model_list'] if m['model_name'] == 'qwen3.6-35b-a3b' - ) - assert ( - qwen_entry['litellm_params']['merge_reasoning_content_in_choices'] - is True - ) - - -def test_pythia_qwen3_6_mixed_profile_resolves_on_4x96(tmp_path: Path) -> None: - deployment = _deployment( - tmp_path, 'pythia-qwen3.6-mixed-4x96', inventory='4x96' - ) - services = {s['service_name']: s for s in deployment['services']} - assert set(services) == {'qwen36-35b', 'pythia-69b', 'pythia-28b'} - - qwen = services['qwen36-35b'] - assert qwen['protocol_mode'] == 'chat' - assert qwen['gpu_indices'] == [0, 1] - assert qwen['tensor_parallel_size'] == 2 - assert qwen['reasoning_enabled'] is True - assert qwen['reasoning_parser'] == 'qwen3' - - p69 = services['pythia-69b'] - assert p69['protocol_mode'] == 'completions' - assert p69['gpu_indices'] == [2] - assert p69['reasoning_enabled'] is False - - p28 = services['pythia-28b'] - assert p28['protocol_mode'] == 'completions' - assert p28['gpu_indices'] == [3] - - -def test_pythia_qwen3_6_mixed_profile_renders_compose_and_litellm( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'pythia-qwen3.6-mixed-4x96', inventory='4x96' - ) - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - blocks = _split_compose_blocks(compose_text) - - # Three vLLM services and two Postgres services. - infer_stacks = [name for name in blocks if name.startswith('vllm-')] - assert sorted(infer_stacks) == [ - 'vllm-pythia-28b', - 'vllm-pythia-69b', - 'vllm-qwen36-35b', - ] - assert 'postgres-open-webui' in blocks - assert 'postgres-litellm' in blocks - assert 'postgres-init' not in blocks - - cfg_doc = yaml.safe_load( - (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').read_text() - ) - by_alias = { - m['model_name']: m['litellm_params']['model'] - for m in cfg_doc['model_list'] - } - assert by_alias['qwen3.6-35b-a3b'].startswith('openai/') - assert ( - by_alias['eleutherai/pythia-6.9b'] - == 'text-completion-openai/eleutherai/pythia-6.9b' - ) - assert ( - by_alias['eleutherai/pythia-2.8b-v0'] - == 'text-completion-openai/eleutherai/pythia-2.8b-v0' - ) - - -def test_qwen3_6_dual_tp2_4x96_profile_resolves_and_renders( - tmp_path: Path, -) -> None: - deployment = _deployment( - tmp_path, 'qwen3.6-35b-a3b-dual-tp2-4x96', inventory='4x96' - ) - services = {s['service_name']: s for s in deployment['services']} - assert set(services) == {'qwen36-35b-gpu01', 'qwen36-35b-gpu23'} - - gpu01 = services['qwen36-35b-gpu01'] - gpu23 = services['qwen36-35b-gpu23'] - assert gpu01['gpu_indices'] == [0, 1] - assert gpu23['gpu_indices'] == [2, 3] - assert ( - gpu01['tensor_parallel_size'] == 2 - and gpu23['tensor_parallel_size'] == 2 - ) - assert gpu01['protocol_mode'] == 'chat' and gpu23['protocol_mode'] == 'chat' - assert ( - gpu01['reasoning_enabled'] is True - and gpu01['reasoning_parser'] == 'qwen3' - ) - - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - blocks = _split_compose_blocks(compose_text) - assert 'vllm-qwen36-35b-gpu01' in blocks - assert 'vllm-qwen36-35b-gpu23' in blocks - for name in ('vllm-qwen36-35b-gpu01', 'vllm-qwen36-35b-gpu23'): - assert '--reasoning-parser' in blocks[name] - assert '"qwen3"' in blocks[name] - assert '--language-model-only' in blocks[name] - assert '--enable-reasoning' not in blocks[name] - - cfg_doc = yaml.safe_load( - (tmp_path / 'state' / 'runtime' / 'litellm_config.yaml').read_text() - ) - by_alias = { - m['model_name']: m['litellm_params']['model'] - for m in cfg_doc['model_list'] - } - assert by_alias['qwen3.6-35b-a3b-gpu01'].startswith('openai/') - assert by_alias['qwen3.6-35b-a3b-gpu23'].startswith('openai/') - - -def test_kubeai_protocol_validation_applies(tmp_path: Path) -> None: - from infer_stack.config import initial_config - - cfg = initial_config() - cfg['backend'] = 'kubeai' - cfg['resource_profiles'] = { - 'gpu-single-default': { - 'node_selector': {'nvidia.com/gpu.product': 'X'}, - 'requests': {'nvidia.com/gpu': 1}, - 'limits': {'nvidia.com/gpu': 1}, - }, - } - cfg['profiles'] = { - 'broken-pythia-chat-kubeai': { - 'description': 'Synthetic invalid profile: chat on a completions-only model.', - 'base_model': 'pythia-1b-v0', - 'public_name': 'broken-pythia-chat-kubeai', - 'protocol_mode': 'chat', - 'resource_profile': 'gpu-single-default:1', - } - } - deployment = resolve( - cfg, - inventory=simulate_inventory('1x96'), - profile_name='broken-pythia-chat-kubeai', - ) - report = validate_resolved(deployment) - assert report['ok'] is False - joined = ' | '.join(report['errors']) - assert 'broken-pythia-chat-kubeai' in joined - assert 'completions' in joined - - -def test_load_profile_contract_uses_public_loader_for_gpt_oss_variants( - tmp_path: Path, -) -> None: - _write_root_config(tmp_path) - completions = load_profile_contract( - 'gpt-oss-20b-completions', - simulate_hardware_spec='1x96', - ) - chat = load_profile_contract( - 'gpt-oss-20b-chat', - simulate_hardware_spec='1x96', - ) - assert completions['services'][0]['protocol']['mode'] == 'completions' - assert chat['services'][0]['protocol']['mode'] == 'chat' - - -def test_compose_infer_stack_persists_caches_and_uses_host_ipc( - tmp_path: Path, -) -> None: - """Warm-restart caches and shared memory must be wired into every vLLM service. - - - Hugging Face cache mount (existing behaviour) avoids re-downloading weights. - - vLLM cache mount + ``VLLM_CACHE_ROOT`` env let torch.compile artifacts - survive container restarts. - - ``ipc: host`` is required by vLLM's Docker guidance to give the engine - adequate shared memory for tensor-parallel and worker IPC. - """ - deployment = _deployment(tmp_path, 'gpt-oss-20b-chat') - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - blocks = _split_compose_blocks(compose_text) - vllm_blocks = { - name: body for name, body in blocks.items() if name.startswith('vllm-') - } - assert vllm_blocks, 'expected at least one rendered vLLM service' - - expected_hf_mount = f'{tmp_path}/state/hf-cache:/root/.cache/huggingface' - expected_vllm_mount = f'{tmp_path}/state/vllm-cache:/root/.cache/vllm' - for name, body in vllm_blocks.items(): - assert 'VLLM_CACHE_ROOT: /root/.cache/vllm' in body, name - assert expected_hf_mount in body, name - assert expected_vllm_mount in body, name - assert 'ipc: host' in body, name - - -def test_compose_non_infer_stacks_do_not_get_vllm_cache_mount( - tmp_path: Path, -) -> None: - """The vLLM cache mount and host IPC must not leak into Postgres / litellm / open-webui.""" - deployment = _deployment(tmp_path, 'gpt-oss-20b-chat') - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - blocks = _split_compose_blocks(compose_text) - for name in ( - 'postgres-open-webui', - 'postgres-litellm', - 'litellm', - 'open-webui', - ): - body = blocks[name] - assert '/root/.cache/vllm' not in body, name - assert 'VLLM_CACHE_ROOT' not in body, name - assert 'ipc: host' not in body, name - - -def test_default_state_paths_include_vllm_cache(tmp_path: Path) -> None: - from infer_stack.config import default_state_paths, normalized_state - - # data_root() is pinned to tmp_path by the autouse fixture. - paths = default_state_paths() - assert 'vllm_cache' in paths - assert paths['vllm_cache'].endswith('/vllm-cache') - assert paths['vllm_cache'].startswith(str(tmp_path)) - - normalized = normalized_state({'vllm_cache': 'custom/vllm'}) - assert normalized['vllm_cache'] == str(tmp_path / 'custom' / 'vllm') - - -def test_allowed_gpus_filters_inventory_to_real_indices(tmp_path: Path) -> None: - """``effective_inventory(--allowed-gpus 1,3)`` against a 4-GPU sim keeps real indices. - - ``placement.strategy: first_fit`` then picks from {1, 3}; the rendered - compose stack pins ``device_ids`` to those physical indices (no - renumbering). - """ - from infer_stack.cli import effective_inventory - - inv = effective_inventory( - {'simulate_hardware': '4x24', 'allowed_gpus': '1,3'} - ) - assert inv is not None - assert inv['gpu_count'] == 2 - assert [g['index'] for g in inv['gpus']] == [1, 3] - - cfg = _cfg(tmp_path) - deployment = resolve( - cfg, - inventory=inv, - profile_name='test-multi-gpu', - ) - svc = deployment['services'][0] - assert svc['gpu_indices'] == [1, 3] - assert svc['tensor_parallel_size'] == 2 - - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - assert 'device_ids: ["1", "3"]' in compose_text - - -def test_allowed_gpus_too_few_for_profile_surfaces_placement_error( - tmp_path: Path, -) -> None: - """Asking for 2-GPU placement when only 1 GPU is allowed must fail validation cleanly.""" - from infer_stack.cli import effective_inventory - - inv = effective_inventory( - {'simulate_hardware': '4x24', 'allowed_gpus': '1'} - ) - assert inv is not None and inv['gpu_count'] == 1 - - cfg = _cfg(tmp_path) - deployment = resolve(cfg, inventory=inv, profile_name='test-multi-gpu') - report = validate_resolved(deployment) - assert report['ok'] is False - joined = ' | '.join(report['errors']) - assert 'need 2 GPUs but only 1 available' in joined - - -def test_output_generated_dir_redirects_compose_render(tmp_path: Path) -> None: - """Configured output.generated_dir takes precedence over the default.""" - cfg = _cfg(tmp_path) - cfg['output'] = {'generated_dir': str(tmp_path / 'custom-out')} - deployment = resolve( - cfg, - inventory=simulate_inventory('1x96'), - profile_name='qwen2-5-7b-instruct-turbo-default', - ) - render_compose_artifacts({'deployment': deployment}) - assert (tmp_path / 'custom-out' / 'docker-compose.yml').exists() - assert (tmp_path / 'custom-out' / '.env').exists() - # The repo-local generated/ is not touched. - assert not (tmp_path / 'generated').exists() - - -def test_output_generated_dir_redirects_kubeai_render(tmp_path: Path) -> None: - cfg = _cfg(tmp_path, backend='kubeai') - cfg['output'] = {'generated_dir': str(tmp_path / 'custom-out')} - deployment = resolve( - cfg, - inventory=simulate_inventory('1x96'), - profile_name='qwen2-5-7b-instruct-turbo-default', - ) - render_kubeai_artifacts({'deployment': deployment}) - assert (tmp_path / 'custom-out' / 'kubeai' / 'models.yaml').exists() - assert (tmp_path / 'custom-out' / 'kubeai' / 'kubeai-values.yaml').exists() - assert not (tmp_path / 'generated').exists() - - -def test_openwebui_auth_defaults_to_disabled_in_compose_render( - tmp_path: Path, -) -> None: - """Auth-less Open WebUI is the out-of-the-box experience for compose.""" - deployment = _deployment(tmp_path, 'gpt-oss-20b-chat') - assert deployment['open_webui']['auth'] is False - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - blocks = _split_compose_blocks(compose_text) - assert 'WEBUI_AUTH: "False"' in blocks['open-webui'] - - -def test_openwebui_auth_can_be_enabled_via_config(tmp_path: Path) -> None: - cfg = _cfg(tmp_path) - cfg['open_webui'] = {'auth': True} - deployment = resolve( - cfg, - inventory=simulate_inventory('1x96'), - profile_name='qwen2-5-7b-instruct-turbo-default', - ) - assert deployment['open_webui']['auth'] is True - render_compose_artifacts({'deployment': deployment}) - compose_text = (tmp_path / 'generated' / 'docker-compose.yml').read_text() - blocks = _split_compose_blocks(compose_text) - assert 'WEBUI_AUTH: "True"' in blocks['open-webui'] - - -def test_normalized_output_anchors_relative_paths_on_data_root( - tmp_path: Path, -) -> None: - from infer_stack.config import normalized_output - - # data_root() is pinned to tmp_path by the autouse fixture. - normalized = normalized_output({'generated_dir': 'out'}) - assert normalized['generated_dir'] == str(tmp_path / 'out') - - normalized = normalized_output({'generated_dir': '/abs/path'}) - assert normalized['generated_dir'] == '/abs/path' - - -def test_catalog_model_path_env_and_directory_overlays( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - from infer_stack.config import catalog_files, merged_catalogs - - save_yaml( - tmp_path / 'models.yaml', - { - 'vllm_models': { - 'base-only': {'hf_model_id': 'example/base-only'}, - 'shared': {'hf_model_id': 'example/base-shared'}, - }, - 'profiles': { - 'base-profile': {'description': 'from base models.yaml'} - }, - }, - ) - config_path_dir = tmp_path / 'repo-catalogs' - config_path_dir.mkdir() - save_yaml( - config_path_dir / 'incubilate.models.yaml', - { - 'vllm_models': { - 'config-only': {'hf_model_id': 'example/config-only'}, - 'shared': {'hf_model_id': 'example/config-shared'}, - }, - 'profiles': { - 'config-profile': {'description': 'from config model_path'} - }, - }, - ) - # Directory expansion is intentionally narrow; unrelated YAML should not be - # treated as a catalog just because it lives beside catalog overlays. - save_yaml( - config_path_dir / 'config.yaml', - {'vllm_models': {'should-not-load': {'hf_model_id': 'bad/bad'}}}, - ) - - env_path_dir = tmp_path / 'env-catalogs' - env_path_dir.mkdir() - save_yaml( - env_path_dir / 'experiment.profiles.yaml', - { - 'vllm_models': { - 'env-only': {'hf_model_id': 'example/env-only'}, - 'shared': {'hf_model_id': 'example/env-shared'}, - }, - 'profiles': { - 'env-profile': {'description': 'from INFER_STACK_MODEL_PATH'} - }, - }, - ) - - cfg = initial_config() - cfg['catalog']['model_path'] = 'repo-catalogs' - monkeypatch.setenv('INFER_STACK_MODEL_PATH', str(env_path_dir)) - - catalog_paths = [path.name for path in catalog_files(cfg)] - assert catalog_paths == [ - 'models.yaml', - 'incubilate.models.yaml', - 'experiment.profiles.yaml', - ] - - cats = merged_catalogs(cfg) - assert 'base-only' in cats['vllm_models'] - assert 'config-only' in cats['vllm_models'] - assert 'env-only' in cats['vllm_models'] - assert 'should-not-load' not in cats['vllm_models'] - assert cats['vllm_models']['shared']['hf_model_id'] == 'example/env-shared' - assert 'base-profile' in cats['profiles'] - assert 'config-profile' in cats['profiles'] - assert 'env-profile' in cats['profiles'] diff --git a/tests/test_tui.py b/tests/test_tui.py new file mode 100644 index 00000000..12bfcdb7 --- /dev/null +++ b/tests/test_tui.py @@ -0,0 +1,921 @@ +"""Headless tests for the optional Textual TUI (skipped if textual absent).""" + +from __future__ import annotations + +import asyncio +import json + +import pytest + +pytest.importorskip('textual') + +CATALOG = { + 'models': {'qc': {'source': 'hf://Qwen/Qwen2.5-Coder-32B-Instruct'}}, + 'endpoints': { + 'qwen-coder': {'engine': 'vllm', 'model': 'qc'}, + 'qwen-fast': {'engine': 'vllm', 'model': 'qc'}, + }, +} + + +def _ctx(): + from infer_stack.leasing import ( + Catalog, + Controller, + Ledger, + NullBackend, + SqliteStore, + ) + + catalog = Catalog.from_dict(CATALOG) + controller = Controller(Ledger(SqliteStore(':memory:')), NullBackend()) + return controller, catalog + + +class _FakeProc: + """Stands in for a `docker compose logs -f` process.""" + + def __init__(self, lines): + self.stdout = iter(lines) + self.terminated = False + + def terminate(self): + self.terminated = True + + +def _run(scenario): + asyncio.run(scenario()) + + +def test_tui_panes_list_catalog_leases_and_deployments(): + from textual.widgets import DataTable + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + controller.acquire('alice', catalog.resolve_names(['qwen-coder'])) + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + eps = app.query_one('#endpoints', DataTable) + assert eps.row_count == 2 # both catalog endpoints + assert app._endpoint_names == ['qwen-coder', 'qwen-fast'] + assert app.query_one('#models', DataTable).row_count == 1 + assert app.query_one('#leases', DataTable).row_count == 1 + assert app.query_one('#deployments', DataTable).row_count == 1 + + _run(scenario) + + +def test_tui_relationship_columns_link_lease_to_deployment(): + from textual.widgets import DataTable + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + out = controller.acquire('alice', catalog.resolve_names(['qwen-coder'])) + gid = out.lease.deployment_ids[0] + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + leases = app.query_one('#leases', DataTable) + deployments = app.query_one('#deployments', DataTable) + assert 'deployment' in [str(c.label) for c in leases.columns.values()] + glabels = [str(c.label) for c in deployments.columns.values()] + assert 'leases' in glabels and 'held by' in glabels + # the lease row names the deployment id; the deployment row counts + # the lease (1) and shows the owner — the many-to-one join. + assert gid in leases.get_row_at(0) + assert '1' in deployments.get_row_at(0) + assert 'alice' in deployments.get_row_at(0) + # selecting a lease explains the link in the status bar + app.query_one('#leases').focus() + app.query_one('#leases', DataTable).move_cursor(row=0) + await pilot.pause() + status = str(app.query_one('#status').render()) + assert 'deployment' in status + + _run(scenario) + + +def test_tui_acquire_from_catalog_creates_a_lease(): + from infer_stack.leasing import LeaseState + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + app.query_one('#endpoints').focus() + await pilot.press('s') # acquire the selected endpoint + await app.workers.wait_for_complete() + await pilot.pause() + + _run(scenario) + leases, _ = controller.ledger.status() + assert len(leases) == 1 + assert leases[0].state == LeaseState.ACTIVE + assert 'qwen-coder' in leases[0].endpoints # first row, sorted + + +def test_tui_panes_are_keyboard_resizable(): + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + w0, h0 = app._sidebar_w, app._log_h + await pilot.press('right_square_bracket') # wider sidebar + await pilot.press('minus') # shorter logs + await pilot.pause() + assert app._sidebar_w == w0 + 4 + assert app._log_h == h0 - 2 + + _run(scenario) + + +def test_tui_docker_pane_has_logs_and_containers_tabs(): + from textual.widgets import Collapsible, DataTable, TabbedContent + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + # docker is a collapsible pane with Logs/Containers/Control tabs; + # system is its own collapsed pane; API is a top-level tab now. + tabs = app.query_one('#docker-tabs', TabbedContent) + assert {p.id for p in tabs.query('TabPane')} == { + 'tab-logs', 'tab-containers', 'tab-control' + } + assert app.query_one('#docker', Collapsible) + assert app.query_one('#system', Collapsible).collapsed + top = app.query_one('#top', TabbedContent) + assert 'tab-api' in {p.id for p in top.query('TabPane')} + # the containers ps view carries the docker-ps columns + ps = app.query_one('#ps', DataTable) + labels = [str(c.label) for c in ps.columns.values()] + assert 'status (uptime)' in labels + assert 'created' in labels + assert 'container id' in labels + + _run(scenario) + + +def test_tui_endpoint_action_buttons_fit_the_sidebar(): + # Regression: 4 buttons at width:1fr were defeated by Button's default + # min-width (16) and overflowed off the narrow sidebar — Edit/Remove only + # appeared after widening. They must all fit at the default sidebar width. + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test(size=(120, 40)) as pilot: + await pilot.pause() + sb = app.query_one('#sidebar').region + for bid in ('#btn-acquire', '#btn-add-endpoint', '#btn-edit-endpoint', + '#btn-remove-endpoint'): + r = app.query_one(bid).region + assert r.width > 0 and r.x >= sb.x and \ + r.x + r.width <= sb.x + sb.width, f'{bid} overflows sidebar' + + _run(scenario) + + +def test_tui_renders_when_action_buttons_are_squeezed_narrow(): + # Regression: the min-width:0 action buttons can shrink until their content + # box is ~2 cells. Textual's Button carries line-pad:1 and folds the label + # at (width - line_pad*2); at width 2 that is 0 and rich's chop_cells does + # range(0, n, 0) -> ValueError, crashing the whole render. text-wrap:nowrap + # on the compact buttons skips that fold path. Sweep the sidebar across the + # crash zone (region width ~4 / content ~2) and force a real composite. + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test(size=(120, 40)) as pilot: + await pilot.pause() + sidebar = app.query_one('#sidebar') + for width in range(10, 25): + app._sidebar_w = width + sidebar.styles.width = width + await pilot.pause() + # export_screenshot drives the full StylesCache render path that + # raised in the field; it must not blow up at any narrow width. + app.export_screenshot() + + _run(scenario) + + +def test_tui_expanding_system_pane_polls_gpus(): + # Regression: the polling gate trusted Collapsible.Toggled (which doesn't + # fire on every path), so expanding System never flipped the gate and the + # GPU table stayed empty. _sync_pane_state reads the live state instead. + from textual.widgets import Collapsible, DataTable + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + app._gpu_rows = lambda: [['0', 'RTX', '15', '5000', '24576', '56']] + app.query_one('#system', Collapsible).collapsed = False # expand + app._sync_pane_state() + assert app._collapsed['system'] is False + assert 'gpus' in app._collect() # now polled + app._refresh_now() + assert app.query_one('#gpus', DataTable).row_count == 1 + + _run(scenario) + + +def test_tui_monitor_panes_are_collapsible(): + from textual.widgets import Collapsible + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + # the four monitor panes collapse (docker/system already did) + for pid in ('#leases-pane', '#deployments-pane', '#docker', '#system'): + pane = app.query_one(pid, Collapsible) + pane.collapsed = True + await pilot.pause() + assert app.query_one('#leases-pane', Collapsible).collapsed + assert app.query_one('#deployments-pane', Collapsible).collapsed + + _run(scenario) + + +def test_tui_panes_drag_resize(): + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + w0, h0 = app._sidebar_w, app._log_h + m0 = app._models_h + app._drag_sidebar(6) # pull the vertical splitter right + app._drag_logs(3) # pull the horizontal splitter down + app._drag_models(2) # catalog endpoints|models splitter + assert app._sidebar_w == w0 + 6 + assert app._log_h == h0 - 3 # down = shorter logs + assert app._models_h == m0 - 2 # drag down = bar down = models shorter + + _run(scenario) + + +def test_tui_dividers_have_a_grab_area(): + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test(size=(120, 40)) as pilot: + await pilot.pause() + # A 0-size divider can't be grabbed; both must span their cross-axis. + assert app.query_one('#vsplit').region.height > 1 + assert app.query_one('#hsplit').region.width > 1 + assert app.query_one('#csplit').region.width > 1 # endpoints|models + + _run(scenario) + + +def test_tui_drag_resizes_sidebar(): + from textual import events + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test(size=(120, 40)) as pilot: + await pilot.pause() + div = app.query_one('#vsplit') + w0 = app._sidebar_w + await pilot.mouse_down('#vsplit') + assert app.mouse_captured is div # the bar grabbed the mouse + div.post_message(events.MouseMove( + widget=div, x=0, y=0, delta_x=5, delta_y=0, button=0, + shift=False, meta=False, ctrl=False, + screen_x=div.region.x + 5, screen_y=div.region.y, + )) + await pilot.pause() + await pilot.mouse_up('#vsplit') + assert app._sidebar_w == w0 + 5 + + _run(scenario) + + +def test_tui_pane_scoped_action_buttons(): + from textual.widgets import Button + + from infer_stack.leasing import LeaseState + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + # Acquire lives under the catalog; release/evict under their tables. + assert app.query_one('#btn-acquire', Button) + assert app.query_one('#btn-release', Button) + assert app.query_one('#btn-evict', Button) + app.query_one('#endpoints').move_cursor(row=0) + await pilot.click('#btn-acquire') + await app.workers.wait_for_complete() + await pilot.pause() + + _run(scenario) + leases, _ = controller.ledger.status() + assert len(leases) == 1 and leases[0].state == LeaseState.ACTIVE + + +def test_tui_add_model_wizard_writes_catalog(tmp_path): + import yaml + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + catalog_path = tmp_path / 'catalog.yaml' + catalog_path.write_text(yaml.safe_dump(CATALOG)) + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None, + catalog_path=str(catalog_path)) + async with app.run_test() as pilot: + await pilot.pause() + app._on_add_model({'name': 'newmod', + 'source': 'hf://org/NewModel'}) + await pilot.pause() + assert 'newmod' in app.catalog.models # reloaded in memory + + _run(scenario) + on_disk = yaml.safe_load(catalog_path.read_text()) + assert on_disk['models']['newmod']['source'] == 'hf://org/NewModel' + + +def test_tui_has_top_level_dashboard_and_settings_tabs(): + from textual.widgets import TabbedContent + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + top = app.query_one('#top', TabbedContent) + ids = {p.id for p in top.query('TabPane')} + assert {'tab-dashboard', 'tab-settings'} <= ids + # dashboard widgets still resolve (composed via the helper) + assert app.query_one('#endpoints') + assert app.query_one('#set-backend') # settings form present + + _run(scenario) + + +def test_tui_settings_save_writes_yaml(tmp_path): + import yaml + from textual.widgets import Input, Select + + from infer_stack import paths + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + paths.set_config_root(tmp_path) + paths.set_data_root(tmp_path) + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + app.query_one('#set-backend', Select).value = 'compose' + app.query_one('#set-data-dir', Input).value = str(tmp_path / 'data') + app.query_one('#set-ui', Select).value = 'off' + app._on_save_settings() + await pilot.pause() + + try: + _run(scenario) + saved = yaml.safe_load((tmp_path / 'settings.yaml').read_text()) + assert saved['backend'] == 'compose' + assert saved['data_dir'] == str(tmp_path / 'data') + assert saved['ui'] is False + finally: + paths.set_config_root(None) + paths.set_data_root(None) + + +def test_tui_endpoint_entry_builds_runtime_from_advanced_params(): + from infer_stack.tui import InferStackTUI + + entry = InferStackTUI._endpoint_entry({ + 'model': 'qc', 'engine': 'vllm', + 'tensor_parallel': 2, 'max_model_len': 8192, 'gpu_mem': 0.4, + 'extra_args': '--dtype=half --enforce-eager', 'reclaim': 'keep-warm', + }) + assert entry['runtime']['tensor_parallel_size'] == 2 + assert entry['runtime']['max_model_len'] == 8192 + assert entry['runtime']['gpu_memory_utilization'] == 0.4 + assert entry['runtime']['extra_args'] == ['--dtype=half', '--enforce-eager'] + assert entry['reclaim'] == {'policy': 'keep-warm'} + + +def test_tui_endpoint_entry_data_parallel_and_ollama(): + from infer_stack.tui import InferStackTUI + + v = InferStackTUI._endpoint_entry({ + 'model': 'm', 'engine': 'vllm', 'data_parallel': 2, + 'prefix_caching': 'on', 'max_num_seqs': 64, + }) + assert v['runtime']['data_parallel_size'] == 2 + assert v['runtime']['enable_prefix_caching'] is True + assert v['runtime']['max_num_seqs'] == 64 + + o = InferStackTUI._endpoint_entry({ + 'model': 'qwen', 'engine': 'ollama', 'host': 'oll', + 'ollama_runtime': 'num_ctx=8192 keep_alive=5m', + }) + assert o['engine'] == 'ollama' and o['host'] == 'oll' + assert o['runtime']['num_ctx'] == 8192 + assert o['runtime']['keep_alive'] == '5m' + + +def test_tui_endpoint_wizard_is_engine_adaptive_and_labeled(): + from textual.widgets import Label, Select + + from infer_stack.tui import InferStackTUI, _AddEndpointScreen + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + screen = _AddEndpointScreen(['qc']) + app.push_screen(screen) + await pilot.pause() + # vLLM knobs visible by default; ollama hidden + assert screen.query_one('#vllm-opts').display is True + assert screen.query_one('#ollama-opts').display is False + assert screen.query_one('#e-dp') # data-parallel field exists + # fields are labeled (the "blank page" complaint) + labels = [str(lbl.render()) for lbl in screen.query(Label)] + assert any('tensor-parallel' in x for x in labels) + assert any('data-parallel' in x for x in labels) + # switching engine swaps the field groups + screen.query_one('#e-engine', Select).value = 'ollama' + await pilot.pause() + assert screen.query_one('#vllm-opts').display is False + assert screen.query_one('#ollama-opts').display is True + + _run(scenario) + + +def test_tui_add_endpoint_writes_advanced_params(tmp_path): + import yaml + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + catalog_path = tmp_path / 'catalog.yaml' + catalog_path.write_text(yaml.safe_dump(CATALOG)) + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None, + catalog_path=str(catalog_path)) + async with app.run_test() as pilot: + await pilot.pause() + app._on_add_endpoint({ + 'name': 'big', 'model': 'qc', 'engine': 'vllm', + 'tensor_parallel': 2, 'max_model_len': None, 'gpu_mem': None, + 'extra_args': '', 'reclaim': '', + }) + await pilot.pause() + + _run(scenario) + on_disk = yaml.safe_load(catalog_path.read_text()) + assert on_disk['endpoints']['big']['runtime']['tensor_parallel_size'] == 2 + + +def test_tui_edit_blocked_while_served(tmp_path): + import yaml + from textual.widgets import DataTable + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + catalog_path = tmp_path / 'catalog.yaml' + catalog_path.write_text(yaml.safe_dump(CATALOG)) + controller.acquire('me', catalog.resolve_names(['qwen-coder'])) # serve it + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None, + catalog_path=str(catalog_path)) + async with app.run_test() as pilot: + await pilot.pause() + app.query_one('#endpoints', DataTable).move_cursor(row=0) # qwen-coder + app.action_edit_endpoint() + await pilot.pause() + assert 'served' in str(app.query_one('#status').render()) + + _run(scenario) + + +def test_tui_remove_endpoint_writes_catalog(tmp_path): + import yaml + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + catalog_path = tmp_path / 'catalog.yaml' + catalog_path.write_text(yaml.safe_dump(CATALOG)) + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None, + catalog_path=str(catalog_path)) + async with app.run_test() as pilot: + await pilot.pause() + app._do_remove('endpoints', 'qwen-fast') # confirm bypassed + await pilot.pause() + assert 'qwen-fast' not in app.catalog.endpoints + + _run(scenario) + on_disk = yaml.safe_load(catalog_path.read_text()) + assert 'qwen-fast' not in on_disk['endpoints'] + + +def test_tui_empty_catalog_shows_suggest_hint(): + from textual.widgets import Static + + from infer_stack.leasing import ( + Catalog, + Controller, + Ledger, + NullBackend, + SqliteStore, + ) + from infer_stack.tui import InferStackTUI + + controller = Controller(Ledger(SqliteStore(':memory:')), NullBackend()) + catalog = Catalog.from_dict({'models': {}, 'endpoints': {}}) + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + help_text = str(app.query_one('#catalog-help', Static).render()) + assert 'suggest' in help_text.lower() + + _run(scenario) + + +def test_tui_ps_rows_parse_status_created_and_id(): + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + sample = json.dumps([ + {'Service': 'litellm', 'Status': 'Up 3 minutes', 'State': 'running', + 'CreatedAt': '2026-06-19 00:00:00 -0400', 'ID': 'abcdef1234567890', + 'Publishers': [{'PublishedPort': 14042, 'TargetPort': 4000}]}, + ]) + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + # stub the backend's compose seam + backend = controller.backend + import tempfile + f = tempfile.NamedTemporaryFile('w', suffix='.yml', delete=False) + f.write('services: {}\n') + f.close() + backend.compose_file = f.name + backend.project = 'infer-stack' + backend.run = lambda args: sample + rows = app._compose_ps_rows() + assert rows[0]['status'] == 'Up 3 minutes' + assert rows[0]['created'].startswith('2026-06-19') + assert rows[0]['id'] == 'abcdef123456' # truncated to 12 + assert '14042->4000' in rows[0]['ports'] + + _run(scenario) + + +def test_tui_system_tab_renders_without_nvidia_smi(): + from textual.widgets import DataTable + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + app._fill_gpus(None) # nvidia-smi unavailable -> hint row + gpus = app.query_one('#gpus', DataTable) + assert gpus.row_count == 1 + assert 'cpus' in app._system_line() + + _run(scenario) + + +def test_tui_collapsed_console_skips_expensive_polling(): + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + app._active_tab = 'tab-containers' + app._collapsed['docker'] = True + assert 'ps' not in app._collect() # collapsed -> no ps poll + app._collapsed['docker'] = False + assert 'ps' in app._collect() # visible -> polled + app._active_tab = 'tab-logs' + assert 'ps' not in app._collect() # other tab -> no ps poll + app._collapsed['system'] = True + assert 'gpus' not in app._collect() # system collapsed + app._collapsed['system'] = False + assert 'gpus' in app._collect() # system expanded -> polled + + _run(scenario) + + +def test_tui_open_builds_openwebui_url(): + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + controller.backend.ui_port = 13000 + assert app._ui_url('qwen-coder') == ( + 'http://localhost:13000/?models=qwen-coder' + ) + + _run(scenario) + + +def test_tui_api_tester_sends_via_injected_http(): + from textual.widgets import Select + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + class _Resp: + def raise_for_status(self): + pass + + def json(self): + return {'choices': [{'message': {'content': 'pong'}}]} + + class _HTTP: + def __init__(self): + self.calls = [] + + def post(self, url, json=None, headers=None, timeout=None): + self.calls.append((url, json)) + return _Resp() + + http = _HTTP() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None, http=http) + async with app.run_test() as pilot: + await pilot.pause() + controller.backend.litellm_port = 14042 + # only ready (running) models are offered; simulate one being ready + app._sync_api_models(['qwen-coder']) + assert app.query_one('#api-model', Select).value == 'qwen-coder' + app.action_api_send() + await app.workers.wait_for_complete() + await pilot.pause() + assert any('pong' in line for line in app._api_lines) + + _run(scenario) + assert http.calls and http.calls[0][0].endswith('/v1/chat/completions') + + +def test_tui_api_urls_render_without_markup_error(): + # Regression: the URLs were rendered with Textual [link=URL] markup, which + # rejects the ':' in http:// and crashed on get_content_height. + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + controller.backend.litellm_port = 14042 + controller.backend.ui_port = 13000 + app._update_api_urls() + await pilot.pause() + text = str(app.query_one('#api-urls').render()) # must not raise + assert '14042' in text and '13000' in text + + _run(scenario) + + +def test_tui_api_list_models_and_curl(): + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + class _Resp: + def raise_for_status(self): + pass + + def json(self): + return {'data': [{'id': 'qwen-coder'}, {'id': 'qwen-fast'}]} + + class _HTTP: + def get(self, url, headers=None, timeout=None): + self.url = url + return _Resp() + + http = _HTTP() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None, http=http) + async with app.run_test() as pilot: + await pilot.pause() + controller.backend.litellm_port = 14042 + app._sync_api_models(['qwen-coder']) + app._update_api_curl() + curl = str(app.query_one('#api-curl').render()) + assert 'curl' in curl and '/v1/chat/completions' in curl + assert 'qwen-coder' in curl + app.action_api_copy_curl() # must not raise + app.action_api_list_models() + await app.workers.wait_for_complete() + await pilot.pause() + assert any('qwen-coder' in line for line in app._api_lines) + + _run(scenario) + assert http.url.endswith('/v1/models') + + +def test_tui_api_lists_only_ready_models(): + from textual.widgets import Select + + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + # NullBackend observes nothing running -> no ready models listed, + # even though the catalog has endpoints. + assert app._ready_endpoints == [] + assert app.query_one('#api-model', Select).value is Select.NULL + + _run(scenario) + + +def test_tui_cleanup_prunes_released_and_stopped(tmp_path): + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + # an active lease, then released -> it becomes a RELEASED tail entry + out = controller.acquire('bob', catalog.resolve_names(['qwen-coder'])) + controller.release(out.lease.id) + leases, _ = controller.ledger.status() + assert any(str(le.state) == 'released' for le in leases) + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + app.action_cleanup() + await app.workers.wait_for_complete() + await pilot.pause() + + _run(scenario) + leases, _ = controller.ledger.status() + assert not any(str(le.state) == 'released' for le in leases) + + +def test_tui_model_cached_label(tmp_path): + from infer_stack.tui import InferStackTUI + + hub = tmp_path / 'hub' + (hub / 'models--Org--Model').mkdir(parents=True) + assert InferStackTUI._cached_label('hf://Org/Model', hub) == 'yes' + assert InferStackTUI._cached_label('hf://Org/Other', hub) == 'no' + assert InferStackTUI._cached_label('hf://Org/Model', None) == '?' + assert InferStackTUI._cached_label('', hub) == '-' + + +def test_tui_compose_control_runs_up(tmp_path): + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + compose_file = tmp_path / 'docker-compose.yml' + compose_file.write_text('services: {}\n') + calls = [] + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: None) + async with app.run_test() as pilot: + await pilot.pause() + backend = controller.backend + backend.compose_file = compose_file + backend.project = 'infer-stack' + backend.run = lambda args: calls.append(args) or '' + app.action_compose_up() + await app.workers.wait_for_complete() + await pilot.pause() + + _run(scenario) + assert calls and calls[0][:2] == ['docker', 'compose'] and 'up' in calls[0] + + +def test_tui_logs_stream_from_injected_source(): + from infer_stack.tui import InferStackTUI + + controller, catalog = _ctx() + lines = ['litellm | started', 'litellm | ready'] + + async def scenario(): + app = InferStackTUI(controller, catalog, interval=999, + proc_factory=lambda svc: _FakeProc(lines)) + async with app.run_test() as pilot: + await pilot.pause() + await app.workers.wait_for_complete() # drain the log stream + await pilot.pause() + assert any('ready' in line for line in app._log_lines) + + _run(scenario) diff --git a/uv.lock b/uv.lock index 7feac46a..70886bc1 100644 --- a/uv.lock +++ b/uv.lock @@ -9,8 +9,7 @@ resolution-markers = [ ] [options] -exclude-newer = "0001-01-01T00:00:00Z" # This has no effect and is included for backwards compatibility when using relative exclude-newer values. -exclude-newer-span = "P14D" +exclude-newer = "2026-06-04T04:00:00Z" [[package]] name = "certifi" @@ -158,10 +157,10 @@ wheels = [ [[package]] name = "infer-stack" -version = "0.5.0" source = { editable = "." } dependencies = [ { name = "jinja2" }, + { name = "loguru" }, { name = "pyyaml" }, { name = "requests" }, { name = "rich" }, @@ -174,19 +173,24 @@ tests = [ { name = "pytest" }, { name = "pytest-codeblocks" }, ] +tui = [ + { name = "textual" }, +] [package.metadata] requires-dist = [ { name = "jinja2", specifier = ">=3.1" }, + { name = "loguru", specifier = ">=0.7" }, { name = "pytest", marker = "extra == 'tests'", specifier = ">=7.0" }, { name = "pytest-codeblocks", marker = "extra == 'tests'", specifier = ">=0.17" }, { name = "pyyaml", specifier = ">=6.0" }, { name = "requests", specifier = ">=2.31" }, { name = "rich", specifier = ">=13.0" }, { name = "scriptconfig", specifier = ">=0.9" }, + { name = "textual", marker = "extra == 'tui'", specifier = ">=0.50" }, { name = "ubelt", specifier = ">=1.3" }, ] -provides-extras = ["tests"] +provides-extras = ["tests", "tui"] [[package]] name = "iniconfig" @@ -209,6 +213,31 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, ] +[[package]] +name = "linkify-it-py" +version = "2.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "uc-micro-py" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2e/c9/06ea13676ef354f0af6169587ae292d3e2406e212876a413bf9eece4eb23/linkify_it_py-2.1.0.tar.gz", hash = "sha256:43360231720999c10e9328dc3691160e27a718e280673d444c38d7d3aaa3b98b", size = 29158, upload-time = "2026-03-01T07:48:47.683Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b4/de/88b3be5c31b22333b3ca2f6ff1de4e863d8fe45aaea7485f591970ec1d3e/linkify_it_py-2.1.0-py3-none-any.whl", hash = "sha256:0d252c1594ecba2ecedc444053db5d3a9b7ec1b0dd929c8f1d74dce89f86c05e", size = 19878, upload-time = "2026-03-01T07:48:46.098Z" }, +] + +[[package]] +name = "loguru" +version = "0.7.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "win32-setctime", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/3a/05/a1dae3dffd1116099471c643b8924f5aa6524411dc6c63fdae648c4f1aca/loguru-0.7.3.tar.gz", hash = "sha256:19480589e77d47b8d85b2c827ad95d49bf31b0dcde16593892eb51dd18706eb6", size = 63559, upload-time = "2024-12-06T11:20:56.608Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0c/29/0348de65b8cc732daa3e33e67806420b2ae89bdce2b04af740289c5c6c8c/loguru-0.7.3-py3-none-any.whl", hash = "sha256:31a33c10c8e1e10422bfd431aeb5d351c7cf7fa671e3c4df004162264b28220c", size = 61595, upload-time = "2024-12-06T11:20:54.538Z" }, +] + [[package]] name = "markdown-it-py" version = "4.2.0" @@ -221,6 +250,11 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" }, ] +[package.optional-dependencies] +linkify = [ + { name = "linkify-it-py" }, +] + [[package]] name = "markupsafe" version = "3.0.3" @@ -306,6 +340,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/70/bc/6f1c2f612465f5fa89b95bead1f44dcb607670fd42891d8fdcd5d039f4f4/markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa", size = 14146, upload-time = "2025-09-27T18:37:28.327Z" }, ] +[[package]] +name = "mdit-py-plugins" +version = "0.6.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/59/fc/f8d0863f8862f25602c0404d75568e89fb6b4109804645e5cdfb1be5cf56/mdit_py_plugins-0.6.1.tar.gz", hash = "sha256:a2bca0f039f39dbd35fb74ae1b5f998608c437463371f0ff7f49a19a17a114d0", size = 56114, upload-time = "2026-05-13T09:03:38.91Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a5/69/6da5581c6a7fede7dc261bf4e67d6adca4196f176b43288b55b3db395b6e/mdit_py_plugins-0.6.1-py3-none-any.whl", hash = "sha256:214c82fb2ac524472ab6a5bcab1de80f73b50443e187f401bfd77efbc7c6481d", size = 66663, upload-time = "2026-05-13T09:03:37.76Z" }, +] + [[package]] name = "mdurl" version = "0.1.2" @@ -324,6 +370,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" }, ] +[[package]] +name = "platformdirs" +version = "4.10.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/47/e4501f49c178ae1d9f4a75073fda4204f52647993f075a9db4d14930e0c5/platformdirs-4.10.0.tar.gz", hash = "sha256:31e761a6a0ca04faf7353ea759bdba55652be214725111e5aac52dfa29d4bef7", size = 31224, upload-time = "2026-05-28T03:32:53.587Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/81/e6/cd9575ac904136b3cbf7aa7ee819ef86eedb7274e46f230e94ea4342e729/platformdirs-4.10.0-py3-none-any.whl", hash = "sha256:fb516cdb12eb0d857d0cd85a7c57cea4d060bee4578d6cf5a14dfdf8cbf8784a", size = 22743, upload-time = "2026-05-28T03:32:52.175Z" }, +] + [[package]] name = "pluggy" version = "1.6.0" @@ -477,6 +532,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/57/00/2bf691dfc2b0170013d466fcdbbfc9d036e26d6049d74efaf7775401321d/scriptconfig-0.9.1-py3-none-any.whl", hash = "sha256:4b69e2f4eb681ec4480e9bb5d6c82d2f8c82a9badc703a261ee8940736101d86", size = 87103, upload-time = "2026-03-02T17:33:31.394Z" }, ] +[[package]] +name = "textual" +version = "8.2.7" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py", extra = ["linkify"] }, + { name = "mdit-py-plugins" }, + { name = "platformdirs" }, + { name = "pygments" }, + { name = "rich" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9b/7a/c519db0aba5024f86e71e9631810bfdd6866ed2c8695bd7fa34b90e7ef59/textual-8.2.7.tar.gz", hash = "sha256:658f568ff81e30ed43890c3e07520390e5cf1b4763822006e060656b0a88f105", size = 1859249, upload-time = "2026-05-19T10:52:49.531Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a8/f5/c1e18bc0707300a0e90204343abbf7d7acd6fb7ebe03a6d4893b99a234b8/textual-8.2.7-py3-none-any.whl", hash = "sha256:4caaa13a90bc4cf9c6c862c067ccd34fe84e9c161710a2a907a8026313b6bd73", size = 731129, upload-time = "2026-05-19T10:52:51.773Z" }, +] + [[package]] name = "tomli" version = "2.4.1" @@ -549,6 +621,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/01/b9/b6ee2a03975b398b428b1ed8ce2f1a598e2b9cc01ad2c6d49cfce3bb5a9b/ubelt-1.4.2-py3-none-any.whl", hash = "sha256:6cdbcbba75cd7c3e1ef9556a13b7c0e29a55f67cb454f022a981e42808f2b06e", size = 231820, upload-time = "2026-04-13T02:38:13.094Z" }, ] +[[package]] +name = "uc-micro-py" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/78/67/9a363818028526e2d4579334460df777115bdec1bb77c08f9db88f6389f2/uc_micro_py-2.0.0.tar.gz", hash = "sha256:c53691e495c8db60e16ffc4861a35469b0ba0821fe409a8a7a0a71864d33a811", size = 6611, upload-time = "2026-03-01T06:31:27.526Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/61/73/d21edf5b204d1467e06500080a50f79d49ef2b997c79123a536d4a17d97c/uc_micro_py-2.0.0-py3-none-any.whl", hash = "sha256:3603a3859af53e5a39bc7677713c78ea6589ff188d70f4fee165db88e22b242c", size = 6383, upload-time = "2026-03-01T06:31:26.257Z" }, +] + [[package]] name = "urllib3" version = "2.7.0" @@ -557,3 +638,12 @@ sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e wheels = [ { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, ] + +[[package]] +name = "win32-setctime" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b3/8f/705086c9d734d3b663af0e9bb3d4de6578d08f46b1b101c2442fd9aecaa2/win32_setctime-1.2.0.tar.gz", hash = "sha256:ae1fdf948f5640aae05c511ade119313fb6a30d7eabe25fef9764dca5873c4c0", size = 4867, upload-time = "2024-12-07T15:28:28.314Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e1/07/c6fe3ad3e685340704d314d765b7912993bcb8dc198f0e7a89382d37974b/win32_setctime-1.2.0-py3-none-any.whl", hash = "sha256:95d644c4e708aba81dc3704a116d8cbc974d70b3bdb8be1d150e36be6e9d1390", size = 4083, upload-time = "2024-12-07T15:28:26.465Z" }, +]