Conversation
A GPU-less check on k3s: KubeAI's cpu profile runs real vLLM on CPU, and kubeai_k3s.sh drives doctor, `run` (acquire, one chat completion, release) and checks the released Model is pruned. First real run of the backend since it landed: it passed unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
On KubeAI a client had to name a model by its Model name, a DNS slug of the served name; cards send the endpoint alias and got HTTP 404 (verified on k3s). The kubeai backend now owns a gateway-only ComposeBackend (own state dir, project infer-stack-gateway) and feeds it a new generic `upstream` route row per alias: the KubeAI URL plus the Model name. Clients get the same base_url, managed key and alias as on compose; readiness is judged through the gateway by alias; `secrets rotate` works unchanged. The gateway reaches KubeAI at the Service's cluster IP by default (kubeai_gateway_upstream overrides). --no-litellm keeps direct access. dev/kubeai_e2e.sh (the existing script; a duplicate I had started is folded in) sends the alias as a card does, gains GATEWAY=0 and a CPU profile, and now fails on a failed generation: the old check sat in a `&&` list where `set -e` does not apply, so a 404 printed PASS. Verified on k3s + KubeAI 0.23.4 (real vLLM on CPU): GATEWAY=1 passes, GATEWAY=0 fails with the 404, rotation accepts the new key and rejects the old, and no Model is left behind either way. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
render_kubeai_artifacts had no callers once the profile path was excised; it was a second KubeAI renderer beside the backend. The doc claimed it and a kubeai_ops.py (already gone) were kept for reference. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
K2: KubeaiBackend.residency() builds the same Residency as compose from `kubectl get pods` (KubeAI copies the Model's infer-stack labels onto its pods), mapping pod states onto the Docker vocabulary and keeping the Kubernetes reason; a kubectl failure raises ResidencyUnknown. `gc --orphans` and `network migrate` stop using "has residency" to mean compose. K4: the crash diagnosis moves to leasing/diagnosis.py (diagnose_startup(instances, read_logs)); compose re-exports the old names. The kubeai backend feeds it pods and pod logs (the previous run last, where the crash is), probes fail fast with the engine's words, and a not-ready wait names the pod reason (Unschedulable, ImagePullBackOff). `error: unrecognized arguments` is now a fatal signature on both backends. Verified on k3s + KubeAI: a Model whose vLLM rejected a flag failed its acquire in 52 s (timeout 800 s), quoting the error and its likely cause; live residency read running, starting and crashed pods correctly. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
leasing.models.served_name replaces five copies of the fallback chain. Two LiteLLM route builders had used `or deployment.id` where the engine used the first served alias, so without served_model_name the route named a model the engine did not serve. KubeAI's resource-profile units come from placement.required_gpu_count, and runtime.env reaches the Model's spec.env (templates filled, reserved names enforced at catalog load); command and mounts stay compose-only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Brings the CI fixes (textual-free doctest, ty errors) onto the branch. The branch's own KubeAI code passes ty as merged; flake8 0; tests pass on 3.14 and 3.10 without textual (831) and with it (898). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
compose.py mixed engine containers with the front door. Two new modules: - leasing/naming.py: dns_slug, the vLLM/Ollama service-name rules and the engine container ports. Engines, gateway routes and the kubeai backend all derive names and upstream addresses from these. - leasing/gateway.py: route rows and the LiteLLM model_list renderers, the route registry functions, the LiteLLM/Postgres/Open WebUI/nginx service renderers, the dynamic-routing constants, and set_master_key. Moved unchanged. ENGINE_LABEL joins the other labels in residency.py. compose keeps importing what it uses (so `from infer_stack.leasing.compose import dns_slug` still works); importers of gateway names now import them from gateway. No behaviour change: suite 898 passed, ty and flake8 clean. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
The gateway's methods move off ComposeBackend into leasing.gateway.Gateway: keys (master_key, rotate, restore_env, gateway_accepts, db_password), the route registry (load, merge, save), dynamic-route reconciliation (_reconcile_routes, _list_managed_routes, _post_route), and access(). ComposeBackend owns one (`backend.gateway`), keeps thin public delegations (master_key, access, litellm_port, ...), and stores the settings the gateway reads (litellm, ui, dynamic_routing, reverse proxy, ports, http, clock, sleep) on it through properties, so there is one copy of each. The registry merge no longer reads backend state: each backend supplies its rows (compose from its catalog and deployments; kubeai persists its upstream rows with merge_route_registry, the `routes seed` path), and the gateway merges, persists and renders them. The upstream_routes side-channel is gone. compose.py 2818 -> 2483 lines. Suite 898 passed; ty and flake8 clean. Live on the guest: a gateway-only compose apply answered 200 with its key, and `secrets rotate` accepted the new key and rejected the old. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Brings the weight-floor doctest fix (sparse files, cleans up after itself). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
render_compose renders the engine services, then merges in what render_front_door returns: the LiteLLM service and config (static, registry or dynamic routes), Postgres for dynamic routing, Open WebUI and the reverse proxy. The engines pass only what the front door needs from them (their service names for the legacy depends_on, and in-network URLs for a UI without a gateway). Verified byte-identical: six configurations (bare, gateway, gateway+UI+ proxy, route registry with an upstream row, dynamic routing, UI only), 27 services, rendered with the code before and after. Suite 898 passed x3, ty and flake8 clean. compose.py 2483 -> 2360 lines. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Rule (user decision): a keep-warm model without a lease is always a candidate for eviction when a model with a lease needs a resource. Compose admission already enforces it when placing. Where the runtime schedules (KubeAI), a probe now reports Readiness.needs_room for an Unschedulable pod, and Controller.wait_ready evicts the longest-idle deployment through the normal locked evict, one per ROOM_COOLDOWN_S (30 s), never a leased one. The step runs after the deadline check: a test caught it evicting a second model after the wait had already given up. Verified on k3s: with a profile only one Model fits (cpu-half, added to the test values), acquiring a leased Model evicted the idle keep-warm one within 10 s and was ready in 107 s; `E2E_MAKE_ROOM=1 dev/kubeai_e2e.sh` passes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
SqliteStore shares one sqlite3 connection across threads
(check_same_thread=False) but only write transactions took its lock, on the
belief that sqlite's serialized mode makes concurrent reads safe. It does
not for a shared Python connection: two threads stepping the same cached
statement interleave. The TUI reads the ledger from its refresh worker and
the UI thread at once, and intermittently got a lease whose deployment ids
included None (TypeError drawing the leases table; seen as a flaky TUI
test).
Every statement now goes through _SerializedConnection, which holds the
store's re-entrant lock until the statement has run and its rows are
fetched, so no cursor outlives the lock and transaction() nests. New test:
eight threads reading one ledger 150 times each; without the fix it fails
with InterfaceError('bad parameter or other API misuse') and more.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Measured with dev/profile_tui.py (headless and on a real terminal) and a real `infer-stack tui` session in tmux: - The lease and deployment panes were repainted every refresh tick: Textual repaints a widget whenever its border title is set, even to the same text. Titles are now set only on change. Idle terminal output fell from 6922 to 75 bytes/s (the header clock), and from 5223 to 442 with the docker pane open, which is what an SSH session feels. - Streamed log lines went to the UI one call_from_thread per line, capping a loading engine at ~1800 lines/s. Workers now append to a deque the UI drains every 100 ms, at most 200 lines per tick (2000 in one tick stalled the loop 218 ms), keeping the newest lines when a flood outruns the pane. The line mirror is trimmed instead of growing without bound. - The compose file's service list is parsed only when the file changes, not every tick. - A watchdog reports any UI-thread stall over 0.5 s in the TUI log, with the stack sampled while it was stuck written to the error log file, so a freeze on the host comes with what caused it. The scroll-offset test waits for the mount-time refresh; its intermittent failure was the ledger race fixed in the previous commit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Brings the shared-ledger-connection fix (serialize every statement). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
The user's report: launch felt laggy, and a slow action gave no sign it was working. Measured on a real terminal with the new `infer-stack tui --exit_after_paint` (quits once the first frame is drawn, then prints when that was after process start): - Activity line: while an action runs in a background worker (acquire, release, evict, apply, save endpoint, inspecting GPUs, API calls), a line above the status bar shows a spinner, what, for how long, and the newest backend progress (image-pull layers). Driven by worker state changes, so no action handler had to change; status hints cannot overwrite it. - `requests` is imported on first use instead of when the backend is built: opening the controller went from 127 ms to 33 ms. - The API, UI and Settings tabs are built right after the first frame (textual Lazy) instead of before it. - On exit, in-flight read-only docker queries (ps, inspect, logs, ...) are killed so quitting does not wait for the background refresh's `docker compose ps`. Mutating commands are never killed. Real terminal, guest: launch+quit 1.15 s -> ~1.0 s, first frame ~0.74 s -> ~0.69 s. The rest is imports (textual, the CLI) and Textual's own mount. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
kwconf is scriptconfig's successor and what the sibling packages
(aiq-magnet, cmd_queue, kwdagger) already use. Mechanical where the APIs
match: `import kwconf as kw`, scfg.Value -> kw.Value, scfg.ModalCLI ->
kw.ModalCLI, scfg.DataConfig -> kw.Config (233 uses in 9 files).
Differences that needed a change:
- kwconf's automatic conversion reads `null` as None, so `--backend null`
failed its choices; the 8 options with string choices get type=str.
- ModalCLI names the program after the class ("ManageCLI"); `__prog__`
keeps `infer-stack` in usage and error messages.
- The TUI's activity_label doctest needs textual, which CI does not
install; it is marked `+REQUIRES(module:textual)`.
Dependency: kwconf>=0.11.0. The repo's uv exclude-newer cutoff
(2026-06-04) predates every kwconf release, so kwconf alone is admitted up
to 2026-08-08 (0.11.0, the version tested) via exclude-newer-package;
uv.lock and the exported test lock only gain kwconf and lose scriptconfig.
Checked: full suite 909 passed; CI's checks (flake8, ty, the package plus
tests with only the tests extra) pass on 3.14 and 3.10 (838 each); CLI
smoke: help, --version, subcommands, --no-wait, unknown flags rejected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
The 2026-06-04 cutoff predated every kwconf release and needed a per-package exception. Removed at the user's request, with the exception. Relocked without --upgrade: uv.lock loses only the cutoff header, the locked versions and requirements/locks/tests.txt are unchanged. A fresh unpinned install now gets kwconf 0.12.0, and the CI-equivalent suite passes on it (838 passed, 3.14, tests extra only). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Both Clean up buttons ran the same ledger prune. It is now a single x binding in the footer, backed by a new `infer-stack gc --forget`. The TUI log also names the command for refresh, settings save and the API tab. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
Each pair is laid out by one helper that builds either tabs or the old draggable split from the same pane functions, so either layout is one class switch. Tab labels carry the lease and deployment counts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
docs/backend-parity.md states the intended relationship (KubeAI is Compose plus a scheduler: more information supplied, more tools installed, the same catalog, verbs, env file and TUI) and records row by row where that holds, where it does not, and which differences are deliberate. docs/planning/backend-parity-roadmap.md plans the rest in phases with exit criteria: one acquire path (K3), day-2 verbs and the TUI through the backend seam, gateway feature parity, placement information, and the gateway inside the cluster for a multi-workstation run. The README's KubeAI section described the pre-leasing setup/deploy workflow and a live kubectl patch that is no longer needed; it now points at the current docs and keeps the cluster prerequisites and debugging checks. The readiness-diagnosis subsection used commands that no longer exist. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
KubeAI gains an in-memory preview (and plan_on_idle_host) over one shared render, and converge accepts placement=, so Controller._admission_mode() is true for both real backends. An unrenderable acquire now writes no lease and runs no kubectl; a queued one fails at once. Duplicate authorities removed on the way: - "does this backend allocate GPUs" is allocates_gpus(), read by the controller's accounting and the CLI's status view; on KubeAI every deployment commits an empty allocation, so nothing is unresolved. - the approval digest and pre-approval move from ComposeBackend into ConvergeScaffold. Compose's stable-address network and container adoption are handed only to a backend that declares them. The roadmap splits P1 after review: P1b (delete the legacy branch, which only test fakes use) goes with P6; deferred duplicates are tabled there. Verified: full suite, and dev/kubeai_e2e.sh on k3s with a new admission step and the make-room step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
docs/queue.md orders what can be done and verified without a GPU or a second machine, with a done-when per item, a rule for adding unforeseen items, and a UX audit that must pass before the queue is finished. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9TxTZLpgCyS6bJe2URhQe
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.