feat(api): runtime model load/unload endpoints - #22
Open
dusterbloom wants to merge 4 commits into
Open
Conversation
Concurrent requests to co-resident models each ran mlx::eval on their own
spawn_blocking thread under a fresh with_new_default_stream(Stream::new()),
racing on MLX's shared Metal CommandEncoder (the output-array table in
set_output_array) -> EXC_BAD_ACCESS/SIGSEGV. The per-model Mutex<AnyModel>
only serializes a single model, not the co-resident set (e.g. an SLM trio).
Add a process-wide GPU gate acquired by Engine::{generate_with_thinking,
generate_streaming_with_thinking, embed}. A single-GPU host has no eval
parallelism to lose and the trio is sequential, so the cost is ~nil.
Poison-recovering so a mid-eval panic can't wedge inference.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit fbdcd2f)
Add POST /v1/models and DELETE /v1/models/{name} so operators can load and
unload MLX models while the server runs, without a restart. Opt-in via
local.allow_runtime_model_load (default off; gate behind server.api_key).
Changes are in-memory only -- the TOML config stays the source of truth.
Router.local_engines becomes an RwLock<HashMap>. resolve()/list take a read
lock and clone the Arc<Engine> out, so an in-flight request is decoupled from
map membership and a concurrent unload can never free a model mid-request.
Unload removes the map entry, drains to sole ownership, then drops (detaches
past a 30s timeout -> 202). Load resolves the path non-interactively and runs
the blocking weight load in spawn_blocking; a shared state::build_engine is
reused by both startup loading and the endpoint. Unloading the auto-router
model is refused (it holds a separate Arc).
Adds ServerError::{Conflict, Forbidden}, a doctor capability warning, the
init-template + README docs, and unit/integration coverage (guards, the
load/list/route/unload round-trip, and the drain-to-sole-ownership logic).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gate runtime mutations behind authentication, constrain model paths, and retain load and resident permits through blocking work and unload drains.
Require authenticated runtime model control, constrain local roots, and preserve runtime quotas through unload. Propagate auto-route failures and document/test the guarded behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds two opt-in admin endpoints so models can be changed while the server runs, no restart:
POST/v1/models[[models]]entry;pathrequired).200+ model object,409collision,400if not cached locally,403when disabled.DELETE/v1/models/{name}204once freed,202if a request is still draining,404unknown,409for the auto-router model.GET/v1/modelsOpt-in via
local.allow_runtime_model_load(default off; gate behindserver.api_key). In-memory only — TOML stays the source of truth.How
Router.local_engines→RwLock<HashMap>.resolve()/list take a read lock and clone theArc<Engine>out, so an in-flight request is decoupled from map membership; a concurrent unload can't free a model mid-request.Arc::try_unwrap), then drops; past a 30s timeout it detaches the final free →202. Drop is ungated — teardown frees buffers but neverevals, so no race with the cross-model output-array table.spawn_blocking; sharedstate::build_engineis reused by startup + the endpoint.ServerError::{Conflict, Forbidden}, adoctorwarning, init-template + README docs.Base note
Includes a cherry-pick of
fbdcd2f7(serialize GPU eval across models to stop SIGSEGV) as a prerequisite — runtime loading is all about co-resident models, the exact case that fix makes safe.Testing
fmt --check,clippy(nursery) clean,cargo test -p higgs -- --test-threads=1: 583 passed / 0 failed (13 new unit + 2 integration).409→ unauth401→ unload204(memory freed) → unknown404. No SIGSEGV under co-resident inference.🤖 Generated with Claude Code