Skip to content

Add async request queue to llm-server - #250

Merged
stikves merged 2 commits into
apple:mainfrom
stikves:sukru/serve-v2-public
Sep 14, 2026
Merged

stikves merged 2 commits into
apple:mainfrom
stikves:sukru/serve-v2-public

Conversation

@stikves

@stikves stikves commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Summary

The llm-server currently gates concurrency with a binary busy flag: while one
request is generating, any concurrent request is rejected immediately with
429. This PR replaces that gate with an async RequestQueue, so concurrent
requests wait in a bounded queue and are rejected only when the queue is full.

Changes

  • RequestQueue (CoreAILMCommon): an async semaphore built on
    withCheckedThrowingContinuation with an acquired / rejected / queued state
    machine. Replaces tryAcquire()/release() and the generating flag.
    acquire() returns a QueuePermit that frees the slot exactly once — on
    explicit release() or, as a backstop, on deallocation, so a dropped
    streaming response can never leak the slot and wedge the server.
  • Queued waiters are cancellation-aware: if the awaiting task is cancelled
    while queued (e.g. on shutdown), the waiter is removed and its acquire()
    throws CancellationError instead of leaking its continuation or being handed
    the slot after the fact.
  • --max-queue-depth (default 16): how many requests may wait before the
    server returns 429 with a queue_full error. Rejected as invalid if
    negative; 0 means no queuing (second concurrent request gets an immediate
    429).
  • hasRecurrentState on InferenceEngine (default false; overridden by
    the sequential and pipelined engines): recurrent (SSM / hybrid) models can't
    reuse a token prefix, so the server skips the prefix-reuse fast path and counts
    a miss. Engine correctness on rewind is already enforced by the engine itself;
    this only keeps the server's reuse stats and logs accurate.
  • ServerError.queueFull(depth:) for the rejection path.

Behavior

Scenario Before After
Single request Served Served (unchanged)
Concurrent requests Immediate 429 Queued, served FIFO
Queue at capacity Immediate 429 429 (queue_full)

/ready now reports busy based on queue activity rather than the removed
generating flag.

Testing

  • RequestQueue unit coverage (in CoreAILMCommonTests): immediate acquire,
    concurrent waiting, FIFO wakeup, rejection at maxDepth, maxDepth == 0,
    negative-depth clamp, idempotent permit release, and cancellation of a queued
    waiter. CoreAILMCommonTests (52) pass; llm-server builds.
  • Load-tested end to end against muse_glimmer_30b_4bit_dynamic:
    • Single request: served.
    • 8 concurrent (< depth): all served FIFO, no premature 429.
    • 30 concurrent (> capacity): 17 served, 13 rejected with queue_full. This
      matches 1 in-flight + 16 queued at the default depth.
    • 20-request soak: no errors; server returned to idle with no stuck slots.

@stikves stikves self-assigned this Sep 14, 2026
@stikves
stikves marked this pull request as ready for review September 14, 2026 14:41
@stikves
stikves force-pushed the sukru/serve-v2-public branch 4 times, most recently from 465b7bc to 0a24579 Compare September 14, 2026 16:11
Replace the binary busy flag with a bounded async RequestQueue: concurrent
requests wait FIFO and get a 429 only when the queue is full.

- acquire() returns a QueuePermit that frees the slot exactly once, on scope
  exit or deinit, so a dropped streaming response can't leak it and wedge the
  server
- Cancel queued waiters cleanly: remove the waiter and throw CancellationError
- --max-queue-depth (default 16); reject negative values
- hasRecurrentState skips prefix-reuse accounting for recurrent (SSM/hybrid)
  models
- RequestQueue, QueuePermit and ServerError live in CoreAILMCommon, unit-tested
  via CoreAILMCommonTests
Comment thread swift/Sources/CoreAILMCommon/RequestQueue.swift Outdated
Replace the Mutex<Bool> once-guard with a lock-free
Atomic<Bool>.compareExchange; the release path takes the RequestQueue
Mutex, so relaxed ordering is sufficient. Add a concurrent-release unit
test that races 50 release() calls against a single queued waiter.
@stikves
stikves merged commit 257008b into apple:main Sep 14, 2026
3 checks passed
@stikves
stikves deleted the sukru/serve-v2-public branch September 14, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants