Repository navigation
Conversation
…n ends
A client that hung up on a request kept its slot in the --socket limit
forever: the write-error path in HttpSession::send_chunk_data closed the
socket but had its active_connections_ decrement commented out, and other
abandoned-request paths never released either. After 10 such clients the
server rejected every connection ("Connection limit reached (10)") and
had to be restarted, while idle at 0% CPU.
Add HttpSession::release_connection(), idempotent via an atomic flag, use
it at every close path (including the stream write error), and call it
from the destructor so a session can never outlive its slot.
Add test_connections.py to the openai_api_test suite: drop more clients
than the limit allows (mid-stream, before the first chunk, non-streaming),
finish and malform connections, then check that limit-1 idle sockets leave
room for a request and limit idle sockets do not (catches leaks and double
releases). 0/6 pass on the unfixed server, 6/6 with the fix; the existing
openai_api_test suite still passes (87/87).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After clients hang up on requests,
flm servecan stop answering everything (evenGET /v1/models), permanently, until restarted. The process is alive and idle (~0% CPU) and the log only repeatsConnection limit reached (10), rejecting new connection.Root cause
Each connection holds a slot in
active_connections_(limit--socket, default 10). Several paths never gave it back when the client went away. The clearest one is the streaming write-error path inHttpSession::send_chunk_data: it cancels the request and closes the socket, but itsactive_connections_.fetch_sub(1)was commented out. After 10 leaked slots the accept loop rejects every new connection.Reproduced deterministically without images, prefix cache, or load: with the default limit, drop 10 requests (streaming hung up before the first chunk, non-streaming hung up while generating), then even
GET /v1/modelsis reset. The log shows the accounting: 21 connections established, 11 closed before the first rejection.Fix
HttpSession::release_connection()releases the slot exactly once (atomic flag). It is used at every close path, including the stream write error, and called from the destructor so a session can never outlive its slot.Tests
New
src/test/openai_api_test/test_connections.py(added torun_all.pyand the README table). It drops more clients than the limit allows, finishes requests, and sends idle and malformed connections. It then checks thatlimit-1idle sockets leave room for one request andlimitidle sockets do not, which catches both leaked slots and double releases. A barrier request (the server runs one at a time) makes it deterministic.openai_api_testsuite on the fixed build (qwen3.5:2b): 87 passed, 0 failed./v1/chat/completionsand/api/chat, at three points in the request (just after sending, after first bytes, mid-response), completed requests, idle/garbage connections, and 4 rounds of 14 concurrent clients mixing completions and drops. Each was followed by the exact-limit probe. No leaks, no underflow.Server-code-only change; no binaries or xclbins touched. Tested on Linux only (Windows shares the same code path).
🤖 Generated with Claude Code