serve: ACCEPT frame — oversized prompt is a clean 400, not a broken 200 stream (#597 item 6) - #609
Merged
Merged
Conversation
…en 200 stream (#597 item 6) A streaming chat request committed its HTTP 200 (SSE headers) before the engine tokenised the prompt and checked it against CTX. An over-long prompt was only then found -- ERROR CONTEXT_EXCEEDED -- far too late to send the 400 the client needs, so the client saw a broken/empty stream instead of a clean context_length_exceeded. (Non-streaming was already correct: it sends its 200 only after generation.) Add an acceptance frame to the engine<->gateway protocol (mkelcb's design in #597): - c/colibri.c mux_submit(): after all cheap validation (encode, non-empty, fits CTX) and BEFORE prefill, emit `ACCEPT <id> <ntok>`. Every early failure already returns first with an ERROR, so a request yields exactly one of ACCEPT or an early ERROR. - Engine._dispatch_stdout(): parse ACCEPT -> ("accept", {...}) without popping pending. - Engine.generate(): new on_accept callback; commit exactly once on the first of ACCEPT / DATA / DONE. DATA/DONE before an ACCEPT still commit (an older engine with no ACCEPT frame keeps working); a second ACCEPT is a protocol error. - APIHandler.generation() streaming: defer the SSE 200 + keepalive into start_stream(), fired via on_accept. An ERROR that arrives before ACCEPT now propagates as an APIError with nothing committed, so do_POST returns a normal JSON 400. Tests: 4 Engine-level cases over a fake engine process (ACCEPT-before-DATA fires on_accept first; ERROR-before-ACCEPT raises 400 and never commits; DATA-before-ACCEPT still commits for an old engine; duplicate ACCEPT is a protocol error) + 1 end-to-end HTTP case (streaming context-exceeded returns a clean 400 with context_length_exceeded, not a 200 stream). Full server suite 83/83; the C engine builds clean. Completes the #597 series (items 1/2/4/5 already on dev via #535/#606/#607). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #597 item 6 — @mkelcb's diagnosis and protocol design.
The bug
A streaming chat request committed its HTTP
200(SSE headers) before the engine tokenised the prompt and checked it againstCTX. An oversized prompt was only discovered afterwards (ERROR CONTEXT_EXCEEDED) — too late to send the400the client needs, so the client got a broken/empty stream instead of a cleancontext_length_exceeded. (Non-streaming was already correct: it sends200only after generation completes.)The fix — an acceptance frame in the engine↔gateway protocol
c/colibri.cmux_submit()— after all cheap validation (encode, non-empty, fitsCTX) and before prefill, emitACCEPT <id> <ntok>. Every early failure alreadyreturns first with anERROR, so a request yields exactly one ofACCEPTor an earlyERROR.Engine._dispatch_stdout()— parseACCEPT→("accept", …)without popping the pending request (DATA/DONE still follow).Engine.generate()— newon_acceptcallback; commit exactly once on the first of ACCEPT / DATA / DONE. DATA/DONE before an ACCEPT still commit, so an older engine with no ACCEPT frame keeps working; a second ACCEPT is a protocol error.APIHandler.generation()streaming — defer the SSE200+ keepalive intostart_stream(), fired byon_accept. AnERRORbeforeACCEPTnow propagates as anAPIErrorwith nothing committed →do_POSTreturns a normal JSON400.Tests
4 Engine-level (fake engine process): ACCEPT-before-DATA fires
on_acceptfirst; ERROR-before-ACCEPT raises400and never commits; DATA-before-ACCEPT still commits (old engine); duplicate ACCEPT is a protocol error.1 end-to-end HTTP: a streaming context-exceeded request returns a clean
400context_length_exceeded, not a200stream.Full server suite 83/83; the C engine builds clean.
#597 is now complete
<|user|>turn terminationreasoning_contentCONTEXT_EXCEEDEDafter 200