Repository navigation
fix(gateway): Read the model from any body the gateway accepts - #1247
Merged
kotevcode merged 2 commits intoOct 17, 2026
Merged
Conversation
The gateway only read a request body's model from its first 4 MiB, so a
larger body (a chat with a few MiB image, which Open WebUI inlines as
base64) arrived at the buyer with the literal `@preset/<slug>` as its
model. The buyer answered 502 model_not_found ('No policy-allowed peer
currently serves model "@preset/..."') although sellers within the caps
existed, and the request log recorded no model.
The preset's price caps, the model allow list and the accounting all read
that model, so all three were wrong for such a request.
The gateway now reads the model exactly as the buyer does: `service`
before `model`, trimmed, from the whole body, multipart forms included
through the buyer's own `parseMultipartFormFields`. A shared reader
cannot drift; the 4 MiB cliff and the hand-rolled multipart sniff are
gone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The gateway only read a request body's model from its first 4 MiB (
MAX_MODEL_SNIFF_BYTES), and treated anything larger as "no model". A body over that size therefore skipped the whole@preset/<slug>block:presetSlug(null)returned null, so no preset was applied and nopreset_not_found/preset_requires_jsonguard ran. The literal alias went to the buyer as the model name, which answered502 model_not_found:This was seen on a live gateway. A chat with a multi-megabyte image attached, which the client inlines as base64, produced a body past the limit; every failed request in that chat carried it. Sellers well inside the preset's caps existed the whole time — the caps were never read. The same blind spot made the model allow list and the request log wrong in the same way: those rows recorded no model at all.
The gateway and the buyer were two readers of one body with different limits, and only the buyer's was uncapped. This removes the divergence rather than raising the limit:
Changes
parseJsonBodyno longer drops a body over 4 MiB; every body the gateway accepts (up toMAX_BODY_BYTES) is parsed.MAX_MODEL_SNIFF_BYTESis gone.parseMultipartFormFieldsinstead of a hand-rolled regex over the first 4 MiB, so amodelfield past the first bytes of an upload is found, and the gateway and the buyer cannot disagree about which field wins.effectiveRequestModeltherefore returns exactly what the buyer routes by (service ?? model, trimmed) for every accepted body, so the preset, the allow list and the accounting all see the model that is actually bought.A partial JSON scanner was the alternative; it was rejected as a new correctness risk in a billing and allow-list path. Parsing at the 64 MiB bound costs ~32-36 ms and ~+103-121 MB RSS, which is what the buyer and the dashboard already do for the same bodies on the same host.
Verified end to end against a copy of a deployed build: before, a 5 MiB body reached the buyer as the literal alias with no routing-policy header and a null model in the ledger; after, the buyer receives the resolved model with the preset's caps in the header and the ledger records that model. The three new tests fail before the change (the buyer receives the alias) and pass after; the full CLI suite is green.
Release Notes
@preset/<slug>now applies to every request body the gateway accepts, however large, and an alias that cannot be resolved is refused instead of forwarded. Previously a body over 4 MiB (a chat with a multi-megabyte image attached, which the client inlines as base64) reached the buyer with the literal@preset/<slug>as its model, so the buyer answered502 model_not_foundand the preset's price caps, the model allow list and the request log's model were all wrong for that request.