Skip to content

fix(gateway): Read the model from any body the gateway accepts - #1247

Merged
kotevcode merged 2 commits into
Antseed:mainfrom
Luciddeft:fix/gateway-preset-large-bodies
Oct 17, 2026
Merged

kotevcode merged 2 commits into
Antseed:mainfrom
Luciddeft:fix/gateway-preset-large-bodies

Conversation

@Luciddeft

@Luciddeft Luciddeft commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Description

The gateway only read a request body's model from its first 4 MiB (MAX_MODEL_SNIFF_BYTES), and treated anything larger as "no model". A body over that size therefore skipped the whole @preset/<slug> block: presetSlug(null) returned null, so no preset was applied and no preset_not_found/preset_requires_json guard ran. The literal alias went to the buyer as the model name, which answered 502 model_not_found:

No policy-allowed peer currently serves model "@preset/<slug>".

This was seen on a live gateway. A chat with a multi-megabyte image attached, which the client inlines as base64, produced a body past the limit; every failed request in that chat carried it. Sellers well inside the preset's caps existed the whole time — the caps were never read. The same blind spot made the model allow list and the request log wrong in the same way: those rows recorded no model at all.

The gateway and the buyer were two readers of one body with different limits, and only the buyer's was uncapped. This removes the divergence rather than raising the limit:

Changes

  • parseJsonBody no longer drops a body over 4 MiB; every body the gateway accepts (up to MAX_BODY_BYTES) is parsed. MAX_MODEL_SNIFF_BYTES is gone.
  • Multipart forms are read with the buyer's own parseMultipartFormFields instead of a hand-rolled regex over the first 4 MiB, so a model field past the first bytes of an upload is found, and the gateway and the buyer cannot disagree about which field wins.
  • effectiveRequestModel therefore returns exactly what the buyer routes by (service ?? model, trimmed) for every accepted body, so the preset, the allow list and the accounting all see the model that is actually bought.

A partial JSON scanner was the alternative; it was rejected as a new correctness risk in a billing and allow-list path. Parsing at the 64 MiB bound costs ~32-36 ms and ~+103-121 MB RSS, which is what the buyer and the dashboard already do for the same bodies on the same host.

Verified end to end against a copy of a deployed build: before, a 5 MiB body reached the buyer as the literal alias with no routing-policy header and a null model in the ledger; after, the buyer receives the resolved model with the preset's caps in the header and the ledger records that model. The three new tests fail before the change (the buyer receives the alias) and pass after; the full CLI suite is green.

Release Notes

  • CLI gateway: @preset/<slug> now applies to every request body the gateway accepts, however large, and an alias that cannot be resolved is refused instead of forwarded. Previously a body over 4 MiB (a chat with a multi-megabyte image attached, which the client inlines as base64) reached the buyer with the literal @preset/<slug> as its model, so the buyer answered 502 model_not_found and the preset's price caps, the model allow list and the request log's model were all wrong for that request.

Luciddeft and others added 2 commits October 16, 2026 17:47
The gateway only read a request body's model from its first 4 MiB, so a
larger body (a chat with a few MiB image, which Open WebUI inlines as
base64) arrived at the buyer with the literal `@preset/<slug>` as its
model. The buyer answered 502 model_not_found ('No policy-allowed peer
currently serves model "@preset/..."') although sellers within the caps
existed, and the request log recorded no model.

The preset's price caps, the model allow list and the accounting all read
that model, so all three were wrong for such a request.

The gateway now reads the model exactly as the buyer does: `service`
before `model`, trimmed, from the whole body, multipart forms included
through the buyer's own `parseMultipartFormFields`. A shared reader
cannot drift; the 4 MiB cliff and the hand-rolled multipart sniff are
gone.
@kotevcode
kotevcode merged commit dfadf84 into Antseed:main Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants