fix(media): async submit (no more 524) + forward gpt-image size; remove dead KIE provider#15
Merged
Merged
Conversation
…e dead KIE provider Novita media proxy (`infer_media_proxy`) blocked the `/inference` response for the entire render while polling the upstream task to completion. For renders over ~100s this tripped Cloudflare's origin timeout (524) even though the generation succeeded server-side. Now the proxy returns the `task_id` as soon as the task is submitted and runs the poll + result callback in a detached task; a new failure callback reports async errors so the gen settles/refunds instead of hanging in `pending`. Also: - gpt-image-2 t2i/edit: add `size` to the request-shape whitelist so the chosen dimension actually reaches Novita (was silently dropped). - Remove the unused KIE provider. No active model routes through it and every media variant uses a proxy, so the `None`-proxy KIE media fallback was dead. Deletes `providers/kie.rs` and the `ProviderConfig::KIE` / `ProviderType::KIE` / `KIEKind` / `KIEProviderTypeConfig` plumbing; media variants without a proxy now return an explicit config error. - Add the missing `media_generation: Default::default()` to test `InferenceParams` initializers so the test target compiles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
I have read the Contributor License Agreement (CLA) and hereby sign the CLA. You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two production issues on media generation (surfaced by
openai/gpt-image-2text-to-image), plus a dead-code removal.1. 524 on media generation
NovitaProvider::infer_media_proxyblocked the/inferenceHTTP response for the entire render, polling the upstream async task to completion before returning. Renders over ~100s tripped Cloudflare's origin timeout (524) even though the generation succeeded server-side and the user was charged.Fix: return the
task_idas soon as the task is submitted, and runpoll_async_result+ the result callback in a detachedspawn_ignoring_shutdowntask. This matches the callback-driven contract the backend already implements (submit →pending→ callback). A newpost_media_callback_failurereports async failures so a gen that errors after submit settles/refunds instead of hanging inpending.2. gpt-image
sizedroppedThe
GptImageTextToImage/GptImageEditrequest-shape whitelists omittedsize, so the chosen dimension never reached Novita (and tier pricing fell back to the flat rate on the RouterBase side). Addedsizeto both.3. Remove unused KIE provider
No active model routes through KIE (
type = "kie"is used by zero models; every media variant has a.proxy), so theNone-proxy KIE media fallback was dead. Removesproviders/kie.rsand theProviderConfig::KIE/ProviderType::KIE/KIEKind/KIEProviderTypeConfigplumbing. Media variants without a proxy now return an explicit config error instead of silently falling back.Also
media_generation: Default::default()to testInferenceParamsinitializers (the field was added earlier but ~28 test sites weren't updated), so the test target compiles again.Verification
cargo check --package tensorzero-core✅cargo check --package tensorzero-core --tests✅cargo fmt(unrelated workspace-wide reformatting reverted to keep the diff focused)🤖 Generated with Claude Code