perf(ai): streaming requests stop pinning a full clone of their wire payload - #899
Open
santhreal wants to merge 6 commits into
Open
perf(ai): streaming requests stop pinning a full clone of their wire payload#899santhreal wants to merge 6 commits into
santhreal wants to merge 6 commits into
Conversation
openai-completions, openai-responses and azure-openai-responses kept the parsed request object in the diagnostic dump from the moment headers left until the stream ended, so a large context stayed resident twice for the life of every request. The dump now retains only the exact sent bytes and materializes its body through the new materializeDumpBody helper when a 400/413 dump is actually built. The pi-native client serializes its body once and hands the payload hook an isolated parse of those bytes, reusing them on the wire when the hook leaves the payload alone instead of serializing the full context a second time.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The completions-family request builders deep-cloned the whole params graph per attempt just to give the payload hook an isolated object, then serialized the result: on a 32MiB context that is 82ms of clone-plus-stringify against 9ms for serialize-once, paid on every submit before the first byte. The hook now gets an isolated parse of the single serialization, an untouched payload reuses those bytes on the wire, and only a genuinely replaced payload costs a second pass.
VEYYON_DEBUG_STARTUP=1 now writes one synchronous stderr line per submit phase — compaction check, plan arm, context build, memory context, before_agent_start, pre-prompt compaction — so a 'submit feels slow' report names the phase that spent the time instead of offering a guess.
…object Anthropic, Google Generative AI/Vertex, Gemini CLI, Bedrock, Ollama and Codex kept their parsed request object in rawRequestDump for the whole stream; they now record the exact sent bytes and materialize the dump body through materializeDumpBody when a 400/413 dump is built.
The codex websocket/SSE prepare path deep-cloned the whole request graph per attempt for payload-hook isolation. Serialize-once with an isolated parse replaces it: untouched payloads reuse the recorded bytes on the wire, replaced ones pay a second pass.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three changes to how in-flight provider requests hold memory:
openai-completions, openai-responses, azure-openai-responses: the diagnostic rawRequestDump.body held the parsed request object from the moment the request left until the stream ended — so a large context stayed resident twice for the life of every request (the live context plus the dump's clone). The dump now retains only the exact sent bytes (wireBodyJson), and a new materializeDumpBody helper parses those bytes back into body at error time, when a 400/413 dump is actually built. Bytes that never sent or never parse leave body unset rather than fabricating one.
pi-native-client: serialized its full body once for the payload hook and again for fetch. It now serializes once, hands the hook an isolated parse of those bytes, and reuses them on the wire when the hook leaves the payload alone. The hook-rejection wrapping in PiNativePayloadHookError is preserved.
The reasoning-effort fallback retry paths no longer eagerly copy params into the dump; the retried attempt's own serialization records them.
Why
For a session carrying a multi-megabyte context, every concurrent stream — including side streams and advisors — kept an entire context-sized object graph alive doing nothing. This was pure diagnostic insurance that almost never pays out (only 400/413 dumps read it).
Testing
Memory effect
Per in-flight stream: peak retention drops by one full serialized-context object graph (typically ~2–4× the JSON byte size) for openai-completions / responses / azure responses requests, and by one whole-body JSON string + parse cycle per attempt on pi-native.