Skip to content

Stream the buddy agent turn live from the AI service - #219

Merged
kiranfin merged 2 commits into
devfrom
ai-streaming-fixes
Oct 7, 2026
Merged

kiranfin merged 2 commits into
devfrom
ai-streaming-fixes

Conversation

@kiranfin

@kiranfin kiranfin commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Short summary

Streams the agentic buddy turn live from the AI service and carries the model's reasoning through backend tool round trips.

Streaming the agent turn

  • onboarding/buddy_agent.py: the agent loop is now the generator stream_agent_turn. It yields AgentReasoning and AgentToken fragments as the model produces them, an AgentToolUse before each local search (search_docs, grep) and finally the AgentTurnResult. run_agent_turn now just runs that generator to the end and returns the result, so the sync behaviour stays the same.
  • Each hop now uses llm.chat_stream, and the forced final answer (when the step budget is used up) uses llm.stream instead of llm.generate. That means reasoning from the forced hop now shows up in reasoning too.
  • When the reader set a filter and the turn could still end with NO_FILTERED_RESULTS_MESSAGE, the hop's text is held back. It goes out if the hop then searches, or the message replaces it if the hop turns out to be the answer, so the reader never sees an answer the filter message is about to replace.
  • Text from consecutive hops is joined with a blank line so the stream doesn't read as "...let me check.The deploy runs on...".
  • New endpoint POST /onboarding/buddy/agent/stream (SSE). It takes the same request body as /onboarding/buddy/agent and emits reasoning, token, tool_use and one closing result event that has the same fields as BuddyAgentResponse. If the model is unavailable or something fails mid-turn, the stream ends with an error event.
  • api/routes/buddy.py: the response building and the vocabulary mapping moved into shared helpers (_agent_response, _vocabulary) that both endpoints use.

Reasoning through backend tool round trips

  • BuddyAgentMessageSchema has two new optional fields, reasoning and reasoning_details, and _to_message / _from_message map them in both directions.
  • This fixes the hop that resumes after a backend tool. A provider with extended thinking rejected that hop with a 400 because the signed thinking blocks were missing.

Tests

  • New tests/onboarding/test_buddy_agent_stream.py covers event order, held text under a filter, the hop separator and the forced final answer.
  • tests/api/test_buddy.py covers the stream endpoint (events, result payload, error events) and checks that reasoning makes it through a backend tool round trip.
  • The test that made sure reasoning stayed out of the wire messages is replaced by tests that check it is carried out and handed back on resume. The existing tests now use streaming fakes.

Checks

  • I verified the code makes sense intuitively
  • The PR changes affect only this issue, no unrelated/unwanted code changes to other modules/code segments
  • CI runs (./gradlew clean build, keycloack)
  • New business logic is unit tested, including WebMvcTest for api controllers
  • The new functionality is tested manually

- Turn the agent loop into a generator that streams reasoning, tokens and search announcements per hop
- Add POST /onboarding/buddy/agent/stream; the sync endpoint drains the same generator
- Hold back text that a filter override could replace, and join hop texts with a blank line

@daniilperkin daniilperkin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm lgtm lgtm

- Add reasoning and reasoning_details to the buddy agent wire message and map them both ways
- Fix the resume hop after a backend tool, which a thinking-enabled provider rejected with a 400
- Replace the test that pinned reasoning out of messages with carry and resume tests
@kiranfin
kiranfin merged commit bd6e6f8 into dev Oct 7, 2026
4 checks passed
@kiranfin
kiranfin deleted the ai-streaming-fixes branch October 7, 2026 00:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants