BZ-2517: feat(breeze-buddy): emit voice-to-chat-redirect RTVI event for restricted tools#723
BZ-2517: feat(breeze-buddy): emit voice-to-chat-redirect RTVI event for restricted tools#723titan-juspay wants to merge 1 commit into
Conversation
…or restricted tools When the LLM attempts to call a voice-restricted tool (e.g. initiateAgenticLoop, runSRAnalysis) during a Daily.co voice session, intercept it via the LLM's on_function_calls_started event, speak a short TTS redirect message, and emit a custom 'voice-to-chat-redirect' RTVI server message to the Lighthouse frontend.
WalkthroughA new voice-mode restriction system has been implemented for Breeze Buddy. A utility module defines restricted tools and a redirect message, while the agent now intercepts restricted tool calls during voice sessions and redirects users to chat mode with audio feedback. Changes
Sequence DiagramsequenceDiagram
participant LLM
participant Agent as Agent<br/>(Event Handler)
participant RTVI as RTVI Processor
participant TTS as TTS Frame Queue
LLM->>Agent: on_function_calls_started
Note over Agent: Check: daily mode?<br/>RTVI processor present?<br/>Tool restricted?
alt Restricted Tool Detected
Agent->>TTS: Queue TTSSpeakFrame<br/>(redirect message)
Agent->>RTVI: Emit voice-to-chat-redirect<br/>(tool name, reason)
Agent->>Agent: Break processing loop
else Unrestricted Tool
Agent->>LLM: Continue function calls
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds a Breeze Buddy voice-mode safeguard that detects attempts to invoke long-running “chat-only” tools and redirects the user to chat by speaking a short TTS message and emitting an RTVI voice-to-chat-redirect server event (Daily non-stream mode only).
Changes:
- Introduces
VOICE_RESTRICTED_TOOLSandVOICE_REDIRECT_TTS_MESSAGEconstants for centralized tool restriction configuration. - Stores the LLM service reference on the agent instance and registers an
on_function_calls_startedhandler to trigger the voice-to-chat redirect behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| app/ai/voice/agents/breeze_buddy/utils/voice_restrictions.py | Defines the restricted tool set and the TTS redirect message. |
| app/ai/voice/agents/breeze_buddy/agent/init.py | Hooks an LLM function-call-start handler to speak a redirect and emit an RTVI event when restricted tools are invoked. |
| """Intercept voice-restricted tool calls and redirect to chat.""" | ||
| for fc in function_calls: | ||
| if fc.function_name not in VOICE_RESTRICTED_TOOLS: | ||
| continue | ||
|
|
||
| logger.info( | ||
| f"[VOICE_REDIRECT] Intercepted restricted tool '{fc.function_name}' " | ||
| f"in voice mode, emitting redirect event. " | ||
| f"conversation_id={self.conversation_id}" | ||
| ) | ||
|
|
||
| # Speak the redirect message via TTS | ||
| if self.task: | ||
| try: | ||
| await self.task.queue_frame( | ||
| TTSSpeakFrame(text=VOICE_REDIRECT_TTS_MESSAGE) | ||
| ) | ||
| except Exception as e: | ||
| logger.warning( | ||
| f"[VOICE_REDIRECT] Failed to queue TTS redirect message: {e}" | ||
| ) | ||
|
|
||
| # Emit the RTVI event to Lighthouse frontend | ||
| await self._emit_rtvi_event( | ||
| "voice-to-chat-redirect", | ||
| { | ||
| "tool": fc.function_name, | ||
| "reason": "voice_restricted", | ||
| "conversation_id": self.conversation_id or "", | ||
| }, | ||
| ) | ||
| break # Handle only the first restricted tool per batch | ||
|
|
There was a problem hiding this comment.
The on_function_calls_started handler only emits a redirect message/event but does not prevent the restricted tool from executing. If these tools are long-running (as the docstring suggests), they can still consume LLM context/compute even after the redirect. To actually enforce the restriction in voice mode, block execution (e.g., don’t register these tools for voice sessions, or short-circuit their handlers to immediately return a safe result and skip the real work, and/or cancel/terminate the current pipeline after emitting the redirect).
| """Intercept voice-restricted tool calls and redirect to chat.""" | |
| for fc in function_calls: | |
| if fc.function_name not in VOICE_RESTRICTED_TOOLS: | |
| continue | |
| logger.info( | |
| f"[VOICE_REDIRECT] Intercepted restricted tool '{fc.function_name}' " | |
| f"in voice mode, emitting redirect event. " | |
| f"conversation_id={self.conversation_id}" | |
| ) | |
| # Speak the redirect message via TTS | |
| if self.task: | |
| try: | |
| await self.task.queue_frame( | |
| TTSSpeakFrame(text=VOICE_REDIRECT_TTS_MESSAGE) | |
| ) | |
| except Exception as e: | |
| logger.warning( | |
| f"[VOICE_REDIRECT] Failed to queue TTS redirect message: {e}" | |
| ) | |
| # Emit the RTVI event to Lighthouse frontend | |
| await self._emit_rtvi_event( | |
| "voice-to-chat-redirect", | |
| { | |
| "tool": fc.function_name, | |
| "reason": "voice_restricted", | |
| "conversation_id": self.conversation_id or "", | |
| }, | |
| ) | |
| break # Handle only the first restricted tool per batch | |
| """Intercept voice-restricted tool calls, block them, and redirect to chat.""" | |
| restricted_calls = [ | |
| fc | |
| for fc in function_calls | |
| if fc.function_name in VOICE_RESTRICTED_TOOLS | |
| ] | |
| if not restricted_calls: | |
| return | |
| allowed_calls = [ | |
| fc | |
| for fc in function_calls | |
| if fc.function_name not in VOICE_RESTRICTED_TOOLS | |
| ] | |
| try: | |
| function_calls[:] = allowed_calls | |
| except Exception as e: | |
| logger.warning( | |
| f"[VOICE_REDIRECT] Failed to remove restricted tools from function call batch: {e}" | |
| ) | |
| return | |
| first_blocked_call = restricted_calls[0] | |
| blocked_tools = [fc.function_name for fc in restricted_calls] | |
| logger.info( | |
| f"[VOICE_REDIRECT] Blocked restricted tool(s) {blocked_tools} " | |
| f"in voice mode and emitted redirect event. " | |
| f"conversation_id={self.conversation_id}" | |
| ) | |
| # Speak the redirect message via TTS | |
| if self.task: | |
| try: | |
| await self.task.queue_frame( | |
| TTSSpeakFrame(text=VOICE_REDIRECT_TTS_MESSAGE) | |
| ) | |
| except Exception as e: | |
| logger.warning( | |
| f"[VOICE_REDIRECT] Failed to queue TTS redirect message: {e}" | |
| ) | |
| # Emit the RTVI event to Lighthouse frontend | |
| await self._emit_rtvi_event( | |
| "voice-to-chat-redirect", | |
| { | |
| "tool": first_blocked_call.function_name, | |
| "reason": "voice_restricted", | |
| "conversation_id": self.conversation_id or "", | |
| }, | |
| ) | |
| if not allowed_calls: | |
| return |
| # Voice-to-chat redirect: intercept restricted tool calls in daily mode | ||
| if self.is_daily_mode and self._rtvi_processor and not self.is_stream_mode and self._llm: | ||
|
|
||
| @self._llm.event_handler("on_function_calls_started") | ||
| async def on_function_calls_started(service, function_calls): | ||
| """Intercept voice-restricted tool calls and redirect to chat.""" | ||
| for fc in function_calls: | ||
| if fc.function_name not in VOICE_RESTRICTED_TOOLS: | ||
| continue | ||
|
|
||
| logger.info( | ||
| f"[VOICE_REDIRECT] Intercepted restricted tool '{fc.function_name}' " | ||
| f"in voice mode, emitting redirect event. " | ||
| f"conversation_id={self.conversation_id}" | ||
| ) | ||
|
|
||
| # Speak the redirect message via TTS | ||
| if self.task: | ||
| try: | ||
| await self.task.queue_frame( | ||
| TTSSpeakFrame(text=VOICE_REDIRECT_TTS_MESSAGE) | ||
| ) | ||
| except Exception as e: | ||
| logger.warning( | ||
| f"[VOICE_REDIRECT] Failed to queue TTS redirect message: {e}" | ||
| ) | ||
|
|
||
| # Emit the RTVI event to Lighthouse frontend | ||
| await self._emit_rtvi_event( | ||
| "voice-to-chat-redirect", | ||
| { | ||
| "tool": fc.function_name, | ||
| "reason": "voice_restricted", | ||
| "conversation_id": self.conversation_id or "", | ||
| }, | ||
| ) | ||
| break # Handle only the first restricted tool per batch |
There was a problem hiding this comment.
After a restricted tool is detected, the handler emits a redirect but doesn’t guard against subsequent restricted tool calls later in the same session. This can lead to repeated TTS messages and duplicate voice-to-chat-redirect events if the LLM retries or calls multiple restricted tools across turns. Consider adding an idempotency flag (emit once per conversation) and/or ending the voice session after emitting the redirect.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@app/ai/voice/agents/breeze_buddy/agent/__init__.py`:
- Around line 654-655: The long if-condition "if self.is_daily_mode and
self._rtvi_processor and not self.is_stream_mode and self._llm:" causes Black
formatting failure; split the conditional across multiple lines using
parentheses and one boolean expression per line to satisfy Black (refer to the
conditional that checks self.is_daily_mode, self._rtvi_processor, not
self.is_stream_mode, and self._llm in the __init__ of the Breeze Buddy agent) so
the line length is reduced and Black passes.
- Around line 657-690: The on_function_calls_started handler currently only
notifies about voice-restricted tools (VOICE_RESTRICTED_TOOLS) but does not stop
their execution; update the flow so restricted tools do not run in voice mode by
either (A) filtering them out of the incoming function_calls list or signaling
cancellation from on_function_calls_started (so the LLM engine will not invoke
them), or (B) registering voice-mode stubs for the five entries in
VOICE_RESTRICTED_TOOLS when building global functions
(GlobalFunctionRegistry.build / global_functions) that immediately emit the same
redirect via _emit_rtvi_event and enqueue TTSSpeakFrame via task.queue_frame and
then return early; modify on_function_calls_started to both emit the RTVI/TTS
and ensure the function call is prevented (by removing from function_calls or
returning the cancellation indicator) so the original tool implementation is
never executed in voice mode.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ca8ffde0-dd40-42ac-8ccd-442869a8057e
📒 Files selected for processing (2)
app/ai/voice/agents/breeze_buddy/agent/__init__.pyapp/ai/voice/agents/breeze_buddy/utils/voice_restrictions.py
| # Voice-to-chat redirect: intercept restricted tool calls in daily mode | ||
| if self.is_daily_mode and self._rtvi_processor and not self.is_stream_mode and self._llm: |
There was a problem hiding this comment.
Fix the Black formatting failure.
The CI failure points at this long conditional. Splitting it keeps the file Black-compliant.
🎨 Proposed formatting fix
- if self.is_daily_mode and self._rtvi_processor and not self.is_stream_mode and self._llm:
+ if (
+ self.is_daily_mode
+ and self._rtvi_processor
+ and not self.is_stream_mode
+ and self._llm
+ ):🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@app/ai/voice/agents/breeze_buddy/agent/__init__.py` around lines 654 - 655,
The long if-condition "if self.is_daily_mode and self._rtvi_processor and not
self.is_stream_mode and self._llm:" causes Black formatting failure; split the
conditional across multiple lines using parentheses and one boolean expression
per line to satisfy Black (refer to the conditional that checks
self.is_daily_mode, self._rtvi_processor, not self.is_stream_mode, and self._llm
in the __init__ of the Breeze Buddy agent) so the line length is reduced and
Black passes.
| @self._llm.event_handler("on_function_calls_started") | ||
| async def on_function_calls_started(service, function_calls): | ||
| """Intercept voice-restricted tool calls and redirect to chat.""" | ||
| for fc in function_calls: | ||
| if fc.function_name not in VOICE_RESTRICTED_TOOLS: | ||
| continue | ||
|
|
||
| logger.info( | ||
| f"[VOICE_REDIRECT] Intercepted restricted tool '{fc.function_name}' " | ||
| f"in voice mode, emitting redirect event. " | ||
| f"conversation_id={self.conversation_id}" | ||
| ) | ||
|
|
||
| # Speak the redirect message via TTS | ||
| if self.task: | ||
| try: | ||
| await self.task.queue_frame( | ||
| TTSSpeakFrame(text=VOICE_REDIRECT_TTS_MESSAGE) | ||
| ) | ||
| except Exception as e: | ||
| logger.warning( | ||
| f"[VOICE_REDIRECT] Failed to queue TTS redirect message: {e}" | ||
| ) | ||
|
|
||
| # Emit the RTVI event to Lighthouse frontend | ||
| await self._emit_rtvi_event( | ||
| "voice-to-chat-redirect", | ||
| { | ||
| "tool": fc.function_name, | ||
| "reason": "voice_restricted", | ||
| "conversation_id": self.conversation_id or "", | ||
| }, | ||
| ) | ||
| break # Handle only the first restricted tool per batch |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description:
# Inspect Pipecat/FlowManager function-call wiring and all usages of the new
# restricted-tool hook. Expected result: either a documented cancellation path
# from on_function_calls_started, or restricted tools being removed/replaced
# before execution in Daily voice mode.
set -euo pipefail
echo "== on_function_calls_started usages =="
rg -n -C4 'on_function_calls_started|function_calls_started'
echo
echo "== function registration/execution paths =="
rg -n -C4 'register_function|FunctionCallParams|function_name|handler_map|setup_flow_manager'
echo
echo "== voice restriction usages =="
rg -n -C4 'VOICE_RESTRICTED_TOOLS|voice-to-chat-redirect|VOICE_REDIRECT_TTS_MESSAGE'Repository: juspay/clairvoyance
Length of output: 50378
🏁 Script executed:
#!/bin/bash
# Find where VOICE_RESTRICTED_TOOLS is defined and how restricted tools are registered
echo "=== VOICE_RESTRICTED_TOOLS definition ==="
rg -n 'VOICE_RESTRICTED_TOOLS\s*=' -A 5
echo
echo "=== Handler return value / cancellation patterns ==="
rg -n 'on_function_calls_started|event_handler' -A 10 app/ai/voice/agents/breeze_buddy/agent/__init__.py | head -80
echo
echo "=== Check if tools are registered conditionally ==="
rg -n 'register_function|register_direct_function' app/ai/voice/agents/breeze_buddy/agent/__init__.py | head -20Repository: juspay/clairvoyance
Length of output: 4274
🏁 Script executed:
#!/bin/bash
# Search for VOICE_RESTRICTED_TOOLS more broadly, including imports
echo "=== VOICE_RESTRICTED_TOOLS anywhere in codebase ==="
rg -n 'VOICE_RESTRICTED_TOOLS' --type py
echo
echo "=== Check imports in breeze_buddy agent __init__.py ==="
head -100 app/ai/voice/agents/breeze_buddy/agent/__init__.py | grep -E 'import|from'
echo
echo "=== Flow.py to see how tools are registered ==="
rg -n 'register.*function|VOICE_RESTRICTED' app/ai/voice/agents/breeze_buddy/agent/flow.py | head -20Repository: juspay/clairvoyance
Length of output: 3005
🏁 Script executed:
#!/bin/bash
# Get the actual definition of VOICE_RESTRICTED_TOOLS
echo "=== voice_restrictions.py content ==="
cat app/ai/voice/agents/breeze_buddy/utils/voice_restrictions.py
echo
echo "=== Flow registration - how are restricted tools registered? ==="
rg -n 'register_function|register_direct_function' app/ai/voice/agents/breeze_buddy/agent/flow.py -B 2 -A 8 | head -100Repository: juspay/clairvoyance
Length of output: 1059
🏁 Script executed:
#!/bin/bash
# Broader search for function registrations
echo "=== All register_function calls in flow.py ==="
rg -n 'register' app/ai/voice/agents/breeze_buddy/agent/flow.py -B 1 -A 3
echo
echo "=== Check if restricted tools appear in any registration ==="
rg -n 'initiateAgenticLoop|generateAdsPerformanceReport|generateReport' app/ai/voice/agents/breeze_buddy/ -r
echo
echo "=== Check on_function_calls_started full handler and whether it returns anything ==="
sed -n '657,690p' app/ai/voice/agents/breeze_buddy/agent/__init__.pyRepository: juspay/clairvoyance
Length of output: 2346
🏁 Script executed:
#!/bin/bash
# Check where functions are made available to LLM
echo "=== Flow builder function schema generation ==="
rg -n 'FlowsFunctionSchema|_build_function_schema|function_schema' app/ai/voice/agents/breeze_buddy/agent/flow.py -B 2 -A 5 | head -60
echo
echo "=== Check if restricted tools are excluded from schema generation ==="
rg -n 'VOICE_RESTRICTED_TOOLS' app/ai/voice/agents/breeze_buddy/ -B 3 -A 3
echo
echo "=== Check FlowManager setup - how functions are registered ==="
rg -n 'setup_flow_manager|FlowManager|_setup_llm' app/ai/voice/agents/breeze_buddy/agent/__init__.py -B 1 -A 5 | head -80Repository: juspay/clairvoyance
Length of output: 4543
🏁 Script executed:
#!/bin/bash
# Look at setup_flow_manager implementation
echo "=== setup_flow_manager function ==="
rg -n 'def setup_flow_manager' app/ai/voice/agents/breeze_buddy/agent/flow.py -A 50
echo
echo "=== Check if restricted tools are filtered during global_functions generation ==="
rg -n 'global_functions|build_function_schema' app/ai/voice/agents/breeze_buddy/template/builder.py -B 2 -A 8 | head -100Repository: juspay/clairvoyance
Length of output: 3876
🏁 Script executed:
#!/bin/bash
# Find GlobalFunctionRegistry to see if it filters restricted tools
echo "=== GlobalFunctionRegistry definition and build method ==="
rg -n 'class GlobalFunctionRegistry|def build' app/ai/voice/agents/breeze_buddy/template/builder.py -A 30 | head -80
echo
echo "=== Check if restricted tools list is used anywhere for filtering ==="
rg -n 'if.*VOICE_RESTRICTED|not in VOICE_RESTRICTED|in VOICE_RESTRICTED' app/ai/voice/agents/breeze_buddy/ -B 2 -A 2Repository: juspay/clairvoyance
Length of output: 3461
🏁 Script executed:
#!/bin/bash
# Find GlobalFunctionRegistry implementation
echo "=== Find GlobalFunctionRegistry class ==="
fd -type f -name '*.py' -path '*breeze_buddy*' | xargs grep -l 'class GlobalFunctionRegistry'
echo
echo "=== If found, show the build method ==="
rg -n 'class GlobalFunctionRegistry' app/ai/voice/agents/breeze_buddy/ -A 100 | grep -A 100 'def build'Repository: juspay/clairvoyance
Length of output: 5323
🏁 Script executed:
#!/bin/bash
# Check if restricted tools are in global_functions config and look for any other guard mechanisms
echo "=== Check template examples for global_functions config ==="
rg -n 'global_functions' app/ai/voice/agents/breeze_buddy/ --type py -B 2 -A 5 | head -80
echo
echo "=== Check if there are any stubs/wrappers for restricted tools ==="
rg -n 'initiateAgenticLoop|generateAdsPerformanceReport|generateReport|generateShopifyBreezeGoLiveReport|runSRAnalysis' app/ai/voice/agents/breeze_buddy/ -B 2 -A 2Repository: juspay/clairvoyance
Length of output: 7494
🏁 Script executed:
#!/bin/bash
# Look for handler implementations and check if restricted tools are actually available
echo "=== Search for handler implementations ==="
rg -n 'async def initiateAgenticLoop|def initiateAgenticLoop|async def generateAdsPerformanceReport|def generateAdsPerformanceReport' app/ai/voice/agents/breeze_buddy/ -A 3
echo
echo "=== Check if there are any function wrappers that might prevent execution ==="
rg -n 'def wrapped_|wrapper|guard|prevent' app/ai/voice/agents/breeze_buddy/template/ --type py -B 1 -A 5 | head -80
echo
echo "=== Look at the actual on_function_calls_started handler more carefully - check if it modifies function_calls ==="
sed -n '655,695p' app/ai/voice/agents/breeze_buddy/agent/__init__.pyRepository: juspay/clairvoyance
Length of output: 9711
The on_function_calls_started handler does not prevent execution of restricted tools; it only notifies.
The handler at lines 657–690 is notification-only: it logs the interception, queues a TTS redirect message, and emits an RTVI event, but does not modify the function_calls list, return a cancellation signal, or prevent execution. If these tools are registered in the template's global_functions config (and GlobalFunctionRegistry.build() applies no filtering), they will execute after the handler fires, potentially exhausting LLM context during voice conversations.
Either exclude restricted tools from registration in voice mode, or register voice-mode stubs for the five tools in VOICE_RESTRICTED_TOOLS that only emit the redirect response and return early.
🧰 Tools
🪛 Ruff (0.15.10)
[warning] 676-676: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@app/ai/voice/agents/breeze_buddy/agent/__init__.py` around lines 657 - 690,
The on_function_calls_started handler currently only notifies about
voice-restricted tools (VOICE_RESTRICTED_TOOLS) but does not stop their
execution; update the flow so restricted tools do not run in voice mode by
either (A) filtering them out of the incoming function_calls list or signaling
cancellation from on_function_calls_started (so the LLM engine will not invoke
them), or (B) registering voice-mode stubs for the five entries in
VOICE_RESTRICTED_TOOLS when building global functions
(GlobalFunctionRegistry.build / global_functions) that immediately emit the same
redirect via _emit_rtvi_event and enqueue TTSSpeakFrame via task.queue_frame and
then return early; modify on_function_calls_started to both emit the RTVI/TTS
and ensure the function call is prevented (by removing from function_calls or
returning the cancellation indicator) so the original tool implementation is
never executed in voice mode.
Summary
VOICE_RESTRICTED_TOOLSfrozenset andVOICE_REDIRECT_TTS_MESSAGEconstant in a newutils/voice_restrictions.pymoduleon_function_calls_startedhandler on the LLM service in daily (non-stream) mode: when the LLM invokes any voice-restricted tool, the agent speaks a short TTS redirect message and emits avoice-to-chat-redirectRTVI server event to the Lighthouse frontendinitiateAgenticLoop,generateAdsPerformanceReport,generateShopifyBreezeGoLiveReport,runSRAnalysis,generateReportFiles Changed
app/ai/voice/agents/breeze_buddy/utils/voice_restrictions.py(created) — definesVOICE_RESTRICTED_TOOLSandVOICE_REDIRECT_TTS_MESSAGEapp/ai/voice/agents/breeze_buddy/agent/__init__.py(modified) — imports voice restriction constants, addsself._llmattribute, stores LLM reference aftercreate_services(), registerson_function_calls_startedhandler in_register_event_handlers()Notes
on_function_calls_startedon the LLM service (same pattern as the Automatic agent) rather than an RTVI processor event, because RTVIProcessor does not expose a function-call intercept eventis_daily_mode and not is_stream_mode— telephony and stream-mode sessions are unaffected[VOICE_REDIRECT]prefix for easy filteringDiscussion Thread
https://slack.com/archives/C08P35EAER0/p1776697311985229
Summary by CodeRabbit