Skip to content

BZ-2517: feat(breeze-buddy): emit voice-to-chat-redirect RTVI event for restricted tools #723

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
titan-juspay wants to merge 1 commit into
base: release
Choose a base branch
from
+73 −0
Open

BZ-2517: feat(breeze-buddy): emit voice-to-chat-redirect RTVI event for restricted tools #723

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions app/ai/voice/agents/breeze_buddy/agent/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,10 @@
create_background_sound_mixer,
track_error,
)
from app.ai.voice.agents.breeze_buddy.utils.voice_restrictions import (
VOICE_REDIRECT_TTS_MESSAGE,
VOICE_RESTRICTED_TOOLS,
)
from app.ai.voice.agents.breeze_buddy.utils.transport.websockets import (
close_websocket_safely,
)
Expand Down Expand Up @@ -164,6 +168,9 @@ def __init__(
# RTVI processor for daily mode real-time events
self._rtvi_processor: Any = None

# LLM service (set after create_services; None in stream mode)
self._llm: Any = None

# Stream mode transcript collector (replaces LLMContext for transcription)
self._transcript_collector: Optional[TranscriptCollectorProcessor] = None

Expand Down Expand Up @@ -644,6 +651,44 @@ async def on_client_ready(rtvi):
RTVIServerMessageFrame(data={"type": "bot-ready"})
)

# Voice-to-chat redirect: intercept restricted tool calls in daily mode
if self.is_daily_mode and self._rtvi_processor and not self.is_stream_mode and self._llm:
Comment on lines +654 to +655
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Fix the Black formatting failure.

The CI failure points at this long conditional. Splitting it keeps the file Black-compliant.

🎨 Proposed formatting fix 
-        if self.is_daily_mode and self._rtvi_processor and not self.is_stream_mode and self._llm:
+        if (
+            self.is_daily_mode
+            and self._rtvi_processor
+            and not self.is_stream_mode
+            and self._llm
+        ):
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@app/ai/voice/agents/breeze_buddy/agent/__init__.py` around lines 654 - 655,
The long if-condition "if self.is_daily_mode and self._rtvi_processor and not
self.is_stream_mode and self._llm:" causes Black formatting failure; split the
conditional across multiple lines using parentheses and one boolean expression
per line to satisfy Black (refer to the conditional that checks
self.is_daily_mode, self._rtvi_processor, not self.is_stream_mode, and self._llm
in the __init__ of the Breeze Buddy agent) so the line length is reduced and
Black passes.


@self._llm.event_handler("on_function_calls_started")
async def on_function_calls_started(service, function_calls):
"""Intercept voice-restricted tool calls and redirect to chat."""
for fc in function_calls:
if fc.function_name not in VOICE_RESTRICTED_TOOLS:
continue

logger.info(
f"[VOICE_REDIRECT] Intercepted restricted tool '{fc.function_name}' "
f"in voice mode, emitting redirect event. "
f"conversation_id={self.conversation_id}"
)

# Speak the redirect message via TTS
if self.task:
try:
await self.task.queue_frame(
TTSSpeakFrame(text=VOICE_REDIRECT_TTS_MESSAGE)
)
except Exception as e:
logger.warning(
f"[VOICE_REDIRECT] Failed to queue TTS redirect message: {e}"
)

# Emit the RTVI event to Lighthouse frontend
await self._emit_rtvi_event(
"voice-to-chat-redirect",
{
"tool": fc.function_name,
"reason": "voice_restricted",
"conversation_id": self.conversation_id or "",
},
)
break # Handle only the first restricted tool per batch
Comment on lines +654 to +690
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a restricted tool is detected, the handler emits a redirect but doesn’t guard against subsequent restricted tool calls later in the same session. This can lead to repeated TTS messages and duplicate voice-to-chat-redirect events if the LLM retries or calls multiple restricted tools across turns. Consider adding an idempotency flag (emit once per conversation) and/or ending the voice session after emitting the redirect.

Copilot uses AI. Check for mistakes.
Comment on lines +657 to +690
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description:
# Inspect Pipecat/FlowManager function-call wiring and all usages of the new
# restricted-tool hook. Expected result: either a documented cancellation path
# from on_function_calls_started, or restricted tools being removed/replaced
# before execution in Daily voice mode.

set -euo pipefail

echo "== on_function_calls_started usages =="
rg -n -C4 'on_function_calls_started|function_calls_started'

echo
echo "== function registration/execution paths =="
rg -n -C4 'register_function|FunctionCallParams|function_name|handler_map|setup_flow_manager'

echo
echo "== voice restriction usages =="
rg -n -C4 'VOICE_RESTRICTED_TOOLS|voice-to-chat-redirect|VOICE_REDIRECT_TTS_MESSAGE'

Repository: juspay/clairvoyance

Length of output: 50378

🏁 Script executed:

#!/bin/bash
# Find where VOICE_RESTRICTED_TOOLS is defined and how restricted tools are registered

echo "=== VOICE_RESTRICTED_TOOLS definition ==="
rg -n 'VOICE_RESTRICTED_TOOLS\s*=' -A 5

echo
echo "=== Handler return value / cancellation patterns ==="
rg -n 'on_function_calls_started|event_handler' -A 10 app/ai/voice/agents/breeze_buddy/agent/__init__.py | head -80

echo
echo "=== Check if tools are registered conditionally ==="
rg -n 'register_function|register_direct_function' app/ai/voice/agents/breeze_buddy/agent/__init__.py | head -20

Repository: juspay/clairvoyance

Length of output: 4274

🏁 Script executed:

#!/bin/bash
# Search for VOICE_RESTRICTED_TOOLS more broadly, including imports

echo "=== VOICE_RESTRICTED_TOOLS anywhere in codebase ==="
rg -n 'VOICE_RESTRICTED_TOOLS' --type py

echo
echo "=== Check imports in breeze_buddy agent __init__.py ==="
head -100 app/ai/voice/agents/breeze_buddy/agent/__init__.py | grep -E 'import|from'

echo
echo "=== Flow.py to see how tools are registered ==="
rg -n 'register.*function|VOICE_RESTRICTED' app/ai/voice/agents/breeze_buddy/agent/flow.py | head -20

Repository: juspay/clairvoyance

Length of output: 3005

🏁 Script executed:

#!/bin/bash
# Get the actual definition of VOICE_RESTRICTED_TOOLS

echo "=== voice_restrictions.py content ==="
cat app/ai/voice/agents/breeze_buddy/utils/voice_restrictions.py

echo
echo "=== Flow registration - how are restricted tools registered? ==="
rg -n 'register_function|register_direct_function' app/ai/voice/agents/breeze_buddy/agent/flow.py -B 2 -A 8 | head -100

Repository: juspay/clairvoyance

Length of output: 1059

🏁 Script executed:

#!/bin/bash
# Broader search for function registrations

echo "=== All register_function calls in flow.py ==="
rg -n 'register' app/ai/voice/agents/breeze_buddy/agent/flow.py -B 1 -A 3

echo
echo "=== Check if restricted tools appear in any registration ==="
rg -n 'initiateAgenticLoop|generateAdsPerformanceReport|generateReport' app/ai/voice/agents/breeze_buddy/ -r

echo
echo "=== Check on_function_calls_started full handler and whether it returns anything ==="
sed -n '657,690p' app/ai/voice/agents/breeze_buddy/agent/__init__.py

Repository: juspay/clairvoyance

Length of output: 2346

🏁 Script executed:

#!/bin/bash
# Check where functions are made available to LLM

echo "=== Flow builder function schema generation ==="
rg -n 'FlowsFunctionSchema|_build_function_schema|function_schema' app/ai/voice/agents/breeze_buddy/agent/flow.py -B 2 -A 5 | head -60

echo
echo "=== Check if restricted tools are excluded from schema generation ==="
rg -n 'VOICE_RESTRICTED_TOOLS' app/ai/voice/agents/breeze_buddy/ -B 3 -A 3

echo
echo "=== Check FlowManager setup - how functions are registered ==="
rg -n 'setup_flow_manager|FlowManager|_setup_llm' app/ai/voice/agents/breeze_buddy/agent/__init__.py -B 1 -A 5 | head -80

Repository: juspay/clairvoyance

Length of output: 4543

🏁 Script executed:

#!/bin/bash
# Look at setup_flow_manager implementation

echo "=== setup_flow_manager function ==="
rg -n 'def setup_flow_manager' app/ai/voice/agents/breeze_buddy/agent/flow.py -A 50

echo
echo "=== Check if restricted tools are filtered during global_functions generation ==="
rg -n 'global_functions|build_function_schema' app/ai/voice/agents/breeze_buddy/template/builder.py -B 2 -A 8 | head -100

Repository: juspay/clairvoyance

Length of output: 3876

🏁 Script executed:

#!/bin/bash
# Find GlobalFunctionRegistry to see if it filters restricted tools

echo "=== GlobalFunctionRegistry definition and build method ==="
rg -n 'class GlobalFunctionRegistry|def build' app/ai/voice/agents/breeze_buddy/template/builder.py -A 30 | head -80

echo
echo "=== Check if restricted tools list is used anywhere for filtering ==="
rg -n 'if.*VOICE_RESTRICTED|not in VOICE_RESTRICTED|in VOICE_RESTRICTED' app/ai/voice/agents/breeze_buddy/ -B 2 -A 2

Repository: juspay/clairvoyance

Length of output: 3461

🏁 Script executed:

#!/bin/bash
# Find GlobalFunctionRegistry implementation

echo "=== Find GlobalFunctionRegistry class ==="
fd -type f -name '*.py' -path '*breeze_buddy*' | xargs grep -l 'class GlobalFunctionRegistry'

echo
echo "=== If found, show the build method ==="
rg -n 'class GlobalFunctionRegistry' app/ai/voice/agents/breeze_buddy/ -A 100 | grep -A 100 'def build'

Repository: juspay/clairvoyance

Length of output: 5323

🏁 Script executed:

#!/bin/bash
# Check if restricted tools are in global_functions config and look for any other guard mechanisms

echo "=== Check template examples for global_functions config ==="
rg -n 'global_functions' app/ai/voice/agents/breeze_buddy/ --type py -B 2 -A 5 | head -80

echo
echo "=== Check if there are any stubs/wrappers for restricted tools ==="
rg -n 'initiateAgenticLoop|generateAdsPerformanceReport|generateReport|generateShopifyBreezeGoLiveReport|runSRAnalysis' app/ai/voice/agents/breeze_buddy/ -B 2 -A 2

Repository: juspay/clairvoyance

Length of output: 7494

🏁 Script executed:

#!/bin/bash
# Look for handler implementations and check if restricted tools are actually available

echo "=== Search for handler implementations ==="
rg -n 'async def initiateAgenticLoop|def initiateAgenticLoop|async def generateAdsPerformanceReport|def generateAdsPerformanceReport' app/ai/voice/agents/breeze_buddy/ -A 3

echo
echo "=== Check if there are any function wrappers that might prevent execution ==="
rg -n 'def wrapped_|wrapper|guard|prevent' app/ai/voice/agents/breeze_buddy/template/ --type py -B 1 -A 5 | head -80

echo
echo "=== Look at the actual on_function_calls_started handler more carefully - check if it modifies function_calls ==="
sed -n '655,695p' app/ai/voice/agents/breeze_buddy/agent/__init__.py

Repository: juspay/clairvoyance

Length of output: 9711

The on_function_calls_started handler does not prevent execution of restricted tools; it only notifies.

The handler at lines 657–690 is notification-only: it logs the interception, queues a TTS redirect message, and emits an RTVI event, but does not modify the function_calls list, return a cancellation signal, or prevent execution. If these tools are registered in the template's global_functions config (and GlobalFunctionRegistry.build() applies no filtering), they will execute after the handler fires, potentially exhausting LLM context during voice conversations.

Either exclude restricted tools from registration in voice mode, or register voice-mode stubs for the five tools in VOICE_RESTRICTED_TOOLS that only emit the redirect response and return early.

🧰 Tools
🪛 Ruff (0.15.10)

[warning] 676-676: Do not catch blind exception: Exception

(BLE001)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@app/ai/voice/agents/breeze_buddy/agent/__init__.py` around lines 657 - 690,
The on_function_calls_started handler currently only notifies about
voice-restricted tools (VOICE_RESTRICTED_TOOLS) but does not stop their
execution; update the flow so restricted tools do not run in voice mode by
either (A) filtering them out of the incoming function_calls list or signaling
cancellation from on_function_calls_started (so the LLM engine will not invoke
them), or (B) registering voice-mode stubs for the five entries in
VOICE_RESTRICTED_TOOLS when building global functions
(GlobalFunctionRegistry.build / global_functions) that immediately emit the same
redirect via _emit_rtvi_event and enqueue TTSSpeakFrame via task.queue_frame and
then return early; modify on_function_calls_started to both emit the RTVI/TTS
and ensure the function call is prevented (by removing from function_calls or
returning the cancellation indicator) so the original tool implementation is
never executed in voice mode.


Comment on lines +659 to +691
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The on_function_calls_started handler only emits a redirect message/event but does not prevent the restricted tool from executing. If these tools are long-running (as the docstring suggests), they can still consume LLM context/compute even after the redirect. To actually enforce the restriction in voice mode, block execution (e.g., don’t register these tools for voice sessions, or short-circuit their handlers to immediately return a safe result and skip the real work, and/or cancel/terminate the current pipeline after emitting the redirect).

Suggested change
"""Intercept voice-restricted tool calls and redirect to chat."""
for fc in function_calls:
if fc.function_name not in VOICE_RESTRICTED_TOOLS:
continue
logger.info(
f"[VOICE_REDIRECT] Intercepted restricted tool '{fc.function_name}' "
f"in voice mode, emitting redirect event. "
f"conversation_id={self.conversation_id}"
)
# Speak the redirect message via TTS
if self.task:
try:
await self.task.queue_frame(
TTSSpeakFrame(text=VOICE_REDIRECT_TTS_MESSAGE)
)
except Exception as e:
logger.warning(
f"[VOICE_REDIRECT] Failed to queue TTS redirect message: {e}"
)
# Emit the RTVI event to Lighthouse frontend
await self._emit_rtvi_event(
"voice-to-chat-redirect",
{
"tool": fc.function_name,
"reason": "voice_restricted",
"conversation_id": self.conversation_id or "",
},
)
break # Handle only the first restricted tool per batch
"""Intercept voice-restricted tool calls, block them, and redirect to chat."""
restricted_calls = [
fc
for fc in function_calls
if fc.function_name in VOICE_RESTRICTED_TOOLS
]
if not restricted_calls:
return
allowed_calls = [
fc
for fc in function_calls
if fc.function_name not in VOICE_RESTRICTED_TOOLS
]
try:
function_calls[:] = allowed_calls
except Exception as e:
logger.warning(
f"[VOICE_REDIRECT] Failed to remove restricted tools from function call batch: {e}"
)
return
first_blocked_call = restricted_calls[0]
blocked_tools = [fc.function_name for fc in restricted_calls]
logger.info(
f"[VOICE_REDIRECT] Blocked restricted tool(s) {blocked_tools} "
f"in voice mode and emitted redirect event. "
f"conversation_id={self.conversation_id}"
)
# Speak the redirect message via TTS
if self.task:
try:
await self.task.queue_frame(
TTSSpeakFrame(text=VOICE_REDIRECT_TTS_MESSAGE)
)
except Exception as e:
logger.warning(
f"[VOICE_REDIRECT] Failed to queue TTS redirect message: {e}"
)
# Emit the RTVI event to Lighthouse frontend
await self._emit_rtvi_event(
"voice-to-chat-redirect",
{
"tool": first_blocked_call.function_name,
"reason": "voice_restricted",
"conversation_id": self.conversation_id or "",
},
)
if not allowed_calls:
return

Copilot uses AI. Check for mistakes.
# Stream mode: accept tts-speak via RTVI client-message (PipecatClient SDK)
if self.is_stream_mode and self._rtvi_processor:

Expand Down Expand Up @@ -808,6 +853,7 @@ async def run(self, runner_args: Optional[RunnerArguments] = None) -> None:
stt, llm, tts = await create_services(
self.configurations, include_llm=not is_stream
)
self._llm = llm # May be None in stream mode
if not is_stream:
assert llm is not None, "LLM is required in agent mode"

Expand Down
27 changes: 27 additions & 0 deletions app/ai/voice/agents/breeze_buddy/utils/voice_restrictions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
"""Voice-restricted tool definitions for Breeze Buddy.

Tools listed here are available in chat mode but not in voice mode
because they are long-running agentic loops that would exhaust the
LLM context window during a real-time voice conversation.
"""

from typing import FrozenSet

# Tools that are only available in chat mode.
# When the LLM invokes one of these during a voice session, the agent
# should speak a redirect message and emit a 'voice-to-chat-redirect'
# RTVI event to the Lighthouse frontend.
VOICE_RESTRICTED_TOOLS: FrozenSet[str] = frozenset(
[
"initiateAgenticLoop",
"generateAdsPerformanceReport",
"generateShopifyBreezeGoLiveReport",
"runSRAnalysis",
"generateReport",
]
)

VOICE_REDIRECT_TTS_MESSAGE = (
"This feature is only available in chat mode. "
"I'll take you there now so you can get the full analysis."
)
Loading