(fix): audio gaps in tts#749
Open
narsimhaReddyJuspay wants to merge 1 commit into
Open
Conversation
WalkthroughThis PR mitigates audible TTS audio gaps in Breeze Buddy's Daily WebRTC mode by introducing a new audio pre-buffering processor, reducing transport chunk size to 20ms, and conditionally wiring these into the agent pipeline for daily mode only. ChangesDaily Mode Audio Gap Fixes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR targets audible TTS audio gaps in Breeze Buddy’s Daily (WebRTC) mode by improving the cadence and startup buffering of outbound audio, and documents the investigation/root causes.
Changes:
- Reduce Daily outbound write granularity by setting
audio_out_10ms_chunks=2(20ms chunks). - Add an
AudioPreBufferProcessorand insert it (Daily agent-mode only) between TTS andtransport.output()to pre-buffer initial audio frames per bot turn. - Add a root-cause analysis document describing confirmed/eliminated causes and a verification plan.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/TTS_AUDIO_GAP_ROOT_CAUSE_ANALYSIS.md | Adds RCA + proposed fixes and verification plan for Daily-mode TTS gaps. |
| app/ai/voice/agents/breeze_buddy/processors/audio_pre_buffer.py | Introduces a new processor that buffers the first N OutputAudioRawFrames per bot turn. |
| app/ai/voice/agents/breeze_buddy/processors/init.py | Exports AudioPreBufferProcessor from the processors package. |
| app/ai/voice/agents/breeze_buddy/agent/transport.py | Configures Daily transport to emit smaller audio chunks (audio_out_10ms_chunks=2). |
| app/ai/voice/agents/breeze_buddy/agent/pipeline.py | Adds is_daily_mode pipeline flag and conditionally inserts the pre-buffer processor. |
| app/ai/voice/agents/breeze_buddy/agent/init.py | Passes is_daily_mode into build_pipeline() when constructing the agent pipeline. |
|
|
||
| ## Diagnostic Tests Run | ||
|
|
||
| 13 tests across 6 test classes in `tests/test_audio_gap_diagnosis.py`. All pass. |
Comment on lines
+23
to
+31
| | Test Class | What It Tests | Result | | ||
| |------------|---------------|--------| | ||
| | TestEventLoopContention | Does BB's extra tasks starve audio output? | **ELIMINATED** — max gap 1.1ms even with 80 competing tasks | | ||
| | TestSOXRResampler | Does resampler state clearing cause discontinuities? | **ELIMINATED** — discontinuity ratio 0.19 (threshold 2.0) | | ||
| | TestDailySDKCallbackLatency | Does `call_soon` / `write_frames` callback delay? | **ELIMINATED** — P99 0.35ms under 30 competing tasks | | ||
| | TestBaselinePipelineJitter | Baseline frame delivery jitter | **ELIMINATED** — sub-ms jitter in ideal conditions | | ||
| | TestInterContextSilence | 500ms silence between TTS audio contexts | **CONFIRMED** (now fixed by pipecat 1.1.0) | | ||
| | TestAggregateSentencesImpact | aggregate_sentences creates more context boundaries | **REVISED** — not a cause post-rebase (sentence aggregation is beneficial) | | ||
|
|
Comment on lines
194
to
204
| async def build_pipeline( | ||
| transport: Any, | ||
| stt: Optional[Any], | ||
| llm: Optional[Any], | ||
| tts: Optional[Any], | ||
| vad_analyzer: Optional[SileroVADAnalyzer] = None, | ||
| configurations: Optional[ConfigurationModel] = None, | ||
| on_user_idle_timeout: Optional[Callable[[int], Any]] = None, | ||
| mode: Literal["agent", "stream"] = "agent", | ||
| is_daily_mode: bool = False, | ||
| ) -> tuple[ |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/ai/voice/agents/breeze_buddy/processors/audio_pre_buffer.py`:
- Around line 20-24: Remove the extra blank line between the import block and
the class definition so Black formatting passes: ensure there are exactly two
consecutive newlines between the import statements (including "from
pipecat.processors.frame_processor import FrameDirection, FrameProcessor") and
the "class AudioPreBufferProcessor(FrameProcessor):" declaration by deleting one
of the three blank lines.
In `@docs/TTS_AUDIO_GAP_ROOT_CAUSE_ANALYSIS.md`:
- Around line 158-165: Update the doc block to reflect the actual
implementation: replace references to the non-existent utils/audio_pacing.py
transport-wrapper with the implemented pipeline FrameProcessor in
app/ai/voice/agents/breeze_buddy/processors/audio_pre_buffer.py (mention the
FrameProcessor/AudioPreBuffer processor name and that it buffers 2–3 initial
audio chunks before forwarding to the Daily SDK rather than wrapping
write_audio_frame). Keep the rest of the text (why, risk, rollback) but change
the "File" and "Approach" lines to point to
app/ai/voice/agents/breeze_buddy/processors/audio_pre_buffer.py and describe the
implemented buffering pipeline stage.
- Around line 87-100: The fenced code block containing the audio flow should
include a language tag to silence markdownlint MD040; change the opening fence
from ``` to ```text for the block that starts with "TTS audio frames" (which
documents BaseOutputTransport.MediaSender.handle_audio_frame(), _next_frame(),
_audio_task_handler(), DailyOutputTransport.write_audio_frame(), and
CustomAudioSource.write_frames()) so the block is treated as plain text by the
linter.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 76bbbe12-a9dd-4fd0-9b7b-ca7d84911cff
📒 Files selected for processing (6)
app/ai/voice/agents/breeze_buddy/agent/__init__.pyapp/ai/voice/agents/breeze_buddy/agent/pipeline.pyapp/ai/voice/agents/breeze_buddy/agent/transport.pyapp/ai/voice/agents/breeze_buddy/processors/__init__.pyapp/ai/voice/agents/breeze_buddy/processors/audio_pre_buffer.pydocs/TTS_AUDIO_GAP_ROOT_CAUSE_ANALYSIS.md
Comment on lines
+20
to
+24
| from pipecat.processors.frame_processor import FrameDirection, FrameProcessor | ||
|
|
||
|
|
||
|
|
||
| class AudioPreBufferProcessor(FrameProcessor): |
There was a problem hiding this comment.
Fix the extra blank line to unblock CI (Black formatting failure).
There are three consecutive blank lines between the import block and the class definition (lines 21–23). Black requires exactly two, and the pipeline fails on this file.
🛠️ Proposed fix
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
-
class AudioPreBufferProcessor(FrameProcessor):📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| from pipecat.processors.frame_processor import FrameDirection, FrameProcessor | |
| class AudioPreBufferProcessor(FrameProcessor): | |
| from pipecat.processors.frame_processor import FrameDirection, FrameProcessor | |
| class AudioPreBufferProcessor(FrameProcessor): |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@app/ai/voice/agents/breeze_buddy/processors/audio_pre_buffer.py` around lines
20 - 24, Remove the extra blank line between the import block and the class
definition so Black formatting passes: ensure there are exactly two consecutive
newlines between the import statements (including "from
pipecat.processors.frame_processor import FrameDirection, FrameProcessor") and
the "class AudioPreBufferProcessor(FrameProcessor):" declaration by deleting one
of the three blank lines.
Comment on lines
+87
to
+100
| **Audio flow**: | ||
| ``` | ||
| TTS audio frames | ||
| → BaseOutputTransport.MediaSender.handle_audio_frame() | ||
| → resample + buffer in bytearray | ||
| → chunk into 40ms pieces (4 x 10ms default) | ||
| → put into _audio_queue | ||
| → _next_frame() / without_mixer | ||
| → yield immediately (NO pacing) | ||
| → _audio_task_handler() | ||
| → DailyOutputTransport.write_audio_frame() | ||
| → CustomAudioSource.write_frames() [daily-python native] | ||
| → await completion callback | ||
| ``` |
There was a problem hiding this comment.
Add a language tag to the fenced code block to suppress the markdownlint MD040 warning.
📝 Proposed fix
-```
+```text
TTS audio frames
→ BaseOutputTransport.MediaSender.handle_audio_frame()🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 88-88: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/TTS_AUDIO_GAP_ROOT_CAUSE_ANALYSIS.md` around lines 87 - 100, The fenced
code block containing the audio flow should include a language tag to silence
markdownlint MD040; change the opening fence from ``` to ```text for the block
that starts with "TTS audio frames" (which documents
BaseOutputTransport.MediaSender.handle_audio_frame(), _next_frame(),
_audio_task_handler(), DailyOutputTransport.write_audio_frame(), and
CustomAudioSource.write_frames()) so the block is treated as plain text by the
linter.
Comment on lines
+158
to
+165
| ### Fix 2: Add Pre-Buffering Before Audio Output (Priority: MEDIUM, Effort: MEDIUM) | ||
|
|
||
| **What**: Buffer 2-3 audio chunks (40-60ms) before starting to write to Daily SDK | ||
| **Why**: Gives the Daily SDK a head start so small delivery delays don't starve the WebRTC play cursor. The SDK has an internal buffer that absorbs jitter once it has some runway — the problem is the initial burst where the play cursor starts immediately with zero buffer. | ||
| **File**: New file `app/ai/voice/agents/breeze_buddy/utils/audio_pacing.py` | ||
| **Approach**: Wrap the Daily transport's `write_audio_frame` to buffer initial chunks before streaming | ||
| **Risk**: Adds 40-60ms initial TTS latency; acceptable for voice calls | ||
| **Rollback**: Remove the wrapper |
There was a problem hiding this comment.
Fix 2 description is stale — references a file and approach that don't match the actual implementation.
The doc describes an unimplemented transport-wrapper approach at app/ai/voice/agents/breeze_buddy/utils/audio_pacing.py, but the PR actually ships a pipeline FrameProcessor at app/ai/voice/agents/breeze_buddy/processors/audio_pre_buffer.py. Someone reading this document later will look for utils/audio_pacing.py and find nothing.
Consider updating lines 158–165 to reflect what was actually implemented.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/TTS_AUDIO_GAP_ROOT_CAUSE_ANALYSIS.md` around lines 158 - 165, Update the
doc block to reflect the actual implementation: replace references to the
non-existent utils/audio_pacing.py transport-wrapper with the implemented
pipeline FrameProcessor in
app/ai/voice/agents/breeze_buddy/processors/audio_pre_buffer.py (mention the
FrameProcessor/AudioPreBuffer processor name and that it buffers 2–3 initial
audio chunks before forwarding to the Daily SDK rather than wrapping
write_audio_frame). Keep the rest of the text (why, risk, rollback) but change
the "File" and "Approach" lines to point to
app/ai/voice/agents/breeze_buddy/processors/audio_pre_buffer.py and describe the
implemented buffering pipeline stage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Documentation