Voice calls: model interrupts itself on speakerphone due to mic echo

[asavs]

Problem

During voice calls (Gemini Live API), the model frequently cuts itself off mid-sentence when the iPhone is on speakerphone. The cause is server-side VAD: Gemini hears its own audio leaking back through the mic and treats it as a user interruption, so it stops generating.

Google's own cookbook acknowledges this — Get_started_LiveAPI.py and Get_started_LiveAPI_NativeAudio.py say:

Important: Use headphones. This script uses the system default audio input and output, which often won't include echo cancellation. So to prevent the model from interrupting itself it is important that you use headphones.

We can't ship a product that requires headphones. We need to handle this in the app.

Possible fixes (ranked by practicality for iOS)

1. iOS hardware AEC via .voiceChat mode — set AVAudioSession mode to .voiceChat + .defaultToSpeaker, check isEchoCancelledInputAvailable at runtime. Native platform solution.
2. Client-side mic suppression — stop sending audio frames to the WebSocket while playback is active, resume ~200–500ms after playback stops. Simple half-duplex; prevents barge-in.
3. NO_INTERRUPTION activity handling — set activityHandling: NO_INTERRUPTION in the setup config. Model never gets interrupted but the user can't interrupt either.
4. Disable auto-VAD + manual control — automaticActivityDetection.disabled: true, then send ActivityStart/ActivityEnd manually. We already know when playback is happening, so we can suppress activity signals during echo.
5. Tune VAD sensitivity — startOfSpeechSensitivity: LOW. Reduces false positives but community reports this alone is insufficient for speakerphone.
6. Proactive Audio (preview) — new feature where the model distinguishes speech directed at the device vs. background. Could help; unconfirmed and in preview.

Recommended layered strategy

• Ensure voice call audio session uses .voiceChat mode (hardware AEC)
• Tune VAD startOfSpeechSensitivity to LOW as a baseline
• If still insufficient: disable auto-VAD and implement echo-aware manual turn detection — we already track when playback is active in VoiceCore
• Fall back to mic suppression during playback if AEC proves insufficient on speakerphone

References

• Gemini Native Audio Model Stops Midway While Speaking google-gemini/live-api-web-console#117 — model stopping mid-sentence
• https://discuss.ai.google.dev/t/disable-interruptions-for-audio-streaming-for-multimodal-live-api/61689
• https://discuss.ai.google.dev/t/how-do-i-prevent-the-live-api-from-discarding-audio-when-its-given-audio-while-it-speaks/73795
• https://community.openai.com/t/realtime-api-starts-to-answer-itself-with-mic-speaker-setup/977801