Two-way audio over WebRTC

[jekahamster]

Reading docs/design/remote-webrtc.md and docs/design/architecture.md (§5.2), the design already spec's an audio track (mic + speaker, two-way for telepresence) alongside the video track in the WebRTC session. The hardware side looks ready too — the TLV320AIC3104 codec and mic capture path (deploy/audio/aic3104-init.sh) are already configured and used by the sounds crate for the robot's own voice output.

But mediad/src/* currently only implements the video pipeline — I don't see an audio track wired into the WebRTC session yet.

Is two-way audio streaming (receiving mic audio from the robot, and sending audio to its speaker) on the near-term roadmap? If so, is there a rough timeframe or milestone it's tied to?

Also curious about mic/speaker quality: would it be good enough for a voice-assistant use case — e.g., streaming audio and video to a PC, processing it there, sending the response back to the robot, and asking the API to speak the reply? Or is the current setup tuned more for basic telepresence than for that kind of close-talk voice interaction?

[Vallhalen]

Second use case for the same gap, and one piece of the picture that this thread has not covered yet: the IPC side.

We are building a duck as the physical endpoint of an assistant that already runs locally — speech-to-text and a cloned-voice TTS on our own machine. The shape is exactly the one described above: mic out of the robot, inference on a box on the LAN, generated audio back into the robot's speaker. Everything except that last hop already works.

On the IPC side there is no path for arbitrary audio either, so this is not only a mediad question. Reading duck-ipc-proto/src/lib.rs at API_VERSION = 27:

• robot.sound (v10) takes SoundParams { tag: SoundTag, hold: Option<bool> }. SoundTag is the rendered bank — Alarm, Greet, Inquire, Peck, Chirp, Coo, Wheee — and hold only means anything for Wheee.
• No method in the protocol accepts a buffer or a stream of samples. robot.mouth moves the beak but carries no audio.
• mediad/src/lib.rs names the mic in its module doc ("camera, mic, WebRTC, and the remote gateway"), and stream.rs mentions that a session "carries audio" as a possibility, but as this issue says, nothing is wired.

So today a robot cannot say a sentence that was not rendered into its bank before it shipped, by any route.

The question that would help most is which surface is meant to carry it, because the two lead to different clients:

1. an inbound audio track in the WebRTC session, where a peer sends audio and the robot plays it, or
2. an IPC method that takes a buffer or a short stream, alongside robot.sound.

Option 1 is the natural home for telepresence and matches docs/design/remote-webrtc.md. Option 2 is what a LAN-side assistant would reach for, since it never wanted a browser session in the first place — it has a WAV and a socket. If the intended answer is 1, integrators can start building the peer side now and stop looking for a method that is not coming.

A related design question, if the answer is 2: what happens when the robot's own behaviour fires a Chirp while a spoken sentence is playing? For a bank of short sounds that is a non-issue, but a sentence is seconds long, and whichever way it resolves is something a client has to know rather than discover.

Happy to test any of this on hardware and report back — our duck is ordered, and in the meantime the whole loop runs in simulation.