A hands-free local voice assistant. Say "hey gemma", talk, and it answers out
loud — wake word, speech in, speech out, and barge-in, all on-device. The wake
phrase is optional (--kwd-off), and the microphone does not have to be
attached to the machine doing the work (--dev-remote).
Everything runs locally: Gemma 4 for the reply, its audio encoder for understanding speech (no separate ASR anywhere — nothing here transcribes you), VibeVoice for the voice, LocalVQE for echo cancellation and noise suppression.
Voice only. Audio in, audio out. No text endpoint, no images, no page. A second way in would behave differently from the first, and every bug worth chasing here has been in the audio path.
Developed on Apple Silicon. The CUDA/aarch64 path in docker/ is built for
Jetson Orin and DGX Spark; LocalVQE's Linux build script
(tools/build_localvqe_linux.sh) is written but has not been run.
[idle — say "hey gemma" to start]
[↩ wake]
[listening — pause ~500 ms to send]
[gemma] The capital of France is Paris, and it is famous for its art and history.
[enc 60 tok | llm 17 tok | ttft 89 ms | ttfa 331 ms | tts 4.7 s | rtf 0.42]
Once it has replied, the conversation stays open for 5 seconds — keep talking and it answers again without the wake word. Talk over a reply and it stops.
Three dependencies, all submodules, ~2 GB together:
git submodule update --init --recursiveTwo of them are forks pinned to a branch, and upstream will not work for
either. The streaming mtmd_audio_stream_* API and the load_audio param
live only on that llama.cpp branch; the streaming
VibeVoice API and the crispasr-tts target live only on that CrispASR
branch.
Build the two out-of-tree pieces, fetch the weights (~5.6 GB), then build:
cd vendor/CrispASR && cmake -B build -S . && cmake --build build --target crispasr-tts && cd ../..
tools/build_localvqe.sh # build_localvqe_linux.sh off macOS
tools/fetch-models.sh # --list to see what it would download
cmake -S . -B build && cmake --build build --target gemma-live
./build/gemma-livegl-session is the model half on its own. -DGL_BUILD_APPS=OFF builds that
and its headers without the application, the offline driver or the tests,
which is what embedding this as a submodule wants.
Two halves, deliberately kept apart:
src/session.{h,cpp}— the model half. Audio in viapush_audio, tokens and synthesised audio out via callbacks. No audio device, no terminal. Owns the LLM, the mtmd audio encoder, MTP speculative decoding and the TTS stream.src/main.cpp— the app. miniaudio capture and playback, the AEC chain, three detectors, and the IDLE → LISTENING → REPLYING → AWAITING_FOLLOWUP state machine.
Plus three header-only pieces: vqe.h (LocalVQE wrapper), barge.h
(double-talk barge-in), transcript.h (rolling-ASR text helpers).
Three detectors, each owning one phase:
| when | what it watches | |
|---|---|---|
| moonshine | IDLE | rolling transcript, matched against keywords/wake.txt |
| firered-vad (eou) | LISTENING | silence after speech → send the turn |
barge.h |
REPLYING | AEC residual energy → user is talking over the reply |
| firered-vad (fup) | AWAITING_FOLLOWUP | voice start → another turn, no wake word |
Written down because each one cost real time to find.
Barge-in is not a VAD. Echo cancellation suppresses whatever correlates
with the reference, and during double-talk it takes the user's voice down with
the echo — too mangled for any speech classifier. Raw-mic energy fails the
other way, because echo is speech. barge.h instead learns the echo-residual
floor and fires at 3x it. Measured: echo alone leaves ~0.002 RMS, real
double-talk ~0.16 — a ~95x gap.
The AEC reference is tapped at the DAC, not where TTS produces audio. TTS synthesises faster than realtime, so at the producer it runs ahead of the speaker, and a barge-in throws that queue away — leaving the canceller aligned against sound nobody heard. Silence must be published too, or the render timeline stops advancing while the mic's keeps going.
MTP is a property of the model pair, not a knob. The draft head only
predicts a trunk it matches; a head from a different quantisation pipeline
dropped acceptance to 33-48% and made speculation a slowdown. Keep trunk and
head from the same repo. n=1 is the setting — llama.cpp's default of 3 is a
regression here. Acceptance is noisy (71-81% across runs on one pair); average
at least three before concluding anything.
An interrupted turn is rolled back entirely. A reply that stops mid-sentence would otherwise be closed as though the model chose to stop there, and a few of those in context teach it to truncate itself — measured, ten interrupted turns shrink the next clean reply to a fragment.
models/tokenizer.bin is a hard dependency with no reference in this repo.
moonshine resolves it as dirname(model)/tokenizer.bin. Without it, startup
fails complaining about the model.
LocalVQE's noise gate takes a quiet room to exactly zero, and that silence is what keeps the wake detector's own gate shut. Do not "optimise" by bypassing LocalVQE while idle — it re-opens that gate and costs far more than it saves.
Everything is a command-line flag, grouped by the same three-letter prefixes the
startup block prints — so the line describing a subsystem tells you which flags
tune it. --help lists all of them with their defaults.
./build/gemma-live --help
./build/gemma-live --mtp-off --vad-silence 350 --tts-chunk 2 llm --llm-model --llm-mmproj --llm-threads --llm-ctx --llm-predict --llm-temp
sys --sys-prompt
mtp --mtp-off --mtp-model --mtp-draft
tts --tts-model --tts-voice --tts-cfg --tts-steps --tts-anchor
--tts-chunk --tts-rms --tts-cpu --tts-prime --tts-filler
aec --aec-model --aec-threads --aec-gate
kwd --kwd-off --kwd-model --kwd-wake --kwd-step --kwd-window --kwd-gpu
--kwd-ratio --kwd-floor --kwd-nogate --kwd-duck --kwd-debug
vad --vad-model --vad-silence --vad-empty --vad-debug
nod --nod-off --nod-phrases --nod-after --nod-gap --nod-mono
--nod-per-turn --nod-len --nod-gain --nod-debug --nod-dump
dev --dev-remote
web --web-host --web-port --web-token --web-idle
(--dev-remote only)
fup --fup-timeout --fup-hops --fup-gate
brg --brg-ratio --brg-floor --brg-sustain --brg-debug
The defaults are measured rather than guessed; the reasoning behind the
non-obvious ones is in src/session.h and src/barge.h.
--kwd-debug, --vad-debug and --brg-debug print what each detector is
seeing, which is the fastest way to tell "correctly quiet" from "stuck".
--verbosity 2 restores the full ggml/llama diagnostics that are filtered out
by default.
Short "mm-hm" sounds while you are still talking, so a long turn does not happen into total silence. They never take the floor: no LLM call, nothing added to the KV cache, no state change — the turn carries on exactly as it would have.
The trigger reuses evidence the system already computes. The keyword worker
runs ends_mid_thought() on the rolling transcript to stretch the
end-of-turn threshold from 500 ms to 900 ms when you pause on a word an
English sentence cannot end on ("and", "because", "the"). That stretch is a
window where the system has already decided you are not finished, which is
exactly where a nod belongs. Two tiers fire:
confident the transcript ends mid-thought
monologue no transcript evidence, but you have been talking past --nod-mono
Only the confident tier is safe enough to fire on short exchanges, and on its own it is so rare you would not notice the feature; the monologue tier is what makes it audible.
A nod goes in early in a pause or not at all — before ~200 ms it clips a word you are still finishing, and after ~450 ms you are probably done and it is pure delay in front of the answer. Firing one also holds end-of-turn off until the clip finishes plus a beat, so the reply cannot land on top of the nod, and so you get the moment the nod just offered you.
Clips are pre-rendered at startup in the assistant's own voice (~1.5 s of
boot) and played from memory. The streaming path's ~335 ms time-to-first-
audio is far too slow: a backchannel that late has missed the moment it was
reacting to. They are trimmed, capped at --nod-len, and played at
--nod-gain — at full level a nod does not read as a listener signal, it
reads as an interruption.
Getting a nod wrong is cheap. Fired at a real turn end it produces "Mm-hm. The capital of France is Paris.", which is how people talk. The failure that matters is firing mid-word, which is what the pause window guards against.
--nod-debug logs every nod with its position and the transcript word that
justified it, and every near miss with the reason it was rejected.
Phrases can only really be judged by ear, so --nod-dump DIR renders the
current --nod-phrases to WAV and exits without opening the microphone.
What lands on disk is what reaches the speaker — trimmed, capped, and at
--nod-gain, so the files are deliberately quiet.
./build/gemma-live --nod-phrases "Mm-hm.,Sure.,Okay." --nod-dump /tmp/nodsThere is one program. By default it opens the sound card in front of it;
--dev-remote and it takes audio from a socket instead.
./build/gemma-live # the microphone on this machine
./build/gemma-live --dev-remote # ws://127.0.0.1:8927/api/liveNothing else changes. The wake word, the echo canceller, end-of-utterance detection, barge-in, backchannels and the follow-up window are the same code either way, because they are written against a device and a socket can be one — see The wire protocol for what goes over it.
That is the whole reason it is built this way. The previous arrangement had a second program for the socket, with a turn machine of its own, and the two front ends behaved differently in ways nobody could keep track of.
There is no page. A plain HTTP request gets 426 Upgrade Required and a
line saying where the socket is. What talks to this is your program: shipping
a client alongside the server meant the two drifted together into something
that only worked as a pair.
There is no text, and no images. Gemma is given audio and answers with audio. A text endpoint would be a second way in that behaved differently from the first, and every bug worth chasing here has been about the audio path.
prompts/chat.txt is the default, and it says voice-only on every line —
one sentence, no markdown, numbers in words. --sys-prompt overrides it.
prompts/echo.txt is a second, for diagnosis rather than use: it tells the
model to repeat what it heard verbatim and nothing else, which turns a turn
into a straight read-back of the audio path.
./build/gemma-live --sys-prompt prompts/echo.txtThere used to be two prompts here, and a rule for choosing between them: a long one for typed turns and a short one for spoken. With text gone there is one channel, so there is one prompt, and it is written for the only way out — everything it produces gets read aloud. That is why it bans markdown outright rather than asking for restraint: asterisks are not emphasis to a text-to-speech engine, they are silence or a spoken "asterisk".
Every reply is spoken with "Ahh," or "Well," in front of it, alternating.
--tts-filler sets the list, --tts-filler "" turns it off. It is spoken and
never shown: on_token fires first and unchanged, so the terminal and a
client's txt events carry the model's actual reply.
It is there for two measured reasons.
VibeVoice mangles short utterances. Eight seeds of "Welcome!" left six with
broadband noise where the word should be, one of them 41 dB down. The same
eight seeds of "Ahh, welcome!" left the word untouched in all eight, and
"Well, welcome!" likewise. The damage does not disappear — it lands on the
filler instead, which takes 16 to 92 dB there — and that is the point: a rough
"Ahh" reads as someone drawing breath, a rough "Welcome" reads as a broken
machine. This replaced an earlier rewrite that spoke ! as . on short
sentences, which worked but cost the exclamatory prosody.
And it covers the chunk-boundary gap. That gap is fixed by latent frames, not words, so it still falls about 400 ms in — but with a filler in front, 400 ms in is the pause between the filler and the first word, where hesitating is what a person does anyway. Measured on an M4 Max:
stutters/reply ttfa
no filler, prime 0 1.20 430 ms
no filler, prime 300 0.12 691 ms
filler, prime 300 0.36 563 ms
filler, prime 0 0.62 276 ms
The stutter count rises as the buffer comes down, but what those stutters
sound like has changed. Judged by ear at prime 0 they are not audible at all,
which is why that is the default: 276 ms to the first sound, better than the
430 ms this ever had before the stutter was addressed, and with the artefacts
gone too. --tts-prime is still there for a slower machine, where the deficit
is larger and a filled pause may not cover it.
Two rewrites on the way to the synthesiser, in tts_text.h. The txt events
a client receives are the model's actual reply; this is only about the audio.
Whitespace collapses, because VibeVoice tokenises every space and newline as a small pause and a blank line becomes a multi-second breath.
And ! is spoken as . at the end of a short sentence, which is a
measurement rather than a preference. VibeVoice returns broadband noise where
the word should be on short exclamatory utterances, and the damage is
confined to one- and two-word ones. Eight seeds per phrase, everything else
held fixed:
"Welcome!" 1 word 6/8 bad, worst 41 dB down
"Hey there!" 2 words 1/8
"That sounds great!" 3 words 0/8
"That sounds really great to me!" 6 words 0/8
"I think that sounds like a
really wonderful idea to try!" 11 words 0/8
Same phrases with a full stop: Welcome. 1/8, Awesome! 3/8 against
Awesome. 0/8. So anything of three words or more keeps its mark and its
exclamatory prosody — at that length the model delivers it. The count is per
sentence, not per reply, so Welcome! That sounds really great to me! gets
the first defused and the second left alone.
Neither knob you would reach for first touches this. --tts-chunk gives
byte-identical output at 3, 6 and 12 latent frames, so it is not the
streaming schedule. --tts-steps goes from 7 bad clips of 24 at 3 steps to 6
at 6 — it changes which seeds fail, not how many.
A post-filter does not fix it either, which is worth recording since it was tried: DeepFilterNet3 does not repair a bad generation, it mutes it — 41 dB off the whole word. A dropped "Welcome" instead of a noisy one, for 6.3% of realtime.
/api/live is the only endpoint. Everything else gets 426 Upgrade Required.
Audio in is interleaved stereo pcm16 at the rate ready states, as binary
frames — channel 0 the microphone, channel 1 whatever your speaker is
playing. The second channel is the echo canceller's far-end reference, and
it must come off the same clock as the microphone; see
Where in-pre comes from. Send it continuously,
silence included: a gap is not "no echo", it is a hole with nothing to
subtract.
Send your microphone unprocessed — no echo cancellation, no noise suppression, no gain control. All three are done here, and a client that does them first leaves two cancellers in series.
Audio out is mono pcm16 at out_rate, binary, and only while there is
something to say. Silence is not sent.
Events out, JSON on the text opcode:
{"t":"ready","in_rate":24000,"out_rate":24000,"channels":2}
{"t":"eou"} you stopped talking; a reply is coming
{"t":"start"} reply begins
{"t":"txt","s":"..."} what is being said, as it is said
{"t":"end","cancelled":false,"out_tok":8,"ms":300}
One verb in:
{"t":"cancel"} stop the reply
That is the whole protocol, and the short list is the point. There is no
played and no barge. Both existed because the old server had no canceller
and needed the client to referee — to say when its speaker had gone quiet, and
whether a sound was the user or her own echo. The canceller is here now and
the barge-in detector reads its residual, so neither question is the client's
to answer.
There is no text verb and no {"t":"end"} to force a turn. This is a voice
program: the microphone opens turns and silence closes them.
Connecting is the wake word. A microphone in a room should not always be listening, which is what the wake phrase is for. A client that opened a socket has already answered that question, so the session starts on connect and stays open — gating on a phrase as well would mean that once the follow-up window closed, a client still connected and still talking was heard by nobody until it disconnected and came back.
--kwd-off removes the phrase entirely, on either front end: it listens from
the start. That also stops moonshine being loaded and run, which is the
largest standing cost in the app — a 1.5 s window encoded several times a
second, for as long as the process is up.
Both off by default, because on loopback neither earns its keep. Together they are what makes a tunnel safe to hand out.
./build/gemma-live --dev-remote --web-token s3cret --web-idle 300--web-token requires ?t= on every route, checked in the handshake:
an unauthorised peer gets 401 and is never upgraded. The page passes on
whatever token it was opened with, so https://host/?t=s3cret is the whole
setup — a browser cannot put a header on a WebSocket, which is why the query
string carries it for the socket and the fetches alike. It is a shared
secret, not per-user auth; Cloudflare Access does the real thing for free if
you want it.
--web-idle reclaims a session after that many seconds without speech —
not without traffic, because the microphone streams continuously and a
connection is never quiet while it is open. What counts is any audio sample
above the silence floor, deliberately not the VAD's verdict: this is a reaper
for an abandoned session, not a judgement about what is speech, and it must
not depend on the VAD being right. It waits for any reply in flight to finish
— start and end mark that span — so a long answer is never cut off. This
matters more than the token for a public link: the server takes one session
at a time, so without it a single forgotten tab locks everyone else out
indefinitely.
Bound anywhere but loopback, gemma-live warns at startup about each of these that is missing.
docker/ runs the same image on a Jetson Orin and a DGX Spark. The base
compose is the LAN setup: Caddy terminating TLS on 8443 with an mkcert
certificate, which browsers trust only where your local CA is installed. For
a public deployment there is an overlay that swaps in ACME:
export GEMMA_LIVE_DOMAIN=voice.example.com
export GEMMA_LIVE_ACME_EMAIL=you@example.com
export GEMMA_LIVE_ARGS="--web-token $(openssl rand -hex 16) --web-idle 900"
docker compose -f compose.yaml -f compose.public.yaml up -dPort 80 has to be reachable even though nothing is served on it: that is where the HTTP-01 challenge is answered. Certificates live in a named volume, because asking Let's Encrypt for a fresh one on every restart reaches the rate limit — five per domain per week — and then the site is simply down until it expires.
gemma-live itself is not published on the host. Caddy reaches it by
service name over the compose network; publishing 8927 as well would put a
plaintext, tokenless copy of the same server on every interface, beside the
front door rather than behind it. To look at it directly while debugging,
publish it to loopback only:
A line per turn at the default verbosity, the same whether the microphone is local or a socket:
[enc 41 tok | llm 7 tok @ 65.8 tok/s | ttft 140 ms | tts 2.51 s | ttfa 379 ms | rtf 0.44 | mtp 4/4 acc 100%]
ttft is end of input to first sampled token, ttfa to the first audio
reaching the speaker, and rtf is synthesis wall time over audio produced —
above 1 it cannot keep up. UNDERRUN appears when playback ran dry, which is
audible as a gap.
Note what these do not include: the end-of-turn wait. ttft is measured
from the moment the turn was declared over, and deciding that takes ~1050 ms
of its own — see the end-of-turn wait.
A turn that feels slow with good numbers here is almost always that, not the
pipe.
The turn opens on speech onset, not at the end, so the
audio encoder runs while you are still talking. Before that it did not, and
the whole encode sat in front of the reply — 30 ms per second of speech,
plus a tail finalise that pushed ttft up with it:
speech ttfa buffered ttfa streaming
1.6 s 395 ms 288 ms
4.9 s 506 ms 270 ms
9.8 s 777 ms 363 ms
The cost is that the turn lock is held from speech onset to the end of the
reply, which shows up as wait in the stats line.
One session at a time. VoiceSession owns a single llama context and is
not thread-safe, so a second conversation would need a second copy of every
model. A client arriving while another is connected gets close code 1013
rather than an unbounded wait. For more than one person, run more than one
process — 128 GB holds about twenty.
No transcript of the user, anywhere. Gemma consumes audio as tokens
through the mtmd encoder and never produces one. What it says back is streamed
as txt — those are the sampled tokens — but what you said exists only as
audio. --cap-dir writes the microphone so you can listen for yourself.
No text, no images, no page. Audio in, audio out. A text endpoint would be a second way in that behaved differently from the first, and every bug worth chasing here has been in the audio path.
Settings are fixed at startup. The system prompt, voice, temperature and
reply length belong to the loaded VoiceSession and come from the flags.
There is no verb to change them mid-session: they would have to be applied to
a context already conditioned on the conversation so far.
Cancel drops the whole turn. {"t":"cancel"} rolls the KV cache back past
the entire assistant turn rather than truncating it where playback stopped.
Coarser, and deliberately so: leaving a half-finished reply in context teaches
the model to truncate itself on later turns.
pcm16 only. No G.711, no Opus.
Left alone, VibeVoice puts roughly 800 ms of digital silence in front of any reply longer than about two words. It is trimmed by default; this is here because the number is surprising and someone will want to check it.
words lead
1 140ms 'Yes.'
2 151ms 'Good morning.'
5 807ms 'I am doing well today.'
8 804ms 'I am doing well, thank you for asking.'
17 806ms 'I am doing well, thank you for asking, and I hope…'
A step, not a slope — and invariant to everything on this side of it:
config lead reply
default (steps 3, cfg 1.7) 804ms 2.70s
--tts-chunk 1 804ms 2.70s
--tts-chunk 6 804ms 2.70s
steps 5, cfg 1.5 806ms 2.97s
--tts-cfg 1.3 807ms 3.21s
The reply length moves with the sampling settings; the lead-in does not. So it is the model's own output, not chunk scheduling, and not something the tuning above introduced. It is also literally zeros — measured peak 0 to 1/32768, against speech at around -16 dBFS — which is what makes dropping it safe rather than a judgement about what counts as quiet.
Worth removing because of where it sits: 800 ms of dead air the user waits through after the answer has already been generated. That is more than speculative decoding or a smaller first chunk can give back. Trimmed in the one place all TTS audio passes through, so both binaries get it, and only until the first real sample of a turn — a pause later in a reply is prosody and is left alone.
Measured end to end: 804 ms of lead becomes 33 ms, and a 2.70 s reply becomes 1.93 s of the same words.
The other half of perceived latency, and the one people misattribute to the
model. --vad-silence is not the whole wait. The detector asks "is there
speech anywhere in the last 800 ms", so after you stop talking the window
keeps answering yes until your speech has slid out of it — only then does the
silence timer start:
(800 ms window - 250 ms MIN_SPEECH_SEC) + --vad-silence + up to 100 ms hop
which is 1050-1150 ms at the default 500, and measures 1002-1165 ms. Halving
--vad-silence therefore buys 250 ms, not half the wait. tools/cap-report.py
reports both numbers per turn from a capture directory, which is the way to
tell which of the two you are actually hearing.
VibeVoice doubles its chunk sizes: the second chunk is twice the first, so the time to GENERATE it grows exactly as fast as the time to PLAY the first. Above a realtime factor of about 0.5 the second chunk is therefore always late, and it is late at the same place every time — one boundary in, a word or two after she starts. That is the stutter.
--tts-chunk does not fix it, which is the interesting part: a bigger first
chunk buys more playing time and an equally bigger second chunk to generate.
Measured across 36 replies each, stutters per reply: chunk 3 gave 1.22, chunk 6
gave 0.44, chunk 9 gave 1.27. It moves the number around without removing the
cause.
The cause is in the fork, and is now half fixed there: the emit chunk grows by three halves rather than doubling, so a chunk's generation fits inside the previous chunk's playback. That does not make synthesis faster — more emits means more of the fixed per-emit re-decode, and rtf rises from 0.60 to 0.72 — but it shrinks the deficit the buffer below has to cover, which is where the latency comes back.
--tts-prime banks audio before the speaker starts, which covers the boundary
directly. Same measurement, at the default chunk size:
--tts-prime 0 200 300 500
doubling stutters 1.20 0.83 - 0.17
ttfa (ms) 430 527 - 740
three halves stutters 1.13 0.29 0.12 -
ttfa (ms) 374 602 691 -
It costs roughly its own length in time-to-first-audio, and it is the same milliseconds the gap would have cost — moved to before the first word instead of into the middle of the second. 300 ms is the default. Set it to 0 to hear what this is about.
Two things that do NOT help, both measured rather than assumed. --tts-cfg 1.0
is slower, not faster: rtf goes 0.65 to 0.96 and ttfa 396 to 578, so the guided
path is better optimised than the unguided one. And --tts-steps is already at
its floor — the runtime setter clamps to 4, so asking for 3 gets 4.
Below a realtime factor of about 0.5 none of this is needed: the gap does not open and the buffer can be 0. That is the Spark, at 0.24. It is not this laptop, at 0.6 to 0.7, and no amount of tuning gets there — it would take a faster synthesis path.
One trap if you change it: the pre-roll has to be released when synthesis finishes, not only when the buffer fills, or a reply shorter than the prime waits for audio that is never coming and never plays at all.
One suppressor, wherever the microphone is.
LocalVQE v1.3 — echo cancellation, noise suppression and
dereverb in one pass, and the strongest model in that family; everything below
it trades noise handling for CPU. Its only tuning knob is --aec-gate, a hard
gate on output hops (default -45 dBFS) that also mutes genuinely quiet speech,
so tighten it only if distant talking is not something you need.
What LocalVQE does not do is rumble. It replaced WebRTC's AudioProcessing,
which ran a high-pass; nothing since did, so HVAC, footfall, desk knocks and
plosives passed straight through to the two things that read this audio — the
VAD's energy detector, which decides when your turns begin and end, and the
model, which takes audio as tokens and cannot tell you it was distracted.
--aec-hpf puts that back: fourth-order Butterworth, 80 Hz by default.
20 Hz -48 dB 150 Hz -0.02 dB
40 Hz -24 dB 300 Hz -0.00 dB
50 Hz -16 dB 1 kHz -0.15 dB
It runs on the AEC's output, deliberately. Filtering the microphone ahead of
the canceller would leave the far-end reference unfiltered, and the canceller
would then be subtracting a low band from a signal that no longer has one.
--aec-hpf 0 turns it off.
--tts-cpu depends on the machine, and the two NVIDIA targets disagree.
VibeVoice runs on the GPU by default. On Jetson/Tegra that is wrong: the
per-frame graph is rebuilt as n_past grows, and the churn measured ~8-10x
SLOWER than the CPU backend, which is why the flag exists.
A DGX Spark (GB10) does the opposite. Measured over four turns each:
rtf ttfa
GPU 0.18-0.24 151-289 ms
--tts-cpu 0.83-1.06 419-931 ms
The GPU is ~4x faster, and CPU synthesis touches rtf 1.06 — above 1, which
means it is slower than the audio it is producing and playback has to wait.
So: leave it off on a Spark, turn it on for an Orin, and measure anywhere
else rather than assuming either.
A remote client must not process its own audio. With --dev-remote the
canceller is here, and it can only work on a microphone nothing has already
touched: a client that leaves the browser's echoCancellation on puts two
cancellers in series, each subtracting an echo the other partly removed.
autoGainControl matters as much — it rides the level in response to the
echo, so the reference stops matching what the microphone heard.
The client sends its own playback back as capture channel 1. That is the far
end, and the canceller needs it: see
Where in-pre comes from.
Neither of these should be trusted without measuring. Capture a turn in the
room that is actually giving you trouble and compare in-pre with in-post;
Whether Gemma still answers sensibly is the check that suppression has not
started removing speech along with the noise — there is no transcript to
compare against, so the reply is the signal. Stronger is not automatically better
here — Gemma consumes audio as tokens rather than a transcript, so artefacts
can cost more than the noise they replaced.
Voice bugs are the worst kind to chase live. "It answered its own echo",
"it cut me off mid-sentence", "it heard something nobody said" are all
claims about audio that no longer exists by the time anyone looks at it.
--cap-dir keeps it. Both binaries take it:
./build/gemma-live --cap-dir capturesEvery run gets its own folder, and every turn up to four files in it:
captures/20260903-120512/
turn-0001-in-pre.wav the microphone before echo cancellation
turn-0001-in-post.wav the audio the model was actually given
turn-0001-out.wav what was spoken back
transcript.jsonl what was said, on both sides
transcript.jsonl is one line per turn:
{"turn":1,"said":"The weather on Mars is generally cold and dusty…",
"in_s":12.81,"out_s":12.03,"listen_ms":0,"reply_ms":2280,
"interrupted":false}said is the reply's own tokens. There is no field for what the user said,
because nothing here produces one — the microphone audio beside it is the
record, and it is the more honest one: a transcript would be a second model's
opinion of the same WAV, and when the two disagreed you would open the WAV
anyway.
Turns that were interrupted, or that failed, are kept and flagged — those are usually the ones being investigated. Input is recorded only from speech onset: the microphone runs continuously, and ungated every turn would open with the tail of the previous reply.
The two binaries cancel echo in different places, so the pair means slightly different things.
gemma-live runs LocalVQE in-process. in-pre and in-post are the
same sample range — LocalVQE is 1:1 and is fed whole hops, so the two
line up in an editor with no offset to hunt for, and the difference between
them is exactly the AEC's work. If the two ever cannot be matched, the pre
side is dropped rather than written misaligned: a pair that does not line up
reads as an AEC bug that is not there.
--dev-remote never sees a local speaker, so the tap that provides the
reference on hardware is the wrong one: it is what was sent, and the
client's speaker plays it some unknown time later, after the network and a
jitter buffer.
So the client sends its own playback back as capture channel 1. Both
halves come off one AudioContext through a channel merger, which makes them
aligned by construction rather than by estimate — one clock, one buffer, no
drift over a long session. The pair reaching the canceller is then the same
shape as on hardware, and in-pre/in-post is a true 1:1 pair either way.
A reference reconstructed server-side from what was sent would have been adrift by the network and the client's buffering, and would drift further the longer a session ran. Measured with the aligned pair: 11.7 dB of a synthetic echo removed, speech untouched at 0.2 dB. With the reference resampled from the wrong clock it was 0.8 dB — which is what that mistake looks like, and it is silent.
A client that ignores the offer keeps sending bare pcm16 and gets
in-post only. The tag is never inferred: guessing wrong takes a byte off
the front of every frame, and pcm16 shifted by one byte is noise that still
decodes. The flag flips on the reader thread, in wire order, for the same
reason.
None of this runs on the audio path. The capture appends to a buffer and nothing else; WAV encoding and transcription happen on a writer thread once the turn is over, because a debug mode that changes the timing of the thing it is measuring describes a system that only exists while debugging. Off by default, and off costs nothing.
Eight unit tests run with the build and need no models:
cd build && ctestbarge double-talk barge-in on the AEC residual
nod when a backchannel is and is not appropriate
transcript wake-phrase matching and transcript assembly
hpf the microphone high-pass: rumble down, speech band flat
ttstrim dropping VibeVoice's lead-in without clipping the onset
capture the recorder: arming, alignment, escaping, bounds
remote_audio the queues behind the remote device, and their backpressure
remote_backend those queues driven through real miniaudio device machinery
The socket is not reachable from those: they never open one, so nothing notices when an event stops being emitted, an audio frame changes shape, or a turn stops ending. So:
./build/gemma-live --dev-remote &
tools/protocol-test.py # or --port / --tokenIt synthesises its own speech, so there are no fixtures — but it needs the models loaded, which is why it is not a ctest target. It asserts the greeting and its rates, the event sequence, that reply audio arrives as binary frames, that cancel ends a turn and says so, that plain HTTP gets 426, that an unknown endpoint and a second client are both turned away, and that the retired verbs stay retired.
A plain tone does not work as test audio, which is worth knowing before writing another one: LocalVQE does noise suppression as well as echo cancellation, and a steady sine is exactly what it removes — it reaches the VAD as silence and no turn ever opens. The synthesised voice has a glottal buzz, three formants and a syllable-rate envelope for that reason.
cmake --build build --target gemma-live gl-offline
cd build && ctestgl-offline drives the model half from WAV files with no microphone — the only
way to measure any of this reproducibly:
./build/gl-offline turn1.wav turn2.wav turn3.wavPass one WAV per turn, not the same one repeated: identical audio makes MTP look
far better than it is. Run several turns, because a speculative-decoding cache
bug shows up on turn 2, not turn 1. GL_ABORT_MS=250 reproduces repeated
barge-in.
All three binaries share gl-session, so build them together — a stale
gl-offline silently measures the old code.
Gemma 4 E4B QAT (UD-Q4_K_XL) on an M4 Max, versus Google's own LiteRT-LM runtime on the same model and machine:
| gemma-live | LiteRT-LM | |
|---|---|---|
| decode, speculative | 104 tok/s | 93 tok/s |
| decode, plain | 88 tok/s | 72 tok/s |
| prefill | 1222 tok/s | 1437 tok/s |
| model load | 0.7 s | ~3 s |
Per turn: ~90 ms to first token, ~310 ms to first audio, ~5.6 GB resident. Perceived latency is dominated by the 500 ms end-of-utterance wait and TTS, not by the LLM.