refactor: audio wake word detector and device capture components

[chcavignx]

No description provided.

[qodo-code-review]

PR Summary by Qodo

Refactor: Replace openWakeWord with direct ONNX Runtime wake word detection
✨ Enhancement 🧪 Tests 🕐 40+ Minutes

Description

• Replaces openwakeword with direct onnxruntime inference for wake-word detection.
• Extends wake-word config to manage separate melspec/embedding model paths and resolved download
 roots.
• Prevents stale audio processing by draining queues on ASR/WakeWordDetector restarts.
• Updates example agent to pause listening during TTS and add simple multi-turn behavior.
• Refreshes unit/coverage tests and dependencies (onnxruntime>=1.18, Ruff py311).

Diagram

graph TD
    MIC["🎤 Microphone"] --> CAP["_capture_loop"] --> Q["audio_queue"] --> DET["_detect_loop"] --> FEAT["ONNXAudioFeatures"]
    FEAT --> MS[("melspec.onnx")]
    FEAT --> EMB[("embedding.onnx")]
    DET --> WW[("wakeword.onnx")] --> CB["on_detected"] --> AG["VoiceAgent"]

    subgraph Legend
      direction LR
      _ext{{"External"}} ~~~ _thr["Thread/Loop"] ~~~ _onnx[("ONNX Model")]
    end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Keep `openwakeword` as the integration layer

• ➕ Less custom inference/feature code to maintain
• ➕ Upstream fixes and model compatibility handled by library
• ➖ Extra dependency/packaging overhead on constrained devices
• ➖ Less control over ORT session options, warmup behavior, and threading/queueing

2. Adopt a single fused ONNX KWS model (audio→score)

• ➕ Simpler runtime: one session, fewer model files
• ➕ Lower orchestration complexity and fewer moving parts
• ➖ Requires new/exported checkpoints; not a drop-in for existing openWakeWord model set
• ➖ Gives up reusable embedding pipeline for other tasks

Recommendation: Direct ONNX Runtime integration is a good fit for embedded/offline deployments: it reduces dependency surface and gives tight control over session options and streaming behavior. Ensure model file packaging/documentation clearly covers the three required ONNX artifacts (melspec/embedding/wakeword) and their expected naming/paths.

Files changed (13) +478 / -209

Enhancement (2) +68 / -26

voice_agent_offline.pyAvoid self-hearing during TTS and add basic multi-turn controls +37/-13

Avoid self-hearing during TTS and add basic multi-turn controls

• Stops the active listener (wake or ASR) before blocking TTS playback, then resumes in the appropriate mode. Adds exit-word logic to decide whether to continue the conversation in ASR or return to wake mode; enriches canned responses with time/date and additional intents.

examples/VAD/voice_agent_offline.py

config.pyExpand WakeConfig model-path handling and improve path resolution +31/-13

Expand WakeConfig model-path handling and improve path resolution

• Adds 'tmp_path' to PathConfig, resolves ASR/TTS model paths consistently with '.resolve()', and extends 'WakeConfig' with melspec/embedding/silero model names + computed paths. Removes 'vad_threshold' and simplifies wake download path computation.

src/utils/config.py

Bug fix (2) +9 / -2

asr.pyDrain audio queue on start to prevent stale/sentinel carryover +8/-1

Drain audio queue on start to prevent stale/sentinel carryover

• Clears any leftover queue items (including a prior 'None' sentinel) at the start of 'ASR.start()' before opening a new stream.

src/audio/asr.py

log_filters.pyFix max-level filter boundary condition +1/-1

Fix max-level filter boundary condition

• Changes log filter from '< max_level' to '<= max_level' so records exactly at the max level are retained.

src/utils/log_filters.py

Refactor (2) +295 / -113

audio_utils.pyTighten sounddevice import validation +5/-8

Tighten sounddevice import validation

• Simplifies 'get_audio_backend()' by explicitly raising a concise ImportError with exception chaining when 'sounddevice' is missing.

src/audio/audio_utils.py

wake_word.pyRewrite wake-word detector to use ONNX Runtime directly +290/-105

Rewrite wake-word detector to use ONNX Runtime directly

• Introduces 'ONNXAudioFeatures' for streaming melspectrogram and embedding computation and updates 'WakeWordDetector' to manage three ONNX sessions (melspec, embedding, wakeword). Adds queue draining on 'start()', larger queue capacity, detection warm-up (initial predictions forced to 0), and more robust stop/unload cleanup.

src/audio/wake_word.py

Tests (4) +94 / -56

test_audio_engine_units.pyUpdate wake-word unit tests for ORT sessions +38/-17

Update wake-word unit tests for ORT sessions

• Replaces openwakeword mocking with 'onnxruntime.InferenceSession' mocks and adds coverage for the new session-based load path and detection loop behavior.

tests/audio/test_audio_engine_units.py

test_vad_wakeword_units.pyRefactor wake-word tests to use ORT session mocks +22/-12

Refactor wake-word tests to use ORT session mocks

• Updates wake-word load and detect-loop tests to stub ORT sessions and the preprocessor feature extraction instead of openwakeword's Model/prediction buffer.

tests/audio/test_vad_wakeword_units.py

test_wake_word_coverage.pyRevise wake_word coverage tests for ONNX runtime integration +34/-26

Revise wake_word coverage tests for ONNX runtime integration

• Renames the import-error test to onnxruntime, updates assertions to session fields, and rewrites detection-loop tests to validate ORT '.run()' calls rather than openwakeword '.predict()'.

tests/audio/test_wake_word_coverage.py

test_config.pyAlign WakeConfig default assertions with new schema +0/-1

Align WakeConfig default assertions with new schema

• Removes the 'wake.vad_threshold' default assertion since the field no longer exists.

tests/utils/test_config.py

Other (3) +12 / -12

config.yamlUpdate wake-word defaults and audio chunk sizing +9/-9

Update wake-word defaults and audio chunk sizing

• Changes wake word to 'hello_jarvis', adds 'backend: wakeword', updates model name, and sets a concrete wakeword 'download_root'. Increases input/output chunk duration and size to 300ms/1024 and removes the wake 'vad_threshold' config.

config.yaml

log.jsonReduce stdout logging verbosity +1/-1

Reduce stdout logging verbosity

• Changes the stdout handler level from INFO to WARNING to cut runtime log noise.

log.json

pyproject.tomlAdd explicit onnxruntime dependency and bump Ruff target +2/-2

Add explicit onnxruntime dependency and bump Ruff target

• Adds 'onnxruntime>=1.18.0' to runtime dependencies and updates Ruff 'target-version' to Python 3.11.

pyproject.toml

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (0) 📜 Skill insights (0)

1. Hardcoded ONNX input names 🐞 Bug ≡ Correctness

Description

ONNXAudioFeatures calls ONNX Runtime with fixed input keys ('input' and 'input_1') instead of using
the model-declared input names, so inference can fail at runtime if the shipped ONNX models use
different tensor names. This would break feature extraction and prevent wake-word detection from
working.

Code

src/audio/wake_word.py[R123-140]

+        outputs = cast("list[NDArray[np.float32]]", self.melspec_session.run(None, {'input': arr}))  # pyright: ignore[reportUnknownMemberType]
+        spec = outputs[0]
+
+        if spec.ndim == 4:
+            spec = np.squeeze(spec, axis=(0, 1))
+
+        spec = melspec_transform(spec)
+        return spec
+
+    def _get_embeddings_from_melspec(self, melspec: NDArray[np.float32]) -> NDArray[np.float32]:
+        """Compute the Google speech embedding features from a mel-spectrogram."""
+        if melspec.ndim == 2:
+            melspec = np.expand_dims(melspec, axis=0)
+        if melspec.ndim == 3:
+            melspec = np.expand_dims(melspec, axis=-1)
+
+        res = cast("list[NDArray[np.float32]]", self.embedding_session.run(None, {'input_1': melspec}))[0]  # pyright: ignore[reportUnknownMemberType]
+        return np.reshape(res, (melspec.shape[0], 96))

Evidence

The preprocessing sessions are invoked with hardcoded input keys, while the wake-word session
explicitly queries the real input name, showing that dynamic input-name handling is
expected/available and missing for preprocessing models.

src/audio/wake_word.py[106-140]
src/audio/wake_word.py[511-515]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`ONNXAudioFeatures` hardcodes ONNX input tensor names when calling `InferenceSession.run()`, which is brittle: ONNX exported graphs frequently use different input names.

### Issue Context
`WakeWordDetector` already queries the wake-word model input name dynamically (`get_inputs()[0].name`), but the melspectrogram and embedding sessions do not.

### Fix Focus Areas
- src/audio/wake_word.py[106-140]

### Suggested fix
- In `ONNXAudioFeatures.__init__`, store the actual input names:
 - `self._melspec_input_name = melspec_session.get_inputs()[0].name`
 - `self._embedding_input_name = embedding_session.get_inputs()[0].name`
- Use those names in `.run()` calls instead of `'input'` / `'input_1'`.
- Optionally validate expected input shapes at load time and raise a clear error if incompatible.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Forced GC on stop() 🐞 Bug ➹ Performance

Description

WakeWordDetector.stop() unconditionally runs gc.collect(), adding synchronous work to a lifecycle
method that is invoked repeatedly during normal agent operation (mode switches / TTS gating). This
can introduce avoidable latency spikes during interactive use.

Code

src/audio/wake_word.py[R418-420]

        logger.info("Wake word detector stopped")
-
-    # ====================================================================
-    # Capture thread
-    # ====================================================================
+        _ = gc.collect()

Evidence

The GC call is executed at the end of every stop, and the voice agent calls stop() in several
normal-flow transitions (during TTS and when switching modes), making this a repeated non-essential
cost.

src/audio/wake_word.py[387-420]
examples/VAD/voice_agent_offline.py[112-195]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`WakeWordDetector.stop()` calls `gc.collect()` every time the detector stops.

### Issue Context
The offline voice agent stops/starts the wake detector frequently when switching between wake/ASR modes and during TTS playback.

### Fix Focus Areas
- src/audio/wake_word.py[387-420]
- examples/VAD/voice_agent_offline.py[112-195]

### Suggested fix
- Remove `gc.collect()` from `stop()`.
- If GC is needed to mitigate a specific teardown leak, move it to `unload()` (less frequent) or guard it behind a config/debug flag (e.g., `config.platform.debug_gc`), and add a comment explaining the motivating leak/issue.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Info logs hidden from stdout 🐞 Bug ◔ Observability

Description

The logging configuration sets the stdout handler level to WARNING, so INFO-level operational logs
are no longer emitted to stdout and may be missed in deployments that only collect stdout/stderr.
This reduces visibility into normal lifecycle events (model load, mode switches, etc.).

Code

log.json[R18-23]

        "stdout": {
            "class": "logging.StreamHandler",
-            "level": "INFO",
+            "level": "WARNING",
            "filters": ["max_warning"],
            "formatter": "simple",
            "stream": "ext://sys.stdout"

Evidence

The stdout handler is set to WARNING and the root logger uses stdout/stderr/file; therefore INFO
records will not appear on either stdout or stderr, only in the file handler output.

log.json[17-46]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`log.json` raises the stdout handler level to `WARNING`, removing `INFO` logs from console output.

### Issue Context
With root `level=DEBUG` and handlers `stderr` (ERROR) + `stdout` (WARNING) + file, INFO logs go only to the rotating file under `.tmp`, which may not be collected in containerized/hosted environments.

### Fix Focus Areas
- log.json[17-46]
- src/utils/log_filters.py[6-16]

### Suggested fix
- If you want to keep errors on stderr but still see INFO on stdout:
 - Set `stdout.level` back to `INFO`.
 - Keep the `MaxLevelFilter(max_level=WARNING)` so stdout gets INFO/WARNING but not ERROR+.
- Alternatively add a dedicated INFO handler to stdout and keep the existing WARNING split, depending on desired routing.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

1. Hardcoded onnx input names 🐞 Bug ≡ Correctness

ONNXAudioFeatures calls ONNX Runtime with fixed input keys ('input' and 'input_1') instead of using
the model-declared input names, so inference can fail at runtime if the shipped ONNX models use
different tensor names. This would break feature extraction and prevent wake-word detection from
working.

Agent Prompt

### Issue description
`ONNXAudioFeatures` hardcodes ONNX input tensor names when calling `InferenceSession.run()`, which is brittle: ONNX exported graphs frequently use different input names.

### Issue Context
`WakeWordDetector` already queries the wake-word model input name dynamically (`get_inputs()[0].name`), but the melspectrogram and embedding sessions do not.

### Fix Focus Areas
- src/audio/wake_word.py[106-140]

### Suggested fix
- In `ONNXAudioFeatures.__init__`, store the actual input names:
  - `self._melspec_input_name = melspec_session.get_inputs()[0].name`
  - `self._embedding_input_name = embedding_session.get_inputs()[0].name`
- Use those names in `.run()` calls instead of `'input'` / `'input_1'`.
- Optionally validate expected input shapes at load time and raise a clear error if incompatible.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools