Wake word training pipeline (M0-M5): commercially-licensed, reproducible - #22
Open
keithtmccartney wants to merge 11 commits into
Open
keithtmccartney wants to merge 11 commits into
keithtmccartney wants to merge 11 commits into
Conversation
Establish an isolated, commercially-licensed training project for Qube and Hey Qube wake words with fail-closed license verification, config-driven scripts, and pinned training environment. Co-authored-by: Cursor <cursoragent@cursor.com>
Generate (N, 16, 96) openWakeWord embedding features from commercially-licensed audio (LibriSpeech/MUSAN), replacing the notebook's non-commercial ACAV100M features. Adds streaming clip stacking, a memory-safe shard-then-merge memmap writer, provenance-stamped outputs, and tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Wire download_datasets.py end-to-end so the M2 feature precompute has real LibriSpeech / MUSAN inputs to consume: - Declarative FOSS dataset registry (lib/datasets.py): LibriSpeech, MUSAN, MIT-RIR, FMA-commercial - all CC-BY-4.0 / commercial-allowlisted - with named profiles (m2-min, m2-full, all). Adding a source is a data change. - Downloader supports resumable HTTP (OpenSLR) and HF snapshot sources, path-traversal-safe tar/zip extraction, and --list/--dry-run/--only. - Dataset-level provenance manifests (datasets/licenses/<key>.license.json) plus a trust-on-first-use reproducibility lock (manifest.lock.json) that records + verifies archive sha256s. - Run ends with the fail-closed commercial license gate. - Tests cover registry integrity, selection resolution, extraction + traversal rejection, manifest gating, and lock behavior (full suite: 35 passing). - Docs/roadmap updated; M1 marked done. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Lands the M3 milestone for Feature dagaza#42: the synthetic training-data and phonetic pilot-planning layer, directly implementing Dan's two biggest data-quality levers (hard-negative mining + multi-speaker positive diversity). New library modules (pure/testable, heavy deps lazily imported): - lib/phonetics.py curated /kjuːb/ confusable library (cube, cute, tube, queue, youtube, cuban, cubed, "ice cube", ...) merged with each config's adversarial_phrases; normalization + dedupe; never emits the wake phrase. - lib/tts.py deterministic multi-speaker Piper synthesis plan (even speaker spread + cycled rate/noise variation) + injectable synth backend; real Piper call and voice download behind lazy imports. - lib/experiments.py pilot-variant expansion + operating-point selection rule (recall subject to FP/hr, tie-break on noisy-room robustness). New pipeline stages: - generate_positives.py real multi-speaker positive synthesis + provenance manifest (Piper MIT + LibriTTS-R CC-BY-4.0) that keeps the gate green. - hard_negative_mining.py synthesizes phonetically-similar hard negatives distributed across the confusable set. - run_pilot_sweep.py plan/data/rank stages across configs/experiments.yaml; data stage runnable today, rank stage consumes M5 eval metrics. Docs/config: README, WAKEWORD_PIPELINE.md, docs/roadmap.md updated (M3 done, augment/train/export -> M4, evaluate -> M5); experiments.yaml driver comment points at run_pilot_sweep.py; augment.py stub relabelled M4. Tests: +38 covering confusable generation, synthesis planning, variant expansion + winner selection, and the positives/hard-negative/sweep orchestrators via injected synthesis. Full suite 73 passing, lint clean. Co-authored-by: Cursor <cursoragent@cursor.com>
Conversation context (used only for the agent's understanding) leaked a reviewer's name into docstrings, config comments, a Dockerfile comment, and in-repo docs. Reword to attribute the substance rather than a person. Also add .cursor/rules/code-commentary.mdc (alwaysApply) so committed comments, docstrings, config comments, and docs never carry names inferred from chat. No functional change; full wakeword/ suite still 73 passing. Co-authored-by: Cursor <cursoragent@cursor.com>
Lands the M4 milestone for Feature dagaza#42: the pipeline now trains a Qube-loadable wake word model end-to-end. openWakeWord 0.4.0 ships no trainer (its auto-train lived in the repo/notebook at a pinned commit), so M4 implements the classifier directly against the exact runtime contract of the shipped models (input (batch,16,96); Flatten -> FC stack -> Linear(1) -> Sigmoid; output (batch,1)), verified by inspecting a pretrained *.onnx. New library modules (pure/testable; torch/tf lazily imported): - lib/augment.py pure-NumPy RIR convolution (peak-aligned, loudness-preserving) + SNR-targeted noise/music mixing + deterministic per-clip augmentation plan. - lib/model.py classifier architecture matching the shipped ONNX layout. - lib/training.py training-spec resolution, class-balanced batch sampling, and false-penalty loss weighting (pure) + the lazy weighted-BCE/Adam/early-stop loop. - lib/model_card.py auditable provenance record (config hash, dataset versions + checksums, params, seed, oww commit, hardware, duration, metrics, license tier). - lib/export.py torch->ONNX export with a dynamic-batch (?,16,96)->(?,1) contract and a real onnxruntime verification pass; optional non-fatal TFLite via onnx2tf. Pipeline stages (were stubs, now implemented): - augment.py far-field reverb + noise/music over the M3 positives -> manifest. - train.py gate -> assemble positive/negative/validation features -> train -> models/<id>/<version>/checkpoint.pt + model_card.json. - export.py checkpoint -> <id>.onnx (verified) + optional .tflite + provenance. Docs/env: README, WAKEWORD_PIPELINE.md, docs/roadmap.md updated (M4 done, only M5 evaluate.py remains a stub); requirements-training.txt notes optional onnx2tf. Tests: +27 covering augmentation math (SNR mix + RIR), model-card provenance, training-spec + sampler + false-penalty weighting, and augment/train/export orchestration with injected I/O. Full suite 100 passing, lint clean. Co-authored-by: Cursor <cursoragent@cursor.com>
Adds the final pipeline stage — operating-point evaluation against a held-out, real-voice corpus — completing the M0-M5 training pipeline. - lib/metrics.py: pure operating-point math — recall/FRR, false-accepts per hour, precision, adversarial false-accept rate, DET/ROC points, latency percentiles, quiet-vs-noisy robustness, and the ship rule (max recall s.t. FP/hr <= target, tie-break to higher threshold). - lib/corpus.py: parse evaluation/corpus.json into typed positives / adversarial / long-form negatives; paths resolved relative to index. - evaluate.py: stream <id>.onnx over the corpus (lazy openWakeWord), sweep thresholds 0.3-0.7, select recommended threshold, write results/<id>/<version>/eval.json + eval.md with a pass/fail verdict. --emit-sweep-metrics writes a rank-stage row so run_pilot_sweep.py --stage rank ranks variants on real numbers (closes the M3 loop). - configs: add evaluation ship-criteria block (FP/hr cap + recall floor). - tests: 18 new (metrics math, corpus parsing, evaluate orchestration with injected scorer) — 118 total, lint clean, heavy deps lazy. - docs: README, WAKEWORD_PIPELINE, roadmap, evaluation.md mark M5 done. All pipeline code is now implemented and tested; remaining ship gates are recording the corpus and Test Lab sign-off (data + human, not code). Co-authored-by: Cursor <cursoragent@cursor.com>
Prove the pipeline wiring (not model quality) on tiny synthetic data so a developer can validate train -> export -> evaluate in seconds before committing to multi-GB downloads. - test_smoke_contract_onnx_evaluate: builds a contract-shaped (batch,16,96)->(batch,1) ONNX directly and drives export.verify_onnx + evaluate.evaluate_corpus through a real onnxruntime session. Runs anywhere onnx/onnxruntime are installed (incl. CI), guarding the model -> scorer -> metrics path the unit tests only cover with injected scorers. - test_smoke_train_export_evaluate: the real heavy path — torch training loop -> torch.onnx export -> contract verification -> onnx-backed evaluate. torch-gated (skips where the pinned training env isn't installed), so it runs green in the training env as a fast pre-flight. Assertions check structure/finiteness, not accuracy, so they aren't flaky. README quick start gains a 1b pre-flight step. Suite: 119 passed, 1 skipped (the torch-gated smoke test), lint clean. Co-authored-by: Cursor <cursoragent@cursor.com>
Adds a dedicated GitHub Actions workflow for the wakeword pipeline, kept independent of the main app CI and triggered only when wakeword/** (or the workflow itself) changes. - .github/workflows/wakeword-ci.yml: on PRs + pushes to main/master/dev, runs the fail-closed license gate (verify_licenses.py --require-commercial --allow-empty) and the full pipeline test suite on ubuntu-latest / py3.13. - environment/requirements-ci.txt: a light CI dependency set (numpy, onnx, onnxruntime, PyYAML, pytest) — deliberately NOT the pinned torch/tf/oww training env. Every heavy dep stays behind a lazy import or importorskip, so the torch-free ONNX contract+evaluate smoke test runs while the full train->export smoke test skips. Verified in a clean venv with only requirements-ci.txt installed: 119 passed, 1 skipped (the torch-gated smoke test). Co-authored-by: Cursor <cursoragent@cursor.com>
keithtmccartney
changed the title
|
This pull request has been inactive for 21 days and will be closed in 7 days if there is no further activity. Push an update or comment to keep it open. |
Owner
|
Commenting to keep PR active. |
|
This pull request has been inactive for 21 days and will be closed in 7 days if there is no further activity. Push an update or comment to keep it open. |
Owner
|
Commenting to keep PR active. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an isolated, commercially-licensed, reproducible wake word training pipeline
under
wakeword/for the Qube and "Hey Qube" wake words (work item #42). The pipeline iskept entirely separate from the PyQt runtime (its own pinned training env) and is now
code-complete end-to-end (M0–M5) — the only remaining ship gates are recording a
real-voice evaluation corpus and Test Lab sign-off (data + human, not code).
Central design doc:
wakeword/WAKEWORD_PIPELINE.md.Why
The reference Colab notebook pulls datasets whose licenses prohibit commercial use, which
is incompatible with Qube's MIT license. This pipeline keeps the notebook's architecture as
a guide but replaces every dataset with a FOSS (CC0/CC-BY/MIT/Apache/BSD/PD) source and
enforces compliance with a fail-closed license gate + machine-readable provenance
manifests.
What's included
precompute_features.py— regenerate(N,16,96)negative/validation embeddings from LibriSpeech/MUSAN (the ACAV100M replacement), memory-safe shard→merge memmapPlus:
test_smoke_e2e.py): a torch-free ONNX contract+evaluatetest that runs in CI, and a torch-gated
train→export→evaluatepre-flight for the pinned env..github/workflows/wakeword-ci.yml): path-filtered onwakeword/**; runs thelicense gate + full test suite on a light dependency set (no torch/tf).
The exported model matches Qube's
openwakeword==0.4.0runtime contract ((batch,16,96)→(batch,1)), so it drops straight intocore/wakeword_manager.py.Test plan
cd wakeword && python -m pytest tests→ 119 passed, 1 skipped (the torch-gated smoke test)environment/requirements-ci.txt(no torch/tf) → same resultpython scripts/verify_licenses.py --datasets datasets --require-commercial --allow-emptytrain.pyrefuses to train in the commercial tier without licensed datasetsevaluation/RECORDING_PROTOCOL.md)download → precompute → generate → augment → train → export → evaluaterun in the pinned envOut of scope (by design)
Real-voice corpus recording, the full multi-GB training run, and Test Lab sign-off are
data/human steps that happen after this lands. No app runtime code is changed by this PR.