Ambient-sound classifier that runs 100% in the browser. No server. Your audio never leaves your device.
▶ Live demo: https://echolens-pied.vercel.app/
EchoLens classifies 10 categories of urban sound in real time from your microphone — trained on UrbanSound8K:
air_conditioner · car_horn · children_playing · dog_bark · drilling ·
engine_idling · gun_shot · jackhammer · siren · street_music
The entire pipeline runs client-side:
microphone → AudioWorklet (log-mel features) → ONNX model (onnxruntime-web) → label
The model is trained offline in Python and shipped as a small ONNX file; the browser reruns the exact same feature extraction and inference at runtime.
BUILD TIME (Python, training/)
UrbanSound8K ─▶ PyTorch AudioCNN ─▶ ONNX export ─▶ model.onnx + mel_filterbank.json
(36 KB FP32) (frozen feature contract)
RUN TIME (browser, web/) — nothing leaves the device
🎙 mic ─▶ AudioWorklet ─▶ sliding window ─▶ onnxruntime-web ─▶ label + probability bars
log-mel DSP ~4 s / 173 frames WASM, FP32 EMA-smoothed UI
(audio thread) (main thread)
Feature extraction lives in an AudioWorklet on the real-time audio thread, so
the DSP is never blocked by main-thread work (layout, GC, or the inference it
feeds). The main thread runs a sliding-window classifier (~4 classifications/s)
and smooths the output over time. Full diagrams and rationale are in
docs/architecture.md.
The two planes and the frozen artifacts that connect them (source in docs/architecture.mmd):
flowchart TB
subgraph TRAIN["Training plane · Python (training/)"]
direction LR
DS[UrbanSound8K] --> MEL[melspec.py]
MEL --> CNN[AudioCNN]
CNN --> TRN[PyTorch training]
TRN --> CKPT[checkpoint_best.pt]
CKPT --> EXP[export_onnx.py]
end
subgraph IFACE["Frozen interface · build artifacts"]
direction LR
M[model.onnx]
FB[mel_filterbank.json]
NS[norm_stats.json]
end
subgraph AUDIO["Inference · Browser — audio thread (AudioWorklet)"]
direction LR
MIC[mic] --> FE[feature-extractor.worklet.ts]
FE --> RING[ring buffer · N_FFT=1024]
RING --> FRAME[mel frame · 64 values]
end
subgraph MAIN["Inference · Browser — main thread (pipeline.ts / main.ts)"]
direction LR
WIN[sliding window · 173 frames ≈ 4 s] --> TENS[framesToTensor]
TENS --> ORT[onnxruntime-web · WASM FP32]
ORT --> SM[softmax]
SM --> EMA[EMA smoothing]
EMA --> UI[UI · label + bars + spectrum]
end
EXP --> M
MEL --> FB
TRN --> NS
M --> ORT
FB --> FE
NS --> TENS
FRAME -->|postMessage| WIN
| Metric | Value |
|---|---|
| Accuracy (10-fold CV, UrbanSound8K) | 81.9% |
| Macro-F1 (fold 10) | 0.877 |
| Inference p50 / p95 (desktop, WASM) | 6.0 ms / 35.1 ms |
| Inference p50 (mobile Safari) | 27.0 ms |
| Model size (FP32 ONNX) | 36 KB |
| Backend | WASM 1-thread (fastest for this model size) |
Full numbers, per-class accuracy, and the cross-device benchmark are in docs/metrics.md.
- JS↔Python numerical parity verified at max |Δ log-mel| ≈ 4.8e-7 — the browser features are bit-for-bit equivalent to what the model trained on.
- The mel filterbank is exported from Python and reused verbatim in JS rather than recomputed — the single biggest source of cross-language drift (ADR-0005).
- WASM outperforms WebGPU for a model this small: GPU transfer overhead dominates the tiny compute, measured across backends (ADR-0007).
- INT8 quantization was evaluated and rejected — it made the model ~9× larger (not smaller) for this architecture (ADR-0004).
- DSP runs in an AudioWorklet, off the main thread, so continuous inference never causes dropped audio or UI glitches (ADR-0006).
- Train the model: see
training/README.md. - Run the web demo locally: see
web/README.md. - Architecture decisions:
docs/adr/— ADR-0001 through ADR-0008.
A deeper write-up of the design, benchmarks, and trade-offs lives in docs/report.md.
Released under the MIT License. See LICENSE.