Mel Spec

Fast Rust mel spectrogram and VAD primitives for ASR systems.

Release note: 0.4.0 moves CPU mel and Kaldi fbank computation onto sparse filterbank projection derived from the same dense reference matrices. On the Parakeet/NeMo JFK benchmark, the pure Rust frontend is now close to C/libtorch CPU trace performance while preserving fixture parity.

mel-spec contains low-cost, predictable components for speech pipelines. The components are:

• STFT
• Whisper-compatible log-mel features
• Kaldi-style filterbanks
• TGA spectrogram interchange
• a lightweight VAD that reuses the same mel and STFT features.

Main Features

Feature	What it is for
Whisper-compatible mel	Log-mel spectrograms aligned with whisper.cpp, PyTorch, and librosa.
Kaldi-compatible fbank	80-bin Kaldi-style filterbank features for speaker and audio models.
Streaming STFT	Overlap-and-save STFT for live audio pipelines.
Interleaved PCM	Downmix stereo or multichannel PCM during batch feature extraction.
Model-free VAD	Fast speech/non-speech decisions and timestamps from mel spectrogram structure.
TGA mel images	Store and pass quantized mel spectrograms as simple 8-bit TGA files.
Local Whisper WASM	Hush uses mel-spec mel tensors/TGA segments for fully local browser Whisper transcription.
Native GPU backends	Experimental CUDA and wgpu paths for batched native mel generation.
Browser worker demo	WASM worker and SharedArrayBuffer example for live browser audio.

Quick Start

use mel_spec::prelude::*;

let samples = vec![0.0_f32; 16_000];
let mel_frames = Spectrogram::compute_mel_spectrogram_cpu(
    &samples,
    400,
    160,
    80,
    16_000.0,
);

println!("frames={}", mel_frames.len());

Use BatchLogMelSpectrogram::compute_interleaved for interleaved audio. The method averages all input channels before feature extraction.

The focused API examples are kept in the example READMEs so the top-level README stays readable:

Example	Description
browser	Stream microphone or WAV audio to a WASM mel worker.
mel_tga	Convert raw audio to TGA mel spectrogram images.
tga_whisper	Transcribe precomputed TGA mel spectrograms with whisper.cpp.
stream_whisper	Stream ffmpeg audio through mel, VAD, and Whisper.
vad_ten_eval	Evaluate mel-spec VAD against the vendored TEN-VAD testset.

Voice Activity Detection

mel-spec includes a lightweight, model-free VAD. It does not load a neural VAD runtime. It looks for speech-like Sobel edge structure in mel spectrogram frames and can attach STFT-derived timestamps to each decision.

Current balanced default on the checked-in TEN-VAD testset:

System	Setting	Macro precision	Macro recall	Macro F1	Macro FPR	RTFx
mel-spec	balanced default	0.8751	0.8785	0.8566	0.3946	819.6
mel-spec	high-F1 sweep result	0.8165	0.9635	0.8769	0.6459	828.9
Silero	tuned threshold 0.13	0.8897	0.9388	0.9088	0.3602	110.3
Silero	default threshold 0.50	0.9379	0.8630	0.8826	0.1778	110.6

The balanced default is not trying to beat learned VADs at strict endpointing. It is a fast built-in option that reuses ASR mel features and avoids another model dependency. Tuned Silero is still more accurate overall. TEN-VAD is the source of the labels and upstream reports stronger precision/recall than Silero and WebRTC on the same testset.

Detailed method, provenance, commands, speed notes, and per-file results are in doc/vad/README.md.

TGA Spectrograms

TGA spectrograms are useful when you want a simple interchange format for mel features. They can be inspected as images, spliced, stored, and passed to the Whisper examples without keeping the original audio around.

This path is live in Hush as local browser ASR. The browser uses the Whisper-compatible log-mel output from mel-spec. It stores captured speech as compact 8-bit TGA images. It decodes the images to an 80-mel Float32Array and passes the tensor to whisper_set_mel in a custom whisper.cpp WASM binding. The active Hush deployment verifies that local WASM Whisper can transcribe from the mel tensor without posting microphone audio to a server. This uses the proposed direct-mel entry point for whisper.cpp, not the stock browser example that feeds PCM audio into whisper.wasm.

"the quest for peace."

Mel spectrograms are also robust under heavy quantization. Whisper does not need high-precision PCM after projection into mel space. Eight-bit TGA images preserve the information that the model sees. Coarse mel-value rounding can also retain useful transcription quality.

Original: [0.158, 0.266, 0.076, 0.196, 0.167, ...]
Rounded:  [0.2,   0.3,   0.1,   0.2,   0.2,   ...]

(top: original mel values, bottom: values rounded to 1.0e-1 before image quantization)

Performance

Benchmarks on Apple M1 Pro, single-threaded release build:

Audio Length	Frames	Time	Throughput
10s	997	21ms	476x realtime
60s	5997	124ms	484x realtime
300s	29997	622ms	482x realtime

mel() and the Kaldi filterbank builder still produce dense filterbank matrices for reference, fixture comparison, and interchange with other toolchains. Runtime mel/fbank computation derives sparse projection tables from these dense matrices. Thus, tests compare the executed math with the same reference weights. A separate filterbank definition is not necessary.

Parakeet/NeMo Frontend Check

asr-api also uses mel-spec filterbanks in its Parakeet/TDT frontend. We compared that Rust frontend with a CPU TorchScript trace of the original NeMo Parakeet featurizer (featurizer_cpu.pt). The test used the 11-second, mono 16 kHz JFK sample on the same M1 Mac.

The benchmark is useful for two reasons:

• It catches frontend contract drift: the first run found a one-frame mismatch (128x1100 vs 128x1101) caused by dropping NeMo's final centered/padded frame.
• After matching the frame count, the Rust features are numerically very close to the traced NeMo frontend.

Featurizer	Shape	Mean	p50	p95	RTFx
Rust Parakeet frontend using mel-spec	128x1101	2.341ms	2.334ms	2.406ms	4699.62
TorchScript CPU trace	128x1101	2.244ms	2.206ms	2.813ms	4902.22

Feature comparison across the full tensor:

Metric	Value
MAE	0.001183
RMSE	0.023699
Max absolute error	3.965733
Correlation	0.999719

The Rust Parakeet path uses BatchLogMelSpectrogram from the existing mel module. It keeps FFT scratch buffers alive between calls and applies the mel filterbank sparsely. This brings the pure Rust frontend close to the traced CPU frontend while avoiding the libtorch/PyTorch runtime dependency. The comparison harness lives in asr-torch as parakeet_featurizer_bench.

The CPU path is the default and is already fast enough for many streaming and batch workloads. Experimental native GPU backends are available behind feature flags:

Feature	Backend
cuda	NVIDIA-only cuFFT plus CUDA mel projection.
wgpu	Native Rust GPU backend for Metal, Vulkan, and DX12 systems.

Build Checks

The top-level library tests do not automatically build every standalone example crate. Use these commands when changing examples:

cargo test --release
cargo build --release --manifest-path examples/mel_tga/Cargo.toml
cargo build --release --manifest-path examples/stream_whisper/Cargo.toml
cargo build --release --manifest-path examples/tga_whisper/Cargo.toml
cargo run --release --manifest-path examples/vad_ten_eval/Cargo.toml
(cd examples/browser && npm ci && npm test)

The Whisper examples compile against the wavey-ai/whisper-rs fork and require a GGML Whisper model to run inference.

Hush Demo

The Hush live browser demo is active at:

https://wavey.ai/code/hush/?v=20260515-35

The source remains at wavey-ai/hush. With the current tuned settings it works as a live browser VAD, spectrogram debugging view, and local Whisper WASM transcription demo. It exposes mel structure, Sobel edges, ridge tracks, candidate speech regions, and the local transcript in real time. The VAD itself should still be treated as experimental rather than a drop-in replacement for a learned VAD. One strong use case is as a browser-side feature/debugging front end or cheap prefilter before a stronger VAD/ASR model.