Unified multitask audio modeling — one language model, many audio tasks.
Independent researcher working toward a single framework that handles audio enhancement, separation, conversion and generation with shared weights, instead of maintaining a zoo of single-task models. I'm opinionated about three things, in order:
- Consolidation — one model conditioned on a task, not ten networks to keep alive.
- Reproducibility — seeds, from-scratch maths, and nothing to download to run the demos.
- Honest evaluation — the comparison is the claim; a single averaged score hides more than it shows.
Two halves of the same thesis: a unified model, and the harness that keeps its claims honest. Both are small, dependency-light reference implementations meant to make the idea easy to read and reproduce.
| Project | What it is |
|---|---|
| audiomux | One decoder-only autoregressive model over discrete audio tokens. A classical codec (mu-law + k-means VQ) turns a waveform into tokens; a reserved task token — enhance, separate or generate — conditions the shared weights. Pure NumPy, with the forward and backward passes written by hand and checked against finite differences, so the test suite doubles as a correctness proof. |
| polyaudio | A multitask audio evaluation suite. It drives the same model object across heterogeneous tasks (keyword spotting, language ID, tagging, valence regression, mini-ASR), scores each with the metric that task actually cares about, and prints them side by side against trivial baselines. NumPy-only, because it scores predictions — not a leaderboard number, a table. |
audiomux is the argument that one set of weights can do many audio tasks.
polyaudio is the discipline that keeps that argument honest: if a unified model
can't beat prior or majority on a task, the table says so plainly. Building
both together is the point — a model is only as good as the comparison you're
willing to show next to it.
From-scratch autoregressive decoders over discrete audio tokens · finite-difference gradient checks · declarative TOML evaluation suites · per-task metrics (WER, mAP, Pearson, accuracy) over a shared harness.
- Growing
audiomuxfrom toy synthetic signals toward more realistic audio, keeping the plumbing and maths readable at every step. - Expanding
polyaudiowith more task kinds and baselines, so unified-model claims stay falsifiable. - Interested in: task-conditioned generation, discrete audio codecs, and evaluation that a skeptic would trust.