Skip to content

Stage 3: in-plugin autoencoder training + dataset building #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gustavokch merged 3 commits into from
Jun 16, 2026
Merged

Stage 3: in-plugin autoencoder training + dataset building #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0a9cb8e
feat(plugin): in-plugin autoencoder training + dataset building (Stag…
gustavokch Jun 16, 2026
c5209d0
style: satisfy clippy 1.96 lints (needless_range_loop, abs_diff, vec_…
gustavokch Jun 16, 2026
d6e3939
fix(training): retire hot-swapped decoders off the audio thread
gustavokch Jun 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 41 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ blog post [here](https://mct-master.github.io/masters-thesis/2024/05/14/alexanjw
| Stage | What | State |
|-------|------|-------|
| **Stage 1** | Original openFrameworks standalone, building/running on Linux x86_64 (runtime only — offline training UI disabled) | Builds & runs on CachyOS/Arch with gcc 16 + openFrameworks 0.12.1. See [`docs/BUILDING-linux.md`](docs/BUILDING-linux.md). |
| **Stage 2** | Rust rewrite as a CLAP + VST3 **MIDI-generator** plugin (nih-plug). Reuses no C++/Pd/Python at runtime. | Working. 16 cargo + 8 pytest tests, clippy clean, `clap-validator` 18/0/3, `pluginval` (VST3) SUCCESS, and headless CLAP + VST3 host tests passing all 14 scales. CI green. |
| **Stage 2** | Rust rewrite as a CLAP + VST3 **MIDI-generator** plugin (nih-plug). Reuses no C++/Pd/Python at runtime. | Working. 34 cargo + 8 pytest tests, clippy clean, `clap-validator` 18/0/3, `pluginval` (VST3) SUCCESS, and headless CLAP + VST3 host tests passing all 14 scales. CI green. |
| **Stage 3** | **In-plugin training**: build a dataset and train the autoencoder live in the DAW, then play *your* model. The Python training pipeline reimplemented from scratch in Rust. | Working. Trains off the audio thread, hot-swaps the result, persists with the session. See [`docs/plans/2026-06-16-runtime-training-design.md`](docs/plans/2026-06-16-runtime-training-design.md). |

The two stages share no runtime code. Stage 1 is the behavioural reference; Stage 2
is the plugin you actually install in a DAW.
Expand All @@ -30,14 +31,39 @@ A **MIDI generator**: it emits notes; your host/synth makes the sound. It has **
internal clock** — it follows the **host transport** (tempo + playhead). Press play
in your DAW and it sequences.

**How it works.** A frozen, offline-trained autoencoder **decoder** turns 4 latent
parameters into a 16-step pattern (which steps fire + a per-step "groove" sub-step
offset). The sequencer plays that pattern at 4 steps/beat (16 per bar), quantising
each step's pitch to a selected scale + key.
**How it works.** An autoencoder **decoder** turns 4 latent parameters into a 16-step
pattern (which steps fire + a per-step "groove" sub-step offset). The sequencer plays
that pattern at 4 steps/beat (16 per bar), quantising each step's pitch to a selected
scale + key. The plugin ships with a frozen, offline-trained decoder as the default, and
you can **train your own model in-session** (see *Training*, below).

**Custom GUI** (egui editor): a 16-step grid with a live playhead and click-to-toggle
cells (a click forces a step on/off, overriding the decoder until the next latent-driven
regeneration), plus sliders for the latent vector, per-step pitches, timing, and tuning.
regeneration), plus sliders for the latent vector, per-step pitches, timing, and tuning,
and a **Training** panel.

## Training (in-plugin)

The plugin reimplements the original's autoencoder *training* in pure Rust — no Python,
no offline step. In the **Training** panel:

1. **Build a dataset.** *Capture pattern* snapshots the current 16-step grid (steps +
sub-step offsets) as a training sample, and/or *Add audio…* loads `.wav`/`.flac` files,
detects onsets (spectral-flux), and encodes each file as a sample. The dataset
accumulates across both sources.
2. **Train.** Set epochs/batch and press *Train*. Training runs on a background thread
(never the audio thread), showing a live epoch/loss progress bar; *Cancel* stops it.
3. **Play your model.** On finish the new decoder is **hot-swapped** into the audio path
(lock-free) and drives the latent sliders immediately. *Encode pattern → latent* runs
the encoder on the current grid to set the 4 latents to that pattern's latent code.

The trained model is **saved with the DAW session** (and travels with presets); reloading
restores it. The baked default decoder remains the fallback when no model has been trained.

> Faithfulness note: the Rust training fixes two bugs in the original Python (broken Adam
> bias correction; no batch shuffle), so it converges better but does not reproduce the
> Python numerically. The offline Python pipeline under `Deep_Steps_project/tools/` still
> exists for reference.

**Parameters** (also host-automatable): Latent A–D, Gate length (ms),
Sub-step scale, Sequence length (1–16), Key (0–11), Scale (14 options: Chromatic,
Expand Down Expand Up @@ -93,14 +119,14 @@ clock and sequences off **incoming MIDI clock**.
- **Step toggles are not preset-persisted.** Grid clicks override the decoder at
runtime but are not saved in presets (they are runtime state, not params) and a
latent move regenerates over them. Promoting them to params is a possible later pass.
- **Shipped weights are from a synthetic dataset.** The original never shipped trained
weights (it random-inits and only becomes meaningful after in-session training).
This port freezes an **offline-trained** decoder, but the committed
`deepsteps-plugin/weights/decoder.json` was trained on a deterministic *synthetic*
corpus (`Deep_Steps_project/tools/make_synth_dataset.py`), so patterns are
reproducible but not musically trained. Train your own from audio with
`Deep_Steps_project/tools/build_dataset.py` + `Deep_Steps_project/tools/train_export.py`
(uses [librosa](https://librosa.org) for onset detection).
- **Default weights are from a synthetic dataset.** The committed default decoder
`deepsteps-plugin/weights/decoder.json` was trained offline on a deterministic
*synthetic* corpus (`Deep_Steps_project/tools/make_synth_dataset.py`), so out-of-the-box
patterns are reproducible but not musically trained. Train your own **in the plugin**
(see *Training*) — or, offline, with the Python tools
`Deep_Steps_project/tools/build_dataset.py` + `train_export.py` ([librosa](https://librosa.org)
onsets). The in-plugin onset detector is spectral-flux based and intentionally not a
librosa clone, so audio-derived datasets are approximate.
- **Two sequencer timing approximations** (flagged for A/B in
[`deepsteps-plugin/NOTES-sequencer.md`](deepsteps-plugin/NOTES-sequencer.md) and
[`VALIDATION.md`](deepsteps-plugin/VALIDATION.md)): the sub-step offset uses a
Expand All @@ -111,7 +137,7 @@ clock and sequences off **incoming MIDI clock**.
## Validation

See [`deepsteps-plugin/VALIDATION.md`](deepsteps-plugin/VALIDATION.md). Automated:
`cargo test` (16), `clap-validator` (18/0/3) and `pluginval` (VST3, strictness 8,
`cargo test` (34), `clap-validator` (18/0/3) and `pluginval` (VST3, strictness 8,
SUCCESS), plus headless host scale tests that load the **shipped** binaries and assert
all 14 scales quantise correctly through both plugin formats — `clap-host-test/` (CLAP)
and `vst3-host-test/` (VST3). All run in CI on every push/PR.
Expand Down
Loading