semantic-synth-control

semantic-synth-control is an ESP32-S31 synthesizer that uses an LLM to interpret spoken natural-language requests, assisted by deterministic reference-patch retrieval, and convert them into validated synth parameter changes.

The repository contains two independently buildable applications:

• firmware/: ESP-IDF firmware for MIDI input, audio capture, synthesis, and Cloud API communication.
• server/: a Cloudflare Worker that runs speech recognition and command interpretation with Workers AI.

USB MIDI keyboard -> ESP32-S31 synth -> I2S codec -> audio output
On-board mic -> Cloudflare Worker -> validated command -> synth parameters

Model output is parsed and validated against the command schema and parameter ranges before it can reach the DSP.

What "semantic" means

The server translates descriptions such as "darker", "wider", or "a soft ambient pad" into concrete oscillator, filter, envelope, LFO, and effect values. It does not ask the LLM to invent parameter names or ranges.

flowchart LR
    A[Voice command] --> B[Speech recognition]
    B --> C[Transcript]
    C --> D[Deterministic retrieval]
    D --> E{Adjustment or new sound?}
    D --> F[Top 3 reference patches]

    G[Current synth state] --> H[LLM context]
    I[Parameter names and ranges] --> H
    J[Available presets] --> H
    E --> H
    F --> H
    C --> H

    H --> K[LLM tool call]
    K --> L[Schema and range validation]
    L --> M[Command JSON]
    M --> N[Firmware validation]
    N --> O[Synth parameters]

Loading

The deterministic retrieval stage matches musical facets such as instrument type, brightness, attack, width, texture, and mood. It then decides whether the request should preserve the current patch (relative) or construct a different sound (redesign), and selects up to three reference patches. The reference-patch set is intentionally small, so retrieval uses deterministic lexical scoring rather than embeddings or RAG.

The LLM receives the transcript together with that retrieval result, the current device state, the available presets, and the complete parameter table. For a relative request it changes only the necessary values. For a redesign it uses the reference patches as points in the synth's parameter space and emits absolute final values through the apply_synth_command tool.

Only two action types can leave the server:

• load_preset: load a named built-in preset.
• set_parameter: set a known parameter to a validated absolute value.

The server rejects unknown names and out-of-range values. The firmware repeats validation before applying the command, so LLM output never reaches the audio engine directly. ASR language selection is configured with ASR_LANGUAGE, and Whisper receives a synth-specific initial prompt. The semantic retrieval vocabulary currently covers Japanese and English; additional languages can be added in server/src/retrieval.ts.

Requirements

To run the cloud API:

• Node.js 20 or later
• npm
• A Cloudflare account with Workers AI access

To build and flash the firmware, also install:

• Windows 11 and PowerShell
• An ESP-IDF release that supports the preview esp32s31 target
• An ESP32-S31 CoreBoard, a USB MIDI keyboard, and an audio output connected to the board codec

The firmware has been built and tested for the esp32-s31-function-coreboard-1. Other ESP32-S31 boards may require changes to GPIO assignments, codec configuration, or peripheral setup.

Clone and install

git clone --recurse-submodules <repository-url>
Set-Location semantic-synth-control
Set-Location server
npm ci
Set-Location ..

For an existing clone without submodules, run git submodule update --init --recursive.

Run the cloud API locally

Authenticate Wrangler and confirm the active Cloudflare account:

Set-Location server
npx wrangler login
npx wrangler whoami

Create server/.dev.vars with a random shared token. Set ASR_LANGUAGE to your ISO 639-1 language code, such as en or ja:

API_TOKEN=replace-with-a-long-random-value
ASR_LANGUAGE=your-language-code

If ASR_LANGUAGE is omitted, Whisper uses automatic language detection.

Start the Worker with npm run dev. Workers AI executes remotely during local development. A workers.dev subdomain must be enabled for the account, and requests consume Workers AI usage. The API is available at http://localhost:8787 unless Wrangler prints a different address.

In another PowerShell session, verify the health endpoint:

Invoke-RestMethod http://localhost:8787/health

Then send a text command. Text input skips speech recognition and is intended for setup verification:

$headers = @{ Authorization = "Bearer replace-with-a-long-random-value" }
Invoke-RestMethod -Method Post `
  -Uri http://localhost:8787/v1/synth/command `
  -Headers $headers `
  -ContentType 'text/plain; charset=utf-8' `
  -Body '明るいパッドの音にして'

To exercise speech recognition, replace the text body with a WAV file:

Invoke-RestMethod -Method Post `
  -Uri http://localhost:8787/v1/synth/command `
  -Headers $headers `
  -ContentType 'audio/wav' `
  -InFile .\command.wav

Deploy to Cloudflare

Set the production API token through Wrangler's interactive prompt, then validate and deploy:

Set-Location server
npx wrangler secret put API_TOKEN
npx wrangler secret put ASR_LANGUAGE
npx wrangler deploy --dry-run
npm run deploy

Wrangler prints the deployed workers.dev URL. Verify it with:

Invoke-RestMethod https://<worker-name>.<subdomain>.workers.dev/health

Do not commit .dev.vars, Cloudflare credentials, Wi-Fi credentials, or the shared API token.

Build and flash the firmware

Open an ESP-IDF PowerShell environment. If IDF_PATH is not already set, create the ignored file scripts/idf_env.local.ps1:

$env:IDF_PATH = 'C:\path\to\esp-idf'

Build, flash, and monitor the board, replacing <your-port> with its serial port:

scripts\build.ps1 firmware
scripts\build.ps1 firmware -p <your-port> flash
scripts\build.ps1 firmware -p <your-port> monitor

The firmware uses BLE provisioning for Wi-Fi credentials. In the serial console, run bt init, provision the device with Espressif's provisioning app, then restart and run wifi start. Configure the Worker and verify connectivity:

api url https://<worker-name>.<subdomain>.workers.dev
api token <same-token-used-by-the-worker>
api health

Checks

scripts\check_all.ps1
scripts\check_all.ps1 -WithFirmware

The first command runs server type checking and tests. The second also builds the firmware and requires the ESP-IDF environment and hardware-specific SDK.

API

POST /v1/synth/command accepts an audio/wav body. A text/plain body skips speech recognition and can be used for development. Every command request requires Authorization: Bearer <token>.

The optional X-Synth-State header contains the current synth state as JSON so relative instructions can be resolved. Responses use one of three statuses: ok, clarify, or no_action.

{
  "status": "ok",
  "actions": [
    { "type": "load_preset", "preset": "Warm Pad" },
    { "type": "set_parameter", "parameter": "filter_cutoff", "value": 3600 }
  ]
}

Unknown actions, parameters, and out-of-range values are discarded before DSP state is changed. GET /health is available without authentication and does not invoke Workers AI.

Third-party licenses are listed in THIRD_PARTY_NOTICES.md.

License

This project is licensed under the MIT License.