Orpheus TTS Fine-Tuning — Custom Voice Generation with Llama-3.2-3B

This project demonstrates how to fine-tune a 3B parameter LLM for text-to-speech using the Orpheus TTS architecture.

Instead of generating text tokens, the model generates SNAC audio tokens, which are decoded into waveform audio.

The result is a lightweight API that can generate natural sounding speech in seconds.

---

🔊 Demo

Live GPU Demo:

https://8000-01kk41cxfsbkbzcx7bd7zyqyf5.cloudspaces.litng.ai/

The demo allows you to:

• Enter text
• Generate speech
• Download the generated .wav audio

---

🧠 Key Idea

Traditional TTS pipelines look like:

text → phonemes → spectrogram → vocoder → audio

Orpheus takes a different approach:

text → LLM → SNAC audio tokens → decoder → waveform

The language model learns to generate sound tokens directly.

---

📦 Model

Fine-tuned model:

https://huggingface.co/yuv008/orpheus-elise-merged

Base model:

Orpheus TTS (Llama-3.2-3B)

---

📊 Dataset

Dataset used for fine-tuning:

https://huggingface.co/datasets/Jinsaryko/Elise

Dataset details:

• ~1,200 audio-text pairs
• single speaker
• voice cloning dataset

---

⚙️ Training Setup

Fine-tuning method:

LoRA (16-bit) via Unsloth

Training notebook:

https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Orpheus_(3B)-TTS.ipynb

Training configuration:

Epochs: 1
Steps: 299
GPU: Single GPU
Training time: ~19 minutes

---

🧩 Training Pipeline

audio files
↓
SNAC codec encoding
↓
audio token sequences
↓
causal language modelling
↓
fine-tuned Orpheus model

---

📦 Model Export

After training:

1. LoRA adapters merged
2. Model converted to GGUF
3. Quantized using Q4_K_M

Model size reduction:

FP16: ~6.3 GB
GGUF Q4_K_M: ~2 GB

---

🚀 Inference Architecture

The model is served through a FastAPI backend.

User text
↓
FastAPI endpoint
↓
llama.cpp inference
↓
SNAC token decoding
↓
WAV audio output

Features:

• dynamic max_tokens scaling
• audio persistence
• downloadable .wav files
• simple web UI

---

☁️ Deployment Experiments

Round 1 — CPU Deployment

Environment:

Azure VM (D4s_v3)

Performance:

~7.5 tokens/sec
~95 seconds per sentence

CPU inference was extremely slow.

---

Round 2 — Optimization

Optimizations attempted:

• rebuilt llama.cpp with AVX2 + FMA
• attempted INT8 dynamic quantization (OOM killed)
• fixed SNAC decoding offsets

SNAC decoding requires subtracting position-specific offsets:

0
4096
8192
12288
16384
20480
24576

Without this correction the model generates silent audio.

---

Round 3 — GPU Deployment

Moved inference to T4 GPU on Lightning.ai

Performance:

~65 tokens/sec
~3 seconds for short prompts
~30× faster than CPU

---

📊 Performance Comparison

Setup	Tokens/sec	Latency
CPU (Azure)	~7.5	~95s
GPU (T4)	~65	~3s

---

💡 Lessons Learned

CPU inference bottlenecks

Large models on CPU are often limited by:

memory bandwidth

not compute.

---

SNAC decoding quirks

Audio tokens require position-aware offsets.

Incorrect decoding results in:

perfect silence

---

Quantization is essential

GGUF quantization enables:

• smaller model size
• faster inference
• deployable models

---

🧰 Tools Used

• Orpheus TTS
• Unsloth
• llama.cpp
• SNAC codec
• FastAPI
• Lightning.ai
• Hugging Face

---

📁 Project Structure

.
├── server.py
├── requirements.txt
├── README.md
└── outputs/

Model weights are hosted externally on Hugging Face.

---

🧪 Example Usage

Run the API:

uvicorn server:app --host 0.0.0.0 --port 8000

Generate speech:

curl "http://localhost:8000/tts?text=Hello world" --output speech.wav

---

🔮 Future Improvements

• streaming audio generation
• real-time voice agents
• multi-speaker fine-tuning
• phoneme-balanced datasets

---