diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000..c409b0c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,69 @@ +name: Bug report +description: Something is broken in VoiceForge +title: "[bug] " +labels: ["bug"] +body: + - type: markdown + attributes: + value: | + **Do not attach:** private voice samples, generated impersonation audio, API tokens, + unredacted databases, copyrighted datasets, model weights you cannot redistribute, + or consent evidence containing personal data. + - type: textarea + id: summary + attributes: + label: Summary + description: What went wrong? + validations: + required: true + - type: textarea + id: reproduce + attributes: + label: Steps to reproduce + description: Minimal steps without private audio. Prefer the synthetic fixture path if relevant. + validations: + required: true + - type: dropdown + id: engine + attributes: + label: Engine (if applicable) + options: + - openvoice-v2 + - f5-tts + - xtts-v2 + - chatterbox + - qwen3-tts + - rvc + - fish-speech + - cosyvoice-3 + - indextts-2 + - n/a (API / Studio / Docker) + - type: input + id: version + attributes: + label: VoiceForge version / commit + placeholder: e.g. 0.3.0 or git SHA + - type: dropdown + id: hardware + attributes: + label: Hardware path + options: + - CPU Docker + - GPU Docker + - Local Python + - Modal / cloud GPU + - Other + - type: textarea + id: logs + attributes: + label: Logs (redacted) + description: Paste relevant log lines with secrets removed. + - type: checkboxes + id: checklist + attributes: + label: Checklist + options: + - label: I did not upload private voice samples, tokens, or personal data + required: true + - label: I read RESPONSIBLE_USE.md and SECURITY.md where relevant + required: true diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..2fc11e4 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: false +contact_links: + - name: Security report + url: https://github.com/mohitkale/voiceforge/blob/main/SECURITY.md + about: Report vulnerabilities privately — do not open a public issue with secrets or voice samples. + - name: Responsible use + url: https://github.com/mohitkale/voiceforge/blob/main/RESPONSIBLE_USE.md + about: Read before filing issues about cloning, consent, or misuse. diff --git a/.github/ISSUE_TEMPLATE/engine_request.yml b/.github/ISSUE_TEMPLATE/engine_request.yml new file mode 100644 index 0000000..507bc3d --- /dev/null +++ b/.github/ISSUE_TEMPLATE/engine_request.yml @@ -0,0 +1,94 @@ +name: Engine request +description: Propose adding a new cloning / TTS engine adapter +title: "[engine] " +labels: ["engine"] +body: + - type: markdown + attributes: + value: | + Adapters are not “supported” merely because code exists. Include licence and hardware facts. + Do not attach model weights, private samples, or tokens. + - type: input + id: repo + attributes: + label: Engine repository / homepage + validations: + required: true + - type: textarea + id: impl_license + attributes: + label: Implementation licence + description: Licence of the engine software / library. + validations: + required: true + - type: textarea + id: model_license + attributes: + label: Model-weight licence + description: Include commercial-use restrictions if any. + validations: + required: true + - type: textarea + id: hardware + attributes: + label: Hardware requirements + description: CPU viability, GPU/VRAM needs, disk size. + validations: + required: true + - type: dropdown + id: commercial + attributes: + label: Commercial-use status (as you understand it) + options: + - Permissive / generally allowed (confirm upstream) + - Non-commercial / research only + - Unclear — needs review + - Forbidden for our intended use + validations: + required: true + - type: textarea + id: install + attributes: + label: Installation method + description: pip, git, Docker sidecar, isolated venv, etc. + validations: + required: true + - type: textarea + id: api + attributes: + label: API or Python interface + description: How VoiceForge would call clone + synth. + validations: + required: true + - type: textarea + id: test_model + attributes: + label: Test model availability + description: Can CI use a tiny fixture? Any redistribution limits? + - type: dropdown + id: isolation + attributes: + label: Isolation required? + options: + - No (fits main venv) + - Yes (isolated worker venv) + - Yes (HTTP sidecar) + - Unknown + - type: textarea + id: languages + attributes: + label: Expected language support + - type: textarea + id: benchmarks + attributes: + label: Source of any quality/speed claims + description: Links only — do not invent numbers. + - type: checkboxes + id: checklist + attributes: + label: Checklist + options: + - label: I have not attached model weights, private audio, or API tokens + required: true + - label: I understand VoiceForge will document this engine’s licence separately from MIT app code + required: true diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000..4ad7b77 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,30 @@ +name: Feature request +description: Suggest an improvement to VoiceForge +title: "[feature] " +labels: ["enhancement"] +body: + - type: markdown + attributes: + value: | + For new speech engines, use the **Engine request** template instead. + - type: textarea + id: problem + attributes: + label: Problem + description: What user problem does this solve? + validations: + required: true + - type: textarea + id: proposal + attributes: + label: Proposal + description: Describe the change without attaching private audio or tokens. + validations: + required: true + - type: checkboxes + id: checklist + attributes: + label: Checklist + options: + - label: This request does not require shipping third-party voice samples or secrets + required: true diff --git a/.github/ISSUE_TEMPLATE/model_licensing.yml b/.github/ISSUE_TEMPLATE/model_licensing.yml new file mode 100644 index 0000000..fd9e8cf --- /dev/null +++ b/.github/ISSUE_TEMPLATE/model_licensing.yml @@ -0,0 +1,47 @@ +name: Model licensing clarification +description: Flag unclear or incorrect engine/model licence documentation +title: "[licensing] " +labels: ["licensing"] +body: + - type: markdown + attributes: + value: | + Help keep NOTICE.md / docs/ENGINE_LICENSING.md accurate. This is not legal advice. + Do not paste copyrighted datasets or redistributable-restricted weights into the issue. + - type: dropdown + id: engine + attributes: + label: Engine + options: + - openvoice-v2 + - f5-tts + - xtts-v2 + - chatterbox + - qwen3-tts + - rvc + - fish-speech + - cosyvoice-3 + - indextts-2 + - other / multiple + validations: + required: true + - type: textarea + id: issue + attributes: + label: What is inaccurate or unclear? + validations: + required: true + - type: textarea + id: sources + attributes: + label: Upstream sources + description: Links to model cards, licence files, or NOTICE entries. + validations: + required: true + - type: checkboxes + id: checklist + attributes: + label: Checklist + options: + - label: I did not upload model weights or private audio + required: true diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..40565b8 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,20 @@ +## Summary + + + +## Checklist + +- [ ] I did **not** commit private voice samples, API tokens, `.env`, unredacted DBs, or restricted model weights +- [ ] Docs/licence claims match [NOTICE.md](../NOTICE.md) / [docs/ENGINE_LICENSING.md](../docs/ENGINE_LICENSING.md) +- [ ] Responsible-use / consent language is not overstated +- [ ] `ruff check app/ scripts/ tests/` passes +- [ ] `pytest tests/ -q` passes +- [ ] New engines include readiness + licence notes (if applicable) + +## Test plan + + + +## Related issues + + diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..0f246da --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,47 @@ +# Changelog + +All notable changes to this project are documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/). + +## [Unreleased] + +### Added + +- Open-source launch documentation: responsible use, consent, engines matrix, + licensing matrix, first-clone guide, self-hosting, Reel Studio notes, + watermarking docs, demo capture process, contributor/community templates. +- Makefile helpers: `start-cpu`, `start-gpu`, `stop`, `logs`, `smoke-openvoice`. +- Studio consent checkbox; Studio loads engines/voices without a token when the + API is open (localhost); default engine highlight for `openvoice-v2`. +- Authentic Studio screenshot and engine-comparison diagram under `docs/assets/`. + +### Fixed + +- CI Ruff failures (`S110` / import order) in ASR and Chatterbox paths. + +## [0.3.0] — 2026-07-10 + +### Added + +- Additional clone engines (Chatterbox, Qwen3-TTS, Fish Speech sidecar, + CosyVoice 3, IndexTTS2). +- Studio web UI with in-browser recording. +- Modal deploy path and deployment guides. +- SVG architecture / clone-flow diagrams. + +## [0.2.0] — earlier + +### Added + +- GitHub CI, structured logging, `/v1/metrics`, optional synth watermark, + OpenAPI polish, NOTICE.md / SECURITY.md. +- RVC high-fidelity worker, F5-TTS, OpenVoice V2, Docker model pre-download, + CPU e2e smoke tests. + +## [0.1.0] — earlier + +### Added + +- Initial FastAPI service, XTTS-v2 cloning, secured API, Docker CPU/GPU images. diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..4167a36 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,38 @@ +# Contributor Covenant Code of Conduct + +## Our pledge + +We pledge to make participation in VoiceForge a harassment-free experience for +everyone, regardless of age, body size, disability, ethnicity, sex +characteristics, gender identity and expression, experience level, education, +socio-economic status, nationality, personal appearance, race, religion, or +sexual identity and orientation. + +## Our standards + +Examples of behaviour that contributes to a positive environment: + +- Using welcoming and inclusive language +- Being respectful of differing viewpoints +- Accepting constructive criticism gracefully +- Focusing on what is best for the community + +Examples of unacceptable behaviour: + +- Harassment, trolling, or personal attacks +- Publishing others’ private information without permission +- Sharing non-consensual or deceptive voice clones targeting real people +- Uploading private voice samples, API tokens, or personal data in issues/PRs + +## Enforcement + +Project maintainers may remove, edit, or reject contributions that violate this +Code of Conduct, and may ban contributors for behaviours they deem inappropriate. + +Report concerns privately to the repository maintainer (see SECURITY.md for +security-sensitive reports). + +## Attribution + +Adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), +version 2.1. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..29bad75 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,38 @@ +# Contributing + +Thanks for helping improve VoiceForge. + +## Before you start + +1. Read [RESPONSIBLE_USE.md](RESPONSIBLE_USE.md) and [docs/CONSENT.md](docs/CONSENT.md). +2. Read [docs/ENGINE_LICENSING.md](docs/ENGINE_LICENSING.md) — do not imply all + models are commercially usable. +3. Do not commit voice samples of real people, API tokens, `.env`, or `data/`. + +## Development setup + +```bash +python3.11 -m venv .venv && source .venv/bin/activate +pip install -e ".[dev]" +pytest +ruff check app tests scripts +``` + +ML engines are optional for unit tests (a `FakeEngine` covers API layers). + +## Pull requests + +- Keep changes focused; prefer docs/engine adapters in separate PRs when large. +- Run `ruff check app/ scripts/ tests/` and `pytest tests/ -q` before pushing. +- Do not add telemetry, cloud TTS vendors, or analytics. +- New engines: implement `CloneEngine`, register in `app/engines/registry.py`, + document readiness and licences (use the engine-request issue template). + +## Code of conduct + +See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). + +## Security + +See [SECURITY.md](SECURITY.md). Report vulnerabilities privately — never attach +private audio or tokens to public issues. diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..10b58d9 --- /dev/null +++ b/Makefile @@ -0,0 +1,21 @@ +# Makefile helpers for common Docker workflows. +# Prefer these for accessibility; equivalent compose commands remain supported. + +COMPOSE := docker compose -f docker/docker-compose.yml + +.PHONY: start-cpu start-gpu stop logs smoke-openvoice + +start-cpu: + $(COMPOSE) --profile cpu up --build + +start-gpu: + $(COMPOSE) --profile gpu up --build + +stop: + $(COMPOSE) --profile cpu --profile gpu --profile fish down + +logs: + $(COMPOSE) --profile cpu logs -f --tail=200 + +smoke-openvoice: + python scripts/e2e_smoke_test.py --engine openvoice-v2 diff --git a/NOTICE.md b/NOTICE.md index 7cdc3a7..f6b5059 100644 --- a/NOTICE.md +++ b/NOTICE.md @@ -52,3 +52,5 @@ automated e2e tests (not a recording of a real person). Voice cloning can produce convincing audio of real people. Only clone voices you have the legal right to clone. This project requires `consent=true` on voice creation; that is an application control, not a substitute for law. + +See [RESPONSIBLE_USE.md](RESPONSIBLE_USE.md) and [docs/CONSENT.md](docs/CONSENT.md). diff --git a/README.md b/README.md index c38bb23..c5c4602 100644 --- a/README.md +++ b/README.md @@ -1,763 +1,375 @@ # VoiceForge -A local-first, open-source, multi-engine voice cloning service. Record or -upload a short sample, clone the voice, and turn arbitrary text into speech — -entirely on hardware you control, no cloud TTS vendor, no API keys, no -per-character billing. - -Built as a standalone service consumed over HTTP by anything (a web app, a -CLI, curl) — including [Reel Studio](#reel-studio-integration), but with no -dependency on it. - -## Status - -Milestones **M0–M8** are complete (see [Roadmap](#roadmap)). M6 lives in the -client (e.g. Reel Studio); this repo is the cloning API. - -- Full service scaffold: FastAPI, SQLite, bearer auth, CORS, jobs, SSE, - metrics, optional synth watermark, GitHub CI. -- Zero-shot engines: **OpenVoice V2**, **F5-TTS**, **XTTS-v2**, **Chatterbox**, - **Qwen3-TTS 1.7B**, **Fish Speech** (self-hosted), **CosyVoice 3**, - **IndexTTS2**. Pick per voice via `engine_id`. -- High-fidelity: **RVC** (`tier=high_fidelity`, GPU + isolated worker). -- CPU and GPU Docker images; model pre-download + e2e smoke tests. - -**CPU tip:** OpenVoice is the lightest zero-shot path. XTTS / F5 / Qwen3 / -Chatterbox / CosyVoice / IndexTTS / RVC are much happier on a GPU session — -see [GPU cloud deployment](#gpu-cloud-deployment). - -Fish Speech talks to a **local** fish-speech HTTP sidecar -(`VOICEFORGE_FISH_SPEECH_URL`) — not the Fish Audio cloud API. - -## Mission & non-negotiables - -- **100% local-first.** No cloud vendor, no API keys required, no rate - limits. Runs on a dev machine or a self-hosted VPS/GPU box. -- **Audio-only cloning.** No video processing. -- **Multi-engine, pluggable.** One interface (`CloneEngine`), implemented - once per engine, selected per voice. -- **API-first.** Every capability is exposed over HTTP. -- **Non-commercial / demo project.** Built for a portfolio, not a SaaS. Some - engines (XTTS-v2) carry non-commercial model licenses — see - [Licensing & responsible use](#licensing--responsible-use). -- **GPU available, CPU fallback.** Everything also runs on CPU, just slower. +[![CI](https://github.com/mohitkale/voiceforge/actions/workflows/ci.yml/badge.svg)](https://github.com/mohitkale/voiceforge/actions/workflows/ci.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) -## Architecture +**Clone a voice locally. Generate speech privately.** -![System architecture](docs/architecture.svg) +VoiceForge is a self-hosted voice cloning studio and API: record or upload a +short sample, create a reusable voice, and synthesize speech with **multiple +local engines** — switching engines without rebuilding your workflow. -Single Python process (FastAPI + Uvicorn), `asyncio` background tasks for -voice processing, SQLite for metadata, and a `data/` directory on disk for -audio samples and per-voice artifacts (cached conditioning latents, etc.). -Optional engines (CosyVoice 3, IndexTTS2, RVC) run in isolated worker venvs -when their dependency pins conflict with the main app. +Self-hosted by design. Run it on your workstation, private server, or GPU +environment. Voice samples and generated audio go only to the infrastructure +you choose. No cloud TTS account. No per-character billing. -### Clone and synthesize flow +> **Project status: Early public release** +> +> The local Studio, API, engine registry, Docker setup, and supported cloning +> workflows are available. Engine installation, quality, and hardware +> requirements vary, and some integrations still need broader hardware testing. +> Milestone history lives in [ROADMAP.md](ROADMAP.md). -![Clone flow](docs/clone-flow.svg) +**VoiceForge application code is MIT licensed. Each speech engine and model has +its own terms** — see [NOTICE.md](NOTICE.md) and +[docs/ENGINE_LICENSING.md](docs/ENGINE_LICENSING.md). -### The `CloneEngine` interface +## Studio -Every engine implements one interface (`app/engines/base.py`); the API and -job runner talk only to it, never to a vendor SDK directly: +![VoiceForge Studio](docs/assets/voiceforge-studio.png) -```python -class CloneEngine(Protocol): - id: str - label: str - capabilities: CloneCapabilities - - def is_ready(self) -> bool: ... - async def create_voice(self, voice_id, sample_paths, tier, language, on_progress=None) -> VoiceArtifact: ... - async def synthesize(self, voice_id, artifact, text, opts) -> bytes: ... -``` +[First clone guide](docs/FIRST_CLONE.md) · +[Engine matrix](docs/ENGINES.md) · +[Demo capture](docs/DEMO_CAPTURE.md) -Adding a new engine = one new file in `app/engines/` + one entry in -`app/engines/registry.py`. Nothing else changes. +### Audio examples -## Quickstart (Docker) +Engine comparison audio should be generated by a running VoiceForge instance +(same synthetic reference + neutral sentence). Captured samples are listed in +[docs/assets/AUDIO_SOURCES.md](docs/assets/AUDIO_SOURCES.md). -Everything runs **inside Docker** so model crashes stay in the container — -your Mac host only needs Docker Desktop. +| Engine | Hardware | Sample | Notes | +|--------|----------|--------|-------| +| OpenVoice V2 | CPU friendly | *capture pending* | Fast local starting point | +| F5-TTS | GPU preferred | *capture pending* | Higher resource use | +| XTTS-v2 | GPU preferred | *capture pending* | Model licence restrictions (CPML) | +| Qwen3-TTS / Chatterbox | GPU preferred | *capture pending* | Extra install / worker | -```bash -cp .env.example .env -# Optional: set VOICEFORGE_API_TOKEN if you will expose beyond localhost. +Do not treat any engine as universally “best” — quality depends on sample, +language, and hardware. See [DEMO_CAPTURE.md](docs/DEMO_CAPTURE.md) to produce +verified WAVs/MP3s for this table. -# CPU (recommended on macOS — no NVIDIA GPU required): -docker compose -f docker/docker-compose.yml --profile cpu up --build -``` +### Multi-engine overview -Then open in your browser: +One Studio/API, many local engines. This is **not** a quality ranking — engines +differ by licence, hardware, and verification status +([docs/ENGINES.md](docs/ENGINES.md)). -| URL | What | -|-----|------| -| **http://localhost:8089/** | **VoiceForge Studio** — pick an engine, upload a sample or record one in-browser (with a read-aloud script), clone, synthesize, play audio | -| http://localhost:8089/docs | Interactive OpenAPI (Swagger) | -| http://localhost:8089/healthz | Liveness JSON | +```mermaid +flowchart LR + subgraph Clients + A[Studio / curl / Reel Studio] + end -Compose caps the container at **4 CPUs / 6 GB RAM** by default so a heavy -model load cannot thrash the whole laptop. Data lives in `./data` and model -weights in the `models-cache` Docker volume (not scattered across your Mac). + subgraph VoiceForge + B[FastAPI + CloneEngine registry] + end -**Mac tip:** start with engine `openvoice-v2` in the Studio UI — it is the -lightest zero-shot path on CPU. First request for any engine may download -checkpoints into the Docker volume (can take several minutes). + subgraph Engines + C1[openvoice-v2
CPU starter] + C2[f5-tts / xtts-v2
chatterbox / qwen3-tts] + C3[rvc / fish-speech
cosyvoice-3 / indextts-2] + end -```bash -# Pre-download built-in engine models (optional, still inside Docker): -docker compose -f docker/docker-compose.yml --profile cpu run --rm voiceforge-download - -# GPU (requires NVIDIA Container Toolkit — not typical on Mac): -# docker compose -f docker/docker-compose.yml --profile gpu up --build + A -->|HTTP + SSE| B + B --> C1 + B --> C2 + B --> C3 ``` -End-to-end smoke test (service must be running; run from a dev venv): - -```bash -docker compose -f docker/docker-compose.yml --profile cpu up --build -d -python scripts/e2e_smoke_test.py --engine openvoice-v2 -python scripts/e2e_smoke_test.py --engine xtts-v2 -python scripts/e2e_smoke_test.py --engine f5-tts +## How it works + +1. **Choose an engine** (`openvoice-v2` recommended on CPU). +2. **Upload or record** a short reference clip. +3. **Confirm consent** (Studio checkbox / API `consent=true`). +4. **Clone** — VoiceForge builds a reusable voice artifact. +5. **Synthesize** text → WAV on your infrastructure. +6. **Delete** the voice anytime (`DELETE /v1/voices/{id}` removes samples + artifacts). + +```mermaid +flowchart TD + U[Upload or record sample] --> P[Validate + preprocess] + P --> C{consent=true?} + C -->|no| X[Reject 422] + C -->|yes| V[create_voice on selected engine] + V --> R[Voice ready · SSE progress] + R --> S[POST /v1/synthesize] + S --> W[Optional watermark] + W --> O[audio/wav bytes] + R --> D[DELETE voice · wipe samples + artifacts] ``` -Or use the Studio UI at `http://localhost:8089/` instead of curl. +SVG copies (open as files if you prefer): [clone flow](docs/clone-flow.svg) · +[engine comparison](docs/assets/engine-comparison.svg) -### CPU vs. GPU +## Responsible use -| | CPU (`Dockerfile.cpu`) | GPU (`Dockerfile.gpu`) | -|---|---|---| -| Works on | Any host, incl. a cheap VPS | Host with an NVIDIA GPU + Container Toolkit | -| XTTS-v2 synthesis | Several seconds per sentence | Sub-second to a few seconds | -| Verified in this repo's own testing | ✅ Full e2e (all three engines) via `scripts/e2e_smoke_test.py` | ⚠️ Written and reviewed carefully, but **not** run against a real GPU while building this — no GPU was available. Please verify on your hardware. RVC requires the GPU image. | +Clone **only** your own voice or a voice for which you have clear permission. -### RVC setup (high-fidelity tier) +VoiceForge requires an explicit consent acknowledgement when creating a voice, +but software **cannot** independently verify ownership or legal authority. +Operators are responsible for obtaining and retaining valid permission. -RVC training uses **fairseq** and **numpy<2**, which conflict with -coqui-tts in the main VoiceForge venv. Training/inference therefore run in an -**isolated worker venv** (`/opt/rvc-venv` in the GPU Docker image, or install -manually from `requirements-rvc.txt`). +Do **not** use VoiceForge for impersonation, fraud, harassment, bypassing +identity verification, or misleading people about the origin of audio. -```bash -# GPU Docker (recommended) — worker venv is pre-installed: -docker compose -f docker/docker-compose.yml --profile gpu up --build +Optional watermarking is a weak fingerprint — not tamper-proof detection. +Details: [RESPONSIBLE_USE.md](RESPONSIBLE_USE.md) · +[docs/CONSENT.md](docs/CONSENT.md) · +[docs/WATERMARKING.md](docs/WATERMARKING.md). -# Create a high-fidelity RVC voice (≥3 min of clean speech recommended): -curl -F "name=Studio Voice" -F "engine_id=rvc" -F "tier=high_fidelity" \ - -F "consent=true" -F "files=@vocals.wav" http://localhost:8089/v1/voices +## Supported engines -# Upgrade an existing voice to high-fidelity (re-trains if engine supports it): -curl -X POST http://localhost:8089/v1/voices//upgrade +### Easiest starting point -# Pre-download RVC base weights (HuBERT, RMVPE): -docker compose -f docker/docker-compose.yml --profile gpu run --rm voiceforge-download --engine rvc -``` +- **`openvoice-v2`** — lightest zero-shot path on CPU; CPU e2e verified -Set `VOICEFORGE_RVC_PYTHON=/path/to/rvc-venv/bin/python` when running outside -the GPU image. Tune training with `VOICEFORGE_RVC_EPOCHS` (default `50`) and -`VOICEFORGE_RVC_BATCH_SIZE` (default `4`). +### Higher-quality / heavier zero-shot -### M8 engines (Chatterbox, Qwen3-TTS, Fish Speech, CosyVoice 3, IndexTTS2) +- `f5-tts`, `xtts-v2` (CPML non-commercial), `chatterbox`, `qwen3-tts` -| Engine | Install | Notes | -|--------|---------|--------| -| `chatterbox` | Isolated worker (`/opt/chatterbox-venv` in CPU image) | MIT; numpy/torch pins conflict with main venv | -| `qwen3-tts` | `pip install -r requirements-qwen3.txt` (not in default CPU image) | In-process; ~6GB+ VRAM recommended | -| `fish-speech` | Run fish-speech server; set `VOICEFORGE_FISH_SPEECH_URL` | Self-hosted only — no cloud API keys | -| `cosyvoice-3` | Worker venv + `VOICEFORGE_COSYVOICE_PYTHON` | See `requirements-cosyvoice.txt` / `scripts/cosyvoice_worker.py` | -| `indextts-2` | Worker venv + `VOICEFORGE_INDEXTTS_PYTHON` | See `requirements-indextts.txt` / `scripts/indextts_worker.py` | +### Advanced / externally managed -```bash -# Pre-download (when the matching extra / worker is installed): -python scripts/download_models.py --engine chatterbox -python scripts/download_models.py --engine qwen3-tts -python scripts/download_models.py --engine cosyvoice-3 -python scripts/download_models.py --engine indextts-2 - -# Smoke (service running; ASR engines need intelligible speech — use the F5 fixture): -python scripts/e2e_smoke_test.py --engine chatterbox -python scripts/e2e_smoke_test.py --engine qwen3-tts \ - --reference-wav scripts/fixtures/f5_reference_en.wav -``` +- `rvc` (GPU + training), `fish-speech` (self-hosted sidecar), + `cosyvoice-3`, `indextts-2` -## API reference +Full matrix (CPU/GPU, Docker defaults, licences, verification labels): +**[docs/ENGINES.md](docs/ENGINES.md)**. -Full interactive docs at `/docs`. Summary: +| Engine | Clone type | CPU | GPU | Default CPU image | Licence note | Verification | +|--------|------------|-----|-----|-------------------|--------------|--------------| +| `openvoice-v2` | Zero-shot | Yes | Yes | Yes | MIT VC + verify YourTTS base | Verified (CPU e2e) | +| `f5-tts` | Zero-shot | Slow | Recommended | Yes | Apache-2.0 / CC (upstream) | Verified (CPU e2e) | +| `xtts-v2` | Zero-shot | Slow | Recommended | Yes | **CPML non-commercial** | Verified (CPU e2e) | +| `chatterbox` | Zero-shot | Worker | Optional | Yes (worker venv) | MIT | Smoke-documented | +| `qwen3-tts` | Zero-shot | Limited | Recommended | Yes | Apache-2.0 | Smoke-documented | +| `rvc` | Trained | No | Required | No (GPU image) | MIT architecture | GPU verification needed | +| `fish-speech` | Sidecar | Depends | Recommended | External | Check upstream | External sidecar | +| `cosyvoice-3` | Zero-shot | Limited | Recommended | Extra install | Apache-2.0 | GPU verification needed | +| `indextts-2` | Zero-shot | Limited | Recommended | Extra install | Check upstream | GPU verification needed | -``` -GET /healthz -> { status, service, version, enginesReady, enginesTotal } (no auth) -GET /v1/metrics -> { uptimeSeconds, voicesCreated, synthRequests, ... } (auth) +GPU Docker paths were written carefully but **not** run against real NVIDIA +hardware during development — please verify on your host. -GET /v1/engines -> [{ id, label, capabilities, ready, configured }] - -POST /v1/voices (multipart/form-data) - fields: name, engine_id, tier ("instant" | "high_fidelity"), - consent=true (required), language, files[] (audio) - -> { id, name, engineId, tier, status, ... } - -GET /v1/voices -> [VoiceSummary] -GET /v1/voices/{id} -> VoiceDetail (incl. errorMessage, readyAt) -DELETE /v1/voices/{id} -> 204, deletes voice + samples + artifacts - -POST /v1/voices/{id}/samples (multipart) -> add more reference audio -POST /v1/voices/{id}/upgrade -> re-train at high_fidelity tier (fine-tunable engines) -GET /v1/voices/{id}/events (SSE) -> processing progress events -GET /v1/voices/{id}/preview -> a short cached WAV clip - -POST /v1/synthesize - body: { voiceId, text, sampleRate?, speed?, language? } - -> audio/wav bytes -``` - -Every `/v1/*` route requires `Authorization: Bearer ` -**when a token is configured**. See [Security](#security). - -## Configuration - -All settings are environment variables, prefixed `VOICEFORGE_` (see -`.env.example` for the full list with defaults): - -| Variable | Purpose | -|---|---| -| `VOICEFORGE_API_TOKEN` | Bearer token required on `/v1/*`. Required beyond localhost. | -| `VOICEFORGE_CORS_ORIGINS` | Comma-separated allowed browser origins. | -| `VOICEFORGE_DATA_DIR` / `VOICEFORGE_MODELS_DIR` | Where samples/DB and model checkpoints live. | -| `VOICEFORGE_DEVICE` | `cpu`, `cuda`, or `auto`. | -| `VOICEFORGE_MAX_UPLOAD_MB` / `_MAX_SAMPLES_PER_VOICE` / `_MAX_SYNTH_CHARS` | Abuse/resource-exhaustion limits. | -| `VOICEFORGE_PREPROCESS_SAMPLES` | Trim/normalize reference uploads before cloning (default: true). | -| `VOICEFORGE_RVC_PYTHON` / `_RVC_EPOCHS` / `_RVC_BATCH_SIZE` | Isolated RVC worker (high-fidelity). | -| `VOICEFORGE_FISH_SPEECH_URL` | Local Fish Speech open-weights API base URL (e.g. `http://127.0.0.1:8080`). | -| `VOICEFORGE_COSYVOICE_PYTHON` | CosyVoice 3 worker interpreter (default `/opt/cosyvoice-venv/bin/python`). | -| `VOICEFORGE_INDEXTTS_PYTHON` | IndexTTS2 worker interpreter (default `/opt/indextts-venv/bin/python`). | -| `VOICEFORGE_CHATTERBOX_PYTHON` | Chatterbox worker interpreter (default `/opt/chatterbox-venv/bin/python`). | -| `VOICEFORGE_ENABLED_ENGINES` | Comma-separated engine ids enabled for clone/synth (empty = all). Modal sets via `MODAL_ENABLED_ENGINES`. | -| `VOICEFORGE_WARMUP_ENGINES` | Comma-separated ids to preload at startup (e.g. `qwen3-tts,chatterbox`). | -| `VOICEFORGE_MODAL_DATA_VOLUME` | Modal volume name to commit after voice writes (Modal deploy only). | -| `VOICEFORGE_LOG_FORMAT` | `text` (default) or `json` for structured log lines. | -| `VOICEFORGE_WATERMARK_ENABLED` | Mix a quiet voice-specific fingerprint into synth output (default: false). | -| `VOICEFORGE_WATERMARK_STRENGTH` | Watermark amplitude 0–1 (default: `0.004`). | -| `COQUI_TOS_AGREED` | Non-interactive acceptance of Coqui's CPML for the XTTS-v2 checkpoint download. See licensing below. | - -## Security - -This is a local-first tool, but it's built defensively since it's designed -to be exposed on a home network / VPS: - -- **Auth:** single bearer token (`VOICEFORGE_API_TOKEN`), compared in - constant time (`secrets.compare_digest`), required on every `/v1/*` route - when set. The app logs a loud warning at startup if it's unset (dev/pure - localhost use only — you are responsible for not exposing an - unauthenticated instance). -- **CORS:** locked to an explicit origin allowlist (`VOICEFORGE_CORS_ORIGINS`); - empty by default (no cross-origin browser access at all). -- **Upload validation:** every uploaded file is sniffed as real audio via - `libsndfile` (not trusted by extension/Content-Type), size-capped - (`VOICEFORGE_MAX_UPLOAD_MB`), duration-bounded, and count-capped per voice. - Filenames are never used to build filesystem paths — every stored file - gets a server-generated UUID name, which rules out path traversal via a - crafted upload filename. -- **Resource limits:** a global semaphore (`VOICEFORGE_MAX_CONCURRENT_JOBS`, - default 1) caps concurrent heavy ML jobs so one CPU-only or small-GPU host - can't be overwhelmed by concurrent requests; text length is capped for - `/synthesize`. -- **Checkpoint loading:** cached conditioning-latent tensors are loaded with - `torch.load(..., weights_only=True)` — these files only ever contain - tensors this service wrote itself, so arbitrary pickle deserialization is - never enabled. -- **Docker:** both images run as a non-root user, install only the OS - packages actually needed (`libsndfile1`, `ffmpeg`, `curl`), and ship a - `HEALTHCHECK`. -- **Dependencies:** core and zero-shot engine deps are version-pinned - (`requirements*.txt` / `pyproject.toml`) from PyPI or the official - PyTorch wheel index, and re-checked with `pip-audit` (see - [Development](#development)). The **optional RVC worker** may install - one git dependency (`rvc-no-gui`) into an isolated venv only — see - `NOTICE.md`. - -**Known accepted risk:** `torch==2.8.0` (the newest version confirmed -compatible with `coqui-tts` without requiring the extra `torchcodec` -dependency at `torch>=2.9`) has a small number of open, low-severity PyTorch -advisories (e.g. `GHSA-c678-jfcj-6jmf`, `GHSA-f4hp-rmr7-r7v8`, -`GHSA-x3gm-94wq-g975`, `GHSA-rrmf-rvhw-rf47`). All require local/authenticated -access and direct invocation of specific low-level APIs -(`torch.jit.script`, `pad_packed_sequence`, quantized ops, JIT futures) that -this service's request-handling code never calls. If you have a strict -compliance requirement, upgrade to `torch>=2.9`/`2.10` and add the -`torchcodec` dependency yourself (see `requirements-xtts.txt`). - -## Licensing & responsible use - -- **This repository's code** is MIT-licensed (`LICENSE`). -- **Model weights are separate** — see **`NOTICE.md`** for the full table. - Short version: - | Engine | Weights license (typical) | Commercial? | - |--------|---------------------------|-------------| - | `openvoice-v2` | OpenVoice VC: MIT; YourTTS base: verify Coqui card | VC yes; verify base | - | `f5-tts` | Apache-2.0 / CC (upstream) | Usually yes — confirm | - | `xtts-v2` | **CPML — non-commercial / research only** | **No** | - | `rvc` | RVC architecture MIT; your fine-tune is yours | Yes (architecture) | - | `chatterbox` | MIT | Yes | - | `qwen3-tts` | Apache-2.0 | Yes | - | `fish-speech` | Check fish-speech upstream (self-hosted) | Verify | - | `cosyvoice-3` | Apache-2.0 | Yes | - | `indextts-2` | Check IndexTTS upstream | Verify | -- Application controls (not a substitute for law): - - Every voice creation requires `consent: true`. - - `GET /v1/engines` exposes `capabilities.license` for client UIs. -- This is a **voice-cloning tool** — only clone voices you have the right to - clone. - -## Development - -Requires Python 3.11+ (the ML stack is verified against 3.11 specifically, -matching the Docker images). +## Quick start ```bash -python3.11 -m venv .venv && source .venv/bin/activate -pip install -e ".[dev]" # core + dev tooling, no ML stack -pip install -r requirements-xtts.txt -r requirements-f5.txt \ - --extra-index-url https://download.pytorch.org/whl/cpu # both engines locally - -pytest # full test suite (mocks the ML engine — no GPU/torch download required) -ruff check app tests scripts -pip-audit # dependency vulnerability scan +cp .env.example .env +# Optional: set VOICEFORGE_API_TOKEN if you will expose beyond localhost. -# Optional: score a clone (speaker similarity + WER) — needs torch + benchmark extras: -pip install -e ".[xtts,benchmark]" -python scripts/benchmark_quality.py \\ - --reference sample.wav --generated synth.wav \\ - --text "The sentence that was synthesized." +make start-cpu +# equivalent: +# docker compose -f docker/docker-compose.yml --profile cpu up --build ``` -Tests never require torch/coqui-tts: a lightweight in-memory `FakeEngine` -(pure numpy/soundfile) is registered alongside the real `xtts-v2` engine so -the API, DB, validation, and auth layers are fully exercised without a -multi-GB model download. The XTTS-v2 engine itself was verified separately, -end-to-end, inside the CPU Docker image (real upload → clone → synthesize → -valid WAV output). - -## Roadmap - -- [x] **M0 — Scaffold:** FastAPI skeleton, `/healthz`, SQLite models, Docker - CPU image, engine registry, README. -- [x] **M1 — MVP clone (XTTS-v2):** upload/record → instant zero-shot voice - → `/synthesize` returns WAV. Manually + automatically verified. -- [x] **M2 — Quality benchmarking:** `scripts/benchmark_quality.py` - (speaker-similarity + WER), reference-audio preprocessing on upload. -- [x] **M3a — F5-TTS engine:** permissive-license zero-shot engine behind - `CloneEngine`. -- [x] **M3b — OpenVoice V2 engine:** MIT-licensed zero-shot engine (Coqui VC - + YourTTS base) behind `CloneEngine`. -- [x] **M4 — High-fidelity tier:** RVC training pipeline (isolated worker), - SSE training-progress events, `POST /v1/voices/{id}/upgrade` flow. -- [x] **M5 — Docker hardening:** GPU + CPU images, model-cache volume, - `voiceforge-download` compose service, `scripts/e2e_smoke_test.py` - (CPU e2e verified for openvoice-v2, xtts-v2, f5-tts). -- [x] **M6 — Reel Studio integration:** `VoiceProvider` + clone UI in the - client repo (verified); see - [Integrating with Reel Studio or any custom app](#integrating-with-reel-studio-or-any-custom-app). -- [x] **M7 — Polish & release:** GitHub CI, structured JSON logging, - `/v1/metrics`, enhanced `/healthz`, optional synth watermarking, - OpenAPI tag docs, `NOTICE.md` / `SECURITY.md` (v0.2.0). -- [x] **M8 — Additional engines:** Chatterbox, Qwen3-TTS 1.7B, Fish Speech - (self-hosted sidecar), CosyVoice 3 + IndexTTS2 (isolated workers), - SVG architecture docs. - -## Integrating with Reel Studio or any custom app - -VoiceForge is a **standalone HTTP service**. Reel Studio (or any other app) -talks to it over REST + SSE — no shared code, no merged Docker compose -required. M6 is implemented **in the client repo** (e.g. a new -`VoiceProvider` in Reel Studio); this repo only runs the cloning API. - -### Architecture - -See the SVG diagrams at the top of this README -([architecture](docs/architecture.svg), [clone flow](docs/clone-flow.svg)). -Reel Studio (or any client) calls VoiceForge over HTTP with -`VOICEFORGE_SERVICE_URL` — no shared process, no merged Docker compose. - -- **List voices / synthesize:** call VoiceForge from your **backend** (Reel - Studio's `VoiceProvider.synth()` pattern) — no browser CORS issues. -- **Clone upload from browser:** either set `VOICEFORGE_CORS_ORIGINS` or (recommended) - add a Next.js API route that proxies multipart uploads so the token stays - on the server. - ---- - -### Step 1 — Start VoiceForge - -**Docker (recommended):** - -```bash -cd /path/to/audio-cloning -cp .env.example .env -# Optional but recommended if Reel Studio or other clients send a token: -# VOICEFORGE_API_TOKEN=$(python -c "import secrets; print(secrets.token_urlsafe(32))") +| URL | What | +|-----|------| +| **http://localhost:8089/** | **Studio** — engine picker, upload/record, consent, clone, synth | +| http://localhost:8089/docs | OpenAPI | +| http://localhost:8089/healthz | Liveness | -# Pre-download models (first run otherwise downloads multi-GB on first request): -docker compose -f docker/docker-compose.yml --profile cpu run --rm voiceforge-download +After the first build, `make start-cpu` (or compose `up` without `--build`) is +enough. Helpers: `make stop`, `make logs`, `make smoke-openvoice`. -# Start the API (CPU — works everywhere; GPU profile for RVC / faster synth): -docker compose -f docker/docker-compose.yml --profile cpu up -d --build -``` +Compose caps the CPU service at **4 CPUs / 6 GB RAM** by default. Data → `./data`; +weights → `models-cache` Docker volume. -**Verify:** +**First clone:** follow **[docs/FIRST_CLONE.md](docs/FIRST_CLONE.md)** +(sample length, quiet room, expected download time, deletion). ```bash -curl http://localhost:8089/healthz -# -> {"status":"ok","service":"voiceforge","version":"0.2.0","enginesReady":3,...} +# Optional model pre-download: +docker compose -f docker/docker-compose.yml --profile cpu run --rm voiceforge-download -open http://localhost:8089/docs # interactive OpenAPI +# GPU (NVIDIA Container Toolkit — not typical on Mac): +# make start-gpu ``` -Default base URL: **`http://localhost:8089`** +### CPU vs GPU ---- +| | CPU | GPU | +|---|-----|-----| +| Best for | Laptops, cheap VPS, first clone (`openvoice-v2`) | Heavier engines, RVC, faster synth | +| Verified here | CPU e2e for openvoice-v2, xtts-v2, f5-tts | Image exists; **real-GPU verification still needed** | -### Step 2 — Configure auth and CORS +## Studio usage -| Scenario | What to set | -|----------|-------------| -| Local dev, server-to-server only | Leave `VOICEFORGE_API_TOKEN` empty; Reel Studio calls from Node without auth. | -| Token required | Same token in VoiceForge `.env` and Reel Studio `VOICEFORGE_API_TOKEN`. | -| Browser uploads directly to VoiceForge | Add Reel Studio origin to `VOICEFORGE_CORS_ORIGINS=http://localhost:3000` | +1. Open http://localhost:8089/ +2. Paste `VOICEFORGE_API_TOKEN` only if the API requires it (localhost can be open) +3. Select **openvoice-v2**, upload/record, check **consent**, clone +4. Synthesize and play; delete voices you no longer need -Generate a token: +## API usage -```bash -python -c "import secrets; print(secrets.token_urlsafe(32))" -``` - ---- - -### Step 3 — Smoke-test the API (curl) - -**List engines** (pick `engine_id` + read `capabilities.license` for UI): - -```bash -curl -s http://localhost:8089/v1/engines | python3 -m json.tool -``` - -**Create a cloned voice** (instant zero-shot example with OpenVoice): +When `VOICEFORGE_API_TOKEN` is set, send `Authorization: Bearer ` on `/v1/*`. ```bash curl -s -X POST http://localhost:8089/v1/voices \ - -H "Authorization: Bearer $VOICEFORGE_API_TOKEN" \ -F "name=Demo Clone" \ -F "engine_id=openvoice-v2" \ -F "tier=instant" \ -F "consent=true" \ -F "language=en" \ - -F "files=@reference.wav" \ - | python3 -m json.tool -``` - -**Poll until ready** (or use SSE — next section): - -```bash -curl -s http://localhost:8089/v1/voices/ \ - -H "Authorization: Bearer $VOICEFORGE_API_TOKEN" -``` - -**Synthesize:** + -F "files=@sample.wav" -```bash curl -s -X POST http://localhost:8089/v1/synthesize \ - -H "Authorization: Bearer $VOICEFORGE_API_TOKEN" \ -H "Content-Type: application/json" \ - -d '{"voiceId":"","text":"Hello from VoiceForge."}' \ + -d '{"voiceId":"","text":"Hello from VoiceForge."}' \ -o out.wav ``` -Or run the bundled e2e script (from a dev venv with `httpx`): - -```bash -pip install -e ".[dev]" -python scripts/e2e_smoke_test.py --engine openvoice-v2 ``` - ---- - -### Step 4 — Wire Reel Studio (env vars) - -In the **reel-studio** repo `.env.local`: - -```bash -VOICEFORGE_SERVICE_URL=http://localhost:8089 -VOICEFORGE_API_TOKEN= # same as VoiceForge .env when set; empty for open localhost dev +GET /healthz +GET /v1/engines +POST /v1/voices # consent=true required +GET /v1/voices/{id}/events # SSE progress +DELETE /v1/voices/{id} # deletes samples + artifacts +POST /v1/synthesize # → audio/wav +GET /v1/metrics ``` -Reel Studio already documents the provider pattern in -`docs/voice-clone-service-BRIEF.md` §11. Implementation lives entirely in -reel-studio — **not in this repo**. - ---- - -### Step 5 — What to build in Reel Studio (M6 checklist) +Interactive docs: `/docs`. Configuration: `.env.example` (`VOICEFORGE_*`). -| File / area | Action | -|-------------|--------| -| `src/providers/voice/types.ts` | Add `"voiceforge"` to `PROVIDER_IDS`. | -| `src/providers/voice/voiceforge.ts` | New `VoiceProvider`: `listVoices`, `listModels`, `synth`. | -| `src/providers/voice/registry.ts` | Register `createVoiceforgeProvider`. | -| `.env.example` | Document `VOICEFORGE_SERVICE_URL`, `VOICEFORGE_API_TOKEN`. | -| Clone UI (new route or `/voices` panel) | Record/upload → `POST /v1/voices` → SSE progress → voice appears in browser. | -| Optional API proxy | `src/app/api/voiceforge/...` to proxy clone uploads without exposing token to the browser. | +## Reel Studio integration -**Provider mapping (VoiceForge → Reel Studio):** +VoiceForge stays audio-only. Reel Studio (or any client) calls HTTP: -| VoiceForge | Reel Studio `VoiceProvider` | -|------------|----------------------------| -| `GET /v1/voices` | `listVoices()` → map each to `{ id, name, category: "cloned", previewUrl }` | -| `GET /v1/voices/{id}/preview` | prefix with `VOICEFORGE_SERVICE_URL` for `previewUrl` | -| `GET /v1/engines` | `listModels()` → `{ id: engine.id, label: engine.label }` | -| `POST /v1/synthesize` JSON `{ voiceId, text, ... }` | `synth()` → parse WAV with `parseWav` from `@/lib/wav` | -| `POST /v1/voices` multipart | Clone UI only (not part of `VoiceProvider`) | - -Use `providerFetch()` from `src/providers/voice/http.ts` for consistent errors. -Set `maxConcurrency: 1` (CPU-bound local ML, same as `kokoro-server`). - -**Engine choice for cloning (surface in UI):** - -| `engine_id` | Tier | License | Notes | -|-------------|------|---------|-------| -| `openvoice-v2` | `instant` | MIT | Good default; fast zero-shot | -| `f5-tts` | `instant` | Apache/CC | Needs intelligible speech in reference | -| `xtts-v2` | `instant` | CPML | Non-commercial only | -| `rvc` | `high_fidelity` | MIT | GPU + long reference audio; training takes minutes | - ---- - -### Step 6 — SSE progress (clone UI) - -Subscribe while a voice is processing: - -```bash -curl -N -H "Authorization: Bearer $VOICEFORGE_API_TOKEN" \ - http://localhost:8089/v1/voices//events +```mermaid +flowchart LR + RS[Reel Studio
scripts · scenes · video] -->|HTTP| VF[VoiceForge API] + VF --> E[Selected local engine] + E --> WAV[WAV audio] ``` -Events: - -- `event: progress` — `data: {"type":"progress","message":"loading_model",...}` -- `event: status` — `data: {"status":"ready"}` or `"failed"` with error details +See **[docs/REEL_STUDIO.md](docs/REEL_STUDIO.md)**. VoiceForge remains usable with +curl, Studio, and other apps without Reel Studio. -In the browser, use `EventSource` (with a Next.js proxy if you need auth -headers — native `EventSource` cannot set `Authorization`). +## Security and self-hosting ---- +- Bearer auth when token configured; CORS allowlist; upload sniffing; path-safe + storage; job concurrency caps; non-root Docker; pinned deps +- Full guide: **[docs/SELF_HOSTING.md](docs/SELF_HOSTING.md)** +- Report vulnerabilities privately: **[SECURITY.md](SECURITY.md)** -### Step 7 — Integrate from any other app (minimal examples) +## Watermarking -**TypeScript (Node 18+):** +Optional (`VOICEFORGE_WATERMARK_ENABLED`, default **false**): mixes a quiet, +voice-id-seeded noise fingerprint into synth WAVs. Not forensic proof; not a +substitute for consent. **[docs/WATERMARKING.md](docs/WATERMARKING.md)** -```typescript -const base = process.env.VOICEFORGE_SERVICE_URL ?? "http://localhost:8089"; -const headers = { - Authorization: `Bearer ${process.env.VOICEFORGE_API_TOKEN ?? ""}`, -}; - -// List cloned voices -const voices = await fetch(`${base}/v1/voices`, { headers }).then((r) => r.json()); +## Architecture -// Synthesize -const wav = await fetch(`${base}/v1/synthesize`, { - method: "POST", - headers: { ...headers, "Content-Type": "application/json" }, - body: JSON.stringify({ voiceId: voices[0].id, text: "Hello world." }), -}).then((r) => r.arrayBuffer()); +Single FastAPI process, SQLite metadata, `data/` for samples/artifacts. Optional +engines run in isolated workers or a Fish Speech sidecar when dependency pins +conflict. + +```mermaid +flowchart TB + subgraph Clients + CLI[Studio / curl / custom apps] + end + + subgraph Service["VoiceForge process"] + API[FastAPI
voices · synth · engines · SSE · metrics] + REG[Engine registry] + JOBS[Background jobs] + API --> REG + API --> JOBS + end + + subgraph Storage + DB[(SQLite)] + DISK[data/voices · samples · artifacts] + end + + subgraph EngineLayer["Engines"] + INPROC[In-process
openvoice · f5 · xtts · qwen3] + WORKER[Isolated workers
rvc · chatterbox · cosyvoice · indextts] + SIDE[Fish Speech sidecar] + end + + CLI -->|REST + SSE| API + API --> DB + API --> DISK + REG --> INPROC + REG --> WORKER + REG --> SIDE ``` -**Python:** - -```python -import httpx, os - -base = os.environ.get("VOICEFORGE_SERVICE_URL", "http://localhost:8089") -token = os.environ.get("VOICEFORGE_API_TOKEN", "") -headers = {"Authorization": f"Bearer {token}"} if token else {} - -with httpx.Client(base_url=base, headers=headers, timeout=600.0) as client: - voices = client.get("/v1/voices").json() - wav = client.post("/v1/synthesize", json={ - "voiceId": voices[0]["id"], - "text": "Hello world.", - }).content - open("out.wav", "wb").write(wav) -``` - -JSON field names on the wire are **camelCase** (`voiceId`, `engineId`, -`errorMessage`) to match Reel Studio conventions. - ---- - -### Cursor agent prompt for Reel Studio (copy-paste) - -Open the **reel-studio** project in Cursor and start a new agent chat with: - ---- - -**Implement M6 — VoiceForge voice cloning provider** - -VoiceForge is running at `http://localhost:8089` (OpenAPI at `/docs`). Implement integration per `docs/voice-clone-service-BRIEF.md` §11 and the VoiceForge README section "Integrating with Reel Studio". - -**Tasks:** - -1. Add `"voiceforge"` to `PROVIDER_IDS` in `src/providers/voice/types.ts`. -2. Create `src/providers/voice/voiceforge.ts` implementing `VoiceProvider`: - - `id: "voiceforge"`, `label: "VoiceForge (local clone)"`, `runtime: "server"`, `keyless: true` when only `VOICEFORGE_SERVICE_URL` is set (use bearer token from `VOICEFORGE_API_TOKEN` when present). - - `isConfigured()` → `VOICEFORGE_SERVICE_URL` is set. - - `listModels()` → `GET {base}/v1/engines` → `{ id, label }[]`. - - `listVoices()` → `GET {base}/v1/voices`, filter `status === "ready"`, map to `VoiceSummary` with `category: "cloned"`, `previewUrl: {base}/v1/voices/{id}/preview`. - - `synth(opts)` → `POST {base}/v1/synthesize` with `{ voiceId, text, sampleRate, speed, language }`, parse WAV via `parseWav` like `cartesia.ts`. - - `maxConcurrency: 1`. - - Use `providerFetch()` from `./http`. -3. Register in `src/providers/voice/registry.ts`. -4. Add `VOICEFORGE_SERVICE_URL` and `VOICEFORGE_API_TOKEN` to `.env.example`. -5. Add a **Clone a voice** UI (new page or section on `/voices`): - - File upload + optional mic record (`MediaRecorder`). - - Engine dropdown from `/v1/engines`, tier `instant` vs `high_fidelity`, consent checkbox. - - Submit `POST /v1/voices` (prefer a Next.js API route proxy so the bearer token is not exposed to the browser). - - Show progress via SSE (`GET /v1/voices/{id}/events`) or polling. - - On `ready`, refresh voice list so the new voice appears under VoiceForge in the voice browser and voiceover panel. -6. Tests: extend `src/providers/voice/providers.test.ts` with mocked fetch for voiceforge (mirror cartesia/elevenlabs tests). - -**Do not** merge VoiceForge's docker-compose into reel-studio — two separate services linked by URL only. - -Reference existing providers: `kokoro-server.ts`, `cartesia.ts`, `elevenlabs.ts`. - ---- +SVG: [docs/architecture.svg](docs/architecture.svg) · [docs/clone-flow.svg](docs/clone-flow.svg) -### Troubleshooting +### `CloneEngine` interface -| Symptom | Fix | -|---------|-----| -| `503 Engine not ready` | Run `voiceforge-download`; first request downloads models. | -| `401 Missing bearer token` | Set matching `VOICEFORGE_API_TOKEN` on both services. | -| CORS error from browser | Use a Next.js proxy route, or set `VOICEFORGE_CORS_ORIGINS`. | -| `422 tier=instant` on RVC | RVC requires `tier=high_fidelity`. | -| Reel Studio can't reach service | Use `host.docker.internal:8089` if Reel Studio runs in Docker on Mac/Windows. | -| Slow synthesis on CPU | Expected; use GPU compose profile or pick lighter engines. | - ---- - -## GPU cloud deployment - -Host VoiceForge on free/low-cost GPU platforms for faster engines and a public -URL (e.g. for Reel Studio). **Your laptop only needs the Modal CLI or a -browser** — inference runs in the cloud. - -| Platform | Public URL | Reel Studio | Guide | -|----------|------------|-------------|-------| -| **Modal.com** | Yes (automatic) | Yes | [docs/deploy-modal.md](docs/deploy-modal.md) | -| **Lightning.ai** | Yes (port proxy) | Yes | [docs/deploy-lightning.md](docs/deploy-lightning.md) | -| **Kaggle** | No (notebooks only) | No | [docs/deploy-kaggle.md](docs/deploy-kaggle.md) | - -**Local Python / PATH issues** (Modal CLI, dev venv): [docs/local-python-setup.md](docs/local-python-setup.md) - -Quick Modal path (after [Python setup](docs/local-python-setup.md)): - -```bash -cd audio-cloning && source .venv/bin/activate # see docs/local-python-setup.md -pip install 'cbor2>=5.9.0' modal && modal setup -modal secret create voiceforge-secrets VOICEFORGE_API_TOKEN="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')" -modal deploy modal_app.py -modal run modal_app.py::download_models -# Open the URL printed by `modal deploy` — VoiceForge Studio + /docs -``` - -Point Reel Studio at the hosted URL: +```python +class CloneEngine(Protocol): + id: str + label: str + capabilities: CloneCapabilities -```bash -VOICEFORGE_SERVICE_URL=https://your-workspace--voiceforge-web.modal.run -VOICEFORGE_API_TOKEN= + def is_ready(self) -> bool: ... + async def create_voice(self, voice_id, sample_paths, tier, language, on_progress=None) -> VoiceArtifact: ... + async def synthesize(self, voice_id, artifact, text, opts) -> bytes: ... ``` -Each platform guide includes a **troubleshooting log** section for known issues -and workarounds — update those docs when you hit platform-specific failures. +Adding an engine = one file in `app/engines/` + one entry in +`app/engines/registry.py`. ---- +## Roadmap -## Deploy on Lightning AI / GPU clouds (summary) +See **[ROADMAP.md](ROADMAP.md)** (completed M0–M8 + next ideas). -Full step-by-step: **[docs/deploy-lightning.md](docs/deploy-lightning.md)**. +## Contributing -VoiceForge is a normal Docker/HTTP service — it fits session-based GPU -providers the same way as any FastAPI app. +See **[CONTRIBUTING.md](CONTRIBUTING.md)** and **[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)**. +Issue/PR templates live under `.github/`. -### Recommended setup on Lightning AI +## Licensing -1. **Create a GPU Studio / job** with an NVIDIA GPU and Docker (or a Python - environment with CUDA). -2. **Clone this repo** into the session (do not upload your local `data/` - folder — it may contain personal voice samples). -3. **Start with the GPU image** (or install deps + set `VOICEFORGE_DEVICE=cuda`): +- **Application code:** MIT (`LICENSE`) +- **Engines / weights:** vary — **[NOTICE.md](NOTICE.md)**, + **[docs/ENGINE_LICENSING.md](docs/ENGINE_LICENSING.md)** +- XTTS-v2 weights are **CPML non-commercial / research**; set `COQUI_TOS_AGREED=1` + for non-interactive download only after you accept those terms -```bash -cp .env.example .env -# Required when the session is reachable from the internet: -# VOICEFORGE_API_TOKEN= -# VOICEFORGE_CORS_ORIGINS=https://your-reel-studio-origin # if browser calls directly +## Documentation index -docker compose -f docker/docker-compose.yml --profile gpu up --build -d -# or: docker compose ... --profile gpu run --rm voiceforge-download -``` +| Doc | Topic | +|-----|-------| +| [FIRST_CLONE.md](docs/FIRST_CLONE.md) | First successful clone | +| [ENGINES.md](docs/ENGINES.md) | Support matrix | +| [ENGINE_LICENSING.md](docs/ENGINE_LICENSING.md) | Licence matrix | +| [RESPONSIBLE_USE.md](RESPONSIBLE_USE.md) / [CONSENT.md](docs/CONSENT.md) | Safety & consent | +| [WATERMARKING.md](docs/WATERMARKING.md) | Fingerprint limitations | +| [SELF_HOSTING.md](docs/SELF_HOSTING.md) | Deploy securely | +| [REEL_STUDIO.md](docs/REEL_STUDIO.md) | Client integration | +| [DEMO_CAPTURE.md](docs/DEMO_CAPTURE.md) | Screenshots & audio | +| [GITHUB_METADATA.md](docs/GITHUB_METADATA.md) | Topics / social preview | +| [deploy-modal.md](docs/deploy-modal.md) / [Lightning](docs/deploy-lightning.md) / [Kaggle](docs/deploy-kaggle.md) | GPU cloud | +| [local-python-setup.md](docs/local-python-setup.md) | Host Python / Modal CLI | -4. **Expose port `8089`** via the provider's public URL / proxy. -5. **Point Reel Studio** (or any client) at that URL: +## Development ```bash -VOICEFORGE_SERVICE_URL=https:// -VOICEFORGE_API_TOKEN= +python3.11 -m venv .venv && source .venv/bin/activate +pip install -e ".[dev]" +pytest +ruff check app tests scripts +pip-audit +docker compose -f docker/docker-compose.yml config ``` -6. **When the session ends**, treat `data/` as disposable (cloned voices and - SQLite live there). Re-download models on the next session or attach a - persistent volume to `/app/models` if the provider supports it. - -### Engine choice on free GPU hours - -| Goal | Engine | Why | -|------|--------|-----| -| Fastest / lightest | `openvoice-v2` | Works on CPU; still fine on GPU | -| Best zero-shot quality (permissive) | `f5-tts` | Happier on GPU | -| Multilingual zero-shot | `xtts-v2` | GPU recommended; **CPML non-commercial** | -| Highest fidelity | `rvc` + `high_fidelity` | Needs GPU + longer reference audio | - -### Session hygiene - -- Always set `VOICEFORGE_API_TOKEN` on public endpoints. -- Never commit `.env` or `data/`. -- Wipe or unmount voice data before sharing a Studio snapshot. - ---- - -## Reel Studio integration (summary) - -M6 code changes belong in the **reel-studio** repository. VoiceForge exposes -a stable HTTP API (documented above and at `/docs`). Follow the Cursor prompt -in the previous section to implement the provider and clone UI there. +Unit tests use a `FakeEngine` (no multi-GB downloads). Optional ML extras are +in `requirements-*.txt` / `pyproject.toml` optional deps. ## Repo layout ``` voiceforge/ -├── app/ -│ ├── main.py, config.py, db.py, db_models.py, schemas.py, security.py, storage.py -│ ├── api/ # voices.py, engines.py, synth.py, events.py, metrics.py -│ ├── engines/ # CloneEngine impls + registry (xtts, f5, openvoice, rvc, -│ │ # chatterbox, qwen3, fish, cosyvoice, indextts) -│ ├── static/ # Docker-hosted Studio UI (http://localhost:8089/) -│ └── jobs/ # background processing + SSE event bus -├── docs/ # architecture.svg, clone-flow.svg, deployment guides -│ ├── deploy-modal.md -│ ├── deploy-kaggle.md -│ ├── deploy-lightning.md -│ └── local-python-setup.md -├── docker/ # Dockerfile.cpu, Dockerfile.gpu, docker-compose.yml -├── scripts/ # download_models.py, *_worker.py, e2e_smoke_test.py +├── app/ # FastAPI, engines, Studio static UI +├── docs/ # Guides + assets +├── docker/ # CPU/GPU images + compose +├── scripts/ # download, workers, e2e smoke ├── tests/ -├── data/ # git-ignored: db.sqlite, voices/{id}/{samples,artifacts} -└── models/ # git-ignored: downloaded model checkpoints +├── data/ # gitignored runtime data +└── models/ # gitignored checkpoints ``` diff --git a/RESPONSIBLE_USE.md b/RESPONSIBLE_USE.md new file mode 100644 index 0000000..8d7e3ae --- /dev/null +++ b/RESPONSIBLE_USE.md @@ -0,0 +1,58 @@ +# Responsible use + +VoiceForge is a self-hosted voice cloning studio and API. Cloning technology +can be misused for impersonation, fraud, harassment, and deception. + +## Allowed use + +Clone **only**: + +- your own voice, or +- a voice for which you have **clear, informed, and valid permission** + +Operators (anyone running VoiceForge) are responsible for obtaining and +retaining evidence of that permission. The application does not verify identity +or legal authority. + +## Prohibited use + +Do **not** use VoiceForge to: + +- impersonate someone without permission +- commit fraud or social-engineering attacks +- harass, threaten, or coerce anyone +- bypass identity verification or voice-based authentication +- mislead people about the origin of generated audio +- create or distribute deceptive deepfakes of public figures or private individuals + +## Consent declaration (application control) + +Creating a voice requires `consent=true` (API) or the Studio consent checkbox. +That field records the caller’s **declaration**, not proof of ownership. + +See [docs/CONSENT.md](docs/CONSENT.md) for details and operator guidance. + +## Disclosure + +Where context requires it (publications, products, social media, customer +communications), disclose that speech was generated or cloned with AI. + +## Privacy and deletion + +- Voice samples and generated artifacts stay on infrastructure **you** choose + (workstation, private server, or GPU environment you control). +- Deleting a voice via `DELETE /v1/voices/{id}` removes the database row and the + on-disk voice directory (samples, artifacts, preview). +- Wipe Docker volumes / `data/` when retiring a deployment. + +## Watermarking + +Optional synth watermarking is a **weak, voice-specific fingerprint**. It is +**not** forensic proof, not a substitute for consent, and not guaranteed to +survive recompression or editing. See [docs/WATERMARKING.md](docs/WATERMARKING.md). + +## Security incidents + +If you discover a vulnerability or a voice-data incident related to this +project, report it privately — see [SECURITY.md](SECURITY.md). Do not attach +private voice samples or API tokens to public issues. diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..d5c106c --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,41 @@ +# Roadmap + +Milestones **M0–M8** described below are complete in this repository (M6 client +integration lives in apps such as Reel Studio). Future work is listed under +**Next**. + +## Project status (user-facing) + +**Early public release.** The local Studio, API, engine registry, Docker setup, +and supported cloning workflows are available. Engine installation, quality, and +hardware requirements vary. Some integrations still need broader hardware +testing (especially GPU paths). + +## Completed milestones + +- [x] **M0 — Scaffold:** FastAPI skeleton, `/healthz`, SQLite, Docker CPU image, + engine registry, README. +- [x] **M1 — MVP clone (XTTS-v2):** upload → instant zero-shot → `/synthesize` WAV. +- [x] **M2 — Quality benchmarking:** `scripts/benchmark_quality.py`, reference + preprocessing. +- [x] **M3a — F5-TTS engine:** permissive-license zero-shot behind `CloneEngine`. +- [x] **M3b — OpenVoice V2 engine:** MIT-oriented zero-shot path behind `CloneEngine`. +- [x] **M4 — High-fidelity tier:** RVC isolated worker, SSE progress, upgrade API. +- [x] **M5 — Docker hardening:** CPU/GPU images, model cache, e2e smoke + (CPU verified for openvoice-v2, xtts-v2, f5-tts). +- [x] **M6 — Reel Studio integration:** client-side `VoiceProvider` + clone UI + (in the client repo). +- [x] **M7 — Polish:** CI, structured logging, metrics, optional watermark, + OpenAPI polish, NOTICE/SECURITY. +- [x] **M8 — Additional engines:** Chatterbox, Qwen3-TTS, Fish Speech sidecar, + CosyVoice 3, IndexTTS2, architecture SVGs. + +## Next + +- [ ] Broader GPU hardware verification for RVC / heavy engines +- [ ] Optional consent metadata fields (source/note/intended use) if needed +- [ ] Cross-platform launcher (`./voiceforge start --cpu`) if demand warrants +- [ ] Additional watermark robustness tests (recompression) — documentation-only until measured +- [ ] Community-contributed engine adapters with licence review via issue template + +See also [docs/ENGINES.md](docs/ENGINES.md) for readiness labels. diff --git a/SECURITY.md b/SECURITY.md index ba32d26..aa9dcb9 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -2,27 +2,39 @@ ## Supported versions -This project is a local-first portfolio / self-hosted service. Security fixes -are applied on the `main` branch. +Security fixes are applied on the `main` branch of this self-hosted service. ## Reporting a vulnerability -Please **do not** open a public GitHub issue for security-sensitive reports. +**Do not** open a public GitHub issue for security-sensitive reports. -Email or privately message the repository maintainer with: +Contact the repository maintainer privately with: - A short description of the issue - Steps to reproduce - Impact (e.g. unauthenticated access, path traversal, model/pickle risk) +**Do not attach:** private voice samples, API tokens, unredacted databases, or +consent evidence containing personal data. + +## Voice data incidents + +Treat uploaded reference audio and generated speech as sensitive. If a +deployment leaks `data/` or tokens, rotate credentials, wipe affected voice +directories, and notify affected operators/users as appropriate. + ## Hardening checklist for public / GPU-cloud deployments -When exposing VoiceForge beyond localhost (VPS, Lightning AI, RunPod, etc.): +When exposing VoiceForge beyond localhost (VPS, Lightning AI, RunPod, Modal, etc.): 1. Set a strong `VOICEFORGE_API_TOKEN` and require `Authorization: Bearer …` 2. Restrict `VOICEFORGE_CORS_ORIGINS` to known frontends (or leave empty and proxy only from your backend) -3. Do not commit `.env`, `data/`, or `models/` -4. Prefer the Docker images (non-root user, healthcheck) -5. Treat uploaded reference audio as sensitive — wipe `data/voices/` when done -6. Remember XTTS-v2 weights are **non-commercial (CPML)** — see `NOTICE.md` +3. Terminate TLS at a reverse proxy +4. Do not commit `.env`, `data/`, or `models/` +5. Prefer the Docker images (non-root user, healthcheck, resource limits) +6. Wipe `data/voices/` when done or when permission ends +7. Remember model licences vary — especially XTTS-v2 **CPML non-commercial** — + see `NOTICE.md` and `docs/ENGINE_LICENSING.md` + +More: [docs/SELF_HOSTING.md](docs/SELF_HOSTING.md), [RESPONSIBLE_USE.md](RESPONSIBLE_USE.md). diff --git a/app/engines/asr.py b/app/engines/asr.py index d2167d5..c6f7370 100644 --- a/app/engines/asr.py +++ b/app/engines/asr.py @@ -114,6 +114,7 @@ def release_asr_model() -> None: if torch.cuda.is_available(): torch.cuda.empty_cache() except Exception: - pass + # Best-effort cleanup — free references even if CUDA teardown fails. + logger.debug("ASR CUDA cleanup failed", exc_info=True) _asr_processor = None _asr_model = None diff --git a/app/engines/chatterbox.py b/app/engines/chatterbox.py index 5a509e0..20fcde7 100644 --- a/app/engines/chatterbox.py +++ b/app/engines/chatterbox.py @@ -24,8 +24,8 @@ Tier, VoiceArtifact, ) -from app.engines.wav_output import to_wav_bytes from app.engines.chatterbox_daemon import synthesize_via_daemon +from app.engines.wav_output import to_wav_bytes from app.engines.worker_client import ping_worker from app.storage import artifacts_dir diff --git a/app/static/index.html b/app/static/index.html index 299c296..baf35d7 100644 --- a/app/static/index.html +++ b/app/static/index.html @@ -256,11 +256,11 @@

VoiceForge Studio

1. Clone

- + @@ -308,11 +308,20 @@

1. Clone

+ +

All supported engines are listed below. ready means - bundled in this deployment; not in this deployment means you - can clone with it elsewhere (Docker GPU) but not here. Failed voices - in the list are usually old attempts — delete them. + usable in this process; other labels mean extra install, GPU, or a + sidecar is required. Start with openvoice-v2 on CPU. + Failed voices in the list are usually old attempts — delete them.

@@ -346,7 +355,7 @@

Voices