feat(mcp): clone_voice tool — clone a new voice from reference audio (#1194) - #1195
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdds a ChangesVoice cloning via MCP
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant MCPClient
participant clone_voice
participant ProfilesAPI
MCPClient->>clone_voice: Send base64 reference audio and metadata
clone_voice->>clone_voice: Decode and validate audio
clone_voice->>ProfilesAPI: POST /profiles as multipart form data
ProfilesAPI-->>clone_voice: Return created profile
clone_voice-->>MCPClient: Return profile_id, name, and kind
Possibly related PRs
🚥 Pre-merge checks | ✅ 7 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (7 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
| Filename | Overview |
|---|---|
| backend/mcp_server.py | Adds the clone_voice MCP tool (~50 lines) following the base64-audio pattern of transcribe. Empty optional fields (ref_text, instruct) are unconditionally serialised into the POST body, inconsistent with the guarded field inclusion in every other tool in this file. |
| tests/test_mcp_mount.py | Updates the tool-surface assertion to include clone_voice; minimal and correct change. |
| CHANGELOG.md | Adds a ### Added changelog entry for clone_voice with correct attribution and issue reference. |
| docs/mcp.md | Updates the MCP tool table and intro sentence to include clone_voice; no issues. |
Sequence Diagram
%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Agent as AI Agent
participant MCP as clone_voice (MCP tool)
participant BE as POST /profiles
Agent->>MCP: name, ref_audio_base64, [ref_text, instruct, language]
MCP->>MCP: "len(base64) > 200 MB? → error"
MCP->>MCP: "base64.b64decode(validate=True) → raw"
MCP->>MCP: raw empty? → error
MCP->>BE: multipart (ref_audio + form data)
BE-->>MCP: "{id, name, kind}"
MCP-->>Agent: "JSON {profile_id, name, kind}"
Agent->>Agent: "generate_speech(profile_id=...)"
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Agent as AI Agent
participant MCP as clone_voice (MCP tool)
participant BE as POST /profiles
Agent->>MCP: name, ref_audio_base64, [ref_text, instruct, language]
MCP->>MCP: "len(base64) > 200 MB? → error"
MCP->>MCP: "base64.b64decode(validate=True) → raw"
MCP->>MCP: raw empty? → error
MCP->>BE: multipart (ref_audio + form data)
BE-->>MCP: "{id, name, kind}"
MCP-->>Agent: "JSON {profile_id, name, kind}"
Agent->>Agent: "generate_speech(profile_id=...)"
Reviews (3): Last reviewed commit: "docs(mcp): refresh module docstring tool..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| r = await _api_post_form( | ||
| "/profiles", | ||
| data={ | ||
| "name": name, | ||
| "kind": "clone", | ||
| "ref_text": ref_text, | ||
| "instruct": instruct, | ||
| "language": language, | ||
| }, | ||
| files={"ref_audio": ("ref_audio.wav", raw, "application/octet-stream")}, | ||
| ) | ||
| profile = r.json() | ||
| import json as _json | ||
| return _json.dumps({ | ||
| "profile_id": profile.get("id", "unknown"), | ||
| "name": profile.get("name", name), | ||
| "kind": profile.get("kind", "clone"), | ||
| }) |
There was a problem hiding this comment.
_api_post_form calls r.raise_for_status() (line 89), so any 4xx/5xx from /profiles — duplicate name, audio too short, backend validation failure — throws httpx.HTTPStatusError that is completely uncaught here. Voice cloning commonly fails at the quality-gate stage, so this is a very live failure mode. The calling agent gets a generic framework exception string instead of the structured {"error": "..."} JSON it expects. r.json() on line 300 is also unguarded; a non-JSON body (e.g. a proxy error page) throws JSONDecodeError.
| "instruct": instruct, | ||
| "language": language, | ||
| }, | ||
| files={"ref_audio": ("ref_audio.wav", raw, "application/octet-stream")}, |
There was a problem hiding this comment.
ref_audio.wav even when the caller sends MP3 or FLAC. Some backends (and the underlying libsndfile/ffmpeg dispatch) use the filename extension as a format hint before falling back to magic-byte detection. If the backend relies on the extension for its initial format sniff, a FLAC sent as ref_audio.wav can produce a silent decode error. Using a neutral name like ref_audio.bin (paired with application/octet-stream) makes the intent explicit.
| files={"ref_audio": ("ref_audio.wav", raw, "application/octet-stream")}, | |
| files={"ref_audio": ("ref_audio.bin", raw, "application/octet-stream")}, |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/mcp_server.py`:
- Around line 282-285: Update the ref_audio_base64 decoding logic in the
surrounding handler to remove an optional data-URI prefix before decoding, then
catch only the specific base64 decoding exception raised by base64.b64decode
instead of broad Exception. Preserve the existing invalid-input JSON error
response for decoding failures.
- Line 298: Update the multipart file construction in the relevant backend
request flow to detect the input audio format from raw magic bytes and choose
the matching filename extension instead of always using ref_audio.wav. Preserve
the existing upload content and field name, and support the MP3, FLAC, and WAV
formats documented by the surrounding API.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: c530bd26-e83a-49f8-8c16-5833f1d27ce9
📒 Files selected for processing (3)
CHANGELOG.mdbackend/mcp_server.pytests/test_mcp_mount.py
| try: | ||
| raw = base64.b64decode(ref_audio_base64, validate=True) | ||
| except Exception: | ||
| return '{"error":"ref_audio_base64 is not valid base64"}' |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Narrow the blind exception and strip data URIs.
Catching the blind Exception from base64 decoding masks other errors and is flagged by static analysis. Additionally, LLM agents frequently prepend data URIs (e.g., data:audio/wav;base64,...) when interacting with file-upload tools; stripping this prefix before decoding prevents a round-trip failure.
🛠️ Proposed fix
+ import binascii
+
+ if ref_audio_base64.startswith("data:"):
+ ref_audio_base64 = ref_audio_base64.split(",", 1)[-1]
+
try:
raw = base64.b64decode(ref_audio_base64, validate=True)
- except Exception:
+ except (binascii.Error, ValueError):
return '{"error":"ref_audio_base64 is not valid base64"}'📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| try: | |
| raw = base64.b64decode(ref_audio_base64, validate=True) | |
| except Exception: | |
| return '{"error":"ref_audio_base64 is not valid base64"}' | |
| import binascii | |
| if ref_audio_base64.startswith("data:"): | |
| ref_audio_base64 = ref_audio_base64.split(",", 1)[-1] | |
| try: | |
| raw = base64.b64decode(ref_audio_base64, validate=True) | |
| except (binascii.Error, ValueError): | |
| return '{"error":"ref_audio_base64 is not valid base64"}' |
🧰 Tools
🪛 Ruff (0.15.21)
[warning] 284-284: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/mcp_server.py` around lines 282 - 285, Update the ref_audio_base64
decoding logic in the surrounding handler to remove an optional data-URI prefix
before decoding, then catch only the specific base64 decoding exception raised
by base64.b64decode instead of broad Exception. Preserve the existing
invalid-input JSON error response for decoding failures.
Source: Linters/SAST tools
| "instruct": instruct, | ||
| "language": language, | ||
| }, | ||
| files={"ref_audio": ("ref_audio.wav", raw, "application/octet-stream")}, |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Sniff magic bytes to ensure correct file extensions for non-WAV inputs.
Mismatched container extensions (e.g., saving an MP3 bitstream as .wav) can break strict HTML5 <audio> playback on Safari when the UI fetches the profile reference, or cause failures in downstream ffmpeg pipelines that rely on extension hints. Since the docstring explicitly invites agents to send MP3 or FLAC, sniff the magic bytes to provide the correct file extension to the backend form.
🎵 Proposed fix
+ ext = ".wav"
+ if raw.startswith(b"fLaC"):
+ ext = ".flac"
+ elif raw.startswith(b"ID3") or raw.startswith(b"\xff\xfb") or raw.startswith(b"\xff\xf3"):
+ ext = ".mp3"
+ elif raw.startswith(b"OggS"):
+ ext = ".ogg"
+ elif raw[4:8] == b"ftyp":
+ ext = ".m4a"
+
- files={"ref_audio": ("ref_audio.wav", raw, "application/octet-stream")},
+ files={"ref_audio": (f"ref_audio{ext}", raw, "application/octet-stream")},📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| files={"ref_audio": ("ref_audio.wav", raw, "application/octet-stream")}, | |
| ext = ".wav" | |
| if raw.startswith(b"fLaC"): | |
| ext = ".flac" | |
| elif raw.startswith(b"ID3") or raw.startswith(b"\xff\xfb") or raw.startswith(b"\xff\xf3"): | |
| ext = ".mp3" | |
| elif raw.startswith(b"OggS"): | |
| ext = ".ogg" | |
| elif raw[4:8] == b"ftyp": | |
| ext = ".m4a" | |
| files={"ref_audio": (f"ref_audio{ext}", raw, "application/octet-stream")}, |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/mcp_server.py` at line 298, Update the multipart file construction in
the relevant backend request flow to detect the input audio format from raw
magic bytes and choose the matching filename extension instead of always using
ref_audio.wav. Preserve the existing upload content and field name, and support
the MP3, FLAC, and WAV formats documented by the surrounding API.
…ebpalash#1194) AI agents driving OmniVoice via MCP could use and list voices but couldn't create one. Add a clone_voice MCP tool that takes a base64-encoded reference audio sample (consistent with transcribe's audio_base64 pattern), decodes it, and POSTs it as a multipart ref_audio to POST /profiles (kind=clone). Returns the new profile_id so the agent can immediately use it with generate_speech. Update test_mcp_mount.py to include clone_voice in the asserted tool surface. CHANGELOG entry.
4f40ad9 to
9b49a3b
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
docs/mcp.md (1)
14-14: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winDocument the consent boundary for voice cloning.
The row makes cloning immediately actionable but does not tell users to obtain the speaker’s explicit permission. Add a concise warning that
clone_voicemust only be used with consent-verified reference audio.Based on learnings, prefer a consent-verified voice profile for any agent that speaks as a person.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/mcp.md` at line 14, Update the clone_voice entry in the MCP documentation to add a concise warning that cloning requires the speaker’s explicit consent and consent-verified reference audio. Also state the preference for consent-verified voice profiles when an agent speaks as a person.Source: Learnings
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@docs/mcp.md`:
- Line 14: Update the clone_voice entry in the MCP documentation to add a
concise warning that cloning requires the speaker’s explicit consent and
consent-verified reference audio. Also state the preference for consent-verified
voice profiles when an agent speaks as a person.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 19a5056f-c838-4d53-8e77-d3b3b5d01bf9
📒 Files selected for processing (4)
CHANGELOG.mdbackend/mcp_server.pydocs/mcp.mdtests/test_mcp_mount.py
🚧 Files skipped from review as they are similar to previous changes (2)
- tests/test_mcp_mount.py
- CHANGELOG.md
…changelog Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… + reviewer configs Harvested and verified every CodeRabbit/Greptile finding from PRs debpalash#1175, debpalash#1189, debpalash#1192, debpalash#1195: 16 real ones fixed (fallback ASR preflight bypass, VRAM release on stream exit, typed 409 parity, uv env independence, path-privacy in errors, MCP clone_voice hardening, CaptureWidget WS guard, test hygiene), 4 refuted with evidence, rest documented as deliberate design or deferred. Deterministic CI replaces hand-enforcement: tests/test_changelog_style.py (quiet one-liner format) and tests/test_locale_parity.py (21-locale key/placeholder lockstep with a ratchet baseline) — the latter surfaced and fixes 151 already-broken locale strings. CodeRabbit/Greptile carry the house rules via .coderabbit.yaml + greptile.json; CLAUDE.md gains the harvest-before-merge and never-accept-as-is rules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
Closes #1194.
What
Adds a
clone_voiceMCP tool so AI agents can create a voice profile from a reference audio sample — not just use existing ones. The new tool:name+ref_audio_base64(base64-encoded WAV/MP3/FLAC, 5-30s clean speech) + optionalref_text/instruct/language.ref_audiofile to the existingPOST /profilesendpoint (kind=clone).{profile_id, name, kind}so the agent can immediately use the voice withgenerate_speech.Consistent with the existing
transcribetool's base64-audio pattern (validated decode + 200 MB cap +application/octet-streamcontent-type).Review
Self-reviewed against the
transcribetool (same base64-audio transport): caught and fixed three robustness gaps — missing base64 decode error handling, missing size cap, and wrong content-type (hardcodedaudio/wavinstead ofapplication/octet-stream). The full bot review (CodeRabbit/Greptile) will run on push.Files
backend/mcp_server.py— one new@mcp.tool()(~50 lines, follows thegenerate_speechpattern).tests/test_mcp_mount.py— addedclone_voiceto the asserted tool surface.CHANGELOG.md—### Addedentry.Summary
Adds a
clone_voiceMCP tool that lets agents create voice profiles from base64-encoded reference audio (WAV/MP3/FLAC).clone_voice(name, ref_audio_base64, ref_text="", instruct="", language="Auto").POST /profilesas a multipartref_audiofile withkind=cloneand content typeapplication/octet-stream, along with optionalref_text,instruct, andlanguage.{profile_id, name, kind}) as a JSON string for immediate use withgenerate_speech.docs/mcp.md; also adds aCHANGELOG.mdentry.