fix: dub with a real translation, any TTS provider, and lines placed on time

[kiyeonjeon21]

Bug

vibe audio dub had three problems:

• No real translation: it called Claude's storyboard analysis (analyzeContent) and read each returned scene description as a translated line. The output was either unrelated text or, through a silent fallback, the untranslated source.
• Timing ignored: it concatenated the spoken lines one after another, ignoring when each was originally said.
• Too many keys: it required both ElevenLabs and Anthropic keys.

Fix

• translateTexts, extracted from translate-srt, translates in batches with Claude, or OpenAI when there's no Anthropic key. Replies are matched by their [N] index, so a skipped, reordered, or chatty reply cannot shift later lines; missing lines keep the source text. translate-srt uses it too, so both share one implementation over providerRequest.
• Any TTS provider: each line is spoken through tts-resolve (new --tts auto|elevenlabs|openai|kokoro, also on the audio_dub tool).
• Lines placed on time: each line is placed at its segment's start time with ffmpeg (adelay + amix), and sped up (at most 1.5x) when it would overrun its slot.
• Fewer keys: only OPENAI_API_KEY (Whisper) is required. Temp files go in a temp dir that is always removed.

Verification

• Tests: new translateTexts tests cover index matching with out-of-order, missing, and preamble lines, the OpenAI fallback, batching, and a missing key as auth.
• Checks: pnpm build, lint, typecheck, tests (CLI 1347 including integration), pre-push gate.
• Live: vibe audio dub long.mp3 -l ko --tts openai --json on a 7.5 s English clip with two lines:
  • Both lines translated correctly ("해 질 녘 종이배 한 척이 연못을 가로질러 떠간다").
  • Output 7.34 s.
  • Silence detection shows the second line starting right after a gap ending at 3.83 s, matching its source start of 3.76 s.
  • Transcribing the dub back gives the Korean text.

https://claude.ai/code/session_011S1BLTaFctZNQzGERpkCuA

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
vibeframe	Ready	Preview	Oct 5, 2026 8:14am UTC