🎬 Audio-to-Video Synchronization Pipeline

An automated, modular pipeline that transforms an audio track, lyrics, and a set of images into a beat-synchronized, lyric-aligned video with dynamic, word-by-word karaoke animated subtitles.

---

📑 Table of Contents

1. System Architecture
2. Directory Structure
3. Master Orchestrator (pipeline.py)
4. Module 1: Lyric Forced Alignment (modules/aligner.py)
5. Module 2: Beat & Rhythm Detector (modules/beat_detector.py)
6. Module 3: Timeline & Motion Planner (modules/timeline_planner.py)
7. Module 4: Karaoke Subtitle Generator (modules/subtitles.py)
8. Module 5: Video Assembly Engine (modules/video_engine.py)
9. Common Recipes & Customization

---

🏗 System Architecture

                       ┌─────────────────────────┐
                       │   Input Audio & Lyrics   │
                       └────────────┬────────────┘
                                    │
             ┌──────────────────────┴──────────────────────┐
             ▼                                             ▼
  [Stage 1: Aligner (Whisper)]                [Stage 2: Beat Detector (Librosa)]
     Extracts word & line                         Extracts tempo (BPM), beats,
     millisecond timestamps                       downbeats & energy onsets
             │                                             │
             └──────────────────────┬──────────────────────┘
                                    │
                                    ▼
                     [Stage 3: Timeline Planner]
                     • Merges beats + lyrics
                     • Maps 180 images naturally
                     • Assigns Ken Burns pan/zoom
                     • Sets beat-synced transitions
                                    │
             ┌──────────────────────┴──────────────────────┐
             ▼                                             ▼
  [Stage 4: Subtitle Generator]               [Stage 5: FFmpeg Video Engine]
   Produces animated .ASS with                • Animates images (Ken Burns)
   word-by-word \k karaoke tags               • Applies transitions (xfade)
             │                                • Burns in styled subtitles
             └──────────────────────┬──────────────────────┘
                                    │
                                    ▼
                       ┌─────────────────────────┐
                       │    Final Output Video    │
                       │    (assets/output/*.mp4) │
                       └─────────────────────────┘

---

📁 Directory Structure

ImageToVideo/
│
├── assets/                       # 📥 Main workspace directory for data
│   ├── audio.mp3                 # Input music / audio track
│   ├── lyrics.txt                # Raw lyrics text
│   ├── images/                   # Directory containing image sequence (.png, .jpg, .webp)
│   │
│   └── output/                   # 📤 Generated outputs & intermediate files
│       ├── alignments.json       # Stage 1: Word & line timestamps
│       ├── beats.json            # Stage 2: Beat grid, downbeats & BPM
│       ├── timeline.json         # Stage 3: Frame cuts, motions & transitions
│       ├── lyrics.ass            # Stage 4: Word-highlighting karaoke subtitles
│       └── final_video.mp4       # Stage 5: Synchronized final MP4 video
│
├── modules/                      # Independent processing modules
│   ├── __init__.py
│   ├── aligner.py                # Module 1
│   ├── beat_detector.py          # Module 2
│   ├── timeline_planner.py       # Module 3
│   ├── subtitles.py              # Module 4
│   └── video_engine.py           # Module 5
│
├── pipeline.py                   # Master End-to-End Orchestrator
├── requirements.txt              # Dependencies specification
└── README.md                     # Documentation

---

🚀 Master Orchestrator (pipeline.py)

Commands:

1. Quick Zero-Config Command (Auto-discovers from assets/):

python pipeline.py

2. Full Explicit Standalone Command (All options specified):

python pipeline.py --audio assets/audio.mp3 --lyrics assets/lyrics.txt --images assets/images --output assets/output/final_video.mp4 --outdir assets/output --mode hybrid --style karaoke --res 1080x1920 --fps 30 --model base --lang en

--res 1080x1920 : makes portrait video --res 1920x1080 : makes landscape video

Multi-line format (PowerShell on Windows):

python pipeline.py `
  --audio assets/audio.mp3 `
  --lyrics assets/lyrics.txt `
  --images assets/images `
  --output assets/output/final_video.mp4 `
  --outdir assets/output `
  --mode hybrid `
  --style karaoke `
  --res 1080x1920 `
  --fps 30 `
  --model base `
  --lang en

Argument	Flag	Type	Default Value	Description
--audio	-a	str	assets/audio.mp3	Path to soundtrack audio file (.mp3, .wav)
--images	-i	str	assets/images	Directory containing image files
--lyrics	-l	str	assets/lyrics.txt	Path to text file containing lyrics (optional)
--output	-o	str	assets/output/final_video.mp4	Output video destination file path
--outdir	-d	str	assets/output	Folder for intermediate JSON & subtitle files
--mode	-m	str	hybrid	Timing sync mode: hybrid, beat_sync, or lyric_sync
--style	-s	str	karaoke	Subtitle style: karaoke, neon, clean, or none
--res		str	1080x1920	Output resolution WIDTHxHEIGHT (e.g. 1080x1920 or 1920x1080)
--fps		int	30	Video frame rate in frames per second
--model		str	base	Whisper model: tiny, base, small, medium, large
--lang		str	None	Language code (e.g., en, es, hi) or auto-detect

---

🧩 Module 1: Lyric Forced Alignment (modules/aligner.py)

Aligns the audio track with the lyrics text to produce exact millisecond timestamps for each line and each word.

Standalone Command:

python -m modules.aligner --audio assets/audio.mp3 --lyrics assets/lyrics.txt --output assets/output/alignments.json

Arguments & Defaults:

• --audio, -a: Path to input audio (Default: assets/audio.mp3)
• --lyrics, -l: Path to lyrics text (Default: assets/lyrics.txt)
• --output, -o: Output JSON file (Default: assets/output/alignments.json)
• --model, -m: Whisper model size (tiny, base, small, medium, large) (Default: base)
• --lang: Optional language code (e.g., en, es, hi)

Output Schema (alignments.json):

{
  "audio_path": "D:/repos/ImageToVideo/assets/audio.mp3",
  "language": "en",
  "total_segments": 12,
  "segments": [
    {
      "id": 0,
      "start": 1.24,
      "end": 4.86,
      "text": "Singing the morning prayers",
      "words": [
        { "word": "Singing", "start": 1.24, "end": 1.78, "probability": 0.98 },
        { "word": "the", "start": 1.82, "end": 2.05, "probability": 0.99 },
        { "word": "morning", "start": 2.10, "end": 2.85, "probability": 0.97 },
        { "word": "prayers", "start": 2.92, "end": 4.86, "probability": 0.96 }
      ]
    }
  ]
}

---

🥁 Module 2: Beat & Rhythm Detector (modules/beat_detector.py)

Analyzes audio digital signal processing (DSP) using librosa to compute tempo (BPM), beat frames, downbeats (bars), and onset energy peaks.

Standalone Command:

python -m modules.beat_detector --audio assets/audio.mp3 --output assets/output/beats.json

Arguments & Defaults:

• --audio, -a: Path to input audio (Default: assets/audio.mp3)
• --output, -o: Output JSON file (Default: assets/output/beats.json)
• --bar, -b: Number of beats per bar/measure for downbeat grouping (Default: 4)

Output Schema (beats.json):

{
  "audio_path": "D:/repos/ImageToVideo/assets/audio.mp3",
  "duration": 184.25,
  "tempo_bpm": 128.0,
  "total_beats": 384,
  "total_onsets": 512,
  "beats_per_bar": 4,
  "beat_times": [0.46, 0.93, 1.40, 1.87, 2.34, 2.81, 3.28],
  "downbeats": [0.46, 2.34, 4.21, 6.09],
  "onset_times": [0.12, 0.46, 0.93, 1.20, 1.40]
}

---

📐 Module 3: Timeline & Motion Planner (modules/timeline_planner.py)

Merges the images with the beat timestamps and lyric cues. Sorts images using natural alphanumeric ordering (0a_cell_01.png, 1_cell_01.png, 2_cell_01.png, 10_cell_01.png) and schedules image switches, camera pan/zoom directions, and transition overlaps.

Standalone Command:

python -m modules.timeline_planner --images assets/images --beats assets/output/beats.json --lyrics assets/output/alignments.json --output assets/output/timeline.json

Arguments & Defaults:

• --images, -i: Directory containing images (Default: assets/images)
• --beats, -b: Path to beats.json (Default: assets/output/beats.json)
• --lyrics, -l: Path to alignments.json (Optional)
• --output, -o: Output timeline JSON (Default: assets/output/timeline.json)
• --mode, -m: Sync mode: hybrid (recommended), beat_sync, or lyric_sync (Default: hybrid)
• --transition, -t: Overlap transition length in seconds (Default: 0.5)
• --beats-per-cut: Number of beats per image in beat_sync mode (Default: 4)

Timing Modes:

• hybrid (Default): Cuts image on major lyric sentence starts snapped to the nearest musical beat, and fills instrumental gaps with beat-spaced image cuts.
• beat_sync: Cuts strictly on musical downbeats/bars (e.g. every 4 beats or 8 beats).
• lyric_sync: Cuts only when a new lyric segment begins.

Output Schema (timeline.json):

{
  "total_duration": 184.25,
  "mode": "hybrid",
  "clip_count": 46,
  "clips": [
    {
      "index": 0,
      "image_path": "D:/repos/ImageToVideo/assets/images/0a_cell_01.png",
      "start_time": 0.0,
      "end_time": 4.21,
      "duration": 4.21,
      "motion": "zoom_in",
      "transition": "none",
      "transition_duration": 0.0
    },
    {
      "index": 1,
      "image_path": "D:/repos/ImageToVideo/assets/images/0a_cell_02.png",
      "start_time": 4.21,
      "end_time": 8.15,
      "duration": 3.94,
      "motion": "pan_left",
      "transition": "dissolve",
      "transition_duration": 0.5
    }
  ]
}

---

🎤 Module 4: Karaoke Subtitle Generator (modules/subtitles.py)

Converts word timestamps from alignments.json into an Advanced SubStation Alpha (.ass) or .srt subtitle file. Uses \k<centiseconds> tags for real-time word highlighting and styling.

Standalone Command:

python -m modules.subtitles --alignments assets/output/alignments.json --output assets/output/lyrics.ass --style karaoke

Arguments & Defaults:

• --alignments, -a: Path to alignments JSON (Default: assets/output/alignments.json)
• --output, -o: Subtitle output file (Default: assets/output/lyrics.ass)
• --format, -f: ass (supports karaoke word highlight) or srt (Default: ass)
• --style, -s: Visual preset: karaoke, neon, clean (Default: karaoke)
• --font: Font family name (Default: Arial)
• --font-size: Font size in pixels (Default: 56)
• --res-x: Video width (Default: 1080)
• --res-y: Video height (Default: 1920)

Visual Styles:

• karaoke: White text with glowing golden-yellow highlighting as each word is sung.
• neon: White text with vivid cyan/electric glow highlighting.
• clean: Clean subtitle captions with crisp outline and soft drop-shadow.

---

🎥 Module 5: Video Assembly Engine (modules/video_engine.py)

Takes the timeline.json, audio, and subtitle file, scales images to target resolution, applies motion and transitions, burns in the subtitles, and muxes high-quality audio using FFmpeg.

Standalone Command:

python -m modules.video_engine --timeline assets/output/timeline.json --audio assets/audio.mp3 --subtitles assets/output/lyrics.ass --output assets/output/final_video.mp4

Arguments & Defaults:

• --timeline, -t: Path to timeline.json (Default: assets/output/timeline.json)
• --audio, -a: Path to audio file (Default: assets/audio.mp3)
• --subtitles, -s: Path to subtitle file (Default: assets/output/lyrics.ass)
• --output, -o: Output MP4 file path (Default: assets/output/final_video.mp4)
• --width: Output width in pixels (Default: 1080)
• --height: Output height in pixels (Default: 1920)
• --fps: Output framerate (Default: 30)

---

💡 Common Recipes & Customization

1. Vertical 9:16 (Instagram Reels / YouTube Shorts / TikTok)

python pipeline.py --res 1080x1920 --style karaoke

2. Landscape 16:9 (Standard YouTube Videos)

python pipeline.py --res 1920x1080 --style karaoke

3. Fast Downbeat Cuts for High-Energy Music

python pipeline.py --mode beat_sync --beats-per-cut 2

4. Video Without Subtitles

python pipeline.py --style none

5. High-Accuracy Singing Vocal Transcription

python pipeline.py --model medium --lang en