Skip to content
0xjgrPublic

About

Python pipeline that turns a plain-text script into an edited video: voice, images, subtitles and assembly

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Garabato

Versión en español

You write the script, out comes the edited video.

I built this so I wouldn't have to edit every video for a faceless YouTube channel by hand: narrated explainer videos on psychology and behavior, with stick-figure illustrations and burned-in subtitles. I write the script and the pipeline does the rest: voice, images, subtitles and final assembly.

Garabato is Spanish for "doodle".

From script to frame

Each sentence in the script is preceded by a line of fields describing its image. The fields are a closed vocabulary, so the same line always produces the same frame. Those two shot types are drawn by the repo itself with PIL, no GPU and no model, when CLIP_SOURCE is comfyui.

The idea was to fix the real bottleneck. Recording the voice and finding or drawing an image for every sentence is what makes an eight-minute video cost a whole day. Everything else (script, judgment, review) is still mine, because that's where the value is.

How it works

A plain-text script goes in through input/script.txt and a final_video.mp4 comes out:

input/script.txt
  ↓ parse_script.py            segments.json
  ↓ generate_voice.py          narration.mp3 + beat_timings.json
  ↓ generate_image_prompts.py  image_prompts.json
  ↓ generate_clips.py          clips/seg_NNN.mp4
  ↓ generate_subtitles.py      subtitles.srt + subtitles.ass
  ↓ assemble_video.py          final_video.mp4
  ↓ check_audio.py             validation report
python run_pipeline.py [slug]

Every step is idempotent: if you run it again it skips what's already done, so you can stop halfway and resume, or redo just one stretch.

That list is the execution order, but it leaves out the two decisions that actually hold the pipeline together:

Architecture: the single clock and the image fork

The clock. generate_voice.py concatenates the audio with no gaps and writes the real start and end of each beat to beat_timings.json. Images and subtitles don't keep time on their own: both read that file. That's why they never drift from the audio, and why a video generated before that file existed can't be synced without redoing it.

The fork. Where each image comes from isn't decided when generating the clips but earlier, when building the prompts (generate_image_prompts.py:750). With gemini or manual, a natural-language prompt is sent for every beat, and the model also illustrates the title cards and diagrams. With comfyui, those two types get an empty prompt and PIL draws them, and only scene shots go through the closed catalog. Same script, two different paths: that's why title cards look different depending on the backend.

The script format

The script isn't loose prose. Each narration sentence is preceded by a line of fields in parentheses that describes its image:

TITLE: Why Habits Change Everything

NARRATION:
[SEGMENT 1]
(shot: medium | char: 1, worried | pose: arms-crossed | prop: red-X right | bg: plain-white)
Every time you try to change, your brain treats it like a threat.

(shot: text-frame | text: 'STATUS QUO BIAS')
Scientists call this the status quo bias.

[SEGMENT 2]
(shot: diagram | left: question-mark | right: lightbulb | arrow: right yellow | bg: plain-white | desc: a question mark on the left turning into a glowing lightbulb on the right)
A simple question turns into a real discovery.

The fields are a closed vocabulary, not free text. scripts/_catalog.py is the source of truth and prompts/beat_format.md the reference for writing.

This started out as free prose, and that was the project's biggest mistake. A fast diffusion model pays attention to three or four elements per image and that's it: give it a paragraph and it picks which ones. With closed fields the prompt is built with a dictionary lookup, comes out the same twice in a row and can be validated before spending a single generation.

The desc: field is the deliberate exception: a literal description of what's in the frame, a single frozen moment. It's what makes the image illustrate the sentence instead of settling for a generic pose. Without desc:, the beat falls back to the catalog, so older scripts keep working.

Before generating anything:

python scripts/lint_script.py [slug]

The linter warns but doesn't block: it checks that poses are in the catalog, that text: fields don't exceed four words and that the mix of shot types resembles the one that works.

Requirements

Piece Note
Python 3.10 or later
FFmpeg with h264_nvenc. The binary I use ships without libx264
Voice Chatterbox, local, on GPU
Images Gemini 2.5 Flash Image via API, or ComfyUI with Flux Schnell locally
Subtitles stable-ts, which downloads the Whisper base model the first time
Fonts None needed. It uses whatever you have; see Fonts
pip install -r requirements.txt

All configuration lives in config.py: voice, style, paths, image backend. There are no settings scattered across the scripts.

The API key

Never in a versioned file. Either the GEMINI_API_KEY environment variable, or a secrets_local.py in the root (it's in .gitignore):

GEMINI_API_KEY = "..."

config.py checks the environment first and then the file.

Fonts

Nothing to install. config.py searches by file name in the Linux, macOS and Windows font directories, and ends with a generic bold (DejaVu Sans, Liberation Sans, Noto Sans, Arial) that exists on any machine. If it finds none, the renderers fall back to PIL's default font instead of crashing.

To use your own, either:

  1. Drop the .ttf into assets/fonts/. Whatever is there wins over everything else, with no install and no code changes. That folder is in .gitignore.
  2. Put the file name, without extension, at the start of FONT_PATHS_BOLD or FONT_PATHS_REGULAR in config.py.

The handwritten fonts at the top of the list (Caveat, Architects Daughter, Patrick Hand) are optional: if you ever install them, they're picked up automatically and the text fits the stick figure better.

Subtitles are separate, because the one painting them when burning them in is libass, which resolves by family name, not by path: that's SUBTITLE_STYLE["font"].

The three image backends

CLIP_SOURCE in config.py picks one of three paths:

  • "manual": scripts/export_prompts.py writes you the list of prompts with the exact file name for each one. You generate them by hand on the web, save them as clips/seg_NNN_src.png and the pipeline builds the clips from whatever it finds. It's free and can be done in batches.
  • "gemini": the same, but automated via API. It needs billing enabled; the image API has no free tier. It comes to about three euros per video.
  • "comfyui": ComfyUI with Flux Schnell locally. It's the original backend and I keep it as a fallback.

To keep the stick figure the same in every shot, assets/character_ref.png is passed as a reference in every image with a character. That model sheet is generated once with the prompt in assets/character_ref_prompt.txt.

Structure

config.py                     All configuration
run_pipeline.py               Orchestrator for the seven steps
│
├── scripts/
│   ├── _catalog.py           Closed vocabulary: field → prompt fragment
│   ├── parse_script.py       script.txt → segments.json
│   ├── generate_voice.py     Chatterbox → narration.mp3
│   ├── generate_image_prompts.py  Audio timing + prompt assembly
│   ├── generate_clips.py     Dispatches to the right backend
│   ├── _gemini.py            Gemini backend with retries and throttling
│   ├── _comfyui.py           Batched ComfyUI backend + fades
│   ├── _text_frame.py        PIL rendering of title cards
│   ├── _diagram_frame.py     PIL rendering of diagrams
│   ├── _sprites.py           Icon compositing over the image
│   ├── generate_subtitles.py Forced alignment → .srt + .ass
│   ├── assemble_video.py     Concatenation, audio, music and burn-in
│   ├── check_audio.py        Validates LUFS, duration and words
│   └── lint_script.py        Script validator, non-blocking
│
├── prompts/
│   ├── script_template.md    Instructions for writing a full script
│   ├── beat_format.md        Field and catalog specification
│   └── generate.md           Script generation prompt
│
├── tools/
│   └── make_readme_figures.py  Regenerates docs/img/ with the repo's renderers
│
├── docs/
│   ├── decisions.md          Why each thing is the way it is
│   └── img/                  README figures
│
├── assets/
│   ├── sprites/              PNG icons with alpha
│   ├── fonts/                Your font, if you want one (not versioned)
│   └── character_ref.png     The stick figure's model sheet
│
└── output/<slug>/            Everything the pipeline generates

The figures in this README aren't hand-made screenshots: they come from python tools/make_readme_figures.py. The frames figure calls _text_frame.py and _diagram_frame.py, the same modules the pipeline uses, so if you change the style you regenerate it and the README stops lying. The diagram is drawn by the same script, in two versions for GitHub's light and dark themes.

Decisions you can't see by reading the code

They're in docs/decisions.md: why the negative prompt is inert and writing one is pointless, why the seed is computed per segment and not per global index, why I removed the Ken Burns effect, why the character description is injected automatically instead of written in the script.

Known issues

The encoder. My FFmpeg is built without libx264, so everything goes through h264_nvenc. It's already set in _comfyui.py and assemble_video.py, but if someone takes this to another machine it's the first thing to check.

The GPU crashes with ComfyUI. Generating images locally at full power took my card down. Two things fixed it almost entirely: lowering the power limit to 80-85%, which costs 10% speed and drops 10 to 15 degrees, and closing the browser before running the pipeline, because hardware acceleration competes for VRAM. If it still crashes you need a full reboot; restarting the driver doesn't bring it back. This doesn't happen with the Gemini backend.

Encoding on Windows. You need to set PYTHONIOENCODING=utf-8 and PYTHONUTF8=1 or the console blows up on accented characters. And if you write input/script.txt from PowerShell with Set-Content -Encoding utf8 it adds a BOM that parse_script.py can't handle.

Videos from before the audio fix. Videos generated before the voice concatenation became gapless have longer narration and no beat_timings.json, so images and subtitles drift from the audio. To fix one: delete narration.mp3, narration_processed.mp3, image_prompts.json and the subtitles.* files, and run it again. Per-beat audio is kept and reused.

Status

The pipeline works end to end. What's solved:

  • Structured beat format, closed catalogs and a linter.
  • Gapless voice between beats, with beat_timings.json as the single timeline for images and subtitles. I measured 0.05 seconds of drift over 83 beats.
  • Character-aware prompts: the stick figure only appears when the beat asks for it, and object shots carry an explicit instruction that nobody should appear.
  • Karaoke-style subtitles, split at beat boundaries so the text never overlaps an image change, with the bottom fifth of the frame kept clear so nothing sits behind them.
  • Audio post-processing with EQ and normalization to -14 LUFS.

What's missing or half-done:

  • The voice is Chatterbox's stock one. I still need to record or clone my own and point CHATTERBOX_REF_AUDIO at it.
  • Background music (BACKGROUND_MUSIC_PATH) isn't set.
  • The LoRA training in tools/lora_training/ became obsolete when I switched to generating images with a character reference. I'm keeping it because the dataset prep code and the experiments are useful, but it's no longer part of the pipeline.

Constraints I don't touch

  • TTS_SPEED = 1.0. Time-stretching below 0.9x sounds robotic.
  • h264_nvenc for all encoding.
  • No web UI and no database. Files and the command line.
  • With ComfyUI, Flux Schnell runs at cfg=1.0 with the euler sampler. At cfg=1.0 the negative prompt does nothing, so quality is controlled only by what's asked for in the positive prompt.

License

MIT, in LICENSE. The models each backend uses come with their own licenses.

About

Python pipeline that turns a plain-text script into an edited video: voice, images, subtitles and assembly

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages