video-gen — Prompt → Animation (Copilot CLI plugin)

Note

Unofficial Plugin: This is an unofficial community plugin and is not affiliated with, sponsored by, or endorsed by GitHub.

Turn a natural-language prompt into a runnable video/animation project, with the plugin automatically picking the right tool for the use case.

It ships seven engines across three categories and a deterministic recommender, so "a rotating 3D globe" scaffolds Three.js, "render a 10-second MP4 from props" scaffolds Remotion, and "animate the proof of a² + b² = c²" scaffolds Manim — each as a working starter you can preview or render.

Once a project exists, the /render-video skill exports it to a real .mp4 (any tool — web ones via headless capture) and can add a local-model voiceover.

The toolbox

Tool	Category	Output	Best for
Motion (Framer Motion)	Modern Web	Live web	React/JS UI transitions, layout morph, gestures
GSAP (GreenSock)	Modern Web	Live web	Complex timelines, SVG morphing, scroll effects
Three.js	Modern Web	Live 3D	3D/WebGL scenes, particles, cameras, shaders
Motion Canvas	Video / MoGraph	Video + live preview	Programmatic motion graphics synced to audio
Remotion	Video / MoGraph	MP4 / WebM / GIF file	Real video files from React, data-driven
Manim	Educational / Math	MP4 file	Math/geometry explainers, equations (LaTeX)
p5.js / Processing	Educational / Creative	Live sketch	Creative coding, generative art, teaching

Install

Install directly from this repo with the Copilot CLI plugin command:

/plugin install abhi-singhs/copilot-video-gen-plugin

Then start a new session (the plugin announces itself on session start). Verify with /plugin list.

Usage

Slash command:

/video-gen a neon logo that bounces in and morphs into the word "LAUNCH"

Or just ask naturally — the video-gen skill triggers on requests to make an animation, video, motion graphic, 3D scene, explainer, or creative-coding sketch:

"make me a 3D galaxy of particles I can orbit" "animate a bar chart racing to a finish line and export an mp4" "build a Perlin-noise flow field to teach beginners about loops"

What happens

1. The prompt is scored against the tool registry to recommend a tool (scripts/recommend.mjs). If it's a close call, you're asked to confirm.
2. The plugin reads the matching reference guide and scaffolds a runnable starter (scripts/scaffold.sh).
3. The starter is customized to your prompt, then previewed (web) or the render command is provided (Remotion/Manim/Motion Canvas).
4. Render & narrate with the /render-video skill — export any tool to an .mp4 and optionally add a local voiceover (see below).

Render to a file & add a voiceover

After generating, the render-video skill (/render-video) turns the project into a real video file — and can narrate it with a local model (no cloud):

# Render any tool to mp4 (web tools are captured headlessly with Playwright)
skills/video-gen/scripts/render.sh my-globe --out out/video.mp4

# Add a local-model voiceover: plan a script, synthesize it, then render with it
node skills/video-gen/scripts/narrate.mjs template "a tour of the globe" --out script.json
skills/video-gen/scripts/voiceover.sh script.json                 # → voiceover/voiceover.wav (+ timing manifest)
skills/video-gen/scripts/render.sh my-globe --audio voiceover/voiceover.wav

• Every tool → mp4. Remotion/Manim/Motion Canvas use native renderers; Motion, GSAP, Three.js and p5 are recorded headlessly (Chromium + ffmpeg).
• Voiceover is local-first. Default engine is Piper (offline neural, auto-downloaded); it falls back to macOS say / espeak-ng. voiceover.sh emits voiceover.json with per-segment timings so the animation can be synced to the narration, and narrate.mjs frames converts those to per-tool snippets.
• Plan the script first when you want narration, so the visuals line up with the spoken track. Full guide: references/render.md, references/voiceover.md.

Examples

# Pick a tool for a prompt (deterministic; --json for machine output)
node skills/video-gen/scripts/recommend.mjs "svg logo morph along a path on scroll"
#  → GSAP

# Scaffold a runnable starter and preview it (web tools need zero install)
skills/video-gen/scripts/scaffold.sh threejs my-globe --prompt "rotating 3D globe"
skills/video-gen/scripts/preview.sh my-globe        # opens http://localhost:4321/

# Render a real video file (any tool → mp4, via the unified renderer)
skills/video-gen/scripts/scaffold.sh remotion promo --prompt "data-driven title card"
skills/video-gen/scripts/render.sh promo --out out/video.mp4

# …or render a web tool to mp4 with a local voiceover muxed in
skills/video-gen/scripts/voiceover.sh script.json
skills/video-gen/scripts/render.sh my-globe --audio voiceover/voiceover.wav

The web starters (Motion, GSAP, Three.js, p5.js) are single-file index.html projects that run instantly from a CDN. Motion Canvas and Remotion generate correct minimal npm projects; Manim generates a Python scene + requirements.txt.

How it's built

plugin.json                     # manifest (skills + hooks)
hooks.json                      # sessionStart announcement
skills/
  video-gen/
    SKILL.md                    # router: decision matrix + end-to-end workflow
    scripts/
      tools.json                # canonical tool registry (single source of truth)
      recommend.mjs             # prompt → tool recommender (keyword scoring)
      scaffold.sh               # generate a runnable starter per tool
      preview.sh                # local static server for web starters
      render.sh                 # any project → mp4 (native render or headless capture)
      web-capture.mjs           # Playwright headless recorder for web tools
      voiceover.sh              # local-model TTS → voiceover.wav + timing manifest
      narrate.mjs               # script template + manifest → per-tool sync hints
    references/
      tool-selection.md         # full decision guide
      motion.md  gsap.md  threejs.md
      motion-canvas.md  remotion.md
      manim.md  p5.md           # per-tool: scaffold, prompt→code, render
      render.md  voiceover.md   # render pipelines + local voiceover guide
  render-video/
    SKILL.md                    # /render-video: render to mp4 (+ optional voiceover)

The recommender and scaffolder are plain Node/Bash with no dependencies, so they also work standalone if you clone the repo. The render and voiceover scripts are dependency-free at rest too — they only fetch heavy tools (Playwright/Chromium, Piper, ffmpeg) on demand when you actually render or narrate.

scaffold.sh reference

scripts/scaffold.sh <tool> <project-name> [options]
  tools:   motion | gsap | threejs | motion-canvas | remotion | manim | p5
  --prompt "<text>"   description embedded in the starter + README
  --dir <parent>      where to create the project (default: current dir)
  --install           run npm install / create a venv automatically

render.sh reference

scripts/render.sh <project-dir> [options]
  --out <file>        output path (default: <project>/out/video.mp4)
  --audio <file>      mux a voiceover/music track
  --fps <n>           web capture/assembly fps (default 30)
  --duration <sec>    web capture length (default: audio length, else 10)
  --resolution WxH    web capture size (default 1920x1080)
  --composition <id>  Remotion composition (default: auto-detected)
  --scene <Class>     Manim scene class (default: auto-detected)
  --quality l|m|h|k   Manim quality (default h)
  --dry-run           print the detected tool + pipeline, then exit

voiceover.sh reference

scripts/voiceover.sh <script-file> [options]   # script.json or plain text
  --out <dir>         output dir (default: <script-dir>/voiceover)
  --engine <name>     piper | say | espeak (default: auto)
  --voice <name>      Piper/`say`/espeak voice
  --dry-run           print the plan and exit
# helper: node scripts/narrate.mjs template "<prompt>" --out script.json
#         node scripts/narrate.mjs frames voiceover/voiceover.json --fps 30

Requirements

• Web tools (Motion, GSAP, Three.js, p5.js): a browser + python3 or npx for the preview server. Zero install otherwise.
• Motion Canvas / Remotion: Node.js + npm.
• Manim: Python 3, ffmpeg, and a LaTeX distribution for equations.
• Rendering (/render-video): ffmpeg for capture/muxing; Playwright + Chromium (auto-installed, cached in ~/.cache/video-gen) to record web tools.
• Voiceover: Piper (auto-downloaded to ~/.cache/video-gen) for offline neural TTS, or the OS engine (macOS say / espeak-ng) as a fallback. python3 is used to assemble the track and timing manifest.

Platform support

The helper scripts (scaffold.sh, preview.sh, render.sh, voiceover.sh) are POSIX bash scripts, so they run anywhere you have a bash shell. Rendering and voiceover are otherwise built on cross-platform tools (Node, Python, ffmpeg, Playwright, Piper).

Platform	Render	Voiceover	Notes
macOS (Apple Silicon + Intel)	✅ native	✅ Piper (auto-download) → say fallback	First-class. say/afconvert are built in.
*Linux / nix (x86_64 + arm64)	✅ native	✅ Piper (auto-download) → espeak-ng fallback	First-class. Install espeak-ng for the fallback.
Windows	✅ via WSL2 or Git Bash / MSYS2	✅ via WSL2; Git Bash works if a TTS engine is present	Use WSL2 for the smoothest experience (behaves like Linux). Native PowerShell-only shells can't run the .sh scripts. Piper's auto-download targets macOS/Linux, so on native Windows (Git Bash) install Piper manually or run under WSL2.

The cross-platform tool layer (Playwright + Chromium, ffmpeg, Remotion, Manim, Motion Canvas) works on all three OSes; only the bash entry points and the macOS say / Linux espeak-ng fallbacks are OS-specific.

License

MIT — see LICENSE.