Video Analysis using vision models like Llama3.2 Vision and OpenAI's Whisper Models

A video analysis tool that combines vision models like Llama's 11B vision model and Whisper to create a description by taking key frames, feeding them to the vision model to get details. It uses the details from each frame and the transcript, if available, to describe what's happening in the video.

Features

💻 Can run completely locally - no cloud services or API keys needed
☁️ Or, leverage any OpenAI API compatible LLM service (openrouter, openai, etc) for speed and scale
🎬 Intelligent key frame extraction from videos
🔊 Audio transcription with dual backends:
- Local: faster-whisper (OpenAI Whisper models) running on your hardware
- API: OpenAI-compatible speech recognition (DashScope Qwen-ASR, OpenAI Whisper API, etc.) with automatic chunking for long audio
👁️ Frame analysis using vision LLMs (Llama 3.2 Vision, Qwen-VL, GPT-4o, etc.)
📝 Natural language descriptions of video content
🔄 Automatic handling of poor quality audio
📊 Detailed JSON output of analysis results
⚙️ Highly configurable through command line arguments or config file
📦 Batch processing script for multiple videos

Design

The system operates in three stages:

Frame Extraction & Audio Processing
- Uses OpenCV to extract key frames
- Processes audio via local faster-whisper or OpenAI-compatible API
- Automatic audio chunking for API backend (configurable chunk length)
- Handles poor quality audio with confidence checks
Frame Analysis
- Analyzes each frame using vision LLM
- Each analysis includes context from previous frames
- Maintains chronological progression
- Uses frame_analysis.txt prompt template
Video Reconstruction
- Combines frame analyses chronologically
- Integrates audio transcript
- Uses first frame to set the scene
- Creates comprehensive video description

Requirements

System Requirements

Python 3.11 or higher
FFmpeg (required for audio processing)
When running LLMs locally (not necessary when using openrouter)
- At least 16GB RAM (32GB recommended)
- GPU at least 12GB of VRAM or Apple M Series with at least 32GB

Installation

Clone the repository:

git clone https://github.com/byjlw/video-analyzer.git
cd video-analyzer

Create and activate a virtual environment:

python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

Install the package:

pip install .  # For regular installation
# OR
pip install -e .  # For development installation

Install FFmpeg:

Ubuntu/Debian:

sudo apt-get update && sudo apt-get install -y ffmpeg

macOS:
```
brew install ffmpeg
```
Windows:
```
choco install ffmpeg
```

Ollama Setup

Install Ollama following the instructions at ollama.ai
Pull the default vision model:

ollama pull llama3.2-vision

Start the Ollama service:

ollama serve

OpenAI-compatible API Setup (Optional)

You can use OpenAI-compatible APIs for vision (frame analysis) and/or audio (transcription) — independently configured.

Vision API

Get an API key from your provider:
- OpenRouter
- OpenAI
- DashScope (Alibaba Cloud)

Configure via command line:

# For OpenRouter
video-analyzer video.mp4 --client openai_api --api-key your-key --api-url https://openrouter.ai/api/v1 --model gpt-4o

Or add to config/config.json:

{
  "vision": {
    "default": "openai_api",
    "openai_api": {
      "api_key": "your-api-key",
      "api_url": "https://openrouter.ai/api/v1",
      "model": "meta-llama/llama-3.2-11b-vision-instruct"
    }
  }
}

Audio API (OpenAI-compatible transcription)

Set audio.backend to "api" and provide credentials in audio.api:

{
  "audio": {
    "backend": "api",
    "api": {
      "api_key": "your-api-key",
      "api_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
      "model": "qwen3-asr-flash",
      "chunk_length": 30,
      "language": "zh"
    }
  }
}

chunk_length — max seconds per API request (default 30). Long audio is auto-split to stay under the 10 MB base64 limit.
language — ISO language code (e.g. "zh", "en"). Omit for auto-detection.
Supported models: any OpenAI-compatible speech endpoint (DashScope Qwen-ASR, OpenAI whisper-1, etc.)
Audio and vision can use different providers — they're independently configured.

Note: With OpenRouter, you can use llama 3.2 11b vision for free by adding :free to the model name

Design

For detailed information about the project's design and implementation, including how to make changes, see docs/DESIGN.md.

Usage

For detailed usage instructions and all available options, see docs/USAGES.md.

Quick Start

# Local analysis with Ollama + local Whisper (default)
video-analyzer video.mp4

# Cloud vision + cloud audio transcription (e.g. DashScope)
video-analyzer video.mp4 \
    --client openai_api \
    --api-key your-key \
    --api-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
    --model qwen3.5-flash \
    --transcription-backend api \
    --transcription-model qwen3-asr-flash \
    --language zh

# Local vision (Ollama) + API audio transcription
video-analyzer video.mp4 \
    --transcription-backend api \
    --asr-api-key your-key \
    --asr-api-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
    --transcription-model qwen3-asr-flash

# Batch process all videos in input/ folder
.\analyze_all.ps1
.\analyze_all.ps1 -PerMinute 10 -MaxFrames 30 -Language zh

Output

The tool generates a JSON file (output\analysis.json) containing:

Metadata about the analysis
Audio transcript (if available)
Frame-by-frame analysis
Final video description

Sample Output

The video begins with a person with long blonde hair, wearing a pink t-shirt and yellow shorts, standing in front of a black plastic tub or container on wheels. The ground appears to be covered in wood chips.\n\nAs the video progresses, the person remains facing away from the camera, looking down at something inside the tub. ........

full sample output in docs/sample_analysis.json

Configuration

The tool uses a cascading configuration system: CLI args > user config (config/config.json) > default config.

Configuration is split into two independent blocks — vision (frame analysis) and audio (transcription):

{
  "vision": {
    "default": "ollama",
    "temperature": 0.0,
    "ollama": {
      "url": "http://localhost:11434",
      "model": "llama3.2-vision"
    },
    "openai_api": {
      "api_key": "",
      "api_url": "https://openrouter.ai/api/v1",
      "model": "meta-llama/llama-3.2-11b-vision-instruct"
    }
  },
  "audio": {
    "backend": "local",
    "local": {
      "whisper_model": "medium",
      "device": "cpu",
      "language": "en"
    },
    "api": {
      "api_key": "",
      "api_url": "",
      "model": "qwen3-asr-flash",
      "chunk_length": 30,
      "language": "en"
    }
  },
  "frames": {
    "per_minute": 60,
    "max_frames": 2147483647
  },
  "output_dir": "output"
}

Section	Key	Description
`vision`	`default`	Active vision client (`"ollama"` or `"openai_api"`)
`vision.{client}`	`api_key`, `api_url`, `model`	Provider credentials and model name
`audio`	`backend`	`"local"` (faster-whisper) or `"api"` (OpenAI-compatible)
`audio.local`	`whisper_model`	Whisper size: `tiny`, `base`, `small`, `medium`, `large`
`audio.local`	`device`	`"cpu"` or `"cuda"`
`audio.api`	`api_key`, `api_url`	API credentials (required when `backend="api"`)
`audio.api`	`model`	Model name for speech recognition
`audio.api`	`chunk_length`	Max seconds per API call (default 30)
`audio.{backend}`	`language`	ISO language code (`"zh"`, `"en"`, etc.) or `""` for auto
`frames`	`per_minute`	Frames sampled per minute of video

See docs/USAGES.md for all available options.

Batch Processing

analyze_all.ps1 is a Windows-only PowerShell script that processes all videos in the input/ folder in one go. It auto-activates the virtual environment, scans for video files, runs video-analyzer on each one sequentially, and writes a unified log.

⚠️ Windows only. This script relies on PowerShell (ps1) and cmd.exe to run — it will not work on macOS or Linux.

Prerequisites

Windows with PowerShell 5.1+
.venv virtual environment set up in the project root
Video files placed in the input/ folder
config/ folder with your config.json

Quick Start

# Put your videos in input/, then:
.\analyze_all.ps1

# With custom frame sampling
.\analyze_all.ps1 -PerMinute 10 -MaxFrames 30

# With language and custom duration
.\analyze_all.ps1 -Language zh -Duration 120

Parameters

Parameter	Type	Default	Description
`-KeepFrames`	switch	off	Keep extracted frames after analysis
`-LogLevel`	string	`INFO`	Log level: `DEBUG`, `INFO`, `WARNING`, `ERROR`
`-Duration`	float	(full video)	Process only the first N seconds of each video
`-StartStage`	int	`1`	Resume from a specific stage (1–3)
`-MaxFrames`	int	`10`	Maximum number of frames to process
`-PerMinute`	int	`5`	Frames sampled per minute of video
`-Prompt`	string	(default)	Custom question to ask about each video
`-Language`	string	(auto)	Language code for transcription (`zh`, `en`, etc.)
`-TranscriptionBackend`	string	(config)	`local` or `api`
`-TranscriptionModel`	string	(config)	Model name for speech recognition
`-AsrApiKey`	string	(config)	API key for audio transcription
`-AsrApiUrl`	string	(config)	API URL for audio transcription

Output

Analysis results are saved to output/<video_name>/ for each video
A unified log is written to output/analyze_all.log
At the end, a summary is printed: total, success, and failed counts

Prompt Tuning

The prompts that drive frame analysis and video reconstruction can be automatically optimized for your specific content and use case using video-analyzer-tune.

pip install video-analyzer-tune

Run video-analyzer on a few representative videos, edit the outputs to show what ideal results look like, then let DSPy MIPROv2 find better prompt instructions automatically. The tuned prompts are written as new files you point to via your config — the main package is unaffected.

See video-analyzer-tune/README.md for full instructions.

Uninstallation

To uninstall the package:

pip uninstall video-analyzer

License

Apache License

Contributing

We welcome contributions! Please see docs/CONTRIBUTING.md for detailed guidelines on how to:

Review the project design
Propose changes through GitHub Discussions
Submit pull requests

Name		Name	Last commit message	Last commit date
Latest commit History 55 Commits
.claude/worktrees		.claude/worktrees
.github		.github
docs		docs
video-analyzer-tune		video-analyzer-tune
video-analyzer-ui		video-analyzer-ui
video_analyzer		video_analyzer
.gitignore		.gitignore
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
analyze_all.ps1		analyze_all.ps1
readme.md		readme.md
requirements.txt		requirements.txt
setup.py		setup.py
test_prompt_loading.py		test_prompt_loading.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Video Analysis using vision models like Llama3.2 Vision and OpenAI's Whisper Models

Table of Contents

Features

Design

Requirements

System Requirements

Installation

Ollama Setup

OpenAI-compatible API Setup (Optional)

Vision API

Audio API (OpenAI-compatible transcription)

Design

Usage

Quick Start

Output

Sample Output

Configuration

Batch Processing

Prerequisites

Quick Start

Parameters

Output

Prompt Tuning

Uninstallation

License

Contributing

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

Video Analysis using vision models like Llama3.2 Vision and OpenAI's Whisper Models

Table of Contents

Features

Design

Requirements

System Requirements

Installation

Ollama Setup

OpenAI-compatible API Setup (Optional)

Vision API

Audio API (OpenAI-compatible transcription)

Design

Usage

Quick Start

Output

Sample Output

Configuration

Batch Processing

Prerequisites

Quick Start

Parameters

Output

Prompt Tuning

Uninstallation

License

Contributing

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages