Skip to content

DESIGN.md

Latest commit

 

History

History
245 lines (202 loc) · 8.27 KB

DESIGN.md

File metadata and controls

245 lines (202 loc) · 8.27 KB
Raw

Video Analyzer Design

Design

Core Workflow

  1. Frame Extraction

    • Uses OpenCV to extract frames from video
    • Calculates frame differences to identify key moments
    • Saves frames as JPEGs for LLM analysis
    • Adaptive sampling based on video length and target frames per minute

    Frame Selection Algorithm

    1. Target Frame Calculation

      • Calculates target frames based on video duration and frames_per_minute
      • Respects optional max_frames limit
      • Ensures at least 1 frame and no more than total video frames

    2. Adaptive Sampling

      • Uses sampling interval = total_frames / (target_frames * 2)
      • Reduces processing load while maintaining coverage
      • Samples more frequently than target to ensure enough candidates

    3. Frame Difference Analysis

      • Converts frames to grayscale for efficient comparison
      • Uses OpenCV's absdiff to calculate absolute difference
      • Compares against FRAME_DIFFERENCE_THRESHOLD (default 10.0)
      • Stores frame number, image data, and difference score

    4. Final Selection Process

      • Selects frames with highest difference scores
      • Takes top N frames based on target frame count
      • If max_frames specified, samples evenly across selected frames
      • Ensures most significant changes are captured

    Limitations

    • Frames between sampling intervals may be missed
    • Rapid sequences may only have one frame selected
    • High-scoring frames may be excluded if outranked by others
    • Even sampling with max_frames may skip some significant changes

  2. Audio Processing

    • Extracts audio using FFmpeg
    • Uses Whisper for transcription (local faster-whisper or DashScope API)
    • Uses YAMNet ONNX for environmental sound detection (default on)
    • Handles poor quality audio by checking confidence scores
    • Segments audio for better context in final analysis

  3. Frame Analysis

    • Each frame is analyzed independently using vision LLM
    • Uses frame_analysis.txt prompt to guide LLM analysis
    • Captures timestamp, visual elements, and actions
    • Maintains chronological order for narrative flow

  4. Video Reconstruction

    • Combines frame analyses chronologically
    • Integrates audio transcript if available
    • Uses video_reconstruction.txt prompt to create technical description
    • Uses narrate_storyteller.txt to transform into engaging narrative

  5. Sound Detection (YAMNet)

    • Reuses the extracted audio.wav (16 kHz mono PCM)
    • Downloads ONNX model (~15 MB) once, caches locally
    • Computes log-mel spectrogram via numpy FFT (no extra deps)
    • Slides 0.96s windows with 0.48s hop → 521-class score vectors
    • Aggregates consecutive windows with identical dominant sounds
    • Excludes speech classes (handled by ASR) to avoid redundancy
    • Feeds time-ranged events into post-processing via {SOUNDS} token

  6. Post-Processing

    • Optional stage controlled by post_process.enabled config
    • Merges: video description + transcript + frame analyses + sound events
    • Uses reverse_script.txt prompt to generate formatted Chinese script
    • Supports context compression to bound per-frame token growth

LLM Integration

Base Client (llm_client.py)

class LLMClient:
    def encode_image(self, image_path: str) -> str:
        # Common base64 encoding for all clients
        with open(image_path, "rb") as image_file:
            return base64.b64encode(image_file.read()).decode('utf-8')

    @abstractmethod
    def generate(self,
        prompt: str,
        image_path: Optional[str] = None,
        stream: bool = False,
        model: str = "llama3.2-vision",
        temperature: float = 0.2,
        num_predict: int = 256) -> Dict[Any, Any]:
        pass

Client Implementations

  1. Ollama (ollama.py)

    • Uses local Ollama API
    • Sends images as base64 in "images" array
    • Returns raw response from Ollama

  2. Generic OpenAI API (generic_openai_api.py)

    • Compatible with OpenAI-style APIs (OpenAI, OpenRouter, etc.)
    • Configurable API URL (e.g. OpenRouter: https://openrouter.ai/api/v1, OpenAI: https://api.openai.com/v1)
    • Sends images as content array with type "image_url"
    • Requires API key and service URL
    • Returns standardized response format

Configuration System

Uses cascade priority:

  1. Command line args
  2. User config (config.json)
  3. Default config (default_config.json)

Key configuration groups:

{
    "clients": {
        "default": "ollama",
        "ollama": {
            "url": "http://localhost:11434",
            "model": "llama3.2-vision"
        },
        "openai_api": {
            "api_key": "",
            "api_url": "https://openrouter.ai/api/v1",
            "model": "meta-llama/llama-3.2-11b-vision-instruct:free"
        }
    },
    "frames": {
        "per_minute": 60,
        "analysis_threshold": 10.0,
        "min_difference": 5.0,
        "max_count": 30
    }
}

Prompt System

Prompt Files

Four key prompts:

  1. frame_analysis.txt — Per-frame visual analysis

    • Includes {PREVIOUS_FRAMES} context (compressed if enabled)
    • Supports user questions through {prompt} token

  2. describe.txt — Video reconstruction

    • Combines all frame analyses + transcript
    • Creates a cohesive video description

  3. compress_context.txt — Context compression (used when compression_interval > 0)

    • Compresses accumulated frame analyses into structured summary
    • Tokens: {COMPRESSED_CONTEXT}, {NEW_ANALYSES}
    • Output format: 场景 / 在场角色 / 近期关键动作 / 对白/文本 / 叙事线索

  4. reverse_script.txt — Post-processing script generation

    • Merges all data sources into formatted Chinese script
    • Tokens: {DESCRIPTION}, {TRANSCRIPT}, {SOUNDS}, {FRAME_ANALYSES}, {METADATA}
    • Output: standard script format with scene headers, action lines, dialogue

The {prompt} token in frame and reconstruction prompts is injected with the user's --prompt flag value (prefixed with "I want to know").

Prompt Loading System

The prompt loading system supports flexible prompt file locations and custom prompts:

  1. Path Resolution:

    • User-specified directory (via config prompt_dir):
      • Absolute paths: /path/to/prompts
      • User home paths: ~/prompts
      • Relative paths: Checked against:
        1. Current working directory
        2. Package root directory
    • Package resources (fallback)

  2. Development Workflow:

    • Install in dev mode: pip install -e .
    • Modify prompt files directly
    • Changes reflect immediately without reinstall
    • Works from any directory

  3. Configuration:

{
    "prompt_dir": "/absolute/path/to/prompts",  // Absolute path
    // or "~/prompts"                          // User home directory
    // or "prompts"                            // Relative path
    // or ""                                   // Use package prompts only
    "prompts": [
        {
            "name": "Frame Analysis",
            "path": "frame_analysis/frame_analysis.txt"
        },
        {
            "name": "Video Reconstruction",
            "path": "frame_analysis/describe.txt"
        }
    ]
}

The system prioritizes user-specified prompts over package prompts, enabling customization while maintaining reliable fallbacks.

Sample output

Sample Output

Common Issues & Solutions

  1. Frame Analysis Failures

    • Ollama: Check if service is running and model is loaded
    • OpenRouter: Verify API key and check response format
    • Both: Ensure image encoding is correct for each API

  2. Memory Usage

    • Adjust frames_per_minute based on video length
    • Clean up frames after analysis
    • Use appropriate Whisper model size

  3. Poor Analysis Quality

    • Check frame extraction threshold
    • Verify prompt templates
    • Ensure correct model is being used

Adding New Features

  1. New LLM Provider

    • Inherit from LLMClient
    • Implement correct image format for API
    • Add client config to default_config.json
    • Update create_client() in cli.py

  2. Custom Analysis

    • Add new prompt template
    • Update VideoAnalyzer methods
    • Modify output format in results

  3. New Audio Processor

    • Add detection class (see sound_detector.py for ONNX pattern)
    • Hook into Stage 1 of cli.py pipeline
    • Store results in analysis.json
    • Pass to post-processing via new {TOKEN} in prompt template