Veo Prompt ML

A machine learning system for generating and managing Veo video prompts using transformer models, PEFT fine-tuning, and semantic search.

Features

• Prompt Extraction: Fine-tuned Flan-T5 model to extract structured data from raw text
• Semantic Search: FAISS-based vector search using BGE embeddings
• Prompt Composition: AI-powered prompt generation from user intent
• Quality Scoring: Reward model for ranking generated prompts
• REST API: FastAPI endpoints for integration
• N8N Integration: Workflow automation support

Project Structure

veo-prompt-ml/
├── api/                 # FastAPI application
├── data/               # Data files and indexes
├── models/             # Trained models
├── n8n/                # N8N workflow files
├── notebooks/          # Jupyter notebooks
├── scripts/            # Training and deployment scripts
├── src/                # Source code
└── requirements.txt    # Python dependencies

Setup

1. Virtual Environment

# Create virtual environment
python3 -m venv venv

# Activate virtual environment
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install Dependencies

pip install -r requirements.txt

3. Prepare Data

Create data/raw_prompts.txt with one prompt per line:

neon-lit street market at blue hour, handheld camera, 24fps
sunrise over foggy mountains, dolly shot, cinematic lighting
rainy city streets at night, sodium vapor lamps, gimbal movement

4. Extract Structured Data

python src/boot_extract.py

This creates data/prompts_seed.jsonl with structured prompt data.

5. Train Models

Configure Accelerate (first time only)

accelerate config

Run Complete Training Pipeline

sh scripts/train_all.sh

Or run individually:

# Train extractor model
accelerate launch src/train_extractor.py

# Build FAISS index
python src/index_faiss.py --build

# Train reward model
python src/train_reward.py

6. Start API

sh scripts/run_api.sh

The API will be available at http://localhost:8000

Usage

API Endpoints

Health Check

curl http://localhost:8000/health

Extract Structured Data

curl -X POST http://localhost:8000/extract \
  -H "Content-Type: application/json" \
  -d '{"text": "neon-lit street market at blue hour"}'

Search Similar Prompts

curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "neon lighting", "k": 5}'

Generate Suggestions

curl -X POST http://localhost:8000/suggest \
  -H "Content-Type: application/json" \
  -d '{"intent": "neon rain street market at blue hour", "k": 6, "n": 6}'

Command Line Tools

Predict with Extractor

python src/predict_extractor.py "your raw prompt text here"

Build FAISS Index

python src/index_faiss.py --build

Search FAISS Index

python src/index_faiss.py --search "your search query" --k 8

Train Reward Model

python src/train_reward.py

Score Prompt Quality

python src/train_reward.py --score "your prompt text"

N8N Integration

Import n8n/veo_suggest_flow.json into N8N to create automated workflows.

Note: If not running in Docker, change the host from host.docker.internal:8000 to localhost:8000 in the HTTP Request node.

Development

Testing

Use the Jupyter notebook notebooks/quicktest.ipynb for quick testing and experimentation.

Logging

The API logs all /suggest calls to data/online_log.jsonl for analysis and future training.

Model Caching

Models are cached in memory using global singletons to avoid reloading on each request.

Docker

Build and run with Docker:

docker build -t veo-prompt-ml .
docker run -p 8000:8000 veo-prompt-ml

Troubleshooting

Common Issues

1. Model not found: Ensure you've run the training scripts first
2. CUDA out of memory: Reduce batch sizes in training scripts
3. Import errors: Check that virtual environment is activated

File Dependencies

• models/extractor/: Required for /extract endpoint
• models/reward/: Required for /suggest endpoint
• data/faiss.index: Required for /search endpoint
• data/metas.json: Required for /search endpoint

License

MIT License