chessvideo2pgn

Convert screen recordings of chess games into PGN notation using vision AI.

Record a chess game on your phone, run one command, and get a PGN file you can import into Lichess for full analysis with Stockfish.

How it works

1. Frame extraction -- Samples the video at 2 fps and uses pixel-diffing to detect when a move is played, extracting stable "before" and "after" frame pairs for each move.
2. Move recognition -- For each move, sends the current board state (as text) plus the "after" screenshot to a vision model (GPT-4o by default). The model picks from the list of legal moves, so every detected move is guaranteed to be valid.
3. PGN generation -- Builds a standard PGN file that can be imported into any chess tool.

Recording your game

Before running the tool, you need a screen recording of your chess game:

1. Start screen recording on your phone before you begin the game
   • iPhone: Swipe down from the top-right corner, tap the screen recording button
   • Android: Swipe down from the top, tap "Screen Record"
2. Play your full game in the Duolingo chess app (or any chess app)
3. Stop the recording after the game ends
4. Transfer the video to your computer (AirDrop, email, USB, cloud storage, etc.)

The video should capture the entire game from the first move to the last. The tool detects moves by watching for changes on the board, so make sure the board is fully visible throughout the recording.

Quick start

git clone https://github.com/anandiyer/chessvideo2pgn.git
cd chessvideo2pgn

# Create virtual environment and install dependencies
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

# Add your API key
cp .env.example .env
# Edit .env and add your OpenAI API key (OPENAI_API_KEY=sk-proj-...)

# Run it
.venv/bin/python3 video2pgn.py path/to/your/video.mov

Example

Here's what a real run looks like:

$ .venv/bin/python3 video2pgn.py game.mov --start-fen "r1bqk1nr/pppp1ppp/1n6/1b2p3/4P3/1N3N2/PPPP1PPP/R1BQKB1R"

[1/3] Extracting move frames from video...
  Video: 452s, 60 fps, sampling every 0.5s
  Found 66 moves

[2/3] Using provided starting position
  FEN: r1bqk1nr/pppp1ppp/1n6/1b2p3/4P3/1N3N2/PPPP1PPP/R1BQKB1R

[3/3] Recognizing 66 moves...
    1/66 (W) t=0s->3s ... Ng5
    2/66 (B) t=3s->7s ... Bxf1
    3/66 (W) t=7s->8s ... Nxh7
    ...
   64/66 (B) t=424s->426s ... Rxe7
   65/66 (W) t=439s->441s ... Qxe7+
   66/66 (B) t=442s->444s ... Kc8

========================================
Moves detected: 62/66
Skipped: 4
PGN written to: game.pgn
========================================

Output PGN:

[Event "Chess Video Analysis"]
[FEN "r1bqk1nr/pppp1ppp/1n6/1b2p3/4P3/1N3N2/PPPP1PPP/R1BQKB1R w KQkq - 0 1"]
[SetUp "1"]

1. Ng5 Bxf1 2. Nxh7 Bd3 3. Ng5 c6 4. Nxf7 Bxe4 5. Nd6+ Ke7
6. Nxe4 Nc4 7. Nc3 Nf6 8. Nd5+ Nxd5 9. Nc5 e4 10. Nxe4 Qe8
11. Nc5 Qf7 12. d4 d6 13. Ne4 Nxb2 14. Bxb2 Nf6 15. O-O Bf5
16. Nxf6 Be4 17. Nxe4 g5 18. Ng3 Qg6 19. Bc3 Qxc2 20. Nf5+ Qxf5
21. Qd3 Qxf2+ 22. Rxf2 c5 23. Qe4+ Kd7 24. Raf1 Raf8 25. d5 Kc7
26. Bd4 b6 27. Bxc5 Rh7 28. Qe6 Re8 29. Rf7+ Re7 30. Rxe7+ Rxe7
31. Qxe7+ Kc8 *

Analyzing your game on Lichess

Once you have the PGN, you can import it into Lichess for full computer analysis:

1. Go to lichess.org/paste
2. Paste the contents of your game.pgn file
3. Click Import game

From there you can:

• Get Stockfish analysis -- Lichess runs Stockfish (depth 40+) on every move, showing you where you blundered, missed tactics, or played brilliantly
• See accuracy percentages -- Get an accuracy score for both players
• Explore alternatives -- Click any move to see what the engine recommended instead
• Review the opening -- Lichess identifies the opening name and shows you where you deviated from theory
• Share the game -- Get a permanent link to share with friends or a coach

This turns any casual phone chess game into a fully analyzed, reviewable game -- even from apps that don't have built-in analysis or export features.

Options

-o, --output FILE           Output PGN file (default: game.pgn)
--model MODEL               Vision model to use (default: gpt-4o)
--start-fen FEN             Manually provide starting position (see below)
--change-threshold N        Pixel sensitivity for move detection (default: 40)
--crop Y1,Y2,X1,X2         Board crop region in pixels

Supported models

Model	Provider	Quality	Speed	Cost
gpt-4o	OpenAI	Best	Medium	~$0.50/game
gpt-4o-mini	OpenAI	Good	Fast	~$0.05/game
claude-sonnet-4-6	Anthropic	Fair	Fast	~$0.30/game
claude-opus-4-6	Anthropic	Good	Slow	~$2.00/game

Fixing the starting position

The tool auto-detects the starting position from the first frame, but this can sometimes be inaccurate with stylized piece sets. If the generated PGN looks wrong when you import it, provide the correct starting FEN manually:

.venv/bin/python3 video2pgn.py video.mov \
  --start-fen "r1bqk1nr/pppp1ppp/1n6/1b2p3/4P3/1N3N2/PPPP1PPP/R1BQKB1R"

You can figure out the correct FEN by pausing the video at the first frame and setting up the position on lichess.org/editor.

Board crop

The default crop is calibrated for the Duolingo chess app on iPhone (1206x2622 screen recordings). For other apps or screen sizes:

1. Take a screenshot of your chess app
2. Find the pixel coordinates of the board area
3. Pass as --crop Y_TOP,Y_BOTTOM,X_LEFT,X_RIGHT

.venv/bin/python3 video2pgn.py video.mov --crop 200,1400,50,1050

Advanced: two-step workflow

For more control, you can run extraction and recognition separately:

# Step 1: Extract frames (no API calls, instant)
.venv/bin/python3 step1_extract_frames.py video.mov -o frames/

# Inspect frames/ visually to verify quality

# Step 2: Recognize moves (API calls)
.venv/bin/python3 step2_recognize_moves.py frames/ -o game.pgn --model gpt-4o

# Resume if interrupted (progress is auto-saved)
.venv/bin/python3 step2_recognize_moves.py frames/ -o game.pgn --resume

Requirements

• Python 3.9+
• An OpenAI API key (recommended) or Anthropic API key
• A screen recording of a chess game

Limitations

• Accuracy depends on the vision model's ability to read the specific chess piece style. GPT-4o handles cartoon/stylized pieces well.
• Very fast games (bullet) may have moves too quick to capture between frames.
• The starting position auto-detection can be inaccurate -- use --start-fen if the PGN looks wrong.
• Currently calibrated for the Duolingo chess app on iPhone. Other apps need --crop adjustment.

License

MIT