________ ______ __
/ ____/ /_ ___ __________/ ____/___ _____ ___ / /
/ / / __ \/ _ \/ ___/ ___/ / / __ `/ __ `__ \/ /
/ /___/ / / / __(__ |__ ) /___/ /_/ / / / / / /_/
\____/_/ /_/\___/____/____/\____/\__,_/_/ /_/ /_(_)
Computer vision system that detects chess moves from a physical chessboard using YOLO and image processing.
Amit Butbul
Electrical & Computer Engineering Student – Technion
Additionally, you can see our system handling Captures, and Illegal Moves, complex moves like En-Passant, and Error Recovery!
You can read more about the development process in ChessCam_Project_Report.pdf, or see our poster.
(Full disclosure: our system does not handle moves like castling or promotion, since the project had limited scope).
- An intel realsense D435 camera's color stream is accessed using pyrealsense
- Every frame, Ultralytics' YOLOv11s provides piece classifications
- We track the pieces using a SORT (simple online realtime tracking) algorithm we implemented, and associate the new detections to tracked piece objects.
- Using OpenCV's functions we localize the pieces to their appropriate square using a Homography.
- We assemble a Python-Chess board object with the current game state, perform legality checks, and query stockfish for a move recommendation.
Training was done in Roboflow using The Following Dataset. We did not allocate images to a test set considering the system's performance itself is the test. About 20% of the images were used for validation. The dataset is comprised off of a mix of Roboflow's dataset, and one we made on our own using the recorder.py script that generates images from the realsense camera stream. We made sure to add both null images, and images with occlusions, since those situations are common in our use case (people move pieces and their hands occlude them frequently).
A homography is a linear transformation that maps between two planes, and we can compute it with matching points. Once we have a homography, we use the tracked pieces' bounding box parameters to compute a point that represents accurately the square the pieces are on. Here's how we do it:
And you can Watch the Following Animation to see how we compute the Homography using OpenCV's findChessBoardCorners, (we then sort them to row major to map to a planar board's corners), and use the computed homography to map pieces to their correct squares.
You can read more about the development process in ChessCam_Project_Report.pdf, or see our poster.
| File | Description |
|---|---|
| src/scripts/online_inference.py | Main file for preforming the inference of chess position, and tracking |
| src/scripts/recorder.py | A file for extracting images from the camera using pyrealsense, used to generate training images |
| src/scripts/train_yolo11_object_detection_on_custom_dataset.ipynb | modified Ultralytics' notebook for training the YOLOv11s model |
| src/scripts/utils/supporting_structs.py | Supporting structures for object tracking, such as PieceManager, and Associator. |
| src/scripts/utils/chess_supporting_structs.py | Supporting structures for chess logic, interfacing with stockfish and move inference. |
| src/Dockerfile and src/Docker-compose.yaml | set up the image and container for the development environment |
We want to work in an environment that has both git, github, VSCode, and Docker. To do that, we first create a github account, download a VScode git extension, download docker, and a VScode extension for docker too.
- we select an isolated folder where a localized instance of our project takes place.
- Using
ctrl+shift+pand then selecting the commandDocker: Add Docker files to Workspacewe get a few empty docker files - we need to give the container permission to access the camera, this is done with the command
xhost +local:docker - make sure you have stockfish installed and that the adress in
chess_supporting_structsmatches its location on your machine.