Learn basic principles of 3D computer vision by building a Structure-from-Motion (SfM) pipeline for spatial reconstruction from a set of images. The pipeline ingests a bunch of photos of one static object from different angles and outputs a (sparse) 3D point cloud model of the object and the estimated camera poses (for each photo).
- Do better than existing SfM pipelines (e.g. GTSfM or COLMAP) in terms of accuracy or efficiency.
- Implement a full photogrammetry pipeline (SfM + MVS) for dense reconstruction.
- Implement a full SLAM pipeline for spatial reconstruction from video sequences.
# Run SfM pipeline on the `statue_orbit` dataset (installs deps if not already installed)
uv run sfm.py --dataset statue_orbit
# To specifiy feature extraction and keypoint matching methods
uv run sfm.py --dataset statue_orbit --cfg.features.type sift --cfg.matcher.type bf
# Read CLI help for more details
uv run sfm.py --helpFor feature (keypoint descriptor) extraction, I tested the classical SIFT features, which are still used in research and serve as a good baseline, and the learned neural DISK features (via kornia library).
For keypoint descriptor (feature) matching, I employed a classical Brute Force matcher (via the OpenCV's BFMatcher) as well as a learned neural LightGlue matcher (via kornia library).
I created my own statue_orbit dataset (at data/raw/statue_orbit) by taking 15 full-resolution (3060 x 4080) photos of a kneeling archer miniature statue from the Chinese terracotta army with my Redmi phone.
The statue measures about 12.5 cm in height, and the base is roughly 4.5 x 4.5 cm.
The photos have the head region slightly out of focus and photos of the statue's back exhibit, compared to the rest, a marked drop in illumination.
Both of these present a bit of a challenge for the SfM reconstruction pipeline.
I wanted to code something that will work with real data, not just pristine lab data made with professional rigs.
uv run sfm.py --dataset statue_orbit --cfg.features.type sift --cfg.matcher.type bf --cfg.features.num 2000- Number of features limited to maximum
num_features=2000 - The Brute-Force (BF) matcher computes symmetric matches (via
cross_check=True) - See full parameter config in
assets/statue_orbit_sift_bf_config.log
Note: This setup does badly with 1000 features. The effect of illumination drop in the statue back photos results in implausible camera pose estimates and the point cloud is disjointed.
DISK+LG: DISK Features with LightGlue Matcher
uv run sfm.py --dataset statue_orbit --cfg.features.type disk --cfg.matcher.type lg --cfg.features.num 1000- Number of features limited to maximum
num_features=1000 - Only LightGlue matches with confidence above
lg_min_conf=0.1are retained - See full parameter config in
assets/statue_orbit_disk_lg_config.log
Both of the following show a sparse point cloud reconstruction of the kneeling archer statue. The estimated statue points are colored by the corresponding pixel value in the images. The final estimated camera poses are shown as 5-point red pyramids representing the camera frustums.
Figure: SIFT+BF: Note the pronounced presence of fliers around the head, which is likely due to the out-of-focus head region in the dataset images.
Figure: DISK+LG: Note the drastically reduced presence of fliers around the head compared to the SIFT+BF setup. The point cloud is also much denser despite the fact that max number of DISK features was limited to half of the max number of SIFT features.
The reconstruction is extremely sensitive to small perturbation in the camera intrinsics. Below I compare reconstructions with original (calibrated) and perturbed camera intrinsics.
Figure: SIFT+BF setup comparing the orignal reconstruction with camera intrinsics $ K $ (green) and reconstruction with perturbed instrinsics $ K'=0.999K $ (red). Slight deviation in intrinsics has decisive effect on the quality of the reconstructed point cloud as well as the estimated camera poses.
In the below figures I compare the resulting point cloud reconstruction pre- and post bundle adjustment for both feature+matcher setups. The bundle adjustment is done only once at end of the SfM estimation to further refine the object points and the camera poses (the intrisics are fixed during optimization).
Figure: SIFT+BF setup: After bundle adjustment the original camera poses out of SfM pipeline (red) are refined (green), while the points are largely unaffected.
Ceres Solver Report:
Iterations: 47,
Initial cost: 2.435764e+03,
Final cost: 7.472441e+02,
Termination: CONVERGENCE
Full solver log in assets/statue_orbit_sift_bf_ba.log
Figure: DISK+LightGlue setup: After bundle adjustment the original camera poses out of SfM pipeline (red) are refined (green), while the points are largely unaffected.
Ceres Solver Report:
Iterations: 35,
Initial cost: 6.514872e+03,
Final cost: 1.699766e+03,
Termination: CONVERGENCE
Full solver log in assets/statue_orbit_disk_lg_ba.log
uv run sfm.py --dataset corridor --cfg.features.type disk --cfg.matcher.type lg --cfg.features.num 1000Out of curiosity, I wanted to see how my pipeline performs on a real benchmarking dataset where the camera doesn't orbit around an object. I chose TUM-VI at first because it comes with IMU measurements, which I was hoping to use in related visual-inertial odometry learning project, but that has been put on ice due to time constraints. In any case, I picked a sequence of 20 uniformly sampled frames between indices 540 and 640, which, at the frame rate of 20Hz, implies frame sampling frequency of 4Hz (250 ms between frames). This subsequence contains enough motion so that a sufficient baseline is ensured. Compared to the phone photos, TUM-VI provides additional challenge because the frames are distorted due to the fisheye cameras used by the recording rig. The further difficulty is the presence of many planar surfaces such as walls, which could cause problems for the camera pose estimation (via PnP or essential matrix decomposition).
With the original distorted images, we get reconstruction that has more points.
The camera pose estimates are plausible given the frame sequence.
Wall points that should be (ideally) estimated as coplanar look warped due to the fact that cv.triangulatePoints is effectively unable to account for the fisheye camera distortion, as it just assumes a simple pinhole camera projection matrices in its arguments.
Figure: Reconstruction on select frames of corridor4 TUM-VI sequence on the orignal distorted frames: the corridor is apparent and the estimated camera pose sequence looks plausible.
The effect of image undistortion on the reconstruction is compared in the following figures. I used DISK features limited to num_features=1000 with LightGlue matcher.
The undistortion procedure has a limiting effect on the field of view of the resulting images so that we end up with less points in the reconstructed point cloud.
The camera pose estimates are still plausible, but are noticeably different from the ones obtained with the original distorted frames.
Figure: Reconstruction on select frames of corridor4 TUM-VI sequence using the undistorted frames: the corridor is no longer apparent while the estimated camera pose sequence remains plausible.