A real-time AR pipeline built with OpenCV. One binary, eight tasks — detect a checkerboard target, calibrate the camera, estimate pose, and render virtual 3-D objects that track the board as the camera or target moves. Includes Harris corner detection and ORB-based AR on arbitrary flat targets.
.
├── main.cpp # Entry point — parses task number, dispatches to task runner
├── config.h # Shared constants (board size, square size, sub-pixel params)
├── task1.h / .cpp # Detect and extract checkerboard corners
├── task2.h / .cpp # Collect calibration images (saves calib_N.png)
├── task3.h / .cpp # Camera calibration + pose helpers (saves calibration.yml)
├── task4.h / .cpp # Camera position via solvePnP (prints R/T vectors)
├── task5.h / .cpp # Reproject outside corners (reprojection check)
├── task6.h / .cpp # Virtual 3-D shapes: wireframe + OpenGL shaded rendering
├── task7.h / .cpp # Harris corner feature detection with live controls
├── task8.h / .cpp # Custom target AR via ORB feature matching
├── ball.png # Sample reference image for Task 8
├── board.png # Sample reference image for Task 8
├── teabag.png # Sample reference image for Task 8
└── CMakeLists.txt
Each task has its own .h/.cpp pair. main.cpp includes all eight headers
and routes to the correct runTaskN() function based on the first argument.
Note:
estimatePose(),loadCalibration(), anddrawAxes()are defined intask3.h / task3.cppand shared by Tasks 4, 5, 6, and 8.
OpenCV 4.x with development headers, plus OpenGL and GLFW (required by Task 6 shaded mode only — wireframe mode works without them).
macOS (Homebrew)
brew install opencv glfwUbuntu / Debian
sudo apt update && sudo apt install libopencv-dev libglfw3-dev libglu1-mesa-devWindows
Download the OpenCV installer from https://opencv.org/releases and install GLFW
from https://www.glfw.org. Ensure pkg-config is on your PATH, or point CMake
at each install directory manually.
CMake (recommended — works on all platforms)
cmake -B build
cmake --build buildBinary: build/ar_project (or build/ar_project.exe on Windows)
Direct compile (Linux / macOS)
g++ -std=c++17 main.cpp task1.cpp task2.cpp task3.cpp task4.cpp \
task5.cpp task6.cpp task7.cpp task8.cpp \
-o ar_project \
$(pkg-config --cflags --libs opencv4) \
-lGL -lGLU -lglfw./ar_project <task> [camera_index | video_file]
| Argument | Meaning |
|---|---|
1 |
Task 1 — detect and display checkerboard corners |
2 |
Task 2 — collect calibration images, save calib_N.png |
3 |
Task 3 — calibrate camera, save calibration.yml (requires calib images) |
4 |
Task 4 — estimate current camera position, print R/T (requires calibration.yml) |
5 |
Task 5 — reproject outside board corners (requires calibration.yml) |
6 |
Task 6 — render virtual 3-D shapes on board (requires calibration.yml) |
7 |
Task 7 — Harris corner feature detection with live controls |
8 <reference_image> |
Task 8 — AR on a custom flat target (requires calibration.yml and a reference image path as 2nd arg) |
| (optional) numeric | Webcam index to use (default 0) |
| (optional) string | Path to a video file instead of webcam |
Examples
./ar_project 1 # Task 1, default webcam
./ar_project 2 1 # Task 2, webcam index 1
./ar_project 3 # Task 3, calibrate from saved calib_N.png images
./ar_project 4 # Task 4, default webcam
./ar_project 5 # Task 5, default webcam
./ar_project 6 # Task 6, default webcam
./ar_project 7 # Task 7, default webcam
./ar_project 8 ball.png # Task 8, ball.png as reference, default webcam
./ar_project 8 teabag.png 1 # Task 8, teabag.png, webcam index 1
./ar_project 8 reference.jpg video.mp4 # Task 8, from a video file| Key | Action | Tasks |
|---|---|---|
q or ESC |
Quit | All |
s |
Save current annotated frame as a PNG snapshot | All |
s |
Save calibration frame (uses last detected corners) | Task 2 & 3 |
c |
Run calibration (requires ≥ 5 saved frames) | Task 3 only |
1–4 |
Select virtual shape | Task 6 only |
Tab |
Toggle wireframe / shaded render mode | Task 6 only |
Note on Task 2 vs Task 3: In Task 2, press
sto save a calibration frame. In Task 3, presssto add a live frame to the calibration set, then presscto run calibration (minimum 5 frames required).
Detects the checkerboard in real time and refines corners to sub-pixel accuracy.
Key functions
| Function | Purpose |
|---|---|
detectCorners(gray, corners) |
Runs findChessboardCorners() + cornerSubPix(), returns true if all corners found |
drawCornersOverlay(frame, corners, found) |
Annotates the frame with colour-coded corners, count, and first-corner coordinates |
runTask1(argc, argv) |
Interactive loop |
Detection flags used inside findChessboardCorners()
CALIB_CB_ADAPTIVE_THRESH— local thresholding, robust to uneven lightingCALIB_CB_NORMALIZE_IMAGE— compensates for ambient brightness variationCALIB_CB_FAST_CHECK— skips full search when no corners are likely present
Sub-pixel refinement (cornerSubPix)
- Search window: 11 × 11 px
- Zero zone: (−1, −1) — disabled
- Termination: ε < 0.001 or 30 iterations
Board configuration (edit in config.h to change)
constexpr int BOARD_COLS = 9; // internal corners, horizontal
constexpr int BOARD_ROWS = 6; // internal corners, vertical
constexpr float SQUARE_SIZE = 1.0f; // each square = 1 world unitcorner_set[0] is the upper-left internal corner when the board's long
axis is horizontal and lighting is unambiguous.
Captures board-detected frames and saves them as calib_1.png, calib_2.png, …
for use by Task 3.
Recommended workflow
- Run
./ar_project 2 - Move the board to distinct positions and orientations
- Press
seach time the corners are detected to save a frame - Collect at least 5 frames (the code enforces this minimum in Task 3); 10–15 frames from varied angles gives a better calibration
- Press
qwhen done — the images are ready for Task 3
The display shows a cached corner overlay ("showing cached corners") when the board is momentarily lost, so you can judge the last detection before saving.
Loads saved calib_N.png images on startup, lets the user add more live frames
with s, then runs cv::calibrateCamera() when c is pressed.
Writes intrinsics and distortion coefficients to calibration.yml.
Key functions
| Function | Purpose |
|---|---|
buildPointSet() |
Returns the 54 3-D board-space corner coordinates (Z = 0 plane, X = col, Y = −row) |
calibrateFromImages(argc, argv, cameraMatrix, distCoeffs, outputFile) |
Loads calib images, runs OpenCV calibration, prints before/after matrix and RMS error, saves YAML |
loadCalibration(filename, cameraMatrix, distCoeffs) |
Reads calibration.yml — shared by Tasks 4–8 |
estimatePose(corners, K, D, rvec, tvec) |
Calls cv::solvePnP() — shared by Tasks 4, 5, 6, 8 |
drawAxes(frame, K, D, rvec, tvec, axisLength) |
Draws R/G/B coordinate axes at the board origin — shared by Tasks 3, 8 |
3D world point convention (from buildPointSet())
| Corner | World coordinate |
|---|---|
| Top-left (row 0, col 0) | (0, 0, 0) |
| Top row, col 1 | (1, 0, 0) |
| Top-right (row 0, col 8) | (8, 0, 0) |
| Row 1, col 0 | (0, −1, 0) |
| Bottom-right (row 5, col 8) | (8, −5, 0) |
Check the printed RMS re-projection error — values below 1.0 px indicate a good calibration; below 0.5 px is excellent.
Output file: calibration.yml
camera_matrix: [fx, 0, cx, 0, fy, cy, 0, 0, 1]
dist_coeffs: [k1, k2, p1, p2, k3]
rms_error: 0.312
image_width: 1280
image_height: 720After calibration, Task 3 automatically enters a live pose-estimation loop that draws colour-coded axes at the board origin (red = +X, green = +Y, blue = +Z).
Loads calibration.yml, then each frame calls estimatePose() (defined in
task3.cpp) on the detected checkerboard. R and T are overlaid on the frame
and printed to the terminal every 15 frames.
Key functions
| Function | Purpose |
|---|---|
runTask4(argc, argv) |
Interactive loop (loads calibration.yml on startup) |
estimatePose(...) |
Defined in task3.cpp; called here via task3.h |
Interpreting rvec and tvec (units = board squares)
| Component | Meaning |
|---|---|
rvec |
Rodrigues angle-axis vector — direction = rotation axis, magnitude = rotation angle (radians) |
tvec[0] |
Lateral X — increases as camera moves right of the board |
tvec[1] |
Vertical Y — negative when board origin is above the camera's horizontal plane |
tvec[2] |
Depth Z — increases as the camera backs away from the board |
Uses projectPoints() to reproject the four extreme internal corners of the
board (TL, TR, BL, BR) back onto the image each frame as a visual
sanity-check of calibration and pose.
Key functions
| Function | Purpose |
|---|---|
projectOutsideCorners(frame, K, D, rvec, tvec) |
Projects the 4 world-space corners and draws the labelled outline quad |
runTask5(argc, argv) |
Interactive loop |
Corner world positions and display colors
| Label | World point | Dot color |
|---|---|---|
| TL | (0, 0, 0) |
Yellow |
| TR | (8, 0, 0) |
Green |
| BL | (0, −5, 0) |
Blue |
| BR | (8, −5, 0) |
Orange |
Requires calibration.yml.
Renders one of four selectable shapes floating above the checkerboard in two
switchable render modes. Requires calibration.yml.
Shapes (press 1–4 or click the on-screen panel)
| Key | Shape | Description |
|---|---|---|
1 |
Eiffel Tower | Tapering 4-tier lattice with cross-bracing; apex leans in X/Y for asymmetry. Color gradient: grey-blue → teal. |
2 |
DNA Helix | Two opposing helical strands (4 full turns, radius 1.3 units) connected by rungs. Strand A: blue→cyan; Strand B: magenta→gold. |
3 |
Icosahedron | 30 edges in 6 latitude color zones: hot red → amber → green → sky blue → violet → magenta. |
4 |
UFO | Hull rings, cockpit dome, gold rim band, 3 landing legs, and 8-ray tractor beam. Multi-zone colors. |
Render modes (press Tab or click the on-screen toggle)
- Wireframe — pure OpenCV:
projectPoints()+cv::line(), no extra dependencies - Shaded — OpenGL fixed-function Phong shading via GLFW offscreen render,
composited back onto the webcam frame with
glReadPixels
Key functions
| Function | Purpose |
|---|---|
buildEiffelTower(cx, cy, scale) |
Returns vector<Line3D> wireframe |
buildDNA(cx, cy) |
Returns vector<Line3D> wireframe |
buildIcosahedron(cx, cy, radius) |
Returns vector<Line3D> wireframe |
buildUFO(cx, cy, scale) |
Returns vector<Line3D> wireframe |
drawVirtualObject(frame, lines, K, D, rvec, tvec) |
Projects all line endpoints and draws with cv::LINE_AA |
runTask6(argc, argv) |
Interactive loop with GUI panel and OpenGL shaded mode |
All objects are centered at (cx, cy) = ((BOARD_COLS−1)/2, −(BOARD_ROWS−1)/2)
— the center of the board — and float above it at negative Z values.
Implements Harris corner detection on a live video stream with three parameters adjustable in real time via clickable ▲ / ▼ buttons drawn on the video frame.
Harris response formula:
R = det(M) − k × trace(M)², k = 0.04 (fixed)
Adjustable parameters
| Parameter | Range | Effect |
|---|---|---|
| Block size | 2–15 | Neighbourhood size for gradient matrix M. Larger = smoother. |
| Aperture | 3 / 5 / 7 | Sobel kernel size. Larger = finer edge sensitivity. |
| Threshold % | 0–100 | Fraction of peak response to count as corner. Lower = more detections. |
Two windows are shown simultaneously:
- "Task 7 - Harris Features" — live frame with corner dots and the control panel
- "Task 7 - Response Map" — Harris R map false-coloured with
COLORMAP_INFERNO
Key functions
| Function | Purpose |
|---|---|
detectHarrisCorners(gray, blockSize, apertureSize, kHarris, threshold) |
Returns vector<cv::Point> of detected corner positions |
drawHarrisResults(frame, corners, response, responseVis) |
Annotates frame with strength-scaled dots and produces heat-map |
runTask7(argc, argv) |
Interactive loop with ▲/▼ GUI buttons |
Tracks any flat, texture-rich target using ORB feature matching and a RANSAC
homography, then overlays a UFO wireframe anchored to the detected target.
Requires calibration.yml and a reference image passed as argv[2].
./ar_project 8 <reference_image> [camera_index | video_file]Pipeline per frame
CLAHE(reference) → ORB(4000 kp) ──────────────────────────────┐
CLAHE(frame) → ORB features → BFMatcher (Hamming) │
↓ │
Lowe ratio test (0.75) │
↓ │
findHomography (RANSAC, 3 px threshold) │
↓ │
refCorners [TL,TR,BR,BL] → perspectiveTransform │
↓ │
worldPts (flat plane, physW × physH) → solvePnP │
↓ │
drawTargetOverlay + drawVirtualObject(UFO) ┘
Good targets: book covers, posters, magazines, printed photos, box faces. Poor targets: plain coloured walls, reflective/transparent surfaces, low-contrast objects.
Three sample reference images are included: ball.png, board.png, teabag.png.
Validity checks applied before pose estimation
| Check | Condition |
|---|---|
| Minimum inliers | ≥ 8 RANSAC inliers required |
| Area check | Detected quad area must be 0.3 %–98 % of frame area |
| Convex hull | Projected 4 corners must form a convex quadrilateral (exactly 4 hull vertices) |
| Bounds check | All 4 corners must lie within a 50 px border around the frame |
Key functions
| Function | Purpose |
|---|---|
loadCustomTarget(imagePath, physW, physH) |
Loads reference image, applies CLAHE, computes ORB features |
detectCustomTarget(frame, gray, target, K, D, rvec, tvec, trackedCorners, outMatchCount) |
kNN + ratio test + RANSAC homography + validity checks + solvePnP |
drawTargetOverlay(frame, corners, K, D, rvec, tvec) |
Draws boundary quad, TL/TR/BR/BL labels, and 3-D axes |
runTask8(argc, argv) |
Interactive loop |
| Task | Needs calibration.yml |
|---|---|
| 1 | No |
| 2 | No |
| 3 | No (produces it) |
| 4 | Yes |
| 5 | Yes |
| 6 | Yes |
| 7 | No |
| 8 | Yes |
- Partial occlusion — all 54 internal corners must be visible simultaneously; any occlusion causes detection to fail for that frame
- Oblique angles — checkerboard detection degrades beyond roughly 70° from frontal
- Screen glare — displaying the board on a phone/tablet can introduce glare that obscures parts of the pattern at certain angles
- Symmetry ambiguity — the checkerboard is symmetric under 180° rotation, so corner ordering can flip; using an ArUco pattern avoids this
- Distance extremes — board partially out of frame (too close) or too small in the image (too far) both prevent detection
- Calibration quality — Tasks 4–6 and 8 depend on a good calibration; ensure diverse board poses and an RMS error below 1.0 px (< 0.5 px is excellent)
- Task 8 target texture — targets with low contrast or repetitive texture produce insufficient ORB matches and tracking will fail
- Task 6 OpenGL dependency — GLFW and OpenGL must be installed for shaded mode; the wireframe mode works without them