FitCoachAR is a lightweight, real-time fitness coaching system that monitors exercises through 2D pose estimation, detects common form errors, and provides adaptive feedback through augmented-reality overlays and LLM-driven natural language coaching.
This project implements the concepts from:
- Lecture 5: Human Kinematic Modeling - Body-relative coordinate frames, forward kinematics, angular features
- AIFit (CVPR 2021) - Exercise modeling, repetition segmentation, active/passive features
- Modern AR and LLM technologies - Real-time visual feedback and natural language generation
- Real-Time Pose Analysis: Track push-ups, squats, and bicep curls using MediaPipe Pose
- Personalized Calibration: Adapt thresholds using user's "best-form" repetitions
- Dynamic AR Visualization: Arrows, colored joints, angle indicators for instant feedback
- LLM-Driven Coaching: Natural language feedback generated from quantitative error analysis
- Session Summaries: Comprehensive post-workout reports with recommendations
pose_backends/: Pluggable pose-processing engines (MediaPipe 2D default, 3D-ready scaffold)main.py: WebSocket server that streams frames to the active pose backendkinematics.py: Body-relative frames, angular feature extraction, forward kinematicsllm_feedback.py: Template-based and API-driven natural language generationfilters.py: Kalman filtering for landmark smoothing
App.jsx: Main application flow (calibration → workout → summary)AROverlay.jsx: Dynamic AR visualization with colored feedbackAvatar.jsx: 3D skeleton rendering using Three.js
Implemented in kinematics.py:
class BodyRelativeFrame:
"""
- Origin: pelvis/hip center
- X: Left-right (mediolateral)
- Y: Vertical (superior-inferior)
- Z: Front-back (anteroposterior)
"""This normalization removes dependency on global orientation, making angle measurements consistent regardless of camera position.
Extracts active (elbow, knee angles) and passive (spine, hip stability) features:
features = AngularFeatureExtractor.extract_all_features(landmarks)
# Returns: elbow_angle, knee_angle, spine_angle, hip_tilt, etc.State machine detects phase transitions (up → down → up):
segmenter = RepetitionSegmenter(exercise_type='bicep_curls')
new_rep = segmenter.update(angle, thresholds)Two-tier approach for low latency:
- Real-time: Template-based feedback (<100ms)
- Summary: Full LLM-generated session report
feedback = llm_feedback.generate_feedback({
"exercise": "bicep_curls",
"errors": [{"joint": "right_elbow", "deviation_deg": 12}],
"critic_level": 0.5
})
# Output: "Nice pace! Try curling a bit higher to finish each rep."Real-time overlay features:
- ✅ Green joints: Correct form
- ❌ Red joints: Error detected
- 📐 Angle arcs: Visual angle indicators
- ➡️ Yellow arrows: Correction guidance
- Python 3.8+
- Node.js 16+
- Webcam
cd fitcoachar/backend
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txtRequired packages (requirements.txt):
fastapi
uvicorn[standard]
uvloop
opencv-python
mediapipe
numpy
pykalman
scipy
The backend now loads pose processors dynamically. Configure via the POSE_BACKEND
environment variable (defaults to mediapipe_2d):
export POSE_BACKEND=mediapipe_2d # default 2.5D MediaPipe pipeline
# or
export POSE_BACKEND=mediapipe_3d # MediaPipe world-landmark 3D pipeline
# Optional (requires extra deps/config):
export POSE_BACKEND=movenet_3d # Calls external MoveNet microservice (see below)
export POSE_BACKEND=mmpose_poselifter # MMPose PoseLifter (needs MMPOSE_CONFIG/MMPOSE_CHECKPOINT)
uvicorn main:app --host 0.0.0.0 --port 8001 --reloadGET / reports the active backend plus all registered options so clients/frontends
can react accordingly.
Note:
mediapipe_3dshares the existing calibration logic and is ready to use. Themmpose_poselifterbackend is still a scaffold and needs a full 2D detector + lifter integration before it produces results.
Every exercise now supports two runtime modes:
- Common mode – everyday training with the currently selected calibration.
- Calibration mode – review previous captures or record a new baseline.
When you record a new calibration the backend stores:
- Extended/contracted angles (or squat up/down)
- Per-joint deviation parameters (η)
- Critic thresholds (δ) for common and calibration modes
- Base64 snapshots of the captured poses for later review
Use the mode toggle in the UI to switch between common and calibration workflows, adjust the critic level for each mode, and replay past calibrations (including the captured snapshots and deviation metrics).
TensorFlow’s macOS build pins older typing-extensions/numpy, so we run MoveNet in a
separate environment and call it over HTTP.
- Create a TensorFlow environment
conda create -n movenet python=3.10 conda activate movenet pip install tensorflow-macos==2.13.1 tensorflow-metal==1.0.0 pip install numpy==1.24.3 typing-extensions<4.6 opencv-python==4.7.0.72 flask - Download a TFLite MoveNet model (e.g. MultiPose Lightning LiteRT) and note its path.
- Run the service:
The service exposes
python backend/services/movenet_service.py \ --model /path/to/movenet_3d.tflite \ --host 127.0.0.1 --port 8502
POST /inferand stays running in this environment. - Back in the main FitCoachAR environment, point the backend at the service:
export POSE_BACKEND=movenet_3d export MOVENET_SERVICE_URL=http://127.0.0.1:8502/infer uvicorn main:app --host 0.0.0.0 --port 8001 --reload
With that setup, the backend streams frames to the MoveNet service and receives 17 keypoints plus scores, while the main FastAPI process keeps using modern dependencies.
cd fitcoachar/frontend
npm installcd backend
source venv/bin/activate
uvicorn main:app --host 0.0.0.0 --port 8001 --reloadcd frontend
npm run devNavigate to http://localhost:5173
Choose between Bicep Curls or Squats
-
For Bicep Curls:
- Extend arm fully down → Record
- Curl to maximum height → Record
-
For Squats:
- Stand straight → Record
- Descend to deepest squat → Record
This creates personalized thresholds adapted to your range of motion.
- Perform your exercise
- Watch for:
- Rep counter: Automatically increments
- Form feedback: "Keep elbow stable!" or "Go deeper!"
- LLM coaching: Natural language tips
- AR overlay: Visual error indicators
Review your session:
- Total reps completed
- Success rate
- Common mistakes identified
- Recommendations for next session
| Metric | Description | Target | Status |
|---|---|---|---|
| Segmentation IoU | Overlap of detected vs. manual rep boundaries | ≥ 0.70 | ✅ Implemented |
| Latency | End-to-end delay (camera → feedback) | < 100 ms | ✅ Template-based LLM |
| Error Detection F1 | Accuracy of form error detection | > 0.80 | |
| Personalization Gain | Improvement after calibration | +10% | ✅ Implemented |
| Feature | AIFit | FitCoachAR |
|---|---|---|
| Processing | Offline (complete video) | Real-time streaming |
| Pose Estimation | 3D (MubyNet) | 2D (MediaPipe) |
| Accessibility | Motion capture equipment | Webcam only |
| Personalization | Expert instructor baseline | User-calibrated thresholds |
| Feedback | Static text + images | Dynamic AR + LLM coaching |
| Latency | Post-session | <100ms real-time |
fitcoachar/
├── backend/
│ ├── main.py # FastAPI WebSocket server
│ ├── kinematics.py # Body frames, angular features, FK
│ ├── llm_feedback.py # Natural language generation
│ ├── filters.py # Kalman smoothing
│ └── requirements.txt # Python dependencies
├── frontend/
│ ├── src/
│ │ ├── App.jsx # Main application
│ │ ├── AROverlay.jsx # Dynamic AR visualization
│ │ ├── Avatar.jsx # 3D skeleton rendering
│ │ └── App.css # Styling
│ ├── index.html # Entry point
│ └── package.json # Node dependencies
└── README.md # This file
- Upper Body: shoulders (11,12), elbows (13,14), wrists (15,16)
- Core: hips (23,24), pelvis center (computed)
- Lower Body: knees (25,26), ankles (27,28)
Using 3D vector math for joint articulation:
def compute_joint_angle(a, b, c):
"""Angle at joint b, formed by points a-b-c"""
ba = a - b
bc = c - b
cosine = dot(ba, bc) / (norm(ba) * norm(bc))
return arccos(cosine) * 180/πInstead of fixed angles (e.g., "elbow must reach 45°"), we use:
threshold = user_calibrated_value ± hysteresis
This accounts for individual differences in flexibility and body proportions.
This project demonstrates:
- Human Kinematic Modeling: Practical application of joint hierarchies, DoF, coordinate frames
- Real-Time Computer Vision: Streaming pose estimation with <100ms latency
- State Machine Design: Finite state automaton for repetition detection
- LLM Integration: Prompt engineering for context-aware feedback
- Full-Stack Development: React + FastAPI + WebSocket architecture
- Add plank exercise support
- Implement tempo analysis (rep speed)
- Export workout history to CSV
- Full LLM API integration (GPT-4-mini)
- Multi-person support
- Progressive workout plans
- Mobile app (React Native + AR Foundation)
- 3D pose reconstruction for depth analysis
- Social features (share workouts, leaderboards)
- Fieraru et al. (2021). AIFit: Automatic 3D Human-Interpretable Feedback Models for Fitness Training. CVPR 2021.
- Lecture 5: Human Kinematics. McMaster University, Fall 2025.
- MediaPipe Pose. Google Research. https://google.github.io/mediapipe/solutions/pose
- Cao et al. (2017). Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. CVPR 2017.
Course Project: Mobile Data Analytics, McMaster University, Fall 2025
This project is for educational purposes. See course guidelines for usage restrictions.
- AIFit team for the foundational methodology
- MediaPipe team for the pose estimation framework
- Course instructors for guidance on kinematic modeling