Skip to content

About

Real-time 3D pose tracking with MediaPipe BlazePose - tracks left arm joints with dual visualization and Parquet data export

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

10 Commits

Folders and files

Repository files navigation

3D Pose Tracker

Real-time 3D pose tracking and visualization using MediaPipe BlazePose. Captures left arm joint positions (shoulder, elbow, wrist) with live 3D visualization and saves data to Parquet format.

Features

  • Real-time 3D Pose Tracking: Uses MediaPipe BlazePose for accurate 3D pose estimation
  • Dual Visualization:
    • 2D video feed with pose overlay (OpenCV)
    • Real-time 3D spatial viewer (Matplotlib - macOS stable)
  • macOS Optimized: Uses matplotlib for 3D visualization - no display issues
  • Optimized Performance: Configurable frame rate, efficient processing pipeline
  • Data Persistence: Saves joint positions to Parquet format with Snappy compression
  • Interactive Playback: Replay and analyze captured motion with playback controls
  • Color-coded Joints:
    • Left Shoulder: Red
    • Left Elbow: Green
    • Left Wrist: Blue

Installation

Prerequisites

  • Conda (Miniconda or Anaconda)
  • Webcam (for live capture)
  • macOS, Linux, or Windows

Setup

  1. Clone the repository:

    git clone https://github.com/danielesiegel/3d-pose-tracker.git
    cd 3d-pose-tracker
  2. Create conda environment:

    conda env create -f environment.yml
  3. Activate environment:

    conda activate pose-tracker

Environment Details

  • Python: 3.10 (optimal compatibility)
  • Key Dependencies:
    • MediaPipe 0.10.9 (BlazePose model)
    • OpenCV 4.8.1 (video capture)
    • Matplotlib 3.7+ (3D visualization for capture - macOS stable)
    • Open3D 0.18.0+ (3D visualization for viewer)
    • Pandas 2.0+ + PyArrow 12.0+ (Parquet I/O)
    • Click 8.1.7 (CLI framework)

Usage

Capture Mode

Start capturing pose data from your webcam:

python src/capture.py --frames 30

Options:

  • --frames, -f: Frames per second to capture (default: 30)
  • --output, -o: Output directory for saved data (default: output/)
  • --no-3d: Disable 3D visualization (useful for display issues on macOS)

Controls:

  • Press q to quit and save data

Examples:

# Capture at 60 FPS
python src/capture.py --frames 60

# Save to custom directory
python src/capture.py --frames 30 --output data/

# Disable 3D viewer (if experiencing display issues)
python src/capture.py --frames 30 --no-3d

Viewer Mode

Replay and analyze saved pose data:

python src/viewer.py output/pose_data_20231129_153045.parquet

Options:

  • --speed, -s: Playback speed multiplier (default: 1.0)
  • --loop, -l: Loop playback continuously
  • --static: Show static trajectory view

Interactive Controls:

  • SPACE: Play/Pause
  • R: Reset to beginning
  • Mouse: Rotate view
  • ESC: Exit

Examples:

# Play at 2x speed
python src/viewer.py output/pose_data.parquet --speed 2.0

# Loop continuously
python src/viewer.py output/pose_data.parquet --loop

# Static trajectory view
python src/viewer.py output/pose_data.parquet --static

Project Structure

3d-pose-tracker/
├── src/
│   ├── capture.py          # Main capture script
│   └── viewer.py           # Playback viewer
├── output/                 # Saved Parquet files (auto-created)
├── data/                   # Optional data directory
├── environment.yml         # Conda environment specification
├── .gitignore             # Git ignore rules
└── README.md              # This file

Data Format

Captured data is saved in Parquet format with the following schema:

Column Type Description
timestamp string ISO 8601 timestamp
frame int64 Frame number
shoulder_x float64 Left shoulder X coordinate
shoulder_y float64 Left shoulder Y coordinate
shoulder_z float64 Left shoulder Z coordinate
elbow_x float64 Left elbow X coordinate
elbow_y float64 Left elbow Y coordinate
elbow_z float64 Left elbow Z coordinate
wrist_x float64 Left wrist X coordinate
wrist_y float64 Left wrist Y coordinate
wrist_z float64 Left wrist Z coordinate

Coordinate System

  • MediaPipe uses a right-handed coordinate system
  • Origin is at the center of the detected pose
  • Units are in meters relative to hip center
  • Z-axis points toward the camera

Sample Data for Collaborators

For researchers and developers looking to analyze human motion data, we provide output/datademo.parquet as a sample dataset.

What it contains: Real 3D kinematic data capturing human left arm motion over time. Each row represents a temporal snapshot (frame) of the spatial configuration of three skeletal keypoints (shoulder, elbow, wrist) as they move through 3D space.

Data represents: The geometric trajectory and spatial evolution of left arm joint positions during natural human movement, sampled at 30 Hz. This biomechanical time-series data encodes:

  • Position vectors (x, y, z) for each joint in world coordinates (meters)
  • Temporal sequence of arm configurations during motion
  • Kinematic state suitable for calculating velocity, acceleration, and joint angles
  • 3D spatial relationships between shoulder, elbow, and wrist throughout the motion

Use cases:

  • Motion analysis and biomechanics research
  • Gesture recognition algorithm development
  • Kinematic modeling and animation
  • Motor control pattern analysis
  • Machine learning training data for pose prediction
# Quick start with the sample data
import pandas as pd

# Load sample motion data
df = pd.read_parquet('output/datademo.parquet')

# Each frame contains 3D positions of left arm joints
# Calculate joint angles, velocities, or train ML models
print(f"Captured {len(df)} frames of motion")
print(f"Duration: {len(df) / 30:.2f} seconds at 30 FPS")

Technical Details

MediaPipe BlazePose

  • Model Complexity: 2 (heavy model for best 3D accuracy)
  • Landmark Format: 33 3D landmarks (COCO format)
  • Left Arm Indices:
    • Shoulder: 11
    • Elbow: 13
    • Wrist: 15

Performance Optimization

  • Efficient frame processing with configurable FPS
  • Non-blocking 3D visualization using threading
  • Parquet with Snappy compression for fast I/O
  • Minimal memory footprint

Visualization

  • 2D View: OpenCV window (640x480)
  • 3D View (Capture): Matplotlib 3D renderer (macOS stable)
    • Interactive rotation with mouse
    • Auto-scaling view limits
    • Color-coded joints and bones
  • 3D View (Viewer): Open3D renderer for playback
    • High-performance trajectory visualization
    • Playback controls

Troubleshooting

Webcam Issues

If webcam doesn't open:

# Check available cameras (macOS/Linux)
ls /dev/video*

# Try different camera index in capture.py
cap = cv2.VideoCapture(1)  # Change from 0 to 1

3D Visualization Issues

If experiencing any 3D display issues:

# Run without 3D visualization
python src/capture.py --frames 30 --no-3d

The capture script now uses matplotlib which is stable on macOS. The --no-3d flag is available if you want to disable the 3D viewer entirely.

Performance Issues

If experiencing lag:

  • Reduce frame rate: --frames 15
  • Close other applications
  • Ensure good lighting for better tracking
  • Try disabling 3D view: --no-3d

Import Errors

If packages are missing:

conda activate pose-tracker
conda env update -f environment.yml

References

MediaPipe

Open3D

Apache Parquet

License

MIT License - feel free to use and modify for your projects.

Contributing

Contributions welcome! Please open issues or pull requests.

Future Enhancements

Potential additions:

  • Full body tracking (all 33 landmarks)
  • Multiple person support
  • Export to additional formats (CSV, JSON, HDF5)
  • Real-time metrics (joint angles, velocity)
  • Web-based viewer using Three.js
  • Video file input support
  • GPU acceleration for processing

Acknowledgments

  • Google MediaPipe team for BlazePose
  • Open3D contributors
  • Apache Arrow/Parquet communities

Disclaimer: This project was created in part with AI-assisted code and is still a work in progress.

About

Real-time 3D pose tracking with MediaPipe BlazePose - tracks left arm joints with dual visualization and Parquet data export

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages