Mesh2Sim — usage

Markerless mesh-based biomechanics: video → MHR mesh (SAM 3D Body) → OpenSim joint angles, monocular and multi-view, one shared back-end. This README is the practical "how to run it" guide. Project rationale and locked design decisions live in CLAUDE.md; the staged plan in docs/.

You can process the same trial two ways and compare the results:

• Monocular — one camera (e.g. cam01) → test_data/results/mono_cam01/
• Multi-view — cameras cam01..cam04 fused → test_data/results/multi_cam4/

Each run produces a scaled .osim model and an IK .mot motion you can open side by side in the OpenSim GUI.

---

1. Requirements

• WSL2 (Ubuntu) on Windows, or native Linux.
• NVIDIA GPU + driver, with the container/CUDA stack working in WSL2 (nvidia-smi OK).
• conda / miniconda.
• The SAM 3D Body checkpoint at /home/fdela/FastSAM3DToOpenSim/checkpoints/sam-3d-body-dinov3 (rig assets/mhr_model.pt).

Environments (one per stage — they are mutually incompatible, never merge them)

The stages talk to each other only through serialized contract files, so each lives in its own conda env. Two are needed to run the pipeline end to end:

env	Python	holds	used by
mesh2sim-frontend-mhr	3.11	PyTorch CUDA + vendored SAM 3D Body rig	T1 front-end, T3–T7, T5 fusion, orchestrators
mesh2sim-biomech-bridge	3.9	nimblephysics (OpenSim scale + IK)	T8 biomech bridge

Create them from the pinned specs (once):

conda env create -f stages/frontend_mhr/environment.yml     # → mesh2sim-frontend-mhr
conda env create -f stages/biomech_bridge/environment.yml    # → mesh2sim-biomech-bridge

⚠️ Do not install nimblephysics and the torch stack in the same env, nor JAX with torch, nor the mmcv stack with the rest. One env per stage — this is a hard project rule.

The shared contracts package (pydantic + numpy only) is installed in every env.

---

2. Input data layout

test_data/
  cam01.mp4  cam02.mp4  cam03.mp4  cam04.mp4      # the 4 synchronized views
  results/
    camera_rig_metric/                            # calibration ingested as a CameraRig (§3)
    mono_cam01/                                   # monocular result folder      (§5)
    multi_cam4/                                   # multi-view result folder     (§6)

The four videos must be time-synchronized (see §4 if they are not).

---

3. Calibration → CameraRig

Multi-view needs calibrated cameras. We ingest a Pose2Sim-format Calib.toml (metric, real-world distances) — e.g. the lab dynamic-calibrator output, or a Kineo NLF calibration exported to the same format — into a serialized CameraRig contract:

conda run -n mesh2sim-frontend-mhr python - <<'PY'
import sys; sys.path.insert(0, "stages/calibration/src")
from mesh2sim_calibration.parse import parse_pose2sim_calib
from mesh2sim.contracts import save
calib = "/home/fdela/extrinsic-camera-calibration/lab-camera-dynamic-calibrator/output/demo_Pose2Sim/results/Calib_scene_calibrated.toml"
rig = parse_pose2sim_calib(calib, rig_id="labcalib_metric")
save(rig, "test_data/results/camera_rig_metric")
print("views:", [c.view_id for c in rig.cameras])
PY

This CameraRig (metric extrinsics + intrinsics per view) is reused by both the mono and multi runs — mono pulls one view's intrinsics K, multi uses all four for fusion. Because the calibration carries real distances, the fused world is metric, so the multi .osim scale is physically correct (not just the joint angles).

---

4. (optional) Multi-camera synchronization

If your four videos are not already frame-aligned, align them with the GUI (scrub each view to the same instant, e.g. a clap / heel-strike):

conda run -n mesh2sim-frontend-mhr python -m mesh2sim_calibration.manual_sync \
    --dir test_data --fps 60

A Pose2Sim-style automatic sync (vertical-speed cross-correlation) also exists in mesh2sim_calibration.auto_sync.

---

5. Run MONOCULAR (one camera)

conda run -n mesh2sim-frontend-mhr python orchestrator/run_mono_view.py \
    --video test_data/cam01.mp4 \
    --rig   test_data/results/camera_rig_metric \
    --view  cam01 \
    --fps   60 \
    --out   test_data/results/mono_cam01 \
    --subject cam4subj --trial mono_cam01

Chain: T1 front-end (per-frame MHR) → T4 shape-lock → C2 personalized .osim → T3 adapter → de-flip + ground-anchor (mono depth-ambiguity handling) → T7 → T8 (nimblephysics scale

• marker offsets + IK, launched automatically in the biomech env). Runs ~4 min on GPU for 100 frames.

---

6. Run MULTI-VIEW (cam01..cam04 fused)

conda run -n mesh2sim-frontend-mhr python orchestrator/demo_pipeline_multi.py \
    --rig    test_data/results/camera_rig_metric \
    --videos test_data/cam01.mp4 test_data/cam02.mp4 test_data/cam03.mp4 test_data/cam04.mp4 \
    --views  cam01 cam02 cam03 cam04 \
    --frames 0 100 --fps 60 \
    --out    test_data/results/multi_cam4 \
    --subject cam4subj --trial multi_cam4

Chain: T1 front-end on each view (with that view's intrinsics) → T5 fusion (fuse_multiview: one shared MHR fit by dense multi-view surface reprojection into the calibrated cameras, visibility-weighted, with shape/scale prior + temporal smoothness — no triangulation) → fused per-frame trajectory (skin markers + joint centres) → reorient to OpenSim Y-up + floor → C2 personalized .osim → T8.

The shared morphology (shape + scale) is fit jointly against all four views (seeded from the median across every view), not taken from a single camera.

---

7. Outputs & comparison

Each result folder contains:

results/<run>/
  demo_subject.osim / subject.osim     # C2 per-subject OpenSim model (pre-scale)
  trajectory_*/                        # serialized AnatomicalTrajectory (world markers)
  <subject>_<trial>.trc                # observed markers (openable in OpenSim)
  t8_out/
    <subject>_<trial>.mot              # ← IK joint angles (the biomechanics result)
    *_scaled.osim                      # ← scaled subject model
    biomech_fit/                       # serialized BiomechFit (RMSE, offsets, provenance)

Open in the OpenSim GUI: load the *_scaled.osim, then File ▸ Load Motion on the .mot. Do this for both mono_cam01/ and multi_cam4/ to overlay and compare the two joint- angle traces of the same trial.

Marker-fit RMSE is printed at the end of each run and stored in biomech_fit (mono cam01 ≈ 2 cm on 100 frames).

---

8. Pipeline stages (reference)

id	stage	env	role
T1	frontend_mhr	frontend	SAM 3D Body → per-frame MHR BodyEstimate
T3	anatomical_adapter	frontend	BodyEstimate → AnatomicalObservation (markers + joint centres via correspondence map)
T4	shape-lock	frontend	freeze subject morphology for the trial
C2	subject_osim_gen	frontend	personalized .osim from locked shape (Mesh2Marker)
T5	fusion_multiview	frontend	multi-view fusion into one shared MHR (reprojection, no triangulation)
T6a	calibration	frontend	ingest Calib.toml → CameraRig; sync GUIs
T7	ground_anchor / pose_refine	frontend	de-flip, foot-lock, floor, world Y-up
T8	biomech_bridge	biomech	nimblephysics scale + marker offsets + IK → .osim + .mot

---

9. Status & caveats

• Validation: no optoelectronic reference data yet, so no Bland-Altman accuracy numbers. The validation stage / harness will be wired once opto data is available. For now, compare mono vs multi against each other and check plausibility in the OpenSim GUI.
• Multi world frame: the fused mesh is placed in the calibration's world frame and reoriented to OpenSim Y-up (floored). With a metric calibration the .osim scale is correct; even without one, IK joint angles are scale-invariant so the .mot stays valid.
• Multi coherence: multi-view resolves the monocular depth ambiguity, so the mono-only de-flip / foot-lock hacks are not applied to the fused motion (fusion + temporal term keep it coherent).