3D Reconstruction Pipeline

About this work: This research was conducted at Forschungszentrum Jülich, Germany, under the Institute for Advanced Simulation (IAS-8), utilizing data provided by the Institute of Bio- and Geosciences (IBG-2).

Data availability: For inquiries regarding access to the cassava root imaging dataset, please contact Tobias Wojciechowski (IBG-2, Forschungszentrum Jülich).

Method-related questions: For questions about the reconstruction methodology, please create an issue on this repository or contact the author.

Overview

The 3D Reconstruction project performs automated single-plant 3D reconstruction from stereo video data. The pipeline processes paired video inputs through 8 stages to generate a complete 3D mesh with phenotyping metrics.

Key Features:

🎥 Stereo video input processing
🌱 AI-powered plant and stem segmentation
🏗️ High-quality COLMAP-based 3D reconstruction
📊 Automated phenotyping and root system analysis
💾 RAM-based workspace for fast processing
🔄 Resume from incomplete runs
🖥️ Support for both local and HPC environments

Sample Data: We provide sample video files in the data/ directory so you can test the pipeline immediately. See the Example Commands section for how to run the pipeline with this data.

Project Structure

3dreconst/
├── data/                        # Sample video inputs for testing
├── report/                      # Analysis notebooks and final PDF report
├── segmentation/                # Segmentation backends (REMBG, SAM, etc.)
├── scripts/                     # Stage implementations and utilities
│   ├── backproject.py          # Stage 6: Backprojection
│   ├── frames.py               # Stage 1: Plant segmentation
│   ├── mesh_analyzer.py        # Stage 7: Mesh analysis & phenotyping
│   ├── refine.py               # Stage 5: Outlier removal
│   ├── stem_segmentation.py    # Stage 2: Stem segmentation
│   └── utilities.py            # Stage execution wrappers
├── utils/                       # Helper modules and configuration
│   ├── config.json             # Pipeline configuration
│   ├── workspace.py            # RAM workspace management
│   └── helpers.py              # Utility functions
├── tests/                       # Comprehensive test suite
│   ├── stages/                 # Tests for each pipeline stage
│   └── unit/                   # Tests for utilities and workspace
├── 3dreconst_venv/             # Environment setup scripts
│   ├── activate.sh             # Activate environment (use this!)
│   ├── setup.sh                # Initial setup (run once)
│   ├── setup_embree_filtered.sh # Embree library config (auto-sourced)
│   ├── requirements.local.txt  # Local workstation dependencies
│   └── requirements.hpc.txt    # HPC cluster dependencies
├── main.py                      # Single-plant reconstruction entry point
├── multinode_reconstruct.py     # Multi-node HPC orchestration
└── README.md                    # This file

Prerequisites

Local Workstation

Python: 3.10 or later
COLMAP: Installed and accessible in PATH or at known location
GPU: CUDA-capable GPU recommended (optional but faster)
RAM: 32GB+ recommended (pipeline uses RAM workspace)
Disk Space: ~2GB per plant reconstruction

HPC Cluster

Python: 3.10 or later (often via module)
COLMAP: Available via module or custom installation
CUDA: Available via module (e.g., CUDA/12.1.1)
Modules: OpenCV, GCC, CMake (typically provided by cluster)

Installation & Setup

Local Workstation Setup

Clone the repository

git clone https://github.com/eyildiz-ugoe/3dreconst.git
cd 3dreconst

Run the setup script (one-time installation)

bash 3dreconst_venv/setup.sh local

This will:

Create a Python virtual environment
Install all required dependencies from requirements.local.txt
Set up pyembree for mesh processing

Activate the environment (every session)

source 3dreconst_venv/activate.sh

The activation script automatically:

Activates the virtual environment
Configures Embree libraries (with TBB conflict resolution)
Sets up COLMAP paths if available

HPC Cluster Setup

Clone the repository

git clone https://github.com/eyildiz-ugoe/3dreconst.git
cd 3dreconst

Load required modules (cluster-specific)

module load Python/3.10.8
module load CUDA/12.1.1
module load OpenCV/4.8.0
module load GCC/11.3.0

Run the HPC setup script (one-time installation)

bash 3dreconst_venv/setup.sh hpc

This installs minimal dependencies, relying on cluster-provided modules for heavy libraries.

Activate the environment (every session)

export THREEDS_RECONST_PROFILE=hpc  # Optional: tells activate.sh to use HPC mode
source 3dreconst_venv/activate.sh

On HPC, the activation script will automatically load required modules from 3dreconst_venv/modules.sh.

Usage

Basic Command Structure

python main.py \
    --videoA <path_to_camera0_video> \
    --videoB <path_to_camera3_video> \
    [OPTIONS]

Required Arguments

--videoA: Path to first camera video (typically ending in "...0.MOV")
--videoB: Path to second camera video (typically ending in "...3.MOV")

Optional Arguments

--stages: Space-separated stage numbers to run (default: 1 2 3 4 5 6 7 8)
--work-dir-path: Path to existing workspace to resume from
--clean-work-dir: Remove existing workspace before starting
--dry-run: Simulate execution without running stages

Example Commands

Run full pipeline with provided sample data:

source 3dreconst_venv/activate.sh

python main.py \
    --videoA "data/40004600100001000020_9-week-old_cam 081021001450.MOV" \
    --videoB "data/40004600100001000020_9-week-old_cam 081021001453.MOV" \
    --stages 1 2 3 4 5 6 7 8 \
    --clean-work-dir

Resume from interrupted run (only run stages 7-8):

python main.py \
    --videoA "data/40004600100001000020_9-week-old_cam 081021001450.MOV" \
    --videoB "data/40004600100001000020_9-week-old_cam 081021001453.MOV" \
    --work-dir-path /dev/shm/40004600100001000020_9-week \
    --stages 7 8

Run only dense reconstruction (Stage 4):

python main.py \
    --videoA "data/40004600100001000020_9-week-old_cam 081021001450.MOV" \
    --videoB "data/40004600100001000020_9-week-old_cam 081021001453.MOV" \
    --work-dir-path /dev/shm/40004600100001000020_9-week \
    --stages 4

Dry run to check configuration:

python main.py \
    --videoA "data/40004600100001000020_9-week-old_cam 081021001450.MOV" \
    --videoB "data/40004600100001000020_9-week-old_cam 081021001453.MOV" \
    --dry-run

HPC Batch Submission Example

Create a SLURM job script (job_reconstruct.sh):

#!/bin/bash
#SBATCH --job-name=3dreconst
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --gres=gpu:1
#SBATCH --mem=64G
#SBATCH --time=8:00:00
#SBATCH --output=reconstruct_%j.log

# Load modules
module load Python/3.10.8
module load CUDA/12.1.1
module load OpenCV/4.8.0

# Activate environment
cd /path/to/3dreconst
export THREEDS_RECONST_PROFILE=hpc
source 3dreconst_venv/activate.sh

# Run reconstruction
python main.py \
    --videoA "data/40004600100001000020_9-week-old_cam 081021001450.MOV" \
    --videoB "data/40004600100001000020_9-week-old_cam 081021001453.MOV" \
    --stages 1 2 3 4 5 6 7 8 \
    --clean-work-dir

Submit the job:

sbatch job_reconstruct.sh

Pipeline Stages

The reconstruction pipeline consists of 8 sequential stages:

Stage 1: Plant Segmentation (3-5 minutes)

Extracts frames from both video inputs
Uses REMBG (BiRefNet-DIS model) for AI-powered plant segmentation
Generates masked images with background removed
Outputs: images/, masked_images/

Stage 2: Stem Segmentation (2-4 minutes)

Segments the plant stem from the root system
Uses color-based segmentation with adaptive thresholding
Generates stem masks for root/stem separation
Outputs: stem_segmentation/, stem_data.json

Stage 3: Sparse Reconstruction (20-30 minutes)

COLMAP feature extraction and matching
Sparse 3D point cloud generation
Camera pose estimation
Multiple trials to ensure convergence
Outputs: sparse/, database.db

Stage 4: Dense Reconstruction (2-3 hours) ⏰

COLMAP dense stereo reconstruction
Depth map fusion
Point cloud densification and mesh generation
Outputs: dense/fused.ply, dense/mesh.ply

Stage 5: Outlier Removal (1-2 minutes)

Filters spurious points from dense reconstruction
Statistical outlier removal
Mesh refinement
Outputs: dense/refined.ply

Stage 6: Backprojection (1-2 minutes)

Projects 3D points back to 2D images
Validates point visibility across camera views
Removes occluded/unreliable points (~12% filtering)
Outputs: backprojection/, backprojection_filtered.ply

Stage 7: Mesh Analyzer (10-70 minutes)

Generates watertight manifold mesh
Computes root system skeleton
Extracts phenotyping metrics (root count, branching, volume, etc.)
Generates interactive 3D visualizations
Outputs: mesh_manifold.ply, skeleton.swc, phenotyping_data.*

Stage 8: Compression (1-2 minutes)

Creates compressed archive of all outputs
Syncs final results from RAM to disk
Cleans up temporary workspace
Outputs: <plant_id>_<week>-week.zip

Total execution time: ~4 hours (varies by hardware and video quality)

Output Directory Structure

After successful completion, the output directory contains:

plant_001_8-week/
├── images/                              # Extracted frames (987 images)
│   ├── image0.jpg
│   ├── image1.jpg
│   └── ...
├── masked_images/                       # Segmented plant images (987 images)
│   ├── image0.jpg
│   ├── image1.jpg
│   └── ...
├── stem_segmentation/                   # Stem masks (987 images)
│   ├── image0.jpg
│   ├── image1.jpg
│   └── ...
├── sparse/                              # COLMAP sparse reconstruction
│   └── 0/
│       ├── cameras.bin
│       ├── images.bin
│       └── points3D.bin
├── dense/                               # Dense reconstruction outputs
│   ├── fused.ply                       # Fused point cloud (~70MB)
│   ├── mesh.ply                        # Initial mesh
│   ├── refined.ply                     # Outlier-filtered mesh
│   └── stereo/                         # Depth maps
├── backprojection/                      # Backprojection validation (987 images + PLY)
│   ├── image0.jpg                      # Projected points overlay
│   ├── overlay_image0.jpg              # Point visualization
│   ├── mat_image0.jpg                  # Original masked image
│   ├── ...
│   └── backprojection_filtered.ply     # Visibility-filtered points (~70MB)
├── mesh_manifold.ply                    # Watertight manifold mesh (~3.8MB)
├── mesh_manifoldWS.ply                  # Mesh with skeleton (~3.6MB)
├── skeleton.swc                         # Root skeleton in SWC format (~95KB)
├── root_tips.html                       # Interactive 3D visualization (~6.8MB)
├── skeleton_swc.html                    # Interactive skeleton view (~4.6MB)
├── phenotyping_data.json                # Phenotyping metrics (JSON format)
├── phenotyping_data.csv                 # Phenotyping metrics (CSV format)
├── phenotyping_data.xlsx                # Phenotyping metrics (Excel format)
├── stem_data.json                       # Stem segmentation metadata
├── database.db                          # COLMAP database (~77MB)
├── sparse_log_0.txt                     # Sparse reconstruction logs
├── plant_001_8-week_logging.log        # Complete pipeline log
└── plant_001_8-week_status.json        # Stage execution status

# Compressed archive
plant_001_8-week.zip                     # Complete workspace archive (~840MB)

Key Output Files

3D Models:

mesh_manifold.ply - Final watertight mesh for volume/surface analysis
backprojection_filtered.ply - Filtered point cloud with reliable points only
dense/fused.ply - Full dense point cloud before filtering

Phenotyping Data:

phenotyping_data.json/csv/xlsx - Root metrics including:
- Number of root tips
- Number of branching points
- Root system volume
- Maximum depth/width/height
- Storage vs. fibrous root counts
- Total root length

Visualizations:

root_tips.html - Interactive 3D plot showing root tips and branching
skeleton_swc.html - Interactive skeleton visualization

Metadata:

plant_001_8-week_status.json - Execution status for each stage
plant_001_8-week_logging.log - Detailed execution logs with timing

Configuration

Edit utils/config.json to customize pipeline behavior:

{
  "3d_reconstruction": {
    "thick_roots_only": true,        // Focus on storage roots
    "quality": "high",               // Reconstruction quality: low/medium/high
    "fibrous_threshold": 10000,      // Threshold for fibrous root classification
    "sparse_trial": 20               // Number of sparse reconstruction attempts
  },
  "stages": {
    "stages_using_gpu": [1, 3, 4],  // Stages requiring GPU
    "stage_1": {
      "rembg_model": "birefnet-general"  // Segmentation model
    }
    // ... stage-specific configs
  }
}

RAM Workspace Lifecycle

The pipeline uses a RAM-based workspace for fast I/O during processing:

Workspace Creation: At startup, a workspace is created in /dev/shm/<plant_id>_<week>-week
Resume Support: If resuming, existing data is copied from disk to RAM
Incremental Sync: After each stage completes, results are synced back to disk (non-destructive merge)
Final Sync: Complete results are synced to the final output directory
Cleanup: RAM workspace is removed after successful completion

Benefits:

⚡ 10-100x faster I/O compared to disk
💾 Automatic backup to disk after each stage
🔄 Safe resume from interruptions
🛡️ Data preserved even if RAM workspace is lost

Note: The sync logic uses non-destructive merging (as of commit 7cee73b), preventing data loss when resuming work.

Testing

The project includes comprehensive unit tests covering all stages and utilities:

# Activate environment
source 3dreconst_venv/activate.sh

# Run all tests
pytest tests/ -v

# Run specific test categories
pytest tests/unit/ -v                    # Utility tests
pytest tests/stages/ -v                  # Stage tests
pytest tests/stages/test_stage1_segmentation.py -v  # Specific stage

# Run with coverage
pytest tests/ --cov=scripts --cov=utils --cov-report=html

Tests use lightweight fakes and run entirely offline without GPU or heavy dependencies.

Troubleshooting

Common Issues

Issue: "ModuleNotFoundError: No module named 'embreex'"

Solution: Make sure to use source 3dreconst_venv/activate.sh instead of directly activating the venv. The activate script sets up Embree paths correctly.

Issue: "ImportError: libembree.so.2: cannot open shared object file"

Solution: The activate script should handle this automatically. If it persists, check that embree-2.17.7.x86_64.linux/ exists in the project root.

Issue: "TBB threading layer requires TBB version 2021 update 6 or later"

Status: This is a warning, not an error. The filtered Embree setup intentionally uses system TBB to avoid conflicts with bpy (Blender). Performance impact is minimal.

Issue: "COLMAP: command not found"

Solution Local: Install COLMAP and ensure it's in PATH, or set COLMAP_PATH environment variable
Solution HPC: Load COLMAP module or update 3dreconst_venv/modules.sh

Issue: Stage 7 reports "Reconstruction failed. Saving 2D metrics only"

Cause: No valid mesh could be generated (sparse reconstruction failure or insufficient points)
Solution: Check that sparse reconstruction (Stage 3) completed successfully. Review sparse_log_*.txt for COLMAP errors.

Issue: Pipeline runs out of RAM

Solution: Reduce video resolution, or process fewer frames. The pipeline needs ~32GB RAM for typical cassava plant videos.

Issue: Workspace sync takes very long

Cause: Large dataset being copied from RAM to disk
Expected: This is normal for datasets with 1000+ images and dense point clouds. Stage 4 and final sync can take several minutes.

Getting Help

Check the execution log: <plant_id>_<week>-week_logging.log
Check stage status: <plant_id>_<week>-week_status.json
Review COLMAP logs: sparse_log_*.txt
Run with --dry-run to check configuration without execution

Performance Benchmarks

Typical execution times on different hardware configurations:

Local Workstation (32 cores, RTX 3090, 64GB RAM):

Stage 1: 3 min
Stage 2: 2 min
Stage 3: 25 min
Stage 4: 2 hours
Stage 5: 1 min
Stage 6: 1 min
Stage 7: 1 hour
Stage 8: 1 min
Total: ~4 hours

HPC Node (16 cores, Tesla V100, 64GB RAM):

Stage 1: 4 min
Stage 2: 3 min
Stage 3: 30 min
Stage 4: 2.5 hours
Stage 5: 1.5 min
Stage 6: 1.5 min
Stage 7: 1.5 hours
Stage 8: 2 min
Total: ~5 hours

Note: Times vary based on video quality, resolution, and plant complexity.

Citation

If you use this pipeline in your research, please cite:

@software{3dreconst2025,
  author = {Yildiz, Erenus},
  title = {3D Reconstruction Pipeline for Plant Phenotyping},
  year = {2025},
  url = {https://github.com/eyildiz-ugoe/3dreconst}
}

Acknowledgments

COLMAP - Structure-from-Motion and Multi-View Stereo
REMBG - Background removal and segmentation
Open3D - 3D data processing
Trimesh - Mesh processing and analysis

License

This work is licensed under the Creative Commons Attribution 4.0 International License.

See the LICENSE file for details.

Author

Erenus Yildiz
erenus.yildiz@hotmail.com

Contributing

Contributions are welcome! Please:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

HPC File Maintenance

Important for HPC users: HPC systems typically delete files that haven't been accessed for 90 days. To prevent data loss, run the maintenance script monthly:

./touch_files_monthly.sh

This script updates access times for files older than 30 days in both the data and project directories. Set a monthly reminder to run this script to ensure your data and results are preserved.

Changelog

Recent Updates

2025-11-06: Added file maintenance script for HPC cleanup prevention
2025-11-06: Fixed numpy version conflict (venv vs system packages)
2025-11-06: Improved corrupted video file detection and error messages
2025-11-05: Fixed folder recreation after Stage 8 compression
2025-10-24: Fixed critical data loss bug in workspace sync (commit 7cee73b)
2025-10-24: Resolved Embree/TBB library conflicts with Blender
2025-10-24: Switched plant segmentation to REMBG (BiRefNet-DIS model)
2025-10-23: Added comprehensive test suite for all stages
2025-10-23: Unified environment setup with profile-based configuration

Name		Name	Last commit message	Last commit date
Latest commit History 613 Commits
3dreconst_venv		3dreconst_venv
data		data
debug		debug
report		report
scripts		scripts
segmentation		segmentation
tests		tests
utils		utils
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
install_3dreconst.sh		install_3dreconst.sh
install_pyembree.sh		install_pyembree.sh
main.py		main.py
multinode_reconstruct.py		multinode_reconstruct.py
pytest.ini		pytest.ini
test_pyembree_job.sh		test_pyembree_job.sh
touch_files_monthly.sh		touch_files_monthly.sh

License

eyildiz-ugoe/3dreconst

Folders and files

Latest commit

History

Repository files navigation

3D Reconstruction Pipeline

Overview

Project Structure

Prerequisites

Local Workstation

HPC Cluster

Installation & Setup

Local Workstation Setup

HPC Cluster Setup

Usage

Basic Command Structure

Required Arguments

Optional Arguments

Example Commands

HPC Batch Submission Example

Pipeline Stages

Stage 1: Plant Segmentation (3-5 minutes)

Stage 2: Stem Segmentation (2-4 minutes)

Stage 3: Sparse Reconstruction (20-30 minutes)

Stage 4: Dense Reconstruction (2-3 hours) ⏰

Stage 5: Outlier Removal (1-2 minutes)

Stage 6: Backprojection (1-2 minutes)

Stage 7: Mesh Analyzer (10-70 minutes)

Stage 8: Compression (1-2 minutes)

Output Directory Structure

Key Output Files

Configuration

RAM Workspace Lifecycle

Testing

Troubleshooting

Common Issues

Getting Help

Performance Benchmarks

Citation

Acknowledgments

License

Author

Contributing

HPC File Maintenance

Changelog

Recent Updates

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Languages