Paper Reproduction Repository

A comprehensive template and toolkit for reproducing research papers, running experiments, and extending methodologies with ablation studies.

📋 Overview

This repository provides a structured approach to:

• Reproduce original paper experiments with exact hyperparameters
• Validate results across different random seeds
• Extend research with custom ablations and modifications
• Track experiments with organized configs and reproducibility checklists
• Automate testing with CI/CD smoke tests

Key Features

• ✅ Minimal working baseline (PyTorch, CIFAR-10)
• ✅ YAML configuration system for easy hyperparameter management
• ✅ Reproducibility checklist and documentation
• ✅ Automated CI smoke tests via GitHub Actions
• ✅ Scripts for data download and training execution
• ✅ Modular code structure for easy customization

---

🚀 Quick Start

Prerequisites

• Python 3.8+
• pip or conda
• Git

Installation & Smoke Test (2 minutes)

# Clone and setup environment
git clone https://github.com/ChethanNazre/Paper-Repro.git
cd Paper-Repro

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Run smoke test (downloads CIFAR-10 automatically)
bash scripts/run_smoke.sh

This smoke test validates that:

• Dependencies are correctly installed
• Data download pipeline works
• Training loop executes without errors
• Model produces predictions

---

📖 Running Full Experiments

Configuration-Based Training

1. Review the baseline config:

   cat configs/baseline.yaml

2. Update hyperparameters in configs/baseline.yaml to match the paper's specifications

3. Run training:

   python src/train.py --config configs/baseline.yaml --seed 123

4. Run multiple seeds for statistical significance:

   for seed in 42 123 456 789 1000; do
       python src/train.py --config configs/baseline.yaml --seed $seed
   done

Custom Configurations

Create new configs for ablation studies:

# Copy and modify baseline
cp configs/baseline.yaml configs/ablation_no_dropout.yaml

# Edit as needed
nano configs/ablation_no_dropout.yaml

# Train with custom config
python src/train.py --config configs/ablation_no_dropout.yaml --seed 42

---

📁 Repository Structure

Paper-Repro/
├── configs/                    # Experiment configurations
│   ├── baseline.yaml          # Original paper hyperparameters
│   └── ablations/             # Custom ablation configs
├── src/                        # Core training and evaluation code
│   ├── train.py               # Main training script
│   ├── model.py               # Model architecture
│   ├── data.py                # Data loading and preprocessing
│   ├── metrics.py             # Evaluation metrics
│   └── utils.py               # Helper functions
├── scripts/                    # Convenience and automation scripts
│   ├── run_smoke.sh           # Smoke test script
│   ├── download_data.sh       # Dataset download
│   └── run_experiment.sh      # Batch experiment runner
├── docs/                       # Documentation and reports
│   ├── REPRODUCIBILITY.md     # Reproducibility checklist
│   ├── EXPERIMENT_LOG.md      # Experiment results and notes
│   └── CITATION.md            # Paper citation
├── .github/
│   └── workflows/
│       └── ci_smoke_test.yml  # Automated CI pipeline
├── requirements.txt            # Python dependencies
├── README.md                   # This file
└── .gitignore                  # Git ignore rules

---

🔧 Customizing for Your Paper

1. Update Model Architecture

Edit src/model.py to implement the paper's specific architecture:

class PaperModel(nn.Module):
    def __init__(self, config):
        super().__init__()
        # Add your model layers here
        pass
    
    def forward(self, x):
        # Implement forward pass
        pass

2. Configure Hyperparameters

Update configs/baseline.yaml with paper parameters:

model:
  architecture: "paper_model"
  hidden_dim: 256
  num_layers: 3

training:
  epochs: 100
  batch_size: 32
  learning_rate: 0.001
  optimizer: "adam"
  loss_fn: "cross_entropy"

3. Add Custom Preprocessing

Modify src/data.py for paper-specific data handling:

def get_dataloaders(config):
    # Implement paper's data augmentation
    # Handle custom train/val/test splits
    # Apply domain-specific preprocessing
    pass

4. Implement Paper Metrics

Add evaluation metrics in src/metrics.py:

def paper_specific_metric(predictions, targets):
    """Compute the paper's custom evaluation metric"""
    pass

5. Update Documentation

• Replace "Paper Title" placeholders in configs and docs
• Add paper-specific notes in docs/EXPERIMENT_LOG.md
• Include citation in docs/CITATION.md

---

📊 Running Experiments

Single Experiment

python src/train.py --config configs/baseline.yaml --seed 42 --output results/exp1

Batch Experiments with Multiple Seeds

bash scripts/run_experiment.sh configs/baseline.yaml 5 results/baseline

Ablation Studies

# Run all ablation configs
for config in configs/ablations/*.yaml; do
    python src/train.py --config $config --seed 42
done

---

✅ Reproducibility Checklist

Before publishing results, verify:

• Code runs on clean environment
• Random seeds are fixed
• Exact paper hyperparameters are used
• Dataset versions are documented
• GPU/hardware specifications are noted
• All results are logged and reproducible
• Ablation studies are documented
• Hyperparameter search methodology is explained

See docs/REPRODUCIBILITY.md for detailed checklist.

---

🤖 Automated Testing

The repository includes GitHub Actions CI pipeline that:

• Runs on every push
• Installs dependencies
• Executes smoke test
• Validates code quality

View workflow: .github/workflows/ci_smoke_test.yml

---

📦 Dependencies

Core packages:

• PyTorch 2.0+ - Deep learning framework
• torchvision - Computer vision datasets and models
• numpy - Numerical computing
• pandas - Data analysis
• matplotlib/seaborn - Visualization
• pyyaml - Config file handling
• tensorboard - Experiment tracking (optional)

Full list: see requirements.txt

Install all:

pip install -r requirements.txt

---

📝 Experiment Tracking

Log results in docs/EXPERIMENT_LOG.md:

## Experiment 1: Baseline Reproduction
- **Config**: configs/baseline.yaml
- **Seeds**: 42, 123, 456, 789, 1000
- **Best Accuracy**: 95.2% ± 0.3%
- **Notes**: Reproduced paper results within 0.5%

---

🔍 Troubleshooting

CUDA Out of Memory

• Reduce batch_size in config
• Use torch.cuda.empty_cache()
• Disable gradient checkpointing if enabled

Data Download Issues

bash scripts/download_data.sh

Dependency Conflicts

pip install --upgrade -r requirements.txt

Reproducibility Issues

• Verify seed setting in src/train.py
• Check CUDA version compatibility
• Confirm GPU/CPU is consistent

---

📜 License & Citation

This template is licensed under the MIT License.

When using this template, cite the original paper:

@article{author2024papertitle,
  title={Paper Title},
  author={Author Name},
  journal={Journal Name},
  year={2024}
}

See docs/CITATION.md for full citation details.

---

👤 Author

ChethanNazre

• GitHub: @ChethanNazre
• Repository: Paper-Repro

---

🤝 Contributing

To extend this template:

1. Fork the repository
2. Create a feature branch
3. Commit changes with clear messages
4. Submit a pull request

---

📚 Additional Resources

• PyTorch Documentation
• Reproducibility in ML
• YAML Configuration Guide
• GitHub Actions Docs

---

❓ FAQ

Q: How do I modify the model architecture? A: Edit src/model.py and update the model class to match your paper's design.

Q: Can I use a different dataset? A: Yes, modify src/data.py to load your dataset and update configs accordingly.

Q: How do I track hyperparameter search? A: Create multiple config files in configs/ and run experiments with each.

Q: What if results don't match the paper? A: Check docs/REPRODUCIBILITY.md, verify hyperparameters, seeds, and data preprocessing.

---

Last Updated: August 2026 | Template Version: 1.0