Welcome to the ATHENA project! This guide provides comprehensive information for developers who want to contribute to, understand, or extend the ATHENA bioinformatics pipeline.
- Project Overview
- Architecture
- Development Environment Setup
- Build System
- Code Organization
- Development Workflow
- Testing
- Code Style Guidelines
- Adding New Features
- Performance Considerations
- Debugging
- Contributing Guidelines
ATHENA (Advanced Tool for High-throughput Experimental NGS Analysis) is a C++17 bioinformatics pipeline that automates NGS data preprocessing. The project integrates external tools (FastQC, Trimmomatic) with a modern C++ wrapper to provide:
- Quality Control: FastQC integration with parallel processing and custom reporting
- Read Trimming: Trimmomatic integration with configurable parameters
- Pipeline Management: Automated workflow orchestration
- User Interface: CLI11-based command-line interface
- Testing: Comprehensive Python-based test suite
- Modularity: Separate modules for each bioinformatics tool
- Performance: Multi-threaded processing and parallel execution
- Usability: Clear command structure and comprehensive error handling
- Maintainability: Clean separation of concerns and well-documented code
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ CLI Layer │ │ Core Logic │ │ Tool Modules │
│ │ │ │ │ │
│ • CLI11 Parser │───▶│ • Athena Class │───▶│ • FastQC │
│ • Argument │ │ • Command │ │ • Trimmomatic │
│ Validation │ │ Dispatch │ │ • Utilities │
│ • Help System │ │ • Pipeline │ │ │
└─────────────────┘ │ Orchestration │ └─────────────────┘
└─────────────────┘
│
┌─────────────────┐
│ System Layer │
│ │
│ • File I/O │
│ • Process Exec │
│ • Threading │
│ • Error Handling│
└─────────────────┘
int main(int argc, char** argv) {
Athena athena;
return athena.run(argc, argv);
}- Simple entry point that creates the main application instance
- Passes command-line arguments to the core application
Key Responsibilities:
- Command-line parsing using CLI11
- Subcommand routing and dispatch
- Pipeline orchestration and coordination
- Error handling and user feedback
Class Structure:
class Athena {
public:
Athena();
int run(int argc, char** argv);
private:
// Core commands
void start(const std::string& input1, const std::string& input2,
const std::string& output_dir, bool generate_report = false);
void clean();
void showHelp();
// Tool integrations
void runFastQC(const std::string& input1, const std::string& input2,
const std::string& output_dir, bool verbose = true,
bool generate_report = false);
void runTrimmomatic(const std::string& input1, const std::string& input2,
const std::string& output_dir, bool generate_report = false,
bool verbose = false);
// Utilities
void generateFastQCReport(const std::string& output_dir);
std::map<std::string, std::string> parseFastQCData(const std::string& fastqc_data_file);
std::string processZipFile(const std::string& zip_file);
// Configuration
int threads = 4;
};fastqc.cpp: Main FastQC execution and coordinationparsing.cpp: FastQC output parsing and data extractionprocess_zip.cpp: Parallel processing of FastQC ZIP archives
Key Features:
- Multi-threaded execution
- Custom report generation
- ZIP file processing with futures/async
- Comprehensive error handling
trim.cpp: Trimmomatic execution and parameter management
Key Features:
- Configurable trimming parameters
- Paired-end read processing
- Output file naming conventions
- Java process management
clean.cpp: Resource cleanup implementationhelp.cpp: Comprehensive help system
// File validation
int check_file_exists(const std::string& input1, const std::string& input2);
// Directory management
int check_output_directory(const std::string& output_dir);Command Line Input
│
▼
CLI11 Parser
│
▼
Athena::run()
│
▼
Command Dispatch
│
┌───┴───┐
▼ ▼
Single Pipeline
Command (start)
│ │
▼ ▼
Execute Orchestrate
Tool(s) Multiple
Tools
- Main Thread: CLI parsing, coordination, user interaction
- FastQC Threads: Configurable threading for FastQC execution
- Report Processing: Parallel ZIP file processing using
std::async - Trimmomatic Threads: Configurable threading for Trimmomatic
# Ubuntu/Debian
sudo apt update
sudo apt install build-essential cmake git python3 python3-pip
# CentOS/RHEL
sudo yum groupinstall "Development Tools"
sudo yum install cmake git python3 python3-pip
# macOS
xcode-select --install
brew install cmake git python3# Ubuntu/Debian
sudo apt install fastqc trimmomatic default-jre
# CentOS/RHEL
sudo yum install fastqc java-1.8.0-openjdk
# Trimmomatic manual installation required
# macOS
brew install fastqc trimmomaticgit clone https://github.com/1337-R-D/Athena.git
cd Athena
# Verify external dependencies are present
ls external/CLI11.hpp
# Check system dependencies
which fastqc
which java
java -version# Development build with debug info
mkdir -p build-debug
cd build-debug
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-g -O0"
make
# Release build for performance testing
mkdir -p build-release
cd build-release
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-O3 -DNDEBUG"
makeThe project uses CMake with a modular structure:
#### 3. IDE Configuration
##### VS Code Setup
```json
// .vscode/settings.json
{
"C_Cpp.default.configurationProvider": "ms-vscode.cmake-tools",
"cmake.buildDirectory": "${workspaceFolder}/build-debug",
"files.associations": {
"*.hpp": "cpp",
"*.cpp": "cpp"
}
}
// .vscode/tasks.json
{
"version": "2.0.0",
"tasks": [
{
"label": "Build Debug",
"type": "shell",
"command": "cmake --build build-debug",
"group": "build"
},
{
"label": "Run Tests",
"type": "shell",
"command": "python3 test_cmd.py",
"group": "test"
}
]
}The project uses CMake with a modular structure:
# CMakeLists.txt overview
cmake_minimum_required(VERSION 3.12)
project(ATHENA VERSION 1.0.0 LANGUAGES CXX)
# C++17 standard
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
# Source organization
set(SOURCES src/main.cpp src/athena.cpp)
set(FASTQC_SOURCES src/fastqc/fastqc.cpp src/fastqc/parsing.cpp src/fastqc/process_zip.cpp)
set(TRIM_SOURCES src/trim/trim.cpp)
set(UTILS_SOURCES utils/tools1.cpp)
# Custom targets
add_custom_target(run COMMAND ${CMAKE_BINARY_DIR}/athena)
add_custom_target(test-commands COMMAND python3 test_cmd.py)# Primary executable
make athena
# Convenience targets
make run # Build and run
make test-commands # Build and test
# Installation
make install # Install system-wide- Symbols included (
-g) - No optimization (
-O0) - Assertions enabled
- Verbose error reporting
cmake .. -DCMAKE_BUILD_TYPE=Debug- Full optimization (
-O3) - No debug symbols
- Assertions disabled (
-DNDEBUG) - Performance-focused
cmake .. -DCMAKE_BUILD_TYPE=Release# Parallel builds
make -j$(nproc)
# Verbose builds for debugging
make VERBOSE=1
# Clean builds
make clean
# or
rm -rf build-* && mkdir build && cd build && cmake .. && makesrc/
├── main.cpp # Entry point - minimal, delegates to Athena class
├── athena.cpp # Core application logic and command dispatch
├── cmd/ # Individual command implementations
│ ├── clean.cpp # Cleanup functionality
│ └── help.cpp # Help system
├── fastqc/ # FastQC integration module
│ ├── fastqc.cpp # Main FastQC execution
│ ├── parsing.cpp # Data parsing and extraction
│ └── process_zip.cpp # ZIP file processing
└── trim/ # Trimmomatic integration module
└── trim.cpp # Trimmomatic execution
Each command is implemented as a method in the Athena class:
// In athena.hpp
void start(const std::string& input1, const std::string& input2,
const std::string& output_dir, bool generate_report = false);
void clean();
void showHelp();Different tools (FastQC, Trimmomatic) implement similar interfaces:
// Similar signatures for tool execution
void runFastQC(args...);
void runTrimmomatic(args...);CLI11 creates command objects based on input:
auto start_cmd = app.add_subcommand("start", "Start the full ATHENA pipeline");
auto clean_cmd = app.add_subcommand("clean", "Clean up ATHENA resources");athena.hpp: Main class declarationtools1.hpp: Utility function declarations- Keep headers minimal - declarations only
- Use include guards or
#pragma once
- One class per file (where applicable)
- Logical grouping in subdirectories
- Clear dependencies - avoid circular includes
- Standalone functions that don't belong to classes
- System interactions (file validation, directory operations)
- Cross-module helpers
# Feature development
git checkout -b feature/new-feature-name
# ... make changes ...
git add .
git commit -m "feat: add new feature description"
git push origin feature/new-feature-nameUse conventional commits:
feat:- New featuresfix:- Bug fixesdocs:- Documentation changesstyle:- Code style changesrefactor:- Code refactoringtest:- Test additions/changeschore:- Build/tooling changes
- Plan: Create issue or discussion for significant changes
- Branch: Create feature branch from main
- Develop: Implement changes with tests
- Test: Run full test suite
- Document: Update documentation as needed
- Review: Submit pull request
- Merge: After review and CI passes
# Quick development cycle
edit src/athena.cpp
make -C build-debug
python3 test_cmd.py
# Full testing cycle
make -C build-debug clean
make -C build-debug
python3 test_cmd.py
./build-debug/athena start -1 test_data/input1.fastq -2 test_data/input2.fastq -o test_resultsThe project uses a multi-layered testing approach:
# Test structure
class TestResult:
def __init__(self, command, expected_keywords, success, output, error=""):
# Test result container
def test_command(executable, args, expected_keywords, test_name):
# Individual command testing
def main():
# Test orchestration and reporting# Full pipeline tests
test_cases = [
{
"name": "Complete Pipeline",
"args": ["start", "-1", "input1.fastq", "-2", "input2.fastq", "-o", "output"],
"keywords": ["Starting ATHENA", "completed successfully"]
}
]# Manual system tests
./athena start --help
./athena clean
./athena --version# Run all tests
python3 test_cmd.py
# Test specific builds
./build-debug/athena help
./build-release/athena --version
# CMake test target
make test-commands# Create test data
mkdir -p test_manual
echo -e "@test\nACGT\n+\n!!!!" > test_manual/R1.fastq
echo -e "@test\nTGCA\n+\n!!!!" > test_manual/R2.fastq
# Test pipeline
./build/athena start -1 test_manual/R1.fastq -2 test_manual/R2.fastq -o test_manual/output# Create realistic test data
mkdir -p test_data_extended
# Generate larger FASTQ files for performance testing
python3 -c "
with open('test_data_extended/large_R1.fastq', 'w') as f:
for i in range(1000):
f.write(f'@read_{i}\n')
f.write('ACGTACGTACGTACGTACGTACGTACGTACGT\n')
f.write('+\n')
f.write('IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII\n')
"# Timing tests
time ./athena start -1 large_R1.fastq -2 large_R2.fastq -o perf_test
# Memory profiling
valgrind --tool=massif ./athena start -1 input1.fastq -2 input2.fastq -o mem_test
# Threading analysis
htop # Monitor during execution// Classes: PascalCase
class Athena {
// Methods: camelCase
void runFastQC();
void showHelp();
// Variables: snake_case or camelCase (be consistent)
int thread_count;
std::string output_dir;
// Constants: ALL_CAPS
static const int DEFAULT_THREADS = 4;
};
// Functions: camelCase
int checkFileExists();
// Files: snake_case.cpp/.hpp// Header guards
#ifndef ATHENA_HPP
#define ATHENA_HPP
// Includes - system first, then project
#include <iostream>
#include <string>
#include <vector>
#include "../include/tools1.hpp"
#include "CLI11.hpp"
// Class definitions
class Athena {
public:
// Public interface first
Athena();
int run(int argc, char** argv);
private:
// Private implementation
void start(/*...*/);
// Member variables last
int threads = 4;
};
#endif // ATHENA_HPP// Function documentation
/**
* @brief Execute FastQC on input files with optional reporting
* @param input1 First FASTQ file path
* @param input2 Second FASTQ file path (for paired-end)
* @param output_dir Output directory for results
* @param verbose Enable detailed output
* @param generate_report Create terminal summary report
*/
void Athena::runFastQC(const std::string& input1, const std::string& input2,
const std::string& output_dir, bool verbose,
bool generate_report) {
// Input validation first
if (check_file_exists(input1, input2) == 0) {
return;
}
// Main logic
std::string command = "fastqc";
// ... build command
// Execution
int result = std::system(command.c_str());
// Result handling
if (result == 0) {
std::cout << "\033[1;32mFastQC completed successfully!\033[0m" << std::endl;
} else {
std::cerr << "\033[1;31mFastQC failed with exit code: \033[0m" << result << std::endl;
}
}// Validation with early return
if (check_file_exists(input1, input2) == 0) {
return; // Error already reported by check function
}
// System command execution
int result = std::system(command.c_str());
if (result != 0) {
std::cerr << "\033[1;31mCommand failed with exit code: \033[0m" << result << std::endl;
std::exit(result); // Propagate error code
}
// File operations
std::ifstream file(filename);
if (!file.is_open()) {
std::cerr << "\033[1;31mUnable to open file: \033[0m" << filename << std::endl;
return;
}/**
* @brief Brief description of the function
* @param parameter_name Description of parameter
* @return Description of return value
* @throws ExceptionType When this exception is thrown
*
* Detailed description of function behavior,
* usage examples, and any special considerations.
*/- Clear section hierarchy with table of contents
- Code blocks with language specification
- Examples for all major functionality
Before implementing:
- Research tool command-line interface
- Plan parameter configuration
- Design error handling approach
- Consider output processing needs
# Create module directory
mkdir src/newtool/
# Create implementation file
touch src/newtool/newtool.cpp
# Update CMakeLists.txt
# Add to NEWTOOL_SOURCES variable// In athena.hpp
void runNewTool(const std::string& input1, const std::string& input2,
const std::string& output_dir, bool verbose = false);
// In athena.cpp - command setup
auto newtool_cmd = app.add_subcommand("newtool", "Run NewTool analysis");
std::string nt_input1, nt_input2, nt_output_dir;
bool nt_verbose = false;
newtool_cmd->add_option("-1,--input1", nt_input1, "First input file")->required();
newtool_cmd->add_option("-2,--input2", nt_input2, "Second input file")->required();
newtool_cmd->add_option("-o,--output", nt_output_dir, "Output directory")->required();
newtool_cmd->add_flag("-v,--verbose", nt_verbose, "Verbose output");
// In command dispatch
if (newtool_cmd->parsed())
runNewTool(nt_input1, nt_input2, nt_output_dir, nt_verbose);// In src/newtool/newtool.cpp
#include "../../include/athena.hpp"
void Athena::runNewTool(const std::string& input1, const std::string& input2,
const std::string& output_dir, bool verbose) {
if (verbose) {
std::cout << "\033[96mRunning NewTool analysis...\033[0m" << std::endl;
std::cout << "Input file 1: " << input1 << std::endl;
std::cout << "Input file 2: " << input2 << std::endl;
std::cout << "Output directory: " << output_dir << std::endl;
}
// Validate inputs
if (check_file_exists(input1, input2) == 0) {
return;
}
if (check_output_directory(output_dir) == 0) {
return;
}
// Build command
std::string command = "newtool";
command += " -i " + input1 + " " + input2;
command += " -o " + output_dir;
if (!verbose) {
command += " > /dev/null 2>&1";
}
// Execute
if (verbose) {
std::cout << "Command: " << command << std::endl;
}
int result = std::system(command.c_str());
if (result == 0) {
std::cout << "\033[92mNewTool completed successfully!\033[0m" << std::endl;
} else {
std::cerr << "\033[91mNewTool failed with exit code:\033[0m " << result << std::endl;
std::exit(result);
}
}# In CMakeLists.txt
set(NEWTOOL_SOURCES
src/newtool/newtool.cpp
)
# Add to main executable
add_executable(athena ${SOURCES} ${FASTQC_SOURCES} ${TRIM_SOURCES} ${NEWTOOL_SOURCES} ${UTILS_SOURCES})# In test_cmd.py
test_cases.append({
"name": "NewTool Command Help",
"args": ["newtool", "--help"],
"keywords": ["Run NewTool analysis", "input1", "input2", "output"]
})
# Add integration test if tool is available
if os.system("which newtool > /dev/null 2>&1") == 0:
test_cases.append({
"name": "NewTool Execution",
"args": ["newtool", "-1", "test_data/input1.fastq", "-2", "test_data/input2.fastq", "-o", "newtool_output"],
"keywords": ["Running NewTool", "completed successfully"]
})- Add command to help system
- Update README with new command examples
- Create specific documentation if needed
// In athena.cpp run() method
app.add_flag("-q,--quiet", quiet_mode, "Suppress output messages");
app.add_option("-t,--threads", global_threads, "Number of threads to use");// For existing commands
fastqc_cmd->add_flag("--no-report", skip_report, "Skip report generation");
fastqc_cmd->add_option("--memory", memory_limit, "Memory limit in GB");// In athena.hpp
void generateCustomReport(const std::string& data_dir, const std::string& format);
// Implementation pattern
void Athena::generateCustomReport(const std::string& data_dir, const std::string& format) {
if (format == "json") {
// Generate JSON report
} else if (format == "xml") {
// Generate XML report
} else {
// Default format
}
}// FastQC threading (user configurable)
int threads = this->threads; // Default: 4
std::cout << "Enter number of threads (default 2): ";
// ... user input handling
// Report processing threading
int num_threads = std::thread::hardware_concurrency() - 4;- I/O Bound Operations: Use more threads than CPU cores
- CPU Bound Operations: Use ≤ CPU core count
- Mixed Workloads: Reserve cores for system processes
// Use RAII for thread management
std::vector<std::future<std::string>> futures;
for (const auto& file : files) {
futures.push_back(std::async(std::launch::async, [this, file]() {
return this->processFile(file);
}));
}
// Collect results
for (auto& future : futures) {
std::string result = future.get();
// Process result
}// Stream processing for large files
std::ifstream file(filename, std::ios::binary);
std::vector<char> buffer(4096);
while (file.read(buffer.data(), buffer.size())) {
// Process chunk
processChunk(buffer, file.gcount());
}# Valgrind memory analysis
valgrind --tool=massif ./athena start -1 large.fastq -2 large.fastq -o output
# Monitor actual usage
/usr/bin/time -v ./athena start -1 input1.fastq -2 input2.fastq -o output// Prefer iostream for C++
std::ifstream file(filename, std::ios::binary | std::ios::ate);
std::streamsize size = file.tellg();
file.seekg(0, std::ios::beg);
// Buffer I/O operations
file.rdbuf()->pubsetbuf(buffer.data(), buffer.size());
#### Process Execution
```cpp
// Consider alternatives to system() for better control
// Using fork/exec or process libraries for production code
## 🐛 Debugging
### Debug Build Configuration
```bash
# Create debug build
mkdir build-debug
cd build-debug
cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-g -O0 -fsanitize=address"
make
# Basic debugging
gdb ./athena
(gdb) run start -1 input1.fastq -2 input2.fastq -o output
(gdb) bt # Backtrace on crash
(gdb) print variable_name
(gdb) step # Step through code# Memory leak detection
valgrind --leak-check=full --show-leak-kinds=all ./athena start -1 input1.fastq -2 input2.fastq -o output
# Thread error detection
valgrind --tool=helgrind ./athena start -1 input1.fastq -2 input2.fastq -o output# Compile with sanitizer
cmake .. -DCMAKE_CXX_FLAGS="-fsanitize=address -fno-omit-frame-pointer -g"
make
# Run with sanitizer
./athena start -1 input1.fastq -2 input2.fastq -o output// Add debug output in athena.cpp
if (start_cmd->parsed()) {
std::cerr << "DEBUG: start command parsed with args:" << std::endl;
std::cerr << " input1: " << input1 << std::endl;
std::cerr << " input2: " << input2 << std::endl;
std::cerr << " output_dir: " << output_dir << std::endl;
start(input1, input2, output_dir, start_report);
}// Debug system command execution
std::cout << "DEBUG: Executing command: " << command << std::endl;
int result = std::system(command.c_str());
std::cout << "DEBUG: Command exit code: " << result << std::endl;// Add file validation debugging
if (!std::filesystem::exists(filename)) {
std::cerr << "DEBUG: File does not exist: " << filename << std::endl;
std::cerr << "DEBUG: Current working directory: "
<< std::filesystem::current_path() << std::endl;
}For production debugging, consider adding structured logging:
// Simple logging utility
enum LogLevel { DEBUG, INFO, WARNING, ERROR };
void log(LogLevel level, const std::string& message) {
std::string prefix;
switch(level) {
case DEBUG: prefix = "[DEBUG] "; break;
case INFO: prefix = "[INFO] "; break;
case WARNING: prefix = "[WARN] "; break;
case ERROR: prefix = "[ERROR] "; break;
}
std::cerr << prefix << message << std::endl;
}-
Fork the repository on GitHub
-
Clone your fork locally:
git clone https://github.com/YOUR-USERNAME/Athena.git cd Athena git remote add upstream https://github.com/1337-R-D/Athena.git -
Set up development environment (see Development Environment Setup)
-
Create a feature branch:
git checkout -b feature/your-feature-name
- Check existing issues and discussions
- For significant changes, create an issue first
- Discuss approach with maintainers
- Keep changes focused: One feature/fix per PR
- Write tests: Add or update tests for your changes
- Update documentation: Keep docs current
- Follow style guidelines: Maintain code consistency
-
Test thoroughly:
python3 test_cmd.py # Manual testing ./build/athena start -1 test_data/input1.fastq -2 test_data/input2.fastq -o test_output -
Commit with clear messages:
git add . git commit -m "feat: add parallel processing for FastQC reports - Implement async processing of ZIP files - Reduce report generation time by 60% - Add threading configuration options - Update tests for parallel execution Closes #123"
-
Push and create PR:
git push origin feature/your-feature-name
## Description
Brief description of changes and motivation.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Refactoring (no functional changes)
## Testing
- [ ] Tests pass locally
- [ ] Added tests for new functionality
- [ ] Manual testing completed
## Checklist
- [ ] Code follows project style guidelines
- [ ] Self-review completed
- [ ] Documentation updated
- [ ] No breaking changes (or clearly documented)- Automated checks must pass
- Peer review by maintainers
- Testing on multiple platforms if applicable
- Documentation review for user-facing changes
## Bug Description
Clear description of the issue.
## Steps to Reproduce
1. Run command: `./athena start -1 file1.fastq -2 file2.fastq -o output`
2. Observe error: ...
## Expected Behavior
What should happen instead.
## Environment
- OS: Ubuntu 20.04
- Compiler: GCC 9.4.0
- CMake: 3.16.3
- FastQC: 0.11.9
- Java: OpenJDK 11.0.11## Feature Description
Clear description of proposed feature.
## Use Case
Why this feature would be useful.
## Proposed Implementation
Ideas for how to implement (optional).
## Alternatives Considered
Other approaches considered.- GitHub Issues: Bug reports, feature requests
- GitHub Discussions: Questions, ideas, general discussion
- Pull Requests: Code review and discussion
- Documentation: Always keep current
Contributors are recognized in:
README.mdcontributors section- Release notes for significant contributions
- GitHub contributor graphs
Thank you for contributing to ATHENA! Your efforts help make bioinformatics data processing more accessible and efficient for researchers worldwide.