Hello developers,
I'm performing batch iRep calculation for 648 metagenomic samples against 402 oral microbial reference genomes, with a total of ~260,000 tasks.My pipeline: bwa aln → samse → samtools sort → iRep.
The problem:
Initial estimated runtime: ~7 days
Actual progress: only a small portion finished after 8 days
The speed keeps getting slower over time (both 6-task parallel and single-thread serial modes show the same issue)
I would appreciate your help with these questions:
What is the main cause of the gradual slowdown? Is it related to iRep’s internal computation, disk I/O, or temporary SAM/BAM files?
Are there any parameters or optimizations to speed up large-scale batch iRep runs?
Do you have any best practices for high-throughput iRep analysis?
Thank you very much for your work and help!
Hello developers,
I'm performing batch iRep calculation for 648 metagenomic samples against 402 oral microbial reference genomes, with a total of ~260,000 tasks.My pipeline: bwa aln → samse → samtools sort → iRep.
The problem:
Initial estimated runtime: ~7 days
Actual progress: only a small portion finished after 8 days
The speed keeps getting slower over time (both 6-task parallel and single-thread serial modes show the same issue)
I would appreciate your help with these questions:
What is the main cause of the gradual slowdown? Is it related to iRep’s internal computation, disk I/O, or temporary SAM/BAM files?
Are there any parameters or optimizations to speed up large-scale batch iRep runs?
Do you have any best practices for high-throughput iRep analysis?
Thank you very much for your work and help!