Slurm: Nextflow Script and Code Integration with OSMP Stack

[jennyziyi-xu]

I tried to run Hannah's PR, but ran into an error when making a basic gene query. That PR contains a lot of code changes and it's hard to find and fix issues. To break Hannah's PR into smaller steps:

• Send a dummy job from OSMP backend to Slurm. Verify that result can be seen on the working directory in the Slurm node.
• Send a dummy job from OSMP backend that kickstarts a Nextflow job.
• Further develop Nextflow script, have a working version of it.
• When gene size is larger than 0.6 mbps, kickstart the Nextflow job.
• TBD

[jennyziyi-xu]

Running these commands in the terminal can start the nextflow job successfully:

ssh [user]@slogin.ccm.sickkids.ca
srun --pty bash
module load nextflow/20200707 Singularity/3.7.0_slurm
cd osmp-annotation/jobs  (from my home directory)
nextflow run annotation.nf -profile annotation

On the branch jenny/slurm, the [Slurm API executes the script slurm-script-1.sh] (

osmp/server/src/resolvers/getVariantsResolver/getVariantsResolver.ts

Line 94 in bcccb31

script: '#!/bin/bash\n./slurm-script-1.sh',

). slurm-script-1.sh has the following content:

#!/bin/bash
module () {
	eval `/usr/bin/modulecmd bash $*`
}
module load nextflow Singularity
cd osmp-annotation/jobs
nextflow run annotation_test.nf -profile annotation
echo "Done executing"

• Calling this REST API will however result in a permission error related to NextFlow.
• Kevin Lu has suggested sourcing .bashrc before executing the script: link