Error only when using SLURM (-S) with ASHS 2.0.0: Validity check at end of stage 2 detected missing files #9
Description
Hi,
I recently downloaded ASHS 2.0.0 (the release from March 2 2022).
I ran ashs_main.sh with the test data you provide and the UPENN PMC Atlas (ashs_atlas_upennpmc_20170810) on our HPC Center that supports Slurm.
When I run ashs_main.sh both with the serial option and the parallel option (-P):
ashs_main.sh -I sub07 -a $ASHS_ATLAS_UPENN_PMC -g sub07_mprage.nii.gz -f sub07_tse.nii.gz -w sub07_output_slurm
or
ashs_main.sh -I sub07 -P -a $ASHS_ATLAS_UPENN_PMC -g sub07_mprage.nii.gz -f sub07_tse.nii.gz -w sub07_output_slurm
the test run is successful (please see the generated log here, https://gist.github.com/npavlovikj/9b089f11283ed98dbe1cfddfa6d6a6b2).
However, when I run ashs_main.sh with the Slurm option (-S) on the same dataset with:
ashs_main.sh -I sub07 -S -q "--mem=60gb --time=168:00:00" -a $ASHS_ATLAS_UPENN_PMC -g sub07_mprage.nii.gz -f sub07_tse.nii.gz -w sub07_output_slurm_parallel
the run fails (please see the generated log here, https://gist.github.com/npavlovikj/4d22d26e713d7406961ee42540927515) with:
**************** !! ERROR !! *******************
Validity check at end of stage 2 detected
missing files.
(multiatlas/tseg_left_train000/greedy_atlas_to_s
ubj_affine.mat and 231 other files).
************************************************
When I submit the job, I can see many ashs_mul* jobs being submitted and running, but ultimately the run fails with the error from above.
Do you know why this error happens only when I use the Slurm option?
Also, do you have any suggestions on how to fix it?
Please let me know if you need any additional information.
Thank you,
Natasha