stderr - lustre cannot find ScanData file

[sayonidas03]

lustre cannot find ScanData file although it seems to exist after the job dies

[2018/08/06 13:54:54] [DEBUG] Found active running jobs are :  (so any_running_jobs_wanted is : )
Mon Aug  6 13:54:54 2018 : Jobs complete - will now return at /lustre/scratch/scratch/ucbtdas/Cath-Gemma/lib/Cath/Gemma/Executor/SpawnHpcSgeRunner.pm line 238.
[2018/08/06 13:54:54] [ INFO] About to load the all-vs-all scans between the starting clusters of the tree
No such ScanData file /lustre/scratch/scratch/ucbtdas/gemma_data/scans/1.10.10.10/working_1.l1st_224576245a17bf0fb19ccbc0029edd15.hhconsensus.scan exists at /lustre/scratch/scratch/ucb
tdas/Cath-Gemma/lib/Cath/Gemma/Scan/ScanData.pm line 53.
        Cath::Gemma::Scan::ScanData::read_from_file("Cath::Gemma::Scan::ScanData", Path::Tiny=ARRAY(0x73ae7708)) called at /lustre/scratch/scratch/ucbtdas/Cath-Gemma/lib/Cath/Gemma/Sca
n/ScansDataFactory.pm line 48
        Cath::Gemma::Scan::ScansDataFactory::load_scans_data_of_starting_clusters_and_dir("Cath::Gemma::Scan::ScansDataFactory", Path::Tiny=ARRAY(0x30d0af8), "hhconsensus", ARRAY(0x2e3
5f10)) called at /lustre/scratch/scratch/ucbtdas/Cath-Gemma/lib/Cath/Gemma/Scan/ScansDataFactory.pm line 66
        Cath::Gemma::Scan::ScansDataFactory::load_scans_data_of_gemma_dir_set("Cath::Gemma::Scan::ScansDataFactory", Cath::Gemma::Disk::GemmaDirSet=HASH(0x30d0a98), "hhconsensus", ARRA
Y(0x2e35f10)) called at /lustre/scratch/scratch/ucbtdas/Cath-Gemma/lib/Cath/Gemma/TreeBuilder.pm line 119
        Cath::Gemma::TreeBuilder::__ANON__(CODE(0x2addf10), Cath::Gemma::TreeBuilder::WindowedTreeBuilder=HASH(0x30d0e40), Cath::Gemma::Disk::Executables=HASH(0x1c17e60), Cath::Gemma::
Executor::SpawnExecutor=HASH(0x2c7ee30), ARRAY(0x2e35f10), Cath::Gemma::Disk::GemmaDirSet=HASH(0x30d0a98), "hhconsensus", "simple_ordering") called at (eval 519) line 1
        Cath::Gemma::TreeBuilder::WindowedTreeBuilder::__ANON__(Cath::Gemma::TreeBuilder::WindowedTreeBuilder=HASH(0x30d0e40), Cath::Gemma::Disk::Executables=HASH(0x1c17e60), Cath::Gem
ma::Executor::SpawnExecutor=HASH(0x2c7ee30), ARRAY(0x2e35f10), Cath::Gemma::Disk::GemmaDirSet=HASH(0x30d0a98), "hhconsensus", "simple_ordering") called at (eval 521) line 2
        Cath::Gemma::TreeBuilder::WindowedTreeBuilder::build_tree(Cath::Gemma::TreeBuilder::WindowedTreeBuilder=HASH(0x30d0e40), Cath::Gemma::Disk::Executables=HASH(0x1c17e60), Cath::G
emma::Executor::SpawnExecutor=HASH(0x2c7ee30), ARRAY(0x2e35f10), Cath::Gemma::Disk::GemmaDirSet=HASH(0x30d0a98), "hhconsensus", "simple_ordering") called at /lustre/scratch/scratch/ucb
tdas/Cath-Gemma/lib/Cath/Gemma/Compute/Task/BuildTreeTask.pm line 355
        Cath::Gemma::Compute::Task::BuildTreeTask::execute_task(Cath::Gemma::Compute::Task::BuildTreeTask=HASH(0x2e54600), Cath::Gemma::Disk::Executables=HASH(0x1c17e60), Cath::Gemma::
Executor::SpawnExecutor=HASH(0x2c7ee30)) called at (eval 541) line 6
        Cath::Gemma::Compute::Task::BuildTreeTask::execute_task(Cath::Gemma::Compute::Task::BuildTreeTask=HASH(0x2e54600), Cath::Gemma::Disk::Executables=HASH(0x1c17e60), Cath::Gemma::
Executor::SpawnExecutor=HASH(0x2c7ee30)) called at /lustre/scratch/scratch/ucbtdas/Cath-Gemma/lib/Cath/Gemma/Compute/WorkBatch.pm line 329
        Cath::Gemma::Compute::WorkBatch::execute_task(Cath::Gemma::Compute::WorkBatch=HASH(0x10b7ec8), Cath::Gemma::Disk::Executables=HASH(0x1c17e60), Cath::Gemma::Executor::SpawnExecu
tor=HASH(0x2c7ee30)) called at /lustre/scratch/scratch/ucbtdas/Cath-Gemma/lib/Cath/Gemma/Compute/WorkBatch.pm line 428
        Cath::Gemma::Compute::WorkBatch::execute_from_file("Cath::Gemma::Compute::WorkBatch", Path::Tiny=ARRAY(0xe5edc8), Cath::Gemma::Disk::Executables=HASH(0x1c17e60), Cath::Gemma::E
xecutor::SpawnExecutor=HASH(0x2c7ee30)) called at /lustre/scratch/scratch/ucbtdas/Cath-Gemma/script/execute_work_batch.pl line 67

[sillitoe]

Definitely encountered this error more than once.

Initial guess is that there's a race condition where we check for the existence of the ScanData file a fraction before it is available on lustre.

Will be difficult to recreate, so I'm going to add a loop to check for this file a few times before dying.

[sillitoe]

@sayonidas03 - just fixed an issue that may have produced the same error message for a different reason.

After the myriad upgrade, the qstat format changed slightly - some of the job ids had white space at the start of the numbers. The code was originally expecting no white space so it effectively failed to recognise any job ids that had less than 6 characters. The following should fix this.

573d421

The error occurred as the parent process was waiting around for expected job ids to finish - the bug meant the parent process thought everything had finished when it hadn't. It tried to start processing the ScanData before it was ready.

[sayonidas03]

Thanks Ian. I will update my repo.

[toluadeyelu]

I found similar error as lustre keeps searching for the ScanData file. Then it exits and dies thereafter. However, upon reloading the job, it starts from where it previously stopped. I found this by looking at the scan folder.
Here is a similar error message to the above:

[2018/08/29 19:02:10] [ WARN] Failed to find ScanData file '/lustre/scratch/scratch/ucbttad/gemma_data/scans/kinase/working_2365.l1st_4df8fa40a24dc0c35216f9ba0fa77aa9.hhconsensus.scan'. It seems that files are not always available immediately on the lustre file system. Will wait a second then try again (count 1) ...
[2018/08/29 19:02:11] [ WARN] Failed to find ScanData file '/lustre/scratch/scratch/ucbttad/gemma_data/scans/kinase/working_2365.l1st_4df8fa40a24dc0c35216f9ba0fa77aa9.hhconsensus.scan'. It seems that files are not always available immediately on the lustre file system. Will wait a second then try again (count 0) ...
/bin/df: ‘/lustre/scratch/scratch/ucbttad/gemma_data/scans/kinase/working_2365.l1st_4df8fa40a24dc0c35216f9ba0fa77aa9.hhconsensus.scan’: No such file or directory
Failed to get ScanData file '/lustre/scratch/scratch/ucbttad/gemma_data/scans/kinase/working_2365.l1st_4df8fa40a24dc0c35216f9ba0fa77aa9.hhconsensus.scan'
HOST: node-h00a-013.myriad.ucl.ac.uk
Results of `/bin/d '/lustre/scratch/scratch/ucbttad/gemma_data/scans/kinase/working_2365.l1st_4df8fa40a24dc0c35216f9ba0fa77aa9.hhconsensus.scan'`:
 at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Scan/ScanData.pm line 82.
Cath::Gemma::Scan::ScanData::read_from_file("Cath::Gemma::Scan::ScanData", Path::Tiny=ARRAY(0xbd1e2658)) called at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Scan/ScansDataFactory.pm line 48
Cath::Gemma::Scan::ScansDataFactory::load_scans_data_of_starting_clusters_and_dir("Cath::Gemma::Scan::ScansDataFactory", Path::Tiny=ARRAY(0x3ea6e60), "hhconsensus", ARRAY(0x3bfbe98)) called at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Scan/ScansDataFactory.pm line 66
Cath::Gemma::Scan::ScansDataFactory::load_scans_data_of_gemma_dir_set("Cath::Gemma::Scan::ScansDataFactory", Cath::Gemma::Disk::GemmaDirSet=HASH(0x3ea6e18), "hhconsensus", ARRAY(0x3bfbe98)) called at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/TreeBuilder.pm line 119 Cath::Gemma::TreeBuilder::__ANON__(CODE(0x3912350), Cath::Gemma::TreeBuilder::WindowedTreeBuilder=HASH(0x3c14e88), Cath::Gemma::Disk::Executables=HASH(0x29ae7b0), Cath::Gemma::Executor::SpawnExecutor=HASH(0x3b3ea38), ARRAY(0x3bfbe98), Cath::Gemma::Disk::GemmaDirSet=HASH(0x3ea6e18), "hhconsensus", "simple_ordering") called at (eval 519) line 1
Cath::Gemma::TreeBuilder::WindowedTreeBuilder::__ANON__(Cath::Gemma::TreeBuilder::WindowedTreeBuilder=HASH(0x3c14e88), Cath::Gemma::Disk::Executables=HASH(0x29ae7b0), Cath::Gemma::Executor::SpawnExecutor=HASH(0x3b3ea38), ARRAY(0x3bfbe98), Cath::Gemma::Disk::GemmaDirSet=HASH(0x3ea6e18), "hhconsensus", "simple_ordering") called at (eval 521) line 2
Cath::Gemma::TreeBuilder::WindowedTreeBuilder::build_tree(Cath::Gemma::TreeBuilder::WindowedTreeBuilder=HASH(0x3c14e88), Cath::Gemma::Disk::Executables=HASH(0x29ae7b0), Cath::Gemma::Executor::SpawnExecutor=HASH(0x3b3ea38), ARRAY(0x3bfbe98), Cath::Gemma::Disk::GemmaDirSet=HASH(0x3ea6e18), "hhconsensus", "simple_ordering") called at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Compute/Task/BuildTreeTask.pm line 355
Cath::Gemma::Compute::Task::BuildTreeTask::execute_task(Cath::Gemma::Compute::Task::BuildTreeTask=HASH(0x1c27c60), Cath::Gemma::Disk::Executables=HASH(0x29ae7b0), Cath::Gemma::Executor::SpawnExecutor=HASH(0x3b3ea38)) called at (eval 541) line 6
Cath::Gemma::Compute::Task::BuildTreeTask::execute_task(Cath::Gemma::Compute::Task::BuildTreeTask=HASH(0x1c27c60), Cath::Gemma::Disk::Executables=HASH(0x29ae7b0), Cath::Gemma::Executor::SpawnExecutor=HASH(0x3b3ea38)) called at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Compute/WorkBatch.pm line 329     Cath::Gemma::Compute::WorkBatch::execute_task(Cath::Gemma::Compute::WorkBatch=HASH(0x3bd4db8), Cath::Gemma::Disk::Executables=HASH(0x29ae7b0), Cath::Gemma::Executor::SpawnExecutor=HASH(0x3b3ea38)) called at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Compute/WorkBatch.pm line 428
Cath::Gemma::Compute::WorkBatch::execute_from_file("Cath::Gemma::Compute::WorkBatch", Path::Tiny=ARRAY(0x1bf83e0), Cath::Gemma::Disk::Executables=HASH(0x29ae7b0), Cath::Gemma::Executor::SpawnExecutor=HASH(0x3b3ea38)) called at /lustre/scratch/scratch/ucbttad/Cath-Gemma/script/execute_work_batch.pl line 67