make it clear when a project is still running / has completed successfully

[sillitoe]

Each GeMMA job is run within a project directory:

data/
data/projects.txt
data/starting_clusters/<project_id>/*.faa

It would be really useful to have a reliable way of knowing the current status of this project - ie is this job still running? if not, did it complete successfully?

Effectively, look at a data directory and identify the status from one of the following:

• invalid data (projects.txt, starting_clusters/, etc)
• not started
• running (current progress?)
• finished with error
• success

Each project data directory should only have one parent process running at a time. This would suggest that a lock file would be useful (containing info like parent PID, submit_dir, etc). A complicating factor is when should we remove this lock file? Each process can spawn multiple child processes and these are pretty independent.

Current plan:

• prepare_research_data.pl produces a lock file when run
• if the lock file exists, this needs a --force argument to overwrite
• the lock file is only removed if a job has completed successfully

[sayonidas03]

Two lines in .stderr files that look like they indicate the status of jobs are:

==> submit_dir.fMY85R4foqn/97c25e4e202ef4a5919213e1642658b9.job_6619138.task_1.stderr <==
Thu Aug 16 12:59:59 2018 : Jobs complete - will now return at /scratch/scratch/ucbtdas/Cath-Gemma/lib/Cath/Gemma/Executor/SpawnHpcSgeRunner.pm line 249.

==> submit_dir.goMUJTbUbc3/d4219f508d89a1592b59fda791663176.job_6586229.task_1.stderr <==
[2018/08/10 11:23:31] [ INFO] Completed processing batch file /scratch/scratch/ucbtdas/gemma_data/submit_dir.goMUJTbUbc3/d4219f508d89a1592b59fda791663176.batch_8a3843e07c790d2f15628824d0ed55c5.freeze