qstat_stderr : no matching hostkey found for key ED25519 SHA256

[toluadeyelu]

While running the run_gemma.sh script from either Legion/Myriad, I am getting the following qstat error:

Tue Jul 31 16:39:10 2018 : \$qstat_stderr : no matching hostkey found for key ED25519 SHA256:waqBYWAb+g1lwUGWz8ku4M48McIBWGCdpMoU8l8j6tU
ssh_keysign: no reply
sign using hostkey ssh-ed25519 SHA256:waqBYWAb+g1lwUGWz8ku4M48McIBWGCdpMoU8l8j6tU failed

Previously, I had RSA (2048) ssh keys set up for logging into both Legion and Myriad.

After getting this error, I generated a ED25519 ssh key using the following command ssh-keygen -t ed25519 and added it to the ~/.ssh/authorized_keys in both Legion/Myriad.

Even after this, I am getting an ssh key related error.

My detailed stderr file looks like this:

2018/07/31 16:39:10] [DEBUG] qstat command is : ssh myriad.rc.ucl.ac.uk qstat
Tue Jul 31 16:39:10 2018 : About to run     command ssh myriad.rc.ucl.ac.uk qstat at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Executor/SpawnHpcSgeRunner.pm line 201.
Tue Jul 31 16:39:10 2018 : Finished running command ssh myriad.rc.ucl.ac.uk qstat at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Executor/SpawnHpcSgeRunner.pm line 205.
Tue Jul 31 16:39:10 2018 : \$qstat_stdout : job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
  29189 2.75020 QRLOGIN    ucbttad      r     07/31/2018 16:37:14 Bagpuss@node-h00a-021              1        
  29191 2.75020 CathGemma. ucbttad      r     07/31/2018 16:38:04 Bagpuss@node-h00a-021              1 1
  29192 2.31124 CathGemma. ucbttad      r     07/31/2018 16:38:13 Bagpuss@node-h00a-021              1 1
Tue Jul 31 16:39:10 2018 : \$qstat_exit   : 0 at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Executor/SpawnHpcSgeRunner.pm line 207.
Tue Jul 31 16:39:10 2018 : \$qstat_stderr : no matching hostkey found for key ED25519 SHA256:waqBYWAb+g1lwUGWz8ku4M48McIBWGCdpMoU8l8j6tU
ssh_keysign: no reply
sign using hostkey ssh-ed25519 SHA256:waqBYWAb+g1lwUGWz8ku4M48McIBWGCdpMoU8l8j6tU failed
[2018/07/31 16:39:10] [DEBUG] Dumper is $VAR1 = {
          'exit' => 0,
          'stdout' => 'job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
  29189 2.75020 QRLOGIN    ucbttad      r     07/31/2018 16:37:14 Bagpuss@node-h00a-021              1        
  29191 2.75020 CathGemma. ucbttad      r     07/31/2018 16:38:04 Bagpuss@node-h00a-021              1 1
  29192 2.31124 CathGemma. ucbttad      r     07/31/2018 16:38:13 Bagpuss@node-h00a-021              1 1
',
          'command_str' => 'ssh myriad.rc.ucl.ac.uk qstat',
          'command_arr' => [
                             'ssh',
                             'myriad.rc.ucl.ac.uk',
                             'qstat'
                           ],
          'stderr' => 'no matching hostkey found for key ED25519 SHA256:waqBYWAb+g1lwUGWz8ku4M48McIBWGCdpMoU8l8j6tU
ssh_keysign: no reply
sign using hostkey ssh-ed25519 SHA256:waqBYWAb+g1lwUGWz8ku4M48McIBWGCdpMoU8l8j6tU failed
'
        };

[2018/07/31 16:39:10] [DEBUG] Found active running jobs are :  (so any_running_jobs_wanted is : )
Tue Jul 31 16:39:10 2018 : Jobs complete - will now return at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Executor/SpawnHpcSgeRunner.pm line 238.
[2018/07/31 16:39:10] [ INFO] About to load the all-vs-all scans between the starting clusters of the tree
No such ScanData file /lustre/scratch/scratch/ucbttad/gemma_data/scans/3.40.50.1580/working_21.l1st_b64312c5ca5948c6367efcbb2dc743c7.hhconsensus.scan exists at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Scan/ScanData.pm li
ne 53.
        Cath::Gemma::Scan::ScanData::read_from_file("Cath::Gemma::Scan::ScanData", Path::Tiny=ARRAY(0x3fa91e0)) called at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Scan/ScansDataFactory.pm line 48
        Cath::Gemma::Scan::ScansDataFactory::load_scans_data_of_starting_clusters_and_dir("Cath::Gemma::Scan::ScansDataFactory", Path::Tiny=ARRAY(0x3b4e400), "hhconsensus", ARRAY(0x3b60458)) called at /lustre/scratch/scratch/ucbttad/Cath
-Gemma/lib/Cath/Gemma/Scan/ScansDataFactory.pm line 66
        Cath::Gemma::Scan::ScansDataFactory::load_scans_data_of_gemma_dir_set("Cath::Gemma::Scan::ScansDataFactory", Cath::Gemma::Disk::GemmaDirSet=HASH(0x3bdd958), "hhconsensus", ARRAY(0x3b60458)) called at /lustre/scratch/scratch/ucbtt
ad/Cath-Gemma/lib/Cath/Gemma/TreeBuilder.pm line 119
        Cath::Gemma::TreeBuilder::__ANON__(CODE(0x38cc690), Cath::Gemma::TreeBuilder::WindowedTreeBuilder=HASH(0x3d13f88), Cath::Gemma::Disk::Executables=HASH(0x2945958), Cath::Gemma::Executor::SpawnExecutor=HASH(0x39acc18), ARRAY(0x3b60
458), Cath::Gemma::Disk::GemmaDirSet=HASH(0x3bdd958), "hhconsensus", "simple_ordering") called at (eval 519) line 1
        Cath::Gemma::TreeBuilder::WindowedTreeBuilder::__ANON__(Cath::Gemma::TreeBuilder::WindowedTreeBuilder=HASH(0x3d13f88), Cath::Gemma::Disk::Executables=HASH(0x2945958), Cath::Gemma::Executor::SpawnExecutor=HASH(0x39acc18), ARRAY(0x
3b60458), Cath::Gemma::Disk::GemmaDirSet=HASH(0x3bdd958), "hhconsensus", "simple_ordering") called at (eval 521) line 2
        Cath::Gemma::TreeBuilder::WindowedTreeBuilder::build_tree(Cath::Gemma::TreeBuilder::WindowedTreeBuilder=HASH(0x3d13f88), Cath::Gemma::Disk::Executables=HASH(0x2945958), Cath::Gemma::Executor::SpawnExecutor=HASH(0x39acc18), ARRAY(
0x3b60458), Cath::Gemma::Disk::GemmaDirSet=HASH(0x3bdd958), "hhconsensus", "simple_ordering") called at /lustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Compute/Task/BuildTreeTask.pm line 355
        Cath::Gemma::Compute::Task::BuildTreeTask::execute_task(Cath::Gemma::Compute::Task::BuildTreeTask=HASH(0x3b67260), Cath::Gemma::Disk::Executables=HASH(0x2945958), Cath::Gemma::Executor::SpawnExecutor=HASH(0x39acc18)) called at (e
val 541) line 6
        Cath::Gemma::Compute::Task::BuildTreeTask::execute_task(Cath::Gemma::Compute::Task::BuildTreeTask=HASH(0x3b67260), Cath::Gemma::Disk::Executables=HASH(0x2945958), Cath::Gemma::Executor::SpawnExecutor=HASH(0x39acc18)) called at /l
ustre/scratch/scratch/ucbttad/Cath-Gemma/lib/Cath/Gemma/Compute/WorkBatch.pm line 329
        Cath::Gemma::Compute::WorkBatch::execute_task(Cath::Gemma::Compute::WorkBatch=HASH(0x3b66c00), Cath::Gemma::Disk::Executables=HASH(0x2945958), Cath::Gemma::Executor::SpawnExecutor=HASH(0x39acc18)) called at /lustre/scratch/scratc
h/ucbttad/Cath-Gemma/lib/Cath/Gemma/Compute/WorkBatch.pm line 428
        Cath::Gemma::Compute::WorkBatch::execute_from_file("Cath::Gemma::Compute::WorkBatch", Path::Tiny=ARRAY(0x1b8ce28), Cath::Gemma::Disk::Executables=HASH(0x2945958), Cath::Gemma::Executor::SpawnExecutor=HASH(0x39acc18)) called at /l
ustre/scratch/scratch/ucbttad/Cath-Gemma/script/execute_work_batch.pl line 67

[toluadeyelu]

Please find attached the stderror and stdout of the job submitted.
error_msg_cathGeMMA.log

[sillitoe]

Good ticket, thanks.

In airport and plane is about to arrive - will try to replicate tomorrow. Will figure out whether it's a local issue for you or shared for other users.

Did you get this error before the recent work on myriad?

[toluadeyelu]

The error was gotten on both Legion and Myriad. It was after the recent update on Myriad.

[sillitoe]

Hey Tolu,

I can't replicate this error on my account (when submitting to myriad).

I think this is good news - hopefully it means that this is just an issue with your local setup and we can get it to work without too much hassle.

I just deleted the section in the original question that quoted the contents of your authorized keys just to be safe (figured probably best not to have that info on github). However it looks like that's where the problem lies - I shouldn't have been so quick to delete.

You need to make sure that you can ssh back to the login node from the compute node (without password).

# log on to myriad
ssh myriad.rc.ucl.ac.uk
# log on to a compute node 
qrsh -verbose -l h_rt=1:0:0,mem=1G
# log back onto submit node
ssh myriad

Based on the error you pasted, I'm assuming you won't be able to do that last step without typing your password. If so, go through the steps that allowed you to log in from your desktop - but do it from the compute node - something like:

ssh myriad.rc.ucl.ac.uk
ssh-keygen -b 4096 -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Hopefully you can now qrsh onto the compute node, then ssh back onto myriad without a password prompt. If you can do that then you should avoid this error.

[sillitoe]

I can reproduce this, but it looks like this is a warning rather than an error, ie this message is reported to STDERR, but ssh login is still successful.

Probably caused by the upgrade on myriad - might be that ED25519 keys are preferred by default (though other keys are still accepted as fallback).