Support host key certificates

[lanshubo]

Resolving issue416.

Improve server host key verification to support OpenSSH certificates
Main changes:

1. Expose PublicKeyOrCertificate struct and update client::Handler::check_server_key parameter from PublicKey to PublicKeyOrCertificate, allowing callers verify server with host certificates.
2. Add certificate algorithm handling in kex.rs to correctly process certificate-based host keys during the key exchange workflow.
3. During algorithm negotiation, automatically append OpenSSH certificate variants to preferred.keys, enabling certificate-based host key algorithms to be advertised to the server.

Some related PRs to make our full workflow functions properly:
#679
Eugeny/RustCrypto-SSH#3

[maximilian-maisel-bl]

I tried your code with my test application and server and get the following error:

[DEBUG russh::client] ssh id = Standard("SSH-2.0-russh_0.57.0")
[DEBUG russh::client] beginning re-key
[DEBUG russh::sshbuffer] > msg type 20, len 1183
[DEBUG russh::client] < msg type 20, seqn 1, len 715
[DEBUG russh::negotiation] strict kex enabled
[DEBUG russh::client::kex] negotiated algorithms: Names { kex: Name("curve25519-sha256"), key: Rsa { hash: Some(Sha512) }, cipher: Name("aes256-gcm@openssh.com"), client_mac: Name("hmac-sha2-512-etm@openssh.com"), ser
ver_mac: Name("hmac-sha2-512-etm@openssh.com"), server_compression: None, client_compression: None, ignore_guessed: false, strict_kex: true }
[DEBUG russh::sshbuffer] > msg type 30, len 37
[DEBUG russh::client] kex impl continues: ClientKex { cause: Initial, state: "waiting for DH response" }
[DEBUG russh::client] < msg type 31, seqn 2, len 2308
[DEBUG russh::client] drop session
[DEBUG russh::client] kex_done_signal sender was dropped RecvError(())

The SSH server is configured to support only host certificates. The relevant sshd_config line is
HostKeyAlgorithms rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com.
With regular host keys, everything works fine with your change.

[lanshubo]

Hi @maximilian-maisel-bl, the fix is working for my host certificate only SSH server.
This PR mainly fixes the algorithm negotiation err and cert converting err during KEXINIT, which from your log it should be working, or you should see something like this:
Error: russh error for server: No common Key algorithm - ours: ["ssh-ed25519", "ecdsa-sha2-nistp256", "ecdsa-sha2-nistp384", "ecdsa-sha2-nistp521", "rsa-sha2-512", "rsa-sha2-256", "ssh-rsa"], theirs: ["ssh-rsa-cert-v01@openssh.com"]

In your case seems like something error happened after the stage I fixed. Maybe there are some other issue that I haven't met in my environment.

Logs from my working application:
[2026-03-03T08:46:26Z DEBUG russh::client] ssh id = Standard("SSH-2.0-russh_0.57.0")
[2026-03-03T08:46:26Z DEBUG russh::client] beginning re-key
[2026-03-03T08:46:26Z DEBUG russh::sshbuffer] > msg type 20, len 1183
[2026-03-03T08:46:26Z DEBUG russh::client] < msg type 20, seqn 1, len 513
[2026-03-03T08:46:26Z DEBUG russh::client::kex] negotiated algorithms: Names { kex: Name("curve25519-sha256@libssh.org"), key: Rsa { hash: None }, cipher: Name("chacha20-poly1305@openssh.com"), client_mac: Name("hmac-sha2-256-etm@openssh.com"), server_mac: Name("hmac-sha2-256-etm@openssh.com"), server_compression: None, client_compression: None, ignore_guessed: false, strict_kex: false }
[2026-03-03T08:46:26Z DEBUG russh::sshbuffer] > msg type 30, len 37
[2026-03-03T08:46:26Z DEBUG russh::client] kex impl continues: ClientKex { cause: Initial, state: "waiting for DH response" }
[2026-03-03T08:46:26Z DEBUG russh::client] < msg type 31, seqn 2, len 2288
[2026-03-03T08:46:26Z DEBUG russh::sshbuffer] > msg type 21, len 1
[2026-03-03T08:46:26Z DEBUG russh::client] kex impl continues: ClientKex { cause: Initial, state: "waiting for NEWKEYS" }
[2026-03-03T08:46:26Z DEBUG russh::client] < msg type 21, seqn 3, len 1
[2026-03-03T08:46:26Z DEBUG russh::client] kex impl has completed
[2026-03-03T08:46:27Z DEBUG russh::client] kex done

[maximilian-maisel-bl]

Hi @lanshubo, may you share a minimal example program and which SSH server you are using?

I'm using this minimal test program, together with OpenSSH 9.9p2.

[lanshubo]

Hi @maximilian-maisel-bl, I'm connecting to a internal SSH server, so I cannot share the case here.
But I've notice in your case, the check_server_key function is returning false, which will reject all connections.

[maximilian-maisel-bl]

As far as I understand the log output in my case, it already fails before the check_server_key function is called, since the eprintln!("Check server key"); does not print.

[lanshubo]

My use case on client side is basicly the same as your minimal test program besides the return of check_server_key. You may need to add some extra log in russh source code to see where exactly does the program fails, for further debug.

[gvz]

I opened a very similar PR 3 weeks ago, with tests and example server: #641

[Eugeny]

Closing in favor of #748