Skip to content

Harden sshd: localhost bind, key before listen - #12099

Closed
chessing234 wants to merge 5 commits into
omacom:quattrofrom
chessing234:fold/sshd-harden-localhost-key-before-listen
Closed

chessing234 wants to merge 5 commits into
omacom:quattrofrom
chessing234:fold/sshd-harden-localhost-key-before-listen

Conversation

@chessing234

@chessing234 chessing234 commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

closing in favor of #9463 (also integrated by #9474), whose current source validates key-only access before starting sshd and preserves prior configuration during rollback. this branch still has listener-exclusivity and failed-rerun bugs, and its loopback-only default conflicts with remote setup; the previous passing-test claim is withdrawn because the test lacks a host-key-generation stub and its daemon output no longer matches the implementation.

#8363 is covered by the older contribution; the broader policy in #8248 remains a separate decision. i inspected the current implementation but did not replay its native VM validation.

Fold key-before-listen ordering from omacom#8365 onto the localhost bind from omacom#11513 so a rejected key never enables sshd.
Write localhost bind and PasswordAuthentication no before enable --now so the first listen is never password-open or world-reachable.
@chessing234
chessing234 force-pushed the fold/sshd-harden-localhost-key-before-listen branch from 51bf446 to 89e20be Compare September 25, 2026 10:42
@llstrk

llstrk commented Sep 27, 2026

Copy link
Copy Markdown

Automated AI review

Community review: Independent automated community review, unaffiliated with the Omarchy team, intended to help prepare PRs for their review.

Verified: the key-before-listen reordering closes the gap from #8363. With --key=not-a-key, a cancelled menu or paste prompt, an empty or invalid paste, an empty GitHub username, a failed GitHub fetch, or GitHub keys that are all invalid, the head exits 1 before anything is installed, written or started. Base had already started sshd on all addresses with password login on, and run ufw limit 22/tcp, before failing on those paths. When host keys already exist, the first start at head already has PasswordAuthentication no.

The main issue is that the new order fails on machines where sshd has never started. The loopback-only default is also a larger behavior change than the rest of the repository reflects.

Setup fails when sshd has never started (no host keys yet)

sshd -t and sshd -T now run before the first systemctl enable --now sshd.service. OpenSSH exits early when no host key loads (sshd.c at V_10_5_P1, "no hostkeys available -- exiting."), before either check completes. On Arch, host keys are created only by sshdgenkeys.service (ssh-keygen -A), which sshd.service pulls in through Wants=/After= when it starts. Omarchy ships sshd disabled, and factory reset deletes the host keys (bin/omarchy-system-factory-reset).

base                                  head
enable --now sshd  -> host keys made  authorize key
ufw limit 22/tcp                      bind_localhost: sshd -t   -> exit 1, drop-in removed
authorize key                         disable_password_auth: sshd -t -> exit 1, removed
sshd -t / sshd -T  -> pass            (start_sshd never reached)  -> exit 1
reload             -> exit 0

Observed in an isolated run with real OpenSSH 10.5p1 against a private config tree, all privileged commands stubbed, and the unit's key-generation step modelled: with a valid key through --key, --gh-keys or paste, the head prints "sshd rejected the localhost bind config; removing it." and then "sshd rejected the hardening config; removing it and leaving passwords on.", exits 1, and never enables sshd. Base exits 0 in the same scenarios. Once host keys exist, the head flow succeeds. This was not run on a real VM with systemd.

Impact: Setup > Security > SSHD fails on fresh installs and after a factory reset, wherever sshd has not run yet, and its messages blame the drop-ins rather than the missing host keys. The shell test cannot see this because its sshd stub accepts -t whether or not sshd has started or host keys exist. The same constraint was raised by the bot on closed PR #8365.

Suggested change: generate host keys before validating, for example sudo systemctl start sshdgenkeys.service (the unit sshd.service itself pulls in) or sudo ssh-keygen -A, then validate, then enable --now.

Loopback-only default removes remote access; the rest of the repository still assumes it

After setup, sshd listens only on 127.0.0.1:22 and [::1]:22, yet the command still ends with "You can now connect with: ssh $USER@$(hostname)". Re-running it on a machine that is already set up writes the drop-in and reloads sshd. Established sessions survive the reload, but new connections from other machines are refused, and the command prints "SSH firewall port left closed" although the earlier ufw limit 22/tcp rule is still in place.

Places that still describe or depend on remote access:

  • omarchy:summary (line 3) still says "open the firewall". The help text says "by default", but no option selects anything else.
  • manual/35-networking.md and manual/48-security.md say the setup opens port 22.
  • test/acceptance.d/security-test.sh asserts a 22/tcp LIMIT ufw rule when ufw is present, so it fails with this change (read, not run).
  • plans/server.md relies on this command for a machine that is reachable before first login.
  • migrations/1788124236.sh tells users with sshd enabled to run this command, which would now switch their server to loopback.
  • bin/omarchy-remove-security-sshd does not remove the new 20-omarchy-localhost.conf.
  • The omarchy-iso test harness reaches the VM over forwarded guest port 22 with a new connection per call, so artifact collection after the acceptance suite would likely be refused (inferred from omarchy-iso source, not run).

Impact: the command that exists to enable SSH access produces a server no other machine can reach. For #8248, the omarchy-iso unattended path still enables sshd on all interfaces with ufw allow ssh, and existing machines are not migrated, so "Fixes #8248" covers only this opt-in command.

Suggested change: whether loopback should be the default is a maintainer decision. If it is, update the summary, success message, manuals, acceptance test, removal command and callers above together. If not, keep remote access as the default and offer loopback as an explicit option.

"Localhost bind only" is printed without being checked

  • When sshd -t rejects the new drop-in, bind_localhost removes it and returns 0, and setup continues. With an existing AddressFamily inet, OpenSSH rejects ListenAddress ::1. In the isolated run (ufw present) the command exited 0 and printed "localhost bind only" while sshd's effective config (sshd -T) listed only 0.0.0.0:22.
  • ListenAddress entries accumulate rather than first-wins (servconf.c). An existing ListenAddress 0.0.0.0 in any drop-in still gives 0.0.0.0:22 plus loopback, and the same message is printed.
  • When ufw is absent, open_firewall now prints nothing; the old "UFW is not installed" line is gone.

Impact: the command can report a loopback-only server while sshd's effective config still listens on all addresses.

Suggested change: check the effective listeners with sshd -T (as is already done for password settings), and report or fail when they are not loopback-only.

A failed re-run leaves the loopback drop-in for the next restart

The 20- drop-in is written before disable_password_auth, and nothing removes it when that step fails. In the isolated run, a re-run on an active server with an existing administrator PasswordAuthentication yes exited 1 and left the 20- drop-in on disk without a reload. Base leaves no Omarchy drop-in behind in the same scenario.

Impact: the next sshd restart (a reboot, or Arch's 70-openssh-restart-sshd.hook on an openssh upgrade) would apply loopback-only without notice, after a run that reported failure.

Suggested change: remove the 20- drop-in when a later step fails, or write it only after disable_password_auth succeeds and reload an already-running sshd after writing it (the existing reload inside disable_password_auth would then run too early).

Key validation before install: valid_key uses ssh-keygen, which comes from the openssh package that is now installed only after the key step. Stock Omarchy already has openssh, so this matters only where it is missing: there, a valid key is rejected (simulated by removing ssh-keygen from PATH until the package stub runs), so setup cannot proceed. Suggested change: call install_openssh before the key step; installing the package does not start sshd.

Optional test improvement: the repository test passes at head (6 checks) and at base, and the head test does fail against the base script, so the reordering is detected. Three gaps showed up with mutants of the head script:

Mutant Head test
start_sshd moved after the key step but before bind_localhost and disable_password_auth passes all 6 checks, despite the "starts sshd only after hardening" assertion
sshd already running is never reloaded passes: no case sets SSHD_ACTIVE=1
sudo ufw allow 22/tcp added before the key step passes

The runtime firewall assertion cannot fail: no stub logs ufw calls, and the omarchy-cmd-missing stub always reports ufw as missing. The static grep only matches the literal sudo ufw limit 22 spelling. Checking call order in the log (drop-ins before enable --now, which needs the sudo stub to log install calls), adding an SSHD_ACTIVE=1 case, and a ufw stub that logs, with omarchy-cmd-missing reporting ufw as present in at least one case, would cover these.

Related work: open drafts #9463 and #9474 (#9474 contains #9463) rewrite the same setup path with the same ordering goal and conflict with this PR in both files. #9307 conflicts in the script and #9506 in the test.


Review information

Test scope: head 89e20be1, with both changed files unchanged on the current quattro. Repository shell test, head/base cross runs, test mutants, real sshd -t/-T (OpenSSH 10.5p1) and end-to-end head and base runs of the script, all in an isolated sandbox with privileged, network and firewall commands stubbed. Host-key generation was modelled from the Arch unit files; no VM or real systemd, and no network listeners. The acceptance test and omarchy-iso harness were read, not run. IPv6-disabled kernels were not tested.

AI process: Opus 5.5 Medium coordination and synthesis, Opus 5.5 Xhigh technical review and final fact check, GPT 6 Sol Xhigh search for related issues, Opus 5.5 Medium editorial check.

Opt out: To stop receiving these reviews, reply to this comment saying so.

@chessing234

Copy link
Copy Markdown
Contributor Author

generate host keys with ssh-keygen -A before sshd -t, verify listenaddress via sshd -T, and remove the localhost drop-in if password hardening fails later.

@omarchybot omarchybot added the enhancement New feature or request label Sep 27, 2026
@llstrk

llstrk commented Sep 28, 2026

Copy link
Copy Markdown

Automated AI review

Community review: Independent automated community review, unaffiliated with the Omarchy team, intended to help prepare PRs for their review.

Follow-up to the earlier review and the author's reply, checked at head de72071b.

Outcome: two earlier findings are resolved and the other five remain. The new sshd -T check confirms that the loopback addresses are present, but it does not detect other listeners. Two new findings: the unchanged repository test now fails at this head, and a failed re-run removes an earlier run's loopback drop-in.

Earlier finding Status at de72071b
Setup fails when sshd has never started Resolved
A failed re-run leaves the loopback drop-in Resolved (one new edge case below)
"Localhost bind only" printed without being checked Still present
Loopback-only default vs. the rest of the repository Unchanged
Key validation before install_openssh Unchanged
Test gaps shown by mutants Unchanged, see the new test finding
Overlapping open PRs #9463, #9474, #9307, #9506 Unchanged: all still open and still conflicting

Verified:

  • Host keys before the first check. sudo ssh-keygen -A now runs after the install and before the first sshd -t. With real OpenSSH 10.5p1 and no host keys, the head exits 0 and ends with listeners on 127.0.0.1:22 and [::1]:22 only, with the hardening drop-in in place. The previous head exited 1 in the same scenario. ssh-keygen -A is the same command Arch's sshdgenkeys.service runs.
  • Cleanup after a failed hardening step. When disable_password_auth fails, the 20- drop-in is now removed. On a re-run against an active server with an administrator PasswordAuthentication yes, the head exits 1 with no 20- file and no reload. The previous head left the file behind.

Of the author's three reply points, host-key generation and drop-in removal hold as described. The ListenAddress check is covered in the next section.

The listener check tests presence, not exclusivity

bind_localhost now requires one ListenAddress line for 127.0.0.1 and one for ::1 in sshd -T output, then prints "Localhost bind only." It never rejects other addresses. OpenSSH accumulates ListenAddress entries, so an existing administrator entry stays active next to the new ones:

existing admin setting          head output             effective listeners (sshd -T)
ListenAddress 0.0.0.0           "Localhost bind only."  0.0.0.0:22, 127.0.0.1:22, [::1]:22
ListenAddress ::                "Localhost bind only."  127.0.0.1:22, [::1]:22, [::]:22
AddressFamily inet              drop-in rejected,       0.0.0.0:22
                                removed, exit 0

In the AddressFamily inet row, sshd -t rejects ListenAddress ::1. Because of the "keep soft reject" commit, the script removes the drop-in and continues. It then prints "SSH firewall port left closed (localhost bind only)" and "Perfect!". The new check never runs on this path. When ufw is absent, no firewall line is printed at all, as at the previous head.

Impact: the command can still report a loopback-only server while sshd also listens on non-loopback addresses. The red "sshd is not listening on loopback only" message describes a check the code does not perform.

Suggested change: reject any listener that is not loopback. For example, fail when the following filter is non-empty (tested only on captured sshd -T output, not inside the script):

grep -Ei '^listenaddress[[:space:]]' <<<"$effective_config" |
  grep -Eiv '^listenaddress[[:space:]]+(127\.0\.0\.1|\[::1\]):[0-9]+$'

It is empty for the stock, fresh-install and Port 2222 cases, and it returns the extra line in the 0.0.0.0 and :: cases. Also consider making the sshd -t rejection a failure, or at least not printing "localhost bind only" after the drop-in has been removed.

The repository test fails at this head and runs the real ssh-keygen -A

bash test/shell.d/setup-security-sshd-test.sh (unchanged since the earlier review) exits 1 at de72071b on its first case, with no ok lines. Its sshd -T stub prints only the password settings, so the new listener check fails and the script exits. The PR's test plan still marks this test as passing.

The test's sudo stub also passes unknown commands to exec "$@", so the new sudo ssh-keygen -A runs the real binary against /etc/ssh. In a read-only sandbox this printed four "Could not save your private key in /etc/ssh/..." errors. Run as root where host keys are missing, for example in a container, it would create them in that machine's /etc/ssh (inferred from source, not run).

Impact: the shell suite fails on this branch. Running it can also touch system host keys.

Suggested change: add ListenAddress 127.0.0.1:22 and ListenAddress [::1]:22 to the -T) branch of the sshd stub, and add an ssh-keygen) case to the sudo stub that logs the call instead of running it. With these two changes all 6 checks pass. However, mutants that remove ensure_host_keys, remove the 20- cleanup, disable the listener check, or start sshd before key generation also pass all 6. Cases for missing host keys (key generation before the first sshd -t), an extra listener in -T output, a hardening failure that removes 20-, sshd -t rejecting only the bind drop-in, a failing sshd -T, and SSHD_ACTIVE=1 would cover the new paths.

A failed re-run now also removes an earlier run's loopback drop-in

The new cleanup deletes 20-omarchy-localhost.conf whether or not this run created it. Test setup: a machine already configured by an earlier run, with sshd active, and an administrator then adding PasswordAuthentication yes. A re-run prints "Localhost bind only.", then fails hardening and exits 1. It removes both Omarchy drop-ins without mentioning the bind file. The next restart (a reboot, or the openssh upgrade hook) listens on 0.0.0.0:22 and [::]:22.

Impact: a failed repair silently widens the listener that an earlier successful run had set, although on a stock setup ufw still blocks port 22 unless it was opened earlier.

Suggested change: record whether the 20- file existed before writing it, and on failure restore the previous content instead of deleting it.

Note: ssh-keygen -A exits 0 even when it cannot write keys, for example on a read-only /etc/ssh. ensure_host_keys therefore cannot detect the failure, and the later messages blame the drop-ins. This is rare, and the script still exits 1.

Unchanged findings: the loopback default still conflicts with the summary line ("open the firewall"), the success message, manual/35-networking.md, manual/48-security.md, the 22/tcp LIMIT assertion in test/acceptance.d/security-test.sh, bin/omarchy-remove-security-sshd, plans/server.md and migrations/1788124236.sh. "Fixes #8248" still covers only this opt-in command. Key validation still runs before install_openssh. The details in the earlier review still apply.


Review information

Test scope: head de72071b, with both changed files unchanged on the current upstream quattro branch. The head, previous head (89e20be1) and base scripts were run end to end with real OpenSSH 10.5p1 sshd -t/-T and ssh-keygen -A against private config trees. Also run: the repository shell test at each revision, a patched test and test mutants. Everything ran in an isolated sandbox, with privileged, systemd, package and firewall commands stubbed and the host-key generation of Arch's sshd.service modelled. No VM, real systemd or network listeners. The acceptance test and omarchy-iso harness were read, not run, and IPv6-disabled kernels were assessed from source only.

AI process: Opus 5.5 Medium coordination and synthesis, Opus 5.5 Xhigh technical review and final fact check, GPT 6 Sol Xhigh search for related issues, Opus 5.5 Medium editorial check.

Opt out: To stop receiving these reviews, reply to this comment saying so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants