fix(squash-lib): mirror host merged-usr symlinks in the squash loader

[prabhakarpujeri]

Problem

The squash loader (squash-lib) assembles its own root directory and creates usr/bin, usr/sbin and usr/lib as plain directories, only symlinking the top-level bin/sbin/lib to their usr counterparts:

for _dir in squash usr/bin usr/sbin usr/lib; do
    mkdir -p "$squashdir/$_dir"
    [[ $_dir == usr/* ]] && ln_r "/$_dir" "${_dir#usr}"
done

On merged-usr hosts where /usr/sbin itself is a symlink (/usr/sbin -> bin, e.g. Fedora 42+), binaries resolved on the host's unified tree are installed into only one of the two split trees in the loader. In particular the busybox module creates its applet links with a shell-side ln_r /usr/bin/busybox "$_path" (no symlink resolution by dracut-install in that path), so sh physically lands in the loader's real usr/sbin/ directory while /bin/sh == /usr/bin/sh stays missing. The kernel then cannot execute the shebang of init-squash.sh:

Run /init as init process
Failed to execute /init (error -2)

This is what breaks the kdump squashfs image on Fedora 42+ in #2454. The main initramfs path does not hit this because create_directories() in dracut.sh mirrors the host layout (if [ -L "/$d" ] → install as symlink).

Fix

Mirror the host layout in squash_install(): if the host directory is a symlink, replicate the same symlink in the loader directory; otherwise create a real directory as before. Binaries installed to either merged tree (usr/sbin/foo or usr/bin/foo) then always land in the single shared tree, matching host semantics.

Root-cause alternative to #2455: instead of special-casing the busybox sh symlink, this fixes the directory structure itself, so it covers every binary — not just sh — for both the busybox and the non-busybox loader paths.

Testing

Reproduced the exact failure from #2454 and verified the fix on two Fedora 44 systems (qemu/KVM guest boot on real hardware, busybox package installed, squashfs handler available, PATH with /usr/sbin first as in the kdump build environment):

check	before	after
loader layout	usr/sbin real dir, sh -> ../bin/busybox stranded there, usr/bin/sh absent	usr/sbin -> bin, usr/bin/sh -> busybox
exec test (chroot into extracted loader tree — the kernel's ENOENT path)	failed to run command '/init': No such file or directory (error -2)	/bin/sh executes (exit 0)
qemu boot of the image	Run /init as init process → Failed to execute /init (error -2)	/init executes, squashfs mounts, switch_root into the inner systemd initramfs, boot proceeds to initrd-switch-root.target (ends there only because the synthetic test image has no real root= device)

Boot logs were captured from the VM serial console for both runs.

shellcheck modules.d/88squash-lib/module-setup.sh clean.

Fixes: #2454

[devkontrol]

The main initramfs path does not hit this because create_directories() in dracut.sh mirrors the host layout
Mirror the host layout in squash_install()

Why not use the exact same code/steps in squash_install that is used by create_directories ?

Can we use inst_symlink instead of ln -sfn ?

I do not mind duplicating some code if we need to.

[devkontrol]

CC @challvy

[prabhakarpujeri]

Thanks for the review! I tried both suggestions; unfortunately they don't work in this stage of the build:

1. inst_symlink() always installs with -D $initdir and does not honor $dstdir; for a symlink source it first installs the resolved target into the main initdir and creates the link there — a plain call cannot stage the link inside $squashdir (which mvs to the final image root).
2. create_directories "/squash_root" ... does route through inst_symlink, but the links it produces via dracut-install's convert_abs_rel() are relative between the staging paths (e.g. usr/sbin -> ../../usr/bin). That text only resolves at boot time thanks to root clamping; during the build it points outside $squashdir, so the subsequent busybox applet loop (ln_r /usr/bin/busybox "$_path") fails for every link: ln: failed to create symbolic link: No such file or directory — empirically 0 of the 247 applet links end up in the image.
3. Overriding initdir locally for the call is not possible either: initdir is readonly (dracut.sh:1469).

What this change does instead is the same intent expressed stage-locally: replicate the host's exact symlink (copy the readlink text) into the loader tree, which is valid both while the loader directory is being populated and at boot, and which makes the loader layout match the main initramfs bit-for-bit. Verified by image listings (247 busybox applet links, usr/sbin -> bin) and by qemu boots of the loader before (Failed to execute /init (error -2)) and after (init runs, image mounts, inner systemd initramfs reaches initrd-switch-root.target).

Happy to converge on a different shape if you prefer — as you said, duplicating a few lines here seemed acceptable, and the prefix/indirection layers of create_directories()/inst_symlink() turned out not to match this use.

[devkontrol]

Thanks @prabhakarpujeri . LGTM. Plan to approve the PR once GitHub is up for the task.

[pvalena]

LGTM.

[bdrung]

create_directories could be changed to not use inst_symlink and then create_directories can be used here as well to avoid duplication.

The difference between ln -snf and inst_symlink is that it will also create the target directory.

While looking at the code I found an issue in create_directories: #2619

[bdrung]

What happens if the symlinks points to a directory that this loop does not create?

[prabhakarpujeri]

Good question. The loop covers exactly the four paths merged-usr symlinks can point into — usr/bin, usr/sbin, usr/lib, squash — so a host symlink's target is always one the loop creates. E.g. on Fedora the host has /usr/sbin -> bin; the loop sees usr/sbin is a link, recreates it as usr/sbin -> bin in the loader, and usr/bin (created by the same loop, either as a real dir or as a symlink as the host demands) is the target bin resolves to. State after the loop on this host: usr/bin dir, usr/sbin -> bin, and busybox + sh reachable from both — which is also verified by the empirical before/after tests in the description (init boots into the inner systemd initramfs with the fix; Failed to execute /init (error -2) without).

Exotic host symlinks pointing outside the loop's set (e.g. /usr/sbin -> ../opt/sbin) would become a dangling symlink in the loader — same as today's behavior, and not harmful (nothing installs through that path, so the link is inert).

[prabhakarpujeri]

Agreed, that's a good follow-up direction. With #2619 (thanks for fixing that) the remaining obstacle for reusing create_directories here is its internal use of inst_symlink — which always stages source resolution with -D $initdir and produces link text between staging paths (usr/sbin -> ../../usr/bin after convert_abs_rel). That text only resolves correctly at boot time, so the busybox applet loop, which installs into $squashdir/usr/sbin/... during the build, hits No such file or directory for every link.

Untangling create_directories from inst_symlink (your idea, not for this PR) would let the loader call it with the squash prefix — happy to fold the loader over to that helper once the coupling is untangled.