Image-layer hardlinks are not preserved and may be corrupted in MSB VMs

[jakub300]

Hi, thank you for creating MSB. It is a great tool. Recently, I ran into an image that did not work correctly under MSB. While investigating it, I found two issues that appear to be related.

First, MSB does not seem to preserve hard links from image layers. When an image contains hard links, Docker preserves them as hard links in containers. In MSB-created VMs, based on my tests, those hard links are materialized as separate regular files.

Second, in some cases, this appears to lead to data corruption. I encountered this while working with a custom image built on top of the official Rails devcontainer image. It is reproducible with that image by simply trying to use Git:

msb run ghcr.io/rails/devcontainer/images/ruby:2.3.1-3.4.9 \
  -- git clone https://github.com/superradcompany/skills.git /tmp/skills

In the broken state, Git reports unusual errors related to git-remote-https. I confirmed this on two MacBooks with MSB 0.4.6, and also in a Linux VM.

I also asked Codex to create a minimal reproduction on top of your base Debian image. It was able to do so, although the reproduction is unusual.

The image is available in my GHCR, and the Dockerfile is below.

To reproduce the problem, the image had to match some of the complexity of the original image. It creates a large number of hard links and pads the binaries to specific sizes. Only certain combinations of these factors seem to trigger the corruption.

Dockerfile

# syntax=docker/dockerfile:1.7
FROM ghcr.io/superradcompany/debian-systemd:12-2026-05-11

# Install Debian's official Git package. The repro does not build Git from source.
RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates git \
    && rm -rf /var/lib/apt/lists/*

# Build both hardlink groups in one layer. Splitting this into multiple layers
# changes the failure shape and no longer reproduces the helper corruption.
RUN <<'EOF'
set -eu

# Create directories that mirror Git's usual /usr/local install layout.
mkdir -p /usr/local/bin /usr/local/libexec/git-core

# Copy the official Debian git binary into /usr/local so the first hardlink
# group is controlled and independent from the distro package files.
cp /usr/bin/git /usr/local/bin/git
chmod 755 /usr/local/bin/git

# Pad the first hardlink-group target to the size class that triggers the MSB
# corruption. Appending zero bytes keeps the binary executable for `git --version`.
current="$(stat -c %s /usr/local/bin/git)"
target=19834024
head -c "$((target - current))" /dev/zero >> /usr/local/bin/git

# Create a large hardlink group for /usr/local/bin/git. This preceding group is
# needed for MSB to corrupt the later helper group in this repro.
for i in $(seq -w 1 150); do
  ln /usr/local/bin/git "/usr/local/libexec/git-core/git-alias-$i"
done

# Copy Debian's official remote helper and pad it to a larger helper size. The
# helper remains a valid ELF binary under Docker.
cp /usr/lib/git-core/git-remote-http /usr/local/libexec/git-core/git-remote-ftp
current="$(stat -c %s /usr/local/libexec/git-core/git-remote-ftp)"
target=11710936
head -c "$((target - current))" /dev/zero >> /usr/local/libexec/git-core/git-remote-ftp
chmod 755 /usr/local/libexec/git-core/git-remote-ftp

# Create the second hardlink group. Docker preserves valid bytes for all four
# helpers; MSB corrupts this later group after the large preceding group above.
cd /usr/local/libexec/git-core
ln git-remote-ftp git-remote-ftps
ln git-remote-ftp git-remote-http
ln git-remote-ftp git-remote-https
EOF

Reproduction with this image uses a test script that shows both the loss of hard links and the corruption.

Test script

set -e

stat -c "%n inode=%i links=%h size=%s" \
  /usr/local/bin/git \
  /usr/local/libexec/git-core/git-alias-001 \
  /usr/local/libexec/git-core/git-alias-150 \
  /usr/local/libexec/git-core/git-remote-ftp \
  /usr/local/libexec/git-core/git-remote-ftps \
  /usr/local/libexec/git-core/git-remote-http \
  /usr/local/libexec/git-core/git-remote-https

sha256sum \
  /usr/local/bin/git \
  /usr/local/libexec/git-core/git-alias-001 \
  /usr/local/libexec/git-core/git-alias-150 \
  /usr/local/libexec/git-core/git-remote-ftp \
  /usr/local/libexec/git-core/git-remote-ftps \
  /usr/local/libexec/git-core/git-remote-http \
  /usr/local/libexec/git-core/git-remote-https

od -An -tx1 -N8 /usr/local/libexec/git-core/git-remote-https

Save the script as test.sh and run it in both Docker and MSB to compare the results.

Results on my MacBook Pro with MSB 0.4.6:

Results: Docker vs MSB

$ docker run --rm ghcr.io/jakub300/container-images/msb-git-hardlink-corruption-repro:sha-ee82e6c bash -c "$(cat test.sh)"
/usr/local/bin/git inode=305251 links=151 size=19834024
/usr/local/libexec/git-core/git-alias-001 inode=305251 links=151 size=19834024
/usr/local/libexec/git-core/git-alias-150 inode=305251 links=151 size=19834024
/usr/local/libexec/git-core/git-remote-ftp inode=305254 links=4 size=11710936
/usr/local/libexec/git-core/git-remote-ftps inode=305254 links=4 size=11710936
/usr/local/libexec/git-core/git-remote-http inode=305254 links=4 size=11710936
/usr/local/libexec/git-core/git-remote-https inode=305254 links=4 size=11710936
c67250cf55e0b5518459b4a7278c164d54cc8f625535b1f0fefd319d41f82ef1  /usr/local/bin/git
c67250cf55e0b5518459b4a7278c164d54cc8f625535b1f0fefd319d41f82ef1  /usr/local/libexec/git-core/git-alias-001
c67250cf55e0b5518459b4a7278c164d54cc8f625535b1f0fefd319d41f82ef1  /usr/local/libexec/git-core/git-alias-150
c2fafba835badc5856fd06ad555d278dab42342ca204ae233e45ce0aa1a26a80  /usr/local/libexec/git-core/git-remote-ftp
c2fafba835badc5856fd06ad555d278dab42342ca204ae233e45ce0aa1a26a80  /usr/local/libexec/git-core/git-remote-ftps
c2fafba835badc5856fd06ad555d278dab42342ca204ae233e45ce0aa1a26a80  /usr/local/libexec/git-core/git-remote-http
c2fafba835badc5856fd06ad555d278dab42342ca204ae233e45ce0aa1a26a80  /usr/local/libexec/git-core/git-remote-https
 7f 45 4c 46 02 01 01 00

$ msb run ghcr.io/jakub300/container-images/msb-git-hardlink-corruption-repro:sha-ee82e6c -- bash -c "$(cat test.sh)"     
/usr/local/bin/git inode=22477 links=151 size=19834024
/usr/local/libexec/git-core/git-alias-001 inode=23708 links=2 size=19834024
/usr/local/libexec/git-core/git-alias-150 inode=204445 links=151 size=19834024
/usr/local/libexec/git-core/git-remote-ftp inode=205658 links=4 size=11710936
/usr/local/libexec/git-core/git-remote-ftps inode=206375 links=2 size=11710936
/usr/local/libexec/git-core/git-remote-http inode=207092 links=3 size=11710936
/usr/local/libexec/git-core/git-remote-https inode=207809 links=4 size=11710936
c67250cf55e0b5518459b4a7278c164d54cc8f625535b1f0fefd319d41f82ef1  /usr/local/bin/git
c67250cf55e0b5518459b4a7278c164d54cc8f625535b1f0fefd319d41f82ef1  /usr/local/libexec/git-core/git-alias-001
2fd0ab4485a099e2e9bd8e839ada331ad8dea7bd87ea82d5b28b2ebacf490dff  /usr/local/libexec/git-core/git-alias-150
0cf331f80c1d83c8616c286d81feab8e9b2c980fbb9466ba13a90c884a813877  /usr/local/libexec/git-core/git-remote-ftp
472c627f5b855368dddca6a563f91abc198d4dc082bef9de899a5f21f6cc1409  /usr/local/libexec/git-core/git-remote-ftps
bc3d64ed83f07f0c03307b89c2dc042634ecc2da9a2cfc3ab335a8485ebcfcdb  /usr/local/libexec/git-core/git-remote-http
89aa4c69e80e8004f2c7ccb0cfd45ff7472cc0555fb52de95d4dad5374fe02d4  /usr/local/libexec/git-core/git-remote-https
 00 00 00 00 00 00 00 00