Issue Description
I am having issues running buildah in my CI environment.
- Gitlab Kubernetes Executor
- EKS with EC2 on AL2023
- buildah version 1.44.0 (via Nix)
After a lot of debugging with some assistance from Claude, I've generated the following summary of what I think is going on.
Problem
When running buildah build --isolation=chroot --storage-driver=vfs inside a Kubernetes pod (GitLab Kubernetes executor, EKS, cgroupv2, AL2023 kernel 6.1), the build fails before any Dockerfile steps execute:
Error: mounting an overlay over build context directory: creating overlay scaffolding
for build context directory: mount overlay:/var/tmp/buildah-context-xxx/overlay/xxx/merge,
data: lowerdir=/builds/myproject,upperdir=.../upper,workdir=.../work,userxattr: invalid argument
The pod runs on an overlay rootfs (containerd snapshots) without /dev/fuse access. The kernel rejects the userxattr overlay mount with EINVAL.
Root Cause
platformSetupContextDirectoryOverlay in imagebuildah/build.go is called unconditionally before any Dockerfile stage is parsed. It attempts to mount an overlay over the build context directory regardless of --isolation or --storage-driver flags. These are separate code paths — --storage-driver=vfs controls image layer storage via containers/storage but has no effect on the build context overlay, which uses pkg/overlay.MountWithOptions directly.
Why This Is Unexpected
Both --isolation=chroot and --storage-driver=vfs signal that the caller has opted out of overlayfs machinery — either because the environment doesn't support it or because they've explicitly chosen simpler semantics. The build context overlay ignores both signals.
Chroot isolation specifically does not benefit from CoW semantics over the build context the way rootless isolation does. Skipping platformSetupContextDirectoryOverlay when --isolation=chroot is set would be consistent with the existing opt-out semantics of that flag.
Steps to Reproduce
- Run
buildah build --isolation=chroot --storage-driver=vfs inside a Kubernetes pod where:
- The pod rootfs is an overlay mount (standard containerd setup)
/dev/fuse is not available (no FUSE device plugin)
- cgroupv2 eBPF device filtering is active
- Observe failure before STEP 1 executes
Debugging Notes
- Setting
BUILDAH_ISOLATION=chroot via environment variable has no effect — the error occurs in the Nix-packaged binary's startup sequence before flags are parsed
--storage-driver=vfs has no effect on the context overlay — confirmed via source inspection that these are separate code paths
- No
containers.conf option or environment variable exists to disable platformSetupContextDirectoryOverlay
-The issue does not reproduce with apt buildah 1.33.7 (Ubuntu noble), which predates the build context overlay feature introduced in the 1.34–1.38 range
-Workarounds tried: --no-pivot, BUILDAH_TMPDIR=/tmp, copying build context to /tmp — none affect the context overlay mount
privileged: true on the pod resolves the issue by bypassing cgroupv2 device filtering, confirming the root cause is /dev/fuse access
Related ### Issues
#4715 — overlay mount failure in GitLab runner, similar environment constraints, predates the build context overlay feature but shows the same class of problem
#4727 — userxattr overlay error on overlay rootfs, same underlying kernel rejection
#6116 — fuse-overlayfs incompatibility in 1.39, related overlay machinery
Steps to reproduce the issue
Steps to reproduce the issue
1.
2.
3.
Describe the results you received
Describe the results you received
Describe the results you expected
Describe the results you expected
buildah version output
If you are unable to run `buildah version` for any reason, please provide the output of `rpm -q buildah` or `apt list buildah`.
buildah info output
If you are unable to run `buildah info` for any reason, please provide the operating system and its version and the architecture you are running.
Provide your storage.conf
no explicit storage.conf file
Upstream Latest Release
Yes
Additional environment details
AWS, EKS with EC2, AL2023, Gitlab Runner Kubernetes executor
Additional information
No response
Issue Description
I am having issues running
buildahin my CI environment.After a lot of debugging with some assistance from Claude, I've generated the following summary of what I think is going on.
Problem
When running
buildah build --isolation=chroot --storage-driver=vfsinside a Kubernetes pod (GitLab Kubernetes executor, EKS, cgroupv2, AL2023 kernel 6.1), the build fails before any Dockerfile steps execute:Error: mounting an overlay over build context directory: creating overlay scaffolding
for build context directory: mount overlay:/var/tmp/buildah-context-xxx/overlay/xxx/merge,
data: lowerdir=/builds/myproject,upperdir=.../upper,workdir=.../work,userxattr: invalid argument
The pod runs on an overlay rootfs (containerd snapshots) without /dev/fuse access. The kernel rejects the userxattr overlay mount with EINVAL.
Root Cause
platformSetupContextDirectoryOverlayinimagebuildah/build.gois called unconditionally before any Dockerfile stage is parsed. It attempts to mount an overlay over the build context directory regardless of--isolationor--storage-driverflags. These are separate code paths —--storage-driver=vfscontrols image layer storage viacontainers/storagebut has no effect on the build context overlay, which usespkg/overlay.MountWithOptionsdirectly.Why This Is Unexpected
Both
--isolation=chrootand--storage-driver=vfssignal that the caller has opted out of overlayfs machinery — either because the environment doesn't support it or because they've explicitly chosen simpler semantics. The build context overlay ignores both signals.Chroot isolation specifically does not benefit from CoW semantics over the build context the way rootless isolation does. Skipping
platformSetupContextDirectoryOverlaywhen--isolation=chrootis set would be consistent with the existing opt-out semantics of that flag.Steps to Reproduce
buildah build --isolation=chroot --storage-driver=vfsinside a Kubernetes pod where:/dev/fuseis not available (no FUSE device plugin)Debugging Notes
BUILDAH_ISOLATION=chrootvia environment variable has no effect — the error occurs in the Nix-packaged binary's startup sequence before flags are parsed--storage-driver=vfshas no effect on the context overlay — confirmed via source inspection that these are separate code pathscontainers.confoption or environment variable exists to disableplatformSetupContextDirectoryOverlay-The issue does not reproduce with apt buildah 1.33.7 (Ubuntu noble), which predates the build context overlay feature introduced in the 1.34–1.38 range
-Workarounds tried:
--no-pivot,BUILDAH_TMPDIR=/tmp, copying build context to/tmp— none affect the context overlay mountprivileged: trueon the pod resolves the issue by bypassing cgroupv2 device filtering, confirming the root cause is/dev/fuseaccessRelated ### Issues
#4715 — overlay mount failure in GitLab runner, similar environment constraints, predates the build context overlay feature but shows the same class of problem
#4727 —
userxattroverlay error on overlay rootfs, same underlying kernel rejection#6116 — fuse-overlayfs incompatibility in 1.39, related overlay machinery
Steps to reproduce the issue
Steps to reproduce the issue
1.
2.
3.
Describe the results you received
Describe the results you received
Describe the results you expected
Describe the results you expected
buildah version output
If you are unable to run `buildah version` for any reason, please provide the output of `rpm -q buildah` or `apt list buildah`.buildah info output
If you are unable to run `buildah info` for any reason, please provide the operating system and its version and the architecture you are running.Provide your storage.conf
no explicit storage.conf fileUpstream Latest Release
Yes
Additional environment details
AWS, EKS with EC2, AL2023, Gitlab Runner Kubernetes executor
Additional information
No response