Skip to content

platformSetupContextDirectoryOverlay should be skipped when --isolation=chroot is set #6910

Description

@rpf3

Issue Description

I am having issues running buildah in my CI environment.

  • Gitlab Kubernetes Executor
  • EKS with EC2 on AL2023
  • buildah version 1.44.0 (via Nix)

After a lot of debugging with some assistance from Claude, I've generated the following summary of what I think is going on.


Problem

When running buildah build --isolation=chroot --storage-driver=vfs inside a Kubernetes pod (GitLab Kubernetes executor, EKS, cgroupv2, AL2023 kernel 6.1), the build fails before any Dockerfile steps execute:
Error: mounting an overlay over build context directory: creating overlay scaffolding
for build context directory: mount overlay:/var/tmp/buildah-context-xxx/overlay/xxx/merge,
data: lowerdir=/builds/myproject,upperdir=.../upper,workdir=.../work,userxattr: invalid argument
The pod runs on an overlay rootfs (containerd snapshots) without /dev/fuse access. The kernel rejects the userxattr overlay mount with EINVAL.

Root Cause

platformSetupContextDirectoryOverlay in imagebuildah/build.go is called unconditionally before any Dockerfile stage is parsed. It attempts to mount an overlay over the build context directory regardless of --isolation or --storage-driver flags. These are separate code paths — --storage-driver=vfs controls image layer storage via containers/storage but has no effect on the build context overlay, which uses pkg/overlay.MountWithOptions directly.

Why This Is Unexpected

Both --isolation=chroot and --storage-driver=vfs signal that the caller has opted out of overlayfs machinery — either because the environment doesn't support it or because they've explicitly chosen simpler semantics. The build context overlay ignores both signals.
Chroot isolation specifically does not benefit from CoW semantics over the build context the way rootless isolation does. Skipping platformSetupContextDirectoryOverlay when --isolation=chroot is set would be consistent with the existing opt-out semantics of that flag.

Steps to Reproduce

  1. Run buildah build --isolation=chroot --storage-driver=vfs inside a Kubernetes pod where:
  • The pod rootfs is an overlay mount (standard containerd setup)
  • /dev/fuse is not available (no FUSE device plugin)
  • cgroupv2 eBPF device filtering is active
  1. Observe failure before STEP 1 executes

Debugging Notes

  • Setting BUILDAH_ISOLATION=chroot via environment variable has no effect — the error occurs in the Nix-packaged binary's startup sequence before flags are parsed
  • --storage-driver=vfs has no effect on the context overlay — confirmed via source inspection that these are separate code paths
  • No containers.conf option or environment variable exists to disable platformSetupContextDirectoryOverlay
    -The issue does not reproduce with apt buildah 1.33.7 (Ubuntu noble), which predates the build context overlay feature introduced in the 1.34–1.38 range
    -Workarounds tried: --no-pivot, BUILDAH_TMPDIR=/tmp, copying build context to /tmp — none affect the context overlay mount
  • privileged: true on the pod resolves the issue by bypassing cgroupv2 device filtering, confirming the root cause is /dev/fuse access

Related ### Issues

#4715 — overlay mount failure in GitLab runner, similar environment constraints, predates the build context overlay feature but shows the same class of problem
#4727userxattr overlay error on overlay rootfs, same underlying kernel rejection
#6116 — fuse-overlayfs incompatibility in 1.39, related overlay machinery

Steps to reproduce the issue

Steps to reproduce the issue
1.
2.
3.

Describe the results you received

Describe the results you received

Describe the results you expected

Describe the results you expected

buildah version output

If you are unable to run `buildah version` for any reason, please provide the output of `rpm -q buildah` or `apt list buildah`.

buildah info output

If you are unable to run `buildah info` for any reason, please provide the operating system and its version and the architecture you are running.

Provide your storage.conf

no explicit storage.conf file

Upstream Latest Release

Yes

Additional environment details

AWS, EKS with EC2, AL2023, Gitlab Runner Kubernetes executor

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions