Build-context overlay in v1.44 makes using buildah in container harder

[chmeliik]

Issue Description

We use buildah containerized (in a Kubernetes Pod, via Tekton) with native overlay. Up to and including v1.43.1, this only required mounting a host directory over /var/lib/containers/storage or the equivalent rootless path (assuming the host directory is on a filesystem that supports overlay).

Version 1.44 adds an overlay over the build context directory and puts the upper and work directories to /var/tmp by default, which creates another obstacle for containerized usage (perhaps unnecessarily?).

Workarounds:

1. Mount a host dir over /var/tmp as well
2. Set engine.image_copy_tmp_dir = "/var/lib/containers/storage" (or equivalent rootless path) in containers.conf

Looks like a good workaround but doesn't work:

1. Set engine.image_copy_tmp_dir = "storage" in containers.conf
   • this makes buildah attempt to put the overlay under <containers storage path>/tmp, but fails with Error: mounting an overlay over build context directory: creating temporary directory: stat /var/lib/containers/storage/tmp: no such file or directory

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a Containerfile for installing buildah + dependencies:

   FROM registry.access.redhat.com/ubi10/go-toolset:1.26.3 AS go-build
   
   ARG BUILDAH_TAG=v1.44.0
   
   USER 0
   
   # Install dependencies for compiling buildah
   RUN dnf -y install \
           bzip2 \
           glib2-devel \
           gpgme-devel \
           libassuan-devel \
           libseccomp-devel
   
   WORKDIR /build
   
   RUN git clone --branch=$BUILDAH_TAG https://github.com/podman-container-tools/buildah && \
       cd buildah && \
       make bin/buildah
   
   
   FROM registry.access.redhat.com/ubi10/ubi-minimal:10.1
   
   USER 0
   
   RUN microdnf -y install \
           containers-common-extra \
           gpgme \
           libassuan \
           libseccomp
   
   COPY --from=go-build /build/buildah/bin/buildah /usr/local/bin/buildah
   
   COPY <<EOF /etc/containers/storage.conf
   [storage]
   driver = "overlay"
   EOF
   
   ENV BUILDAH_ISOLATION=chroot

2. Build it: buildah build -t buildah:1.44.0 .

3. Try to run a build:

   podman run --rm -i -v /var/lib/containers/storage buildah:1.44.0 bash <<EOF
   mkdir /tmp/buildcontext
   cd /tmp/buildcontext
   echo 'FROM scratch' > Containerfile
   buildah build
   EOF

   Error: mounting an overlay over build context directory: creating overlay scaffolding for build context directory: mount overlay:/var/tmp/buildah-context-855737399/overlay/890669428/merge, data: lowerdir=/tmp/buildcontext,upperdir=/var/tmp/buildah-context-855737399/overlay/890669428/upper,workdir=/var/tmp/buildah-context-855737399/overlay/890669428/work,userxattr: invalid argument
   

4. Try again with image_copy_tmp_dir = "storage":

   podman run --rm -i -v /var/lib/containers/storage buildah:1.44.0 bash <<EOF
   mkdir /tmp/buildcontext
   cd /tmp/buildcontext
   echo 'FROM scratch' > Containerfile
   
   mkdir /etc/containers/containers.conf.d
   printf '[engine]\nimage_copy_tmp_dir = "storage"' >/etc/containers/containers.conf.d/set-tmp-dir.conf
   
   buildah build
   EOF

   Error: mounting an overlay over build context directory: creating temporary directory: stat /var/lib/containers/storage/tmp: no such file or directory
   

Describe the results you received

Containerized usage now requires a non-obvious volume mount or configuration change

Describe the results you expected

Ideally, expected the build context overlay to be created in the container storage dir by default and just work.
Alternatively, expected the engine.image_copy_tmp_dir = "storage" setting to solve the problem.

buildah version output

Version:         1.44.0
Go Version:      go1.26.3 (Red Hat 1.26.3-4.el10_2)
Image Spec:      1.1.1
Runtime Spec:    1.3.0
image Version:   5.40.0
Git Commit:      30a4189415e5d0a3a0c5971307a6432d8ed1a097
Built:           Wed Jun  3 13:55:30 2026
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

buildah info output

{
    "host": {
        "Distribution": {
            "distribution": "\"rhel\"",
            "version": "10.1"
        },
        "MemFree": 3450511360,
        "MemTotal": 33055125504,
        "OCIRuntime": "crun",
        "SwapFree": 8589930496,
        "SwapTotal": 8589930496,
        "arch": "amd64",
        "cpus": 14,
        "hostname": "123b86d8a1a5",
        "kernel": "7.0.9-102.fc43.x86_64",
        "os": "linux",
        "rootless": true,
        "uptime": "7h 40m 7.94s (Approximately 0.29 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphImageStore": "",
        "GraphOptions": null,
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "btrfs",
            "Native Overlay Diff": "true",
            "Supports d_type": "true",
            "Supports shifting": "false",
            "Supports volatile": "true",
            "Using metacopy": "false"
        },
        "GraphTransientStore": false,
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/containers/storage"
    }
}

Provide your storage.conf

[storage]
driver = "overlay"

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[very-doge-wow]

We're hitting the same problem with STORAGE_DRIVER=vfs and BUILDAH_ISOLATION=chroot in a GitLab CI runner (Kubernetes executor).

Our setup uses a custom image based on Wolfi with buildah, where we build OCI images inside the CI job. With v1.44, the unconditional overlay mount over the build context fails even though we explicitly use the vfs storage driver:

Error: mounting an overlay over build context directory: creating overlay scaffolding for build context directory: mount overlay:/var/tmp/buildah-context-852256633/overlay/377000136/merge, data: lowerdir=/builds/...,upperdir=/var/tmp/buildah-context-852256633/overlay/377000136/upper,workdir=/var/tmp/buildah-context-852256633/overlay/377000136/work: invalid argument

The build context overlay appears to bypass the storage driver setting entirely - setting STORAGE_DRIVER=vfs, passing --storage-driver vfs on the CLI, or pointing CONTAINERS_STORAGE_CONF to a custom config with driver = "vfs" all have no effect. The overlay mount is attempted regardless.

We had to pin buildah to 1.43.1 as a workaround. It would be great if the build context overlay either respected the configured storage driver or had a fallback/opt-out when the overlay mount fails.

[prime000000]

Another issue I found with this overlay mount is that it is not marked as volatile (does not have volatile mount option). Thus, unmount of this overlay on a busy CI host is taking really long time.