Unable to build some images from Docker - While performing build: conveyor failed to get: open

[asiomchen]

Before you report an issue...

• Have you searched for a duplicate report?
• Have you replicated the problem on the latest release?
• Have you checked the latest documentation?
• Are you using an up-to-date system?

Version of Singularity

What version of Singularity are you using? Run:

$ singularity --version
singularity-ce version 4.4.1

Describe the bug

Trying to build some images from Docker images fails e.g pixi and pixi-derived images

To Reproduce

Steps to reproduce the behavior:

docker pull ghcr.io/prefix-dev/pixi:latest
singularity build pixi.sif docker-daemon:ghcr.io/prefix-dev/pixi:latest

Current output:

INFO:    Starting build...
INFO:    Fetching OCI image...
FATAL:   While performing build: conveyor failed to get: open /home/anton/.singularity/cache/blob/blobs/sha256/7574239a4ba2e6a23ea4ff665e3a4ca9f01f8f2dafb4ff6b1e4808e06df0c8e5: no such file or directory

Expected behavior

Build finishes succesfully

OS / Linux Distribution

Which Linux distribution are you using? Is it up-to-date?
It is WSL2 Ubuntu

PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Installation Method

Compiled from source, the error also exists if installed from DEB

[dtrudg]

There is an incompatibility between the google/go-containerregistry library SingularityCE uses to access the docker-daemon, and newer versions of docker.

Which version of docker are you using (docker version)? As this would allow us to confirm the issue.

As a workaround use docker save to export the image, and then run the singularity build from using docker-archive:<filename>

[asiomchen]

Thank you for the prompt reply.

Docker version 28.5.1, build e180ab8

Confirmed, the code below works:

docker save ghcr.io/prefix-dev/pixi:focal -o pixi-image.docker
singularity build pixi.sif docker-archive:pixi-image.docker

Output:

INFO:    Starting build...
INFO:    Fetching OCI image...
INFO:    Extracting OCI image...
INFO:    Inserting Singularity configuration...
INFO:    Creating SIF file...
INFO:    Build complete: pixi.sif

@dtrudg can you share some more context, what is the affected docker version range? In there some issues on google/go-containerregistry I can track to learn about the fix process?

[dtrudg]

Glad that the workaround worked.

@dtrudg can you share some more context, what is the affected docker version range? In there some issues on google/go-containerregistry I can track to learn about the fix process?

It is a repeat issue, which occurs for different docker versions as the docker daemon / client API changes relatively frequently. See e.g.

• ggcr: docker v29.0.0 breaking changes google/go-containerregistry#2160
• ggcr: github.com/docker/docker v28.0.1 breaks interface google/go-containerregistry#2072

Sometimes the streaming output from the daemon, which the ggcr library consumes, also changes with these API changes.

Unfortunately the situation around docker v28 / v29 is especially confusing, as it seems some breaking changes were implemented and then reverted (I'm not sure exactly when).

General advice is to just build directly from docker:// sources with singularity or do the docker save singularity build docker-archive: dance - as the daemon source is fragile, outside of our control upstream. We do not test against multiple docker versions routinely - only the version in the current CircleCI ubuntu machine image is routinely tested as a docker-daemon source.

[dtrudg]

FWIW - everything is working on a local machine, for me, with docker v29.4.0 - which is the latest. So it appears that with the ggcr update to use the v29 docker client, use against docker versions <29 may now be having issues. This isn't something we can address I'm afraid.

I'll look into getting the version compatibility issues mentioned in documentation / singularity help text etc.

02:47 pm $ docker pull ghcr.io/prefix-dev/pixi:latest
latest: Pulling from prefix-dev/pixi
689b91d88a0f: Pull complete 
1420face9adb: Pull complete 
18c4f161c388: Pull complete 
72b65e0f9413: Pull complete 
Digest: sha256:d2ba1b9be8edc147d2ef5fbf25db78718e0a80daef3b9417b36e05cb16452a52
Status: Downloaded newer image for ghcr.io/prefix-dev/pixi:latest
ghcr.io/prefix-dev/pixi:latest

02:48 pm $ singularity build pixi.sif docker-daemon:ghcr.io/prefix-dev/pixi:latest
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Fetching OCI image...
INFO:    Extracting OCI image...
INFO:    Inserting Singularity configuration...
INFO:    Creating SIF file...
INFO:    Build complete: pixi.sif

02:48 pm $ docker version
Client: Docker Engine - Community
 Version:           29.4.0
 API version:       1.54
 Go version:        go1.26.1
 Git commit:        9d7ad9f
 Built:             Tue Apr  7 08:35:57 2026
 OS/Arch:           linux/amd64
 Context:           default

[dtrudg]

Further digging suggests it might be related to containerd being used for snapshots... google/go-containerregistry#1954 (closed stale, but seems likely this is the real issue).

Reason then for different behaviour on my system would be docker has been updated over time... so didn't switch to the containerd snapshotter (see https://docs.docker.com/engine/storage/containerd/)

@asiomchen - does your docker version output confirm it's using containerd snapshotter? e.g.

docker info -f '{{ .DriverStatus }}'
[[driver-type io.containerd.snapshotter.v1]]

If confirmed.. this is still an upstream issue we can't really work around, short of scanning and manually fixing the layout dir we use to cache blobs. This would be an unreasonably lengthy / IO intensive step on any system that has pulled more than a couple of images.

[asiomchen]

Yes, this is exactly the case for me

➜  ~ docker --version
Docker version 29.4.0, build 9d7ad9f
➜  ~ docker info -f '{{ .DriverStatus }}'
[[driver-type io.containerd.snapshotter.v1]]

Even after docker update and singularity/docker cache cleanup the error still exists in my system. (at least workaround still works)

[dtrudg]

Thanks. That does appear to be the underlying issue then. We can't really tackle it short of fixing the dependency upstream or forking it. I'll try to propose a PR upstream at some point.