Error while building a bootc-generic-iso: Failed to mount image (125): Error: layer not known

[sprat]

I would like the build an ISO from 2 bootc container images (installer + payload), but it currently fails during the build with a Failed to mount image (125): Error: layer not known error, and I am currently trying to understand the root cause.

FYI I run the image-builder CLI inside a container using the official ghcr.io/osbuild/image-builder-cli:latest image, and I think the root cause of the problem is more or less due to the double sandboxing.

However, I think there a problem with the manifest generated by the image-builder CLI, listed here: there's a duplicate org.osbuild.container-deploy stage in the first and second pipeline and I don't understand why. Is that a bug?

Since my build fails in this second duplicate stage, I guess my problem would disappear by fixing the manifest (but I didn't tried yet)

[supakeen]

The duplicate stage is not a bug; we deploy a container for the build pipeline which sets up the buildroot (we use the tooling from this buildroot to perform any further actions). If no separate --bootc-build-ref is passed we use container passed in --bootc-ref, hence the duplication here.

In the case of the generic ISO we also deploy the container in the os-tree pipeline to serve as the filesystem that we will put onto the ISO.

---

I've seen the error about missing layers before and it was flakyness that disappeared after the container itself got rebuilt and then pulled again, I think it was specifically a container that used chunkah though that might be entirely unrelated.

Could you try to do so to see if that 'fixes' the issue. If not it might be worth looking into it deeper because you're definitely not the first one to see that error.

---

I've edited the title to mention the error that's more important here, hope that's OK.

[sprat]

@supakeen, thanks for the feedback, it makes sense. So my initial guess about duplicate stages is completely wrong.

I've already rebuilt both bootc containers multiple times by trashing the build cache & images with podman system prune. But I've tried again to make sure: I removed all the images from the podman image cache (to pull again every base image), rebuilt all my bootc containers and tried to rebuild the ISO again: same error.

I use podman build to build the bootc images, using podman-remote from a WSL distro connected to the rootful socket of Podman Desktop running in Windows.

$ podman system connection list
Name                         URI                                                                     Identity    Default     ReadWrite
podman-machine-default-root  unix:///mnt/wsl/podman-sockets/podman-machine-default/podman-root.sock              true        true

There's no flakyness in my case, I can reproduce the problem all the time.

My observations:

• the first container-deploy always succeeds
• the second container-deployer always fail
• that's strange because they run in separate sandboxes, so some parts of the podman context seems to be shared between the 2 runs.

I don't know how to troubleshoot further, please advise.

Here is the full log: bootc-generic-iso.log.

I also have the same problem with the bootc-installer target as they are doing the same things in the first pipelines & stages.

NOTE: I realize that the bug should probably be moved in the osbuild/osbuild repo.

[supakeen]

Thanks for the additional information. Having the bug here is fine, if anywhere I'd move it to osbuild/image-builder-cli since that's the tool it shows up in. I'm going to report it in Red Hat's JIRA as well so it has more visibility: https://redhat.atlassian.net/browse/HMS-10692

Definitely needs some looking into from our side.

[sprat]

I've double-checked on a Linux machine, same problem, so it's not related to the Windows+WSL environment.