SCOS 4.22 4.22.0-okd-scos.5 upgrade leaves system booting to `grub>` prompt because `EFI/centos/grub.cfg` is missing

[madpearl]

Describe the bug

Title

SCOS 4.22 upgrade leaves system booting to grub> prompt because EFI/centos/grub.cfg is missing

Body

Describe the bug

After upgrading an OKD worker to 4.22.0-okd-scos.5, the node no longer booted automatically.

Instead, the machine stopped at a bare GRUB prompt:

grub>

The OS deployment itself was healthy and could be booted manually.

The problem was eventually traced to the EFI partition: the firmware booted \EFI\centos\shimx64.efi, but there was no accompanying \EFI\centos\grub.cfg (nor bootuuid.cfg).

Creating these files restored normal unattended boot.

---

Environment

• OKD release:
  4.22.0-okd-scos.5

• SCOS image:

CentOS Stream CoreOS 10.0.20260614-0
Kernel 6.12.0-233.el10.x86_64

• UEFI boot

---

Symptoms

After reboot:

grub>

No menu appeared.

The system could be booted manually using:

set root=(hd1,gpt3)
set prefix=(hd1,gpt3)/grub2
insmod normal
normal

After that, GRUB loaded normally and booted the installed SCOS deployment.

---

Investigation

The deployed OSTree deployment was healthy.

rpm-ostree status

showed:

State: idle
Version: 10.0.20260614-0

ostree admin status:

* fedora-coreos e4202...

bootupctl status:

Component EFI
Installed: grub2-efi-x64
Update: At latest version

The BLS configuration was present:

/boot/grub2/grub.cfg

contains:

blscfg

The BLS entry:

/boot/loader.1/entries/ostree-1.conf

contained the expected kernel and ostree arguments.

---

EFI contents

EFI partition contained:

EFI/
 ├── BOOT/
 ├── centos/
 │    grubx64.efi
 │    shimx64.efi
 │    shim.efi
 │    ...
 └── fedora/
      grub.cfg
      bootuuid.cfg

Notably missing:

EFI/centos/grub.cfg
EFI/centos/bootuuid.cfg

---

Workaround

Copying the Fedora stub configuration into the CentOS directory fixed the problem:

mount /dev/<efi-partition> /mnt/efi

cp /mnt/efi/EFI/fedora/grub.cfg \
   /mnt/efi/EFI/centos/grub.cfg

cp /mnt/efi/EFI/fedora/bootuuid.cfg \
   /mnt/efi/EFI/centos/bootuuid.cfg

After reboot the machine booted normally without manual GRUB interaction.

---

Current status after workaround

BootCurrent: 0004
BootOrder: 0004,...

Kernel command line:

ostree=/ostree/boot.1/fedora-coreos/...

Node boots normally.

---

Question

Is the SCOS migration expected to populate:

EFI/centos/grub.cfg
EFI/centos/bootuuid.cfg

when the firmware boot entry points to:

\EFI\centos\shimx64.efi

or is this a bug in the migration/update process?

---

I'd also attach:

• efibootmgr -v
• ls -R /boot
• ls -R /mnt/efi/EFI
• rpm-ostree status
• bootupctl status

Those will give the maintainers everything they need to reproduce or diagnose the issue.

One final note: because this occurred immediately after upgrading to 4.22.0-okd-scos.5, it would also be worth checking whether other users have reported similar SCOS boot issues before filing a new issue, and if not, this report should provide enough detail to start the investigation.

Version

4.22.0-okd-scos.5

Reproducibility

No response

Log Bundle

No response

[madpearl]

output :

efibootmgr -v
ls -R /boot
ls -R /mnt/efi/EFI
rpm-ostree status
bootupctl status

BootCurrent: 0004
BootOrder: 0004,0005,0000,0003,0006
Boot0000* Integrated NIC 1 Port 1 Partition 1	VenHw(3a191845-5f86-4e78-8fce-c4cff59f9daa)
      dp: 01 04 14 00 45 18 19 3a 86 5f 78 4e 8f ce c4 cf f5 9f 9d aa / 7f ff 04 00
Boot0001* Hard drive C:	VenHw(d6c0639f-c705-4eb9-aa4f-5802d8823de6)feff6500000000000000000000000104f000e0cb02000000cc00e0cb4002e0cb00000000000000000000000000000000000000000000000000000000000000000000000000120000000000000000001c0002010c00d041030a050000000101060000040101060000007fff040050004500520043002000480037003500350020002000460072006f006e007400200020002800620075007300200036003500200064006500760020003000300029000000
      dp: 01 04 14 00 9f 63 c0 d6 05 c7 b9 4e aa 4f 58 02 d8 82 3d e6 / 7f ff 04 00
    data: fe ff 65 00 00 00 00 00 00 00 00 00 00 00 01 04 f0 00 e0 cb 02 00 00 00 cc 00 e0 cb 40 02 e0 cb 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 12 00 00 00 00 00 00 00 00 00 1c 00 02 01 0c 00 d0 41 03 0a 05 00 00 00 01 01 06 00 00 04 01 01 06 00 00 00 7f ff 04 00 50 00 45 00 52 00 43 00 20 00 48 00 37 00 35 00 35 00 20 00 20 00 46 00 72 00 6f 00 6e 00 74 00 20 00 20 00 28 00 62 00 75 00 73 00 20 00 36 00 35 00 20 00 64 00 65 00 76 00 20 00 30 00 30 00 29 00 00 00
Boot0002* BRCM MBA Slot 0400 v21.6.0	BBS(128,BRCM MBA Slot 0400 v21.6.0,0x0)feff0400000000000000000000000200660300c880000000800000c84b0300c800000000000000000000000000000000000000000000000000000000000000000000000000110000020000000000001c0002010c00d041030a0000000001010600051c0101060000007fff04004200520043004d0020004d0042004100200053006c006f0074002000300034003000300020007600320031002e0036002e0030000000
      dp: 05 01 23 00 80 00 00 00 42 52 43 4d 20 4d 42 41 20 53 6c 6f 74 20 30 34 30 30 20 76 32 31 2e 36 2e 30 00 / 7f ff 04 00
    data: fe ff 04 00 00 00 00 00 00 00 00 00 00 00 02 00 66 03 00 c8 80 00 00 00 80 00 00 c8 4b 03 00 c8 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 00 00 02 00 00 00 00 00 00 1c 00 02 01 0c 00 d0 41 03 0a 00 00 00 00 01 01 06 00 05 1c 01 01 06 00 00 00 7f ff 04 00 42 00 52 00 43 00 4d 00 20 00 4d 00 42 00 41 00 20 00 53 00 6c 00 6f 00 74 00 20 00 30 00 34 00 30 00 30 00 20 00 76 00 32 00 31 00 2e 00 36 00 2e 00 30 00 00 00
Boot0003* Integrated NIC 1 Port 1 Partition 1	VenHw(986d1755-b9d0-4f8d-a0da-d1db18672045)
      dp: 01 04 14 00 55 17 6d 98 d0 b9 8d 4f a0 da d1 db 18 67 20 45 / 7f ff 04 00
Boot0004* CoreOS	HD(2,GPT,914f7b4f-7a6e-44fd-80d1-fbbef2802fca,0x1000,0x3f800)/\EFI\centos\shimx64.efi
      dp: 04 01 2a 00 02 00 00 00 00 10 00 00 00 00 00 00 00 f8 03 00 00 00 00 00 4f 7b 4f 91 6e 7a fd 44 80 d1 fb be f2 80 2f ca 02 02 / 04 04 34 00 5c 00 45 00 46 00 49 00 5c 00 63 00 65 00 6e 00 74 00 6f 00 73 00 5c 00 73 00 68 00 69 00 6d 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
Boot0005* debian	HD(1,GPT,cda6f3db-bcd7-425e-bf93-cd52d161bd58,0x800,0x100000)/\EFI\debian\shimx64.efi
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 00 10 00 00 00 00 00 db f3 a6 cd d7 bc 5e 42 bf 93 cd 52 d1 61 bd 58 02 02 / 04 04 34 00 5c 00 45 00 46 00 49 00 5c 00 64 00 65 00 62 00 69 00 61 00 6e 00 5c 00 73 00 68 00 69 00 6d 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
Boot0006* Fedora	HD(2,GPT,914f7b4f-7a6e-44fd-80d1-fbbef2802fca,0x1000,0x3f800)/\EFI\fedora\shimx64.efi
      dp: 04 01 2a 00 02 00 00 00 00 10 00 00 00 00 00 00 00 f8 03 00 00 00 00 00 4f 7b 4f 91 6e 7a fd 44 80 d1 fb be f2 80 2f ca 02 02 / 04 04 34 00 5c 00 45 00 46 00 49 00 5c 00 66 00 65 00 64 00 6f 00 72 00 61 00 5c 00 73 00 68 00 69 00 6d 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
MirrorStatus: Platform does not support address range mirror
DesiredMirroredPercentageAbove4G: 0.00
DesiredMirrorMemoryBelow4GB: false
/boot:
boot  bootupd-state.json  coreos  efi  grub2  loader  loader.1  lost+found  ostree

/boot/coreos:
platforms.json

/boot/efi:

/boot/grub2:
bootuuid.cfg  fonts  grub.cfg  grubenv  i386-pc  locale

/boot/grub2/fonts:
unicode.pf2

/boot/grub2/i386-pc:
acpi.mod              cmostest.mod       f2fs.mod               gcry_sm4.mod              legacycfg.mod    mul_test.mod         pcidump.mod         sfs.mod             usbserial_common.mod
adler32.mod           cmp.mod            fat.mod                gcry_stribog.mod          linux.mod        multiboot.mod        pgp.mod             shift_test.mod      usbserial_ftdi.mod
affs.mod              cmp_test.mod       file.mod               gcry_tiger.mod            loadenv.mod      multiboot2.mod       plainmount.mod      signature_test.mod  usbserial_pl2303.mod
afs.mod               command.lst        font.mod               gcry_twofish.mod          loopback.mod     nativedisk.mod       plan9.mod           sleep.mod           usbserial_usbdebug.mod
afsplitter.mod        configfile.mod     freedos.mod            gcry_whirlpool.mod        ls.mod           net.mod              play.mod            sleep_test.mod      usbtest.mod
ahci.mod              core.img           fs.lst                 gdb.mod                   lsacpi.mod       newc.mod             png.mod             smbios.mod          vbe.mod
all_video.mod         cpio.mod           fshelp.mod             geli.mod                  lsapm.mod        nilfs2.mod           priority_queue.mod  spkmodem.mod        version.mod
aout.mod              cpio_be.mod        functional_test.mod    gettext.mod               lsmmap.mod       normal.mod           probe.mod           squash4.mod         vga.mod
archelp.mod           cpuid.mod          gcry_arcfour.mod       gfxmenu.mod               lspci.mod        ntfs.mod             procfs.mod          strtoull_test.mod   vga_text.mod
asn1.mod              crc64.mod          gcry_aria.mod          gfxterm.mod               luks.mod         ntfscomp.mod         progress.mod        syslinuxcfg.mod     video.lst
at_keyboard.mod       crypto.lst         gcry_blake2.mod        gfxterm_background.mod    luks2.mod        ntldr.mod            pubkey.mod          tar.mod             video.mod
ata.mod               crypto.mod         gcry_blowfish.mod      gfxterm_menu.mod          lvm.mod          odc.mod              pxe.mod             terminal.lst        video_bochs.mod
backtrace.mod         cryptodisk.mod     gcry_camellia.mod      gptsync.mod               lzopio.mod       offsetio.mod         pxechain.mod        terminal.mod        video_cirrus.mod
bfs.mod               cs5536.mod         gcry_cast5.mod         gzio.mod                  macbless.mod     ohci.mod             raid5rec.mod        terminfo.mod        video_colors.mod
biosdisk.mod          ctz_test.mod       gcry_crc.mod           halt.mod                  macho.mod        part_acorn.mod       raid6rec.mod        test.mod            video_fb.mod
bitmap.mod            date.mod           gcry_des.mod           hashsum.mod               mda_text.mod     part_amiga.mod       random.mod          test_asn1.mod       videoinfo.mod
bitmap_scale.mod      datehook.mod       gcry_dsa.mod           hdparm.mod                mdraid09.mod     part_apple.mod       rdmsr.mod           test_blockarg.mod   videotest.mod
blocklist.mod         datetime.mod       gcry_gost28147.mod     hello.mod                 mdraid09_be.mod  part_bsd.mod         read.mod            testload.mod        videotest_checksum.mod
blscfg.mod            disk.mod           gcry_gostr3411_94.mod  help.mod                  mdraid1x.mod     part_dfly.mod        reboot.mod          testspeed.mod       wrmsr.mod
boot.img              diskfilter.mod     gcry_idea.mod          hexdump.mod               memdisk.mod      part_dvh.mod         regexp.mod          tftp.mod            xfs.mod
boot.mod              div.mod            gcry_keccak.mod        hfs.mod                   memrw.mod        part_gpt.mod         reiserfs.mod        tga.mod             xnu.mod
bsd.mod               div_test.mod       gcry_md4.mod           hfsplus.mod               minicmd.mod      part_msdos.mod       relocator.mod       time.mod            xnu_uuid.mod
bswap_test.mod        dm_nv.mod          gcry_md5.mod           hfspluscomp.mod           minix.mod        part_plan.mod        romfs.mod           tr.mod              xnu_uuid_test.mod
btrfs.mod             drivemap.mod       gcry_rfc2268.mod       http.mod                  minix2.mod       part_sun.mod         rsa_sexp_test.mod   trig.mod            xzio.mod
bufio.mod             dsa_sexp_test.mod  gcry_rijndael.mod      increment.mod             minix2_be.mod    part_sunpc.mod       scsi.mod            true.mod            zfs.mod
cat.mod               echo.mod           gcry_rmd160.mod        iorw.mod                  minix3.mod       partmap.lst          search.mod          truecrypt.mod       zfscrypt.mod
cbfs.mod              efiemu.mod         gcry_rsa.mod           iso9660.mod               minix3_be.mod    parttool.lst         search_fs_file.mod  udf.mod             zfsinfo.mod
cbls.mod              ehci.mod           gcry_salsa20.mod       jfs.mod                   minix_be.mod     parttool.mod         search_fs_uuid.mod  ufs1.mod            zstd.mod
cbmemc.mod            elf.mod            gcry_seed.mod          jpeg.mod                  mmap.mod         password.mod         search_label.mod    ufs1_be.mod
cbtable.mod           eval.mod           gcry_serpent.mod       json.mod                  moddep.lst       password_pbkdf2.mod  sendkey.mod         ufs2.mod
cbtime.mod            exfat.mod          gcry_sha1.mod          keylayouts.mod            modinfo.sh       pata.mod             serial.mod          uhci.mod
chain.mod             exfctest.mod       gcry_sha256.mod        keystatus.mod             morse.mod        pbkdf2.mod           setjmp.mod          usb.mod
cmdline_cat_test.mod  ext2.mod           gcry_sha512.mod        ldm.mod                   mpi.mod          pbkdf2_test.mod      setjmp_test.mod     usb_keyboard.mod
cmosdump.mod          extcmd.mod         gcry_sm3.mod           legacy_password_test.mod  msdospart.mod    pci.mod              setpci.mod          usbms.mod

/boot/grub2/locale:
ast.mo  da.mo  de@hebrew.mo  en@arabic.mo    en@greek.mo   en@piglatin.mo  eo.mo  fi.mo  gl.mo  hr.mo  id.mo  ja.mo  ko.mo  lt.mo  nl.mo  pl.mo  pt_BR.mo  ru.mo  sr.mo  tr.mo  vi.mo     zh_TW.mo
ca.mo   de.mo  de_CH.mo      en@cyrillic.mo  en@hebrew.mo  en@quot.mo      es.mo  fr.mo  he.mo  hu.mo  it.mo  ka.mo  lg.mo  nb.mo  pa.mo  pt.mo  ro.mo     sl.mo  sv.mo  uk.mo  zh_CN.mo

/boot/loader.1:
entries

/boot/loader.1/entries:
ostree-1.conf

/boot/lost+found:

/boot/ostree:
fedora-coreos-edd0ed54acaa09ea3dc2e92cca8a7678aa28c0e5aaf358106ddbd47826818727

/boot/ostree/fedora-coreos-edd0ed54acaa09ea3dc2e92cca8a7678aa28c0e5aaf358106ddbd47826818727:
initramfs-6.12.0-233.el10.x86_64.img  vmlinuz-6.12.0-233.el10.x86_64
/mnt/efi/EFI:
BOOT  Dell  centos  fedora

/mnt/efi/EFI/BOOT:
BOOTX64.EFI  fbx64.efi

/mnt/efi/EFI/Dell:
BootOptionCache

/mnt/efi/EFI/Dell/BootOptionCache:
BootOptionCache.dat

/mnt/efi/EFI/centos:
BOOTX64.CSV  bootuuid.cfg  grub.cfg  grubx64.efi  mmx64.efi  shim.efi  shimx64-centos.efi  shimx64.efi

/mnt/efi/EFI/fedora:
bootuuid.cfg  grub.cfg
State: idle
Deployments:
● ostree-unverified-registry:quay.io/okd/scos-content@sha256:ea9c3617059d50829367a3b6fad651bfee2513986828cf1a181bfea5f0818643
                   Digest: sha256:d4b0f31c0bb70cb056fc8f0ee06296f1ff489728a4b2109d126f6bad837b4284
                  Version: 10.0.20260614-0 (2026-06-17T07:56:04Z)
Running as unit: bootupd.service
Component BIOS
  Installed: grub2-tools-1:2.12-48.el10.x86_64
  Update: At latest version
Component EFI
  Installed: grub2-efi-x64-1:2.12-48.el10.x86_64,shim-x64-16.1-1.el10.x86_64
  Update: At latest version
No components are adoptable.
CoreOS aleph version: 37.20230401.3.0
Boot method: EFI
[root@worker-beta05 ~]#

[root@srv-okd ~]# oc get nodes -owide
NAME                           STATUS   ROLES                  AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                          KERNEL-VERSION           CONTAINER-RUNTIME
control-beta01.okd.gebit.dev   Ready    control-plane,master   3y59d   v1.35.3   10.11.1.1     <none>        CentOS Stream CoreOS 10.0.20260614-0 (Coughlan)   6.12.0-233.el10.x86_64   cri-o://1.36.1
control-beta02.okd.gebit.dev   Ready    control-plane,master   3y59d   v1.35.3   10.11.1.2     <none>        CentOS Stream CoreOS 10.0.20260614-0 (Coughlan)   6.12.0-233.el10.x86_64   cri-o://1.36.1
control-beta03.okd.gebit.dev   Ready    control-plane,master   3y59d   v1.35.3   10.11.1.3     <none>        CentOS Stream CoreOS 10.0.20260614-0 (Coughlan)   6.12.0-233.el10.x86_64   cri-o://1.36.1
worker-beta01.okd.gebit.dev    Ready    worker                 3y59d   v1.35.3   10.11.2.1     <none>        CentOS Stream CoreOS 10.0.20260614-0 (Coughlan)   6.12.0-233.el10.x86_64   cri-o://1.36.1
worker-beta02.okd.gebit.dev    Ready    worker                 3y59d   v1.35.3   10.11.2.2     <none>        CentOS Stream CoreOS 10.0.20260614-0 (Coughlan)   6.12.0-233.el10.x86_64   cri-o://1.36.1
worker-beta03.okd.gebit.dev    Ready    worker                 450d    v1.35.3   10.11.2.3     <none>        CentOS Stream CoreOS 10.0.20260614-0 (Coughlan)   6.12.0-233.el10.x86_64   cri-o://1.36.1
worker-beta05.okd.gebit.dev    Ready    worker                 2y66d   v1.35.3   10.11.2.5     <none>        CentOS Stream CoreOS 10.0.20260614-0 (Coughlan)   6.12.0-233.el10.x86_64   cri-o://1.36.1
[root@srv-okd ~]#

[ArthurVardevanyan]

Also running into same issue, on 4.22.0-okd-scos.6 and 4.22.0-okd-scos.7

[nate-duke]

Is there anywhere else we can look for the status of this? We're contemplating an update in early fall to try and catch back up (currently on 4.20.0-okd-scos.17) but would certainly like to avoid a fleet of unbootable systems on getting to 4.22.

i.e. it seems like there should be some way to mitigate and/or fix this before whichever update it is that breaks grub.

Digging around some it feels like coreos/bootupd#1024 is related but I haven't fully internalized all the bits and pieces involved.

[madpearl]

it happens when you have an old boot image on your okd nodes ( fedora core os) . To avoid it I think you have to upgrade the boot image, each node should have a new boot image with CentOS scos10. That is my opinion. You have to check the slack okd group for more information. Usually each upgrade didn't upgrade boot image, it upgrade the OS.

[nate-duke]

I'm assuming the thread referred to is this one. I don't quite understand the proposed solution though.

The boot image is not upgradable during the OS upgrade. The nodes will keep the same Fedora CoreOS image they were originally deployed with (the old OKD image). To apply the new boot image, you must replace the nodes.

There's no way in the SCOS update system to avoid ending up with unbootable nodes? Everyone updating from some mysterious "old" version is going to have to replace all of their nodes at some mysterious point between 4.21 and 4.22?

I don't intend this as harshly as it comes across, It just feels like this is a fundamental flaw in the update logic for CoreOS or maybe a left over artifact from the transition from FCOS to SCOS in this project which should (IMO) be addressed with a revision of the update instructions or more robust processing during big OS updates.

As long as the workaround works that's great it just seems like there's broken process somewhere and the project would be healthier if it were fixed.

[forzamehlano]

So is the workaround to essentially initiate the upgrade and then fix within the grub prompt once it fails or is there a set of proactive steps we can take to create the required structure prior to kicking off the upgrade?

[nate-duke]

@forzamehlano I've not gone through these updates yet so take this with a grain of salt.

The Operating System vendor changed back in the 4.15/16 switch from FCOS to SCOS so this has been lingering since at least then. I tasked ShatGPT with digging through release notes and issues and it seems to believe that the issue is triggered by the updates to shim and/or GRUB in 4.22.0-okd-scos.4 and perhaps contributed to (if not outright caused) by coreos/bootupd#1024.

I believe the workaround can be applied to any system post switch to SCOS. I'm still working on verifying all this stuff first hand or waiting for someone who has already been through this to confirm.

[ibotty]

So is the workaround to essentially initiate the upgrade and then fix within the grub prompt once it fails or is there a set of proactive steps we can take to create the required structure prior to kicking off the upgrade?

You can do these steps before the update. That way, the nodes will boot correctly.

[risochras]

I just ran into this while upgrading an older cluster using the following command (due to another bug):
oc adm upgrade --to-image=registry.ci.openshift.org/origin/release-scos@sha256:65f272bc84b5e235aa7f3026f08d69f00adebc3fd1228ea7f29e99c5580165af --allow-explicit-upgrade --force

I was also able to get the node to boot using the following (grub>) commands:
set root=(hd0,gpt3)
set prefix=(hd0,gpt3)/grub2
insmod normal
normal

[risochras]

I can also verify the fix with copying over the .cfg files from the fedora to the centos folder.
In my case the sda2 Partition had the "EFI-SYSTEM" Label so the mount command was:
mount /dev/sda2 /mnt/efi

Then copied over the 2 .cfg files:
cp /mnt/efi/EFI/fedora/*.cfg /mnt/efi/EFI/centos/

And the node came back without any manual interaction. I think this should be handled automatically when upgrading.