Skip to content

Keep the first-boot rebuild owed until Limine confirms it - #14872

Open
scottjones wants to merge 5 commits into
omacom:quattrofrom
scottjones:fix/provision-rebuild-verify
Open

scottjones wants to merge 5 commits into
omacom:quattrofrom
scottjones:fix/provision-rebuild-verify

Conversation

@scottjones

@scottjones scottjones commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

An image's first-boot hardware setup now checks that Limine actually rebuilt the boot image, and when it can't confirm that, owner setup asks the owner to fix it before the first login. Greptile flagged this on #14675; the owner-setup half follows @maralcbr's suggestion here.

limine-mkinitcpio exits 0 even when it skipped a kernel: a module mkinitcpio can't build, or a boot partition too full for the new image. omarchy-provision-hardware took that as done, so the machine kept booting the image's generic initramfs, which in the worst case lacks a keyboard module the LUKS prompt needs.

The rebuild now counts only when Limine's output shows every image it started succeeded and nothing reported an error, mkinitcpio's and limine-entry-tool's own error lines included. The check is shared from install/helpers/image-target.sh. Retrying on later boots would fail the same way, so an unconfirmed rebuild finishes the hardware setup with a warning and leaves a marker. That exit 0 also lets a Mac's first boot carry on instead of stopping. Owner setup then rebuilds and checks again before handing off, and while it still can't confirm, the retry screen points the owner to the console to fix it.

A machine whose owner setup has already finished, or an image without one, only gets the warning in the logs, as on quattro today.

Evidence

test/shell.d/image-deferred-hardware-test.sh (Limine stub that prints a real run's output and exits 0) and a new test/shell.d/provision-owner-boot-image-test.sh.

  • Before (the first commit, c41df62): not ok - a kernel Limine skipped is left for owner setup to confirm: the first boot finishes, so a platform's first boot carries on to owner setup
  • After: 49/49 and 5/5. These cover each unconfirmed case leaving a marker, owner setup failing the attempt with the boot-image message, a retry clearing the marker once confirmed, and an earlier failure keeping the usual message. The full suite prints the same not ok lines with and without this change (the count depends on the environment).

Test plan

Repo and temp fixtures (no root, nothing on the machine changes):

  • bash test/shell.d/image-deferred-hardware-test.sh and bash test/shell.d/provision-owner-boot-image-test.sh pass
  • ./test/shell fails only what unchanged quattro fails on the same machine

Real machine (an image-built install on a spare machine or VM with Limine, never a daily machine):

  • A first boot that rebuilds normally leaves omarchy-provision-hardware.service disabled and no marker
  • With the boot partition too full for the new image, the hardware setup finishes, the logs say it couldn't confirm the rebuild, and owner setup's retry screen says to choose Drop to console and fix it
  • From that console, free space on the boot partition and run omarchy-provision-owner: setup finishes and the marker is gone
  • On a Mac image, the same case reaches owner setup instead of "First boot stopped"

Merge Danger

Door: two-way.

Blast Radius: small. It only affects image-built machines whose first-boot rebuild goes through Limine. If a future Limine changes its output wording, or an image ships a module the owner can't fix, owner setup can't confirm the rebuild and comes back on every boot. The owner account and desktop still work (the account is made before this check), but the console doesn't clear the marker: only fixing the cause, or removing /var/lib/omarchy/image/boot-rebuild-unconfirmed, and then running omarchy-provision-owner does.

🤖 Generated with Claude Code

@maralcbr
maralcbr requested a review from omarchybot October 11, 2026 00:57
@maralcbr
maralcbr marked this pull request as ready for review October 11, 2026 01:33
@greptile-apps

greptile-apps Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new blocking issue was found.

Summary

The PR keeps an unconfirmed first-boot rebuild owed until owner setup can confirm it.

  • First-boot setup checks that Limine rebuilt the boot image.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Hardware setup rebuilds boot images] --> B{Rebuild confirmed?}
  B -->|Yes| C[Flush images, then clear marker]
  B -->|No| D[Warn and leave marker]
  D --> E[Owner setup retries rebuild]
  E --> F{Rebuild confirmed?}
  F -->|No| G[Show retry or console]
  G --> E
  F -->|Yes| C
  C --> H[Owner setup can finish]
Loading

Reviews (6) · Last reviewed commit: "Count a killed Limine command as an unco..." · Reviewed by Greptile

@maralcbr maralcbr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how I'd want an unconfirmed Limine rebuild handled, and three tries is the right cap: bounded before the login screen, then today's behaviour with a warning.

I ran it on an Apple Silicon test machine (Arch Linux ARM, mkinitcpio 42.2, limine-mkinitcpio-hook 1.39.0): all 47 tests pass, and a real limine-mkinitcpio run (UKI) is confirmed by limine_rebuild_confirmed, so a healthy rebuild isn't retried.

Two optional follow-ups, neither a regression against quattro (inline).

Comment thread bin/omarchy-provision-hardware Outdated
local output uki initcpio built_uki built_initcpio

output=$(sed 's/\x1b\[[0-9;]*m//g' <<<"$1")
! grep -Eq '^((==> )?(ERROR|Error|error)\b|Failed to |Command failed:|The file: |Invalid '"'"'initrd=|Repeated hash mismatches)' <<<"$output" || return 1

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: limine-entry-tool has two more add-path errors this doesn't catch, Kernel name: '…' is invalid due to FAT32 naming rules… and FAT32 boot partition not found…. Both are in the 1.39 binary. Adding Kernel name: |FAT32 boot partition not found to the alternation covers them. Omarchy's kernel names never hit the first one, so this isn't blocking.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in b3bd7c2, thanks. Kernel name: |FAT32 boot partition not found is in the alternation now, with a test row for each, using the text from the limine-entry-tool 1.40.0 binary. The check itself moved to install/helpers/image-target.sh (omarchy_image_limine_rebuild_confirmed) so owner setup can use it too.

Comment thread bin/omarchy-provision-hardware Outdated
local tries=0

[[ ! -f $rebuild_unconfirmed ]] || tries=$(<"$rebuild_unconfirmed")
[[ $tries =~ ^[0-9]+$ ]] || tries=0

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: ^[0-9]+$ accepts 08, and $(( tries + 1 )) then aborts under set -e ("value too great for base") on every boot. Only this script writes the file, and only 1 or 2, so it takes corruption to get there, but [[ $tries =~ ^[12]$ ]] || tries=0 closes it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moot now, since b3bd7c2 drops the retry counter. The marker file only holds a timestamp line, and nothing reads it. Only whether it exists matters. Good catch though.

@maralcbr

Copy link
Copy Markdown
Member

What's the real case where this PR comes into play? It seems like we'd want the user to resolve before rebooting

No field report behind it. Greptile flagged it on #14675, and it's a narrow gap.

The case: an image install defers hardware steps to first boot, and omarchy-provision-hardware rebuilds the initramfs afterwards if a step changed it (MODULES drop-ins like the T2/Surface keyboard or NVIDIA early-KMS ones, or the kernel command line). limine-mkinitcpio exits 0 even when it skipped the kernel: a mkinitcpio error on the new modules, or limine-entry-tool failing to copy to the ESP. So we clear the queue, disable the service, and the machine keeps booting the image's generic initramfs. Worst case is a keyboard module that was needed at the LUKS prompt.

Agreed on resolving it before reboot. Both failures are deterministic, so retrying on the next two boots doesn't help. And there's a natural place for it: owner setup runs right after this on tty1 with the owner there, and it already has a Try again / Drop to console prompt.

Suggestion for @scottjones: keep the detection (limine_rebuild_confirmed) and drop the retry counter. On an unconfirmed rebuild, leave a marker that owner setup shows before it hands off, something like "The boot image rebuild couldn't be confirmed (see /var/log/omarchy-install.log). Fix it before rebooting, then run sudo limine-mkinitcpio", with the same Try again option. This replaces what my approval said about the three-try cap.

scottjones added a commit to scottjones/omarchy-mx-mac that referenced this pull request Oct 12, 2026
Retrying an unconfirmed Limine rebuild on later boots fails the same way
each time, and exiting 1 for it stops a Mac's first boot. The deferred
hardware setup now finishes instead: it warns in the journal and the
install log and leaves a marker. Owner setup, with the owner at the
screen, rebuilds and checks again before it hands off, and fails the
attempt while the rebuild still can't be confirmed, so the owner gets
Try again or a console with a message that says what to fix. The check
moves to install/helpers/image-target.sh so both share it, and it now
also catches limine-entry-tool's FAT32 naming and missing boot partition
errors.

Suggested by Marcelo on omacom#14872.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread install/helpers/image-target.sh Resolved
scottjones added a commit to scottjones/omarchy-mx-mac that referenced this pull request Oct 12, 2026
The boot images live on another filesystem than the marker, so a power cut
after the marker's removal reached disk could leave owner setup skipping a
rebuild whose images were never written. The shared rebuild now syncs
before it removes the marker, which also covers owner setup, where nothing
synced afterwards.

Found by Greptile on omacom#14872.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@scottjones

Copy link
Copy Markdown
Contributor Author

Thanks Marcelo, this is a better shape. I pushed b3bd7c2 on top of 440ae5b.

The retry counter is gone. The hardware setup still checks Limine's output (the check now lives in install/helpers/image-target.sh so owner setup can share it). When it can't confirm the rebuild, it warns in the journal and the install log, leaves /var/lib/omarchy/image/boot-rebuild-unconfirmed, and finishes. Owner setup then rebuilds and checks again before it hands off, and fails the attempt while it still can't confirm, as you suggested.

The retry screen then says: "The boot image rebuild couldn't be confirmed (see /var/log/omarchy-install.log). Choose Drop to console to fix what it reports, such as a full boot partition, then run omarchy-provision-owner." That's a bit off your wording. Try again just repeats the same failure, and running limine-mkinitcpio by hand wouldn't clear the marker, since only a confirmed check does. omarchy-provision-owner reruns setup and the check.

One gap I haven't closed: the marker only shows while owner setup is still pending. If the rebuild happens after owner setup has finished (a deferred step exited 75 and the rebuild ran on a later boot), or the image has no pending owner setup, it's a log-only warning, as on quattro today.

One question for you. Right now there's no way past an unconfirmed rebuild. If a future Limine changes the wording this check relies on, or an image ships a module bug the owner can't fix, the check fails on healthy machines and every owner goes through setup again on every boot. The account and desktop still work, since the account is made before the check, but the only way out is the console, deleting the marker by hand. Do you want a third choice on the retry screen, "Continue anyway", that clears the marker and finishes with a warning, like quattro does today? I haven't built it. Your call.

@omarchybot

Copy link
Copy Markdown
Collaborator

I read this pull request and would call it a bug fix: on quattro, omarchy-provision-hardware treats a limine-mkinitcpio run that exits 0 as a confirmed rebuild, even when Limine skipped a kernel. It is not labelled yet, because the classification needs a second opinion from Codex Medium and today's review budget was already spent when it was asked, so the class is unsettled rather than agreed. Nothing was run on a worker and no tests were executed in this pass; it will be classified, reviewed and verified on a later pass.

One thing that is already yours: GitHub reports the branch as conflicting with quattro, which has moved 161 commits since it was branched, so it needs a merge of quattro before it can land.

scottjones and others added 3 commits October 18, 2026 23:18
limine-mkinitcpio skips a kernel it cannot build or install, a full boot
partition for one, and still exits 0. First-boot hardware setup took that
as done: it cleared the rebuild request and disarmed its service, so the
machine kept booting an image without the hardware changes. The rebuild
now counts only when every image Limine started reports success and
nothing reported an error, limine-entry-tool's own "Failed to" lines
included. Otherwise it stays owed for the next boot, for up to three
tries, after which setup finishes with a warning in the journal and the
install log rather than holding every boot on a rebuild.

Found by Greptile on omacom#14675.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Retrying an unconfirmed Limine rebuild on later boots fails the same way
each time, and exiting 1 for it stops a Mac's first boot. The deferred
hardware setup now finishes instead: it warns in the journal and the
install log and leaves a marker. Owner setup, with the owner at the
screen, rebuilds and checks again before it hands off, and fails the
attempt while the rebuild still can't be confirmed, so the owner gets
Try again or a console with a message that says what to fix. The check
moves to install/helpers/image-target.sh so both share it, and it now
also catches limine-entry-tool's FAT32 naming and missing boot partition
errors.

Suggested by Marcelo on omacom#14872.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The boot images live on another filesystem than the marker, so a power cut
after the marker's removal reached disk could leave owner setup skipping a
rebuild whose images were never written. The shared rebuild now syncs
before it removes the marker, which also covers owner setup, where nothing
synced afterwards.

Found by Greptile on omacom#14872.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@scottjones
scottjones force-pushed the fix/provision-rebuild-verify branch from 9b0b309 to fe150ae Compare October 19, 2026 03:29
@scottjones
scottjones requested a review from maralcbr October 19, 2026 03:43
@omarchybot omarchybot added the bug Something isn't working label Oct 21, 2026
omarchybot and others added 2 commits October 21, 2026 22:21
limine-entry-tool 1.40.0 throws an uncaught Java exception when it can't read limine.conf (invalid UTF-8, for one), before it installs the new image. limine-mkinitcpio ignores install_uki's status and exits 0, and mkinitcpio has already reported both images built, so the check confirmed a rebuild whose image never reached the boot partition and cleared the marker. Seen on a worker with real Limine: the UKI on the ESP kept its old hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Codex Medium <noreply@openai.com>
A command killed outright, as the OOM killer would kill limine-entry-tool, prints no error of its own, and limine-mkinitcpio ignores the status and exits 0 after mkinitcpio has already reported the images built. Bash still reports the death as "<script>: line N: <pid> Killed ...", which is the line a worker printed when the entry tool was killed during --add-uki with real Limine; the UKI on the ESP kept its old hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Codex Medium <noreply@openai.com>
@omarchybot omarchybot added verified Omarchy Triage has verified that this issue is ready for final review ready Good to merge labels Oct 21, 2026
@omarchybot

Copy link
Copy Markdown
Collaborator

Reviewed and run on a disposable Omarchy worker with real Limine (limine-mkinitcpio-hook 1.40.0, mkinitcpio 42.2, x86 UKI). The bug is real and this fixes it. I pushed two small commits that close failure modes the output check missed.

Reproduction. I filled the ESP and changed an initramfs input, the way a deferred step would. limine-mkinitcpio exited 0 after printing Failed to copy: … No space left on device and left no UKI on the ESP at all. On quattro, omarchy-provision-hardware then reported "Deferred hardware setup is complete" and left no trace. On this branch, the same run warns and leaves boot-rebuild-unconfirmed. With the omarchy-provision-hardware change reverted, the silent failure comes back. On the owner-setup side, the shared rebuild helper kept the marker while the ESP was still full and cleared it once there was space and the UKI was back. A healthy rebuild on the same machine is confirmed.

Pushed to your branch:

  • 10bfe65: an uncaught Java exception from limine-entry-tool now counts as unconfirmed. With a limine.conf the tool can't decode, it throws MalformedInputException in addUki before installing the image. limine-mkinitcpio still exits 0 and mkinitcpio has already reported both images built, so the check confirmed a rebuild whose UKI never reached the ESP (its hash was unchanged on the worker).
  • 1ba708f: a Limine command killed outright now counts too. Killed during --add-uki (as the OOM killer would kill it), the entry tool prints nothing of its own, and bash's /usr/bin/limine-entry-tool: line 116: <pid> Killed … is the only sign. The pattern matches that bash report for any script Limine runs.

Each fix comes with a regression case in image-deferred-hardware-test.sh. Both cases failed without the pattern and pass with it.

Tests, on the worker: image-deferred-hardware-test.sh 51/51 and provision-owner-boot-image-test.sh 5/5. test/cli passes. The full test/shell suite fails only the seven shell, screenshot and locale tests that fail identically on quattro there. It also passes on a local merge of this branch with current quattro, which changed omarchy-provision-owner since you branched and merges cleanly.

Second opinion: Codex Medium reviewed it over three rounds. It found both issues above, which I then reproduced with real Limine before fixing. Its last round, on 1ba708f, found nothing left to act on. That round agreed with the fixes I had already written up for it, so treat that agreement as not independent.

For the maintainer: two things are yours to weigh, and neither blocks this as written. First, a check that reads another tool's output can't see a failure that prints nothing at all. The complete fix is upstream: limine-mkinitcpio-hook should propagate install_uki's status. Second, Scott's open question about a "Continue anyway" choice on the retry screen. Without it, an unconfirmed rebuild can only be cleared from the console, so a future Limine wording change would send every new owner through setup on every boot until someone removes the marker.

Waiting on the maintainer.

@omarchybot omarchybot removed verified Omarchy Triage has verified that this issue is ready for final review ready Good to merge labels Oct 21, 2026
omarchybot added a commit to kill74/omarchy that referenced this pull request Nov 17, 2026
limine-entry-tool can report an installation error and still exit successfully, including with limine-mkinitcpio-hook 1.41.0. Use Scott Jones's confirmation predicate from omacom#14872 in both deferred hardware setup and its repair migration, so failed installations, incomplete builds and empty runs cannot write the completion marker. Keep the existing retry behavior; the owner-setup handoff in omacom#14872 remains a separate maintainer decision.

Co-Authored-By: Scott Jones <scottajones@gmail.com>
Co-Authored-By: Codex High <noreply@openai.com>
Co-Authored-By: Opus 5.5 High <noreply@anthropic.com>
Co-Authored-By: muse-wonder via pi, thinking high <noreply@openrouter.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants