This repository shows how to configure and install NixOS on a cluster of Compute Blade boards.
The repository includes:
- Guide on how to install and maintenance NixOS on your Compute Blade nodes.
- Custom installer images for Compute Module 4 and 5.
- Host configurations for individual Compute Blade nodes.
It is based on ,
which provides NixOS support for Raspberry Pi Compute Modules.
Note
This guide expects that you are an experienced NixOS user, it doesn't cover how to create flake based configuration.
This guide explains how to boot a temporary NixOS installer image and then install the final NixOS system onto the Compute Blade's SSD.
- Uptime Compute Blade DEV Board
- Raspberry Pi Compute Module 4 or 5
- NVMe SSD
- SD Card, only if your Compute Module doesn't have eMMC
- Linux machine with Nix
- (optional) Network Switch with POE+, but it is possible to power Blade with the USB-C
Clone the repository wherever you want to keep your cluster configuration files.
I usually keep mine under ~/.config..
git clone https://github.com/thatwhichisdev/blazing-cluster.git ~/.config/blazing-clusterFor the best experience, add your public SSH key to the configuration. This allows you to SSH into the installer and final system without typing a password.
Modify following sections in /modules/openssh.nix
users.users.nixos.openssh.authorizedKeys.keys = [
"<your public ssh key here>"
];
users.users.root.openssh.authorizedKeys.keys = [
"<your public ssh key here>"
];You can also add extra tools or settings to the installer image if needed.
Keep in mind that this installer system is not the final system. It is only used
to boot the board and run nixos-anywhere.
This repository defines separate installer images for Compute Module 4 and Compute Module 5.
Run following command to generate installer, replace cmX with desired
instalation target cm4 or cm5.
nix --accept-flake-config build .#installer-cmXAfter the build finishes, the generated image will be available under
result/sd-image/nixos-installer-cmX.img.zst
Use this method only if your Compute Module does not have eMMC.
Connect the SD card to your Linux machine and find its device name using lsblk
command. It usually appears as something like /dev/sdX or /dev/mmcblkX. No
mounting needed.
Flash the image with two possible set of tools and replace cmX and target
device with yours.
Flashing with bmaptool:
sudo bmaptool copy --nobmap result/sd-image/nixos-installer-cmX.img.zst /dev/sdXFlashing with zstd and dd:
First, decompress the image
zstd --decompress --force result/sd-image/nixos-installer-cmX.img.zst --output result/sd-image/nixos-installer-cmX.imgSecond, flash the decompressed image to the target device:
sudo dd if=result/sd-image/nixos-installer-cmX.img of=/dev/sdX bs=16M status=progress conv=fsyncWarning
Be careful: use the whole disk, for example /dev/sdX, not a partition like
/dev/sdX1.
As confirmation that flashing completed successfully you will see metrics in the console:
385+1 records in
385+1 records out
1618251606 bytes (1.6 GB, 1.5 GiB) copied, 49.1864 s, 32.9 MB/sUse this method if your Compute Module has onboard eMMC.
This process requires putting the Compute Module into USB boot mode and flashing the eMMC from your Linux machine.
First, we need to install official raspberry's
tool to boot our compute
module over USB using
mass-storage-gadget extension.
You can clone the official repository and build it following the official guide
or a more simple way would be to install it from nixpkgs via nix-shell, for
that just run following command:
nix-shell -p rpibootNow, run rpiboot as sudo to start listening for the compute module over USB:
sudo rpibootYou will see following output
RPIBOOT: build-date 2026/05/09 pkg-version 20250908~162618~bookworm
Please fit the EMMC_DISABLE / nRPIBOOT jumper before connecting the power and USB cables to the target device.
If the device fails to connect then please see https://rpltd.co/rpiboot for debugging tips.
Waiting for BCM2835/6/7/2711/2712...Move the USB switch on the Compute Blade to the USB Type-C position. Then, while holding down the nRPIBOOT button on the blade connect the USB Type-C cable. You will see following:
Directory not specified - trying default /nix/store/5knggxch09wj8qf7pgx6m19z3ib22b18-rpiboot-20250908-162618-bookworm/share/rpiboot/mass-storage-gadget64/
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7/2711/2712...
Second stage boot server
File read: mcb.bin
File read: memsys00.bin
File read: memsys01.bin
File read: memsys02.bin
File read: memsys03.bin
File read: memsys04.bin
File read: memsys05.bin
File read: memsys06.bin
File read: memsys07.bin
File read: memsys08.bin
File read: bootmain
Loading: /nix/store/5knggxch09wj8qf7pgx6m19z3ib22b18-rpiboot-20250908-162618-bookworm/share/rpiboot/mass-storage-gadget64//config.txt
File read: config.txt
Loading: /nix/store/5knggxch09wj8qf7pgx6m19z3ib22b18-rpiboot-20250908-162618-bookworm/share/rpiboot/mass-storage-gadget64//boot.img
File read: boot.img
Second stage boot server doneIf everything finishes successfully, you should be able to see your eMMC via
lsblk:
[nix-shell:~/development/thatwhichisdev/blazing-cluster]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 7.3G 0 disk
├─sda1 8:1 1 1G 0 part
└─sda2 8:2 1 6.3G 0 partFlash the image with two possible set of tools and replace cmX and target
device with yours.
Flashing with bmaptool:
sudo bmaptool copy --nobmap result/sd-image/nixos-installer-cmX.img.zst /dev/sdXFlashing with zstd and dd:
First, decompress the image
zstd --decompress --force result/sd-image/nixos-installer-cmX.img.zst --output result/sd-image/nixos-installer-cmX.imgSecond, flash the decompressed image to the target device:
sudo dd if=result/sd-image/nixos-installer-cmX.img of=/dev/sdX bs=16M status=progress conv=fsyncWarning
Be careful: use the whole disk, for example /dev/sdX, not a partition like
/dev/sdX1.
As confirmation that flashing completed successfully you will see metrics in the console:
385+1 records in
385+1 records out
1618251606 bytes (1.6 GB, 1.5 GiB) copied, 49.1864 s, 32.9 MB/sAfter flashing the installer image, power on the board, if you're using SD card don't forget to insert it.
Once the board boots, find it's IP address on your local network.
You can SSH into the installer with:
ssh root@<hostname>If you added your SSH key to the installer configuration, key-based login should work automatically, otherwise you need to connect external display via HDMI cable and copy the password from the welcome message.
After the board has booted into the temporary installer image, install the final NixOS system with nixos-anywhere.
The final system is installed to the SSD. Disk partitioning and ZFS setup are
handled by disko using the configuration from /modules/disko.nix.
To install system run:
nix run github:nix-community/nixos-anywhere -- --flake .#<system> root@<hostname>Example:
nix run github:nix-community/nixos-anywhere -- --flake .#cb1 root@192.168.0.161When the installation finishes, you should see following in the console:
kernel boot files installed for nixos generation '1-default'
removing obsolete generations in /boot/firmware/nixos...
generational bootloader installed
installation finished!
### Rebooting ###
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '192.168.0.161' (ED25519) to the list of known hosts.
umount: /mnt/var/lib (rpool/safe/var/lib) unmounted
umount: /mnt/var (rpool/system/var) unmounted
umount: /mnt/nix (rpool/local/nix) unmounted
umount: /mnt/home (rpool/safe/home) unmounted
umount: /mnt/boot/firmware unmounted
umount: /mnt/boot unmounted
umount: /mnt (rpool/system/root) unmounted
### Waiting for the machine to become unreachable due to reboot ###
Warning: Permanently added '192.168.0.161' (ED25519) to the list of known hosts.
Warning: Permanently added '192.168.0.161' (ED25519) to the list of known hosts.
Warning: Permanently added '192.168.0.161' (ED25519) to the list of known hosts.
Warning: Permanently added '192.168.0.161' (ED25519) to the list of known hosts.
Warning: Permanently added '192.168.0.161' (ED25519) to the list of known hosts.
ssh: connect to host 192.168.0.161 port 22: Connection refused
### Done! ###Now, connect via SSH and enjoy!
ssh nixos@<hostname>Before building a system configuration, you can check the flake for evaluation errors and other issues.
nix flake checkTo format repository simply run
nix fmtTo change configuration of the running system you can simply run:
nixos-rebuild switch --flake .#<system> --target-host root@<hostname>Otherwise you can clone this repository to the compute blade and run
nixos-rebuild directly on it, in my case I'm running Asahi Linux on M1 Mac, so
I have aarch64 on my main machine and building remotely usually is best choice
for me. If you have different architecture on your main machine builds might
take very long time to build, in that case you can indeed try to apply changes
within the blade itself.
If you wish to update the bootloader EEPROM of your Compute Module boards you
can follow the steps below, unlike the booting Compute Module over USB in
Flash the image onto eMMC section, we actually want to reference to most
recent commits, so for that we'll have to clone the repository and build the
tool.
Clone the official
repository.
git clone https://github.com/raspberrypi/usbboot ~/usbbootUpdate the EEPROM submodule
git submodule update --initInstall the neccessary tools in order to build the binary
nix-shell -p pkg-config libusb1 gcc gnumakeBuild the binary
makeNow you can update the EEPROM
For Compute Module 4:
sudo rpiboot -d recoveryFor Compute Module 5:
sudo rpiboot -d recovery5You will see following:
[nix-shell:~/development/raspberrypi/usbboot]$ sudo ./rpiboot -v -d recovery
RPIBOOT: build-date 2026/07/13 pkg-version local 87d6e032
Please fit the EMMC_DISABLE / nRPIBOOT jumper before connecting the power and USB cables to the target device.
If the device fails to connect then please see https://rpltd.co/rpiboot for debugging tips.
Boot directory 'recovery'
Loading: recovery/bootcode4.bin
Waiting for BCM2835/6/7/2711/2712...Move the USB switch on the Compute Blade to the USB Type-C position. Then, while holding down the nRPIBOOT button on the blade connect the USB Type-C cable. You will see following:
Device located successfully
Loading: recovery/bootcode4.bin
Initialised device correctly
Found serial number 3
last_serial -1 serial 3
Sending bootcode.bin
libusb_bulk_transfer sent 24 bytes; returned 0
Writing 100140 bytes
libusb_bulk_transfer sent 100140 bytes; returned 0
Successful read 4 bytes
Waiting for BCM2835/6/7/2711/2712...
Device located successfully
Loading: recovery/bootcode4.bin
Initialised device correctly
Found serial number 4
last_serial -1 serial 4
Second stage boot server
Received message GetFileSize: config.txt
Loading: recovery/config.txt
File size = 254 bytes
Received message ReadFile: config.txt
File read: config.txt
libusb_bulk_transfer sent 254 bytes; returned 0
Received message GetFileSize: pieeprom.bin
libusb_bulk_transfer sent 0 bytes; returned 0
Cannot open file pieeprom.bin
Received message GetFileSize: pieeprom.upd
libusb_bulk_transfer sent 0 bytes; returned 0
Cannot open file pieeprom.upd
Received message GetFileSize: *USER_SERIAL_NUM*7b335499
{
"USER_SERIAL_NUM": "7b335499"libusb_bulk_transfer sent 0 bytes; returned 0
Received message GetFileSize: *MAC_ADDR*2c:cf:67:22:08:f4
,
"MAC_ADDR": "2c:cf:67:22:08:f4"libusb_bulk_transfer sent 0 bytes; returned 0
Received message GetFileSize: *CUSTOMER_KEY_HASH*0000000000000000000000000000000000000000000000000000000000000000
,
"CUSTOMER_KEY_HASH": "0000000000000000000000000000000000000000000000000000000000000000"libusb_bulk_transfer sent 0 bytes; returned 0
Received message GetFileSize: *BOOT_ROM*000048b0
,
"BOOT_ROM": "000048b0"libusb_bulk_transfer sent 0 bytes; returned 0
Received message GetFileSize: *BOARD_ATTR*40000000
,
"BOARD_ATTR": "40000000"libusb_bulk_transfer sent 0 bytes; returned 0
Received message GetFileSize: *USER_BOARDREV*c03141
,
"USER_BOARDREV": "c03141"libusb_bulk_transfer sent 0 bytes; returned 0
Received message GetFileSize: *JTAG_LOCKED*0
,
"JTAG_LOCKED": "0"libusb_bulk_transfer sent 0 bytes; returned 0
Received message GetFileSize: *ADVANCED_BOOT*0000e8e8
,
"ADVANCED_BOOT": "0000e8e8"libusb_bulk_transfer sent 0 bytes; returned 0
Received message Done: *ADVANCED_BOOT*0000e8e8
CMD exit
}
Second stage boot server doneCongratz, you successfully updated the bootloader EEPROM.
If you wish to collect metrics and logs from compute blade then it is possible
to achieve by enabling opentelemetry-collector and configuring it with your
desired otel dashboard.
Please explore modules/opentelemetry.nix to see how to configure one.
Example of configuration file, I'm using agenix to encrypt it since it contains
auth token, replace <TOKEN> with the token from your otel dashboard.
extensions:
file_storage:
directory: /var/lib/opentelemetry-collector/storage
create_directory: true
recreate: true
receivers:
hostmetrics:
collection_interval: 1m
scrapers:
cpu:
metrics:
system.cpu.utilization:
enabled: true
memory:
metrics:
system.memory.utilization:
enabled: true
load:
network:
exclude:
interfaces:
- lo
match_type: strict
disk:
filesystem:
exclude_fs_types:
fs_types:
- autofs
- devtmpfs
- overlay
- proc
- sysfs
- tmpfs
match_type: strict
exclude_mount_points:
mount_points:
- /boot
- /boot/firmware
match_type: strict
metrics:
system.filesystem.utilization:
enabled: true
paging:
processes:
journald:
journalctl_path: journalctl
start_at: end
priority: info
storage: file_storage
retry_on_failure:
enabled: true
initial_interval: 1s
max_interval: 30s
max_elapsed_time: 0
processors:
memory_limiter:
check_interval: 5s
limit_mib: 128
spike_limit_mib: 32
resourcedetection:
detectors:
- env
- system
timeout: 2s
override: false
batch:
timeout: 5s
send_batch_size: 512
exporters:
otlphttp/dash0:
endpoint: https://ingress.europe-west4.gcp.dash0.com
headers:
Authorization: Bearer <TOKEN>
Dash0-Dataset: default
compression: gzip
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 0
sending_queue:
enabled: true
storage: file_storage
queue_size: 1000
num_consumers: 2
service:
extensions:
- file_storage
pipelines:
metrics:
receivers:
- hostmetrics
processors:
- memory_limiter
- resourcedetection
- batch
exporters:
- otlphttp/dash0
logs:
receivers:
- journald
processors:
- memory_limiter
- resourcedetection
- batch
exporters:
- otlphttp/dash0
telemetry:
logs:
level: infoIf you experiecing issues, it always handy to explore system logs and try to figure out what could be wrong, you can use following commands in such cases:
Diagnostic messages is the go-to tool for viewing low-level hardware detection, driver initialization, and kernel error messages.
sudo dmesg -T -LThe code in this project is licensed under MIT license. Check LICENSE for further details.