A custom embedded Linux distribution built with Yocto Project for Raspberry Pi Zero 2 W, featuring custom machine configuration, kernel optimizations, device tree overlays, and kernel modules.
Big thanks to Techleef that gives me this opportunity.
Disclaimer: This repository serves as both a functional Yocto project and comprehensive documentation of my learning journey. The content includes detailed notes and troubleshooting steps.
Let's say it's Yocto/Embedded Linux Hacking ^_^.
This repository contains my first Yocto project implementation, documenting the complete process of development. The project uses KAS for build automation , Docker for containerized development environment and No Poky.
- Raspberry Pi Zero 2 W
- MicroSD card (16GB)
- FT232RL
- openembedded-core / bitbake / meta-raspberrypi: under meta/layers
- Custom layer: meta-aero-bsp
- KAS yml file: for automatic configuration and building instead of sourcing oe script
- Container image files and directories: checking setup
- HTTP server: host /images via docker, accessible in local network and shared between team
- KAS yml files for all layers and local configuration
- Commands:
- Build:
kas-container build kas-prj.yml - Execute bitbake command:
kas-container shell kas-prj.yml
- Build:
- Successfully generated first image with default raspberrypi2w config
- Manual approach:
- Host directory using
docker run -it - Add
-vfor file sharing between container and local machine
- Host directory using
- KAS-container approach: using kas-container instead of kas + manual docker configuration
- Use
local_conf_headerin kas file to define custom paths for SSTATE and DL_DIR
- Add to local_conf_header:
INHERIT += "buildhistory" BUILDHISTORY_COMMIT = "1"
- Developed script to setup docker HTTP server over packages deploy directory
- Added new machine called
aero-RPI
bunzip2 -f core-image-minimal-aero-rsp.rootfs.wic.bz2
sudo dd if=core-image-minimal-aero-rsp.rootfs.wic of=/dev/mmcblk0 bs=1M status=progress conv=fsync- Serial connection:
picocom /dev/mmcblk0 -b 115200- Issue with picocom input login, switched to
screeninstead
- Issue with picocom input login, switched to
- UART setup: Add
ENABLE_UART = "1"in kas file for FTDI usage - Alternative flashing: Balena Etcher proved easier and safer than dd
bitbake -e | grep '^MACHINE_FEATURES='Result: MACHINE_FEATURES=" apm usbhost keyboard vfat ext2 screen touchscreen alsa bluetooth wifi sdio vc4graphics qemu-usermode"
MACHINE_FEATURES:remove = 'apm usbhost keyboard screen touchscreen alsa bluetooth sdio vc4graphics qemu-usermode'
Result: MACHINE_FEATURES=" vfat ext2 wifi"
- Attempted:
IMAGE_INSTALL:remove = "kernel-image kernel-devicetree kernel-base kernel-modules" - Status: Not working - raspberrypi2w config automatically adds kernel to rootfs
- Note: Need more investigation
- Auxiliary display support: [n]
- Changed default hostname
- Enabled optimize for size
- Disabled sound support
Results:
- Old kernel image: 7.3M kernel7.img
- New kernel image: 4.4M kernel7.img
diff ./linux-raspberrypi/6.6.63+git/sources-unpack/defconfig ./linux-raspberrypi/6.6.63+git/linux-aero_rsp-standard-build/defconfig
grep SOUND .config # Result: # CONFIG_SOUND is not set- Added
led.dtsfile under files directory - Used
do_configure:append()to copymy-led-overlay.dtsinto overlays dts and add to MAKEFILE - Verified overlay location:
./build/tmp/work-shared/aero-rsp/kernel-source/arch/arm/boot/dts/overlays/myled-overlay.dts
KERNEL_DEVICETREE:append = " overlays/myled.dtbo"
RPI_EXTRA_CONFIG = "dtoverlay=myled"
- Successfully controlled LED:
echo 1 > /sys/class/leds/simple_led/brightness
- Created
my_moduledirectory structure - Added
my-module.bbrecipe - Created
filessubdirectory withhello.candMakefile - Added to configuration:
IMAGE_INSTALL:append = " my-module"
ls /tmp/wic_rootfs/lib/modules/6.6.63-v7/updates/
# Result: hello.ko.xz- Now let's test out module, in this case our module is a considered as standard kernel module , so i m using modprobe instead of insmod.
-
- Very minimal kernel defconfig (already did, maybe adding more stuff) I found that I missed this one CONFIG_USB_SUPPORT.
- First reading the boot process messages, and I found that the boot is 8 seconds (that's too much)
[ 0.200167] calling deferred_probe_initcall+0x0/0x9c @ 1
[ 0.205851] probe of 3f101000.cprman returned 0 after 5512 usecs
[ 0.206178] uart-pl011 3f201000.serial: cts_event_workaround enabled
[ 0.206382] probe of 3f201000.serial:0 returned 0 after 30 usecs
[ 0.206485] probe of 3f201000.serial:0.0 returned 0 after 28 usecs
[ 0.206512] 3f201000.serial: ttyAMA1 at MMIO 0x3f201000 (irq = 114, base_baud = 0) is a PL011 rev2
[ 0.206746] serial serial0: tty port ttyAMA1 registered
[ 0.206835] probe of 3f201000.serial returned 0 after 945 usecs
[ 0.207494] probe of 3f215000.aux returned 0 after 588 usecs
[ 0.207569] bcm2835-aux-uart 3f215040.serial: there is not valid maps for state default
[ 0.207968] printk: console [ttyS0] disabled
[ 0.208276] probe of 3f215040.serial:0 returned 0 after 27 usecs
[ 0.208372] probe of 3f215040.serial:0.0 returned 0 after 26 usecs
[ 0.208399] 3f215040.serial: ttyS0 at MMIO 0x3f215040 (irq = 86, base_baud = 50000000) is a 16550
[ 0.208435] printk: console [ttyS0] enabled
[ 7.852532] probe of 3f215040.serial returned 0 after 7644975 usecs
[ 7.859468] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[ 7.865947] probe of bcm2835-wdt returned 0 after 6750 usecs
[ 7.871960] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver
[ 7.879299] probe of bcm2835-power returned 0 after 7469 usecs
[ 7.885261] probe of 3f100000.watchdog returned 0 after 26243 usecs
[ 7.892078] probe of 3f212000.thermal returned 0 after 370 usecs
[ 7.898704] mmc-bcm2835 3f300000.mmcnr: mmc_debug:0 mmc_debug2:0
[ 7.904810] mmc-bcm2835 3f300000.mmcnr: DMA channel allocated
[ 7.931793] probe of 3f300000.mmcnr returned 0 after 33522 usecs
[ 7.938490] sdhost: log_buf @ 37041243 (d7d43000)
[ 7.990895] mmc0: sdhost-bcm2835 loaded - DMA enabled (>1)
[ 7.996685] probe of 3f202000.mmc returned 0 after 58719 usecs
[ 8.002764] initcall deferred_probe_initcall+0x0/0x9c returned 0 after 7802583 usecs
-
bcm2835-aux-uart 3f215040.serial: there is not valid maps for state default, I noticed that there is a problem with initialization of serial driver, so I have reset cmdline.txt to default value using "bitbake -c clean rpi-cmdline" and changed from console=serial0 to ttyS0
-
Now the boot time is 1.7 seconds
-
For now I still don't know how I can change configs to cmdline.txt using my machine config, so I'm editing it manually. I hope I can find a solution ASAP.
-
Generated boot time graph with help of bootline documentation you find step and svg in Doc dir.
-
Adding these to my machine config:
# Disable all unnecessary features
disable_splash=1
boot_delay=0
disable_overscan=1
# Minimal GPU memory
gpu_mem=16
# Disable audio completely
dtparam=audio=off
# Disable camera
start_x=0
# Disable HDMI
hdmi_blanking=2
- sadly, I couldn't see any difference in boot time for now.
- I couldn't a proper way to remove unecessary services, while im using sysvinit (default init manager), so i should remove manually that services from rootfs , to do that i need to create a recipe, not a package recipe like before but an image recipe, before creating it we need to know the process of image recipes.
- do_fetch → do_configure → do_compile → do_install → do_package
- do_rootfs → ROOTFS_POSTPROCESS_COMMAND → do_image
- That's why i created a new recipe called core-image-minimal.bbappend so i can append rootfs process by remove unecessary serices:
remove_unwanted_services(){
rm -f ${IMAGE_ROOTFS}/etc/init.d/networking
echo "-->removing ${IMAGE_ROOTFS}/rootfs/etc/init.d/networking"
rm -f ${IMAGE_ROOTFS}/etc/init.d/banner.sh
rm -f ${IMAGE_ROOTFS}/etc/rc*.d/*networking
rm -f ${IMAGE_ROOTFS}/etc/rc*.d/*banner
}
ROOTFS_POSTPROCESS_COMMAND:append = " \
remove_unwanted_services; \
"
- clearing core-image-minimal do_rootfs then building again fix it as expected :
bitbake core-image-minimal -f -c do_rootfs
bitbake core-image-minimal
ls ./tmp/work/aero_rsp-oe-linux-gnueabi/core-image-minimal/1.0/rootfs/etc/init.d
alignment.sh checkroot.sh functions modutils.sh populate-volatile.sh read-only-rootfs-hook.sh save-rtc.sh stop-bootlogd udev urandom
bootlogd devpts.sh halt mountall.sh rc reboot sendsigs sysfs.sh umountfs
bootmisc.sh dmesg.sh hostname.sh mountnfs.sh rcS rmnologin.sh single syslog umountnfs.sh
- YAY they are really removed.(it may could be another elegant method but this what i got for now).
-Since I’m using a Raspberry Pi, the default bootloader is not U-Boot; it’s the GPU firmware bootloader, which uses these files: start*.elf, bootcode.bin, and kernel*.img. It’s easy to use, easy to configure, and requires no compilation, but it is very specific to the Raspberry Pi. That’s why I’m thinking of switching to U-Boot. Let’s do some U-Boot hacking ^_^.
-PS: It's 2025 not 2024 hehe, we will fix it later....
-Im planning to use falcon mode , but i don't found a good documentation for how to use it, we will get back to it later.
- meta-aero-distro
- INIT_MANAGER bitbake variable can be set to systemd or sysvinit. I have tested both, but sysvinit is faster by almost 2 seconds.
- Using busybox instead of core-utils + bash, because busybox is a single binary about 1-2MB while core-utils + bash almost 14 MB with max of GPL3 and GPL2 , so busybox is the king here.
- Let's check that busybox is Added or not : Yes it's valid as the image shows.
DISTRO_FEATURES = "ext2"
# This feature is added by DISTRO_FEATURES_BACKFILL
DISTRO_FEATURES:remove = "pulseaudio"
Of course!
# CVE report generated under: build/tmp/deploy/cve
INHERIT += "cve-check"
INHERIT is used to include the cve-check bbclass. If we examine the cve-check.bbclass, we can see that it contains "addtask cve_check before do_build".
INCOMPATIBLE_LICENSE:pn-core-image-minimal = "AGPL-3.0-only AGPL-3.0-or-later GPL-3.0-only GPL-3.0-or-later LGPL-3.0-only"
This needs to be made specific to our target core-image-minimal; otherwise, the build will fail.
- Adding
INHERIT += "image-buildinfo"- still not sure how to implement this properly.
# Enable build information integration for all images:
find ./build/tmp/work/aero_rsp-oe-linux-gnueabi/core-image-minimal/ -name buildinfo
./build/tmp/work/aero_rsp-oe-linux-gnueabi/core-image-minimal/1.0/rootfs/etc/buildinfo
export BB_ENV_PASSTHROUGH_ADDITIONS="$BB_ENV_PASSTHROUGH_ADDITIONS IMAGE_TYPE"
IMAGE_TYPE="fab" bitbake core-image-minimal: generates a fab imageIMAGE_TYPE="dev" bitbake core-image-minimal: generates a dev image
IMAGE_FEATURES="dbg-pkgs dev-pkgs empty-root-password"# IMAGE_FEATURES not configured for production buildsUsing IMAGE_INSTALL to add necessary packages including custom kernel modules:
IMAGE_INSTALL += "my-module"IMAGE_FEATURES += "dbg-pkg"IMAGE_INSTALL += " \
gdb \
strace \
ldd \
"IMAGE_INSTALL += " \
gcc \
g++ \
make \
"IMAGE_FEATURES += "empty-root-password"Created a custom packagegroup recipe called basic-packagegroup inspired by packagegroup-core-boot:
- Contains only necessary configurations
- Includes French keyboard layout:
KEYMAP ?= "fr" - Used by both development and production images via
IMAGE_INSTALL
Using WKS (WIC Kickstart) files for partition management:
- Base file:
sdimage-raspberrypi.wks(contains/bootand/rootpartitions) - Custom file:
dev-image.wks(adds/datapartition for development image)
- Size: 80MB
- Source: Empty
- Usage: Development image only
After booting, partition layout can be verified:
sudo fdisk -l /dev/mmcblk0Expected Output:
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 * 8192 274431 266240 130M c W95 FAT32 (LBA)
/dev/mmcblk0p2 278528 5922815 5644288 2.7G 83 Linux
/dev/mmcblk0p3 5922816 6086655 163840 80M 83 Linux
The /data partition (/dev/mmcblk0p3) is not automatically recognized by the system:
blkid
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL_FATBOOT="boot" LABEL="boot" UUID="0F4C-F39B" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="076c4a2a-01"
/dev/mmcblk0p2: LABEL="root" UUID="b43c603a-2050-46e5-bbc6-b7597e2916d8" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="076c4a2a-02"
/dev/mmcblk0p3: PARTUUID="076c4a2a-03"-
Resolution Required: Need to define the source directory for the data partition in the WKS file.
-
Solved : After removing --source empty thre problem is solved.
- Dev image :
- IMAGE_FEATURES += "empty-root-password" => remove root password
- Fab image :
# set image root password
ROOT_PASSWORD = "root"
DEV_PASSWORD = "mrrobot"
#-m : add home dir to elliot
EXTRA_USERS_PARAMS = "groupadd developers; \
useradd -m -G developers -p '$(openssl passwd ${DEV_PASSWORD})' elliot; \
usermod -p '$(openssl passwd ${ROOT_PASSWORD})' root;"
- Already implemented in fab-image and dev-image custom recipes.
- Creating a new recipe called sudo-config.bb :
- Customizing do_install :
install -d ${D}/etc/sudoers.d
echo "elliot ALL=(ALL) ALL" > ${D}/etc/sudoers.d/devs
- Need to be completed. (skiped for now ....)
- This is kinda a hard part for me, but it's challenging at the same time. This a simple diagram represent how rauc update works in our project.
- From https://github.com/rauc/meta-rauc-community/tree/master/meta-rauc-raspberrypi
- we need to add :
IMAGE_INSTALL:append = " rauc"IMAGE_FSTYPES:append = " ext4"DISTRO_FEATURES:append = " rauc"- It is recommended to use systemd as init manager but i prefer sysvinit at this moment.
- Adding system.conf :
- we need to add :
[system]
compatible=aero-rsp
bootloader=uboot
data-directory=/data/
[keyring]
path=/etc/rauc/ca.cert.pem
[slot.rootfs.0]
device=/dev/mmcblk0p2
type=ext4
bootname=A
[slot.rootfs.1]
device=/dev/mmcblk0p3
type=ext4
bootname=B
DESCRIPTION = "RAUC bundle generator"
LICENSE = "CLOSED"
inherit bundle
RAUC_BUNDLE_COMPATIBLE = "${MACHINE}"
RAUC_BUNDLE_SLOTS = "rootfs"
RAUC_SLOT_rootfs = "dev-image"
-
After installing the bundle and reboot, we got an issue , failed to boot from B partition.
-
The boot is crashing in uboot with "Failed to load 'boot/Image' Bad Linux ARM64 Image magic! Boot failed (err=1) Card did not respond to voltage select!"
-
After installing rauc bundle and before restarting i tried to mount /dev/mmcblk0p3 /tmp/rootfs and i could find Image under /tmp/rootfs.
-
It turns out that I need to add custom fstab recipe that mount /boot every boot:
/dev/mmcblk0p1 /boot vfat defaults 0 0 -
Explanation : after installing the bundle the uboot.env is modified with new changes like boot order from A -> B or B -> A.
┌───(aeropop@unkown)-[/media/aeropop/boot]
└─$ strings uboot.env | grep ORDER
BOOT_ORDER=B A- Finally it works !
- Adding RAUC_BUNDLE_FORMAT = "crypt" in bundle recipe.
- Generating bundle key and cert using :
openssl req -new -x509 \\n -key bundle-encryption.key \\n -out bundle-encryption.cert \\n -days 3650 \\n -subj "/CN=RAUC Bundle Encryption/O=YourCompany/C=US"Adding these in layer.conf:
- RAUC_ENCRYPTION_KEY ?= "${LAYERDIR}/recipes-core/rauc/files/bundle-encryption.key"
- RAUC_ENCRYPTION_CERT ?= "${LAYERDIR}/recipes-core/rauc/files/bundle-encryption.cert"
-
Using these implementations , bundle install fails:
-
- Using
RAUC_SLOT_rootfs[adaptive] = "block-hash-index"i was able to Adaptive update.
- Already done under /docker dir, a script is responsible for that.
- Adding a recipe called
update-bundlethat use the object andde.pengutronix.rauc.Installerinterface .
path = input("Bundle path: ")
# Call the InstallBundle method
print("Installing bundle ...")
ret = interface.InstallBundle(path, {})- It communicate successfully with rauc daemon but somehow the bundle is installed(still investigating ...)
- To be tested !
- First we need to create a initramfs image, to do that we can use embedded initramfs in our kernel.
INITRAMFS_IMAGE = "base-initramfs"
INITRAMFS_IMAGE_BUNDLE = "1"
- Creating a custom initramfs image recipe
base-initramfs.bb - Creating our init script
boot.shand a recipe that copy that script as init in our initramfs image. - The first problem that i faced was that every boot, the initramfs was skipped kernel is mounting rootfs then sbin/init was executed.
- After investigation i figured out that there is some kernel arguments are passed using
cmdline, that make my initramfs from loading. - In our case it's
root=/dev/mmcblk0p2which leads to directly mount that rootfs , also this can also happen using init=/sbin/init (cyber sec people may know this withinit=/bin/shas an argument in grub to get root hehe). - Using
CMDLINE_ROOTFS = ""solved the problem. - Second issue , that RAUC integration is loading kernel from rootfs, where kernel-initramfs image is not there,
- To fix this, i copied kernel initramfs bin file to boot partition and modify RAUC uboot script to load kernel from boot partition instead.
Replacing Image (kernel without initramfs) with Image-initramfs-aero-rsp.bin:
IMAGE_BOOT_FILES:append = " Image-initramfs-aero-rsp.bin;Image "
replace :
load ${BOOT_DEV} ${kernel_addr_r} boot/@@KERNEL_IMAGETYPE@@
with :
load mmc 0:1 ${kernel_addr_r} Image
- This is a very interesting part, that's i've designed a diagram to show the difference between booting without initramfs , with initramfs and with initramfs and encrypted storage (dm-crypt).
- Using cryptsetup tool i was able to encrypt and decrypt my storage (/dev/mmcblk0p2) , by formatting it with luks format, then copy partition content on it.
- To make initramfs script encrypting and decrypting operations easy I have placed The key in boot partition for now called
pass.txt. - After couple of tries and a bunch of kernel panics :
-
Finally my init script is running and it's end of kernel panics.
Error: The postinstall intercept hook 'update_gio_module_cache' failed
Solution: git restore meta-layers repos (still don't know why there is modifications in those repo)
sudo mount -o loop,offset=$((278528*512)) core-image-minimal-aero-rsp.rootfs.wic /tmp/root
sudo mount -o loop,offset=$((8192*512)) core-image-minimal-aero-rsp.rootfs.wic /tmp/boot- Clone this repository
- Ensure Docker is installed and running
- Run the KAS build:
kas-container build kas-prj.yml
- Flash the generated image to SD card using Balena Etcher or dd command
- Connect FTDI adapter and boot the system
This is a learning project documenting my Yocto development journey. Feel free to suggest improvements or share your own experiences!
This project is for educational purposes.
