Storage primitives#17
Closed
philbudden wants to merge 20 commits into
Closed
Conversation
Introduces new 'blueberry' image that builds FROM blueberry-minimal with
storage primitives and observability tooling for NAS workloads and
general-purpose server use.
Package additions:
- Storage primitives: smartctl, hdparm, lvm2, mdadm, cockpit-storaged,
filesystem tools (xfsprogs, dosfstools, exfatprogs, e2fsprogs)
- Observability: pcp-zeroconf (Performance Co-Pilot)
Architecture:
- Builds FROM blueberry-minimal:${FEDORA_VERSION}
- Follows same directory structure and patterns as blueberry-minimal
- Clean layering: Fedora IoT → blueberry-minimal → blueberry
CI updates:
- Split build workflow into sequential jobs (build-minimal, build-blueberry)
- Enforces build order to ensure blueberry-minimal exists before building blueberry
- Triggers on changes to blueberry/ directory
- Consistent ARCH parameter across both jobs
System configuration:
- PCP tmpfiles.d configuration for /var/lib/pcp and /var/log/pcp directories
- Enables pmcd and pmlogger services for performance monitoring
Documentation:
- Updated README with image variants, layering architecture
- Included rebase instructions and package justifications
- Added RAID usage guidance (RAID1 recommended, RAID0 not for USB storage)
Design philosophy:
- Primitives not policy - no SMB/NFS/backup daemons on host
- Container-first - stateful services run in containers
- SBC-appropriate - minimal overhead, constrained hardware aware
- Follows AGENTS.md principles
No modifications to blueberry-minimal (verified).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous approach of separate sequential jobs failed because: - build-blueberry ran on a different runner than build-minimal - Tried to pull blueberry-minimal from GHCR before it existed - Failed on first build and PR builds (where images aren't pushed) Solution: - Use matrix strategy to build both images - Both run on same runner, so blueberry can use locally built blueberry-minimal - blueberry-minimal is built first (alphabetically in matrix) - blueberry build attempts to pull from registry (for cache) but falls back to local This matches the original single-job pattern used before the split, but with explicit matrix values for clarity. Fixes: Error 'manifest unknown' when pulling blueberry-minimal:43 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
After building blueberry-minimal, tag it with the full registry path (ghcr.io/philbudden/blueberry-minimal:44) so that the subsequent blueberry build can find it locally. Without this, when blueberry's Containerfile tries: FROM ghcr.io/philbudden/blueberry-minimal:44 Buildah tries to pull from GHCR instead of using the locally built image, causing 'manifest unknown' errors. The tag step only runs for blueberry-minimal builds, creating a local alias that matches what the blueberry Containerfile expects. Fixes: Error 'reading manifest 44 in ghcr.io/philbudden/blueberry-minimal' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Matrix strategy builds jobs in parallel by default, causing blueberry to start building before blueberry-minimal completes and gets tagged. Setting max-parallel: 1 forces sequential execution: 1. blueberry-minimal builds completely 2. Gets tagged with registry prefix 3. Then blueberry starts and can find the tagged local image This ensures the dependency order is respected even within a matrix. Alternative considered: separate jobs with needs dependency, but that requires different runners and registry pushes between jobs. Fixes: blueberry build fails with 'manifest unknown' because it starts before blueberry-minimal is tagged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
blueberry-minimal's Containerfile uses IMAGE_REGISTRY to pull the
Fedora IoT base image:
FROM ${IMAGE_REGISTRY}/fedora/fedora-iot:${FEDORA_VERSION}-${ARCH}
When we pass IMAGE_REGISTRY=ghcr.io/philbudden to blueberry-minimal,
it tries to pull from ghcr.io/philbudden/fedora/fedora-iot which
doesn't exist, causing a 403 Forbidden error.
Solution:
- blueberry-minimal: Use default IMAGE_REGISTRY=quay.io (don't pass it)
- blueberry: Pass IMAGE_REGISTRY=ghcr.io/philbudden (needs it for base)
This is done with a conditional in the build-args:
${{ matrix.image_name == 'blueberry' && format('...') || '' }}
Fixes: 403 Forbidden when pulling ghcr.io/philbudden/fedora/fedora-iot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The tagging step failed with 'image not known' when trying to tag blueberry-minimal:43 after the build. Buildah-build creates multiple tags (sha, date, version, latest) but they may not be immediately available by version number. The 'latest' tag is guaranteed to exist after the build completes. Changed from: podman tag blueberry-minimal:43 ghcr.io/.../blueberry-minimal:43 To: podman tag blueberry-minimal:latest ghcr.io/.../blueberry-minimal:43 This creates the registry-prefixed tag that blueberry's Containerfile expects, using a source tag we know exists. Fixes: Error 'blueberry-minimal:43: image not known' during tagging Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
buildah-build action creates images with 'localhost/' prefix by default. The tagging step was failing because it tried to reference: blueberry-minimal:latest But the actual image is stored as: localhost/blueberry-minimal:latest Changed to use the correct image reference with localhost/ prefix. This is a standard behavior of buildah/podman when building images locally - they are stored in the localhost registry namespace. Fixes: Error 'blueberry-minimal:latest: image not known' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of guessing the image name (localhost/, no prefix, etc), use the actual output from the buildah-build step. Added debug step to list all images and build outputs to diagnose what's actually being created. The buildah-build action provides: - steps.build.outputs.image (the base image name) - steps.build.outputs.tags (all applied tags) Using the output directly ensures we reference the image correctly, regardless of how buildah stores it internally. Debug output will show exactly what images exist and help identify any remaining naming issues. Fixes: Continued 'image not known' errors due to incorrect image ref Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CRITICAL FIX: The workflow was fundamentally broken for PR builds. The Problem: - blueberry Containerfile: FROM ghcr.io/philbudden/blueberry-minimal:44 - On PR builds, images are NOT pushed to GHCR - blueberry tried to pull from GHCR, but image doesn't exist there - First build on main also fails - nothing in registry yet The Solution: - Pass IMAGE_REGISTRY=localhost for blueberry builds - Tag blueberry-minimal as localhost/blueberry-minimal:44 - blueberry's FROM becomes: FROM localhost/blueberry-minimal:44 - Uses the locally built image, no network/registry needed Changes: 1. IMAGE_REGISTRY=localhost (not ghcr.io/philbudden) 2. Tag as localhost/blueberry-minimal:44 (matches what blueberry expects) 3. Use :latest as source tag (guaranteed to exist after build) This works for: - ✅ PR builds (never touch GHCR) - ✅ First build on main (no prior registry images) - ✅ All subsequent builds (uses local, fast) The GHCR registry is only for distribution, not for build dependencies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The issue: FROM localhost/foo in a Containerfile tries to PULL from a registry at 'localhost', not use local storage. How buildah/podman works: - FROM ghcr.io/foo/bar:tag -> checks local storage FIRST, then pulls - FROM localhost/bar:tag -> treats 'localhost' as registry URL, tries HTTP The fix: - Don't pass IMAGE_REGISTRY=localhost (that made it try to pull) - Tag image as ghcr.io/philbudden/blueberry-minimal:44 locally - blueberry Containerfile uses default: FROM ghcr.io/philbudden/blueberry-minimal:44 - Buildah finds it in local storage (tagged with that name) - No network/registry access needed This is the correct pattern: 1. Build image 2. Tag it with the SAME name the next Containerfile will reference 3. Buildah finds it locally, doesn't try to pull Works for PR builds, first builds, all scenarios - no GHCR dependency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The build step outputs only the image name without a tag, so we need to use the normalized image_name variable instead of build.outputs.image when tagging the local image for use by the blueberry build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
philbudden
force-pushed
the
storage-primitives
branch
from
February 15, 2026 10:47
93a45da to
120ae78
Compare
Implement comprehensive three-tier testing strategy to validate GitHub Actions workflows and image builds locally before pushing to remote CI. Architecture: - Tier 1: Fast syntax validation (~5 seconds) - Tier 2: Workflow simulation with act (~5-10 minutes) - Tier 3: Build testing with QEMU ARM64 emulation (variable) DevContainer enhancements: - Mount Docker socket from host (not DinD for simplicity) - Auto-install act, actionlint, yamllint, buildah - Configure QEMU for multi-arch support Testing scripts: - scripts/validate-workflows.sh - YAML/workflow linting - scripts/test-workflow.sh - Run GitHub Actions locally - scripts/test-build.sh - Test image builds with dependencies Justfile integration: - just validate-workflows - Fast syntax check - just test-workflow <name> - Dry-run workflow - just test-workflow-run <name> - Full execution - just test-build <variant> - Test single variant - just test-build-all - Test full dependency chain - just pre-push - Run all pre-push validations Configuration: - .actrc - act runner settings - .yamllint - YAML linting rules - .github/workflows/validate.yml - CI validation Documentation: - docs/TESTING.md - Comprehensive guide (7.5KB) - docs/TESTING-QUICKREF.md - Quick reference card - README.md - Added Development & Testing section Design decisions: - Docker socket binding over DinD (lower complexity) - QEMU for ARM64 local testing, GitHub Actions for production - Industry-standard tools (act, actionlint) - Pragmatic trade-offs aligned with AGENTS.md philosophy Usage: just pre-push # Before every commit just test-workflow build-43.yml # Test workflow changes just test-build blueberry-minimal # Test build changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove trailing spaces from reusable-build.yml - Fix line length violations by refactoring long lines - Add proper quoting around shell variable expansions - Fix shellcheck warnings (SC2086) by quoting variables - Refactor long URL in validate.yml using variables All workflow validation tests now pass: - yamllint: PASSED - actionlint: PASSED - shellcheck: PASSED Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The test-workflow.sh script was passing "$@" to act after already consuming and processing the first two arguments (workflow file and event). This caused act to receive duplicate arguments, resulting in the error: "accepts at most 1 arg(s), received 3". Root cause: - Line 75 passed all original args to act via "$@" - But args 1 and 2 were already extracted and used in ACT_FLAGS Fix: - Remove "$@" from act invocation - Script already processes required arguments explicitly Tested: - just test-workflow build-43.yml (dry-run) - PASSED - just test-workflow validate.yml pull_request - PASSED Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GitHub Actions CI was failing with error:
Error: blueberry-minimal:latest: image not known
Error: Process completed with exit code 125.
Root cause:
Incorrect shell quoting split the image reference into invalid parts:
"${IMG}":latest
This creates: "blueberry-minimal" :latest (broken reference)
The shell interprets this as:
- A quoted string: "blueberry-minimal"
- An unquoted literal: :latest
- Buildah receives an invalid image name
Fix:
Move the colon inside the quotes to create a valid image reference:
"${IMG}:latest"
This creates: "blueberry-minimal:latest" (valid reference)
Before: sudo buildah tag "${IMG}":latest "${REGISTRY}/${IMG}:${FEDORA_VER}"
After: sudo buildah tag "${IMG}:latest" "${REGISTRY}/${IMG}:${FEDORA_VER}"
This follows shell quoting best practices where the entire value
(including punctuation) should be quoted as a unit.
Verified:
- yamllint: PASSED
- actionlint: PASSED
- shellcheck: PASSED (via actionlint)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The tag step was failing because it referenced the image as
${IMG}:latest, but buildah stores the image under the name from
steps.build.outputs.image. Now uses the actual built image reference.
Fixes: Error: blueberry-minimal:latest: image not known
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The buildah-build action outputs image-id which reliably references the built image regardless of tag format. Using image:tag was failing because the tags output is multiline and the constructed reference didn't match buildah's internal storage. Also fixes trailing whitespace linting error. Fixes: Error: blueberry-minimal:latest: image not known Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The buildah-build action does not provide an image-id output. Available outputs are: image, tags, and image-with-tag. Using image-with-tag which provides the image name with first tag applied, giving us a valid reference to the built image. Fixes: Error: repository name must have at least one component Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The buildah-build action stores images in user storage, not root storage. Using 'sudo buildah tag' tries to access root storage where the image doesn't exist, causing 'image not known' errors. Removed sudo from buildah tag command to access user storage where the buildah-build action stored the image. Also added debug output to show both user and root storage contents. Fixes: Error: blueberry-minimal:pr-17-ea7f628: image not known Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.