Skip to content

Latest commit

 

History

History
232 lines (164 loc) · 11.4 KB

bumping-kernel.md

File metadata and controls

232 lines (164 loc) · 11.4 KB

Bumping the kernel pin

The procedure when moving linux-smoothkernel to a new kernel series (7.2 → 7.3, 6.x → 7.x). Point releases within a series (7.2.10 → 7.2.14) are automatic; see below.

One kernel line across all four flavors, so one bump is a single coordinated edit — no per-flavor version skew.

Point releases are automatic

A point release of the series SmoothKernel builds (7.2.10 → 7.2.14) needs no one. The Follow stable workflow (.github/workflows/follow-stable.yml) checks kernel.org every hour, commits the new KERNEL_VERSION to main and starts the Release workflow. smoothapt ingests the release within the hour, and every Smooth* system installs it through unattended-upgrades. The patch lanes and configs carry over: lanes keep the name of the release they were rebased on.

If a point release breaks a lane, that release's build fails and nothing is published; fix the lane on main and the next build releases it. The steps below are for a new series (7.2 → 7.3), which the workflow only reports.

When to bump

Bump when the latest stable kernel that the must-have DKMS set supports AND our vendored patch lanes are ready for moves forward.

For Smooth* the must-have DKMS set is OpenZFS (required by SmoothNAS). Rule: latest stable kernel ≤ OpenZFS Linux-Maximum AND our vendored patch lanes are ready for that kernel.

Check OpenZFS:

$ curl -fsSL https://github.com/openzfs/zfs/raw/zfs-2.4.4/META | grep ^Linux
Linux-Maximum: 7.2
Linux-Minimum: 4.18

Linux-Maximum is a declaration, not a measurement

Treat the Linux-Maximum line as OpenZFS telling you what they have tested, not as evidence that the code fails above it. Twice now the cap has lagged reality: the bumps to both 7.0 and 7.1 were one-line META commits with no accompanying code change (7.1 was PR #18682). By contrast the 6.19 bump did need real code (duplicate GCM assembly symbols). So the cap moving is sometimes a paperwork event and sometimes a engineering one, and you cannot tell which from the version number alone.

Before concluding a kernel is out of reach, check whether the cap is the only thing in the way:

  1. Look at the commit that raised the cap upstream. One line touched? Likely no code was needed.
  2. Compile it. Build the OpenZFS release against the candidate kernel with --enable-linux-experimental (an upstream-supported flag that bypasses the cap) and read the warnings, not just the exit status. Implicit-declaration and incompatible-pointer warnings mean real API drift; objtool __noreturn noise does not.

This is how the 7.0.11 → 7.1.8 bump was resolved: OpenZFS 2.4.3 compiled clean against 7.1.8 even though its META said 7.0, so the bump proceeded with a one-line carry patch instead of waiting.

Do not ship --enable-linux-experimental as the fix. It works for a local build but cannot reach the DKMS path — scripts/dkms.mkconf hardcodes the configure arguments used when openzfs-zfs-dkms rebuilds on the target, so the flag never reaches an appliance and every on-target rebuild would fail. Carry a patches/zfs-<version>/ META patch instead, so the raised cap travels with the packaged source. See PATCHES.md and patches/zfs-2.4.3/README.md.

OpenZFS 2.4.4 now supplies the Linux 7.2 compatibility code (including the sget_fc() conversion) and declares Linux-Maximum: 7.2 upstream. The current 7.2.10 line therefore clears ZFS_PATCHSET; the archived 2.4.3 META patch stays available for reproducing older builds. Packaging serialization, gzip RPM payloads, and the linux-headers virtual Provides fix remain required.

Check your downstream sources:

$ git log -- patches/cachyos-* patches/nobara-picks patches/post-nobara-* | head

If the downstream patch material for your target kernel is not ready yet, wait or do the rebase work first. SmoothKernel vendors the exact lane it builds.

Always pick a stable point release (x.y.z), never -rc or -next.

Steps

1. Pick the new version

Cross-reference:

  • kernel.org stable
  • OpenZFS Linux-Maximum (for NAS-supporting kernels)
  • CachyOS patch-series availability

Update build.env:

KERNEL_VERSION=7.2.10
LOCALVERSION=-smoothkernel          # never changes under the one-kernel model
CACHYOS_PATCHSET=cachyos-7.2.10
NOBARA_PATCHSET=nobara-picks
POST_NOBARA_PATCHSET=post-nobara-7.2.10
ZFS_VERSION=2.4.4                   # native Linux 7.2 support
ZFS_PATCHSET=                       # no carry needed for 2.4.4

2. Vendor the patch lanes

Refresh the ordered patch lanes and commit them:

mkdir -p patches/cachyos-$KERNEL_VERSION patches/post-nobara-$KERNEL_VERSION
# copy or regenerate the base lane, starting with the pristine-kernel-safe BORE patch
# refresh any Nobara picks in patches/nobara-picks/
# copy or regenerate any post-Nobara carry patches
git add patches/cachyos-$KERNEL_VERSION patches/nobara-picks patches/post-nobara-$KERNEL_VERSION
git commit -m "patches: refresh kernel lanes for $KERNEL_VERSION"

Vendoring keeps builds reproducible and CI-offline-safe. The current pristine-kernel base lane uses 0001-bore.patch, not 0001-bore-cachy.patch.

3. Update the canonical config

make kernel-config-update-all # runs `make olddefconfig` against the new kernel for each arch

Review the diff in configs/smooth-*.config. New CONFIG_* symbols default to the kernel's preference; check that defaults don't regress the invariants in kernel-config.md (PREEMPT=y, HZ=1000, SCHED_BORE=y, etc.). make kernel-config-update-all also refreshes configs/<kernel-version>/smooth-*.config.

4. Build and verify

make kernel DEB_ARCH=amd64
make kernel DEB_ARCH=arm64
make zfs DEB_ARCH=amd64
make zfs DEB_ARCH=arm64
find out -maxdepth 2 -name '*.deb'

Run make check-zfs-dkms DEB_ARCH=<arch> on each native builder after both package builds. It extracts the generated openzfs-zfs-dkms package and invokes its normal configure/build hook against the complete matching kernel tree, checks zfs.ko/spl.ko vermagic, and refuses an experimental-cap bypass. CI runs this before uploading or publishing packages. It does not install or load modules.

Both architectures must build clean. Kernel build failures are usually patch-conflict or config drift; ZFS build failures are usually a kernel-API break that OpenZFS hasn't caught up to yet.

Local builds are sufficient for patch/config bring-up. What users get always comes from the CI release path in CI_RELEASES.md, which attests every package; signing is optional (see signing.md).

5. Update out-of-tree module compat shims

For each consuming repo with an out-of-tree module (smoothfs in SmoothNAS today; others later):

  1. Edit the module's compat.h:
    • Bump KERNEL_FLOOR_MAJOR / KERNEL_FLOOR_MINOR to the new floor.
    • Sweep dead pre-floor branches (anything inside #if LINUX_VERSION_CODE < KERNEL_VERSION(<old_floor>, …) is now unreachable — delete).
    • For new kernel APIs the new floor makes available and you want to adopt: add #if LINUX_VERSION_CODE >= KERNEL_VERSION(<new_floor>, …) blocks; expose them via <prefix>_compat_*() helpers.
  2. Update the module's dkms.conf BUILD_EXCLUSIVE_KERNEL regex if needed.

The detailed DKMS consumer contract is in DKMS.md.

6. Deploy to a test box

scp out/<arch>/*.deb test-box:/tmp/
ssh test-box 'sudo dpkg -i /tmp/linux-*_*.deb /tmp/zfs*.deb /tmp/lib*.deb'
ssh test-box 'sudo reboot'

7. Validate

Per-flavor smoke tests. The minimum publish gate is one representative target per flavor class, not just "the author's test box":

  • SmoothNAS: real or virtual machine with a ZFS pool and SMB export path
  • SmoothRouter: dual-NIC box or VM with WAN/LAN traffic path and WireGuard enabled
  • SmoothHTPC: Intel or AMD GPU box that boots to smoothtv and exercises hardware video decode
  • SmoothDesktop: desktop-class box that boots to Plasma and exercises Rakuen Mesa userspace

Per target, the minimum smoke tests are:

  • SmoothNAS: mount a ZFS pool, export an SMB share, confirm tierd starts + UI loads.
  • SmoothRouter: pass packets WAN↔LAN, verify nftables rules applied, wireguard handshake completes.
  • SmoothHTPC: boot to smoothtv, launch Kodi, verify VA-API hardware decode.
  • SmoothDesktop: boot to Plasma, launch Firefox and Steam Big Picture, verify GPU acceleration.

NVIDIA is a conditional lane rather than a universal gate: run it whenever the kernel bump also changes NVIDIA-relevant DKMS or graphics packaging assumptions.

Flavor-specific smoke tests live in their respective repos' docs/OPERATIONS.md.

8. Promote to common main

Promotion takes the packages of the CI release for the bump, never a local build. smoothapt ingests them from the GitHub Release into the common suite, after verifying each package's build-provenance attestation (see CI_RELEASES.md). Systems follow through the linux-image-smoothkernel and linux-headers-smoothkernel metapackages on their next apt upgrade.

9. Secure-Boot verify (signed releases)

When the release was built with signing material, before promoting it:

  • verify a CI-produced kernel package reports a non-empty signer on one packaged module
  • verify a Secure-Boot-enabled test machine enrolled via smooth-secureboot can rebuild and load ZFS
  • verify an unsigned ad-hoc module is rejected

See signing.md for the trust model.

Tradeoffs to remember

  • Major bumps (6.x → 7.0) historically have more API churn and surface more shim work. Wait for 7.1 minimum before shipping to users.
  • Point bumps within an LTS (6.18.22 → 6.18.30) usually only surface config drift or out-of-tree module fallout.
  • Cross-line bumps (6.18 → 6.19) often involve real API changes. Plan for shim work in any DKMS consumer.
  • Patch-lane lag: if the base lane, Nobara picks, or post-Nobara carry patches are not ready for your target kernel, wait or schedule the rebase work.
  • OpenZFS lag: typically 1–3 months behind mainline stable. Same "wait" rule if NAS is in-scope.

When a patch lane skips a kernel version

Sometimes the downstream material you are following skips a point release, or a carry patch only rebases cleanly on the next one. In that case we can skip that version too. Document the skip in the bump PR's changelog:

bump kernel: 6.19.6 → 6.19.9 (skipping .7 and .8; downstream patch lanes were not ready)

When a major kernel bump requires patch-series rework

Major version bumps (6 → 7) often require real rebase work across the base lane, Nobara picks, and any post-Nobara carry patches. The rule is unchanged: wait. The conservative x.1 minimum for shipping to users gives downstream patch material time to stabilize against the new kernel line too.

Single-flavor rollback

The one-kernel model means we can't selectively roll back the kernel for one flavor. If kernel X breaks SmoothDesktop but is fine on SmoothNAS, both roll back. In practice this has been rare (most kernel regressions affect either all workloads or none), but it's a trade we accept for the maintenance savings.

If a flavor-specific rollback becomes chronic, that's a strong signal to reconsider the one-kernel decision. See ARCHITECTURE.md and kernel-config.md.