Upstream unit tests in the image build; patches 0044–0054, cbt history windows, tier read errors

docs/RPC-CONTRACT.md

@@ -27,7 +27,7 @@ No params, read-only, idempotent. Returns `boot_id` (per-process uuid), `tier_sb
 | `bdev_tier_retire_band` | not an md-mirror band, else `-EBUSY` | idempotent: a re-run re-persists and re-closes | async; acks only once the superblock is durable. `rc ≠ 0` means retry |
 | `bdev_tier_resync_md` | target is a DEGRADED md leg; a healthy source leg exists | re-runnable; the leg stays DEGRADED on failure | copies under an md-range quiesce; acks after activate + persist |
 | `bdev_tier_delete` | — | `-ENODEV` if absent | unregister, then destruct |
-| `bdev_tier_get_bands`, `bdev_tier_read_sb` | — | read-only | `read_sb` returns the highest-seq valid slot plus `generation_uuid` |
+| `bdev_tier_get_bands`, `bdev_tier_read_sb` | — | read-only | `read_sb` returns the highest-seq valid slot plus `generation_uuid`; `valid: false` for a disk with no valid superblock, and an error (its errno) for a read that failed, which says nothing about the disk |
 
 **Assembly rules.** `bdev_tier_read_sb` exposes `version`, `seq`, `generation_uuid`, `created_epoch_sec`. Read every candidate disk's superblock, group by `generation_uuid` (this fences stale disks from a previous instance), take the highest `seq` per band, and when the two md legs disagree on `seq`, assemble the higher one ACTIVE and the other DEGRADED, then `bdev_tier_resync_md`. The fork persists DEGRADED but cannot arbitrate a split-brain across disks; that is the control-plane's.
 

images/spdk/Dockerfile

@@ -4,9 +4,11 @@
 # ============================================================================
 #
 # Multi-stage build:
-#   Stage 1 (builder)   — Compiles SPDK from source on Fedora 43
-#   Stage 2 (collector) — Collects exact runtime closure (binary + shared libs)
-#   Stage 3 (final)     — Scratch-based, no shell, no package manager
+#   Stage 1 (builder)    — Compiles SPDK from source on Fedora 43
+#   Stage 1b (unit-tests) — Runs the upstream unit tests of what the series
+#                           patches, on the tree built in stage 1
+#   Stage 2 (collector)  — Collects exact runtime closure (binary + shared libs)
+#   Stage 3 (final)      — Scratch-based, no shell, no package manager
 #
 # Build:
 #   docker build -t ghcr.io/evariops/spdk:v1.0.0 .
@@ -59,18 +61,10 @@ RUN git clone --branch "${SPDK_VERSION}" --depth 1 --recurse-submodules \
 
 WORKDIR /build/spdk
 
-# ── Inject CBT + tier modules and patches ──
-COPY module/bdev/cbt/ module/bdev/cbt/
-COPY module/bdev/tier/ module/bdev/tier/
-COPY patches/ /build/patches/
-
-# Register CBT + tier in the bdev module build, link list, and deps & apply patches
-# hadolint ignore=SC2016
-RUN sed -i '/^DIRS-y += delay/s/$/ cbt tier/' module/bdev/Makefile && \
-    sed -i '/^BLOCKDEV_MODULES_LIST += bdev_zone_block/a BLOCKDEV_MODULES_LIST += bdev_cbt\nBLOCKDEV_MODULES_LIST += bdev_tier' mk/spdk.modules.mk && \
-    sed -i '/^DEPDIRS-bdev_passthru/a DEPDIRS-bdev_cbt := $(BDEV_DEPS_THREAD)\nDEPDIRS-bdev_tier := $(BDEV_DEPS_THREAD)' mk/spdk.lib_deps.mk && \
-    sed -i '/^DEPDIRS-bdev_lvol/s/$/ bdev_tier/' mk/spdk.lib_deps.mk && \
-    for p in /build/patches/*.patch; do echo "Applying ${p}" && git apply "${p}" || exit 1; done
+# ── Build dependencies — BEFORE the modules and patches ──
+# They depend on the upstream tree alone (pkgdep.sh is upstream's), so a change
+# to a patch or a module no longer invalidates these layers: only the apply,
+# configure and make below run again.
 
 # Fedora's CMake rejects cmake_minimum_required(<3.5) in SPDK subprojects (ISA-L, etc.)
 ENV CMAKE_POLICY_VERSION_MINIMUM=3.5
@@ -88,6 +82,19 @@ RUN ./scripts/pkgdep.sh -d
 RUN dnf install -y --setopt=install_weak_deps=False --nodocs liburing-devel libaio-devel && \
     dnf clean all && rm -rf /var/cache/dnf
 
+# ── Inject CBT + tier modules and patches ──
+COPY module/bdev/cbt/ module/bdev/cbt/
+COPY module/bdev/tier/ module/bdev/tier/
+COPY patches/ /build/patches/
+
+# Register CBT + tier in the bdev module build, link list, and deps & apply patches
+# hadolint ignore=SC2016
+RUN sed -i '/^DIRS-y += delay/s/$/ cbt tier/' module/bdev/Makefile && \
+    sed -i '/^BLOCKDEV_MODULES_LIST += bdev_zone_block/a BLOCKDEV_MODULES_LIST += bdev_cbt\nBLOCKDEV_MODULES_LIST += bdev_tier' mk/spdk.modules.mk && \
+    sed -i '/^DEPDIRS-bdev_passthru/a DEPDIRS-bdev_cbt := $(BDEV_DEPS_THREAD)\nDEPDIRS-bdev_tier := $(BDEV_DEPS_THREAD)' mk/spdk.lib_deps.mk && \
+    sed -i '/^DEPDIRS-bdev_lvol/s/$/ bdev_tier/' mk/spdk.lib_deps.mk && \
+    for p in /build/patches/*.patch; do echo "Applying ${p}" && git apply "${p}" || exit 1; done
+
 # Configure: static SPDK libs + minimal module set.
 #
 # --without-shared      All SPDK code linked statically into spdk_tgt.
@@ -138,10 +145,35 @@ RUN if [ "${BUILD_TYPE}" != "debug" ]; then \
     fi
 
 
+# ============================================================================
+# Stage 1b: Upstream unit tests of what the series patches
+# ============================================================================
+# On the tree just built, not a second build: the suites are compiled against
+# the libraries above and run here. Both images derive from this stage (the
+# collector below, and the debug image's COPY). The suites run in the DEBUG
+# build (asserts on, the configuration upstream runs them in), on each arch; a
+# failing suite fails that build, and the release publish needs every build of
+# the matrix. The release build passes through at no cost.
+# images/spdk/unit-tests.map says which suites a patched file owes; see
+# patches/README.md ("Unit tests").
+FROM builder AS unit-tests
+
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+ARG BUILD_TYPE=release
+
+COPY images/spdk/unit-tests.sh images/spdk/unit-tests.map /build/unit-tests/
+RUN if [ "${BUILD_TYPE}" = "debug" ]; then \
+      /build/unit-tests/unit-tests.sh /build/patches /build/unit-tests/unit-tests.map; \
+    else \
+      echo "Unit tests run in the debug build of this pipeline (asserts on)."; \
+    fi
+
+
 # ============================================================================
 # Stage 2: Collect the exact runtime closure
 # ============================================================================
-FROM builder AS collector
+FROM unit-tests AS collector
 
 SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 
@@ -221,16 +253,17 @@ RUN dnf install -y --setopt=install_weak_deps=False --nodocs \
         jq \
     && dnf clean all && rm -rf /var/cache/dnf
 
-# Binary only (runtime .so provided by Fedora packages above)
-COPY --from=builder /build/spdk/build/bin/spdk_tgt /usr/local/bin/spdk_tgt
+# Binary only (runtime .so provided by Fedora packages above). Copied from the
+# unit-tests stage, so the debug image too is built only once its suites pass.
+COPY --from=unit-tests /build/spdk/build/bin/spdk_tgt /usr/local/bin/spdk_tgt
 
 # spdk_dd is the raid5f bench's full-stripe writer: raid5f only accepts full-stripe
 # writes, and remote initiators are MDTS-bounded, so the writer must run INSIDE the
 # target pod. Patch 0024 makes its exit code honest.
-COPY --from=builder /build/spdk/build/bin/spdk_dd /usr/local/bin/spdk_dd
+COPY --from=unit-tests /build/spdk/build/bin/spdk_dd /usr/local/bin/spdk_dd
 
 # SPDK RPC scripts (useful for live debugging: rpc.py, spdkcli, etc.)
-COPY --from=builder /build/spdk/scripts/ /usr/local/share/spdk/scripts/
+COPY --from=unit-tests /build/spdk/scripts/ /usr/local/share/spdk/scripts/
 
 LABEL org.opencontainers.image.title="SPDK Data Engine (debug)" \
       org.opencontainers.image.description="Fedora-based SPDK debug image with gdb, strace, perf, and RPC scripts" \

images/spdk/unit-tests.map

@@ -0,0 +1,30 @@
+# Which upstream unit-test suites a patched source file owes (see
+# unit-tests.sh, and patches/README.md, "Unit tests").
+#
+# One line: <path prefix> <suite> [<suite> ...]
+#   - the prefix is matched against the files the series patches (the
+#     `+++ b/` lines of patches/*.patch); the LONGEST matching prefix wins;
+#   - a suite is a leaf directory under test/unit/ holding one *_ut.c, whose
+#     binary is that file without its .c;
+#   - "-" says upstream has no suite for it: said, not guessed.
+#
+# Only .c files under lib/, module/ and app/ are looked up. Headers, symbol
+# maps and Makefiles are covered by the suites of the .c files beside them. A
+# patched .c file that matches no line fails the stage: a component is mapped
+# when its first patch lands. A file the series patches under test/unit/ adds
+# its own suite, so a patch's own tests always run.
+
+lib/bdev/                   lib/bdev/bdev.c lib/bdev/mt/bdev.c lib/bdev/part.c
+lib/blob/                   lib/blob/blob.c lib/blob/blob_bdev.c
+lib/jsonrpc/                lib/jsonrpc/jsonrpc_server.c
+lib/nvme/nvme_tcp.c         lib/nvme/nvme_tcp.c
+lib/nvmf/subsystem.c        lib/nvmf/subsystem.c
+lib/nvmf/nvmf_rpc.c         -
+lib/nvmf/nvmf_pause_rpc.c   -
+lib/rpc/                    lib/rpc/rpc.c
+lib/util/bit_array.c        lib/util/bit_array.c
+module/bdev/lvol/           lib/bdev/vbdev_lvol.c
+module/bdev/nvme/           lib/bdev/nvme/bdev_nvme.c
+module/bdev/raid/           lib/bdev/raid/bdev_raid.c lib/bdev/raid/bdev_raid_sb.c lib/bdev/raid/raid0.c lib/bdev/raid/raid1.c lib/bdev/raid/raid5f.c lib/bdev/raid/concat.c
+module/bdev/uring/          -
+app/spdk_dd/                -

images/spdk/unit-tests.sh

@@ -0,0 +1,136 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2026 Evariops.
+#
+# unit-tests.sh <patches_dir> <map>
+#
+# Run from the root of an SPDK tree that is already built (the image's builder
+# stage). Builds and runs the upstream unit-test suites the patch series owes:
+# unit-tests.map says which suites a patched file names.
+#
+# The suites are compiled against the libraries of that build: SPDK is not
+# built a second time. Each suite is built from its own leaf directory, because
+# the parent Makefiles skip some suites with a mere warning (blob.c wants
+# CUnit 2.1-3), and a skip must not read as a pass. A suite whose binary is
+# missing after its build fails the stage, as does a suite that fails.
+set -euo pipefail
+
+patches_dir="${1:?usage: unit-tests.sh <patches_dir> <map>}"
+map="${2:?usage: unit-tests.sh <patches_dir> <map>}"
+
+# Plain bash 3 on purpose (no mapfile, no associative arrays): the selection is
+# checkable on any workstation, not only in the builder.
+
+# ── The files the series patches ──
+touched=()
+while IFS= read -r f; do
+	touched+=("${f}")
+done < <(sed -n 's|^+++ b/||p' "${patches_dir}"/[0-9]*.patch | sort -u)
+
+# ── The map ──
+prefixes=()
+suites_of=()
+while read -r prefix rest; do
+	[[ -z "${prefix}" || "${prefix}" == \#* ]] && continue
+	prefixes+=("${prefix}")
+	suites_of+=("${rest}")
+done < "${map}"
+
+# ── Suites owed ──
+owed=()
+unmapped=()
+for f in "${touched[@]}"; do
+	case "${f}" in
+		test/unit/*)
+			# A patch's own tests: the leaf directory of the file it touches.
+			owed+=("$(dirname "${f#test/unit/}")")
+			continue
+			;;
+		lib/*.c | module/*.c | app/*.c) ;;
+		*) continue ;;
+	esac
+
+	best=-1
+	best_len=0
+	for i in "${!prefixes[@]}"; do
+		p="${prefixes[$i]}"
+		if [[ "${f}" == "${p}"* && ${#p} -gt ${best_len} ]]; then
+			best=${i}
+			best_len=${#p}
+		fi
+	done
+	if ((best < 0)); then
+		unmapped+=("${f}")
+		continue
+	fi
+	for s in ${suites_of[$best]}; do
+		[[ "${s}" == "-" ]] || owed+=("${s}")
+	done
+done
+
+if ((${#unmapped[@]} > 0)); then
+	echo "FATAL: patched file(s) with no line in unit-tests.map (map them; \"-\" when upstream has no suite):" >&2
+	printf '  %s\n' "${unmapped[@]}" >&2
+	exit 1
+fi
+
+suites=()
+while IFS= read -r s; do
+	suites+=("${s}")
+done < <(printf '%s\n' "${owed[@]}" | sort -u)
+echo "Unit-test suites owed by the series (${#suites[@]}):"
+printf '  %s\n' "${suites[@]}"
+
+# ── Build ──
+# The ut library is built with the tests only (lib/Makefile), and the image
+# configures them off. Every suite is built even after one fails, and so run
+# below: one pass reports every broken suite, not the first one in the list.
+make -C lib/ut -j"$(nproc)"
+logs="$(mktemp -d)"
+failed=()
+for s in "${suites[@]}"; do
+	if [[ ! -d "test/unit/${s}" ]]; then
+		echo "FATAL: unit-tests.map names test/unit/${s}, which does not exist" >&2
+		exit 1
+	fi
+	if ! make -C "test/unit/${s}" -j"$(nproc)" > "${logs}/${s//\//_}.build.log" 2>&1; then
+		echo "FAIL ${s}: does not build"
+		{ grep -E 'error|undefined reference' "${logs}/${s//\//_}.build.log" | head -n 40 | sed 's/^/     /'; } || true
+		failed+=("${s}")
+	fi
+done
+
+# ── Run ──
+for s in "${suites[@]}"; do
+	src="$(find "test/unit/${s}" -maxdepth 1 -name '*_ut.c' | head -n 1)"
+	if [[ -z "${src}" ]]; then
+		echo "FATAL: test/unit/${s} holds no *_ut.c" >&2
+		exit 1
+	fi
+	bin="${src%.c}"
+	log="${logs}/${s//\//_}.log"
+	if [[ ! -x "${bin}" ]]; then
+		# A build that failed is reported above; a build that "succeeded"
+		# without its binary (a skip) is reported here.
+		[[ " ${failed[*]} " == *" ${s} "* ]] || {
+			echo "FAIL ${s}: ${bin} was not built"
+			failed+=("${s}")
+		}
+		continue
+	fi
+	start=${SECONDS}
+	if "${bin}" > "${log}" 2>&1; then
+		echo "PASS ${s} ($((SECONDS - start)) s)"
+		grep -E '^\s+(suites|tests|asserts)\s' "${log}" | sed 's/^/     /' || true
+	else
+		echo "FAIL ${s} ($((SECONDS - start)) s)"
+		tail -n 80 "${log}" | sed 's/^/     /'
+		failed+=("${s}")
+	fi
+done
+
+if ((${#failed[@]} > 0)); then
+	echo "FAILED: ${#failed[@]} of ${#suites[@]} suites: ${failed[*]}" >&2
+	exit 1
+fi
+echo "All ${#suites[@]} suites passed."

module/bdev/cbt/README.md

@@ -55,6 +55,8 @@ The module cannot converge on its own when the write rate approaches rebuild ban
 
 Clearing is reset-driven: there is no automatic clear. Once the backend is re-added and all backends are synchronized, the orchestrator MUST call `bdev_cbt_reset`, or the bitmap grows monotonically and "partial" rebuilds degrade toward full-surface copies.
 
+A reset needs a moment when every backend is known in sync, and a member that leaves without notice gives none: the epoch the raid opens at its ejection (`vbdev_cbt_auto_epoch_open`) cannot clear anything, since the writes the member just missed are in the live bitmap. So the live bitmap is also kept in windows: `bdev_cbt_rotate`, called periodically while no epoch is live, moves it to a previous-window bitmap and starts it again empty, and the epoch at ejection takes the previous window back before it opens. Its delta then holds one to two windows of writes before the departure, and everything after, instead of every write since the last reset. Two rotations are at least `CBT_ROTATE_MIN_INTERVAL_US` (10 s) apart whoever asks, so a write missed a few milliseconds before the ejection is always in one of the two bitmaps. A reset clears both.
+
 ## RAID integration
 
 The companion patch (`patches/0001-raid-add-skip_rebuild-parameter.patch`) adds a `skip_rebuild` boolean to `bdev_raid_add_base_bdev`. When true, the RAID module skips its full surface rebuild and instead quiesces the raid, opens `base_channel[slot]` on every existing I/O channel for the re-added bdev — without this, existing channels would never write to the backend — then unquiesces and writes the superblock.