Shared BATS test infrastructure for R-fx Networks projects.
Consumed as a git submodule at tests/infra/ in each project.
Copyright (C) 2002-2026 R-fx Networks proj@rfxn.com
-
Add batsman as a submodule:
cd your-project git submodule add https://github.com/rfxn/batsman.git tests/infra git submodule update --init --recursive -
Create a project Dockerfile (
tests/Dockerfile):ARG BASE_IMAGE=myproject-base-debian12 FROM ${BASE_IMAGE} RUN apt-get update && apt-get install -y --no-install-recommends your-packages COPY . /opt/project-src/ RUN cd /opt/project-src && sh install.sh COPY tests/ /opt/tests/ WORKDIR /opt/tests CMD ["bats", "--formatter", "tap", "/opt/tests/"]
-
Create
tests/run-tests.sh(thin wrapper):#!/bin/bash set -e SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BATSMAN_PROJECT="myproject" BATSMAN_PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" BATSMAN_TESTS_DIR="$SCRIPT_DIR" BATSMAN_INFRA_DIR="$SCRIPT_DIR/infra" BATSMAN_DEFAULT_OS="debian12" BATSMAN_CONTAINER_TEST_PATH="/opt/tests" BATSMAN_SUPPORTED_OS="debian12 centos7 rocky8 rocky9 ubuntu2004 ubuntu2404" source "$BATSMAN_INFRA_DIR/lib/run-tests-core.sh" batsman_run "$@"
-
Create
tests/Makefile:BATSMAN_OS_MODERN := debian12 rocky9 ubuntu2404 BATSMAN_OS_LEGACY := centos7 rocky8 ubuntu2004 BATSMAN_OS_DEEP := BATSMAN_OS_EXTRA := BATSMAN_OS_ALL := $(BATSMAN_OS_MODERN) $(BATSMAN_OS_LEGACY) $(BATSMAN_OS_DEEP) $(BATSMAN_OS_EXTRA) BATSMAN_RUN_TESTS := ./run-tests.sh BATSMAN_PROJECT := myproject include infra/include/Makefile.tests
-
Run tests:
make -C tests test
batsman uses a two-phase Docker build to separate base OS infrastructure from project-specific dependencies:
-
Phase 1 (base image): Built from batsman's
dockerfiles/Dockerfile.<os>. Installs system packages, common utilities, and BATS. Tagged as<project>-base-<os>(e.g.,apf-base-debian12). -
Phase 2 (project image): Built from the project's own
tests/Dockerfile.<os>. UsesARG BASE_IMAGE/FROM ${BASE_IMAGE}to layer on project-specific packages, install the project, and copy test files.
This separation means base images are cached and shared across CI runs, while project images rebuild only when project code changes.
lib/run-tests-core.sh is a sourced library that provides:
- Round-robin distribution:
.batsfiles are distributed across N Docker containers (default:nproc). Each container runs a subset of tests independently. - TAP aggregation: Output from all containers is collected and merged into a single TAP stream.
- Named containers: Each container gets a deterministic name
(
<project>-<os>-<pid>-g<N>) for easy debugging. Containers are cleaned up on exit, including onSIGINT/SIGTERM. - Formatter restriction: Parallel mode forces
tapformatter for TAP stream aggregation. The--formatteroption applies only to sequential and direct modes. Usemake test-verbose(sequential) for pretty-formatted output. - Sequential fallback: When
--parallelis not passed, tests run in a single container.
Docker images accumulate across test runs. batsman provides opt-in cleanup:
- Auto-prune: After every build, dangling images (orphaned by tag replacement) are automatically pruned. This is silent and always safe.
--cleanflag: Removes the base and test images for the target OS after the test run completes. Test exit codes are preserved — cleanup never masks failures.batsman_clean(): Public function callable from scripts with three modes: no args (current OS),--all(all project images),--dangling-only.- Makefile targets:
clean(current project),clean-all(all batsman projects),clean-dangling(dangling only).
.github/workflows/test.yml is a reusable GitHub Actions workflow called via
workflow_call. It:
- Checks out the project with
submodules: recursive - Sets up Docker Buildx with the
docker-containerdriver - Builds the base image with GHA cache (
type=gha) - Builds the project image with plain
docker build(sees the loaded base) - Runs tests in the project image
The hybrid approach (docker-container for cached base build, plain docker for
project build) works around limitations with type=gha cache on older Docker
versions.
JUnit reporting: When reports: true (default), each OS job generates a
JUnit XML report via --report-formatter junit, uploads it as a GitHub
artifact (14-day retention), and writes a test summary table to the job
summary page.
Concurrency control: Each OS job runs in a concurrency group keyed by
<project-name>-<git-ref>-<os>. Rapid pushes to the same branch auto-cancel
superseded runs per-OS. Callers can override the prefix with the
concurrency-group input.
batsman provides 9 base OS images spanning three tiers plus an extra tier:
| Target | Base Image | Package Manager | Tier | Notes |
|---|---|---|---|---|
| debian12 | debian:12-slim |
apt-get | Modern | Default target |
| rocky9 | rockylinux:9-minimal |
microdnf | Modern | |
| ubuntu2404 | ubuntu:24.04 |
apt-get | Modern | |
| centos7 | centos:7 |
yum | Legacy | EOL, vault repos |
| rocky8 | rockylinux:8-minimal |
microdnf | Legacy | |
| ubuntu2004 | ubuntu:20.04 |
apt-get | Legacy | |
| centos6 | centos:6 |
yum | Deep Legacy | EOL, vault repos, TLS fallback |
| ubuntu1204 | ubuntu:12.04 |
apt-get | Deep Legacy | EOL, old-releases repos, TLS fallback |
| rocky10 | rockylinux:10 |
dnf | Extra |
Tiers group OS targets by age and compatibility characteristics:
Modern — Current production targets. Full TLS support, modern package managers, Bash 5.x. These run in CI by default for all projects.
Legacy — Older but still commonly deployed. EOL repositories may be needed
(CentOS 7 uses vault.centos.org). Bash 4.2+. These run in CI to catch
compatibility regressions.
Deep Legacy — CentOS 6 (Bash 4.1, kernel 2.6.32) and Ubuntu 12.04 (Bash 4.2). These define the portability floor:
wgetmay not support TLS 1.2+ —install-bats.shprovides TLS fallback modes usingwget --no-check-certificateandcurl -sSL -kwith OS-specific ordering (seeTLS_FALLBACKin Configuration Reference). SHA256 checksums verify download integrity regardless of TLS mode.- EOL repositories:
vault.centos.orgfor CentOS 6,old-releases.ubuntu.comfor Ubuntu 12.04. - No systemd — SysV init only.
Extra — Targets not included in CI by default. Available for manual testing
via make -C tests test-<os>. Rocky 10 is in this tier pending stable release.
Projects may also use Extra for non-OS variants (e.g., LMD's yara-x target).
| OS Family | Manager | Install Command | Notes |
|---|---|---|---|
| Debian/Ubuntu | apt-get | apt-get install -y --no-install-recommends |
|
| CentOS 6/7 | yum | yum install -y |
|
| Rocky 8/9 (minimal) | microdnf | microdnf install -y |
No --allowerasing |
| Rocky 10 | dnf | dnf install -y --allowerasing |
Full dnf |
Note: Rocky 8/9 minimal images ship coreutils-single which conflicts
with coreutils via microdnf. Omit coreutils from package lists on these
targets — coreutils-single provides equivalent commands.
Each OS needs a project Dockerfile. The default target (debian12) uses
tests/Dockerfile; others use tests/Dockerfile.<os>.
ARG BASE_IMAGE=myproject-base-debian12
FROM ${BASE_IMAGE}
# Project-specific packages only — base utilities are in the base image
RUN apt-get update && apt-get install -y --no-install-recommends \
iptables iproute2
# Install project
COPY . /opt/project-src/
RUN cd /opt/project-src && sh install.sh
# Copy tests
COPY tests/ /opt/tests/
WORKDIR /opt/tests
CMD ["bats", "--formatter", "tap", "/opt/tests/"]For RHEL-family targets, use the appropriate package manager:
# Rocky 8/9 (microdnf)
RUN microdnf install -y iproute && microdnf clean all
# Rocky 10 (dnf)
RUN dnf install -y --allowerasing iproute && dnf clean all
# CentOS 6/7 (yum)
RUN yum install -y iproute && yum clean allThe wrapper sets project-specific variables and sources the orchestration engine. It should be ~20-30 lines.
#!/bin/bash
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Required variables
BATSMAN_PROJECT="myproject"
BATSMAN_PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
BATSMAN_TESTS_DIR="$SCRIPT_DIR"
BATSMAN_INFRA_DIR="$SCRIPT_DIR/infra"
BATSMAN_CONTAINER_TEST_PATH="/opt/tests"
BATSMAN_SUPPORTED_OS="debian12 centos6 centos7 rocky8 rocky9 rocky10 ubuntu1204 ubuntu2004 ubuntu2404"
# Optional variables
BATSMAN_DOCKER_FLAGS="--privileged" # Only if needed (e.g., iptables tests)
BATSMAN_DEFAULT_OS="debian12"
BATSMAN_BASE_OS_MAP="" # e.g., "yara-x=debian12" for variants
source "$BATSMAN_INFRA_DIR/lib/run-tests-core.sh"
batsman_run "$@"The project Makefile defines tier groupings and includes Makefile.tests:
BATSMAN_OS_MODERN := debian12 rocky9 ubuntu2404
BATSMAN_OS_LEGACY := centos7 rocky8 ubuntu2004
BATSMAN_OS_DEEP := centos6 ubuntu1204
BATSMAN_OS_EXTRA := rocky10
BATSMAN_OS_ALL := $(BATSMAN_OS_MODERN) $(BATSMAN_OS_LEGACY) $(BATSMAN_OS_DEEP) $(BATSMAN_OS_EXTRA)
BATSMAN_RUN_TESTS := ./run-tests.sh
BATSMAN_PROJECT := myproject
include infra/include/Makefile.testsProjects call the reusable workflow from their own CI configuration:
name: Tests
on:
push:
branches: [master, '2.*']
pull_request:
branches: [master]
jobs:
test:
uses: rfxn/batsman/.github/workflows/test.yml@v1.0.3
with:
project-name: myproject
os-matrix: '["debian12","centos7","rocky8","rocky9","ubuntu2004","ubuntu2404"]'
docker-run-flags: '--privileged' # omit if not needed
# test-path: '/opt/custom/tests' # override if non-standard (default: /opt/tests)
# reports: false # disable JUnit reports (default: true)| Variable | Default | Purpose |
|---|---|---|
BATS_VERSION |
1.13.0 |
bats-core version |
BATS_SUPPORT_VERSION |
0.3.0 |
bats-support version |
BATS_ASSERT_VERSION |
2.1.0 |
bats-assert version |
BATS_CORE_SHA256 |
(matches pinned version) | SHA256 checksum for bats-core tarball |
BATS_SUPPORT_SHA256 |
(matches pinned version) | SHA256 checksum for bats-support tarball |
BATS_ASSERT_SHA256 |
(matches pinned version) | SHA256 checksum for bats-assert tarball |
TLS_FALLBACK |
0 |
TLS mode: 0=standard wget, 1=wget --no-check-certificate with curl fallback, 2=curl primary with wget fallback |
| Variable | Required | Purpose |
|---|---|---|
BATSMAN_PROJECT |
yes | Image tag prefix, container naming |
BATSMAN_PROJECT_DIR |
yes | Docker build context root |
BATSMAN_TESTS_DIR |
yes | Directory containing .bats files |
BATSMAN_INFRA_DIR |
yes | Path to batsman submodule |
BATSMAN_DOCKER_FLAGS |
no | Extra docker run flags (e.g. --privileged) |
BATSMAN_DEFAULT_OS |
no | Default OS when --os omitted (default: debian12) |
BATSMAN_CONTAINER_TEST_PATH |
yes | Test directory path inside container |
BATSMAN_SUPPORTED_OS |
yes | Space-separated list of supported OS targets |
BATSMAN_BASE_OS_MAP |
no | Variant-to-base mappings (e.g. "yara-x=debian12") |
BATSMAN_TEST_TIMEOUT |
no | Per-test timeout in seconds; overridden by --timeout CLI flag |
BATSMAN_REPORT_DIR |
no | Host directory for JUnit XML reports; overridden by --report-dir CLI flag |
| Variable | Required | Purpose |
|---|---|---|
BATSMAN_OS_MODERN |
yes | Modern tier OS list |
BATSMAN_OS_LEGACY |
yes | Legacy tier OS list |
BATSMAN_OS_DEEP |
no | Deep legacy tier OS list |
BATSMAN_OS_EXTRA |
no | Extra OS targets (e.g. rocky10, yara-x) |
BATSMAN_OS_ALL |
yes | Combined full OS list |
BATSMAN_RUN_TESTS |
yes | Path to project run-tests.sh |
BATSMAN_PROJECT |
no* | Project name for image tags (required for clean targets) |
PARALLEL_JOBS |
no | Cross-OS parallel job count for xargs -P (default: nproc; override via make ... PARALLEL_JOBS=N) |
| Input | Required | Default | Purpose |
|---|---|---|---|
project-name |
yes | — | Project name for image tags |
os-matrix |
yes | — | JSON array of OS targets |
docker-run-flags |
no | "" |
Extra docker run flags |
timeout |
no | 15 |
Job timeout in minutes |
dockerfile-dir |
no | tests |
Directory containing project Dockerfiles |
concurrency-group |
no | "" |
Concurrency group prefix (empty = default per-project grouping) |
test-path |
no | /opt/tests |
Test directory path inside container |
reports |
no | true |
Generate JUnit XML reports and upload as artifacts |
batsman includes its own test suite that validates the orchestration engine. It bootstraps itself — the engine builds a Docker image from its own base Dockerfiles, copies its library into the container, and runs BATS tests against it. Unit tests cover argument parsing, variable validation, variant mapping, parallel distribution, and file discovery.
make -C tests test # parallel (default)
make -C tests test-verbose # pretty output (sequential)| Target | Description |
|---|---|
test |
Default OS, parallel (default goal) |
test-serial |
Default OS, sequential (single container) |
test-verbose |
Default OS, pretty formatter (sequential) |
test-report |
Default OS, parallel, JUnit XML in reports/ |
test-<os> |
Specific OS, parallel |
test-modern |
Modern tier, sequential across OS |
test-legacy |
Legacy tier, sequential across OS |
test-deep-legacy |
Deep legacy tier, sequential across OS |
test-all |
All tiers, sequential across OS |
test-modern-parallel |
Modern tier, parallel across OS |
test-legacy-parallel |
Legacy tier, parallel across OS |
test-deep-legacy-parallel |
Deep legacy tier, parallel across OS |
test-all-parallel |
All tiers, parallel across OS |
clean |
Remove all images for current project |
clean-all |
Remove all batsman project images across all projects |
clean-dangling |
Prune dangling images only (always safe) |
# Run on default OS (parallel)
./tests/run-tests.sh --parallel
# Run on a specific OS
./tests/run-tests.sh --os rocky9 --parallel
# Filter tests by name
./tests/run-tests.sh --filter "install" --parallel
# Run a specific .bats file
./tests/run-tests.sh /opt/tests/01-install.bats
# Custom parallelism level
./tests/run-tests.sh --parallel 4
# Pretty output (sequential only)
./tests/run-tests.sh --formatter pretty
# Per-test timeout (30 seconds)
./tests/run-tests.sh --timeout 30 --parallel
# Filter tests by tag
./tests/run-tests.sh --filter-tags "smoke" --parallel
# Exclude slow-tagged tests
./tests/run-tests.sh --filter-tags '!slow' --parallel
# Stop on first failure
./tests/run-tests.sh --abort --parallel
# Generate JUnit XML reports
./tests/run-tests.sh --report-dir /tmp/reports --parallel
# Clean up project images after test run
./tests/run-tests.sh --os rocky9 --clean --parallel
# Show batsman version
./tests/run-tests.sh --versionbatsman can be used by any Bash project that needs cross-OS BATS testing.
- Docker (with BuildKit support)
- GNU Make
- Bash 4.1+
- Git (for submodule)
For a project with a single OS target and no special requirements:
tests/Dockerfile:
ARG BASE_IMAGE=mylib-base-debian12
FROM ${BASE_IMAGE}
COPY . /opt/src/
COPY tests/ /opt/tests/
WORKDIR /opt/tests
CMD ["bats", "--formatter", "tap", "/opt/tests/"]tests/run-tests.sh:
#!/bin/bash
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BATSMAN_PROJECT="mylib"
BATSMAN_PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
BATSMAN_TESTS_DIR="$SCRIPT_DIR"
BATSMAN_INFRA_DIR="$SCRIPT_DIR/infra"
BATSMAN_CONTAINER_TEST_PATH="/opt/tests"
BATSMAN_SUPPORTED_OS="debian12"
source "$BATSMAN_INFRA_DIR/lib/run-tests-core.sh"
batsman_run "$@"tests/Makefile:
BATSMAN_OS_MODERN := debian12
BATSMAN_OS_LEGACY :=
BATSMAN_OS_DEEP :=
BATSMAN_OS_EXTRA :=
BATSMAN_OS_ALL := $(BATSMAN_OS_MODERN)
BATSMAN_RUN_TESTS := ./run-tests.sh
BATSMAN_PROJECT := mylib
include infra/include/Makefile.testsThen make -C tests test builds the base image, builds the project image,
and runs your BATS tests.
BATSMAN_DOCKER_FLAGS— Set to--privilegedif your tests need iptables, kernel modules, or network namespaces. Leave empty otherwise.BATSMAN_BASE_OS_MAP— Map variant names to base OS images. For example,"yara-x=debian12"means theyara-xvariant reuses thedebian12base image but has its own project Dockerfile (tests/Dockerfile.yara-x).- Tier groupings — Adjust
BATSMAN_OS_MODERN,BATSMAN_OS_LEGACY, etc. to match your project's support matrix. - TLS fallback — Deep legacy Dockerfiles set
TLS_FALLBACK=2in theinstall-bats.shinvocation to handle systems wherewgetcannot connect to GitHub over TLS 1.2+.
Pin the submodule to a specific tag for reproducibility:
cd tests/infra
git fetch --tags
git checkout v1.0.3
cd ../..
git add tests/infra
git commit -m "Pin batsman submodule to v1.0.3"In CI workflow callers, reference the same tag:
uses: rfxn/batsman/.github/workflows/test.yml@v1.0.3Upgrade notes for consumer projects. Newest version first.
SHA256 checksum verification (transparent)
install-bats.sh now verifies SHA256 checksums for all three downloaded
tarballs (bats-core, bats-support, bats-assert) after download and before
extraction. This is transparent for standard usage — the checksums match the
pinned versions shipped with batsman. If you override BATS_VERSION,
BATS_SUPPORT_VERSION, or BATS_ASSERT_VERSION via environment variables,
you must also set the corresponding BATS_CORE_SHA256, BATS_SUPPORT_SHA256,
or BATS_ASSERT_SHA256 to match your custom tarballs. A mismatch aborts the
build with expected vs actual hash output.
Default parallelism reduced
Default parallel container count changed from nproc*2 to nproc — both for
intra-OS containers (run-tests-core.sh) and cross-OS parallel jobs
(Makefile.tests PARALLEL_JOBS). This prevents compounding container storms
on test-all-parallel targets. To restore the previous behavior, pass an
explicit count:
# Intra-OS: override via CLI
./tests/run-tests.sh --parallel $(($(nproc) * 2))
# Cross-OS: override via Make variable
make -C tests test-all-parallel PARALLEL_JOBS=$(($(nproc) * 2))CLI argument parsing stricter
Flags that require a value (--os, --filter, --filter-tags, --formatter,
--timeout, --report-dir) now error when the trailing argument is missing.
--timeout rejects non-numeric values. Unknown --flags emit a warning
instead of silently routing to direct mode. These changes only affect incorrect
invocations — correct usage is unaffected.
BATS 1.13.0 run-variable unset — BREAKING CHANGE
BATS was upgraded from 1.11.0 to 1.13.0. Starting with BATS 1.12.0, the run
command unsets $output, $lines, $stderr, and $stderr_lines at the start
of each invocation. Tests that rely on stale values from a previous run call
will silently produce incorrect results.
Broken pattern:
@test "example" {
run some_command
run another_command
# BUG: $output now contains only another_command's output
# In BATS < 1.12.0, if another_command produced no output,
# $output would still hold some_command's output (crosstalk)
assert_output --partial "from some_command" # FAILS
}Fixed pattern:
@test "example" {
run some_command
local first_output="$output"
run another_command
assert_output --partial "from another_command"
[[ "$first_output" == *"from some_command"* ]]
}Audit your existing test suites for this pattern:
grep -n 'run ' tests/*.bats | grep -B1 'assert_output\|assert_line\|\$output\|\$lines'New CLI options available
v1.0.2 added --timeout, --abort, --filter-tags, --report-dir, --clean,
and --version. See the Script CLI section for usage.
CI workflow changes
The reusable workflow gained test-path (default /opt/tests), reports
(default true), and concurrency-group inputs. JUnit XML reports are uploaded
as artifacts with 14-day retention and written to the job summary.
Update your CI caller reference:
# Before
uses: rfxn/batsman/.github/workflows/test.yml@v1.0.1
# After
uses: rfxn/batsman/.github/workflows/test.yml@v1.0.2Update your submodule pin:
cd tests/infra
git fetch origin --tags --force
git checkout v1.0.2
cd ../..
git add tests/infra
git commit -m "Pin batsman submodule to v1.0.2"Variant mapping (new capability)
v1.0.1 introduced BATSMAN_BASE_OS_MAP for mapping non-OS variant names to
base OS images (e.g., "yara-x=debian12"). No action required unless you want
to use this feature.
Makefile include and CI workflow introduced
v1.0.1 added include/Makefile.tests and .github/workflows/test.yml. Projects
upgrading from v1.0.0 need to create a tests/Makefile and CI workflow caller.
See the Integration Guide section for templates.
Rocky 10 image name fix (transparent)
The Rocky 10 base image was corrected from rockylinux:10 to
rockylinux/rockylinux:10. No action required.
| Project | Docker Flags | Container Test Path | OS Targets | Notable | Repository |
|---|---|---|---|---|---|
| APF | --privileged |
/opt/tests |
9 | iptables/netfilter tests | rfxn/apf |
| BFD | (none) | /opt/tests |
9 | rfxn/bfd | |
| LMD | (none) | /opt/tests |
9 + yara-x | BATSMAN_BASE_OS_MAP for yara-x variant | rfxn/lmd |
| tlog_lib | (none) | /opt/tests |
9 | Zero project packages needed | rfxn/tlog_lib |
GNU General Public License v2 — see LICENSE.