We've now merged #1770 to add support for integration tests 馃帀 However, buildx has been around for several years before these tests, so we don't have anywhere near the amount of test coverage that we want. These tests are essential to track regressions, detect changes in behavior as we refactor and build new functionality, and add new modes and drivers.
In general, all new functionality should be added with tests wherever feasible, but we need a plan to work through the existing backlog of code with no tests. This issue is to track what broad groups of tests we need, and how far along we are. This list isn't currently complete -- but as a general rule of thumb, if we document the behavior on https://docs.docker.com, we should be testing it.
Ideally, each sub-point in the list would end up as a PR, though maybe it makes sense to group some of the smaller ones together, or split some of the larger ones up.
(note to maintainers, feel free to add anything that you think is missing 馃コ)
The first type of tests that we need are for the top-level commands package - at the time of writing, these currently reside in the top-level tests package. These are designed to test each of the buildx subcommands, such as build, bake, inspect, etc.
The areas that we need coverage for:
The second type of tests that we need are for the top-level driver package. These drivers allow connecting to buildkit running in different configurations. Each driver supports lots of different types of options, and so we should have tests that test for configuring and setting up buildkit with these different options. Currently, we don't have any tests of this format.
The areas that we need coverage for:
We've now merged #1770 to add support for integration tests 馃帀 However, buildx has been around for several years before these tests, so we don't have anywhere near the amount of test coverage that we want. These tests are essential to track regressions, detect changes in behavior as we refactor and build new functionality, and add new modes and drivers.
In general, all new functionality should be added with tests wherever feasible, but we need a plan to work through the existing backlog of code with no tests. This issue is to track what broad groups of tests we need, and how far along we are. This list isn't currently complete -- but as a general rule of thumb, if we document the behavior on https://docs.docker.com, we should be testing it.
Ideally, each sub-point in the list would end up as a PR, though maybe it makes sense to group some of the smaller ones together, or split some of the larger ones up.
(note to maintainers, feel free to add anything that you think is missing 馃コ)
The first type of tests that we need are for the top-level
commandspackage - at the time of writing, these currently reside in the top-leveltestspackage. These are designed to test each of the buildx subcommands, such asbuild,bake,inspect, etc.The areas that we need coverage for:
buildx build/buildx bake--platform) tests: build multiplatform聽#2620--cache-to/--cache-from/--no-cache/--no-cache-filter)--output/--load/--push/--tag) build: handle push/load shorthands for multi exporters聽#2302--build-arg/--build-context) build: validate build args and labels聽#2040--metadata-file/--iidfile)--print) test: build print聽#2423--secret/--ssh) tests: build secret聽#2381--shm-size) test: build shm-size and ulimit聽#2279--ulimit) test: build shm-size and ulimit聽#2279--label) build: validate build args and labels聽#2040--attest/--provenance/--sbom) build: error on attests on non-multiplatform driver聽#1988--progress) tests: add basic build progress test聽#1925BUILDX_EXPERIMENTALenv var,--detachflag) tests: test with buildx experimental聽#2376buildx imagetoolsbuildx imagetools inspect) imagetools: Allow annotations for OCI image index聽#1965 tests: add integration test for imagetools create聽#1978buildx imagetools create) tests: add imagetools tests for copying manifests and indexes聽#2013--invokeflag--invokeflag (debug-shell,on-error, shell commands)attach/list/help/etc)buildx du/buildx prunebuildx version(Integration test for docker buildx version聽#1985)buildx create/buildx rm/buildx ls/buildx stop/buildx useThe second type of tests that we need are for the top-level
driverpackage. These drivers allow connecting to buildkit running in different configurations. Each driver supports lots of different types of options, and so we should have tests that test for configuring and setting up buildkit with these different options. Currently, we don't have any tests of this format.The areas that we need coverage for:
dockerdriverdocker-containerdriver (see docs)kubernetesdriver (see docs)remotedriver (see docs)