Add python wheel packaging#1
Open
CavRiley wants to merge 14 commits into
Open
Conversation
Add Wrapping/Packaging/, the wheel definitions that build ITK's Python wheels
from this repository by slicing a single pre-built, Python-wrapped ITK tree into
per-group wheels, with zero ITK recompilation per wheel.
ITK already emits a per-module install component
(<Module>PythonWheelRuntimeLibraries) when configured with
WRAP_ITK_INSTALL_COMPONENT_PER_MODULE=ON; that is the slicing seam. This change
adds the wheel definitions on top of it:
* one per-group pyproject.toml (core, numerics, io, filtering, registration,
segmentation) plus a platform/ABI-tagged "itk" umbrella, each a standard
PEP 517 project consumed by scikit-build-core;
* the module placement and per-component install CMake (relocated from
ITKPythonPackage, Apache-2.0), driven by ITKPythonPackage_WHEEL_NAME;
* a shared CMakeLists.txt that reuses a pre-built ITK tree
(ITKPythonPackage_ITK_BINARY_REUSE) instead of recompiling.
Each group wheel tracks ITK's version; the "itk" meta wheel carries platform and
ABI tags (matching the PyPI itk wheel) rather than py3-none-any so that
"pip install itk" only resolves where the group wheels exist.
Add the local orchestration for the in-tree wheel build:
* Utilities/Maintenance/build_wheels.py builds the wrapped ITK tree once then
loops over the Wrapping/Packaging/ group definitions, slicing each wheel
(and the meta wheel) by passing the ITK_BINARY_DIR reuse defines to
scikit-build-core. It guards against an unbuilt ITK tree and accepts a
subset of groups to build.
* a [tool.pixi.feature.wheel] environment plus configure-wheels,
build-wrapped-itk, and build-wheels tasks, so the full flow is
"pixi run -e wheel build-wrapped-itk" then "pixi run -e wheel build-wheels".
configure-wheels bakes in the three settings the slice requires
(WRAP_ITK_INSTALL_COMPONENT_PER_MODULE, the PythonWheel component identifier,
and a relative PY_SITE_PACKAGES_PATH so the install lands in the wheel).
Sketch a GitHub Actions workflow that builds the in-tree ITK Python wheels with cibuildwheel. One job per platform compiles the wrapped ITK tree once inside the build container (CIBW_BEFORE_ALL, guarded so the remaining wheel builds reuse it) then slices each Wrapping/Packaging/ group, letting cibuildwheel handle the manylinux/macOS containers, wheel repair, the abi3 tag, and the import test. The ~8.5 GB wrapped ITK tree cannot be shared across runners as an artifact, so it is built and consumed in place within each platform job rather than in a per-wheel matrix. This is a starting point; image, arch, and path details (and a Windows variant) need iteration in real CI.
Add Wrapping/Packaging/README.md describing the in-tree wheel build: the slicing architecture, the one-time wrapped-ITK build plus per-wheel slice commands, the per-group module placement, and platform findings (static linking, the runtime location, the relative install prefix, and a stale-ninja-log gotcha). Add an "In-Tree Wheel Build" section to the contributing Python packaging guide so the in-tree mechanism is discoverable alongside the existing ITKPythonPackage workflow, pointing at the pixi tasks and the Wrapping/Packaging/ README.
github-actions
Bot
added
area:Python wrapping
type:Infrastructure
area:Documentation
labels
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Thank you for contributing a pull request! 🙏
Welcome to the ITK community! 🤗👋☀️
We are glad you are here and appreciate your contribution. Please keep in mind our community participation guidelines. 📜
More support and guidance on the contribution process can be found in our contributing guide. 📖
This is an automatic message. Allow for time for the ITK community to be able to read the pull request and comment
on it.
The configure-wheels task used bash conditional expansion
(${WHEEL_CC:+...}) to add the compiler defines only when set. Pixi
parses task scripts with deno_task_shell, which accepts only bare $VAR
and rejects every brace form, so the task failed to parse before running
cmake and every Python-wheels CI job died in the CIBW_BEFORE_ALL hook.
Move the conditional-compiler logic into a Python driver,
Utilities/Maintenance/configure_wrapped_itk.py, mirroring build_wheels.py
in the same directory. The WHEEL_CC / WHEEL_CXX override interface is
preserved exactly: the C/C++ compiler defines are appended only when the
variables are set.
Each group wheel is built from the shared Wrapping/Packaging source, which requires ITKPythonPackage_WHEEL_NAME to pick the module group. The build_wheels.py driver passes it per group, but the cibuildwheel CI drives a bare `python -m build` whose one shared CIBW_CONFIG_SETTINGS cannot vary per group, so configure failed with "ITKPythonPackage_WHEEL_NAME must be defined". Default the name from SKBUILD_PROJECT_NAME when the driver did not set it. Each group's pyproject already declares name = itk-<group>; scikit-build-core normalizes that to itk_<group>, so convert underscores back to the itk-<group> form ipp_wheel_to_group expects. An explicit define still wins via the DEFINED guard, and a plain `pip wheel` of a group now configures without the driver.
…wheel The macOS matrix sets ITK_BINARY_DIR to $GITHUB_WORKSPACE/build-python. cibuildwheel shell-expands CIBW_ENVIRONMENT but passes CIBW_CONFIG_SETTINGS verbatim, so the cmake.define carried the literal string $GITHUB_WORKSPACE and configure failed with "ITK_BINARY_DIR must point at a pre-built ITK build tree". Drop the config-settings define and read ITK_BINARY_DIR from the environment in the shared wheel CMakeLists when the driver did not pass it. build_wheels.py still passes it as a define, which wins via the DEFINED guard; Linux keeps working because its /project path needs no expansion.
auditwheel rejected the Linux wheels ("too-recent versioned symbols") because
the CIBW_BEFORE_ALL hook compiled ITK's static libraries with the pixi
conda-forge compiler (g++ 14.3.0), whose libstdc++/glibc symbols are newer than
the manylinux_2_28 baseline, while cibuildwheel compiles each extension with the
image's gcc-toolset (g++ 14.2.1). The mismatched symbols propagated from the
static libs into the extension.
In the manylinux container, build the wrapped ITK tree with the image's own
gcc-toolset (via WHEEL_CC/WHEEL_CXX and pip-provided cmake/ninja) instead of
pixi, so ITK and the extensions share one ABI baseline. Native macOS builds keep
using the pixi env, where delocate handles the dylib ABI.
The wheels targeted cp39, but ITK 6 requires Python >= 3.10 (CMake/ITKSetPython3Vars.cmake sets PYTHON_VERSION_MIN 3.10) and its Python support uses 3.10-only syntax: itkConfig.py annotates with PEP 604 unions (message: Warning | str), which evaluate at import and raise TypeError on 3.9. Both platforms built, repaired, and installed the wheel, then failed the `import itk` smoke test on the cp39 runner. Raise the floor to 3.10 everywhere: CIBW_BUILD cp310-*, the manylinux interpreter (/opt/python/cp310-cp310), the abi3 tag (wheel.py-api cp310), and requires-python >=3.10 in every group and the meta package.
CavRiley
marked this pull request as ready for review
July 4, 2026 02:35
The PR-triggered build compiles the branch merged into main, so the wheels run main's Python support, which requires 3.11: itk/support/types.py does an unconditional `from typing import Self` (Self is 3.11+), and main sets PYTHON_VERSION_MIN 3.11. The cp310 wheels built, repaired, and installed, then failed the `import itk` smoke test with "cannot import name 'Self' from typing". A stale 3.10 floor in the branch's own CMake had misled the earlier bump. Raise the floor to 3.11: CIBW_BUILD cp311-*, the manylinux interpreter (/opt/python/cp311-cp311), the abi3 tag (wheel.py-api cp311), and requires-python >=3.11 across every group and the meta package.
cibuildwheel ran the import test after each group wheel in an isolated venv, but the group wheels depend on one another (itk-numerics needs itk-core==6.0b2) and those versions are not on PyPI, so the per-wheel install failed to resolve the sibling: "No matching distribution found for itk-core==6.0b2". Core passed because it only needs numpy. Drop CIBW_TEST_COMMAND and add a final step that installs the meta package and its six group wheels from the local wheelhouse (--no-index --find-links, with numpy from PyPI) and imports itk once — the correct integration test for a set of interdependent, unpublished wheels.
cibuildwheel builds musllinux by default, but ITK ships glibc (manylinux) wheels only, and the musl/Alpine container cannot execute the glibc vendored swig binary — the before-all failed there with "Failed to run vendored swig" while the manylinux build succeeded. Skip musllinux with CIBW_SKIP. The meta package is pure Python (no binaries), so auditwheel/delocate abort with "Failed to find any binary with the required architecture". Build it with CIBW_REPAIR_WHEEL_COMMAND empty so its platform+ABI tag is kept without repair.
…ainers Each group wheel is a separate cibuildwheel invocation, so on Linux every one started a fresh container and the before-all recompiled ITK — seven builds that overran the 6h job limit. macOS reused the build because its native invocations share the workspace. Mount one named Docker volume into every Linux container (CIBW_CONTAINER_ENGINE) and build wrapped ITK into it (ITK_BINARY_DIR=/itk-build/build-python). The before-all guard then finds the tree the first wheel built and every later wheel reuses it. cibw-build-wrapped-itk.sh takes the build location from ITK_BINARY_DIR, and configure_wrapped_itk.py honors WHEEL_ITK_BUILD_DIR; macOS and local builds keep using build-python unchanged.
All target platforms now build green (Linux reuses the one ITK build in ~1.5h, macOS arm64 succeeds). Remove the macos-x86_64 (macos-13) job: Intel macOS runners are being retired and it routinely timed out waiting in the runner queue. Add a "Wheel build summary" step that writes a table of every wheel in the wheelhouse to the job's Summary page, so the whole set is visible at once rather than only the last cibuildwheel invocation (the meta wheel) at the end of the log.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Checklist
Refer to the ITK Software Guide for
further development details if necessary.