Repository navigation
Conversation
Co-Authored-By: Cavan Riley <cavan-riley@uiowa.edu>
Co-Authored-By: Cavan Riley <cavan-riley@uiowa.edu>
Co-Authored-By: Cavan Riley <cavan-riley@uiowa.edu>
Co-Authored-By: Cavan Riley <cavan-riley@uiowa.edu>
Co-Authored-By: Cavan Riley <cavan-riley@uiowa.edu>
Document the build driver's full command line surface so a coding agent can compose a correct invocation without reading the source. CLAUDE.md redirects to AGENTS.md, which is authoritative. The worked example covers the common request shape: a local macOS build against an unmerged ITK pull request, with custom compiler flags and a non-default deployment target. It fetches the pull request ref by hand, because the driver fetches tags only and a failed checkout warns and keeps the commit already present.
Every build_wheels.py example omitted the pixi environment selector, so following the README landed in the default environment and aborted. Also corrects the dockcross image tag defaults, the directory the publish script searches for tarballs, the step count, and the --cleanup description, which read as the opposite of what the flag does. Adds the expected wheel tags and how to confirm a macOS binary matches its tag.
The ITK C++ build ran ninja with -j<cpu_count> -l<cpu_count>. On a 16-thread machine with 32 GB that is 2 GB per job; the template-heavy wrapping units peak at 3.9 GB in a single cl.exe, and the build was killed for memory pressure in 2 of 3 clean attempts. Jobs are now min(cpu_count, floor(ram_gb / 2.5)). The budget comes from a three-point sweep on that machine, working sets sampled every 0.5 s over the whole build: -j16 step 02 4716 s killed in 2 of 3 clean builds -j12 step 02 4842 s peak build-process sum 15.7 GB, min free 5.75 GB -j8 step 02 5291 s peak build-process sum 12.2 GB, min free 8.96 GB Above 12 there is at most 2.7% to gain; below it the cost climbs to 12%. The marginal cost between -j8 and -j12 was 0.84 GB per job because peaks rarely coincide, so 2.5 GB/job lands on the measured optimum (floor(31.8 / 2.5) = 12) with ~5 GB of floor. The value floors rather than rounds so it cannot land one job past the tested point. -l stays at the thread count and must never follow the job count. ninja honours -l on Windows (1.13.2: eight 3 s jobs under `-l 0.5` serialise to 25 s), so -l<jobs> on a box with more threads than jobs throttles ninja to one compiler whenever a saturating compile pushes load past <jobs>. Links are not the pressure -- link.exe peaked at 1.45 GB and the whole link phase is ~1.3 min of an ~80 min step -- so there is no link pool. ITK_BUILD_JOBS and ITK_BUILD_LOAD_LIMIT override for CI runner sizing. physical_memory_gb() is dependency-free so it works in every build env; unknown memory falls back to the previous cpu_count behaviour.
…s are pixi's conda-forge vs2022 activation exports CMAKE_GENERATOR="Visual Studio 17 2022". scikit-build-core honours it, so the seven ITK wheel builds in step 03 ran under MSBuild while ITK itself was configured with -G Ninja. The remote-module wheel build already passes -GNinja explicitly for this reason; do the same for the ITK wheels. Measured on a 16-thread Windows box: step 03 went from 190-216 s under the Visual Studio generator to 159-168 s under Ninja. An explicit -G also makes CMake ignore the ambient CMAKE_GENERATOR_PLATFORM/TOOLSET that the same activation exports and that Ninja would otherwise reject.
The reusable GitHub Actions workflow fetches this file by URL to learn which container image to pre-pull. Removing it left the action without a source for the image tags, and the file is the only place those coordinates are stated once for both architectures. The manylinux2014 branch is dropped: ITK 6 targets manylinux_2_28 only, so that path now reports an unknown version rather than resolving.
Nothing references docs/make.bat; docs/Makefile remains for building the documentation. The cache-path note claimed the packaging scripts enforce the build path convention without saying that build_wheels.py does not. Its default build root matches the convention only on Windows, so a cache built by calling it directly is unusable by the reusable workflow unless --build-dir-root is set. State the paths as a table and give the command to check a finished cache.
Add --module-itk-dir {build,install} to build_wheels.py. The default keeps
passing the ITK build tree as ITK_DIR. With "install", a new step
06_install_wrapped_itk_cplusplus runs cmake --install on the build tree
into a sibling <build tree>-install prefix, and the module is configured
with ITK_DIR=<prefix>/lib/cmake/ITK-<version>. The install tree is only
accepted when it carries ITKModuleExternal.cmake and the wrapping
infrastructure, which an ITK without ITK_INSTALL_WRAPPING_DEVELOPMENT_FILES
does not install; the failure names the option instead of letting the
module's first include() fail.
Every ITK configure now passes ITK_INSTALL_WRAPPING_DEVELOPMENT_FILES=ON so a
cached build tree can be installed later; an ITK without the option reports
it unused. The Windows module script forwards MODULE_ITK_DIR.
make_windows_zip.ps1 pinned the build root to C:\BDR and refused to run anywhere else. The root is now the directory holding the IPP checkout, so the same <root>\IPP, <root>\ITK, <root>\build layout works on any drive; the layout check still requires the checkout to be named IPP, which the cache and CI both assume. The module script's root becomes an environment override, DASHBOARD_BUILD_DIRECTORY, defaulting to C:\BDR. A cache is only consumable at the root it was built under, because the ITK build tree records absolute paths; both scripts say so, and CI keeps extracting at C:\BDR.
…elease Invoke-WebRequest has no timeout by default, so when pixi.sh was unreachable the Windows build hung at "Installing pixi..." with no error. Give the installer download a 60 second bound and, when it fails, fetch the release binary from GitHub directly, which is what the installer script does anyway. Both Windows entry points get the same treatment, and both now fail loudly if no pixi.exe exists afterwards.
build_superbuild_support_components checked the configure but not the cmake --build that fetches and builds oneTBB. When that download failed, the step was recorded as done and the run stopped in step 02 with "Could not find a package configuration file provided by TBB", two steps away from the cause. Check the build too.
build_wheels.py --platform-env manylinux228-py311 run on the host left MANYLINUX_VERSION empty, skipped auditwheel and produced linux_x86_64 wheels that PyPI rejects. The manylinux version now follows from the environment name, so the existing container check stops such a build and prints the dockcross command to use instead.
The builder enables the compiler launcher only for USE_CCACHE=ON, but build_wheels.py ignored --use-ccache and passed USE_CCACHE=1 through unchanged. Without ccache on PATH the launcher became the string None.
Inside the manylinux container the SSH remote of a mounted ITK checkout is unreachable, and the build stopped although ITK_GIT_TAG was already present. A failed fetch is now fatal only when the revision is missing.
pixi refuses environments that moved, so the checkout's .pixi created on the host cannot be used at /work in the container. Mount a .pixi from the manylinux build directory instead, and keep the pixi cache there because the container user has no writable home directory.
dockcross-manylinux-build-wheels.sh required TARGET_ARCH. Document it and ITK_PACKAGE_VERSION with the other variables of the script.
The regular expression missed modules whose itk-module.cmake mentions itk_module() in a comment or names the module through a variable, so those modules had no group and no wrapping detection. Load each file with a stand-in itk_module() and record the module directory.
ITK marks them EXCLUDE_FROM_DEFAULT, so the wheels left them out. A positional -DModule_<name>:BOOL=OFF still turns one off.
PyPI rejects files over 100 MB, which otherwise surfaces only at upload.
With the ingested modules, itk-filtering reached 148 MB on Linux x86_64. ITK_WHEEL_<group>_MODULE_OVERRIDES replaces the hard-coded dispatch of ITKBridgeNumPy and ITKVTK, and configuring warns when a wrapped module's dependency is in a wheel its own wheel does not require.
ITKIOTransformBase depends on ITKDisplacementField and ITKIOSpatialObjects on ITKSpatialObjects, both packaged in itk-numerics, but itk-io required only itk-core.
build_wheels.py never runs docker and never read args.use_sudo or the NO_SUDO value it copied into package_env_config. The manylinux shell scripts that run docker still honor NO_SUDO.
pixi run release-wheels <itk-ref> [<package-version>] builds every wheel the host can build: Windows x64 on Windows, macOS arm64 and Linux aarch64 on an Apple Silicon Mac, and Linux x86_64 or aarch64 on Linux, the Linux wheels in the manylinux container. It keeps its own ITK clone, resolves the ref to one commit for all targets and takes the package version from a tag. Linux hosts cannot build macOS wheels: those need Apple's SDK and a macOS host for the import test.
dict(image) in MixingITKAndNumPy.py reads the deprecated 'origin', 'spacing' and 'direction' keys, emitting six DeprecationWarnings with ITK 6. The example stays brief; the test filters those warnings.
Port of the SimpleITK_DEAD_STRIP option from SimpleITK (SimpleITK/SimpleITK#2784, which followed the link-time optimization study in SimpleITK/SimpleITK#2772). On Linux it compiles with -ffunction-sections -fdata-sections and links with gold --gc-sections --icf=safe. Off by default; other platforms are unchanged. A/B on ITK 957816d, manylinux_2_28 (GCC 14.2, gold 1.15), cp311 abi3: wheel off on itk_core 89.5 MB 87.0 MB itk_filtering 89.3 MB 86.4 MB itk_filtering_extra 60.3 MB 57.8 MB itk_numerics 64.7 MB 62.9 MB .text across the 96 wrapper modules fell 4.4%, against 15% in SimpleITK: _ITKCommonPython exports 33,625 symbols from its statically linked ITK libraries, and the linker must keep them all. ITK already links its wrapper modules with -flto, so link cost barely moved: the three largest modules, linked alone, took 1-4% longer and used the same peak memory (9.05 GB for _ITKImageIntensityPython). Seventeen Python cases gave bit-identical output in both arms; BSpline/LBFGSB registration ran 5-7% slower, as it did in SimpleITK, and the rest stayed within run-to-run noise. ld64 -dead_strip changed the macOS arm64 wheels by under 0.1% and is not applied there.
The wrapper modules link ITK statically, and ITK's static libraries keep default visibility on Linux, so _ITKCommonPython exported 33,625 symbols. The linker had to keep every one of them, which limited --gc-sections to a 4% saving. SimpleITK avoids this with a version script on its Python module (SimpleITK/SimpleITK#2784 and the visibility settings in SimpleITK's SuperBuild/External_ITK.cmake); this applies the same script to every ITK Python module. The modules share ITK's global state through the _ITKCommonPython capsule, not through exported symbols, and Python loads them RTLD_LOCAL. A/B on ITK 957816d, manylinux_2_28, cp311 abi3 (MB): wheel off --gc-sections + this itk_core 89.5 87.0 70.7 itk_filtering 89.3 86.4 67.8 itk_filtering_extra 60.3 57.8 39.3 itk_numerics 64.7 62.9 52.3 itk_io 36.9 34.9 17.6 itk_registration 37.3 36.0 27.6 itk_segmentation 19.7 18.9 14.3 Each module now exports 84 symbols. Seventeen Python cases gave bit-identical output; the BSpline/LBFGSB slowdown seen with --gc-sections alone is gone (2% faster than off), and the rest stayed within run-to-run noise. Thread defaults, the global warning flag, object factories, observers, down-casts, exceptions, and a pipeline through four modules all behave across modules as before.
With the PyInit_*-only version script, --dead-strip cut the manylinux wheels by 19-52% (itk_core 89.5 -> 70.7 MB), kept every benchmark output bit-identical, and left run time unchanged. It is a no-op on macOS and Windows, so the default changes only Linux builds; --no-dead-strip restores the previous link. macOS keeps ld64's default link. -dead_strip alone changed the wheels by under 0.1%; with exports limited to _PyInit_* it reduced one module by 6.9%, but no full macOS wheel build has verified that combination.
…er build build_wheels.py skips steps its build_log_<env>.json marks done, so a rerun with a different ITK commit only rebuilt the tarball and left the earlier wheels in dist, which release-wheels then listed as its output. Remove that log before each target, list only wheels the build wrote, and fail a target that writes none.
echo_check_call captured each command's output and printed it after the command exited, so a 30-minute ninja build showed no progress. It now echoes each line (stderr merged into stdout) as the command runs; the git and interpreter probes that parse output still capture it.
hjmjohnson
force-pushed
the
itk6-build-system-v2
branch
3 times, most recently
from
November 20, 2026 12:40
3170cdc to
76efc31
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.