Unify coverage tooling: ship xmsconan_coverage + GitHub support + Python coverage

[gagelarsen]

Background

xmsconan already has GitLab coverage support gated by [ci] coverage = true in build.toml, but it has several gaps. We just built a richer Phase 1 coverage workflow in xmsgrid that addresses them, and the natural next step is to lift it into xmsconan so every library benefits.

Reference implementation: xmsgrid PR #196 — see dev/coverage.sh, .github/workflows/Coverage.yaml, and the coverage block in build.toml's extra_cmake_text.

Revised after review feedback. The original plan ported the xmsgrid two-build shape (one Debug+testing build, one Release+pybind build) into xmsconan_coverage and deferred cross-credit to a Phase 2. wdolinar's feedback was correct that a single instrumented testing=True, pybind=True build is the cleaner long-term shape — it halves cold runtime, feeds both ctest and pytest from one gcda set, and gets Python-tests-cover-C++ credit for free. This issue now plans the one-build shape from day one (which collapses Phase 2 into Phase 1). Three xmsvtk-specific asks (xvfb, image selection, no sharding) are also folded in.

What xmsconan does today

• [ci] coverage = true adds a Coverage: stage to .gitlab-ci.yml.
• Stage uses an in-repo conan_profiles/linux_testing_debug_coverage.txt and a CMake coverage preset (both expected to be hand-committed in each consuming repo — xmsconan does not generate them).
• Runs ./build/<lib>-coverage/Debug/runner directly, then one gcovr call producing coverage.xml (cobertura) + coverage.html (single-page).
• GitLab pages stage publishes the HTML.

Gaps to close

• C++ only. No Python coverage is collected anywhere.
• No threshold gating. The job emits a coverage badge but never fails on regression.
• Single-page HTML. Uses --html rather than --html-details, so reviewers can't drill into per-file misses.
• GitHub workflow not generated. ci_coverage only affects the GitLab template.
• Per-repo profile/preset files. Consumers must hand-create conan_profiles/linux_testing_debug_coverage.txt and a CMake preset, neither of which xmsconan supplies. Adoption friction.

Proposed plan

Replace the existing GitLab implementation in the same release; introduce a generic command that both CI templates and developers call.

1. Conanfile / CMake refactor: one build, both test paths

To make the rest of the design clean, the generated conanfile and CMakeLists need to allow testing=True and pybind=True in the same configuration. Today they're mutually exclusive.

Changes to xmsconan templates:

• CMakeLists.txt.jinja — drop if (IS_PYTHON_BUILD AND BUILD_TESTING) message(FATAL_ERROR …). Both the test runner target and the pybind module target should configure and build together.
• xms_conan2_file.py — remove the configure_options() clause that deletes pybind when build_type != Release. Allow pybind=True with any build type. Restructure build() from if testing: … elif pybind: … into two independent if blocks so both test paths run when both options are true.
• xms_conan2_file.py (cont.) — extend run_python_tests to install pytest-cov and pass --cov=<python_namespaced_dir> / --cov-report=xml:cov-py.xml / --cov-report=html:build/coverage-html-py / --cov-report=json:build/cov-py-summary.json when XMS_COVERAGE is set in the environment. Non-coverage runs are unaffected. Honors XMS_COVERAGE_PIP_INDEX for the pip extra-index URL so consumers behind alternative devpi configurations can override.
• Existing non-coverage build matrix entries continue to set only one of the options at a time, so this is purely additive — no behavior change for current consumers' regular CI.

Consequence: a single instrumented build with testing=True, pybind=True, build_type=Debug produces both the runner and the pybind module, and ctest + pytest both write into the same gcda set. Python tests automatically credit the underlying C++ in the C++ coverage report — no separate "Phase 2 cross-credit" work needed.

2. New console entry point: xmsconan_coverage

Ship a Python implementation alongside xmsconan_gen and xmsconan_wheel_repair. CI templates and local devs invoke the same command:

xmsconan_coverage --build-toml build.toml

Behavior (single instrumented build, both test paths):

• Run xmsconan_gen to render build artifacts.
• If [ci] xvfb = true, re-exec under xvfb-run -a -s "-screen 0 1280x1024x24" as the very first action so both the C++ runner and pybind GL contexts have a display. Same command works locally in the Linux container and in CI.
• Run one instrumented build:

  XMS_COVERAGE=1 python build.py \
      --filter='{"build_type":"Debug","testing":true,"pybind":true}'
  

  This produces both the test runner and the pybind module from a single compile. ctest runs (via the conanfile's existing run_cxx_tests) and pytest runs (via the conanfile's existing run_python_tests) write gcda into the same Conan build folder. Sharding options (split_tests / test_shards) are explicitly ignored: coverage is a single measurement per commit, and gcda merging across shards adds complexity for a job that's not on the critical path.
• Walk the Conan cache to locate the build folder and matching source folder containing the library subtree.
• gcovr --root <source> --html-details build/coverage-html-cpp/index.html --xml cov-cpp.xml --json-summary build/cov-cpp-summary.json <build_folder>.
• Python coverage is collected by the conanfile's own run_python_tests (see section 1.a below). The pytest invocation inside that method picks up pytest-cov and --cov=<library_namespace> / --cov-report=… flags when XMS_COVERAGE is set, writing cov-py.xml / build/cov-py-summary.json / build/coverage-html-py/ into the build folder. xmsconan_coverage copies the JSON summary up to the workspace root so the threshold-check step can read it.
• Round actuals to one decimal, compare to per-layer thresholds, exit non-zero if either fails.
• Append a markdown summary table to $GITHUB_STEP_SUMMARY when present (threshold + actual + status per layer).

3. CMake instrumentation moves into the template

Add an env-var-gated coverage block to CMakeLists.txt.jinja, rendered when ci.coverage = true:

if (DEFINED ENV{XMS_COVERAGE} AND NOT "$ENV{XMS_COVERAGE}" STREQUAL "")
    if (MSVC)
        message(FATAL_ERROR "XMS_COVERAGE is GCC/Clang only")
    endif ()
    add_compile_options(--coverage -O0 -g)
    add_link_options(--coverage)
endif ()

Use a generic env var name (XMS_COVERAGE) rather than <LIB>_COVERAGE since the same env reads for any library. Frees up extra_cmake_text for repos that previously had to land it themselves.

4. build.toml schema additions

[ci]
coverage = true             # (existing flag, behavior generalized)
xvfb = true                 # (existing flag, now also honored by xmsconan_coverage)

[coverage]
cpp_threshold = 70          # optional, default 0 (no gate)
python_threshold = 70       # optional, default 0
filters = ["<library>/"]    # default: library_name + "/"
excludes = [                # sensible defaults baked in if omitted
    ".*\\.t\\.h$",
    ".*/<library>/python/.*",
    ".*/_package/tests/.*",
]

Most repos opt in with one line (coverage = true).

5. GitHub workflow generation

When ci.coverage = true and ci_type = "github", render a Coverage.yaml next to the existing <Library>-CI.yaml. Job:

• Linux container, image selected the same way as <Library>-CI.yaml: conan-gcc13-x11-gdal-py3.13 when [ci] xvfb = true, otherwise conan-gcc13-py3.13. Honors docker_image override.
• Installs xmsconan + gcovr; runs xmsconan_conan_setup.
• Calls xmsconan_coverage.
• Uploads coverage-html (HTML reports) and coverage-xml (Cobertura XMLs) as artifacts with if: always().
• Threshold env vars read from build.toml or set explicitly in the workflow.

6. GitLab template replacement

Replace the existing Coverage: stage body with a single call to xmsconan_coverage. Image selection follows the same xvfb-aware rule. Keep artifacts in the same shape (cobertura at cov-cpp.xml/cov-py.xml, HTML directories) so the existing pages: stage (and any external dashboards keying off the cobertura) continues to work. Note the change in release notes since any library currently relying on the hand-rolled CMake preset will be affected.

Migration for consumers

After the xmsconan release lands, an existing repo migrates by:

1. Bumping xmsconan dep version in build.toml.
2. Adding [ci] coverage = true and [coverage] cpp_threshold = N / python_threshold = N.
3. Deleting any local dev/coverage.sh, hand-written Coverage.yaml, coverage block in extra_cmake_text, or *_coverage.txt conan profiles and coverage* CMake presets that pre-dated this work.
4. Re-running xmsconan_gen.

For xmsgrid: ~200 lines of repo-local plumbing collapse to ~6 lines of build.toml. For xmsvtk: drop two *_coverage.txt profiles, drop the Coverage: stage from .gitlab-ci.yml (regen handles it), drop the coverage* block from CMakePresets.json, add [coverage] cpp_threshold = 0, python_threshold = 0, regen. Ratchet thresholds once a green baseline lands.

Risks / open questions

• Wheel optimization level. Today the user-facing pybind wheel comes from a Release build. Allowing pybind=True in non-Release builds is additive — the Release wheel build path stays intact for normal CI. But worth a quick check that no downstream tooling assumes pybind is always Release.
• Existing GitLab consumers. Anyone using the current coverage = true will see their hand-rolled CMake preset / conan profile path become unused. Worth a clear note in release notes.

[wdolinar]

xmsvtk perspective on this proposal

xmsvtk is one of the heavier consumers of coverage = true — two hand-committed *_coverage.txt profiles, a coverage/coverage-debug/coverage-release block in CMakePresets.json, and a bespoke Coverage: stage in .gitlab-ci.yml. Lifting all of that into xmsconan_coverage plus a [coverage] block in build.toml is a clear win. Strong +1 on the direction.

Three xmsvtk-specific things the design should handle, all derivable from existing build.toml:

1. Xvfb. xmsvtk has [ci] xvfb = true because every test opens a real GL context (C++ runner and the pybind GraphicsView/viewer tests both need DISPLAY). Cleanest place to handle it is inside xmsconan_coverage — when [ci] xvfb is set, re-exec under xvfb-run -a -s "-screen 0 1280x1024x24" as the first action. Same command works for local devs in the Linux container and in CI, no per-step wrapping inside the pytest venv.

2. Image selection in generated Coverage.yaml. When [ci] xvfb = true, the workflow needs the X11/GDAL image (xmsvtk's regular CI uses conan-gcc13-x11-gdal-py3.13), not the plain conan-gcc13-py3.13 the xmsgrid PR uses. Same picker xmsconan already does for <Library>-CI.yaml.

3. Sharding is off for coverage. xmsvtk has [ci] split_tests = true, test_shards = 4 for regular CI; xmsconan_coverage should ignore those and execute the suite once. Coverage is a single measurement per commit; sharding adds .gcda merge complexity for wall-clock savings on a job that's already off the critical path.

One structural point on the xmsgrid reference itself: dev/coverage.sh runs the build twice per commit — one instrumented build_type=Debug, testing=True invocation for C++ gcda, then a separate non-instrumented build_type=Release, pybind=True invocation to produce the wheel that pytest installs. That's two full Conan dependency resolutions and two dependency link cycles per commit, doubling cold-job runtime to satisfy two artifact shapes that no measurement actually requires to be separate. A single instrumented build with testing=True, pybind=True can produce both the test runner and the pybind module from one compile, feeding ctest for C++ and pytest for Python from the same gcda set. That collapses the job to one run per commit, halves cold runtime, and as a free side-benefit sets up the Phase 2 cross-credit path (Python tests covering C++) without any further xmsconan changes. Worth designing xmsconan_coverage around the single-build shape from day one rather than porting the two-build pattern over.

Migration cost for xmsvtk after the xmsconan release: delete the two *_coverage.txt profiles, drop the Coverage: stage from .gitlab-ci.yml (generated — falls out on regen), add [coverage] cpp_threshold = 0, python_threshold = 0, regen. Ratchet thresholds in a follow-up after one green baseline run.

Happy to drive the xmsvtk-side migration once xmsconan ships.

[gagelarsen]

@wdolinar thanks — all four points incorporated. Issue body updated with the revised plan.

Quick recap of the changes:

• One-build shape from day one. New section 1 calls out the conanfile/CMake refactor (drop the BUILD_TESTING + IS_PYTHON_BUILD FATAL_ERROR, drop the configure_options() deletion of pybind for non-Release, restructure build()'s if/elif into independent ifs). Single instrumented testing=True, pybind=True build feeds both ctest and pytest from the same gcda set. Phase 2 cross-credit collapses into Phase 1 as you noted.
• Xvfb. xmsconan_coverage re-execs under xvfb-run -a -s "-screen 0 1280x1024x24" as its first action when [ci] xvfb = true. Local + CI both go through the same wrapper.
• Image selection. Generated Coverage.yaml (and the updated GitLab Coverage: stage) pick conan-gcc13-x11-gdal-py3.13 when xvfb is on, falling back to conan-gcc13-py3.13 otherwise — same rule the existing <Library>-CI.yaml generation uses, honoring docker_image override.
• No sharding. xmsconan_coverage explicitly ignores [ci] split_tests / test_shards. One run per commit.

One open question I flagged in the Risks section that affects xmsvtk in particular: where pytest-cov gets wired in. Two options — extend the conanfile's run_python_tests to install pytest-cov and pass --cov flags when an env signal is set, or have xmsconan_coverage run pytest a second time in its own venv against the just-installed wheel (matches the xmsgrid reference). The first is cleaner (truly one build, one test run); the second is more isolated and avoids touching the production test runner. I'd lean (a) — opinions welcome before the implementation PR picks one.

Happy to have you drive the xmsvtk migration once xmsconan ships.

[gagelarsen]

Decision: go with option (a) — wire pytest-cov into the conanfile's run_python_tests directly, gated on XMS_COVERAGE. Issue body updated: section 1 now spells out the run_python_tests extension (install pytest-cov, pass --cov flags including xml/html/json reports, honor XMS_COVERAGE_PIP_INDEX for the extra-index override), and the open question is removed from the Risks section. Plan is now fully spec'd and ready for implementation once the team signs off.

[gagelarsen]

Closed by #58, merged as 3ce6274. Coverage badge, gated thresholds, GitHub workflow + GitLab pages landing page all live on master.