Skip to content

A missing pytest-cov report reads as a green 0.0% Python coverage #107

Description

@aclark-aquaveo

Raised in review of #105 and deliberately left out of that PR, which closed one producer of a missing Python coverage report while leaving the detector blind.

A missing Python report is not just silent — it is green

  • coverage_tools/coverage_generator.py:581py_raw = _py_percent_from_summary(py_summary) if py_summary.exists() else 0.0. A missing report is indistinguishable from a genuine 0%.
  • generator_tools/ci_file_generator.py:140python_threshold defaults to 0. So 0.0 >= 0.0py_pass = True → exit 0, printing Python: 0.0% (threshold 0.0%) -> PASS for any library that has not set a threshold.
  • coverage_generator.py:559-576 — three if X is not None: copies with no else, no warning, no log. Nothing distinguishes "copied the report" from "found nothing to copy". _find_pytest_cov_artifact's docstring justifies None as legitimate when pybind=False, but at this call site pybind is True by construction (:520-527), so None can only mean something broke.
  • ci_templates/github-coverage.yaml.jinja:83,93 sets if-no-files-found: ignore (the default is warn), removing the last place an operator would see a red flag. gitlab-ci.yml.jinja:379-389 drops the Python link with no echo, and docs/USAGE.md:591 documents that as intended.

The C++ half does not accept this asymmetry: _assert_gcovr_collected_data (:392) raises a three-cause diagnostic on zero instrumented lines. The Python half has no equivalent — and _py_percent_from_summary (:459) raises on schema drift with a docstring saying it does so "rather than collapsing to an indistinguishable 0%", while the file being absent entirely collapses to exactly that.

#105 closed one route in (the Linux Debug pybind build skipped its wheel, so no Python tests ran). Every other route still lands green: XMS_COVERAGE not reaching the recipe, a cache hit with no build folder, a pytest-cov output path change.

Suggested scope

  1. Raise on an absent cov-py-summary.json when the pybind coverage build was expected to produce one, with a diagnostic naming the candidate causes — mirroring _assert_gcovr_collected_data.
  2. Log explicitly in the three if X is not None: arms when there was nothing to copy.
  3. Reconsider the 0 threshold defaults, or separate "no threshold set" from "0% is acceptable" so a missing report cannot report PASS.
  4. Revisit if-no-files-found: ignore in the GitHub coverage template.

Two smaller adjacent items

  • xms_conan2_file.py:526-533 runs _save_test_artifacts() in a bare finally, so an unguarded rmtree/copytree failure there replaces a propagating test failure. Give the Debug pybind module the _d name its consumers link #105 makes run_python_tests() reachable on the coverage path, where CI sets XMS_TEST_ARTIFACTS_DIR.
  • xms_conan2_file.py:350-356 is an except ConanException: pass on python_version propagation, so a typo'd override is discarded silently and later surfaces as No pybind=True... package found from _find_coverage_package, pointing at the wrong component.

No repo sets [ci].coverage = true today, so nothing is failing right now — the same "luck, not safety" that #105's second change was about.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions