Skip to content

test(unit): ~26 hardware-gated tests report as passed instead of skipped #421

Description

@jamesburton

Observation

Found while addressing review feedback on #315, where one test used this shape:

if (!Avx2.IsSupported) return;   // reports as PASSED on hardware without AVX2

A test that returns early without asserting is reported as passed, not skipped. On a CI agent or
developer machine lacking the required ISA, the suite goes green while the coverage silently
evaporates — which is the failure mode a test suite exists to prevent.

grep puts roughly 26 further occurrences across tests/DotLLM.Tests.Unit, gated on things like
Avx2.IsSupported, Avx512F.IsSupported, CUDA availability and Vulkan device presence.

Why it matters here specifically

This repo's correctness story leans on parity tests — SIMD kernels verified against a scalar
reference, backends verified against CPU. Those are exactly the tests that carry hardware gates, so
the pattern concentrates in the highest-value tests rather than the incidental ones.

It also interacts badly with heterogeneous hardware: a contributor on a machine without AVX-512 sees
green, pushes, and only discovers the gap if someone else happens to run it on a machine that has it.

The convention already exists

tests/DotLLM.Tests.Unit already references Xunit.SkippableFact and uses it in the CUDA tests:

[SkippableFact]
public void Something()
{
    Skip.IfNot(Avx2.IsSupported, "Requires AVX2.");
    ...
}

So this is applying an established convention consistently, not introducing one.

Proposed change

  1. Convert hardware-gated early returns to [SkippableFact]/[SkippableTheory] + Skip.IfNot.
  2. Where a test runs BOTH a scalar and a vector path, gate only the vector half — skipping the whole
    test would discard the scalar coverage, which does run everywhere. (This distinction was applied
    in kernels(cpu)(matmul): F32 outer-product tiled GEMM kernel for prefill #315 and is the reason that PR did not convert two of its tests.)
  3. Leave a short comment at any site where the early return is genuinely intentional.

Acceptance criteria

  • No test in tests/DotLLM.Tests.Unit returns early on a hardware capability check without
    either skipping or retaining an assertion that runs everywhere.
  • Skipped counts appear in the run summary on hardware lacking each capability, so the gap is
    visible rather than silent.
  • Tests exercising both scalar and vector paths still assert the scalar path when the vector
    path is unavailable.
  • No change to which assertions run on fully-capable hardware — the pass count there is
    unchanged.

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions