test(integration): silent no-op passes survive #421 — 74 of 86 'passing' Vulkan integration tests do nothing; CrossBackendQuantGate is 100% inert

[jamesburton]

The #421 fix was incomplete

Issue #421 (upstream kkokosa#421, PR #428) fixed hardware-gated tests that report passed instead of skipped. That fix only touched tests/DotLLM.Tests.Unit/Cpu/Kernels/. The identical anti-pattern survives across the integration suite, where it matters more, because those are the tests that verify real models.

The pattern, inside a [SkippableFact]:

_output.WriteLine("[SKIP] fixture not found");
return;                       // <-- reports PASSED

instead of Skip.If(...).

Surviving instances, measured on this box (gfx1151, Vulkan available):

suite	reported	actually executed	silent no-ops
RealGgufVulkanParityTests	9 passed	3	6
other integration Vulkan + CrossBackend*	77 passed	9	68

So of 86 "passing" integration tests, 74 did nothing. On a fixture-less CI runner the number would be higher still, and the suite would report green.

Roughly lines 69, 88, 111, 218, 247, 273, 291 of tests/DotLLM.Tests.Integration/Vulkan/RealGgufVulkanParityTests.cs, plus CrossBackendQuantGateTests and RealHfSafetensorsEndToEndVulkanTests.

This looks unintentional rather than a decision: the same class already uses Skip.If correctly for its Vulkan/SPIR-V availability gates. Only the fixture gates use the silent form.

The worst case: CrossBackendQuantGateTests is 100% inert

This is the class specifically designed to be the format × backend end-to-end matrix — CPU vs CUDA vs Vulkan perplexity agreement per QuantizationType. 105 cases, zero executed. 42 report skipped, 63 report passed.

No fixture exists for any of them. Worse, .docs/corpora/ — the directory its own skip message names as the place documenting how to generate them — does not exist in the repo.

So the intended safety net for cross-backend correctness is entirely notional, and reports green.

Acceptance criteria

• Every fixture-gated integration test uses Skip.If/Skip.IfNot, so absent fixtures report skipped.
• A run with no fixtures present reports 0 passed for these suites, not 63.
• CrossBackendQuantGateTests' referenced fixture-generation recipe either exists at the path its skip message names, or the message points somewhere real.
• Ideally a lint/analyzer or a test-of-tests that fails on return; following a [SKIP] log line inside a [SkippableFact], since this has now recurred once after being fixed.

References

• upstream test(unit): ~26 hardware-gated tests report as passed instead of skipped kkokosa/dotLLM#421 / test(quant): IQ4_NL has no llama.cpp-transcription anchor — codebook interior is unpinned, and Nemotron GGUFs are ~91% IQ4_NL #428 — the original fix, unit-tests only
• tests/DotLLM.Tests.Integration/Vulkan/RealGgufVulkanParityTests.cs
• tests/DotLLM.Tests.Integration/CrossBackendQuantGateTests.cs

[jamesburton]

Fixed and merged — commits e320739 (`test: convert silent no-op skips to Skip.If and add shared fixture resolver (#307, #308)`) and 8bb9d9c (`test: catch two unlogged silent skips and document the detector's known gap (#307)`), landed via PR merge b0b4630. Already on `dev`.

Verified independently:

• `CrossBackendQuantGateTests.cs` now uses `Skip.If`/`Skip.IfNot` throughout — no silent `[SKIP]`-log-then-`return` sites remain.
• `docs/QUANT_FIXTURES.md` (tracked) replaces the unreachable `.docs/corpora/QUANT_FIXTURES.md` the old skip message pointed at.
• `SilentSkipConventionTests` (the detector-of-detectors) exists and is documented as catching a planted violation.
• Ran the full `Backend_AgreesWithCpu` matrix for real (42 cases, CUDA + Vulkan × 21 quant types): 36 genuinely executed and passed, 6 legitimately skipped (BF16/F16/Q8_0 — missing quant-ladder fixtures, correctly reported as skipped rather than a fake pass). Zero silent no-ops. This is the first time this gate has produced a real number.

The residual gap (bare if (!X.IsSupported) return; with no [SKIP] marker, ~46 sites) was correctly scoped out to #319 rather than bundled here. Closing.