test(ci): 314 of 462 integration tests skip unmonitored — no mechanism detects coverage silently evaporating

[jamesburton]

Follow-up surfaced while reviewing #392 / PR #412.

Problem

The integration suite reports Skipped: 314 of Total: 462 on CI — roughly 68% of integration tests never execute there, and nothing asserts that this is the intended number.

Skips are individually legitimate (gated model fixtures, GPU-only tests, checkpoint-dependent paths). The gap is that no mechanism notices when a test starts skipping that previously ran. A renamed env var, a moved fixture path, a broken File.Exists probe, or a refactor that inverts a guard all present identically to a correctly-gated skip: green CI, quietly reduced coverage.

This is the same failure class the repo already tracks elsewhere:

• Three RealHfSafetensorsEndToEndTests methods use [Fact] instead of [SkippableFact] — SkipException counted as FAIL #392 — SkipException from [Fact] counted as failure (the loud version; fixed in fix(tests): use [SkippableFact] for gated safetensors E2E tests (#392) #412).
• Sweep: ~42 further test classes share the File.Exists-only checkpoint-probe pattern #384 fixed in 12 #393 — ~42 classes share a File.Exists-only checkpoint probe, which Checkpoint-gated test classes hard-FAIL with IOException on partial/locked checkpoint files instead of skipping #384 replaced with CheckpointGuard.LoadOrSkip<T> in only 12.
• SilentSkipConventionTests already exists, but it guards a different thing — silent pass via early-return, not silent skip.

So the repo has caught the loud form and the probe-correctness form, but not "coverage silently evaporated".

Why it matters now

Two model-support programmes (Qwen3.8-27B, Nemotron-3.5-Lightning) are about to lean on the integration suite as the correctness gate for new architectures and kernels. A suite where two thirds of tests skip, unmonitored, cannot serve as that gate — and a newly skipping test would be indistinguishable from the 314 that are meant to.

Possible shapes (not prescriptive — pick one in the PR)

1. Baseline assertion. Commit an expected skip-count (or per-class skip manifest); fail CI when actual skips exceed it. Cheap; needs updating when tests are legitimately added.
2. Skip-reason taxonomy. Require every skip to carry a machine-readable reason (fixture-missing, gpu-required, platform). Report counts per category; alert when a category grows.
3. Environment-declared expectations. CI declares what it has (no GPU, no gated checkpoints) and asserts exactly the matching set skips — anything else skipping is a failure.

Option 3 is the strongest signal and the most work; option 1 is a day and catches most regressions.

Acceptance criteria

• A test starting to skip when it previously ran causes CI to fail, not pass quietly.
• The expected-skip set is explicit and reviewable in-repo, not implied by whatever CI happens to do.
• Legitimately adding a gated test is a small, obvious diff — the mechanism must not be so annoying it gets disabled.
• Documented in the testing docs so the next person adding a gated test knows the contract.

Context

Numbers from PR #412's CI run: Failed: 3, Passed: 145, Skipped: 314, Total: 462. Related: #392, #393, #384.