docs(process): require tests to be DEMONSTRATED to fail — discrimination is asserted but not proven

[jamesburton]

Problem

CLAUDE.md:252 already requires regression tests that discriminate between broken and fixed forms. It states the rule but not how to prove it was met — so in practice discrimination is asserted rather than demonstrated, and the assertion is often wrong.

Three instances from a single day:

1. PR fix(tokenizers): add qwen2 + qwen35 pre-tokenizer pipelines; complete the #373/#387 pre-tokenizer work #410 (pre-tokenizers). The flagship real-GGUF parity test was proven non-discriminating: loading both GGUFs and forcibly swapping the pre-tokenizer pipeline produced identical token ids, so the test passes even when qwen35 is misrouted to qwen2 — the exact regression it exists to catch. Root cause was measurable: neither Qwen vocab contains a single ASCII multi-digit token, so \p{N} vs \p{N}{1,3} is unobservable. It passed CI, the author's own testing, and an advisor review. Only adversarial substitution found it.
2. PR feat(tokenizers): Jinja loop.previtem / loop.nextitem #411 (Jinja loop vars). A reviewer applied 7 mutants and re-ran the suite. Six died; one survived (is undefined returning true for a legitimately null value) — behaviour that was correct in the code but guarded by nothing.
3. Nemotron-H applies RoPE that neither llama.cpp nor HF applies — attention wrong on CPU and Vulkan #372 (NemotronH RoPE). dotLLM applied RoPE where neither llama.cpp nor HF does. It survived for months because the test only asserted logits were finite and non-constant.

#344 already mandates this for quantisation ("every type ships a negative control — the test must be shown to fail against a deliberately broken implementation"). That standard is right and should not be confined to one subsystem.

Proposal

Extend CLAUDE.md's testing rule so that for any test guarding a correctness invariant, the author demonstrates the test can fail — and records how.

Two accepted methods, both used successfully this week:

• Mutation. Apply a deliberate wrong implementation, re-run, confirm the test fails. Best for logic.
• Substitution. Swap the component for a plausible wrong alternative on real data and confirm the output actually differs. Best for data-dependent paths — this is what caught fix(tokenizers): add qwen2 + qwen35 pre-tokenizer pipelines; complete the #373/#387 pre-tokenizer work #410, where the fixture data, not the assertion, was the weak link.

Record the result in the PR body: which mutants were applied and which died. A surviving mutant is a finding, not a failure — #411's surviving mutant produced a one-line test that now guards real behaviour.

Deliberately NOT proposed

• No coverage-percentage target. This is about assertion sensitivity, not line coverage.
• No mutation-testing framework or CI gate. Manual, targeted mutants are what worked; automated whole-suite mutation would be slow, noisy, and would not have caught fix(tokenizers): add qwen2 + qwen35 pre-tokenizer pipelines; complete the #373/#387 pre-tokenizer work #410 at all (the defect was in fixture data, not code).
• Not for smoke tests. A smoke test is allowed to be a smoke test. This applies to tests whose name or issue claims a correctness guarantee.

Acceptance criteria

• CLAUDE.md testing rule extended with the demonstrate-it-can-fail requirement and the two methods.
• States explicitly that data-dependent tests need substitution, because a strong assertion over weak fixture data still cannot fail (fix(tokenizers): add qwen2 + qwen35 pre-tokenizer pipelines; complete the #373/#387 pre-tokenizer work #410's exact shape).
• Cross-references Vulkan expands 5 quantization types to F32 (5.3x-7.5x) for want of kernels — umbrella #344's existing negative-control standard so the two do not drift apart.
• Scoped: correctness gates only, not smoke tests.

Context

An audit of the existing suite's discriminating power is underway; this issue is the standing rule, independent of what that audit finds. Related: #344, #410, #411, #372, #261.