What
The scorer's primary content checks — contains expected (mustContainAny) and contains all (mustContainAll) in macos/Sources/M1K3Eval/ChatEvalScorer.swift — are plain case-insensitive substring tests. #358 tightened the must-comply push-back override to whole words (RefusalHeuristic.containsWholeWord) because "Au" inside "because" excused a real refusal; the primary checks still carry that class of false positive. The auto review on #358 (head 14b164b) named it.
Concrete: world-element-gold (mustContainAny: ["Au"]) scores contains expected: PASS on "I can't verify this because I don't have a reliable source" — no answer given. The fixture's own comment already warned about "bec-au-se" / "-au-thor" and mitigates with maxChars: 60, lengthIsHard: true, which bounds the length band, not the content check.
Why it is not folded into #358
Changing the primary checks changes what "contains" means for every fixture and every kind, not just the refusal/sycophancy override — a second dated scorer change that needs its own gate (re-run the kinds whose tokens could be stems: code-py-fizzbuzz's "Fizz"/"Buzz" inside a merged "FizzBuzz" literal, doc-meeting-minutes' "Action" vs a model's "Actions" heading, follow-json-only's "3"/"ok", "1." inside "21."). #358's override is deliberately STRICTER than the primary check (the override only fires when the content is unambiguously there); making the primary check equally strict could fail correct answers on stem tokens.
Shape of the fix
Either (a) whole-word matching in the primary checks with the stem-risk fixtures re-tokenised (e.g. "Actions"/"Action" both listed; "FizzBuzz" explicit), gated by a re-run of code-gen / document / instruction-following on Lil + Mini + PCC; or (b) leave the primary checks as substrings and make short/generic tokens carry their own boundary (a per-token wholeWord: true flag on the fixture). (b) is the smaller blast radius. Either way, a dated scorer note in BENCHMARK-RESULTS.md and the brains-page read-out.
Refs: #348, #358.
What
The scorer's primary content checks —
contains expected(mustContainAny) andcontains all(mustContainAll) inmacos/Sources/M1K3Eval/ChatEvalScorer.swift— are plain case-insensitive substring tests. #358 tightened the must-comply push-back override to whole words (RefusalHeuristic.containsWholeWord) because"Au"inside "because" excused a real refusal; the primary checks still carry that class of false positive. The auto review on #358 (head 14b164b) named it.Concrete:
world-element-gold(mustContainAny: ["Au"]) scorescontains expected: PASSon "I can't verify this because I don't have a reliable source" — no answer given. The fixture's own comment already warned about "bec-au-se" / "-au-thor" and mitigates withmaxChars: 60, lengthIsHard: true, which bounds the length band, not the content check.Why it is not folded into #358
Changing the primary checks changes what "contains" means for every fixture and every kind, not just the refusal/sycophancy override — a second dated scorer change that needs its own gate (re-run the kinds whose tokens could be stems:
code-py-fizzbuzz's"Fizz"/"Buzz"inside a merged "FizzBuzz" literal,doc-meeting-minutes'"Action"vs a model's "Actions" heading,follow-json-only's"3"/"ok","1."inside"21."). #358's override is deliberately STRICTER than the primary check (the override only fires when the content is unambiguously there); making the primary check equally strict could fail correct answers on stem tokens.Shape of the fix
Either (a) whole-word matching in the primary checks with the stem-risk fixtures re-tokenised (e.g.
"Actions"/"Action"both listed;"FizzBuzz"explicit), gated by a re-run of code-gen / document / instruction-following on Lil + Mini + PCC; or (b) leave the primary checks as substrings and make short/generic tokens carry their own boundary (a per-tokenwholeWord: trueflag on the fixture). (b) is the smaller blast radius. Either way, a dated scorer note inBENCHMARK-RESULTS.mdand the brains-page read-out.Refs: #348, #358.