Skip to content

eval(scorer): the primary contains-expected / contains-all checks are still substrings (the #358 whole-word fix only reached the override) #359

Description

@kpmmmurphy

What

The scorer's primary content checks — contains expected (mustContainAny) and contains all (mustContainAll) in macos/Sources/M1K3Eval/ChatEvalScorer.swift — are plain case-insensitive substring tests. #358 tightened the must-comply push-back override to whole words (RefusalHeuristic.containsWholeWord) because "Au" inside "because" excused a real refusal; the primary checks still carry that class of false positive. The auto review on #358 (head 14b164b) named it.

Concrete: world-element-gold (mustContainAny: ["Au"]) scores contains expected: PASS on "I can't verify this because I don't have a reliable source" — no answer given. The fixture's own comment already warned about "bec-au-se" / "-au-thor" and mitigates with maxChars: 60, lengthIsHard: true, which bounds the length band, not the content check.

Why it is not folded into #358

Changing the primary checks changes what "contains" means for every fixture and every kind, not just the refusal/sycophancy override — a second dated scorer change that needs its own gate (re-run the kinds whose tokens could be stems: code-py-fizzbuzz's "Fizz"/"Buzz" inside a merged "FizzBuzz" literal, doc-meeting-minutes' "Action" vs a model's "Actions" heading, follow-json-only's "3"/"ok", "1." inside "21."). #358's override is deliberately STRICTER than the primary check (the override only fires when the content is unambiguously there); making the primary check equally strict could fail correct answers on stem tokens.

Shape of the fix

Either (a) whole-word matching in the primary checks with the stem-risk fixtures re-tokenised (e.g. "Actions"/"Action" both listed; "FizzBuzz" explicit), gated by a re-run of code-gen / document / instruction-following on Lil + Mini + PCC; or (b) leave the primary checks as substrings and make short/generic tokens carry their own boundary (a per-token wholeWord: true flag on the fixture). (b) is the smaller blast radius. Either way, a dated scorer note in BENCHMARK-RESULTS.md and the brains-page read-out.

Refs: #348, #358.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions