Description
ROUGE's fallback path documents itself as unigram recall, but computes recall over word types rather than word occurrences. Both sides are deduplicated into sets before the overlap is counted, so a term appearing three times in the reference counts once.
openagent_eval/metrics/generation/rouge.py:68 — "Simple unigram recall fallback"
openagent_eval/metrics/generation/rouge.py:69-80 — set-based deduplication, then overlap
The practical effect is that the score is insensitive to repetition. A candidate recovering one instance of a term repeated many times in the reference scores identically to one recovering all of them, which is not what unigram recall means.
Expected Behavior
Either the fallback computes unigram recall over occurrences (multiset/counter intersection over reference token count), or the docstring states that it measures word-type recall. Which of those is right is a product decision about what this metric is for, so I have not assumed one.
Discovered During
Found while verifying #281 — a sweep for metrics whose documented behaviour differs from their implementation, prompted by the same class of defect in ExactMatch (#225).
Suggested Fix
Unverified: if occurrence-sensitive recall is intended, collections.Counter intersection over the reference's total token count gives it without changing the surrounding structure. If type recall is intended, the one-line docstring change is the whole fix.
Description
ROUGE's fallback path documents itself as unigram recall, but computes recall over word types rather than word occurrences. Both sides are deduplicated into sets before the overlap is counted, so a term appearing three times in the reference counts once.The practical effect is that the score is insensitive to repetition. A candidate recovering one instance of a term repeated many times in the reference scores identically to one recovering all of them, which is not what unigram recall means.
Expected Behavior
Either the fallback computes unigram recall over occurrences (multiset/counter intersection over reference token count), or the docstring states that it measures word-type recall. Which of those is right is a product decision about what this metric is for, so I have not assumed one.
Discovered During
Found while verifying #281 — a sweep for metrics whose documented behaviour differs from their implementation, prompted by the same class of defect in
ExactMatch(#225).Suggested Fix
Unverified: if occurrence-sensitive recall is intended,
collections.Counterintersection over the reference's total token count gives it without changing the surrounding structure. If type recall is intended, the one-line docstring change is the whole fix.