Skip to content

Fix: scorers crash on empty input (UnboundLocalError / ZeroDivisionError) - #52

Open
jimdawdy-hub wants to merge 1 commit into
HazyResearch:mainfrom
jimdawdy-hub:fix/empty-input-guards
Open

Fix: scorers crash on empty input (UnboundLocalError / ZeroDivisionError)#52
jimdawdy-hub wants to merge 1 commit into
HazyResearch:mainfrom
jimdawdy-hub:fix/empty-input-guards

Conversation

@jimdawdy-hub

Copy link
Copy Markdown

Problem

Two scorers raise on an empty generations list instead of returning a score:

  • evaluate_successor_liability binds f1 only inside the per-sample loop → UnboundLocalError when generations == [].
  • evaluate_definition_extraction ends in correct / totalZeroDivisionError when total == 0.

An empty list is a normal degenerate case (a task whose generations all failed to parse, a zero-length slice, etc.).

Fix

Return 0.0 in both when there are no generations.

Two scorers raise on an empty `generations` list:

- `evaluate_successor_liability` binds `f1` only inside the per-sample loop, so
  it raises `UnboundLocalError` when there are no generations.
- `evaluate_definition_extraction` ends in `correct / total`, raising
  `ZeroDivisionError` when `total == 0`.

An empty list is a normal degenerate case (e.g. a task whose generations all
failed to produce, or a zero-length eval slice). Return 0.0 in both instead of
throwing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jimdawdy-hub

Copy link
Copy Markdown
Author

Reproduction (captured output)

# repro.py
from evaluation import evaluate_successor_liability, evaluate_definition_extraction
for name, fn in [("evaluate_successor_liability", evaluate_successor_liability),
                 ("evaluate_definition_extraction", evaluate_definition_extraction)]:
    try:
        print(f"{name}([], []) -> {fn([], [])}")
    except Exception as e:
        print(f"{name}([], []) -> {type(e).__name__}: {e}")

BEFORE (upstream main):

evaluate_successor_liability([], []) -> UnboundLocalError: cannot access local variable 'f1' where it is not associated with a value
evaluate_definition_extraction([], []) -> ZeroDivisionError: division by zero

AFTER (this branch):

evaluate_successor_liability([], []) -> 0.0
evaluate_definition_extraction([], []) -> 0.0

@jimdawdy-hub

Copy link
Copy Markdown
Author

@neelguha Friendly ping on this one as well — another small scorer hardening fix.

What it fixes: Two scorers crash on empty generations instead of returning a score:

  • evaluate_successor_liabilityUnboundLocalError (F1 only bound inside the loop)
  • evaluate_definition_extractionZeroDivisionError when total == 0

Scope: One commit; returns 0.0 for the empty-input degenerate case.

Relation to #50 / #51: Independent. Useful on its own for robust eval pipelines (failed parses, empty slices, etc.).

Let me know if you’d prefer these three successor_liability-related PRs squashed — happy to do that. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant