Skip to content

Add PerspectiveGap community eval tasks - #2

Open
WhymustIhaveaname wants to merge 5 commits into
zai-org:mainfrom
WhymustIhaveaname:feature/perspectivegap-glm-adapter
Open

WhymustIhaveaname wants to merge 5 commits into
zai-org:mainfrom
WhymustIhaveaname:feature/perspectivegap-glm-adapter

Conversation

@WhymustIhaveaname

@WhymustIhaveaname WhymustIhaveaname commented Jul 7, 2026

Copy link
Copy Markdown

Summary

Adds PerspectiveGap as two community reference tasks:

  • role_assignment
  • prompt_writing

The tasks load either a local JSONL file at data/perspective_gap/evaluations.jsonl or the Hugging Face sun1245/PerspectiveGap test split, call the configured sampler with the task prompt, and score with the official deterministic perspective_gap.scoring functions.

Notes

  • Registers both tasks in evaluate.py.
  • Pins the PerspectiveGap scorer dependency to commit 9c6921b3337ff3e6a6a453f68d117a8c1663135e.
  • Fails fast when the scorer dependency is missing, before sampler calls start.
  • Keeps sampler and runtime scorer errors as per-sample failures.
  • Adds a root test_perspectivegap_smoke.py so default pytest discovery includes the smoke tests.
  • Updates both README.md and README_zh.md.

Validation

uv run --no-project --python 3.13 --with pre-commit pre-commit run --files README.md README_zh.md evals/perspectivegap_eval.py evaluate.py requirements.txt test_perspectivegap_smoke.py
PYTHONPATH=. uv run --no-project --python 3.13 --with pytest pytest -p no:cacheprovider
git diff --check

Result: 7 smoke tests passed. Pre-commit and whitespace checks passed.

Caveat

This is a community reference contribution. The maintainer noted that this repository is not actively maintained, so review may be slow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant