load_problems skips anything whose problem_type is not diff, because
exact-match scoring cannot grade a problem with many valid outputs. In a sample
of 25 problems, 5 were checker — so this is roughly 20% of the corpus excluded
from eval, and the excluded slice is not random.
Codeforces ships a checker program per problem. open-r1/codeforces-cots has a
checker_interactor config that appears to carry them.
What this needs:
- Load the checker source alongside the problem
- Run it as
checker(input, expected_output, actual_output) -> accept/reject
in the same sandbox the solutions run in
- A
Verdict path for "checker itself crashed", which is a harness failure and
must not be silently counted as a wrong answer
Until this lands, every published pass@1 is measured on the exact-match subset
only, and the README should keep saying so.
load_problemsskips anything whoseproblem_typeis notdiff, becauseexact-match scoring cannot grade a problem with many valid outputs. In a sample
of 25 problems, 5 were
checker— so this is roughly 20% of the corpus excludedfrom eval, and the excluded slice is not random.
Codeforces ships a checker program per problem.
open-r1/codeforces-cotshas achecker_interactorconfig that appears to carry them.What this needs:
checker(input, expected_output, actual_output) -> accept/rejectin the same sandbox the solutions run in
Verdictpath for "checker itself crashed", which is a harness failure andmust not be silently counted as a wrong answer
Until this lands, every published pass@1 is measured on the exact-match subset
only, and the README should keep saying so.