Add StructEval benchmark - #57
Conversation
|
The paper is legitimate — arXiv 2505.20139, TMLR 2025, and the repo is Apache-2.0 and actively pushed. No concerns there. The problem is fit. This inserts into §9 · Agent-specific evaluation, whose scope is trajectories, tool use, multi-turn, world state, multi-agent, and localization. StructEval measures single-shot structured-output generation across 18 formats: no agent, no trajectory, no tool use, no multi-turn. I can't find a theme in that section it matches, and I don't think it fits the other sections either — this list is specifically about evaluating agents, and structured-output generation is a model capability benchmark. If you think I'm drawing that line wrong, make the case — what's the agent-evaluation angle? The PR body describes what StructEval is but not why it belongs here, and CONTRIBUTING asks for that explicitly. Also please add a disclosure: you're an author on the paper, which you noted on #54 but not here or on #64. Leaving this open for now in case you want to argue the fit, but my inclination is that it's out of scope. |
|
Correcting my scope argument — the reason I gave was wrong. I wrote that "this list is specifically about evaluating agents, and structured-output generation is a model capability benchmark." §9 itself already carries Humanity's Last Exam (README:384) and ARC-AGI-2 (README:392), both single-shot with no tools and no trajectory. So the line I drew isn't the line the list actually draws, and it wasn't fair to apply it to you. The real pattern, as far as I can reconstruct it: non-agent benchmarks get carried when they teach a methodological lesson — MMLU-Redux for label error, LiveCodeBench for contamination resistance. Pure capability benchmarks generally don't. So the question isn't "is StructEval about agents" but "does it teach something methodological that this list is missing." If you think it does — cross-format validation, or the visual/structural checking approach — make that case and I'll reconsider on the merits. One more correction: I said you'd disclosed on #54 but not here. What you disclosed on #54 was ClawBench maintainership, which is a different resource — so there was nothing inconsistent about this PR. It would still be good to note your StructEval authorship explicitly. |
Adds StructEval, an open benchmark for structured-output generation and cross-format conversion, to the benchmark resources section.