Skip to content

Add StructEval benchmark - #57

Open
reacher-z wants to merge 1 commit into
benchflow-ai:mainfrom
reacher-z:add-structeval-benchmark
Open

Add StructEval benchmark#57
reacher-z wants to merge 1 commit into
benchflow-ai:mainfrom
reacher-z:add-structeval-benchmark

Conversation

@reacher-z

Copy link
Copy Markdown

Adds StructEval, an open benchmark for structured-output generation and cross-format conversion, to the benchmark resources section.

@xdotli

xdotli commented Jul 30, 2026

Copy link
Copy Markdown
Member

The paper is legitimate — arXiv 2505.20139, TMLR 2025, and the repo is Apache-2.0 and actively pushed. No concerns there.

The problem is fit. This inserts into §9 · Agent-specific evaluation, whose scope is trajectories, tool use, multi-turn, world state, multi-agent, and localization. StructEval measures single-shot structured-output generation across 18 formats: no agent, no trajectory, no tool use, no multi-turn. I can't find a theme in that section it matches, and I don't think it fits the other sections either — this list is specifically about evaluating agents, and structured-output generation is a model capability benchmark.

If you think I'm drawing that line wrong, make the case — what's the agent-evaluation angle? The PR body describes what StructEval is but not why it belongs here, and CONTRIBUTING asks for that explicitly.

Also please add a disclosure: you're an author on the paper, which you noted on #54 but not here or on #64.

Leaving this open for now in case you want to argue the fit, but my inclination is that it's out of scope.

@xdotli

xdotli commented Jul 30, 2026

Copy link
Copy Markdown
Member

Correcting my scope argument — the reason I gave was wrong.

I wrote that "this list is specifically about evaluating agents, and structured-output generation is a model capability benchmark." §9 itself already carries Humanity's Last Exam (README:384) and ARC-AGI-2 (README:392), both single-shot with no tools and no trajectory. So the line I drew isn't the line the list actually draws, and it wasn't fair to apply it to you.

The real pattern, as far as I can reconstruct it: non-agent benchmarks get carried when they teach a methodological lesson — MMLU-Redux for label error, LiveCodeBench for contamination resistance. Pure capability benchmarks generally don't.

So the question isn't "is StructEval about agents" but "does it teach something methodological that this list is missing." If you think it does — cross-format validation, or the visual/structural checking approach — make that case and I'll reconsider on the merits.

One more correction: I said you'd disclosed on #54 but not here. What you disclosed on #54 was ClawBench maintainership, which is a different resource — so there was nothing inconsistent about this PR. It would still be good to note your StructEval authorship explicitly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants