Is any evaluation prove it won‘t damage the quality of output?

[Haodong-Lei-Ray]

I think researchers should make an evaluation on some benchmark, such as Humaneval, Locomo, and SWE-bench. Just for proving😭🙋.

Thank you for your work.

[DietrichGebert]

Thanks! Good news, this already exists.

The benchmark in benchmarks/agentic/ doesn't just count lines, it runs the generated code against adversarial input (the safety tier), so it can catch a skill that "wins" on size by breaking the code. On top of that, #83 added a robustness audit, 12 edge-case traps (is_prime(1), factorial(0), 1900 is not a leap year, subtractive Roman numerals, etc.) on weak models, with every check self-verified before it scores, and ponytail holds baseline parity. And #171 just added a completeness judge so an arm can't win on fewer lines by quietly doing less.

I haven't run HumanEval or SWE-bench specifically. Those are single-function / repo-task suites, and the agentic bench is closer to the real multi-file editing where bloat actually shows up. But the core worry, "does making it lazy make it worse," is exactly what the bench is built to be able to disprove, not just confirm.

Closing as answered. Happy to keep talking if you want.