Submission Eligibility: Agentic Systems

[alirezamshi]

Hi guys,

I wanted to check whether TabArena allows agentic ML systems to be submitted to the leaderboard.

Would this be eligible for the leaderboard, perhaps as an external system or reference pipeline?

Thanks,

[LennartPurucker]

Heyho @alirezamshi!

TabArena supports benchmarking (agentic) ML systems, which you can already use to compare with the SOTA. Check out this example: https://github.com/autogluon/tabarena/blob/main/examples/advanced/run_quickstart_tabarena_external_system.py

We are still determining how to add such systems to the leaderboard. By design, TabArena is a benchmark for model-specific pipelines.
Hence, our current goal is to add a new view of the existing leaderboard or an additional leaderboard that also shows systems and agents.
We will likely also re-frame and remove reference pipelines going forward and instead focus only on systems vs. models.

The hard part is understanding how to mitigate presenting wrong conclusions on the leaderboard due to leakage from LLM-based systems (https://arxiv.org/abs/2602.04031, https://arxiv.org/abs/2404.06209) and cheating in closed-source APIs.

What are your thoughts on the topic?