[Meta] custom post-hoc ensembling

[mglowacki100]

Hi @LennartPurucker and @Innixma

Are there any examples or recommended patterns for implementing custom post-hoc ensembling with tabarena?

I recently came across two interesting papers that I’d like to reimplement/check:
https://arxiv.org/pdf/2508.05144
https://arxiv.org/pdf/2512.22284

Thanks in advance for any pointers or guidance.

[Innixma]

Hi @mglowacki100, thanks for your interest!

You can refer to this code which I recently wrote to test out new post-hoc calibration techniques: https://github.com/autogluon/tabarena/blob/main/examples/tmp/run_scratch_add_calibration_logic.py

You should be able to run this code to get an idea of what is going on.

Note: Before running the code, you need to do: pip install probmetrics pytorch-minimize (and following the TabArena install instructions)

You can refer to the ensemble_cls=EnsembleScorerCalibrated line. This is where you would put your custom class that performs post-hoc ensembling.

The class you want to inherit from is EnsembleScorerMaxModels: https://github.com/autogluon/tabarena/blob/main/tabarena/tabarena/simulation/ensemble_selection_config_scorer.py#L230-L305

You can completely override all the methods.

The main code you want to edit is here: https://github.com/autogluon/tabarena/blob/main/tabarena/tabarena/simulation/ensemble_selection_config_scorer.py#L157-L163

I'd recommend using a debugger when running the script and putting breakpoints in this class to better understand what is going on.

If you want to simplify the code for yourself, I'd suggest forcing use_fast_metrics=False during your class init.

Note: To evaluate ensembling you'll need to download one of the method's processed results that have multiple configs, such as LightGBM. This will be a relatively large download (~17 GB per method), so ensure you have a stable and fast internet connection and sufficient disk space, then you can try setting run_only_small_methods=False. Once they have been downloaded, you won't need to redownload them and everything should run quickly.

Let me know if I missed anything, cheers!