[tabarena] HillClimbingEnsembler for #4505 ensembling investigation

[a11to1n3]

#4505 investigation

Question: Does continuous hill climbing improve post-hoc ensembling over Caruana Greedy on TabRepo / TabArena OOF?

Answer: no clear improvement. HC is near-parity with Greedy (tiny edges depending on metric; not significant). Not enough to change the default ensembler.

Code

• Optional HillClimbingEnsembler (global-best step + Caruana grid weight)
• Unit tests + RankScorer fix
• Default remains GreedyEnsembler (no behavior change unless opted in)

repo.evaluate_ensemble(..., ensemble_kwargs={
    "ensembler_cls": HillClimbingEnsembler,
    "ensembler_kwargs": {"precision": 0.02, "max_rounds": 40, "include_caruana_step": True},
})

Evidence

1) Paper all-config Portfolio-N200 (main result)

Protocol (evaluate_baselines / zeroshot spirit):

• Full paper config pool: 16 methods, 3016 configs
• Zeroshot Portfolio-N200, ensemble E=40, 4h runtime filter
• 51 datasets × 3 folds (153 tasks)
• Swap only the ensembler (Greedy vs HC)

method	mean rank ↓	mean norm-error	mean #models	H2H W/L/T
hill_climbing	208.7	0.246	9.2	83 / 69 / 1
greedy	210.4	0.249	9.7	—

• Median Δ(test error HC−Greedy) ≈ 0 (−1.7e−5)
• Fold-level sign test on wins: not significant (p ≈ 0.29)
• Pairwise Elo (Greedy fixed at 1000): HC ≈ +25–32 (bootstrap 95% CI includes 0)

2) TabArena-Lite style table (fold 0, RF-calibrated Elo)

Same published methods as the AG 1.6 release table, plus our Portfolio-N200 rows. Elo anchor: RF (default) = 1000.

method	elo	rank	improvability (%)	train s	infer s
AutoGluon 1.6 noncommercial	1858	2.7	4.5	160	7.2
TabFM (default)	1814	3.1	3.5	613	138
AutoGluon 1.6 extreme	1732	3.9	5.4	139	6.1
AutoGluon 1.5 extreme (4h)	1641	4.9	6.5	3758	27
TabPFN-3 (default)	1605	5.3	7.7	26	2.4
Portfolio-N200-HC (ours)	1577	5.7	10.0	9098*	412
Portfolio-N200-Greedy (ours)	1563	5.8	10.0	9098*	393
AutoGluon 1.4 best (4h)	1483	6.7	11.1	8869	34
RealMLP (tuned + ensemble)	1430	7.2	11.8	31982	34

*Portfolio train s is summed config train time in the zeroshot pool, not a single AutoML wall-clock fit.

Published Elo ordering matches the AG 1.6 release table (absolute Elo shifts slightly because the opponent set includes Portfolio-N200 + RF).

Interpretation

• On clean TabArena OOF, continuous HC and Caruana Greedy solve essentially the same job; HC is a viable alternative, not a free accuracy upgrade.
• Portfolio-N200 is a mid-tier zeroshot system on a Lite-style board (~1570 Elo), well below AG 1.6 noncommercial (~1820) — as expected (post-hoc blend of fixed configs vs full AutoML + foundation models).
• Default stays Greedy.

Scope / non-goals

• Not claiming SOTA or AG 1.6-level Elo
• Not replacing GreedyEnsembler by default
• No experimental benchmark scripts in the PR (local re-bench artifacts only)

[LennartPurucker]

Thank you for the submission! I will leave this to @Innixma to review.

From my POV, a small comment: not sure we want to move the experimental script files into the main branch. We could just keep them here in the PR.

[a11to1n3]

Hello @LennartPurucker,

well-noted, I will move out the script files. Looking forward to your reviews.

Cheers,
Duy