[TabArena-v0.X] Try better fillna / cat encoding strategies for RF/XT/KNN

[Innixma]

Try better fillna / cat encoding strategies for RF/XT/KNN

Currently ordinal encoding for RF/XT and fillna 0 for RF/XT/KNN.
KNN ignores categorical features.

Figure out what missing categories are assigned to for RF/XT.

Maybe try mean fillna + indicator for numeric.

[dholzmueller]

With the new seed logic we could potentially use different random shuffles of the ordinal encoding between splits.

[LennartPurucker]

@dholzmueller Would you expect this to make a significant difference because of how tree splits would interpret numbers? Interesting. We might need to benchmark a DT-learning algorithm that supports cat columns natively, but I do not think sklearn has this so far.

[dholzmueller]

I would expect trees to have a bias to give more similar predictions to categories that are encoded closer to each other, and proposed to use permutations to break this bias.

I just benchmarked default RF with 8-fold inner CV on TALENT classification in pytabkit, with and without permutation of the ordinals. Permuting is a bit better than not permuting in terms of win-rate, but there seem to be some datasets where not permuting is much better (churn, kr-vs-k, kropt), presumably because the categories are already encoded in a meaningful order, which is destroyed by the permutation. Hence, not permuting is better in some other aggregate metrics.

So in total, it might not be worth going for the permutations.