Hello,
I recently used TabArena to make a benchmark on some basic ensemble models (hard voting, blending, ...). To do this, I made some code that might be useful to other people. I was wondering if it might interest you to integrate some parts directly in TabArena (in which case I would need to better adapt to it).
What I have:
-
Ensemble methods that work with TabArena's 3-fold precomputed results for classification
- Soft Voting (+ Median / Max / Min Rules)
- Hard Voting with vote proportion for probabilitic classification
- Blending (Stacking with base learners only trained once and a small meta-dataset)
- Stacking with some caveats
- Uses two sets of 3-fold CV
- Data leakage: Base learners used for the meta-dataset have seen the test set
- (NeuralEnsembler too but it's nearly just copy pasted from RegularizedNeuralEnsembler)
-
Some specific TaskEvaluator children
- One that doesn't preprocess wrong probabilities out of existence during prediction (necessary for Hard Voting and methods requiring normalization)
- One that makes a nested 3-fold CV on the test data
- 1/9 of data to train just the ensemble method (like blending)
- 2/9 of data to evaluate
- One specifically for Stacking
-
Some multi-family CASH simulation with time budget and iteration budget
- Random Search
- Bandit-based methods (UCB, MaxUCB)
If anything looks interesting, I will need to adapt my code to quite a bit to better integrate it in your project. Notably, I didn't use probability calibration for my code, worked with the 3-fold CV, only ever used the "native" mode, and didn't work on regression.
Have a nice day !
Hello,
I recently used TabArena to make a benchmark on some basic ensemble models (hard voting, blending, ...). To do this, I made some code that might be useful to other people. I was wondering if it might interest you to integrate some parts directly in TabArena (in which case I would need to better adapt to it).
What I have:
Ensemble methods that work with TabArena's 3-fold precomputed results for classification
Some specific TaskEvaluator children
Some multi-family CASH simulation with time budget and iteration budget
If anything looks interesting, I will need to adapt my code to quite a bit to better integrate it in your project. Notably, I didn't use probability calibration for my code, worked with the 3-fold CV, only ever used the "native" mode, and didn't work on regression.
Have a nice day !