Add AutoGluon-resolved validation wrappers (V2) and preset-driven model-specific preprocessing

[Innixma]

What

Two related additions for running AutoGluon presets through the TabArena benchmark framework.

V2 wrappers: AutoGluon resolves the validation splits

AGWrapperV2 / AGSingleWrapperV2 / AGSingleBagWrapperV2 (plus the AGExperimentV2 and AGModelBagExperimentV2 experiment flavours that fix them) declare a task's grouped / temporal structure to TabularPredictor.fit(validation_structure=...) instead of resolving folds in TabArena and passing custom_splits. This is the path for full-predictor preset runs on non-IID data, where one fit trains many models and TabArena cannot pre-resolve splits per model. TabArenaExperimentBundle and generate_bag_experiments gain a bag_experiment_cls / experiment_cls knob to opt into the V2 flavour. This is the infrastructure the AutoGluon 1.6 release benchmark runs (TabArena-Full and BeyondArena-Core) used.

Preset-driven runs now get the model-specific preprocessing

AGExperiment._apply_model_specific_preprocessing previously skipped injection whenever fit_kwargs["hyperparameters"] was not a dict, which is exactly what a preset-driven run looks like (the shipped presets carry a named portfolio string such as "noncommercial_2026_08_05"). Every model then silently missed the model-specific step, most importantly the NoCatAsStringCategoryFeatureGenerator encoding of string columns. Models whose encoders only handle category dtype crashed on raw object columns: in the 1.6 release benchmark, TabDPT-Turbo failed deterministically on the 5 TabArena datasets that carry object-dtype columns (seismic-bumps, in_vehicle_coupon_recommendation, coil2000_insurance_policies, bank-marketing, HR_Analytics_Job_Change_of_Data_Scientists), 90 model skips across both arms.

The method now resolves the hyperparameters the same way TabularPredictor.fit does before injecting:

• a named config string is expanded via get_hyperparameter_config (a typo raises the valid-names ValueError at experiment-preprocessing time instead of inside the fit)
• a bare presets= run takes the hyperparameters entry from the preset dict(s), with AutoGluon's exact merge semantics: aliases resolve, presets apply first-to-last with the last one setting the key winning, and an explicit hyperparameters key (even None) blocks the presets' value
• nothing from either source falls back to fit's own "default"

The resolved dict is written back to the per-run copy of fit_kwargs. Explicit fit kwargs take precedence over presets in AutoGluon and the value equals what the preset would have resolved to, so the fit itself is unchanged; it only gains the per-model preprocessing. Shapes that cannot be resolved (for example an int) warn instead of passing silently.

Verification

• New tests in tests/tabarena/benchmark/experiment/test_build_config_pipeline.py cover the named-config expansion, the typo error, bare presets with alias resolution, a preset without a hyperparameters entry (falls back to "default"), the explicit-None-blocks-preset precedence, last-preset-wins on preset lists, and the unresolvable-shape warning. 459 tests pass across the experiment and preprocessing suites.
• End-to-end check with the release benchmark construction (AGExperimentV2 plus a deep-copied noncommercial preset): all portfolio configs get wrapped, TabDPT-Turbo receives the Identity + NoCatAsStringCategoryFeatureGenerator stack, and PrepLightGBM composes the default stage with its own TabPrep pipeline instead of losing either.
• The experiment object's stored fit_kwargs still carries the original string (resolution happens on the per-run deepcopy), so cache identity is unaffected.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ELdutHiUqkvynzEP7EnPsi

[LennartPurucker]

Mhm, I am not sure we would want to support it in this way. We dont want to have more diverging pipelines for how models are benchmarked. IMO a bigger discussion. If this is just for running AutoGluon, then we can go to the systems API. If this is for benchmarking, we shoud talk more about it

[Innixma]

Note: I haven't actually looked over this code myself very closely, but it was what I was using for running AG on BeyondArena

[LennartPurucker]

Should be fine for now, but I would argue against committing this for long-term use IMO.
The new system's API is better suited then for fairness and our own sanity to avoid more divering pipelines