OneTrainer Windows-ROCm profile installs bitsandbytes 0.50.0.dev0, which OneTrainer doesn't support yet — bnb 8-bit optimizers crash

[dxqb]

OneTrainerWindowsRocmProfile.cs currently installs bitsandbytes from
0xDELUXA/bitsandbytes_win_rocm release 0.50.0.dev0-py3.12-rocm7.15-win_amd64_all.

OneTrainer pins bitsandbytes==0.49.1 and doesn't support 0.50 yet: 0.50 removed the block_wise= and percentile_clipping= arguments from all optimizer constructors, which OneTrainer still passes. Any training run using a bitsandbytes optimizer (AdamW8bit etc.) fails at startup:

File "...\modules\util\create.py", line 251, in create_optimizer
  optimizer = bnb.optim.AdamW8bit(
TypeError: AdamW8bit.__init__() got an unexpected keyword argument 'percentile_clipping'

Details on the API break and the planned OneTrainer-side upgrade are here:
Nerogar/OneTrainer#1519 (comment)

This looks like it started with the July 15 wheel bump ("Update bitsandbytes wheel for ROCm 7.13/7.14/7.15 compatibility") — the previously used 0.49-based wheel wouldn't have hit this.

Until OneTrainer supports 0.50, could the profile pin a 0.49-based wheel instead? The same repo has v0.49.2.dev0-py312-rocm7.12 — though it's built against ROCm 7.12 rather than 7.15, so it may not fit the torch build the profile installs. If no compatible 0.49 wheel exists for ROCm-on-Windows, it might be worth noting in the profile that bnb optimizers are currently broken there; OneTrainer will catch up once the 0.50 migration lands.

Drafted by Claude

[dxqb]

side question - do you have a working AMD install for OneTrainer?
we've been struggling with finding one: Nerogar/OneTrainer#1559

if you have, we could "downstream" it so you don't have to manage your own requirements for it but it's part of OneTrainer

[NeuralFault]

OneTrainerWindowsRocmProfile.cs currently installs bitsandbytes from 0xDELUXA/bitsandbytes_win_rocm release 0.50.0.dev0-py3.12-rocm7.15-win_amd64_all.

OneTrainer pins bitsandbytes==0.49.1 and doesn't support 0.50 yet: 0.50 removed the block_wise= and percentile_clipping= arguments from all optimizer constructors, which OneTrainer still passes. Any training run using a bitsandbytes optimizer (AdamW8bit etc.) fails at startup:

File "...\modules\util\create.py", line 251, in create_optimizer
  optimizer = bnb.optim.AdamW8bit(
TypeError: AdamW8bit.__init__() got an unexpected keyword argument 'percentile_clipping'

Details on the API break and the planned OneTrainer-side upgrade are here: Nerogar/OneTrainer#1519 (comment)

This looks like it started with the July 15 wheel bump ("Update bitsandbytes wheel for ROCm 7.13/7.14/7.15 compatibility") — the previously used 0.49-based wheel wouldn't have hit this.

Until OneTrainer supports 0.50, could the profile pin a 0.49-based wheel instead? The same repo has v0.49.2.dev0-py312-rocm7.12 — though it's built against ROCm 7.12 rather than 7.15, so it may not fit the torch build the profile installs. If no compatible 0.49 wheel exists for ROCm-on-Windows, it might be worth noting in the profile that bnb optimizers are currently broken there; OneTrainer will catch up once the 0.50 migration lands.

Drafted by Claude

Thanks for the report. There is no availability of a bitsandbytes wheel version .49 for the current production version of ROCm.
Going back to that version would involve reverting the ROCm install logic to ROCm 3.12 from the (soon to be deprecated and no longer updated) v2 repo as there is not any version lesser than 7.13 in the current multi-arch repo.

Upstream b'n'b also has only added support starting with 7.14 with version .50 also. (which will switch to using when verified for windows compatibility).

Having a carve out for the OneTrainer package would require some modification of the ROCm helper to accommodate with a decent bit of logic specific for it for something that is only temporary.

Think best option would be just disabling the automatic install for bnb for the time being until the .50 pin update in upstream OneTrainer. Importing of the regular .49 bnb will fail at every package launch... but should be handled gracefully. Along with a note in the package card in the SM UI, or in the console output as a message at launch-time about it not working temporarily.

In regards to a "working AMD install", this is all facilitated by custom helper framework for ROCm on Windows done in-line additional to the base install of OneTrainer.

It is effectively the same process as what my GPU detect and ROCm install/launch/updating does in this fork, same fundamental process as I had proposed in the OneTrainer discord previously (except any baseline optimization/compatibility envioronment variables that SM passes to all ROCm helper compatible package launches)

[0xDELUXA]

Fixed in #1723 by pinning the 0.49.1 wheel. I opened Nerogar/OneTrainer#1708 to migrate OneTrainer to 0.50.2, but that is not on the roadmap right now, so the wheel moves to match the pin instead.