微调模型时加入validation出现问题

[symhsym]

我在使用sft脚本多卡训练模型的时候，如果使用validation数据集就总是会出现NCCL超时的问题，请问一下有没有人了解类似情况和解决办法，推测可能是在验证集生成音频样本的时候对模型unwrapped_model.to(torch.bfloat16)导致的？

[liuxin99]

推测有可能推理耗时比较长？尝试在这个地方：

VoxCPM/scripts/train_voxcpm_finetune.py

Lines 375 to 386 in 7aadc6c

	if writer is not None and val_ds is not None and audio_vae is not None and accelerator.rank == 0:
	try:
	generate_sample_audio(model, val_ds, audio_vae, writer, step, accelerator, sample_rate,
	val_texts=val_texts, tokenizer=tokenizer, valid_interval=valid_interval,
	tracker=tracker)
	except Exception as e:
	tracker.print(f"[Warning] Failed to generate sample audio: {e}")
	import traceback
	import io
	buf = io.StringIO()
	traceback.print_exc(file=buf)
	tracker.print(buf.getvalue())

改成：

    if val_ds is not None and audio_vae is not None:
        if accelerator.rank == 0 and writer is not None:
            try:
                generate_sample_audio(...)
            except Exception as e:
                tracker.print(f"[Warning] Failed to generate sample audio: {e}")
        accelerator.barrier()

看能否解决问题呢？

[symhsym]

谢谢回复，问题已经解决了，不是推理耗时过长的问题，问题出在进行推理时generate_sample_audio函数中进行的unwrapped_model.to(torch.bfloat16)和unwrapped_model.to(torch.float32)这里，可能是模型量化转换的时候发生一些数值浮动，导致validate跑完后再进行训练时卡住，删掉这两个部分然后修改推理部分的代码为：

autocast_ctx = (
                torch.autocast(device_type="cuda", dtype=torch.bfloat16)
                if torch.cuda.is_available()
                else contextlib.nullcontext()
            )

            with torch.no_grad():
                with autocast_ctx:
                    if use_interleave_generate:
                        if not hasattr(unwrapped_model, "val_interleave_generate"):
                            raise AttributeError(
                                "Model has no method `val_interleave_generate`. "
                                "Please add it to VoxCPMModel first."
                            )
                        generated = unwrapped_model.val_interleave_generate(
                            target_text=text,
                            chunk_chars=interleave_chunk_chars,
                            inference_timesteps=inference_timesteps,
                            cfg_value=cfg_value,
                        )
                    else:
                        # fallback old path
                        generated = unwrapped_model.generate(
                            target_text=text,
                            inference_timesteps=inference_timesteps,
                            cfg_value=cfg_value,
                        )

就可以了
在训练时加上验证集音频播放真是很方便的工作！

[liuxin99]

好的，感谢指正，后续会修一下