无法启动

[rain2307]

[startup] Initializing VAE
Traceback (most recent call last):
  File "/home/aaa/models/Lance/inference_lance.py", line 671, in <module>
    main()
  File "/home/aaa/models/Lance/inference_lance.py", line 468, in main
    vae_model = WanVideoVAE()
                ^^^^^^^^^^^^^
  File "/home/aaa/models/Lance/modeling/vae/wan/model.py", line 46, in __init__
    self.configure_vae_model()
  File "/home/aaa/models/Lance/modeling/vae/wan/model.py", line 69, in configure_vae_model
    self.vae: Wan2_2_VAE = Wan2_2_VAE(vae_pth=vae_path, device=device, dtype=self.dtype)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaa/models/Lance/modeling/vae/wan/vae2_2.py", line 985, in __init__
    self.scale = [mean, 1.0 / std]
                        ~~~~^~~~~
  File "/home/aaa/app/miniconda3/envs/Lance/lib/python3.11/site-packages/torch/_tensor.py", line 39, in wrapped
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/aaa/app/miniconda3/envs/Lance/lib/python3.11/site-packages/torch/_tensor.py", line 1077, in __rdiv__
    return self.reciprocal() * other
           ^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Traceback (most recent call last):
  File "/home/aaa/app/miniconda3/envs/Lance/bin/accelerate", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/aaa/app/miniconda3/envs/Lance/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
    args.func(args)
  File "/home/aaa/app/miniconda3/envs/Lance/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1405, in launch_command
    simple_launcher(args)
  File "/home/aaa/app/miniconda3/envs/Lance/lib/python3.11/site-packages/accelerate/commands/launch.py", line 993, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/aaa/app/miniconda3/envs/Lance/bin/python3.11', 'inference_lance.py', '--model_path', 'downloads/Lance_3B_Video', '--vit_type', 'qwen_2_5_vl_original', '--llm_qk_norm', 'true', '--llm_qk_norm_und', 'true', '--llm_qk_norm_gen', 'true', '--tie_word_embeddings', 'false', '--validation_num_timesteps', '30', '--validation_timestep_shift', '3.5', '--copy_init_moe', 'true', '--max_num_frames', '121', '--max_latent_size', '64', '--latent_patch_size', '1', '1', '1', '--visual_und', 'true', '--visual_gen', 'true', '--vae_model_type', 'wan', '--apply_qwen_2_5_vl_pos_emb', 'true', '--apply_chat_template', 'false', '--cfg_type', '0', '--validation_data_seed', '42', '--video_height', '768', '--video_width', '768', '--num_frames', '50', '--task', 'x2t_image', '--save_path_gen', 'results/x2t_image_sample_ts30_tts3.5_seed42_cfg4.0_kvcache_20260524_092540', '--resolution', 'video_480p', '--text_template', 'true', '--cfg_text_scale', '4.0', '--use_KVcache', 'true']' returned non-zero exit status 1.

操作系统 : ubuntu26.04
显卡: blackwell 显卡 NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2

按照readme一步步执行的

[fengyifu2000]

This may be a PyTorch/CUDA binary compatibility issue on Blackwell GPUs, rather than a code bug.

The error happens at:

self.scale = [mean, 1.0 / std]

This line triggers a CUDA tensor operation because std is on the GPU. The error:

CUDA error: no kernel image is available for execution on the device

usually means that the installed PyTorch/CUDA wheel does not contain kernels compiled for the current GPU architecture.

Lance currently pins the recommended environment to:

torch==2.5.1+cu124
torchvision==0.20.1+cu124
torchaudio==2.5.1+cu124

This works in our A100 reference environment because A100 is sm_80, and the installed PyTorch wheel includes sm_80.

Our working reference environment:

GPU: NVIDIA A100-SXM4-80GB
Driver: 535.261.03
nvidia-smi CUDA Version: 12.4
OS: Debian GNU/Linux 12
Python: 3.11.15
PyTorch: 2.5.1+cu124
torch.version.cuda: 12.4
GPU capability: (8, 0)
torch arch list: ['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
basic CUDA op: passed

For the failing Blackwell machine, could you please run the following diagnostic script?

python - <<'PY'
import torch

print("torch:", torch.__version__)
print("torch.version.cuda:", torch.version.cuda)
print("cuda available:", torch.cuda.is_available())
print("gpu:", torch.cuda.get_device_name(0))
print("capability:", torch.cuda.get_device_capability(0))
print("torch arch list:", torch.cuda.get_arch_list())

x = torch.tensor([1.0], device="cuda")
print("basic cuda op:", (1.0 / x).cpu())
PY

If the Blackwell GPU reports a capability such as (12, 0), but torch.cuda.get_arch_list() does not include the corresponding Blackwell architecture, for example sm_120, then the installed PyTorch wheel likely does not support this GPU architecture.

If this is confirmed to be a GPU architecture mismatch, I would suggest not directly relying on the fixed torch==2.5.1+cu124 environment from the Lance README for the Blackwell machine. Instead, please set up a PyTorch/CUDA environment that is compatible with the local GPU, driver, and CUDA support.

The key point is to use an environment supported by the local hardware, and make sure PyTorch, the CUDA runtime, and CUDA extensions such as xformers, flash-attn, and sageattention are all compatible with that environment.