A Serious Feature Extraction Bug

[kellyiss]

Hi, authors. I used the code in the spatial reasoning branch.
But I found a severe bug exists in the current implementation of the Wan model's input handling.
The wan model expects input tensors of shape [B, C_in, F, H, W] (Batch, Channels, Frames, Height, Width).

VEGA-3D/src/qwen_vl/model/generative_encoders/wan/modules/model.py

Line 506 in ca64583

x (List[Tensor]):

However, the current implementation incorrectly maps the frame dimension (F) to the batch dimension (B).
For a 32-frame input, the model currently receives:
Input shape: [32, C_in, 1, H, W]

The model should receive:
Input shape: [1, C_in, 32, H, W]

This dimension mapping error causes the model to treat each frame as an independent video, resulting in:

Extracting features from 32 individual single images instead of processing them as a coherent multi-view video sequence

Please double check if I'm correct.

[HyperbolicCurve]

Thanks for the careful read. You're correct that we feed [32, C, 1, H, W] rather than [1, C, 32, H, W]. This is intentional rather than a shape bug, for two reasons:

1. WAN's VAE has temporal stride 4 — packing 32 frames into F collapses them to ~9 latent slots, which then no longer aligns 1-to-1 with the 32 frame tokens our FeatureFusionModule expects. Per-frame encoding preserves that alignment without an extra temporal upsampling step.
2. We aren't sampling video from noise; we run a single forward at a fixed timestep and read an intermediate block. In this single-step feature-extraction regime, the marginal contribution of cross-frame self-attention is much smaller than in iterative denoising — empirically the per-frame WAN-DiT activations already provide useful diffusion priors over the SigLIP baseline.

We agree the per-frame batching is also somewhat inefficient and plan to revisit in a follow-up.

[kellyiss]

Actually, you can still encode the VAE latent as in the current version and forward the 32 latents to the WAN model to extract features of the exact same shape by reshaping [32, C, 1, H, W] to [1, C, 32, H, W]. However, the main issue is: if what you're extracting is not multi-view features, then the 3D claim argued in the paper becomes very weak. And the analysis in Fig. 2(a) seems to reflect a different approach than per-frame batching.

[HyperbolicCurve]

Thanks for pushing on this. A few clarifications🙏:

The observation originated from SEVA / Vmem, which are Novel-View-Synthesis models and more close to multi-view 3D priors. In scripts/3d/preprocessing/extract_seva_vmem_offline_step1_features.py#L183-L280, all 32 frames go through a single forward with per-frame Plücker rays, c2w/w2c, intrinsics K, and a MultiviewCFG guider, genuine cross-view conditioning. The features we read out are the cross-view attended. That's the basis of the 3D claim in the paper.

Wan2.1-T2V and Wan2.1-VACE were added afterward as DiT extensions. These towers don't natively accept multi-view geometric conditioning (no Plücker/extrinsics input), so we use them as per-frame diffusion-prior encoders and were positively surprised the same downstream gains transfer, not a claim that the Wan tower itself does multi-view attention.

Fig. 2(a) is also produced from per-frame batching as the released visualization scripts wan_t2v_adjacent_correspondence_matching.pyon the main branch. So the cross-view consistency in that figure is already obtainable from per-frame WAN-DiT features, a true [1, C, F, H, W] forward should only strengthen it.

Next step. We'll tighten the per-frame-vs-multi-view wording, and run an ablation with the batched form [1, C, F, H, W] following your suggestions. Thank you very much!