Frame Alignment for Sparse Camera Motion Annotations

[sustecher]

I have a question about frame alignment for camera motion annotation.
To reduce annotation costs, we usually use sparse labels: for a 100-frame video, only 25 frames have camera motion annotations.

During training, do you

1. downsample the video or
2. interpolate the sparse annotations (which seems more reasonable for preserving temporal resolution)

If using interpolation, what interpolation method is applied?

Thanks!

[ChefTom]

Are you using LingBot-World-Base (Cam) and (Act) variants, or just one?
Without a true annotation path planner,it's best to break this down exactly by considering three scenarios, answering the model choice and the sparse → dense densification decision strategy.

• 1. Static subject (e.g. smooth orbital fly-around of a parked car — pure cinematic camera operator circling a fixed object)
  Recommended model: Cam-only (pure 6-DoF pose control).

  The GitHub explicitly lists this under Cam use cases: “orbiting static objects” and “camera fly-through”. No embodied agent is needed, so poses.npy + intrinsics.npy give you perfect independent camera movement without any discrete action noise.
  Densification strategy:
Always interpolate to full temporal resolution (never downsample). The model was trained on dense, per-frame poses at the original 16 FPS video rate (synchronized GT from Unreal + MegaSAM pseudo-labels). Downsampling would break the temporal consistency the diffusion transformer expects.
  Best interpolation method (to preserve natural human-operator feel):
• Cubic Bézier spline for translation (positions)
• SLERP for rotation (quaternions inside the 4×4 matrix)
This matches the “multi-point interpolation” we use internally in the Unreal trajectory generator for smooth orbits. Zero jitter, constant velocity feel, and it’s the exact style you see in the official fly.mp4 demo (generated from ViPE-extracted poses).
• 2. Slow-moving subject (e.g. person walking down the street; camera follows from behind, then pans across and overtakes, human-like POV filming)
  Recommended model: Cam-only.
Again, GitHub directly calls this out: “following moving subjects”. The camera is an independent operator (not the walking person’s own eyes), so pure pose control is cleaner and avoids the discrete W/A/S/D quantization that Act introduces. (If you wanted POV first-person “I am the walker” footage, then Act would be better; but that’s not your description.)
  Densification strategy: Same as above, interpolate to full 16 FPS. The paper stresses dense temporal alignment for long-horizon consistency; sparse labels fed directly would cause drift in the follow-cam.
  Best interpolation method:
• Catmull-Rom spline for the camera path (gives natural easing and C1 continuity — perfect for gentle acceleration when overtaking)
• SLERP for rotation
This produces the buttery “human operator with gimbal” feel some camera POV demands. It’s the same continuity we get from ViPE on real stabilized footage.
• 3. High-speed subject (e.g. helicopter flying at speed; camera in a chase plane circles it for a movie-style shot — fast relative motion)
  Recommended model: Cam-only.
GitHub explicitly includes “high-speed chase scenes” as a Cam use case. The relative velocity is handled entirely by the dense pose sequence; Act’s discrete actions would introduce stepping artifacts at high speeds.
  Densification strategy: Interpolate to full temporal resolution (mandatory for stability). The model’s long-term memory and Plücker embeddings were trained on full-rate trajectories — downsampling would amplify any aliasing in fast motion.
  Best interpolation method:
• Cubic Bézier (with tight handle control for the sharp circular path) + SLERP for rotation
• Add a small constant-velocity bias on the translation spline to keep the circling speed uniform (exactly how our Unreal synthetic pipeline handles fast trajectories before exporting dense poses).
  Quick Implementation Tip (Works for All Three)
  Since the repo requires full dense poses.npy, use this exact pipeline (zero extra cost):
• Define 5–10 keyframes (position + quaternion).
• Apply the spline/SLERP combo above (scipy.interpolate.CubicSpline + scipy.spatial.transform.Slerp — <0.1 s on CPU).
• Save the dense 4×4 matrices + intrinsics → drop into --action_path.
  This is 100 % compatible with the official Cam model (including the 4-bit quantized version) and gives the same quality as our internal ViPE + MegaSAM workflow. The Act model is only needed when you want an embodied agent inside the world (e.g., controlling a character’s legs + head look).
  You can create a Python snippet tailored to one of your three scenarios (or a Blender/Unreal export template), it really depends upon your scenario.