Overlapping Explorations and Open Questions
Context
We have independently explored a closely related research direction: learning continuous latent dynamics for world modeling. Below we describe the components that overlap with PT-Flow, along with the difficulties we encountered. We hope this can serve as a starting point for discussion and would greatly appreciate insights on the open questions at the end.
Core Overlap: Latent ODE Predictor
We implemented a latent ODE dynamics function of the form dz/dt = f_int(z) + Σ_k G_k · proj_k(z) · s_k with RK4 integration, variable step sizes, and arbitrary Δt prediction. This shares the same paradigm as PT-Flow's dz/dt = v_θ(z,t). The key difference is structural: we decompose the velocity field into an intrinsic component f_int = W·z + MLP_residual plus learnable operator directions G_k with soft gating and consolidation, whereas PT-Flow uses a monolithic MLP with FiLM time conditioning. We found the decomposed form more interpretable but are uncertain whether it sacrifices expressiveness compared to the unconstrained MLP.
Related Explorations via Different Mechanisms
1. Continuous-Time Transition
PT-Flow approach: Numerical ODE integration z_T = z_0 + ∫v_θ dt.
Our approach: Closed-form solution for linear systems via matrix exponential z_{t+Δt} = exp(A·Δt) · z_t. Verified on a linear Markov-closed system S=(q,v), q'=q+v: training loss 3.6e-7, OOD generalization of 2082× when extrapolating to velocities 2× outside the training range. However, we were unable to extend this to general nonlinear dynamics — the closed-form solution is restricted to linear/piecewise-linear systems.
2. Static/Dynamic Information Separation
PT-Flow approach: Initial-state conditioning — the dynamics encoder f_dyn(s_t; s_0) and decoder g_dyn(z_t; s_0) are both conditioned on the initial state s_0, forcing z_t to encode only temporal changes.
Our approach: Dimensional split into z_inv/z_var — the latent vector is partitioned into an invariant component (z_inv, supervised via InfoNCE to maintain temporal consistency) and a varying component (z_var, supervised with speed/direction heads to capture evolution). An orthogonality loss constrains the two to remain decoupled. We found this explicit split easier to supervise but acknowledge that the boundary between "invariant" and "varying" is not always clean.
3. Continuous Representation Space
PT-Flow approach: Single-token compression z_t ∈ R^{1×768} extracted via cross-attention from DINO features.
Our approach: Continuous kernel field Z(q) = Σ_j K(q - q_j) v_j, where each atom produces a coordinate q_j = P_φ(a_j) and content v_j. Field values at arbitrary query coordinates are obtained via Gaussian kernel interpolation. This supports arbitrary-location readout and leave-one-out addressability verification. The representation is richer but significantly more expensive than the single-token approach.
Difficulties Encountered
1. Poor nonlinear generalization.
Linear dynamics generalize remarkably well (2082× OOD); replacing the transition function with an MLP degrades generalization substantially. Our interpretation is that the Jacobian of nonlinear systems changes rapidly with state, causing local linear approximations to break down quickly. PT-Flow sidesteps the perceptual nonlinearity via the frozen DINO encoder, but the dynamical nonlinearity at the latent level remains.
2. Unstable velocity supervision.
Direct velocity prediction required log-space MSE, and we found it necessary to supervise speed and direction separately. PT-Flow's JVP projection (Eq. 4: ż_t = JVP(f_dyn, ṡ_t)) is an elegant alternative, but we are uncertain whether this supervision remains effective when the encoder Jacobian is near-zero or rank-deficient in certain regions.
3. Long-horizon error accumulation.
Single-step MSE of 3.5e-7 grows to 6.3e-4 over 12-step rollouts (1800× amplification). ODE integration truncation error grows with horizon, and the growth is faster for more nonlinear systems.
4. Representation collapse requires multi-signal constraints.
Any single loss (reconstruction or velocity) leads to collapse. PT-Flow uses reconstruction + velocity dual loss for mutual constraint; we used InfoNCE + von Neumann entropy + orthogonality + speed/direction supervision. Both approaches rely on "multiple complementary signals to prevent degeneration," but neither provides a proof that collapse is eliminated rather than merely delayed.
Open Questions
We would be grateful for any insights on the following:
-
Temporal extrapolation. Training time τ is sampled at discrete frame offsets; at inference, FiLM conditions on continuous τ values. Is there evidence that FiLM generalizes beyond the training time range (e.g., training on 1–64 frames, predicting at 128+ frames)? In our similar time-conditioning scheme, the velocity field degraded outside the training time range.
-
JVP dependence on encoder quality. Eq. 4 assumes the encoder Jacobian accurately reflects local dynamics. PT-Flow ensures this via the frozen DINO encoder. For non-pretrained or jointly trained encoders, does the Jacobian accuracy still hold? What happens when the encoder Jacobian approaches zero or becomes rank-deficient in certain input regions?
-
Non-smooth dynamics. PT-Flow's robotic tasks involve contact-push-contact sequences with discontinuous forces. Continuous ODEs require a Lipschitz-continuous velocity field, but contact and collision dynamics are inherently discontinuous. For more severe discontinuities (multi-body collision, fluid dynamics), does the velocity field require additional regularization or architectural constraints to maintain physical consistency?
-
Long-horizon rollout bounds. PT-Flow demonstrates high-quality 64-frame predictions. Are there results or theoretical bounds on error accumulation over longer horizons (e.g., 256/512 frames)? In our experiments, the gap between single-step accuracy and multi-step rollout quality widens rapidly; we would be very interested to know whether PT-Flow exhibits the same behavior.
-
Physical reversibility of backward prediction. PT-Flow claims that reversing the velocity sign (v → −v) produces physically plausible sequences. However, most physical processes are irreversible (friction, plastic deformation, dissipation). Is backward prediction intended only for conservative systems, or does it also produce meaningful results in dissipative settings?
Summary
Our current approach has moved toward discrete Hebbian transitions and structured multi-slot states, largely because of the difficulties described above. However, we are very interested in understanding whether and how the continuous ODE paradigm can address these challenges. Any guidance or discussion would be sincerely appreciated.
Overlapping Explorations and Open Questions
Context
We have independently explored a closely related research direction: learning continuous latent dynamics for world modeling. Below we describe the components that overlap with PT-Flow, along with the difficulties we encountered. We hope this can serve as a starting point for discussion and would greatly appreciate insights on the open questions at the end.
Core Overlap: Latent ODE Predictor
We implemented a latent ODE dynamics function of the form
dz/dt = f_int(z) + Σ_k G_k · proj_k(z) · s_kwith RK4 integration, variable step sizes, and arbitrary Δt prediction. This shares the same paradigm as PT-Flow'sdz/dt = v_θ(z,t). The key difference is structural: we decompose the velocity field into an intrinsic componentf_int = W·z + MLP_residualplus learnable operator directions G_k with soft gating and consolidation, whereas PT-Flow uses a monolithic MLP with FiLM time conditioning. We found the decomposed form more interpretable but are uncertain whether it sacrifices expressiveness compared to the unconstrained MLP.Related Explorations via Different Mechanisms
1. Continuous-Time Transition
PT-Flow approach: Numerical ODE integration
z_T = z_0 + ∫v_θ dt.Our approach: Closed-form solution for linear systems via matrix exponential
z_{t+Δt} = exp(A·Δt) · z_t. Verified on a linear Markov-closed systemS=(q,v), q'=q+v: training loss 3.6e-7, OOD generalization of 2082× when extrapolating to velocities 2× outside the training range. However, we were unable to extend this to general nonlinear dynamics — the closed-form solution is restricted to linear/piecewise-linear systems.2. Static/Dynamic Information Separation
PT-Flow approach: Initial-state conditioning — the dynamics encoder
f_dyn(s_t; s_0)and decoderg_dyn(z_t; s_0)are both conditioned on the initial state s_0, forcing z_t to encode only temporal changes.Our approach: Dimensional split into z_inv/z_var — the latent vector is partitioned into an invariant component (z_inv, supervised via InfoNCE to maintain temporal consistency) and a varying component (z_var, supervised with speed/direction heads to capture evolution). An orthogonality loss constrains the two to remain decoupled. We found this explicit split easier to supervise but acknowledge that the boundary between "invariant" and "varying" is not always clean.
3. Continuous Representation Space
PT-Flow approach: Single-token compression
z_t ∈ R^{1×768}extracted via cross-attention from DINO features.Our approach: Continuous kernel field
Z(q) = Σ_j K(q - q_j) v_j, where each atom produces a coordinateq_j = P_φ(a_j)and contentv_j. Field values at arbitrary query coordinates are obtained via Gaussian kernel interpolation. This supports arbitrary-location readout and leave-one-out addressability verification. The representation is richer but significantly more expensive than the single-token approach.Difficulties Encountered
1. Poor nonlinear generalization.
Linear dynamics generalize remarkably well (2082× OOD); replacing the transition function with an MLP degrades generalization substantially. Our interpretation is that the Jacobian of nonlinear systems changes rapidly with state, causing local linear approximations to break down quickly. PT-Flow sidesteps the perceptual nonlinearity via the frozen DINO encoder, but the dynamical nonlinearity at the latent level remains.
2. Unstable velocity supervision.
Direct velocity prediction required log-space MSE, and we found it necessary to supervise speed and direction separately. PT-Flow's JVP projection (Eq. 4:
ż_t = JVP(f_dyn, ṡ_t)) is an elegant alternative, but we are uncertain whether this supervision remains effective when the encoder Jacobian is near-zero or rank-deficient in certain regions.3. Long-horizon error accumulation.
Single-step MSE of 3.5e-7 grows to 6.3e-4 over 12-step rollouts (1800× amplification). ODE integration truncation error grows with horizon, and the growth is faster for more nonlinear systems.
4. Representation collapse requires multi-signal constraints.
Any single loss (reconstruction or velocity) leads to collapse. PT-Flow uses reconstruction + velocity dual loss for mutual constraint; we used InfoNCE + von Neumann entropy + orthogonality + speed/direction supervision. Both approaches rely on "multiple complementary signals to prevent degeneration," but neither provides a proof that collapse is eliminated rather than merely delayed.
Open Questions
We would be grateful for any insights on the following:
Temporal extrapolation. Training time τ is sampled at discrete frame offsets; at inference, FiLM conditions on continuous τ values. Is there evidence that FiLM generalizes beyond the training time range (e.g., training on 1–64 frames, predicting at 128+ frames)? In our similar time-conditioning scheme, the velocity field degraded outside the training time range.
JVP dependence on encoder quality. Eq. 4 assumes the encoder Jacobian accurately reflects local dynamics. PT-Flow ensures this via the frozen DINO encoder. For non-pretrained or jointly trained encoders, does the Jacobian accuracy still hold? What happens when the encoder Jacobian approaches zero or becomes rank-deficient in certain input regions?
Non-smooth dynamics. PT-Flow's robotic tasks involve contact-push-contact sequences with discontinuous forces. Continuous ODEs require a Lipschitz-continuous velocity field, but contact and collision dynamics are inherently discontinuous. For more severe discontinuities (multi-body collision, fluid dynamics), does the velocity field require additional regularization or architectural constraints to maintain physical consistency?
Long-horizon rollout bounds. PT-Flow demonstrates high-quality 64-frame predictions. Are there results or theoretical bounds on error accumulation over longer horizons (e.g., 256/512 frames)? In our experiments, the gap between single-step accuracy and multi-step rollout quality widens rapidly; we would be very interested to know whether PT-Flow exhibits the same behavior.
Physical reversibility of backward prediction. PT-Flow claims that reversing the velocity sign (v → −v) produces physically plausible sequences. However, most physical processes are irreversible (friction, plastic deformation, dissipation). Is backward prediction intended only for conservative systems, or does it also produce meaningful results in dissipative settings?
Summary
Our current approach has moved toward discrete Hebbian transitions and structured multi-slot states, largely because of the difficulties described above. However, we are very interested in understanding whether and how the continuous ODE paradigm can address these challenges. Any guidance or discussion would be sincerely appreciated.