The one-sentence idea: today's video models see a clip as a list of frames and have no idea whether two frames are 40 ms or 4 seconds apart; continuum makes a model's internal state evolve over real elapsed time, so it perceives motion the same way whether the video was sampled at 4 fps or 24 fps.
Small on purpose — a 549K-parameter network that trains on one Apple-Silicon Mac — and built to be falsifiable: the evaluation criteria were frozen and committed before any training run existed, and the results below report a missed primary claim with the same prominence as the comparative finding that held.
The point is motion. Speed = distance ÷ time, so a model that can't tell 4 fps from 24 fps can't tell how fast things move — feed it the same motion at two rates and an index-clocked model misreads speed by roughly the rate ratio. Continuum fixes this at the source by evolving its state over real elapsed seconds: at 24 fps it reads motion-speed 2.5× more accurately than its identical index-clocked twin (speed error 0.16 vs 0.41; speed-class 0.46 vs 0.20).
The effect is sharpest on the purest time quantity — event timing (when a bounce happened): continuum stays near-flat (0.057 → 0.073 across 4→24 fps) while the twin explodes 5.7× (0.057 → 0.327), and feeding Δt as a plain input number only reaches 0.199 — so the sense of time has to be integrated into the state, not bolted on. (Perceptual physical time — how fast / when from sampled video — not linguistic temporal reasoning; synthetic physics, 549K params, one seed; under mixed-rate training the gap narrows, so the edge lives in off-rate and irregular capture.)
A video is really a continuous event h(t) that we sample at instants t_0, t_1, …. Every
mainstream video model throws the timestamps away and feeds the frames in as a numbered list.
Concretely, inside a video state-space model the state is advanced one step per frame — the
step is tied to the frame's position in the list, not to the time it represents. So:
- The model literally cannot tell 4 fps from 24 fps. Feed the same motion at both rates and an index-clocked model produces bit-identical output (we prove this as a unit test).
- Physical quantities that depend on time — speed = distance / time, event timing — get silently confounded with the sampling rate. A ball moves far between frames at 4 fps and barely moves at 24 fps; an index-clocked model trained at one rate misreads speed by roughly the rate ratio at another.
The frame_timestamps fields shipped in real video pipelines are metadata — they never enter
the computation.
Continuum is a selective state-space model (Mamba-style) with a single change: the
discretization step of observation k is
step_k = softplus(δ(x_k)) · Δt_k # Δt_k = t_k − t_{k−1}, in SECONDS
so the state evolves as exp(A · step_k) between observations — a genuine exponential
integrator over physical time, modulated by the input. The integral uses an exact
zero-order-hold (ZOH), not the usual first-order Δ approximation. Consequences hold by
construction (each is a CI unit test in tests/, today):
- Partition invariance — integrating the same time interval in 4 steps or 8 lands the state in the same place. This is the mathematical seed of sampling-rate invariance.
- Irregular timestamps are native — jittered, bursty, or dropped frames need no special casing.
- Constant memory in stream length (it's a recurrence): O(1) per step.
Flip one flag (time_aware=False, Δt frozen to its training-set mean) and you get the exact
field-standard model with identical parameters — the protocol's primary ablation. That is the
honest comparison: the same network, the only difference being whether physical time enters.
Four arms, identical data/seeds/parameters; continuum and status_quo are the same
549K-parameter network differing only in whether physical Δt enters the SSM. Trained at 4 fps
uniform only; evaluated at 2–24 fps. Normalized speed-MAE (lower better) / 4-way class accuracy.
Full grid (uniform/jittered/bursty) in results/RESULTS.md.
| eval (uniform) | continuum | status_quo (the field) | transformer | dt_feature |
|---|---|---|---|---|
| 4 fps (train rate) | 0.045 / 0.88 | 0.045 / 0.88 | 0.060 / 0.80 | 0.046 / 0.87 |
| 8 fps | 0.100 / 0.68 | 0.243 / 0.24 | 0.223 / 0.28 | 0.255 / 0.23 |
| 12 fps | 0.131 / 0.58 | 0.330 / 0.20 | 0.239 / 0.30 | 0.350 / 0.20 |
| 24 fps | 0.161 / 0.46 | 0.407 / 0.20 | 0.248 / 0.30 | 0.414 / 0.21 |
What the numbers say — the win first, the miss kept at equal prominence:
- Physical time wins big — and it's the same network. At every off-train rate continuum
degrades 2.5–4.5× more gracefully than every index-clocked arm, and
status_quois continuum with one flag flipped — so this is the mechanism, not a bigger model. Sharpest on event timing: continuum reads when a bounce happened near-flat (0.057 → 0.073 across 4→24 fps) while its identical index-clocked twin explodes 5.7× (0.057 → 0.327). - It's the dynamics, not a free time input. The negative control
dt_feature— Δt fed as an ordinary input number — is the worst arm at 24 fps. The benefit comes specifically from integrating physical time into the state, not appending it. (Fixed-rate regime; the mixed-rate exploration below shows this distinction narrows once training spans rates — reported, not buried.) - What we did not get — stated with equal prominence: invariance. The pre-registered claim was flat (±2%) accuracy across rates. Continuum is not flat — it degrades with distance from the training rate. The honest headline is graceful degradation, not invariance; we publish that miss with the prominence a success would have received.
- Diagnosis: the integrator is partition-exact (unit-tested), but the selective gating
δ(x)learns the observation statistics of the training rate. Architectural time-awareness, trained at one rate, buys robustness, not invariance. (A post-hoc mixed-rate exploration — below — asks whether varied-rate training changes this; it is never merged with the pre-registered run.)
The pre-registered run trains at one rate. The natural follow-up: train across varied rates
({2,4,8,12,24}fps × {uniform,jittered}) — is physical time then exploited, not just survived?
Same four arms, same budget; one seed; exploratory. Full grid in
results/RESULTS_mixed.md. Two findings, at different confidence:
-
Robust — timing information is decisive on irregular capture. On bursty (irregular) schedules both timing-aware arms beat both timing-blind arms by a wide margin, and the blind arms decay as frames are added (more irregularly-spaced frames confuse an index clock) while the timing-aware arms keep improving:
bursty, class acc continuum dt_feature status_quo transformer 8 fps 0.65 0.64 0.56 0.48 12 fps 0.68 0.68 0.54 0.46 24 fps 0.78 0.74 0.53 0.49 -
Honest complication — the "dynamics vs input" distinction largely washes out here. Under fixed-rate training the negative control
dt_featureis the worst arm; under mixed-rate training it nearly matches continuum, trailing only at the highest frame rate (bursty@24: 0.74 vs 0.78 acc; 0.079 vs 0.066 speed-MAE). So whether the model sees Δt dominates; how (state dynamics vs input feature) matters far less once training spans rates. The clean-control result is training-regime-dependent, not universal.
Takeaway: physical-time dynamics remain the single strongest arm, but their signature edge over a plain Δt-input baseline is modest under mixed-rate training. This tempers, not overturns, the tier-1 story — and it is the kind of result this repo reports out loud.
The natural question: is "time as an index" just a toy-model artifact, or does it afflict a real
vision-language model? We tested Gemma 4 12B directly — no training, no weight changes —
by inserting continuum's ZOH resampling as a front-end that chooses which frames the frozen
model sees. Full method and honesty in continuum_gemma/README.md.
First, the blindness is real at scale: asked to rate motion speed from clean frames, Gemma 4 12B correlates ≈ 0 with true speed — it cannot read motion magnitude off frames. It manages only a coarse binary judgment (did the ball stay local, or roam the box?), which is what we benchmark.
Task: binary local vs roams, Gemma 4 12B frozen, K = 6 frames composited to one overlay, N = 120 episodes, exact ground truth (full writeup):
| capture schedule | front-end | overall acc | roaming-ball acc |
|---|---|---|---|
| uniform (clean control) | vanilla — index subsample | 0.817 | 0.833 |
| uniform (clean control) | continuum — physical-time ZOH | 0.758 | 0.783 |
| bursty (irregular) | vanilla — index subsample | 0.767 | 0.650 |
| bursty (irregular) | continuum — physical-time ZOH | 0.808 | 0.817 |
On the roaming balls the mechanism targets, shifting uniform → bursty capture makes index subsampling collapse 0.83 → 0.65 (z = 2.3, p ≈ 0.02) — the fast ball, sampled inside the burst, looks static (left) — while physical-time resampling holds at 0.82 and wins the bursty-roams cell head-to-head (z = 2.1, p ≈ 0.04). Difference-in-differences +0.22.
The shape repeats the small-model story: physical-time handling does not make Gemma better in absolute terms (on the clean control it's a wash) — it makes it degrade gracefully when the capture schedule turns irregular, exactly where index-based frame selection makes a fast ball look static. The effect is modest but significant (≈ a fifth of roaming balls flip, p ≈ 0.02–0.04 at N = 120) — real, not large. A front-end can't install machinery a frozen model lacks; the decisive fix is architectural (a trained continuum SSM in the vision path) — the documented next step.
- Mechanical guarantees, not assertions. Seeds are pinned; every figure regenerates from a
clean clone via
experiments/; the integrator's partition invariance and the ablation's time-blindness are continuously-run unit tests. - Built small on purpose (sub-10M params, one Mac) in the lineage of property-first results the field later scaled: NeRF, RoPE, Mamba, ToMe, Perceiver.
- Prior art, honestly (
RELATED_WORK.md): physical-Δt SSM discretization itself has a precedent in longitudinal medical imaging (Δt-Mamba3D, Oct 2025 — exams months apart, no video, no invariance protocol). What no published work does is the combination here: physical-time selective-SSM video perception under a pre-registered train-one-rate / test-all-rates protocol. Find prior work that does and the wording gets corrected, not defended.
- ARCHITECTURE.md — how the physical-time SSM works, the module layout, the four arms, and how continuum integrates into a model: the training-free ZOH resampling front-end and the trained state → projector → frozen-LLM graft.
- REPRODUCE.md — install and reproduce every number (construction tests, the ~3-minute smoke pass, the pre-registered grid, the Gemma side-by-side, the mixed-rate exploration).
continuum/protocol/PROTOCOL.md— success criteria, frozen before training.
MIT license. Results are exactly as loud as the curves allow.