EgoDex: temporal action rates as Daft window expressions

[everettVT]

What

Follow-up to #46. The pose-feature stage moves to one row per frame, and every temporal action rate moves out of NumPy-in-a-UDF and into the query plan as Daft window expressions.

• EgoDexPipeline.frame_features() — an episode-level UDF computes instantaneous hand geometry and explodes it into per-frame rows keyed by (task, episode_id, frame_index).
• EgoDexPipeline.temporal_features() / new temporal.py — every rate is an expression over Window().partition_by("task", "episode_id").order_by("frame_index"):
  • lead(1) diffs → curl_rate (grasping), wrist_vert_vel (lifting), arm_ext_rate (reaching)
  • euclidean_distance with lead(1) → wrist_speed (stillness), articulation (in-hand)
  • forearm_roll — the one custom UDF, fed by the window (this frame's rotation + next frame's via lead(1)), smoothed with a centered rows_between(-2, 2) mean (twisting)
• calculate_features() composes both stages; TemporalFeatureComputer is deleted.
• calibrate() becomes Daft percentile aggregations over the pooled per-frame rows.
• query.py scenarios are unchanged — _match_episodes regroups frames into per-episode tracks via groupby.agg(list_agg), re-sorted by frame_index inside the match UDF (list_agg does not guarantee order).

Why

• The temporal math stays in the query plan — no collect, visible to the optimizer.
• The per-frame + window data model matches how frame-level robot data actually ships (LeRobot-style parquet, fleet logs), instead of being an HDF5-episode special case.
• Restores the windowed-features story the EgoDex blog post tells; the post's snippets map ~1:1 onto temporal.py now.

Semantics

Window semantics are pinned to the old NumPy behavior: fill_null(0.0) reproduces the 0-at-last-frame convention, and rows_between(-2, 2) shrinks at episode edges exactly like the old centered mean.

Tests

• New test_temporal_window_rates_match_numpy_reference: all six rates vs an independent NumPy reference at 1e-9 tolerance, on scrambled row order across two episodes (proves order_by recovers frame order).
• All existing query-behavior assertions (openness/twisting/text/combined hits) pass unchanged — the refactor is behavior-preserving end to end.
• make lint + ruff format --check clean; 5/5 in tests/test_egodex.py.

Breaking

The features table (and features_dir parquet from run_all) is now per-frame instead of one-row-per-episode; POSE_FEATURES_DTYPE and num_frames are gone. calibrate/query/pose_search signatures are unchanged.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a4ddb57cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve actual frame indexes when building segments

When the per-frame feature table has gaps or non-zero-based frame_index values (for example after filtering rows, downsampling, or using an external per-frame parquet), sorting by frame_index does not make mask position i equal frame i. The subsequent np.flatnonzero(mask)/segment stitching returns positional offsets instead of the original frame indexes, so reported segments point at the wrong video frames and combined text+pose queries can miss sampled frames whose index is beyond len(mask). Map matching positions back through the sorted frame_index values or enforce a dense frame index contract before querying.

Useful? React with 👍 / 👎.