Clean EgoDex scenario search workflow#46
Merged
Conversation
Read Apple's EgoDex through Daft's LeRobot v3 reader, precompute per-frame geometric states + Daft-window action rates and SigLIP-2 semantic embeddings, and serve a query UI that combines a geometric filter with semantic ranking. Dataset is CC-BY-NC-ND: scripts contain no EgoDex data; users convert their own licensed copy locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tures Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Avoids colliding with the real daft.datasets.hdf5 reader shipping in Daft soon, and makes the conversion example self-contained (imports the local copy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…df5File Add the thin facade the blog/notebook import (convert -> add_state_features -> add_skeleton_features -> embed_frames -> query) and a copy-paste egodex_demo.py showing pose-only, text-only, and combined pose x text queries. Switch egodex_lerobot.py to read HDF5 via Daft's native Hdf5File type (one row per episode) instead of the vendored hdf5_example reader. Add requirements-egodex.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… precompute Drop the NumPy grip predicates (is_writing_grip / is_hammer_grip / calibrate_grip_thresholds) from skeleton_features.py: grip detection now lives only in the Daft layer (egodex._is_writing_grip / _is_hammer_grip as @daft.func, with thresholds from egodex.calibrate at query time). skeleton_features stays pure NumPy geometry. Collapse run_pose_features.py onto the facade (read -> add_state_features -> add_skeleton_features -> write_parquet); it now writes continuous geometry only, no baked sc_* booleans. Refactor query_ui.py to source its predicate/thresholds/segments/text-encoder from the facade and read the continuous-only features, so the whole repo computes scenarios at query time from one implementation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build -> query walkthrough: read, add_state_features, add_skeleton_features, embed_frames, write; then calibrate, pose query, semantic text query, overlay, and launch the query_ui dashboard. Saved with cleared outputs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pass numpy frames directly to the SigLIP processor, run embeddings before pose features in the demo, and bump daft/gradio deps. Co-authored-by: Cursor <cursoragent@cursor.com>
Consolidate facade and feature modules under egodex_lib/, drop vendored readers and standalone run scripts in favor of PEP 723 demos, and update the README and query UI for the new layout.
… cleanup Switch HDF5 conversion to process_transforms via daft.File, add notebook download skip logic with .data caching, use nightly Daft in PEP 723 demos, and drop egodex_lib/__init__ in favor of direct submodule imports.
Replace the manual batch loop with from_files → process_egodex_episode → LeRobotDatasetWriter, fix the episode struct return dtype, and restore output-dir cleanup before writing.
…entual-Inc/daft-examples into shreyas/lerobot-pose-pipeline we need this# the commit.
Move HDF5 read + feature build into the writer UDF, switch write_lerobot to from_glob_path over .data, and tidy the demo notebook/script.
Switch the demo to read raw EgoDex HDF5 directly via read_egodex (native Hdf5File) instead of a LeRobot round-trip, and fix issues found while running it end to end: - _decode_sibling_mp4: strip Daft's file:// URI scheme before av.open - SiglipEmbedder.embed_image: guard empty batches (HF processor does images[0].device and errors on []) - overlay: glob with recursive=True so `**` matches nested dirs and episode_index ordering matches read_egodex Rewrite README for the egodex_lib/ layout and the raw-HDF5 pipeline, with a note on Daft laziness and the explode barrier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eline Reconcile with Everett's EgoDexLeRobotDatasetWriter commit. The two branches took different directions in convert_egodex_to_lerobot.py (LeRobot-writer vs raw-HDF5 read_egodex); keep the read_egodex pipeline, notebook, demo.py, and README from this branch. Everett's writer refactor stays reachable in history, and his egodex_video.py formatting is preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1480b1dd89
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+109
to
+110
| sums = np.convolve(values, kernel, mode="same") | ||
| counts = np.convolve(np.ones_like(values), kernel, mode="same") |
There was a problem hiding this comment.
When an episode has fewer than the 5-frame smoothing kernel, np.convolve(..., mode="same") returns the larger of the input and kernel lengths, so this method returns 5 samples for 1–4 frame episodes. That makes roll_L/R longer than num_frames, and pose-only twisting queries can report segments for frames that do not exist in the episode; trim the convolution result back to len(values) or handle short tracks explicitly.
Useful? React with 👍 / 👎.
…og-workflow # Conflicts: # examples/files/hdf5_func.py # examples/files/hdf5_udf.py
Add editable egodex/notebook extras, rename the pipeline module so notebook kernels can import the package from datasets/egodex, and replace the broken Daft embed_image path with a SigLIP batch UDF plus Daft-native pose search helpers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
egodexpackage over raw HDF5 episodesValidation
uv run --extra test pytest tests/test_egodex.py -quv run --extra lint ruff check datasets/egodex tests/test_egodex.py datasets/egodex/egodex_demo.ipynbgit diff --checkNotes
daft.functions.embed_imagewith post-normalization