feat(datasets): browse a local dataset's episodes in-app - #68
Conversation
A dataset on disk currently can't be viewed at all. The viewer is a deep-link to the hosted lerobot/visualize_dataset Space, which renders from a Hub repo path, so Landing sends local datasets to /upload instead — you have to push a dataset to the Hub before you can look at what you just recorded. /edit-dataset has been a routed placeholder. Fill it in: pick a dataset, page its episodes, watch every camera frame by frame, with a film strip and an aggregate joint-motion trace. Backend — episode_media.py reads the v3.0 layout (meta/episodes/*.parquet + videos/) directly and decodes with PyAV. It deliberately avoids LeRobotDataset so browsing never builds a robot config or imports torch. No new dependencies: PyAV, numpy, pyarrow and Pillow all arrive with lerobot[core_scripts], which is already required. Notes on two things that are easy to get wrong: - Episodes share an mp4. Locating one resolves the file *and* its from_/to_timestamp window inside it. /dataset-video is therefore addressed by chunk/file, not by episode, so the URL stays stable as you page between episodes: the player seeks instead of reloading, and playback survives the switch. An episode-addressed URL hands the browser a new identifier for the same bytes and silently drops playback on every click. - duration_s is length/fps, not to_timestamp - from_timestamp. An episode accepted early stops its parquet at accept while the camera ran to the configured limit, so the window over-reports the demonstration. The window is still exposed for the transport. Frontend — the Landing picker now opens local/both datasets in the browser rather than diverting to /upload; Hub-only datasets still go to the Space, since there's nothing here to decode. Upload stays reachable from the browse page. Tests build a real (tiny) v3.0 dataset on disk — a genuine PyAV-encoded mp4 plus parquet metadata — rather than mocking the readers: the point of the module is that it agrees with the on-disk format, which a mock can't check.
|
Nice work! Thumbnail index/png misalignment In Suggested fix: keep the index bound to its frame instead of relying on positional alignment. e.g. have Copyright header
Your two follow-up PR offers — both agreed
Everything else, path-traversal guard, chunk/file video addressing, the duration_s = length/fps rule, sync def routes keeping PyAV decode off the event loop, looks very good. |
extract_thumbnails dropped an undecodable frame with `continue` and returned a plain list of pngs, which handle_episode_thumbnails paired back to the requested indices positionally. A single mid-strip decode failure then shifted every later tile up one slot: a tile labelled "frame 8" showed a later frame's image, clicking it seeked the player somewhere the picture never showed, and the strip ended short of the last frame — all with success:true and valid PNGs. It surfaces on a corrupt/truncated recording, which is exactly when the strip is being scrutinised. Return (frame_index, png) pairs so the index travels with its image; a dropped frame is simply absent rather than a positional gap. Regression tests at the unit and route level inject a mid-strip decode failure and assert the survivors keep their true indices. Reported by @nicolas-rabault in review. Also joins the episode_media.py copyright with HuggingFace's, per review.
|
Sounds good! |
…owser # Conflicts: # CLAUDE.md
nicolas-rabault
left a comment
nicolas-rabault
left a comment
There was a problem hiding this comment.
Solid work,
I just Requesting changes on two coupled points, both on /dataset-frame.
| """One decoded frame as PNG bytes. Raises on any miss; the route maps to 404.""" | ||
| dataset_dir = episode_media.resolve_dataset_dir(repo_id) | ||
| location = episode_media.locate_episode_video(dataset_dir, episode_index, camera=camera) | ||
| return episode_media.extract_frame_png(location, frame_index, max_width=max_width) |
There was a problem hiding this comment.
extract_frame_png computes from_timestamp + frame_index/fps, and _decode_at returns the last decoded frame on overshoot. On a packed mp4 a frame index past the episode end returns the next episode's footage with a 200.
Repro against your own fixture (episodes=[(0, 12), (1, 8), (2, 15)]):
ep0 window=0.0..1.2
ep0 frame11 pixel=33 ep0 frame14 pixel=40 ep1 frame2 pixel=40
Frame 14 of ep0 is byte-identical to frame 2 of ep1. frame_index=10_000_000 also returns 200.
The viewer clamps to detail.length - 1 so the UI is safe, but the route is public. Suggest:
rows = episode_media.read_episode_index(dataset_dir)
row = next((r for r in rows if r.episode_idx == episode_index), None)
if row and row.length and not 0 <= frame_index < row.length:
raise episode_media.EpisodeNotFoundError(f"Frame {frame_index} out of range")Wants a test. test_frame_route_404s_on_unknown_episode only covers a bad episode index.
There was a problem hiding this comment.
Reproduced, addressed, added test. There are two checks now: the route rejects anything past the row's length, and extract_frame_png itself refuses targets outside the episode's video window, which also covers rows with no usable length where the route check never fires. Negative indices always 404. Your frame-14 repro is pinned in test_frame_route_404s_past_episode_end. While throwing bad inputs at the route I also hit two 500s (negative max_width, NUL byte in repo_id), fixed in the same commit.
| } | ||
|
|
||
| /** URL for a single decoded frame as a PNG. Goes into <img src>. */ | ||
| export function frameUrl( |
There was a problem hiding this comment.
No call sites. The viewer uses videoUrl and getThumbnails, and /dataset-frame is only hit by tests.
Either give it a use (poster frame while the video is paused is the obvious one) or drop the helper, the route and their tests. If it stays, the out-of-range issue needs fixing first.
There was a problem hiding this comment.
Sounds good, included the poster frame in this PR. Both viewer <video>s now use the episode's first frame as their poster, so the box shows the scene while the mp4 downloads instead of sitting black. One decode per episode+camera, browser-cached.
…owser # Conflicts: # lelab/server.py
…frame Out-of-range frame indices used to decode cleanly into the NEXT episode's footage on a packed mp4 and come back as a 200. Now bounded twice: the route checks the episode's length, and extract_frame_png itself refuses targets outside the episode's video window — which also covers rows with no usable length, where the route check never fires. Negative indices are rejected unconditionally. Pinned by test_frame_route_404s_past_episode_end, test_frame_route_404s_past_end_without_a_usable_length and test_extract_frame_png_refuses_frames_outside_the_window. frameUrl gets its call site: the viewer's video elements use the episode's first frame as their poster, so the box shows the scene while the mp4 downloads instead of sitting black. Two adjacent 500s found while fuzzing the route are fixed too: a non-positive max_width is a no-op instead of a PIL ValueError, and a NUL byte in repo_id maps to 404 instead of an unhandled ValueError out of Path.resolve(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
seekToFrame seeked to the frame's boundary timestamp; the timeupdate recompute (floor((t - fromTs) * fps)) then sat epsilon below the boundary and floored back a frame — ArrowRight moved +1 and snapped back, ArrowLeft jumped 2. Seeking to (frame + 0.5) / fps makes the floor immune to float error in either direction; the scrubber and film-strip clicks ride the same path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
The gap
A dataset that exists on disk can't currently be viewed in LeLab.
The viewer is a
window.opendeep-link to the hostedlerobot/visualize_datasetSpace, which renders from a Hub repo path.Landing.tsxencodes the consequence:handlePickExistingroutessource: "local" | "both"datasets to/uploadinstead. So the dataset you just recorded has to be pushed to the Hub before you can look at it — which is backwards, since checking it is usually why you'd decide whether to keep it./edit-datasethas been a routed 15-line placeholder ("This page is under construction") with no backend. This fills it in.What this adds
Pick a dataset, page through its episodes, and watch any camera frame by frame — with a film strip and an aggregate joint-motion trace, entirely against local files.
GET /dataset-episodesGET /dataset-episodeGET /dataset-frameGET /dataset-thumbnailsGET /dataset-motionGET /dataset-videoinline+ Range-capableThe Landing picker now opens local datasets in the browser instead of diverting to
/upload. Hub-only datasets still hand off to the Space — there's nothing local to decode. This changes existing behaviour, so it's the piece most worth your opinion; Upload stays reachable from the browse page.No new dependencies
PyAV, numpy, pyarrow and Pillow all arrive via
lerobot[core_scripts,feetech,training], whichpyproject.tomlalready requires (core_scripts→dataset→av-dep). Nothing was added.episode_media.pyreads the v3.0 layout (meta/episodes/*.parquet+videos/) directly and decodes in-process with PyAV. It deliberately avoidsLeRobotDataset, so browsing a dataset never constructs a robot config or imports torch — listing and previewing stay cheap.Two details that are easy to get wrong
Episodes share an mp4. Locating one means resolving the file and its
from_/to_timestampwindow inside it./dataset-videois therefore addressed by chunk/file rather than by episode, so the URL stays byte-identical as you page between episodes: the player seeks instead of reloading, and playback survives the switch. An episode-addressed URL hands the browser a new identifier for the same bytes, forcing a reload and silently dropping playback on every click. A test pins this.duration_sislength / fps, notto_timestamp - from_timestamp. An episode accepted early stops its parquet at accept while the camera ran to the configured limit, so the window over-reports the demonstration (a ~2.6s episode reads as the configured 10s). The window is still exposed for the transport.Testing
223 passed(187 existing + 36 new).ruff check,ruff format,tscandeslintare clean, andnpm run buildpasses.The new tests build a real (tiny) v3.0 dataset on disk — a genuine PyAV-encoded mp4 plus parquet metadata — rather than mocking the readers. The whole point of the module is that it agrees with the on-disk format, which a mock can't check. Covered: path traversal, camera/episode resolution, the packed and pre-packing layouts, the
duration_srule above, decode-lands-in-the-right-window,inline+ Range on the video route, and the shared-URL contract.Verified by hand against two datasets:
lerobot/svla_so101_pickplace(50 episodes, 30fps, camerasside/up) and an unrelated in-house one (20fps, three cameras, 7-DoF), so the format assumptions aren't tuned to one producer.frontend/distis intentionally not included —build_frontend.ymlrebuilds it on merge to main.Documentation
README.mdgains a Browse row;CLAUDE.mdgainsdatasets.pyandepisode_media.pyin the feature-module list, including the chunk/file addressing rationale. A stale comment inserver.py("Replay is rendered by the embeddedvisualize_datasetSpace") is corrected — it's a link-out, not an embed, and local datasets no longer go there.Unrelated and not touched: the README's ⏪ Replay — "Re-run any recorded episode" row looks stale.
ba0ddfband8a72410merged record/replay and dropped the/replay-datasetroute, so nothing re-runs an episode on hardware any more. Happy to fix in a separate PR if you agree.Open question
lelab --devhardcodes:8000/:8080with no override, which collides with anything already on:8000. I worked around it locally with a--portuvicorn and the?api=override. Worth a flag? Separate PR if so.Contributed by VibeCuisine. Draft — happy to adjust the Landing behaviour change, the route naming, or the scope before this goes green.