Repository navigation
Conversation
This was referenced Sep 24, 2026
Contributor
Author
This was referenced Sep 24, 2026
The pi0.5 converters stop doing rotation math. Actions now reach the algo
already normalized and already in their model-native layout, so to32_norm_6d /
from32_norm_6d are a pure rearrange into (and out of) openpi's 32-slot block
layout: RobotBimanual's native 20D [L xyz 6d g | R xyz 6d g] IS that layout and
only needs padding, HumanBimanual's 18D inserts a zero gripper slot per arm.
BaseActionConverter raises on both, so an unregistered embodiment fails loudly.
The four single-arm converters (RobotLeft/Right, HumanLeft/Right) are deleted
along with PI._extract_xyz, which existed for the cam-frame MSE. PI's loss
reduction and sample post-processing move into _reduce_loss and
_postprocess_sampled_actions.
The ypr->6D conversion is now a data transform. Eva gains cartesian_6d and
cartesian_wristframe_6d, Human the same pair, each 6D-encoding the action AND
the proprio ee_pose: normalized YPR saturates yaw/roll at +-pi (wraparound), so
per-dim normalization is only meaningful on the continuous representation.
get_revert_transform_list / the _build_*_revert_6d{,_wristframe}_ lists map a
prediction batch back to cam-frame ypr for the viz overlay.
Eva defaults to cartesian_wristframe_6d (eva, cotrain_pi_base, industry_eva_pi;
cotrain_pi_lang_wrist now composes the same as cotrain_pi_lang). Human data
configs move to cartesian_6d with pad_proprio_gripper, which pads the proprio
ee_pose 18 -> 20 (zero grip slots at 9/19) so the pi0.5 State: prompt bins line
up positionally with the robot 20D layout; UnpadGripperZeros inverts it in the
revert lists. HPT dims follow: eva 14 -> 20, human 12 -> proprio 20 / action 18.
With the 6D modes in place, a discontinuous rotation stops being a legal
training target. get_transform_list raises on the cartesian ypr modes and on
Eva's cartesian_wristframe_quat unless the caller passes an explicit
allow_legacy_rotation=True, which only the data/rollout boundary does (the
viz tutorial, rollout.py's legacy-checkpoint path, and the fixtures that use a
cartesian pipeline as scaffolding). test_data_configs_compose asserts that no
shipped data config selects such a mode or sets the opt-out, so the guard
cannot be bypassed from a config. The five *_pickplace_qwen* data configs were
the only ones still on Euler rotations; they and the two hpt_bc_pickplace_qwen
model configs paired with them are deleted. The keypoint modes keep their ypr
and quat variants for now -- the keypoints_*_6d modes do not exist yet.
Also here, because they are the same widening: SplitKeys rejects a layout whose
widths do not sum to the input (an evaluator revert list built for 12-dim ypr
fed an 18/20-dim 6D batch used to slice silently), and vendor-tagged human
embodiment names (MECKA_/SCALE_/ARIA_/LIGHTWHEEL_*) alias onto HUMAN_* through
canonical_embodiment_name, so pins and embodiment checks accept the legacy
metadata. Robot (EVA_*) names are never aliased.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
… zarrs compute_hand_pose_xyzquat built the palm normal as cross(thumb_dir, pinky_dir). Thumb and pinky are anatomically swapped between hands, so that cross product already mirrors chirality for free; applying rot_left on top of it flipped x/y a second time and landed the LEFT hand on the RIGHT hand's spatial convention. Both hands now use the same rot_right relabel. Reconverting every affected zarr would be the obvious fix, so this avoids it: rot_left == rot_right @ diag(-1, -1, 1), which makes right-multiplying the stored left pose by Rz(180 deg) exactly equivalent to reconverting with rot_right. RotateLocalFrame does that right-multiply (R_new = R_old @ R_fix), relabelling the pose's own axes without moving its origin, on (7,) poses and (T, 7) chunks alike, and leaves zero-norm quaternions alone because those mark padded frames. Human.get_transform_list(fix_left_wrist_convention=True) prepends it to the raw LEFT pose keys before any frame math: ee_pose in the cartesian modes, wrist_pose in the keypoints modes, since the converter wrote the same rotation into both. The flag is a property of the DATA, not of the vendor: measured against the production set (1833 of 1838 episodes probed, 2026-09-18), every mecka episode AND all 20 scale episodes still carry the double-mirrored left wrist, with the right-hand control clean, so data/mecka.yaml and data/scale.yaml both opt in. Data reconverted after this commit must not, or it double-flips. Because that correctness lives in the data, left_wrist_convention() re-derives the verdict from an episode's own keypoints -- recomputing both wrists under each relabel and comparing with the stored quaternions, with the RIGHT hand as the control -- and S3EpisodeResolver samples a few resolved episodes at startup. Only the double-flip raises: the flag on against post-fix data would correct what the converter already corrected, which is a fresh error from a stale setting. The flag off against pre-fix data is what every run did before this commit, and the generic human configs (aria, human, the cotrains) have never set it, so that warns and names the knob rather than turning their startup into a crash. Five legacy 2025 episodes match neither convention (a third converter wrote them); their control fails, so they warn too. data/mecka_scale_cotrain_pi.yaml and data/industry_eva_pi.yaml are deleted: both merge mecka and scale into ONE human_bimanual dataset under a single transform_list, and the correction is per-lab, so neither file has a correct value for the flag. model/pi0.5_cotrain_mecka_scale.yaml goes with them. test_wrist6d_roundtrip runs the real transform lists on synthetic world-frame poses, parametrized over the flag, and compares every stage against plain numpy/scipy SE(3) math that shares no code with the pipeline; the convention checker is tested the same way, on synthetic keypoints. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…an default
Human data trains on hand keypoints instead of a palm-origin cartesian pose.
The action is 144-D wrist-first, per hand
[wrist xyz (3) | wrist rot6d (6) | 21 MANO keypoints (63)] x {left, right},
built by the keypoints_{wrist,head}frame_6d modes: the existing ypr keypoint
builders produce the 138-D layout and KeypointsYPRToRot6D re-encodes just the
wrist rotation, leaving the keypoint blocks untouched. Both the action and the
observations.state.keypoints proprio are converted, so neither carries a
wrapping euler angle. MANO keypoint 0 IS the wrist, so per hand the wrist
position rides at both ch 0-2 and ch 9-11; that is kept deliberately (21
keypoints is the MANO interface every consumer reshapes to) and written down
in the bimanual_keypoint_layout docstring.
include_ee_pose additionally builds the cartesian head-frame ee_pose next to
the keypoint action (keymap + transform list), so a keypoint pi0.5 run
discretizes the same State: prompt block as the cartesian runs and differs
from them only in the action space. The cartesian builder runs with
delete_target_world=False to keep obs_head_pose alive for the keypoint
builder, and its actions_cartesian by-product is dropped.
144 does not fit openpi's action vector, whose PyTorch port hard-codes 32-wide
action projections regardless of Pi0Config.action_dim. PI now reads action_dim
and, when it is not 32, swaps in action_dim-wide action_in_proj /
action_out_proj AFTER the strict base-weight load: only those two layers start
fresh, everything else comes from the checkpoint, and sample_actions already
draws noise at config.action_dim. pad_to_width zero-pads narrower embodiments
(eva's 20-D) into the same vector and raises rather than truncating a wider
one. _reduce_loss then restricts the mean to the slots the converter actually
packs, so eva does not average its loss over 112 zero-padded dims and
contribute a fraction of the gradient signal it had at width 32.
HumanBimanualKeypoints is identity-in, slice-out.
HPT human configs move to a 144-D state_keypoints stem. hpt_bc_flow_human
holds the body -- there is no vendor notion at the embodiment level, so the
canonical config is the embodiment one -- and hpt_bc_flow_{mecka,aria,scale}
are two-line aliases of it. HPTModel.add_stem drops null stem entries so a
child config can delete a stem its base declares, which hydra merges cannot
express, and hpt_cotrain_flow_seperate_head uses it to drop state_ee_pose.
With the keypoints_*_6d modes in place, the guard from the previous commit
widens to the keypoints ypr and quat modes, so every remaining discontinuous
rotation target needs an explicit allow_legacy_rotation=True (the aria->MANO
and mecka keypoint viz tutorials, which read pre-6D data). The cartesian
configs that were still pinned to the ypr widths move with it:
hpt_bc_flow_human_cartesian and the cotrain base to human 20/18, the
shared-head cotrains to a 20-wide shared action.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Wrist-relative actions made a latent division safe-looking: the t=0 cell of a wrist-frame chunk is exactly the identity pose, so its quantile range is 0 and the historical `(q99 - q1) + 1e-6` denominator scaled any off-convention value by 1e6. A channel whose range is below NORM_MIN_RANGE (1e-4) is now treated as constant -- it normalizes to 0 and unnormalizes to its centre rather than dividing by epsilon. Diffusion Policy / robomimic ignore ranges under 1e-4 and GR00T maps min == max to 0; this is the same rule, applied in the shared _apply_norm_one / _apply_unnorm_one so the dataset and the tests agree. norm_stats.pool_horizon (default false, so nothing changes unless asked) computes one set of stats per channel over every timestep of the chunk, the way openpi does, instead of one per (timestep, channel) cell. The pooled stats are tiled back to the (T, D) shape, so normalize, the bounds check and checkpoints see the same layout either way. It is part of the norm-stat cache key, so flipping it cannot silently reuse stats computed the other way. precompute_norm_stats.py computes and caches the stats for a recipe outside a training run, keyed by the same file + content hash. Which of the two is better is unsettled: the literature is split (LBM per-timestep with a stated rationale, openpi / RDT / GR00T / Diffusion Policy / UMI / lerobot / Octo pooled) and nothing here has compared them on a real recipe, so the default stays the inherited per-timestep one and the config comment says what would make it worth flipping. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
A 6D rotation is two columns of a rotation matrix: already bounded in [-1, 1], and per-channel centre-and-scale pulls the pair off the manifold without buying any resolution. The two projects that anchor an action chunk at near-identity the way this repo does -- TRI's LBM and UMI -- both exclude rotation from normalization for exactly that reason (LBM: "We avoid this normalization procedure for the 6D rotation in the poses to avoid corrupting the rotation matrix"; UMI sends its rotation channels through an identity normalizer). Per timestep AND normalized was the one combination with no precedent, and it is what manufactures the t=0 zero-range rotation cells NORM_MIN_RANGE exists to catch -- this removes them at the source. rot6d_channels() reads the indices off the existing layout tables, so the rule follows the layout rather than a second hardcoded list: the cartesian 18/20 widths and the 144-D keypoint layout, action and proprio alike. The ypr widths keep the old behaviour; an Euler angle is not a matrix column and those modes are legacy-only now. _apply_norm_one / _apply_unnorm_one take the channels and pass them through untouched, so the dataset path, the evaluator revert and the rollout revert all follow from one place, and the degenerate-channel guard is left to do what it is actually good at: the structural kp0 / kp9 position zeros. Norm-stat files stay valid -- what changes is how they are applied -- but a run before this commit and a run after it are not comparable, and the loss is now weighted differently between translation (unit scale) and rotation (raw). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Three independent ways MultiDataset could fail a whole run on data that was
fine, or on data it should have rejected up front.
Bounds check. Quantile bounds on a rotation channel are meaningless: euler ypr
wraps at +-pi and rot6d columns are matrix entries, so neither respects a
quantile interval. _bounds_check_channels uses the layout tables to check only
translation, gripper and keypoint channels; unrecognized widths still get the
full-vector check, and the NaN/Inf check always covers everything. The bounds
themselves get slack -- 1e-6 absolute plus half of each cell's own quantile
range -- because the check is any-cell over ~13 200 cells per keypoint chunk.
Measured on the 2815-frame flagship episode through the real
keypoints_wristframe_6d pipeline (zscore, stats from sample_frac 0.10 over its
own frames, 403 probe frames, any-key rejection): 39.5 % of frames rejected at
slack 0, 17.9 % at 0.05, 11.4 % at 0.1, 5.2 % at 0.25, 3.2 % at 0.5, 1.0 % at
1.0. The 132-channel action chunk drives all of it. Half a range is the knee,
not a cushion -- though the absolute numbers move with which frames the stats
were drawn from, so the comment says how to reproduce them.
Corrupt values sit orders of magnitude outside the range and are still caught.
Cells the normalizer treats as constant are exempt, since a tiny offset there
cannot blow up the loss -- and which stat decides that is norm_mode's business
(std under zscore, the quantile range under quantile), so the exemption asks
_degenerate instead of hardcoding q99 - q1; every shipped config is zscore, and
a channel that is 0 in 99.6 % of samples and 1.0 in the rest has q99 - q1 = 0
but std = 0.063, i.e. it is still divided by and must still be checked.
quantile_0_01 / quantile_99_99 now snap to observed samples ("lower" /
"higher") instead of interpolating: under 10k samples the linear percentile
lands strictly inside the sample range, so the extreme frames of the very
episodes the stats came from were rejected (3-16 % of a single episode in the
overfit runs). A stats/sample shape mismatch is logged once per key instead of
silently disabling the check.
Sampler. Retries stayed inside the failing episode until they exhausted it,
so one wholly-bad episode killed the run and NCCL-timed-out every other rank.
Retries now widen to the global index space after 25 attempts, and 100
consecutive failures raise. The cap is a systemic-failure detector, not a
tolerance dial: attempts resets per __getitem__, so with p = the bad fraction
of the whole index space it fires with probability p**(N - 25), a step function
in p. At N = 100 the firing band sits at p >~ 0.8, so everything it can reach
is systemic; at N = 1000 it needed ~99 % of the store to be bad and got there
through ~975 sequential zarr/S3 reads per call per worker, which reads as a
hang rather than a failure. Retry logging was one WARNING per attempt -- tens
of thousands of lines per run -- now the first 20 per worker and then one in
500.
Annotation cutoff. Many Scale-"completed" episodes have no usable annotation
span because the injection step lagged, which makes the cutoff a silent no-op
on exactly the runs whose point is to clamp chunks at annotation boundaries.
S3AnnotationCutoffEpisodeResolver(require_annotations=True) drops them, logs
the count it dropped and the count it kept, and raises if nothing is left. The
per-episode span line drops to DEBUG: one line per episode per rank is
thousands of lines on a real split, and the resolver's aggregate says the same
thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Metrics were paired MSE over whatever the action vector happened to be, which mixes metres and rotation-matrix entries into one number that means nothing in either unit. action_metrics dispatches on the layout instead: cartesian widths report xyz MSE (m^2), geodesic rotation error in degrees (after Gram-Schmidt, so it survives a non-orthonormal prediction), raw rot6d MSE as a training-tracking diagnostic, DTW and Frechet; keypoint widths report keypoint L2 in metres, wrist xyz / rotation error and keypoint DTW. Each also splits early / mid / late (0-10 / 10-50 / 50-100 % of the chunk), because a chunk model can be right at t=0 and drift, and one average hides that. DTW forgives temporal misalignment that paired MSE punishes: a correct trajectory executed slightly early scores near zero. These are all frame-invariant -- the cam-frame revert is a rigid transform applied to prediction and ground truth alike -- so the revert no longer runs for metrics at all, only when rendering the overlay. That also removes the cam-frame MSE, which was the same number as the native one. Rendering is CPU-bound at roughly a second a frame and dominated viz-epoch wall time (unbounded rendering added ~2h per viz epoch on the combined pretrain), so viz_every_n_epochs gates it on Lightning's (current_epoch + 1) convention, the one validation actually runs on, with the final epoch always rendering so short runs get a video; viz_max_batches caps frames per rank. Metrics still run on every batch of every pass. Video fps is divided by world_size, since distributed validation deals indices round-robin and rank 0's consecutive frames are world_size source frames apart -- they were being written as a world_size-times timelapse. train_viz adds a second val loader over the train split (metrics train_viz/, videos videos_train_viz/), on by default for training runs with an evaluator, so train and val curves come from the same evaluator on the same axes. A third unseen_op_valid loader and data.valid_prefix let the held-out-operator splits log seen_op_valid/ and unseen_op_valid/ separately, instead of the seen operators landing in the train_viz slot and the unseen ones in Valid/. ModelWrapper dispatches on val_loader_names by dataloader_idx. val_at_start runs a pre-fit baseline validation, skipped on resumes because trainer.validate there would score the un-resumed base model. Reverse KL is removed: M forward passes per val batch for a number nothing was read off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fold recipes this stack was built for: mecka_all_6d, mecka_fold_freeform_6d
and its opsplit variant (wrist frame; cam frame is a transform_list.mode
override rather than a near-duplicate config, and test_data_configs_compose
rejects any data config that differs from its parent only in that mode),
mecka_fold_flagship_opsplit_hpt_6d, mecka_fold_flagship_topop_hpt_6d and
mecka_fold_freeform_opsplit_hpt_keypoints, plus the two 300M HPT models and
pi0.5_bc_mecka_6d.
Defaults, each for a failure seen in a run:
- checkpoints every 50 epochs, saved on train-epoch end. With
check_val_every_n_epoch > 1 the validation-end hook only fires on val epochs,
so every_n_epochs silently skipped the others.
- ddp_pi inference_mode: false. The compiled pi sampler does in-place setup on
its static buffers during cudagraph capture, which is illegal on inference
tensors ("Inplace update to inference tensor outside InferenceMode"); no_grad
is equivalent for eval and creates none.
- ddp_pi validates every 20 epochs instead of 200, and eval_hpt renders viz
every 200.
- pi0.5 gets a constant LR after warmup. The cosine schedule decayed to 0 over
num_training_steps=60000, so runs past ~60k steps trained at ~0 lr and
shorter runs never saw the configured lr for most of training.
- The Qwen stem resolves a repo id to its cached snapshot directory when
offline: transformers >= 4.57 probes the Hub while loading a tokenizer by
repo id even under HF_HUB_OFFLINE, which raises on compute nodes with no
internet. Loading from the directory skips the probe. Offline is read the
way huggingface_hub reads it -- 1 / ON / YES / TRUE in any case, from the
live environment -- rather than by matching the literal "1", which left
HF_HUB_OFFLINE=true hitting the probe the helper exists to avoid.
viz_mecka_episode.py and viz_eva_episode.py render an episode straight from
its zarr, which is how the left-wrist convention bug was found. Both take the
dataset dir from egomimic.utils.paths.default_dataset_dir(), which reads
paths/default.yaml, instead of each re-stating the cluster path.
HumanBimanualCartesianEuler / RobotBimanualCartesianEuler are renamed to
HumanBimanualCartesian6D / RobotBimanualCartesian6D: both pack xyz + rot6d and
have not touched an Euler angle since the 6D conversion, and this is the commit
that makes the human one the converter for a new default recipe.
Also prunes unit tests whose assertions another test already makes on the same
code path, and parametrized cases that repeat a stricter test's compose call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…evert its outputs NOT TESTED ON HARDWARE. The rollout hard-coded cartesian_wristframe_ypr inputs and used the policy's predictions as-is. Now that eva checkpoints can be trained in several modes (cartesian, cartesian_6d, cartesian_wristframe_6d), that silently feeds a model the wrong inputs and sends the wrong frame to the arms. It now reads data.train_datasets.<name>.resolver.transform_list.mode off the checkpoint's own config, builds the matching input pipeline, and maps predictions back to camera-frame xyz+ypr+gripper through Eva.get_revert_transform_list. The revert needs the un-normalized proprio, which process_batch_for_training normalizes in place, so the raw ee_pose is cloned before that call. A checkpoint whose config records no mode keeps exactly the old behaviour -- legacy inputs, no revert -- and says so loudly, so this cannot silently change what an existing checkpoint does. test_eva_revert_for_rollout_recovers_camframe_ypr covers the revert math by cross-checking the wrist-frame revert against the cam-frame cartesian_6d one, which never enters the wrist frame -- so the frame composition is genuinely validated, though not against independently written SE(3) math. The hardware path itself still needs a real rollout. Both get_transform_list calls here pass allow_legacy_rotation=True: this is inference, and a checkpoint that predates the 6D conversion records exactly the modes the training-time guard rejects. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
The unshuffled val loaders were truncated by limit_val_batches (80 x 64 x W frames) to their leading episode-hash-sorted slice, so at W = 1 the seen-op head scored 25 of 70 episodes and 6 of 11 operators, and the unseen head 33 of 191 episodes with an 81/19 operator mix on a 52/48 split (lane-0 diagnostic, 2026-09-15, measured single-rank). The loaders are unshuffled only so the overlay video is coherent, and the video renders just the first viz_max_batches (10) batches anyway. Split each val head in two: * metric loader = EvenStrideDataset(split, frames_per_episode=K) with K from data.metric_frames_per_episode[head], so every episode -- and so every operator -- is scored, at the same 80 batches and with no rendering; * video loader = <head>_video, the head's split restricted to the episode hashes pinned in data.video_episodes[head] (one per operator), contiguous and unsubsampled, logging no metrics. The pinned episodes are taken out of the already-resolved split, so resolve-once still holds. EvalVideo.on_validation_step gains a mode argument (metrics / video / both); ModelWrapper.validation_step routes <head>_video to the same evaluator in video mode and skips it entirely on a non-viz epoch, before the forward pass. Heads with no pins are called exactly as before, so a data config that declares neither key behaves as today. K = floor(5120 / episodes-in-split), from the measured splits: opsplit valid 73 (70 eps), train_viz 3 (1346 eps), unseen_op_valid 26 (191 eps); topop valid 256 (20 eps), train_viz 13 (385 eps). Those are SINGLE-RANK numbers and trainHydra scales them by trainer.devices x num_nodes, because DistributedSampler strides rather than chunks: the W ranks between them walk the whole split, so the pre-split window was already 80 x 64 x W frames wide. Without the scaling the shipped 4-GPU ddp_pi run would score 5,110 frames of the seen-val head where it previously scored all 13,744 -- a regression on the one head small enough to score whole. EvenStrideDataset keeps an episode entire when K exceeds its length, so the scaling cannot overrun a short split. Expect a shift in the per-batch Frechet _min/_max curves: they now span many episodes instead of ~2 s of one, so they are not comparable with older runs. Eval.on_validation_step, the abstract base, takes the mode kwarg too: the metrics/video path passes it unconditionally, so an evaluator that predates the split would have raised TypeError at the first val step of any config that declares video_episodes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…split Three-operator twin of mecka_fold_flagship_opsplit_hpt_6d (and the wider sibling of the single-operator topop config): train on the 3 largest flagship fold-clothes operators (1,079 eps / 206,276 frames, 65.3 % of flagship), none of them held out. Seen-op val is the seeded 5 % episode split within those operators (53 eps / 9,826 frames); the unseen-operator head is inherited unchanged, so the runs stay comparable. Lane A keys measured from the real split on a compute node 2026-09-15: metric_frames_per_episode valid=floor(5120/53)=96, train_viz=floor(5120/1026)=4 (single-rank values; trainHydra scales them by the world size) (unseen 26 inherited); video_episodes pins one seen-val episode per train operator (191+223+226 = 640 f = exactly the 10-batch video budget) and inherits the train_viz and unseen pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…only grad-norm MAD Validating one checkpoint twice now gives identical prompts, actions and metrics. * HPT gets a per-pass RNG seed, base + rank * stride + pass counter, where the counter advances once per forward_eval call and is rewound by ModelWrapper.on_validation_start. forward_eval builds a torch.Generator from it and threads it down HPTModel.forward -> DenoisingPolicy.forward -> sample_action, and forks/seeds the global RNG around the BC val-loss call (DenoisingPolicy.predict draws its own noise and timestep, out of reach of any generator kwarg). Every generator kwarg defaults to None, so training and all other callers are untouched. The val-loss seed is offset by EVAL_LOSS_STRIDE rather than by 1: a torch.Generator seeded with S draws the same stream as a global manual_seed(S), so +1 would have handed batch i's val loss exactly the noise batch i+1 samples with. eval_base_seed is a real __init__ kwarg, so a second eval pass can be made to draw different noise on purpose. * _build_prompts picks annotations with that same seeded RNG whenever the algo is in eval mode, regardless of annotation_sampling_mode, so eval keeps prompt variety instead of collapsing to sample[0]. Train mode is unchanged. * The MAD grad-norm spike detector in on_after_backward is now log-only: it still logs the raw norm, the threshold, the flag and the [GRAD_NORM_SPIKE] line, but never rescales gradients, and every step (flagged or not) goes into the rolling median/MAD window. Fixed clipping moves to Lightning: trainer/default.yaml gains gradient_clip_val (null for now) and gradient_clip_algorithm: norm. The on_before_optimizer_step hook goes with it: Precision._after_closure calls that hook BEFORE _clip_gradients, so "policy_grad_norms_clipped" was the pre-clip norm under another name -- bit-identical to policy_grad_norms_raw, and no more informative once gradient_clip_val is set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
torchvision draws one random parameter set per call, so applying the configured train_image_augs to the stacked (B, 3, H, W) camera batch gave every image in the batch the same jitter. Wrap train_image_augs in a new PerSampleAugs module so each image draws its own parameters, and move the inline aug block of _robomimic_to_hpt_data into _apply_image_augs with the same gating. Deterministic eval augs stay batched. _apply_image_augs also checks train_image_augs before calling it: __init__ accepts None and train mode would have raised TypeError. The loop is not free: measured on an A40 at batch 64 with the shipped ColorJitter + Normalize, 13.8 -> 92.2 ms at 224x224 and 62.2 -> 105.5 ms at the camera's 360x640, per camera per training step. Correctness first here; the cost is removed further up the stack, where the jitter itself is vectorised over the batch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
… LR group Two defects that would confound any backbone comparison: * a frozen `ResNet` backbone kept its BatchNorm layers in train mode, so it normalized with batch statistics and kept updating its running stats. `requires_grad = False` does not touch either. `train()` now puts a frozen backbone in eval mode. Latent today -- no shipped config freezes the ResNet -- so no run to date was affected. * the pretrained backbone and the randomly initialised trunk shared one LR. The new `model.backbone_lr_scale` key gives the encoders' backbones (`backbone_parameters()`) their own param group at `lr * scale`; at the default 1.0 the optimizer is built exactly as before, as a single group over `trainer.model.parameters()`. Nothing shipped sets it to anything else, so the two-group path is exercised only by its tests for now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
ZarrDataset emits a real action_pad_mask (1.0 per real frame, 0.0 per repeat-last pad, honouring the annotation-cutoff EOS clamp) next to every horizoned action chunk. A new InterpolatePadMask transform carries it through the 30 -> 100 upsampling on the interpolators' own grid: an output step is real only if both source frames it interpolates between are real. HPT gains a use_pad_mask algo key (default false, declared in the two flagship model yamls; every other HPT config takes the __init__ default). With it on, process_batch_for_training builds pad_mask (B, S, 1) from the dataset mask instead of ones; DenoisingPolicy.loss_fn takes the mask and means the squared error over the valid steps only. With it off nothing changes: the mask is dropped from the batch and pad_mask stays all ones, which is exactly F.mse_loss. It ships off to match openpi, which asks LeRobotDataset for action_is_pad and then ignores it (Pi0.compute_loss means over the whole horizon). LeRobot's own policies do mask, but reduce with .mean() over all elements including the zeroed ones, so their loss shrinks as padding grows; this divides by the valid count instead, which keeps the gradient magnitude constant. Measured on the three pinned flagship episodes, 6.4-7.6 % of supervised action steps are repeat-last padding and 12.8-15.2 % of samples contain some, so turning it on changes the loss: a run before and a run after are not comparable, and the layout-aware eval metrics would have to be masked in the same change or train and val would be measuring different things. Padding is still repeat-last and no tail samples are dropped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…mbedding) The proprio keys the model consumes read K steps instead of one. Every change is gated on K > 1 (data) or history_len > 1 (model), and K = 1 / stride 1 is bit-for-bit today's pipeline. Data: Human.get_keymap gains proprio_history and history_stride; for K > 1 it marks the proprio entries with "history": K plus the stride, and ZarrDataset.__getitem__ reads the window idx - s*(K-1) ... idx in steps of s, clamped at the episode start and front-padded by repeating the first real frame, so the key is always (K, D) with the current frame last. A proprio_history_mask (1 = real, 0 = front padding) travels with it. The stride is why this ships on rather than at K = 1. Measured on a real flagship episode (30 fps, fold_clothes), the median wrist translation is 6.8 mm over 1 frame, 25.7 mm over 3, 42.9 mm over 5 and 80.8 mm over 10: an unstrided K = 3 window is 67 ms of past, close enough to a duplicate of the current pose to be mostly sensor noise, while the action chunk it conditions spans a full second. The flagship recipe therefore ships proprio_history: 3 / history_stride: 5 -- 0.33 s of past -- with the stem at history_len: 3. That changes the observation, so runs before and after this commit are not comparable, and history_dropout: 0.2 stops being inert: it is now a real hyperparameter, and the mitigation for the copycat/causal-confusion risk that proprio history classically brings. Transforms: the cartesian builders already treat a leading axis as batch; ActionChunkCoordinateFrameTransform / PoseCoordinateFrameTransform gain an opt-in target_history so a proprio target frame collapses to the CURRENT step (the action chunk must not move when history is added), and Reshape / QuaternionPoseToYPR tolerate the extra axis. observations.state.ee_pose is (K, 20). Eval: every human revert transform list starts with a SelectCurrentStep on its proprio key, so the val metrics, the cam-frame revert and the overlay video see the single current pose they always saw. No-op at K = 1 and idempotent. Norm stats are the baseline's: per-channel (D,) stats broadcast over (K, D), and both the training and the precompute stats pass force proprio_history=1 so K > 1 and K = 1 runs share one set. Model: MLPPolicyStem gains history_len / history_dropout. For history_len > 1 it owns a zero-init learned time embedding added after the projection, and HPT.stem_process skips the input-side sinusoid for such a stem (a constant [0, 1, 0, 1, ...] on normalized values is amplitude-1 noise on [-1, 1] data). history_dropout replaces the whole past by the current step per sample in training, exactly what an episode-start sample looks like after front padding. The K = 1 baseline stays one override away (proprio_history=1, history_stride=1, history_len=1) and the tests pin that it is bit-identical. Every other shipped config is untouched at K = 1, including the keypoint arm, whose wrist-frame proprio would need per-step framing before it can carry history. A compose test pins every train_*.yaml recipe's data K against its stem's history_len, because a recipe that inherits one model's data config and overrides the model is exactly how the two drift apart, and nothing else composes both halves. _stem_history_len takes the domain rather than matching a stem name by suffix: a cotrain run registers one state_ee_pose stem per domain, and a suffix match returns whichever comes first, so the human side was checked against eva's history_len. Latent while everything shipped K = 1; a blocker now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
… padding `HPTModel.finalize_modules` ran `self.apply(self._init_weights)` over the whole model, which xavier-randomised the pretrained HF encoders inside the text stems (the frozen Qwen3-Embedding weights were overwritten in place and their biases zeroed). Modules that own pretrained weights now declare them through the `PretrainedWeights` mixin, and the init pass walks the tree without ever entering such a subtree, so the ResNet backbone, `T5Encoder` and any future pretrained stem are covered by the same mechanism. `HPT.__init__` also called `finalize_modules` twice; the first call is gone. Once the model is built, moved and upcast, `verify_pretrained_weights` hashes every declared pretrained submodule and compares it with the checkpoint read straight off the local HF snapshot, failing the run on a mismatch. It costs a second full copy of the encoder (~1.1 GB fp16 at 0.6B) and ~3 s of hashing per rank, so `verify_pretrained: false` turns it off; it is on by default, because a silently re-initialised encoder trains to a plausible wrong result. Only stems that own an HF snapshot are checked -- `ResNet` returns no reference, so it is protected from the init pass but not hash-verified, which the mixin docstring now says. Finally, `QwenPerTokenEncoder.compute_latent` now passes the tokenizer attention mask into its cross-attention. Padded positions are zeroed before the biased `proj`, so they were the learned constant `proj.bias`, not zero, and got real attention weight: text-conditioned results from before this commit are not comparable with results after it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
The annotation vocabulary is tiny next to the number of training steps, so a frozen text encoder re-runs the same 600M-parameter forward every step. `_Qwen3BaseEncoder` now keeps a lazy in-memory cache of the per-token hidden states keyed by the exact prompt string: misses go through the tokenizer + encoder in ONE sub-batch of unique strings, their un-padded (L_i, D) rows are stored detached, and the batch is re-assembled with the same LEFT padding the tokenizer produces, so `forward_with_mask` / the pooled path are unchanged. Only legal when the encoder is frozen - a trainable encoder must stay online. Knobs on the stem: `cache_text_features` (default true) and `text_cache_max_entries` (default 8192, LRU); both added to the model yamls with a Qwen stem. The cap counts entries, and an entry is the prompt's un-padded (L, D) states -- 4-25 KB for real annotations, 262 KB at max_length 128 -- so the default is ~100-200 MB of device memory in practice and 2 GB in the worst case; the docstring says so next to the knob. A batch carrying more unique prompts than the cap reads its rows from the miss pass rather than from the cache, because the LRU trim runs first and would otherwise evict entries the very next line looks up. `_apply` (to/cuda/float) clears the cache, and `clear_text_feature_cache()` / `text_feature_cache_stats()` are exposed. Padded positions come back as zeros instead of the encoder's pad-token output; every consumer masks them out. Cached-vs-online on 50 real prompts (fp16, each warmed alone: worst-case padding mismatch): max |diff| 3.6e-3 on the latents and 9.4e-2 on raw hidden states, exactly equal to the online path's own batch-padding noise (9.4e-2); in fp32 the same comparison is 1.1e-6 / 1.6e-4. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
RDT(HPT) swaps the policy network: no Perceiver pooling; stems hand full token sequences to the head; the DiT blocks live in the trunk slot, shared across embodiments, and cross-attend to language / image tokens in alternating blocks; each FMPolicy head holds one embodiment's action adaptor, position table and output layer. DINOv3 ViT-B/L (fine-tuned at 0.1x) + frozen Qwen3-Embedding, 2-frame image history, ctrl-freq token, per-stream condition dropout. Configs: model/rdt_*; train_zarr_mecka_flagship_6d_rdt. The (past, current) pair goes through the train augs as (B * 2, 3, H, W) with frames=2: one jitter draw per sample covers both frames, and every op runs per frame. Laid side by side instead, ColorJitter's contrast took one mean over both frames (the past frame shifted by up to 0.20 with the current frame's content) and a Resize or CenterCrop would have mixed pixels across the seam. The pair also gets HPT's per-sample absent-camera guard, since both go through _apply_image_augs. compile.enabled=true leaves the trunk off HPT's compile list (no_trunk), but the DiT is still compiled: every head's denoiser calls it, so dynamo traces its blocks into the head's graph (asserted in test_rdt with a capturing backend). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019DbPWJCKtVg5c2PNHs7egw
A cancelled DDP job left GPUs allocated with nvidia-smi blaming PIDs absent from /proc. The driver is fine: the memory is held by live processes reparented to init. gpu19 had 126 pt_data_worker survivors across three cancellations, each holding /dev/nvidia0-7, pinning 7 GPUs and ~1.7 TB of RAM. Workers fork after CUDA init, so they inherit the rank's device fds, and ranks open every visible GPU -- one survivor pins the node. A worker also inherits both ends of its result pipe, so a dead rank never sends EPIPE; it blocks in pipe_write and never reaches PyTorch's ppid check. Whole ranks leak the same way: one wedged in torch.cuda.synchronize() inside inductor cudagraph_trees ignores SIGTERM until Slurm gives up (UnkillableStepTimeout) and drops the job with its ranks still spinning. ProctrackType=proctrack/linuxproc walks the PPID tree, so it never sweeps anything reparented to init. egomimic/utils/gpu_orphans.py: * orphan_guarded/die_with_parent put PR_SET_PDEATHSIG on every worker, so the kernel kills it with its rank whatever it is blocked on. Wired into the train and val loaders and the norm-stats pass. * arm_rank_pdeathsig does the same for the rank process. A process whose parent is already gone when it arms exits 1, after one line on stderr saying why (os._exit skips logging's flush). SLURM_JOB_ID is also set inside a salloc shell, so a run started there with nohup is left unarmed (SIGHUP ignored): armed, it died with the shell it was meant to outlive. * reap_orphans runs from trainHydra.main before CUDA init and clears what an earlier job left, so a leak cannot outlive the job that made it. It only signals processes that are ours, reparented to init and holding an nvidia fd. Slurm-gated; EGOMIMIC_REAP_GPU_ORPHANS=0 (or false/no) disables it, any truthy spelling keeps it on, and anything else raises. reap_gpu_orphans.sh is the manual version for a node we are not submitting to. It reads PPid from /proc/<pid>/status, as the module does: in /proc/<pid>/stat, field 2 is comm, and a comm with spaces shifted field 4 far enough to read a live process's PPid as 1. The PDEATHSIG test waits for the unguarded worker to block on its pipe, which Linux >= 5.5 reports as pipe_write and 5.4 (sky2) as pipe_wait. It accepts either and skips on anything else, killing the worker first: failing there used to leave it running under init. The kill-switch test never scans the real /proc. The cluster-level fix is ProctrackType=proctrack/cgroup, which needs admin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DF1j2AzwzFztRsqvbTC35b
…e-rank trainer/ddp.yaml and ddp_pi.yaml set devices to gpus_per_node * nodes next to num_nodes, but Lightning reads devices per node and multiplies by num_nodes itself, so a 2-node run asked each node for twice the GPUs it has. The submitit launchers made the same mistake in gres, which Slurm also reads per node. Both now use gpus_per_node; single-node runs are unchanged. trainHydra scales each val head's frames-per-episode K by devices * num_nodes, but eval mode forces devices=1 only after the loaders are built. An eval run of a multi-GPU recipe therefore kept W times the frames per episode that its single rank's limit_val_batches window holds, and the window ran out before the last episodes. _trainer_world_size now returns 1 in eval mode. Both predate this stack (the ddp devices expression is from 2025-01). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UkAZwERXcFKjueZQKfRF6J
Each mp4 EvalVideo writes is also logged to any WandbLogger under <video dir>/<embodiment> (videos/, videos_train_viz/, videos_unseen_op_valid/), captioned with the epoch, so runs can be reviewed without copying files off the cluster. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01415567AQYskZLeSpSMsgwL
viz_mode=replay plays each action chunk twice over the frames it spans: first the ground-truth hand at each step, then the predicted one. Each step is drawn on the frame it belongs to, with the chunk mapped into that frame's camera by a rigid fit of the GT keypoints (the same hand at the same instant in both frames), so head motion needs no head pose. The 126-D keypoint layout draws both hands' 21-keypoint skeletons; other layouts fall back to the embodiment's own viz over a short trail. Chunks don't overlap and span horizon * action_stride frames; frames of an incomplete chunk carry over to the next flush. viz_max_batches must cover at least one chunk (a 100-step horizon needs ~13 batches of 8). _visualize_preds moves from HPTEvalVideo / PIEvalVideo (identical copies) into EvalVideo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01516MaELbiPNmJzkjPGebhu
Replay becomes EvalVideo's default. It needs consecutive frames, so it runs only on a pinned video loader; a head without one keeps the overlay video, and EvalLatent stays on overlay. The replay video loader is capped at replay_chunks (5) chunks instead of viz_max_batches, which at 8 frames a batch could not cover one 100-step chunk; viz_max_batches still bounds a loader that yields no clips. action_stride defaults to None and is inferred from the GT keypoints (step k of a chunk rigidly matches frame k*stride's own step 0): data configs use 1 or 3 and cotrain mixes them under one embodiment. TrainVizEvalVideo passes the replay settings through and tells its base which mode each step is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01516MaELbiPNmJzkjPGebhu
2,866 of 41,607 local Mecka episodes carry one head pose for the whole episode (no tracking) and 142 have single-frame head jumps of 0.3-3.7 m. Both corrupt the head-frame action chunks: with a frozen pose, head motion leaks into the hand trajectory. The vendor's egomotion is passed through unchanged by mecka_to_zarr, so the fault is upstream. - DatasetFilter(exclude_hashes=...): a list, a file, or a blocklist name under egomimic/resources/blocklists/. Excluded episodes never match; an excluded pin is dropped with a warning (an all-excluded pin list raises, since no pins means no filter). - scan_egomotion.py classifies every local episode's obs_head_pose and writes the blocklist; mecka_bad_egomotion.txt holds its 3,008 hashes. - mecka_all_6d applies it (this reshuffles its seeded train/valid split). - mecka_20pct_6d: 393 of 4,471 train pins drop out. Its valid and unseen_op_valid video episodes were two of the frozen-pose episodes; both are repinned to packaging_food episodes with tracked egomotion (69b5691e... seen operator, 69b525ff... held-out operator). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01516MaELbiPNmJzkjPGebhu
Norm stats were cached per split, so a new episode set (20% -> full, an operator hold-out, newly collected data) resampled every episode. Each episode's sampled proprio/action rows are now cached on their own under <cache_dir>/episodes/, keyed by the resolver recipe and reader code but not by the split, and a split's stats are computed from the union of its episodes' rows. Frames sit on a power-of-two stride with a per-episode phase, so a file cached at stride s serves every coarser stride. Adds norm_stats.max_samples (now part of the split key; KEY_VERSION 3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0118RhoJmzcWC75B3vtc3zkF
…andas Startup on the 41.6k-episode full-mecka split spent ~20 min before training: populate_from_datasets read a full sample from every episode to learn the post-transform keys, though a resolver hands every episode the same key_map and transform objects. It now probes one episode per (embodiment, key_map, transform) group, trying up to 8 before falling back to the raw key_map. Resolving pinned filters was ~50 s each: a row-wise df.apply(row.to_dict()) over the ~458k-row episode table plus a per-pin .loc in _check_pins. Pinned filters now narrow the table to their hashes first, rows are filtered as records, and _check_pins builds one dict of pinned rows. Same paths on the real table; 41.6k pins 2.5 s, val pins ~0 s, unpinned lab filter 32 s -> 13 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0118RhoJmzcWC75B3vtc3zkF
aidang3019
force-pushed
the
ryanco/fast-dataset-startup
branch
from
September 30, 2026 01:07
04f0a1b to
abb03bf
Compare
RyanPCo
force-pushed
the
ryanco/norm-stats-per-episode
branch
from
October 12, 2026 20:26
1c8e707 to
e51b3b4
Compare
Contributor
Author
|
Folded into #687 to keep the stack under Graphite's 50-branch cap; the changes are unchanged there. |
This was referenced Oct 12, 2026
Open
This was referenced Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Startup on the 41.6k-episode full-mecka split spent ~20 min before training:
populate_from_datasets read a full sample from every episode to learn the
post-transform keys, though a resolver hands every episode the same key_map
and transform objects. It now probes one episode per (embodiment, key_map,
transform) group, trying up to 8 before falling back to the raw key_map.
Resolving pinned filters was ~50 s each: a row-wise df.apply(row.to_dict())
over the ~458k-row episode table plus a per-pin .loc in _check_pins. Pinned
filters now narrow the table to their hashes first, rows are filtered as
records, and _check_pins builds one dict of pinned rows. Same paths on the real
table; 41.6k pins 2.5 s, val pins ~0 s, unpinned lab filter 32 s -> 13 s.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_0118RhoJmzcWC75B3vtc3zkF