[RyanPCo]

Fixes two problems on the <head>_video val loaders. Every rank wrote the same mp4, which is a correctness bug. The pinned video batches were also decoded on every validation pass, including the ones that render nothing.

Behaviour changes

• Only rank 0 writes overlay videos. Metrics are unchanged.
• trainHydra refuses a head that has pinned video episodes and an evaluator with viz_max_batches=None. Every recipe that pins video episodes already caps it at 10. The evaluators that leave it unset (eval_hpt, eval_hpt_cartesian, eval_hpt_wrist) pair with configs that pin nothing, so they keep the ungated path, where every rank renders together as before.

What changed

• Rank guard in EvalVideo._write_video. The val heads come back from val_dataloader() inside CombinedLoaders, which Lightning does not wrap in a DistributedSampler. Every rank iterated every val batch, rendered the same frames and wrote the same mp4 path, so the file was whichever rank finished last and could be torn.
• _VizGatedLoader (pl_utils/pl_data_utils.py) yields nothing when its gate is closed, which is on a non-viz pass and on every rank but 0. Before, ModelWrapper.validation_step returned before the forward on a non-viz pass, but Lightning had already built the batch. So limit_val_batches x 64 pinned frames were decoded and discarded on every rank on the ~90% of passes that render nothing: ~1.7 min a pass, ~4% of a 1000-step cycle.
  • The val loaders are built once (_EvaluationLoop.setup_data returns early once _combined_loader is set), so the gate can't be a length. It is an empty iterator, which _Sequential already reads as "move to the next loader".
  • __len__ answers without creating an iterator. Creating one forks the loader's workers and starts them prefetching, which would do the skipped work at setup and never free it. It asks a fresh _MaxSizeCycle for the length; on lightning 2.6.1 that stores its loaders and limits and builds no iterator until __iter__.
  • The gate falls open when a link is missing (eval-only mode, tests, an evaluator without _should_viz).

DDP safety

Ranks now iterate different numbers of val batches. That is safe because what they log is symmetric: the metric loaders run everywhere and log the same keys, and the video loader logs nothing, so the sync_dist reduction matches. The other ranks do wait in that reduction while rank 0 renders. Rendering took 8.4 min for 76 batches back when all four ranks rendered at once, well inside the 30-minute process-group timeout. That timeout is now the ceiling on evaluator.viz_max_batches, which is why the fail-fast above exists.

Tests

• A desync here would hang a campaign rather than fail it, so it is tested against real DDP. tests/fixtures/ddp_video_gate.py runs a 2-process gloo fit with the same collective pattern, at num_workers > 0 so prefetch is visible. In test_two_rank_ddp_fit_survives_the_rank_divergence, both ranks finish; rank 0 steps the video loader on the viz epoch and rank 1 never touches the pinned episode.
• New in tests/unit/test_eval_loaders.py: rank-0-only writes, the closed gate yielding nothing, the gate's fallbacks, __len__ opening no iterator, a real fit building video batches only on viz passes, and test_video_head_without_a_viz_cap_is_rejected.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw

[RyanPCo]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• pi: initialize from PaliGemma with a scratch action expert #677 : 2 dependent PRs (#678 , #680 )
• perf: vectorized image augs and a torch.compile knob for HPT and Pi; Pi clips gradients like openpi #676
• loader: one zarr read per key, not one per keymap entry #674
• eval: build the pinned-episode video batches only where they are used #673 👈 (View in Graphite)
• checkpoints: last.ckpt as a symlink, not a second full copy #672
• trainHydra: metric_frames_per_episode: auto, and warn on empty prompts #671
• transforms: per-step wrist framing for proprio history #670
• viz: one keypoint renderer, the horizon trace #669
• configs: history and backbone-LR knobs on the keypoint wrist HPT model #668
• trainer: ddp_bf16_compress, DDP with a bf16 gradient all-reduce #667
• eval: overlay videos play at the source rate under DDP #666
• hpt: build the policy on CPU and let Lightning place it #664 : 1 other dependent PR (#665 )
• hpt: never re-initialise pretrained stems; hash-check them; mask text padding #645 : 1 other dependent PR (#646 )
• obs: proprio history (K proprio steps, one token each, learned time embedding) #650
• loss: mask the repeat-last action padding out of the HPT loss #649
• resnet: keep a frozen backbone in eval mode; give backbones their own LR group #652
• augs: per-sample image augmentation for HPT train-mode augs #651
• eval: deterministic validation (seeded prompts + flow sampling); log-only grad-norm MAD #644
• data: mecka_fold_flagship_top3_hpt_6d, the 3-operator flagship train split #643
• val: subsampled metric loaders + pinned contiguous video loaders #642
• infra: mecka 6D / keypoint recipes, 300M models, defaults, and the rollout revert #640
• eval: layout-aware frame-invariant metrics, and a train-split viz head #659
• data: stop letting one bad episode or a rotation channel take a run down #658
• norm: guard degenerate channels, horizon pooling, and no rot6d normalization #657
• keypoints: the 144-D wrist-first hand-keypoint action becomes the human default #656
• mecka: fix the double-mirrored LEFT wrist frame, and correct existing zarrs #655
• 6d: PI goes 6D-only; Eva and Human gain the continuous-rotation modes #654
• 6d: continuous rotation representation, bimanual layouts, shared norm helpers #653
• train: give each Slurm slurm job gets its own cache dir to avoid conflict #638 : 1 other dependent PR (#639 )
• write git version into the run dir for reproducibility #637
• train: don't load PI base weights on resume and verify that checkpoint path exists #636
• configs: declare proprio and actions dims once per HPT model config #634 : 1 other dependent PR (#635 )
• cache norm stats for data configs to minimize recomputation #633
• import cloud packages lazily to support local training #631 : 1 other dependent PR (#632 )
• cache dataset resolver for faster training initialization #630
• validate episode hashes in data config before training #628
• modify pi camera map to match data #629
• add tests to ensure training works between every model and data vendor #626
• run CI tests on every PR #625
• add env variables for user-specific paths #624
• build: set pyproject and uv as the dependency managers #623
• fix(configs): fix paths for data hydra yaml configs #622
• refactor(algo): delete ACT #627 : 1 other dependent PR (#621 )
• tests: create dedicated test folder with unit and integration tests #620
• add pyright type checker #619
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[RyanPCo]

Automated adversarial review (read-only; findings were checked by a second skeptical pass before posting): 1 low, 2 nit.

[RyanPCo]

low (simplification): _VizGatedLoader accepts mode only to reject everything but 'max_size_cycle', and its single caller passes that literal. Drop the parameter and call super().init(iterables, "max_size_cycle"). More importantly, len and the module import depend on Lightning privates (_MaxSizeCycle, _iterator, flattened, limits), and lightning is unpinned in requirements.txt and pyproject.toml. An upgrade that renames _MaxSizeCycle breaks the import of pl_data_utils itself. Pin the Lightning version, or at least name the version these privates were checked against.

[RyanPCo]

nit (comments): Keep the invariant in _video_gate's docstring (the video loader logs nothing, so the sync_dist reduction stays symmetric, and viz_max_batches bounds rank 0's render against the process-group timeout). Drop the measurements ('8.4 min for 76 batches', and '~0.1 s/frame, 4864 frames' in eval_video.py). The 30-minute figure is a configurable default, not a constant.

[RyanPCo]

nit (dead-code): _Module stores pinned but never reads it, since main() reads pinned.reads directly. Drop the parameter and attribute.

[RyanPCo]

Review findings: status after the fixup pass. The fixes are squashed into this PR's commits, and the whole stack was force-pushed.

Fixed

• _VizGatedLoader takes a mode it only allows one value for; relies on Lightning privates: Removed the mode parameter, and the gate is now functools.partial. A comment names lightning 2.6.1 as the version the private API was checked against. Lightning is not pinned in requirements: that is a project-wide decision outside this PR.
• Docstrings carry dated wall-clock measurements and a hard-coded timeout: The _video_gate docstring keeps only the invariant. Removed the 8.4 min and 30-minute figures, and the ~0.1 s/frame note in eval_video.
• _Module.pinned stored but never read: Removed the parameter and the attribute.