feat: held-out validation during training

[ZouzouWP]

Until now, training loss was the only signal available during a run — no way to tell a policy was starting to overfit without stopping and evaluating it separately afterward, by which point the ideal checkpoint was already steps behind.

What changed

• New "Validation" section in training's advanced config: a single toggle drives dataset.eval_split (fraction of episodes held out from training) and eval_steps (evaluation cadence) together, since lerobot requires both or neither.
• Backend parses step N: eval_loss=X lines (emitted by lerobot_train whenever eval_steps > 0) into a new eval_loss field, exposed on both the live training status and the persisted metrics history (so the curve survives a page reload).
• The monitoring chart plots eval_loss as a second, sparser line alongside the training loss, so overfitting becomes visible while training is still running instead of only in hindsight.

flowchart TB
    A["Held-out episodes\n(e.g. 10%)"] --> B["Periodic evaluation"]
    C["Training loop"] --> D["Training loss\n(every step)"]
    B --> E["Validation loss\n(every N steps)"]
    D --> F["Monitoring chart"]
    E --> F

Loading

Testing

Implemented and code-reviewed against the log-parsing regex and the chart's dedupe/merge logic, but not yet run end-to-end with eval_steps > 0 against a real training job — needs a validation pass to confirm the parsed values match lerobot_train's actual output format before merge.

[nicolas-rabault]

Thanks for this, and thanks especially for saying plainly in the description that the log parsing had not been run end to end. I checked that part: the step N: eval_loss=X format matches what lerobot_train actually emits at our pinned v0.6.0, all three flags you emit exist there, and the split is per episode rather than per frame, which is the thing that would have made the whole feature meaningless. I ran your parsing over synthetic logs at both aligned and unaligned eval cadences and it holds up.

Two things block, both in TrainingRequest.
Good feature,

[nicolas-rabault]

Unbounded. lerobot validates this as 0.0 <= x < 1.0 (configs/default.py:55), but the request model does not, so dataset_eval_split=1.5 is accepted here, the job spawns, and it dies inside make_train_eval_datasets about a minute later with the reason buried in the training log.

Field(ge=0.0, lt=1.0) makes it a 422 on the request, before anything starts.

[nicolas-rabault]

This is the half of the pairing lerobot does not enforce, and it is the half that fails silently.

configs/train.py:255 rejects eval_steps > 0 with eval_split == 0.0, loudly. The reverse is perfectly legal: dataset_eval_split=0.1 with eval_steps=0 emits --dataset.eval_split 0.1 and nothing else, so make_train_eval_datasets holds out 10% of the episodes, the training set silently shrinks, and no eval ever runs. The user pays for it and gets no curve back, with nothing in the log explaining why.

I hit this by calling build_training_command directly with only dataset_eval_split set, so it is reachable, not hypothetical. The Switch comment in AdvancedCard.tsx already has the right mental model (both or neither); it just needs to live in TrainingRequest as well, as a model_validator, so the invariant holds for every caller of /training/start and not only for the form.