Evaluation uses params before training update

[jselvaraaj]

Looking at the training loop in rec_mappo.py (and other algorithms):

learner_output = learn(learner_state)

# Evaluation uses learner_state (input), not learner_output.learner_state(output)
trained_params = unreplicate_batch_dim(learner_state.params.actor_params)
eval_metrics = evaluator(trained_params, ...)

learner_state = learner_output.learner_state  # Update for next iteration

The evaluation runs on learner_state.params (the params before learn() was called), not learner_output.learner_state.params (the params after training).

This means:

• First eval uses random/initial params (no training has happened yet)
• Every eval is "one step behind" the actual trained params
• Final eval doesn't reflect the last training update (unless absolute metrics is enabled)

Is this intentional? It seems like we'd want to evaluate the params we just trained, not the ones from the previous iteration.

[sash-a]

It is intentional as we usually train with the absolute metric turned on and we'd like the evaluations to start from 0, as if your num_evaluations is low you may miss a the initial spike if you're only looking from the first eval.

But I do see your point, it would make more sense to do an initial eval outside the loop and then evaluate with the current params