REINFROCE or GRPO

[QingyuanWuNothing]

Nice work! I’m a bit confused about the usage of REINFORCE and GRPO in the example.

1. The report states that multi-turn REINFORCE is used as the baseline (Algorithm 1).
2. In the code, the default for reinforce_update is set to False (link).
3. The provided example does not enable reinforce_update=True (link).
4. In train.py, the learning_step of Learner is not modified (link).

Did I miss something? Thanks!

[lkevinzc]

Hey @QingyuanWuNothing , thanks for your interest and your great questions!

To clarify, reinforce_update in oat is an old flag I used earlier to ablate the PPO's clipping operation, and it is not relevant to the multi-turn REINFORCE we presented in this work.

Instead, the multi-turn REINFORCE is achieved by:

1. considering transition level training samples (just as normal RL does) and compute turn-level returns via monte carlo (link).
2. directly passing the computed returns as sample "reward" from actor to learner (a bit hacky, but in this way we can conveniently use this "reward" to reweigh policy gradient in learner, link)
3. on the learner side, we receive the reward at here, and processed here, then converted into advantage here.
4. In train_oat.py, we modify compute_monte_carlo_advantages (code) to normalize the returns over the whole batch (as stated in Algo1).

[QingyuanWuNothing]

Thanks! I am really into your works.

1. The MC return and advantage processing pipeline is cool and clear.
2. So the loss calculation and learning_step in Learner used in this project is here? Please point to the correct one if I am wrong.

[lkevinzc]

Thank you very much for your kind words!

For the question: yep, because in train_oat.py we inherit PPOLearner from oat and customize it for the batch normalization.

I am really glad that you find our work useful. Please let me know any questions and I am happy to have further discussion!

[QingyuanWuNothing]

Thanks! No more questions here.