Discussion on reproducing the problem

[Alexsrty]

I'm also trying to reproduce this work. How can I contact you to have a discussion?While offline reinforcement learning has improved to some extent, it still cannot achieve the effect of a thesis report.

[LZH-97]

I'm also trying to reproduce this paper. I found that IQL training showed a small improvement on the pure expert dataset. Then, when I added failed real machine data and iterated, IQL still hasn't trained any gain, even though I reduced AM-Q's threshold_radio to 0.005. I'm checking if there's something wrong with my IQL code.

[Starsshine21]

I'm also trying to reproduce this work. How can I contact you to have a discussion?While offline reinforcement learning has improved to some extent, it still cannot achieve the effect of a thesis report.

Thank you for your comment. If you want to discuss with me, you can do so directly here. I often check GitHub. Most of the current performance improvement comes from the IL effect brought by dataset expansion after rollout. The current problem lies in the AMQ algorithm. When I decrease the learning rate, the new policy struggles to outperform OPE and is always rejected. When I increase the learning rate, although it is accepted, the sampling success rate in the subsequent rollout drops to 0, indicating that the policy drift is a bit too large. You can run a complete training session to view the output logs, or we can check the code together. Looking forward to discussing with you!

[LZH-97]

I'm also trying to reproduce this work. How can I contact you to have a discussion?While offline reinforcement learning has improved to some extent, it still cannot achieve the effect of a thesis report.

Thank you for your comment. If you want to discuss with me, you can do so directly here. I often check GitHub. Most of the current performance improvement comes from the IL effect brought by dataset expansion after rollout. The current problem lies in the AMQ algorithm. When I decrease the learning rate, the new policy struggles to outperform OPE and is always rejected. When I increase the learning rate, although it is accepted, the sampling success rate in the subsequent rollout drops to 0, indicating that the policy drift is a bit too large. You can run a complete training session to view the output logs, or we can check the code together. Looking forward to discussing with you!

I'm encountering the same problem. If the PPO learning rate is set too high, it easily triggers the early termination mechanism in my code (triggered when KL > 0.5). When the learning rate is set too low, AM-Q keeps rejecting model updates, and sometimes the new policy score is slightly lower than the old one. Are there any good suggestions or tricks to achieve a balance?

My training parameters are as follows:
python train_iql.py \
--config-name=iql.yaml \
task=pick_place_rl \
dataloader.batch_size=128 \
iql.clip_ratio=0.20 \
iql.policy_lr=5e-6 \
iql.critic_epochs=200 \
iql.policy_epochs=50 \
iql.ppo_inner_steps=2 \
iql.max_batches_per_epoch=80 \
iql.expectile_tau=0.8 \
iql.kl_early_stop=0.5 \
iql.amq_gating.threshold_ratio=0.005 \
iql.amq_gating.n_eval_batches=50

here is my log

[Starsshine21]

I'm also trying to reproduce this work. How can I contact you to have a discussion?While offline reinforcement learning has improved to some extent, it still cannot achieve the effect of a thesis report.

Thank you for your comment. If you want to discuss with me, you can do so directly here. I often check GitHub. Most of the current performance improvement comes from the IL effect brought by dataset expansion after rollout. The current problem lies in the AMQ algorithm. When I decrease the learning rate, the new policy struggles to outperform OPE and is always rejected. When I increase the learning rate, although it is accepted, the sampling success rate in the subsequent rollout drops to 0, indicating that the policy drift is a bit too large. You can run a complete training session to view the output logs, or we can check the code together. Looking forward to discussing with you!

I'm encountering the same problem. If the PPO learning rate is set too high, it easily triggers the early termination mechanism in my code (triggered when KL > 0.5). When the learning rate is set too low, AM-Q keeps rejecting model updates, and sometimes the new policy score is slightly lower than the old one. Are there any good suggestions or tricks to achieve a balance?

My training parameters are as follows: python train_iql.py \ --config-name=iql.yaml \ task=pick_place_rl \ dataloader.batch_size=128 \ iql.clip_ratio=0.20 \ iql.policy_lr=5e-6 \ iql.critic_epochs=200 \ iql.policy_epochs=50 \ iql.ppo_inner_steps=2 \ iql.max_batches_per_epoch=80 \ iql.expectile_tau=0.8 \ iql.kl_early_stop=0.5 \ iql.amq_gating.threshold_ratio=0.005 \ iql.amq_gating.n_eval_batches=50

here is my log

Yes, that's the problem I encountered as well. However, the configuration of my QV network is different from yours. I stopped training after 20 epochs because the loss had dropped to a very low level. The code for my iql algorithm is in the repository. We can refer to each other's code

[LZH-97]

I'm also trying to reproduce this work. How can I contact you to have a discussion?While offline reinforcement learning has improved to some extent, it still cannot achieve the effect of a thesis report.

Thank you for your comment. If you want to discuss with me, you can do so directly here. I often check GitHub. Most of the current performance improvement comes from the IL effect brought by dataset expansion after rollout. The current problem lies in the AMQ algorithm. When I decrease the learning rate, the new policy struggles to outperform OPE and is always rejected. When I increase the learning rate, although it is accepted, the sampling success rate in the subsequent rollout drops to 0, indicating that the policy drift is a bit too large. You can run a complete training session to view the output logs, or we can check the code together. Looking forward to discussing with you!

I'm encountering the same problem. If the PPO learning rate is set too high, it easily triggers the early termination mechanism in my code (triggered when KL > 0.5). When the learning rate is set too low, AM-Q keeps rejecting model updates, and sometimes the new policy score is slightly lower than the old one. Are there any good suggestions or tricks to achieve a balance?
My training parameters are as follows: python train_iql.py \ --config-name=iql.yaml \ task=pick_place_rl \ dataloader.batch_size=128 \ iql.clip_ratio=0.20 \ iql.policy_lr=5e-6 \ iql.critic_epochs=200 \ iql.policy_epochs=50 \ iql.ppo_inner_steps=2 \ iql.max_batches_per_epoch=80 \ iql.expectile_tau=0.8 \ iql.kl_early_stop=0.5 \ iql.amq_gating.threshold_ratio=0.005 \ iql.amq_gating.n_eval_batches=50
here is my log

Yes, that's the problem I encountered as well. However, the configuration of my QV network is different from yours. I stopped training after 20 epochs because the loss had dropped to a very low level. The code for my iql algorithm is in the repository. We can refer to each other's code

thanks for your reply. I'm comparing the code and rerunning it to see the results. I'll share any new conclusions or better methods in the comments section.