Problem about reward and loss in online training

[Enjia]

Hi,
Thank you for your dedicated work of PCC-Uspace.
When I followed the instruction in Deep_Learning_Readme.md, I found that values of both Reward and Ewma Reward were so high as the snapshot below:
Reward: 1360096.79, Ewma Reward: 21968834.42
Reward: 1013840.44, Ewma Reward: 21759284.48
Reward: 425067.66, Ewma Reward: 21545942.31
Reward: 327400.01, Ewma Reward: 21333756.89
Reward: 154455.32, Ewma Reward: 21121963.88
Reward: 115554.43, Ewma Reward: 20911899.78
Reward: 140730.04, Ewma Reward: 20704188.08
Reward: 112697.73, Ewma Reward: 20498273.18
Reward: 107894.34, Ewma Reward: 20294369.39
...
Worsestill, values of loss_vf_loss were also unexpected, one of which reached "4512207000000.0 ".
Did you ever stumbled across this problem and could you please tell me the possible reason behind this phenomenon? Thanks!

[nathanhjay]

Hmm. That is much higher than I would expect. The loss_vf_loss doesn't surprise me since the rewards are so high, because it's measuring value function estimation loss when trying to esimtate rewards, so if rewards get much larger, but they don't become much more predicatble, that value should go up.

Looking at the scale of rewards, I thought maybe this was just a difference in units, but it doesn't appear to be. Both offline and online training use units of "packets" and "seconds" to compute reward. Since it's not clear from the code immediately, I'll have to try running it and replicating the problem. What commands did you use to run the training? I'm most interested in the arguments given to the pccclient program and the stable_solve script. Also, what are the network parameters you're testing on? If you are testing on localhost (i.e. 127.0.0.1) with no bandwidth limiters or latency mechanisms, you might see very high numbers, but I'd still like to replicate it.

[Enjia]

Thanks for your reply！
For offline training, I just run "python stable_solve.py as you mentioned in PCC-RL repo, and the result was normal actually. For online training, I just followed the commands in Deep_Learning_Readme.md, that is:
"Start a server as usual: cd src export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:pwd/core/ ./app/pccserver recv 9000"
"Start the gym side of the training environment from the pcc-rl repo: cd pcc-rl/gym/training/online/ python3 shim_solver.py"
"Start the udt side of the training environment (on the same machine as the shim_solver): cd src export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:pwd/core/ ./app/pccclient send 127.0.0.1 9000 --pcc-rate-control=python -pyhelper=shim -path=/path/to/pcc-rl/src/udt-plugins/training/ --history-len=10 --pcc-utility-calc=linear"
And I replace the argument "127.0.0.1" of my own localhost and others remain unchanged. I wonder how to set bandwidth limiters or latency mechanisms as you mentioned. Besides, I think the main problem may be the scale of rewards, because after I read the code of network_shim.py in PCC-RL, there is a code line says:
"reward = (10.0 * throughput / (8 * BYTES_PER_PACKET) - 1e3 * latency - 2e3 * loss)
and I didn't see any similiar code lines or value transformation in shim_env.py, which directly output the initial rewards in function "reset()" that reached really high value. Do you have idea of the difference of reward expression in PCC-RL network_shim.py and shim_env.py?

Thanks!

Best Wishes

[tianbochen1]

Hi All,
Can I know that is the unit of throughput is mbps or packets?

Thanks