How to calculate the tpf and tsp?

[guoyd1999]

Thanks for your great work.
Do you have a tool/code to calculate metrics like token per forward and token per second for AR and dLLMs models?

[snyhlxde1]

Hi there. Thanks for your interest.
We have provided example profiling script to generate answers for HumanEval here:
https://github.com/hao-ai-lab/JacobiForcing/blob/main/JacobiForcing/jacobi_forcing_inference_MR_humaneval.py

If you run this script, it will report avg. iterations/token from this line, you can therefore take the inverse tokens/iter to get TPF, and tokens/s (TPS) from this line.