PRM Data Construct

[Aaronhuang-778]

Hi,
Thanks for your nice work!

May I know how did you generate the PRM data based on Math-Shepherd (I am curious about how did you construct the MC process for this kind of data). Have you planed to open the code for data generation?

Thanks :-)

[hanningzhang]

Thank you for your interest!

To construct the MC process, given a question: {Question... Step 1.. Step 2.. Step3 ..},
We may generate completions from Question... Step 1...
and then we can estimate the quality of Step 1 based on the completions.

We may then generate completions from Question... Step 1... Step 2..
and then we can estimate the quality of Step 2 based on the completions.

We follow this to construct the MC process

[yarinshaked]

Hey guys, thank you very much for the work.
Can you please open the code for data generation?

[hanningzhang]

Hi. Thank you for your question. We are sorry that we did not save the exact code for data generation, but the logic is similar to https://github.com/RLHFlow/Online-DPO-R1/blob/main/generation/gen_hf.py.

When you have your constructed dataset (Question + partial solutions), you may just simply use vllm to sample multiple solutions and compare the solution with the ground truth (you may use https://github.com/RLHFlow/Online-DPO-R1/blob/main/eval_math/grader.py to compare the answer with the ground-truth). And then you could calculate the process reward for a specific step.