Problem Regarding Evaluation

[chrisliu298]

Hi,

Thank you for developing this fantastic dataset for code generation!

I had a quick question about the evaluation script (compute_metric.py). In my experience, it takes about 17 hours complete, even with the provided example generation. When I tried switching to parallel evaluation, the process finished much faster (in seconds), but all the results returned as zero. During evaluation, I used the original code an generated solutions provided in this repo.

Is this expected behavior, or am I missing something? I would appreciate any guidance you could provide.

[sssszh]

@chrisliu298 Hi, have you solved this problem? I’ve encountered the same issue.

[CaioRhoden]

Hello,

I'm facing the same issue here, each generation was taking too long to be evaluated (about 5 min)

Currently I'm trying to make the parallel evaluation works, it seems to be calling the function to compile in parallel as expected and it's always returning the code -2, but I don't know why yet.

I would appreciate it if anyone has suggestions or updates about this topic. If I achieve progress, I will report here

[2450292765]

Hello,

I'm facing the same issue here, each generation was taking too long to be evaluated (about 5 min)

Currently I'm trying to make the parallel evaluation works, it seems to be calling the function to compile in parallel as expected and it's always returning the code -2, but I don't know why yet.

I would appreciate it if anyone has suggestions or updates about this topic. If I achieve progress, I will report here

Hi, have you solved this problem? I’ve encountered the same issue.