Release of fine-tuning data and eval configuration

[pulkitkumar95]

Hey,

Congrats on the great work! I had two questions:

1. Can the 467k caption/QA video-paired data that was used for training your model be released? It would help save time in running the data curation pipeline.

2. For evaluation results reported in Table 1, at what fps/number of frames was each video sampled at?

Also, a suggestion: in your HuggingFace dataset space, renaming "hands_eval" to "daily_eval" might be a good idea, as the paper reports by the split name "Daily".

Best,
Pulkit

[sunrainyg]

1. Yes, we will release it soon.
2. Four frames.

Thank you for the suggestion, we’ll update this!

[pulkitkumar95]

1. Is there a potential date by when it would be released? Just curious, as it would help me with an upcoming deadline
2. So, 4 frames were sampled uniformly? That seems way less. Was there a reason for it? Also, I was looking at the Gemini demo code more closely, and it seems like you sampled at 1 fps for it. Is that true? If so, there might be a discrepancy in Table 1 numbers.

[Lyken17]

we will release after new year holiday. Expected to be sometime in mid Jan.

and it seems like you sampled at 1 fps for it

The FPS should be 4 and seems the code is not the latest version. Will update the code and double check the results.