Training Time Discrepancies

[AlexTurner90]

Hi @farewellthree,

Thank you for sharing your work. I've been trying to training your STLLM models using the configurations provided - specifically the following two: instructblipbase_stllm_conversation.yaml and instructblipbase_stllm_qa.yaml on 8 A100 GPUs. I observed the training time for the conversation model to be around 16 hours, whereas the paper suggests approximately "6 hours for 2 epochs using Deepspeed's zero-2 setting".

Here is the command I use to initiate training:
deepspeed --master_port=20000 --include=localhost:0,1,2,3,4,5,6,7

I'm looking for a clarification on whether there might be any configuration adjustments (e.g., batch size, optimizer settings) that could help align the training time more closely with what is in the paper. Additionally, could you provide the expected training duration for tboth configurations?

Any suggestions to improve training efficiency would be greatly appreciated. Thank you.

[farewellthree]

Sorry for the confusion. The training time mentioned in the paper is for the qa_config, and training with the conversation_config does take considerably more time. However, if you exclude the caption_webvid data, you can save a significant amount of time with minimal performance loss.