Hi authors,
Thanks for the interesting work. I have a question about the reported REC/Grounding results on the CoIN benchmark.
In the original CoIN paper, the Grounding performance of LLaVA right after learning the Grounding task is 31.27%, and the final performance after sequentially training all tasks drops to 0.83%. However, in your paper, SAME reports a final REC accuracy of 59.87%, which is much higher than even the “right-after-learning” Grounding result reported by CoIN.
I also tried single-task fine-tuning on the Grounding data only, with a trainable multimodal projector and LoRA on the LLM. We used the publicly released CoIN Instructions_Original as the instruction data for testing. The Grounding accuracy is 39.68%, which is still far below the final continual-learning REC result of SAME, 59.87%. Intuitively, if the model is trained only on this task, its performance should not be much worse than the final result after continual learning over multiple tasks.
Could you please provide the accuracy of each task right after it is learned, i.e., $A_{s,s}$, especially the REC/Grounding result right after learning REC? It would also be helpful to clarify whether the REC result in Table 1 is the final result after all tasks, $A_{REC,T}$, or the result right after learning REC, $A_{REC,REC}$.
Thanks!
Hi authors,
Thanks for the interesting work. I have a question about the reported REC/Grounding results on the CoIN benchmark.
In the original CoIN paper, the Grounding performance of LLaVA right after learning the Grounding task is 31.27%, and the final performance after sequentially training all tasks drops to 0.83%. However, in your paper, SAME reports a final REC accuracy of 59.87%, which is much higher than even the “right-after-learning” Grounding result reported by CoIN.
I also tried single-task fine-tuning on the Grounding data only, with a trainable multimodal projector and LoRA on the LLM. We used the publicly released CoIN Instructions_Original as the instruction data for testing. The Grounding accuracy is 39.68%, which is still far below the final continual-learning REC result of SAME, 59.87%. Intuitively, if the model is trained only on this task, its performance should not be much worse than the final result after continual learning over multiple tasks.
Could you please provide the accuracy of each task right after it is learned, i.e.,$A_{s,s}$ , especially the REC/Grounding result right after learning REC? It would also be helpful to clarify whether the REC result in Table 1 is the final result after all tasks, $A_{REC,T}$ , or the result right after learning REC, $A_{REC,REC}$ .
Thanks!