Hi, thanks for sharing this excellent framework — it is very helpful for CLIP-based continual learning research.
While using C3Box to reproduce several CLIP-based continual learning baselines, I carefully cross-checked the implementation against both the original papers and the prior LAMDA framework, such as LAMDA-PILOT, and noticed a potential inconsistency in the evaluation settings of DualPrompt and CODA-Prompt.
Specifically, there appear to be two key differences from the original rehearsal-free formulations.
- Replay memory is enabled by default
- Both DualPrompt and CODA-Prompt include a
memory_size, such as 2000, and memory_per_class.
- This introduces rehearsal samples during training.
- However, in the original papers and prior LAMDA-PILOT implementation, these methods are defined under a rehearsal-free, or no exemplar replay, setting.
- Old-class logits masking is disabled
In the original CLIP-CIL evaluation setup, it is common to mask old classes during training:
logits[:, :self.last_valid_out_dim] = -float("inf")
However, in C3Box this line appears to be commented out:
# logits[:, :self._known_classes] = float("-inf")
As a result, the model performs joint optimization over all seen classes rather than strictly task-incremental isolation, which changes the learning dynamics compared to the original formulation.
I also reviewed the prior LAMDA framework, such as LAMDA-PILOT, and the original papers for these methods. They consistently describe DualPrompt and CODA-Prompt as rehearsal-free baselines without exemplar replay. This cross-check further suggests that the current implementation difference may significantly affect reproducibility comparisons.
Thanks again for your great work on C3Box — it is a very useful and well-structured benchmark for CLIP-based continual learning.
Hi, thanks for sharing this excellent framework — it is very helpful for CLIP-based continual learning research.
While using C3Box to reproduce several CLIP-based continual learning baselines, I carefully cross-checked the implementation against both the original papers and the prior LAMDA framework, such as LAMDA-PILOT, and noticed a potential inconsistency in the evaluation settings of DualPrompt and CODA-Prompt.
Specifically, there appear to be two key differences from the original rehearsal-free formulations.
memory_size, such as2000, andmemory_per_class.In the original CLIP-CIL evaluation setup, it is common to mask old classes during training:
logits[:, :self.last_valid_out_dim] = -float("inf")However, in C3Box this line appears to be commented out:
# logits[:, :self._known_classes] = float("-inf")As a result, the model performs joint optimization over all seen classes rather than strictly task-incremental isolation, which changes the learning dynamics compared to the original formulation.
I also reviewed the prior LAMDA framework, such as LAMDA-PILOT, and the original papers for these methods. They consistently describe DualPrompt and CODA-Prompt as rehearsal-free baselines without exemplar replay. This cross-check further suggests that the current implementation difference may significantly affect reproducibility comparisons.
Thanks again for your great work on C3Box — it is a very useful and well-structured benchmark for CLIP-based continual learning.