Question about SLT Training Logs

[lucasnhandang]

Hi Geo-Sign authors,

Thanks for open-sourcing Geo-Sign. I’m training the SLT task and observed an unusual pattern: train loss decreases, validation loss decreases yet increase again after some epochs, but BLEU/ROUGE still improve.

Could you share the original SLT training logs or loss/metric curves for comparison? I’d like to know whether this loss–metric mismatch is expected in Geo-Sign training, or whether it may indicate a configuration issue on my side.

Thanks!

[ed-fish]

Hi - yes I also observed this behaviour - it seems to be some grokking artifact from the regularisation - in fact the performance continues improving but I didn't want to train longer than uni-sign for fair comparison. I'm not sure why this happens but we have observed with other methods as well using the same decoder.

[lucasnhandang]

Thank you. I would like to ask why the model was finetuned in Stage 3 for 40 epochs instead of 20 epochs like Uni-Sign. Is it possible that 20 epochs are insufficient for the improvements of Geo-Sign to take effect?

[ed-fish]

oh yes you make a good point! I was sure that Uni-Sign did 40 epochs of fine-tuning but in fact looking at the code/paper it seems they did 20 epochs or pre-training and 20 epochs of fine-tuning - thanks for spotting this, I will make a correction in the paper. For Geo-Sign the value of alpha will have an impact on the performance with respect to the number of epochs - since the alpha will introduce more/less regularisation as detailed in the supplementary material section E.2. All other results in the paper include the same training epochs (40) - so you can take the Euclidean Pooled results to be a reflection of the Uni-Sign method over 40 epochs with some additional regularisation which explains the small increase in performance.