Does the project include code for language model decoding?

[Feiyunobug]

Hi, thank you again for your great work!

I have a few questions regarding the use of a language model in this project:

Does the current implementation include any decoding code that uses a transformer-based language model?

Did you train or fine-tune a language model separately for decoding purposes?

Would it be possible for you to share any code or guidance on how to integrate a transformer-based language model for decoding with your model outputs?

[snoop2head]

@Feiyunobug

Thank you for your interest in our work! Below is a brief summary, followed by additional background information.

TL;DR

We rely on an RNN-based language model checkpoint, which you can find here.

Additional Details

Our approach is largely inspired by AutoAVSR. In our paper, we adopt a training framework similar to AutoAVSR but introduce an audio reconstruction loss term.

AutoAVSR itself builds upon Visual Speech Recognition for Multiple Languages in the Wild (often referred to as CM-Aux), while adding training on synthetically generated transcripts. Consequently, we inherited many of the settings (including the choice of language model) from these projects. They host a selection of language models in this model zoo. For our work, we specifically used the same RNN-based checkpoint as described in the Raven repository, a work from the same group of authors.

Notes on Transformer-Based Language Models

• We have not trained or fine-tuned a transformer-based language model for decoding in our current implementation.
• If you wish to integrate a transformer-based model, you could adapt the outputs from our system to feed into your own or a publicly available transformer-based decoder. However, we do not currently provide a direct tutorial or code snippet for this.

Please let us know if you have any further questions!