Support for Structured Languages like Code and Context-Free Grammars

[QinengWang-Aiden]

I've been exploring SONAR's multilingual capabilities and am impressed by its ability to handle diverse languages through its encoder-decoder architecture. I'm wondering if it would be possible to extend SONAR to support structured languages, such as programming languages or other context-free grammars, by treating them as new languages in the system.

Given SONNAR's language-agnostic design and the use of SentencePiece tokenization, it seems theoretically possible to train SONAR to handle structured languages by defining them as new language codes (e.g., "py_Code" for Python, "java_Code" for Java, or "cfg_Form" for formal grammars).

Therefore, may I ask if it is possible to do the following:

1. training the structured language as a new 'language' to encode and decode the expression?
2. Will any modifications be needed to handle strict syntactic rules?
   If so, may I further ask how I can possibly add a new language to SONNAR, i.e., the training recipe?

Looking forward to hearing from you :)

[avidale]

Hi @QinengWang-Aiden, and thank you for an interesting question!

1. is it possible to ... training the structured language as a new 'language' to encode and decode the expression?

Yes, absolutely. To train a new encoder or decoder, you need a parallel dataset of sentences, where one side is in a language already supported by a SONAR encoder, and another side is the new language. This is how the text encoders for new languages were trained, for example, in Multilingual Representation Distillation with Contrastive Learning, and speech encoders were trained in the SONAR paper. To perform the training, you encode the known-language side with an existing encoder ("teacher"), and then update the new encoder ("student") to embed the sentences in the new language similarly (using MSE or cosine loss or their combination; some contrastive loss may also help), or update the new decoder to decode the sentences in the new language from the teacher embeddings (using the standard cross-entropy loss).

2. Will any modifications be needed to handle strict syntactic rules?

On the encoder side, probably no modification will be required. On the decoder side, you may want to implement some kind of structured/constrained decoding that adheres to the syntax by design, but I would recommend first trying unconstrained decoding (and maybe it would be sufficient).

If so, may I further ask how I can possibly add a new language to SONNAR, i.e., the training recipe?

The training recipe would largely depend on the training framework that you choose (e.g. Huggingface transformers, fairseq2, or maybe pure pytorch).