the number of MLS model parameters and polish dev loss curve fluctuations

[lhw26]

Question

I have read the paper and have some questions. In thisr paper MLS: A LARGE-SCALE MULTILINGUAL DATASET FOR SPEECH RESEARCH, a 36-layers transformer was used to train the monolingual model. I would like to know the model size. A 1GB acoustic model is provided in the mls folder, but I want to know the number of parameters of the model. Besides, when reproducing the monolingual results in this paper for Polish, the dev loss always fluctuate seriously, but this did not happen in Portuguese and Italian. Even after adjusting the learning rate, it will still fluctuate. When I shuffle the order of train and dev and redistribute the two datasets, the loss of dev can converge well. How can I check the problem?

[ljluestc]

import torch
from torch import nn

class TransformerModel(nn.Module):
    def __init__(self, num_layers=36, hidden_size=1024, ff_size=4096, vocab_size=10000):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, hidden_size)
        self.layers = nn.ModuleList([
            nn.TransformerEncoderLayer(d_model=hidden_size, nhead=8, dim_feedforward=ff_size)
            for _ in range(num_layers)
        ])
        self.fc_out = nn.Linear(hidden_size, vocab_size)

    def forward(self, x):
        x = self.embedding(x)
        for layer in self.layers:
            x = layer(x)
        return self.fc_out(x)

model = TransformerModel()
total_params = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total_params}")