What should be ideal range of Loss while training PL-ERT (any new language)

[SandyPanda-MLDL]

I am training PLBERT for 5000000 number of epochs but my loss in not decreasing and always giving 8.something. Please let us know the ideal range of loss for PLBERT training.

Step [2233480/5000000], Loss: 8.51781, Vocab Loss: 6.68871, Token Loss: 1.66403
Step [2233490/5000000], Loss: 8.26963, Vocab Loss: 6.07282, Token Loss: 1.52627
Step [2233500/5000000], Loss: 8.77817, Vocab Loss: 5.29348, Token Loss: 2.13625
Step [2233510/5000000], Loss: 8.51580, Vocab Loss: 6.69523, Token Loss: 2.22209
Step [2233520/5000000], Loss: 8.72296, Vocab Loss: 6.78622, Token Loss: 2.18462
Step [2233530/5000000], Loss: 8.26051, Vocab Loss: 6.93161, Token Loss: 1.61516

[Siim]

I'm getting significantly different loss values. I'm using the bert-base-multilingual-cased tokenizer for Estonian because it seems like it should work, according to multilingual-pl-bert.

Note that I added some debug input/output to check what is going on.

The output is not correctly decoded—for example, it is not mapped back to the actual values using token_map. However, it already provides a good hint that the predictions are performing well.

Validation Example:
Example 20:
Input Phonemes: 531566351158166115747471581652156435569165215663506263611661156515115861165615663143143516116156726069165315643165615672375146634616615815751465755156471586215860515416
Words - Predicted: 傅 傅 傅 傅 傅 [PAD] സ സ സ സ സ [PAD] 砌 砌 砌 砌 砌 [PAD] [unused55] [unused55] [unused55] [unused55] [unused55] [unused55] [unused55] [PAD] 洩 洩 洩 洩 洩 洩 [PAD] ỗ ỗ ỗ ỗ ỗ ỗ ỗ [PAD] ] ] ] ] [PAD] 紧 紧 紧 [PAD] tiểu tiểu tiểu tiểu tiểu tiểu tiểu tiểu [PAD] x x x x x x x x x x x x x x x [PAD]
Words - Actual: 傅 傅 傅 傅 傅 [PAD] സ സ സ സ സ [PAD] 砌 砌 砌 砌 砌 [PAD] [unused55] [unused55] [unused55] [unused55] [unused55] [unused55] [unused55] [PAD] 洩 洩 洩 洩 洩 洩 [PAD] ỗ ỗ ỗ ỗ ỗ ỗ ỗ [PAD] ] ] ] ] [PAD] 紧 紧 紧 [PAD] tiểu tiểu tiểu tiểu tiểu tiểu tiểu tiểu [PAD] x x x x x x x x x x x x x x x [PAD]
Tokens - Predicted: <UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK>
Tokens - Actual: <UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK>
--------------------------------------------------

Step [192940/1000000], Loss: 0.06243, Vocab Loss: 0.01753, Token Loss: 0.03885, LR: 0.0000236
Current LR: 2.3613016762571927e-05

Validation Example:
Example 3:
Input Phonemes: 58156475361431661156431586443461615651531584316561574747158461653476116501565715814351644346165815647435447161564756464316531564316156471565343156156615016156515586545153161566750102615357564616
Words - Predicted: ##xexexexexexe [PAD] Đ Đ Đ Đ Đ Đ Đ [PAD] 蹈 蹈 蹈 蹈 蹈 [PAD] 哉 哉 哉 哉 哉 哉 [PAD] Association Association Association [PAD] Ó Ó Ó Ó Ó Ó Ó Ó Ó [PAD] 埡 埡 埡 埡 埡 埡 [PAD] 朐 朐 朐 朐 朐 [PAD] ဂ ဂ ဂ [PAD] 别 别 别 别 别 别 别 别 别 [PAD] 哉 哉 哉 哉 哉 哉 哉 [PAD] ગામમાં ગામમાં ગામમાં ગામમાં ગામમાં ગામમાં ગામમાં ગામમાં ગામમાં [PAD]
Words - Actual: ##xexexexexexe [PAD] Đ Đ Đ Đ Đ Đ Đ [PAD] 蹈 蹈 蹈 蹈 蹈 [PAD] 哉 哉 哉 哉 哉 哉 [PAD] Association Association Association [PAD] Ó Ó Ó Ó Ó Ó Ó Ó Ó [PAD] 埡 埡 埡 埡 埡 埡 [PAD] 朐 朐 朐 朐 朐 [PAD] ဂ ဂ ဂ [PAD] 别 别 别 别 别 别 别 别 别 [PAD] 哉 哉 哉 哉 哉 哉 哉 [PAD] ગામમાં ગામમાં ગામમાં ગામમાં ગામમાં ગામમાં ગામમાં ગામમાં ગામમાં [PAD]
Tokens - Predicted: <UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK>
Tokens - Actual: <UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK>
--------------------------------------------------


Validation Example:
Example 36:
Input Phonemes: 5615747564647165815643448660511574753615857606216292916531566751143143166115663631586016531566351461661156511586253431576346631615747511581615757548616611574754544716641564746135165615672506916
Words - Predicted: 蹲 蹲 蹲 蹲 蹲 蹲 [PAD] Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж [PAD] вже вже [PAD] 缓 缓 缓 缓 缓 缓 [PAD] 儼 儼 儼 儼 儼 儼 [PAD] [unused75] [unused75] [unused75] [unused75] [unused75] [PAD] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [PAD]ีีีี [PAD]όόόό [PAD] 汲 汲 汲 汲 汲 汲 [PAD] z z z z z [PAD] ğ ğ ğ ğ ğ [PAD]
Words - Actual: 蹲 蹲 蹲 蹲 蹲 蹲 [PAD] Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж Ж [PAD] вже вже [PAD] 缓 缓 缓 缓 缓 缓 [PAD] 儼 儼 儼 儼 儼 儼 [PAD] [unused75] [unused75] [unused75] [unused75] [unused75] [PAD] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [unused26] [PAD]ีีีี [PAD]όόόό [PAD] 汲 汲 汲 汲 汲 汲 [PAD] z z z z z [PAD] ğ ğ ğ ğ ğ [PAD]
Tokens - Predicted: <UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK>
Tokens - Actual: <UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK>
--------------------------------------------------


Validation Example:
Example 17:
Input Phonemes: 601574792102616260156471586051625116531565754551654156511585315854636115711756564762636162165515751545447611661156435115816641565192696162157434643165315663631586116292929292929292916
Words - Predicted: めて めて めて めて めて めて めて めて めて めて めて めて めて めて めて [PAD] costa costa costa costa costa [PAD] перед перед перед перед перед перед перед перед перед перед перед перед перед перед перед перед перед перед [PAD] ھ ھ ھ ھ ھ ھ ھ [PAD] Ѕ Ѕ Ѕ Ѕ Ѕ [PAD] ☉ ☉ ☉ ☉ ☉ ☉ ☉ ☉ ☉ ☉ ☉ [PAD]harharharharharhar [PAD] Sarupo Sarupo Sarupo Sarupo Sarupo Sarupo Sarupo Sarupo [PAD]
Words - Actual: めて めて めて めて めて めて めて めて めて めて めて めて めて めて めて [PAD] costa costa costa costa costa [PAD] перед перед перед перед перед перед перед перед перед перед перед перед перед перед перед перед перед перед [PAD] ھ ھ ھ ھ ھ ھ ھ [PAD] Ѕ Ѕ Ѕ Ѕ Ѕ [PAD] ☉ ☉ ☉ ☉ ☉ ☉ ☉ ☉ ☉ ☉ ☉ [PAD]harharharharharhar [PAD] Sarupo Sarupo Sarupo Sarupo Sarupo Sarupo Sarupo Sarupo [PAD]
Tokens - Predicted: <UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK>2<UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK>
Tokens - Actual: <UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK>2<UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK><UNK>
--------------------------------------------------

Step [192950/1000000], Loss: 0.06471, Vocab Loss: 0.02408, Token Loss: 0.04235, LR: 0.0000236
Current LR: 2.3609053790342755e-05

[Rahul-Chittora]

@Siim I guess, as most of the tokens are marked as "UNK" it is predicting everything as "UNK" and hence loss is very low.
@SandyPanda-MLDL We are also getting similar loss values. How does it work when you used with TTS model? was it able to perform in line with english model.

[Islomov49]

I had the same problem as well. But as I later figured out, the issue was with my data. It's likely the same here—most probably something with the tokenizer. I had to train my own language tokenizer, using good normalization scripts.

What I see now is that the loss looks like a healthy training process, and it’s decreasing normally.

Previously, the author commented that his token loss was close to 1. And, like with any large language model (such as BERT models), the training should continue as much as you can afford