Word substitution hallucinations during synthesis

[DragonbornElric]

Environment: VoXtream v0.2.0, herimor/voxtream2 model, RTX 3060 12GB, CUDA 12.8, Ubuntu Linux, Python 3.12

Description:
The model occasionally substitutes entirely unrelated words in the output. For example, the word "small" in the input text was spoken as "fish" in the audio output. These are not mispronunciations — they are completely different words with no phonetic similarity.

Steps to reproduce:
Intermittent — no reliable reproduction steps identified yet. Occurs during normal conversational synthesis with a voice-cloned reference WAV.

Expected behavior:
The spoken output should match the input text.

[herimor]

Hi @DragonbornElric, thank you for reporting the issue.
I think such hallucination might be due to the filler word handling in VoXtream2. To automatically insert filler words, I remove them from the transcript during training, but keep them in the audio. Since the Emilia dataset is pseudo-labelled, some words might be non-transcribed as well, but are presented in the audio. I assume the model learns to insert not only filler words, but some random words as well.
I'm not sure I have a quick solution for that, probably retraining the model without filler words label should help, as I noticed it reduces WER.