Canary-v2 streamatt speech processor

[azziko]

Hello,

I implemented a support for decoding_input_ids and tested it to see if it works. Although it's not merged yet, I'm working towards contributing canary with streamatt to the repo.

I have a question about the implementation of audio history. I see from the base_streamatt implementation that the audio is supposed to be stored in mel-features. Although it is possible to extract the mel-features first in NeMo framework, it's much easier to work with raw waveform history. I was thinking of some options how to tweak the implementation so that the raw history update is supported:

1. In self.audio_subsampling_factor put subsampling_factor * MEL_HOP_SAMPLES. This maps one encoder frame -> raw form. I was worried this might mess up the semantics of the code a bit.
2. The second option would be to override _update_audio_history completely, but then there is repeating code.

How would you see this implemented best?

[mgaido91]

Thank you very much @azziko ! Honestly, I think that if possible working with mel features would be preferred for a couple of reasons:

1. When you decide how much audio to retain with the attention matrix, you do this selection on the network input (ie. mel features). If you want to keep the waveform you have to map this back to the waveform.
2. Extracting mel features, although not the most expesive thing, has some compute overhead, which is not negligible. So if we can avoid doing it multiple times, the whole thing is a bit more efficient.

Thank you!

[azziko]

Thanks and sorry for a late reply, it's a busy time with the iwslt2026 submission deadline ahead.

Understood, I will try to look and see if something can be done. It would be great if passing them twice through the preprocessing would be smart. Otherwise, transcribe() is not an option and the operation would need to be done through manual encoder -> decoder passes.

            processed_signal, processed_signal_length = self.preprocessor(
                input_signal=input_signal, length=input_signal_length
            )

[mgaido91]

Mmmhhh it should be possible if we feed transcribe with a DataLoader that does not run the preprocessor but just build the batch from the features.

[azziko]

Not directly, since here in the pipeline it's not expected that after loading the audio is preprocessed for a forward call:

https://github.com/NVIDIA-NeMo/NeMo/blob/0ae9f88b19479db12605d1bd0889ce4a1f4c0f20/nemo/collections/asr/models/aed_multitask_models.py#L986

[mgaido91]

I see, you're right. We would have to create a parent class and override _transcribe_forward but that's far from ideal. We probably have to fallback to keep the audio history in waveform as you were suggesting. To avoid duplicating a lot of code, we can add an abstract property to the BaseStreamAtt that tells whether to use raw audio of frames in the audio history and edit the few places where it needs to be updated based on this flag. it is not ideal as it has some (minor) slowdown due to the extra preprocessing needed and I am also a bit afraid regarding the stability of the outputs, since truncating the waveform and recomputing the filterbanks may lead to a bit of "dirty" vectors at the beginning and end, but I see no other option avoiding to rely on private methods in NeMo, which is not great. Thanks for your efforts on this!

PS ofc let me know if there is something I can help with.

[azziko]

Thank you, I implemented it as you suggested in the linked PR

I am also a bit afraid regarding the stability of the outputs, since truncating the waveform and recomputing the filterbanks may lead to a bit of "dirty" vectors at the beginning and end

I agree. Also, for more accurate history clearing it's best to use chunk sizes that maps to whole number of encoder features, i.e. 16000(Hz) * X (chunk size in s) / 160 (hop size) / 8 is a whole number