Multimodal temporal wrapper

[kjtheron]

I would like to use a temporal multi modal dataset to fine tune terrmind. However, i am having trouble with data shapes when adding the temporal wrapper functionality to the multi modal example notebook (specifically the flooding extent mapping using S1 and S2 inputs).

Any guidance or examples on how to use the temporal wrapper to with multiple modalities?

[blumenstiel]

Hi @kjtheron, do you use the generic multimodal datamodule or a custom one? with the generic one, I implemented some things but I never really had a dataset and the time to test it, so please share any errors!

If you have a custom one, you want to have the modalities split into a dictionary like {"mod1": tensor.tensor(...), "mod2": ...} with the tensor shape being [batch, channel, time, H, W]. Then it should work with the temporal wrapper.

Let me know if that helps. Otherwise, could you test if the temporal wrapper works for you for a single modality?

[kjtheron]

Hi @blumenstiel, thanks for your response.

I am using this notebook: https://github.com/IBM/terramind/blob/main/notebooks/terramind_v1_small_sen1floods11.ipynb

And trying to add the temporal wrapper code from: https://github.com/IBM/terramind/blob/main/notebooks/terramind_v1_small_multitemporal_crop.ipynb

However, there is constant data shape issues. I had to eventually create a new utility to manage the multimodal temporal data for the backbone. This utility manages the normalization and shape (the expand_temporal_dimension flag). But I am sure there are better ways of doing this.

[blumenstiel]

Was the issue in the data loading or in the backbone? Just trying to understand which part I have to fix:) If you already have a good fix, feel free to post/create an pr for TT.