Skip to content

plot_stimuli crashes on silent / speechless video clips #46

@sepehrheravi

Description

@sepehrheravi

Describe the bug

PlotBrain.plot_timesteps(..., show_stimuli=True) crashes when the input
video clip has no speech or no audio track, because
BasePlotBrain.plot_stimuli assumes both (a) get_audio(...) returns a
non-None object and (b) every segment.events dataframe contains a
type column (i.e. at least one Word event).

Both assumptions break in realistic workflows — e.g. screen recordings,
silent clips, or any video where WhisperX produces zero words. The model
itself handles the missing-modality case correctly (main.py auto-drops
the text extractor when no Word events exist), so this is strictly
a plotting-side bug.

Reproduction

import pandas as pd
from pathlib import Path
from tribev2.demo_utils import TribeModel, get_audio_and_text_events

model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
df = get_audio_and_text_events(
    pd.DataFrame([{
        "type": "Video", "filepath": "silent_clip.mp4",
        "start": 0, "timeline": "default", "subject": "default",
    }]),
    audio_only=True,
)
preds, segments = model.predict(events=df)
model._plotter.plot_timesteps(preds[:5], segments=segments[:5], show_stimuli=True)
# AttributeError: 'NoneType' object has no attribute 'to_soundarray'
# OR  KeyError: 'type'  when events is None / missing the column 

Expected behaviour

plot_stimuli should gracefully skip the audio waveform / word-overlay
subplots when those modalities are not present in the segments, mirroring
the model's existing "auto-drop missing extractor" behaviour.

Environment

OS: Ubuntu 24.04 aarch64 (NVIDIA DGX Spark, GB10)
Python 3.12.3
torch 2.11.0+cu130 (Blackwell sm_121)

Proposed fix

Two defensive guards in: tribev2/plotting/base.py::BasePlotBrain.plot_stimuli

Only plot the audio waveform when get_audio(...) returns non-None.
Only iterate words when segment.events is not None and the dataframe has a type column.
No change in behaviour when audio / words ARE present.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions