ComfyUI Audio Segment Plugin (ComfyUI-AudioSegment)

中文说明 (Chinese Readme)

This is a custom node suite for ComfyUI designed for advanced audio processing. It allows users to segment a long audio file into multiple shorter clips in two ways:

1. Silence-based: Intelligently splits the audio into a list of shorter clips based on silence detection and a set maximum duration.
2. Paragraph-based: Accurately aligns and splits the audio based on a provided reference text using AI-powered speech recognition.

Features

This plugin suite includes five nodes that work together:

1. Audio Segment By Paragraph

This is the new core feature node. It accurately segments audio by aligning it with user-provided text paragraphs using a Whisper ASR model.

• Inputs:
  • audio: The original ComfyUI audio object.
  • text: A multiline string where each line represents a paragraph. The node splits paragraphs based on newlines.
  • model: A dropdown menu to select the Whisper model to use. It supports all standard models (e.g., base, large-v3), local .pt files, and Hugging Face model folders located in ComfyUI/models/stt/whisper/. If a selected model is not found locally, it will be downloaded automatically.
• Outputs:
  • AudioClip list: A custom list of AudioClip objects. Each AudioClip contains the audio (as a ComfyUI audio object), startTime (in milliseconds), endTime (in milliseconds), and text (the corresponding original text paragraph).
  • size: The number of AudioClip objects in the list.
  • srt: An SRT-formatted subtitle string generated from the segmentation results.

2. Select AudioClip From List

This node allows you to select a specific AudioClip from the list generated by Audio Segment By Paragraph for further processing.

• Inputs:
  • audio_clip_list: The list of AudioClip objects from the Audio Segment By Paragraph node.
  • index: The index of the AudioClip to select (starting from 0).
• Outputs:
  • audio: The ComfyUI audio object from the selected AudioClip.
  • text: The text string from the selected AudioClip.

3. Audio Segmenter (Silence-based)

The original core node, which segments audio based on silence.

• Inputs:
  • audio: The original ComfyUI audio object.
  • max_length_s: The maximum length of each output clip in seconds.
  • silence_thresh_db: The silence threshold in decibels (dB). Anything below this volume is considered silence.
• Outputs:
  • audio_list: A list of individual audio clips.
  • count: The number of audio clips in the list.

4. Select Audio From List

This node allows you to select a specific audio clip from the list generated by Audio Segmenter.

• Inputs:
  • audio_list: The audio list from the Audio Segmenter node.
  • index: The index of the audio clip to select (starting from 0).
• Outputs:
  • AUDIO: The single audio clip at the specified index.

5. Audio List to Batch

This node converts a list of audio clips back into the standard ComfyUI batch audio format, making it compatible with other nodes that require batch input (like ASR models).

• Inputs:
  • audio_list: The audio list from the Audio Segmenter node.
• Outputs:
  • AUDIO: A batched audio object containing all clips from the list. All clips are padded to match the length of the longest clip.

Installation

1. Navigate to the custom_nodes directory: Open a terminal and go to the custom_nodes folder in your ComfyUI installation directory.

   cd path/to/ComfyUI/custom_nodes/

2. Clone the repository: Clone this repository into the custom_nodes directory.

   git clone git@gitee.com:dreamidea/comfyui-audio-segment.git

3. Install dependencies: Navigate into the plugin directory and install the required libraries using the pip from your ComfyUI environment.

   cd comfyui-audio-segment
   pip install -r requirements.txt

   (Note: If you are running ComfyUI in a virtual environment (e.g., venv or conda), make sure to activate it first.)

4. Restart ComfyUI: Completely shut down and restart ComfyUI. You will then find the new nodes under the "Audio/Segmentation" category.

Implementation Details

Silence-Based Segmentation

The original segmentation logic is in audio_segmenter.py and relies on the pydub library. It first splits the audio stream based on silence detection and then intelligently merges or splits the resulting chunks to ensure they are as close as possible to the user-defined maximum length.

Paragraph-Based Segmentation

The new core logic resides in audio_paragraph_segmenter.py and follows this workflow:

1. Speech Recognition (ASR):

   • It uses the openai-whisper library for high-accuracy speech recognition, complete with word-level timestamps.
   • It supports dynamic model loading for various Whisper models, including standard versions (e.g., base, large-v3), local .pt files, and Hugging Face models stored in ComfyUI/models/stt/whisper/. The system automatically downloads models if they are not found locally.

2. Text Alignment:

   • To handle discrepancies between the user's reference text and the ASR results (e.g., homophones, traditional/simplified Chinese, missing words), the plugin implements a robust text alignment algorithm.
   • The algorithm first uses the opencc-python-reimplemented library to normalize both the reference text and the ASR output to simplified Chinese.
   • It then uses Python's difflib.SequenceMatcher to find all matching blocks between the two texts. By calculating a cumulative similarity score, it can tolerate scattered ASR errors and accurately determine the start and end times for each paragraph.

3. Audio Slicing and Data Encapsulation:

   • Based on the timestamps from the alignment algorithm, pydub is used to precisely slice the original audio.
   • Each audio slice is then encapsulated into a custom AudioClip object, along with its corresponding text paragraph and start/end times, making it easy to use with downstream nodes like Select AudioClip From List.