Streaming LLM Response with Real-Time TTS

A beginner-friendly Python project that demonstrates how to:

Stream responses from a local LLM
Display generated text in real time
Detect complete sentences from streaming output
Convert generated sentences into speech
Run TTS in the background
Clean Markdown and unwanted characters before speech
Process audio using FFmpeg
Add fade-in and fade-out effects
Control speech speed
Run everything locally on CPU

1. Project Overview

Normally, when we call an LLM using invoke(), Python waits until the complete response is generated.

Example:

response = llm.invoke("Hello")
print(response.content)

The flow looks like this:

User Prompt
    ↓
LLM starts generating
    ↓
Wait...
    ↓
Wait...
    ↓
Complete response generated
    ↓
Display full response

This project uses streaming instead.

With streaming, the response is displayed piece by piece while the model is still generating.

User Prompt
    ↓
LLM starts generating
    ↓
"Hello"
    ↓
" there"
    ↓
"! How"
    ↓
" can I"
    ↓
" help?"

At the same time, complete sentences are sent to a Text-to-Speech system.

The final architecture is:

User
  ↓
ChatOllama
  ↓
Streaming LLM Response
  ↓
Token / Chunk Buffer
  ↓
Sentence Detection
  ↓
Text Cleaning
  ↓
TTS Queue
  ↓
Piper TTS
  ↓
FFmpeg Audio Processing
  ↓
Speaker

2. Features

This project includes:

Local LLM using Ollama
Streaming response using ChatOllama
Local CPU-based TTS using Piper
Background TTS worker thread
Thread-safe queue
Sentence-by-sentence speech generation
Markdown cleaning
Emoji removal
URL removal
FFmpeg audio processing
TTS speed control
Fade-in effect
Fade-out effect
Graceful application shutdown

3. Technologies Used

Python

Python is used to connect all components together.

Recommended version:

Python 3.10+

Ollama

Ollama runs the LLM locally on your computer.

Example model:

gemma3:4b

LangChain Ollama

The langchain-ollama package allows Python to communicate with Ollama using LangChain.

Piper TTS

Piper is a lightweight local Text-to-Speech engine.

It is useful for:

CPU-based TTS
Offline speech generation
Fast synthesis
Local AI assistants

FFmpeg

FFmpeg processes the generated audio.

In this project it is used for:

Speech speed control
Fade-in
Fade-out
Smooth audio processing

SoundDevice

The sounddevice package plays generated audio through the computer speakers.

4. Project Structure

Recommended folder structure:

streaming-llm-tts/
│
├── main.py
│
├── streaming_tts.py
│
├── requirements.txt
│
├── README.md
│
└── voices/
    ├── en_US-kristin-medium.onnx
    └── en_US-kristin-medium.onnx.json

Explanation:

main.py

Handles:

User input
LLM connection
LLM streaming
Sending tokens to TTS

streaming_tts.py

Handles:

Sentence buffering
Text cleaning
TTS queue
Piper speech generation
FFmpeg processing
Audio playback

voices/

Contains the Piper voice model files.

5. Installation

Step 1: Clone the Repository

git clone https://github.com/Rahul3998/Streaming_AI_Reponse_and_TTS.git
cd Streaming_AI_Reponse_and_TTS

Step 2: Create a Virtual Environment

Windows

python -m venv .venv

Activate it:

.venv\Scripts\activate

Linux / macOS

python3 -m venv .venv
source .venv/bin/activate

Step 3: Install Python Dependencies

pip install langchain-ollama
pip install piper-tts
pip install numpy
pip install sounddevice

Or install everything using:

pip install -r requirements.txt

6. requirements.txt

Create a file named:

requirements.txt

Add:

langchain-ollama
piper-tts
numpy
sounddevice

Then install:

pip install -r requirements.txt

7. Install Ollama

Install Ollama on your computer.

After installation, download the model:

ollama pull gemma3:4b

Check installed models:

ollama list

Test the model:

ollama run gemma3:4b

8. Install FFmpeg

This project uses FFmpeg for audio processing.

Example Windows location:

C:\your\location\ffmpeg.exe

Your Python configuration can use:

FFMPEG_PATH = r"C:\your\location\ffmpeg.exe"

Why use r before the string?

r"C:\your\location\ffmpeg.exe"

This creates a raw string and prevents Windows backslashes from being interpreted as escape characters.

9. Add a Piper Voice

Create a folder:

voices/

Add both Piper voice files:

voices/
├── en_US-kristin-medium.onnx
└── en_US-kristin-medium.onnx.json

Both files are important.

The .onnx file contains the voice model.

The .json file contains voice configuration information.

10. Understanding Normal LLM Response

A normal LLM request uses:

from langchain_ollama import ChatOllama

llm = ChatOllama(
    model="gemma3:4b"
)

response = llm.invoke(
    "Hello"
)

print(
    response.content
)

Problem:

User sends prompt
      ↓
Wait for complete generation
      ↓
Display complete answer

For long responses, the user sees nothing until generation finishes.

11. Understanding Streaming LLM Response

Instead of:

llm.invoke()

we use:

llm.stream()

Example:

from langchain_ollama import ChatOllama

llm = ChatOllama(
    model="gemma3:4b"
)

user_prompt = input(
    "USER :: "
)

print(
    "ASSISTANT :: ",
    end="",
    flush=True
)

for chunk in llm.stream(
    user_prompt
):
    print(
        chunk.content,
        end="",
        flush=True
    )

print()

The important code is:

for chunk in llm.stream(user_prompt):

Every chunk contains a small part of the generated response.

We access the text using:

chunk.content

12. Why Use `end=""`?

Normally:

print("Hello")
print("World")

Output:

Hello
World

But streaming requires continuous text.

So we use:

print(
    chunk.content,
    end=""
)

Output:

Hello World

instead of:

Hello
World

13. Why Use `flush=True`?

Streaming output should appear immediately.

print(
    chunk.content,
    end="",
    flush=True
)

flush=True tells Python:

Display this output immediately.
Do not wait for the output buffer.

This makes the terminal response feel real-time.

14. Why Not Send Every Token Directly to TTS?

Suppose the LLM generates:

"Hel"
"lo"
" Rah"
"ul"
"."

If every chunk is sent directly to TTS:

TTS("Hel")
TTS("lo")
TTS("Rah")
TTS("ul")

The speech will sound broken and unnatural.

Instead, we create a text buffer.

"Hel"
   ↓
Buffer = "Hel"

"lo"
   ↓
Buffer = "Hello"

" Rahul"
   ↓
Buffer = "Hello Rahul"

"."
   ↓
Buffer = "Hello Rahul."

Now a complete sentence exists.

We send:

"Hello Rahul."

to TTS.

15. Sentence Detection

A simple sentence extraction function:

import re


def extract_sentences(buffer):

    sentences = []

    pattern = r"(.+?[.!?])(?:\s+|$)"

    matches = list(
        re.finditer(
            pattern,
            buffer
        )
    )

    last_end = 0

    for match in matches:

        sentence = (
            match
            .group(1)
            .strip()
        )

        if sentence:

            sentences.append(
                sentence
            )

        last_end = match.end()

    remaining = buffer[last_end:]

    return sentences, remaining

Example input:

Hello Rahul. How are

Output:

sentences = [
    "Hello Rahul."
]

Remaining buffer:

How are

When more text arrives:

you today?

The buffer becomes:

How are you today?

Now it can be sent to TTS.

16. Why Clean Text Before TTS?

LLMs often generate Markdown.

Example:

As an AI, I don't really *have* days in the same way humans do!

The * characters are useful for Markdown formatting.

But they should not be sent to TTS.

Clean output:

As an AI, I don't really have days in the same way humans do!

Another example:

**Hello Rahul!** 👋

Clean version:

Hello Rahul!

17. Text Cleaning Function

import re


def clean_text_for_tts(text: str) -> str:

    if not text:
        return ""

    # Markdown links:
    # [Google](url) -> Google
    text = re.sub(
        r"\[([^\]]+)\]\([^)]+\)",
        r"\1",
        text
    )

    # Remove URLs
    text = re.sub(
        r"https?://\S+|www\.\S+",
        "",
        text
    )

    # Remove Markdown stars
    text = re.sub(
        r"\*+",
        "",
        text
    )

    # Remove headings
    text = re.sub(
        r"#+\s*",
        "",
        text
    )

    # Remove underscores
    text = re.sub(
        r"_+",
        " ",
        text
    )

    # Remove backticks
    text = re.sub(
        r"`+",
        "",
        text
    )

    # Remove brackets
    text = re.sub(
        r"[\[\]{}]",
        "",
        text
    )

    # Remove decorative symbols
    text = re.sub(
        r"[~^|<>]",
        "",
        text
    )

    # Remove emojis
    text = re.sub(
        r"[\U0001F300-\U0001FAFF]",
        "",
        text
    )

    # Normalize spaces
    text = re.sub(
        r"\s+",
        " ",
        text
    )

    return text.strip()

18. Why Use a Queue?

The LLM and TTS have different speeds.

Example:

LLM generates sentence 1
        ↓
LLM generates sentence 2
        ↓
LLM generates sentence 3

But TTS may still be speaking sentence 1.

Without a queue, the application becomes difficult to manage.

We use:

import queue

tts_queue = queue.Queue()

Now:

LLM
 ↓
Sentence 1
 ↓
Queue

LLM
 ↓
Sentence 2
 ↓
Queue

LLM
 ↓
Sentence 3
 ↓
Queue

The TTS worker processes them in order:

Sentence 1
    ↓
Sentence 2
    ↓
Sentence 3

19. Why Use a Background Thread?

If TTS runs in the main thread:

Generate sentence
      ↓
Speak sentence
      ↓
Wait
      ↓
Continue LLM

This blocks streaming.

Instead:

Main Thread
    ↓
LLM Streaming

Background Thread
    ↓
TTS Processing

Python example:

worker = threading.Thread(
    target=tts_worker,
    daemon=True
)

worker.start()

Now LLM streaming and TTS processing can overlap.

20. Piper TTS Generation

Load the Piper voice:

from piper import PiperVoice

voice = PiperVoice.load(
    "voices/en_US-kristin-medium.onnx"
)

Generate speech:

for audio_chunk in voice.synthesize(
    text
):

    audio = np.frombuffer(
        audio_chunk.audio_int16_bytes,
        dtype=np.int16
    )

Piper may return multiple audio chunks.

For smoother processing, combine them:

audio_parts = []

for audio_chunk in voice.synthesize(text):

    chunk_audio = np.frombuffer(
        audio_chunk.audio_int16_bytes,
        dtype=np.int16
    )

    audio_parts.append(
        chunk_audio.copy()
    )

complete_audio = np.concatenate(
    audio_parts
)

21. Why Combine Piper Audio Chunks?

A sentence may be generated as multiple internal chunks.

If fade-in and fade-out are applied separately to every small chunk:

Chunk 1
fade in → audio → fade out

Chunk 2
fade in → audio → fade out

Chunk 3
fade in → audio → fade out

This can sound unnatural.

Better approach:

Chunk 1
Chunk 2
Chunk 3
   ↓
Combine
   ↓
Complete sentence audio
   ↓
Apply FFmpeg once

This produces smoother speech.

22. FFmpeg Audio Processing

The project uses:

atempo=0.85

This changes speech tempo.

Examples:

1.00 = normal speed
0.95 = slightly slower
0.85 = slower
1.10 = faster
1.25 = much faster

Current project setting:

TTS_SPEED = 0.85

23. Fade-In and Fade-Out

Fade-in gradually increases volume at the beginning.

silence
  ↓
low volume
  ↓
medium volume
  ↓
normal volume

Fade-out gradually reduces volume at the end.

normal volume
  ↓
medium volume
  ↓
low volume
  ↓
silence

Example configuration:

FADE_IN_DURATION = 0.08
FADE_OUT_DURATION = 0.12

This means:

Fade-in  = 80 milliseconds
Fade-out = 120 milliseconds

24. StreamingTTS Class

The recommended design is to move all TTS logic into one reusable class:

class StreamingTTS:

    def __init__(
        self,
        model_path,
        ffmpeg_path,
        speed=0.85
    ):
        pass

    def add_token(
        self,
        token
    ):
        pass

    def flush(
        self
    ):
        pass

    def wait_until_done(
        self
    ):
        pass

    def interrupt(
        self
    ):
        pass

    def close(
        self
    ):
        pass

The main application should not need to understand:

Piper internals
FFmpeg commands
Queue management
Thread management
Sentence extraction
Text cleaning

The class handles these responsibilities.

25. Using StreamingTTS from main.py

Example:

from langchain_ollama import ChatOllama

from streaming_tts import StreamingTTS


llm = ChatOllama(
    model="gemma3:4b"
)


tts = StreamingTTS(
    model_path=(
        "voices/"
        "en_US-kristin-medium.onnx"
    ),
    ffmpeg_path=(
        r"C:\your\location\ffmpeg.exe"
    ),
    speed=0.85,
    fade_in_duration=0.08,
    fade_out_duration=0.12,
    debug=True
)


try:

    while True:

        user_prompt = input(
            "\nUSER :: "
        )

        if user_prompt.lower() in {
            "exit",
            "quit"
        }:
            break

        print(
            "ASSISTANT :: ",
            end="",
            flush=True
        )

        for chunk in llm.stream(
            user_prompt
        ):

            token = chunk.content

            # Display immediately
            print(
                token,
                end="",
                flush=True
            )

            # Send to TTS
            tts.add_token(
                token
            )

        # Send remaining text
        tts.flush()

        print()


finally:

    tts.close(
        wait=True
    )

26. Understanding `add_token()`

During streaming:

for chunk in llm.stream(user_prompt):

    token = chunk.content

    tts.add_token(token)

Suppose tokens arrive like:

"Hello"
" Rahul"
". How"
" are"
" you?"

The class internally builds:

Hello Rahul.

It sends that sentence to TTS.

Remaining buffer:

How are you?

Then that sentence is also sent to TTS.

27. Understanding `flush()`

Sometimes the LLM response does not end with punctuation.

Example:

That is the answer

There is no:

.
?
!

So the sentence detector may keep it in the buffer.

After streaming finishes, call:

tts.flush()

This sends any remaining text to the TTS queue.

28. Understanding `close()`

When the application exits:

tts.close(
    wait=True
)

This can:

Flush remaining text
Wait for queued speech
Stop the worker safely
Clean up resources

29. Complete Runtime Flow

Suppose the user enters:

USER :: Explain Python simply

The LLM starts generating:

Python is a programming language.

The terminal displays tokens immediately:

Python
Python is
Python is a
Python is a programming
Python is a programming language.

Internally:

Token 1
   ↓
Buffer

Token 2
   ↓
Buffer

Token 3
   ↓
Buffer

"." detected
   ↓
Complete sentence

The sentence:

Python is a programming language.

goes through:

clean_text_for_tts()
        ↓
Piper
        ↓
Raw PCM audio
        ↓
FFmpeg
        ↓
Speed 0.85
        ↓
Fade-in
        ↓
Fade-out
        ↓
sounddevice
        ↓
Speaker

30. Important Limitation

In a simple one-worker architecture:

TTS Worker
    ↓
Generate audio
    ↓
Play audio
    ↓
sd.wait()
    ↓
Generate next audio

While sentence 1 is playing, sentence 2 cannot be synthesized by the same worker.

For even lower latency, use two workers:

LLM Stream
    ↓
Text Queue
    ↓
TTS Generation Worker
    ↓
Audio Queue
    ↓
Audio Playback Worker
    ↓
Speaker

This allows:

Sentence 1 is playing

while:

Sentence 2 is being generated

This is a better architecture for real-time AI assistants.

31. Future Improvements

Possible improvements:

Separate synthesis and playback workers
Add interruption support
Add microphone input
Add Voice Activity Detection
Add Speech-to-Text
Add conversation memory
Add FastAPI backend
Add WebSocket streaming
Add React frontend
Add Streamlit interface
Add configurable voices
Add volume normalization
Add crossfade between sentences
Add smarter sentence chunking
Add abbreviations handling
Add number-to-speech normalization

A complete voice assistant architecture could be:

Microphone
    ↓
Speech-to-Text
    ↓
User Prompt
    ↓
Local LLM
    ↓
Streaming Response
    ↓
Sentence Buffer
    ↓
Text Cleaner
    ↓
Piper TTS
    ↓
FFmpeg
    ↓
Speaker

32. Common Errors

Ollama model not found

Error:

model "gemma3:4b" not found

Solution:

ollama pull gemma3:4b

Piper voice not found

Error:

FileNotFoundError

Check:

voices/
├── en_US-kristin-medium.onnx
└── en_US-kristin-medium.onnx.json

FFmpeg not found

Error:

The system cannot find the file specified

Check:

ffmpeg_path = (
    r"C:\your\location\ffmpeg.exe"
)

No audio output

Check available devices:

import sounddevice as sd

print(
    sd.query_devices()
)

TTS sounds broken

Do not send every LLM token directly to Piper.

Bad:

for chunk in llm.stream(prompt):

    voice.synthesize(
        chunk.content
    )

Better:

LLM chunks
    ↓
Buffer
    ↓
Complete sentence
    ↓
Piper

33. Key Concepts Learned

After completing this project, you should understand:

What LLM streaming is
Difference between invoke() and stream()
How token/chunk streaming works
Why buffering is required
How sentence detection works
Why queues are useful
Why background threads are useful
How local TTS works
How Piper generates audio
How FFmpeg processes audio
How fade-in and fade-out work
How to connect an LLM stream with a TTS pipeline

34. Summary

The complete system works like this:

User Prompt
    ↓
ChatOllama.stream()
    ↓
Receive chunk.content
    ↓
Print chunk immediately
    ↓
Add chunk to text buffer
    ↓
Detect complete sentence
    ↓
Clean Markdown and symbols
    ↓
Add sentence to TTS queue
    ↓
Background TTS worker
    ↓
Piper speech generation
    ↓
Combine audio chunks
    ↓
FFmpeg processing
    ↓
Speed = 0.85
    ↓
Fade-in
    ↓
Fade-out
    ↓
Play through speaker

The main idea is simple:

Do not wait for the complete LLM response. Stream the response, collect meaningful sentences, and speak them in the background.

This creates a much more responsive foundation for:

AI voice assistants
AI teachers
Local chatbots
Desktop assistants
Accessibility tools
Interactive learning systems
Real-time conversational AI

Contributions

Contributions are welcome.

Possible contribution areas:

Better sentence detection
More natural TTS
Lower latency
Async support
FastAPI integration
WebSocket streaming
React frontend
Better interruption handling

Name		Name	Last commit message	Last commit date
voices		voices
.gitignore		.gitignore
1_llm_streaming.py		1_llm_streaming.py
2_streaming_llm_and_tts.py		2_streaming_llm_and_tts.py
README.md		README.md
main.py		main.py
requirements.txt		requirements.txt
streaming_tts.py		streaming_tts.py

Latest commit

History

Folders and files

Repository files navigation

Streaming LLM Response with Real-Time TTS

1. Project Overview

2. Features

3. Technologies Used

Python

Ollama

LangChain Ollama

Piper TTS

FFmpeg

SoundDevice

4. Project Structure

5. Installation

Step 1: Clone the Repository

Step 2: Create a Virtual Environment

Windows

Linux / macOS

Step 3: Install Python Dependencies

6. requirements.txt

7. Install Ollama

8. Install FFmpeg

9. Add a Piper Voice

10. Understanding Normal LLM Response

11. Understanding Streaming LLM Response

12. Why Use end=""?

13. Why Use flush=True?

14. Why Not Send Every Token Directly to TTS?

15. Sentence Detection

16. Why Clean Text Before TTS?

17. Text Cleaning Function

18. Why Use a Queue?

19. Why Use a Background Thread?

20. Piper TTS Generation

21. Why Combine Piper Audio Chunks?

22. FFmpeg Audio Processing

23. Fade-In and Fade-Out

24. StreamingTTS Class

25. Using StreamingTTS from main.py

26. Understanding add_token()

27. Understanding flush()

28. Understanding close()

29. Complete Runtime Flow

30. Important Limitation

31. Future Improvements

32. Common Errors

Ollama model not found

Piper voice not found

FFmpeg not found

No audio output

TTS sounds broken

33. Key Concepts Learned

34. Summary

Contributions

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

12. Why Use `end=""`?

13. Why Use `flush=True`?

26. Understanding `add_token()`

27. Understanding `flush()`

28. Understanding `close()`