Real-Time Voice Agent Orchestration Engine

Repository: Agent_Builder_v2

A real-time voice-agent orchestration prototype combining REST-based agent configuration, document ingestion, local FAISS retrieval, WebSocket audio streaming, speech-to-text, LLM reasoning, and text-to-speech.

The original concept was an agent-builder platform where users could configure multiple agents with text and voice capabilities. The current repository is best understood as the real-time voice + RAG orchestration layer of that broader concept.

Project status: Independent MVP/prototype. It is not presented as a production-scale multi-user platform.

What it demonstrates

• bidirectional WebSocket audio
• authenticated streaming connections
• STT → retrieval → LLM → TTS orchestration
• per-agent knowledge ingestion
• semantic chunking and embeddings
• local FAISS retrieval
• REST APIs for agent/chat configuration
• separation between controllers and services

Architecture

Runtime voice pipeline

User Microphone
      |
      | audio chunks
      v
+-----------------------+
| WebSocket Gateway     |
| connection auth       |
+-----------+-----------+
            |
            v
+-----------------------+
| Speech-to-Text        |
| streaming transcript  |
+-----------+-----------+
            |
            v
+-----------------------+
| Agent Controller      |
| context + retrieval   |
+-----+-------------+---+
      |             |
      |             v
      |       +-------------+
      |       | FAISS Index |
      |       | vector RAG  |
      |       +-------------+
      |             |
      +------+------+
             |
             v
       +-----------+
       |    LLM    |
       +-----+-----+
             |
             v
       +-----------+
       |    TTS    |
       +-----+-----+
             |
             v
      Audio to Client

Knowledge-ingestion pipeline

PDF / Text Document
        |
        v
Document Upload
        |
        v
Parsing / Normalization
        |
        v
Chunking
        |
        v
Embeddings
        |
        v
FAISS Index
        |
        +----> Runtime retrieval

Runtime workflow

1. Client establishes a WebSocket connection.
2. Connection is authenticated before audio buffering.
3. Audio chunks are streamed to the backend.
4. STT converts speech into text.
5. Agent controller combines conversation context with retrieved knowledge.
6. LLM generates the response.
7. TTS converts the response back into audio.
8. Audio is streamed back to the client.

Why local FAISS?

For the MVP, local FAISS keeps retrieval simple and avoids adding a managed vector-database dependency.

That trade-off also creates a clear scaling boundary: in-memory indexes are straightforward for a small number of agents but increasingly expensive as the number and size of knowledge bases grow.

Latency considerations

The system is designed around real-time interaction rather than ordinary request/response chat.

The repository reports an under-800ms pipeline measurement/target for its configured path. This is an environment-specific prototype figure, not a production SLA.

Security boundary

WebSocket authentication is performed before audio buffering.

WebSocket Upgrade
       |
       v
Authentication
       |
       +---- invalid ----> reject
       |
       v
Audio streaming

Engineering decisions

WebSockets for voice

Continuous bidirectional audio requires persistent streaming transport rather than independent HTTP requests.

Retrieval outside the LLM

Uploaded knowledge is retrieved at runtime and supplied as context rather than being treated as model memory.

Service-oriented Node.js structure

Controllers are separated from STT, TTS, FAISS, and agent services, making individual pipeline components replaceable.

Local-first MVP

Distributed infrastructure was intentionally avoided while experimenting with the core voice orchestration problem.

Known limitations

• Runtime is centered around a single Node.js server.
• FAISS indexes are local/in-memory.
• Heavy concurrent audio processing can pressure the Node.js process.
• Third-party STT, TTS, and LLM latency affects end-to-end response time.
• No distributed session management is demonstrated.
• No production-scale traffic is claimed.

Production evolution

A production-scale version could introduce managed/distributed vector storage, worker processes, distributed WebSocket/session infrastructure, durable agent configuration, provider fallback, tenant isolation, rate limiting, latency/cost telemetry, and automated voice/retrieval evaluations.

What I learned

Real-time voice AI is not simply an LLM application with a microphone attached. Transport, buffering, provider latency, retrieval, state management, and failure handling all affect whether the conversation actually feels real-time.

Current scope

The most precise description of this repository is:

A real-time voice-agent orchestration engine with RAG capabilities, built as the core of a broader agent-builder concept.

Local demo

The repository includes examples/voice-chat-client.html for initiating a local voice session with a configured agent.

Links

• GitHub: https://github.com/Reh1t/Agent_Builder_v2
• LinkedIn: https://www.linkedin.com/in/rehantariqbhatti