Digital Human

A local-first, realtime Vietnamese-English assistant with WebRTC media, hands-free turn detection, interruption, streaming speech, and a SyncTalk_2D avatar renderer.

What works

• Bidirectional browser audio and synchronized avatar audio/video over WebRTC
• Hands-free voice activity detection and immediate barge-in
• Local faster-whisper ASR, vLLM/Liquid LFM2 LLM, and VieNeu-TTS adapters
• External, persistent SyncTalk_2D research worker
• One bounded GPU session with typed REST and WebSocket events
• Responsive React/TypeScript interface with dark and light themes
• Mock mode for development and CI without models or a GPU

Quick start

Requirements: Python 3.10–3.12, UV, Node.js 20+, npm, and FFmpeg.

cp .env.example .env
make install

Run the gateway and web client in separate terminals:

make dev-api
make dev-web

Open http://localhost:5173. Mock mode exercises the same media, session, event, and interruption paths while replacing model output with deterministic placeholders.

Local GPU mode

Target: Linux, 64 GB RAM, and one NVIDIA GPU with 16 GB VRAM.

1. Install AI dependencies with make install-ai and the pinned renderer with make install-synctalk.
2. Download the official LiquidAI/LFM2-2.6B checkpoint and start it with make llm-up.
3. Prepare and train an identity-specific SyncTalk_2D avatar, then run make synctalk-worker.
4. Set DIGITAL_HUMAN_MOCK_MODE=false in .env and run uv run digital-human.

SyncTalk_2D is not vendored or relicensed by this Apache-2.0 repository. It is an external research dependency supplied by the user.

Run every real service with only the avatar mocked:

docker compose up --build

Open http://localhost:8010. Later starts can use docker compose up; stop with Ctrl+C and remove the services with docker compose down.

Avatar onboarding

Use a consented, well-lit 4–7 minute capture with one visible speaker and a stationary camera.

make avatar-prepare AVATAR=may VIDEO=/path/to/may.mp4
make avatar-train AVATAR=may
make avatar-verify AVATAR=may

avatar-train stops the Compose stack first and defaults to batch size 1, zero data-loader workers, and CUDA mixed precision. This avoids competing with vLLM and prevents upstream's fixed 16/32-worker loaders from exhausting RAM.

Generated media, checkpoints, and biometric avatar assets are excluded from Git.

Quality checks

make test       # Python and component tests
make check      # Ruff, MyPy, Bandit, tests, and production web build
make gpu-smoke  # opt-in local model integration tests

See the complete local GPU deployment guide, architecture, API reference, and environment setup.