Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions README.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,22 @@

*A local LLM and a zero-shot voice-cloning TTS take turns on one 8 GB GPU. Use them as local HTTP services, or as an offline voice server for ESP32 boards running xiaozhi firmware.*

[Results](#results) · [How it works](#how-it-works) · [Requirements](#requirements) · [Quick start](#quick-start) · [Connect a board](#connect-a-board-mode-b) · [Known limitations](#known-limitations) · [Upstream fixes](#fixes-sent-upstream) · [Docs](#documentation) · [32 pitfalls](docs/pitfalls.md)

[![CI](https://github.com/korexil/gpu-duet/actions/workflows/ci.yml/badge.svg)](https://github.com/korexil/gpu-duet/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/korexil/gpu-duet)](https://github.com/korexil/gpu-duet/releases/latest)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
[![Bilibili](https://img.shields.io/badge/Demo_video-Bilibili-00A1D6)](https://www.bilibili.com/video/BV15Cpr6eEMx)

<a href="https://www.bilibili.com/video/BV15Cpr6eEMx"><img src="docs/img/demo-video.jpg" alt="Demo video: same RTX 4060 Laptop, same ESP32 board, going from a stall on every sentence to long answers with no gaps (Bilibili, 3 min 28 s)" width="640"></a>
| First word | Total gaps | VRAM |
|:-:|:-:|:-:|
| 6.2&nbsp;→&nbsp;**2.1&nbsp;s** | 4.22&nbsp;→&nbsp;**0.17&nbsp;s** | **7.3&nbsp;GB**&nbsp;/&nbsp;8&nbsp;GB |

<a href="https://www.bilibili.com/video/BV15Cpr6eEMx"><img src="docs/img/before-after.en.webp" alt="Live recording, same laptop, same question. Before: first word at 5.95 s, then a stall in almost every sentence. After: first word at 1.92 s, one 0.1 s stall in the whole answer" width="760"></a>

<sub>Live recording at 2× speed, same laptop and same question; red bars are where the audio stalled. Click for the full video (Bilibili, 3 min 28 s, in Chinese). The questions were asked live and their audio was later replaced with a synthesized voice; the board's answers are the live audio.</sub>

<sub>Real-device recording. The questions were asked live; their audio was later replaced with a synthesized voice. The board's answers are the live audio.</sub>
**Try it without a board:** Windows + an 8 GB NVIDIA GPU + a 3–10 s voice recording. Follow [Quick start](#quick-start) steps 1–5; the last step prints the total gap time.

## What it does

Expand Down Expand Up @@ -118,7 +126,7 @@ python tools\turn_taking_sim.py --mode turns --question "说说你最喜欢的

At this point you have mode A: your own programs call `http://127.0.0.1:18766/stream` to synthesize speech. API reference: [tts-service/README.md](tts-service/README.md).

**6. Connect a board (mode B).**
### Connect a board (mode B)

```powershell
pip install -r standalone\requirements.txt
Expand Down
14 changes: 11 additions & 3 deletions README.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,22 @@

*ローカル LLM とゼロショット音声クローン TTS が、8 GB の GPU 1 枚を交代で使います。ローカル HTTP サービスとして使うことも、xiaozhi ファームウェアを載せた ESP32 ボード向けのオフライン音声サーバーとして使うこともできます。*

[効果](#効果) · [仕組み](#仕組み) · [必要なもの](#必要なもの) · [クイックスタート](#クイックスタート) · [ボードをつなぐ](#ボードをつなぐ使い方-b) · [既知の制限](#既知の制限) · [上流への修正](#上流への修正) · [ドキュメント](#ドキュメント) · [ハマりどころ 32 件](docs/pitfalls.md)

[![CI](https://github.com/korexil/gpu-duet/actions/workflows/ci.yml/badge.svg)](https://github.com/korexil/gpu-duet/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/korexil/gpu-duet)](https://github.com/korexil/gpu-duet/releases/latest)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
[![Bilibili](https://img.shields.io/badge/デモ動画-Bilibili-00A1D6)](https://www.bilibili.com/video/BV15Cpr6eEMx)

<a href="https://www.bilibili.com/video/BV15Cpr6eEMx"><img src="docs/img/demo-video.jpg" alt="デモ動画:同じ RTX 4060 Laptop と同じ ESP32 ボードで、毎文ごとに途切れていた状態から長い回答でも音切れしない状態へ(Bilibili、3 分 28 秒)" width="640"></a>
| 最初の一声 | 音切れ合計 | VRAM |
|:-:|:-:|:-:|
| 6.2&nbsp;→&nbsp;**2.1&nbsp;s** | 4.22&nbsp;→&nbsp;**0.17&nbsp;s** | **7.3&nbsp;GB**&nbsp;/&nbsp;8&nbsp;GB |

<a href="https://www.bilibili.com/video/BV15Cpr6eEMx"><img src="docs/img/before-after.ja.webp" alt="実機の録画、同じノート PC・同じ質問。改善前は最初の一声 5.95 s、その後ほぼ毎文で途切れる。改善後は 1.92 s、回答全体で途切れは 1 回(0.1 s)だけ" width="760"></a>

<sub>実機の録画(2 倍速)、同じノート PC・同じ質問。赤いバーは音が途切れたところです。画像をクリックすると完全版の動画(Bilibili、3 分 28 秒、中国語)。質問はその場でリアルタイムに話したもので、動画では質問の声だけ後から合成音声に差し替えています。ボードの回答はその場の音声そのままです。</sub>

<sub>実機での録画です。質問はその場でリアルタイムに話したもので、動画では質問の声だけ後から合成音声に差し替えています。ボードの回答はその場の音声そのままです。</sub>
**ボードなしで試す:** Windows + NVIDIA 8 GB GPU + 3~10 秒の録音があれば、[クイックスタート](#クイックスタート)の手順 1~5 で動きます。最後の手順で音切れの合計が表示されます。

## できること

Expand Down Expand Up @@ -118,7 +126,7 @@ python tools\turn_taking_sim.py --mode turns --question "说说你最喜欢的

ここまでで使い方 A の状態です。自作プログラムから `http://127.0.0.1:18766/stream` を呼べば音声を合成できます。API は [tts-service/README.md](tts-service/README.md) を参照してください。

**6. ボードをつなぐ(使い方 B)。**
### ボードをつなぐ(使い方 B)

```powershell
pip install -r standalone\requirements.txt
Expand Down
14 changes: 11 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,22 @@

*A local LLM and a zero-shot voice-cloning TTS take turns on one 8 GB GPU. Use them as local HTTP services, or as an offline voice server for ESP32 boards running xiaozhi firmware.*

[效果](#效果) · [原理](#原理) · [需要什么](#需要什么) · [快速开始](#快速开始) · [接板子](#接板子用法-b) · [已知限制](#已知限制) · [上游修复](#提给上游的修复) · [文档](#文档) · [32 条踩坑](docs/pitfalls.md)

[![CI](https://github.com/korexil/gpu-duet/actions/workflows/ci.yml/badge.svg)](https://github.com/korexil/gpu-duet/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/korexil/gpu-duet)](https://github.com/korexil/gpu-duet/releases/latest)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
[![Bilibili](https://img.shields.io/badge/演示视频-Bilibili-00A1D6)](https://www.bilibili.com/video/BV15Cpr6eEMx)

<a href="https://www.bilibili.com/video/BV15Cpr6eEMx"><img src="docs/img/demo-video.jpg" alt="演示视频:同一台 RTX 4060 Laptop 和同一块 ESP32 板子上,从每句都卡到长回答不断音(B 站,3 分 28 秒)" width="640"></a>
| 首声 | 断音合计 | 显存 |
|:-:|:-:|:-:|
| 6.2&nbsp;→&nbsp;**2.1&nbsp;s** | 4.22&nbsp;→&nbsp;**0.17&nbsp;s** | **7.3&nbsp;GB**&nbsp;/&nbsp;8&nbsp;GB |

<a href="https://www.bilibili.com/video/BV15Cpr6eEMx"><img src="docs/img/before-after.webp" alt="实机录像:同一台笔记本、同一个问题。之前首声 5.95 s、红色断音竖条越来越多;之后首声 1.92 s,一路是绿色波形" width="760"></a>

<sub>实机录像,2 倍速,红色竖条是声音断掉的地方。点图看完整视频(B 站 3 分 28 秒):提问是现场实时问的,视频里的提问声后来换成了合成声;板子的回答是现场原声。</sub>

<sub>实机录像。提问是现场实时问的,视频里的提问声后来换成了合成声;板子的回答是现场原声。</sub>
**不接板子先试一下:** Windows + NVIDIA 8 GB 显卡 + 一段 3~10 秒的录音,按[快速开始](#快速开始)第 1~5 步走,最后一步直接打印断音合计。

## 能做什么

Expand Down Expand Up @@ -117,7 +125,7 @@ python tools\turn_taking_sim.py --mode turns --question "说说你最喜欢的

到这一步就是用法 A:自己的程序调 `http://127.0.0.1:18766/stream` 合成语音,接口见 [tts-service/README.md](tts-service/README.md)。

**6. 接板子(用法 B)。**
### 接板子(用法 B)

```powershell
pip install -r standalone\requirements.txt
Expand Down
Binary file added docs/img/before-after.en.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/img/before-after.ja.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/img/before-after.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/img/demo-video.jpg
Binary file not shown.
Binary file added docs/img/social-preview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading