Skip to content

feat(vtuber): add OpenAI-compatible adapter#1234

Open
canyugs wants to merge 21 commits into
openabdev:mainfrom
canyugs:feat/vtuber-adapter
Open

feat(vtuber): add OpenAI-compatible adapter#1234
canyugs wants to merge 21 commits into
openabdev:mainfrom
canyugs:feat/vtuber-adapter

Conversation

@canyugs

@canyugs canyugs commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

What problem does this solve?

Desktop character apps such as AniCompanion, Open-LLM-VTuber, and ChatVRM already speak OpenAI chat completions, but they usually connect to a raw LLM. This PR lets those skins point at OpenAB instead, so the same UI gets ACP-backed tool use, code editing, memory, MCP, and existing OpenAB steering.

Closes #1233

Discord Discussion URL: https://discord.com/channels/1491295327620169908/1520790210320011274

Architecture

The VTuber adapter now runs inside the unified OpenAB binary:

Skin (AniCompanion / Open-LLM-VTuber / ChatVRM)
  |
  |-- POST /v1/chat/completions  (stream:true, Bearer key)
  |     choices[].delta.content, including inline [emotion] tags
  |
OpenAB unified binary
  |-- crates/openab-gateway/src/adapters/vtuber.rs
  |-- src/unified_adapter.rs
  |-- src/acp -> coding agent (codex / claude / kiro)

No separate gateway process or adapter config block is required for VTuber in unified mode. Set VTUBER_ENABLED=true on the OpenAB process, and the unified HTTP listener exposes the OpenAI-compatible endpoint.

Proposed Solution

OpenAI-Compatible SSE

  • POST /v1/chat/completions streams OpenAI-compatible chat.completion.chunk events.
  • Inline [emotion] tags pass through for skins that already parse and strip them before TTS.
  • A stable vtb_persistent channel reuses one warm ACP session for VTuber traffic, avoiding per-turn cold starts.
  • Requests are serialized with a stream-owned lock; concurrent requests return 429 with Retry-After: 3.
  • OpenAI-compatible skins may still send full messages[]; the adapter forwards system/developer instructions plus the latest user turn so assistant history is not duplicated inside the persistent ACP session.

Why this approach?

  • Zero client changes for skins that already support OpenAI-compatible chat completions.
  • Unified binary deployment matches the current OpenAB platform-adapter model.
  • Persistent session reuse preserves the latency win from Add Gemini CLI support with separate Docker image #15 while keeping stream lifetime locking and prompt forwarding explicit.

Validation

  • cargo test -p openab-gateway vtuber — 7 VTuber tests passed
  • cargo test -p openab has_unified_platform_env_checks --features unified
  • cargo check -p openab --features unified
  • cargo clippy -- -D warnings
  • cargo test

Notes

  • docs/vtuber.md documents unified-mode setup only.
  • docs/config-reference.md lists the VTuber unified environment variables.
  • Tier-2 WebSocket side-channel work was deferred out of this PR and should land separately when there is a concrete client workflow.

Expose POST /v1/chat/completions (SSE) backed by the OAB agent, so any
OpenAI-compatible character skin (AniCompanion, Open-LLM-VTuber, …) gets a
real agent with zero client changes. messages[] is flattened into the agent
prompt; the agent's streamed reply is re-emitted as OpenAI chat.completion.chunk
deltas via a per-request channel.id registry drained by the /ws recv loop.
Inline [emotion] tags pass through untouched. Tier 1 of RFC openabdev#1233.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m
@smallgun01

Copy link
Copy Markdown
Contributor

Additional: Frontend WebSocket Client Demo

Added examples/vtuber-demo/index.html — a minimal WebSocket client reference implementation for VTuber skins.

What it does:

Connects to OAB Gateway via raw WebSocket (ws://host:8080/ws?token=)
Sends openab.gateway.event.v1 (text messages)
Receives openab.gateway.reply.v1 (streaming reply with cursor animation)
Dark theme UI, settings persisted in localStorage

Usage:

Open index.html in browser, configure WS URL and token, connect and chat.

Note:

Currently the gateway's handle_oab_connection does not forward GatewayEvents from WS clients to OAB core — backend update needed for full functionality.

Code:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>OAB VTuber Client</title>
  <style>
    * { box-sizing: border-box; margin: 0; padding: 0; }
    body { font-family: monospace; background: #1a1a2e; color: #e0e0e0; display: flex; flex-direction: column; height: 100vh; }
    #header { padding: 12px 16px; background: #16213e; border-bottom: 1px solid #0f3460; display: flex; align-items: center; gap: 12px; }
    #header h1 { font-size: 16px; color: #e94560; }
    #status { font-size: 12px; padding: 2px 8px; border-radius: 4px; }
    .disconnected { background: #555; }
    .connected { background: #2d6a4f; }
    .error { background: #9b2226; }
    #config { padding: 8px 16px; background: #16213e; border-bottom: 1px solid #0f3460; display: flex; gap: 8px; flex-wrap: wrap; align-items: center; font-size: 12px; }
    #config input { background: #0f3460; border: 1px solid #533483; color: #e0e0e0; padding: 4px 8px; border-radius: 4px; font-family: monospace; font-size: 12px; }
    #config button { background: #533483; border: none; color: #e0e0e0; padding: 4px 12px; border-radius: 4px; cursor: pointer; font-size: 12px; }
    #config button:hover { background: #e94560; }
    #chat { flex: 1; overflow-y: auto; padding: 16px; display: flex; flex-direction: column; gap: 8px; }
    .msg { padding: 8px 12px; border-radius: 6px; max-width: 80%; white-space: pre-wrap; word-break: break-word; font-size: 14px; line-height: 1.5; }
    .msg.user { background: #533483; align-self: flex-end; }
    .msg.assistant { background: #0f3460; align-self: flex-start; }
    .msg.system { background: #2d2d2d; align-self: center; font-size: 11px; color: #888; }
    .msg.error { background: #9b2226; align-self: center; font-size: 12px; }
    #input-area { padding: 12px 16px; background: #16213e; border-top: 1px solid #0f3460; display: flex; gap: 8px; }
    #msg-input { flex: 1; background: #0f3460; border: 1px solid #533483; color: #e0e0e0; padding: 8px 12px; border-radius: 6px; font-family: monospace; font-size: 14px; resize: none; }
    #send-btn { background: #e94560; border: none; color: #fff; padding: 8px 20px; border-radius: 6px; cursor: pointer; font-size: 14px; font-weight: bold; }
    #send-btn:hover { background: #c81d4e; }
    #send-btn:disabled { background: #555; cursor: not-allowed; }
    .cursor { display: inline-block; width: 8px; height: 14px; background: #e94560; animation: blink 0.8s infinite; vertical-align: text-bottom; }
    @keyframes blink { 0%, 100% { opacity: 1; } 50% { opacity: 0; } }
  </style>
</head>
<body>
  <div id="header">
    <h1>OAB VTuber Client</h1>
    <span id="status" class="disconnected">Disconnected</span>
  </div>
  <div id="config">
    <label>WS: <input type="text" id="ws-url" size="40"></label>
    <label>Token: <input type="password" id="ws-token" placeholder="API key" size="20"></label>
    <button id="connect-btn" onclick="toggleConnection()">Connect</button>
  </div>
  <div id="chat"></div>
  <div id="input-area">
    <textarea id="msg-input" rows="2" placeholder="Type a message..." onkeydown="handleKey(event)"></textarea>
    <button id="send-btn" onclick="sendMessage()" disabled>Send</button>
  </div>

  <script>
    const STORAGE_KEY = 'oab-vtuber-config';
    let ws = null;
    let currentAssistantEl = null;
    let cursorEl = null;

    const chatEl = document.getElementById('chat');
    const msgInput = document.getElementById('msg-input');
    const sendBtn = document.getElementById('send-btn');
    const connectBtn = document.getElementById('connect-btn');
    const statusEl = document.getElementById('status');
    const wsUrlInput = document.getElementById('ws-url');
    const wsTokenInput = document.getElementById('ws-token');

    function loadConfig() {
      try {
        const saved = JSON.parse(localStorage.getItem(STORAGE_KEY));
        if (saved) {
          wsUrlInput.value = saved.wsUrl || 'ws://localhost:8080/ws';
          wsTokenInput.value = saved.token || '';
        }
      } catch {
        wsUrlInput.value = 'ws://localhost:8080/ws';
      }
    }

    function saveConfig() {
      localStorage.setItem(STORAGE_KEY, JSON.stringify({
        wsUrl: wsUrlInput.value.trim(),
        token: wsTokenInput.value.trim()
      }));
    }

    loadConfig();

    function addMessage(text, role) {
      const el = document.createElement('div');
      el.className = `msg ${role}`;
      el.textContent = text;
      chatEl.appendChild(el);
      chatEl.scrollTop = chatEl.scrollHeight;
      return el;
    }

    function showCursor(parentEl) {
      removeCursor();
      cursorEl = document.createElement('span');
      cursorEl.className = 'cursor';
      parentEl.appendChild(cursorEl);
    }

    function removeCursor() {
      if (cursorEl) {
        cursorEl.remove();
        cursorEl = null;
      }
    }

    function setStatus(state, text) {
      statusEl.className = state;
      statusEl.textContent = text;
    }

    function toggleConnection() {
      if (ws && ws.readyState === WebSocket.OPEN) {
        ws.close();
      } else {
        connectWS();
      }
    }

    function connectWS() {
      const baseUrl = wsUrlInput.value.trim();
      const token = wsTokenInput.value.trim();
      if (!baseUrl) return addMessage('Please enter WebSocket URL', 'error');

      saveConfig();
      const url = token ? `${baseUrl}?token=${encodeURIComponent(token)}` : baseUrl;
      addMessage(`Connecting to ${url}...`, 'system');

      try {
        ws = new WebSocket(url);
      } catch (e) {
        addMessage(`Connection error: ${e.message}`, 'error');
        return;
      }

      ws.onopen = () => {
        setStatus('connected', 'Connected');
        connectBtn.textContent = 'Disconnect';
        sendBtn.disabled = false;
        addMessage('Connected to OAB Gateway', 'system');
      };

      ws.onmessage = (event) => {
        try {
          const reply = JSON.parse(event.data);

          if (reply.schema === 'openab.gateway.reply.v1') {
            const content = reply.content;
            if (content && content.type === 'text' && content.text) {
              if (!currentAssistantEl) {
                currentAssistantEl = addMessage('', 'assistant');
              }
              currentAssistantEl.textContent += content.text;
              showCursor(currentAssistantEl);
              chatEl.scrollTop = chatEl.scrollHeight;
            }

            if (reply.done) {
              removeCursor();
              currentAssistantEl = null;
            }
          } else if (reply.type === 'done' || reply.done === true) {
            removeCursor();
            currentAssistantEl = null;
          } else {
            addMessage(`[Unknown reply] ${event.data}`, 'system');
          }
        } catch {
          if (!currentAssistantEl) {
            currentAssistantEl = addMessage('', 'assistant');
          }
          currentAssistantEl.textContent += event.data;
          showCursor(currentAssistantEl);
          chatEl.scrollTop = chatEl.scrollHeight;
        }
      };

      ws.onclose = () => {
        setStatus('disconnected', 'Disconnected');
        connectBtn.textContent = 'Connect';
        sendBtn.disabled = true;
        removeCursor();
        currentAssistantEl = null;
        addMessage('Disconnected', 'system');
      };

      ws.onerror = () => {
        setStatus('error', 'Error');
        addMessage('WebSocket error', 'error');
      };
    }

    function sendMessage() {
      const text = msgInput.value.trim();
      if (!text || !ws || ws.readyState !== WebSocket.OPEN) return;

      addMessage(text, 'user');
      currentAssistantEl = null;
      removeCursor();

      const payload = {
        schema: 'openab.gateway.event.v1',
        platform: 'vtuber',
        content: {
          type: 'text',
          text: text
        }
      };

      ws.send(JSON.stringify(payload));
      msgInput.value = '';
    }

    function handleKey(e) {
      if (e.key === 'Enter' && !e.shiftKey) {
        e.preventDefault();
        sendMessage();
      }
    }
  </script>
</body>
</html>

@canyugs
canyugs marked this pull request as ready for review June 28, 2026 17:33
@canyugs
canyugs requested a review from thepagent as a code owner June 28, 2026 17:33
Copilot AI review requested due to automatic review settings June 28, 2026 17:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@canyugs

canyugs commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

Tier-1 complete ✅

All validation items checked:

  • Unit tests (config, flatten_messages, delta_suffix incl. multibyte) — 257 passed
  • Full e2e against real gateway binary (fake agent)
  • Real ACP agent e2e (kiro-cli driving live LLM)
  • Cloud deployment e2e (Zeabur, Tencent Tokyo) — curl -sN with Bearer auth → streamed OpenAI deltas → [DONE]
  • AniCompanion e2e (macOS VRM app) — zero code changes, pointed at cloud gateway, sent messages and received replies from claude-agent-acp

CI: all checks and smoke tests passing.

Next: Tier-2

Tier-2 RFC opened as #1235 — WebSocket side-channel (/v1/vtuber/ws) for agent-state push, tool visibility, emotion, and ambient notifications. Design is informed by prior art from Open-LLM-VTuber, clawd-on-desk, and VTube Studio API.

@chaodu-agent

This comment has been minimized.

Tier-2 adds GET /v1/vtuber/ws — a persistent WebSocket that pushes
agent_state, emotion, and notification events derived from GatewayReply
commands. VTuber skins connect once and receive real-time state updates
(thinking/working/idle, tool usage, emotion tags) without polling.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m
@canyugs

canyugs commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

Tier-2 WebSocket event stream pushed (acd3e38). Pending: e2e testing with a live VTuber skin.

@chaodu-agent

This comment has been minimized.

Replace Vec<WsClient> with HashMap<u64, WsClient> and an AtomicU64
counter. The old Vec+index scheme broke when broadcast() called
swap_remove on dead clients — surviving clients' stored indices became
stale, routing subscribe/pong to the wrong connection.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m
@chaodu-agent

This comment has been minimized.

F2: Cap in-flight /v1/chat/completions at 32 by checking
vtuber_pending size before accepting — returns 429 when full.

F3: Emit SSE comment `: waiting for agent` after 10s of silence,
giving clients an early signal before the 180s hard timeout.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m
@canyugs

canyugs commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Review findings addressed:

  • F1 (client index bug): Fixed in c154dc1Vec<WsClient> + index → HashMap<u64, WsClient> + AtomicU64. No more stale index after swap_remove.
  • F2 (rate limit): Fixed in 50c252e — cap at 32 in-flight requests via vtuber_pending.len() check, returns 429.
  • F3 (idle warning): Fixed in 50c252e — SSE comment : waiting for agent emitted after 10s of silence, before the 180s hard timeout.
  • F1 (duplicate AppState): Deferred — this pattern is shared across all adapters; refactoring serve() to delegate to AppState::from_env() should be a separate PR.
  • F3 (emotion intensity): Deferred — hardcoded 1.0 is sufficient until a skin needs variable intensity ([tag:0.8] format).

@chaodu-agent

This comment has been minimized.

@canyugs

canyugs commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

All findings resolved, clippy CI fixed in c1b9495. Ready for merge — pending e2e testing with a live VTuber skin.

@chaodu-agent

This comment has been minimized.

@github-actions

Copy link
Copy Markdown

Caution

This PR has been waiting on the author for more than 2 days (labeled pending-contributor since 2026-07-12).
It will be automatically closed in 24 hours if there is no update.

@canyugs — You must add a new comment on this PR to remove the closing-soon label and keep it open. Pushing commits alone is not sufficient. Feel free to reopen a new PR later if it gets closed and you want to pick it back up.

Bring the branch up to openab-0.9.0. Conflict resolutions:

- Cargo.toml: take main's default feature set (adds `filestore`); the
  `vtuber` feature and its slot in `unified` are unchanged.
- main.rs `has_unified_platform`: keep main's config-first Google Chat
  resolver and re-attach the env-signaled vtuber arm after it.
- main.rs unified routing: adopt main's `gw_state.*_webhook_path` fields
  and the shared trust-registry gating (openabdev#1391), re-attaching the vtuber
  `/v1/chat/completions` route after the Google Chat route.
- main.rs tests: `has_unified_platform_env` was renamed upstream to
  `has_unified_platform(&cfg)`; port the two vtuber cases over.
- vtuber.rs test AppState: add the three new webhook path fields.

Verified: cargo test --workspace --features vtuber (all vtuber tests
pass). `secrets::tests::resolve_exec_nonzero_exit` fails identically on
clean upstream main — pre-existing, unrelated to this merge.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@chaodu-agent

Copy link
Copy Markdown
Collaborator

Important

CHANGES REQUESTED ⚠️ — The adapter has a disconnect-time response-routing bug and permits an enabled endpoint to run without authentication.

What This PR Does

This PR adds an OpenAI-compatible /v1/chat/completions SSE adapter so VTuber clients can use OpenAB's ACP-backed agent instead of connecting directly to an LLM. It wires the adapter into the unified binary, adds persistent-session prompt handling, configuration documentation, and focused tests.

How It Works

The endpoint accepts streaming chat requests, maps system/developer instructions plus the latest user-like message into a gateway event, and routes gateway replies back as OpenAI-compatible SSE chunks. A shared vtb_persistent channel and a stream-owned mutex serialize turns and reuse the warm ACP session.

Findings

# Severity Finding Location
1 🔴 Critical A dropped SSE stream can leave a stale fixed-channel sender, allowing a late reply from the old turn to be delivered to the next request. crates/openab-gateway/src/adapters/vtuber.rs:275
2 🟡 Important VTUBER_AUTH_KEY is optional, so enabling the endpoint without it exposes an unauthenticated agent/tool endpoint. crates/openab-gateway/src/adapters/vtuber.rs:220
3 🟢 Praise The adapter has focused coverage for streaming-only requests, concurrency rejection, auth comparison, prompt flattening, and tail-idle cleanup. crates/openab-gateway/src/adapters/vtuber.rs
Finding Details

🔴 F1: Clean up request routing when the client disconnects

The request stores its sender under the constant vtb_persistent key at line 275, while registry cleanup only runs in the normal stream state machine at line 430. If the HTTP client disconnects, Axum drops the SSE stream before that cleanup phase; the request guard is released, so the next request can insert the same key and replace the old sender. Gateway replies from the old ACP turn still use that channel ID, and handle_reply then forwards a late old reply into the new request's stream.

Requested change: Make cancellation/disconnect cleanup remove the request's own registry entry, using compare-and-remove/ownership protection so it cannot remove a replacement; also ensure late replies from an abandoned turn cannot match a subsequent request (for example, use a per-request correlation key while retaining the persistent ACP session separately).

🟡 F2: Require authentication for an enabled endpoint

VtuberConfig::from_env logs a warning and continues when VTUBER_AUTH_KEY is absent. Because the route is mounted whenever VTUBER_ENABLED=true, a deployment that forgets the key exposes an endpoint capable of invoking the agent and its tools without authentication. The setup docs present the key as recommended even though the endpoint may be internet-facing.

Requested change: Fail startup or refuse requests whenever the adapter is enabled without a non-empty VTUBER_AUTH_KEY (or require an explicit, clearly named local-only unauthenticated opt-in). Update the configuration documentation to match the enforced security contract.

🟢 F3: Focused protocol and lifecycle tests

The added tests cover the streaming-only contract, busy-request behavior, exact auth-key comparison, prompt shaping, and normal tail-idle registry cleanup. These provide a useful base for adding the missing disconnect/cancellation regression case.

Baseline Check
  • PR opened: 2026-06-28
  • Main already has: generic OpenAI client usage and existing gateway adapters, but no VTuber adapter or /v1/chat/completions gateway route.
  • Net-new value: a unified OpenAB endpoint for OpenAI-compatible VTuber clients with ACP session reuse and SSE streaming.
What's Good (🟢)
  • The implementation is feature-gated and integrated into both standalone gateway and unified startup paths.
  • The stream protocol emits role/content/final chunks and [DONE] in a client-compatible shape.
  • Constant-time API-key comparison is used when a key is configured.
  • Existing CI unified smoke tests passed; git diff --check is clean. Local Rust tests could not be rerun here because cargo is unavailable in the review environment.

Addressing External Reviewer Feedback

No external reviewer feedback was present when this review round was prepared.

5️⃣ Three Reasons We Might Not Need This PR

  1. The client ecosystem may already support a generic OpenAI proxy — if deployment can use an existing gateway, maintaining another protocol adapter may not justify its long-term surface area.
  2. A single persistent ACP session limits concurrency — VTuber traffic is intentionally serialized, so higher-throughput clients would need a different session-pooling design.
  3. The feature expands the public attack surface — without a strict authentication and cancellation contract, the unified binary gains a route that can invoke powerful agent capabilities; those operational requirements may outweigh zero-client-change integration.

@chaodu-agent chaodu-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

CHANGES REQUESTED ⚠️ — The adapter has a disconnect-time response-routing bug and permits an enabled endpoint to run without authentication.

Consolidated review: #1234 (comment)

.insert(axum::http::header::RETRY_AFTER, "3".parse().unwrap());
return resp;
}
pending.insert(channel_id.clone(), tx);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

� F1 � Clean up request routing when the client disconnects

If the SSE client disconnects, the stream is dropped before the cleanup at line 430. Since this request uses the fixed vtb_persistent key, the next request can replace this sender and receive late replies from the abandoned turn.

Requested change: Add cancellation-safe, ownership-aware cleanup and a per-request correlation mechanism so an abandoned turn cannot be routed into a later request.

.into_response();
};

if let Some(expected) = cfg.auth_key.as_ref() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

� F2 � Require authentication when the adapter is enabled

This branch only validates a key when one exists. VTUBER_ENABLED=true with no VTUBER_AUTH_KEY therefore mounts an unauthenticated endpoint that can invoke the agent and its tools.

Requested change: Fail startup/refuse requests without a non-empty key, or require an explicit local-only unauthenticated opt-in, and document that contract.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

closing-soon PR missing Discord Discussion URL — will auto-close in 24 hours. pending-contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC: VTuber adapter — OpenAI-compatible skin frontend for OAB

5 participants