Skip to content

feat: send audio as native WeChat voice bubbles - #282

Open
kenhuangus wants to merge 13 commits into
Tencent:mainfrom
kenhuangus:feat/outbound-native-voice
Open

kenhuangus wants to merge 13 commits into
Tencent:mainfrom
kenhuangus:feat/outbound-native-voice

Conversation

@kenhuangus

@kenhuangus kenhuangus commented Sep 2, 2026

Copy link
Copy Markdown

Summary

  • Send audio/* as a native WeChat voice bubble: ffmpeg + silk-wasm to Tencent SILK_V3 (\x02#!SILK_V3), upload media_type=VOICE (4), send MessageItemType.VOICE (3).
  • Outbound defaults match a real inbound personal-WeChat 语音条: encode_type=4, sample_rate=16000, no size, voice aes_key as base64(hex string) per parseAesKey, encrypt_type omitted.
  • Native encode/upload/send failures fall back to FILE. asVoice / forceDocument honored.

Test plan

  • npm test
  • Live iLink: plugin-encoded MP3 and echo of Ken's inbound SILK both return sendVoiceMessageWeixin success
  • Combined TEXT+VOICE in one item_list is rejected (invalid arguments); synthesized media.full_url CDN GET returns 400 (not shipped)
  • Ken confirms a green 语音条 on personal WeChat

Related: #91, #215, #192. If the client still drops the bubble, that is the known iLink limitation; this PR still ships the native VOICE path plus FILE fallback.

ken huang added 3 commits September 1, 2026 23:08
Route audio through SILK_V3 + VOICE upload/item instead of a file attachment, with FILE fallback if encode or send fails. OpenClaw asVoice/forceDocument flags are honored.
WeChat clients expect \x02#!SILK_V3. silk-wasm already emits it; this also prefixes raw pass-through SILK.
Needed so we can echo a real client voice payload back as a native bubble.
@kenhuangus

Copy link
Copy Markdown
Author

Live test notes (ken-mac, personal WeChat, plugin 2.4.8 + this branch)

What already works

  • audio/* native path: ffmpeg 24 kHz WAV -> silk-wasm SILK_V3 (Tencent 0x02 prefix, 24 KB / 6120 ms) -> getuploadurl media_type=4 -> sendVoiceMessageWeixin encode_type=6 returned success.
  • FILE attachments still deliver (control).
  • getConfig + sendTyping still succeed on the same context_token after sendMessage starts failing.

What is not verified yet

Send window

  • After several outbound messages on one inbound turn, sendMessage returns ret=-2 errmsg=prepare failed until the user sends a new WeChat message. That blocked follow-up payload variants (encode_type=4 / 16 kHz inbound clone, echo of a real inbound SILK).

This branch also

  • Always Tencent-prefixes outbound SILK (ensureTencentSilk).
  • Downloads inbound VOICE even when voice_item.text transcription is present, and logs encode_type / sample_rate / magic bytes so we can echo a real client payload.

Will update this thread when a real-device 语音条 is confirmed or ruled out.

ken huang added 2 commits September 1, 2026 23:36
Real inbound voice bubbles use 16 kHz SILK bytes with encode_type=4. Default prepareOutboundVoice and sendVoiceMessageWeixin to that shape so TTS/MP3 outbound matches what the personal WeChat client already renders.
@kenhuangus

Copy link
Copy Markdown
Author

Follow-up: default outbound native voice now matches a real inbound voice bubble from Ken's personal WeChat:

  • sample_rate=16000
  • encode_type=4 (client sends SILK bytes with type 4; proto comments saying 4=speex / 6=silk are not what the client emits)
  • Tencent SILK prefix 0x02 #!SILK_V3

Live echo of Ken's own inbound SILK (6576 bytes, playtime 3637 ms) was sent as native VOICE at 23:40 ET, clientId openclaw-weixin:1788320467959-024de542. Waiting on phone confirmation that a green voice bar rendered above the [ECHO] marker.

Keep outbound defaults at 16 kHz / encode_type=4. Expand the optional inbound dump so the next real 语音条 can be compared on encrypt_type and CDN fields without writing secrets.
@kenhuangus

Copy link
Copy Markdown
Author

Live test update (Ken personal WeChat, iLink bot d2dabd8a1c40-im-bot):

Send Codec encode_type sample_rate API Client (pending Ken)
23:06 native MP3->SILK Tencent SILK 24 kB / 6120 ms 6 24000 sendVoiceMessageWeixin success unconfirmed
23:40 ECHO of Ken inbound 语音条 same 6576-byte SILK Ken just sent 4 16000 VARIANT_OK ECHO look above [ECHO]
23:47 plugin prepareOutboundVoice MP3 SILK 23338 B / 6120 ms 4 16000 success 1788320868133-3fd99ff1 look above [VOICE-16K]

The ECHO used Ken's exact inbound bytes (\x02#!SILK_V3, encode_type=4, 16 kHz). If that still has no green voice bar, this is the same iLink bot limitation as #91 / #215, not a missing SILK prefix or sample-rate mismatch.

Outbound defaults on this branch: 16 kHz + encode_type=4, FILE fallback if encode/upload/send fails. No gateway restart while a WeChat turn is in flight; live dist copied for the next process start.

Ken's inbound 语音条 had size unset. Combined TEXT+VOICE in one sendMessage is rejected (invalid arguments). Voice-only without size is accepted.
@kenhuangus

Copy link
Copy Markdown
Author

Follow-up live send: voice_item.size omitted (Ken inbound had size unset).

  • Combined TEXT+VOICE in one item_list -> ret=-2 invalid arguments
  • Voice-only, encode_type=4, 16 kHz, inbound SILK bytes, no size -> API success openclaw-weixin:1788321177032-d02b883d
  • Marker: [VOICE-NOSIZE]

Please check whether a green voice bar is immediately above that marker.

uploaded.aeskey is hex; Buffer.from(hex) was treating hex digits as UTF-8. Inbound images already convert hex with Buffer.from(aeskey, "hex").toString("base64"). Voice bubbles likely require that 24-char form.
@kenhuangus

Copy link
Copy Markdown
Author

Follow-up: encode voice/image/video/file media.aes_key as base64(raw 16-byte key), not base64(utf8(hex)).

Live send of inbound SILK with corrected aes_key: look above [VOICE-AES].

…t_type.

pic-decrypt parseAesKey: images are base64(raw 16 bytes); file/voice/video inbound is base64 of the 32-char hex string. Sending the image form on VOICE was likely wrong.
@kenhuangus

Copy link
Copy Markdown
Author

Follow-up: voice media.aes_key now matches inbound file/voice format (base64 of hex string, 44 chars). encrypt_type omitted on VOICE.

Live marker: [VOICE-HEX].

@kenhuangus

Copy link
Copy Markdown
Author

Did not ship media.full_url: constructing download URL via buildCdnDownloadUrl on an outbound x-encrypted-param returns CDN 400. Reverted. Plugin remains at encode_type=4 / 16 kHz / no size / voice aes_key=base64(hex). Still waiting on phone confirmation of markers [VOICE-HEX] [VOICE-AES] [VOICE-NOSIZE] [VOICE-16K] [ECHO].

ChannelMessageSendMediaContext uses audioAsVoice, not asVoice. The plugin only read asVoice, so TTS sends with asVoice=true (including .pcm) were uploaded as FILE. Map audioAsVoice and recognize pcm/aiff so the native VOICE path actually runs.
@kenhuangus

Copy link
Copy Markdown
Author

Deep fix: OpenClaw sendMedia ctx field is audioAsVoice, not asVoice. Live agent TTS used asVoice=true with a .pcm and the plugin uploaded it as FILE (0-byte attachment). Plugin now reads audioAsVoice and treats pcm/aiff as audio.

Needed to distinguish upload/encoding bugs from iLink dropping BOT+VOICE. Dump is env-gated (WEIXIN_DUMP_INBOUND_VOICE=1).
@kenhuangus

Copy link
Copy Markdown
Author

Next approach is protocol-level, not another SILK tweak.

Independent iLink SDKs (lith0924 Java SDK issue 11, 2026-07-02) report Tencent changed the protocol: bot-sent VOICE is accepted (ret=0) but personal WeChat no longer renders a bubble. Hermes and issue 215 show the same.

Remaining discriminators on this PR:

  1. Replay inbound CDN refs (encrypt_query_param + aes_key the client already rendered) without re-upload. Dump of those refs is now env-gated in media-download + process-message.
  2. Send the same VOICE item with message_type=USER instead of BOT. If the filter is on BOT+VOICE, USER may still render.

Also: OpenClaw sendMedia ctx field is audioAsVoice (not asVoice). Live TTS was uploading 0-byte .pcm as FILE.

iLink accepts BOT+VOICE but personal WeChat does not render a bubble. Stop calling sendVoice from sendWeixinMediaFile so TTS cannot emit a silent VOICE item.
@kenhuangus

Copy link
Copy Markdown
Author

Outbound native VOICE is now disabled. sendWeixinMediaFile routes audio as FILE only.

Inbound voice still downloads. Outbound audio (TTS, say, asVoice) is converted to a text message because iLink does not render bot voice bubbles.
@seaurching

Copy link
Copy Markdown

Same result here, reproducing with the official Tencent PR #9 implementation (direct MP3 upload as VOICE(4), encode_type=7, playtime = size/160).

We patched the exact PR #9 code into the plugin dist and sent 6 live voice messages to a real personal-WeChat account via a scanned iLink bot session: SILK encode_type=6/24 kHz, 5/8 kHz, and MP3 7/as-is, with playtime as real ms and as size/160. Every attempt: getUploadUrl(media_type=4) → CDN PUT → sendMessage(type=3) returned success, yet the client rendered nothing (silent drop). Text ✅, FILE(4)+FILE(4) MP3 attachment ✅, and the VOICE CDN upload ✅ all work on the same account — only the VOICE item vanishes.

Also ruled out: playtime unit (both real-ms and size/160 accepted, neither rendered) and SILK validity (our silk-wasm output decodes fine with kn007 silk-v3-decoder).

Detailed matrix: #215 (comment)

This includes the official PR #9 path, not just community guesses — I'd also like confirmation from the team whether outbound VOICE is supported for normal iLink bot accounts, or should be documented as unsupported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants