feat(intelligence): hands-free continuous voice mode + real location_search wiring - #243
Merged
Wintercom merged 12 commits intoAug 14, 2026
Conversation
Backend Tier 1 of the natural-stop migration: the intelligence/realtime agent module (end_conversation tool, session-end delivery after the farewell audio finishes) plus the untouched gateway/websocket handlers and messages it flows through.
Backend Tier 2 of the natural-stop migration: reconciles the continuous- mode voice_mode handshake field against main's independently-added latitude/longitude/coordinate_system fields on the same session.hello payload.
Frontend port of the pure-addition files for continuous voice mode: the phone icon, push-to-talk bar, full-screen VoiceCallScreen, the app-foreground/background lifecycle port, plus the paused phase on ConversationTurn and the voice.session.end contract message that the rest of the migration wires up.
…ed transport Reimplements the hands-free continuous conversation orchestration (startTurn/endTurn, idle-timeout auto-hangup, manual pause/resume, background auto-mute, voice.session.end handling) against main's shared AuthenticatedWebSocketClient/AuthenticatedVoiceTransport connect() contract instead of the old branch's own per-service connect(url).
Threads voice_mode through AuthenticatedVoiceTransport/ AuthenticatedWebSocketClient's connect(), adds the optional togglePause() to AssistantApplicationPort and the hook, teaches AssistantVoiceOverlay the paused status/title and the full-screen call layer, and swaps HomeScreen/AppRoot over to the phone-icon entry point alongside push-to-talk.
Ending a continuous-mode stream while the vendor is still replying used to just cancel the pump task, leaving the vendor generating into the same socket a session that gets reused across turns -- those frames, and a stale open-response count, would otherwise bleed into the next turn.
…and cutoffs Several bugs found in manual testing of hands-free continuous mode: - startTurn()/endTurn() had no reentrancy guard, so a double-tap or an overlapping server-initiated hangup could corrupt connection/waiter state or repeat the teardown steps. - messages arriving mid-endTurn() could re-open state that was already settling back to idle. - native playback lifecycle calls (startStream/pushChunk/endStream/stop) were fired independently with no ordering guarantee, letting audio race ahead of the stream that was supposed to configure it first. - handleClose() nulled the connection before unsubscribing, leaking listeners on the shared transport when a real disconnect landed mid-teardown. - startTurn() failed to reset a stale mute flag inherited from a call that was auto-hung-up while paused. - capture.stop()/connection teardown failures in endTurn() could abort the hangup partway, leaving the service stuck mid-call.
…in miss Three defects found testing continuous mode on a real device, all against the vendor session reused across turns: - end_conversation asked the vendor for a follow-up reply after the model had already said goodbye, producing a second, often truncated one. send_tool_result() now takes respond=False for a tool that has nothing left to say. - A continuous-mode pump that exits early on a vendor error was never noticed: handle_audio() waited on the client's still-open microphone stream, which nothing was going to close. The mic forward and the pump are now raced with asyncio.wait(), and a failure in continuous mode now ends the call the same way end_conversation does, instead of leaving the call screen up around a microphone nobody is reading any more. - The vendor generates audio faster than it plays back, so response.done -- which closed the barge-in window -- fired well before the phone finished sounding a reply out. A barge-in landing in that gap did nothing. The window now stays open for the estimated remaining playback time, and the client is told to stop audio even if this turn's own bookkeeping has already settled.
…sed Agents Replaces the location_search placeholder with the real Tencent-backed module (PR 1024XEngineer#236) across both agents, plumbing a session's client location from the WebSocket handshake through to the tool, plus a round of fixes found testing it live on a device. Wiring (issue 1024XEngineer#235): - Composition root builds a shared httpx.AsyncClient + TencentMapsLocationPort + LocationSearchService, gated on voice_agent_mode=="1" and Tencent creds being configured; closed in lifespan. - Client latitude/longitude/coordinate_system -- captured at session.hello but previously dropped -- now flow as raw primitives through StreamContext to RealtimeAgent, which assembles a validated ClientLocation once per held session (same "fixed at session-open time" rule timezone already followed). - ToolBox registers real location_search (schema shared with the location module, query-only) and degrades to provider_unavailable -- never withheld from the schema -- whenever no location/service is available. - Composed Agent's tool registry gets the same real implementation for parity, though it still is not wired into /ws -- a separate, unrelated gap. - Fixed a latent circular import between intelligence.location and intelligence.conversation (pre-existing from PR 1024XEngineer#236, closed by this change's new edge) with TYPE_CHECKING plus a deferred import. Fixes found live-testing on a device: - schedule_create/update rejected latitude/longitude/reminder_offset_minutes whenever the model quoted a value it had copied verbatim from a location_search candidate -- reproduced retrying the exact same quoted value four times without self-correcting. Numeric strings are now coerced rather than rejected. - A location_search prepare() failure (a transient Tencent outage) used to stick for the rest of a held session, since it was computed once and cached forever; it is now retried on every call until it succeeds once, so a provider recovering mid-call is usable again immediately instead of only after the app reconnects. - Prompt gained default values for fields the model left unspecified (title, times, reminder), disambiguation rules for 0/1/2 location_search candidates spoken aloud (no candidate-card UI) rather than narrated as an unstructured reply, and a rule to copy a chosen candidate's coordinates verbatim into schedule_create/update.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![fennoai[bot]](https://avatars.githubusercontent.com/in/1924301?s=60&v=4){kind=small}
There was a problem hiding this comment.
Found one startup cleanup issue in continuous voice mode. Focused review covered code quality, performance, security, and documentation. Automated tests were not run because the mounted workspace has neither frontend dependencies nor the backend virtual environment installed.
…ce tests CI's format:check caught this; the file itself was untouched otherwise.
LUPENGHAN
changed the title
…s to start startTurn() sends voice.stream.start and awaits voice.stream.started before calling capture.start() -- by that point the server already considers this session's stream active. If native recording then rejects (device busy, permission revoked mid-flow), the old code let the rejection propagate straight out, leaving the WebSocket connection and the server-side stream alive. The UI stayed on the error path, and retrying reused a session whose next voice.stream.start the server would refuse as already active. Wraps capture.start() the same way the push-to-talk sibling service already does: send voice.stream.end for the stream that was opened, then unsubscribe and close the connection so a retry opens a genuinely fresh one, matching endTurn()'s own teardown. Found via automated PR review (fennoai bot on 1024XEngineer#243).
Wintercom
reviewed
Aug 14, 2026
Collaborator
Collaborator
|
【测试验收缺口 / 阻塞合并】Issue #242 的验收要求“新增的连续对话服务/UI 有对应 vitest 单测覆盖”，但本 PR 新增 |
Wintercom
requested changes
Aug 14, 2026
Wintercom
left a comment
Wintercom
left a comment
Collaborator
There was a problem hiding this comment.
按照 git_rules.txt 的合并标准，本轮存在以下阻塞项：
- #235 明确禁止在未实现可信候选绑定时让 Agent 直接写裸经纬度；当前 Prompt 却要求模型直接复制坐标，下游也不验证 provider_id/coordinate_system，违反关联 Issue 的安全与一致性约束。
- PR 自述合并了“两块本该分开的东西”，实际修改 56 个文件、+4177/-330 行，混合 #242 连续对话与 #235 地图 Agent 接入，不符合范围可控、小而频繁的要求；请拆分。
- #242 要求连续对话服务/UI 有测试覆盖，但新增通话 UI 与 AppState Provider 没有对应组件/生命周期测试。
- PR 未设置 Milestone，#235 也未纳入 Milestone，迭代追踪不完整。
具体内容已使用 gh 逐条评论。已有 reviewer 提出的 capture.start 清理问题已在 93f1b61 修复，本轮未重复评论。
VoiceCallScreen, PushToTalkBar, PhoneCallIcon, and RNAppStateProvider shipped in 1024XEngineer#242 without their own tests.
Contributor
Author
|
确实违反了拆分原则，时间紧不是合理理由。这次因为免提通话和 location_search 接入共享同一条 |
Contributor
Author
|
属实，已补上：
共 14 例，覆盖录音/暂停波形展示、收起/打断/结束/暂停继续按钮交互、后台切换事件转发、图标 props。已推到本分支（bc0690e）。 |
Contributor
Author
LUPENGHAN
added a commit
to LUPENGHAN/timeflow
that referenced
this pull request
Aug 14, 2026
Rebased onto upstream/main now that 1024XEngineer#243 is merged. Same content as the original merge commit (fcc9f15) plus the reminder_offset_minutes=15 fix (6a4bafd), replayed cleanly on top of 1024XEngineer#240/1024XEngineer#241's schedule UI redesign (no overlap, verified via git merge-tree before rebasing). 将 gac0812 fork（提醒引擎、原生闹钟、地理围栏、系统通知/震动、权限申请）合 并进来，解决 12 个真实冲突，并把两套组合根手工合成一套：保留 1024XEngineer#243 的认证/ WebSocket/日程视图骨架，reminderPorts 换成 fork 的真实实现 （NativeAlarmScheduler/NativeDeviceCapability/ExpoAudioPlayback/ NativeLocationMonitor），reminder 从 MockReminderApplication 换成真的 LocalReminderApplication。 冲突解决之外真实接入的部分： - app.json 转成 app.config.js：百度定位 Key 走 process.env，合并双方的 iOS 权限说明/Android 权限并集，去重重复键。 - 新建 SqliteLocalScheduleReader（attach/detach 延迟绑定，供 reminder 引擎 读真实本地日程）和 SqliteReminderStateStore（提醒运行时状态真正落 SQLite，替换掉纯内存的 MemoryReminderStateStore——原来进程一杀已触发的 提醒状态就丢，下次启动会整批重新弹一遍）。 - LocalScheduleWriter 语音写入成功后触发 SqliteLocalScheduleReader.refresh()。 - AppProviders 里接上权限申请 hook + reminder.rebuild()，按认证态门控。 - LocalReminderApplication 的时间型提醒投递加了原生闹钟归属判断：只要排上 了原生精确闹钟，弹窗和语音就完全交给原生 RingActivity/AlarmSoundService， JS 不再重复弹——避免真机上应用内弹窗和全屏响铃互相抢、谁都关不掉的问题。 - 新增 ExpoLocationMonitor：用 expo-location 的系统原生地理围栏 （GeofencingClient/CLCircularRegion）接围栏检测，不依赖百度定位 SDK 的 账号/Key 绑定；NativeLocationMonitor（百度）保留在仓库里，createAppServices 换个 import 就能切回去。 - exact_alarm 权限跳过确认弹窗直接跳系统设置页（没有系统授权框，多一次点 击没有意义）。 - 修了 @irvingouj/expo-audio-stream 的 build.gradle patch：AGP>=8 时 javac/kotlinc 目标版本对不上导致编译失败。 - reminder_offset_minutes 默认值从 200 分钟改成 15 分钟（backend/ instructions.py，200 分钟太长，真机测试等不到触发）。 删掉了 3 个测试老版本原生适配器 API 的过期单测（useReminderPermissionsOnLaunch/ nativeAlarmScheduler/nativeDeviceCapability），新增 SqliteLocalScheduleReader 和 LocalScheduleWriter 刷新触发的集成测试。 已知缺口：云端日程同步（SqliteScheduleSyncService）这次没接，生产代码里还 是零调用点，只接了语音写入这条刷新路径；geofence_radius_meters 本地表没有 这一列，硬编码 200 米。
LUPENGHAN
added a commit
to LUPENGHAN/timeflow
that referenced
this pull request
Aug 14, 2026
Rebased onto upstream/main now that 1024XEngineer#243 is merged. Same content as the original merge commit (fcc9f15) plus the reminder_offset_minutes=15 fix (6a4bafd), replayed cleanly on top of 1024XEngineer#240/1024XEngineer#241's schedule UI redesign (no overlap, verified via git merge-tree before rebasing). 将 gac0812 fork（提醒引擎、原生闹钟、地理围栏、系统通知/震动、权限申请）合 并进来，解决 12 个真实冲突，并把两套组合根手工合成一套：保留 1024XEngineer#243 的认证/ WebSocket/日程视图骨架，reminderPorts 换成 fork 的真实实现 （NativeAlarmScheduler/NativeDeviceCapability/ExpoAudioPlayback/ NativeLocationMonitor），reminder 从 MockReminderApplication 换成真的 LocalReminderApplication。 冲突解决之外真实接入的部分： - app.json 转成 app.config.js：百度定位 Key 走 process.env，合并双方的 iOS 权限说明/Android 权限并集，去重重复键。 - 新建 SqliteLocalScheduleReader（attach/detach 延迟绑定，供 reminder 引擎 读真实本地日程）和 SqliteReminderStateStore（提醒运行时状态真正落 SQLite，替换掉纯内存的 MemoryReminderStateStore——原来进程一杀已触发的 提醒状态就丢，下次启动会整批重新弹一遍）。 - LocalScheduleWriter 语音写入成功后触发 SqliteLocalScheduleReader.refresh()。 - AppProviders 里接上权限申请 hook + reminder.rebuild()，按认证态门控。 - LocalReminderApplication 的时间型提醒投递加了原生闹钟归属判断：只要排上 了原生精确闹钟，弹窗和语音就完全交给原生 RingActivity/AlarmSoundService， JS 不再重复弹——避免真机上应用内弹窗和全屏响铃互相抢、谁都关不掉的问题。 - 新增 ExpoLocationMonitor：用 expo-location 的系统原生地理围栏 （GeofencingClient/CLCircularRegion）接围栏检测，不依赖百度定位 SDK 的 账号/Key 绑定；NativeLocationMonitor（百度）保留在仓库里，createAppServices 换个 import 就能切回去。 - exact_alarm 权限跳过确认弹窗直接跳系统设置页（没有系统授权框，多一次点 击没有意义）。 - 修了 @irvingouj/expo-audio-stream 的 build.gradle patch：AGP>=8 时 javac/kotlinc 目标版本对不上导致编译失败。 - reminder_offset_minutes 默认值从 200 分钟改成 15 分钟（backend/ instructions.py，200 分钟太长，真机测试等不到触发）。 删掉了 3 个测试老版本原生适配器 API 的过期单测（useReminderPermissionsOnLaunch/ nativeAlarmScheduler/nativeDeviceCapability），新增 SqliteLocalScheduleReader 和 LocalScheduleWriter 刷新触发的集成测试。 已知缺口：云端日程同步（SqliteScheduleSyncService）这次没接，生产代码里还 是零调用点，只接了语音写入这条刷新路径；geofence_radius_meters 本地表没有 这一列，硬编码 200 米。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #242
Part of #235（Agent 接入部分做完了，可信坐标写入日程数据库那块——issue 里第 5 点——按 issue 自己允许的范围明确留白，没碰
business/calendar/contracts.py或数据库，那是 2 号的领域）概述
一个 PR 装了两块本该分开的东西，时间关系合并提交：
main当前的共享 transport 架构（AuthenticatedWebSocketClient/AuthenticatedVoiceTransport）重新实现，不是从旧的feature/realtime-continuous-mode分支直接搬过来的。location_search（feat(backend): 接入腾讯地图位置检索 Function 与 Agent #235）：把 PR feat(backend): 增加腾讯地图位置检索模块 #236 已经合并、但还没接进任何 Agent 的腾讯地图位置检索模块，真正接进 Realtime Agent（/ws上真正在跑的那条），Composed Agent 的工具层也做了同样的替换（但那个 Agent 本身还没接/ws，是另一个不相关的缺口，这次没顺带做）。改动
免提连续对话（#242 六条范围逐条落地）
AssistantContinuousConversationService.ts：跟按住说话（AssistantConversationService）并列的独立编排类，复用同一条AuthenticatedVoiceTransport连接end_conversation工具，_finish_reply()等告别语音完整播完（voice.tts.end）才发voice.session.end，不截断告别话；gateway/websocket/messages/voice.py新增VoiceSessionEnd消息类型SESSION_IDLE_TIMEOUT_MS = 180_000）、手动静音/恢复（复用通话圆圈，不新增按钮）、切后台自动静音（新增AppStateProvider.ts，react-nativeAppState首次接入）PhoneCallIcon.tsx+VoiceCallScreen.tsx，接入HomeScreen.tsx，跟按住说话入口并存end_conversation之后模型又被要求回一次话，说完告别语又被截断第二次；continuous 模式下 pump 提前退出（vendor 报错）没人发现，导致客户端卡在等一个没人再读的麦克风；打断（barge-in）落在"生成已完成但播放还没追上"这个窗口内会被漏掉，因为 vendor 生成比播放快location_search接入（#235）httpx.AsyncClient+TencentMapsLocationPort+LocationSearchService，按voice_agent_mode=="1"且腾讯凭据齐全为条件装配，lifespan里关闭session.hello早就在发的latitude/longitude/coordinate_system，之前后端接住了但一直没往下传——现在作为裸值一路透传到RealtimeAgent，在 session 开启那一刻组装一次校验过的ClientLocation（跟timezone同样"开的时候定死"的语义），gateway 全程不导入intelligence（架构测试验证过没破坏分层）ToolBox注册真正的location_search（schema 跟地图模块共用一份），没有位置/没配置时降级成provider_unavailable，不从 schema 里拿掉这个工具/wsintelligence.location↔intelligence.conversation）location_search返回的高精度坐标抄进schedule_create时有时会带上引号变成字符串，被严格的数字校验拒绝，模型重试 4 次都没跳出这个坑——现在改成能识别数字字符串自动转换；location_search第一次准备位置上下文失败后（比如腾讯接口临时不可用）之前会卡死一整通电话，现在改成每次调用都重试，不是只试一次然后永久放弃验证
bash scripts/check.sh：ruff / mypy / 架构测试 / 全量 pytest（799 passed, 41 skipped），覆盖率 96.6%npx tsc --noEmit、jest、vitest、eslint 全过location_search真实腾讯地图 key 下搜索、多候选口头消歧、日程带坐标创建成功本次不含
/ws——voice_agent_mode=2现在还是直接 raise，这个和地图功能无关，是更大的独立缺口