Repository navigation
feat(mobile): show chat visuals inline in the phone's native chat - #26071
Conversation
…s visuals folder
A shared grammar for the ::orca-visual{file="..." title="..."} reply line,
the per-chat visuals folder location on the owning host, and the
agentSession.readVisual runtime method that reads one visual with lexical and
canonical containment, a 512 KiB bounded read and UTF-8 refusal.
One string builder every client wraps a visual's HTML with: the policy (CDN assets only, no fetch, frames, workers, forms or base rewrites), the theme variables, and a prelude that reports height, routes links to the parent and refuses navigation. Also the validated frame-to-parent message reader and the live theme message.
A finished assistant reply's ::orca-visual line now shows the visual inline, read from the chat's owning host through agentSession.readVisual. The visual runs in an opaque sandboxed child of a trusted host document inside the WebView; the app accepts only a token-checked height and an http(s) link opened under user activation. Navigation away from the visual's own document is refused, a dead web process reloads once, and the visual opens full screen. The hybrid shell's page renders it sealed.
Native-chat assistant replies render a ::orca-visual{...} line as the chat's
HTML visual in an opaque, scripts-only sandboxed frame: CSP first, the host
frame navigation guard registered before content runs, live theme without a
reload, fitted height, links opened in the viewer's browser only from a real
gesture, lazy mount, and one muted line when the visual cannot be shown.
Open in sidebar shows the same frame in the right sidebar, widened while it
is open and restored after.
…er' into brennanb2025/inline-visuals-mobile
…y message the app Review round 1: - use PR 1's shared visual shell and height governor instead of a second builder; the app decides heights and pushes them to the host page - react-native-webview patch: the message channel accepts only string messages from the main frame on iOS and Android, so a visual in its sandboxed child cannot reach it (or crash Android with a non-string) - structured replies grow in place: while a turn works, a row holds back a directive still being typed at its tail; finished lines mount - links need child focus + activation, one per activation window - a refused read takes the visual down and drops its cached bytes - in-page anchors load; text spelling a placeholder renders no visuals
The phone bundles only src/shared, so the governor the desktop frame uses moves there unchanged and the mobile frame shares it.
- host page relays only messages on the visual's own channel, as its own copy, so a visual cannot push oversized fields through it; relay rate halved; a link is validated before it uses up the link window - the frame denies camera, microphone, geolocation, clipboard and display capture; iOS media capture requests are denied - react-native-webview patch: the iOS history-shim handler also accepts only main-frame string messages - only the newest assistant row of a working turn holds back an unfinished directive; earlier finished rows show their visuals - an error reply or older host is not a verdict: it keeps a visual on screen and its cache, and retries; only a host refusal takes it down
iOS gives a scrollable frame its own scroll view, which took the drag; the inline frame is sized to its content, so it no longer scrolls. Full screen still does.
- hold back only the last block of the newest assistant row, and not while a question or approval is open, so a visual followed by a tool call or a pending question shows at once - an older host (method_not_found) reads as unavailable without retries - drop a stray @pnpm/exe lockfile block; only the patch hash changes
…mpt-controller harness
Review summaryHead: Review roundsRound 1 (correctness + a separate security pass):
Round 2:
QA, between rounds:
Round 3:
Round 4:
Live QAiOS Simulator (iPhone 17 Pro, iOS 26.5). The dev client was built from this branch, including the native patch, and rebuilt after each native change. The app was paired with the repo's mobile mock host serving a structured chat with three visual lines; that mock scenario is QA tooling and is not committed. Verified:
Gates run locally
CI, classified (head
|
…streaming hold Registers agentSession.readVisual from the methods index so the structured method file stays under its line budget, replaces reflective reads with checked narrowing, moves the pure height governor to src/shared for mobile, and holds a half-written directive tail while the turn works (structured text rows carry no running state).
Re-checks after the open that the chat's visuals folder is still the real directory at Orca's path, reports unexpected filesystem faults by code without host paths, and lets one click in a visual open at most one page.
…er' into brennanb2025/inline-visuals-mobile
… review fixes One shared helper drops visual lines (outside fenced code) from reply text where it becomes plain text: the structured status summary that feeds the sidebar row, dashboard, notifications, phone rows and handoffs, and AI Vault reply previews. Review fixes: height also counts a pinned body's overflow, only the live frontier row holds a half-written visual line, the runaway-height stop needs the same step repeated, and any host refusal evicts the cached revision.
…nal-paths helper Main removed the per-chat journal paths and the journal database's state directory; the visuals folder keeps the same sha256 layout on its own and the read method uses the profile state directory the chat host is opened in.
…er' into brennanb2025/inline-visuals-mobile
…sage text A reply's ::orca-visual line renders only in the transcript. The worktree list's agent row and the message actions sheet's copy text now drop it with the shared helper; a reply that is only a visual falls back to the prompt, as an empty one does.
…al lines Reply previews in Agent Session History drop visual lines per text part before lines are folded; the frame adds a body's overflow only when the body really overflows; fence tracking follows CommonMark closers and openers; the copy button copies a reply without visual lines; a coded read fault keeps its cause.
…er' into brennanb2025/inline-visuals-mobile
…er' into brennanb2025/inline-visuals-mobile
… visuals folder pair at once
…er' into brennanb2025/inline-visuals-mobile
…on agent-session surface
…er' into brennanb2025/inline-visuals-mobile
Removing visual lines now closes only the gap each removal leaves, instead of collapsing blank lines across the whole reply and trimming its indentation; the visuals folder is checked parent first again so a broken path answers the same way every time.
…er' into brennanb2025/inline-visuals-mobile
|
Important Review skippedReview was skipped as selected files did not have any reviewable changes. ⛔ Files ignored due to path filters (1)
⚙️ Run configuration
⛔ Files ignored due to path filters (1)
You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe change adds native-chat visual directives across desktop and mobile chat surfaces. It defines shared parsing, validation, filtering, sandboxing, sizing, and frame-message handling. It adds secure visual-file reads through Priority: ➖ Normal Merge Risk: 🔵 Low · up to Inline chat visuals are mostly well contained. Session previews can still do full-length work on very long assistant replies. On mobile, a visual can show as unavailable after one process crash instead of reloading. Both fixes are small follow-ups. 🚥 Pre-merge checks | ✅ 4 | ❓ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 37.84% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 74 functions across 50 files. (37 skipped: 3 unsupported, 34 over the file limit.) ✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
74ca2c8b-64f0-42e0-8394-f248cd46949e
⛔ Files ignored due to path filters (2)
mobile/pnpm-lock.yamlis excluded by!**/pnpm-lock.yamlsrc/shared/rpc-contract/rpc-params-catalog.generated.tsis excluded by!**/*.generated.*
📒 Files selected for processing (87)
mobile/patches/react-native-webview@13.16.2.patchmobile/src/components/MobileMarkdown.tsxmobile/src/components/WorktreeAgentRow.tsxmobile/src/components/WorktreeAgentRow.visual-line.test.tsxmobile/src/components/inline-script-json.tsmobile/src/components/mobile-markdown-parser.tsmobile/src/components/mobile-markdown-visual-lines.test.tsmobile/src/components/mobile-markdown-visual-lines.tsmobile/src/components/pr-sidebar/MermaidDiagram.tsxmobile/src/components/use-mobile-markdown-blocks.tsmobile/src/session/MobileNativeChatMessage.tsxmobile/src/session/MobileNativeChatOverlay.test.tsmobile/src/session/MobileNativeChatOverlay.tsxmobile/src/session/MobileNativeChatVisual.tsxmobile/src/session/MobileNativeChatVisualFrame.tsxmobile/src/session/MobileNativeChatVisualFrame.web.tsxmobile/src/session/__mocks__/mobile-prompt-controller.tsmobile/src/session/mobile-native-chat-controller-contract.tsmobile/src/session/mobile-native-chat-merged-snapshot-parity-hooks.test.tsxmobile/src/session/mobile-native-chat-message-plain-text.test.tsmobile/src/session/mobile-native-chat-message-plain-text.tsmobile/src/session/mobile-native-chat-message-visuals.test.tsmobile/src/session/mobile-native-chat-visual-bridge.test.tsmobile/src/session/mobile-native-chat-visual-bridge.tsmobile/src/session/mobile-native-chat-visual-context.tsmobile/src/session/mobile-native-chat-visual-host-document.tsmobile/src/session/mobile-native-chat-visual-read.test.tsmobile/src/session/mobile-native-chat-visual-read.tsmobile/src/session/mobile-native-chat-visual-theme.tsmobile/src/session/use-mobile-native-chat-controller.tsmobile/src/session/use-mobile-native-chat-turn-disclosure-growing-row.test.tsxmobile/src/session/use-mobile-native-chat-turn-disclosure.tsmobile/src/session/use-mobile-native-chat-visual.test.tsxmobile/src/session/use-mobile-native-chat-visual.tsmobile/src/session/use-mobile-structured-agent-session.tsmobile/web-entry/web-overrides.jsonsrc/main/ai-vault/session-scanner-accumulator.tssrc/main/ai-vault/session-scanner-text-normalization.tssrc/main/ai-vault/session-scanner-values.test.tssrc/main/native-chat/native-chat-visual-file-read.test.tssrc/main/native-chat/native-chat-visual-file-read.tssrc/main/native-chat/native-chat-visuals-folder.tssrc/main/runtime/mobile-rpc-allowlist.test.tssrc/main/runtime/rpc/methods/index.tssrc/main/runtime/rpc/methods/structured-agent-session-rpc.test-fixture.tssrc/main/runtime/rpc/methods/structured-agent-session-visual.test.tssrc/main/runtime/rpc/methods/structured-agent-session-visual.tssrc/main/runtime/runtime-rpc/runtime-rpc-mobile-agent-session-methods.tssrc/main/window/host-frame-navigation-guard.test.tssrc/main/window/host-frame-navigation-guard.tssrc/main/window/main-window-webview-security.test.tssrc/main/window/main-window-webview-security.tssrc/renderer/src/components/native-chat/NativeChatInlineVisual.tsxsrc/renderer/src/components/native-chat/NativeChatMarkdown.tsxsrc/renderer/src/components/native-chat/NativeChatMarkdown.visual.test.tsxsrc/renderer/src/components/native-chat/NativeChatMessageRow.test.tsxsrc/renderer/src/components/native-chat/NativeChatMessageRow.tsxsrc/renderer/src/components/native-chat/NativeChatTranscriptChrome.tsxsrc/renderer/src/components/native-chat/NativeChatView.tsxsrc/renderer/src/components/native-chat/NativeChatVisualFrame.test.tsxsrc/renderer/src/components/native-chat/NativeChatVisualFrame.tsxsrc/renderer/src/components/native-chat/NativeChatVisualPanel.tsxsrc/renderer/src/components/native-chat/native-chat-visual-markdown-extension.tsxsrc/renderer/src/components/native-chat/native-chat-visual-markdown-syntax.test.tsxsrc/renderer/src/components/native-chat/native-chat-visual-markdown-syntax.tssrc/renderer/src/components/native-chat/native-chat-visual-owner.tsxsrc/renderer/src/components/native-chat/native-chat-visual-read-client.test.tssrc/renderer/src/components/native-chat/native-chat-visual-read-client.tssrc/renderer/src/components/native-chat/use-native-chat-visual-document.tssrc/renderer/src/components/native-chat/use-native-chat-visual-theme.tssrc/renderer/src/components/right-sidebar/index.tsxsrc/renderer/src/components/right-sidebar/right-sidebar-panel-content.tsxsrc/renderer/src/components/right-sidebar/right-sidebar-width.tssrc/renderer/src/components/sidebar/CommentMarkdown.tsxsrc/renderer/src/i18n/locales/en.jsonsrc/renderer/src/store/slices/editor/actions/right-sidebar-state.tssrc/renderer/src/store/slices/editor/actions/right-sidebar-visual-state.test.tssrc/shared/native-chat-visual-directive.test.tssrc/shared/native-chat-visual-directive.tssrc/shared/native-chat-visual-height-governor.test.tssrc/shared/native-chat-visual-height-governor.tssrc/shared/native-chat-visual-shell.test.tssrc/shared/native-chat-visual-shell.tssrc/shared/rpc-contract/agent-session-visual-params.tssrc/shared/structured-agent-session-latest-request.test.tssrc/shared/structured-agent-session-latest-request.tstests/e2e/cross-version-wire/structured-agent-session-surface-manifest.ts
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.
| const restart = useCallback(() => { | ||
| if (restarts.current >= MAX_PROCESS_RESTARTS) { | ||
| onFailed() | ||
| return | ||
| } | ||
| restarts.current += 1 | ||
| setHeight(MOBILE_NATIVE_CHAT_VISUAL_INITIAL_HEIGHT) | ||
| setGeneration((value) => value + 1) | ||
| }, [onFailed]) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Reset restarts when built changes.
restarts is a component-lifetime ref. The component never resets it when html changes. Suppose the web process dies once for one revision. A later revision then reaches restart with restarts.current === 1, so its first process crash calls onFailed() without the automatic reload. The fix: reset the counter in the existing [built, generation] effect when built changes, or in a separate effect keyed on built.
🐛 Proposed fix
+ useEffect(() => {
+ restarts.current = 0
+ }, [built])| } | ||
|
|
||
| function previewSource(text: string, dropVisualLines: boolean): string { | ||
| return dropVisualLines ? withoutNativeChatVisualDirectiveLines(text) : text |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
🔎 Supported by static analysis
🏁 Script executed:
set -u
printf '%s\n' '--- changed file ---'
nl -ba src/main/ai-vault/session-scanner-text-normalization.ts
printf '%s\n' '--- references ---'
rg -n -F -- 'extractPreviewContentText(' . --glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' || test "$?" -eq 1
rg -n -F -- 'normalizePreviewText(' . --glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' || test "$?" -eq 1
rg -n -F -- 'previewSource' src test tests . --glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' || test "$?" -eq 1
printf '%s\n' '--- diff from merge base ---'
git diff --no-ext-diff --unified=80 c8dcffce46d92ddbb1f511baf88df60d322f4ac1 c37b189a78cffdcf114158d26274d67c555ee8c7 -- src/main/ai-vault/session-scanner-text-normalization.ts src/main/ai-vaultRepository: stablyai/orca
Length of output: 41223
🏁 Script executed:
set -u
printf '%s\n' '--- directive helper ---'
rg -n -F -- 'withoutNativeChatVisualDirectiveLines' src shared . --glob='!node_modules/**' --glob='!dist/**' --glob='!build/**' 2>/dev/null || test "$?" -eq 1
printf '%s\n' '--- accumulator call paths ---'
nl -ba src/main/ai-vault/session-scanner-accumulator.ts | sed -n '210,290p'
printf '%s\n' '--- parser call sites ---'
for f in \
src/main/ai-vault/session-scanner-codex-message-records.ts \
src/main/ai-vault/session-scanner-droid-parser.ts \
src/main/ai-vault/session-scanner-opencode-parser.ts \
src/main/ai-vault/session-scanner-grok-parser.ts \
src/main/ai-vault/session-scanner-kimi-parser.ts; do
if [ -f "$f" ]; then
printf '%s\n' "--- $f ---"
nl -ba "$f" | sed -n '1,180p'
fi
done
printf '%s\n' '--- bounds and helper usages ---'
rg -n -e 'boundedText|cap.*Preview|PREVIEW.*LIMIT|preview.*slice|withoutNativeChatVisualDirectiveLines' src shared --glob='!node_modules/**' --glob='!dist/**' --glob='!build/**' 2>/dev/null || test "$?" -eq 1Repository: stablyai/orca
Length of output: 45239
🏁 Script executed:
set -u
printf '%s\n' '--- native-chat directive implementation ---'
nl -ba src/shared/native-chat-visual-directive.ts | sed -n '130,215p'
printf '%s\n' '--- bounded text implementation ---'
nl -ba src/main/ai-vault/session-transcript-message-content.ts | sed -n '145,175p'
printf '%s\n' '--- relevant preview exports ---'
rg -n -F -- 'export {' src/main/ai-vault/session-scanner-values.ts src/main/ai-vault/session-scanner-text-normalization.ts 2>/dev/null || trueRepository: stablyai/orca
Length of output: 5044
Keep assistant preview filtering bounded.
previewSource calls withoutNativeChatVisualDirectiveLines before the 220-character limit applies. That helper scans the full input and, when it finds a marker, splits, stores, and rejoins the full reply. The content path passes raw provider content into this operation without an upstream bound.
Filter incrementally and stop after collecting enough visible text for the preview.
ELI5
When an agent in a native chat wants to show a chart, it writes a small web page into the chat's own visuals folder and puts one line in its reply, like
::orca-visual{file="usage-chart.html" title="Usage by day"}. The desktop learns to draw that page inside the reply in the base PR. On the phone, that line still showed up as raw text. This PR makes the phone draw the chart inside the reply too, lets you open it full screen, and keeps the agent's page locked in a box it cannot get out of.What Changed
Before: in a structured chat on the phone, the visual line was shown as literal text (
::orca-visual{file="…"}), and the chart never appeared.After (iOS and Android app):
Mechanism:
src/shared/native-chat-visual-directive.ts). Only top-level lines outside code fences count (not quotes, lists, inline code or indented code), at most 8 per message. The line is lifted out before the phone's HTML clean-up step so a title is never rewritten. Only assistant rows of structured chats get this; every other markdown surface is unchanged.agentSession.readVisualon the host that owns the chat (now on the mobile method allowlist). The phone sends only the chat id and a bare file name; no path ever leaves the host. A small in-memory cache (16 entries, 4 MB of text) revalidates on every mount by content revision, shares concurrent reads, and drops a visual the host now refuses. Lost contact retries twice automatically, then waits for a tap; nothing latches.sandbox="allow-scripts"frame inside it, with an opaque origin: no storage, cookies, same-origin access, popups, forms or top navigation. It uses the base PR's shared shell (src/shared/native-chat-visual-shell.ts): content policy first, theme variables, height and link reporting. The host page relays only messages from that frame, stamps them with a random per-frame token the agent's page never sees, and asks only two things of the app: a height, which the shared height governor (moved tosrc/sharedso the phone can bundle it) clamps, rate-limits and stops from running away, and an http(s) link, which needs frame focus and user activation.react-native-webviewexposes its message channel to every frame on both platforms, including the sandboxed one. On Android a non-string message from there would crash the app. The repo's existing patch to that library now drops any message that is not a string from the main frame, on iOS and Android. That is the same rule the app's own shell bridge already enforces. The token check stays as a second fence.about:blank,about:srcdocand in-page anchors may load. The WebView whitelist is*on purpose: anything outside the library's whitelist is opened in the system browser by the library itself, with no gesture check. If the agent's frame ever loads a second document, the host page removes it and the visual shows as unavailable.Why
sandbox, and a<meta>policy cannot sandbox it. The common pattern serves the page over HTTP with a sandbox policy in a response header. Orca's phone has no per-file HTTP endpoint to the host (it reads over the runtime connection), so the frame sandbox is the equivalent isolation.Differences from the common pattern
Known residual risks (accepted, not fixed here)
clipboard-write 'none'covers the async clipboard API, notexecCommand('copy')during a tap.Linked Issue
Stacked on the base PR (branch
brennanb2025/inline-visuals-render), which adds the grammar, the host read, the shared shell and the desktop rendering. A sibling PR teaches the agents to write visuals.Visual Proof
iOS Simulator (iPhone 17 Pro, iOS 26.5), a dev client built from this branch, paired to the repo's mobile mock host serving a structured chat whose replies contain visual lines.
null, storage blockedAfter killing the visual's web processes, both visuals reloaded:
Testing
New tests:
Gates run locally:
tscand the tests-typecheck ratchettc:nodeand the web typecheckNot tested on Android: this machine has no JDK to build an Android dev client. The Android half of the native patch was checked against the androidx.webkit 1.14.0 API (
getType(),TYPE_STRING) but not compiled.AI Disclosure
Review
Agent skill upstream boundary
docs/reference/agent-skill-sharing-upstream-boundary.mdand copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.Notes
method_not_found, and the visual shows as unavailable.react-native-webviewpatch ships with the app binary.Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)