Repository navigation
feat(agent): grab_frame tool + Gemini Vision + image-in-chat - #96
Merged
Merged
Conversation
…pport Enables the camera-vision demo: "what do the cameras see?" in the Chat tab calls grab_frame for each sensor node, injects JPEG frames into the next Gemini turn, and returns one coherent natural-language analysis. Architecture: - Camera node (rtsp-camera) owns encoding: kornia-rs RGBA→RGB→resize→JPEG - Agent grab_frame tool is a thin Zenoh query to bubbaloop/local/.../grab_frame - Images are injected out-of-band after tool_results via user_with_labeled_images - Gemini role-coalescing merges consecutive user messages (required for HTTP 200) New primitives: - ContentBlock::Image + ImageSource (provider-agnostic, Anthropic wire format) - Message::user_with_labeled_images (interleaved text labels + images) - GeminiPart::InlineData + role-coalescing in to_gemini_contents - Dispatcher image_stash (Mutex<HashMap<tool_use_id, ImageAttachment>>) - PlatformOperations::zenoh_session() for grab_frame Zenoh access - agent/camera.rs: Zenoh query client returning (receipt_json, ImageAttachment) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
…-use fix **Image in chat (dashboard)** - `AgentEvent` gains `image` + `media_type` optional fields and a new `Image` variant in `AgentEventType`. - After draining the image stash, the agent loop emits one SSE `image` event per captured frame so the dashboard can render them inline. - `ChatView` handles `image` events: renders `<img>` tags (max 420px wide) styled to match the existing tool-call row aesthetics. - `agent.ts` extends `AgentEvent` and `AgentEventType` with the new fields. - CLI (`agent_client.rs`, `StdoutSink`) silently ignores image events. **Instance-name resolution for grab_frame** - `PlatformOperations` gains `get_node_instance_name()`: maps the registry name (e.g. `tapo-terrace`) to the Zenoh instance name the node publishes under (e.g. `tapo_terrace_camera`), which is what the queryable key uses. - `NodeManager::get_instance_name()` returns `published_instance_name` or falls back to `effective_name()`. - `DaemonPlatform` and `MockPlatform` implement the new trait method. - `dispatch.rs` resolves the name before building the grab_frame query key. **Dangling function-call sanitization** - `sanitize_dangling_tool_use()` is now called at the *start* of every `run_agent_turn()`, not only on cancel/timeout. This prevents a Gemini HTTP 400 "function response must come immediately after function call" error when an SSE connection drops after the assistant message is stored but before the tool results are pushed into short-term memory. **Dashboard agent discovery fix** - `queryWithTimeout` races the zenoh-js receiver against a hard deadline (`timeoutMs + 500 ms`). Without this, the `for await` loop hangs because zenoh-ts receivers don't always close when the server-side timeout fires. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
edgarriba
changed the title
Two bugs caused "function response must come immediately after function
call turn" errors on the second grab_frame call in a conversation:
1. **functionResponse.name mismatch** — to_gemini_contents was using the
internal tool_use_id UUID ("gemini-abc123") as the functionResponse.name,
but Gemini requires it to match the functionCall.name ("grab_frame").
Added tool_name_map() to scan ToolUse blocks and look up real function
names when building FunctionResponse parts.
2. **base64 blob accumulation** — labeled_images were pushed permanently to
short_term. On subsequent grab_frame calls, the accumulated image messages
caused role-sequencing inconsistencies for Gemini's validator. Fixed by
making the image message transient: it's queued in pending_image_msg for
the analysis provider call only, then discarded. Dashboard images still
stream via SSE events as before.
Adds regression test: to_gemini_contents_function_response_uses_function_name_not_id.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The agent was calling grab_frame proactively (e.g. on "Hi"), sending unsolicited camera frames to the dashboard. Added explicit guidance to the system prompt: only call grab_frame when the user's message explicitly requests visual information (see, show, visualize, analyze a camera feed). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Added explicit rule at top of the "Rules" section: only call tools when the user explicitly requests an action or investigation. Greetings, chitchat, and conversational messages get a plain text reply — no system_status, grab_frame, or any other tool calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add explicit instruction in tool description and query param hint to combine related keywords into a single broad query rather than firing one memory_search call per keyword. Reduces 3-turn memory lookups to 1 turn for typical "check memories about X" requests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Tool calls collapse to a single expandable line showing tool name, turn number, and episode count; result fills in when it arrives - System context events (memory episodes, turn N/M) are no longer shown as chat rows — metadata is captured in refs and surfaced inside tool call lines only - Consecutive image entries are grouped into a horizontal thumbnail strip (110×62px, nowrap, overflow-x scroll); single image renders at natural size up to 320px; click any thumbnail opens a fullscreen lightbox (ESC or click outside to dismiss) - Scroll-to-bottom floating button appears when user has scrolled up more than 80px; disappears when back at bottom; sending a message resets to auto-scroll Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extract agentColor() helper and AgentTriggerContent component to deduplicate color/name/model rendering across single-agent chip, dropdown trigger, and pill - Move agent-panel-header inline style to .agent-panel-title class - Replace [...prev].reverse().findIndex() with a backward for-loop in tool_result handler (no array copy, no index arithmetic, no cast) - Lift entries render IIFE into renderEntries() top-level helper Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- streamingRef (not just state) guards against concurrent handleSend invocations before setStreaming(true) commits - mountedRef prevents setState after unmount; setStreaming(false) gated in finally; error/connection_lost gated in catch - programmaticScrollRef 600ms window stops smooth-scroll feedback loop from flipping userScrolledRef and showing scroll button mid-stream - handleSelectAgent aborts in-flight stream when user switches agents - runDiscovery preserves current selection on refresh; falls back to default/first only when selected agent is no longer in the list - Token trimmed before both saveToken() and setToken() to prevent trailing-whitespace Bearer header - episodesRef only updated when parseEpisodes() returns > 0 so unrelated system messages don't clobber a valid count - Network TypeError classified as connection_lost not raw error - IME composition guard on Enter key (CJK input) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous pattern "message " also matched "user's message explicitly" in the grab_frame prompt rule, giving count 6 instead of 5. Switch to per-index contains() check so only log entries are counted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This was referenced May 18, 2026
edgarriba
added a commit
that referenced
this pull request
May 19, 2026
Ships 10 commits worth of features and fixes since v0.0.13: Features - agent: grab_frame tool + Gemini Vision provider + image-in-chat (#96) - dashboard: Chat tab with HTTP+SSE agent gateway (#91) - agent: client-side turn cancellation via Zenoh cancel topic (#94, #95) - chat: multi-provider login status, 404 on unknown agent, Claude OAuth risk warning (#93) - mcp: 'bubbaloop mcp --token' prints bearer token for .mcp.json (#90) Fixes - agent: grab_frame reads binary JPEG payload + JSON metadata attachment from camera nodes (#99) - chat: clear Responding state after multi-camera grab_frame (#98) - mcp: eliminate token race between gateway and agent runtime (#89) Chores - dashboard: remove LibraryView component (#92, replaced by Chat tab) Docs - Comprehensive sync across README, CHANGELOG (backfilled v0.0.8 to v0.0.13), CLAUDE.md, ARCHITECTURE.md, ROADMAP.md, concept/reference/ dashboard docs to match the new feature surface (#100) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Imagevariant in the provider model; Gemini converter maps it toinline_dataparts; consecutive same-role messages coalesced to satisfy Gemini's strict role-alternation requirementfunctionResponse.namenow uses the actual function name (grab_frame) looked up viatool_name_map(), not the internal UUID; fixes HTTP 400 on second camera request in the same sessionpending_image_msgfor one provider call only, never stored inshort_term; prevents base64 blob accumulation across turnsgrab_frameonly fires when user explicitly asks to see camera feeds⚙ tool_name · turn N · X ep; system context events hidden from chat rows110×62pxstrip; click to open fullscreen lightbox (ESC / click-outside to dismiss)↓button appears when user scrolls up >80px; disappears at bottom; send resets auto-scrollTest plan
memory_searchcall with combined query↓button appears; click it → snaps to bottom…, done shows✓🤖 Generated with Claude Code