Repository navigation
Conversation
…pport Enables the camera-vision demo: "what do the cameras see?" in the Chat tab calls grab_frame for each sensor node, injects JPEG frames into the next Gemini turn, and returns one coherent natural-language analysis. Architecture: - Camera node (rtsp-camera) owns encoding: kornia-rs RGBA→RGB→resize→JPEG - Agent grab_frame tool is a thin Zenoh query to bubbaloop/local/.../grab_frame - Images are injected out-of-band after tool_results via user_with_labeled_images - Gemini role-coalescing merges consecutive user messages (required for HTTP 200) New primitives: - ContentBlock::Image + ImageSource (provider-agnostic, Anthropic wire format) - Message::user_with_labeled_images (interleaved text labels + images) - GeminiPart::InlineData + role-coalescing in to_gemini_contents - Dispatcher image_stash (Mutex<HashMap<tool_use_id, ImageAttachment>>) - PlatformOperations::zenoh_session() for grab_frame Zenoh access - agent/camera.rs: Zenoh query client returning (receipt_json, ImageAttachment) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-use fix **Image in chat (dashboard)** - `AgentEvent` gains `image` + `media_type` optional fields and a new `Image` variant in `AgentEventType`. - After draining the image stash, the agent loop emits one SSE `image` event per captured frame so the dashboard can render them inline. - `ChatView` handles `image` events: renders `<img>` tags (max 420px wide) styled to match the existing tool-call row aesthetics. - `agent.ts` extends `AgentEvent` and `AgentEventType` with the new fields. - CLI (`agent_client.rs`, `StdoutSink`) silently ignores image events. **Instance-name resolution for grab_frame** - `PlatformOperations` gains `get_node_instance_name()`: maps the registry name (e.g. `tapo-terrace`) to the Zenoh instance name the node publishes under (e.g. `tapo_terrace_camera`), which is what the queryable key uses. - `NodeManager::get_instance_name()` returns `published_instance_name` or falls back to `effective_name()`. - `DaemonPlatform` and `MockPlatform` implement the new trait method. - `dispatch.rs` resolves the name before building the grab_frame query key. **Dangling function-call sanitization** - `sanitize_dangling_tool_use()` is now called at the *start* of every `run_agent_turn()`, not only on cancel/timeout. This prevents a Gemini HTTP 400 "function response must come immediately after function call" error when an SSE connection drops after the assistant message is stored but before the tool results are pushed into short-term memory. **Dashboard agent discovery fix** - `queryWithTimeout` races the zenoh-js receiver against a hard deadline (`timeoutMs + 500 ms`). Without this, the `for await` loop hangs because zenoh-ts receivers don't always close when the server-side timeout fires. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs caused "function response must come immediately after function
call turn" errors on the second grab_frame call in a conversation:
1. **functionResponse.name mismatch** — to_gemini_contents was using the
internal tool_use_id UUID ("gemini-abc123") as the functionResponse.name,
but Gemini requires it to match the functionCall.name ("grab_frame").
Added tool_name_map() to scan ToolUse blocks and look up real function
names when building FunctionResponse parts.
2. **base64 blob accumulation** — labeled_images were pushed permanently to
short_term. On subsequent grab_frame calls, the accumulated image messages
caused role-sequencing inconsistencies for Gemini's validator. Fixed by
making the image message transient: it's queued in pending_image_msg for
the analysis provider call only, then discarded. Dashboard images still
stream via SSE events as before.
Adds regression test: to_gemini_contents_function_response_uses_function_name_not_id.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The agent was calling grab_frame proactively (e.g. on "Hi"), sending unsolicited camera frames to the dashboard. Added explicit guidance to the system prompt: only call grab_frame when the user's message explicitly requests visual information (see, show, visualize, analyze a camera feed). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Added explicit rule at top of the "Rules" section: only call tools when the user explicitly requests an action or investigation. Greetings, chitchat, and conversational messages get a plain text reply — no system_status, grab_frame, or any other tool calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add explicit instruction in tool description and query param hint to combine related keywords into a single broad query rather than firing one memory_search call per keyword. Reduces 3-turn memory lookups to 1 turn for typical "check memories about X" requests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Tool calls collapse to a single expandable line showing tool name, turn number, and episode count; result fills in when it arrives - System context events (memory episodes, turn N/M) are no longer shown as chat rows — metadata is captured in refs and surfaced inside tool call lines only - Consecutive image entries are grouped into a horizontal thumbnail strip (110×62px, nowrap, overflow-x scroll); single image renders at natural size up to 320px; click any thumbnail opens a fullscreen lightbox (ESC or click outside to dismiss) - Scroll-to-bottom floating button appears when user has scrolled up more than 80px; disappears when back at bottom; sending a message resets to auto-scroll Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extract agentColor() helper and AgentTriggerContent component to deduplicate color/name/model rendering across single-agent chip, dropdown trigger, and pill - Move agent-panel-header inline style to .agent-panel-title class - Replace [...prev].reverse().findIndex() with a backward for-loop in tool_result handler (no array copy, no index arithmetic, no cast) - Lift entries render IIFE into renderEntries() top-level helper Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- streamingRef (not just state) guards against concurrent handleSend invocations before setStreaming(true) commits - mountedRef prevents setState after unmount; setStreaming(false) gated in finally; error/connection_lost gated in catch - programmaticScrollRef 600ms window stops smooth-scroll feedback loop from flipping userScrolledRef and showing scroll button mid-stream - handleSelectAgent aborts in-flight stream when user switches agents - runDiscovery preserves current selection on refresh; falls back to default/first only when selected agent is no longer in the list - Token trimmed before both saveToken() and setToken() to prevent trailing-whitespace Bearer header - episodesRef only updated when parseEpisodes() returns > 0 so unrelated system messages don't clobber a valid count - Network TypeError classified as connection_lost not raw error - IME composition guard on Enter key (CJK input) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous pattern "message " also matched "user's message explicitly" in the grab_frame prompt rule, giving count 6 instead of 5. Switch to per-index contains() check so only log entries are counted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix React StrictMode bug: mountedRef.current was permanently false after StrictMode's double-invoke cleanup, so setStreaming(false) in the finally block was silently skipped. Reset mountedRef.current=true in effect setup; drop the guard entirely (React 18 safe no-op). - Fix SSE generator hang: parseSseStream now accepts AbortSignal and races every reader.read() against it. Calling ctrl.abort() on done/ error immediately unblocks any pending read so generator.return() can complete and the outer finally runs. - Fix labeled-break no-op: break inside switch only exits the switch, not the for-await. Changed to sseLoop: label + break sseLoop. - Fix Stop button: ctrl.abort() now reliably interrupts the in-flight reader.read() via the abort promise race. - grab_frame: remove specific camera name examples from tool schema (biased LLM toward tapo-terrace). System prompt now instructs agent to call grab_frame for ALL camera nodes when no specific camera is named by the user. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
React.StrictModedouble-invokes effect cleanups in dev, permanently settingmountedRef.current = false— soif (mountedRef.current) setStreaming(false)in thefinallyblock was always skipped. Fix resets the ref in the effect setup and drops the guard (React 18 makessetStateon unmounted components a safe no-op).breakinsideswitchonly exits theswitch, not thefor awaitloop. AddedsseLoop:label +break sseLoop.parseSseStreamnow acceptsAbortSignaland races everyreader.read()against it —ctrl.abort()ondone/errorimmediately unblocks any pending read sogenerator.return()completes and the outerfinallyruns.reader.read()through the Vite proxy keep-alive connection.tapo-terrace). System prompt now instructs agent to callgrab_framefor all camera nodes when no specific camera is named.Test plan
🤖 Generated with Claude Code