[AGENT-2144] VNC agent tool — vision + interaction via VNC protocol

[dnviti]

Code: AGENT-2144
Priority: LOW
Section: Orchestration — Agent Orchestration Gateway (Phase 5: Advanced — RDP/VNC Vision Agents)
Dependencies: AGENT-2142, AGENT-2143

Description

Extend the RDP vision and interaction tools to work with VNC connections. Since both RDP and VNC are proxied through Guacamole, the underlying Guacamole protocol is identical — the same screenshot capture, keyboard injection, and mouse click mechanisms work for both protocols. This task adapts the RDP tools for VNC-specific session establishment and settings while reusing the core Guacamole interaction layer.

Technical Details

Server — VNC Tool Adapter (server/src/services/agentTools/vncTool.service.ts)

• Extends/wraps the RDP vision and interaction tools for VNC connections
• VNC session establishment:
  • Use existing vnc.service.ts → generateVncGuacamoleToken() for session creation
  • VNC-specific settings: viewOnly mode, color depth, cursor handling
  • Same headless pattern as RDP agent sessions
• Tool registration:
  • vnc_screenshot — reuses rdpScreenshot.service.ts screenshot capture (Guacamole protocol is identical)
  • vnc_click — reuses rdpInput.service.ts mouse injection
  • vnc_type — reuses rdpInput.service.ts text input
  • vnc_key — reuses rdpInput.service.ts key combination
  • vnc_scroll — reuses rdpInput.service.ts scroll
• VNC-specific handling:
  • Resolution detection: VNC may have different default resolution than RDP
  • Encoding differences: Guacamole abstracts these, but screenshot quality may differ
  • viewOnly mode: if connection is configured as viewOnly, deny all write/execute tools

Server — Shared Guacamole Agent Layer

• Extract shared code from RDP tools into server/src/services/agentTools/guacamoleAgent.service.ts:
  • createAgentGuacSession(connectionId, protocol: 'rdp' | 'vnc') — unified session creation
  • captureFrame(sessionId) — protocol-agnostic screenshot
  • sendInput(sessionId, input) — protocol-agnostic input injection
  • analyzeWithVision(agentId, screenshot, question?) — shared LLM vision call
• Both RDP and VNC tools delegate to this shared layer
• Reduces code duplication and ensures consistent behavior

Server — Tool Schemas

• All VNC tools mirror their RDP counterparts:
  • vnc_screenshot: "Capture and analyze the current VNC remote desktop screen."
  • vnc_click: "Click at coordinates on the VNC remote desktop."
  • vnc_type: "Type text on the VNC remote desktop."
  • vnc_key: "Send a key combination on the VNC remote desktop."
  • vnc_scroll: "Scroll at coordinates on the VNC remote desktop."

Server — Permission and Security

• Same permission model as RDP tools (read for screenshot/scroll, write for click/type, execute for key combos)
• Same DLP enforcement
• Same keystroke inspection
• VNC viewOnly connections: auto-deny write/execute tools regardless of agent permissions

Files Involved

• CREATE server/src/services/agentTools/vncTool.service.ts — VNC tool adapter
• CREATE server/src/services/agentTools/guacamoleAgent.service.ts — Shared Guacamole agent layer
• MODIFY server/src/services/agentTools/rdpVisionTool.service.ts — Delegate to shared layer
• MODIFY server/src/services/agentTools/rdpInteractionTool.service.ts — Delegate to shared layer
• MODIFY server/src/services/agentToolRegistry.service.ts — Register vnc_screenshot, vnc_click, vnc_type, vnc_key, vnc_scroll tools

Reference: Agent Orchestration Gateway — Phase 5.3