[AGENT-2142] RDP agent tool — screenshot capture from Guacamole stream + LLM vision analysis

[dnviti]

Code: AGENT-2142
Priority: LOW
Section: Orchestration — Agent Orchestration Gateway (Phase 5: Advanced — RDP/VNC Vision Agents)
Dependencies: AGENT-2126

Description

Implement the first half of the RDP agent tool: screenshot capture from an active Guacamole RDP session and LLM vision analysis. This enables agents to "see" what's on a remote desktop by capturing the current screen state and sending it to a multimodal LLM for analysis. The agent can then decide what actions to take based on what it sees.

This is the foundation for vision-based RDP automation — agents interact with graphical desktops the same way a human would: look at the screen, understand what's there, and act accordingly.

Technical Details

Server — Screenshot Capture Service (server/src/services/agentTools/rdpScreenshot.service.ts)

• captureScreenshot(sessionId): Promise<Buffer>
  • Capture current frame from Guacamole session
  • Option 1: Use Guacamole screenshot instruction (screenshot opcode) if supported by guacamole-lite
  • Option 2: Use recording stream — capture a single frame from the live Guac protocol stream and render to PNG using guacamole-common-js Canvas rendering server-side (via node-canvas or sharp)
  • Option 3: Tap into the guacd recording file and extract the latest frame
  • Return PNG buffer (compressed, max 1920x1080)
• analyzeScreen(agentId, screenshot: Buffer, question?: string): Promise<ScreenAnalysis>
  • Send screenshot to agent's resolved LLM provider (must support vision/multimodal)
  • System prompt: "You are analyzing a remote desktop screenshot. Describe what you see: window layout, application states, dialog boxes, error messages, buttons, text content. If asked a specific question, answer it based on what you see."
  • If question provided: append to prompt (e.g., "Is the installation complete?", "What error message is showing?")
  • Return: { description, elements: [{ type, text, location, confidence }], rawResponse }
  • Token counting: include image tokens in agent's budget tracking

Server — Vision Tool Handler (server/src/services/agentTools/rdpVisionTool.service.ts)

• rdp_screenshot tool:
  • Input: { connectionId, question? }
  • Establishes RDP session if not open (reuse existing if available)
  • Captures screenshot
  • Sends to LLM for analysis
  • Returns: { description, elements[], imageBase64? }
  • Log AgentActionLog: action='rdp_screenshot', input=question, output=description
• Permission check: agent must have 'read' capability on the connection

Server — Session Management

• RDP sessions for agents: established via existing rdp.service.ts token generation + guacamole-lite
• Agent sessions are headless — no client WebSocket, server consumes the Guacamole protocol stream directly
• Session keepalive: periodic NOP instructions to prevent idle timeout
• Session recording: enabled for audit trail

Server — LLM Provider Validation

• Before attempting vision: verify resolved LLM supports multimodal input
  • Anthropic: Claude 3+ models support vision
  • OpenAI: GPT-4V+ models support vision
  • Ollama: depends on model (llava, etc.)
• If provider doesn't support vision: return error with message "LLM provider does not support vision analysis"

Files Involved

• CREATE server/src/services/agentTools/rdpScreenshot.service.ts — Screenshot capture from Guacamole
• CREATE server/src/services/agentTools/rdpVisionTool.service.ts — RDP vision tool handler
• MODIFY server/src/services/agentToolRegistry.service.ts — Register rdp_screenshot tool
• MODIFY server/src/types/llm.types.ts — Add ImageContent message type for multimodal

Reference: Agent Orchestration Gateway — Phase 5.1