[AGENT-2143] RDP agent tool — keyboard/mouse injection via Guacamole protocol commands

[dnviti]

Code: AGENT-2143
Priority: LOW
Section: Orchestration — Agent Orchestration Gateway (Phase 5: Advanced — RDP/VNC Vision Agents)
Dependencies: AGENT-2142

Description

Implement the second half of the RDP agent tool: keyboard and mouse input injection via the Guacamole protocol. Combined with the screenshot/vision capability (AGENT-2142), this enables complete vision-based RDP automation — agents can see the screen, decide where to click or what to type, and execute those actions programmatically through the Guacamole protocol.

Technical Details

Server — Input Injection Service (server/src/services/agentTools/rdpInput.service.ts)

• sendKeyPress(sessionId, key: string | number): Promise<void>
  • Send Guacamole key instruction (keycode + pressed/released events)
  • Support named keys: 'Enter', 'Tab', 'Escape', 'Backspace', 'Delete', arrow keys, modifier keys
  • Support key combinations: Ctrl+C, Alt+Tab, Ctrl+Alt+Delete
• sendText(sessionId, text: string): Promise<void>
  • Send text as sequence of key press/release events
  • Handle special characters and Unicode
  • Configurable inter-key delay (default 50ms) for realistic typing
• sendMouseClick(sessionId, x: number, y: number, button?: 'left' | 'right' | 'middle'): Promise<void>
  • Send Guacamole mouse instruction with coordinates and button mask
  • Sequence: move → button down → brief delay → button up
• sendMouseDoubleClick(sessionId, x: number, y: number): Promise<void>
  • Two click sequences with standard double-click interval
• sendMouseDrag(sessionId, startX, startY, endX, endY): Promise<void>
  • Mouse down at start, move to end, mouse up
• sendMouseScroll(sessionId, x: number, y: number, direction: 'up' | 'down', amount?: number): Promise<void>
  • Scroll wheel instruction at coordinates

Server — RDP Interaction Tool (server/src/services/agentTools/rdpInteractionTool.service.ts)

• rdp_click tool:
  • Input: { connectionId, x, y, button?, doubleClick? }
  • Permission: 'write' capability required
  • Sends click, waits briefly, captures screenshot, returns visual confirmation
• rdp_type tool:
  • Input: { connectionId, text, pressEnter? }
  • Permission: 'write' capability required
  • Types text, optionally presses Enter
  • DLP check: if disablePaste is true, this simulates typing (not paste), so allowed
• rdp_key tool:
  • Input: { connectionId, key, modifiers?: string[] }
  • Permission: 'execute' capability required (for key combos like Ctrl+Alt+Delete)
  • Sends key combination
• rdp_scroll tool:
  • Input: { connectionId, x, y, direction, amount? }
  • Permission: 'read' capability sufficient

Server — Safety Controls

• Keystroke inspection: agent keyboard input piped through keystroke inspection (if enabled for connection)
• Rate limiting: max input events per second (default 20) to prevent accidental input storms
• Dangerous key combo protection: Ctrl+Alt+Delete, Alt+F4, Windows key sequences require 'execute' permission
• DLP: clipboard operations (Ctrl+C/V) respect DLP policy

Server — Coordinate System

• Screenshot coordinates match Guacamole's coordinate system (0,0 = top-left)
• LLM vision analysis returns approximate element locations from screenshot
• Agent uses these coordinates for click targets
• Allow small coordinate offsets for retry on missed clicks

Files Involved

• CREATE server/src/services/agentTools/rdpInput.service.ts — Guacamole input injection
• CREATE server/src/services/agentTools/rdpInteractionTool.service.ts — RDP interaction tools
• MODIFY server/src/services/agentToolRegistry.service.ts — Register rdp_click, rdp_type, rdp_key, rdp_scroll tools

Reference: Agent Orchestration Gateway — Phase 5.2

[m13v]

the Guacamole protocol approach for RDP agent automation is solid for remote desktop scenarios. one thing to consider for the coordinate-based interaction: since you're working through a remote display protocol, you don't have access to the remote machine's accessibility tree (UI Automation elements with roles and labels). this means the agent is purely vision-based for element targeting.

to improve click reliability, consider adding an element detection layer on top of the screenshots. OCR the screenshot to find text labels, then click relative to the detected text rather than relying on absolute coordinates. text positions are more stable than specific pixel coordinates when the remote desktop resolution or DPI changes.

for the sendText implementation, the inter-key delay approach works but watch out for input method editors on the remote machine. if the remote Windows has CJK input enabled, individual key events will get intercepted by the IME. a more reliable approach for arbitrary text: inject the text via the clipboard (Ctrl+V) through the Guacamole protocol. send Ctrl+C to copy the text to the remote clipboard, then Ctrl+V to paste. this bypasses IME issues entirely.

the screen annotation overlay for showing planned actions is a great UX decision. in our desktop automation work we found that showing the agent's planned click target before executing gives the operator confidence and catches misclicks before they happen.

[m13v]

for the local desktop automation case (non-RDP), our cross-platform element interaction layer that handles both accessibility-based and coordinate-based clicking with the pre-click validation pattern: https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/element.rs

and the core automation library with the clipboard-based text injection that bypasses IME issues: https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/lib.rs