Add software-KVM: drive the receiver's native desktop from the sender's keyboard/mouse

[DrDavidL]

Summary

Adds a software KVM to TargetBridge: a sender-side toggle, "Control receiver Mac (keyboard & mouse)", that re-routes the sender's physical keyboard/mouse to the receiver Mac's native macOS desktop over the existing session — then back to the streamed display. Synergy / Universal-Control-style, on the link-local connection you already have.

Verified working end-to-end on real hardware (MacBook Pro sender → OCLP iMac receiver): toggle on → the receiver window minimizes to expose its native desktop → the MacBook keyboard/mouse drive the iMac (move / click / type / ⌘-shortcuts) → escape chord returns control and streaming resumes.

MVP scope (agreed): relative mouse (receiver owns/clamps the cursor), mouse + keyboard only, the toggle also hides the receiver window to reveal its desktop, and a keyboard escape (Control-Option-Command-K) that always returns control.

How to run it

Receiver (the Mac being controlled):

1. Build it as an app bundle: TargetBridge-Receiver/scripts/build_tbreceiver_c_app.sh → produces build/TargetBridge Receiver.app.
2. Drag it into /Applications and launch it from Finder. ⚠️ Do not run the bare ./tbreceiver from a terminal/VSCode — a bare-binary launch inherits the terminal's TCC/launch context and silently breaks the link-local data path (stream connects, sends the display profile, then dies with ENOTCONN / NWError 54 reset by peer). This is unrelated to the KVM code — pristine main fails identically from a terminal.
3. Add TargetBridge Receiver.app to System Settings → Privacy & Security → Accessibility (it injects events into the native session). Injection silently no-ops until granted.

Sender (the Mac you drive from):

1. Run the app; grant Screen Recording (for streaming) and, when you flip the KVM toggle, Accessibility (for the input tap). Add Input Monitoring too if keys don't come through.
2. Stream as usual, then flip "Control receiver Mac (keyboard & mouse)." The receiver window minimizes, and you're driving its native desktop. The on-screen indicator shows Control-Option-Command-K to return.

How it works

Protocol (TBMonitorProtocol.swift + receiver proto.h): five new JSON packet types inputControl / inputMouseMove / inputMouseButton / inputScroll / inputKey (0x33–0x37), reusing the existing length-prefixed framing.

Sender (TBKVMController.swift): an active kCGSessionEventTap on a dedicated thread + CFRunLoop (never the main runloop — a slow callback on an active tap would freeze system-wide input). It consumes keyboard/mouse, coalesces mouse-move deltas at ~120 Hz, parks the local cursor, and forwards over the session's NWConnection. Escape chord recognized inside the tap. Un-park failsafes cover tap-disabled re-enable, resign-active, terminate, connection drop, and an atexit cursor re-associate for the crash case. While active it declares local user activity every 30 s so the controlling Mac doesn't idle into the screensaver/lock mid-session (even while passively watching remote video).

Receiver (input.c / main.c): re-synthesizes events via CGEventCreate* + CGEventPost(kCGHIDEventTap, …), owning a virtual cursor clamped to CGMainDisplayID bounds (drag-vs-move by button state). inputControl minimizes/restores the SDL window to reveal/hide the native desktop; modifiers are reset on each KVM toggle so the escape chord can't leave keys stuck. Disconnect auto-exits KVM. Input init is lazy (first KVM engage) so the normal streaming path is identical to a build without KVM.

Commits

1. 223d000 — the feature (protocol + sender tap/UI + receiver injection/window-hide).
2. 5beadf6 — lazy receiver input init (keep the non-KVM path identical to a no-KVM build).
3. 54b859e — drop the ApplicationServices link / AXIsProcessTrusted probe (use CoreGraphics only; receiver Accessibility is granted manually).
4. 75e747f — spell out the escape chord as Control-Option-Command-K in the indicator.
5. 20d54b5 — keep the controlling Mac awake while KVM is active (screensaver/lock fix).
6. 9a512ed — fix a CI type-check timeout (Xcode 16.4) by building the CGEventMask with a loop instead of a 14-term OR-chain.
7. 39e19b1 — receiver CPU fix: the main loop busy-spun a core during streaming (constant fan); now poll()s the socket with a short timeout — same latency, idle CPU drops to ~nothing. Pre-existing behavior, not KVM-specific, but rides along here.

Known limitations / caveats

• Run the receiver as the .app, not a bare binary (see above) — the one thing most likely to bite.
• Ad-hoc signing keys TCC grants to a signature that changes every rebuild, so Screen Recording / Accessibility grants must be re-added after each rebuild. A stable self-signed identity would fix it (follow-up).
• Receiver Accessibility is added manually (no auto-prompt).

Test plan

• Both apps build clean; sender + receiver.
• End-to-end on hardware: control the iMac's native desktop (move/click/type/shortcuts); escape returns control.
• Screensaver: controlling Mac stays awake during KVM (no mid-session lockout) — fixed in 20d54b5.
• CI green on macOS-15 (sender Apple Silicon + receiver arm64/x86_64).
• Receiver CPU (39e19b1): confirm the iMac no longer pegs a core / runs the fan during streaming (rebuild the receiver .app).
• Failsafe drills: quit sender mid-KVM → local cursor returns (atexit); pull the link → KVM auto-exits; no stuck modifiers after escaping via the chord.
• Load: SD video + fast mouse during KVM → no input flood, no added stutter.

Out of scope (v2)

Clipboard sharing, drag-and-drop fidelity, double-click timing, multi-display + edge-flip, absolute-coordinate mode, binary input packets, configurable hotkey, stable code-signing for persistent grants.

🤖 Generated with Claude Code

[DrDavidL]

Still working on this...

[swellweb]

Thanks a lot for working on this — the software-KVM direction is definitely valuable.

I do not want to merge this directly onto main as-is, mainly because the current release work is already happening on feature/input-dockstation, and that branch now carries a lot of input-control / permissions / clipboard / brightness / receiver-master logic on top of newer protocol changes.

So the right path here is to rebase / rework the useful parts against feature/input-dockstation, rather than trying to land this patchset directly on top of main.

The main reason is to avoid regressions and protocol collisions:

• the current release branch already uses the input-related packet range
• there is already an Input Dockstation control flow in place
• permissions / receiver status / clipboard / brightness handling have all moved forward there

So: thank you — this is important work, but the fixes need to be adapted on feature/input-dockstation before they can be considered for the release branch.

[swellweb]

Small follow-up: I am actively working through this myself on feature/input-dockstation so we can adapt the KVM pieces against the current release branch carefully and avoid regressions. That is why I would prefer keeping the integration work there instead of landing a parallel direct-to-main variant.

[swellweb]

Quick note to keep the integration path clear: any follow-up changes for this work should be opened against feature/input-dockstation, not directly against main or a parallel branch.

That branch already contains the current input-dockstation / receiver-master flow, protocol updates, permission handling, cursor/clipboard behavior, and the ongoing fixes needed to avoid regressions. Keeping the work there is the safest way to converge everything before merge.