RFC: AI-forward input drivers for editable-layers (vision, text, audio)

[charlieforward9]

Motivation

Closing #534 made the direction clearer: the most interesting AI work in this repo is not a cross-cutting skills module — it is treating the mode systems we already own as the lowering target for every AI input channel.

editable-layers modes are an input-agnostic state machine. Today the only drivers are mouse and touch. That is under-ambitious for where interfaces are going:

• Vision-channel AI — MediaPipe hand / face / pose models already produce high-level gesture semantics today (see NEW-HEAT/thor.gl: hand tracking → NAV / PICK / SIGNAL channels driving a deck.gl globe). "Fist to delete the selected feature" is a one-line binding away.
• Text-channel AI — LLMs with tool-calls can express structured edits ("split this polygon along the river") that would take dozens of clicks manually.
• Audio-channel AI — speech-to-intent (Whisper + a thin classifier, or realtime multimodal models) collapses the text path further — you talk, the map edits.
• Future channels — eye-gaze targeting, stylus pressure semantics, controller input in XR. All same pattern.

Every one of these channels wants to produce the same thing: a well-typed, validated transition of a FeatureCollection. If each channel hand-rolls its own path, we get four diverging codebases and four diverging bug surfaces. If the mode vocabulary is the stable contract and each channel is a driver lowering into it, we get one surface that every AI modality can feed.

Target demo integration: ilyabo/mapcanv, consuming both a chat-driven tool surface and thor.gl's vision channel against the same editing primitives.

Scope

In scope

• A typed tool-call vocabulary that is the semantic contract for every input driver — one tool per operation, not per mode.
• Drop-in factory returning Vercel-AI-SDK-shaped tools for text-channel use today.
• Reference integration in mapcanv — text driver (chat) + vision driver (thor.gl) both producing the same transitions.
• A compliance test suite asserting direct-geometry tool execution matches mouse-driven mode replay, so drivers cannot silently diverge.

Out of scope

• Audio / speech driver implementation (prove the contract with text + vision first; audio is a driver swap later).
• Non-edit layer construction (see graph-layers RFC and arrow-layers tracker Tracker: arrow-layers module #79).
• Snap backends / spatial index as a general concern.

Proposed API

import { createEditTools } from "@deck.gl-community/editable-layers";

const tools = createEditTools({
  getFeatureCollection: () => features,
  onFeatureCollectionChange: (fc) => setFeatures(fc),
  crs: "EPSG:4326",
});
// tools is a record of Vercel-AI-SDK-shaped tools, droppable into:
//   await streamText({ tools, ... })

Vocabulary — one tool per semantic operation, not per mode:

Tool	Args	Maps to mode
draw_point	{ position: [lng,lat] }	DrawPointMode
draw_line_string	{ vertices: [lng,lat][] }	DrawLineStringMode
draw_polygon	{ vertices: [lng,lat][], holes?: [lng,lat][][] }	DrawPolygonMode
draw_rectangle	{ corner1, corner2 }	DrawRectangleMode
modify_feature	{ featureIndex, edits: VertexEdit[] } where VertexEdit = { op: "move"|"insert"|"delete", vertexIndex, position? }	ModifyMode
translate_feature	{ featureIndex, deltaMeters: [dx,dy] }	TranslateMode
rotate_feature	{ featureIndex, angleDegrees, pivot? }	RotateMode
scale_feature	{ featureIndex, factor, pivot? }	ScaleMode
split_polygon	{ featureIndex, splitLine: [lng,lat][] }	SplitPolygonMode
duplicate_feature	{ featureIndex, offsetMeters? }	—
delete_feature	{ featureIndex }	—

Uniform return shape:

type EditResult =
  | { ok: true; featureIndex: number; featureCollection: FeatureCollection }
  | { ok: false; reason: "self_intersecting" | "out_of_bounds" | "feature_not_found" | "invalid_geometry" | "crs_mismatch" };

Key design choice — direct geometry v. mode replay
v1 executes tools via direct geometry ops (turf / jsts) rather than replaying the mode state machine. Trades perfect fidelity for ~10× simpler code. The compliance test suite closes the gap: for every tool, assert tool(fc, args).featureCollection equals the FC you'd get from driving the matching mode with synthetic mouse events. Drivers cannot silently diverge.

Thor.gl binding — no API changes to editable-layers needed:

thor.on("fist", () => tools.delete_feature.execute({ featureIndex: hoveredIndex }));
thor.on("pinch-pan", ({ dx, dy }) => /* accumulate, debounce, commit as translate_feature */);

SIGNALS map to discrete tools; NAV accumulates into continuous edits with a commit threshold.

Open questions

1. Tool shape — default to Vercel AI SDK ({ description, parameters: Zod, execute }) since most agent runtimes consume it; MCP adapter can come later.
2. History / undo — out of scope for v1; one tool per turn. Multi-turn transactions are a v2 concern.
3. Continuous drivers (gesture drag, stylus) — where does debouncing / commit-threshold logic live? In the driver, or as a helper in this module?
4. CRS handling — accept only WGS84 in v1, or surface a crs field and let tools project?

Backlog to address alongside

Editable-layers items that should be triaged as the surface stabilizes enough to expose to every AI channel:

• [Feat] Snap cursor to edges while editing #586 — Snap cursor to edges while editing
• [Bug] Drawing models not working on overlay mode #570 — Drawing models not working on overlay mode
• PathOutlineLayer / PathMarkerLayer: incomplete v9 port, example non-functional #485 — PathOutlineLayer / PathMarkerLayer incomplete v9 port
• [Bug] Map moves when dragging handles in ModifyMode with MapboxOverlay interleaved #457 — Map moves when dragging handles in ModifyMode with MapboxOverlay interleaved
• [Bug] ModifyMode does not preserve rotation of a rectangle #252 — ModifyMode does not preserve rotation of a rectangle
• [Bug] DrawPolygonByDraggingMode should not allow drawing with middle or right mouse buttons #247 — DrawPolygonByDraggingMode mouse button filtering
• [Bug] Non-standard FeatureCollection type in editable-layers #202 — Non-standard FeatureCollection type in editable-layers

Related

• CC: @ilyabo
• Closed: feat: add @deck.gl-community/visgl-skills — AI agent skills for deck.gl #534 — visgl-skills
• External: ilyabo/mapcanv, NEW-HEAT/thor.gl