What
Give agents "eyes": a way to visually inspect footage and the current timeline state, extending the analyze pipeline that already provides the "ears".
From the Show HN thread (spottedmarley): "clever use of vision and maybe waveform analysis to give the model eyes and ears". v1.3's fablecut_analyze_reference shipped the ears (shots, beats, BPM, energy curve, drop detection). This issue is the other half.
Ideas (scope to be cut down)
GET /api/frame?src=/media/x.mp4&t=3.2 — server extracts a single frame via ffmpeg, returns JPEG. An MCP tool wraps it so a multimodal agent can look at any moment of any source clip before choosing in points.- Same for the composed timeline: render the program monitor at time t (harder — the compositor lives in the browser; could round-trip through the open UI like fast export does).
- Contact-sheet mode: one call returns an N-up grid of frames across a clip, cheap way for an agent to survey footage.
The first bullet alone is probably 30 lines of server code + one MCP tool and would already let agents pick shots by looking instead of guessing.
Done when
An agent can request a frame from any registered media at any timestamp through MCP and receives an image it can reason about.
What
Give agents "eyes": a way to visually inspect footage and the current timeline state, extending the analyze pipeline that already provides the "ears".
From the Show HN thread (spottedmarley): "clever use of vision and maybe waveform analysis to give the model eyes and ears". v1.3's
fablecut_analyze_referenceshipped the ears (shots, beats, BPM, energy curve, drop detection). This issue is the other half.Ideas (scope to be cut down)
GET /api/frame?src=/media/x.mp4&t=3.2— server extracts a single frame via ffmpeg, returns JPEG. An MCP tool wraps it so a multimodal agent can look at any moment of any source clip before choosinginpoints.The first bullet alone is probably 30 lines of server code + one MCP tool and would already let agents pick shots by looking instead of guessing.
Done when
An agent can request a frame from any registered media at any timestamp through MCP and receives an image it can reason about.