Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion electron/mcp/media-tools.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,7 @@ describe("pointerSamples", () => {
});

describe("setPointerTrack", () => {
it("gives a script-driven take a pointer, and a zoom on each click", async () => {
it("gives a script-driven take a pointer, and zooms on its clicks", async () => {
// Driven through the DOM: the real pointer never moved, so nothing to zoom on.
await fs.rm(`${videoPath}.cursor.json`);
const take = await recordTake();
Expand Down
10 changes: 5 additions & 5 deletions electron/mcp/media-tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ const READ_ONLY = { readOnlyHint: true, destructiveHint: false, openWorldHint: f
export const MEDIA_PREAMBLE = [
"OpenScreen records the screen, turns each take into a project you edit with the document tools, and exports it as a video.",
"A demo goes: listSources, then startRecording, then drive the app, then stopRecording. It returns the projectId and a checkpointId. Edit with the document tools, always passing that projectId: every edit is saved to the project straight away, and restoreCheckpoint with the checkpointId takes the project back to the raw take in one step. Then exportVideo, and getFrames with the projectId to check the export before you hand it over: it takes the same ruler seconds as the document tools, so a trim or a speed region never shifts what you look at.",
"Drive the app with the real mouse when you can: OpenScreen records the pointer, draws a clean cursor and zooms on each click. If you drive it from a script instead (Playwright, CDP), the screen pointer never moves: log each action as the script runs it, then hand the log to setPointerTrack after stopRecording.",
"Drive the app with the real mouse when you can: OpenScreen records the pointer, draws a clean cursor and zooms on the clicks as the editor does: clicks close in time and on screen share one zoom, and the first seconds and the end stay on the whole screen. If you drive it from a script instead (Playwright, CDP), the screen pointer never moves: log each action as the script runs it, then hand the log to setPointerTrack after stopRecording.",
"These projects are in the OpenScreen app's project list, so the user can open one in the editor. Do not edit a project while the user has it open there: the editor would keep its own copy.",
"",
].join("\n");
Expand Down Expand Up @@ -214,7 +214,7 @@ export function registerMediaTools(server: McpServer, deps: MediaToolDeps): void
{},
);
if (!seeded.ok || !seeded.document) throw new Error(seeded.resultJson);
// The zooms a recording made in the app starts with: one on each recorded click.
// The zooms a recording made in the app starts with, planned from its clicks (zoom-suggestions.ts).
const suggestions = await collectAutoZoomSuggestionsForDocument(
seeded.document,
deps.readCursor,
Expand Down Expand Up @@ -253,7 +253,7 @@ export function registerMediaTools(server: McpServer, deps: MediaToolDeps): void
"startRecording",
{
description:
"Start recording one window (preferred: nothing else on the desktop is captured) or one display. Returns once capture has started; then drive the app and call stopRecording. The pointer is recorded as data, so the export draws a clean cursor and zooms on the clicks. On Linux the desktop's own picker asks the user which window to record.",
"Start recording one window (preferred: nothing else on the desktop is captured) or one display. Returns once capture has started; then drive the app and call stopRecording. The pointer is recorded as data, so the export draws a clean cursor and the zooms planned from the clicks. On Linux the desktop's own picker asks the user which window to record.",
inputSchema: z.object({
window: z
.string()
Expand Down Expand Up @@ -318,7 +318,7 @@ export function registerMediaTools(server: McpServer, deps: MediaToolDeps): void
"stopRecording",
{
description:
"Stop the recording and turn the take into a new project, laid out as the editor lays out a fresh recording: one clip, plus a zoom on each recorded click. Returns the projectId to edit, a checkpointId that restores the raw take, and the take's videoPath.",
"Stop the recording and turn the take into a new project, laid out as the editor lays out a fresh recording: one clip, plus the zooms planned from the recorded clicks. Not one per click: clicks close in time and on screen share one zoom, a click in the first seconds or the one that stopped the take gets none, and autoZooms says how many there are. Returns the projectId to edit, a checkpointId that restores the raw take, and the take's videoPath.",
inputSchema: z.object({
title: z.string().optional().describe("The project's title in OpenScreen."),
}),
Expand Down Expand Up @@ -388,7 +388,7 @@ export function registerMediaTools(server: McpServer, deps: MediaToolDeps): void
"setPointerTrack",
{
description: [
"For a take driven by a script (Playwright, CDP, a browser extension) rather than the real mouse: the screen pointer never moved, so the video has no cursor and no zoom. Give the pointer's path here and OpenScreen draws it, gliding between your actions, and zooms on its clicks as for a real mouse. It replaces the pointer recorded with the take, then adds a zoom on each new click. Only for a take recorded by this server.",
"For a take driven by a script (Playwright, CDP, a browser extension) rather than the real mouse: the screen pointer never moved, so the video has no cursor and no zoom. Give the pointer's path here and OpenScreen draws it, gliding between your actions, and zooms on its clicks as for a real mouse. It replaces the pointer recorded with the take, then plans zooms on the new clicks as stopRecording does, clear of the zooms already there; autoZoomsAdded says how many. Only for a take recorded by this server.",
"events, one per action: `at` is Date.now() when your script did it (log it as it runs; the take's recordingStartedAt is on the same clock). `x` and `y` are fractions (0 to 1) of the recorded window. For a point (clientX, clientY) of a page shown in that window, from inside the page: border = (outerWidth - innerWidth) / 2; x = (border + clientX) / outerWidth; y = (outerHeight - innerHeight - border + clientY) / outerHeight.",
].join("\n"),
inputSchema: z.object({
Expand Down
4 changes: 2 additions & 2 deletions skills/openscreen-demo/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Write a storyboard and show it to the user in a few lines, then carry on unless
3. `stopRecording` with a `title`. Keep what it returns: `projectId`, `checkpointId`, `videoPath`, `recordingStartedAt`.

**How you drive the app decides the cursor.**
- **Real mouse** (computer-use, or any tool that moves the actual pointer): OpenScreen records the pointer, draws a clean cursor and already zooms on each click. If your mouse tool cannot click in the app's window (some only read browsers, for one), drive it from a script instead.
- **Real mouse** (computer-use, or any tool that moves the actual pointer): OpenScreen records the pointer, draws a clean cursor and already zooms on the clicks (see *Zoom on the action* below). If your mouse tool cannot click in the app's window (some only read browsers, for one), drive it from a script instead.
- **A script** (Playwright, CDP): the screen pointer never moves, so the take has no cursor and no zoom. In the script, log every action as it runs: `at` is `Date.now()`, `x` and `y` are the action's position as fractions of the window, `click` is true for a click. Hand the log to `setPointerTrack` after `stopRecording`; its description gives the formula from a page's coordinates. A script also gives the cleanest take: same timing on every run, no hesitation.

## 5. Check the take
Expand All @@ -80,7 +80,7 @@ Use the document tools, always with the `projectId`. Read the project first with

- **Format first:** `setAspectRatio` for where the video will be shown, `16:9` for X. Do it before placing text.
- **Cut dead time:** the setup at the start, the stop at the end, any wait. Use `addTrims`, or `addSpeed` to fast-forward a wait instead of cutting it.
- **Zoom on the action:** clicks close together share one zoom, and the first 2.5 s stay wide. Check them, then add or move zooms with `addZooms` and `setZoom` where a result deserves its own. Hold a zoom at least 1.5 s.
- **Zoom on the action:** clicks close in time and on screen share one zoom, and the first 2.5 s stay wide. Check them, then add or move zooms with `addZooms` and `setZoom` where a result deserves its own. Hold a zoom at least 1.5 s.
- **Callouts:** one `addAnnotation` per beat, short, placed clear of the action. Text sits on the output frame and stays put when the picture zooms. Use its look for hierarchy: a large title with `animation: "typewriter"`, step callouts with `animation: "pop"`. `setAnnotation` moves or restyles one.
- **Undo:** `restoreCheckpoint` with the `checkpointId` takes the project back to the raw take in one step.

Expand Down
4 changes: 2 additions & 2 deletions technical-documentation/testing/manual-e2e-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -575,7 +575,7 @@ Two servers. The **in-app** one is HTTP, in *AI settings* → *MCP server*, off
- [ ] In *AI settings*, confirm *Enable MCP server* and *Allow MCP clients to edit your projects* are off by default; enable the server and confirm the status reads *Running* with a *Port*, an *Access token* (Show, Copy, Regenerate) and the Claude Code and Codex snippets. Change the port, *Apply*, and confirm a client reconnects on it.
- [ ] With edits not allowed, confirm an edit tool is refused; allow them and confirm the edit lands in the open editor and undoes with `Ctrl/Cmd+Z`.
- [ ] Confirm `"<exe>" help` lists `mcp`, register the stdio server in an MCP client, and confirm `listSources` lists displays, windows and microphones.
- [ ] Run `startRecording` on a window, click inside it a few times, `stopRecording`, and confirm the returned project is in the editor's project list with one clip and a zoom per click. On macOS 15.2+, confirm Apple's picker never appears; on Linux, confirm the portal asks.
- [ ] Run `startRecording` on a window, click inside it a few times, `stopRecording`, and confirm the returned project is in the editor's project list with one clip and the zooms its `autoZooms` count announces. Clicks close in time (3 s or less) and on screen share one zoom, and a click in the first seconds or the stopping click gets none, so there can be fewer zooms than clicks. On macOS 15.2+, confirm Apple's picker never appears; on Linux, confirm the portal asks.
- [ ] Edit that project while it is closed, `restoreCheckpoint` with the returned checkpoint, and confirm the project is back to the raw take. Confirm an edit to a closed project with no checkpoint is refused.
- [ ] Open the project in the editor, then send an MCP edit to it as a closed project, and confirm it is refused rather than overwriting the editor's copy.
- [ ] `exportVideo`, then `getFrames` at a few ruler seconds after a trim, and confirm the frames show the moments asked for; edit the project again and confirm `getFrames` warns that the export is stale.
Expand Down Expand Up @@ -655,7 +655,7 @@ The inspector's rail holds five facets: Composition (background, format, frame,
- [ ] Pick each frame *Style* (Window, Laptop, Phone, Screen) and each *Theme* (Light, Dark), and confirm the preview draws it around the recording; export and confirm the frame is in the file.
- [ ] With a zoom on a 3D camera in the project, confirm *Depth of field* appears under Motion (and is absent without one); turn it on and confirm the tilted zoom changes in the preview and in an export.
- [ ] Open the Camera layout facet and choose each *Preset* (Picture in picture, Dual frame, Vertical stack, No webcam); confirm the preview arrangement changes.
- [ ] Change *Mirror webcam*, *Shrink on zoom*, *Camera shape* (Rectangle, Square) and its roundness, *Webcam size*, and each *Position*; confirm each change is visible.
- [ ] Change *Mirror webcam*, *Shrink on zoom*, *Camera shape* (Square, Original) and its roundness, *Webcam size*, and each *Position*; confirm each change is visible.
- [ ] In *Webcam crop*, drag the frame to move the camera's framing and a corner to zoom it; confirm the preview follows.
- [ ] Open the Audio facet, change *Output level*, and confirm the preview and an export follow it; *Reset audio* brings it back.
- [ ] Open the Cursor facet and toggle *Show cursor* and *Auto-hide when inactive*; confirm the preview changes.
Expand Down
2 changes: 1 addition & 1 deletion website/docs/ai-editing.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,6 @@ codex mcp add openscreen -- "/Applications/Openscreen.app/Contents/MacOS/Openscr
The executable on each system is listed in [Running the CLI](./cli.md#running-the-cli).

- **No port, no token, no setting.** The agent starts the process itself, with your rights, as it would run `openscreen record`. So it can record your screen and edit without the **Project edits** switch.
- **Each take becomes a new project** in your project list, with a zoom on each recorded click, as a recording made in the app. Do not edit it in the editor while the agent is still working on it: the editor keeps its own copy.
- **Each take becomes a new project** in your project list, zoomed on its clicks as a recording made in the app is: clicks close in time and on screen share one zoom, and the first seconds and the end stay on the whole screen. Do not edit it in the editor while the agent is still working on it: the editor keeps its own copy.
- **A take driven by a script** (Playwright, for example) never moves the screen pointer. The agent hands OpenScreen its own log of the actions with `setPointerTrack`, and the video gets a cursor and its zooms all the same.
- **The `openscreen-demo` skill** teaches an agent the whole job, from the storyboard to the finished video. It then attaches the video to a pull request, so reviewers watch the change in place, or drafts a post on X that it publishes only once you say yes. Add it with `npx skills add getopenscreen/openscreen --skill openscreen-demo`. The recipe is [`skills/openscreen-demo/SKILL.md`](https://github.com/getopenscreen/openscreen/blob/main/skills/openscreen-demo/SKILL.md).
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,6 @@ codex mcp add openscreen -- "/Applications/Openscreen.app/Contents/MacOS/Openscr
Die ausführbare Datei für jedes System steht unter [Die CLI ausführen](./cli.md#running-the-cli).

- **Kein Port, kein Token, keine Einstellung.** Der Agent startet den Prozess selbst, mit deinen Rechten, so wie er `openscreen record` ausführen würde. Deshalb kann er deinen Bildschirm aufnehmen und bearbeiten, ohne den Schalter **Project edits**.
- **Jeder Take wird zu einem neuen Projekt** in deiner Projektliste, mit einem Zoom auf jeden aufgezeichneten Klick, wie eine in der App gemachte Aufnahme. Bearbeite es nicht im Editor, solange der Agent noch daran arbeitet: Der Editor hält seine eigene Kopie.
- **Jeder Take wird zu einem neuen Projekt** in deiner Projektliste, mit Zooms auf seine Klicks, wie eine in der App gemachte Aufnahme: Zeitlich und räumlich nah beieinanderliegende Klicks teilen sich einen Zoom, und die ersten Sekunden und das Ende bleiben auf dem ganzen Bildschirm. Bearbeite es nicht im Editor, solange der Agent noch daran arbeitet: Der Editor hält seine eigene Kopie.
- **Ein per Skript gesteuerter Take** (zum Beispiel mit Playwright) bewegt nie den Zeiger auf dem Bildschirm. Der Agent übergibt OpenScreen mit `setPointerTrack` sein eigenes Protokoll der Aktionen, und das Video bekommt trotzdem einen Cursor und dessen Zooms.
- **Der Skill `openscreen-demo`** bringt einem Agenten die ganze Arbeit bei, vom Storyboard bis zum fertigen Video. Danach hängt er das Video an einen Pull Request, damit die Reviewer die Änderung direkt dort sehen, oder entwirft einen Post auf X, den er erst veröffentlicht, wenn du Ja sagst. Füge ihn mit `npx skills add getopenscreen/openscreen --skill openscreen-demo` hinzu. Das Rezept ist [`skills/openscreen-demo/SKILL.md`](https://github.com/getopenscreen/openscreen/blob/main/skills/openscreen-demo/SKILL.md).
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,6 @@ codex mcp add openscreen -- "/Applications/Openscreen.app/Contents/MacOS/Openscr
El ejecutable de cada sistema aparece en [Ejecutar la CLI](./cli.md#running-the-cli).

- **Sin puerto, sin token, sin ajuste.** El agente inicia el proceso él mismo, con tus permisos, como ejecutaría `openscreen record`. Así puede grabar tu pantalla y editar sin el interruptor **Ediciones del proyecto**.
- **Cada toma se convierte en un proyecto nuevo** en tu lista de proyectos, con un zoom en cada clic grabado, como una grabación hecha en la app. No lo edites en el editor mientras el agente siga trabajando en él: el editor conserva su propia copia.
- **Cada toma se convierte en un proyecto nuevo** en tu lista de proyectos, con zooms en sus clics, como una grabación hecha en la app: los clics cercanos en el tiempo y en la pantalla comparten un zoom, y los primeros segundos y el final se quedan en la pantalla completa. No lo edites en el editor mientras el agente siga trabajando en él: el editor conserva su propia copia.
- **Una toma controlada por un script** (Playwright, por ejemplo) nunca mueve el puntero de la pantalla. El agente le pasa a OpenScreen su propio registro de las acciones con `setPointerTrack`, y el video obtiene igualmente un cursor y sus zooms.
- **La skill `openscreen-demo`** le enseña a un agente todo el trabajo, desde el storyboard hasta el video terminado. Después adjunta el video a un pull request, para que los revisores vean el cambio ahí mismo, o redacta un post en X que solo publica cuando dices que sí. Agrégala con `npx skills add getopenscreen/openscreen --skill openscreen-demo`. La receta es [`skills/openscreen-demo/SKILL.md`](https://github.com/getopenscreen/openscreen/blob/main/skills/openscreen-demo/SKILL.md).
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,6 @@ codex mcp add openscreen -- "/Applications/Openscreen.app/Contents/MacOS/Openscr
L'exécutable de chaque système est indiqué dans [Lancer la CLI](./cli.md#running-the-cli).

- **Ni port, ni jeton, ni réglage.** L'agent lance lui-même le processus, avec vos droits, comme il lancerait `openscreen record`. Il peut donc enregistrer votre écran et faire des modifications sans l'interrupteur **Modifications du projet**.
- **Chaque prise devient un nouveau projet** dans votre liste de projets, avec un zoom sur chaque clic enregistré, comme un enregistrement fait dans l'application. Ne le modifiez pas dans l'éditeur tant que l'agent y travaille encore : l'éditeur garde sa propre copie.
- **Chaque prise devient un nouveau projet** dans votre liste de projets, zoomé sur ses clics comme un enregistrement fait dans l'application : des clics rapprochés dans le temps et à l'écran partagent un zoom, et les premières secondes comme la fin restent sur l'écran entier. Ne le modifiez pas dans l'éditeur tant que l'agent y travaille encore : l'éditeur garde sa propre copie.
- **Une prise pilotée par un script** (Playwright, par exemple) ne déplace jamais le pointeur de l'écran. L'agent transmet à OpenScreen son propre journal des actions avec `setPointerTrack`, et la vidéo a malgré tout un curseur et ses zooms.
- **La skill `openscreen-demo`** enseigne à un agent tout le travail, du storyboard jusqu'à la vidéo finie. Il joint ensuite la vidéo à une pull request, pour que les relecteurs voient le changement sur place, ou rédige un post sur X qu'il ne publie qu'une fois que vous avez dit oui. Ajoutez-la avec `npx skills add getopenscreen/openscreen --skill openscreen-demo`. La recette est [`skills/openscreen-demo/SKILL.md`](https://github.com/getopenscreen/openscreen/blob/main/skills/openscreen-demo/SKILL.md).
Loading
Loading