From af2111a070346dccfacd59aa9afb88b274caf9bc Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Wed, 14 Oct 2026 17:37:14 +0200 Subject: [PATCH 1/2] docs(testing): log the macOS e2e pass on main build run 38581408372 Partial pass on macOS 26.5 (M1): MCP section on both servers, #1194, #1180 tools, CLI captions and a scoped app slice under the legacy picker. Also records which process owns the macOS permission prompts and that a launch from Terminal files them under Terminal. --- technical-documentation/testing/manual-e2e-checklist.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/technical-documentation/testing/manual-e2e-checklist.md b/technical-documentation/testing/manual-e2e-checklist.md index 9b212b5e06..69b898d475 100644 --- a/technical-documentation/testing/manual-e2e-checklist.md +++ b/technical-documentation/testing/manual-e2e-checklist.md @@ -31,6 +31,8 @@ Sections and checks marked **post-2.0.0** cover what has landed on `main` since - the OS settings app — needed to change display scaling, which is how DPI checks are run (see AGENTS.md; "the machine is at 100%" is not a reason to skip them), and on macOS to grant permissions; - on macOS the permission prompts, and on Linux the portal's source picker. Neither is an OpenScreen window, and which process owns each is **to confirm on the first run**: the screenshot note names a masked window's process, so ask for that name, then write it here. + On macOS 26 the Screen Recording and Accessibility prompts are windows of `universalAccessAuthWarn` (measured 2026-10-14). It is not an installed app, so `request_access` cannot grant it: the prompt stays masked and the operator answers it. `UserNotificationCenter` resolves and is worth asking for, for the Automation and other system alerts. The save panel shows nothing in screenshots either; typing the base name and pressing Return still saves. + **Apple's source picker (macOS 15.2+) cannot be granted.** It is drawn by Control Centre, which computer-use refuses, so an agent sees the picker and cannot click it: every take started through it waits for a human (rc.8, rc.12 and rc.13 runs). Do not try to grant it. Run the pass on the app's own picker instead, and keep Apple's picker for a short human slice: [Running the pass unattended on macOS](#running-the-pass-unattended-on-macos). **Name them the way the Start menu does, in the system's own language.** The resolver matches installed-app display names, not executables: on a French Windows the shell is `Explorateur de fichiers` and `explorer.exe` fails outright — `notInstalled`, with a nonsense suggestion attached — which then voids every other name in the same call. On an English install it is `File Explorer`. When unsure, ask rather than guess; the tool lists the installed names it knows. @@ -85,7 +87,7 @@ gh run download -R getopenscreen/openscreen -n openscreen-windows-x64 - The DMG is signed with the Developer ID, notarized and stapled only when the run has all six Apple secrets (`MAC_CERTIFICATE_P12`, `MAC_CERTIFICATE_PASSWORD`, `MAC_CSC_NAME`, `APPLE_ID`, `APPLE_TEAM_ID`, `APPLE_APP_SPECIFIC_PASSWORD`). Without them the `.app` is ad-hoc signed and the DMG is neither signed nor notarized; the job's *Notarize DMG* step then shows as skipped. On the Mac, `spctl -a -vv -t install ` answers `accepted` and `source=Notarized Developer ID` for a notarized DMG. - Gatekeeper refuses an unnotarized DMG that a browser downloaded. Clearing its quarantine attribute is a security override the Mac's owner decides on, not the agent. macOS ties permission grants to the code signature, and an ad-hoc signature changes with every build, so grants made to one candidate may not carry to the next. - Open the DMG and drag Openscreen into Applications. -- On macOS 26 and later no flag is needed: launch it as a user would. On macOS 13 to 15 the flag takes a launch from Terminal, `OPENSCREEN_DISABLE_CONTENT_PROTECTION=1 /Applications/Openscreen.app/Contents/MacOS/Openscreen`. Whether macOS then files the permission prompts under Terminal rather than OpenScreen is **to confirm on the first run**. +- On macOS 26 and later no flag is needed: launch it as a user would. On macOS 13 to 15 the flag takes a launch from Terminal, `OPENSCREEN_DISABLE_CONTENT_PROTECTION=1 /Applications/Openscreen.app/Contents/MacOS/Openscreen`. Started that way, macOS files the permission prompts under **Terminal**, not OpenScreen (2026-10-14, macOS 26.5): with OpenScreen allowed under *Screen & System Audio Recording* and Terminal not, the legacy launch still raised the Screen Recording prompt. Launch with `open -a /Applications/Openscreen.app --env OPENSCREEN_DISABLE_CONTENT_PROTECTION=1 --env OPENSCREEN_DIAGNOSTIC=1 --stdout --stderr ` instead (add `--env OPENSCREEN_MAC_SOURCE_PICKER=legacy` for the unattended run): OpenScreen stays its own responsible process, the env reaches only it, and the main log lands in the file. While an `Openscreen mcp` stdio server runs, `open` says the app is already running and starts nothing; add `-n`. - Grant `Openscreen`, plus the picker and prompt processes from step 4. **Linux.** @@ -960,3 +962,4 @@ These checks are the human slice: an agent cannot click Apple's picker (see abov | 2026-10-13 | `release/v2.1.0` @ `912b44800` (fast-forwarded to `main`: #1144, #1145, #1150, #1152, #1153, #1154, #1156, #1161, #1163, #1165), CI artifact `openscreen-windows-x64` from build run 38370716184 (NSIS sha512 matched `update-info-win-x64.json`; the run's macOS x64 job failed once on a notarization-staple network error, `NSURLErrorDomain -1005`, and passed on re-run), silent per-user install, About still `2.0.0` (intermediate build) | Windows 11 Home 26200 (x64), Ryzen 7 5800X, RTX 4070 Ti, 1920×1080 @ 100 %; Logitech StreamCam | **Pass — scoped re-run of the fixes from the 2026-10-12 row, 1 new minor** | Real OS input through computer-use, measurements on the files. **#945 / #1165:** 10:57 full-screen take, StreamCam 1080p30 + system audio, while 44 CLI exports ran back to back: screen 657.535 s, audio 657.536 s, webcam 657.537 s (2026-10-12: screen −187 ms and −221 ms); HUD-timer flips filmed at 21, 301, 451, 601 and 651 s all sit at +0.033 to +0.050 s, so the screen holds the wall clock to one frame over the whole take (2026-10-12: −217 ms after 5:00); TTS markers drift 0 ms over 567 s. **#1163:** CLI export of the two-clip project at 60 fps gave 4886 frames against 4886.8 predicted from the clips' real video length (48.765 s each); the progress total said 4890, the residual the PR documents. **#1154:** with the project open in the editor, stdio `listProjects` reports it `open: true`, `createCheckpoint` + `addTrim` is refused ("open in the OpenScreen editor …") and the file is untouched; `.open-in-editor.json` carries the GUI's pid. **#1145:** an installed `Openscreen.exe mcp` locks its own `cli-sessions\-\Local Storage`, never the GUI profile, with no cache error, and dead runs' folders are swept on the next run. The GUI-side symptom (language prompt on every launch) could not be re-tested: ten older dev `electron … mcp` servers from another worktree, built before the fix, still hold `%APPDATA%\openscreen\Local Storage\leveldb\LOCK`. **#1150/#1153:** `listSources`, `startRecording`'s not-found error and `stopRecording` now say what is not listed and how clicks share zooms; the not-found error still carries a JavaScript stack trace. **#1152:** a project's first chat message shows `11:16:44`, like the reply. **#1156:** Enter in the Open project search opens the highlighted single match. **#1161:** HUD dropped at y≈200, the language menu and Device settings each push the bar down while open and it returns to y≈200 once they close. **#1144:** two identical 1.5× regions, the second dragged into the first, stop flush at 25.173 s with the length kept (4.983 s). **New, minor:** adding a region with its key while the playhead is inside an existing region of the same kind creates an overlapping one, whatever the values (2× 20.190–25.173 then 1.5× 23.253–28.236), the creation counterpart of #1133. Not re-run: microphone (no working input on this machine), everything outside the fixed points. | | 2026-10-14 | `main` @ `b668b2c8d` (MCP unification #1174: #1175, #1176, #1179; editor parity #1180: #1181 to #1186), CI artifact `openscreen-windows-x64` from build run 38538195250 (NSIS sha512 matched `update-info-win-x64.json`), silent per-user install, About still `2.0.0` (intermediate build) | Windows 11 Home 26200 (x64), Ryzen 7 5800X, RTX 4070 Ti, 1920×1080 @ 100 %; Logitech StreamCam | **Partial — MCP server section only, 2 defects** | Real OS input through computer-use; MCP calls from an SDK client (`@modelcontextprotocol/sdk`) to the installed `Openscreen.exe mcp`, with the app running unless stated. **Tools:** `listTools` lists all 51 tools (record, export, frames, settings, captions, transcription, clips, projects, media). **Record:** `startRecording` on a Notepad window with `camera: "StreamCam"` and system audio, then real clicks and `stopRecording`, gives an 890×596 60 fps take with a webcam track, a cursor sidecar and 1 zoom for 3 close clicks. `cursor: "system"` gives no zoom and the session says `system`. A take dies with its stdio server: a client that disconnects after `startRecording` leaves an 83-byte MP4 and an empty webcam file. **Closed project, app running:** `restoreCheckpoint` with the checkpoint from another client's `stopRecording` works, because it lives in the app. **Media, transcription, captions:** `importMedia` of a spoken 17.2 s MP4, then `insertClip`. `transcribe` finds 40 English words in about 15 s, and `getCaptionTranslationUnits fr` returns 6 units. `putCaptionTranslation` writes all 6, and `setCaptionSettings {enabled, style: bold, language: fr}` shows them. **Look:** `setEditorSettings` applies a laptop frame, a wallpaper, 16:9 and a camera blur cut-out, and turns format fill on as the editor does. The `inactive` reasons are read back. Also tested: `addAnnotation` for a 72 px text and a mosaic blur. **Export:** `exportVideo` at `fps: 30` gives 1920×1080 at 30 fps, 1472 frames for 49.07 s. The frames at 5 and 35 s show the blur, the webcam cut-out top-right, the title and the French captions. `gifLoop` on an `.mp4` is refused. **Open in the editor:** `listProjects` reports `open: true`. `addZoom` and a wallpaper change land live, and each comes off with one Ctrl+Z. `restoreCheckpoint` resets the take live, and one Ctrl+Z brings every edit back. **App quit and relaunch:** one client session spans an app quit and a relaunch. With the app gone, an edit asks for a checkpoint and lands in the file after `createCheckpoint`. Once the app is relaunched and the project reopened, the same session's edit shows live, with no client restart. **Other tools:** `setProjectInfo`, `duplicateClip`, a clip `cropRegion`, `addAutoZooms` (0 new, telemetry reported per asset) and `importMedia` of an MP3 work. `removeMedia` refuses an asset still on the timeline and names its clips, and `createProject` from a video returns a checkpoint. **CLI:** `Openscreen.exe captions --max-words 5` writes a 40-word transcript, turns captions on, and leaves no `auto-caption` annotation. **Defect 1 (P1): every GIF export over MCP fails.** It fails with the app running and without it: `exportVideo` to `.gif` → "--quality is only supported for MP4 exports", because `electron/mcp/media-tools.ts` always sends `quality: "good"` and the export runner refuses a quality on a GIF. **Defect 2 (P2): the recording bar loses always-on-top** after the source picker or the language prompt closes. Its window style goes from `0x280028` to `0x200000` and it sinks behind other windows; Ctrl+Shift+O brings it forward, still not topmost. **Minor:** `camera: true` picks the first listed camera, a virtual "VCam Camera" here, which wrote a 0-byte webcam file that the take dropped without a word. The language prompt shows on every launch, the #1145 symptom: ten older dev `mcp` servers from another worktree still hold the profile. The first two MCP takes had silent system audio (−91 dB) although the narration played; a take from the bar and three later MCP takes, with and without StreamCam, all had it (−3 dB), so this is not attributed. Not run: the in-app HTTP server, because copying its access token is refused to the agent; the microphone, with no working input on this machine; everything outside the MCP section. | | 2026-10-14 | `main` @ `5bc559198` (adds #1192 GIF export over MCP, #1194 recording bar kept on top, #1197 GIF plays once), CI artifact `openscreen-windows-x64` from build run 38564622152 (NSIS sha512 matched `update-info-win-x64.json`), silent per-user install, About still `2.0.0` (intermediate build) | Windows 11 Home 26200 (x64), Ryzen 7 5800X, RTX 4070 Ti, 1920×1080 @ 100 %; Logitech StreamCam | **Pass — MCP server section re-run, both earlier defects fixed** | Same scope and method as the earlier row of this day (run 38538195250). **#1194:** the recording bar keeps `WS_EX_TOPMOST` (`0x280028`) after the language prompt and after the source picker, and stays above a Notepad window brought to the front. **#1192 / #1197:** `exportVideo` to a `.gif` with `gifSize: "small"` gives 852×480 at 15 fps; with `gifLoop: false` the file carries no NETSCAPE block, so it plays once, and with looping on its count is 0. **Record:** `startRecording` on a Notepad window with `camera: "StreamCam"` and system audio gives a 40.6 s 1166×888 60 fps take with a webcam file, system audio at −3.6 dB and 1 zoom. **Transcript and captions:** `transcribe` finds 18 English words, and `putCaptionTranslation` writes 3 French units. **Look and annotations:** `setCaptionSettings` shows the French captions in the `bold` style. `setEditorSettings` applies a laptop frame, 16:9 and a camera blur cut-out, and `addAnnotation` adds a 72 px title and a mosaic blur. **MP4 export:** 1920×1080 at 30 fps, 1219 frames for 40.63 s; the 4 s frame shows the title, the French caption, the blur and the cut-out webcam. **Open project:** `listProjects` reports `open: true`. A zoom and a wallpaper change land live, and each comes off with one Ctrl+Z. **One client session across an app quit and relaunch:** with the app gone, an edit asks for a checkpoint and lands in the file after `createCheckpoint`. Back in the reopened editor, the next edit shows live. **Media, clips, checkpoints:** `importMedia` of an MP4 and an MP3, `insertClip`, `addAudio` with a fade, `duplicateClip`, a clip `cropRegion` and `setProjectInfo` work. `removeMedia` refuses an asset still on the timeline. `restoreCheckpoint` on the open project resets it, and one Ctrl+Z brings every edit back. **Without the app:** `createProject` from a video returns a checkpoint, and `Openscreen.exe captions` writes a 40-word transcript with captions on and no `auto-caption` annotation. Not run: the in-app HTTP server (copying its access token is refused to the agent), the microphone (no working input on this machine), everything outside the MCP section. | +| 2026-10-14 | `main` @ `0bed36852` (adds #1192, #1194, #1197 over the 2026-10-14 Windows rows; c541fffaa, the recording-state fix, landed after this build and is not in it), CI artifact `openscreen-mac-arm64` from build run 38581408372, DMG `Openscreen-macOS-Apple-Silicon-2.0.0.dmg` signed and notarized (`spctl`: accepted, Notarized Developer ID, M4LK7C6S84), installed over `/Applications/Openscreen.app`; About reports `2.0.0 (dmg)`, Electron 41.2.1, Chromium 146.0.7680.188, Node 24.14.1, darwin arm64 | macOS 26.5 (25F71), Mac mini M1 (arm64), one 3840×2160 display at 2× (UI 1920×1080), no camera (Scaleway MDM `allowCamera=false`), BlackHole 2ch as the only input | **Partial, 4 defects; MCP section and #1194 first, rest scoped** | Real OS mouse and keyboard through computer-use; MCP calls from `@modelcontextprotocol/sdk` 1.x (stdio to `Openscreen mcp`, StreamableHTTP to `127.0.0.1:/mcp` with the token copied from AI settings); numbers are ffprobe/ffmpeg on the files. **Pickers:** the first launch (from Terminal, no `OPENSCREEN_MAC_SOURCE_PICKER`) only served the permissions window; every take used the legacy selector (`open --env OPENSCREEN_MAC_SOURCE_PICKER=legacy`) or the MCP servers, none went through Apple's picker. **#1194:** `keepOnTop` is a no-op off Windows, so the bar stays at window level 3 (floating), under the Dock (20) and the menu bar (24); it stays above a frontmost Finder window after the source selector, Device settings and the language menu, keeps its tooltips and clicks, stops under the menu bar and above the Dock when dragged, is pushed down by the language menu near the top and comes back (#1161), and does not float over another app's full-screen Space, as `macSpaces.ts` documents. **MCP stdio:** `help` lists `mcp`; `listTools` 51 tools; `listSources` displays, windows, both BlackHole inputs, no camera; window take 1840×872 with no Apple picker, 4 clicks 5 s apart gave 1 zoom (the no-yo-yo rule of `resolveShots`, not a defect); edit to a closed project, `restoreCheckpoint` to the raw take, edit without checkpoint refused; `exportVideo` 30 fps after a 6 s trim, 1085 frames for 36.17 s as predicted; `getFrames` maps ruler 27 s to file 21 s and warns once the project changed; `cursor: "system"` take has the system pointer and 0 zooms; `camera: true` refused (MDM); `setPointerTrack` 391 samples, 1 zoom, cursor drawn in the export. **#1174:** in-app server off, project open: `listProjects` `open: true`, a stdio edit shows live and one Cmd+Z takes it back, an editor change keeps it, `restoreCheckpoint` lands live and one Cmd+Z restores every edit; app quit: the edit asks for a checkpoint, then lands in the file after `createCheckpoint`; relaunch and reopen: the same client session edits live again. **In-app server:** off by default with *MCP edits* off; *Running* with port, token (Show, Copy, Regenerate) and snippets; edit, `startRecording` and `exportVideo` refused naming *MCP edits*; once allowed, an edit lands live and undoes with Cmd+Z; port 47821 → 47822 then *Apply*: the old URL fails, a client reconnects on the new one. **#1180 tools, both servers:** `createProject`, `setProjectInfo`, `transcribe` (35 English words, about 10 s), `getCaptions`, `setCaptionSettings` (bold), `getCaptionTranslationUnits fr` (4 units), `putCaptionTranslation` with a French translation written by hand and shown, `setEditorSettings` (laptop or window frame, wallpaper, 16:9, `webcamBackgroundMode: transparent` stored with the reason "No clip on the timeline has a camera"), `addAnnotation` (72 px title, mosaic and smooth blurs), `importMedia` (MP4 and M4A), `insertClip`, `duplicateClip`, `setClipRange` `cropRegion` (cropped frame in the export), `removeMedia` refused while on the timeline and naming the clips, then removing an unused asset, `addAutoZooms` (0 added, telemetry per asset). **Exports:** MP4 at `fps: 30`, 1920×1080, 721 frames for 24.03 s, and 1280×720, 1687 frames for 56.23 s, both as predicted; GIF `gifSize: "small"`, `gifLoop: false`: 852×480, 15 fps, no NETSCAPE2.0 block (#1197); `gifLoop` on an `.mp4` refused; MP4 opened in QuickTime and GIF in Preview, judged by their windows only (the viewers' grant was refused). **CLI:** `Openscreen captions ` transcribes on Metal (11.7 s in 8.4 s), turns captions on, no `auto-caption` annotation. **App, legacy picker:** permissions window lists the 15.2+ rows (System audio, Accessibility, Microphone, Camera), then Screen & system audio (Required) first under `legacy`; Device settings lists the inputs and "No camera found" with no quality row; camera toggle refused with "Camera not found."; Record mode rows (Recordings folder last, Auto-zoom only with Editable cursor), *Start recording* hands over to the bar with the same settings; pause stops the timer and resume restarts it; stop opens Edit with 3 zooms; full-screen take 3840×2160 edge to edge, bar and desktop icons absent, system audio −12.6 dB; zoom-in and zoom-out trail tooltips; Glass Lens in preview and in a 720p30 export (1234 frames, true peak −1.6 dBTP, −18.5 LUFS on the clipped fixture voice); *Mouse clicks* off by default, *Click volume* once on, a click at 34.76 s in silence; export dialog (formats, sizes, rates, no codec), progress, *Saved to*, *Show in folder*; Cmd+O lists MCP projects; #997 holds the last clip's last frame at 0:24.0; app menu and menu bar item menus; About; Save Diagnostics writes a bundle. **Defects. (1) P1, MCP in-app `startRecording` (same as Windows, fixed after this build by c541fffaa):** the bar records within 5 s but the call returns after 120 s with "capture had not started", and on macOS the take is then lost: `restore()` ends it, no file is left, no editor opens, and `stopRecording` says "No recording is running." **(2) P2, launch blocked by the stdio server:** while an `Openscreen mcp` client session runs, opening OpenScreen from Finder or `open` starts no GUI (exit 0, no window); LaunchServices activates the `mcp` process instead, which turns `Foreground` with its own Dock icon after its first take. `open -n` works around it. **(3) P3, `getFrames` cannot read a GIF:** with `path` to a GIF, or with `projectId` after a GIF export, both servers answer "Chromium could not decode the video (media error 4)", so the documented check after `exportVideo` fails for GIFs. **(4) P3, system audio on a window take:** an MCP window take with `systemAudio: true` is silent (−91 dB) while sound plays from another app, because `SCContentFilter(desktopIndependentWindow:)` keeps only the window's own app; a display take of the same sound reads −10.5 dB. Windows records the whole mix. **Minor:** the `camera: true` error still carries a JavaScript stack trace; the stop click gets a click sound at the end of the export (40.9 s) although the zoom planner discards it. **Skipped:** Apple's picker slice and the permissions window's prompt rows (operator chose the unattended legacy run; Screen Recording already granted); the system-language prompt (system in English, so it never shows); Spaces switch (one Space); every camera and Full Camera check (MDM); in-app `startRecording`/`stopRecording` end to end, its zooms, settings restore and the unsaved-changes refusal (blocked by defect 1, the bar's settings did come back after the timeout); microphone audio (no source plays into BlackHole); Windows, Linux, macOS 13/14, Intel; not run for time: chat, names and terms, transcript editing, clip anchoring, audio tracks, voice-over, caption placement, persistence, editor bounds, GIF timing against v2.0.0, export cancel. | From 58c874672a9cd1b56f3dcd015a064ba437e0e3b1 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Wed, 14 Oct 2026 17:53:43 +0200 Subject: [PATCH 2/2] docs(testing): say what a UserNotificationCenter grant does and does not give --- technical-documentation/testing/manual-e2e-checklist.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/technical-documentation/testing/manual-e2e-checklist.md b/technical-documentation/testing/manual-e2e-checklist.md index 3f82f1987e..c10217118a 100644 --- a/technical-documentation/testing/manual-e2e-checklist.md +++ b/technical-documentation/testing/manual-e2e-checklist.md @@ -31,7 +31,7 @@ Sections and checks marked **post-2.0.0** cover what has landed on `main` since - the OS settings app — needed to change display scaling, which is how DPI checks are run (see AGENTS.md; "the machine is at 100%" is not a reason to skip them), and on macOS to grant permissions; - on macOS the permission prompts, and on Linux the portal's source picker. Neither is an OpenScreen window, and which process owns each is **to confirm on the first run**: the screenshot note names a masked window's process, so ask for that name, then write it here. - On macOS 26 the Screen Recording and Accessibility prompts are windows of `universalAccessAuthWarn` (measured 2026-10-14). It is not an installed app, so `request_access` cannot grant it: the prompt stays masked and the operator answers it. `UserNotificationCenter` resolves and is worth asking for, for the Automation and other system alerts. The save panel shows nothing in screenshots either; typing the base name and pressing Return still saves. + On macOS 26 the Screen Recording and Accessibility prompts are windows of `universalAccessAuthWarn` (measured 2026-10-14). It is not an installed app, so `request_access` cannot grant it: the prompt stays masked and the operator answers it. `UserNotificationCenter` is an installed app that `request_access` does resolve: granting it unmasks the system alerts it draws, an Automation consent dialog among them, so the agent can see what is blocking its clicks. That grant is computer-use access to those windows, not the Automation (Apple Events) permission itself, which stays the operator's answer to the prompt. The save panel shows nothing in screenshots either; typing the base name and pressing Return still saves. **Apple's source picker (macOS 15.2+) cannot be granted.** It is drawn by Control Centre, which computer-use refuses, so an agent sees the picker and cannot click it: every take started through it waits for a human (rc.8, rc.12 and rc.13 runs). Do not try to grant it. Run the pass on the app's own picker instead, and keep Apple's picker for a short human slice: [Running the pass unattended on macOS](#running-the-pass-unattended-on-macos).