Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion technical-documentation/testing/manual-e2e-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ Sections and checks marked **post-2.0.0** cover what has landed on `main` since
- the OS settings app — needed to change display scaling, which is how DPI checks are run (see AGENTS.md; "the machine is at 100%" is not a reason to skip them), and on macOS to grant permissions;
- on macOS the permission prompts, and on Linux the portal's source picker. Neither is an OpenScreen window, and which process owns each is **to confirm on the first run**: the screenshot note names a masked window's process, so ask for that name, then write it here.

On macOS 26 the Screen Recording and Accessibility prompts are windows of `universalAccessAuthWarn` (measured 2026-10-14). It is not an installed app, so `request_access` cannot grant it: the prompt stays masked and the operator answers it. `UserNotificationCenter` is an installed app that `request_access` does resolve: granting it unmasks the system alerts it draws, an Automation consent dialog among them, so the agent can see what is blocking its clicks. That grant is computer-use access to those windows, not the Automation (Apple Events) permission itself, which stays the operator's answer to the prompt. The save panel shows nothing in screenshots either; typing the base name and pressing Return still saves.

**Apple's source picker (macOS 15.2+) cannot be granted.** It is drawn by Control Centre, which computer-use refuses, so an agent sees the picker and cannot click it: every take started through it waits for a human (rc.8, rc.12 and rc.13 runs). Do not try to grant it. Run the pass on the app's own picker instead, and keep Apple's picker for a short human slice: [Running the pass unattended on macOS](#running-the-pass-unattended-on-macos).

**Name them the way the Start menu does, in the system's own language.** The resolver matches installed-app display names, not executables: on a French Windows the shell is `Explorateur de fichiers` and `explorer.exe` fails outright — `notInstalled`, with a nonsense suggestion attached — which then voids every other name in the same call. On an English install it is `File Explorer`. When unsure, ask rather than guess; the tool lists the installed names it knows.
Expand Down Expand Up @@ -85,7 +87,7 @@ gh run download <run-id> -R getopenscreen/openscreen -n openscreen-windows-x64
- The DMG is signed with the Developer ID, notarized and stapled only when the run has all six Apple secrets (`MAC_CERTIFICATE_P12`, `MAC_CERTIFICATE_PASSWORD`, `MAC_CSC_NAME`, `APPLE_ID`, `APPLE_TEAM_ID`, `APPLE_APP_SPECIFIC_PASSWORD`). Without them the `.app` is ad-hoc signed and the DMG is neither signed nor notarized; the job's *Notarize DMG* step then shows as skipped. On the Mac, `spctl -a -vv -t install <dmg>` answers `accepted` and `source=Notarized Developer ID` for a notarized DMG.
- Gatekeeper refuses an unnotarized DMG that a browser downloaded. Clearing its quarantine attribute is a security override the Mac's owner decides on, not the agent. macOS ties permission grants to the code signature, and an ad-hoc signature changes with every build, so grants made to one candidate may not carry to the next.
- Open the DMG and drag Openscreen into Applications.
- On macOS 26 and later no flag is needed: launch it as a user would. On macOS 13 to 15 the flag takes a launch from Terminal, `OPENSCREEN_DISABLE_CONTENT_PROTECTION=1 /Applications/Openscreen.app/Contents/MacOS/Openscreen`. Whether macOS then files the permission prompts under Terminal rather than OpenScreen is **to confirm on the first run**.
- On macOS 26 and later no flag is needed: launch it as a user would. On macOS 13 to 15 the flag takes a launch from Terminal, `OPENSCREEN_DISABLE_CONTENT_PROTECTION=1 /Applications/Openscreen.app/Contents/MacOS/Openscreen`. Started that way, macOS files the permission prompts under **Terminal**, not OpenScreen (2026-10-14, macOS 26.5): with OpenScreen allowed under *Screen & System Audio Recording* and Terminal not, the legacy launch still raised the Screen Recording prompt. Launch with `open -a /Applications/Openscreen.app --env OPENSCREEN_DISABLE_CONTENT_PROTECTION=1 --env OPENSCREEN_DIAGNOSTIC=1 --stdout <log> --stderr <log>` instead (add `--env OPENSCREEN_MAC_SOURCE_PICKER=legacy` for the unattended run): OpenScreen stays its own responsible process, the env reaches only it, and the main log lands in the file. While an `Openscreen mcp` stdio server runs, `open` says the app is already running and starts nothing; add `-n`.
- Grant `Openscreen`, plus the picker and prompt processes from step 4.

**Linux.**
Expand Down Expand Up @@ -962,3 +964,4 @@ These checks are the human slice: an agent cannot click Apple's picker (see abov
| 2026-10-14 | `main` @ `5bc559198` (adds #1192 GIF export over MCP, #1194 recording bar kept on top, #1197 GIF plays once), CI artifact `openscreen-windows-x64` from build run 38564622152 (NSIS sha512 matched `update-info-win-x64.json`), silent per-user install, About still `2.0.0` (intermediate build) | Windows 11 Home 26200 (x64), Ryzen 7 5800X, RTX 4070 Ti, 1920×1080 @ 100 %; Logitech StreamCam | **Pass — MCP server section re-run, both earlier defects fixed** | Same scope and method as the earlier row of this day (run 38538195250). **#1194:** the recording bar keeps `WS_EX_TOPMOST` (`0x280028`) after the language prompt and after the source picker, and stays above a Notepad window brought to the front. **#1192 / #1197:** `exportVideo` to a `.gif` with `gifSize: "small"` gives 852×480 at 15 fps; with `gifLoop: false` the file carries no NETSCAPE block, so it plays once, and with looping on its count is 0. **Record:** `startRecording` on a Notepad window with `camera: "StreamCam"` and system audio gives a 40.6 s 1166×888 60 fps take with a webcam file, system audio at −3.6 dB and 1 zoom. **Transcript and captions:** `transcribe` finds 18 English words, and `putCaptionTranslation` writes 3 French units. **Look and annotations:** `setCaptionSettings` shows the French captions in the `bold` style. `setEditorSettings` applies a laptop frame, 16:9 and a camera blur cut-out, and `addAnnotation` adds a 72 px title and a mosaic blur. **MP4 export:** 1920×1080 at 30 fps, 1219 frames for 40.63 s; the 4 s frame shows the title, the French caption, the blur and the cut-out webcam. **Open project:** `listProjects` reports `open: true`. A zoom and a wallpaper change land live, and each comes off with one Ctrl+Z. **One client session across an app quit and relaunch:** with the app gone, an edit asks for a checkpoint and lands in the file after `createCheckpoint`. Back in the reopened editor, the next edit shows live. **Media, clips, checkpoints:** `importMedia` of an MP4 and an MP3, `insertClip`, `addAudio` with a fade, `duplicateClip`, a clip `cropRegion` and `setProjectInfo` work. `removeMedia` refuses an asset still on the timeline. `restoreCheckpoint` on the open project resets it, and one Ctrl+Z brings every edit back. **Without the app:** `createProject` from a video returns a checkpoint, and `Openscreen.exe captions` writes a 40-word transcript with captions on and no `auto-caption` annotation. Not run: the in-app HTTP server (copying its access token is refused to the agent), the microphone (no working input on this machine), everything outside the MCP section. |
| 2026-10-14 | `main` @ `5bc559198`, CI artifact `openscreen-windows-x64` from build run 38564622152; then `fix/mcp-native-take-start` @ `bdfa71641` (#1205, now merged), build run 38582471723. NSIS sha512 matched `update-info-win-x64.json` for both, silent per-user install, About still `2.0.0` (intermediate build) | Windows 11 Home 26200 (x64), Ryzen 7 5800X, RTX 4070 Ti, 1920×1080 @ 100 %; Logitech StreamCam | **Pass after #1205 — in-app MCP server; 1 defect found and fixed** | Real OS input through computer-use. Calls come from an SDK client over `StreamableHTTPClientTransport` to `http://127.0.0.1:<port>/mcp`, with the Bearer token copied from *AI settings* and read from the clipboard (the server is ephemeral in dev). **Gate:** with *MCP edits* off, `addZoom` returns `consent_required`, and its guidance names *MCP edits*; `startRecording` and `exportVideo` are refused, also naming it. **Live edit:** with *MCP edits* on, an `addSpeed` on the open project lands live and one Ctrl+Z takes it back. **Export:** `exportVideo` at `fps: 30`, `quality: "medium"` gives 900×720 at 30 fps. To a `.gif` with `gifSize: "small"` it gives 600×480 at 15 fps, and `getFrames` returns the asked frame. A relative path is refused ("The path must be absolute"). **Port:** set to 47822 and applied, the old port refuses connections, the new one serves the same token, and the Claude Code snippet shows the new port. **Defect (P1, #1204):** on `main` @ `5bc559198`, `startRecording` on a window closes the editor and the recording bar records after its countdown, with only the app's `wgc-capture.exe` running, but the call waits 120 s and answers "capture had not started". `stopRecording` then finds no take, and the bar keeps recording until stopped by hand. Cause: the native capture paths never told the MCP recorder a take had started. **After #1205:** `startRecording` answers `recording: true` in 2 s, and `stopRecording` returns the 29.2 s take with its webcam and 1 zoom, open in the editor under the requested title. The bar's recording settings come back to what they were before the take. Not run: *Enable MCP server* and *MCP edits* off by default, because this profile had already enabled the server; the refusal while the editor has unsaved changes; the microphone, with no working input on this machine. |
| 2026-10-14 | `main` @ `5bc559198`, CI artifact `openscreen-linux` from build run 38564622152. The `Openscreen-Linux-2.0.0.AppImage` sha512 matched `latest-linux.yml`; it was extracted (`--appimage-extract`), because WSL has no libfuse2 and no passwordless sudo for the `.deb`. About not read, since the GUI was not driven. | WSL2 Ubuntu 24.04 (kernel 5.15.146.1) under WSLg on Windows 11 Home 26200, Ryzen 7 5800X, RTX 4070 Ti. Chromium's GPU process fails to start under WSLg and falls back; Mesa/ZINK warnings. | **Partial — headless only (MCP stdio + CLI), 1 defect** | WSLg draws the Linux windows through `msrdc.exe`. Granting it to computer-use was declined, so nothing in the GUI was driven: no HUD, editor, tray or settings. The app was launched under WSLg (`--no-sandbox`) while an SDK MCP client drove `openscreen mcp` with `ELECTRON_DISABLE_SANDBOX=1`. The CLI rejects `--no-sandbox` after a subcommand, and the SDK's stdio transport passes no `DISPLAY` unless given the full environment. **Works:** - `listSources` lists one display, no camera (WSL has none) and the `RDP Source` microphone; - `createProject` from a 17.2 s video; - `setEditorSettings` with a laptop frame, a wallpaper and 16:9; - `setCaptionSettings`; - `addAnnotation` with a 72 px title and a mosaic blur; - `addZoom`; - `exportVideo` at 30 fps: 1920×1080, 516 frames for 17.2 s, and the 4 s frame shows frame, title and blur; - a `.gif` with `gifSize: "small"` and `gifLoop: false`: 852×480 at 15 fps with no NETSCAPE block; - CLI `export --fps 24`: 413 frames for 17.21 s. **Defect (P1, Linux): transcription fails over MCP and the CLI.** The Linux package ships the ffmpeg libraries (`helper-ffmpeg/`) but no ffmpeg executable. So `transcribe` returns "stt:native-extraction-unavailable: no ffmpeg binary for native audio extraction", and `getCaptionTranslationUnits` has nothing to translate. `Openscreen captions <project>` falls back to the renderer decode, which refuses the path: "Transcription failed: File path is not approved or is not a supported video file". A system `/usr/bin/ffmpeg` is on `PATH` and is not used. Not run: everything in the GUI (grant declined); screen and window recording (no xdg-desktop-portal ScreenCast or PipeWire in WSLg); the webcam (none in WSL); the tray (not shown by WSLg); the in-app HTTP server (its token is read from the GUI); transcription in the editor, which may still work through its own fallback. |
| 2026-10-14 | `main` @ `0bed36852` (adds #1192, #1194, #1197 over the 2026-10-14 Windows rows; c541fffaa, the recording-state fix, landed after this build and is not in it), CI artifact `openscreen-mac-arm64` from build run 38581408372, DMG `Openscreen-macOS-Apple-Silicon-2.0.0.dmg` signed and notarized (`spctl`: accepted, Notarized Developer ID, M4LK7C6S84), installed over `/Applications/Openscreen.app`; About reports `2.0.0 (dmg)`, Electron 41.2.1, Chromium 146.0.7680.188, Node 24.14.1, darwin arm64 | macOS 26.5 (25F71), Mac mini M1 (arm64), one 3840×2160 display at 2× (UI 1920×1080), no camera (Scaleway MDM `allowCamera=false`), BlackHole 2ch as the only input | **Partial, 4 defects; MCP section and #1194 first, rest scoped** | Real OS mouse and keyboard through computer-use; MCP calls from `@modelcontextprotocol/sdk` 1.x (stdio to `Openscreen mcp`, StreamableHTTP to `127.0.0.1:<port>/mcp` with the token copied from AI settings); numbers are ffprobe/ffmpeg on the files. **Pickers:** the first launch (from Terminal, no `OPENSCREEN_MAC_SOURCE_PICKER`) only served the permissions window; every take used the legacy selector (`open --env OPENSCREEN_MAC_SOURCE_PICKER=legacy`) or the MCP servers, none went through Apple's picker. **#1194:** `keepOnTop` is a no-op off Windows, so the bar stays at window level 3 (floating), under the Dock (20) and the menu bar (24); it stays above a frontmost Finder window after the source selector, Device settings and the language menu, keeps its tooltips and clicks, stops under the menu bar and above the Dock when dragged, is pushed down by the language menu near the top and comes back (#1161), and does not float over another app's full-screen Space, as `macSpaces.ts` documents. **MCP stdio:** `help` lists `mcp`; `listTools` 51 tools; `listSources` displays, windows, both BlackHole inputs, no camera; window take 1840×872 with no Apple picker, 4 clicks 5 s apart gave 1 zoom (the no-yo-yo rule of `resolveShots`, not a defect); edit to a closed project, `restoreCheckpoint` to the raw take, edit without checkpoint refused; `exportVideo` 30 fps after a 6 s trim, 1085 frames for 36.17 s as predicted; `getFrames` maps ruler 27 s to file 21 s and warns once the project changed; `cursor: "system"` take has the system pointer and 0 zooms; `camera: true` refused (MDM); `setPointerTrack` 391 samples, 1 zoom, cursor drawn in the export. **#1174:** in-app server off, project open: `listProjects` `open: true`, a stdio edit shows live and one Cmd+Z takes it back, an editor change keeps it, `restoreCheckpoint` lands live and one Cmd+Z restores every edit; app quit: the edit asks for a checkpoint, then lands in the file after `createCheckpoint`; relaunch and reopen: the same client session edits live again. **In-app server:** off by default with *MCP edits* off; *Running* with port, token (Show, Copy, Regenerate) and snippets; edit, `startRecording` and `exportVideo` refused naming *MCP edits*; once allowed, an edit lands live and undoes with Cmd+Z; port 47821 → 47822 then *Apply*: the old URL fails, a client reconnects on the new one. **#1180 tools, both servers:** `createProject`, `setProjectInfo`, `transcribe` (35 English words, about 10 s), `getCaptions`, `setCaptionSettings` (bold), `getCaptionTranslationUnits fr` (4 units), `putCaptionTranslation` with a French translation written by hand and shown, `setEditorSettings` (laptop or window frame, wallpaper, 16:9, `webcamBackgroundMode: transparent` stored with the reason "No clip on the timeline has a camera"), `addAnnotation` (72 px title, mosaic and smooth blurs), `importMedia` (MP4 and M4A), `insertClip`, `duplicateClip`, `setClipRange` `cropRegion` (cropped frame in the export), `removeMedia` refused while on the timeline and naming the clips, then removing an unused asset, `addAutoZooms` (0 added, telemetry per asset). **Exports:** MP4 at `fps: 30`, 1920×1080, 721 frames for 24.03 s, and 1280×720, 1687 frames for 56.23 s, both as predicted; GIF `gifSize: "small"`, `gifLoop: false`: 852×480, 15 fps, no NETSCAPE2.0 block (#1197); `gifLoop` on an `.mp4` refused; MP4 opened in QuickTime and GIF in Preview, judged by their windows only (the viewers' grant was refused). **CLI:** `Openscreen captions <project>` transcribes on Metal (11.7 s in 8.4 s), turns captions on, no `auto-caption` annotation. **App, legacy picker:** permissions window lists the 15.2+ rows (System audio, Accessibility, Microphone, Camera), then Screen & system audio (Required) first under `legacy`; Device settings lists the inputs and "No camera found" with no quality row; camera toggle refused with "Camera not found."; Record mode rows (Recordings folder last, Auto-zoom only with Editable cursor), *Start recording* hands over to the bar with the same settings; pause stops the timer and resume restarts it; stop opens Edit with 3 zooms; full-screen take 3840×2160 edge to edge, bar and desktop icons absent, system audio −12.6 dB; zoom-in and zoom-out trail tooltips; Glass Lens in preview and in a 720p30 export (1234 frames, true peak −1.6 dBTP, −18.5 LUFS on the clipped fixture voice); *Mouse clicks* off by default, *Click volume* once on, a click at 34.76 s in silence; export dialog (formats, sizes, rates, no codec), progress, *Saved to*, *Show in folder*; Cmd+O lists MCP projects; #997 holds the last clip's last frame at 0:24.0; app menu and menu bar item menus; About; Save Diagnostics writes a bundle. **Defects. (1) P1, MCP in-app `startRecording` (same as Windows, fixed after this build by c541fffaa):** the bar records within 5 s but the call returns after 120 s with "capture had not started", and on macOS the take is then lost: `restore()` ends it, no file is left, no editor opens, and `stopRecording` says "No recording is running." **(2) P2, launch blocked by the stdio server:** while an `Openscreen mcp` client session runs, opening OpenScreen from Finder or `open` starts no GUI (exit 0, no window); LaunchServices activates the `mcp` process instead, which turns `Foreground` with its own Dock icon after its first take. `open -n` works around it. **(3) P3, `getFrames` cannot read a GIF:** with `path` to a GIF, or with `projectId` after a GIF export, both servers answer "Chromium could not decode the video (media error 4)", so the documented check after `exportVideo` fails for GIFs. **(4) P3, system audio on a window take:** an MCP window take with `systemAudio: true` is silent (−91 dB) while sound plays from another app, because `SCContentFilter(desktopIndependentWindow:)` keeps only the window's own app; a display take of the same sound reads −10.5 dB. Windows records the whole mix. **Minor:** the `camera: true` error still carries a JavaScript stack trace; the stop click gets a click sound at the end of the export (40.9 s) although the zoom planner discards it. **Skipped:** Apple's picker slice and the permissions window's prompt rows (operator chose the unattended legacy run; Screen Recording already granted); the system-language prompt (system in English, so it never shows); Spaces switch (one Space); every camera and Full Camera check (MDM); in-app `startRecording`/`stopRecording` end to end, its zooms, settings restore and the unsaved-changes refusal (blocked by defect 1, the bar's settings did come back after the timeout); microphone audio (no source plays into BlackHole); Windows, Linux, macOS 13/14, Intel; not run for time: chat, names and terms, transcript editing, clip anchoring, audio tracks, voice-over, caption placement, persistence, editor bounds, GIF timing against v2.0.0, export cancel. |
Loading