From f1fd2bdd332f7ad918f5f161fa119dc7c91b16b6 Mon Sep 17 00:00:00 2001 From: m1 Date: Thu, 30 Jul 2026 13:35:08 +0200 Subject: [PATCH 1/7] fix(stt): make the macOS whisper helper actually shippable, and report its real device MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Building the whisper.cpp helper on macOS produced a staged directory that only worked on the machine that built it, and misreported what it had done. Three defects, all in the blind spot left by unit tests that mock `fetch`: - The sidecar glob (`ggml*.*`) matched only Windows DLL spelling. CMake emits `libggml-base.dylib` / `libggml-base.so.0` elsewhere, so on macOS and Linux every ggml library was skipped and the helper died in dyld before main(). It also swept up `.lib`/`.exp`/`.pdb` build by-products into the Windows installer. The glob now covers all three platforms' naming (27/27 real sidecars, 0 by-products) and `cp -a` keeps the version symlink farm instead of dereferencing each link into a full copy. - The staged binary's only LC_RPATH was an absolute path into `.cache/whisper-stt-build`. Deleting the build cache — or downloading the CI artifact onto another runner, which is what build.yml does — left dyld unable to find libwhisper: SIGABRT, exit 134, before main(). Now rewritten to `@loader_path` and re-signed ad-hoc. - `detect_active_backend()` matched the string "Metal", but ggml's Metal *device* is named from GGML_METAL_NAME ("MTL0"), so every Apple Silicon run reported `whispercpp-cpu` while Metal was bound. It now matches MTL and filters on device type GPU/IGPU, which also excludes ggml-blas. Also: `detected_language` echoed the request parameter. The renderer always sends "auto", so the field was permanently the literal "auto" — displayed in the media stage's "detected language" line and written onto AxcutTranscript.language. It now reports whisper_full_lang_id(). stage-whisper-stt.sh's load check could not catch any of this: it asserted the helper printed *something*, which is a Windows-shaped test (that loader dies mute) — dyld's four-line error satisfied it. It now requires the `[whisper-stt] boot:` line main() emits, and no longer depends on `timeout`, which is absent from a stock macOS runner and made the check pass without ever starting the binary. Adds scripts/test-whisper-stt.mjs (`npm run test:whisper-stt`), which round-trips a real clip through a real helper and asserts the contract. Verified on macOS arm64/Metal: WER 0.0000 on an English and a French clip, rtf 0.18-0.25 on an M1, from a relocated copy with the build tree deleted. Vulkan on Windows and Linux is unverified since the backend-detection change. Fixes the `nproc` fallback that built with -j4 on every Mac. --- electron/native/whisper-stt/src/main.cpp | 43 ++- package.json | 1 + scripts/build-whisper-stt.sh | 101 +++++- scripts/stage-whisper-stt.sh | 32 +- scripts/test-whisper-stt.mjs | 326 ++++++++++++++++++ .../transcription-and-captions.md | 103 +++++- 6 files changed, 587 insertions(+), 19 deletions(-) create mode 100644 scripts/test-whisper-stt.mjs diff --git a/electron/native/whisper-stt/src/main.cpp b/electron/native/whisper-stt/src/main.cpp index 1cc067697..339f759b1 100644 --- a/electron/native/whisper-stt/src/main.cpp +++ b/electron/native/whisper-stt/src/main.cpp @@ -175,14 +175,30 @@ std::string detect_active_backend() { for (size_t i = 0; i < n; ++i) { ggml_backend_dev_t dev = ggml_backend_dev_get(i); if (!dev) continue; + // Only compute devices are candidates. This drops ggml-cpu (type CPU) + // and ggml-blas (type ACCEL — Accelerate on macOS), neither of which is + // the GPU offload this field is meant to report. IGPU is accepted + // because that is how an integrated adapter enumerates. + // `auto`: ggml_backend_dev_type names both a C enum and the accessor + // function, so spelling the type out needs the `enum` tag to disambiguate. + const auto type = ggml_backend_dev_type(dev); + if (type != GGML_BACKEND_DEVICE_TYPE_GPU && type != GGML_BACKEND_DEVICE_TYPE_IGPU) { + continue; + } const char* name = ggml_backend_dev_name(dev); if (!name) continue; std::string s = name; - // ggml-cpu shows up as "CPU" — skip it on the first pass, only return - // it as a last resort. - if (s == "CPU" || s == "cpu") continue; + // These are ggml *device* names, which carry a device index: + // "Vulkan0", "CUDA0", "MTL0". Matching the bare backend title is why + // Metal never matched — ggml-metal's device name is built from + // GGML_METAL_NAME ("MTL"), and the string "Metal" appears nowhere in + // it, so every Apple Silicon run reported `whispercpp-cpu` while the + // Metal backend was in fact bound. `backend` is documented as the + // source of truth for which device ran, so this was the one number a + // consumer could not get right on macOS. if (s.find("Vulkan") != std::string::npos) return "whispercpp-vulkan"; if (s.find("CUDA") != std::string::npos) return "whispercpp-cuda"; + if (s.find("MTL") != std::string::npos) return "whispercpp-metal"; if (s.find("Metal") != std::string::npos) return "whispercpp-metal"; } // Last resort: the CPU device, or a hard "cpu" if ggml has nothing @@ -464,9 +480,26 @@ int main(int argc, char** argv) { } // ---- Emit verbose_json (CT2-compatible shape + backend + timing) ---- + // + // Report the language whisper actually decoded with, not the one the + // request asked for. `language` is the raw request parameter, and the + // renderer sends "auto" on every call (there is no language selector in + // the UI yet), so echoing it back meant `detected_language` never + // carried a language at all: the media stage's "detected language" line + // rendered the literal string "auto", and AxcutTranscript.language was + // set to it via transcribe.ts. whisper_full_lang_id() returns the id + // whisper resolved — the detected one under "auto", and the forced one + // otherwise, which is correct for both paths. + std::string resolved_language = language; + const int lang_id = whisper_full_lang_id(ctx); + if (lang_id >= 0) { + if (const char* lang_str = whisper_lang_str(lang_id)) { + resolved_language = lang_str; + } + } nlohmann::json reply; - reply["language"] = language; - reply["detected_language"] = language; + reply["language"] = resolved_language; + reply["detected_language"] = resolved_language; reply["backend"] = active_backend; reply["timing"] = { {"elapsed_s", elapsed_s}, diff --git a/package.json b/package.json index 4bb0f3d65..4646e9184 100644 --- a/package.json +++ b/package.json @@ -36,6 +36,7 @@ "build:win:store": "npm run build:native:win && npm run fetch:ffmpeg && npm run build:native:compositor && tsc && vite build && electron-builder --win appx --config.npmRebuild=false", "build:linux": "tsc && vite build && electron-builder --linux AppImage deb pacman --config.npmRebuild=false", "build:whisper-binaries": "bash scripts/build-whisper-stt.sh", + "test:whisper-stt": "node scripts/test-whisper-stt.mjs", "test": "vitest --run", "test:watch": "vitest", "test:cursor-native:win": "node scripts/test-windows-native-cursor.mjs", diff --git a/scripts/build-whisper-stt.sh b/scripts/build-whisper-stt.sh index 2e8ec5ffe..16d6a4f7e 100644 --- a/scripts/build-whisper-stt.sh +++ b/scripts/build-whisper-stt.sh @@ -87,6 +87,59 @@ backend_flag_for_host() { esac } +# --------------------------------------------------------------------------- +# macOS: make the staged tree relocatable. +# +# CMake links the helper and every ggml/whisper dylib against `@rpath/...` and +# then bakes a single absolute LC_RPATH pointing at its own build tree +# (`.cache/whisper-stt-build/build--/bin`). That resolves on the +# machine that ran the build and nowhere else: delete the build cache, or +# download the CI artifact onto a different runner, and dyld cannot find +# libwhisper at all — the process aborts (SIGABRT, exit 134) before main(). +# +# This is the macOS twin of the Windows 0xC0000135 incident that shipped in +# 1.8.0-rc.1..rc.3 (see the OpenSSL note in the CMakeLists). Swapping the +# absolute rpath for `@loader_path` makes each binary look for its siblings +# next to itself, which is exactly how the staged directory and the packaged +# `resources/electron/native/bin//` are laid out. +# +# Editing a Mach-O header invalidates its code signature, and arm64 refuses to +# execute an unsigned-but-modified image ("killed: 9"), so each patched file is +# re-signed ad-hoc afterwards. +# --------------------------------------------------------------------------- +relocate_macos_rpaths() { + local out_dir="$1" + + local f rp + for f in "${out_dir}"/*; do + # Symlinks share the payload of their target; patching the target is + # enough, and codesign would follow the link and sign it twice. + [[ -L "${f}" ]] && continue + [[ -f "${f}" ]] || continue + # Mach-O only: skips *.metal shader sidecars and anything else non-binary. + file -b "${f}" | grep -q "Mach-O" || continue + + # Drop every *absolute* LC_RPATH. An absolute rpath in a shipped artifact + # is always a build-machine path; relative ones (@loader_path, + # @executable_path) are already correct and must survive. + while read -r rp; do + [[ "${rp}" == /* ]] || continue + install_name_tool -delete_rpath "${rp}" "${f}" 2>/dev/null || true + done < <(otool -l "${f}" | awk '/cmd LC_RPATH/{n=1;next} n&&/^ *path /{print $2;n=0}') + + # Idempotent: a second run would otherwise stack duplicate rpaths. + if ! otool -l "${f}" | awk '/cmd LC_RPATH/{n=1;next} n&&/^ *path /{print $2;n=0}' | + grep -qx "@loader_path"; then + install_name_tool -add_rpath "@loader_path" "${f}" + fi + + codesign --force --sign - --timestamp=none "${f}" 2>/dev/null || + echo "[whisper-stt] WARN: could not re-sign ${f##*/}" >&2 + done + + echo "[whisper-stt] rewrote macOS rpaths to @loader_path in ${out_dir}" +} + build_variant() { local variant_name="$1" shift @@ -109,7 +162,10 @@ build_variant() { ${extra_cmake_flags[@]+"${extra_cmake_flags[@]}"} echo "[whisper-stt] building ${variant_name}" - cmake --build "${build_dir}" --config Release -j "$(nproc 2>/dev/null || echo 4)" + # ponytail: macOS has no `nproc` (it is GNU coreutils), so this silently built + # with -j4 on an 8-core Mac. `sysctl -n hw.ncpu` is the BSD equivalent. + cmake --build "${build_dir}" --config Release \ + -j "$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)" local bin_name="whisper-stt-server" if [[ "${OS_ARCH}" == win32-* ]]; then @@ -152,14 +208,36 @@ build_variant() { # Stage any shared libraries / backend sidecars that CMake produced. # Copy everything that looks like a ggml/whisper shared library, plus any # .metal shader files (only produced when Metal is not embedded). + # + # ponytail: the pattern list used to be `ggml*.*|whisper.*|libwhisper.*`, + # which only ever matched on Windows. CMake names shared libraries + # `ggml-base.dll` on Windows but `libggml-base.dylib` / `libggml-base.so.0` + # everywhere else, so on macOS and Linux every ggml sidecar was skipped and + # only libwhisper was staged — leaving a helper that cannot resolve + # @rpath/libggml.0.dylib and dies in dyld before main(). The `lib` prefix and + # the `.so.` version suffixes are what the globs below add. + # + # -a preserves the symlink farm (libggml.dylib -> libggml.0.dylib -> + # libggml.0.15.1.dylib); plain `cp` dereferences each one into a full copy of + # the same payload, which tripled the staged size for no benefit. local found_libs=0 for lib_dir in "${search_dirs[@]}"; do if [[ -d "${lib_dir}" ]]; then for f in "${lib_dir}"/*; do - if [[ -f "${f}" ]]; then + # -f follows symlinks (true for both link and target); -L catches the + # links themselves so the farm is staged intact. + if [[ -f "${f}" || -L "${f}" ]]; then + # Patterns, in order: Windows DLLs; macOS dylibs; Linux .so plus its + # versioned forms (libggml-base.so.0, libggml-base.so.0.15.1); Metal + # shader sidecars. A comment cannot be placed inside a `case` pattern + # list — a `| \` continuation would swallow it as part of the pattern. case "${f##*/}" in - ggml*.*|whisper.*|whisper.dylib|whisper.dll|libwhisper.*|*.metal) - cp "${f}" "${OUT_DIR}/" + ggml*.dll|whisper.dll|parakeet.dll|\ +libggml*.dylib|libwhisper*.dylib|libparakeet*.dylib|\ +libggml*.so|libggml*.so.*|libwhisper*.so|libwhisper*.so.*|\ +libparakeet*.so|libparakeet*.so.*|\ +*.metal) + cp -a "${f}" "${OUT_DIR}/" found_libs=1 ;; esac @@ -168,6 +246,21 @@ build_variant() { fi done + # Fail here rather than in the installer. `found_libs` was computed and then + # never read, so a staging glob that matched nothing — which is precisely what + # happened on macOS and Linux — still exited 0 and produced a green build. + if [[ "${found_libs}" -eq 0 ]]; then + echo "FATAL: staged no shared libraries alongside ${out_bin_name}." >&2 + echo " Searched: ${search_dirs[*]}" >&2 + echo " The helper links ggml/whisper as shared libraries; without them" >&2 + echo " it cannot start. Check the sidecar glob in this script." >&2 + exit 1 + fi + + if [[ "${OS_ARCH}" == darwin-* ]]; then + relocate_macos_rpaths "${OUT_DIR}" + fi + echo "[whisper-stt] built ${variant_name} -> ${OUT_DIR}/${out_bin_name}" ls -la "${OUT_DIR}" } diff --git a/scripts/stage-whisper-stt.sh b/scripts/stage-whisper-stt.sh index ffe9b45a7..37883d34e 100755 --- a/scripts/stage-whisper-stt.sh +++ b/scripts/stage-whisper-stt.sh @@ -86,9 +86,19 @@ if [ "${TAG#win32}" != "${TAG}" ]; then else MINIMAL_PATH="/usr/bin:/bin" fi +# `timeout` is GNU coreutils: absent from a stock macOS runner. Unqualified, it +# made the whole check evaporate there — `env: timeout: No such file or +# directory` is itself output, so the "did it print anything" test below passed +# without ever starting the binary. Use it only when it exists. +if command -v timeout >/dev/null 2>&1; then + TIMEOUT_CMD="timeout 60" +else + TIMEOUT_CMD="" +fi # A bad --model makes it fail fast; we only care that it got far enough to speak. set +e -LOAD_OUT="$(cd "${DEST}" && timeout 60 env PATH="${MINIMAL_PATH}" \ +# shellcheck disable=SC2086 # TIMEOUT_CMD is deliberately word-split (or empty). +LOAD_OUT="$(cd "${DEST}" && ${TIMEOUT_CMD} env PATH="${MINIMAL_PATH}" \ "./$(basename "${BIN}")" --model "__staging_load_check__" 2>&1)" LOAD_CODE=$? set -e @@ -98,6 +108,26 @@ if [ -z "${LOAD_OUT}" ]; then echo " Windows exit 3221225781 = 0xC0000135 STATUS_DLL_NOT_FOUND." >&2 exit 1 fi + +# "It printed something" is a Windows-shaped test: the loader there dies mute. +# dyld and ld.so are chatty, so on macOS/Linux a helper that never reached +# main() still satisfies the check above — which is exactly how a macOS build +# missing every libggml*.dylib passed staging. Demand the marker that main() +# itself prints (electron/native/whisper-stt/src/main.cpp), so the gate proves +# execution rather than mere noise. +case "${LOAD_OUT}" in + *"[whisper-stt] boot:"*) ;; + *) + echo "FATAL: $(basename "${BIN}") never reached main() (exit ${LOAD_CODE})." >&2 + echo " Expected its '[whisper-stt] boot:' line; got:" >&2 + printf '%s\n' "${LOAD_OUT}" | sed 's/^/ /' | head -n 12 >&2 + echo " A dynamic-loader error here means a sidecar is missing from the" >&2 + echo " artifact, or the binary carries an absolute rpath/RUNPATH into" >&2 + echo " the build machine's tree. Both are bugs in" >&2 + echo " scripts/build-whisper-stt.sh's staging step." >&2 + exit 1 + ;; +esac echo "Load check OK (exit ${LOAD_CODE}): $(printf '%s' "${LOAD_OUT}" | head -n 1)" echo "Staged $(basename "${BIN}") + $(( $(ls -1 "${DEST}" | wc -l) - 1 )) sidecar(s) -> ${DEST}" diff --git a/scripts/test-whisper-stt.mjs b/scripts/test-whisper-stt.mjs new file mode 100644 index 000000000..b1f9b4504 --- /dev/null +++ b/scripts/test-whisper-stt.mjs @@ -0,0 +1,326 @@ +// Round-trips a real WAV through a real `whisper-stt-server` and checks the +// response against the invariants in +// technical-documentation/architecture/transcription-and-captions.md. +// +// WHY THIS EXISTS. The helper's unit tests mock `fetch`, so every assertion +// about the wire format was made against a hand-written fixture rather than the +// binary. Three defects lived in that blind spot at once on macOS: no ggml +// dylib was staged next to the executable, the executable carried an absolute +// rpath into the build tree, and `backend`/`detected_language` reported values +// the helper had never actually resolved. All three are things only a real +// request can see. +// +// Usage: +// node scripts/test-whisper-stt.mjs # macOS: synthesizes speech with `say` +// node scripts/test-whisper-stt.mjs --wav a.wav # any platform: bring your own clip +// node scripts/test-whisper-stt.mjs --wav a.wav --ref "expected words" +// node scripts/test-whisper-stt.mjs --language fr # force a language instead of auto +// +// Env overrides: +// OPENSCREEN_WHISPER_SERVER_EXE helper binary (default: the staged one for this host) +// OPENSCREEN_WHISPER_MODEL GGML model (default: the userData cache location) + +import { spawn, spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const ROOT = path.join(__dirname, ".."); + +const argOf = (flag) => { + const i = process.argv.indexOf(flag); + return i >= 0 && i + 1 < process.argv.length ? process.argv[i + 1] : null; +}; + +const TAG = `${process.platform}-${process.arch}`; +const BIN = + process.env.OPENSCREEN_WHISPER_SERVER_EXE ?? + path.join( + ROOT, + "electron", + "native", + "bin", + TAG, + process.platform === "win32" ? "whisper-stt-server.exe" : "whisper-stt-server", + ); + +// Mirrors modelManager.ts's cache location. Electron's userData dir is +// "Electron" in dev and the productName once packaged. +const MODEL = process.env.OPENSCREEN_WHISPER_MODEL ?? defaultModelPath(); +const LANGUAGE = argOf("--language") ?? "auto"; +const WAV_ARG = argOf("--wav"); +const REF_ARG = argOf("--ref"); + +function defaultModelPath() { + const file = path.join("stt-models", "whisper-ggml", "ggml-small-q8_0.bin"); + if (process.platform === "darwin") { + return path.join(os.homedir(), "Library", "Application Support", "Electron", file); + } + if (process.platform === "win32") { + return path.join(process.env.APPDATA ?? "", "Electron", file); + } + return path.join(os.homedir(), ".config", "Electron", file); +} + +const failures = []; +const check = (ok, label, detail = "") => { + console.log(` ${ok ? "PASS" : "FAIL"} ${label}${detail ? ` — ${detail}` : ""}`); + if (!ok) failures.push(label); +}; + +/** Minimal RIFF/WAVE parse: enough to assert the clip is what the helper expects. */ +function readWavMeta(file) { + const buf = fs.readFileSync(file); + if (buf.toString("ascii", 0, 4) !== "RIFF" || buf.toString("ascii", 8, 12) !== "WAVE") { + throw new Error(`${file} is not a RIFF/WAVE file`); + } + let offset = 12; + let fmt = null; + let dataBytes = null; + while (offset + 8 <= buf.length) { + const id = buf.toString("ascii", offset, offset + 4); + const size = buf.readUInt32LE(offset + 4); + if (id === "fmt ") { + fmt = { + channels: buf.readUInt16LE(offset + 10), + sampleRate: buf.readUInt32LE(offset + 12), + bitsPerSample: buf.readUInt16LE(offset + 22), + }; + } else if (id === "data") { + dataBytes = size; + } + offset += 8 + size + (size % 2); + } + if (!fmt || dataBytes == null) throw new Error(`${file} has no fmt/data chunk`); + const bytesPerFrame = (fmt.bitsPerSample / 8) * fmt.channels; + return { ...fmt, durationSec: dataBytes / bytesPerFrame / fmt.sampleRate }; +} + +/** macOS-only: synthesize a clip with a known transcript so WER is meaningful. */ +function synthesizeWav() { + const text = + "And so my fellow Americans, ask not what your country can do for you, " + + "ask what you can do for your country."; + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "whisper-stt-test-")); + const aiff = path.join(dir, "speech.aiff"); + const wav = path.join(dir, "speech.wav"); + for (const [cmd, args] of [ + ["say", ["-o", aiff, text]], + ["afconvert", ["-f", "WAVE", "-d", "LEI16@16000", "-c", "1", aiff, wav]], + ]) { + const r = spawnSync(cmd, args, { stdio: "inherit" }); + if (r.status !== 0) throw new Error(`${cmd} failed (status ${r.status})`); + } + return { wav, ref: text, cleanup: () => fs.rmSync(dir, { recursive: true, force: true }) }; +} + +const normalize = (s) => + s + .toLowerCase() + .replace(/[^\p{L}\p{N}\s']/gu, " ") + .split(/\s+/) + .filter(Boolean); + +/** Word error rate: Levenshtein over word arrays, (S+D+I)/N. */ +function wer(refText, hypText) { + const ref = normalize(refText); + const hyp = normalize(hypText); + if (ref.length === 0) return hyp.length === 0 ? 0 : 1; + let prev = Array.from({ length: hyp.length + 1 }, (_, j) => j); + for (let i = 1; i <= ref.length; i++) { + const cur = [i]; + for (let j = 1; j <= hyp.length; j++) { + cur[j] = Math.min( + prev[j] + 1, + cur[j - 1] + 1, + prev[j - 1] + (ref[i - 1] === hyp[j - 1] ? 0 : 1), + ); + } + prev = cur; + } + return prev[hyp.length] / ref.length; +} + +async function waitForReady(baseUrl, timeoutMs = 120_000) { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + try { + const res = await fetch(baseUrl, { method: "GET" }); + if (res.ok) return; + } catch { + // not up yet + } + await new Promise((r) => setTimeout(r, 250)); + } + throw new Error(`helper did not become ready within ${timeoutMs}ms`); +} + +async function main() { + for (const [label, p] of [ + ["helper binary", BIN], + ["GGML model", MODEL], + ]) { + if (!fs.existsSync(p)) { + console.error(`FATAL: ${label} not found at ${p}`); + if (label === "helper binary") { + console.error(" Build it: bash scripts/build-whisper-stt.sh"); + } else { + console.error( + " Run a transcription in the app once, or set OPENSCREEN_WHISPER_MODEL.", + ); + } + process.exit(1); + } + } + + let wavPath = WAV_ARG; + let refText = REF_ARG; + // No-op unless we synthesize a clip into a temp dir that needs removing. + let cleanup = () => undefined; + if (!wavPath) { + if (process.platform !== "darwin") { + console.error("FATAL: --wav is required off macOS (no `say` to synthesize with)."); + process.exit(2); + } + const made = synthesizeWav(); + wavPath = made.wav; + refText = refText ?? made.ref; + cleanup = made.cleanup; + } + + const meta = readWavMeta(wavPath); + console.log(`\nclip : ${wavPath}`); + console.log( + `format : ${meta.channels}ch ${meta.sampleRate}Hz ${meta.bitsPerSample}-bit, ` + + `${meta.durationSec.toFixed(2)}s`, + ); + console.log(`helper : ${BIN}`); + console.log(`model : ${MODEL}\n`); + + const port = 20100 + Math.floor(process.pid % 400); + const child = spawn( + BIN, + [ + "--model", + MODEL, + "--port", + String(port), + "--host", + "127.0.0.1", + "--threads", + String(Math.max(1, os.cpus().length)), + ], + { stdio: ["ignore", "pipe", "pipe"] }, + ); + let stderr = ""; + child.stderr.on("data", (d) => { + stderr += String(d); + }); + child.on("exit", (code, signal) => { + if (code !== 0 && code !== null) { + console.error(`\nhelper exited early (code ${code}, signal ${signal}):`); + console.error(stderr.split("\n").slice(-15).join("\n")); + } + }); + + let json; + try { + await waitForReady(`http://127.0.0.1:${port}/`); + const form = new FormData(); + form.append("file", new Blob([fs.readFileSync(wavPath)]), path.basename(wavPath)); + form.append("language", LANGUAGE); + form.append("response_format", "verbose_json"); + const res = await fetch(`http://127.0.0.1:${port}/inference`, { + method: "POST", + body: form, + }); + if (!res.ok) throw new Error(`/inference returned ${res.status}: ${await res.text()}`); + json = await res.json(); + } finally { + child.kill(); + cleanup(); + } + + const segments = json.segments ?? []; + const words = segments.flatMap((s) => s.words ?? []); + const text = segments + .map((s) => s.text ?? "") + .join("") + .trim(); + + console.log(`text : ${text}\n`); + console.log("assertions:"); + + check(segments.length > 0, "response carries phrase segments", `${segments.length}`); + check(words.length > 0, "response carries per-word timings", `${words.length} words`); + + // The DTW guardrail lives in the helper; its stderr line is the only place + // the result is reported, and a FAIL there is a 500 rather than a bad body. + check(/§4\.1 guardrail: PASS/.test(stderr), "DTW guardrail passed in the helper"); + + // `backend` is documented as the source of truth for the device that ran. + // Metal reported as CPU is the exact bug this catches: ggml's Metal *device* + // is named "MTL0", so a matcher looking for "Metal" never fires. + const gpuExpected = + (process.platform === "darwin" && process.arch === "arm64") || process.platform === "win32"; + check( + typeof json.backend === "string" && json.backend.startsWith("whispercpp-"), + "backend is a contract value", + String(json.backend), + ); + if (gpuExpected) { + check( + json.backend !== "whispercpp-cpu", + "backend reports GPU offload on a GPU-capable host", + String(json.backend), + ); + } + + // "auto" is a request value, never a response value: the helper must report + // the language whisper resolved. + check( + typeof json.detected_language === "string" && + json.detected_language !== "auto" && + json.detected_language.length > 0, + "detected_language is a resolved language, not the request echo", + String(json.detected_language), + ); + + // Contract invariant: absolute seconds in the source recording, [0, duration). + const tolerance = 0.5; // whisper rounds segment ends up to the chunk edge + const inRange = words.every( + (w) => w.start >= 0 && w.end >= w.start && w.end <= meta.durationSec + tolerance, + ); + check(inRange, "word times lie within the clip", `clip is ${meta.durationSec.toFixed(2)}s`); + + let monotonic = true; + for (let i = 1; i < words.length; i++) { + if (words[i].start < words[i - 1].start) monotonic = false; + } + check(monotonic, "word starts are monotonic non-decreasing"); + + if (refText) { + const rate = wer(refText, text); + check(rate <= 0.15, "WER within tolerance", `${rate.toFixed(4)}`); + } + + if (json.timing) { + console.log( + `\ntiming : ${json.timing.elapsed_s?.toFixed?.(2)}s for ` + + `${json.timing.audio_s?.toFixed?.(2)}s audio (rtf ${json.timing.rtf?.toFixed?.(3)})`, + ); + } + + if (failures.length > 0) { + console.error(`\n${failures.length} check(s) failed: ${failures.join("; ")}`); + process.exit(1); + } + console.log("\nAll checks passed."); +} + +main().catch((err) => { + console.error(`\nFATAL: ${err.message}`); + process.exit(1); +}); diff --git a/technical-documentation/architecture/transcription-and-captions.md b/technical-documentation/architecture/transcription-and-captions.md index 1ae45bc0e..c9694ec79 100644 --- a/technical-documentation/architecture/transcription-and-captions.md +++ b/technical-documentation/architecture/transcription-and-captions.md @@ -55,17 +55,32 @@ factors — lives in ### Per-platform backend -| Platform | Helper binary | Runtime device candidates | +The binary is named `whisper-stt-server` (`.exe` on Windows) on **every** +platform — the backend is a build-time flag, not a file name, and +`binaryNameForBackend` +([`electron/stt/gpuDetector.ts:60`](../../electron/stt/gpuDetector.ts:60)) +returns that single name. What varies is which backend was compiled in: + +| Platform | Backend compiled in | Runtime device candidates | |---|---|---| -| macOS arm64 | `whisper-stt-server-metal` | Metal, CPU | -| macOS x64 | `whisper-stt-server-cpu` | CPU | -| Windows x64 | `whisper-stt-server-vulkan` | Vulkan, CPU | -| Linux x64 | `whisper-stt-server-vulkan` | Vulkan, CPU | +| macOS arm64 | Metal (`-DOSC_ENABLE_METAL=ON`) | Metal, CPU | +| macOS x64 | CPU only | CPU | +| Windows x64 | Vulkan (`-DOSC_ENABLE_VULKAN=ON`) | Vulkan, CPU | +| Linux x64 | Vulkan (`-DOSC_ENABLE_VULKAN=ON`) | Vulkan, CPU | whisper.cpp chooses the device in `whisper_backend_init`; the helper reports what actually bound (via `ggml_backend_dev_name()`) and `SttManager` returns it verbatim in the response. +> `ggml_backend_dev_name()` returns a *device* name carrying an index — +> `Vulkan0`, `CUDA0`, `MTL0` — not the backend's title. Metal is the trap: +> ggml-metal builds its device name from `GGML_METAL_NAME` (`"MTL"`), so the +> string `"Metal"` appears nowhere in it. `detect_active_backend()` matched +> `"Metal"` and therefore reported `whispercpp-cpu` on every Apple Silicon run +> while the Metal backend was in fact bound. It now matches `MTL` and filters on +> `ggml_backend_dev_type()` being GPU/IGPU, which also drops ggml-blas +> (Accelerate on macOS, device type ACCEL) from consideration. + ### Word-level alignment (DTW token timestamps) 1. **Decode** — `whisper_full()` runs with the default sampling parameters @@ -170,6 +185,61 @@ curl -X POST -F "file=@test.wav" -F "language=auto" -F "response_format=verbose_ The helper's stderr logs the actual backend it bound (`whispercpp-vulkan` / `-metal` / `-cpu`). +`npm run test:whisper-stt` round-trips a real clip through a real helper and +checks the response against the invariants below — segments and per-word +timings present, the DTW guardrail passed, `backend` reporting GPU offload on a +GPU-capable host, `detected_language` resolved rather than echoed, word times +monotonic and inside the clip, and WER against a reference. On macOS it +synthesizes its own clip with `say`, so it needs no fixture; elsewhere pass +`--wav ` (and optionally `--ref ""`). This is the check +that the unit tests structurally cannot make: they mock `fetch`, so they assert +against a hand-written fixture rather than the binary. + +#### What the staged directory has to contain + +`electron/native/bin/-/` is not just the executable. whisper.cpp is +built as **shared** libraries, so the helper is dynamically linked against +`whisper` plus the ggml backends, and every one of them has to sit next to the +binary: + +- **Windows** — `whisper.dll`, `ggml.dll`, `ggml-base.dll`, `ggml-cpu.dll`, + `ggml-vulkan.dll`. +- **macOS** — `libwhisper.*.dylib`, `libggml{,-base,-cpu,-blas,-metal}.*.dylib`, + plus the version symlink farm (`libggml.dylib` → `libggml.0.dylib` → + `libggml.0.15.1.dylib`). +- **Linux** — the `.so` equivalents, including the `.so.` and + `.so.` links. + +Two macOS/Linux-specific hazards, both of which shipped silently: + +1. **The sidecar glob.** CMake emits `ggml-base.dll` on Windows but + `libggml-base.dylib` / `libggml-base.so.0` elsewhere. A glob written as + `ggml*.*` matches only the Windows spelling, so the staging step copied + nothing but `libwhisper` on macOS and Linux and the helper died in dyld + before `main()`. `build_variant()` now globs the `lib`-prefixed and + `.so.`-suffixed forms too, uses `cp -a` to keep the symlink farm intact + instead of dereferencing each link into a full copy, and **fails the build** + when it stages zero libraries. +2. **Absolute rpaths.** CMake bakes an `LC_RPATH` pointing at its own build + tree, which resolves on the machine that built it and nowhere else — delete + `.cache/`, or download the CI artifact onto a different runner, and the + helper aborts (SIGABRT, exit 134) in the loader. + `relocate_macos_rpaths()` strips every absolute `LC_RPATH` from the + executable and each dylib, adds `@loader_path`, and re-signs ad-hoc + (editing a Mach-O header invalidates its signature, and arm64 refuses to + execute a modified unsigned image). + +`scripts/stage-whisper-stt.sh` gates the installer on actually loading the +binary with a scrubbed `PATH`. That gate was written for Windows, where the +loader dies mute, so it asserted only that the process *printed something* — +and dyld and `ld.so` are chatty, so a macOS helper missing every ggml dylib +printed a four-line loader error and passed. It now requires the +`[whisper-stt] boot:` line that `main()` itself emits, which proves execution +rather than noise, and it no longer depends on `timeout` (GNU coreutils, absent +from a stock macOS runner — unqualified, `env: timeout: No such file or +directory` was itself enough "output" to satisfy the old check without ever +starting the binary). + ## The transcription contract The wire types in `electron/stt/transcriptionContract.ts` are shared across @@ -390,16 +460,31 @@ it deletes data first window and slightly improve WER. Needs a UI control wired through `setTranscript` and a mapping from the UI string to a whisper.cpp language token. + + Note that `"auto"` is a *request* value only. The helper used to echo the + request straight back into `detected_language`, so with no selector the field + was permanently the literal string `"auto"`: the media stage's "detected + language" line ([`src/components/ai-edition/Modals.tsx:1763`](../../src/components/ai-edition/Modals.tsx:1763)) + displayed it verbatim, and `transcribe.ts` wrote it onto + `AxcutTranscript.language`. It now reports `whisper_full_lang_id()` — the + detected language under `"auto"`, the forced one otherwise. - **No CUDA build.** Vulkan already accelerates NVIDIA on Windows and Linux, but a dedicated CUDA build can be added later via `OSC_ENABLE_CUDA=ON`; the build script and CMake already accept the flag, the default matrix doesn't build it yet. - **No CoreML/ANE encoder.** Metal already covers Apple GPU; CoreML is a future perf refinement. -- **HTTP integration test not on CI.** No automated `POST /inference` - round-trip against a known WAV on the macOS-ARM/Metal, - Ubuntu-x64/Vulkan, Windows-x64/Vulkan matrix. The native build - matrix is the right place to land it. +- **HTTP integration test not on CI.** `npm run test:whisper-stt` does the + `POST /inference` round-trip and asserts the contract, but nothing runs it on + the macOS-ARM/Metal, Ubuntu-x64/Vulkan, Windows-x64/Vulkan matrix. The native + build matrix (`build-whisper-stt.yml`) is the right place to land it — each + job already has the freshly built helper on disk, so it only needs a clip and + the model. Off macOS the harness needs `--wav`, so a committed fixture (or a + runner-side TTS) is the missing piece. Until then the three GPU paths are + verified only by whoever runs it locally: **macOS arm64/Metal was verified + this way on 2026-07-30 (WER 0.0000 on an English and a French clip, rtf + 0.18–0.25 on an M1); Vulkan on Windows and Linux has not been re-checked + since the backend-detection fix.** - **No C++ unit tests.** The WAV reader and the DTW-inactive guardrail in `electron/native/whisper-stt/src/main.cpp` are exercised only at runtime. From d344fba1426e7cadded71bcc7ec923c2a46ed600 Mon Sep 17 00:00:00 2001 From: m1 Date: Thu, 30 Jul 2026 13:50:17 +0200 Subject: [PATCH 2/7] fix(stt): resolve the model cache from the app's userData name, not "Electron" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The harness hardcoded ~/Library/Application Support/Electron, but the leaf of app.getPath("userData") is app.getName() — "openscreen" in dev, the productName once packaged. "Electron" is only what a bare `electron .` with no app name uses, so the default pointed somewhere the app never writes. Probe the plausible names and take the one that exists. --- scripts/test-whisper-stt.mjs | 29 ++++++++++++++++++++--------- 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/scripts/test-whisper-stt.mjs b/scripts/test-whisper-stt.mjs index b1f9b4504..702d06cef 100644 --- a/scripts/test-whisper-stt.mjs +++ b/scripts/test-whisper-stt.mjs @@ -46,22 +46,33 @@ const BIN = process.platform === "win32" ? "whisper-stt-server.exe" : "whisper-stt-server", ); -// Mirrors modelManager.ts's cache location. Electron's userData dir is -// "Electron" in dev and the productName once packaged. const MODEL = process.env.OPENSCREEN_WHISPER_MODEL ?? defaultModelPath(); const LANGUAGE = argOf("--language") ?? "auto"; const WAV_ARG = argOf("--wav"); const REF_ARG = argOf("--ref"); +/** + * Mirrors `SttManager`'s cache location: `app.getPath("userData")/stt-models` + * ([electron/stt/index.ts:69](../electron/stt/index.ts)). + * + * The leaf of that userData path is `app.getName()`, which is package.json's + * `name` ("openscreen") in dev and the productName once packaged — *not* + * "Electron", which is only what a bare `electron .` with no app name would + * use. Probe the plausible names and take the one that exists, so this works + * whether the model was cached by the app or by a bare helper run. + */ function defaultModelPath() { const file = path.join("stt-models", "whisper-ggml", "ggml-small-q8_0.bin"); - if (process.platform === "darwin") { - return path.join(os.homedir(), "Library", "Application Support", "Electron", file); - } - if (process.platform === "win32") { - return path.join(process.env.APPDATA ?? "", "Electron", file); - } - return path.join(os.homedir(), ".config", "Electron", file); + const roots = + process.platform === "darwin" + ? [path.join(os.homedir(), "Library", "Application Support")] + : process.platform === "win32" + ? [process.env.APPDATA ?? ""] + : [process.env.XDG_CONFIG_HOME || path.join(os.homedir(), ".config")]; + const candidates = roots.flatMap((root) => + ["openscreen", "Openscreen", "Electron"].map((appName) => path.join(root, appName, file)), + ); + return candidates.find((p) => fs.existsSync(p)) ?? candidates[0]; } const failures = []; From 308da0e52bf754ef415eefaa117a3cbfb949c829 Mon Sep 17 00:00:00 2001 From: m1 Date: Thu, 30 Jul 2026 13:57:12 +0200 Subject: [PATCH 3/7] fix(i18n): transcription does not run in the browser MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit transcript.whisperHint told users "runs in your browser". Whisper runs in the main process: SttManager spawns the native whisper-stt-server helper (whisper.cpp linked statically) and talks to it over loopback HTTP. The renderer only ships it audio and maps the result back. See technical-documentation/architecture/transcription-and-captions.md. Reworded to "runs on your computer" (and the per-language equivalent) across all 13 locales. The "no data leaves the device" half was already accurate and is untouched — as is the rest of each sentence, since only the offending clause was replaced. "computer" rather than "device" because the second half already says data never leaves the device, and repeating it reads as a stutter in most of these languages. Only whisperHint made the claim; nothing else in the locales, the docs or the website says transcription is a browser feature. npm run i18n:check passes; the diff is 13 lines across 13 files. --- src/i18n/locales/ar/settings.json | 2 +- src/i18n/locales/en/settings.json | 2 +- src/i18n/locales/es/settings.json | 2 +- src/i18n/locales/fr/settings.json | 2 +- src/i18n/locales/it/settings.json | 2 +- src/i18n/locales/ja-JP/settings.json | 2 +- src/i18n/locales/ko-KR/settings.json | 2 +- src/i18n/locales/pt-BR/settings.json | 2 +- src/i18n/locales/ru/settings.json | 2 +- src/i18n/locales/tr/settings.json | 2 +- src/i18n/locales/vi/settings.json | 2 +- src/i18n/locales/zh-CN/settings.json | 2 +- src/i18n/locales/zh-TW/settings.json | 2 +- 13 files changed, 13 insertions(+), 13 deletions(-) diff --git a/src/i18n/locales/ar/settings.json b/src/i18n/locales/ar/settings.json index 242fd8c82..39e893a6c 100644 --- a/src/i18n/locales/ar/settings.json +++ b/src/i18n/locales/ar/settings.json @@ -245,7 +245,7 @@ "help": "نص مجمّع لكل المقاطع على المخطط الزمني. اضغط Backspace / Delete على كلمة أو تحديد لتمييزه كمتخطّى (بالأحمر). مرّر المؤشر فوق المقطع الأحمر لاستعادته.", "noClips": "لا توجد مقاطع بعد", "noTranscript": "لا يوجد نص بعد", - "whisperHint": "يستخدم التفريغ النصي Whisper محليًا — يعمل داخل متصفحك، ولا تغادر أي بيانات جهازك.", + "whisperHint": "يستخدم التفريغ النصي Whisper محليًا — يعمل على حاسوبك، ولا تغادر أي بيانات جهازك.", "transcribeNow": "فرّغ النص الآن", "transcribing": "جارٍ التفريغ…", "clipLabel": "المقطع {{index}}", diff --git a/src/i18n/locales/en/settings.json b/src/i18n/locales/en/settings.json index 19acd784f..8ebfb79ed 100644 --- a/src/i18n/locales/en/settings.json +++ b/src/i18n/locales/en/settings.json @@ -245,7 +245,7 @@ "help": "Aggregated transcript of every clip on the timeline. Backspace / Delete a word or selection to mark it as skipped (red). Hover a red span to restore it.", "noClips": "No clips yet", "noTranscript": "No transcript yet", - "whisperHint": "Transcribe uses local Whisper — runs in your browser, no data leaves the device.", + "whisperHint": "Transcribe uses local Whisper — runs on your computer, no data leaves the device.", "transcribeNow": "Transcribe now", "transcribing": "Transcribing…", "clipLabel": "Clip {{index}}", diff --git a/src/i18n/locales/es/settings.json b/src/i18n/locales/es/settings.json index c0a5a113c..68bd83ec0 100644 --- a/src/i18n/locales/es/settings.json +++ b/src/i18n/locales/es/settings.json @@ -245,7 +245,7 @@ "help": "Transcripción agregada de todos los clips de la línea de tiempo. Retroceso / Supr sobre una palabra o selección la marca como omitida (en rojo). Pasa el cursor sobre un fragmento rojo para restaurarlo.", "noClips": "Aún no hay clips", "noTranscript": "Aún no hay transcripción", - "whisperHint": "La transcripción usa Whisper local: se ejecuta en tu navegador y ningún dato sale del dispositivo.", + "whisperHint": "La transcripción usa Whisper local: se ejecuta en tu equipo y ningún dato sale del dispositivo.", "transcribeNow": "Transcribir ahora", "transcribing": "Transcribiendo…", "clipLabel": "Clip {{index}}", diff --git a/src/i18n/locales/fr/settings.json b/src/i18n/locales/fr/settings.json index e78e96a83..13f650f5f 100644 --- a/src/i18n/locales/fr/settings.json +++ b/src/i18n/locales/fr/settings.json @@ -245,7 +245,7 @@ "help": "Transcription agrégée de tous les clips de la timeline. Retour arrière / Suppr sur un mot ou une sélection le marque comme ignoré (en rouge). Survolez un passage rouge pour le restaurer.", "noClips": "Aucun clip pour l'instant", "noTranscript": "Aucune transcription pour l'instant", - "whisperHint": "La transcription utilise Whisper en local — tout s'exécute dans votre navigateur, aucune donnée ne quitte l'appareil.", + "whisperHint": "La transcription utilise Whisper en local — tout s'exécute sur votre ordinateur, aucune donnée ne quitte l'appareil.", "transcribeNow": "Transcrire maintenant", "transcribing": "Transcription…", "clipLabel": "Clip {{index}}", diff --git a/src/i18n/locales/it/settings.json b/src/i18n/locales/it/settings.json index 6361208a3..9f6f82393 100644 --- a/src/i18n/locales/it/settings.json +++ b/src/i18n/locales/it/settings.json @@ -245,7 +245,7 @@ "help": "Trascrizione aggregata di tutti i clip sulla timeline. Backspace / Canc su una parola o selezione la segna come saltata (in rosso). Passa sopra un tratto rosso per ripristinarlo.", "noClips": "Ancora nessun clip", "noTranscript": "Ancora nessuna trascrizione", - "whisperHint": "La trascrizione usa Whisper in locale: gira nel tuo browser, nessun dato lascia il dispositivo.", + "whisperHint": "La trascrizione usa Whisper in locale: gira sul tuo computer, nessun dato lascia il dispositivo.", "transcribeNow": "Trascrivi ora", "transcribing": "Trascrizione…", "clipLabel": "Clip {{index}}", diff --git a/src/i18n/locales/ja-JP/settings.json b/src/i18n/locales/ja-JP/settings.json index 2317b864c..e7e4bb5dc 100644 --- a/src/i18n/locales/ja-JP/settings.json +++ b/src/i18n/locales/ja-JP/settings.json @@ -245,7 +245,7 @@ "help": "タイムライン上のすべてのクリップをまとめた文字起こしです。単語や選択範囲を Backspace / Delete でスキップ（赤色）にできます。赤い部分にカーソルを合わせると元に戻せます。", "noClips": "クリップがまだありません", "noTranscript": "文字起こしがまだありません", - "whisperHint": "文字起こしはローカルの Whisper を使用します。ブラウザ内で実行され、データが端末外に出ることはありません。", + "whisperHint": "文字起こしはローカルの Whisper を使用します。お使いのパソコン上で実行され、データが端末外に出ることはありません。", "transcribeNow": "今すぐ文字起こし", "transcribing": "文字起こし中…", "clipLabel": "クリップ {{index}}", diff --git a/src/i18n/locales/ko-KR/settings.json b/src/i18n/locales/ko-KR/settings.json index 21b064645..4e1d23ca9 100644 --- a/src/i18n/locales/ko-KR/settings.json +++ b/src/i18n/locales/ko-KR/settings.json @@ -245,7 +245,7 @@ "help": "타임라인의 모든 클립을 합친 전사입니다. 단어나 선택 영역에서 Backspace / Delete를 누르면 건너뛴 것으로 표시됩니다(빨간색). 빨간 부분에 마우스를 올리면 복원할 수 있습니다.", "noClips": "아직 클립이 없습니다", "noTranscript": "아직 전사가 없습니다", - "whisperHint": "전사는 로컬 Whisper를 사용합니다. 브라우저에서 실행되며 데이터가 기기를 벗어나지 않습니다.", + "whisperHint": "전사는 로컬 Whisper를 사용합니다. 이 컴퓨터에서 실행되며 데이터가 기기를 벗어나지 않습니다.", "transcribeNow": "지금 전사하기", "transcribing": "전사 중…", "clipLabel": "클립 {{index}}", diff --git a/src/i18n/locales/pt-BR/settings.json b/src/i18n/locales/pt-BR/settings.json index 080b45edf..95f63653e 100644 --- a/src/i18n/locales/pt-BR/settings.json +++ b/src/i18n/locales/pt-BR/settings.json @@ -245,7 +245,7 @@ "help": "Transcrição agregada de todos os clipes da linha do tempo. Backspace / Delete em uma palavra ou seleção marca como ignorada (em vermelho). Passe o mouse sobre um trecho vermelho para restaurá-lo.", "noClips": "Nenhum clipe ainda", "noTranscript": "Nenhuma transcrição ainda", - "whisperHint": "A transcrição usa o Whisper local — roda no seu navegador, nenhum dado sai do dispositivo.", + "whisperHint": "A transcrição usa o Whisper local — roda no seu computador, nenhum dado sai do dispositivo.", "transcribeNow": "Transcrever agora", "transcribing": "Transcrevendo…", "clipLabel": "Clipe {{index}}", diff --git a/src/i18n/locales/ru/settings.json b/src/i18n/locales/ru/settings.json index aa3d0209b..ccec01ca2 100644 --- a/src/i18n/locales/ru/settings.json +++ b/src/i18n/locales/ru/settings.json @@ -245,7 +245,7 @@ "help": "Сводная расшифровка всех клипов на таймлайне. Backspace / Delete по слову или выделению помечает его как пропущенное (красным). Наведите курсор на красный фрагмент, чтобы вернуть его.", "noClips": "Клипов пока нет", "noTranscript": "Расшифровки пока нет", - "whisperHint": "Расшифровка использует локальный Whisper — работает в вашем браузере, данные не покидают устройство.", + "whisperHint": "Расшифровка использует локальный Whisper — работает на вашем компьютере, данные не покидают устройство.", "transcribeNow": "Расшифровать сейчас", "transcribing": "Расшифровка…", "clipLabel": "Клип {{index}}", diff --git a/src/i18n/locales/tr/settings.json b/src/i18n/locales/tr/settings.json index 2f23d40e4..d81852dd4 100644 --- a/src/i18n/locales/tr/settings.json +++ b/src/i18n/locales/tr/settings.json @@ -245,7 +245,7 @@ "help": "Zaman çizelgesindeki tüm kliplerin birleşik dökümü. Bir kelimede veya seçimde Backspace / Delete tuşuna basmak onu atlanmış (kırmızı) olarak işaretler. Kırmızı bölümün üzerine gelerek geri alabilirsiniz.", "noClips": "Henüz klip yok", "noTranscript": "Henüz döküm yok", - "whisperHint": "Döküm yerel Whisper kullanır — tarayıcınızda çalışır, hiçbir veri cihazdan çıkmaz.", + "whisperHint": "Döküm yerel Whisper kullanır — bilgisayarınızda çalışır, hiçbir veri cihazdan çıkmaz.", "transcribeNow": "Şimdi dökümünü çıkar", "transcribing": "Döküm çıkarılıyor…", "clipLabel": "Klip {{index}}", diff --git a/src/i18n/locales/vi/settings.json b/src/i18n/locales/vi/settings.json index f7c47b113..c2fc75d7b 100644 --- a/src/i18n/locales/vi/settings.json +++ b/src/i18n/locales/vi/settings.json @@ -245,7 +245,7 @@ "help": "Bản chép lời gộp của mọi clip trên dòng thời gian. Nhấn Backspace / Delete trên một từ hoặc vùng chọn để đánh dấu là bỏ qua (màu đỏ). Di chuột lên đoạn màu đỏ để khôi phục.", "noClips": "Chưa có clip nào", "noTranscript": "Chưa có bản chép lời", - "whisperHint": "Việc chép lời dùng Whisper cục bộ — chạy ngay trong trình duyệt, không dữ liệu nào rời khỏi thiết bị.", + "whisperHint": "Việc chép lời dùng Whisper cục bộ — chạy ngay trên máy của bạn, không dữ liệu nào rời khỏi thiết bị.", "transcribeNow": "Chép lời ngay", "transcribing": "Đang chép lời…", "clipLabel": "Clip {{index}}", diff --git a/src/i18n/locales/zh-CN/settings.json b/src/i18n/locales/zh-CN/settings.json index 536db980e..1fd9f2a83 100644 --- a/src/i18n/locales/zh-CN/settings.json +++ b/src/i18n/locales/zh-CN/settings.json @@ -245,7 +245,7 @@ "help": "时间轴上所有片段的合并转录。对某个词或选区按退格 / Delete 会将其标记为跳过（红色）。将鼠标悬停在红色片段上可恢复。", "noClips": "暂无片段", "noTranscript": "暂无转录", - "whisperHint": "转录使用本地 Whisper — 在您的浏览器中运行，数据不会离开本设备。", + "whisperHint": "转录使用本地 Whisper — 在您的电脑上运行，数据不会离开本设备。", "transcribeNow": "立即转录", "transcribing": "转录中…", "clipLabel": "片段 {{index}}", diff --git a/src/i18n/locales/zh-TW/settings.json b/src/i18n/locales/zh-TW/settings.json index 8483874ac..1210a1db0 100644 --- a/src/i18n/locales/zh-TW/settings.json +++ b/src/i18n/locales/zh-TW/settings.json @@ -246,7 +246,7 @@ "help": "時間軸上所有片段的合併逐字稿。對某個字或選取範圍按 Backspace / Delete 會標記為略過（紅色）。將滑鼠移到紅色片段上即可還原。", "noClips": "尚無片段", "noTranscript": "尚無逐字稿", - "whisperHint": "逐字稿使用本機 Whisper — 在您的瀏覽器中執行，資料不會離開本裝置。", + "whisperHint": "逐字稿使用本機 Whisper — 在您的電腦上執行，資料不會離開本裝置。", "transcribeNow": "立即產生逐字稿", "transcribing": "轉錄中…", "clipLabel": "片段 {{index}}", From 2092dace8a4a2eb665b79940d8f137448d6702e2 Mon Sep 17 00:00:00 2001 From: m1 Date: Thu, 30 Jul 2026 15:10:31 +0200 Subject: [PATCH 4/7] =?UTF-8?q?fix(macos):=20don't=20content-protect=20win?= =?UTF-8?q?dows=20on=20macOS=2026=20=E2=80=94=20it=20makes=20them=20invisi?= =?UTF-8?q?ble?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On macOS 26, a window that has had setContentProtection(true) applied is never displayed. Not "excluded from screen captures", which is the documented behaviour and the reason we call it — never painted for the user either. The app started with a tray icon, a healthy renderer and no window on screen, and both the tray click and Cmd+Shift+O were inert because Electron already considered the window visible. Confirmed on macOS 26.5 / Electron 41.2.1 against the HUD: ready-to-show fired, show() was called, geometry was on-screen on the main display, React had mounted with no console errors. Unsetting protection makes it appear instantly. Reordering protect-vs-show changes nothing, so it is the call itself, not the timing. setContentProtection maps to NSWindow.sharingType = NSWindowSharingNone, a path Electron has churned on repeatedly (electron/electron#45990; PR #46886 reverted a macOS content-protection refactor). Gated on the macOS major version, not darwin wholesale: the breakage is only confirmed on 26.x, and dropping protection on older macOS where it may work would be a privacy regression made on no evidence. The check reads process.getSystemVersion() (macOS "26.5.0"), NOT os.release() (Darwin "25.5.0") — feeding it the latter would silently never match. OPENSCREEN_FORCE_CONTENT_PROTECTION=1 re-enables it to re-test against a future Electron. KNOWN TRADE-OFF: the HUD and Notes windows can now appear in captures on macOS 26. Apple already made that partly unavoidable — ScreenCaptureKit ignores sharingType, so any SCK-based recorder captures them regardless, including our own helper. The durable fix is to exclude our windows via SCContentFilter's excludingWindows:, which electron/native/screencapturekit currently passes as []. Not done here: it needs a recording to verify, which needs screen tooling I don't have in this session. --- AGENTS.md | 1 + electron/windows.ts | 52 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index b4b873979..1f1a63d40 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,6 +60,7 @@ Unit/browser tests can't exercise real capture (native screen recording, a physi **The HUD widget** (recording controller) - **It is invisible in screenshots by default.** The HUD (and the Notes window) call `setContentProtection(true)` so the recording controls never end up baked into a recording — the same `SetWindowDisplayAffinity` that WGC honours also hides them from *your* screenshots. The window is there, and clicks land, but you are aiming blind at a rectangle you cannot see. Set **`OPENSCREEN_DISABLE_CONTENT_PROTECTION=1`** in the app's environment to turn it off for a session; every skipped window logs a warning. Unset it before recording anything real, or the HUD ends up in the video. +- **On macOS 26+ content protection is auto-disabled, so the HUD *is* visible and screenshottable with no flag.** That OS never displays a content-protected window at all — not just absent from captures, but never painted, leaving a tray icon, a live renderer and nothing on screen (confirmed on macOS 26.5 / Electron 41.2.1). `applyContentProtection` therefore skips the call there and logs a warning per window; the trade-off is that the HUD can appear in recordings on that OS until the ScreenCaptureKit helper excludes our own windows via `SCContentFilter(excludingWindows:)`, which it currently passes as `[]`. `OPENSCREEN_FORCE_CONTENT_PROTECTION=1` re-enables it to re-test against a future Electron. - The HUD is what opens the editor (clapper icon, tooltip *Open Studio*), so without that flag a whole slice of the app is unreachable from automation: killing the app to redeploy a native addon leaves you unable to reopen a project. - Frameless, transparent, always-on-top, `skipTaskbar`, centered at the **bottom of the primary display** (`createHudOverlayWindow`, 600×160). It is **click-through** (`setIgnoreMouseEvents(true, { forward: true })`): moving the real cursor over an interactive control makes that region clickable and shows its tooltip, so `mouse_move` → screenshot → `left_click` works; a blind click on empty HUD area passes through to the desktop. - Control row (left→right): layout preset, **source** button (`Screen`/`Window` → label becomes the picked source), system-audio toggle, mic toggle, **webcam toggle** (shows the detected camera name), cursor-highlight toggle, **record**, notes, open-editor, language, minimize, close. The record button is disabled until a source is chosen (tooltip: "Please select a source to record"). diff --git a/electron/windows.ts b/electron/windows.ts index 129eb8757..32046fe36 100644 --- a/electron/windows.ts +++ b/electron/windows.ts @@ -19,6 +19,49 @@ const HEADLESS = process.env["HEADLESS"] === "true"; // while it is set WILL contain the HUD. const CONTENT_PROTECTION_DISABLED = process.env["OPENSCREEN_DISABLE_CONTENT_PROTECTION"] === "1"; +// Forces protection back on where it is auto-disabled below, so the macOS +// regression can be re-tested against a future Electron without editing code. +const CONTENT_PROTECTION_FORCED = process.env["OPENSCREEN_FORCE_CONTENT_PROTECTION"] === "1"; + +/** + * macOS 26 (Darwin 25) never displays a window that has had + * `setContentProtection(true)` applied to it. + * + * Not "excludes it from captures" — which is the documented behaviour and the + * whole point — but never paints it for the user either. Confirmed on Darwin + * 25.5 / Electron 41.2.1 with the HUD: tray icon present, renderer alive and + * painting (React mounted, no console errors), `ready-to-show` fired, `show()` + * called, window at valid on-screen coordinates on the main display — and + * nothing on screen. `showMainWindow()` from the tray and the global shortcut + * were equally inert, because the window was already "visible" as far as + * Electron was concerned. Unsetting protection makes it appear instantly; the + * ordering of protect-vs-show makes no difference, so it is the call itself. + * + * `setContentProtection` maps to `NSWindow.sharingType = NSWindowSharingNone`, + * a path Electron has repeatedly churned on (electron/electron#45990, and + * PR #46886 reverting a macOS content-protection refactor). + * + * Gated on the Darwin major version rather than `darwin` wholesale: the + * breakage is only *confirmed* on 25.x, and silently dropping protection on + * older macOS — where it may well work — would be a privacy regression made on + * no evidence. + * + * NOTE: this leaves the HUD capturable on macOS 26. Apple already made that + * partly true regardless — ScreenCaptureKit ignores `sharingType`, so any + * SCK-based recorder (including *ours*, see + * `electron/native/screencapturekit/`) captures these windows anyway. The + * durable fix is to exclude our own windows via `SCContentFilter`'s + * `excludingWindows:`, which that helper currently passes as `[]`. + */ +const CONTENT_PROTECTION_BREAKS_DISPLAY = (() => { + if (process.platform !== "darwin") return false; + // getSystemVersion() reports the *macOS* version on darwin, not the Darwin + // kernel version: measured "26.5.0" here where `os.release()` is "25.5.0". + // So the affected major is 26, and the check must not be fed os.release(). + const macOSMajor = Number.parseInt(process.getSystemVersion().split(".")[0] ?? "", 10); + return Number.isFinite(macOSMajor) && macOSMajor >= 26; +})(); + function applyContentProtection(win: BrowserWindow, label: string) { if (CONTENT_PROTECTION_DISABLED) { console.warn( @@ -28,6 +71,15 @@ function applyContentProtection(win: BrowserWindow, label: string) { ); return; } + if (CONTENT_PROTECTION_BREAKS_DISPLAY && !CONTENT_PROTECTION_FORCED) { + console.warn( + `[content-protection] OFF for the ${label} window — macOS ` + + `${process.getSystemVersion()} never displays a content-protected window, so ` + + "enabling it would make this window permanently invisible. It may therefore appear " + + "in screen captures. Set OPENSCREEN_FORCE_CONTENT_PROTECTION=1 to re-test.", + ); + return; + } win.setContentProtection(true); } From e0c21b9c4640a51e9596aa88662f23ea42ae6ea6 Mon Sep 17 00:00:00 2001 From: m1 Date: Thu, 30 Jul 2026 15:44:19 +0200 Subject: [PATCH 5/7] test(stt): split the word-timing assertions, and stop failing on raw inverted ranges MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified against a real 24.7s French screen recording (WER-free run: 10 segments, 57 words, fr detected, rtf 0.076 on M1/Metal). Two problems with the single bundled `every()`: - It could not say which property broke. The real clip tripped it and the output read "word times lie within the clip — clip is 24.70s" when nothing was outside the clip: max word end was 24.00s. The actual trigger was `end < start` on three tokens. - Asserting `end >= start` on the raw helper response is wrong for this layer. whisper.cpp gives a segment's last word the segment's own t1 as its end, while the word's DTW start runs 80-150ms late, so a token emitted near the boundary (standalone punctuation: "?", "!", "!") lands after it and inverts. That is anticipated and handled one layer up, on purpose — whisperServer.ts clamps to Math.max(startSec + 0.02, endSec), and snapWordBoundaries.ts keeps "degenerate words (whisper sometimes reports end <= start) non-empty". The document never sees an inverted range. Now three separate checks (starts non-negative, ends within clip, starts monotonic) plus an informational count of raw inversions. --- scripts/test-whisper-stt.mjs | 40 +++++++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/scripts/test-whisper-stt.mjs b/scripts/test-whisper-stt.mjs index 702d06cef..7e52bac2b 100644 --- a/scripts/test-whisper-stt.mjs +++ b/scripts/test-whisper-stt.mjs @@ -300,17 +300,43 @@ async function main() { ); // Contract invariant: absolute seconds in the source recording, [0, duration). + // + // These are checked separately rather than as one `every()`. Bundled, a + // failure could not say which property broke: a real French clip tripped the + // combined check and read as "word times outside the clip" when nothing was + // outside the clip at all — three punctuation tokens had end < start. const tolerance = 0.5; // whisper rounds segment ends up to the chunk edge - const inRange = words.every( - (w) => w.start >= 0 && w.end >= w.start && w.end <= meta.durationSec + tolerance, + check( + words.every((w) => w.start >= 0), + "word starts are non-negative", + ); + check( + words.every((w) => w.end <= meta.durationSec + tolerance), + "word ends lie within the clip", + `clip is ${meta.durationSec.toFixed(2)}s`, + ); + check( + words.every((w, i) => i === 0 || w.start >= words[i - 1].start), + "word starts are monotonic non-decreasing", ); - check(inRange, "word times lie within the clip", `clip is ${meta.durationSec.toFixed(2)}s`); - let monotonic = true; - for (let i = 1; i < words.length; i++) { - if (words[i].start < words[i - 1].start) monotonic = false; + // NOT a failure. whisper.cpp gives the last word of a segment the segment's + // own t1 as its end, while the word's DTW start runs 80–150 ms late — so a + // token emitted near the boundary (typically standalone punctuation) can land + // after it and invert. The contract's [startSec, endSec) guarantee is enforced + // one layer up, deliberately: whisperServer.ts clamps to + // `Math.max(startSec + 0.02, endSec)` and snapWordBoundaries.ts keeps + // "degenerate words (whisper sometimes reports end <= start) non-empty". This + // harness speaks to the raw helper, below that clamp, so it reports the count + // as information instead of asserting on it. + const inverted = words.filter((w) => w.end < w.start); + if (inverted.length > 0) { + console.log( + ` info ${inverted.length}/${words.length} raw word(s) have end < start ` + + `(${inverted.map((w) => JSON.stringify(w.word)).join(", ")}) — expected at segment ` + + "boundaries; whisperServer.ts clamps these before they reach the document.", + ); } - check(monotonic, "word starts are monotonic non-decreasing"); if (refText) { const rate = wer(refText, text); From de07e32dd0c213933253d874d19348bab66921c2 Mon Sep 17 00:00:00 2001 From: m1 Date: Thu, 30 Jul 2026 16:24:40 +0200 Subject: [PATCH 6/7] build(macos): strip the build-machine path from the shipped compositor addon MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit cargo stamps the cdylib's own id (LC_ID_DYLIB) with the absolute path it was built at — crates/target/release/deps/libcompositor_view.dylib. That is not a load failure: require() dlopens the .node by path and never reads LC_ID_DYLIB, and every real dependency was already rewritten to @rpath. But it ships a build-machine path inside a release artefact, complete with the builder's home directory and checkout name, which defeats reproducible builds and leaks a filesystem layout to anyone running `otool -D` on the installed app. Set it to @rpath/compositor_view.node and re-sign. The existing post-condition could not have caught it: it grepped otool -L output for `lib(av|sw)` only, so it saw neither the id nor any future non-ffmpeg dependency arriving absolute. Replaced with the real invariant — no reference, id or dependency, may be an absolute path outside /usr/lib and /System — and it now runs over the vendored dylibs too, not just the addon. Verified: shipped copy's id is @rpath/compositor_view.node with a valid signature; the dev copy under electron/native/compositor-view/build/ still carries its absolute id, which the new check correctly identifies as the value it would reject (that copy never leaves the checkout, by design). --- scripts/build-macos-compositor-addon.mjs | 45 ++++++++++++++++++++---- 1 file changed, 39 insertions(+), 6 deletions(-) diff --git a/scripts/build-macos-compositor-addon.mjs b/scripts/build-macos-compositor-addon.mjs index a66bb2803..200b9cd87 100644 --- a/scripts/build-macos-compositor-addon.mjs +++ b/scripts/build-macos-compositor-addon.mjs @@ -190,14 +190,47 @@ function vendorFfmpegDylibs(nodePath, ffmpegDir) { execFileSync("codesign", ["--force", "--sign", "-", target]); } - const remaining = execFileSync("otool", ["-L", nodePath], { encoding: "utf8" }) - .split("\n") - .map((l) => l.trim().split(" ")[0]) - .filter((p) => p.startsWith("/") && /lib(av|sw)/.test(p)); - if (remaining.length > 0) { - throw new Error(`Still absolute after rewriting: ${remaining.join(", ")}`); + // cargo stamps the cdylib's own id with the absolute path it was built at + // (`crates/target/release/deps/libcompositor_view.dylib`). Nothing resolves + // through it — `require()` dlopens the file by path and never reads + // LC_ID_DYLIB — so this is not a load failure. It is a build-machine path, + // complete with the builder's home directory and checkout name, shipped + // inside a release artefact: it defeats reproducible builds and leaks a + // filesystem layout to anyone who runs `otool -D` on the installed app. + execFileSync("install_name_tool", ["-id", `@rpath/${path.basename(nodePath)}`, nodePath]); + execFileSync("codesign", ["--force", "--sign", "-", nodePath]); + + // The old check only grepped for `lib(av|sw)`, so it could not have caught + // the id above — nor any future non-ffmpeg dependency that arrives absolute. + // Assert the real invariant instead: nothing outside the OS's own prefixes + // may be referenced by absolute path. + const absolutePaths = (file) => { + const id = execFileSync("otool", ["-D", file], { encoding: "utf8" }) + .split("\n") + .slice(1) // first line is the filename echoed back + .map((l) => l.trim()) + .filter(Boolean); + const deps = execFileSync("otool", ["-L", file], { encoding: "utf8" }) + .split("\n") + .slice(1) // ditto + .map((l) => l.trim().split(" ")[0]) + .filter(Boolean); + return [...id, ...deps].filter( + (p) => p.startsWith("/") && !p.startsWith("/usr/lib/") && !p.startsWith("/System/"), + ); + }; + + for (const file of [nodePath, ...names.map((n) => path.join(outDir, n))]) { + const remaining = absolutePaths(file); + if (remaining.length > 0) { + throw new Error( + `${path.basename(file)} still references build-machine paths after rewriting: ` + + remaining.join(", "), + ); + } } console.log(`Vendored ${names.length} ffmpeg dylibs next to ${path.basename(nodePath)}`); + console.log(`No absolute build-machine paths remain in ${path.basename(nodePath)} or its dylibs`); } /** From 47a523acb9084052fbeefa5303208fa1f55de720 Mon Sep 17 00:00:00 2001 From: m1 Date: Thu, 30 Jul 2026 17:06:19 +0200 Subject: [PATCH 7/7] build(macos): package the compositor addon, and refuse an incomplete .app MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The macOS CI job built the ScreenCaptureKit helper and staged the whisper-stt binaries, but never ran fetch:ffmpeg:mac or build:native:compositor:mac. So the .app it produced had no compositor_view.node at all: preview and export dead in the installed app, with nothing in any log to say why. Windows does not drift this way because its job just runs `npm run build:win`, which chains both; the macOS job spells its steps out (it needs --dir plus a hand-rolled DMG and signing) and fell behind the `build:mac` recipe it was supposed to mirror. Nothing caught it, because before-pack.cjs — the hook written precisely to stop a broken compositor shipping — returned early on every non-win32 platform. Two changes: - build.yml gains the two missing steps, both cached: the LGPL ffmpeg tree (built from source, ~5 min, keyed on the fetch script so the cache busts when the pin moves) and the Metal addon. - before-pack.cjs now runs on darwin. It asserts the *shipped* payload in electron/native/bin/darwin-/ — the directory mac.extraResources copies wholesale — contains the compositor addon, the ffmpeg dylibs it links, the whisper-stt helper, the ggml dylibs that helper links, and the ScreenCaptureKit helper, naming what breaks for each. It then applies the existing staleness check to the arch-tagged addon rather than the dev copy, since the arch-tagged one is what actually gets packaged. Verified both directions: passes on a complete tree; with compositor_view.node removed it names the addon, with libggml*.dylib removed it names the ggml dylibs, exit 1 both times. build.yml re-parses as valid YAML and the macOS job's step order is unchanged apart from the insertions. NOT touched: the job's `if: false`. Its comment marks it release-branch-only for 1.8.0 shipping Windows-only, and says explicitly not to let it reach main. So these fixes make the job correct but do not make it run — that is a release decision, and it is now flagged in build-and-packaging.md. --- .github/workflows/build.yml | 34 +++++ scripts/before-pack.cjs | 137 ++++++++++++++++-- .../engineering/build-and-packaging.md | 26 +++- 3 files changed, 182 insertions(+), 15 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index dc3249c91..923a5c435 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -169,6 +169,40 @@ jobs: env: OPENSCREEN_MAC_HELPER_ARCHS: ${{ matrix.arch }} + # The two steps below are what `npm run build:mac` does and this job did not. + # Windows gets them for free because its job just runs `npm run build:win`, + # which chains fetch:ffmpeg + build:native:compositor; macOS spells its steps + # out (it needs `--dir` plus a hand-rolled DMG and signing) and drifted. The + # result was a .app with no compositor addon — preview and export dead in the + # installed app, silently. `scripts/before-pack.cjs` now refuses to package + # that, so this is also what keeps the job from failing at the pack step. + - name: Cache LGPL ffmpeg tree + uses: actions/cache@v4 + with: + # fetch-ffmpeg-macos.mjs BUILDS ffmpeg from source (~5 min): BtbN ships no + # macOS target and every circulating macOS build is GPL, which would + # relicense this MIT app. The script pins the release and checksums it, so + # keying on the script itself busts the cache when the pin moves. + path: crates/thirdparty + key: ffmpeg-macos-${{ matrix.arch }}-${{ hashFiles('scripts/fetch-ffmpeg-macos.mjs') }} + + - name: Vendor LGPL ffmpeg + run: npm run fetch:ffmpeg:mac + + - name: Cache cargo + compositor build tree + uses: actions/cache@v4 + with: + path: | + ~/.cargo/registry + ~/.cargo/git + crates/target + key: cargo-macos-${{ matrix.arch }}-${{ hashFiles('crates/Cargo.lock') }} + restore-keys: | + cargo-macos-${{ matrix.arch }}- + + - name: Build Metal compositor addon + run: npm run build:native:compositor:mac + - name: Package .app bundle run: npx electron-builder --mac --${{ matrix.arch }} --dir --publish never env: diff --git a/scripts/before-pack.cjs b/scripts/before-pack.cjs index e64ba780e..c38cea87f 100644 --- a/scripts/before-pack.cjs +++ b/scripts/before-pack.cjs @@ -38,6 +38,99 @@ const SOURCE_PATHS = [ const FIX = "Rebuild it with:\n\n npm run build:native:compositor\n\nor use `npm run build:win`, which does that for you."; +const FIX_MAC = + "Rebuild it with:\n\n npm run build:native:compositor:mac\n\nor use `npm run build:mac`, which does that for you."; + +/** + * Everything that has to be inside `electron/native/bin/darwin-/` for the .app to + * work, keyed by what breaks when it is absent. + * + * This list exists because the macOS deliverable had no guard at all: `beforePack` + * returned early on any non-win32 platform, so a mac package built without the compositor + * addon shipped silently — the preview and the export come up dead, with nothing in any + * log to say why. That is precisely the failure mode the staleness check below was written + * to prevent, and the platform guard was letting it through on the other OS. + * + * `mac.extraResources` ships this directory wholesale (`filter: ["darwin-*​/*"]`), so + * "present here" is the same thing as "present in the installed app". + */ +const MAC_REQUIRED = [ + { + match: (name) => name === "compositor_view.node", + what: "the Metal compositor addon", + breaks: "the preview and every export render nothing", + fix: FIX_MAC, + }, + { + match: (name) => /^libav(codec|format|util)\.\d+\.dylib$/.test(name), + what: "the LGPL ffmpeg dylibs the compositor links", + breaks: "the compositor addon cannot be loaded at all (dyld error at require())", + fix: FIX_MAC, + atLeast: 3, + }, + { + match: (name) => name === "whisper-stt-server", + what: "the whisper.cpp STT helper", + breaks: "transcription and captions fail with a developer error shown to end users", + fix: "Build it with:\n\n npm run build:whisper-binaries\n\nor stage CI's with `bash scripts/stage-whisper-stt.sh darwin-`.", + }, + { + match: (name) => /^libggml.*\.dylib$/.test(name), + what: "the ggml backend dylibs the STT helper links", + breaks: "whisper-stt-server dies in dyld before main(), so STT times out with no diagnostic", + fix: "Build it with:\n\n npm run build:whisper-binaries", + atLeast: 1, + }, + { + match: (name) => name === "openscreen-screencapturekit-helper", + what: "the ScreenCaptureKit capture helper", + breaks: "native screen capture is unavailable", + fix: "Build it with:\n\n npm run build:native:mac", + }, +]; + +/** electron-builder passes `context.arch` as a numeric enum; map it to our directory tag. */ +function archTagFor(context) { + const BY_INDEX = { 0: "ia32", 1: "x64", 2: "armv7l", 3: "arm64", 4: "universal" }; + const name = BY_INDEX[context?.arch]; + return name && name !== "universal" ? name : process.arch; +} + +function checkMacNativePayload(context) { + const tag = `darwin-${archTagFor(context)}`; + const dir = path.join(ROOT, "electron", "native", "bin", tag); + if (!fs.existsSync(dir)) { + throw new Error( + `Refusing to package: ${path.relative(ROOT, dir)} does not exist, so the .app would ` + + "ship with no native modules at all.\n\n" + + `${FIX_MAC}\n\nThe STT helper and the capture helper are separate builds — see\n` + + "technical-documentation/engineering/build-and-packaging.md.", + ); + } + + const present = fs.readdirSync(dir); + const missing = MAC_REQUIRED.filter( + (req) => present.filter((name) => req.match(name)).length < (req.atLeast ?? 1), + ); + if (missing.length === 0) { + return; + } + + const detail = missing + .map( + (req) => + ` - ${req.what}\n without it: ${req.breaks}\n ${req.fix.replace(/\n+/g, " ")}`, + ) + .join("\n"); + throw new Error( + `Refusing to package an incomplete macOS payload.\n\n` + + ` looked in: ${path.relative(ROOT, dir)}\n\n` + + `Missing:\n${detail}\n\n` + + "Every one of these fails silently or as an unactionable timeout in the installed\n" + + "app, which is why this is a hard error at pack time rather than a warning.", + ); +} + /** Newest mtime under `target` (file or directory), or 0 if it does not exist. */ function newestMtimeMs(target) { let stat; @@ -56,14 +149,14 @@ function newestMtimeMs(target) { return newest; } -function checkCompositorAddonFreshness() { - if (!fs.existsSync(ADDON)) { +function checkCompositorAddonFreshness(addon = ADDON, fix = FIX, label = "D3D11") { + if (!fs.existsSync(addon)) { throw new Error( - `Refusing to package: the D3D11 compositor addon is missing.\n\n expected: ${ADDON}\n\n${FIX}`, + `Refusing to package: the ${label} compositor addon is missing.\n\n expected: ${addon}\n\n${fix}`, ); } - const addonMs = fs.statSync(ADDON).mtimeMs; + const addonMs = fs.statSync(addon).mtimeMs; const stale = SOURCE_PATHS.map((source) => ({ source, ms: newestMtimeMs(source) })).filter( (entry) => entry.ms > addonMs, ); @@ -73,29 +166,51 @@ function checkCompositorAddonFreshness() { const newest = stale.reduce((a, b) => (a.ms > b.ms ? a : b)); throw new Error( - "Refusing to package a stale D3D11 compositor addon.\n\n" + + `Refusing to package a stale ${label} compositor addon.\n\n` + + ` addon: ${path.relative(ROOT, addon)}\n` + ` addon built: ${new Date(addonMs).toISOString()}\n` + ` newer source: ${path.relative(ROOT, newest.source)} (${new Date(newest.ms).toISOString()})\n\n` + "Packaging this would silently ship an addon that ignores newer scene fields\n" + "(they are #[serde(default)], so it falls back instead of erroring).\n\n" + - FIX, + fix, ); } exports.default = async function beforePack(context) { - // The compositor addon is Windows-only; skip when packing any other target. const platform = context?.electronPlatformName ?? process.platform; - if (platform !== "win32") { + if (platform === "win32") { + checkCompositorAddonFreshness(); return; } - checkCompositorAddonFreshness(); + if (platform === "darwin") { + // The addon that actually ships is the arch-tagged copy under + // electron/native/bin/ (mac.extraResources), not the dev copy this hook used to + // be the sole guardian of — so that is the one whose freshness matters. + const tag = `darwin-${archTagFor(context)}`; + const shipped = path.join(ROOT, "electron", "native", "bin", tag, "compositor_view.node"); + checkMacNativePayload(context); + checkCompositorAddonFreshness(shipped, FIX_MAC, "Metal"); + return; + } + // Linux ships no native addon of its own; nothing to assert. }; // Runnable on its own for debugging: `node scripts/before-pack.cjs` if (require.main === module) { try { - checkCompositorAddonFreshness(); - console.log("compositor addon is up to date with its Rust sources."); + if (process.platform === "darwin") { + checkMacNativePayload({ arch: undefined }); + const tag = `darwin-${process.arch}`; + checkCompositorAddonFreshness( + path.join(ROOT, "electron", "native", "bin", tag, "compositor_view.node"), + FIX_MAC, + "Metal", + ); + console.log(`macOS native payload complete in electron/native/bin/${tag}, addon up to date.`); + } else { + checkCompositorAddonFreshness(); + console.log("compositor addon is up to date with its Rust sources."); + } } catch (err) { console.error(err.message); process.exit(1); diff --git a/technical-documentation/engineering/build-and-packaging.md b/technical-documentation/engineering/build-and-packaging.md index fb45dc8be..c1e623ed5 100644 --- a/technical-documentation/engineering/build-and-packaging.md +++ b/technical-documentation/engineering/build-and-packaging.md @@ -33,7 +33,8 @@ A usable full package depends on generated artifacts that are not committed: | macOS ScreenCaptureKit capture helper and cursor helper | `electron/native/bin/darwin-arm64/` or `darwin-x64/` | Full Xcode, Swift, SwiftPM; Command Line Tools alone may be insufficient | | Whisper STT server and ggml/whisper backend libraries | `electron/native/bin/-/` | CMake plus host compiler; Metal on Apple Silicon, Vulkan SDK on supported Windows/Linux builds, CPU fallback, optional CUDA | | Native D3D11 compositor addon | `electron/native/compositor-view/build/compositor_view.node` | Rust MSVC toolchain, Visual Studio/Windows SDK, LLVM/libclang, and the exact pinned shared FFmpeg SDK | -| FFmpeg runtime files | matching `electron/native/bin/-/` directory | Downloaded by `fetch:ffmpeg` for the Windows build path | +| Native Metal compositor addon | `electron/native/bin/darwin-/compositor_view.node` (plus a dev copy under `electron/native/compositor-view/build/`) | Rust, Xcode, and the LGPL FFmpeg tree from `fetch:ffmpeg:mac` | +| FFmpeg runtime files | matching `electron/native/bin/-/` directory | Downloaded by `fetch:ffmpeg` on Windows; **built from source** by `fetch:ffmpeg:mac` on macOS (~5 min) — BtbN publishes no macOS target and every circulating macOS build is GPL, which would relicense this MIT app | Electron-builder copies only the matching `electron/native/bin/-/` directory into each package. The compositor `.node` file is included by the Windows `files` rule and unpacked from ASAR because native addons cannot be loaded from inside the archive. @@ -50,13 +51,28 @@ Electron-builder copies only the matching `electron/native/bin/- A stale addon **fails silently rather than erroring**. Scene fields are `#[serde(default)]` on the Rust side, so an addon that predates a contract change does not reject the payload: it ignores the unknown key, takes the default, and falls back to older behaviour. Nothing appears in any log, and the symptom ("the feature does nothing") is indistinguishable from a TypeScript bug. This is not hypothetical — on 2026-07-27 a build shipped an addon three days older than the commit adding `cursorSprites`, custom cursor themes silently rendered as the built-in art, and the resulting investigation blamed the wrong layer entirely. -`scripts/before-pack.cjs` runs as electron-builder's `beforePack` hook and fails the build when the addon is older than `crates/compositor/src/`, `crates/compositor-view-napi/src/`, or either `Cargo.toml`. It only checks when packaging for Windows. The fix it prints is: +`scripts/before-pack.cjs` runs as electron-builder's `beforePack` hook and fails the build when the addon is older than `crates/compositor/src/`, `crates/compositor-view-napi/src/`, or either `Cargo.toml`. The fix it prints is: ```bash -npm run build:native:compositor +npm run build:native:compositor # Windows +npm run build:native:compositor:mac # macOS ``` -CI is unaffected: `.github/workflows/build.yml` already uses `build:win`, which rebuilds before packaging. +**On macOS it also asserts the payload is complete**, which is a stronger check than freshness and exists because the hook used to return early on every non-Windows platform. That gap was not theoretical: the macOS CI job built the ScreenCaptureKit helper but never ran `fetch:ffmpeg:mac` or `build:native:compositor:mac`, so the `.app` it produced had no compositor addon — preview and export dead in the installed app, with nothing in any log. Nothing caught it, because the one guard that would have was Windows-only. + +The hook now reads `electron/native/bin/darwin-/` — the directory `mac.extraResources` ships wholesale, so "present here" means "present in the installed app" — and refuses to package unless it holds all of: + +| Required | Without it | +|---|---| +| `compositor_view.node` | preview and every export render nothing | +| `libavcodec/libavformat/libavutil.*.dylib` | the addon cannot load at all (dyld error at `require()`) | +| `whisper-stt-server` | transcription and captions fail with a developer error shown to end users | +| `libggml*.dylib` | the helper dies in dyld before `main()`; STT times out with no diagnostic | +| `openscreen-screencapturekit-helper` | native screen capture unavailable | + +It then applies the same staleness comparison to the **shipped** addon (the arch-tagged copy), not the dev copy under `electron/native/compositor-view/build/`, since the arch-tagged one is what electron-builder actually packages. + +CI: the Windows job runs `build:win`, which rebuilds before packaging. The macOS job spells its steps out — it needs `--dir` plus a hand-rolled DMG and signing — and had drifted from the `build:mac` recipe; it now vendors the LGPL ffmpeg tree and builds the Metal addon before packing, both cached. The check compares modification times, so `git checkout` (which restamps source files) can occasionally flag an addon that is genuinely fine. That trade is deliberate — a false alarm costs one rebuild, whereas a missed stale addon ships a broken installer. Run `node scripts/before-pack.cjs` on its own to see the verdict without packaging. @@ -70,6 +86,8 @@ The default electron-builder target is NSIS, with an assisted installer that all ### macOS +> **The macOS job is currently disabled** (`if: false` in `build.yml`) because 1.8.0 ships Windows-only. That flag is release-branch-only and must not reach `main` when promoting, or every later release becomes Windows-only too. Until it is lifted, the macOS packaging path — including the compositor and ffmpeg steps described above — is exercised only by `npm run build:mac` locally. + Electron-builder targets DMG for both `arm64` and `x64`, enables hardened runtime, and applies `macos.entitlements` to the app and inherited code. The entitlements allow Electron JIT/native library loading and audio, camera, and screen capture. The configuration itself sets `notarize: false`; release CI packages the `.app`, creates and signs the DMG manually, submits stable tags to `notarytool`, staples the ticket, and validates Gatekeeper. Pre-release tags skip DMG signing/notarization, and missing Apple credentials produce an unsigned artifact. ### Linux and Nix