Skip to content

development.md

Latest commit

 

History

History
408 lines (291 loc) · 14.8 KB

development.md

File metadata and controls

408 lines (291 loc) · 14.8 KB
Raw

Audio8 Transcriber — Development Notes

This document describes the implementation that currently exists in the repository. It is intended for contributors working on the macOS app, menu-bar helper, local ASR pipeline, packaging, and release verification.

For installation and user-facing behavior, start with README.md.

1. Product contract

Audio8 Transcriber provides two local speech-to-text paths:

  1. Global recording: press ⌘⇧U, click the menu-bar item, or use the Record tab; capture 0.25–30 seconds; transcribe in the background helper; copy and optionally paste into the previously frontmost app.
  2. Audio file: choose a local clip in the Dock app; validate the strict 30-second limit; transcribe in the helper; display the result without auto-paste.

Non-negotiable properties:

  • macOS 15+, Apple silicon.
  • Audio and inference remain local.
  • No TCP/HTTP listener or cloud service.
  • A single warm ASR engine lives in the helper process.
  • Temporary microphone audio is deleted on every terminal path.
  • Audio longer than 30.000 seconds is rejected.
  • The helper remains useful when the Dock window is closed.

2. Process architecture

The application bundle contains two executables.

Audio8Transcriber — Dock application

Responsibilities:

  • SwiftUI window and three input tabs.
  • XPC client lifecycle and reconnect behavior.
  • File picker and security-scoped bookmark creation.
  • Rendering agent state, transcription results, metrics, and diagnostics.
  • Installing/updating the per-user LaunchAgent.
  • Permission guidance and microphone selection UI.

The Dock process does not load the model or transcribe audio.

Audio8Agent — menu-bar helper

Responsibilities:

  • NSStatusItem menu-bar UI.
  • Carbon global hotkey registration.
  • Microphone permission and AVFoundation recording.
  • Input-device selection and restoration.
  • The only ASRTranscriber instance and model warm-up.
  • Clipboard and optional synthetic-paste behavior.
  • Mach XPC listener and client notifications.
  • Temporary-file cleanup.

The helper is an LSUIElement application nested under:

Audio8 Transcriber.app/
  Contents/
    MacOS/Audio8Transcriber
    Library/
      LaunchAgents/audio8-agent.plist
      LoginItems/Audio8Agent.app/
        Contents/
          MacOS/Audio8Agent
          Frameworks/onnxruntime.framework
          Resources/ASRModels.bundle

The model and ONNX Runtime exist only in the helper. Duplicating them in the Dock app adds roughly 480 MB without providing another execution path.

3. Source layout

Audio8Package/SpeechKit/
  Sources/
    Audio8Transcriber/
      Audio8TranscriberApp.swift   # SwiftUI app, model, Record/File UI
      AgentConnection.swift        # XPC client and LaunchAgent startup
      SettingsView.swift           # Setup and diagnostics UI
    Audio8Agent/
      Audio8AgentMain.swift        # LSUIElement entry point
      AgentSession.swift           # Agent state machine/orchestrator
      AgentRecordingController.swift
      AgentTranscriber.swift
      AgentXPCService.swift
      AudioInputDeviceManager.swift
      FrontmostPasteController.swift
      HotkeyController.swift
      StatusItemController.swift
    Audio8XPC/
      Audio8XPCProtocols.swift      # Objective-C-compatible XPC contracts
      Audio8XPCTypes.swift          # NSSecureCoding DTOs/constants
      Audio8LaunchAgentInstaller.swift
      Audio8XPCEndpointStore.swift  # legacy/diagnostic support
    ASRKit/                         # Audio8 inference pipeline
    SpeechCore/                     # assets, resampling, shared errors
    asrkit-cli/                     # command-line transcription tool
  Tests/SpeechKitTests/

Root packaging files:

  • build-app.sh — release build, bundle assembly, signing, verification.
  • Info.plist — Dock app metadata.
  • AgentInfo.plist — helper metadata and microphone usage string.
  • Entitlements.plist / AgentEntitlements.plist — current local ad-hoc entitlement sets.
  • audio8-agent.plist — embedded LaunchAgent template.
  • Audio8Package/dist/ASRModels.bundle — Git LFS model assets.

4. Runtime state flow

AgentSession is the authoritative state machine. Its phases are:

  • preparing — resolving, verifying, and warming the model.
  • idle — ready for a recording or file job.
  • recording — microphone capture active; elapsed time published.
  • busy — validation/transcription/paste in progress.

State changes are published to:

  1. The agent's status item.
  2. Every connected Dock client over XPC.

The UI must not invent local recording state. Buttons issue an XPC command, then render the agent's published state.

Start recording

  1. Reject while preparing, busy, or already requesting permission.
  2. Require the warmed transcriber.
  3. Request microphone access from the helper's TCC identity.
  4. Capture the current frontmost app as the potential paste target.
  5. Temporarily select the preferred input device when configured.
  6. Create a UUID-named 16 kHz mono 16-bit PCM WAV in the helper's temporary directory.
  7. Start AVAudioRecorder with record(forDuration: 30).
  8. Publish elapsed time at approximately 20 Hz.

Stop recording

Stop is triggered by:

  • Second hotkey press.
  • Second status-item click.
  • Record-button press.
  • AVAudioRecorder's duration limit.
  • Timer observation at 30 seconds.

The session then:

  1. Stops capture and restores the previous default input device.
  2. Rejects clips shorter than 0.25 seconds.
  3. Reads/resamples with the same strict 30-second ceiling used for files.
  4. Transcribes with the warm helper-owned engine.
  5. Deletes the temporary WAV with defer-style terminal cleanup.
  6. Copies the transcript and attempts optional paste.
  7. Publishes the result and timings to connected UI clients.
  8. Returns to idle.

Do not add a code path that leaves recordingURL owned ambiguously. The recorder returns ownership to the caller on stop; the caller must delete it.

5. Audio validation and model pipeline

Input contract

  • Target sample rate: 16,000 Hz.
  • Channels: mono after conversion.
  • Minimum duration: 0.25 seconds.
  • Maximum duration: 30.000 seconds.
  • Maximum samples after conversion: 480,000.

For files, validation occurs twice:

  1. Read lightweight file metadata and reject clearly oversized input before allocating/decoding the full clip.
  2. Enforce the exact converted sample-count boundary after resampling.

Do not introduce duration tolerances such as + 0.01; one sample over the limit must be rejected.

Inference stages

  1. AVFoundation decode/resample.
  2. Log-mel feature extraction.
  3. Core ML audio tower on Apple Neural Engine.
  4. Projection into decoder embeddings.
  5. Quantized int4 ONNX decoder on CPU.
  6. Token-to-text reconstruction.

AgentTranscriber.ensureLoaded() verifies the asset manifest, sizes, and SHA-256 hashes before constructing and warming the engine. A load failure remains sticky for that helper lifetime and is surfaced through diagnostics.

6. Mach XPC

The service name is defined in Audio8XPCConstants and advertised by the per-user LaunchAgent. There is no socket or anonymous endpoint discovery protocol.

The Dock app sends:

  • warm-up request
  • recording toggle
  • current-state request
  • file transcription request
  • diagnostics request
  • input-device list/selection
  • microphone test
  • microphone/Accessibility permission requests

The agent pushes:

  • state changes
  • completed transcription results
  • user-facing failures

All DTOs crossing XPC must conform to NSSecureCoding, use explicit interfaces, and remain Objective-C representable.

Caller authentication

AgentXPCService derives and caches the designated signing requirement of the main app embedded beside the helper when the helper starts. For each incoming XPC connection it:

  1. Resolves the caller's dynamic SecCode using the connection PID.
  2. Checks the caller against the cached exact-main-app requirement with SecCodeCheckValidity.
  3. Rejects the connection unless it matches.

Caching the requirement at startup prevents later mutation of bundle files from changing trust decisions inside the running helper.

For ad-hoc builds, the requirement contains the current build's CDHash. Rebuilding changes that hash, which is expected; launch the matching app/helper pair from the same bundle.

If a future Developer ID build is introduced, retain caller verification and validate the production Team ID/designated requirement deliberately. Do not weaken the listener to accept all same-user clients.

7. File handoff

The main app creates a security-scoped bookmark from the NSOpenPanel URL and sends bookmark data over XPC. The helper resolves the bookmark, begins resource access when available, reads the file, and ends access afterward.

File transcriptions set didAttemptPaste to false. They update the window but never inject text into another application.

Never send arbitrary raw paths as the public XPC contract; bookmarks preserve user intent and are compatible with future sandboxing work.

8. Clipboard and Accessibility

At recording start, the helper captures NSWorkspace.shared.frontmostApplication before the Audio8 UI can take focus.

After successful transcription:

  1. Clear and write NSPasteboard.general.
  2. If the saved application still exists, check PostEvent/Accessibility access.
  3. Activate the saved app.
  4. Wait briefly for focus restoration.
  5. Post Command-V with CGEvent.

If any paste step fails, keep the transcript on the clipboard and report a user-actionable fallback. Transcription success must not be converted into a failure merely because Accessibility is denied.

The helper is intentionally unsandboxed because reliable cross-application CGEvent.post is not an App Sandbox design. Any Mac App Store variant needs a different product contract, likely clipboard-only output.

9. Input-device handling

The Setup tab discovers microphones with AVFoundation and maps them to Core Audio device UIDs.

When a preferred device differs from the system default:

  1. Save the current default input device ID.
  2. Set the preferred device for the recording session.
  3. Restore the original default on stop, failure, microphone-test completion, or termination.

Treat restoration as mandatory cleanup. Never leave the user's global default device changed after Audio8 finishes.

10. LaunchAgent lifecycle

Audio8LaunchAgentInstaller writes:

~/Library/LaunchAgents/audio8-agent.plist

The plist contains:

  • absolute path to the helper inside the current main app bundle
  • Mach service registration
  • RunAtLoad
  • per-user interactive process settings

This is why users should move the app to /Applications before first launch. Moving a running app does not update its cached bundle path. Opening the app at its final location rewrites the LaunchAgent when the path changes.

--unregister-agent removes the LaunchAgent and endpoint state before deletion.

11. UI structure

Record tab

  • 30-second progress ring.
  • Record/Stop button.
  • precise elapsed/maximum timer.
  • guidance for menu-bar and global-shortcut capture.

Audio File tab

  • supported-format guidance.
  • file picker.
  • explicit local-processing and no-auto-paste labels.

Setup tab

  • agent/model/permission/hotkey/login checks.
  • preferred-device picker.
  • microphone level test.
  • permission requests and System Settings links.

Transcript card

  • source label.
  • selectable transcript.
  • copy and clear actions.
  • audio duration, inference time, speed ratio, and stage timing.

Avoid displaying absolute model paths or user directory names in public-facing UI. Diagnostics should show ASRModels.bundle · bundled locally unless a developer-only diagnostic export is explicitly requested.

12. Build and signing

Run:

./build-app.sh

The script resolves Swift from the active Xcode toolchain using xcrun, builds both products, assembles the nested app layout, embeds the model/runtime only in the helper, and signs inside-out:

  1. helper frameworks
  2. helper app with helper entitlements
  3. outer app

Do not use a broad codesign --deep --force re-sign step with the outer entitlements. It can overwrite nested helper signing/entitlements and break microphone or paste behavior. --deep is used only for verification.

Current output uses ad-hoc signatures. A distributable binary release requires:

  • stable reverse-DNS bundle IDs
  • Developer ID Application certificate
  • hardened runtime review
  • notarization and stapling
  • a tested update strategy
  • deliberate handling of the helper/LaunchAgent signing requirement

13. Tests and quality gates

Required before committing a release change:

swift test -c release --package-path Audio8Package/SpeechKit
./build-app.sh
codesign --verify --deep --strict --verbose=2 \
  "build/Audio8 Transcriber.app"
plutil -lint Info.plist AgentInfo.plist \
  Entitlements.plist AgentEntitlements.plist \
  audio8-agent.plist

Also verify:

  • model manifest/hash validation
  • exactly 30 seconds accepted
  • one sample over rejected
  • 31-second metadata rejection
  • short-recording cleanup
  • helper model discovery after moving the complete app while it is not running
  • XPC connection succeeds for the matching app and rejects a nonmatching client
  • no network sockets during warm-up/transcription
  • microphone default device restored
  • clipboard-only fallback without Accessibility
  • README screenshots contain no usernames, private paths, or permission secrets

Documentation screenshot hooks

The Dock app supports QA-only arguments:

  • --test-mode Record|Setup
  • --test-file <path>
  • --capture-window <png-path>
  • --test-report <json-path>
  • --delete-test-file

Use -ApplePersistenceIgnoreState YES during screenshot runs so a previously closed window is not suppressed by AppKit restoration. These hooks do not fabricate results; they capture the real rendered state and, for file tests, real inference output.

14. Git and large assets

The model files are tracked with Git LFS under Audio8Package/.gitattributes. Contributors must run:

git lfs install
git lfs pull
git lfs fsck

Never commit:

  • .build/
  • build/
  • .DS_Store
  • temporary audio
  • signing identities/private keys
  • provisioning profiles
  • environment files or credentials

The model license is CC BY-NC 4.0. Preserve attribution and license files when redistributing the repository.

15. Current limitations and future work

  • Fixed ⌘⇧U shortcut; no shortcut editor.
  • English-focused bundled model.
  • No streaming or partial transcripts.
  • No transcript history database.
  • No notarized binary release pipeline.
  • Local LaunchAgent installer rather than an App Store-compatible helper lifecycle.
  • Clipboard/auto-paste behavior can interact with third-party clipboard managers.

Potential future work should preserve the local-only, strict-duration, authenticated-IPC, and cleanup guarantees described above.