diff --git a/CONTEXT.md b/CONTEXT.md index 516769a3..c1ca286b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -57,6 +57,8 @@ S15/S16 target terms below refine product intent. Definitions explicitly marked **Shared client (S16, API16-01):** The browser/native-safe transport and ephemeral projection state over the existing protected Jit service. It submits exact owner commands, reads currently authorized snapshots/control events, recovers the original send identity and reviews exact actions. It is not a second authority, engine or store. The [client contract](docs/contracts/s16-client-v1.md) supports text/tool work and exact decisions. Explicit file composition additionally supports the first [UTF-8 file-to-artifact workflow](docs/contracts/s16-files-v1.md) and React `/s16` client; an explicit strict parser configuration additionally supports [isolated PDF/PNG/JPEG inputs](docs/contracts/s16-binary-files-v1.md) through actual guarded Schway image sends. Explicit [sourced file memory](docs/contracts/s16-file-memory-v1.md) additionally carries exact original/representation pins through later work and inference history, with withdrawal/correction/forget and current recipient/private-scope checks; live steering remains unavailable; Linux isolation and live-account vision evidence remain separate pending acceptance. +**Voice draft (S16, VOICE16-01):** A finite owner-initiated microphone clip and editable transcription under explicit protected-host local speech configuration ([ADR 0096](docs/adr/0096-s16-bounded-local-dictation-and-playback.md)). Capture reserves local audio-second quota, not Bot work. The transcript enters CHAT only through ordinary Send; spoken approval words cannot decide an action. Cancel discards the draft independently of work. Only a currently published reply with exact conversation/request/revision/text pins can be synthesized. Interrupt stops client audio, without implying a Bot or effect stopped. Raw audio/drafts are ephemeral; bounded metadata-only quota survives restart with conservative unknown charges and no speech replay. The [voice contract](docs/contracts/s16-voice-v1.md) does not deliver #423 continuous calls or #431 physical-device acceptance. + **File lineage pin (S16, MEM16-01A):** Immutable metadata naming an exact original source/resource generation and original/validated-representation hashes. MEM/TEAM context, claim and artifact ancestry retains these pins; owning FileStore status and selected bytes are rechecked before release. A pin is data, never a principal or permission. Explicit opt-in freezes new file-memory ingress; old accepted file commands retain their prior semantics. **Work item:** A proposed or owner-requested obligation with a status, explicit completion rule, origin and evidence. A work item may involve multiple Bots and conversations without becoming a Project. diff --git a/README.md b/README.md index c4654b08..38678aa1 100644 --- a/README.md +++ b/README.md @@ -109,6 +109,7 @@ Project access levels, scripted replies, `jit ask`/`pending`, signed webhooks an | --- | --- | | [Service install and upgrade](docs/runbooks/ops-service.md) | Build, plan, install and upgrade the systemd service | | [S15 install](docs/runbooks/s15-install.md) | Opt-in S15 installation (`install.sh --s15`), upgrade, rollback, uninstall and owner steps | +| [Local voice](docs/runbooks/s16-local-voice.md) | Explicit local dictation/playback, pinned speech artifacts, finite usage and opt-in proof | | [Execution box](docs/runbooks/execution-box.md) | Rootless Podman setup and `box-doctor` | | [Pi live round](docs/runbooks/pi-live-round.md) | Host, subscription and reboot evidence on a Pi | | [Load drill](docs/runbooks/ops-load-drill.md) | Capacity and release-evidence drills | diff --git a/docs/adr/0096-s16-bounded-local-dictation-and-playback.md b/docs/adr/0096-s16-bounded-local-dictation-and-playback.md new file mode 100644 index 00000000..6be9ce6b --- /dev/null +++ b/docs/adr/0096-s16-bounded-local-dictation-and-playback.md @@ -0,0 +1,63 @@ +--- +status: implemented optional local dictation/playback; installation and full voice acceptance remain separate +date: 2026-11-18 +scope: VOICE16-01 (#422); S16-AC15; FR-06/11/13/14/19/24/25/37/38; NFR-05/06/07/09/11 +--- + +# Bounded local dictation and reply playback + +## Required outcome and current baseline + +S16-AC15 and the authored [voice experience](../design/S16-rakazo-experience.md#voice-and-clients) require intentional microphone capture, visible recording/transcription, an editable transcript, cancellable playback and finite usage. Text remains usable. Audio/transcribed approval words cannot invoke an exact protected decision. Continuous calls belong to #423; native account/device evidence belongs to #431. The primary agent owns implementation/design under the current AGENTS agreement. + +At main `be21d747d25f4a6a234f057d8b35362501b03a7f`, the shared client (#395) and connections (#417) prerequisites are merged/closed. The React composer, guarded owner projections, command identities and exact decision cards are available. There is no audio implementation; `tests/chat-voice*` tests prose style. #422 was unassigned with no competing branch/PR; it is now owned on an issue-specific worktree. Existing worktrees are untouched. + +## Alternatives and source observations + +| Choice | Benefits and limits | Selection | +| --- | --- | --- | +| Browser/system speech recognition and synthesis | Little server infrastructure. Recognition may use a browser/vendor service or lack a language/platform; synthesis voices can also be remote. A system API is not evidence of local custody | Do not silently use it as local speech. A future explicit device-local adapter must prove its locality and availability | +| Protected provider STT/TTS | Broad language/voice coverage; requires exact account/recipient, billing scope, current private-source checks and uncertain-usage handling | Keep as a future adapter option, not an automatic fallback or a prerequisite for configured local voice | +| Local CLI STT/TTS on the protected Jit host | Actual speech without a new external recipient or paid account. Operator supplies verified binaries/model; resource limits and lifecycle are Jit's responsibility. Recognition/voice quality varies | Select `whisper.cpp` transcription and `espeak-ng` synthesis on Linux, with an explicit `say` synthesis adapter for macOS development/evidence | + +The selected local route satisfies #422's configured-local-or-provider choice. It does not claim every provider or device is supported. The host uses fixed argument arrays, private bounded files and finite processes, never a model-provided command or a working-VM execution backend. Schway continues to own model/tool execution; this owner-client audio adapter introduces no provider loop or private Schway import. + +Rakazo inspection: repository `elie222/rakazo`, immutable commit `abf3c08145a05cba92a912d6c24fdfb5f569d0e0`, tree `f8ca5adc96366b722bb60186c21cb29cec0f5390` (complete tree, not truncated). Read without executing upstream scripts: + +| Source | SHA-256 | Observation | +| --- | --- | --- | +| `packages/adapters/src/openai-compatible-voice.ts` | `33c9fd666df55f36305b26e1a0f9b6deaf0e73e36e07e0382c64f51d5f154c76` | Explicit compatible server/model credentials, bounded transcription/synthesis calls; not Jit's recipient/budget authority | +| `packages/adapters/src/voice-http.ts` | `ceda3d36f626a5893e808f8e94ceb2edd7ff853a234c718165697843359fb80e` | Finite request deadlines and response byte limits are useful concepts | +| `apps/desktop/native/speech/main.swift` | `099d8862dae69ebbd077b352c30afcbf21e949c7008cba1c08c56f0225295b30` | Requests on-device recognition when supported, otherwise permits system service; that fallback cannot establish Jit's local-only route | + +These are comparison observations, not upstream vulnerability claims. No upstream source/assets/dependencies are copied. Existing Apache-2.0 inventory/notices remain unchanged; future copied files follow ADR 0074/0080. CLI binaries, voice data and model weights are separately operator-supplied artifacts; each installation must retain their applicable licenses/provenance rather than infer them from Jit's MIT license. + +## Selected ownership and custody + +The implemented [s16-voice/v1](../contracts/s16-voice-v1.md) owns finite client capture sessions and local speech usage. It adds no Bot tool, message admission or approval authority. The configured voice route is installation-owned, opt-in and contains no credentials. Its artifact paths/hashes are protected configuration; clients get labels, capabilities and limits only. No automatic binary/model download or installation activation occurs. + +The owner starts dictation with a deliberate Mic press. The authenticated service reserves finite local quota before capture. The browser then requests microphone permission, captures bounded PCM and stops every track/audio node on Stop, Cancel, permission loss, page/target/session change or deadline. Cancelled/late permission promises immediately stop any subsequently supplied stream. No browser speech-recognition service is contacted. + +Stop submits a bounded canonical WAV to the same authenticated Jit service for one fixed local transcription. A transcript is untrusted draft data. It is not a conversation message until the owner uses ordinary Send; editing and cancellation create no Bot work. Spoken/quoted "approve" cannot call `decide`, `decideAction` or change purpose. Existing file-bearing/steering semantics remain explicit rather than being inferred from speech. + +Playback selects the exact currently guarded reply by conversation/request/revision/text digest; it does not accept arbitrary unguarded text from the page. Recheck owner/session and source publication before local synthesis and before publishing audio. Interrupt stops playback immediately and fences pending playback responses; underlying Bot work keeps its existing state. Local synthesis is not evidence that an effect succeeded. + +## Bounds, uncertainty and compatibility + +Use finite per-capture duration, WAV bytes, transcript/reply bytes, local process time, synthesized audio duration/bytes and persistent per-day local speech quota. Configuration can narrow published hard limits. Unknown/failed process usage retains its conservative reservation; do not reset quota on restart or confuse audio seconds with measured CPU/GPU time, provider tokens or a zero monetary bill. Local route loss/budget exhaustion ends voice with a safe code while ordinary text remains usable. No fallback creates a new account/recipient or raises limits. + +Review refinement: recording and submission have separate deadlines. Slow microphone permission can consume the original grace, so the capture receipt exposes `recordingEndsAt` and a later finite `expiresAt` with 15 seconds for cleanup/upload. The browser never records into that submission window; the service accepts a completed bounded clip until submission expiry. Actual subprocess barrier/settlement JSON is published by atomic rename after the complete file is written, so file existence cannot expose a partial receipt. + +Audio/transcript scratch is private and short-lived, never in model prompts, working VM, exports, raw errors or logs. Submitted text uses existing conversation retention. Metadata-only quota receipts remain protected; abandoned pending operations are terminal after restart and cannot silently replay. Shutdown kills and actually joins local children before removing scratch/releasing capacity. Browser stop and server settlement are separate facts. + +This is a new optional voice journal/port, not a migration of CHAT, TEAM, MEM, FILE or policy meaning. S14 stays default; S15 opt-in and omitted-runtime compatibility stay. No existing owner host/account/policy is changed by implementation. Packaging verifies the optional feature without bundling a model/binary or asserting device acceptance. #422 closes only after actual composed client/service and native speech evidence plus required checks/review; this document alone does not complete it. + +The [operator runbook](../runbooks/s16-local-voice.md) describes explicit configuration, artifact licenses/pins, usage and recovery. Runtime source reaches the shipped package through the existing `src` inventory, including the independent supervisor and fixed worklet. No new package dependency or imported upstream asset is required. Health metadata caching is descriptive only: each actual speech job rehashes executable/model pins. A valid bounded macOS `say` WAV may include `FLLR` padding; synthesis validation accepts it without weakening the canonical capture format. + +[Local speech evidence](../evidence/s16-local-voice-proof.md) records actual macOS/Linux ARM64 speech, the real English/Thai bundle, process-crash/unknown-usage fixtures, exact artifact hashes, failed experiments and remaining acceptance. Linux eSpeak NG required the separately pinned `base.en` model to pass the unchanged known-speech assertions; the smaller `tiny.en` failure is retained as a quality limit. This is not full continuous-voice/device acceptance. + +## Verification obligations + +Use deterministic PCM/command fixtures, fake clocks and barriers for quota, cancellation, lost responses, stale generations and current owner/publication checks. Browser tests exercise the real bundle in English/Thai and mobile/keyboard layouts, with fake media devices rather than a real microphone. Native opt-in evidence must transcribe actual known speech and synthesize an actual guarded reply through configured local executables, recording exact binary/model/build hashes. Label recognition accuracy, device permissions and all untested platforms honestly. #423 retains continuous-call/barge-in/reconnect behavior. + +Protocol references inspected 2026-11-18: [Media Capture and Streams](https://www.w3.org/TR/mediacapture-streams/), [whisper.cpp CLI implementation](https://github.com/ggml-org/whisper.cpp/blob/master/examples/cli/cli.cpp), [espeak-ng command reference](https://github.com/espeak-ng/espeak-ng/blob/master/src/espeak-ng.1.ronn). Implementations must validate their exact supplied build; these moving references are not installation pins. diff --git a/docs/contracts/s16-voice-v1.md b/docs/contracts/s16-voice-v1.md new file mode 100644 index 00000000..41fe58d0 --- /dev/null +++ b/docs/contracts/s16-voice-v1.md @@ -0,0 +1,65 @@ +# s16-voice/v1 + +Status: implemented optional local dictation/playback for #422 ([ADR 0096](../adr/0096-s16-bounded-local-dictation-and-playback.md)). Explicit protected-host configuration is required; omission keeps text behavior. #423 owns continuous calls. The [runbook](../runbooks/s16-local-voice.md) records setup, opt-in evidence and platform/device limits. + +## Protected host route + +Compose an optional local voice service in S15 and add an optional `voice` capability to the existing shared client. Older services/clients keep text behavior. No configured route means voice unavailable, never an implicit browser/cloud service. A route names pinned executable/model artifacts, STT language, synthesis kind/voice and finite local quota. Protected paths and raw CLI output never enter capability JSON. Configuration is explicit host-owned state, not a model/tool payload. + +Linux route: `whisper.cpp` CLI with fixed model/file/output/language/no-GPU/thread arguments, plus `espeak-ng` file-to-WAV synthesis. macOS evidence may explicitly select `say` file-to-WAV synthesis; this does not claim Linux availability. No shell parsing, arbitrary flags, URL input, inherited provider credentials or automatic model download. Verify exact configured artifact hashes before use. Operator-supplied libraries/model/voice data remain a trusted installation dependency; do not claim the CLI wrapper is a sandbox against a malicious executable. + +## Owner-client operations + +All operations use the current shared-client owner session, exact Host/Origin, CSRF for POST, no cross-site access, no-store responses, finite streamed bodies and reauthentication after asynchronous boundaries. No voice endpoint accepts an approval command or serialized actor. + +| Operation | Binding and result | +| --- | --- | +| Read capability/usage | Actual configured route health, finite limits, local audio-second quota and pending/unknown reservations. Read starts no capture/process | +| Begin dictation | One exact command ID, conversation and current session generation; reserve bounded local capture quota. Returns opaque capture identity, expiry and permitted duration/bytes; starts no Bot work | +| Transcribe | One capture identity and strict PCM WAV bytes; identity/hash conflict refuses. Actual owner/route/capacity checks precede process start and transcript publication. Returns editable untrusted draft text plus observed/unknown usage | +| Cancel | Fence current capture/transcription, stop and join actual local process, discard scratch/draft. A late result cannot revive it. Does not cancel a Bot request | +| Synthesize reply | One identity with exact conversation/request/revision/text hash. Reacquire current guarded owner projection; synthesize only its published result. Return bounded WAV after current authorization/source checks | +| Interrupt playback | Stop client audio immediately, revoke its object URL and fence pending synthesis/play responses; underlying work remains separate | + +The protected HTTP paths and shared `JitClient` methods are: + +| HTTP | Client method / result | +| --- | --- | +| `GET /api/s16/client/voice` | `voiceStatus` → `VoiceStatus`, including current `serverTime`, limits and local usage | +| `POST /api/s16/client/conversations/:conversationId/voice/captures` | `prepareDictation(serverTime)` creates a `VoiceCaptureCommand`; `beginDictation` → `VoiceCapture` | +| `GET /api/s16/client/voice/:id` | `dictationStatus` → current `VoiceCapture` metadata | +| `POST /api/s16/client/voice/:id/transcribe` | `transcribeDictation(capture, wav)` sends bounded `application/octet-stream`; → `VoiceTranscript` | +| `POST /api/s16/client/voice/:id/cancel` | `cancelVoice` sends `{}`; → terminal/current `VoiceCapture` metadata | +| `POST /api/s16/client/voice/playback` | `preparePlayback` creates the exact `VoicePlaybackCommand`; `playReply` → `VoicePlayback` | + +Types are exported through `src/s16/client`; the owning shapes live in `src/s16/voice/contracts.ts`. The additive bootstrap capability states that the API is configured; `voiceStatus.available` reports actual local route health. It does not grant microphone or work authority. Command JSON accepts exact keys; raw capture accepts no client-provided duration or transcript. Playback returns bounded `wavBase64` for a same-origin-client Blob, its actual duration and usage. Draft/audio results are private, no-store and memory-only. + +Client retries never invent an identity or automatically replay speech. No transcript is automatically submitted to CHAT. Owner Send uses the ordinary frozen message command/receipt and unchanged exact-decision surface. Cancel before Send creates no conversation request/work. `useVoice` owns one client lifecycle shared by the composer and reply buttons; `startVoiceRecording` uses the authenticated fixed `/s16/voice-worklet.js`, finite PCM buffers and a reservation-relative deadline. There is no browser recognition/synthesis fallback. View/conversation changes, session bootstrap/clear, owner rejection, track loss, cancellation and limits fence pending responses and dispose native resources. English/Thai controls keep a separately labelled editable transcript before Add to message and ordinary Send. + +## Audio and finite usage + +Capture uses explicit `getUserMedia` after a deliberate Mic action, with visible Recording/elapsed time/Stop/Cancel. A bounded PCM conversion emits canonical RIFF/WAVE, PCM format 1, mono, 16-bit little-endian at 16,000 Hz. Validate RIFF length, exact format/data layout, alignment, sample count and hard duration/byte limits before CLI input. Reject malformed/compressed/oversized or trailing data rather than trusting MIME or client duration. No private audio decoding library or arbitrary attachment parser runs for capture. + +Hard per-capture ceiling: 30,000 ms / 960,044 canonical WAV bytes. Transcript is at most 16 KiB UTF-8 and still subject to ordinary message limits. Synthesis accepts at most 2,000 characters / 8 KiB UTF-8 of one current result, with at most 60 seconds / 4 MiB output and a finite process deadline. Longer replies remain readable; do not silently label truncated speech as a complete result. Final implementation may narrow bounds and must publish the actual limits before capture/playback. + +Capture receipts separate `recordingEndsAt` from the final `expiresAt` submission deadline. The recording/permission deadline is the reservation time plus the configured capture bound and 15 seconds; submission expires another 15 seconds later. The client stops by the earlier deadline (conservatively subtracting request elapsed time), leaving cleanup/upload time even after slow permission reduces the recording duration. The server still validates actual PCM duration and refuses a new transcription at/after submission expiry. This finite grace never extends microphone capture or admits another request identity. + +Configured quota is local audio seconds per UTC day, persisted outside the working VM. Reserve worst-case capacity before capture/synthesis; update observed accepted/generated audio seconds only after real evidence. Unknown processing keeps its conservative charge/reservation, including restart. Local audio seconds are not measured compute time, provider billing or tokens. A refused/not-started operation can release unused capacity; cancellation after actual processing is not proof of zero usage. Quota exhaustion or route failure stops voice without disabling text or switching provider. + +Only one owner capture/playback lifecycle is active per client generation. Permission denial/revocation, track `ended`/unusable capture, cancellation, limit, target change, logout and unmount stop all tracks/nodes and clear buffers. An unresolved permission prompt can outlive a cancellation: any later stream must be stopped without adoption. Stop does not rely on a `MediaStreamTrack` ended event firing for application-initiated stop. Browser resources and pending responses are fenced independently of server process settlement. + +## Custody, publication and recovery + +Raw audio stays in client memory and protected service scratch. The local engine receives only bounded private files and a minimal environment; raw output is never returned as an error. Scratch is removed after actual process settlement; startup reaps abandoned scratch without following arbitrary paths. No audio/transcript is placed in the working VM, native model history, logs or ordinary exports. A submitted transcript becomes ordinary owner text under existing CHAT custody. Browser/device-local memory cannot be remotely recalled after disclosure. + +The service owns a separate versioned voice/quota journal with bounded metadata and no audio. It retains original command identities and conservative charges across process loss. A restarted pending local operation becomes terminal/unknown, with no reconstructed execution. Missing/expired draft/audio results are reported honestly; automatic re-recording, re-synthesis or re-submission is forbidden. #423 will extend continuity for confirmed continuous-call turns using the existing durable CHAT receipt, rather than treating a voice response as work acceptance. + +The implementation uses `VoiceService.open` with an exclusive protected journal lease before its engine factory opens scratch. `begin`, `inspect`, `transcribe`, `playback`, `cancel`, `status`, final synchronous `publishTranscript`/`publishPlayback` and memoized `close` accept trusted current-owner ingress closures; they are not model tools or serialized actor APIs. The HTTP handler finishes streamed body cleanup before processing/publication and runs the final publisher in the same synchronous turn as response serialization. A source withdrawn after synthesis resolves still cannot release cached audio. The optional composed service reacquires owning conversation/result projections and closes voice before releasing its base protected ledgers. Command IDs embed issuance time (`voiceRequestId`); first admission permits five minutes of age and thirty seconds of future skew. Retained retries reuse the original binding. Terminal metadata may expire after seven days, but an old timestamped identity still cannot become a new capture. A persisted clock high-water prevents clock rewind from resetting a quota day. At most 4,096 retained operations and 8 MiB / sixteen short-lived response-cache entries are permitted; storage exhaustion refuses new voice work honestly. + +Accepted input duration is known from strict PCM bytes and rounded up to quota milliseconds when transcription starts. Synthesis starts with its conservative maximum and narrows only after a valid actual WAV result. Failed/unknown started processing does not refund the charge. The independent local supervisor retains a private stdin liveness lease; controller loss, cancellation or its own deadline kills and joins its CLI before writing a metadata-only settlement receipt. A reopened engine cleans abandoned scratch only when no command could have been supplied or the matching supervisor receipt proves settlement. An unproven orphan keeps the route unavailable. Runtime composition must place scratch in protected temporary storage outside application state/backups, while the metadata-only journal remains durable. No raw audio/draft is an ordinary export. + +Private reply publication is always reacquired through the owning conversation service, including before returning generated audio. Source withdrawal, owner/session revocation, changed result/revision, changed route artifact or generation prevents stale publication. Interruption invalidates client playback without claiming remote work or effects stopped. Audio content, transcripts, quoted commands and model output cannot approve, change scope or configure a route. + +## Delivery evidence + +Tests must exercise the actual configured local adapter/service and ordinary Send/decision ports, not only a reducer or a mock successful transcript. Deterministic fixtures cover strict WAV parsing, finite quotas across restart, duplicate/conflicting commands, late permission/result rejection, denied/revoked capture, exact reply/current-source checks, provider/process loss and no work before Send. Real English/Thai bundle tests cover keyboard/mobile/error states, interrupted playback, text fallback and transcribed approval words. Opt-in local speech evidence records actual known speech transcription and reply audio with exact executable/model hashes. Physical native-device/permission acceptance remains #431; source presence or this contract is not completion. diff --git a/docs/evidence/s16-local-voice-proof.md b/docs/evidence/s16-local-voice-proof.md new file mode 100644 index 00000000..72fc42e6 --- /dev/null +++ b/docs/evidence/s16-local-voice-proof.md @@ -0,0 +1,61 @@ +# VOICE16-01 local dictation and guarded playback + +Date: 2026-11-18. Runtime source: `60679513fdf70d0f9b32e409387023e7068dafae`. Scope: #422, the bounded-dictation/playback portion of S16-AC15; FR-06/11/13/14/19/24/25/37/38; NFR-05/06/07/09/11. Contract: [s16-voice/v1](../contracts/s16-voice-v1.md), [ADR 0096](../adr/0096-s16-bounded-local-dictation-and-playback.md), [operator runbook](../runbooks/s16-local-voice.md). + +This evidence uses the actual optional composed voice service, shared client, local process supervisor and React/AudioWorklet bundle. Model, interpretation and working-VM work remain scripted fixtures. Native speech uses actual local executables/model files; no microphone, private recording, provider credential, metered API or owner-host activation is used. Installation paths and full private receipts remain ignored `.local` records. Public facts below contain build/artifact hashes and outcomes. + +## Actual local speech + +`tests/s16-voice.native.test.ts` synthesizes the original public test sentence, "The quick brown fox jumps over the lazy dog.", converts its actual PCM to canonical capture WAV, transcribes through the actual configured whisper CLI, and verifies the known words without changing assertions between platforms. The composed request count and model history stay empty until ordinary Send. An explicit text edit adds a diagnostic instruction; scripted model/VM work then publishes the real guarded result, "Completed with observed tool evidence.", which the local adapter synthesizes. Final usage has no outstanding reservation or conservative charge. + +| Observation | macOS ARM64 | Linux ARM64 | +| --- | --- | --- | +| Receipt UTC | `2026-11-18T05:45:53.063Z` | `2026-11-18T05:48:28.227Z` | +| Runtime | Bun `1.4.2` (`43848b5a7`) | Ubuntu 24.04.5, Bun `1.4.2` (`744846f84`) | +| STT | whisper.cpp v1.9.2, `tiny.en`, language `en`, CPU-only fixed arguments | whisper.cpp v1.9.2 CPU build, `base.en`, language `en`, same fixed arguments | +| TTS | Explicit local `say`, installed `Samantha`, rate 165 | eSpeak NG `1.51+dfsg-12build1`, installed `en-us`, rate 165 | +| Canonical input duration | 2,670.75 ms | 3,091.1875 ms | +| Observed transcript | Exact test sentence | Exact test sentence | +| Actual reply duration | 2,194 ms | 2,531 ms | +| Charged local audio | 4,865 ms | 5,623 ms | +| Native suite | 1 pass, 0 fail, 10 assertions | 1 pass, 0 fail, 10 assertions | + +Usage dates are intentionally `1970-01-01` under the composed fixture's deterministic clock; receipt UTC is the actual proof time. Audio seconds measure accepted/generated audio, not compute time or monetary cost. The Linux proof ran inside a disposable controller VM and does not assert default Linux/KVM or owner-host acceptance. + +| SHA-256 artifact | macOS ARM64 | Linux ARM64 | +| --- | --- | --- | +| `whisper-cli` | `40bca494d49af736058eb3f33cbcebaa020eacf6d0087b623f334946e1ab2128` | `651c5ca81bbd648da226598e0349ec097ffb1c74dd04f8e8f249e807b1758bb6` | +| Synthesis executable | `29f947458332d87ae788ba874728bfbe5fc39a30b6249522569ddbd11bd2bb5c` | `89402b6a13d29ab2edb0570c809796751b22a5d031828897cfb1b370dafa9c29` | +| Model | `921e4cf8686fdd993dcd081a5da5b6c365bfde1162e72b08d75ac75289920b1f` | `a03779c86df3323075f5e796cb2ce5029f00ec8869eee3fdfb897afe36c6d002` | +| Canonical input WAV | `dec53bc57d9f07d9b5c133aa91c069c91e2d2b3050f85c90ff032037ad87d3af` | `75b061d3b158a198b8fc7b9a029dba7c2d2eee0af2b5223a585f4500b6d15a7b` | +| Actual reply WAV | `2b8f1312b9fa85aabf4351cb8e9494d286b94e66f742441b713051a8ad04510a` | `620a065674469b24d8e2761b3117e352088938981b2c43aca761197ba95a7228` | + +The Linux source archive is [whisper.cpp commit `306c88f4d1286aec1bf96e544632897886af5501`](https://github.com/ggml-org/whisper.cpp/tree/306c88f4d1286aec1bf96e544632897886af5501), archive SHA-256 `f3585ebf64df3e41b26c45d93bdb38b423ca1de0bf40cc4b6d320254867a75df`. It was built with CMake `3.28.3-1build7`, GCC package `4:13.2.0-7ubuntu1`, Release, two build jobs, `GGML_NATIVE/OPENMP/BLAS/CUDA/VULKAN=OFF`, `WHISPER_CURL=OFF`, `WHISPER_BUILD_TESTS=OFF`, target `whisper-cli`. Runtime still explicitly passes `--no-gpu`. Artifact pins cover the executable/model, not an attestation of every dynamic dependency or an OS sandbox. Source, binaries, voice data and weights are separately trusted operator dependencies with their own licenses; Jit imports/bundles none of them. + +Both public models came from Hugging Face repository `ggerganov/whisper.cpp`, immutable revision `5359861c739e955e79d9a303bcbc70fb988958b1`; downloaded bytes were checked against its LFS SHA-256/size metadata. Model links, bytes and hashes are in the runbook. No training was performed. + +## Browser and owning-service behavior + +`JIT_S16_BROWSER_TESTS=1 bun test tests/s16-voice.browser.test.ts`: **7 pass, 0 fail, 71 assertions**. The real English 1440 px and Thai 390 px React layouts use native Chrome fake media devices and the actual fixed AudioWorklet over a loopback HTTP server. Sound is muted; no physical microphone is opened. Keyboard-started reply playback, no horizontal overflow, elapsed capture, Stop, editable transcript, Add to message and ordinary Send are exercised. Before Send there is no CHAT request/model history; Cancel creates no additional work. Playback interruption and later local route loss stop client audio without stopping the underlying conversation request. Controlled permission denial/track-mute, late permission after navigation and actual owner-cookie rejection exercise cleanup/text fallback and private-draft clearing. The session-negative case's HTTP 401 is expected. Fake wall-clock barriers also cover permission delayed beyond fifteen seconds, automatic recording stop followed by successful upload, an expired recording receipt opening no microphone prompt, and a late recording-health response preserving the completed editable transcript. + +Chrome AudioWorklet fetches bypassed the older Playwright request-route harness; the final test uses a real isolated HTTP server rather than substituting a fake worklet. The successful native capture uses the actual browser AudioContext/AudioWorklet classes. Screenshots were inspected for desktop/mobile layout; they are private test artifacts, not physical-device evidence. + +`tests/s16-voice-client.test.ts` additionally exercises the actual HTTP/composed ports: approval words remain ordinary text while a Google send slip stays pending; streaming reauthentication and malformed/oversized capture refuse before engine start; client clear rejects late private JSON; withdrawal after the synthesis promise resolves still prevents final HTTP audio release; omitted voice preserves text and creates no voice journal; the worklet requires owner authentication and no-store/CSP headers. + +## Durability, bounds and regression coverage + +At the source revision above, `bun test tests/s16-voice-audio.test.ts tests/s16-voice-local.test.ts tests/s16-voice-service.test.ts tests/s16-voice-client.test.ts` passed **30 tests / 164 assertions** on both macOS and Linux ARM64. This includes the authenticated-worklet case (seven client tests / 35 assertions) and a completed clip submitted during the separate finite submission window. The core covers strict actual PCM bytes, 30-second limits, native `say` filler without relaxing capture input, pinned artifacts, minimal process environment, actual cancellation/deadline/close joining, actual controller SIGKILL plus supervisor settlement, journal reopen/unknown quota/no replay, route-health races, stale generation and reply/source guards. These process-crash fixtures use actual subprocesses with controlled CLI output; they are distinct from the actual speech proof above. + +The composed/runtime/package regression at the final runtime source above passed **254 tests / 1,479 assertions across thirteen files**. Frozen install, typecheck, upstream provenance and committed diff checks passed; the PR records its final head (including subsequent evidence-only changes). Linux ARM64 typecheck passed after its repeated native proof; provenance passed on both platforms. Package tests exercise the existing source inventory, which ships the supervisor/worklet without bundling speech models or binaries. + +## Review and CI corrections + +Review identified that slow permission could let recording end at submission expiry. Capture receipts now separate `recordingEndsAt` and a submission deadline fifteen seconds later. Expired recording receipts refuse before asking for microphone access. A delayed capture-health failure can update availability without discarding an already completed transcript; owner/authentication rejection still withholds it. + +The initial ARM64 CI shard 5 failed when a crash-test marker became visible before its JSON write completed. The controlled marker is now published by atomic rename, and the production supervisor similarly publishes its complete settlement receipt only after joining the actual CLI. No timeout/assertion was weakened. The exact failing shard command passed locally: **641 pass, 17 opt-in skips, 0 fail, 66,627 assertions across 52 files**. Updated native/controller-crash suites passed on Linux ARM64 as recorded above; the fresh PR CI run must still pass on its own final head before merge. + +## Failed experiments and remaining acceptance + +The first macOS actual WAV included an `FLLR` padding chunk and was refused by synthesis validation. The corrected parser accepts that bounded padding; an independent fixture keeps canonical capture strict. The Linux source build initially produced a group-writable executable, correctly refused by artifact validation; the supplied artifact's mode was corrected before proof, without changing its hash. Linux `tiny.en` misrecognized the eSpeak NG sentence (including "quick brown fox"). That run failed the unchanged assertions and is not accepted evidence. Selecting the separately pinned `base.en` model passed the same test. These results do not establish broad recognition quality. + +Thai copy/layout is tested; Thai speech accuracy is unverified because these proof models/voices are English. Operators must validate their selected multilingual model/voice. Physical microphone permission/revocation, real mobile audio/background behavior, other speech platforms/languages, accounts and owner-host cutover remain #431/#432. Linux x64 deterministic CI is separate from actual ARM64 speech proof. Continuous call, barge-in and reconnect work remains #423. Parent #392 and full S16-AC15/AC22 acceptance stay open. No production deployment, standing-policy change, default-runtime cutover, historical scope migration or private-data disclosure follows from this proof. diff --git a/docs/runbooks/s16-local-voice.md b/docs/runbooks/s16-local-voice.md new file mode 100644 index 00000000..1335c5f9 --- /dev/null +++ b/docs/runbooks/s16-local-voice.md @@ -0,0 +1,81 @@ +# Optional local dictation and playback + +VOICE16-01 ([ADR 0096](../adr/0096-s16-bounded-local-dictation-and-playback.md), [contract](../contracts/s16-voice-v1.md)) adds bounded dictation/playback to the explicit S15 service and shared `/s16` client. Omitted configuration keeps text available and creates no voice journal. Installation, model download, device permission and owner-host activation never happen automatically. S14/default/legacy services and installed policy are unchanged. Continuous calls belong to #423; physical-device and owner-host acceptance remain #431/#432. + +## Supply trusted local artifacts + +On Linux, supply `whisper.cpp`'s `whisper-cli`, a compatible public GGML model, `espeak-ng` and its local voice data. macOS development/evidence may explicitly select `/usr/bin/say`. Use canonical absolute paths with no symlink components, no group/world-writable artifacts, and executable bits for binaries. Hash each actual executable/model with SHA-256 after resolving its path. Retain its source/build provenance and applicable licenses; Jit's MIT license does not relicense weights, libraries or voice data. Pins do not sandbox malicious executables or attest every dynamic dependency. + +The proof uses whisper.cpp v1.9.2 commit `306c88f4d1286aec1bf96e544632897886af5501` and public GGML models at Hugging Face revision `5359861c739e955e79d9a303bcbc70fb988958b1` (upstream labels MIT): + +| Model | Bytes | SHA-256 | Observed proof | +| --- | --- | --- | --- | +| [base.en](https://huggingface.co/ggerganov/whisper.cpp/blob/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-base.en.bin) | 147,964,211 | `a03779c86df3323075f5e796cb2ce5029f00ec8869eee3fdfb897afe36c6d002` | Linux eSpeak NG synthetic sentence passes the unchanged known-word assertions | +| [tiny.en](https://huggingface.co/ggerganov/whisper.cpp/blob/5359861c739e955e79d9a303bcbc70fb988958b1/ggml-tiny.en.bin) | 77,704,715 | `921e4cf8686fdd993dcd081a5da5b6c365bfde1162e72b08d75ac75289920b1f` | macOS `say` sentence passes; Linux eSpeak NG pronunciation was misrecognized, so it is not the validated Linux example | + +These English-only models do not establish Thai recognition accuracy. Choose and verify a compatible multilingual model/language for Thai. Browser Thai copy is separate evidence. No model/binary is bundled with Jit. + +Add an explicit `voice` entry to a reviewed S15 runtime configuration. This template's paths/hashes must be replaced with the supplied artifacts; it is not an activation command: + +```json +{ + "voice": { + "local": { + "transcription": { + "executable": { "path": "/protected/voice/bin/whisper-cli", "sha256": "" }, + "model": { "path": "/protected/voice/models/ggml-base.en.bin", "sha256": "a03779c86df3323075f5e796cb2ce5029f00ec8869eee3fdfb897afe36c6d002" }, + "language": "en" + }, + "synthesis": { + "kind": "espeak", + "executable": { "path": "/usr/bin/espeak-ng", "sha256": "" }, + "voice": "en-us" + }, + "timeoutMs": 60000 + }, + "quota": { "dailyMs": 120000, "captureMs": 30000 } + } +} +``` + +The macOS evidence alternative is `kind: "say"` with the verified canonical `say` executable and an already installed local voice such as `Samantha`. No download/fallback is requested. Configuration contains no remote URL, credentials or arbitrary flags. The local supervisor uses fixed arguments, no shell, CPU-only whisper flags and a minimal environment. A healthy status read starts no speech; unchanged artifact metadata may cache health, but each actual speech job rehashes all three pins. Keep installation paths/config/receipts in ignored `.local` records. + +`captureMs` is an integer from 1 through 30,000; `dailyMs` is finite, at least the configured capture bound and at most 3,600,000. Playback needs its full 60,000 ms reservation before synthesis. `timeoutMs` is 1–120,000. These are local audio-second limits, not compute-time or monetary billing measurements. Process/model limits also depend on the trusted host; the wrapper is not an OS memory sandbox. + +## Use and recovery + +Open `/s16` through a browser secure context (HTTPS, or browser-trusted loopback). Dictate reserves quota before asking for microphone access. Recording displays elapsed time, Stop and Discard. Stop produces an editable transcript; Add to message copies it to the ordinary composer. Only Send admits work. The transcript, including the word "approve", cannot press an exact decision card. Permission denial/revocation, unavailable local artifacts or exhausted quota leave ordinary text available. No browser/vendor speech service is selected. + +The server supplies separate recording and submission deadlines. Recording ends no later than the capture bound plus 15 seconds from reservation; slower permission shortens the remaining recording time. A further 15-second window remains for cleanup/upload before submission expires. It does not permit longer audio or automatic retry. A connection that cannot submit within the finite window reports expiry honestly. + +Listen to this reply binds the exact currently guarded result. Stop listening interrupts audio and pending voice responses, independently of the Bot's Stop action. A changed/withdrawn source or session prevents fresh audio publication. The client clears voice on conversation/view/session refresh and disposes microphone tracks, audio nodes and object URLs; a cancelled permission prompt's later stream is stopped immediately. + +The service owns metadata-only `voice.sqlite` under the existing protected state manifest/exclusive lease. Audio/text scratch uses a private installation-specific system temporary directory outside application state/backups; audio/transcripts are never journal fields. Completed response caches are memory-only, at most sixteen entries/8 MiB/five minutes. A transcript deliberately sent to CHAT follows ordinary CHAT retention. + +Known accepted PCM duration is charged when STT starts. Synthesis keeps its worst-case charge until a valid actual WAV narrows it. Unknown/failed started processing is not refunded or replayed after restart. The supervisor's stdin liveness lease kills and joins the CLI after controller loss. Startup removes scratch only when no job could have started or a matching settlement receipt proves physical CLI exit; an unproven orphan makes voice unavailable. Do not delete the voice journal or orphan evidence to bypass usage/recovery. Read safe codes and retain private installation receipts for investigation; no raw CLI stderr is exposed. + +## Verify an exact build + +Deterministic suites: + +```sh +bun test tests/s16-voice-audio.test.ts tests/s16-voice-local.test.ts tests/s16-voice-service.test.ts tests/s16-voice-client.test.ts +bun run typecheck +``` + +Real React/AudioWorklet browser tests use an isolated loopback test server on port 3099, fake native media devices and muted audio. They never open the owner's microphone. Supply an installed Chrome path with `JIT_BROWSER_EXECUTABLE` when needed: + +```sh +JIT_S16_BROWSER_TESTS=1 bun test tests/s16-voice.browser.test.ts +``` + +For opt-in actual local speech, put only the `local` object in an ignored configuration file; its supplied binaries/model must match their pins. The harness creates its own public synthetic sentence and scripted conversation/VM fixture, transcribes via actual whisper, checks that no work exists before ordinary Send, then synthesizes the currently guarded reply. It does not use a subscription/account or a private recording: + +```sh +JIT_S16_NATIVE_VOICE_TESTS=1 \ +JIT_S16_NATIVE_VOICE_CONFIG=.local/voice-config.json \ +JIT_S16_NATIVE_VOICE_RECEIPT=.local/voice-receipt.json \ +bun test tests/s16-voice.native.test.ts +``` + +Keep exact build/artifact hashes, durations, outcomes and limitations with release evidence. This native harness does not prove physical microphone permission, recognition quality in every language, mobile background/continuity, real account policy or owner-host activation. diff --git a/src/i18n/en.ts b/src/i18n/en.ts index 7f556616..496f2e57 100644 --- a/src/i18n/en.ts +++ b/src/i18n/en.ts @@ -6504,4 +6504,27 @@ export const en: Record = { "s16.connections.card.note": "Nothing is connected or approved until you finish on the protected form. Never paste secrets into chat.", "s16.connections.card.open": "Open the connection form", "s16.connections.card.dismiss": "Dismiss", + "s16.voice.title": "Voice", + "s16.voice.dictate": "Dictate", + "s16.voice.stop": "Stop recording", + "s16.voice.cancel": "Discard dictation", + "s16.voice.interrupt": "Stop listening", + "s16.voice.preparing": "Preparing voice…", + "s16.voice.permission": "Waiting for microphone permission…", + "s16.voice.recording": "Recording", + "s16.voice.transcribing": "Transcribing…", + "s16.voice.loading": "Preparing audio…", + "s16.voice.playing": "Listening to reply", + "s16.voice.checking": "Checking local voice…", + "s16.voice.unavailable": "Local voice is unavailable. You can keep using text.", + "s16.voice.local": "Audio is processed locally on your Jit server.", + "s16.voice.usage": "{used} s processed · {reserved} s reserved · {limit} s daily limit", + "s16.voice.conservative": "{seconds} s remain conservatively charged after an interrupted operation.", + "s16.voice.transcript": "Edit your transcript", + "s16.voice.review": "Review this text before adding it to your message. Speech never approves an action.", + "s16.voice.useText": "Add to message", + "s16.voice.listen": "Listen to this reply", + "s16.voice.permissionDenied": "Microphone access stopped or was denied. Your text is still available.", + "s16.voice.budget": "The local voice limit is reached. Continue with text.", + "s16.voice.failed": "Voice stopped. You can continue with text.", }; diff --git a/src/i18n/th.ts b/src/i18n/th.ts index 2718a6af..6e793322 100644 --- a/src/i18n/th.ts +++ b/src/i18n/th.ts @@ -6508,4 +6508,27 @@ export const th = { "s16.connections.card.note": "ยังไม่มีอะไรถูกเชื่อมหรืออนุมัติจนกว่าคุณทำให้เสร็จในฟอร์มที่ปลอดภัย อย่าวางข้อมูลลับในแชท", "s16.connections.card.open": "เปิดฟอร์มเชื่อมต่อ", "s16.connections.card.dismiss": "ปิด", + "s16.voice.title": "เสียง", + "s16.voice.dictate": "พิมพ์ด้วยเสียง", + "s16.voice.stop": "หยุดอัดเสียง", + "s16.voice.cancel": "ทิ้งข้อความเสียง", + "s16.voice.interrupt": "หยุดฟัง", + "s16.voice.preparing": "กำลังเตรียมเสียง…", + "s16.voice.permission": "กำลังรอสิทธิ์ใช้ไมโครโฟน…", + "s16.voice.recording": "กำลังอัดเสียง", + "s16.voice.transcribing": "กำลังถอดเสียง…", + "s16.voice.loading": "กำลังเตรียมเสียงอ่าน…", + "s16.voice.playing": "กำลังฟังคำตอบ", + "s16.voice.checking": "กำลังตรวจเสียงบนเครื่อง…", + "s16.voice.unavailable": "เสียงบนเครื่องยังใช้ไม่ได้ คุณยังใช้ข้อความต่อได้", + "s16.voice.local": "ประมวลผลเสียงบนเซิร์ฟเวอร์ Jit ของคุณ", + "s16.voice.usage": "ประมวลผลแล้ว {used} วิ · สำรอง {reserved} วิ · จำกัดวันละ {limit} วิ", + "s16.voice.conservative": "ยังนับยอดสำรอง {seconds} วิ จากรายการที่ถูกขัดจังหวะ", + "s16.voice.transcript": "แก้ไขข้อความที่ถอดเสียง", + "s16.voice.review": "ตรวจข้อความก่อนเพิ่มในข้อความที่จะส่ง เสียงพูดไม่อนุมัติการกระทำใด ๆ", + "s16.voice.useText": "เพิ่มในข้อความ", + "s16.voice.listen": "ฟังคำตอบนี้", + "s16.voice.permissionDenied": "สิทธิ์ไมโครโฟนถูกปฏิเสธหรือหยุดแล้ว คุณยังใช้ข้อความได้", + "s16.voice.budget": "ใช้เสียงถึงขีดจำกัดแล้ว ใช้ข้อความต่อได้", + "s16.voice.failed": "เสียงหยุดแล้ว คุณยังใช้ข้อความต่อได้", } as const; diff --git a/src/s15/composition.ts b/src/s15/composition.ts index 2bc4ba4a..3f8e586b 100644 --- a/src/s15/composition.ts +++ b/src/s15/composition.ts @@ -1,10 +1,14 @@ import { SkillDirectory } from "../s16/skills/service"; +import { VoiceService } from "../s16/voice/service"; +import { openLocalVoice, type LocalVoiceConfig, type LocalVoiceEngine } from "../s16/voice/local"; +import type { VoiceQuotaConfig } from "../s16/voice/contracts"; +import { tmpdir } from 'node:os'; import { createSkillRuntime, skillVmSessions } from "../s16/skills/runtime"; import { skillDisclosure } from "../s16/skills/disclosure"; import { disclosureMemory } from "./disclosure/data-memory"; import { ComputerContinuations } from "../s16/computers/continuation"; import { websiteHostSensitive } from "./browser/sites"; -import { fsyncSync, lstatSync, mkdirSync, openSync, closeSync, readFileSync, renameSync, rmSync, writeFileSync } from 'node:fs'; +import { fsyncSync, lstatSync, mkdirSync, openSync, closeSync, readFileSync, realpathSync, renameSync, rmSync, writeFileSync } from 'node:fs'; import { isAbsolute, join, resolve } from 'node:path'; import { createS16ModelsPort } from '../s16/models'; import { createS16ClientHandler } from '../s16/http'; @@ -39,7 +43,7 @@ import { createHostCredentialLifecycle, type HostAuthFetch, type HostClock, type import { AppError } from '../errors'; import { openPolicyService, type ActionAdapter, type AuthenticatedOwner, type CatalogPort, type OwnerActor, type PolicyInput, type ReadHandler, type Ref, type S15PolicyService, type SubscriptionModels, type WorkActor } from './policy'; import { canonical, digest, equal, unknownUsage } from './policy/validation'; -import { ConversationApplication, ConversationStore, conversationRequestView, createConversationHandler, createConversationPolicy, createConversationVmTools, type ConversationPolicyWork, type ConversationToolPort } from './conversation'; +import { ConversationApplication, ConversationStore, conversationOwnerView, conversationRequestView, createConversationHandler, createConversationPolicy, createConversationVmTools, type ConversationPolicyWork, type ConversationToolPort } from './conversation'; import { WorkingVmService, VmEnrollment, VmManagers, VmPolicyIntegration, VmPool, VmStore, LocalVmProvisioning, createVmApi, isVmLocalManagement, placeWorkingVm, probeLocalVmHost, startLocalVmControl, type LocalVmControlOptions, type WorkingVmOptions } from './vm'; import { privateLocalVmResources } from './vm/local'; import { registerProtectedBrowser, type BrowserOptions, type BrowserRegistration } from './browser'; @@ -106,6 +110,9 @@ export interface S15CompositionOptions { connections?: { vault: import('../exec/connections').ConnectionVault; network?: SourceNetworkPorts }; /** Explicit SKILL16-01 pinned inert packages and guarded working-computer use; requires DATA15. */ skills?: { network?: SourceNetworkPorts }; + /** Explicit local owner voice, outside Bot execution. Scratch defaults outside state/backups; injected engine/scratch + * are trusted test/host seams, never runtime-client fields. A configured route changes no standing policy. */ + voice?: { local: LocalVoiceConfig; quota: VoiceQuotaConfig; engine?: () => Promise; scratchDirectory?: string }; /** Selected host transition. Status is read-only; its activity guard fences work and records released * activity durably. Composition cannot activate/release or import old authority through this port. */ adoption?: { service: Pick; authenticate(owner: OwnerActor): import('./adoption').AdoptionOwner }; @@ -175,7 +182,7 @@ export async function composeS15(options: S15CompositionOptions) { let skillCopyCurrent: (vmId: string, resource: Ref) => boolean = () => false; if (options.skills && options.disclosure) options = { ...options, disclosure: skillDisclosure(options.disclosure, { directory: () => skills, ownerGeneration: options.owner.generation, computer: (work, vmId) => skillComputer(work, vmId), copyCurrent: (vmId, resource) => skillCopyCurrent(vmId, resource), work: { ...s15DefaultRefs.work }, interpretation: { ...s15DefaultRefs.interpretation }, now: options.clock?.now ?? Date.now }) }; const root = options.directory, manifestPath = join(root, 'composition.json'); - const storeNames = ['conversation.sqlite', 'policy.sqlite', 'model.sqlite', 'work.sqlite', 'vm.sqlite', 'team.sqlite', 'memory.sqlite', ...(options.disclosure ? ['disclosure.sqlite'] : []), ...(options.browser ? ['browser.sqlite'] : []), ...(options.github || webGithubEnabled ? ['github.sqlite'] : []), ...(options.files ? ['files.sqlite'] : []), ...(options.routines ? ['routines.sqlite'] : []), ...(options.notifications ? ['notifications.sqlite'] : []), ...(options.bots ? ['bots.sqlite'] : []), ...(options.connections ? ['connections.sqlite'] : []), ...(options.skills ? ['skills.sqlite'] : [])]; + const storeNames = ['conversation.sqlite', 'policy.sqlite', 'model.sqlite', 'work.sqlite', 'vm.sqlite', 'team.sqlite', 'memory.sqlite', ...(options.disclosure ? ['disclosure.sqlite'] : []), ...(options.browser ? ['browser.sqlite'] : []), ...(options.github || webGithubEnabled ? ['github.sqlite'] : []), ...(options.files ? ['files.sqlite'] : []), ...(options.routines ? ['routines.sqlite'] : []), ...(options.notifications ? ['notifications.sqlite'] : []), ...(options.bots ? ['bots.sqlite'] : []), ...(options.connections ? ['connections.sqlite'] : []), ...(options.skills ? ['skills.sqlite'] : []), ...(options.voice ? ['voice.sqlite'] : [])]; const manifest = { contract: 's15-composition/v1', installationId: options.installationId, stores: storeNames }; if (options.create) { // mkdir without recursive prevents accidentally accepting an existing S14/legacy/partial directory. @@ -188,7 +195,7 @@ export async function composeS15(options: S15CompositionOptions) { // earlier manifest, which an earlier release needs to start) and may hold an armed effect, or a complete fresh // store (creation is atomic). Only a store this boot created is removed if the boot fails. A store the manifest // already lists must still exist; a missing listed ledger always refuses. - const additive = [...(options.browser?.approved ? ['browser.sqlite'] : []), ...(webGithubEnabled ? ['github.sqlite'] : []), ...(options.files ? ['files.sqlite'] : []), ...(options.routines ? ['routines.sqlite'] : []), ...(options.notifications ? ['notifications.sqlite'] : []), ...(options.bots ? ['bots.sqlite'] : []), ...(options.connections ? ['connections.sqlite'] : []), ...(options.skills ? ['skills.sqlite'] : [])]; + const additive = [...(options.browser?.approved ? ['browser.sqlite'] : []), ...(webGithubEnabled ? ['github.sqlite'] : []), ...(options.files ? ['files.sqlite'] : []), ...(options.routines ? ['routines.sqlite'] : []), ...(options.notifications ? ['notifications.sqlite'] : []), ...(options.bots ? ['bots.sqlite'] : []), ...(options.connections ? ['connections.sqlite'] : []), ...(options.skills ? ['skills.sqlite'] : []), ...(options.voice ? ['voice.sqlite'] : [])]; let added: string[] = []; const created: string[] = []; if (!options.create) { @@ -836,7 +843,13 @@ export async function composeS15(options: S15CompositionOptions) { return (['owner_named_send', 'github_default'] as const).some(mode => proposed.rules.some(r => r.mode === mode) && !current.rules.some(r => r.mode === mode)); } catch { return false; } }; - const clientHandler = createS16ClientHandler(conversation, { origin: options.origin, cookieName, authentication, memory, policyUpdate, models: createS16ModelsPort(models, options.model.registry, policy), ...(routines ? { routines } : {}), ...((routines || computerContinuations) ? { wake: () => { void composed.drain().catch(() => {}); } } : {}), ...(notifications ? { notifications } : {}), ...(computers ? { computers } : {}), ...(screen ? { screen } : {}), ...(browserCards ? { browserCards } : {}), ...(needsYou ? { needsYou } : {}), ...(bots ? { bots } : {}), ...(connections ? { connections } : {}), ...(skills ? { skills } : {}), ...(files ? { files, imageAvailable: () => { const route = options.model.registry.current(options.conversation.selection); return !!route && route.availability === "available" && !!options.files?.imageProbes?.some(probe => probe.verdict === "pass" && probe.route === route.provider && probe.model === route.modelId); } } : {}) }); + const voice = options.voice ? await VoiceService.open({ path: join(root, 'voice.sqlite'), installationId: options.installationId, create: options.create || created.includes('voice.sqlite'), limits: options.voice.quota, now: clockNow, + requireOwner, ownerGeneration: options.owner.generation, + conversation: (owner, id) => { conversationOwnerView(conversation, owner, id); }, + reply: (owner, id, requestId) => { const row = conversationRequestView(conversation, owner, requestId); if (row.receipt.conversationId !== id || !row.result) throw new AppError('voice_result_unavailable', 409); return { revision: row.revision, text: row.result.text }; }, + engine: options.voice.engine ?? (() => openLocalVoice(options.voice!.local, options.voice!.scratchDirectory ?? join(realpathSync(tmpdir()), 'jit-voice-' + new Bun.CryptoHasher('sha256').update(options.installationId + '\n' + realpathSync(root)).digest('hex').slice(0, 24)))) }) : undefined; + if (voice) cleanup.push(() => voice.close()); + const clientHandler = createS16ClientHandler(conversation, { origin: options.origin, cookieName, authentication, memory, policyUpdate, ...(voice ? { voice } : {}), models: createS16ModelsPort(models, options.model.registry, policy), ...(routines ? { routines } : {}), ...((routines || computerContinuations) ? { wake: () => { void composed.drain().catch(() => {}); } } : {}), ...(notifications ? { notifications } : {}), ...(computers ? { computers } : {}), ...(screen ? { screen } : {}), ...(browserCards ? { browserCards } : {}), ...(needsYou ? { needsYou } : {}), ...(bots ? { bots } : {}), ...(connections ? { connections } : {}), ...(skills ? { skills } : {}), ...(files ? { files, imageAvailable: () => { const route = options.model.registry.current(options.conversation.selection); return !!route && route.availability === "available" && !!options.files?.imageProbes?.some(probe => probe.verdict === "pass" && probe.route === route.provider && probe.model === route.modelId); } } : {}) }); const s16Web = await createS16WebHandler(clientHandler); const vmHandler = createVmApi(enrollment, async request => authenticate(request), { service: vm, ...(managers instanceof VmManagers ? { managers } : {}) }); const modelHandler = createModelProjectionHandler(models, authenticate), workHandler = createResponsibilityProjectionHandler(work, authenticate); @@ -854,7 +867,7 @@ export async function composeS15(options: S15CompositionOptions) { const dir = openSync(root, 'r'); try { fsyncSync(dir); } finally { closeSync(dir); } added = []; } - const composed = { policy, conversation, team, memory, files, bots, connections, skills, sources: sourceRuntime, disclosure, adoptionStatus, githubConnections, vm, workingVm, vmStore, enrollment, vmManagers: managers, vmPool: pool, provisioning, computers, computerScopes: scopes, screen, models, work, github, browser, authentication, resumes, routines, computerContinuations, + const composed = { policy, conversation, team, memory, files, bots, connections, skills, voice, sources: sourceRuntime, disclosure, adoptionStatus, githubConnections, vm, workingVm, vmStore, enrollment, vmManagers: managers, vmPool: pool, provisioning, computers, computerScopes: scopes, screen, models, work, github, browser, authentication, resumes, routines, computerContinuations, /** One protected scheduler wake; waiting records alone hold no VM/native session. Stall-reply resumes * settle after CHAT. */ drain(): Promise { if (closing) return Promise.resolve(); try { computerContinuations?.pump(); } catch (error) { return Promise.reject(error); } return draining ??= (async () => { try { reconcileVm(); try { routines?.tick(); routines?.dispatch(); } catch { /* Committed occurrences retry next wake. */ } await conversation.drain(); computerContinuations?.pump(); await conversation.drain(); computerContinuations?.pump(); try { routines?.observe(); } catch { /* Observed next wake. */ } try { resumes.settle(); } catch { /* Pending bindings stay pending. */ } await github?.drain(); } finally { draining = null; } })(); }, diff --git a/src/s15/main.ts b/src/s15/main.ts index 446402d3..ab0ea07c 100644 --- a/src/s15/main.ts +++ b/src/s15/main.ts @@ -133,6 +133,7 @@ export async function runS15(args: string[], env: NodeJS.ProcessEnv = process.en // CONN16-01/MCP16-01/OPEN16-01: apps and tool sources the owner connects; secrets stay in the protected vault. ...(config.connections ? { connections: { vault } } : {}), ...(config.skills ? { skills: {} } : {}), + ...(config.voice ? { voice: config.voice } : {}), ...(config.files ? { files: { ...(typeof config.files === 'object' ? { parserRuntime: binaryFileRuntime(config.files, config.directory, env), imageProbes: config.files.imageProbes } : {}), ...(config.fileMemory ? { memory: true } : {}) } } : {}), recipients, owner: { authenticate: token => { const auth = ownerApp.authenticate(token); return { proof: auth.actor, csrf: auth.csrf }; }, generation, verifyOwner: requireOwner, verifyCsrf: (expected, supplied) => ownerApp.verifyCsrf(expected, supplied) }, catalogs, diff --git a/src/s15/runtime-config.ts b/src/s15/runtime-config.ts index 7c496092..0600dba4 100644 --- a/src/s15/runtime-config.ts +++ b/src/s15/runtime-config.ts @@ -10,6 +10,9 @@ import type { GithubRuntimeConfig } from './github/runtime'; import type { RuntimeDisclosureConfig } from './disclosure/runtime'; import type { S15CompositionOptions } from './composition'; import { validateBinaryFileConfig, type BinaryFileConfig } from '../s16/files/config'; +import { validateLocalVoiceConfig, type LocalVoiceConfig } from '../s16/voice/local'; +import { validateVoiceQuota } from '../s16/voice/service'; +import type { VoiceQuotaConfig } from '../s16/voice/contracts'; /** Credential-free explicit source/disposable-service configuration; never an existing omitted-runtime service file. */ export interface S15RuntimeConfig { @@ -43,6 +46,8 @@ export interface S15RuntimeConfig { connections?: boolean; /** SKILL16-01 pinned package import/review and working-computer use; requires DATA15. */ skills?: boolean; + /** Explicit configured local owner speech. No credential, provider URL, arbitrary flags or automatic installation. */ + voice?: { local: LocalVoiceConfig; quota: VoiceQuotaConfig }; recurring?: import('./policy/recurring').RecurringRuleOptions; /** * The team's working-computer pool (POOL15-01). `id` names the pool and its default member; with `local` the @@ -91,6 +96,10 @@ export function parseS15RuntimeConfig(raw: unknown): S15RuntimeConfig { if (value.bots !== undefined && typeof value.bots !== 'boolean') throw new AppError('invalid_s15_config'); if (value.skills !== undefined && (typeof value.skills !== 'boolean' || value.skills && !value.disclosure)) throw new AppError('invalid_s15_config'); if (value.connections !== undefined && (typeof value.connections !== 'boolean' || value.connections && !value.disclosure)) throw new AppError('invalid_s15_config'); + if (value.voice !== undefined) { + try { if (!value.voice || typeof value.voice !== 'object' || Object.keys(value.voice).sort().join() !== 'local,quota') throw Error(); validateLocalVoiceConfig(value.voice.local); validateVoiceQuota(value.voice.quota); } + catch { throw new AppError('invalid_s15_config'); } + } if (value.policy && (!value.disclosure || value.policy.resources.some(r => !value.disclosure!.sources.some(s => equal(s.resource, r))))) throw new AppError('classification_unknown'); return value; } diff --git a/src/s16/client/client.ts b/src/s16/client/client.ts index bddccb0f..a4ab923f 100644 --- a/src/s16/client/client.ts +++ b/src/s16/client/client.ts @@ -1,4 +1,6 @@ import type { SkillCommand, SkillCommandStatus, SkillDependencyBinding, SkillFile, SkillReceipt } from "../skills/types"; +import { voiceRequestId, type VoiceCapture, type VoiceCaptureCommand, type VoicePlayback, type VoicePlaybackCommand, type VoiceStatus, type VoiceTranscript } from "../voice/contracts"; +import { captureWavInfo, VoiceAudioError } from "../voice/wav"; import type { SkillDetail, SkillsView } from "../skills/owner-view"; import type { ComputerNeedsYouCard, ComputerNeedsYouCommand, ComputerNeedsYouResult } from "../computers/needs-you-types"; import type { ClientModelsView } from "./contracts"; @@ -63,6 +65,61 @@ export class JitClient { if (!count(data.budget.unsettled)) return invalid(); return data as unknown as ClientModelsView; } + /** Read configured local speech health/quota without microphone, model or process activity. */ + async voiceStatus(signal?: AbortSignal): Promise { + const data = await this.call('/voice', 'GET', undefined, signal); + if (!record(data) || data.contract !== 's16-voice/v1' || typeof data.available !== 'boolean' || data.route !== 'local' || !count(data.serverTime) || !record(data.limits) || !count(data.limits.captureMs) || !count(data.limits.dailyMs) || !this.voiceUsage(data.usage)) return invalid(); + return data as unknown as VoiceStatus; + } + /** Freeze one intentional Mic identity using the service's clock sample, retained through uncertain responses. */ + prepareDictation(serverTime: number): VoiceCaptureCommand { return Object.freeze({ contract: 's16-voice-capture/v1', requestId: voiceRequestId(serverTime) }); } + /** Explicit reserve only; the caller requests microphone permission separately after seeing this receipt. */ + async beginDictation(conversationId: string, command: VoiceCaptureCommand, signal?: AbortSignal): Promise { + const input = structuredClone(command); id(conversationId); id(input.requestId); + const data = await this.call(`/conversations/${conversationId}/voice/captures`, 'POST', input, signal, input.requestId); + if (!this.voiceCapture(data, input.requestId, conversationId)) return invalid(); return data as unknown as VoiceCapture; + } + /** Inspect an original draft; no microphone capture or transcription is reconstructed. */ + async dictationStatus(capture: Pick, signal?: AbortSignal): Promise { + const input = structuredClone(capture); id(input.id); id(input.conversationId); + const data = await this.call(`/voice/${input.id}`, 'GET', undefined, signal); + if (!this.voiceCapture(data, input.id, input.conversationId)) return invalid(); return data as unknown as VoiceCapture; + } + /** Submit strict snapshotted PCM once. The transcript remains editable data; this never calls Send or Decide. */ + async transcribeDictation(capture: Pick, bytes: Uint8Array, signal?: AbortSignal): Promise { + const input = structuredClone(capture), wav = new Uint8Array(bytes); id(input.id); id(input.conversationId); + try { captureWavInfo(wav); } catch (error) { throw new JitClientError(error instanceof VoiceAudioError ? error.code : 'voice_audio_invalid', null, 'not_sent', input.id); } + const data = await this.call(`/voice/${input.id}/transcribe`, 'POST', undefined, signal, input.id, wav); + if (!record(data) || data.contract !== 's16-voice-transcript/v1' || !this.voiceCapture(data.capture, input.id, input.conversationId) || data.capture.state !== 'ready' || typeof data.text !== 'string' || new TextEncoder().encode(data.text).length > 16384 || !this.voiceUsage(data.usage)) return invalid(); + return data as unknown as VoiceTranscript; + } + /** Stop only voice; underlying conversation work and exact approval state remain unchanged. */ + async cancelVoice(operation: { id: string; conversationId: string }, signal?: AbortSignal): Promise { + const input = structuredClone(operation); id(input.id); id(input.conversationId); + const data = await this.call(`/voice/${input.id}/cancel`, 'POST', {}, signal, input.id); + if (!this.voiceCapture(data, input.id, input.conversationId) || data.state !== 'cancelled') return invalid(); return data as unknown as VoiceCapture; + } + /** Freeze a specific currently displayed reply; hashing is local and does not start synthesis or work. */ + async preparePlayback(serverTime: number, conversationId: string, requestId: string, revision: number, text: string): Promise { + const generation = this.#sessionGeneration; id(conversationId); id(requestId); + if (!count(revision) || !revision || typeof text !== 'string' || !text.trim() || text.length > 2000 || new TextEncoder().encode(text).length > 8192) throw new JitClientError('voice_text_limit', null, 'not_sent'); + const textSha256 = Array.from(new Uint8Array(await crypto.subtle.digest('SHA-256', new TextEncoder().encode(text))), byte => byte.toString(16).padStart(2, '0')).join(''); + if (generation !== this.#sessionGeneration) throw new JitClientError('client_response_stale', null, 'not_sent'); + return Object.freeze({ contract: 's16-voice-playback/v1', requestId: voiceRequestId(serverTime), conversationId, targetRequestId: requestId, targetRevision: revision, textSha256 }); + } + /** Synthesize once; late/stale replies use the same transport generation fences as ordinary owner projections. */ + async playReply(command: VoicePlaybackCommand, signal?: AbortSignal): Promise { + const input = structuredClone(command); id(input.requestId); id(input.conversationId); id(input.targetRequestId); + const data = await this.call('/voice/playback', 'POST', input, signal, input.requestId); + if (!record(data) || data.contract !== 's16-voice-playback/v1' || data.id !== input.requestId || data.conversationId !== input.conversationId || data.targetRequestId !== input.targetRequestId || !count(data.durationMs) || (data.durationMs as number) > 60000 || typeof data.wavBase64 !== 'string' || data.wavBase64.length > 5592408 || !this.voiceUsage(data.usage)) return invalid(); + try { const decoded = atob(data.wavBase64); if (!decoded.length || decoded.length > 4 * 1024 * 1024 || btoa(decoded) !== data.wavBase64 || decoded.slice(0, 4) !== 'RIFF' || decoded.slice(8, 12) !== 'WAVE') return invalid(); } + catch { return invalid(); } + return data as unknown as VoicePlayback; + } + private voiceUsage(value: unknown): boolean { return record(value) && value.unit === 'local_audio_ms' && typeof value.day === 'string' && /^\d{4}-\d\d-\d\d$/.test(value.day) && [value.limitMs, value.chargedMs, value.reservedMs, value.conservativeMs].every(count); } + private voiceCapture(value: unknown, expectedId: string, conversationId: string): value is VoiceCapture { + return record(value) && value.id === expectedId && value.conversationId === conversationId && ['capturing', 'processing', 'ready', 'cancelled', 'failed', 'unknown', 'expired'].includes(String(value.state)) && count(value.recordingEndsAt) && count(value.expiresAt) && (value.recordingEndsAt as number) <= (value.expiresAt as number) && count(value.maxDurationMs) && (value.maxDurationMs as number) <= 60000; + } /** Creates one local immutable send identity; callers retain this command through response loss. */ prepareMessage(conversationId: string, text: string, files: readonly FileReference[] = []): ClientMessage { id(conversationId); diff --git a/src/s16/client/contracts.ts b/src/s16/client/contracts.ts index 52eac874..3276ee70 100644 --- a/src/s16/client/contracts.ts +++ b/src/s16/client/contracts.ts @@ -34,6 +34,8 @@ export interface ClientCapabilities { skills?: { available: false; code: "skills_unavailable" } | { available: true; contract: "s16-client-skills/v1" }; /** MODEL16-01: configured routes and recorded work usage; reads never start provider work. */ models?: { available: true; contract: "s16-client-models/v1" }; + /** Owner audio API is configured; actual local binary/model health and quota are read from its status endpoint. */ + voice?: { available: false; code: "voice_unavailable" } | { available: true; contract: "s16-voice/v1"; route: "local" }; } /** Authenticated, no-store bootstrap. CSRF stays in the transport, outside persisted client state. */ export interface ClientBootstrap { diff --git a/src/s16/client/index.ts b/src/s16/client/index.ts index 8f36aa4d..aa96a641 100644 --- a/src/s16/client/index.ts +++ b/src/s16/client/index.ts @@ -2,6 +2,7 @@ export { JitClient, JitClientError } from "./client"; export { beginConversation, emptyConversationState, receiveConversation, withholdConversation } from "./state"; export type * from "./contracts"; +export type * from "../voice/contracts"; export type { ConversationClientState } from "./state"; export { applicationServerKey, disableNotifications, enableNotifications, endpointSha256, notificationLocale, notificationStatus, notificationSupport, notificationWorker, sameApplicationServerKey } from "./notifications"; export type { NotificationEnvironment, NotificationStatus, NotificationSupport } from "./notifications"; diff --git a/src/s16/client/voice-recording.ts b/src/s16/client/voice-recording.ts new file mode 100644 index 00000000..73a8d52d --- /dev/null +++ b/src/s16/client/voice-recording.ts @@ -0,0 +1,70 @@ +import { encodeCaptureWav } from '../voice/wav'; + +/** Fixed authenticated same-origin AudioWorklet module; callers never supply scripts or a remote recognizer. */ +export const voiceWorkletPath = '/s16/voice-worklet.js'; +/** One finite local microphone lease. Stop produces a draft clip; cancel discards it and never submits text. */ +export interface VoiceRecording { done: Promise; stop(): void; cancel(): Promise } +const failure = (code: string) => new Error(code); +const stopTracks = (stream?: MediaStream) => { for (const track of stream?.getTracks() ?? []) track.stop(); }; + +/** Explicit gesture-only capture with late-permission fencing, finite PCM buffers and native track/node cleanup. */ +export async function startVoiceRecording(options: { signal: AbortSignal; durationMs: number; deadline?: number; progress(ms: number): void }): Promise { + if (!navigator.mediaDevices?.getUserMedia || typeof AudioContext !== 'function' || typeof AudioWorkletNode !== 'function') throw failure('voice_capture_unsupported'); + if (!Number.isFinite(options.durationMs) || options.durationMs < 1 || options.durationMs > 30000) throw failure('voice_capture_expired'); + if (options.signal.aborted) throw failure('voice_cancelled'); + let stream: MediaStream | undefined, context: AudioContext | undefined, source: MediaStreamAudioSourceNode | undefined, node: AudioWorkletNode | undefined, silent: GainNode | undefined; + const deadline = options.deadline ?? performance.now() + options.durationMs + 15000; + if (!Number.isFinite(deadline) || deadline <= performance.now()) throw failure('voice_capture_expired'); + let stopping = false, finished = false, frames = 0, maximum = Math.floor(options.durationMs * 16), chunks: Float32Array[] = []; + let timeout: ReturnType | undefined, watchdog: ReturnType | undefined; + const complete = Promise.withResolvers(); void complete.promise.catch(() => {}); + const interrupted = Promise.withResolvers(); void interrupted.promise.catch(() => {}); + let closing: Promise | null = null; + const close = (): Promise => { + stopping = true; stopTracks(stream); clearTimeout(timeout); clearTimeout(watchdog); + if (node) { node.port.onmessage = null; node.port.close(); node.disconnect(); } source?.disconnect(); silent?.disconnect(); + return closing ??= (async () => { if (context && context.state !== 'closed') await context.close().catch(() => {}); })(); + }; + const refuse = async (code: string) => { interrupted.reject(failure(code)); if (finished) return; finished = true; chunks = []; await close(); complete.reject(failure(code)); }; + const abort = () => { void refuse('voice_cancelled'); }; + options.signal.addEventListener('abort', abort, { once: true }); + const done = complete.promise.finally(() => options.signal.removeEventListener('abort', abort)); void done.catch(() => {}); + timeout = setTimeout(() => { void refuse('voice_capture_expired'); }, Math.max(0, deadline - performance.now())); + try { + const pending = navigator.mediaDevices.getUserMedia({ audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true }, video: false }); + // A cancelled permission prompt may resolve later. Its stream is stopped even when this caller already returned. + void pending.then(value => { if (options.signal.aborted || finished) stopTracks(value); }, () => {}); + stream = await Promise.race([pending, interrupted.promise]); + if (options.signal.aborted || finished) { stopTracks(stream); throw failure('voice_cancelled'); } + const lost = () => { if (!stopping) void refuse('voice_permission_lost'); }; + for (const track of stream.getAudioTracks()) { track.addEventListener('ended', lost, { once: true }); track.addEventListener('mute', lost, { once: true }); } + context = new AudioContext(); await Promise.race([context.audioWorklet.addModule(voiceWorkletPath, { credentials: 'same-origin' }), interrupted.promise]); + if (options.signal.aborted || finished) throw failure('voice_cancelled'); + maximum = Math.floor(Math.min(options.durationMs, deadline - performance.now()) * 16); if (maximum < 1) throw failure('voice_capture_expired'); + node = new AudioWorkletNode(context, 'jit-voice-capture', { numberOfInputs: 1, numberOfOutputs: 1, outputChannelCount: [1], processorOptions: { maximumFrames: maximum } }); + const finish = async () => { + if (finished) return; finished = true; + try { + if (!frames) throw failure('voice_audio_empty'); + const samples = new Float32Array(frames); let at = 0; for (const chunk of chunks) { samples.set(chunk, at); at += chunk.length; } + chunks = []; const wav = encodeCaptureWav(samples); await close(); + if (options.signal.aborted) throw failure('voice_cancelled'); complete.resolve(wav); + } catch (error) { chunks = []; await close(); complete.reject(error); } + }; + node.port.onmessage = event => { + if (finished || options.signal.aborted) return; + const data = event.data; + if (data?.kind === 'samples' && data.samples instanceof Float32Array && data.samples.length > 0 && data.samples.length <= 1024 && frames + data.samples.length <= maximum) { chunks.push(data.samples); frames += data.samples.length; options.progress(frames / 16); } + else if (data?.kind === 'ended') void finish(); + else void refuse('voice_audio_invalid'); + }; + source = context.createMediaStreamSource(stream); silent = context.createGain(); silent.gain.value = 0; source.connect(node); node.connect(silent); silent.connect(context.destination); + await Promise.race([context.resume(), interrupted.promise]); if (options.signal.aborted) throw failure('voice_cancelled'); + const stop = () => { if (finished || stopping) return; stopping = true; stopTracks(stream); clearTimeout(timeout); node!.port.postMessage({ kind: 'stop' }); watchdog = setTimeout(() => { void refuse('voice_capture_unavailable'); }, 1500); }; + if (!finished) { clearTimeout(timeout); timeout = setTimeout(stop, Math.max(0, Math.min(options.durationMs, deadline - performance.now()))); } + return { done, stop, async cancel() { await refuse('voice_cancelled'); await close(); } }; + } catch (error) { + const name = error instanceof Error ? error.name : '', code = error instanceof Error && /^voice_[a-z_]+$/.test(error.message) ? error.message : name === 'NotAllowedError' ? 'voice_permission_denied' : ['NotFoundError', 'NotReadableError'].includes(name) ? 'voice_microphone_unavailable' : 'voice_capture_unavailable'; + await refuse(code); await close(); throw failure(code); + } +} diff --git a/src/s16/http.ts b/src/s16/http.ts index ef86b986..789c858a 100644 --- a/src/s16/http.ts +++ b/src/s16/http.ts @@ -1,4 +1,7 @@ import type { SkillDirectory } from "./skills/service"; +import type { VoiceService } from "./voice/service"; +import type { VoiceCaptureCommand, VoicePlaybackCommand } from "./voice/contracts"; +import { captureAudio, VoiceAudioError } from "./voice/wav"; import { skillDetail, skillFile, skillsView } from "./skills/owner-view"; import { skillLimits } from "./skills/types"; import { AppError } from "../errors"; @@ -30,7 +33,7 @@ import { connectionLimits } from "./connections/types"; export const artifactPreviewPolicy = "sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src 'none'; form-action 'none'; base-uri 'none'; frame-ancestors 'self'"; /** Protected shared client adapter. It adds no engine, database, approval authority or automatic retry. */ -export function createS16ClientHandler(app: ConversationApplication, options: { origin: string; cookieName: string; /** AUTH16-01 banner: true while a newer default proposal awaits the owner. */ policyUpdate?: (actor: Actor) => boolean; authentication: ConversationHttpAuthentication; models?: S16ModelsPort; files?: FileStore; imageAvailable?: () => boolean; memory?: S16MemoryPort; routines?: RoutineService; wake?: () => void; notifications?: NotificationService; computers?: ComputerDirectory; /** DESK16-01 watch and take over a working computer's screen (`s16-computer-screen/v1`). */ screen?: ScreenControl; browserCards?: BrowserActionCards; needsYou?: ComputerNeedsYou; /** BOT16-01 persistent Bots (`s16-client-bots/v1`). */ bots?: BotDirectory; /** CONN16-01 connections (`s16-client-connections/v1`). */ connections?: ConnectionDirectory; /** SKILL16-01 inert owner review and pin/receipt recovery. */ skills?: SkillDirectory }): (request: Request) => Promise { +export function createS16ClientHandler(app: ConversationApplication, options: { origin: string; cookieName: string; /** AUTH16-01 banner: true while a newer default proposal awaits the owner. */ policyUpdate?: (actor: Actor) => boolean; authentication: ConversationHttpAuthentication; models?: S16ModelsPort; files?: FileStore; imageAvailable?: () => boolean; memory?: S16MemoryPort; routines?: RoutineService; wake?: () => void; notifications?: NotificationService; computers?: ComputerDirectory; /** DESK16-01 watch and take over a working computer's screen (`s16-computer-screen/v1`). */ screen?: ScreenControl; browserCards?: BrowserActionCards; needsYou?: ComputerNeedsYou; /** BOT16-01 persistent Bots (`s16-client-bots/v1`). */ bots?: BotDirectory; /** CONN16-01 connections (`s16-client-connections/v1`). */ connections?: ConnectionDirectory; /** SKILL16-01 inert owner review and pin/receipt recovery. */ skills?: SkillDirectory; /** Local owner audio; creates no Bot work or decisions. */ voice?: VoiceService }): (request: Request) => Promise { const origin = new URL(options.origin); if (origin.origin !== options.origin || !/^[A-Za-z0-9_-]+$/.test(options.cookieName)) throw new AppError("invalid_configuration"); const json = (value: unknown, status = 200): Response => { @@ -65,6 +68,7 @@ export function createS16ClientHandler(app: ConversationApplication computers: options.computers ? { available: true, contract: "s16-client-computers/v1", idleStopMs: options.computers.options.vm.idleStopAfterMs, files: !!(files && app.options.files), ...(options.screen ? { screen: true } : {}) } : { available: false, code: "computers_unavailable" }, bots: options.bots ? { available: true, contract: "s16-client-bots/v1", delegation: true } : { available: false, code: "bots_unavailable" }, skills: options.skills ? { available: true, contract: "s16-client-skills/v1" } : { available: false, code: "skills_unavailable" }, + voice: options.voice ? { available: true, contract: "s16-voice/v1", route: "local" } : { available: false, code: "voice_unavailable" }, connections: options.connections ? { available: true, contract: "s16-client-connections/v1", kinds: ["mcp", "openapi", "graphql"], github: !!options.connections.options.github, oauth: true } : { available: false, code: "connections_unavailable" } }; }; return async request => { @@ -85,6 +89,32 @@ export function createS16ClientHandler(app: ConversationApplication if (request.method === "POST") options.authentication.verifyCsrf(current.csrf, request.headers.get("x-csrf-token")); return current.actor; }; + const voicePath = /^\/api\/s16\/client\/voice(?:\/(playback|[A-Za-z0-9_-]{1,128})(?:\/(transcribe|cancel))?)?$/.exec(url.pathname); + const capturePath = /^\/api\/s16\/client\/conversations\/([A-Za-z0-9_-]{1,128})\/voice\/captures$/.exec(url.pathname); + if (voicePath || capturePath) { + if (!options.voice) throw new AppError('voice_unavailable', 503); + if (url.search) throw new AppError('invalid_body'); + if (capturePath && request.method === 'POST') { const capture = await options.voice.begin(currentOwner, capturePath[1]!, await body(request) as VoiceCaptureCommand, request.signal); return json(options.voice.inspect(currentOwner, capture.id), 201); } + if (voicePath && request.method === 'GET' && !voicePath[1]) { const status = await options.voice.status(currentOwner); currentOwner(); return json(status); } + if (voicePath?.[1] === 'playback' && !voicePath[2] && request.method === 'POST') { const input = await body(request) as VoicePlaybackCommand; await options.voice.playback(currentOwner, input, request.signal); return options.voice.publishPlayback(currentOwner, input, value => json(value)); } + if (voicePath?.[1] && voicePath[1] !== 'playback') { + const captureId = voicePath[1]; + if (request.method === 'GET' && !voicePath[2]) return json(options.voice.inspect(currentOwner, captureId)); + if (request.method === 'POST' && voicePath[2] === 'cancel') { exact(await body(request), []); const cancelled = await options.voice.cancel(currentOwner, captureId); currentOwner(); return json(cancelled); } + if (request.method === 'POST' && voicePath[2] === 'transcribe') { + if (request.headers.get('content-type') !== 'application/octet-stream' || !request.body) throw new AppError('unsupported_content_type', 415); + const capture = options.voice.inspect(currentOwner, captureId); if (capture.state === 'expired') throw new AppError('voice_capture_expired', 409); + const reader = request.body.getReader(), chunks: Uint8Array[] = []; let length = 0, wav: Buffer; + try { + for (;;) { const chunk = await reader.read(); currentOwner(); if (chunk.done) break; length += chunk.value.byteLength; if (length > captureAudio.bytes) throw new AppError('body_too_large', 413); chunks.push(chunk.value); } + currentOwner(); wav = Buffer.concat(chunks); + } finally { await reader.cancel().catch(() => {}); reader.releaseLock(); } + currentOwner(); await options.voice.transcribe(currentOwner, captureId, wav!, request.signal); + return options.voice.publishTranscript(currentOwner, captureId, value => json(value)); + } + } + throw new AppError('method_rejected', 405); + } const uploadCreate = /^\/api\/s16\/client\/conversations\/([A-Za-z0-9_-]{1,128})\/uploads$/.exec(url.pathname); if (uploadCreate && request.method === "POST") { if (!options.files || !app.options.files) throw new AppError("upload_unavailable", 503); @@ -329,6 +359,6 @@ export function createS16ClientHandler(app: ConversationApplication app.stop(current.actor, work[1]!); return json({ state: "stop_requested" }); } throw new AppError("not_found", 404); - } catch (error) { return json({ error: error instanceof AppError ? error.code : "client_unavailable" }, error instanceof AppError ? error.status : 503); } + } catch (error) { return json({ error: error instanceof AppError || error instanceof VoiceAudioError ? error.code : "client_unavailable" }, error instanceof AppError ? error.status : error instanceof VoiceAudioError ? 400 : 503); } }; } diff --git a/src/s16/voice/contracts.ts b/src/s16/voice/contracts.ts new file mode 100644 index 00000000..e86446c6 --- /dev/null +++ b/src/s16/voice/contracts.ts @@ -0,0 +1,23 @@ +/** Finite owner-client voice support; audio seconds do not measure compute time or provider billing. */ +export const voiceLimits = Object.freeze({ captureMs: 30000, permissionMs: 15000, submissionMs: 15000, synthesisMs: 60000, transcriptBytes: 16384, responseCacheBytes: 8 * 1024 * 1024, responseCacheMs: 300000, retentionMs: 7 * 86400000, requestAgeMs: 300000, operations: 4096 }); +/** Original request identity embeds its issuance time, so a pruned old command cannot become a fresh capture. */ +export function voiceRequestId(serverTime: number): string { + if (!Number.isSafeInteger(serverTime) || serverTime < 0) throw Error('voice_time_invalid'); + return `v_${serverTime}_${crypto.randomUUID().replaceAll('-', '')}`; +} +/** Explicit local audio budget. Values can narrow, never exceed, published per-operation bounds. */ +export interface VoiceQuotaConfig { dailyMs: number; captureMs: number } +/** Metadata-only local quota; unknown processing retains its conservative charge. All milliseconds round up. */ +export interface VoiceUsage { day: string; unit: 'local_audio_ms'; limitMs: number; chargedMs: number; reservedMs: number; conservativeMs: number } +/** Capture is a private draft operation, never work/approval. Stop by recordingEndsAt; expiresAt separately bounds submission. */ +export interface VoiceCapture { id: string; conversationId: string; state: 'capturing' | 'processing' | 'ready' | 'cancelled' | 'failed' | 'unknown' | 'expired'; recordingEndsAt: number; expiresAt: number; maxDurationMs: number } +/** Read-only configured capability and current local quota; paths, credentials and model data are absent. */ +export interface VoiceStatus { contract: 's16-voice/v1'; available: boolean; serverTime: number; route: 'local'; limits: VoiceQuotaConfig; usage: VoiceUsage } +/** The owner's explicit Mic intent. It starts no conversation request or model operation. */ +export interface VoiceCaptureCommand { contract: 's16-voice-capture/v1'; requestId: string } +/** Only the exact currently published result can be synthesized; client text is never the source. */ +export interface VoicePlaybackCommand { contract: 's16-voice-playback/v1'; requestId: string; conversationId: string; targetRequestId: string; targetRevision: number; textSha256: string } +/** Editable untrusted draft and its original capture identity. Ordinary Send remains a separate command. */ +export interface VoiceTranscript { contract: 's16-voice-transcript/v1'; capture: VoiceCapture; text: string; usage: VoiceUsage } +/** Bounded audio for one currently guarded result; interruption never implies work/effect cancellation. */ +export interface VoicePlayback { contract: 's16-voice-playback/v1'; id: string; conversationId: string; targetRequestId: string; wavBase64: string; durationMs: number; usage: VoiceUsage } diff --git a/src/s16/voice/local-worker.ts b/src/s16/voice/local-worker.ts new file mode 100644 index 00000000..9f1ad669 --- /dev/null +++ b/src/s16/voice/local-worker.ts @@ -0,0 +1,50 @@ +// Private finite process supervisor. Only its parent supplies a job over a fresh anonymous pipe. +// Keeping stdin open is the parent's liveness lease: EOF/SIGTERM/deadline stops and joins the CLI. +import { lstatSync, renameSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; + +if (import.meta.main) { + if (process.argv[2] !== '--local-voice-worker') process.exit(2); + let child: ReturnType | undefined, reason: string | null = null, finished = false, directory: string | undefined = process.cwd(); + const stop = (code: string) => { reason ??= code; if (child && !finished) { try { child.kill('SIGKILL'); } catch { /* Exit may have won the race. */ } } }; + process.on('SIGTERM', () => stop('cancelled')); process.on('SIGINT', () => stop('cancelled')); + const respond = (state: string) => { + finished = true; + // A restarted service may remove this private scratch only after the actual CLI was joined. + if (directory) { + const temporary = join(directory, 'settled.tmp'); + writeFileSync(temporary, JSON.stringify({ state, worker: process.pid, child: child?.pid ?? null }), { mode: 0o600 }); + renameSync(temporary, join(directory, 'settled.json')); + } + process.stdout.write(JSON.stringify({ state }) + '\n'); process.exit(0); + }; + const run = async (raw: string) => { + let deadline: ReturnType | undefined, size: ReturnType | undefined; + try { + const job = JSON.parse(raw) as { argv: string[]; cwd: string; timeoutMs: number; output: string; bytes: number }; + if (!job || Object.keys(job).sort().join() !== 'argv,bytes,cwd,output,timeoutMs' || !Array.isArray(job.argv) || !job.argv.length || job.argv.length > 32 + || job.argv.some(value => typeof value !== 'string' || !value || value.length > 4096 || value.includes('\0')) || !job.argv[0]!.startsWith('/') + || typeof job.cwd !== 'string' || !job.cwd.startsWith('/') || typeof job.output !== 'string' || !job.output.startsWith(job.cwd + '/') + || !Number.isSafeInteger(job.timeoutMs) || job.timeoutMs < 1 || job.timeoutMs > 120000 || !Number.isSafeInteger(job.bytes) || job.bytes < 1 || job.bytes > 4 * 1024 * 1024) return respond('failed'); + directory = job.cwd; + if (reason) return respond(reason); + child = Bun.spawn(job.argv, { cwd: job.cwd, stdin: 'ignore', stdout: 'ignore', stderr: 'ignore', env: { PATH: '/usr/bin:/bin', LANG: 'C.UTF-8', LC_ALL: 'C.UTF-8', TMPDIR: job.cwd } }); + deadline = setTimeout(() => stop('timeout'), job.timeoutMs); + size = setInterval(() => { try { if (lstatSync(job.output).size > job.bytes) stop('limit'); } catch { /* The CLI may not have created its output yet. */ } }, 20); + const code = await child.exited; + respond(reason ?? (code === 0 ? 'completed' : 'failed')); + } catch { stop('failed'); if (child) await child.exited; respond(reason ?? 'failed'); } + finally { clearTimeout(deadline); clearInterval(size); } + }; + let input = '', started = false; const decoder = new TextDecoder('utf-8', { fatal: true }); + try { + for await (const chunk of Bun.stdin.stream()) { + if (started) { if (chunk.byteLength) stop('failed'); continue; } + input += decoder.decode(chunk, { stream: true }); + if (Buffer.byteLength(input) > 16384) { stop('failed'); respond('failed'); } + const end = input.indexOf('\n'); + if (end >= 0) { started = true; if (input.slice(end + 1)) process.exit(2); void run(input.slice(0, end)); input = ''; } + } + stop('cancelled'); if (!started) respond('cancelled'); + } catch { stop('failed'); if (!started) respond('failed'); } +} diff --git a/src/s16/voice/local.ts b/src/s16/voice/local.ts new file mode 100644 index 00000000..fd286c8d --- /dev/null +++ b/src/s16/voice/local.ts @@ -0,0 +1,180 @@ +import { createHash } from 'node:crypto'; +import { chmod, lstat, mkdir, mkdtemp, open, readdir, realpath, rm, writeFile } from 'node:fs/promises'; +import { constants, writeFileSync } from 'node:fs'; +import { isAbsolute, join, resolve } from 'node:path'; +import { AppError } from '../../errors'; +import { captureWavInfo } from './wav'; + +/** Operator-supplied trusted artifact. A hash pin is not a sandbox or a dependency-tree attestation. */ +export interface VoiceArtifact { path: string; sha256: string } +/** Explicit local-only route: no URL, provider credential, shell command, custom flags or fallback. */ +export interface LocalVoiceConfig { + transcription: { executable: VoiceArtifact; model: VoiceArtifact; language: string }; + synthesis: { kind: 'espeak' | 'say'; executable: VoiceArtifact; voice: string }; + timeoutMs: number; +} +/** Host-only final checks and durable process-start accounting; never supplied by a client/model. */ +export interface VoiceProcessContext { signal: AbortSignal; guard(): void; started(): void } +/** Finite local adapter. Raw audio/text stays in memory/private scratch; errors contain safe codes only. */ +export interface LocalVoiceEngine { + available(): Promise; + transcribe(wav: Uint8Array, context: VoiceProcessContext): Promise; + synthesize(text: string, context: VoiceProcessContext): Promise<{ wav: Uint8Array; durationMs: number }>; + close(): Promise; +} +/** Local synthesis/output bounds, independent of owner-configured quota. */ +export const localSpeechLimits = Object.freeze({ textCharacters: 2000, textBytes: 8192, transcriptBytes: 16384, audioBytes: 4 * 1024 * 1024, durationMs: 60000 }); +const reject = (code: string): never => { throw new AppError(code, 503); }; +const record = (value: unknown, keys: string[]): value is Record => !!value && typeof value === 'object' && !Array.isArray(value) && Object.getPrototypeOf(value) === Object.prototype && Object.keys(value).sort().join() === keys.sort().join(); +const artifact = (value: unknown): value is VoiceArtifact => record(value, ['path', 'sha256']) && typeof value.path === 'string' && isAbsolute(value.path) && resolve(value.path) === value.path && !/[\x00-\x1f\x7f]/.test(value.path) && value.path.length < 2048 && typeof value.sha256 === 'string' && /^[a-f0-9]{64}$/.test(value.sha256); + +async function boundedFile(path: string, max: number): Promise { + const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await handle.stat(); if (!stat.isFile() || stat.size < 1 || stat.size > max) reject('voice_audio_limit'); + const buffer = Buffer.alloc(Math.min(65536, max + 1)), chunks: Buffer[] = []; let total = 0; + for (;;) { const read = await handle.read(buffer, 0, Math.min(buffer.length, max + 1 - total), null); if (!read.bytesRead) break; total += read.bytesRead; if (total > max) reject('voice_audio_limit'); chunks.push(Buffer.from(buffer.subarray(0, read.bytesRead))); } + if (total !== stat.size) reject('voice_audio_limit'); return Buffer.concat(chunks); + } finally { await handle.close(); } +} + +/** Dependency-free configuration shape validation; actual regular-file hashes are checked before every process. */ +export function validateLocalVoiceConfig(value: unknown): asserts value is LocalVoiceConfig { + if (!record(value, ['transcription', 'synthesis', 'timeoutMs']) || !record(value.transcription, ['executable', 'model', 'language']) || !record(value.synthesis, ['kind', 'executable', 'voice']) + || !artifact(value.transcription.executable) || !artifact(value.transcription.model) || !artifact(value.synthesis.executable) + || typeof value.transcription.language !== 'string' || !/^[a-z]{2,3}(?:-[A-Za-z]{2,8})?$|^auto$/.test(value.transcription.language) + || !['espeak', 'say'].includes(value.synthesis.kind as string) || typeof value.synthesis.voice !== 'string' || !/^[A-Za-z][A-Za-z0-9 _-]{0,79}$/.test(value.synthesis.voice) + || !Number.isSafeInteger(value.timeoutMs) || (value.timeoutMs as number) < 1 || (value.timeoutMs as number) > 120000) throw new AppError('invalid_voice_config'); +} + +async function verifyArtifact(value: VoiceArtifact, executable: boolean, signal: AbortSignal, health?: Map): Promise { + if (await realpath(value.path) !== value.path) reject('voice_artifact_changed'); + const handle = await open(value.path, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await handle.stat(); + if (!stat.isFile() || stat.size < 1 || stat.size > (executable ? 256 * 1024 * 1024 : 1024 * 1024 * 1024) || stat.mode & 0o022 || executable && !(stat.mode & 0o111)) reject('voice_artifact_changed'); + const key = value.path + '\n' + value.sha256, fingerprint = [stat.dev, stat.ino, stat.size, stat.mode, stat.mtimeMs, stat.ctimeMs].join(':'); + if (health?.get(key) === fingerprint) return; + const hash = createHash('sha256'), buffer = Buffer.alloc(65536); let total = 0; + for (;;) { signal.throwIfAborted(); const read = await handle.read(buffer, 0, buffer.length, null); if (!read.bytesRead) break; total += read.bytesRead; if (total > stat.size) reject('voice_artifact_changed'); hash.update(buffer.subarray(0, read.bytesRead)); } + if (total !== stat.size || hash.digest('hex') !== value.sha256) reject('voice_artifact_changed'); + health?.set(key, fingerprint); + } finally { await handle.close(); } +} + +/** Validate bounded PCM WAV output, allowing the ordinary padded RIFF chunks emitted by local synthesizers. */ +export function speechWavInfo(bytes: Uint8Array): { durationMs: number } { + const invalid = () => reject('voice_audio_invalid'); + if (bytes.length < 46 || bytes.length > localSpeechLimits.audioBytes) return invalid(); + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength), tag = (at: number) => String.fromCharCode(...bytes.subarray(at, at + 4)); + if (tag(0) !== 'RIFF' || tag(8) !== 'WAVE' || view.getUint32(4, true) !== bytes.length - 8) return invalid(); + let at = 12, format: { rate: number; alignment: number } | undefined, size: number | undefined, chunks = 0; + while (at < bytes.length) { + if (++chunks > 32 || at + 8 > bytes.length) return invalid(); + const kind = tag(at), length = view.getUint32(at + 4, true), start = at + 8, end = start + length; + if (end > bytes.length) return invalid(); + if (kind === 'fmt ') { + if (format || length !== 16 && length !== 18 || view.getUint16(start, true) !== 1 || view.getUint16(start + 2, true) !== 1 || view.getUint16(start + 14, true) !== 16) return invalid(); + const rate = view.getUint32(start + 4, true), alignment = view.getUint16(start + 12, true); + if (rate < 8000 || rate > 48000 || alignment !== 2 || view.getUint32(start + 8, true) !== rate * alignment || length === 18 && view.getUint16(start + 16, true) !== 0) return invalid(); + format = { rate, alignment }; + } else if (kind === 'data') { if (size !== undefined || !length || length % 2) return invalid(); size = length; } + else if (!['JUNK', 'FLLR', 'LIST', 'fact'].includes(kind)) return invalid(); + at = end + length % 2; + } + if (at !== bytes.length || !format || !size) return invalid(); + const durationMs = size / format.alignment / format.rate * 1000; + if (durationMs > localSpeechLimits.durationMs) reject('voice_audio_limit'); + return { durationMs }; +} + +/** Open an opt-in protected-host adapter; no process/model download or speech starts while checking availability. */ +export async function openLocalVoice(configured: LocalVoiceConfig, scratchPath: string): Promise { + validateLocalVoiceConfig(configured); const config = structuredClone(configured); + if (!isAbsolute(scratchPath) || resolve(scratchPath) !== scratchPath) throw new AppError('invalid_voice_config'); + await mkdir(scratchPath, { recursive: true, mode: 0o700 }); + if (await realpath(scratchPath) !== scratchPath || (await lstat(scratchPath)).mode & 0o077) throw new AppError('invalid_voice_config'); + const shutdown = new AbortController(), jobs = new Set>(); let closed = false; + let recovered = false, recovery: Promise | null = null; + const recover = (): Promise => { + if (recovered) return Promise.resolve(true); + return recovery ??= (async () => { + for (const name of await readdir(scratchPath)) { + if (!/^localvoice-[A-Za-z0-9]+$/.test(name)) reject('voice_recovery_pending'); + const directory = join(scratchPath, name), stat = await lstat(directory); + if (!stat.isDirectory() || stat.isSymbolicLink() || stat.mode & 0o077) reject('voice_recovery_pending'); + let intent: { state?: string; worker?: number } | undefined; + try { const data = await boundedFile(join(directory, 'intent.json'), 256); intent = JSON.parse(data.toString('utf8')); } + catch (error) { if ((error as NodeJS.ErrnoException).code !== 'ENOENT') return false; } + if (intent && intent.state !== 'preparing') { + try { + const data = await boundedFile(join(directory, 'settled.json'), 512); + const settled = JSON.parse(data.toString('utf8')) as { state?: string; worker?: number }; + if (intent.state !== 'worker' || !Number.isSafeInteger(intent.worker) || settled.worker !== intent.worker || !['completed', 'cancelled', 'failed', 'timeout', 'limit'].includes(settled.state ?? '')) return false; + } catch { return false; } + } + // No intent (or preparing) means no command could have been supplied. Otherwise only the + // supervisor's receipt, written after actually joining its child, permits scratch deletion. + await rm(directory, { recursive: true, force: true }); + } + recovered = true; return true; + })().finally(() => { recovery = null; }); + }; + const health = new Map(); + const verify = async (signal: AbortSignal, cache?: Map) => { if (config.synthesis.kind === 'say' && process.platform !== 'darwin') reject('voice_engine_unavailable'); await verifyArtifact(config.transcription.executable, true, signal, cache); await verifyArtifact(config.transcription.model, false, signal, cache); await verifyArtifact(config.synthesis.executable, true, signal, cache); }; + const job = (context: VoiceProcessContext, body: (directory: string, context: VoiceProcessContext) => Promise): Promise => { + const signal = AbortSignal.any([context.signal, shutdown.signal]), check = () => { if (closed || signal.aborted) reject('voice_cancelled'); context.guard(); }; + const running = (async () => { + let directory: string | undefined; + try { check(); if (!await recover()) reject('voice_recovery_pending'); await verify(signal); check(); directory = await mkdtemp(join(scratchPath, 'localvoice-')); await chmod(directory, 0o700); check(); return await body(directory, { ...context, signal, guard: check }); } + catch (error) { if (signal.aborted) return reject('voice_cancelled'); if (error instanceof AppError) throw error; return reject('voice_engine_unavailable'); } + finally { if (directory) await rm(directory, { recursive: true, force: true }); } + })(); + jobs.add(running); void running.then(() => jobs.delete(running), () => jobs.delete(running)); return running; + }; + const execute = async (argv: string[], directory: string, output: string, bytes: number, context: VoiceProcessContext) => { + await writeFile(join(directory, 'intent.json'), JSON.stringify({ state: 'preparing' }), { mode: 0o600 }); + context.guard(); + const worker = Bun.spawn([process.execPath, new URL('./local-worker.ts', import.meta.url).pathname, '--local-voice-worker'], { cwd: directory, stdin: 'pipe', stdout: 'pipe', stderr: 'ignore', env: { PATH: '/usr/bin:/bin', LANG: 'C.UTF-8', LC_ALL: 'C.UTF-8', TMPDIR: directory } }); + const stop = () => { try { worker.stdin.end(); } catch { /* The worker may have exited already. */ } }; + context.signal.addEventListener('abort', stop, { once: true }); + try { + writeFileSync(join(directory, 'intent.json'), JSON.stringify({ state: 'worker', worker: worker.pid }), { mode: 0o600 }); + context.guard(); context.started(); context.guard(); + worker.stdin.write(JSON.stringify({ argv, cwd: directory, timeoutMs: config.timeoutMs, output, bytes }) + '\n'); + const [code, raw] = await Promise.all([worker.exited, new Response(worker.stdout).text()]); + context.guard(); + const state = code === 0 && raw.length < 256 ? (JSON.parse(raw) as { state?: string }).state : 'failed'; + if (state !== 'completed') reject(state === 'timeout' ? 'voice_process_timeout' : state === 'limit' ? 'voice_audio_limit' : state === 'cancelled' ? 'voice_cancelled' : 'voice_engine_unavailable'); + } finally { context.signal.removeEventListener('abort', stop); stop(); await worker.exited; } + }; + return { + // Health reads can reuse unchanged file metadata. Every actual speech job still rehashes all pinned artifacts. + async available() { if (closed) return false; try { if (!await recover()) return false; await verify(shutdown.signal, health); return !closed; } catch { return false; } }, + transcribe(input, context) { + const wav = input.slice(); captureWavInfo(wav); + return job(context, async (directory, current) => { + const source = join(directory, 'input.wav'), prefix = join(directory, 'transcript'), output = prefix + '.json'; + await writeFile(source, wav, { mode: 0o600 }); current.guard(); + await execute([config.transcription.executable.path, '--model', config.transcription.model.path, '--file', source, '--language', config.transcription.language, '--no-gpu', '--threads', '2', '--processors', '1', '--output-json', '--output-file', prefix, '--no-prints', '--no-timestamps'], directory, output, 131072, current); + const parsed = JSON.parse((await boundedFile(output, 131072)).toString('utf8')) as { transcription?: { text?: unknown }[] }; + const rows = parsed.transcription; + if (!Array.isArray(rows) || rows.length > 256 || rows.some(row => !row || typeof row.text !== 'string')) return reject('voice_transcript_invalid'); + const text = rows.map(row => row.text).join('').trim(); + if (!text || !text.isWellFormed() || Buffer.byteLength(text) > localSpeechLimits.transcriptBytes || /[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]/.test(text)) reject('voice_transcript_invalid'); + current.guard(); return text; + }); + }, + synthesize(text, context) { + if (typeof text !== 'string' || !text.trim() || !text.isWellFormed() || text.length > localSpeechLimits.textCharacters || Buffer.byteLength(text) > localSpeechLimits.textBytes || /[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]/.test(text)) throw new AppError('voice_text_limit'); + if (config.synthesis.kind === 'say' && text.includes('[[')) throw new AppError('voice_text_unsupported'); + return job(context, async (directory, current) => { + const input = join(directory, 'reply.txt'), output = join(directory, 'speech.wav'); await writeFile(input, text, { mode: 0o600 }); current.guard(); + const argv = config.synthesis.kind === 'say' ? ['-v', config.synthesis.voice, '-r', '165', '--file-format=WAVE', '--data-format=LEI16@16000', '--channels=1', '-f', input, '-o', output] : ['-v', config.synthesis.voice, '-s', '165', '-f', input, '-w', output]; + await execute([config.synthesis.executable.path, ...argv], directory, output, localSpeechLimits.audioBytes, current); + const wav = await boundedFile(output, localSpeechLimits.audioBytes), info = speechWavInfo(wav); current.guard(); return { wav: Uint8Array.from(wav), ...info }; + }); + }, + async close() { if (!closed) { closed = true; shutdown.abort(); } await Promise.allSettled([...jobs]); }, + }; +} diff --git a/src/s16/voice/service.ts b/src/s16/voice/service.ts new file mode 100644 index 00000000..8e2acfab --- /dev/null +++ b/src/s16/voice/service.ts @@ -0,0 +1,206 @@ +import { createHash } from 'node:crypto'; +import { AppError } from '../../errors'; +import { DurableJournal } from '../../s15/durable-journal'; +import { canonical, exact, nat } from '../../s15/policy/validation'; +import { identity } from '../../s15/conversation/validation'; +import type { LocalVoiceEngine, VoiceProcessContext } from './local'; +import { captureWavInfo } from './wav'; +import { voiceLimits, type VoiceCapture, type VoiceCaptureCommand, type VoicePlayback, type VoicePlaybackCommand, type VoiceQuotaConfig, type VoiceStatus, type VoiceTranscript, type VoiceUsage } from './contracts'; + +/** Authenticated ingress reacquired after every wait. This host closure is never serialized or offered as a tool. */ +export type VoiceOwner = () => Actor; +/** Existing owning projections supply conversation access and the current guarded result, never cached client text. */ +export interface VoiceServiceOptions { + path: string; installationId: string; create: boolean; limits: VoiceQuotaConfig; + engine(): Promise; + requireOwner(actor: Actor): void; ownerGeneration(): number; + conversation(actor: Actor, id: string): void; + reply(actor: Actor, conversationId: string, requestId: string): { revision: number; text: string }; + now?: () => number; +} +interface Operation { + id: string; kind: 'capture' | 'playback'; conversationId: string; generation: number; commandHash: string; + state: VoiceCapture['state']; createdAt: number; expiresAt: number; day: string; boundMs: number; + reservedMs: number; chargedMs: number; conservative: boolean; started: boolean; inputHash: string | null; inputMs: number | null; +} +interface Active { abort: AbortController; promise: Promise } +const fail = (code: string, status = 409): never => { throw new AppError(code, status); }; +const sha = (value: string | Uint8Array) => createHash('sha256').update(value).digest('hex'); +const day = (at: number) => new Date(at).toISOString().slice(0, 10); +const stamp = (id: string): number => { const match = /^v_(0|[1-9][0-9]{0,15})_[a-f0-9]{32}$/.exec(id); if (!match || !Number.isSafeInteger(Number(match[1]))) return fail('voice_request_invalid', 400); return Number(match[1]); }; + +/** Validate only explicit finite local quotas. Configuration never creates provider/account/billing authority. */ +export function validateVoiceQuota(value: unknown): asserts value is VoiceQuotaConfig { + exact(value, ['dailyMs', 'captureMs']); nat(value.dailyMs, true); nat(value.captureMs, true); + if (value.dailyMs > 3600000 || value.captureMs > voiceLimits.captureMs || value.captureMs > value.dailyMs) fail('invalid_voice_config', 400); +} + +/** Protected owner-only voice journal. It admits no CHAT work and exposes no actor, approval or provider tool. */ +export class VoiceService { + private readonly active = new Map(); + private readonly results = new Map(); + private closed = false; + private closing: Promise | null = null; + private constructor(readonly options: VoiceServiceOptions, private readonly journal: DurableJournal, private readonly engine: LocalVoiceEngine) {} + + /** Acquire the exclusive journal lease before opening local scratch/engine, then retain interrupted usage without replay. */ + static async open(options: VoiceServiceOptions): Promise> { + validateVoiceQuota(options.limits); + options = { ...options, limits: { ...options.limits } }; + const journal = new DurableJournal(options.path, options.installationId, options.create, 0x4a565631); + let engine: LocalVoiceEngine | undefined; + try { + if (options.create) journal.db.exec('CREATE TABLE voice_operations (id TEXT PRIMARY KEY, created_at INTEGER NOT NULL, record_json TEXT NOT NULL)'); + engine = await options.engine(); const service = new VoiceService(options, journal, engine); + journal.tx(() => { + for (const record of service.all()) if (['capturing', 'processing'].includes(record.state)) { record.state = record.started ? 'unknown' : 'expired'; record.reservedMs = 0; service.save(record); } + }); + return service; + } catch (error) { await engine?.close(); journal.close(); throw error; } + } + private all(): Operation[] { return this.journal.db.query<{ record_json: string }, []>('SELECT record_json FROM voice_operations ORDER BY rowid').all().map(row => JSON.parse(row.record_json) as Operation); } + private get(id: string): Operation { stamp(id); const row = this.journal.db.query<{ record_json: string }, [string]>('SELECT record_json FROM voice_operations WHERE id=?').get(id); if (!row) return fail('voice_not_found', 404); return JSON.parse(row.record_json) as Operation; } + private find(id: string): Operation | null { stamp(id); const row = this.journal.db.query<{ record_json: string }, [string]>('SELECT record_json FROM voice_operations WHERE id=?').get(id); return row ? JSON.parse(row.record_json) as Operation : null; } + private save(record: Operation): void { const text = canonical(record); if (Buffer.byteLength(text) > 4096) fail('voice_storage_limit'); this.journal.db.query('INSERT INTO voice_operations VALUES(?,?,?) ON CONFLICT(id) DO UPDATE SET record_json=excluded.record_json').run(record.id, record.createdAt, text); } + private now(): number { + const wall = (this.options.now ?? Date.now)(); if (!Number.isSafeInteger(wall) || wall < 0 || wall > 8640000000000000) return fail('voice_time_invalid'); + const clock = this.journal.db.query<{ json: string }, []>("SELECT json FROM records WHERE id='voice_clock'").get(); + return Math.max(wall, clock ? (JSON.parse(clock.json) as { at: number }).at : 0); + } + private clock(): number { const at = this.now(); this.journal.put('voice_clock', { at }); return at; } + private owner(owner: VoiceOwner, record?: Operation): Actor { + if (this.closed) return fail('voice_unavailable', 503); + const actor = owner(); this.options.requireOwner(actor); + if (record) { if (record.generation !== this.options.ownerGeneration()) return fail('voice_stale', 403); this.options.conversation(actor, record.conversationId); } + return actor; + } + private prune(at: number): void { + // Expired recording reservations sent no CLI command. Processing capacity is released only by its real completion. + for (const record of this.all()) if (record.state === 'capturing' && record.expiresAt <= at) { record.state = 'expired'; record.reservedMs = 0; this.save(record); } + this.journal.db.query("DELETE FROM voice_operations WHERE created_at row.day === today); + return { day: today, unit: 'local_audio_ms', limitMs: this.options.limits.dailyMs, chargedMs: rows.reduce((n, row) => n + row.chargedMs, 0), reservedMs: rows.reduce((n, row) => n + (row.state === 'capturing' && row.expiresAt <= at ? 0 : row.reservedMs), 0), conservativeMs: rows.reduce((n, row) => n + (row.conservative ? row.chargedMs : 0), 0) }; + } + private capture(record: Operation): VoiceCapture { return { id: record.id, conversationId: record.conversationId, state: record.state, recordingEndsAt: record.kind === 'capture' ? record.expiresAt - voiceLimits.submissionMs : record.expiresAt, expiresAt: record.expiresAt, maxDurationMs: record.boundMs }; } + private cached(id: string, owner: VoiceOwner): T { + const record = this.get(id); this.owner(owner, record); + const result = this.results.get(id); if (!result || result.until <= this.now()) return fail(record.state === 'unknown' ? 'voice_outcome_unknown' : 'voice_result_expired'); + return { ...structuredClone(result.value), usage: this.usageAt(this.now()) } as T; + } + private cache(id: string, value: VoiceTranscript | VoicePlayback): void { + const bytes = Buffer.byteLength(JSON.stringify(value)); if (bytes > voiceLimits.responseCacheBytes) fail('voice_audio_limit'); + this.results.set(id, { value: structuredClone(value), bytes, until: this.now() + voiceLimits.responseCacheMs }); + let total = [...this.results.values()].reduce((n, entry) => n + entry.bytes, 0); + for (const [key, entry] of this.results) { if (total <= voiceLimits.responseCacheBytes && this.results.size <= 16) break; this.results.delete(key); total -= entry.bytes; } + } + private fresh(id: string, at: number): void { const issued = stamp(id); if (issued > at + 30000 || issued < at - voiceLimits.requestAgeMs) fail('voice_request_expired'); } + private capacity(at: number, upper: number): void { + this.prune(at); + if (this.all().some(row => ['capturing', 'processing'].includes(row.state))) fail('voice_busy'); + if (this.all().length >= voiceLimits.operations) fail('voice_storage_limit'); + const usage = this.usageAt(at); if (upper > usage.limitMs - usage.chargedMs - usage.reservedMs) fail('voice_budget_exhausted'); + } + + /** Metadata-only status; health inspection never starts capture, speech or Bot work. */ + async status(owner: VoiceOwner): Promise { + this.owner(owner); const available = await this.engine.available(); this.owner(owner); + const at = this.now(); return { contract: 's16-voice/v1', available, serverTime: at, route: 'local', limits: { ...this.options.limits }, usage: this.usageAt(at) }; + } + /** Reserve a bounded draft. Retried/expired identities cannot silently create another capture. */ + async begin(owner: VoiceOwner, conversationId: string, input: VoiceCaptureCommand, signal?: AbortSignal): Promise { + if (signal?.aborted) fail('voice_cancelled'); + exact(input, ['contract', 'requestId']); if (input.contract !== 's16-voice-capture/v1') fail('voice_request_invalid', 400); identity(conversationId); stamp(input.requestId); + const command = structuredClone(input), hash = sha(canonical({ ...command, conversationId })); + this.options.conversation(this.owner(owner), conversationId); + if (!await this.engine.available()) fail('voice_unavailable', 503); + return this.journal.tx(() => { + if (signal?.aborted) fail('voice_cancelled'); + this.options.conversation(this.owner(owner), conversationId); const at = this.clock(), old = this.find(command.requestId); + if (old) { if (old.commandHash !== hash) fail('request_id_conflict'); this.owner(owner, old); this.prune(at); return this.capture(this.get(old.id)); } + this.fresh(command.requestId, at); this.capacity(at, this.options.limits.captureMs); + const record: Operation = { id: command.requestId, kind: 'capture', conversationId, generation: this.options.ownerGeneration(), commandHash: hash, state: 'capturing', createdAt: at, expiresAt: at + this.options.limits.captureMs + voiceLimits.permissionMs + voiceLimits.submissionMs, day: day(at), boundMs: this.options.limits.captureMs, reservedMs: this.options.limits.captureMs, chargedMs: 0, conservative: false, started: false, inputHash: null, inputMs: null }; + this.save(record); return this.capture(record); + }); + } + /** Inspect current capture metadata without exposing its transcript/audio or admitting new work. */ + inspect(owner: VoiceOwner, id: string): VoiceCapture { this.owner(owner); const record = this.get(id); this.owner(owner, record); return { ...this.capture(record), state: record.state === 'capturing' && record.expiresAt <= this.now() ? 'expired' : record.state }; } + + /** Final synchronous draft release, after every transport wait. A transcript is never an approval or message. */ + publishTranscript(owner: VoiceOwner, id: string, publish: (value: VoiceTranscript) => T): T { + this.owner(owner); if (this.get(id).kind !== 'capture') fail('voice_request_invalid'); + return publish(this.cached(id, owner)); + } + /** Reacquire current reply/source permission in the same synchronous turn that the owner transport serializes audio. */ + publishPlayback(owner: VoiceOwner, command: VoicePlaybackCommand, publish: (value: VoicePlayback) => T): T { + this.owner(owner); const record = this.get(command.requestId); this.owner(owner, record); + if (record.kind !== 'playback' || record.commandHash !== sha(canonical(command))) fail('request_id_conflict'); + const reply = this.options.reply(this.owner(owner), command.conversationId, command.targetRequestId); + if (reply.revision !== command.targetRevision || sha(reply.text) !== command.textSha256) fail('voice_result_changed'); + return publish(this.cached(record.id, owner)); + } + + private run(owner: VoiceOwner, record: Operation, signal: AbortSignal, body: (context: VoiceProcessContext) => Promise, finalGuard?: () => void): Promise { + const abort = new AbortController(), stop = () => abort.abort(); signal.addEventListener('abort', stop, { once: true }); if (signal.aborted) stop(); + const guard = () => { if (abort.signal.aborted) fail('voice_cancelled'); const current = this.get(record.id); this.owner(owner, current); if (current.state !== 'processing') fail('voice_cancelled'); }; + const promise = Promise.resolve().then(async () => { + try { + guard(); const result = await body({ signal: abort.signal, guard, started: () => this.journal.tx(() => { guard(); const current = this.get(record.id); if (current.started) fail('voice_process_replay'); current.started = true; current.reservedMs = 0; current.chargedMs = current.kind === 'capture' ? current.inputMs! : current.boundMs; current.conservative = current.kind === 'playback'; this.save(current); }) }); + if (abort.signal.aborted) fail('voice_cancelled'); + const current = this.get(record.id); this.owner(owner, current); if (current.state !== 'ready') fail('voice_cancelled'); + finalGuard?.(); this.cache(record.id, result); return structuredClone(result); + } catch (error) { + this.journal.tx(() => { const current = this.get(record.id); current.state = abort.signal.aborted ? 'cancelled' : current.started ? 'unknown' : 'failed'; current.reservedMs = 0; this.save(current); }); + this.results.delete(record.id); throw error; + } finally { signal.removeEventListener('abort', stop); this.active.delete(record.id); } + }); + this.active.set(record.id, { abort, promise }); return promise; + } + + /** Strict raw WAV is snapshotted/bound once; cancellation or a lost result never replays transcription. */ + transcribe(owner: VoiceOwner, id: string, input: Uint8Array, signal: AbortSignal): Promise { + this.owner(owner); + const wav = input.slice(), info = captureWavInfo(wav), hash = sha(wav); let record = this.get(id); this.owner(owner, record); + if (record.kind !== 'capture' || info.durationMs > record.boundMs) return fail('voice_audio_limit', 400); + if (record.inputHash && record.inputHash !== hash) return fail('request_id_conflict'); + const active = this.active.get(id); if (active) return active.promise.then(() => this.cached(id, owner)); + if (record.state !== 'capturing') return Promise.resolve(this.cached(id, owner)); + record = this.journal.tx(() => { const current = this.get(id), at = this.clock(); this.owner(owner, current); if (at >= current.expiresAt) fail('voice_capture_expired'); current.state = 'processing'; current.inputHash = hash; current.inputMs = Math.ceil(info.durationMs); this.save(current); return current; }); + return this.run(owner, record, signal, async context => { + const text = await this.engine.transcribe(wav, context); context.guard(); + return this.journal.tx(() => { context.guard(); const current = this.get(id); current.state = 'ready'; current.reservedMs = 0; current.chargedMs = current.inputMs!; current.conservative = false; this.save(current); return { contract: 's16-voice-transcript/v1', capture: this.capture(current), text, usage: this.usageAt(this.clock()) }; }); + }); + } + /** Reacquire exact current result before processing and before publishing bounded local reply audio. */ + async playback(owner: VoiceOwner, input: VoicePlaybackCommand, signal: AbortSignal): Promise { + exact(input, ['contract', 'requestId', 'conversationId', 'targetRequestId', 'targetRevision', 'textSha256']); + if (input.contract !== 's16-voice-playback/v1' || !/^[a-f0-9]{64}$/.test(input.textSha256)) fail('voice_request_invalid', 400); + identity(input.conversationId); identity(input.targetRequestId); nat(input.targetRevision, true); stamp(input.requestId); + const command = structuredClone(input), hash = sha(canonical(command)); + const reply = () => { const actor = this.owner(owner), value = this.options.reply(actor, command.conversationId, command.targetRequestId); if (value.revision !== command.targetRevision || sha(value.text) !== command.textSha256) fail('voice_result_changed'); return value.text; }; + const text = reply(), existing = this.find(command.requestId); + if (existing) { if (existing.commandHash !== hash) fail('request_id_conflict'); const active = this.active.get(existing.id); if (active) await active.promise; reply(); return this.cached(existing.id, owner); } + if (!await this.engine.available()) fail('voice_unavailable', 503); + const admitted = this.journal.tx(() => { + const old = this.find(command.requestId); + if (old) { if (old.commandHash !== hash) fail('request_id_conflict'); return { record: old, existing: true }; } + reply(); const at = this.clock(); this.fresh(command.requestId, at); this.capacity(at, voiceLimits.synthesisMs); + const value: Operation = { id: command.requestId, kind: 'playback', conversationId: command.conversationId, generation: this.options.ownerGeneration(), commandHash: hash, state: 'processing', createdAt: at, expiresAt: at + 120000, day: day(at), boundMs: voiceLimits.synthesisMs, reservedMs: voiceLimits.synthesisMs, chargedMs: 0, conservative: false, started: false, inputHash: command.textSha256, inputMs: null }; this.save(value); return { record: value, existing: false }; + }); + const record = admitted.record; + if (admitted.existing) { const active = this.active.get(record.id); if (active) await active.promise; reply(); return this.cached(record.id, owner); } + return this.run(owner, record, signal, async context => { + const audio = await this.engine.synthesize(text, { ...context, guard() { context.guard(); reply(); } }); context.guard(); reply(); + return this.journal.tx(() => { context.guard(); reply(); const current = this.get(record.id); current.state = 'ready'; current.reservedMs = 0; current.chargedMs = Math.ceil(audio.durationMs); current.conservative = false; this.save(current); return { contract: 's16-voice-playback/v1', id: current.id, conversationId: command.conversationId, targetRequestId: command.targetRequestId, wavBase64: Buffer.from(audio.wav).toString('base64'), durationMs: Math.ceil(audio.durationMs), usage: this.usageAt(this.clock()) }; }); + }, reply); + } + /** Cancel fences private drafts/playback only; it never stops or approves underlying CHAT work. */ + async cancel(owner: VoiceOwner, id: string): Promise { + this.owner(owner); this.owner(owner, this.get(id)); const active = this.active.get(id); active?.abort.abort(); if (active) await active.promise.catch(() => {}); + return this.journal.tx(() => { this.owner(owner); this.clock(); const record = this.get(id); this.owner(owner, record); record.state = 'cancelled'; record.reservedMs = 0; this.results.delete(id); this.save(record); return this.capture(record); }); + } + /** Native engine closure joins all children before journal/capacity custody closes. */ + close(): Promise { return this.closing ??= (async () => { this.closed = true; for (const active of this.active.values()) active.abort.abort(); await this.engine.close(); await Promise.allSettled([...this.active.values()].map(value => value.promise)); this.results.clear(); this.journal.close(); })(); } +} diff --git a/src/s16/voice/wav.ts b/src/s16/voice/wav.ts new file mode 100644 index 00000000..59e11e08 --- /dev/null +++ b/src/s16/voice/wav.ts @@ -0,0 +1,43 @@ +/** Canonical capture format: mono PCM16 at 16 kHz, bounded independently of client MIME/duration. */ +export const captureAudio = Object.freeze({ sampleRate: 16000, channels: 1, bits: 16, durationMs: 30000, bytes: 960044 }); + +/** Safe public failure code; never carries audio, transcript or private host paths. */ +export class VoiceAudioError extends Error { + constructor(readonly code: 'voice_audio_invalid' | 'voice_audio_limit') { super(code); } +} + +const tag = (bytes: Uint8Array, offset: number, value: string): boolean => [...value].every((letter, index) => bytes[offset + index] === letter.charCodeAt(0)); +const invalid = (): never => { throw new VoiceAudioError('voice_audio_invalid'); }; +const limit = (): never => { throw new VoiceAudioError('voice_audio_limit'); }; + +/** + * Validate the complete canonical WAV before a local engine sees bytes. Rejects other codecs/chunks, + * false lengths, alignment, trailing data and excess samples. Metadata is derived from actual bytes; + * the caller still owns snapshotting its input before an async boundary. + */ +export function captureWavInfo(bytes: Uint8Array, maxDurationMs: number = captureAudio.durationMs): { frames: number; durationMs: number; bytes: number } { + if (!Number.isSafeInteger(maxDurationMs) || maxDurationMs < 1 || maxDurationMs > captureAudio.durationMs) invalid(); + if (bytes.byteLength > captureAudio.bytes || bytes.byteLength > 44 + Math.floor(maxDurationMs * captureAudio.sampleRate / 1000) * 2) limit(); + if (bytes.byteLength < 46 || !tag(bytes, 0, 'RIFF') || !tag(bytes, 8, 'WAVE') || !tag(bytes, 12, 'fmt ') || !tag(bytes, 36, 'data')) invalid(); + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength), size = bytes.byteLength - 44; + if (view.getUint32(4, true) !== bytes.byteLength - 8 || view.getUint32(16, true) !== 16 || view.getUint16(20, true) !== 1 + || view.getUint16(22, true) !== 1 || view.getUint32(24, true) !== captureAudio.sampleRate || view.getUint32(28, true) !== 32000 + || view.getUint16(32, true) !== 2 || view.getUint16(34, true) !== 16 || view.getUint32(40, true) !== size || size % 2 !== 0) invalid(); + return { frames: size / 2, durationMs: size / 2 * 1000 / captureAudio.sampleRate, bytes: bytes.byteLength }; +} + +/** Encode already resampled mono 16 kHz capture. Finite samples are clipped to PCM16; no codec/dependency is used. */ +export function encodeCaptureWav(samples: Float32Array): Uint8Array { + if (samples.length > captureAudio.sampleRate * captureAudio.durationMs / 1000) limit(); + if (!samples.length) invalid(); + const bytes = new Uint8Array(44 + samples.length * 2), view = new DataView(bytes.buffer); + const write = (offset: number, text: string) => { for (let index = 0; index < text.length; index++) bytes[offset + index] = text.charCodeAt(index); }; + write(0, 'RIFF'); view.setUint32(4, bytes.length - 8, true); write(8, 'WAVE'); write(12, 'fmt '); + view.setUint32(16, 16, true); view.setUint16(20, 1, true); view.setUint16(22, 1, true); view.setUint32(24, captureAudio.sampleRate, true); + view.setUint32(28, 32000, true); view.setUint16(32, 2, true); view.setUint16(34, 16, true); write(36, 'data'); view.setUint32(40, samples.length * 2, true); + for (let index = 0; index < samples.length; index++) { + const sample = samples[index]!; if (!Number.isFinite(sample)) invalid(); + const clipped = Math.min(1, Math.max(-1, sample)); view.setInt16(44 + index * 2, Math.round(clipped * (clipped < 0 ? 32768 : 32767)), true); + } + return bytes; +} diff --git a/src/s16/web/VoiceControls.tsx b/src/s16/web/VoiceControls.tsx new file mode 100644 index 00000000..0c944599 --- /dev/null +++ b/src/s16/web/VoiceControls.tsx @@ -0,0 +1,99 @@ +import { useEffect, useId, useLayoutEffect, useRef, useState } from 'react'; +import { JitClient, JitClientError } from '../client'; +import { startVoiceRecording, type VoiceRecording } from '../client/voice-recording'; +import type { VoiceStatus } from '../voice/contracts'; +import type { Copy } from './copy'; + +type Phase = 'idle' | 'reserving' | 'permission' | 'recording' | 'transcribing' | 'draft' | 'loading' | 'playing' | 'error'; +interface Slot { kind: 'capture' | 'playback'; scope: string; conversationId: string; stop: AbortController; id?: string; recorder?: VoiceRecording; audio?: HTMLAudioElement; url?: string; timer?: ReturnType; target?: { requestId: string; revision: number; text: string } } +const code = (error: unknown) => error instanceof JitClientError ? error.code : error instanceof Error && /^[a-z][a-z0-9_]{0,127}$/.test(error.message) ? error.message : 'voice_unavailable'; + +/** One client voice lifecycle shared by the composer and reply buttons; it never sends a transcript or votes. */ +export function useVoice(options: { client: JitClient; enabled: boolean; canCapture: boolean; conversationId: string; scope: string; onTranscript(text: string): void; onFailure(error: unknown): void }) { + const latest = useRef(options); latest.current = options; + const slot = useRef(null), turn = useRef(0), mounted = useRef(true); + const [phase, setPhase] = useState('idle'), [status, setStatus] = useState(null), [error, setError] = useState(null), [elapsed, setElapsed] = useState(0), [draft, setDraft] = useState(''); + const alive = (value: Slot) => mounted.current && slot.current === value && latest.current.enabled && latest.current.scope === value.scope; + const dispose = async (value: Slot | null, remote = true) => { + if (!value) return; + clearInterval(value.timer); delete value.timer; value.stop.abort(); value.audio?.pause(); + if (value.audio) { value.audio.onended = null; value.audio.onerror = null; value.audio.removeAttribute('src'); value.audio.load(); } + if (value.url) URL.revokeObjectURL(value.url); + await Promise.allSettled([value.recorder?.cancel(), remote && value.id ? latest.current.client.cancelVoice({ id: value.id, conversationId: value.conversationId }) : undefined]); + }; + const cancel = () => { turn.current++; const previous = slot.current; slot.current = null; setDraft(''); setElapsed(0); setError(null); setPhase('idle'); void dispose(previous); }; + const failed = async (value: Slot, failure: unknown) => { + if (!alive(value)) return; + slot.current = null; setDraft(''); setError(code(failure)); setPhase('error'); const cleanup = dispose(value); + if (mounted.current && latest.current.scope === value.scope && failure instanceof JitClientError && ([401, 403].includes(failure.status ?? 0) || ['voice_result_changed', 'voice_result_unavailable'].includes(failure.code))) latest.current.onFailure(failure); + await cleanup; + }; + const inspect = async (value: Slot) => { const result = await latest.current.client.voiceStatus(value.stop.signal); if (!alive(value)) throw Error('voice_cancelled'); setStatus(result); if (!result.available) throw Error('voice_unavailable'); return result; }; + const monitor = (value: Slot) => { let checking = false; const timer = value.timer = setInterval(() => { if (!alive(value) || checking) return; checking = true; void inspect(value).catch(error => { if (value.timer === timer || error instanceof JitClientError && [401, 403].includes(error.status ?? 0)) return failed(value, error); }).finally(() => { checking = false; }); }, 2000); }; + const begin = async (kind: Slot['kind']): Promise => { + const sequence = ++turn.current, scope = latest.current.scope, previous = slot.current; slot.current = null; + setDraft(''); setError(null); setElapsed(0); setPhase(kind === 'capture' ? 'reserving' : 'loading'); await dispose(previous); + if (!mounted.current || turn.current !== sequence || latest.current.scope !== scope || !latest.current.enabled || kind === 'capture' && !latest.current.canCapture) return null; + const value: Slot = { kind, scope, conversationId: latest.current.conversationId, stop: new AbortController() }; slot.current = value; return value; + }; + const record = async () => { + const value = await begin('capture'); if (!value) return; + try { + const requested = performance.now(), status = await inspect(value), command = latest.current.client.prepareDictation(status.serverTime); value.id = command.requestId; + const capture = await latest.current.client.beginDictation(value.conversationId, command, value.stop.signal); + if (!alive(value)) return; if (capture.state !== 'capturing') throw Error('voice_capture_expired'); + setPhase('permission'); monitor(value); + const recorder = await startVoiceRecording({ signal: value.stop.signal, durationMs: capture.maxDurationMs, deadline: requested + capture.recordingEndsAt - status.serverTime, progress(ms) { if (alive(value)) setElapsed(old => Math.floor(old / 1000) === Math.floor(ms / 1000) ? old : ms); } }); + value.recorder = recorder; + if (!alive(value)) { await recorder.cancel(); return; } + setPhase('recording'); const wav = await recorder.done; + delete value.recorder; + if (!alive(value)) return; clearInterval(value.timer); delete value.timer; setPhase('transcribing'); + const transcript = await latest.current.client.transcribeDictation(capture, wav, value.stop.signal); + if (!alive(value)) return; setDraft(transcript.text); setPhase('draft'); setStatus(old => old ? { ...old, usage: transcript.usage } : old); + void latest.current.client.voiceStatus(value.stop.signal).then(fresh => { if (alive(value)) setStatus(fresh); }).catch(error => { if (!alive(value)) return; if (error instanceof JitClientError && [401, 403].includes(error.status ?? 0)) void failed(value, error); else setError(code(error)); }); + } catch (error) { await failed(value, error); } + }; + const play = async (target: { requestId: string; revision: number; text: string }) => { + const value = await begin('playback'); if (!value) return; value.target = { ...target }; + try { + const status = await inspect(value); monitor(value); + const command = await latest.current.client.preparePlayback(status.serverTime, value.conversationId, target.requestId, target.revision, target.text); value.id = command.requestId; + const result = await latest.current.client.playReply(command, value.stop.signal); if (!alive(value)) return; + const bytes = Uint8Array.from(atob(result.wavBase64), character => character.charCodeAt(0)); + value.url = URL.createObjectURL(new Blob([bytes], { type: 'audio/wav' })); value.audio = new Audio(value.url); + value.audio.onended = () => { if (!alive(value)) return; slot.current = null; setPhase('idle'); void dispose(value, false); }; + value.audio.onerror = () => { void failed(value, Error('voice_playback_unavailable')); }; + await value.audio.play(); if (!alive(value)) { value.audio.pause(); return; } + setStatus(old => old ? { ...old, usage: result.usage } : old); setPhase('playing'); + } catch (error) { await failed(value, error); } + }; + const apply = () => { if (!draft.trim() || !slot.current || !alive(slot.current)) return; try { latest.current.onTranscript(draft); cancel(); } catch (error) { setError(code(error)); } }; + useLayoutEffect(() => { + mounted.current = true; let fresh = true; const stop = new AbortController(); + setDraft(''); setError(null); setElapsed(0); setPhase('idle'); setStatus(null); + if (options.enabled) void options.client.voiceStatus(stop.signal).then(status => { if (fresh) setStatus(status); }).catch(error => { if (fresh && !stop.signal.aborted) { setError(code(error)); if (error instanceof JitClientError && [401, 403].includes(error.status ?? 0)) latest.current.onFailure(error); } }); else setStatus(null); + return () => { fresh = false; mounted.current = false; stop.abort(); turn.current++; const previous = slot.current; slot.current = null; void dispose(previous); }; + }, [options.enabled, options.scope, options.client]); + useEffect(() => { if (!options.canCapture && slot.current?.kind === 'capture') cancel(); }, [options.canCapture]); + return { phase, status, error, elapsed, draft, setDraft, record, play, cancel, apply, stop: () => slot.current?.recorder?.stop(), playing: slot.current?.kind === 'playback' ? slot.current.target?.requestId : null, + currentReply: (rows: { requestId: string; revision: number; result: { text: string } | null }[]) => { const current = slot.current; if (current?.kind === 'playback' && current.target && !rows.some(row => row.requestId === current.target!.requestId && row.revision === current.target!.revision && row.result?.text === current.target!.text)) cancel(); } }; +} + +/** Visible recording/transcription/playback states; transcript editing precedes ordinary composer Send. */ +export function VoiceControls({ voice, t, canCapture }: { voice: ReturnType; t: Copy; canCapture: boolean }) { + const transcript = useRef(null), transcriptId = useId(); + useEffect(() => { if (voice.phase === 'draft') transcript.current?.focus(); }, [voice.phase]); + const busy = !['idle', 'error', 'draft'].includes(voice.phase), seconds = Math.floor(voice.elapsed / 1000), usage = voice.status?.usage; + return
+
+ {voice.phase === 'recording' && <>{String(Math.floor(seconds / 60)).padStart(2, '0')}:{String(seconds % 60).padStart(2, '0')}} + {(busy || voice.phase === 'draft') && } + {busy && {t(voice.phase === 'reserving' ? 's16.voice.preparing' : voice.phase === 'permission' ? 's16.voice.permission' : voice.phase === 'recording' ? 's16.voice.recording' : voice.phase === 'transcribing' ? 's16.voice.transcribing' : voice.phase === 'loading' ? 's16.voice.loading' : 's16.voice.playing')}} +
+ {!voice.status ? t('s16.voice.checking') : !voice.status.available ? t('s16.voice.unavailable') : t('s16.voice.local')} + {usage && {t('s16.voice.usage', { used: (usage.chargedMs / 1000).toFixed(1), reserved: (usage.reservedMs / 1000).toFixed(1), limit: (usage.limitMs / 1000).toFixed(0) })}{usage.conservativeMs > 0 && <> {t('s16.voice.conservative', { seconds: (usage.conservativeMs / 1000).toFixed(1) })}}} + {voice.phase === 'draft' &&